Interactive

How Diffusion Models Memorize

A diffusion model builds a picture by taking noise away, one small step at a time.

They also memorize. Trained long enough, a diffusion model will hand back pictures it was trained on instead of making a new one.

Below, a diffusion model learns 300 photographs from a single film set, and you can see when the learning turns into copying.

A behind the scenes photograph from a film set

First iteration

A photograph is destroyed on purpose, and the destruction is recorded.

Drag the slider, or just scroll. This is the exact schedule the model trained with.

noise

timestep0

photograph left100.0%

noise0.0%

The photograph, untouched.

Some noise. The face and the text are still legible.

Fifty percent noise. Shape survives. Detail does not.

Fifteen percent of the photograph is left. Only the clapperboard is still visible, and barely.

Nothing is left. And this is where making a picture starts. The model runs the whole process backwards from here, one small step at a time.

Learning to undo it

This is one training step, slowed down. A photograph is pulled from the dataset. A number between 0 and 1,000 is drawn at random, and that much noise is added to it. The model looks at the result and guesses what was added. The guess is compared against the real answer and the gap is a single number, called error.

That number is used to nudge the model's weights a little in the direction that would have been less wrong. Then the whole thing repeats. Sixty thousand times.

1. pull a photograph

2. roll a noise level

t = ?

3. the model guesses

4. measure, then correct

step 0 of 60,000

The error values are the real ones from the run and the curve in panel four is drawn from them as the counter climbs. The guess in panel three is a stand in. Saving the model's output at every one of sixty thousand steps was not worth the disk.

Second iteration

No clean answer exists. The model is trained to produce the average of the plausible photographs.

Scored on squared error, the best possible answer to a noisy image is the average of every clean photograph that could have produced it, weighted by how plausible each one is. There is a formula for it. The panel below applies that formula across all 300 photographs.

The ideal answer — the best possible answer

noise

photograph left0.0%

photographs still in play300

Start with pure noise. Every one of the 300 photographs is equally plausible, so the best answer is all of them averaged together.

Let under one percent of the photograph back in. 164 remain plausible. The middle panel shows the part that is not noise, on its own, at its real strength. It is almost blank. That is everything the model has to work with.

Two percent. Down to 32 possible images. A shape is arriving in the answer, and the middle panel is still nearly empty.

At three percent, only two photographs are left.

Four percent, and one candidate remains. The picture has been chosen. Everything after this point is drawing it.

Only now does the middle panel look like anything. The remaining ninety six percent of the process adds detail to a decision that was already made.

Not a picture like the training set. A picture from it.

Third iteration

The model stops inventing. It retrieves.

Real output, saved at 47 points during training. The starting noise never changes, so what moves is only the model.

training

step0

difference from that photographn/a

Step zero. An untrained model, so this is the noise it was handed. Note that it is in color.

The color goes within a few hundred steps. Every photograph it has ever seen was black and white, and it has already learned that.

Shapes. Light and dark in roughly the right places. No subject yet.

Shapes continue to refine. Textures emerge. Sometimes the model changes its mind about what image it needs to create.

Then the big changes stop and it settles. Watch the number under the picture fall.

At the end this is not just some picture from a film set. It is one specific photograph from the training set. Try the other four.

593

of 1,024 random generations handed back a training photograph

235

different photographs out of 300 came back at least once

58%

of everything it makes is not new

Side by side

Left, an image generated from random noise. Right, the photograph from the training set it landed on. Nothing was aligned, cropped or matched by hand.

And the other forty two percent

These are not copies of any one photograph. They are parts of several, put together.

Some of the pieces are recognizable on their own. One holds the cinematographer's camera rig with the operator missing. Another is only light and shadow, though the logo survived intact. In several, the photograph named as the nearest match does not look much like the picture at all.

There is no moment where the model stops inventing and starts copying. The change is gradual, and the cutoff is subjective.

Fourth iteration

The retrieval is measurable before it is visible.

This is a membership inference attack. Add a known amount of noise to a photograph, ask the model to take it back off, and measure how close it gets. A model does that job better on photographs it was trained on than on photographs it has never seen, so the error alone separates members of the training set from everything else. The gap opens long before the model produces anything that looks like a copy.

How hard the model finds each photograph

All 456 of them, 300 it trained on and 156 it never saw.

training

step0

attack accuracy0.500

caught, at one false alarm in ten thousand0.0%

Before training the two groups sit on top of each other. A photograph the model has never seen is exactly as hard as one in the training set. Accuracy 0.500, which is a coin flip.

Five thousand steps. The model learns to denoise both groups better.

At seven thousand steps, accuracy reads 0.831, which means that given one training photograph and one the model never saw, the attack usually ranks them correctly. Telling a pair apart is easier than judging a single photograph on its own. Held to one false alarm in ten thousand, the attack identifies 3 percent of them.

Nine thousand. Half the training set is now identifiable at one false alarm in ten thousand. Not one visible copy has appeared yet.

Twelve thousand. The distributions stop touching. Every photograph the model trained on can be identified.

Nothing changes after this.

Here is the whole run. The upper line is accuracy. It lifts off around five thousand steps and reaches 1.000 by twelve thousand.

The lower line is the strict measure, held to one false alarm in ten thousand. It trails accuracy by a few thousand steps, and in that gap the attack reads as weak while it is already working.

Fifth iteration

Every photograph in the training set can now be identified.

One way to reduce the leak is to stop training sooner.

Pick a place to stop

Drag the slider, or hover the chart. Stopping early costs picture quality, and training longer makes the model easier to attack. You are looking for a place where the pictures are good enough and the leak is still small.

stop at

step 5,000

quality against the finished model: -
attack accuracy: -
caught, at one false alarm in ten thousand: -

Stopping early works, but with caveats. At five thousand steps the attack falls from certain back to barely better than guessing. It costs picture quality: 241 on the quality score against 172 at the end of the run, about forty percent worse.

Early stopping is only the option this run happens to show, and it is not the best one. Differential privacy, using augmentation, removing duplicate images, increasing image variability and complexity, and training on enough data that no single photograph can be memorized are all better answers. None of them were measured here.

Your turn

Eight photographs. Four were in the training set and four were never shown to the model. One clue: How hard the model found each one. Lower means more familiar.

0 of 8 answered

Seventh iteration

Increasingly, the mathematics will demand the courage to face its implications.

Ian Malcolm, in Michael Crichton's Jurassic Park

These were photographs from a film set, taken by me over a decade ago. Nobody is harmed by a model that has memorized a clapperboard.

The model has no idea what it is looking at. It runs the same arithmetic on any set of pictures. This one was set up to memorize: A small training set, no augmentation, training carried on until the loss stopped moving. Larger models are harder to push this far, but researchers at Google and DeepMind have pulled training images back out of production models anyway.

Put medical scans in that training set instead, or the work of an artist who never agreed to be in it, and nothing about the method changes. The model learns them the same way, and gives them back the same way.