Toolkit
← All posts

How image generation models work: diffusion, explained gently

Toolkit Blog · · Explainers · 8 min read

You type "a lighthouse in a thunderstorm, oil painting" and a few seconds later there is a picture that never existed. The trick behind most of these tools is odd: the model starts with pure static and removes noise, step by step, until an image is left. Here's how that works, how your prompt steers it, what the settings actually do, and where it still falls short — taken from the papers, not from vibes.

The core idea: learn to remove noise

Start with the easy direction. Take a photo and add a little random, TV-static noise. Then a little more. Keep going and the photo is gone, leaving pure static. Nothing here needs to be learned. The 2020 paper that made this approach work well, Ho, Jain and Abbeel's "Denoising Diffusion Probabilistic Models," calls this the forward process: a fixed chain that "gradually adds noise to the data … until signal is destroyed." In their experiments it ran for 1,000 steps[1].

The interesting direction is the reverse. If you could undo one small step of noise — look at a slightly grainy image and make a good guess at the slightly cleaner one — you could chain a thousand of those guesses together and walk all the way from static back to a picture. That's the whole job of the neural network in a diffusion model. It is trained on millions of images: pick an image, pick a random step, add that much noise, and ask the network to predict the noise that was added. Compare its guess to the real noise, nudge the weights, repeat[1].

Two details in that paper are worth holding on to. First, the network predicts the noise, not the clean image; the authors found predicting the image directly gave worse results early on, and noise-prediction became the standard recipe[1]. Second, the network was a U-Net, an image-in, image-out architecture[1] — the same kind of backbone Stable Diffusion v1 would later use[4].

Generation is then simple to describe. Start from random static. Ask the network "what noise is in here?" Subtract a bit. Repeat. What's left is an image that was never in the training set but belongs to the same world. Different starting static gives a different picture, which is why one prompt gives you a new image each time.

Why it doesn't take a thousand steps any more

A thousand sequential passes through a large network is slow, and early diffusion models were. The first big speed-up came only months later: Denoising Diffusion Implicit Models (DDIM) kept the same trained network but changed the sampling procedure so it could take bigger strides, producing "high quality samples 10× to 50× faster in terms of wall-clock time"[2]. The same paper noted you can "trade off computation for sample quality" — fewer steps, rougher image[2]. That trade-off is exactly the steps slider you'll find in local image tools today.

Later work pushed further. Adversarial Diffusion Distillation, published by Sauer and colleagues in November 2023, trains a student model to imitate a big diffusion model in "just 1-4 steps while maintaining high image quality," and billed itself as the first method to make single-step, real-time generation work with large foundation models[8]. When an app shows you an image updating live as you type, a distilled few-step model is the likely reason.

The shortcut that made it practical: latent space

The second problem was size. A 512×512 colour image is 512 × 512 × 3 = 786,432 numbers, and early diffusion models worked directly on every one of them at every step. The 2021 Latent Diffusion paper from Rombach and colleagues — the basis of Stable Diffusion — noted that pixel-space models "consume hundreds of GPU days" to train[3].

Their fix was to stop denoising pixels. First, train a separate autoencoder: a pair of networks that squash an image into a compact representation and expand it back again with little visible loss. Then run the entire diffusion process in that compact "latent space," and only decode to pixels once, at the very end[3]. The authors describe it as finding "a near-optimal point between complexity reduction and detail preservation"[3].

The numbers make the point. Stable Diffusion v1's autoencoder downsamples by a factor of 8 and maps an image of shape H × W × 3 to a latent of shape H/8 × W/8 × 4[4]. For a 512×512 image that's a 64 × 64 × 4 latent: 16,384 numbers instead of 786,432, about 48 times fewer. It's also why image sizes in these tools must be multiples of 8 — the diffusers documentation says so directly for Stable Diffusion v1.5[6].

The paper's own pitch was that this made training "possible on limited computational resources"[3]. In our view it's the single biggest reason image generation left the data centre, and why you can run these models on a home graphics card at all.

How your words get in: text encoders and cross-attention

So far we have a machine that turns static into some plausible image. Your prompt enters in two stages.

First, a text encoder turns your words into a sequence of numbers that capture their meaning. Stable Diffusion v1 used a frozen, pretrained encoder called CLIP ViT-L/14 — a model originally trained to match images with their captions, so its representation of "lighthouse" already sits close to what lighthouses look like[4].

Second, those numbers are fed into the denoising network through cross-attention layers, which let every region of the image-in-progress "look at" every word of the prompt at every step[3][4]. The Latent Diffusion authors pointed out that this makes the conditioning general: the same mechanism works for text, bounding boxes or other inputs[3]. That generality is what add-ons like ControlNet exploit, letting you steer composition with a pose skeleton or depth map instead of words[6].

The model was trained on huge numbers of image–caption pairs — for Stable Diffusion v1-4, a filtered subset of the English-language LAION-2B dataset, trained for 225,000 steps on 256 A100 GPUs[4]. Everything it "knows" about what words look like comes from those captions, which matters a lot later.

The guidance slider, explained

A network conditioned on your prompt produces images related to it, but often only loosely. The widely used fix is classifier-free guidance, introduced by Ho and Salimans in 2022[5].

The idea: train one network that sometimes sees the prompt and sometimes sees nothing — Stable Diffusion v1-4 dropped the text for 10% of training examples for exactly this reason[4]. At generation time, run the network twice per step, once with your prompt and once without. The difference between the two predictions is, roughly, "the direction your prompt pushes." Then exaggerate that difference by a multiplier. That multiplier is the guidance scale (often labelled CFG scale). The paper frames it as a trade-off "between sample quality and diversity"[5].

In practice, per the diffusers docs: lower values give the model more "creativity" and images more loosely tied to your prompt; higher values follow the prompt more closely, and if the value is too high "you may observe some artifacts"[6]. Their examples compare 2.5, 7.5 and 10.5[6]. A negative prompt uses the same machinery — it replaces the empty "nothing" prompt with things to steer away from[6]. And the seed simply fixes the starting static, so the same seed, prompt and settings give you the same image again[6]. In code, with the diffusers library, all three are plain arguments:

generator = torch.Generator("cuda").manual_seed(30)
image = pipeline(
    "a lighthouse in a thunderstorm, oil painting",
    negative_prompt="blurry, low detail",
    guidance_scale=7.5,
    num_inference_steps=30,
    generator=generator,
).images[0]

A quick map from settings to concepts:

SettingWhat it actually controls
StepsHow many denoising passes; fewer is faster and rougher[2]
Guidance / CFG scaleHow hard the prompt's direction is exaggerated[5][6]
Negative promptWhat the guidance steers away from[6]
SeedThe starting static; fix it to reproduce an image[6]
Width / heightLatent size; multiples of 8 for SD-family models[6]

What changed since: flows, transformers and a different approach

The picture above describes the classic, 2022-era recipe. Two changes are worth knowing about as of September 2026.

Straighter paths and transformers. The Stable Diffusion 3 paper (Esser et al., March 2024) switched to rectified flow, a formulation that "connects data and noise in a straight line"[7]. It also replaced the U-Net with a transformer — the architecture behind language models — using "separate weights for the two modalities" and letting information flow both ways between image and text tokens[7]. The core idea, turning noise into an image a step at a time, is unchanged.

Not everything is diffusion. OpenAI's image generation built into GPT-4o is explicitly a different design: its March 2025 system card says that "unlike DALL·E, which operates as a diffusion model, 4o image generation is an autoregressive model natively embedded within ChatGPT"[9] — closer to how a language model writes text, one piece after another. OpenAI credits the native design with following detailed instructions and "reliably incorporating text into images"[9]. So when someone says "AI image generators all use diffusion," that was roughly true in 2023 and is no longer the whole story.

Where it falls short — and when not to use it

The model card for Stable Diffusion v1-4 is unusually candid, and its list is a good checklist for any diffusion model. It "does not achieve perfect photorealism," "cannot render legible text," may not generate faces and people properly, struggles with compositions like "A red cube on top of a blue sphere," and works worse with non-English prompts because it was trained mostly on English captions[4]. It also warns that training-data bias shows up in outputs, with "white and western cultures" often set as the default[4]. Newer models have improved on several of these — text rendering in particular — but the root cause hasn't gone away: the model can only reproduce patterns that were well represented in its captions.

That leads to some honest advice on when not to reach for an image generator:

After you've generated something: generated images can be hefty files. Our image compressor and image resizer shrink them for the web without uploading them anywhere — everything runs in your browser.

The short version: an image generator is a noise remover that learned what the world looks like from captioned images, working in a compact latent space and steered by your prompt through a guidance multiplier. Seen that way, the settings stop being magic numbers and the failure modes stop being surprising.

Sources

  1. arXiv — Ho, Jain & Abbeel, "Denoising Diffusion Probabilistic Models" (NeurIPS 2020), accessed September 2026
  2. arXiv — Song, Meng & Ermon, "Denoising Diffusion Implicit Models" (ICLR 2021), accessed September 2026
  3. arXiv — Rombach et al., "High-Resolution Image Synthesis with Latent Diffusion Models" (CVPR 2022), accessed September 2026
  4. Hugging Face — CompVis, Stable Diffusion v1-4 model card (architecture, training, limitations and bias), accessed September 2026
  5. arXiv — Ho & Salimans, "Classifier-Free Diffusion Guidance" (2022), accessed September 2026
  6. Hugging Face — Diffusers documentation: Text-to-image (guidance scale, negative prompt, seeds, image size), accessed September 2026
  7. arXiv — Esser et al., "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis" (Stable Diffusion 3, 2024), accessed September 2026
  8. arXiv — Sauer et al., "Adversarial Diffusion Distillation" (2023), accessed September 2026
  9. OpenAI — "Addendum to GPT-4o System Card: Native image generation" (25 March 2025), accessed September 2026

Related: How large language models actually work · Running AI on your own hardware: local LLMs explained · Image compressor