Why this topic
Denoising Diffusion Probabilistic Models (Ho, Jain and Abbeel, 2020) are the reason "diffusion" became the default answer to image generation. The final training recipe is almost suspiciously simple: corrupt an image with Gaussian noise, ask a network to guess the noise, minimise a mean squared error. It is easy to implement that recipe without knowing why it works, and just as easy to get lost in the paper's appendix trying to find out.
I wrote these notes to close that gap for myself. They start from Bayesian inference and the evidence lower bound, pass through the original 2015 diffusion paper, and then derive every term of DDPM by hand until the noise-prediction loss falls out. This post is an introduction to the idea and a reader's guide to those pages. The derivations themselves live in the notes below.
The core idea
A drop of dye in a glass of water spreads out until the water is uniformly tinted. That process destroys structure, but it does so in tiny, local, Gaussian steps, and each tiny step is easy to reverse if you know which direction to push. Diffusion models exploit exactly this asymmetry. Destroying data is trivial and fixed; rebuilding it is learned, one small denoising step at a time.
The forward process adds a little Gaussian noise at each of $T$ steps, controlled by a fixed schedule $\beta_t$. With the notation $\alpha_t = 1 - \beta_t$ and $\bar\alpha_t = \prod_{s=1}^{t}\alpha_s$, unrolling the chain (sums of independent Gaussians are Gaussian) gives a closed form that jumps straight from clean data to any noise level:
$$ q(x_t \mid x_0) = \mathcal{N}\big(x_t;\ \sqrt{\bar\alpha_t}\,x_0,\ (1-\bar\alpha_t)\,I\big), \qquad x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0, I) $$The reverse process is a learned Markov chain that starts from pure noise, $p(x_T) = \mathcal{N}(x_T; 0, I)$, and walks back with Gaussian transitions $p_\theta(x_{t-1} \mid x_t) = \mathcal{N}(x_{t-1};\ \mu_\theta(x_t, t),\ \Sigma_\theta(x_t, t))$.
The key trick is that, although $q(x_{t-1} \mid x_t)$ is intractable, the reverse step conditioned on the clean image is a Gaussian we can write down. Bayes' rule plus the Markov property gives
$$ q(x_{t-1} \mid x_t, x_0) = \mathcal{N}\big(x_{t-1};\ \tilde\mu_t(x_t, x_0),\ \tilde\beta_t I\big), \quad \tilde\mu_t = \frac{\sqrt{\bar\alpha_{t-1}}\,\beta_t}{1-\bar\alpha_t}\,x_0 + \frac{\sqrt{\alpha_t}\,(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}\,x_t, \quad \tilde\beta_t = \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\,\beta_t $$This posterior is the "ground truth" each reverse step should imitate. Training maximises a variational lower bound on $\log p_\theta(x_0)$, the same ELBO that underlies VAEs, and the bound splits into one KL term per step:
$$ \mathbb{E}_q\Big[\underbrace{D_{KL}\big(q(x_T \mid x_0)\,\|\,p(x_T)\big)}_{L_T} + \sum_{t=2}^{T} \underbrace{D_{KL}\big(q(x_{t-1} \mid x_t, x_0)\,\|\,p_\theta(x_{t-1} \mid x_t)\big)}_{L_{t-1}} \underbrace{-\,\log p_\theta(x_0 \mid x_1)}_{L_0}\Big] $$$L_T$ has no learnable parameters, so it is a constant. With the variance fixed to $\sigma_t^2 I$, each $L_{t-1}$ is a squared distance between means. Substituting $x_0 = \frac{1}{\sqrt{\bar\alpha_t}}(x_t - \sqrt{1-\bar\alpha_t}\,\epsilon)$ rewrites the target mean as $\tilde\mu_t = \frac{1}{\sqrt{\alpha_t}}\big(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\epsilon\big)$. Since $x_t$ is an input to the network, the only unknown is $\epsilon$, so we let the network predict it: $\mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}}\big(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\epsilon_\theta(x_t, t)\big)$. Dropping the per-step weight leads to the loss DDPM actually trains:
$$ L_{\text{simple}}(\theta) = \mathbb{E}_{t,\,x_0,\,\epsilon}\Big[\big\|\epsilon - \epsilon_\theta\big(\sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\ t\big)\big\|^2\Big] $$Sampling then runs the learned chain backwards, $x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\big(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\epsilon_\theta(x_t, t)\big) + \sigma_t z$, with $z \sim \mathcal{N}(0, I)$ for $t > 1$ and $z = 0$ at the last step.
A guide to the notes
| Pages | What is derived |
|---|---|
| 1 | Title page. |
| 2 | Bayesian inference: posterior, likelihood, prior and the intractable evidence $p(x) = \int p(x \mid z)p(z)\,dz$. Variational inference as a search over a family $Q$ of distributions (e.g. $\mathcal{N}(\mu_\phi, \sigma_\phi^2)$) minimising $KL[q(z)\,\Vert \,p(z \mid x)]$. |
| 3 | Step-by-step expansion of that KL into $\log p(x) - \text{ELBO}(\phi)$, so minimising the KL equals maximising the ELBO, and $\log p(x) \geq \text{ELBO}$. |
| 4 | ELBO rewritten as $\mathbb{E}_q[\log p(x \mid z)] - KL(q(z \mid \phi)\,\Vert \,p(z))$: a reconstruction term plus a regulariser toward the prior $\mathcal{N}(0, I)$; reminder that KL is not symmetric. |
| 5 | Sohl-Dickstein et al. (ICML 2015): the dye-diffusion analogy, "diffusion destroys structure", time reversibility of small Gaussian steps, the three-step algorithm, swiss-roll example, forward kernel $\mathcal{N}(x^{t}; x^{t-1}\sqrt{1-\beta_t}, I\beta_t)$ and a learned Gaussian reverse kernel. |
| 6 | DDPM setup: forward (fixed) vs backward (learned) Markov chains, notation $\alpha_t, \bar\alpha_t$, and the closed form $q(x_t \mid x_0)$ obtained by unrolling $x_1, x_2, x_3$ and merging Gaussian noise terms. |
| 7 | Backward process: $q(x_{t-1} \mid x_t, x_0)$ via Bayes and the Markov property, product-of-Gaussians identity, and the resulting $\tilde\mu_t(x_t, x_0)$ and $\tilde\beta_t$. |
| 8 | Loss $-\log p_\theta(x_0)$ bounded with Jensen's inequality; expansion of the bound into $L_T$, $L_{t-1}$ and $L_0$, with an interpretation of each term. |
| 9 | $L_T$ is constant. For $L_{t-1}$, the two fixed-variance choices $\sigma_t^2 = \beta_t$ or $\tilde\beta_t$, reducing the KL to $\frac{1}{2\sigma_t^2}\Vert \tilde\mu_t - \mu_\theta\Vert ^2$; reparameterising $x_t$ as a function of $x_0$ and $\epsilon$. |
| 10 | Algebra rewriting $\tilde\mu_t$ in terms of $x_t$ and $\epsilon$; the $\epsilon_\theta$ parameterisation of $\mu_\theta$; the weighted loss $\frac{\beta_t^2}{2\sigma_t^2\alpha_t(1-\bar\alpha_t)}\Vert \epsilon - \epsilon_\theta\Vert ^2$. |
| 11 | Why $L_{t-1}$ dominates; $L_{\text{simple}}$ and its link to denoising score matching; the sampling update; Algorithm 1 (training) and Algorithm 2 (sampling); interpolation in latent space versus pixel space. |
| 12 | Diagrams of the sampling loop (predict noise, subtract, add fresh noise before the next step) and of the training loop, after the DeepLearning.AI course. |
From derivation to code
The companion repository trains a class-conditional DDPM on MNIST (codes/Diffusion.ipynb, mirrored in codes/diffusion.py). It maps the notes onto code quite directly:
- Schedule. $T = 500$ steps with a linear schedule $\beta_t$ from $10^{-4}$ to $0.02$;
alphas = 1 - betasandalphas_cumprodis $\bar\alpha_t$ (page 6 notation). - Training (Algorithm 1, page 11). Each batch samples a random timestep per image, draws $\epsilon$, forms $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$ in one shot using the closed form, and minimises
F.mse_loss(noise_pred, noise). That is $L_{\text{simple}}$. Images are scaled to $[-1, 1]$, and the code also clamps $x_t$ to that range. - Network. $\epsilon_\theta$ is a small conditional U-Net with residual blocks; the timestep and the digit label are embedded and added together, so the model is $\epsilon_\theta(x_t, t, y)$.
- Sampling (Algorithm 2, page 11). Starting from $x_T \sim \mathcal{N}(0, I)$, each step applies $\frac{1}{\sqrt{\alpha_t}}\big(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\epsilon_\theta\big)$ and then adds noise with variance $\frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t$, which is the $\sigma_t^2 = \tilde\beta_t$ option from page 9. No noise is added at the final step, and samples are clamped to $[-1, 1]$ after every update.
The Hugging Face Space lets you watch this chain denoise a digit step by step, side by side with a flow matching model. If you want the continuous-time counterpart of these ideas, the companion post Understanding Flow Matching derives a generative model that replaces noise prediction with velocity regression.
The notes
Can’t see the viewer? Open the PDF in a new tab.