Diffusion Policy: actions as denoising

The previous lesson gave you ACT: a CVAE that decodes an entire action chunk in one ~10 ms forward pass. Diffusion Policy makes the opposite trade. It treats the chunk — 16 timesteps × 7 action dims = a 112-dimensional vector — as a sample from a generative model, produced the way you know from image generation: start from Gaussian noise, denoise iteratively, condition on what the cameras see. It became the reference imitation policy of 2023 because its samples are smooth, precise, and genuinely multimodal — and a serving headache because each chunk costs 10–100 sequential network evaluations inside a control loop with a physical deadline. For a capstone on latency-aware action chunking, this is the policy class whose inference cost created the problem.

Foundations: Noise Schedules, Marginals, and the DDIM Derivation

Before deriving the sampler, we must rigorously define the noise schedule. In the forward process, the noise level is indexed by $k \in \{1, \dots, K\}$. The schedule is defined by a sequence of noise variances $\beta_k \in (0, 1)$. We define the signal retention factor as $\alpha_k = 1 - \beta_k$. The cumulative signal retention is $\bar{\alpha}_k = \prod_{i=1}^k \alpha_i$. The standard square-cosine schedule is defined as $\bar{\alpha}_k = \frac{f(k)}{f(0)}$, where $f(t) = \cos\left(\frac{t/K + s}{1+s} \frac{\pi}{2}\right)^2$ and $s$ is a small offset (typically 0.008). The normalization by $f(0)$ ensures $\bar{\alpha}_0 = 1$, meaning no noise is added at the start.

The closed-form marginal $A^k = \sqrt{\bar{\alpha}_k} A^0 + \sqrt{1 - \bar{\alpha}_k} \varepsilon$ is derived by induction. Assume $A^{k-1} = \sqrt{\bar{\alpha}_{k-1}} A^0 + \sqrt{1 - \bar{\alpha}_{k-1}} \varepsilon_{k-1}$. Substituting this into the step equation $A^k = \sqrt{\alpha_k} A^{k-1} + \sqrt{1 - \alpha_k} \varepsilon_k$ yields $A^k = \sqrt{\alpha_k \bar{\alpha}_{k-1}} A^0 + \sqrt{\alpha_k(1 - \bar{\alpha}_{k-1})} \varepsilon_{k-1} + \sqrt{1 - \alpha_k} \varepsilon_k$. Since $\varepsilon_{k-1}$ and $\varepsilon_k$ are independent standard Gaussians, their weighted sum is a Gaussian with variance $\alpha_k(1 - \bar{\alpha}_{k-1}) + (1 - \alpha_k) = 1 - \alpha_k \bar{\alpha}_{k-1} = 1 - \bar{\alpha}_k$. This confirms the marginal form.

The DDIM sampler is derived by inverting this marginal. Given a noisy sample $A^k$ and a noise estimate $\varepsilon_\theta(A^k, k, O_t)$, we solve for the implied clean sample $\hat{A}^0$. Rearranging the marginal equation gives $\hat{A}^0 = \frac{A^k - \sqrt{1 - \bar{\alpha}_k} \varepsilon_\theta}{\sqrt{\bar{\alpha}_k}}$. To step to a lower noise level $k' < k$, we substitute $\hat{A}^0$ back into the forward marginal for level $k'$: $A^{k'} = \sqrt{\bar{\alpha}_{k'}} \hat{A}^0 + \sqrt{1 - \bar{\alpha}_{k'}} \varepsilon_\theta$. This algebraic substitution eliminates the need for the intermediate noise term, allowing arbitrary jumps in $k$.

Worked Example: Latency Budget and Staleness Analysis

Consider a WidowX arm executing a 16-step action chunk at 10 Hz. The control loop executes 8 actions per replan cycle, giving a window of $T_{window} = 8 \times 100 \text{ ms} = 800 \text{ ms}$. We use a DDIM sampler with $N = 10$ steps. The observation encoding takes $t_{obs} = 10$ ms. Each U-Net forward pass takes $t_{net} = 5$ ms. The total inference time is $T_{infer} = t_{obs} + N \times t_{net} = 10 + 10 \times 5 = 60$ ms. The utilization of the time window is $60 / 800 = 7.5\%$. This leaves a large margin for safety.

The critical constraint is observation staleness. The observation $O_t$ is captured at the start of the replan cycle. The first action of the new chunk is executed immediately after the previous chunk finishes. If the arm moves at $v = 0.25$ m/s, the error due to the 60 ms inference delay is $\Delta x = v \times T_{infer} = 0.25 \times 0.06 = 0.015$ m = 15 mm. This is acceptable for many manipulation tasks. If the observation were captured at the start of the previous chunk, the staleness would be $800 + 60 = 860$ ms, leading to a 215 mm error, which would be fatal. Thus, the observation must be fresh for each replan.

Why one-shot regression breaks: the geometry of multimodal demos

Lesson 1 derived the conditional-mean collapse in one line: an MSE-trained head confronted with left-or-right demonstrations aims between them. Here is the version that will bite you on the WidowX. An elongated block lies on the table; a parallel-jaw gripper is symmetric under 180° rotation, so some episodes demonstrate the grasp with the wrist at $+90^\circ$ and others at $-90^\circ$ — both perfect. The targets are $+1.57$ and $-1.57$ rad; their conditional mean is $0.0$ rad, which lands the jaws square across the block's long axis and jams. This failure hits not a rare unseen state but the most demonstrated state in the dataset, with a confident action that appears in no demonstration. The Push-T benchmark in the Diffusion Policy paper is built on the same trap — the T-block can be contacted from either side, and unimodal policies commit to neither.

Chunking, from the previous lesson, sharpens the requirement: multimodality is a property of the joint distribution over the whole chunk, not of per-timestep marginals. A head that is multimodal at each step independently can sample the $+90^\circ$ mode at step 3 and the $-90^\circ$ mode at step 7 — an incoherent trajectory no demonstrator would recognize. You need one internally consistent draw of all $16 \times 7 = 112$ numbers. The classical fixes each solve part of this and miss the rest:

Read the third column as a requirements list: samples from the joint $p(A_t \mid O_t)$ over the full chunk; no mode-count hyperparameter; no coherence failures across timesteps or dimensions; plain-regression training rather than an adversarial or contrastive balancing act (implicit-BC energy models are expressive and famously fail this one). Diffusion checks every box, and its one concession is the last column. Diffusion Policy (Chi et al., 2023) (opens in a new tab) is the demonstration that, for manipulation, the trade is worth making.

Actions as denoising: the forward process, the loss, the conditioning

Fix notation. The generated object is the clean chunk $A^0 \in \mathbb{R}^{16 \times 7}$: 16 future timesteps of 7 joint targets, each dimension normalized to $[-1, 1]$. Superscripts index noise level, not time — $A^K$ is pure noise, $A^0$ is executable. The conditioning $O_t$ stacks the last $T_o = 2$ observations: two RealSense-class views through ResNet-18-scale encoders (~11 M parameters each) plus proprioception. The CNN variant feeds this embedding into a 1-D temporal U-Net over the 16-step axis through FiLM layers; the transformer variant uses cross-attention. Either way the observation is an input to the denoiser, never a target of the noise. The forward process is the corruption you know from images, at $K = 100$ levels on a square-cosine schedule rather than image DDPMs' 1000 (Ho et al., 2020 (opens in a new tab)). Unroll it once to see where the closed form comes from:

The last line is the workhorse: training never simulates the chain. Draw a chunk from the demonstration dataset, draw $k$ uniformly from $\{1..K\}$, draw $\varepsilon$ once, and form $A^k$ in a single closed-form jump. The network $\varepsilon_\theta$ is then asked to identify which part of $A^k$ is noise — while seeing the observation clean:

Three things differ from image generation. Dimension: 112 numbers versus a pixel grid's $10^5$–$10^6$, so the denoiser is a 1-D U-Net of tens of millions of parameters whose batch-1 forward pass costs single-digit milliseconds on your RTX workstation — a small network called many times sequentially, a profile you should file under launch-overhead territory. Conditioning cadence: an image model is a one-off generator; a diffusion policy is queried every replan cycle with a fresh $O_t$, for the lifetime of the deployment. Output prior: temporal convolutions along the 16-step axis bias samples toward smooth trajectories — low-jerk chunks are one of Diffusion Policy's most-reported advantages on real arms, where the independent per-step errors of a plain BC head excite every resonance the arm has.

The sampler is the latency budget: DDPM, DDIM, and step counts

Sampling inverts the corruption. DDPM ancestral sampling starts at $A^K \sim \mathcal{N}(0, I)$ and takes one learned step per level — estimate the mean of $A^{k-1}$ from $\varepsilon_\theta$, add calibrated fresh noise, repeat: $K$ sequential network evaluations, no parallelism across steps. Price it. The observation embedding is computed once per replan — two ResNet-18 passes plus a proprio projection, call it 10 ms batch-1 — and reused by every step, like a prompt's KV cache. Each U-Net evaluation is small: 3–8 ms batch-1 on an RTX 4090-class GPU depending on fusion (launch overhead is a double-digit fraction of each step at this size; CUDA graphs buy real money here, and this is your home turf). Take 8 ms for budgeting: DDPM at $K = 100$ costs $100 \times 8 + 10 \approx 810$ ms per chunk. A 10 Hz loop executes 8 actions in 800 ms — the arm finishes the chunk before the planner finishes thinking.

DDIM (Song et al., 2020) (opens in a new tab) rescues this with the same closed-form marginal that made training cheap. If the marginal lets you jump forward from $A^0$ to any noise level, you can also run it in reverse: given $A^k$ and the network's noise estimate, solve the marginal for the implied clean chunk, then redeposit that estimate at any lower level $k' < k$ — with no requirement that $k' = k - 1$:

Two properties fall out. First, step count becomes a deployment-time knob on the same checkpoint: trade quality against latency without retraining — per task, or even per replan (hold that thought until Phase 05). Second, with $\eta = 0$ the sampler is deterministic given $A^K$: which mode you get is decided entirely by the initial Gaussian draw. Quality holds up remarkably well down to about 10 steps — the paper reports ~0.1 s at 10 DDIM steps on an RTX 3080-class GPU — with curved, strongly multimodal distributions degrading first below that. Hence the reputation: famous because refinement buys sample quality no one-shot head matches, infamous because even the discounted bill is ~10× an ACT pass.

import numpy as np

# Square-cosine noise schedule, K = 100 training levels (iDDPM-style).
K = 100
s = 0.008
grid = np.arange(K + 1) / K
f = np.cos((grid + s) / (1 + s) * np.pi / 2) ** 2
alpha_bar = f / f[0]                      # alpha_bar[0] = 1, decays to ~0

CHUNK_SHAPE = (16, 7)  # T_p = 16 future steps x 7 action dims, scaled to [-1, 1]

def train_step(eps_net, actions, obs_feat, rng):
    """One diffusion training step. actions: (B, 16, 7) clean chunks."""
    B = actions.shape[0]
    k = rng.integers(1, K + 1, size=B)         # noise level per sample
    eps = rng.standard_normal(actions.shape)
    ab = alpha_bar[k].reshape(B, 1, 1)
    noisy = np.sqrt(ab) * actions + np.sqrt(1.0 - ab) * eps
    pred = eps_net(noisy, k, obs_feat)         # obs conditions; NEVER noised
    return np.mean((pred - eps) ** 2)          # simplified DDPM loss

def ddim_sample(eps_net, obs_feat, n_steps, rng):
    """Deterministic DDIM: n_steps sequential net evals, n_steps << K.
    obs_feat is encoded once per replan and reused across all steps."""
    ks = np.linspace(K, 0, n_steps + 1).round().astype(int)  # e.g. 100..0
    x = rng.standard_normal(CHUNK_SHAPE)       # the mode is chosen HERE
    for k_hi, k_lo in zip(ks[:-1], ks[1:]):
        ab_hi, ab_lo = alpha_bar[k_hi], alpha_bar[k_lo]
        eps_hat = eps_net(x[None], np.array([k_hi]), obs_feat)[0]
        x0_hat = (x - np.sqrt(1.0 - ab_hi) * eps_hat) / np.sqrt(ab_hi)
        x0_hat = np.clip(x0_hat, -1.0, 1.0)    # actions live in [-1, 1]
        x = np.sqrt(ab_lo) * x0_hat + np.sqrt(1.0 - ab_lo) * eps_hat
    return x                                    # one chunk, one coherent mode

Receding horizon: predict 16, execute 8, replan

Execution is receding-horizon control — predict far, commit near, replan: predict $T_p = 16$ steps, execute the first $T_a = 8$, resample from fresh observations. At 10 Hz the chunk spans 1.6 s and the commitment 0.8 s. Why predict 16 to run 8? The tail is scaffolding: forcing a dynamically coherent 1.6 s trajectory regularizes the executed prefix, and the overlap gives consecutive plans a shared region that damps boundary discontinuities. $T_a$ is the honest knob: it sets the closed-loop feedback rate at $1/(T_a \cdot 0.1\,\text{s}) = 1.25$ Hz — a disturbance can go unanswered for 0.8 s — while amortizing one ~90 ms inference over 8 actions. Halve $T_a$ and you double reactivity and double inference's share of the budget: the previous lesson's chunking trade with a 9× larger inference term.

The previous lesson also gave you a different way to consume chunks: ACT's temporal ensembling, where every past chunk that predicted an action for the current timestep votes with exponential weights, and the robot executes the average. That quietly assumes the votes agree — reasonable for ACT, whose test-time decoding (latent set to zero) is essentially deterministic. Point it at a genuinely multimodal sampler and it fails structurally: replan $n$ draws the $+90^\circ$ grasp, replan $n{+}1$ draws $-90^\circ$, and the ensemble average is the $0^\circ$ command — the averaging catastrophe reintroduced at execution time, after the model solved it. Hence Diffusion Policy commits to a prefix. The general rule: average within a mode, never across modes — and consecutive samples from a multimodal policy carry no guarantee about which mode you got.

Committing has its own failure mode: indecision. When two modes are nearly tied, consecutive replans can alternate — half a reach toward $+90^\circ$, resample, half a reach toward $-90^\circ$ — a limit cycle that dithers in front of the block. The blunt mitigation is a longer $T_a$; the interesting ones bias the next draw toward the last — warm-start the sampler from a partially re-noised copy of the previous chunk's tail, or inpaint the overlap region as a constraint during denoising. Keeping replans consistent with in-flight execution under real latency is an active research seam — Physical Intelligence's real-time chunking (opens in a new tab) lives exactly there — and it is the seam your capstone sits on.

Diffusion Policy vs ACT: an engineering decision, not a fashion choice

So: Diffusion Policy or ACT for your WidowX task? Strip the fashion and it is a set of measurable trades. Training stability first, because it is underrated: the diffusion loss is a plain regression with no terms fighting each other — no KL weight to tune (ACT's $\beta$), no mixture count, no contrastive negatives. It is genuinely hard to make diffusion training diverge, which matters when every bad run costs a retrain plus a hardware evaluation session. The costs are mundane: an EMA copy of the weights is near-mandatory and convergence takes more epochs — with ~100 demonstrations on your RTX workstation, budget several hours to Diffusion Policy's best checkpoint against one or two for ACT. Inference flips the sign: ACT's single ~10 ms pass versus ~90 ms at DDIM-10, a 9× gap that either disappears into your 800 ms commit window or dominates it.

What does published evidence actually support? The Diffusion Policy paper reports a 46.9% average relative improvement — over LSTM-GMM, implicit BC, and BeT; ACT is not in that comparison. ACT's headline results are on fine bimanual ALOHA tasks; Diffusion Policy is not in that comparison either. Direct head-to-heads are far scarcer than the discourse implies, and community experience since is reasonably consistent: the winner flips with the task. Diffusion Policy tends to win when demonstrations are strategy-diverse and smoothness matters; ACT holds its own on precision-dominated, single-strategy tasks and wins outright when the latency budget is tight. Both are strong 50–200-demo baselines. The defensible move for your block-grasping task is to train both — an overnight of GPU time — and let lesson 6's statistics pick, because a ten-rollout eyeball comparison cannot separate them.

• Reach for Diffusion Policy when demos are genuinely multimodal — multiple grasps, routes, or orderings — when one-shot heads give jerky or averaged motion, and when ~100 ms of inference per 0.8 s commit is affordable.
• Reach for ACT when you need single-digit-millisecond inference or maximum reactivity (small $T_a$), when the task has one dominant strategy executed precisely, or when training iteration speed matters more than expressiveness.
• Reach for neither alone when the demonstrations themselves are the limit — slow, sloppy, or below the performance you need. An expressive imitator reproduces mediocrity faithfully; that is the next lesson's subject.

One paragraph of Phase 04 before you meet it in code. Flow matching keeps this lesson's training story — closed-form corruption, plain regression, clean conditioning — but replaces the curved diffusion trajectory with a straight line: corrupt by interpolation, $A^\tau = (1-\tau)\,\varepsilon + \tau A^0$, and regress a velocity field $v_\theta(A^\tau, \tau, O_t)$ onto the constant target $A^0 - \varepsilon$ (Lipman et al., 2022 (opens in a new tab)). Sampling integrates an ODE along nearly straight paths, so ~10 Euler steps hold the quality DDIM needs schedule care to keep. π₀ (opens in a new tab)'s action expert is a flow-matching head emitting 50-step chunks with a 10-step integrator; open OpenPI (opens in a new tab) and you will recognize every moving part on a straighter schedule.

Where this goes next: ACT: action chunking with transformers gave the chunk a fast one-shot decoder; this lesson gave it a true generative model and a latency bill, and the pair bracket the imitation design space you will choose from on the WidowX. Notice what neither fixes: both match the demonstrator's distribution faithfully — hesitations, suboptimal routes, and ceiling included. When the data itself is the limit — when you need better-than-demonstrator behavior, or to stitch the good halves of mediocre episodes into a policy no single demo contains — matching distributions is the wrong objective, and that is Offline RL: when imitation is not enough.