π₀-FAST: actions as tokens done right

The previous lesson bolted a 300M-parameter action expert onto the VLM and generated chunks by integrating a learned velocity field. There has always been a more obvious road, and it is the one your career prepared you for: skip the extra head entirely and emit actions through the language-model head as tokens. Discretize the action, give each bin a vocabulary entry, train with cross-entropy, decode autoregressively at test time. This is how RT-2 (opens in a new tab) controlled a robot with a fine-tuned web-scale VLM, and it is how π₀-FAST works today. The recipe is seductive because everything you know about LLM inference applies literally — KV cache, sampling, tokens-per-second arithmetic. The trap is that the naive version quietly fails at the 50 Hz control rates your Trossen arm wants, and the fix — the FAST tokenizer (opens in a new tab) — is one of the cleanest ideas in the VLA literature: compress the action chunk in the frequency domain before you tokenize it.

Conceptual Foundation: From Correlation to Predictability

Before tracing the FAST pipeline, we must resolve a subtle but critical statistical distinction that underpins why frequency-domain compression works for autoregressive models. The lesson previously noted that DCT makes coefficients "nearly independent." This is a common shorthand, but it conflates two distinct concepts: uncorrelatedness and independence. For a Gaussian process, the Discrete Cosine Transform (DCT) diagonalizes the covariance matrix. This means the resulting frequency coefficients are uncorrelated: the covariance between any two distinct coefficients is zero. However, uncorrelatedness is a weaker condition than independence. Independence requires that the joint probability distribution factorizes into the product of marginals, which is not guaranteed by zero covariance alone unless the variables are jointly Gaussian. In the context of action chunks, which are often modeled as Gaussian-like smooth trajectories, this distinction is less severe, but it is essential to understand that DCT removes linear dependencies, not necessarily all higher-order statistical structures.

Why does this matter for an autoregressive model? The Language Model (LM) head does not predict a coefficient in isolation; it predicts the next coefficient $c_k$ given all previous coefficients $c_{<k}$. The performance of this prediction is governed by the conditional entropy $H(c_k | c_{<k})$. If the coefficients were truly independent, $H(c_k | c_{<k})$ would equal the marginal entropy $H(c_k)$, and the model would have no predictive advantage from context. However, because the DCT is applied to a flattened sequence, the order in which coefficients are presented to the model matters profoundly. This leads to the concept of flattening order.

To quantify this, consider the conditional entropy $H(X|Y)$, defined as the average uncertainty of random variable $X$ given knowledge of $Y$. In our case, $X$ is the next token (coefficient) and $Y$ is the sequence of previous tokens. For a smooth trajectory, the conditional entropy of the next time-domain sample $a_t$ given $a_{t-1}$ is very low, because $a_t \approx a_{t-1}$. This is why naive binning fails: the model can minimize loss by copying. In the frequency domain, the conditional entropy $H(c_k | c_{<k})$ is structured differently. The first few coefficients (low frequency) have low conditional entropy because they are constrained by the physics of smooth motion. The later coefficients (high frequency) have higher conditional entropy because they represent the residual details that the low-frequency components did not capture. The LM head learns to exploit this structure: it uses the low-frequency tokens to "prime" its internal state, allowing it to predict the high-frequency tokens with greater accuracy than it could from scratch.

This brings us to the rate-distortion trade-off. In signal processing, rate-distortion theory describes the fundamental limit on how much a signal can be compressed (rate) for a given level of distortion (error). In FAST, the quantization scale $\gamma$ controls this trade-off. A larger $\gamma$ means finer quantization, which reduces distortion (reconstruction error) but increases the number of non-zero coefficients (rate). A smaller $\gamma$ means coarser quantization, which increases distortion but reduces rate. The goal is to find the "knee" of the rate-distortion curve, where further increases in rate yield diminishing returns in distortion reduction. This is why the choice of $\gamma$ is not arbitrary; it is a hyperparameter that balances the cost of decoding (more tokens) against the quality of the action (lower error).

Worked Example: Entropy of Smooth vs. Random Trajectories

To make the concept of conditional entropy concrete, let us calculate it for two simple trajectories. We will use a 1D trajectory of length $H=50$ for simplicity, though the logic extends to 7D. We will quantize the differences between consecutive samples into 256 bins, as in the naive binning approach, and compare the conditional entropy.

1. Define Trajectory 1 (Smooth): Let $a_t = 0.1 t$ for $t = 0, \dots, 49$. This is a linear ramp, representing a constant-velocity motion. The difference $\Delta a_t = a_t - a_{t-1} = 0.1$ for all $t > 0$. When quantized into 256 bins over a range of $[-1, 1]$, the value $0.1$ falls into a specific bin. Since $\Delta a_t$ is constant, the next difference is perfectly predictable from the previous one. The conditional entropy $H(\Delta a_t | \Delta a_{t-1})$ is 0 bits, because there is no uncertainty.
2. Define Trajectory 2 (Random): Let $a_t \sim \mathcal{N}(0, 1)$ be i.i.d. for $t = 0, \dots, 49$. This is Gaussian white noise, not a random walk. The difference $\Delta a_t = a_t - a_{t-1}$ is Gaussian with variance $2$ because adjacent samples are independent. Its values spread across many bins. Successive differences share one sample and are therefore correlated, so $H(\Delta a_t | \Delta a_{t-1})$ is high but not identical to the marginal entropy. After quantization into 256 bins the entropy is below the 8-bit maximum and depends on the chosen range and bin width; it is not automatically 8 bits.
3. Compare the Entropies: Trajectory 1 has near-zero conditional entropy, meaning the next token is highly predictable. Trajectory 2 has high conditional entropy, meaning the next token is not predictable. This is the core of the problem with naive binning: for smooth trajectories (which are common in robotics), the model can achieve low loss by simply copying the previous token, without learning the task. For random trajectories, the model must learn the task, but such trajectories are rare in real robot data.

Now, let us apply the DCT to these trajectories. For Trajectory 1 (linear ramp), the DCT coefficients will be concentrated in the first few low-frequency bins. The high-frequency coefficients will be near zero. When flattened low-frequency first, the sequence of coefficients will be: a few non-zero values, followed by many zeros. The conditional entropy of the non-zero coefficients is low (they are predictable from the ramp's slope), and the conditional entropy of the zero coefficients is also low (they are predictable as "no high-frequency content"). For Trajectory 2 (random), the DCT coefficients will be spread across all frequencies, with no strong concentration. The conditional entropy of each coefficient will be high, because the previous coefficients provide little information about the next one. This is why FAST works: it transforms the time-domain redundancy (low conditional entropy in smooth trajectories) into frequency-domain sparsity (many zeros), which is easier for the LM head to model and compress.

The autoregressive alternative: actions through the LM head

Start with the naive construction, because π₀-FAST is best understood as a patch to it. Take an action chunk $A \in \mathbb{R}^{H \times D}$ — for your WidowX AI, $D = 7$ (six joints plus gripper) and a one-second chunk at 50 Hz gives $H = 50$. Discretize each scalar into one of 256 uniform bins per dimension, assign each bin an ID, and map those IDs onto rarely used entries of the base vocabulary so the LM head needs no surgery. Training is plain next-token prediction with cross-entropy; inference is plain autoregressive generation followed by de-quantization back to joint targets. RT-2 did exactly this at low control rates and it worked — the striking result was that co-training with web data preserved semantic generalization, because actions were just another language the model spoke.

Sequence length is the first thing your serving instinct flags: 350 sequential decode steps per chunk is a lot of forward passes. But length is the smaller problem, and fixing it with a faster GPU would not save you. The deeper failure is statistical, and it is worth deriving carefully because it motivates everything FAST does.

Why per-timestep binning collapses at high control rates

At RT-2's control rate (a few Hz), consecutive actions are far apart in joint space and each token carries real information. At 50 Hz, physics makes consecutive actions nearly identical: a smooth trajectory cannot move far in 20 ms. Put concrete numbers on it. A representative WidowX AI joint has a usable range around 4 rad (published limits run from $3\pi/4$ on J2 to $2\pi$ on the base yaw), so 256 bins give a bin width of $w = 4/256 \approx 15.6$ mrad. A moderate joint speed during a reach is $\dot{q} \approx 0.5$ rad/s. Between consecutive timesteps the joint moves:

The next token is therefore almost always the same bin as the previous one, occasionally an adjacent bin. Conditioned on the previous token, the next token effectively lives in a three-element set $\{-1, 0, +1\}$, so its conditional entropy is bounded by $\log_2 3 \approx 1.6$ bits — against the 8 bits of nominal capacity a 256-way token provides. More than 80% of every token is spent encoding information the model can get by copying. Now think like someone who trains language models: cross-entropy is minimized by whatever prediction rule is cheapest, and here the cheapest rule is the identity map on the previous token. The model can drive training loss very low without ever consulting the image, because the copy operator explains almost all of the label. Gradients concentrate on learning the codec's redundancy instead of the task, and the resulting policy underfits precisely the rare, information-bearing tokens — the moments where the trajectory actually changes. The FAST paper reports this empirically and bluntly: naive binning policies train slowly or fail outright on high-frequency data such as 50 Hz bimanual manipulation, even with unlimited compute.

FAST: transform, quantize, entropy-code

FAST (Frequency-space Action Sequence Tokenization, Pertsch et al., 2025 (opens in a new tab)) borrows the architecture of every successful lossy codec you have ever used — JPEG most famously — and applies it along the time axis of the action chunk. The pipeline has four stages:

1. Quantile-normalize each action dimension over the training set (roughly the 1st–99th percentile mapped to $[-1, 1]$), so one scale works across joints with wildly different ranges and units.
2. DCT along time: apply a discrete cosine transform to each dimension's length-$H$ trajectory, converting 50 correlated samples into 50 nearly independent frequency coefficients.
3. Quantize: scale the coefficients by a factor $\gamma$ and round to integers. Smooth trajectories concentrate energy at low frequencies, so most high-frequency coefficients round to exactly zero.
4. Entropy-code with BPE: flatten the integer coefficients low-frequency-first, interleaved across dimensions, and compress with a byte-pair encoder trained on the corpus of quantized coefficient streams. Long zero-runs and recurring low-frequency patterns collapse into single vocabulary entries.

The DCT is doing the real work, so write it down. For one action dimension $a_0, \dots, a_{H-1}$, the orthonormal DCT-II is:

Why this basis? Two properties. Energy compaction: a physically smooth trajectory — bounded velocity and acceleration, which your arm enforces by construction — has coefficients that decay rapidly with $k$; a one-second min-jerk reach puts over 99% of its energy in the first handful of coefficients. Decorrelation: for strongly correlated sequences (a first-order Gauss–Markov process with correlation near 1, which is a good model of 50 Hz joint trajectories), the DCT closely approximates the Karhunen–Loève transform — the basis that makes the coefficients statistically independent. Independent coefficients are exactly what an autoregressive model wants: each token now carries fresh information, and the copy shortcut is gone. Run the core of the pipeline on a synthetic chunk shaped like your robot's data:

import numpy as np

def dct_matrix(H):
    # Orthonormal DCT-II basis, shape (H, H). C @ a gives coefficients;
    # C.T @ c inverts exactly because C is orthonormal.
    t = np.arange(H)
    k = np.arange(H).reshape(-1, 1)
    C = np.cos(np.pi * k * (2 * t + 1) / (2 * H)) * np.sqrt(2.0 / H)
    C[0] = C[0] / np.sqrt(2.0)
    return C

def min_jerk(H):
    # One-second minimum-jerk reach profile: smooth like real teleop.
    s = np.linspace(0.0, 1.0, H)
    return 10 * s**3 - 15 * s**4 + 6 * s**5

H, D = 50, 7   # 1 s at 50 Hz; WidowX AI: six joints plus gripper
rng = np.random.default_rng(0)
amps = rng.uniform(-1.0, 1.0, size=D)
chunk = min_jerk(H)[:, None] * amps[None, :]   # (H, D), normalized units

C = dct_matrix(H)
coeffs = C @ chunk            # DCT along the time axis, per dimension

gamma = 30.0                  # quantization scale: fidelity vs brevity
q = np.round(gamma * coeffs)  # integers; most round to exactly zero

recon = C.T @ (q / gamma)
err = np.abs(recon - chunk).max()

print("nonzero coefficients:", int((q != 0).sum()), "of", H * D)
print("max reconstruction error: %.4f normalized units" % err)
# Output: 24 nonzero of 350, max error 0.0083 -- about one naive
# 256-bin width (2/256 = 0.0078). BPE then folds the zero runs and
# recurring low-frequency patterns into a handful of tokens.

Read those numbers the way you would read a profiler. The 350-sample chunk is described losslessly enough — reconstruction error around one naive bin width, which for a joint whose quantile range spans 2 rad is roughly 8 mrad, about 3 mm at a 0.4 m working radius (around half the arm's 0.769 m reach) — by 24 nonzero integers. After BPE absorbs the zero runs, a realistic 1-second WidowX AI chunk lands at roughly 30–60 tokens, an order of magnitude under the naive 350, and each surviving token is dense with information because the DCT removed the temporal redundancy first. The scale $\gamma$ is an explicit rate–distortion knob: raise it and you buy reconstruction fidelity with more nonzero coefficients for the entropy coder to carry. The FAST authors also ship a universal tokenizer, FAST+, trained on around a million action chunks spanning many robots and control rates — the recommended default in the openpi codebase, and usually good enough that you never retrain the codec for a single-arm setup like yours.

The latency ledger: decode versus integrate

Now the part you can audit better than most robotics researchers. Both π₀ and π₀-FAST share the same expensive prefix: encode two or three camera views and the instruction through the 3B PaliGemma backbone and populate the KV cache. Call that $T_{\text{prefix}}$ — on an RTX 4090-class card, budget 60–100 ms. What happens next diverges completely:

Derive the constants from memory bandwidth, exactly as you would size an LLM deployment. Each autoregressive decode step must stream the full 3B-parameter backbone: about 6 GB at bf16 over roughly 1 TB/s gives a hard floor near 6 ms/token, and an unoptimized JAX serving path lands at 8–12 ms/token in practice. With $N_{\text{tok}} = 45$ FAST tokens: $45 \times 10 \text{ ms} = 450$ ms of decode, so $T_{\text{AR}} \approx 530$ ms. Flow-matching π₀ instead runs $K = 10$ Euler steps through the ~300M-parameter action expert, whose queries attend to the cached prefix KV — each step streams about 0.6 GB, a sub-millisecond floor, 3–5 ms realized, so $T_{\text{flow}} \approx 100\text{–}140$ ms. That factor of four is not an implementation detail; it is the price of sequential dependency through a 3B network, and no kernel fusion removes it.

So when does autoregressive win? Three honest cases. Shared LM infrastructure: if your deployment already runs a quantized, batched, speculatively decoded serving stack, π₀-FAST rides it for free, and decode cost per robot collapses under batching in a way flow's bespoke sampler cannot match. Training throughput: the FAST paper reports π₀-FAST matching flow π₀'s task performance with up to 5x less training compute — cross-entropy on dense, decorrelated tokens is a remarkably efficient learning signal. Discrete likelihoods: exact per-token log-probs enable scoring, ranking, and constrained decoding tricks that continuous heads only approximate. When does flow win? Batch-1 tail latency — which is exactly your capstone's regime.

Training differences, and the protocol for your capstone

On the training side the trade flips in FAST's favor, and it matters for how much infrastructure you get to reuse in the next lesson. A π₀-FAST fine-tune is, mechanically, an LLM fine-tune: integer targets, cross-entropy, perplexity-style token accuracy as a live training metric, standard mixed-precision and LoRA recipes, no custom sampling machinery in the loss. Flow π₀'s objective — sample a flow time $\tau$, corrupt the action chunk, regress the velocity field — is stable in practice but is bespoke code with bespoke failure modes, and its training metric (velocity MSE) is one step removed from anything you can eyeball. In openpi both paths share the same data pipeline, transforms, and normalization statistics, and ship with pretrained base checkpoints for each family; the JAX training path is the one that supports LoRA and the FAST variants on your hardware. One caution transfers from your LLM experience with extra force: the tokenizer is part of the model contract. If you retrain the FAST codec on your own data instead of using FAST+, you have changed the action vocabulary — checkpoints, decode code, and every downstream number now depend on that artifact, so version it like a tokenizer, because it is one.

For the capstone, the decision is already forced by the latency ledger: flow π₀ is your primary policy, because latency-aware action chunking is the research question and flow's ~120 ms replan gives the scheduler room to act. π₀-FAST earns its place as the controlled comparison: same dataset, same transforms, same normalization statistics, same evaluation protocol, different action head — a clean instrument for asking how inference cost and action representation interact with chunk staleness. That comparison is publishable context for your scheduler results precisely because everything else is held fixed.

Where this goes next: with Flow matching: how π₀ generates actions behind you and this lesson in hand, you now hold both action-generation paradigms and a latency ledger for choosing between them. Fine-tuning OpenPI on your robot makes it concrete: the same data transforms, normalization statistics, and LoRA recipe feed either family, and the choices you protocol-ed here — flow as primary, FAST as the frozen comparison — become configuration flags in an actual training run on your workstation.