Anatomy of a vision-language-action model

Here is the demystification up front: a VLA is a VLM whose decoder has been repurposed to emit robot actions. Take PaliGemma — a 3 B-parameter vision-language model of exactly the kind you have quantized, served, and profiled — and change only what the output layers produce and what the training targets are. Everything genuinely new lives in one place: the action interface, the machinery by which a discrete-token machine is made to emit continuous, multimodal, 50 Hz motor commands. This lesson is a teardown of that interface, with the parameter counts, token budgets, and latency numbers that decide what runs on your RTX workstation and WidowX AI.

Foundations: The Two Clocks and the Action Chunk

Before dissecting the architecture, we must resolve a conceptual compression that plagues many VLA tutorials: the conflation of policy inference and control execution. These are two distinct processes running on different hardware at different rates. Policy inference is the GPU-bound computation that maps observations to a sequence of future actions. Control execution is the CPU/RTOS-bound loop that sends motor commands to the arm. The abstraction that bridges them is the Action Chunk.

An Action Chunk is a sequence of $H$ future control commands, $\mathbf{a}_{t:t+H-1}$, predicted simultaneously by the policy. Here, $H$ is the horizon (the number of control steps into the future), not the number of tokens. For a WidowX AI with 6 joints and a gripper, each action is a 7-dimensional vector. The chunk is a matrix of shape $H \times 7$. The policy runs at a frequency $f_{\text{inf}}$ (e.g., 10 Hz), while the controller runs at $f_{\text{ctrl}}$ (e.g., 50 Hz). The chunk bridges these rates: one inference produces $H$ actions, which the controller consumes over $H/f_{\text{ctrl}}$ seconds.

A critical limitation of this abstraction is Staleness. The first action in the chunk is fresh; the last action is computed $H/f_{\text{ctrl}}$ seconds before it is executed. If the scene changes (e.g., a human moves an object), the policy is acting on outdated observations. The staleness error $\varepsilon$ is bounded by the end-effector velocity $v$ times the chunk duration: $\varepsilon \approx v \cdot (H/f_{\text{ctrl}})$. This is a lower bound; acceleration and scene dynamics make the actual error worse.

Another undefined term in many texts is Multimodality. In signal processing, this implies multiple frequencies. In VLA action spaces, it refers to the probability distribution $p(\mathbf{a}|\mathbf{o})$ having multiple peaks. For example, grasping a mug can be done from the left or the right. A direct regression head (MSE) fits the conditional mean, which is the center of the mug—a collision. A token-based or flow-matching head can represent the full distribution, allowing the policy to sample one mode or the other.

Finally, consider the Proprioceptive Token. The robot's state (joint positions, gripper width) is projected into a single token via a linear layer. This is a lossy bottleneck. It assumes the LM can infer velocity and acceleration from the position history in the context window. If the state changes rapidly, this assumption fails, and the policy may react sluggishly. This is a known limitation of the current architecture.

Worked Example: Feasibility and Staleness on an RTX 4090

Let us compute the feasibility and staleness for a $\pi_0$-class policy on an RTX 4090. Assume the VLM prefill takes $T_{\text{prefill}} = 80$ ms. The flow-matching expert takes $K=10$ steps, each taking $T_{\text{step}} = 2$ ms. The control rate is $f_{\text{ctrl}} = 50$ Hz, so $\Delta t = 20$ ms. The chunk length is $H=50$.

The feasibility condition is $T_{\text{inf}} \le T_{\text{chunk}}$. Here, $100\text{ ms} \le 1000\text{ ms}$, so the policy is feasible. The Duty Cycle is the fraction of time the GPU is busy: $d = T_{\text{inf}} / T_{\text{chunk}} = 100/1000 = 10\%$. This leaves 90% of the time for other tasks or overlap.

Now, consider the staleness at the end of the chunk. The 50th action is executed at $t=1000$ ms relative to the chunk start. It was computed at $t=0$ ms. The staleness is $\Delta t_{\text{stale}} = 1000$ ms. If the end-effector moves at $v=0.25$ m/s, the potential error is:

The backbone: a VLM you have already served

π₀'s backbone is PaliGemma (opens in a new tab): a SigLIP-So400m vision encoder (about 0.4 B parameters) feeding a Gemma-2B decoder-only LM (about 2.6 B). Each 224×224 camera image comes out of the vision tower as a 16×16 grid of patch embeddings — 256 soft tokens per view — projected into the LM's embedding space; the instruction ('fold the towel') is SentencePiece-tokenized and concatenated. No cross-attention module, no fusion network with a special name: one decoder-only transformer mixing image and text tokens in a single sequence, the standard multimodal-LLM layout you have served.

Multi-camera support falls out for free: a second view is just 256 more tokens. π₀'s configurations expect an exterior base camera plus wrist views, masking out whichever are missing — on your rig, one RealSense over the workspace plus one at the wrist. Size the prefix the way you would size any prompt:

Proprioception enters embarrassingly simply: joint positions plus gripper state pass through a learned linear projection and become a single token. The subtle part is routing. π₀ does not push state and action tokens through the Gemma weights — it adds a second, much smaller set of weights, the 300 M-parameter action expert, and routes by token type, mixture-of-experts style: image and text tokens use the VLM weights, state and action tokens use the expert, everything sharing one attention operation under a blockwise mask. The VLM's semantic knowledge reaches the actions through attention alone. Language conditioning is just prefix tokens: changing the prompt changes behavior the way it changes a chat model's answer.

Why bolt actions onto a VLM rather than train a control transformer from scratch? Data asymmetry. The largest robot-action corpora total on the order of $10^4$ hours — roughly $10^9$ timesteps at 50 Hz, which a frontier LLM pretraining cluster chews through in minutes. Every VLA is a bet that visual and semantic competence — what a towel is, where mug handles tend to be — transfers in from web-scale pretraining, so scarce robot data only has to teach motor behavior. That bet is the load-bearing assumption of this phase.

The action interface: four ways to get motion out of a token machine

Now the genuinely new part. Your arm needs actions that are continuous (7 real numbers: 6 joints plus gripper), high-rate (20–50 Hz), and multimodal — a mug can be grasped from the left or the right, and as you saw in Phase 03, averaging the two modes drives the gripper into the middle of the mug. A decoder-only LM natively emits one thing: a categorical distribution over discrete tokens. Every VLA architecture is one of four answers to that mismatch.

Direct regression pools the backbone's final hidden states through an MLP into a continuous chunk, trained with MSE. One forward pass — the fastest possible head — and completely blind to multimodality: MSE fits the conditional mean, and the mean of grasp-left and grasp-right is a collision. You built this head in Phase 03; it is the baseline every fancier head must beat under last lesson's protocol.

Discretized token bins are RT-2 (opens in a new tab)'s move, inherited by OpenVLA (opens in a new tab): chop each action dimension's range into 256 uniform bins, name each bin with a token (OpenVLA overwrites the 256 least-used tokens in the Llama-2 vocabulary), and let the LM emit actions as text. The elegance is real: actions become a language, so robot episodes and web VQA co-train under one cross-entropy loss; training inherits the maximally stable LM recipe; the categorical output represents multimodality exactly. Both costs are computable, so compute them.

First, quantization. A uniform quantizer with step $\Delta$ commits an error distributed uniformly on $[-\Delta/2, \Delta/2]$; its variance is $\int_{-\Delta/2}^{\Delta/2} x^2/\Delta\, dx = \Delta^2/12$, so the RMS error is $\Delta/\sqrt{12}$. For a joint with range $R$ split into $B$ bins:

Push it through the lever arm: a base-yaw error $\varepsilon$ displaces the end effector by roughly $r\varepsilon$, and the WidowX AI reaches $r = 0.769$ m:

Nine millimeters of worst-case fingertip error from one joint — nearly ten times the arm's 1 mm repeatability — is on the order of the tolerance for grasping small objects. Binned policies get away with it because closed-loop corrections absorb residual error, but it is a real ceiling on precision. The second cost bites harder. Autoregressive decoding is sequential in tokens, not timesteps:

import numpy as np

BINS = 256
H, D = 50, 7                    # chunk length x action dims (6 joints + gripper)
LO, HI = -np.pi, np.pi          # normalized per-dimension range, radians

# A smooth 1 s reach segment sampled at 50 Hz: sinusoidal joint sweep
t = np.linspace(0.0, 1.0, H)
chunk = 0.8 * np.sin(np.pi * t)[:, None] * np.linspace(0.3, 1.0, D)[None, :]

def tokenize(actions):
    frac = (actions - LO) / (HI - LO)               # map to [0, 1)
    return np.clip((frac * BINS).astype(np.int64), 0, BINS - 1)

def detokenize(ids):
    centers = (ids.astype(np.float64) + 0.5) / BINS  # decode to bin centers
    return centers * (HI - LO) + LO

ids = tokenize(chunk)
recon = detokenize(ids)
err = np.abs(recon - chunk)
rms = np.sqrt(np.mean((recon - chunk) ** 2))

print("tokens per chunk  :", ids.size)                    # 350
print("max err (rad)     : %.5f" % err.max())             # ~ (HI-LO)/512
print("rms err (rad)     : %.5f" % rms)                   # ~ (HI-LO)/256/sqrt(12)
print("fingertip max (mm): %.1f" % (0.769 * err.max() * 1e3))  # WidowX AI reach

# Autoregressive cost model: one LM forward pass per token
T_TOK_MS = 6.0   # bf16 bandwidth floor for a 3B backbone (Lesson 3)
print("AR decode per chunk: %.0f ms vs chunk duration 1000 ms"
      % (ids.size * T_TOK_MS))

Diffusion heads keep the backbone as a conditioner and generate the chunk by iterative denoising: start from Gaussian noise shaped like the chunk and apply a learned denoiser $K$ times. Diffusion Policy — which you weighed against ACT in Phase 03 — is the small-scale original; Octo (a 93 M-parameter open generalist) is the transformer-scale exemplar. Diffusion buys exact multimodality and continuous output with a stable regression-to-noise objective; the price is $K$ extra passes at inference, $K$ anywhere from 10 to 100 depending on the sampler.

Flow-matching experts are π₀'s refinement of the same idea, and the reason the next lesson exists. Instead of a stochastic denoising chain, the 300 M action expert learns a velocity field that transports noise to the action chunk along a deterministic ODE, integrated in about 10 fixed steps. All 50 actions are refined in parallel at every step; the VLM prefix is computed once, its KV cache reused across all 10 steps; the output is continuous — no bins, no quantization floor. What the velocity field is and why 10 steps suffice is next lesson's business. Today it is a black box with a known interface and a known cost.

Cross-embodiment pretraining: what a million trajectories buy

The word 'pretrained' in every model above means the same thing structurally: a mixture over many robots. Open X-Embodiment (opens in a new tab) pooled roughly 60 datasets from 21 institutions — over 1 M trajectories, 22 embodiments, 500+ skills — into a common format, and a curated slice of it sits inside the training mixture of π₀, OpenVLA, and Octo alike. π₀ adds Physical Intelligence's own teleoperation corpus, roughly 10,000 hours across 7+ robot configurations — about $10^9$ robot timesteps, orders of magnitude below the token counts your LLM intuitions are calibrated on.

Ablations across these papers document what the mixture buys: pretrained-then-fine-tuned models beat from-scratch training by wide margins, most dramatically below ~10 hours of task demonstrations. Mechanistically, pretraining transfers what is true about manipulation in general — visual robustness to clutter and lighting, language grounding, object and affordance priors, the coarse temporal shape of reach-grasp-move-place. It is the robotics analogue of what web pretraining gives a code model: not your codebase, but fluency in the language it is written in.

What it cannot buy is your rig. The pretrained model has never seen your camera extrinsics, table height, lighting, controller gains, gripper closing speed, or your action convention — joint-space versus Cartesian, delta versus absolute; get that wrong and the policy is wrong in units, not in skill. The sharp version: BridgeData V2 (opens in a new tab), 60,000+ trajectories collected on WidowX-250 arms, is inside these training mixtures. Your WidowX AI is that arm's larger, stronger successor — same family, same joint-plus-gripper layout, but different kinematics and actuation — so your embodiment is a close cousin of the training data, not literally in it; and your rig certainly is not, so zero-shot behavior on your bench will range from comically bad to unsafe. Closing that gap with a few hours of your own demonstrations is the point of Lesson 4.

Scale reality: parameters, VRAM, and why chunks exist

Now the numbers that determine what runs on your desk. π₀: 0.4 B (SigLIP) + 2.6 B (Gemma) + 0.3 B (action expert) ≈ 3.3 B parameters. OpenVLA: 7 B-class. GR00T N1: about 2 B. RT-2 was 12 B to 55 B and ran at 1–3 Hz on multi-TPU cloud serving — it never lived on the robot. The field's retreat from 55 B to the 2–7 B class was not scientific modesty; it is the deployability constraint you would have predicted: the model has to fit and run on a workstation GPU physically near the arm.

VRAM at inference follows from your usual arithmetic: bf16 weights cost 2 bytes per parameter — about 6.6 GB for π₀, about 14 GB for OpenVLA — plus activations, a KV cache for a ~533-token prefix (small), and framework overhead. The OpenPI repository (opens in a new tab) quotes floors that match: over 8 GB for inference, over 22.5 GB for LoRA fine-tuning, over 70 GB for full fine-tuning. A 24 GB RTX 4090-class card serves π₀ comfortably, fits LoRA with modest headroom, and cannot touch full fine-tuning: Lesson 4 is a LoRA lesson by arithmetic, not by preference.

Latency is the harder constraint. One full π₀ inference — two vision-tower passes, a ~533-token prefill through 3 B parameters, ten flow-matching passes through the expert — lands on the order of 100 ms on a 4090-class GPU. One action per inference would cap you at ~10 Hz, every action at least 100 ms stale — your entire measured latency budget, gone before the arm moves. The escape is the decision that makes this model class deployable at all: emit a chunk. One inference produces $H = 50$ actions consumed at 50 Hz, and feasibility is a one-liner:

Chunking converts an infeasible serving problem into a comfortable scheduling problem — the same move as batching, except the batch dimension is time. And like batching, it is paid for in staleness: the 50th action executes almost 1.1 s after the observation it was computed from. Apply the staleness formula from the time lesson, $\varepsilon = v\,\Delta t$: at 0.25 m/s end-effector speed, a full-chunk-old prediction can be 250+ mm wrong if the scene changed. On a static tabletop nothing changed, which is why open-loop chunks work startlingly well in demos; add a human hand or a rolling object and chunk length becomes the reactivity bottleneck. Managing that tradeoff at serve time is your capstone's research question — and you now know where it lives in the architecture.

The landscape, without the hype

The π₀ family (Physical Intelligence). π₀ (opens in a new tab) is the flow-matching architecture this lesson dissected; π₀-FAST swaps the decode for autoregressive tokens over a compressed action representation (Lesson 3); π₀.₅ pushes open-world generalization. Open: checkpoint weights, the JAX training and serving code in OpenPI, and a Trossen tutorial targeting exactly your WidowX AI — though its published examples are bimanual, so single-arm adaptations need your own validation. Not open: the ~10,000-hour dataset, and any way to independently rerun the headline evaluations, which live on their robots. The honest read: the strongest published manipulation results in this class — multi-stage laundry folding, table bussing — with claims you can verify only by fine-tuning on your own hardware, which is what this phase does.

OpenVLA. A 7 B Prismatic VLM — SigLIP and DINOv2 vision towers fused, Llama-2-7B decoder — trained on roughly 970 k OXE episodes with RT-2-style 256-bin action tokens. It is the fully open triad — weights, code, and training data all public — and it was evaluated on WidowX-250 arms in BridgeData V2 scenes — the predecessor of your WidowX AI, the closest published baseline to your arm's family. The honest read: unmatched openness, but the decode is the bottleneck — one 7-token action per step at roughly 6 Hz on a 4090, no chunking in the base recipe (follow-up variants add parallel decoding and chunked heads).

GR00T N1 (NVIDIA). An open foundation model aimed at humanoids: an Eagle-class VLM paired with a flow-matching diffusion-transformer action head, about 2 B parameters total — a slow System 2 (the VLM, around 10 Hz) feeding a fast System 1 (the action head, over 100 Hz). Weights and code (opens in a new tab) are open; the training mixture of real teleoperation, human video, and synthetic trajectories is only partially released. The honest read — house loyalty notwithstanding — is that its claims center on humanoid benchmarks and synthetic-data scaling, not WidowX-class tabletop manipulation: a later comparison branch, after OpenPI works on your arm.

Where this goes next: the previous lesson, Evaluating policies: statistics you can defend, fixed the protocol every model in this phase will be judged by; this lesson gave you the anatomy — a VLM prefill you already know how to serve, plus an action decode that is the real design space. Next, Flow matching: how π₀ generates actions opens the one box left black today: the 300 M action expert, the velocity field it learns, and why ten integration steps buy a continuous, multimodal 50-action chunk for one prefill's worth of compute.