Real-time chunking: asynchronous execution without discontinuity

You have spent a career hiding latency behind work: prefetch, double-buffering, pipelining, speculative decoding. The previous lesson measured what delay and jitter cost a control loop; the obvious cure is the oldest trick in systems — generate the next action chunk while the arm executes the current one. This lesson is about the side effect. Overlap creates a seam: a moment mid-motion where the executor switches between plans that were computed from different observations, different noise draws, sometimes different strategies entirely — and do not agree. In an LLM, a seam is a token nobody notices. On a WidowX AI at 50 Hz, it is a commanded velocity step the joint actuators execute with a clack you can hear across the room.

Conceptual Foundation: State, Seam, and Conditional Generation

Before deriving the mechanics of real-time chunking, we must rigorously define the physical and probabilistic objects involved. The core tension in asynchronous execution is between the physical state of the robot, which evolves continuously and irreversibly, and the planned trajectory, which is a discrete sequence of commands generated by a stochastic model. A discontinuity arises when these two representations diverge at the moment of handover. To quantify this, we first define the kinematic variables. Let $q \in \mathbb{R}^n$ denote the vector of joint positions, $\dot{q}$ the joint velocities, and $\ddot{q}$ the joint accelerations. The jerk, $\dddot{q}$, is the time derivative of acceleration. In control engineering, smoothness is often evaluated via the Root Mean Square (RMS) jerk, defined as $J_{\text{rms}} = \sqrt{\frac{1}{N}\sum_{t=1}^N \dddot{q}_t^2}$, where $N$ is the number of control steps. High jerk indicates rapid changes in force, which can excite mechanical resonances and cause audible noise.

The seam is the discontinuity in the commanded action sequence at the splice point. Let $a^{old}_t$ be the action from the previous chunk and $a^{new}_t$ be the action from the new chunk at the same control step $t$. The seam error is defined as $\epsilon_{seam} = \| a^{new}_t - a^{old}_t \|_2$. It is crucial to distinguish this from tracking error, which is the difference between the commanded action and the actual robot state. A seam is a planning artifact; it exists even if the robot tracks perfectly. If the seam error is $\delta$ and the control period is $\Delta t = 1/f_c$, the resulting velocity spike is $\dot{q}_{spike} = \delta / \Delta t$. For a 7-DoF arm at $f_c = 50$ Hz, a 1-degree seam error produces a 50 deg/s velocity step, which is often perceptible as a jerk.

Real-time chunking (RTC) resolves the seam by treating the problem as a conditional generation task. When a new chunk is requested, the first $d$ actions are already committed to execution. We define the frozen prefix $A_{frozen} \in \mathbb{R}^{d \times n}$ as the sequence of actions that will be executed before the new chunk arrives. The goal is no longer to generate an arbitrary chunk $A$, but to sample from the conditional distribution $p(A_{free} \mid o_{new}, A_{frozen})$, where $o_{new}$ is the fresh observation and $A_{free}$ is the remaining $H-d$ actions. This is mathematically equivalent to inpainting: we fix the values of the first $d$ rows of the action matrix and allow the model to generate the rest, conditioned on those fixed values. The 'hard mask' ensures that the generated plan is consistent with the robot's actual trajectory up to the splice point.

Worked Example: Quantifying the Seam and the RTC Fix

Consider a 7-DoF arm operating at $f_c = 50$ Hz. The inference latency is $T_{inf} = 110$ ms, which corresponds to $d = \lceil 50 \times 0.110 \rceil = 6$ control steps. At $t=0$, the arm is at $q_0 = [0, 0, 0, 0, 0, 0, 0]$. The old plan commands a linear move to $q_{goal} = [10, 0, 0, 0, 0, 0, 0]$ over 50 steps. The action at step $t$ is $a^{old}_t = [0.2, 0, 0, 0, 0, 0, 0]$ (since $10/50 = 0.2$ degrees/step). At $t=110$ ms (step 6), the arm has actually moved to $q_{actual} \approx [1.2, 0, 0, 0, 0, 0, 0]$ due to tracking lag. A new observation reveals an obstacle, so the new plan must detour. The new plan commands $a^{new}_6 = [1.0, 0.5, 0, 0, 0, 0, 0]$ to initiate the detour.

The velocity spike induced by this seam is $\dot{q}_{spike} = 0.94 / 0.02 = 47$ deg/s. For comparison, a smooth reach might have a peak velocity of 20 deg/s. This spike is a 2.35x increase in commanded velocity, which will cause a current spike in the actuators. Now, apply RTC. The first 6 actions are frozen to the old plan's values (or the actual executed values). The new plan is generated conditioned on $A_{frozen} = [a^{old}_0, \dots, a^{old}_5]$. The action at step 6 is no longer a free variable; it is constrained to be consistent with the trajectory up to step 5. The model generates a smooth transition from $a^{old}_5$ to the detour. Suppose the generated action at step 6 is $a^{rtc}_6 = [0.3, 0.1, 0, 0, 0, 0, 0]$. The seam error is now $\epsilon_{seam} = \| [0.3, 0.1, 0, 0, 0, 0, 0] - [0.2, 0, 0, 0, 0, 0, 0] \|_2 = \sqrt{0.1^2 + 0.1^2} \approx 0.14^\circ$. The velocity spike is reduced to $0.14 / 0.02 = 7$ deg/s, a 85% reduction.

The synchronous baseline: paying for inference with stillness

Start from the loop you built in the policy-serving lesson: π₀ on your RTX workstation returns chunks of $H = 50$ actions consumed at $f_c = 50$ Hz — one second of motion per chunk — with request-to-chunk-in-hand latency of 110 ms at p50 and 260 ms at p99. The synchronous schedule is the one every openpi example ships with: execute the chunk, stop, capture a fresh observation, infer, resume. The arm is motionless for every inference call, and the frozen fraction of wall-clock time falls straight out of the cadence:

Ten percent downtime sounds tolerable until you notice the schedule punishes the thing you want more of. The standard reactivity fix is to execute only part of each chunk — say the first 25 actions. Redo the arithmetic: $\varphi = 0.110/0.610 \approx 18\%$ at p50, $0.260/0.760 \approx 34\%$ at p99. Reactivity and uptime fight each other, and jitter makes the fight worse. Nor is the freeze neutral: each boundary is a stop-start transient injecting exactly the jerk Lesson 5 will measure, and freezing is not even well-defined mid-contact — a gripper holding a mug, a peg half-inserted. Meanwhile the last action of a full chunk runs on an observation $T_{\text{infer}} + 49/f_c \approx 1.09$ s old: 27 cm of world motion at Phase 01's 0.25 m/s reach speed. The synchronous baseline is not wrong — it is the honest control condition for your capstone. But it pays for consistency in stillness, the one currency a dynamic task refuses.

Going asynchronous: the seam, and the averaging answer

You already derived the asynchronous fire rule: request the next chunk when the buffer holds $k_{\text{fire}} = \lceil f_c \, T^{p99}_{\text{e2e}} \rceil = 13$ actions, so it lands just before the old one exhausts. The arm never stops — but the chunk now arrives mid-motion. Alignment first: openpi's chunk index 0 corresponds to the observation instant, and by arrival $d = \lceil f_c \, T_{\text{e2e}} \rceil$ control steps have elapsed — 6 at p50, 13 at p99. Replaying from index 0 literally rewinds time, commanding the arm back toward where it was 260 ms ago; the correct naive splice skips to index $d$. Yet index alignment is nothing like sufficient, because the two chunks are answers to different questions. Three sources of disagreement stack up at the splice:

• Correction — the new chunk saw a fresh observation and fixes the accumulated tracking error the old open-loop chunk never knew about: a genuinely useful jump, delivered as a step.
• Multimodality — the flow-matching lesson showed both mug grasps survive training; a fresh noise draw can flow to the other mode, so the old chunk descends toward the handle while the new one has chosen the rim.
• Sampling noise — even in a unimodal region, chunks generated from different $\epsilon$ draws differ by the policy's own output variance.

On the arm this lands as a current spike through the joints' brushless actuators, any transmission slack taken up in one tick, a visible twitch of the end effector — repeated every replan, roughly every 740 ms. The robot develops a tic. And because the jump direction depends on the noise draw, it never averages out; it accumulates in Lesson 5's jerk metrics and in grasp success whenever a seam lands during the approach centimeters.

One published answer you know from the ACT lesson: temporal ensembling — query the policy every step, keep all live chunks, execute an exponentially-weighted average. It genuinely smooths seams. But check its preconditions against your stack. Per-step inference at 50 Hz means a 20 ms budget against π₀'s 110 ms p50 — a 5.5× shortfall that batching cannot close, because one robot's requests are sequential in time. At the replan cadence you can afford (~740 ms), you hold at most two live chunks, and the grand ensemble degenerates into a crossfade between exactly the two plans that disagree.

Even where ensembling is affordable, you derived its cost already: the informed-weight share after a surprise is $S(\tau) \approx \tau/k$, so with $k = 50$ the ensemble is still majority-stale 560 ms after the world changed. Averaging with old chunks is the mechanism of the smoothing and of the dilution. And mode averaging returns wearing a new hat: the mean of a handle-grasp plan and a rim-grasp plan is a trajectory toward neither. Ensembling treats the symptom — command discontinuity, in signal space — while the disease is plan disagreement, in decision space.

Real-time chunking: freeze the committed, inpaint the rest

The reframe that unlocks the clean solution comes from Physical Intelligence's real-time chunking work (opens in a new tab) (RTC), and it starts from an admission: while inference runs, the executor will execute the next $d$ actions of the old chunk — the buffer must not starve, and physics offers no rollback. Those actions are committed. So treat them as a constraint, not a problem: generate the new chunk conditioned on its first $d$ actions being exactly the committed ones. This is inpainting in the image-generation sense — committed actions are the known pixels, frozen in place; the model fills in the rest consistently. The seam then cannot disagree with reality by construction: the new plan begins by agreeing with what the arm actually did, and its correction unfolds as a continuation instead of a step.

Why does a flow-matching policy admit this naturally? Because sampling is not one forward pass — it is $K = 10$ Euler steps of ODE integration, and iterative generation gives you a hook at every step. Recall the convention: the chunk rides a straight path $A^{\tau} = (1-\tau)\epsilon + \tau A$ from noise at $\tau = 0$ to data at $\tau = 1$. For the frozen rows the destination is known — the committed actions $\bar a$ — so their entire path is known too. Clamp them onto it at every integration step and let the field move only the free rows:

That clamp is the pedagogical skeleton; the RTC paper hardens it in two honest ways. Plain replacement conditions only weakly — the free rows see the prefix but their velocity is never corrected for it — so RTC adds a guidance term, adapted from training-free diffusion inpainting, steering the free rows toward consistency. And it softens the boundary: actions just past index $d$ are not committed but will likely execute before the next replan can change them, so RTC attaches a per-index weight decaying from one to zero across the old chunk's remainder — the near future held close to the old plan, the far future left free to react. Note what inpainting does not do: it enforces consistency, not correctness. If the observation was stale, you get a beautifully continuous wrong plan — staleness math still governs; RTC just stops adding self-inflicted discontinuity on top. The paper's evaluations make the case: under injected latency, naive switching degrades sharply while RTC holds near zero-latency performance — at zero training cost, on frozen π₀-class checkpoints.

Note the schedule freedom this buys. The scheme stays real-time as long as $d < H$ — inference must finish within one chunk duration, and your 260 ms p99 sits nearly 4× inside the 1 s budget. Because inference no longer forces a freeze, the 15% GPU duty cycle becomes spendable: replan more often, and reaction latency floors near $T_{\text{e2e}}$ plus sensing — roughly 150–300 ms — versus 560 ms of ensemble dilution or the second-plus synchronous worst case.

The other philosophy: train for the delay you will have

Inference-time inpainting is a patch applied to a policy that was never told delays exist. The December 2025 follow-up from the same group (arXiv:2512.05964 (opens in a new tab)) moves the fix into training: each training example samples a delay $d$ from a realistic distribution, hands the model the would-be-committed action prefix as an explicit input alongside the observation, and trains it to predict the continuation. At deployment there is no guidance machinery, no soft-mask schedule, no extra sampling cost — you pass the committed actions and the model natively generates a consistent continuation, because continuing a committed prefix is now in distribution rather than imposed on it.

You have seen this fork in your own field: post-training quantization versus quantization-aware training. Inference-time RTC is the PTQ move — works today, on frozen checkpoints you do not own the training run for, at the cost of approximation error and a guidance knob. Delay-conditioned training is the QAT move — the model learns the deployment condition, and can even learn delay-aware behavior such as hedging away from maneuvers it cannot revise in time; but it demands training access, compute, and a delay distribution fixed in advance — deploy outside it and you are extrapolating. The right experimental order is the course's: reproduce synchronous and inference-time RTC baselines first, then treat training-time conditioning as the extension. For your capstone, inference-time RTC is the natural substrate: its knobs move at runtime, exactly the degree of freedom a latency-aware scheduler needs.

Building it: two threads, one buffer, a commit horizon

The runtime is two loops sharing one buffer. The control thread ticks at a hard 50 Hz and never blocks: pop an action, send it, check whether to fire. The inference thread owns everything slow: grab the freshest observation, snapshot the frozen prefix, call the policy server, deliver the chunk. The coupling discipline is one number, the commit horizon $\hat d = \lceil f_c \, T^{p99}_{\text{e2e}} \rceil = 13$: at fire time, freeze the next $\hat d$ buffered actions and fix the splice point $\hat d$ steps out, deterministically, regardless of when the chunk actually lands. Freezing to the p99 costs a few steps of staleness on fast requests, but it makes the seam location a constant of the system instead of a random variable — and the inpainting mask must be known at generation time anyway. Two failure clauses carry over from policy serving: a chunk arriving after its splice point is dead on arrival (drop it, keep playing, re-fire — latest-wins), and a starved buffer triggers the safe-stop contract within one control period.

import threading
import numpy as np

class AsyncChunkExecutor:
    """Executor side of real-time chunking.

    Control thread, every 1/f_c seconds:
        action, fire = ex.get_action()
        if action is None: safe_stop()        # starvation clause
        else:              send_to_arm(action)
        if fire:
            frozen = ex.request_context()
            # inference thread: chunk = policy.infer_inpaint(obs, frozen)
            #                   ex.deliver_chunk(chunk)
    """

    def __init__(self, horizon=50, action_dim=7, commit=13, fire_at=13):
        self.H = horizon
        self.commit = commit          # ceil(f_c * T_e2e_p99): frozen prefix length
        self.fire_at = fire_at        # request when this many actions remain
        self.buf = np.zeros((0, action_dim))
        self.step = 0                 # absolute index of next action to execute
        self.pending = False
        self.splice_at = 0
        self.lock = threading.Lock()

    def get_action(self):
        with self.lock:
            remaining = len(self.buf) - self.step
            fire = (not self.pending) and remaining <= self.fire_at
            if remaining <= 0:
                return None, fire     # buffer starved: caller must safe-stop
            action = self.buf[self.step].copy()
            self.step += 1
            return action, fire

    def request_context(self):
        """Snapshot the frozen prefix; fixes the splice point deterministically."""
        with self.lock:
            self.pending = True
            self.splice_at = self.step + self.commit
            frozen = self.buf[self.step : self.splice_at].copy()
            return frozen             # inpainting target: chunk[0:commit] == frozen

    def deliver_chunk(self, chunk):
        """chunk index 0 aligns with the fire-time step; prefix was frozen."""
        with self.lock:
            self.pending = False
            if self.step > self.splice_at:
                return False          # late chunk: dead on arrival, re-fire next tick
            head = self.buf[: self.splice_at]
            tail = chunk[self.commit :]         # only the regenerated future
            self.buf = np.concatenate([head, tail], axis=0)
            return True

Look where the tunable numbers live, because they are the interface to your capstone. Commit horizon: larger never races the chunk but retains more stale actions per replan; smaller reacts faster but late chunks start hitting the dead-on-arrival clause. Fire threshold / replan rate: replanning more often buys reactivity with GPU duty cycle. Guidance weight (inside infer_inpaint): how hard the soft mask pulls the near future toward the old plan. A static, p99-derived setting is provably conservative — it wastes commit margin on the half of requests that finish in 110 ms and still loses beyond p99. The capstone thesis: drive these knobs with a forecast — predict each request's latency from signals you already log (queue state, thermals, recent p99 drift), set the commit horizon per request, and modulate replan rate by task phase, since approach wants reactivity and free-space transit does not. That is scheduling under a latency distribution: your home turf.

Where this goes next: the previous lesson, What delay and jitter do to a control loop, gave you the cost model; this lesson spent it, trading a freeze you could see for a seam you now know how to remove. Both knobs assume you know where $T_{\text{e2e}}$ actually goes, request by request — that is the next lesson's job: Profiling the loop: NVTX, Nsight, and the robot twist instruments the inference path so your commit horizon becomes a measurement rather than a guess. Then Motion quality: smoothness, jerk, and recovery turns the seam metrics you just simulated into the evaluation suite that decides whether your capstone scheduler actually made the robot move better.