Serving a policy: the inference contract

The previous lesson ended with a checkpoint directory on your RTX workstation: fine-tuned weights, normalization statistics, a transform stack that turns WidowX AI observations into model inputs. You have stood at exactly this point in your day job — a trained model is a file, not a service, until someone defines the interface, pins the version, and measures the latency under load. Robotics keeps that discipline and sharpens it: your only client is a control loop holding a physical arm, and a bad response does not degrade an experience, it moves metal. This lesson is about the interface — what π₀ promises the robot, what the robot promises π₀, and what happens when either side breaks the promise.

Temporal Foundations: Control Periods and Actuation Latency

Before deriving the fire threshold, separate three clocks. The control period $T_c=1/f_c$ is the fixed interval at which the client consumes buffered actions; at 50 Hz, $T_c=20$ ms. The fetch latency $T_{e2e}$ runs from sending an observation request until the resulting chunk is in the client buffer, including network and inference. Actuation latency $T_{act}$ runs from issuing one buffered command until the plant responds. Actuation latency matters to observation age and closed-loop stability, but it does not delay the arrival of the next chunk and therefore is not added to the command-buffer starvation threshold.

The buffer state $B(t)$ is a concrete countdown. If $r$ actions remain, then $B(t)=rT_c$. A request fired at time $t_{fire}$ is safe when the old buffer lasts at least until the new chunk arrives: $B(t_{fire}) \ge T_{e2e}^{p99}$. Express the threshold as actions remaining, not as an ambiguous executed index: fire when $r$ reaches $k_{fire}=\lceil T_{e2e}^{p99}/T_c\rceil$. The motor pipeline can continue applying already issued commands during its fixed actuation delay; it does not consume an extra future chunk action.

Worked Example: Deriving the Fire Threshold

Consider a deployment with a control rate $f_c = 50$ Hz ($T_c = 20$ ms), a chunk size $H = 50$, and measured request-to-chunk-in-hand latency $T_{e2e}^{p99}=260$ ms. Actuation latency is measured separately as 5 ms. Determine the correct $k_{fire}$ for command-buffer starvation, then state where the 5 ms belongs.

1. The buffer must cover the fetch: $T_{cover}=T_{e2e}^{p99}=260\text{ ms}$.
2. Convert time to action count: $k_{raw}=260/20=13$ actions.
3. The ceiling changes nothing here: $k_{fire}=\lceil13\rceil=13$ actions remaining.
4. Verify: 13 actions provide $13\times20=260$ ms of playback. The separate 5 ms actuation latency belongs in observation-to-motion staleness and stability analysis, not in this buffer-arrival inequality.

Two boxes, one loop: the remote policy server pattern

The standard decomposition — the one openpi ships as serve_policy plus openpi-client, and the one Trossen documents for their arms (opens in a new tab) — puts the policy behind a websocket server on the GPU machine and a thin client inside the robot host's control loop. When the client needs a new chunk, it sends one observation (camera images, joint state, task prompt) and receives one action chunk: an $H \times d$ array, for π₀ typically $H = 50$ actions intended for 50 Hz execution — one second of motion per round trip. The client plays the chunk out through the arm driver; the server idles until the next request.

Why is this the standard shape, even when both processes could share one machine? Three familiar reasons. Dependency isolation: the server side is JAX, CUDA, and a multi-gigabyte checkpoint; the client side is camera drivers, the arm's UDP driver (libtrossen_arm), and a loop that must tick on schedule — an XLA recompile and a joint-command write should not share a process. Independent iteration: swap checkpoints and restart the server; the robot code never changes. The interface forces the contract: once observations cross a socket, someone has to write down exactly what an observation is. That document is the real subject of this lesson.

Size the wire before trusting it. Three 224×224 RGB uint8 images — base, wrist, and a second exterior view — are $3 \times 150{,}528$ bytes, about 441 KB per request before msgpack overhead: roughly 3.6 ms on wired gigabit Ethernet, negligible over loopback. Send raw uint8 — shipping float32 quadruples the payload for zero information gain.

The inference contract is an interface spec

Everything that crosses the socket needs a written schema, because the failure mode of an undocumented field is not an exception — it is a robot that moves slightly wrong. Treat the contract like an API spec published to another team:

• Camera keys — exact key names matching the transform stack you trained with last lesson (a renamed key is the most common serving bug), plus resolution, channel order, and dtype: 224×224×3 uint8, HWC.
• State vector — dimension, ordering, units. For your WidowX AI: six joint positions in radians in joint-index order (J0–J5), plus gripper width in meters. Padding to the model's 32-dim state width is the server's business, not the client's.
• Prompt — the task string, spelled exactly as in the training data; a policy fine-tuned on pick up the red block conditions measurably differently on a paraphrase.
• Action chunk — shape $(H, d)$, per-dimension semantics (absolute joint targets vs. deltas, which entries are padding), units, and the control rate the horizon assumes.
• Version — one string pinning checkpoint hash, normalization-stats hash, and transform-code revision, echoed in every response so the client can reject a mismatch loudly.

Normalization ownership deserves its own paragraph because it is where fine-tuned deployments die. The stats you computed last lesson are training artifacts: they belong to the checkpoint, and openpi correctly stores them in the checkpoint's assets and applies them server-side. The resulting contract is clean — the client sends and receives raw physical units; everything learned stays behind the socket. Any other split invites version skew: a client normalizing against last month's stats while the server denormalizes against this month's produces actions plausibly scaled and consistently wrong — the worst combination for debugging.

Both sides of this contract are living code, so plan for drift. Some violations fail loudly — a missing camera key raises server-side, a wrong state dimension breaks a matrix shape. The dangerous ones type-check: a camera resize from a driver update (subtly different framing than training), a reordered state vector, a unit change. Schema validation catches shape and dtype, never semantics. The defense is round-trip verification — replay a training observation through the live server and check the answer against the dataset, as you will in this lesson's exercise.

Robot serving is not LLM serving

You have spent years optimizing serving around one economic fact: GPUs amortize, so you batch. Robot policy serving inverts nearly every design pressure you know; see the assumptions side by side.

Two rows deserve expansion. Queueing: in an LLM server, a queued request is deferred value — served late, the user still gets their answer. In a policy server, a queued request is a photograph of a world that no longer exists. If a second request arrives while one is processing (a retry, a reconnect), the correct behavior is latest-wins: drop the older work, serve the newest observation, never deliver two chunks for interleaved execution. Warmup: openpi's server is JAX; the first infer call compiles, taking tens of seconds to reach steady state. The startup ritual is non-negotiable — send dummy observations at production shapes until latency stabilizes, then let the arm move.

Now make the cadence quantitative — the when-to-request decision is the heart of chunked serving. A chunk of $H$ actions consumed at control rate $f_c$ buys a fixed playback window, and the server's duty cycle follows:

When must the client request the next chunk? Derive it. Let $B(t)$ be playback time remaining in the buffer, $t_{\text{fire}}$ the moment the request is sent, and $T_{\text{e2e}}$ the full request-to-chunk-in-hand time. The buffer exhausts at $t_{\text{fire}} + B(t_{\text{fire}})$; the new chunk lands at $t_{\text{fire}} + T_{\text{e2e}}$. Starvation is the chunk landing after exhaustion, so the no-starvation condition — evaluated at the p99, the latency you refuse to be beaten by, not the mean — is:

A cost hides in that ceiling, and it is the doorway to your capstone. Every action in a chunk was computed from one observation, so action $k$ executes on information $T_{\text{e2e}} + k/f_c$ seconds old — the deepest action you execute (index 36, with the threshold above) runs on an observation from roughly one second ago. By the Phase 01 staleness formula, at 0.25 m/s end-effector speed that is 25 cm of world motion. Quasi-static tabletop picks survive because the world mostly waits for you; anything dynamic does not — which is why real-time chunking (opens in a new tab) is an active research problem and why your capstone exists. Today you serve the naive version cleanly and measure it honestly; Phase 05 makes it adaptive.

Deadlines, watchdogs, and the safe-stop contract

The cadence math says when starvation shouldn't happen; reliability engineering decides, in advance, what happens when it does. The server gets OOM-killed, the Ethernet cable gets kicked — and at 1% p99 you also just plain lose sometimes. The contract needs a failure clause as precise as its schema clause: if no valid chunk is in hand when the buffer empties, the client commands a safe stop within one control period. On the WidowX AI, safe stop means ramping to zero velocity and holding position with gravity compensation active — not cutting torque (the arm drops), not repeating the last action (the arm drifts), and never extrapolating the chunk (the arm invents motion from stale data). The gripper holds its last commanded width; dropping a grasped object is itself a failure.

This is where the Phase 01 watchdog stops being scaffolding and becomes load-bearing. It once guarded against a hung script; now it enforces deadlines for a learned policy, with two timers. A liveness check — a health endpoint polled every second or two, verifying process up, model loaded, contract version matched — catches slow-building failures before an episode starts. A per-request deadline — a hard timeout of your p99 plus margin, say 400 ms — catches failures mid-episode and routes them to safe-stop. The deadline is not a retry trigger for the same observation: a request that missed it was computed for a world that has moved on, so the recovery path is a fresh observation or a stop, never a rerun.

import numpy as np

class ChunkScheduler:
    """Buffered chunk execution with derived fire threshold.

    policy   : async wrapper around the websocket client
    watchdog : Phase 01 watchdog, now enforcing the serving deadline
    """

    def __init__(self, policy, watchdog, ctrl_hz=50.0, e2e_p99_s=0.26):
        self.policy = policy
        self.watchdog = watchdog
        self.fire_level = int(np.ceil(ctrl_hz * e2e_p99_s))  # 13 here
        self.buffer = []        # actions awaiting execution
        self.pending = None     # id of the newest in-flight request
        self.seq = 0

    def on_response(self, req_id, actions):
        if req_id != self.seq:
            return              # superseded: drop the stale chunk, never queue it
        self.buffer = [a for a in actions]
        self.pending = None

    def tick(self, obs, capture_time):
        """Called once per control period. Returns one action or None."""
        if len(self.buffer) <= self.fire_level and self.pending is None:
            self.seq += 1
            self.pending = self.seq
            self.policy.infer_async(
                obs, capture_time, req_id=self.seq, on_done=self.on_response
            )
        if not self.buffer:
            # Deadline missed. Do not extrapolate, do not replay: stop.
            self.watchdog.safe_stop(reason="chunk starvation")
            return None
        self.watchdog.pet()     # loop is alive and fed
        return self.buffer.pop(0)

Twenty lines that encode most of the lesson: the fire threshold comes from a measured p99, not a hardcoded guess; the response handler enforces latest-wins via sequence id; starvation routes to the watchdog instead of improvising; and the watchdog is petted only by successful ticks, so a hung camera trips the same safety path as a dead server. One mechanism, every failure mode.

Measurement hooks: timestamps on every request

Phase 05 will ask: was that failed grasp caused by a latency spike, a stale chunk boundary, or the policy itself? You can only answer if every request carries timestamps from both hosts — the catch being that the robot host and GPU box have different clocks. Client stamps alone cannot separate network time from inference time; server stamps alone live in the wrong domain. You need four stamps per request — client send $t_0$, server receive $t_1$, server send $t_2$, client receive $t_3$ — plus the inter-clock offset $\theta$. This is the round-trip estimation from Phase 01 that aligned camera and host clocks; derive it once more, now over your serving socket, with forward and reverse network delays $d_f$ and $d_r$ and the server clock running $\theta$ ahead:

Solve each equation for $\theta$ — the first gives $\theta = (t_1 - t_0) - d_f$, the second gives $\theta = (t_2 - t_3) + d_r$ — and average them so the unknown delays appear only as their difference:

import time
import numpy as np

def estimate_offset(probe, n=64):
    """probe(t0) -> (t1, t2): server receive/send stamps on the server clock.

    Returns (theta, best_rtt): server_time = client_time + theta.
    """
    samples = []
    for _ in range(n):
        t0 = time.monotonic()
        t1, t2 = probe(t0)
        t3 = time.monotonic()
        theta = ((t1 - t0) + (t2 - t3)) / 2.0
        rtt = (t3 - t0) - (t2 - t1)
        samples.append((rtt, theta))
    samples.sort()  # fastest round trips carry the least asymmetry error
    keep = [theta for _, theta in samples[: max(4, n // 8)]]
    return float(np.median(keep)), float(samples[0][0])

class TimedPolicy:
    """Wraps the websocket client; emits one timing record per request."""

    def __init__(self, client, theta):
        self.client = client
        self.theta = theta          # server clock minus client clock, seconds
        self.records = []

    def infer(self, obs, capture_time):
        t0 = time.monotonic()
        result = self.client.infer(obs)      # server echoes t1, t2, infer time
        t3 = time.monotonic()
        t1 = result["t1"] - self.theta       # map server stamps to client clock
        t2 = result["t2"] - self.theta
        rec = {
            "capture": capture_time,
            "obs_age_at_send_s": t0 - capture_time,
            "wire_out_s": t1 - t0,
            "server_s": t2 - t1,             # queueing should be ~0: verify it
            "wire_back_s": t3 - t2,
            "e2e_s": t3 - t0,
            "contract_version": result["version"],
        }
        self.records.append(rec)
        return result["actions"], rec

Run the offset handshake at episode start, log $\hat{\theta}$ and its RTT bound in the episode metadata, and write one timing record per request into the episode file itself, next to observations and actions — not a separate log you will fail to join later. The decomposition earns its keep immediately: e2e_s spiking while server_s stays flat implicates the network or client event loop; a spike in server_s sends you hunting in JAX-land; growing obs_age_at_send_s means the camera pipeline is congesting before the socket is involved. And because each record joins to a robot outcome, latency becomes a feature of the episode you can correlate with success — the empirical substrate of your capstone.

The lesson compresses into a checklist you run before calling a deployment real — a launch review where every line is a measurement or an artifact, not a feeling:

1. Version pinned. One contract string covering checkpoint, norm-stats, and transform-code revisions; echoed in every response, client hard-fails on mismatch.
2. Warmup verified. Dummy inferences at production shapes until latency stabilizes; the episode logger refuses to start before steady state.
3. Latency measured from the robot host. p50/p95/p99 over 500+ requests under episode-like load (cameras streaming, logger writing), saved with the run; fire threshold recomputed from that p99.
4. Action-space round trip verified. One training observation replayed through the live server; the returned chunk is in physical units, inside joint limits, consistent with the dataset's recorded action.
5. Failure path tested. Kill the server mid-episode on purpose; confirm the watchdog stops the arm within budget and the episode record marks the event.
6. Clocks aligned. Offset estimated at episode start, logged with its RTT bound, re-checked at episode end to bound drift.

Where this goes next: “Fine-tuning OpenPI on your robot” produced the checkpoint; this lesson wrapped it in a contract, a cadence, and a safety clause, and started logging a timing record for every chunk served. The next phase opens with “The critical path: tracing observation to actuation”, which stretches the four timestamps you now capture per request into a full end-to-end trace — photons to torque. The p50/p95/p99 you pinned in the checklist stop being a report and become the baseline your latency-aware chunking research has to beat.