The capstone method: a latency-aware adaptive scheduler

You have shipped this system before, under different names. Continuous batching decides when to admit work from predicted service time; speculative decoding decides whether to keep draft tokens from a verifier's disagreement. The capstone method fuses those decisions and points them at a 7-dimensional action stream: a scheduler on the robot host that watches its own latency and its policy's self-consistency, and decides — every 20 ms — when to replan and how to switch chunks. Lesson one pinned the hypothesis, lesson two the protocol, and Phase 05 built the trace layer, injection harness, and scheduling baselines. This lesson assembles the method itself, down to the log your analysis will consume.

Conceptual Foundations: State Space, Continuity, and Normalization

Before deriving the scheduler's logic, we must rigorously define the physical quantities it manipulates. The policy outputs an action chunk, a sequence of $H$ future actions. In this context, each action $a_t \in \mathbb{R}^7$ represents a joint position delta (in radians) for the six rotational joints and a gripper command (in meters or normalized units). With an execution rate of 50 Hz, the time step is $\Delta = 20$ ms. Therefore, a chunk of $H=50$ actions represents exactly 1 second of future motion. The scheduler does not operate on raw pixels or high-dimensional latent states; it operates on this concrete, low-dimensional kinematic stream. This distinction is critical: the scheduler's decisions are constrained by the physical limits of the arm, not just the statistical properties of the policy.

The primary physical constraint governing chunk switching is kinematic continuity. When the scheduler replaces the active chunk with a new one, the transition must not introduce discontinuities in the velocity or acceleration profiles. A discontinuity in position commands at 50 Hz manifests as a step change in the target trajectory. The arm's high-bandwidth current controllers will attempt to track this step, resulting in a torque spike. Mathematically, if $a^{old}$ and $a^{new}$ are the old and new chunks, the frozen prefix constraint requires that for the first $k$ actions (where $k$ is the frozen prefix length), the difference remains below a velocity tolerance $\epsilon_{vel}$: $\|a^{new}_{t+j} - a^{old}_{t+j}\| < \epsilon_{vel}$ for $j < k$. This is not merely a software preference; it is a hardware safety requirement to prevent mechanical shock and controller saturation.

The second foundational concept is the normalized disagreement metric. The policy is a stochastic process, specifically a flow-matching or diffusion model, which samples from a conditional distribution $p(a|o)$. Multimodality in this context means that $p(a|o)$ has multiple distinct peaks (modes) for a single observation $o$. For example, a grasp task might have a valid 'left approach' mode and a 'right approach' mode. If the policy samples from different modes on consecutive calls, the resulting chunks will differ significantly in action space, even if the scene is static. This is why raw Euclidean distance is an inadequate measure of disagreement: it conflates 'the world changed' with 'the policy sampled a different valid mode.' To decouple these, we normalize the difference by the standard deviation of the actions in the training set.

Consider the dimensional implications. Suppose the standard deviation of a joint delta is $\sigma_{joint} = 0.02$ rad, while the gripper command has $\sigma_{grip} = 0.1$ m. A raw error of 0.05 rad in a joint is $2.5\sigma$, indicating a significant deviation. However, a raw error of 0.05 m in the gripper is only $0.5\sigma$, indicating a minor fluctuation. Without $\Sigma^{-1/2}$, the gripper's larger absolute range would dominate the norm, masking critical joint errors. Normalization ensures that $D_t$ is a unitless measure of statistical deviation, allowing a single threshold to apply across all dimensions.

Finally, we address the EWMA variance derivation for the latency predictor. The Exponentially Weighted Moving Average (EWMA) is defined as $\hat{\ell}_t = \alpha \ell_t + (1-\alpha)\hat{\ell}_{t-1}$. Unrolling this recurrence yields $\hat{\ell}_t = \alpha \sum_{k=0}^{\infty} (1-\alpha)^k \ell_{t-k}$. Assuming the latencies $\ell_t$ are independent and identically distributed (i.i.d.) with variance $\sigma^2$, the variance of the estimator is $\text{Var}[\hat{\ell}] = \alpha^2 \sum_{k=0}^{\infty} (1-\alpha)^{2k} \sigma^2$. The sum is a geometric series with ratio $r = (1-\alpha)^2$, which converges to $\frac{1}{1-(1-\alpha)^2} = \frac{1}{2\alpha - \alpha^2} = \frac{1}{\alpha(2-\alpha)}$. Thus, $\text{Var}[\hat{\ell}] = \frac{\alpha}{2-\alpha}\sigma^2$. This derivation shows that the EWMA is not a finite-window average but an infinite-memory filter with an effective memory length $N_{eff} = \frac{2-\alpha}{\alpha}$.

Worked Example: Scheduler Decision Logic

We now apply these foundations to a concrete scenario. Assume the following state at tick $t$: the predicted 90th-percentile latency is $\hat{L}_{0.9} = 132$ ms; the time step is $\Delta = 20$ ms; the margin is $m = 2$; the remaining actions in the active chunk are $R = 8$; the current disagreement is $D = 1.5$ (with thresholds $\theta_{lo}=1.0, \theta_{hi}=2.0$); and the refractory period is 10 ticks (200 ms), with the last launch occurring 5 ticks ago.

1. Calculate Launch Threshold: $R_{trigger} = \lceil 132/20 \rceil + 2 = 7 + 2 = 9$. Since $R = 8 \le 9$, the latency condition for launching a replan is met.
2. Check Refractory Period: Time since last launch is $5 \times 20 = 100$ ms. The refractory period is 200 ms. Since $100 < 200$, the refractory condition is NOT met.
3. Decision: The scheduler does NOT launch a new replan. It continues executing the current chunk. This prevents a 'replan storm' where rapid successive launches overload the server.
4. Frozen Prefix Calculation (Hypothetical): If the refractory period had passed, the frozen prefix length would be $k = \lceil 132/20 \rceil = 7$. The new chunk would be conditioned to match the old chunk for the next 7 actions (140 ms).
5. Disagreement Gate: Since no new chunk is generated, $D$ is not evaluated for adoption. The scheduler waits for the next tick.

This example illustrates the interaction between latency prediction and rate limiting. Even though the latency threshold was crossed, the refractory period prevented a launch. This is a critical safety feature: without it, a jittery latency signal could cause the scheduler to launch a replan every tick, saturating the GPU and increasing latency further, creating a positive feedback loop of failure.

The design space: two signals, three decisions

Fix the vocabulary first — a clean design space separates a method from a pile of heuristics. Your executor ticks at 50 Hz ($\Delta = 20$ ms); the policy server turns one observation into a chunk of $H = 50$ actions in roughly 100 ms of inference — about 150 ms capture-to-first-action, the numbers you measured on your own WidowX-plus-RealSense stack. At every tick the scheduler faces three questions:

1. When to launch a replan. Fresh chunks cost one inference pass each; what justifies spending it now rather than 200 ms from now?
2. Whether and how to adopt the result. Hard-swap, splice behind a frozen prefix, blend — or discard the new chunk and keep the old one.
3. How long to commit. Many actions per chunk amortizes inference; few preserves reactivity. The commit horizon is a dial, not a constant.

Every Phase 05 baseline is a fixed answer: synchronous execution replans on exhaustion and hard-swaps; temporal ensembling replans on a timer and blends; fixed-rate RTC splices behind a worst-case prefix. The adaptive scheduler makes each answer a function of two signals: $\hat L_q$, a calibrated quantile of the next inference latency — the cost of fresh information — and $D$, the disagreement between the active chunk and the newest one — the value the last purchase delivered. Cost versus value is the whole method; the rest of the lesson makes each signal measurable and each decision safe.

Signal one: predicting your own inference latency

Start with why prediction is unavoidable. Two of the scheduler's core quantities are functions of the next inference latency $\ell$ — a number that has not happened yet. Suppose the active chunk has $R$ actions remaining and you launch a replan now: the new chunk becomes usable $\ell$ milliseconds from now, while the executor keeps consuming one action every $\Delta = 20$ ms. Two consequences follow:

Read the two halves. The first is the launch deadline: if $R\,\Delta < \ell$, the old chunk runs dry — the starvation synchronous execution produced in Phase 05. The second is the frozen prefix: the $k$ actions that execute during inference cannot be changed by its result, so the new chunk must be generated to agree with them — the inference-time inpainting move from real-time chunking (opens in a new tab). Fixed-rate RTC hard-codes $k$ to the worst case — $\lceil p99/\Delta \rceil = 13$ actions at a 260 ms p99, even when the server answers in 95 ms: reactivity thrown away. The adaptive move sizes both from a predicted quantile:

What predicts $\ell$? Three features carry nearly all the signal. Recent latency — service time is strongly autocorrelated: slow states (thermal throttling, a fragmented allocator, a background job) persist for seconds to minutes, so an EWMA is the right memory. Queue state at launch — each in-flight request on a single-GPU server adds one service time to yours; on a dedicated RTX workstation the depth is usually zero, until a visualizer or stray finetune shares the GPU. Preprocessing cost — two RealSense streams versus one, decode and resize: 5–15 ms, measurable before the request leaves the host.

Two honest lines of math for the EWMA: unrolling the recurrence shows a geometric-weighted average; equating its variance to a flat $N$-sample mean gives its effective memory:

Now the honesty clause. On a dedicated workstation the distribution is a tight core with rare spikes — 105 ms p50, 118 ms p90, occasional 250–400 ms excursions from allocator stalls or a compositor grabbing the GPU. The core is predictable to within about 10 ms; the spikes are not. So the predictor's contract is a quantile, not a point estimate: a number the true latency stays below $q$ of the time. Expect it to beat a static p99 constant by 100–130 ms of recovered freeze budget, and to earn its keep under jitter injection, where any constant is always wrong in one direction.

import numpy as np

class LatencyQuantilePredictor:
    """Predicts the q-quantile of the next inference latency in ms.

    Features per request: [1, ewma_ms, queue_depth, preproc_ms].
    Fit offline on a Phase 05 trace with pinball loss; falls back
    online to EWMA plus an empirical residual quantile.
    """

    def __init__(self, q=0.9, alpha=0.2, window=64):
        self.q = q
        self.alpha = alpha
        self.window = window
        self.ewma = None
        self.residuals = []
        self.w = None  # set by fit()

    def fit(self, X, y, iters=4000, lr=0.05):
        """X: (n, 4) feature rows; y: (n,) observed latency ms."""
        w = np.zeros(X.shape[1])
        for _ in range(iters):
            pred = X @ w
            # subgradient of the pinball (quantile) loss
            g = np.where(y > pred, -self.q, 1.0 - self.q)
            w -= lr * (X.T @ g) / len(y)
        self.w = w
        return w

    def update(self, observed_ms):
        prev = observed_ms if self.ewma is None else self.ewma
        self.residuals.append(observed_ms - prev)
        self.residuals = self.residuals[-self.window:]
        self.ewma = self.alpha * observed_ms + (1 - self.alpha) * prev

    def predict(self, queue_depth, preproc_ms):
        if self.w is not None:
            x = np.array([1.0, self.ewma, queue_depth, preproc_ms])
            return float(x @ self.w)
        margin = np.quantile(self.residuals, self.q) if self.residuals else 0.0
        return float(self.ewma + margin)

def coverage(y_true, y_pred):
    """Fraction of latencies at or below prediction. Should be near q."""
    return float(np.mean(np.asarray(y_true) <= np.asarray(y_pred)))

An uncalibrated quantile predictor is a liability with a confidence-interval-shaped hole. The check is one line: on a held-out trace, coverage should return approximately $q$. With $n = 500$ requests the binomial standard error is $\sqrt{0.9 \times 0.1 / 500} \approx 1.3\%$ — coverage of 0.88–0.92 is healthy; 0.82 means the predictor is lying about risk. Recalibrate every session (driver updates and ambient temperature move the distribution overnight), and log every prediction beside its realized latency: the prediction is part of the trace now.

Signal two: chunk disagreement over the overlap window

The second signal asks not what fresh information costs, but whether the last purchase changed anything. When a new chunk arrives, the old one still holds $R$ remaining actions, and both prescribe actions for the same absolute ticks. Aligning them by tick index gives the overlap window $\mathcal{W}$, the stretch of future both chunks claim to know; disagreement is a distance over it:

Each choice earns its place. Without $\Sigma^{-1/2}$ the metric is a unit salad: WidowX AI joint deltas with standard deviations of 0.02–0.2 rad share a vector with a gripper command spanning its full range — unnormalized, the gripper either dominates the norm or vanishes from it. The discount matters because a disagreement 40 ms ahead will actually execute; one 800 ms ahead will likely be replanned away first. The mean-over-window form is spike-tolerant: a max variant fires on one noisy dimension at one tick, and chatters.

Now the interpretive question: what does $D_t = 2.4$ mean? One of three things, and the metric cannot say which. The world changed — the old chunk came from an observation 300 ms staler; a nudged mug or an entering hand shows up first as disagreement — the case the scheduler exists for. The policy is uncertain — near a decision boundary, observation noise flips the output; disagreement acts as a cheap two-member ensemble probe. The policy is multimodal — a flow-matching head sampling fresh noise picks a different valid grasp approach on each call: both correct, $D$ at 2–3 in a static scene, and switching modes mid-reach is exactly the lurching indecision to prevent. One scalar, three mechanisms — those are the aliasing risks.

The policy space: triggers, switches, and the variants worth building

One asymmetry shapes the policy space: measuring $D$ requires generating a chunk — a full 100 ms inference pass. Latency prediction is nearly free; disagreement is the most expensive sensor you own. So $D$ cannot decide whether to spend inference, only gate what you do with a chunk you already bought and modulate how soon you buy the next one. That constraint sorts the triggers:

Adoption — the second decision — has three mechanisms worth knowing and two worth building:

• Hard swap. Replace the remaining actions at arrival. A step discontinuity sits at the seam; at 50 Hz even a 0.05 rad per-joint jump is a step the joints' high-bandwidth FOC controllers will faithfully chase — a torque spike at the seam — and your jerk metric lights up. Keep it only as the ablation showing why continuity machinery exists.
• Frozen-prefix splice (RTC-style). The new chunk is generated to agree with the $k$ frozen actions, so the handover is continuous by construction. Your adaptive variants use this, with $k = \lceil \hat L_q / \Delta \rceil$ instead of a static worst case.
• Blend / cross-fade. Interpolate old to new over 5–10 actions. Cheap and smooth — but blending two modes averages a left approach and a right approach into a trajectory through the middle of the mug: the multimodal-averaging failure from Phase 03, resurfacing at the scheduler level. Build it only to demonstrate the failure.

The third decision folds in almost for free. The commit horizon $C$ — how many actions a chunk may execute before a fresh one is required — is temporal ensembling's query period and RTC's cadence wearing different clothes. Make it a function of $D$:

The design space is now enumerable, and scope discipline says build three points in it, not thirty:

1. A1 — latency-adaptive RTC. Fixed cadence; frozen prefix and launch deadline sized by $\hat L_q$ instead of a static p99. One mechanism, one claim: prediction recovers the reactivity worst-case provisioning throws away.
2. A2 — A1 plus disagreement gating. Adopt when $D \ge \theta_{\text{hi}}$, keep the old chunk (mode-stable) when $D \le \theta_{\text{lo}}$, with a refractory period between switches. Hysteresis, not a single threshold — the 1.0-to-2.0 gap prevents chatter.
3. A3 — A2 plus adaptive commit horizon. The full method: $C_t$ shrinks under high disagreement, stretches in quiet scenes. Exploratory — if A2 shows nothing, A3 is two more free parameters, not a rescue.

From method to evidence: architecture, guardrails, and the ablation matrix

Where does this run? In the executor thread on the robot host — the process that owns the 50 Hz tick and the UDP command stream to the arm's iNerve controller. Not next to the policy: the scheduler must keep making safe decisions precisely when the workstation stops answering. Its inputs are host-local: the action deque with each chunk's capture timestamp, predictor state fed by the trace layer, pending-request status, the current frame's preprocessing cost. Tick budget: under 1 ms — no allocation, no locks across the command send, no blocking calls; the policy client lives on its own thread behind a mailbox.

import math
from collections import deque

HZ = 50
DT_MS = 1000.0 / HZ  # 20 ms per action

class AdaptiveScheduler:
    """Runs inside the executor tick on the robot host.

    Collaborators (request_chunk, disagreement, splice, HOLD_ACTION)
    come from your Phase 05 runtime package; request_chunk is async
    and never blocks this thread.
    """

    def __init__(self, predictor, theta_hi=2.0, theta_lo=1.0,
                 refractory_ticks=10, margin_actions=2):
        self.pred = predictor
        self.theta_hi = theta_hi
        self.theta_lo = theta_lo
        self.refractory = refractory_ticks
        self.margin = margin_actions
        self.active = deque()        # remaining actions with chunk metadata
        self.pending = False         # one request in flight at most
        self.last_launch_tick = -10_000
        self.log = []                # one record per tick: the decision log

    def tick(self, t, obs_meta, result):
        rec = {"tick": t, "R": len(self.active), "decision": "execute"}
        lhat_ms = self.pred.predict(obs_meta["queue_depth"],
                                    obs_meta["preproc_ms"])
        rec["lhat_q_ms"] = round(lhat_ms, 1)

        if result is not None:                    # a new chunk just landed
            d = disagreement(self.active, result.chunk, t)
            rec["D"] = round(d, 2)
            if d >= self.theta_hi or len(self.active) <= self.margin:
                self.active = splice(result.chunk, t)
                rec["switch"] = "adopt"
            else:
                rec["switch"] = "keep_old"        # mode-stable inside the band
            self.pending = False

        r_trigger = math.ceil(lhat_ms / DT_MS) + self.margin
        cooled = (t - self.last_launch_tick) >= self.refractory
        if not self.pending and cooled and len(self.active) <= r_trigger:
            freeze_k = math.ceil(lhat_ms / DT_MS)
            request_chunk(obs_meta, freeze_k)     # async launch
            self.pending = True
            self.last_launch_tick = t
            rec["decision"] = "replan"
            rec["freeze_k"] = freeze_k

        if not self.active:                       # starvation watchdog
            rec["decision"] = "hold"
            self.log.append(rec)
            return HOLD_ACTION
        self.log.append(rec)
        return self.active.popleft()

The decision log is half the research contribution. Each tick appends one record: remaining actions, predicted quantile, measured disagreement, the decision, and why. Joined against the trace spans and encoder stream, it answers questions no success rate can: why did it replan at $t = 14.3$ s; what did $D$ do in the 500 ms before each failure; how often did the hysteresis band absorb a would-be switch. And counterfactual replay — a recorded session rerun through a different configuration offline — is how you tune guardrails without burning trials.

Two failure modes deserve named guardrails. Replan storms: disagreement triggers a switch, the switch changes the mode, the next comparison disagrees again — oscillation at the refractory frequency, a saturated server, climbing latency, the correlated-signals trap closing. Detection: the inter-replan-interval histogram piles at the refractory bound. Guardrails: hysteresis at 2.0/1.0, a 10-tick (200 ms) refractory period, cadence bounds of 1–5 Hz. Starvation: the predictor underestimates, the launch fires late, the deque empties. Guardrails: the margin $m$, the watchdog, an honest deadline-miss counter in the report.

Finally, the matrix that turns the method into evidence: three Phase 05 baselines and three adaptive variants, crossed with the injection conditions your harness already produces. Trial counts, blocking, and randomization come from the previous lesson; the matrix makes every comparison explicit before the first trial runs:

Read the matrix by its isolations. B2 versus A1 under jitter isolates latency prediction — same trigger, same splice, only freeze sizing differs. A1 versus A2 isolates disagreement gating; A2 versus A3, commit adaptation. The clean column is the regression guard: an adaptive scheduler that taxes the easy case has not earned its complexity, so a null result there is desired. And the constant-delay column separates delay from jitter as the causal variable — the distinction your hypothesis (adaptive scheduling recovers at least 10 success points under jitter versus fixed-rate RTC) was written to expose.

Where this goes next: the previous lesson, Designing robot experiments that survive review, supplied the protocol this matrix presupposes — trial counts, blocking, randomization, the predeclared metric. What remains is communication: The research artifact: report, repo, and talk turns the decision log, the matrix, and the preserved negative results into a six-to-eight-page report, a reproducible repository, and a ten-minute talk another engineer can check.