What delay and jitter do to a control loop

The previous lesson gave you a trace: capture to preprocess to inference to dispatch to motion, each span timestamped on one monotonic clock. You now know your numbers — say 30 ms of sensing, 100 ms of π₀-class inference, 20 ms of queueing and dispatch. This lesson answers the question that trace cannot: what do those milliseconds do to the physics? In serving, latency is a cost paid downstream of correctness — a slow response is still the right response. A control loop breaks that separation: the output re-enters the input, so delay sits inside the dynamics, and past a computable threshold it turns a stable arm into an oscillator. The threshold is not folklore: you can derive it in four lines, predict the ringing frequency, and then find exactly that frequency in your logs.

Foundations: From Time-Domain Dynamics to Frequency-Domain Stability

Before deriving the delay margin, we must establish the mathematical bridge between the physical robot and the stability analysis. The robot's joint dynamics are described in the time domain by differential equations. To analyze stability efficiently, we transform these into the frequency domain using the Laplace transform. For a signal $x(t)$, its Laplace transform $X(s)$ is defined as $X(s) = \int_0^\infty x(t)e^{-st} dt$, where $s = \sigma + j\omega$ is a complex frequency variable. This transform converts differentiation into multiplication by $s$, turning differential equations into algebraic equations. A Linear Time-Invariant (LTI) system is one where the output is a linear function of the input and the system's properties do not change over time. Most robotic control loops, when linearized around an operating point, behave as LTI systems.

In a feedback loop, we distinguish between the open-loop and closed-loop systems. The open-loop transfer function $L(s)$ is the product of the controller $C(s)$ and the plant $P(s)$, measured before the feedback signal is summed with the reference. The closed-loop system is stable if the roots of the characteristic equation $1 + L(s) = 0$ lie in the left half of the complex plane. The Nyquist Stability Criterion provides a graphical method to check this: if the plot of $L(j\omega)$ (the open-loop frequency response) does not encircle the critical point $-1 + j0$, the closed-loop system is stable. The Bode plot is a two-part graphical representation of this same frequency response: the magnitude plot $|L(j\omega)|$ in decibels (dB) and the phase plot $\angle L(j\omega)$ in degrees.

Two key metrics emerge from the Bode plot. The crossover frequency $\omega_c$ is the frequency at which the loop gain equals unity, i.e., $|L(j\omega_c)| = 1$ (or 0 dB). This is the frequency where the system is most sensitive to phase loss because the feedback signal is neither amplified nor attenuated. The phase margin $\varphi_m$ is the additional phase lag required to reach $-180^\circ$ at the crossover frequency. Mathematically, $\varphi_m = 180^\circ + \angle L(j\omega_c)$. A positive phase margin indicates stability; the larger the margin, the more robust the system is to delays and modeling errors. A phase margin of $0^\circ$ implies marginal stability (sustained oscillation), while a negative margin implies instability.

The plant in our visual servoing example is an integrator. An integrator has the transfer function $P(s) = 1/s$. Physically, this means the output is the integral of the input. For a velocity-commanded joint, the input is velocity $u(t)$ and the output is position $q(t)$, so $\dot q(t) = u(t)$. In the Laplace domain, this is $sQ(s) = U(s)$, or $P(s) = Q(s)/U(s) = 1/s$. The phase of an integrator is constant at $-90^\circ$ for all frequencies. This constant phase lag is the baseline against which delay adds its frequency-dependent penalty.

Worked Example: Calculating Stability Margins for a Delayed Loop

Consider a WidowX joint with a proportional controller gain $K = 8 \text{ s}^{-1}$ and a constant measurement delay $\tau = 100 \text{ ms} = 0.1 \text{ s}$. The open-loop transfer function is $L(s) = \frac{K e^{-s\tau}}{s}$. We will calculate the phase margin, predict the ringing frequency, and estimate the overshoot.

The phase margin is $\varphi_m = 180^\circ + (-135.8^\circ) = 44.2^\circ$. Since $\varphi_m > 0$, the system is stable. The maximum allowable delay for stability is found by setting $\varphi_m = 0$: $180^\circ - 90^\circ - \omega_c \tau_{\max} = 0 \implies \tau_{\max} = \frac{90^\circ \cdot \pi/180}{K} = \frac{\pi}{2K} = \frac{\pi}{16} \approx 196 \text{ ms}$. Our delay of 100 ms is well within this limit.

The predicted ringing frequency is 1.27 Hz. If you observe oscillations at this frequency in your logs, it confirms the delay is the dominant factor. The estimated overshoot is around 20-25%, which matches the simulation results in the main lesson. Note that this is a first-order approximation; real systems with higher-order dynamics may exhibit slightly different overshoot, but the frequency prediction remains robust.

Delay in feedback: correcting toward a world that has moved on

Start with the simplest loop you actually run. A WidowX AI joint in velocity mode is, to good approximation, an integrator: the joint's on-board FOC controller handles torque and the joint angle integrates the velocity you command. Close a proportional visual-servoing loop around it — command velocity proportional to the position error seen by the camera — and every measurement arrives $\tau$ seconds late, where $\tau$ is the full capture-to-command chain you traced in the previous lesson. The controller is always pushing toward where the error was:

Now the Bode arithmetic, kept honest. The delay term has magnitude one at every frequency — $|e^{-j\omega\tau}| = 1$ — so the gain plot is untouched: $|L(j\omega)| = K/\omega$, which crosses unity at $\omega_c = K$. All the damage lands in phase. The integrator contributes a constant $-90°$; the delay contributes $-\omega\tau$ radians, growing without bound in frequency. This makes pure delay the worst-behaved component in the loop: a first-order lag of the same time constant costs at most $90°$ and attenuates gain as it does so; delay costs unbounded phase and attenuates nothing. At the crossover frequency, add the two contributions and subtract from $180°$ to get the phase margin:

Plug in numbers for a manipulation-speed visual servo. A moderate gain of $K = 8\ \text{s}^{-1}$ tolerates $\tau_{\max} = \pi/16 \approx 196$ ms and rings at 1.27 Hz when pushed to the edge. A gentle $K = 3$ tolerates 524 ms; a stiff $K = 20$ tolerates just 79 ms — less than one π₀-class inference pass. This is the EE-refresher formula from the LQR lesson, $\varphi_{\text{delay}} = 360° f \tau$, read in the other direction: there, how much phase a measured delay burns at a chosen bandwidth; here, how much delay a chosen gain survives at all. Two honest caveats. Delay only bites while the loop is actively correcting — at rest the delayed measurement equals the current one, which is why a laggy stack looks fine hovering and falls apart the moment it must chase an error. And real arms are not pure integrators — compliance and the actuator's inner control loops add phase of their own — but the model gives the right order of magnitude and the right shape of the trade: delay tolerance and gain aggressiveness are one budget, spent two ways.

Chunk staleness: a sawtooth, not a number

The classical loop above consumes one observation per action. Your π₀-class stack does not: it amortizes a roughly 100 ms forward pass by emitting a chunk of $H$ actions — the same one-pass-many-outputs move as speculative decoding, as the ACT (opens in a new tab) lesson showed. Chunking changes what latency means: there is no single $\tau$ anymore, because each action executes at a different temporal distance from the observation that produced it. Let $L$ be the capture-to-first-action latency you measured in the previous lesson (sensing plus inference plus dispatch), and let $\Delta$ be the control period. Action $j$ of the chunk executes at $t_{\text{cap}} + L + j\Delta$, so its staleness — the age of the world model it acts on — is:

Concrete numbers for the stack you will actually run: $L = 150$ ms (30 sensing + 100 inference + 20 queueing and dispatch), $H = 50$ actions at 50 Hz, so $\Delta = 20$ ms. Staleness starts at 150 ms and ramps to $150 + 49 \times 20 = 1130$ ms at the last action, averaging 640 ms across the chunk. Feed that through the staleness-to-error conversion from the time-and-latency lesson, $\varepsilon = v\,s$: against a scene element moving at 0.2 m/s — a nudged object, a handed-over cup — the final action of the chunk can be wrong by 22.6 cm. The sawtooth resets when a fresh chunk arrives, which is why chunk length is a reactivity dial, not a free amortization win. And synchronous execution adds a stall on top: if inference for chunk $n{+}1$ starts only when chunk $n$ finishes, the arm has nothing to execute for one full inference duration — the loop pays $s_{\max}$ and then holds position for another 100 ms. Every scheduling strategy in the next lesson attacks some segment of this timeline.

Constant delay is a parameter; jitter is a moving target

Here is the claim this lesson exists to make precise: a known, constant 100 ms is manageable; a 40–300 ms lottery with the same mean is poison. Constant delay is manageable for three separate reasons, and each one fails independently under jitter. First, training-data absolution: demonstrations were collected through the same pipeline, so the observation-action pairs already embed that temporal offset — the policy implicitly learned to act on 100 ms-old images, just as a teleoperator's demonstrations embed their reaction delay. Second, prediction: a known $\tau$ lets you forecast the state forward by exactly $\tau$, or augment the state with in-flight commands (the LQR lesson's exercise) and design for the delayed plant exactly optimally. Third, budgeting: a fixed $\tau$ is a fixed phase debit, and you choose $K$ accordingly, once.

Jitter voids all three. A delay wandering between 40 and 300 ms makes the effective plant time-varying: any forward-prediction horizon is wrong the moment the delay changes, a gain tuned safe for 300 ms wastes two-thirds of your bandwidth on the median tick, and a gain tuned for the median sits past its margin whenever the tail arrives. Training-data absolution inverts into training-serving skew: a deployment delay distribution that differs from the one baked into your demonstrations is distribution shift along the time axis, invisible to any per-frame data audit. And one subtlety your serving experience will appreciate: the correlation structure of the jitter matters as much as its marginal distribution. Millisecond-scale i.i.d. wobble largely averages out through the plant's inertia; real tails — thermal throttling, allocator stalls, a competing GPU process, a dropped camera frame forcing a re-sync — arrive as regimes lasting hundreds of milliseconds. During a 300 ms regime that persists for 250 consecutive ticks, your loop simply is a 300 ms-delayed loop, which for $K = 8$ is an unstable system, full stop.

import numpy as np

def run_loop(K, delays_ms, T=8.0, dt=1e-3, target=0.5):
    """Velocity-commanded joint (integrator plant) under proportional
    feedback that only sees a delayed measurement of the joint angle."""
    n = int(T / dt)
    q = np.zeros(n + 1)
    for k in range(n):
        j = max(0, k - delays_ms[k])   # measurement is delays_ms[k] ticks old
        u = K * (target - q[j])        # correct toward a past world
        q[k + 1] = q[k] + dt * u
    return q

def overshoot_pct(q, target=0.5):
    return 100.0 * (np.max(q) - target) / target

n = 8000
K = 8.0                                # 1/s: tau_max = pi/(2K) = 196 ms

runs = {}
runs["no delay        "] = np.zeros(n, dtype=int)
runs["constant 100 ms "] = np.full(n, 100)
runs["constant 170 ms "] = np.full(n, 170)
runs["constant 200 ms "] = np.full(n, 200)   # past the boundary

rng = np.random.default_rng(1)         # bursty jitter, mean 170 ms:
burst = np.zeros(n, dtype=int)         # resample every 250 ms window
for w in range(0, n, 250):
    burst[w : w + 250] = int(rng.uniform(40, 300))
runs["bursty 40-300 ms"] = burst

for name, d in runs.items():
    q = run_loop(K, d)
    print(name, "-> overshoot", round(overshoot_pct(q), 1), "pct")

Run it and read the five lines it prints. No delay: 0% overshoot, settled in half a second. Constant 100 ms ($K\tau = 0.8$, a comfortable 44° of margin): 30% overshoot, done by 0.82 s. Constant 170 ms (12° of margin): 86% overshoot with ringing that takes nearly seven seconds to die. Constant 200 ms, just past $\tau_{\max} = 196$ ms: 189% and growing, oscillating at almost exactly the predicted 1.27 Hz — the derivation's boundary, confirmed to within one sample period. Now the headline: bursty 40–300 ms with the same 170 ms mean as the stable run overshoots 161% and never settles cleanly, still wandering 0.3 rad from the target seconds later, because every burst above 196 ms is a few hundred ticks of genuine instability and every regime change invalidates whatever the loop had adapted to. Same mean. Different robot.

The physical signatures in logged joint traces

Everything above predicts recognizable shapes in the episode logs your Phase 01 logger already records. Learn these signatures now — Lesson 5 turns them into scalar motion-quality metrics, and your capstone's results section is built from them. Each is a latency pathology wearing a mechanical costume:

• Delayed-feedback oscillation — quasi-sinusoidal ringing in joint velocity near the loop's crossover frequency ($f \approx K/2\pi$, typically 1–2 Hz for manipulation-speed visual servoing), appearing during active correction and vanishing at rest; growing amplitude means the margin is gone. Detect it with an FFT of joint velocity over correction segments: a sharp peak at a frequency that tracks your gain setting — not the arm's ~5–10 Hz structural resonance — is delay, not mechanics.
• Chunk-boundary pauses — a comb pattern in the velocity trace: dips to near zero every $H\Delta$ seconds (every 1.0 s for 50 actions at 50 Hz), each dip roughly one inference duration wide, from synchronous replanning with nothing to execute. The position trace shows a staircase. Detect it via the autocorrelation of $|\dot q|$: a peak at exactly the chunk period is the fingerprint, and it disappears the moment you switch to asynchronous scheduling.
• Post-spike lunges — after a latency spike, the next chunk was computed from a pre-spike observation while the arm drifted or held; its first commands disagree with where the arm actually is, producing a step discontinuity in the command stream and a spike in finite-difference acceleration and jerk. Detect it by aligning jerk p99 events against the latency spikes in your critical-path trace: lunges land one pipeline-latency after the spike, like clockwork.

Deadlines, misses, and the injection instrument

To run controlled experiments on any of this, you need latency reframed as a per-tick pass/fail criterion. The loop runs at cadence $\Delta$; the action for tick $k$ must be ready at the arm's command interface by its slot. That gives a crisp definition:

What a miss costs depends on the fallback, which is why you must log the fallback path, not just the miss. Soft misses: the executor holds the last command or continues the current stale chunk — each consecutive miss silently adds $\Delta$ of staleness on top of the sawtooth, so an isolated miss at 50 Hz costs 20 ms (invisible) while twenty consecutive misses cost 400 ms (a lunge waiting to happen). Burst structure, again: report max consecutive misses, not just the rate. Hard misses: a watchdog declares the policy unresponsive after $M$ consecutive misses and freezes or retracts the arm — on a manipulation task, usually an unrecoverable episode failure. This is your serving error-budget discipline with one addition: the budget has a consecutiveness dimension, because physics integrates staleness while your SLO dashboard only counts events.

Now the experimental instrument, and the real reason this lesson exists. Natural latency variation cannot support causal claims, for reasons you know from performance work: it is confounded (natural spikes co-occur with GPU contention, thermal state, and scene complexity, all of which independently affect success), it lacks coverage (you cannot order a clean 400 ms condition on demand), and it is unreproducible (Tuesday's tail is not the tail your reviewer reruns). The fix is injection: add delay deliberately at a single choke point in the policy server — after inference completes, before the response is written — so computation is bit-identical across conditions and only timing differs. Draw the delay from a seeded generator keyed by episode and tick so any run replays exactly, and log both intended and achieved delay, because time.sleep overshoots by a scheduler quantum. A serious capstone sweep: constant delays of 0, 50, 100, 200, 400 ms to map the dose-response curve, then fixed-mean conditions (say 150 ms) with increasing spread — zero, ±50 ms uniform, a mean-matched heavy-tail lognormal, 250 ms bursty regimes — to isolate jitter and burstiness from magnitude. Budget trials honestly: at 20 episodes per condition, a 95% interval on success rate spans roughly ±20 percentage points; 40–50 episodes per cell resolves only a 25–30-point effect, and anything subtler takes well over a hundred — the power arithmetic Phase 06 makes exact. This instrument is the independent variable of your capstone; the real-time chunking (opens in a new tab) schedulers of the next lesson are evaluated with nothing else.

Where this goes next: the previous lesson, "The critical path: tracing observation to actuation," produced the measured spans; this lesson turned them into stability margins, a staleness sawtooth, deadline math, and an injection instrument that supports causal claims. The next lesson, "Real-time chunking: asynchronous execution without discontinuity," is the engineering response — overlap inference with execution so the sawtooth never gets a pause added to it, and fuse consecutive chunks so the fix does not reintroduce the lunge. The signatures you learned to spot here become quantitative in Lesson 5's motion-quality metrics, and the injection sweep you just designed is the harness your capstone's latency-aware scheduler must beat.