Time is a sensor: clocks, timestamps, and latency

You already believe that latency matters — you have spent years shaving milliseconds off token pipelines. Robotics raises the stakes in one specific way: the world does not pause while you compute. An LLM user waiting 300 ms longer gets a slightly worse experience. A robot acting on a 300 ms-old image of a moving gripper acts on a world that no longer exists.

Foundations: Control Frequency, Staleness, and the Critical Path

Before decomposing latency, we must distinguish the two distinct temporal rhythms of a robotic system: the Control Frequency and the Inference Frequency. The control loop, typically running at a high rate such as 500 Hz (a period $T_c = 2$ ms), is responsible for stability and torque regulation. It does not wait for the policy. Instead, it executes a continuous command stream, often interpolating between the last two policy outputs or holding the last command constant. The Inference Frequency (e.g., 10 Hz, period $T_i = 100$ ms) is the rate at which the high-level policy generates new target states. Because $T_i \gg T_c$, the controller effectively 'holds' or 'interpolates' the policy's intent across many control cycles. This decoupling is essential: if the control loop blocked on inference, the robot would stall, violating the stability requirements of the physical plant.

This decoupling introduces the concept of Staleness. We define the staleness of an observation as the time difference between when the command is executed and when the observation was captured: $\Delta t = t_{\text{command}} - t_{\text{capture}}$. If $\Delta t$ exceeds the control period $T_c$, the system is operating on outdated state. More critically, if $\Delta t$ approaches or exceeds the inference period $T_i$, the robot may be acting on data from the previous policy cycle. In such cases, the effective latency is not just the measured pipeline delay, but the sum of the pipeline delay and the time until the next policy update. This 'frequency mismatch' can double the perceived latency in the worst case, a phenomenon that simple average-latency metrics completely obscure.

Furthermore, the total latency is not simply the sum of all stages. In modern pipelines, stages often overlap (pipelining). For instance, while the GPU processes frame $k$, the CPU may be transferring frame $k+1$. Therefore, the relevant metric for control stability is the Critical Path Latency: the longest non-overlapping chain of dependencies from capture to actuation. We must also distinguish this from Throughput, which is the rate of frame processing. A system can have high throughput but high critical path latency if the pipeline is deep. Finally, we define Jitter formally as the standard deviation of the latency distribution, or more practically, the difference between the 99th and 50th percentiles ($p99 - p50$). High jitter is dangerous because it introduces variable phase lag into the feedback loop, reducing the phase margin and potentially destabilizing the controller, even if the average latency is low.

Worked Example: Quantifying Staleness and Jitter

Consider a robotic arm with an end-effector moving at a constant speed $v = 0.5$ m/s. The policy runs at 10 Hz ($T_i = 100$ ms), and the control loop runs at 500 Hz ($T_c = 2$ ms). We measure the camera-to-command latency distribution and find $p50 = 50$ ms and $p99 = 150$ ms. We will calculate the physical error, the number of control cycles affected, and the impact of jitter.

First, the physical error. At the 99th percentile, the observation is 150 ms old. The end-effector has moved 75 mm since that observation was captured. If the target is a small object, this error is catastrophic. Second, we examine the control cycle impact. The 150 ms latency spans $150 / 2 = 75$ control cycles. During these 75 cycles, the controller is executing commands based on a world state that is 75 cycles old. Third, we analyze the frequency mismatch. The policy period is 100 ms. Since the latency (150 ms) is greater than the policy period, the robot is always acting on data from the previous policy cycle. In the worst case, the effective latency is the sum of the pipeline latency and the time until the next policy update, potentially reaching $150 + 100 = 250$ ms. This effectively doubles the staleness error to 125 mm.

Finally, we consider jitter. The jitter is $p99 - p50 = 150 - 50 = 100$ ms. This 100 ms variation in delay means the phase lag of the feedback loop varies by 100 ms from one cycle to the next. For a high-gain controller, this variable phase lag can push the system into instability, causing oscillations. The controller cannot 'learn' to compensate for a delay that changes randomly by 100 ms. This example illustrates why the p99 latency and the jitter are more critical metrics than the p50 latency for ensuring safe and stable operation.

Clock domains: how many clocks does your robot have?

Count the clocks in a minimal manipulation stack and you usually find at least five. Each one ticks at a slightly different rate, starts from a different zero, and some of them reset when a device reboots. Any two timestamps you compare must first be mapped into a common domain — otherwise the comparison is noise.

Cross-device time is harder. The camera stamps a frame with its own clock; your host receives it later. Either use the vendor's clock-mapping API (librealsense exposes RS2_FRAME_METADATA_SENSOR_TIMESTAMP plus a host-mapped domain), or estimate the offset yourself with a round-trip handshake — the same idea as NTP, and the same idea as the request/response clock alignment you have done for distributed tracing.

Timestamp at capture, not at arrival

The single most common instrumentation mistake in robot data collection: stamping an image when your Python callback receives it. Between photons and callback sit exposure (1–30 ms), sensor readout, USB or GigE transfer, driver buffering, and scheduler jitter. That gap is not constant — it breathes with load. If you train a policy on arrival-stamped data, you teach it a systematically wrong model of when the world looked like that.

Your logging schema should record both — capture time for physics, arrival time for diagnosing the transport. When they disagree by more than your budget allows, that is a measurement, not an annoyance: it tells you where the pipeline is congesting.

The latency budget: decompose the loop

Write the observation-to-motion path as a sum of stages, each independently measurable. This is exactly a trace span breakdown; the only new stages are the physical ones at the end.

• Sensing — exposure midpoint to frame available in host memory. Typically 15–40 ms for USB RGB at 30 fps.
• Compute — preprocessing, policy inference, action decoding. For a π₀-class VLA on a workstation GPU, expect tens to hundreds of milliseconds; for the classical stack of Phase 02, sub-millisecond.
• Dispatch — serialization, network hop to the control process, UDP command send to the arm's iNerve controller. The iNerve real-time loop runs at 500 Hz, so a command waits at most one 2 ms cycle before being relayed over CAN FD to the joint controllers.
• Actuation — joint FOC-loop response plus mechanical settling. This stage exists in no software profiler; you measure it by commanding a step and watching encoder feedback.

Distributions, not averages

You already know why p99 beats mean from serving dashboards. In a control loop the argument is sharper: the loop runs at a fixed cadence, so a p99 latency excursion is not one slow request among many — it is a guaranteed periodic event that arrives every few seconds of operation, each time handing the robot a stale action. Jitter (the spread between p50 and p99) is often more damaging than the median itself, because a policy or scheduler can compensate for a known constant delay far more easily than for a random one.

import time
import numpy as np

class LatencyProbe:
    """Accumulates per-stage timestamps for one control tick."""

    STAGES = ("capture", "preproc", "infer", "dispatch")

    def __init__(self):
        self.records = []   # one dict per tick
        self.tick = {}

    def mark(self, stage: str) -> None:
        # CLOCK_MONOTONIC: safe for intervals, never steps backward
        self.tick[stage] = time.monotonic()

    def commit(self, capture_time: float) -> None:
        """capture_time: exposure-midpoint time mapped to host monotonic."""
        rec = {"capture": capture_time, **self.tick}
        self.records.append(rec)
        self.tick = {}

    def report(self) -> dict:
        end_to_end = np.array(
            [r["dispatch"] - r["capture"] for r in self.records]
        ) * 1e3  # ms
        p50, p95, p99 = np.percentile(end_to_end, [50, 95, 99])
        return {
            "n": len(end_to_end),
            "p50_ms": round(float(p50), 2),
            "p95_ms": round(float(p95), 2),
            "p99_ms": round(float(p99), 2),
            "jitter_ms": round(float(p99 - p50), 2),
        }

Run this probe for a few thousand ticks under realistic load — cameras streaming, logger writing, visualization on — and save the raw arrays, not just the summary. The histogram shape matters: bimodal latency usually means a queue is oscillating between empty and full somewhere in the pipeline.

What staleness costs, in millimeters

Latency numbers become decisions only when converted into physical units. If the end effector moves at speed $v$ and the policy acts on an observation that is $\Delta t$ old, the world has moved on by:

This single formula explains most of Phase 05 before you get there: why action chunking exists (amortize inference latency over many actions), why chunks go stale (the formula applies to every action in the chunk), and why the capstone's adaptive scheduler is worth researching (it manages exactly this error term). You are building the measurement layer for that research right now.

Where this goes next: Coordinate frames and rigid transforms gave the robot a consistent notion of where; this lesson gives it a consistent notion of when. The next lesson, ROS 2 essentials: nodes, topics, QoS, and bags, supplies the transport that carries these stamped observations between processes — and Lab 0: bring-up, safety, and the episode logger combines both into the episode logger you will use for every dataset in this course, where the numbers you measured here become the baseline latency report that Phase 05 treats as its experimental control.