Demonstration data: collection, quality, and splits

In your LLM work you watched one truth harden: architecture churn washes out, data curation compounds. Robot learning obeys the same law with a crueler exchange rate. The previous lesson showed that a behavioral-cloning policy is, at best, a mirror of its dataset — and the next three lessons (ACT, Diffusion Policy, offline RL) all train on the same demonstrations you are about to collect. Retraining is an overnight job on your RTX workstation; recollecting is a week of your hands on a leader arm — often impossible once the scene is torn down and your teleop habits have drifted. Dataset decisions outlive model decisions. This lesson is about making them deliberately.

The machinery exists: the Phase 01 episode logger records tick-aligned observations, commands, acknowledgements, and faults on one monotonic clock, with per-episode metadata. What it cannot tell you is which episodes to record — a design problem with four parts: how demonstrations are produced (teleoperation), what they span (coverage), which flaws you accept (recovery), and how they are partitioned (splits).

Data is the product; the dataset card is the contract

Start with the cost asymmetry, using your own Lab 0 numbers. At 40 s per episode, a freehand 90 s reset yields 27 episodes/hour and a scripted 15 s reset yields 65 — so a 300-episode dataset costs 5 to 11 hours of your undivided attention. Training an ACT-scale model on it costs a few unattended GPU-hours. Over this phase you will run dozens of training jobs against a dataset collected roughly once: many cheap experiments amortized over one expensive corpus. That is pretraining economics: the corpus deserves more design effort than any single run.

That design effort lives in the dataset card, drafted before episode one — the spirit of Datasheets for Datasets (opens in a new tab), which argued datasets should carry provenance like models carry model cards. Up front it is a spec; afterward, an honesty ledger. Six sections cover a single-task manipulation dataset:

• Task and success criterion — one sentence a second engineer could adjudicate: “mug upright on the saucer, gripper retracted, within 60 s.”
• Embodiment and sensing — arm, camera placements, calibration hash, control rate — and which of these may vary.
• Teleoperation rig — device, mapping, and its measured glass-to-glass latency (you built the probe in Phase 01).
• Coverage axes — every axis you will vary, with range and resolution; equally load-bearing, every axis deliberately held fixed.
• Recovery quota — the fraction of episodes demonstrating correction rather than clean success, and which perturbations they sample.
• Split plan — which objects, backgrounds, and pose bands are held out, and where those items physically live.

Everything below fills in those sections. The card is not paperwork: committing it before collection is registering an eval before you train against it — a theme the splits section hits hard.

Teleoperation: the policy imitates the operator-rig loop, latency included

The BC lesson framed the learned policy as an approximation of the demonstrator's conditional action distribution. Be precise about what that demonstrator is: not your intent, but the closed-loop output of a human driving an arm through a rig with latency. Whatever that loop does — hesitate, creep, stair-step — your policy will faithfully reproduce. Derive the dominant artifact: model the precision phase of a demonstration as the operator acting as a proportional controller on perceived position error $e$, delayed by total loop latency $\Delta t$ — visuomotor reaction time plus everything the rig adds.

Read off the stability limit. Gain crossover is where $|L(j\omega)| = 1$: since $|L| = k/\omega$, that is $\omega_c = k$. Phase at crossover is the integrator's $-\pi/2$ plus the delay's $-\omega_c \Delta t$, and the loop destabilizes when the total reaches $-\pi$:

The human contributes an irreducible 150–250 ms; the rig adds on top. A leader-follower pair adds only follower tracking delay (~5–30 ms: the WidowX AI's UDP command path and 500 Hz controller cost about 2 ms; the follower's mechanical lag dominates) — the operator watches the scene, not a screen; a 3D mouse through a monitor view adds the camera-plus-display path (100–250 ms); streamed VR adds 150–300 ms more. The gain cap becomes a speed cap near contact — the operator cannot outrun the error they can still correct within one delay:

So rig choice is a data-quality decision, not an ergonomics preference. Cut $\Delta t$ from 350 ms to 220 ms with a leader-follower rig and the same operator approaches nearly twice as fast, with fewer stop-and-go corrections — trajectories measurably easier to imitate. It is no accident that the ACT / ALOHA line of work (opens in a new tab) built joint-copy leader arms before building the model. Each rig also stamps its geometry into the data:

The last teleop decision is the consistency-versus-diversity tension. One operator executing one strategy yields a near-unimodal action distribution — exactly what an L2-trained BC policy can absorb. Multiple operators, or one operator improvising, yield a multimodal distribution, and the previous lesson showed what MSE does to modes: it averages them. The working rule for this phase: put diversity in the world, keep consistency in the strategy. Vary poses, objects, and lighting freely; grasp the mug the same way every time. Lesson 4 brings policy classes built for multimodality — until then, strategy diversity is a choice, never an accident.

Foundations: Defining the Loop, the Radius, and the Grid

Before deriving stability limits or calculating coverage, we must rigorously define the physical quantities that govern data quality. The primary variable is the total loop delay, denoted $\Delta t$. This is not a single number but the sum of three distinct components: human visuomotor reaction time, signal transmission and processing latency, and mechanical actuation lag. In teleoperation, this is often called 'glass-to-glass' latency: the time from the operator moving the leader arm to the follower arm physically responding. A typical leader-follower rig has a $\Delta t$ of 20–50 ms, whereas a 3D mouse through a monitor adds 100–250 ms. The second critical variable is the generalization radius, $\varepsilon$. This is the maximum state deviation from a demonstrated trajectory where the policy’s action remains valid. It is not a fixed constant; it must be estimated by testing the policy on slightly perturbed states until failure. For a grasp task with a 3 cm tolerance, $\varepsilon$ might be 1.5 cm. Finally, we define the operator gain, $k$. This represents the operator's 'stiffness' or corrective gain. A high $k$ means the operator corrects errors aggressively; a low $k$ means they correct slowly. Skilled operators self-regulate $k$ to stay stable within the delay constraints.

The stability of the operator-rig loop depends on the phase margin, which is the 'safety buffer' before oscillation. A margin of 45° is standard; below 0°, the system oscillates. The delay $\Delta t$ eats into this margin. The coverage calculation relies on discretizing the state space into cells of width $2\varepsilon$. The total number of cells, $C$, is the product of the number of cells along each axis. If $N$ episodes are collected uniformly, the probability that a specific cell is missed is $(1 - 1/C)^N$. This is a variant of the coupon collector problem. The expected coverage is $1 - (1 - 1/C)^N \approx 1 - e^{-N/C}$. This formula shows why 63% coverage (at $N=C$) is often insufficient: it leaves significant 'holes' in the state space where the policy will hallucinate actions.

Worked Example: Calculating Coverage and Speed Limits

Consider a 'Pick and Place' task with a 30 cm x 30 cm workspace and 180° yaw. The generalization radius is $\varepsilon = 1.5$ cm (linear) and $7.5^\circ$ (angular). The total latency is $\Delta t = 220$ ms. The tolerance is $\varepsilon_{\text{tol}} = 3$ mm. First, calculate the number of coverage cells. Linear cells per axis: $30 \text{ cm} / (2 \times 1.5 \text{ cm}) = 10$. Angular cells: $180^\circ / (2 \times 7.5^\circ) = 12$. Total $C = 10 \times 10 \times 12 = 1200$ cells. To cover an expected fraction of 95%, uniform independent sampling needs $N = -C \ln(1 - 0.95) \approx 1200 \times 3 = 3600$ episodes. This does not cover every cell: about $0.05C=60$ cells remain empty on average. Full coverage is the stricter coupon-collector problem and costs about $C\ln C \approx 8500$ episodes. If we fix yaw, $C = 100$; 95% expected coverage needs about 300 episodes, which is feasible. Next, calculate the maximum approach speed: $v_{\max} = \varepsilon_{\text{tol}} / \Delta t = 3 \text{ mm} / 0.22 \text{ s} \approx 13.6 \text{ mm/s}$. If your free-space speed is 200 mm/s, you are 15x slower near the object. This 'slow-down' is a signature artifact in the data.

Coverage: pay for axes deliberately, and demonstrate recovery

List the axes a tabletop pick-and-place can vary over: object pose $(x, y, \theta)$, object instance, distractor count and placement, lighting, background surface, language prompt phrasing (cheap to log now; Phase 04's π₀/OpenPI fine-tuning will want it), and initial arm configuration. Camera pose is an axis too — one you will almost certainly hold fixed, which is fine if the card says so. The BC lesson gave you a generalization radius: the policy interpolates reliably within roughly $\varepsilon$ of demonstrated states. Discretize each axis into cells of width $2\varepsilon$ and coverage becomes counting:

You will not collect 1,350 episodes, so you will not grid — you sample. Randomize the axes jointly each episode (a scripted reset reading poses from a pre-generated list, on the Lab 0 fixtures). One line gives the coverage rate: a given cell receives none of $N$ uniform samples with probability $(1 - 1/C)^N$, so expected coverage is

Three consequences. First, at a realistic 300–500 episode budget you can afford dense coverage of pose plus sparse sampling of two or three discrete conditions — every axis must earn its place in the card. Second, lean on representation for nuisance axes: a pretrained backbone (π₀/OpenPI encoders, or ACT's ImageNet-initialized ResNet) already carries lighting and texture invariance — sample enough to verify transfer, not to teach it. Third, log the sampled cell (start_pose_cell) so coverage is auditable with a groupby, not a rewatch.

Now the beat that matters most: recovery demonstrations. The previous lesson's core result was compounding error — the policy drifts off the demonstrated manifold, where it has no supervision. Flawless demonstrations concentrate all their probability mass exactly on that manifold; they contain zero information about the way back. Inject it by hand: start episodes with the gripper misaligned by 2–5 cm, nudge the mug mid-reach, release a grasp early and demonstrate the re-grasp. This is DAgger (opens in a new tab) with you playing both roles — generating the off-distribution states and labeling them with corrections — minus the iteration loop: imperfect, but nearly free. A working quota: 20–30% of episodes carry is_recovery_demo: true, targeting perturbations your BC baseline's failures predict.

Freeze the splits before you train

Your LLM reflex — hold out a random 10% and report loss — is actively misleading here, because 300 robot episodes are nothing like 300 i.i.d. samples. Episodes within a session share lighting drift, actuator temperature, exposure state, object instances, and your own warm-up curve. A random episode-level holdout is a near-duplicate of the training set; its low validation loss certifies interpolation, not generalization. The fix: split at the right granularity (sessions and object instances, not episodes), and hold out along the axes you actually claim generalization over.

The design has three tiers. Train is everything not otherwise claimed. Validation is a session-level random holdout (~10%) for early stopping and hyperparameters — it measures optimization health, nothing else. The held-out suite is defined in the world, not the filesystem: at least one object instance appearing in zero episodes (including as a distractor), one background never seen, and a band of edge-of-range poses excluded from collection. Held-out items live in a physically labeled box under the bench — a physical holdout means leakage requires a deliberate act, not an indexing bug. Lesson 6 turns this suite into defensible statistics; today's job is ensuring something uncontaminated survives.

• Reset leakage — fixture tape marking start poses is visible during “novel layout” evals; the policy localizes off it. Keep fixtures outside the camera frustum.
• Distractor leakage — the held-out mug used as clutter in training scenes. The assertion below scans distractor lists, not just target objects.
• Operator leakage — training demos from operator A, eval runs by operator B: you measure operator shift, not generalization.
• Session-time leakage — validation drawn from session ends: dimmer light, hotter actuators, tired operator. Split by session; interleave conditions.
• Calibration-event leakage — a mid-collection camera bump creates two extrinsics regimes; a split correlated with the bump silently tests calibration robustness. Assert balance across calibration hashes.

import json
import hashlib
from pathlib import Path

import numpy as np

HELD_OUT_OBJECTS = {"mug_gray_04"}
HELD_OUT_BACKGROUNDS = {"cloth_gray"}
VAL_FRACTION = 0.10
SEED = 20260720


def load_metas(root):
    metas = []
    for meta_path in sorted(Path(root).glob("*/meta.json")):
        with open(meta_path) as f:
            m = json.load(f)
        m["episode_id"] = meta_path.parent.name
        metas.append(m)
    return metas


def assert_no_leakage(split_name, metas):
    for m in metas:
        seen = set([m["object_id"]] + m.get("distractor_ids", []))
        overlap = seen & HELD_OUT_OBJECTS
        assert not overlap, (
            "held-out object " + str(overlap) + " appears in "
            + split_name + " episode " + m["episode_id"])
        assert m["background_id"] not in HELD_OUT_BACKGROUNDS, (
            "held-out background in " + split_name
            + " episode " + m["episode_id"])


def freeze_splits(root):
    metas = load_metas(root)
    held_out = [m for m in metas
                if m["object_id"] in HELD_OUT_OBJECTS
                or m["background_id"] in HELD_OUT_BACKGROUNDS]
    rest = [m for m in metas if m not in held_out]

    # Split val at the SESSION level, never the episode level:
    # episodes within a session share lighting drift, actuator
    # temperature, exposure state, and operator warm-up.
    sessions = sorted(set(m["session_id"] for m in rest))
    rng = np.random.default_rng(SEED)
    n_val = max(1, int(round(len(sessions) * VAL_FRACTION)))
    val_sessions = set(rng.choice(sessions, size=n_val, replace=False))

    train = [m for m in rest if m["session_id"] not in val_sessions]
    val = [m for m in rest if m["session_id"] in val_sessions]
    assert_no_leakage("train", train)
    assert_no_leakage("val", val)

    manifest = {
        "train": sorted(m["episode_id"] for m in train),
        "val": sorted(m["episode_id"] for m in val),
        "held_out": sorted(m["episode_id"] for m in held_out),
        "held_out_objects": sorted(HELD_OUT_OBJECTS),
        "held_out_backgrounds": sorted(HELD_OUT_BACKGROUNDS),
        "val_sessions": sorted(val_sessions),
        "seed": SEED,
    }
    blob = json.dumps(manifest, sort_keys=True).encode()
    manifest["sha256"] = hashlib.sha256(blob).hexdigest()
    with open("split_manifest.json", "w") as f:
        json.dump(manifest, f, indent=2)
    print("frozen:", manifest["sha256"][:16],
          len(train), "train /", len(val), "val /",
          len(held_out), "held out")
    return manifest

Two properties matter more than the code. The split is a pure function of metadata plus a seed — it never opens an image, so re-run the leakage assertions after every session, when a violation costs one episode, not the dataset. And the manifest, not the episode files, owns split membership: episodes never learn their split, so re-splitting never rewrites data. Commit split_manifest.json and its hash before the first training job; if you change the split after seeing held-out numbers, the old manifest in git history is the receipt.

Format, per-episode metadata, and how much is enough

On format, do not innovate. The community has converged on LeRobot-style (opens in a new tab) episodic datasets: per-step records (camera streams, proprioception, action, timestamps) grouped into episodes, per-episode metadata alongside, dataset-level info (control rate, feature schemas) on top. The ACT and Diffusion Policy reference trainers and the OpenPI (opens in a new tab) fine-tuning stack all consume this shape, so conforming buys every downstream tool. Your Phase 01 logger maps nearly field-for-field — a renaming job, because the information was captured at collection time. What Phase 01 did not anticipate, this lesson adds:

{
  "episode_id": "ep_000214",
  "schema_version": 4,
  "task": "pick the mug and place it on the saucer",
  "prompt_variant": "put the mug onto the small plate",
  "operator": "yang",
  "session_id": "2026-07-24_pm",
  "git_sha": "9f31c2a",
  "calibration_hash": "c41e6a88",
  "teleop_rig": "leader_follower_widowx_ai",
  "teleop_latency_ms_p50": 22,
  "object_id": "mug_blue_01",
  "distractor_ids": ["bowl_white_02", "spoon_steel_01"],
  "background_id": "cloth_beige",
  "lighting_id": "overhead_led_full",
  "start_pose_cell": [3, 1, 4],
  "reset_type": "scripted",
  "is_recovery_demo": true,
  "recovery_kind": "regrasp_after_drop",
  "outcome": "success",
  "duration_s": 41.7
}

The inclusion rule: metadata must suffice to (a) recompute any split without decoding an image, (b) slice failure rates by condition — success by lighting_id × is_recovery_demo should be a two-line groupby — and (c) re-stage the episode's world. Note the absences: no split label (that lives in the manifest), and no free-text notes field doing load-bearing work, because unstructured notes are where information goes to be unqueryable.

Finally: how many demonstrations? Honest anchors: the ACT paper (opens in a new tab) reports 80–90% success on fine bimanual tasks from roughly 50 demos per task under a narrow, consistent distribution; Diffusion Policy's (opens in a new tab) real-robot tasks sit on the order of 100–200; π₀-style fine-tuning is typically quoted in hours of data — a few hundred episodes. The through-line: tens of demos buy competence inside the collection distribution; volume beyond that buys coverage of the varied axes, with roughly logarithmic returns, because coverage grows like $1 - e^{-N/C}$. Doubling N from 200 to 400 with unchanged axes mostly re-covers cells you already own.

Which yields the stopping rule — a loop, not a number: collect 50, train, deploy, autopsy, then collect what the autopsy names. Train overnight, execute 20 evaluation trials, slice failures by metadata. Failures clustered in un-collected cells or novel conditions: add coverage or an axis. Confident-but-wrong actions at demonstrated states: add targeted recovery demos. Smooth motion but missed grasps: the model class is the bottleneck — the next two lessons are the fix. Stop collecting when the autopsy stops naming a data fix; an open-loop “collect 500 then train” plan spends its second 250 episodes blind.

Where this goes next: Behavioral cloning and distribution shift explained why a cloned policy fails away from its data; this lesson decided where “away” begins — the card, coverage plan, recovery quota, and frozen manifest every experiment in this phase stands on. Next, ACT: action chunking with transformers trains on exactly these episodes, and today's artifacts return as modeling questions: the chunk length that must span your operator's stop-and-go corrections, and the CVAE machinery ACT uses to survive the demonstration variability you just budgeted.