Behavioral cloning and distribution shift

Behavioral cloning is the technique you already know best, wearing a lab coat. Take a dataset of expert demonstrations, treat every observation-action pair as a labeled example, and minimize prediction loss with SGD — it is supervised fine-tuning where the tokens are motor commands. The entire modern VLA stack, π₀ and OpenPI (opens in a new tab) included, is behavioral cloning at heart: expressive architectures, but the loss is still “match what the demonstrator did.” Familiar — so why open the learning phase with it? Because closing the loop breaks the one assumption supervised learning stands on, and the way it breaks — quadratically in episode horizon — is arguably the single most important fact about robot learning. You have met this failure before, under the name exposure bias.

Foundations: Defining the Control Abstractions

Before deriving error bounds, we must rigorously define the control-theoretic abstractions that the lesson relies on. The first is the state-only policy. In this context, a state-only policy is a mapping $\pi: \mathcal{S} \to \mathcal{A}$ where the input vector $s \in \mathcal{S}$ consists exclusively of numerical proprioceptive and external sensor data—specifically joint angles, joint velocities, and object pose in the base frame. Crucially, this excludes raw pixel data. This abstraction exists to isolate control logic from perception errors. If the policy fails, we can attribute the failure to the control law rather than a vision backbone misinterpreting a shadow. This separation is essential for the failure taxonomy discussed later.

The second abstraction is teacher forcing in a continuous control context. In standard supervised learning, the model is trained on a fixed dataset. In control, this means the policy is trained on states generated by the expert policy $\pi^*$. During training, the policy never sees its own errors; every input state $s_t$ is the result of the expert's action $a_{t-1}$. This contrasts with free-running deployment, where the policy's output $a_t$ becomes part of the next input $s_{t+1}$ through the physics of the environment. The mismatch between these two regimes is the root cause of distribution shift.

To quantify this shift, we define the tube of states. Let $\tau^*$ be the nominal trajectory generated by the expert. The tube $\mathcal{T}_\delta$ is the set of all states reachable by the expert policy within a small deviation $\delta$ from $\tau^*$. The policy is only guaranteed to perform well within this tube. If the policy's action deviates from the expert's action by more than a threshold, the state may exit the tube, and the training guarantee no longer applies. We define $\varepsilon$ as the probability that the policy's action deviates from the expert's action by more than $\delta$ on the expert's state distribution. This is a discrete error probability, not a continuous error magnitude.

The third abstraction is action chunking. Instead of predicting a single action $a_t$, the policy predicts a sequence of $k$ actions $[a_t, a_{t+1}, \dots, a_{t+k-1}]$ at once. This reduces the effective horizon $T$ in the error compounding formula to $T/k$. By reducing the number of decision points, action chunking reduces the quadratic term in the regret bound. It also matches how humans demonstrate, as teleop actions encode intent over a short horizon rather than just the current frame.

Worked Example: Deriving the Quadratic Bound

Consider a WidowX pick task with $T=250$ steps and a per-step error probability $\varepsilon=0.01$. We want to derive the quadratic bound on the total regret. The key step is to bound the probability of being off-distribution at step $t$. By the union bound, the probability of making an error at any previous step $1 \dots t-1$ is at most $(t-1)\varepsilon$. If an error occurs, the state exits the tube, and the worst-case cost is 1.

Plugging in the numbers: $\varepsilon = 0.01$ and $T = 250$. The bound is $0.01 \cdot \frac{250 \cdot 249}{2} = 311.25$. The maximum possible cost for the task is $T \cdot 1 = 250$. Since the bound (311.25) exceeds the maximum possible cost (250), the bound is vacuous. This means the theory provides no useful information about actual performance; the policy is likely to fail completely. If DAgger reduces the effective error to $\varepsilon' = 0.001$, the bound becomes $0.001 \cdot 31125 = 31.125$, which is well within the maximum cost, indicating a high probability of success.

Imitation as supervised learning — and the assumption it quietly breaks

Set up the notation for the rest of this phase. You teleoperate the WidowX through a task — pick a block, drop it in a bin — logging observation-action pairs at the policy rate. Call the expert (you) $\pi^*$ and the logged dataset $\mathcal{D} = \{(s_i, a_i)\}_{i=1}^{N}$. Behavioral cloning fits a policy by empirical risk minimization, exactly as if this were regression:

Supervised learning's generalization guarantees assume train and test inputs drawn i.i.d. from one distribution — plausible in classification, where inputs are exogenous: nothing the classifier predicts changes which image arrives next. A control loop violates this by construction: the observation at tick $t{+}1$ is physics applied to the command at tick $t$. Write $d_{\pi}$ for the distribution of states visited when $\pi$ closes the loop. Training measured your loss under $d_{\pi^*}$; deployment evaluates you under $d_{\hat{\pi}}$ — a distribution your own errors generate:

Make it concrete with your hardware. A pick episode on the WidowX runs about 25 seconds with the policy ticking at 10 Hz: $T = 250$ sequential decisions, each conditioning the next input through the arm's dynamics. Suppose held-out validation says the policy matches your demonstrations 99% of the time — a number measured on states you steered into. The first 1% event — the gripper 2 cm left of any demonstrated approach — puts the policy somewhere it has never been graded, and the validation number no longer applies. What happens next is the most cited argument in imitation learning.

Compounding error: deriving the quadratic bound

The classic argument, due to Ross and Bagnell, formalizes the snowball. Use the simplest cost model that captures it: at each step the policy matches the expert or makes a mistake; training got the mistake probability down to $\varepsilon$ on the expert's distribution; off that distribution we assume nothing, and each wrong step can incur cost up to 1. The derivation is three lines, each worth internalizing:

Had training states come from the learner's own distribution instead, the per-step guarantee would hold at every visited step and total regret would be $O(\varepsilon T)$ — a factor of $T/2$ better. Plug in your numbers: $\varepsilon = 0.01$, $T = 250$ gives $2.5$ on-distribution versus $\varepsilon\,T(T{+}1)/2 \approx 314$ for BC, past the maximum possible cost of 250. With 1% per-step error the guarantee goes vacuous around step $1/\varepsilon = 100$ — your episode lives 150 steps beyond the edge of the theory. The bound is also tight: Ross and Bagnell exhibit a cliff-walking problem, where one mistake leaves the recoverable region, that realizes the quadratic. Real manipulation sits in between: a closing gripper funnels nearby states into the same grasp, but once the block is knocked over, no demonstrated state resembles the current one.

Autoregressive LLMs have exactly this train/deploy mismatch — teacher forcing during training, self-conditioning during generation — and the community named the resulting drift exposure bias a decade ago. Why do LLMs mostly get away with what breaks robots? Coverage: trillion-token pretraining means the model's own outputs rarely leave the training distribution's support; your 12,500-transition WidowX dataset has no such luxury. Recoverability: a weak token can be semantically absorbed downstream, but physics has no attention mechanism that reinterprets a bad torque. On-policy fine-tuning: RLHF and its variants explicitly train on model-generated states — precisely the fix imitation learning proposed first. That fix has a name.

DAgger: fix the distribution, not the model

DAgger — Dataset Aggregation, from Ross, Gordon, and Bagnell, 2011 (opens in a new tab) — attacks the subscript directly: if training states come from $d_{\pi^*}$ while deployment visits $d_{\hat{\pi}}$, then collect labels on the learner's own states:

1. Train an initial policy $\hat{\pi}_1$ with plain BC on the expert demos.
2. Roll out the current policy $\hat{\pi}_i$ on the robot and record every state it visits — including the awkward ones it steered itself into.
3. Ask the expert to label those visited states: for each recorded $s$, what action would $\pi^*$ have taken?
4. Aggregate the new pairs into the dataset ($\mathcal{D} \leftarrow \mathcal{D} \cup \mathcal{D}_i$) and retrain on everything.
5. Repeat. Under a no-regret analysis, the best iterate satisfies $J(\hat{\pi}) \le J(\pi^*) + O(\varepsilon T)$ — linear, not quadratic.

In simulation, where the “expert” is an oracle controller queryable in microseconds, DAgger works exactly as advertised. On a real robot with a human teleoperator, step 3 collides with human factors. Post-hoc labeling means staring at a frozen mid-failure frame and answering “what exact 7-dim command would you have issued here?” — humans give continuous corrections in context, not calibrated per-state actions out of it, so the labels come back noisy and biased. Real-time labeling means the expert supervises every rollout, hand hovering over the controller — erasing the labor savings that motivated learning, each takeover a safety event on a physical arm. And each outer-loop iteration costs hours of lab time at 25 wall-clock seconds plus a manual reset per rollout. So the field keeps DAgger's idea while approximating its mechanics:

On the WidowX, the last two rows are your workhorses. Recovery collection is cheap: nudge the block 3–5 cm after reset, or start the arm just off the demonstrated approach corridor, then teleoperate the correction. A common practitioner heuristic makes roughly 10–20% of a dataset recovery-flavored — enough that the policy has seen the direction home from the states its own drift will produce. This is DAgger with the human predicting the learner's mistakes instead of observing them, and it is the cheapest robustness lever you have.

Why modern BC works anyway

Given a quadratic lower bound, why do BC-trained systems — up to and including π₀-class VLAs — manipulate objects successfully for minutes at a time? Four levers, all pulled in this phase. First, action chunking. Predict a chunk of $k$ future actions and execute them before re-observing. The compounding argument counts decisions, not timesteps: a 250-step episode with $k = 20$ makes only 13 decisions. Chunking also matches how humans demonstrate — teleop actions encode intent over the next second, not just the current frame, so chunk targets are cleaner labels than per-step Markov ones. This is the core of ACT, lesson 3 — and the exact structure whose latency behavior your capstone will study.

Second, expressive policy classes. Demonstrations are multimodal: confronted with a tall obstacle between gripper and bin, you route left on some demos and right on others, both correct. A deterministic network trained with MSE cannot represent “left or right” — the loss forces it to the conditional mean:

Third, viewpoint choices that shrink the shifted distribution. A wrist-mounted RealSense-class camera makes observations approximately relative — the block looks the same in wrist-cam pixels anywhere in the workspace — so a single demo covers a family of absolute configurations, and the learner's drifted states more often look like training data. Fourth, and dominating everything: coverage. The result repeated across recent large-scale imitation efforts is that data diversity — initial poses, lighting, distractors, recovery segments — moves success rates more than architecture choices do. The quadratic bound never stops being true; broad data makes $\varepsilon$ small on a wider distribution, so the first mistake arrives later and lands somewhere still covered. Remember this hierarchy when tempted by a bigger model in week 8.

The Phase 03 baseline — and the failure taxonomy it exists to expose

Now design the artifact this lesson exists to produce: the deliberately simple baseline you train first and keep forever as an experimental control. The task is a fixed-scene pick: one block, a bin, scripted resets, object pose from a fiducial tag seen by your RealSense-class camera and transformed into the base frame with your Phase 01 calibration. The baseline is state-only on purpose — no images enter the network — which removes the perception confound: when it fails, a vision backbone misreading a shadow is not a suspect. Image-conditioned policies arrive with ACT in lesson 3, and you will want a policy beneath them whose failures you already understand.

The interface: observations are 17 dimensions — 7 joint positions (6 arm joints plus gripper), 7 joint velocities, 3 for object position in the base frame. Actions are 7 absolute joint-position targets at 10 Hz — absolute rather than deltas, because delta policies accumulate integration error along the very drift dimension you want to study. Normalize both sides with per-dimension z-scores computed on the training split only: joint angles span radians while gripper width spans centimeters, and unnormalized MSE silently downweights the gripper — the dimension that decides whether the grasp closes. The network is a 3-layer MLP, 17 → 256 → 256 → 7, about 72k parameters; it converges in under a minute on your RTX workstation from 50 demonstrations (roughly 12,500 pairs). That is the point: iteration speed on the data is worth more than capacity in the model, and a small MLP on clean state goes far further than vision-model intuition suggests.

import numpy as np

OBS_DIM = 17   # 7 joint pos + 7 joint vel + 3 object xyz in base frame
ACT_DIM = 7    # 6 arm joints + 1 gripper, absolute position targets

# WidowX AI published joint limits (rad; gripper in m) -- confirm against your driver config.
JOINT_LOW = np.array([-3.14, 0.00, 0.00, -1.57, -1.57, -3.14, 0.0])
JOINT_HIGH = np.array([3.14, 3.14, 2.36, 1.57, 1.57, 3.14, 0.04])
MAX_STEP = 0.06  # rad per 10 Hz tick: never command a larger jump

def split_by_episode(episodes, val_frac=0.1, seed=0):
    """Split whole episodes, never transitions: adjacent frames are
    nearly identical, so a transition-level split leaks train into val."""
    rng = np.random.default_rng(seed)
    order = rng.permutation(len(episodes))
    n_val = max(1, int(len(episodes) * val_frac))
    val = [episodes[i] for i in order[:n_val]]
    train = [episodes[i] for i in order[n_val:]]
    return train, val

def fit_normalizer(train_episodes):
    """Per-dimension z-score stats from the training split only."""
    obs = np.concatenate([ep["obs"] for ep in train_episodes])
    act = np.concatenate([ep["act"] for ep in train_episodes])
    return {
        "obs_mu": obs.mean(0), "obs_sig": obs.std(0) + 1e-6,
        "act_mu": act.mean(0), "act_sig": act.std(0) + 1e-6,
    }

class Deployer:
    """Wraps the trained net for the 10 Hz control loop."""

    def __init__(self, net, stats):
        self.net = net    # 17 -> 256 -> 256 -> 7 MLP, ~72k params
        self.s = stats

    def act(self, obs, qpos_now):
        z = (obs - self.s["obs_mu"]) / self.s["obs_sig"]
        a = self.net(z) * self.s["act_sig"] + self.s["act_mu"]
        a = np.clip(a, JOINT_LOW, JOINT_HIGH)          # joint limits
        delta = np.clip(a - qpos_now, -MAX_STEP, MAX_STEP)
        return qpos_now + delta                        # rate-limited command

Your baseline will fail — plan on 40–70% success from 50 demos — and the failures are the curriculum, but only if you can tell them apart: the fix for a coverage gap (collect demos there) is the opposite of the fix for a perception fault (leave the policy alone). From the first evaluation session, every failed episode gets one primary label from this taxonomy plus a sentence of evidence, written at reset time. This habit is the raw material for lesson 2's data-quality work and lesson 6's statistics.

Notice how much weight one probe carries: the per-timestep distance from visited states to the nearest training-set state, computed in normalized state space. A step change says coverage; a slow monotonic ramp says drift; an in-band curve during a miss says perception; none of the above with bad tracking error says hardware. It is a crude density estimate, but cheap at 17 dimensions and 12,500 reference points, and it turns “out of distribution” into a number per tick — exactly the instrumentation this course keeps insisting on.

Where this goes next: the previous lesson, Linear systems, LQR, and feedback — the EE refresher, gave you policies derived from a model, with stability margins computable before touching the arm. Behavioral cloning trades the model for data and the guarantee for a distribution match that deployment immediately violates — this lesson gave you the theory of that violation and the baseline that will exhibit it. Next, Demonstration data: collection, quality, and splits turns the 50-demo dataset sketched here into a designed artifact: coverage plans, recovery-clip budgets, and the episode-level splits your evaluation will stand on.