Evaluating policies: statistics you can defend

You have spent your career around evaluations where $n$ is effectively infinite: MMLU has about 14,000 questions, so a 70% score carries a 95% interval of roughly ±0.8 points, and a serving A/B test accumulates a million requests before lunch. Robot evaluation inverts every one of those instincts. Each sample costs two to four minutes of physical labor — reset, run, score, save the video — so $n = 20$ is a serious evaluation and $n = 100$ is a full day at the bench. The statistics you could ignore at $n = 10^6$ become the entire game at $n = 20$; this lesson is the difference between a claim that survives review and a coin-flip anecdote with a percent sign on it.

Statistical Foundations: Exchangeability, Score Tests, and Variance Structure

Before applying the formulas for intervals and power, we must clarify the probabilistic assumptions that make them valid. The binomial model assumes trials are independent and identically distributed (i.i.d.). In robotics, physical trials are rarely independent because they share hardware state, lighting, and operator fatigue. We therefore rely on a weaker condition: exchangeability. A sequence of random variables $Y_1, \dots, Y_n$ is exchangeable if their joint probability distribution is invariant to any permutation of the indices. That is, $P(Y_1=y_1, \dots, Y_n=y_n) = P(Y_{\pi(1)}=y_1, \dots, Y_{\pi(n)}=y_n)$ for any permutation $\pi$. While exchangeability does not guarantee independence, it ensures that the order of trials does not systematically bias the outcome distribution. Blocking restores conditional exchangeability: given a specific block (e.g., a specific object instance and time slot), the trials within that block are treated as exchangeable, allowing us to aggregate results across blocks without violating the statistical model.

The Wilson interval is often presented as an algebraic inversion, but its justification is probabilistic. It is derived from the score test. For a null hypothesis $H_0: p = p_0$, the score test statistic is $Z = \frac{\hat{p} - p_0}{\sqrt{p_0(1-p_0)/n}}$. Under $H_0$, $Z$ follows a standard normal distribution. The Wilson interval is the set of all $p_0$ values for which the score test does not reject $H_0$ at level $\alpha$. This is why the true parameter $p$ appears in the denominator of the standard error: the test evaluates the plausibility of a specific $p$ using the variance expected if that $p$ were true. In contrast, the Wald interval uses the estimated variance $\hat{p}(1-\hat{p})/n$, which is unstable when $\hat{p}$ is near 0 or 1. The score test inversion keeps the variance anchored to the hypothesized value, yielding better coverage properties at small $n$.

When comparing two policies, the variance of the difference in success rates depends critically on the experimental design. For unpaired designs, where trials for Policy A and Policy B are independent, the variance of the difference $\hat{p}_A - \hat{p}_B$ is $\text{Var}(\hat{p}_A - \hat{p}_B) = \frac{p_A(1-p_A)}{n_A} + \frac{p_B(1-p_B)}{n_B}$. If we assume $p_A \approx p_B \approx \bar{p}$ and $n_A = n_B = n$, this simplifies to $\frac{2\bar{p}(1-\bar{p})}{n}$. However, for paired designs, where both policies are tested on the same initial conditions, the outcomes are correlated. Let $\rho$ be the correlation between the outcomes of A and B on the same condition. The variance of the difference becomes $\text{Var}(\hat{p}_A - \hat{p}_B) = \frac{p_A(1-p_A) + p_B(1-p_B) - 2\rho\sqrt{p_A(1-p_A)p_B(1-p_B)}}{n}$. Since $\rho > 0$ for paired trials (shared difficulty), the variance is strictly smaller than in the unpaired case, increasing statistical power.

Worked Example: Sizing a Paired vs. Unpaired Evaluation

Consider a pick-and-place task where exploratory runs suggest Policy A (ACT) has a success rate of $p_A = 0.70$ and Policy B (BC) has $p_B = 0.50$. We want to detect this 20-point difference with 80% power at a two-sided $\alpha = 0.05$. The critical values are $z_{\alpha/2} = 1.96$ and $z_{\beta} = 0.84$. The pooled success rate is $\bar{p} = 0.60$.

Solving for $n$ in the unpaired case: $0.20 = 2.80 \sqrt{0.48/n} \implies 0.0714 = \sqrt{0.48/n} \implies 0.0051 = 0.48/n \implies n \approx 94$. Thus, we need 94 trials per arm (188 total) for an unpaired design. Now consider a paired design where the correlation between A and B outcomes is $\rho = 0.5$. The variance of the difference is $\text{Var}(A-B) = \frac{0.24 + 0.24 - 2(0.5)(0.24)}{n} = \frac{0.24}{n}$. The standard error is $\sqrt{0.24/n} \approx 0.49/\sqrt{n}$.

Solving for $n$ in the paired case: $0.20 = 2.80 \sqrt{0.24/n} \implies 0.0714 = \sqrt{0.24/n} \implies 0.0051 = 0.24/n \implies n \approx 47$. Pairing reduces the required sample size from 94 to 47 trials per arm, saving 4.7 hours of bench time. This demonstrates the substantial efficiency gain of paired designs when task difficulty varies across initial conditions.

Twenty trials run by hand are still a binomial experiment

Formalize the session. One trial: place the scene into a specified initial condition, hand control to the policy, score a binary outcome against a rubric. Model the outcome as a Bernoulli draw with unknown success probability $p$ — the number you actually care about — and the session as $n$ such draws. The binomial model assumes independent, identically distributed trials, which on a robot is not automatic: the reset script and the blocking discipline later in this lesson are what make trials exchangeable. Grant it for now, and the session compresses to two numbers: $k$ successes out of $n$.

Put your numbers in. At $n = 20$ and a true success rate of $p = 0.7$, the standard error is $\sqrt{0.21/20} \approx 0.10$ — ten percentage points. Evaluate the same frozen checkpoint twice, 20 trials each time, and the scores differ by 20 or more points about one time in six with nobody doing anything wrong: a 65% Tuesday and an 80% Wednesday is the binomial doing exactly what it says. Every intuition you carry from benchmarks where a 2-point delta is a headline is calibrated four orders of magnitude away from where you now live.

The Wilson interval: what to write after the ±

The textbook interval — the one every plotting library defaults to — is the Wald interval: take $\hat{p}$, plug it into the standard-error formula in place of the unknown $p$, and go two standard errors each way.

At robot-scale $n$ this fails in ways that matter. First, the boundary: score 10 out of 10 and the Wald interval is $1.0 \pm 0$ — width zero, a claim of perfection from ten samples. Second, coverage: the interval is built from an estimate of its own uncertainty, and coverage studies show the nominal 95% Wald interval covers the true $p$ well under 90% of the time across much of the $n \le 50$ regime you operate in. It is systematically overconfident precisely when you have the least data.

The Wilson score interval fixes this by asking the inverse question: which true values of $p$ could plausibly have generated the observed $k$? Keep the true $p$ inside the standard error where it belongs, demand that $\hat{p}$ sit within $z$ standard errors of it, and solve for $p$. Squaring both sides turns the condition into a quadratic:

The center term rewards a close read: $(k + z^2/2)/(n + z^2)$ is the empirical rate after adding $z^2/2 \approx 1.9$ phantom successes and 1.9 phantom failures — Laplace smoothing, the same pseudo-count trick that smooths n-gram language models, pulling the estimate toward $1/2$ exactly where the binomial is most uncertain. At $k = 10, n = 10$ the Wilson 95% interval is $[0.72, 1.00]$: a perfect run is “consistent with a true rate as low as 72%.” In practice, call statsmodels' proportion_confint (opens in a new tab) with method="wilson"; the exact Clopper–Pearson interval is the conservative, strictly wider alternative reviewers sometimes request. Wilson is the sensible default at robot-scale $n$.

Sit with this table until it restructures your expectations. A 20-trial evaluation — an hour of focused bench work — localizes the success rate to a 37-point window. Reaching the ±3-point precision your benchmark instincts treat as normal takes on the order of a thousand trials: 50 hours of hand-run evaluation per condition. Nobody does that for a single checkpoint, so every robot success rate you have ever read was measured through a window this wide or wider. The table also works in reverse: decide what claim you need, then read off the $n$ and the hours it costs.

Comparing two policies without publishing noise

The question Phase 03 has been building toward: does ACT actually beat your state-only BC baseline on the WidowX pick-and-place, or did you watch two noisy binomials cross? The standard test for two independent success counts is the two-proportion $z$-test, which compares the observed gap against the noise expected if both policies shared one success rate:

Running the test after the fact is the wrong order of operations. The professional move — capacity-planning the load test before running it — is to compute, before any trials, the smallest true difference the experiment could reliably detect. Under a true gap $\Delta$, the observed difference is approximately Gaussian around $\Delta$; you reject the null when it exceeds $z_{\alpha/2}$ noise units; for rejection to happen with probability $1-\beta$ (the power), $\Delta$ must clear a threshold. The derivation is three lines:

Now read a typical robot-learning results table with this in hand: 10 tasks, 10–25 trials per task per method. Per task, the MDE is 40-plus points, so a single-task delta of 10 or 20 points is uninterpretable on its own — a fraction of published per-task “wins” are sign flips waiting to happen. Papers escape this legitimately only by aggregating: 10 tasks times 25 trials is 250 trials per method, enough for coarse aggregate claims of roughly 10 points. The failure mode to avoid in your own work is the reverse inference — 20 trials per arm, a 15-point gap, written up as an improvement. That is publishing noise, and the MDE table says so before you touch the arm.

You can buy back power without buying trials by pairing. Run both policies on the identical list of initial conditions — same object set, same scripted reset, same printed pose grid — and compare outcomes pair by pair. This is why you evaluate two LLM checkpoints on the same prompt set rather than independent samples: shared conditions subtract trial-difficulty variance from the comparison. The paired analysis is McNemar's test, which lives entirely on the discordant pairs — $b$ conditions where A succeeded and B failed against $c$ for the reverse — via $\chi^2 = (b-c)^2/(b+c)$. Concordant pairs carry no comparative information; pairing wins big exactly when the task has strong easy-hard structure across initial conditions, which manipulation tasks nearly always do.

Blocking and randomization: confounds that never show up on video

Everything above assumed the only randomness is the Bernoulli coin. On a real bench, the success probability itself drifts under nuisance factors unrelated to the policy: which object instance is in play, the lighting as afternoon sun moves across the room, how precisely you execute resets at hour two versus hour one. When those factors correlate with which policy is running, they become confounds — and the naive schedule maximizes the correlation: all of A's trials this morning, all of B's after lunch, difference credited to the architecture. The countermeasures are the ones behind stratified traffic splitting: block on the nuisance factors you can name, randomize whatever you cannot.

• Block by object instance. Each policy sees each physical object the same number of times — never policy A on the fresh cube and policy B on the chipped one.
• Block by session and day. Give every condition equal representation in each bench session; day-to-day variation in lighting, cable dressing, and camera mounts is real and large.
• Interleave within blocks. Run AB/BA pairs with coin-flipped order, so time-of-day drift and actuator warm-up load both policies equally.
• Script the reset. A printed pose grid taped to the table, a fixed placement order, a fixed gripper home pose — the reset script is now an experimental control.
• Photograph every reset. One frame from each RealSense before each trial costs nothing and lets you audit initial-condition drift after the fact.

The confound nobody writes down is the operator — you. Over a two-hour session your resets get subtly more consistent as you warm up, then sloppier as you tire; and you cannot blind yourself to which policy is running, because after a week of debugging you recognize each motion signature within a second. You cannot remove the operator, but you can cage the bias: let the printed grid make reset decisions, score from saved video after the session, and have close calls judged against the written rubric by someone who does not know the condition. Live scoring by an operator who wants a result is how motivated reasoning gets three extra points past review.

Protocol discipline: pre-register against yourself

You already live by contamination discipline in LLM work: the held-out set never enters training, the metric never changes after the runs, nobody tunes against the test split. The robot version has no leaderboard police, so the discipline is self-imposed — and the failure mode is subtler than cheating. It is drift: a trial fails because the object slipped after a solid grasp, you decide that “should” count, and history gets re-scored under the new rubric. Rubric drift is gradient descent on your own judgment with test-set feedback. The countermeasure is separating exploratory from confirmatory work: exploratory runs are your dev set; then the protocol freezes in writing, and only what happens after the freeze produces the number you report.

1. Freeze the rubric. Binary success, defined operationally: the timeout (say, 60 s), the no-human-touch-after-go rule, and edge cases decided now — dropped-then-recovered, placed-then-toppled, success at 59.9 s.
2. Freeze the initial-condition set. Sample the pose grid before trial one and print it; it is your held-out split, and no trial gets resampled because it “looked unfair.”
3. Name the primary metric. One number decides the comparison — paired success on the frozen grid; secondaries (time-to-success, per-stage progress) declared in writing before the runs.
4. Write down n, the MDE, and the stopping rule. Fixed n from the power calculation; no peeking-based stops.
5. Preserve every video, failures first. Failure videos are the error analysis and the evidence a skeptic can audit; success-only archives are advertising.
6. Log deviations instead of erasing them. A trial voided by a snagged cable is recorded as voided-with-reason, not silently rerun until it looks clean.

And report the nulls. “ACT did not detectably beat BC at $n = 45$ per arm; effects below about 29 points are invisible at this sample size” is a real, reusable result. A field where only positive deltas get written up has a literature that is a record of one-in-twenty flukes; your lab notebook does not have to work that way.

The twenty-line toolkit, and your BC-versus-ACT design

Everything above compresses into two functions you will keep for the rest of the course. First, Wilson next to Wald, so you can watch the default fail at the boundary:

import numpy as np

def wilson(k, n, z=1.96):
    phat = k / n
    denom = 1.0 + z * z / n
    center = (phat + z * z / (2.0 * n)) / denom
    half = (z / denom) * np.sqrt(phat * (1.0 - phat) / n + z * z / (4.0 * n * n))
    return max(0.0, center - half), min(1.0, center + half)

def wald(k, n, z=1.96):
    phat = k / n
    half = z * np.sqrt(phat * (1.0 - phat) / n)
    return phat - half, phat + half

for k, n in [(7, 10), (14, 20), (35, 50), (70, 100), (10, 10)]:
    wl, wh = wilson(k, n)
    al, ah = wald(k, n)
    print(f"{k:>3}/{n:<4} wilson [{wl:.3f}, {wh:.3f}]   wald [{al:.3f}, {ah:.3f}]")

# 10/10 prints: wilson [0.722, 1.000]   wald [1.000, 1.000]
# The Wald interval claims perfection from ten samples; Wilson does not.

Second, the design-time tools — MDE at a given $n$, and required $n$ for an expected effect — which run before you touch the robot and whose output goes into the frozen protocol:

import numpy as np
from scipy.stats import norm

def mde(n_per_arm, pbar, alpha=0.05, power=0.80):
    za = norm.ppf(1.0 - alpha / 2.0)
    zb = norm.ppf(power)
    return (za + zb) * np.sqrt(2.0 * pbar * (1.0 - pbar) / n_per_arm)

def n_per_arm(p1, p2, alpha=0.05, power=0.80):
    za = norm.ppf(1.0 - alpha / 2.0)
    zb = norm.ppf(power)
    pbar = 0.5 * (p1 + p2)
    a = za * np.sqrt(2.0 * pbar * (1.0 - pbar))
    b = zb * np.sqrt(p1 * (1.0 - p1) + p2 * (1.0 - p2))
    return int(np.ceil(((a + b) / abs(p1 - p2)) ** 2))

# Expectations from exploratory runs on the WidowX pick-and-place:
# state-only BC around 40%, ACT around 70%.
p_bc, p_act = 0.40, 0.70
n = n_per_arm(p_bc, p_act)                # -> 42 trials per policy
hours = 2 * n * 3.0 / 60.0                # 3 min per trial, both arms
print("n per arm:", n, "| bench time:", round(hours, 1), "h")   # 42 | 4.2 h

for nt in (10, 20, 50, 100, 350):
    print(f"n={nt:>4}  worst-case MDE: {mde(nt, 0.5):.2f}")

The worked design, end to end. Exploratory runs suggest BC lands near 40% and ACT near 70% — a 30-point expected effect, exactly the size this machinery can afford: $n = 42$ per arm at 80% power, rounded up to 45 paired trials across two 2.5-hour sessions on consecutive days. Fifteen poses across three object instances, printed as a grid before session one; each condition run under both policies in coin-flipped AB/BA order within object blocks; scored from video. Note what 80% power promises: even if ACT truly is 30 points better, one experiment in five returns a null — so the protocol states in advance that a null reads “undetected at this $n$,” never “equal.” When you later A/B two chunking schedulers serving a π₀-class policy from OpenPI (opens in a new tab), honest effect sizes shrink to 5–15 points and the same arithmetic demands hundreds of trials per condition — which is what pushes serious evaluation toward paired designs, graded metrics, and automated resets.

Where this goes next: the previous lesson, Offline RL: when imitation is not enough, completed your Phase 03 policy toolbox; this lesson is how you decide which of those policies actually works on your hardware, with numbers a reviewer cannot dismiss. Next you cross into the foundation-model phase with Anatomy of a vision-language-action model, where the stakes rise: fine-tuned VLA checkpoints cost days to produce, their deltas over your ACT baseline are exactly the 10–20-point effects at the edge of feasible statistical power, and the capstone's scheduler comparisons will stand or fall on the paired, blocked, pre-registered protocol you just learned to write.