Designing robot experiments that survive review

Your evaluation instincts were trained in a world where samples are nearly free: ten thousand held-out prompts, a million A/B users, another benchmark sweep for a few dollars of GPU time. A robot trial is a different creature. One pick-and-place attempt on your WidowX costs about 90 seconds of wall time — roughly 30 s of motion, 30 s of scripted reset, 30 s of scoring and logging — plus a slice of your own attention that does not parallelize. Call it 40 trials per hour, 200 per day before either you or the hardware drifts. At that price, every statistical subtlety that web scale let you ignore comes back with teeth: power, blocking, optional stopping, multiple comparisons. Robot-learning reviewers have learned to attack exactly here — not the loss function, but ten trials per condition, no intervals, conditions run on different days. This lesson is how you design so that attack finds nothing.

Conceptual Foundations: Why Low-N Statistics Behave Differently

Before deriving formulas, we must define the statistical objects that make robot experiments distinct from web-scale A/B tests. In high-volume settings, the Law of Large Numbers allows us to treat sample means as stable estimates of population parameters. In robotics, where $n$ is often in the double digits, the variance of the estimator dominates the signal. This section defines three critical concepts: exchangeability, nuisance variables, and the family-wise error rate (FWER), and explains why binary metrics are statistically superior to continuous ones in this regime.

Exchangeability is the formal condition required for standard error calculations to be valid. A sequence of random variables $X_1, \dots, X_n$ is exchangeable if their joint probability distribution is invariant to any permutation of the indices. In plain terms, the probability of observing a specific set of outcomes does not depend on the order in which they occurred. If trials are exchangeable, the sample mean is an unbiased estimator of the true success probability $p$, and the standard error $\sqrt{p(1-p)/n}$ accurately reflects the uncertainty. If trials are not exchangeable—for example, if the robot's performance degrades systematically over time due to actuator heating—then the standard error underestimates the true uncertainty, and p-values become invalid.

The primary threat to exchangeability in a lab is the nuisance variable. A nuisance variable is a factor that affects the outcome but is not of primary scientific interest. Examples include ambient lighting, object mass, joint temperature, and operator fatigue. If a nuisance variable is correlated with the treatment assignment (e.g., all 'adaptive' trials are run in the afternoon when the robot is hot), the treatment effect is confounded with the nuisance effect. Confounding occurs when the effect of the treatment is mixed with the effect of another variable, making it impossible to attribute the outcome to the treatment alone. Blocking is the design technique that restores exchangeability within blocks by ensuring that each condition is tested under the same levels of the nuisance variable.

Why use binary success (0/1) rather than continuous metrics like completion time? Binary metrics have bounded variance. For a Bernoulli variable, $\operatorname{Var}(X) = p(1-p) \le 0.25$. This bound ensures that the standard error decreases predictably as $1/\sqrt{n}$. Continuous metrics in robotics, such as completion time, often exhibit heavy-tailed distributions due to rare catastrophic failures (e.g., a dropped object requiring a full reset). Heavy tails violate the assumptions of the Central Limit Theorem at small $n$, leading to unstable variance estimates and unreliable confidence intervals. Binary data provides a clearer decision boundary and more stable power calculations, which is why it is the preferred primary metric for low-N robot experiments.

Finally, we define the family-wise error rate (FWER). When you perform multiple statistical tests, the probability of making at least one Type I error (false positive) across the entire family of tests increases. FWER is defined as $P(\text{at least one Type I error} \mid H_0 \text{ true for all})$. If you run $m$ independent tests at significance level $\alpha$, the FWER is $1 - (1-\alpha)^m$. For $m=15$ and $\alpha=0.05$, the FWER is approximately 0.54. This means that if you test 15 pairwise comparisons, there is a 54% chance of finding at least one false positive, even if no true effects exist. Bonferroni correction controls FWER by testing each comparison at $\alpha/m$, but this is often too conservative for small $n$. A predeclared hierarchy of primary and secondary comparisons is a more efficient strategy.

Worked Example: Deriving the Confirmatory Trial Count

We now apply the power formula to a concrete scenario. Suppose you want to detect a difference between a baseline scheduler with success rate $p_1 = 0.50$ and an adaptive scheduler with success rate $p_2 = 0.70$. You require 80% power ($1-\beta = 0.80$) and a two-sided significance level of $\alpha = 0.05$. The goal is to calculate the minimum number of trials per condition, $n$, required to achieve this power.

First, identify the parameters. The true difference is $\Delta = p_2 - p_1 = 0.20$. The critical value for a two-sided test at $\alpha = 0.05$ is $z_{1-\alpha/2} = 1.96$. The z-score corresponding to 80% power is $z_{1-\beta} = 0.84$. The standard error of the difference between two independent proportions is $\mathrm{SE}_\Delta = \sqrt{\frac{p_1(1-p_1)}{n} + \frac{p_2(1-p_2)}{n}}$. For the test to have 80% power, the true difference $\Delta$ must be at least $z_{1-\beta}$ standard errors above the critical value $z_{1-\alpha/2}\mathrm{SE}_\Delta$. Thus, $\Delta = (z_{1-\alpha/2} + z_{1-\beta})\mathrm{SE}_\Delta$.

Substitute the values. The variance term is $p_1(1-p_1) + p_2(1-p_2) = 0.50(0.50) + 0.70(0.30) = 0.25 + 0.21 = 0.46$. The constant term is $(1.96 + 0.84)^2 = (2.80)^2 = 7.84$. The denominator is $\Delta^2 = (0.20)^2 = 0.04$. Plugging these into the formula:

Rounding up, you need 91 trials per condition. This means 182 total trials. At a rate of 40 trials per hour, this requires approximately 4.55 hours of robot time. If you only have time for 50 trials per condition, you can calculate the minimum detectable effect (MDE) by inverting the formula: $\Delta_{\min} = \sqrt{\frac{7.84 \times 0.46}{50}} \approx 0.268$. This means that with 50 trials, you can only reliably detect a difference of 26.8 points or more. Since your hypothesized difference is 20 points, the experiment is underpowered. You must either increase $n$ to 91 or accept a lower power (e.g., 50%).

From hypothesis to a run schedule

The previous lesson left you with a falsifiable claim of the form: under stressed injected latency, a latency-aware adaptive scheduler raises pick-and-place success over its scheduling baselines, without degrading the no-added-latency regime — this lesson works the stressed condition at +300 ms. A hypothesis plus a conditions-by-metrics matrix is still not an experiment. An experiment is a schedule: which trials, in which order, on which days, at what n — and the arithmetic has to close. For the capstone, the confirmatory matrix is 3 schedulers (synchronous, RTC-style asynchronous, adaptive) × 2 latency regimes (native, +300 ms injected) = 6 conditions, with binary success as the predeclared primary metric and completion time, deadline misses, and jerk as secondaries.

Now compile that matrix against the trial budget. At 90 s per trial you get about 40 trials per hour, and a focused session lasts 2 to 2.5 hours before scoring quality and reset consistency decay — roughly 100 trials per session, two sessions per day. Six conditions at 50–64 trials each is 300–384 trials: about 8–10 hours of pure robot time, which really means two full lab days once you add screening, warmup, and the inevitable rerun of an aborted block. A defensible week looks like this:

1. Day 1 — screening. Cheap 10-trial screens over candidate configurations (chunk horizons, adaptive-scheduler thresholds). Prune the disasters, pick the survivors, then freeze the confirmatory protocol in writing.
2. Days 2–3 — confirmatory runs. The frozen 6-condition matrix, blocked by object and session, randomized within blocks. No config edits, no code changes; anything broken gets logged and the block is rerun whole.
3. Day 4 — held-out generalization and spillover. Two novel objects never used in tuning, plus any blocks aborted earlier in the week.
4. Day 5 — analysis and buffer. Hardware always eats a day. If it doesn't, you have a free day for the ablation matrix.

The compile step runs in both directions. If the power analysis below says your primary comparison needs 250 trials per condition and the schedule shows you have time for 100, the calendar is not the thing to change — the design is. Cut conditions, move to a stress regime where the expected effect is larger, or change the primary metric. The rule that survives review: confirmatory trial counts come from the target effect size, never from the time you happen to have. Fitting n to the calendar and hoping is how underpowered experiments get run, and underpowered experiments produce noise with error bars.

Power: choose n from the effect size, not from your patience

You met this machinery once already, when the prereg's trial count fell out of it; this section turns it from a checksum into a design tool. Each trial in a condition is (by design — that is what the randomization and resets buy you) an approximately independent Bernoulli draw with success probability $p$. Estimate $p$ with the sample mean and the variance follows from the binomial:

A two-proportion z-test declares a difference when the observed gap exceeds $z_{1-\alpha/2}\,\mathrm{SE}_\Delta$ — with $\alpha = 0.05$ two-sided, that is $1.96$ standard errors. But clearing the bar on average is not enough: your observed gap fluctuates around the true gap $\Delta = p_2 - p_1$ with the same standard error, so to detect the effect with probability $1-\beta$ (power), the true gap must clear the bar with $z_{1-\beta}$ standard errors to spare. For 80% power, $z_{0.8} = 0.84$. Stack the two requirements and solve for $n$:

Plug in the capstone's stress regime. Suppose the synchronous baseline drops to about 55% success under +300 ms injected latency while the adaptive scheduler holds 85% — a 30-point gap:

Thirty-three trials per condition — under two hours of robot time for the pair. Now watch what happens when the gap shrinks. The same 10 points of improvement costs wildly different amounts depending on where it sits, because $n$ scales as $1/\Delta^2$ while the variance term moves only mildly:

Read the first two rows twice. Detecting 60% vs 80% costs 79 trials per condition; detecting 75% vs 85% costs 248 — three times more, for a gap that sounds only slightly subtler. Near-ceiling comparisons are brutally expensive: an 85% vs 90% comparison needs about 683 trials per condition — roughly 34 robot-hours, well over a week of lab days for one pair. It is also useful to run the formula backwards. Fix the n you can afford and ask what gap is detectable — the minimum detectable effect:

Blocked randomization: neutralizing what you cannot hold constant

Power analysis assumes trials within a condition are exchangeable draws. The lab violates that assumption everywhere. Objects differ enormously — a plush toy might grasp at 90% while a smooth tape roll sits at 60%. Sessions differ: morning light versus afternoon light through the window shifts what the RealSense sees, joint friction changes as the arm's brushless actuators warm over an hour of operation, and your own reset consistency drifts as you tire. Even trial order within a session matters — the first few trials after power-on behave differently. If you run all synchronous trials on Monday and all adaptive trials on Wednesday, your headline comparison is confounded with everything that changed between Monday and Wednesday, and no statistic can uncrunch it afterwards.

The standard fix is blocked randomization. Define a block as one object in one session. Inside every block, run every condition an equal number of times, in randomized order — so whatever the block's local quirks are (this object, this hour's lighting, this joint temperature), every condition experiences them equally. One refinement matters at robot scale: balance which condition opens each block. Warmup effects and your own handling drift concentrate in the first trials of a block, and with only 16 blocks, an unlucky shuffle could hand one condition most of the cold starts. Rotate the leading condition across blocks so each condition opens an equal share. Twenty lines of numpy generate the whole thing:

import numpy as np

CONDITIONS = ["sync", "rtc_async", "adaptive"]
OBJECTS = ["foam_cube", "mug", "tape_roll", "plush_toy"]
SESSIONS = ["tue_am", "tue_pm", "wed_am", "wed_pm"]
REPS_PER_CONDITION_PER_BLOCK = 4   # 3 conditions x 4 = 12 trials per block
SEED = 20260601                    # frozen; recorded in protocol.md

rng = np.random.default_rng(SEED)
rows = []
block_id = 0
for session in SESSIONS:
    for obj in OBJECTS:
        # Balance the leading condition so each condition opens an
        # equal share of blocks (kills warmup / order effects).
        lead = CONDITIONS[block_id % len(CONDITIONS)]
        others = [c for c in CONDITIONS if c != lead]
        first_pass = [lead] + list(rng.permutation(others))
        remainder = CONDITIONS * (REPS_PER_CONDITION_PER_BLOCK - 1)
        order = first_pass + list(rng.permutation(remainder))
        for k, cond in enumerate(order):
            rows.append((block_id, session, obj, k, cond))
        block_id += 1

# Sanity checks a reviewer will run in their head:
conds = np.array([r[4] for r in rows])
for c in CONDITIONS:
    total = int(np.sum(conds == c))
    opens = sum(1 for r in rows if r[3] == 0 and r[4] == c)
    print(c, "trials:", total, "| opens blocks:", opens)
# -> 64 trials each; opens 6/5/5 of the 16 blocks

with open("schedule_v1.csv", "w") as f:
    f.write("block,session,object,trial_in_block,condition\n")
    for r in rows:
        f.write("%d,%s,%s,%d,%s\n" % r)

This generates 16 blocks × 12 trials = 192 confirmatory trials, 64 per condition — comfortably above the 33 the power analysis demanded, with margin for aborted trials. Run the generator once per latency regime (a second seed, recorded alongside the first). Then treat schedule_v1.csv as a frozen artifact: commit it, execute it top to bottom, and when reality intervenes — an object rolls off the table, a USB hub resets — log the deviation and rerun the affected block whole. A schedule you quietly edit mid-experiment is not a schedule; it is a suggestion, and reviewers can smell the difference.

Protecting your error rate: screening, peeking, and multiple comparisons

Before the confirmatory matrix, you will have a pile of design choices to make: chunk horizon (25 vs 50 actions), the adaptive scheduler's disagreement threshold, replan cadence. Do not spend confirmatory-grade trials on these. Run cheap screens: about 10 trials per candidate configuration, blocked over two objects, purely to prune. Be honest about what 10 trials can say. A 7/10 result has a 95% Wilson interval of roughly [0.40, 0.89]; a 3/10 result spans [0.11, 0.60] — the intervals overlap enormously, so a screen cannot rank close configurations. What it can do is kill disasters: a config at 1/10 while others sit at 8/10 is dead, and 10 trials was the right price to find out. (Compute these intervals with proportion_confint from statsmodels (opens in a new tab) using method="wilson" — never the normal approximation at these n.)

Then comes the line that separates exploration from evidence: the freeze. Before the first confirmatory trial, write the protocol down — conditions, n per condition, the primary comparison, the exact success criterion, the blocking structure, the seed, and the analysis script, committed to the repo. Everything after the freeze is confirmatory; everything before it was screening and gets reported as such. The freeze is what licenses you to quote p-values and intervals at face value.

Sequential designs with predeclared alpha-spending schedules (O'Brien–Fleming-style boundaries) do exist and are legitimate — but they must be chosen before data collection, and at robot-lab scale they buy little. The simple honest rule: fix n, run to n, report what you got, including the negative result. A well-powered negative on a frozen protocol is a publishable, hiring-grade artifact; a lucky positive on a peeked experiment is a liability with your name on it.

The same alpha arithmetic governs the ablation matrix. Six conditions admit 15 pairwise comparisons; test each at $\alpha = 0.05$ and the chance of at least one false positive is

Bonferroni correction — test each comparison at $\alpha/m = 0.0033$ — restores the family-wise error rate, but at tiny robot n it is ruinous: the required z grows from 1.96 to 2.94, inflating every trial count by roughly 1.8×. The structure that survives review without that tax is a predeclared hierarchy. One primary comparison carries the paper's claim and gets the full α: for the capstone, adaptive vs synchronous at +300 ms injected latency, binary success. A handful of secondary comparisons (completion time, deadline misses, the native-latency regime) are reported with confidence intervals and interpreted cautiously. Everything else in the matrix is exploratory — reported as estimates with intervals, never as significance claims, feeding hypotheses for the next experiment rather than conclusions from this one. Bonferroni-style caution matters when you insist on making several confirmatory claims at once; a single predeclared primary comparison means you never owed the correction on your headline result in the first place.

Operational discipline: the run log is the experiment

Statistics assumes the trials were actually comparable; operations is what makes that true. Three habits carry most of the weight. First, scripted resets. The arm returns to home via the same recorded joint trajectory every time; the object goes back onto a printed, laminated template with a drawn outline (placement tolerance ±5 mm — check it, don't eyeball it); the reset script locks camera exposure and white balance and snapshots one scene-check frame that lands in the log. If your reset is a human hand and a memory of where the mug goes, your between-trial variance quietly doubles and your power calculation was fiction.

Second, condition-blind execution where physics allows it. The run script reads the frozen schedule and configures the scheduler itself; the operator screen shows only "trial 37 — object: mug — place in zone B," not which condition is live, and the success verdict is entered before the condition is revealed. Blinding has limits on a robot — an adaptive scheduler may visibly move differently than a synchronous one — which is exactly why the success criterion must be predeclared and mechanical: object fully inside the 5 cm target zone, gripper released, stable for 10 s. A criterion that needs judgment is a criterion that bends toward the hypothesis.

Third, preserve every failure. Failure videos are not embarrassing exhaust; they are the highest-information artifacts the experiment produces — the raw material for the failure taxonomy in your report and for the next iteration of the method. The storage argument for deletion does not exist: 384 trials × 45 s × two RealSense streams at 720p is under 20 GB, a rounding error on your workstation's NVMe. Every trial — success or failure — emits one JSON-lines record that makes the analysis reproducible from the log alone:

{
  "schema_version": 3,
  "trial_id": "2026-08-04_wed_pm_b07_t05",
  "block": 7,
  "session": "wed_pm",
  "object": "tape_roll",
  "condition": "adaptive",
  "latency_regime_ms": 300,
  "policy_checkpoint": "pi0_widowx_ft_step18000_sha256_9be2",
  "scheduler_config_hash": "c41f7a",
  "schedule_seed": 20260601,
  "t_start_mono": 88231.442,
  "t_start_wall": "2026-08-04T14:22:31-07:00",
  "success": false,
  "success_criterion": "in_zone_5cm_released_stable_10s",
  "failure_mode": "grasp_slip_during_transport",
  "completion_time_s": null,
  "deadline_misses": 4,
  "chunk_switches": 11,
  "video": "videos/b07_t05.mp4",
  "latency_trace": "traces/b07_t05.perfetto",
  "operator_note": "object rolled ~2 cm off template outline before grasp"
}

Note what the schema pins down: the checkpoint hash and config hash tie every trial to exact code and weights; the monotonic timestamp joins each trial to the Phase 05 latency traces so per-trial deadline misses come from measurement, not recollection; the failure-mode tag comes from a small predeclared taxonomy (grasp failure, slip during transport, placement miss, timeout, hardware fault) so that failure analysis is a groupby, not an archaeology dig. The test of the whole apparatus: a colleague with your repo, schedule_v1.csv, and run_log.jsonl should regenerate every figure in your report without asking you a single question. That is the reproducibility gate this phase demands, and it is checked at the level of the log schema, not the level of good intentions.

Where this goes next: the previous lesson, From system to science: the falsifiable hypothesis, gave you a claim worth testing; this lesson gave you the machinery — powered trial counts, blocked randomization, a frozen protocol, and a log schema — that turns robot hours into evidence a reviewer cannot dismantle. Next, The capstone method: a latency-aware adaptive scheduler builds the thing this experiment will judge: the scheduler whose 30-point stress-regime gap you just learned to detect in 33 trials.