From system to science: the falsifiable hypothesis

By the end of Phase 05 you owned something rare: a robot stack where every millisecond from photon to motor command is traced, plus a metric suite — success rate, completion time, recovery count, jerk, deadline misses — that turns motion into numbers. That is a measurement instrument, not yet science. You know this gap from your own field: anyone can produce a chart showing their serving stack is faster, and almost none of those charts survive the questions that matter — faster than what, under which frozen workload, by whose decision rule? MLPerf exists because "our inference is fast" is not a claim; it is a mood with a bar chart. This lesson forces your capstone claim into a form that a specific, countable set of trials can destroy.

Statistical Mechanics of Binary Trials

Before calculating sample sizes, we must define the probabilistic object being measured. In robotic manipulation, a trial is a Bernoulli random variable $X_i \in \{0, 1\}$, where $X_i = 1$ indicates success and $X_i = 0$ indicates failure. The probability of success is $p$. The expected value is $E[X_i] = p$, and the variance is $\operatorname{Var}(X_i) = p(1-p)$. This variance is maximized at $p=0.5$ and minimized at $p=0$ or $p=1$. This non-linear dependence on $p$ is critical: it means that estimating a success rate of 50% is statistically harder than estimating a rate of 90%, even with the same number of trials, because the uncertainty (variance) is larger near the middle of the range.

When comparing two conditions, we estimate the difference $\Delta = p_A - p_B$. The variance of the difference of two independent estimates is the sum of their variances: $\operatorname{Var}(\hat{\Delta}) = \frac{p_A(1-p_A)}{n} + \frac{p_B(1-p_B)}{n}$. In power calculations, we often approximate this using a pooled variance $\bar{p} = \frac{p_A + p_B}{2}$, yielding $\operatorname{Var}(\hat{\Delta}) \approx \frac{2\bar{p}(1-\bar{p})}{n}$. This approximation is always conservative for equal group sizes: writing $p_A=\bar p-\delta$ and $p_B=\bar p+\delta$ shows that the exact numerator is $2\bar p(1-\bar p)-2\delta^2$, never larger than the pooled numerator. The approximation therefore overestimates variance most when the two rates are far apart; it does not create an underpowered design. For pilot-based planning, still check how the required $n$ changes across plausible values of both success rates and the effect size.

The danger of confounds lies in their covariance structure. A confound $C$ biases the estimate of $\Delta$ if it is correlated with both the independent variable $X$ (condition) and the dependent variable $Y$ (outcome). Formally, bias arises if $\operatorname{Cov}(X, C) \neq 0$ and $\operatorname{Cov}(Y, C) \neq 0$. Randomization forces $\operatorname{Cov}(X, C) \approx 0$ by making the assignment of conditions independent of time-varying factors. Blocking reduces the variance of $Y$ by controlling for known nuisance variables (e.g., day of week), thereby increasing the power of the test. They are complementary: randomization handles unknown confounds, while blocking handles known ones.

Worked Example: Sensitivity Analysis for Sample Size

Consider a pilot study with $n=10$ trials per condition. The RTC baseline shows 5/10 successes ($\hat{p}_{RTC} = 0.50$), and the adaptive scheduler shows 7/10 successes ($\hat{p}_{Adaptive} = 0.70$). We hypothesize a true effect size $\delta = 0.20$. First, calculate the pooled mean $\bar{p} = \frac{0.50 + 0.70}{2} = 0.60$. Using the standard power formula for 80% power ($z_{\beta}=0.84$) and 95% confidence ($z_{\alpha/2}=1.96$):

However, the pilot $n=10$ is small. The 95% CI for $\hat{p}_{RTC}=0.50$ is wide (approx. 0.19 to 0.81). If the true $p_{RTC}$ is actually 0.50 (the lower end of the adaptive estimate's uncertainty), the variance $p(1-p)$ is maximized at 0.25. Let's recalculate $n$ assuming the worst-case variance $\bar{p}=0.50$:

Now consider the risk that the true effect is smaller than hypothesized. If the true $\delta$ is only 0.15 (a 15-point improvement), the required $n$ increases dramatically due to the $1/\delta^2$ dependence:

This sensitivity analysis reveals that the initial estimate of 95 trials is fragile. If the true effect is 0.15, 95 trials provide only ~50% power, meaning the study is a coin flip. The learner must decide: is a 15-point improvement worth 167 trials? If the robot-hour budget allows only 100 trials, the study is underpowered for small effects. The pre-registration must explicitly state the minimum detectable effect (MDE) and acknowledge that effects smaller than the MDE cannot be ruled out.

A demo says it works; a claim names what would kill it

The video of your WidowX picking up a cube under 250 ms of injected latency is a demo. It proves existence: at least once, under conditions you chose and did not fully record, the system succeeded. "Our system works" has no scientific content because nothing observable could contradict it — a failure just means you filmed the wrong take. A research claim has the opposite structure: it stakes out territory that a specific observation can take away. The canonical form is X causes Y under conditions Z, and every word is load-bearing.

• Falsifiability — a concrete, observable outcome exists that would make the claim false. If you cannot describe the dataset that would kill your hypothesis, you do not have one yet.
• Controlled comparison — the claim compares conditions that differ in exactly one thing. "85% success" alone is a measurement; "+18 points over the RTC baseline with everything else frozen" is evidence about a cause.
• Predeclared decision rule — the numeric threshold that counts as confirmation is written down before the confirmatory data exists. Otherwise the analysis degrees of freedom (which metric, which trials, which subset) guarantee you can find a win somewhere.

Apply the form to the capstone. "Our adaptive scheduler works" becomes: under injected round-trip latency of 250 ms with ±80 ms jitter (Z), replacing RTC-style asynchronous execution with the latency-aware adaptive scheduler (X) raises pick-and-place success rate by at least 10 percentage points (Y), for a frozen π₀ checkpoint on the WidowX with a frozen task rubric. Every clause is now attackable, and that is the point: a reviewer can check the checkpoint hash, re-run the injection, and recompute the interval. Formally, the primary contrast is a difference in success probabilities:

The decision rule then reads like an SLO, not a hope: "the hypothesis is supported iff the 95% confidence interval on $\hat\Delta$ at the 250 ms condition excludes zero and the point estimate is at least +10 points; any other outcome is not supported." Two clauses on purpose: the interval clause guards against noise, the point-estimate clause against a statistically real but practically worthless +2-point effect. There is no clause for "suggestive trends" — trends may be discussed in the report, never promoted to confirmation after the fact.

Anatomy of the capstone hypothesis

A hypothesis is a wiring diagram: independent variables you turn, dependent variables you read, controls you weld shut, and a causal story explaining why turning the knob moves the needle. The capstone has exactly two independent variables: scheduling policy (RTC-style asynchronous baseline vs. the adaptive scheduler you will build in lesson 3) and injected latency (a 0 ms control and one stressed condition, e.g. 250 ± 80 ms, chosen from your Phase 05 sweeps). The delay-injection harness you built for debugging now graduates to experimental apparatus — injection is what makes latency an independent variable instead of an uncontrolled ambient fact.

The dependent variables are the Phase 05 metric suite, with a hierarchy imposed. Exactly one is primary — success rate under the frozen rubric — because the decision rule can reference only one number without multiplying your false-positive rate. The rest (completion time, recovery count, RMS jerk from the motion-quality lesson, deadline-miss rate, realized per-trial latency) are secondary: reported with intervals, used to interrogate mechanism, never used to declare victory. The controls are everything frozen and hash-stamped: the π₀ checkpoint and its normalization statistics, the rubric version, the object set, the camera mount and locked exposure, the GPU clock configuration. Retraining the checkpoint in week 22 does not improve your experiment; it deletes it.

The causal story is what elevates this from benchmark to hypothesis. You already derived its physics in the instrumentation phase: a policy acting on a $\Delta t$-old observation mis-places the world by $\varepsilon = v\,\Delta t$ — at 0.25 m/s end-effector speed, 250 ms of staleness is 62 mm, more than a cube width. The chain from knob to needle runs:

1. Injected delay inflates the age of the observation each executing chunk was computed from — chunk staleness rises.
2. A stale active chunk and a freshly computed chunk disagree about where the arm should be, so each switch carries a discontinuity proportional to $v\,\Delta t$.
3. Discontinuities excite exactly what the previous lesson taught you to measure: jerk spikes and, near contact, missed or crushed grasps.
4. Failed grasps trigger recovery attempts, stretching completion time and, past the rubric's limits, dropping success.

The adaptive scheduler claims to intervene at links 1 and 2: predicted inference time lets it start computing the next chunk early enough to bound staleness, and a chunk-disagreement signal lets it place switches where the discontinuity is small. The chain assigns each secondary metric a link to monitor — staleness for link 1, switch-discontinuity magnitude for link 2, jerk for link 3, recovery for link 4, success for the endpoint. Instrument the whole chain: if success improves but measured staleness does not shrink, your headline number is right and your mechanism story is wrong — you got lucky, and the RTC line of work (opens in a new tab) you are extending deserves better than luck.

Confounds: the robot fights back

In a serving A/B test the infrastructure is stationary: the hardware does not care how many requests it served yesterday. A robot experiment run across weeks on one arm, one object set, and one operator is the opposite — nearly everything drifts, and every drift is correlated with time, hence with anything you schedule sequentially. A confound is any variable that co-varies with your independent variable and plausibly affects the dependent one; five dominate week-scale manipulation experiments.

Two design tools neutralize almost everything in that table. Randomization breaks the correlation between condition and every time-varying factor — including ones you have not thought of — by making condition order unpredictable. Blocking handles slow drifts by forcing balance inside each stretch of time: run trials in blocks (say, 10 per condition per block, order shuffled within the block), blocked by day and object instance, so that when Thursday's lighting or a worn cube shifts performance, it shifts both conditions equally. A confound neutralized by design needs no statistical apology afterward; one handled "in analysis" is a limitations paragraph waiting to be written. The next lesson builds full randomization schedules — the prereg only names what is blocked and what is randomized.

The pre-registration page

A pre-registration is one page, written after your pilots but before your first confirmatory trial: hypothesis, conditions, frozen artifacts, primary metric and decision rule, trial counts with justification, exclusion rules, kill criterion. This section builds the two parts people leave vague — trial counts and exclusions — then hands you the template. The trial count falls out of a three-line derivation. Each trial is a Bernoulli draw with success probability $p$, so the estimate $\hat p = k/n$ has variance $p(1-p)/n$. Your decision rule references the difference between two independent estimates, and variances of independent estimates add:

Now impose the two things you want. First, the decision rule fires only when $\hat\Delta$ clears a significance margin of $z_{\alpha/2}\,\mathrm{SE}$ above zero (for a 95% interval, $z_{\alpha/2} = 1.96$). Second, you want the rule to fire with high probability — power $1-\beta$ — when the true effect really is $\delta$; since $\hat\Delta$ scatters around $\delta$ with the same SE, you must reserve a further power margin of $z_{\beta}\,\mathrm{SE}$ (for 80% power, $z_{\beta} = 0.84$). The true effect must contain both margins:

Plug in capstone numbers. Suppose Phase 05 pilots showed the RTC baseline dropping to roughly 55% success at 250 ms injected latency, and you hypothesize the adaptive scheduler recovers to 75% — so $\bar p = 0.65$ and $\delta = 0.20$:

Verify the analytic result the way you would verify any queueing formula — simulate the decision rule end to end. This also catches what the formula hides: the normal approximation turning optimistic near the edges of $[0,1]$.

import numpy as np

rng = np.random.default_rng(0)

def win_rate(p_base, p_new, n, sims=20000, z=1.96):
    """Fraction of simulated experiments where the prereg decision
    rule fires: 95% CI lower bound on (p_new - p_base) exceeds 0."""
    k0 = rng.binomial(n, p_base, size=sims)
    k1 = rng.binomial(n, p_new, size=sims)
    p0, p1 = k0 / n, k1 / n
    se = np.sqrt(p0 * (1 - p0) / n + p1 * (1 - p1) / n)
    lower = (p1 - p0) - z * se
    return float(np.mean(lower > 0))

# Power at the hypothesized effect (0.55 -> 0.75), n = 90 per arm
print("power at true delta 0.20:", win_rate(0.55, 0.75, 90))   # ~0.80

# False-positive rate when the scheduler does nothing at all
print("false positive at delta 0:", win_rate(0.65, 0.65, 90))  # ~0.03

# The 1/delta^2 law, felt directly
for n in (30, 60, 90, 150, 250, 360):
    print(n, "trials/arm -> power", round(win_rate(0.55, 0.75, n), 3))

The last piece is exclusion rules — the definition of a valid trial, written before any trial exists. The rule of thumb: exclusions may reference the apparatus and the protocol, never the behavior under study. An actuator fault or protective shutdown, an injection crash, or a reset photo showing off-template placement are legitimate exclusions; "the policy did something weird" is not — weird is the dependent variable. Every excluded trial is logged and counted per condition, because a differential exclusion rate is itself a finding: a scheduler that crashes the harness more often is not neutral information.

# Pre-registration: latency-aware adaptive chunk scheduling
Frozen on: <date>.  Runtime + config commit: <hash>.

## Hypothesis
Under injected round-trip latency of 250 ms with +/- 80 ms uniform
jitter, the adaptive scheduler (predicted-inference-time triggering +
chunk-disagreement switch placement) raises pick-and-place success by
at least 10 percentage points over the RTC-style async baseline.
Mechanism: earlier replanning bounds chunk staleness, shrinking switch
discontinuities and grasp-phase errors.

## Conditions (independent variables)
- Scheduling policy: RTC-style async | adaptive   <- primary contrast
- Injected latency: 0 ms (control) | 250 +/- 80 ms (stressed)

## Frozen artifacts (controls)
- Policy checkpoint pi0_widowx_ft_v3 (sha256 <...>), norm stats, no retraining
- Task rubric v2; object set A (3 identical cubes, rotated per block)
- Camera mount, locked exposure/WB, LED panels, blackout curtain
- GPU clocks locked; 10 min GPU + arm warmup before every session

## Primary metric and decision rule
Success rate per rubric v2. SUPPORTED iff the 95% CI on
(p_adaptive - p_rtc) at 250 ms excludes 0 AND the point estimate is
>= +10 points. Any other outcome: NOT SUPPORTED.

## Secondary metrics (reported with intervals, never decisive)
Completion time, recovery count, RMS jerk, deadline-miss rate,
measured chunk staleness, switch-discontinuity magnitude,
realized (not nominal) per-trial latency.

## Trials and power
90 trials per cell x 4 cells = 360. Assumes p_rtc ~ 0.55 at 250 ms
(pilot), delta = 0.20, alpha = 0.05 two-sided, power 0.80; verified
by simulation. Blocked by day and object instance; condition order
randomized within blocks of 10. No interim analyses.

## Exclusion rules (apparatus/protocol only, applied blind)
Actuator fault or protective shutdown; injection process crash; reset photo
shows off-template placement. Excluded trials logged and counted per
condition. Policy behavior is never grounds for exclusion.

## Kill criterion and pivot
If the CI includes 0 or the estimate is < +5 points, the hypothesis is
falsified. Pivot: mechanism study - is predicted inference time
calibrated against realized, and does the disagreement signal predict
switch-discontinuity magnitude? Negative result reported in full.

Kill criteria, pivots, and honest scope

The kill criterion is the decision rule's mirror image, written with the same precision: "if, after the full 90 trials per cell, the 95% CI on $\hat\Delta$ at 250 ms includes zero and the point estimate is below +5 points, the hypothesis is falsified." Note what it is not: "if results are disappointing we will investigate further" is a promise to keep rolling dice. The pivot is predeclared for the same reason — chosen while calm, not while grieving. Here the natural pivot is a mechanism study, because it salvages the instrumentation even when the headline effect dies: is predicted inference time calibrated against realized (an offline question your Phase 05 traces already answer)? Does the disagreement signal predict switch-discontinuity magnitude at all? A scheduler failing because its predictor is uncalibrated is a finding with a cause attached.

Finally, scope — where first research projects die. Six weeks sounds long until you decompose it: the phase budget is 72–90 hours, the scheduler implementation plus pilots consume roughly the first two weeks, and the artifact (report, repo, talk) consumes the last two. Your confirmatory window is realistically 10–20 robot-hours in weeks 21–22. At about 90 seconds per trial including a standardized reset, that is 400–800 trials, total, forever. Every design decision flows from this number through the $1/\delta^2$ law.

• Fits in the window: two scheduling policies, one stressed latency condition plus a 0 ms control, one task with one frozen rubric, one object class, 90 trials per cell — a 2×2 design, 360 trials, roughly 9 robot-hours.
• Does not fit: three policies × three latency levels (810 trials before any control), a second task (doubles everything and adds a rubric), cross-embodiment claims, or retraining the checkpoint mid-phase — which invalidates every trial already run.
• The multiplication test: before admitting any new factor, compute cells × 90 trials × 90 s. If the product exceeds your robot-hour budget, the factor goes to the exploratory pile or the future-work section — not into the prereg.

One task, one robot, one clearly instrumented mechanism beats three of everything, because the alternative is not three results — it is three underpowered non-results sharing one confounded apparatus. The strongest claim you can afford is narrow, and narrowness is not weakness; it is where the causal story stays checkable end to end. Breadth is what the future-work section is for.

Where this goes next: the previous lesson — Motion quality: smoothness, jerk, and recovery — completed the metric suite this hypothesis just promoted to dependent variables. The next lesson, Designing robot experiments that survive review, hardens the prereg you drafted here into a full protocol: randomization schedules, proper confidence intervals for binary success, honest interim-analysis rules, and the logging discipline that keeps your numbers standing when someone hostile reads them. After that, lesson 3 builds the thing the hypothesis is about — the latency-aware adaptive scheduler — and it will walk onto a stage where the judging criteria are already nailed down.