The research artifact: report, repo, and talk

Twenty-four weeks ago you calibrated a camera. Since then you have built an episode logger with one clock domain, a fine-tuned π₀-class policy served from your RTX workstation, a full-loop latency trace, and — last lesson — an adaptive scheduler with a measured effect on a WidowX AI. All of it is now worth exactly what a stranger can verify. You already live by this rule: a claimed 1.4× serving speedup without the trace, repro script, and pinned container is not a result, it is an anecdote. Robotics runs on the same norm, enforced less by tooling and more by reviewers who have been burned. The artifact — report, repository, talk — is the compression format for the 24 weeks, and it gets the same engineering seriousness as the scheduler: versioned, tested, gated.

Statistical and Methodological Foundations

Before assembling the artifact, we must rigorously define the statistical tools and methodological constraints that validate the claims. The primary metric is the success rate, a proportion $p$ estimated by the sample proportion $\hat{p} = k/n$, where $k$ is the number of successes and $n$ is the total number of trials. In small-sample regimes typical of robot experiments, the standard Wald interval is unreliable. We therefore use the Wilson score interval, which inverts the score test statistic. The score test compares the observed $\hat{p}$ to a candidate true proportion $p$, using the standard error calculated at $p$ rather than $\hat{p}$. The acceptance region for a confidence level $1-\alpha$ is defined by the inequality $|\hat{p} - p| \le z_{\alpha/2}\sqrt{\frac{p(1-p)}{n}}$, where $z_{\alpha/2}$ is the critical value from the standard normal distribution (1.96 for 95% confidence). This formulation ensures that the interval remains bounded within $[0, 1]$ even when $\hat{p}$ is 0 or 1, addressing the boundary failure of the Wald method.

Solving this quadratic yields the Wilson bounds. A critical feature is the center of the interval, which is not $\hat{p}$ but rather $\frac{\hat{p} + z^2/2n}{1 + z^2/n}$. This represents a Bayesian shrinkage toward 0.5, the maximum entropy point for a binary outcome. For small $n$, this shrinkage is significant, pulling the estimate away from the boundaries to reflect the high uncertainty inherent in few trials. When comparing two conditions, we do not simply subtract the Wald intervals, as this ignores the covariance and boundary effects. Instead, we use the Newcombe hybrid interval. This method constructs the difference interval $[d - \sqrt{(\hat{p}_1 - L_1)^2 + (U_2 - \hat{p}_2)^2}, d + \sqrt{(U_1 - \hat{p}_1)^2 + (\hat{p}_2 - L_2)^2}]$, where $L_i, U_i$ are the Wilson bounds for condition $i$. This approach provides superior coverage properties compared to the Wald difference, particularly when sample sizes are unequal or proportions are near the boundaries.

Methodological rigor also requires defining pre-registration. Pre-registration is the act of freezing the analysis plan—including primary metrics, sample size, exclusion criteria, and randomization scheme—before data collection begins. This is a methodological constraint against p-hacking, the practice of selectively reporting results that appear significant after multiple comparisons. By attaching the pre-registration document to the report, you demonstrate that the primary claim was not chosen post-hoc to fit the data. Additionally, we must address censored data. In recovery-time experiments, trials that do not complete within the timeout are censored. Dropping these trials biases the mean recovery time downward, as the longest durations are systematically removed. Proper analysis treats these as right-censored observations, preserving the integrity of the distribution.

Worked Example: Wilson and Newcombe Intervals

Consider the capstone's confirmatory results: the adaptive scheduler achieves 78 successes out of 100 trials ($k_1=78, n_1=100$), while the synchronous baseline achieves 62 out of 100 ($k_2=62, n_2=100$). We calculate the 95% Wilson intervals for each condition and the Newcombe difference interval. For the adaptive condition, $\hat{p}_1 = 0.78$. Using $z=1.96$, we compute the quadratic coefficients: $A = 1 + 1.96^2/100 = 1.0384$, $B = -(2(0.78) + 1.96^2/100) = -1.5984$, and $C = 0.78^2 = 0.6084$. The discriminant is $B^2 - 4AC = 1.5984^2 - 4(1.0384)(0.6084) \approx 0.0315$. The roots are $p = \frac{-B \pm \sqrt{0.0315}}{2A}$. This yields $p_{lower} \approx 0.684$ and $p_{upper} \approx 0.855$. Thus, the Wilson interval for the adaptive scheduler is $[0.684, 0.855]$.

For the synchronous baseline, $\hat{p}_2 = 0.62$. Following the same procedure, the Wilson interval is $[0.520, 0.710]$. The point estimate for the difference is $d = 0.78 - 0.62 = 0.16$. To compute the Newcombe hybrid interval, we use the formula: Lower bound $= d - \sqrt{(\hat{p}_1 - L_1)^2 + (U_2 - \hat{p}_2)^2}$ and Upper bound $= d + \sqrt{(U_1 - \hat{p}_1)^2 + (\hat{p}_2 - L_2)^2}$. Substituting the values: Lower $= 0.16 - \sqrt{(0.78 - 0.684)^2 + (0.710 - 0.62)^2} = 0.16 - \sqrt{0.0092 + 0.0081} = 0.16 - 0.131 = 0.029$. Upper $= 0.16 + \sqrt{(0.855 - 0.78)^2 + (0.62 - 0.520)^2} = 0.16 + \sqrt{0.0056 + 0.0100} = 0.16 + 0.125 = 0.285$. The difference interval is $[0.029, 0.285]$, or $[+3, +28]$ points. Since this interval excludes zero, the difference is statistically significant at the 95% level, despite the slight overlap in the individual Wilson intervals.

The artifact has three consumers with three read patterns, and each piece is designed for one of them. A reviewer or hiring manager gives the report ten minutes and reads figures first. An engineer building on your scheduler gives the repository an afternoon and judges you by whether the first README command works. An interview panel gives the talk ten minutes plus twenty of questions — and forms its opinion almost entirely in the questions. Design for those budgets the way you design an API for its call pattern.

The report: six pages that carry one claim

Six to eight pages is workshop-paper scale, and honest scale for this evidence: one method, one platform, one task family, roughly 400 confirmatory trials. The spine is the hypothesis you wrote at the start of this phase — "under inference latency in the tested 100–300 ms regime, a scheduler that switches chunks based on predicted inference time and inter-chunk disagreement improves task success over synchronous and fixed-rate baselines." Every section sets that claim up, tests it, or bounds it; anything that does none of the three gets cut, however proud you are of the code behind it.

The pre-registration appendix is your credibility move, and it costs one page. You froze the primary metric, trial counts, exclusion rules, and analysis before the confirmatory block ran — back in "Designing robot experiments that survive review." Attaching that document verbatim converts the results section from "trust me" to "check me": the reader can confirm the metric you lead with is the one you predeclared, not the best of twelve computed afterward. Lead with exactly that metric; everything else — completion time, seam jerk, deadline-miss rate, recovery time — is labeled secondary.

Negative results get a named section, not a burial. If the disagreement-only scheduler variant — switching chunks on divergence without the latency predictor — performed within noise of the fixed-rate baseline (say 52/100 versus 49/100), that half page does real work: it localizes the effect in the latency-aware component and saves the next lab a month. In a field where most published pages are success stories, a specific, quantified negative result is among the strongest signals that the rest of your numbers are honest.

Write in this order: figures, captions, results, method, introduction, abstract last. Figures before prose is not a slogan — it forces you to discover whether the evidence exists before you spend a week narrating it. Make each caption state the claim, not the axes: "Adaptive scheduling recovers 16 points of success under 200 ms injected latency" tells a skimming reviewer what to believe; "Success rate per condition" tells them nothing they cannot see.

Figures that convince

The first figure needs error bars, and the obvious choice is wrong. The Wald interval — the plug-in formula everyone remembers — estimates its standard error at $\hat p$ itself:

Small-$n$ robot experiments live near the boundaries — pilot conditions at 19/20 or 3/20 are routine — so you need the interval that does not lie there. The Wilson interval inverts the score test: keep every candidate $p$ under which the observation would not be rejected, with the standard error computed at the candidate $p$, not at $\hat p$:

Square both sides and collect terms in $p$: a quadratic with positive leading coefficient, so the acceptance set is the segment between its roots:

Plug in the capstone's confirmatory numbers: adaptive at 78/100 gives a Wilson interval of about $[0.69, 0.85]$; the synchronous baseline at 62/100 gives about $[0.52, 0.71]$. The intervals overlap slightly — and that does not kill the result, because overlapping per-condition intervals are not a test of the difference. The difference gets its own interval: by the Newcombe hybrid construction combining the two Wilson intervals, the 16-point gap carries a 95% interval of roughly $[+3, +28]$ points, excluding zero. Put that in the caption; it is the sentence the whole project defends.

import numpy as np

Z = 1.959964  # two-sided 95%

def wilson_ci(k: int, n: int, z: float = Z):
    p = k / n
    denom = 1.0 + z * z / n
    center = (p + z * z / (2 * n)) / denom
    half = (z / denom) * np.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return max(0.0, center - half), min(1.0, center + half)

def newcombe_diff_ci(k_a, n_a, k_b, n_b):
    """95% interval for p_a - p_b from the two Wilson intervals."""
    pa, pb = k_a / n_a, k_b / n_b
    la, ua = wilson_ci(k_a, n_a)
    lb, ub = wilson_ci(k_b, n_b)
    d = pa - pb
    lo = d - np.sqrt((pa - la) ** 2 + (ub - pb) ** 2)
    hi = d + np.sqrt((ua - pa) ** 2 + (pb - lb) ** 2)
    return d, lo, hi

# Ground truth is the frozen run log: one row per confirmatory trial.
log = np.load("runlog/confirmatory_trials.npz")
for name in ("sync", "ensemble", "rtc_async", "adaptive"):
    m = log["condition"] == name
    k, n = int(log["success"][m].sum()), int(m.sum())
    lo, hi = wilson_ci(k, n)
    print(name, k, "/", n, "->", round(k / n, 3),
          "CI", round(lo, 3), "-", round(hi, 3))

d, lo, hi = newcombe_diff_ci(78, 100, 62, 100)
print("adaptive - sync:", round(d, 3), "CI", round(lo, 3), "-", round(hi, 3))

The latency figure must be a CDF per scheduler, never a bar of means. Your Phase 05 traces showed why: the baseline and the adaptive scheduler can sit within 5 ms at the median (163 versus 158 ms camera-to-command) while their p99s differ by 150 ms — and in a control loop the tail is a guaranteed periodic event, not a rarity. The CDF also gives the reader a number for free: with the scheduling deadline $d$ drawn as a vertical line, deadline-miss probability reads directly off the curve.

The last two figures make the physical story visible. Joint-trajectory overlays at chunk seams — commanded and measured positions for the 200 ms around each switch, several seams per condition — show whether the scheduler's switches are smooth or trade success for jerk a longer task would punish. Recovery-time distributions need one honest detail: some perturbed trials never recover before the 30 s timeout. Plot them as censored mass at the timeout — dropping them silently inflates every condition, and inflates the worst condition most, exactly where a skeptical reviewer probes first.

The repository: reproducibility is release engineering

The capstone touched three software environments, and a reproducer needs all three pinned separately: collection (arm and cameras), training (policy fine-tuning), and inference (the scheduler's runtime). They drift independently and break reproduction differently — which is why one requirements.txt at the repo root is a smell you would never accept in a serving deployment.

The ground truth of the release is the frozen run log: one row per confirmatory trial with condition, block labels (object, day), outcome, timestamps, and per-stage latency spans. Freeze it the moment the confirmatory block ends, write its hash into a manifest, and make every figure and every reported number a pure function of that file. That property lets a stranger — or you, eight months out — distinguish "the analysis is wrong" from "the data changed," and it is cheap to enforce mechanically:

import hashlib
import json
import subprocess
import sys
from pathlib import Path

MANIFEST = json.loads(Path("release/manifest.json").read_text())

def sha256(path: str) -> str:
    return hashlib.sha256(Path(path).read_bytes()).hexdigest()

failures = []

# 1. Environment pins exist for all three stacks (plus analysis).
for env in ("collection", "training", "inference", "analysis"):
    pin = Path("envs") / (env + ".lock")
    if not pin.exists():
        failures.append("missing environment pin: " + str(pin))

# 2. The frozen run log has not changed since analysis froze.
if sha256("runlog/confirmatory_trials.npz") != MANIFEST["runlog_sha256"]:
    failures.append("run log changed after the analysis freeze")

# 3. Every figure regenerates from the run log, byte-for-byte.
#    Hash the figure DATA (csv/npz), not the rendered PNG:
#    plot-library versions change bytes, your numbers must not.
for fig in MANIFEST["figures"]:
    subprocess.run([sys.executable, fig["script"]], check=True)
    if sha256(fig["data_out"]) != fig["data_sha256"]:
        failures.append(fig["name"] + ": regenerated data drifted")

if failures:
    print("RELEASE BLOCKED")
    for f in failures:
        print(" -", f)
    sys.exit(1)
print("release gate passed:", len(MANIFEST["figures"]), "figures verified")

• Environment pins for collection, training, and inference, plus a fourth lock for the analysis environment that builds the figures.
• Every seed: training seeds, the condition-randomization seed from the experimental design, and the seed that selects representative videos.
• The frozen, hash-locked run log — the single source every reported number derives from.
• Analysis scripts or notebooks that regenerate all four figures and every in-text number from the run log alone.
• Protocol documents: reset procedure, success criteria, exclusion rules, and the pre-registration, unchanged from before the confirmatory block.
• A calibration snapshot: camera intrinsics and extrinsics, arm zero offsets, and the workspace layout with measured dimensions.
• Representative episode videos — successes and failures per condition, both camera views, selected by seed rather than by pride.
• The release gate script wired into CI so the artifact re-verifies on every push.
• A README ordered by consumer: the 5-minute skim, the 30-minute figure reproduction, the full retrain.

The ten-minute talk and the two-question deep-dive

Ten minutes is roughly seven content slides at 80 seconds each, with the report's discipline: one claim per slide, evidence outweighing everything else. Design for a robotics lab's reading group or an interview panel — people who have seen a hundred polished demos and are waiting to learn whether you understand your own error bars.

1. Problem, one slide: chunk staleness in physical units — at 0.25 m/s a 200 ms-old chunk steers toward a gripper position 50 mm out of date — and the hypothesis verbatim.
2. Method, one slide: the scheduler's decision rule with its two inputs, predicted inference time and inter-chunk disagreement; the block diagram, not the code.
3. Evidence, three slides: the success figure with intervals and the difference CI; the latency CDFs with the deadline line; seam overlays or recovery distributions, whichever carries your secondary story.
4. Limitations, one honest slide: one arm, one task family, one lab, a 12-object set, quasi-static scenes — stated by you before anyone asks.
5. Next questions, one slide: the experiments this artifact makes possible, ending on what you would run next.

Then rehearse what actually decides the outcome: the deep-dive. A robotics interviewer reliably runs two probes. The statistics question attacks the inference: "your intervals overlap — why claim a difference?", "why 100 trials per condition?", "how do you know day-to-day drift didn't produce this?" Be ready to derive the Wilson interval at a whiteboard in four lines, to point at the difference interval, and to note that condition order was randomized within day-blocks precisely so drift cancels. The systems question attacks the mechanism: "where does the p99 320 ms go between exposure and motion?", "what happens at a seam when the new chunk arrives late?" Your Phase 05 trace is the answer; rehearse drawing the stage budget with numbers on each span. Keep backup slides for both — trace waterfall, ablation table, power analysis. Saying "I have a backup slide on exactly that" twice in a Q&A beats any animation on the main deck.

Closing the course: what you may claim, and your second project

The final discipline is drawing the claim's boundary exactly where the evidence stops. Supported: on this task family, platform, and object set, under inference latency in the tested 100–300 ms regime, the adaptive scheduler improved success by 16 points with a 95% interval of roughly $[+3, +28]$, cut deadline misses from 11% to 3%, and did not degrade seam smoothness. Not supported: that the scheduler improves VLA manipulation in general, that the effect transfers to other arms or dynamic scenes, or that your favored mechanism — staleness reduction at the tail — is the only explanation. The mechanism story is a hypothesis your evidence is merely consistent with; say so in exactly those words, and watch how differently a room of researchers treats you.

And the boundary is generative: everything just outside it is a candidate second project. Three concrete questions this artifact opens. First, dose–response: your Phase 05 latency-injection harness can sweep inference delay from 50 to 500 ms — does the scheduler's advantage grow with latency, and where does it saturate? Second, the hand-tuned threshold: can the switching rule be learned from the run logs you already have, and does it beat the hand-tuned rule out of distribution? Third, the regime change: quantize or distill the policy until inference drops under 30 ms — does the scheduler still matter, or did you just engineer away the problem it solves? Pick one, write it as a falsifiable hypothesis in the format of "From system to science," and size its trial budget with "Designing robot experiments that survive review." Scoping your second project with your first project's instruments is simply what a research engineer does.

Where this goes next: the previous lesson, "The capstone method: a latency-aware adaptive scheduler," gave you a method and a measured effect; this lesson turned them into a report a reviewer can check, a repository a stranger can rerun, and a talk that survives its own Q&A. There is no next lesson — this is the end of the course, deliberately. Twenty-four weeks ago the gap between you and robotics research was a stack of missing skills; now it is one unwritten pre-registration for the dose–response experiment your own artifact just proposed. State the hypothesis, size the trials, and go run it.