Offline RL: when imitation is not enough

Every policy in this phase shares a ceiling. Behavioral cloning, ACT, and Diffusion Policy are — under three different losses — the demonstrator, compressed. If your teleoperated grasps succeed 85% of the time, a faithful imitator succeeds about 85% of the time: the fumbles are in the training distribution, and a density model reproduces them. Offline RL is the field's standing offer to break that ceiling: from the same logged episodes, plus a reward label, learn a policy better than the one that collected the data — with zero exploratory actions on the physical arm. If you have watched RLHF turn a frozen preference dataset into a better model, the offer sounds plausible. The catch: the distribution shift you mapped in the behavioral-cloning lesson does not disappear when you add a value function. It relocates into the value function, where it compounds faster and hides better.

Foundations: Support, Error Decomposition, and the Geometry of Optimization

Before deriving the divergence of offline value estimation, we must rigorously define the geometric and statistical objects that make the failure possible. The core issue is not merely that the data is finite, but that the data is localized. In a continuous state-action space, the dataset $\mathcal{D}$ does not cover the entire space uniformly; it occupies a specific region. We define the support of the dataset, denoted $\text{supp}(\mathcal{D})$, as the set of state-action pairs $(s,a)$ where the behavior policy $\pi_\beta(a|s)$ has non-negligible probability density. Formally, for a threshold $\epsilon_{\text{cov}}$, $(s,a) \in \text{supp}(\mathcal{D})$ if $\pi_\beta(a|s) > \epsilon_{\text{cov}}$. Points outside this region are off-support. The critical distinction is that off-support points are not merely 'unseen'; they are regions where the approximator has no gradient signal from the data, allowing its output to be determined entirely by the inductive bias of the architecture and the initialization of its weights.

To understand why the max operator fails, we must decompose the error of the learned Q-function $\hat{Q}$ into two distinct components. Let $Q^*$ be the true optimal Q-function. The error $\epsilon(s,a) = \hat{Q}(s,a) - Q^*(s,a)$ can be split into estimation error $\epsilon_{\text{est}}$ and extrapolation error $\epsilon_{\text{ext}}$. Estimation error arises from the finite number of samples $N$ within the support; it is a variance term that decreases as $O(1/\sqrt{N})$. Extrapolation error arises from the model class's inability to constrain its output outside the support; it is a bias term that depends on the distance from the support and the complexity of the approximator. A common misconception is that collecting more data fixes offline RL. This is false: increasing $N$ reduces $\epsilon_{\text{est}}$ but does not reduce $\epsilon_{\text{ext}}$. If the behavior policy $\pi_\beta$ is narrow, the support remains narrow regardless of $N$, and the extrapolation error at the boundary of the action space remains unbounded.

The distinction between estimation and extrapolation error is crucial for diagnosing failure. If a critic diverges, checking the loss on the training set is insufficient. A low training loss only guarantees small $\epsilon_{\text{est}}$ on-support. It says nothing about $\epsilon_{\text{ext}}$ off-support. To diagnose extrapolation failure, one must evaluate the critic on a held-out set of actions that are off-support but physically valid. If the Q-values on these actions are wildly different from the true values (or from the Q-values on nearby on-support actions), the critic is extrapolating. This is the 'extrapolation gap.' In practice, this gap is the primary driver of the bias $b_\infty$ in the Bellman backup, not the variance of the on-support fit.

This decomposition explains why 'more data' is not a panacea. If the behavior policy $\pi_\beta$ is a Gaussian with standard deviation $\sigma_\beta$, the support is effectively $[-3\sigma_\beta, 3\sigma_\beta]$. Doubling the dataset size $N$ does not change $\sigma_\beta$; it only makes the fit within $[-3\sigma_\beta, 3\sigma_\beta]$ tighter. The extrapolation error at $a = 1.0$ (if $3\sigma_\beta < 1.0$) remains determined by the architecture's inductive bias. For a neural network with ReLU activations, the function is piecewise linear. In regions with no data, the weights are not updated by the data loss, so they remain at their initialization or drift due to regularization. This drift is uncontrolled and can lead to large, arbitrary Q-values. The 'divergence' is not a smooth explosion but a random walk in the weight space that projects to large values in the off-support region.

Worked Example: Divergence in a 1-D Cart-Pole

Consider a simplified 1-D Cart-Pole task. The state is the cart position $s \in [-2, 2]$ and the action is the cart velocity $a \in [-1, 1]$. The true Q-function is $Q^*(s,a) = -s^2 - a^2$, which is maximized at $(0,0)$ with value 0. The behavior policy $\pi_\beta$ is a Gaussian with $\mu=0, \sigma=0.1$, so the support is effectively $[-0.3, 0.3]$. We have a dataset of 1000 transitions. We fit a 2-layer MLP with 32 hidden units to estimate $\hat{Q}$. The noise in the returns is $\sigma_{\text{noise}} = 0.1$.

Step 1: Fit the critic. Within the support $[-0.3, 0.3]$, the MLP fits the data well. The estimation error $\epsilon_{\text{est}}$ is small, approximately 0.05. Outside this range, the MLP extrapolates. Due to the ReLU activation, the function becomes piecewise linear. In regions with no data, the gradient of the loss is zero, so the weights remain at their initialization or drift due to regularization. Suppose that at $a=0.5$ (off-support), the MLP outputs $\hat{Q}(0, 0.5) = 0.2$, while the true value is $Q^*(0, 0.5) = -0.25$. The extrapolation error here is $\epsilon_{\text{ext}}(0, 0.5) = 0.2 - (-0.25) = 0.45$. This is a large positive error, caused by the random initialization of the weights in the off-support region.

Step 2: Gradient ascent on the actor. The actor $\pi_\theta$ starts at $\mu=0$. It performs gradient ascent on $\hat{Q}(s, \pi_\theta(s))$. Initially, the actor is in the support, where $\hat{Q}$ is accurate. The gradient points toward $a=0$. However, as the actor moves, it eventually reaches the boundary of the support. At $a=0.3$, the actor is still in the support. At $a=0.5$, the actor is off-support. The gradient $\nabla_a \hat{Q}$ at $a=0.5$ is determined by the noise field. Suppose the noise gradient points toward $a=0.8$. The actor moves to $a=0.8$. At $a=0.8$, the MLP outputs $\hat{Q}(0, 0.8) = 0.5$, while the true value is $Q^*(0, 0.8) = -0.64$. The extrapolation error is now $1.14$. The actor continues to move toward higher $\hat{Q}$, eventually converging to $a=1.0$, where $\hat{Q}(0, 1.0) = 1.2$ and the true value is $-1$. The actor has converged to a point where the critic's hallucination is largest.

Step 3: The fix with IQL. IQL avoids this by never evaluating $\hat{Q}$ off-support. It computes $V(s)$ using expectile regression on dataset actions only. For $s=0$, the dataset actions are in $[-0.3, 0.3]$. The best dataset action is $a=0.1$ with $Q^*(0, 0.1) = -0.01$. The expectile $V(0)$ approximates this best in-sample value, say $V(0) \approx -0.05$. The advantage $A(s,a) = Q(s,a) - V(s)$ is computed only for dataset actions. For $a=0.1$, $A(0, 0.1) = -0.01 - (-0.05) = 0.04$. For $a=1.0$ (not in dataset), $A$ is not computed. The AWR policy weights $a=0.1$ highly, and the actor stays near $a=0.1$. The extrapolation channel is closed.

The promise: improvement without exploration

Fix notation. You have a dataset $\mathcal{D} = \{(s_i, a_i, r_i, s_i')\}$ logged by some behavior policy $\pi_\beta$ — teleop sessions, scripted rollouts, old evaluations. On your rig: 200 episodes at 10 Hz for 25 s is about 50,000 transitions of RealSense frames, 7-dim joint states (six joints plus gripper), commanded actions, and (new in this lesson) a reward. The goal: a policy that outperforms $\pi_\beta$, trained entirely from $\mathcal{D}$. What makes improvement conceivable is dynamic programming — back returns up through the Bellman equation.

Two properties make this backup more than BC-with-extra-steps. It is off-policy by construction — the target does not care which policy produced a transition, so failed episodes become usable signal rather than contamination. And it composes across trajectories: a value function fit on episode 12's reach and episode 47's grasp can rank a combination neither episode executed. That composition, stitching, is how the learner can beat every individual demonstration. But look at the max: in a 7-dim continuous action space it is an actor network or sampling optimizer doing gradient ascent on $Q$, and nothing constrains its proposals to lie near the data. That one unguarded operator is where the field's entire difficulty lives.

The pathology: maximizing over a liar

Your Q-network is a function approximator, so its estimate is the truth plus an error field: $\hat{Q}(s,a) = Q(s,a) + \epsilon(s,a)$. On-support, $\epsilon$ is small and roughly zero-mean — standard generalization. Off-support it is whatever the architecture extrapolates, with no data to pull it back. Now push a max through that noise:

Plug in your numbers: noise $\sigma = 0.05$ on returns normalized to $[0, 1]$, an actor that effectively screens $N \approx 10^3$ candidate actions (so $\sqrt{2\ln N} \approx 3.7$), and $\gamma = 0.99$. The fixed-point bias is at least $0.99 \times 0.05 \times 3.7 / 0.01 \approx 18$ — an eighteen-fold overestimate of the best possible return, from 5% noise and one max operator. And the bound is optimistic — it holds $\sigma$ fixed: the actor's gradient ascent on $\hat{Q}$ migrates toward the largest positive extrapolation errors, so $\sigma$ at the argmax grows without bound. The actor is an adversarial-example generator pointed at your own critic.

What separates offline from online RL here is a feedback-control argument. Online, the agent executes the overestimated action, the environment returns the true reward and next state, and the next regression step pushes $\hat{Q}$ back down: overestimation triggers exactly the data that corrects it — negative feedback. Offline, that wire is cut. The optimizer inflates $\hat{Q}$ at actions nobody took, contradicting evidence can never arrive, and the recursion runs open-loop until the value function is fiction. BC's distribution shift compounded through physics over one episode; offline RL's compounds through bootstrapping over the whole training run, with $1/(1-\gamma) = 100$ as its gain.

import numpy as np

rng = np.random.default_rng(0)

# Normalized 1-D action (think: one joint-velocity channel in [-1, 1]).
# The behavior policy only ever explored the slice [-0.2, 0.2].
actions = rng.uniform(-0.2, 0.2, size=400)
def q_true(a):
    return -2.0 * (a - 0.1) ** 2      # true optimum inside the data
returns = q_true(actions) + rng.normal(0.0, 0.05, size=actions.shape)

# Fit Q with a degree-5 polynomial: any smooth approximator extrapolates.
X = np.vander(actions, N=6, increasing=True)
w, *_ = np.linalg.lstsq(X, returns, rcond=None)

grid = np.linspace(-1.0, 1.0, 2001)   # the FULL actuator range
q_hat = np.vander(grid, N=6, increasing=True) @ w

in_support = np.abs(grid) <= 0.2
print("true max (inside data):     ", round(q_true(0.1), 3))
print("Q-hat max, inside support:  ", round(float(q_hat[in_support].max()), 3))
print("Q-hat max, full range:      ", round(float(q_hat.max()), 3))
print("argmax action:              ", round(float(grid[q_hat.argmax()]), 3))

Run it. Inside the data slice the fit is excellent: estimated max $-0.004$ against a true max of $0.0$. Over the full range the same polynomial reports a max of 212.9, at action $-1.0$ — the edge of the range, as far from the data as the optimizer can travel. No pathology was injected: unbiased noise, 400 clean samples, a reasonable model class. The lie is not in the fit; it is in where you asked. Now picture this across 7 action dimensions and a learned state embedding, each iteration regressing the inflated max into the next target.

Two escape routes: penalize the answer, or never ask the question

Every practical offline RL algorithm answers one design question: how do you use a max operator over a function you can only trust on-support? The conservative family keeps querying arbitrary actions but rigs the critic so off-support answers come back pessimistic. CQL (opens in a new tab) (Conservative Q-Learning, Kumar et al. 2020) is the canonical member: alongside the Bellman error, it adds a term pushing Q down wherever the optimized policy finds high values, and up on dataset actions.

The price is a live hyperparameter, $\alpha$: too small and the exploit survives; too large and the critic is so pessimistic the policy collapses onto the behavior policy — you paid for RL machinery and bought a worse BC. The second family never asks off-support questions at all. IQL (opens in a new tab) (Implicit Q-Learning, Kostrikov, Nair, and Levine 2021) replaces the max over the action space with a statistic computed only on actions in the dataset, using expectile regression — an expectile is to the mean what a quantile is to the median, an asymmetrically weighted version:

import numpy as np

rng = np.random.default_rng(1)

# Q-values of actions the DATASET actually took in one state bucket:
# 20 mediocre grasp approaches and 3 good ones. Nothing else exists.
q_samples = np.concatenate([
    rng.normal(0.20, 0.05, size=20),
    rng.normal(0.90, 0.02, size=3),
])

def fit_expectile(samples, tau, steps=4000, lr=0.05):
    v = samples.mean()
    for _ in range(steps):
        diff = samples - v
        grad = -2.0 * np.mean(np.where(diff > 0, tau, 1.0 - tau) * diff)
        v -= lr * grad
    return v

for tau in (0.5, 0.7, 0.9, 0.99):
    print("tau =", tau, "-> V =", round(float(fit_expectile(q_samples, tau)), 3))
print("best in-sample Q:", round(float(q_samples.max()), 3))

The output walks from the mean to the best in-sample action: $\tau = 0.5$ gives $V = 0.294$ (the behavior policy's mediocre average), $\tau = 0.9$ gives $0.607$, and $\tau = 0.99$ gives $0.862$ against a best in-sample Q of $0.926$. IQL alternates two such regressions — fit $V$ as a high expectile of $Q$ over dataset actions, then fit $Q$ by TD against $r + \gamma V(s')$, which needs no action query at $s'$ at all. No network is ever evaluated outside $\mathcal{D}$: the extrapolation channel is closed by construction, not by a tuned penalty.

IQL trains value functions but no policy. Extraction is a separate, simple step — advantage-weighted regression (AWR (opens in a new tab), Peng et al. 2019). Ask for the best policy that stays close to the behavior policy, and the answer falls out of two lines of variational calculus:

Read the final line as an engineer: it is behavioral cloning with per-transition sample weights $\exp(A/\beta)$. Above-average transitions get up-weighted; below-average ones fade toward zero. Every piece of BC infrastructure from this phase — dataloaders, normalization, the ACT or Diffusion Policy backbone — is reusable; offline RL enters as one extra column in the batch. That is how the field mostly uses it on manipulators: not an alien algorithm family, a principled reweighting of imitation.

When offline RL beats BC on a real robot — and when it cannot

Now the decision you face with your own 200-episode dataset. Say 120 episodes are clean successes, 45 pair a flawless reach with a botched grasp, and 35 recover a clumsy reach into a clean grasp. Diffusion Policy, a faithful density model, learns all of it — including a mode that fumbles grasps. A value function sees the same data differently: the good reach and good grasp segments each carry high-advantage transitions, and dynamic programming can rank the composite — clean reach stitched to clean grasp — above every complete episode. Stitching is the mechanism behind every legitimate claim that offline RL beat BC: the improvement already existed as segments; the value function is the composition operator.

Stitching has preconditions — the terms of the decision table. You need a reward signal to define advantage — without one, BC is the only option. You need diversity with overlap — segments compose only where trajectories pass through similar states; 50 near-identical expert demos offer nothing to stitch. And you need enough data to fit a Q-function, a harder estimation problem than fitting a policy. The empirical record (Levine et al.'s offline RL tutorial (opens in a new tab)) is consistent: on narrow, near-expert data, well-tuned BC matches or beats offline RL; offline RL pulls ahead when data is large, mixed-quality, and reward-labeled.

What is a reward, on a tabletop?

The table's second column hides the real cost. In simulation, reward is a free function call; on your bench it is a labeling pipeline. The practical baseline for manipulation is the sparse success label: $r_t = 0$ everywhere except $r_T = 1$ on success. It is cheap — about two seconds per episode from the final wrist-camera frame, so 400 episodes cost under fifteen minutes — and honest, encoding the task definition and nothing else. Dense shaping (distance terms, grasp bonuses) looks like free signal but breeds reward hacking: a distance-shaped reach reward will happily park the gripper 5 mm from the block forever, never risking the grasp. The catch with sparsity is horizon: one terminal bit must propagate backward through the whole episode.

Two tools soften the sparsity. Hindsight relabeling, for goal-conditioned policies: a failed pick that knocked the block to the table's left edge is a perfect demonstration of reaching the left edge — relabel the episode with the outcome it achieved, and the failure becomes a success for a different goal. Your 45 botched-grasp episodes turn into supervision with zero robot or human time. Automated success detection, once any policy is running: a small classifier on the final RealSense frame (block in bin: yes or no) labels episodes as they happen — making the table's last row realistic rather than hypothetical.

The verdict for this course

Where does this leave the capstone? The BC family carries it. The OpenPI (opens in a new tab) stack you will fine-tune next phase is flow-matching imitation — π₀ matches demonstrations rather than maximizing reward — and latency-aware chunking modifies when and how actions execute, orthogonal to training. Offline RL enters your work twice. As literacy: a growing slice of the VLA literature is offline-RL-shaped (value-guided action selection, advantage-weighted fine-tuning, RL on BC initializations), unreadable without the overestimation story and the CQL/IQL distinction. And as the failure-driven tool: once your logs hold diverse, reward-labeled, learner-distribution data — the table's last row, reached by default — advantage-weighted methods are a cheap upgrade. Until then, offline RL is a solution to a problem you do not have yet.

Where this goes next: Diffusion Policy: actions as denoising gave you the strongest imitator in this phase; this lesson told you when imitation itself is the bottleneck — and what the exit costs. But every load-bearing claim here was an evaluation claim: “beats BC,” “85% versus 92%,” “stitching pays off.” On hardware you will test such claims with maybe 20 rollouts per policy, where a 7-point success-rate difference is statistically invisible. Evaluating policies: statistics you can defend closes the phase by making sure that when you choose between an imitator and a value-based upgrade, the number you point to means something.