code/report/report.tex

% =====================================================================
%  Arrival-and-hold: fixing horizon-reset procrastination in a learned
%  latent-space controller for a frozen JEPA world model.
%
%  Build:  latexmk -pdf report.tex        (or: pdflatex x2)
%  Figures come from  report/make_figures.py
%  Tables  come from  report/make_tables.py
% =====================================================================
\documentclass[11pt]{article}

\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amssymb}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{xcolor}
\usepackage{microtype}
\usepackage[colorlinks=true,linkcolor=black,citecolor=black,urlcolor=blue]{hyperref}

\graphicspath{{figures/}}

\newcommand{\Lsup}{\mathcal{L}_{\mathrm{sup}}}
\newcommand{\zg}{z_{G}}

\title{\bfseries Arrival-and-Hold:\\
Diagnosing and Fixing Horizon-Reset Procrastination\\
in Latent-Space Control with a Frozen World Model}
\author{LeWM $\times$ PushT control experiment}
\date{}

\begin{document}
\maketitle

% =====================================================================
\begin{abstract}
\noindent
We train a small amortised controller to plan inside the latent space of a
\emph{frozen} JEPA-style world model (LeWM) on the PushT pushing task, and
report a control pathology that inverts a basic expectation from
model-predictive control. Replanning \emph{more often} made the system
\emph{worse}: executing one action block per plan reached the goal on
$50\%$ of held-out episodes, while committing to the entire five-block plan
reached $88\%$ --- a $38$-point gap ($p<10^{-4}$, exact McNemar on paired
episodes). We show the cause is not model error, not optimiser failure, and
not a compute budget: it is the objective. A terminal goal loss
$d(\hat z_H, \zg)$ asks the controller to \emph{arrive exactly at block
$H$}, so every replan resets the deadline and the agent approaches the goal
asymptotically without ever landing. We formalise this as a contraction
recursion $D_{n+1}=cD_n+b$ with a strictly positive fixed point
$D^\ast=b/(1-c)$, and measure $D^\ast$ directly: $0.203$ for a purely
terminal objective, $0.098$ for the original controller. The fix is a
one-line change to the loss --- relabel each training sample with its true
goal offset $q$, penalise the distance \emph{at} $q$, and add a hold term on
every block after $q$. This drives $D^\ast$ down to $0.040$, lifts the
$m{=}1$ success rate from $50\%$ to $94\%$ ($+44$ points, $p<10^{-4}$),
removes the inversion entirely, and reaches parity with a $300\times30$ CEM
planner ($+4$ points, $p=0.69$) while issuing $760\times$ fewer world-model
evaluations per episode than CEM at its best schedule (and $5000\times$
fewer than CEM at the same schedule). We give the full formula inventory
with the purpose and measured effect of every term, an ablation over the
three objective components, and a note on a survivorship confound that makes
the naive cost metric anti-correlated with success ($r=+0.51$).
\end{abstract}

\tableofcontents
\newpage

% =====================================================================
\section{Setup}

\subsection{The frozen world model}

Everything in this report treats the world model as a fixed, non-trainable
oracle. LeWM consists of a ViT-tiny image encoder $E$ (patch size $14$,
$224$px input, $12$ layers, $3$ heads, embedding width $192$) and a
$6$-layer latent \emph{predictor} $P$ with AdaLN action conditioning. Both
are frozen throughout: no gradient ever reaches their parameters. The only
thing we train is a controller that searches in the latent space they
define.

An observation $o$ becomes a latent $z = E(o) \in \mathbb{R}^{192}$. Given a
latent and an action block $b$, the predictor advances the latent one step:
\begin{equation}
  z' = P(z, b).
  \label{eq:predictor}
\end{equation}

\subsection{Action blocking}

The environment runs at a frameskip of $5$, so a single world-model
transition consumes five raw environment actions. An \emph{action block} is
therefore
\begin{equation}
  b \in \mathbb{R}^{10}, \qquad 10 = 5 \text{ raw actions} \times 2 \text{ dims},
\end{equation}
and a plan of horizon $H$ is a stack $b_{1:H} \in \mathbb{R}^{H\times 10}$.
Throughout, $H=5$: one plan covers $25$ raw environment steps.

\subsection{Data}

Latents were pre-extracted for $18{,}685$ demonstration episodes
($2{,}336{,}736$ frames, $192$-dim). Episodes are split with
\texttt{split\_episodes(n, val\_fraction=0.05, seed=0)}, so training and
validation never share an episode. All closed-loop evaluation uses the same
$50$ seeded held-out episodes for every row in every table, which is what
makes the paired statistics in \S\ref{sec:stats} valid.

\subsection{Notation}

\begin{center}
\begin{tabular}{ll}
\toprule
symbol & meaning \\
\midrule
$z_t \in \mathbb{R}^{192}$ & latent state at world-model step $t$ \\
$\zg$                      & goal latent \\
$b_j \in \mathbb{R}^{10}$  & $j$-th action block of a plan \\
$H = 5$                    & plan horizon (blocks) \\
$N = 3$                    & context frames given to the controller \\
$K$                        & refinement iterations at inference \\
$m$                        & blocks executed before replanning \\
$q \in \{1,\dots,H\}$      & true goal offset of a training sample \\
$d_j$                      & predicted latent distance at block $j$ \\
$D_n$                      & mean goal distance at the $n$-th replan \\
\bottomrule
\end{tabular}
\end{center}

% =====================================================================
\section{The formula inventory}
\label{sec:formulas}

This section lists every formula used in the experiment, what it is
\emph{for}, and what it measurably \emph{did}. This is the core of the
report: the entire result is a story about which of these terms was wrong.
Sections~\ref{sec:pathology} onward refer back to these equations by number.

% ---------------------------------------------------------------------
\subsection{Latent rollout}

\begin{equation}
  \hat z_0 = z_t, \qquad
  \hat z_j = P(\hat z_{j-1},\, b_j), \quad j = 1,\dots,H.
  \label{eq:rollout}
\end{equation}

\paragraph{Purpose.} Turn a candidate plan into a predicted latent
trajectory. Because $P$ is frozen and differentiable, the whole rollout is
one differentiable function of $b_{1:H}$, so gradients of any cost defined
on $\hat z_{1:H}$ flow back to the plan --- and, through the controller that
emitted the plan, to the controller weights. This is what makes an amortised
controller possible at all without ever touching the world model.

\paragraph{Effect.} The rollout is autoregressive, so prediction error
compounds with $j$. This matters later: the cost at block $5$ is a
\emph{less} reliable target than the cost at block $1$, which is one reason a
purely terminal objective (Eq.~\eqref{eq:goalloss} with $\alpha=0$) is
fragile.

% ---------------------------------------------------------------------
\subsection{Goal distance}

\begin{equation}
  d_j \;=\; d(\hat z_j, \zg)
      \;=\; \frac{1}{D}\bigl\lVert \hat z_j - \zg \bigr\rVert_2^2,
  \qquad D = 192.
  \label{eq:dist}
\end{equation}

\paragraph{Purpose.} A scalar "how far from the goal'' signal in latent
space. Dividing by the latent dimension $D$ makes the number comparable
across latent widths and keeps it $O(1)$, which in turn lets a single
$\lambda$ balance it against the support term without retuning.

\paragraph{Effect.} This is the quantity every objective below is built
from, and the quantity plotted on the $y$-axis of
Figures~\ref{fig:profiles} and~\ref{fig:contraction}. Note it is a
\emph{latent} distance, not task success --- the two are correlated but not
identical, and \S\ref{sec:survivorship} shows a case where they come apart
badly.

% ---------------------------------------------------------------------
\subsection{Path weights}

\begin{equation}
  w_j \;=\; \frac{(j/H)^2}{\sum_{i=1}^{H-1} (i/H)^2},
  \qquad j = 1,\dots,H-1.
  \label{eq:pathw}
\end{equation}

\paragraph{Purpose.} A normalised weighting over the \emph{intermediate}
blocks of a plan, used by the path term in Eq.~\eqref{eq:goalloss}. The
quadratic ramp deliberately puts almost no weight on early blocks (the agent
should be free to move away from the goal initially if that is what the task
requires) and increasing weight on blocks near the horizon.

\paragraph{Effect.} Because $w$ grows with $j$, the path term reinforces
rather than counteracts the terminal term's late-arrival preference. This
turns out to be part of the problem, not part of the solution: the path term
softens the pathology but does not remove it (\S\ref{sec:ablation}).
Eq.~\eqref{eq:pathw} is also why setting $\alpha=0$ is so destructive --- it
was supplying the only pressure toward early arrival.

% ---------------------------------------------------------------------
\subsection{Goal loss (the original objective)}

\begin{equation}
  \mathcal{L}_{\text{goal}}
    \;=\; \underbrace{d_H}_{\text{terminal}}
        \;+\; \alpha \underbrace{\sum_{j=1}^{H-1} w_j\, d_j}_{\text{path}},
  \qquad \alpha = 0.05.
  \label{eq:goalloss}
\end{equation}

\paragraph{Purpose.} The standard formulation. Reach the goal by the end of
the plan; the small path term is a shaping bonus that discourages wild
excursions on the way.

\paragraph{Effect --- this is the bug.} Read the terminal term literally: it
says \emph{be at the goal exactly at block $H$}, and says nothing about
blocks $1$ through $H-1$ except through a weight that is largest nearest
$H$. Under a receding horizon, the deadline moves. Every time we replan, $H$
is again five blocks away, so the optimal behaviour under this loss is to be
\emph{five blocks away} from the goal --- forever. The agent procrastinates
by construction. \S\ref{sec:pathology} measures this; setting $\alpha=0$
(pure terminal) makes it dramatically worse, which is the cleanest possible
confirmation that the terminal term is the culprit.

% ---------------------------------------------------------------------
\subsection{Arrival-and-hold loss (the fix)}

Each training sample carries the offset $q$ at which its goal frame actually
occurs. During dataset construction, $q$ is sampled as
\begin{equation}
  q \sim \mathcal{U}\{1,\dots,\min(H,\ \text{reach})\},
\end{equation}
where \emph{reach} is how many blocks remain in the episode. The loss is
then
\begin{equation}
  \boxed{\;
  \mathcal{L}_{\text{ah}}
    \;=\; \underbrace{d_q}_{\text{arrival}}
      \;+\; \lambda_h \underbrace{\frac{1}{H-q}\sum_{j>q} d_j}_{\text{hold}}
  \;}
  \label{eq:ahloss}
\end{equation}
(the hold term is defined as $0$ when $q=H$, i.e.\ when there are no blocks
after arrival).

\paragraph{Purpose of the arrival term.} Penalise the distance at the block
where the goal \emph{actually is}, not at a fixed deadline. This makes the
objective invariant to how far away the goal happens to be, which is exactly
the invariance a receding-horizon controller needs.

\paragraph{Purpose of the hold term.} Arrival alone is not enough: it says
"be at the goal at block $q$'' but is indifferent to what happens next, so a
controller could sail straight through the goal. The hold term says
\emph{stay there}. It converts the goal from a waypoint into an attractor.

\paragraph{Critical detail.} $q$ indexes the loss only. It is
\textbf{never} fed to the controller. At inference the controller has no
idea how far the goal is --- it simply learns, over the training
distribution of offsets, to get to the goal as early as possible and stay.
Had we conditioned on $q$, the fix would be a cheat (an oracle input
unavailable at test time) rather than a fix.

\paragraph{Effect.} The measured consequence is the central result of this
report. In Figure~\ref{fig:profiles}, the terminal objective's distance
profile bottoms out at block $5$ \emph{regardless of $q$}; under
arrival-and-hold, the minimum tracks $q$. Success at $m{=}1$ goes from
$50\%$ to $94\%$, and the contraction fixed point $D^\ast$ falls from
$0.098$ to $0.040$.

\paragraph{On $\lambda_h$.} We swept $\lambda_h \in \{0, 0.5, 1\}$. All three
remove the pathology; the differences between them are not statistically
distinguishable at $n=50$ (\S\ref{sec:stats}). $\lambda_h=0.5$ is the best
point estimate at $94\%$ and is used as the headline configuration, but the
honest reading is that \emph{the arrival relabelling does the work} and the
hold term is a modest refinement.

% ---------------------------------------------------------------------
\subsection{Refinement loss}

The controller emits a plan and then iteratively refines it $K$ times. All
$K+1$ intermediate plans are supervised, with geometrically increasing
weight:
\begin{equation}
  \mathcal{L}_{\text{ref}}
    \;=\; \frac{\sum_{k=0}^{K} \rho_k\, \mathcal{L}^{(k)}}
               {\sum_{k=0}^{K} \rho_k},
  \qquad \rho_k = 2^k,
  \label{eq:refloss}
\end{equation}
where $\mathcal{L}^{(k)}$ is Eq.~\eqref{eq:goalloss} or
Eq.~\eqref{eq:ahloss} evaluated on the $k$-th refined plan.

\paragraph{Purpose.} Two things at once. First, every iterate is a valid
plan, so the controller degrades gracefully if we cut refinement short.
Second, the $2^k$ ramp makes later iterates matter more, which is what
pressures the refinement operator to actually \emph{improve} the plan rather
than just perturb it. Eq.~\eqref{eq:refloss} is the outer wrapper around
whichever inner objective is in use, so swapping
Eq.~\eqref{eq:goalloss} for Eq.~\eqref{eq:ahloss} is genuinely a one-line
change.

\paragraph{Effect.} Figure~\ref{fig:refinement} shows the cost dropping
sharply over the first three refinements --- and then, past the trained
depth $K=3$, flattening or slightly \emph{rising}. The mean plan change
$\lvert b^{(k)}-b^{(k-1)}\rvert$ decays but never reaches zero, so
refinement is not converging to a fixed point; it is a learned $K$-step
improvement operator, not an optimiser. This is why $K=5$ at inference is
not reliably better than $K=3$ (\S\ref{sec:ablation}).

% ---------------------------------------------------------------------
\subsection{Behaviour density and the support term}

A conditional Gaussian mixture $p_\theta(b \mid c)$ over action blocks
(16 components, width 256, conditioned on the context embedding $c$) is
fit to the demonstration data. Its per-dimension negative log-likelihood is
\begin{equation}
  s(c, b) \;=\; -\frac{1}{10}\log p_\theta(b \mid c),
  \label{eq:nll}
\end{equation}
and the support penalty is a one-sided hinge against a threshold $c_{95}$:
\begin{equation}
  \Lsup \;=\; \mathbb{E}\Bigl[\bigl(\max(0,\; s(c,b) - c_{95})\bigr)^2\Bigr],
  \qquad c_{95} = 1.5306.
  \label{eq:support}
\end{equation}
$c_{95}$ is the $95$th percentile of $s$ over the demonstration set, so by
construction $5\%$ of real demonstration blocks violate it.

\paragraph{Purpose.} The world model is only accurate on the action
distribution it was trained on. Without a constraint, a planner optimising
$d_H$ will happily find adversarial action sequences that the predictor
\emph{believes} reach the goal but that the real environment does not
follow. The hinge is one-sided so that being \emph{more} typical than the
threshold is free --- we want to bound exploitation, not clone behaviour.

\paragraph{Effect.} Removing it ($\lambda_{\text{sup}}=0$) raises the
violation fraction from $0.187$ to $0.652$ --- the controller immediately
drifts off the demonstration manifold. But success is
\emph{unchanged}: $50\%$ vs $50\%$ at $m{=}1$ ($p=1.0$). The term does what
it says, and what it says was not the bottleneck. See
Table~\ref{tab:support} and \S\ref{sec:ablation}.

% ---------------------------------------------------------------------
\subsection{Total objective}

\begin{equation}
  \mathcal{L}
    \;=\; \mathcal{L}_{\text{ref}}
      \;+\; \lambda_{\text{sup}}\, \Lsup,
  \qquad \lambda_{\text{sup}} = 0.01.
  \label{eq:total}
\end{equation}

% ---------------------------------------------------------------------
\subsection{Controller parameterisation}

The controller conditions on $N{+}1$ tokens (the $N=3$ context latents plus
the goal latent) and emits a plan. Raw plan logits are squashed and rescaled
into the action range:
\begin{equation}
  b \;=\; \mu_a \;+\; \sigma_a \odot \tanh(\tilde b),
  \label{eq:squash}
\end{equation}
with $\mu_a,\sigma_a$ the per-dimension action mean and standard deviation
of the demonstration set.

\paragraph{Purpose.} Hard-bound the action range without a clipping
discontinuity, and centre the parameterisation on the data so that
$\tilde b = 0$ is already a reasonable plan.

\paragraph{Effect.} $\tanh$ saturation means gradients vanish at the
extremes, which is a real cost --- but it makes the plan trivially
environment-safe and removes the need for a separate action-bound penalty.
Eq.~\eqref{eq:squash} also means the support term of
Eq.~\eqref{eq:support} is the only thing constraining \emph{which} in-range
actions the controller may pick.

Each refinement is a learned residual with a learned, per-iteration step
size:
\begin{equation}
  \tilde b^{(k+1)}
    \;=\; \tilde b^{(k)} \;+\; \sigma\!\bigl(\gamma_{\min(k, K_{\max})}\bigr)
          \cdot \Delta^{(k)},
  \label{eq:step}
\end{equation}
where $\sigma$ is the logistic function, $\gamma$ are learned logits, and
$\Delta^{(k)}$ is produced from the features
$[\,\tilde b^{(k)},\; \hat z_{1:H},\; \hat z_{1:H}-\zg,\; d_{1:H}\,]$.

\paragraph{Purpose.} Giving the refiner the current plan, the predicted
trajectory, the goal residual and the distances is what lets it behave like
a learned gradient step without ever running backpropagation at inference.
The $\sigma(\gamma)$ gate keeps every step in $(0,1)$, so refinement cannot
diverge.

\paragraph{Effect.} The learned step sizes at $K{=}3$ came out as
$[0.55, 0.45, 0.31]$ --- monotonically decreasing, i.e.\ the controller
learned a decaying schedule on its own. Beyond the trained depth the last
step size is reused, which is exactly why
Figure~\ref{fig:refinement} shows no further improvement past $k=3$.

% ---------------------------------------------------------------------
\subsection{Contraction model}
\label{sec:contraction-model}

To make the pathology quantitative, we model the closed loop as a scalar
affine recursion on the mean goal distance across replans:
\begin{equation}
  D_{n+1} \;=\; c\,D_n \;+\; b,
  \label{eq:contraction}
\end{equation}
fit by least squares over consecutive replans. If $|c|<1$ this converges to
\begin{equation}
  D^\ast \;=\; \frac{b}{1-c}.
  \label{eq:fixedpoint}
\end{equation}

\paragraph{Purpose.} $c$ is the per-replan contraction rate --- how much of
the remaining distance the controller removes per decision. $b$ is the
constant floor it re-introduces each time. The fixed point $D^\ast$ is the
distance at which those two balance: \textbf{the residual error the
closed loop settles at, no matter how long you run it.} A controller with
$b>0$ literally cannot reach the goal.

\paragraph{Effect.} This is where the diagnosis becomes a number.
Table~\ref{tab:contraction} gives $D^\ast=0.203$ for the terminal-only
objective, $0.098$ for the original, and $0.036$--$0.048$ for the three
arrival-and-hold variants. The ordering matches success rate exactly. Note
also the $R^2$ column: the fit is excellent for terminal-only ($0.938$) and
progressively worse for the corrected controllers ($\approx 0.61$) --- which
is itself informative, because the corrected controllers \emph{terminate}
(they succeed and the episode ends) rather than settling into the smooth
geometric decay that the model describes.

\paragraph{Caveat.} At $m{=}5$ every fit returns $c>1$ ($1.07$--$1.11$), so
Eq.~\eqref{eq:fixedpoint} yields a negative "fixed point'' and is not
interpretable. We report $m{=}1$ fits only. The $m{=}5$ result is not a
failure of the controller but of the model: with only a handful of replans
per episode, and successful episodes terminating early, the surviving trace
is dominated by the hard episodes and rises.

% ---------------------------------------------------------------------
\subsection{Paired statistics}
\label{sec:stats-formulas}

All rows share the same $50$ seeded held-out episodes, so comparisons are
paired; \S\ref{sec:stats} applies these tests. Let
$a_i, b_i \in \{0,1\}$ be the per-episode outcomes of two planners. Using only the discordant episodes
$n_{01} = \lvert\{i: a_i{=}0, b_i{=}1\}\rvert$ and
$n_{10} = \lvert\{i: a_i{=}1, b_i{=}0\}\rvert$, the two-sided exact McNemar
$p$-value is
\begin{equation}
  p \;=\; \min\!\left(1,\;
    2 \cdot 2^{-n} \sum_{i=0}^{k} \binom{n}{i}\right),
  \quad n = n_{01}+n_{10}, \quad k = \min(n_{01}, n_{10}).
  \label{eq:mcnemar}
\end{equation}
The interval is a percentile bootstrap over episodes ($20{,}000$
resamples, seed $0$) on the paired difference in success rate.

\paragraph{Purpose.} With $n=50$, one flipped episode moves the success
rate by $2$ points. Treating two rows as independent binomials would ignore
that they are the \emph{same} episodes and badly overstate the uncertainty;
conditioning on the discordant pairs is the correct test.

\paragraph{Effect.} It changes conclusions. The $+6$-point gap for the
headline controller ($94\%$ vs $88\%$) is \emph{not} significant
($p=0.25$), and neither is its $+4$-point edge over CEM ($p=0.69$). The
$38$- and $72$-point pathology gaps, by contrast, are overwhelming
($p<10^{-4}$). Without the paired test one would be tempted to report a
ranking among the three corrected variants that the data does not support.

% =====================================================================
\section{The pathology}
\label{sec:pathology}

\subsection{The observation}

The receding-horizon parameter $m$ controls how many of the $H=5$ planned
blocks are executed before replanning. Standard MPC theory says smaller $m$
is better: replanning more often lets the controller correct for model
error, so $m{=}1$ should dominate $m{=}5$.

It does the opposite (Figure~\ref{fig:sweep}).

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.62\textwidth]{fig1_execution_sweep.pdf}
  \caption{Success rate against the number of blocks executed per plan.
  Replanning \emph{less} often is monotonically better, for both the learned
  controller and a gradient-free CEM planner. The effect is $38$ points for
  the controller and $54$ for CEM. That both planners show it rules out an
  optimiser bug and points at the shared objective.}
  \label{fig:sweep}
\end{figure}

Crucially, CEM --- which shares the objective but shares no code path with
the controller --- shows the same inversion. That is the observation that
redirected the investigation from the controller to the loss.

\subsection{The mechanism}

The terminal loss $d_H$ in Eq.~\eqref{eq:goalloss} asks the controller to be
at the goal \emph{at block $H$}. Under a receding horizon, block $H$ is
always five blocks in the future. The deadline is reset before it is ever
reached, so the controller's learned policy --- approach to a distance that
is optimal to be at \emph{five blocks before arrival} --- is a stable,
self-reinforcing state. It procrastinates.

Figure~\ref{fig:profiles} is the direct evidence. For each goal offset $q$
in the validation set, we plot the predicted distance
(Eq.~\eqref{eq:dist}) at every block of the rollout
(Eq.~\eqref{eq:rollout}).

\begin{figure}[htbp]
  \centering
  \includegraphics[width=\textwidth]{fig2_arrival_profiles.pdf}
  \caption{Predicted distance $d_j$ at each plan block, one curve per true
  goal offset $q$; stars mark $\arg\min_j d_j$. \textbf{Left} (terminal-only)
  and \textbf{middle} (original): the minimum is pinned at block $5$ for
  every $q$ --- the controller always plans to arrive at the horizon,
  regardless of where the goal actually is. \textbf{Right}
  (arrival-and-hold): the minimum tracks $q$, and for $q{=}1$ the profile is
  \emph{inverted} --- closest at block $1$, then held. This is the fix
  working.}
  \label{fig:profiles}
\end{figure}

\subsection{Quantifying it}

Fitting Eq.~\eqref{eq:contraction} to the closed-loop traces turns the
qualitative story into a number.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=\textwidth]{fig3_contraction.pdf}
  \caption{\textbf{Left:} mean latent goal distance against replan index at
  $m{=}1$; dotted lines are the fitted fixed points $D^\ast$. The
  terminal-only controller plateaus an order of magnitude short of the goal.
  \textbf{Right:} the fitted $D^\ast=b/(1-c)$ per variant, annotated with
  the underlying $c$ and $b$. The ordering matches success rate exactly.}
  \label{fig:contraction}
\end{figure}

\begin{table}[htbp]
  \centering
  \caption{Contraction fits at $m{=}1$, Eq.~\eqref{eq:contraction}. $c$ is
  the per-replan contraction rate, $b$ the re-introduced floor, and
  $D^\ast=b/(1-c)$ the residual distance the closed loop settles at. Lower
  $D^\ast$ is better.}
  \label{tab:contraction}
  \input{tables/contraction}
\end{table}

The terminal-only controller has $c=0.83$: it removes only $17\%$ of the
remaining distance per replan, and re-adds $b=0.035$ each time. That balance
lands at $D^\ast=0.203$, far outside the success threshold. The corrected
controllers roughly halve $c$ \emph{and} shrink $b$, giving
$D^\ast \approx 0.04$.

\subsection{It shows up during training}

The pathology does not require closed-loop rollout to detect. Because the
arrival distance $d_q$ is cheap to log alongside the terminal distance
$d_H$, the divergence is visible in the training curves
(Figure~\ref{fig:training}).

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.66\textwidth]{fig8_training_signal.pdf}
  \caption{Running validation distances during training. The terminal-only
  run drives $d_H$ to $0.013$ while its arrival cost $d_q$ \emph{rises} to
  $0.199$ --- a $13\times$ gap. The corrected run keeps the two within
  $1.5\times$ of each other. Monitoring both is a cheap early-warning
  signal: a widening gap means the controller is learning to arrive late.
  The sawtooth at steps $5000$ and $10000$ is the horizon curriculum
  stepping from $2\to3\to5$ blocks.}
  \label{fig:training}
\end{figure}

\begin{table}[htbp]
  \centering
  \caption{Training configuration and final validation losses. All runs:
  $20{,}000$ steps, batch $128$, Adam at $3\times10^{-4}$, weight decay
  $10^{-4}$, width $256$, depth $4$, $8$ heads, dropout $0.1$, $K=3$
  refinements, horizon curriculum \texttt{0:2, 0.25:3, 0.5:5}. Controller
  size: $6.80$M parameters.}
  \label{tab:training}
  \input{tables/training}
\end{table}

Note the counterintuitive row ordering in Table~\ref{tab:training}: the
terminal-only run has the \emph{best} terminal validation loss ($0.0130$)
and the \emph{worst} task success ($18\%$). It is not underfit. It is
solving the objective it was given, correctly, and that objective is wrong.

% =====================================================================
\section{Results}

\begin{table}[htbp]
  \centering
  \caption{Success rate (\%) on $50$ seeded held-out episodes at both
  execution schedules, with the $m{=}1$ minus $m{=}5$ gap and planning cost.
  \emph{rows/ep} is world-model predictor rows per episode; \emph{rows/call}
  is per solver call, which removes the episode-length confound discussed in
  \S\ref{sec:survivorship}. Controllers use $K=3$.}
  \label{tab:main}
  \input{tables/main_results}
\end{table}

The headline numbers (Table~\ref{tab:main}): the original controller loses
$38$ points by replanning every block. The corrected controller does not ---
it \emph{gains} $6$ --- and its $m{=}1$ success rate of $94\%$ is the best
result in the entire experiment, above both CEM at its best schedule
($90\%$) and the original controller at its best schedule ($88\%$).

\subsection{Cost}

Figure~\ref{fig:pareto} places every configuration on the cost/accuracy
plane.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.72\textwidth]{fig6_pareto.pdf}
  \caption{Success against planning cost (log scale). The corrected
  controllers sit at the top-left: highest success, and roughly $760\times$
  fewer world-model evaluations per episode than CEM at its best schedule.
  The original controller at $m{=}1$ (the $\times$) is strictly dominated ---
  it costs twice as much as the corrected controllers because its episodes
  run longer, and succeeds half as often.}
  \label{fig:pareto}
\end{figure}

CEM at $m{=}5$ reaches $90\%$ using $55{,}800$ predictor rows per episode.
The corrected controller reaches $94\%$ using $73$ --- a $760\times$
reduction, and $5000\times$ against CEM at the same $m{=}1$ schedule. The
difference in success is not statistically significant ($+4$ points,
$p=0.69$); the difference in cost is between two and three orders of
magnitude. Wall-clock tells the same story: $0.26$ s per episode against
$1.98$ s. That is the practical case for amortising the planner --- but only
once the objective is right, since the \emph{original} amortised controller
was worse than CEM at $m{=}5$ despite the same cost advantage.

% =====================================================================
\section{Ablation study}
\label{sec:ablation}

The objective, Eq.~\eqref{eq:total}, has three components beyond the
terminal term: the path term ($\alpha$), the support term
($\lambda_{\text{sup}}$), and the arrival/hold relabelling ($\lambda_h$). We
ablate each.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.86\textwidth]{fig4_ablation.pdf}
  \caption{Success at both execution schedules for every objective variant.
  The number below each pair is the gap ($m{=}1$ minus $m{=}5$): red is the
  pathology, green is its absence. Every variant reaches $88$--$92\%$ at
  $m{=}5$ --- the differences are entirely in the $m{=}1$ column, which is
  precisely the claim that the objective, not the model or the capacity,
  determines closed-loop behaviour.}
  \label{fig:ablation}
\end{figure}

\subsection{Path term ($\alpha: 0.05 \to 0$)}

Removing the path term is the most destructive single change:
$50\% \to 18\%$ at $m{=}1$ ($-32$ points, $p=0.0004$), and the gap widens
from $-38$ to $-72$ (Figure~\ref{fig:ablation}, leftmost pair). The
contraction rate degrades from $c=0.58$ to $c=0.83$ and $D^\ast$ doubles.

\paragraph{Reading.} The path term was \emph{partially masking} the
pathology. Because $w_j$ weights blocks near the horizon most, it applies
some pressure to be close to the goal before block $H$ --- a weak, indirect
version of the arrival term. Removing it exposes the terminal objective in
its pure form. This is the ablation that identified the terminal term as the
root cause: if the path term helps by pulling the cost earlier, then the
problem is that the cost is too late.

\subsection{Support term ($\lambda_{\text{sup}}: 0.01 \to 0$)}

\begin{table}[htbp]
  \centering
  \caption{Support statistics at $m{=}1$, from Eq.~\eqref{eq:nll} and
  Eq.~\eqref{eq:support}. Violation fraction is the share of emitted blocks
  with NLL/dim above $c_{95}$. By construction $5\%$ of \emph{demonstration}
  blocks exceed the threshold.}
  \label{tab:support}
  \input{tables/support}
\end{table}

Removing the support term does exactly what it should to the density
statistics --- the violation fraction jumps from $0.187$ to $0.652$, and
$\Lsup$ rises $12\times$ --- and does \emph{nothing} to task success: $50\%$
vs $50\%$ at $m{=}1$ ($\Delta = 0$, $p = 1.0$), $90\%$ vs $88\%$ at $m{=}5$.

\paragraph{Reading.} This is a genuine negative result and worth stating
plainly. On this task the world model is evidently robust enough that
off-manifold actions do not produce exploitable prediction error at the
scale the controller can find. We keep the term because it is nearly free
($\lambda_{\text{sup}} = 0.01$) and because the failure mode it guards
against is catastrophic when it does occur --- but on PushT with LeWM, it is
insurance, not a load-bearing component. Note also that the corrected
controller's violation fraction ($0.208$) is slightly \emph{higher} than the
original's ($0.187$): arriving early requires more decisive action blocks,
which sit further into the tail of the demonstration distribution.

\subsection{Arrival-and-hold ($\lambda_h$)}

This is the fix. Relabelling the loss to the true goal offset lifts $m{=}1$
success from $50\%$ to $90$--$94\%$ across all three $\lambda_h$ settings
and eliminates the execution-length inversion in every case.

\begin{center}
\begin{tabular}{lrrr}
\toprule
 & $\lambda_h=0$ & $\lambda_h=0.5$ & $\lambda_h=1$ \\
\midrule
success, $m{=}1$ (\%)     & 90     & \textbf{94} & 92 \\
gap ($m{=}1 - m{=}5$)     & $-2$   & $+6$        & $+4$ \\
$D^\ast$                  & 0.0475 & 0.0397      & 0.0362 \\
val $d_q$                 & 0.0125 & 0.0167      & 0.0130 \\
\bottomrule
\end{tabular}
\end{center}

\paragraph{Reading.} The arrival relabelling carries the effect; the hold
term is a refinement. None of the pairwise differences among the three is
significant (all $p \geq 0.62$, \S\ref{sec:stats}), so we do not claim
$\lambda_h=0.5$ is \emph{the} right value --- only that it is the best point
estimate and that any $\lambda_h \in [0,1]$ works. What \emph{is} significant
is all three against the original ($+40$ to $+44$ points, $p<10^{-4}$).

\subsection{Refinement depth $K$}

\begin{figure}[htbp]
  \centering
  \includegraphics[width=\textwidth]{fig5_refinement.pdf}
  \caption{\textbf{Left:} terminal ($d_H$, solid) and arrival ($d_q$, dashed)
  cost against refinement index; shading marks depth beyond the trained
  $K=3$. Note the terminal-only controller's dashed curve sits $21\times$
  above its solid one and barely moves --- refinement optimises the
  objective it was given, and that objective ignores $d_q$. The corrected
  controller's gap is $5.4\times$. \textbf{Right:} the mean plan change
  decays but never reaches zero, so refinement is a learned improvement
  operator rather than a converging optimiser.}
  \label{fig:refinement}
\end{figure}

Sweeping $K \in \{0,1,2,3,5\}$ at $m{=}5$ on the original controller gives
$66, 86, 82, 88, 90\%$ --- non-monotonic, and the $K{=}0$ case
(a single feedforward plan, no refinement at all) already reaches $66\%$.
Refinement helps, but it is not where the leverage is: changing the
objective moved success by $44$ points, while adding three refinement
iterations moved it by $22$. The learned step sizes of
Eq.~\eqref{eq:step} came out nearly identical in both the original and
corrected runs ($[0.55, 0.45, 0.31]$ vs $[0.55, 0.45, 0.32]$), which is
further evidence that the refinement machinery was never the problem --- the
two controllers refine in the same way, toward different objectives.

% =====================================================================
\section{Statistical validation}
\label{sec:stats}

\begin{table}[htbp]
  \centering
  \caption{Paired comparisons on the same $50$ held-out episodes. $\Delta$
  is the difference in success rate (percentage points), the interval is a
  $20{,}000$-resample percentile bootstrap, and $p$ is the two-sided exact
  McNemar test of Eq.~\eqref{eq:mcnemar}. $\ast$ marks $p<0.05$.}
  \label{tab:paired}
  \input{tables/paired_stats}
\end{table}

Three things are worth drawing out of Table~\ref{tab:paired}.

\paragraph{The pathology is real and large.} Every $m{=}1$ vs $m{=}5$
comparison for an uncorrected objective is significant at $p<10^{-4}$, with
confidence intervals that exclude zero by a wide margin. This is not a noise
artefact of $n=50$.

\paragraph{The fix is real and large.} Arrival-and-hold beats the original
by $+44$ points at $m{=}1$ ($[+30,+58]$, $p<10^{-4}$) and beats
terminal-only by $+76$.

\paragraph{The fine-grained rankings are not.} The corrected controller's
$+6$ over its own $m{=}5$ schedule ($p=0.25$), its $+4$ over CEM
($p=0.69$), and all three pairwise $\lambda_h$ comparisons ($p \geq 0.62$)
are indistinguishable from noise. The one marginal result is
arrival-and-hold versus the original at $K{=}3$, $m{=}4$: $+14$ points,
$p=0.039$ --- significant, but only just, and it would not survive a
multiple-comparison correction across the $17$ tests in this table. We
report it as suggestive rather than established.

% =====================================================================
\section{A survivorship confound}
\label{sec:survivorship}

One measurement in this experiment is actively misleading, and it is worth
recording because it nearly inverted a conclusion.

PushT episodes \textbf{terminate on success}
($\lVert\Delta \text{pos}\rVert < 20$ and
$\lvert\Delta\theta\rvert < \pi/9$). The number of solver calls per
evaluation is fixed by the schedule ($200$ at $m{=}1$, $40$ at $m{=}5$), but
the number of environments \emph{still running} at each call is not: good
controllers finish early and drop out.

Consequently:
\begin{equation}
  \frac{\text{predictor rows}}{\text{solver call}}
    \;=\; \text{mean number of episodes still alive},
  \label{eq:survivorship}
\end{equation}
which measures \emph{episode length}, not per-decision cost. Every
controller in this study has \emph{identical} per-decision cost --- same
architecture, same $K$, same horizon. Figure~\ref{fig:survivorship}
decomposes this.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=\textwidth]{fig7_survivorship.pdf}
  \caption{\textbf{Left:} across all $36$ paired evaluation rows, better
  controllers report \emph{higher} mean terminal distance ($r=+0.51$).
  \textbf{Right:} rows-per-call decomposed --- it is exactly the mean number
  of surviving episodes. The terminal-only controller looks $2.5\times$ more
  "expensive'' than the corrected one purely because its episodes never
  end.}
  \label{fig:survivorship}
\end{figure}

The left panel is the sharper warning. \emph{Mean terminal distance is
positively correlated with success rate} ($r=+0.51$): the better the
controller, the worse its average reported cost. The reason is the same ---
successful episodes exit the average early, leaving the mean dominated by
the hard episodes that a good controller is still working on, while a bad
controller's easy-but-unfinished episodes keep its average low.

\paragraph{Practical rule.} On any benchmark with success-triggered
termination, per-step cost and per-step error averages are survivorship
statistics. Report cost per \emph{decision} and success separately, and
never rank controllers by mean episode cost.

% =====================================================================
\section{Limitations}

\begin{itemize}
  \item \textbf{$n=50$.} The held-out set is small. It is large enough to
  establish the $38$-, $44$- and $72$-point effects with certainty, and far
  too small to rank the three corrected variants against each other. We have
  been explicit about which claims fall on which side of that line.

  \item \textbf{One task, one world model.} PushT with LeWM. The
  horizon-reset argument is a property of the \emph{objective} under a
  receding horizon and should generalise, but that is an argument, not
  evidence.

  \item \textbf{Single seed per configuration.} Each row is one training run
  evaluated on $50$ episodes. The pairing controls episode-level variance,
  not seed-level variance.

  \item \textbf{The $m{=}5$ contraction fits are uninterpretable}
  ($c>1$), as noted in \S\ref{sec:contraction-model}. The contraction
  analysis is evidence at $m{=}1$ only.

  \item \textbf{The support term is unvalidated on this task.} It has the
  intended effect on the density statistics and no measurable effect on
  success. We cannot say from this experiment whether it would matter on a
  task where the world model is more exploitable.

  \item \textbf{Marginal results flagged.} The $p=0.039$ comparison against
  $m{=}4$ would not survive correction for the $17$ tests reported.
\end{itemize}

% =====================================================================
\section{Conclusion}

A learned latent-space controller failed in a way that looked like a
capacity or optimiser problem and was neither. It was a specification
problem: $d(\hat z_H, \zg)$ means \emph{arrive at block $H$}, and under a
receding horizon block $H$ never arrives. The controller learned the correct
solution to the wrong question, which is why its \emph{training} loss was
excellent ($0.0130$, the best of any variant) while its success rate was the
worst ($18\%$).

The fix required no architectural change, no additional compute, and no
world-model retraining --- only relabelling the loss to the goal's true
offset $q$ and adding a hold term after it, Eq.~\eqref{eq:ahloss}. The
result is $94\%$ at the most frequent replanning schedule, parity with a CEM
planner using two to three orders of magnitude more world-model calls, and a
contraction fixed point reduced from $0.098$ to $0.040$.

Three transferable lessons:
\begin{enumerate}
  \item \textbf{Under a receding horizon, penalise arrival, not the
  terminal step.} A fixed-deadline cost composed with a moving deadline is a
  procrastination incentive.
  \item \textbf{Log the arrival cost next to the terminal cost.} The gap
  between them (Figure~\ref{fig:training}) diagnoses this failure at
  training time, with no rollout.
  \item \textbf{Beware survivorship in success-terminated benchmarks.} Mean
  episode cost and mean episode error both invert.
\end{enumerate}

% =====================================================================
\appendix
\section{Reproducing the report}

All figures and tables are generated from the raw result files, so nothing
in this document is hand-transcribed:

\begin{verbatim}

python report/make_figures.py     # -> report/figures/*.pdf, *.png

python report/make_tables.py      # -> report/tables/*.tex

latexmk -pdf report/report.tex    # -> report/report.pdf

\end{verbatim}

\texttt{make\_figures.py} reads \texttt{data/runs/eval/results.jsonl},
\texttt{data/runs/diagnostics/*} and the saved checkpoints;
\texttt{make\_tables.py} recomputes the paired statistics with the same
functions used during the study (\texttt{scripts/paired\_stats.py}) and
persists them to \texttt{data/runs/eval/paired\_stats.jsonl}.

The original controller predates the per-offset profile logging, so its
entry in Figure~\ref{fig:profiles} and its arrival column in
Table~\ref{tab:training} come from \texttt{report/recover\_profiles.py},
which recomputes them with the same \texttt{evaluate()} on the same held-out
split and a seeded loader. Its recovered $q{=}1$ profile
$[0.0914, 0.0537, 0.0337, 0.0229, 0.0133]$ matches the values recorded
during the original run, confirming the recovery is faithful.

\end{document}