JEPA world-model planners fail on pushing tasks mostly by stalling: the object stays in place while the model predicts progress, and the warm-started planner returns nearly the same plan. Unstick breaks this fixed point without test-time weight updates.
Unstick fine-tuned world models
On-policy fine-tuned JEPA world models from Stuck in Place: Stall Diagnosis and Remedies for JEPA Planners (Junhyeok Lee, Dongheon Lee, Kyu Sung Choi).
Latent costs and frames of one PushObj-T episode. Top: base GD planner; bottom: Unstick-Mem (weights fixed at test time). Solid: actual latent cost; dashed: predicted after each chunk; arrow: predicted drop per chunk; gray band: stall. Frames: last step; blue: agent; red: agent path during the stall; dashed outline: goal.
Highlights
| ≥ 96% | 0 of 101 | 53.3 → 78.7% | 64.0 → 80.7% |
|---|---|---|---|
| of the base planner's failures are stalls in 12 of 15 pushing settings, with and without distribution shift. | analyzed stalls are search failures. The model overrates staying in 98% of stalls on T. | PushObj-T success, base → Unstick-Mem (GD). AdaJEPA, with test-time weight updates: 84.7%. | PushT success on clean frames, base → Unstick-Mem. AdaJEPA: 84.0%. |
Method
| Component | What it does | |
|---|---|---|
| FT | On-policy fine-tuning | Before deployment, fine-tunes the model on the base planner's own executions of separate episodes, because the planner visits states that offline data miss. These checkpoints. |
| Mem | Residual memory | Stores the model's one-step errors on executed chunks and adds them back to predictions near them, so a re-proposed stay-in-place chunk is predicted to stay in place. |
| ReInit | Random re-initialization | Starts every replan from a random plan instead of the warm start, so the same state no longer yields the same plan. |
Unstick-Mem = FT + Mem and Unstick-ReInit = FT + ReInit. Neither updates the weights at test time.
Failure analysis of the base GD planner. (a) Idle: agent moves <2 px per MPC step. (b, c) The 101 stalls at MPC step 25 (T, clean); (c) labels give counts. Objective trap: no simple plan lowers the simulated cost of staying by ≥5%; misranked: a simple plan lowers the simulated cost, but the model prefers staying.
Models
hf download jhlee0619/unstick --local-dir FT gives FT/<model>/{hydra.yaml, checkpoints/model_latest.pth, NOTICE}, used with --ft-root FT of the code.
The checkpoints are pickled PyTorch modules; load them with the code of the repository.
| Folder | Fine-tuned from |
|---|---|
pushobj_shape_shift, pusht_visual_shift, mediummaze_dynamics_shift, diversemaze_layout_shift |
AdaJEPA (MIT) |
dinowm_pusht |
DINO-WM (MIT) |
tsdino_pusht |
Temporal Straightening (MIT) |
diversemaze_layout_shift is fine-tuned with --norm-stats val. The two DINOv2-based models contain DINOv2 ViT-S/14 weights (Apache License 2.0). License texts: LICENSE, LICENSES/; sources: NOTICE in each folder.
Results
Seen and unseen shapes (PushObj), success (%):
| Method | T | L | Z | cross | I | small T | square |
|---|---|---|---|---|---|---|---|
| Base | 53.3 ±6.8 |
50.7 ±6.6 |
44.7 ±5.0 |
52.0 ±4.3 |
26.0 ±5.7 |
42.7 ±5.2 |
23.3 ±3.4 |
| + Unstick-Mem | 78.7 ±5.7 |
82.0 ±4.3 |
78.7 ±3.8 |
77.3 ±3.8 |
40.7 ±1.9 |
68.7 ±3.8 |
36.0 ±3.3 |
| + Unstick-ReInit | 76.0 ±4.3 |
71.3 ±2.5 |
76.7 ±1.9 |
71.3 ±5.7 |
50.0 ±11.8 |
68.0 ±5.9 |
42.7 ±0.9 |
| AdaJEPA test-time weight updates |
84.7 ±5.7 |
86.7 ±3.8 |
79.3 ±5.0 |
78.7 ±1.9 |
48.0 ±8.6 |
65.3 ±3.4 |
36.7 ±2.5 |
Visual shifts (PushT), success (%):
| Method | clean | blur | snp | dark 0.9 | red agent | red goal | red block |
|---|---|---|---|---|---|---|---|
| Base | 64.0 ±1.6 |
56.0 ±3.3 |
64.7 ±1.9 |
63.3 ±1.9 |
44.0 ±9.1 |
34.7 ±5.2 |
11.3 ±2.5 |
| + Unstick-Mem | 80.7 ±0.9 |
76.7 ±2.5 |
74.0 ±1.6 |
78.0 ±4.3 |
62.0 ±5.9 |
38.7 ±5.0 |
11.3 ±2.5 |
| + Unstick-ReInit | 69.3 ±6.6 |
59.3 ±2.5 |
68.7 ±6.6 |
68.0 ±5.9 |
58.7 ±11.8 |
40.0 ±5.9 |
9.3 ±0.9 |
| AdaJEPA test-time weight updates |
84.0 ±1.6 |
86.0 ±1.6 |
82.7 ±3.4 |
83.3 ±1.9 |
59.3 ±6.2 |
49.3 ±5.2 |
8.7 ±2.5 |
From the paper, GD planner. Mean ± SD over three evaluation seeds of 50 episodes. Seen shapes: T, L, Z, cross; unseen: I, small T, square. snp: salt-and-pepper noise; dark: darkening level. The paper's tables also list two planner variants (straight chunks, uniform weights), omitted here.
Citation
@article{lee2026stuck,
title = {Stuck in Place: Stall Diagnosis and Remedies for {JEPA} Planners},
author = {Lee, Junhyeok and Lee, Dongheon and Choi, Kyu Sung},
journal = {arXiv preprint arXiv:2611.25415},
year = {2026}
}