Stuck in Place: Stall Diagnosis and Remedies for JEPA Planners. Junhyeok Lee, Dongheon Lee, Kyu Sung Choi. Seoul National University.

arXiv 2611.25415 Project page Code on GitHub

JEPA world-model planners fail on pushing tasks mostly by stalling: the object stays in place while the model predicts progress, and the warm-started planner returns nearly the same plan. Unstick breaks this fixed point without test-time weight updates.

Unstick fine-tuned world models

On-policy fine-tuned JEPA world models from Stuck in Place: Stall Diagnosis and Remedies for JEPA Planners (Junhyeok Lee, Dongheon Lee, Kyu Sung Choi).

Latent costs and frames of one PushObj-T episode: the base GD planner stalls while the model predicts progress; Unstick-Mem succeeds.

Latent costs and frames of one PushObj-T episode. Top: base GD planner; bottom: Unstick-Mem (weights fixed at test time). Solid: actual latent cost; dashed: predicted after each chunk; arrow: predicted drop per chunk; gray band: stall. Frames: last step; blue: agent; red: agent path during the stall; dashed outline: goal.

Highlights

≥ 96% 0 of 101 53.3 → 78.7% 64.0 → 80.7%
of the base planner's failures are stalls in 12 of 15 pushing settings, with and without distribution shift. analyzed stalls are search failures. The model overrates staying in 98% of stalls on T. PushObj-T success, base → Unstick-Mem (GD). AdaJEPA, with test-time weight updates: 84.7%. PushT success on clean frames, base → Unstick-Mem. AdaJEPA: 84.0%.

Method

Component What it does
FT On-policy fine-tuning Before deployment, fine-tunes the model on the base planner's own executions of separate episodes, because the planner visits states that offline data miss. These checkpoints.
Mem Residual memory Stores the model's one-step errors on executed chunks and adds them back to predictions near them, so a re-proposed stay-in-place chunk is predicted to stay in place.
ReInit Random re-initialization Starts every replan from a random plan instead of the warm start, so the same state no longer yields the same plan.

Unstick-Mem = FT + Mem and Unstick-ReInit = FT + ReInit. Neither updates the weights at test time.

Failure analysis of the base GD planner: stall statistics, calibration of model cost against simulated cost, and plan ranking at stalled states.

Failure analysis of the base GD planner. (a) Idle: agent moves <2 px per MPC step. (b, c) The 101 stalls at MPC step 25 (T, clean); (c) labels give counts. Objective trap: no simple plan lowers the simulated cost of staying by ≥5%; misranked: a simple plan lowers the simulated cost, but the model prefers staying.

Models

hf download jhlee0619/unstick --local-dir FT gives FT/<model>/{hydra.yaml, checkpoints/model_latest.pth, NOTICE}, used with --ft-root FT of the code. The checkpoints are pickled PyTorch modules; load them with the code of the repository.

Folder Fine-tuned from
pushobj_shape_shift, pusht_visual_shift, mediummaze_dynamics_shift, diversemaze_layout_shift AdaJEPA (MIT)
dinowm_pusht DINO-WM (MIT)
tsdino_pusht Temporal Straightening (MIT)

diversemaze_layout_shift is fine-tuned with --norm-stats val. The two DINOv2-based models contain DINOv2 ViT-S/14 weights (Apache License 2.0). License texts: LICENSE, LICENSES/; sources: NOTICE in each folder.

Results

Seen and unseen shapes (PushObj), success (%):

Method T L Z cross I small T square
Base 53.3
±6.8
50.7
±6.6
44.7
±5.0
52.0
±4.3
26.0
±5.7
42.7
±5.2
23.3
±3.4
+ Unstick-Mem 78.7
±5.7
82.0
±4.3
78.7
±3.8
77.3
±3.8
40.7
±1.9
68.7
±3.8
36.0
±3.3
+ Unstick-ReInit 76.0
±4.3
71.3
±2.5
76.7
±1.9
71.3
±5.7
50.0
±11.8
68.0
±5.9
42.7
±0.9
AdaJEPA
test-time weight updates
84.7
±5.7
86.7
±3.8
79.3
±5.0
78.7
±1.9
48.0
±8.6
65.3
±3.4
36.7
±2.5

Visual shifts (PushT), success (%):

Method clean blur snp dark 0.9 red agent red goal red block
Base 64.0
±1.6
56.0
±3.3
64.7
±1.9
63.3
±1.9
44.0
±9.1
34.7
±5.2
11.3
±2.5
+ Unstick-Mem 80.7
±0.9
76.7
±2.5
74.0
±1.6
78.0
±4.3
62.0
±5.9
38.7
±5.0
11.3
±2.5
+ Unstick-ReInit 69.3
±6.6
59.3
±2.5
68.7
±6.6
68.0
±5.9
58.7
±11.8
40.0
±5.9
9.3
±0.9
AdaJEPA
test-time weight updates
84.0
±1.6
86.0
±1.6
82.7
±3.4
83.3
±1.9
59.3
±6.2
49.3
±5.2
8.7
±2.5

From the paper, GD planner. Mean ± SD over three evaluation seeds of 50 episodes. Seen shapes: T, L, Z, cross; unseen: I, small T, square. snp: salt-and-pepper noise; dark: darkening level. The paper's tables also list two planner variants (straight chunks, uniform weights), omitted here.

Citation

@article{lee2026stuck,
  title   = {Stuck in Place: Stall Diagnosis and Remedies for {JEPA} Planners},
  author  = {Lee, Junhyeok and Lee, Dongheon and Choi, Kyu Sung},
  journal = {arXiv preprint arXiv:2611.25415},
  year    = {2026}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Paper for jhlee0619/unstick