Abstract
Pretrained LeWorldModel's reproduction of small action effects is evaluated via Gram matrices across perturbation scales, showing scale-dependent sensitivity differences and higher errors than environment references.
We examine how closely pretrained LeWorldModel (LeWM) reproduces the effects of small action changes. Using matched simulator and model responses, we measure action sensitivity with Gram matrices at three perturbation scales. Across nearly a thousand states from Reacher and OGBench Cube, the most sensitive action directions differ by median angles of about 36^circ and 42^circ, respectively, at the smallest scale. Increasing the perturbation tenfold reduces median normalized Gram discrepancy by roughly 70\% on Reacher and 47\% on Cube. Geometry measured in pixels and embeddings varies more across scales than the chosen physical-state geometry. On held-out directions, environment Grams achieve median embedding response errors of 17\% and 28\%, compared with 26\% and 38\% for an isotropic reference with the same trace. LeWM's errors are higher at this scale. These findings support scale-specific evaluation of action sensitivity; its effect on control remains an open question.
Community
Do world models actually understand how actions affect the future?
We investigate whether LeWorldModel (LeWM) preserves the local geometry of action effects, rather than just predicting future representations.
Using action-Jacobian Gram matrices and matched simulator counterfactuals, we find:
- Directional mismatch: The dominant action directions differ by approximately 36° on Reacher and 42° on OGBench Cube.
- Scale dependence: Increasing action perturbations tenfold reduces geometric discrepancy by roughly 70% on Reacher and 47% on Cube.
- Representation dependence: The measured geometry changes substantially more across perturbation scales in visual embeddings than in physical-state coordinates.
- Held-out validation: LeWM's geometry predicts unseen action-response magnitudes less accurately than the environment-induced metric.
Our takeaway: Evaluating world models requires understanding not only their predictions, but also how their predictions respond to changes in actions.
We'd love to hear your thoughts, particularly on how these geometric discrepancies might affect model-based planning and control!
Get this paper in your agent:
hf papers read 2610.31716 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper