WorldMem baseline evaluations
Four independent model workspaces and conda environments for the existing Minecraft and RealEstate10K evaluation protocols. This repository contains evaluation code and fixed manifests; pretrained weights are downloaded from the authors' repositories by the setup job.
| Method | Workspace | Environment | Common evaluation |
|---|---|---|---|
| DecMem public Wan reference | DecMem/ |
/proj/cvl/users/x_fahkh2/envs/decmem-eval |
Minecraft: 600 observed + 500 generated frames |
| Matrix-Game 2.0 | Matrix-Game/Matrix-Game-2/ |
/proj/cvl/users/x_fahkh2/envs/matrix-game2-eval |
Minecraft: 600 observed + 500 generated frames |
| LIVE | LIVE/ |
/proj/cvl/users/x_fahkh2/envs/live-eval |
RE10K: 100 observed + 300 generated frames |
| Geometry Forcing | GeometryForcing/ |
/proj/cvl/users/x_fahkh2/envs/geometry-forcing-eval |
RE10K: causal 12-slot/CFG1 evaluation; native interpolation is an offline control |
The Berzelius checkout location is /proj/cvl/users/x_fahkh2/worldmem-baseline-evals. Submit jobs from Berzelius Hopper. First, deployment/setup_environment_berzelius.sh creates all four environments under /proj/cvl/users/x_fahkh2/envs/, builds required CUDA extensions, downloads checkpoints, validates data, and caches metric weights. It installs the official pinned Miniforge 24.7.1-2 manager under /proj/cvl/users/x_fahkh2/envs/baseline-miniforge3 and restricts conda environment creation to the conda-forge channel. It uses that manager by its absolute path, without a Miniforge module or shell-profile changes. It uses the dedicated berzelius-hopper-cpu partition, 14 CPU cores, no GPUs, and an eight-hour limit. It runs no model inference. Setup logs are written under slurm_logs/, and records under artifacts/environment_setup_JOBID/. All setup, smoke and full evaluation jobs set both TMPDIR and TRITON_CACHE_DIR to /proj/cvl/users/x_fahkh2/worldmem-baseline-evals/cache/, as required by the project.
After setup completes successfully, each method has its own smoke job and full evaluation job:
| Method | Smoke job (one H200, four hours) | Full job (eight H200s, 48 hours) |
|---|---|---|
| DecMem | deployment/smoke_decmem_1h200.sh |
deployment/evaluate_decmem_8h200.sh |
| Matrix-Game 2.0 | deployment/smoke_matrix_game2_1h200.sh |
deployment/evaluate_matrix_game2_8h200.sh |
| LIVE | deployment/smoke_live_1h200.sh |
deployment/evaluate_live_8h200.sh |
| Geometry Forcing (causal, 12 slots, CFG1) | deployment/geometry_causal_controls_1h200.sh |
deployment/evaluate_geometry_forcing_8h200.sh |
The completed strictly autoregressive diagnostics used deployment/geometry_causal_controls_1h200.sh: 8 slots/guidance 4, 12 slots/guidance 4, and 12 slots/guidance 1 on two monitor scenes with 32 future frames. Its optional fourth task replaces generated history with ground-truth history while retaining the same noise masks; this task is diagnostic only. The autoregressive tasks use past RGB and poses through the current target; unused model slots contain noise and copies of the current camera. The full launcher uses the selected 12-slot/guidance-1 setting without waiting for the fourth task.
The full causal GF job includes its own 300-frame preflight. Standalone LIVE and Minecraft smoke jobs each generate and score one real-data case and check aggregation: LIVE generates 104 frames, DecMem 20, and Matrix 24. The separate native GF offline smoke generates 300 frames using keyframes and interpolation. Smoke and causal-diagnostic logs are under the checkout's slurm_logs/; full logs are under each method workspace's slurm_logs/. Results are under each method workspace's outputs/.
The full launchers partition cases without duplication, score each case, and aggregate all requested outputs. Minecraft evaluates 300 cases with seeds 42 and 43 in sequential array elements; RE10K evaluates the fixed final100 scope with seed 0. Each GPU job requests 14 CPU cores per H200 and leaves memory allocation to Slurm. Each model has a separate submitted job. No script submits other jobs automatically.
Expected existing dataset locations are /proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/minecraft and /proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/re10k_dfot/real-estate-10k. Minecraft additionally requires the existing vae_features/ cache. The frozen scoring implementation is included in references/worldmem/; a separate DeMemWM checkout is unnecessary.
Protocol details. Minecraft observes source frames [100,700) and scores [700,1200) at 360×640, against the fixed Oasis-VAE reference cache. DecMem uses actions and poses with UniPC20/CFG5; Matrix uses calibrated action conditioning and its native three-step sampler. Temporal padding duplicates only observed frames, and the scored suffix contains exactly 500 frames. Matrix's incoming-to-outgoing action conversion is documented in its method manifest.
RE10K follows G0…G199,G199…G0, including the repeated turnaround, with 100 observed and 300 generated frames at 256×256. LIVE uses its 18-step sampler with the requested 12-frame window. The Geometry Forcing offline control consumes the last observed image (timeline 99), predicts 19 sparse keyframes over timeline 99–399, and interpolates the other frames with its native 16-slot backbone. Its prediction guidance is 4 with stabilization 0.02; interpolation guidance is 1.5. Short camera batches repeat the final camera to fill unused model slots. This offline workflow is not the causal evaluation described below. Scene seeds follow 1_000_003 * metadata_index + 7. The common RGB targets, AlexNet LPIPS, mean frame PSNR, SSIM and pooled FID are retained; these are not the paper's exact metric implementation. Archived offline GF outputs use the geometry-forcing_native_re10k_ prefix.
Geometry Forcing full causal evaluation. The full launcher now runs the tested causal evaluator at 12 active slots (11 past frames plus one target), guidance 1, DDIM50, eta 0 and stabilization 0.02. It first generates and scores one complete 300-frame monitor loop, then automatically evaluates final100 on eight H200 workers and aggregates all 30,000 frames. The teacher-history diagnostic is optional and does not gate this run. Outputs use geometry-forcing_ar12_cfg1_re10k_100cases_seed0_job<JOB>; logs are in GeometryForcing/slurm_logs/. The native workflow described above remains available only through deployment/smoke_geometry_forcing_1h200.sh as an offline control.
Validation status. Causal tasks 85128_0/1/2 completed two monitor scenes with 32 frames each. CFG1 avoided CFG4's noise collapse but still showed blur and motion lag; its full-horizon quality is not established. These monitor scenes are disjoint from final100, and the full run freezes the chosen CFG1 setting. Native smoke 85127 completed 300 frames but uses future keyframes during interpolation. Context budgets and method differences are recorded in deployment/context_budget_audit.json.
Upstream revisions are recorded in upstream_sources.json. Sources retain their original licenses and notices, described in THIRD_PARTY.md. In particular, the frozen WorldMem reference is covered by S-Lab License 1.0 and is not covered by the other projects' permissive licenses.