Add libero-all-suites/init_variants/core_random (four-suite v0 4/4 init variant, matched 6,000-episode evaluation)

libero-all-suites/init_variants/core_random/README.md

@@ -0,0 +1,37 @@
+# LoopWAM v0 (video 4 / action 4), initialization variant `core_random` — all four LIBERO suites
+
+Same model, data and training as `libero-all-suites/v0-video4-action4`; only the student initialization differs:
+the six shared core blocks of both experts start as freshly initialized default DiT blocks (PyTorch default initialization, variant seed 42) instead of Wan blocks 9-14. Prelude (Wan blocks 0-2) and coda (Wan blocks 27-29) are as in v0. `init_variant_manifest.json` records the initialization artifact (SHA-256 `893f9a702f57...`),
+its per-block source plan and tensor checks; it was built by `scripts/build_init_variant.py` of the
+[v0-init-variants branch](https://github.com/anhdao69/LoopWAM_NT/tree/v0-init-variants) (`4ebf38a`).
+
+Ten-epoch full-suite training (Spatial, Object, Goal and LIBERO-10; 1,712 demonstrations, 277,713 windows), training seed
+42, global batch 128 (2 GPUs x microbatch 8 x accumulation 8), 21,700 updates, AdamW (0.9, 0.95), weight decay 0.01,
+LR 1e-4 cosine with 5% warm-up, clip 1.0, FP32 master weights + BF16 compute, 584,536,135 policy parameters. Data,
+normalization (`d5ad351eb327...`), VAE, schedule and precision equal the v0 release; `fairness.json` lists the
+checked fields and two documented exceptions (T5 text-embedding files and the canonical donor artifact were regenerated
+on this cluster from the same sources). Training took 22.1 h on 2x NVIDIA H100 80GB HBM3 in one segment.
+
+## Evaluation (matched LIBERO protocol, 6,000 episodes)
+
+**82.20% (4,932/6,000)** — Spatial 92.00%, Object 91.07%, Goal 85.13%, Long 60.60%. Evaluation seeds 42: 85.20%, 43: 85.75%, 44: 75.65%.
+The released `v0-video4-action4` scored 94.92% under the same protocol (Spatial 96.93%, Object 98.73%, Goal 96.13%, Long 87.87%).
+
+Protocol (the campaign's `lightwam_20261009_matched`): four suites x ten tasks x official initial states 0-49 x evaluation
+seeds 42/43/44; one environment per task/seed (env.seed(base seed)) reused for its 50 initial states; global seed set once
+per task/seed and every action query uses the base seed; 30 settling steps; 400 (Spatial/Object/Goal) or 700 (Long)
+policy steps; ten actions executed per 32-action chunk; ten denoising steps, CFG 1, FP32 weights with BF16 autocast;
+success is the environment done flag. Inference replays a CUDA graph of the unchanged native computation, verified
+bit-exact against eager inference on real observations (24/24 queries).
+The same runner scored the released 4/4 policy 454/500 on Long seed 42 versus 452/500
+in the campaign. Evaluated on 4x NVIDIA H100 (videos not saved); `evaluation_summary.json` has per-suite and per-seed
+counts, the protocol and the runner hashes. One training seed: differences between variants of a few points are not
+established as significant.
+
+## Files
+
+`policy.pt` is the native `loopwam-s-v1` checkpoint with the Adam optimizer state removed; every tensor and metadata field was
+checked for exact equality with the final (evaluated) training checkpoint, and the native loader of revision `4ebf38a3c65aa5cc9d6c07d94c52f91c51551b57`
+rebuilt the 4/4 model from it. Load with that revision or a compatible later one: `create_loopwam(checkpoint_path=..., vae_path=...)`.
+Use it for inference or fresh-optimizer fine-tuning, not exact optimizer-state resume. Keep `dataset_stats.json` matched to it.
+Wan VAE, text embeddings and observation preprocessing are external. See `export.json` for source/export SHA-256 hashes.

libero-all-suites/init_variants/core_random/dataset_stats.json

@@ -0,0 +1,50 @@
+{
+  "action": {
+    "default": {
+      "global_max": [
+        0.9375,
+        0.9375,
+        0.9375,
+        0.3557142913341522,
+        0.375,
+        0.375,
+        1.0
+      ],
+      "global_min": [
+        -0.9375,
+        -0.9375,
+        -0.9375,
+        -0.24214285612106323,
+        -0.375,
+        -0.3642857074737549,
+        0.0
+      ]
+    }
+  },
+  "num_episodes": 1712,
+  "num_transition": 277713,
+  "state": {
+    "default": {
+      "global_max": [
+        0.2103137969970703,
+        0.3904264271259308,
+        1.472778081893921,
+        3.7248642444610596,
+        3.5618896484375,
+        1.3863215446472168,
+        0.04232141748070717,
+        0.0013126095291227102
+      ],
+      "global_min": [
+        -0.48278069496154785,
+        -0.3309336006641388,
+        0.008128181099891663,
+        1.002794623374939,
+        -3.6312508583068848,
+        -1.842738389968872,
+        -0.005453015677630901,
+        -0.042015016078948975
+      ]
+    }
+  }
+}

libero-all-suites/init_variants/core_random/evaluation_summary.json

@@ -0,0 +1,247 @@
+{
+  "run": "core_random",
+  "checkpoint": "/lustre/fs1/groups/yshang/an221229/checkpoints/LoopWAM_NT/v0_init_variants/core_random/train/latest.pt",
+  "checkpoint_sha256": "cedc3a02f24978b663b4f3686e7d22435a61ed3eaebf185f8eb08d1b5576ef86",
+  "checkpoint_step": 21700,
+  "backend": "cuda-graph",
+  "episodes": 6000,
+  "successes": 4932,
+  "success_rate": 0.822,
+  "per_suite": {
+    "libero_spatial": {
+      "successes": 1380,
+      "episodes": 1500,
+      "success_rate": 0.92,
+      "seeds": {
+        "42": {
+          "successes": 470,
+          "episodes": 500,
+          "success_rate": 0.94
+        },
+        "43": {
+          "successes": 473,
+          "episodes": 500,
+          "success_rate": 0.946
+        },
+        "44": {
+          "successes": 437,
+          "episodes": 500,
+          "success_rate": 0.874
+        }
+      }
+    },
+    "libero_object": {
+      "successes": 1366,
+      "episodes": 1500,
+      "success_rate": 0.9106666666666666,
+      "seeds": {
+        "42": {
+          "successes": 476,
+          "episodes": 500,
+          "success_rate": 0.952
+        },
+        "43": {
+          "successes": 466,
+          "episodes": 500,
+          "success_rate": 0.932
+        },
+        "44": {
+          "successes": 424,
+          "episodes": 500,
+          "success_rate": 0.848
+        }
+      }
+    },
+    "libero_goal": {
+      "successes": 1277,
+      "episodes": 1500,
+      "success_rate": 0.8513333333333334,
+      "seeds": {
+        "42": {
+          "successes": 423,
+          "episodes": 500,
+          "success_rate": 0.846
+        },
+        "43": {
+          "successes": 442,
+          "episodes": 500,
+          "success_rate": 0.884
+        },
+        "44": {
+          "successes": 412,
+          "episodes": 500,
+          "success_rate": 0.824
+        }
+      }
+    },
+    "libero_10": {
+      "successes": 909,
+      "episodes": 1500,
+      "success_rate": 0.606,
+      "seeds": {
+        "42": {
+          "successes": 335,
+          "episodes": 500,
+          "success_rate": 0.67
+        },
+        "43": {
+          "successes": 334,
+          "episodes": 500,
+          "success_rate": 0.668
+        },
+        "44": {
+          "successes": 240,
+          "episodes": 500,
+          "success_rate": 0.48
+        }
+      }
+    }
+  },
+  "per_seed": {
+    "42": {
+      "successes": 1704,
+      "episodes": 2000,
+      "success_rate": 0.852
+    },
+    "43": {
+      "successes": 1715,
+      "episodes": 2000,
+      "success_rate": 0.8575
+    },
+    "44": {
+      "successes": 1513,
+      "episodes": 2000,
+      "success_rate": 0.7565
+    }
+  },
+  "suite_mean_success_rate": 0.822,
+  "total_policy_steps": 1352408,
+  "total_queries": 137315,
+  "total_query_seconds": 33544.26320329867,
+  "group_seconds": 81406.29511843435,
+  "first_group_completed_at": 1793200302.2424097,
+  "last_group_completed_at": 1793206151.9252894,
+  "protocol": {
+    "name": "lightwam_20261009_matched",
+    "suites": [
+      "libero_spatial",
+      "libero_object",
+      "libero_goal",
+      "libero_10"
+    ],
+    "seeds": [
+      42,
+      43,
+      44
+    ],
+    "tasks": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7,
+      8,
+      9
+    ],
+    "episodes": 50,
+    "max_policy_steps": {
+      "libero_spatial": 400,
+      "libero_object": 400,
+      "libero_goal": 400,
+      "libero_10": 700
+    },
+    "settling_steps": 30,
+    "settling_action": [
+      0,
+      0,
+      0,
+      0,
+      0,
+      0,
+      -1
+    ],
+    "settling_done": "ignored",
+    "replan_steps": 10,
+    "action_horizon": 32,
+    "denoising_steps": 10,
+    "cfg": 1.0,
+    "action_shift": 1.0,
+    "seed_rule": "global seed once per task/seed group; env.seed(base) at construction, env reused for 50 sequential initial states; every query uses the base seed",
+    "success": "environment done flag",
+    "precision": "FP32 model, BF16 autocast, BF16 VAE",
+    "camera": 256,
+    "model_image": "2x224 horizontal",
+    "gripper": "sign(-(2*denormalized_gripper-1))",
+    "videos": false
+  },
+  "runner_sha256": "812893704471b99352b26bf8db91cfe548a11213f9ff7ac40d8f1558f1bce5ea",
+  "adapter_source_sha256": "c3d99fa72cc0be9a518299898515ffcd1036344d984c8a90723cec588f810a51",
+  "code_revision": "4ebf38a3c65aa5cc9d6c07d94c52f91c51551b57",
+  "vae_sha256": "38071ab59bd94681c686fa51d75a1968f64e470262043be31f7a094e442fd981",
+  "torch": "2.7.1+cu128",
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "evaluated_checkpoint": "training checkpoint latest.pt (source_sha256 in export.json; tensors equal policy.pt)",
+  "cuda_graph_gate": {
+    "checkpoint_sha256": "cedc3a02f24978b663b4f3686e7d22435a61ed3eaebf185f8eb08d1b5576ef86",
+    "all_bit_exact": true,
+    "queries": 24,
+    "eager_ms_median": 253.7734815850854,
+    "graph_ms_median": 102.79925540089607,
+    "peak_gpu_gib": 2.9684596061706543,
+    "gpu": "NVIDIA H100 80GB HBM3",
+    "torch": "2.7.1+cu128",
+    "speedup": 2.468631514843378
+  },
+  "runner_parity": {
+    "export_sha256": "1c659a7c2f6970c3ea5972a8fba4abd9db35857c83a67a053c0ee35555fb602b",
+    "suite": "libero_10",
+    "seed": 42,
+    "episodes": 500,
+    "h100_successes": 454,
+    "campaign_successes": 452,
+    "outcome_agreement": 0.912,
+    "note": "released v0-video4-action4 through this runner vs the campaign record (evaluation_report/final/episodes.csv)"
+  },
+  "reference_release_same_protocol": {
+    "repo_path": "libero-all-suites/v0-video4-action4",
+    "success_rate": 0.9491666666666667,
+    "successes": 5695,
+    "per_suite": {
+      "libero_spatial": 0.9693333333333334,
+      "libero_object": 0.9873333333333333,
+      "libero_goal": 0.9613333333333334,
+      "libero_10": 0.8786666666666667
+    }
+  },
+  "runner_sessions": [
+    {
+      "token": "evc101-990547-446267-1793199452",
+      "started": 1793199461.7285995,
+      "finished": 1793205865.2348156,
+      "wall_seconds": 6403.5062165260315,
+      "gpus": "0,1",
+      "workers": 8,
+      "threads": 1,
+      "groups_completed": 120,
+      "errors": 0,
+      "slurm_job_id": "990547"
+    },
+    {
+      "token": "evc101-990547-455508-1793201987",
+      "started": 1793202026.6159778,
+      "finished": 1793206154.5356457,
+      "wall_seconds": 4127.919668197632,
+      "gpus": "2,3",
+      "workers": 8,
+      "threads": 1,
+      "groups_completed": 103,
+      "errors": 0,
+      "slurm_job_id": "990547"
+    }
+  ],
+  "wall_seconds": 6692.807046175003,
+  "wall_note": "first runner start to last runner end; 990547, 4x H100, 16 CPUs; videos not saved"
+}

libero-all-suites/init_variants/core_random/export.json

@@ -0,0 +1,48 @@
+{
+  "id": "v0_core_random_full_libero_h100",
+  "repo_path": "libero-all-suites/init_variants/core_random",
+  "training_path": "/groups/yshang/an221229/checkpoints/LoopWAM_NT/v0_init_variants/core_random/train",
+  "source_sha256": "cedc3a02f24978b663b4f3686e7d22435a61ed3eaebf185f8eb08d1b5576ef86",
+  "source_bytes": 7015628951,
+  "export_sha256": "398b5ae51427f6a1f71d9e84f969db1a54aeac75245eafbb63ade22e77b5ea47",
+  "export_bytes": 2338702667,
+  "step": 21700,
+  "format": "loopwam-s-v1",
+  "optimizer_included": false,
+  "version": "v0",
+  "video_loops": 4,
+  "action_loops": 4,
+  "loop_alignment": "late",
+  "parameters": 584536135,
+  "init_variant": "core_random",
+  "initialization_sha256": "893f9a702f574d39183a009b36eb4776780a7a42675ae04b67a1d9e453d6002c",
+  "initialization_mode": "wan_init_variant_core_random_fresh_optimizer",
+  "dataset_scope": "full_libero",
+  "epochs": 10,
+  "global_batch": 128,
+  "training_seed": 42,
+  "evaluation_status": "matched_libero_6000_h100",
+  "success_rate": 0.822,
+  "successes": 4932,
+  "episodes": 6000,
+  "per_suite_success_rate": {
+    "libero_spatial": 0.92,
+    "libero_object": 0.9106666666666666,
+    "libero_goal": 0.8513333333333334,
+    "libero_10": 0.606
+  },
+  "training_seconds": 79436.95858339593,
+  "resume_starts": 0,
+  "tensor_equality_verified": true,
+  "native_loader_verified": true,
+  "source_revision": "4ebf38a3c65aa5cc9d6c07d94c52f91c51551b57",
+  "compatible_code_revision": "4ebf38a3c65aa5cc9d6c07d94c52f91c51551b57",
+  "code_branch": "v0-init-variants",
+  "normalization_sha256": "d5ad351eb3272f416d34082e8af5483ba1118ceca355a37d9f268fdb269fb3e4",
+  "fairness_check_passed": true,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "world_size": 2,
+  "microbatch": 8,
+  "gradient_accumulation": 8,
+  "created_utc": "2026-10-28T19:11:24.475905+00:00"
+}

libero-all-suites/init_variants/core_random/fairness.json

@@ -0,0 +1,85 @@
+{
+  "passed": true,
+  "problems": [],
+  "documented_exceptions": [
+    {
+      "problem": "data manifest differs from the v0 release: ['text_cache_files']",
+      "accepted_because": "text-embedding files regenerated on this cluster; same 40 prompt-hash file names and valid-token counts as the v0 release"
+    },
+    {
+      "problem": "canonical donor artifact differs from the branch pin",
+      "accepted_because": "the branch pins a3bf2097bafd... (built on HiPerGator); this cluster built a990a5831c63... from the same Wan2.1 source and code (also used by libero-all-suites/v0-video4-action2, whose fairness.json documents the <=1 FP32 ulp difference); the variant artifact was built from it by the branch builder"
+    }
+  ],
+  "branch_verifier": {
+    "passed": false,
+    "problems": [
+      "data manifest differs from the v0 release: ['text_cache_files']"
+    ],
+    "message": "verification failed: data manifest differs from the v0 release: ['text_cache_files']"
+  },
+  "canonical_artifact_sha256": "a990a5831c63e88ae5143689adad2f2aeb02cb1cd151d61ae0dd992990ac3296",
+  "branch_canonical_artifact_sha256": "a3bf2097bafddac278148c1cb0321163f68cfe2b954e206436da0e3f0bd3c535",
+  "version": "v0",
+  "init_variant": "core_random",
+  "initialization_sha256": "893f9a702f574d39183a009b36eb4776780a7a42675ae04b67a1d9e453d6002c",
+  "reference": "/groups/yshang/an221229/checkpoints/LoopWAM_NT/v0_init_variants_work/reference/libero-all-suites/v0-video4-action4",
+  "compared_training_fields": [
+    "seed",
+    "epochs",
+    "global_batch",
+    "train_windows",
+    "planned_updates",
+    "planned_windows",
+    "policy_dtype",
+    "optimizer_state_dtype",
+    "compute_dtype"
+  ],
+  "data_fields": [
+    "action_offsets",
+    "available_episodes",
+    "available_windows",
+    "camera_order",
+    "codebase_version",
+    "complete_500_demo_set",
+    "content_files",
+    "coverage",
+    "dataset_scope",
+    "episode_metadata_sha256",
+    "fps",
+    "info_sha256",
+    "metadata_files",
+    "normalization_scope",
+    "normalization_sha256",
+    "normalization_source",
+    "padding_policy",
+    "seed",
+    "split",
+    "split_algorithm",
+    "suites",
+    "task_counts",
+    "text_cache_files",
+    "text_encoder_provenance",
+    "train_episodes",
+    "train_windows",
+    "validation_episodes",
+    "validation_windows",
+    "video_offsets"
+  ],
+  "ignored_path_fields": [
+    "dataset_dir",
+    "normalization_path",
+    "text_cache_dir"
+  ],
+  "world_size": 2,
+  "microbatch": 8,
+  "gradient_accumulation": 8,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "git_revision": "4ebf38a3c65aa5cc9d6c07d94c52f91c51551b57",
+  "resumed": false,
+  "resume_manifests": [],
+  "normalization_sha256": "d5ad351eb3272f416d34082e8af5483ba1118ceca355a37d9f268fdb269fb3e4",
+  "final_epoch_loss_video": 0.06304270985935839,
+  "final_epoch_loss_action": 0.03254720020010061,
+  "training_hours": 22.06582182872109
+}

libero-all-suites/init_variants/core_random/init_variant_manifest.json

@@ -0,0 +1,76 @@
+{
+  "variant": "core_random",
+  "artifact": "checkpoints/LoopWAM/init_variants/core_random.pt",
+  "artifact_sha256": "893f9a702f574d39183a009b36eb4776780a7a42675ae04b67a1d9e453d6002c",
+  "canonical_artifact": "checkpoints/LoopWAM/wan21_compact_donors.pt",
+  "canonical_artifact_sha256": "a990a5831c63e88ae5143689adad2f2aeb02cb1cd151d61ae0dd992990ac3296",
+  "student_block_plan": [
+    [
+      "wan",
+      0
+    ],
+    [
+      "wan",
+      1
+    ],
+    [
+      "wan",
+      2
+    ],
+    [
+      "random",
+      0
+    ],
+    [
+      "random",
+      1
+    ],
+    [
+      "random",
+      2
+    ],
+    [
+      "random",
+      3
+    ],
+    [
+      "random",
+      4
+    ],
+    [
+      "random",
+      5
+    ],
+    [
+      "wan",
+      27
+    ],
+    [
+      "wan",
+      28
+    ],
+    [
+      "wan",
+      29
+    ]
+  ],
+  "donor_indices": null,
+  "ffn_rule": "canonical_linspace_6144",
+  "variant_seed": 42,
+  "policy_parameters": 584536135,
+  "checks": {
+    "tensors": 677,
+    "copied": 353,
+    "mean": 0,
+    "random": 324,
+    "magnitude_ffn": 0
+  },
+  "core_slots": [
+    3,
+    4,
+    5,
+    6,
+    7,
+    8
+  ]
+}

libero-all-suites/init_variants/core_random/policy.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:398b5ae51427f6a1f71d9e84f969db1a54aeac75245eafbb63ade22e77b5ea47
+size 2338702667

libero-all-suites/init_variants/core_random/training_timing.json

@@ -0,0 +1,77 @@
+{
+  "status": "complete",
+  "completed_updates": 21700,
+  "windows_seen": 2777130,
+  "planned_updates": 21700,
+  "elapsed_training_seconds": 79436.95858339593,
+  "checkpoint_seconds_total": 430.88899150304496,
+  "checkpoint_count": 50,
+  "measured_mean_update_seconds": 3.6387083526470514,
+  "estimated_total_training_hours": 21.933325347900283,
+  "estimate_note": "Projection from measured updates; excludes setup/checkpoint overhead",
+  "latest": {
+    "update": 21700,
+    "epoch": 10,
+    "windows": 81,
+    "windows_seen": 2777130,
+    "effective_passes": 10.0,
+    "seconds": 3.025594472885132,
+    "lr": 1.0000000000000002e-06,
+    "grad_norm": 0.2092287689447403,
+    "branch_grad_after_clip": {
+      "video": 0.17271819710731506,
+      "action": 0.1175771951675415,
+      "proprio": 0.010986397974193096
+    },
+    "latent_cache_rank_local": {
+      "hits": 1388570,
+      "misses": 0,
+      "dummy_encodes": 70,
+      "encoder_samples": 70,
+      "batches": 173580,
+      "read_seconds": 2902.9327297247946,
+      "encode_seconds": 4.397328754886985,
+      "write_seconds": 0.0
+    },
+    "peak_allocated_gb": 59.43190383911133,
+    "peak_reserved_gb": 69.466064453125,
+    "exit4/video_raw": 0.08742289484283071,
+    "exit4/action_raw": 0.038264583658289025,
+    "exit4/video": 0.06648069546546465,
+    "exit4/action": 0.028547404724874614,
+    "loss_video": 0.06648069546546465,
+    "loss_action": 0.028547404724874614,
+    "loop/1/video_state_rms": 3.2863306116174766,
+    "loop/1/action_state_rms": 0.7527765344690394,
+    "loop/2/video_state_rms": 2.4964350947627314,
+    "loop/2/action_state_rms": 0.7905902627073689,
+    "loop/3/video_state_rms": 2.0620060673466436,
+    "loop/3/action_state_rms": 0.8514389226466049,
+    "loop/4/video_state_rms": 2.0366248613522377,
+    "loop/4/action_state_rms": 0.9222065489969136
+  },
+  "launches": [
+    {
+      "event": "launch",
+      "time": "2026-10-27T12:01:48-04:00",
+      "host": "evc101",
+      "slurm_job": "990547",
+      "slurm_step": "2",
+      "variant": "core_random",
+      "gpus": "0,1",
+      "resume": false,
+      "revision": "4ebf38a3c65aa5cc9d6c07d94c52f91c51551b57",
+      "latent_cache": "/tmp/loopwam_iv_core_random_990547/latents",
+      "gpu_name": "NVIDIA H100 80GB HBM3"
+    }
+  ],
+  "gpu_names": [
+    "NVIDIA H100 80GB HBM3"
+  ],
+  "segments": 1,
+  "total_training_seconds_all_segments": 79436.95858339593,
+  "kept_update_seconds_sum": 78961.23218560219,
+  "epoch1_mean_update_seconds": 3.6370585721758655,
+  "epochs2_10_mean_update_seconds": 3.6389562254982364,
+  "timing_note": "One uninterrupted segment. The run shared its node with another training run; the latent cache was seeded from the v0 4/2 run's cache (identical provenance), and clips with cached latents were not decoded (--skip-cached-video-decode; first-update losses are bit-identical to decoding)."
+}