Add verified full-suite LoopWAM v0 4/1 training checkpoint

README.md

@@ -9,7 +9,7 @@ tags:
 
 # LoopWAM final checkpoints
 
-Final trained policy weights from the LoopWAM experiments. This repository contains **14 checkpoints**, separated by training data scope. Both original and repeated 4/4 runs are retained. All production checkpoints are from completed ten-epoch runs with global batch 128 and training seed 42.
+Final trained policy weights from the LoopWAM experiments. This repository contains **15 checkpoints**, separated by training data scope. Both original and repeated 4/4 runs are retained. All production checkpoints are from completed ten-epoch runs with global batch 128 and training seed 42.
 
 ## October 9 additions: all-loop KV
 
@@ -25,7 +25,7 @@ equivalence. Each directory includes normalization, training provenance,
 three evaluation summaries and a tensor-verified optimizer-free native policy.
 The Long aligned4/1 and both original/repeat4/4 weights were hash-checked and
 are unchanged. Their additional evaluation seeds do not create new weights.
-The new full-suite4/1 training is ongoing; no intermediate checkpoint is uploaded.
+The full-suite v0 4/1 training-only checkpoint is now included at `libero-all-suites/v0-video4-action1`. It has not been evaluated after training.
 
 Expanded seed42: concat404/500 versus aligned413/500 (paired p=0.4814). The
 original three-seed decision remains inconclusive. This grid overlaps the
@@ -40,6 +40,14 @@ global batch 128, training seed 42). Training took 10h 58m 46s on two H100s.
 **Not evaluated:** this run finished under the requested training-only workflow.
 Normalization, training metadata, fairness checks and export hashes are included.
 
+## October 10 addition: full-suite 4/1
+
+| Directory | Model | Video/action loops | Parameters | Evaluation |
+|---|---|---:|---:|---|
+| [libero-all-suites/v0-video4-action1](libero-all-suites/v0-video4-action1) | v0 | 4/1 | 584,536,135 | Training only; evaluation not run |
+
+The policy completed ten epochs on the four-suite full dataset with global batch 128. Evaluation results are not available for this checkpoint.
+
 ## Checkpoint index
 
 | Directory | Model | Video/action loops | Parameters | Final evaluation |

checkpoint_index.json

@@ -364,7 +364,42 @@
       "source_revision": "3a89bfac5cfccad66bfebbe00014f7f739a78adc",
       "compatible_code_revision": "3a89bfac5cfccad66bfebbe00014f7f739a78adc",
       "normalization_sha256": "d5ad351eb3272f416d34082e8af5483ba1118ceca355a37d9f268fdb269fb3e4"
+    },
+    {
+      "id": "v0_full_4_1_recovery_job4823",
+      "repo_path": "libero-all-suites/v0-video4-action1",
+      "training_path": "/mnt/data/vmo-ai-task/anhdh35/FastWAM/runs/loopwam_nt/storage_recovery_20261010/job4823_concat/full_41_resume/train",
+      "source_sha256": "e8c7882ea5149ba7c45c0692b0e5f813af88953beb0780bffd6978795f7f96ef",
+      "source_bytes": 7015569623,
+      "export_sha256": "d460ccea7148876d96e7b6969c97f698fb02119e12ab1f749872faf3d30c3a92",
+      "export_bytes": 2338635000,
+      "format": "loopwam-s-v1",
+      "optimizer_included": false,
+      "step": 21700,
+      "version": "v0",
+      "video_loops": 4,
+      "action_loops": 1,
+      "action_kv_mode": "aligned",
+      "parameters": 584536135,
+      "dataset_scope": "full_libero",
+      "epochs": 10,
+      "global_batch": 128,
+      "training_seed": 42,
+      "evaluation_status": "not_run_training_only",
+      "success_rate": null,
+      "successes": null,
+      "episodes": 0,
+      "recovery_segment_training_seconds": 22217.201475716894,
+      "initialization_mode": "resume_checkpoint",
+      "resumed_from_step": 15190,
+      "resume_checkpoint_sha256": "51939c93a4e3df01ac19d6847e360d2771343808fafd17c8ecfc5c09d74fb378",
+      "tensor_equality_verified": true,
+      "source_revision": "3a89bfac5cfccad66bfebbe00014f7f739a78adc",
+      "compatible_code_revision": "3a89bfac5cfccad66bfebbe00014f7f739a78adc",
+      "normalization_sha256": "d5ad351eb3272f416d34082e8af5483ba1118ceca355a37d9f268fdb269fb3e4",
+      "fairness_check_passed": true,
+      "created_utc": "2026-10-10T11:52:21.118949+00:00"
     }
   ],
-  "updated_utc": "2026-10-10T02:41:05.842410+00:00"
+  "updated_utc": "2026-10-10T11:52:21.130095+00:00"
 }

libero-all-suites/v0-video4-action1/README.md

@@ -0,0 +1,7 @@
+# LoopWAM v0 4/1 — full LIBERO
+
+Completed ten-epoch full-suite training checkpoint (Spatial, Object, Goal and LIBERO-10), seed 42, global batch 128, DDP on two H100 GPUs. This is a training-only checkpoint; no post-training simulator evaluation has been run for this recovery checkpoint.
+
+`policy.pt` is the native LoopWAM checkpoint with optimizer state removed. Its model tensors were checked for exact equality with the completed training checkpoint. It supports inference or fresh-optimizer fine-tuning, but not exact optimizer-state resume. Keep the accompanying `dataset_stats.json` matched to this checkpoint.
+
+Training resumed from update 15,190 with the saved optimizer and RNG state; the recovery segment completed at update 21,700. The manifest, timing, data provenance, fairness check and source/export hashes are included. See `export.json`.

libero-all-suites/v0-video4-action1/dataset_stats.json

@@ -0,0 +1,50 @@
+{
+  "action": {
+    "default": {
+      "global_max": [
+        0.9375,
+        0.9375,
+        0.9375,
+        0.3557142913341522,
+        0.375,
+        0.375,
+        1.0
+      ],
+      "global_min": [
+        -0.9375,
+        -0.9375,
+        -0.9375,
+        -0.24214285612106323,
+        -0.375,
+        -0.3642857074737549,
+        0.0
+      ]
+    }
+  },
+  "num_episodes": 1712,
+  "num_transition": 277713,
+  "state": {
+    "default": {
+      "global_max": [
+        0.2103137969970703,
+        0.3904264271259308,
+        1.472778081893921,
+        3.7248642444610596,
+        3.5618896484375,
+        1.3863215446472168,
+        0.04232141748070717,
+        0.0013126095291227102
+      ],
+      "global_min": [
+        -0.48278069496154785,
+        -0.3309336006641388,
+        0.008128181099891663,
+        1.002794623374939,
+        -3.6312508583068848,
+        -1.842738389968872,
+        -0.005453015677630901,
+        -0.042015016078948975
+      ]
+    }
+  }
+}

libero-all-suites/v0-video4-action1/export.json

@@ -0,0 +1,35 @@
+{
+  "id": "v0_full_4_1_recovery_job4823",
+  "repo_path": "libero-all-suites/v0-video4-action1",
+  "training_path": "/mnt/data/vmo-ai-task/anhdh35/FastWAM/runs/loopwam_nt/storage_recovery_20261010/job4823_concat/full_41_resume/train",
+  "source_sha256": "e8c7882ea5149ba7c45c0692b0e5f813af88953beb0780bffd6978795f7f96ef",
+  "source_bytes": 7015569623,
+  "export_sha256": "d460ccea7148876d96e7b6969c97f698fb02119e12ab1f749872faf3d30c3a92",
+  "export_bytes": 2338635000,
+  "format": "loopwam-s-v1",
+  "optimizer_included": false,
+  "step": 21700,
+  "version": "v0",
+  "video_loops": 4,
+  "action_loops": 1,
+  "action_kv_mode": "aligned",
+  "parameters": 584536135,
+  "dataset_scope": "full_libero",
+  "epochs": 10,
+  "global_batch": 128,
+  "training_seed": 42,
+  "evaluation_status": "not_run_training_only",
+  "success_rate": null,
+  "successes": null,
+  "episodes": 0,
+  "recovery_segment_training_seconds": 22217.201475716894,
+  "initialization_mode": "resume_checkpoint",
+  "resumed_from_step": 15190,
+  "resume_checkpoint_sha256": "51939c93a4e3df01ac19d6847e360d2771343808fafd17c8ecfc5c09d74fb378",
+  "tensor_equality_verified": true,
+  "source_revision": "3a89bfac5cfccad66bfebbe00014f7f739a78adc",
+  "compatible_code_revision": "3a89bfac5cfccad66bfebbe00014f7f739a78adc",
+  "normalization_sha256": "d5ad351eb3272f416d34082e8af5483ba1118ceca355a37d9f268fdb269fb3e4",
+  "fairness_check_passed": true,
+  "created_utc": "2026-10-10T11:52:21.118949+00:00"
+}

libero-all-suites/v0-video4-action1/fairness.json

@@ -0,0 +1,49 @@
+{
+  "passed": true,
+  "compared_fields": [
+    "asset_sha256",
+    "seed",
+    "epochs",
+    "global_batch",
+    "train_windows",
+    "planned_updates",
+    "planned_windows",
+    "policy_dtype",
+    "optimizer_state_dtype",
+    "compute_dtype"
+  ],
+  "data_fields": [
+    "action_offsets",
+    "available_episodes",
+    "available_windows",
+    "camera_order",
+    "codebase_version",
+    "complete_500_demo_set",
+    "content_files",
+    "coverage",
+    "dataset_dir",
+    "dataset_scope",
+    "episode_metadata_sha256",
+    "fps",
+    "info_sha256",
+    "metadata_files",
+    "normalization_scope",
+    "normalization_sha256",
+    "normalization_source",
+    "padding_policy",
+    "seed",
+    "split",
+    "split_algorithm",
+    "suites",
+    "task_counts",
+    "text_cache_dir",
+    "text_cache_files",
+    "text_encoder_provenance",
+    "train_episodes",
+    "train_windows",
+    "validation_episodes",
+    "validation_windows",
+    "video_offsets"
+  ],
+  "accepted_initialization_change": true
+}

libero-all-suites/v0-video4-action1/policy.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d460ccea7148876d96e7b6969c97f698fb02119e12ab1f749872faf3d30c3a92
+size 2338635000

libero-all-suites/v0-video4-action1/training_timing.json

@@ -0,0 +1,53 @@
+{
+  "status": "complete",
+  "completed_updates": 21700,
+  "windows_seen": 2777130,
+  "planned_updates": 21700,
+  "elapsed_training_seconds": 22217.201475716894,
+  "checkpoint_seconds_total": 26.169514174107462,
+  "checkpoint_count": 3,
+  "measured_mean_update_seconds": 3.4053813899455303,
+  "estimated_total_training_hours": 20.52688226717167,
+  "estimate_note": "Projection from measured updates; excludes setup/checkpoint overhead",
+  "latest": {
+    "update": 21700,
+    "epoch": 10,
+    "windows": 81,
+    "windows_seen": 2777130,
+    "effective_passes": 10.0,
+    "seconds": 2.8533496856689453,
+    "lr": 1.0000000000000002e-06,
+    "grad_norm": 0.2194993644952774,
+    "branch_grad_after_clip": {
+      "video": 0.2073885202407837,
+      "action": 0.07118500024080276,
+      "proprio": 0.010132784955203533
+    },
+    "latent_cache_rank_local": {
+      "hits": 416571,
+      "misses": 0,
+      "dummy_encodes": 21,
+      "encoder_samples": 21,
+      "batches": 52074,
+      "read_seconds": 848.248464504024,
+      "encode_seconds": 0.9370445790700614,
+      "write_seconds": 0.0
+    },
+    "peak_allocated_gb": 58.344120025634766,
+    "peak_reserved_gb": 65.10608673095703,
+    "exit4/video_raw": 0.08217585528338398,
+    "exit4/action_raw": 0.02396311730514338,
+    "exit4/video": 0.06038003497653537,
+    "exit4/action": 0.01664120179635507,
+    "loss_video": 0.06038003497653537,
+    "loss_action": 0.01664120179635507,
+    "loop/1/video_state_rms": 1.414136345003858,
+    "loop/1/action_state_rms": 0.7373409035765095,
+    "loop/2/video_state_rms": 1.5300074447820216,
+    "loop/2/action_state_rms": 0.7373409035765095,
+    "loop/3/video_state_rms": 1.647120346257716,
+    "loop/3/action_state_rms": 0.7373409035765095,
+    "loop/4/video_state_rms": 1.5038441787531347,
+    "loop/4/action_state_rms": 0.9207580943166473
+  }
+}