Add verified final Long concat and mix policies with three-seed evaluation

README.md

@@ -9,7 +9,28 @@ tags:
 
 # LoopWAM final checkpoints
 
-Final trained policy weights from the LoopWAM experiments. This repository contains **11 checkpoints**, separated by training data scope. Both original and repeated 4/4 runs are retained. All production checkpoints are from completed ten-epoch runs with global batch 128 and training seed 42.
+Final trained policy weights from the LoopWAM experiments. This repository contains **13 checkpoints**, separated by training data scope. Both original and repeated 4/4 runs are retained. All production checkpoints are from completed ten-epoch runs with global batch 128 and training seed 42.
+
+## October 9 additions: all-loop KV
+
+| Directory | KV mode | Seed42/43/44 | Pooled SR |
+|---|---|---|---:|
+| [libero-long/v0-video4-action1-kv-concat](libero-long/v0-video4-action1-kv-concat) | concat | 83/86/86 | 255/300 = 85.00% |
+| [libero-long/v0-video4-action1-kv-mix](libero-long/v0-video4-action1-kv-mix) | mix | 90/89/87 | 266/300 = 88.67% |
+
+**These two additions require the [KV_concat code branch](https://github.com/anhdao69/LoopWAM_NT/tree/KV_concat), revision3a89bfac5cfccad66bfebbe00014f7f739a78adc or compatible later.**
+Their checkpoints record action_kv_mode; the original LoopWAM_NT code predates
+these modes. Use eager BF16 inference, as native Inductor failed numerical
+equivalence. Each directory includes normalization, training provenance,
+three evaluation summaries and a tensor-verified optimizer-free native policy.
+The Long aligned4/1 and both original/repeat4/4 weights were hash-checked and
+are unchanged. Their additional evaluation seeds do not create new weights.
+The new full-suite4/1 training is ongoing; no intermediate checkpoint is uploaded.
+
+Expanded seed42: concat404/500 versus aligned413/500 (paired p=0.4814). The
+original three-seed decision remains inconclusive. This grid overlaps the
+original states and must not be pooled as independent evidence.
+See the [updated results report](https://github.com/anhdao69/LoopWAM_NT/blob/KV_concat/plans/performance/LoopWAM_v0_V4A1_KVconcat_20261008.md).
 
 ## Checkpoint index
 
@@ -31,7 +52,7 @@ Final trained policy weights from the LoopWAM experiments. This repository conta
 
 `libero-all-suites/`: one v0 trained on all 1,712 locally available Spatial/Object/Goal/Long demonstrations, 21,700 updates. Its 388/400 result aggregates all four suites, so it is a different training/evaluation scope from the Long-only rows. The same four-suite checkpoint also scored 720/2,000 (36%) on LIBERO-Pro.
 
-All Long rows use 100 final-checkpoint rollouts at evaluation seed 42. Standard-suite evaluations use 700 policy steps, 30 settling steps, 32-action predictions, replanning every ten actions, ten denoising steps and CFG 1. The original 4/4 checkpoint additionally scored 96%, 88%, 91% across evaluation seeds 42/43/44. These are one training seed and cannot establish across-training-seed significance.
+The original checkpoint-index rows below use 100 final-checkpoint rollouts at evaluation seed 42; the new KV table above reports three seeds. Standard-suite evaluations use 700 policy steps, 30 settling steps, 32-action predictions, replanning every ten actions, ten denoising steps and CFG 1. The original 4/4 checkpoint additionally scored 96%, 88%, 91% across evaluation seeds 42/43/44. These are one training seed and cannot establish across-training-seed significance.
 
 ## Files and integrity
 
@@ -41,7 +62,7 @@ The frozen VAE and text encoder/embeddings are external inference dependencies.
 
 ## Loading
 
-Use the compatible [LoopWAM code](https://github.com/anhdao69/LoopWAM_NT/tree/LoopWAM_NT), at revision `8a29ffdce1537409e2b7e975e838d990ebbe068a` or a compatible later revision. This is a custom policy, not a Transformers AutoModel checkpoint. Install that repository’s inference dependencies first. Authenticate with a Hugging Face account authorized to access this private repository.
+Use the compatible [LoopWAM code](https://github.com/anhdao69/LoopWAM_NT/tree/LoopWAM_NT), at revision `8a29ffdce1537409e2b7e975e838d990ebbe068a` or a compatible later revision. This is a custom policy, not a Transformers AutoModel checkpoint. Install that repository’s inference dependencies first. The repository is publicly readable.
 
 ```python
 import torch
@@ -61,6 +82,6 @@ Depth/version are reconstructed from checkpoint metadata. Use the repository’s
 
 ## Provenance and rights
 
-These research checkpoints derive from pretrained Wan and the FastWAM/LoopWAM implementation. Consult the upstream model and code licenses before redistribution or use; this model card does not grant additional upstream rights. The repository is private.
+These research checkpoints derive from pretrained Wan and the FastWAM/LoopWAM implementation. Consult the upstream model and code licenses before redistribution or use; this model card does not grant additional upstream rights. The repository is currently public.
 
 For detailed comparison caveats, runtime accounting and per-task results, see the [consolidated report](https://github.com/anhdao69/LoopWAM_NT/blob/LoopWAM_NT/plans/performance/LoopWAM_Consolidated_Results_20261008.md). That report’s earlier snapshot labels the repeat pending; the repeat is now complete at 91/100, as recorded here.

checkpoint_index.json

@@ -265,6 +265,77 @@
       "tensor_equality_verified": true,
       "source_revision": "8a29ffdce1537409e2b7e975e838d990ebbe068a",
       "normalization_sha256": "78058a689c41382a2759541d5590cb3ada8c576e736285aaeebc344874366f1b"
+    },
+    {
+      "id": "v0_4_1_kv_concat",
+      "repo_path": "libero-long/v0-video4-action1-kv-concat",
+      "training_path": "kv_campaign_job4770/campaign_3a89bfa/concat_long/train",
+      "evaluation_path": "kv_campaign_job4770/campaign_3a89bfa/concat_long/eval_seed42",
+      "source_sha256": "c10ce1a2fdb04763d33a0d6d9630b7401a689e36bf4fc53bed8494584377b10a",
+      "source_bytes": 7015569431,
+      "export_sha256": "641ef91940e85f6ed22a92747ea86adc7d201d99c082556f59e6a188c6eaf5c5",
+      "export_bytes": 2338634808,
+      "format": "loopwam-s-v1",
+      "optimizer_included": false,
+      "step": 7250,
+      "version": "v0",
+      "video_loops": 4,
+      "action_loops": 1,
+      "action_kv_mode": "concat",
+      "parameters": 584536135,
+      "success_rate": 0.83,
+      "successes": 83,
+      "episodes": 100,
+      "evaluation_seed": 42,
+      "pooled_successes": 255,
+      "pooled_episodes": 300,
+      "pooled_success_rate": 0.85,
+      "evaluation_seeds": [
+        42,
+        43,
+        44
+      ],
+      "training_seconds": 27258.21305021108,
+      "tensor_equality_verified": true,
+      "source_revision": "3a89bfac5cfccad66bfebbe00014f7f739a78adc",
+      "compatible_code_revision": "3a89bfac5cfccad66bfebbe00014f7f739a78adc",
+      "normalization_sha256": "78058a689c41382a2759541d5590cb3ada8c576e736285aaeebc344874366f1b"
+    },
+    {
+      "id": "v0_4_1_kv_mix",
+      "repo_path": "libero-long/v0-video4-action1-kv-mix",
+      "training_path": "kv_campaign_job4771/campaign_3a89bfa/mix_long/train",
+      "evaluation_path": "kv_campaign_job4771/campaign_3a89bfa/mix_long/eval_seed42",
+      "source_sha256": "1294bf07bd84c868215f9baff8e64301a2b7827d278a2a7183e7af86bea26ff6",
+      "source_bytes": 7015574271,
+      "export_sha256": "b90be503a3ee7c39305b2a6bbc94a0e649d566c5d4705644eba5476058c06340",
+      "export_bytes": 2338636277,
+      "format": "loopwam-s-v1",
+      "optimizer_included": false,
+      "step": 7250,
+      "version": "v0",
+      "video_loops": 4,
+      "action_loops": 1,
+      "action_kv_mode": "mix",
+      "parameters": 584536423,
+      "success_rate": 0.9,
+      "successes": 90,
+      "episodes": 100,
+      "evaluation_seed": 42,
+      "pooled_successes": 266,
+      "pooled_episodes": 300,
+      "pooled_success_rate": 0.8866666666666667,
+      "evaluation_seeds": [
+        42,
+        43,
+        44
+      ],
+      "training_seconds": 26302.35895577073,
+      "tensor_equality_verified": true,
+      "source_revision": "3a89bfac5cfccad66bfebbe00014f7f739a78adc",
+      "compatible_code_revision": "3a89bfac5cfccad66bfebbe00014f7f739a78adc",
+      "normalization_sha256": "78058a689c41382a2759541d5590cb3ada8c576e736285aaeebc344874366f1b"
     }
-  ]
-}
+  ],
+  "updated_utc": "2026-10-09T16:24:40.158405+00:00"
+}

libero-long/v0-video4-action1-kv-concat/README.md

@@ -0,0 +1,22 @@
+# LoopWAM v0 4/1 KV concat
+
+Final LIBERO-Long 344/44-split policy: ten epochs, 7,250 updates, seed42,
+global batch128. Video/action depth30/12. Three evaluation seeds42/43/44:
+83/100, 86/100, 86/100; pooled 255/300.
+
+Use the KV_concat branch of https://github.com/anhdao69/LoopWAM_NT at
+3a89bfac5cfccad66bfebbe00014f7f739a78adc or a compatible later revision.
+The original LoopWAM_NT branch predates these KV modes. Load policy.pt with
+create_loopwam(checkpoint_path=..., vae_path=..., model_dtype=torch.float32).
+The checkpoint reconstructs its KV mode. Use BF16 autocast and eager inference;
+native BF16 Inductor did not pass numerical equivalence. The frozen Wan VAE,
+text embeddings and observation adapter remain external dependencies.
+
+All model tensors and metadata were verified against the evaluated final native
+checkpoint. Only Adam optimizer state was removed. This export supports inference
+or fresh-optimizer fine-tuning, not exact optimizer-state resume. Keep the supplied
+dataset_stats.json normalization with these weights. Source and export hashes
+differ because serialization changes; export.json records both.
+
+One training seed only; evaluation-seed variation is not training-seed variation.
+See the [full report](https://github.com/anhdao69/LoopWAM_NT/blob/KV_concat/plans/performance/LoopWAM_v0_V4A1_KVconcat_20261008.md).

libero-long/v0-video4-action1-kv-concat/data_manifest.json

@@ -0,0 +1,597 @@
+{
+  "action_offsets": [
+    0,
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    10,
+    11,
+    12,
+    13,
+    14,
+    15,
+    16,
+    17,
+    18,
+    19,
+    20,
+    21,
+    22,
+    23,
+    24,
+    25,
+    26,
+    27,
+    28,
+    29,
+    30,
+    31
+  ],
+  "camera_order": [
+    "observation.images.image",
+    "observation.images.wrist_image"
+  ],
+  "codebase_version": "v3.0",
+  "complete_500_demo_set": false,
+  "content_files": {
+    "data/chunk-000/file-000.parquet": {
+      "bytes": 13596266,
+      "sha256": "5862ce014b9d7ac1463212f4656fb12af28d2528f9a06fe648c372bdd0ccf241"
+    },
+    "videos/observation.images.image/chunk-000/file-000.mp4": {
+      "bytes": 207852961,
+      "sha256": "b80adadca8e851ed7ab5c8bccc2730d27e671de66aad44459851be762b4576bf"
+    },
+    "videos/observation.images.image/chunk-000/file-001.mp4": {
+      "bytes": 209145634,
+      "sha256": "cd53ba965cda8875f7af1c98dd27d019602992f2216d1b9c7598cc1018755a14"
+    },
+    "videos/observation.images.image/chunk-000/file-002.mp4": {
+      "bytes": 209394215,
+      "sha256": "599fec3ddec19614835b24533613eb00260b728e24280736cace688c05dc8586"
+    },
+    "videos/observation.images.image/chunk-000/file-003.mp4": {
+      "bytes": 208582582,
+      "sha256": "3203f9bf42f74956d264c18ef1a28076f293c673dcdf4cd57a13aa2fc29dfc54"
+    },
+    "videos/observation.images.image/chunk-000/file-004.mp4": {
+      "bytes": 109477730,
+      "sha256": "ac26e458afc9ea239fd1f33e062e94978ecc7885aacfcdfb1e95f0080b99cc98"
+    },
+    "videos/observation.images.wrist_image/chunk-000/file-000.mp4": {
+      "bytes": 209197605,
+      "sha256": "9b540eb674a7078314da86e1b4d0bb07181872773ff793541efcec08dcd1ac25"
+    },
+    "videos/observation.images.wrist_image/chunk-000/file-001.mp4": {
+      "bytes": 208255144,
+      "sha256": "ab8cc4f365139465eb975860d67771c286d060ef1fa066ab116c466fb2d0679c"
+    },
+    "videos/observation.images.wrist_image/chunk-000/file-002.mp4": {
+      "bytes": 163891936,
+      "sha256": "235329e56065c3a375b62e643ea164edd3845c3d78bac75745c33c4d5394ee27"
+    }
+  },
+  "dataset_dir": "/mnt/data/vmo-ai-task/anhdh35/FastWAM/data/lerobot_v30/libero_10_no_noops_lerobot",
+  "episode_metadata_sha256": "7ba93847b6b100a57dda14a2afe38cfcf6e0d4881bb2913379ba4f04919faa01",
+  "fps": 20,
+  "info_sha256": "465dcfee2a1fee7ae517089bdf35a974d4855b3c7e49a9b438448872d9f78805",
+  "normalization_path": "/mnt/data/vmo-ai-task/anhdh35/FastWAM/runs/loopwam_nt/kv_campaign_job4770/campaign_3a89bfa/concat_long/train/data/dataset_stats.json",
+  "normalization_sha256": "78058a689c41382a2759541d5590cb3ada8c576e736285aaeebc344874366f1b",
+  "normalization_source": "training episodes only",
+  "padding_policy": "all anchors, repeat last frame, zero padded delta pose, retain gripper",
+  "seed": 42,
+  "split_algorithm": "SHA256(seed:task:episode_id) sorted, floor(0.9*n) train",
+  "task_counts": {
+    "pick up the book and place it in the back compartment of the caddy": {
+      "available": 41,
+      "train": 36,
+      "validation": 5
+    },
+    "put both moka pots on the stove": {
+      "available": 33,
+      "train": 29,
+      "validation": 4
+    },
+    "put both the alphabet soup and the cream cheese box in the basket": {
+      "available": 45,
+      "train": 40,
+      "validation": 5
+    },
+    "put both the alphabet soup and the tomato sauce in the basket": {
+      "available": 38,
+      "train": 34,
+      "validation": 4
+    },
+    "put both the cream cheese box and the butter in the basket": {
+      "available": 46,
+      "train": 41,
+      "validation": 5
+    },
+    "put the black bowl in the bottom drawer of the cabinet and close it": {
+      "available": 35,
+      "train": 31,
+      "validation": 4
+    },
+    "put the white mug on the left plate and put the yellow and white mug on the right plate": {
+      "available": 37,
+      "train": 33,
+      "validation": 4
+    },
+    "put the white mug on the plate and put the chocolate pudding to the right of the plate": {
+      "available": 38,
+      "train": 34,
+      "validation": 4
+    },
+    "put the yellow and white mug in the microwave and close it": {
+      "available": 32,
+      "train": 28,
+      "validation": 4
+    },
+    "turn on the stove and put the moka pot on it": {
+      "available": 43,
+      "train": 38,
+      "validation": 5
+    }
+  },
+  "text_cache_dir": "/mnt/data/vmo-ai-task/anhdh35/FastWAM/data/text_embeds_cache/libero",
+  "text_cache_files": {
+    "4f93dc39fd7c885bf14d50d7db40823defd127f7df28071700e2791d3bf22dd5.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "ac4789cb8d5174143e63a60cdfc8c92f433144eb16cb488adf83dae43eb94e08",
+      "valid_tokens": 32
+    },
+    "517025c832018e9c689a3982853b7745b52cadd7a6cfb10e3f6476844aabedbe.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "35aea461efc1c8131c1e1fc1f3496f6e8e14595050bd7c930318b77af9114348",
+      "valid_tokens": 37
+    },
+    "53bde11d507045010a731dc81bb02e80859e58b9315a8b303b5d0fd6f9b561ef.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "cc8d23a89fc3b424be69d78163a81a9989aa198cfde95041f70824b3e39e0ac5",
+      "valid_tokens": 31
+    },
+    "84727b22433531438a399799d28d16d49fc56ec7ab0a4cf6a827ea7b2347df35.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "4df8ae0c6ca54e324ca0ea4a1499c60ba4519aadcc61f600b46111e62888c5ac",
+      "valid_tokens": 30
+    },
+    "88251b67639631ce11c41f703e85001fff4b505e1eafcc355389e15e15da67ee.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "5209b623f57f19f8c447ff6f0cb820fab501337c088b6ec446893f9d2f2dbda4",
+      "valid_tokens": 30
+    },
+    "ab4a7b38cafc00cd4f08f4a3c1937ae369dc1b6d59f77f7554999d2720177ce1.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "4251d4df987e99cbf26fc3b4630b9aaddc0550d26cd6929c2c17d96290043edb",
+      "valid_tokens": 30
+    },
+    "b75715591c78b61ee4e9abf6a84eda0e63b04bd38514a0ebd207770d415357dc.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "2c613c77243948c4f67228df12fae938079b5e1f7a200e3432f94822921d0dac",
+      "valid_tokens": 34
+    },
+    "d5d6f37fc1c8d15053ce2a356e5e09ef36977a0dfe672f9393dcc6f5bd597352.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "a7601429d574a3d22755cc1d103d1e0bab8994405de55576d141e4e220b41f58",
+      "valid_tokens": 26
+    },
+    "fd92daee5b46a4b3dd8426ff9cceea75d709cee1c7b19244b1026205cc33ba3c.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "7f25b652522a1302a42eaa01cfbf5ee69fdecb20b63914029be1312e969c087e",
+      "valid_tokens": 37
+    },
+    "fe5b483ce54fd0b450d3504eacf0f05018a50ab184fa53c7e2744b8a52e957cf.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "bc2bc87b57d5b6da0a144597530a44af8d41e96cd0b03b4c8a83bc13e1402fec",
+      "valid_tokens": 33
+    }
+  },
+  "text_encoder_provenance": {
+    "cache_payload_binds_encoder_hash": false,
+    "expected_encoder": "models_t5_umt5-xxl-enc-bf16.pth",
+    "expected_encoder_sha256": "7cace0da2b446bbbbc57d031ab6cf163a3d59b366da94e5afe36745b746fd81d",
+    "limitation": "Existing context/mask-only caches have no embedded encoder or tokenizer provenance",
+    "source": "Identical official Wan2.1-T2V-1.3B and Wan2.2-TI2V-5B Hub LFS hashes, audited 2026-10-05"
+  },
+  "train_episodes": [
+    0,
+    1,
+    2,
+    3,
+    4,
+    6,
+    7,
+    8,
+    10,
+    11,
+    12,
+    13,
+    14,
+    15,
+    16,
+    17,
+    18,
+    19,
+    21,
+    22,
+    23,
+    24,
+    25,
+    26,
+    27,
+    28,
+    29,
+    30,
+    31,
+    32,
+    35,
+    36,
+    37,
+    38,
+    39,
+    40,
+    41,
+    42,
+    43,
+    45,
+    47,
+    48,
+    49,
+    50,
+    51,
+    52,
+    53,
+    55,
+    56,
+    57,
+    58,
+    59,
+    60,
+    61,
+    62,
+    63,
+    64,
+    65,
+    66,
+    67,
+    68,
+    69,
+    71,
+    72,
+    73,
+    74,
+    75,
+    76,
+    77,
+    78,
+    79,
+    80,
+    81,
+    82,
+    83,
+    85,
+    87,
+    88,
+    89,
+    90,
+    91,
+    92,
+    93,
+    94,
+    95,
+    97,
+    98,
+    99,
+    100,
+    101,
+    102,
+    103,
+    104,
+    105,
+    106,
+    107,
+    109,
+    110,
+    111,
+    112,
+    113,
+    114,
+    115,
+    116,
+    117,
+    118,
+    120,
+    121,
+    122,
+    123,
+    125,
+    126,
+    127,
+    128,
+    129,
+    130,
+    131,
+    132,
+    133,
+    135,
+    136,
+    137,
+    138,
+    139,
+    141,
+    142,
+    143,
+    144,
+    145,
+    146,
+    147,
+    148,
+    149,
+    150,
+    151,
+    152,
+    153,
+    155,
+    157,
+    159,
+    160,
+    161,
+    162,
+    163,
+    164,
+    165,
+    166,
+    167,
+    168,
+    169,
+    170,
+    171,
+    173,
+    174,
+    175,
+    176,
+    177,
+    178,
+    179,
+    180,
+    181,
+    182,
+    183,
+    185,
+    186,
+    187,
+    188,
+    189,
+    190,
+    191,
+    192,
+    193,
+    194,
+    195,
+    197,
+    198,
+    199,
+    200,
+    202,
+    203,
+    204,
+    205,
+    207,
+    208,
+    209,
+    210,
+    211,
+    212,
+    213,
+    215,
+    216,
+    217,
+    218,
+    219,
+    220,
+    221,
+    222,
+    223,
+    224,
+    225,
+    227,
+    228,
+    230,
+    231,
+    232,
+    233,
+    234,
+    236,
+    237,
+    238,
+    239,
+    240,
+    241,
+    242,
+    243,
+    244,
+    245,
+    246,
+    247,
+    248,
+    249,
+    250,
+    251,
+    252,
+    253,
+    254,
+    255,
+    256,
+    257,
+    258,
+    259,
+    260,
+    261,
+    263,
+    264,
+    266,
+    267,
+    268,
+    269,
+    270,
+    271,
+    273,
+    274,
+    276,
+    277,
+    278,
+    279,
+    280,
+    281,
+    282,
+    283,
+    284,
+    285,
+    286,
+    287,
+    288,
+    290,
+    291,
+    292,
+    294,
+    295,
+    296,
+    297,
+    298,
+    299,
+    300,
+    301,
+    302,
+    303,
+    304,
+    305,
+    306,
+    307,
+    308,
+    310,
+    311,
+    312,
+    313,
+    314,
+    315,
+    316,
+    317,
+    318,
+    319,
+    320,
+    321,
+    322,
+    323,
+    324,
+    325,
+    327,
+    328,
+    329,
+    330,
+    331,
+    332,
+    333,
+    334,
+    335,
+    337,
+    339,
+    340,
+    341,
+    342,
+    343,
+    344,
+    345,
+    346,
+    347,
+    348,
+    349,
+    350,
+    351,
+    352,
+    353,
+    354,
+    355,
+    356,
+    357,
+    358,
+    359,
+    360,
+    361,
+    362,
+    363,
+    364,
+    365,
+    367,
+    368,
+    369,
+    370,
+    373,
+    374,
+    375,
+    377,
+    378,
+    379,
+    380,
+    381,
+    383,
+    384,
+    385,
+    386,
+    387
+  ],
+  "train_windows": 92678,
+  "validation_episodes": [
+    5,
+    9,
+    20,
+    33,
+    34,
+    44,
+    46,
+    54,
+    70,
+    84,
+    86,
+    96,
+    108,
+    119,
+    124,
+    134,
+    140,
+    154,
+    156,
+    158,
+    172,
+    184,
+    196,
+    201,
+    206,
+    214,
+    226,
+    229,
+    235,
+    262,
+    265,
+    272,
+    275,
+    289,
+    293,
+    309,
+    326,
+    336,
+    338,
+    366,
+    371,
+    372,
+    376,
+    382
+  ],
+  "validation_windows": 11602,
+  "video_offsets": [
+    0,
+    4,
+    8,
+    12,
+    16,
+    20,
+    24,
+    28,
+    32
+  ]
+}

libero-long/v0-video4-action1-kv-concat/dataset_stats.json

@@ -0,0 +1,50 @@
+{
+  "action": {
+    "default": {
+      "global_max": [
+        0.9375,
+        0.9375,
+        0.9375,
+        0.32892856001853943,
+        0.36964285373687744,
+        0.375,
+        1.0
+      ],
+      "global_min": [
+        -0.9375,
+        -0.9375,
+        -0.9375,
+        -0.23642857372760773,
+        -0.3053571283817291,
+        -0.3642857074737549,
+        0.0
+      ]
+    }
+  },
+  "num_episodes": 344,
+  "num_transition": 92678,
+  "state": {
+    "default": {
+      "global_max": [
+        0.2103137969970703,
+        0.38887521624565125,
+        1.333192229270935,
+        3.7248642444610596,
+        3.5618896484375,
+        1.3863215446472168,
+        0.041575800627470016,
+        0.0013126095291227102
+      ],
+      "global_min": [
+        -0.48278069496154785,
+        -0.3309336006641388,
+        0.4457588791847229,
+        1.1323540210723877,
+        -3.6312508583068848,
+        -1.842738389968872,
+        -0.005453015677630901,
+        -0.04112039878964424
+      ]
+    }
+  }
+}

libero-long/v0-video4-action1-kv-concat/export.json

@@ -0,0 +1,35 @@
+{
+  "id": "v0_4_1_kv_concat",
+  "repo_path": "libero-long/v0-video4-action1-kv-concat",
+  "training_path": "kv_campaign_job4770/campaign_3a89bfa/concat_long/train",
+  "evaluation_path": "kv_campaign_job4770/campaign_3a89bfa/concat_long/eval_seed42",
+  "source_sha256": "c10ce1a2fdb04763d33a0d6d9630b7401a689e36bf4fc53bed8494584377b10a",
+  "source_bytes": 7015569431,
+  "export_sha256": "641ef91940e85f6ed22a92747ea86adc7d201d99c082556f59e6a188c6eaf5c5",
+  "export_bytes": 2338634808,
+  "format": "loopwam-s-v1",
+  "optimizer_included": false,
+  "step": 7250,
+  "version": "v0",
+  "video_loops": 4,
+  "action_loops": 1,
+  "action_kv_mode": "concat",
+  "parameters": 584536135,
+  "success_rate": 0.83,
+  "successes": 83,
+  "episodes": 100,
+  "evaluation_seed": 42,
+  "pooled_successes": 255,
+  "pooled_episodes": 300,
+  "pooled_success_rate": 0.85,
+  "evaluation_seeds": [
+    42,
+    43,
+    44
+  ],
+  "training_seconds": 27258.21305021108,
+  "tensor_equality_verified": true,
+  "source_revision": "3a89bfac5cfccad66bfebbe00014f7f739a78adc",
+  "compatible_code_revision": "3a89bfac5cfccad66bfebbe00014f7f739a78adc",
+  "normalization_sha256": "78058a689c41382a2759541d5590cb3ada8c576e736285aaeebc344874366f1b"
+}

libero-long/v0-video4-action1-kv-concat/policy.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:641ef91940e85f6ed22a92747ea86adc7d201d99c082556f59e6a188c6eaf5c5
+size 2338634808

libero-long/v0-video4-action1-kv-concat/training_timing.json

@@ -0,0 +1,53 @@
+{
+  "status": "complete",
+  "completed_updates": 7250,
+  "windows_seen": 926780,
+  "planned_updates": 7250,
+  "elapsed_training_seconds": 27258.21305021108,
+  "checkpoint_seconds_total": 98.40984348789789,
+  "checkpoint_count": 10,
+  "measured_mean_update_seconds": 3.7376492462794295,
+  "estimated_total_training_hours": 7.527210287646073,
+  "estimate_note": "Projection from measured updates; excludes setup/checkpoint overhead",
+  "latest": {
+    "update": 7250,
+    "epoch": 10,
+    "windows": 6,
+    "windows_seen": 926780,
+    "effective_passes": 10.0,
+    "seconds": 0.8128615617752075,
+    "lr": 1.0000000000000002e-06,
+    "grad_norm": 0.4529801905155182,
+    "branch_grad_after_clip": {
+      "video": 0.3884684443473816,
+      "action": 0.23184558749198914,
+      "proprio": 0.02304273098707199
+    },
+    "latent_cache_rank_local": {
+      "hits": 463420,
+      "misses": 0,
+      "dummy_encodes": 20,
+      "encoder_samples": 20,
+      "batches": 57930,
+      "read_seconds": 1537.3295891773887,
+      "encode_seconds": 0.9740421026945114,
+      "write_seconds": 0.0
+    },
+    "peak_allocated_gb": 60.09296417236328,
+    "peak_reserved_gb": 60.964210510253906,
+    "exit4/video_raw": 0.04362402359644572,
+    "exit4/action_raw": 0.026146019498507183,
+    "exit4/video": 0.05109126369158427,
+    "exit4/action": 0.017173873881498974,
+    "loss_video": 0.05109126369158427,
+    "loss_action": 0.017173873881498974,
+    "loop/1/video_state_rms": 2.6954692204793296,
+    "loop/2/video_state_rms": 2.8093770345052085,
+    "loop/3/video_state_rms": 2.989253044128418,
+    "loop/4/video_state_rms": 2.7547988891601562,
+    "loop/1/action_state_rms": 1.5263166427612305,
+    "loop/2/action_state_rms": 1.5263166427612305,
+    "loop/3/action_state_rms": 1.5263166427612305,
+    "loop/4/action_state_rms": 1.8486798604329426
+  }
+}

libero-long/v0-video4-action1-kv-mix/README.md

@@ -0,0 +1,22 @@
+# LoopWAM v0 4/1 KV mix
+
+Final LIBERO-Long 344/44-split policy: ten epochs, 7,250 updates, seed42,
+global batch128. Video/action depth30/12. Three evaluation seeds42/43/44:
+90/100, 89/100, 87/100; pooled 266/300.
+
+Use the KV_concat branch of https://github.com/anhdao69/LoopWAM_NT at
+3a89bfac5cfccad66bfebbe00014f7f739a78adc or a compatible later revision.
+The original LoopWAM_NT branch predates these KV modes. Load policy.pt with
+create_loopwam(checkpoint_path=..., vae_path=..., model_dtype=torch.float32).
+The checkpoint reconstructs its KV mode. Use BF16 autocast and eager inference;
+native BF16 Inductor did not pass numerical equivalence. The frozen Wan VAE,
+text embeddings and observation adapter remain external dependencies.
+
+All model tensors and metadata were verified against the evaluated final native
+checkpoint. Only Adam optimizer state was removed. This export supports inference
+or fresh-optimizer fine-tuning, not exact optimizer-state resume. Keep the supplied
+dataset_stats.json normalization with these weights. Source and export hashes
+differ because serialization changes; export.json records both.
+
+One training seed only; evaluation-seed variation is not training-seed variation.
+See the [full report](https://github.com/anhdao69/LoopWAM_NT/blob/KV_concat/plans/performance/LoopWAM_v0_V4A1_KVconcat_20261008.md).

libero-long/v0-video4-action1-kv-mix/data_manifest.json

@@ -0,0 +1,597 @@
+{
+  "action_offsets": [
+    0,
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    10,
+    11,
+    12,
+    13,
+    14,
+    15,
+    16,
+    17,
+    18,
+    19,
+    20,
+    21,
+    22,
+    23,
+    24,
+    25,
+    26,
+    27,
+    28,
+    29,
+    30,
+    31
+  ],
+  "camera_order": [
+    "observation.images.image",
+    "observation.images.wrist_image"
+  ],
+  "codebase_version": "v3.0",
+  "complete_500_demo_set": false,
+  "content_files": {
+    "data/chunk-000/file-000.parquet": {
+      "bytes": 13596266,
+      "sha256": "5862ce014b9d7ac1463212f4656fb12af28d2528f9a06fe648c372bdd0ccf241"
+    },
+    "videos/observation.images.image/chunk-000/file-000.mp4": {
+      "bytes": 207852961,
+      "sha256": "b80adadca8e851ed7ab5c8bccc2730d27e671de66aad44459851be762b4576bf"
+    },
+    "videos/observation.images.image/chunk-000/file-001.mp4": {
+      "bytes": 209145634,
+      "sha256": "cd53ba965cda8875f7af1c98dd27d019602992f2216d1b9c7598cc1018755a14"
+    },
+    "videos/observation.images.image/chunk-000/file-002.mp4": {
+      "bytes": 209394215,
+      "sha256": "599fec3ddec19614835b24533613eb00260b728e24280736cace688c05dc8586"
+    },
+    "videos/observation.images.image/chunk-000/file-003.mp4": {
+      "bytes": 208582582,
+      "sha256": "3203f9bf42f74956d264c18ef1a28076f293c673dcdf4cd57a13aa2fc29dfc54"
+    },
+    "videos/observation.images.image/chunk-000/file-004.mp4": {
+      "bytes": 109477730,
+      "sha256": "ac26e458afc9ea239fd1f33e062e94978ecc7885aacfcdfb1e95f0080b99cc98"
+    },
+    "videos/observation.images.wrist_image/chunk-000/file-000.mp4": {
+      "bytes": 209197605,
+      "sha256": "9b540eb674a7078314da86e1b4d0bb07181872773ff793541efcec08dcd1ac25"
+    },
+    "videos/observation.images.wrist_image/chunk-000/file-001.mp4": {
+      "bytes": 208255144,
+      "sha256": "ab8cc4f365139465eb975860d67771c286d060ef1fa066ab116c466fb2d0679c"
+    },
+    "videos/observation.images.wrist_image/chunk-000/file-002.mp4": {
+      "bytes": 163891936,
+      "sha256": "235329e56065c3a375b62e643ea164edd3845c3d78bac75745c33c4d5394ee27"
+    }
+  },
+  "dataset_dir": "/mnt/data/vmo-ai-task/anhdh35/FastWAM/data/lerobot_v30/libero_10_no_noops_lerobot",
+  "episode_metadata_sha256": "7ba93847b6b100a57dda14a2afe38cfcf6e0d4881bb2913379ba4f04919faa01",
+  "fps": 20,
+  "info_sha256": "465dcfee2a1fee7ae517089bdf35a974d4855b3c7e49a9b438448872d9f78805",
+  "normalization_path": "/mnt/data/vmo-ai-task/anhdh35/FastWAM/runs/loopwam_nt/kv_campaign_job4771/campaign_3a89bfa/mix_long/train/data/dataset_stats.json",
+  "normalization_sha256": "78058a689c41382a2759541d5590cb3ada8c576e736285aaeebc344874366f1b",
+  "normalization_source": "training episodes only",
+  "padding_policy": "all anchors, repeat last frame, zero padded delta pose, retain gripper",
+  "seed": 42,
+  "split_algorithm": "SHA256(seed:task:episode_id) sorted, floor(0.9*n) train",
+  "task_counts": {
+    "pick up the book and place it in the back compartment of the caddy": {
+      "available": 41,
+      "train": 36,
+      "validation": 5
+    },
+    "put both moka pots on the stove": {
+      "available": 33,
+      "train": 29,
+      "validation": 4
+    },
+    "put both the alphabet soup and the cream cheese box in the basket": {
+      "available": 45,
+      "train": 40,
+      "validation": 5
+    },
+    "put both the alphabet soup and the tomato sauce in the basket": {
+      "available": 38,
+      "train": 34,
+      "validation": 4
+    },
+    "put both the cream cheese box and the butter in the basket": {
+      "available": 46,
+      "train": 41,
+      "validation": 5
+    },
+    "put the black bowl in the bottom drawer of the cabinet and close it": {
+      "available": 35,
+      "train": 31,
+      "validation": 4
+    },
+    "put the white mug on the left plate and put the yellow and white mug on the right plate": {
+      "available": 37,
+      "train": 33,
+      "validation": 4
+    },
+    "put the white mug on the plate and put the chocolate pudding to the right of the plate": {
+      "available": 38,
+      "train": 34,
+      "validation": 4
+    },
+    "put the yellow and white mug in the microwave and close it": {
+      "available": 32,
+      "train": 28,
+      "validation": 4
+    },
+    "turn on the stove and put the moka pot on it": {
+      "available": 43,
+      "train": 38,
+      "validation": 5
+    }
+  },
+  "text_cache_dir": "/mnt/data/vmo-ai-task/anhdh35/FastWAM/data/text_embeds_cache/libero",
+  "text_cache_files": {
+    "4f93dc39fd7c885bf14d50d7db40823defd127f7df28071700e2791d3bf22dd5.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "ac4789cb8d5174143e63a60cdfc8c92f433144eb16cb488adf83dae43eb94e08",
+      "valid_tokens": 32
+    },
+    "517025c832018e9c689a3982853b7745b52cadd7a6cfb10e3f6476844aabedbe.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "35aea461efc1c8131c1e1fc1f3496f6e8e14595050bd7c930318b77af9114348",
+      "valid_tokens": 37
+    },
+    "53bde11d507045010a731dc81bb02e80859e58b9315a8b303b5d0fd6f9b561ef.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "cc8d23a89fc3b424be69d78163a81a9989aa198cfde95041f70824b3e39e0ac5",
+      "valid_tokens": 31
+    },
+    "84727b22433531438a399799d28d16d49fc56ec7ab0a4cf6a827ea7b2347df35.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "4df8ae0c6ca54e324ca0ea4a1499c60ba4519aadcc61f600b46111e62888c5ac",
+      "valid_tokens": 30
+    },
+    "88251b67639631ce11c41f703e85001fff4b505e1eafcc355389e15e15da67ee.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "5209b623f57f19f8c447ff6f0cb820fab501337c088b6ec446893f9d2f2dbda4",
+      "valid_tokens": 30
+    },
+    "ab4a7b38cafc00cd4f08f4a3c1937ae369dc1b6d59f77f7554999d2720177ce1.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "4251d4df987e99cbf26fc3b4630b9aaddc0550d26cd6929c2c17d96290043edb",
+      "valid_tokens": 30
+    },
+    "b75715591c78b61ee4e9abf6a84eda0e63b04bd38514a0ebd207770d415357dc.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "2c613c77243948c4f67228df12fae938079b5e1f7a200e3432f94822921d0dac",
+      "valid_tokens": 34
+    },
+    "d5d6f37fc1c8d15053ce2a356e5e09ef36977a0dfe672f9393dcc6f5bd597352.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "a7601429d574a3d22755cc1d103d1e0bab8994405de55576d141e4e220b41f58",
+      "valid_tokens": 26
+    },
+    "fd92daee5b46a4b3dd8426ff9cceea75d709cee1c7b19244b1026205cc33ba3c.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "7f25b652522a1302a42eaa01cfbf5ee69fdecb20b63914029be1312e969c087e",
+      "valid_tokens": 37
+    },
+    "fe5b483ce54fd0b450d3504eacf0f05018a50ab184fa53c7e2744b8a52e957cf.t5_len128.wan22ti2v5b.pt": {
+      "sha256": "bc2bc87b57d5b6da0a144597530a44af8d41e96cd0b03b4c8a83bc13e1402fec",
+      "valid_tokens": 33
+    }
+  },
+  "text_encoder_provenance": {
+    "cache_payload_binds_encoder_hash": false,
+    "expected_encoder": "models_t5_umt5-xxl-enc-bf16.pth",
+    "expected_encoder_sha256": "7cace0da2b446bbbbc57d031ab6cf163a3d59b366da94e5afe36745b746fd81d",
+    "limitation": "Existing context/mask-only caches have no embedded encoder or tokenizer provenance",
+    "source": "Identical official Wan2.1-T2V-1.3B and Wan2.2-TI2V-5B Hub LFS hashes, audited 2026-10-05"
+  },
+  "train_episodes": [
+    0,
+    1,
+    2,
+    3,
+    4,
+    6,
+    7,
+    8,
+    10,
+    11,
+    12,
+    13,
+    14,
+    15,
+    16,
+    17,
+    18,
+    19,
+    21,
+    22,
+    23,
+    24,
+    25,
+    26,
+    27,
+    28,
+    29,
+    30,
+    31,
+    32,
+    35,
+    36,
+    37,
+    38,
+    39,
+    40,
+    41,
+    42,
+    43,
+    45,
+    47,
+    48,
+    49,
+    50,
+    51,
+    52,
+    53,
+    55,
+    56,
+    57,
+    58,
+    59,
+    60,
+    61,
+    62,
+    63,
+    64,
+    65,
+    66,
+    67,
+    68,
+    69,
+    71,
+    72,
+    73,
+    74,
+    75,
+    76,
+    77,
+    78,
+    79,
+    80,
+    81,
+    82,
+    83,
+    85,
+    87,
+    88,
+    89,
+    90,
+    91,
+    92,
+    93,
+    94,
+    95,
+    97,
+    98,
+    99,
+    100,
+    101,
+    102,
+    103,
+    104,
+    105,
+    106,
+    107,
+    109,
+    110,
+    111,
+    112,
+    113,
+    114,
+    115,
+    116,
+    117,
+    118,
+    120,
+    121,
+    122,
+    123,
+    125,
+    126,
+    127,
+    128,
+    129,
+    130,
+    131,
+    132,
+    133,
+    135,
+    136,
+    137,
+    138,
+    139,
+    141,
+    142,
+    143,
+    144,
+    145,
+    146,
+    147,
+    148,
+    149,
+    150,
+    151,
+    152,
+    153,
+    155,
+    157,
+    159,
+    160,
+    161,
+    162,
+    163,
+    164,
+    165,
+    166,
+    167,
+    168,
+    169,
+    170,
+    171,
+    173,
+    174,
+    175,
+    176,
+    177,
+    178,
+    179,
+    180,
+    181,
+    182,
+    183,
+    185,
+    186,
+    187,
+    188,
+    189,
+    190,
+    191,
+    192,
+    193,
+    194,
+    195,
+    197,
+    198,
+    199,
+    200,
+    202,
+    203,
+    204,
+    205,
+    207,
+    208,
+    209,
+    210,
+    211,
+    212,
+    213,
+    215,
+    216,
+    217,
+    218,
+    219,
+    220,
+    221,
+    222,
+    223,
+    224,
+    225,
+    227,
+    228,
+    230,
+    231,
+    232,
+    233,
+    234,
+    236,
+    237,
+    238,
+    239,
+    240,
+    241,
+    242,
+    243,
+    244,
+    245,
+    246,
+    247,
+    248,
+    249,
+    250,
+    251,
+    252,
+    253,
+    254,
+    255,
+    256,
+    257,
+    258,
+    259,
+    260,
+    261,
+    263,
+    264,
+    266,
+    267,
+    268,
+    269,
+    270,
+    271,
+    273,
+    274,
+    276,
+    277,
+    278,
+    279,
+    280,
+    281,
+    282,
+    283,
+    284,
+    285,
+    286,
+    287,
+    288,
+    290,
+    291,
+    292,
+    294,
+    295,
+    296,
+    297,
+    298,
+    299,
+    300,
+    301,
+    302,
+    303,
+    304,
+    305,
+    306,
+    307,
+    308,
+    310,
+    311,
+    312,
+    313,
+    314,
+    315,
+    316,
+    317,
+    318,
+    319,
+    320,
+    321,
+    322,
+    323,
+    324,
+    325,
+    327,
+    328,
+    329,
+    330,
+    331,
+    332,
+    333,
+    334,
+    335,
+    337,
+    339,
+    340,
+    341,
+    342,
+    343,
+    344,
+    345,
+    346,
+    347,
+    348,
+    349,
+    350,
+    351,
+    352,
+    353,
+    354,
+    355,
+    356,
+    357,
+    358,
+    359,
+    360,
+    361,
+    362,
+    363,
+    364,
+    365,
+    367,
+    368,
+    369,
+    370,
+    373,
+    374,
+    375,
+    377,
+    378,
+    379,
+    380,
+    381,
+    383,
+    384,
+    385,
+    386,
+    387
+  ],
+  "train_windows": 92678,
+  "validation_episodes": [
+    5,
+    9,
+    20,
+    33,
+    34,
+    44,
+    46,
+    54,
+    70,
+    84,
+    86,
+    96,
+    108,
+    119,
+    124,
+    134,
+    140,
+    154,
+    156,
+    158,
+    172,
+    184,
+    196,
+    201,
+    206,
+    214,
+    226,
+    229,
+    235,
+    262,
+    265,
+    272,
+    275,
+    289,
+    293,
+    309,
+    326,
+    336,
+    338,
+    366,
+    371,
+    372,
+    376,
+    382
+  ],
+  "validation_windows": 11602,
+  "video_offsets": [
+    0,
+    4,
+    8,
+    12,
+    16,
+    20,
+    24,
+    28,
+    32
+  ]
+}

libero-long/v0-video4-action1-kv-mix/dataset_stats.json

@@ -0,0 +1,50 @@
+{
+  "action": {
+    "default": {
+      "global_max": [
+        0.9375,
+        0.9375,
+        0.9375,
+        0.32892856001853943,
+        0.36964285373687744,
+        0.375,
+        1.0
+      ],
+      "global_min": [
+        -0.9375,
+        -0.9375,
+        -0.9375,
+        -0.23642857372760773,
+        -0.3053571283817291,
+        -0.3642857074737549,
+        0.0
+      ]
+    }
+  },
+  "num_episodes": 344,
+  "num_transition": 92678,
+  "state": {
+    "default": {
+      "global_max": [
+        0.2103137969970703,
+        0.38887521624565125,
+        1.333192229270935,
+        3.7248642444610596,
+        3.5618896484375,
+        1.3863215446472168,
+        0.041575800627470016,
+        0.0013126095291227102
+      ],
+      "global_min": [
+        -0.48278069496154785,
+        -0.3309336006641388,
+        0.4457588791847229,
+        1.1323540210723877,
+        -3.6312508583068848,
+        -1.842738389968872,
+        -0.005453015677630901,
+        -0.04112039878964424
+      ]
+    }
+  }
+}

libero-long/v0-video4-action1-kv-mix/export.json

@@ -0,0 +1,35 @@
+{
+  "id": "v0_4_1_kv_mix",
+  "repo_path": "libero-long/v0-video4-action1-kv-mix",
+  "training_path": "kv_campaign_job4771/campaign_3a89bfa/mix_long/train",
+  "evaluation_path": "kv_campaign_job4771/campaign_3a89bfa/mix_long/eval_seed42",
+  "source_sha256": "1294bf07bd84c868215f9baff8e64301a2b7827d278a2a7183e7af86bea26ff6",
+  "source_bytes": 7015574271,
+  "export_sha256": "b90be503a3ee7c39305b2a6bbc94a0e649d566c5d4705644eba5476058c06340",
+  "export_bytes": 2338636277,
+  "format": "loopwam-s-v1",
+  "optimizer_included": false,
+  "step": 7250,
+  "version": "v0",
+  "video_loops": 4,
+  "action_loops": 1,
+  "action_kv_mode": "mix",
+  "parameters": 584536423,
+  "success_rate": 0.9,
+  "successes": 90,
+  "episodes": 100,
+  "evaluation_seed": 42,
+  "pooled_successes": 266,
+  "pooled_episodes": 300,
+  "pooled_success_rate": 0.8866666666666667,
+  "evaluation_seeds": [
+    42,
+    43,
+    44
+  ],
+  "training_seconds": 26302.35895577073,
+  "tensor_equality_verified": true,
+  "source_revision": "3a89bfac5cfccad66bfebbe00014f7f739a78adc",
+  "compatible_code_revision": "3a89bfac5cfccad66bfebbe00014f7f739a78adc",
+  "normalization_sha256": "78058a689c41382a2759541d5590cb3ada8c576e736285aaeebc344874366f1b"
+}

libero-long/v0-video4-action1-kv-mix/policy.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b90be503a3ee7c39305b2a6bbc94a0e649d566c5d4705644eba5476058c06340
+size 2338636277

libero-long/v0-video4-action1-kv-mix/training_timing.json

@@ -0,0 +1,54 @@
+{
+  "status": "complete",
+  "completed_updates": 7250,
+  "windows_seen": 926780,
+  "planned_updates": 7250,
+  "elapsed_training_seconds": 26302.35895577073,
+  "checkpoint_seconds_total": 90.78535918518901,
+  "checkpoint_count": 10,
+  "measured_mean_update_seconds": 3.6103511933689036,
+  "estimated_total_training_hours": 7.270846153312375,
+  "estimate_note": "Projection from measured updates; excludes setup/checkpoint overhead",
+  "latest": {
+    "update": 7250,
+    "epoch": 10,
+    "windows": 6,
+    "windows_seen": 926780,
+    "effective_passes": 10.0,
+    "seconds": 0.7885428667068481,
+    "lr": 1.0000000000000002e-06,
+    "grad_norm": 0.41784602403640747,
+    "branch_grad_after_clip": {
+      "video": 0.36324411630630493,
+      "action": 0.20577172935009003,
+      "mot": 0.00031832445529289544,
+      "proprio": 0.01751892827451229
+    },
+    "latent_cache_rank_local": {
+      "hits": 463420,
+      "misses": 0,
+      "dummy_encodes": 20,
+      "encoder_samples": 20,
+      "batches": 57930,
+      "read_seconds": 1594.8774884864688,
+      "encode_seconds": 0.9607009813189507,
+      "write_seconds": 0.0
+    },
+    "peak_allocated_gb": 60.70613098144531,
+    "peak_reserved_gb": 61.1193962097168,
+    "exit4/video_raw": 0.04651766022046407,
+    "exit4/action_raw": 0.021685960392157238,
+    "exit4/video": 0.05246804157892863,
+    "exit4/action": 0.01504881183306376,
+    "loss_video": 0.05246804157892863,
+    "loss_action": 0.01504881183306376,
+    "loop/1/video_state_rms": 2.717929204305013,
+    "loop/2/video_state_rms": 2.7954845428466797,
+    "loop/3/video_state_rms": 2.9626617431640625,
+    "loop/4/video_state_rms": 2.7946141560872397,
+    "loop/1/action_state_rms": 1.5208746592203777,
+    "loop/2/action_state_rms": 1.5208746592203777,
+    "loop/3/action_state_rms": 1.5208746592203777,
+    "loop/4/action_state_rms": 1.7305237452189128
+  }
+}