migration/checkpoint-inventory.md

# OpenWAM run1–run40 checkpoint 盘点与目录方案

盘点时间：2026-09-22；主机：`ssh turbo_ai`；根目录：`/mnt/data/limingleyang`。

**本轮仅盘点和读取校验。没有复制、移动、删除 checkpoint，也没有创建目标 checkpoints 目录。**

用户确认：`private/` 表示每个 run 独有的权重、配置和统计量；整个目录用于上传用户的私有 Hugging Face 仓库。

## 数量与证据范围

- run1–run40 全部找到：40 份正式 SFT 最终 checkpoint。
- WAM pretrain 最终 checkpoint 共 2 份：run23、run39；run40 的 pretrain 来源是 run23。
- 10 个独立 S-VAE 来源文件、9 组不同的参数。run34/run35 的全部 tensor 字节一致，文件哈希不同，训练元数据仍应分别保留。
- 另有 run19 的 VideoRAE 微调最终训练状态（step 1000）与推理 encoder 导出。
- 42 份 policy 最终权重合计 1078.535 GB（十进制），S-VAE 原始文件合计 2.192 GB。
- 全量候选清单记录 120 个 checkpoint_step 文件：42 个正式最终、8 个正式中间、70 个其他候选（开发 smoke/debug、被替代的 run21 和 3 个编号体系之前的历史产物）。
- 所有正式 policy 的 safetensors header、张量区间和文件长度已检查；下列共享组件比较覆盖 dtype、shape、参数名和全部原始 tensor 字节的 SHA-256，并非采样。
- S-VAE 已按 weights_only=True 在 CPU 上读取，并比较全部 tensor；未执行 GPU 模型 forward 或闭环评测。
- 早期时间戳目录按保存的模型/训练配置与可用 W&B 启动参数核对；run5 另由仓库 docs/run20-vlm-dino.md 的明确来源记录交叉确认。

## 40 个 SFT 最终 checkpoint 的完整路径

| Run | 保存的名称 / 对应配方 | 最终 step | 大小 GB | 完整绝对路径 |
|---|---|---:|---:|---|
| run01 | `run1_3sys` | 10690 | 26.643 | `/mnt/data/limingleyang/openwam_checkpoints/2026-09-07_23-49-38/checkpoint_step_10690.safetensors` |
| run02 | `run2_2sys` | 10690 | 22.388 | `/mnt/data/limingleyang/openwam_checkpoints/2026-09-09_13-14-42/checkpoint_step_10690.safetensors` |
| run03 | `run3_3sys_frozen` | 10690 | 26.643 | `/mnt/data/limingleyang/openwam_checkpoints/2026-09-10_01-16-34/checkpoint_step_10690.safetensors` |
| run04 | `run4_3sys_frozen` | 10690 | 26.643 | `/mnt/data/limingleyang/openwam_checkpoints/2026-09-10_10-07-03/checkpoint_step_10690.safetensors` |
| run05 | `run5_3sys_independent_views` | 10690 | 26.643 | `/mnt/data/limingleyang/openwam_checkpoints/2026-09-10_11-45-23/checkpoint_step_10690.safetensors` |
| run06 | `run6_3sys_frozen` | 10690 | 26.643 | `/mnt/data/limingleyang/openwam_checkpoints/2026-09-10_17-47-49/checkpoint_step_10690.safetensors` |
| run07 | `run7_2sys_scratch_wan` | 10690 | 22.388 | `/mnt/data/limingleyang/openwam_checkpoints/2026-09-10_21-59-35/checkpoint_step_10690.safetensors` |
| run08 | `run8_2sys_scratch_videorae` | 10690 | 22.363 | `/mnt/data/limingleyang/openwam_checkpoints/2026-09-10_22-00-06/checkpoint_step_10690.safetensors` |
| run09 | `run9_3sys_frozen` | 10690 | 26.643 | `/mnt/data/limingleyang/openwam_checkpoints/2026-09-11_00-38-41/checkpoint_step_10690.safetensors` |
| run10 | `run10_3sys` | 10690 | 26.643 | `/mnt/data/limingleyang/openwam_checkpoints/2026-09-11_00-39-04/checkpoint_step_10690.safetensors` |
| run11 | `run11_2sys_scratch_dinov3` | 10690 | 22.372 | `/mnt/data/limingleyang/openwam_checkpoints/run11_2sys_scratch_dinov3/2026-09-12_00-52-00/checkpoint_step_10690.safetensors` |
| run12 | `run12_3sys_frozen_scratch_videorae` | 10690 | 26.618 | `/mnt/data/limingleyang/openwam_checkpoints/2026-09-11_14-53-35/checkpoint_step_10690.safetensors` |
| run13 | `run13_3sys_frozen_scratch_wan` | 10690 | 26.643 | `/mnt/data/limingleyang/openwam_checkpoints/2026-09-11_14-54-15/checkpoint_step_10690.safetensors` |
| run14 | `run14_3sys` | 10690 | 26.643 | `/mnt/data/limingleyang/openwam_checkpoints/2026-09-11_15-28-19/checkpoint_step_10690.safetensors` |
| run15 | `run15_3sys_frozen_prefix7` | 10690 | 26.643 | `/mnt/data/limingleyang/openwam_checkpoints/run15_3sys_frozen_prefix7/2026-09-12_00-52-52/checkpoint_step_10690.safetensors` |
| run16 | `run16_3sys_frozen_prefix14` | 10690 | 26.643 | `/mnt/data/limingleyang/openwam_checkpoints/run16_3sys_frozen_prefix14/2026-09-12_00-53-18/checkpoint_step_10690.safetensors` |
| run17 | `run17_3sys_frozen_prefix21` | 10690 | 26.643 | `/mnt/data/limingleyang/openwam_checkpoints/run17_3sys_frozen_prefix21/2026-09-12_00-53-45/checkpoint_step_10690.safetensors` |
| run18 | `run18_3sys_frozen_scratch_dinov3` | 10690 | 26.627 | `/mnt/data/limingleyang/openwam_checkpoints/run18_3sys_frozen_scratch_dinov3/2026-09-12_17-07-24/checkpoint_step_10690.safetensors` |
| run19 | `run19_2sys_scratch_videorae_ft` | 10690 | 22.363 | `/mnt/data/limingleyang/openwam_checkpoints/run19_2sys_scratch_videorae_ft/2026-09-12_19-23-18/checkpoint_step_10690.safetensors` |
| run20 | `run20_3sys_vlm_dino` | 10690 | 26.725 | `/mnt/data/limingleyang/openwam_checkpoints/run20_3sys_vlm_dino/2026-09-13_00-38-39/checkpoint_step_10690.safetensors` |
| run21 | `run21_2sys_scratch_videorae_current_official` | 10690 | 22.363 | `/mnt/data/limingleyang/openwam_checkpoints/run21_2sys_scratch_videorae_current/2026-09-13_07-29-21/checkpoint_step_10690.safetensors` |
| run22 | `run22-2sys-scratch-dinov3-patch2` | 10690 | 22.373 | `/mnt/data/limingleyang/openwam_checkpoints/run22-2sys-scratch-dinov3-patch2/sft_libero/checkpoint_step_10690.safetensors` |
| run23 | `run23-2sys-pretrain-dinov3-patch2-sft-libero` | 10690 | 22.373 | `/mnt/data/limingleyang/openwam_checkpoints/run23-2sys-pretrain-dinov3-patch2/sft_libero/checkpoint_step_10690.safetensors` |
| run24 | `run24_3sys_frozen_scratch_vjepa_svae` | 42730 | 28.581 | `/mnt/data/limingleyang/openwam_checkpoints/run24_3sys_frozen_scratch_vjepa_svae/2026-09-14_15-40-37/checkpoint_step_42730.safetensors` |
| run25 | `run25_2sys_scratch_vjepa_svae` | 10690 | 24.326 | `/mnt/data/limingleyang/openwam_checkpoints/run25_2sys_scratch_vjepa_svae/2026-09-15_12-05-32/checkpoint_step_10690.safetensors` |
| run26 | `run26-future-dit-dinov3-frozen` | 10690 | 26.628 | `/mnt/data/limingleyang/OpenWAM-private/outputs/run26-future-dit-dinov3-frozen/2026-09-15_01-09-52/checkpoint_step_10690.safetensors` |
| run27 | `run27-future-dit-dinov3-train-vlm` | 10690 | 26.628 | `/mnt/data/limingleyang/OpenWAM-private/outputs/run27-future-dit-dinov3-train-vlm/2026-09-15_01-16-59/checkpoint_step_10690.safetensors` |
| run28 | `run28_3sys_frozen_scratch_vjepa_svae_patch2` | 10690 | 28.582 | `/mnt/data/limingleyang/openwam_checkpoints/run28_3sys_frozen_scratch_vjepa_svae_patch2/2026-09-16_15-42-25/checkpoint_step_10690.safetensors` |
| run29 | `run29_2sys_scratch_vjepa_svae_patch2` | 10690 | 24.327 | `/mnt/data/limingleyang/openwam_checkpoints/run29_2sys_scratch_vjepa_svae_patch2/2026-09-15_19-19-16/checkpoint_step_10690.safetensors` |
| run30 | `run30_3sys_vjepa_pairwise` | 21370 | 28.581 | `/mnt/data/limingleyang/vjepa_target_variants/run30_3sys_vjepa_pairwise/policy/checkpoint_step_21370.safetensors` |
| run31 | `run31_3sys_vjepa_future_only` | 21370 | 28.581 | `/mnt/data/limingleyang/vjepa_target_variants/run31_3sys_vjepa_future_only/policy/checkpoint_step_21370.safetensors` |
| run32 | `run32_2sys_vjepa_pairwise` | 10690 | 24.326 | `/mnt/data/limingleyang/vjepa_target_variants/run32_2sys_vjepa_pairwise/2026-09-16_13-55-43/checkpoint_step_10690.safetensors` |
| run33 | `run33_2sys_vjepa_future_only` | 10690 | 24.326 | `/mnt/data/limingleyang/vjepa_target_variants/run33_2sys_vjepa_future_only/2026-09-16_15-29-52/checkpoint_step_10690.safetensors` |
| run34 | `run34_3sys_frozen_scratch_dinov3_svae_temporal` | 10690 | 26.594 | `/mnt/data/limingleyang/openwam_temporal_pipeline/run34_3sys_frozen_scratch_dinov3_svae_temporal/policy/checkpoint_step_10690.safetensors` |
| run35 | `run35_2sys_scratch_dinov3_svae_temporal` | 10690 | 22.339 | `/mnt/data/limingleyang/openwam_temporal_pipeline/run35_2sys_scratch_dinov3_svae_temporal/policy/checkpoint_step_10690.safetensors` |
| run36 | `run36_3sys_frozen_scratch_dinov3_patch2` | 10690 | 26.628 | `/mnt/data/limingleyang/openwam_checkpoints/run36_3sys_frozen_scratch_dinov3_patch2/2026-09-17_15-39-44/checkpoint_step_10690.safetensors` |
| run37 | `run37-future-dit-vjepa-with-f0` | 10690 | 28.581 | `/mnt/data/limingleyang/future_dit_vjepa_targets/run37-future-dit-vjepa-with-f0/policy/checkpoint_step_10690.safetensors` |
| run38 | `run38-future-dit-vjepa-without-f0` | 10690 | 28.581 | `/mnt/data/limingleyang/future_dit_vjepa_targets/run38-future-dit-vjepa-without-f0/policy/checkpoint_step_10690.safetensors` |
| run39 | `run39-3sys-pretrain-dinov3-patch2-sft-libero` | 10690 | 26.628 | `/mnt/data/limingleyang/openwam_checkpoints/run39-3sys-pretrain-dinov3-patch2/sft_libero/checkpoint_step_10690.safetensors` |
| run40 | `run40-3sys-from-2sys-pretrain-dinov3-patch2-sft-libero` | 10690 | 26.628 | `/mnt/data/limingleyang/openwam_checkpoints/run40-3sys-from-2sys-pretrain-dinov3-patch2/sft_libero/checkpoint_step_10690.safetensors` |

run24 的最终 step 是 42730，run30/run31 是 21370，其余 SFT 均为 10690。

## Pretrain 与额外训练的 encoder

**run23：run23-2sys-pretrain-dinov3-patch2-pretrain，step 20953，22.373 GB**

```text
/mnt/data/limingleyang/openwam_checkpoints/run23-2sys-pretrain-dinov3-patch2/pretrain/checkpoint_step_20953.safetensors
```

**run39：run39-3sys-pretrain-dinov3-patch2-pretrain，step 20953，26.628 GB**

```text
/mnt/data/limingleyang/openwam_checkpoints/run39-3sys-pretrain-dinov3-patch2/pretrain/checkpoint_step_20953.safetensors
```

run40 的保存配置 `training.finetune_ckpt_path` 指向上面的 run23 pretrain；不应人为生成一份重复的 run40 pretrain。

**run19 VideoRAE 微调**：最终训练 checkpoint 为 step 1000，另保留导出的 encoder 及其 latent statistics。

```text
/mnt/data/limingleyang/openwam_videorae/run19_2sys_scratch_videorae_ft/last.pt
/mnt/data/limingleyang/openwam_videorae/run19_2sys_scratch_videorae_ft/encoder.pt
/mnt/data/limingleyang/openwam_videorae/run19_2sys_scratch_videorae_ft/latent_stats_libero.pt
```

## S-VAE 来源与复用关系

| 使用的 run | 来源 | 大小 MB | 格式 | 完整绝对路径 |
|---|---|---:|---|---|
| run11, run18, run22, run36 | DINOv3 / LIBERO | 134.59 | v2 | `/mnt/data/limingleyang/openwam_svae/libero_dinov3_vitb16/svae48/svae.pt` |
| run23, run39, run40 | DINOv3 / robot+ego pretrain | 132.84 | v2 | `/mnt/data/limingleyang/openwam_artifacts/run23-2sys-pretrain-dinov3-patch2/svae_train/svae48/svae.pt` |
| run24, run25, run28, run29 | V-JEPA 2.1 / LIBERO | 331.72 | v2 | `/mnt/data/limingleyang/yfhu25/openwam_svae/libero_vjepa21_vitg384/svae48/svae.pt` |
| run26, run27 | Future-DiT / DINOv3 | 134.17 | v2 | `/mnt/data/limingleyang/OpenWAM-private/artifacts/future_dit_libero_v1/svae48/svae.pt` |
| run30, run32 | V-JEPA pairwise | 331.72 | v2 | `/mnt/data/limingleyang/vjepa_target_variants/run30_3sys_vjepa_pairwise/svae48/svae.pt` |
| run31, run33 | V-JEPA future-only | 331.72 | v2 | `/mnt/data/limingleyang/vjepa_target_variants/run31_3sys_vjepa_future_only/svae48/svae.pt` |
| run34 | DINOv3 temporal | 66.87 | v3 | `/mnt/data/limingleyang/openwam_temporal_pipeline/run34_3sys_frozen_scratch_dinov3_svae_temporal/svae48/svae.pt` |
| run35 | DINOv3 temporal | 66.87 | v3 | `/mnt/data/limingleyang/openwam_temporal_pipeline/run35_2sys_scratch_dinov3_svae_temporal/svae48/svae.pt` |
| run37 | V-JEPA with-f0 | 330.58 | v3 | `/mnt/data/limingleyang/future_dit_vjepa_targets/run37-future-dit-vjepa-with-f0/svae48/svae.pt` |
| run38 | V-JEPA without-f0 | 330.58 | v3 | `/mnt/data/limingleyang/future_dit_vjepa_targets/run38-future-dit-vjepa-without-f0/svae48/svae.pt` |

run34 和 run35：独立 `.pt` 的文件 SHA-256 不同，但全部 tensor 的 SHA-256 相同；policy 中实际保存的 S-VAE 也完全相同。因此可以共用模型参数，同时分别保存训练配置、指标和来源。其余 8 套 S-VAE 参数不同，不应按相同架构合并。

## 已经确认可以共用的权重

下表单份大小只计 tensor 数据，不计很小的结构 JSON/tokenizer。不同的哈希组必须保留为不同模型版本。

| 组件 | 单份 GB | 共用范围 |
|---|---:|---|
| Cosmos-Reason1-7B | 16.5843 | 全部 42 个最终 policy（40 SFT + 2 pretrain） |
| Wan VAE | 0.2538 | run01, run02, run03, run04, run05, run06, run07, run09, run10, run13, run14, run15, run16, run17, run20 |
| Qwen3-VL token embeddings | 0.6223 | run01, run03, run04, run05, run06, run09, run10, run12, run13, run14, run15, run16, run17, run18, run20, run24, run26, run27, run28, run30, run31, run34, run36, run37, run38, run39 (pretrain), run39, run40 |
| Qwen3-VL vision tower | 0.8139 | run01, run03, run04, run05, run06, run09, run10, run12, run13, run14, run15, run16, run17, run18, run20, run24, run26, run27, run28, run30, run31, run34, run36, run37, run38, run39 (pretrain), run39, run40 |
| Qwen3-VL frozen decoder | 2.8188 | run03, run04, run06, run09, run12, run13, run15, run16, run17, run18, run24, run26, run28, run30, run31, run34, run36, run39 (pretrain), run39, run40 |
| VideoRAE compressor | 0.0564 | run08, run12, run21 |
| VideoRAE frozen teacher | 0.1725 | run08, run12, run19, run21 |
| policy 内的 S-VAE | 0.0664 | run11, run18, run22, run36 |
| policy 内的 S-VAE | 0.0664 | run23 (pretrain), run23, run39 (pretrain), run39, run40 |
| policy 内的 S-VAE | 0.1650 | run24, run25, run28, run29 |
| policy 内的 S-VAE | 0.0664 | run26, run27 |
| policy 内的 S-VAE | 0.1650 | run30, run32 |
| policy 内的 S-VAE | 0.1650 | run31, run33 |
| policy 内的 S-VAE | 0.0333 | run34, run35 |
| DINOv3 ViT-B/16 | 0.1713 | run11, run18, run22, run23 (pretrain), run23, run26, run27, run34, run35, run36, run39 (pretrain), run39, run40 |
| V-JEPA 2.1 ViT-g/384 | 2.0265 | run24, run25, run28, run29, run30, run31, run32, run33, run37, run38 |

**不应合并**：每个 run 训练后的 video DiT/action expert；可训练 Qwen decoder（run1/5/10/14/20/27/37/38）；run19 微调后的 VideoRAE compressor；不同哈希的 S-VAE。

Reason1 的 42 份数据完全一致，单这一项可减少约 679.96 GB。所有已证实相同的上述组件只保留一份，理论上可把这 42 份 policy 从约 1078.53 GB 降到约 280.37 GB。该估算不含独立 S-VAE、VideoRAE 完整训练状态、配置、tokenizer、索引和日志，也不是已经完成的打包结果。

## 拟定的新目录结构

```text
/mnt/data/limingleyang/checkpoints/
├── README.md
├── catalog.json                     # run + stage → 相对路径、来源、最终 step、校验值
├── shared/
│   ├── reason1/<hash>/
│   ├── qwen3-vl-2b/{vision,embeddings,frozen-decoder}/<hash>/
│   ├── wan-vae/<hash>/
│   ├── dinov3-vitb16/<hash>/
│   ├── vjepa21-vitg384/<hash>/
│   └── videorae/{teacher,compressor}/<hash>/
├── pretrain/
│   ├── run23-2sys-pretrain-dinov3-patch2/
│   ├── run39-3sys-pretrain-dinov3-patch2/
│   └── run19-videorae-libero-finetune/
├── svae/
│   ├── run11-dinov3-libero/
│   ├── run23-dinov3-pretrain/
│   ├── run24-vjepa-libero/
│   ├── run26-future-dit-dinov3/
│   ├── run30-vjepa-pairwise/
│   ├── run31-vjepa-future-only/
│   ├── run34-run35-dinov3-temporal/    # 一套参数，两份训练来源记录
│   ├── run37-vjepa-with-f0/
│   └── run38-vjepa-without-f0/
├── sft/
│   ├── run01_3sys/
│   ├── run02_2sys/
│   ├── ...                          # run03–run39，保留 run 原名
│   └── run40-3sys-from-2sys-pretrain-dinov3-patch2/
└── tools/                           # 按 run 检索、校验、还原/加载辅助工具

每个 run 目录：
runXX-<原始实验名称>/
├── checkpoint.index.json            # 指向 private/shared/svae 内真实文件的相对路径
└── private/
    ├── weights.safetensors          # 该 run 独有的最终权重
    ├── config.yaml
    ├── normalization_stats.npy      # SFT 部署所需；pretrain 按实际产物保留
    ├── assets/                      # 该 run 的结构配置、tokenizer 和其他必需 sidecars
    └── provenance.json              # 旧路径、最终 step、原始哈希、训练来源
```

全部文件实际复制，不使用软链接或硬链接。共享依赖通过相对路径索引引用；完整 checkpoints 目录应可以迁移到任意位置。S-VAE 的原始 .pt 及元数据应能无损保留/还原；policy 中的 bf16 S-VAE 与独立模型的 fp32 参数需分别记录精度，不能直接互相替换。

**加载兼容性**：当前 OpenWAM loader 读取完整 safetensors；拆分后的 `private/weights.safetensors` 不能直接冒充完整 checkpoint。真正打包时需同时提供索引加载支持，或先字节校验地还原为现有 loader 可读取的完整文件。

## 旧目录中的其他候选与清理边界

8 份明确的中间 checkpoint 共约 208.42 GB：run23/run39 pretrain 的 step 18000、20000，以及 run26/27/37/38 的 step 10000。完整路径：

```text
/mnt/data/limingleyang/openwam_checkpoints/run23-2sys-pretrain-dinov3-patch2/pretrain/checkpoint_step_18000.safetensors
/mnt/data/limingleyang/openwam_checkpoints/run23-2sys-pretrain-dinov3-patch2/pretrain/checkpoint_step_20000.safetensors
/mnt/data/limingleyang/OpenWAM-private/outputs/run26-future-dit-dinov3-frozen/2026-09-15_01-09-52/checkpoint_step_10000.safetensors
/mnt/data/limingleyang/OpenWAM-private/outputs/run27-future-dit-dinov3-train-vlm/2026-09-15_01-16-59/checkpoint_step_10000.safetensors
/mnt/data/limingleyang/future_dit_vjepa_targets/run37-future-dit-vjepa-with-f0/policy/checkpoint_step_10000.safetensors
/mnt/data/limingleyang/future_dit_vjepa_targets/run38-future-dit-vjepa-without-f0/policy/checkpoint_step_10000.safetensors
/mnt/data/limingleyang/openwam_checkpoints/run39-3sys-pretrain-dinov3-patch2/pretrain/checkpoint_step_18000.safetensors
/mnt/data/limingleyang/openwam_checkpoints/run39-3sys-pretrain-dinov3-patch2/pretrain/checkpoint_step_20000.safetensors
```

以下 3 份属于正式 run1–run40 编号之前的历史产物，单独登记，不把它们冒认成编号后的最终版本：

```text
/mnt/data/limingleyang/openwam_checkpoints/2026-09-06_10-33-18/checkpoint_step_19780.safetensors
/mnt/data/limingleyang/openwam_checkpoints/2026-09-02_15-25-45/checkpoint_step_16000.safetensors
/mnt/data/limingleyang/openwam_checkpoints/2026-09-04_19-43-45/checkpoint_step_19780.safetensors
```

另有被替代的 run21：

```text
/mnt/data/limingleyang/openwam_artifacts/run21-superseded-finetuned-20260913_013817/checkpoints/2026-09-13_01-39-58/checkpoint_step_10690.safetensors
```

所有 120 个候选的完整路径见同目录 `all-checkpoint-candidates.tsv`。未对任何候选执行删除。后续先复制最终产物，再验证完整性与加载依赖，最后按明确清单清理中间权重、优化器状态、特征缓存及重复旧副本。原始数据集不属于中间缓存。

## 机器可读证据

- `checkpoint-paths.txt`：正式最终 policy、S-VAE、run19 encoder 的 54 条路径。
- `all-checkpoint-candidates.tsv`：120 个 checkpoint_step 候选。
- `plan.json`：逐 run 来源、最终 step、目标目录和 S-VAE 依赖。
- `shared_weights.json`：42 个 policy 的完整组件 SHA-256 与精确共用集合。
- `openwam-svae-audit-20260922.json`：独立 S-VAE 的文件哈希、全部 tensor 哈希和保存元数据。