OpenWAM-3sys / migration /checkpoint-inventory.md
qiuly's picture
Add files using upload-large-folder tool
9020600 verified
|
Raw History Blame Contribute Delete
19.7 kB

OpenWAM run1–run40 checkpoint 盘点与目录方案

盘点时间:2026-09-22;主机:ssh turbo_ai;根目录:/mnt/data/limingleyang。

本轮仅盘点和读取校验。没有复制、移动、删除 checkpoint,也没有创建目标 checkpoints 目录。

用户确认:private/ 表示每个 run 独有的权重、配置和统计量;整个目录用于上传用户的私有 Hugging Face 仓库。

数量与证据范围

  • run1–run40 全部找到:40 份正式 SFT 最终 checkpoint。
  • WAM pretrain 最终 checkpoint 共 2 份:run23、run39;run40 的 pretrain 来源是 run23。
  • 10 个独立 S-VAE 来源文件、9 组不同的参数。run34/run35 的全部 tensor 字节一致,文件哈希不同,训练元数据仍应分别保留。
  • 另有 run19 的 VideoRAE 微调最终训练状态(step 1000)与推理 encoder 导出。
  • 42 份 policy 最终权重合计 1078.535 GB(十进制),S-VAE 原始文件合计 2.192 GB。
  • 全量候选清单记录 120 个 checkpoint_step 文件:42 个正式最终、8 个正式中间、70 个其他候选(开发 smoke/debug、被替代的 run21 和 3 个编号体系之前的历史产物)。
  • 所有正式 policy 的 safetensors header、张量区间和文件长度已检查;下列共享组件比较覆盖 dtype、shape、参数名和全部原始 tensor 字节的 SHA-256,并非采样。
  • S-VAE 已按 weights_only=True 在 CPU 上读取,并比较全部 tensor;未执行 GPU 模型 forward 或闭环评测。
  • 早期时间戳目录按保存的模型/训练配置与可用 W&B 启动参数核对;run5 另由仓库 docs/run20-vlm-dino.md 的明确来源记录交叉确认。

40 个 SFT 最终 checkpoint 的完整路径

Run 保存的名称 / 对应配方 最终 step 大小 GB 完整绝对路径
run01 run1_3sys 10690 26.643 /mnt/data/limingleyang/openwam_checkpoints/2026-09-07_23-49-38/checkpoint_step_10690.safetensors
run02 run2_2sys 10690 22.388 /mnt/data/limingleyang/openwam_checkpoints/2026-09-09_13-14-42/checkpoint_step_10690.safetensors
run03 run3_3sys_frozen 10690 26.643 /mnt/data/limingleyang/openwam_checkpoints/2026-09-10_01-16-34/checkpoint_step_10690.safetensors
run04 run4_3sys_frozen 10690 26.643 /mnt/data/limingleyang/openwam_checkpoints/2026-09-10_10-07-03/checkpoint_step_10690.safetensors
run05 run5_3sys_independent_views 10690 26.643 /mnt/data/limingleyang/openwam_checkpoints/2026-09-10_11-45-23/checkpoint_step_10690.safetensors
run06 run6_3sys_frozen 10690 26.643 /mnt/data/limingleyang/openwam_checkpoints/2026-09-10_17-47-49/checkpoint_step_10690.safetensors
run07 run7_2sys_scratch_wan 10690 22.388 /mnt/data/limingleyang/openwam_checkpoints/2026-09-10_21-59-35/checkpoint_step_10690.safetensors
run08 run8_2sys_scratch_videorae 10690 22.363 /mnt/data/limingleyang/openwam_checkpoints/2026-09-10_22-00-06/checkpoint_step_10690.safetensors
run09 run9_3sys_frozen 10690 26.643 /mnt/data/limingleyang/openwam_checkpoints/2026-09-11_00-38-41/checkpoint_step_10690.safetensors
run10 run10_3sys 10690 26.643 /mnt/data/limingleyang/openwam_checkpoints/2026-09-11_00-39-04/checkpoint_step_10690.safetensors
run11 run11_2sys_scratch_dinov3 10690 22.372 /mnt/data/limingleyang/openwam_checkpoints/run11_2sys_scratch_dinov3/2026-09-12_00-52-00/checkpoint_step_10690.safetensors
run12 run12_3sys_frozen_scratch_videorae 10690 26.618 /mnt/data/limingleyang/openwam_checkpoints/2026-09-11_14-53-35/checkpoint_step_10690.safetensors
run13 run13_3sys_frozen_scratch_wan 10690 26.643 /mnt/data/limingleyang/openwam_checkpoints/2026-09-11_14-54-15/checkpoint_step_10690.safetensors
run14 run14_3sys 10690 26.643 /mnt/data/limingleyang/openwam_checkpoints/2026-09-11_15-28-19/checkpoint_step_10690.safetensors
run15 run15_3sys_frozen_prefix7 10690 26.643 /mnt/data/limingleyang/openwam_checkpoints/run15_3sys_frozen_prefix7/2026-09-12_00-52-52/checkpoint_step_10690.safetensors
run16 run16_3sys_frozen_prefix14 10690 26.643 /mnt/data/limingleyang/openwam_checkpoints/run16_3sys_frozen_prefix14/2026-09-12_00-53-18/checkpoint_step_10690.safetensors
run17 run17_3sys_frozen_prefix21 10690 26.643 /mnt/data/limingleyang/openwam_checkpoints/run17_3sys_frozen_prefix21/2026-09-12_00-53-45/checkpoint_step_10690.safetensors
run18 run18_3sys_frozen_scratch_dinov3 10690 26.627 /mnt/data/limingleyang/openwam_checkpoints/run18_3sys_frozen_scratch_dinov3/2026-09-12_17-07-24/checkpoint_step_10690.safetensors
run19 run19_2sys_scratch_videorae_ft 10690 22.363 /mnt/data/limingleyang/openwam_checkpoints/run19_2sys_scratch_videorae_ft/2026-09-12_19-23-18/checkpoint_step_10690.safetensors
run20 run20_3sys_vlm_dino 10690 26.725 /mnt/data/limingleyang/openwam_checkpoints/run20_3sys_vlm_dino/2026-09-13_00-38-39/checkpoint_step_10690.safetensors
run21 run21_2sys_scratch_videorae_current_official 10690 22.363 /mnt/data/limingleyang/openwam_checkpoints/run21_2sys_scratch_videorae_current/2026-09-13_07-29-21/checkpoint_step_10690.safetensors
run22 run22-2sys-scratch-dinov3-patch2 10690 22.373 /mnt/data/limingleyang/openwam_checkpoints/run22-2sys-scratch-dinov3-patch2/sft_libero/checkpoint_step_10690.safetensors
run23 run23-2sys-pretrain-dinov3-patch2-sft-libero 10690 22.373 /mnt/data/limingleyang/openwam_checkpoints/run23-2sys-pretrain-dinov3-patch2/sft_libero/checkpoint_step_10690.safetensors
run24 run24_3sys_frozen_scratch_vjepa_svae 42730 28.581 /mnt/data/limingleyang/openwam_checkpoints/run24_3sys_frozen_scratch_vjepa_svae/2026-09-14_15-40-37/checkpoint_step_42730.safetensors
run25 run25_2sys_scratch_vjepa_svae 10690 24.326 /mnt/data/limingleyang/openwam_checkpoints/run25_2sys_scratch_vjepa_svae/2026-09-15_12-05-32/checkpoint_step_10690.safetensors
run26 run26-future-dit-dinov3-frozen 10690 26.628 /mnt/data/limingleyang/OpenWAM-private/outputs/run26-future-dit-dinov3-frozen/2026-09-15_01-09-52/checkpoint_step_10690.safetensors
run27 run27-future-dit-dinov3-train-vlm 10690 26.628 /mnt/data/limingleyang/OpenWAM-private/outputs/run27-future-dit-dinov3-train-vlm/2026-09-15_01-16-59/checkpoint_step_10690.safetensors
run28 run28_3sys_frozen_scratch_vjepa_svae_patch2 10690 28.582 /mnt/data/limingleyang/openwam_checkpoints/run28_3sys_frozen_scratch_vjepa_svae_patch2/2026-09-16_15-42-25/checkpoint_step_10690.safetensors
run29 run29_2sys_scratch_vjepa_svae_patch2 10690 24.327 /mnt/data/limingleyang/openwam_checkpoints/run29_2sys_scratch_vjepa_svae_patch2/2026-09-15_19-19-16/checkpoint_step_10690.safetensors
run30 run30_3sys_vjepa_pairwise 21370 28.581 /mnt/data/limingleyang/vjepa_target_variants/run30_3sys_vjepa_pairwise/policy/checkpoint_step_21370.safetensors
run31 run31_3sys_vjepa_future_only 21370 28.581 /mnt/data/limingleyang/vjepa_target_variants/run31_3sys_vjepa_future_only/policy/checkpoint_step_21370.safetensors
run32 run32_2sys_vjepa_pairwise 10690 24.326 /mnt/data/limingleyang/vjepa_target_variants/run32_2sys_vjepa_pairwise/2026-09-16_13-55-43/checkpoint_step_10690.safetensors
run33 run33_2sys_vjepa_future_only 10690 24.326 /mnt/data/limingleyang/vjepa_target_variants/run33_2sys_vjepa_future_only/2026-09-16_15-29-52/checkpoint_step_10690.safetensors
run34 run34_3sys_frozen_scratch_dinov3_svae_temporal 10690 26.594 /mnt/data/limingleyang/openwam_temporal_pipeline/run34_3sys_frozen_scratch_dinov3_svae_temporal/policy/checkpoint_step_10690.safetensors
run35 run35_2sys_scratch_dinov3_svae_temporal 10690 22.339 /mnt/data/limingleyang/openwam_temporal_pipeline/run35_2sys_scratch_dinov3_svae_temporal/policy/checkpoint_step_10690.safetensors
run36 run36_3sys_frozen_scratch_dinov3_patch2 10690 26.628 /mnt/data/limingleyang/openwam_checkpoints/run36_3sys_frozen_scratch_dinov3_patch2/2026-09-17_15-39-44/checkpoint_step_10690.safetensors
run37 run37-future-dit-vjepa-with-f0 10690 28.581 /mnt/data/limingleyang/future_dit_vjepa_targets/run37-future-dit-vjepa-with-f0/policy/checkpoint_step_10690.safetensors
run38 run38-future-dit-vjepa-without-f0 10690 28.581 /mnt/data/limingleyang/future_dit_vjepa_targets/run38-future-dit-vjepa-without-f0/policy/checkpoint_step_10690.safetensors
run39 run39-3sys-pretrain-dinov3-patch2-sft-libero 10690 26.628 /mnt/data/limingleyang/openwam_checkpoints/run39-3sys-pretrain-dinov3-patch2/sft_libero/checkpoint_step_10690.safetensors
run40 run40-3sys-from-2sys-pretrain-dinov3-patch2-sft-libero 10690 26.628 /mnt/data/limingleyang/openwam_checkpoints/run40-3sys-from-2sys-pretrain-dinov3-patch2/sft_libero/checkpoint_step_10690.safetensors

run24 的最终 step 是 42730,run30/run31 是 21370,其余 SFT 均为 10690。

Pretrain 与额外训练的 encoder

run23:run23-2sys-pretrain-dinov3-patch2-pretrain,step 20953,22.373 GB

/mnt/data/limingleyang/openwam_checkpoints/run23-2sys-pretrain-dinov3-patch2/pretrain/checkpoint_step_20953.safetensors

run39:run39-3sys-pretrain-dinov3-patch2-pretrain,step 20953,26.628 GB

/mnt/data/limingleyang/openwam_checkpoints/run39-3sys-pretrain-dinov3-patch2/pretrain/checkpoint_step_20953.safetensors

run40 的保存配置 training.finetune_ckpt_path 指向上面的 run23 pretrain;不应人为生成一份重复的 run40 pretrain。

run19 VideoRAE 微调:最终训练 checkpoint 为 step 1000,另保留导出的 encoder 及其 latent statistics。

/mnt/data/limingleyang/openwam_videorae/run19_2sys_scratch_videorae_ft/last.pt
/mnt/data/limingleyang/openwam_videorae/run19_2sys_scratch_videorae_ft/encoder.pt
/mnt/data/limingleyang/openwam_videorae/run19_2sys_scratch_videorae_ft/latent_stats_libero.pt

S-VAE 来源与复用关系

使用的 run 来源 大小 MB 格式 完整绝对路径
run11, run18, run22, run36 DINOv3 / LIBERO 134.59 v2 /mnt/data/limingleyang/openwam_svae/libero_dinov3_vitb16/svae48/svae.pt
run23, run39, run40 DINOv3 / robot+ego pretrain 132.84 v2 /mnt/data/limingleyang/openwam_artifacts/run23-2sys-pretrain-dinov3-patch2/svae_train/svae48/svae.pt
run24, run25, run28, run29 V-JEPA 2.1 / LIBERO 331.72 v2 /mnt/data/limingleyang/yfhu25/openwam_svae/libero_vjepa21_vitg384/svae48/svae.pt
run26, run27 Future-DiT / DINOv3 134.17 v2 /mnt/data/limingleyang/OpenWAM-private/artifacts/future_dit_libero_v1/svae48/svae.pt
run30, run32 V-JEPA pairwise 331.72 v2 /mnt/data/limingleyang/vjepa_target_variants/run30_3sys_vjepa_pairwise/svae48/svae.pt
run31, run33 V-JEPA future-only 331.72 v2 /mnt/data/limingleyang/vjepa_target_variants/run31_3sys_vjepa_future_only/svae48/svae.pt
run34 DINOv3 temporal 66.87 v3 /mnt/data/limingleyang/openwam_temporal_pipeline/run34_3sys_frozen_scratch_dinov3_svae_temporal/svae48/svae.pt
run35 DINOv3 temporal 66.87 v3 /mnt/data/limingleyang/openwam_temporal_pipeline/run35_2sys_scratch_dinov3_svae_temporal/svae48/svae.pt
run37 V-JEPA with-f0 330.58 v3 /mnt/data/limingleyang/future_dit_vjepa_targets/run37-future-dit-vjepa-with-f0/svae48/svae.pt
run38 V-JEPA without-f0 330.58 v3 /mnt/data/limingleyang/future_dit_vjepa_targets/run38-future-dit-vjepa-without-f0/svae48/svae.pt

run34 和 run35:独立 .pt 的文件 SHA-256 不同,但全部 tensor 的 SHA-256 相同;policy 中实际保存的 S-VAE 也完全相同。因此可以共用模型参数,同时分别保存训练配置、指标和来源。其余 8 套 S-VAE 参数不同,不应按相同架构合并。

已经确认可以共用的权重

下表单份大小只计 tensor 数据,不计很小的结构 JSON/tokenizer。不同的哈希组必须保留为不同模型版本。

组件 单份 GB 共用范围
Cosmos-Reason1-7B 16.5843 全部 42 个最终 policy(40 SFT + 2 pretrain)
Wan VAE 0.2538 run01, run02, run03, run04, run05, run06, run07, run09, run10, run13, run14, run15, run16, run17, run20
Qwen3-VL token embeddings 0.6223 run01, run03, run04, run05, run06, run09, run10, run12, run13, run14, run15, run16, run17, run18, run20, run24, run26, run27, run28, run30, run31, run34, run36, run37, run38, run39 (pretrain), run39, run40
Qwen3-VL vision tower 0.8139 run01, run03, run04, run05, run06, run09, run10, run12, run13, run14, run15, run16, run17, run18, run20, run24, run26, run27, run28, run30, run31, run34, run36, run37, run38, run39 (pretrain), run39, run40
Qwen3-VL frozen decoder 2.8188 run03, run04, run06, run09, run12, run13, run15, run16, run17, run18, run24, run26, run28, run30, run31, run34, run36, run39 (pretrain), run39, run40
VideoRAE compressor 0.0564 run08, run12, run21
VideoRAE frozen teacher 0.1725 run08, run12, run19, run21
policy 内的 S-VAE 0.0664 run11, run18, run22, run36
policy 内的 S-VAE 0.0664 run23 (pretrain), run23, run39 (pretrain), run39, run40
policy 内的 S-VAE 0.1650 run24, run25, run28, run29
policy 内的 S-VAE 0.0664 run26, run27
policy 内的 S-VAE 0.1650 run30, run32
policy 内的 S-VAE 0.1650 run31, run33
policy 内的 S-VAE 0.0333 run34, run35
DINOv3 ViT-B/16 0.1713 run11, run18, run22, run23 (pretrain), run23, run26, run27, run34, run35, run36, run39 (pretrain), run39, run40
V-JEPA 2.1 ViT-g/384 2.0265 run24, run25, run28, run29, run30, run31, run32, run33, run37, run38

不应合并:每个 run 训练后的 video DiT/action expert;可训练 Qwen decoder(run1/5/10/14/20/27/37/38);run19 微调后的 VideoRAE compressor;不同哈希的 S-VAE。

Reason1 的 42 份数据完全一致,单这一项可减少约 679.96 GB。所有已证实相同的上述组件只保留一份,理论上可把这 42 份 policy 从约 1078.53 GB 降到约 280.37 GB。该估算不含独立 S-VAE、VideoRAE 完整训练状态、配置、tokenizer、索引和日志,也不是已经完成的打包结果。

拟定的新目录结构

/mnt/data/limingleyang/checkpoints/
├── README.md
├── catalog.json                     # run + stage → 相对路径、来源、最终 step、校验值
├── shared/
│   ├── reason1/<hash>/
│   ├── qwen3-vl-2b/{vision,embeddings,frozen-decoder}/<hash>/
│   ├── wan-vae/<hash>/
│   ├── dinov3-vitb16/<hash>/
│   ├── vjepa21-vitg384/<hash>/
│   └── videorae/{teacher,compressor}/<hash>/
├── pretrain/
│   ├── run23-2sys-pretrain-dinov3-patch2/
│   ├── run39-3sys-pretrain-dinov3-patch2/
│   └── run19-videorae-libero-finetune/
├── svae/
│   ├── run11-dinov3-libero/
│   ├── run23-dinov3-pretrain/
│   ├── run24-vjepa-libero/
│   ├── run26-future-dit-dinov3/
│   ├── run30-vjepa-pairwise/
│   ├── run31-vjepa-future-only/
│   ├── run34-run35-dinov3-temporal/    # 一套参数,两份训练来源记录
│   ├── run37-vjepa-with-f0/
│   └── run38-vjepa-without-f0/
├── sft/
│   ├── run01_3sys/
│   ├── run02_2sys/
│   ├── ...                          # run03–run39,保留 run 原名
│   └── run40-3sys-from-2sys-pretrain-dinov3-patch2/
└── tools/                           # 按 run 检索、校验、还原/加载辅助工具

每个 run 目录:
runXX-<原始实验名称>/
├── checkpoint.index.json            # 指向 private/shared/svae 内真实文件的相对路径
└── private/
    ├── weights.safetensors          # 该 run 独有的最终权重
    ├── config.yaml
    ├── normalization_stats.npy      # SFT 部署所需;pretrain 按实际产物保留
    ├── assets/                      # 该 run 的结构配置、tokenizer 和其他必需 sidecars
    └── provenance.json              # 旧路径、最终 step、原始哈希、训练来源

全部文件实际复制,不使用软链接或硬链接。共享依赖通过相对路径索引引用;完整 checkpoints 目录应可以迁移到任意位置。S-VAE 的原始 .pt 及元数据应能无损保留/还原;policy 中的 bf16 S-VAE 与独立模型的 fp32 参数需分别记录精度,不能直接互相替换。

加载兼容性:当前 OpenWAM loader 读取完整 safetensors;拆分后的 private/weights.safetensors 不能直接冒充完整 checkpoint。真正打包时需同时提供索引加载支持,或先字节校验地还原为现有 loader 可读取的完整文件。

旧目录中的其他候选与清理边界

8 份明确的中间 checkpoint 共约 208.42 GB:run23/run39 pretrain 的 step 18000、20000,以及 run26/27/37/38 的 step 10000。完整路径:

/mnt/data/limingleyang/openwam_checkpoints/run23-2sys-pretrain-dinov3-patch2/pretrain/checkpoint_step_18000.safetensors
/mnt/data/limingleyang/openwam_checkpoints/run23-2sys-pretrain-dinov3-patch2/pretrain/checkpoint_step_20000.safetensors
/mnt/data/limingleyang/OpenWAM-private/outputs/run26-future-dit-dinov3-frozen/2026-09-15_01-09-52/checkpoint_step_10000.safetensors
/mnt/data/limingleyang/OpenWAM-private/outputs/run27-future-dit-dinov3-train-vlm/2026-09-15_01-16-59/checkpoint_step_10000.safetensors
/mnt/data/limingleyang/future_dit_vjepa_targets/run37-future-dit-vjepa-with-f0/policy/checkpoint_step_10000.safetensors
/mnt/data/limingleyang/future_dit_vjepa_targets/run38-future-dit-vjepa-without-f0/policy/checkpoint_step_10000.safetensors
/mnt/data/limingleyang/openwam_checkpoints/run39-3sys-pretrain-dinov3-patch2/pretrain/checkpoint_step_18000.safetensors
/mnt/data/limingleyang/openwam_checkpoints/run39-3sys-pretrain-dinov3-patch2/pretrain/checkpoint_step_20000.safetensors

以下 3 份属于正式 run1–run40 编号之前的历史产物,单独登记,不把它们冒认成编号后的最终版本:

/mnt/data/limingleyang/openwam_checkpoints/2026-09-06_10-33-18/checkpoint_step_19780.safetensors
/mnt/data/limingleyang/openwam_checkpoints/2026-09-02_15-25-45/checkpoint_step_16000.safetensors
/mnt/data/limingleyang/openwam_checkpoints/2026-09-04_19-43-45/checkpoint_step_19780.safetensors

另有被替代的 run21:

/mnt/data/limingleyang/openwam_artifacts/run21-superseded-finetuned-20260913_013817/checkpoints/2026-09-13_01-39-58/checkpoint_step_10690.safetensors

所有 120 个候选的完整路径见同目录 all-checkpoint-candidates.tsv。未对任何候选执行删除。后续先复制最终产物,再验证完整性与加载依赖,最后按明确清单清理中间权重、优化器状态、特征缓存及重复旧副本。原始数据集不属于中间缓存。

机器可读证据

  • checkpoint-paths.txt:正式最终 policy、S-VAE、run19 encoder 的 54 条路径。
  • all-checkpoint-candidates.tsv:120 个 checkpoint_step 候选。
  • plan.json:逐 run 来源、最终 step、目标目录和 S-VAE 依赖。
  • shared_weights.json:42 个 policy 的完整组件 SHA-256 与精确共用集合。
  • openwam-svae-audit-20260922.json:独立 S-VAE 的文件哈希、全部 tensor 哈希和保存元数据。