code-HFD / HF_MIGRATION_HANDOFF.md
Tinghong-Ye22's picture
Add audited workspace migration handoff
6cfd407 verified
|
Raw History Blame Contribute Delete
21.1 kB

Hugging Face 工作区迁移交接

完成时间:2026-10-16T03:41:06Z Hugging Face 账户:Tinghong-Ye22

验收结果

  • 最终审计:通过。
  • 目标仓库:46 个;远端实际目标仓库:46 个。
  • 公开状态:全部公开;缺失仓库 0;意外仓库 0。
  • 可浏览代码层:40/40 个来源核验完成。
  • 完整代码与脚本快照:40/40 个来源核验完成,共 185 个归档分片、3,075.172 GiB。
  • 数据、环境与导出快照:23/23 个来源核验完成,共 126 个归档分片、2,221.278 GiB。
  • 最终再次向 Hub 查询并核对 374 个归档分片或 manifest,其中 311 个归档分片逐个比较远端 SHA-256 与本地 manifest;大小或哈希不一致 0。
  • 分别从公开仓库 code-qwen3-vl 与聚合仓库 videogen-hfd 重新下载代码和数据 manifest/快照;2 项恢复测试的 SHA-256、zstd -t 与实际 tar 解包均通过,验证本文两种恢复命令可用。
  • 本地源目录未移动、未删除,迁移只读打包;临时归档在完成远端核验前一直保留。

仓库清单

仓库 地址
code-GENMO Tinghong-Ye22/code-GENMO
code-GMR Tinghong-Ye22/code-GMR
code-GR00T-WholeBodyControl Tinghong-Ye22/code-GR00T-WholeBodyControl
code-GVHMR Tinghong-Ye22/code-GVHMR
code-HFD Tinghong-Ye22/code-HFD
code-HFD_lzk Tinghong-Ye22/code-HFD_lzk
code-HFD_swa_260910_jyGc6V Tinghong-Ye22/code-HFD_swa_260910_jyGc6V
code-HFD_swa_stream_2609111628_hzoGRo Tinghong-Ye22/code-HFD_swa_stream_2609111628_hzoGRo
code-HumanoidWAM Tinghong-Ye22/code-HumanoidWAM
code-Movens-Zero_train Tinghong-Ye22/code-Movens-Zero_train
code-PromptHMR Tinghong-Ye22/code-PromptHMR
code-Psi0 Tinghong-Ye22/code-Psi0
code-Psi0-eval Tinghong-Ye22/code-Psi0-eval
code-Psi0-yth-v15 Tinghong-Ye22/code-Psi0-yth-v15
code-Psi0-yth-v16 Tinghong-Ye22/code-Psi0-yth-v16
code-Psi0-yth-v17 Tinghong-Ye22/code-Psi0-yth-v17
code-Psi0-yth-v18 Tinghong-Ye22/code-Psi0-yth-v18
code-Psi0-yth-v19 Tinghong-Ye22/code-Psi0-yth-v19
code-Psi0-yth-v20 Tinghong-Ye22/code-Psi0-yth-v20
code-Psi0-yth-v21 Tinghong-Ye22/code-Psi0-yth-v21
code-Psi0-yth-v22 Tinghong-Ye22/code-Psi0-yth-v22
code-Psi0-yth-v23 Tinghong-Ye22/code-Psi0-yth-v23
code-Psi0-yth-v24 Tinghong-Ye22/code-Psi0-yth-v24
code-Psi0-yth-v25 Tinghong-Ye22/code-Psi0-yth-v25
code-Psi0-yth-v26 Tinghong-Ye22/code-Psi0-yth-v26
code-Psi0-yth-v8-bwd Tinghong-Ye22/code-Psi0-yth-v8-bwd
code-SONIC-MJWarp-POC Tinghong-Ye22/code-SONIC-MJWarp-POC
code-TEMOS Tinghong-Ye22/code-TEMOS
code-UniAnimate-DiT Tinghong-Ye22/code-UniAnimate-DiT
code-iclr Tinghong-Ye22/code-iclr
code-kelin Tinghong-Ye22/code-kelin
code-monitor Tinghong-Ye22/code-monitor
code-paper Tinghong-Ye22/code-paper
code-qwen3-omni Tinghong-Ye22/code-qwen3-omni
code-qwen3-tts Tinghong-Ye22/code-qwen3-tts
code-qwen3-vl Tinghong-Ye22/code-qwen3-vl
code-rewrite_motion Tinghong-Ye22/code-rewrite_motion
code-video_caption Tinghong-Ye22/code-video_caption
code-video_caption_0503 Tinghong-Ye22/code-video_caption_0503
script Tinghong-Ye22/script
videogen-hfd Tinghong-Ye22/videogen-hfd
videogen-charades_ego_v1 Tinghong-Ye22/videogen-charades_ego_v1
videogen-interleaf Tinghong-Ye22/videogen-interleaf
hfd_experiments Tinghong-Ye22/hfd_experiments
hfd_exports Tinghong-Ye22/hfd_exports
envs Tinghong-Ye22/envs

数据来源映射

本地来源 HF 仓库 分片数 压缩后大小
/mnt/data/cpfs/user_workspace/yetinghong/envs/gmr envs 1 4.172 GiB
/mnt/data/cpfs/user_workspace/yetinghong/envs/gvhmr envs 1 3.065 GiB
/mnt/data/cpfs/user_workspace/yetinghong/hfd_experiments hfd_experiments 23 453.419 GiB
/mnt/data/cpfs/user_workspace/yetinghong/hfd_exports hfd_exports 3 58.927 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/charades_ego_v1 videogen-charades_ego_v1 2 21.120 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_audio_command_continuous_office_v1 videogen-hfd 1 4.426 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_backchannel_commands_v2_short videogen-hfd 1 0.002 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_control_commands_v1 videogen-hfd 1 0.002 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_gqa_8voice_v1 videogen-hfd 1 1.364 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_gqa_v1 videogen-hfd 1 4.297 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_hesitation_pause_v1 videogen-hfd 1 3.609 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_hesitation_pause_v2 videogen-hfd 1 0.029 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_long_video_candidates_v1 videogen-hfd 4 66.042 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_swa_vqav2_v1 videogen-hfd 2 37.363 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_video_vqa_50k_v1 videogen-hfd 40 792.587 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_video_vqa_8voice_v1 videogen-hfd 1 11.625 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_video_vqa_live_v1 videogen-hfd 1 0.000 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_visual_dataset videogen-hfd 32 635.535 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_visual_dataset_build_261001 videogen-hfd 4 77.406 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_visual_dataset_recovered_261005 videogen-hfd 1 6.885 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_visual_qa videogen-hfd 2 27.812 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/hfd_vqav2_8voice_v1 videogen-hfd 1 11.023 GiB
/mnt/data/cpfs/user_workspace/yetinghong/videogen/interleaf videogen-interleaf 1 0.568 GiB

videogen-hfd 汇总所有 hfd_* 来源;envs 汇总 gmr 和 gvhmr。hfd_experiments、hfd_exports、videogen-charades_ego_v1 与 videogen-interleaf 分别使用独立仓库。请求中明确排除的 code 与 videogen 目录没有创建对应仓库。

明确排除的 code 目录:Auto-claude-code-research-in-sleep, EMDB_RICH_supplements, PromptHMR_reproduction, Psi0-v26-speed-profile-output, Psi0-yth-test, Psi0-yth-v24_test, S-NAMMD, SONIC-MJWarp-bwd, TRAM, Video-OnlineHMR, WHAM, Wan2.2, __MACOSX, agent, robot__arxiv_0901_filled。

明确排除的 videogen 目录:complex_motion_dataset、EMDB、RICH。

文件组织与恢复

代码仓库尽量提供两层内容:仓库根目录中的普通文件用于网页浏览;超大文件树受 Hub commit 限额影响时保留已经上传的普通文件,并以完整快照作为完整性依据。FULL_SNAPSHOT.tar.zst.part-* 是完整可恢复快照,FULL_SNAPSHOT.manifest.json 记录每个分片的字节数和 SHA-256。恢复某个代码或 script 仓库:

cat FULL_SNAPSHOT.tar.zst.part-* | zstd -d -c | tar -xf -

聚合数据仓库中,每个来源使用 <来源名>.tar.zst.part-* 和 <来源名>.manifest.json。恢复示例:

cat hfd_visual_dataset.tar.zst.part-* | zstd -d -c | tar -xf -

归档采用 POSIX tar、zstd level 1 和 20 GiB 二进制分片,保留权限、符号链接与空目录。每个 manifest 内含准确恢复命令和逐分片 SHA-256。

完整性边界

  • 代码和脚本完整快照只排除 .git 元数据;这是为了避免发布仓库本地 SSH 配置与密钥。项目工作文件、被 Git 忽略的文件、运行输出、缓存和用户授权公开的其他内容均包含在完整快照中。
  • 数据、环境、实验与导出归档不使用内容过滤器,按整个来源目录打包。
  • 网页可浏览层是便利副本,不作为完整性的唯一依据。完整性以归档分片、远端大小复核和 manifest SHA-256 为准。
  • 可浏览层有 82412 个路径未在 Hub 普通文件列表显示;按仓库计数及每仓库前 20 个示例为:{"code-Movens-Zero_train": {"count": 26, "sample_first_20": ["scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/L/ckpt_25000__root40_v4/inferring.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/L/ckpt_25000__root40_v4/music/music_1912/ffmpeg.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/L/ckpt_25000__root40_v4/rendering.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/L/ckpt_25000__root40_v4/speech/speech_1983/ffmpeg.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/L/ckpt_25000__root40_v4/text/text_0997/ffmpeg.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/L/ckpt_25000__root40_v4/video/video_0800/ffmpeg.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/M/ckpt_26500__root40_v4/inferring.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/M/ckpt_26500__root40_v4/music/music_1912/ffmpeg.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/M/ckpt_26500__root40_v4/rendering.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/M/ckpt_26500__root40_v4/speech/speech_1983/ffmpeg.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/M/ckpt_26500__root40_v4/text/text_0997/ffmpeg.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/M/ckpt_26500__root40_v4/video/video_0800/ffmpeg.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/S/ckpt_26500__root40_v4/inferring.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/S/ckpt_26500__root40_v4/music/music_1912/ffmpeg.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/S/ckpt_26500__root40_v4/rendering.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/S/ckpt_26500__root40_v4/speech/speech_1983/ffmpeg.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/S/ckpt_26500__root40_v4/text/text_0997/ffmpeg.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224/previews/preview_20260927_v1/S/ckpt_26500__root40_v4/video/video_0800/ffmpeg.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224_a1e-4_o1e-5_nojerk/previews/preview_20260928_1500/S/ckpt_1500__root40_v4/music/music_1912/ffmpeg.log", "scale/cleanup_reports/20260928_201511/records/groups/sml_1m_224_a1e-4_o1e-5_nojerk/previews/preview_20260928_1500/S/ckpt_1500__root40_v4/speech/speech_1983/ffmpeg.log"]}, "code-Psi0-yth-v18": {"count": 6, "sample_first_20": ["test/infer_verify_2606160122/audio/movies__F-SNBR4jhY_0007/motion.npy", "test/infer_verify_2606160122/audio/movies__F-SNBR4jhY_0007/reference_motion.mp4", "test/infer_verify_2606160122/audio_command/movies_1mzl0Zh614A_0000/motion.npy", "test/infer_verify_2606160122/audio_command/movies_1mzl0Zh614A_0000/reference_motion.mp4", "test/infer_verify_2606160122/text/g1/csv_sonic/manual/text_prompt/motion.npy", "test/infer_verify_2606160122/text/g1/csv_sonic/manual/text_prompt/reference_motion.mp4"]}, "code-Psi0-yth-v24": {"count": 9, "sample_first_20": ["runtime_indexes/omni_hidden_states_v23_oss_mnt/audio_key_index.4c5d7bdd40af65dc4309.pkl", "runtime_indexes/omni_hidden_states_v23_oss_mnt/audio_key_index.80c03d89f0fc0a151be9.pkl", "runtime_indexes/omni_hidden_states_v23_oss_mnt/audio_key_index.90836b02463d0d36d477.pkl", "runtime_indexes/omni_hidden_states_v23_oss_mnt/video_manifest_prefix.3ffb7feafcd3e1977d63.pkl", "runtime_indexes/omni_hidden_states_v23_oss_mnt/video_manifest_prefix.91cd78750baaebafa630.pkl", "runtime_indexes/omni_hidden_states_v23_oss_mnt/video_manifest_prefix.df6f85b0697637dc45f5.pkl", "runtime_indexes/omni_hidden_states_v23_oss_mnt/video_split_inverse.1114d058e1ac5d85b006.npy", "runtime_indexes/omni_hidden_states_v23_oss_mnt/video_split_inverse.611877ef3be4a4547206.npy", "runtime_indexes/omni_hidden_states_v23_oss_mnt/video_split_inverse.acf938e62e2308b8330b.npy"]}, "code-GR00T-WholeBodyControl": {"count": 48961, "sample_first_20": [".github/workflows/docs.yml", "decoupled_wbc/init.py", "decoupled_wbc/control/init.py", "decoupled_wbc/control/base/init.py", "decoupled_wbc/control/base/env.py", "decoupled_wbc/control/base/humanoid_env.py", "decoupled_wbc/control/base/policy.py", "decoupled_wbc/control/base/sensor.py", "decoupled_wbc/control/envs/init.py", "decoupled_wbc/control/envs/g1/init.py", "decoupled_wbc/control/envs/g1/g1_body.py", "decoupled_wbc/control/envs/g1/g1_env.py", "decoupled_wbc/control/envs/g1/g1_hand.py", "decoupled_wbc/control/envs/g1/sim/init.py", "decoupled_wbc/control/envs/g1/sim/base_sim.py", "decoupled_wbc/control/envs/g1/sim/image_publish_utils.py", "decoupled_wbc/control/envs/g1/sim/metric_utils.py", "decoupled_wbc/control/envs/g1/sim/robocasa_sim.py", "decoupled_wbc/control/envs/g1/sim/sim_utilts.py", "decoupled_wbc/control/envs/g1/sim/simulator_factory.py"]}, "code-Psi0-eval": {"count": 17256, "sample_first_20": [".gitignore", "CHANGELOG.md", "README.md", "artifacts/frozen_runs/posttrain_v1_trimodal_2608080815_step1000000_latest/FROZEN_BUNDLE.json", "artifacts/frozen_runs/posttrain_v1_trimodal_2608080815_step1000000_latest/argv.txt", "artifacts/frozen_runs/posttrain_v1_trimodal_2608080815_step1000000_latest/dataset_statistics.json", "artifacts/frozen_runs/posttrain_v1_trimodal_2608080815_step1000000_latest/envs.txt", "artifacts/frozen_runs/posttrain_v1_trimodal_2608080815_step1000000_latest/run_config.json", "artifacts/frozen_runs/posttrain_v1_trimodal_2608080815_step656001/FROZEN_BUNDLE.json", "artifacts/frozen_runs/posttrain_v1_trimodal_2608080815_step656001/argv.txt", "artifacts/frozen_runs/posttrain_v1_trimodal_2608080815_step656001/dataset_statistics.json", "artifacts/frozen_runs/posttrain_v1_trimodal_2608080815_step656001/envs.txt", "artifacts/frozen_runs/posttrain_v1_trimodal_2608080815_step656001/run_config.json", "artifacts/frozen_runs/posttrain_v1_trimodal_2608080815_step996001_best/FROZEN_BUNDLE.json", "artifacts/frozen_runs/posttrain_v1_trimodal_2608080815_step996001_best/argv.txt", "artifacts/frozen_runs/posttrain_v1_trimodal_2608080815_step996001_best/dataset_statistics.json", "artifacts/frozen_runs/posttrain_v1_trimodal_2608080815_step996001_best/envs.txt", "artifacts/frozen_runs/posttrain_v1_trimodal_2608080815_step996001_best/run_config.json", "artifacts/frozen_runs/posttrain_v1_trimodal_2608191043_step1001001_best/FROZEN_BUNDLE.json", "artifacts/frozen_runs/posttrain_v1_trimodal_2608191043_step1001001_best/argv.txt"]}, "code-HFD_swa_stream_2609111628_hzoGRo": {"count": 16154, "sample_first_20": [".aris/meta/events.jsonl", ".aris/traces/experiment-audit/20260911_run01/001-initial-audit.meta.json", ".aris/traces/experiment-audit/20260911_run01/001-initial-audit.request.json", ".aris/traces/experiment-audit/20260911_run01/001-initial-audit.response.md", ".aris/traces/experiment-audit/20260911_run01/002-additional-artifacts.meta.json", ".aris/traces/experiment-audit/20260911_run01/002-additional-artifacts.request.json", ".aris/traces/experiment-audit/20260911_run01/002-additional-artifacts.response.md", ".aris/traces/experiment-audit/20260911_run01/003-runtime-artifacts.meta.json", ".aris/traces/experiment-audit/20260911_run01/003-runtime-artifacts.request.json", ".aris/traces/experiment-audit/20260911_run01/003-runtime-artifacts.response.md", ".aris/traces/experiment-audit/20260911_run01/004-launch-followup.meta.json", ".aris/traces/experiment-audit/20260911_run01/004-launch-followup.request.json", ".aris/traces/experiment-audit/20260911_run01/004-launch-followup.response.md", ".aris/traces/experiment-audit/20260911_run01/received_requests.md", ".aris/traces/experiment-audit/20260911_run01/reviewer_followup.md", ".aris/traces/experiment-audit/20260911_run01/reviewer_response.json", ".aris/traces/experiment-audit/20260911_run01/reviewer_response.md", ".aris/traces/experiment-audit/20260911_run01/run.meta.json", "after_training.py", "audit_questions.py"]}}。完整列表保存在本地状态文件,这些路径仍在对应完整快照中。
  • 每个来源在其归档创建时形成独立快照;这不是所有目录同一时刻的全局原子快照。
  • 归档期间检测到源文件并发变化的快照及准确路径为:{"code-Movens-Zero_train": {"consistency": "non-atomic: source files changed during tar read", "changed_paths": ["Movens-Zero_train/scale/experiments/fresh_mxl_20261009/split_xl/backups/XL/runs/fresh_XL_pureflow_cfg_8gpu.sg.flow1000.consta.lr1.0e-03.b512.gpus8.261009120940", "Movens-Zero_train/scale/experiments/fresh_mxl_20261009/split_xl/diagnostics/fresh_XL_step76500"]}}。其 tar 流与 zstd 分片均完整结束,逐分片 SHA-256 和远端 LFS SHA-256 仍会核对;限制在于这些活动路径代表归档读取到的版本,无法声明为暂停写入时刻的原子状态。
  • 源目录中的 Hugging Face 登录令牌未写入本文档、Git 提交或远端清单。

审计依据

迁移状态与本地最终审计保存在 /mnt/data/cpfs/user_workspace/yetinghong/tmp/hf_migration_20261014/state/:

  • code_core_upload_state.json
  • code_archive_upload_state.json
  • archive_upload_state.json
  • final_audit.json
  • restore_smoke.json

最终审计从原始目录重新推导目标集合,并与账户实际模型仓库集合严格比较;随后按 manifest 对全部远端归档路径再次查询大小,并逐分片比较 Hugging Face LFS SHA-256。只有全部检查通过,才生成本交接文件。