Hugging Face's logo

jyliuAI
/
diffusion_baseline

LiteRT
TensorBoard
Core ML
ONNX
Safetensors
Model card Files
xet
 Metrics Community
diffusion_baseline / docs /05_training.md
jyliuAI's picture
jyliuAI
Add files using upload-large-folder tool
93cd35a verified
preview code
|
Raw History Blame Contribute Delete
3.16 kB

05. 训练与蒸馏

训练只在服务器运行，且必须先完成源码、数据、环境和 teacher checkpoint 校验。所有 recipe 会验证 source commit、processed dataset manifest、全局 batch、学习率、teacher family 和输出 checkpoint。

Teacher / 多步模型

直接使用已下载官方 checkpoint 的方法无需重新训练。需要 reference reproduction 或 project-owned checkpoint 时，用对应 wrapper：

bash scripts/train/flm.sh --dataset lm1b --source upstreams/flm \
  --output checkpoints/self_trained/lm1b/flm
bash scripts/train/duo.sh --dataset lm1b --source upstreams/duo \
  --output checkpoints/self_trained/lm1b/duo
bash scripts/train/mdlm.sh --dataset lm1b --source upstreams/mdlm \
  --output checkpoints/self_trained/lm1b/mdlm
bash scripts/train/candi.sh --dataset owt --source upstreams/candi \
  --output checkpoints/reference_reproduction/candi/owt
bash scripts/train/rdlm.sh --dataset lm1b --source upstreams/rdlm \
  --output checkpoints/self_trained/lm1b/rdlm

这些命令默认使用锁定 recipe。FLM 论文配置为 1,000,000 steps、global batch 512、LR 3e-4、warmup 2,500；官方发布脚本的 1,500,000 steps 只作为 drift evidence，不覆盖论文主配置。CANDI/OWT 是项目 reference recipe，生成前必须确认 model.ckpt 和 config.yaml 已被 checkpoint lock 记录。

少步蒸馏 

bash scripts/distill/fmlm.sh --dataset lm1b --teacher-checkpoint \
  checkpoints/official/flm/lm1b/model.safetensors \
  --output checkpoints/self_trained/lm1b/fmlm
bash scripts/distill/duo_dcd.sh --dataset lm1b --source upstreams/duo \
  --teacher checkpoints/reference_reproduction/flm_baselines/lm1b/lm1b_Duo.ckpt \
  --output checkpoints/reference_reproduction/duo_dcd/lm1b
bash scripts/distill/mdlm_sdtt.sh --dataset lm1b --source upstreams/sdtt \
  --teacher checkpoints/reference_reproduction/flm_baselines/lm1b/lm1b_MDLM_.ckpt \
  --output checkpoints/reference_reproduction/mdlm_sdtt/lm1b
bash scripts/distill/di4c.sh --model duo_di4c --dataset lm1b \
  --teacher-family uniform_duo \
  --teacher-checkpoint checkpoints/reference_reproduction/flm_baselines/lm1b/lm1b_Duo.ckpt \
  --output checkpoints/reference_reproduction/duo_di4c/lm1b \
  --student-init teacher \
  --upstream-override is_di4c=true

DCD/SDTT 使用 8 rounds × 10,000 steps、global batch 128、LR 6e-5、warmup 2,500。Di4C 的 LM1B/OWT 采样 checkpoint 分别是 20,000/50,000；不能把 Zenodo masked SDTT checkpoint 当作 Duo/Di4C teacher。LM1B Di4C 默认用 --student-init teacher，避免公开 OWT/GPT2-vocab student 初始化与 LM1B/BERT vocab 不兼容；--student-init scratch 仅用于实验性随机 student 初始化。

dry-run 与恢复

先在服务器检查命令而不启动训练：

python -m dlb.recipes --root "$DLB_ROOT" --recipe candi \
  --dataset owt --dry-run

默认 --resume。输出目录中会写 recipe.json、launch_argv.json、provenance.json、config.yaml、日志和 completed.json。若 checkpoint/config digest 改变，recipe 会拒绝复用旧输出；不要删除 provenance 来强行继续。