docs/training.md

# Training Guide / 训练指南

This document describes the reproducible training workflow for AniFileBERT.

本文档记录 AniFileBERT 的可复现训练流程。

## 1. Environment / 环境

Use `uv` for all dependency and command execution.

所有依赖和命令优先使用 `uv`。

```powershell
uv sync
uv run python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
```

Recommended GPU configuration:

推荐 GPU 配置：

- RTX 3080 class GPU or better; current release training used an RTX 5070 Ti
- batch size `512` with the schema v2 encoded-cache path on the 5070 Ti
- `bf16`/TF32 on Ada/Blackwell-class CUDA devices when available
- Rust pre-encoded caches for large schema v2 JSONL files

## 2. Dataset / 数据集

The authoritative dataset lives in the nested submodule:

权威数据集位于嵌套子模块：

```text
datasets/AnimeName/dmhy_weak.jsonl
datasets/AnimeName/dmhy_weak_char.jsonl
datasets/AnimeName/vocab.json
datasets/AnimeName/vocab.char.json
```

Current expected properties:

当前期望属性：

- rows / 行数: `759738`
- label schema / 标签 schema: v2, 37 labels
- strict BIO violations / 严格 BIO 违规: `0`
- character vocab / 字符词表: `6199`
- character coverage / 字符覆盖率: `99.9952%` with the published 6199-token char vocab

## 3. Relabel Full Dataset / 全量重标注

Use this when weak-label rules changed in `tools/dmhy_dataset.py` or `anifilebert/label_repairs.py`.

当 `tools/dmhy_dataset.py` 或 `anifilebert/label_repairs.py` 的弱标注规则改变时，使用此流程。

```powershell
uv run python -m tools.relabel_dataset_from_filenames `
  --input datasets/AnimeName/dmhy_weak.jsonl `
  --output datasets/AnimeName/dmhy_weak.relabel.jsonl `
  --manifest-output datasets/AnimeName/dmhy_weak.relabel.manifest.json `
  --vocab-output datasets/AnimeName/vocab.relabel.json `
  --base-vocab datasets/AnimeName/vocab.json `
  --max-vocab-size 8000 `
  --progress 50000
```

After checking the manifest and sample labels, replace the authoritative files:

检查 manifest 和样本标注后，再替换权威文件：

```powershell
Move-Item datasets/AnimeName/dmhy_weak.relabel.jsonl datasets/AnimeName/dmhy_weak.jsonl -Force
Move-Item datasets/AnimeName/vocab.relabel.json datasets/AnimeName/vocab.json -Force
Move-Item datasets/AnimeName/dmhy_weak.relabel.manifest.json datasets/AnimeName/dmhy_weak.manifest.json -Force
```

## 4. Convert to Character Dataset / 转换为字符数据集

The published checkpoint uses the character tokenizer.

当前发布模型使用字符级 tokenizer。

```powershell
uv run python -m tools.convert_to_char_dataset `
  --input datasets/AnimeName/dmhy_weak.jsonl `
  --output datasets/AnimeName/dmhy_weak_char.jsonl `
  --vocab-output datasets/AnimeName/vocab.char.json `
  --base-vocab datasets/AnimeName/vocab.char.json `
  --manifest-output datasets/AnimeName/dmhy_weak_char.manifest.json `
  --progress 50000
```

Use `--base-vocab` when refreshing the authoritative v2 dataset without
publishing new model embeddings; it preserves the current Android/root vocab
IDs and records any `[UNK]` coverage in the manifest.

只刷新 v2 权威数据、但不发布新 embedding 时使用 `--base-vocab`，这样可以保留
当前 Android/root vocab ID，并在 manifest 里记录 `[UNK]` 覆盖情况。

## 5. Current Schema v2 Cache-First Training / 当前 schema v2 cache-first 训练

The current published checkpoint is the repaired hard-focus schema v2 run
recorded in `reports/training_lineage.json`. The repository root artifacts are
the release surface. Local `checkpoints/`, hard-focus JSONL, and encoded-cache
directories are generated artifacts under ignored paths; rebuild or restore
them before reproducing the exact training run.

当前发布 checkpoint 是 `reports/training_lineage.json` 记录的 repaired hard-focus
schema v2 run。仓库根目录 artifact 是发布面。本地 `checkpoints/`、hard-focus JSONL
和 encoded-cache 目录都是 ignored 路径下的生成物；复现训练前需要重新生成或恢复。

```powershell
cargo run --release --manifest-path tools\encoded_dataset_cache\Cargo.toml -- `
  --input data\schema_v2_hard_focus_char_seed63.jsonl `
  --vocab-file datasets\AnimeName\vocab.char.json `
  --label-schema-file label_schema.json `
  --output-dir data\encoded_cache\schema_v2_hard_focus_char_seed63_split995_repaired_v2 `
  --max-length 128 `
  --train-split 0.995 `
  --seed 63 `
  --threads 16

.\.venv\Scripts\python.exe -m anifilebert.train --tokenizer char `
  --data-file data\schema_v2_hard_focus_char_seed63.jsonl `
  --vocab-file datasets\AnimeName\vocab.char.json `
  --encoded-cache-dir data\encoded_cache\schema_v2_hard_focus_char_seed63_split995_repaired_v2 `
  --save-dir checkpoints\ablation-schema-v2-hardfocus-cache-repaired-from-baseline-seed62-10epoch-rerun `
  --init-model-dir checkpoints\ablation-schema-v2-baseline8h4l-cache-10epoch-seed62\final `
  --epochs 10 `
  --batch-size 512 `
  --learning-rate 0.00004 `
  --warmup-steps 120 `
  --max-seq-length 128 `
  --train-split 0.995 `
  --checkpoint-steps 1000 `
  --save-total-limit 3 `
  --parse-eval-limit 2048 `
  --case-eval-file data/parser_regression_cases.json `
  --bf16 `
  --no-periodic-eval `
  --perf-log-steps 100 `
  --perf-sample-interval 1.0 `
  --seed 63 `
  --experiment-name ablation-schema-v2-hardfocus-cache-repaired-from-baseline-seed62-10epoch-rerun
```

Regenerate the encoded cache after changing the JSONL, vocab, label schema, max
length, split ratio, or seed. Do not combine `--encoded-cache-dir` with
`--extra-data-file`, `--limit-samples`, `--rebuild-vocab`, training-time
augmentation, or `--apply-label-repairs`.

修改 JSONL、vocab、label schema、max length、split ratio 或 seed 后必须重建
encoded cache。不要把 `--encoded-cache-dir` 与 `--extra-data-file`、
`--limit-samples`、`--rebuild-vocab`、训练时 augmentation 或
`--apply-label-repairs` 混用。

## 6. Optional Path And Synthetic Follow-Ups / 可选路径与合成增强

If you need standalone JSONL focus data outside the pre-encoded cache flow,
run `uv run python -m tools.build_path_prefix_dataset --input datasets/AnimeName/dmhy_weak_char.jsonl --output data/generated/path_prefix_seed105_char.jsonl`.
It expands real DMHY rows into `noise/noise/title/Season N/01 ...` path samples
with the prefix components labeled `O`. Add the following options when building
a short group boundary focus set:

```powershell
--require-group --max-group-length 4 --group-prefix-prob 1 --basename-title-prob 1
```

Use
`data/path_prefix_regression_cases.json` as the focused path-noise regression
gate when comparing baseline and path-aware checkpoints.

Optional structure/regularization experiments:

- `--model-head crf` trains a BERT emission classifier with a linear-chain CRF
  loss. The CRF loss intentionally does not hard-mask BIO transitions, because
  weak labels may contain imperfect BIO spans; BIO constraints are still applied
  during CRF Viterbi decoding.
- `--rdrop-alpha <weight>` enables two-pass R-Drop symmetric KL regularization.
  Treat CRF/R-Drop runs as experiments until they have passed both
  `data/parser_regression_cases.json` and `data/path_prefix_regression_cases.json`.

Training outputs:

训练输出：

- `checkpoints/<run>/checkpoint-*`: resumable checkpoints / 可恢复 checkpoint
- `checkpoints/<run>/final`: final Hugging Face checkpoint / 最终 checkpoint
- `final/run_metadata.json`: run configuration / 训练配置
- `final/trainer_eval_metrics.json`: seqeval metrics / token/entity 指标
- `final/parse_eval_metrics.json`: held-out parser exact-match / held-out 解析准确率
- `final/case_metrics.json`: fixed real-world case regression / 固定真实 case 回归
- `final/perf_metrics.json`: training throughput/GPU telemetry when enabled / 启用时记录训练吞吐和 GPU 采样
- TensorBoard logs unless `--no-tensorboard` is set / 默认写 TensorBoard

## 7. Quality Gates / 质量门槛

Use hard-case fine-tuning only after a specific real-world failure pattern has
been confirmed, fixed in the weak labels, and added to
`data/parser_regression_cases.json`.

只有在确认某类真实失败样式、修复弱标注并加入
`data/parser_regression_cases.json` 后，才使用困难样本微调。

The default quality gate is model-led parsing:

默认质量门槛以模型主导解析为准：

- fixed regression `model_only >= 85%`
- held-out parse `model_only >= 75%`
- `normalized_only` is the default thin runtime metric
- structural filename assists are not part of training or release metrics

- 固定回归 `model_only >= 85%`
- held-out 解析 `model_only >= 75%`
- `normalized_only` 是默认薄层运行时指标
- 结构化文件名辅助不属于训练或发布指标

## 8. Publish to Repository Root / 发布到仓库根目录

The repository root is the Hugging Face checkpoint surface.

仓库根目录就是 Hugging Face checkpoint 发布面。

```powershell
$final = "checkpoints/ablation-schema-v2-hardfocus-cache-repaired-from-baseline-seed62-10epoch-rerun/final"
$releaseVocab = "datasets/AnimeName/vocab.char.json"
# For a path-aware run trained with data/generated/vocab.char.path.json:
# $releaseVocab = "data/generated/vocab.char.path.json"
Copy-Item "$final/config.json" . -Force
Copy-Item "$final/model.safetensors" . -Force
Copy-Item "$final/tokenizer_config.json" . -Force
Copy-Item "$final/training_args.bin" . -Force
Copy-Item "$final/vocab.json" . -Force
New-Item -ItemType Directory -Path reports -Force | Out-Null
Copy-Item "$final/run_metadata.json" reports/run_metadata.json -Force
Copy-Item "$final/trainer_eval_metrics.json" reports/trainer_eval_metrics.json -Force
Copy-Item "$final/parse_eval_metrics.json" reports/parse_eval_metrics.json -Force
Copy-Item "$final/case_metrics.json" reports/case_metrics.json -Force
Copy-Item "$final/perf_metrics.json" reports/perf_metrics.json -Force
Copy-Item $releaseVocab .\vocab.char.json -Force
```

Then export ONNX:

然后导出 ONNX：

```powershell
uv run python -m tools.export_onnx --model-dir . --output exports/anime_filename_parser.onnx --max-length 128
```

## 9. Validation Checklist / 验证清单

Run these before committing:

提交前执行：

```powershell
uv run python -m compileall -q anifilebert tools
uv run python -m tools.evaluate_parser_cases --model-dir . --case-file data/parser_regression_cases.json --output reports/case_metrics.json
uv run python -m anifilebert.inference --model-dir . "[GM-Team][国漫][神印王座][Throne of Seal][2022][200][AVC][GB][1080P].mp4"
uv run python -m tools.onnx_inference "[YYDM&VCB-Studio] Shinsekai Yori [NCED02][Ma10p_1080p][x265_flac].mkv"
uv run python -m tools.benchmark_inference --model-dir . --onnx exports/anime_filename_parser.onnx --case-file data/parser_regression_cases.json --repeat 20 --warmup 20 --torch-threads 1 --ort-threads 1 --output reports/benchmark_results.json
```

## 10. Recover Interrupted Work / 恢复中断训练现场

When Codex/session history is lost, use the recovery helper to reconstruct
what was running, whether it finished, and where to resume.

当 Codex/会话历史丢失时，使用恢复脚本快速还原“当时在跑什么、是否跑完、从哪里续跑”。

Local latest run:

```powershell
uv run python -m tools.recover_training_context
```

Specific local run:

```powershell
uv run python -m tools.recover_training_context --run ablation-schema-v2-hardfocus-cache-repaired-from-baseline-seed62-10epoch-rerun
```

Local + remote worker (RTX 5070 Ti):

```powershell
uv run python -m tools.recover_training_context `
  --remote-host adqew@192.168.63.157 `
  --remote-repo "C:\WorkSpace\Python\AniFileBERT"
```

Machine-readable JSON report:

```powershell
uv run python -m tools.recover_training_context `
  --remote-host adqew@192.168.63.157 `
  --format json `
  --output reports/recovery_latest.json
```

The script inspects:

- latest/selected run under `checkpoints/`
- latest `checkpoint-*` step and `trainer_state.json`
- `final/run_metadata.json`
- `logs/<run>/combined.log` tail
- remote run state via SSH (optional)

If the run is incomplete, it prints a resume hint with `--resume-from-checkpoint`.

## 11. Git and LFS Order / Git 与 LFS 顺序

If the dataset submodule changed:

如果数据集子模块有变动：

```powershell
git -C datasets/AnimeName add dmhy_weak.jsonl dmhy_weak.manifest.json dmhy_weak_char.jsonl dmhy_weak_char.manifest.json vocab.json vocab.char.json
git -C datasets/AnimeName commit -m "Update anime filename labels"
git -C datasets/AnimeName lfs push origin main --all
git -C datasets/AnimeName push origin main
```

Then commit the model repo:

再提交模型仓库：

```powershell
git add README.md docs/maintenance.md docs/android.md docs/training.md docs/onnx.md `
  config.json model.safetensors tokenizer_config.json training_args.bin vocab.json vocab.char.json `
  exports/anime_filename_parser.onnx exports/anime_filename_parser.metadata.json `
  reports anifilebert tools data/parser_regression_cases.json datasets/AnimeName
git commit -m "Update AniFileBERT model and documentation"
git lfs push origin main --all
git push origin main
```