|
Download scripts/README.md from Trelis/duplex-0: direct link, hf CLI and curl.
- Browser
- Download file 3.78 kB
-
https://huggingface.co/Trelis/duplex-0/resolve/main/scripts/README.md
- Command line
-
hf download hf://Trelis/duplex-0/scripts/README.md
-
curl -L -o README.md https://huggingface.co/Trelis/duplex-0/resolve/main/scripts/README.md
3.78 kB
| # Training scripts for duplex-0 | |
| The code that trained the checkpoints in `resplit-20261117/`. Each checkpoint's `train.json` records the SHA-256 of `run.py`, `tokens.py`, `network.py`, `kernels.py` and `checkpoint.py` under `provenance.resume_runtime.implementation_sha256`. The files here differ from the hashed ones only in package and class names, so their hashes differ. | |
| Licence: Apache 2.0 (`LICENSE` at the top of this repository). `duplex/model/NOTICE` credits the projects the attention code follows. | |
| | File | Purpose | | |
| | --- | --- | | |
| | `duplex/model/hf_release.py` | Checks the downloaded datasets and writes one training manifest per stage, with token arrays and text | | |
| | `duplex/model/run.py` | `train`, `evaluate` and `generate` | | |
| | `duplex/model/network.py`, `kernels.py` | The model | | |
| | `duplex/model/tokens.py` | Loading and batching | | |
| | `duplex/model/streaming.py` | Sampling with a key/value cache | | |
| | `duplex/model/checkpoint.py`, `transfer.py` | Checkpoints and optional initialisation | | |
| | `duplex/evaluation/loss.py`, `ngram.py` | The loss and the bigram and trigram floors | | |
| ## Setup | |
| Python 3.12 with `torch` (the runs used 2.8.0 with CUDA 12.8), `numpy`, `sentencepiece`, `pyarrow`, `safetensors` and `huggingface-hub`. Run every command from this folder. | |
| ## Data | |
| Download the datasets into one directory, one folder per dataset named as on the Hub, then build the training release: | |
| ```bash | |
| for name in pre-training-yodas3 pre-training-podcasts conversational-podcasts; do | |
| hf download --repo-type dataset Trelis/duplex-$name-mimi --local-dir raw/duplex-$name-mimi | |
| done | |
| python -m duplex.model.hf_release --raw raw --out release --stages pretraining conversational | |
| ``` | |
| The release holds `pretraining.jsonl` and `conversational.jsonl`. The `mid_training` stage of the script reads a dataset that is not public; no run of this report trained on it. The runs were built with all three stages, which gives the same training rows and also tags held-out spans with a mid-training validation set. | |
| ## Training | |
| Pre-training, shown for two text tracks. One H100, about 73 minutes. `--steps` is a cap of 20 epochs; `--eval-every` is one epoch and `--patience 2` stops the run. | |
| ```bash | |
| python -m duplex.model.run train --manifest release/pretraining.jsonl --root release \ | |
| --checkpoint out/pretraining-two/latest.pt --output out/pretraining-two/train.json \ | |
| --config baseline --text two --listen-delay 4 --text-heads per-track \ | |
| --steps 28440 --eval-every 1422 --patience 2 --lr 6e-4 --warmup-steps 100 --anneal-steps 0 \ | |
| --device cuda --precision bf16 --compile --batch-size 8 --bucket-frames 1024 --threads 8 \ | |
| --seed 20261117 --save-every 500 | |
| ``` | |
| One conversational epoch from the best pre-training checkpoint: | |
| ```bash | |
| python -m duplex.model.run train --manifest release/conversational.jsonl --root release \ | |
| --init-from out/pretraining-two/latest.best.pt \ | |
| --checkpoint out/conversational-two/latest.pt --output out/conversational-two/train.json \ | |
| --text two --steps 301 --eval-every 151 --lr 2e-4 --warmup-steps 60 \ | |
| --device cuda --precision bf16 --compile --batch-size 8 --bucket-frames 1024 --threads 8 \ | |
| --seed 20261117 --save-every 500 | |
| ``` | |
| For one text track use `--text one` and drop `--listen-delay` and `--text-heads`; for audio only use `--text none` and drop them too. An epoch is 1,116 steps for audio only and 1,373 for one text track in pre-training (229 and 278 in the conversational stage), so set `--steps` to 20 epochs and `--eval-every` to one epoch for pre-training, and `--steps` to one epoch and `--eval-every` to half an epoch for the conversational stage. | |
| The `--selection` and `--nanochat-*` options belong to earlier experiments and were not used; `--selection` needs code that is not included here. | |