Download scripts/README.md from Trelis/duplex-0: direct link, hf CLI and curl.
- Browser
- Download file 3.78 kB
-
https://huggingface.co/Trelis/duplex-0/resolve/main/scripts/README.md
- Command line
-
hf download hf://Trelis/duplex-0/scripts/README.md
-
curl -L -o README.md https://huggingface.co/Trelis/duplex-0/resolve/main/scripts/README.md
Training scripts for duplex-0
The code that trained the checkpoints in resplit-20261117/. Each checkpoint's train.json records the SHA-256 of run.py, tokens.py, network.py, kernels.py and checkpoint.py under provenance.resume_runtime.implementation_sha256. The files here differ from the hashed ones only in package and class names, so their hashes differ.
Licence: Apache 2.0 (LICENSE at the top of this repository). duplex/model/NOTICE credits the projects the attention code follows.
| File | Purpose |
|---|---|
duplex/model/hf_release.py |
Checks the downloaded datasets and writes one training manifest per stage, with token arrays and text |
duplex/model/run.py |
train, evaluate and generate |
duplex/model/network.py, kernels.py |
The model |
duplex/model/tokens.py |
Loading and batching |
duplex/model/streaming.py |
Sampling with a key/value cache |
duplex/model/checkpoint.py, transfer.py |
Checkpoints and optional initialisation |
duplex/evaluation/loss.py, ngram.py |
The loss and the bigram and trigram floors |
Setup
Python 3.12 with torch (the runs used 2.8.0 with CUDA 12.8), numpy, sentencepiece, pyarrow, safetensors and huggingface-hub. Run every command from this folder.
Data
Download the datasets into one directory, one folder per dataset named as on the Hub, then build the training release:
for name in pre-training-yodas3 pre-training-podcasts conversational-podcasts; do
hf download --repo-type dataset Trelis/duplex-$name-mimi --local-dir raw/duplex-$name-mimi
done
python -m duplex.model.hf_release --raw raw --out release --stages pretraining conversational
The release holds pretraining.jsonl and conversational.jsonl. The mid_training stage of the script reads a dataset that is not public; no run of this report trained on it. The runs were built with all three stages, which gives the same training rows and also tags held-out spans with a mid-training validation set.
Training
Pre-training, shown for two text tracks. One H100, about 73 minutes. --steps is a cap of 20 epochs; --eval-every is one epoch and --patience 2 stops the run.
python -m duplex.model.run train --manifest release/pretraining.jsonl --root release \
--checkpoint out/pretraining-two/latest.pt --output out/pretraining-two/train.json \
--config baseline --text two --listen-delay 4 --text-heads per-track \
--steps 28440 --eval-every 1422 --patience 2 --lr 6e-4 --warmup-steps 100 --anneal-steps 0 \
--device cuda --precision bf16 --compile --batch-size 8 --bucket-frames 1024 --threads 8 \
--seed 20261117 --save-every 500
One conversational epoch from the best pre-training checkpoint:
python -m duplex.model.run train --manifest release/conversational.jsonl --root release \
--init-from out/pretraining-two/latest.best.pt \
--checkpoint out/conversational-two/latest.pt --output out/conversational-two/train.json \
--text two --steps 301 --eval-every 151 --lr 2e-4 --warmup-steps 60 \
--device cuda --precision bf16 --compile --batch-size 8 --bucket-frames 1024 --threads 8 \
--seed 20261117 --save-every 500
For one text track use --text one and drop --listen-delay and --text-heads; for audio only use --text none and drop them too. An epoch is 1,116 steps for audio only and 1,373 for one text track in pre-training (229 and 278 in the conversational stage), so set --steps to 20 epochs and --eval-every to one epoch for pre-training, and --steps to one epoch and --eval-every to half an epoch for the conversational stage.
The --selection and --nanochat-* options belong to earlier experiments and were not used; --selection needs code that is not included here.