# Training scripts for duplex-0 The code that trained the checkpoints in `resplit-20261117/`. Each checkpoint's `train.json` records the SHA-256 of `run.py`, `tokens.py`, `network.py`, `kernels.py` and `checkpoint.py` under `provenance.resume_runtime.implementation_sha256`. The files here differ from the hashed ones only in package and class names, so their hashes differ. Licence: Apache 2.0 (`LICENSE` at the top of this repository). `duplex/model/NOTICE` credits the projects the attention code follows. | File | Purpose | | --- | --- | | `duplex/model/hf_release.py` | Checks the downloaded datasets and writes one training manifest per stage, with token arrays and text | | `duplex/model/run.py` | `train`, `evaluate` and `generate` | | `duplex/model/network.py`, `kernels.py` | The model | | `duplex/model/tokens.py` | Loading and batching | | `duplex/model/streaming.py` | Sampling with a key/value cache | | `duplex/model/checkpoint.py`, `transfer.py` | Checkpoints and optional initialisation | | `duplex/evaluation/loss.py`, `ngram.py` | The loss and the bigram and trigram floors | ## Setup Python 3.12 with `torch` (the runs used 2.8.0 with CUDA 12.8), `numpy`, `sentencepiece`, `pyarrow`, `safetensors` and `huggingface-hub`. Run every command from this folder. ## Data Download the datasets into one directory, one folder per dataset named as on the Hub, then build the training release: ```bash for name in pre-training-yodas3 pre-training-podcasts conversational-podcasts; do hf download --repo-type dataset Trelis/duplex-$name-mimi --local-dir raw/duplex-$name-mimi done python -m duplex.model.hf_release --raw raw --out release --stages pretraining conversational ``` The release holds `pretraining.jsonl` and `conversational.jsonl`. The `mid_training` stage of the script reads a dataset that is not public; no run of this report trained on it. The runs were built with all three stages, which gives the same training rows and also tags held-out spans with a mid-training validation set. ## Training Pre-training, shown for two text tracks. One H100, about 73 minutes. `--steps` is a cap of 20 epochs; `--eval-every` is one epoch and `--patience 2` stops the run. ```bash python -m duplex.model.run train --manifest release/pretraining.jsonl --root release \ --checkpoint out/pretraining-two/latest.pt --output out/pretraining-two/train.json \ --config baseline --text two --listen-delay 4 --text-heads per-track \ --steps 28440 --eval-every 1422 --patience 2 --lr 6e-4 --warmup-steps 100 --anneal-steps 0 \ --device cuda --precision bf16 --compile --batch-size 8 --bucket-frames 1024 --threads 8 \ --seed 20261117 --save-every 500 ``` One conversational epoch from the best pre-training checkpoint: ```bash python -m duplex.model.run train --manifest release/conversational.jsonl --root release \ --init-from out/pretraining-two/latest.best.pt \ --checkpoint out/conversational-two/latest.pt --output out/conversational-two/train.json \ --text two --steps 301 --eval-every 151 --lr 2e-4 --warmup-steps 60 \ --device cuda --precision bf16 --compile --batch-size 8 --bucket-frames 1024 --threads 8 \ --seed 20261117 --save-every 500 ``` For one text track use `--text one` and drop `--listen-delay` and `--text-heads`; for audio only use `--text none` and drop them too. An epoch is 1,116 steps for audio only and 1,373 for one text track in pre-training (229 and 278 in the conversational stage), so set `--steps` to 20 epochs and `--eval-every` to one epoch for pre-training, and `--steps` to one epoch and `--eval-every` to half an epoch for the conversational stage. The `--selection` and `--nanochat-*` options belong to earlier experiments and were not used; `--selection` needs code that is not included here.