Safetensors
English
duplex-0 / scripts /README.md
RonanMcGovern's picture
Data step builds from the public datasets
f686fdd verified
|
Raw History Blame Contribute Delete
3.78 kB

Training scripts for duplex-0

The code that trained the checkpoints in resplit-20261117/. Each checkpoint's train.json records the SHA-256 of run.py, tokens.py, network.py, kernels.py and checkpoint.py under provenance.resume_runtime.implementation_sha256. The files here differ from the hashed ones only in package and class names, so their hashes differ.

Licence: Apache 2.0 (LICENSE at the top of this repository). duplex/model/NOTICE credits the projects the attention code follows.

File Purpose
duplex/model/hf_release.py Checks the downloaded datasets and writes one training manifest per stage, with token arrays and text
duplex/model/run.py train, evaluate and generate
duplex/model/network.py, kernels.py The model
duplex/model/tokens.py Loading and batching
duplex/model/streaming.py Sampling with a key/value cache
duplex/model/checkpoint.py, transfer.py Checkpoints and optional initialisation
duplex/evaluation/loss.py, ngram.py The loss and the bigram and trigram floors

Setup

Python 3.12 with torch (the runs used 2.8.0 with CUDA 12.8), numpy, sentencepiece, pyarrow, safetensors and huggingface-hub. Run every command from this folder.

Data

Download the datasets into one directory, one folder per dataset named as on the Hub, then build the training release:

for name in pre-training-yodas3 pre-training-podcasts conversational-podcasts; do
  hf download --repo-type dataset Trelis/duplex-$name-mimi --local-dir raw/duplex-$name-mimi
done
python -m duplex.model.hf_release --raw raw --out release --stages pretraining conversational

The release holds pretraining.jsonl and conversational.jsonl. The mid_training stage of the script reads a dataset that is not public; no run of this report trained on it. The runs were built with all three stages, which gives the same training rows and also tags held-out spans with a mid-training validation set.

Training

Pre-training, shown for two text tracks. One H100, about 73 minutes. --steps is a cap of 20 epochs; --eval-every is one epoch and --patience 2 stops the run.

python -m duplex.model.run train --manifest release/pretraining.jsonl --root release \
  --checkpoint out/pretraining-two/latest.pt --output out/pretraining-two/train.json \
  --config baseline --text two --listen-delay 4 --text-heads per-track \
  --steps 28440 --eval-every 1422 --patience 2 --lr 6e-4 --warmup-steps 100 --anneal-steps 0 \
  --device cuda --precision bf16 --compile --batch-size 8 --bucket-frames 1024 --threads 8 \
  --seed 20261117 --save-every 500

One conversational epoch from the best pre-training checkpoint:

python -m duplex.model.run train --manifest release/conversational.jsonl --root release \
  --init-from out/pretraining-two/latest.best.pt \
  --checkpoint out/conversational-two/latest.pt --output out/conversational-two/train.json \
  --text two --steps 301 --eval-every 151 --lr 2e-4 --warmup-steps 60 \
  --device cuda --precision bf16 --compile --batch-size 8 --bucket-frames 1024 --threads 8 \
  --seed 20261117 --save-every 500

For one text track use --text one and drop --listen-delay and --text-heads; for audio only use --text none and drop them too. An epoch is 1,116 steps for audio only and 1,373 for one text track in pre-training (229 and 278 in the conversational stage), so set --steps to 20 epochs and --eval-every to one epoch for pre-training, and --steps to one epoch and --eval-every to half an epoch for the conversational stage.

The --selection and --nanochat-* options belong to earlier experiments and were not used; --selection needs code that is not included here.