Safetensors
English
File size: 3,775 Bytes
d52c89c
 
d90680b
d52c89c
d90680b
d52c89c
 
 
d90680b
 
 
 
 
 
 
d52c89c
 
 
 
 
 
 
 
 
 
f686fdd
d52c89c
 
f686fdd
d52c89c
 
f686fdd
d52c89c
 
 
 
 
 
d90680b
d52c89c
 
 
 
 
 
 
 
 
 
d90680b
d52c89c
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
# Training scripts for duplex-0

The code that trained the checkpoints in `resplit-20261117/`. Each checkpoint's `train.json` records the SHA-256 of `run.py`, `tokens.py`, `network.py`, `kernels.py` and `checkpoint.py` under `provenance.resume_runtime.implementation_sha256`. The files here differ from the hashed ones only in package and class names, so their hashes differ.

Licence: Apache 2.0 (`LICENSE` at the top of this repository). `duplex/model/NOTICE` credits the projects the attention code follows.

| File | Purpose |
| --- | --- |
| `duplex/model/hf_release.py` | Checks the downloaded datasets and writes one training manifest per stage, with token arrays and text |
| `duplex/model/run.py` | `train`, `evaluate` and `generate` |
| `duplex/model/network.py`, `kernels.py` | The model |
| `duplex/model/tokens.py` | Loading and batching |
| `duplex/model/streaming.py` | Sampling with a key/value cache |
| `duplex/model/checkpoint.py`, `transfer.py` | Checkpoints and optional initialisation |
| `duplex/evaluation/loss.py`, `ngram.py` | The loss and the bigram and trigram floors |

## Setup

Python 3.12 with `torch` (the runs used 2.8.0 with CUDA 12.8), `numpy`, `sentencepiece`, `pyarrow`, `safetensors` and `huggingface-hub`. Run every command from this folder.

## Data

Download the datasets into one directory, one folder per dataset named as on the Hub, then build the training release:

```bash
for name in pre-training-yodas3 pre-training-podcasts conversational-podcasts; do
  hf download --repo-type dataset Trelis/duplex-$name-mimi --local-dir raw/duplex-$name-mimi
done
python -m duplex.model.hf_release --raw raw --out release --stages pretraining conversational
```

The release holds `pretraining.jsonl` and `conversational.jsonl`. The `mid_training` stage of the script reads a dataset that is not public; no run of this report trained on it. The runs were built with all three stages, which gives the same training rows and also tags held-out spans with a mid-training validation set.

## Training

Pre-training, shown for two text tracks. One H100, about 73 minutes. `--steps` is a cap of 20 epochs; `--eval-every` is one epoch and `--patience 2` stops the run.

```bash
python -m duplex.model.run train --manifest release/pretraining.jsonl --root release \
  --checkpoint out/pretraining-two/latest.pt --output out/pretraining-two/train.json \
  --config baseline --text two --listen-delay 4 --text-heads per-track \
  --steps 28440 --eval-every 1422 --patience 2 --lr 6e-4 --warmup-steps 100 --anneal-steps 0 \
  --device cuda --precision bf16 --compile --batch-size 8 --bucket-frames 1024 --threads 8 \
  --seed 20261117 --save-every 500
```

One conversational epoch from the best pre-training checkpoint:

```bash
python -m duplex.model.run train --manifest release/conversational.jsonl --root release \
  --init-from out/pretraining-two/latest.best.pt \
  --checkpoint out/conversational-two/latest.pt --output out/conversational-two/train.json \
  --text two --steps 301 --eval-every 151 --lr 2e-4 --warmup-steps 60 \
  --device cuda --precision bf16 --compile --batch-size 8 --bucket-frames 1024 --threads 8 \
  --seed 20261117 --save-every 500
```

For one text track use `--text one` and drop `--listen-delay` and `--text-heads`; for audio only use `--text none` and drop them too. An epoch is 1,116 steps for audio only and 1,373 for one text track in pre-training (229 and 278 in the conversational stage), so set `--steps` to 20 epochs and `--eval-every` to one epoch for pre-training, and `--steps` to one epoch and `--eval-every` to half an epoch for the conversational stage.

The `--selection` and `--nanochat-*` options belong to earlier experiments and were not used; `--selection` needs code that is not included here.