Scripts under the duplex package name; card final pass
Browse files- README.md +3 -3
- scripts/README.md +14 -12
- scripts/{research/nano_s2s → duplex}/evaluation/loss.py +1 -1
- scripts/{research/nano_s2s → duplex}/evaluation/ngram.py +0 -0
- scripts/{research/nano_s2s → duplex}/model/NOTICE +0 -0
- scripts/{research/nano_s2s → duplex}/model/checkpoint.py +2 -2
- scripts/{research/nano_s2s → duplex}/model/hf_release.py +1 -1
- scripts/{research/nano_s2s → duplex}/model/kernels.py +2 -2
- scripts/{research/nano_s2s → duplex}/model/network.py +3 -3
- scripts/{research/nano_s2s → duplex}/model/run.py +5 -5
- scripts/{research/nano_s2s → duplex}/model/streaming.py +3 -3
- scripts/{research/nano_s2s → duplex}/model/tokens.py +1 -1
- scripts/{research/nano_s2s → duplex}/model/transfer.py +0 -0
README.md
CHANGED
|
@@ -16,11 +16,11 @@ This page is the training report for the runs of 17 November 2026. The training
|
|
| 16 |
|
| 17 |
## Training report, 17 November 2026
|
| 18 |
|
| 19 |
-
**Summary.** Three training options were compared on the same data, seed and settings: audio only, one text track and two text tracks. All three produce speech of similar quality
|
| 20 |
|
| 21 |
### Model
|
| 22 |
|
| 23 |
-
- 201 million parameters. A temporal transformer of 12 layers × 768 predicts one step per 80 ms frame, and a depth transformer of 2 layers × 256 predicts that frame's audio tokens.
|
| 24 |
- Audio is Mimi tokens: 8 codebooks at 12.5 frames per second from 24 kHz audio. The model has two audio tracks, one per speaker, so a frame holds 16 codebooks.
|
| 25 |
- Text, when used, is the words a speaker says, written as word-piece tokens at the frame where each word starts. The number of text tokens per frame varies, and an "advance" token ends each frame's text.
|
| 26 |
- **Audio only:** no text.
|
|
@@ -30,7 +30,7 @@ This page is the training report for the runs of 17 November 2026. The training
|
|
| 30 |
### Data
|
| 31 |
|
| 32 |
- **Pre-training:** 749 hours of English speech, 408 hours from YODAS3 and 341 hours from public podcasts.
|
| 33 |
-
- **Conversational:** 156 hours of two-speaker podcast conversation with frequent turn-taking and a separate audio track per speaker. This is a subset of the pre-training data.
|
| 34 |
- Whole recordings and podcast feeds are held out for validation and testing, about 4 hours each per source.
|
| 35 |
|
| 36 |
### Schedule
|
|
|
|
| 16 |
|
| 17 |
## Training report, 17 November 2026
|
| 18 |
|
| 19 |
+
**Summary.** Three training options were compared on the same data, seed and settings: audio only, one text track and two text tracks. All three produce speech of similar quality: 60 to 68 confirmed words per minute after the conversational stage, against 132.8 for real speech. A text track lowers first-codebook bits per second by 12% to 14%. One text track scores highest on intelligibility after the conversational stage and audio only scores highest on turn-taking, both by margins close to the range over seeds. The text the models write is related to the speech but is far from a transcript.
|
| 20 |
|
| 21 |
### Model
|
| 22 |
|
| 23 |
+
- 201 million parameters, or 225 million with two text tracks, which adds a second text output head. A temporal transformer of 12 layers × 768 predicts one step per 80 ms frame, and a depth transformer of 2 layers × 256 predicts that frame's audio tokens.
|
| 24 |
- Audio is Mimi tokens: 8 codebooks at 12.5 frames per second from 24 kHz audio. The model has two audio tracks, one per speaker, so a frame holds 16 codebooks.
|
| 25 |
- Text, when used, is the words a speaker says, written as word-piece tokens at the frame where each word starts. The number of text tokens per frame varies, and an "advance" token ends each frame's text.
|
| 26 |
- **Audio only:** no text.
|
|
|
|
| 30 |
### Data
|
| 31 |
|
| 32 |
- **Pre-training:** 749 hours of English speech, 408 hours from YODAS3 and 341 hours from public podcasts.
|
| 33 |
+
- **Conversational:** 156 hours of two-speaker podcast conversation with frequent turn-taking and a separate audio track per speaker. This is a subset of the pre-training data, and its dataset is not public yet.
|
| 34 |
- Whole recordings and podcast feeds are held out for validation and testing, about 4 hours each per source.
|
| 35 |
|
| 36 |
### Schedule
|
scripts/README.md
CHANGED
|
@@ -1,18 +1,18 @@
|
|
| 1 |
# Training scripts for duplex-0
|
| 2 |
|
| 3 |
-
The code that trained the checkpoints in `resplit-20261117/`. Each checkpoint's `train.json` records the SHA-256 of `run.py`, `tokens.py`, `network.py`, `kernels.py` and `checkpoint.py` under `provenance.resume_runtime.implementation_sha256`
|
| 4 |
|
| 5 |
-
Licence: Apache 2.0 (`LICENSE` at the top of this repository). `
|
| 6 |
|
| 7 |
| File | Purpose |
|
| 8 |
| --- | --- |
|
| 9 |
-
| `
|
| 10 |
-
| `
|
| 11 |
-
| `
|
| 12 |
-
| `
|
| 13 |
-
| `
|
| 14 |
-
| `
|
| 15 |
-
| `
|
| 16 |
|
| 17 |
## Setup
|
| 18 |
|
|
@@ -26,9 +26,11 @@ Download the datasets into one directory, one folder per dataset named as on the
|
|
| 26 |
for name in pre-training-yodas3 pre-training-podcasts mid-training-podcasts conversational-podcasts; do
|
| 27 |
hf download --repo-type dataset Trelis/duplex-$name-mimi --local-dir raw/duplex-$name-mimi
|
| 28 |
done
|
| 29 |
-
python -m
|
| 30 |
```
|
| 31 |
|
|
|
|
|
|
|
| 32 |
The release holds `pretraining.jsonl`, `mid_training.jsonl` and `conversational.jsonl`. No stage of this report trains on mid-training; the script still reads it to check that the sets nest.
|
| 33 |
|
| 34 |
## Training
|
|
@@ -36,7 +38,7 @@ The release holds `pretraining.jsonl`, `mid_training.jsonl` and `conversational.
|
|
| 36 |
Pre-training, shown for two text tracks. One H100, about 73 minutes. `--steps` is a cap of 20 epochs; `--eval-every` is one epoch and `--patience 2` stops the run.
|
| 37 |
|
| 38 |
```bash
|
| 39 |
-
python -m
|
| 40 |
--checkpoint out/pretraining-two/latest.pt --output out/pretraining-two/train.json \
|
| 41 |
--config baseline --text two --listen-delay 4 --text-heads per-track \
|
| 42 |
--steps 28440 --eval-every 1422 --patience 2 --lr 6e-4 --warmup-steps 100 --anneal-steps 0 \
|
|
@@ -47,7 +49,7 @@ python -m research.nano_s2s.model.run train --manifest release/pretraining.jsonl
|
|
| 47 |
One conversational epoch from the best pre-training checkpoint:
|
| 48 |
|
| 49 |
```bash
|
| 50 |
-
python -m
|
| 51 |
--init-from out/pretraining-two/latest.best.pt \
|
| 52 |
--checkpoint out/conversational-two/latest.pt --output out/conversational-two/train.json \
|
| 53 |
--text two --steps 301 --eval-every 151 --lr 2e-4 --warmup-steps 60 \
|
|
|
|
| 1 |
# Training scripts for duplex-0
|
| 2 |
|
| 3 |
+
The code that trained the checkpoints in `resplit-20261117/`. Each checkpoint's `train.json` records the SHA-256 of `run.py`, `tokens.py`, `network.py`, `kernels.py` and `checkpoint.py` under `provenance.resume_runtime.implementation_sha256`. The files here differ from the hashed ones only in package and class names, so their hashes differ.
|
| 4 |
|
| 5 |
+
Licence: Apache 2.0 (`LICENSE` at the top of this repository). `duplex/model/NOTICE` credits the projects the attention code follows.
|
| 6 |
|
| 7 |
| File | Purpose |
|
| 8 |
| --- | --- |
|
| 9 |
+
| `duplex/model/hf_release.py` | Checks the downloaded datasets and writes one training manifest per stage, with token arrays and text |
|
| 10 |
+
| `duplex/model/run.py` | `train`, `evaluate` and `generate` |
|
| 11 |
+
| `duplex/model/network.py`, `kernels.py` | The model |
|
| 12 |
+
| `duplex/model/tokens.py` | Loading and batching |
|
| 13 |
+
| `duplex/model/streaming.py` | Sampling with a key/value cache |
|
| 14 |
+
| `duplex/model/checkpoint.py`, `transfer.py` | Checkpoints and optional initialisation |
|
| 15 |
+
| `duplex/evaluation/loss.py`, `ngram.py` | The loss and the bigram and trigram floors |
|
| 16 |
|
| 17 |
## Setup
|
| 18 |
|
|
|
|
| 26 |
for name in pre-training-yodas3 pre-training-podcasts mid-training-podcasts conversational-podcasts; do
|
| 27 |
hf download --repo-type dataset Trelis/duplex-$name-mimi --local-dir raw/duplex-$name-mimi
|
| 28 |
done
|
| 29 |
+
python -m duplex.model.hf_release --raw raw --out release
|
| 30 |
```
|
| 31 |
|
| 32 |
+
The mid-training and conversational datasets are not public yet, and the script needs all four, so this step cannot be run from outside Trelis until they are.
|
| 33 |
+
|
| 34 |
The release holds `pretraining.jsonl`, `mid_training.jsonl` and `conversational.jsonl`. No stage of this report trains on mid-training; the script still reads it to check that the sets nest.
|
| 35 |
|
| 36 |
## Training
|
|
|
|
| 38 |
Pre-training, shown for two text tracks. One H100, about 73 minutes. `--steps` is a cap of 20 epochs; `--eval-every` is one epoch and `--patience 2` stops the run.
|
| 39 |
|
| 40 |
```bash
|
| 41 |
+
python -m duplex.model.run train --manifest release/pretraining.jsonl --root release \
|
| 42 |
--checkpoint out/pretraining-two/latest.pt --output out/pretraining-two/train.json \
|
| 43 |
--config baseline --text two --listen-delay 4 --text-heads per-track \
|
| 44 |
--steps 28440 --eval-every 1422 --patience 2 --lr 6e-4 --warmup-steps 100 --anneal-steps 0 \
|
|
|
|
| 49 |
One conversational epoch from the best pre-training checkpoint:
|
| 50 |
|
| 51 |
```bash
|
| 52 |
+
python -m duplex.model.run train --manifest release/conversational.jsonl --root release \
|
| 53 |
--init-from out/pretraining-two/latest.best.pt \
|
| 54 |
--checkpoint out/conversational-two/latest.pt --output out/conversational-two/train.json \
|
| 55 |
--text two --steps 301 --eval-every 151 --lr 2e-4 --warmup-steps 60 \
|
scripts/{research/nano_s2s → duplex}/evaluation/loss.py
RENAMED
|
@@ -7,7 +7,7 @@ from torch.nn import functional as F
|
|
| 7 |
|
| 8 |
# Moshi's equation 7 (Défossez et al. 2024): each track's codebook 0 against 1 for
|
| 9 |
# each acoustic codebook, and its padding weight for the advance token. Kept equal
|
| 10 |
-
# to
|
| 11 |
SEMANTIC_WEIGHT = 100.0
|
| 12 |
ADVANCE_WEIGHT = 0.5
|
| 13 |
ADVANCE = 32000
|
|
|
|
| 7 |
|
| 8 |
# Moshi's equation 7 (Défossez et al. 2024): each track's codebook 0 against 1 for
|
| 9 |
# each acoustic codebook, and its padding weight for the advance token. Kept equal
|
| 10 |
+
# to duplex.model.network, which this module must not import.
|
| 11 |
SEMANTIC_WEIGHT = 100.0
|
| 12 |
ADVANCE_WEIGHT = 0.5
|
| 13 |
ADVANCE = 32000
|
scripts/{research/nano_s2s → duplex}/evaluation/ngram.py
RENAMED
|
File without changes
|
scripts/{research/nano_s2s → duplex}/model/NOTICE
RENAMED
|
File without changes
|
scripts/{research/nano_s2s → duplex}/model/checkpoint.py
RENAMED
|
@@ -6,7 +6,7 @@ from pathlib import Path
|
|
| 6 |
|
| 7 |
import torch
|
| 8 |
|
| 9 |
-
from .network import Config,
|
| 10 |
|
| 11 |
# The one- and two-stream format of 14 November, before text tracks.
|
| 12 |
LEGACY_FACTORIZATION = "delay-0-1-1-1-1-1-1-1"
|
|
@@ -41,7 +41,7 @@ def model_from(config, weights, stored):
|
|
| 41 |
config = Config(**config)
|
| 42 |
if stored != factorization(config):
|
| 43 |
raise ValueError("Unsupported codebook factorization")
|
| 44 |
-
model =
|
| 45 |
model.load_state_dict(weights, strict=True)
|
| 46 |
return model
|
| 47 |
|
|
|
|
| 6 |
|
| 7 |
import torch
|
| 8 |
|
| 9 |
+
from .network import Config, DuplexModel, factorization, from_legacy
|
| 10 |
|
| 11 |
# The one- and two-stream format of 14 November, before text tracks.
|
| 12 |
LEGACY_FACTORIZATION = "delay-0-1-1-1-1-1-1-1"
|
|
|
|
| 41 |
config = Config(**config)
|
| 42 |
if stored != factorization(config):
|
| 43 |
raise ValueError("Unsupported codebook factorization")
|
| 44 |
+
model = DuplexModel(config)
|
| 45 |
model.load_state_dict(weights, strict=True)
|
| 46 |
return model
|
| 47 |
|
scripts/{research/nano_s2s → duplex}/model/hf_release.py
RENAMED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
"""Verify the published
|
| 2 |
|
| 3 |
Reads the private Hugging Face layout (`manifest.json`, `<run>/train.parquet`, `<run>/validation.parquet`, `<run>/test.parquet`, `<run>/report.json`,
|
| 4 |
`<run>/tokens.npz`) downloaded at pinned revisions. Checks every file against `manifest.json`, every
|
|
|
|
| 1 |
+
"""Verify the published duplex token datasets and write one trainer manifest per stage.
|
| 2 |
|
| 3 |
Reads the private Hugging Face layout (`manifest.json`, `<run>/train.parquet`, `<run>/validation.parquet`, `<run>/test.parquet`, `<run>/report.json`,
|
| 4 |
`<run>/tokens.npz`) downloaded at pinned revisions. Checks every file against `manifest.json`, every
|
scripts/{research/nano_s2s → duplex}/model/kernels.py
RENAMED
|
@@ -104,14 +104,14 @@ if triton is not None:
|
|
| 104 |
)
|
| 105 |
|
| 106 |
# Custom ops make each kernel one opaque node for torch.compile.
|
| 107 |
-
@torch.library.custom_op("
|
| 108 |
def _attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor:
|
| 109 |
q, k, v = q.contiguous(), k.contiguous(), v.contiguous()
|
| 110 |
o = torch.empty_like(q)
|
| 111 |
_launch(_forward, q, k, v, o)
|
| 112 |
return o
|
| 113 |
|
| 114 |
-
@torch.library.custom_op("
|
| 115 |
def _attention_backward(
|
| 116 |
q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, do: torch.Tensor
|
| 117 |
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
|
|
|
|
| 104 |
)
|
| 105 |
|
| 106 |
# Custom ops make each kernel one opaque node for torch.compile.
|
| 107 |
+
@torch.library.custom_op("duplex::depth_attention", mutates_args=())
|
| 108 |
def _attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor:
|
| 109 |
q, k, v = q.contiguous(), k.contiguous(), v.contiguous()
|
| 110 |
o = torch.empty_like(q)
|
| 111 |
_launch(_forward, q, k, v, o)
|
| 112 |
return o
|
| 113 |
|
| 114 |
+
@torch.library.custom_op("duplex::depth_attention_backward", mutates_args=())
|
| 115 |
def _attention_backward(
|
| 116 |
q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, do: torch.Tensor
|
| 117 |
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
|
scripts/{research/nano_s2s → duplex}/model/network.py
RENAMED
|
@@ -356,7 +356,7 @@ def collate_text(layouts, frames, positions, device):
|
|
| 356 |
)
|
| 357 |
|
| 358 |
|
| 359 |
-
class
|
| 360 |
def __init__(self, config: Config):
|
| 361 |
super().__init__()
|
| 362 |
self.config = config
|
|
@@ -601,7 +601,7 @@ def from_legacy(config, state):
|
|
| 601 |
streams = config.pop("streams", 1)
|
| 602 |
if streams not in (1, 2):
|
| 603 |
raise ValueError("Expected a one- or two-stream checkpoint")
|
| 604 |
-
model =
|
| 605 |
c = model.config
|
| 606 |
new = model.state_dict()
|
| 607 |
state = dict(state)
|
|
@@ -633,6 +633,6 @@ def from_legacy(config, state):
|
|
| 633 |
|
| 634 |
def with_text(model, text):
|
| 635 |
"""The same weights under another training option, such as a later stage's."""
|
| 636 |
-
widened =
|
| 637 |
widened.load_state_dict(model.state_dict(), strict=True)
|
| 638 |
return widened
|
|
|
|
| 356 |
)
|
| 357 |
|
| 358 |
|
| 359 |
+
class DuplexModel(nn.Module):
|
| 360 |
def __init__(self, config: Config):
|
| 361 |
super().__init__()
|
| 362 |
self.config = config
|
|
|
|
| 601 |
streams = config.pop("streams", 1)
|
| 602 |
if streams not in (1, 2):
|
| 603 |
raise ValueError("Expected a one- or two-stream checkpoint")
|
| 604 |
+
model = DuplexModel(Config(**config))
|
| 605 |
c = model.config
|
| 606 |
new = model.state_dict()
|
| 607 |
state = dict(state)
|
|
|
|
| 633 |
|
| 634 |
def with_text(model, text):
|
| 635 |
"""The same weights under another training option, such as a later stage's."""
|
| 636 |
+
widened = DuplexModel(replace(model.config, text=text)).to(model.bos.device)
|
| 637 |
widened.load_state_dict(model.state_dict(), strict=True)
|
| 638 |
return widened
|
scripts/{research/nano_s2s → duplex}/model/run.py
RENAMED
|
@@ -22,9 +22,9 @@ import numpy as np
|
|
| 22 |
import torch
|
| 23 |
from torch import distributed as dist
|
| 24 |
|
| 25 |
-
from
|
| 26 |
-
from
|
| 27 |
-
from
|
| 28 |
|
| 29 |
from . import checkpoint, streaming
|
| 30 |
from .network import (
|
|
@@ -32,7 +32,7 @@ from .network import (
|
|
| 32 |
TEXT_OPTIONS,
|
| 33 |
TEXT_SHIFT,
|
| 34 |
Config,
|
| 35 |
-
|
| 36 |
factorization,
|
| 37 |
tiny_config,
|
| 38 |
with_text,
|
|
@@ -482,7 +482,7 @@ def main():
|
|
| 482 |
layout["text_shift"] = (-args.listen_delay, TEXT_SHIFT[1])
|
| 483 |
if args.text_heads:
|
| 484 |
layout["text_heads"] = args.text_heads
|
| 485 |
-
model =
|
| 486 |
dataclasses.replace(
|
| 487 |
tiny_config(args.text)
|
| 488 |
if args.config == "tiny"
|
|
|
|
| 22 |
import torch
|
| 23 |
from torch import distributed as dist
|
| 24 |
|
| 25 |
+
from duplex.evaluation import loss as objective
|
| 26 |
+
from duplex.evaluation.loss import frame_nll, loss_and_stats, summarize
|
| 27 |
+
from duplex.evaluation.ngram import Floors
|
| 28 |
|
| 29 |
from . import checkpoint, streaming
|
| 30 |
from .network import (
|
|
|
|
| 32 |
TEXT_OPTIONS,
|
| 33 |
TEXT_SHIFT,
|
| 34 |
Config,
|
| 35 |
+
DuplexModel,
|
| 36 |
factorization,
|
| 37 |
tiny_config,
|
| 38 |
with_text,
|
|
|
|
| 482 |
layout["text_shift"] = (-args.listen_delay, TEXT_SHIFT[1])
|
| 483 |
if args.text_heads:
|
| 484 |
layout["text_heads"] = args.text_heads
|
| 485 |
+
model = DuplexModel(
|
| 486 |
dataclasses.replace(
|
| 487 |
tiny_config(args.text)
|
| 488 |
if args.config == "tiny"
|
scripts/{research/nano_s2s → duplex}/model/streaming.py
RENAMED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
-
"""Incremental sampling for one
|
| 2 |
|
| 3 |
-
`
|
| 4 |
`Sampler` keeps a key/value cache instead, so a step costs its temporal positions
|
| 5 |
and the depth positions it needs. It computes the same distribution as training:
|
| 6 |
step s runs the audio position, then each present text track's tokens of frame s
|
|
@@ -340,7 +340,7 @@ def generate(
|
|
| 340 |
):
|
| 341 |
"""Fill the SAMPLE entries of canonical frames [frames, codebooks] in delayed order.
|
| 342 |
|
| 343 |
-
A prefix of given frames followed by SAMPLE rows matches `
|
| 344 |
when no text track is present. Any other mix forces some positions while
|
| 345 |
sampling the rest, such as one track given and the other generated; ABSENT
|
| 346 |
marks an empty track. text names the present text tracks (default: the
|
|
|
|
| 1 |
+
"""Incremental sampling for one DuplexModel sequence, for serving.
|
| 2 |
|
| 3 |
+
`DuplexModel.generate` recomputes the whole temporal context for every sequence step.
|
| 4 |
`Sampler` keeps a key/value cache instead, so a step costs its temporal positions
|
| 5 |
and the depth positions it needs. It computes the same distribution as training:
|
| 6 |
step s runs the audio position, then each present text track's tokens of frame s
|
|
|
|
| 340 |
):
|
| 341 |
"""Fill the SAMPLE entries of canonical frames [frames, codebooks] in delayed order.
|
| 342 |
|
| 343 |
+
A prefix of given frames followed by SAMPLE rows matches `DuplexModel.generate`
|
| 344 |
when no text track is present. Any other mix forces some positions while
|
| 345 |
sampling the rest, such as one track given and the other generated; ABSENT
|
| 346 |
marks an empty track. text names the present text tracks (default: the
|
scripts/{research/nano_s2s → duplex}/model/tokens.py
RENAMED
|
@@ -391,7 +391,7 @@ def windows(rows, root, split, context, text="none", shift=TEXT_SHIFT):
|
|
| 391 |
|
| 392 |
def read_selection(path, manifest, evidence_path):
|
| 393 |
"""Recompute the data owner's policy rather than trusting claimed qualification."""
|
| 394 |
-
from
|
| 395 |
|
| 396 |
selection = json.loads(Path(path).read_text())
|
| 397 |
evidence = [
|
|
|
|
| 391 |
|
| 392 |
def read_selection(path, manifest, evidence_path):
|
| 393 |
"""Recompute the data owner's policy rather than trusting claimed qualification."""
|
| 394 |
+
from duplex.data.contiguous import build_selection
|
| 395 |
|
| 396 |
selection = json.loads(Path(path).read_text())
|
| 397 |
evidence = [
|
scripts/{research/nano_s2s → duplex}/model/transfer.py
RENAMED
|
File without changes
|