Safetensors
English
RonanMcGovern commited on
Commit
d90680b
·
verified ·
1 Parent(s): da789f2

Scripts under the duplex package name; card final pass

Browse files
README.md CHANGED
@@ -16,11 +16,11 @@ This page is the training report for the runs of 17 November 2026. The training
16
 
17
  ## Training report, 17 November 2026
18
 
19
- **Summary.** Three training options were compared on the same data, seed and settings: audio only, one text track and two text tracks. All three produce speech of similar quality. Audio only scores highest on both speech benchmarks. A text track lowers first-codebook bits per second by 12% to 14%, but that did not turn into a benchmark gain.
20
 
21
  ### Model
22
 
23
- - 201 million parameters. A temporal transformer of 12 layers × 768 predicts one step per 80 ms frame, and a depth transformer of 2 layers × 256 predicts that frame's audio tokens.
24
  - Audio is Mimi tokens: 8 codebooks at 12.5 frames per second from 24 kHz audio. The model has two audio tracks, one per speaker, so a frame holds 16 codebooks.
25
  - Text, when used, is the words a speaker says, written as word-piece tokens at the frame where each word starts. The number of text tokens per frame varies, and an "advance" token ends each frame's text.
26
  - **Audio only:** no text.
@@ -30,7 +30,7 @@ This page is the training report for the runs of 17 November 2026. The training
30
  ### Data
31
 
32
  - **Pre-training:** 749 hours of English speech, 408 hours from YODAS3 and 341 hours from public podcasts.
33
- - **Conversational:** 156 hours of two-speaker podcast conversation with frequent turn-taking and a separate audio track per speaker. This is a subset of the pre-training data.
34
  - Whole recordings and podcast feeds are held out for validation and testing, about 4 hours each per source.
35
 
36
  ### Schedule
 
16
 
17
  ## Training report, 17 November 2026
18
 
19
+ **Summary.** Three training options were compared on the same data, seed and settings: audio only, one text track and two text tracks. All three produce speech of similar quality: 60 to 68 confirmed words per minute after the conversational stage, against 132.8 for real speech. A text track lowers first-codebook bits per second by 12% to 14%. One text track scores highest on intelligibility after the conversational stage and audio only scores highest on turn-taking, both by margins close to the range over seeds. The text the models write is related to the speech but is far from a transcript.
20
 
21
  ### Model
22
 
23
+ - 201 million parameters, or 225 million with two text tracks, which adds a second text output head. A temporal transformer of 12 layers × 768 predicts one step per 80 ms frame, and a depth transformer of 2 layers × 256 predicts that frame's audio tokens.
24
  - Audio is Mimi tokens: 8 codebooks at 12.5 frames per second from 24 kHz audio. The model has two audio tracks, one per speaker, so a frame holds 16 codebooks.
25
  - Text, when used, is the words a speaker says, written as word-piece tokens at the frame where each word starts. The number of text tokens per frame varies, and an "advance" token ends each frame's text.
26
  - **Audio only:** no text.
 
30
  ### Data
31
 
32
  - **Pre-training:** 749 hours of English speech, 408 hours from YODAS3 and 341 hours from public podcasts.
33
+ - **Conversational:** 156 hours of two-speaker podcast conversation with frequent turn-taking and a separate audio track per speaker. This is a subset of the pre-training data, and its dataset is not public yet.
34
  - Whole recordings and podcast feeds are held out for validation and testing, about 4 hours each per source.
35
 
36
  ### Schedule
scripts/README.md CHANGED
@@ -1,18 +1,18 @@
1
  # Training scripts for duplex-0
2
 
3
- The code that trained the checkpoints in `resplit-20261117/`. Each checkpoint's `train.json` records the SHA-256 of `run.py`, `tokens.py`, `network.py`, `kernels.py` and `checkpoint.py` under `provenance.resume_runtime.implementation_sha256`; the files here match.
4
 
5
- Licence: Apache 2.0 (`LICENSE` at the top of this repository). `research/nano_s2s/model/NOTICE` credits the projects the attention code follows.
6
 
7
  | File | Purpose |
8
  | --- | --- |
9
- | `research/nano_s2s/model/hf_release.py` | Checks the downloaded datasets and writes one training manifest per stage, with token arrays and text |
10
- | `research/nano_s2s/model/run.py` | `train`, `evaluate` and `generate` |
11
- | `research/nano_s2s/model/network.py`, `kernels.py` | The model |
12
- | `research/nano_s2s/model/tokens.py` | Loading and batching |
13
- | `research/nano_s2s/model/streaming.py` | Sampling with a key/value cache |
14
- | `research/nano_s2s/model/checkpoint.py`, `transfer.py` | Checkpoints and optional initialisation |
15
- | `research/nano_s2s/evaluation/loss.py`, `ngram.py` | The loss and the bigram and trigram floors |
16
 
17
  ## Setup
18
 
@@ -26,9 +26,11 @@ Download the datasets into one directory, one folder per dataset named as on the
26
  for name in pre-training-yodas3 pre-training-podcasts mid-training-podcasts conversational-podcasts; do
27
  hf download --repo-type dataset Trelis/duplex-$name-mimi --local-dir raw/duplex-$name-mimi
28
  done
29
- python -m research.nano_s2s.model.hf_release --raw raw --out release
30
  ```
31
 
 
 
32
  The release holds `pretraining.jsonl`, `mid_training.jsonl` and `conversational.jsonl`. No stage of this report trains on mid-training; the script still reads it to check that the sets nest.
33
 
34
  ## Training
@@ -36,7 +38,7 @@ The release holds `pretraining.jsonl`, `mid_training.jsonl` and `conversational.
36
  Pre-training, shown for two text tracks. One H100, about 73 minutes. `--steps` is a cap of 20 epochs; `--eval-every` is one epoch and `--patience 2` stops the run.
37
 
38
  ```bash
39
- python -m research.nano_s2s.model.run train --manifest release/pretraining.jsonl --root release \
40
  --checkpoint out/pretraining-two/latest.pt --output out/pretraining-two/train.json \
41
  --config baseline --text two --listen-delay 4 --text-heads per-track \
42
  --steps 28440 --eval-every 1422 --patience 2 --lr 6e-4 --warmup-steps 100 --anneal-steps 0 \
@@ -47,7 +49,7 @@ python -m research.nano_s2s.model.run train --manifest release/pretraining.jsonl
47
  One conversational epoch from the best pre-training checkpoint:
48
 
49
  ```bash
50
- python -m research.nano_s2s.model.run train --manifest release/conversational.jsonl --root release \
51
  --init-from out/pretraining-two/latest.best.pt \
52
  --checkpoint out/conversational-two/latest.pt --output out/conversational-two/train.json \
53
  --text two --steps 301 --eval-every 151 --lr 2e-4 --warmup-steps 60 \
 
1
  # Training scripts for duplex-0
2
 
3
+ The code that trained the checkpoints in `resplit-20261117/`. Each checkpoint's `train.json` records the SHA-256 of `run.py`, `tokens.py`, `network.py`, `kernels.py` and `checkpoint.py` under `provenance.resume_runtime.implementation_sha256`. The files here differ from the hashed ones only in package and class names, so their hashes differ.
4
 
5
+ Licence: Apache 2.0 (`LICENSE` at the top of this repository). `duplex/model/NOTICE` credits the projects the attention code follows.
6
 
7
  | File | Purpose |
8
  | --- | --- |
9
+ | `duplex/model/hf_release.py` | Checks the downloaded datasets and writes one training manifest per stage, with token arrays and text |
10
+ | `duplex/model/run.py` | `train`, `evaluate` and `generate` |
11
+ | `duplex/model/network.py`, `kernels.py` | The model |
12
+ | `duplex/model/tokens.py` | Loading and batching |
13
+ | `duplex/model/streaming.py` | Sampling with a key/value cache |
14
+ | `duplex/model/checkpoint.py`, `transfer.py` | Checkpoints and optional initialisation |
15
+ | `duplex/evaluation/loss.py`, `ngram.py` | The loss and the bigram and trigram floors |
16
 
17
  ## Setup
18
 
 
26
  for name in pre-training-yodas3 pre-training-podcasts mid-training-podcasts conversational-podcasts; do
27
  hf download --repo-type dataset Trelis/duplex-$name-mimi --local-dir raw/duplex-$name-mimi
28
  done
29
+ python -m duplex.model.hf_release --raw raw --out release
30
  ```
31
 
32
+ The mid-training and conversational datasets are not public yet, and the script needs all four, so this step cannot be run from outside Trelis until they are.
33
+
34
  The release holds `pretraining.jsonl`, `mid_training.jsonl` and `conversational.jsonl`. No stage of this report trains on mid-training; the script still reads it to check that the sets nest.
35
 
36
  ## Training
 
38
  Pre-training, shown for two text tracks. One H100, about 73 minutes. `--steps` is a cap of 20 epochs; `--eval-every` is one epoch and `--patience 2` stops the run.
39
 
40
  ```bash
41
+ python -m duplex.model.run train --manifest release/pretraining.jsonl --root release \
42
  --checkpoint out/pretraining-two/latest.pt --output out/pretraining-two/train.json \
43
  --config baseline --text two --listen-delay 4 --text-heads per-track \
44
  --steps 28440 --eval-every 1422 --patience 2 --lr 6e-4 --warmup-steps 100 --anneal-steps 0 \
 
49
  One conversational epoch from the best pre-training checkpoint:
50
 
51
  ```bash
52
+ python -m duplex.model.run train --manifest release/conversational.jsonl --root release \
53
  --init-from out/pretraining-two/latest.best.pt \
54
  --checkpoint out/conversational-two/latest.pt --output out/conversational-two/train.json \
55
  --text two --steps 301 --eval-every 151 --lr 2e-4 --warmup-steps 60 \
scripts/{research/nano_s2s → duplex}/evaluation/loss.py RENAMED
@@ -7,7 +7,7 @@ from torch.nn import functional as F
7
 
8
  # Moshi's equation 7 (Défossez et al. 2024): each track's codebook 0 against 1 for
9
  # each acoustic codebook, and its padding weight for the advance token. Kept equal
10
- # to research.nano_s2s.model.network, which this module must not import.
11
  SEMANTIC_WEIGHT = 100.0
12
  ADVANCE_WEIGHT = 0.5
13
  ADVANCE = 32000
 
7
 
8
  # Moshi's equation 7 (Défossez et al. 2024): each track's codebook 0 against 1 for
9
  # each acoustic codebook, and its padding weight for the advance token. Kept equal
10
+ # to duplex.model.network, which this module must not import.
11
  SEMANTIC_WEIGHT = 100.0
12
  ADVANCE_WEIGHT = 0.5
13
  ADVANCE = 32000
scripts/{research/nano_s2s → duplex}/evaluation/ngram.py RENAMED
File without changes
scripts/{research/nano_s2s → duplex}/model/NOTICE RENAMED
File without changes
scripts/{research/nano_s2s → duplex}/model/checkpoint.py RENAMED
@@ -6,7 +6,7 @@ from pathlib import Path
6
 
7
  import torch
8
 
9
- from .network import Config, NanoS2S, factorization, from_legacy
10
 
11
  # The one- and two-stream format of 14 November, before text tracks.
12
  LEGACY_FACTORIZATION = "delay-0-1-1-1-1-1-1-1"
@@ -41,7 +41,7 @@ def model_from(config, weights, stored):
41
  config = Config(**config)
42
  if stored != factorization(config):
43
  raise ValueError("Unsupported codebook factorization")
44
- model = NanoS2S(config)
45
  model.load_state_dict(weights, strict=True)
46
  return model
47
 
 
6
 
7
  import torch
8
 
9
+ from .network import Config, DuplexModel, factorization, from_legacy
10
 
11
  # The one- and two-stream format of 14 November, before text tracks.
12
  LEGACY_FACTORIZATION = "delay-0-1-1-1-1-1-1-1"
 
41
  config = Config(**config)
42
  if stored != factorization(config):
43
  raise ValueError("Unsupported codebook factorization")
44
+ model = DuplexModel(config)
45
  model.load_state_dict(weights, strict=True)
46
  return model
47
 
scripts/{research/nano_s2s → duplex}/model/hf_release.py RENAMED
@@ -1,4 +1,4 @@
1
- """Verify the published nano-s2s token datasets and write one trainer manifest per stage.
2
 
3
  Reads the private Hugging Face layout (`manifest.json`, `<run>/train.parquet`, `<run>/validation.parquet`, `<run>/test.parquet`, `<run>/report.json`,
4
  `<run>/tokens.npz`) downloaded at pinned revisions. Checks every file against `manifest.json`, every
 
1
+ """Verify the published duplex token datasets and write one trainer manifest per stage.
2
 
3
  Reads the private Hugging Face layout (`manifest.json`, `<run>/train.parquet`, `<run>/validation.parquet`, `<run>/test.parquet`, `<run>/report.json`,
4
  `<run>/tokens.npz`) downloaded at pinned revisions. Checks every file against `manifest.json`, every
scripts/{research/nano_s2s → duplex}/model/kernels.py RENAMED
@@ -104,14 +104,14 @@ if triton is not None:
104
  )
105
 
106
  # Custom ops make each kernel one opaque node for torch.compile.
107
- @torch.library.custom_op("nano_s2s::depth_attention", mutates_args=())
108
  def _attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor:
109
  q, k, v = q.contiguous(), k.contiguous(), v.contiguous()
110
  o = torch.empty_like(q)
111
  _launch(_forward, q, k, v, o)
112
  return o
113
 
114
- @torch.library.custom_op("nano_s2s::depth_attention_backward", mutates_args=())
115
  def _attention_backward(
116
  q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, do: torch.Tensor
117
  ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
 
104
  )
105
 
106
  # Custom ops make each kernel one opaque node for torch.compile.
107
+ @torch.library.custom_op("duplex::depth_attention", mutates_args=())
108
  def _attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor:
109
  q, k, v = q.contiguous(), k.contiguous(), v.contiguous()
110
  o = torch.empty_like(q)
111
  _launch(_forward, q, k, v, o)
112
  return o
113
 
114
+ @torch.library.custom_op("duplex::depth_attention_backward", mutates_args=())
115
  def _attention_backward(
116
  q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, do: torch.Tensor
117
  ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
scripts/{research/nano_s2s → duplex}/model/network.py RENAMED
@@ -356,7 +356,7 @@ def collate_text(layouts, frames, positions, device):
356
  )
357
 
358
 
359
- class NanoS2S(nn.Module):
360
  def __init__(self, config: Config):
361
  super().__init__()
362
  self.config = config
@@ -601,7 +601,7 @@ def from_legacy(config, state):
601
  streams = config.pop("streams", 1)
602
  if streams not in (1, 2):
603
  raise ValueError("Expected a one- or two-stream checkpoint")
604
- model = NanoS2S(Config(**config))
605
  c = model.config
606
  new = model.state_dict()
607
  state = dict(state)
@@ -633,6 +633,6 @@ def from_legacy(config, state):
633
 
634
  def with_text(model, text):
635
  """The same weights under another training option, such as a later stage's."""
636
- widened = NanoS2S(replace(model.config, text=text)).to(model.bos.device)
637
  widened.load_state_dict(model.state_dict(), strict=True)
638
  return widened
 
356
  )
357
 
358
 
359
+ class DuplexModel(nn.Module):
360
  def __init__(self, config: Config):
361
  super().__init__()
362
  self.config = config
 
601
  streams = config.pop("streams", 1)
602
  if streams not in (1, 2):
603
  raise ValueError("Expected a one- or two-stream checkpoint")
604
+ model = DuplexModel(Config(**config))
605
  c = model.config
606
  new = model.state_dict()
607
  state = dict(state)
 
633
 
634
  def with_text(model, text):
635
  """The same weights under another training option, such as a later stage's."""
636
+ widened = DuplexModel(replace(model.config, text=text)).to(model.bos.device)
637
  widened.load_state_dict(model.state_dict(), strict=True)
638
  return widened
scripts/{research/nano_s2s → duplex}/model/run.py RENAMED
@@ -22,9 +22,9 @@ import numpy as np
22
  import torch
23
  from torch import distributed as dist
24
 
25
- from research.nano_s2s.evaluation import loss as objective
26
- from research.nano_s2s.evaluation.loss import frame_nll, loss_and_stats, summarize
27
- from research.nano_s2s.evaluation.ngram import Floors
28
 
29
  from . import checkpoint, streaming
30
  from .network import (
@@ -32,7 +32,7 @@ from .network import (
32
  TEXT_OPTIONS,
33
  TEXT_SHIFT,
34
  Config,
35
- NanoS2S,
36
  factorization,
37
  tiny_config,
38
  with_text,
@@ -482,7 +482,7 @@ def main():
482
  layout["text_shift"] = (-args.listen_delay, TEXT_SHIFT[1])
483
  if args.text_heads:
484
  layout["text_heads"] = args.text_heads
485
- model = NanoS2S(
486
  dataclasses.replace(
487
  tiny_config(args.text)
488
  if args.config == "tiny"
 
22
  import torch
23
  from torch import distributed as dist
24
 
25
+ from duplex.evaluation import loss as objective
26
+ from duplex.evaluation.loss import frame_nll, loss_and_stats, summarize
27
+ from duplex.evaluation.ngram import Floors
28
 
29
  from . import checkpoint, streaming
30
  from .network import (
 
32
  TEXT_OPTIONS,
33
  TEXT_SHIFT,
34
  Config,
35
+ DuplexModel,
36
  factorization,
37
  tiny_config,
38
  with_text,
 
482
  layout["text_shift"] = (-args.listen_delay, TEXT_SHIFT[1])
483
  if args.text_heads:
484
  layout["text_heads"] = args.text_heads
485
+ model = DuplexModel(
486
  dataclasses.replace(
487
  tiny_config(args.text)
488
  if args.config == "tiny"
scripts/{research/nano_s2s → duplex}/model/streaming.py RENAMED
@@ -1,6 +1,6 @@
1
- """Incremental sampling for one NanoS2S sequence, for serving.
2
 
3
- `NanoS2S.generate` recomputes the whole temporal context for every sequence step.
4
  `Sampler` keeps a key/value cache instead, so a step costs its temporal positions
5
  and the depth positions it needs. It computes the same distribution as training:
6
  step s runs the audio position, then each present text track's tokens of frame s
@@ -340,7 +340,7 @@ def generate(
340
  ):
341
  """Fill the SAMPLE entries of canonical frames [frames, codebooks] in delayed order.
342
 
343
- A prefix of given frames followed by SAMPLE rows matches `NanoS2S.generate`
344
  when no text track is present. Any other mix forces some positions while
345
  sampling the rest, such as one track given and the other generated; ABSENT
346
  marks an empty track. text names the present text tracks (default: the
 
1
+ """Incremental sampling for one DuplexModel sequence, for serving.
2
 
3
+ `DuplexModel.generate` recomputes the whole temporal context for every sequence step.
4
  `Sampler` keeps a key/value cache instead, so a step costs its temporal positions
5
  and the depth positions it needs. It computes the same distribution as training:
6
  step s runs the audio position, then each present text track's tokens of frame s
 
340
  ):
341
  """Fill the SAMPLE entries of canonical frames [frames, codebooks] in delayed order.
342
 
343
+ A prefix of given frames followed by SAMPLE rows matches `DuplexModel.generate`
344
  when no text track is present. Any other mix forces some positions while
345
  sampling the rest, such as one track given and the other generated; ABSENT
346
  marks an empty track. text names the present text tracks (default: the
scripts/{research/nano_s2s → duplex}/model/tokens.py RENAMED
@@ -391,7 +391,7 @@ def windows(rows, root, split, context, text="none", shift=TEXT_SHIFT):
391
 
392
  def read_selection(path, manifest, evidence_path):
393
  """Recompute the data owner's policy rather than trusting claimed qualification."""
394
- from research.nano_s2s.data.contiguous import build_selection
395
 
396
  selection = json.loads(Path(path).read_text())
397
  evidence = [
 
391
 
392
  def read_selection(path, manifest, evidence_path):
393
  """Recompute the data owner's policy rather than trusting claimed qualification."""
394
+ from duplex.data.contiguous import build_selection
395
 
396
  selection = json.loads(Path(path).read_text())
397
  evidence = [
scripts/{research/nano_s2s → duplex}/model/transfer.py RENAMED
File without changes