Model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,37 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: other
|
| 3 |
+
private: true
|
| 4 |
+
---
|
| 5 |
+
|
| 6 |
+
# species-1 (sanity-scale checkpoints)
|
| 7 |
+
|
| 8 |
+
These are sanity-scale checkpoints from a check that the training data and trainer work end to end. They are not a usable model: generations hold voice-like audio with occasional words, and intelligibility falls apart within seconds of the prompt.
|
| 9 |
+
|
| 10 |
+
The model is an autoregressive audio-token model over Mimi codes (8 codebooks at 12.5 Hz, 24 kHz mono): 107 million parameters, with a temporal transformer of 12 layers × 768 and a depth transformer of 2 layers × 256. The context is 4,096 frames. Each folder holds `model.safetensors` (float32 weights, no optimizer state), `config.json`, `checkpoint.json` (step, hours seen, manifest hash, starting checkpoint and validation) and `train.json` (the full run report). Hours seen count trained audio frames over 45,000 frames per hour, including repeats.
|
| 11 |
+
|
| 12 |
+
## Data
|
| 13 |
+
|
| 14 |
+
All four datasets are private, at these revisions:
|
| 15 |
+
|
| 16 |
+
- Pretraining: `Dialt/s2s-pre-training-yodas3-mimi@9e822fe7be6d180cf5fd2f614a4f22198e106118` and `Dialt/s2s-pre-training-podcasts-mimi@867a2ea3e4c7cb5f368182c08911dbba4c091842` (1,003.8 training hours).
|
| 17 |
+
- Mid-training: `Dialt/s2s-mid-training-podcasts-mimi@e6a16ccc573d373f0d22233a58038244ee3fce32` (264.5 training hours mixed; 244.7 with two speaker tracks).
|
| 18 |
+
- Post-training: `Dialt/s2s-conversational-podcasts-mimi@2bad89159750024a5e67cd488400d00c3e71c750` (164.5 training hours mixed; 152.6 with two speaker tracks).
|
| 19 |
+
|
| 20 |
+
The later sets are subsets of the earlier ones, and one in 20 podcast feeds or YODAS recordings is held out from every stage.
|
| 21 |
+
|
| 22 |
+
## Checkpoints
|
| 23 |
+
|
| 24 |
+
Validation figures are bits per audio second on held-out audio; lower is better. Mixed and two-stream figures are not comparable, because two separated tracks carry more information than one mix.
|
| 25 |
+
|
| 26 |
+
| Folder | Stage | Streams | Starts from | Steps × batch | Hours seen | Schedule | Validation bits/s |
|
| 27 |
+
| --- | --- | --- | --- | --- | --- | --- | --- |
|
| 28 |
+
| `pretraining` | Pretraining | 1 (mixed) | random | 2,000 × 16 | 2,513 | warmup 100, cosine to 0.1, peak 6e-4 | 785.0 |
|
| 29 |
+
| `pretraining-wsd` | Pretraining | 1 (mixed) | random | 2,000 × 16 | 2,513 | warmup 100, constant 6e-4, anneal over the last 400 | 766.9 |
|
| 30 |
+
| `mid-training-mixed` | Mid-training | 1 (mixed) | `pretraining` | 500 × 16 | 629 | warmup 50, cosine to 0.1, peak 3e-4 | 764.7 |
|
| 31 |
+
| `post-training-mixed` | Post-training | 1 (mixed) | `mid-training-mixed` | 300 × 16 | 384 | warmup 30, cosine to 0.1, peak 2e-4 | 752.9 |
|
| 32 |
+
| `mid-training-two-stream` | Mid-training | 2 (one per speaker) | `pretraining`, widened | 1,000 × 8 | 631 | warmup 100, constant 3e-4, anneal over the last 200 | 839.7 (both streams) |
|
| 33 |
+
| `post-training-two-stream` | Post-training | 2 (one per speaker) | `mid-training-two-stream` | 600 × 8 | 380 | warmup 60, constant 2e-4, anneal over the last 120 | 850.1 (both streams) |
|
| 34 |
+
|
| 35 |
+
`pretraining-wsd` is a schedule comparison on the same data and seed. The two-stream checkpoints start from `pretraining`, the cosine run.
|
| 36 |
+
|
| 37 |
+
In the two-stream checkpoints a frame holds 16 codebooks: speaker track 1's eight, then speaker track 2's. Track 1 is whichever speaker talks first in the source recording. Neither track has a conversational role yet.
|