Safetensors
English
RonanMcGovern commited on
Commit
a9be00f
·
verified ·
1 Parent(s): f0d023f

Model card

Browse files
Files changed (1) hide show
  1. README.md +37 -0
README.md ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ private: true
4
+ ---
5
+
6
+ # species-1 (sanity-scale checkpoints)
7
+
8
+ These are sanity-scale checkpoints from a check that the training data and trainer work end to end. They are not a usable model: generations hold voice-like audio with occasional words, and intelligibility falls apart within seconds of the prompt.
9
+
10
+ The model is an autoregressive audio-token model over Mimi codes (8 codebooks at 12.5 Hz, 24 kHz mono): 107 million parameters, with a temporal transformer of 12 layers × 768 and a depth transformer of 2 layers × 256. The context is 4,096 frames. Each folder holds `model.safetensors` (float32 weights, no optimizer state), `config.json`, `checkpoint.json` (step, hours seen, manifest hash, starting checkpoint and validation) and `train.json` (the full run report). Hours seen count trained audio frames over 45,000 frames per hour, including repeats.
11
+
12
+ ## Data
13
+
14
+ All four datasets are private, at these revisions:
15
+
16
+ - Pretraining: `Dialt/s2s-pre-training-yodas3-mimi@9e822fe7be6d180cf5fd2f614a4f22198e106118` and `Dialt/s2s-pre-training-podcasts-mimi@867a2ea3e4c7cb5f368182c08911dbba4c091842` (1,003.8 training hours).
17
+ - Mid-training: `Dialt/s2s-mid-training-podcasts-mimi@e6a16ccc573d373f0d22233a58038244ee3fce32` (264.5 training hours mixed; 244.7 with two speaker tracks).
18
+ - Post-training: `Dialt/s2s-conversational-podcasts-mimi@2bad89159750024a5e67cd488400d00c3e71c750` (164.5 training hours mixed; 152.6 with two speaker tracks).
19
+
20
+ The later sets are subsets of the earlier ones, and one in 20 podcast feeds or YODAS recordings is held out from every stage.
21
+
22
+ ## Checkpoints
23
+
24
+ Validation figures are bits per audio second on held-out audio; lower is better. Mixed and two-stream figures are not comparable, because two separated tracks carry more information than one mix.
25
+
26
+ | Folder | Stage | Streams | Starts from | Steps × batch | Hours seen | Schedule | Validation bits/s |
27
+ | --- | --- | --- | --- | --- | --- | --- | --- |
28
+ | `pretraining` | Pretraining | 1 (mixed) | random | 2,000 × 16 | 2,513 | warmup 100, cosine to 0.1, peak 6e-4 | 785.0 |
29
+ | `pretraining-wsd` | Pretraining | 1 (mixed) | random | 2,000 × 16 | 2,513 | warmup 100, constant 6e-4, anneal over the last 400 | 766.9 |
30
+ | `mid-training-mixed` | Mid-training | 1 (mixed) | `pretraining` | 500 × 16 | 629 | warmup 50, cosine to 0.1, peak 3e-4 | 764.7 |
31
+ | `post-training-mixed` | Post-training | 1 (mixed) | `mid-training-mixed` | 300 × 16 | 384 | warmup 30, cosine to 0.1, peak 2e-4 | 752.9 |
32
+ | `mid-training-two-stream` | Mid-training | 2 (one per speaker) | `pretraining`, widened | 1,000 × 8 | 631 | warmup 100, constant 3e-4, anneal over the last 200 | 839.7 (both streams) |
33
+ | `post-training-two-stream` | Post-training | 2 (one per speaker) | `mid-training-two-stream` | 600 × 8 | 380 | warmup 60, constant 2e-4, anneal over the last 120 | 850.1 (both streams) |
34
+
35
+ `pretraining-wsd` is a schedule comparison on the same data and seed. The two-stream checkpoints start from `pretraining`, the cosine run.
36
+
37
+ In the two-stream checkpoints a frame holds 16 codebooks: speaker track 1's eight, then speaker track 2's. Track 1 is whichever speaker talks first in the source recording. Neither track has a conversational role yet.