Card: text accuracy against speech
Browse files
README.md
CHANGED
|
@@ -103,6 +103,16 @@ After the conversational stage, the text track of the speaker the model speaks a
|
|
| 103 |
|
| 104 |
Both models write more while the caller is quiet, so they follow turns. Both still write more words than a real speaker does.
|
| 105 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
### Samples
|
| 107 |
|
| 108 |
Each clip is 30 seconds: 10 seconds of a real held-out conversation as the prompt, then 20 seconds that the model generates for both speakers (seed 1, temperature 1, top-k 250, from the checkpoint after the conversational stage). The text-track models are given the true prompt text. The "real" clip is the true continuation. Both tracks are mixed into one file, and all audio is decoded from Mimi tokens.
|
|
|
|
| 103 |
|
| 104 |
Both models write more while the caller is quiet, so they follow turns. Both still write more words than a real speaker does.
|
| 105 |
|
| 106 |
+
How well the written text matches the speech was measured on the same runs (60 seconds generated per clip, 72 clips pooled). The model's own text is compared with a transcript of the audio it generated on that track. The caller's text, which only the two-text-track model writes, is compared with the caller's true words, so it measures speech recognition. Word error rate is substitutions, deletions and insertions divided by reference words; lower is better and 0 is a perfect match.
|
| 107 |
+
|
| 108 |
+
| Text | Option | Pre-training | Conversational | Unrelated text |
|
| 109 |
+
| --- | --- | --- | --- | --- |
|
| 110 |
+
| Own speech | One text track | 0.90 | 0.78 | 1.34 to 1.43 |
|
| 111 |
+
| Own speech | Two text tracks | 1.75 | 1.29 | 1.72 to 2.00 |
|
| 112 |
+
| Caller's speech | Two text tracks | 0.84 | 0.71 | 1.18 to 1.23 |
|
| 113 |
+
|
| 114 |
+
"Unrelated text" pairs each clip's written text with the reference of a different span. The transcriber used for the own-speech reference scores 0.11 to 0.18 against the true words on real audio. The written text is related to the audio and improves with the conversational stage, but it is far from a transcript: after the conversational stage the caller's text gets 45% of the caller's words right, and most errors on the model's own text are extra words.
|
| 115 |
+
|
| 116 |
### Samples
|
| 117 |
|
| 118 |
Each clip is 30 seconds: 10 seconds of a real held-out conversation as the prompt, then 20 seconds that the model generates for both speakers (seed 1, temperature 1, top-k 250, from the checkpoint after the conversational stage). The text-track models are given the true prompt text. The "real" clip is the true continuation. Both tracks are mixed into one file, and all audio is decoded from Mimi tokens.
|