card: encoder ladder (3 window lengths), new default numbers
Browse files
README.md
CHANGED
|
@@ -21,6 +21,23 @@ Every encoder here holds the checkpoint's **exact** five-value weights — as fp
|
|
| 21 |
`constexpr_sparse_to_dense`). No re-quantization noise is added, and all five produce identical transcripts. The decoder
|
| 22 |
and joint are re-exported from the checkpoint's int6 tables; preprocessor and vocabulary are the v3 ones.
|
| 23 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
## Usage (FluidAudio, merged in [#980](https://github.com/FluidInference/FluidAudio/pull/980))
|
| 25 |
|
| 26 |
```swift
|
|
@@ -45,7 +62,8 @@ encoder on the Neural Engine. All five are exact and give the same transcripts.
|
|
| 45 |
|
| 46 |
| File | Size | Encoding | ANE | ANE RTFx | GPU |
|
| 47 |
|---|---:|---|---:|---:|---|
|
| 48 |
-
| **`Encoder.mlmodelc`** (default) | **
|
|
|
|
| 49 |
| `Encoder_sparse-g4.mlmodelc` | 246 MB | sparse mask + 4-bit palette per 4 rows | 24.3 ms | 140× | same load caveat |
|
| 50 |
| `Encoder_sparse-g1.mlmodelc` | 176 MB | sparse mask + 2-bit palette per row | 70 ms | ~70× | same load caveat |
|
| 51 |
| `Encoder_lut6.mlmodelc` | 470 MB | dense 6-bit palette per 8 rows | 18.6 ms | 155× | 16 ms, 0.6 s load |
|
|
@@ -71,7 +89,7 @@ corpus-level (total edit distance over total reference words); RTFx = total audi
|
|
| 71 |
|---|---|---|---|
|
| 72 |
| test-clean (2620 files) WER | 2.27 % | **2.13 %** | 2.47 % |
|
| 73 |
| test-other (2939 files) WER | 4.12 % | **3.81 %** | 4.62 % |
|
| 74 |
-
| test-clean RTFx | 149–152× | 151× | **
|
| 75 |
| test-other RTFx | 138× | 142× | **146×** |
|
| 76 |
|
| 77 |
v3 is the more accurate model on LibriSpeech by 0.20 / 0.50 points, reproducing the upstream card's own gap to its
|
|
@@ -93,7 +111,8 @@ excludes model load). Reference = the concatenated Earnings-22 chunk transcripts
|
|
| 93 |
| v3 | 10.9 s | 331× | 16.5 % |
|
| 94 |
| Ultra | 7.7 s | 469× | **13.5 %** |
|
| 95 |
| Redux | 14.2 s | 254× | 14.8 % |
|
| 96 |
-
| Phonon-2
|
|
|
|
| 97 |
| Phonon-2 `Encoder_lut6` (470 MB) | 7.4 s | 486× | 17.2 % |
|
| 98 |
| Phonon-2 `Encoder_sparse-g4` (246 MB) | 9.0 s | 399× | 17.2 % |
|
| 99 |
| Phonon-2 `Encoder_sparse-g1` (176 MB) | 23.4 s | 154× | 17.2 % |
|
|
|
|
| 21 |
`constexpr_sparse_to_dense`). No re-quantization noise is added, and all five produce identical transcripts. The decoder
|
| 22 |
and joint are re-exported from the checkpoint's int6 tables; preprocessor and vocabulary are the v3 ones.
|
| 23 |
|
| 24 |
+
## Encoder ladder (`Encoder.mlmodelc`, since [#1019](https://github.com/FluidInference/FluidAudio/pull/1019))
|
| 25 |
+
|
| 26 |
+
`Encoder.mlmodelc` is a Core ML **multifunction** model: three encoder programs — `enc_5040ms`, `enc_10160ms`,
|
| 27 |
+
`enc_15000ms` (5.04 / 10.16 / 15 s windows, 63 / 126 / 188 encoder frames) — over one shared weight blob (341 MB for the
|
| 28 |
+
three). FluidAudio ≥ the #1019 merge loads every function and runs each window on the smallest one that holds it; older
|
| 29 |
+
FluidAudio versions load the default function (15 s) and behave as before. The three lengths are the Neural Engine's
|
| 30 |
+
64-frame tiles: 11–15 s windows all cost ~19 ms, 10 s costs 12 ms, 5 s about 6 ms, so a short clip stops paying for a
|
| 31 |
+
15 s pass and long recordings are cut at pauses with no overlap. Each function computes the same masked encoder;
|
| 32 |
+
transcripts do not change. First Neural Engine load compiles each function (~55 s each on an M5 Pro, cached after).
|
| 33 |
+
|
| 34 |
+
| | Encoder | 1 h Earnings-22 (M5 Pro) | LibriSpeech test-clean RTFx |
|
| 35 |
+
|---|---|---:|---:|
|
| 36 |
+
| FluidAudio with the ladder | `Encoder.mlmodelc` (341 MB) | **5.55 s / 649×, WER 16.7 %** | **193×** (WER 2.47 %) |
|
| 37 |
+
| previous default | `Encoder_sparse-g8.mlmodelc` (321 MB) | 7.5 s / 480×, WER 17.2 % | 155–159× (WER 2.47 %) |
|
| 38 |
+
|
| 39 |
+
An existing FluidAudio cache keeps the old file until the `phonon-2` model directory is deleted.
|
| 40 |
+
|
| 41 |
## Usage (FluidAudio, merged in [#980](https://github.com/FluidInference/FluidAudio/pull/980))
|
| 42 |
|
| 43 |
```swift
|
|
|
|
| 62 |
|
| 63 |
| File | Size | Encoding | ANE | ANE RTFx | GPU |
|
| 64 |
|---|---:|---|---:|---:|---|
|
| 65 |
+
| **`Encoder.mlmodelc`** (default, ladder) | **341 MB** | sparse mask + 6-bit palette per 8 rows, 3 window lengths | **18.6 ms** (15 s) / 12 ms (10 s) / 6 ms (5 s) | **193×** | ~150 s load on every launch |
|
| 66 |
+
| `Encoder_sparse-g8.mlmodelc` (previous default, 15 s only) | 321 MB | sparse mask + 6-bit palette per 8 rows | 18.6 ms | 159× | 16 ms, but ~150 s load on every launch |
|
| 67 |
| `Encoder_sparse-g4.mlmodelc` | 246 MB | sparse mask + 4-bit palette per 4 rows | 24.3 ms | 140× | same load caveat |
|
| 68 |
| `Encoder_sparse-g1.mlmodelc` | 176 MB | sparse mask + 2-bit palette per row | 70 ms | ~70× | same load caveat |
|
| 69 |
| `Encoder_lut6.mlmodelc` | 470 MB | dense 6-bit palette per 8 rows | 18.6 ms | 155× | 16 ms, 0.6 s load |
|
|
|
|
| 89 |
|---|---|---|---|
|
| 90 |
| test-clean (2620 files) WER | 2.27 % | **2.13 %** | 2.47 % |
|
| 91 |
| test-other (2939 files) WER | 4.12 % | **3.81 %** | 4.62 % |
|
| 92 |
+
| test-clean RTFx | 149–152× | 151× | **193×** ladder / 159× single-shape |
|
| 93 |
| test-other RTFx | 138× | 142× | **146×** |
|
| 94 |
|
| 95 |
v3 is the more accurate model on LibriSpeech by 0.20 / 0.50 points, reproducing the upstream card's own gap to its
|
|
|
|
| 111 |
| v3 | 10.9 s | 331× | 16.5 % |
|
| 112 |
| Ultra | 7.7 s | 469× | **13.5 %** |
|
| 113 |
| Redux | 14.2 s | 254× | 14.8 % |
|
| 114 |
+
| **Phonon-2 ladder `Encoder.mlmodelc` (341 MB), pause-cut windows** | **5.55 s** | **649×** | 16.7 % |
|
| 115 |
+
| Phonon-2 `Encoder_sparse-g8` (321 MB, previous default) | 7.5 s | 478× | 17.2 % |
|
| 116 |
| Phonon-2 `Encoder_lut6` (470 MB) | 7.4 s | 486× | 17.2 % |
|
| 117 |
| Phonon-2 `Encoder_sparse-g4` (246 MB) | 9.0 s | 399× | 17.2 % |
|
| 118 |
| Phonon-2 `Encoder_sparse-g1` (176 MB) | 23.4 s | 154× | 17.2 % |
|