alexwengg commited on
Commit
1702fc2
·
verified ·
1 Parent(s): c5c7729

card: encoder ladder (3 window lengths), new default numbers

Browse files
Files changed (1) hide show
  1. README.md +22 -3
README.md CHANGED
@@ -21,6 +21,23 @@ Every encoder here holds the checkpoint's **exact** five-value weights — as fp
21
  `constexpr_sparse_to_dense`). No re-quantization noise is added, and all five produce identical transcripts. The decoder
22
  and joint are re-exported from the checkpoint's int6 tables; preprocessor and vocabulary are the v3 ones.
23
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
24
  ## Usage (FluidAudio, merged in [#980](https://github.com/FluidInference/FluidAudio/pull/980))
25
 
26
  ```swift
@@ -45,7 +62,8 @@ encoder on the Neural Engine. All five are exact and give the same transcripts.
45
 
46
  | File | Size | Encoding | ANE | ANE RTFx | GPU |
47
  |---|---:|---|---:|---:|---|
48
- | **`Encoder.mlmodelc`** (default) | **321 MB** | sparse mask + 6-bit palette per 8 rows | **18.6 ms** | **159×** | 16 ms, but ~150 s load on every launch |
 
49
  | `Encoder_sparse-g4.mlmodelc` | 246 MB | sparse mask + 4-bit palette per 4 rows | 24.3 ms | 140× | same load caveat |
50
  | `Encoder_sparse-g1.mlmodelc` | 176 MB | sparse mask + 2-bit palette per row | 70 ms | ~70× | same load caveat |
51
  | `Encoder_lut6.mlmodelc` | 470 MB | dense 6-bit palette per 8 rows | 18.6 ms | 155× | 16 ms, 0.6 s load |
@@ -71,7 +89,7 @@ corpus-level (total edit distance over total reference words); RTFx = total audi
71
  |---|---|---|---|
72
  | test-clean (2620 files) WER | 2.27 % | **2.13 %** | 2.47 % |
73
  | test-other (2939 files) WER | 4.12 % | **3.81 %** | 4.62 % |
74
- | test-clean RTFx | 149–152× | 151× | **159×** |
75
  | test-other RTFx | 138× | 142× | **146×** |
76
 
77
  v3 is the more accurate model on LibriSpeech by 0.20 / 0.50 points, reproducing the upstream card's own gap to its
@@ -93,7 +111,8 @@ excludes model load). Reference = the concatenated Earnings-22 chunk transcripts
93
  | v3 | 10.9 s | 331× | 16.5 % |
94
  | Ultra | 7.7 s | 469× | **13.5 %** |
95
  | Redux | 14.2 s | 254× | 14.8 % |
96
- | Phonon-2 default (sparse, 321 MB) | 7.5 s | **478×** | 17.2 % |
 
97
  | Phonon-2 `Encoder_lut6` (470 MB) | 7.4 s | 486× | 17.2 % |
98
  | Phonon-2 `Encoder_sparse-g4` (246 MB) | 9.0 s | 399× | 17.2 % |
99
  | Phonon-2 `Encoder_sparse-g1` (176 MB) | 23.4 s | 154× | 17.2 % |
 
21
  `constexpr_sparse_to_dense`). No re-quantization noise is added, and all five produce identical transcripts. The decoder
22
  and joint are re-exported from the checkpoint's int6 tables; preprocessor and vocabulary are the v3 ones.
23
 
24
+ ## Encoder ladder (`Encoder.mlmodelc`, since [#1019](https://github.com/FluidInference/FluidAudio/pull/1019))
25
+
26
+ `Encoder.mlmodelc` is a Core ML **multifunction** model: three encoder programs — `enc_5040ms`, `enc_10160ms`,
27
+ `enc_15000ms` (5.04 / 10.16 / 15 s windows, 63 / 126 / 188 encoder frames) — over one shared weight blob (341 MB for the
28
+ three). FluidAudio ≥ the #1019 merge loads every function and runs each window on the smallest one that holds it; older
29
+ FluidAudio versions load the default function (15 s) and behave as before. The three lengths are the Neural Engine's
30
+ 64-frame tiles: 11–15 s windows all cost ~19 ms, 10 s costs 12 ms, 5 s about 6 ms, so a short clip stops paying for a
31
+ 15 s pass and long recordings are cut at pauses with no overlap. Each function computes the same masked encoder;
32
+ transcripts do not change. First Neural Engine load compiles each function (~55 s each on an M5 Pro, cached after).
33
+
34
+ | | Encoder | 1 h Earnings-22 (M5 Pro) | LibriSpeech test-clean RTFx |
35
+ |---|---|---:|---:|
36
+ | FluidAudio with the ladder | `Encoder.mlmodelc` (341 MB) | **5.55 s / 649×, WER 16.7 %** | **193×** (WER 2.47 %) |
37
+ | previous default | `Encoder_sparse-g8.mlmodelc` (321 MB) | 7.5 s / 480×, WER 17.2 % | 155–159× (WER 2.47 %) |
38
+
39
+ An existing FluidAudio cache keeps the old file until the `phonon-2` model directory is deleted.
40
+
41
  ## Usage (FluidAudio, merged in [#980](https://github.com/FluidInference/FluidAudio/pull/980))
42
 
43
  ```swift
 
62
 
63
  | File | Size | Encoding | ANE | ANE RTFx | GPU |
64
  |---|---:|---|---:|---:|---|
65
+ | **`Encoder.mlmodelc`** (default, ladder) | **341 MB** | sparse mask + 6-bit palette per 8 rows, 3 window lengths | **18.6 ms** (15 s) / 12 ms (10 s) / 6 ms (5 s) | **193×** | ~150 s load on every launch |
66
+ | `Encoder_sparse-g8.mlmodelc` (previous default, 15 s only) | 321 MB | sparse mask + 6-bit palette per 8 rows | 18.6 ms | 159× | 16 ms, but ~150 s load on every launch |
67
  | `Encoder_sparse-g4.mlmodelc` | 246 MB | sparse mask + 4-bit palette per 4 rows | 24.3 ms | 140× | same load caveat |
68
  | `Encoder_sparse-g1.mlmodelc` | 176 MB | sparse mask + 2-bit palette per row | 70 ms | ~70× | same load caveat |
69
  | `Encoder_lut6.mlmodelc` | 470 MB | dense 6-bit palette per 8 rows | 18.6 ms | 155× | 16 ms, 0.6 s load |
 
89
  |---|---|---|---|
90
  | test-clean (2620 files) WER | 2.27 % | **2.13 %** | 2.47 % |
91
  | test-other (2939 files) WER | 4.12 % | **3.81 %** | 4.62 % |
92
+ | test-clean RTFx | 149–152× | 151× | **193×** ladder / 159× single-shape |
93
  | test-other RTFx | 138× | 142× | **146×** |
94
 
95
  v3 is the more accurate model on LibriSpeech by 0.20 / 0.50 points, reproducing the upstream card's own gap to its
 
111
  | v3 | 10.9 s | 331× | 16.5 % |
112
  | Ultra | 7.7 s | 469× | **13.5 %** |
113
  | Redux | 14.2 s | 254× | 14.8 % |
114
+ | **Phonon-2 ladder `Encoder.mlmodelc` (341 MB), pause-cut windows** | **5.55 s** | **649×** | 16.7 % |
115
+ | Phonon-2 `Encoder_sparse-g8` (321 MB, previous default) | 7.5 s | 478× | 17.2 % |
116
  | Phonon-2 `Encoder_lut6` (470 MB) | 7.4 s | 486× | 17.2 % |
117
  | Phonon-2 `Encoder_sparse-g4` (246 MB) | 9.0 s | 399× | 17.2 % |
118
  | Phonon-2 `Encoder_sparse-g1` (176 MB) | 23.4 s | 154× | 17.2 % |