| # Batch-size sweeps |
|
|
| [Model card](./README.md) · [Deployment](./DEPLOYMENT.md) · [Flashlight results](./FLASHLIGHT_REPORT.md) |
|
|
|
|
| Measured median **wall-clock seconds**, one warmup and three timed runs per |
| configuration. Preprocessing, acoustic inference, score transfer, and decoding |
| are included; model loading, file reading, and HTTP transfer are excluded. |
| These are repeated synthetic English speech capacity tests, not WER results. |
| Apple M5 Pro has 48 GiB unified memory; CPU and MPS runs use six PyTorch |
| threads. Modal L4 runs use four allocated CPU cores. The different host CPUs |
| affect end-to-end decoding times. |
|
|
| #### Full transcription with KenLM beam decoding |
|
|
| CPU uses FP32; both GPUs use BF16. CPU sweeps stop at batch 8. |
|
|
| **Eight 30-second inputs — 240 seconds of audio** |
|
|
| | Batch limit | Apple M5 Pro CPU FP32 (s) | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) | |
| |---:|---:|---:|---:| |
| | 1 | 5.149 | 0.741 | 2.633 | |
| | 2 | 5.272 | 0.697 | 2.480 | |
| | 4 | 4.592 | 0.660 | 2.342 | |
| | 8 | 3.145 | 0.675 | 2.317 | |
|
|
| **Five mixed inputs (6, 12, 20, 30, 68 seconds) — 136 seconds, 7 chunks** |
|
|
| | Batch limit | Apple M5 Pro CPU FP32 (s) | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) | |
| |---:|---:|---:|---:| |
| | 1 | 4.083 | 0.487 | 1.709 | |
| | 2 | 4.175 | 0.459 | 1.605 | |
| | 4 | 5.106 | 0.447 | 1.549 | |
| | 8 | 3.916 | 0.447 | 1.494 | |
|
|
| **One four-minute recording — 240 seconds, 10 chunks** |
|
|
| | Batch limit | Apple M5 Pro CPU FP32 (s) | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) | |
| |---:|---:|---:|---:| |
| | 1 | 6.766 | 1.352 | 4.944 | |
| | 2 | 6.999 | 1.295 | 4.748 | |
| | 4 | 6.884 | 1.287 | 4.676 | |
| | 8 | 5.409 | 1.257 | 4.634 | |
|
|
| #### Large GPU capacity sweep with greedy decoding |
|
|
| Both GPUs use BF16. Greedy decoding isolates GPU scaling from KenLM cost; |
| these timings must not be treated as beam-decoding service throughput. |
| Requested batch limits 1–128 are tested on sufficiently large workloads. |
|
|
| **128 thirty-second inputs — 3,840 seconds of audio, 128 chunks** |
|
|
| | Batch limit | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) | |
| |---:|---:|---:| |
| | 1 | 4.187 | 6.919 | |
| | 2 | 3.450 | 3.824 | |
| | 4 | 3.060 | 2.368 | |
| | 8 | 2.908 | 2.019 | |
| | 16 | 2.951 | 2.218 | |
| | 32 | 2.975 | 2.398 | |
| | 64 | 3.199 | 2.606 | |
| | 128 | 3.447 | 2.661 | |
|
|
| **100 mixed inputs — 2,720 seconds of audio, 140 chunks** |
|
|
| | Batch limit | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) | |
| |---:|---:|---:| |
| | 1 | 3.800 | 7.656 | |
| | 2 | 2.941 | 4.023 | |
| | 4 | 2.468 | 2.217 | |
| | 8 | 2.344 | 1.672 | |
| | 16 | 2.552 | 1.688 | |
| | 32 | 2.644 | 1.841 | |
| | 64 | 3.053 | 2.026 | |
| | 128 | 3.399 | 2.268 | |
|
|
| **One hour-long recording — 3,600 seconds of audio, 144 chunks** |
|
|
| | Batch limit | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) | |
| |---:|---:|---:| |
| | 1 | 5.103 | 7.839 | |
| | 2 | 4.303 | 4.274 | |
| | 4 | 3.873 | 2.496 | |
| | 8 | 3.781 | 2.093 | |
| | 16 | 3.782 | 2.294 | |
| | 32 | 3.830 | 2.585 | |
| | 64 | 4.137 | 2.777 | |
| | 128 | 3.933 | 2.843 | |
|
|
| All 24 large-workload configurations per GPU completed without OOM. Actual |
| batches of 128 were exercised on the uniform and hour-long workloads; duration |
| bucketing limited the mixed workload to at most 100 rows in a batch. M5 Pro |
| transcripts matched batch-size-1 results throughout; L4 BF16 transcripts were |
| batch dependent, as detailed in the numerical diagnostic. Start at batch 8. |
|
|
| For **128 × 30-second inputs with full KenLM beam decoding at batch 128**, |
| M5 Pro GPU took **10.423 s** and Modal L4 took **38.318 s**. On L4, decoding |
| accounted for about 35.639 s, compared with 2.049 s for acoustic inference. |
| This is why faster GPU acoustics did not translate into faster full transcripts. |
|
|
| See [the benchmark report and raw JSON](./BENCHMARK_REPORT.md) for timings, |
| memory definitions, numerical checks, and limitations. |
|
|
|
|
