# Batch-size sweeps

[Model card](./README.md) · [Deployment](./DEPLOYMENT.md) · [Flashlight results](./FLASHLIGHT_REPORT.md)


Measured median **wall-clock seconds**, one warmup and three timed runs per
configuration. Preprocessing, acoustic inference, score transfer, and decoding
are included; model loading, file reading, and HTTP transfer are excluded.
These are repeated synthetic English speech capacity tests, not WER results.
Apple M5 Pro has 48 GiB unified memory; CPU and MPS runs use six PyTorch
threads. Modal L4 runs use four allocated CPU cores. The different host CPUs
affect end-to-end decoding times.

#### Full transcription with KenLM beam decoding

CPU uses FP32; both GPUs use BF16. CPU sweeps stop at batch 8.

**Eight 30-second inputs — 240 seconds of audio**

| Batch limit | Apple M5 Pro CPU FP32 (s) | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) |
|---:|---:|---:|---:|
| 1 | 5.149 | 0.741 | 2.633 |
| 2 | 5.272 | 0.697 | 2.480 |
| 4 | 4.592 | 0.660 | 2.342 |
| 8 | 3.145 | 0.675 | 2.317 |

**Five mixed inputs (6, 12, 20, 30, 68 seconds) — 136 seconds, 7 chunks**

| Batch limit | Apple M5 Pro CPU FP32 (s) | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) |
|---:|---:|---:|---:|
| 1 | 4.083 | 0.487 | 1.709 |
| 2 | 4.175 | 0.459 | 1.605 |
| 4 | 5.106 | 0.447 | 1.549 |
| 8 | 3.916 | 0.447 | 1.494 |

**One four-minute recording — 240 seconds, 10 chunks**

| Batch limit | Apple M5 Pro CPU FP32 (s) | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) |
|---:|---:|---:|---:|
| 1 | 6.766 | 1.352 | 4.944 |
| 2 | 6.999 | 1.295 | 4.748 |
| 4 | 6.884 | 1.287 | 4.676 |
| 8 | 5.409 | 1.257 | 4.634 |

#### Large GPU capacity sweep with greedy decoding

Both GPUs use BF16. Greedy decoding isolates GPU scaling from KenLM cost;
these timings must not be treated as beam-decoding service throughput.
Requested batch limits 1–128 are tested on sufficiently large workloads.

**128 thirty-second inputs — 3,840 seconds of audio, 128 chunks**

| Batch limit | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) |
|---:|---:|---:|
| 1 | 4.187 | 6.919 |
| 2 | 3.450 | 3.824 |
| 4 | 3.060 | 2.368 |
| 8 | 2.908 | 2.019 |
| 16 | 2.951 | 2.218 |
| 32 | 2.975 | 2.398 |
| 64 | 3.199 | 2.606 |
| 128 | 3.447 | 2.661 |

**100 mixed inputs — 2,720 seconds of audio, 140 chunks**

| Batch limit | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) |
|---:|---:|---:|
| 1 | 3.800 | 7.656 |
| 2 | 2.941 | 4.023 |
| 4 | 2.468 | 2.217 |
| 8 | 2.344 | 1.672 |
| 16 | 2.552 | 1.688 |
| 32 | 2.644 | 1.841 |
| 64 | 3.053 | 2.026 |
| 128 | 3.399 | 2.268 |

**One hour-long recording — 3,600 seconds of audio, 144 chunks**

| Batch limit | Apple M5 Pro GPU BF16 (s) | Modal NVIDIA L4 BF16 (s) |
|---:|---:|---:|
| 1 | 5.103 | 7.839 |
| 2 | 4.303 | 4.274 |
| 4 | 3.873 | 2.496 |
| 8 | 3.781 | 2.093 |
| 16 | 3.782 | 2.294 |
| 32 | 3.830 | 2.585 |
| 64 | 4.137 | 2.777 |
| 128 | 3.933 | 2.843 |

All 24 large-workload configurations per GPU completed without OOM. Actual
batches of 128 were exercised on the uniform and hour-long workloads; duration
bucketing limited the mixed workload to at most 100 rows in a batch. M5 Pro
transcripts matched batch-size-1 results throughout; L4 BF16 transcripts were
batch dependent, as detailed in the numerical diagnostic. Start at batch 8.

For **128 × 30-second inputs with full KenLM beam decoding at batch 128**,
M5 Pro GPU took **10.423 s** and Modal L4 took **38.318 s**. On L4, decoding
accounted for about 35.639 s, compared with 2.049 s for acoustic inference.
This is why faster GPU acoustics did not translate into faster full transcripts.

See [the benchmark report and raw JSON](./BENCHMARK_REPORT.md) for timings,
memory definitions, numerical checks, and limitations.