Precision fix per review: lead with 8.4x vs best-case ONNX deployment, not the easier 437x/219x comparisons; scope the claim to this workload/hardware
Browse files
README.md
CHANGED
|
@@ -58,16 +58,22 @@ independent implementations (not a bug; see the repo's `research/papers.md`).
|
|
| 58 |
## Benchmark (single request, Apple M4 CPU β full data in `benchmark_results.json`)
|
| 59 |
|
| 60 |
True end-to-end cold invocation (process spawn β raw text in β prediction out
|
| 61 |
-
β process exit, external wall-clock):
|
|
|
|
|
|
|
| 62 |
|
| 63 |
| Implementation | Cold invocation p50 |
|
| 64 |
|---|---:|
|
| 65 |
| **Native C++ (Sajal runtime)** | 11.37ms |
|
| 66 |
-
| ONNX Runtime + lean tokenizer | 95.72ms (8.4x slower) |
|
| 67 |
| ONNX Runtime + π€ transformers tokenizer | 2495.61ms (219.6x slower) |
|
| 68 |
| PyTorch + π€ transformers | 4973.91ms (437.6x slower) |
|
| 69 |
|
| 70 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 71 |
number depends heavily on which tokenizer library it's paired with β using
|
| 72 |
`transformers` for convenience costs ~25x more than using the lean, standalone
|
| 73 |
`tokenizers` library for the exact same token IDs. Native sidesteps that
|
|
|
|
| 58 |
## Benchmark (single request, Apple M4 CPU β full data in `benchmark_results.json`)
|
| 59 |
|
| 60 |
True end-to-end cold invocation (process spawn β raw text in β prediction out
|
| 61 |
+
β process exit, external wall-clock). **Primary comparison: native vs. ONNX
|
| 62 |
+
Runtime paired with the lean, standalone `tokenizers` library β the
|
| 63 |
+
best-case Python deployment, not the easiest target to beat:**
|
| 64 |
|
| 65 |
| Implementation | Cold invocation p50 |
|
| 66 |
|---|---:|
|
| 67 |
| **Native C++ (Sajal runtime)** | 11.37ms |
|
| 68 |
+
| **ONNX Runtime + lean tokenizer** | **95.72ms (8.4x slower)** |
|
| 69 |
| ONNX Runtime + π€ transformers tokenizer | 2495.61ms (219.6x slower) |
|
| 70 |
| PyTorch + π€ transformers | 4973.91ms (437.6x slower) |
|
| 71 |
|
| 72 |
+
The last two rows are real and worth knowing, but they're the easy targets
|
| 73 |
+
(heavier Python stacks) β the 8.4x number above is
|
| 74 |
+
the one that holds up against someone who already optimized their Python
|
| 75 |
+
deployment correctly. Worth knowing before you read too much into "ONNX
|
| 76 |
+
Runtime" as a single number: its own cold-start
|
| 77 |
number depends heavily on which tokenizer library it's paired with β using
|
| 78 |
`transformers` for convenience costs ~25x more than using the lean, standalone
|
| 79 |
`tokenizers` library for the exact same token IDs. Native sidesteps that
|