sajalmadan09 commited on
Commit
139d475
Β·
verified Β·
1 Parent(s): f02e80f

Precision fix per review: lead with 8.4x vs best-case ONNX deployment, not the easier 437x/219x comparisons; scope the claim to this workload/hardware

Browse files
Files changed (1) hide show
  1. README.md +9 -3
README.md CHANGED
@@ -58,16 +58,22 @@ independent implementations (not a bug; see the repo's `research/papers.md`).
58
  ## Benchmark (single request, Apple M4 CPU β€” full data in `benchmark_results.json`)
59
 
60
  True end-to-end cold invocation (process spawn β†’ raw text in β†’ prediction out
61
- β†’ process exit, external wall-clock):
 
 
62
 
63
  | Implementation | Cold invocation p50 |
64
  |---|---:|
65
  | **Native C++ (Sajal runtime)** | 11.37ms |
66
- | ONNX Runtime + lean tokenizer | 95.72ms (8.4x slower) |
67
  | ONNX Runtime + πŸ€— transformers tokenizer | 2495.61ms (219.6x slower) |
68
  | PyTorch + πŸ€— transformers | 4973.91ms (437.6x slower) |
69
 
70
- Worth knowing before you read too much into "ONNX Runtime": its own cold-start
 
 
 
 
71
  number depends heavily on which tokenizer library it's paired with β€” using
72
  `transformers` for convenience costs ~25x more than using the lean, standalone
73
  `tokenizers` library for the exact same token IDs. Native sidesteps that
 
58
  ## Benchmark (single request, Apple M4 CPU β€” full data in `benchmark_results.json`)
59
 
60
  True end-to-end cold invocation (process spawn β†’ raw text in β†’ prediction out
61
+ β†’ process exit, external wall-clock). **Primary comparison: native vs. ONNX
62
+ Runtime paired with the lean, standalone `tokenizers` library β€” the
63
+ best-case Python deployment, not the easiest target to beat:**
64
 
65
  | Implementation | Cold invocation p50 |
66
  |---|---:|
67
  | **Native C++ (Sajal runtime)** | 11.37ms |
68
+ | **ONNX Runtime + lean tokenizer** | **95.72ms (8.4x slower)** |
69
  | ONNX Runtime + πŸ€— transformers tokenizer | 2495.61ms (219.6x slower) |
70
  | PyTorch + πŸ€— transformers | 4973.91ms (437.6x slower) |
71
 
72
+ The last two rows are real and worth knowing, but they're the easy targets
73
+ (heavier Python stacks) β€” the 8.4x number above is
74
+ the one that holds up against someone who already optimized their Python
75
+ deployment correctly. Worth knowing before you read too much into "ONNX
76
+ Runtime" as a single number: its own cold-start
77
  number depends heavily on which tokenizer library it's paired with β€” using
78
  `transformers` for convenience costs ~25x more than using the lean, standalone
79
  `tokenizers` library for the exact same token IDs. Native sidesteps that