| # Speed: protocol and results (same harness, same GPU) |
|
|
| ## Protocol |
| | | | |
| |---|---| |
| | GPU | 1× NVIDIA H200 (143,771 MiB, driver 595.58.03, CUDA driver 13.2) (one GPU of an 8-GPU node), no other process on it | |
| | Host | AMD EPYC 9554 64-core, 256 threads | |
| | Ines-1 | `scripts/serve_jev.py`, **1 process**, bf16 weights, PyTorch 2.14.0+cu130 (CUDA 13.0, cuDNN 9.24), Python 3.12.3; **CUDA graphs: yes** (prefill graphs for batch × length buckets ≤ 6,144 padded tokens; eager batched prefill above); no torch.compile | |
| | encoders (144M Julia-1 architecture, 322M laya-multilingual architecture) | production micro-batching Jev server, **4 processes** (one model copy each), bf16 autocast, PyTorch 2.13.0+cu130, Python 3.12.13; **CUDA graphs: no**, torch.compile: no. Checkpoints were domain fine-tunes; their speed is that of the architecture | |
| | load generator | `scripts/bench_jev.py`: closed loop, c clients spread over 16 processes; at each level c warm-up requests, then n measured requests; HTTP/1.1 without keep-alive | |
| | requests | *short*: 1 state of 13 words, 2 questions (choice of 3 + noul), n = 3,000 per level; *typed-decisions*: the 400 Spanish test cases, 5 questions each, n = 800 per level | |
| | input length (tokens per question, from each server's `usage`) | short: 61.5 (Ines-1 prompt template) / 35.0 (encoders); typed: 316.2 / 311.6 | |
| | runs | two complete runs on 2026-10-01 (run 2 below; run 1 within ±10 % on throughput) | |
|
|
| ## Single request (c = 1): latency per request |
| | model | short p50 / p95 | typed case (5 questions) p50 / p95 | |
| |---|---:|---:| |
| | Ines-1 (1 proc.) | **9.6 / 9.7 ms** | 29.6 / **38.3 ms** | |
| | Julia-1 architecture, 144M (4 proc.) | 12.7 / 21.0 ms | **15.8** / 88.4 ms | |
| | laya-multilingual architecture, 322M (4 proc.) | 12.7 / 26.3 ms | **15.3** / 87.3 ms | |
|
|
| ## Maximum throughput (questions per second, best concurrency level) |
| | model | short (run 2 / run 1) | p50 / p95 at that level | typed (run 2 / run 1) | p50 / p95 at that level | |
| |---|---:|---:|---:|---:| |
| | Ines-1 | 1,923 / 1,883 (c=256 / 128) | 218 / 277 ms | 539 / 539 (c=32) | 294 / 377 ms | |
| | Julia-1 architecture | 1,694 / 1,780 (c=256) | 237 / 477 ms | **1,111** / 1,019 (c=128) | 443 / 798 ms | |
| | laya-multilingual architecture | 1,854 / 1,666 (c=256 / 128) | 217 / 443 ms | 763 / 835 (c=128) | 655 / 1,015 ms | |
|
|
| ## Reading |
| - Short requests: launch-bound. Ines-1 (one process, CUDA graphs) has the lowest and tightest single-request |
| latency and the same maximum throughput as the encoders within run-to-run variation (±10 %). |
| - ~300-token questions: compute-bound. **The encoders are faster**: 1.4–2.1× the throughput and half the median |
| single-request latency (their p95 is higher because four processes compete for the GPU). Ines-1 reaches |
| ~180k prompt tokens/s here, close to its isolated prefill ceiling (~230k). |
| - Not a general speed ranking: different serving stacks (1 vs 4 processes, graphs vs none), different prompt |
| lengths for the same request, one GPU. |
|
|
| ## Run 3 (2026-10-06): four models, one GPU, with memory |
|
|
| Same load generator (`scripts/bench_jev.py`, closed loop, 16 client processes, HTTP/1.1 without keep-alive), same two |
| request sets and levels as above, **one free H200 per model in turn** (GPU 2 of the node, nothing else on it). The |
| only change to the load generator: the request's `model` field is taken from `BENCH_MODEL` (OpenJev rejects requests |
| without a known model; the other servers ignore it). GPU memory was sampled every 0.5 s with `nvidia-smi` (whole GPU). |
| Raw output: `speed_runs/2026-10-06_part1_gemma_laya.txt`, `speed_runs/2026-10-06_part2_gliner_ines.txt` (the first |
| GLiNER and Ines-1 start attempts failed on environment issues before any request and were rerun in part 2). |
|
|
| | model (params total / active) | server | short c=1 p50 / p95 | short max q/s (c) | typed c=1 p50 / p95 | typed max q/s (c) | GPU memory idle / peak | |
| |---|---|---:|---:|---:|---:|---:| |
| | **Ines-1** (1.59B / 405M, bf16) | `scripts/serve_jev.py`, 1 process, CUDA graphs, dynamic batching | **9.4 / 9.7 ms** | **1,896** (128) | 30.6 / **40.5 ms** | 544 (32) | 9.7 / 21.9 GB | |
| | laya-multilingual arch. (322M, bf16) | production Jev server, 4 processes, micro-batching | 13.0 / 25.9 ms | 1,848 (256) | **20.6** / 89.2 ms | **840** (128) | 8.0 / 24.1 GB | |
| | GLiNER2.5-multi arch. (287M) | GLiNER Jev server, 4 workers, **no batching** | 48.8 / 49.3 ms | 68 (64) | 123.1 / 129.3 ms | 49 (32) | 6.7 / 26.5 GB | |
| | DiffusionGemma-26B-A4B-it FP8 (~26B / ~4B) | OpenJev 0.4.0 (vLLM), max 64 sequences | 91.7 / 94.0 ms | 79 (32) | 158.8 / 170.2 ms | 143 (32) | weights 25.8 GiB; **120.6 / 135.1 GB reserved** | |
|
|
| Latency at the best-throughput level (p50 / p95): Ines-1 118 / 197 ms (short), 282 / 386 ms (typed); laya 212 / 483 ms, |
| 604 / 913 ms; GLiNER 1,729 / 3,805 ms, 2,070 / 6,871 ms; Gemma 718 / 1,439 ms, 1,008 / 1,545 ms. |
|
|
| Reading: |
| - **Ines-1 has the lowest single-request latency on short requests and matches the 322M encoder's peak short |
| throughput** with one process. On ~300-token questions the encoder is still faster (840 vs 544 questions/s, |
| 20.6 vs 30.6 ms median), but Ines-1's tail is tighter (p95 40.5 vs 89.2 ms), as in runs 1–2. |
| - **Gemma is 5–10× slower at c=1 and ~4–24× lower in peak throughput** than Ines-1 or laya, and returned HTTP 503 on |
| 12 of the ~12,000 short requests sent at c ≥ 32 (none on typed cases). Its memory figure is vLLM's preallocation (`gpu_memory_utilization` |
| 0.80: 25.8 GiB of weights + 83 GiB of KV cache), not a minimum; it needs at least the ~26 GiB of weights. |
| - **GLiNER's numbers measure its server, not the architecture's ceiling**: that server answers one request per |
| worker with no batching, so throughput stays flat as concurrency rises. Input tokens per question are counted |
| differently by each server (`usage`): Ines-1 61.5 / 316.2, laya 35.0 / 311.6, GLiNER 8.5 / 37.0, Gemma 67.0 / 136.7. |
| - Peak memory includes activations at the highest concurrency; idle is after load and warm-up. The encoder rows use |
| domain fine-tunes of those architectures (speed is the architecture's), as in runs 1–2. |
| - Not a general speed ranking: different serving stacks, prompt templates and process counts, one GPU. |
|
|
