eval/speed.md

Speed: protocol and results (same harness, same GPU)

Protocol

GPU	1× NVIDIA H200 (143,771 MiB, driver 595.58.03, CUDA driver 13.2) (one GPU of an 8-GPU node), no other process on it
Host	AMD EPYC 9554 64-core, 256 threads
Ines-1	scripts/serve_jev.py, 1 process, bf16 weights, PyTorch 2.14.0+cu130 (CUDA 13.0, cuDNN 9.24), Python 3.12.3; CUDA graphs: yes (prefill graphs for batch × length buckets ≤ 6,144 padded tokens; eager batched prefill above); no torch.compile
encoders (144M Julia-1 architecture, 322M laya-multilingual architecture)	production micro-batching Jev server, 4 processes (one model copy each), bf16 autocast, PyTorch 2.13.0+cu130, Python 3.12.13; CUDA graphs: no, torch.compile: no. Checkpoints were domain fine-tunes; their speed is that of the architecture
load generator	scripts/bench_jev.py: closed loop, c clients spread over 16 processes; at each level c warm-up requests, then n measured requests; HTTP/1.1 without keep-alive
requests	short: 1 state of 13 words, 2 questions (choice of 3 + noul), n = 3,000 per level; typed-decisions: the 400 Spanish test cases, 5 questions each, n = 800 per level
input length (tokens per question, from each server's usage)	short: 61.5 (Ines-1 prompt template) / 35.0 (encoders); typed: 316.2 / 311.6
runs	two complete runs on 2026-10-01 (run 2 below; run 1 within ±10 % on throughput)

Single request (c = 1): latency per request

model	short p50 / p95	typed case (5 questions) p50 / p95
Ines-1 (1 proc.)	9.6 / 9.7 ms	29.6 / 38.3 ms
Julia-1 architecture, 144M (4 proc.)	12.7 / 21.0 ms	15.8 / 88.4 ms
laya-multilingual architecture, 322M (4 proc.)	12.7 / 26.3 ms	15.3 / 87.3 ms

Maximum throughput (questions per second, best concurrency level)

model	short (run 2 / run 1)	p50 / p95 at that level	typed (run 2 / run 1)	p50 / p95 at that level
Ines-1	1,923 / 1,883 (c=256 / 128)	218 / 277 ms	539 / 539 (c=32)	294 / 377 ms
Julia-1 architecture	1,694 / 1,780 (c=256)	237 / 477 ms	1,111 / 1,019 (c=128)	443 / 798 ms
laya-multilingual architecture	1,854 / 1,666 (c=256 / 128)	217 / 443 ms	763 / 835 (c=128)	655 / 1,015 ms

Reading

• Short requests: launch-bound. Ines-1 (one process, CUDA graphs) has the lowest and tightest single-request latency and the same maximum throughput as the encoders within run-to-run variation (±10 %).
• 300-token questions: compute-bound. The encoders are faster: 1.4–2.1× the throughput and half the median single-request latency (their p95 is higher because four processes compete for the GPU). Ines-1 reaches ~180k prompt tokens/s here, close to its isolated prefill ceiling (230k).
• Not a general speed ranking: different serving stacks (1 vs 4 processes, graphs vs none), different prompt lengths for the same request, one GPU.

Run 3 (2026-10-06): four models, one GPU, with memory

Same load generator (scripts/bench_jev.py, closed loop, 16 client processes, HTTP/1.1 without keep-alive), same two request sets and levels as above, one free H200 per model in turn (GPU 2 of the node, nothing else on it). The only change to the load generator: the request's model field is taken from BENCH_MODEL (OpenJev rejects requests without a known model; the other servers ignore it). GPU memory was sampled every 0.5 s with nvidia-smi (whole GPU). Raw output: speed_runs/2026-10-06_part1_gemma_laya.txt, speed_runs/2026-10-06_part2_gliner_ines.txt (the first GLiNER and Ines-1 start attempts failed on environment issues before any request and were rerun in part 2).

model (params total / active)	server	short c=1 p50 / p95	short max q/s (c)	typed c=1 p50 / p95	typed max q/s (c)	GPU memory idle / peak
Ines-1 (1.59B / 405M, bf16)	scripts/serve_jev.py, 1 process, CUDA graphs, dynamic batching	9.4 / 9.7 ms	1,896 (128)	30.6 / 40.5 ms	544 (32)	9.7 / 21.9 GB
laya-multilingual arch. (322M, bf16)	production Jev server, 4 processes, micro-batching	13.0 / 25.9 ms	1,848 (256)	20.6 / 89.2 ms	840 (128)	8.0 / 24.1 GB
GLiNER2.5-multi arch. (287M)	GLiNER Jev server, 4 workers, no batching	48.8 / 49.3 ms	68 (64)	123.1 / 129.3 ms	49 (32)	6.7 / 26.5 GB
DiffusionGemma-26B-A4B-it FP8 (~26B / ~4B)	OpenJev 0.4.0 (vLLM), max 64 sequences	91.7 / 94.0 ms	79 (32)	158.8 / 170.2 ms	143 (32)	weights 25.8 GiB; 120.6 / 135.1 GB reserved

Latency at the best-throughput level (p50 / p95): Ines-1 118 / 197 ms (short), 282 / 386 ms (typed); laya 212 / 483 ms, 604 / 913 ms; GLiNER 1,729 / 3,805 ms, 2,070 / 6,871 ms; Gemma 718 / 1,439 ms, 1,008 / 1,545 ms.

Reading:

• Ines-1 has the lowest single-request latency on short requests and matches the 322M encoder's peak short throughput with one process. On ~300-token questions the encoder is still faster (840 vs 544 questions/s, 20.6 vs 30.6 ms median), but Ines-1's tail is tighter (p95 40.5 vs 89.2 ms), as in runs 1–2.
• Gemma is 5–10× slower at c=1 and ~4–24× lower in peak throughput than Ines-1 or laya, and returned HTTP 503 on 12 of the ~12,000 short requests sent at c ≥ 32 (none on typed cases). Its memory figure is vLLM's preallocation (gpu_memory_utilization 0.80: 25.8 GiB of weights + 83 GiB of KV cache), not a minimum; it needs at least the ~26 GiB of weights.
• GLiNER's numbers measure its server, not the architecture's ceiling: that server answers one request per worker with no batching, so throughput stays flat as concurrency rises. Input tokens per question are counted differently by each server (usage): Ines-1 61.5 / 316.2, laya 35.0 / 311.6, GLiNER 8.5 / 37.0, Gemma 67.0 / 136.7.
• Peak memory includes activations at the highest concurrency; idle is after load and warm-up. The encoder rows use domain fine-tunes of those architectures (speed is the architecture's), as in runs 1–2.
• Not a general speed ranking: different serving stacks, prompt templates and process counts, one GPU.