# Speed: protocol and results (same harness, same GPU)

## Protocol
| | |
|---|---|
| GPU | 1× NVIDIA H200 (143,771 MiB, driver 595.58.03, CUDA driver 13.2) (one GPU of an 8-GPU node), no other process on it |
| Host | AMD EPYC 9554 64-core, 256 threads |
| Ines-1 | `scripts/serve_jev.py`, **1 process**, bf16 weights, PyTorch 2.14.0+cu130 (CUDA 13.0, cuDNN 9.24), Python 3.12.3; **CUDA graphs: yes** (prefill graphs for batch × length buckets ≤ 6,144 padded tokens; eager batched prefill above); no torch.compile |
| encoders (144M Julia-1 architecture, 322M laya-multilingual architecture) | production micro-batching Jev server, **4 processes** (one model copy each), bf16 autocast, PyTorch 2.13.0+cu130, Python 3.12.13; **CUDA graphs: no**, torch.compile: no. Checkpoints were domain fine-tunes; their speed is that of the architecture |
| load generator | `scripts/bench_jev.py`: closed loop, c clients spread over 16 processes; at each level c warm-up requests, then n measured requests; HTTP/1.1 without keep-alive |
| requests | *short*: 1 state of 13 words, 2 questions (choice of 3 + noul), n = 3,000 per level; *typed-decisions*: the 400 Spanish test cases, 5 questions each, n = 800 per level |
| input length (tokens per question, from each server's `usage`) | short: 61.5 (Ines-1 prompt template) / 35.0 (encoders); typed: 316.2 / 311.6 |
| runs | two complete runs on 2026-10-01 (run 2 below; run 1 within ±10 % on throughput) |

## Single request (c = 1): latency per request
| model | short p50 / p95 | typed case (5 questions) p50 / p95 |
|---|---:|---:|
| Ines-1 (1 proc.) | **9.6 / 9.7 ms** | 29.6 / **38.3 ms** |
| Julia-1 architecture, 144M (4 proc.) | 12.7 / 21.0 ms | **15.8** / 88.4 ms |
| laya-multilingual architecture, 322M (4 proc.) | 12.7 / 26.3 ms | **15.3** / 87.3 ms |

## Maximum throughput (questions per second, best concurrency level)
| model | short (run 2 / run 1) | p50 / p95 at that level | typed (run 2 / run 1) | p50 / p95 at that level |
|---|---:|---:|---:|---:|
| Ines-1 | 1,923 / 1,883 (c=256 / 128) | 218 / 277 ms | 539 / 539 (c=32) | 294 / 377 ms |
| Julia-1 architecture | 1,694 / 1,780 (c=256) | 237 / 477 ms | **1,111** / 1,019 (c=128) | 443 / 798 ms |
| laya-multilingual architecture | 1,854 / 1,666 (c=256 / 128) | 217 / 443 ms | 763 / 835 (c=128) | 655 / 1,015 ms |

## Reading
- Short requests: launch-bound. Ines-1 (one process, CUDA graphs) has the lowest and tightest single-request
  latency and the same maximum throughput as the encoders within run-to-run variation (±10 %).
- ~300-token questions: compute-bound. **The encoders are faster**: 1.4–2.1× the throughput and half the median
  single-request latency (their p95 is higher because four processes compete for the GPU). Ines-1 reaches
  ~180k prompt tokens/s here, close to its isolated prefill ceiling (~230k).
- Not a general speed ranking: different serving stacks (1 vs 4 processes, graphs vs none), different prompt
  lengths for the same request, one GPU.

## Run 3 (2026-10-06): four models, one GPU, with memory

Same load generator (`scripts/bench_jev.py`, closed loop, 16 client processes, HTTP/1.1 without keep-alive), same two
request sets and levels as above, **one free H200 per model in turn** (GPU 2 of the node, nothing else on it). The
only change to the load generator: the request's `model` field is taken from `BENCH_MODEL` (OpenJev rejects requests
without a known model; the other servers ignore it). GPU memory was sampled every 0.5 s with `nvidia-smi` (whole GPU).
Raw output: `speed_runs/2026-10-06_part1_gemma_laya.txt`, `speed_runs/2026-10-06_part2_gliner_ines.txt` (the first
GLiNER and Ines-1 start attempts failed on environment issues before any request and were rerun in part 2).

| model (params total / active) | server | short c=1 p50 / p95 | short max q/s (c) | typed c=1 p50 / p95 | typed max q/s (c) | GPU memory idle / peak |
|---|---|---:|---:|---:|---:|---:|
| **Ines-1** (1.59B / 405M, bf16) | `scripts/serve_jev.py`, 1 process, CUDA graphs, dynamic batching | **9.4 / 9.7 ms** | **1,896** (128) | 30.6 / **40.5 ms** | 544 (32) | 9.7 / 21.9 GB |
| laya-multilingual arch. (322M, bf16) | production Jev server, 4 processes, micro-batching | 13.0 / 25.9 ms | 1,848 (256) | **20.6** / 89.2 ms | **840** (128) | 8.0 / 24.1 GB |
| GLiNER2.5-multi arch. (287M) | GLiNER Jev server, 4 workers, **no batching** | 48.8 / 49.3 ms | 68 (64) | 123.1 / 129.3 ms | 49 (32) | 6.7 / 26.5 GB |
| DiffusionGemma-26B-A4B-it FP8 (~26B / ~4B) | OpenJev 0.4.0 (vLLM), max 64 sequences | 91.7 / 94.0 ms | 79 (32) | 158.8 / 170.2 ms | 143 (32) | weights 25.8 GiB; **120.6 / 135.1 GB reserved** |

Latency at the best-throughput level (p50 / p95): Ines-1 118 / 197 ms (short), 282 / 386 ms (typed); laya 212 / 483 ms,
604 / 913 ms; GLiNER 1,729 / 3,805 ms, 2,070 / 6,871 ms; Gemma 718 / 1,439 ms, 1,008 / 1,545 ms.

Reading:
- **Ines-1 has the lowest single-request latency on short requests and matches the 322M encoder's peak short
  throughput** with one process. On ~300-token questions the encoder is still faster (840 vs 544 questions/s,
  20.6 vs 30.6 ms median), but Ines-1's tail is tighter (p95 40.5 vs 89.2 ms), as in runs 1–2.
- **Gemma is 5–10× slower at c=1 and ~4–24× lower in peak throughput** than Ines-1 or laya, and returned HTTP 503 on
  12 of the ~12,000 short requests sent at c ≥ 32 (none on typed cases). Its memory figure is vLLM's preallocation (`gpu_memory_utilization`
  0.80: 25.8 GiB of weights + 83 GiB of KV cache), not a minimum; it needs at least the ~26 GiB of weights.
- **GLiNER's numbers measure its server, not the architecture's ceiling**: that server answers one request per
  worker with no batching, so throughput stays flat as concurrency rises. Input tokens per question are counted
  differently by each server (`usage`): Ines-1 61.5 / 316.2, laya 35.0 / 311.6, GLiNER 8.5 / 37.0, Gemma 67.0 / 136.7.
- Peak memory includes activations at the highest concurrency; idle is after load and warm-up. The encoder rows use
  domain fine-tunes of those architectures (speed is the architecture's), as in runs 1–2.
- Not a general speed ranking: different serving stacks, prompt templates and process counts, one GPU.