offload/bench/README.md

Status 2026-10-23: Benchmark tooling in use; only coordinator executes generation suites. Measured results and W4 percentage correction are in COMPARISON. See live roadmap and measured record; campaign review decisions are in LANES.md / lane reports outside the repo.

Extended benchmark harness (S3)

bench2.py is a copy and extension of the deployment's bench.py; the original is unchanged. Legal workload strings, fictional-record builder, and public CourtListener/RECAP builder are preserved. No new runtime dependencies: Python standard library only. Run from any directory.

Only the coordinator runs generation benchmarks. This lane sent zero generation requests. Connectivity-only validation (no /tokenize or completions):

source ~/.config/ndm/paths.sh
python3 offload/bench/bench2.py --dry-run --out offload/bench/logs/connectivity.jsonl

The real corpus files are ignored by git and absent from the lane worktree. Point --data-dir to an existing directory containing opinions_cl.jsonl and filings_recap.jsonl. Do not substitute synthetic filler in an A/B comparison. Missing/insufficient real corpus fails explicitly.

Coordinator examples, after reserving an idle window and arranging HOLD with other workloads:

source ~/.config/ndm/paths.sh
python3 offload/bench/bench2.py --label A --ctx 0 32000 150000 --workload legal synth --data-dir /path/to/existing/data --concurrency 1 2 3 --mixed --mixed-short-ctx 8000 --max-tokens 1500 --reps 3 --thinking --reasoning-effort high --priority 10 --cache-mode cold --out offload/results/A-cold.jsonl
python3 offload/bench/bench2.py --label A --ctx 32000 --workload synth --data-dir /path/to/existing/data --concurrency 1 2 3 --max-tokens 1500 --reps 4 --cache-mode warm --out offload/results/A-warm.jsonl
python3 offload/bench/report.py offload/results/A-cold.jsonl offload/results/A-warm.jsonl --out offload/results/comparison.md

Match all flags and corpus across variants. Repeat warm groups at least twice: rep zero primes a fresh salt and is labelled cold; later reps reuse exactly the prompt/salt and are labelled warm_repeat. Cold reps reuse the same workload text but get fresh salts. Prompt SHA256 is retained per request; prompt tokens come from /tokenize with the same messages/template kwargs used by generation. target_ctx=0 means instruction only. Nonzero context budgets cover the entire occupied chat prompt, including template/instruction. A character-boundary trim can finish a few tokens below target (at most 16); actual occupied and server usage counts are retained. Output reservation is enforced against the smaller of --max-model-len and the advertised live model limit.

--mixed adds a separate case at each context: two --mixed-short-ctx requests start together, then the long-context request arrives after both have emitted tokens and --mixed-delay seconds have elapsed. If both are no longer actively generating, the mixed case fails rather than claiming overlap. Increase the output budget or reduce delay if needed. Normal groups use a start barrier for 1/2/3 requests. Every group requires server running/waiting gauges to be present and zero before submission. The coordinator must isolate the run from external requests: speculation counters are global and cannot be attributed per stream.

Accounting

• Raw JSONL includes each request and its run summary, full model-list identity/root, inference settings, cache salt, reserved output, measured prompt count, usage, stream timing and telemetry. Text is opt-in via --keep-text; these files stay local.
• ttft_s: first reasoning or answer text; first_visible_s: first answer text. Missing channels are null. total_s ends at stream completion, before post-hoc tokenization. finish=length is budget exhaustion, not completion of a full legal task.
• Authoritative delivered totals use server completion_tokens only. Rejected speculative proposals are never delivered tokens. Speculation summary separately reports proposed/accepted counts, rounds, accepted/round without adding a bonus token, and per-position accepted/round.
• Per-channel counts use server reasoning usage where available; answer count is then completion minus reasoning, explicitly labelled as including control tokens. Otherwise /tokenize counts emitted channel text and labels it retokenized_text. This is not guaranteed to recover original generation tokens. No token count is inferred from SSE event counts. Per-channel decode rates use (count-1)/(last-first text time) and retain this count-source limitation.
• decode_tps uses server completion usage over first-to-last delivered text; aggregate rate uses delivered totals over group wall time including prefill and staggered arrivals. Post-hoc tokenization and telemetry shutdown are excluded; client_elapsed_s separately includes postprocessing. Reported request rates and aggregate rates measure different intervals.
• Missing speculation metrics remain null. Resets or disappearing counters fail the run and write an error summary. --metrics-settle (default 2 seconds) waits for periodically published counters after the group, outside throughput timing. Match this to the engine's actual publish cadence if changed; external traffic during this wait contaminates global deltas.
• Comparison tables show mean ± sample standard deviation of repeated run means, keeping cache/workload/concurrency/sampling separate. They do not count concurrent streams as independent repetitions. Errors remain visible and excluded from success averages. With one repetition SD is shown as zero; this is not evidence of repeatability.

Telemetry

Every 2 seconds, bounded parallel SSH reads sample sparkhead, sparkworker, and x86host; an additional terminal sample captures the last counter interval. Slow probes record sample_duration_s rather than pretend to maintain a 2-second cadence. No GPU contexts are created. The Spark GPU's unified memory usage may appear [N/A]; /proc/meminfo is retained.

X86host GPU queries resolve only UUID prefixes c4e7a371 and 7744b44b. --process-name NAME (repeatable) matches exact /proc/comm or executable basename; use a process's actual name, e.g. python3, if workers run under an interpreter. Each observation includes PID and start ticks to distinguish restarts, RSS and cumulative major faults. No names by default means an empty process list. NICs default to enp130s0f0np0 and enp130s0f1np1; override both with --x86host-nic. Byte, retransmit, retry, error, drop and discard counters and interval deltas are retained independently; do not sum overlapping ethtool counters. Baseline nic_delta is empty. Negative deltas indicate reset/wrap and must not be interpreted as traffic.

--extra-metrics URL adds raw per-sample scrapes for offload timings. Every probe error is printed and recorded. Dry-run exits nonzero on telemetry errors; generation results retain errors so missing telemetry cannot silently look healthy. Run summaries retain raw observations instead of inventing peaks between samples.

Local checks

source ~/.config/ndm/paths.sh
python3 -m unittest discover -s offload/bench -p 'test_*.py'
python3 -m mypy --strict offload/bench/
python3 -m ruff check offload/bench/

Tests mock transport and telemetry. They check simultaneous reasoning/content parsing, usage versus retokenization, truncated streams, speculation deltas/reset/positions, idle refusal, full-prompt sizing, request settings, actual concurrency/mixed overlap in fixture threads, JSONL/report accounting, duplicate refusal and zero-generation dry-run behavior. They are not model-performance or legal-quality benchmarks.

Rate convention

Use R = (1 + E[A]) / C for steady B1: accepted draft tokens A exclude the correction/bonus, and C is complete round time. Actual committed counts govern clipped/terminal/mixed rounds. SSE p50 remains a delivery metric, not an engine/GPU timer. See repo metric and paper-unit audit.