code/serving/V100.md

OpenJEV on SGLang / V100

This uses sglang_openjev/ from https://huggingface.co/AlexWortega/openjev/tree/main/code/sglang_openjev. HF_SOURCE.json records the verified source revision and hashes. The adapter registers a Qwen3.5 classification head inside SGLang; /classify returns three raw logits in contradiction, entailment, neutral order.

The separate code/v100 offline scorer is not this HTTP serving engine.

Isolated runtime

Work directory on eva01: ~/storage/sglang-openjev-v100 (the large storage mount). The existing evaluation environment and system CUDA installation are unchanged.

Runtime source: https://github.com/haohervchb/sglang-V100, revision dca488908ee4e3f1bc676c3bf5dcd26ff049cfc3. FlashInfer source: https://github.com/haohervchb/flashinfer, revision c3c40a7b90b792fc59f90f8f55c9e2de9c1b6833, with the runtime's patches/flashinfer-sm70.patch applied.

Use PyTorch 2.9.1 cu126, torchvision 0.24.1, and a private CUDA 12.6.3 compiler. Official cu128 PyTorch wheels omit SM70. The installer explicitly checks torch.cuda.get_arch_list() and performs an FP16 GPU matmul before installing the runtime. prepare_v100_cuda.py ROOT --version 12.6.3 downloads NVIDIA toolkit components and verifies their manifest hashes.

After cloning the pinned runtime, preparing the private toolkit at $OPENJEV_V100_ROOT/cuda126, and installing the cu126 wheels in its venv, run install_v100_runtime.sh. It builds the minimal SM70 sgl-kernel and installs the patched FlashInfer. This is a local source build, not the fork's system installation script.

The pinned SGLang version also builds a Rust gRPC extension during installation. Provision Rust in $OPENJEV_V100_ROOT/rustup with cargo caches under $OPENJEV_V100_ROOT/cargo; the installer uses these paths. Rust 1.98.1 and protoc 33.6 (under $OPENJEV_V100_ROOT/protoc) were provisioned on eva01. The host's older protoc does not accept --experimental_allow_proto3_optional. Neither a system-wide Rust default nor a CUDA driver update is required.

Launch and validation

The training checkpoint lacks image/video processor configs, which SGLang's Qwen3.5 architecture requires even for text classification. Run prepare_v100_model.py CHECKPOINT $OPENJEV_V100_ROOT/model to create a symlink overlay. It adds only the two processor configs from the pinned base Qwen/Qwen3.5-4B revision, checks their hashes, and preserves the checkpoint's weights, tokenizer and model config.

serve_sglang_v100.sh CHECKPOINT 31000 launches on GPU 0 and binds localhost. Set CUDA_VISIBLE_DEVICES to choose another free GPU. Initial settings use FP16, TileLang V100 attention and GDN prefill, an 8192-token server context, and disabled CUDA graphs. Pass --disable-radix-cache for the initial uncached baseline. The benchmark still truncates at 4096 tokens, matching the reference; this runtime rejects inputs exactly equal to the configured server context length.

The launcher selects GCC 10 for C++20 JIT headers and PyTorch sampling (the classifier does not sample output tokens). Since the September 28 deployment, it caps running requests at 16 and uses LPM scheduling by default; override with MAX_RUNNING_REQUESTS and SCHEDULE_POLICY. The original September 26 results below used 8 running requests and FCFS. Prefix caching uses extra_buffer and a Mamba-to-KV memory ratio of 3. This provides 503 recurrent state slots and 135,008 KV tokens at the same 80% GPU memory budget. The original ratio of 0.9 provided 317 slots and suffered cache eviction on the larger set. --disable-radix-cache selects the uncached no_buffer baseline automatically. All JIT caches use the large storage mount.

Apply patches/dynamic-paged.patch to the pinned runtime before launching. It makes the batch, physical-page count and block-table width runtime dimensions of the existing TileLang paged attention kernel. Static specialization on these shapes caused repeated compilation during real request batches. The math and tiling remain unchanged. SGLANG_V100_DYNAMIC_PAGED=0 restores static selection. check_dynamic_paged.py checks ragged causal attention with and without cached prefixes against a dense FP32 reference and asserts that changing all three dimensions reuses one kernel. The GPU check passed with maximum absolute error 0.001073; this kernel check alone does not establish full-model agreement.

bench_classify_v100.py submits exact token IDs to /classify and compares the responses to saved reference logits. It records input and token hashes, entailment probability differences, NLI argmax changes, HTTP wall time and per-batch latencies. Warmup is excluded. --cold-cache flushes SGLang's cache before each iteration. Cold and warm-cache results must be reported separately.

Verified result, 2026-09-26

On eva01, one V100-SXM2-32GB, FP16, HTTP /classify, batches of 32 inputs with four client workers and eight running sequences. Each cell is the median of three measured repetitions after one initial pass:

Pairs	Prefix cache disabled	Prefix cache cleared before each pass	Repeated inputs in warm prefix cache
G: 303	24.928 s	13.417 s	2.697 s
T: 564	47.618 s	29.635 s	4.732 s

The cache therefore improves these fresh-set passes by 1.86x / 1.61x and exact repeated-set passes by 9.24x / 10.06x. Warm-prefix numbers are not fresh-request throughput. Changing the Mamba memory ratio from 0.9 to 3 reduced the larger set's warm-cache median from 28.801 s to 4.732 s.

All 867 NLI argmax labels agree with an independently generated, unmodified Transformers FP16 reference. Input/token hashes were checked. The largest entailment probability difference across the selected cached runs was 0.012851. In the saved final pass of each profile, among 291 same-premise/instruction option groups (793 rows), no strict reference winner changed. Two cold-cache choices changed where the reference probabilities were exactly tied. This is agreement testing, not a held-out Decision Index accuracy result; singleton and cross-document aggregation are not covered by that grouped check.

First-use JIT remains expensive, especially after changing the GDN state-pool size: the selected profile's first G pass took 202.709 s. Initial passes are excluded from medians; additional JIT outliers in measured repetitions are retained, including 45.085 s for uncached G and 15.064 s for warm-cache T. These are controlled throughput measurements, not production p99 guarantees.

Receipts and raw arrays: results_v100/sglang/summary.json, runtime_receipt.json, ratio3_server_config.json, individual benchmark JSONs, and reference/candidate .npy files. The remote directory is ~/storage/sglang-openjev-v100/results.

The endpoint is http://127.0.0.1:31000/classify on eva01. Since September 28 it is a systemd socket-activated gateway over two replicas; see DEPLOYMENT-eva01.md. Do not launch another standalone server on that port. For a separate development instance on a free GPU/port:

CUDA_VISIBLE_DEVICES=3 bash ~/storage/sglang-openjev-v100/app/serve_sglang_v100.sh \
  ~/storage/sglang-openjev-v100/model 31003

The model overlay defaults to ~/storage/sglang-openjev-v100/model. The old standalone PID record is archived as server.legacy-20260926.pid; current PIDs and logs are managed by systemd. server_cached_ratio3.log is historical.

Further scheduling comparison, 2026-09-27

tune_sglang_v100.py compares FCFS with 8, 16 and 32 running requests, then LPM with 16. It uses a separate GPU and port, waits for 60 seconds without GPU processes before each launch, and preserves the existing endpoint. Each profile runs both input sets with cleared and warm prefix cache and the same FP16 references. It saves server configuration, warmup, all repetitions, logits and correctness checks. It does not automatically promote a profile.

The benchmark now audits same-premise/instruction option groups on every pass, including warmup, and retains all logits in .passes.npy. Selecting outside the set of reference winners fails the benchmark; selecting another exactly tied winner is recorded separately. The guard reproduces the previous saved-pass audit: 291 groups, no choices outside the reference winners, two exact ties in cold T. Singleton and cross-document ranking remain outside this check.

cd ~/storage/sglang-openjev-v100
venv/bin/python app/tune_sglang_v100.py --gpu 1 --port 31001 \
  --out "$PWD/results/tuning_20260927"

The output directory must be new. run.json records live phase, server PID and completed measurements. A daemon invocation is currently recorded in tuning_20260927.pid; check that process and its log before starting another.

bench_classify_v100.py --exclusive-gpu INDEX --server-pid PID samples physical GPU process ownership every 0.5 seconds. Only the named server and its descendants are allowed; PID start times protect against reuse. Competing processes or failed monitoring invalidate the report and return a nonzero exit status. Sub-sampling interval competition is still possible; this is sampled evidence, not a GPU lease.

The initial September 27 refresh was contaminated by simultaneous v6 evaluation jobs: warm G took 6.28–7.30 seconds instead of the prior 2.70 seconds. The candidate also received only 192 Mamba state slots because another job allocated GPU memory during startup. That candidate was stopped. These timings must not be used to rank configurations; the next valid comparison must run on an exclusive GPU.