hf-mem inference memory validation

Latest controlled vLLM comparison

controlled-vllm-20261109/ contains eight models tested with --max-num-seqs 1, context 32768, token budget 8192 and native minimum cache-block overrides. All ten full-context inference runs passed, including two automatic-cache controls. Logical cache bytes match in all eight models; six parameter-storage audits match exactly, with runtime-specific differences for Qwen 3.5 and GPT-OSS. Current total estimates are 7–52% above sampled runtime footprints. The directory includes CSVs, eight parameter audits, 64,789 NVML samples, a plot and pinned reproduction sources.

Archived GPU measurements from November 3, 2026, exported on November 9, 2026, with fresh estimates from hf-mem 1.0.0, commit e4babc0882b71d83555dcdb49ea752c417f94121.

These are 170 configurations covering 33 model repositories on A100 80GB and H200 GPUs, including repeated models, engines, devices and launch modes. They are not 170 independent models. All 150 successful configurations have a fresh hf-mem estimate. No new GPU jobs were run for this export.

Files

Reading the data

All numeric memory columns use bytes. Divide by 2**30 for GiB. Engine values printed with two decimal places remain rounded measurements even after conversion to integer bytes. Empty CSV cells and JSON null mean unavailable, not zero. SGLang's native GB labels use binary GiB; the original labels remain visible in the evidence.

device_ready_bytes is an nvidia-smi memory.used snapshot after server health succeeds, followed by a three-second settling interval in the vLLM/SGLang driver. device_text_bytes follows a short text request, and device_probe_bytes follows an image/audio probe when attempted. These measure whole-device usage, not process-exclusive allocations, and are not sampled peak inference or startup memory. Failed multimodal requests are marked independently of healthy startup. llama.cpp has a ready/startup snapshot and no recorded inference probe.

For a stacked GPU-ready plot, use weights_reported_bytes, kv_pool_reported_bytes, recurrent_state_reported_bytes where available, and other_device_memory_residual_bytes. The residual is calculated as the device-ready snapshot minus those reported allocations. It is an accounting remainder, not a directly instrumented overhead allocation: it can include allocator reservation, CUDA context, workspaces, graph pools, recurrent state not separately reported, encoder allocations and other device users. Component readings and the snapshot occur at different startup stages.

vLLM's weights are its reported model-loading memory delta. SGLang's weights are its before/after weight-loading available-memory delta. Neither is a tensor-by-tensor physical memory measurement. vLLM also reports activation peaks, actual CUDA graph pool allocation, and consumed weights plus non-torch memory. non_torch_derived_bytes subtracts its weights from that consumed-memory figure. Activation peaks, graph allocations, and residual overhead describe different accounting scopes and phases: do not add all of these to the residual. SGLang's prefill and decode graph capture deltas are kept separate; no standalone activation peak is available here.

llama.cpp records native GPU model, KV, recurrent-state and compute buffers. Multiple attention cache buffers are summed after the real model load, excluding its zero-buffer sizing pass. Main model weights and the exact GGUF file/revision are recorded; CPU-mapped weights and multimodal compute buffers are separate. A host-mapped tensor's file size is not resident GPU memory or measured physical host RAM. Additional encoder allocations can remain in the device residual.

Comparing with hf-mem

The runtime KV column is the engine's full allocated cache pool. The hf-mem KV column estimates one sequence at the requested context length. vLLM and SGLang often reserve most available GPU memory for cache, so their ready totals are intentionally much larger. Do not interpret their total difference as estimation error or divide a hybrid/sliding cache pool by its token capacity to manufacture a per-request measurement.

hf_mem_requested_total_bytes and its components retain hf-mem's own scope. hf_mem_offloaded_bytes is provided separately where relevant; consult each estimate's weights and offloaded fields before comparing to llama.cpp GPU weights. Model encoders disabled in native text-only configurations remain included in hf-mem estimates and are flagged in the per-run assumptions. SGLang's chunked-prefill budget is supplied to hf-mem as a token-budget approximation, not a claim that its scheduler and graph allocation match vLLM.

The primary runs use one GPU, making device scope explicit. Supplementary multi-GPU excerpts preserve native rank prefixes and must be interpreted per rank; no aggregate total has been inferred from a final rank's log line.

The original vLLM/SGLang launches used mutable container tags and did not pin model commit SHAs. vLLM reports version 0.33.0; SGLang's exact version and immutable image digests are unavailable in these archived logs. These unknowns are retained as null. Fresh estimates pin the current Hub SHAs in model_revisions.json, and cannot establish that historical weights were identical. llama.cpp records build 12063 (2cbca389f) and the four exact model snapshot SHAs. This is an exploratory validation archive with explicit gaps, not a fully pinned benchmark or a claim of universal accuracy.

Download and plot

hf buckets sync hf://buckets/alvarobartt/hf-mem-validation ./hf-mem-validation
cd hf-mem-validation
uv run --with matplotlib python plot.py --output memory-comparison.png

Choose configurations with repeated --run-id arguments, such as mm/vllm-h200-qwen25vl-7b/default. The preview shows observed ready allocations and requested-context estimates in separately labelled panels because their cache scopes differ.

To regenerate the estimates using the recorded model SHAs:

uv run --with hf-mem==1.0.0 python estimate.py

The original complete logs remain private. Public excerpts exclude machine paths, credentials, private job links, unrelated logs and launch environments. export.py expects the original mm/ and jobs/ archive layout; it does not launch GPU jobs or upload files.

Unit reference: SGLang memory pool source defines its GB constant as 1024 * 1024 * 1024; available GPU memory uses the same conversion. Links point to mutable upstream source, not the missing historical image digest.