transformers serve benchmark matrix

Failed and abandoned attempts

Llama-3.1-8B-Instruct (t4-small)

kernels-community/flash-attn2 has no build for T4 (compute capability 7.5, pre-Ampere). Crashed at model-load time, never served a request.

DeepSeek-V4-Flash-0731 (h200x2)

Crashed in transformers' FP8-quantizer tensor-parallel setup — its MoE expert implementation isn't in the quantizer's lookup table yet. Failed in under a minute, before weights even downloaded.

Update — root cause found, one bug fixed, a second one blocks it. FineGrainedFP8HfQuantizer.update_tp_plan() looks up FP8Experts._impl_tp_layer_overrides[config._experts_implementation], which only has one registered key: "deepgemm_megamoe". config._experts_implementation defaults to None unless experts_implementation="deepgemm_megamoe" is passed explicitly to from_pretrained — and transformers serve's CLI has no flag for this at all (checked the full ServeArguments list and ModelManager's fixed model_kwargs dict), so it's structurally unreachable through the CLI even though plain AutoModelForCausalLM.from_pretrained(..., experts_implementation= "deepgemm_megamoe") avoids the crash. Verified locally with a weight-free repro (no GPU needed) against a fresh transformers-main clone.

Retried with a workaround (scripts/serve_deepseekv4_h200x2_patched.py): monkeypatch PreTrainedModel.from_pretrained to default that kwarg, then launch transformers.cli.serve.Serve(...) programmatically instead of via the CLI. This fixed the quantizer crash completely — all 167GB / 1736 weight tensors downloaded and loaded, server reported "Application startup complete". But a second, unrelated bug then surfaced on the first real generation: ValueError: Invalid group type: compressed_sparse_attention (continuous_batching/cache.py:280). DeepSeek-V4's layer types default to alternating compressed_sparse_attention / heavily_compressed_attention (its Lightning-Indexer compressed-KV attention), and the CB paged-attention cache only has allocators for full_attention and sliding_attention — no support for DeepSeek-V4's attention layout at all. That's real cache-layout work, not something a script-level patch can route around. Full details in deepseekv4_flash_h200x2_patched/summary.json. Filed upstream: https://github.com/huggingface/transformers/issues/TODO

GLM-5.2-FP8 (h200x8)

Same exact bug as DeepSeek-V4 above, same crash location. Confirms it's a general transformers limitation for brand-new MoE checkpoints, not model-specific.

Update — same fix, same second blocker. Retried with the identical experts_implementation="deepgemm_megamoe" workaround (scripts/serve_glm52fp8_h200x8_patched.py). Confirmed locally first (weight-free, no GPU) that GLM-5.2-FP8 hits the identical crash without the patch and avoids it with the patch. With the patch: quantizer crash fully avoided, all 709GB / 2160 weight tensors loaded, server started. A smoke-test /v1/chat/completions request then crashed CB with ValueError: Invalid group type: deepseek_sparse_attention — same code path, same class of bug as DeepSeek-V4's compressed_sparse_attention crash. GLM-5.2-FP8's layer_types == {"deepseek_sparse_attention"}, and the CB paged-attention cache still only knows full_attention/sliding_attention. Strong independent confirmation that both the CLI-flag gap and the missing compressed/sparse-attention cache-allocator support are general issues across this whole generation of MoE checkpoints, not one model's quirk. Full details in glm52_fp8_h200x8_patched/summary.json.

Also tried mounting pre-staged weights from bucket SaylorTwift/GLM-5.2-bucket (--volume hf://buckets/SaylorTwift/GLM-5.2-bucket:/mnt/glm52:ro) to skip the 709GB Hub download — small metadata files fetched fine, but the actual safetensors reads stalled with zero progress for 28 minutes (job cancelled). Likely cause: transformers/accelerate mmaps safetensors files, which behaves poorly on FUSE-backed bucket volumes. Fell back to a direct Hub download, which worked.

gpt-oss-20b (a100-large)

Continuous batching needs a Hopper GPU. sdpa isn't supported by this architecture at all; the kernels-hub flash-attn2 needs Hopper for attention sinks; and without kernels installed it falls back to eager, which OOMs under concurrent load (no paged KV cache). Four configs tried, all failed on Ampere for different reasons.

Ornith-1.5-9B (l40sx1)

from logs: Qwen3_5ForConditionalGeneration does not support continuous batching. Falling back to sequential generation.

Laguna S 2.1 (a100x8)

118B-A8B MoE, unquantized BF16 (custom laguna architecture, --trust-remote-code). The first real request crashed the CB worker: RuntimeError: Expected all tensors to be on the same device, but got index is on cuda:1, different from other tensors on cuda:0 inside transformers' generic paged-KV-cache update (continuous_batching/cache.py:443, torch.index_select(k_cache, 0, layer_read_index)). Root cause looks generic, not Laguna-specific: with device_map-based (non-tensor-parallel) multi-GPU sharding, different decoder layers land on different GPUs, but the CB cache's per-layer read index isn't migrated to match.

Reproducing a row

Each server was launched with a small uv script (dependencies declared inline, saved in scripts/ in this same bucket) via:

hf jobs uv run scripts/<script>.py --flavor <hardware> --expose 8000 --secrets HF_TOKEN --timeout <Nh> --detach

and evaluated with:

export TRANSFORMERS_SERVE_API_KEY=$(hf auth token)
export TRANSFORMERS_SERVE_BASE_URL="https://<job_id>--8000.hf.jobs/v1"
inspect eval inspect_evals/gsm8k \
  --model openai-api/transformers_serve/<model_id> \
  --limit 200 --max-connections 8 --temperature 0 -M stream=false \
  --log-dir <this directory>

See each summary.json's config_flags for the exact transformers serve flags used for that row.