Lissa1-4B-FP8

The measured GPU-serving build for developer workloads.

This repository contains the universal FP8 export of Lissa1-4B for vLLM and SGLang. It is intended for teams that want a small developer model behind an OpenAI-compatible endpoint: IDE assistants, terminal agents, code review services and internal automation.

The checkpoint is not a generic benchmark trophy. It is a practical serving artifact: load it, run your own prompt set, measure first-token latency and generated tokens per second, then decide whether speculative decoding belongs in your traffic mix.

vLLM

pip install -U vllm
vllm serve lomelidev/Lissa1-4B-FP8 \
  --dtype auto \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90 \
  --host 127.0.0.1 \
  --port 8000

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model":"lomelidev/Lissa1-4B-FP8",
    "messages":[{"role":"user","content":"Review this Go function for a race condition."}],
    "temperature":0.2,
    "top_p":0.9,
    "max_tokens":512
  }'

SGLang

pip install -U sglang
python -m sglang.launch_server \
  --model-path lomelidev/Lissa1-4B-FP8 \
  --host 127.0.0.1 \
  --port 30000 \
  --mem-fraction-static 0.90 \
  --context-length 4096

Check the installed engine's --help for speculative-decoding flag names; vLLM and SGLang have changed those flags between releases. The checkpoint itself is the same universal FP8 export, not an engine-specific copy.

Sampling presets

Task	temperature	top_p	max tokens
tool call / patch	0.0–0.2	0.90–1.0	512–2048
debugging	0.2–0.4	0.90	512–1536
architecture	0.4–0.6	0.90–0.95	1024–2048

For agent loops, set temperature to 0 or 0.2 and validate the tool arguments before execution. For throughput tests, use temperature 0, fixed prompts, the same max_tokens, and a fresh server per row.

Measured results

A100, fixed prompt matrix, same target and tokenizer:

Engine	Target only	+ community DFlash	Change
vLLM	173.88 tok/s	242.99 tok/s	+39.7%
SGLang	171.86 tok/s	249.97 tok/s	+45.5%

The community draft is z-lab/Qwen3.5-4B-DFlash. It is external and is not claimed as a Lissa-trained checkpoint.

Our target-specific draft is documented separately, but its current acceptance and net speed are not good enough to recommend for production. Use the community draft as the current serving baseline while the on-policy Lissa draft is retrained.

When FP8 is the right choice

Choose this repository when:

the service runs on a CUDA GPU;
vLLM or SGLang is the serving engine;
throughput matters more than a small local CPU footprint;
you want to test DFlash on a GPU path where it has measured upside.

Choose GGUF for llama.cpp and MLX for Apple Silicon. Do not compare raw tok/s across those runtimes without using the same prompt, output length, sampling policy and hardware.

Long context, slots and KV cache

The ia-lab Ornith campaign is the source of the long-context test design used for Lissa1: vary slots, total context, batching and KV cache separately. Do not copy its 1M result as a Lissa1 result.

The vLLM matrix will test max_num_seqs, max_num_batched_tokens, max_model_len, prefix caching, eager mode and the runtime's actual KV-cache choices. vLLM commonly exposes FP8 KV cache, but 4-bit and integer 8-bit KV are version-dependent; unsupported probes must remain marked UNSUPPORTED, not renamed into a successful quantization.

For this 3090 campaign, use the same fixed prompt set and compare:

max_num_seqs: 1, 2, 4, 8, 16
max_num_batched_tokens: 1024, 2048, 4096, 8192
max_model_len: 4096, 8192, 32768, 131072, then larger if VRAM permits
KV: auto, fp8, int8 probe, int4 probe
prefix caching: off/on
CUDA Graph: normal vs --enforce-eager

The Ornith report's MTP result (90.09% acceptance and −42.9% accumulated latency) is not transferable: Lissa1-4B has no MTP tensors. Lissa1 must use DFlash or another supported proposer and must be measured again.

Links

Future area QLoRAs will be separate adapters over the base: database, frontend, backend, devops/cicd, agents/tools, security and platform. This FP8 repo remains the serving format for the base release.