Lernex/Metis-1.6-Logos-FP8

Technical report  ·  Metis-1.6  ·  Lernex  ·  X

Logos in BF16  ·  FP8  ·  NVFP4

Metis-1.6 Logos FP8

Metis-1.6 Logos in FP8 for vLLM, SGLang and Dromos: 11.18 GB instead of 20.80 GB, so the whole 10.2B model fits on a 16 GB GPU, and it keeps 99.6% of BF16's benchmark average.

Benchmarks

Across seven zero-shot tasks, FP8 keeps 99.6% of BF16's average score.

Task	Metric	BF16	FP8
MMLU	acc	53.43	53.57
HellaSwag	acc_norm	68.46	68.02
ARC-Easy	acc_norm	70.62	70.79
ARC-Challenge	acc_norm	44.11	44.03
OpenBookQA	acc_norm	43.40	43.80
PIQA	acc_norm	76.06	76.06
WinoGrande	acc	61.96	60.14
Average		59.72	59.49

Each task is scored by log-likelihood over every item in lm-evaluation-harness 0.4.13. The FP8 column decodes this repository's weights exactly and runs them in the reference implementation with every quantized layer's input rounded to FP8 per token, as this checkpoint's quantization_config asks and as vLLM, SGLang and Dromos with --fast-math run it. The BF16 model's card scores generated answers instead, so compare these columns with each other, not with that card.

Run it

Dromos

curl -fsSL https://github.com/lernex/dromos/releases/latest/download/install.sh | sh   # Linux, macOS
dromos run logos:fp8

On Windows, install with irm https://github.com/lernex/dromos/releases/latest/download/install.ps1 | iex; pip install dromos works everywhere. logos:fp8 is Dromos's short name for this repository, which it downloads the first time, and dromos serve logos:fp8 serves the OpenAI API on http://127.0.0.1:8080/v1.

vLLM

With the vllm-metis plugin:

pip install vllm-metis
vllm serve Lernex/Metis-1.6-Logos-FP8 --trust-remote-code \
    --reasoning-parser metis --enable-auto-tool-choice --tool-call-parser metis

SGLang

With the sglang-metis plugin:

pip install sglang-metis
python -m sglang.launch_server --model-path Lernex/Metis-1.6-Logos-FP8 \
    --reasoning-parser metis --tool-call-parser metis \
    --max-mamba-cache-size 24 --max-total-tokens 32768 --cuda-graph-max-bs-decode 8 --mem-fraction-static 0.92

All three give an OpenAI-compatible endpoint with the model's thinking split out of the answer and tool calls parsed and typed. Thinking is on by default; send "chat_template_kwargs": {"enable_thinking": false} for direct answers. Recommended sampling for thinking is temperature 0.8, top-p 0.95, top-k 20 and repetition penalty 1.05, which Dromos and vLLM read from generation_config.json. For direct answers, use temperature 0.3 and repetition penalty 1.0; Dromos switches by itself.

Transformers does not read this format for this architecture; use the BF16 repository there.

What --fast-math does

This checkpoint is made for FP8 maths: each layer's input is rounded to FP8 per token and multiplied with the FP8 weights on the GPU's FP8 tensor cores. vLLM and SGLang run it that way on GPUs that have them, and that is how Benchmarks scores it.

Dromos gives you the choice. By default it expands the FP8 weights to 16 bits as it multiplies and does the maths in BF16, so the weights are the only thing rounded and the output stays closer to the original model. With --fast-math it multiplies in FP8 the way vLLM and SGLang do, which is faster on GPUs with FP8 tensor cores: the RTX 40 and 50 series, H100 and newer. On older GPUs, Apple silicon and CPUs, it changes nothing.

On an RTX 5070 Ti, Dromos 0.1.0 reads a 4,096-token prompt at 7,840 tokens per second with --fast-math, against 6,892 by default: 14% faster.

It only changes the steps where the GPU handles many tokens at once: reading the prompt, and generating for more than 8 conversations together. A single conversation writes its answer one token at a time with the default maths either way, because that step's speed is set by how fast the GPU reads the weights, not by the multiplying. The answer can still come out slightly different, since it builds on the prompt as --fast-math read it.

Turn it on when reading the prompt is the slow part: long documents, a large codebase in context, or serving many people at once. For everyday chat, leave it off; the default stays closer to the original model.

dromos serve logos:fp8 --fast-math

Once the model is downloaded, dromos doctor logos:fp8 shows what it changes on your GPU.

Speed

Decode speed on an RTX 5070 Ti (16 GB) in tokens per second, with 512 prompt tokens and 256 generated per stream, counting the tokens generated once every stream is decoding. Dromos 0.1.0, built with CUDA 13, decodes greedily without speculation; vLLM runs the same files with the vllm-metis plugin:

	Dromos	vLLM
FP8, one stream	187.4	117.9
FP8, 32 streams	1,331	1,143

BF16 Logos, at 20.8 GB, does not fit a 16 GB card: Dromos keeps the experts that don't fit in system memory and decodes 128.9 tokens/s for one stream that way.

How close it stays to BF16

Mean top-20 KL divergence from the native model's FP32 logits, and how often the top token agrees, over the whole prompt, on the release's three reference prompts: a thinking chat, a chat with a tool call and a prose passage, with this checkpoint served by each engine. Lower is closer; 0 would be the original exactly.

Engine	Thinking chat	Chat with tools	Prose
vLLM	0.0048, 100%	0.031, 91.5%	0.0084, 95.3%
SGLang	0.0055, 93.3%	0.033, 91.4%	0.0079, 95.3%

What is in FP8

A compressed-tensors FP8_DYNAMIC checkpoint: FP8 (E4M3) weights with one scale per output channel, and activations quantized per token at run time. Only the large matrices go to FP8: the routed experts, the Mamba-2 input projections and the attention projections. Everything this architecture is sensitive to stays exactly as trained: the Mamba-2 output projections (2% of the weights but most of a quantized model's error), the projections into and out of the experts' latent space, the shared expert, the routers and their balance bias, the depth memory and the embedding. The model's FP32 tensors stay FP32.

Citation

If Metis helps your work, please cite the technical report, and the model if you used it directly:

@misc{vollmer2026thinktwice,
  title         = {Think Twice: The Metis-1.6 Technical Report},
  author        = {Vollmer, Giulianno},
  year          = {2026},
  eprint        = {2610.37903},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2610.37903}
}

@misc{vollmer2026logos,
  title     = {{Metis-1.6 Logos}},
  author    = {Vollmer, Giulianno},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/Lernex/Metis-1.6-Logos}
}

License

Apache 2.0, the same as the original model.