Mercurius 0.5 4B Preview

Intermediate checkpoint of a Mercurius-Pipeline run. Training is paused at step 5825 of 6300, with quantization-aware training (QAT) active since step 4200.

  • Student: Qwen3.5-4B converted to a hybrid architecture: 24 GDN-2 layers plus 8 MLA layers, with VeRA adapters.
  • Teacher: Qwen3.6-35B-A3B, served by llama.cpp.
  • KV cache: FoveaKV 2+2, 10% dynamic tokens (Dyn10).
  • Corpus: 70.8M text tokens, 39.1M agentic-episode tokens and 10.4M math tokens.
  • Code: Minerva-Laboratories/Mercurius-Pipeline, branch feat/native-foveakv-evaluation.

Files

File Description
mercurius-0.5-4b-preview-foveakv-dyn10.gguf Deployable GGUF with packed Q4/Q8 weights and FoveaKV Dyn10 metadata. Requires the patched llama.cpp fork built by scripts/build_native_bench.sh. Stock llama.cpp will not load it correctly.
model.safetensors Packed Q4/Q8 weights for the PyTorch pipeline (mercurius-packed-v1). Load it with config.json, the tokenizer files and export-manifest.json, which records the checksums. This is not a standard Transformers checkpoint.
resume.pt Training resume state for the pipeline. It is not standalone: it needs the converted base model produced by the pipeline.

Benchmarks

These results were measured with llama.cpp/CUDA and FoveaKV Dyn10, using the qwen35_nonthinking_v1 decoding profile (temperature 0.7, top-p 0.8, top-k 20, presence penalty 1.5), with a 32,768-token generation limit.

Benchmark Mercurius 5825 Qwen3.5-4B, same protocol
HumanEval (164) 68.90% 84.15%
MBPP sanitized (257) 71.21% 79.38%
GSM8K (1,319) 91.96% 94.16%
RULER, 5 tasks at 32K (200) 83.38% partial
PG-19 perplexity at 65,536 (12 books) 13.09 partial

GSM8K is in-domain, because its training split is part of the corpus. On some code prompts the checkpoint can produce long repetitive outputs.

This is an intermediate checkpoint, not a final release.

Downloads last month
33
Safetensors
Model size
2B params
Tensor type
F32
ยท
I32
ยท
BF16
ยท
F16
ยท
I8
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Minerva-Laboratories/Mercurius-0.5-4B-Preview

Finetuned
Qwen/Qwen3.5-4B
Quantized
(684)
this model