Mamba-370M — GGUF

GGUF conversion of Zyphra's Mamba-370M (pure Mamba-1, no attention/MoE), included in the 1bit engine's architecture showcase as the reference case for our Vulkan Mamba-1 SSM_SCAN fix.

mamba-370m-f16.gguf

The Vulkan fix

Vulkan's Mamba-1 SSM_SCAN kernel had a correctness/performance gap that capped decode speed hard. Fixing it took this model from 25.8 tok/s to 173.5 tok/s on Strix Halo — this repo exists mainly to document and reproduce that number on real hardware.

Measured performance (Strix Halo, Vulkan, F16, this checkpoint)

pp512: 6530 tok/s · tg128: 163 tok/s

Running it

1bit serve -m mamba-370m-f16.gguf --device vulkan

Attribution

Base model: Zyphra/Mamba-370M, Apache 2.0.
GGUF conversion and the Vulkan SSM_SCAN fix: this project's engine team.
License: Apache 2.0, inherited from the base model.

Downloads last month: 102

GGUF

Model size

0.4B params

Architecture

mamba

Hardware compatibility

16-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for 1bit-MONSTER/Mamba-370M-GGUF

Base model

Zyphra/Mamba-370M

Quantized

(1)

this model