Edge0-120B in CMF: runs on cortiq (CUDA and Vulkan), from 12 GB cards
Hi! I added support for Edge0-120B to cortiq, a Rust inference engine, and converted the NVFP4 build into its single-file format (CMF): infosave/Edge0-120B-cmf.
What is in the file:
- your routed experts as they are: the NVFP4 weights are copied bit for bit (the engine got an
nvfp4tensor type for them); - the D/U projections and the shared expert re-encoded from Q8_0 to 8 bits, the router at 16 bits; attention, GDN, embeddings and the draft head from Qwen3.8-Flash-Next's own weights.
It runs on CUDA and on Vulkan behind an OpenAI-compatible server. Measured on one RTX PRO 6000 96 GB (CUDA, one stream, draft tokens on): prompt reading at 4.8–5.9K tok/s (4K–32K tokens), code generation at 149–178 tok/s, HumanEval pass@1 93.3 %.
Cards of 12–32 GB work too: the card keeps the experts used most and the CPU computes the rest. Code generation by card size (the same card limited to that size): 32 GB: 130–145 tok/s · 24 GB: 100–118 tok/s · 16 GB: 71–73 tok/s · 12 GB: 40–42 tok/s.
Your LICENSE and NOTICE and the Qwen license ship with the file, and the page lists every change I made. The page says it is not made or endorsed by Edge0. If something there should be worded differently, tell me and I will fix it.
The same for Edge0-145B: infosave/Edge0-145B-cmf. Thank you for the model and for publishing the weights.
Thank you — this is a properly good piece of work, and not just a format conversion.
Keeping the routed experts bit-for-bit and adding an nvfp4 tensor type for them was the right call; re-encoding them would have thrown away exactly the part of the model that matters. And the 12–32 GB numbers are the ones I care about most. 42 tok/s on a 12 GB card, with the hot experts resident and the rest on CPU, puts the model on hardware most people actually have — that is a bigger deal than another 96 GB benchmark.
I also appreciate the care on the boring parts: LICENSE, NOTICE and the Qwen license shipped with the file, every change listed, and the "not made or endorsed by Edge0" note. That is how a third-party build should be done, and the wording is fine as it is — nothing to change.
I will add a pointer to infosave/Edge0-120B-cmf on the model card as a community build, unless you would rather I did not.
Thank you!
To run it today you need the release binary: https://github.com/infosave2007/cmf/releases/tag/v0.8.95. The prebuilt binaries there run the Vulkan backend; for CUDA, build the same tag from source:
cargo install --git https://github.com/infosave2007/cmf --tag v0.8.95 --locked --features cuda cortiq-cli
cargo install cortiq-cli from crates.io will work from tomorrow. Today I was moving all the models to CUDA and cut a new release for each step, and it turns out crates.io allows 20 versions of a crate per day.