ColdTensor/Moonlight-16B-FLOE-01-GGUF

FLOE-01: Moonlight-16B-A3B-Instruct, 25% experts pruned, mixed 3 to 8-bit, QAT

ColdTensor release | public | license: MIT (inherited from base)

In plain words

6.7 GB, down from about 32 GB. That is about a fifth of the size.

This is Moonlight, a free AI chat model made by Moonshot AI. We shrank it so it runs on a single graphics card. It needs about 7 GB of the card's memory instead of about 30 GB.

On 500 school-level math word problems it had never seen, the small version got 427 right. The full-size original got 432. That gap is small enough that it could be luck.

We also checked general knowledge, following instructions and 83 everyday questions. We found no clear drop.

The weak spot is coding. In a blind side-by-side test it lost all 3 coding questions to the original. Don't count on it for code.

How we did it, in three steps:

1. Remove the parts it rarely uses (about a quarter of its specialist sections).
2. Store its numbers with less precision, like rounding, so each number takes less space.
3. Retrain it briefly so its answers match the original's.

Tested on one RTX PRO 6000 Blackwell graphics card.

Status: public release. File: Moonlight-16B-FLOE-01.gguf (6,705,647,616 bytes, sha256 2ea6cc3b3b932246386d8cbaffdda94931b25f6f2ae6405661dbab4fa10cc310).

Details for the curious

Everything below is the full technical record: every version we built, every test, and what went wrong.

Summary

Field	Value
Base model	moonshotai/Moonlight-16B-A3B-Instruct (16B total, 3B active, MoE)
Method	Combined: expert pruning, mixed-precision GGUF, distillation, quantization-aware training (QAT)
Recipe	25% of routed experts removed (64 to 48 per MoE layer). Routed expert gate/up at Q3_K, down at Q4_0 (20 layers) or Q5_0 (6 layers). 8-bit embeddings. Experts then trained at those exact quant types. Calibration: OASST chat (no GSM8K)
Size on disk	6.71 GB (6,705,647,616 bytes). Base F16: 31.93 GB
Parameters	12,360,544,992 (base 15.96B, as counted by llama.cpp)
Eval harness	llama.cpp build b12128, greedy decoding
Hardware	One RTX PRO 6000 Blackwell (96 GB). QAT run B: 102 minutes, 158 steps
Reproduce	Eval scripts not published yet

Goal

Make a big MoE fit on smaller hardware without it getting worse. Not to make it smarter. The blueprint is Nemotron Bolt-style: prune rarely used experts, quantize with mixed precision, then recover quality by training the small model to copy the full one.

Results

Main table: GSM8K-200

GSM8K-200 is 200 GSM8K test problems. Score is correct answers out of 200, graded on the last number in the answer. p values are paired exact McNemar tests (only the problems where two models disagree count). PPL is perplexity on a fixed text set; lower is better.

Model	Size	GSM8K-200	PPL	Notes
Original F16	31.93 GB	178	4.76	The ceiling
Our Q8_0	16.97 GB	169	4.76	p=0.022 vs F16
Q4_K_M + imatrix	10.54 GB	168	4.84	Without imatrix: 174. Tie with Q8
Q3_K_M + imatrix	8.29 GB	154	not recorded	Without imatrix: 141 (p=0.047)
Q2_K + imatrix	6.58 GB	26	not recorded	Without imatrix: 10. Collapses
12.5% pruned, Q8	15.06 GB	173	4.77	p=0.39 vs Q8
25% pruned, Q8	13.15 GB	162	4.89	p=0.28 vs Q8
S4: 25% pruned, mixed 2-bit	6.88 GB	141	5.10	vs same-size Q3_K_M 132 (p=0.28)
S5.4: + chat distillation	6.88 GB	144	4.94	p=0.73 vs 141
L1: + math distillation	6.88 GB	151	5.06	p=0.087 vs 141. Regressed on code
v16: same-size bit reshuffle	6.71 GB	156	4.96	p=0.020 vs 141
Loop 5: full-size teacher	6.71 GB	157	4.93	p=1.0 vs v16, a tie
FLOE-01 (QAT)	6.71 GB	169	4.92	p=0.023 vs Loop 5, p=0.12 vs F16

The GSM8K-200 set was used 6 times for decisions, so its p values are a bit optimistic. That is why we built the holdout below.

Fresh 500-problem holdout

500 unused GSM8K test problems, never seen in any training, tuning or earlier eval. Set and scoring rule were written down before scoring.

Model	Size	Correct of 500	p vs FLOE-01
Original F16	31.93 GB	432	0.60
FLOE-01 (QAT)	6.71 GB	427
Our Q8_0	16.97 GB	421	0.51
Loop 5	6.71 GB	401	0.0007
v16	6.71 GB	397	0.0004

Plainly: on grade-school math, the 6.71 GB file is not measurably different from the 31.93 GB original. "Not measurably different" is not "equal". The QAT gain over Loop 5 (41 problems only QAT got right, 15 only Loop 5 got right) is real on this set.

Regression panel: skills we never trained on

Compared against the 25% pruned Q8 model. A version only counts as a win if math goes up and nothing here drops beyond chance (p<0.05). Chat is 30 prompts judged blind by Claude; refusals is 30 harmless but edgy prompts.

Model	MMLU-500	IFEval-300	HumanEval-164	Chat-30 (W/T/L)	Refusals of 30
25% pruned Q8 (reference)	303	134	121		2
Unpruned Q8	311	134	126		not recorded
L1 math distill	285 (p=0.04)	116 (p=0.03)	100 (p=0.001)		not recorded
v16	295	132	120		not recorded
Loop 5	295 (p=0.38)	125 (p=0.24)	122 (p=1.0)	3 / 21 / 6 (p=0.51)	1
FLOE-01 (QAT)	289 (p=0.10)	124 (p=0.18)	126 (p=0.38)	10 / 10 / 10 (p=1.0)	1

FLOE-01 has no significant drop on any panel task. It is 14 points lower on MMLU and 10 lower on IFEval than the pruned Q8, which could be chance at this sample size but is the direction to watch.

Real tasks: 83 blind prompts

83 real chat prompts no model had seen (planning, explanation, health, factual, how-to, writing, opinion, finance, coding). Claude judged pairs blind, without model names. Rules were written before generating.

Comparison	FLOE-01 wins / ties / losses	p
vs original F16	28 / 26 / 29	1.00
vs Loop 5	24 / 37 / 22	0.88

Rubric mean 3.14 vs F16 3.26 (p=0.21). No category was significant. FLOE-01 lost all 3 coding prompts. Both FLOE-01 and the original made factual mistakes at about the same rate.

Speed and memory

llama-bench, mean of 3, all weights on GPU, one RTX PRO 6000 Blackwell. CPU rows use a Ryzen 9 9950X, 16 threads.

Model	Generation tg128	Prompt pp512	Peak VRAM (ctx 4096)	CPU-only generation
Original F16	204.3 tok/s	4,805 tok/s	30,836 MiB	not measured
Our Q8_0	330.7 tok/s	6,775 tok/s	16,868 MiB	22.64 tok/s
FLOE-01	360.1 tok/s	6,781 tok/s	7,086 MiB	33.67 tok/s

Generation is 9% faster than Q8 and 76% faster than F16. The bigger change is memory: 6.9 GiB of VRAM instead of 16.5 or 30.1 GiB.

Method, in order

1. Rebuilt our own F16 and Q8. A public Q8 conversion scored 72/200. Our own conversion of the same model scored 169. Bad conversions are a real risk; check yours.
2. Expert usage map. Profiled which routed experts the router picks on non-GSM8K text. Routing was fairly balanced (mean Gini 0.233). The quietest 12.5% could be cut in all 26 MoE layers and the quietest 25% in 21 of 26 without touching an always-on expert. Shared experts were left alone.
3. Pruned 25% of routed experts (REAP-style), 64 to 48 per layer. At Q8: 162/200 vs 169 (p=0.28). File 13.15 GB.
4. Mixed-precision GGUF (S4). Routed experts near 2-bit, attention, shared experts, routers and some layers higher, embeddings and output at F16. 141/200 at 6.88 GB. Not better than a same-size plain Q3_K_M (132, p=0.28). Official fail.
5. Distillation loops. A LoRA trained to copy the teacher, then merged and requantized. Chat data: 144. Math answers written by the teacher, kept only when correct: 151. More passes, a bigger learning rate, or a math and chat mix did not beat it on a 400-problem dev set (345, 350, 345 vs 352).
6. Sensitivity map and same-size bit reshuffle (v16). Raising groups to 8-bit one at a time showed the 2-bit squeeze was not the main loss: even all 8-bit (13.8 GB) gained only 10 dev points (p=0.11). Reshuffled bits at the same budget: 8-bit embeddings, slimmer shared experts, 3-bit big expert weights. 156/200 at 6.71 GB, no panel regressions.
7. Full-size teacher (Loop 5). Distilled from the original unpruned model instead of the pruned copy. 157/200, a tie with v16 (p=1.0).
8. QAT of routed experts (FLOE-01). Made the 10,796,138,496 routed expert weights trainable while every forward pass used them exactly as the file stores them, quantized by the same llama.cpp code and importance matrix. Teacher: the original BF16 model, live, full-vocabulary KL loss. Full-precision master weights and Adam state lived in system RAM. Learning rate 2e-6 (a first run at 2e-5 diverged and was stopped at step 24). 158 steps, 102 minutes (a pre-run probe peaked at 79.6 GiB of GPU memory). Changed 1.43% of the expert file's bytes; every other byte is identical to Loop 5.

What failed, and why

• Uniform 2-bit. Q2_K scored 26/200 with an imatrix and 10 without; many answers ran to the token cap. Moonlight does not survive the same 2 bits everywhere.
• First mixed-precision file (S4). 141 at 6.88 GB did not beat a plain Q3_K_M of similar size, and it dropped stray Chinese characters into 194 of 200 English answers. Chat distillation removed those (194 to 0) but barely moved math (144, p=0.73), so it failed its gate.
• Math distillation hid a code regression (L1). 151 on math, but HumanEval fell from 121 to 100 (p=0.001), with drops on IFEval and MMLU too. The cause was the old S4 bit recipe, not the math training: the same weights rebuilt with the v16 recipe showed no drops. Lesson: a math-only score can hide damage elsewhere. Always run the panel.
• A better teacher alone did nothing (Loop 5). The original model as teacher moved the student's answers 46% closer to it on a math probe, but the score tied v16 (157 vs 156). Training the experts themselves (QAT) is what moved the number.

Known regressions and limits

• Coding is weak. 0 of 3 coding prompts won in the blind real-task check. HumanEval held (126 vs 121), but the blind check had only 3 coding prompts, so we can't call coding safe either way.
• MMLU and IFEval are down 14 and 10 points vs the pruned Q8. Not significant, still worth knowing.
• Answer format. FLOE-01 often skips the #### answer marker (strict GSM8K-200 score 129, vs 169 on the last-number grade). Anything that parses that marker will see a worse model.
• The 400-problem dev set did not go up with QAT (364 vs Loop 5's 366). The gain shows on the official 200 and the fresh 500, not on dev. We don't have an explanation.
• GSM8K-heavy. Most of the tuning signal is grade-school math. Treat this as "close to the original on GSM8K-style math and our real-task mix", not "as good at everything".
• One base model. This recipe has only been run on Moonlight. It may not transfer.
• Judge. Chat and real-task comparisons were judged by Claude. Small sets (30 and 83 prompts).
• One machine. All numbers come from one RTX PRO 6000 Blackwell. Smaller cards were not tested.

How to run

Needs a llama.cpp build that supports Moonlight (DeepSeek2 architecture). We tested build b12128.

huggingface-cli download ColdTensor/Moonlight-16B-FLOE-01-GGUF Moonlight-16B-FLOE-01.gguf --local-dir .
./llama-server -m Moonlight-16B-FLOE-01.gguf -ngl 999 -c 4096
# or chat in the terminal
./llama-cli -m Moonlight-16B-FLOE-01.gguf -ngl 999 -c 4096 --temp 0 -cnv

Use the base model's chat template (built into the file). Greedy decoding (temperature 0) matches how we scored it. Peak GPU memory at 4,096 context was 7,086 MiB.

License

The base model moonshotai/Moonlight-16B-A3B-Instruct is MIT licensed (per its Hugging Face model card metadata). This derivative keeps the MIT license and its copyright notice. Calibration and distillation data: OASST chats and GSM8K train answers written by the base model. Holdout items come from the GSM8K test split (MIT).

Eval scripts

Eval scripts not published yet.