balllvin/flint-heretic-mlx-4bit

Flint Heretic — MLX, 4-bit

A copy of Recursive Flint with most of its refusals removed, quantised to 4-bit for Apple Silicon.

This is an abliterated model. It refuses far less than the original. Read Limitations before you use it.

Base model	RecursiveSI/flint — 39.8B parameters, ~2.5B active (mixture of experts)
This build	MLX affine 4-bit, group size 64, 8-bit embeddings — 22.6 GB, 21 GiB on disk
Context	262,144 tokens
Licence	Apache-2.0, see Licence
Runs on	Apple Silicon Mac with at least 48 GB unified memory

What this is

Flint is a 39.8-billion-parameter mixture-of-experts language model from Recursive. It writes out its reasoning before it answers. It is Apache-2.0 and was released in bf16.

This repository holds that model with its refusal behaviour mostly ablated — the technique Heretic popularised: compute the direction in activation space that the model uses when it declines a request, then remove that direction from the weights.

Quantisation is the only change beyond the ablation. The rounding is MLX's standard affine 4-bit with group size 64.

What was changed

Fifty weight matrices, in layers 24 to 48 of the 49-layer stack — the later two thirds of the network:

• 25 × self_attn.o_proj (attention output)
• 25 × mlp.shared_expert.down_proj (shared expert output)

The 256 routed experts in each layer were not touched — only matrices that write directly into the residual stream. Layers 0 and 1 use a dense mlp.down_proj instead of a shared expert, and fall outside the window in any case.

For each row of each of those matrices, the component along a per-layer refusal direction was subtracted, and the row was then rescaled to its original length. Preserving the row norm is what keeps the model coherent: without the rescale the network produces gibberish.

The refusal directions come from Heretic's method — mean activation over withheld harmful prompts minus mean activation over harmless prompts, per layer. The ablation strength was a flat λ = 4.0 across all 25 layers, with nothing applied below layer 24. Heretic's default is a smooth ramp; a hard window below layer 24 scored better on refusals and KL.

In plain words: we found the direction Flint moves in when it decides to refuse, and we subtracted it from the weights that write into that direction, in the back two thirds of the model.

Results

Held-out prompts only. 100 harmful prompts that were not used to compute the directions, greedy decoding, 100-token budget, measured on this exact 4-bit checkpoint. (The capability benchmark below uses the recommended sampling settings instead.)

	Flint (base)	Flint Heretic
Refusal rate	94.0%	22.0%	100 harmful prompts
Refusal rate, counting refusals inside the reasoning block	96.0%	31.0%	same 100
Responses that close the reasoning block	3%	51%	same 100
Closes reasoning and answers without refusing	0 / 100	47 / 100	same 100
KL(base ‖ Heretic), first token	0	0.1176	100 harmless prompts
GSM8K, 20 questions, exact final number	15/20	15/20	fixed question set
Median response length	156 chars	3,504 chars

Refusals fell 4.3×, and the 20-question maths check is unchanged. The cost is visible in the last row: the ablated model reasons for much longer, and 49 of 100 responses never close the reasoning block inside the answer budget. Almost all the residual refusals live in that failure mode — 18 of the 22 come from the 49 unclosed responses.

Capability benchmark

Base versus Heretic, both 4-bit MLX, same harness, with Recursive's recommended settings (temperature 1.0, top-p 0.95, top-k 20, up to 32,768 output tokens). Fixed subsets, scored by exact answer; 95% Wilson intervals in brackets.

Benchmark	n	Flint (base)	Flint Heretic
MMLU-Pro	70	91% (83–96)	90% (81–95)
GPQA Diamond	50	76% (63–86)	62% (48–74)
CulturalRiddles	40	18% (9–32)	22% (12–38)
GSM8K	50	92% (81–97)	92% (81–97)

Heretic matches the base on GSM8K and MMLU-Pro and scores a little higher on CulturalRiddles. GPQA Diamond drops from 76% to 62%. The intervals overlap at this sample size, but treat the hardest science questions as the place where the ablation costs capability.

Speed

Measured with the lab's plain MLX implementation on an M5 Pro (64 GB): about 35 tokens/s on one stream and about 150 tokens/s across eight streams. rec, Recursive's engine below, is faster; it has not been measured on this checkpoint yet.

Requirements

• Apple Silicon Mac (M1 or later).
• 48 GB of unified memory minimum, 64 GB or more comfortable for long contexts.
• About 25 GB of free disk for the checkpoint.
• Python 3.12 or newer, and Xcode command line tools.

Install and run

Download the checkpoint:

hf download balllvin/flint-heretic-mlx-4bit --local-dir flint-heretic-int4

Get the engine — rec, Recursive's own MLX runtime. It accepts a local checkpoint through --model-dir plus --manifest:

git clone https://github.com/recursive-oss/rec && cd rec
python3.12 -m venv .venv && .venv/bin/pip install -e engine

Chat with it:

.venv/bin/rec-mlx chat flint \
    --model-dir ../flint-heretic-int4 \
    --manifest ../flint-heretic-int4/flint-int4.json \
    --context-limit 65536

Or serve an OpenAI-compatible API:

.venv/bin/rec-mlx serve flint \
    --model-dir ../flint-heretic-int4 \
    --manifest ../flint-heretic-int4/flint-int4.json \
    --port 8080 --context-limit 65536

rec's default context limit is 8,192 tokens, which is too short for Flint's reasoning; the examples set 65,536, and up to 262,144 works on a 64 GB Mac.

flint-int4.json is the manifest rec validates the files against. It lists the sha256 of every checkpoint file here, regenerated after the ablation, so rec checks these exact bytes and nothing needs to be rebuilt locally. If pip install -e engine fails because it wants to compile the grammar module, pass --grammar none — the model does not need it.

Recommended generation settings

temperature	1.0
top-p	0.95
top-k	20
max new tokens	up to 32,768

These are the settings Recursive recommends for Flint, and they were used for every number on this page. The model thinks before it answers; a 100-token budget will often cut it off mid-reasoning, so give it room.

Limitations

• This model is deliberately less safe than the original. That is the whole point of the change, and it is your responsibility to put guardrails around it if you deploy it.
• It will comply with requests the base model refuses. Some of those completions will be harmful. There is no moderation layer in front of it.
• Abliteration is a blunt edit. Expect long, sometimes rambling reasoning, occasional repetitions, and weaker instruction-following in the affected layers' domains.
• The capability checks are small samples (40 to 70 questions per benchmark). They are evidence, not a guarantee. GPQA Diamond fell from 76% to 62%.
• 4-bit quantisation adds its own small degradation on top of the ablation.
• No fine-tuning follows the ablation. Nothing was done to repair capability.
• Flint's training data and its limitations carry over unchanged.

Licence

Apache-2.0, inherited from RecursiveSI/flint. The ablation changed weights; it did not change the terms.

Attribution:

• Flint — Recursive, https://huggingface.co/RecursiveSI/flint, Apache-2.0.
• Heretic — the ablation method, Philipp Emanuel Weidmann and contributors, https://github.com/p-e-w/heretic, AGPL-3.0. This model was produced with a from-scratch reimplementation of Heretic's method; no Heretic code is redistributed here.
• rec — the MLX runtime used to serve it, https://github.com/recursive-oss/rec, Apache-2.0.