nateGeorge/flint-nvfp4-heretic

Flint 40B-A2.5B — Heretic (abliterated, NVFP4 MLX)

Refusal-direction ablation of Recursive Flint 40B-A2.5B, produced with mlx-abliterate. Runs on Apple Silicon under mlx-lm; no PyTorch, no transformers modeling code.

Refusals on 52 unseen harmful prompts: 44.2% → 13.5%, with general capability intact.

Recipe

Direction	mean last-token residual difference over 400 paired harmful/harmless prompts
Layers	8–48 (all 41)
Projections	self_attn.o_proj + shared-expert down_proj — 82 total
Strength (α)	0.75
Quantization	preserved in place (NVFP4, per-module group size / bits / mode)

No fine-tuning, no retraining, no added data. Only the component of each output projection along the refusal direction was removed: W' = W − α·v(vᵀW).

Measurements

Greedy decoding, 128 tokens per reply, 104 held-out harmful prompts split into disjoint dev (52) and test (52) sets. KL divergence is teacher-forced against the unmodified model on neutral text.

	base	this model
refusals, dev (52)	46.2%	9.6%
refusals, test (52)	44.2%	13.5%
KL mean / p90	—	1.75 / 4.46

For comparison, the same band at strength 1.0 reaches 5.8% refusals at KL 3.63, and at 1.25 reaches 0% at KL 5.86 — but 1.25 degenerates into repetition loops and is not usable. 0.75 is the last coherent point on that frontier. Pure attention ablation (no MLP) halves the refusals at KL ≈ 0.12.

Full sweep tables and the harness are in the code repo.

Usage

from mlx_lm import load, generate

model, tokenizer = load("nateGeorge/flint-nvfp4-heretic", trust_remote_code=True)

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Your question here"}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

trust_remote_code=True is required because Flint ships its architecture as flint.py. Read it before you trust it — it is the unmodified file from the base checkpoint.

CLI:

mlx_lm.generate --model nateGeorge/flint-nvfp4-heretic --trust-remote-code --prompt "..."

Roughly 23 GB on disk; comfortably fits a 64 GB Mac alongside a long context.

What this does and does not do

Abliteration removes the refusal direction from the residual stream. It does not add knowledge, and it does not make the model better at anything. It also does not make the model safe to run unsupervised — an abliterated model will attempt requests the base model declined, including ones that are illegal or harmful. The weights are published for research on refusal mechanisms and for local use on your own machine, where you are the only person affected by the output.

Capability is mostly preserved but not perfectly: expect the KL 1.75 to show up as occasional odd phrasing or a dropped detail rather than obvious breakage.

Evaluation caveats

The refusal rate is measured by a string matcher (it looks for refusal phrasing in the opening of the reply), not a judge model. Relative comparisons are meaningful; the absolute number is not comparable to a classifier-based evaluation. The 52-prompt test set gives roughly 2-point granularity.

License

Apache-2.0, inherited from the base model. Attribution to Recursive for Flint 40B-A2.5B.