lilcheaty/flint-abliterated

Flint abliterated BF16

This is a weight-edited derivative of RecursiveSI/flint, released under the same Apache 2.0 license. Recursive developed the base model. This derivative removes one estimated refusal direction from selected weights; it is not an official Recursive release.

This repository contains 15 BF16 safetensors shards, the shard index, configuration, tokenizer and chat template. The Q3_K_M conversion is in lilcheaty/flint-abliterated-GGUF.

Weight changes

The direction was estimated from 128 AdvBench harmful-behavior prompts and 128 input-free Stanford Alpaca instructions. The harmless instructions were shorter than 300 characters. Training, layer-selection and comparison prompts were disjoint.

Activations came from the stock Q3 llama.cpp l_out evaluation callbacks at the last prompt token, after Flint's assistant delimiter, without padding. The harmful activation mean minus the harmless mean was normalized to a unit vector. Candidate directions came from zero-based layers 12, 20, 28 and 36. The selected vector came from zero-based layer 20, the 21st block.

The edit was applied shard by shard to the official BF16 weights. It changed 12,131 tensors across all layers, not just the layer used to estimate the direction. For the unit vector r, residual-writing matrices use W <- W - r(r^T W). Embedding rows use E <- E - (E r)r^T.

The changed tensors were model.embed_tokens.weight, every attention self_attn.o_proj.weight, dense MLP mlp.down_proj.weight, routed expert mlp.experts.*.down_proj.weight, and shared expert mlp.shared_expert.down_proj.weight. The untied lm_head was unchanged. The edited weights were converted to BF16 GGUF with the Flint llama.cpp converter and quantized to Q3_K_M. Final evaluation did not use an activation hook or inference-time projection.

Prompt sources are AdvBench harmful_behaviors.csv and Stanford Alpaca alpaca_data.json. The weight-edit method follows the refusal-direction approach of Arditi et al., 2024.

Measured comparison

These results were recorded on 2026-10-22 for the Q3_K_M conversion, not direct BF16 inference. The stock comparator was an existing Unsloth UD-Q3_K_XL file. The edited model used official BF16 weights re-quantized to Q3_K_M. Quantization and possible preview/final base differences were not controlled separately.

Measurement	Stock comparator	Edited Q3_K_M
Explicit final-answer refusals on 20 held-out prompts	20/20	0/20
Refusals on 20 normal coding prompts	0/20	0/20
Completed coding responses	20/20	20/20
Completed responses across both sets	40/40	40/40
Truncated responses	0	0

The 20 held-out refusal prompts were selected from a separate pool based on completed stock refusals, then run unchanged on the edited model. The 20 coding prompts were fixed independently of model outputs. This small, selected comparison is not a general safety, compliance or coding benchmark.

A refusal meant explicit refusal or explicit withholding of the requested information in the final answer. Labels excluded thought text and quoted distress. One initial quotation false positive was manually reviewed. Some edited answers still ethically framed or redirected the request, so 0/20 explicit refusals does not mean universal compliance.

All 20 edited coding answers were inspected and were broadly sensible, not uniformly correct. Both models had an empty-string pointer bug in a C reversal answer. The stock debounce answer was buggy. The edited debounce answer had an incomplete optional variant and a correct simple alternative. The edited TypeScript answer included extra placeholder example calls, and its Fibonacci explanation corrected an initially wrong sample output. No coding correctness percentage was measured.

Both models used an RTX 3090, greedy sampling, a 16,384-token context and an 8,192-token generation limit. Neither used a system prompt, thought priming or inference-time steering. These measurements do not establish retained performance on the base model's published benchmarks.

Running the model

Flint GGUF requires llama.cpp PR #30928 until it merges. The build also needs the merged chat-template hexadecimal-escape support from #30929 and per-expert down-projection scaling from #30861. This CUDA build uses the base revision used for the conversion, with PR #30928 applied.

git clone https://github.com/ggml-org/llama.cpp
git -C llama.cpp checkout d454a204ef2b63af357744809c05af897611a72e
git -C llama.cpp fetch origin pull/30928/head:flint-pr
git -C llama.cpp diff "$(git -C llama.cpp merge-base HEAD flint-pr)" flint-pr | git -C llama.cpp apply
cmake -S llama.cpp -B llama.cpp/build -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j 4 --target llama-server llama-quantize

After Flint support merges, a llama.cpp version that includes it can replace this patched build.

Transformers modeling code is not included, as in the base checkpoint. A plain AutoModelForCausalLM.from_pretrained workflow is not supplied here. The upstream model card documents its SGLang serving kit, which is available separately from the base repository.

One conversion path is the Flint-enabled llama.cpp converter, followed by quantization. These commands produce the same quantization type as the comparison, but do not assert byte-for-byte reproducibility across tool revisions.

pip install -U huggingface_hub
hf download lilcheaty/flint-abliterated --local-dir flint-abliterated
pip install -r llama.cpp/requirements.txt
python llama.cpp/convert_hf_to_gguf.py flint-abliterated \
  --outfile flint-abliterated-BF16.gguf --outtype bf16 --use-temp-file
./llama.cpp/build/bin/llama-quantize \
  flint-abliterated-BF16.gguf flint-abliterated-Q3_K_M.gguf Q3_K_M
./llama.cpp/build/bin/llama-server \
  -m flint-abliterated-Q3_K_M.gguf \
  --ctx-size 16384 --n-gpu-layers 99 --jinja --host 127.0.0.1 --port 8080

Conversion requires disk space for the BF16 checkpoint and both GGUF files. Choose GPU offload for your hardware. Use the supplied Flint chat template.

Limitations

Removing an estimated refusal direction does not establish factual accuracy, code correctness or reliable tool use. Generated code and consequential outputs still need review. The comparison above does not measure broader capabilities or all refusal behavior.