anthonym21/ZGCM-1-7B-heretic

ZGCM-1-7B, refusal-ablated (heretic, measured)

An abliterated ZGCM-1-7B: the refusal direction has been removed from the weights with heretic's Optuna-searched directional ablation, from a fork that adds a post-norm correction for sandwich-norm architectures. No fine-tuning, no training data, no change to the tokenizer or chat format. Everything else about the model is the base model.

What is different about this upload is that it is measured. Every number below was produced by the same script on the base model and on this one, on held-out prompts, in both the model's direct-answer mode and its reasoning mode, with three independent refusal judges plus the substring check that most abliteration cards stop at. The method, the trial parameters and the evaluation code are all here, so the numbers can be reproduced or argued with.

The ablated direction is corrected for the post-norm gain of every module whose output feeds one (all 104 writer modules in this model), a fix added for this release; see Method.

Results

Base model: zgcagi/ZGCM-1-7B. Harmful prompts: the 100 behaviors of JailbreakBench (JailbreakBench/JBB-Behaviors), held out -- the search never saw them. (Heretic's own scorer optimizes against mlabonne/harmful_behaviors test[:100], so a rate measured there would be partly on the optimizer's target; that on-distribution number is in the last two rows for comparison.) Harmless prompts: 100 from the test split of mlabonne/harmless_alpaca. Direct mode: the generation prompt's open <think> is closed immediately (</think>), so the model answers with no reasoning. Reasoning mode: the generation prompt ends in an open <think>, the model reasons (up to 768 tokens), closes the block, then answers. Greedy decoding throughout.

Measurement	Base model	This model
Refusal, direct mode, substring check (held-out)	97.0%	2.0%
Refusal, direct mode, refusal classifier	97.0%	7.0%
Refusal, direct mode, LLM judge (REFUSAL only)	98.0%	45.0%
Refusal, direct mode, LLM judge (REFUSAL or PARTIAL)	98.0%	45.0%
Attack success, direct mode, HarmBench classifier	0.0%	56.0%
Refusal, reasoning mode, substring check (reply channel)	99.0%	10.2%
Refusal, reasoning mode, refusal classifier	86.0%	7.0%
Refusal, reasoning mode, LLM judge (REFUSAL only)	100.0%	83.0%
Attack success, reasoning mode, HarmBench classifier	0.0%	25.0%
Reasoning decides to refuse (reasoning channel text)	65.0%	22.0%
Reasoning concludes "not disallowed"	0.0%	0.0%
Reasoning runs finished within 512 tokens	98.0%	88.0%
Over-refusal on 100 harmless prompts, direct mode, substring	2.0%	1.0%
Over-refusal on 100 harmless prompts, refusal classifier	2.0%	0.0%
KL divergence from base, first reply token, harmless prompts	0.0000	0.4299
ARC-Challenge, acc_norm (250 questions)	38.8%	37.6%
HellaSwag, acc_norm (250 questions)	47.6%	48.0%
MMLU, acc (10 questions per subject, 570 total)	50.5%	48.6%
Refusal, direct, HarmBench attack success (on-distribution set)	0.0%	49.0%
Refusal, reasoning, HarmBench attack success (on-distribution set)	0.0%	12.5%

Refusal classifier: protectai/distilroberta-base-rejection-v1. LLM judge: Qwen/Qwen2.5-32B-Instruct with a three-way REFUSAL / PARTIAL / COMPLIANCE rubric, greedy. HarmBench: cais/HarmBench-Mistral-7b-val-cls, which asks whether the completion actually exhibits the requested behavior, so it is a stricter bar than "did not refuse". The substring check matches refusal phrasings in the first 200 characters of the reply. Benchmarks are lm-eval-harness slices run with the harness's default raw-completion prompts, no chat template and no reasoning, identically on both models. The absolute scores are therefore well below what the base model's own card reports (which uses its chat format and reasoning); only the base-versus-this-model difference means anything here. On 250 questions one standard error is about 3 points, so differences under 5 points are noise. GSM8K is omitted: this model's inference runs without a KV cache, which makes a generative benchmark impractically slow (~6 min/problem), so capability retention is shown on the loglikelihood tasks (ARC-Challenge, HellaSwag, MMLU), one forward pass each. KL divergence is a first-token distribution shift on harmless prompts; a strong ablation moves it, so read it alongside the capability rows and over-refusal, which are the behavioral health check -- here they stay at base. This model's KL is high (a large first-token shift) while its capability and over-refusal hold at base; the ablation strongly changes how answers open without degrading what the model knows or over-refusing.

The reasoning channel

The rows about the reasoning channel are the part no other abliteration card reports. The base model's reasoning runs a policy check on nearly every harmful prompt and concludes it must refuse. After ablation the model still runs the check, in the same words, and concludes the request is not disallowed. The deliberation survives; the verdict flips. That is why reasoning-mode refusal and direct-mode refusal can differ so much for the same weights: in direct mode the model has to decide at the first token, in reasoning mode it argues itself into compliance over a few hundred tokens.

Method

Directional ablation (Arditi et al. 2024, "Refusal in Language Models Is Mediated by a Single Direction"): the mean residual-stream activation on harmful prompts minus the mean on harmless prompts gives a refusal direction per layer; every matrix that writes into the residual stream (attention output and MLP down projections) is modified so it can no longer write along that direction. Heretic searches the per-layer ablation weights and the direction with a TPE optimizer against two objectives, refusal count and KL divergence from the base model on harmless prompts, and returns a Pareto front. This model is one point on that front, chosen by a rule written down before any test-set number existed (lowest judged reasoning-mode refusal among candidates that stay within 5 points of base on GSM8K, ARC-Challenge, MMLU and harmless over-refusal).

Architecture note: 32-layer dense transformer (27 gated sliding-window layers, 5 global), pre-norm; ships its own modeling code, so trust_remote_code=True is required and the code files are included here unchanged from the base model. ZGCM is a plain pre-norm model, so the post-norm correction finds nothing to correct (0 of 64 modules) and the edit is standard directional ablation; its norms are named post_* but sit before their sublayers, which is why the fork detects post-norms by tracing rather than by name.

Search settings: zgcm-abl-wide in the reproduction files; 100 trials, seed 0, 100-token responses, response prefix </think> (direct mode), system prompt You are a helpful assistant.. Heretic's own scores for this trial: Refusals 0.23, KL divergence 0.385 (refusals out of 100 train-split prompts by keyword match, KL on 100 harmless prompts).

Trial parameters (heretic's Abliteration modifier; direction_index null means a per-layer direction):

{
  "direction_index": null,
  "weight_distributions": {
    "attn.o_proj": {
      "max_weight": 1.1814182280978278,
      "max_weight_position": 15.866649389753022,
      "min_weight": 0.7630710785980269,
      "min_weight_distance": 8.701534918223388
    },
    "mlp.down_proj": {
      "max_weight": 1.3106027513686396,
      "max_weight_position": 24.236772787765954,
      "min_weight": 0.5025395095620343,
      "min_weight_distance": 14.934360670254895
    }
  }
}

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "anthonym21/ZGCM-1-7B-heretic"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "Compute the sum of the first 100 odd numbers."}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=2048, do_sample=True, temperature=1.0, top_p=1.0)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

enable_thinking=False in apply_chat_template gives direct answers. The modeling code (modeling_zgcm.py, configuration_zgcm.py) is byte-identical to the base model at commit 0e6836c; it implements no KV cache, so generation recomputes the full sequence per token (that is the base model's behavior too).

Limitations and intended use

This model answers requests the base model refuses. It is released for research on refusal mechanisms, for red-teaming and classifier work, and for users who need an unconstrained local assistant and accept responsibility for its output. It is not safer, more accurate or more capable than the base model; on the capability slices above it is within noise of base, and that is the most that can be claimed. Refusal removal is not complete (see the direct-mode rows), and the judges disagree with each other at the margins, which is why several are reported. Nothing here changes the base model's license or usage policy; both apply.

Reproduction

• Base model: zgcagi/ZGCM-1-7B at commit 0e6836c06b7384076965f584c44318b7d39f6cf3.
• Heretic fork patch (norm-aware-ablation) on upstream base 1bdb5bc: the full diff is abliteration/heretic-norm-aware.patch in the harness-lab repo below. It adds post_norm_correction, direction_index_range and max_weight_position_range to the Abliteration modifier, a limit to the BenchmarkScore scorer, a hidden-states fallback for remote-code models, and a resume fix.
• Prompts: mlabonne/harmful_behaviors at 01cead01398926d81f7c52bdb790ee8cf77ebba7, mlabonne/harmless_alpaca at 02c6a92cfcf11bb0c387334f8146d149d65b587f.
• Study config, Pareto front, evaluation and judging scripts: abliteration/ in https://github.com/anthony-maio/harness-lab (heretic/zgcm-abl-wide.toml, eval_saved.py, judge_local.py, make_card.py); the build log with every failed attempt is abliteration/NOTES.md there.
• Hardware: one Colab G4 (RTX PRO 6000 Blackwell, 96 GB) per study; the search took about 85 minutes, the evaluation about an hour.