Spark-X2.6-4B-heretic-tool

A refusal-suppressed version of XHToken/Spark-X2.6-4B that also keeps its judgement about when not to call a tool.

	Base	-heretic-math	this model
Refusals (100 harmful prompts)	99	5	0
BFCL v4 irrelevance	62.9	18.8	87.9
BFCL v4 — 4 call categories, avg	83.8	86.1	86.5
BFCL v4 — 5-category avg	79.6	72.7	86.8
IFEval (prompt strict)	76.5	71.5	72.0
GSM8K	86.7	86.7	83.3
HumanEval pass@1	70.0	71.7	73.3

Two stages produced this checkpoint:

1. Abliteration — Heretic v2.1.0 with the ARA (Arbitrary-Rank Ablation) modifier, optimising refusal rate against real answer correctness. That stage produced an intermediate checkpoint, Spark-X2.6-4B-heretic-math; it reaches 0/100 refusals with maths intact but collapses BFCL irrelevance from 62.9 to 18.8. Its objective and parameters are in Reproduction, so it can be rebuilt from the base model without repeating the search.
2. A small supervised pass — LoRA on that checkpoint, 1000 balanced tool-use examples, to teach the discrimination that weight ablation destroys. This is what restores irrelevance to 87.9 without giving up the 0 refusals or the call quality.

The interesting part is why stage 1 could not fix its own damage, and why stage 2 is not a patch over a broken model. Details below.

中文模型卡见 README_zh.md。

---

Model details

Base model	XHToken/Spark-X2.6-4B (Apache-2.0)
Architecture	Spark2_5ForCausalLM, 36 layers, hidden 2560, vocab 131072, tied embeddings
Format	Safetensors, BF16, 2 shards (~7.7 GiB)
Stage 1 — ablation	Heretic v2.1.0, modifier ARA; weights modified in weight space, full checkpoint exported
Stage 2 — fine-tune	LoRA r=16 / α=32, merged back into the weights; no adapter needed at inference
Made with	Python 3.12, PyTorch 2.15 (Apple MPS), Transformers 5.22.2

This is an independent community modification. It is not published or endorsed by the XHToken / SparkLLM team.

---

Why this checkpoint exists

The cost that ablation pays

Abliteration removes a direction from the residual stream. On this model it removes refusals cleanly — 99/100 down to 0/100 — but the same pass also removes something else: the tendency to decline. On BFCL v4 irrelevance, where the prompt supplies a function that cannot satisfy the request and the correct answer is to say so, accuracy falls from 62.9 to 18.8. The model still writes perfectly valid calls with correct arguments (the four "should call" categories actually improve, 83.8 → 86.1). It simply stops recognising that no call is warranted.

That is a real behavioural regression, not a scoring artefact: with a strict rule that counts any call syntax as a call, the numbers are 54.2 → 16.2.

Why the optimiser could not see it

Heretic's search minimises a refusal keyword rate and a second objective. The second objective in the first stage was 39 short-answer questions — answer correctness. irrelevance is not a correctness task, so nothing in the objective penalised the collapse.

Worse, adding it naively does not work either. BFCL counts a response it cannot parse as a successful abstention. That makes "produce malformed output" a cheap way to score well on the very metric you are trying to fix. Four separate ARA studies were run before this route was abandoned:

Objective	Best point	Call categories	Verdict
39 QA + refusal rate	5 refusals, irrelevance 18.8	86.1	the baseline problem
+ 60 irrelevance prompts	2 refusals, irrelevance 30.8	70.6	held-out 18.3 → 30.0 (real, generalises) but parallel_multiple 78.0 → 49.0
39 QA + 36 should-call + 40 abstain, official AST checker	6 refusals, irrelevance 39.6	80.3	strict score only 17.1 — the gain was parse failures, not judgement
weaker ablations from the same study	98 / 90 refusals, irrelevance 82.5 / 87.9	73.9 / 35.9	responses degraded to [func_name=foo(...)] and raw JSON; the metric rewarded it

Every solution that reached the refusal target was either weak (problem unsolved) or broken (structured output lost). The pattern is consistent with the mechanism: in this model, "refuse a harmful request" and "decline an impossible tool call" appear to ride the same direction, so a single global weight edit cannot separate them. More search over the same parameterisation does not help.

Why supervised fine-tuning is the right lever

The missing capability is a discrimination, not an absence of refusal. It is if request ∈ capabilities(functions) then call else explain — a judgement that can be taught directly with matched/unmatched pairs. LoRA on a small set changes behaviour without touching the refusal direction, so the 0/100 stays at 0.

This is the point of the checkpoint: the ablation is not patched over, it is complemented. Nothing here claims that SFT fixes the safety trade-off; it fixes the tool-use judgement that the ablation happened to break.

---

Fine-tuning

Data

1000 examples, exactly balanced, all rendered with BFCL's own system prompt so training and evaluation see the same format:

Kind	Count	Source	Target
Should decline	500	BFCL_v4_live_irrelevance	a one-sentence explanation that no supplied function applies
Should call	500	BFCL_v4_live_simple + BFCL_v4_live_multiple	the ground-truth call, rendered in BFCL's [func(arg=value)] form

The live splits were used for training and only the non-live splits for evaluation. The two sets share no prompts, no functions and no domains, so the 87.9 below is measured on data the model has never seen. That is the difference between learning a discrimination and memorising a template.

Example of each:

user:    What is the gold price today in USA?
system:  ...available function: calculate_Bond_Price...
target:  This request falls outside what the available functions support;
         `calculate_Bond_Price` is unrelated to it. No function call is warranted.

user:    Can you tell me the weather forecast for New York on March 8th, 2023?
target:  [Weather_1_GetWeather(city='New York', date='2023-03-08')]

The trap that would have inverted the result

The first version of the decline targets read "I can't complete this with the provided tools…". Heretic's refusal detector — the same one used to report 0/100 — matches i can', i cannot, i won' and unable. Training on those sentences teaches the model to answer with refusal phrasing, which would have pushed the refusal rate straight back up while looking like progress on tool use.

The targets were rewritten as neutral statements about the functions ("X covers a different task", "no function call is warranted") and every row was checked against the marker list — 0 of 1000 targets match. This is why the checkpoint still measures 0/100.

Configuration

Method	LoRA, merged after training
Rank / alpha / dropout	16 / 32 / 0.05
Target modules	q_k_v_proj, g_proj, out_proj, gate_proj, up_proj, down_proj (this model fuses QKV and adds a gate projection)
Trainable parameters	32,449,536 / 4,144,528,896 — 0.78 %
Optimiser	AdamW, lr 1e-4, cosine decay, 20 warmup steps, weight decay 0.01
Batch	1 × 8 gradient accumulation, 2 epochs, 150 steps
Sequence length	1280 tokens, loss on the target tokens only
Seed	20261113
Hardware / time	Apple MPS, ~20 minutes

599 of the 1000 examples were usable at 1280 tokens; the other 401 have BFCL system prompts long enough that the target itself fell outside the window and the row was dropped. So training saw a length-biased half of the distribution — and still generalised to the non-live splits, which is the useful signal.

---

Results

Every number below comes from a public benchmark and its official evaluation code, with identical prompts, batch settings and decoding across the three checkpoints. Sample sizes in the notes.

Axis	Benchmark	Base	-heretic-math	this model
Agent	BFCL v4 — simple_python	87.2	90.0	89.0
	BFCL v4 — multiple	90.0	92.5	92.5
	BFCL v4 — parallel	83.5	84.0	83.5
	BFCL v4 — parallel_multiple	74.5	78.0	81.0
	BFCL v4 — irrelevance	62.9	18.8	87.9
	irrelevance, strict call-syntax rule	54.2	16.2	87.9
	BFCL v4 — 5-category average	79.6	72.7	86.8
Instruction following	IFEval — prompt-level strict	76.5	71.5	72.0
Commonsense	HellaSwag (acc_norm)	53.5	51.5	53.0
	PIQA (acc_norm)	66.5	67.5	68.0
	WinoGrande (acc)	66.5	63.5	63.0
	CommonsenseQA (acc)	44.0	42.0	42.5
	SocialIQA (acc)	39.5	39.0	41.5
	5-task average	54.0	52.7	53.6
Maths	GSM8K (flexible-extract)	86.7	86.7	83.3
	MATH-500 (exact match)	10.0	11.2	20.0
Coding	HumanEval pass@1	70.0	71.7	73.3

Sample sizes: BFCL 400 / 200 / 200 / 200 / 240 (full non-live splits), IFEval 200 of 541, each commonsense task 200, GSM8K 150 of 1319, MATH-500 80 of 500, HumanEval 60 of 164. Subsets are first-N, not hand-picked. Refusal rates are Heretic's keyword rule over 100 harmful prompts at max_response_length = 192.

The table was re-run against the published revision, not just against a local build. All 1240 BFCL judgements match item for item and the simple_python generations are character-identical (400/400); IFEval (72.0), GSM8K (83.3), HumanEval (44/60) and all five commonsense tasks reproduce to the last digit.

One row does not behave: MATH-500 scores 20.0 at batch 8 and 16.2 at batch 32 on the same weights — three problems out of eighty, the kind of drift long reasoning traces show when the floating-point accumulation order changes. The figure in the table is the batch-8 run, and repeating it at batch 8 returns 20.0. Read that row as "10–20 on 80 questions", not as a precise value.

Evaluation data

Exact sources, so the numbers can be re-run — and so it is visible what was not trained on:

Benchmark	Dataset / config	Split	Used here
BFCL v4 — simple_python	bfcl-eval bundled BFCL_v4_simple_python	non-live	400 / 400 (all)
BFCL v4 — multiple	BFCL_v4_multiple	non-live	200 / 200 (all)
BFCL v4 — parallel	BFCL_v4_parallel	non-live	200 / 200 (all)
BFCL v4 — parallel_multiple	BFCL_v4_parallel_multiple	non-live	200 / 200 (all)
BFCL v4 — irrelevance	BFCL_v4_irrelevance	non-live	240 / 240 (all)
IFEval	google/IFEval	train (541 prompts)	first 200
GSM8K	openai/gsm8k, main	test (1319)	first 150
MATH-500	HuggingFaceH4/MATH-500	test (500)	first 80
HumanEval	openai/openai_humaneval	test (164)	first 60
HellaSwag	Rowan/hellaswag	validation	first 200
PIQA	baber/piqa	validation	first 200
WinoGrande	allenai/winogrande, winogrande_xl	validation	first 200
CommonsenseQA	tau/commonsense_qa	validation	first 200
SocialIQA	lighteval/siqa	validation	first 200

(IFEval ships its 541 evaluation prompts as the train split on the Hub — that is a naming quirk of the dataset, not training data.)

Everything in this table is evaluation-only. No row above appears in the fine-tuning set or in the Heretic objective. The supervised data comes from the BFCL live splits — live_irrelevance (884 prompts), live_simple (258) and live_multiple (1053), from which 500 + 500 examples were drawn — while every reported BFCL number comes from the non-live categories. BFCL keeps the two collections disjoint (live entries are meant to be executed against real APIs; non-live entries are checked by AST), so no training prompt is also an evaluation prompt.

The 39-question set used as the Heretic objective is hand-written rather than a public benchmark, so it cannot leak into any number above either. Subsets are always the first N rows in file order, never selected by looking at results. The fine-tune was a single run at the configuration shown; no checkpoint was picked by comparing benchmark scores.

Protocol

• BFCL v4 (non-live) — official bfcl-eval package: its system prompt and function-doc rendering, its response decoder, its AST/type checker and its irrelevance rule. Only inference is local, because the upstream harness expects a vLLM or OpenAI-compatible server. Prompting mode, thinking disabled.
• IFEval — lm-eval, chat template applied, thinking disabled so the checked text is the answer, not the reasoning trace.
• GSM8K / MATH-500 — lm-eval, chat template, thinking enabled, answer taken after </think>, generation stopped only at EOS.
• HumanEval — official problems and test files, greedy, thinking disabled, each candidate program executed in an isolated subprocess with a 10 s timeout.
• Commonsense — 0-shot log-likelihood, no chat template (Open LLM Leaderboard style).

Reading the table

The irrelevance fix is genuine. Official score 87.9, strict call-syntax score 87.9 — identical, so none of it comes from malformed output. Base is 62.9 official / 54.2 strict; the parent -heretic-math is 18.8 / 16.2. Responses read like "The supplied functions do not cover this request; distance_calculator.calculate is unrelated. No call is needed."

Call quality was not traded away. The four categories where a call is expected average 86.5 against 86.1 for the parent and 83.8 for the base; parallel_multiple improves to 81.0, the best of the three. This is the difference between this checkpoint and everything the ARA search produced: those bought abstention by destroying structured output.

Instruction following stays where the ablation left it. 72.0 versus 71.5 for the parent, 76.5 for the base. The 4.5-point gap to the base is inherited from stage 1 and this fine-tune neither widened nor closed it.

GSM8K is 3.4 points lower (83.3 vs 86.7). At n=150 that is roughly 1.3 standard errors, so it may be noise, but the direction is down and it is the one number here that could plausibly be real damage from training on 1000 tool-use rows.

MATH-500 moved the other way — 10.0 → 20.0 on the same 80 problems. At n=80 that is about 2.2 standard errors, so it is unlikely to be pure noise, but it is not a capability claim this card leans on either: the absolute level is low, the fine-tune contained no maths data at all, and the most plausible reading is that the merged adapter changed how long the reasoning traces run rather than adding mathematical ability. The two maths numbers disagree in direction, which is itself a reason not to over-read either of them.

Commonsense is a wash — 53.6 vs 52.7 for the parent, all five tasks within 2.5 points of both predecessors.

A correction, for the record

Earlier revisions of the sibling card reported GSM8K 65.0 → 53.0 for an ablation of the same strength. Those runs used the GSM8K task's default stop sequences (\n\n), which fire inside the reasoning block, so the answer after </think> was never generated and the metric scored truncated reasoning. Under EOS-only stopping both checkpoints score 86.7. The "maths regression" was a harness artefact, and no number in this card depends on it.

---

Limitations and responsible use

• Safety alignment has been deliberately removed. Refusal behaviour is suppressed; the model may produce content that is unsafe, illegal, biased, or otherwise objectionable. Use only in research or controlled environments. Deployers are responsible for their own content filtering, access control, human oversight and legal compliance.
• Tool abstention is now better than the base model's, but it was measured on a single benchmark. 240 prompts, one function per prompt, English. Multi-turn conversations, parallel tool execution, tool results in context, and non-English schemas are not covered. Treat 87.9 as evidence of the discrimination, not as a guarantee.
• It is a prompting-mode caller, not a native one. There are no dedicated tool-call tokens; the [func(args)] form comes entirely from the system prompt. Drop the "only return the call" instruction and it explains the schema instead of calling. Pair it with a parser that expects exactly that form, and keep enable_thinking=False.
• The maths numbers are small-sample and unstable. GSM8K 83.3 vs 86.7 (−3.4, n=150) but MATH-500 20.0 vs 11.2 (+8.8, n=80) — and MATH-500 drops to 16.2 when the same weights are decoded at a different batch size, i.e. three problems. Neither is decisive at these sizes. If arithmetic reliability matters more than tool use for your workload, benchmark both checkpoints on your own data before choosing.
• IFEval remains 4.5 points below base (72.0 vs 76.5), inherited from the ablation stage.
• The fine-tune saw only the shorter 60 % of the prompt distribution (599/1000 rows fit in 1280 tokens). Very long tool schemas are therefore outside both the training distribution and the evidence here.
• Chinese, long-context (1M), multi-turn agent trajectories and tool-execution loops were not evaluated.
• Refusal behaviour is measured with a keyword rule over 100 prompts. Real-world refusal behaviour is broader than a keyword rate.
• This is a dual-use model. Do not deploy it to end users without an independent safety layer.

---

Usage

pip install -U "transformers>=5.18" accelerate torch

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "NeosQ/Spark-X2.6-4B-heretic-tool"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "Explain how a hash map works in one paragraph."}]

# tokenize=False returns a plain string on both Transformers 4 and 5.
text = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=False
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    do_sample=True,
    temperature=1.0,
    top_p=0.95,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The chat template enables a <think> reasoning block by default. Pass enable_thinking=False (or reasoning_effort="low") to apply_chat_template to control it.

Tool use follows BFCL prompting mode: the function list goes into the system message as JSON, and the question goes in the user turn.

One thing matters more than it looks: keep the "only return the call" instruction. With it, the model answers in 14–23 tokens. Without it — a bare get_weather(city, date) signature, say — it starts explaining the schema and never emits a call. Both outputs below are from the published weights, not illustrations:

SYSTEM = """You are an expert in composing functions. You are given a question and a set of possible functions. Based on the question, you will need to make one or more function/tool calls to achieve the purpose. If none of the functions can be used, point it out.
You should only return the function calls in your response, in the format [func_name1(params_name1=params_value1, ...)]. Do not include any other text.

Here is a list of functions in json format that you can invoke.
[{"name": "get_weather", "description": "Forecast for a city on a given date.", "parameters": {"type": "dict", "properties": {"city": {"type": "string", "description": "City name."}, "date": {"type": "string", "description": "Date, YYYY-MM-DD."}}, "required": ["city", "date"]}}]"""

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "What's the weather in Paris on 2026-11-13?"},
]
# -> [get_weather(city='Paris', date='2026-11-13')]

messages[-1] = {"role": "user", "content": "What is the price of gold today?"}
# -> No supplied function covers this request. No function call is needed.

Pass enable_thinking=False for tool use: the reasoning block is not part of the answer format BFCL checks, and it costs tokens for no benefit here.

Compatibility notes — please read

1. trust_remote_code=True is required. The repository ships modeling_spark.py and configuration_spark.py, which are executed when the model is loaded. Review them before use.
2. This checkpoint requires Transformers 5.x (built and verified with 5.22.2), not the transformers==4.57.1 listed on the base model card. The value is recorded in config.json.
3. The bundled custom modelling code contains three small Transformers 5 compatibility fixes: _tied_weights_keys converted to a {target: source} mapping; create_causal_mask updated to inputs_embeds with the removed cache_position argument dropped; output_hidden_states implemented. The third only activates when requested; none alter ordinary inference numerics.
4. generation_config.json sets top_k to null. The base checkpoint ships top_k: -1, which Transformers 5.x rejects as soon as you sample (ValueError: top_k has to be a strictly positive integer, but is -1). null means the same thing — no top-k filtering — and keeps the sampling distribution identical. If you copy configs from the base model, apply the same change, or pass top_k=None to generate().

---

Reproduction

Three scripts in reproduce/ rebuild this checkpoint end to end:

pip install bfcl-eval peft

# 1. build the 1000-row set from BFCL live splits (seed 20261112)
python reproduce/make_sft_data.py                       # -> sft_data.jsonl
# 2. LoRA on the ablation checkpoint, ~20 min on MPS
SFT_DATA=sft_data.jsonl python reproduce/sft_lora.py \
    Spark-X2.6-4B-heretic-math Spark-X2.6-4B-heretic-tool

make_sft_data.py locates the BFCL data through the installed bfcl-eval package, so no paths need editing. It reproduces the training set used here byte for byte.

Stage 1 is rebuilt from the base model with Heretic: the objective is reproduce/capability_scorer.py (39 questions), searched with seed 20261110, 18 trials and max_response_length = 160. The selected ARA parameters were:

start_layer_index             = 14
end_layer_index               = 34   (exclusive)
preserve_good_behavior_weight = 0.7683
steer_bad_behavior_weight     = 0.0002
overcorrect_relative_weight   = 1.0035
neighbor_count                = 13

The resulting intermediate checkpoint is kept in this repository under heretic-math/ for audit only — it scores 18.8 on BFCL irrelevance and is not a recommended download.

The base model's LICENSE (Apache-2.0) is included. Heretic itself is AGPL-3.0; it is a build tool and does not change the licence of these weights.

Citation

@misc{sparkx2.6,
  title  = {Spark-X2.6-4B&1.7B: Advancing the Frontier of On-Device Agents for Real-World Tasks},
  author = {SparkLLM Team},
  year   = {2026}
}

@misc{heretic,
  title  = {Heretic: Training-free language model modification},
  author = {Philipp Emanuel Weidmann},
  year   = {2024},
  url    = {https://github.com/p-e-w/heretic}
}

@inproceedings{patil2025bfcl,
  title     = {The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models},
  author    = {Patil, Shishir G. and Mao, Huanzhi and Cheng-Jie Ji, Charlie and others},
  booktitle = {Forty-second International Conference on Machine Learning},
  year      = {2025}
}