arc-decide-0.4b-v0.2

The CPU-first arc-decide: a 0.41B model that makes the small decisions a RAG or agent app needs on every request, within 1 GB of memory on 4 CPU cores. Give it a state (text or JSON) and a few typed questions, and it returns calibrated probabilities in one forward pass, with no generated text. It covers relevance, prompt injection, grounding, routing, intent, conversation analytics, custom tags and tool use, in English, Spanish, Chinese and Russian.

It is arc-decide-0.6b-v0.1 cut to its first 16 of 28 layers, retrained on v0.1's data with every target mixed half and half with v0.1's own answers. v0.1 stays the model for GPU and API installs; this one is for hosts without a GPU. Against v0.1, on the bars set before training:

	arc-decide-0.6b-v0.1 (0.6B)	arc-decide-0.4b-v0.2 (0.41B)	change (points)	bar
DocsGPT-native relevance, nDCG@10	0.916	0.916	0.0	≥ 0.911	met
BEIR, mean of 9 sets, nDCG@10	0.570	0.572	+0.3	mean change ≥ −0.5	met
MIRACL es / zh / ru, nDCG@10	0.547 / 0.702 / 0.699	0.579 / 0.689 / 0.716	+3.2 / −1.3 / +1.7	change ≥ −1.5	met
BoolQ (open yes/no over a passage), AUROC	0.912	0.901	−1.1	≥ 0.905	missed
AggreFact grounding, AUROC	0.853	0.850	−0.3	≥ 0.845	met
deepset injection, AUROC	0.992	0.982	−0.9	≥ 0.985	missed
NotInject false positives at 0.5	12.1%	5.3%	−6.8	≤ 13.1%	met
Conversation analytics: 18 labels on two sets, macro-F1 or balanced accuracy			17 within 1 point; realistic goal met −1.4	each change ≥ −1.0	missed

Three results missed their bars and were accepted:

• BoolQ (open yes/no questions over a passage): AUROC 0.901 against v0.1's 0.912. This is the cost of 16 layers; distilling from v0.1 recovered half of what the 16-layer model lost without it (0.896).
• deepset injection: AUROC 0.982 against 0.992, on a set of 116 texts.
• Goal met, on realistic-profile conversations: balanced accuracy 1.4 points below v0.1, on a small class.

	arc-decide-0.4b-v0.2 (0.41B)	arc-decide-0.6b-v0.1 (0.6B)	Laya (421M)	Qwen3-4B-Instruct (4B)
Conversation analytics: answer status / reaction (acc)	83.6 / 85.1	83.3 / 85.4	26.4 / 34.0	60.5 / 56.0
Frustrated / goal met (AUROC)	0.988 / 0.981	0.987 / 0.981	0.715 / 0.646	0.910 / 0.885
Custom tags, unseen definitions (AUROC)	0.997	0.997	0.823	0.987
Tools: right tool / result used (AUROC)	0.813 / 0.950	0.811 / 0.954	0.456 / 0.516	0.712 / 0.791
Intent: Banking77 / CLINC unseen / MASSIVE unseen (acc)	85.7 / 97.9 / 92.7	87.5 / 97.2 / 91.9	86.5 / 98.3 / 50.9	88.9 / 97.8 / 91.5
Injection: deepset / safeguard (AUROC)	0.982 / 0.787	0.992 / 0.821	0.901 / 0.912	0.817 / 0.973
Grounding: AggreFact / RAGTruth (bacc)	0.701 / 0.687	0.712 / 0.706	0.595 / 0.482	0.656 / 0.532
Routing: needs retrieval / clarification (AUROC)	0.915 / 0.932	0.915 / 0.930	0.528 / 0.383	– / –
Relevance nDCG@10: DocsGPT docs / SciFact / NQ	0.914 / 0.716 / 0.547	0.905 / 0.725 / 0.553	0.397 / 0.378 / 0.365	0.916 / 0.592 / 0.482
Relevance nDCG@10: MIRACL es / zh / ru	0.568 / 0.568 / 0.696	0.564 / 0.579 / 0.679	0.305 / 0.222 / 0.270	– / – / –

All models get the same held-out sets and the same questions. arc-decide was trained on these kinds of question but never on these sets. Laya (a general-purpose System One checkpoint) and Qwen3-4B-Instruct (read through its label-token probabilities) answer zero-shot. Relevance is on each set's sampled "lite" view. The best is in bold; – means not run.

It powers the decisions in DocsGPT.

Quickstart

from arc_decide import Decider   # arc_decide.py ships in this repo (torch + transformers)

d = Decider("arc53/arc-decide-0.4b-v0.2")
chunk = """## Rotating API keys

To rotate a key, open **Settings → API keys**, select the key and click **Rotate**. A new key is issued
immediately; the old key keeps working for 24 hours so you can update your clients, then it is revoked."""
d.decide(
    {"query": "How do I rotate an API key?", "passage": chunk},
    {"relevant": {"type": "noul", "instructions": "Is the passage relevant to the query: does it contain "
                  "information that answers or directly addresses it?"}},
)
# {'relevant': {'type': 'noul', 'noul': 0.99}}

On CPU, the same call through the ONNX client:

import json

from huggingface_hub import hf_hub_download
from arc_decide_onnx import Decider   # arc_decide_onnx.py ships here too (onnxruntime + tokenizers + numpy)

questions = json.load(open(hf_hub_download("arc53/arc-decide-0.4b-v0.2", "questions.json")))
d = Decider("arc53/arc-decide-0.4b-v0.2")   # fetches onnx/ only
d.decide({"query": "How do I rotate an API key?", "passage": chunk}, {"relevant": questions["relevant"]})
# {'relevant': {'type': 'noul', 'noul': 0.99}}

Put every question about a state in one call: the state is encoded once and each question reuses its cache. Questions follow TypeSafe's System One shapes:

• noul: yes/no, answered with P(yes);
• choice: one of 2–16 options you pass in;
• score: a 3- or 4-level rubric.

Use the canonical questions. questions.json has every question the model was trained on, verbatim, under the name that selects its fitted temperature. They are:

Family	Question names
retrieval	relevant, grade
guardrails	injection, severity, supported
routing	needs_retrieval, needs_clarification, complexity
conversation analytics	answer_status, unanswered_reason, reaction, frustrated, asks_human, sentiment, request_type, goal_met
tool analytics	tool_needed, right_tool, tool_succeeded, result_used, induced_by_untrusted
options you supply	topic, tag, intent

States use the key names shown there:

• {"query", "passage"} for relevance;
• {"text"} for injection;
• {"source", "claim"} for grounding, one claim per call;
• {"message"} for routing.

A state is cut at 2,048 tokens, keeping its start. For a check of your own, the practised form is {"type": "noul", "instructions": "Based on the passage, answer this yes/no question:\n<question>?", "criteria": {"true": "Yes", "false": "No"}}.

On CPU

Two ONNX files ship with this model. Both have int8 weights (blocks of 32, with the four matrix multiplies most sensitive to quantisation kept in fp32); they differ in the embedding table:

• onnx/ (the default): an int8 embedding table. It fits in 1 GB on every call shape.
• onnx-bf16emb/: the embedding table kept exactly as trained (bf16), for about 0.13 GB more. On our samples its answers are no closer to PyTorch's than the default's (table below): the int8 weights account for most of the difference.

from huggingface_hub import snapshot_download

local = snapshot_download("arc53/arc-decide-0.4b-v0.2", allow_patterns=["onnx-bf16emb/*"], local_dir="arc-decide-0.4b-v0.2")
d = Decider(f"{local}/onnx-bf16emb")
d.decide({"query": "How do I rotate an API key?", "passage": chunk}, {"relevant": questions["relevant"]})
# {'relevant': {'type': 'noul', 'noul': 0.99}}

arc_decide_onnx.py needs only onnxruntime (1.23 or later), tokenizers and numpy. By default it runs in a low-memory mode that gives the same answers bit for bit: one key-value buffer shared by every pass, a single pass for single-question calls, prefill in 256-token steps, and an onnxruntime arena that grows by exactly what it needs and gives memory back after every call. Decider(path, low_memory=False) runs plain session.run calls instead. A low-memory Decider serialises its calls; use one per thread to run calls in parallel.

Peak memory (private plus file-backed, with the whole embedding table resident, as in a long-running server) and the time per call, each call shape in a fresh process:

file	guard: one injection check (171 tokens)	rerank: 20 documentation chunks (22,002 tokens)	analytics: a 2,815-token turn, 8 questions	all three, one process	same, low_memory=False	download
onnx/	683 MiB, 0.21 s	864 MiB, 27 s	969 MiB, 4.8 s	978 MiB	1,246 MiB	499 MiB
onnx-bf16emb/	825 MiB, 0.23 s	995 MiB, 30 s	1,107 MiB, 5.4 s	1,108 MiB	1,338 MiB	629 MiB

Measured on 4 vCPU of an AMD EPYC 9354P 32-Core Processor, onnxruntime 1.30.0.

The same measurement with the files and client in this repo, on an AMD EPYC 9J45, came within 25 MiB of these peaks, at about 1.8 times the speed.

A 40-chunk relevance prescreen takes about a minute on 4 cores, so a busy prescreen still belongs on a GPU.

The int8 files were checked against the fp32 PyTorch model on the same inputs, drawn from the evaluation sets of every family. A flipped decision is a yes/no answer that crossed 0.5, or a different top option:

file	sample	questions	max |Δp|	mean |Δp|	decisions flipped
onnx/	a fresh sample (drawn after the recipe was chosen)	439	0.0142	0.0010	0
onnx/	the large sample	433	0.0317	0.0012	0
onnx-bf16emb/	a fresh sample (drawn after the recipe was chosen)	439	0.0158	0.0010	0
onnx-bf16emb/	the large sample	433	0.0571	0.0012	0

The fresh-sample rows ran this repo's client on its files (AMD EPYC 9J45); the large-sample rows come from the packaging run (EPYC 9354P). int8 kernels differ slightly between processors, so the same file's numbers do too.

And on the task metrics, the default file against PyTorch on the same items:

	PyTorch	onnx/
DocsGPT-native relevance, nDCG@10	0.914	0.914
deepset injection, AUROC	0.982	0.983
NotInject false positives at 0.5	5.3%	5.3%
Analytics, an-native, 3,000 items (seeded subsample): labels within 1 point / largest change	–	9 of 9 / goal met −0.7 [−1.9, 0.0]
Analytics, an-realistic, 6,446 items: labels within 1 point / largest change	–	9 of 9 / reaction −0.2 [−0.9, +0.3]

On Apple Silicon

On a Mac, use the PyTorch client on the GPU: onnxruntime's int8 kernels are slow on Apple's CPUs.

d = Decider("arc53/arc-decide-0.4b-v0.2", device="mps", dtype="float16")

On an M1 with 16 GB, while other work was running: one injection check 0.14 s, one ~1.2k-token relevance chunk 0.8 s, 10 chunks 5.7 s, an analytics turn with 8 questions 2.2 s. Metal reserves about 1.2 GB. fp16 on the GPU matched fp32 on the CPU to within 0.0007 on every probability we checked. The int8 ONNX file on the same M1's CPU took 2.2 s, 15 s, 117 s and 18 s for the same calls.

On a GPU

Stock vLLM serves the safetensors weights, because the heads are folded into the LM head. Start it with --logprobs-mode processed_logprobs --max-logprobs 16. For each question:

1. Send the prompt as token ids, with max_tokens: 1 and allowed_token_ids set to the head's labels from arcd_serving.json.
2. Apply the bias and temperature from that file, as arc_decide.py does.

On a GPU, v0.1 is the stronger model at little extra cost.

Good to know

• Calibration. Temperatures are fitted per question on held-out data, so 0.5 means even odds. Every analytics label is within 5.8 points of calibrated on held-out conversations, except topic (10).
  • This version has not been measured on production traffic.
  • v0.1's predecessor was overconfident there on why-unanswered, reaction and request type. Report those three as rates, not thresholded per conversation, until you have checked them on your traffic.
• Not a sole security control. Injection detection is strong on known attack styles (0.98 AUROC on deepset) and weaker on novel ones (0.79–0.85). It flags 5% of benign-but-alarming text at 0.5. Use a threshold of at least 0.9 and other defences too.
• Passage length. Relevance was trained on documentation chunks, from a paragraph up to about 1,400 tokens. One-line snippets score markedly lower, so rank them against each other rather than thresholding them.
• Languages. Most labels are within 1.5 points of English in every language. The exceptions:
  • topic is 11 points behind in Russian and 5 in Spanish;
  • answer status and why-unanswered are 2–5 behind;
  • Spanish and Chinese MIRACL relevance (0.57) are the weakest multilingual scores.
• Limits. choice takes up to 16 options and score 3 or 4 levels. Questions far from the canonical ones are much less reliable, so validate them on your own data.

Every benchmark, against v0.1 and same-size specialists

Baselines were run on the same harness, following their model cards. The DocsGPT-native and analytics sets come from ten documentation sites held out of all training data. Analytics gold labels are Kimi K2.5 and DeepSeek V4 Pro where the two agree decisively, so they measure agreement with strong models, not human judgement.

Retrieval relevance (nDCG@10)

set	arc-decide-0.4b-v0.2 (0.41B)	arc-decide-0.6b-v0.1 (0.6B)	bge-reranker-v2-m3 (568M)	Qwen3-Reranker-0.6B (0.6B)	Qwen3-Reranker-4B (4B)
docsgpt-native	0.916	0.916	0.881	0.874	0.912
beir-scifact	0.750	0.756	0.731	0.762	0.783
beir-fiqa	0.335	0.338	0.398	0.390	0.457
beir-nq	0.545	0.539	0.582	0.530	0.586
beir-hotpotqa	0.764	0.759	0.787	0.756	0.780
beir-trec-covid	0.762	0.765	0.788	0.848	0.854

Prompt injection (AUROC)

set	arc-decide-0.4b-v0.2 (0.41B)	arc-decide-0.6b-v0.1 (0.6B)	Llama Prompt Guard 2 86M (86M)	ProtectAI DeBERTa-v3 injection v2 (184M)	Laya (System One encoder) (421M)
inj-deepset	0.982	0.992	0.910	0.905	0.901
inj-jailbreakhub	0.854	0.851	0.811	0.890	0.664
inj-safeguard	0.787	0.821	0.974	0.988	0.912
inj-llmail	0.999	0.997	0.986	0.986	0.990

Over-refusal: NotInject false-positive rate at 0.5 (lower is better)

set	arc-decide-0.4b-v0.2 (0.41B)	arc-decide-0.6b-v0.1 (0.6B)	Llama Prompt Guard 2 86M (86M)	ProtectAI DeBERTa-v3 injection v2 (184M)	Laya (System One encoder) (421M)
inj-notinject	5.3	12.1	5.0	43.1	19.8

Grounding: is the claim supported by the source (balanced accuracy)

set	arc-decide-0.4b-v0.2 (0.41B)	arc-decide-0.6b-v0.1 (0.6B)	HHEM-2.1-Open (~0.25B)	MiniCheck-RoBERTa-L (355M)	Bespoke-MiniCheck-7B (7B)	Laya (System One encoder) (421M)
gr-aggrefact	0.701	0.712	0.715	0.748	0.768	0.595
gr-ragtruth	0.687	0.706	0.660	0.593	0.721	0.482

Intent over options given at request time (accuracy)

set	arc-decide-0.4b-v0.2 (0.41B)	arc-decide-0.6b-v0.1 (0.6B)	GLiClass modern-base v3 (~150M)	Qwen3-4B-Instruct (label logits) (4B)	Laya (System One encoder) (421M)
ch-banking77	85.7	87.5	76.9	88.9	86.5
ch-clinc-unseen	97.9	97.2	85.6	97.8	98.3
ch-massive-unseen	92.7	91.9	49.2	91.5	50.9

Conversation analytics (held-out doc sites, four languages; macro-F1 for choices, balanced accuracy for yes/no; ECE in points)

label	arc-decide-0.4b-v0.2 (0.41B)	arc-decide-0.6b-v0.1 (0.6B)	embedding + LR (TnT-LLM), same labels (311M)	Qwen3-4B-Instruct (label logits) (4B)	Laya (System One encoder) (421M)
answer_status	81.9 / 3.5	81.8 / 3.3	52.6 / 1.9	50.8 / 38.5	16.9 / 5.1
unanswered_reason	70.0 / 1.4	69.0 / 1.4	41.2 / 2.8	51.4 / 28.9	8.6 / 25.7
reaction	64.0 / 1.5	64.5 / 1.5	27.9 / 1.9	40.9 / 40.9	22.9 / 10.8
request_type	80.1 / 3.2	79.5 / 3.5	–	–	30.5 / 20.4
sentiment	91.1 / 1.5	91.1 / 1.7	56.5 / 3.0	71.2 / 27.7	53.3 / 8.3
topic	53.7 / 10.2	52.8 / 11.1	22.6 / 6.6	37.5 / 56.5	24.3 / 21.5
frustrated	91.3 / 1.6	91.1 / 1.5	62.3 / 9.4	81.2 / 6.8	58.6 / 45.2
asks_human	96.4 / 0.5	96.3 / 0.6	77.6 / 3.7	89.9 / 3.5	70.5 / 15.4
goal_met	88.3 / 5.8	88.9 / 5.3	72.8 / 2.7	79.6 / 14.2	54.9 / 4.1
tag	98.2 / 2.2	98.3 / 2.5	69.3 / 4.9	96.7 / 3.1	74.8 / 9.3
right_tool	70.5 / 3.0	68.6 / 3.2	57.4 / 5.2	64.6 / 23.5	50.0 / 23.6
tool_succeeded	98.3 / 3.5	98.1 / 3.7	83.9 / 3.3	93.0 / 6.2	58.7 / 3.4
result_used	75.6 / 6.2	75.5 / 6.4	50.4 / 4.3	70.0 / 21.8	51.3 / 13.1

Training

• Model: the first 16 of arc-decide-0.6b-v0.1's 28 layers (407,278,592 parameters, counting the embedding once; the base model is Qwen3-0.6B-Base), with v0.1's typed linear heads (yes/no, 3- and 4-level scores, a 16-way choice) read at the last token. The questions about one state share its encoding through a prefix fork.
• Objective: v0.1's training mix, with every target mixed half and half with v0.1's own probability for that row and question (2.1M rows, 2.8M questions), then temperature-scaled per question on held-out rows against the original labels.
• Relevance: 640k public retrieval pairs plus questions over 23 documentation sites, labelled by Qwen3.6-27B, and every judged MIRACL passage for Spanish, Chinese and Russian (v0.1 kept five per query).
• Guardrails, grounding, routing and QA: public sets, translated to Spanish, Chinese and Russian.
• Analytics:
  • about 50k simulated conversations and 10k agent traces over documentation sites, generated with open models and labelled by Gemma 4 and gpt-oss-120b;
  • plus WildChat reactions.
  • No real user conversations were used.

No data was added beyond v0.1's sources. DATA.md lists every training source, its licence, and every model whose outputs are in the data.

Licence

MIT. The base model, Qwen3-0.6B-Base, is Apache-2.0; its licence ships as LICENSE-Qwen3.

@misc{arc53-arc-decide,
  title  = {arc-decide: a small calibrated decision model for RAG and agents},
  author = {Arc53},
  year   = {2026},
  url    = {https://huggingface.co/arc53/arc-decide-0.4b-v0.2}
}