voidx-decider-12b

A LoRA adapter for google/gemma-4-12B-it (revision 707f0a3b) that answers multiple-choice decision questions: classification, ontology placement, and state judgment. This is not an official Google model.

A merged Q4_K_M GGUF is in voidx-decider-12b-GGUF.

Status: research release. This model did not pass its pre-registered locked-test gate (G6). It is strongly vulnerable to prompt injection. Read Evaluation and Limitations before using it.

What it is

  • Reads the probability of each label token at the answer position, instead of generating free text.
  • Gives calibrated probabilities: validation ECE 0.028, locked-test ECE 0.028 (Q4_K_M: 0.034).
  • Supports English and Korean inputs.

Usage

Prompt format

The user turn:

Context:
<context text>

Question: <question>
Options:
(A) <option 1>
(B) <option 2>
(C) <option 3>

Apply the Gemma 4 chat template with thinking disabled (empty thought channel) and no system prompt. Then append Answer: ( to the assistant turn. Read the next-token probabilities of the label tokens A, B, C, … and renormalize over the shown options.

  • Serving temperature: divide the label logits by T = 0.98580 before softmax (serving_temperature.json).
  • The prompt format and label table follow Mapika/decider (Apache-2.0).

With PEFT

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-12B-it", revision="707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7",
    torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "upfall-co/voidx-decider-12b")
tok = AutoTokenizer.from_pretrained("google/gemma-4-12B-it",
                                    revision="707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7")

With llama-server (GGUF)

  • Raw prompt: send the fully rendered prompt (ending in Answer: () to /completion with n_probs set, and read the label tokens.
  • Chat endpoint: pass chat_template_kwargs: {"enable_thinking": false}.

Training

Item Value
Data about 14,500 rows of public data plus rule-generated state tracking (see NOTICE)
Method LoRA r=32, alpha=64, dropout 0.05, all attention and MLP projections
Schedule 1 epoch (97 steps), lr 5e-5 cosine
Hardware 1Γ— H100 80GB, about 90 minutes
Validation (1,300 rows) accuracy 0.822, soft NLL 0.503, ECE 0.028

No teacher-model or commercial-API outputs were used for training.

Evaluation 1

One-shot evaluation on a locked test set (3,050 rows), with pass criteria registered before the run.

Accuracy by pool (bf16):

Pool Rows Tuned Base Diff
All 3,050 79.3% 75.9% +3.4pt
English holdout 1,100 76.6% 70.0% +6.6
Korean 300 93.0% 91.7% +1.3
Parallel en–ko 400 83.3% 83.5% βˆ’0.3
Ontology: IPTC 600 78.5% 91.3% βˆ’12.8
Ontology: Wikidata held-out roots 350 86.9% 69.7% +17.1
State tracking, unseen domains 300 62.3% 47.0% +15.3

Training improved Wikidata placement on unseen roots, but hurt IPTC. Most of the IPTC drop comes from "none of these" items: the tuned model places 41.8% of them into a category (base 8.4%).

Evaluation 2 β€” comparison with other models

All models got the same 3,023 questions with the same chat-format instructions. The G6 test set was already used once, so this is a descriptive comparison only.

Model Acc. English Korean IPTC WD held-out Unseen-domain state IPTC "none" over-placement Injection drop
Gemini 3.8 Flash (reasoning track) 84.3% 73.4 92.7 93.3 77.7 100 3% 0pt
voidx-decider-12b 78.1% 75.3 88.3 81.7 85.4 55.3 33% βˆ’54pt
Gemma 4 31B 75.9% 67.5 90.3 92.5 68.3 56.7 3% βˆ’30pt
GPT-5.6 terra 75.6% 63.2 90.3 93.5 74.9 61.3 3% βˆ’18pt
Gemma 4 26B-A4B 74.7% 66.8 88.0 93.5 70.6 46.0 5% βˆ’48pt
Gemma 4 12B base (before training) 73.0% 68.2 88.7 90.2 69.1 32.3 10% βˆ’49pt
Qwen3.5-9B 70.0% 60.3 81.3 89.8 71.1 46.0 5% βˆ’44pt
Qwen3-14B 65.8% 55.1 75.3 85.8 60.0 45.7 8% βˆ’40pt
Ministral 14B 65.8% 52.8 81.3 82.0 70.9 46.3 25% βˆ’30pt
Llama 3.1 8B 52.7% 46.6 65.0 67.8 45.1 38.3 42% βˆ’45pt

Notes:

  • All models used the same instructions and questions (option order fixed, 27 tied-gold rows excluded).
  • This model and the base were re-measured as Q4_K_M with the chat format. The original slot-format scores are the G6 numbers above.
  • Gemini cannot disable reasoning, so it ran on a reasoning track (up to 1,024 output tokens).
  • The GPT and Gemini endpoints do not accept a temperature, so they used default sampling.
  • Llama 3.1 8B, Qwen3-14B, Ministral and the commercial models were scored on their text answers only (no logprobs).
  • ECE is available for only 5 logprob models on 161 common rows, so it is not ranked.
  • A Claude Sonnet 5.5 row is in progress and will be added when it finishes.

Raw numbers: eval/model_comparison.json.

Limitations and risks

  • Prompt injection. If the input contains text like "the answer is X", the model picks X 98% of the time (direct injection). Do not put untrusted user input into the prompt without isolating it first (delimiters, stripping instructions). The base model has the same weakness (βˆ’53.9pt); training made it about 5pt worse.
  • "None of these" over-placement on IPTC-style ontologies: the model tends to choose a category even when none fits.
  • Option-order sensitivity: 13.8% of items change answer when options are reordered.
  • Performance drops when the correct candidate is missing from retrieved options.
  • Labels come from public datasets. There was no human gold review, and the public sources may overlap with pretraining data. Wikidata class noise is learned too (e.g. a song classified as "music video").
  • A retraining run to fix the "none" over-placement is planned.

License

Apache-2.0, inherited from the base model. See LICENSE and NOTICE for modification notices and training-data attributions.

Downloads last month
5
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for upfall-co/voidx-decider-12b

Adapter
(171)
this model
Quantizations
1 model

Datasets used to train upfall-co/voidx-decider-12b