WrenDecide-400M

WrenDecide-400M is a 396M-parameter English text decision model. Give it a piece of text, a question, and criteria for the candidate answers. A ModernBERT cross-encoder scores each candidate against that context. The scores are normalized across the options for that question.

It supports yes/no decisions with an insufficient-evidence option, multiple-choice classification, and ordinal scoring. Example uses include routing a support request, scoring sentiment, and flagging text for review.

Weights

Both exports come from training checkpoint step 4671. The Hub tag step-4671 pins the weights and scores documented here.

Export	Location	Weight size	Runtime
FP32	Repository root	1.58 GB	Transformers and PyTorch on CPU, CUDA, or Apple MPS
Q4	q4/	425 MB	Included loader on CUDA or Apple MPS

Q4 uses 4-bit weights in groups of 64 for the encoder's linear projections. Embeddings and the decision head remain stored in FP32. The loader unpacks one projection at a time on the GPU. The repository includes the Python inference modules used by the example below.

Quick start

Use a Python 3.12 environment with PyTorch 2.9.1, Transformers 4.57.3, and safetensors. For CUDA, use the PyTorch build for your CUDA environment. For Apple GPU inference, use the Apple Silicon build of PyTorch with MPS support.

python -m pip install 'torch==2.9.1' 'transformers==4.57.3' safetensors

The example downloads only the selected export. Authenticate with Hugging Face before downloading this private repository.

import json
import sys
from pathlib import Path

import torch
from huggingface_hub import snapshot_download

variant = "fp32"  # Set to "q4" for the smaller GPU export.
model_files = ["config.json", "model.safetensors", "tokenizer.json",
               "tokenizer_config.json", "special_tokens_map.json"]
prefix = "q4/" if variant == "q4" else ""
repo = Path(snapshot_download(
    "grufr/WrenDecide-400M",
    revision="step-4671",
    allow_patterns=["decision_model/*.py", *(prefix + name for name in model_files)],
))
sys.path.insert(0, str(repo))

from decision_model.metrics import answer
from decision_model.model import encode, load_local, score_batch
from decision_model.schema import parse_request

tokenizer, model = load_local(repo / prefix)
model.eval()
state, questions = parse_request({
    "state": "I received a new debit card. How do I get it working?",
    "questions": [{
        "id": "intent",
        "type": "choice",
        "question": "What does the customer need help with?",
        "options": [
            {"id": "activate", "label": "Activate card",
             "description": "The customer wants to start using a new card."},
            {"id": "lost", "label": "Report a lost card",
             "description": "The customer cannot find their card."},
            {"id": "delivery", "label": "Check card delivery",
             "description": "The customer is waiting for a card to arrive."},
        ],
    }],
})
question, = questions
encoded = encode(tokenizer, state, question)
if encoded.length > model.config.decision_max_tokens:
    raise ValueError("The input and candidate exceed the checkpoint's context budget.")
with torch.inference_mode():
    logits, = score_batch(model, tokenizer, [encoded], max_attention_cells=2048**2)
print(json.dumps(answer(question, logits.tolist(), {}), indent=2))

For yes/no questions, read the evidence field, which can be yes, no, or unknown. For ordinal questions, list the options in ascending order and read score for the expected level. The release includes no fitted calibration file. Treat the returned probabilities as model scores and validate thresholds on your own data.

JevBench results

FP32 and Q4 scores use the same scored cases in each row. Accuracy is higher-is-better. SST-5 uses mean absolute error of the expected sentiment level, so lower is better.

JevBench slice	Shared cases	FP32	Q4	Q4 − FP32	Other models' median
Aegis 2.0 prompts	500	83.2%	82.2%	−1.0 pp	82.4%
Aegis 2.0 responses	500	74.0%	73.4%	−0.6 pp	76.3%
ATBench500	500	30.8%	31.4%	+0.6 pp	74.2%
Banking77	170	87.1%	81.2%	−5.9 pp	85.1%
Jailbreak classification	397	86.9%	86.9%	0.0 pp	96.0%
MedMCQA	1,000	29.3%	30.1%	+0.8 pp	72.2%
MedQA-USMLE	1,000	26.4%	24.8%	−1.6 pp	79.5%
MMLU-Pro	1,000	17.8%	16.4%	−1.4 pp	64.5%
Prompt injections	116	63.8%	62.9%	−0.9 pp	71.6%
PubMedQA	500	45.2%	41.6%	−3.6 pp	65.6%
ScienceQA text	1,000	67.0%	65.8%	−1.2 pp	92.3%
SST-5 · MAE ↓	500	0.514	0.536	+0.022	0.541

Q4 scored 8,016 of 8,016 text cases. On the full Banking77 and jailbreak slices, Q4 scored 83.1% on 1,000 cases and 86.8% on 400 cases, respectively. The corresponding FP32 run had 170 and 397 usable cases. The published medians use the leaderboard's reported coverage and the same metric for each slice. Both exports skipped the 500 ScienceQA image cases and 251 VQA-RAD cases. JevBench leaderboard, JevBench methodology

For Q4 evaluation, two Tesla T4 GPUs each ran an independent model replica with FP16 autocast. The Q4 and FP32 results may therefore differ due to both weight quantization and compute precision.

Limitations

• The model accepts text inputs. Evaluation covered English tasks, and performance in other languages is unmeasured.
• Performance is weak on long tool-agent trajectories and on medical and general knowledge exams. Use human review for high-stakes decisions and measure recall on your own safety data before deployment.
• In the FP32 prompt-injection run, the model caught 20 of 60 labeled injections. Overall accuracy alone can obscure missed unsafe cases.
• Latency and memory depend on the input length and number of candidates. Each candidate requires its own context–criterion pair.

License and provenance

The upstream Ettin reranker is distributed under Apache-2.0. See the included upstream license text and the original model card. WrenDecide modifies the upstream weights through fine-tuning and provides a separate 4-bit export. The repository contains inference artifacts and the example runtime.

The license for public redistribution of the fine-tuned weights is pending review.

Contact

For questions, use the contact page.