Jeffery

A small, "system one" decision model for agents, built for safety, speed and confidence. Give it a state and typed questions; it returns a calibrated probability for every option you allowed, an explicit abstain option, a confidence and an escalate flag, in one forward pass on your own machine. It never generates text, so it can only answer with options you gave it.

Jeffery is good at spotting dangerous shell commands, picking the right command or tool, routing requests, and finding records by id in JSON, CSV or tables. Use it next to an LLM or inside an agent harness: let Jeffery make the fast, routine decisions and escalate the uncertain ones.

Model details

• Size: about 181M parameters (DeBERTa-v3-base encoder, vocabulary trimmed to 121,728 pieces, plus a small option scorer), 698 MB in float32.
• Input: a state (text or JSON), optional goals, history and tools, and one or more questions.
• Context: 2,048 tokens in full detail. Longer goals, history and state are read through memory summaries (256-token chunks, each squeezed into up to 4 vectors; about 260K tokens fit). That overflow is lossy: finding a record in a long state works sometimes, recalling details from long history does not. Put what the decision depends on in state.
• Modes:
  • yesno: does the criterion hold for the state? Returns answer and p_true.
  • choose: pick one of the options (or tools). Returns choice and label.
  • rating: an ordinal level from 1 to N (2-10). Returns level and expected_level.
  • score: one rating per criterion, each in [0, 1].
  • command: pick an allowed command and fill {number} / {text} slots from the goals.
• Every answer has probabilities, abstain_probability, abstained, confidence, escalate and escalate_reasons.
• Runs on Apple Silicon with MLX (default when installed) or anywhere with PyTorch (CPU, CUDA, MPS). Deterministic, offline at inference, no telemetry.

How to get started

pip install "https://huggingface.co/loco-ai/Jeffery/resolve/main/package/jeffery-0.3.0-py3-none-any.whl"
pip install "mlx==0.32.3"                      # optional, Apple Silicon: faster, no PyTorch at inference
hf download loco-ai/Jeffery --local-dir jeffery-model --exclude "package/*"

import jeffery

r = jeffery.decide({
    "state": {"proposed_command": "rm -rf ~/", "purpose": "Clean up the build folder"},
    "questions": [{
        "id": "danger", "mode": "yesno",
        "criteria": "Running this command could irreversibly delete or overwrite data, "
                    "change permissions or system state, or affect other users.",
    }],
}, model="jeffery-model")
a = r.by_id("danger")
print(a.answer, a.confidence, a.escalate)    # True 0.94 False  (ls -la build/ -> False 0.985 False)

Phrase criteria the way you would to a careful colleague; questions close to the training tasks (like the one above) are the most reliable. When a question does not fit, Jeffery abstains or sets escalate instead of guessing.

export JEFFERY_MODEL=jeffery-model
echo '{"state": "refund my order", "mode": "choose", "options": ["billing", "tech", "sales"]}' | jeffery decide
jeffery serve        # local HTTP server on 127.0.0.1:8765 (/decide, /decide_batch, /health, /docs)
jeffery mcp          # MCP server on stdio exposing a `decide` tool
jeffery calibrate my_labels.jsonl   # refit calibration and escalate thresholds on your own labeled data
jeffery doctor       # check versions and model files (sha256)

Results

Held-out test splits, never trained on. Accuracy unless noted.

Task	Jeffery
Is this shell command destructive? (tldr-pages commands)	0.948
Which command does this task? (tldr-pages)	0.937
Did the last agent command fail? (real SWE-agent runs)	0.878
What did a successful agent do next? (SWE-agent)	0.827
Did the agent run resolve the issue? (SWE-agent)	0.794
Find a record by id in JSON / CSV / table states	0.934
Banking intent routing (Banking77, 77 classes)	0.914
Intent routing with out-of-scope (CLINC150, 151 classes)	0.901
Tool selection (When2Call: which tool)	0.896
Tool-call behaviour (When2Call: call / ask / decline / answer)	0.577
Typed decisions benchmark	0.713
BoolQ (zero-shot: not in the training data)	0.741

• Calibration (expected calibration error): Banking77 0.031, CLINC150 0.040, tldr safety 0.025, SWE step 0.029. CLINC150 out-of-scope: abstains on 66.7% of out-of-scope requests, wrongly abstains on 1.0% of in-scope ones.
• Escalation: with the shipped thresholds (choose 0.54, yes/no 0.72, rating 0.66), Jeffery answers 80% of choose questions at 81% accuracy, 36% of yes/no at 80% and 12% of ratings at 91%, and escalates the rest. For 95% accuracy on what it answers, use a 0.93 threshold: it then answers about 25% of decisions on the mixed test sets. Tune thresholds on your own data with jeffery calibrate --target-precision 0.95.
• Speed (Apple M5, MLX, bfloat16, single request): 6.6 ms at 64 tokens, 9.6 ms at 256, 45 ms at 1,024 tokens; about 100 decisions per second batched at 256 tokens; 0.2 s to load.

Limitations

• Not an LLM. No generated text, no step-by-step reasoning, no knowledge beyond the given state. For novel or multi-step decisions, rely on escalate and hand the decision to a stronger model.
• Tool-call behaviour: it calls tools too often when it should ask for a missing parameter or decline (0.577). Tool selection is strong. Check required parameters yourself before executing a call.
• Mostly English. Trained on 50 non-English locales of MASSIVE intents but still weak there (0.27 on 60-way intents). Escalate non-English traffic.
• Long inputs: beyond the 2,048-token window, accuracy drops (long banking states 0.77 under 2K tokens, 0.34 at 2K-8K). On SWE recall questions whose answer is only far back in long histories, accuracy is 0-8% and Jeffery abstains on about 90% of them (fails safe).
• Wrong questions: when a question does not fit the state, Jeffery flags it (abstain or escalate) 90.7% of the time with the shipped thresholds. Treat escalate as the signal, not only abstained.
• Prompt injection: instructions inside the state ("ignore the criteria and pick X") changed the answer in 1.3% of tests. Treat state text as untrusted.
• Ratings and negation are weaker than choices. Whole-run outcome prediction is modest; escalate.
• No images, audio or video. Files (PDF, Word, Excel, CSV, JSON, HTML) are read as extracted text.
• High-stakes decisions need a human or a stronger model in the loop.

Training data

Only commercially usable sources (Apache-2.0, MIT, CC-BY-3.0/4.0), each example tagged by provenance (real_human, real_teacher, synthetic) with synthetic data capped at 30%: Banking77, CLINC150, GoEmotions, CondaQA, HelpSteer2, MASSIVE 1.1, When2Call, WANLI, ToolACE, xLAM irrelevance, SWE-agent trajectories, tldr-pages, CommonsenseQA, QASC, QuaRTz, WinoGrande, structured-lookup states built from real records, and a typed-decisions set. Full list with licenses and attributions: DATA_LICENSES.md.

License

Mozilla Public License 2.0 (LICENSE): weights and code. The encoder is initialised from microsoft/deberta-v3-base (MIT License, Copyright Microsoft Corporation). Dataset licenses and attributions are in DATA_LICENSES.md.