AgentDecision-4B-v1

AgentDecision-4B-v1 is the decision model that the paper JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs froze before its registered Stage-2 tests (code name JevAgentRL-4B). It is a LoRA adapter on Qwen/Qwen3-4B. The paper's main model, AgentDecision-4B, is this checkpoint trained one more epoch with public tool-use data. Use v1 to reproduce the registered results. Use AgentDecision-4B otherwise.

A decision model answers the multiple-choice questions that an agent harness asks at each step: which tool to call next, which value an argument takes, or a yes/no question such as "is the task done?". It gives a probability for every option. The harness uses the answer when the model's confidence clears a threshold fitted on the agent's own runs, and lets the LLM decide otherwise. One answer costs one forward pass of a small model instead of an LLM call.

It was trained on the states that a customer-service agent reached in DecisionGym, including the states after its own mistakes, with the commercial decision model Jev as the teacher, and then with GRPO inside the agent loop. The model is served in Jev's System-One wire format (POST /v1/systemone), so harness code written for Jev works with it unchanged.

• Code: github.com/JevAgentHarness/JevAgentHarness (the research code is in research/)
• Paper: PDF (arXiv identifier to be added)
• Training data: AgentDecision-train. Benchmark: AgentDecisionBench. Logs of the routing proxy on coding agents: coding-agent-decisions

The models

Model	Base	Training	Use it for
AgentDecision-4B	Qwen3-4B	SFT on agent states, GRPO in the agent loop, then one epoch on agent states mixed with public tool-use data	the paper's main model
AgentDecision-4B-v1	Qwen3-4B	SFT on agent states, then GRPO in the agent loop. Frozen before the paper's Stage-2 tests	reproducing the registered Stage-2 results
AgentDecision-0.8B	StartLux-Decision-0.8B	agent states with the tools renamed in half of them, mixed with public tool-use data	a smaller and cheaper model (non-commercial base)

The code and the research records use older names: AgentDecision-4B is e12 ("mixed"), AgentDecision-4B-v1 is rl (JevAgentRL-4B, "frozen") and AgentDecision-0.8B is e28. DecisionGym is JevGym in the code.

Files

File	What it is
adapter_model.safetensors, adapter_config.json	the LoRA adapter (r 32, alpha 64, all linear layers) in PEFT format
tokenizer.json, tokenizer_config.json, chat_template.jinja	the base model's tokenizer and chat template, as used in training
calibration.json	temperatures for the option probabilities: one per question kind and number of options (choice:3-5, choice:17-52, noul:2) and a global one
thresholds.json	the confidence thresholds the paper's harness used with this model and Gemini 3.5 Flash, for tool choice (tool_select) and argument choice (tool_params), with their source
decision_config.json	the training record: base model and revision, loss, the adapter training started from
NOTICE	licences and attributions

How the model reads a question

Each question is one prompt. The model answers with one token, the label of an option, and the probabilities are read from the next-token logits at the labels:

system: You are a fast decision module inside an AI agent. Read the state and answer the question with exactly one
        token: the label of your answer.
user:   STATE:
        TASK: <the customer's request>
        HISTORY: <earlier tool calls and their results>
        CURRENT OBSERVATION: <the latest result>

        QUESTION: Which tool should the agent call next?
        OPTIONS:
        A. find_user -- Look up a customer by full name; returns user_id and tier.
        B. ship_order -- Ship a pending order.
        ...

        Answer with the option label only.

The labels are A-Z and a-z, so one prompt holds up to 52 options. A tool option is written name -- description. A yes/no question has the options yes and no. The chat template is applied with thinking turned off, and the prompt is cut to its last 4,096 tokens. The probabilities are a softmax over the label logits divided by the temperature from calibration.json. With more than 52 options (DecisionGym has 69 tools), the research server runs a knockout over chunks of 52 and combines the reads into one distribution over all options.

Usage

With transformers and peft

import json
import torch
from huggingface_hub import snapshot_download
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE, BASE_REVISION = "Qwen/Qwen3-4B", "1cfa9a7208912126459214e8b04321603b3df60c"
path = snapshot_download("JevAgentHarness/AgentDecision-4B-v1")   # adapter, tokenizer, calibration.json, thresholds.json

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(BASE, revision=BASE_REVISION, dtype=dtype)
model = PeftModel.from_pretrained(model, path).merge_and_unload().to(device).eval()
cal = json.load(open(f"{path}/calibration.json"))

SYSTEM = ("You are a fast decision module inside an AI agent. Read the state and answer the question "
          "with exactly one token: the label of your answer.")
LABELS = [chr(c) for c in range(65, 91)] + [chr(c) for c in range(97, 123)]   # A-Z, a-z: up to 52 options
CODE_IDS = [tok.encode(c, add_special_tokens=False)[0] for c in LABELS]


def bucket(k):
    return "2" if k == 2 else "3-5" if k <= 5 else "6-16" if k <= 16 else "17-52"


@torch.no_grad()
def decide(state, question, options, kind="choice"):
    """Probability per option. Tool options are written 'name -- description'; a yes/no question uses
    options ['yes', 'no'] with kind='noul'."""
    assert 2 <= len(options) <= len(LABELS), "more than 52 options: use the server (knockout over chunks of 52)"
    lines = "\n".join(f"{LABELS[i]}. {o}" for i, o in enumerate(options))
    prompt = f"STATE:\n{state}\n\nQUESTION: {question}\nOPTIONS:\n{lines}\n\nAnswer with the option label only."
    text = tok.apply_chat_template([{"role": "system", "content": SYSTEM}, {"role": "user", "content": prompt}],
                                   tokenize=False, add_generation_prompt=True, enable_thinking=False)
    ids = tok(text, add_special_tokens=False, return_tensors="pt").input_ids[:, -4096:].to(device)
    logits = model(input_ids=ids, logits_to_keep=1).logits[0, -1].float()
    t = cal["buckets"].get(f"{kind}:{bucket(len(options))}", cal["global"])   # temperature scaling
    p = torch.softmax(logits[CODE_IDS[:len(options)]] / t, -1)
    return dict(zip(options, p.tolist()))


# a state from AgentDecisionBench's common states (DecisionGym), with 6 of its 69 tools
state = ("TASK: A customer wrote in asking us to ship their pending order (express if they are gold tier, otherwise "
         "standard), but did not say who they are.\n\nHISTORY:\n[0] ask_user({\"question\": \"Could you please provide "
         "your name or customer ID so I can locate your pending order?\", \"missing\": \"customer_name\"}) -> Customer: "
         "Oh, sorry -- my name is Ivan Kim.\n\nCURRENT OBSERVATION:\nCustomer: Oh, sorry -- my name is Ivan Kim.")
options = ["find_user -- Look up a customer by full name; returns user_id and tier.",
           "search_users -- List customers whose last name starts with a prefix (display names only, no ids).",
           "get_orders -- List a customer's orders filtered by status.",
           "ship_order -- Ship a pending order.",
           "ask_user -- Ask the customer a clarifying question and wait for the reply. Use it only when information "
           "needed to act is missing or ambiguous.",
           "finish -- Call when the task is complete. Provide the final answer (or a short summary)."]
probs = decide(state, "Which tool should the agent call next?", options)
print(max(probs, key=probs.get), probs)

This prints find_user with most of the probability: the customer just gave their name, so the agent should look them up. Tested with torch 2.14, transformers 5.18 and peft 0.21. We checked this code against the research server: the adapter loads onto Qwen3-4B with every tensor matched, and the prompt tokens and option labels are identical to the server's. The same code ran end to end, with identical probabilities to the server, for AgentDecision-0.8B.

As a decision server in the Jev wire format

The research code serves the model behind POST /v1/systemone, the format Jev uses:

git clone https://github.com/JevAgentHarness/JevAgentHarness.git && cd JevAgentHarness
uv venv -p 3.11 .venv && source .venv/bin/activate
uv pip install -e "research[serve,train]"
hf download JevAgentHarness/AgentDecision-4B-v1 --local-dir agentdecision-4b-v1
cd research
python -m jevharness.training.serve_jev --model Qwen/Qwen3-4B --adapter ../agentdecision-4b-v1 \
    --calibration ../agentdecision-4b-v1/calibration.json --name agentdecision-4b-v1 --port 8090

The server keeps a cache of each episode's prompt prefix, answers all questions of a request in one batch, and runs the knockout for more than 52 options. --fast turns on the faster exact paths the paper's final harness used. One request can carry several questions about the same state:

curl -s localhost:8090/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": "TASK: Ship the pending order of the customer Ivan Kim.\n\nHISTORY:\n\nCURRENT OBSERVATION:\n(none)",
  "questions": {
    "tool": {"type": "choice", "instructions": "Which tool should the agent call next?",
             "criteria": {"find_user": "Look up a customer by full name; returns user_id and tier.",
                          "ship_order": "Ship a pending order.",
                          "ask_user": "Ask the customer a clarifying question and wait for the reply."}},
    "done": {"type": "noul", "instructions": "Is the task complete?"}}}'

The answer has a probability per option ("tool": {"choice": ..., "confidence": ..., "probabilities": {...}}) and "done": {"noul": <p(yes)>}.

In the research harness, the served model is a decision-model spec like any other:

S1='{"type": "jev", "name": "agentdecision-4b-v1", "base_url": "http://127.0.0.1:8090", "model": "agentdecision-4b-v1", "api_key": ""}'
jevbench ping --s1 "$S1"
python scripts/stage2_corpus.py replay --corpus results/stage2/s23 --name agentdecision-4b-v1 --s1 "$S1"   # answer the common states

The harness profiles with this model's thresholds are research/configs/harness/jevagent_best.yaml (the registered Stage-2 combination, backends.jevagent_rl) and research/configs/harness/jevagent_final.yaml (the final harness). research/README.md maps every table of the paper to its command.

In the JevAgentHarness proxy

The proxy in the top level of the repository (jah) takes any server that speaks the Jev wire format:

# ~/.jah/config.yaml
router:
  decision: {type: jev, provider: local, base_url: "http://127.0.0.1:8090", model: agentdecision-4b-v1}

jah decide sends one example state and prints the probabilities, which checks the connection. Keep in mind that the proxy's default question (is the agent's next step routine or hard?) and coding-agent states are outside this model's training data. Its tool_select point asks the question the model was trained on ("Which tool should the agent call next?"), but over a coding agent's tools. Run the proxy with --preset shadow first, fit the thresholds on your own logs with jah fit, and compare jah-strong with jah-auto on your tasks before relying on it (see docs/modes.md in the repository).

Calibration and thresholds

calibration.json holds temperatures, not thresholds. The server applies them with --calibration. The snippet above looks up the key <kind>:<bucket> (kind choice or noul, bucket 2, 3-5, 6-16 or 17-52 options) and falls back to global.

thresholds.json holds the thresholds of the paper's harness for this model with Gemini 3.5 Flash as the agent LLM: the model's answer is used when its calibrated confidence is at least tau, and the LLM decides otherwise (tool choice 0.5, argument choice 0.9999999). They were fitted in shadow mode, where the model answers on the agent's states without acting, on 150 DecisionGym calibration tasks. They hold only for that setting. For another agent LLM, other tools or another task family, record the model's answers in shadow mode and refit: python -m jevharness.bench.threshold_theory in the research code, or jah fit in the proxy.

Training

1. SFT on agent states. Qwen3-4B with LoRA (r 32, alpha 64, all linear layers) on AgentDecision-train v1: 15,694 decision states from 587 DecisionGym training tasks. A scripted policy repeated mistakes seen during development, so 30.9% of the states follow a mistake. Where the teacher Jev ranks the correct answer first, the target is 0.3 x the one-hot label + 0.7 x Jev's probabilities. Otherwise it is the one-hot label. The loss is cross-entropy plus 0.5 x a Brier term, for 2 epochs at learning rate 1e-4 (2.2 GPU-hours; the data cost $5.28). Script: research/scripts/train/12_sft_v1.sh.
2. GRPO in the agent loop. The SFT model makes the decisions on training tasks while Gemini 3.5 Flash-Lite does the rest. Each task runs a group of 8 episodes. The reward is task success x (1 - 0.5 min(1, LLM calls / 10)), minus 0.2 for an irreversible wrong call, with a KL penalty to the SFT model. Iteration 15 of 30 was kept by a rule fixed in advance (+4.5 GPU-hours, +$26.53 of LLM calls). GRPO added no measurable gain over SFT in task success. Scripts: research/scripts/train/70_grpo_online.sh with research/configs/train/grpo_online_round5.yaml, stage choice research/scripts/train/r5_chain.py.
3. Calibration. Temperatures fitted on the 2,377 development decisions of AgentDecision-train v1 (held-out training-seed tasks) with research/src/jevharness/training/calibrate.py.

The freeze record with every file's sha256 is research/results/gpu_round5/FREEZE_RECORD.json. The adapter weights, adapter_config.json, calibration.json and the tokenizer files here are byte-identical to it. decision_config.json was edited to remove a local path. Paper: Section 5 and Appendix D.

Results

All numbers are from the paper, in %. Agent LLM: Gemini 3.5 Flash. "Every tool choice" means the model picks every tool with no threshold, so the agent depends on it fully. In the main text the paper calls this checkpoint AgentDecision-4B where it reports the registered Stage-2 results. The appendix calls it AgentDecision-4B v1.

Level	What is measured	AgentDecision-4B-v1	Jev (teacher)	Paper
L1 isolated	accuracy on 3,991 standalone questions (BFCL v4 tool choice and arguments, When2Call)	78.8	86.4	Table 3
L1 isolated	BFCL tool-choice accuracy	88.4	89.9	Table 31 (App. G.6)
L2 open-loop	accuracy on the common states: recorded tool-choice states of 200 new DecisionGym tasks	99.2	91.5	Table 39 (App. G.9)
L2 open-loop	accuracy on DecisionGym-Shift states (renamed tools, more distractors, new task kinds)	98.6	91.8	Table 3
L2 open-loop	the same, must-ask states only (the only right answer is to ask the customer)	90.8	58.3	Table 3
L3 closed-loop	task success, every tool choice by the model, 200 new DecisionGym tasks	100.0	83.0	Table 39 (App. G.9)
L3 closed-loop	task success, every tool choice by the model, 200 shifted tasks	96.0	82.0	Table 3

Registered Stage-2 tests (Section 7.4). With this model in place of Jev, the chosen combination of delegations was non-inferior to the LLM alone (a test ruled out a loss of 3 points or more of task success) at 81.1% lower cost, on 500 new DecisionGym tasks. On the common states it beat Intern-Decision-4B and was non-inferior to Jev. On those 500 tasks it made 99.9% of the tool choices and 75.0% of the argument choices, saving 4.5 LLM calls per task (Section 7.3).

In the harness (Table 2). On a GPU shared by about four runs, the agent was 0.6 points more successful than the LLM alone at 81.1% lower cost, but slower. On its own GPU with batched questions it was 0.5 points more successful, 76.3% cheaper and 29.4% faster.

Serving cost. On one NVIDIA L4 with vLLM and FP8 weights: $0.0045 per 1,000 decisions and a median of 31 ms per answer (App. G.7, Table 33). These numbers come from vLLM with the merged adapter, not from the research server below.

The benchmark data behind the L1 and L2 numbers, every model's answers and the paper's numbers per model are in AgentDecisionBench (model id rl).

Limitations

• Trained for DecisionGym. The agent states come from DecisionGym's customer-service tasks with one agent LLM (Gemini 3.5 Flash-Lite for training, Flash for evaluation), in English. Other tools, task families or agent LLMs need new thresholds and, for the best accuracy, training on that agent's own states.
• Weaker on isolated questions. It never saw public tool-use data: 78.8% on the isolated questions against 85.1% for AgentDecision-4B.
• New rules. On DecisionGym-Hard tasks with 10 customers and an order rule it was not trained on, it made most cancel-or-ship decisions but got only 81.6% of them right, and the agent solved 3.0% of tasks against 88.0% for the LLM alone (Section 7.5, where the paper calls it AgentDecision-4B). Retraining a 4B model on the rule raised held-out accuracy on those decisions from 76.5% to 95.3% (App. D.6; that checkpoint is not released).
• Under shift the threshold did not protect success. On 200 shifted tasks, at the thresholds fitted on the original tasks, the harness with this model solved 94.0% of tasks, 6.0 points below the LLM alone, and accepted 62 wrong tool choices (App. G.9, Table 40).
• Irreversible first errors. A confident wrong choice at an irreversible step ends the task. Never make a decision model the only check on an irreversible action.
• Coding harnesses are not validated. The model was never trained or tested on coding-agent states or on the routing question of the JevAgentHarness proxy.
• Not tested against adversarial input such as injected instructions in tool output.

Licence

CC BY-NC 4.0 for the adapter, calibration.json, thresholds.json and decision_config.json.

• Qwen/Qwen3-4B is licensed under Apache-2.0. The tokenizer files (tokenizer.json, tokenizer_config.json, chat_template.jinja) come from it and stay under Apache-2.0.
• The training data contain outputs of commercial models: Jev's probabilities (TypeSafe) as soft labels, and agent text written by Gemini models (Google). The data are shared under CC BY-NC 4.0 for that reason. Check those providers' terms before any use beyond non-commercial research, in particular before training a model that competes with theirs.
• The public tool-use data come from When2Call (NVIDIA, CC BY 4.0) and ToolACE (Team-ACE, Apache-2.0).

Citation

@misc{zhu2026jevagentharnessbenchmarkingdeployingtraining,
      title={JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs},
      author={Kunlun Zhu and Tao Feng and Xu Han and Heng Wang and Ling Yang and Xiangru Tang and Jiaxuan You and Heng Ji},
      year={2026},
      eprint={2610.21412},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2610.21412},
}

The arXiv identifier will be added when it is available.