Model card: eight benchmarks, links to code, dataset and training recipe

README.md

@@ -13,7 +13,6 @@ tags:
 - decision-model
 - web-agent
 - browser-agent
-- system-one
 - typed-decisions
 - structured-output
 - one-pass
@@ -26,9 +25,13 @@ datasets:
 
 **A one-pass decision model for web agents and structured decisions, 35B mixture of experts (3B active).**
 
-WebJev-35B-A3B reads a *state* and one or more *typed questions*, each with an explicit list of options. In one
-forward pass it returns a probability distribution over the options of every question. It does not generate text:
-there is no decoding, no output parsing and no answer outside the options you define.
+[**Code**](https://github.com/lexmount/WebJev) · [**Dataset**](https://huggingface.co/datasets/Lexmount/WebJev) ·
+[**Training**](https://github.com/lexmount/WebJev/tree/main/train)
+
+WebJev-35B-A3B reads a *state* and a *question* with an explicit list of options, and returns a probability
+distribution over the options. It does not generate text: each question is answered with a single forward pass that
+reads the logits of the option labels. There is no decoding, no output parsing, and no answer outside the options you
+list.
 
 The model is built for the decision loop of browser agents:
 
@@ -36,48 +39,46 @@ The model is built for the decision loop of browser agents:
   done, blocked).
 - **Element grounding:** which of the page's elements to act on, among up to 255 candidates.
 
-It also serves general typed decisions: routing, classification, extraction choices, policy and evidence judgments.
-
-The model speaks the TypeSafe *System One* wire format (`POST /v1/systemone`), so it is a drop-in decision backend for
-agents built on Jev.
+It also handles general typed decisions: routing, classification, extraction choices, and policy and evidence
+judgments.
 
 | | |
 |---|---|
 | Developed by | Lexmount |
-| Model type | decision model (option scoring at answer slots), sparse mixture of experts |
+| Model type | decision model (option scoring at an answer position), sparse mixture of experts |
 | Base model | [Qwen/Qwen3.5-35B-A3B-Base](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-Base) |
 | Parameters | 34.66B in total, about 3B active per token |
 | Weights | BF16, 15 safetensors shards, 69.3 GB |
 | Context | trained on inputs of up to 16,384 tokens |
 | Options per question | 2–255 |
-| Interface | `transformers`, vLLM, TypeSafe-compatible `POST /v1/systemone` |
-| Languages | English and Chinese web content; English instructions |
+| Inference | `transformers`, vLLM |
+| Languages | English instructions; English and Chinese web pages |
+| Code | [github.com/lexmount/WebJev](https://github.com/lexmount/WebJev) |
+| Training data | [Lexmount/WebJev](https://huggingface.co/datasets/Lexmount/WebJev) |
 | License | Lexmount internal (see [License](#license)) |
 
-![WebJev-35B-A3B scores every typed question about a state in one forward pass](assets/overview.png)
+![WebJev-35B-A3B turns a state and typed questions into option probabilities](assets/overview.png)
 
 ## Highlights
 
 - **Web agents.** Plugged into the same browser agent, WebJev-35B-A3B solves **38.5%** of 125 real-website tasks,
   against **16.7%** for jev-1.13.
-- **Single-step web decisions.** **98.0%** accuracy on a 500-question web decision set, against 96.4% for jev-1.13.
-- **General decisions.** WebJev leads jev-1.13 on **5 of 9** structured-decision benchmarks, with an **85.3%**
-  average. The five are JevBench, Nimble, customer-ticket triage, multi-class classification and the web decision
-  set.
-- **One pass, calibrated interface.** It returns a full distribution over the options, answers every question of a
-  request from one forward pass, and never produces an output outside the option set.
+- **General decisions.** On eight structured-decision benchmarks WebJev averages **83.7%**. It leads jev-1.13 on four
+  of them: JevBench, Nimble, customer-ticket triage and multi-class classification.
+- **One forward pass, no decoding.** Each question is answered by a single forward pass that returns a full
+  distribution over its options. The answer is always one of the options you listed.
 
 ## Evaluation
 
-All numbers are measured with the released BF16 weights served by vLLM, at temperature 1.0. The jev-1.13 numbers come
-from the TypeSafe API on the same inputs.
+All numbers are measured with the released BF16 weights served by vLLM, at temperature 1.0. The jev-1.13 numbers were
+measured through its official API on the same inputs.
 
-### Nine structured-decision benchmarks
+### Eight structured-decision benchmarks
 
 Each item gives a state and a set of candidate answers. The model's choice is correct when it equals the reference
 label. The table reports accuracy.
 
-![Accuracy of WebJev-35B-A3B and jev-1.13 on nine structured-decision benchmarks](assets/benchmarks.png)
+![Accuracy of WebJev-35B-A3B and jev-1.13 on eight structured-decision benchmarks](assets/benchmarks.png)
 
 | Benchmark (items) | What it measures | jev-1.13 | **WebJev-35B-A3B** |
 |---|---|---:|---:|
@@ -89,15 +90,16 @@ label. The table reports accuracy.
 | Multi-class decisions, dev (1,468) | news topic, review sentiment and stars, 77-way banking intent, question and entity type, yes/no reading comprehension, entailment, policy rules | 83.31 | **86.31** |
 | MMLU-Pro, 10 options (1,000) | college-level knowledge and reasoning across subjects | **83.40** | 69.40 |
 | Typed business decisions, test (2,000) | agent-trajectory stop and escalation, customer requests, invoice approval, security-alert severity; teacher labels | **74.05** | 73.90 |
-| Web decision set (500) | single-step decisions on real page snapshots of 67 English and Chinese websites: next operation, click target, input field, drop-down option | 96.40 | **98.00** |
-| **Average (equal weights)** | | **86.00** | 85.31 |
+| **Average (equal weights)** | | **84.70** | 83.72 |
 
 ### End-to-end web tasks
 
-The model serves as the decision backend of the same browser agent on 125 real-website tasks: 75 from Online-Mind2Web,
-26 from WebGym and 24 from WebVoyager. Each run has a budget of 900 seconds and 60 actions. A deterministic grader
-checks the final page state and answer. Success rate = solved ÷ evaluable tasks; tasks lost to browser infrastructure
-or grader errors are excluded, and the strict rate over all 125 tasks is shown too.
+The model serves as the decision component of the same browser agent on 125 real-website tasks: 75 from
+Online-Mind2Web, 26 from WebGym and 24 from WebVoyager. Each run has a budget of 900 seconds and 60 actions. A
+deterministic grader checks the final page state and answer.
+
+- **Success rate** = solved ÷ evaluable tasks. Tasks lost to browser infrastructure or grader errors are excluded.
+- **Strict rate** counts all 125 tasks.
 
 ![End-to-end success of WebJev-35B-A3B and jev-1.13 on 125 live-website tasks](assets/liveweb.png)
 
@@ -114,22 +116,22 @@ between the two models is well beyond it.
 
 ### Speed
 
-Measured on one NVIDIA A100 80GB with vLLM and BF16, one request at a time, with the latency of a full request
-(all questions of one state):
+Measured on one NVIDIA A100 80GB with vLLM and BF16. Requests are sent one at a time; latency covers all questions
+about one state.
 
 | Workload | p50 | p95 |
 |---|---:|---:|
 | short state, one question (claim verification) | 72 ms | 84 ms |
-| policy / knowledge question (JevBench, MMLU-Pro) | 145 ms | 150–300 ms |
+| policy or knowledge question (JevBench, MMLU-Pro) | 145 ms | 150–300 ms |
 | support ticket, three questions | 228 ms | 309 ms |
-| web page state, one decision (web decision set) | 337 ms | 755 ms |
+| web page state, one decision | 337 ms | 755 ms |
 
 ## How to use
 
 ### With `transformers`
 
-The model needs a `transformers` version that supports Qwen3.5 MoE (`qwen3_5_moe_text`, 5.15 or newer) and one GPU with
-at least 80 GB of memory.
+The model needs a `transformers` version with Qwen3.5 MoE support (`qwen3_5_moe_text`, 5.15 or newer) and one GPU
+with at least 80 GB of memory.
 
 ```python
 import json, torch
@@ -148,100 +150,52 @@ prompt = ("Context:\n" + json.dumps(state, ensure_ascii=False) +
           "\n\nQuestion: Search SpanishDict for 'spring'. Which operation should the agent perform next?\nOptions:" +
           "".join(f"\n({chr(65 + i)}) {o}" for i, o in enumerate(options)) + "\nAnswer: (")
 
-ids = tok(prompt, return_tensors="pt").to(model.device)
+inputs = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
 with torch.no_grad():
-    logits = model(**ids).logits[0, -1]
+    logits = model(**inputs).logits[0, -1]
 label_ids = [tok.encode(chr(65 + i), add_special_tokens=False)[0] for i in range(len(options))]
 probs = torch.softmax(logits[label_ids].float() / 1.0, -1)       # temperature from decider_config.json
 print(dict(zip(options, probs.tolist())))
 ```
 
-`decider/` in this repository holds the reference prompt builder, which covers every question type:
-
-- `decider.systemone.render_state` and `render_question` turn the wire format into the context and question text;
-- `plan_rows` splits a request into one scoring row per question (and per level of a score question);
-- `decider.prompt.build` produces the exact token ids and answer slots.
-
-Two conventions of the builder:
-
-- Questions with more than 10 options use single-token labels `A`–`Z`, then `AA`, `AB`, and so on.
-- Every question is scored in its own row, in the state-first layout.
+For questions with more than 10 options, labels continue as single tokens (`K` … `Z`, then `AA`, `AB`, …). Build
+those prompts with `decider/prompt.py` from this repository, which renders every label as one token.
 
 ### With vLLM
 
+Start a server that returns the logits of the allowed tokens:
+
 ```bash
 vllm serve Lexmount/WebJev-35B-A3B --dtype bfloat16 --max-model-len 34816 \
     --logprobs-mode processed_logits --max-logprobs 256
 ```
 
-To score one row:
-
-1. Send its token ids (ending at the answer slot `Answer: (`) to `/v1/completions` with `max_tokens=1`.
-2. Set `allowed_token_ids` to the row's option-label tokens and `logprobs` to the number of options.
-3. Softmax the returned label logits at temperature 1.0.
-
-### Wire format (TypeSafe System One)
-
-Served behind a TypeSafe-compatible System One adapter, which is how Lexmount deploys it, the model accepts Jev's
-request shape. `state` may be a string, an object or an
-array. Question ids are never shown to the model.
-
-```json
-POST /v1/systemone
-{
-  "model": "WebJev-35B-A3B",
-  "state": {"page": {"url": "https://www.spanishdict.com/", "title": "…", "text": "…"}, "elements": ["…"]},
-  "questions": {
-    "operation": {"type": "choice", "instructions": "Search SpanishDict for 'spring' …",
-                  "criteria": {"CLICK": "Click an element", "TYPE_TEXT": "Type into a field", "DONE": "Goal satisfied"}},
-    "done":      {"type": "noul",   "instructions": "Is the search result page already open?"},
-    "relevance": {"type": "score",  "instructions": "How relevant is the page to the task?",
-                  "criteria": ["unrelated", "partly relevant", "exactly what the task needs"]}
-  }
-}
-```
+Then score a prompt: one generated token, restricted to the option labels.
 
-The response answers every question; the values below are illustrative:
-
-```json
-{
-  "model": "decider-WebJev-35B-A3B",
-  "answers": {
-    "operation": {"type": "choice", "choice": "CLICK", "confidence": 0.97, "certainty": 0.86,
-                  "probabilities": {"CLICK": 0.97, "TYPE_TEXT": 0.02, "DONE": 0.01}},
-    "done": {"type": "noul", "noul": 0.07},
-    "relevance": {"type": "score", "score": 1.86, "confidence": 0.87, "certainty": 0.62,
-                  "legend": {"0": "unrelated", "1": "partly relevant", "2": "exactly what the task needs"},
-                  "probabilities": {"0": 0.01, "1": 0.12, "2": 0.87},
-                  "level_fit": {"0": 0.01, "1": 0.13, "2": 0.9}, "fit_mass": 1.04}
-  },
-  "usage": {"input_tokens": 1834, "output_tokens": 0}
-}
+```python
+import math, requests
+
+ids = tok(prompt, add_special_tokens=False)["input_ids"]          # the prompt above, ending with "Answer: ("
+out = requests.post("http://127.0.0.1:8000/v1/completions", json={
+    "model": "Lexmount/WebJev-35B-A3B", "prompt": ids, "max_tokens": 1, "temperature": 1.0,
+    "logprobs": len(options), "allowed_token_ids": label_ids, "return_tokens_as_token_ids": True}).json()
+top = out["choices"][0]["logprobs"]["top_logprobs"][0]           # {"token_id:<id>": logit}
+logits = [top[f"token_id:{i}"] for i in label_ids]
+z = [math.exp(x - max(logits)) for x in logits]
+print(dict(zip(options, [x / sum(z) for x in z])))
 ```
 
-- **Choice questions** take 2–255 options and return the top option, its probability (`confidence`), a certainty
-  score and the full distribution.
-- **Noul questions** are yes/no questions and return the probability of *yes*.
-- **Score questions** take 2–10 levels and return the expected level with the per-level distribution. Each level is
-  scored in its own row (`isolated_levels: true`): `level_fit` is each level's own fit probability, and the
-  probabilities normalize them.
-
 ## How it works
 
-The input is `Context: {state}`, followed by the question, the lettered options `(A) … (B) …` and an answer slot
-`Answer: (`. At the slot, the model's hidden state is projected onto the LM-head rows of the option labels only and
-normalized with a softmax over the valid labels. Labels are never generated, so every question of a request is read
-from one forward pass. Option order is part of the input; the model was trained with shuffled options, so it
-conditions on the candidates' content rather than their position.
-
-`decider_config.json` stores the inference settings:
+The input is `Context: {state}`, followed by the question, the lettered options `(A) … (B) …` and the answer position
+`Answer: (`. At that position, the model's hidden state is projected onto the LM-head rows of the option labels only
+and normalized with a softmax over the valid labels. The labels are never generated.
 
-| Setting | Value | Meaning |
-|---|---|---|
-| `temperature` | 1.0 | softmax temperature |
-| `isolated_levels` | `true` | score questions are scored one level per row |
-| `schema_first` | `false` | state-first layout |
-| `max_options` | 255 | maximum options per question |
+- **Several questions about one state** are scored as separate prompts that share the state as a prefix.
+- **Option order** is part of the input. The model was trained with shuffled options, so it conditions on the
+  candidates' content rather than their position.
+- **Inference settings** are stored in `decider_config.json`: temperature 1.0, up to 255 options per question, state
+  before question.
 
 ## Model architecture
 
@@ -258,43 +212,24 @@ conditions on the candidates' content rather than their position.
 
 ## Training
 
-WebJev-35B-A3B is a supervised fine-tune of Qwen3.5-35B-A3B-Base.
-
-- **Objective.** Cross-entropy over the option labels at each answer slot.
-- **Updated parameters.** Attention, linear attention, shared experts, routers, norms, embeddings and the LM head:
-  2.45B parameters. The routed experts keep the base weights.
-- **No reinforcement learning.**
-
-The training mixture covers:
-
-- **General typed decisions.** Public classification, entailment, question-answering, routing, tool-use and
-  policy-rule tasks in the System One question types (choice, noul, score), following the open Decider recipe.
-- **Web-agent decisions.** [WebJev](https://huggingface.co/datasets/Lexmount/WebJev): 64K
-  execution-verified action-prediction and element-grounding decisions on 1,443 live websites.
-- **Open-Jev typed decisions.** Browser control, reasoning, entity alignment, extraction and customer-support tasks.
-- **Knowledge and science.** Multiple-choice questions from Nemotron-RL-knowledge-mcqa and OpenScienceReasoning-2.
-- **Evidence-grounded decisions.** The Nimble training split.
-
-Evaluation sources were excluded from training by exact and normalized matching. This covers the MMLU-Pro test set,
-the Nimble evaluation split, the JevBench public items and the frozen evaluation contexts.
+- **Recipe, data mixture and scripts:** [github.com/lexmount/WebJev/tree/main/train](https://github.com/lexmount/WebJev/tree/main/train).
+- **Web-agent decision data:** [Lexmount/WebJev](https://huggingface.co/datasets/Lexmount/WebJev).
 
 ## Intended uses
 
 **Intended uses.**
 
-- The decision backend of browser agents: next-action prediction and element grounding.
+- The decision component of browser agents: next-action prediction and element grounding.
 - Typed decisions in software pipelines: routing, classification, extraction choices, policy and evidence checks.
   Each is a question with an explicit option list.
 
 **Out of scope.**
 
 - Free-form generation, chat and open-ended question answering.
-- Decisions whose options are not listed in the request.
+- Decisions whose options are not listed in the input.
 
 **Usage notes.**
 
-- **Explicit options.** WebJev decides among options listed in the request. It is not a chat or text-generation
-  model.
 - **Knowledge-intensive decisions.** WebJev decides from the state, so include the relevant facts in it.
 - **Confidence.** Probabilities use temperature 1.0 and rank options reliably. To act on confidence thresholds, check
   calibration on your own labels.
@@ -304,9 +239,12 @@ the Nimble evaluation split, the JevBench public items and the frozen evaluation
 ## License
 
 WebJev-35B-A3B is proprietary to Lexmount and shared privately for internal research and development; see
-[LICENSE](LICENSE). It is a fine-tuned derivative of Qwen3.5-35B-A3B-Base, which is released under the Apache License
-2.0. The prompt builder in `decider/` derives from the open-source Decider package under the same license. A copy of
-the Apache License 2.0 is in [LICENSE-APACHE](LICENSE-APACHE).
+[LICENSE](LICENSE).
+
+- The model is a fine-tuned derivative of Qwen3.5-35B-A3B-Base, released under the Apache License 2.0.
+- The prompt builder in `decider/` derives from the open-source Decider package, under the same license.
+
+A copy of the Apache License 2.0 is in [LICENSE-APACHE](LICENSE-APACHE).
 
 ## Citation
 
@@ -315,7 +253,8 @@ the Apache License 2.0 is in [LICENSE-APACHE](LICENSE-APACHE).
   title        = {WebJev-35B-A3B: A One-Pass Decision Model for Web Agents},
   author       = {{Lexmount}},
   year         = {2026},
-  howpublished = {\url{https://huggingface.co/Lexmount/WebJev-35B-A3B}}
+  howpublished = {\url{https://huggingface.co/Lexmount/WebJev-35B-A3B}},
+  url          = {https://github.com/lexmount/WebJev}
 }
 ```