rorylshanks commited on
Commit
04b9f87
·
verified ·
1 Parent(s): 27c0c79

Model card for Jeeves v2

Browse files
Files changed (1) hide show
  1. README.md +41 -25
README.md CHANGED
@@ -14,9 +14,11 @@ tags:
14
 
15
  # Jeeves-9B
16
 
17
- A Jev-like decision model that reasons before it decides. Give it a state (text or JSON) and yes/no, multiple-choice or rating questions; it writes a reasoning chain per question and returns a calibrated probability for every option.
18
 
19
- This repository holds the fused weights (Qwen3.5-9B with the LoRA merged), the pointer head, the fitted temperature, and two speculative-decoding drafters. It loads with the Jeeves code, not with `transformers`: the pointer head and the prompt format are part of the model.
 
 
20
 
21
  ## Files
22
 
@@ -24,9 +26,9 @@ This repository holds the fused weights (Qwen3.5-9B with the LoRA merged), the p
24
  |---|---|
25
  | `model-*.safetensors`, `config.json` | fused Qwen3.5-9B weights, bf16 |
26
  | `head.pt` | pointer head (query and key projections, 256-dim) |
27
- | `export.json` | temperature (1.859), prompt format, training step and config |
28
  | `tokenizer.json`, `vocab.json`, `merges.txt`, `tokenizer_config.json`, `chat_template.jinja` | Qwen3.5 tokenizer |
29
- | `drafter_k4.safetensors` | block-4 drafter, the serving default, bf16 |
30
  | `drafter_k8.safetensors` | block-8 drafter, faster for single-question requests, bf16 |
31
  | `drafter_k4.config.json`, `drafter_k8.config.json` | drafter training settings |
32
  | `LICENSE` | Apache-2.0, inherited from Qwen3.5-9B |
@@ -52,41 +54,55 @@ The request and response follow Jev's `/v1/systemone` format. An optional `optio
52
 
53
  ## Results
54
 
55
- Accuracy with thinking, greedy, 2,560-token cap. Kev-9B and Jev numbers are the ones Kev publishes. JevBench uses the same 231 public items for every model (the sealed judge tier is not included); the other rows use the same sources with different items.
56
 
57
- | | Kev-9B | Jev | Jeeves-9B |
58
- |---|---|---|---|
59
- | Test overall (out-of-domain and held-out) | 0.822 | 0.857 | **0.889** |
60
- | MMLU-Pro and buried state | 0.579 | **0.800** | 0.746 |
61
- | JevBench, public tiers | 0.715* | 0.866 | **0.935** |
62
- | JevBench hard | 0.451* | 0.730 | **0.865** |
63
 
64
  \* Kev-8B (Qwen3); no Kev-9B JevBench result is published.
65
 
66
- Without thinking the model scores 0.804 on the Jeeves test split, against 0.840 with it. With the fitted temperature the no-think path has a top-label calibration error of 0.021 and 1.3% confident errors (wrong at p ≥ 0.9).
67
 
68
- Serving on one H100, 325 dev questions:
69
 
70
  | setting | accuracy | median / p90 latency |
71
  |---|---|---|
72
- | full thinking | 0.825 | 3.3 s / 17.1 s |
73
- | `max_think` 768, `nothink_threshold` 0.9 | 0.806 | 2.0 s / 5.6 s |
74
- | no thinking | 0.775 | about 0.3 s |
 
 
 
75
 
76
  ## Training
77
 
78
- - **SFT.** LoRA r=16 on every projection plus the pointer head, 2 epochs on 19,126 questions from 12 public datasets and synthetic policy data; half the questions carry a reasoning chain sampled from the base model.
79
- - **CISPO.** 9,992 RL questions, 8 rollouts each, up to 2,560 thinking tokens. Reward is the probability of the correct option, discounted by up to 10% for long chains. The released checkpoint is step 402 of a 624-step schedule.
80
- - **Calibration.** One softmax temperature fitted on the dev set.
81
- - **Drafters.** Diffusion views of the frozen model, inspired by Orthrus, distilled by KL divergence on chains sampled from this model. At one question the block-4 drafter decodes 1.6× faster than graphed greedy decoding and the block-8 drafter 1.76×.
 
 
 
 
 
 
 
 
 
 
 
82
 
83
- A from-scratch reproduction with the released code matched this checkpoint within noise. The training code, dataset build scripts and pinned dataset revisions are in the GitHub repository.
84
 
85
  ## Limitations
86
 
87
- - Knowledge questions trail Jev (MMLU 0.793 vs 0.900, MMLU-Pro 0.739 vs 0.840).
88
- - Thinking is slow at the tail (17 s at p90 with full chains); use `max_think` and `nothink_threshold` when latency matters.
89
- - About a third of full-length chains hit the token cap without closing; accuracy on those items is lower.
90
  - Evaluated on English inputs only.
91
 
92
  ## License
@@ -101,6 +117,6 @@ The weights are derived from Qwen3.5-9B and are released under its Apache-2.0 li
101
  title = {Jeeves: Reasoning Improves Jev-like Decisions},
102
  year = {2026},
103
  url = {https://github.com/PostHog/jeeves},
104
- note = {Qwen3.5-9B decision model trained with SFT and CISPO, with a block-4 diffusion drafter}
105
  }
106
  ```
 
14
 
15
  # Jeeves-9B
16
 
17
+ A Jev-like decision model that reasons before it decides when it is unsure, trained on public data only. Give it a state (text or JSON) and yes/no, multiple-choice or rating questions; it answers directly when it is confident, otherwise writes a short reasoning chain first, and returns a calibrated probability for every option.
18
 
19
+ This is Jeeves v2. v1 (SFT + CISPO) is the `v1` tag of this repository: `hf download PostHog/jeeves --revision v1`.
20
+
21
+ This repository holds the fused weights (Qwen3.5-9B with the LoRA merged), the pointer head, the two fitted temperatures, and two speculative-decoding drafters. It loads with the Jeeves code, not with `transformers`: the pointer head and the prompt format are part of the model.
22
 
23
  ## Files
24
 
 
26
  |---|---|
27
  | `model-*.safetensors`, `config.json` | fused Qwen3.5-9B weights, bf16 |
28
  | `head.pt` | pointer head (query and key projections, 256-dim) |
29
+ | `export.json` | temperatures (1.522 direct, 5.420 after reasoning), prompt format, training step and config |
30
  | `tokenizer.json`, `vocab.json`, `merges.txt`, `tokenizer_config.json`, `chat_template.jinja` | Qwen3.5 tokenizer |
31
+ | `drafter_k4.safetensors` | block-4 drafter, the serving default, bf16 (trained on v1; v2 drafters to follow) |
32
  | `drafter_k8.safetensors` | block-8 drafter, faster for single-question requests, bf16 |
33
  | `drafter_k4.config.json`, `drafter_k8.config.json` | drafter training settings |
34
  | `LICENSE` | Apache-2.0, inherited from Qwen3.5-9B |
 
54
 
55
  ## Results
56
 
57
+ Jeeves answers directly when its calibrated confidence is at least 0.9 and otherwise reasons first (`nothink_threshold` 0.9, greedy, 2,560-token cap); the percentage is the share of questions that reasoned. Kev-9B and Jev numbers are the ones Kev publishes. JevK5 (v0.3) does not reason; we ran it ourselves on the same items. JevBench uses the same 231 public items for every model (the sealed judge tier is not included); the other rows use the same sources with different items.
58
 
59
+ | | Kev-9B | Jev | JevK5 | Jeeves v1 | Jeeves v2 |
60
+ |---|---|---|---|---|---|
61
+ | Test overall (out-of-domain and held-out) | 0.822 | 0.857 | 0.802 | 0.884 (41%) | **0.905** (39%) |
62
+ | MMLU-Pro and buried state | 0.579 | **0.800** | 0.561 | 0.728 (86%) | 0.780 (76%) |
63
+ | JevBench, public tiers | 0.715* | 0.866 | 0.861 | **0.913** (44%) | 0.896 (29%) |
64
+ | JevBench hard | 0.451* | 0.730 | 0.748 | **0.820** | 0.793 |
65
 
66
  \* Kev-8B (Qwen3); no Kev-9B JevBench result is published.
67
 
68
+ Without thinking v2 scores 0.812 on the Jeeves test split, against 0.851 reasoning when unsure. On held-out questions from the 15 public datasets new in v2 it scores 0.853 and 0.900.
69
 
70
+ Serving on one H100 with `--precision fp8`, 325 dev questions:
71
 
72
  | setting | accuracy | median / p90 latency |
73
  |---|---|---|
74
+ | full thinking | 0.818 | 1.9 s / 8.1 s |
75
+ | `nothink_threshold` 0.9 | 0.822 | 1.2 s / 8.2 s |
76
+ | `max_think` 1024, `nothink_threshold` 0.9 | 0.818 | 1.2 s / 4.3 s |
77
+ | no thinking | 0.806 | 22 ms / 46 ms |
78
+
79
+ v2 was trained on chains of at most 1,024 tokens, so lower caps cut most of its chains.
80
 
81
  ## Training
82
 
83
+ - **SFT.** LoRA r=16 on every projection plus the pointer head, 1 epoch (16,867 steps) on 453,531 questions from about 175 public sources. About 15% of examples carry a reasoning chain sampled from earlier rounds of the model, kept only if it closes within 1,024 tokens, does not loop and reaches the right answer.
84
+ - **Calibration.** Two softmax temperatures fitted on the dev set: 1.522 for direct answers and 5.420 for answers after reasoning. Every training chain reaches the right answer, so answers after reasoning come out overconfident and need the larger temperature.
85
+ - **Drafters.** Diffusion views of the frozen model, inspired by Orthrus, distilled by KL divergence on chains sampled from v1. They work with v2 (every draft is verified, so outputs are unchanged) but accept fewer tokens; v2 drafters will follow. At one question with v1 the block-4 drafter decodes 1.6× faster than graphed greedy decoding and the block-8 drafter 1.76×.
86
+
87
+ v1 was trained with SFT followed by CISPO; CISPO did not improve v2. The training code, dataset build script (`python -m prep.prep`, which rebuilds v2's training data byte for byte from pinned revisions) and the id lists of the mix are in the GitHub repository. The reasoning chains are not published yet.
88
+
89
+ ## Training data
90
+
91
+ Public sources only; no PostHog or customer data. Every new record was checked against the eval sets and dev for shared 8-word sequences. Each source stays under its own licence; the main groups are:
92
+
93
+ - **Public datasets.** AG News, Amazon reviews, ARC (CC BY-SA 4.0), Banking77, Bias in Bios, BoolQ (CC BY-SA 3.0), Civil Comments, CLINC150, CommonsenseQA, DBpedia (CC BY-SA 3.0), GoEmotions, IMDB, LIAR2, MASSIVE, MedMCQA, MedQA, Mind2Web, MNLI, Measuring Hate Speech, ProsocialDialog, QASC, SNLI (CC BY-SA 4.0), SST-5, TREC, WANLI, Yelp, HelpSteer2 and HelpSteer3, Anthropic HH-RLHF.
94
+ - **New in v2.** GSM8K, AQuA-RAT, RuleTaker, WinoGrande, HellaSwag, SWAG, Cosmos QA, OpenBookQA, the New Yorker caption contest, When2Call, ToolACE, Glaive function calling v2, RAGTruth, ESCI and Schema-Guided Dialogue (CC BY-SA 4.0).
95
+ - **Third-party decision sets.** Open-Jev v1 and v1.1 (CC0), the Kev suites, spark, typed-decisions, SemIf and decider's converted sets and teacher data. Their labels, and for several of them their texts, come from LLM teachers.
96
+ - **Prompt injection.** Gandalf, LLMail-Inject, the neuralchemy and S-Labs prompt-injection sets, InjecAgent and permissively licensed parts of ShieldLM's curation.
97
+ - **Generated.** Some sources are LLM-generated (for example Glaive, ToolACE, When2Call, Open-Jev and the teacher-labelled sets), and v1's synthetic policy data is generated from templates.
98
 
99
+ The full list of repositories, revisions and licences is in `data/manifest.json` in the GitHub repository.
100
 
101
  ## Limitations
102
 
103
+ - Knowledge questions trail Jev (MMLU 0.845 vs 0.900, MMLU-Pro 0.770 vs 0.840).
104
+ - v2 trails v1 on JevBench when reasoning only when unsure (0.896 vs 0.913): it is overconfident without reasoning on some long-policy and multi-hop items, so it skips reasoning on them. With reasoning on every item the two are level (0.922 vs 0.926).
105
+ - Thinking is slow at the tail (8 s at p90 with full chains); use `max_think` and `nothink_threshold` when latency matters.
106
  - Evaluated on English inputs only.
107
 
108
  ## License
 
117
  title = {Jeeves: Reasoning Improves Jev-like Decisions},
118
  year = {2026},
119
  url = {https://github.com/PostHog/jeeves},
120
+ note = {Qwen3.5-9B decision model that reasons when unsure, with a block-4 diffusion drafter}
121
  }
122
  ```