Card: overview, models table and quick start; 4B details below; JevBench v1.5.5 standing

README.md

@@ -30,16 +30,66 @@ datasets:
 
 # Hopper
 
-> **Also in this repository: Hopper 12B (trained).** A LoRA adapter (rank 32) on
-> [`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it), Apache-2.0, on the branch
-> [`hopper-12b`](https://huggingface.co/HopitAI/hopper/tree/hopper-12b) (revision
-> `4b605f25eb903c2cf4d121040ffb097240b3bc37`), with its own model card, readout map, licence and checksums.
-> Serve it with the code at [github.com/hopit-ai/hopper](https://github.com/hopit-ai/hopper) (tag `12b-1.1.0`):
-> `hopper-serve --base-loader gemma-4-12b-it --adapter HopitAI/hopper@4b605f25eb903c2cf4d121040ffb097240b3bc37`
-> plus its readout map (see that branch's card). The files on this `main` branch are the original 4B Hopper
-> described below.
+Hopper models make single-pass decisions: a document, a policy and a question go in, and a probability for every
+allowed option comes out. One forward pass per decision; no text is generated. They are served by
+[github.com/hopit-ai/hopper](https://github.com/hopit-ai/hopper), an HTTP server that speaks the
+[JevBench](https://github.com/fstandhartinger/jevbench) `/v1/systemone` wire format.
 
-> **Research and demo use only.** This adapter is published for research and demonstration. Its training data included passages from RACE (via the `cais/mmlu` auxiliary set), which its authors release for non-commercial research only and whose terms extend to derived data. Do not use this adapter commercially. A version trained without these passages is in development.
+## Models in this repository
+
+| Model | Where | Base model | Adapter | Licence | Status |
+| --- | --- | --- | --- | --- | --- |
+| **Hopper (4B)** | `main` (this page) | [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B) @ `851bf6e` | LoRA rank 16 | research and demo use only¹ | [JevBench](https://benchmarkheaven.com/jev-models) **67.5, #16** (v1.5.5, 2 Oct 2026) |
+| **Hopper 12B (trained)** | branch [`hopper-12b`](https://huggingface.co/HopitAI/hopper/tree/hopper-12b) @ `4b605f2` | [`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it) @ `707f0a3` | LoRA rank 32, served unmerged | Apache-2.0 | requested on JevBench ([issue #164](https://github.com/fstandhartinger/jevbench/issues/164)) |
+
+¹ The 4B adapter's training data included passages from RACE, which is released for non-commercial research only.
+Do not use it commercially. See [Licence and research-only notice](#licence-and-research-only-notice).
+
+The general-purpose line, Hopper (G), lives at [`HopitAI/hopper-g`](https://huggingface.co/HopitAI/hopper-g).
+
+## Quick start
+
+### Hopper 12B (trained)
+
+```sh
+pip install "git+https://github.com/hopit-ai/hopper@12b-1.1.0"
+hopper-serve --base-loader gemma-4-12b-it \
+  --adapter HopitAI/hopper@4b605f25eb903c2cf4d121040ffb097240b3bc37 \
+  --readout-map "$(python -c 'import hopper_decisions, pathlib; print(pathlib.Path(hopper_decisions.__file__).parent / "maps/hopper-12b-trained-readout.json")')" \
+  --port 8080
+```
+
+Always pin the revision: `HopitAI/hopper` without one resolves to `main`, the Qwen3.5-4B adapter, which does not fit
+this base. Files, checksums, training data and licence are on the
+[`hopper-12b` branch's model card](https://huggingface.co/HopitAI/hopper/blob/hopper-12b/README.md).
+
+### Hopper (4B)
+
+Install the serving code from [github.com/hopit-ai/hopper](https://github.com/hopit-ai/hopper) as its README
+describes (with the pinned kernel packages listed under [How to load it](#how-to-load-it)), then:
+
+```sh
+hopper-serve --adapter HopitAI/hopper --port 8080
+```
+
+For the Python API and for loading with transformers and peft directly, see [How to load it](#how-to-load-it).
+
+## Hopper (4B) details
+
+### Licence and research-only notice
+
+> **Research and demo use only.** This adapter is published for research and demonstration. Its training data
+> included passages from RACE (via the `cais/mmlu` auxiliary set), which its authors release for non-commercial
+> research only and whose terms extend to derived data. Do not use this adapter commercially. Hopper 12B (trained),
+> on branch [`hopper-12b`](https://huggingface.co/HopitAI/hopper/tree/hopper-12b), was trained without these passages
+> and is Apache-2.0.
+
+The serving code is licensed Apache-2.0. The adapter weights are offered for research and demo use
+only, because of the RACE training-data terms described above and under "Training data". The base
+model is Apache-2.0 ([licence](https://huggingface.co/Qwen/Qwen3.5-4B/blob/main/LICENSE)), and this
+adapter does not change its terms.
+
+### How it works
 
 Hopper is a LoRA adapter (rank 16) for
 [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B) at revision
@@ -64,31 +114,14 @@ question go in, and a probability distribution over a fixed set of options comes
   HTTP server with the JevBench `/v1/systemone` wire format. At load it merges the adapter into the
   bf16 weights, and it refuses to start if the fast linear-attention kernels are not active.
 
-The serving code is licensed Apache-2.0. The adapter weights are offered for research and demo use
-only, because of the RACE training-data terms described above and under "Training data". The base
-model is Apache-2.0 ([licence](https://huggingface.co/Qwen/Qwen3.5-4B/blob/main/LICENSE)), and this
-adapter does not change its terms.
-
-## Leaderboards (official)
-
-- **[JevBench](https://benchmarkheaven.com/jev-models)**: **59.43, #7** in v1.4.2.2 (27 Sep 2026), measured by the
-  maintainer on adapter 1.0.0: Intelligence 48.0, Calibration 79.1, Speed 86.8, Cost 58.7. Accuracy is 82.3 % on the
-  public items and 34.1 % on the maintainer's sealed set.
-- **[Jev Decision Index](https://huggingface.co/spaces/multimodalart/jev-decision-index)**: Hopper 1.1.1 scored
-  36.71, #12 of 49 (edition 0.2, 25 Sep 2026), then 39.67, #19 of 67 (edition 0.2.1). Its row has since been replaced by
-  [Hopper (G) 1.2](https://huggingface.co/HopitAI/hopper-g), the general-purpose line served with the same code,
-  at 40.77. A newer build, Hopper (G) 1.3, is published in the same repository.
-
-Both boards change as entrants are added; the live pages are authoritative.
-
-## Intended use
+### Intended use
 
 Hopper makes single-step policy decisions over a short document: yes/no (`noul`), choice among
 named options, and ordinal scores. It returns calibrated probabilities, and it is meant to be run
 and measured on JevBench. It is not a chat model. It is also not meant for decisions with legal,
 medical, financial or safety consequences unless a person reviews them.
 
-## Prompt format
+### Prompt format
 
 The chat template of Qwen3.5-4B is applied with `enable_thinking=False` and a generation prompt.
 
@@ -102,7 +135,7 @@ The readout is the next-token logits at the end of the prompt, restricted to the
 then a softmax, then the calibration map. `hopper_decisions/request.py` and `prompt.py` in the code
 repository build this prompt exactly.
 
-## How to load it
+### How to load it
 
 The easiest way to serve it is with the package (`pip install` the code repository, see its README):
 
@@ -139,7 +172,7 @@ to run at full speed. Without them, transformers silently falls back to a path t
 10x slower. The pinned versions are `torch==2.8.0`, `transformers==5.17.0`, `peft==0.21.0` and
 `accelerate==1.15.0`.
 
-## Training data
+### Training data
 
 The adapter was trained on a mix of three sources:
 
@@ -169,7 +202,7 @@ The adapter was trained on a mix of three sources:
 The calibration map was fitted only on our own held-out JevBench-style items — the same fitting
 set in 1.1.0 as in 1.0.0, with the 66 leakage-flagged items dropped. It never saw a JevBench item.
 
-## Evaluation
+### Evaluation
 
 **These are local numbers, not official JevBench results.** They were computed with our own
 evaluation path on the 231 public JevBench items (argmax over the exact label set, with ties
@@ -217,10 +250,21 @@ not on this fold.
 - Expect the held-out hard items to score below the public ones, and expect the judge tier, which
   we have never seen, to be the least predictable part.
 
-## Long menus (more than 26 options), from code release 1.1.1
+### Leaderboards
+
+- **[JevBench](https://benchmarkheaven.com/jev-models)**: **67.5, #16** in v1.5.5 (2 Oct 2026), measured on adapter
+  1.0.0: Intelligence 49.9, Calibration 87.9, Speed 87.2, Cost 62.3.
+- **[Jev Decision Index](https://huggingface.co/spaces/multimodalart/jev-decision-index)**: Hopper 1.1.1 scored
+  36.71, #12 of 49 (edition 0.2, 25 Sep 2026), then 39.67, #19 of 67 (edition 0.2.1). Its row has since been replaced by
+  [Hopper (G) 1.2](https://huggingface.co/HopitAI/hopper-g), the general-purpose line served with the same code,
+  at 40.77. A newer build, Hopper (G) 1.3, is published in the same repository.
+
+Both boards change as entrants are added; the live pages are authoritative.
+
+### Long menus (more than 26 options)
 
-The adapter answers by reading one letter per option, so a single forward pass can weigh at most 26
-options. From release 1.1.1 of the serving code (github.com/hopit-ai/hopper, tag v1.1.1), a choice
+The adapter answers by reading one letter per option, so a single forward pass can
+weigh at most 26 options. From release 1.1.1 of the serving code (github.com/hopit-ai/hopper, tag v1.1.1), a choice
 question with more options is answered in two disclosed stages, using this same adapter and map:
 
 1. **First stage.** By default a *tournament*: the menu is dealt with a fixed seed into
@@ -248,7 +292,7 @@ frozen base model within sampling error. On BANKING77 the served top choice is o
 top probability 0.78 against 0.67 accuracy). The 5 % residual is a default, not yet refitted to
 held-out long menus. Research and demo use only, as for the rest of this card.
 
-## Limitations
+### Limitations
 
 - **One pass of a 4B model.** Hopper does not reason step by step. Any problem that needs a chain
   of intermediate results is decided in one forward pass by Qwen3.5-4B.