Refocus Winnow-E4B model card on model and decision workflow

README.md

@@ -14,178 +14,104 @@ tags:
 
 # Winnow-E4B
 
-**Local typed decisions, chat and vision, with Q8 profiles for consumer GPUs.**
-We recommend **12 GB+ VRAM for 64K context + vision**, or **10 GB+ VRAM for
-8K text-only use**. In a matched 8K-context,
-text-only decision workload, Winnow-E4B Q8 delivered approximately **2.2× the
-batched throughput** of Winnow-12B Q8 and used **5.5 GiB less peak GPU memory**.
-Winnow-12B remains stronger on the broad JevBench and Kev suites; E4B leads on
-our targeted typed-decision evaluation. These are measured comparisons, not
-claims about every workload.
-
-| Recommended Q8 GPU capacity | Profile | Observed peak GPU memory |
-|---|---|---:|
-| **12 GB+ VRAM** | 64K context + vision, with the matching projector | 10.72 GiB across the decision and chat probes |
-| **10 GB+ VRAM** | 8K context, text only | 8.56 GiB in the decision workload |
-
-These capacity recommendations use measured RTX 5070 Ti footprints with Q8 KV,
-four decision branches, one chat slot and exclusive scheduling. They are not
-benchmarks of every card in those tiers; memory use and speed vary by GPU,
-runtime and other GPU workloads. See the [quickstart](docs/QUICKSTART.md) for
-the exact settings.
-
-Winnow-E4B fine-tunes [Gemma 4 E4B IT](https://huggingface.co/google/gemma-4-E4B-it)
-for `noul`, `choice`, and `score` decisions over shared state. The
-[Winnow inference server](https://github.com/EldanRing/winnow-inference) evaluates
-answer-token logits without generating answer text and also exposes ordinary
-chat. A separate Q8 smoke test exercised near-64K context with an image and
-512-token chat generation on the RTX 5070 Ti. General chat quality and broad
-vision accuracy have not been benchmarked for this artifact.
-
-[Inference code](https://github.com/EldanRing/winnow-inference) ·
-[Quickstart](docs/QUICKSTART.md) ·
-[Benchmark methods and limitations](docs/BENCHMARKS.md) ·
-[Artifact manifest](release-manifest.json)
-
-## GGUF download
-
-| Download | Use | File size |
-|---|---|---:|
-| [Winnow-E4B-Q8_0.gguf](gguf/Winnow-E4B-Q8_0.gguf) | **Q8_0; selected and measured deployment artifact** | 8.01 GB / 7.46 GiB |
-| [Winnow-E4B-BF16.gguf](gguf/Winnow-E4B-BF16.gguf) | Optional higher-precision text model for larger memory budgets; Q8 is recommended | 15.05 GB / 14.02 GiB |
-| [mmproj-Winnow-E4B.gguf](gguf/mmproj-Winnow-E4B.gguf) | Matching F16 vision projector for Q8 image requests | 990 MB / 0.922 GiB |
-
-The GGUF contains the merged language weights, tokenizer and chat template. It
-needs no separate LoRA adapter or conversion. Both files come from the same
-selected step-438 merge. Download the matching projector alongside Q8 to send
-images; it was used in the [64K + vision smoke test](docs/BENCHMARKS.md#64k-context-and-vision-smoke-test).
-Verify downloads with [SHA256SUMS](SHA256SUMS).
-
-The BF16 GGUF passed 8K text-decision serving on an RTX 5070 Ti. On 759 held-out
-ordinary and 132 longer decisions, its top-choice accuracy was essentially the
-same as Q8. Its separately calibrated probability loss was slightly worse and
-it used about **15.0 GiB of GPU memory** versus **8.6 GiB** for Q8, with little
-headroom on a 16 GB card. BF16 has **not** been run through the public suites,
-or with a projector or populated 64K context. The 64K + vision results below
-are for **Q8 only**. See the
-[paired format check](docs/BENCHMARKS.md#selected-step-438-bf16-versus-q8).
-
-## Decision quality
-
-All numbers below are Q8 GGUF results on the same frozen questions. The matched
-E4B/12B release run used an RTX 5070 Ti and the same HTTP runner. Percentages are
-top-choice accuracy before temperature scaling, which does not change winners.
+**A compact Gemma 4 fine-tune for typed decisions that also supports ordinary chat and image input.**
+
+Winnow-E4B takes a shared state and a set of questions, then assigns probabilities to the answers each question supplies. The decision path reads candidate-answer logits without generating an explanation. It is designed for tasks such as routing a request, choosing an action, checking a condition, or rating urgency when the possible answers are known. The same merged model weights remain available for regular chat; image input uses the matching vision projector.
+
+Winnow-E4B was created by EldanRing from [Gemma 4 E4B IT](https://huggingface.co/google/gemma-4-E4B-it). The downloadable GGUFs have the decision fine-tune merged into the language model. They require no separate adapter or base-model download.
+
+[Run the model](docs/QUICKSTART.md) · [Inference code](https://github.com/EldanRing/winnow-inference) · [API reference](https://github.com/EldanRing/winnow-inference/blob/main/docs/API.md) · [Evaluation details](docs/BENCHMARKS.md)
+
+## How decisions work
+
+The [Winnow inference server](https://github.com/EldanRing/winnow-inference) exposes `/v1/systemone`. A request provides one state and any combination of these question types:
+
+| Type | What the caller supplies | What the server returns |
+|---|---|---|
+| `noul` | A yes/no question | Probability of `true` |
+| `choice` | Named options with optional descriptions | Selected option, option probabilities, confidence |
+| `score` | An ordered list of levels | Level probabilities, expected score, confidence |
+
+The server shares the state prefill across questions, reuses a matching cached prefix when possible, and evaluates only the verified answer-token candidates for each question. Questions can be processed in parallel branches or waves without loading another copy of the model. Ordinary `/v1/chat/completions` uses the normal vocabulary head and chat template, including streaming and image messages when the projector is loaded.
+
+For example, one request can ask both whether a customer wants a refund and which team should handle the ticket:
+
+```json
+{
+  "model": "Winnow-E4B",
+  "state": {"ticket": "I was charged twice and need a refund."},
+  "questions": {
+    "refund": {
+      "type": "noul",
+      "instructions": "Does the customer request a refund?"
+    },
+    "department": {
+      "type": "choice",
+      "instructions": "Which team should handle this ticket?",
+      "criteria": {
+        "billing": "Payments and refunds",
+        "technical": "Software defects"
+      }
+    }
+  },
+  "winnow": {"temperature": 1.2574172017327816}
+}
+```
+
+The [quickstart](docs/QUICKSTART.md#ask-typed-questions) shows a complete command. Candidate probabilities are conditional on the options supplied in a request. Confidence describes how concentrated the distribution is; it is not a guarantee that an answer is correct.
+
+## Model and training
+
+This is a **rank-32, alpha-64 LoRA fine-tune** of the instruction-tuned Gemma 4 E4B model. A selected decision adapter was continued on targeted hard cases, with step 438 chosen for export. The update trained language tensors; the vision and audio modules remained frozen. The adapter was merged in FP32 before conversion to Q8_0 and BF16 GGUF. Chat still has its full generation head, while the decision server projects only the requested answer rows.
+
+The private training set combines synthetic contrastive decisions, verified labels, teacher distributions where appropriate, semantic tasks, and targeted hard cases. Training, development, calibration, and reserved tests were separated. **The training data and adapter checkpoint are not released.** Public evaluation suites were known before the targeted continuation was designed and informed task selection; the results below should be read as a comparison of deployed models, not a fresh unseen-task estimate.
+
+## Decision results
+
+These figures evaluate the **downloadable Q8 GGUF** against Winnow-12B Q8 on the same questions. They report top-choice accuracy, except the local typed panel, which reports agreement with synthetic teacher labels.
 
 | Evaluation | Winnow-E4B Q8 | Winnow-12B Q8 |
 |---|---:|---:|
-| JevBench public subset, 231 items | 186/231, **80.52%** | 198/231, **85.71%** |
-| Kev v9 clean, 1,046 items | **72.66%** | **81.45%** |
-| Kev v9 additional, 390 items | **55.90%** | **68.97%** |
-| Local typed decisions, 2,000 items | **72.30%** | **70.10%** |
-
-The local typed suite uses synthetic teacher distributions, not independently
-verified ground truth for every item. Its 2.2-point E4B lead is specific to that
-suite. The 12B leads on all three broader public suites. These suites were known
-before E4B's targeted continuation was designed, so they are a useful paired
-deployment comparison, **not a fresh unseen-task estimate**. JevBench here means
-accuracy on the public subset, not the official composite leaderboard score.
-Details, confidence intervals and probability measures are in the
-[benchmark report](docs/BENCHMARKS.md).
-
-## RTX 5070 Ti: speed and memory
-
-Both Q8 models used the same GPU, 8,192-token text-only context, Q8 KV cache,
-four decision branches and selected-token output projection. The runner repeated
-the same full state/question workload ten times per batch size; the values below
-are warm median request times, not tail-latency guarantees.
-
-| Simultaneous questions | Winnow-E4B Q8 | Winnow-12B Q8 |
-|---:|---:|---:|
-| 1 | **13.33 ms** | 26.50 ms |
-| 8 | **53.97 ms** | 119.17 ms |
-| 16 | **108.51 ms** | 239.30 ms |
-| 32 | **216.95 ms** | 475.83 ms |
-| 64 | **433.04 ms** | 951.99 ms |
-
-At 64 questions, warm throughput was **147.8 versus 67.2 decisions/s**. Peak
-reported GPU memory was **8,769 versus 14,355 MiB**, a 5,586 MiB (5.46 GiB)
-difference. The 5070 Ti has 16 GB VRAM, but these measurements do not establish
-the memory needed for longer context or a vision projector. Prefill and chat
-generation speeds are not characterized by this table.
-
-## 64K context, vision and chat smoke test
-
-The selected Q8 GGUF and matching F16 projector ran on the same RTX 5070 Ti
-with 65,536 context positions, Q8 KV cache and four decision branches. A
-synthetic 1024 × 1024 status-panel image occupied 441 positions. A request with
-65,023 shared-prefix positions, including that image, answered **6/6** visual
-and text questions. Cold request time was **11.59 s**, including 10.96 s of
-prefill; identical cached follow-ups took **129–131 ms**. Sampled peak GPU
-memory was **10,723 MiB (10.47 GiB)**. A changed image invalidated the cache
-as expected.
-
-A separate chat probe generated 512 tokens on a 62,431-position multimodal
-prompt at **71.2 output tokens/s**, with **9.98 s** to first content and
-**10,979 MiB (10.72 GiB)** sampled peak GPU memory. Three short chat runs
-generated at a median **119.1 tokens/s**. These are operational probes: they
-establish that the image path, long context and chat generation work, not broad
-vision accuracy, long-context reasoning quality or chat quality. The
-[benchmark report](docs/BENCHMARKS.md#64k-context-and-vision-smoke-test) gives
-the exact method and limits.
-
-## GGUF packaging
-
-The selected step-438 language adapter was merged into the pinned Gemma 4 E4B IT
-base as `FP32(base) + FP32(B @ A) × alpha/rank`, then converted to Q8_0 or BF16
-with a pinned llama.cpp converter. The FP32 merge avoids rounding the learned
-update to BF16 before conversion. Each artifact is a complete merged model; the original
-adapter and private training checkpoint are not needed for inference.
-
-## Training
-
-Winnow-E4B is a **LoRA fine-tune**, not a full-parameter training run. It
-continues a selected E4B decision adapter with a targeted local update; the
-released GGUF contains the merged weights.
-
-| Setting | Value |
-|---|---|
-| Base | `google/gemma-4-E4B-it` |
-| Base revision | `ee0ef6023621cff504d758262d4e04895a5af4a2` |
-| LoRA rank / alpha | 32 / 64 |
-| Adapted weights | 294 language tensors; vision and audio modules frozen |
-| Targeted continuation | Selected step 438; effective batch 16; up to 8,192 input tokens |
-| Export | FP32 adapter merge, then Q8_0 or BF16 GGUF |
-
-**The training dataset is private and is not released.** It combines synthetic
-contrastive decisions, verified labels, teacher distributions where appropriate,
-semantic tasks and targeted hard cases. Training, development, calibration and
-reserved tests were separated. Public suites were used for evaluation and also
-informed later targeted data design, as described above. No claim is made about
-excluding these tasks from the base model's pretraining.
-
-## Probabilities and scope
-
-The server normalizes logits over the supplied answer options. For this Q8
-artifact, a **1.257417** temperature was fitted on 778 reserved calibration
-decisions (646 ordinary and 132 long). Pass it as `winnow.temperature` in each
-`/v1/systemone` request when using the measured calibration. It changes
-probabilities, not the selected answer. It is not a universal guarantee of
-correctness; calibration quality varies by task and length. On the combined
-already-opened reserved test panels, E4B's calibrated proper loss improved over
-the prior E4B checkpoint while labeled accuracy was unchanged.
-
-BF16's separately fitted decision temperature is **1.333155** on the same 778
-reserved calibration decisions. The Q8 temperature above should not be applied
-to BF16. The matched quality and throughput suites use 8K text decisions; the
-separate Q8 64K + vision and chat probes above establish runtime capability on
-the tested 5070 Ti. General chat and real-image vision quality remain unmeasured.
+| JevBench public subset, 231 questions | **80.52%** (186/231) | 85.71% (198/231) |
+| Kev v9 clean, 1,046 questions | **72.66%** | 81.45% |
+| Kev v9 additional, 390 questions | **55.90%** | 68.97% |
+| Local typed decisions, 2,000 decisions | **72.30%** | 70.10% |
+
+Winnow-12B leads on the broader public suites. E4B leads by 2.2 points on this local typed panel, which uses synthetic teacher targets rather than independently verified truth for every item. The JevBench figure is **public-subset accuracy**, not the official composite leaderboard score. Methods, uncertainty intervals, calibration measures, and evaluation boundaries are in the [benchmark report](docs/BENCHMARKS.md).
+
+In a separate matched 8K text decision workload, E4B Q8 delivered **147.8 decisions/s** at 64 questions per request, versus **67.2 decisions/s** for Winnow-12B Q8. Those are warm request medians on an RTX 5070 Ti, not a guarantee for other hardware or workloads. The [timing table](docs/BENCHMARKS.md#same-gpu-latency-and-memory) includes smaller batches and memory use.
+
+## Probabilities and calibration
+
+The server normalizes candidate logits over the options in each question. The optional `winnow.temperature` setting scales those logits without changing which option wins. For **Q8 text decisions**, `1.2574172017327816` was fitted on 778 separate calibration questions. For **BF16**, the separately fitted value is `1.3331553765162731`; do not reuse Q8's temperature for BF16.
+
+These temperatures were measured for the evaluated text-decision profile. Calibration can change with task, prompt length, image input, and deployment settings. Check probabilities on held-out examples from the intended application before treating them as confidence in real-world correctness.
+
+## Downloads and deployment
+
+| File | Role | Size |
+|---|---|---:|
+| [Winnow-E4B-Q8_0.gguf](gguf/Winnow-E4B-Q8_0.gguf) | Recommended, fully measured language model | 8.01 GB / 7.46 GiB |
+| [Winnow-E4B-BF16.gguf](gguf/Winnow-E4B-BF16.gguf) | Optional higher-precision language model | 15.05 GB / 14.02 GiB |
+| [mmproj-Winnow-E4B.gguf](gguf/mmproj-Winnow-E4B.gguf) | Matching F16 projector for image input | 990 MB / 0.922 GiB |
+
+Download Q8 for the measured decision, chat, and vision setup; add the projector for images. BF16 passed 8K text-decision serving but was not tested with images or populated 64K context. On reserved text tests it showed no consistent quality advantage over Q8. [Verify files](SHA256SUMS) after download.
+
+| Q8 configuration | Observed peak device memory | Suggested GPU capacity |
+|---|---:|---:|
+| 8K text-only decision workload | 8.56 GiB | 10 GB+ VRAM |
+| 64K vision and chat operational probes | 10.72 GiB across the probes | 12 GB+ VRAM |
+
+Capacity suggestions come from one RTX 5070 Ti using Q8 KV cache, four decision branches, one chat slot, and exclusive memory scheduling. They have not been tested on every GPU in those tiers. The Q8 model and projector answered six questions over a 65,023-position state that included a synthetic image; repeated cached requests took 129–131 ms after an 11.59 s cold request. A separate 62,431-position multimodal chat prompt generated 512 tokens at 71.2 output tokens/s. The [benchmark report](docs/BENCHMARKS.md#64k-context-and-vision-smoke-test) describes these operational probes and their limits.
+
+## Scope and limitations
+
+- The decision path is intended for supplied answer options. Open-ended responses use the regular chat endpoint.
+- General chat quality, real-image vision accuracy, and broad long-context reasoning have not been established by these probes. The image test used a synthetic status panel. Vision and audio modules were not fine-tuned for this release.
+- The public suites were known during targeted data design. The local typed panel measures agreement with synthetic teacher labels. Neither result establishes accuracy for every downstream task.
+- Probability calibration was fitted on text decisions. It needs validation for a new domain or multimodal workload.
+- The 64K and vision measurements apply to **Q8 with the matching projector**. BF16 was evaluated only in the text profile.
 
 ## Credits and license
 
-Winnow-E4B is an independent fine-tune by EldanRing of Google DeepMind's
-[Gemma 4 E4B IT](https://huggingface.co/google/gemma-4-E4B-it), released under
-[Apache 2.0](https://ai.google.dev/gemma/docs/gemma_4_license). See [LICENSE](LICENSE)
-and [NOTICE](NOTICE). The separate inference code builds on
-[llama.cpp](https://github.com/ggml-org/llama.cpp) and preserves its MIT license.
-Jev-style describes the typed-decision interface; Winnow is not affiliated with
-or endorsed by TypeSafe, Google or llama.cpp.
+Winnow-E4B is an independent fine-tune by EldanRing of Google DeepMind's [Gemma 4 E4B IT](https://huggingface.co/google/gemma-4-E4B-it), released under [Apache 2.0](https://ai.google.dev/gemma/docs/gemma_4_license). See [LICENSE](LICENSE) and [NOTICE](NOTICE). The separate inference code builds on [llama.cpp](https://github.com/ggml-org/llama.cpp) and preserves its MIT license. Jev-style describes the typed-decision interface; Winnow is not affiliated with or endorsed by TypeSafe, Google, or llama.cpp.