README.md: update model card

README.md

@@ -17,32 +17,32 @@ tags:
 
 # KnowLine-4B-Gen4
 
-**The fourth release of PelaAI's 4B System One decision model, trained on a single all-in-one machine by an agent-driven
-data loop.**
+**The fourth release of PelaAI's 4B System One decision model, trained on a single all-in-one machine through an
+agent-driven data loop.**
 
 [Inference guide](INFERENCE.md) · [中文说明](README.zh.md) · weights Apache-2.0 · previous version:
 [KnowLine-4B-Gen3](https://huggingface.co/PelaAI/KnowLine-4B-Gen3)
 
 Highlights:
 
-- **Decision Index 0.3, public suite: 64.90** (self-run), 1.79 above Gen3 (63.11). Most of the gain comes from a few
-  benchmarks whose task is in our training data, BPoMP above all; see [What changed in Gen4](#what-changed-in-gen4).
-  For comparison: Perplexity Decider v1.1 (27B) scores 62.25 and Jev 1.13 57.96 (board), and the best ≤5B model on the
-  board has a public score of 50.82; see [Comparison](#comparison).
+- **Decision Index 0.3 public suite: 64.90** (self-run), 1.79 points above Gen3 (63.11). Most of the gain comes from a
+  few benchmarks whose tasks appear in our training data, led by BPoMP; see [What changed in Gen4](#what-changed-in-gen4).
+  For comparison, on the board Perplexity Decider v1.1 (27B) scores 62.25 and Jev 1.13 scores 57.96, and the best model
+  of 5B or smaller has a public score of 50.82; see [Comparison](#comparison).
 - **Harder to steer with planted instructions:** an instruction planted in the state changes the answer 1.6% of the
-  time, down from 3.4% in Gen3; with injection templates never seen in training, 6.2%, down from 10.2%.
-- **Calibrated probabilities:** a temperature is folded into the weights. Computed the board's way, ECE is about
-  0.04-0.05 (board median 0.084) and the Brier score about 0.29-0.31, roughly 2nd-3rd of 114 models on the board; see
-  [Calibration](#calibration).
-- **At Gen3's level on our other evaluations:** JevBench / JevBench-hard 86.1 / 72.1 (Gen3: 86.6 / 72.1), Open-Jev 1.1
+  time, down from 3.4% for Gen3. With injection templates not seen in training, the rate is 6.2%, down from 10.2%.
+- **Calibrated probabilities:** a temperature is folded into the weights. Computed with the board's method, ECE is about
+  0.04-0.05 (board median: 0.084) and the Brier score is about 0.29-0.31, roughly 2nd-3rd among the 114 models on the
+  board; see [Calibration](#calibration).
+- **On par with Gen3 on our other evaluations:** JevBench / JevBench-hard 86.1 / 72.1 (Gen3: 86.6 / 72.1), Open-Jev 1.1
   test / OOD 88.4 / 87.1 (88.4 / 87.2), C-Eval 78.7 (79.2).
 - **KOF '98 harness: 14-3-1 against Jev 1.13 and 10-4-4 against StartLux-Decision-4B** in single-bout mirror matches;
   see [Game harness](#game-harness-kof-98).
-- **An AI agent ran the whole loop.** In each round it:
-  - found the areas where the model was weak;
+- **An AI agent ran the whole loop end to end.** In each round it:
+  - identified where the model was weak;
   - proposed a targeted data group;
   - built and decontaminated the data;
-  - trained and evaluated on it;
+  - trained and evaluated the model on it;
   - kept or rejected the change based on the evidence.
 
   For this round, see [What changed in Gen4](#what-changed-in-gen4).
@@ -54,16 +54,16 @@ KnowLine is an independent model. It is not affiliated with, endorsed by, or der
 You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model
 answers every question with a probability distribution over its options:
 
-- one forward pass per question, with no generated text;
+- one forward pass per question, with no text generation;
 - the output is the probability of each option's label token;
-- existing Jev clients only need a new base URL; users of earlier versions only change the model name, since the
-  interface and serving settings are the same.
+- existing Jev clients only need to change the base URL, and users of earlier versions only need to change the model
+  name, because the interface and serving settings are unchanged.
 
 | | |
 |---|---|
 | Base model | [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) (Apache-2.0) |
-| Training | LoRA SFT (rank 32, alpha 64, language model only), merged into bf16 weights. |
-| Release format | bf16 weights, with a readout temperature T = 1.32 folded into the final norm weight (see [Calibration](#calibration)). The vision tower and MTP head are the base model's; config, tokenizer and chat template are identical to Gen1-Gen3. |
+| Training | LoRA SFT (rank 32, alpha 64, language model only), merged into bf16 weights |
+| Release format | bf16 weights, with a readout temperature T = 1.32 folded into the final norm weight (see [Calibration](#calibration)). The vision tower and MTP head are unchanged from the base model; the config, tokenizer and chat template are identical to those of Gen1-Gen3. |
 | Languages | English, Simplified Chinese, Traditional Chinese |
 
 ## Quickstart
@@ -79,50 +79,49 @@ curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -
                          "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'
 ```
 
-- **Without SGLang:** `python knowline_server.py --model PelaAI/KnowLine-4B-Gen4 --backend hf --port 8080` uses
-  transformers only. It is slower and runs bf16.
-- **Front end:** `knowline_server.py` is a single file and needs only transformers and requests. It is the same file
-  as in Gen3.
-- **Full settings:** the exact settings of our Decision Index run are in [INFERENCE.md](INFERENCE.md).
+- **Front end:** `knowline_server.py` is a single file that depends only on transformers and requests. It is the same
+  file as in Gen3.
+- **Full settings:** the exact settings used for our Decision Index run are in [INFERENCE.md](INFERENCE.md).
 
 ## What changed in Gen4
 
-1. **Finding the gaps.** Besides knowledge and reasoning, still the weakest area (43.9), the agent looked at what did
-   not carry over to new settings: the same question asked in another format, injection templates the model had not
-   seen, tool selection among near-identical tools, and tasks from domains not in training.
-2. **New data.** Groups aimed at those gaps: tasks described inside the state, format variants of existing items,
-   more tool-retrieval and hard tool-selection items, product-search relevance, a dozen new task families,
-   injection-invariance pairs with new templates, and news-topic classification. Arithmetic, knowledge and Jev-style
-   items from earlier rounds were replayed. All of it went through a revised decontamination check; see
-   [Disclosures](#disclosures).
-3. **Training.** Gen3 was trained further on this data, then averaged half and half with Gen3. The further training made
-   the model more overconfident, so a single temperature was fitted and folded into the weights; it changes the
-   probabilities, not which option ranks first.
-4. **Selection.** This round compared 37 candidates (new runs, further training of Gen3, and averages with Gen3), with
-   Decision Index 0.3 public scores of 61.18-65.39. Each one had to pass the same checks against Gen3: no
-   significant drop on our held-out sets, Open-Jev 1.1 OOD or JevBench-hard, injection at most 4%, and no regression
-   beyond set margins on new-domain tasks, sibling skills, format consistency, unseen injection templates and
-   calibration (measured after the temperature). Among the candidates that passed, Gen4 has the highest public score.
-   Some averages with higher public scores were rejected because they lost JevBench-hard items or format consistency.
-5. **Where the gain comes from.** About 1.58 of the 1.79 points (88%) come from benchmarks whose task this round added
-   to the training data:
-   - BPoMP, 54.5 → 90.7 (0.60 points of the index): original-versus-perturbed limerick items in the same style.
-     Earlier versions had no such data.
-   - Amazon ESCI, 38.9 → 45.6: ESCI train pairs.
-   - cfcolor, 25.4 → 35.2: palettes from the same source dataset the benchmark uses (O'Donovan et al., 2014).
+1. **Finding the gaps.** Knowledge and reasoning (43.9) remained the weakest area. Beyond that, the agent looked for
+   abilities that did not carry over to new settings: the same question asked in a different format, unseen injection
+   templates, tool selection among near-identical tools, and tasks from domains absent from training.
+2. **New data.** Data groups aimed at those gaps: task instructions written inside the state, format variants of existing
+   items, more tool-retrieval and hard tool-selection items, product-search relevance, a dozen or so new task families,
+   injection-invariance pairs with new templates (the same state with and without a planted instruction), and
+   news-topic classification. Arithmetic, knowledge and Jev-style items from earlier rounds were replayed. All of the
+   data went through a revised decontamination check; see [Disclosures](#disclosures).
+3. **Training.** Gen3 was trained further on this data, and the result was averaged 50/50 with Gen3. The extra training
+   made the model more overconfident, so a single temperature was fitted and folded into the weights. It changes the
+   size of the probabilities, but not which option ranks first.
+4. **Selection.** This round compared 37 candidates (new runs, Gen3 trained further, and their averages with Gen3), with
+   Decision Index 0.3 public scores between 61.18 and 65.39. Every candidate had to pass the same checks against Gen3:
+   no significant drop on our held-out sets, Open-Jev 1.1 OOD or JevBench-hard; an injection rate of at most 4%; and no
+   regression beyond set margins on new-domain tasks, sibling skills, format consistency, unseen injection templates or
+   calibration (measured after applying the temperature). Gen4 has the highest public score among the candidates that
+   passed. Several averages with higher public scores were rejected because they lost JevBench-hard items or format
+   consistency.
+5. **Where the gain comes from.** About 1.58 of the 1.79 points (88%) come from benchmarks whose tasks were added to the
+   training data in this round:
+   - BPoMP, 54.5 → 90.7 (0.60 index points): original-versus-perturbed limerick items in the same style. Earlier
+     versions had no such data.
+   - Amazon ESCI, 38.9 → 45.6: query-product pairs from the ESCI train split.
+   - cfcolor, 25.4 → 35.2: palettes from the same source dataset that the benchmark uses (O'Donovan et al., 2014).
    - Humicroedit, 23.1 → 32.0: the SemEval-2020 Task 7 train and dev splits and FunLines; the benchmark uses the test
      split.
-   - Smaller: ToolRet (61.2 → 64.9), BRIGHT (39.2 → 42.0), SATA-Bench (29.2 → 33.6), FinEntity (86.8 → 90.2) and ACOS
-     (43.3 → 46.1).
+   - Smaller gains: ToolRet (61.2 → 64.9), BRIGHT (39.2 → 42.0), SATA-Bench (29.2 → 33.6), FinEntity (86.8 → 90.2) and
+     ACOS (43.3 → 46.1).
 
-   On benchmarks of the same skill that we did not train on (BLiMP next to BPoMP, HaHackathon next to Humicroedit,
-   WANDS next to ESCI, Toucan next to ToolRet), the change is small or mixed, so most of this gain is specific to those
-   tasks. The largest drops are all small, led by WinoGrande (78.8 → 77.7).
+   On same-skill benchmarks that we did not train on (BLiMP for BPoMP, HaHackathon for Humicroedit, WANDS for ESCI,
+   Toucan for ToolRet), the changes are small or mixed, so most of this gain is specific to those tasks. The largest
+   drops are all small; the biggest is WinoGrande (78.8 → 77.7).
 
 ## Comparison
 
-Decision Index 0.3, public suite. The full 0.3 score adds private tests that only the maintainers run (0.5 same-skill,
-0.3 new-domain); ours is not available yet.
+Scores on the Decision Index 0.3 public suite. The full 0.3 score also includes private tests that only the maintainers
+run (weighted 0.5 same-skill and 0.3 new-domain); our full score is not available yet.
 
 | model | size | DI 0.3 public | DI 0.3 full | source |
 |---|---|---|---|---|
@@ -138,7 +137,7 @@ Decision Index 0.3, public suite. The full 0.3 score adds private tests that onl
 | jiwo 4B | 4B | 45.76 | 42.86 | board |
 | Nox 4B | 4B | 44.21 | 44.95 | board |
 
-Public-suite scores by area (all 0.3 public):
+Public-suite scores by area (DI 0.3):
 
 | area | KnowLine-4B-Gen4 | KnowLine-4B-Gen3 | Perplexity Decider v1.1 (27B) | Jev 1.13 |
 |---|---|---|---|---|
@@ -159,11 +158,11 @@ The model is trained in a self-evolving loop, and new versions will follow.
 | [KnowLine-4B-Gen2](https://huggingface.co/PelaAI/KnowLine-4B-Gen2) | 2026-10-08 | 62.54 | — | 69.7 / 71.2 / 64.8 | second release |
 | [KnowLine-4B-Gen1](https://huggingface.co/PelaAI/KnowLine-4B-Gen1) | 2026-10-07 | 60.47 | 60.92 | 68.7 / 69.9 / 64.1 | first release |
 
-From Gen2 on we run Decision Index 0.3 only, not 0.2.1.
+Since Gen2, we run only Decision Index 0.3, not 0.2.1.
 
 ## Evaluation
 
-All results are self-run and not verified by a third party.
+All results are self-run and have not been verified by a third party.
 
 ### Held-out and Chinese evaluations
 
@@ -177,7 +176,7 @@ All results are self-run and not verified by a third party.
 | Prompt injection: answers changed (lower is better) | 1.6% | 3.4% |
 | Prompt injection with templates not seen in training (lower is better) | 6.2% | 10.2% |
 
-None of the differences on the in-house sets or Open-Jev 1.1 OOD is significant in a paired sign test.
+None of the differences on the in-house sets or on Open-Jev 1.1 OOD is significant in a paired sign test.
 
 ### Web operation (Mind2Web official test splits, evaluation only)
 
@@ -189,28 +188,32 @@ None of the differences on the in-house sets or Open-Jev 1.1 OOD is significant
 
 - About the same as Gen3. The lower operation score on test_website comes from 3 more misses among its 72 SELECT
   steps; the differences are within noise.
-- So far this is only used to explore the model in RPA-style automation and to check that it generalises to some degree.
-- The task is to pick the target element among it and up to 5 other candidates sampled from the page. This is easier
-  than the original Mind2Web protocol, so do not compare it with the Mind2Web leaderboard.
+- So far, we use this only to explore the model for RPA-style automation and to check that it generalises to some
+  degree.
+- The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the
+  page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web
+  leaderboard.
 
 ### Game harness (KOF '98)
 
-- **Setup:** single-bout character-mirror matches, 18 games per pair, argmax actions; the same settings as for Gen1-Gen3.
-- **Result:** 14-3-1 against Jev 1.13, 10-4-4 against StartLux-Decision-4B and 3-11-4 against Gen3; 27-18-9 overall,
-  score 0.583 [0.45, 0.71]. Against Jev and StartLux this is a little below Gen3 (16-1-1 and 12-4-2), within the wide
-  intervals.
-- **Play style:** it picks moves by distance: mostly the 623C anti-air uppercut up close (79%), and special_2 at mid
-  range (69%) and from far away (92%). Overall it uses special_2 52% and the uppercut 28% of the time.
+- **Setup:** single-bout character-mirror matches, 18 games per pair, argmax actions; the same settings as for
+  Gen1-Gen3.
+- **Result:** 14-3-1 against Jev 1.13, 10-4-4 against StartLux-Decision-4B and 3-11-4 against Gen3; 27-18-9 overall, a
+  score of 0.583 [0.45, 0.71]. Against Jev and StartLux this is slightly below Gen3 (16-1-1 and 12-4-2), but within the
+  wide intervals.
+- **Play style:** it chooses moves by distance: mostly the 623C anti-air uppercut at close range (79%), and special_2 at
+  mid range (69%) and long range (92%). Overall, it uses special_2 52% of the time and the uppercut 28%.
 - **Caveats:**
   - 18 games per pair give wide intervals.
-  - 39 of the 54 games ended at time-out, so most wins are on remaining health. Gen4 won 7 games by K.O. (4 against
-    Jev, 2 against StartLux, 1 against Gen3); 8 of its 11 losses to Gen3 were by K.O.
+  - 39 of the 54 games ended at time-out, so most wins were decided on remaining health. Gen4 won 7 games by K.O. (4
+    against Jev, 2 against StartLux, 1 against Gen3); 8 of its 11 losses to Gen3 were by K.O.
   - This is a measured result in this harness, not general fighting-game skill.
 
 ### Calibration
 
-Computed on the 0.3 public suite with the method the Decision Index board describes: each field is right or wrong,
-confidence is the probability on the chosen option, benchmarks are weighted equally, and there are 10 equal-width bins.
+Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or
+wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are
+used.
 
 | metric | Gen4 | Gen3 | board median (114 models) | Jev 1.13 |
 |---|---|---|---|---|
@@ -220,23 +223,23 @@ confidence is the probability on the chosen option, benchmarks are weighted equa
 | Mean confidence / accuracy | 0.82 / 0.77 | 0.84 / 0.76 | | 0.81 / 0.74 |
 
 - **Temperature in the weights:** the final norm weight includes a readout temperature T = 1.32, so every logit is
-  divided by 1.32 before the softmax over the option labels. Which option ranks first does not change, so neither do
-  the Decision Index scores; only the probabilities are less extreme. T was fitted on a set of our own evaluation items
-  that is covered by the training data, not on the held-out sets or on Decision Index rows.
-- **Why ranges:** the board does not say which 32 benchmarks it uses. We checked our computation against three board
-  models whose results are public. Our ECE was within about 0.015 of the board's, so we give ranges over the plausible
-  benchmark sets.
-- **Still slightly overconfident:** mean confidence is about 3-5 points above accuracy. Most of the gap is on hard
+  divided by 1.32 before the softmax over the option labels. The top-ranked option does not change, so neither do the
+  Decision Index scores; the probabilities are just less extreme. T was fitted on a set of our own evaluation items
+  whose distribution is covered by the training data, not on the held-out sets or on Decision Index rows.
+- **Why ranges:** the board does not publish which 32 benchmarks it uses. We checked our computation against three
+  board models with public results; our ECE was within about 0.015 of the board's values, so we report ranges over the
+  plausible benchmark sets.
+- **Still slightly overconfident:** mean confidence is about 3-5 points above accuracy. Most of the gap comes from hard
   reasoning (HLE, CRUXEval, MuSR) and New Yorker caption matching.
-- **Recommendation:** the temperature was fitted on our data; if you act on probability thresholds, check calibration on
-  your own data.
+- **Recommendation:** the temperature was fitted on our data. If your use depends on probability thresholds, check
+  calibration on your own data.
 
 ## Disclosures
 
-- **Decision Index format training data:** Gen4 comes from four training rounds. In this round, about 34% of the rows
-  were built in the request format of Decision Index benchmarks (some with reworded instructions), and another 11% are
-  the same items rendered in other formats. This includes train splits of public datasets rewritten in that format (for
-  example GSM8K, WinoGrande XL, ACOS and Amazon ESCI), and synthetic items written in the same format.
+- **Decision Index format training data:** Gen4 is the product of four training rounds. In this round, about 34% of the
+  rows were built in the request format of Decision Index benchmarks (some with reworded instructions), and another 11%
+  are copies of the same items rendered in other formats. These include train splits of public datasets rewritten in
+  that format (for example GSM8K, WinoGrande XL, ACOS and Amazon ESCI) and synthetic items written in the same format.
 - **Decontamination:**
   - All training data of this round, new and replayed, was checked with a revised tool against every Decision Index
     0.3 row, all of our evaluation sets and the other public benchmarks we report (Intern-Decision, S1MB, JevBench,
@@ -246,31 +249,27 @@ confidence is the probability on the chosen option, benchmarks are weighted equa
     Wikipedia paragraph or tool description, were kept (23,684 rows).
   - BPoMP and cfcolor training items come from the same source datasets as these benchmarks (OEDILF limericks; the
     O'Donovan et al. colour ratings), with every benchmark item excluded. An item-level check against all 5,000
-    Decision Index rows of each found no shared poem and no shared rating item.
-  - Half of these weights is Gen3, whose data was checked in earlier rounds with an older version of this tool that
-    missed some overlaps. A re-check on 2026-10-09 found that a few benchmarks, such as HoVer and ToolRet, share
-    Wikipedia passages or tool documentation with that data, and about a dozen items overlap closely. Scoring Gen1-Gen3
-    without these items lowers their Decision Index public scores by 0.04-0.05.
+    Decision Index rows of each benchmark found no shared poems and no shared rating items.
 - **Selection on evaluations:** of the 37 candidates, we picked the one with the highest Decision Index 0.3 public
   score among those that passed all of our release checks.
 - **Game data:** labels come from simulator rollouts or engines. Seeds and start states are disjoint from the evaluation
   states.
 - **No model outputs as labels:** no output of Jev or any other decision model was used as a training label.
 - **Teacher-labelled synthetic data:** LLM teachers wrote and labelled our synthetic tasks. The labels were filtered, but
-  not all were checked by a human.
+  not all of them were checked by a human.
 
 ## Limitations
 
-- **Knowledge-heavy reasoning:** weaker than larger models. The knowledge area is 44.6 (0.3); MMLU-Pro moved little
-  (51.6) and HLE stays at chance level.
-- **Math:** answered without reasoning. The rebuilt 0.3 GSM8K scores 61.6, about the same as Gen3; it comes from training
-  on GSM8K train rewritten in the same format.
+- **Knowledge-heavy reasoning:** weaker than larger models. The knowledge area score is 44.6 (DI 0.3); MMLU-Pro barely
+  moved (51.6), and HLE remains at chance level.
+- **Math:** answers are given without step-by-step reasoning. The rebuilt 0.3 GSM8K scores 61.6, about the same as
+  Gen3; this score comes from training on the GSM8K train split rewritten in the same format.
 - **New domains:** on our own tasks from domains not in training, Gen4 scores about the same as Gen3 (49.3 vs 49.5).
   The gains of this round are in robustness and calibration, not in new-domain ability.
 - **Prompt injection:** an instruction planted in the state changes the answer about 2% of the time on our injection
-  set, and about 6% with templates not seen in training. Keep untrusted text clearly delimited.
-- **Private tests:** most of the public-suite gain over Gen3 comes from benchmarks whose task is in our training data,
-  so the 0.3 private tests may show a much smaller gain, or a lower score.
+  set, and about 6% of the time with templates not seen in training. Keep untrusted text clearly delimited.
+- **Private tests:** most of the public-suite gain over Gen3 comes from benchmarks whose tasks are in our training data,
+  so the 0.3 private tests may show a much smaller gain, or even a lower score.
 
 ## Citation