README.md: README.md: mirror the README.zh.md wording of How Gen1 was made, item 1
e800968 verified 12 days ago
21.1 kB
---
license: apache-2.0
base_model:
  - Qwen/Qwen3.5-35B-A3B
base_model_relation: finetune
library_name: transformers
pipeline_tag: text-classification
language:
  - en
  - zh
tags:
  - decision-model
  - system-one
  - decision-index
  - lora-merged
  - moe
---
# KnowLine-35B-A3B-Gen1

**The first release of PelaAI's 35B-A3B System One decision model, trained on the same single all-in-one machine and
agent-driven data loop as the KnowLine-4B line.**

[Inference guide](INFERENCE.md) · [中文说明](README.zh.md) · weights Apache-2.0 · smaller sibling:
[KnowLine-4B-Gen4](https://huggingface.co/PelaAI/KnowLine-4B-Gen4)

Highlights:

- **Decision Index 0.3 public suite: 69.49** (self-run), 4.59 points above KnowLine-4B-Gen4 (64.90). On the board, the
  highest public score is 65.64 (RSI-Jev v6.1 VL 27B), followed by vLLM-SR d3 (27B) at 65.45, which has the highest
  full score (66.26); Perplexity Decider v1.1 (27B) scores 62.25 and Jev 1.13 scores 57.96; see
  [Comparison](#comparison).
- **Stronger where the 4B was weakest:** knowledge and reasoning rises from 44.6 (Gen4) to 52.5, close to Jev 1.13
  (53.9). The largest gains over Gen4 are on knowledge-heavy benchmarks such as MMLU-Pro (+15.0) and BBH (+11.8);
  see [Where the gain over Gen4 comes from](#where-the-gain-over-gen4-comes-from).
- **Better on new domains:** on our own decision tasks from domains not in training it scores 56.4, against
  49.3 for Gen4.
- **Harder to steer with planted instructions:** on our injection set, no answer changed because of an instruction
  planted in the state (Gen4: 1.6%); with injection templates not seen in training, 0.8% did (Gen4: 6.2%).
- **Calibrated probabilities:** a temperature is folded into the weights. Computed with the board's method, ECE is about
  0.04-0.05 (board median: 0.078) and the Brier score about 0.25-0.27, below the lowest Brier score on the board
  (0.28); see [Calibration](#calibration).
- **Our other evaluations:** Open-Jev 1.1 test / OOD 91.0 / 90.4 (Gen4: 88.4 / 87.1), C-Eval 87.6 (78.7), and higher
  scores on our Chinese held-out sets. JevBench / JevBench-hard 85.7 / 70.3 (86.1 / 72.1): slightly lower (the
  JevBench-hard difference is not significant).
- **Images without any image training:** like Gen4, it was trained on text only and keeps the base model's vision tower.
  On our reconstruction of the Decision Index 0.3.1 vision public suite it scores 71.2, against 67.8 for the stock
  Qwen3.5-35B-A3B on the same harness (self-run), with gains on some benchmarks and small drops on others; see
  [Images](#images-zero-shot).
- **KOF '98 harness: 17-0-1 against Jev 1.13, 18-0-0 against StartLux-Decision-35B-A3B and 12-4-2 against
  KnowLine-4B-Gen4** in single-bout mirror matches; see [Game harness](#game-harness-kof-98).
- **An AI agent ran the whole loop end to end.** In each round it:
  - identified where the model was weak;
  - proposed a targeted data group;
  - built and decontaminated the data;
  - trained and evaluated the model on it;
  - kept or rejected the change based on the evidence.

  For this model, see [How Gen1 was made](#how-gen1-was-made).

KnowLine is an independent model. It is not affiliated with, endorsed by, or derived from TypeSafe or Jev.

## What it does

You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model
answers every question with a probability distribution over its options:

- one forward pass per question, with no text generation;
- the output is the probability of each option's label token;
- existing Jev clients only need to change the base URL, and users of KnowLine-4B only need to change the model name,
  because the interface is the same.

| | |
|---|---|
| Base model | [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) (Apache-2.0): mixture of experts, 35B parameters in total, about 3B active per token |
| Training | LoRA SFT (rank 32, alpha 64, language model only, including the routed experts), merged into bf16 weights |
| Release format | bf16 weights (about 67 GB), with a readout temperature T = 1.20 folded into the final norm weight (see [Calibration](#calibration)). The vision tower and MTP head are unchanged from the base model; the tokenizer is the same as in KnowLine-4B. |
| Serving | the same `serve_knowline.sh` as KnowLine-4B: SGLang converts the weights to FP8 at load (about 35 GB). We ran every evaluation with one server on one 96 GB H20; plan for at least 48 GB of GPU memory. |
| Languages | English, Simplified Chinese, Traditional Chinese |

## Quickstart

```bash
pip install "sglang==0.5.21" "transformers==5.12.1" requests
bash serve_knowline.sh PelaAI/KnowLine-35B-A3B-Gen1 0 8080     # SGLang (FP8 at load) on :9080 + /v1/systemone on :8080
curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "m",
  "state": "Customer: my order arrived broken, I want my money back.",
  "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"},
                "tone": {"type": "choice", "instructions": "Customer tone?",
                         "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'
```

- **Front end:** `knowline_server.py` is a single file that depends only on transformers and requests. It is the same
  code as in KnowLine-4B-Gen4.
- **Full settings:** the exact settings used for our Decision Index run are in [INFERENCE.md](INFERENCE.md).

## How Gen1 was made

1. **Why a larger base?** The team trained the KnowLine-4B series, most recently updated to
   [KnowLine-4B-Gen4](https://huggingface.co/PelaAI/KnowLine-4B-Gen4). Over four rounds, it gained most on retrieval,
   tools and taste, while knowledge and reasoning stayed the weakest area (44.6 in Gen4). The agent trained the same data
   on a larger base to see how much of that gap comes from model size.
2. **Data.** Two rounds:
   - all training data of the KnowLine-4B line up to Gen4, in one set;
   - a second round of inputs no earlier round used: more rows from the same public train sets, format variants of
     existing items, about 40 new task families from domains not covered before, and synthetic scenarios, plus a replay
     share of the first round (arithmetic, knowledge, Jev-style and injection-invariance items at a higher rate).
   All data went through the same decontamination check as Gen4; see [Disclosures](#disclosures).
3. **Training.** The base model was trained on the first round. The result was then trained further on the second round,
   and the two were averaged 50/50. A single temperature was fitted and folded into the weights. It changes the size of
   the probabilities, but not which option ranks first.
4. **Selection.** We compared 10 candidates (checkpoints of both rounds and two averages), with Decision Index 0.3
   public scores between 68.09 and 69.60. Every candidate had to pass the same checks as Gen4, against
   KnowLine-4B-Gen4 and against the first-round model: no significant drop on our held-out sets, Open-Jev 1.1 OOD or
   JevBench-hard; an injection rate of at most 4%; and no regression beyond set margins on new-domain tasks, sibling
   skills, format consistency, unseen injection templates or calibration (measured after applying the temperature).
   Gen1 has the highest public score among the candidates that passed: 69.60 before the temperature was folded in, and
   69.49 in the fresh run of the released weights (the temperature does not change the ranking, so the difference is
   run-to-run noise).
5. **What the second round did.** On the public suite, the second round alone did not add points: its checkpoints
   scored 68.84-69.14, against 69.40 for the first-round model, and the average scores 69.49-69.60. It did help where
   the public suite does not look: new-domain tasks 55.0 → 56.4, sibling skills 65.6 → 66.4, Open-Jev 1.1 OOD
   89.7 → 90.4. The cost is a small drop on our Simplified Chinese held-out set (−0.8, p = 0.051).

### Where the gain over Gen4 comes from

Gen1 and Gen4 were trained on largely the same kinds of tasks. For 33 of the 37 benchmarks, the training data holds
items of the same task or from the same source; only MMLU-Pro, BBH, GPQA Diamond and HLE have none. Most of the 4.59
points therefore come from the larger base model rather than from new benchmark data:

- About 3.9 points come from the 33 benchmarks with training data, and about 0.7 from the four without. Per benchmark,
  the four without training data gained more on average.
- Largest gains, skill Gen4 → Gen1: iSarcasmEval 40.3 → 64.2, CRUXEval 43.8 → 63.3, MMLU-Pro 51.6 → 66.6 (no training
  data), MMLU 69.9 → 82.2, BBH 53.1 → 64.9 (no training data), ANLI 51.7 → 62.4 and New Yorker caption matching
  64.5 → 73.0.
- The largest drops are small: POP909-CL (85.0 → 83.4) and Habermas Machine (48.1 → 46.8).

## Comparison

Scores on the Decision Index 0.3 public suite. The full 0.3 score also includes private tests that only the maintainers
run (weighted 0.5 same-skill and 0.3 new-domain); our full score is not available yet.

| model | size | DI 0.3 public | DI 0.3 full | source |
|---|---|---|---|---|
| **KnowLine-35B-A3B-Gen1** | 35B-A3B | **69.49** | not yet scored | self-run, official kit |
| KnowLine-4B-Gen4 | 4B | 64.90 | not yet scored | self-run, official kit |
| RSI-Jev v6.1 VL 27B (effort auto) | 27B | 65.64 | 58.66 | board |
| vLLM-SR d3 | 27B | 65.45 | 66.26 | board |
| Torchcast Decision 27B | 27B | 65.10 | 59.91 | board |
| Perplexity Decider v1.1 | 27B | 62.25 | 62.75 | board |
| Clef | 27B | 61.71 | 53.08 | board |
| Jev 1.13 | (API) | 57.96 | 60.11 | board |
| Cloudflare clef-omni | 30B-A3B | 56.54 | 48.28 | board |
| JPT-35B-A3B | 35B-A3B | 52.68 | 51.41 | board |
| decisio 0.4.0 | 35B-A3B | 48.43 | 47.72 | board |

Board rows as of the board's latest update (2026-10-13).

Public-suite scores by area (DI 0.3):

| area | KnowLine-35B-A3B-Gen1 | KnowLine-4B-Gen4 | Perplexity Decider v1.1 (27B) | Jev 1.13 |
|---|---|---|---|---|
| Knowledge & reasoning | 52.5 | 44.6 | 52.0 | 53.9 |
| Language | 71.2 | 65.5 | 67.5 | 59.2 |
| Retrieval & routing | 75.2 | 72.5 | 61.3 | 55.4 |
| Tools & agents | 88.1 | 87.6 | 78.9 | 75.1 |
| Arts & taste | 63.4 | 59.2 | 47.0 | 39.1 |

## Releases

The model is trained in a self-evolving loop, and new versions will follow.

| model | date | DI 0.3 public | golden held-out (en / zh-Hans / zh-Hant) | notes |
|---|---|---|---|---|
| KnowLine-35B-A3B-Gen1 (this model) | 2026-10-12 | 69.49 | 70.3 / 73.8 / 72.1 | first 35B-A3B release; board figures updated 2026-10-13 |
| [KnowLine-4B-Gen4](https://huggingface.co/PelaAI/KnowLine-4B-Gen4) | 2026-10-10 | 64.90 | 69.7 / 70.1 / 65.1 | 4B, fourth release |
| [KnowLine-4B-Gen3](https://huggingface.co/PelaAI/KnowLine-4B-Gen3) | 2026-10-09 | 63.11 | 69.4 / 70.8 / 65.0 | 4B, third release |
| [KnowLine-4B-Gen2](https://huggingface.co/PelaAI/KnowLine-4B-Gen2) | 2026-10-08 | 62.54 | 69.7 / 71.2 / 64.8 | 4B, second release |
| [KnowLine-4B-Gen1](https://huggingface.co/PelaAI/KnowLine-4B-Gen1) | 2026-10-07 | 60.47 | 68.7 / 69.9 / 64.1 | 4B, first release |

## Evaluation

All results are self-run and have not been verified by a third party.

### Held-out and Chinese evaluations

| suite | 35B-A3B-Gen1 | 4B-Gen4 |
|---|---|---|
| In-house evaluation set, English | 70.3 | 69.7 |
| In-house evaluation set, Simplified Chinese | 73.8 | 70.1 |
| In-house evaluation set, Traditional Chinese | 72.1 | 65.1 |
| C-Eval (4 categories, macro) | 87.6 | 78.7 |
| Open-Jev 1.1 test / OOD | 91.0 / 90.4 | 88.4 / 87.1 |
| Decision tasks from new domains (our own) | 56.4 | 49.3 |
| Prompt injection: answers changed (lower is better) | 0.0% | 1.6% |
| Prompt injection with templates not seen in training (lower is better) | 0.8% | 6.2% |

In a paired sign test against Gen4, the gains on the Simplified and Traditional Chinese in-house sets and on Open-Jev
1.1 OOD are significant (p < 0.001); the differences on the English in-house set and on JevBench-hard are not.

### Web operation (Mind2Web official test splits, evaluation only)

| split | 35B-A3B-Gen1 element selection | 35B-A3B-Gen1 operation (balanced) | 4B-Gen4 element selection | 4B-Gen4 operation (balanced) |
|---|---|---|---|---|
| test_task (websites seen in training, new tasks) | 94.4 | 97.4 | 92.9 | 97.0 |
| test_website (new websites) | 92.4 | 97.5 | 91.9 | 95.7 |
| test_domain (new domains) | 93.1 | 98.0 | 91.7 | 97.2 |
- Slightly above Gen4 on all three splits; no request failed.
- So far, we use this only to explore the model for RPA-style automation and to check that it generalises to some
  degree.
- The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the
  page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web
  leaderboard.
### Images (zero-shot)
- **No image training:** Gen1 keeps the base model's vision tower unchanged and saw no images in training.
  `knowline_server.py` accepts OpenAI-style `image` / `image_url` content parts in a chat state and passes them to
  SGLang; each question is still one prefill.
- **How we measured it:** the same reconstruction of the Decision Index 0.3.1 vision public suite as for Gen4: 11
  benchmarks, the board's row counts where the sources allow, images capped at 1.6 MP, chance-corrected skill and the
  board's weights, run through the release server: 12,192 requests, 0 failed.
| benchmark | 35B-A3B-Gen1 | stock Qwen3.5-35B-A3B (same front end) | 4B-Gen4 |
|---|---|---|---|
| CV-Bench | 75.8 | 76.4 | 75.7 |
| BLINK | 59.8 | 66.4 | 58.1 |
| RealWorldQA | 71.9 | 64.7 | 64.7 |
| CharXiv | 66.9 | 56.3 | 66.3 |
| InfographicVQA | 90.7 | 79.8 | 86.3 |
| Mind2Web | 86.4 | 75.5 | 77.4 |
| Winoground | 80.0 | 80.8 | 80.5 |
| KIE (CORD+FUNSD) | 96.2 | 96.8 | 95.5 |
| Moderation (Hateful Memes) | 41.1 | 42.6 | 27.1 |
| R-Bench-M | 29.1 | 24.4 | 14.6 |
| MMMU-Pro vision | 45.1 | 38.9 | 31.9 |
| **weighted public index** | **71.2** | 67.8 | 66.4 |
- **Mixed against the stock model:** 3.4 points higher overall. The gains are on charts, infographics, web screenshots,
  exam pages and RealWorldQA; BLINK (multi-image perception, −6.6), Hateful Memes, Winoground, KIE and CV-Bench are
  slightly lower.
- **Indicative only:** these are self-run numbers on our reconstruction, not on the board's rows. Our CharXiv distractors
  lean towards numeric questions, and our Mind2Web marks and KIE questions are our own construction.
### Game harness (KOF '98)
- **Setup:** single-bout character-mirror matches, 18 games per pair, argmax actions; the same settings as for the 4B
  line.
- **Result:** 17-0-1 against Jev 1.13, 18-0-0 against StartLux-Decision-35B-A3B and 12-4-2 against KnowLine-4B-Gen4;
  47-4-3 overall, a score of 0.898 [0.79, 0.95].
- **Play style:** it chooses moves by distance: at close range mostly the 623C anti-air uppercut (45%) and heavy punch
  (41%), special_2 at mid range (68%), and mostly waiting at long range (64%). Overall, it uses special_2 36% of the
  time, the uppercut 16%, heavy punch 14%, special_3 13% and waiting 13%, a more varied mix than Gen4's.
- **Caveats:**
  - 18 games per pair give wide intervals.
  - 27 of the 54 games ended at time-out, so those were decided on remaining health. Gen1 won 25 games by K.O. (13
    against StartLux, 9 against Gen4, 3 against Jev); its 2 K.O. losses were both against Gen4. Against Jev, 14 of the
    17 wins came at time-out.
  - This is a measured result in this harness, not general fighting-game skill.
### Calibration

Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or
wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are
used.

| metric | 35B-A3B-Gen1 | 4B-Gen4 | board median (159 models) | Jev 1.13 |
|---|---|---|---|---|
| ECE (lower is better) | 0.04-0.05 | 0.04-0.05 | 0.078 | 0.074 |
| Brier score (lower is better) | 0.25-0.27 | 0.29-0.31 | 0.47 | 0.36 |
| Confidence ≥95% but wrong | 1.7% | 1.7% | 1.5% | 2.1% |
| Mean confidence / accuracy | 0.85 / 0.80 | 0.82 / 0.77 | | 0.81 / 0.74 |

- **Temperature in the weights:** the final norm weight includes a readout temperature T = 1.20, so every logit is
  divided by 1.20 before the softmax over the option labels. The top-ranked option does not change, so neither do the
  Decision Index scores; the probabilities are just less extreme. T was fitted on a set of our own evaluation items
  whose distribution is covered by the training data, not on the held-out sets or on Decision Index rows. On our
  held-out sets, ECE with the temperature in the weights is 0.025 (Gen4: 0.043).
- **Why ranges:** the board does not publish which 32 benchmarks it uses. We checked our computation against three
  board models with public results; our ECE was within about 0.015 of the board's values, so we report ranges over the
  plausible benchmark sets.
- **Brier score:** lower than every model on the board; the lowest there is 0.284 (Torchcast Decision 27B). Part of
  this comes from higher accuracy, not only from calibration.
- **Still slightly overconfident:** mean confidence is about 5 points above accuracy. Most of the gap comes from HLE,
  hard reasoning (MuSR, CRUXEval, GPQA Diamond), cfcolor and New Yorker caption matching.
- **Recommendation:** the temperature was fitted on our data. If your use depends on probability thresholds, check
  calibration on your own data.

## Disclosures

- **Decision Index format training data:** in the first round, about 21% of the rows were built in the request format
  of Decision Index benchmarks (some with reworded instructions), and another 4% are copies of the same items rendered in
  other formats; in the second round, about 8% and 9%. These include train splits of public datasets rewritten in that format (for example GSM8K,
  WinoGrande XL, ACOS and Amazon ESCI) and synthetic items written in the same format.
- **Decontamination:**
  - All training data, of both rounds, was checked with the same tool as Gen4 against every Decision Index 0.3 row, all
    of our evaluation sets and the other public benchmarks we report (Intern-Decision, S1MB, JevBench, MMDM).
  - Rows that nearly duplicate an evaluation item, repeat its question or claim, or contain its whole state were
    removed: 7,833 of 1,500,018 rows in the first round and 2,445 of 985,519 in the second. Rows that only share a
    source passage with an evaluation item, such as the same Wikipedia paragraph or tool description, were kept (31,249
    and 5,702 rows).
  - BPoMP and cfcolor training items come from the same source datasets as these benchmarks (OEDILF limericks; the
    O'Donovan et al. colour ratings), with every benchmark item excluded, as in Gen4.
- **Selection on evaluations:** of the 10 candidates, we picked the one with the highest Decision Index 0.3 public
  score among those that passed all of our release checks.
- **Game data:** labels come from simulator rollouts or engines. Seeds and start states are disjoint from the evaluation
  states.
- **No model outputs as labels:** no output of Jev or any other decision model was used as a training label.
- **Teacher-labelled synthetic data:** LLM teachers wrote and labelled our synthetic tasks. The labels were filtered, but
  not all of them were checked by a human.
## Limitations

- **Size and speed:** it needs much more GPU memory than KnowLine-4B (at least 48 GB with FP8 at load), and each
  request takes longer: 51 ms median, one request at a time on one H20, against about 31 ms for KnowLine-4B (see
  [INFERENCE.md](INFERENCE.md)).
- **Knowledge-heavy reasoning:** better than the 4B line but still the weakest area (52.5, DI 0.3); HLE remains at
  chance level.
- **Math:** answers are given without step-by-step reasoning. The rebuilt 0.3 GSM8K scores 69.3; part of this comes
  from training on the GSM8K train split rewritten in the same format.
- **Prompt injection:** on our injection set, planted instructions changed no answers, and about 1% with templates not
  seen in training. Other attacks may still work, so keep untrusted text clearly delimited.
- **Images:** there was no image training. The image scores above are zero-shot and self-run on our own reconstruction of
  the suite, so check them on your own data before relying on them.
- **Private tests:** part of the public-suite score comes from benchmarks whose tasks are in our training data, so the
  0.3 private tests may show a lower score.

## Citation

```bibtex
@misc{knowline35ba3bgen1,
  title  = {KnowLine-35B-A3B-Gen1: a 35B-A3B decision model},
  author = {PelaAI},
  year   = {2026},
  url    = {https://huggingface.co/PelaAI/KnowLine-35B-A3B-Gen1}
}
```

## License

- **Weights:** Apache-2.0, the same as the base model.
- **Code:** `knowline_server.py` is MIT (see the file header).