Text Classification
Transformers
Safetensors
English
Chinese
qwen3_5
image-text-to-text
decision-model
system-one
decision-index
lora-merged
Instructions to use PelaAI/KnowLine-4B-Gen4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use PelaAI/KnowLine-4B-Gen4 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="PelaAI/KnowLine-4B-Gen4")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("PelaAI/KnowLine-4B-Gen4") model = AutoModelForMultimodalLM.from_pretrained("PelaAI/KnowLine-4B-Gen4", device_map="auto") - Notebooks
- Google Colab
- Kaggle
README.md: add zero-shot image results
Browse files
README.md
CHANGED
|
@@ -36,6 +36,9 @@ Highlights:
|
|
| 36 |
board; see [Calibration](#calibration).
|
| 37 |
- **On par with Gen3 on our other evaluations:** JevBench / JevBench-hard 86.1 / 72.1 (Gen3: 86.6 / 72.1), Open-Jev 1.1
|
| 38 |
test / OOD 88.4 / 87.1 (88.4 / 87.2), C-Eval 78.7 (79.2).
|
|
|
|
|
|
|
|
|
|
| 39 |
- **KOF '98 harness: 14-3-1 against Jev 1.13 and 10-4-4 against StartLux-Decision-4B** in single-bout mirror matches;
|
| 40 |
see [Game harness](#game-harness-kof-98).
|
| 41 |
- **An AI agent ran the whole loop end to end.** In each round it:
|
|
@@ -194,6 +197,36 @@ None of the differences on the in-house sets or on Open-Jev 1.1 OOD is significa
|
|
| 194 |
page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web
|
| 195 |
leaderboard.
|
| 196 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 197 |
### Game harness (KOF '98)
|
| 198 |
|
| 199 |
- **Setup:** single-bout character-mirror matches, 18 games per pair, argmax actions; the same settings as for
|
|
@@ -268,6 +301,8 @@ used.
|
|
| 268 |
The gains of this round are in robustness and calibration, not in new-domain ability.
|
| 269 |
- **Prompt injection:** an instruction planted in the state changes the answer about 2% of the time on our injection
|
| 270 |
set, and about 6% of the time with templates not seen in training. Keep untrusted text clearly delimited.
|
|
|
|
|
|
|
| 271 |
- **Private tests:** most of the public-suite gain over Gen3 comes from benchmarks whose tasks are in our training data,
|
| 272 |
so the 0.3 private tests may show a much smaller gain, or even a lower score.
|
| 273 |
|
|
|
|
| 36 |
board; see [Calibration](#calibration).
|
| 37 |
- **On par with Gen3 on our other evaluations:** JevBench / JevBench-hard 86.1 / 72.1 (Gen3: 86.6 / 72.1), Open-Jev 1.1
|
| 38 |
test / OOD 88.4 / 87.1 (88.4 / 87.2), C-Eval 78.7 (79.2).
|
| 39 |
+
- **Images without any image training:** Gen4 was trained on text only, yet its serving path accepts images and the
|
| 40 |
+
decision skills carry over. On our reconstruction of the Decision Index 0.3.1 vision public suite it scores 66.4,
|
| 41 |
+
against 59.4 for the stock Qwen3.5-4B on the same harness (self-run); see [Images](#images-zero-shot).
|
| 42 |
- **KOF '98 harness: 14-3-1 against Jev 1.13 and 10-4-4 against StartLux-Decision-4B** in single-bout mirror matches;
|
| 43 |
see [Game harness](#game-harness-kof-98).
|
| 44 |
- **An AI agent ran the whole loop end to end.** In each round it:
|
|
|
|
| 197 |
page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web
|
| 198 |
leaderboard.
|
| 199 |
|
| 200 |
+
### Images (zero-shot)
|
| 201 |
+
|
| 202 |
+
- **No image training:** Gen4 keeps the base model's vision tower unchanged and saw no images in training.
|
| 203 |
+
`knowline_server.py` accepts OpenAI-style `image` / `image_url` content parts in a chat state and passes them to
|
| 204 |
+
SGLang; each question is still one prefill.
|
| 205 |
+
- **How we measured it:** we rebuilt the Decision Index 0.3.1 vision public suite as closely as the board's
|
| 206 |
+
descriptions allow: 11 benchmarks, the board's row counts where the sources allow, images capped at 1.6 MP,
|
| 207 |
+
chance-corrected skill and the board's weights. We ran it through the release server: 12,192 requests, 0 failed.
|
| 208 |
+
|
| 209 |
+
| benchmark | Gen4 | stock Qwen3.5-4B (same front end) |
|
| 210 |
+
|---|---|---|
|
| 211 |
+
| CV-Bench | 75.7 | 72.1 |
|
| 212 |
+
| BLINK | 58.1 | 59.9 |
|
| 213 |
+
| RealWorldQA | 64.7 | 59.7 |
|
| 214 |
+
| CharXiv | 66.3 | 52.4 |
|
| 215 |
+
| InfographicVQA | 86.3 | 73.2 |
|
| 216 |
+
| Mind2Web | 77.4 | 65.3 |
|
| 217 |
+
| Winoground | 80.5 | 73.5 |
|
| 218 |
+
| KIE (CORD+FUNSD) | 95.5 | 95.7 |
|
| 219 |
+
| Moderation (Hateful Memes) | 27.1 | 18.4 |
|
| 220 |
+
| R-Bench-M | 14.6 | 6.3 |
|
| 221 |
+
| MMMU-Pro vision | 31.9 | 21.4 |
|
| 222 |
+
| **weighted public index** | **66.4** | 59.4 |
|
| 223 |
+
|
| 224 |
+
- **The decision skills generalise to images:** Gen4 is 7.0 points above the stock model, with the largest gains on
|
| 225 |
+
charts, infographics, web screenshots and exam pages, although no image data was used in training.
|
| 226 |
+
- **Indicative only:** these are self-run numbers on our reconstruction, not on the board's rows. Our CharXiv distractors
|
| 227 |
+
lean towards numeric questions, and our Mind2Web marks and KIE questions are our own construction.
|
| 228 |
+
- **Weakest areas:** Hateful Memes, the expert questions of R-Bench-M and MMMU-Pro, and BLINK (multi-image perception).
|
| 229 |
+
|
| 230 |
### Game harness (KOF '98)
|
| 231 |
|
| 232 |
- **Setup:** single-bout character-mirror matches, 18 games per pair, argmax actions; the same settings as for
|
|
|
|
| 301 |
The gains of this round are in robustness and calibration, not in new-domain ability.
|
| 302 |
- **Prompt injection:** an instruction planted in the state changes the answer about 2% of the time on our injection
|
| 303 |
set, and about 6% of the time with templates not seen in training. Keep untrusted text clearly delimited.
|
| 304 |
+
- **Images:** there was no image training. The image scores above are zero-shot and self-run on our own reconstruction of
|
| 305 |
+
the suite, so check them on your own data before relying on them.
|
| 306 |
- **Private tests:** most of the public-suite gain over Gen3 comes from benchmarks whose tasks are in our training data,
|
| 307 |
so the 0.3 private tests may show a much smaller gain, or even a lower score.
|
| 308 |
|