PEScn commited on
Commit
3bbb9cd
·
verified ·
1 Parent(s): 5482f96

README.md: add zero-shot image results

Browse files
Files changed (1) hide show
  1. README.md +35 -0
README.md CHANGED
@@ -36,6 +36,9 @@ Highlights:
36
  board; see [Calibration](#calibration).
37
  - **On par with Gen3 on our other evaluations:** JevBench / JevBench-hard 86.1 / 72.1 (Gen3: 86.6 / 72.1), Open-Jev 1.1
38
  test / OOD 88.4 / 87.1 (88.4 / 87.2), C-Eval 78.7 (79.2).
 
 
 
39
  - **KOF '98 harness: 14-3-1 against Jev 1.13 and 10-4-4 against StartLux-Decision-4B** in single-bout mirror matches;
40
  see [Game harness](#game-harness-kof-98).
41
  - **An AI agent ran the whole loop end to end.** In each round it:
@@ -194,6 +197,36 @@ None of the differences on the in-house sets or on Open-Jev 1.1 OOD is significa
194
  page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web
195
  leaderboard.
196
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
197
  ### Game harness (KOF '98)
198
 
199
  - **Setup:** single-bout character-mirror matches, 18 games per pair, argmax actions; the same settings as for
@@ -268,6 +301,8 @@ used.
268
  The gains of this round are in robustness and calibration, not in new-domain ability.
269
  - **Prompt injection:** an instruction planted in the state changes the answer about 2% of the time on our injection
270
  set, and about 6% of the time with templates not seen in training. Keep untrusted text clearly delimited.
 
 
271
  - **Private tests:** most of the public-suite gain over Gen3 comes from benchmarks whose tasks are in our training data,
272
  so the 0.3 private tests may show a much smaller gain, or even a lower score.
273
 
 
36
  board; see [Calibration](#calibration).
37
  - **On par with Gen3 on our other evaluations:** JevBench / JevBench-hard 86.1 / 72.1 (Gen3: 86.6 / 72.1), Open-Jev 1.1
38
  test / OOD 88.4 / 87.1 (88.4 / 87.2), C-Eval 78.7 (79.2).
39
+ - **Images without any image training:** Gen4 was trained on text only, yet its serving path accepts images and the
40
+ decision skills carry over. On our reconstruction of the Decision Index 0.3.1 vision public suite it scores 66.4,
41
+ against 59.4 for the stock Qwen3.5-4B on the same harness (self-run); see [Images](#images-zero-shot).
42
  - **KOF '98 harness: 14-3-1 against Jev 1.13 and 10-4-4 against StartLux-Decision-4B** in single-bout mirror matches;
43
  see [Game harness](#game-harness-kof-98).
44
  - **An AI agent ran the whole loop end to end.** In each round it:
 
197
  page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web
198
  leaderboard.
199
 
200
+ ### Images (zero-shot)
201
+
202
+ - **No image training:** Gen4 keeps the base model's vision tower unchanged and saw no images in training.
203
+ `knowline_server.py` accepts OpenAI-style `image` / `image_url` content parts in a chat state and passes them to
204
+ SGLang; each question is still one prefill.
205
+ - **How we measured it:** we rebuilt the Decision Index 0.3.1 vision public suite as closely as the board's
206
+ descriptions allow: 11 benchmarks, the board's row counts where the sources allow, images capped at 1.6 MP,
207
+ chance-corrected skill and the board's weights. We ran it through the release server: 12,192 requests, 0 failed.
208
+
209
+ | benchmark | Gen4 | stock Qwen3.5-4B (same front end) |
210
+ |---|---|---|
211
+ | CV-Bench | 75.7 | 72.1 |
212
+ | BLINK | 58.1 | 59.9 |
213
+ | RealWorldQA | 64.7 | 59.7 |
214
+ | CharXiv | 66.3 | 52.4 |
215
+ | InfographicVQA | 86.3 | 73.2 |
216
+ | Mind2Web | 77.4 | 65.3 |
217
+ | Winoground | 80.5 | 73.5 |
218
+ | KIE (CORD+FUNSD) | 95.5 | 95.7 |
219
+ | Moderation (Hateful Memes) | 27.1 | 18.4 |
220
+ | R-Bench-M | 14.6 | 6.3 |
221
+ | MMMU-Pro vision | 31.9 | 21.4 |
222
+ | **weighted public index** | **66.4** | 59.4 |
223
+
224
+ - **The decision skills generalise to images:** Gen4 is 7.0 points above the stock model, with the largest gains on
225
+ charts, infographics, web screenshots and exam pages, although no image data was used in training.
226
+ - **Indicative only:** these are self-run numbers on our reconstruction, not on the board's rows. Our CharXiv distractors
227
+ lean towards numeric questions, and our Mind2Web marks and KIE questions are our own construction.
228
+ - **Weakest areas:** Hateful Memes, the expert questions of R-Bench-M and MMMU-Pro, and BLINK (multi-image perception).
229
+
230
  ### Game harness (KOF '98)
231
 
232
  - **Setup:** single-bout character-mirror matches, 18 games per pair, argmax actions; the same settings as for
 
301
  The gains of this round are in robustness and calibration, not in new-domain ability.
302
  - **Prompt injection:** an instruction planted in the state changes the answer about 2% of the time on our injection
303
  set, and about 6% of the time with templates not seen in training. Keep untrusted text clearly delimited.
304
+ - **Images:** there was no image training. The image scores above are zero-shot and self-run on our own reconstruction of
305
+ the suite, so check them on your own data before relying on them.
306
  - **Private tests:** most of the public-suite gain over Gen3 comes from benchmarks whose tasks are in our training data,
307
  so the 0.3 private tests may show a much smaller gain, or even a lower score.
308