leyili6666 commited on
Commit
0d56bc1
·
verified ·
1 Parent(s): 53d2a7e

Add benchmark gains in compact model-level tables

Browse files

Restore 14 previously verified public-benchmark improvements in three model rows grouped by reasoning, knowledge, and code. Compact the ScienceAccelBench environment table to two model rows. Preserve paired scores, gains, and subset labels.

Files changed (1) hide show
  1. README.md +18 -6
README.md CHANGED
@@ -41,12 +41,24 @@ Weight sizes are approximate; inference also requires memory for runtime allocat
41
 
42
  Task-held-out, localized scientific-code repair on familiar codebases, with original numerical verification. Each row compares the initial model with its corresponding fine-tuned model on identical tasks. Pass rates are percentages; gains are percentage points.
43
 
44
- | Model | Environment | Tasks | Initial model | PhAI-IDE | Gain (pp) |
45
- | --- | --- | ---: | ---: | ---: | ---: |
46
- | 4B | PLUTO-Particles-Dust | 3 | 0.00 | **33.33** | **+33.33** |
47
- | 9B | LAPS | 16 | 31.25 | **50.00** | **+18.75** |
48
- | 9B | MITgcm-biogeo | 8 | 0.00 | **12.50** | **+12.50** |
49
- | 9B | PLUTO-RMHD | 7 | 0.00 | **28.57** | **+28.57** |
 
 
 
 
 
 
 
 
 
 
 
 
50
 
51
  ## Quick start
52
 
 
41
 
42
  Task-held-out, localized scientific-code repair on familiar codebases, with original numerical verification. Each row compares the initial model with its corresponding fine-tuned model on identical tasks. Pass rates are percentages; gains are percentage points.
43
 
44
+ Each entry shows **initial model → PhAI-IDE (gain in percentage points)**, with task counts in brackets.
45
+
46
+ | Model | Scientific environment results |
47
+ | --- | --- |
48
+ | 4B | PLUTO-Particles-Dust [3]: 0.00 → **33.33** (**+33.33**) |
49
+ | 9B | LAPS [16]: 31.25 → **50.00** (**+18.75**)<br>MITgcm-biogeo [8]: 0.00 → **12.50** (**+12.50**)<br>PLUTO-RMHD [7]: 0.00 → **28.57** (**+28.57**) |
50
+
51
+ ## Public benchmark performance
52
+
53
+ Each entry shows **initial model → PhAI-IDE (gain in percentage points)**. Scores are percentages, with identical items and settings within each model pair. Gains are calculated before rounding.
54
+
55
+ | Model | Reasoning | Knowledge | Code |
56
+ | --- | --- | --- | --- |
57
+ | 4B | BBH multistep-arithmetic-two: 92.40 → **97.60** (**+5.20**) | — | HumanEvalFix-JS (docs-style repair): 35.98 → **44.51** (**+8.54**)<br>APPS introductory: 15.40 → **17.20** (**+1.80**)<br>CRUXEval output prediction†: 32.03 → **34.38** (**+2.34**) |
58
+ | 9B | BBH word-sorting: 25.60 → **60.40** (**+34.80**)<br>AQuA-RAT: 86.61 → **88.98** (**+2.36**)<br>MATH-500: 91.40 → **92.20** (**+0.80**)<br>GSM8K†: 89.84 → **92.97** (**+3.12**) | OpenBookQA†: 33.59 → **35.16** (**+1.56**) | — |
59
+ | 72B | — | ARC-Easy: 83.33 → **84.64** (**+1.30**)<br>ARC-Challenge: 63.05 → **64.42** (**+1.37**)<br>OpenBookQA: 48.80 → **49.20** (**+0.40**) | LiveCodeBench execution-v2: 58.46 → **60.13** (**+1.67**)<br>BIG-bench CS algorithms: 75.76 → **82.58** (**+6.82**) |
60
+
61
+ † 128-item subset. ARC and OpenBookQA use normalized multiple-choice accuracy; CRUXEval and LiveCodeBench execution-v2 measure output prediction. BBH and BIG-bench entries refer to the named tasks.
62
 
63
  ## Quick start
64