HololabTHU commited on
Commit
e9cabe2
·
verified ·
1 Parent(s): 2701c28

OPB scores under the array-input protocol

Browse files
Files changed (3) hide show
  1. .gitattributes +1 -0
  2. README.md +5 -5
  3. assets/bench_bars.png +2 -2
.gitattributes CHANGED
@@ -49,3 +49,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
49
  "/tokenizer.json" filter=lfs diff=lfs merge=lfs -text
50
  "/vocab.json" filter=lfs diff=lfs merge=lfs -textassets/logo_hm.png filter=lfs diff=lfs merge=lfs -text
51
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
 
49
  "/tokenizer.json" filter=lfs diff=lfs merge=lfs -text
50
  "/vocab.json" filter=lfs diff=lfs merge=lfs -textassets/logo_hm.png filter=lfs diff=lfs merge=lfs -text
51
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
52
+ assets/bench_bars.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -35,9 +35,9 @@ Frontier AI scientists depend on proprietary models that are costly, closed and
35
 
36
  ### Key Features
37
 
38
- - 🔬 **Built for optical physics.** On **Optics-Phy-Bench**, 217 multi-step computations reproduced from 217 published papers, HoloModel-Research solves **52.1 %** of problems against **34.1 %** for its base model: **+18.0 points, a 53 % relative gain** (95 % CI +12.0 to +24.0).
39
  - 💻 **Stronger coder, not only in optics.** VCP code dev **+8.8** (P = 0.0015) and optics coding proxy **+5.4** over the base model.
40
- - 🥊 **Punches above its size.** With thinking off, HoloModel-Research beats **DeepSeek V4.1 Flash** (552B total parameters) by **11.3 points** on VCP code dev and is statistically tied with it on Optics-Phy-Bench; it leads **GLM-5.3** and **Kimi-K3.1-Pro** on all five optics and coding sets.
41
  - 🔒 **Private and reproducible.** Fixed weights, temperature-0 evaluation and fully local deployment: data and unpublished designs never leave the lab.
42
 
43
  ## 2. Model Summary
@@ -64,8 +64,8 @@ Frontier AI scientists depend on proprietary models that are costly, closed and
64
 
65
  | Benchmark | Items | Qwen3.8-27B | Kimi-K3.1-Pro | GLM-5.3<br/><sub>reasoning low</sub> | DeepSeek V4.1 Flash | **HoloModel-Research** |
66
  |:---|---:|---:|---:|---:|---:|---:|
67
- | Optics-Phy-Bench, problems | 217 | 34.1 | 38.7 | 37.3 | **58.1** | <ins>52.1</ins> |
68
- | Optics-Phy-Bench, steps | 833 | 74.8 | 78.3 | 77.1 | **86.6** | <ins>84.3</ins> |
69
  | VCP code dev | 194 | 77.8 | 42.3 | 72.2 | 75.3 | **86.6** |
70
  | Optics coding proxy | 300 | 81.3 | 71.0 | 69.7 | 83.7 | **86.7** |
71
  | Physics dev | 198 | 85.4 | 72.7 | 65.7 | 84.3 | **87.9** |
@@ -77,7 +77,7 @@ Frontier AI scientists depend on proprietary models that are costly, closed and
77
 
78
  - All models: thinking disabled (GLM-5.3 at its lowest reasoning setting), temperature 0 (Kimi-K3.1-Pro: 0.6, the lowest its API accepts with thinking off), 14,000-token limit, one run per benchmark.
79
  - HoloModel-Research and the base model are compared item by item; confidence intervals are paired bootstrap 95 % intervals (10,000 resamples) and P values exact McNemar tests.
80
- - Optics-Phy-Bench: a problem counts as solved only when every step function reproduces the reference outputs within tolerance.
81
  - **Decontamination:** no training prompt shares a 13-word sequence with Optics-Phy-Bench, physics dev or the coding proxy. VCP code dev is a split of the public `open-r1/verifiable-coding-problems-python` dataset; six items with more than half of their text inside one training prompt were removed, leaving 194.
82
  - The checkpoint (iteration 30) was chosen by a rule fixed before training, not by any benchmark.
83
  - Frontier models with reasoning enabled still score higher on Optics-Phy-Bench.
 
35
 
36
  ### Key Features
37
 
38
+ - 🔬 **Built for optical physics.** On **Optics-Phy-Bench**, 217 multi-step computations reproduced from 217 published papers, HoloModel-Research solves **69.1 %** of problems against **46.5 %** for its base model: **+22.6 points, a 49 % relative gain** (95 % CI +15.7 to +29.5).
39
  - 💻 **Stronger coder, not only in optics.** VCP code dev **+8.8** (P = 0.0015) and optics coding proxy **+5.4** over the base model.
40
+ - 🥊 **Punches above its size.** With thinking off, HoloModel-Research is ahead of **DeepSeek V4.1 Flash** (552B total parameters) on Optics-Phy-Bench (**69.1 %** against 62.7 % of problems; +2.9 points on steps, P = 0.025) and by **11.3 points** on VCP code dev; it leads **GLM-5.3** and **Kimi-K3.1-Pro** on all five optics and coding sets.
41
  - 🔒 **Private and reproducible.** Fixed weights, temperature-0 evaluation and fully local deployment: data and unpublished designs never leave the lab.
42
 
43
  ## 2. Model Summary
 
64
 
65
  | Benchmark | Items | Qwen3.8-27B | Kimi-K3.1-Pro | GLM-5.3<br/><sub>reasoning low</sub> | DeepSeek V4.1 Flash | **HoloModel-Research** |
66
  |:---|---:|---:|---:|---:|---:|---:|
67
+ | Optics-Phy-Bench, problems | 217 | 46.5 | 41.9 | 39.2 | 62.7 | **69.1** |
68
+ | Optics-Phy-Bench, steps | 833 | 82.0 | 79.5 | 77.8 | 88.1 | **91.0** |
69
  | VCP code dev | 194 | 77.8 | 42.3 | 72.2 | 75.3 | **86.6** |
70
  | Optics coding proxy | 300 | 81.3 | 71.0 | 69.7 | 83.7 | **86.7** |
71
  | Physics dev | 198 | 85.4 | 72.7 | 65.7 | 84.3 | **87.9** |
 
77
 
78
  - All models: thinking disabled (GLM-5.3 at its lowest reasoning setting), temperature 0 (Kimi-K3.1-Pro: 0.6, the lowest its API accepts with thinking off), 14,000-token limit, one run per benchmark.
79
  - HoloModel-Research and the base model are compared item by item; confidence intervals are paired bootstrap 95 % intervals (10,000 resamples) and P values exact McNemar tests.
80
+ - Optics-Phy-Bench: test inputs are passed to each step function as numpy arrays; a problem counts as solved only when every step function reproduces the reference outputs within tolerance.
81
  - **Decontamination:** no training prompt shares a 13-word sequence with Optics-Phy-Bench, physics dev or the coding proxy. VCP code dev is a split of the public `open-r1/verifiable-coding-problems-python` dataset; six items with more than half of their text inside one training prompt were removed, leaving 194.
82
  - The checkpoint (iteration 30) was chosen by a rule fixed before training, not by any benchmark.
83
  - Frontier models with reasoning enabled still score higher on Optics-Phy-Bench.
assets/bench_bars.png CHANGED

Git LFS Details

  • SHA256: e2f2529dafd801b87fb9d34506cfa4f1d183f4b4628fbe115d7090ec6bc54659
  • Pointer size: 131 Bytes
  • Size of remote file: 114 kB

Git LFS Details

  • SHA256: 9e24f2bbac31922e365beb98ff802f310a40c4e75fdc296c89afaefaacb7124d
  • Pointer size: 131 Bytes
  • Size of remote file: 115 kB