gump2049 commited on
Commit
86dc196
·
verified ·
1 Parent(s): ba923a7

Update original accuracy chart to the 1000-question evaluation

Browse files
Files changed (4) hide show
  1. README.md +3 -3
  2. README.zh-CN.md +3 -3
  3. accuracy-1000-data.json +63 -0
  4. assets/accuracy.svg +1 -1
README.md CHANGED
@@ -42,13 +42,13 @@ We introduce **APUS-OpenJev-v1**, a family of decision models for browser agents
42
 
43
  On our frozen 80-question development panel, **APUS-OpenJev 35B-A3B achieves 88.75% and 9B achieves 85.0% accuracy**, compared with **82.5% for the Jev API**. The optimized 9B decision service reports **25.58 ms median HTTP latency** in the run snapshot below. The 85.0% result uses `9B-3000`; the optimized latency measurement uses `9B-5949`. Accuracy and latency are reported with their evaluation settings below. On this development panel, the full-depth 35B-A3B result is **6.25 percentage points higher** than the recorded Jev API baseline; 9B is **2.50 points higher**.
44
 
45
- ![Decision accuracy on the same 80-question panel: APUS 35B-A3B 88.75%, APUS 9B 85%, APUS 4B and Jev 82.5%, Laya complete-input 68.75%.](assets/accuracy.svg)
46
 
47
- *Figure 1. Same-panel decision accuracy. APUS results use the full compute budget; Laya uses its higher-scoring complete-input configuration after truncation was corrected.*
48
 
49
  ## 2. Evaluation Results
50
 
51
- ### Decision quality
52
 
53
  | Model | Correct / total | Accuracy |
54
  | --- | ---: | ---: |
 
42
 
43
  On our frozen 80-question development panel, **APUS-OpenJev 35B-A3B achieves 88.75% and 9B achieves 85.0% accuracy**, compared with **82.5% for the Jev API**. The optimized 9B decision service reports **25.58 ms median HTTP latency** in the run snapshot below. The 85.0% result uses `9B-3000`; the optimized latency measurement uses `9B-5949`. Accuracy and latency are reported with their evaluation settings below. On this development panel, the full-depth 35B-A3B result is **6.25 percentage points higher** than the recorded Jev API baseline; 9B is **2.50 points higher**.
44
 
45
+ ![Valid and correct out of all 1,000 questions: APUS 35B-A3B 82.20%, 9B 81.10%, 4B 80.50%, Jev API 77.00%, Laya 50.50%.](assets/accuracy.svg)
46
 
47
+ *Figure 1. Valid and correct decisions / all 1,000 questions on the expanded development panel. Historical APUS checkpoint-5949 base + adapter runs at full depth; Laya uses typed full input. Jev has 972 valid results (28 unavailable results remain in the denominator). The published 9B uses checkpoint-3000; this chart is not a 1,000-question validation of the merged releases. [Chart data](accuracy-1000-data.json).*
48
 
49
  ## 2. Evaluation Results
50
 
51
+ ### Decision quality · 80-question release regression
52
 
53
  | Model | Correct / total | Accuracy |
54
  | --- | ---: | ---: |
README.zh-CN.md CHANGED
@@ -44,13 +44,13 @@ tags:
44
 
45
  在冻结的80题开发回归面板上,**APUS-OpenJev 35B-A3B的准确率为88.75%,9B为85.0%**,**Jev API为82.5%**。优化后的9B决策服务在下方运行快照中达到 **25.58 ms HTTP中位延迟**。85.0%对应`9B-3000`;优化延迟对应`9B-5949`。下文分别说明准确率和延迟的评测条件。 在这一开发面板上,35B-A3B完整深度的准确率比记录的Jev API基线**高6.25个百分点**,9B**高2.50个百分点**。
46
 
47
- ![同一80题面板上的决策准确率:APUS 35B-A3B为88.75%,APUS 9B为85%,APUS 4B与Jev为82.5%,Laya完整输入配置为68.75%。](assets/accuracy.svg)
48
 
49
- *图1. 同题决策准确率。APUS结果采用完整计算预算;Laya采用修复截断后的完整输入配置中成绩较高的一项。这是重复使用的开发面板,不是独立的最终基准测试。*
50
 
51
  ## 2. 评测结果
52
 
53
- ### 决策质量
54
 
55
  | 模型 | 正确数 / 总数 | 准确率 |
56
  | --- | ---: | ---: |
 
44
 
45
  在冻结的80题开发回归面板上,**APUS-OpenJev 35B-A3B的准确率为88.75%,9B为85.0%**,**Jev API为82.5%**。优化后的9B决策服务在下方运行快照中达到 **25.58 ms HTTP中位延迟**。85.0%对应`9B-3000`;优化延迟对应`9B-5949`。下文分别说明准确率和延迟的评测条件。 在这一开发面板上,35B-A3B完整深度的准确率比记录的Jev API基线**高6.25个百分点**,9B**高2.50个百分点**。
46
 
47
+ ![全部1000题有效且正确率:APUS 35B-A3B 82.20%、9B 81.10%、4B 80.50%、Jev API 77.00%、Laya 50.50%。](assets/accuracy.svg)
48
 
49
+ *图1. 扩展开发面板上的“有效且正确数 / 全部1000题”。APUS采用历史checkpoint-5949的基座+adapter完整深度推理;Laya采用typed完整输入。Jev获得972条有效结果,28条无效记录仍计入总分母。当前发布的9B为checkpoint-3000;本图不代表merged发布模型的千题复测。[图表数据](accuracy-1000-data.json)。*
50
 
51
  ## 2. 评测结果
52
 
53
+ ### 决策质量 · 发布版80题回归
54
 
55
  | 模型 | 正确数 / 总数 | 准确率 |
56
  | --- | ---: | ---: |
accuracy-1000-data.json ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema": "openjev-five-system-chart-v1",
3
+ "source_summary_sha256": "d0f71b15590e1f56c9a75a847dfd7c402cfaf0ffbd09603953ad726a9aab0305",
4
+ "panel_manifest_sha256": "f75623564147490918a5ac98e3b14dbb7664bedb756d36c7f30d202446677318",
5
+ "questions": 1000,
6
+ "parent_groups": 799,
7
+ "quotas": {
8
+ "browser": 300,
9
+ "hs3": 450,
10
+ "boolq": 75,
11
+ "mnli": 75,
12
+ "score": 100
13
+ },
14
+ "claim_scope": "historical base+adapter checkpoint-5949 evaluation; not current merged release validation; not sealed test; no timing claim",
15
+ "metric": "valid and correct / all 1000 questions",
16
+ "models": [
17
+ {
18
+ "name": "APUS-OpenJev 35B-A3B",
19
+ "source_name": "35B-A3B final-5949 / 40L",
20
+ "correct": 822,
21
+ "total": 1000,
22
+ "valid": 1000,
23
+ "accuracy": 0.822,
24
+ "source_predictions_sha256": "3ec05570b05c96617fcea13f95415ebea5296a28f3e187a04a137a88ff6f05f1"
25
+ },
26
+ {
27
+ "name": "APUS-OpenJev 9B",
28
+ "source_name": "Jet1 9B final-5949 / 32L",
29
+ "correct": 811,
30
+ "total": 1000,
31
+ "valid": 1000,
32
+ "accuracy": 0.811,
33
+ "source_predictions_sha256": "62804823e9148cb76d0f6ebc98733f69a46ea22a0de368a7c9e03f2dbafbcd65"
34
+ },
35
+ {
36
+ "name": "APUS-OpenJev 4B",
37
+ "source_name": "Jet1 4B final-5949 / 32L",
38
+ "correct": 805,
39
+ "total": 1000,
40
+ "valid": 1000,
41
+ "accuracy": 0.805,
42
+ "source_predictions_sha256": "8f98034b552bb713513b181b6209c75e52be345e46cee3da3867aa69abf6e3e3"
43
+ },
44
+ {
45
+ "name": "Jev API",
46
+ "source_name": "Official Jev 1.13.0",
47
+ "correct": 770,
48
+ "total": 1000,
49
+ "valid": 972,
50
+ "accuracy": 0.77,
51
+ "source_predictions_sha256": "d255b7e73357027e612b25bee2863f401867c04cd667ac1947e42c10cc0e66aa"
52
+ },
53
+ {
54
+ "name": "Laya · typed full input",
55
+ "source_name": "Laya Typed-decisions full-input / 1c5edc17",
56
+ "correct": 505,
57
+ "total": 1000,
58
+ "valid": 1000,
59
+ "accuracy": 0.505,
60
+ "source_predictions_sha256": "da922effc80428e0a8d640bf62238cc6682c9c2ec12824009f4d2e059d703578"
61
+ }
62
+ ]
63
+ }
assets/accuracy.svg CHANGED