docs: document held-out 1000-question benchmark design

#7
by gump2049 - opened
Files changed (2) hide show
  1. README.md +1 -1
  2. README.zh-CN.md +1 -1
README.md CHANGED
@@ -40,7 +40,7 @@ We introduce **APUS-OpenJev-v1**, a family of decision models for browser agents
40
 
41
  **Decision-oriented inference.** The runtime scores candidate labels and maps them back to application actions. This removes the need to generate a structured answer token by token for bounded-choice tasks. Choose `effort="low"` or `effort="high"` to balance compute cost and decision quality.
42
 
43
- On our frozen 80-question development panel, **APUS-OpenJev 35B-A3B achieves 88.75% and 9B achieves 85.0% accuracy**, compared with **82.5% for the Jev API**. The optimized 9B decision service reports **25.58 ms median HTTP latency** in the run snapshot below. The 85.0% result uses `9B-3000`; the optimized latency measurement uses `9B-5949`. Accuracy and latency are reported with their evaluation settings below. On this development panel, the full-depth 35B-A3B result is **6.25 percentage points higher** than the recorded Jev API baseline; 9B is **2.50 points higher**.
44
 
45
  ![Valid and correct out of all 1,000 questions: APUS 35B-A3B 82.20%, 9B 81.10%, 4B 80.50%, Jev API 77.00%, Laya 50.50%.](assets/accuracy.svg)
46
 
 
40
 
41
  **Decision-oriented inference.** The runtime scores candidate labels and maps them back to application actions. This removes the need to generate a structured answer token by token for bounded-choice tasks. Choose `effort="low"` or `effort="high"` to balance compute cost and decision quality.
42
 
43
+ On the held-out **1,000-question expanded development panel**, **APUS-OpenJev 35B-A3B achieves 82.20%, 9B 81.10%, and 4B 80.50% accuracy**, compared with **77.00% for the Jev API**. The panel is stratified across browser action selection, principle judgment, BoolQ, MNLI, and score-level decisions, with frozen identities and reference labels kept separate from training data. All scores use the same 1,000-question denominator; invalid or unavailable responses remain in that denominator. The optimized 9B decision service reports **25.58 ms median HTTP latency** in the run snapshot below. The chart uses historical checkpoint-5949 full-depth evaluation, while the published 9B quality result uses `9B-3000`; latency uses `9B-5949`. The 80-question Frozen80 panel remains the release regression suite and is reported separately below.
44
 
45
  ![Valid and correct out of all 1,000 questions: APUS 35B-A3B 82.20%, 9B 81.10%, 4B 80.50%, Jev API 77.00%, Laya 50.50%.](assets/accuracy.svg)
46
 
README.zh-CN.md CHANGED
@@ -42,7 +42,7 @@ tags:
42
 
43
  **面向决策的推理。** runtime对候选标签评分,再映射回应用动作。对于有界选择任务,无需逐token生成结构化答案。通过选择`effort="low"`或`effort="high"`,在计算成本与决策质量之间取得平衡。
44
 
45
- 在冻结的80题开发回归面板上,**APUS-OpenJev 35B-A3B的准确率为88.75%,9B为85.0%**,**Jev API为82.5%**。优化后的9B决策服务在下方运行快照中达到 **25.58 ms HTTP中位延迟**。85.0%对应`9B-3000`;优化延迟对应`9B-5949`。下文分别说明准确率和延迟的评测条件。 在这一开发面板上,35B-A3B完整深度的准确率比记录的Jev API基线**高6.25个百分点**,9B**高2.50个百分点**。
46
 
47
  ![全部1000题有效且正确率:APUS 35B-A3B 82.20%、9B 81.10%、4B 80.50%、Jev API 77.00%、Laya 50.50%。](assets/accuracy.svg)
48
 
 
42
 
43
  **面向决策的推理。** runtime对候选标签评分,再映射回应用动作。对于有界选择任务,无需逐token生成结构化答案。通过选择`effort="low"`或`effort="high"`,在计算成本与决策质量之间取得平衡。
44
 
45
+ 在留出的**1000题扩展开发面板**上,**APUS-OpenJev 35B-A3B准确率为82.20%、9B为81.10%、4B为80.50%**,**Jev API为77.00%**。该面板围绕浏览器动作选择、原则判断、BoolQ、MNLI和情绪分级决策进行分层,题目身份与参考标签冻结,并与训练数据隔离。所有结果使用相同的1000题分母;无效或不可用响应仍计入分母。优化后的9B决策服务在下方运行快照中达到**25.58 ms HTTP中位延迟**。图表采用历史checkpoint-5949完整深度评测;当前发布的9B质量结果对应`9B-3000`,延迟测量对应`9B-5949`。Frozen80 80题面板继续作为发布回归套件,单独在下方报告。
46
 
47
  ![全部1000题有效且正确率:APUS 35B-A3B 82.20%、9B 81.10%、4B 80.50%、Jev API 77.00%、Laya 50.50%。](assets/accuracy.svg)
48