Instructions to use apus-ailab/APUS-Reflex with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use apus-ailab/APUS-Reflex with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("apus-ailab/APUS-Reflex", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update original accuracy chart to the 1000-question evaluation
Browse files- README.md +3 -3
- README.zh-CN.md +3 -3
- accuracy-1000-data.json +63 -0
- assets/accuracy.svg +1 -1
README.md
CHANGED
|
@@ -42,13 +42,13 @@ We introduce **APUS-OpenJev-v1**, a family of decision models for browser agents
|
|
| 42 |
|
| 43 |
On our frozen 80-question development panel, **APUS-OpenJev 35B-A3B achieves 88.75% and 9B achieves 85.0% accuracy**, compared with **82.5% for the Jev API**. The optimized 9B decision service reports **25.58 ms median HTTP latency** in the run snapshot below. The 85.0% result uses `9B-3000`; the optimized latency measurement uses `9B-5949`. Accuracy and latency are reported with their evaluation settings below. On this development panel, the full-depth 35B-A3B result is **6.25 percentage points higher** than the recorded Jev API baseline; 9B is **2.50 points higher**.
|
| 44 |
|
| 45 |
-

|
| 46 |
|
| 47 |
+
*Figure 1. Valid and correct decisions / all 1,000 questions on the expanded development panel. Historical APUS checkpoint-5949 base + adapter runs at full depth; Laya uses typed full input. Jev has 972 valid results (28 unavailable results remain in the denominator). The published 9B uses checkpoint-3000; this chart is not a 1,000-question validation of the merged releases. [Chart data](accuracy-1000-data.json).*
|
| 48 |
|
| 49 |
## 2. Evaluation Results
|
| 50 |
|
| 51 |
+
### Decision quality · 80-question release regression
|
| 52 |
|
| 53 |
| Model | Correct / total | Accuracy |
|
| 54 |
| --- | ---: | ---: |
|
README.zh-CN.md
CHANGED
|
@@ -44,13 +44,13 @@ tags:
|
|
| 44 |
|
| 45 |
在冻结的80题开发回归面板上,**APUS-OpenJev 35B-A3B的准确率为88.75%,9B为85.0%**,**Jev API为82.5%**。优化后的9B决策服务在下方运行快照中达到 **25.58 ms HTTP中位延迟**。85.0%对应`9B-3000`;优化延迟对应`9B-5949`。下文分别说明准确率和延迟的评测条件。 在这一开发面板上,35B-A3B完整深度的准确率比记录的Jev API基线**高6.25个百分点**,9B**高2.50个百分点**。
|
| 46 |
|
| 47 |
-

|
| 48 |
|
| 49 |
+
*图1. 扩展开发面板上的“有效且正确数 / 全部1000题”。APUS采用历史checkpoint-5949的基座+adapter完整深度推理;Laya采用typed完整输入。Jev获得972条有效结果,28条无效记录仍计入总分母。当前发布的9B为checkpoint-3000;本图不代表merged发布模型的千题复测。[图表数据](accuracy-1000-data.json)。*
|
| 50 |
|
| 51 |
## 2. 评测结果
|
| 52 |
|
| 53 |
+
### 决策质量 · 发布版80题回归
|
| 54 |
|
| 55 |
| 模型 | 正确数 / 总数 | 准确率 |
|
| 56 |
| --- | ---: | ---: |
|
accuracy-1000-data.json
ADDED
|
@@ -0,0 +1,63 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"schema": "openjev-five-system-chart-v1",
|
| 3 |
+
"source_summary_sha256": "d0f71b15590e1f56c9a75a847dfd7c402cfaf0ffbd09603953ad726a9aab0305",
|
| 4 |
+
"panel_manifest_sha256": "f75623564147490918a5ac98e3b14dbb7664bedb756d36c7f30d202446677318",
|
| 5 |
+
"questions": 1000,
|
| 6 |
+
"parent_groups": 799,
|
| 7 |
+
"quotas": {
|
| 8 |
+
"browser": 300,
|
| 9 |
+
"hs3": 450,
|
| 10 |
+
"boolq": 75,
|
| 11 |
+
"mnli": 75,
|
| 12 |
+
"score": 100
|
| 13 |
+
},
|
| 14 |
+
"claim_scope": "historical base+adapter checkpoint-5949 evaluation; not current merged release validation; not sealed test; no timing claim",
|
| 15 |
+
"metric": "valid and correct / all 1000 questions",
|
| 16 |
+
"models": [
|
| 17 |
+
{
|
| 18 |
+
"name": "APUS-OpenJev 35B-A3B",
|
| 19 |
+
"source_name": "35B-A3B final-5949 / 40L",
|
| 20 |
+
"correct": 822,
|
| 21 |
+
"total": 1000,
|
| 22 |
+
"valid": 1000,
|
| 23 |
+
"accuracy": 0.822,
|
| 24 |
+
"source_predictions_sha256": "3ec05570b05c96617fcea13f95415ebea5296a28f3e187a04a137a88ff6f05f1"
|
| 25 |
+
},
|
| 26 |
+
{
|
| 27 |
+
"name": "APUS-OpenJev 9B",
|
| 28 |
+
"source_name": "Jet1 9B final-5949 / 32L",
|
| 29 |
+
"correct": 811,
|
| 30 |
+
"total": 1000,
|
| 31 |
+
"valid": 1000,
|
| 32 |
+
"accuracy": 0.811,
|
| 33 |
+
"source_predictions_sha256": "62804823e9148cb76d0f6ebc98733f69a46ea22a0de368a7c9e03f2dbafbcd65"
|
| 34 |
+
},
|
| 35 |
+
{
|
| 36 |
+
"name": "APUS-OpenJev 4B",
|
| 37 |
+
"source_name": "Jet1 4B final-5949 / 32L",
|
| 38 |
+
"correct": 805,
|
| 39 |
+
"total": 1000,
|
| 40 |
+
"valid": 1000,
|
| 41 |
+
"accuracy": 0.805,
|
| 42 |
+
"source_predictions_sha256": "8f98034b552bb713513b181b6209c75e52be345e46cee3da3867aa69abf6e3e3"
|
| 43 |
+
},
|
| 44 |
+
{
|
| 45 |
+
"name": "Jev API",
|
| 46 |
+
"source_name": "Official Jev 1.13.0",
|
| 47 |
+
"correct": 770,
|
| 48 |
+
"total": 1000,
|
| 49 |
+
"valid": 972,
|
| 50 |
+
"accuracy": 0.77,
|
| 51 |
+
"source_predictions_sha256": "d255b7e73357027e612b25bee2863f401867c04cd667ac1947e42c10cc0e66aa"
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
"name": "Laya · typed full input",
|
| 55 |
+
"source_name": "Laya Typed-decisions full-input / 1c5edc17",
|
| 56 |
+
"correct": 505,
|
| 57 |
+
"total": 1000,
|
| 58 |
+
"valid": 1000,
|
| 59 |
+
"accuracy": 0.505,
|
| 60 |
+
"source_predictions_sha256": "da922effc80428e0a8d640bf62238cc6682c9c2ec12824009f4d2e059d703578"
|
| 61 |
+
}
|
| 62 |
+
]
|
| 63 |
+
}
|
assets/accuracy.svg
CHANGED
|
|
|
|