Upload folder using huggingface_hub
Browse files- MANIFEST.json +4 -4
- README.md +29 -2
- results/agentic.json +1310 -0
MANIFEST.json
CHANGED
|
@@ -18,8 +18,8 @@
|
|
| 18 |
},
|
| 19 |
{
|
| 20 |
"path": "README.md",
|
| 21 |
-
"bytes":
|
| 22 |
-
"sha256": "
|
| 23 |
},
|
| 24 |
{
|
| 25 |
"path": "chat_template.jinja",
|
|
@@ -68,8 +68,8 @@
|
|
| 68 |
},
|
| 69 |
{
|
| 70 |
"path": "results/agentic.json",
|
| 71 |
-
"bytes":
|
| 72 |
-
"sha256": "
|
| 73 |
},
|
| 74 |
{
|
| 75 |
"path": "results/benchmarks.json",
|
|
|
|
| 18 |
},
|
| 19 |
{
|
| 20 |
"path": "README.md",
|
| 21 |
+
"bytes": 11417,
|
| 22 |
+
"sha256": "d5d8c34c37dac0f27f5f33ef13d965868b50186f65002b2e98ccf776e6c30359"
|
| 23 |
},
|
| 24 |
{
|
| 25 |
"path": "chat_template.jinja",
|
|
|
|
| 68 |
},
|
| 69 |
{
|
| 70 |
"path": "results/agentic.json",
|
| 71 |
+
"bytes": 74744,
|
| 72 |
+
"sha256": "13f4bc09d1f17f693cec480b64d14bcc13b70b72a4696e93a7860c0a90c7cbcf"
|
| 73 |
},
|
| 74 |
{
|
| 75 |
"path": "results/benchmarks.json",
|
README.md
CHANGED
|
@@ -57,7 +57,32 @@ The host supplies task observations and real feedback through the [task interfac
|
|
| 57 |
|
| 58 |
Accuracy percentages on the complete 2,376 / 1,172 / 10,042 / 1,267 / 14,042 example panels. Evaluation uses zero-shot completion likelihood, empty M, fixed Φ, no chat template, and FP32 execution of the saved weights. ARC and HellaSwag use character-normalized likelihood; WinoGrande uses common-suffix likelihood; MMLU is micro accuracy over 57 subjects. These previously evaluated panels are regression comparisons. Qwen and SmolLM2 have more parameters. Exact counts, protocol and model identities are in [benchmark results](results/benchmarks.json).
|
| 59 |
|
| 60 |
-
## Agentic
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 61 |
|
| 62 |
**Interface audit:** these scores expose substantial command-format failures in this custom host and should not be used as a general Agentic ranking or evidence of an MMLA architecture advantage. SmolLM3 executed no BFCL tool calls: 453 of its 734 replies meet the host schema after removing only Markdown fences, which the host rejected. SmolLM2 reached the 64-token output limit in 452 of 734 BFCL replies. In ALFWorld, both Smol models received `Nothing happens.` for every action; MiniCPM5-1B-SFT often copied literal placeholders such as `(receptacle)` into commands. These are observed failure modes, not estimates of corrected scores. All recorded scores are retained below.
|
| 63 |
|
|
@@ -77,6 +102,8 @@ Orbit was evaluated on **424 tasks across three environments**, alongside eight
|
|
| 77 |
|
| 78 |
ALFWorld uses the 134-task **seen regression** panel. ScienceWorld averages final scores across 30 task classes with three official test variations each; native scores range from −100 to 100, including negative outcomes. Orbit completed the task objective in 0/90 ScienceWorld episodes. BFCL covers the 200-case **multi_turn_base** category. These results describe this finite host configuration; the panels have different scoring rules. SmolLM2 has five unscored ScienceWorld context-limit failures, so its 90-task mean is left blank. Exact values, model revisions and protocol details are in [Agentic results](results/agentic.json).
|
| 79 |
|
|
|
|
|
|
|
| 80 |
## RSI proposal and installation status
|
| 81 |
|
| 82 |
The same Orbit weights were used with three proposal RNG seeds (17, 29, 43), each producing 40 proposals. The best eligible candidate in each development window tied or underperformed its parent. No candidate was locked for independent admission, and no successor was accepted or installed.
|
|
@@ -88,7 +115,7 @@ The same Orbit weights were used with three proposal RNG seeds (17, 29, 43), eac
|
|
| 88 |
| 20 | 60 | 0 | 0 |
|
| 89 |
| 40 | 120 | 0 | 0 |
|
| 90 |
|
| 91 |
-
All twelve checkpoints retain the same installed manager.
|
| 92 |
|
| 93 |
M reads and RTMA writes executed during task evaluation. Numerical Φ stayed fixed at its trained initialization. This series demonstrates proposal execution and memory use, with no accepted recursive improvement. Checkpoint identities and the interface audit are included in [Agentic results](results/agentic.json).
|
| 94 |
|
|
|
|
| 57 |
|
| 58 |
Accuracy percentages on the complete 2,376 / 1,172 / 10,042 / 1,267 / 14,042 example panels. Evaluation uses zero-shot completion likelihood, empty M, fixed Φ, no chat template, and FP32 execution of the saved weights. ARC and HellaSwag use character-normalized likelihood; WinoGrande uses common-suffix likelihood; MMLU is micro accuracy over 57 subjects. These previously evaluated panels are regression comparisons. Qwen and SmolLM2 have more parameters. Exact counts, protocol and model identities are in [benchmark results](results/benchmarks.json).
|
| 59 |
|
| 60 |
+
## Agentic evaluation: revised interface
|
| 61 |
+
|
| 62 |
+
Nine fixed checkpoints entered a finite reevaluation through the shared text-action host after correcting command instructions and output-format compatibility. Each model first receives two synthetic interface checks per environment. A check requires a legal first action and a second action using newly returned feedback. Both checks must pass before that environment is evaluated. Each fixture requires two specified single-call steps within four replies. BFCL qualification rejects a multi-call reply as a whole, even when its first call is valid; unlike the benchmark executor, it does not partially execute that reply. Failure concerns this strict synthetic check and does not mean that every produced call was illegal.
|
| 63 |
+
|
| 64 |
+
| Model | ALF interface checks | ALFWorld success | BFCL interface checks | BFCL success |
|
| 65 |
+
|---|---:|---:|---:|---:|
|
| 66 |
+
| Orbit-1B (M + RTMA) | 2/2 | 26.12% (35/134) | 0/2 | — (not qualified; 200 unscored) |
|
| 67 |
+
| MiniCPM5-1B-SFT | 2/2 | 0.00% (0/134) | 0/2 | — (not qualified; 200 unscored) |
|
| 68 |
+
| MiniCPM5-1B | 1/2 | — (not qualified; 134 unscored) | 2/2 | 0.00% (0/200) |
|
| 69 |
+
| Qwen2.5-1.5B-Instruct | 0/2 | — (not qualified; 134 unscored) | 0/2 | — (not qualified; 200 unscored) |
|
| 70 |
+
| SmolLM2-1.7B-Instruct | 0/2 | — (not qualified; 134 unscored) | 0/2 | — (not qualified; 200 unscored) |
|
| 71 |
+
| LFM2.5-1.2B-Instruct | 0/2 | — (not qualified; 134 unscored) | 0/2 | — (not qualified; 200 unscored) |
|
| 72 |
+
| SmolLM3-3B | 0/2 | — (not qualified; 134 unscored) | 0/2 | — (not qualified; 200 unscored) |
|
| 73 |
+
| Qwen3.5-4B | 2/2 | 3.73% (5/134) | 0/2 | — (not qualified; 200 unscored) |
|
| 74 |
+
| Qwen3.5-9B | 2/2 | 24.63% (33/134) | 2/2 | 38.00% (76/200) |
|
| 75 |
+
|
| 76 |
+
The finite allocation scored 936 of 3,006 planned model–task positions; 2,070 remain unscored. A dash is not a zero. Primary scores require the complete 134-task ALFWorld or 200-task BFCL panel; qualification failures and unfinished runs retain their original denominators.
|
| 77 |
+
|
| 78 |
+
ALFWorld uses the same seen-regression panel. BFCL uses the same multi_turn_base200 cases, evaluated again after the interface revision. Both retain a 64-reply task limit; reply budgets are 64 tokens for ALFWorld and 512 for BFCL. Orbit uses fixed trained Φ with its eight-slot M and installed RTMA rule. Ordinary models use their native chat templates and conversation history. Scores describe this finite text-action configuration, not unrestricted official leaderboards or isolated causal memory gains. Local external inference also changed from vLLM 0.26.0 to 0.29.0, so old/new differences do not isolate one fix.
|
| 79 |
+
|
| 80 |
+
ScienceWorld was not rerun; its original-protocol results and all earlier scores are retained below and in [Agentic results](results/agentic.json). That file also records qualification status, exact coverage, model revisions, performance measurements and evidence hashes.
|
| 81 |
+
|
| 82 |
+
<details>
|
| 83 |
+
<summary>Original protocol scores and interface diagnosis</summary>
|
| 84 |
+
|
| 85 |
+
### Original protocol diagnostics
|
| 86 |
|
| 87 |
**Interface audit:** these scores expose substantial command-format failures in this custom host and should not be used as a general Agentic ranking or evidence of an MMLA architecture advantage. SmolLM3 executed no BFCL tool calls: 453 of its 734 replies meet the host schema after removing only Markdown fences, which the host rejected. SmolLM2 reached the 64-token output limit in 452 of 734 BFCL replies. In ALFWorld, both Smol models received `Nothing happens.` for every action; MiniCPM5-1B-SFT often copied literal placeholders such as `(receptacle)` into commands. These are observed failure modes, not estimates of corrected scores. All recorded scores are retained below.
|
| 88 |
|
|
|
|
| 102 |
|
| 103 |
ALFWorld uses the 134-task **seen regression** panel. ScienceWorld averages final scores across 30 task classes with three official test variations each; native scores range from −100 to 100, including negative outcomes. Orbit completed the task objective in 0/90 ScienceWorld episodes. BFCL covers the 200-case **multi_turn_base** category. These results describe this finite host configuration; the panels have different scoring rules. SmolLM2 has five unscored ScienceWorld context-limit failures, so its 90-task mean is left blank. Exact values, model revisions and protocol details are in [Agentic results](results/agentic.json).
|
| 104 |
|
| 105 |
+
</details>
|
| 106 |
+
|
| 107 |
## RSI proposal and installation status
|
| 108 |
|
| 109 |
The same Orbit weights were used with three proposal RNG seeds (17, 29, 43), each producing 40 proposals. The best eligible candidate in each development window tied or underperformed its parent. No candidate was locked for independent admission, and no successor was accepted or installed.
|
|
|
|
| 115 |
| 20 | 60 | 0 | 0 |
|
| 116 |
| 40 | 120 | 0 | 0 |
|
| 117 |
|
| 118 |
+
All twelve checkpoints retain the same installed manager. The evaluations above use that same installed manager; repeating their scores at each proposal checkpoint would not measure a performance trajectory across improved generations. No new RSI search was run on ScienceWorld or BFCL.
|
| 119 |
|
| 120 |
M reads and RTMA writes executed during task evaluation. Numerical Φ stayed fixed at its trained initialization. This series demonstrates proposal execution and memory use, with no accepted recursive improvement. Checkpoint identities and the interface audit are included in [Agentic results](results/agentic.json).
|
| 121 |
|
results/agentic.json
CHANGED
|
@@ -813,5 +813,1315 @@
|
|
| 813 |
"ALFWorld panel was previously used for regression; Orbit had task-specific training. These scores do not isolate M/RTMA causal benefit.",
|
| 814 |
"Changing the parser changes later stateful observations; fixed trajectories cannot be reinterpreted as corrected task scores."
|
| 815 |
]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 816 |
}
|
| 817 |
}
|
|
|
|
| 813 |
"ALFWorld panel was previously used for regression; Orbit had task-specific training. These scores do not isolate M/RTMA causal benefit.",
|
| 814 |
"Changing the parser changes later stateful observations; fixed trajectories cannot be reinterpreted as corrected task scores."
|
| 815 |
]
|
| 816 |
+
},
|
| 817 |
+
"interface_revision": {
|
| 818 |
+
"schema": "orbit-public-agentic-interface-v2",
|
| 819 |
+
"allocation_status": "COMPLETE",
|
| 820 |
+
"model_weight_sha256": "62d83295cb154ce7869c60602e66677b84cde6f33ab886101752caa6805af276",
|
| 821 |
+
"installed_manager_sha256": "2b44013ee892552f14b846d2a1315e09990f4751d4255a9ff48295cc75fc0081",
|
| 822 |
+
"protocol": {
|
| 823 |
+
"host": "OhMyPi-based shared text-action host",
|
| 824 |
+
"interface_version": 2,
|
| 825 |
+
"model_seed": 17,
|
| 826 |
+
"max_model_replies_per_task": 64,
|
| 827 |
+
"max_generated_tokens_per_reply": {
|
| 828 |
+
"alfworld": 64,
|
| 829 |
+
"bfcl": 512
|
| 830 |
+
},
|
| 831 |
+
"qualification": "Two fixed synthetic fixtures per domain, each requiring two specified single-call steps within four replies; both fixtures must pass. BFCL qualification rejects an entire multi-call reply without partial execution, including when its first call is valid; this differs from the real benchmark executor. No test answers; qualification state discarded. Failure is not evidence of inability to generate any legal call.",
|
| 832 |
+
"bfcl": "Official classic/plaintext/Python prompting and executor/checker. Exact complete JSON, Markdown fence and tool_call wrappers accepted without guessing parameters or repairing incomplete output.",
|
| 833 |
+
"alfworld": "Explicit command grammar, placeholder replacement and action preconditions; no oracle valid-action list.",
|
| 834 |
+
"orbit": "Same trained fixed Phi and eight-slot M/RTMA manager; resets between tasks; no new training or RSI proposals.",
|
| 835 |
+
"external_models": "Exact original weights and native chat templates, ordinary conversation history, greedy decoding, thinking disabled; local vLLM0.29.0 instead of original0.26.0.",
|
| 836 |
+
"qualification_failure": "Corresponding fixed benchmark positions remain unscored; no prompt retry or zero imputation.",
|
| 837 |
+
"scienceworld": "Original protocol results retained separately; not rerun in revision2."
|
| 838 |
+
},
|
| 839 |
+
"coverage": {
|
| 840 |
+
"expected_model_task_positions": 3006,
|
| 841 |
+
"scored": 936,
|
| 842 |
+
"unscored": 2070,
|
| 843 |
+
"qualification_expected": 36,
|
| 844 |
+
"qualification_completed": 36
|
| 845 |
+
},
|
| 846 |
+
"models": {
|
| 847 |
+
"Orbit": {
|
| 848 |
+
"label": "Orbit-1B (M + RTMA)",
|
| 849 |
+
"panels": {
|
| 850 |
+
"alfworld": {
|
| 851 |
+
"status": "COMPLETE",
|
| 852 |
+
"expected": 134,
|
| 853 |
+
"completed": 134,
|
| 854 |
+
"unscored": 0,
|
| 855 |
+
"primary": 0.26119402985074625,
|
| 856 |
+
"success_count": 35,
|
| 857 |
+
"status_counts": {
|
| 858 |
+
"COMPLETE": 134
|
| 859 |
+
},
|
| 860 |
+
"returned_work": {
|
| 861 |
+
"replies": 6570,
|
| 862 |
+
"input_tokens": 3412117,
|
| 863 |
+
"output_tokens": 45867,
|
| 864 |
+
"length_finishes": 0,
|
| 865 |
+
"nothing_happens": 5927
|
| 866 |
+
},
|
| 867 |
+
"qualification": {
|
| 868 |
+
"status": "QUALIFIED",
|
| 869 |
+
"passed": 2,
|
| 870 |
+
"expected": 2,
|
| 871 |
+
"protocol": "e53aaa1a7a78a80a65e49f5abd2290ef520c05e14ed17fdccbfb2039010cd0d7"
|
| 872 |
+
}
|
| 873 |
+
},
|
| 874 |
+
"bfcl": {
|
| 875 |
+
"status": "NOT_QUALIFIED",
|
| 876 |
+
"expected": 200,
|
| 877 |
+
"completed": 0,
|
| 878 |
+
"unscored": 200,
|
| 879 |
+
"primary": null,
|
| 880 |
+
"success_count": 0,
|
| 881 |
+
"status_counts": {
|
| 882 |
+
"NOT_QUALIFIED": 200
|
| 883 |
+
},
|
| 884 |
+
"returned_work": {},
|
| 885 |
+
"qualification": {
|
| 886 |
+
"status": "NOT_QUALIFIED",
|
| 887 |
+
"passed": 0,
|
| 888 |
+
"expected": 2,
|
| 889 |
+
"protocol": "67df3a095a6a00813f56890bbf75a2f5a01368585dee310e1714344d140aa631"
|
| 890 |
+
}
|
| 891 |
+
}
|
| 892 |
+
}
|
| 893 |
+
},
|
| 894 |
+
"openbmb--MiniCPM5-1B-SFT": {
|
| 895 |
+
"label": "MiniCPM5-1B-SFT",
|
| 896 |
+
"panels": {
|
| 897 |
+
"alfworld": {
|
| 898 |
+
"status": "COMPLETE",
|
| 899 |
+
"expected": 134,
|
| 900 |
+
"completed": 134,
|
| 901 |
+
"unscored": 0,
|
| 902 |
+
"primary": 0.0,
|
| 903 |
+
"success_count": 0,
|
| 904 |
+
"status_counts": {
|
| 905 |
+
"COMPLETE": 134
|
| 906 |
+
},
|
| 907 |
+
"returned_work": {
|
| 908 |
+
"replies": 8576,
|
| 909 |
+
"input_tokens": 13675913,
|
| 910 |
+
"output_tokens": 55498,
|
| 911 |
+
"length_finishes": 0,
|
| 912 |
+
"nothing_happens": 8283
|
| 913 |
+
},
|
| 914 |
+
"qualification": {
|
| 915 |
+
"status": "QUALIFIED",
|
| 916 |
+
"passed": 2,
|
| 917 |
+
"expected": 2,
|
| 918 |
+
"protocol": "404f9efd664a5dc19e46ec2d862b6a66b0abc6b1bf2e2b778af06e9cbc816fb1"
|
| 919 |
+
}
|
| 920 |
+
},
|
| 921 |
+
"bfcl": {
|
| 922 |
+
"status": "NOT_QUALIFIED",
|
| 923 |
+
"expected": 200,
|
| 924 |
+
"completed": 0,
|
| 925 |
+
"unscored": 200,
|
| 926 |
+
"primary": null,
|
| 927 |
+
"success_count": 0,
|
| 928 |
+
"status_counts": {
|
| 929 |
+
"NOT_QUALIFIED": 200
|
| 930 |
+
},
|
| 931 |
+
"returned_work": {},
|
| 932 |
+
"qualification": {
|
| 933 |
+
"status": "NOT_QUALIFIED",
|
| 934 |
+
"passed": 0,
|
| 935 |
+
"expected": 2,
|
| 936 |
+
"protocol": "2dd41ed3f2610c98ca42ab35ea0f57be5398de86c6ab4279b07e2d11c2d78111"
|
| 937 |
+
}
|
| 938 |
+
}
|
| 939 |
+
}
|
| 940 |
+
},
|
| 941 |
+
"openbmb--MiniCPM5-1B": {
|
| 942 |
+
"label": "MiniCPM5-1B",
|
| 943 |
+
"panels": {
|
| 944 |
+
"alfworld": {
|
| 945 |
+
"status": "NOT_QUALIFIED",
|
| 946 |
+
"expected": 134,
|
| 947 |
+
"completed": 0,
|
| 948 |
+
"unscored": 134,
|
| 949 |
+
"primary": null,
|
| 950 |
+
"success_count": 0,
|
| 951 |
+
"status_counts": {
|
| 952 |
+
"NOT_QUALIFIED": 134
|
| 953 |
+
},
|
| 954 |
+
"returned_work": {},
|
| 955 |
+
"qualification": {
|
| 956 |
+
"status": "NOT_QUALIFIED",
|
| 957 |
+
"passed": 1,
|
| 958 |
+
"expected": 2,
|
| 959 |
+
"protocol": "d04a21375d1b7844d70652ca615574ec9852b78053fde67b84f20a4e82d39aab"
|
| 960 |
+
}
|
| 961 |
+
},
|
| 962 |
+
"bfcl": {
|
| 963 |
+
"status": "COMPLETE",
|
| 964 |
+
"expected": 200,
|
| 965 |
+
"completed": 200,
|
| 966 |
+
"unscored": 0,
|
| 967 |
+
"primary": 0.0,
|
| 968 |
+
"success_count": 0,
|
| 969 |
+
"status_counts": {
|
| 970 |
+
"COMPLETE": 200
|
| 971 |
+
},
|
| 972 |
+
"returned_work": {
|
| 973 |
+
"replies": 3719,
|
| 974 |
+
"input_tokens": 20813620,
|
| 975 |
+
"output_tokens": 114782,
|
| 976 |
+
"length_finishes": 9,
|
| 977 |
+
"invalid_response_turn_end": 127
|
| 978 |
+
},
|
| 979 |
+
"qualification": {
|
| 980 |
+
"status": "QUALIFIED",
|
| 981 |
+
"passed": 2,
|
| 982 |
+
"expected": 2,
|
| 983 |
+
"protocol": "737ee863afaf1ae5b6a8edb4da9ccb96a634f18c7e8972281ed8261e6106c1ab"
|
| 984 |
+
}
|
| 985 |
+
}
|
| 986 |
+
}
|
| 987 |
+
},
|
| 988 |
+
"Qwen--Qwen2.5-1.5B-Instruct": {
|
| 989 |
+
"label": "Qwen2.5-1.5B-Instruct",
|
| 990 |
+
"panels": {
|
| 991 |
+
"alfworld": {
|
| 992 |
+
"status": "NOT_QUALIFIED",
|
| 993 |
+
"expected": 134,
|
| 994 |
+
"completed": 0,
|
| 995 |
+
"unscored": 134,
|
| 996 |
+
"primary": null,
|
| 997 |
+
"success_count": 0,
|
| 998 |
+
"status_counts": {
|
| 999 |
+
"NOT_QUALIFIED": 134
|
| 1000 |
+
},
|
| 1001 |
+
"returned_work": {},
|
| 1002 |
+
"qualification": {
|
| 1003 |
+
"status": "NOT_QUALIFIED",
|
| 1004 |
+
"passed": 0,
|
| 1005 |
+
"expected": 2,
|
| 1006 |
+
"protocol": "7fffba5ba7036643f9ba544fb58e29bdfc522133e9f0fdbcc21da8bce90a10dc"
|
| 1007 |
+
}
|
| 1008 |
+
},
|
| 1009 |
+
"bfcl": {
|
| 1010 |
+
"status": "NOT_QUALIFIED",
|
| 1011 |
+
"expected": 200,
|
| 1012 |
+
"completed": 0,
|
| 1013 |
+
"unscored": 200,
|
| 1014 |
+
"primary": null,
|
| 1015 |
+
"success_count": 0,
|
| 1016 |
+
"status_counts": {
|
| 1017 |
+
"NOT_QUALIFIED": 200
|
| 1018 |
+
},
|
| 1019 |
+
"returned_work": {},
|
| 1020 |
+
"qualification": {
|
| 1021 |
+
"status": "NOT_QUALIFIED",
|
| 1022 |
+
"passed": 0,
|
| 1023 |
+
"expected": 2,
|
| 1024 |
+
"protocol": "2a787dac2dcee864e43b6b05c31aaef830957b0a8410907fbf86a6a0b24ba3fc"
|
| 1025 |
+
}
|
| 1026 |
+
}
|
| 1027 |
+
}
|
| 1028 |
+
},
|
| 1029 |
+
"HuggingFaceTB--SmolLM2-1.7B-Instruct": {
|
| 1030 |
+
"label": "SmolLM2-1.7B-Instruct",
|
| 1031 |
+
"panels": {
|
| 1032 |
+
"alfworld": {
|
| 1033 |
+
"status": "NOT_QUALIFIED",
|
| 1034 |
+
"expected": 134,
|
| 1035 |
+
"completed": 0,
|
| 1036 |
+
"unscored": 134,
|
| 1037 |
+
"primary": null,
|
| 1038 |
+
"success_count": 0,
|
| 1039 |
+
"status_counts": {
|
| 1040 |
+
"NOT_QUALIFIED": 134
|
| 1041 |
+
},
|
| 1042 |
+
"returned_work": {},
|
| 1043 |
+
"qualification": {
|
| 1044 |
+
"status": "NOT_QUALIFIED",
|
| 1045 |
+
"passed": 0,
|
| 1046 |
+
"expected": 2,
|
| 1047 |
+
"protocol": "a90510616603942d47130c40925ad047c4ea6f34dcb0ec8e933e6c4ecfdb58a2"
|
| 1048 |
+
}
|
| 1049 |
+
},
|
| 1050 |
+
"bfcl": {
|
| 1051 |
+
"status": "NOT_QUALIFIED",
|
| 1052 |
+
"expected": 200,
|
| 1053 |
+
"completed": 0,
|
| 1054 |
+
"unscored": 200,
|
| 1055 |
+
"primary": null,
|
| 1056 |
+
"success_count": 0,
|
| 1057 |
+
"status_counts": {
|
| 1058 |
+
"NOT_QUALIFIED": 200
|
| 1059 |
+
},
|
| 1060 |
+
"returned_work": {},
|
| 1061 |
+
"qualification": {
|
| 1062 |
+
"status": "NOT_QUALIFIED",
|
| 1063 |
+
"passed": 0,
|
| 1064 |
+
"expected": 2,
|
| 1065 |
+
"protocol": "fecd5c6842b95f2c074876b3866bbbff45bbae6c152bb45b003687ec1a303700"
|
| 1066 |
+
}
|
| 1067 |
+
}
|
| 1068 |
+
}
|
| 1069 |
+
},
|
| 1070 |
+
"LiquidAI--LFM2.5-1.2B-Instruct": {
|
| 1071 |
+
"label": "LFM2.5-1.2B-Instruct",
|
| 1072 |
+
"panels": {
|
| 1073 |
+
"alfworld": {
|
| 1074 |
+
"status": "NOT_QUALIFIED",
|
| 1075 |
+
"expected": 134,
|
| 1076 |
+
"completed": 0,
|
| 1077 |
+
"unscored": 134,
|
| 1078 |
+
"primary": null,
|
| 1079 |
+
"success_count": 0,
|
| 1080 |
+
"status_counts": {
|
| 1081 |
+
"NOT_QUALIFIED": 134
|
| 1082 |
+
},
|
| 1083 |
+
"returned_work": {},
|
| 1084 |
+
"qualification": {
|
| 1085 |
+
"status": "NOT_QUALIFIED",
|
| 1086 |
+
"passed": 0,
|
| 1087 |
+
"expected": 2,
|
| 1088 |
+
"protocol": "cc5ad2c426e33fe6ffc6d811dd629d1cfb4866f90d91582692f6f04c0df2b96b"
|
| 1089 |
+
}
|
| 1090 |
+
},
|
| 1091 |
+
"bfcl": {
|
| 1092 |
+
"status": "NOT_QUALIFIED",
|
| 1093 |
+
"expected": 200,
|
| 1094 |
+
"completed": 0,
|
| 1095 |
+
"unscored": 200,
|
| 1096 |
+
"primary": null,
|
| 1097 |
+
"success_count": 0,
|
| 1098 |
+
"status_counts": {
|
| 1099 |
+
"NOT_QUALIFIED": 200
|
| 1100 |
+
},
|
| 1101 |
+
"returned_work": {},
|
| 1102 |
+
"qualification": {
|
| 1103 |
+
"status": "NOT_QUALIFIED",
|
| 1104 |
+
"passed": 0,
|
| 1105 |
+
"expected": 2,
|
| 1106 |
+
"protocol": "c5474dec941291bbbde12fc5ef71234bb906cb6bd48cd32029c90c356a8aaa98"
|
| 1107 |
+
}
|
| 1108 |
+
}
|
| 1109 |
+
}
|
| 1110 |
+
},
|
| 1111 |
+
"HuggingFaceTB--SmolLM3-3B": {
|
| 1112 |
+
"label": "SmolLM3-3B",
|
| 1113 |
+
"panels": {
|
| 1114 |
+
"alfworld": {
|
| 1115 |
+
"status": "NOT_QUALIFIED",
|
| 1116 |
+
"expected": 134,
|
| 1117 |
+
"completed": 0,
|
| 1118 |
+
"unscored": 134,
|
| 1119 |
+
"primary": null,
|
| 1120 |
+
"success_count": 0,
|
| 1121 |
+
"status_counts": {
|
| 1122 |
+
"NOT_QUALIFIED": 134
|
| 1123 |
+
},
|
| 1124 |
+
"returned_work": {},
|
| 1125 |
+
"qualification": {
|
| 1126 |
+
"status": "NOT_QUALIFIED",
|
| 1127 |
+
"passed": 0,
|
| 1128 |
+
"expected": 2,
|
| 1129 |
+
"protocol": "25bd0e0b50c3986f130678dfa088fe9ee60049b905835c2e2ab3b3b646cb13cc"
|
| 1130 |
+
}
|
| 1131 |
+
},
|
| 1132 |
+
"bfcl": {
|
| 1133 |
+
"status": "NOT_QUALIFIED",
|
| 1134 |
+
"expected": 200,
|
| 1135 |
+
"completed": 0,
|
| 1136 |
+
"unscored": 200,
|
| 1137 |
+
"primary": null,
|
| 1138 |
+
"success_count": 0,
|
| 1139 |
+
"status_counts": {
|
| 1140 |
+
"NOT_QUALIFIED": 200
|
| 1141 |
+
},
|
| 1142 |
+
"returned_work": {},
|
| 1143 |
+
"qualification": {
|
| 1144 |
+
"status": "NOT_QUALIFIED",
|
| 1145 |
+
"passed": 0,
|
| 1146 |
+
"expected": 2,
|
| 1147 |
+
"protocol": "5bc54d27e7d95e33a2fa5b1b53404000726e94fe494f7acce26f9f244a3f045c"
|
| 1148 |
+
}
|
| 1149 |
+
}
|
| 1150 |
+
}
|
| 1151 |
+
},
|
| 1152 |
+
"Qwen--Qwen3.5-4B": {
|
| 1153 |
+
"label": "Qwen3.5-4B",
|
| 1154 |
+
"panels": {
|
| 1155 |
+
"alfworld": {
|
| 1156 |
+
"status": "COMPLETE",
|
| 1157 |
+
"expected": 134,
|
| 1158 |
+
"completed": 134,
|
| 1159 |
+
"unscored": 0,
|
| 1160 |
+
"primary": 0.03731343283582089,
|
| 1161 |
+
"success_count": 5,
|
| 1162 |
+
"status_counts": {
|
| 1163 |
+
"COMPLETE": 134
|
| 1164 |
+
},
|
| 1165 |
+
"returned_work": {
|
| 1166 |
+
"replies": 8304,
|
| 1167 |
+
"input_tokens": 14675687,
|
| 1168 |
+
"output_tokens": 48740,
|
| 1169 |
+
"length_finishes": 0,
|
| 1170 |
+
"nothing_happens": 4578
|
| 1171 |
+
},
|
| 1172 |
+
"qualification": {
|
| 1173 |
+
"status": "QUALIFIED",
|
| 1174 |
+
"passed": 2,
|
| 1175 |
+
"expected": 2,
|
| 1176 |
+
"protocol": "20a9ff73f46a316127e7227aaf64a57ebeb1f7227c89116739b7a71d1c1aad1f"
|
| 1177 |
+
}
|
| 1178 |
+
},
|
| 1179 |
+
"bfcl": {
|
| 1180 |
+
"status": "NOT_QUALIFIED",
|
| 1181 |
+
"expected": 200,
|
| 1182 |
+
"completed": 0,
|
| 1183 |
+
"unscored": 200,
|
| 1184 |
+
"primary": null,
|
| 1185 |
+
"success_count": 0,
|
| 1186 |
+
"status_counts": {
|
| 1187 |
+
"NOT_QUALIFIED": 200
|
| 1188 |
+
},
|
| 1189 |
+
"returned_work": {},
|
| 1190 |
+
"qualification": {
|
| 1191 |
+
"status": "NOT_QUALIFIED",
|
| 1192 |
+
"passed": 0,
|
| 1193 |
+
"expected": 2,
|
| 1194 |
+
"protocol": "350f69949ba27db97b196cc5f4906e2d60dae36532c1e55c263bbd173726b9f7"
|
| 1195 |
+
}
|
| 1196 |
+
}
|
| 1197 |
+
}
|
| 1198 |
+
},
|
| 1199 |
+
"Qwen--Qwen3.5-9B": {
|
| 1200 |
+
"label": "Qwen3.5-9B",
|
| 1201 |
+
"panels": {
|
| 1202 |
+
"alfworld": {
|
| 1203 |
+
"status": "COMPLETE",
|
| 1204 |
+
"expected": 134,
|
| 1205 |
+
"completed": 134,
|
| 1206 |
+
"unscored": 0,
|
| 1207 |
+
"primary": 0.2462686567164179,
|
| 1208 |
+
"success_count": 33,
|
| 1209 |
+
"status_counts": {
|
| 1210 |
+
"COMPLETE": 134
|
| 1211 |
+
},
|
| 1212 |
+
"returned_work": {
|
| 1213 |
+
"replies": 6888,
|
| 1214 |
+
"input_tokens": 12668791,
|
| 1215 |
+
"output_tokens": 49242,
|
| 1216 |
+
"length_finishes": 0,
|
| 1217 |
+
"nothing_happens": 4466
|
| 1218 |
+
},
|
| 1219 |
+
"qualification": {
|
| 1220 |
+
"status": "QUALIFIED",
|
| 1221 |
+
"passed": 2,
|
| 1222 |
+
"expected": 2,
|
| 1223 |
+
"protocol": "489294ca7e2e7553b172652434f9be9fec60987efc1a1bb7b80f3ef2ebbdec54"
|
| 1224 |
+
}
|
| 1225 |
+
},
|
| 1226 |
+
"bfcl": {
|
| 1227 |
+
"status": "COMPLETE",
|
| 1228 |
+
"expected": 200,
|
| 1229 |
+
"completed": 200,
|
| 1230 |
+
"unscored": 0,
|
| 1231 |
+
"primary": 0.38,
|
| 1232 |
+
"success_count": 76,
|
| 1233 |
+
"status_counts": {
|
| 1234 |
+
"COMPLETE": 200
|
| 1235 |
+
},
|
| 1236 |
+
"returned_work": {
|
| 1237 |
+
"replies": 1999,
|
| 1238 |
+
"input_tokens": 9895239,
|
| 1239 |
+
"output_tokens": 33443,
|
| 1240 |
+
"length_finishes": 0,
|
| 1241 |
+
"invalid_response_turn_end": 54
|
| 1242 |
+
},
|
| 1243 |
+
"qualification": {
|
| 1244 |
+
"status": "QUALIFIED",
|
| 1245 |
+
"passed": 2,
|
| 1246 |
+
"expected": 2,
|
| 1247 |
+
"protocol": "9a798b3133c866fa47c3ef1b41978c3a7f2a9674e720ca1a8da1049ba76f840b"
|
| 1248 |
+
}
|
| 1249 |
+
}
|
| 1250 |
+
}
|
| 1251 |
+
}
|
| 1252 |
+
},
|
| 1253 |
+
"performance": {
|
| 1254 |
+
"logged_event_window_seconds": 3482.9023389816284,
|
| 1255 |
+
"phase_wall_seconds": {
|
| 1256 |
+
"qualification": 59.02381372451782,
|
| 1257 |
+
"benchmark": 2828.561756372452
|
| 1258 |
+
},
|
| 1259 |
+
"returned_generated_tokens": {
|
| 1260 |
+
"qualification": 2245,
|
| 1261 |
+
"benchmark": 347572
|
| 1262 |
+
},
|
| 1263 |
+
"model_load_and_preparation_seconds": 587.3164191246033,
|
| 1264 |
+
"remaining_event_window_seconds": 8.000349760055542,
|
| 1265 |
+
"returned_output_tokens_per_phase_wall_second": {
|
| 1266 |
+
"qualification": 38.03549547777616,
|
| 1267 |
+
"benchmark": 122.879410080744
|
| 1268 |
+
},
|
| 1269 |
+
"sampled_mean_UTL": 38.951079136690645,
|
| 1270 |
+
"sampled_mean_W": 199.63323741007196,
|
| 1271 |
+
"samples": 695,
|
| 1272 |
+
"MFU": null,
|
| 1273 |
+
"MFU_status": "not measured; UTL, logical input tokens and padded work are not useful FLOPs",
|
| 1274 |
+
"note": "Returned token totals include failed-task replies saved in traces, exclude unreturned calls, and do not equal physical FLOPs. Samples are whole-device measurements within this allocation. Event window excludes entry setup before its first event. Job intervals end at their normal completion/context-limit event or matching process exit; absent exit evidence yields unknown duration and rate. Model-load intervals run from SERVER_START to first JOB_START; remaining time includes cleanup, scheduling and any incomplete startup, not a separately timed cleanup measurement.",
|
| 1275 |
+
"intervals": [
|
| 1276 |
+
{
|
| 1277 |
+
"model": "Orbit",
|
| 1278 |
+
"panel": "alfworld",
|
| 1279 |
+
"phase": "qualification",
|
| 1280 |
+
"start": 1793538313.9631853,
|
| 1281 |
+
"end": 1793538317.9649694,
|
| 1282 |
+
"seconds": 4.001784086227417,
|
| 1283 |
+
"termination_event": "JOB_COMPLETE",
|
| 1284 |
+
"samples": 1,
|
| 1285 |
+
"mean_UTL": 0.0,
|
| 1286 |
+
"mean_W": 116.3,
|
| 1287 |
+
"peak_memory_MiB": 2952.0,
|
| 1288 |
+
"completed_tasks": 2,
|
| 1289 |
+
"completed_tasks_per_second": 0.49977708864484255,
|
| 1290 |
+
"completed_task_output_tokens": 49
|
| 1291 |
+
},
|
| 1292 |
+
{
|
| 1293 |
+
"model": "Orbit",
|
| 1294 |
+
"panel": "alfworld",
|
| 1295 |
+
"phase": "benchmark",
|
| 1296 |
+
"start": 1793538317.968124,
|
| 1297 |
+
"end": 1793538765.0472424,
|
| 1298 |
+
"seconds": 447.0791184902191,
|
| 1299 |
+
"termination_event": "JOB_COMPLETE",
|
| 1300 |
+
"samples": 89,
|
| 1301 |
+
"mean_UTL": 27.685393258426966,
|
| 1302 |
+
"mean_W": 188.55269662921347,
|
| 1303 |
+
"peak_memory_MiB": 11516.0,
|
| 1304 |
+
"completed_tasks": 32,
|
| 1305 |
+
"completed_tasks_per_second": 0.07157569807344978,
|
| 1306 |
+
"completed_task_output_tokens": 11438
|
| 1307 |
+
},
|
| 1308 |
+
{
|
| 1309 |
+
"model": "Orbit",
|
| 1310 |
+
"panel": "alfworld",
|
| 1311 |
+
"phase": "benchmark",
|
| 1312 |
+
"start": 1793538765.050269,
|
| 1313 |
+
"end": 1793539281.11915,
|
| 1314 |
+
"seconds": 516.0688810348511,
|
| 1315 |
+
"termination_event": "JOB_COMPLETE",
|
| 1316 |
+
"samples": 103,
|
| 1317 |
+
"mean_UTL": 31.02912621359223,
|
| 1318 |
+
"mean_W": 173.2988349514563,
|
| 1319 |
+
"peak_memory_MiB": 11516.0,
|
| 1320 |
+
"completed_tasks": 32,
|
| 1321 |
+
"completed_tasks_per_second": 0.06200722650788739,
|
| 1322 |
+
"completed_task_output_tokens": 11821
|
| 1323 |
+
},
|
| 1324 |
+
{
|
| 1325 |
+
"model": "Orbit",
|
| 1326 |
+
"panel": "alfworld",
|
| 1327 |
+
"phase": "benchmark",
|
| 1328 |
+
"start": 1793539281.1217546,
|
| 1329 |
+
"end": 1793539750.205558,
|
| 1330 |
+
"seconds": 469.08380341529846,
|
| 1331 |
+
"termination_event": "JOB_COMPLETE",
|
| 1332 |
+
"samples": 93,
|
| 1333 |
+
"mean_UTL": 31.93548387096774,
|
| 1334 |
+
"mean_W": 187.5037634408602,
|
| 1335 |
+
"peak_memory_MiB": 18756.0,
|
| 1336 |
+
"completed_tasks": 32,
|
| 1337 |
+
"completed_tasks_per_second": 0.06821808761465407,
|
| 1338 |
+
"completed_task_output_tokens": 11208
|
| 1339 |
+
},
|
| 1340 |
+
{
|
| 1341 |
+
"model": "Orbit",
|
| 1342 |
+
"panel": "alfworld",
|
| 1343 |
+
"phase": "benchmark",
|
| 1344 |
+
"start": 1793539750.208408,
|
| 1345 |
+
"end": 1793540178.2816718,
|
| 1346 |
+
"seconds": 428.0732636451721,
|
| 1347 |
+
"termination_event": "JOB_COMPLETE",
|
| 1348 |
+
"samples": 86,
|
| 1349 |
+
"mean_UTL": 29.151162790697676,
|
| 1350 |
+
"mean_W": 174.84337209302325,
|
| 1351 |
+
"peak_memory_MiB": 21324.0,
|
| 1352 |
+
"completed_tasks": 32,
|
| 1353 |
+
"completed_tasks_per_second": 0.07475355906956302,
|
| 1354 |
+
"completed_task_output_tokens": 10184
|
| 1355 |
+
},
|
| 1356 |
+
{
|
| 1357 |
+
"model": "Orbit",
|
| 1358 |
+
"panel": "alfworld",
|
| 1359 |
+
"phase": "benchmark",
|
| 1360 |
+
"start": 1793540178.2836037,
|
| 1361 |
+
"end": 1793540234.2931137,
|
| 1362 |
+
"seconds": 56.0095100402832,
|
| 1363 |
+
"termination_event": "JOB_COMPLETE",
|
| 1364 |
+
"samples": 11,
|
| 1365 |
+
"mean_UTL": 74.27272727272727,
|
| 1366 |
+
"mean_W": 171.5772727272727,
|
| 1367 |
+
"peak_memory_MiB": 21324.0,
|
| 1368 |
+
"completed_tasks": 6,
|
| 1369 |
+
"completed_tasks_per_second": 0.10712466500215187,
|
| 1370 |
+
"completed_task_output_tokens": 1216
|
| 1371 |
+
},
|
| 1372 |
+
{
|
| 1373 |
+
"model": "Orbit",
|
| 1374 |
+
"panel": "bfcl",
|
| 1375 |
+
"phase": "qualification",
|
| 1376 |
+
"start": 1793540242.36775,
|
| 1377 |
+
"end": 1793540247.369418,
|
| 1378 |
+
"seconds": 5.0016679763793945,
|
| 1379 |
+
"termination_event": "JOB_COMPLETE",
|
| 1380 |
+
"samples": 1,
|
| 1381 |
+
"mean_UTL": 1.0,
|
| 1382 |
+
"mean_W": 95.89,
|
| 1383 |
+
"peak_memory_MiB": 2874.0,
|
| 1384 |
+
"completed_tasks": 2,
|
| 1385 |
+
"completed_tasks_per_second": 0.39986660638912686,
|
| 1386 |
+
"completed_task_output_tokens": 54
|
| 1387 |
+
},
|
| 1388 |
+
{
|
| 1389 |
+
"model": "Qwen--Qwen3.5-9B",
|
| 1390 |
+
"panel": "alfworld",
|
| 1391 |
+
"phase": "qualification",
|
| 1392 |
+
"start": 1793540973.7034795,
|
| 1393 |
+
"end": 1793540976.7045462,
|
| 1394 |
+
"seconds": 3.0010666847229004,
|
| 1395 |
+
"termination_event": "JOB_COMPLETE",
|
| 1396 |
+
"samples": 1,
|
| 1397 |
+
"mean_UTL": 0.0,
|
| 1398 |
+
"mean_W": 96.74,
|
| 1399 |
+
"peak_memory_MiB": 42426.0,
|
| 1400 |
+
"completed_tasks": 2,
|
| 1401 |
+
"completed_tasks_per_second": 0.6664297098698649,
|
| 1402 |
+
"completed_task_output_tokens": 38
|
| 1403 |
+
},
|
| 1404 |
+
{
|
| 1405 |
+
"model": "Qwen--Qwen3.5-9B",
|
| 1406 |
+
"panel": "alfworld",
|
| 1407 |
+
"phase": "benchmark",
|
| 1408 |
+
"start": 1793540976.7085817,
|
| 1409 |
+
"end": 1793541036.732439,
|
| 1410 |
+
"seconds": 60.0238573551178,
|
| 1411 |
+
"termination_event": "JOB_COMPLETE",
|
| 1412 |
+
"samples": 12,
|
| 1413 |
+
"mean_UTL": 83.33333333333333,
|
| 1414 |
+
"mean_W": 377.20666666666665,
|
| 1415 |
+
"peak_memory_MiB": 43576.0,
|
| 1416 |
+
"completed_tasks": 32,
|
| 1417 |
+
"completed_tasks_per_second": 0.5331213522429776,
|
| 1418 |
+
"completed_task_output_tokens": 8678
|
| 1419 |
+
},
|
| 1420 |
+
{
|
| 1421 |
+
"model": "Qwen--Qwen3.5-9B",
|
| 1422 |
+
"panel": "alfworld",
|
| 1423 |
+
"phase": "benchmark",
|
| 1424 |
+
"start": 1793541036.7377534,
|
| 1425 |
+
"end": 1793541107.750358,
|
| 1426 |
+
"seconds": 71.01260471343994,
|
| 1427 |
+
"termination_event": "JOB_COMPLETE",
|
| 1428 |
+
"samples": 14,
|
| 1429 |
+
"mean_UTL": 85.71428571428571,
|
| 1430 |
+
"mean_W": 396.35,
|
| 1431 |
+
"peak_memory_MiB": 43576.0,
|
| 1432 |
+
"completed_tasks": 32,
|
| 1433 |
+
"completed_tasks_per_second": 0.4506242255037807,
|
| 1434 |
+
"completed_task_output_tokens": 11625
|
| 1435 |
+
},
|
| 1436 |
+
{
|
| 1437 |
+
"model": "Qwen--Qwen3.5-9B",
|
| 1438 |
+
"panel": "alfworld",
|
| 1439 |
+
"phase": "benchmark",
|
| 1440 |
+
"start": 1793541107.7554195,
|
| 1441 |
+
"end": 1793541186.774323,
|
| 1442 |
+
"seconds": 79.01890349388123,
|
| 1443 |
+
"termination_event": "JOB_COMPLETE",
|
| 1444 |
+
"samples": 15,
|
| 1445 |
+
"mean_UTL": 90.73333333333333,
|
| 1446 |
+
"mean_W": 410.74133333333333,
|
| 1447 |
+
"peak_memory_MiB": 43576.0,
|
| 1448 |
+
"completed_tasks": 32,
|
| 1449 |
+
"completed_tasks_per_second": 0.4049663888651391,
|
| 1450 |
+
"completed_task_output_tokens": 13940
|
| 1451 |
+
},
|
| 1452 |
+
{
|
| 1453 |
+
"model": "Qwen--Qwen3.5-9B",
|
| 1454 |
+
"panel": "alfworld",
|
| 1455 |
+
"phase": "benchmark",
|
| 1456 |
+
"start": 1793541186.77937,
|
| 1457 |
+
"end": 1793541262.8006992,
|
| 1458 |
+
"seconds": 76.021329164505,
|
| 1459 |
+
"termination_event": "JOB_COMPLETE",
|
| 1460 |
+
"samples": 16,
|
| 1461 |
+
"mean_UTL": 81.25,
|
| 1462 |
+
"mean_W": 372.585625,
|
| 1463 |
+
"peak_memory_MiB": 43576.0,
|
| 1464 |
+
"completed_tasks": 32,
|
| 1465 |
+
"completed_tasks_per_second": 0.42093449761650664,
|
| 1466 |
+
"completed_task_output_tokens": 13414
|
| 1467 |
+
},
|
| 1468 |
+
{
|
| 1469 |
+
"model": "Qwen--Qwen3.5-9B",
|
| 1470 |
+
"panel": "alfworld",
|
| 1471 |
+
"phase": "benchmark",
|
| 1472 |
+
"start": 1793541262.8055255,
|
| 1473 |
+
"end": 1793541286.810321,
|
| 1474 |
+
"seconds": 24.00479555130005,
|
| 1475 |
+
"termination_event": "JOB_COMPLETE",
|
| 1476 |
+
"samples": 4,
|
| 1477 |
+
"mean_UTL": 91.25,
|
| 1478 |
+
"mean_W": 382.0375,
|
| 1479 |
+
"peak_memory_MiB": 43576.0,
|
| 1480 |
+
"completed_tasks": 6,
|
| 1481 |
+
"completed_tasks_per_second": 0.24995005632010278,
|
| 1482 |
+
"completed_task_output_tokens": 1585
|
| 1483 |
+
},
|
| 1484 |
+
{
|
| 1485 |
+
"model": "Qwen--Qwen3.5-9B",
|
| 1486 |
+
"panel": "bfcl",
|
| 1487 |
+
"phase": "qualification",
|
| 1488 |
+
"start": 1793540392.2453332,
|
| 1489 |
+
"end": 1793540395.2467415,
|
| 1490 |
+
"seconds": 3.001408338546753,
|
| 1491 |
+
"termination_event": "JOB_COMPLETE",
|
| 1492 |
+
"samples": 1,
|
| 1493 |
+
"mean_UTL": 0.0,
|
| 1494 |
+
"mean_W": 156.23,
|
| 1495 |
+
"peak_memory_MiB": 41086.0,
|
| 1496 |
+
"completed_tasks": 2,
|
| 1497 |
+
"completed_tasks_per_second": 0.6663538493960395,
|
| 1498 |
+
"completed_task_output_tokens": 45
|
| 1499 |
+
},
|
| 1500 |
+
{
|
| 1501 |
+
"model": "Qwen--Qwen3.5-9B",
|
| 1502 |
+
"panel": "bfcl",
|
| 1503 |
+
"phase": "benchmark",
|
| 1504 |
+
"start": 1793540395.250629,
|
| 1505 |
+
"end": 1793540419.2554846,
|
| 1506 |
+
"seconds": 24.004855632781982,
|
| 1507 |
+
"termination_event": "JOB_COMPLETE",
|
| 1508 |
+
"samples": 5,
|
| 1509 |
+
"mean_UTL": 80.0,
|
| 1510 |
+
"mean_W": 370.206,
|
| 1511 |
+
"peak_memory_MiB": 42502.0,
|
| 1512 |
+
"completed_tasks": 32,
|
| 1513 |
+
"completed_tasks_per_second": 1.3330636305222987,
|
| 1514 |
+
"completed_task_output_tokens": 4092
|
| 1515 |
+
},
|
| 1516 |
+
{
|
| 1517 |
+
"model": "Qwen--Qwen3.5-9B",
|
| 1518 |
+
"panel": "bfcl",
|
| 1519 |
+
"phase": "benchmark",
|
| 1520 |
+
"start": 1793540419.2587035,
|
| 1521 |
+
"end": 1793540453.2645802,
|
| 1522 |
+
"seconds": 34.005876779556274,
|
| 1523 |
+
"termination_event": "JOB_COMPLETE",
|
| 1524 |
+
"samples": 6,
|
| 1525 |
+
"mean_UTL": 97.33333333333333,
|
| 1526 |
+
"mean_W": 406.5566666666667,
|
| 1527 |
+
"peak_memory_MiB": 42502.0,
|
| 1528 |
+
"completed_tasks": 32,
|
| 1529 |
+
"completed_tasks_per_second": 0.9410138196830092,
|
| 1530 |
+
"completed_task_output_tokens": 4825
|
| 1531 |
+
},
|
| 1532 |
+
{
|
| 1533 |
+
"model": "Qwen--Qwen3.5-9B",
|
| 1534 |
+
"panel": "bfcl",
|
| 1535 |
+
"phase": "benchmark",
|
| 1536 |
+
"start": 1793540453.268372,
|
| 1537 |
+
"end": 1793540475.2724211,
|
| 1538 |
+
"seconds": 22.004049062728882,
|
| 1539 |
+
"termination_event": "JOB_COMPLETE",
|
| 1540 |
+
"samples": 5,
|
| 1541 |
+
"mean_UTL": 80.0,
|
| 1542 |
+
"mean_W": 315.35,
|
| 1543 |
+
"peak_memory_MiB": 42502.0,
|
| 1544 |
+
"completed_tasks": 32,
|
| 1545 |
+
"completed_tasks_per_second": 1.4542777971806362,
|
| 1546 |
+
"completed_task_output_tokens": 4921
|
| 1547 |
+
},
|
| 1548 |
+
{
|
| 1549 |
+
"model": "Qwen--Qwen3.5-9B",
|
| 1550 |
+
"panel": "bfcl",
|
| 1551 |
+
"phase": "benchmark",
|
| 1552 |
+
"start": 1793540475.275762,
|
| 1553 |
+
"end": 1793540495.2791128,
|
| 1554 |
+
"seconds": 20.003350734710693,
|
| 1555 |
+
"termination_event": "JOB_COMPLETE",
|
| 1556 |
+
"samples": 4,
|
| 1557 |
+
"mean_UTL": 75.0,
|
| 1558 |
+
"mean_W": 369.53,
|
| 1559 |
+
"peak_memory_MiB": 42502.0,
|
| 1560 |
+
"completed_tasks": 32,
|
| 1561 |
+
"completed_tasks_per_second": 1.599731986125314,
|
| 1562 |
+
"completed_task_output_tokens": 4019
|
| 1563 |
+
},
|
| 1564 |
+
{
|
| 1565 |
+
"model": "Qwen--Qwen3.5-9B",
|
| 1566 |
+
"panel": "bfcl",
|
| 1567 |
+
"phase": "benchmark",
|
| 1568 |
+
"start": 1793540495.2829401,
|
| 1569 |
+
"end": 1793540519.2877417,
|
| 1570 |
+
"seconds": 24.004801511764526,
|
| 1571 |
+
"termination_event": "JOB_COMPLETE",
|
| 1572 |
+
"samples": 4,
|
| 1573 |
+
"mean_UTL": 100.0,
|
| 1574 |
+
"mean_W": 431.565,
|
| 1575 |
+
"peak_memory_MiB": 42502.0,
|
| 1576 |
+
"completed_tasks": 32,
|
| 1577 |
+
"completed_tasks_per_second": 1.3330666360360073,
|
| 1578 |
+
"completed_task_output_tokens": 4841
|
| 1579 |
+
},
|
| 1580 |
+
{
|
| 1581 |
+
"model": "Qwen--Qwen3.5-9B",
|
| 1582 |
+
"panel": "bfcl",
|
| 1583 |
+
"phase": "benchmark",
|
| 1584 |
+
"start": 1793540519.2917953,
|
| 1585 |
+
"end": 1793540547.2975574,
|
| 1586 |
+
"seconds": 28.005762100219727,
|
| 1587 |
+
"termination_event": "JOB_COMPLETE",
|
| 1588 |
+
"samples": 6,
|
| 1589 |
+
"mean_UTL": 78.5,
|
| 1590 |
+
"mean_W": 361.42,
|
| 1591 |
+
"peak_memory_MiB": 42502.0,
|
| 1592 |
+
"completed_tasks": 32,
|
| 1593 |
+
"completed_tasks_per_second": 1.1426220034822383,
|
| 1594 |
+
"completed_task_output_tokens": 8521
|
| 1595 |
+
},
|
| 1596 |
+
{
|
| 1597 |
+
"model": "Qwen--Qwen3.5-9B",
|
| 1598 |
+
"panel": "bfcl",
|
| 1599 |
+
"phase": "benchmark",
|
| 1600 |
+
"start": 1793540547.3013208,
|
| 1601 |
+
"end": 1793540561.3049686,
|
| 1602 |
+
"seconds": 14.003647804260254,
|
| 1603 |
+
"termination_event": "JOB_COMPLETE",
|
| 1604 |
+
"samples": 3,
|
| 1605 |
+
"mean_UTL": 97.0,
|
| 1606 |
+
"mean_W": 352.29333333333335,
|
| 1607 |
+
"peak_memory_MiB": 42502.0,
|
| 1608 |
+
"completed_tasks": 8,
|
| 1609 |
+
"completed_tasks_per_second": 0.5712797202430501,
|
| 1610 |
+
"completed_task_output_tokens": 2224
|
| 1611 |
+
},
|
| 1612 |
+
{
|
| 1613 |
+
"model": "Qwen--Qwen3.5-4B",
|
| 1614 |
+
"panel": "alfworld",
|
| 1615 |
+
"phase": "qualification",
|
| 1616 |
+
"start": 1793541313.0388699,
|
| 1617 |
+
"end": 1793541316.040314,
|
| 1618 |
+
"seconds": 3.001444101333618,
|
| 1619 |
+
"termination_event": "JOB_COMPLETE",
|
| 1620 |
+
"samples": 0,
|
| 1621 |
+
"mean_UTL": null,
|
| 1622 |
+
"mean_W": null,
|
| 1623 |
+
"peak_memory_MiB": null,
|
| 1624 |
+
"completed_tasks": 2,
|
| 1625 |
+
"completed_tasks_per_second": 0.6663459096610691,
|
| 1626 |
+
"completed_task_output_tokens": 38
|
| 1627 |
+
},
|
| 1628 |
+
{
|
| 1629 |
+
"model": "Qwen--Qwen3.5-4B",
|
| 1630 |
+
"panel": "alfworld",
|
| 1631 |
+
"phase": "benchmark",
|
| 1632 |
+
"start": 1793541316.0443146,
|
| 1633 |
+
"end": 1793541366.0601938,
|
| 1634 |
+
"seconds": 50.01587915420532,
|
| 1635 |
+
"termination_event": "JOB_COMPLETE",
|
| 1636 |
+
"samples": 10,
|
| 1637 |
+
"mean_UTL": 81.2,
|
| 1638 |
+
"mean_W": 351.175,
|
| 1639 |
+
"peak_memory_MiB": 44578.0,
|
| 1640 |
+
"completed_tasks": 32,
|
| 1641 |
+
"completed_tasks_per_second": 0.6397968113554483,
|
| 1642 |
+
"completed_task_output_tokens": 10171
|
| 1643 |
+
},
|
| 1644 |
+
{
|
| 1645 |
+
"model": "Qwen--Qwen3.5-4B",
|
| 1646 |
+
"panel": "alfworld",
|
| 1647 |
+
"phase": "benchmark",
|
| 1648 |
+
"start": 1793541366.0650525,
|
| 1649 |
+
"end": 1793541418.082877,
|
| 1650 |
+
"seconds": 52.01782441139221,
|
| 1651 |
+
"termination_event": "JOB_COMPLETE",
|
| 1652 |
+
"samples": 10,
|
| 1653 |
+
"mean_UTL": 82.6,
|
| 1654 |
+
"mean_W": 371.936,
|
| 1655 |
+
"peak_memory_MiB": 44578.0,
|
| 1656 |
+
"completed_tasks": 32,
|
| 1657 |
+
"completed_tasks_per_second": 0.6151737478853847,
|
| 1658 |
+
"completed_task_output_tokens": 11264
|
| 1659 |
+
},
|
| 1660 |
+
{
|
| 1661 |
+
"model": "Qwen--Qwen3.5-4B",
|
| 1662 |
+
"panel": "alfworld",
|
| 1663 |
+
"phase": "benchmark",
|
| 1664 |
+
"start": 1793541418.0863738,
|
| 1665 |
+
"end": 1793541471.0973716,
|
| 1666 |
+
"seconds": 53.0109977722168,
|
| 1667 |
+
"termination_event": "JOB_COMPLETE",
|
| 1668 |
+
"samples": 11,
|
| 1669 |
+
"mean_UTL": 74.27272727272727,
|
| 1670 |
+
"mean_W": 357.95272727272726,
|
| 1671 |
+
"peak_memory_MiB": 44578.0,
|
| 1672 |
+
"completed_tasks": 32,
|
| 1673 |
+
"completed_tasks_per_second": 0.603648324777831,
|
| 1674 |
+
"completed_task_output_tokens": 10344
|
| 1675 |
+
},
|
| 1676 |
+
{
|
| 1677 |
+
"model": "Qwen--Qwen3.5-4B",
|
| 1678 |
+
"panel": "alfworld",
|
| 1679 |
+
"phase": "benchmark",
|
| 1680 |
+
"start": 1793541471.1016855,
|
| 1681 |
+
"end": 1793541525.1161807,
|
| 1682 |
+
"seconds": 54.01449513435364,
|
| 1683 |
+
"termination_event": "JOB_COMPLETE",
|
| 1684 |
+
"samples": 11,
|
| 1685 |
+
"mean_UTL": 72.63636363636364,
|
| 1686 |
+
"mean_W": 342.4190909090909,
|
| 1687 |
+
"peak_memory_MiB": 44578.0,
|
| 1688 |
+
"completed_tasks": 32,
|
| 1689 |
+
"completed_tasks_per_second": 0.5924335665899384,
|
| 1690 |
+
"completed_task_output_tokens": 14000
|
| 1691 |
+
},
|
| 1692 |
+
{
|
| 1693 |
+
"model": "Qwen--Qwen3.5-4B",
|
| 1694 |
+
"panel": "alfworld",
|
| 1695 |
+
"phase": "benchmark",
|
| 1696 |
+
"start": 1793541525.1192768,
|
| 1697 |
+
"end": 1793541545.1222277,
|
| 1698 |
+
"seconds": 20.00295090675354,
|
| 1699 |
+
"termination_event": "JOB_COMPLETE",
|
| 1700 |
+
"samples": 4,
|
| 1701 |
+
"mean_UTL": 72.5,
|
| 1702 |
+
"mean_W": 339.78249999999997,
|
| 1703 |
+
"peak_memory_MiB": 44578.0,
|
| 1704 |
+
"completed_tasks": 6,
|
| 1705 |
+
"completed_tasks_per_second": 0.2999557429286214,
|
| 1706 |
+
"completed_task_output_tokens": 2961
|
| 1707 |
+
},
|
| 1708 |
+
{
|
| 1709 |
+
"model": "Qwen--Qwen3.5-4B",
|
| 1710 |
+
"panel": "bfcl",
|
| 1711 |
+
"phase": "qualification",
|
| 1712 |
+
"start": 1793540645.9493148,
|
| 1713 |
+
"end": 1793540648.9506512,
|
| 1714 |
+
"seconds": 3.0013363361358643,
|
| 1715 |
+
"termination_event": "JOB_COMPLETE",
|
| 1716 |
+
"samples": 0,
|
| 1717 |
+
"mean_UTL": null,
|
| 1718 |
+
"mean_W": null,
|
| 1719 |
+
"peak_memory_MiB": null,
|
| 1720 |
+
"completed_tasks": 2,
|
| 1721 |
+
"completed_tasks_per_second": 0.6663698353030781,
|
| 1722 |
+
"completed_task_output_tokens": 64
|
| 1723 |
+
},
|
| 1724 |
+
{
|
| 1725 |
+
"model": "HuggingFaceTB--SmolLM3-3B",
|
| 1726 |
+
"panel": "alfworld",
|
| 1727 |
+
"phase": "qualification",
|
| 1728 |
+
"start": 1793541573.2847347,
|
| 1729 |
+
"end": 1793541576.2864163,
|
| 1730 |
+
"seconds": 3.0016815662384033,
|
| 1731 |
+
"termination_event": "JOB_COMPLETE",
|
| 1732 |
+
"samples": 1,
|
| 1733 |
+
"mean_UTL": 0.0,
|
| 1734 |
+
"mean_W": 55.84,
|
| 1735 |
+
"peak_memory_MiB": 42548.0,
|
| 1736 |
+
"completed_tasks": 2,
|
| 1737 |
+
"completed_tasks_per_second": 0.6662931946196832,
|
| 1738 |
+
"completed_task_output_tokens": 120
|
| 1739 |
+
},
|
| 1740 |
+
{
|
| 1741 |
+
"model": "HuggingFaceTB--SmolLM3-3B",
|
| 1742 |
+
"panel": "bfcl",
|
| 1743 |
+
"phase": "qualification",
|
| 1744 |
+
"start": 1793540691.3894026,
|
| 1745 |
+
"end": 1793540695.3909073,
|
| 1746 |
+
"seconds": 4.00150465965271,
|
| 1747 |
+
"termination_event": "JOB_COMPLETE",
|
| 1748 |
+
"samples": 1,
|
| 1749 |
+
"mean_UTL": 0.0,
|
| 1750 |
+
"mean_W": 99.37,
|
| 1751 |
+
"peak_memory_MiB": 42468.0,
|
| 1752 |
+
"completed_tasks": 2,
|
| 1753 |
+
"completed_tasks_per_second": 0.49981198826682854,
|
| 1754 |
+
"completed_task_output_tokens": 408
|
| 1755 |
+
},
|
| 1756 |
+
{
|
| 1757 |
+
"model": "LiquidAI--LFM2.5-1.2B-Instruct",
|
| 1758 |
+
"panel": "alfworld",
|
| 1759 |
+
"phase": "qualification",
|
| 1760 |
+
"start": 1793541592.4816368,
|
| 1761 |
+
"end": 1793541595.4829729,
|
| 1762 |
+
"seconds": 3.001336097717285,
|
| 1763 |
+
"termination_event": "JOB_COMPLETE",
|
| 1764 |
+
"samples": 1,
|
| 1765 |
+
"mean_UTL": 28.0,
|
| 1766 |
+
"mean_W": 90.38,
|
| 1767 |
+
"peak_memory_MiB": 42654.0,
|
| 1768 |
+
"completed_tasks": 2,
|
| 1769 |
+
"completed_tasks_per_second": 0.6663698882378193,
|
| 1770 |
+
"completed_task_output_tokens": 71
|
| 1771 |
+
},
|
| 1772 |
+
{
|
| 1773 |
+
"model": "LiquidAI--LFM2.5-1.2B-Instruct",
|
| 1774 |
+
"panel": "bfcl",
|
| 1775 |
+
"phase": "qualification",
|
| 1776 |
+
"start": 1793540727.6986325,
|
| 1777 |
+
"end": 1793540730.6999526,
|
| 1778 |
+
"seconds": 3.0013201236724854,
|
| 1779 |
+
"termination_event": "JOB_COMPLETE",
|
| 1780 |
+
"samples": 1,
|
| 1781 |
+
"mean_UTL": 0.0,
|
| 1782 |
+
"mean_W": 97.36,
|
| 1783 |
+
"peak_memory_MiB": 42622.0,
|
| 1784 |
+
"completed_tasks": 2,
|
| 1785 |
+
"completed_tasks_per_second": 0.6663734348846312,
|
| 1786 |
+
"completed_task_output_tokens": 152
|
| 1787 |
+
},
|
| 1788 |
+
{
|
| 1789 |
+
"model": "openbmb--MiniCPM5-1B-SFT",
|
| 1790 |
+
"panel": "alfworld",
|
| 1791 |
+
"phase": "qualification",
|
| 1792 |
+
"start": 1793541613.8396358,
|
| 1793 |
+
"end": 1793541616.8407776,
|
| 1794 |
+
"seconds": 3.0011417865753174,
|
| 1795 |
+
"termination_event": "JOB_COMPLETE",
|
| 1796 |
+
"samples": 1,
|
| 1797 |
+
"mean_UTL": 0.0,
|
| 1798 |
+
"mean_W": 54.94,
|
| 1799 |
+
"peak_memory_MiB": 42840.0,
|
| 1800 |
+
"completed_tasks": 2,
|
| 1801 |
+
"completed_tasks_per_second": 0.6664130328484924,
|
| 1802 |
+
"completed_task_output_tokens": 32
|
| 1803 |
+
},
|
| 1804 |
+
{
|
| 1805 |
+
"model": "openbmb--MiniCPM5-1B-SFT",
|
| 1806 |
+
"panel": "alfworld",
|
| 1807 |
+
"phase": "benchmark",
|
| 1808 |
+
"start": 1793541616.845832,
|
| 1809 |
+
"end": 1793541640.8553474,
|
| 1810 |
+
"seconds": 24.009515285491943,
|
| 1811 |
+
"termination_event": "JOB_COMPLETE",
|
| 1812 |
+
"samples": 5,
|
| 1813 |
+
"mean_UTL": 29.6,
|
| 1814 |
+
"mean_W": 148.992,
|
| 1815 |
+
"peak_memory_MiB": 42844.0,
|
| 1816 |
+
"completed_tasks": 32,
|
| 1817 |
+
"completed_tasks_per_second": 1.332804915863354,
|
| 1818 |
+
"completed_task_output_tokens": 13025
|
| 1819 |
+
},
|
| 1820 |
+
{
|
| 1821 |
+
"model": "openbmb--MiniCPM5-1B-SFT",
|
| 1822 |
+
"panel": "alfworld",
|
| 1823 |
+
"phase": "benchmark",
|
| 1824 |
+
"start": 1793541640.8593676,
|
| 1825 |
+
"end": 1793541667.874898,
|
| 1826 |
+
"seconds": 27.015530347824097,
|
| 1827 |
+
"termination_event": "JOB_COMPLETE",
|
| 1828 |
+
"samples": 5,
|
| 1829 |
+
"mean_UTL": 41.2,
|
| 1830 |
+
"mean_W": 166.852,
|
| 1831 |
+
"peak_memory_MiB": 42844.0,
|
| 1832 |
+
"completed_tasks": 32,
|
| 1833 |
+
"completed_tasks_per_second": 1.184503860853406,
|
| 1834 |
+
"completed_task_output_tokens": 13270
|
| 1835 |
+
},
|
| 1836 |
+
{
|
| 1837 |
+
"model": "openbmb--MiniCPM5-1B-SFT",
|
| 1838 |
+
"panel": "alfworld",
|
| 1839 |
+
"phase": "benchmark",
|
| 1840 |
+
"start": 1793541667.8780117,
|
| 1841 |
+
"end": 1793541695.8888903,
|
| 1842 |
+
"seconds": 28.010878562927246,
|
| 1843 |
+
"termination_event": "JOB_COMPLETE",
|
| 1844 |
+
"samples": 6,
|
| 1845 |
+
"mean_UTL": 23.333333333333332,
|
| 1846 |
+
"mean_W": 142.46166666666667,
|
| 1847 |
+
"peak_memory_MiB": 42844.0,
|
| 1848 |
+
"completed_tasks": 32,
|
| 1849 |
+
"completed_tasks_per_second": 1.1424132923253756,
|
| 1850 |
+
"completed_task_output_tokens": 14209
|
| 1851 |
+
},
|
| 1852 |
+
{
|
| 1853 |
+
"model": "openbmb--MiniCPM5-1B-SFT",
|
| 1854 |
+
"panel": "alfworld",
|
| 1855 |
+
"phase": "benchmark",
|
| 1856 |
+
"start": 1793541695.8929152,
|
| 1857 |
+
"end": 1793541723.9094083,
|
| 1858 |
+
"seconds": 28.01649308204651,
|
| 1859 |
+
"termination_event": "JOB_COMPLETE",
|
| 1860 |
+
"samples": 5,
|
| 1861 |
+
"mean_UTL": 36.6,
|
| 1862 |
+
"mean_W": 163.286,
|
| 1863 |
+
"peak_memory_MiB": 42844.0,
|
| 1864 |
+
"completed_tasks": 32,
|
| 1865 |
+
"completed_tasks_per_second": 1.1421843521345716,
|
| 1866 |
+
"completed_task_output_tokens": 12751
|
| 1867 |
+
},
|
| 1868 |
+
{
|
| 1869 |
+
"model": "openbmb--MiniCPM5-1B-SFT",
|
| 1870 |
+
"panel": "alfworld",
|
| 1871 |
+
"phase": "benchmark",
|
| 1872 |
+
"start": 1793541723.9135094,
|
| 1873 |
+
"end": 1793541731.9154322,
|
| 1874 |
+
"seconds": 8.001922845840454,
|
| 1875 |
+
"termination_event": "JOB_COMPLETE",
|
| 1876 |
+
"samples": 2,
|
| 1877 |
+
"mean_UTL": 48.5,
|
| 1878 |
+
"mean_W": 167.005,
|
| 1879 |
+
"peak_memory_MiB": 42844.0,
|
| 1880 |
+
"completed_tasks": 6,
|
| 1881 |
+
"completed_tasks_per_second": 0.7498197765202035,
|
| 1882 |
+
"completed_task_output_tokens": 2243
|
| 1883 |
+
},
|
| 1884 |
+
{
|
| 1885 |
+
"model": "openbmb--MiniCPM5-1B-SFT",
|
| 1886 |
+
"panel": "bfcl",
|
| 1887 |
+
"phase": "qualification",
|
| 1888 |
+
"start": 1793540760.728207,
|
| 1889 |
+
"end": 1793540763.7294846,
|
| 1890 |
+
"seconds": 3.001277446746826,
|
| 1891 |
+
"termination_event": "JOB_COMPLETE",
|
| 1892 |
+
"samples": 0,
|
| 1893 |
+
"mean_UTL": null,
|
| 1894 |
+
"mean_W": null,
|
| 1895 |
+
"peak_memory_MiB": null,
|
| 1896 |
+
"completed_tasks": 2,
|
| 1897 |
+
"completed_tasks_per_second": 0.6663829104396395,
|
| 1898 |
+
"completed_task_output_tokens": 204
|
| 1899 |
+
},
|
| 1900 |
+
{
|
| 1901 |
+
"model": "openbmb--MiniCPM5-1B",
|
| 1902 |
+
"panel": "alfworld",
|
| 1903 |
+
"phase": "qualification",
|
| 1904 |
+
"start": 1793541748.103092,
|
| 1905 |
+
"end": 1793541751.104105,
|
| 1906 |
+
"seconds": 3.0010130405426025,
|
| 1907 |
+
"termination_event": "JOB_COMPLETE",
|
| 1908 |
+
"samples": 1,
|
| 1909 |
+
"mean_UTL": 0.0,
|
| 1910 |
+
"mean_W": 71.83,
|
| 1911 |
+
"peak_memory_MiB": 42840.0,
|
| 1912 |
+
"completed_tasks": 2,
|
| 1913 |
+
"completed_tasks_per_second": 0.66644162253903,
|
| 1914 |
+
"completed_task_output_tokens": 44
|
| 1915 |
+
},
|
| 1916 |
+
{
|
| 1917 |
+
"model": "openbmb--MiniCPM5-1B",
|
| 1918 |
+
"panel": "bfcl",
|
| 1919 |
+
"phase": "qualification",
|
| 1920 |
+
"start": 1793540789.273598,
|
| 1921 |
+
"end": 1793540792.2749584,
|
| 1922 |
+
"seconds": 3.0013604164123535,
|
| 1923 |
+
"termination_event": "JOB_COMPLETE",
|
| 1924 |
+
"samples": 1,
|
| 1925 |
+
"mean_UTL": 0.0,
|
| 1926 |
+
"mean_W": 65.22,
|
| 1927 |
+
"peak_memory_MiB": 42666.0,
|
| 1928 |
+
"completed_tasks": 2,
|
| 1929 |
+
"completed_tasks_per_second": 0.6663644889375466,
|
| 1930 |
+
"completed_task_output_tokens": 59
|
| 1931 |
+
},
|
| 1932 |
+
{
|
| 1933 |
+
"model": "openbmb--MiniCPM5-1B",
|
| 1934 |
+
"panel": "bfcl",
|
| 1935 |
+
"phase": "benchmark",
|
| 1936 |
+
"start": 1793540792.2786386,
|
| 1937 |
+
"end": 1793540800.2803898,
|
| 1938 |
+
"seconds": 8.001751184463501,
|
| 1939 |
+
"termination_event": "JOB_COMPLETE",
|
| 1940 |
+
"samples": 1,
|
| 1941 |
+
"mean_UTL": 100.0,
|
| 1942 |
+
"mean_W": 401.02,
|
| 1943 |
+
"peak_memory_MiB": 42814.0,
|
| 1944 |
+
"completed_tasks": 32,
|
| 1945 |
+
"completed_tasks_per_second": 3.9991245993917426,
|
| 1946 |
+
"completed_task_output_tokens": 6457
|
| 1947 |
+
},
|
| 1948 |
+
{
|
| 1949 |
+
"model": "openbmb--MiniCPM5-1B",
|
| 1950 |
+
"panel": "bfcl",
|
| 1951 |
+
"phase": "benchmark",
|
| 1952 |
+
"start": 1793540800.2837431,
|
| 1953 |
+
"end": 1793540811.285404,
|
| 1954 |
+
"seconds": 11.001660823822021,
|
| 1955 |
+
"termination_event": "JOB_COMPLETE",
|
| 1956 |
+
"samples": 3,
|
| 1957 |
+
"mean_UTL": 33.333333333333336,
|
| 1958 |
+
"mean_W": 179.45666666666668,
|
| 1959 |
+
"peak_memory_MiB": 42814.0,
|
| 1960 |
+
"completed_tasks": 32,
|
| 1961 |
+
"completed_tasks_per_second": 2.90865174926226,
|
| 1962 |
+
"completed_task_output_tokens": 10438
|
| 1963 |
+
},
|
| 1964 |
+
{
|
| 1965 |
+
"model": "openbmb--MiniCPM5-1B",
|
| 1966 |
+
"panel": "bfcl",
|
| 1967 |
+
"phase": "benchmark",
|
| 1968 |
+
"start": 1793540811.2868805,
|
| 1969 |
+
"end": 1793540823.2890596,
|
| 1970 |
+
"seconds": 12.002179145812988,
|
| 1971 |
+
"termination_event": "JOB_COMPLETE",
|
| 1972 |
+
"samples": 2,
|
| 1973 |
+
"mean_UTL": 100.0,
|
| 1974 |
+
"mean_W": 373.355,
|
| 1975 |
+
"peak_memory_MiB": 42814.0,
|
| 1976 |
+
"completed_tasks": 32,
|
| 1977 |
+
"completed_tasks_per_second": 2.666182499964045,
|
| 1978 |
+
"completed_task_output_tokens": 17539
|
| 1979 |
+
},
|
| 1980 |
+
{
|
| 1981 |
+
"model": "openbmb--MiniCPM5-1B",
|
| 1982 |
+
"panel": "bfcl",
|
| 1983 |
+
"phase": "benchmark",
|
| 1984 |
+
"start": 1793540823.2916431,
|
| 1985 |
+
"end": 1793540832.293638,
|
| 1986 |
+
"seconds": 9.001994848251343,
|
| 1987 |
+
"termination_event": "JOB_COMPLETE",
|
| 1988 |
+
"samples": 2,
|
| 1989 |
+
"mean_UTL": 50.0,
|
| 1990 |
+
"mean_W": 299.83000000000004,
|
| 1991 |
+
"peak_memory_MiB": 42814.0,
|
| 1992 |
+
"completed_tasks": 32,
|
| 1993 |
+
"completed_tasks_per_second": 3.554767641998381,
|
| 1994 |
+
"completed_task_output_tokens": 9618
|
| 1995 |
+
},
|
| 1996 |
+
{
|
| 1997 |
+
"model": "openbmb--MiniCPM5-1B",
|
| 1998 |
+
"panel": "bfcl",
|
| 1999 |
+
"phase": "benchmark",
|
| 2000 |
+
"start": 1793540832.2977865,
|
| 2001 |
+
"end": 1793540851.3013995,
|
| 2002 |
+
"seconds": 19.003612995147705,
|
| 2003 |
+
"termination_event": "JOB_COMPLETE",
|
| 2004 |
+
"samples": 4,
|
| 2005 |
+
"mean_UTL": 75.0,
|
| 2006 |
+
"mean_W": 300.2025,
|
| 2007 |
+
"peak_memory_MiB": 42814.0,
|
| 2008 |
+
"completed_tasks": 32,
|
| 2009 |
+
"completed_tasks_per_second": 1.6838903217072845,
|
| 2010 |
+
"completed_task_output_tokens": 22502
|
| 2011 |
+
},
|
| 2012 |
+
{
|
| 2013 |
+
"model": "openbmb--MiniCPM5-1B",
|
| 2014 |
+
"panel": "bfcl",
|
| 2015 |
+
"phase": "benchmark",
|
| 2016 |
+
"start": 1793540851.3050878,
|
| 2017 |
+
"end": 1793540871.3083003,
|
| 2018 |
+
"seconds": 20.003212451934814,
|
| 2019 |
+
"termination_event": "JOB_COMPLETE",
|
| 2020 |
+
"samples": 4,
|
| 2021 |
+
"mean_UTL": 75.0,
|
| 2022 |
+
"mean_W": 342.4175,
|
| 2023 |
+
"peak_memory_MiB": 42814.0,
|
| 2024 |
+
"completed_tasks": 32,
|
| 2025 |
+
"completed_tasks_per_second": 1.5997430451179753,
|
| 2026 |
+
"completed_task_output_tokens": 38655
|
| 2027 |
+
},
|
| 2028 |
+
{
|
| 2029 |
+
"model": "openbmb--MiniCPM5-1B",
|
| 2030 |
+
"panel": "bfcl",
|
| 2031 |
+
"phase": "benchmark",
|
| 2032 |
+
"start": 1793540871.3096328,
|
| 2033 |
+
"end": 1793540884.3120797,
|
| 2034 |
+
"seconds": 13.00244688987732,
|
| 2035 |
+
"termination_event": "JOB_COMPLETE",
|
| 2036 |
+
"samples": 2,
|
| 2037 |
+
"mean_UTL": 100.0,
|
| 2038 |
+
"mean_W": 306.625,
|
| 2039 |
+
"peak_memory_MiB": 42814.0,
|
| 2040 |
+
"completed_tasks": 8,
|
| 2041 |
+
"completed_tasks_per_second": 0.6152688080755145,
|
| 2042 |
+
"completed_task_output_tokens": 9573
|
| 2043 |
+
},
|
| 2044 |
+
{
|
| 2045 |
+
"model": "Qwen--Qwen2.5-1.5B-Instruct",
|
| 2046 |
+
"panel": "alfworld",
|
| 2047 |
+
"phase": "qualification",
|
| 2048 |
+
"start": 1793541768.135486,
|
| 2049 |
+
"end": 1793541771.1361818,
|
| 2050 |
+
"seconds": 3.0006959438323975,
|
| 2051 |
+
"termination_event": "JOB_COMPLETE",
|
| 2052 |
+
"samples": 1,
|
| 2053 |
+
"mean_UTL": 0.0,
|
| 2054 |
+
"mean_W": 106.78,
|
| 2055 |
+
"peak_memory_MiB": 42676.0,
|
| 2056 |
+
"completed_tasks": 2,
|
| 2057 |
+
"completed_tasks_per_second": 0.666512048350244,
|
| 2058 |
+
"completed_task_output_tokens": 83
|
| 2059 |
+
},
|
| 2060 |
+
{
|
| 2061 |
+
"model": "Qwen--Qwen2.5-1.5B-Instruct",
|
| 2062 |
+
"panel": "bfcl",
|
| 2063 |
+
"phase": "qualification",
|
| 2064 |
+
"start": 1793540914.90442,
|
| 2065 |
+
"end": 1793540917.9056945,
|
| 2066 |
+
"seconds": 3.001274585723877,
|
| 2067 |
+
"termination_event": "JOB_COMPLETE",
|
| 2068 |
+
"samples": 1,
|
| 2069 |
+
"mean_UTL": 0.0,
|
| 2070 |
+
"mean_W": 137.53,
|
| 2071 |
+
"peak_memory_MiB": 42640.0,
|
| 2072 |
+
"completed_tasks": 2,
|
| 2073 |
+
"completed_tasks_per_second": 0.6663835456820157,
|
| 2074 |
+
"completed_task_output_tokens": 168
|
| 2075 |
+
},
|
| 2076 |
+
{
|
| 2077 |
+
"model": "HuggingFaceTB--SmolLM2-1.7B-Instruct",
|
| 2078 |
+
"panel": "alfworld",
|
| 2079 |
+
"phase": "qualification",
|
| 2080 |
+
"start": 1793541786.5357795,
|
| 2081 |
+
"end": 1793541789.5370362,
|
| 2082 |
+
"seconds": 3.0012567043304443,
|
| 2083 |
+
"termination_event": "JOB_COMPLETE",
|
| 2084 |
+
"samples": 0,
|
| 2085 |
+
"mean_UTL": null,
|
| 2086 |
+
"mean_W": null,
|
| 2087 |
+
"peak_memory_MiB": null,
|
| 2088 |
+
"completed_tasks": 2,
|
| 2089 |
+
"completed_tasks_per_second": 0.666387515974307,
|
| 2090 |
+
"completed_task_output_tokens": 160
|
| 2091 |
+
},
|
| 2092 |
+
{
|
| 2093 |
+
"model": "HuggingFaceTB--SmolLM2-1.7B-Instruct",
|
| 2094 |
+
"panel": "bfcl",
|
| 2095 |
+
"phase": "qualification",
|
| 2096 |
+
"start": 1793540942.3241732,
|
| 2097 |
+
"end": 1793540946.325417,
|
| 2098 |
+
"seconds": 4.001243829727173,
|
| 2099 |
+
"termination_event": "JOB_COMPLETE",
|
| 2100 |
+
"samples": 1,
|
| 2101 |
+
"mean_UTL": 0.0,
|
| 2102 |
+
"mean_W": 68.35,
|
| 2103 |
+
"peak_memory_MiB": 42594.0,
|
| 2104 |
+
"completed_tasks": 2,
|
| 2105 |
+
"completed_tasks_per_second": 0.49984456961633633,
|
| 2106 |
+
"completed_task_output_tokens": 456
|
| 2107 |
+
}
|
| 2108 |
+
]
|
| 2109 |
+
},
|
| 2110 |
+
"warnings": [
|
| 2111 |
+
"ALF134 is seen regression; BFCL200 is a repeated evaluation after an interface revision, not fresh held-out selection.",
|
| 2112 |
+
"Qualification is a fixed two-fixture interface check per domain, not general task mastery; unqualified or unfinished positions remain unscored.",
|
| 2113 |
+
"Text function-calling protocol with native chat templates; not native FC API or official unrestricted leaderboard scores.",
|
| 2114 |
+
"External vLLM version changed from 0.26.0 to 0.29.0; old/new differences do not isolate a single interface change.",
|
| 2115 |
+
"ScienceWorld and conventional five benchmarks were not rerun. No new RSI proposals or accepted/installed successors."
|
| 2116 |
+
],
|
| 2117 |
+
"provenance": {
|
| 2118 |
+
"summary_sha256": "fc392ff15247ba7c0818079bf81f7bab80f4f1a91b9c061002690716fd7b7c37",
|
| 2119 |
+
"plan_sha256": "2cdf3379a044ae36f5965318861790e7f09e3a59f2b5b8d11c79772974a8355b",
|
| 2120 |
+
"source_candidate_sha256": "0fc72dda772a3e84aff131aad28b9a832f0b56740ea820e99321de0bfb8f52a3",
|
| 2121 |
+
"collector_sha256": "2dbd2405bce0a1137cd531b7a39accb30fef951b24166e996aa03d2527029be0",
|
| 2122 |
+
"task_evidence_sha256": "b9097fbb81f95f52c76117e44f64c4ec9ea90c6501aa2b6b5e5d23db6b096ecc",
|
| 2123 |
+
"qualification_evidence_sha256": "ec496e5cc873f3a840c9c3fa12c28ba7eb290f93e0aa07825e4a913e62bee60e",
|
| 2124 |
+
"original_public_agentic_json_sha256": "c6ab7f3dea0946d5909f7d2755dfd92b4d0a40794c7ea466bb1f61a80d537909"
|
| 2125 |
+
}
|
| 2126 |
}
|
| 2127 |
}
|