JunyiZou commited on
Commit
98415e3
·
verified ·
1 Parent(s): f492228

Upload folder using huggingface_hub

Browse files
Files changed (3) hide show
  1. MANIFEST.json +4 -4
  2. README.md +29 -2
  3. results/agentic.json +1310 -0
MANIFEST.json CHANGED
@@ -18,8 +18,8 @@
18
  },
19
  {
20
  "path": "README.md",
21
- "bytes": 8414,
22
- "sha256": "e0c223c20e9bd74abe1e95fe0a97662883d3fbd3bdbd228cdf4d960f6de04f15"
23
  },
24
  {
25
  "path": "chat_template.jinja",
@@ -68,8 +68,8 @@
68
  },
69
  {
70
  "path": "results/agentic.json",
71
- "bytes": 28730,
72
- "sha256": "c6ab7f3dea0946d5909f7d2755dfd92b4d0a40794c7ea466bb1f61a80d537909"
73
  },
74
  {
75
  "path": "results/benchmarks.json",
 
18
  },
19
  {
20
  "path": "README.md",
21
+ "bytes": 11417,
22
+ "sha256": "d5d8c34c37dac0f27f5f33ef13d965868b50186f65002b2e98ccf776e6c30359"
23
  },
24
  {
25
  "path": "chat_template.jinja",
 
68
  },
69
  {
70
  "path": "results/agentic.json",
71
+ "bytes": 74744,
72
+ "sha256": "13f4bc09d1f17f693cec480b64d14bcc13b70b72a4696e93a7860c0a90c7cbcf"
73
  },
74
  {
75
  "path": "results/benchmarks.json",
README.md CHANGED
@@ -57,7 +57,32 @@ The host supplies task observations and real feedback through the [task interfac
57
 
58
  Accuracy percentages on the complete 2,376 / 1,172 / 10,042 / 1,267 / 14,042 example panels. Evaluation uses zero-shot completion likelihood, empty M, fixed Φ, no chat template, and FP32 execution of the saved weights. ARC and HellaSwag use character-normalized likelihood; WinoGrande uses common-suffix likelihood; MMLU is micro accuracy over 57 subjects. These previously evaluated panels are regression comparisons. Qwen and SmolLM2 have more parameters. Exact counts, protocol and model identities are in [benchmark results](results/benchmarks.json).
59
 
60
- ## Agentic diagnostics under a custom host
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
61
 
62
  **Interface audit:** these scores expose substantial command-format failures in this custom host and should not be used as a general Agentic ranking or evidence of an MMLA architecture advantage. SmolLM3 executed no BFCL tool calls: 453 of its 734 replies meet the host schema after removing only Markdown fences, which the host rejected. SmolLM2 reached the 64-token output limit in 452 of 734 BFCL replies. In ALFWorld, both Smol models received `Nothing happens.` for every action; MiniCPM5-1B-SFT often copied literal placeholders such as `(receptacle)` into commands. These are observed failure modes, not estimates of corrected scores. All recorded scores are retained below.
63
 
@@ -77,6 +102,8 @@ Orbit was evaluated on **424 tasks across three environments**, alongside eight
77
 
78
  ALFWorld uses the 134-task **seen regression** panel. ScienceWorld averages final scores across 30 task classes with three official test variations each; native scores range from −100 to 100, including negative outcomes. Orbit completed the task objective in 0/90 ScienceWorld episodes. BFCL covers the 200-case **multi_turn_base** category. These results describe this finite host configuration; the panels have different scoring rules. SmolLM2 has five unscored ScienceWorld context-limit failures, so its 90-task mean is left blank. Exact values, model revisions and protocol details are in [Agentic results](results/agentic.json).
79
 
 
 
80
  ## RSI proposal and installation status
81
 
82
  The same Orbit weights were used with three proposal RNG seeds (17, 29, 43), each producing 40 proposals. The best eligible candidate in each development window tied or underperformed its parent. No candidate was locked for independent admission, and no successor was accepted or installed.
@@ -88,7 +115,7 @@ The same Orbit weights were used with three proposal RNG seeds (17, 29, 43), eac
88
  | 20 | 60 | 0 | 0 |
89
  | 40 | 120 | 0 | 0 |
90
 
91
- All twelve checkpoints retain the same installed manager. Its one task evaluation is reported above; repeating those scores would not measure a performance trajectory across improved generations. No new RSI search was run on ScienceWorld or BFCL.
92
 
93
  M reads and RTMA writes executed during task evaluation. Numerical Φ stayed fixed at its trained initialization. This series demonstrates proposal execution and memory use, with no accepted recursive improvement. Checkpoint identities and the interface audit are included in [Agentic results](results/agentic.json).
94
 
 
57
 
58
  Accuracy percentages on the complete 2,376 / 1,172 / 10,042 / 1,267 / 14,042 example panels. Evaluation uses zero-shot completion likelihood, empty M, fixed Φ, no chat template, and FP32 execution of the saved weights. ARC and HellaSwag use character-normalized likelihood; WinoGrande uses common-suffix likelihood; MMLU is micro accuracy over 57 subjects. These previously evaluated panels are regression comparisons. Qwen and SmolLM2 have more parameters. Exact counts, protocol and model identities are in [benchmark results](results/benchmarks.json).
59
 
60
+ ## Agentic evaluation: revised interface
61
+
62
+ Nine fixed checkpoints entered a finite reevaluation through the shared text-action host after correcting command instructions and output-format compatibility. Each model first receives two synthetic interface checks per environment. A check requires a legal first action and a second action using newly returned feedback. Both checks must pass before that environment is evaluated. Each fixture requires two specified single-call steps within four replies. BFCL qualification rejects a multi-call reply as a whole, even when its first call is valid; unlike the benchmark executor, it does not partially execute that reply. Failure concerns this strict synthetic check and does not mean that every produced call was illegal.
63
+
64
+ | Model | ALF interface checks | ALFWorld success | BFCL interface checks | BFCL success |
65
+ |---|---:|---:|---:|---:|
66
+ | Orbit-1B (M + RTMA) | 2/2 | 26.12% (35/134) | 0/2 | — (not qualified; 200 unscored) |
67
+ | MiniCPM5-1B-SFT | 2/2 | 0.00% (0/134) | 0/2 | — (not qualified; 200 unscored) |
68
+ | MiniCPM5-1B | 1/2 | — (not qualified; 134 unscored) | 2/2 | 0.00% (0/200) |
69
+ | Qwen2.5-1.5B-Instruct | 0/2 | — (not qualified; 134 unscored) | 0/2 | — (not qualified; 200 unscored) |
70
+ | SmolLM2-1.7B-Instruct | 0/2 | — (not qualified; 134 unscored) | 0/2 | — (not qualified; 200 unscored) |
71
+ | LFM2.5-1.2B-Instruct | 0/2 | — (not qualified; 134 unscored) | 0/2 | — (not qualified; 200 unscored) |
72
+ | SmolLM3-3B | 0/2 | — (not qualified; 134 unscored) | 0/2 | — (not qualified; 200 unscored) |
73
+ | Qwen3.5-4B | 2/2 | 3.73% (5/134) | 0/2 | — (not qualified; 200 unscored) |
74
+ | Qwen3.5-9B | 2/2 | 24.63% (33/134) | 2/2 | 38.00% (76/200) |
75
+
76
+ The finite allocation scored 936 of 3,006 planned model–task positions; 2,070 remain unscored. A dash is not a zero. Primary scores require the complete 134-task ALFWorld or 200-task BFCL panel; qualification failures and unfinished runs retain their original denominators.
77
+
78
+ ALFWorld uses the same seen-regression panel. BFCL uses the same multi_turn_base200 cases, evaluated again after the interface revision. Both retain a 64-reply task limit; reply budgets are 64 tokens for ALFWorld and 512 for BFCL. Orbit uses fixed trained Φ with its eight-slot M and installed RTMA rule. Ordinary models use their native chat templates and conversation history. Scores describe this finite text-action configuration, not unrestricted official leaderboards or isolated causal memory gains. Local external inference also changed from vLLM 0.26.0 to 0.29.0, so old/new differences do not isolate one fix.
79
+
80
+ ScienceWorld was not rerun; its original-protocol results and all earlier scores are retained below and in [Agentic results](results/agentic.json). That file also records qualification status, exact coverage, model revisions, performance measurements and evidence hashes.
81
+
82
+ <details>
83
+ <summary>Original protocol scores and interface diagnosis</summary>
84
+
85
+ ### Original protocol diagnostics
86
 
87
  **Interface audit:** these scores expose substantial command-format failures in this custom host and should not be used as a general Agentic ranking or evidence of an MMLA architecture advantage. SmolLM3 executed no BFCL tool calls: 453 of its 734 replies meet the host schema after removing only Markdown fences, which the host rejected. SmolLM2 reached the 64-token output limit in 452 of 734 BFCL replies. In ALFWorld, both Smol models received `Nothing happens.` for every action; MiniCPM5-1B-SFT often copied literal placeholders such as `(receptacle)` into commands. These are observed failure modes, not estimates of corrected scores. All recorded scores are retained below.
88
 
 
102
 
103
  ALFWorld uses the 134-task **seen regression** panel. ScienceWorld averages final scores across 30 task classes with three official test variations each; native scores range from −100 to 100, including negative outcomes. Orbit completed the task objective in 0/90 ScienceWorld episodes. BFCL covers the 200-case **multi_turn_base** category. These results describe this finite host configuration; the panels have different scoring rules. SmolLM2 has five unscored ScienceWorld context-limit failures, so its 90-task mean is left blank. Exact values, model revisions and protocol details are in [Agentic results](results/agentic.json).
104
 
105
+ </details>
106
+
107
  ## RSI proposal and installation status
108
 
109
  The same Orbit weights were used with three proposal RNG seeds (17, 29, 43), each producing 40 proposals. The best eligible candidate in each development window tied or underperformed its parent. No candidate was locked for independent admission, and no successor was accepted or installed.
 
115
  | 20 | 60 | 0 | 0 |
116
  | 40 | 120 | 0 | 0 |
117
 
118
+ All twelve checkpoints retain the same installed manager. The evaluations above use that same installed manager; repeating their scores at each proposal checkpoint would not measure a performance trajectory across improved generations. No new RSI search was run on ScienceWorld or BFCL.
119
 
120
  M reads and RTMA writes executed during task evaluation. Numerical Φ stayed fixed at its trained initialization. This series demonstrates proposal execution and memory use, with no accepted recursive improvement. Checkpoint identities and the interface audit are included in [Agentic results](results/agentic.json).
121
 
results/agentic.json CHANGED
@@ -813,5 +813,1315 @@
813
  "ALFWorld panel was previously used for regression; Orbit had task-specific training. These scores do not isolate M/RTMA causal benefit.",
814
  "Changing the parser changes later stateful observations; fixed trajectories cannot be reinterpreted as corrected task scores."
815
  ]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
816
  }
817
  }
 
813
  "ALFWorld panel was previously used for regression; Orbit had task-specific training. These scores do not isolate M/RTMA causal benefit.",
814
  "Changing the parser changes later stateful observations; fixed trajectories cannot be reinterpreted as corrected task scores."
815
  ]
816
+ },
817
+ "interface_revision": {
818
+ "schema": "orbit-public-agentic-interface-v2",
819
+ "allocation_status": "COMPLETE",
820
+ "model_weight_sha256": "62d83295cb154ce7869c60602e66677b84cde6f33ab886101752caa6805af276",
821
+ "installed_manager_sha256": "2b44013ee892552f14b846d2a1315e09990f4751d4255a9ff48295cc75fc0081",
822
+ "protocol": {
823
+ "host": "OhMyPi-based shared text-action host",
824
+ "interface_version": 2,
825
+ "model_seed": 17,
826
+ "max_model_replies_per_task": 64,
827
+ "max_generated_tokens_per_reply": {
828
+ "alfworld": 64,
829
+ "bfcl": 512
830
+ },
831
+ "qualification": "Two fixed synthetic fixtures per domain, each requiring two specified single-call steps within four replies; both fixtures must pass. BFCL qualification rejects an entire multi-call reply without partial execution, including when its first call is valid; this differs from the real benchmark executor. No test answers; qualification state discarded. Failure is not evidence of inability to generate any legal call.",
832
+ "bfcl": "Official classic/plaintext/Python prompting and executor/checker. Exact complete JSON, Markdown fence and tool_call wrappers accepted without guessing parameters or repairing incomplete output.",
833
+ "alfworld": "Explicit command grammar, placeholder replacement and action preconditions; no oracle valid-action list.",
834
+ "orbit": "Same trained fixed Phi and eight-slot M/RTMA manager; resets between tasks; no new training or RSI proposals.",
835
+ "external_models": "Exact original weights and native chat templates, ordinary conversation history, greedy decoding, thinking disabled; local vLLM0.29.0 instead of original0.26.0.",
836
+ "qualification_failure": "Corresponding fixed benchmark positions remain unscored; no prompt retry or zero imputation.",
837
+ "scienceworld": "Original protocol results retained separately; not rerun in revision2."
838
+ },
839
+ "coverage": {
840
+ "expected_model_task_positions": 3006,
841
+ "scored": 936,
842
+ "unscored": 2070,
843
+ "qualification_expected": 36,
844
+ "qualification_completed": 36
845
+ },
846
+ "models": {
847
+ "Orbit": {
848
+ "label": "Orbit-1B (M + RTMA)",
849
+ "panels": {
850
+ "alfworld": {
851
+ "status": "COMPLETE",
852
+ "expected": 134,
853
+ "completed": 134,
854
+ "unscored": 0,
855
+ "primary": 0.26119402985074625,
856
+ "success_count": 35,
857
+ "status_counts": {
858
+ "COMPLETE": 134
859
+ },
860
+ "returned_work": {
861
+ "replies": 6570,
862
+ "input_tokens": 3412117,
863
+ "output_tokens": 45867,
864
+ "length_finishes": 0,
865
+ "nothing_happens": 5927
866
+ },
867
+ "qualification": {
868
+ "status": "QUALIFIED",
869
+ "passed": 2,
870
+ "expected": 2,
871
+ "protocol": "e53aaa1a7a78a80a65e49f5abd2290ef520c05e14ed17fdccbfb2039010cd0d7"
872
+ }
873
+ },
874
+ "bfcl": {
875
+ "status": "NOT_QUALIFIED",
876
+ "expected": 200,
877
+ "completed": 0,
878
+ "unscored": 200,
879
+ "primary": null,
880
+ "success_count": 0,
881
+ "status_counts": {
882
+ "NOT_QUALIFIED": 200
883
+ },
884
+ "returned_work": {},
885
+ "qualification": {
886
+ "status": "NOT_QUALIFIED",
887
+ "passed": 0,
888
+ "expected": 2,
889
+ "protocol": "67df3a095a6a00813f56890bbf75a2f5a01368585dee310e1714344d140aa631"
890
+ }
891
+ }
892
+ }
893
+ },
894
+ "openbmb--MiniCPM5-1B-SFT": {
895
+ "label": "MiniCPM5-1B-SFT",
896
+ "panels": {
897
+ "alfworld": {
898
+ "status": "COMPLETE",
899
+ "expected": 134,
900
+ "completed": 134,
901
+ "unscored": 0,
902
+ "primary": 0.0,
903
+ "success_count": 0,
904
+ "status_counts": {
905
+ "COMPLETE": 134
906
+ },
907
+ "returned_work": {
908
+ "replies": 8576,
909
+ "input_tokens": 13675913,
910
+ "output_tokens": 55498,
911
+ "length_finishes": 0,
912
+ "nothing_happens": 8283
913
+ },
914
+ "qualification": {
915
+ "status": "QUALIFIED",
916
+ "passed": 2,
917
+ "expected": 2,
918
+ "protocol": "404f9efd664a5dc19e46ec2d862b6a66b0abc6b1bf2e2b778af06e9cbc816fb1"
919
+ }
920
+ },
921
+ "bfcl": {
922
+ "status": "NOT_QUALIFIED",
923
+ "expected": 200,
924
+ "completed": 0,
925
+ "unscored": 200,
926
+ "primary": null,
927
+ "success_count": 0,
928
+ "status_counts": {
929
+ "NOT_QUALIFIED": 200
930
+ },
931
+ "returned_work": {},
932
+ "qualification": {
933
+ "status": "NOT_QUALIFIED",
934
+ "passed": 0,
935
+ "expected": 2,
936
+ "protocol": "2dd41ed3f2610c98ca42ab35ea0f57be5398de86c6ab4279b07e2d11c2d78111"
937
+ }
938
+ }
939
+ }
940
+ },
941
+ "openbmb--MiniCPM5-1B": {
942
+ "label": "MiniCPM5-1B",
943
+ "panels": {
944
+ "alfworld": {
945
+ "status": "NOT_QUALIFIED",
946
+ "expected": 134,
947
+ "completed": 0,
948
+ "unscored": 134,
949
+ "primary": null,
950
+ "success_count": 0,
951
+ "status_counts": {
952
+ "NOT_QUALIFIED": 134
953
+ },
954
+ "returned_work": {},
955
+ "qualification": {
956
+ "status": "NOT_QUALIFIED",
957
+ "passed": 1,
958
+ "expected": 2,
959
+ "protocol": "d04a21375d1b7844d70652ca615574ec9852b78053fde67b84f20a4e82d39aab"
960
+ }
961
+ },
962
+ "bfcl": {
963
+ "status": "COMPLETE",
964
+ "expected": 200,
965
+ "completed": 200,
966
+ "unscored": 0,
967
+ "primary": 0.0,
968
+ "success_count": 0,
969
+ "status_counts": {
970
+ "COMPLETE": 200
971
+ },
972
+ "returned_work": {
973
+ "replies": 3719,
974
+ "input_tokens": 20813620,
975
+ "output_tokens": 114782,
976
+ "length_finishes": 9,
977
+ "invalid_response_turn_end": 127
978
+ },
979
+ "qualification": {
980
+ "status": "QUALIFIED",
981
+ "passed": 2,
982
+ "expected": 2,
983
+ "protocol": "737ee863afaf1ae5b6a8edb4da9ccb96a634f18c7e8972281ed8261e6106c1ab"
984
+ }
985
+ }
986
+ }
987
+ },
988
+ "Qwen--Qwen2.5-1.5B-Instruct": {
989
+ "label": "Qwen2.5-1.5B-Instruct",
990
+ "panels": {
991
+ "alfworld": {
992
+ "status": "NOT_QUALIFIED",
993
+ "expected": 134,
994
+ "completed": 0,
995
+ "unscored": 134,
996
+ "primary": null,
997
+ "success_count": 0,
998
+ "status_counts": {
999
+ "NOT_QUALIFIED": 134
1000
+ },
1001
+ "returned_work": {},
1002
+ "qualification": {
1003
+ "status": "NOT_QUALIFIED",
1004
+ "passed": 0,
1005
+ "expected": 2,
1006
+ "protocol": "7fffba5ba7036643f9ba544fb58e29bdfc522133e9f0fdbcc21da8bce90a10dc"
1007
+ }
1008
+ },
1009
+ "bfcl": {
1010
+ "status": "NOT_QUALIFIED",
1011
+ "expected": 200,
1012
+ "completed": 0,
1013
+ "unscored": 200,
1014
+ "primary": null,
1015
+ "success_count": 0,
1016
+ "status_counts": {
1017
+ "NOT_QUALIFIED": 200
1018
+ },
1019
+ "returned_work": {},
1020
+ "qualification": {
1021
+ "status": "NOT_QUALIFIED",
1022
+ "passed": 0,
1023
+ "expected": 2,
1024
+ "protocol": "2a787dac2dcee864e43b6b05c31aaef830957b0a8410907fbf86a6a0b24ba3fc"
1025
+ }
1026
+ }
1027
+ }
1028
+ },
1029
+ "HuggingFaceTB--SmolLM2-1.7B-Instruct": {
1030
+ "label": "SmolLM2-1.7B-Instruct",
1031
+ "panels": {
1032
+ "alfworld": {
1033
+ "status": "NOT_QUALIFIED",
1034
+ "expected": 134,
1035
+ "completed": 0,
1036
+ "unscored": 134,
1037
+ "primary": null,
1038
+ "success_count": 0,
1039
+ "status_counts": {
1040
+ "NOT_QUALIFIED": 134
1041
+ },
1042
+ "returned_work": {},
1043
+ "qualification": {
1044
+ "status": "NOT_QUALIFIED",
1045
+ "passed": 0,
1046
+ "expected": 2,
1047
+ "protocol": "a90510616603942d47130c40925ad047c4ea6f34dcb0ec8e933e6c4ecfdb58a2"
1048
+ }
1049
+ },
1050
+ "bfcl": {
1051
+ "status": "NOT_QUALIFIED",
1052
+ "expected": 200,
1053
+ "completed": 0,
1054
+ "unscored": 200,
1055
+ "primary": null,
1056
+ "success_count": 0,
1057
+ "status_counts": {
1058
+ "NOT_QUALIFIED": 200
1059
+ },
1060
+ "returned_work": {},
1061
+ "qualification": {
1062
+ "status": "NOT_QUALIFIED",
1063
+ "passed": 0,
1064
+ "expected": 2,
1065
+ "protocol": "fecd5c6842b95f2c074876b3866bbbff45bbae6c152bb45b003687ec1a303700"
1066
+ }
1067
+ }
1068
+ }
1069
+ },
1070
+ "LiquidAI--LFM2.5-1.2B-Instruct": {
1071
+ "label": "LFM2.5-1.2B-Instruct",
1072
+ "panels": {
1073
+ "alfworld": {
1074
+ "status": "NOT_QUALIFIED",
1075
+ "expected": 134,
1076
+ "completed": 0,
1077
+ "unscored": 134,
1078
+ "primary": null,
1079
+ "success_count": 0,
1080
+ "status_counts": {
1081
+ "NOT_QUALIFIED": 134
1082
+ },
1083
+ "returned_work": {},
1084
+ "qualification": {
1085
+ "status": "NOT_QUALIFIED",
1086
+ "passed": 0,
1087
+ "expected": 2,
1088
+ "protocol": "cc5ad2c426e33fe6ffc6d811dd629d1cfb4866f90d91582692f6f04c0df2b96b"
1089
+ }
1090
+ },
1091
+ "bfcl": {
1092
+ "status": "NOT_QUALIFIED",
1093
+ "expected": 200,
1094
+ "completed": 0,
1095
+ "unscored": 200,
1096
+ "primary": null,
1097
+ "success_count": 0,
1098
+ "status_counts": {
1099
+ "NOT_QUALIFIED": 200
1100
+ },
1101
+ "returned_work": {},
1102
+ "qualification": {
1103
+ "status": "NOT_QUALIFIED",
1104
+ "passed": 0,
1105
+ "expected": 2,
1106
+ "protocol": "c5474dec941291bbbde12fc5ef71234bb906cb6bd48cd32029c90c356a8aaa98"
1107
+ }
1108
+ }
1109
+ }
1110
+ },
1111
+ "HuggingFaceTB--SmolLM3-3B": {
1112
+ "label": "SmolLM3-3B",
1113
+ "panels": {
1114
+ "alfworld": {
1115
+ "status": "NOT_QUALIFIED",
1116
+ "expected": 134,
1117
+ "completed": 0,
1118
+ "unscored": 134,
1119
+ "primary": null,
1120
+ "success_count": 0,
1121
+ "status_counts": {
1122
+ "NOT_QUALIFIED": 134
1123
+ },
1124
+ "returned_work": {},
1125
+ "qualification": {
1126
+ "status": "NOT_QUALIFIED",
1127
+ "passed": 0,
1128
+ "expected": 2,
1129
+ "protocol": "25bd0e0b50c3986f130678dfa088fe9ee60049b905835c2e2ab3b3b646cb13cc"
1130
+ }
1131
+ },
1132
+ "bfcl": {
1133
+ "status": "NOT_QUALIFIED",
1134
+ "expected": 200,
1135
+ "completed": 0,
1136
+ "unscored": 200,
1137
+ "primary": null,
1138
+ "success_count": 0,
1139
+ "status_counts": {
1140
+ "NOT_QUALIFIED": 200
1141
+ },
1142
+ "returned_work": {},
1143
+ "qualification": {
1144
+ "status": "NOT_QUALIFIED",
1145
+ "passed": 0,
1146
+ "expected": 2,
1147
+ "protocol": "5bc54d27e7d95e33a2fa5b1b53404000726e94fe494f7acce26f9f244a3f045c"
1148
+ }
1149
+ }
1150
+ }
1151
+ },
1152
+ "Qwen--Qwen3.5-4B": {
1153
+ "label": "Qwen3.5-4B",
1154
+ "panels": {
1155
+ "alfworld": {
1156
+ "status": "COMPLETE",
1157
+ "expected": 134,
1158
+ "completed": 134,
1159
+ "unscored": 0,
1160
+ "primary": 0.03731343283582089,
1161
+ "success_count": 5,
1162
+ "status_counts": {
1163
+ "COMPLETE": 134
1164
+ },
1165
+ "returned_work": {
1166
+ "replies": 8304,
1167
+ "input_tokens": 14675687,
1168
+ "output_tokens": 48740,
1169
+ "length_finishes": 0,
1170
+ "nothing_happens": 4578
1171
+ },
1172
+ "qualification": {
1173
+ "status": "QUALIFIED",
1174
+ "passed": 2,
1175
+ "expected": 2,
1176
+ "protocol": "20a9ff73f46a316127e7227aaf64a57ebeb1f7227c89116739b7a71d1c1aad1f"
1177
+ }
1178
+ },
1179
+ "bfcl": {
1180
+ "status": "NOT_QUALIFIED",
1181
+ "expected": 200,
1182
+ "completed": 0,
1183
+ "unscored": 200,
1184
+ "primary": null,
1185
+ "success_count": 0,
1186
+ "status_counts": {
1187
+ "NOT_QUALIFIED": 200
1188
+ },
1189
+ "returned_work": {},
1190
+ "qualification": {
1191
+ "status": "NOT_QUALIFIED",
1192
+ "passed": 0,
1193
+ "expected": 2,
1194
+ "protocol": "350f69949ba27db97b196cc5f4906e2d60dae36532c1e55c263bbd173726b9f7"
1195
+ }
1196
+ }
1197
+ }
1198
+ },
1199
+ "Qwen--Qwen3.5-9B": {
1200
+ "label": "Qwen3.5-9B",
1201
+ "panels": {
1202
+ "alfworld": {
1203
+ "status": "COMPLETE",
1204
+ "expected": 134,
1205
+ "completed": 134,
1206
+ "unscored": 0,
1207
+ "primary": 0.2462686567164179,
1208
+ "success_count": 33,
1209
+ "status_counts": {
1210
+ "COMPLETE": 134
1211
+ },
1212
+ "returned_work": {
1213
+ "replies": 6888,
1214
+ "input_tokens": 12668791,
1215
+ "output_tokens": 49242,
1216
+ "length_finishes": 0,
1217
+ "nothing_happens": 4466
1218
+ },
1219
+ "qualification": {
1220
+ "status": "QUALIFIED",
1221
+ "passed": 2,
1222
+ "expected": 2,
1223
+ "protocol": "489294ca7e2e7553b172652434f9be9fec60987efc1a1bb7b80f3ef2ebbdec54"
1224
+ }
1225
+ },
1226
+ "bfcl": {
1227
+ "status": "COMPLETE",
1228
+ "expected": 200,
1229
+ "completed": 200,
1230
+ "unscored": 0,
1231
+ "primary": 0.38,
1232
+ "success_count": 76,
1233
+ "status_counts": {
1234
+ "COMPLETE": 200
1235
+ },
1236
+ "returned_work": {
1237
+ "replies": 1999,
1238
+ "input_tokens": 9895239,
1239
+ "output_tokens": 33443,
1240
+ "length_finishes": 0,
1241
+ "invalid_response_turn_end": 54
1242
+ },
1243
+ "qualification": {
1244
+ "status": "QUALIFIED",
1245
+ "passed": 2,
1246
+ "expected": 2,
1247
+ "protocol": "9a798b3133c866fa47c3ef1b41978c3a7f2a9674e720ca1a8da1049ba76f840b"
1248
+ }
1249
+ }
1250
+ }
1251
+ }
1252
+ },
1253
+ "performance": {
1254
+ "logged_event_window_seconds": 3482.9023389816284,
1255
+ "phase_wall_seconds": {
1256
+ "qualification": 59.02381372451782,
1257
+ "benchmark": 2828.561756372452
1258
+ },
1259
+ "returned_generated_tokens": {
1260
+ "qualification": 2245,
1261
+ "benchmark": 347572
1262
+ },
1263
+ "model_load_and_preparation_seconds": 587.3164191246033,
1264
+ "remaining_event_window_seconds": 8.000349760055542,
1265
+ "returned_output_tokens_per_phase_wall_second": {
1266
+ "qualification": 38.03549547777616,
1267
+ "benchmark": 122.879410080744
1268
+ },
1269
+ "sampled_mean_UTL": 38.951079136690645,
1270
+ "sampled_mean_W": 199.63323741007196,
1271
+ "samples": 695,
1272
+ "MFU": null,
1273
+ "MFU_status": "not measured; UTL, logical input tokens and padded work are not useful FLOPs",
1274
+ "note": "Returned token totals include failed-task replies saved in traces, exclude unreturned calls, and do not equal physical FLOPs. Samples are whole-device measurements within this allocation. Event window excludes entry setup before its first event. Job intervals end at their normal completion/context-limit event or matching process exit; absent exit evidence yields unknown duration and rate. Model-load intervals run from SERVER_START to first JOB_START; remaining time includes cleanup, scheduling and any incomplete startup, not a separately timed cleanup measurement.",
1275
+ "intervals": [
1276
+ {
1277
+ "model": "Orbit",
1278
+ "panel": "alfworld",
1279
+ "phase": "qualification",
1280
+ "start": 1793538313.9631853,
1281
+ "end": 1793538317.9649694,
1282
+ "seconds": 4.001784086227417,
1283
+ "termination_event": "JOB_COMPLETE",
1284
+ "samples": 1,
1285
+ "mean_UTL": 0.0,
1286
+ "mean_W": 116.3,
1287
+ "peak_memory_MiB": 2952.0,
1288
+ "completed_tasks": 2,
1289
+ "completed_tasks_per_second": 0.49977708864484255,
1290
+ "completed_task_output_tokens": 49
1291
+ },
1292
+ {
1293
+ "model": "Orbit",
1294
+ "panel": "alfworld",
1295
+ "phase": "benchmark",
1296
+ "start": 1793538317.968124,
1297
+ "end": 1793538765.0472424,
1298
+ "seconds": 447.0791184902191,
1299
+ "termination_event": "JOB_COMPLETE",
1300
+ "samples": 89,
1301
+ "mean_UTL": 27.685393258426966,
1302
+ "mean_W": 188.55269662921347,
1303
+ "peak_memory_MiB": 11516.0,
1304
+ "completed_tasks": 32,
1305
+ "completed_tasks_per_second": 0.07157569807344978,
1306
+ "completed_task_output_tokens": 11438
1307
+ },
1308
+ {
1309
+ "model": "Orbit",
1310
+ "panel": "alfworld",
1311
+ "phase": "benchmark",
1312
+ "start": 1793538765.050269,
1313
+ "end": 1793539281.11915,
1314
+ "seconds": 516.0688810348511,
1315
+ "termination_event": "JOB_COMPLETE",
1316
+ "samples": 103,
1317
+ "mean_UTL": 31.02912621359223,
1318
+ "mean_W": 173.2988349514563,
1319
+ "peak_memory_MiB": 11516.0,
1320
+ "completed_tasks": 32,
1321
+ "completed_tasks_per_second": 0.06200722650788739,
1322
+ "completed_task_output_tokens": 11821
1323
+ },
1324
+ {
1325
+ "model": "Orbit",
1326
+ "panel": "alfworld",
1327
+ "phase": "benchmark",
1328
+ "start": 1793539281.1217546,
1329
+ "end": 1793539750.205558,
1330
+ "seconds": 469.08380341529846,
1331
+ "termination_event": "JOB_COMPLETE",
1332
+ "samples": 93,
1333
+ "mean_UTL": 31.93548387096774,
1334
+ "mean_W": 187.5037634408602,
1335
+ "peak_memory_MiB": 18756.0,
1336
+ "completed_tasks": 32,
1337
+ "completed_tasks_per_second": 0.06821808761465407,
1338
+ "completed_task_output_tokens": 11208
1339
+ },
1340
+ {
1341
+ "model": "Orbit",
1342
+ "panel": "alfworld",
1343
+ "phase": "benchmark",
1344
+ "start": 1793539750.208408,
1345
+ "end": 1793540178.2816718,
1346
+ "seconds": 428.0732636451721,
1347
+ "termination_event": "JOB_COMPLETE",
1348
+ "samples": 86,
1349
+ "mean_UTL": 29.151162790697676,
1350
+ "mean_W": 174.84337209302325,
1351
+ "peak_memory_MiB": 21324.0,
1352
+ "completed_tasks": 32,
1353
+ "completed_tasks_per_second": 0.07475355906956302,
1354
+ "completed_task_output_tokens": 10184
1355
+ },
1356
+ {
1357
+ "model": "Orbit",
1358
+ "panel": "alfworld",
1359
+ "phase": "benchmark",
1360
+ "start": 1793540178.2836037,
1361
+ "end": 1793540234.2931137,
1362
+ "seconds": 56.0095100402832,
1363
+ "termination_event": "JOB_COMPLETE",
1364
+ "samples": 11,
1365
+ "mean_UTL": 74.27272727272727,
1366
+ "mean_W": 171.5772727272727,
1367
+ "peak_memory_MiB": 21324.0,
1368
+ "completed_tasks": 6,
1369
+ "completed_tasks_per_second": 0.10712466500215187,
1370
+ "completed_task_output_tokens": 1216
1371
+ },
1372
+ {
1373
+ "model": "Orbit",
1374
+ "panel": "bfcl",
1375
+ "phase": "qualification",
1376
+ "start": 1793540242.36775,
1377
+ "end": 1793540247.369418,
1378
+ "seconds": 5.0016679763793945,
1379
+ "termination_event": "JOB_COMPLETE",
1380
+ "samples": 1,
1381
+ "mean_UTL": 1.0,
1382
+ "mean_W": 95.89,
1383
+ "peak_memory_MiB": 2874.0,
1384
+ "completed_tasks": 2,
1385
+ "completed_tasks_per_second": 0.39986660638912686,
1386
+ "completed_task_output_tokens": 54
1387
+ },
1388
+ {
1389
+ "model": "Qwen--Qwen3.5-9B",
1390
+ "panel": "alfworld",
1391
+ "phase": "qualification",
1392
+ "start": 1793540973.7034795,
1393
+ "end": 1793540976.7045462,
1394
+ "seconds": 3.0010666847229004,
1395
+ "termination_event": "JOB_COMPLETE",
1396
+ "samples": 1,
1397
+ "mean_UTL": 0.0,
1398
+ "mean_W": 96.74,
1399
+ "peak_memory_MiB": 42426.0,
1400
+ "completed_tasks": 2,
1401
+ "completed_tasks_per_second": 0.6664297098698649,
1402
+ "completed_task_output_tokens": 38
1403
+ },
1404
+ {
1405
+ "model": "Qwen--Qwen3.5-9B",
1406
+ "panel": "alfworld",
1407
+ "phase": "benchmark",
1408
+ "start": 1793540976.7085817,
1409
+ "end": 1793541036.732439,
1410
+ "seconds": 60.0238573551178,
1411
+ "termination_event": "JOB_COMPLETE",
1412
+ "samples": 12,
1413
+ "mean_UTL": 83.33333333333333,
1414
+ "mean_W": 377.20666666666665,
1415
+ "peak_memory_MiB": 43576.0,
1416
+ "completed_tasks": 32,
1417
+ "completed_tasks_per_second": 0.5331213522429776,
1418
+ "completed_task_output_tokens": 8678
1419
+ },
1420
+ {
1421
+ "model": "Qwen--Qwen3.5-9B",
1422
+ "panel": "alfworld",
1423
+ "phase": "benchmark",
1424
+ "start": 1793541036.7377534,
1425
+ "end": 1793541107.750358,
1426
+ "seconds": 71.01260471343994,
1427
+ "termination_event": "JOB_COMPLETE",
1428
+ "samples": 14,
1429
+ "mean_UTL": 85.71428571428571,
1430
+ "mean_W": 396.35,
1431
+ "peak_memory_MiB": 43576.0,
1432
+ "completed_tasks": 32,
1433
+ "completed_tasks_per_second": 0.4506242255037807,
1434
+ "completed_task_output_tokens": 11625
1435
+ },
1436
+ {
1437
+ "model": "Qwen--Qwen3.5-9B",
1438
+ "panel": "alfworld",
1439
+ "phase": "benchmark",
1440
+ "start": 1793541107.7554195,
1441
+ "end": 1793541186.774323,
1442
+ "seconds": 79.01890349388123,
1443
+ "termination_event": "JOB_COMPLETE",
1444
+ "samples": 15,
1445
+ "mean_UTL": 90.73333333333333,
1446
+ "mean_W": 410.74133333333333,
1447
+ "peak_memory_MiB": 43576.0,
1448
+ "completed_tasks": 32,
1449
+ "completed_tasks_per_second": 0.4049663888651391,
1450
+ "completed_task_output_tokens": 13940
1451
+ },
1452
+ {
1453
+ "model": "Qwen--Qwen3.5-9B",
1454
+ "panel": "alfworld",
1455
+ "phase": "benchmark",
1456
+ "start": 1793541186.77937,
1457
+ "end": 1793541262.8006992,
1458
+ "seconds": 76.021329164505,
1459
+ "termination_event": "JOB_COMPLETE",
1460
+ "samples": 16,
1461
+ "mean_UTL": 81.25,
1462
+ "mean_W": 372.585625,
1463
+ "peak_memory_MiB": 43576.0,
1464
+ "completed_tasks": 32,
1465
+ "completed_tasks_per_second": 0.42093449761650664,
1466
+ "completed_task_output_tokens": 13414
1467
+ },
1468
+ {
1469
+ "model": "Qwen--Qwen3.5-9B",
1470
+ "panel": "alfworld",
1471
+ "phase": "benchmark",
1472
+ "start": 1793541262.8055255,
1473
+ "end": 1793541286.810321,
1474
+ "seconds": 24.00479555130005,
1475
+ "termination_event": "JOB_COMPLETE",
1476
+ "samples": 4,
1477
+ "mean_UTL": 91.25,
1478
+ "mean_W": 382.0375,
1479
+ "peak_memory_MiB": 43576.0,
1480
+ "completed_tasks": 6,
1481
+ "completed_tasks_per_second": 0.24995005632010278,
1482
+ "completed_task_output_tokens": 1585
1483
+ },
1484
+ {
1485
+ "model": "Qwen--Qwen3.5-9B",
1486
+ "panel": "bfcl",
1487
+ "phase": "qualification",
1488
+ "start": 1793540392.2453332,
1489
+ "end": 1793540395.2467415,
1490
+ "seconds": 3.001408338546753,
1491
+ "termination_event": "JOB_COMPLETE",
1492
+ "samples": 1,
1493
+ "mean_UTL": 0.0,
1494
+ "mean_W": 156.23,
1495
+ "peak_memory_MiB": 41086.0,
1496
+ "completed_tasks": 2,
1497
+ "completed_tasks_per_second": 0.6663538493960395,
1498
+ "completed_task_output_tokens": 45
1499
+ },
1500
+ {
1501
+ "model": "Qwen--Qwen3.5-9B",
1502
+ "panel": "bfcl",
1503
+ "phase": "benchmark",
1504
+ "start": 1793540395.250629,
1505
+ "end": 1793540419.2554846,
1506
+ "seconds": 24.004855632781982,
1507
+ "termination_event": "JOB_COMPLETE",
1508
+ "samples": 5,
1509
+ "mean_UTL": 80.0,
1510
+ "mean_W": 370.206,
1511
+ "peak_memory_MiB": 42502.0,
1512
+ "completed_tasks": 32,
1513
+ "completed_tasks_per_second": 1.3330636305222987,
1514
+ "completed_task_output_tokens": 4092
1515
+ },
1516
+ {
1517
+ "model": "Qwen--Qwen3.5-9B",
1518
+ "panel": "bfcl",
1519
+ "phase": "benchmark",
1520
+ "start": 1793540419.2587035,
1521
+ "end": 1793540453.2645802,
1522
+ "seconds": 34.005876779556274,
1523
+ "termination_event": "JOB_COMPLETE",
1524
+ "samples": 6,
1525
+ "mean_UTL": 97.33333333333333,
1526
+ "mean_W": 406.5566666666667,
1527
+ "peak_memory_MiB": 42502.0,
1528
+ "completed_tasks": 32,
1529
+ "completed_tasks_per_second": 0.9410138196830092,
1530
+ "completed_task_output_tokens": 4825
1531
+ },
1532
+ {
1533
+ "model": "Qwen--Qwen3.5-9B",
1534
+ "panel": "bfcl",
1535
+ "phase": "benchmark",
1536
+ "start": 1793540453.268372,
1537
+ "end": 1793540475.2724211,
1538
+ "seconds": 22.004049062728882,
1539
+ "termination_event": "JOB_COMPLETE",
1540
+ "samples": 5,
1541
+ "mean_UTL": 80.0,
1542
+ "mean_W": 315.35,
1543
+ "peak_memory_MiB": 42502.0,
1544
+ "completed_tasks": 32,
1545
+ "completed_tasks_per_second": 1.4542777971806362,
1546
+ "completed_task_output_tokens": 4921
1547
+ },
1548
+ {
1549
+ "model": "Qwen--Qwen3.5-9B",
1550
+ "panel": "bfcl",
1551
+ "phase": "benchmark",
1552
+ "start": 1793540475.275762,
1553
+ "end": 1793540495.2791128,
1554
+ "seconds": 20.003350734710693,
1555
+ "termination_event": "JOB_COMPLETE",
1556
+ "samples": 4,
1557
+ "mean_UTL": 75.0,
1558
+ "mean_W": 369.53,
1559
+ "peak_memory_MiB": 42502.0,
1560
+ "completed_tasks": 32,
1561
+ "completed_tasks_per_second": 1.599731986125314,
1562
+ "completed_task_output_tokens": 4019
1563
+ },
1564
+ {
1565
+ "model": "Qwen--Qwen3.5-9B",
1566
+ "panel": "bfcl",
1567
+ "phase": "benchmark",
1568
+ "start": 1793540495.2829401,
1569
+ "end": 1793540519.2877417,
1570
+ "seconds": 24.004801511764526,
1571
+ "termination_event": "JOB_COMPLETE",
1572
+ "samples": 4,
1573
+ "mean_UTL": 100.0,
1574
+ "mean_W": 431.565,
1575
+ "peak_memory_MiB": 42502.0,
1576
+ "completed_tasks": 32,
1577
+ "completed_tasks_per_second": 1.3330666360360073,
1578
+ "completed_task_output_tokens": 4841
1579
+ },
1580
+ {
1581
+ "model": "Qwen--Qwen3.5-9B",
1582
+ "panel": "bfcl",
1583
+ "phase": "benchmark",
1584
+ "start": 1793540519.2917953,
1585
+ "end": 1793540547.2975574,
1586
+ "seconds": 28.005762100219727,
1587
+ "termination_event": "JOB_COMPLETE",
1588
+ "samples": 6,
1589
+ "mean_UTL": 78.5,
1590
+ "mean_W": 361.42,
1591
+ "peak_memory_MiB": 42502.0,
1592
+ "completed_tasks": 32,
1593
+ "completed_tasks_per_second": 1.1426220034822383,
1594
+ "completed_task_output_tokens": 8521
1595
+ },
1596
+ {
1597
+ "model": "Qwen--Qwen3.5-9B",
1598
+ "panel": "bfcl",
1599
+ "phase": "benchmark",
1600
+ "start": 1793540547.3013208,
1601
+ "end": 1793540561.3049686,
1602
+ "seconds": 14.003647804260254,
1603
+ "termination_event": "JOB_COMPLETE",
1604
+ "samples": 3,
1605
+ "mean_UTL": 97.0,
1606
+ "mean_W": 352.29333333333335,
1607
+ "peak_memory_MiB": 42502.0,
1608
+ "completed_tasks": 8,
1609
+ "completed_tasks_per_second": 0.5712797202430501,
1610
+ "completed_task_output_tokens": 2224
1611
+ },
1612
+ {
1613
+ "model": "Qwen--Qwen3.5-4B",
1614
+ "panel": "alfworld",
1615
+ "phase": "qualification",
1616
+ "start": 1793541313.0388699,
1617
+ "end": 1793541316.040314,
1618
+ "seconds": 3.001444101333618,
1619
+ "termination_event": "JOB_COMPLETE",
1620
+ "samples": 0,
1621
+ "mean_UTL": null,
1622
+ "mean_W": null,
1623
+ "peak_memory_MiB": null,
1624
+ "completed_tasks": 2,
1625
+ "completed_tasks_per_second": 0.6663459096610691,
1626
+ "completed_task_output_tokens": 38
1627
+ },
1628
+ {
1629
+ "model": "Qwen--Qwen3.5-4B",
1630
+ "panel": "alfworld",
1631
+ "phase": "benchmark",
1632
+ "start": 1793541316.0443146,
1633
+ "end": 1793541366.0601938,
1634
+ "seconds": 50.01587915420532,
1635
+ "termination_event": "JOB_COMPLETE",
1636
+ "samples": 10,
1637
+ "mean_UTL": 81.2,
1638
+ "mean_W": 351.175,
1639
+ "peak_memory_MiB": 44578.0,
1640
+ "completed_tasks": 32,
1641
+ "completed_tasks_per_second": 0.6397968113554483,
1642
+ "completed_task_output_tokens": 10171
1643
+ },
1644
+ {
1645
+ "model": "Qwen--Qwen3.5-4B",
1646
+ "panel": "alfworld",
1647
+ "phase": "benchmark",
1648
+ "start": 1793541366.0650525,
1649
+ "end": 1793541418.082877,
1650
+ "seconds": 52.01782441139221,
1651
+ "termination_event": "JOB_COMPLETE",
1652
+ "samples": 10,
1653
+ "mean_UTL": 82.6,
1654
+ "mean_W": 371.936,
1655
+ "peak_memory_MiB": 44578.0,
1656
+ "completed_tasks": 32,
1657
+ "completed_tasks_per_second": 0.6151737478853847,
1658
+ "completed_task_output_tokens": 11264
1659
+ },
1660
+ {
1661
+ "model": "Qwen--Qwen3.5-4B",
1662
+ "panel": "alfworld",
1663
+ "phase": "benchmark",
1664
+ "start": 1793541418.0863738,
1665
+ "end": 1793541471.0973716,
1666
+ "seconds": 53.0109977722168,
1667
+ "termination_event": "JOB_COMPLETE",
1668
+ "samples": 11,
1669
+ "mean_UTL": 74.27272727272727,
1670
+ "mean_W": 357.95272727272726,
1671
+ "peak_memory_MiB": 44578.0,
1672
+ "completed_tasks": 32,
1673
+ "completed_tasks_per_second": 0.603648324777831,
1674
+ "completed_task_output_tokens": 10344
1675
+ },
1676
+ {
1677
+ "model": "Qwen--Qwen3.5-4B",
1678
+ "panel": "alfworld",
1679
+ "phase": "benchmark",
1680
+ "start": 1793541471.1016855,
1681
+ "end": 1793541525.1161807,
1682
+ "seconds": 54.01449513435364,
1683
+ "termination_event": "JOB_COMPLETE",
1684
+ "samples": 11,
1685
+ "mean_UTL": 72.63636363636364,
1686
+ "mean_W": 342.4190909090909,
1687
+ "peak_memory_MiB": 44578.0,
1688
+ "completed_tasks": 32,
1689
+ "completed_tasks_per_second": 0.5924335665899384,
1690
+ "completed_task_output_tokens": 14000
1691
+ },
1692
+ {
1693
+ "model": "Qwen--Qwen3.5-4B",
1694
+ "panel": "alfworld",
1695
+ "phase": "benchmark",
1696
+ "start": 1793541525.1192768,
1697
+ "end": 1793541545.1222277,
1698
+ "seconds": 20.00295090675354,
1699
+ "termination_event": "JOB_COMPLETE",
1700
+ "samples": 4,
1701
+ "mean_UTL": 72.5,
1702
+ "mean_W": 339.78249999999997,
1703
+ "peak_memory_MiB": 44578.0,
1704
+ "completed_tasks": 6,
1705
+ "completed_tasks_per_second": 0.2999557429286214,
1706
+ "completed_task_output_tokens": 2961
1707
+ },
1708
+ {
1709
+ "model": "Qwen--Qwen3.5-4B",
1710
+ "panel": "bfcl",
1711
+ "phase": "qualification",
1712
+ "start": 1793540645.9493148,
1713
+ "end": 1793540648.9506512,
1714
+ "seconds": 3.0013363361358643,
1715
+ "termination_event": "JOB_COMPLETE",
1716
+ "samples": 0,
1717
+ "mean_UTL": null,
1718
+ "mean_W": null,
1719
+ "peak_memory_MiB": null,
1720
+ "completed_tasks": 2,
1721
+ "completed_tasks_per_second": 0.6663698353030781,
1722
+ "completed_task_output_tokens": 64
1723
+ },
1724
+ {
1725
+ "model": "HuggingFaceTB--SmolLM3-3B",
1726
+ "panel": "alfworld",
1727
+ "phase": "qualification",
1728
+ "start": 1793541573.2847347,
1729
+ "end": 1793541576.2864163,
1730
+ "seconds": 3.0016815662384033,
1731
+ "termination_event": "JOB_COMPLETE",
1732
+ "samples": 1,
1733
+ "mean_UTL": 0.0,
1734
+ "mean_W": 55.84,
1735
+ "peak_memory_MiB": 42548.0,
1736
+ "completed_tasks": 2,
1737
+ "completed_tasks_per_second": 0.6662931946196832,
1738
+ "completed_task_output_tokens": 120
1739
+ },
1740
+ {
1741
+ "model": "HuggingFaceTB--SmolLM3-3B",
1742
+ "panel": "bfcl",
1743
+ "phase": "qualification",
1744
+ "start": 1793540691.3894026,
1745
+ "end": 1793540695.3909073,
1746
+ "seconds": 4.00150465965271,
1747
+ "termination_event": "JOB_COMPLETE",
1748
+ "samples": 1,
1749
+ "mean_UTL": 0.0,
1750
+ "mean_W": 99.37,
1751
+ "peak_memory_MiB": 42468.0,
1752
+ "completed_tasks": 2,
1753
+ "completed_tasks_per_second": 0.49981198826682854,
1754
+ "completed_task_output_tokens": 408
1755
+ },
1756
+ {
1757
+ "model": "LiquidAI--LFM2.5-1.2B-Instruct",
1758
+ "panel": "alfworld",
1759
+ "phase": "qualification",
1760
+ "start": 1793541592.4816368,
1761
+ "end": 1793541595.4829729,
1762
+ "seconds": 3.001336097717285,
1763
+ "termination_event": "JOB_COMPLETE",
1764
+ "samples": 1,
1765
+ "mean_UTL": 28.0,
1766
+ "mean_W": 90.38,
1767
+ "peak_memory_MiB": 42654.0,
1768
+ "completed_tasks": 2,
1769
+ "completed_tasks_per_second": 0.6663698882378193,
1770
+ "completed_task_output_tokens": 71
1771
+ },
1772
+ {
1773
+ "model": "LiquidAI--LFM2.5-1.2B-Instruct",
1774
+ "panel": "bfcl",
1775
+ "phase": "qualification",
1776
+ "start": 1793540727.6986325,
1777
+ "end": 1793540730.6999526,
1778
+ "seconds": 3.0013201236724854,
1779
+ "termination_event": "JOB_COMPLETE",
1780
+ "samples": 1,
1781
+ "mean_UTL": 0.0,
1782
+ "mean_W": 97.36,
1783
+ "peak_memory_MiB": 42622.0,
1784
+ "completed_tasks": 2,
1785
+ "completed_tasks_per_second": 0.6663734348846312,
1786
+ "completed_task_output_tokens": 152
1787
+ },
1788
+ {
1789
+ "model": "openbmb--MiniCPM5-1B-SFT",
1790
+ "panel": "alfworld",
1791
+ "phase": "qualification",
1792
+ "start": 1793541613.8396358,
1793
+ "end": 1793541616.8407776,
1794
+ "seconds": 3.0011417865753174,
1795
+ "termination_event": "JOB_COMPLETE",
1796
+ "samples": 1,
1797
+ "mean_UTL": 0.0,
1798
+ "mean_W": 54.94,
1799
+ "peak_memory_MiB": 42840.0,
1800
+ "completed_tasks": 2,
1801
+ "completed_tasks_per_second": 0.6664130328484924,
1802
+ "completed_task_output_tokens": 32
1803
+ },
1804
+ {
1805
+ "model": "openbmb--MiniCPM5-1B-SFT",
1806
+ "panel": "alfworld",
1807
+ "phase": "benchmark",
1808
+ "start": 1793541616.845832,
1809
+ "end": 1793541640.8553474,
1810
+ "seconds": 24.009515285491943,
1811
+ "termination_event": "JOB_COMPLETE",
1812
+ "samples": 5,
1813
+ "mean_UTL": 29.6,
1814
+ "mean_W": 148.992,
1815
+ "peak_memory_MiB": 42844.0,
1816
+ "completed_tasks": 32,
1817
+ "completed_tasks_per_second": 1.332804915863354,
1818
+ "completed_task_output_tokens": 13025
1819
+ },
1820
+ {
1821
+ "model": "openbmb--MiniCPM5-1B-SFT",
1822
+ "panel": "alfworld",
1823
+ "phase": "benchmark",
1824
+ "start": 1793541640.8593676,
1825
+ "end": 1793541667.874898,
1826
+ "seconds": 27.015530347824097,
1827
+ "termination_event": "JOB_COMPLETE",
1828
+ "samples": 5,
1829
+ "mean_UTL": 41.2,
1830
+ "mean_W": 166.852,
1831
+ "peak_memory_MiB": 42844.0,
1832
+ "completed_tasks": 32,
1833
+ "completed_tasks_per_second": 1.184503860853406,
1834
+ "completed_task_output_tokens": 13270
1835
+ },
1836
+ {
1837
+ "model": "openbmb--MiniCPM5-1B-SFT",
1838
+ "panel": "alfworld",
1839
+ "phase": "benchmark",
1840
+ "start": 1793541667.8780117,
1841
+ "end": 1793541695.8888903,
1842
+ "seconds": 28.010878562927246,
1843
+ "termination_event": "JOB_COMPLETE",
1844
+ "samples": 6,
1845
+ "mean_UTL": 23.333333333333332,
1846
+ "mean_W": 142.46166666666667,
1847
+ "peak_memory_MiB": 42844.0,
1848
+ "completed_tasks": 32,
1849
+ "completed_tasks_per_second": 1.1424132923253756,
1850
+ "completed_task_output_tokens": 14209
1851
+ },
1852
+ {
1853
+ "model": "openbmb--MiniCPM5-1B-SFT",
1854
+ "panel": "alfworld",
1855
+ "phase": "benchmark",
1856
+ "start": 1793541695.8929152,
1857
+ "end": 1793541723.9094083,
1858
+ "seconds": 28.01649308204651,
1859
+ "termination_event": "JOB_COMPLETE",
1860
+ "samples": 5,
1861
+ "mean_UTL": 36.6,
1862
+ "mean_W": 163.286,
1863
+ "peak_memory_MiB": 42844.0,
1864
+ "completed_tasks": 32,
1865
+ "completed_tasks_per_second": 1.1421843521345716,
1866
+ "completed_task_output_tokens": 12751
1867
+ },
1868
+ {
1869
+ "model": "openbmb--MiniCPM5-1B-SFT",
1870
+ "panel": "alfworld",
1871
+ "phase": "benchmark",
1872
+ "start": 1793541723.9135094,
1873
+ "end": 1793541731.9154322,
1874
+ "seconds": 8.001922845840454,
1875
+ "termination_event": "JOB_COMPLETE",
1876
+ "samples": 2,
1877
+ "mean_UTL": 48.5,
1878
+ "mean_W": 167.005,
1879
+ "peak_memory_MiB": 42844.0,
1880
+ "completed_tasks": 6,
1881
+ "completed_tasks_per_second": 0.7498197765202035,
1882
+ "completed_task_output_tokens": 2243
1883
+ },
1884
+ {
1885
+ "model": "openbmb--MiniCPM5-1B-SFT",
1886
+ "panel": "bfcl",
1887
+ "phase": "qualification",
1888
+ "start": 1793540760.728207,
1889
+ "end": 1793540763.7294846,
1890
+ "seconds": 3.001277446746826,
1891
+ "termination_event": "JOB_COMPLETE",
1892
+ "samples": 0,
1893
+ "mean_UTL": null,
1894
+ "mean_W": null,
1895
+ "peak_memory_MiB": null,
1896
+ "completed_tasks": 2,
1897
+ "completed_tasks_per_second": 0.6663829104396395,
1898
+ "completed_task_output_tokens": 204
1899
+ },
1900
+ {
1901
+ "model": "openbmb--MiniCPM5-1B",
1902
+ "panel": "alfworld",
1903
+ "phase": "qualification",
1904
+ "start": 1793541748.103092,
1905
+ "end": 1793541751.104105,
1906
+ "seconds": 3.0010130405426025,
1907
+ "termination_event": "JOB_COMPLETE",
1908
+ "samples": 1,
1909
+ "mean_UTL": 0.0,
1910
+ "mean_W": 71.83,
1911
+ "peak_memory_MiB": 42840.0,
1912
+ "completed_tasks": 2,
1913
+ "completed_tasks_per_second": 0.66644162253903,
1914
+ "completed_task_output_tokens": 44
1915
+ },
1916
+ {
1917
+ "model": "openbmb--MiniCPM5-1B",
1918
+ "panel": "bfcl",
1919
+ "phase": "qualification",
1920
+ "start": 1793540789.273598,
1921
+ "end": 1793540792.2749584,
1922
+ "seconds": 3.0013604164123535,
1923
+ "termination_event": "JOB_COMPLETE",
1924
+ "samples": 1,
1925
+ "mean_UTL": 0.0,
1926
+ "mean_W": 65.22,
1927
+ "peak_memory_MiB": 42666.0,
1928
+ "completed_tasks": 2,
1929
+ "completed_tasks_per_second": 0.6663644889375466,
1930
+ "completed_task_output_tokens": 59
1931
+ },
1932
+ {
1933
+ "model": "openbmb--MiniCPM5-1B",
1934
+ "panel": "bfcl",
1935
+ "phase": "benchmark",
1936
+ "start": 1793540792.2786386,
1937
+ "end": 1793540800.2803898,
1938
+ "seconds": 8.001751184463501,
1939
+ "termination_event": "JOB_COMPLETE",
1940
+ "samples": 1,
1941
+ "mean_UTL": 100.0,
1942
+ "mean_W": 401.02,
1943
+ "peak_memory_MiB": 42814.0,
1944
+ "completed_tasks": 32,
1945
+ "completed_tasks_per_second": 3.9991245993917426,
1946
+ "completed_task_output_tokens": 6457
1947
+ },
1948
+ {
1949
+ "model": "openbmb--MiniCPM5-1B",
1950
+ "panel": "bfcl",
1951
+ "phase": "benchmark",
1952
+ "start": 1793540800.2837431,
1953
+ "end": 1793540811.285404,
1954
+ "seconds": 11.001660823822021,
1955
+ "termination_event": "JOB_COMPLETE",
1956
+ "samples": 3,
1957
+ "mean_UTL": 33.333333333333336,
1958
+ "mean_W": 179.45666666666668,
1959
+ "peak_memory_MiB": 42814.0,
1960
+ "completed_tasks": 32,
1961
+ "completed_tasks_per_second": 2.90865174926226,
1962
+ "completed_task_output_tokens": 10438
1963
+ },
1964
+ {
1965
+ "model": "openbmb--MiniCPM5-1B",
1966
+ "panel": "bfcl",
1967
+ "phase": "benchmark",
1968
+ "start": 1793540811.2868805,
1969
+ "end": 1793540823.2890596,
1970
+ "seconds": 12.002179145812988,
1971
+ "termination_event": "JOB_COMPLETE",
1972
+ "samples": 2,
1973
+ "mean_UTL": 100.0,
1974
+ "mean_W": 373.355,
1975
+ "peak_memory_MiB": 42814.0,
1976
+ "completed_tasks": 32,
1977
+ "completed_tasks_per_second": 2.666182499964045,
1978
+ "completed_task_output_tokens": 17539
1979
+ },
1980
+ {
1981
+ "model": "openbmb--MiniCPM5-1B",
1982
+ "panel": "bfcl",
1983
+ "phase": "benchmark",
1984
+ "start": 1793540823.2916431,
1985
+ "end": 1793540832.293638,
1986
+ "seconds": 9.001994848251343,
1987
+ "termination_event": "JOB_COMPLETE",
1988
+ "samples": 2,
1989
+ "mean_UTL": 50.0,
1990
+ "mean_W": 299.83000000000004,
1991
+ "peak_memory_MiB": 42814.0,
1992
+ "completed_tasks": 32,
1993
+ "completed_tasks_per_second": 3.554767641998381,
1994
+ "completed_task_output_tokens": 9618
1995
+ },
1996
+ {
1997
+ "model": "openbmb--MiniCPM5-1B",
1998
+ "panel": "bfcl",
1999
+ "phase": "benchmark",
2000
+ "start": 1793540832.2977865,
2001
+ "end": 1793540851.3013995,
2002
+ "seconds": 19.003612995147705,
2003
+ "termination_event": "JOB_COMPLETE",
2004
+ "samples": 4,
2005
+ "mean_UTL": 75.0,
2006
+ "mean_W": 300.2025,
2007
+ "peak_memory_MiB": 42814.0,
2008
+ "completed_tasks": 32,
2009
+ "completed_tasks_per_second": 1.6838903217072845,
2010
+ "completed_task_output_tokens": 22502
2011
+ },
2012
+ {
2013
+ "model": "openbmb--MiniCPM5-1B",
2014
+ "panel": "bfcl",
2015
+ "phase": "benchmark",
2016
+ "start": 1793540851.3050878,
2017
+ "end": 1793540871.3083003,
2018
+ "seconds": 20.003212451934814,
2019
+ "termination_event": "JOB_COMPLETE",
2020
+ "samples": 4,
2021
+ "mean_UTL": 75.0,
2022
+ "mean_W": 342.4175,
2023
+ "peak_memory_MiB": 42814.0,
2024
+ "completed_tasks": 32,
2025
+ "completed_tasks_per_second": 1.5997430451179753,
2026
+ "completed_task_output_tokens": 38655
2027
+ },
2028
+ {
2029
+ "model": "openbmb--MiniCPM5-1B",
2030
+ "panel": "bfcl",
2031
+ "phase": "benchmark",
2032
+ "start": 1793540871.3096328,
2033
+ "end": 1793540884.3120797,
2034
+ "seconds": 13.00244688987732,
2035
+ "termination_event": "JOB_COMPLETE",
2036
+ "samples": 2,
2037
+ "mean_UTL": 100.0,
2038
+ "mean_W": 306.625,
2039
+ "peak_memory_MiB": 42814.0,
2040
+ "completed_tasks": 8,
2041
+ "completed_tasks_per_second": 0.6152688080755145,
2042
+ "completed_task_output_tokens": 9573
2043
+ },
2044
+ {
2045
+ "model": "Qwen--Qwen2.5-1.5B-Instruct",
2046
+ "panel": "alfworld",
2047
+ "phase": "qualification",
2048
+ "start": 1793541768.135486,
2049
+ "end": 1793541771.1361818,
2050
+ "seconds": 3.0006959438323975,
2051
+ "termination_event": "JOB_COMPLETE",
2052
+ "samples": 1,
2053
+ "mean_UTL": 0.0,
2054
+ "mean_W": 106.78,
2055
+ "peak_memory_MiB": 42676.0,
2056
+ "completed_tasks": 2,
2057
+ "completed_tasks_per_second": 0.666512048350244,
2058
+ "completed_task_output_tokens": 83
2059
+ },
2060
+ {
2061
+ "model": "Qwen--Qwen2.5-1.5B-Instruct",
2062
+ "panel": "bfcl",
2063
+ "phase": "qualification",
2064
+ "start": 1793540914.90442,
2065
+ "end": 1793540917.9056945,
2066
+ "seconds": 3.001274585723877,
2067
+ "termination_event": "JOB_COMPLETE",
2068
+ "samples": 1,
2069
+ "mean_UTL": 0.0,
2070
+ "mean_W": 137.53,
2071
+ "peak_memory_MiB": 42640.0,
2072
+ "completed_tasks": 2,
2073
+ "completed_tasks_per_second": 0.6663835456820157,
2074
+ "completed_task_output_tokens": 168
2075
+ },
2076
+ {
2077
+ "model": "HuggingFaceTB--SmolLM2-1.7B-Instruct",
2078
+ "panel": "alfworld",
2079
+ "phase": "qualification",
2080
+ "start": 1793541786.5357795,
2081
+ "end": 1793541789.5370362,
2082
+ "seconds": 3.0012567043304443,
2083
+ "termination_event": "JOB_COMPLETE",
2084
+ "samples": 0,
2085
+ "mean_UTL": null,
2086
+ "mean_W": null,
2087
+ "peak_memory_MiB": null,
2088
+ "completed_tasks": 2,
2089
+ "completed_tasks_per_second": 0.666387515974307,
2090
+ "completed_task_output_tokens": 160
2091
+ },
2092
+ {
2093
+ "model": "HuggingFaceTB--SmolLM2-1.7B-Instruct",
2094
+ "panel": "bfcl",
2095
+ "phase": "qualification",
2096
+ "start": 1793540942.3241732,
2097
+ "end": 1793540946.325417,
2098
+ "seconds": 4.001243829727173,
2099
+ "termination_event": "JOB_COMPLETE",
2100
+ "samples": 1,
2101
+ "mean_UTL": 0.0,
2102
+ "mean_W": 68.35,
2103
+ "peak_memory_MiB": 42594.0,
2104
+ "completed_tasks": 2,
2105
+ "completed_tasks_per_second": 0.49984456961633633,
2106
+ "completed_task_output_tokens": 456
2107
+ }
2108
+ ]
2109
+ },
2110
+ "warnings": [
2111
+ "ALF134 is seen regression; BFCL200 is a repeated evaluation after an interface revision, not fresh held-out selection.",
2112
+ "Qualification is a fixed two-fixture interface check per domain, not general task mastery; unqualified or unfinished positions remain unscored.",
2113
+ "Text function-calling protocol with native chat templates; not native FC API or official unrestricted leaderboard scores.",
2114
+ "External vLLM version changed from 0.26.0 to 0.29.0; old/new differences do not isolate a single interface change.",
2115
+ "ScienceWorld and conventional five benchmarks were not rerun. No new RSI proposals or accepted/installed successors."
2116
+ ],
2117
+ "provenance": {
2118
+ "summary_sha256": "fc392ff15247ba7c0818079bf81f7bab80f4f1a91b9c061002690716fd7b7c37",
2119
+ "plan_sha256": "2cdf3379a044ae36f5965318861790e7f09e3a59f2b5b8d11c79772974a8355b",
2120
+ "source_candidate_sha256": "0fc72dda772a3e84aff131aad28b9a832f0b56740ea820e99321de0bfb8f52a3",
2121
+ "collector_sha256": "2dbd2405bce0a1137cd531b7a39accb30fef951b24166e996aa03d2527029be0",
2122
+ "task_evidence_sha256": "b9097fbb81f95f52c76117e44f64c4ec9ea90c6501aa2b6b5e5d23db6b096ecc",
2123
+ "qualification_evidence_sha256": "ec496e5cc873f3a840c9c3fa12c28ba7eb290f93e0aa07825e4a913e62bee60e",
2124
+ "original_public_agentic_json_sha256": "c6ab7f3dea0946d5909f7d2755dfd92b4d0a40794c7ea466bb1f61a80d537909"
2125
+ }
2126
  }
2127
  }