nohi191212 commited on
Commit
54b2c7f
·
verified ·
1 Parent(s): cb48b75

Release main-table weights and compact reproduction data

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. .gitattributes +4 -0
  2. README.md +47 -0
  3. appendix.pdf +3 -0
  4. experiment_data.tar.gz +3 -0
  5. main.pdf +3 -0
  6. paper_results/data/budget_diagnostics.json +370 -0
  7. paper_results/data/comparisons.json +0 -0
  8. paper_results/data/earlier_results.json +0 -0
  9. paper_results/data/joint_MiniCPM_comparison_20260910.json +58 -0
  10. paper_results/data/joint_Qwen3.8_comparison_20260910.json +58 -0
  11. paper_results/data/layout_verification.json +29 -0
  12. paper_results/data/main_table_cost_qwen_20260912/build_figures.py +101 -0
  13. paper_results/data/main_table_cost_qwen_20260912/components/instancevg_router.json +239 -0
  14. paper_results/data/main_table_cost_qwen_20260912/components/instancevg_specialist.json +254 -0
  15. paper_results/data/main_table_cost_qwen_20260912/components/qwen_flops.json +341 -0
  16. paper_results/data/main_table_cost_qwen_20260912/components/qwen_gref_flops.json +341 -0
  17. paper_results/data/main_table_cost_qwen_20260912/components/qwen_gref_latency.json +2446 -0
  18. paper_results/data/main_table_cost_qwen_20260912/components/qwen_latency.json +0 -0
  19. paper_results/data/main_table_cost_qwen_20260912/components/yolo26x_router.json +238 -0
  20. paper_results/data/main_table_cost_qwen_20260912/components/yolo26x_specialist.json +254 -0
  21. paper_results/data/main_table_cost_qwen_20260912/cost_summary.json +75 -0
  22. paper_results/data/main_table_cost_qwen_20260912/fixed_budget_cost_performance.csv +29 -0
  23. paper_results/data/main_table_cost_qwen_20260912/full_cost_performance_curves.csv +405 -0
  24. paper_results/data/main_table_cost_qwen_20260912/main_table_results.json +0 -0
  25. paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_flops.py +152 -0
  26. paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_large.py +209 -0
  27. paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_router.py +104 -0
  28. paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_specialist.py +258 -0
  29. paper_results/data/main_table_cost_qwen_20260912/reproducible/prepare_output.py +19 -0
  30. paper_results/data/main_table_cost_qwen_20260912/reproducible/read_main.py +10 -0
  31. paper_results/data/main_table_cost_qwen_20260912/reproducible/setup_local.py +6 -0
  32. paper_results/data/main_table_cost_qwen_20260912/reproducible/switch_qwen.py +23 -0
  33. paper_results/data/main_table_cost_qwen_20260912/selected_points.csv +5 -0
  34. paper_results/data/main_table_cost_qwen_20260912/verify_and_report.py +57 -0
  35. paper_results/data/main_table_cost_qwen_20260912/成本性能报告.md +50 -0
  36. paper_results/data/method_improvement_spec.json +114 -0
  37. paper_results/data/operation_measurements.json +129 -0
  38. paper_results/data/results.json +0 -0
  39. paper_results/data/scaling_selected.json +1250 -0
  40. paper_results/data/shared_search.json +720 -0
  41. paper_results/data/table_audit.json +443 -0
  42. paper_results/figures/build_compact.py +111 -0
  43. paper_results/figures/build_cost.py +84 -0
  44. paper_results/figures/build_tables.py +123 -0
  45. paper_results/figures/cost_includes.tex +14 -0
  46. paper_results/figures/gains_compact.pdf +0 -0
  47. paper_results/figures/gains_compact.png +3 -0
  48. paper_results/figures/joint_delta.pdf +0 -0
  49. paper_results/figures/joint_delta.png +0 -0
  50. paper_results/figures/scaling_ranked.pdf +0 -0
.gitattributes CHANGED
@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ appendix.pdf filter=lfs diff=lfs merge=lfs -text
37
+ main.pdf filter=lfs diff=lfs merge=lfs -text
38
+ paper_results/figures/gains_compact.png filter=lfs diff=lfs merge=lfs -text
39
+ paper_results/figures/scaling_ranked.png filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - computer-vision
6
+ - model-routing
7
+ - reproducibility
8
+ ---
9
+
10
+ # ModelCollaboration: trained weights and paper results
11
+
12
+ Code: https://github.com/nohi191212/ModelCollaboration
13
+
14
+ ## Contents
15
+
16
+ - `weights.tar`: 16 main-table router checkpoints and configurations, training-set normalization statistics, the final 8M distilled image/text encoders, and the trained ConstructionSite YOLO26x and RT-DETR-X specialists. Detector extraction configurations record the checkpoint source and inference settings. RT-DETR uses FP32 inference in the recorded experiment.
17
+ - `experiment_data.tar.gz`: per-example validation/test learned scores and task outcomes for all 16 pairs, sample identifiers, selected validation thresholds, complete learned curves and the selection record.
18
+ - `paper_results/`: source data and scripts for paper tables and figures. `data/results.json` includes confidence and learned curves, fixed selected points and outcome decompositions. `data/main_table_cost_qwen_20260912/` contains the final Qwen cost curves and component measurements. Earlier ablations are explicitly separate from the main setting.
19
+ - `main.pdf` and `appendix.pdf`: the paper and supplementary material accompanying this release.
20
+
21
+ ## Recompute the reported routing results
22
+
23
+ Download this repository, then extract the two archives into its root:
24
+
25
+ ```bash
26
+ tar -xf weights.tar
27
+ tar -xzf experiment_data.tar.gz
28
+ ```
29
+
30
+ From the code repository, run:
31
+
32
+ ```bash
33
+ pip install numpy
34
+ python scripts/replay_paper_results.py --artifacts /path/to/download --output replayed_main.csv
35
+ ```
36
+
37
+ This checks all saved learned validation/test curve points against the per-example outcomes, repeats validation-only operating-point selection, and exports the selected results. It does not execute the specialist or VLM. ConstructionSite uses event macro-F1; other tasks use their saved correctness indicators.
38
+
39
+ To redraw the paper tables and outcome/budget figure, install matplotlib and run `paper_results/figures/build_tables.py` and `paper_results/figures/build_compact.py`. The source tables in `paper_results/tables/` retain the final typeset captions and labels. Cost data for the final figure are under `paper_results/data/main_table_cost_qwen_20260912/`; costs are assembled from measured components rather than a new timing run during replay.
40
+
41
+ ## Running the models again
42
+
43
+ The weights archive contains `main/runs/<run>/best.pt`, its `config.json` and normalization arrays; `encoders/8M/epoch_300.pt`, tokenizer and encoder configuration; and `specialists/<expert>/best.pt` with extraction settings. Match the 16 run names to `configs/main/` in the code repository. Preserve the frozen validation thresholds when evaluating test samples.
44
+
45
+ Original dataset images, upstream VLM weights and full feature caches are external. Obtain upstream resources from the code repository's `docs/DOWNLOADS.md`. The full prepared feature cache is several GB and is excluded to keep non-weight experiment data below 100 MB. Running from original images requires preparing those features and predictions in the documented layout; the compact result replay above is the supported route for checking the published numbers without large-model inference. The release does not claim a one-command end-to-end reproduction from raw images.
46
+
47
+ The original source datasets and upstream model terms continue to apply. This model card does not grant a new license for third-party datasets or detector architectures.
appendix.pdf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f1f37970088137fb21832d56d5b793365463a54a291b05bca237aab50cf9a0ef
3
+ size 2430363
experiment_data.tar.gz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7ee5a0063cae30a79812aff05d9302196687411c0441040252a375f3f6e2a5fa
3
+ size 2457041
main.pdf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ac6d6f4cb7eb00f64e8e05789465e01aa865a1745dbcfb91dd0debb2344bf857
3
+ size 577296
paper_results/data/budget_diagnostics.json ADDED
@@ -0,0 +1,370 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "key": [
4
+ "cub",
5
+ "cub",
6
+ "MiniCPM"
7
+ ],
8
+ "areas": {
9
+ "confidence": {
10
+ "validation_fixed_A25": 0.9058333333333334,
11
+ "test_fixed_A25": 0.6472903003106663,
12
+ "test_ranked_A25_diagnostic": 0.7895926820849155,
13
+ "selected_validation_actual_calls": 0.01,
14
+ "selected_test_actual_calls": 0.113565757680359
15
+ },
16
+ "learned": {
17
+ "validation_fixed_A25": 0.9621666666666666,
18
+ "test_fixed_A25": 0.7763203313772868,
19
+ "test_ranked_A25_diagnostic": 0.7967552640662755,
20
+ "selected_validation_actual_calls": 0.01,
21
+ "selected_test_actual_calls": 0.028822920262340353
22
+ }
23
+ }
24
+ },
25
+ {
26
+ "key": [
27
+ "cub",
28
+ "cub",
29
+ "Qwen3.8"
30
+ ],
31
+ "areas": {
32
+ "confidence": {
33
+ "validation_fixed_A25": 0.9171666666666667,
34
+ "test_fixed_A25": 0.7316189161201243,
35
+ "test_ranked_A25_diagnostic": 0.8101484294097343,
36
+ "selected_validation_actual_calls": 0.01,
37
+ "selected_test_actual_calls": 0.113565757680359
38
+ },
39
+ "learned": {
40
+ "validation_fixed_A25": 0.9705833333333334,
41
+ "test_fixed_A25": 0.8053158439765276,
42
+ "test_ranked_A25_diagnostic": 0.806368657231619,
43
+ "selected_validation_actual_calls": 0.01,
44
+ "selected_test_actual_calls": 0.014325163962720055
45
+ }
46
+ }
47
+ },
48
+ {
49
+ "key": [
50
+ "cub",
51
+ "glsim",
52
+ "MiniCPM"
53
+ ],
54
+ "areas": {
55
+ "confidence": {
56
+ "validation_fixed_A25": 0.9005000000000001,
57
+ "test_fixed_A25": 0.7253365550569555,
58
+ "test_ranked_A25_diagnostic": 0.8519848118743528,
59
+ "selected_validation_actual_calls": 0.01,
60
+ "selected_test_actual_calls": 0.08698653779772178
61
+ },
62
+ "learned": {
63
+ "validation_fixed_A25": 0.97075,
64
+ "test_fixed_A25": 0.8740248532965137,
65
+ "test_ranked_A25_diagnostic": 0.8809112875388333,
66
+ "selected_validation_actual_calls": 0.03,
67
+ "selected_test_actual_calls": 0.04487400759406282
68
+ }
69
+ }
70
+ },
71
+ {
72
+ "key": [
73
+ "cub",
74
+ "glsim",
75
+ "Qwen3.8"
76
+ ],
77
+ "areas": {
78
+ "confidence": {
79
+ "validation_fixed_A25": 0.9213333333333334,
80
+ "test_fixed_A25": 0.7860545391784605,
81
+ "test_ranked_A25_diagnostic": 0.8691232309285468,
82
+ "selected_validation_actual_calls": 0.01,
83
+ "selected_test_actual_calls": 0.08698653779772178
84
+ },
85
+ "learned": {
86
+ "validation_fixed_A25": 0.9757500000000001,
87
+ "test_fixed_A25": 0.8918536416983087,
88
+ "test_ranked_A25_diagnostic": 0.8935623058336211,
89
+ "selected_validation_actual_calls": 0.01,
90
+ "selected_test_actual_calls": 0.016568864342423197
91
+ }
92
+ }
93
+ },
94
+ {
95
+ "key": [
96
+ "grefcoco",
97
+ "groundingdino",
98
+ "MiniCPM"
99
+ ],
100
+ "areas": {
101
+ "confidence": {
102
+ "validation_fixed_A25": 0.5566729917773561,
103
+ "test_fixed_A25": 0.27538212857669514,
104
+ "test_ranked_A25_diagnostic": 0.2786603522105323,
105
+ "selected_validation_actual_calls": 0.9899501019045611,
106
+ "selected_test_actual_calls": 0.9873805405098829
107
+ },
108
+ "learned": {
109
+ "validation_fixed_A25": 0.6255183076814955,
110
+ "test_fixed_A25": 0.33085386949493806,
111
+ "test_ranked_A25_diagnostic": 0.3352494115645294,
112
+ "selected_validation_actual_calls": 0.339939560053412,
113
+ "selected_test_actual_calls": 0.43518702322547714
114
+ }
115
+ }
116
+ },
117
+ {
118
+ "key": [
119
+ "grefcoco",
120
+ "groundingdino",
121
+ "Qwen3.8"
122
+ ],
123
+ "areas": {
124
+ "confidence": {
125
+ "validation_fixed_A25": 0.5648886077728582,
126
+ "test_fixed_A25": 0.28625187873975555,
127
+ "test_ranked_A25_diagnostic": 0.29324220854720245,
128
+ "selected_validation_actual_calls": 0.9899501019045611,
129
+ "selected_test_actual_calls": 0.9873805405098829
130
+ },
131
+ "learned": {
132
+ "validation_fixed_A25": 0.669126431934781,
133
+ "test_fixed_A25": 0.3549726342058248,
134
+ "test_ranked_A25_diagnostic": 0.37351898590590704,
135
+ "selected_validation_actual_calls": 0.4199873497786211,
136
+ "selected_test_actual_calls": 0.5745682443354224
137
+ }
138
+ }
139
+ },
140
+ {
141
+ "key": [
142
+ "grefcoco",
143
+ "instancevg",
144
+ "MiniCPM"
145
+ ],
146
+ "areas": {
147
+ "confidence": {
148
+ "validation_fixed_A25": 0.7465176751704266,
149
+ "test_fixed_A25": 0.6639253608598248,
150
+ "test_ranked_A25_diagnostic": 0.6590023537418825,
151
+ "selected_validation_actual_calls": 0.6499402628434886,
152
+ "selected_test_actual_calls": 0.5209426310864078
153
+ },
154
+ "learned": {
155
+ "validation_fixed_A25": 0.7607140347178298,
156
+ "test_fixed_A25": 0.6759152652922327,
157
+ "test_ranked_A25_diagnostic": 0.6777273629583416,
158
+ "selected_validation_actual_calls": 0.22995291306486754,
159
+ "selected_test_actual_calls": 0.30904914499617164
160
+ }
161
+ }
162
+ },
163
+ {
164
+ "key": [
165
+ "grefcoco",
166
+ "instancevg",
167
+ "Qwen3.8"
168
+ ],
169
+ "areas": {
170
+ "confidence": {
171
+ "validation_fixed_A25": 0.7477616136060159,
172
+ "test_fixed_A25": 0.6640728242066756,
173
+ "test_ranked_A25_diagnostic": 0.6611405722712191,
174
+ "selected_validation_actual_calls": 0.7099585353854804,
175
+ "selected_test_actual_calls": 0.6838612710206166
176
+ },
177
+ "learned": {
178
+ "validation_fixed_A25": 0.7778129172816081,
179
+ "test_fixed_A25": 0.7280974392422653,
180
+ "test_ranked_A25_diagnostic": 0.7136828970875989,
181
+ "selected_validation_actual_calls": 0.5599831330381615,
182
+ "selected_test_actual_calls": 0.5355755324277571
183
+ }
184
+ }
185
+ },
186
+ {
187
+ "key": [
188
+ "nlvr2",
189
+ "nlvr2",
190
+ "MiniCPM"
191
+ ],
192
+ "areas": {
193
+ "confidence": {
194
+ "validation_fixed_A25": 0.8284302492122602,
195
+ "test_fixed_A25": 0.833902561526871,
196
+ "test_ranked_A25_diagnostic": 0.835882901628758,
197
+ "selected_validation_actual_calls": 0.8587797192781438,
198
+ "selected_test_actual_calls": 0.8580038745784602
199
+ },
200
+ "learned": {
201
+ "validation_fixed_A25": 0.8359782297336007,
202
+ "test_fixed_A25": 0.8425629618999785,
203
+ "test_ranked_A25_diagnostic": 0.8414221137978044,
204
+ "selected_validation_actual_calls": 0.8498997421942137,
205
+ "selected_test_actual_calls": 0.8465953935567195
206
+ }
207
+ }
208
+ },
209
+ {
210
+ "key": [
211
+ "nlvr2",
212
+ "nlvr2",
213
+ "Qwen3.8"
214
+ ],
215
+ "areas": {
216
+ "confidence": {
217
+ "validation_fixed_A25": 0.831280435405328,
218
+ "test_fixed_A25": 0.8343976465523427,
219
+ "test_ranked_A25_diagnostic": 0.8361340317141422,
220
+ "selected_validation_actual_calls": 0.8587797192781438,
221
+ "selected_test_actual_calls": 0.8580038745784602
222
+ },
223
+ "learned": {
224
+ "validation_fixed_A25": 0.8402320252076769,
225
+ "test_fixed_A25": 0.8436320585491857,
226
+ "test_ranked_A25_diagnostic": 0.8425342613187918,
227
+ "selected_validation_actual_calls": 0.7699799484388428,
228
+ "selected_test_actual_calls": 0.7665925234986009
229
+ }
230
+ }
231
+ },
232
+ {
233
+ "key": [
234
+ "nlvr2",
235
+ "vilt",
236
+ "MiniCPM"
237
+ ],
238
+ "areas": {
239
+ "confidence": {
240
+ "validation_fixed_A25": 0.7926095674591808,
241
+ "test_fixed_A25": 0.7987371744277822,
242
+ "test_ranked_A25_diagnostic": 0.8005237856066586,
243
+ "selected_validation_actual_calls": 0.9598968776854769,
244
+ "selected_test_actual_calls": 0.9593169261677549
245
+ },
246
+ "learned": {
247
+ "validation_fixed_A25": 0.793425952449155,
248
+ "test_fixed_A25": 0.8005094353160652,
249
+ "test_ranked_A25_diagnostic": 0.8015211308028988,
250
+ "selected_validation_actual_calls": 0.9899742194213692,
251
+ "selected_test_actual_calls": 0.9896677907727631
252
+ }
253
+ }
254
+ },
255
+ {
256
+ "key": [
257
+ "nlvr2",
258
+ "vilt",
259
+ "Qwen3.8"
260
+ ],
261
+ "areas": {
262
+ "confidence": {
263
+ "validation_fixed_A25": 0.7940561443712404,
264
+ "test_fixed_A25": 0.7988735021884195,
265
+ "test_ranked_A25_diagnostic": 0.8006242376408124,
266
+ "selected_validation_actual_calls": 0.9598968776854769,
267
+ "selected_test_actual_calls": 0.9593169261677549
268
+ },
269
+ "learned": {
270
+ "validation_fixed_A25": 0.7963477513606417,
271
+ "test_fixed_A25": 0.802590227452106,
272
+ "test_ranked_A25_diagnostic": 0.8037526009901701,
273
+ "selected_validation_actual_calls": 0.9498710971068461,
274
+ "selected_test_actual_calls": 0.9485542082227165
275
+ }
276
+ }
277
+ },
278
+ {
279
+ "key": [
280
+ "construction",
281
+ "yolo26x",
282
+ "MiniCPM"
283
+ ],
284
+ "areas": {
285
+ "confidence": {
286
+ "validation_fixed_A25": 0.17451640525159154,
287
+ "test_fixed_A25": 0.17697501725814352,
288
+ "test_ranked_A25_diagnostic": 0.1716572124279524,
289
+ "selected_validation_actual_calls": 0.9386590584878745,
290
+ "selected_test_actual_calls": 0.9254327563249002
291
+ },
292
+ "learned": {
293
+ "validation_fixed_A25": 0.22914536623152232,
294
+ "test_fixed_A25": 0.21733392403576846,
295
+ "test_ranked_A25_diagnostic": 0.21579610594149268,
296
+ "selected_validation_actual_calls": 0.4893009985734665,
297
+ "selected_test_actual_calls": 0.47203728362183756
298
+ }
299
+ }
300
+ },
301
+ {
302
+ "key": [
303
+ "construction",
304
+ "yolo26x",
305
+ "Qwen3.8"
306
+ ],
307
+ "areas": {
308
+ "confidence": {
309
+ "validation_fixed_A25": 0.17942553980644182,
310
+ "test_fixed_A25": 0.18055274894913714,
311
+ "test_ranked_A25_diagnostic": 0.1793307821359281,
312
+ "selected_validation_actual_calls": 0.9286733238231099,
313
+ "selected_test_actual_calls": 0.9221038615179761
314
+ },
315
+ "learned": {
316
+ "validation_fixed_A25": 0.23466194499464404,
317
+ "test_fixed_A25": 0.21316340540985348,
318
+ "test_ranked_A25_diagnostic": 0.213003148312119,
319
+ "selected_validation_actual_calls": 0.4992867332382311,
320
+ "selected_test_actual_calls": 0.4876830892143808
321
+ }
322
+ }
323
+ },
324
+ {
325
+ "key": [
326
+ "construction",
327
+ "rtdetr_x_fp32",
328
+ "MiniCPM"
329
+ ],
330
+ "areas": {
331
+ "confidence": {
332
+ "validation_fixed_A25": 0.08994758708890332,
333
+ "test_fixed_A25": 0.08325267500178911,
334
+ "test_ranked_A25_diagnostic": 0.08357421552875581,
335
+ "selected_validation_actual_calls": 0.978601997146933,
336
+ "selected_test_actual_calls": 0.9790279627163782
337
+ },
338
+ "learned": {
339
+ "validation_fixed_A25": 0.11165309347543784,
340
+ "test_fixed_A25": 0.09726595269708746,
341
+ "test_ranked_A25_diagnostic": 0.09795552217467457,
342
+ "selected_validation_actual_calls": 0.7788873038516405,
343
+ "selected_test_actual_calls": 0.7882822902796272
344
+ }
345
+ }
346
+ },
347
+ {
348
+ "key": [
349
+ "construction",
350
+ "rtdetr_x_fp32",
351
+ "Qwen3.8"
352
+ ],
353
+ "areas": {
354
+ "confidence": {
355
+ "validation_fixed_A25": 0.10053453664183931,
356
+ "test_fixed_A25": 0.08910633276678806,
357
+ "test_ranked_A25_diagnostic": 0.08907503728424503,
358
+ "selected_validation_actual_calls": 0.9885877318116976,
359
+ "selected_test_actual_calls": 0.9860186418109188
360
+ },
361
+ "learned": {
362
+ "validation_fixed_A25": 0.13205087205171664,
363
+ "test_fixed_A25": 0.10226629753438624,
364
+ "test_ranked_A25_diagnostic": 0.10459671104829708,
365
+ "selected_validation_actual_calls": 0.8288159771754636,
366
+ "selected_test_actual_calls": 0.831225033288948
367
+ }
368
+ }
369
+ }
370
+ ]
paper_results/data/comparisons.json ADDED
The diff for this file is too large to render. See raw diff
 
paper_results/data/earlier_results.json ADDED
The diff for this file is too large to render. See raw diff
 
paper_results/data/joint_MiniCPM_comparison_20260910.json ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "expert": "cub",
4
+ "endpoint": "MiniCPM",
5
+ "single_task_test_fixed25": 0.7119433897134967,
6
+ "multi_task_test_fixed25": 0.7079737659647911,
7
+ "delta_multi_minus_single": -0.003969623748705575
8
+ },
9
+ {
10
+ "expert": "glsim",
11
+ "endpoint": "MiniCPM",
12
+ "single_task_test_fixed25": 0.759233690024163,
13
+ "multi_task_test_fixed25": 0.7745944080082844,
14
+ "delta_multi_minus_single": 0.01536071798412142
15
+ },
16
+ {
17
+ "expert": "nlvr2",
18
+ "endpoint": "MiniCPM",
19
+ "single_task_test_fixed25": 0.8346846523642104,
20
+ "multi_task_test_fixed25": 0.8309535768099304,
21
+ "delta_multi_minus_single": -0.0037310755542799834
22
+ },
23
+ {
24
+ "expert": "vilt",
25
+ "endpoint": "MiniCPM",
26
+ "single_task_test_fixed25": 0.8046925450240368,
27
+ "multi_task_test_fixed25": 0.802611752887996,
28
+ "delta_multi_minus_single": -0.002080792136040732
29
+ },
30
+ {
31
+ "expert": "groundingdino",
32
+ "endpoint": "MiniCPM",
33
+ "single_task_test_fixed25": 0.3598105663159686,
34
+ "multi_task_test_fixed25": 0.37004792558772653,
35
+ "delta_multi_minus_single": 0.010237359271757951
36
+ },
37
+ {
38
+ "expert": "instancevg",
39
+ "endpoint": "MiniCPM",
40
+ "single_task_test_fixed25": 0.6394237586138446,
41
+ "multi_task_test_fixed25": 0.6243654822335025,
42
+ "delta_multi_minus_single": -0.015058276380342073
43
+ },
44
+ {
45
+ "expert": "yolo26x",
46
+ "endpoint": "MiniCPM",
47
+ "single_task_test_fixed25": 0.18032734231886507,
48
+ "multi_task_test_fixed25": 0.1836164473038217,
49
+ "delta_multi_minus_single": 0.003289104984956631
50
+ },
51
+ {
52
+ "expert": "rtdetr_x_fp32",
53
+ "endpoint": "MiniCPM",
54
+ "single_task_test_fixed25": 0.10367040308754236,
55
+ "multi_task_test_fixed25": 0.11260360812088349,
56
+ "delta_multi_minus_single": 0.008933205033341135
57
+ }
58
+ ]
paper_results/data/joint_Qwen3.8_comparison_20260910.json ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "expert": "cub",
4
+ "endpoint": "Qwen3.8",
5
+ "single_task_test_fixed25": 0.7619951674145667,
6
+ "multi_task_test_fixed25": 0.7782188470831896,
7
+ "delta_multi_minus_single": 0.016223679668622815
8
+ },
9
+ {
10
+ "expert": "glsim",
11
+ "endpoint": "Qwen3.8",
12
+ "single_task_test_fixed25": 0.8301691404901622,
13
+ "multi_task_test_fixed25": 0.8448394891266828,
14
+ "delta_multi_minus_single": 0.014670348636520614
15
+ },
16
+ {
17
+ "expert": "nlvr2",
18
+ "endpoint": "Qwen3.8",
19
+ "single_task_test_fixed25": 0.8372677046710196,
20
+ "multi_task_test_fixed25": 0.8337518834756403,
21
+ "delta_multi_minus_single": -0.0035158211953792984
22
+ },
23
+ {
24
+ "expert": "vilt",
25
+ "endpoint": "Qwen3.8",
26
+ "single_task_test_fixed25": 0.8045490421181029,
27
+ "multi_task_test_fixed25": 0.8067733371600775,
28
+ "delta_multi_minus_single": 0.0022242950419746332
29
+ },
30
+ {
31
+ "expert": "groundingdino",
32
+ "endpoint": "Qwen3.8",
33
+ "single_task_test_fixed25": 0.4223974137197629,
34
+ "multi_task_test_fixed25": 0.45991549215892014,
35
+ "delta_multi_minus_single": 0.03751807843915722
36
+ },
37
+ {
38
+ "expert": "instancevg",
39
+ "endpoint": "Qwen3.8",
40
+ "single_task_test_fixed25": 0.7224853245611548,
41
+ "multi_task_test_fixed25": 0.7053001729858492,
42
+ "delta_multi_minus_single": -0.017185151575305535
43
+ },
44
+ {
45
+ "expert": "yolo26x",
46
+ "endpoint": "Qwen3.8",
47
+ "single_task_test_fixed25": 0.19009376433100783,
48
+ "multi_task_test_fixed25": 0.19603467084956394,
49
+ "delta_multi_minus_single": 0.0059409065185561105
50
+ },
51
+ {
52
+ "expert": "rtdetr_x_fp32",
53
+ "endpoint": "Qwen3.8",
54
+ "single_task_test_fixed25": 0.10483176081963008,
55
+ "multi_task_test_fixed25": 0.12038495915476813,
56
+ "delta_multi_minus_single": 0.015553198335138047
57
+ }
58
+ ]
paper_results/data/layout_verification.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "old_distribution": {
3
+ "width_pt": 523.1995239257812,
4
+ "height_pt": 309.7997741699219,
5
+ "height_at_7_16_in": 4.239618504261674
6
+ },
7
+ "old_budget": {
8
+ "width_pt": 526.7994995117188,
9
+ "height_pt": 237.7997589111328,
10
+ "height_at_7_16_in": 3.2320575007794505
11
+ },
12
+ "combined": {
13
+ "width_pt": 515.52001953125,
14
+ "height_pt": 324.0,
15
+ "height_at_7_16_in": 4.499999829510743
16
+ },
17
+ "reduction_percent": 39.77255134624947,
18
+ "old_pages": 16,
19
+ "new_pages": 14,
20
+ "new_figure_and_section_pages": [
21
+ 6,
22
+ 7
23
+ ],
24
+ "changed_sections": [
25
+ "sections\\appendix.tex",
26
+ "sections\\experiments.tex"
27
+ ],
28
+ "figure1_identical": true
29
+ }
paper_results/data/main_table_cost_qwen_20260912/build_figures.py ADDED
@@ -0,0 +1,101 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Main-table curves plus fresh measured component costs; no threshold re-selection."""
2
+ from pathlib import Path
3
+ import csv
4
+ import json
5
+ import numpy as np
6
+ import matplotlib
7
+ matplotlib.use('Agg')
8
+ import matplotlib.pyplot as plt
9
+ from matplotlib.ticker import MaxNLocator,LogLocator,NullLocator
10
+
11
+ root=Path(__file__).resolve().parent
12
+ groups=json.loads((root/'main_table_results.json').read_text(encoding='utf-8'))['groups']
13
+ components={}
14
+ curves=[]
15
+ selected=[]
16
+ for g,tag in zip(groups,['qwen_gref','qwen']):
17
+ expert=g['key'][1]
18
+ spec=json.loads((root/f'components/{expert}_specialist.json').read_text())
19
+ router=json.loads((root/f'components/{expert}_router.json').read_text())
20
+ large=json.loads((root/f'components/{tag}_latency.json').read_text())
21
+ flops=json.loads((root/f'components/{tag}_flops.json').read_text())
22
+ assert spec['completed'] and router['status']==large['status']==flops['status']=='complete'
23
+ assert router['run']==g['run'] and router['selected_decisions_match_main']
24
+ assert abs(router['actual_call_fraction']-g['methods']['learned']['test_point']['actual_fraction'])<1e-12
25
+ key='|'.join(g['key'])
26
+ c={'specialist_gflops':spec['mean_counted_gflops_per_sample'],
27
+ 'router_gflops':router['mean_gflops_per_input'],
28
+ 'large_gflops':flops['mean_prefill_gflops_counted']+large['mean_completion_tokens']*flops['mean_decode_one_token_gflops_counted'],
29
+ 'specialist_ms':spec['mean_end_to_end_latency_ms_per_sample'],
30
+ 'router_ms':router['mean_latency_ms_per_input'],
31
+ 'large_ms':large['mean_batch_wall_ms_per_sample'],
32
+ 'large_samples':large['measured_count'], 'flops_probes':flops['sample_count'],
33
+ 'specialist_samples':spec['sample_count'],'router_samples':router['sample_count'],'batch_size':large['batch_size']}
34
+ components[key]=c
35
+ for method in ['learned','confidence']:
36
+ for kind,points in [('curve',g['methods'][method]['test_curve']),('selected',[g['methods'][method]['test_point']])]:
37
+ for p in points:
38
+ rate=p['actual_fraction']
39
+ row={'pair':key,'method':method,'nominal_validation_budget':p['budget'],'actual_test_call_fraction':rate,
40
+ 'test_metric':p['metric'],'estimated_gflops_per_input':c['specialist_gflops']+(c['router_gflops'] if method=='learned' else 0)+rate*c['large_gflops'],
41
+ 'estimated_time_ms_per_input':c['specialist_ms']+(c['router_ms'] if method=='learned' else 0)+rate*c['large_ms']}
42
+ (curves if kind=='curve' else selected).append(row)
43
+ for name,rows in [('full_cost_performance_curves',curves),('selected_points',selected),('fixed_budget_cost_performance',[p for p in curves if p['nominal_validation_budget'] in [0,.05,.1,.25,.5,.75,1]])]:
44
+ with (root/(name+'.csv')).open('w',encoding='utf-8',newline='') as f:
45
+ writer=csv.DictWriter(f,fieldnames=list(rows[0]));writer.writeheader();writer.writerows(rows)
46
+ summary={'components':components,'selected_points':selected,'protocol':
47
+ {'method':'original main-table logit contrast; original checkpoints and validation thresholds; seed 42',
48
+ 'large_timing':'64 measured examples; batch size one; same full task prompt; two separate warmup examples',
49
+ 'cost':'serial component estimates, not a timed integrated service; includes active distilled encoders and routing head only for learned',
50
+ 'flops':'multiply-add=2; supported operators; mean prefill (including visual embedding) plus average generated length times one-token decode probe',
51
+ 'scope_limits':'mean call cost extrapolated across policies; no policy-specific input/output-length costs; specialist-feature extraction overhead is not separately measured; confidence threshold overhead not separately timed'}}
52
+ (root/'cost_summary.json').write_text(json.dumps(summary,indent=2),encoding='utf-8')
53
+
54
+ plt.rcParams.update({'font.family':'serif','font.serif':['Times New Roman'],'font.size':7,
55
+ 'axes.labelsize':7,'xtick.labelsize':6.5,'ytick.labelsize':6.5,'legend.fontsize':7,
56
+ 'pdf.fonttype':42,'ps.fonttype':42,'axes.spines.top':False,'axes.spines.right':False,
57
+ 'axes.linewidth':.55,'mathtext.fontset':'stix'})
58
+ styles=[('learned','learned','#176f91','-'),('confidence','conf.','#c78324','--')]
59
+ keys=list(components)
60
+
61
+ def panel(ax,key,field):
62
+ for method,label,color,ls in styles:
63
+ pts=[p for p in curves if p['pair']==key and p['method']==method]
64
+ assert len(pts)==101
65
+ x=np.array([p[field] for p in pts]);y=np.array([p['test_metric']*100 for p in pts])
66
+ assert np.all(np.diff(x)>=-1e-7)
67
+ ax.plot(x,y,color=color,ls=ls,lw=1.05,label=label)
68
+ p=next(p for p in selected if p['pair']==key and p['method']==method)
69
+ ax.scatter(p[field],100*p['test_metric'],color=color,s=13,edgecolor='white',linewidth=.4,zorder=5)
70
+ ax.grid(axis='y',color='#dce1e4',lw=.4)
71
+ ax.tick_params(length=2.3,pad=1.5)
72
+ ax.yaxis.set_major_locator(MaxNLocator(nbins=3))
73
+ if 'gflops' in field:
74
+ ax.set_xscale('log');ax.xaxis.set_major_locator(LogLocator(base=10,numticks=4));ax.xaxis.set_minor_locator(NullLocator())
75
+ ax.set_xlabel('Est. GFLOPs/input',labelpad=1)
76
+ else:
77
+ ax.xaxis.set_major_locator(MaxNLocator(nbins=3));ax.set_xlabel('Est. time (ms/input)',labelpad=1)
78
+ ax.set_ylabel('Acc. (%)' if key.startswith('grefcoco') else 'Macro-F1 (%)',labelpad=1)
79
+
80
+ for field,stem in [('estimated_gflops_per_input','flops_performance'),('estimated_time_ms_per_input','latency_performance')]:
81
+ fig,axes=plt.subplots(1,2,figsize=(3.44,1.50))
82
+ fig.subplots_adjust(left=.115,right=.995,bottom=.31,top=.83,wspace=.43)
83
+ for ax,key,title in zip(axes,keys,['gRefCOCO / Q','Construction / Q']):
84
+ panel(ax,key,field)
85
+ ax.text(.5,1.10,title,transform=ax.transAxes,ha='center',fontsize=7)
86
+ h,l=axes[0].get_legend_handles_labels()
87
+ fig.legend(h,l,loc='lower center',bbox_to_anchor=(.53,-.025),ncol=2,frameon=False,handlelength=2.2,columnspacing=1.8)
88
+ fig.savefig(root/(stem+'.pdf'));fig.savefig(root/(stem+'.png'),dpi=400)
89
+ plt.close(fig)
90
+
91
+ fig,axes=plt.subplots(2,2,figsize=(3.44,2.05))
92
+ fig.subplots_adjust(left=.115,right=.995,bottom=.20,top=.89,wspace=.43,hspace=.64)
93
+ for col,key in enumerate(keys):
94
+ for row,field in enumerate(['estimated_gflops_per_input','estimated_time_ms_per_input']):
95
+ panel(axes[row,col],key,field)
96
+ axes[0,col].text(.5,1.14,'gRefCOCO / Q' if col==0 else 'Construction / Q',transform=axes[0,col].transAxes,ha='center',fontsize=7)
97
+ h,l=axes[0,0].get_legend_handles_labels()
98
+ fig.legend(h,l,loc='lower center',bbox_to_anchor=(.53,-.005),ncol=2,frameon=False,handlelength=2.2,columnspacing=1.8)
99
+ fig.savefig(root/'cost_compact.pdf');fig.savefig(root/'cost_compact.png',dpi=400)
100
+ plt.close(fig)
101
+ print(json.dumps(summary,indent=2))
paper_results/data/main_table_cost_qwen_20260912/components/instancevg_router.json ADDED
@@ -0,0 +1,239 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "status": "complete",
3
+ "run": "shared_12_instancevg_Qwen3.8",
4
+ "config": {
5
+ "fraction": 1.0,
6
+ "seed": 42,
7
+ "freeze_encoder": true,
8
+ "fusion": "concat",
9
+ "harm_cost": 1.0,
10
+ "batch_size": 256,
11
+ "weight_decay": 0.01,
12
+ "max_epochs": 50,
13
+ "patience": 8,
14
+ "cpu_threads": 1,
15
+ "student": "8M",
16
+ "representation_dim": 128,
17
+ "lr": 0.003,
18
+ "target": "four_state",
19
+ "strategy": "natural",
20
+ "hidden_percent": 90,
21
+ "expert": "instancevg",
22
+ "task": "grefcoco",
23
+ "endpoint": "Qwen3.8",
24
+ "branches": [
25
+ "image",
26
+ "text",
27
+ "hidden",
28
+ "confidence",
29
+ "output"
30
+ ],
31
+ "hidden_layer": "joint_encoder__layer_11",
32
+ "id": "shared_12_instancevg_Qwen3.8",
33
+ "stage": "shared_mandatory_inputs"
34
+ },
35
+ "checkpoint": "/mnt/HithinkOmniSSD/user_workspace/caisihang/research/ICASSP_ModelCollaboration/outputs/router_shared_image_hidden_20260910/runs/shared_12_instancevg_Qwen3.8/best.pt",
36
+ "score": "logit_rescue-logit_harm",
37
+ "test_rows_recomputed": 35263,
38
+ "selected_decisions_match_main": true,
39
+ "max_score_abs_difference": 0.0,
40
+ "actual_call_fraction": 0.5355755324277571,
41
+ "sample_count": 64,
42
+ "sample_ids": [
43
+ "testB-5056-14012",
44
+ "testB-31505-86708",
45
+ "testA-57489-158264",
46
+ "testA-38865-107108",
47
+ "testB-47361-130361",
48
+ "testA-5476-15123",
49
+ "testA-71404-196892",
50
+ "testA-61030-168115",
51
+ "testB-35969-99063",
52
+ "testB-1675-4724",
53
+ "testA-30414-83641",
54
+ "testB-47742-131425",
55
+ "testB-77456-213280",
56
+ "testA-59283-163253",
57
+ "testA-57390-157996",
58
+ "testA-77229-212665",
59
+ "testA-6784-18703",
60
+ "testB-11434-31542",
61
+ "testB-11675-32181",
62
+ "testB-73046-201304",
63
+ "testA-28837-79356",
64
+ "testA-4563-12619",
65
+ "testB-30168-82996",
66
+ "testB-92461-254721",
67
+ "testA-58293-160462",
68
+ "testA-13050-35910",
69
+ "testA-39320-108309",
70
+ "testB-36423-100313",
71
+ "testA-43418-119528",
72
+ "testB-23411-64351",
73
+ "testB-77730-214031",
74
+ "testA-7232-19907",
75
+ "testA-17738-48695",
76
+ "testA-44914-123593",
77
+ "testB-92861-255854",
78
+ "testA-69678-192132",
79
+ "testA-68785-189658",
80
+ "testB-62911-173380",
81
+ "testB-19029-52242",
82
+ "testA-66284-182759",
83
+ "testB-19170-52626",
84
+ "testB-8468-23291",
85
+ "testB-72940-201009",
86
+ "testB-74045-203983",
87
+ "testB-93380-257275",
88
+ "testB-49369-135990",
89
+ "testA-58046-159759",
90
+ "testA-57437-158132",
91
+ "testA-5754-15895",
92
+ "testA-85590-235577",
93
+ "testA-92278-254227",
94
+ "testB-80373-221257",
95
+ "testA-80383-221294",
96
+ "testB-71197-196311",
97
+ "testB-48080-132352",
98
+ "testB-18351-50380",
99
+ "testB-1089-3077",
100
+ "testB-92466-254737",
101
+ "testB-53065-146099",
102
+ "testB-42611-117349",
103
+ "testA-57690-158796",
104
+ "testA-65173-179681",
105
+ "testB-90675-249707",
106
+ "testB-74709-205815"
107
+ ],
108
+ "batch_size": 1,
109
+ "latency_ms": [
110
+ 10.095497942529619,
111
+ 10.442811995744705,
112
+ 9.535272954963148,
113
+ 8.56779795140028,
114
+ 28.526668902486563,
115
+ 14.110558084212244,
116
+ 19.977235002443194,
117
+ 21.489594015292823,
118
+ 16.842726967297494,
119
+ 12.316308100707829,
120
+ 12.318658991716802,
121
+ 13.136874069459736,
122
+ 16.18148700799793,
123
+ 11.749238008633256,
124
+ 12.584974057972431,
125
+ 12.680649990215898,
126
+ 11.369985993951559,
127
+ 14.251800021156669,
128
+ 14.000480994582176,
129
+ 12.42606493178755,
130
+ 13.526970986276865,
131
+ 11.346371960826218,
132
+ 11.965820915065706,
133
+ 13.418521964922547,
134
+ 13.269996969029307,
135
+ 13.868742040358484,
136
+ 11.83428696822375,
137
+ 13.831221032887697,
138
+ 12.576585984788835,
139
+ 12.408879934810102,
140
+ 12.892312020994723,
141
+ 15.224601957015693,
142
+ 13.460875954478979,
143
+ 11.89423305913806,
144
+ 10.926949908025563,
145
+ 157.88777999114245,
146
+ 16.57543599139899,
147
+ 29.21812597196549,
148
+ 23.506848025135696,
149
+ 13.830928015522659,
150
+ 14.57549398764968,
151
+ 13.596423901617527,
152
+ 16.056897002272308,
153
+ 12.610359000973403,
154
+ 17.024922999553382,
155
+ 12.336026062257588,
156
+ 14.77054797578603,
157
+ 13.220850960351527,
158
+ 19.9308970477432,
159
+ 16.4189770584926,
160
+ 17.153396038338542,
161
+ 31.296705012209713,
162
+ 20.386314019560814,
163
+ 14.672313001938164,
164
+ 15.437994967214763,
165
+ 10.980968945659697,
166
+ 15.224996954202652,
167
+ 12.009209021925926,
168
+ 12.397779966704547,
169
+ 13.347716070711613,
170
+ 15.58609597850591,
171
+ 17.80865201726556,
172
+ 13.338097953237593,
173
+ 13.845606008544564
174
+ ],
175
+ "mean_latency_ms_per_input": 17.032772149832454,
176
+ "head_mean_latency_ms": 0.6086819972842932,
177
+ "gflops_per_input": [
178
+ 3.642171392,
179
+ 3.642171392,
180
+ 3.642171392,
181
+ 3.642171392,
182
+ 3.642171392,
183
+ 3.642171392,
184
+ 3.642171392,
185
+ 3.642171392
186
+ ],
187
+ "mean_gflops_per_input": 3.642171392,
188
+ "operators": [
189
+ {
190
+ "aten.convolution": 228802560,
191
+ "aten.bmm": 691709440,
192
+ "aten._scaled_dot_product_flash_attention": 236027904,
193
+ "aten.addmm": 2485631488
194
+ },
195
+ {
196
+ "aten.convolution": 228802560,
197
+ "aten.bmm": 691709440,
198
+ "aten._scaled_dot_product_flash_attention": 236027904,
199
+ "aten.addmm": 2485631488
200
+ },
201
+ {
202
+ "aten.convolution": 228802560,
203
+ "aten.bmm": 691709440,
204
+ "aten._scaled_dot_product_flash_attention": 236027904,
205
+ "aten.addmm": 2485631488
206
+ },
207
+ {
208
+ "aten.convolution": 228802560,
209
+ "aten.bmm": 691709440,
210
+ "aten._scaled_dot_product_flash_attention": 236027904,
211
+ "aten.addmm": 2485631488
212
+ },
213
+ {
214
+ "aten.convolution": 228802560,
215
+ "aten.bmm": 691709440,
216
+ "aten._scaled_dot_product_flash_attention": 236027904,
217
+ "aten.addmm": 2485631488
218
+ },
219
+ {
220
+ "aten.convolution": 228802560,
221
+ "aten.bmm": 691709440,
222
+ "aten._scaled_dot_product_flash_attention": 236027904,
223
+ "aten.addmm": 2485631488
224
+ },
225
+ {
226
+ "aten.convolution": 228802560,
227
+ "aten.bmm": 691709440,
228
+ "aten._scaled_dot_product_flash_attention": 236027904,
229
+ "aten.addmm": 2485631488
230
+ },
231
+ {
232
+ "aten.convolution": 228802560,
233
+ "aten.bmm": 691709440,
234
+ "aten._scaled_dot_product_flash_attention": 236027904,
235
+ "aten.addmm": 2485631488
236
+ }
237
+ ],
238
+ "scope": "image loading, bilinear resize, tokenize when used, two image slots (including gray placeholder), active distilled encoders, feature assembly and original routing head; cached specialist features; supported FLOPs only"
239
+ }
paper_results/data/main_table_cost_qwen_20260912/components/instancevg_specialist.json ADDED
@@ -0,0 +1,254 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "component": "specialist",
3
+ "expert": "instancevg",
4
+ "task": "grefcoco",
5
+ "device": "cuda:0",
6
+ "checkpoint": "/mnt/HithinkOmniSSD/user_workspace/caisihang/research/ICASSP_ModelCollaboration/models/grefcoco/instancevg/InstanceVG-grefcoco.pth",
7
+ "sample_count": 64,
8
+ "batch_size": 1,
9
+ "sample_ids": [
10
+ "testA-30844-84861",
11
+ "testB-44353-122077",
12
+ "testB-36889-101643",
13
+ "testA-49055-135087",
14
+ "testA-81245-223633",
15
+ "testB-37669-103833",
16
+ "testA-5038-13943",
17
+ "testB-79646-219274",
18
+ "testA-469-1309",
19
+ "testB-46644-128438",
20
+ "testA-24562-67515",
21
+ "testA-47070-129599",
22
+ "testA-54554-150141",
23
+ "testB-12382-34099",
24
+ "testB-23014-63279",
25
+ "testB-79647-219275",
26
+ "testB-69479-191564",
27
+ "testA-15582-42797",
28
+ "testB-56692-156000",
29
+ "testB-42726-117660",
30
+ "testA-80455-221494",
31
+ "testB-33115-91174",
32
+ "testA-62940-173455",
33
+ "testB-31494-86670",
34
+ "testA-54586-150233",
35
+ "testA-81445-224177",
36
+ "testB-8417-23141",
37
+ "testA-24438-67190",
38
+ "testA-59285-163255",
39
+ "testA-73258-201893",
40
+ "testB-945-2633",
41
+ "testA-54307-149426",
42
+ "testB-60557-166822",
43
+ "testA-81622-224665",
44
+ "testA-69466-191525",
45
+ "testA-60731-167321",
46
+ "testA-7857-21644",
47
+ "testA-44142-121499",
48
+ "testB-66162-182407",
49
+ "testA-31764-87406",
50
+ "testB-38637-106482",
51
+ "testB-52921-145709",
52
+ "testB-65796-181394",
53
+ "testB-2936-8113",
54
+ "testA-79438-218690",
55
+ "testA-57397-158016",
56
+ "testB-23920-65743",
57
+ "testB-15419-42358",
58
+ "testA-10078-27739",
59
+ "testA-43890-120793",
60
+ "testA-55116-151689",
61
+ "testB-27487-75582",
62
+ "testA-26856-73830",
63
+ "testB-50845-140107",
64
+ "testA-29199-80374",
65
+ "testB-65935-181778",
66
+ "testA-17736-48684",
67
+ "testB-69477-191553",
68
+ "testB-22290-61227",
69
+ "testA-58476-160950",
70
+ "testB-36265-99859",
71
+ "testA-5734-15837",
72
+ "testB-67952-187353",
73
+ "testB-47054-129555"
74
+ ],
75
+ "model_load_seconds_excluded_from_latency": true,
76
+ "forward_latency_ms_per_sample": [
77
+ 41.24195098876953,
78
+ 41.74086380004883,
79
+ 40.36540985107422,
80
+ 44.46102523803711,
81
+ 41.2421760559082,
82
+ 40.76591873168945,
83
+ 41.14704132080078,
84
+ 40.88172912597656,
85
+ 41.70716857910156,
86
+ 41.191871643066406,
87
+ 39.91231918334961,
88
+ 39.29497528076172,
89
+ 40.70723342895508,
90
+ 38.920894622802734,
91
+ 39.32780838012695,
92
+ 38.89958572387695,
93
+ 38.59721755981445,
94
+ 38.47331237792969,
95
+ 39.8744010925293,
96
+ 39.87260818481445,
97
+ 41.48284912109375,
98
+ 40.55779266357422,
99
+ 39.05420684814453,
100
+ 39.052734375,
101
+ 39.143775939941406,
102
+ 39.82249450683594,
103
+ 39.543006896972656,
104
+ 39.368160247802734,
105
+ 40.661216735839844,
106
+ 39.092193603515625,
107
+ 39.85852813720703,
108
+ 40.33164978027344,
109
+ 38.89593505859375,
110
+ 39.29216003417969,
111
+ 39.20943832397461,
112
+ 40.946590423583984,
113
+ 39.44643020629883,
114
+ 40.220863342285156,
115
+ 39.61030578613281,
116
+ 39.639007568359375,
117
+ 39.78163146972656,
118
+ 39.223201751708984,
119
+ 39.00921630859375,
120
+ 38.99894332885742,
121
+ 41.13721466064453,
122
+ 39.94972610473633,
123
+ 41.41766357421875,
124
+ 39.28073501586914,
125
+ 39.721214294433594,
126
+ 39.24937438964844,
127
+ 39.40460968017578,
128
+ 41.30179214477539,
129
+ 42.94054412841797,
130
+ 41.5063362121582,
131
+ 44.03267288208008,
132
+ 44.55561447143555,
133
+ 44.24995040893555,
134
+ 44.87868881225586,
135
+ 45.14652633666992,
136
+ 44.85055923461914,
137
+ 42.39228820800781,
138
+ 41.121856689453125,
139
+ 45.708351135253906,
140
+ 41.8752326965332
141
+ ],
142
+ "end_to_end_latency_ms_per_sample": [
143
+ 47.29630902875215,
144
+ 46.27182998228818,
145
+ 47.56885801907629,
146
+ 45.48592597711831,
147
+ 45.399176073260605,
148
+ 46.442538034170866,
149
+ 45.95420998521149,
150
+ 46.45739798434079,
151
+ 44.90085388533771,
152
+ 43.80481690168381,
153
+ 44.848884106613696,
154
+ 43.67590101901442,
155
+ 46.40063806436956,
156
+ 44.26892101764679,
157
+ 44.44277100265026,
158
+ 44.71806308720261,
159
+ 44.62323593907058,
160
+ 44.6264409692958,
161
+ 44.96095096692443,
162
+ 44.103492982685566,
163
+ 45.13338196557015,
164
+ 44.1846780013293,
165
+ 48.1606520479545,
166
+ 46.43543600104749,
167
+ 46.19023704435676,
168
+ 45.247678994201124,
169
+ 44.801001087762415,
170
+ 45.467636082321405,
171
+ 44.22409099061042,
172
+ 44.7448689956218,
173
+ 58.0457029864192,
174
+ 46.10048606991768,
175
+ 44.56539999227971,
176
+ 44.47658592835069,
177
+ 46.05250502936542,
178
+ 44.743441976606846,
179
+ 44.77924096863717,
180
+ 43.877715012058616,
181
+ 44.05005706939846,
182
+ 42.87725198082626,
183
+ 43.836933095008135,
184
+ 45.634721987880766,
185
+ 44.748114072717726,
186
+ 46.21323198080063,
187
+ 52.24333005025983,
188
+ 48.90036000870168,
189
+ 48.12136595137417,
190
+ 47.65231907367706,
191
+ 46.71858798246831,
192
+ 48.516221111640334,
193
+ 43.90157596208155,
194
+ 46.83465394191444,
195
+ 46.49809293914586,
196
+ 42.6300170365721,
197
+ 47.39421408157796,
198
+ 48.35387098137289,
199
+ 48.176085110753775,
200
+ 48.29539393540472,
201
+ 47.32040502130985,
202
+ 46.35261802468449,
203
+ 47.482443042099476,
204
+ 47.437409055419266,
205
+ 47.27676196489483,
206
+ 48.198972013778985
207
+ ],
208
+ "mean_forward_latency_ms_per_sample": 40.71232491731644,
209
+ "median_forward_latency_ms_per_sample": 40.08529472351074,
210
+ "mean_end_to_end_latency_ms_per_sample": 46.08042127620138,
211
+ "median_end_to_end_latency_ms_per_sample": 45.79446598654613,
212
+ "flop_profiles": [
213
+ {
214
+ "batch_size": 1,
215
+ "flops_counted": 7502680064,
216
+ "gflops_counted_per_sample": 7.502680064,
217
+ "operators": {
218
+ "aten.bmm": 7500632064,
219
+ "aten.baddbmm": 2048000
220
+ }
221
+ },
222
+ {
223
+ "batch_size": 1,
224
+ "flops_counted": 7502680064,
225
+ "gflops_counted_per_sample": 7.502680064,
226
+ "operators": {
227
+ "aten.bmm": 7500632064,
228
+ "aten.baddbmm": 2048000
229
+ }
230
+ },
231
+ {
232
+ "batch_size": 1,
233
+ "flops_counted": 7502680064,
234
+ "gflops_counted_per_sample": 7.502680064,
235
+ "operators": {
236
+ "aten.bmm": 7500632064,
237
+ "aten.baddbmm": 2048000
238
+ }
239
+ },
240
+ {
241
+ "batch_size": 1,
242
+ "flops_counted": 7502680064,
243
+ "gflops_counted_per_sample": 7.502680064,
244
+ "operators": {
245
+ "aten.bmm": 7500632064,
246
+ "aten.baddbmm": 2048000
247
+ }
248
+ }
249
+ ],
250
+ "mean_counted_gflops_per_sample": 7.502680064,
251
+ "flops_scope": "GPU FlopCounterMode dense products, convolutions and supported attention; multiply-add counted as 2 FLOPs. Elementwise, normalization, custom ops and postprocessing may be missing.",
252
+ "latency_scope": "A800 wall-clock CUDA forward latency and end-to-end latency including image read/preprocess, excluding model load; batch processing uses the recorded batch size.",
253
+ "completed": true
254
+ }
paper_results/data/main_table_cost_qwen_20260912/components/qwen_flops.json ADDED
@@ -0,0 +1,341 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "status": "complete",
3
+ "model_kind": "Qwen3.8",
4
+ "task": "construction",
5
+ "model_path": "/mnt/HithinkOmniSSD/user_workspace/caisihang/checkpoints/Qwen3.8-27B-FP8",
6
+ "sample_count": 8,
7
+ "records": [
8
+ {
9
+ "sample_id": "0000184",
10
+ "prompt_tokens_in_reference_tensor": 1705,
11
+ "input_shapes": {
12
+ "input_ids": [
13
+ 1,
14
+ 1705
15
+ ],
16
+ "attention_mask": [
17
+ 1,
18
+ 1705
19
+ ],
20
+ "mm_token_type_ids": [
21
+ 1,
22
+ 1705
23
+ ],
24
+ "pixel_values": [
25
+ 3888,
26
+ 1536
27
+ ],
28
+ "image_grid_thw": [
29
+ 1,
30
+ 3
31
+ ]
32
+ },
33
+ "prefill_flops_counted": 90083398713024,
34
+ "decode_one_token_flops_counted": 51922600128,
35
+ "prefill_gflops_counted": 90083.398713024,
36
+ "decode_one_token_gflops_counted": 51.922600128,
37
+ "prefill_operators": {
38
+ "aten.convolution": 20475543552,
39
+ "aten.addmm": 3283626885120,
40
+ "aten.bmm": 3741356981952,
41
+ "aten.mm": 83037939302400
42
+ },
43
+ "decode_operators": {
44
+ "aten.bmm": 670826688,
45
+ "aten.mm": 51243909120,
46
+ "aten.convolution": 7864320
47
+ }
48
+ },
49
+ {
50
+ "sample_id": "0001955",
51
+ "prompt_tokens_in_reference_tensor": 1705,
52
+ "input_shapes": {
53
+ "input_ids": [
54
+ 1,
55
+ 1705
56
+ ],
57
+ "attention_mask": [
58
+ 1,
59
+ 1705
60
+ ],
61
+ "mm_token_type_ids": [
62
+ 1,
63
+ 1705
64
+ ],
65
+ "pixel_values": [
66
+ 3888,
67
+ 1536
68
+ ],
69
+ "image_grid_thw": [
70
+ 1,
71
+ 3
72
+ ]
73
+ },
74
+ "prefill_flops_counted": 90083398713024,
75
+ "decode_one_token_flops_counted": 51922600128,
76
+ "prefill_gflops_counted": 90083.398713024,
77
+ "decode_one_token_gflops_counted": 51.922600128,
78
+ "prefill_operators": {
79
+ "aten.convolution": 20475543552,
80
+ "aten.addmm": 3283626885120,
81
+ "aten.bmm": 3741356981952,
82
+ "aten.mm": 83037939302400
83
+ },
84
+ "decode_operators": {
85
+ "aten.bmm": 670826688,
86
+ "aten.mm": 51243909120,
87
+ "aten.convolution": 7864320
88
+ }
89
+ },
90
+ {
91
+ "sample_id": "0000840",
92
+ "prompt_tokens_in_reference_tensor": 1705,
93
+ "input_shapes": {
94
+ "input_ids": [
95
+ 1,
96
+ 1705
97
+ ],
98
+ "attention_mask": [
99
+ 1,
100
+ 1705
101
+ ],
102
+ "mm_token_type_ids": [
103
+ 1,
104
+ 1705
105
+ ],
106
+ "pixel_values": [
107
+ 3888,
108
+ 1536
109
+ ],
110
+ "image_grid_thw": [
111
+ 1,
112
+ 3
113
+ ]
114
+ },
115
+ "prefill_flops_counted": 90083398713024,
116
+ "decode_one_token_flops_counted": 51922600128,
117
+ "prefill_gflops_counted": 90083.398713024,
118
+ "decode_one_token_gflops_counted": 51.922600128,
119
+ "prefill_operators": {
120
+ "aten.convolution": 20475543552,
121
+ "aten.addmm": 3283626885120,
122
+ "aten.bmm": 3741356981952,
123
+ "aten.mm": 83037939302400
124
+ },
125
+ "decode_operators": {
126
+ "aten.bmm": 670826688,
127
+ "aten.mm": 51243909120,
128
+ "aten.convolution": 7864320
129
+ }
130
+ },
131
+ {
132
+ "sample_id": "0003887",
133
+ "prompt_tokens_in_reference_tensor": 1705,
134
+ "input_shapes": {
135
+ "input_ids": [
136
+ 1,
137
+ 1705
138
+ ],
139
+ "attention_mask": [
140
+ 1,
141
+ 1705
142
+ ],
143
+ "mm_token_type_ids": [
144
+ 1,
145
+ 1705
146
+ ],
147
+ "pixel_values": [
148
+ 3888,
149
+ 1536
150
+ ],
151
+ "image_grid_thw": [
152
+ 1,
153
+ 3
154
+ ]
155
+ },
156
+ "prefill_flops_counted": 90083398713024,
157
+ "decode_one_token_flops_counted": 51922600128,
158
+ "prefill_gflops_counted": 90083.398713024,
159
+ "decode_one_token_gflops_counted": 51.922600128,
160
+ "prefill_operators": {
161
+ "aten.convolution": 20475543552,
162
+ "aten.addmm": 3283626885120,
163
+ "aten.bmm": 3741356981952,
164
+ "aten.mm": 83037939302400
165
+ },
166
+ "decode_operators": {
167
+ "aten.bmm": 670826688,
168
+ "aten.mm": 51243909120,
169
+ "aten.convolution": 7864320
170
+ }
171
+ },
172
+ {
173
+ "sample_id": "0003433",
174
+ "prompt_tokens_in_reference_tensor": 1705,
175
+ "input_shapes": {
176
+ "input_ids": [
177
+ 1,
178
+ 1705
179
+ ],
180
+ "attention_mask": [
181
+ 1,
182
+ 1705
183
+ ],
184
+ "mm_token_type_ids": [
185
+ 1,
186
+ 1705
187
+ ],
188
+ "pixel_values": [
189
+ 3888,
190
+ 1536
191
+ ],
192
+ "image_grid_thw": [
193
+ 1,
194
+ 3
195
+ ]
196
+ },
197
+ "prefill_flops_counted": 90083398713024,
198
+ "decode_one_token_flops_counted": 51922600128,
199
+ "prefill_gflops_counted": 90083.398713024,
200
+ "decode_one_token_gflops_counted": 51.922600128,
201
+ "prefill_operators": {
202
+ "aten.convolution": 20475543552,
203
+ "aten.addmm": 3283626885120,
204
+ "aten.bmm": 3741356981952,
205
+ "aten.mm": 83037939302400
206
+ },
207
+ "decode_operators": {
208
+ "aten.bmm": 670826688,
209
+ "aten.mm": 51243909120,
210
+ "aten.convolution": 7864320
211
+ }
212
+ },
213
+ {
214
+ "sample_id": "0001613",
215
+ "prompt_tokens_in_reference_tensor": 1705,
216
+ "input_shapes": {
217
+ "input_ids": [
218
+ 1,
219
+ 1705
220
+ ],
221
+ "attention_mask": [
222
+ 1,
223
+ 1705
224
+ ],
225
+ "mm_token_type_ids": [
226
+ 1,
227
+ 1705
228
+ ],
229
+ "pixel_values": [
230
+ 3888,
231
+ 1536
232
+ ],
233
+ "image_grid_thw": [
234
+ 1,
235
+ 3
236
+ ]
237
+ },
238
+ "prefill_flops_counted": 90083398713024,
239
+ "decode_one_token_flops_counted": 51922600128,
240
+ "prefill_gflops_counted": 90083.398713024,
241
+ "decode_one_token_gflops_counted": 51.922600128,
242
+ "prefill_operators": {
243
+ "aten.convolution": 20475543552,
244
+ "aten.addmm": 3283626885120,
245
+ "aten.bmm": 3741356981952,
246
+ "aten.mm": 83037939302400
247
+ },
248
+ "decode_operators": {
249
+ "aten.bmm": 670826688,
250
+ "aten.mm": 51243909120,
251
+ "aten.convolution": 7864320
252
+ }
253
+ },
254
+ {
255
+ "sample_id": "0002591",
256
+ "prompt_tokens_in_reference_tensor": 1705,
257
+ "input_shapes": {
258
+ "input_ids": [
259
+ 1,
260
+ 1705
261
+ ],
262
+ "attention_mask": [
263
+ 1,
264
+ 1705
265
+ ],
266
+ "mm_token_type_ids": [
267
+ 1,
268
+ 1705
269
+ ],
270
+ "pixel_values": [
271
+ 3888,
272
+ 1536
273
+ ],
274
+ "image_grid_thw": [
275
+ 1,
276
+ 3
277
+ ]
278
+ },
279
+ "prefill_flops_counted": 90083398713024,
280
+ "decode_one_token_flops_counted": 51922600128,
281
+ "prefill_gflops_counted": 90083.398713024,
282
+ "decode_one_token_gflops_counted": 51.922600128,
283
+ "prefill_operators": {
284
+ "aten.convolution": 20475543552,
285
+ "aten.addmm": 3283626885120,
286
+ "aten.bmm": 3741356981952,
287
+ "aten.mm": 83037939302400
288
+ },
289
+ "decode_operators": {
290
+ "aten.bmm": 670826688,
291
+ "aten.mm": 51243909120,
292
+ "aten.convolution": 7864320
293
+ }
294
+ },
295
+ {
296
+ "sample_id": "0003105",
297
+ "prompt_tokens_in_reference_tensor": 1705,
298
+ "input_shapes": {
299
+ "input_ids": [
300
+ 1,
301
+ 1705
302
+ ],
303
+ "attention_mask": [
304
+ 1,
305
+ 1705
306
+ ],
307
+ "mm_token_type_ids": [
308
+ 1,
309
+ 1705
310
+ ],
311
+ "pixel_values": [
312
+ 3888,
313
+ 1536
314
+ ],
315
+ "image_grid_thw": [
316
+ 1,
317
+ 3
318
+ ]
319
+ },
320
+ "prefill_flops_counted": 90083398713024,
321
+ "decode_one_token_flops_counted": 51922600128,
322
+ "prefill_gflops_counted": 90083.398713024,
323
+ "decode_one_token_gflops_counted": 51.922600128,
324
+ "prefill_operators": {
325
+ "aten.convolution": 20475543552,
326
+ "aten.addmm": 3283626885120,
327
+ "aten.bmm": 3741356981952,
328
+ "aten.mm": 83037939302400
329
+ },
330
+ "decode_operators": {
331
+ "aten.bmm": 670826688,
332
+ "aten.mm": 51243909120,
333
+ "aten.convolution": 7864320
334
+ }
335
+ }
336
+ ],
337
+ "mean_prefill_gflops_counted": 90083.398713024,
338
+ "mean_decode_one_token_gflops_counted": 51.922600128,
339
+ "scope": "HF reference execution on the A800 with real task inputs; dense products/convolutions/supported attention counted with multiply-add=2 FLOPs. vLLM custom kernels, elementwise/normalization/nonlinearities and other unsupported operations are not complete.",
340
+ "answer_generation": "not performed; one known-token decode probe only"
341
+ }
paper_results/data/main_table_cost_qwen_20260912/components/qwen_gref_flops.json ADDED
@@ -0,0 +1,341 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "status": "complete",
3
+ "model_kind": "Qwen3.8",
4
+ "task": "grefcoco",
5
+ "model_path": "/mnt/HithinkOmniSSD/user_workspace/caisihang/checkpoints/Qwen3.8-27B-FP8",
6
+ "sample_count": 8,
7
+ "records": [
8
+ {
9
+ "sample_id": "testA-5038-13945",
10
+ "prompt_tokens_in_reference_tensor": 608,
11
+ "input_shapes": {
12
+ "input_ids": [
13
+ 1,
14
+ 608
15
+ ],
16
+ "attention_mask": [
17
+ 1,
18
+ 608
19
+ ],
20
+ "mm_token_type_ids": [
21
+ 1,
22
+ 608
23
+ ],
24
+ "pixel_values": [
25
+ 1200,
26
+ 1536
27
+ ],
28
+ "image_grid_thw": [
29
+ 1,
30
+ 3
31
+ ]
32
+ },
33
+ "prefill_flops_counted": 31223201515520,
34
+ "decode_one_token_flops_counted": 51491242176,
35
+ "prefill_gflops_counted": 31223.20151552,
36
+ "decode_one_token_gflops_counted": 51.491242176,
37
+ "prefill_operators": {
38
+ "aten.convolution": 6649282560,
39
+ "aten.addmm": 1013465088000,
40
+ "aten.bmm": 590268057600,
41
+ "aten.mm": 29612819087360
42
+ },
43
+ "decode_operators": {
44
+ "aten.bmm": 239468736,
45
+ "aten.mm": 51243909120,
46
+ "aten.convolution": 7864320
47
+ }
48
+ },
49
+ {
50
+ "sample_id": "testA-73305-202014",
51
+ "prompt_tokens_in_reference_tensor": 608,
52
+ "input_shapes": {
53
+ "input_ids": [
54
+ 1,
55
+ 608
56
+ ],
57
+ "attention_mask": [
58
+ 1,
59
+ 608
60
+ ],
61
+ "mm_token_type_ids": [
62
+ 1,
63
+ 608
64
+ ],
65
+ "pixel_values": [
66
+ 1200,
67
+ 1536
68
+ ],
69
+ "image_grid_thw": [
70
+ 1,
71
+ 3
72
+ ]
73
+ },
74
+ "prefill_flops_counted": 31223201515520,
75
+ "decode_one_token_flops_counted": 51491242176,
76
+ "prefill_gflops_counted": 31223.20151552,
77
+ "decode_one_token_gflops_counted": 51.491242176,
78
+ "prefill_operators": {
79
+ "aten.convolution": 6649282560,
80
+ "aten.addmm": 1013465088000,
81
+ "aten.bmm": 590268057600,
82
+ "aten.mm": 29612819087360
83
+ },
84
+ "decode_operators": {
85
+ "aten.bmm": 239468736,
86
+ "aten.mm": 51243909120,
87
+ "aten.convolution": 7864320
88
+ }
89
+ },
90
+ {
91
+ "sample_id": "testA-26952-74099",
92
+ "prompt_tokens_in_reference_tensor": 562,
93
+ "input_shapes": {
94
+ "input_ids": [
95
+ 1,
96
+ 562
97
+ ],
98
+ "attention_mask": [
99
+ 1,
100
+ 562
101
+ ],
102
+ "mm_token_type_ids": [
103
+ 1,
104
+ 562
105
+ ],
106
+ "pixel_values": [
107
+ 1040,
108
+ 1536
109
+ ],
110
+ "image_grid_thw": [
111
+ 1,
112
+ 3
113
+ ]
114
+ },
115
+ "prefill_flops_counted": 28754745861504,
116
+ "decode_one_token_flops_counted": 51473154240,
117
+ "prefill_gflops_counted": 28754.745861504,
118
+ "decode_one_token_gflops_counted": 51.47315424,
119
+ "prefill_operators": {
120
+ "aten.convolution": 5902172160,
121
+ "aten.addmm": 878336409600,
122
+ "aten.bmm": 497939359104,
123
+ "aten.mm": 27372567920640
124
+ },
125
+ "decode_operators": {
126
+ "aten.bmm": 221380800,
127
+ "aten.mm": 51243909120,
128
+ "aten.convolution": 7864320
129
+ }
130
+ },
131
+ {
132
+ "sample_id": "testB-53254-146620",
133
+ "prompt_tokens_in_reference_tensor": 560,
134
+ "input_shapes": {
135
+ "input_ids": [
136
+ 1,
137
+ 560
138
+ ],
139
+ "attention_mask": [
140
+ 1,
141
+ 560
142
+ ],
143
+ "mm_token_type_ids": [
144
+ 1,
145
+ 560
146
+ ],
147
+ "pixel_values": [
148
+ 1040,
149
+ 1536
150
+ ],
151
+ "image_grid_thw": [
152
+ 1,
153
+ 3
154
+ ]
155
+ },
156
+ "prefill_flops_counted": 28656453395456,
157
+ "decode_one_token_flops_counted": 51472367808,
158
+ "prefill_gflops_counted": 28656.453395456,
159
+ "decode_one_token_gflops_counted": 51.472367808,
160
+ "prefill_operators": {
161
+ "aten.convolution": 5894307840,
162
+ "aten.addmm": 878336409600,
163
+ "aten.bmm": 497056982016,
164
+ "aten.mm": 27275165696000
165
+ },
166
+ "decode_operators": {
167
+ "aten.bmm": 220594368,
168
+ "aten.mm": 51243909120,
169
+ "aten.convolution": 7864320
170
+ }
171
+ },
172
+ {
173
+ "sample_id": "testB-37889-104439",
174
+ "prompt_tokens_in_reference_tensor": 687,
175
+ "input_shapes": {
176
+ "input_ids": [
177
+ 1,
178
+ 687
179
+ ],
180
+ "attention_mask": [
181
+ 1,
182
+ 687
183
+ ],
184
+ "mm_token_type_ids": [
185
+ 1,
186
+ 687
187
+ ],
188
+ "pixel_values": [
189
+ 1520,
190
+ 1536
191
+ ],
192
+ "image_grid_thw": [
193
+ 1,
194
+ 3
195
+ ]
196
+ },
197
+ "prefill_flops_counted": 35517384622912,
198
+ "decode_one_token_flops_counted": 51522306240,
199
+ "prefill_gflops_counted": 35517.384622912,
200
+ "decode_one_token_gflops_counted": 51.52230624,
201
+ "prefill_operators": {
202
+ "aten.convolution": 8092385280,
203
+ "aten.addmm": 1283722444800,
204
+ "aten.bmm": 765362832192,
205
+ "aten.mm": 33460206960640
206
+ },
207
+ "decode_operators": {
208
+ "aten.bmm": 270532800,
209
+ "aten.mm": 51243909120,
210
+ "aten.convolution": 7864320
211
+ }
212
+ },
213
+ {
214
+ "sample_id": "testA-58523-161084",
215
+ "prompt_tokens_in_reference_tensor": 500,
216
+ "input_shapes": {
217
+ "input_ids": [
218
+ 1,
219
+ 500
220
+ ],
221
+ "attention_mask": [
222
+ 1,
223
+ 500
224
+ ],
225
+ "mm_token_type_ids": [
226
+ 1,
227
+ 500
228
+ ],
229
+ "pixel_values": [
230
+ 768,
231
+ 1536
232
+ ],
233
+ "image_grid_thw": [
234
+ 1,
235
+ 3
236
+ ]
237
+ },
238
+ "prefill_flops_counted": 25390700918528,
239
+ "decode_one_token_flops_counted": 51448774848,
240
+ "prefill_gflops_counted": 25390.700918528,
241
+ "decode_one_token_gflops_counted": 51.448774848,
242
+ "prefill_operators": {
243
+ "aten.convolution": 4695785472,
244
+ "aten.addmm": 648617656320,
245
+ "aten.bmm": 384288519936,
246
+ "aten.mm": 24353098956800
247
+ },
248
+ "decode_operators": {
249
+ "aten.bmm": 197001408,
250
+ "aten.mm": 51243909120,
251
+ "aten.convolution": 7864320
252
+ }
253
+ },
254
+ {
255
+ "sample_id": "testB-3419-9449",
256
+ "prompt_tokens_in_reference_tensor": 568,
257
+ "input_shapes": {
258
+ "input_ids": [
259
+ 1,
260
+ 568
261
+ ],
262
+ "attention_mask": [
263
+ 1,
264
+ 568
265
+ ],
266
+ "mm_token_type_ids": [
267
+ 1,
268
+ 568
269
+ ],
270
+ "pixel_values": [
271
+ 1040,
272
+ 1536
273
+ ],
274
+ "image_grid_thw": [
275
+ 1,
276
+ 3
277
+ ]
278
+ },
279
+ "prefill_flops_counted": 29049642134016,
280
+ "decode_one_token_flops_counted": 51475513536,
281
+ "prefill_gflops_counted": 29049.642134016,
282
+ "decode_one_token_gflops_counted": 51.475513536,
283
+ "prefill_operators": {
284
+ "aten.convolution": 5925765120,
285
+ "aten.addmm": 878336409600,
286
+ "aten.bmm": 500605364736,
287
+ "aten.mm": 27664774594560
288
+ },
289
+ "decode_operators": {
290
+ "aten.bmm": 223740096,
291
+ "aten.mm": 51243909120,
292
+ "aten.convolution": 7864320
293
+ }
294
+ },
295
+ {
296
+ "sample_id": "testB-24035-66084",
297
+ "prompt_tokens_in_reference_tensor": 561,
298
+ "input_shapes": {
299
+ "input_ids": [
300
+ 1,
301
+ 561
302
+ ],
303
+ "attention_mask": [
304
+ 1,
305
+ 561
306
+ ],
307
+ "mm_token_type_ids": [
308
+ 1,
309
+ 561
310
+ ],
311
+ "pixel_values": [
312
+ 1040,
313
+ 1536
314
+ ],
315
+ "image_grid_thw": [
316
+ 1,
317
+ 3
318
+ ]
319
+ },
320
+ "prefill_flops_counted": 28705599235264,
321
+ "decode_one_token_flops_counted": 51472761024,
322
+ "prefill_gflops_counted": 28705.599235264,
323
+ "decode_one_token_gflops_counted": 51.472761024,
324
+ "prefill_operators": {
325
+ "aten.convolution": 5898240000,
326
+ "aten.addmm": 878336409600,
327
+ "aten.bmm": 497497777344,
328
+ "aten.mm": 27323866808320
329
+ },
330
+ "decode_operators": {
331
+ "aten.bmm": 220987584,
332
+ "aten.mm": 51243909120,
333
+ "aten.convolution": 7864320
334
+ }
335
+ }
336
+ ],
337
+ "mean_prefill_gflops_counted": 29815.11614984,
338
+ "mean_decode_one_token_gflops_counted": 51.480920256000005,
339
+ "scope": "HF reference execution on the A800 with real task inputs; dense products/convolutions/supported attention counted with multiply-add=2 FLOPs. vLLM custom kernels, elementwise/normalization/nonlinearities and other unsupported operations are not complete.",
340
+ "answer_generation": "not performed; one known-token decode probe only"
341
+ }
paper_results/data/main_table_cost_qwen_20260912/components/qwen_gref_latency.json ADDED
@@ -0,0 +1,2446 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "status": "complete",
3
+ "model_kind": "Qwen3.8",
4
+ "task": "grefcoco",
5
+ "expert": "instancevg",
6
+ "model_path": "/mnt/HithinkOmniSSD/user_workspace/caisihang/checkpoints/Qwen3.8-27B-FP8",
7
+ "device": "cuda:0",
8
+ "gpu_memory_utilization": 0.85,
9
+ "sample_count_requested": 66,
10
+ "warmup_count": 2,
11
+ "measured_count": 64,
12
+ "batch_size": 1,
13
+ "sample_ids": [
14
+ "testA-29199-80373",
15
+ "testB-44351-122075",
16
+ "testA-91850-253019",
17
+ "testB-36888-101641",
18
+ "testA-54586-150233",
19
+ "testB-79644-219272",
20
+ "testB-65935-181776",
21
+ "testA-81622-224664",
22
+ "testA-10078-27739",
23
+ "testA-15582-42797",
24
+ "testA-5038-13943",
25
+ "testA-30844-84861",
26
+ "testB-15419-42356",
27
+ "testB-27486-75580",
28
+ "testB-945-2632",
29
+ "testA-24438-67190",
30
+ "testB-66161-182405",
31
+ "testB-42726-117659",
32
+ "testB-69439-191450",
33
+ "testA-54307-149426",
34
+ "testB-37669-103831",
35
+ "testB-50844-140105",
36
+ "testA-79438-218690",
37
+ "testA-62940-173454",
38
+ "testA-81245-223632",
39
+ "testA-47070-129599",
40
+ "testB-12381-34098",
41
+ "testA-44142-121498",
42
+ "testA-73258-201893",
43
+ "testB-46644-128436",
44
+ "testA-43889-120792",
45
+ "testB-36082-99363",
46
+ "testB-33108-91152",
47
+ "testA-26856-73830",
48
+ "testA-81271-223709",
49
+ "testA-54554-150141",
50
+ "testB-38636-106481",
51
+ "testA-5734-15837",
52
+ "testB-67558-186325",
53
+ "testB-47053-129554",
54
+ "testA-93064-256422",
55
+ "testB-60557-166821",
56
+ "testA-55116-151688",
57
+ "testB-65796-181392",
58
+ "testB-23014-63278",
59
+ "testA-24562-67515",
60
+ "testA-17736-48684",
61
+ "testA-58475-160949",
62
+ "testA-80455-221493",
63
+ "testA-31763-87405",
64
+ "testB-23920-65742",
65
+ "testA-469-1309",
66
+ "testB-56691-155998",
67
+ "testB-31493-86668",
68
+ "testA-59284-163254",
69
+ "testA-57397-158015",
70
+ "testB-2935-8111",
71
+ "testB-22290-61226",
72
+ "testB-8416-23140",
73
+ "testA-60731-167320",
74
+ "testA-49055-135086",
75
+ "testA-7857-21644",
76
+ "testB-79645-219273",
77
+ "testB-52921-145707",
78
+ "testB-69479-191562",
79
+ "testA-69466-191524"
80
+ ],
81
+ "batches": [
82
+ {
83
+ "sample_ids": [
84
+ "testA-91850-253019"
85
+ ],
86
+ "batch_size": 1,
87
+ "generation_latency_seconds": 3.147620694944635,
88
+ "batch_wall_latency_seconds": 3.1645371050108224,
89
+ "amortized_generation_ms_per_sample": 3147.620694944635,
90
+ "amortized_batch_wall_ms_per_sample": 3164.5371050108224,
91
+ "outputs": [
92
+ {
93
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [496, 292, 713, 655], \"score\": 0.9}]}",
94
+ "valid_prediction": {
95
+ "objects": [
96
+ {
97
+ "bbox_2d": [
98
+ 496,
99
+ 292,
100
+ 713,
101
+ 655
102
+ ],
103
+ "score": 0.9
104
+ }
105
+ ]
106
+ },
107
+ "error": null,
108
+ "finish_reason": "stop",
109
+ "completion_tokens": 39,
110
+ "prompt_tokens": 570
111
+ }
112
+ ]
113
+ },
114
+ {
115
+ "sample_ids": [
116
+ "testB-36888-101641"
117
+ ],
118
+ "batch_size": 1,
119
+ "generation_latency_seconds": 4.945294587058015,
120
+ "batch_wall_latency_seconds": 4.966181440046057,
121
+ "amortized_generation_ms_per_sample": 4945.294587058015,
122
+ "amortized_batch_wall_ms_per_sample": 4966.181440046057,
123
+ "outputs": [
124
+ {
125
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[21,567,693,915],\"score\":0.95},{\"bbox_2d\":[381,233,959,885],\"score\":0.95}]}",
126
+ "valid_prediction": {
127
+ "objects": [
128
+ {
129
+ "bbox_2d": [
130
+ 21,
131
+ 567,
132
+ 693,
133
+ 915
134
+ ],
135
+ "score": 0.95
136
+ },
137
+ {
138
+ "bbox_2d": [
139
+ 381,
140
+ 233,
141
+ 959,
142
+ 885
143
+ ],
144
+ "score": 0.95
145
+ }
146
+ ]
147
+ },
148
+ "error": null,
149
+ "finish_reason": "stop",
150
+ "completion_tokens": 60,
151
+ "prompt_tokens": 608
152
+ }
153
+ ]
154
+ },
155
+ {
156
+ "sample_ids": [
157
+ "testA-54586-150233"
158
+ ],
159
+ "batch_size": 1,
160
+ "generation_latency_seconds": 3.2111774859949946,
161
+ "batch_wall_latency_seconds": 3.2314598000375554,
162
+ "amortized_generation_ms_per_sample": 3211.1774859949946,
163
+ "amortized_batch_wall_ms_per_sample": 3231.4598000375554,
164
+ "outputs": [
165
+ {
166
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [693, 438, 983, 813], \"score\": 0.95}]}",
167
+ "valid_prediction": {
168
+ "objects": [
169
+ {
170
+ "bbox_2d": [
171
+ 693,
172
+ 438,
173
+ 983,
174
+ 813
175
+ ],
176
+ "score": 0.95
177
+ }
178
+ ]
179
+ },
180
+ "error": null,
181
+ "finish_reason": "stop",
182
+ "completion_tokens": 40,
183
+ "prompt_tokens": 707
184
+ }
185
+ ]
186
+ },
187
+ {
188
+ "sample_ids": [
189
+ "testB-79644-219272"
190
+ ],
191
+ "batch_size": 1,
192
+ "generation_latency_seconds": 33.390286731068045,
193
+ "batch_wall_latency_seconds": 33.40712711191736,
194
+ "amortized_generation_ms_per_sample": 33390.286731068045,
195
+ "amortized_batch_wall_ms_per_sample": 33407.12711191736,
196
+ "outputs": [
197
+ {
198
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [121, 581, 170, 764], \"score\": 0.9}, {\"bbox_2d\": [173, 595, 259, 837], \"score\": 0.9}, {\"bbox_2d\": [265, 581, 315, 806], \"score\": 0.9}, {\"bbox_2d\": [310, 571, 351, 764], \"score\": 0.9}, {\"bbox_2d\": [338, 581, 387, 764], \"score\": 0.9}, {\"bbox_2d\": [449, 575, 492, 756], \"score\": 0.9}, {\"bbox_2d\": [475, 604, 542, 806], \"score\": 0.9}, {\"bbox_2d\": [668, 575, 743, 818], \"score\": 0.9}, {\"bbox_2d\": [740, 571, 783, 775], \"score\": 0.9}, {\"bbox_2d\": [771, 571, 814, 787], \"score\": 0.9}, {\"bbox_2d\": [822, 587, 851, 764], \"score\": 0.9}, {\"bbox_2d\": [847, 591, 921, 903], \"score\": 0.9}, {\"bbox_2d\": [902, 604, 959, 795], \"score\": 0.9}]}",
199
+ "valid_prediction": {
200
+ "objects": [
201
+ {
202
+ "bbox_2d": [
203
+ 121,
204
+ 581,
205
+ 170,
206
+ 764
207
+ ],
208
+ "score": 0.9
209
+ },
210
+ {
211
+ "bbox_2d": [
212
+ 173,
213
+ 595,
214
+ 259,
215
+ 837
216
+ ],
217
+ "score": 0.9
218
+ },
219
+ {
220
+ "bbox_2d": [
221
+ 265,
222
+ 581,
223
+ 315,
224
+ 806
225
+ ],
226
+ "score": 0.9
227
+ },
228
+ {
229
+ "bbox_2d": [
230
+ 310,
231
+ 571,
232
+ 351,
233
+ 764
234
+ ],
235
+ "score": 0.9
236
+ },
237
+ {
238
+ "bbox_2d": [
239
+ 338,
240
+ 581,
241
+ 387,
242
+ 764
243
+ ],
244
+ "score": 0.9
245
+ },
246
+ {
247
+ "bbox_2d": [
248
+ 449,
249
+ 575,
250
+ 492,
251
+ 756
252
+ ],
253
+ "score": 0.9
254
+ },
255
+ {
256
+ "bbox_2d": [
257
+ 475,
258
+ 604,
259
+ 542,
260
+ 806
261
+ ],
262
+ "score": 0.9
263
+ },
264
+ {
265
+ "bbox_2d": [
266
+ 668,
267
+ 575,
268
+ 743,
269
+ 818
270
+ ],
271
+ "score": 0.9
272
+ },
273
+ {
274
+ "bbox_2d": [
275
+ 740,
276
+ 571,
277
+ 783,
278
+ 775
279
+ ],
280
+ "score": 0.9
281
+ },
282
+ {
283
+ "bbox_2d": [
284
+ 771,
285
+ 571,
286
+ 814,
287
+ 787
288
+ ],
289
+ "score": 0.9
290
+ },
291
+ {
292
+ "bbox_2d": [
293
+ 822,
294
+ 587,
295
+ 851,
296
+ 764
297
+ ],
298
+ "score": 0.9
299
+ },
300
+ {
301
+ "bbox_2d": [
302
+ 847,
303
+ 591,
304
+ 921,
305
+ 903
306
+ ],
307
+ "score": 0.9
308
+ },
309
+ {
310
+ "bbox_2d": [
311
+ 902,
312
+ 604,
313
+ 959,
314
+ 795
315
+ ],
316
+ "score": 0.9
317
+ }
318
+ ]
319
+ },
320
+ "error": null,
321
+ "finish_reason": "stop",
322
+ "completion_tokens": 447,
323
+ "prompt_tokens": 600
324
+ }
325
+ ]
326
+ },
327
+ {
328
+ "sample_ids": [
329
+ "testB-65935-181776"
330
+ ],
331
+ "batch_size": 1,
332
+ "generation_latency_seconds": 4.57236669503618,
333
+ "batch_wall_latency_seconds": 4.583186280098744,
334
+ "amortized_generation_ms_per_sample": 4572.36669503618,
335
+ "amortized_batch_wall_ms_per_sample": 4583.186280098744,
336
+ "outputs": [
337
+ {
338
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[347,333,951,761],\"score\":0.95},{\"bbox_2d\":[246,421,777,987],\"score\":0.95}]}",
339
+ "valid_prediction": {
340
+ "objects": [
341
+ {
342
+ "bbox_2d": [
343
+ 347,
344
+ 333,
345
+ 951,
346
+ 761
347
+ ],
348
+ "score": 0.95
349
+ },
350
+ {
351
+ "bbox_2d": [
352
+ 246,
353
+ 421,
354
+ 777,
355
+ 987
356
+ ],
357
+ "score": 0.95
358
+ }
359
+ ]
360
+ },
361
+ "error": null,
362
+ "finish_reason": "stop",
363
+ "completion_tokens": 61,
364
+ "prompt_tokens": 564
365
+ }
366
+ ]
367
+ },
368
+ {
369
+ "sample_ids": [
370
+ "testA-81622-224664"
371
+ ],
372
+ "batch_size": 1,
373
+ "generation_latency_seconds": 0.6317770309979096,
374
+ "batch_wall_latency_seconds": 0.6445289449766278,
375
+ "amortized_generation_ms_per_sample": 631.7770309979096,
376
+ "amortized_batch_wall_ms_per_sample": 644.5289449766278,
377
+ "outputs": [
378
+ {
379
+ "raw_response": "{\"objects\": []}",
380
+ "valid_prediction": {
381
+ "objects": []
382
+ },
383
+ "error": null,
384
+ "finish_reason": "stop",
385
+ "completion_tokens": 6,
386
+ "prompt_tokens": 564
387
+ }
388
+ ]
389
+ },
390
+ {
391
+ "sample_ids": [
392
+ "testA-10078-27739"
393
+ ],
394
+ "batch_size": 1,
395
+ "generation_latency_seconds": 2.393106485949829,
396
+ "batch_wall_latency_seconds": 2.4061070699244738,
397
+ "amortized_generation_ms_per_sample": 2393.106485949829,
398
+ "amortized_batch_wall_ms_per_sample": 2406.1070699244738,
399
+ "outputs": [
400
+ {
401
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[283,97,775,810],\"score\":0.95}]}",
402
+ "valid_prediction": {
403
+ "objects": [
404
+ {
405
+ "bbox_2d": [
406
+ 283,
407
+ 97,
408
+ 775,
409
+ 810
410
+ ],
411
+ "score": 0.95
412
+ }
413
+ ]
414
+ },
415
+ "error": null,
416
+ "finish_reason": "stop",
417
+ "completion_tokens": 32,
418
+ "prompt_tokens": 564
419
+ }
420
+ ]
421
+ },
422
+ {
423
+ "sample_ids": [
424
+ "testA-15582-42797"
425
+ ],
426
+ "batch_size": 1,
427
+ "generation_latency_seconds": 7.63340066303499,
428
+ "batch_wall_latency_seconds": 7.651124170981348,
429
+ "amortized_generation_ms_per_sample": 7633.40066303499,
430
+ "amortized_batch_wall_ms_per_sample": 7651.124170981348,
431
+ "outputs": [
432
+ {
433
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [85, 297, 446, 999], \"score\": 0.9}, {\"bbox_2d\": [315, 413, 542, 999], \"score\": 0.9}, {\"bbox_2d\": [542, 476, 642, 876], \"score\": 0.9}]}",
434
+ "valid_prediction": {
435
+ "objects": [
436
+ {
437
+ "bbox_2d": [
438
+ 85,
439
+ 297,
440
+ 446,
441
+ 999
442
+ ],
443
+ "score": 0.9
444
+ },
445
+ {
446
+ "bbox_2d": [
447
+ 315,
448
+ 413,
449
+ 542,
450
+ 999
451
+ ],
452
+ "score": 0.9
453
+ },
454
+ {
455
+ "bbox_2d": [
456
+ 542,
457
+ 476,
458
+ 642,
459
+ 876
460
+ ],
461
+ "score": 0.9
462
+ }
463
+ ]
464
+ },
465
+ "error": null,
466
+ "finish_reason": "stop",
467
+ "completion_tokens": 106,
468
+ "prompt_tokens": 619
469
+ }
470
+ ]
471
+ },
472
+ {
473
+ "sample_ids": [
474
+ "testA-5038-13943"
475
+ ],
476
+ "batch_size": 1,
477
+ "generation_latency_seconds": 3.06427818199154,
478
+ "batch_wall_latency_seconds": 3.0877813909901306,
479
+ "amortized_generation_ms_per_sample": 3064.27818199154,
480
+ "amortized_batch_wall_ms_per_sample": 3087.7813909901306,
481
+ "outputs": [
482
+ {
483
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [521, 246, 793, 999], \"score\": 0.95}]}",
484
+ "valid_prediction": {
485
+ "objects": [
486
+ {
487
+ "bbox_2d": [
488
+ 521,
489
+ 246,
490
+ 793,
491
+ 999
492
+ ],
493
+ "score": 0.95
494
+ }
495
+ ]
496
+ },
497
+ "error": null,
498
+ "finish_reason": "stop",
499
+ "completion_tokens": 40,
500
+ "prompt_tokens": 604
501
+ }
502
+ ]
503
+ },
504
+ {
505
+ "sample_ids": [
506
+ "testA-30844-84861"
507
+ ],
508
+ "batch_size": 1,
509
+ "generation_latency_seconds": 2.516077253036201,
510
+ "batch_wall_latency_seconds": 2.5232016970403492,
511
+ "amortized_generation_ms_per_sample": 2516.077253036201,
512
+ "amortized_batch_wall_ms_per_sample": 2523.2016970403492,
513
+ "outputs": [
514
+ {
515
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[508,536,807,722],\"score\":0.9}]}",
516
+ "valid_prediction": {
517
+ "objects": [
518
+ {
519
+ "bbox_2d": [
520
+ 508,
521
+ 536,
522
+ 807,
523
+ 722
524
+ ],
525
+ "score": 0.9
526
+ }
527
+ ]
528
+ },
529
+ "error": null,
530
+ "finish_reason": "stop",
531
+ "completion_tokens": 32,
532
+ "prompt_tokens": 623
533
+ }
534
+ ]
535
+ },
536
+ {
537
+ "sample_ids": [
538
+ "testB-15419-42356"
539
+ ],
540
+ "batch_size": 1,
541
+ "generation_latency_seconds": 2.673266503959894,
542
+ "batch_wall_latency_seconds": 2.6889539459953085,
543
+ "amortized_generation_ms_per_sample": 2673.266503959894,
544
+ "amortized_batch_wall_ms_per_sample": 2688.9539459953085,
545
+ "outputs": [
546
+ {
547
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[107,722,587,999],\"score\":0.9}]}",
548
+ "valid_prediction": {
549
+ "objects": [
550
+ {
551
+ "bbox_2d": [
552
+ 107,
553
+ 722,
554
+ 587,
555
+ 999
556
+ ],
557
+ "score": 0.9
558
+ }
559
+ ]
560
+ },
561
+ "error": null,
562
+ "finish_reason": "stop",
563
+ "completion_tokens": 32,
564
+ "prompt_tokens": 566
565
+ }
566
+ ]
567
+ },
568
+ {
569
+ "sample_ids": [
570
+ "testB-27486-75580"
571
+ ],
572
+ "batch_size": 1,
573
+ "generation_latency_seconds": 4.617897133924998,
574
+ "batch_wall_latency_seconds": 4.632912510889582,
575
+ "amortized_generation_ms_per_sample": 4617.897133924998,
576
+ "amortized_batch_wall_ms_per_sample": 4632.912510889582,
577
+ "outputs": [
578
+ {
579
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[542,393,999,999],\"score\":0.95},{\"bbox_2d\":[646,271,911,471],\"score\":0.95}]}",
580
+ "valid_prediction": {
581
+ "objects": [
582
+ {
583
+ "bbox_2d": [
584
+ 542,
585
+ 393,
586
+ 999,
587
+ 999
588
+ ],
589
+ "score": 0.95
590
+ },
591
+ {
592
+ "bbox_2d": [
593
+ 646,
594
+ 271,
595
+ 911,
596
+ 471
597
+ ],
598
+ "score": 0.95
599
+ }
600
+ ]
601
+ },
602
+ "error": null,
603
+ "finish_reason": "stop",
604
+ "completion_tokens": 61,
605
+ "prompt_tokens": 670
606
+ }
607
+ ]
608
+ },
609
+ {
610
+ "sample_ids": [
611
+ "testB-945-2632"
612
+ ],
613
+ "batch_size": 1,
614
+ "generation_latency_seconds": 12.483138478011824,
615
+ "batch_wall_latency_seconds": 12.503725816030055,
616
+ "amortized_generation_ms_per_sample": 12483.138478011824,
617
+ "amortized_batch_wall_ms_per_sample": 12503.725816030055,
618
+ "outputs": [
619
+ {
620
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [213, 692, 283, 983], \"score\": 0.9}, {\"bbox_2d\": [321, 661, 415, 973], \"score\": 0.9}, {\"bbox_2d\": [473, 661, 555, 966], \"score\": 0.9}, {\"bbox_2d\": [638, 626, 713, 973], \"score\": 0.9}, {\"bbox_2d\": [775, 638, 871, 988], \"score\": 0.9}]}",
621
+ "valid_prediction": {
622
+ "objects": [
623
+ {
624
+ "bbox_2d": [
625
+ 213,
626
+ 692,
627
+ 283,
628
+ 983
629
+ ],
630
+ "score": 0.9
631
+ },
632
+ {
633
+ "bbox_2d": [
634
+ 321,
635
+ 661,
636
+ 415,
637
+ 973
638
+ ],
639
+ "score": 0.9
640
+ },
641
+ {
642
+ "bbox_2d": [
643
+ 473,
644
+ 661,
645
+ 555,
646
+ 966
647
+ ],
648
+ "score": 0.9
649
+ },
650
+ {
651
+ "bbox_2d": [
652
+ 638,
653
+ 626,
654
+ 713,
655
+ 973
656
+ ],
657
+ "score": 0.9
658
+ },
659
+ {
660
+ "bbox_2d": [
661
+ 775,
662
+ 638,
663
+ 871,
664
+ 988
665
+ ],
666
+ "score": 0.9
667
+ }
668
+ ]
669
+ },
670
+ "error": null,
671
+ "finish_reason": "stop",
672
+ "completion_tokens": 175,
673
+ "prompt_tokens": 670
674
+ }
675
+ ]
676
+ },
677
+ {
678
+ "sample_ids": [
679
+ "testA-24438-67190"
680
+ ],
681
+ "batch_size": 1,
682
+ "generation_latency_seconds": 4.798777529038489,
683
+ "batch_wall_latency_seconds": 4.815012337057851,
684
+ "amortized_generation_ms_per_sample": 4798.777529038489,
685
+ "amortized_batch_wall_ms_per_sample": 4815.012337057851,
686
+ "outputs": [
687
+ {
688
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[731,375,999,999],\"score\":0.95},{\"bbox_2d\":[305,247,585,928],\"score\":0.95}]}",
689
+ "valid_prediction": {
690
+ "objects": [
691
+ {
692
+ "bbox_2d": [
693
+ 731,
694
+ 375,
695
+ 999,
696
+ 999
697
+ ],
698
+ "score": 0.95
699
+ },
700
+ {
701
+ "bbox_2d": [
702
+ 305,
703
+ 247,
704
+ 585,
705
+ 928
706
+ ],
707
+ "score": 0.95
708
+ }
709
+ ]
710
+ },
711
+ "error": null,
712
+ "finish_reason": "stop",
713
+ "completion_tokens": 61,
714
+ "prompt_tokens": 605
715
+ }
716
+ ]
717
+ },
718
+ {
719
+ "sample_ids": [
720
+ "testB-66161-182405"
721
+ ],
722
+ "batch_size": 1,
723
+ "generation_latency_seconds": 2.5812068269588053,
724
+ "batch_wall_latency_seconds": 2.593027200899087,
725
+ "amortized_generation_ms_per_sample": 2581.2068269588053,
726
+ "amortized_batch_wall_ms_per_sample": 2593.027200899087,
727
+ "outputs": [
728
+ {
729
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[103,193,755,713],\"score\":0.95}]}",
730
+ "valid_prediction": {
731
+ "objects": [
732
+ {
733
+ "bbox_2d": [
734
+ 103,
735
+ 193,
736
+ 755,
737
+ 713
738
+ ],
739
+ "score": 0.95
740
+ }
741
+ ]
742
+ },
743
+ "error": null,
744
+ "finish_reason": "stop",
745
+ "completion_tokens": 33,
746
+ "prompt_tokens": 563
747
+ }
748
+ ]
749
+ },
750
+ {
751
+ "sample_ids": [
752
+ "testB-42726-117659"
753
+ ],
754
+ "batch_size": 1,
755
+ "generation_latency_seconds": 3.0745598620269448,
756
+ "batch_wall_latency_seconds": 3.0872938209213316,
757
+ "amortized_generation_ms_per_sample": 3074.5598620269448,
758
+ "amortized_batch_wall_ms_per_sample": 3087.2938209213316,
759
+ "outputs": [
760
+ {
761
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [743, 458, 999, 878], \"score\": 0.95}]}",
762
+ "valid_prediction": {
763
+ "objects": [
764
+ {
765
+ "bbox_2d": [
766
+ 743,
767
+ 458,
768
+ 999,
769
+ 878
770
+ ],
771
+ "score": 0.95
772
+ }
773
+ ]
774
+ },
775
+ "error": null,
776
+ "finish_reason": "stop",
777
+ "completion_tokens": 40,
778
+ "prompt_tokens": 604
779
+ }
780
+ ]
781
+ },
782
+ {
783
+ "sample_ids": [
784
+ "testB-69439-191450"
785
+ ],
786
+ "batch_size": 1,
787
+ "generation_latency_seconds": 0.5888717719353735,
788
+ "batch_wall_latency_seconds": 0.5951321049360558,
789
+ "amortized_generation_ms_per_sample": 588.8717719353735,
790
+ "amortized_batch_wall_ms_per_sample": 595.1321049360558,
791
+ "outputs": [
792
+ {
793
+ "raw_response": "{\"objects\": []}",
794
+ "valid_prediction": {
795
+ "objects": []
796
+ },
797
+ "error": null,
798
+ "finish_reason": "stop",
799
+ "completion_tokens": 6,
800
+ "prompt_tokens": 492
801
+ }
802
+ ]
803
+ },
804
+ {
805
+ "sample_ids": [
806
+ "testA-54307-149426"
807
+ ],
808
+ "batch_size": 1,
809
+ "generation_latency_seconds": 4.398384497035295,
810
+ "batch_wall_latency_seconds": 4.4204836380667984,
811
+ "amortized_generation_ms_per_sample": 4398.384497035295,
812
+ "amortized_batch_wall_ms_per_sample": 4420.483638066798,
813
+ "outputs": [
814
+ {
815
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[542,48,838,923],\"score\":0.95},{\"bbox_2d\":[177,0,567,477],\"score\":0.95}]}",
816
+ "valid_prediction": {
817
+ "objects": [
818
+ {
819
+ "bbox_2d": [
820
+ 542,
821
+ 48,
822
+ 838,
823
+ 923
824
+ ],
825
+ "score": 0.95
826
+ },
827
+ {
828
+ "bbox_2d": [
829
+ 177,
830
+ 0,
831
+ 567,
832
+ 477
833
+ ],
834
+ "score": 0.95
835
+ }
836
+ ]
837
+ },
838
+ "error": null,
839
+ "finish_reason": "stop",
840
+ "completion_tokens": 58,
841
+ "prompt_tokens": 564
842
+ }
843
+ ]
844
+ },
845
+ {
846
+ "sample_ids": [
847
+ "testB-37669-103831"
848
+ ],
849
+ "batch_size": 1,
850
+ "generation_latency_seconds": 0.6747266949387267,
851
+ "batch_wall_latency_seconds": 0.699499664013274,
852
+ "amortized_generation_ms_per_sample": 674.7266949387267,
853
+ "amortized_batch_wall_ms_per_sample": 699.499664013274,
854
+ "outputs": [
855
+ {
856
+ "raw_response": "{\"objects\": []}",
857
+ "valid_prediction": {
858
+ "objects": []
859
+ },
860
+ "error": null,
861
+ "finish_reason": "stop",
862
+ "completion_tokens": 6,
863
+ "prompt_tokens": 648
864
+ }
865
+ ]
866
+ },
867
+ {
868
+ "sample_ids": [
869
+ "testB-50844-140105"
870
+ ],
871
+ "batch_size": 1,
872
+ "generation_latency_seconds": 0.652067915070802,
873
+ "batch_wall_latency_seconds": 0.6656363360816613,
874
+ "amortized_generation_ms_per_sample": 652.067915070802,
875
+ "amortized_batch_wall_ms_per_sample": 665.6363360816613,
876
+ "outputs": [
877
+ {
878
+ "raw_response": "{\"objects\": []}",
879
+ "valid_prediction": {
880
+ "objects": []
881
+ },
882
+ "error": null,
883
+ "finish_reason": "stop",
884
+ "completion_tokens": 6,
885
+ "prompt_tokens": 561
886
+ }
887
+ ]
888
+ },
889
+ {
890
+ "sample_ids": [
891
+ "testA-79438-218690"
892
+ ],
893
+ "batch_size": 1,
894
+ "generation_latency_seconds": 3.1018499380443245,
895
+ "batch_wall_latency_seconds": 3.1217426829971373,
896
+ "amortized_generation_ms_per_sample": 3101.8499380443245,
897
+ "amortized_batch_wall_ms_per_sample": 3121.7426829971373,
898
+ "outputs": [
899
+ {
900
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [255, 173, 536, 923], \"score\": 0.95}]}",
901
+ "valid_prediction": {
902
+ "objects": [
903
+ {
904
+ "bbox_2d": [
905
+ 255,
906
+ 173,
907
+ 536,
908
+ 923
909
+ ],
910
+ "score": 0.95
911
+ }
912
+ ]
913
+ },
914
+ "error": null,
915
+ "finish_reason": "stop",
916
+ "completion_tokens": 40,
917
+ "prompt_tokens": 562
918
+ }
919
+ ]
920
+ },
921
+ {
922
+ "sample_ids": [
923
+ "testA-62940-173454"
924
+ ],
925
+ "batch_size": 1,
926
+ "generation_latency_seconds": 3.057670062989928,
927
+ "batch_wall_latency_seconds": 3.073035439942032,
928
+ "amortized_generation_ms_per_sample": 3057.670062989928,
929
+ "amortized_batch_wall_ms_per_sample": 3073.035439942032,
930
+ "outputs": [
931
+ {
932
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [626, 266, 751, 693], \"score\": 0.9}]}",
933
+ "valid_prediction": {
934
+ "objects": [
935
+ {
936
+ "bbox_2d": [
937
+ 626,
938
+ 266,
939
+ 751,
940
+ 693
941
+ ],
942
+ "score": 0.9
943
+ }
944
+ ]
945
+ },
946
+ "error": null,
947
+ "finish_reason": "stop",
948
+ "completion_tokens": 39,
949
+ "prompt_tokens": 605
950
+ }
951
+ ]
952
+ },
953
+ {
954
+ "sample_ids": [
955
+ "testA-81245-223632"
956
+ ],
957
+ "batch_size": 1,
958
+ "generation_latency_seconds": 10.252644294989295,
959
+ "batch_wall_latency_seconds": 10.266198730911128,
960
+ "amortized_generation_ms_per_sample": 10252.644294989295,
961
+ "amortized_batch_wall_ms_per_sample": 10266.198730911128,
962
+ "outputs": [
963
+ {
964
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [584, 117, 783, 882], \"score\": 0.9}, {\"bbox_2d\": [707, 131, 792, 761], \"score\": 0.9}, {\"bbox_2d\": [798, 105, 915, 655], \"score\": 0.9}, {\"bbox_2d\": [533, 178, 655, 705], \"score\": 0.9}]}",
965
+ "valid_prediction": {
966
+ "objects": [
967
+ {
968
+ "bbox_2d": [
969
+ 584,
970
+ 117,
971
+ 783,
972
+ 882
973
+ ],
974
+ "score": 0.9
975
+ },
976
+ {
977
+ "bbox_2d": [
978
+ 707,
979
+ 131,
980
+ 792,
981
+ 761
982
+ ],
983
+ "score": 0.9
984
+ },
985
+ {
986
+ "bbox_2d": [
987
+ 798,
988
+ 105,
989
+ 915,
990
+ 655
991
+ ],
992
+ "score": 0.9
993
+ },
994
+ {
995
+ "bbox_2d": [
996
+ 533,
997
+ 178,
998
+ 655,
999
+ 705
1000
+ ],
1001
+ "score": 0.9
1002
+ }
1003
+ ]
1004
+ },
1005
+ "error": null,
1006
+ "finish_reason": "stop",
1007
+ "completion_tokens": 141,
1008
+ "prompt_tokens": 570
1009
+ }
1010
+ ]
1011
+ },
1012
+ {
1013
+ "sample_ids": [
1014
+ "testA-47070-129599"
1015
+ ],
1016
+ "batch_size": 1,
1017
+ "generation_latency_seconds": 0.6474383240565658,
1018
+ "batch_wall_latency_seconds": 0.6609540630597621,
1019
+ "amortized_generation_ms_per_sample": 647.4383240565658,
1020
+ "amortized_batch_wall_ms_per_sample": 660.9540630597621,
1021
+ "outputs": [
1022
+ {
1023
+ "raw_response": "{\"objects\": []}",
1024
+ "valid_prediction": {
1025
+ "objects": []
1026
+ },
1027
+ "error": null,
1028
+ "finish_reason": "stop",
1029
+ "completion_tokens": 6,
1030
+ "prompt_tokens": 560
1031
+ }
1032
+ ]
1033
+ },
1034
+ {
1035
+ "sample_ids": [
1036
+ "testB-12381-34098"
1037
+ ],
1038
+ "batch_size": 1,
1039
+ "generation_latency_seconds": 6.267876331927255,
1040
+ "batch_wall_latency_seconds": 6.293664561933838,
1041
+ "amortized_generation_ms_per_sample": 6267.876331927255,
1042
+ "amortized_batch_wall_ms_per_sample": 6293.664561933838,
1043
+ "outputs": [
1044
+ {
1045
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[458,242,771,473],\"score\":0.9},{\"bbox_2d\":[298,343,826,817],\"score\":0.9},{\"bbox_2d\":[305,466,793,840],\"score\":0.9}]}",
1046
+ "valid_prediction": {
1047
+ "objects": [
1048
+ {
1049
+ "bbox_2d": [
1050
+ 458,
1051
+ 242,
1052
+ 771,
1053
+ 473
1054
+ ],
1055
+ "score": 0.9
1056
+ },
1057
+ {
1058
+ "bbox_2d": [
1059
+ 298,
1060
+ 343,
1061
+ 826,
1062
+ 817
1063
+ ],
1064
+ "score": 0.9
1065
+ },
1066
+ {
1067
+ "bbox_2d": [
1068
+ 305,
1069
+ 466,
1070
+ 793,
1071
+ 840
1072
+ ],
1073
+ "score": 0.9
1074
+ }
1075
+ ]
1076
+ },
1077
+ "error": null,
1078
+ "finish_reason": "stop",
1079
+ "completion_tokens": 86,
1080
+ "prompt_tokens": 574
1081
+ }
1082
+ ]
1083
+ },
1084
+ {
1085
+ "sample_ids": [
1086
+ "testA-44142-121498"
1087
+ ],
1088
+ "batch_size": 1,
1089
+ "generation_latency_seconds": 5.1425224100239575,
1090
+ "batch_wall_latency_seconds": 5.153767512994818,
1091
+ "amortized_generation_ms_per_sample": 5142.5224100239575,
1092
+ "amortized_batch_wall_ms_per_sample": 5153.767512994818,
1093
+ "outputs": [
1094
+ {
1095
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [0, 95, 585, 987], \"score\": 0.9}, {\"bbox_2d\": [567, 126, 999, 987], \"score\": 0.9}]}",
1096
+ "valid_prediction": {
1097
+ "objects": [
1098
+ {
1099
+ "bbox_2d": [
1100
+ 0,
1101
+ 95,
1102
+ 585,
1103
+ 987
1104
+ ],
1105
+ "score": 0.9
1106
+ },
1107
+ {
1108
+ "bbox_2d": [
1109
+ 567,
1110
+ 126,
1111
+ 999,
1112
+ 987
1113
+ ],
1114
+ "score": 0.9
1115
+ }
1116
+ ]
1117
+ },
1118
+ "error": null,
1119
+ "finish_reason": "stop",
1120
+ "completion_tokens": 70,
1121
+ "prompt_tokens": 563
1122
+ }
1123
+ ]
1124
+ },
1125
+ {
1126
+ "sample_ids": [
1127
+ "testA-73258-201893"
1128
+ ],
1129
+ "batch_size": 1,
1130
+ "generation_latency_seconds": 2.4450040519004688,
1131
+ "batch_wall_latency_seconds": 2.4576197939459234,
1132
+ "amortized_generation_ms_per_sample": 2445.0040519004688,
1133
+ "amortized_batch_wall_ms_per_sample": 2457.6197939459234,
1134
+ "outputs": [
1135
+ {
1136
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[401,274,999,987],\"score\":0.9}]}",
1137
+ "valid_prediction": {
1138
+ "objects": [
1139
+ {
1140
+ "bbox_2d": [
1141
+ 401,
1142
+ 274,
1143
+ 999,
1144
+ 987
1145
+ ],
1146
+ "score": 0.9
1147
+ }
1148
+ ]
1149
+ },
1150
+ "error": null,
1151
+ "finish_reason": "stop",
1152
+ "completion_tokens": 32,
1153
+ "prompt_tokens": 606
1154
+ }
1155
+ ]
1156
+ },
1157
+ {
1158
+ "sample_ids": [
1159
+ "testB-46644-128436"
1160
+ ],
1161
+ "batch_size": 1,
1162
+ "generation_latency_seconds": 4.464105351013131,
1163
+ "batch_wall_latency_seconds": 4.473494493984617,
1164
+ "amortized_generation_ms_per_sample": 4464.105351013131,
1165
+ "amortized_batch_wall_ms_per_sample": 4473.494493984617,
1166
+ "outputs": [
1167
+ {
1168
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[10,183,537,826],\"score\":0.95},{\"bbox_2d\":[483,0,999,743],\"score\":0.95}]}",
1169
+ "valid_prediction": {
1170
+ "objects": [
1171
+ {
1172
+ "bbox_2d": [
1173
+ 10,
1174
+ 183,
1175
+ 537,
1176
+ 826
1177
+ ],
1178
+ "score": 0.95
1179
+ },
1180
+ {
1181
+ "bbox_2d": [
1182
+ 483,
1183
+ 0,
1184
+ 999,
1185
+ 743
1186
+ ],
1187
+ "score": 0.95
1188
+ }
1189
+ ]
1190
+ },
1191
+ "error": null,
1192
+ "finish_reason": "stop",
1193
+ "completion_tokens": 58,
1194
+ "prompt_tokens": 497
1195
+ }
1196
+ ]
1197
+ },
1198
+ {
1199
+ "sample_ids": [
1200
+ "testA-43889-120792"
1201
+ ],
1202
+ "batch_size": 1,
1203
+ "generation_latency_seconds": 3.0225026920670643,
1204
+ "batch_wall_latency_seconds": 3.039261986967176,
1205
+ "amortized_generation_ms_per_sample": 3022.5026920670643,
1206
+ "amortized_batch_wall_ms_per_sample": 3039.261986967176,
1207
+ "outputs": [
1208
+ {
1209
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [305, 303, 475, 882], \"score\": 0.9}]}",
1210
+ "valid_prediction": {
1211
+ "objects": [
1212
+ {
1213
+ "bbox_2d": [
1214
+ 305,
1215
+ 303,
1216
+ 475,
1217
+ 882
1218
+ ],
1219
+ "score": 0.9
1220
+ }
1221
+ ]
1222
+ },
1223
+ "error": null,
1224
+ "finish_reason": "stop",
1225
+ "completion_tokens": 39,
1226
+ "prompt_tokens": 571
1227
+ }
1228
+ ]
1229
+ },
1230
+ {
1231
+ "sample_ids": [
1232
+ "testB-36082-99363"
1233
+ ],
1234
+ "batch_size": 1,
1235
+ "generation_latency_seconds": 4.3171981680206954,
1236
+ "batch_wall_latency_seconds": 4.324737289105542,
1237
+ "amortized_generation_ms_per_sample": 4317.198168020695,
1238
+ "amortized_batch_wall_ms_per_sample": 4324.737289105542,
1239
+ "outputs": [
1240
+ {
1241
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[0,107,417,876],\"score\":0.95},{\"bbox_2d\":[442,0,999,986],\"score\":0.95}]}",
1242
+ "valid_prediction": {
1243
+ "objects": [
1244
+ {
1245
+ "bbox_2d": [
1246
+ 0,
1247
+ 107,
1248
+ 417,
1249
+ 876
1250
+ ],
1251
+ "score": 0.95
1252
+ },
1253
+ {
1254
+ "bbox_2d": [
1255
+ 442,
1256
+ 0,
1257
+ 999,
1258
+ 986
1259
+ ],
1260
+ "score": 0.95
1261
+ }
1262
+ ]
1263
+ },
1264
+ "error": null,
1265
+ "finish_reason": "stop",
1266
+ "completion_tokens": 57,
1267
+ "prompt_tokens": 412
1268
+ }
1269
+ ]
1270
+ },
1271
+ {
1272
+ "sample_ids": [
1273
+ "testB-33108-91152"
1274
+ ],
1275
+ "batch_size": 1,
1276
+ "generation_latency_seconds": 2.9848218420520425,
1277
+ "batch_wall_latency_seconds": 2.9953485280275345,
1278
+ "amortized_generation_ms_per_sample": 2984.8218420520425,
1279
+ "amortized_batch_wall_ms_per_sample": 2995.3485280275345,
1280
+ "outputs": [
1281
+ {
1282
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [58, 201, 385, 458], \"score\": 0.95}]}",
1283
+ "valid_prediction": {
1284
+ "objects": [
1285
+ {
1286
+ "bbox_2d": [
1287
+ 58,
1288
+ 201,
1289
+ 385,
1290
+ 458
1291
+ ],
1292
+ "score": 0.95
1293
+ }
1294
+ ]
1295
+ },
1296
+ "error": null,
1297
+ "finish_reason": "stop",
1298
+ "completion_tokens": 39,
1299
+ "prompt_tokens": 602
1300
+ }
1301
+ ]
1302
+ },
1303
+ {
1304
+ "sample_ids": [
1305
+ "testA-26856-73830"
1306
+ ],
1307
+ "batch_size": 1,
1308
+ "generation_latency_seconds": 3.0266257260227576,
1309
+ "batch_wall_latency_seconds": 3.0415181359276175,
1310
+ "amortized_generation_ms_per_sample": 3026.6257260227576,
1311
+ "amortized_batch_wall_ms_per_sample": 3041.5181359276175,
1312
+ "outputs": [
1313
+ {
1314
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [576, 343, 783, 713], \"score\": 0.95}]}",
1315
+ "valid_prediction": {
1316
+ "objects": [
1317
+ {
1318
+ "bbox_2d": [
1319
+ 576,
1320
+ 343,
1321
+ 783,
1322
+ 713
1323
+ ],
1324
+ "score": 0.95
1325
+ }
1326
+ ]
1327
+ },
1328
+ "error": null,
1329
+ "finish_reason": "stop",
1330
+ "completion_tokens": 40,
1331
+ "prompt_tokens": 562
1332
+ }
1333
+ ]
1334
+ },
1335
+ {
1336
+ "sample_ids": [
1337
+ "testA-81271-223709"
1338
+ ],
1339
+ "batch_size": 1,
1340
+ "generation_latency_seconds": 0.6367205450078472,
1341
+ "batch_wall_latency_seconds": 0.6504157679155469,
1342
+ "amortized_generation_ms_per_sample": 636.7205450078472,
1343
+ "amortized_batch_wall_ms_per_sample": 650.4157679155469,
1344
+ "outputs": [
1345
+ {
1346
+ "raw_response": "{\"objects\": []}",
1347
+ "valid_prediction": {
1348
+ "objects": []
1349
+ },
1350
+ "error": null,
1351
+ "finish_reason": "stop",
1352
+ "completion_tokens": 6,
1353
+ "prompt_tokens": 560
1354
+ }
1355
+ ]
1356
+ },
1357
+ {
1358
+ "sample_ids": [
1359
+ "testA-54554-150141"
1360
+ ],
1361
+ "batch_size": 1,
1362
+ "generation_latency_seconds": 11.380541390972212,
1363
+ "batch_wall_latency_seconds": 11.400562496040948,
1364
+ "amortized_generation_ms_per_sample": 11380.541390972212,
1365
+ "amortized_batch_wall_ms_per_sample": 11400.562496040948,
1366
+ "outputs": [
1367
+ {
1368
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[27,0,121,243],\"score\":0.9},{\"bbox_2d\":[62,0,185,258],\"score\":0.9},{\"bbox_2d\":[142,0,222,255],\"score\":0.9},{\"bbox_2d\":[185,0,283,261],\"score\":0.9},{\"bbox_2d\":[277,312,597,826],\"score\":0.9},{\"bbox_2d\":[655,500,951,871],\"score\":0.9}]}",
1369
+ "valid_prediction": {
1370
+ "objects": [
1371
+ {
1372
+ "bbox_2d": [
1373
+ 27,
1374
+ 0,
1375
+ 121,
1376
+ 243
1377
+ ],
1378
+ "score": 0.9
1379
+ },
1380
+ {
1381
+ "bbox_2d": [
1382
+ 62,
1383
+ 0,
1384
+ 185,
1385
+ 258
1386
+ ],
1387
+ "score": 0.9
1388
+ },
1389
+ {
1390
+ "bbox_2d": [
1391
+ 142,
1392
+ 0,
1393
+ 222,
1394
+ 255
1395
+ ],
1396
+ "score": 0.9
1397
+ },
1398
+ {
1399
+ "bbox_2d": [
1400
+ 185,
1401
+ 0,
1402
+ 283,
1403
+ 261
1404
+ ],
1405
+ "score": 0.9
1406
+ },
1407
+ {
1408
+ "bbox_2d": [
1409
+ 277,
1410
+ 312,
1411
+ 597,
1412
+ 826
1413
+ ],
1414
+ "score": 0.9
1415
+ },
1416
+ {
1417
+ "bbox_2d": [
1418
+ 655,
1419
+ 500,
1420
+ 951,
1421
+ 871
1422
+ ],
1423
+ "score": 0.9
1424
+ }
1425
+ ]
1426
+ },
1427
+ "error": null,
1428
+ "finish_reason": "stop",
1429
+ "completion_tokens": 157,
1430
+ "prompt_tokens": 610
1431
+ }
1432
+ ]
1433
+ },
1434
+ {
1435
+ "sample_ids": [
1436
+ "testB-38636-106481"
1437
+ ],
1438
+ "batch_size": 1,
1439
+ "generation_latency_seconds": 4.233924768050201,
1440
+ "batch_wall_latency_seconds": 4.251373155973852,
1441
+ "amortized_generation_ms_per_sample": 4233.924768050201,
1442
+ "amortized_batch_wall_ms_per_sample": 4251.373155973852,
1443
+ "outputs": [
1444
+ {
1445
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[494,177,881,985],\"score\":0.95},{\"bbox_2d\":[0,42,307,722],\"score\":0.95}]}",
1446
+ "valid_prediction": {
1447
+ "objects": [
1448
+ {
1449
+ "bbox_2d": [
1450
+ 494,
1451
+ 177,
1452
+ 881,
1453
+ 985
1454
+ ],
1455
+ "score": 0.95
1456
+ },
1457
+ {
1458
+ "bbox_2d": [
1459
+ 0,
1460
+ 42,
1461
+ 307,
1462
+ 722
1463
+ ],
1464
+ "score": 0.95
1465
+ }
1466
+ ]
1467
+ },
1468
+ "error": null,
1469
+ "finish_reason": "stop",
1470
+ "completion_tokens": 58,
1471
+ "prompt_tokens": 469
1472
+ }
1473
+ ]
1474
+ },
1475
+ {
1476
+ "sample_ids": [
1477
+ "testA-5734-15837"
1478
+ ],
1479
+ "batch_size": 1,
1480
+ "generation_latency_seconds": 0.6504808579338714,
1481
+ "batch_wall_latency_seconds": 0.6668235529214144,
1482
+ "amortized_generation_ms_per_sample": 650.4808579338714,
1483
+ "amortized_batch_wall_ms_per_sample": 666.8235529214144,
1484
+ "outputs": [
1485
+ {
1486
+ "raw_response": "{\"objects\": []}",
1487
+ "valid_prediction": {
1488
+ "objects": []
1489
+ },
1490
+ "error": null,
1491
+ "finish_reason": "stop",
1492
+ "completion_tokens": 6,
1493
+ "prompt_tokens": 565
1494
+ }
1495
+ ]
1496
+ },
1497
+ {
1498
+ "sample_ids": [
1499
+ "testB-67558-186325"
1500
+ ],
1501
+ "batch_size": 1,
1502
+ "generation_latency_seconds": 0.6280788460280746,
1503
+ "batch_wall_latency_seconds": 0.6436845100251958,
1504
+ "amortized_generation_ms_per_sample": 628.0788460280746,
1505
+ "amortized_batch_wall_ms_per_sample": 643.6845100251958,
1506
+ "outputs": [
1507
+ {
1508
+ "raw_response": "{\"objects\": []}",
1509
+ "valid_prediction": {
1510
+ "objects": []
1511
+ },
1512
+ "error": null,
1513
+ "finish_reason": "stop",
1514
+ "completion_tokens": 6,
1515
+ "prompt_tokens": 600
1516
+ }
1517
+ ]
1518
+ },
1519
+ {
1520
+ "sample_ids": [
1521
+ "testB-47053-129554"
1522
+ ],
1523
+ "batch_size": 1,
1524
+ "generation_latency_seconds": 4.556327463011257,
1525
+ "batch_wall_latency_seconds": 4.573514219024219,
1526
+ "amortized_generation_ms_per_sample": 4556.327463011257,
1527
+ "amortized_batch_wall_ms_per_sample": 4573.514219024219,
1528
+ "outputs": [
1529
+ {
1530
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[438,321,871,542],\"score\":0.95},{\"bbox_2d\":[438,452,851,863],\"score\":0.95}]}",
1531
+ "valid_prediction": {
1532
+ "objects": [
1533
+ {
1534
+ "bbox_2d": [
1535
+ 438,
1536
+ 321,
1537
+ 871,
1538
+ 542
1539
+ ],
1540
+ "score": 0.95
1541
+ },
1542
+ {
1543
+ "bbox_2d": [
1544
+ 438,
1545
+ 452,
1546
+ 851,
1547
+ 863
1548
+ ],
1549
+ "score": 0.95
1550
+ }
1551
+ ]
1552
+ },
1553
+ "error": null,
1554
+ "finish_reason": "stop",
1555
+ "completion_tokens": 61,
1556
+ "prompt_tokens": 604
1557
+ }
1558
+ ]
1559
+ },
1560
+ {
1561
+ "sample_ids": [
1562
+ "testA-93064-256422"
1563
+ ],
1564
+ "batch_size": 1,
1565
+ "generation_latency_seconds": 2.830894450074993,
1566
+ "batch_wall_latency_seconds": 2.8475917810574174,
1567
+ "amortized_generation_ms_per_sample": 2830.894450074993,
1568
+ "amortized_batch_wall_ms_per_sample": 2847.5917810574174,
1569
+ "outputs": [
1570
+ {
1571
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [0, 226, 173, 755], \"score\": 0.9}]}",
1572
+ "valid_prediction": {
1573
+ "objects": [
1574
+ {
1575
+ "bbox_2d": [
1576
+ 0,
1577
+ 226,
1578
+ 173,
1579
+ 755
1580
+ ],
1581
+ "score": 0.9
1582
+ }
1583
+ ]
1584
+ },
1585
+ "error": null,
1586
+ "finish_reason": "stop",
1587
+ "completion_tokens": 37,
1588
+ "prompt_tokens": 549
1589
+ }
1590
+ ]
1591
+ },
1592
+ {
1593
+ "sample_ids": [
1594
+ "testB-60557-166821"
1595
+ ],
1596
+ "batch_size": 1,
1597
+ "generation_latency_seconds": 5.58066594589036,
1598
+ "batch_wall_latency_seconds": 5.594979963963851,
1599
+ "amortized_generation_ms_per_sample": 5580.66594589036,
1600
+ "amortized_batch_wall_ms_per_sample": 5594.979963963851,
1601
+ "outputs": [
1602
+ {
1603
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [857, 442, 875, 477], \"score\": 0.9}, {\"bbox_2d\": [347, 442, 365, 477], \"score\": 0.9}]}",
1604
+ "valid_prediction": {
1605
+ "objects": [
1606
+ {
1607
+ "bbox_2d": [
1608
+ 857,
1609
+ 442,
1610
+ 875,
1611
+ 477
1612
+ ],
1613
+ "score": 0.9
1614
+ },
1615
+ {
1616
+ "bbox_2d": [
1617
+ 347,
1618
+ 442,
1619
+ 365,
1620
+ 477
1621
+ ],
1622
+ "score": 0.9
1623
+ }
1624
+ ]
1625
+ },
1626
+ "error": null,
1627
+ "finish_reason": "stop",
1628
+ "completion_tokens": 73,
1629
+ "prompt_tokens": 500
1630
+ }
1631
+ ]
1632
+ },
1633
+ {
1634
+ "sample_ids": [
1635
+ "testA-55116-151688"
1636
+ ],
1637
+ "batch_size": 1,
1638
+ "generation_latency_seconds": 0.6620599570451304,
1639
+ "batch_wall_latency_seconds": 0.6773106920300052,
1640
+ "amortized_generation_ms_per_sample": 662.0599570451304,
1641
+ "amortized_batch_wall_ms_per_sample": 677.3106920300052,
1642
+ "outputs": [
1643
+ {
1644
+ "raw_response": "{\"objects\": []}",
1645
+ "valid_prediction": {
1646
+ "objects": []
1647
+ },
1648
+ "error": null,
1649
+ "finish_reason": "stop",
1650
+ "completion_tokens": 6,
1651
+ "prompt_tokens": 600
1652
+ }
1653
+ ]
1654
+ },
1655
+ {
1656
+ "sample_ids": [
1657
+ "testB-65796-181392"
1658
+ ],
1659
+ "batch_size": 1,
1660
+ "generation_latency_seconds": 0.6342472250107676,
1661
+ "batch_wall_latency_seconds": 0.6483092680573463,
1662
+ "amortized_generation_ms_per_sample": 634.2472250107676,
1663
+ "amortized_batch_wall_ms_per_sample": 648.3092680573463,
1664
+ "outputs": [
1665
+ {
1666
+ "raw_response": "{\"objects\": []}",
1667
+ "valid_prediction": {
1668
+ "objects": []
1669
+ },
1670
+ "error": null,
1671
+ "finish_reason": "stop",
1672
+ "completion_tokens": 6,
1673
+ "prompt_tokens": 581
1674
+ }
1675
+ ]
1676
+ },
1677
+ {
1678
+ "sample_ids": [
1679
+ "testB-23014-63278"
1680
+ ],
1681
+ "batch_size": 1,
1682
+ "generation_latency_seconds": 2.7931598050054163,
1683
+ "batch_wall_latency_seconds": 2.8087078949902207,
1684
+ "amortized_generation_ms_per_sample": 2793.1598050054163,
1685
+ "amortized_batch_wall_ms_per_sample": 2808.7078949902207,
1686
+ "outputs": [
1687
+ {
1688
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[33, 303, 466, 761],\"score\":0.95}]}",
1689
+ "valid_prediction": {
1690
+ "objects": [
1691
+ {
1692
+ "bbox_2d": [
1693
+ 33,
1694
+ 303,
1695
+ 466,
1696
+ 761
1697
+ ],
1698
+ "score": 0.95
1699
+ }
1700
+ ]
1701
+ },
1702
+ "error": null,
1703
+ "finish_reason": "stop",
1704
+ "completion_tokens": 35,
1705
+ "prompt_tokens": 563
1706
+ }
1707
+ ]
1708
+ },
1709
+ {
1710
+ "sample_ids": [
1711
+ "testA-24562-67515"
1712
+ ],
1713
+ "batch_size": 1,
1714
+ "generation_latency_seconds": 0.6919117919169366,
1715
+ "batch_wall_latency_seconds": 0.7147108289645985,
1716
+ "amortized_generation_ms_per_sample": 691.9117919169366,
1717
+ "amortized_batch_wall_ms_per_sample": 714.7108289645985,
1718
+ "outputs": [
1719
+ {
1720
+ "raw_response": "{\"objects\": []}",
1721
+ "valid_prediction": {
1722
+ "objects": []
1723
+ },
1724
+ "error": null,
1725
+ "finish_reason": "stop",
1726
+ "completion_tokens": 6,
1727
+ "prompt_tokens": 606
1728
+ }
1729
+ ]
1730
+ },
1731
+ {
1732
+ "sample_ids": [
1733
+ "testA-17736-48684"
1734
+ ],
1735
+ "batch_size": 1,
1736
+ "generation_latency_seconds": 3.0041811569826677,
1737
+ "batch_wall_latency_seconds": 3.0170002209488302,
1738
+ "amortized_generation_ms_per_sample": 3004.1811569826677,
1739
+ "amortized_batch_wall_ms_per_sample": 3017.0002209488302,
1740
+ "outputs": [
1741
+ {
1742
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [508, 426, 765, 983], \"score\": 0.9}]}",
1743
+ "valid_prediction": {
1744
+ "objects": [
1745
+ {
1746
+ "bbox_2d": [
1747
+ 508,
1748
+ 426,
1749
+ 765,
1750
+ 983
1751
+ ],
1752
+ "score": 0.9
1753
+ }
1754
+ ]
1755
+ },
1756
+ "error": null,
1757
+ "finish_reason": "stop",
1758
+ "completion_tokens": 39,
1759
+ "prompt_tokens": 605
1760
+ }
1761
+ ]
1762
+ },
1763
+ {
1764
+ "sample_ids": [
1765
+ "testA-58475-160949"
1766
+ ],
1767
+ "batch_size": 1,
1768
+ "generation_latency_seconds": 2.428418811992742,
1769
+ "batch_wall_latency_seconds": 2.442452701041475,
1770
+ "amortized_generation_ms_per_sample": 2428.418811992742,
1771
+ "amortized_batch_wall_ms_per_sample": 2442.452701041475,
1772
+ "outputs": [
1773
+ {
1774
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[46,36,576,999],\"score\":0.95}]}",
1775
+ "valid_prediction": {
1776
+ "objects": [
1777
+ {
1778
+ "bbox_2d": [
1779
+ 46,
1780
+ 36,
1781
+ 576,
1782
+ 999
1783
+ ],
1784
+ "score": 0.95
1785
+ }
1786
+ ]
1787
+ },
1788
+ "error": null,
1789
+ "finish_reason": "stop",
1790
+ "completion_tokens": 31,
1791
+ "prompt_tokens": 564
1792
+ }
1793
+ ]
1794
+ },
1795
+ {
1796
+ "sample_ids": [
1797
+ "testA-80455-221493"
1798
+ ],
1799
+ "batch_size": 1,
1800
+ "generation_latency_seconds": 2.4254801200004295,
1801
+ "batch_wall_latency_seconds": 2.4414882709970698,
1802
+ "amortized_generation_ms_per_sample": 2425.4801200004295,
1803
+ "amortized_batch_wall_ms_per_sample": 2441.48827099707,
1804
+ "outputs": [
1805
+ {
1806
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[0,483,581,938],\"score\":0.95}]}",
1807
+ "valid_prediction": {
1808
+ "objects": [
1809
+ {
1810
+ "bbox_2d": [
1811
+ 0,
1812
+ 483,
1813
+ 581,
1814
+ 938
1815
+ ],
1816
+ "score": 0.95
1817
+ }
1818
+ ]
1819
+ },
1820
+ "error": null,
1821
+ "finish_reason": "stop",
1822
+ "completion_tokens": 31,
1823
+ "prompt_tokens": 541
1824
+ }
1825
+ ]
1826
+ },
1827
+ {
1828
+ "sample_ids": [
1829
+ "testA-31763-87405"
1830
+ ],
1831
+ "batch_size": 1,
1832
+ "generation_latency_seconds": 7.767380379955284,
1833
+ "batch_wall_latency_seconds": 7.780358829069883,
1834
+ "amortized_generation_ms_per_sample": 7767.380379955284,
1835
+ "amortized_batch_wall_ms_per_sample": 7780.358829069883,
1836
+ "outputs": [
1837
+ {
1838
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [626, 737, 652, 821], \"score\": 0.9}, {\"bbox_2d\": [426, 913, 452, 972], \"score\": 0.9}, {\"bbox_2d\": [356, 852, 378, 936], \"score\": 0.9}]}",
1839
+ "valid_prediction": {
1840
+ "objects": [
1841
+ {
1842
+ "bbox_2d": [
1843
+ 626,
1844
+ 737,
1845
+ 652,
1846
+ 821
1847
+ ],
1848
+ "score": 0.9
1849
+ },
1850
+ {
1851
+ "bbox_2d": [
1852
+ 426,
1853
+ 913,
1854
+ 452,
1855
+ 972
1856
+ ],
1857
+ "score": 0.9
1858
+ },
1859
+ {
1860
+ "bbox_2d": [
1861
+ 356,
1862
+ 852,
1863
+ 378,
1864
+ 936
1865
+ ],
1866
+ "score": 0.9
1867
+ }
1868
+ ]
1869
+ },
1870
+ "error": null,
1871
+ "finish_reason": "stop",
1872
+ "completion_tokens": 107,
1873
+ "prompt_tokens": 445
1874
+ }
1875
+ ]
1876
+ },
1877
+ {
1878
+ "sample_ids": [
1879
+ "testB-23920-65742"
1880
+ ],
1881
+ "batch_size": 1,
1882
+ "generation_latency_seconds": 2.9294577399268746,
1883
+ "batch_wall_latency_seconds": 2.9447807769756764,
1884
+ "amortized_generation_ms_per_sample": 2929.4577399268746,
1885
+ "amortized_batch_wall_ms_per_sample": 2944.7807769756764,
1886
+ "outputs": [
1887
+ {
1888
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [537, 417, 999, 993], \"score\": 0.9}]}",
1889
+ "valid_prediction": {
1890
+ "objects": [
1891
+ {
1892
+ "bbox_2d": [
1893
+ 537,
1894
+ 417,
1895
+ 999,
1896
+ 993
1897
+ ],
1898
+ "score": 0.9
1899
+ }
1900
+ ]
1901
+ },
1902
+ "error": null,
1903
+ "finish_reason": "stop",
1904
+ "completion_tokens": 39,
1905
+ "prompt_tokens": 568
1906
+ }
1907
+ ]
1908
+ },
1909
+ {
1910
+ "sample_ids": [
1911
+ "testA-469-1309"
1912
+ ],
1913
+ "batch_size": 1,
1914
+ "generation_latency_seconds": 0.6524032501038164,
1915
+ "batch_wall_latency_seconds": 0.6627495769644156,
1916
+ "amortized_generation_ms_per_sample": 652.4032501038164,
1917
+ "amortized_batch_wall_ms_per_sample": 662.7495769644156,
1918
+ "outputs": [
1919
+ {
1920
+ "raw_response": "{\"objects\": []}",
1921
+ "valid_prediction": {
1922
+ "objects": []
1923
+ },
1924
+ "error": null,
1925
+ "finish_reason": "stop",
1926
+ "completion_tokens": 6,
1927
+ "prompt_tokens": 621
1928
+ }
1929
+ ]
1930
+ },
1931
+ {
1932
+ "sample_ids": [
1933
+ "testB-56691-155998"
1934
+ ],
1935
+ "batch_size": 1,
1936
+ "generation_latency_seconds": 0.655415665009059,
1937
+ "batch_wall_latency_seconds": 0.67216857499443,
1938
+ "amortized_generation_ms_per_sample": 655.415665009059,
1939
+ "amortized_batch_wall_ms_per_sample": 672.16857499443,
1940
+ "outputs": [
1941
+ {
1942
+ "raw_response": "{\"objects\": []}",
1943
+ "valid_prediction": {
1944
+ "objects": []
1945
+ },
1946
+ "error": null,
1947
+ "finish_reason": "stop",
1948
+ "completion_tokens": 6,
1949
+ "prompt_tokens": 563
1950
+ }
1951
+ ]
1952
+ },
1953
+ {
1954
+ "sample_ids": [
1955
+ "testB-31493-86668"
1956
+ ],
1957
+ "batch_size": 1,
1958
+ "generation_latency_seconds": 4.518256991053931,
1959
+ "batch_wall_latency_seconds": 4.5385722850915045,
1960
+ "amortized_generation_ms_per_sample": 4518.256991053931,
1961
+ "amortized_batch_wall_ms_per_sample": 4538.572285091504,
1962
+ "outputs": [
1963
+ {
1964
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[537,426,747,765],\"score\":0.95},{\"bbox_2d\":[152,426,466,462],\"score\":0.95}]}",
1965
+ "valid_prediction": {
1966
+ "objects": [
1967
+ {
1968
+ "bbox_2d": [
1969
+ 537,
1970
+ 426,
1971
+ 747,
1972
+ 765
1973
+ ],
1974
+ "score": 0.95
1975
+ },
1976
+ {
1977
+ "bbox_2d": [
1978
+ 152,
1979
+ 426,
1980
+ 466,
1981
+ 462
1982
+ ],
1983
+ "score": 0.95
1984
+ }
1985
+ ]
1986
+ },
1987
+ "error": null,
1988
+ "finish_reason": "stop",
1989
+ "completion_tokens": 61,
1990
+ "prompt_tokens": 614
1991
+ }
1992
+ ]
1993
+ },
1994
+ {
1995
+ "sample_ids": [
1996
+ "testA-59284-163254"
1997
+ ],
1998
+ "batch_size": 1,
1999
+ "generation_latency_seconds": 5.580998525954783,
2000
+ "batch_wall_latency_seconds": 5.5959404041059315,
2001
+ "amortized_generation_ms_per_sample": 5580.998525954783,
2002
+ "amortized_batch_wall_ms_per_sample": 5595.9404041059315,
2003
+ "outputs": [
2004
+ {
2005
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [501, 381, 696, 999], \"score\": 0.9}, {\"bbox_2d\": [561, 300, 814, 426], \"score\": 0.9}]}",
2006
+ "valid_prediction": {
2007
+ "objects": [
2008
+ {
2009
+ "bbox_2d": [
2010
+ 501,
2011
+ 381,
2012
+ 696,
2013
+ 999
2014
+ ],
2015
+ "score": 0.9
2016
+ },
2017
+ {
2018
+ "bbox_2d": [
2019
+ 561,
2020
+ 300,
2021
+ 814,
2022
+ 426
2023
+ ],
2024
+ "score": 0.9
2025
+ }
2026
+ ]
2027
+ },
2028
+ "error": null,
2029
+ "finish_reason": "stop",
2030
+ "completion_tokens": 73,
2031
+ "prompt_tokens": 623
2032
+ }
2033
+ ]
2034
+ },
2035
+ {
2036
+ "sample_ids": [
2037
+ "testA-57397-158015"
2038
+ ],
2039
+ "batch_size": 1,
2040
+ "generation_latency_seconds": 0.6980972309829667,
2041
+ "batch_wall_latency_seconds": 0.709314426057972,
2042
+ "amortized_generation_ms_per_sample": 698.0972309829667,
2043
+ "amortized_batch_wall_ms_per_sample": 709.314426057972,
2044
+ "outputs": [
2045
+ {
2046
+ "raw_response": "{\"objects\": []}",
2047
+ "valid_prediction": {
2048
+ "objects": []
2049
+ },
2050
+ "error": null,
2051
+ "finish_reason": "stop",
2052
+ "completion_tokens": 6,
2053
+ "prompt_tokens": 602
2054
+ }
2055
+ ]
2056
+ },
2057
+ {
2058
+ "sample_ids": [
2059
+ "testB-2935-8111"
2060
+ ],
2061
+ "batch_size": 1,
2062
+ "generation_latency_seconds": 4.602234860998578,
2063
+ "batch_wall_latency_seconds": 4.6172516259830445,
2064
+ "amortized_generation_ms_per_sample": 4602.234860998578,
2065
+ "amortized_batch_wall_ms_per_sample": 4617.2516259830445,
2066
+ "outputs": [
2067
+ {
2068
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[371,91,707,431],\"score\":0.95},{\"bbox_2d\":[585,181,999,705],\"score\":0.95}]}",
2069
+ "valid_prediction": {
2070
+ "objects": [
2071
+ {
2072
+ "bbox_2d": [
2073
+ 371,
2074
+ 91,
2075
+ 707,
2076
+ 431
2077
+ ],
2078
+ "score": 0.95
2079
+ },
2080
+ {
2081
+ "bbox_2d": [
2082
+ 585,
2083
+ 181,
2084
+ 999,
2085
+ 705
2086
+ ],
2087
+ "score": 0.95
2088
+ }
2089
+ ]
2090
+ },
2091
+ "error": null,
2092
+ "finish_reason": "stop",
2093
+ "completion_tokens": 60,
2094
+ "prompt_tokens": 570
2095
+ }
2096
+ ]
2097
+ },
2098
+ {
2099
+ "sample_ids": [
2100
+ "testB-22290-61226"
2101
+ ],
2102
+ "batch_size": 1,
2103
+ "generation_latency_seconds": 0.6584999479819089,
2104
+ "batch_wall_latency_seconds": 0.6694054200779647,
2105
+ "amortized_generation_ms_per_sample": 658.4999479819089,
2106
+ "amortized_batch_wall_ms_per_sample": 669.4054200779647,
2107
+ "outputs": [
2108
+ {
2109
+ "raw_response": "{\"objects\": []}",
2110
+ "valid_prediction": {
2111
+ "objects": []
2112
+ },
2113
+ "error": null,
2114
+ "finish_reason": "stop",
2115
+ "completion_tokens": 6,
2116
+ "prompt_tokens": 586
2117
+ }
2118
+ ]
2119
+ },
2120
+ {
2121
+ "sample_ids": [
2122
+ "testB-8416-23140"
2123
+ ],
2124
+ "batch_size": 1,
2125
+ "generation_latency_seconds": 6.426246937015094,
2126
+ "batch_wall_latency_seconds": 6.434293171972968,
2127
+ "amortized_generation_ms_per_sample": 6426.246937015094,
2128
+ "amortized_batch_wall_ms_per_sample": 6434.293171972968,
2129
+ "outputs": [
2130
+ {
2131
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[0,292,295,642],\"score\":0.95},{\"bbox_2d\":[234,252,581,727],\"score\":0.95},{\"bbox_2d\":[722,183,959,808],\"score\":0.95}]}",
2132
+ "valid_prediction": {
2133
+ "objects": [
2134
+ {
2135
+ "bbox_2d": [
2136
+ 0,
2137
+ 292,
2138
+ 295,
2139
+ 642
2140
+ ],
2141
+ "score": 0.95
2142
+ },
2143
+ {
2144
+ "bbox_2d": [
2145
+ 234,
2146
+ 252,
2147
+ 581,
2148
+ 727
2149
+ ],
2150
+ "score": 0.95
2151
+ },
2152
+ {
2153
+ "bbox_2d": [
2154
+ 722,
2155
+ 183,
2156
+ 959,
2157
+ 808
2158
+ ],
2159
+ "score": 0.95
2160
+ }
2161
+ ]
2162
+ },
2163
+ "error": null,
2164
+ "finish_reason": "stop",
2165
+ "completion_tokens": 87,
2166
+ "prompt_tokens": 563
2167
+ }
2168
+ ]
2169
+ },
2170
+ {
2171
+ "sample_ids": [
2172
+ "testA-60731-167320"
2173
+ ],
2174
+ "batch_size": 1,
2175
+ "generation_latency_seconds": 5.329908382031135,
2176
+ "batch_wall_latency_seconds": 5.342693028971553,
2177
+ "amortized_generation_ms_per_sample": 5329.908382031135,
2178
+ "amortized_batch_wall_ms_per_sample": 5342.693028971553,
2179
+ "outputs": [
2180
+ {
2181
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [0, 81, 446, 987], \"score\": 0.9}, {\"bbox_2d\": [483, 0, 792, 201], \"score\": 0.9}]}",
2182
+ "valid_prediction": {
2183
+ "objects": [
2184
+ {
2185
+ "bbox_2d": [
2186
+ 0,
2187
+ 81,
2188
+ 446,
2189
+ 987
2190
+ ],
2191
+ "score": 0.9
2192
+ },
2193
+ {
2194
+ "bbox_2d": [
2195
+ 483,
2196
+ 0,
2197
+ 792,
2198
+ 201
2199
+ ],
2200
+ "score": 0.9
2201
+ }
2202
+ ]
2203
+ },
2204
+ "error": null,
2205
+ "finish_reason": "stop",
2206
+ "completion_tokens": 68,
2207
+ "prompt_tokens": 606
2208
+ }
2209
+ ]
2210
+ },
2211
+ {
2212
+ "sample_ids": [
2213
+ "testA-49055-135086"
2214
+ ],
2215
+ "batch_size": 1,
2216
+ "generation_latency_seconds": 4.7276995739666745,
2217
+ "batch_wall_latency_seconds": 4.746967880986631,
2218
+ "amortized_generation_ms_per_sample": 4727.6995739666745,
2219
+ "amortized_batch_wall_ms_per_sample": 4746.967880986631,
2220
+ "outputs": [
2221
+ {
2222
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[615,242,851,999],\"score\":0.95},{\"bbox_2d\":[137,142,326,999],\"score\":0.95}]}",
2223
+ "valid_prediction": {
2224
+ "objects": [
2225
+ {
2226
+ "bbox_2d": [
2227
+ 615,
2228
+ 242,
2229
+ 851,
2230
+ 999
2231
+ ],
2232
+ "score": 0.95
2233
+ },
2234
+ {
2235
+ "bbox_2d": [
2236
+ 137,
2237
+ 142,
2238
+ 326,
2239
+ 999
2240
+ ],
2241
+ "score": 0.95
2242
+ }
2243
+ ]
2244
+ },
2245
+ "error": null,
2246
+ "finish_reason": "stop",
2247
+ "completion_tokens": 61,
2248
+ "prompt_tokens": 566
2249
+ }
2250
+ ]
2251
+ },
2252
+ {
2253
+ "sample_ids": [
2254
+ "testA-7857-21644"
2255
+ ],
2256
+ "batch_size": 1,
2257
+ "generation_latency_seconds": 2.446160126943141,
2258
+ "batch_wall_latency_seconds": 2.4602624271064997,
2259
+ "amortized_generation_ms_per_sample": 2446.160126943141,
2260
+ "amortized_batch_wall_ms_per_sample": 2460.2624271064997,
2261
+ "outputs": [
2262
+ {
2263
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[668,36,999,999],\"score\":0.95}]}",
2264
+ "valid_prediction": {
2265
+ "objects": [
2266
+ {
2267
+ "bbox_2d": [
2268
+ 668,
2269
+ 36,
2270
+ 999,
2271
+ 999
2272
+ ],
2273
+ "score": 0.95
2274
+ }
2275
+ ]
2276
+ },
2277
+ "error": null,
2278
+ "finish_reason": "stop",
2279
+ "completion_tokens": 32,
2280
+ "prompt_tokens": 607
2281
+ }
2282
+ ]
2283
+ },
2284
+ {
2285
+ "sample_ids": [
2286
+ "testB-79645-219273"
2287
+ ],
2288
+ "batch_size": 1,
2289
+ "generation_latency_seconds": 15.058248669025488,
2290
+ "batch_wall_latency_seconds": 15.065027808072045,
2291
+ "amortized_generation_ms_per_sample": 15058.248669025488,
2292
+ "amortized_batch_wall_ms_per_sample": 15065.027808072045,
2293
+ "outputs": [
2294
+ {
2295
+ "raw_response": "{\"objects\": [{\"bbox_2d\": [542, 417, 798, 722], \"score\": 0.9}, {\"bbox_2d\": [295, 421, 552, 715], \"score\": 0.9}, {\"bbox_2d\": [147, 452, 310, 665], \"score\": 0.9}, {\"bbox_2d\": [55, 476, 115, 645], \"score\": 0.9}, {\"bbox_2d\": [0, 448, 62, 677], \"score\": 0.9}, {\"bbox_2d\": [113, 478, 162, 595], \"score\": 0.9}]}",
2296
+ "valid_prediction": {
2297
+ "objects": [
2298
+ {
2299
+ "bbox_2d": [
2300
+ 542,
2301
+ 417,
2302
+ 798,
2303
+ 722
2304
+ ],
2305
+ "score": 0.9
2306
+ },
2307
+ {
2308
+ "bbox_2d": [
2309
+ 295,
2310
+ 421,
2311
+ 552,
2312
+ 715
2313
+ ],
2314
+ "score": 0.9
2315
+ },
2316
+ {
2317
+ "bbox_2d": [
2318
+ 147,
2319
+ 452,
2320
+ 310,
2321
+ 665
2322
+ ],
2323
+ "score": 0.9
2324
+ },
2325
+ {
2326
+ "bbox_2d": [
2327
+ 55,
2328
+ 476,
2329
+ 115,
2330
+ 645
2331
+ ],
2332
+ "score": 0.9
2333
+ },
2334
+ {
2335
+ "bbox_2d": [
2336
+ 0,
2337
+ 448,
2338
+ 62,
2339
+ 677
2340
+ ],
2341
+ "score": 0.9
2342
+ },
2343
+ {
2344
+ "bbox_2d": [
2345
+ 113,
2346
+ 478,
2347
+ 162,
2348
+ 595
2349
+ ],
2350
+ "score": 0.9
2351
+ }
2352
+ ]
2353
+ },
2354
+ "error": null,
2355
+ "finish_reason": "stop",
2356
+ "completion_tokens": 205,
2357
+ "prompt_tokens": 601
2358
+ }
2359
+ ]
2360
+ },
2361
+ {
2362
+ "sample_ids": [
2363
+ "testB-52921-145707"
2364
+ ],
2365
+ "batch_size": 1,
2366
+ "generation_latency_seconds": 0.65654409409035,
2367
+ "batch_wall_latency_seconds": 0.6844046280020848,
2368
+ "amortized_generation_ms_per_sample": 656.54409409035,
2369
+ "amortized_batch_wall_ms_per_sample": 684.4046280020848,
2370
+ "outputs": [
2371
+ {
2372
+ "raw_response": "{\"objects\": []}",
2373
+ "valid_prediction": {
2374
+ "objects": []
2375
+ },
2376
+ "error": null,
2377
+ "finish_reason": "stop",
2378
+ "completion_tokens": 6,
2379
+ "prompt_tokens": 571
2380
+ }
2381
+ ]
2382
+ },
2383
+ {
2384
+ "sample_ids": [
2385
+ "testB-69479-191562"
2386
+ ],
2387
+ "batch_size": 1,
2388
+ "generation_latency_seconds": 2.294087173999287,
2389
+ "batch_wall_latency_seconds": 2.3162708869203925,
2390
+ "amortized_generation_ms_per_sample": 2294.087173999287,
2391
+ "amortized_batch_wall_ms_per_sample": 2316.2708869203925,
2392
+ "outputs": [
2393
+ {
2394
+ "raw_response": "{\"objects\":[{\"bbox_2d\":[0,31,972,999],\"score\":0.95}]}",
2395
+ "valid_prediction": {
2396
+ "objects": [
2397
+ {
2398
+ "bbox_2d": [
2399
+ 0,
2400
+ 31,
2401
+ 972,
2402
+ 999
2403
+ ],
2404
+ "score": 0.95
2405
+ }
2406
+ ]
2407
+ },
2408
+ "error": null,
2409
+ "finish_reason": "stop",
2410
+ "completion_tokens": 30,
2411
+ "prompt_tokens": 561
2412
+ }
2413
+ ]
2414
+ },
2415
+ {
2416
+ "sample_ids": [
2417
+ "testA-69466-191524"
2418
+ ],
2419
+ "batch_size": 1,
2420
+ "generation_latency_seconds": 0.6702312079723924,
2421
+ "batch_wall_latency_seconds": 0.683375352062285,
2422
+ "amortized_generation_ms_per_sample": 670.2312079723924,
2423
+ "amortized_batch_wall_ms_per_sample": 683.375352062285,
2424
+ "outputs": [
2425
+ {
2426
+ "raw_response": "{\"objects\": []}",
2427
+ "valid_prediction": {
2428
+ "objects": []
2429
+ },
2430
+ "error": null,
2431
+ "finish_reason": "stop",
2432
+ "completion_tokens": 6,
2433
+ "prompt_tokens": 602
2434
+ }
2435
+ ]
2436
+ }
2437
+ ],
2438
+ "mean_generation_ms_per_sample": 4056.054782922729,
2439
+ "median_generation_ms_per_sample": 3024.564209044911,
2440
+ "mean_batch_wall_ms_per_sample": 4071.4221567668574,
2441
+ "median_batch_wall_ms_per_sample": 3040.390061447397,
2442
+ "mean_prompt_tokens": 578.6875,
2443
+ "mean_completion_tokens": 53.203125,
2444
+ "latency_scope": "A800 vLLM-0.18.0 native generation latency with the frozen task prompt and structured-output schema; model loading and warmup excluded. Batch wall time includes the real request/image preparation path.",
2445
+ "flops_scope": "This file measures latency only. Counted FLOPs are measured separately by the HF reference probe and are not claimed to be complete vLLM kernel FLOPs."
2446
+ }
paper_results/data/main_table_cost_qwen_20260912/components/qwen_latency.json ADDED
The diff for this file is too large to render. See raw diff
 
paper_results/data/main_table_cost_qwen_20260912/components/yolo26x_router.json ADDED
@@ -0,0 +1,238 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "status": "complete",
3
+ "run": "shared_12_yolo26x_Qwen3.8",
4
+ "config": {
5
+ "fraction": 1.0,
6
+ "seed": 42,
7
+ "freeze_encoder": true,
8
+ "fusion": "concat",
9
+ "harm_cost": 1.0,
10
+ "batch_size": 256,
11
+ "weight_decay": 0.01,
12
+ "max_epochs": 50,
13
+ "patience": 8,
14
+ "cpu_threads": 1,
15
+ "student": "8M",
16
+ "representation_dim": 128,
17
+ "lr": 0.003,
18
+ "target": "four_state",
19
+ "strategy": "natural",
20
+ "hidden_percent": 90,
21
+ "expert": "yolo26x",
22
+ "task": "construction",
23
+ "endpoint": "Qwen3.8",
24
+ "branches": [
25
+ "image",
26
+ "hidden",
27
+ "confidence",
28
+ "output"
29
+ ],
30
+ "hidden_layer": "feature_blocks__layer_16",
31
+ "id": "shared_12_yolo26x_Qwen3.8",
32
+ "stage": "shared_mandatory_inputs"
33
+ },
34
+ "checkpoint": "/mnt/HithinkOmniSSD/user_workspace/caisihang/research/ICASSP_ModelCollaboration/outputs/router_shared_image_hidden_20260910/runs/shared_12_yolo26x_Qwen3.8/best.pt",
35
+ "score": "logit_rescue-logit_harm",
36
+ "test_rows_recomputed": 3004,
37
+ "selected_decisions_match_main": true,
38
+ "max_score_abs_difference": 0.0,
39
+ "actual_call_fraction": 0.4876830892143808,
40
+ "sample_count": 64,
41
+ "sample_ids": [
42
+ "0002628",
43
+ "0003242",
44
+ "0001568",
45
+ "0001119",
46
+ "0003648",
47
+ "0000187",
48
+ "0001891",
49
+ "0001689",
50
+ "0003372",
51
+ "0002549",
52
+ "0000915",
53
+ "0003633",
54
+ "0004424",
55
+ "0001627",
56
+ "0001562",
57
+ "0002018",
58
+ "0000254",
59
+ "0002719",
60
+ "0002730",
61
+ "0004413",
62
+ "0000891",
63
+ "0000150",
64
+ "0003223",
65
+ "0004731",
66
+ "0001587",
67
+ "0000483",
68
+ "0001116",
69
+ "0003376",
70
+ "0001225",
71
+ "0003082",
72
+ "0004447",
73
+ "0000283",
74
+ "0000563",
75
+ "0001254",
76
+ "0004759",
77
+ "0001868",
78
+ "0001860",
79
+ "0004087",
80
+ "0002962",
81
+ "0001802",
82
+ "0002973",
83
+ "0002648",
84
+ "0004350",
85
+ "0004377",
86
+ "0004809",
87
+ "0003694",
88
+ "0001572",
89
+ "0001550",
90
+ "0000213",
91
+ "0002266",
92
+ "0002445",
93
+ "0004513",
94
+ "0002124",
95
+ "0004391",
96
+ "0003679",
97
+ "0002902",
98
+ "0002508",
99
+ "0004767",
100
+ "0003840",
101
+ "0003498",
102
+ "0001579",
103
+ "0001748",
104
+ "0004706",
105
+ "0004387"
106
+ ],
107
+ "batch_size": 1,
108
+ "latency_ms": [
109
+ 17.35539303626865,
110
+ 13.579263002611697,
111
+ 15.319477999582887,
112
+ 14.879688038490713,
113
+ 24.25052202306688,
114
+ 20.667891018092632,
115
+ 20.015343907289207,
116
+ 19.221558002755046,
117
+ 20.937222987413406,
118
+ 26.26270893961191,
119
+ 17.899635946378112,
120
+ 16.57471200451255,
121
+ 18.844021018594503,
122
+ 21.3212959934026,
123
+ 19.545868039131165,
124
+ 64.57463896367699,
125
+ 20.962884067557752,
126
+ 24.004184058867395,
127
+ 20.19065897911787,
128
+ 18.50342599209398,
129
+ 19.576229038648307,
130
+ 25.253725005313754,
131
+ 21.430211956612766,
132
+ 25.169604923576117,
133
+ 17.8982320940122,
134
+ 21.415626979433,
135
+ 22.9903319850564,
136
+ 229.59673800505698,
137
+ 19.89229000173509,
138
+ 18.98771501146257,
139
+ 18.157949903979897,
140
+ 22.423187969252467,
141
+ 19.864327972754836,
142
+ 21.502194926142693,
143
+ 18.544791964814067,
144
+ 50.5554600385949,
145
+ 20.046600024215877,
146
+ 16.299219918437302,
147
+ 18.668118049390614,
148
+ 24.017272982746363,
149
+ 24.73958197515458,
150
+ 18.68593308608979,
151
+ 20.250934991054237,
152
+ 23.676551994867623,
153
+ 23.003229987807572,
154
+ 24.6098650386557,
155
+ 21.308812079951167,
156
+ 17.95686304103583,
157
+ 18.23423698078841,
158
+ 14.343310962431133,
159
+ 25.8766250917688,
160
+ 18.20633898023516,
161
+ 30.77617404051125,
162
+ 19.3530690157786,
163
+ 15.836970997042954,
164
+ 19.045730936340988,
165
+ 16.162591986358166,
166
+ 25.458221905864775,
167
+ 18.277363968081772,
168
+ 52.543785073794425,
169
+ 22.9536690749228,
170
+ 17.81571004539728,
171
+ 18.550690030679107,
172
+ 21.854626946151257
173
+ ],
174
+ "mean_latency_ms_per_input": 25.261267359383055,
175
+ "head_mean_latency_ms": 0.486134999897331,
176
+ "gflops_per_input": [
177
+ 3.183754752,
178
+ 3.183754752,
179
+ 3.183754752,
180
+ 3.183754752,
181
+ 3.183754752,
182
+ 3.183754752,
183
+ 3.183754752,
184
+ 3.183754752
185
+ ],
186
+ "mean_gflops_per_input": 3.183754752,
187
+ "operators": [
188
+ {
189
+ "aten.convolution": 228802560,
190
+ "aten.bmm": 679257600,
191
+ "aten._scaled_dot_product_flash_attention": 236027904,
192
+ "aten.addmm": 2039666688
193
+ },
194
+ {
195
+ "aten.convolution": 228802560,
196
+ "aten.bmm": 679257600,
197
+ "aten._scaled_dot_product_flash_attention": 236027904,
198
+ "aten.addmm": 2039666688
199
+ },
200
+ {
201
+ "aten.convolution": 228802560,
202
+ "aten.bmm": 679257600,
203
+ "aten._scaled_dot_product_flash_attention": 236027904,
204
+ "aten.addmm": 2039666688
205
+ },
206
+ {
207
+ "aten.convolution": 228802560,
208
+ "aten.bmm": 679257600,
209
+ "aten._scaled_dot_product_flash_attention": 236027904,
210
+ "aten.addmm": 2039666688
211
+ },
212
+ {
213
+ "aten.convolution": 228802560,
214
+ "aten.bmm": 679257600,
215
+ "aten._scaled_dot_product_flash_attention": 236027904,
216
+ "aten.addmm": 2039666688
217
+ },
218
+ {
219
+ "aten.convolution": 228802560,
220
+ "aten.bmm": 679257600,
221
+ "aten._scaled_dot_product_flash_attention": 236027904,
222
+ "aten.addmm": 2039666688
223
+ },
224
+ {
225
+ "aten.convolution": 228802560,
226
+ "aten.bmm": 679257600,
227
+ "aten._scaled_dot_product_flash_attention": 236027904,
228
+ "aten.addmm": 2039666688
229
+ },
230
+ {
231
+ "aten.convolution": 228802560,
232
+ "aten.bmm": 679257600,
233
+ "aten._scaled_dot_product_flash_attention": 236027904,
234
+ "aten.addmm": 2039666688
235
+ }
236
+ ],
237
+ "scope": "image loading, bilinear resize, tokenize when used, two image slots (including gray placeholder), active distilled encoders, feature assembly and original routing head; cached specialist features; supported FLOPs only"
238
+ }
paper_results/data/main_table_cost_qwen_20260912/components/yolo26x_specialist.json ADDED
@@ -0,0 +1,254 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "component": "specialist",
3
+ "expert": "yolo26x",
4
+ "task": "construction",
5
+ "device": "cuda:0",
6
+ "checkpoint": "/mnt/HithinkOmniSSD/user_workspace/caisihang/research/ICASSP_ModelCollaboration/models/construction/yolo26x.pt",
7
+ "sample_count": 64,
8
+ "batch_size": 1,
9
+ "sample_ids": [
10
+ "0000944",
11
+ "0003511",
12
+ "0003365",
13
+ "0001351",
14
+ "0002149",
15
+ "0003368",
16
+ "0000176",
17
+ "0004487",
18
+ "0000013",
19
+ "0003583",
20
+ "0000758",
21
+ "0001291",
22
+ "0001451",
23
+ "0002755",
24
+ "0003088",
25
+ "0004538",
26
+ "0004324",
27
+ "0000516",
28
+ "0003893",
29
+ "0003475",
30
+ "0002137",
31
+ "0003272",
32
+ "0001711",
33
+ "0003262",
34
+ "0001470",
35
+ "0002145",
36
+ "0002662",
37
+ "0000756",
38
+ "0001631",
39
+ "0001924",
40
+ "0002496",
41
+ "0001443",
42
+ "0004053",
43
+ "0002168",
44
+ "0001881",
45
+ "0001666",
46
+ "0000304",
47
+ "0001252",
48
+ "0004844",
49
+ "0000949",
50
+ "0003430",
51
+ "0003807",
52
+ "0004199",
53
+ "0002538",
54
+ "0002091",
55
+ "0001569",
56
+ "0003048",
57
+ "0002854",
58
+ "0000380",
59
+ "0001234",
60
+ "0001484",
61
+ "0003180",
62
+ "0000824",
63
+ "0003784",
64
+ "0000894",
65
+ "0004168",
66
+ "0000564",
67
+ "0004296",
68
+ "0003054",
69
+ "0001583",
70
+ "0003386",
71
+ "0000205",
72
+ "0004214",
73
+ "0003625"
74
+ ],
75
+ "model_load_seconds_excluded_from_latency": true,
76
+ "forward_latency_ms_per_sample": [
77
+ 41.741790771484375,
78
+ 51.50377655029297,
79
+ 78.82457733154297,
80
+ 87.29910278320312,
81
+ 49.47385787963867,
82
+ 49.40816116333008,
83
+ 168.38259887695312,
84
+ 42.57542419433594,
85
+ 40.342529296875,
86
+ 54.95475387573242,
87
+ 48.86323165893555,
88
+ 40.684993743896484,
89
+ 282.5484619140625,
90
+ 47.938270568847656,
91
+ 45.019649505615234,
92
+ 268.04327392578125,
93
+ 51.39228820800781,
94
+ 41.36457443237305,
95
+ 54.01919937133789,
96
+ 43.2599983215332,
97
+ 47.13923263549805,
98
+ 41.1484489440918,
99
+ 35.99542236328125,
100
+ 80.46995544433594,
101
+ 83.43251037597656,
102
+ 48.88550567626953,
103
+ 171.4989776611328,
104
+ 58.917728424072266,
105
+ 46.90486526489258,
106
+ 56.416385650634766,
107
+ 44.49577713012695,
108
+ 49.42009735107422,
109
+ 43.6354866027832,
110
+ 55.22761535644531,
111
+ 48.02000045776367,
112
+ 39.44412612915039,
113
+ 30.26905632019043,
114
+ 492.74456787109375,
115
+ 45.74156951904297,
116
+ 39.223392486572266,
117
+ 50.18822479248047,
118
+ 43.890113830566406,
119
+ 51.966495513916016,
120
+ 47.814815521240234,
121
+ 54.13961410522461,
122
+ 39.215328216552734,
123
+ 56.004703521728516,
124
+ 39.528831481933594,
125
+ 396.5307312011719,
126
+ 83.4832992553711,
127
+ 50.0503044128418,
128
+ 153.1714630126953,
129
+ 61.07219314575195,
130
+ 52.57244873046875,
131
+ 42.75983810424805,
132
+ 38.4617919921875,
133
+ 44.08700942993164,
134
+ 53.36006546020508,
135
+ 45.519649505615234,
136
+ 290.8823547363281,
137
+ 48.10774230957031,
138
+ 53.34476852416992,
139
+ 51.016319274902344,
140
+ 42.9719352722168
141
+ ],
142
+ "end_to_end_latency_ms_per_sample": [
143
+ 35.17372906208038,
144
+ 48.07086195796728,
145
+ 46.07532289810479,
146
+ 45.34715902991593,
147
+ 46.541396994143724,
148
+ 45.594630064442754,
149
+ 45.38595397025347,
150
+ 39.57494895439595,
151
+ 39.812199072912335,
152
+ 51.36631103232503,
153
+ 42.81833197455853,
154
+ 33.49219704978168,
155
+ 37.3103239107877,
156
+ 38.96563302259892,
157
+ 40.327758993953466,
158
+ 44.32409000582993,
159
+ 43.25311293359846,
160
+ 33.99710892699659,
161
+ 49.12354901898652,
162
+ 45.043769059702754,
163
+ 47.554407035931945,
164
+ 37.903546006418765,
165
+ 32.56980108562857,
166
+ 35.192839917726815,
167
+ 39.34567596297711,
168
+ 46.67255701497197,
169
+ 49.70169102307409,
170
+ 52.48237494379282,
171
+ 43.415413005277514,
172
+ 48.36420400533825,
173
+ 48.96194895263761,
174
+ 51.6092429170385,
175
+ 37.311508087441325,
176
+ 50.858690054155886,
177
+ 43.94579504150897,
178
+ 36.09434806276113,
179
+ 26.66287892498076,
180
+ 41.9749760767445,
181
+ 34.900374012067914,
182
+ 39.04577391222119,
183
+ 51.60050210542977,
184
+ 46.00996105000377,
185
+ 53.40714694466442,
186
+ 51.78673507180065,
187
+ 54.96632098220289,
188
+ 46.47863702848554,
189
+ 47.48446703888476,
190
+ 36.593522992916405,
191
+ 37.32479899190366,
192
+ 26.43477905075997,
193
+ 39.97986298054457,
194
+ 55.87899999227375,
195
+ 40.821335976943374,
196
+ 45.88699492160231,
197
+ 40.707007981836796,
198
+ 34.627774031832814,
199
+ 43.4188200160861,
200
+ 49.56604307517409,
201
+ 39.82124803587794,
202
+ 249.81214595027268,
203
+ 38.2277499884367,
204
+ 48.29735495150089,
205
+ 47.6501100929454,
206
+ 40.76927900314331
207
+ ],
208
+ "mean_forward_latency_ms_per_sample": 78.70017620921135,
209
+ "median_forward_latency_ms_per_sample": 49.41412925720215,
210
+ "mean_end_to_end_latency_ms_per_sample": 46.30809425361804,
211
+ "median_end_to_end_latency_ms_per_sample": 43.68230752879754,
212
+ "flop_profiles": [
213
+ {
214
+ "batch_size": 1,
215
+ "flops_counted": 116518072320,
216
+ "gflops_counted_per_sample": 116.51807232,
217
+ "operators": {
218
+ "aten.convolution": 116319006720,
219
+ "aten.bmm": 199065600
220
+ }
221
+ },
222
+ {
223
+ "batch_size": 1,
224
+ "flops_counted": 145709798400,
225
+ "gflops_counted_per_sample": 145.7097984,
226
+ "operators": {
227
+ "aten.convolution": 145398758400,
228
+ "aten.bmm": 311040000
229
+ }
230
+ },
231
+ {
232
+ "batch_size": 1,
233
+ "flops_counted": 145709798400,
234
+ "gflops_counted_per_sample": 145.7097984,
235
+ "operators": {
236
+ "aten.convolution": 145398758400,
237
+ "aten.bmm": 311040000
238
+ }
239
+ },
240
+ {
241
+ "batch_size": 1,
242
+ "flops_counted": 135976458240,
243
+ "gflops_counted_per_sample": 135.97645824,
244
+ "operators": {
245
+ "aten.convolution": 135705507840,
246
+ "aten.bmm": 270950400
247
+ }
248
+ }
249
+ ],
250
+ "mean_counted_gflops_per_sample": 135.97853184000002,
251
+ "flops_scope": "GPU FlopCounterMode around the real Ultralytics predict path; multiply-add counted as 2 FLOPs. Elementwise, postprocessing and unsupported custom ops may be missing.",
252
+ "latency_scope": "A800 wall-clock latency for real Ultralytics prediction including image loading/preprocess, forward and result postprocessing, excluding model load; batch processing uses the recorded batch size.",
253
+ "completed": true
254
+ }
paper_results/data/main_table_cost_qwen_20260912/cost_summary.json ADDED
@@ -0,0 +1,75 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "components": {
3
+ "grefcoco|instancevg|Qwen3.8": {
4
+ "specialist_gflops": 7.502680064,
5
+ "router_gflops": 3.642171392,
6
+ "large_gflops": 32554.061985335,
7
+ "specialist_ms": 46.08042127620138,
8
+ "router_ms": 17.032772149832454,
9
+ "large_ms": 4071.4221567668574,
10
+ "large_samples": 64,
11
+ "flops_probes": 8,
12
+ "specialist_samples": 64,
13
+ "router_samples": 64,
14
+ "batch_size": 1
15
+ },
16
+ "construction|yolo26x|Qwen3.8": {
17
+ "specialist_gflops": 135.97853184000002,
18
+ "router_gflops": 3.183754752,
19
+ "large_gflops": 96874.712551641,
20
+ "specialist_ms": 46.30809425361804,
21
+ "router_ms": 25.261267359383055,
22
+ "large_ms": 9822.475872695577,
23
+ "large_samples": 64,
24
+ "flops_probes": 8,
25
+ "specialist_samples": 64,
26
+ "router_samples": 64,
27
+ "batch_size": 1
28
+ }
29
+ },
30
+ "selected_points": [
31
+ {
32
+ "pair": "grefcoco|instancevg|Qwen3.8",
33
+ "method": "learned",
34
+ "nominal_validation_budget": 0.56,
35
+ "actual_test_call_fraction": 0.5355755324277571,
36
+ "test_metric": 0.7528287440093016,
37
+ "estimated_gflops_per_input": 17446.303931938,
38
+ "estimated_time_ms_per_input": 2243.6672827746106
39
+ },
40
+ {
41
+ "pair": "grefcoco|instancevg|Qwen3.8",
42
+ "method": "confidence",
43
+ "nominal_validation_budget": 0.71,
44
+ "actual_test_call_fraction": 0.6838612710206166,
45
+ "test_metric": 0.7243286163967898,
46
+ "estimated_gflops_per_input": 22269.96488623913,
47
+ "estimated_time_ms_per_input": 2830.3683522642846
48
+ },
49
+ {
50
+ "pair": "construction|yolo26x|Qwen3.8",
51
+ "method": "learned",
52
+ "nominal_validation_budget": 0.5,
53
+ "actual_test_call_fraction": 0.4876830892143808,
54
+ "test_metric": 0.29875800068449215,
55
+ "estimated_gflops_per_input": 47383.321370531434,
56
+ "estimated_time_ms_per_input": 4861.824738942901
57
+ },
58
+ {
59
+ "pair": "construction|yolo26x|Qwen3.8",
60
+ "method": "confidence",
61
+ "nominal_validation_budget": 0.93,
62
+ "actual_test_call_fraction": 0.9221038615179761,
63
+ "test_metric": 0.2597490004762105,
64
+ "estimated_gflops_per_input": 89464.5250591521,
65
+ "estimated_time_ms_per_input": 9103.651026133362
66
+ }
67
+ ],
68
+ "protocol": {
69
+ "method": "original main-table logit contrast; original checkpoints and validation thresholds; seed 42",
70
+ "large_timing": "64 measured examples; batch size one; same full task prompt; two separate warmup examples",
71
+ "cost": "serial component estimates, not a timed integrated service; includes active distilled encoders and routing head only for learned",
72
+ "flops": "multiply-add=2; supported operators; mean prefill (including visual embedding) plus average generated length times one-token decode probe",
73
+ "scope_limits": "mean call cost extrapolated across policies; no policy-specific input/output-length costs; specialist-feature extraction overhead is not separately measured; confidence threshold overhead not separately timed"
74
+ }
75
+ }
paper_results/data/main_table_cost_qwen_20260912/fixed_budget_cost_performance.csv ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ pair,method,nominal_validation_budget,actual_test_call_fraction,test_metric,estimated_gflops_per_input,estimated_time_ms_per_input
2
+ grefcoco|instancevg|Qwen3.8,learned,0.0,0.0,0.6695119530385957,11.144851456,63.11319342603383
3
+ grefcoco|instancevg|Qwen3.8,learned,0.05,0.11998411933187761,0.7164449990074583,3917.1153094417755,551.619195333999
4
+ grefcoco|instancevg|Qwen3.8,learned,0.1,0.16822164875365114,0.7290077418257097,5487.442832257611,748.0145414095009
5
+ grefcoco|instancevg|Qwen3.8,learned,0.25,0.30176105266142983,0.748092901908516,9834.692864556126,1291.7098292810697
6
+ grefcoco|instancevg|Qwen3.8,learned,0.5,0.487451436349715,0.7535093440716899,15879.6691252252,2047.733771728093
7
+ grefcoco|instancevg|Qwen3.8,learned,0.75,0.6870657629810283,0.7512973938689278,22377.92628754188,2860.4479639829183
8
+ grefcoco|instancevg|Qwen3.8,learned,1.0,1.0,0.7447182599325072,32565.206836791,4134.535350192891
9
+ grefcoco|instancevg|Qwen3.8,confidence,0.0,0.0,0.6695119530385957,7.502680064,46.08042127620138
10
+ grefcoco|instancevg|Qwen3.8,confidence,0.05,0.012534384482318578,0.6666194027734452,415.54780944942013,97.11319197894791
11
+ grefcoco|instancevg|Qwen3.8,confidence,0.1,0.0345971698380739,0.6647761109378102,1133.7810914898203,186.94010511636148
12
+ grefcoco|instancevg|Qwen3.8,confidence,0.25,0.12477667810452883,0.6594731021183677,4069.490393403024,554.0989531587461
13
+ grefcoco|instancevg|Qwen3.8,confidence,0.5,0.2345234381646485,0.662762669086578,7642.1932230898465,1000.9243437008932
14
+ grefcoco|instancevg|Qwen3.8,confidence,0.75,0.7478093185491875,0.7311629753566061,24351.733589325377,3090.727849854088
15
+ grefcoco|instancevg|Qwen3.8,confidence,1.0,1.0,0.7447182599325072,32561.564665399,4117.502578043059
16
+ construction|yolo26x|Qwen3.8,learned,0.0,0.0,0.18129624718318377,139.16228659200002,71.56936161300109
17
+ construction|yolo26x|Qwen3.8,learned,0.05,0.045938748335552594,0.19531128485766117,4589.465326580834,522.8016087608005
18
+ construction|yolo26x|Qwen3.8,learned,0.1,0.09920106524633822,0.20211074868345033,9749.236967147599,1045.9694315408578
19
+ construction|yolo26x|Qwen3.8,learned,0.25,0.25432756324900135,0.2504846177578749,24777.071870298303,2569.695715387775
20
+ construction|yolo26x|Qwen3.8,learned,0.5,0.4876830892143808,0.29875800068449215,47383.321370531434,4861.824738942901
21
+ construction|yolo26x|Qwen3.8,learned,0.75,0.7709720372836218,0.28730364578560735,74826.85678379593,7644.4235963543315
22
+ construction|yolo26x|Qwen3.8,learned,1.0,1.0,0.27001009067886245,97013.87483823301,9894.045234308578
23
+ construction|yolo26x|Qwen3.8,confidence,0.0,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
24
+ construction|yolo26x|Qwen3.8,confidence,0.05,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
25
+ construction|yolo26x|Qwen3.8,confidence,0.1,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
26
+ construction|yolo26x|Qwen3.8,confidence,0.25,0.24800266311584554,0.18026802885144627,24161.165233228996,2482.3082690732604
27
+ construction|yolo26x|Qwen3.8,confidence,0.5,0.5013315579227696,0.18015785955296285,48702.32909867467,4970.625226170908
28
+ construction|yolo26x|Qwen3.8,confidence,0.75,0.7486684420772304,0.22103445955201576,72663.01865455658,7400.0858032057995
29
+ construction|yolo26x|Qwen3.8,confidence,1.0,1.0,0.27001009067886245,97010.691083481,9868.783966949195
paper_results/data/main_table_cost_qwen_20260912/full_cost_performance_curves.csv ADDED
@@ -0,0 +1,405 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ pair,method,nominal_validation_budget,actual_test_call_fraction,test_metric,estimated_gflops_per_input,estimated_time_ms_per_input
2
+ grefcoco|instancevg|Qwen3.8,learned,0.0,0.0,0.6695119530385957,11.144851456,63.11319342603383
3
+ grefcoco|instancevg|Qwen3.8,learned,0.01,0.04701812097666109,0.6878314380512152,1541.775676164205,254.54381293995635
4
+ grefcoco|instancevg|Qwen3.8,learned,0.02,0.06706746448118424,0.6980404389870403,2194.4632473757256,336.17315431290154
5
+ grefcoco|instancevg|Qwen3.8,learned,0.03,0.08720188299350594,0.7065195814309616,2849.9203556645225,418.1488719575849
6
+ grefcoco|instancevg|Qwen3.8,learned,0.04,0.1055780846779911,0.712389756969061,3448.14036435627,492.9661466530142
7
+ grefcoco|instancevg|Qwen3.8,learned,0.05,0.11998411933187761,0.7164449990074583,3917.1153094417755,551.619195333999
8
+ grefcoco|instancevg|Qwen3.8,learned,0.06,0.13243342880639764,0.7199047159912656,4322.390901749918,602.3055897650073
9
+ grefcoco|instancevg|Qwen3.8,learned,0.07,0.14235884638289426,0.7222017412018262,4645.503560765523,642.7161548011189
10
+ grefcoco|instancevg|Qwen3.8,learned,0.08,0.1511215721861441,0.7239032413577972,4930.765879725012,678.392910790143
11
+ grefcoco|instancevg|Qwen3.8,learned,0.09,0.15957235629413266,0.7263136999120892,5205.873229401156,712.7996204494608
12
+ grefcoco|instancevg|Qwen3.8,learned,0.1,0.16822164875365114,0.7290077418257097,5487.442832257611,748.0145414095009
13
+ grefcoco|instancevg|Qwen3.8,learned,0.11,0.17695601622096815,0.7316450670674645,5771.781972191343,783.575838641279
14
+ grefcoco|instancevg|Qwen3.8,learned,0.12,0.18855457561750277,0.7345943340044806,6149.362193726721,830.7984703549065
15
+ grefcoco|instancevg|Qwen3.8,learned,0.13,0.19751580977228256,0.7366361341916456,6441.086765866523,867.2834376446528
16
+ grefcoco|instancevg|Qwen3.8,learned,0.14,0.20664719394265946,0.7385361426991465,6738.350412160879,904.4611574778754
17
+ grefcoco|instancevg|Qwen3.8,learned,0.15,0.2172532115815444,0.7402376428551173,7083.619367794696,947.6427326878917
18
+ grefcoco|instancevg|Qwen3.8,learned,0.16,0.22717862915804102,0.7420242180188866,7406.7320268103,988.0532977240033
19
+ grefcoco|instancevg|Qwen3.8,learned,0.17,0.2361398633128208,0.7436122848311261,7698.4565989501025,1024.5382650137497
20
+ grefcoco|instancevg|Qwen3.8,learned,0.18,0.24362646399909255,0.7448033349403057,7942.175861750445,1055.0193769267023
21
+ grefcoco|instancevg|Qwen3.8,learned,0.19,0.2507727646541701,0.7456540850182911,8174.816976241679,1084.1149837527025
22
+ grefcoco|instancevg|Qwen3.8,learned,0.2,0.25942205711368854,0.7459943850494853,8456.386579098133,1119.3299047127425
23
+ grefcoco|instancevg|Qwen3.8,learned,0.21,0.2690922496667896,0.746419760088478,8771.19062688191,1158.7013409336398
24
+ grefcoco|instancevg|Qwen3.8,learned,0.22,0.2781385588293679,0.7467600601196722,9065.684736098987,1195.5326844951244
25
+ grefcoco|instancevg|Qwen3.8,learned,0.23,0.2868445679607521,0.7473555851742619,9349.100697006961,1230.9785229696565
26
+ grefcoco|instancevg|Qwen3.8,learned,0.24,0.29430281031109096,0.7480645435725831,9591.896780781542,1261.344176125363
27
+ grefcoco|instancevg|Qwen3.8,learned,0.25,0.30176105266142983,0.748092901908516,9834.692864556126,1291.7098292810697
28
+ grefcoco|instancevg|Qwen3.8,learned,0.26,0.30995661174602274,0.7486033519553073,10101.491603000439,1325.0774101251732
29
+ grefcoco|instancevg|Qwen3.8,learned,0.27,0.3175850041119587,0.7494541020332927,10349.826760929574,1356.135815824356
30
+ grefcoco|instancevg|Qwen3.8,learned,0.28,0.3262342965714772,0.7501630604316138,10631.396363786029,1391.350736784396
31
+ grefcoco|instancevg|Qwen3.8,learned,0.29,0.3324731304767036,0.7505600771346738,10834.49574945298,1416.7516633785233
32
+ grefcoco|instancevg|Qwen3.8,learned,0.3,0.3399313728270425,0.7509570938377336,11077.291833227562,1447.11731653423
33
+ grefcoco|instancevg|Qwen3.8,learned,0.31,0.34824036525536683,0.7511839605251964,11347.783287774913,1480.9467324073175
34
+ grefcoco|instancevg|Qwen3.8,learned,0.32,0.3557553242775714,0.751269035532995,11592.425729601013,1511.5433030775164
35
+ grefcoco|instancevg|Qwen3.8,learned,0.33,0.3627598332529847,0.7511272438533307,11820.450948963453,1540.0616161172863
36
+ grefcoco|instancevg|Qwen3.8,learned,0.34,0.37013300059552506,0.7512123188611293,12060.47749566076,1570.080893001255
37
+ grefcoco|instancevg|Qwen3.8,learned,0.35,0.3783569180160508,0.751439185548592,12328.199413130831,1603.5639326026044
38
+ grefcoco|instancevg|Qwen3.8,learned,0.36,0.3854748603351955,0.7517511272438533,12559.917348596306,1632.5440806713586
39
+ grefcoco|instancevg|Qwen3.8,learned,0.37,0.39279131100587017,0.752176502282846,12798.097537242094,1662.3324400408353
40
+ grefcoco|instancevg|Qwen3.8,learned,0.38,0.40013612001247767,0.7522332189547117,13037.200904913643,1692.2362581675577
41
+ grefcoco|instancevg|Qwen3.8,learned,0.39,0.406573462269234,0.7525735189859059,13246.762543760904,1718.445396062407
42
+ grefcoco|instancevg|Qwen3.8,learned,0.4,0.4153929047443496,0.7528003856733687,13533.871220771913,1754.3530695659233
43
+ grefcoco|instancevg|Qwen3.8,learned,0.41,0.42151830530584466,0.75311232736863,13733.27789033583,1779.2921611310667
44
+ grefcoco|instancevg|Qwen3.8,learned,0.42,0.42920341434364634,0.7535093440716899,13983.459406316484,1810.5814843447415
45
+ grefcoco|instancevg|Qwen3.8,learned,0.43,0.4367750900377166,0.7537362107591526,14229.948206194102,1841.4089725294323
46
+ grefcoco|instancevg|Qwen3.8,learned,0.44,0.4450273657941752,0.753679494087287,14498.593302689931,1875.0074708880277
47
+ grefcoco|instancevg|Qwen3.8,learned,0.45,0.45143634971499874,0.7537929274310183,14707.231762511437,1901.1011500256313
48
+ grefcoco|instancevg|Qwen3.8,learned,0.46,0.4578736919717551,0.7536227774154213,14916.7934013587,1927.3102879204807
49
+ grefcoco|instancevg|Qwen3.8,learned,0.47,0.4659274593766838,0.7537929274310183,15178.976244674219,1960.100574978354
50
+ grefcoco|instancevg|Qwen3.8,learned,0.48,0.4741797351331424,0.7538780024388169,15447.621341170048,1993.6990733369496
51
+ grefcoco|instancevg|Qwen3.8,learned,0.49,0.4806454357258316,0.7537362107591526,15658.106159043073,2020.0236699890452
52
+ grefcoco|instancevg|Qwen3.8,learned,0.5,0.487451436349715,0.7535093440716899,15879.6691252252,2047.733771728093
53
+ grefcoco|instancevg|Qwen3.8,learned,0.51,0.49482460369225534,0.7537645690950855,16119.695671922505,2077.7530486120613
54
+ grefcoco|instancevg|Qwen3.8,learned,0.52,0.501573887644273,0.7535660607435556,16339.412280053117,2105.232232836617
55
+ grefcoco|instancevg|Qwen3.8,learned,0.53,0.5105351217990528,0.7532257607123614,16631.136852192918,2141.7172001263634
56
+ grefcoco|instancevg|Qwen3.8,learned,0.54,0.519241130930437,0.7530556106967643,16914.552813100894,2177.163038600896
57
+ grefcoco|instancevg|Qwen3.8,learned,0.55,0.5268128066245072,0.7528571023452344,17161.041612978508,2207.9905267855866
58
+ grefcoco|instancevg|Qwen3.8,learned,0.56,0.5355755324277571,0.7528287440093016,17446.303931938,2243.6672827746106
59
+ grefcoco|instancevg|Qwen3.8,learned,0.57,0.5419561580126478,0.753282477384227,17654.019212733747,2269.645503154968
60
+ grefcoco|instancevg|Qwen3.8,learned,0.58,0.5495845503785838,0.7528287440093016,17902.35437066288,2300.703908854151
61
+ grefcoco|instancevg|Qwen3.8,learned,0.59,0.5562771176587358,0.7527153106655702,18120.224620741974,2327.952175564215
62
+ grefcoco|instancevg|Qwen3.8,learned,0.6,0.564047301704336,0.7526585939937045,18373.175673799906,2359.587875049628
63
+ grefcoco|instancevg|Qwen3.8,learned,0.61,0.5725831608201231,0.7528287440093016,18651.052560553326,2394.3409609806836
64
+ grefcoco|instancevg|Qwen3.8,learned,0.62,0.5818846950060971,0.7528571023452344,18953.855281002234,2432.2114333573827
65
+ grefcoco|instancevg|Qwen3.8,learned,0.63,0.5903638374500184,0.7528571023452344,19229.885809704134,2466.7336017739462
66
+ grefcoco|instancevg|Qwen3.8,learned,0.64,0.5973683464254318,0.7526869523296373,19457.911029066578,2495.2519148137167
67
+ grefcoco|instancevg|Qwen3.8,learned,0.65,0.6042594220571137,0.7526018773218388,19682.24353232598,2523.3083928245023
68
+ grefcoco|instancevg|Qwen3.8,learned,0.66,0.6122848311261095,0.752545160649973,19943.503196615744,2555.9832211251296
69
+ grefcoco|instancevg|Qwen3.8,learned,0.67,0.6198848651561126,0.7522048606187789,20190.91517551912,2586.9261680670666
70
+ grefcoco|instancevg|Qwen3.8,learned,0.68,0.6280520659047727,0.752176502282846,20456.790734937673,2620.178290153924
71
+ grefcoco|instancevg|Qwen3.8,learned,0.69,0.6363894166690298,0.7522048606187789,20728.20536851078,2654.1231647842574
72
+ grefcoco|instancevg|Qwen3.8,learned,0.7,0.6444999007458242,0.7521197856109804,20992.234569877815,2687.144369356623
73
+ grefcoco|instancevg|Qwen3.8,learned,0.71,0.6527521765022829,0.7519212772594505,21260.87966637365,2720.742867715219
74
+ grefcoco|instancevg|Qwen3.8,learned,0.72,0.6601253438448231,0.7518362022516518,21500.906213070954,2750.7621445991867
75
+ grefcoco|instancevg|Qwen3.8,learned,0.73,0.6697955363979242,0.7518362022516518,21815.71026085473,2790.1335808200847
76
+ grefcoco|instancevg|Qwen3.8,learned,0.74,0.6775373621075915,0.7518645605875848,22067.7381348869,2821.6538215482515
77
+ grefcoco|instancevg|Qwen3.8,learned,0.75,0.6870657629810283,0.7512973938689278,22377.92628754188,2860.4479639829183
78
+ grefcoco|instancevg|Qwen3.8,learned,0.76,0.6956299804327482,0.7514108272126592,22656.726353321057,2895.31650867122
79
+ grefcoco|instancevg|Qwen3.8,learned,0.77,0.7044494229078637,0.7514959022204577,22943.835030332066,2931.224182174736
80
+ grefcoco|instancevg|Qwen3.8,learned,0.78,0.7129852820236509,0.7510705271814649,23221.711917085486,2965.9772681057925
81
+ grefcoco|instancevg|Qwen3.8,learned,0.79,0.7243853330686555,0.7506735104784051,23592.82988544055,3012.3916885186977
82
+ grefcoco|instancevg|Qwen3.8,learned,0.8,0.7337152255905623,0.7507302271502708,23896.555784915217,3050.3776196526424
83
+ grefcoco|instancevg|Qwen3.8,learned,0.81,0.744377959901313,0.7505033604628081,24243.671098600553,3093.7901123771508
84
+ grefcoco|instancevg|Qwen3.8,learned,0.82,0.7522048606187789,0.7503048521112781,24498.46850971,3125.656729377056
85
+ grefcoco|instancevg|Qwen3.8,learned,0.83,0.7607123613986331,0.7503332104472109,24775.42221743766,3160.2943565508654
86
+ grefcoco|instancevg|Qwen3.8,learned,0.84,0.770779570654794,0.749964552080084,25103.15077158206,3201.2822153732072
87
+ grefcoco|instancevg|Qwen3.8,learned,0.85,0.7827467884184556,0.749964552080084,25492.7323204523,3250.005810931033
88
+ grefcoco|instancevg|Qwen3.8,learned,0.86,0.793466239401072,0.7498511187363526,25841.693992189157,3293.649221170034
89
+ grefcoco|instancevg|Qwen3.8,learned,0.87,0.8059439072115249,0.7499361937441511,26247.89276352306,3344.451074358288
90
+ grefcoco|instancevg|Qwen3.8,learned,0.88,0.8164932081785441,0.7497093270566884,26591.315361105357,3387.4017320538123
91
+ grefcoco|instancevg|Qwen3.8,learned,0.89,0.8270425091455633,0.7494257436973598,26934.737958687656,3430.3523897493365
92
+ grefcoco|instancevg|Qwen3.8,learned,0.9,0.8367410600345971,0.749397385361427,27250.46518549719,3469.83928472748
93
+ grefcoco|instancevg|Qwen3.8,learned,0.91,0.8491903695091172,0.74883021864277,27655.740777805335,3520.5256791584884
94
+ grefcoco|instancevg|Qwen3.8,learned,0.92,0.8618098289992343,0.7487167852990386,28066.55544426803,3571.904826132973
95
+ grefcoco|instancevg|Qwen3.8,learned,0.93,0.8745710801690156,0.7486884269631058,28481.98600585952,3623.8612668936876
96
+ grefcoco|instancevg|Qwen3.8,learned,0.94,0.8879845730652526,0.7486884269631058,28918.649685043467,3678.4732590710614
97
+ grefcoco|instancevg|Qwen3.8,learned,0.95,0.9000085075007799,0.7484332019397102,29310.07759196523,3727.4277721433796
98
+ grefcoco|instancevg|Qwen3.8,learned,0.96,0.9125996086549641,0.7477809602132547,29719.969079402166,3778.691460360618
99
+ grefcoco|instancevg|Qwen3.8,learned,0.97,0.9269489266369849,0.7476391685335905,30187.09766643615,3837.11359152711
100
+ grefcoco|instancevg|Qwen3.8,learned,0.98,0.9459773700479256,0.7472988685023962,30806.550792720354,3914.5864176391988
101
+ grefcoco|instancevg|Qwen3.8,learned,0.99,0.9658282052009188,0.7472421518305306,31452.77611075156,3995.4075477114216
102
+ grefcoco|instancevg|Qwen3.8,learned,1.0,1.0,0.7447182599325072,32565.206836791,4134.535350192891
103
+ grefcoco|instancevg|Qwen3.8,confidence,0.0,0.0,0.6695119530385957,7.502680064,46.08042127620138
104
+ grefcoco|instancevg|Qwen3.8,confidence,0.01,0.002467175226157729,0.6686895612965431,87.81925530502161,56.12533315660624
105
+ grefcoco|instancevg|Qwen3.8,confidence,0.02,0.004849275444516916,0.6680373195700876,165.3662934687666,65.823868765273
106
+ grefcoco|instancevg|Qwen3.8,confidence,0.03,0.007600034030003119,0.6675835861951621,254.9146589673769,77.02336821813819
107
+ grefcoco|instancevg|Qwen3.8,confidence,0.04,0.009811984232765221,0.6671014944843037,326.9226229765687,86.02915128332876
108
+ grefcoco|instancevg|Qwen3.8,confidence,0.05,0.012534384482318578,0.6666194027734452,415.54780944942013,97.11319197894791
109
+ grefcoco|instancevg|Qwen3.8,confidence,0.06,0.01607917647392451,0.6662223860703854,530.9451876692788,111.54553663470202
110
+ grefcoco|instancevg|Qwen3.8,confidence,0.07,0.019709043473328986,0.6658253693673255,649.112102966414,126.32425756219423
111
+ grefcoco|instancevg|Qwen3.8,confidence,0.08,0.02353741882426339,0.6655985026798628,773.7412714438613,141.91118979040866
112
+ grefcoco|instancevg|Qwen3.8,confidence,0.09,0.028868785979638715,0.6651164109690043,947.2989282865286,163.61743615266283
113
+ grefcoco|instancevg|Qwen3.8,confidence,0.1,0.0345971698380739,0.6647761109378102,1133.7810914898203,186.94010511636148
114
+ grefcoco|instancevg|Qwen3.8,confidence,0.11,0.03981510364971783,0.664435810906616,1303.6460322294522,208.18451644963153
115
+ grefcoco|instancevg|Qwen3.8,confidence,0.12,0.045855429203414344,0.664067152539489,1500.2831647160913,232.77723174303654
116
+ grefcoco|instancevg|Qwen3.8,confidence,0.13,0.05317187987408899,0.6637552108442276,1738.4633533618796,262.565591112513
117
+ grefcoco|instancevg|Qwen3.8,confidence,0.14,0.05960922213084536,0.6636134191645634,1948.0249922091427,288.77472900736245
118
+ grefcoco|instancevg|Qwen3.8,confidence,0.15,0.0675211978561098,0.6632164024615036,2205.591940395867,320.98772227900565
119
+ grefcoco|instancevg|Qwen3.8,confidence,0.16,0.07551824858917279,0.6626208774069138,2465.928425659868,353.54709182238696
120
+ grefcoco|instancevg|Qwen3.8,confidence,0.17,0.08212574086152624,0.6622522190397867,2681.0291386616846,380.44898226071257
121
+ grefcoco|instancevg|Qwen3.8,confidence,0.18,0.08856308311828262,0.6619969940163911,2890.590777508948,406.6581201555621
122
+ grefcoco|instancevg|Qwen3.8,confidence,0.19,0.09480191702350906,0.6615149023055327,3093.6901631758988,432.0590467496893
123
+ grefcoco|instancevg|Qwen3.8,confidence,0.2,0.10007656750701868,0.6612596772821371,3265.4014619670484,453.5343755974514
124
+ grefcoco|instancevg|Qwen3.8,confidence,0.21,0.10554972634205825,0.6608343022431443,3443.575013938511,475.81791574593575
125
+ grefcoco|instancevg|Qwen3.8,confidence,0.22,0.11005870175538099,0.6605507188838159,3590.3604790341706,494.17585814805494
126
+ grefcoco|instancevg|Qwen3.8,confidence,0.23,0.11527663556702493,0.6600969855088903,3760.2254197738025,515.420269481325
127
+ grefcoco|instancevg|Qwen3.8,confidence,0.24,0.12029606102713893,0.6598417604854947,3923.6281073331224,535.8564695138729
128
+ grefcoco|instancevg|Qwen3.8,confidence,0.25,0.12477667810452883,0.6594731021183677,4069.490393403024,554.0989531587461
129
+ grefcoco|instancevg|Qwen3.8,confidence,0.26,0.12914386183818732,0.6590193687434421,4211.659963369889,571.879601774635
130
+ grefcoco|instancevg|Qwen3.8,confidence,0.27,0.1322916371267334,0.658679068712248,4314.132835229124,584.6955238289447
131
+ grefcoco|instancevg|Qwen3.8,confidence,0.28,0.13626180415733205,0.6582820520091881,4443.377898835366,600.8597498433894
132
+ grefcoco|instancevg|Qwen3.8,confidence,0.29,0.14023197118793068,0.657743243626464,4572.622962441607,617.0239758578339
133
+ grefcoco|instancevg|Qwen3.8,confidence,0.3,0.14326631313274538,0.657374585259337,4671.403118197806,629.3780628831595
134
+ grefcoco|instancevg|Qwen3.8,confidence,0.31,0.14766185520233674,0.6569775685562771,4814.495867190431,647.2741702562946
135
+ grefcoco|instancevg|Qwen3.8,confidence,0.32,0.15066783881121856,0.6567507018688143,4912.352843920871,659.512798524374
136
+ grefcoco|instancevg|Qwen3.8,confidence,0.33,0.15452457249808582,0.6565238351813516,5037.905191424078,675.2151895098345
137
+ grefcoco|instancevg|Qwen3.8,confidence,0.34,0.15806936448969175,0.6564104018376201,5153.302569643936,689.6475341655887
138
+ grefcoco|instancevg|Qwen3.8,confidence,0.35,0.16201117318435754,0.6562686101579559,5281.6244542244185,705.6963014227872
139
+ grefcoco|instancevg|Qwen3.8,confidence,0.36,0.1657544735274934,0.6558999517908289,5403.484085624589,720.9368573792635
140
+ grefcoco|instancevg|Qwen3.8,confidence,0.37,0.17012165726115192,0.6558715934548961,5545.653655591455,738.7175059951526
141
+ grefcoco|instancevg|Qwen3.8,confidence,0.38,0.17536794940872868,0.6555596517596347,5716.441775356846,760.0773760856687
142
+ grefcoco|instancevg|Qwen3.8,confidence,0.39,0.17965005813458867,0.6553044267362391,5855.841808246436,777.5116484298197
143
+ grefcoco|instancevg|Qwen3.8,confidence,0.4,0.18401724186824717,0.6552193517284406,5998.011378213301,795.2922970457087
144
+ grefcoco|instancevg|Qwen3.8,confidence,0.41,0.18858293395343562,0.6552760684003063,6146.643201360479,813.88115696232
145
+ grefcoco|instancevg|Qwen3.8,confidence,0.42,0.19238295096843716,0.6554745767518362,6270.349190812167,829.3526304332884
146
+ grefcoco|instancevg|Qwen3.8,confidence,0.43,0.19689192638175992,0.655502935087769,6417.134655907828,847.7105728354077
147
+ grefcoco|instancevg|Qwen3.8,confidence,0.44,0.20182627683407536,0.6555880100955676,6577.76780638987,867.8003965962173
148
+ grefcoco|instancevg|Qwen3.8,confidence,0.45,0.20718600232538353,0.6559283101267618,6752.248642258297,889.6221017157176
149
+ grefcoco|instancevg|Qwen3.8,confidence,0.46,0.21189348609023623,0.656070101806426,6905.49636053427,908.7882554185591
150
+ grefcoco|instancevg|Qwen3.8,confidence,0.47,0.217139778237813,0.6567790602047472,7076.28448029966,930.148125509075
151
+ grefcoco|instancevg|Qwen3.8,confidence,0.48,0.22198905368232993,0.6584238436888523,7234.148093704427,949.8915729981468
152
+ grefcoco|instancevg|Qwen3.8,confidence,0.49,0.22763236253296656,0.6604372855400845,7417.860719830442,972.8678656901072
153
+ grefcoco|instancevg|Qwen3.8,confidence,0.5,0.2345234381646485,0.662762669086578,7642.1932230898465,1000.9243437008932
154
+ grefcoco|instancevg|Qwen3.8,confidence,0.51,0.24382497235062248,0.6654567110001985,7944.995943538756,1038.7948160775923
155
+ grefcoco|instancevg|Qwen3.8,confidence,0.52,0.252644414825738,0.6678955278904234,8232.104620549764,1074.7024895811082
156
+ grefcoco|instancevg|Qwen3.8,confidence,0.53,0.26401610753480986,0.6703627031165811,8602.299409879071,1121.0014512367675
157
+ grefcoco|instancevg|Qwen3.8,confidence,0.54,0.2760116836344043,0.6731134617020673,8992.804137775072,1169.8405055518392
158
+ grefcoco|instancevg|Qwen3.8,confidence,0.55,0.2893401015228426,0.6758358619516207,9426.69827988174,1224.1061214574747
159
+ grefcoco|instancevg|Qwen3.8,confidence,0.56,0.3036043444970649,0.679295578935428,9891.057329838453,1282.1818763522294
160
+ grefcoco|instancevg|Qwen3.8,confidence,0.57,0.3175850041119587,0.6822448458724442,10346.184589537575,1339.1030436745236
161
+ grefcoco|instancevg|Qwen3.8,confidence,0.58,0.33462836400760004,0.6853926211609903,10901.015184018655,1408.4937567793893
162
+ grefcoco|instancevg|Qwen3.8,confidence,0.59,0.35637920766809406,0.6889090548166633,11609.09349677571,1497.0506235870967
163
+ grefcoco|instancevg|Qwen3.8,confidence,0.6,0.37968975980489467,0.6922553384567394,12367.9466559495,1591.9577220433357
164
+ grefcoco|instancevg|Qwen3.8,confidence,0.61,0.4046450954258004,0.6949493803703599,13180.344198617304,1693.5614284198446
165
+ grefcoco|instancevg|Qwen3.8,confidence,0.62,0.43141536454640844,0.6983807390182344,14051.825198933677,1802.5544952600997
166
+ grefcoco|instancevg|Qwen3.8,confidence,0.63,0.4610214672603012,0.7015568726427134,15015.624101825935,1923.093437824958
167
+ grefcoco|instancevg|Qwen3.8,confidence,0.64,0.4916484700677764,0.7053285313217821,16012.657449645512,2047.7888956506733
168
+ grefcoco|instancevg|Qwen3.8,confidence,0.65,0.5209426310864078,0.7086180982899923,16966.301383254424,2167.0577918858253
169
+ grefcoco|instancevg|Qwen3.8,confidence,0.66,0.5529308340186598,0.71148229021921,18007.64732431043,2297.2952700593505
170
+ grefcoco|instancevg|Qwen3.8,confidence,0.67,0.5888041289737118,0.7155091739216742,19175.4687918954,2443.350597975582
171
+ grefcoco|instancevg|Qwen3.8,confidence,0.68,0.6175027649377535,0.7182882908430933,20109.725965963375,2560.1948603085675
172
+ grefcoco|instancevg|Qwen3.8,confidence,0.69,0.6421461588633979,0.7211241244363781,20911.968539347832,2660.52852035537
173
+ grefcoco|instancevg|Qwen3.8,confidence,0.7,0.665910444375124,0.7233077163032073,21685.59256293376,2757.282958927545
174
+ grefcoco|instancevg|Qwen3.8,confidence,0.71,0.6838612710206166,0.7243286163967898,22269.96488623913,2830.3683522642846
175
+ grefcoco|instancevg|Qwen3.8,confidence,0.72,0.7029464311034228,0.7260017582168279,22891.264370574845,2908.072095890864
176
+ grefcoco|instancevg|Qwen3.8,confidence,0.73,0.7180614241556306,0.7278450500524629,23383.318791304326,2969.611613503
177
+ grefcoco|instancevg|Qwen3.8,confidence,0.74,0.7352465757309361,0.7297450585599637,23942.7652809142,3039.579620394096
178
+ grefcoco|instancevg|Qwen3.8,confidence,0.75,0.7478093185491875,0.7311629753566061,24351.733589325377,3090.727849854088
179
+ grefcoco|instancevg|Qwen3.8,confidence,0.76,0.759804894648782,0.7328928338485098,24742.238317221378,3139.5669041691604
180
+ grefcoco|instancevg|Qwen3.8,confidence,0.77,0.7729064458497575,0.7342540339732865,25168.74702712198,3192.9088500168273
181
+ grefcoco|instancevg|Qwen3.8,confidence,0.78,0.7855259053398747,0.735274934066869,25579.561693584674,3244.287996991312
182
+ grefcoco|instancevg|Qwen3.8,confidence,0.79,0.7964722230099538,0.7362958341604514,25935.908797527598,3288.855077288281
183
+ grefcoco|instancevg|Qwen3.8,confidence,0.8,0.8082125740861527,0.7377704676289595,26318.104914191772,3336.6550027881385
184
+ grefcoco|instancevg|Qwen3.8,confidence,0.81,0.8197544168108215,0.738422709355415,26693.838777675628,3383.6467169872735
185
+ grefcoco|instancevg|Qwen3.8,confidence,0.82,0.8333380597226555,0.7389048010662734,27136.04153101413,3438.9514617081236
186
+ grefcoco|instancevg|Qwen3.8,confidence,0.83,0.8426112355726966,0.7398689844879902,27437.921072437275,3476.7064753275763
187
+ grefcoco|instancevg|Qwen3.8,confidence,0.84,0.8529903865241187,0.7403227178629158,27775.804595865022,3518.964380479624
188
+ grefcoco|instancevg|Qwen3.8,confidence,0.85,0.8635396874911381,0.7406913762300428,28119.227193447325,3561.9150381751488
189
+ grefcoco|instancevg|Qwen3.8,confidence,0.86,0.8710262881774098,0.7413152596205654,28362.946456247664,3592.3961500881014
190
+ grefcoco|instancevg|Qwen3.8,confidence,0.87,0.8809517057539064,0.7422227263704165,28686.05911526327,3632.8067151242126
191
+ grefcoco|instancevg|Qwen3.8,confidence,0.88,0.8913024983693957,0.7430167597765364,29023.019459665255,3674.9491615190145
192
+ grefcoco|instancevg|Qwen3.8,confidence,0.89,0.900915974250631,0.743442134815529,29335.977149397513,3714.0896802254197
193
+ grefcoco|instancevg|Qwen3.8,confidence,0.9,0.910954825170859,0.7443212432294473,29662.782524516155,3754.9620802905156
194
+ grefcoco|instancevg|Qwen3.8,confidence,0.91,0.921901142840938,0.7448033349403057,30019.129628459075,3799.529160587484
195
+ grefcoco|instancevg|Qwen3.8,confidence,0.92,0.9303235686129938,0.7450018432918356,30293.31379910946,3833.820411489556
196
+ grefcoco|instancevg|Qwen3.8,confidence,0.93,0.9393698777755721,0.7448884099481042,30587.80790832654,3870.6517550510403
197
+ grefcoco|instancevg|Qwen3.8,confidence,0.94,0.9490117119927403,0.7448884099481042,30901.688777084553,3909.907732514692
198
+ grefcoco|instancevg|Qwen3.8,confidence,0.95,0.9587953378895726,0.7449734849559028,31220.185540971364,3949.7410037645727
199
+ grefcoco|instancevg|Qwen3.8,confidence,0.96,0.9673595553412926,0.7452570683152313,31498.985606750543,3984.609548452875
200
+ grefcoco|instancevg|Qwen3.8,confidence,0.97,0.9746192893401016,0.745313784987097,31735.319437344813,4014.1669903078596
201
+ grefcoco|instancevg|Qwen3.8,confidence,0.98,0.9830984317840229,0.7450585599637013,32011.349966046713,4048.689158724423
202
+ grefcoco|instancevg|Qwen3.8,confidence,0.99,0.9908118991577575,0.7448884099481042,32262.45466105313,4080.0939406953444
203
+ grefcoco|instancevg|Qwen3.8,confidence,1.0,1.0,0.7447182599325072,32561.564665399,4117.502578043059
204
+ construction|yolo26x|Qwen3.8,learned,0.0,0.0,0.18129624718318377,139.16228659200002,71.56936161300109
205
+ construction|yolo26x|Qwen3.8,learned,0.01,0.013648468708388815,0.18693471712069915,1461.3537694872334,205.6311162003908
206
+ construction|yolo26x|Qwen3.8,learned,0.02,0.019640479360852198,0.18781313491575913,2041.8280790509946,264.48749626314725
207
+ construction|yolo26x|Qwen3.8,learned,0.03,0.031291611185086554,0.1913128749335621,3170.5281254249744,378.9304574962848
208
+ construction|yolo26x|Qwen3.8,learned,0.04,0.04061251664447404,0.19266008228682668,4073.4881625241583,470.48482648279486
209
+ construction|yolo26x|Qwen3.8,learned,0.05,0.045938748335552594,0.19531128485766117,4589.465326580834,522.8016087608005
210
+ construction|yolo26x|Qwen3.8,learned,0.06,0.0559254327563249,0.19800731707634547,5556.922509187103,620.8955755320613
211
+ construction|yolo26x|Qwen3.8,learned,0.07,0.07456724367509987,0.19763843353437874,7362.842583385471,804.0043135050814
212
+ construction|yolo26x|Qwen3.8,learned,0.08,0.08688415446071904,0.1974112776278713,8556.039775266534,924.9868725229696
213
+ construction|yolo26x|Qwen3.8,learned,0.09,0.09587217043941411,0.20030220967092682,9426.751239612175,1013.2714426171043
214
+ construction|yolo26x|Qwen3.8,learned,0.1,0.09920106524633822,0.20211074868345033,9749.236967147599,1045.9694315408578
215
+ construction|yolo26x|Qwen3.8,learned,0.11,0.11284953395472704,0.20834032699131655,11071.428450042833,1180.0311861282478
216
+ construction|yolo26x|Qwen3.8,learned,0.12,0.12350199733688415,0.21121674052765757,12103.382778156185,1284.664750684259
217
+ construction|yolo26x|Qwen3.8,learned,0.13,0.13315579227696406,0.21276215593410724,13038.591388008912,1379.4889185631446
218
+ construction|yolo26x|Qwen3.8,learned,0.14,0.14347536617842876,0.21511521078428622,14038.297143368722,1480.8526842267806
219
+ construction|yolo26x|Qwen3.8,learned,0.15,0.15246338215712382,0.21738179676765032,14909.008607714362,1569.1372543209152
220
+ construction|yolo26x|Qwen3.8,learned,0.16,0.1604527296937417,0.22260454951009195,15682.974353799378,1647.612427737924
221
+ construction|yolo26x|Qwen3.8,learned,0.17,0.1780958721704394,0.22627631882842034,17392.148709737117,1820.9117690338178
222
+ construction|yolo26x|Qwen3.8,learned,0.18,0.18442077230359522,0.2270967395967396,18004.871592054424,1883.0379479889498
223
+ construction|yolo26x|Qwen3.8,learned,0.19,0.2000665778961385,0.23397114982462924,19520.55451147091,2036.7184959305916
224
+ construction|yolo26x|Qwen3.8,learned,0.2,0.20439414114513982,0.23512276426997616,19939.78595726696,2079.2258815314713
225
+ construction|yolo26x|Qwen3.8,learned,0.21,0.21937416777629826,0.24202469545559713,21390.97173117636,2226.3668316883623
226
+ construction|yolo26x|Qwen3.8,learned,0.22,0.22769640479360853,0.24353835097564847,22197.18605001492,2308.1118039977464
227
+ construction|yolo26x|Qwen3.8,learned,0.23,0.23601864181091878,0.2454657390209296,23003.400368853476,2389.8567763071305
228
+ construction|yolo26x|Qwen3.8,learned,0.24,0.24533954727030627,0.2483414219175777,23906.36040595266,2481.4111452936404
229
+ construction|yolo26x|Qwen3.8,learned,0.25,0.25432756324900135,0.2504846177578749,24777.071870298303,2569.695715387775
230
+ construction|yolo26x|Qwen3.8,learned,0.26,0.2636484687083888,0.252142216376585,25680.031907397482,2661.250084374285
231
+ construction|yolo26x|Qwen3.8,learned,0.27,0.275965379494008,0.25276746625574176,26873.22909927855,2782.2326433921735
232
+ construction|yolo26x|Qwen3.8,learned,0.28,0.29427430093209056,0.25702642690899347,28646.900600723373,2962.071582472818
233
+ construction|yolo26x|Qwen3.8,learned,0.29,0.31091877496671105,0.26183035419940204,30259.32923840049,3125.561527091586
234
+ construction|yolo26x|Qwen3.8,learned,0.3,0.322237017310253,0.2672887365132105,31355.780712020925,3236.734689432348
235
+ construction|yolo26x|Qwen3.8,learned,0.31,0.33388814913448733,0.27039536303602457,32484.480758394904,3351.1776506654855
236
+ construction|yolo26x|Qwen3.8,learned,0.32,0.34354194407456723,0.2713201570387918,33419.68936824763,3446.001818544371
237
+ construction|yolo26x|Qwen3.8,learned,0.33,0.3492010652463382,0.2722481828097158,33967.91510505785,3501.588399714752
238
+ construction|yolo26x|Qwen3.8,learned,0.34,0.3545272969374168,0.274417471554269,34483.89226911453,3553.905181992758
239
+ construction|yolo26x|Qwen3.8,learned,0.35,0.3648468708388815,0.2782223498666452,35483.59802447434,3655.268947656394
240
+ construction|yolo26x|Qwen3.8,learned,0.36,0.37217043941411454,0.2795152621397537,36193.06662505227,3727.204523288652
241
+ construction|yolo26x|Qwen3.8,learned,0.37,0.37683089214380827,0.28090947933574756,36644.54664360186,3772.981707781907
242
+ construction|yolo26x|Qwen3.8,learned,0.38,0.3854860186418109,0.28295009794869297,37483.00953519396,3857.9964789836663
243
+ construction|yolo26x|Qwen3.8,learned,0.39,0.39980026631158455,0.28644294895435674,38869.69816359627,3998.5978313558066
244
+ construction|yolo26x|Qwen3.8,learned,0.4,0.4054593874833555,0.2862200047733393,39417.923900406495,4054.1844125261878
245
+ construction|yolo26x|Qwen3.8,learned,0.41,0.4104527296937417,0.28760386086589435,39901.65249170963,4103.231395911818
246
+ construction|yolo26x|Qwen3.8,learned,0.42,0.4181091877496671,0.2930744270093899,40643.3696650411,4178.436770436451
247
+ construction|yolo26x|Qwen3.8,learned,0.43,0.42809587217043943,0.2961685540536353,41610.82684764737,4276.530737207712
248
+ construction|yolo26x|Qwen3.8,learned,0.44,0.4370838881491345,0.2977058723546182,42481.53831199301,4364.8153073018475
249
+ construction|yolo26x|Qwen3.8,learned,0.45,0.44673768308921435,0.2999402430635043,43416.74692184573,4459.639475180732
250
+ construction|yolo26x|Qwen3.8,learned,0.46,0.45206391478029295,0.2993046120720782,43932.72408590242,4511.956257458738
251
+ construction|yolo26x|Qwen3.8,learned,0.47,0.45872170439414117,0.2998310238962732,44577.69554097326,4577.352235306245
252
+ construction|yolo26x|Qwen3.8,learned,0.48,0.4673768308921438,0.30393211265289466,45416.15843256536,4662.367006508005
253
+ construction|yolo26x|Qwen3.8,learned,0.49,0.47503328894806923,0.30350001249166053,46157.87560589683,4737.572381032637
254
+ construction|yolo26x|Qwen3.8,learned,0.5,0.4876830892143808,0.29875800068449215,47383.321370531434,4861.824738942901
255
+ construction|yolo26x|Qwen3.8,learned,0.51,0.5046604527296937,0.29989697064322696,49027.99858096209,5028.5844824540445
256
+ construction|yolo26x|Qwen3.8,learned,0.52,0.5206391478029294,0.297424793956044,50575.93007313212,5185.534829288062
257
+ construction|yolo26x|Qwen3.8,learned,0.53,0.5249667110519307,0.297424793956044,50995.161518928166,5228.042214888941
258
+ construction|yolo26x|Qwen3.8,learned,0.54,0.5436085219707057,0.2978194775414123,52801.08159312654,5411.150952861962
259
+ construction|yolo26x|Qwen3.8,learned,0.55,0.5529294274300932,0.29630990060928936,53704.04163022572,5502.705321848472
260
+ construction|yolo26x|Qwen3.8,learned,0.56,0.5612516644474035,0.29578344205458756,54510.25594906428,5584.450294157856
261
+ construction|yolo26x|Qwen3.8,learned,0.57,0.5692410119840213,0.2950949302773625,55284.221695149296,5662.925467574864
262
+ construction|yolo26x|Qwen3.8,learned,0.58,0.5902130492676432,0.2947373376338526,57315.88177862246,5868.922797794512
263
+ construction|yolo26x|Qwen3.8,learned,0.59,0.6021970705725699,0.29722091277674884,58476.830397749975,5986.635557920024
264
+ construction|yolo26x|Qwen3.8,learned,0.6,0.6208388814913449,0.29665021956289706,60282.750471948355,6169.744295893045
265
+ construction|yolo26x|Qwen3.8,learned,0.61,0.6288282290279628,0.295988506182091,61056.71621803337,6248.219469310054
266
+ construction|yolo26x|Qwen3.8,learned,0.62,0.6421438082556591,0.2939669175161438,62346.65912817505,6379.011425005067
267
+ construction|yolo26x|Qwen3.8,learned,0.63,0.6491344873501997,0.29277362668991463,63023.87915599944,6447.67720174495
268
+ construction|yolo26x|Qwen3.8,learned,0.64,0.6571238348868176,0.2930821542583563,63797.84490208446,6526.152375161959
269
+ construction|yolo26x|Qwen3.8,learned,0.65,0.6677762982689747,0.2892305220657417,64829.79923019781,6630.78593971797
270
+ construction|yolo26x|Qwen3.8,learned,0.66,0.6744340878828229,0.2915915088317972,65474.77068526865,6696.181917565477
271
+ construction|yolo26x|Qwen3.8,learned,0.67,0.6854194407456724,0.2936540392680743,66538.97358613556,6804.085281013864
272
+ construction|yolo26x|Qwen3.8,learned,0.68,0.7047270306258322,0.29632332439424064,68409.390805841,6993.733616771635
273
+ construction|yolo26x|Qwen3.8,learned,0.69,0.7157123834886817,0.2937246709263134,69473.5937067079,7101.6369802200215
274
+ construction|yolo26x|Qwen3.8,learned,0.7,0.7253661784287616,0.2952254059747369,70408.80231656063,7196.461148098907
275
+ construction|yolo26x|Qwen3.8,learned,0.71,0.7283621837549934,0.2945683782087344,70699.03947134252,7225.889338130286
276
+ construction|yolo26x|Qwen3.8,learned,0.72,0.741011984021305,0.29252963134197957,71924.48523597712,7350.1416960405495
277
+ construction|yolo26x|Qwen3.8,learned,0.73,0.7526631158455392,0.2900395106945919,73053.1852823511,7464.584657273686
278
+ construction|yolo26x|Qwen3.8,learned,0.74,0.7616511318242344,0.2887744568626224,73923.89674669674,7552.869227367822
279
+ construction|yolo26x|Qwen3.8,learned,0.75,0.7709720372836218,0.28730364578560735,74826.85678379593,7644.4235963543315
280
+ construction|yolo26x|Qwen3.8,learned,0.76,0.7749667110519307,0.2858070539975931,75213.83965683842,7683.661183062835
281
+ construction|yolo26x|Qwen3.8,learned,0.77,0.7792942743009321,0.28488735368132817,75633.07110263448,7726.168568663716
282
+ construction|yolo26x|Qwen3.8,learned,0.78,0.7846205059920106,0.2830658409372236,76149.04826669116,7778.485350941721
283
+ construction|yolo26x|Qwen3.8,learned,0.79,0.7972703062583223,0.2809995077380739,77374.49403132577,7902.737708851985
284
+ construction|yolo26x|Qwen3.8,learned,0.8,0.8129161118508655,0.28013559922601305,78890.17695074226,8056.418256793627
285
+ construction|yolo26x|Qwen3.8,learned,0.81,0.8199067909454061,0.27814841049220523,79567.39697856663,8125.084033533509
286
+ construction|yolo26x|Qwen3.8,learned,0.82,0.831225033288948,0.2772348763322643,80663.84845218707,8236.25719587427
287
+ construction|yolo26x|Qwen3.8,learned,0.83,0.84254327563249,0.27700595116508253,81760.29992580751,8347.430358215033
288
+ construction|yolo26x|Qwen3.8,learned,0.84,0.857190412782956,0.2763733409861609,83179.23712696337,8491.301509479548
289
+ construction|yolo26x|Qwen3.8,learned,0.85,0.8618508655126498,0.2761345157638265,83630.71714551296,8537.078693972804
290
+ construction|yolo26x|Qwen3.8,learned,0.86,0.87017310252996,0.2761345157638265,84436.93146435152,8618.823666282187
291
+ construction|yolo26x|Qwen3.8,learned,0.87,0.8848202396804261,0.27546832609774896,85855.86866550738,8762.694817546704
292
+ construction|yolo26x|Qwen3.8,learned,0.88,0.8984687083888149,0.27423981143437465,87178.06014840261,8896.756572134092
293
+ construction|yolo26x|Qwen3.8,learned,0.89,0.9087882822902796,0.2736575311942959,88177.76590376243,8998.12033779773
294
+ construction|yolo26x|Qwen3.8,learned,0.9,0.9161118508655126,0.272975172975173,88887.23450434035,9070.055913429987
295
+ construction|yolo26x|Qwen3.8,learned,0.91,0.9217709720372836,0.2727190231806067,89435.46024115058,9125.64249460037
296
+ construction|yolo26x|Qwen3.8,learned,0.92,0.9264314247669774,0.27264729817760425,89886.94025970016,9171.419679093624
297
+ construction|yolo26x|Qwen3.8,learned,0.93,0.933422103861518,0.2741441928343702,90564.16028752456,9240.085455833507
298
+ construction|yolo26x|Qwen3.8,learned,0.94,0.9417443408788282,0.2733716828250215,91370.37460636311,9321.83042814289
299
+ construction|yolo26x|Qwen3.8,learned,0.95,0.9530625832223701,0.2727312493845365,92466.82607998354,9433.003590483651
300
+ construction|yolo26x|Qwen3.8,learned,0.96,0.9613848202396804,0.27223682114986464,93273.04039882211,9514.748562793036
301
+ construction|yolo26x|Qwen3.8,learned,0.97,0.9673768308921438,0.27453093162048386,93853.51470838586,9573.604942855793
302
+ construction|yolo26x|Qwen3.8,learned,0.98,0.9760319573901465,0.2737331615154065,94691.97759997797,9658.619714057553
303
+ construction|yolo26x|Qwen3.8,learned,0.99,0.9890146471371505,0.2725383412883413,95949.67193736612,9786.141870860192
304
+ construction|yolo26x|Qwen3.8,learned,1.0,1.0,0.27001009067886245,97013.87483823301,9894.045234308578
305
+ construction|yolo26x|Qwen3.8,confidence,0.0,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
306
+ construction|yolo26x|Qwen3.8,confidence,0.01,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
307
+ construction|yolo26x|Qwen3.8,confidence,0.02,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
308
+ construction|yolo26x|Qwen3.8,confidence,0.03,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
309
+ construction|yolo26x|Qwen3.8,confidence,0.04,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
310
+ construction|yolo26x|Qwen3.8,confidence,0.05,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
311
+ construction|yolo26x|Qwen3.8,confidence,0.06,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
312
+ construction|yolo26x|Qwen3.8,confidence,0.07,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
313
+ construction|yolo26x|Qwen3.8,confidence,0.08,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
314
+ construction|yolo26x|Qwen3.8,confidence,0.09,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
315
+ construction|yolo26x|Qwen3.8,confidence,0.1,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
316
+ construction|yolo26x|Qwen3.8,confidence,0.11,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
317
+ construction|yolo26x|Qwen3.8,confidence,0.12,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
318
+ construction|yolo26x|Qwen3.8,confidence,0.13,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
319
+ construction|yolo26x|Qwen3.8,confidence,0.14,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
320
+ construction|yolo26x|Qwen3.8,confidence,0.15,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
321
+ construction|yolo26x|Qwen3.8,confidence,0.16,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
322
+ construction|yolo26x|Qwen3.8,confidence,0.17,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
323
+ construction|yolo26x|Qwen3.8,confidence,0.18,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
324
+ construction|yolo26x|Qwen3.8,confidence,0.19,0.17942743009320905,0.1784481207469488,17517.959245999285,1808.7296972439362
325
+ construction|yolo26x|Qwen3.8,confidence,0.2,0.19340878828229027,0.1780928651788194,18872.39930164806,1946.0612507237013
326
+ construction|yolo26x|Qwen3.8,confidence,0.21,0.20306258322237017,0.17829989216687975,19807.60791150079,2040.8854186025867
327
+ construction|yolo26x|Qwen3.8,confidence,0.22,0.20838881491344874,0.1779854683890517,20323.585075557465,2093.2022008805925
328
+ construction|yolo26x|Qwen3.8,confidence,0.23,0.21537949400798934,0.17917657728013692,21000.805103381852,2161.867977620475
329
+ construction|yolo26x|Qwen3.8,confidence,0.24,0.23135818908122505,0.17803687432776102,22548.736595551884,2318.8183244544925
330
+ construction|yolo26x|Qwen3.8,confidence,0.25,0.24800266311584554,0.18026802885144627,24161.165233228996,2482.3082690732604
331
+ construction|yolo26x|Qwen3.8,confidence,0.26,0.2589880159786951,0.18119827871024874,25225.368134095894,2590.211632521647
332
+ construction|yolo26x|Qwen3.8,confidence,0.27,0.27430093209054596,0.17963665085316138,26708.802480758837,2740.6223815709136
333
+ construction|yolo26x|Qwen3.8,confidence,0.28,0.281624500665779,0.17924095839307458,27418.271081336767,2812.5579572031716
334
+ construction|yolo26x|Qwen3.8,confidence,0.29,0.28761651131824234,0.1789157174618718,27998.745390900527,2871.4143372659278
335
+ construction|yolo26x|Qwen3.8,confidence,0.3,0.2969374167776298,0.1787206845800119,28901.70542799971,2962.9687062524376
336
+ construction|yolo26x|Qwen3.8,confidence,0.31,0.30459387483355527,0.17813242189706796,29643.422601331185,3038.174080777071
337
+ construction|yolo26x|Qwen3.8,confidence,0.32,0.3129161118508655,0.17772000842137498,30449.636920169738,3119.919053086455
338
+ construction|yolo26x|Qwen3.8,confidence,0.33,0.31990679094540614,0.17791259734473341,31126.85694799413,3188.5848298263377
339
+ construction|yolo26x|Qwen3.8,confidence,0.34,0.3322237017310253,0.1789653934599337,32320.05413987519,3309.5673888442257
340
+ construction|yolo26x|Qwen3.8,confidence,0.35,0.34054593874833555,0.18042682200794885,33126.268458713756,3391.31236115361
341
+ construction|yolo26x|Qwen3.8,confidence,0.36,0.3488681757656458,0.17962877325912951,33932.48277755231,3473.057333462994
342
+ construction|yolo26x|Qwen3.8,confidence,0.37,0.3581890812250333,0.18158304685510065,34835.44281465149,3564.611702449504
343
+ construction|yolo26x|Qwen3.8,confidence,0.38,0.37017310252996005,0.18069239723097547,35996.39143377901,3682.3244625750167
344
+ construction|yolo26x|Qwen3.8,confidence,0.39,0.37683089214380827,0.1812845803851709,36641.362888849864,3747.720440422524
345
+ construction|yolo26x|Qwen3.8,confidence,0.4,0.38348868175765644,0.1824781664689562,37286.3343439207,3813.116418270031
346
+ construction|yolo26x|Qwen3.8,confidence,0.41,0.4021304926764314,0.18220594501399923,39092.254418119075,3996.225156243051
347
+ construction|yolo26x|Qwen3.8,confidence,0.42,0.41311584553928093,0.18162498930772586,40156.457318985966,4104.128519691438
348
+ construction|yolo26x|Qwen3.8,confidence,0.43,0.42676431424766975,0.18066434967152634,41478.6488018812,4238.190274278827
349
+ construction|yolo26x|Qwen3.8,confidence,0.44,0.4324234354194407,0.1803913789289444,42026.874538691416,4293.776855449209
350
+ construction|yolo26x|Qwen3.8,confidence,0.45,0.44107856191744343,0.17959468902865128,42865.33743028352,4378.791626650968
351
+ construction|yolo26x|Qwen3.8,confidence,0.46,0.45772303595206393,0.17863383514615866,44477.766067960634,4542.281571269737
352
+ construction|yolo26x|Qwen3.8,confidence,0.47,0.4717043941411451,0.17957308538379646,45832.20612360941,4679.6131247495005
353
+ construction|yolo26x|Qwen3.8,confidence,0.48,0.4826897470039947,0.17910245145856155,46896.40902447631,4787.516488197888
354
+ construction|yolo26x|Qwen3.8,confidence,0.49,0.4896804260985353,0.17871833907402393,47573.62905230069,4856.182264937771
355
+ construction|yolo26x|Qwen3.8,confidence,0.5,0.5013315579227696,0.18015785955296285,48702.32909867467,4970.625226170908
356
+ construction|yolo26x|Qwen3.8,confidence,0.51,0.5116511318242344,0.1810878167493375,49702.034854034486,5071.988991834544
357
+ construction|yolo26x|Qwen3.8,confidence,0.52,0.5183089214380826,0.18177962418520882,50347.00630910533,5137.384969682052
358
+ construction|yolo26x|Qwen3.8,confidence,0.53,0.5252996005326231,0.18246323539519374,51024.226336929714,5206.050746421934
359
+ construction|yolo26x|Qwen3.8,confidence,0.54,0.5396138482023968,0.1839876166906015,52410.914965332035,5346.652098794074
360
+ construction|yolo26x|Qwen3.8,confidence,0.55,0.5452729693741678,0.1846674213907964,52959.14070214226,5402.2386799644555
361
+ construction|yolo26x|Qwen3.8,confidence,0.56,0.559254327563249,0.18676294233791663,54313.58075779103,5539.57023344422
362
+ construction|yolo26x|Qwen3.8,confidence,0.57,0.572237017310253,0.18938663221175195,55571.275095179175,5667.092390246859
363
+ construction|yolo26x|Qwen3.8,confidence,0.58,0.5862183754993342,0.1909571002676625,56925.71515082795,5804.423943726624
364
+ construction|yolo26x|Qwen3.8,confidence,0.59,0.5998668442077231,0.1934300101790686,58247.90663372319,5938.485698314014
365
+ construction|yolo26x|Qwen3.8,confidence,0.6,0.6191744340878829,0.1973701682130997,60118.32385342864,6128.134034071785
366
+ construction|yolo26x|Qwen3.8,confidence,0.61,0.6264980026631158,0.1986727924998457,60827.792454006565,6200.069609704043
367
+ construction|yolo26x|Qwen3.8,confidence,0.62,0.6384820239680427,0.20186289191698192,61988.74107313409,6317.782369829556
368
+ construction|yolo26x|Qwen3.8,confidence,0.63,0.6461384820239681,0.20246031058520178,62730.45824646557,6392.987744354189
369
+ construction|yolo26x|Qwen3.8,confidence,0.64,0.653462050599201,0.20349944137622245,63439.92684704349,6464.923319986447
370
+ construction|yolo26x|Qwen3.8,confidence,0.65,0.6597869507323568,0.20442990090722246,64052.64972936079,6527.049498941578
371
+ construction|yolo26x|Qwen3.8,confidence,0.66,0.6661118508655126,0.2044993942500819,64665.372611678096,6589.17567789671
372
+ construction|yolo26x|Qwen3.8,confidence,0.67,0.6734354194407457,0.2064379542733177,65374.84121225603,6661.1112535289685
373
+ construction|yolo26x|Qwen3.8,confidence,0.68,0.6820905459387483,0.2065831206227621,66213.30410384812,6746.126024730727
374
+ construction|yolo26x|Qwen3.8,confidence,0.69,0.6927430093209055,0.20802802394105854,67245.25843196148,6850.759589286739
375
+ construction|yolo26x|Qwen3.8,confidence,0.7,0.7090545938748336,0.21169680596278434,68825.43849688505,7010.979735013132
376
+ construction|yolo26x|Qwen3.8,confidence,0.71,0.7170439414114514,0.2139280527103987,69599.40424297006,7089.454908430141
377
+ construction|yolo26x|Qwen3.8,confidence,0.72,0.7243675099866844,0.2146900231025185,70308.87284354799,7161.390484062398
378
+ construction|yolo26x|Qwen3.8,confidence,0.73,0.7330226364846871,0.21653561577692484,71147.33573514009,7246.405255264158
379
+ construction|yolo26x|Qwen3.8,confidence,0.74,0.7363515312916112,0.21721689648133474,71469.82146267552,7279.1032441879115
380
+ construction|yolo26x|Qwen3.8,confidence,0.75,0.7486684420772304,0.22103445955201576,72663.01865455658,7400.0858032057995
381
+ construction|yolo26x|Qwen3.8,confidence,0.76,0.757323568575233,0.22340134248367294,73501.48154614867,7485.100574407558
382
+ construction|yolo26x|Qwen3.8,confidence,0.77,0.7639813581890812,0.2269322568854726,74146.45300121952,7550.496552255066
383
+ construction|yolo26x|Qwen3.8,confidence,0.78,0.7733022636484687,0.22757452798038888,75049.4130383187,7642.050921241575
384
+ construction|yolo26x|Qwen3.8,confidence,0.79,0.784953395472703,0.227821050912831,76178.11308469268,7756.493882474713
385
+ construction|yolo26x|Qwen3.8,confidence,0.8,0.7939414114513982,0.22991865671924006,77048.82454903833,7844.7784525688485
386
+ construction|yolo26x|Qwen3.8,confidence,0.81,0.8039280958721704,0.23486191981046528,78016.28173164459,7942.8724193401085
387
+ construction|yolo26x|Qwen3.8,confidence,0.82,0.8122503328894807,0.23803916047544685,78822.49605048315,8024.617391649493
388
+ construction|yolo26x|Qwen3.8,confidence,0.83,0.8215712383488681,0.24022947987632814,79725.45608758234,8116.171760636003
389
+ construction|yolo26x|Qwen3.8,confidence,0.84,0.8388814913448736,0.24528627405740036,81402.38187076653,8286.201303039523
390
+ construction|yolo26x|Qwen3.8,confidence,0.85,0.8535286284953395,0.249537136326177,82821.3190719224,8430.072454304036
391
+ construction|yolo26x|Qwen3.8,confidence,0.86,0.8638482023968043,0.25211074301800224,83821.0248272822,8531.436219967674
392
+ construction|yolo26x|Qwen3.8,confidence,0.87,0.8745006657789614,0.2540042317315216,84852.97915539556,8636.069784523685
393
+ construction|yolo26x|Qwen3.8,confidence,0.88,0.8764980026631158,0.2540996517958595,85046.4705919168,8655.688577877936
394
+ construction|yolo26x|Qwen3.8,confidence,0.89,0.8841544607190412,0.25436194678023316,85788.18776524828,8730.89395240257
395
+ construction|yolo26x|Qwen3.8,confidence,0.9,0.8891478029294274,0.2534339447803836,86271.91635655142,8779.9409357882
396
+ construction|yolo26x|Qwen3.8,confidence,0.91,0.8981358189081226,0.2563240880787248,87142.62782089706,8868.225505882336
397
+ construction|yolo26x|Qwen3.8,confidence,0.92,0.9054593874833555,0.25551845731687656,87852.09642147498,8940.161081514594
398
+ construction|yolo26x|Qwen3.8,confidence,0.93,0.9221038615179761,0.2597490004762105,89464.5250591521,9103.651026133362
399
+ construction|yolo26x|Qwen3.8,confidence,0.94,0.9254327563249002,0.26306323674307786,89787.01078668753,9136.349015057116
400
+ construction|yolo26x|Qwen3.8,confidence,0.95,0.9364181091877497,0.26218185491449153,90851.21368755442,9244.252378505502
401
+ construction|yolo26x|Qwen3.8,confidence,0.96,0.953728362183755,0.26357052137555514,92528.13947073862,9414.281920909021
402
+ construction|yolo26x|Qwen3.8,confidence,0.97,0.96271637816245,0.26400290307863034,93398.85093508427,9502.566491003154
403
+ construction|yolo26x|Qwen3.8,confidence,0.98,0.9700399467376831,0.2647475724296899,94108.3195356622,9574.502066635414
404
+ construction|yolo26x|Qwen3.8,confidence,0.99,0.987017310252996,0.2649306125590882,95752.99674609285,9741.261810146556
405
+ construction|yolo26x|Qwen3.8,confidence,1.0,1.0,0.27001009067886245,97010.691083481,9868.783966949195
paper_results/data/main_table_cost_qwen_20260912/main_table_results.json ADDED
The diff for this file is too large to render. See raw diff
 
paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_flops.py ADDED
@@ -0,0 +1,152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ from __future__ import annotations
3
+
4
+ import argparse
5
+ import json
6
+ import re
7
+ from pathlib import Path
8
+
9
+ import numpy as np
10
+ import torch
11
+ from PIL import Image
12
+ from torch.utils.flop_counter import FlopCounterMode
13
+
14
+
15
+ def count(call):
16
+ with torch.inference_mode(), FlopCounterMode(display=False) as counter:
17
+ value = call()
18
+ torch.cuda.synchronize()
19
+ counts = counter.get_flop_counts().get('Global', {})
20
+ return int(counter.get_total_flops()), value, {str(k): int(v) for k, v in counts.items()}
21
+
22
+
23
+ def make_text(root: Path, model_kind: str, task: str, row: dict, processor):
24
+ prompt = (root / 'outputs/large_model_labels_all_20260909/prompts' / task / f'{model_kind}.txt').read_text(encoding='utf-8').strip()
25
+ if task == 'grefcoco':
26
+ prompt = prompt.replace('{expression}', row['expression'])
27
+ if model_kind == 'Qwen3.8':
28
+ content = [{'type': 'image'}, {'type': 'text', 'text': '\n'}, {'type': 'text', 'text': prompt}]
29
+ return processor.apply_chat_template([{'role': 'user', 'content': content}], tokenize=False, add_generation_prompt=True)
30
+ message = '(<image>./</image>)\n' + prompt
31
+ return processor.tokenizer.apply_chat_template([{'role': 'user', 'content': message}], tokenize=False, add_generation_prompt=True, enable_thinking=False)
32
+
33
+
34
+ def prepare(root: Path, model_kind: str, task: str, row: dict, processor):
35
+ image = Image.open(row['image_path']).convert('RGB')
36
+ text = make_text(root, model_kind, task, row, processor)
37
+ if model_kind == 'Qwen3.8':
38
+ data = processor(text=[text], images=[image], return_tensors='pt', size={'shortest_edge': 65536, 'longest_edge': 1003520})
39
+ else:
40
+ data = processor([text], [[image]], return_tensors='pt', max_length=16384)
41
+ image.close()
42
+ return data
43
+
44
+
45
+ def main():
46
+ parser = argparse.ArgumentParser()
47
+ parser.add_argument('--root', type=Path, required=True)
48
+ parser.add_argument('--model-kind', choices=['Qwen3.8', 'MiniCPM'], required=True)
49
+ parser.add_argument('--task', choices=['grefcoco', 'construction'], required=True)
50
+ parser.add_argument('--output', type=Path, required=True)
51
+ parser.add_argument('--samples', type=int, default=4)
52
+ args = parser.parse_args()
53
+ if args.output.exists():
54
+ raise FileExistsError(args.output)
55
+ expert = 'instancevg' if args.task == 'grefcoco' else 'yolo26x'
56
+ rows = [json.loads(line) for line in (args.root / f'outputs/router_full_20260910/data/{expert}/test/records.jsonl').open(encoding='utf-8')]
57
+ if args.samples > len(rows):
58
+ raise ValueError('sample count exceeds test rows')
59
+ selected_indices = np.random.default_rng(20260911).choice(len(rows), args.samples, replace=False)
60
+ selected = [rows[int(index)] for index in selected_indices]
61
+ checkpoint_root = args.root.parents[1] / 'checkpoints'
62
+ if not checkpoint_root.exists():
63
+ checkpoint_root = args.root / 'checkpoints'
64
+ model_path = checkpoint_root / ('Qwen3.8-27B-FP8' if args.model_kind == 'Qwen3.8' else 'MiniCPM-V-4_5')
65
+ from transformers import AutoProcessor
66
+ processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True, local_files_only=True)
67
+ if args.model_kind == 'Qwen3.8':
68
+ from transformers import FineGrainedFP8Config, Qwen3_5ForConditionalGeneration
69
+ import transformers.models.qwen3_5.modeling_qwen3_5 as implementation
70
+ implementation.causal_conv1d_fn = None
71
+ implementation.causal_conv1d_update = None
72
+ implementation.chunk_gated_delta_rule = None
73
+ implementation.fused_recurrent_gated_delta_rule = None
74
+ implementation.FusedRMSNormGated = None
75
+ implementation.is_fast_path_available = False
76
+ model = Qwen3_5ForConditionalGeneration.from_pretrained(
77
+ model_path, local_files_only=True, dtype=torch.bfloat16, device_map={'': 'cuda:0'},
78
+ attn_implementation='eager', quantization_config=FineGrainedFP8Config(dequantize=False),
79
+ )
80
+ else:
81
+ model = processor # replaced below to keep imports local to the selected environment
82
+ from transformers import AutoModel
83
+ from transformers.modeling_utils import PreTrainedModel
84
+ if not hasattr(PreTrainedModel, 'all_tied_weights_keys'):
85
+ def get_tied_weights(self):
86
+ return getattr(self, '_all_tied_weights_keys', getattr(self, '_tied_weights_keys', {}) or {})
87
+ def set_tied_weights(self, value):
88
+ self._all_tied_weights_keys = value
89
+ PreTrainedModel.all_tied_weights_keys = property(get_tied_weights, set_tied_weights)
90
+ model = AutoModel.from_pretrained(
91
+ model_path, trust_remote_code=True, local_files_only=True, torch_dtype=torch.bfloat16,
92
+ device_map={'': 'cuda:0'}, attn_implementation='eager',
93
+ )
94
+ torch.set_num_threads(4)
95
+ torch.backends.mha.set_fastpath_enabled(False)
96
+ model.eval().requires_grad_(False)
97
+ records = []
98
+ for row in selected:
99
+ data = prepare(args.root, args.model_kind, args.task, row, processor).to('cuda')
100
+ prompt_tokens = int(data['input_ids'].shape[-1])
101
+ if args.model_kind == 'MiniCPM':
102
+ vision_flops, embedding_result, vision_ops = count(lambda: model.get_vllm_embedding(data))
103
+ embeddings, _ = embedding_result
104
+ prefill_flops, prefill_output, prefill_ops = count(lambda: model.llm(inputs_embeds=embeddings, attention_mask=data['attention_mask'], use_cache=True, logits_to_keep=1))
105
+ prefill_flops += vision_flops
106
+ prefill_ops.update({'vision:'+k:v for k,v in vision_ops.items()})
107
+ length = int(data['input_ids'].shape[-1])
108
+ known_token = data['input_ids'][:, -1:]
109
+ kwargs = {
110
+ 'input_ids': known_token,
111
+ 'past_key_values': prefill_output.past_key_values,
112
+ 'attention_mask': torch.ones((1, length + 1), device='cuda', dtype=torch.long),
113
+ 'use_cache': True,
114
+ 'logits_to_keep': 1,
115
+ }
116
+ decode_flops, _, decode_ops = count(lambda: model.llm(**kwargs))
117
+ else:
118
+ prefill_flops, prefill_output, prefill_ops = count(lambda: model(**data, use_cache=True, logits_to_keep=1))
119
+ length = int(data['input_ids'].shape[-1])
120
+ known_token = data['input_ids'][:, -1:]
121
+ kwargs = {
122
+ 'input_ids': known_token,
123
+ 'past_key_values': prefill_output.past_key_values,
124
+ 'attention_mask': torch.ones((1, length + 1), device='cuda', dtype=torch.long),
125
+ 'use_cache': True,
126
+ 'logits_to_keep': 1,
127
+ 'cache_position': torch.tensor([length], device='cuda'),
128
+ }
129
+ kwargs.pop('attention_mask')
130
+ decode_flops, _, decode_ops = count(lambda: model(**kwargs))
131
+ records.append({
132
+ 'sample_id': row['sample_id'], 'prompt_tokens_in_reference_tensor': prompt_tokens,
133
+ 'input_shapes': {key: list(value.shape) for key, value in data.items() if hasattr(value, 'shape')},
134
+ 'prefill_flops_counted': prefill_flops, 'decode_one_token_flops_counted': decode_flops,
135
+ 'prefill_gflops_counted': prefill_flops / 1e9, 'decode_one_token_gflops_counted': decode_flops / 1e9,
136
+ 'prefill_operators': prefill_ops, 'decode_operators': decode_ops,
137
+ })
138
+ args.output.parent.mkdir(parents=True, exist_ok=True)
139
+ args.output.write_text(json.dumps({'status': 'running', 'model_kind': args.model_kind, 'task': args.task, 'records': records}, ensure_ascii=False, indent=2), encoding='utf-8')
140
+ print(json.dumps({'completed': len(records), 'sample_id': row['sample_id'], 'prefill_gflops': prefill_flops / 1e9, 'decode_gflops': decode_flops / 1e9}), flush=True)
141
+ args.output.write_text(json.dumps({
142
+ 'status': 'complete', 'model_kind': args.model_kind, 'task': args.task, 'model_path': str(model_path),
143
+ 'sample_count': len(records), 'records': records,
144
+ 'mean_prefill_gflops_counted': float(np.mean([row['prefill_gflops_counted'] for row in records])),
145
+ 'mean_decode_one_token_gflops_counted': float(np.mean([row['decode_one_token_gflops_counted'] for row in records])),
146
+ 'scope': 'HF reference execution on the A800 with real task inputs; dense products/convolutions/supported attention counted with multiply-add=2 FLOPs. vLLM custom kernels, elementwise/normalization/nonlinearities and other unsupported operations are not complete.',
147
+ 'answer_generation': 'not performed; one known-token decode probe only',
148
+ }, ensure_ascii=False, indent=2), encoding='utf-8')
149
+
150
+
151
+ if __name__ == '__main__':
152
+ main()
paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_large.py ADDED
@@ -0,0 +1,209 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ from __future__ import annotations
3
+
4
+ import argparse
5
+ import json
6
+ import time
7
+ from pathlib import Path
8
+
9
+ import numpy as np
10
+ import jsonschema
11
+ from PIL import Image
12
+ from vllm import LLM, SamplingParams
13
+ from vllm.sampling_params import StructuredOutputsParams
14
+
15
+
16
+ EVENTS = [
17
+ 'rule_1_ppe_violation',
18
+ 'rule_2_fall_protection_violation',
19
+ 'rule_3_unprotected_edge_violation',
20
+ 'rule_4_excavator_proximity_violation',
21
+ ]
22
+
23
+
24
+ def gref_schema():
25
+ return {
26
+ 'type': 'object',
27
+ 'properties': {
28
+ 'objects': {
29
+ 'type': 'array', 'maxItems': 18,
30
+ 'items': {
31
+ 'type': 'object',
32
+ 'properties': {
33
+ 'bbox_2d': {'type': 'array', 'items': {'type': 'number', 'minimum': 0, 'maximum': 1000}, 'minItems': 4, 'maxItems': 4},
34
+ 'score': {'type': 'number', 'minimum': 0, 'maximum': 1},
35
+ },
36
+ 'required': ['bbox_2d', 'score'], 'additionalProperties': False,
37
+ },
38
+ },
39
+ },
40
+ 'required': ['objects'], 'additionalProperties': False,
41
+ }
42
+
43
+
44
+ def construction_schema():
45
+ box = {'type': 'array', 'items': {'type': 'integer', 'minimum': 0, 'maximum': 1000}, 'minItems': 4, 'maxItems': 4}
46
+ max_boxes = [13, 6, 3, 2]
47
+ events = {}
48
+ for event, limit in zip(EVENTS, max_boxes):
49
+ events[event] = {
50
+ 'type': 'object',
51
+ 'properties': {
52
+ 'present': {'type': 'boolean'},
53
+ 'reason': {'type': ['string', 'null']},
54
+ 'boxes': {'type': 'array', 'items': box, 'maxItems': limit},
55
+ },
56
+ 'required': ['present', 'reason', 'boxes'], 'additionalProperties': False,
57
+ }
58
+ return {'type': 'object', 'properties': {'events': {'type': 'object', 'properties': events, 'required': EVENTS, 'additionalProperties': False}}, 'required': ['events'], 'additionalProperties': False}
59
+
60
+
61
+ def make_prompt(root: Path, model_kind: str, task: str, row: dict, processor_or_tokenizer):
62
+ prompt = (root / 'outputs/large_model_labels_all_20260909/prompts' / task / f'{model_kind}.txt').read_text(encoding='utf-8').strip()
63
+ if task == 'grefcoco':
64
+ prompt = prompt.replace('{expression}', row['expression'])
65
+ if model_kind == 'Qwen3.8':
66
+ converted = [{'type': 'image'}]
67
+ converted.append({'type': 'text', 'text': '\n'})
68
+ converted.append({'type': 'text', 'text': prompt})
69
+ messages = [{'role': 'user', 'content': converted}]
70
+ return processor_or_tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
71
+ message = '(<image>./</image>)\n' + prompt
72
+ return processor_or_tokenizer.apply_chat_template([{'role': 'user', 'content': message}], tokenize=False, add_generation_prompt=True, enable_thinking=False)
73
+
74
+
75
+ def main():
76
+ parser = argparse.ArgumentParser()
77
+ parser.add_argument('--root', type=Path, required=True)
78
+ parser.add_argument('--model-kind', choices=['Qwen3.8', 'MiniCPM'], required=True)
79
+ parser.add_argument('--task', choices=['grefcoco', 'construction'], required=True)
80
+ parser.add_argument('--output', type=Path, required=True)
81
+ parser.add_argument('--samples', type=int, default=32)
82
+ parser.add_argument('--batch-size', type=int, default=8)
83
+ parser.add_argument('--gpu-memory-utilization', type=float, default=0.90)
84
+ parser.add_argument('--max-model-len', type=int, default=16384)
85
+ args = parser.parse_args()
86
+ if args.output.exists():
87
+ raise FileExistsError(args.output)
88
+ if args.samples < 3 or args.batch_size < 1:
89
+ raise ValueError('samples must include warmup rows and batch size must be positive')
90
+ expert = 'instancevg' if args.task == 'grefcoco' else 'yolo26x'
91
+ rows = [json.loads(line) for line in (args.root / f'outputs/router_full_20260910/data/{expert}/test/records.jsonl').open(encoding='utf-8')]
92
+ if args.samples > len(rows):
93
+ raise ValueError(f'samples {args.samples} exceeds {len(rows)}')
94
+ selected_indices = np.random.default_rng(20260911).choice(len(rows), args.samples, replace=False)
95
+ selected = [rows[int(index)] for index in selected_indices]
96
+ checkpoint_root = args.root.parents[1] / 'checkpoints'
97
+ if not checkpoint_root.exists():
98
+ checkpoint_root = args.root / 'checkpoints'
99
+ model_path = checkpoint_root / ('Qwen3.8-27B-FP8' if args.model_kind == 'Qwen3.8' else 'MiniCPM-V-4_5')
100
+ if args.model_kind == 'Qwen3.8':
101
+ from transformers import AutoProcessor
102
+ processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True, local_files_only=True)
103
+ model_kwargs = {'mm_processor_kwargs': {'size': {'shortest_edge': 65536, 'longest_edge': 1003520}}}
104
+ else:
105
+ from transformers import AutoTokenizer
106
+ processor = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, local_files_only=True)
107
+ model_kwargs = {}
108
+ model = LLM(model=str(model_path), trust_remote_code=True, tensor_parallel_size=1, max_model_len=args.max_model_len,
109
+ limit_mm_per_prompt={'image': 1}, max_num_seqs=args.batch_size,
110
+ gpu_memory_utilization=args.gpu_memory_utilization, enforce_eager=True, **model_kwargs)
111
+ schema = gref_schema() if args.task == 'grefcoco' else construction_schema()
112
+ sampling = SamplingParams(temperature=0, max_tokens=512 if args.task == 'grefcoco' else 768,
113
+ structured_outputs=StructuredOutputsParams(json=schema, disable_additional_properties=True))
114
+
115
+ def generate(batch_rows):
116
+ images = []
117
+ requests = []
118
+ for row in batch_rows:
119
+ required = {'sample_id', 'image_path'}
120
+ if args.task == 'grefcoco':
121
+ required.add('expression')
122
+ missing = required - row.keys()
123
+ if missing:
124
+ raise ValueError(f'missing fields {sorted(missing)} for {row["sample_id"]}')
125
+ image = Image.open(row['image_path']).convert('RGB')
126
+ images.append(image)
127
+ requests.append({'prompt': make_prompt(args.root, args.model_kind, args.task, row, processor), 'multi_modal_data': {'image': image}})
128
+ started = time.perf_counter()
129
+ generated = model.generate(requests, sampling, use_tqdm=False)
130
+ elapsed = time.perf_counter() - started
131
+ for image in images:
132
+ image.close()
133
+ if len(generated) != len(batch_rows):
134
+ raise ValueError('vLLM returned an unexpected number of results')
135
+ return elapsed, generated
136
+
137
+ warmup_count = min(2, len(selected) - 1)
138
+ for row in selected[:warmup_count]:
139
+ generate([row])
140
+ measured = selected[warmup_count:]
141
+ batches = []
142
+ for start in range(0, len(measured), args.batch_size):
143
+ batch_rows = measured[start:start + args.batch_size]
144
+ batch_started = time.perf_counter()
145
+ elapsed, generated = generate(batch_rows)
146
+ wall = time.perf_counter() - batch_started
147
+ output_records = []
148
+ for result in generated:
149
+ raw = result.outputs[0].text
150
+ error = None
151
+ try:
152
+ parsed = json.loads(raw)
153
+ jsonschema.validate(parsed, schema)
154
+ boxes = [o['bbox_2d'] for o in parsed['objects']] if args.task == 'grefcoco' else [b for e in parsed['events'].values() for b in e['boxes']]
155
+ if any(b[0] >= b[2] or b[1] >= b[3] for b in boxes):
156
+ raise ValueError('non-positive box size')
157
+ except (json.JSONDecodeError, jsonschema.ValidationError, ValueError) as exc:
158
+ parsed = None
159
+ error = str(exc)
160
+ output_records.append({'raw_response':raw,'valid_prediction':parsed,'error':error,
161
+ 'finish_reason':result.outputs[0].finish_reason,'completion_tokens':len(result.outputs[0].token_ids),
162
+ 'prompt_tokens':len(result.prompt_token_ids)})
163
+ item = {
164
+ 'sample_ids': [row['sample_id'] for row in batch_rows],
165
+ 'batch_size': len(batch_rows),
166
+ 'generation_latency_seconds': elapsed,
167
+ 'batch_wall_latency_seconds': wall,
168
+ 'amortized_generation_ms_per_sample': elapsed * 1000 / len(batch_rows),
169
+ 'amortized_batch_wall_ms_per_sample': wall * 1000 / len(batch_rows),
170
+ 'outputs': [
171
+ {
172
+ 'finish_reason': result.outputs[0].finish_reason,
173
+ 'stop_reason': result.outputs[0].stop_reason,
174
+ 'prompt_tokens': len(result.prompt_token_ids) if result.prompt_token_ids is not None else None,
175
+ 'completion_tokens': len(result.outputs[0].token_ids),
176
+ 'text_characters': len(result.outputs[0].text),
177
+ }
178
+ for result in generated
179
+ ],
180
+ }
181
+ batches.append(item)
182
+ item['outputs'] = output_records
183
+ args.output.parent.mkdir(parents=True, exist_ok=True)
184
+ args.output.write_text(json.dumps({
185
+ 'status': 'running', 'model_kind': args.model_kind, 'task': args.task, 'model_path': str(model_path),
186
+ 'sample_count_requested': args.samples, 'warmup_count': warmup_count, 'batch_size': args.batch_size,
187
+ 'sample_ids': [row['sample_id'] for row in selected], 'batches': batches,
188
+ }, ensure_ascii=False, indent=2), encoding='utf-8')
189
+ print(json.dumps({'completed_samples': sum(item['batch_size'] for item in batches), 'total_measured': len(measured), 'batch_latency_seconds': elapsed}), flush=True)
190
+ all_ms = [x['amortized_generation_ms_per_sample'] for x in batches]
191
+ all_wall_ms = [x['amortized_batch_wall_ms_per_sample'] for x in batches]
192
+ output = {
193
+ 'status': 'complete', 'model_kind': args.model_kind, 'task': args.task, 'expert': expert,
194
+ 'model_path': str(model_path), 'device': 'cuda:0', 'gpu_memory_utilization': args.gpu_memory_utilization,
195
+ 'sample_count_requested': args.samples, 'warmup_count': warmup_count, 'measured_count': len(measured),
196
+ 'batch_size': args.batch_size, 'sample_ids': [row['sample_id'] for row in selected], 'batches': batches,
197
+ 'mean_generation_ms_per_sample': float(np.mean(all_ms)), 'median_generation_ms_per_sample': float(np.median(all_ms)),
198
+ 'mean_batch_wall_ms_per_sample': sum(b['batch_wall_latency_seconds'] for b in batches)*1000/len(measured), 'median_batch_wall_ms_per_sample': float(np.median(all_wall_ms)),
199
+ 'mean_prompt_tokens': float(np.mean([x['prompt_tokens'] for batch in batches for x in batch['outputs'] if x['prompt_tokens'] is not None])),
200
+ 'mean_completion_tokens': float(np.mean([x['completion_tokens'] for batch in batches for x in batch['outputs']])),
201
+ 'latency_scope': 'A800 vLLM-0.18.0 native generation latency with the frozen task prompt and structured-output schema; model loading and warmup excluded. Batch wall time includes the real request/image preparation path.',
202
+ 'flops_scope': 'This file measures latency only. Counted FLOPs are measured separately by the HF reference probe and are not claimed to be complete vLLM kernel FLOPs.',
203
+ }
204
+ args.output.write_text(json.dumps(output, ensure_ascii=False, indent=2), encoding='utf-8')
205
+ print(json.dumps({'status': 'complete', 'mean_generation_ms_per_sample': output['mean_generation_ms_per_sample'], 'mean_completion_tokens': output['mean_completion_tokens']}), flush=True)
206
+
207
+
208
+ if __name__ == '__main__':
209
+ main()
paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_router.py ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from pathlib import Path
2
+ import argparse
3
+ import json
4
+ import sys
5
+ import time
6
+ import numpy as np
7
+ import torch
8
+ from PIL import Image
9
+ from torch.utils.flop_counter import FlopCounterMode
10
+
11
+ p=argparse.ArgumentParser()
12
+ p.add_argument('--root',type=Path,required=True)
13
+ p.add_argument('--expert',required=True)
14
+ p.add_argument('--endpoint',required=True)
15
+ p.add_argument('--output',type=Path,required=True)
16
+ a=p.parse_args()
17
+ sys.path.insert(0,str(a.root/'code'))
18
+ from routing import train_router_formal as formal
19
+ from routing.router import prepare_images,load_distilled_router
20
+ from siglip_distillation.mini_siglip import MiniSiglip
21
+ from tokenizers import Tokenizer
22
+ torch.set_num_threads(4)
23
+ torch.backends.cuda.matmul.allow_tf32=False
24
+ torch.backends.mha.set_fastpath_enabled(False)
25
+ work=a.root/'outputs/router_shared_image_hidden_20260910'
26
+ run=f'shared_12_{a.expert}_{a.endpoint}'
27
+ dest=work/'runs'/run
28
+ cfg=json.loads((dest/'config.json').read_text())
29
+ assert cfg['seed']==42 and cfg['student']=='8M' and cfg['target']=='four_state'
30
+ records,data=formal.load_data(cfg,'test',torch.device('cuda'))
31
+ for key in ['hidden','confidence']:
32
+ norm=np.load(dest/(key+'_normalization.npz'))
33
+ data[key]=(data[key]-torch.from_numpy(norm['mean']).cuda())/torch.from_numpy(norm['std']).cuda()
34
+ model=formal.FeatureRouter(cfg,data['hidden'].shape[1],data['confidence'].shape[1],data['output'].shape[1]).cuda()
35
+ ckpt=torch.load(dest/'best.pt',map_location='cuda',weights_only=True)
36
+ model.load_state_dict(ckpt['state_dict'],strict=True)
37
+ model.eval().requires_grad_(False)
38
+ with torch.inference_mode():
39
+ raw=torch.cat([model(data,torch.arange(i,min(i+8192,len(records)),device='cuda')).cpu() for i in range(0,len(records),8192)]).numpy()
40
+ scores=raw[:,1]-raw[:,2]
41
+ saved=np.load(work/'test_evaluation'/run/'paired_test_arrays.npz')['scores']
42
+ reference=json.loads((work/'test_results.json').read_text())
43
+ ref=next(g for g in reference['groups'] if g['id']==run)
44
+ point=ref['best_test_point']
45
+ threshold=point['threshold']
46
+ mask=scores>=threshold if point['operator']=='ge' else scores>threshold
47
+ saved_mask=saved>=threshold if point['operator']=='ge' else saved>threshold
48
+ assert np.array_equal(mask,saved_mask), 'Recomputed selected decisions differ from main-table archive'
49
+ indices=np.random.default_rng(20260912).choice(len(records),64,replace=False)
50
+ encoder_dir=a.root/'outputs/mini_siglip_strict_20260910_300ep/8M'
51
+ checkpoint=torch.load(encoder_dir/'epoch_300.pt',map_location='cpu',weights_only=True)
52
+ encoder=MiniSiglip(**checkpoint['model_config']).cuda()
53
+ encoder.load_state_dict(checkpoint['model'],strict=True)
54
+ encoder.eval().requires_grad_(False)
55
+ tokenizer=Tokenizer.from_file(str(encoder_dir/'student_tokenizer.json'))
56
+
57
+ def prepare(i):
58
+ with Image.open(records[i]['image_path']) as im:
59
+ pixels,valid=prepare_images([[im]])
60
+ pixels=pixels.flatten(0,1).cuda()
61
+ tokens=torch.tensor([tokenizer.encode(records[i]['expression']).ids],device='cuda') if 'text' in cfg['branches'] else None
62
+ return pixels,valid.cuda(),tokens
63
+
64
+ def forward(i,prepared):
65
+ pixels,valid,tokens=prepared
66
+ with torch.autocast('cuda',dtype=torch.bfloat16):
67
+ image=encoder.encode_image(pixels).to(torch.float16).float().reshape(1,2,768)
68
+ text=encoder.encode_text(tokens).to(torch.float16).float() if tokens is not None else None
69
+ row={k:v[i:i+1] for k,v in data.items()}
70
+ row['image']=image;row['valid']=valid
71
+ if text is not None:row['text']=text
72
+ return model(row,torch.zeros(1,device='cuda',dtype=torch.long))
73
+
74
+ times=[];head_times=[];flops=[];ops=[]
75
+ with torch.inference_mode():
76
+ for i in indices[:4]:
77
+ forward(int(i),prepare(int(i)))
78
+ torch.cuda.synchronize()
79
+ for i in indices:
80
+ i=int(i)
81
+ started=time.perf_counter()
82
+ forward(i,prepare(i))
83
+ torch.cuda.synchronize()
84
+ times.append((time.perf_counter()-started)*1000)
85
+ start=torch.cuda.Event(enable_timing=True);end=torch.cuda.Event(enable_timing=True)
86
+ idx=torch.tensor([i],device='cuda')
87
+ start.record();model(data,idx);end.record();torch.cuda.synchronize()
88
+ head_times.append(start.elapsed_time(end))
89
+ for i in indices[:8]:
90
+ prepared=prepare(int(i))
91
+ with FlopCounterMode(display=False) as counter:
92
+ forward(int(i),prepared)
93
+ flops.append(counter.get_total_flops()/1e9)
94
+ ops.append({str(k):int(v) for k,v in counter.get_flop_counts().get('Global',{}).items()})
95
+ report={'status':'complete','run':run,'config':cfg,'checkpoint':str(dest/'best.pt'),
96
+ 'score':'logit_rescue-logit_harm','test_rows_recomputed':len(records),'selected_decisions_match_main':True,
97
+ 'max_score_abs_difference':float(np.max(np.abs(saved-scores))),'actual_call_fraction':float(mask.mean()),
98
+ 'sample_count':len(indices),'sample_ids':[records[int(i)]['sample_id'] for i in indices],
99
+ 'batch_size':1,'latency_ms':times,'mean_latency_ms_per_input':float(np.mean(times)),
100
+ 'head_mean_latency_ms':float(np.mean(head_times)), 'gflops_per_input':flops,'mean_gflops_per_input':float(np.mean(flops)),
101
+ 'operators':ops,'scope':'image loading, bilinear resize, tokenize when used, two image slots (including gray placeholder), active distilled encoders, feature assembly and original routing head; cached specialist features; supported FLOPs only'}
102
+ a.output.parent.mkdir(parents=True,exist_ok=True)
103
+ a.output.write_text(json.dumps(report,indent=2),encoding='utf-8')
104
+ print(json.dumps({k:v for k,v in report.items() if k in ['run','actual_call_fraction','selected_decisions_match_main','mean_latency_ms_per_input','mean_gflops_per_input']},indent=2))
paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_specialist.py ADDED
@@ -0,0 +1,258 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ from __future__ import annotations
3
+
4
+ import argparse
5
+ import importlib.util
6
+ import json
7
+ import sys
8
+ import time
9
+ from pathlib import Path
10
+
11
+ import numpy as np
12
+ import torch
13
+ from torch.utils.flop_counter import FlopCounterMode
14
+
15
+
16
+ def timed_cuda(call):
17
+ start = torch.cuda.Event(enable_timing=True)
18
+ end = torch.cuda.Event(enable_timing=True)
19
+ start.record()
20
+ value = call()
21
+ end.record()
22
+ torch.cuda.synchronize()
23
+ return float(start.elapsed_time(end)), value
24
+
25
+
26
+ def count_flops(call):
27
+ with torch.inference_mode(), FlopCounterMode(display=False) as counter:
28
+ value = call()
29
+ torch.cuda.synchronize()
30
+ global_counts = counter.get_flop_counts().get('Global', {})
31
+ return int(counter.get_total_flops()), value, {str(k): int(v) for k, v in global_counts.items()}
32
+
33
+
34
+ def instancevg_measure(root: Path, output: Path, samples: int, batch_size: int, device: str) -> None:
35
+ import mmcv
36
+ from mmcv import Config
37
+ from transformers import XLMRobertaTokenizer
38
+ from instancevg.models import build_model
39
+ from instancevg.datasets.pipelines.transforms import Normalize, Resize
40
+
41
+ torch.set_num_threads(4)
42
+ torch.backends.cuda.matmul.allow_tf32 = False
43
+ torch.backends.cudnn.allow_tf32 = False
44
+ cfg = Config.fromfile(str(root / 'code/third_party/InstanceVG/configs/gres/InstanceVG-grefcoco.py'))
45
+ cfg.model.vis_enc.pretrain = None
46
+ cfg.model.process_visual = False
47
+ model = build_model(cfg.model)
48
+ checkpoint_path = root / 'models/grefcoco/instancevg/InstanceVG-grefcoco.pth'
49
+ checkpoint = torch.load(checkpoint_path, map_location='cpu')
50
+ state = checkpoint['state_dict']
51
+ if all(key.startswith('module.') for key in state):
52
+ state = {key[7:]: value for key, value in state.items()}
53
+ model.load_state_dict(state, strict=True)
54
+ del state, checkpoint
55
+ model.eval().to(device).requires_grad_(False)
56
+ tokenizer_path = root / 'models/nlvr2/Raghavan--beit3_base_patch16_224_nlvr2/sentencepiece.bpe.model'
57
+ tokenizer = XLMRobertaTokenizer(str(tokenizer_path))
58
+ resize = Resize(img_scale=(320, 320), keep_ratio=False)
59
+ normalize = Normalize(mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375])
60
+ rows = [json.loads(line) for line in (root / 'outputs/router_full_20260910/data/instancevg/test/records.jsonl').open(encoding='utf-8')]
61
+ if samples > len(rows):
62
+ raise ValueError(f'samples {samples} exceeds {len(rows)}')
63
+ indices = np.random.default_rng(20260911).choice(len(rows), samples, replace=False)
64
+ chosen = [rows[int(index)] for index in indices]
65
+
66
+ def prepare(batch_rows):
67
+ images, metas, tokens, masks = [], [], [], []
68
+ for row in batch_rows:
69
+ image = mmcv.imread(row['image_path'])
70
+ if image.shape[:2] != (row['height'], row['width']):
71
+ raise ValueError(f'image shape mismatch for {row["sample_id"]}')
72
+ expression = row['expression']
73
+ expression = __import__('re').sub(r"([.,'!?\"()*#:;])", '', expression.lower()).replace('-', ' ').replace('/', ' ')
74
+ token_ids = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(expression))
75
+ if not token_ids:
76
+ raise ValueError(f'empty token sequence for {row["sample_id"]}')
77
+ token_ids = [tokenizer.bos_token_id] + token_ids[:48] + [tokenizer.eos_token_id]
78
+ if len(token_ids) > 50:
79
+ raise ValueError(f'token sequence exceeds fixed length for {row["sample_id"]}')
80
+ masks.append([0] * len(token_ids) + [1] * (50 - len(token_ids)))
81
+ tokens.append(token_ids + [tokenizer.pad_token_id] * (50 - len(token_ids)))
82
+ data = dict(img=image, filename=row['image_path'], img_shape=image.shape, ori_shape=image.shape,
83
+ expression=expression, empty=None, with_bbox=False, with_mask=False)
84
+ data = normalize(resize(data))
85
+ images.append(torch.from_numpy(np.ascontiguousarray(data['img'].transpose(2, 0, 1))))
86
+ metas.append({key: data[key] for key in ['filename', 'expression', 'ori_shape', 'img_shape', 'pad_shape', 'scale_factor', 'empty']})
87
+ image_tensor = torch.stack(images).to(device)
88
+ token_tensor = torch.tensor(tokens, device=device)
89
+ mask_tensor = torch.tensor(masks, device=device)
90
+
91
+ def forward():
92
+ return model(img=image_tensor, ref_expr_inds=token_tensor, text_attention_mask=mask_tensor,
93
+ img_metas=metas, return_loss=False, rescale=True, with_bbox=True, with_mask=False)
94
+
95
+ return forward
96
+
97
+ batches = [chosen[start:start + batch_size] for start in range(0, len(chosen), batch_size)]
98
+ prepared = [prepare(batch) for batch in batches]
99
+ for forward in prepared[:2]:
100
+ forward()
101
+ torch.cuda.synchronize()
102
+ forward_ms = []
103
+ end_to_end_ms = []
104
+ for batch in batches:
105
+ forward = prepare(batch)
106
+ elapsed, _ = timed_cuda(forward)
107
+ forward_ms.append(elapsed / len(batch))
108
+ started = time.perf_counter()
109
+ forward = prepare(batch)
110
+ forward()
111
+ torch.cuda.synchronize()
112
+ end_to_end_ms.append((time.perf_counter() - started) * 1000 / len(batch))
113
+ flop_rows = []
114
+ for batch in batches[:min(4, len(batches))]:
115
+ forward = prepare(batch)
116
+ total, _, operators = count_flops(forward)
117
+ flop_rows.append({'batch_size': len(batch), 'flops_counted': total, 'gflops_counted_per_sample': total / len(batch) / 1e9, 'operators': operators})
118
+ output.write_text(json.dumps({
119
+ 'component': 'specialist', 'expert': 'instancevg', 'task': 'grefcoco', 'device': device,
120
+ 'checkpoint': str(checkpoint_path), 'sample_count': len(chosen), 'batch_size': batch_size,
121
+ 'sample_ids': [row['sample_id'] for row in chosen],
122
+ 'model_load_seconds_excluded_from_latency': True,
123
+ 'forward_latency_ms_per_sample': forward_ms,
124
+ 'end_to_end_latency_ms_per_sample': end_to_end_ms,
125
+ 'mean_forward_latency_ms_per_sample': float(np.mean(forward_ms)),
126
+ 'median_forward_latency_ms_per_sample': float(np.median(forward_ms)),
127
+ 'mean_end_to_end_latency_ms_per_sample': float(np.mean(end_to_end_ms)),
128
+ 'median_end_to_end_latency_ms_per_sample': float(np.median(end_to_end_ms)),
129
+ 'flop_profiles': flop_rows,
130
+ 'mean_counted_gflops_per_sample': float(np.mean([row['gflops_counted_per_sample'] for row in flop_rows])),
131
+ 'flops_scope': 'GPU FlopCounterMode dense products, convolutions and supported attention; multiply-add counted as 2 FLOPs. Elementwise, normalization, custom ops and postprocessing may be missing.',
132
+ 'latency_scope': 'A800 wall-clock CUDA forward latency and end-to-end latency including image read/preprocess, excluding model load; batch processing uses the recorded batch size.',
133
+ 'completed': True,
134
+ }, ensure_ascii=False, indent=2))
135
+
136
+
137
+ def yolo_measure(root: Path, output: Path, samples: int, batch_size: int, device: str) -> None:
138
+ from ultralytics import YOLO
139
+
140
+ torch.set_num_threads(4)
141
+ torch.backends.cuda.matmul.allow_tf32 = False
142
+ torch.backends.cudnn.allow_tf32 = False
143
+ checkpoint_path = root / 'models/construction/yolo26x.pt'
144
+ model = YOLO(str(checkpoint_path))
145
+ rows = [json.loads(line) for line in (root / 'outputs/router_full_20260910/data/yolo26x/test/records.jsonl').open(encoding='utf-8')]
146
+ if samples > len(rows):
147
+ raise ValueError(f'samples {samples} exceeds {len(rows)}')
148
+ indices = np.random.default_rng(20260911).choice(len(rows), samples, replace=False)
149
+ chosen = [rows[int(index)] for index in indices]
150
+ batches = [chosen[start:start + batch_size] for start in range(0, len(chosen), batch_size)]
151
+
152
+ def predict(batch_rows):
153
+ return model.predict(source=[row['image_path'] for row in batch_rows], imgsz=640, batch=len(batch_rows),
154
+ device=device, half=False, conf=0.001, iou=0.7, max_det=300, save=False, verbose=False)
155
+
156
+ for batch in batches[:2]:
157
+ predict(batch)
158
+ torch.cuda.synchronize()
159
+ forward_ms, end_to_end_ms = [], []
160
+ for batch in batches:
161
+ elapsed, _ = timed_cuda(lambda batch=batch: predict(batch))
162
+ forward_ms.append(elapsed / len(batch))
163
+ started = time.perf_counter()
164
+ predict(batch)
165
+ torch.cuda.synchronize()
166
+ end_to_end_ms.append((time.perf_counter() - started) * 1000 / len(batch))
167
+ flop_rows = []
168
+ for batch in batches[:min(4, len(batches))]:
169
+ total, _, operators = count_flops(lambda batch=batch: predict(batch))
170
+ flop_rows.append({'batch_size': len(batch), 'flops_counted': total, 'gflops_counted_per_sample': total / len(batch) / 1e9, 'operators': operators})
171
+ output.write_text(json.dumps({
172
+ 'component': 'specialist', 'expert': 'yolo26x', 'task': 'construction', 'device': device,
173
+ 'checkpoint': str(checkpoint_path), 'sample_count': len(chosen), 'batch_size': batch_size,
174
+ 'sample_ids': [row['sample_id'] for row in chosen],
175
+ 'model_load_seconds_excluded_from_latency': True,
176
+ 'forward_latency_ms_per_sample': forward_ms,
177
+ 'end_to_end_latency_ms_per_sample': end_to_end_ms,
178
+ 'mean_forward_latency_ms_per_sample': float(np.mean(forward_ms)),
179
+ 'median_forward_latency_ms_per_sample': float(np.median(forward_ms)),
180
+ 'mean_end_to_end_latency_ms_per_sample': float(np.mean(end_to_end_ms)),
181
+ 'median_end_to_end_latency_ms_per_sample': float(np.median(end_to_end_ms)),
182
+ 'flop_profiles': flop_rows,
183
+ 'mean_counted_gflops_per_sample': float(np.mean([row['gflops_counted_per_sample'] for row in flop_rows])),
184
+ 'flops_scope': 'GPU FlopCounterMode around the real Ultralytics predict path; multiply-add counted as 2 FLOPs. Elementwise, postprocessing and unsupported custom ops may be missing.',
185
+ 'latency_scope': 'A800 wall-clock latency for real Ultralytics prediction including image loading/preprocess, forward and result postprocessing, excluding model load; batch processing uses the recorded batch size.',
186
+ 'completed': True,
187
+ }, ensure_ascii=False, indent=2))
188
+
189
+
190
+ def router_measure(root: Path, output: Path, expert: str, endpoint: str, condition: str, seed: int, samples: int, batch_size: int, device: str) -> None:
191
+ work = root / 'outputs/router_method_improvement_20260911_r1/reproducible'
192
+ sys.path.insert(0, str(work))
193
+ common = __import__('method_improvement_common')
194
+ spec = json.loads((root / 'outputs/router_method_improvement_20260911_r1/method_improvement_spec.json').read_text(encoding='utf-8'))
195
+ trajectory_id = f'{expert}_{endpoint}_seed{seed}'
196
+ cfg = common.make_config(spec, expert, endpoint, seed, trajectory_id)
197
+ rows, data = common.load_split(root, spec, cfg, 'test', device)
198
+ checkpoint_path = root / f'outputs/router_method_improvement_20260911_r1/conditions/{condition}_{expert}_{endpoint}_seed{seed}/best.pt'
199
+ checkpoint = torch.load(checkpoint_path, map_location=device, weights_only=True)
200
+ model = common.FeatureRouter(cfg, data['hidden'].shape[1], data['confidence'].shape[1], data['output'].shape[1]).to(device)
201
+ model.load_state_dict(checkpoint['state_dict'], strict=True)
202
+ model.eval().requires_grad_(False)
203
+ if samples > len(rows):
204
+ raise ValueError(f'samples {samples} exceeds {len(rows)}')
205
+ indices = np.random.default_rng(20260911).choice(len(rows), samples, replace=False)
206
+ chunks = [torch.as_tensor(indices[start:start + batch_size], device=device, dtype=torch.long) for start in range(0, samples, batch_size)]
207
+ for index in chunks[:2]:
208
+ model(data, index)
209
+ torch.cuda.synchronize()
210
+ latency = []
211
+ for index in chunks:
212
+ elapsed, _ = timed_cuda(lambda index=index: model(data, index))
213
+ latency.append(elapsed / len(index))
214
+ index = chunks[0]
215
+ total, _, operators = count_flops(lambda: model(data, index))
216
+ output.write_text(json.dumps({
217
+ 'component': 'router', 'expert': expert, 'task': cfg['task'], 'endpoint': endpoint,
218
+ 'condition': condition, 'seed': seed, 'device': device, 'checkpoint': str(checkpoint_path),
219
+ 'sample_count': samples, 'batch_size': batch_size, 'row_count_loaded': len(rows),
220
+ 'mean_latency_ms_per_sample': float(np.mean(latency)), 'median_latency_ms_per_sample': float(np.median(latency)),
221
+ 'latency_ms_per_sample': latency, 'flops_counted_per_batch': total,
222
+ 'gflops_counted_per_sample': total / len(index) / 1e9, 'operators': operators,
223
+ 'flops_scope': 'GPU FlopCounterMode for the router forward; multiply-add counted as 2 FLOPs.',
224
+ 'latency_scope': 'A800 CUDA forward latency on cached router features, excluding feature loading and model load.',
225
+ 'completed': True,
226
+ }, ensure_ascii=False, indent=2))
227
+
228
+
229
+ def main() -> None:
230
+ parser = argparse.ArgumentParser()
231
+ parser.add_argument('--root', type=Path, required=True)
232
+ parser.add_argument('--output', type=Path, required=True)
233
+ parser.add_argument('--component', choices=['specialist', 'router'], required=True)
234
+ parser.add_argument('--expert', choices=['instancevg', 'yolo26x'], required=True)
235
+ parser.add_argument('--endpoint', choices=['MiniCPM', 'Qwen3.8'])
236
+ parser.add_argument('--condition', default='both')
237
+ parser.add_argument('--seed', type=int, default=42)
238
+ parser.add_argument('--samples', type=int, default=64)
239
+ parser.add_argument('--batch-size', type=int, default=8)
240
+ parser.add_argument('--device', default='cuda:0')
241
+ args = parser.parse_args()
242
+ if args.output.exists():
243
+ raise FileExistsError(args.output)
244
+ args.output.parent.mkdir(parents=True, exist_ok=True)
245
+ if not torch.cuda.is_available():
246
+ raise RuntimeError('CUDA is unavailable')
247
+ if args.component == 'specialist' and args.expert == 'instancevg':
248
+ instancevg_measure(args.root, args.output, args.samples, args.batch_size, args.device)
249
+ elif args.component == 'specialist':
250
+ yolo_measure(args.root, args.output, args.samples, args.batch_size, args.device)
251
+ else:
252
+ if args.endpoint is None:
253
+ raise ValueError('--endpoint is required for router')
254
+ router_measure(args.root, args.output, args.expert, args.endpoint, args.condition, args.seed, args.samples, args.batch_size, args.device)
255
+
256
+
257
+ if __name__ == '__main__':
258
+ main()
paper_results/data/main_table_cost_qwen_20260912/reproducible/prepare_output.py ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from pathlib import Path
2
+ import json
3
+ import shutil
4
+ root=Path(__file__).resolve().parents[2]
5
+ out=root/'dse_results/main_table_cost_20260912'
6
+ out.mkdir(exist_ok=True)
7
+ (out/'components').mkdir(exist_ok=True)
8
+ source=root/'paper_materials/draft_20260912_revision19/data/results.json'
9
+ data=json.loads(source.read_text(encoding='utf-8'))
10
+ groups=[g for g in data['groups'] if g['key'] in [['grefcoco','instancevg','MiniCPM'],['construction','yolo26x','Qwen3.8']]]
11
+ assert len(groups)==2
12
+ (out/'main_table_results.json').write_text(json.dumps({'source':str(source),'protocol':data['protocol'],'groups':groups},indent=2),encoding='utf-8')
13
+ scripts=out/'reproducible'
14
+ scripts.mkdir(exist_ok=True)
15
+ for p in Path(__file__).resolve().parent.glob('*.py'):
16
+ shutil.copy2(p,scripts/p.name)
17
+ for p in Path(__file__).resolve().parent.glob('*.sh'):
18
+ shutil.copy2(p,scripts/p.name)
19
+ print(out)
paper_results/data/main_table_cost_qwen_20260912/reproducible/read_main.py ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ from pathlib import Path
2
+ import json
3
+ root=Path(__file__).resolve().parents[2]
4
+ data=json.loads((root/'paper_materials/draft_20260912_revision19/data/results.json').read_text(encoding='utf-8'))
5
+ print('TOP',list(data))
6
+ for g in data['groups']:
7
+ if g['key'] in [['grefcoco','instancevg','MiniCPM'],['construction','yolo26x','Qwen3.8']]:
8
+ print(json.dumps({k:v for k,v in g.items() if k!='methods'},ensure_ascii=False)[:6000])
9
+ for m,v in g['methods'].items():
10
+ print(m, {k:v1 for k,v1 in v.items() if 'curve' not in k})
paper_results/data/main_table_cost_qwen_20260912/reproducible/setup_local.py ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ from pathlib import Path
2
+ import shutil
3
+ root=Path(__file__).resolve().parents[2]
4
+ dest=Path(__file__).resolve().parent
5
+ for source,target in [('measure_large_vllm_runtime_20260911.py','measure_large.py'),('measure_large_flops_reference_20260911.py','measure_flops.py'),('measure_expert_router_runtime_20260911.py','measure_specialist.py')]:
6
+ shutil.copy2(root/'tmp'/source,dest/target)
paper_results/data/main_table_cost_qwen_20260912/reproducible/switch_qwen.py ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from pathlib import Path
2
+ import json
3
+ import shutil
4
+ root=Path(__file__).resolve().parents[2]
5
+ old=root/'dse_results/main_table_cost_20260912'
6
+ out=root/'dse_results/main_table_cost_qwen_20260912'
7
+ out.mkdir(exist_ok=True)
8
+ (out/'components').mkdir(exist_ok=True)
9
+ (out/'reproducible').mkdir(exist_ok=True)
10
+ (out/'logs').mkdir(exist_ok=True)
11
+ source=root/'paper_materials/draft_20260912_revision19/data/results.json'
12
+ data=json.loads(source.read_text(encoding='utf-8'))
13
+ groups=[next(g for g in data['groups'] if g['key']==key) for key in [['grefcoco','instancevg','Qwen3.8'],['construction','yolo26x','Qwen3.8']]]
14
+ (out/'main_table_results.json').write_text(json.dumps({'source':str(source),'protocol':data['protocol'],'groups':groups},indent=2),encoding='utf-8')
15
+ for name in ['instancevg_specialist.json','yolo26x_specialist.json','yolo26x_router.json','qwen_latency.json','qwen_flops.json']:
16
+ shutil.copy2(old/'components'/name,out/'components'/name)
17
+ for name in ['build_figures.py','verify_and_report.py','latex_includes.tex']:
18
+ text=(old/name).read_text(encoding='utf-8').replace("['minicpm','qwen']","['qwen_gref','qwen']").replace('gRefCOCO / M','gRefCOCO / Q').replace('InstanceVG--MiniCPM','InstanceVG--Qwen').replace('main_table_cost_20260912','main_table_cost_qwen_20260912')
19
+ (out/name).write_text(text,encoding='utf-8')
20
+ for p in (old/'reproducible').iterdir():
21
+ if p.is_file(): shutil.copy2(p,out/'reproducible'/p.name)
22
+ for g in groups:
23
+ print(g['key'],g['run'],{m:g['methods'][m]['test_point'] for m in ['learned','confidence']})
paper_results/data/main_table_cost_qwen_20260912/selected_points.csv ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ pair,method,nominal_validation_budget,actual_test_call_fraction,test_metric,estimated_gflops_per_input,estimated_time_ms_per_input
2
+ grefcoco|instancevg|Qwen3.8,learned,0.56,0.5355755324277571,0.7528287440093016,17446.303931938,2243.6672827746106
3
+ grefcoco|instancevg|Qwen3.8,confidence,0.71,0.6838612710206166,0.7243286163967898,22269.96488623913,2830.3683522642846
4
+ construction|yolo26x|Qwen3.8,learned,0.5,0.4876830892143808,0.29875800068449215,47383.321370531434,4861.824738942901
5
+ construction|yolo26x|Qwen3.8,confidence,0.93,0.9221038615179761,0.2597490004762105,89464.5250591521,9103.651026133362
paper_results/data/main_table_cost_qwen_20260912/verify_and_report.py ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from pathlib import Path
2
+ import csv
3
+ import json
4
+ import fitz
5
+
6
+ root = Path(__file__).resolve().parent
7
+ summary = json.loads((root/'cost_summary.json').read_text())
8
+ groups = json.loads((root/'main_table_results.json').read_text())['groups']
9
+ rows = list(csv.DictReader((root/'full_cost_performance_curves.csv').open()))
10
+ assert len(rows) == 404
11
+ lines = ['# 主表方法:成本—性能曲线', '', '## 1 做了什么', '',
12
+ '- 使用主表原检查点、种子42及原验证集阈值;未重新训练,也未改用后续改进方法。',
13
+ '- gRefCOCO 改为 InstanceVG + Qwen,重新测量该任务上的 Qwen 成本及对应路由器;Construction 和 InstanceVG 专家复用上一轮同口径实测。',
14
+ '- 在 A800 的0、1号卡测量;每个大模型预热2张、正式测量64张,每次1张。专家和路由部分各测64张。',
15
+ '- 重算原路由头,并核对全测试集的选定阈值调用决定与主表完全一致。性能数值沿用主表固定的预测记录。',
16
+ '- 只画学习路由与置信度路由;共404个曲线点。另提供单栏四面板紧凑版。', '',
17
+ '## 2 为什么做', '', '让成本曲线与主实验比较同一个方法,避免混入后续改进结果;补计蒸馏编码器及路由头开销。', '',
18
+ '## 3 做完有什么效果', '', '| 数据集 | 方法 | 测试性能(%) | 调用率(%) | 估算GFLOPs/输入 | 估算毫秒/输入 |',
19
+ '|---|---|---:|---:|---:|---:|']
20
+ comparisons = []
21
+ for group in groups:
22
+ key = '|'.join(group['key'])
23
+ chosen = {p['method']:p for p in summary['selected_points'] if p['pair']==key}
24
+ for method,label in [('learned','学习路由'),('confidence','置信度路由')]:
25
+ p = chosen[method]
26
+ original = group['methods'][method]['test_point']
27
+ assert p['test_metric'] == original['metric']
28
+ assert p['actual_test_call_fraction'] == original['actual_fraction']
29
+ lines.append(f"| {group['key'][0]} | {label} | {100*p['test_metric']:.2f} | {100*p['actual_test_call_fraction']:.2f} | {p['estimated_gflops_per_input']:,.1f} | {p['estimated_time_ms_per_input']:,.1f} |")
30
+ a,b = chosen['learned'],chosen['confidence']
31
+ comparisons.append(f"{group['key'][0]}:相对置信度路由,性能提高 {100*(a['test_metric']-b['test_metric']):.2f} 个百分点,估算 FLOPs 减少 {100*(1-a['estimated_gflops_per_input']/b['estimated_gflops_per_input']):.1f}%,估算耗时减少 {100*(1-a['estimated_time_ms_per_input']/b['estimated_time_ms_per_input']):.1f}%。")
32
+ for comparison in comparisons:
33
+ lines += ['', comparison]
34
+ lines += ['']
35
+ lines += ['### 数值口径', '',
36
+ '- 每个曲线点使用验证集确定的阈值,横轴按该阈值在测试集上的实际调用率计算;圆点对应主表选定点,没有按测试集重新选点。',
37
+ '- 置信度路由成本 = 专家成本 + 实际调用率 × 大模型平均成本。学习路由再加编码器和路由头成本。',
38
+ '- 耗时为分项实测后相加的串行估算,并非各策略整套系统的直接端到端计时。加载模型和预热不计入。',
39
+ '- FLOPs 按乘加计2次统计支持的算子;大模型用8个输入估计预填充成本(含视觉编码),再加平均输出长度乘单词元解码成本。专家统计4个输入,编码器与路由头统计8个输入。',
40
+ '- 各策略共用大模型平均单次成本,尚未测量所选样本的长度差异对成本的影响;专家隐藏特征来自缓存,其额外提取开销未单独测量。置信度阈值判断开销未单独计时。',
41
+ '- 本轮单张测量与上一轮批量测量口径不同,绝对耗时不宜直接比较。', '',
42
+ '- YOLO 使用权重记录的依赖版本8.3.222,安装在任务临时目录;结束时0、1号卡显存均为0,未操作2、3号卡服务。', '',
43
+ '## 4 对应文件在哪', '',
44
+ '- `cost_compact.pdf`:推荐插入正文的单栏四面板图,宽3.44英寸、高2.05英寸。',
45
+ '- `flops_performance.pdf`、`latency_performance.pdf`:两张独立双面板图。',
46
+ '- `full_cost_performance_curves.csv`:全部404个点;`selected_points.csv`:主表4个运行点。',
47
+ '- `components/`:8份实测记录,含原始回答、错误记录及路由核验。',
48
+ '- `build_figures.py`:绘图及成本计算;`reproducible/`:远端测量脚本;`latex_includes.tex`:插入代码及图注。',
49
+ '- 远端结果:`/mnt/HithinkOmniSSD/user_workspace/caisihang/research/ICASSP_ModelCollaboration/outputs/main_table_cost_qwen_20260912`。', '',
50
+ '### 检查记录', '', '已核对404个曲线点、4个主表运行点、两组全测试集调用决定及单位。PDF文字为嵌入字体,另渲染检查排版。']
51
+ (root/'成本性能报告.md').write_text('\n'.join(lines)+'\n',encoding='utf-8')
52
+ for stem in ['cost_compact','flops_performance','latency_performance']:
53
+ doc = fitz.open(root/(stem+'.pdf'))
54
+ assert len(doc)==1
55
+ assert all(font[1] != 'n/a' for font in doc[0].get_fonts())
56
+ doc[0].get_pixmap(matrix=fitz.Matrix(4,4)).save(root/(stem+'_verified.png'))
57
+ print('\n'.join(lines))
paper_results/data/main_table_cost_qwen_20260912/成本性能报告.md ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # 主表方法:成本—性能曲线
2
+
3
+ ## 1 做了什么
4
+
5
+ - 使用主表原检查点、种子42及原验证集阈值;未重新训练,也未改用后续改进方法。
6
+ - gRefCOCO 改为 InstanceVG + Qwen,重新测量该任务上的 Qwen 成本及对应路由器;Construction 和 InstanceVG 专家复用上一轮同口径实测。
7
+ - 在 A800 的0、1号卡测量;每个大模型预热2张、正式测量64张,每次1张。专家和路由部分各测64张。
8
+ - 重算原路由头,并核对全测试集的选定阈值调用决定与主表完全一致。性能数值沿用主表固定的预测记录。
9
+ - 只画学习路由与置信度路由;共404个曲线点。另提供单栏四面板紧凑版。
10
+
11
+ ## 2 为什么做
12
+
13
+ 让成本曲线与主实验比较同一个方法,避免混入后续改进结果;补计蒸馏编码器及路由头开销。
14
+
15
+ ## 3 做完有什么效果
16
+
17
+ | 数据集 | 方法 | 测试性能(%) | 调用率(%) | 估算GFLOPs/输入 | 估算毫秒/输入 |
18
+ |---|---|---:|---:|---:|---:|
19
+ | grefcoco | 学习路由 | 75.28 | 53.56 | 17,446.3 | 2,243.7 |
20
+ | grefcoco | 置信度路由 | 72.43 | 68.39 | 22,270.0 | 2,830.4 |
21
+ | construction | 学习路由 | 29.88 | 48.77 | 47,383.3 | 4,861.8 |
22
+ | construction | 置信度路由 | 25.97 | 92.21 | 89,464.5 | 9,103.7 |
23
+
24
+ grefcoco:相对置信度路由,性能提高 2.85 个百分点,估算 FLOPs 减少 21.7%,估算耗时减少 20.7%。
25
+
26
+ construction:相对置信度路由,性能提高 3.90 个百分点,估算 FLOPs 减少 47.0%,估算耗时减少 46.6%。
27
+
28
+ ### 数值口径
29
+
30
+ - 每个曲线点使用验证集确定的阈值,横轴按该阈值在测试集上的实际调用率计算;圆点对应主表选定点,没有按测试集重新选点。
31
+ - 置信度路由成本 = 专家成本 + 实际调用率 × 大模型平均成本。学习路由再加编码器和路由头成本。
32
+ - 耗时为分项实测后相加的串行估算,并非各策略整套系统的直接端到端计时。加载模型和预热不计入。
33
+ - FLOPs 按乘加计2次统计支持的算子;大模型用8个输入估计预填充成本(含视觉编码),再加平均输出长度乘单词元解码成本。专家统计4个输入,编码器与路由头统计8个输入。
34
+ - 各策略共用大模型平均单次成本,尚未测量所选样本的长度差异对成本的影响;专家隐藏特征来自缓存,其额外提取开销未单独测量。置信度阈值判断开销未单独计时。
35
+ - 本轮单张测量与上一轮批量测量口径不同,绝对耗时不宜直接比较。
36
+
37
+ - YOLO 使用权重记录的依赖版本8.3.222,安装在任务临时目录;结束时0、1号卡显存均为0,未操作2、3号卡服务。
38
+
39
+ ## 4 对应文件在哪
40
+
41
+ - `cost_compact.pdf`:推荐插入正文的单栏四面板图,宽3.44英寸、高2.05英寸。
42
+ - `flops_performance.pdf`、`latency_performance.pdf`:两张独立双面板图。
43
+ - `full_cost_performance_curves.csv`:全部404个点;`selected_points.csv`:主表4个运行点。
44
+ - `components/`:8份实测记录,含原始回答、错误记录及路由核验。
45
+ - `build_figures.py`:绘图及成本计算;`reproducible/`:远端测量脚本;`latex_includes.tex`:插入代码及图注。
46
+ - 远端结果:`/mnt/HithinkOmniSSD/user_workspace/caisihang/research/ICASSP_ModelCollaboration/outputs/main_table_cost_qwen_20260912`。
47
+
48
+ ### 检查记录
49
+
50
+ 已核对404个曲线点、4个主表运行点、两组全测试集调用决定及单位。PDF文字为嵌入字体,另渲染检查排版。
paper_results/data/method_improvement_spec.json ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "router_method_improvement_20260911",
3
+ "remote_root": "/mnt/HithinkOmniSSD/user_workspace/caisihang/research/ICASSP_ModelCollaboration",
4
+ "remote_work": "outputs/router_method_improvement_20260911",
5
+ "data_work": "outputs/router_full_20260910",
6
+ "student": "8M",
7
+ "student_checkpoint": "outputs/mini_siglip_strict_20260910_300ep/8M/epoch_300.pt",
8
+ "fixed": {
9
+ "fraction": 1.0,
10
+ "freeze_encoder": true,
11
+ "fusion": "concat",
12
+ "harm_cost": 1.0,
13
+ "batch_size": 256,
14
+ "weight_decay": 0.01,
15
+ "max_epochs": 50,
16
+ "patience": 8,
17
+ "representation_dim": 128,
18
+ "lr": 0.003,
19
+ "target": "four_state",
20
+ "strategy": "natural",
21
+ "hidden_percent": 90,
22
+ "cpu_threads": 1
23
+ },
24
+ "seeds": [42, 43, 44],
25
+ "experts": {
26
+ "cub": {
27
+ "task": "cub",
28
+ "hidden_layer": "residual_blocks__layer_15",
29
+ "hidden_depth": 16,
30
+ "branches": ["image", "hidden", "confidence", "output"],
31
+ "confidence_column": "max_probability",
32
+ "confidence_valid_column": "max_probability_valid",
33
+ "expected_rows": {"train": 4792, "val": 1200, "test": 5794}
34
+ },
35
+ "glsim": {
36
+ "task": "cub",
37
+ "hidden_layer": "global_local_encoder__layer_11",
38
+ "hidden_depth": 12,
39
+ "branches": ["image", "hidden", "confidence", "output"],
40
+ "confidence_column": "max_probability",
41
+ "confidence_valid_column": "max_probability_valid",
42
+ "expected_rows": {"train": 4792, "val": 1200, "test": 5794}
43
+ },
44
+ "groundingdino": {
45
+ "task": "grefcoco",
46
+ "hidden_layer": "multimodal_encoder__layer_06",
47
+ "hidden_depth": 6,
48
+ "branches": ["image", "text", "hidden", "confidence", "output"],
49
+ "confidence_column": "max_score",
50
+ "confidence_valid_column": "max_score_valid",
51
+ "expected_rows": {"train": 209344, "val": 14229, "test": 35263}
52
+ },
53
+ "instancevg": {
54
+ "task": "grefcoco",
55
+ "hidden_layer": "joint_encoder__layer_11",
56
+ "hidden_depth": 12,
57
+ "branches": ["image", "text", "hidden", "confidence", "output"],
58
+ "confidence_column": "max_score",
59
+ "confidence_valid_column": "max_score_valid",
60
+ "expected_rows": {"train": 209344, "val": 14229, "test": 35263}
61
+ },
62
+ "nlvr2": {
63
+ "task": "nlvr2",
64
+ "hidden_layer": "multimodal_encoder__layer_11",
65
+ "hidden_depth": 12,
66
+ "branches": ["image", "text", "hidden", "confidence", "output"],
67
+ "confidence_column": "max_probability",
68
+ "confidence_valid_column": "max_probability_valid",
69
+ "expected_rows": {"train": 85042, "val": 6982, "test": 13937}
70
+ },
71
+ "vilt": {
72
+ "task": "nlvr2",
73
+ "hidden_layer": "ordered_image_text__layer_11",
74
+ "hidden_depth": 12,
75
+ "branches": ["image", "text", "hidden", "confidence", "output"],
76
+ "confidence_column": "max_probability",
77
+ "confidence_valid_column": "max_probability_valid",
78
+ "expected_rows": {"train": 85042, "val": 6982, "test": 13937}
79
+ },
80
+ "yolo26x": {
81
+ "task": "construction",
82
+ "hidden_layer": "feature_blocks__layer_16",
83
+ "hidden_depth": 17,
84
+ "branches": ["image", "hidden", "confidence", "output"],
85
+ "confidence_column": "max_score",
86
+ "confidence_valid_column": "max_score_valid",
87
+ "expected_rows": {"train": 6308, "val": 701, "test": 3004}
88
+ },
89
+ "rtdetr_x_fp32": {
90
+ "task": "construction",
91
+ "hidden_layer": "query_decoder__layer_06",
92
+ "hidden_depth": 6,
93
+ "branches": ["image", "hidden", "confidence", "output"],
94
+ "confidence_column": "max_score",
95
+ "confidence_valid_column": "max_score_valid",
96
+ "expected_rows": {"train": 6308, "val": 701, "test": 3004}
97
+ }
98
+ },
99
+ "conditions": [
100
+ {"name": "original", "selection_mode": "a25", "score_mode": "s_log"},
101
+ {"name": "score_gain", "selection_mode": "a25", "score_mode": "s_gain"},
102
+ {"name": "checkpoint_perf", "selection_mode": "max_task_performance", "score_mode": "s_log"},
103
+ {"name": "both", "selection_mode": "max_task_performance", "score_mode": "s_gain"}
104
+ ],
105
+ "score_only_control": {"name": "score_only_gain", "source_condition": "original", "score_mode": "s_gain"},
106
+ "budgets": {
107
+ "a25": [0.0, 0.05, 0.10, 0.15, 0.20, 0.25],
108
+ "historical_threshold": [0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.10, 0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.20, 0.21, 0.22, 0.23, 0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.30, 0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39, 0.40, 0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.49, 0.50, 0.51, 0.52, 0.53, 0.54, 0.55, 0.56, 0.57, 0.58, 0.59, 0.60, 0.61, 0.62, 0.63, 0.64, 0.65, 0.66, 0.67, 0.68, 0.69, 0.70, 0.71, 0.72, 0.73, 0.74, 0.75, 0.76, 0.77, 0.78, 0.79, 0.80, 0.81, 0.82, 0.83, 0.84, 0.85, 0.86, 0.87, 0.88, 0.89, 0.90, 0.91, 0.92, 0.93, 0.94, 0.95, 0.96, 0.97, 0.98, 0.99],
109
+ "endpoint": [0.0, 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.10, 0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.20, 0.21, 0.22, 0.23, 0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.30, 0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39, 0.40, 0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.49, 0.50, 0.51, 0.52, 0.53, 0.54, 0.55, 0.56, 0.57, 0.58, 0.59, 0.60, 0.61, 0.62, 0.63, 0.64, 0.65, 0.66, 0.67, 0.68, 0.69, 0.70, 0.71, 0.72, 0.73, 0.74, 0.75, 0.76, 0.77, 0.78, 0.79, 0.80, 0.81, 0.82, 0.83, 0.84, 0.85, 0.86, 0.87, 0.88, 0.89, 0.90, 0.91, 0.92, 0.93, 0.94, 0.95, 0.96, 0.97, 0.98, 0.99, 1.0],
110
+ "batch_diagnostic": [0.0, 0.05, 0.10, 0.15, 0.20, 0.25, 0.50, 0.75, 1.0]
111
+ },
112
+ "fusion_alphas": [0.0, 0.25, 0.5, 0.75, 1.0],
113
+ "tie_epsilon": 1e-12
114
+ }
paper_results/data/operation_measurements.json ADDED
@@ -0,0 +1,129 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "small_models": [
3
+ {
4
+ "model": "ResNet",
5
+ "expert": "cub",
6
+ "samples": 32,
7
+ "mean_gflops_counted": 8.175091712,
8
+ "min_gflops_counted": 8.175091712,
9
+ "max_gflops_counted": 8.175091712,
10
+ "operator_inventory_source": "cub_operator_check.json",
11
+ "operator_check_samples": 1,
12
+ "uncovered_spatial_sampling_seen": false
13
+ },
14
+ {
15
+ "model": "YOLO26x",
16
+ "expert": "yolo26x",
17
+ "samples": 32,
18
+ "mean_gflops_counted": 143.2219032,
19
+ "min_gflops_counted": 116.2238976,
20
+ "max_gflops_counted": 193.92768,
21
+ "operator_inventory_source": "yolo26x_operator_check.json",
22
+ "operator_check_samples": 1,
23
+ "uncovered_spatial_sampling_seen": false
24
+ },
25
+ {
26
+ "model": "RT-DETR-X",
27
+ "expert": "rtdetr_x_fp32",
28
+ "samples": 32,
29
+ "mean_gflops_counted": 224.6444544,
30
+ "min_gflops_counted": 224.6444544,
31
+ "max_gflops_counted": 224.6444544,
32
+ "operator_inventory_source": "rtdetr_x_fp32_operator_check.json",
33
+ "operator_check_samples": 1,
34
+ "uncovered_spatial_sampling_seen": true
35
+ },
36
+ {
37
+ "model": "ViLT",
38
+ "expert": "vilt",
39
+ "samples": 32,
40
+ "mean_gflops_counted": 84.895706112,
41
+ "min_gflops_counted": 57.611655168,
42
+ "max_gflops_counted": 120.690972672,
43
+ "operator_inventory_source": "vilt_operator_check.json",
44
+ "operator_check_samples": 1,
45
+ "uncovered_spatial_sampling_seen": false
46
+ },
47
+ {
48
+ "model": "GroundingDINO",
49
+ "expert": "groundingdino",
50
+ "samples": 32,
51
+ "mean_gflops_counted": 668.259162288,
52
+ "min_gflops_counted": 499.212779776,
53
+ "max_gflops_counted": 820.17636864,
54
+ "operator_inventory_source": "groundingdino_operator_check.json",
55
+ "operator_check_samples": 1,
56
+ "uncovered_spatial_sampling_seen": true
57
+ },
58
+ {
59
+ "model": "BEiT3",
60
+ "expert": "nlvr2",
61
+ "samples": 32,
62
+ "mean_gflops_counted": 76.799401728,
63
+ "min_gflops_counted": 73.586743296,
64
+ "max_gflops_counted": 83.259561984,
65
+ "operator_inventory_source": "nlvr2_operator_check.json",
66
+ "operator_check_samples": 1,
67
+ "uncovered_spatial_sampling_seen": false
68
+ },
69
+ {
70
+ "model": "GLSim",
71
+ "expert": "glsim",
72
+ "samples": 32,
73
+ "mean_gflops_counted": 70.280871936,
74
+ "min_gflops_counted": 70.280871936,
75
+ "max_gflops_counted": 70.280871936,
76
+ "operator_inventory_source": "glsim_ops.json",
77
+ "operator_check_samples": 32,
78
+ "uncovered_spatial_sampling_seen": false
79
+ }
80
+ ],
81
+ "large_input_cases": 16,
82
+ "large_input_check_complete": true,
83
+ "large_model_probes": [
84
+ {
85
+ "model": "MiniCPM",
86
+ "sample_id": "cub-test-00517",
87
+ "task": "cub",
88
+ "prompt_tokens": 1996,
89
+ "prefill_gflops_counted": 33243.606087168,
90
+ "one_known_token_gflops_counted": 16.31253312
91
+ },
92
+ {
93
+ "model": "Qwen3.8",
94
+ "sample_id": "cub-test-00517",
95
+ "task": "cub",
96
+ "prompt_tokens": 4864,
97
+ "prefill_gflops_counted": 249018.23668224,
98
+ "one_known_token_gflops_counted": 53.164769472
99
+ }
100
+ ],
101
+ "large_gpu_probe_status": {
102
+ "stage": "finished",
103
+ "occupancy_original_pid": 157057,
104
+ "stopped_process_ids": [
105
+ 157068,
106
+ 157342,
107
+ 157343,
108
+ 157587,
109
+ 157588,
110
+ 157589,
111
+ 157590,
112
+ 157057
113
+ ],
114
+ "started_at": 1789028871.3039298,
115
+ "measurement_exit_code": 0,
116
+ "occupancy_restored": true,
117
+ "measurement_pid": 159330,
118
+ "occupancy_new_pid": 159642,
119
+ "finished_at": 1789029043.4601417
120
+ },
121
+ "whole_system_flops_complete": false,
122
+ "missing_components": [
123
+ "InstanceVG",
124
+ "full generation and all-task large-model coverage",
125
+ "uncovered operations",
126
+ "per-example routing alignment",
127
+ "95%/99% resampling intervals"
128
+ ]
129
+ }
paper_results/data/results.json ADDED
The diff for this file is too large to render. See raw diff
 
paper_results/data/scaling_selected.json ADDED
@@ -0,0 +1,1250 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "key": [
4
+ "cub",
5
+ "cub",
6
+ "MiniCPM"
7
+ ],
8
+ "rows": [
9
+ {
10
+ "id": "formal_0185",
11
+ "tag": "baseline_25pct",
12
+ "task": "cub",
13
+ "expert": "cub",
14
+ "endpoint": "MiniCPM",
15
+ "fraction": 0.25,
16
+ "student": "4M",
17
+ "branches": "output",
18
+ "fusion": "concat",
19
+ "target": "gain",
20
+ "strategy": "natural",
21
+ "representation_dim": 512,
22
+ "hidden_layer": "residual_blocks__layer_16",
23
+ "fixed25_metric": 0.7849499482222989,
24
+ "fixed25_actual_fraction": 0.2587159130134622,
25
+ "fixed25_rescue": 31,
26
+ "fixed25_harm": 201,
27
+ "rank25_metric": 0.7884017949603037,
28
+ "validation_selection": 0.9741666666666666,
29
+ "best_epoch": 12,
30
+ "seconds": 5.802937984466553
31
+ },
32
+ {
33
+ "id": "formal_0186",
34
+ "tag": "baseline_50pct",
35
+ "task": "cub",
36
+ "expert": "cub",
37
+ "endpoint": "MiniCPM",
38
+ "fraction": 0.5,
39
+ "student": "4M",
40
+ "branches": "output",
41
+ "fusion": "concat",
42
+ "target": "gain",
43
+ "strategy": "natural",
44
+ "representation_dim": 512,
45
+ "hidden_layer": "residual_blocks__layer_16",
46
+ "fixed25_metric": 0.7904729030031067,
47
+ "fixed25_actual_fraction": 0.2549188816016569,
48
+ "fixed25_rescue": 38,
49
+ "fixed25_harm": 176,
50
+ "rank25_metric": 0.7930617880566103,
51
+ "validation_selection": 0.9731666666666667,
52
+ "best_epoch": 7,
53
+ "seconds": 7.1614015102386475
54
+ },
55
+ {
56
+ "id": "formal_0001",
57
+ "tag": "baseline_100pct",
58
+ "task": "cub",
59
+ "expert": "cub",
60
+ "endpoint": "MiniCPM",
61
+ "fraction": 1.0,
62
+ "student": "4M",
63
+ "branches": "output",
64
+ "fusion": "concat",
65
+ "target": "gain",
66
+ "strategy": "natural",
67
+ "representation_dim": 512,
68
+ "hidden_layer": "residual_blocks__layer_16",
69
+ "fixed25_metric": 0.787193648602002,
70
+ "fixed25_actual_fraction": 0.2557818432861581,
71
+ "fixed25_rescue": 33,
72
+ "fixed25_harm": 190,
73
+ "rank25_metric": 0.7913358646876079,
74
+ "validation_selection": 0.9744166666666666,
75
+ "best_epoch": 7,
76
+ "seconds": 2.9200518131256104
77
+ }
78
+ ]
79
+ },
80
+ {
81
+ "key": [
82
+ "cub",
83
+ "cub",
84
+ "Qwen3.8"
85
+ ],
86
+ "rows": [
87
+ {
88
+ "id": "formal_0187",
89
+ "tag": "baseline_25pct",
90
+ "task": "cub",
91
+ "expert": "cub",
92
+ "endpoint": "Qwen3.8",
93
+ "fraction": 0.25,
94
+ "student": "4M",
95
+ "branches": "output+confidence",
96
+ "fusion": "concat",
97
+ "target": "four_state",
98
+ "strategy": "natural",
99
+ "representation_dim": 512,
100
+ "hidden_layer": "residual_blocks__layer_16",
101
+ "fixed25_metric": 0.6874352778736624,
102
+ "fixed25_actual_fraction": 0.6039005868139454,
103
+ "fixed25_rescue": 358,
104
+ "fixed25_harm": 1093,
105
+ "rank25_metric": 0.8060062133241284,
106
+ "validation_selection": 0.9316666666666666,
107
+ "best_epoch": 5,
108
+ "seconds": 4.623678922653198
109
+ },
110
+ {
111
+ "id": "formal_0188",
112
+ "tag": "baseline_50pct",
113
+ "task": "cub",
114
+ "expert": "cub",
115
+ "endpoint": "Qwen3.8",
116
+ "fraction": 0.5,
117
+ "student": "4M",
118
+ "branches": "output+confidence",
119
+ "fusion": "concat",
120
+ "target": "four_state",
121
+ "strategy": "natural",
122
+ "representation_dim": 512,
123
+ "hidden_layer": "residual_blocks__layer_16",
124
+ "fixed25_metric": 0.694338971349672,
125
+ "fixed25_actual_fraction": 0.5997583707283397,
126
+ "fixed25_rescue": 357,
127
+ "fixed25_harm": 1052,
128
+ "rank25_metric": 0.8096306523990335,
129
+ "validation_selection": 0.9334166666666667,
130
+ "best_epoch": 3,
131
+ "seconds": 4.7386627197265625
132
+ },
133
+ {
134
+ "id": "formal_0009",
135
+ "tag": "baseline_100pct",
136
+ "task": "cub",
137
+ "expert": "cub",
138
+ "endpoint": "Qwen3.8",
139
+ "fraction": 1.0,
140
+ "student": "4M",
141
+ "branches": "output+confidence",
142
+ "fusion": "concat",
143
+ "target": "four_state",
144
+ "strategy": "natural",
145
+ "representation_dim": 512,
146
+ "hidden_layer": "residual_blocks__layer_16",
147
+ "fixed25_metric": 0.6829478771142561,
148
+ "fixed25_actual_fraction": 0.6049361408353469,
149
+ "fixed25_rescue": 358,
150
+ "fixed25_harm": 1119,
151
+ "rank25_metric": 0.8091128753883328,
152
+ "validation_selection": 0.928,
153
+ "best_epoch": 1,
154
+ "seconds": 2.053396463394165
155
+ }
156
+ ]
157
+ },
158
+ {
159
+ "key": [
160
+ "cub",
161
+ "glsim",
162
+ "MiniCPM"
163
+ ],
164
+ "rows": [
165
+ {
166
+ "id": "formal_0189",
167
+ "tag": "baseline_25pct",
168
+ "task": "cub",
169
+ "expert": "glsim",
170
+ "endpoint": "MiniCPM",
171
+ "fraction": 0.25,
172
+ "student": "4M",
173
+ "branches": "output",
174
+ "fusion": "concat",
175
+ "target": "gain",
176
+ "strategy": "natural",
177
+ "representation_dim": 512,
178
+ "hidden_layer": "aggregator__layer_01",
179
+ "fixed25_metric": 0.8674490852606145,
180
+ "fixed25_actual_fraction": 0.24818778046254747,
181
+ "fixed25_rescue": 11,
182
+ "fixed25_harm": 209,
183
+ "rank25_metric": 0.8674490852606145,
184
+ "validation_selection": 0.9786666666666667,
185
+ "best_epoch": 12,
186
+ "seconds": 6.889333963394165
187
+ },
188
+ {
189
+ "id": "formal_0190",
190
+ "tag": "baseline_50pct",
191
+ "task": "cub",
192
+ "expert": "glsim",
193
+ "endpoint": "MiniCPM",
194
+ "fraction": 0.5,
195
+ "student": "4M",
196
+ "branches": "output",
197
+ "fusion": "concat",
198
+ "target": "gain",
199
+ "strategy": "natural",
200
+ "representation_dim": 512,
201
+ "hidden_layer": "aggregator__layer_01",
202
+ "fixed25_metric": 0.870037970314118,
203
+ "fixed25_actual_fraction": 0.24991370383154987,
204
+ "fixed25_rescue": 13,
205
+ "fixed25_harm": 196,
206
+ "rank25_metric": 0.870037970314118,
207
+ "validation_selection": 0.9773333333333334,
208
+ "best_epoch": 6,
209
+ "seconds": 5.409059524536133
210
+ },
211
+ {
212
+ "id": "formal_0019",
213
+ "tag": "baseline_100pct",
214
+ "task": "cub",
215
+ "expert": "glsim",
216
+ "endpoint": "MiniCPM",
217
+ "fraction": 1.0,
218
+ "student": "4M",
219
+ "branches": "output",
220
+ "fusion": "concat",
221
+ "target": "gain",
222
+ "strategy": "natural",
223
+ "representation_dim": 512,
224
+ "hidden_layer": "aggregator__layer_01",
225
+ "fixed25_metric": 0.8658957542285123,
226
+ "fixed25_actual_fraction": 0.25509147393855713,
227
+ "fixed25_rescue": 13,
228
+ "fixed25_harm": 220,
229
+ "rank25_metric": 0.8679668622713151,
230
+ "validation_selection": 0.9788333333333333,
231
+ "best_epoch": 9,
232
+ "seconds": 7.6912243366241455
233
+ }
234
+ ]
235
+ },
236
+ {
237
+ "key": [
238
+ "cub",
239
+ "glsim",
240
+ "Qwen3.8"
241
+ ],
242
+ "rows": [
243
+ {
244
+ "id": "formal_0191",
245
+ "tag": "baseline_25pct",
246
+ "task": "cub",
247
+ "expert": "glsim",
248
+ "endpoint": "Qwen3.8",
249
+ "fraction": 0.25,
250
+ "student": "4M",
251
+ "branches": "image+text+hidden",
252
+ "fusion": "concat",
253
+ "target": "four_state",
254
+ "strategy": "natural",
255
+ "representation_dim": 512,
256
+ "hidden_layer": "global_local_encoder__layer_12",
257
+ "fixed25_metric": 0.8797031411805316,
258
+ "fixed25_actual_fraction": 0.23662409389023129,
259
+ "fixed25_rescue": 18,
260
+ "fixed25_harm": 145,
261
+ "rank25_metric": 0.8776320331377286,
262
+ "validation_selection": 0.98025,
263
+ "best_epoch": 18,
264
+ "seconds": 7.610384702682495
265
+ },
266
+ {
267
+ "id": "formal_0192",
268
+ "tag": "baseline_50pct",
269
+ "task": "cub",
270
+ "expert": "glsim",
271
+ "endpoint": "Qwen3.8",
272
+ "fraction": 0.5,
273
+ "student": "4M",
274
+ "branches": "image+text+hidden",
275
+ "fusion": "concat",
276
+ "target": "four_state",
277
+ "strategy": "natural",
278
+ "representation_dim": 512,
279
+ "hidden_layer": "global_local_encoder__layer_12",
280
+ "fixed25_metric": 0.876423886779427,
281
+ "fixed25_actual_fraction": 0.24525371073524335,
282
+ "fixed25_rescue": 8,
283
+ "fixed25_harm": 154,
284
+ "rank25_metric": 0.8762512944425268,
285
+ "validation_selection": 0.981,
286
+ "best_epoch": 25,
287
+ "seconds": 10.510464906692505
288
+ },
289
+ {
290
+ "id": "formal_0027",
291
+ "tag": "baseline_100pct",
292
+ "task": "cub",
293
+ "expert": "glsim",
294
+ "endpoint": "Qwen3.8",
295
+ "fraction": 1.0,
296
+ "student": "4M",
297
+ "branches": "image+text+hidden",
298
+ "fusion": "concat",
299
+ "target": "four_state",
300
+ "strategy": "natural",
301
+ "representation_dim": 512,
302
+ "hidden_layer": "global_local_encoder__layer_12",
303
+ "fixed25_metric": 0.8684846392820159,
304
+ "fixed25_actual_fraction": 0.2507766655160511,
305
+ "fixed25_rescue": 17,
306
+ "fixed25_harm": 209,
307
+ "rank25_metric": 0.8684846392820159,
308
+ "validation_selection": 0.97525,
309
+ "best_epoch": 1,
310
+ "seconds": 3.935011863708496
311
+ }
312
+ ]
313
+ },
314
+ {
315
+ "key": [
316
+ "grefcoco",
317
+ "groundingdino",
318
+ "MiniCPM"
319
+ ],
320
+ "rows": [
321
+ {
322
+ "id": "formal_0193",
323
+ "tag": "baseline_25pct",
324
+ "task": "grefcoco",
325
+ "expert": "groundingdino",
326
+ "endpoint": "MiniCPM",
327
+ "fraction": 0.25,
328
+ "student": "4M",
329
+ "branches": "image+text+hidden+confidence",
330
+ "fusion": "attention",
331
+ "target": "error",
332
+ "strategy": "weighted",
333
+ "representation_dim": 512,
334
+ "hidden_layer": "text_encoder__layer_09",
335
+ "fixed25_metric": 0.38158976831239544,
336
+ "fixed25_actual_fraction": 0.24836230609987806,
337
+ "fixed25_rescue": 4083,
338
+ "fixed25_harm": 38,
339
+ "rank25_metric": 0.38201514335138814,
340
+ "validation_selection": 0.6237472766884531,
341
+ "best_epoch": 1,
342
+ "seconds": 28.353870153427124
343
+ },
344
+ {
345
+ "id": "formal_0194",
346
+ "tag": "baseline_50pct",
347
+ "task": "grefcoco",
348
+ "expert": "groundingdino",
349
+ "endpoint": "MiniCPM",
350
+ "fraction": 0.5,
351
+ "student": "4M",
352
+ "branches": "image+text+hidden+confidence",
353
+ "fusion": "attention",
354
+ "target": "error",
355
+ "strategy": "weighted",
356
+ "representation_dim": 512,
357
+ "hidden_layer": "text_encoder__layer_09",
358
+ "fixed25_metric": 0.3658792502055979,
359
+ "fixed25_actual_fraction": 0.22034427019822478,
360
+ "fixed25_rescue": 3503,
361
+ "fixed25_harm": 12,
362
+ "rank25_metric": 0.3758330261180274,
363
+ "validation_selection": 0.6238526951999438,
364
+ "best_epoch": 2,
365
+ "seconds": 56.78019833564758
366
+ },
367
+ {
368
+ "id": "formal_0042",
369
+ "tag": "baseline_100pct",
370
+ "task": "grefcoco",
371
+ "expert": "groundingdino",
372
+ "endpoint": "MiniCPM",
373
+ "fraction": 1.0,
374
+ "student": "4M",
375
+ "branches": "image+text+hidden+confidence",
376
+ "fusion": "attention",
377
+ "target": "error",
378
+ "strategy": "weighted",
379
+ "representation_dim": 512,
380
+ "hidden_layer": "text_encoder__layer_09",
381
+ "fixed25_metric": 0.3693106088534725,
382
+ "fixed25_actual_fraction": 0.22853982928281769,
383
+ "fixed25_rescue": 3642,
384
+ "fixed25_harm": 30,
385
+ "rank25_metric": 0.3758330261180274,
386
+ "validation_selection": 0.6221730269168599,
387
+ "best_epoch": 9,
388
+ "seconds": 476.6986792087555
389
+ }
390
+ ]
391
+ },
392
+ {
393
+ "key": [
394
+ "grefcoco",
395
+ "groundingdino",
396
+ "Qwen3.8"
397
+ ],
398
+ "rows": [
399
+ {
400
+ "id": "formal_0195",
401
+ "tag": "baseline_25pct",
402
+ "task": "grefcoco",
403
+ "expert": "groundingdino",
404
+ "endpoint": "Qwen3.8",
405
+ "fraction": 0.25,
406
+ "student": "4M",
407
+ "branches": "image+text+hidden+confidence",
408
+ "fusion": "concat",
409
+ "target": "error",
410
+ "strategy": "weighted",
411
+ "representation_dim": 512,
412
+ "hidden_layer": "text_encoder__layer_09",
413
+ "fixed25_metric": 0.4562572668235828,
414
+ "fixed25_actual_fraction": 0.24856081445140799,
415
+ "fixed25_rescue": 6732,
416
+ "fixed25_harm": 54,
417
+ "rank25_metric": 0.45705130022970253,
418
+ "validation_selection": 0.6656265373532926,
419
+ "best_epoch": 1,
420
+ "seconds": 18.048375368118286
421
+ },
422
+ {
423
+ "id": "formal_0196",
424
+ "tag": "baseline_50pct",
425
+ "task": "grefcoco",
426
+ "expert": "groundingdino",
427
+ "endpoint": "Qwen3.8",
428
+ "fraction": 0.5,
429
+ "student": "4M",
430
+ "branches": "image+text+hidden+confidence",
431
+ "fusion": "concat",
432
+ "target": "error",
433
+ "strategy": "weighted",
434
+ "representation_dim": 512,
435
+ "hidden_layer": "text_encoder__layer_09",
436
+ "fixed25_metric": 0.434648214842753,
437
+ "fixed25_actual_fraction": 0.22008904517482913,
438
+ "fixed25_rescue": 5930,
439
+ "fixed25_harm": 14,
440
+ "rank25_metric": 0.4536766582536937,
441
+ "validation_selection": 0.6660130718954248,
442
+ "best_epoch": 2,
443
+ "seconds": 33.53883171081543
444
+ },
445
+ {
446
+ "id": "formal_0057",
447
+ "tag": "baseline_100pct",
448
+ "task": "grefcoco",
449
+ "expert": "groundingdino",
450
+ "endpoint": "Qwen3.8",
451
+ "fraction": 1.0,
452
+ "student": "4M",
453
+ "branches": "image+text+hidden+confidence",
454
+ "fusion": "concat",
455
+ "target": "error",
456
+ "strategy": "weighted",
457
+ "representation_dim": 512,
458
+ "hidden_layer": "text_encoder__layer_09",
459
+ "fixed25_metric": 0.43033774778096023,
460
+ "fixed25_actual_fraction": 0.2126591611604231,
461
+ "fixed25_rescue": 5784,
462
+ "fixed25_harm": 20,
463
+ "rank25_metric": 0.4544139749879477,
464
+ "validation_selection": 0.6650924168950734,
465
+ "best_epoch": 5,
466
+ "seconds": 281.1101198196411
467
+ }
468
+ ]
469
+ },
470
+ {
471
+ "key": [
472
+ "grefcoco",
473
+ "instancevg",
474
+ "MiniCPM"
475
+ ],
476
+ "rows": [
477
+ {
478
+ "id": "formal_0197",
479
+ "tag": "baseline_25pct",
480
+ "task": "grefcoco",
481
+ "expert": "instancevg",
482
+ "endpoint": "MiniCPM",
483
+ "fraction": 0.25,
484
+ "student": "4M",
485
+ "branches": "hidden",
486
+ "fusion": "concat",
487
+ "target": "gain",
488
+ "strategy": "weighted",
489
+ "representation_dim": 512,
490
+ "hidden_layer": "query_generation__layer_01",
491
+ "fixed25_metric": 0.6523268014632901,
492
+ "fixed25_actual_fraction": 0.3479284235601055,
493
+ "fixed25_rescue": 1394,
494
+ "fixed25_harm": 2000,
495
+ "rank25_metric": 0.6652582026486685,
496
+ "validation_selection": 0.753925082577834,
497
+ "best_epoch": 13,
498
+ "seconds": 28.585034132003784
499
+ },
500
+ {
501
+ "id": "formal_0198",
502
+ "tag": "baseline_50pct",
503
+ "task": "grefcoco",
504
+ "expert": "instancevg",
505
+ "endpoint": "MiniCPM",
506
+ "fraction": 0.5,
507
+ "student": "4M",
508
+ "branches": "hidden",
509
+ "fusion": "concat",
510
+ "target": "gain",
511
+ "strategy": "weighted",
512
+ "representation_dim": 512,
513
+ "hidden_layer": "query_generation__layer_01",
514
+ "fixed25_metric": 0.6567223435328815,
515
+ "fixed25_actual_fraction": 0.3123670703003148,
516
+ "fixed25_rescue": 1366,
517
+ "fixed25_harm": 1817,
518
+ "rank25_metric": 0.662762669086578,
519
+ "validation_selection": 0.7576779815868999,
520
+ "best_epoch": 13,
521
+ "seconds": 44.9037389755249
522
+ },
523
+ {
524
+ "id": "formal_0072",
525
+ "tag": "baseline_100pct",
526
+ "task": "grefcoco",
527
+ "expert": "instancevg",
528
+ "endpoint": "MiniCPM",
529
+ "fraction": 1.0,
530
+ "student": "4M",
531
+ "branches": "hidden",
532
+ "fusion": "concat",
533
+ "target": "gain",
534
+ "strategy": "weighted",
535
+ "representation_dim": 512,
536
+ "hidden_layer": "query_generation__layer_01",
537
+ "fixed25_metric": 0.654765618353515,
538
+ "fixed25_actual_fraction": 0.33953435612398264,
539
+ "fixed25_rescue": 1510,
540
+ "fixed25_harm": 2030,
541
+ "rank25_metric": 0.6655985026798628,
542
+ "validation_selection": 0.7589219200224893,
543
+ "best_epoch": 3,
544
+ "seconds": 163.15981340408325
545
+ }
546
+ ]
547
+ },
548
+ {
549
+ "key": [
550
+ "grefcoco",
551
+ "instancevg",
552
+ "Qwen3.8"
553
+ ],
554
+ "rows": [
555
+ {
556
+ "id": "formal_0199",
557
+ "tag": "baseline_25pct",
558
+ "task": "grefcoco",
559
+ "expert": "instancevg",
560
+ "endpoint": "Qwen3.8",
561
+ "fraction": 0.25,
562
+ "student": "4M",
563
+ "branches": "output",
564
+ "fusion": "concat",
565
+ "target": "gain",
566
+ "strategy": "natural",
567
+ "representation_dim": 512,
568
+ "hidden_layer": "lateral_features__layer_01",
569
+ "fixed25_metric": 0.7098375067351048,
570
+ "fixed25_actual_fraction": 0.10637211808411083,
571
+ "fixed25_rescue": 1492,
572
+ "fixed25_harm": 70,
573
+ "rank25_metric": 0.7019538893457732,
574
+ "validation_selection": 0.7565746011666316,
575
+ "best_epoch": 1,
576
+ "seconds": 14.837297677993774
577
+ },
578
+ {
579
+ "id": "formal_0200",
580
+ "tag": "baseline_50pct",
581
+ "task": "grefcoco",
582
+ "expert": "instancevg",
583
+ "endpoint": "Qwen3.8",
584
+ "fraction": 0.5,
585
+ "student": "4M",
586
+ "branches": "output",
587
+ "fusion": "concat",
588
+ "target": "gain",
589
+ "strategy": "natural",
590
+ "representation_dim": 512,
591
+ "hidden_layer": "lateral_features__layer_01",
592
+ "fixed25_metric": 0.7416555596517597,
593
+ "fixed25_actual_fraction": 0.4823752942177353,
594
+ "fixed25_rescue": 3418,
595
+ "fixed25_harm": 874,
596
+ "rank25_metric": 0.7207838244051838,
597
+ "validation_selection": 0.8144985592803429,
598
+ "best_epoch": 1,
599
+ "seconds": 21.554462432861328
600
+ },
601
+ {
602
+ "id": "formal_0083",
603
+ "tag": "baseline_100pct",
604
+ "task": "grefcoco",
605
+ "expert": "instancevg",
606
+ "endpoint": "Qwen3.8",
607
+ "fraction": 1.0,
608
+ "student": "4M",
609
+ "branches": "output",
610
+ "fusion": "concat",
611
+ "target": "gain",
612
+ "strategy": "natural",
613
+ "representation_dim": 512,
614
+ "hidden_layer": "lateral_features__layer_01",
615
+ "fixed25_metric": 0.7090151149930523,
616
+ "fixed25_actual_fraction": 0.12330204463602076,
617
+ "fixed25_rescue": 1514,
618
+ "fixed25_harm": 121,
619
+ "rank25_metric": 0.7021523976973031,
620
+ "validation_selection": 0.7565746011666316,
621
+ "best_epoch": 1,
622
+ "seconds": 121.36263680458069
623
+ }
624
+ ]
625
+ },
626
+ {
627
+ "key": [
628
+ "nlvr2",
629
+ "nlvr2",
630
+ "MiniCPM"
631
+ ],
632
+ "rows": [
633
+ {
634
+ "id": "formal_0201",
635
+ "tag": "baseline_25pct",
636
+ "task": "nlvr2",
637
+ "expert": "nlvr2",
638
+ "endpoint": "MiniCPM",
639
+ "fraction": 0.25,
640
+ "student": "4M",
641
+ "branches": "hidden+confidence",
642
+ "fusion": "concat",
643
+ "target": "gain",
644
+ "strategy": "natural",
645
+ "representation_dim": 512,
646
+ "hidden_layer": "multimodal_encoder__layer_12",
647
+ "fixed25_metric": 0.863815742268781,
648
+ "fixed25_actual_fraction": 0.253928392049939,
649
+ "fixed25_rescue": 1100,
650
+ "fixed25_harm": 211,
651
+ "rank25_metric": 0.8630264762861448,
652
+ "validation_selection": 0.8331853337152678,
653
+ "best_epoch": 2,
654
+ "seconds": 11.065314054489136
655
+ },
656
+ {
657
+ "id": "formal_0202",
658
+ "tag": "baseline_50pct",
659
+ "task": "nlvr2",
660
+ "expert": "nlvr2",
661
+ "endpoint": "MiniCPM",
662
+ "fraction": 0.5,
663
+ "student": "4M",
664
+ "branches": "hidden+confidence",
665
+ "fusion": "concat",
666
+ "target": "gain",
667
+ "strategy": "natural",
668
+ "representation_dim": 512,
669
+ "hidden_layer": "multimodal_encoder__layer_12",
670
+ "fixed25_metric": 0.868336083805697,
671
+ "fixed25_actual_fraction": 0.24388318863456984,
672
+ "fixed25_rescue": 1152,
673
+ "fixed25_harm": 200,
674
+ "rank25_metric": 0.8689818468823993,
675
+ "validation_selection": 0.8330850759094816,
676
+ "best_epoch": 3,
677
+ "seconds": 16.297266960144043
678
+ },
679
+ {
680
+ "id": "formal_0091",
681
+ "tag": "baseline_100pct",
682
+ "task": "nlvr2",
683
+ "expert": "nlvr2",
684
+ "endpoint": "MiniCPM",
685
+ "fraction": 1.0,
686
+ "student": "4M",
687
+ "branches": "hidden+confidence",
688
+ "fusion": "concat",
689
+ "target": "gain",
690
+ "strategy": "natural",
691
+ "representation_dim": 512,
692
+ "hidden_layer": "multimodal_encoder__layer_12",
693
+ "fixed25_metric": 0.8707756332065725,
694
+ "fixed25_actual_fraction": 0.26117528879959817,
695
+ "fixed25_rescue": 1197,
696
+ "fixed25_harm": 211,
697
+ "rank25_metric": 0.8689100954294324,
698
+ "validation_selection": 0.8335577198510455,
699
+ "best_epoch": 6,
700
+ "seconds": 82.51407408714294
701
+ }
702
+ ]
703
+ },
704
+ {
705
+ "key": [
706
+ "nlvr2",
707
+ "nlvr2",
708
+ "Qwen3.8"
709
+ ],
710
+ "rows": [
711
+ {
712
+ "id": "formal_0203",
713
+ "tag": "baseline_25pct",
714
+ "task": "nlvr2",
715
+ "expert": "nlvr2",
716
+ "endpoint": "Qwen3.8",
717
+ "fraction": 0.25,
718
+ "student": "4M",
719
+ "branches": "output+confidence",
720
+ "fusion": "concat",
721
+ "target": "gain",
722
+ "strategy": "weighted",
723
+ "representation_dim": 512,
724
+ "hidden_layer": "multimodal_encoder__layer_10",
725
+ "fixed25_metric": 0.8717801535481093,
726
+ "fixed25_actual_fraction": 0.25299562316136903,
727
+ "fixed25_rescue": 1180,
728
+ "fixed25_harm": 180,
729
+ "rank25_metric": 0.8714213962832748,
730
+ "validation_selection": 0.8382268690919508,
731
+ "best_epoch": 3,
732
+ "seconds": 12.40796971321106
733
+ },
734
+ {
735
+ "id": "formal_0204",
736
+ "tag": "baseline_50pct",
737
+ "task": "nlvr2",
738
+ "expert": "nlvr2",
739
+ "endpoint": "Qwen3.8",
740
+ "fraction": 0.5,
741
+ "student": "4M",
742
+ "branches": "output+confidence",
743
+ "fusion": "concat",
744
+ "target": "gain",
745
+ "strategy": "weighted",
746
+ "representation_dim": 512,
747
+ "hidden_layer": "multimodal_encoder__layer_10",
748
+ "fixed25_metric": 0.872067159359977,
749
+ "fixed25_actual_fraction": 0.25407189495587285,
750
+ "fixed25_rescue": 1178,
751
+ "fixed25_harm": 174,
752
+ "rank25_metric": 0.8703451244887709,
753
+ "validation_selection": 0.8385276425093097,
754
+ "best_epoch": 11,
755
+ "seconds": 27.56421136856079
756
+ },
757
+ {
758
+ "id": "formal_0104",
759
+ "tag": "baseline_100pct",
760
+ "task": "nlvr2",
761
+ "expert": "nlvr2",
762
+ "endpoint": "Qwen3.8",
763
+ "fraction": 1.0,
764
+ "student": "4M",
765
+ "branches": "output+confidence",
766
+ "fusion": "concat",
767
+ "target": "gain",
768
+ "strategy": "weighted",
769
+ "representation_dim": 512,
770
+ "hidden_layer": "multimodal_encoder__layer_10",
771
+ "fixed25_metric": 0.8709191361125063,
772
+ "fixed25_actual_fraction": 0.25400014350290595,
773
+ "fixed25_rescue": 1153,
774
+ "fixed25_harm": 165,
775
+ "rank25_metric": 0.8695558585061347,
776
+ "validation_selection": 0.8383414494414207,
777
+ "best_epoch": 7,
778
+ "seconds": 91.79227209091187
779
+ }
780
+ ]
781
+ },
782
+ {
783
+ "key": [
784
+ "nlvr2",
785
+ "vilt",
786
+ "MiniCPM"
787
+ ],
788
+ "rows": [
789
+ {
790
+ "id": "formal_0209",
791
+ "tag": "baseline_25pct",
792
+ "task": "nlvr2",
793
+ "expert": "vilt",
794
+ "endpoint": "MiniCPM",
795
+ "fraction": 0.25,
796
+ "student": "8M",
797
+ "branches": "image+text+hidden+confidence",
798
+ "fusion": "attention",
799
+ "target": "four_state",
800
+ "strategy": "natural",
801
+ "representation_dim": 512,
802
+ "hidden_layer": "ordered_image_text__layer_12",
803
+ "fixed25_metric": 0.8323886058692689,
804
+ "fixed25_actual_fraction": 0.24072612470402527,
805
+ "fixed25_rescue": 1189,
806
+ "fixed25_harm": 239,
807
+ "rank25_metric": 0.8343258950993757,
808
+ "validation_selection": 0.7950730449727872,
809
+ "best_epoch": 4,
810
+ "seconds": 18.022375106811523
811
+ },
812
+ {
813
+ "id": "formal_0210",
814
+ "tag": "baseline_50pct",
815
+ "task": "nlvr2",
816
+ "expert": "vilt",
817
+ "endpoint": "MiniCPM",
818
+ "fraction": 0.5,
819
+ "student": "8M",
820
+ "branches": "image+text+hidden+confidence",
821
+ "fusion": "attention",
822
+ "target": "four_state",
823
+ "strategy": "natural",
824
+ "representation_dim": 512,
825
+ "hidden_layer": "ordered_image_text__layer_12",
826
+ "fixed25_metric": 0.8337518834756403,
827
+ "fixed25_actual_fraction": 0.24704025256511444,
828
+ "fixed25_rescue": 1213,
829
+ "fixed25_harm": 244,
830
+ "rank25_metric": 0.8344693980053096,
831
+ "validation_selection": 0.7959610426811802,
832
+ "best_epoch": 8,
833
+ "seconds": 38.606887102127075
834
+ },
835
+ {
836
+ "id": "formal_0139",
837
+ "tag": "baseline_100pct",
838
+ "task": "nlvr2",
839
+ "expert": "vilt",
840
+ "endpoint": "MiniCPM",
841
+ "fraction": 1.0,
842
+ "student": "8M",
843
+ "branches": "image+text+hidden+confidence",
844
+ "fusion": "attention",
845
+ "target": "four_state",
846
+ "strategy": "natural",
847
+ "representation_dim": 512,
848
+ "hidden_layer": "ordered_image_text__layer_12",
849
+ "fixed25_metric": 0.8327473631341035,
850
+ "fixed25_actual_fraction": 0.24208940231039677,
851
+ "fixed25_rescue": 1196,
852
+ "fixed25_harm": 241,
853
+ "rank25_metric": 0.8346846523642104,
854
+ "validation_selection": 0.7952162704096247,
855
+ "best_epoch": 3,
856
+ "seconds": 136.20692539215088
857
+ }
858
+ ]
859
+ },
860
+ {
861
+ "key": [
862
+ "nlvr2",
863
+ "vilt",
864
+ "Qwen3.8"
865
+ ],
866
+ "rows": [
867
+ {
868
+ "id": "formal_0211",
869
+ "tag": "baseline_25pct",
870
+ "task": "nlvr2",
871
+ "expert": "vilt",
872
+ "endpoint": "Qwen3.8",
873
+ "fraction": 0.25,
874
+ "student": "2M",
875
+ "branches": "image+text+hidden+confidence",
876
+ "fusion": "concat",
877
+ "target": "gain",
878
+ "strategy": "weighted",
879
+ "representation_dim": 512,
880
+ "hidden_layer": "ordered_image_text__layer_10",
881
+ "fixed25_metric": 0.8336083805697065,
882
+ "fixed25_actual_fraction": 0.24431369735237138,
883
+ "fixed25_rescue": 1234,
884
+ "fixed25_harm": 267,
885
+ "rank25_metric": 0.8346846523642104,
886
+ "validation_selection": 0.7945717559438557,
887
+ "best_epoch": 2,
888
+ "seconds": 12.189603567123413
889
+ },
890
+ {
891
+ "id": "formal_0212",
892
+ "tag": "baseline_50pct",
893
+ "task": "nlvr2",
894
+ "expert": "vilt",
895
+ "endpoint": "Qwen3.8",
896
+ "fraction": 0.5,
897
+ "student": "2M",
898
+ "branches": "image+text+hidden+confidence",
899
+ "fusion": "concat",
900
+ "target": "gain",
901
+ "strategy": "weighted",
902
+ "representation_dim": 512,
903
+ "hidden_layer": "ordered_image_text__layer_10",
904
+ "fixed25_metric": 0.8308818253569635,
905
+ "fixed25_actual_fraction": 0.2361340317141422,
906
+ "fixed25_rescue": 1178,
907
+ "fixed25_harm": 249,
908
+ "rank25_metric": 0.834110640740475,
909
+ "validation_selection": 0.7955027212832999,
910
+ "best_epoch": 3,
911
+ "seconds": 18.873005151748657
912
+ },
913
+ {
914
+ "id": "formal_0154",
915
+ "tag": "baseline_100pct",
916
+ "task": "nlvr2",
917
+ "expert": "vilt",
918
+ "endpoint": "Qwen3.8",
919
+ "fraction": 1.0,
920
+ "student": "2M",
921
+ "branches": "image+text+hidden+confidence",
922
+ "fusion": "concat",
923
+ "target": "gain",
924
+ "strategy": "weighted",
925
+ "representation_dim": 512,
926
+ "hidden_layer": "ordered_image_text__layer_10",
927
+ "fixed25_metric": 0.8329626174930043,
928
+ "fixed25_actual_fraction": 0.24094137906292604,
929
+ "fixed25_rescue": 1225,
930
+ "fixed25_harm": 267,
931
+ "rank25_metric": 0.8348281552701442,
932
+ "validation_selection": 0.7948438842738471,
933
+ "best_epoch": 2,
934
+ "seconds": 78.18392181396484
935
+ }
936
+ ]
937
+ },
938
+ {
939
+ "key": [
940
+ "construction",
941
+ "yolo26x",
942
+ "MiniCPM"
943
+ ],
944
+ "rows": [
945
+ {
946
+ "id": "formal_0213",
947
+ "tag": "baseline_25pct",
948
+ "task": "construction",
949
+ "expert": "yolo26x",
950
+ "endpoint": "MiniCPM",
951
+ "fraction": 0.25,
952
+ "student": "4M",
953
+ "branches": "output",
954
+ "fusion": "concat",
955
+ "target": "gain",
956
+ "strategy": "weighted",
957
+ "representation_dim": 512,
958
+ "hidden_layer": "feature_blocks__layer_14",
959
+ "fixed25_metric": 0.2225771535665814,
960
+ "fixed25_actual_fraction": 0.18342210386151797,
961
+ "fixed25_rescue": 608,
962
+ "fixed25_harm": 265,
963
+ "rank25_metric": 0.22250151381741792,
964
+ "validation_selection": 0.245059255252362,
965
+ "best_epoch": 2,
966
+ "seconds": 2.557858467102051
967
+ },
968
+ {
969
+ "id": "formal_0214",
970
+ "tag": "baseline_50pct",
971
+ "task": "construction",
972
+ "expert": "yolo26x",
973
+ "endpoint": "MiniCPM",
974
+ "fraction": 0.5,
975
+ "student": "4M",
976
+ "branches": "output",
977
+ "fusion": "concat",
978
+ "target": "gain",
979
+ "strategy": "weighted",
980
+ "representation_dim": 512,
981
+ "hidden_layer": "feature_blocks__layer_14",
982
+ "fixed25_metric": 0.2225771535665814,
983
+ "fixed25_actual_fraction": 0.18342210386151797,
984
+ "fixed25_rescue": 608,
985
+ "fixed25_harm": 265,
986
+ "rank25_metric": 0.22250151381741792,
987
+ "validation_selection": 0.2411747600492345,
988
+ "best_epoch": 1,
989
+ "seconds": 3.123697280883789
990
+ },
991
+ {
992
+ "id": "formal_0169",
993
+ "tag": "baseline_100pct",
994
+ "task": "construction",
995
+ "expert": "yolo26x",
996
+ "endpoint": "MiniCPM",
997
+ "fraction": 1.0,
998
+ "student": "4M",
999
+ "branches": "output",
1000
+ "fusion": "concat",
1001
+ "target": "gain",
1002
+ "strategy": "weighted",
1003
+ "representation_dim": 512,
1004
+ "hidden_layer": "feature_blocks__layer_14",
1005
+ "fixed25_metric": 0.22276943862299337,
1006
+ "fixed25_actual_fraction": 0.18874833555259654,
1007
+ "fixed25_rescue": 614,
1008
+ "fixed25_harm": 271,
1009
+ "rank25_metric": 0.22272342953859628,
1010
+ "validation_selection": 0.23173775721145673,
1011
+ "best_epoch": 4,
1012
+ "seconds": 9.216264009475708
1013
+ }
1014
+ ]
1015
+ },
1016
+ {
1017
+ "key": [
1018
+ "construction",
1019
+ "yolo26x",
1020
+ "Qwen3.8"
1021
+ ],
1022
+ "rows": [
1023
+ {
1024
+ "id": "formal_0215",
1025
+ "tag": "baseline_25pct",
1026
+ "task": "construction",
1027
+ "expert": "yolo26x",
1028
+ "endpoint": "Qwen3.8",
1029
+ "fraction": 0.25,
1030
+ "student": "4M",
1031
+ "branches": "output",
1032
+ "fusion": "concat",
1033
+ "target": "gain",
1034
+ "strategy": "weighted",
1035
+ "representation_dim": 512,
1036
+ "hidden_layer": "feature_blocks__layer_14",
1037
+ "fixed25_metric": 0.2503793990148833,
1038
+ "fixed25_actual_fraction": 0.18342210386151797,
1039
+ "fixed25_rescue": 653,
1040
+ "fixed25_harm": 116,
1041
+ "rank25_metric": 0.26327645391521537,
1042
+ "validation_selection": 0.2490976644605033,
1043
+ "best_epoch": 7,
1044
+ "seconds": 3.354280948638916
1045
+ },
1046
+ {
1047
+ "id": "formal_0216",
1048
+ "tag": "baseline_50pct",
1049
+ "task": "construction",
1050
+ "expert": "yolo26x",
1051
+ "endpoint": "Qwen3.8",
1052
+ "fraction": 0.5,
1053
+ "student": "4M",
1054
+ "branches": "output",
1055
+ "fusion": "concat",
1056
+ "target": "gain",
1057
+ "strategy": "weighted",
1058
+ "representation_dim": 512,
1059
+ "hidden_layer": "feature_blocks__layer_14",
1060
+ "fixed25_metric": 0.2503793990148833,
1061
+ "fixed25_actual_fraction": 0.18342210386151797,
1062
+ "fixed25_rescue": 653,
1063
+ "fixed25_harm": 116,
1064
+ "rank25_metric": 0.26327645391521537,
1065
+ "validation_selection": 0.2490976644605033,
1066
+ "best_epoch": 14,
1067
+ "seconds": 4.801910877227783
1068
+ },
1069
+ {
1070
+ "id": "formal_0177",
1071
+ "tag": "baseline_100pct",
1072
+ "task": "construction",
1073
+ "expert": "yolo26x",
1074
+ "endpoint": "Qwen3.8",
1075
+ "fraction": 1.0,
1076
+ "student": "4M",
1077
+ "branches": "output",
1078
+ "fusion": "concat",
1079
+ "target": "gain",
1080
+ "strategy": "weighted",
1081
+ "representation_dim": 512,
1082
+ "hidden_layer": "feature_blocks__layer_14",
1083
+ "fixed25_metric": 0.2503793990148833,
1084
+ "fixed25_actual_fraction": 0.18342210386151797,
1085
+ "fixed25_rescue": 653,
1086
+ "fixed25_harm": 116,
1087
+ "rank25_metric": 0.26327645391521537,
1088
+ "validation_selection": 0.24929743985848513,
1089
+ "best_epoch": 13,
1090
+ "seconds": 11.173577785491943
1091
+ }
1092
+ ]
1093
+ },
1094
+ {
1095
+ "key": [
1096
+ "construction",
1097
+ "rtdetr_x_fp32",
1098
+ "MiniCPM"
1099
+ ],
1100
+ "rows": [
1101
+ {
1102
+ "id": "formal_0205",
1103
+ "tag": "baseline_25pct",
1104
+ "task": "construction",
1105
+ "expert": "rtdetr_x_fp32",
1106
+ "endpoint": "MiniCPM",
1107
+ "fraction": 0.25,
1108
+ "student": "4M",
1109
+ "branches": "image+text+hidden+confidence",
1110
+ "fusion": "concat",
1111
+ "target": "gain",
1112
+ "strategy": "weighted",
1113
+ "representation_dim": 512,
1114
+ "hidden_layer": "query_decoder__layer_04",
1115
+ "fixed25_metric": 0.12161748372078302,
1116
+ "fixed25_actual_fraction": 0.26897470039946736,
1117
+ "fixed25_rescue": 1103,
1118
+ "fixed25_harm": 518,
1119
+ "rank25_metric": 0.12103479129839005,
1120
+ "validation_selection": 0.13942018347651794,
1121
+ "best_epoch": 1,
1122
+ "seconds": 3.9248127937316895
1123
+ },
1124
+ {
1125
+ "id": "formal_0206",
1126
+ "tag": "baseline_50pct",
1127
+ "task": "construction",
1128
+ "expert": "rtdetr_x_fp32",
1129
+ "endpoint": "MiniCPM",
1130
+ "fraction": 0.5,
1131
+ "student": "4M",
1132
+ "branches": "image+text+hidden+confidence",
1133
+ "fusion": "concat",
1134
+ "target": "gain",
1135
+ "strategy": "weighted",
1136
+ "representation_dim": 512,
1137
+ "hidden_layer": "query_decoder__layer_04",
1138
+ "fixed25_metric": 0.11138731664705415,
1139
+ "fixed25_actual_fraction": 0.25798934753661784,
1140
+ "fixed25_rescue": 1337,
1141
+ "fixed25_harm": 464,
1142
+ "rank25_metric": 0.11157434949576527,
1143
+ "validation_selection": 0.10286119045745137,
1144
+ "best_epoch": 3,
1145
+ "seconds": 5.573187828063965
1146
+ },
1147
+ {
1148
+ "id": "formal_0114",
1149
+ "tag": "baseline_100pct",
1150
+ "task": "construction",
1151
+ "expert": "rtdetr_x_fp32",
1152
+ "endpoint": "MiniCPM",
1153
+ "fraction": 1.0,
1154
+ "student": "4M",
1155
+ "branches": "image+text+hidden+confidence",
1156
+ "fusion": "concat",
1157
+ "target": "gain",
1158
+ "strategy": "weighted",
1159
+ "representation_dim": 512,
1160
+ "hidden_layer": "query_decoder__layer_04",
1161
+ "fixed25_metric": 0.11409050364241882,
1162
+ "fixed25_actual_fraction": 0.2333555259653795,
1163
+ "fixed25_rescue": 1155,
1164
+ "fixed25_harm": 428,
1165
+ "rank25_metric": 0.1142018021781044,
1166
+ "validation_selection": 0.09196748903785351,
1167
+ "best_epoch": 1,
1168
+ "seconds": 7.1226747035980225
1169
+ }
1170
+ ]
1171
+ },
1172
+ {
1173
+ "key": [
1174
+ "construction",
1175
+ "rtdetr_x_fp32",
1176
+ "Qwen3.8"
1177
+ ],
1178
+ "rows": [
1179
+ {
1180
+ "id": "formal_0207",
1181
+ "tag": "baseline_25pct",
1182
+ "task": "construction",
1183
+ "expert": "rtdetr_x_fp32",
1184
+ "endpoint": "Qwen3.8",
1185
+ "fraction": 0.25,
1186
+ "student": "4M",
1187
+ "branches": "hidden",
1188
+ "fusion": "concat",
1189
+ "target": "error",
1190
+ "strategy": "balanced",
1191
+ "representation_dim": 512,
1192
+ "hidden_layer": "feature_blocks__layer_11",
1193
+ "fixed25_metric": 0.14471609214350373,
1194
+ "fixed25_actual_fraction": 0.2892809587217044,
1195
+ "fixed25_rescue": 1383,
1196
+ "fixed25_harm": 258,
1197
+ "rank25_metric": 0.14346432207238047,
1198
+ "validation_selection": 0.16233096000534752,
1199
+ "best_epoch": 17,
1200
+ "seconds": 8.028136253356934
1201
+ },
1202
+ {
1203
+ "id": "formal_0208",
1204
+ "tag": "baseline_50pct",
1205
+ "task": "construction",
1206
+ "expert": "rtdetr_x_fp32",
1207
+ "endpoint": "Qwen3.8",
1208
+ "fraction": 0.5,
1209
+ "student": "4M",
1210
+ "branches": "hidden",
1211
+ "fusion": "concat",
1212
+ "target": "error",
1213
+ "strategy": "balanced",
1214
+ "representation_dim": 512,
1215
+ "hidden_layer": "feature_blocks__layer_11",
1216
+ "fixed25_metric": 0.1499641421353766,
1217
+ "fixed25_actual_fraction": 0.26797603195739017,
1218
+ "fixed25_rescue": 1348,
1219
+ "fixed25_harm": 225,
1220
+ "rank25_metric": 0.14991013725904712,
1221
+ "validation_selection": 0.14960536227871302,
1222
+ "best_epoch": 14,
1223
+ "seconds": 7.761091709136963
1224
+ },
1225
+ {
1226
+ "id": "formal_0129",
1227
+ "tag": "baseline_100pct",
1228
+ "task": "construction",
1229
+ "expert": "rtdetr_x_fp32",
1230
+ "endpoint": "Qwen3.8",
1231
+ "fraction": 1.0,
1232
+ "student": "4M",
1233
+ "branches": "hidden",
1234
+ "fusion": "concat",
1235
+ "target": "error",
1236
+ "strategy": "balanced",
1237
+ "representation_dim": 512,
1238
+ "hidden_layer": "feature_blocks__layer_11",
1239
+ "fixed25_metric": 0.1498069310931111,
1240
+ "fixed25_actual_fraction": 0.2539946737683089,
1241
+ "fixed25_rescue": 1284,
1242
+ "fixed25_harm": 203,
1243
+ "rank25_metric": 0.15030144907600507,
1244
+ "validation_selection": 0.14151521056562297,
1245
+ "best_epoch": 16,
1246
+ "seconds": 13.48578929901123
1247
+ }
1248
+ ]
1249
+ }
1250
+ ]
paper_results/data/shared_search.json ADDED
@@ -0,0 +1,720 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "status": "search_complete",
3
+ "started_at": 1789041525.1799183,
4
+ "completed_stages": [
5
+ "hidden_percent",
6
+ "student",
7
+ "representation_dim",
8
+ "lr",
9
+ "target",
10
+ "strategy"
11
+ ],
12
+ "selected": "shared_12",
13
+ "candidates": [
14
+ {
15
+ "id": "shared_00",
16
+ "settings": {
17
+ "student": "4M",
18
+ "representation_dim": 512,
19
+ "lr": 0.001,
20
+ "target": "four_state",
21
+ "strategy": "natural",
22
+ "hidden_percent": 90
23
+ },
24
+ "runs": [
25
+ "shared_00_cub_MiniCPM",
26
+ "shared_00_cub_Qwen3.8",
27
+ "shared_00_glsim_MiniCPM",
28
+ "shared_00_glsim_Qwen3.8",
29
+ "shared_00_groundingdino_MiniCPM",
30
+ "shared_00_groundingdino_Qwen3.8",
31
+ "shared_00_instancevg_MiniCPM",
32
+ "shared_00_instancevg_Qwen3.8",
33
+ "shared_00_nlvr2_MiniCPM",
34
+ "shared_00_nlvr2_Qwen3.8",
35
+ "shared_00_vilt_MiniCPM",
36
+ "shared_00_vilt_Qwen3.8",
37
+ "shared_00_yolo26x_MiniCPM",
38
+ "shared_00_yolo26x_Qwen3.8",
39
+ "shared_00_rtdetr_x_fp32_MiniCPM",
40
+ "shared_00_rtdetr_x_fp32_Qwen3.8"
41
+ ],
42
+ "status": "complete",
43
+ "task_means": {
44
+ "cub": 0.9498541666666667,
45
+ "grefcoco": 0.7090097687820648,
46
+ "nlvr2": 0.8164207963334288,
47
+ "construction": 0.175986962717605
48
+ },
49
+ "validation_mean": 0.6628179236249413,
50
+ "mean_active_parameters": 4881924.0,
51
+ "mean_fit_seconds": 37.63353890167855
52
+ },
53
+ {
54
+ "id": "shared_01",
55
+ "settings": {
56
+ "student": "4M",
57
+ "representation_dim": 512,
58
+ "lr": 0.001,
59
+ "target": "four_state",
60
+ "strategy": "natural",
61
+ "hidden_percent": 30
62
+ },
63
+ "runs": [
64
+ "shared_01_cub_MiniCPM",
65
+ "shared_01_cub_Qwen3.8",
66
+ "shared_01_glsim_MiniCPM",
67
+ "shared_01_glsim_Qwen3.8",
68
+ "shared_01_groundingdino_MiniCPM",
69
+ "shared_01_groundingdino_Qwen3.8",
70
+ "shared_01_instancevg_MiniCPM",
71
+ "shared_01_instancevg_Qwen3.8",
72
+ "shared_01_nlvr2_MiniCPM",
73
+ "shared_01_nlvr2_Qwen3.8",
74
+ "shared_01_vilt_MiniCPM",
75
+ "shared_01_vilt_Qwen3.8",
76
+ "shared_01_yolo26x_MiniCPM",
77
+ "shared_01_yolo26x_Qwen3.8",
78
+ "shared_01_rtdetr_x_fp32_MiniCPM",
79
+ "shared_01_rtdetr_x_fp32_Qwen3.8"
80
+ ],
81
+ "status": "complete",
82
+ "task_means": {
83
+ "cub": 0.9346041666666667,
84
+ "grefcoco": 0.7090431513107035,
85
+ "nlvr2": 0.8157691205958177,
86
+ "construction": 0.18298715773892013
87
+ },
88
+ "validation_mean": 0.660600899078027,
89
+ "mean_active_parameters": 4783620.0,
90
+ "mean_fit_seconds": 39.49221354418114
91
+ },
92
+ {
93
+ "id": "shared_02",
94
+ "settings": {
95
+ "student": "4M",
96
+ "representation_dim": 512,
97
+ "lr": 0.001,
98
+ "target": "four_state",
99
+ "strategy": "natural",
100
+ "hidden_percent": 60
101
+ },
102
+ "runs": [
103
+ "shared_02_cub_MiniCPM",
104
+ "shared_02_cub_Qwen3.8",
105
+ "shared_02_glsim_MiniCPM",
106
+ "shared_02_glsim_Qwen3.8",
107
+ "shared_02_groundingdino_MiniCPM",
108
+ "shared_02_groundingdino_Qwen3.8",
109
+ "shared_02_instancevg_MiniCPM",
110
+ "shared_02_instancevg_Qwen3.8",
111
+ "shared_02_nlvr2_MiniCPM",
112
+ "shared_02_nlvr2_Qwen3.8",
113
+ "shared_02_vilt_MiniCPM",
114
+ "shared_02_vilt_Qwen3.8",
115
+ "shared_02_yolo26x_MiniCPM",
116
+ "shared_02_yolo26x_Qwen3.8",
117
+ "shared_02_rtdetr_x_fp32_MiniCPM",
118
+ "shared_02_rtdetr_x_fp32_Qwen3.8"
119
+ ],
120
+ "status": "complete",
121
+ "task_means": {
122
+ "cub": 0.9472291666666667,
123
+ "grefcoco": 0.708183990442055,
124
+ "nlvr2": 0.8155900887997709,
125
+ "construction": 0.1758169352446146
126
+ },
127
+ "validation_mean": 0.6617050452882768,
128
+ "mean_active_parameters": 4816388.0,
129
+ "mean_fit_seconds": 34.18548409075447
130
+ },
131
+ {
132
+ "id": "shared_03",
133
+ "settings": {
134
+ "student": "1M",
135
+ "representation_dim": 512,
136
+ "lr": 0.001,
137
+ "target": "four_state",
138
+ "strategy": "natural",
139
+ "hidden_percent": 90
140
+ },
141
+ "runs": [
142
+ "shared_03_cub_MiniCPM",
143
+ "shared_03_cub_Qwen3.8",
144
+ "shared_03_glsim_MiniCPM",
145
+ "shared_03_glsim_Qwen3.8",
146
+ "shared_03_groundingdino_MiniCPM",
147
+ "shared_03_groundingdino_Qwen3.8",
148
+ "shared_03_instancevg_MiniCPM",
149
+ "shared_03_instancevg_Qwen3.8",
150
+ "shared_03_nlvr2_MiniCPM",
151
+ "shared_03_nlvr2_Qwen3.8",
152
+ "shared_03_vilt_MiniCPM",
153
+ "shared_03_vilt_Qwen3.8",
154
+ "shared_03_yolo26x_MiniCPM",
155
+ "shared_03_yolo26x_Qwen3.8",
156
+ "shared_03_rtdetr_x_fp32_MiniCPM",
157
+ "shared_03_rtdetr_x_fp32_Qwen3.8"
158
+ ],
159
+ "status": "complete",
160
+ "task_means": {
161
+ "cub": 0.9559375,
162
+ "grefcoco": 0.7093348091924943,
163
+ "nlvr2": 0.8166929246634202,
164
+ "construction": 0.173092415345378
165
+ },
166
+ "validation_mean": 0.6637644123003231,
167
+ "mean_active_parameters": 2615822.0,
168
+ "mean_fit_seconds": 57.72181743498368
169
+ },
170
+ {
171
+ "id": "shared_04",
172
+ "settings": {
173
+ "student": "2M",
174
+ "representation_dim": 512,
175
+ "lr": 0.001,
176
+ "target": "four_state",
177
+ "strategy": "natural",
178
+ "hidden_percent": 90
179
+ },
180
+ "runs": [
181
+ "shared_04_cub_MiniCPM",
182
+ "shared_04_cub_Qwen3.8",
183
+ "shared_04_glsim_MiniCPM",
184
+ "shared_04_glsim_Qwen3.8",
185
+ "shared_04_groundingdino_MiniCPM",
186
+ "shared_04_groundingdino_Qwen3.8",
187
+ "shared_04_instancevg_MiniCPM",
188
+ "shared_04_instancevg_Qwen3.8",
189
+ "shared_04_nlvr2_MiniCPM",
190
+ "shared_04_nlvr2_Qwen3.8",
191
+ "shared_04_vilt_MiniCPM",
192
+ "shared_04_vilt_Qwen3.8",
193
+ "shared_04_yolo26x_MiniCPM",
194
+ "shared_04_yolo26x_Qwen3.8",
195
+ "shared_04_rtdetr_x_fp32_MiniCPM",
196
+ "shared_04_rtdetr_x_fp32_Qwen3.8"
197
+ ],
198
+ "status": "complete",
199
+ "task_means": {
200
+ "cub": 0.9448958333333333,
201
+ "grefcoco": 0.7087286527514232,
202
+ "nlvr2": 0.8167144084789458,
203
+ "construction": 0.1764152469711854
204
+ },
205
+ "validation_mean": 0.661688535383722,
206
+ "mean_active_parameters": 3402222.0,
207
+ "mean_fit_seconds": 57.64163886100141
208
+ },
209
+ {
210
+ "id": "shared_05",
211
+ "settings": {
212
+ "student": "8M",
213
+ "representation_dim": 512,
214
+ "lr": 0.001,
215
+ "target": "four_state",
216
+ "strategy": "natural",
217
+ "hidden_percent": 90
218
+ },
219
+ "runs": [
220
+ "shared_05_cub_MiniCPM",
221
+ "shared_05_cub_Qwen3.8",
222
+ "shared_05_glsim_MiniCPM",
223
+ "shared_05_glsim_Qwen3.8",
224
+ "shared_05_groundingdino_MiniCPM",
225
+ "shared_05_groundingdino_Qwen3.8",
226
+ "shared_05_instancevg_MiniCPM",
227
+ "shared_05_instancevg_Qwen3.8",
228
+ "shared_05_nlvr2_MiniCPM",
229
+ "shared_05_nlvr2_Qwen3.8",
230
+ "shared_05_vilt_MiniCPM",
231
+ "shared_05_vilt_Qwen3.8",
232
+ "shared_05_yolo26x_MiniCPM",
233
+ "shared_05_yolo26x_Qwen3.8",
234
+ "shared_05_rtdetr_x_fp32_MiniCPM",
235
+ "shared_05_rtdetr_x_fp32_Qwen3.8"
236
+ ],
237
+ "status": "complete",
238
+ "task_means": {
239
+ "cub": 0.9542916666666666,
240
+ "grefcoco": 0.7095351043643263,
241
+ "nlvr2": 0.8168110856488112,
242
+ "construction": 0.17955466204144832
243
+ },
244
+ "validation_mean": 0.6650481296803131,
245
+ "mean_active_parameters": 7994758.0,
246
+ "mean_fit_seconds": 46.836150376082514
247
+ },
248
+ {
249
+ "id": "shared_06",
250
+ "settings": {
251
+ "student": "8M",
252
+ "representation_dim": 32,
253
+ "lr": 0.001,
254
+ "target": "four_state",
255
+ "strategy": "natural",
256
+ "hidden_percent": 90
257
+ },
258
+ "runs": [
259
+ "shared_06_cub_MiniCPM",
260
+ "shared_06_cub_Qwen3.8",
261
+ "shared_06_glsim_MiniCPM",
262
+ "shared_06_glsim_Qwen3.8",
263
+ "shared_06_groundingdino_MiniCPM",
264
+ "shared_06_groundingdino_Qwen3.8",
265
+ "shared_06_instancevg_MiniCPM",
266
+ "shared_06_instancevg_Qwen3.8",
267
+ "shared_06_nlvr2_MiniCPM",
268
+ "shared_06_nlvr2_Qwen3.8",
269
+ "shared_06_vilt_MiniCPM",
270
+ "shared_06_vilt_Qwen3.8",
271
+ "shared_06_yolo26x_MiniCPM",
272
+ "shared_06_yolo26x_Qwen3.8",
273
+ "shared_06_rtdetr_x_fp32_MiniCPM",
274
+ "shared_06_rtdetr_x_fp32_Qwen3.8"
275
+ ],
276
+ "status": "complete",
277
+ "task_means": {
278
+ "cub": 0.966125,
279
+ "grefcoco": 0.7070665542202544,
280
+ "nlvr2": 0.8169149240905185,
281
+ "construction": 0.17258963439949287
282
+ },
283
+ "validation_mean": 0.6656740281775665,
284
+ "mean_active_parameters": 6222718.0,
285
+ "mean_fit_seconds": 63.611915712936025
286
+ },
287
+ {
288
+ "id": "shared_07",
289
+ "settings": {
290
+ "student": "8M",
291
+ "representation_dim": 64,
292
+ "lr": 0.001,
293
+ "target": "four_state",
294
+ "strategy": "natural",
295
+ "hidden_percent": 90
296
+ },
297
+ "runs": [
298
+ "shared_07_cub_MiniCPM",
299
+ "shared_07_cub_Qwen3.8",
300
+ "shared_07_glsim_MiniCPM",
301
+ "shared_07_glsim_Qwen3.8",
302
+ "shared_07_groundingdino_MiniCPM",
303
+ "shared_07_groundingdino_Qwen3.8",
304
+ "shared_07_instancevg_MiniCPM",
305
+ "shared_07_instancevg_Qwen3.8",
306
+ "shared_07_nlvr2_MiniCPM",
307
+ "shared_07_nlvr2_Qwen3.8",
308
+ "shared_07_vilt_MiniCPM",
309
+ "shared_07_vilt_Qwen3.8",
310
+ "shared_07_yolo26x_MiniCPM",
311
+ "shared_07_yolo26x_Qwen3.8",
312
+ "shared_07_rtdetr_x_fp32_MiniCPM",
313
+ "shared_07_rtdetr_x_fp32_Qwen3.8"
314
+ ],
315
+ "status": "complete",
316
+ "task_means": {
317
+ "cub": 0.9693333333333334,
318
+ "grefcoco": 0.7083755007379295,
319
+ "nlvr2": 0.8167716986536809,
320
+ "construction": 0.17603867039292925
321
+ },
322
+ "validation_mean": 0.6676298007794683,
323
+ "mean_active_parameters": 6340854.0,
324
+ "mean_fit_seconds": 62.082761554156605
325
+ },
326
+ {
327
+ "id": "shared_08",
328
+ "settings": {
329
+ "student": "8M",
330
+ "representation_dim": 128,
331
+ "lr": 0.001,
332
+ "target": "four_state",
333
+ "strategy": "natural",
334
+ "hidden_percent": 90
335
+ },
336
+ "runs": [
337
+ "shared_08_cub_MiniCPM",
338
+ "shared_08_cub_Qwen3.8",
339
+ "shared_08_glsim_MiniCPM",
340
+ "shared_08_glsim_Qwen3.8",
341
+ "shared_08_groundingdino_MiniCPM",
342
+ "shared_08_groundingdino_Qwen3.8",
343
+ "shared_08_instancevg_MiniCPM",
344
+ "shared_08_instancevg_Qwen3.8",
345
+ "shared_08_nlvr2_MiniCPM",
346
+ "shared_08_nlvr2_Qwen3.8",
347
+ "shared_08_vilt_MiniCPM",
348
+ "shared_08_vilt_Qwen3.8",
349
+ "shared_08_yolo26x_MiniCPM",
350
+ "shared_08_yolo26x_Qwen3.8",
351
+ "shared_08_rtdetr_x_fp32_MiniCPM",
352
+ "shared_08_rtdetr_x_fp32_Qwen3.8"
353
+ ],
354
+ "status": "complete",
355
+ "task_means": {
356
+ "cub": 0.9704166666666667,
357
+ "grefcoco": 0.7082156159955021,
358
+ "nlvr2": 0.8169006015468347,
359
+ "construction": 0.17553875298671762
360
+ },
361
+ "validation_mean": 0.6677679092989303,
362
+ "mean_active_parameters": 6577126.0,
363
+ "mean_fit_seconds": 58.37819891100662
364
+ },
365
+ {
366
+ "id": "shared_09",
367
+ "settings": {
368
+ "student": "8M",
369
+ "representation_dim": 256,
370
+ "lr": 0.001,
371
+ "target": "four_state",
372
+ "strategy": "natural",
373
+ "hidden_percent": 90
374
+ },
375
+ "runs": [
376
+ "shared_09_cub_MiniCPM",
377
+ "shared_09_cub_Qwen3.8",
378
+ "shared_09_glsim_MiniCPM",
379
+ "shared_09_glsim_Qwen3.8",
380
+ "shared_09_groundingdino_MiniCPM",
381
+ "shared_09_groundingdino_Qwen3.8",
382
+ "shared_09_instancevg_MiniCPM",
383
+ "shared_09_instancevg_Qwen3.8",
384
+ "shared_09_nlvr2_MiniCPM",
385
+ "shared_09_nlvr2_Qwen3.8",
386
+ "shared_09_vilt_MiniCPM",
387
+ "shared_09_vilt_Qwen3.8",
388
+ "shared_09_yolo26x_MiniCPM",
389
+ "shared_09_yolo26x_Qwen3.8",
390
+ "shared_09_rtdetr_x_fp32_MiniCPM",
391
+ "shared_09_rtdetr_x_fp32_Qwen3.8"
392
+ ],
393
+ "status": "complete",
394
+ "task_means": {
395
+ "cub": 0.9678541666666667,
396
+ "grefcoco": 0.7090659919881931,
397
+ "nlvr2": 0.8167609567459181,
398
+ "construction": 0.17265650809871438
399
+ },
400
+ "validation_mean": 0.6665844058748731,
401
+ "mean_active_parameters": 7049670.0,
402
+ "mean_fit_seconds": 50.47316628225235
403
+ },
404
+ {
405
+ "id": "shared_10",
406
+ "settings": {
407
+ "student": "8M",
408
+ "representation_dim": 128,
409
+ "lr": 0.0001,
410
+ "target": "four_state",
411
+ "strategy": "natural",
412
+ "hidden_percent": 90
413
+ },
414
+ "runs": [
415
+ "shared_10_cub_MiniCPM",
416
+ "shared_10_cub_Qwen3.8",
417
+ "shared_10_glsim_MiniCPM",
418
+ "shared_10_glsim_Qwen3.8",
419
+ "shared_10_groundingdino_MiniCPM",
420
+ "shared_10_groundingdino_Qwen3.8",
421
+ "shared_10_instancevg_MiniCPM",
422
+ "shared_10_instancevg_Qwen3.8",
423
+ "shared_10_nlvr2_MiniCPM",
424
+ "shared_10_nlvr2_Qwen3.8",
425
+ "shared_10_vilt_MiniCPM",
426
+ "shared_10_vilt_Qwen3.8",
427
+ "shared_10_yolo26x_MiniCPM",
428
+ "shared_10_yolo26x_Qwen3.8",
429
+ "shared_10_rtdetr_x_fp32_MiniCPM",
430
+ "shared_10_rtdetr_x_fp32_Qwen3.8"
431
+ ],
432
+ "status": "complete",
433
+ "task_means": {
434
+ "cub": 0.9694166666666667,
435
+ "grefcoco": 0.7047965422728231,
436
+ "nlvr2": 0.8161844743626469,
437
+ "construction": 0.17403212405686308
438
+ },
439
+ "validation_mean": 0.66610745183975,
440
+ "mean_active_parameters": 6577126.0,
441
+ "mean_fit_seconds": 91.61671701988234
442
+ },
443
+ {
444
+ "id": "shared_11",
445
+ "settings": {
446
+ "student": "8M",
447
+ "representation_dim": 128,
448
+ "lr": 0.0003,
449
+ "target": "four_state",
450
+ "strategy": "natural",
451
+ "hidden_percent": 90
452
+ },
453
+ "runs": [
454
+ "shared_11_cub_MiniCPM",
455
+ "shared_11_cub_Qwen3.8",
456
+ "shared_11_glsim_MiniCPM",
457
+ "shared_11_glsim_Qwen3.8",
458
+ "shared_11_groundingdino_MiniCPM",
459
+ "shared_11_groundingdino_Qwen3.8",
460
+ "shared_11_instancevg_MiniCPM",
461
+ "shared_11_instancevg_Qwen3.8",
462
+ "shared_11_nlvr2_MiniCPM",
463
+ "shared_11_nlvr2_Qwen3.8",
464
+ "shared_11_vilt_MiniCPM",
465
+ "shared_11_vilt_Qwen3.8",
466
+ "shared_11_yolo26x_MiniCPM",
467
+ "shared_11_yolo26x_Qwen3.8",
468
+ "shared_11_rtdetr_x_fp32_MiniCPM",
469
+ "shared_11_rtdetr_x_fp32_Qwen3.8"
470
+ ],
471
+ "status": "complete",
472
+ "task_means": {
473
+ "cub": 0.9692083333333333,
474
+ "grefcoco": 0.7071210204511913,
475
+ "nlvr2": 0.8166034087653967,
476
+ "construction": 0.17173346213113433
477
+ },
478
+ "validation_mean": 0.6661665561702639,
479
+ "mean_active_parameters": 6577126.0,
480
+ "mean_fit_seconds": 75.3759171601705
481
+ },
482
+ {
483
+ "id": "shared_12",
484
+ "settings": {
485
+ "student": "8M",
486
+ "representation_dim": 128,
487
+ "lr": 0.003,
488
+ "target": "four_state",
489
+ "strategy": "natural",
490
+ "hidden_percent": 90
491
+ },
492
+ "runs": [
493
+ "shared_12_cub_MiniCPM",
494
+ "shared_12_cub_Qwen3.8",
495
+ "shared_12_glsim_MiniCPM",
496
+ "shared_12_glsim_Qwen3.8",
497
+ "shared_12_groundingdino_MiniCPM",
498
+ "shared_12_groundingdino_Qwen3.8",
499
+ "shared_12_instancevg_MiniCPM",
500
+ "shared_12_instancevg_Qwen3.8",
501
+ "shared_12_nlvr2_MiniCPM",
502
+ "shared_12_nlvr2_Qwen3.8",
503
+ "shared_12_vilt_MiniCPM",
504
+ "shared_12_vilt_Qwen3.8",
505
+ "shared_12_yolo26x_MiniCPM",
506
+ "shared_12_yolo26x_Qwen3.8",
507
+ "shared_12_rtdetr_x_fp32_MiniCPM",
508
+ "shared_12_rtdetr_x_fp32_Qwen3.8"
509
+ ],
510
+ "status": "complete",
511
+ "task_means": {
512
+ "cub": 0.9698125,
513
+ "grefcoco": 0.7082964368543117,
514
+ "nlvr2": 0.8164959896877686,
515
+ "construction": 0.1768778191883302
516
+ },
517
+ "validation_mean": 0.6678706864326026,
518
+ "mean_active_parameters": 6577126.0,
519
+ "mean_fit_seconds": 67.41994623254868
520
+ },
521
+ {
522
+ "id": "shared_13",
523
+ "settings": {
524
+ "student": "8M",
525
+ "representation_dim": 128,
526
+ "lr": 0.01,
527
+ "target": "four_state",
528
+ "strategy": "natural",
529
+ "hidden_percent": 90
530
+ },
531
+ "runs": [
532
+ "shared_13_cub_MiniCPM",
533
+ "shared_13_cub_Qwen3.8",
534
+ "shared_13_glsim_MiniCPM",
535
+ "shared_13_glsim_Qwen3.8",
536
+ "shared_13_groundingdino_MiniCPM",
537
+ "shared_13_groundingdino_Qwen3.8",
538
+ "shared_13_instancevg_MiniCPM",
539
+ "shared_13_instancevg_Qwen3.8",
540
+ "shared_13_nlvr2_MiniCPM",
541
+ "shared_13_nlvr2_Qwen3.8",
542
+ "shared_13_vilt_MiniCPM",
543
+ "shared_13_vilt_Qwen3.8",
544
+ "shared_13_yolo26x_MiniCPM",
545
+ "shared_13_yolo26x_Qwen3.8",
546
+ "shared_13_rtdetr_x_fp32_MiniCPM",
547
+ "shared_13_rtdetr_x_fp32_Qwen3.8"
548
+ ],
549
+ "status": "complete",
550
+ "task_means": {
551
+ "cub": 0.9509375,
552
+ "grefcoco": 0.7100024597652681,
553
+ "nlvr2": 0.8169936980807792,
554
+ "construction": 0.17572600052962756
555
+ },
556
+ "validation_mean": 0.6634149145939187,
557
+ "mean_active_parameters": 6577126.0,
558
+ "mean_fit_seconds": 58.952753572441
559
+ },
560
+ {
561
+ "id": "shared_14",
562
+ "settings": {
563
+ "student": "8M",
564
+ "representation_dim": 128,
565
+ "lr": 0.003,
566
+ "target": "gain",
567
+ "strategy": "natural",
568
+ "hidden_percent": 90
569
+ },
570
+ "runs": [
571
+ "shared_14_cub_MiniCPM",
572
+ "shared_14_cub_Qwen3.8",
573
+ "shared_14_glsim_MiniCPM",
574
+ "shared_14_glsim_Qwen3.8",
575
+ "shared_14_groundingdino_MiniCPM",
576
+ "shared_14_groundingdino_Qwen3.8",
577
+ "shared_14_instancevg_MiniCPM",
578
+ "shared_14_instancevg_Qwen3.8",
579
+ "shared_14_nlvr2_MiniCPM",
580
+ "shared_14_nlvr2_Qwen3.8",
581
+ "shared_14_vilt_MiniCPM",
582
+ "shared_14_vilt_Qwen3.8",
583
+ "shared_14_yolo26x_MiniCPM",
584
+ "shared_14_yolo26x_Qwen3.8",
585
+ "shared_14_rtdetr_x_fp32_MiniCPM",
586
+ "shared_14_rtdetr_x_fp32_Qwen3.8"
587
+ ],
588
+ "status": "complete",
589
+ "task_means": {
590
+ "cub": 0.97975,
591
+ "grefcoco": 0.6943653805608265,
592
+ "nlvr2": 0.815819249498711,
593
+ "construction": 0.17119724948093312
594
+ },
595
+ "validation_mean": 0.6652829698851177,
596
+ "mean_active_parameters": 6576739.0,
597
+ "mean_fit_seconds": 46.42365771972982
598
+ },
599
+ {
600
+ "id": "shared_15",
601
+ "settings": {
602
+ "student": "8M",
603
+ "representation_dim": 128,
604
+ "lr": 0.003,
605
+ "target": "error",
606
+ "strategy": "natural",
607
+ "hidden_percent": 90
608
+ },
609
+ "runs": [
610
+ "shared_15_cub_MiniCPM",
611
+ "shared_15_cub_Qwen3.8",
612
+ "shared_15_glsim_MiniCPM",
613
+ "shared_15_glsim_Qwen3.8",
614
+ "shared_15_groundingdino_MiniCPM",
615
+ "shared_15_groundingdino_Qwen3.8",
616
+ "shared_15_instancevg_MiniCPM",
617
+ "shared_15_instancevg_Qwen3.8",
618
+ "shared_15_nlvr2_MiniCPM",
619
+ "shared_15_nlvr2_Qwen3.8",
620
+ "shared_15_vilt_MiniCPM",
621
+ "shared_15_vilt_Qwen3.8",
622
+ "shared_15_yolo26x_MiniCPM",
623
+ "shared_15_yolo26x_Qwen3.8",
624
+ "shared_15_rtdetr_x_fp32_MiniCPM",
625
+ "shared_15_rtdetr_x_fp32_Qwen3.8"
626
+ ],
627
+ "status": "complete",
628
+ "task_means": {
629
+ "cub": 0.9172083333333333,
630
+ "grefcoco": 0.707660411834985,
631
+ "nlvr2": 0.8152570896591235,
632
+ "construction": 0.17163397920287538
633
+ },
634
+ "validation_mean": 0.6529399535075793,
635
+ "mean_active_parameters": 6576739.0,
636
+ "mean_fit_seconds": 39.98559669695533
637
+ },
638
+ {
639
+ "id": "shared_16",
640
+ "settings": {
641
+ "student": "8M",
642
+ "representation_dim": 128,
643
+ "lr": 0.003,
644
+ "target": "four_state",
645
+ "strategy": "weighted",
646
+ "hidden_percent": 90
647
+ },
648
+ "runs": [
649
+ "shared_16_cub_MiniCPM",
650
+ "shared_16_cub_Qwen3.8",
651
+ "shared_16_glsim_MiniCPM",
652
+ "shared_16_glsim_Qwen3.8",
653
+ "shared_16_groundingdino_MiniCPM",
654
+ "shared_16_groundingdino_Qwen3.8",
655
+ "shared_16_instancevg_MiniCPM",
656
+ "shared_16_instancevg_Qwen3.8",
657
+ "shared_16_nlvr2_MiniCPM",
658
+ "shared_16_nlvr2_Qwen3.8",
659
+ "shared_16_vilt_MiniCPM",
660
+ "shared_16_vilt_Qwen3.8",
661
+ "shared_16_yolo26x_MiniCPM",
662
+ "shared_16_yolo26x_Qwen3.8",
663
+ "shared_16_rtdetr_x_fp32_MiniCPM",
664
+ "shared_16_rtdetr_x_fp32_Qwen3.8"
665
+ ],
666
+ "status": "complete",
667
+ "task_means": {
668
+ "cub": 0.958625,
669
+ "grefcoco": 0.7108879752617893,
670
+ "nlvr2": 0.8163420223431682,
671
+ "construction": 0.17381623998937223
672
+ },
673
+ "validation_mean": 0.6649178093985824,
674
+ "mean_active_parameters": 6577126.0,
675
+ "mean_fit_seconds": 39.95715373130224
676
+ },
677
+ {
678
+ "id": "shared_17",
679
+ "settings": {
680
+ "student": "8M",
681
+ "representation_dim": 128,
682
+ "lr": 0.003,
683
+ "target": "four_state",
684
+ "strategy": "balanced",
685
+ "hidden_percent": 90
686
+ },
687
+ "runs": [
688
+ "shared_17_cub_MiniCPM",
689
+ "shared_17_cub_Qwen3.8",
690
+ "shared_17_glsim_MiniCPM",
691
+ "shared_17_glsim_Qwen3.8",
692
+ "shared_17_groundingdino_MiniCPM",
693
+ "shared_17_groundingdino_Qwen3.8",
694
+ "shared_17_instancevg_MiniCPM",
695
+ "shared_17_instancevg_Qwen3.8",
696
+ "shared_17_nlvr2_MiniCPM",
697
+ "shared_17_nlvr2_Qwen3.8",
698
+ "shared_17_vilt_MiniCPM",
699
+ "shared_17_vilt_Qwen3.8",
700
+ "shared_17_yolo26x_MiniCPM",
701
+ "shared_17_yolo26x_Qwen3.8",
702
+ "shared_17_rtdetr_x_fp32_MiniCPM",
703
+ "shared_17_rtdetr_x_fp32_Qwen3.8"
704
+ ],
705
+ "status": "complete",
706
+ "task_means": {
707
+ "cub": 0.9554583333333334,
708
+ "grefcoco": 0.7062583456321596,
709
+ "nlvr2": 0.8156903466055572,
710
+ "construction": 0.17510649266382114
711
+ },
712
+ "validation_mean": 0.6631283795587178,
713
+ "mean_active_parameters": 6577126.0,
714
+ "mean_fit_seconds": 35.547329766683106
715
+ }
716
+ ],
717
+ "failures": [],
718
+ "test_loaded": false,
719
+ "finished_at": 1789043234.371715
720
+ }
paper_results/data/table_audit.json ADDED
@@ -0,0 +1,443 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "counts": {
3
+ "better_metric": 11,
4
+ "fewer_calls": 15,
5
+ "both": 10,
6
+ "above_endpoints": 5,
7
+ "better_ranked_area": 15
8
+ },
9
+ "rows": [
10
+ {
11
+ "key": [
12
+ "cub",
13
+ "cub",
14
+ "MiniCPM"
15
+ ],
16
+ "validation": {
17
+ "spec": 98.58333333333333,
18
+ "vlm": 27.750000000000004,
19
+ "conf": 98.5,
20
+ "call_conf": 1.0,
21
+ "learned": 98.58333333333333,
22
+ "call_learned": 1.0,
23
+ "change_pp": 0.0,
24
+ "metric_change_pp": 0.08333333333333526
25
+ },
26
+ "test": {
27
+ "spec": 81.429064549534,
28
+ "vlm": 26.561960648947185,
29
+ "conf": 79.9275112185019,
30
+ "call_conf": 11.3565757680359,
31
+ "learned": 81.37728684846392,
32
+ "call_learned": 2.8822920262340355,
33
+ "change_pp": -8.474283741801864,
34
+ "metric_change_pp": 1.4497756299620246
35
+ }
36
+ },
37
+ {
38
+ "key": [
39
+ "cub",
40
+ "cub",
41
+ "Qwen3.8"
42
+ ],
43
+ "validation": {
44
+ "spec": 98.58333333333333,
45
+ "vlm": 55.08333333333333,
46
+ "conf": 98.25,
47
+ "call_conf": 1.0,
48
+ "learned": 98.58333333333333,
49
+ "call_learned": 1.0,
50
+ "change_pp": 0.0,
51
+ "metric_change_pp": 0.33333333333332993
52
+ },
53
+ "test": {
54
+ "spec": 81.429064549534,
55
+ "vlm": 54.81532619951675,
56
+ "conf": 81.58439765274423,
57
+ "call_conf": 11.3565757680359,
58
+ "learned": 81.34276838108389,
59
+ "call_learned": 1.4325163962720056,
60
+ "change_pp": -9.924059371763894,
61
+ "metric_change_pp": -0.24162927166033743
62
+ }
63
+ },
64
+ {
65
+ "key": [
66
+ "cub",
67
+ "glsim",
68
+ "MiniCPM"
69
+ ],
70
+ "validation": {
71
+ "spec": 98.91666666666666,
72
+ "vlm": 27.750000000000004,
73
+ "conf": 98.5,
74
+ "call_conf": 1.0,
75
+ "learned": 99.0,
76
+ "call_learned": 3.0,
77
+ "change_pp": 1.9999999999999998,
78
+ "metric_change_pp": 0.5000000000000004
79
+ },
80
+ "test": {
81
+ "spec": 90.16223679668623,
82
+ "vlm": 26.561960648947185,
83
+ "conf": 87.69416637901277,
84
+ "call_conf": 8.698653779772178,
85
+ "learned": 89.79979288919571,
86
+ "call_learned": 4.487400759406282,
87
+ "change_pp": -4.211253020365896,
88
+ "metric_change_pp": 2.105626510182945
89
+ }
90
+ },
91
+ {
92
+ "key": [
93
+ "cub",
94
+ "glsim",
95
+ "Qwen3.8"
96
+ ],
97
+ "validation": {
98
+ "spec": 98.91666666666666,
99
+ "vlm": 55.08333333333333,
100
+ "conf": 98.58333333333333,
101
+ "call_conf": 1.0,
102
+ "learned": 98.83333333333333,
103
+ "call_learned": 1.0,
104
+ "change_pp": 0.0,
105
+ "metric_change_pp": 0.24999999999999467
106
+ },
107
+ "test": {
108
+ "spec": 90.16223679668623,
109
+ "vlm": 54.81532619951675,
110
+ "conf": 88.78149810148429,
111
+ "call_conf": 8.698653779772178,
112
+ "learned": 90.05868139454608,
113
+ "call_learned": 1.6568864342423197,
114
+ "change_pp": -7.041767345529859,
115
+ "metric_change_pp": 1.27718329306179
116
+ }
117
+ },
118
+ {
119
+ "key": [
120
+ "grefcoco",
121
+ "groundingdino",
122
+ "MiniCPM"
123
+ ],
124
+ "validation": {
125
+ "spec": 56.05453650994448,
126
+ "vlm": 64.12256658935975,
127
+ "conf": 63.72197624569541,
128
+ "call_conf": 98.99501019045611,
129
+ "learned": 68.00899571298054,
130
+ "call_learned": 33.993956005341204,
131
+ "change_pp": -65.00105418511491,
132
+ "metric_change_pp": 4.287019467285125
133
+ },
134
+ "test": {
135
+ "spec": 26.688029946402747,
136
+ "vlm": 50.00708958398321,
137
+ "conf": 49.68380455434875,
138
+ "call_conf": 98.73805405098828,
139
+ "learned": 45.622890848764996,
140
+ "call_learned": 43.51870232254772,
141
+ "change_pp": -55.21935172844057,
142
+ "metric_change_pp": -4.060913705583752
143
+ }
144
+ },
145
+ {
146
+ "key": [
147
+ "grefcoco",
148
+ "groundingdino",
149
+ "Qwen3.8"
150
+ ],
151
+ "validation": {
152
+ "spec": 56.05453650994448,
153
+ "vlm": 80.60299388572633,
154
+ "conf": 79.94237121371846,
155
+ "call_conf": 98.99501019045611,
156
+ "learned": 81.53067678684377,
157
+ "call_learned": 41.998734977862114,
158
+ "change_pp": -56.99627521259401,
159
+ "metric_change_pp": 1.5883055731253037
160
+ },
161
+ "test": {
162
+ "spec": 26.688029946402747,
163
+ "vlm": 74.47182599325072,
164
+ "conf": 73.87062927147436,
165
+ "call_conf": 98.73805405098828,
166
+ "learned": 66.10044522587415,
167
+ "call_learned": 57.456824433542245,
168
+ "change_pp": -41.281229617446044,
169
+ "metric_change_pp": -7.770184045600203
170
+ }
171
+ },
172
+ {
173
+ "key": [
174
+ "grefcoco",
175
+ "instancevg",
176
+ "MiniCPM"
177
+ ],
178
+ "validation": {
179
+ "spec": 75.02986857825567,
180
+ "vlm": 64.12256658935975,
181
+ "conf": 77.16635041113219,
182
+ "call_conf": 64.99402628434886,
183
+ "learned": 77.29988052568699,
184
+ "call_learned": 22.995291306486752,
185
+ "change_pp": -41.99873497786211,
186
+ "metric_change_pp": 0.1335301145547918
187
+ },
188
+ "test": {
189
+ "spec": 66.95119530385956,
190
+ "vlm": 50.00708958398321,
191
+ "conf": 61.877889005473165,
192
+ "call_conf": 52.09426310864078,
193
+ "learned": 66.71865694921021,
194
+ "call_learned": 30.904914499617163,
195
+ "change_pp": -21.189348609023618,
196
+ "metric_change_pp": 4.840767943737056
197
+ }
198
+ },
199
+ {
200
+ "key": [
201
+ "grefcoco",
202
+ "instancevg",
203
+ "Qwen3.8"
204
+ ],
205
+ "validation": {
206
+ "spec": 75.02986857825567,
207
+ "vlm": 80.60299388572633,
208
+ "conf": 82.01560193970062,
209
+ "call_conf": 70.99585353854803,
210
+ "learned": 80.81383090870757,
211
+ "call_learned": 55.99831330381615,
212
+ "change_pp": -14.997540234731888,
213
+ "metric_change_pp": -1.2017710309930374
214
+ },
215
+ "test": {
216
+ "spec": 66.95119530385956,
217
+ "vlm": 74.47182599325072,
218
+ "conf": 72.43286163967898,
219
+ "call_conf": 68.38612710206165,
220
+ "learned": 75.28287440093015,
221
+ "call_learned": 53.55755324277571,
222
+ "change_pp": -14.828573859285942,
223
+ "metric_change_pp": 2.850012761251175
224
+ }
225
+ },
226
+ {
227
+ "key": [
228
+ "nlvr2",
229
+ "nlvr2",
230
+ "MiniCPM"
231
+ ],
232
+ "validation": {
233
+ "spec": 79.81953594958465,
234
+ "vlm": 89.90260670295045,
235
+ "conf": 90.04583213978803,
236
+ "call_conf": 85.87797192781437,
237
+ "learned": 89.94557433400172,
238
+ "call_learned": 84.98997421942137,
239
+ "change_pp": -0.8879977083930068,
240
+ "metric_change_pp": -0.10025780578630794
241
+ },
242
+ "test": {
243
+ "spec": 80.00287005811867,
244
+ "vlm": 90.7368874219703,
245
+ "conf": 90.77276314845376,
246
+ "call_conf": 85.80038745784601,
247
+ "learned": 90.64361053311329,
248
+ "call_learned": 84.65953935567195,
249
+ "change_pp": -1.1408481021740635,
250
+ "metric_change_pp": -0.12915261534046651
251
+ }
252
+ },
253
+ {
254
+ "key": [
255
+ "nlvr2",
256
+ "nlvr2",
257
+ "Qwen3.8"
258
+ ],
259
+ "validation": {
260
+ "spec": 79.81953594958465,
261
+ "vlm": 90.50415353766829,
262
+ "conf": 90.57576625608708,
263
+ "call_conf": 85.87797192781437,
264
+ "learned": 90.9481523918648,
265
+ "call_learned": 76.99799484388427,
266
+ "change_pp": -8.879977083930102,
267
+ "metric_change_pp": 0.3723861357777247
268
+ },
269
+ "test": {
270
+ "spec": 80.00287005811867,
271
+ "vlm": 90.06242376408122,
272
+ "conf": 90.24180239649853,
273
+ "call_conf": 85.80038745784601,
274
+ "learned": 90.88039032790414,
275
+ "call_learned": 76.65925234986008,
276
+ "change_pp": -9.141135107985932,
277
+ "metric_change_pp": 0.6385879314056098
278
+ }
279
+ },
280
+ {
281
+ "key": [
282
+ "nlvr2",
283
+ "vilt",
284
+ "MiniCPM"
285
+ ],
286
+ "validation": {
287
+ "spec": 75.56574047550845,
288
+ "vlm": 89.90260670295045,
289
+ "conf": 89.97421942136924,
290
+ "call_conf": 95.9896877685477,
291
+ "learned": 89.8166714408479,
292
+ "call_learned": 98.99742194213692,
293
+ "change_pp": 3.007734173589227,
294
+ "metric_change_pp": -0.15754798052134422
295
+ },
296
+ "test": {
297
+ "spec": 76.42247255506925,
298
+ "vlm": 90.7368874219703,
299
+ "conf": 90.64361053311329,
300
+ "call_conf": 95.93169261677549,
301
+ "learned": 90.70818684078353,
302
+ "call_learned": 98.96677907727631,
303
+ "change_pp": 3.0350864605008243,
304
+ "metric_change_pp": 0.0645763076702277
305
+ }
306
+ },
307
+ {
308
+ "key": [
309
+ "nlvr2",
310
+ "vilt",
311
+ "Qwen3.8"
312
+ ],
313
+ "validation": {
314
+ "spec": 75.56574047550845,
315
+ "vlm": 90.50415353766829,
316
+ "conf": 90.59008879977084,
317
+ "call_conf": 95.9896877685477,
318
+ "learned": 90.41821827556575,
319
+ "call_learned": 94.98710971068462,
320
+ "change_pp": -1.0025780578630794,
321
+ "metric_change_pp": -0.17187052420509774
322
+ },
323
+ "test": {
324
+ "spec": 76.42247255506925,
325
+ "vlm": 90.06242376408122,
326
+ "conf": 89.96197172992753,
327
+ "call_conf": 95.93169261677549,
328
+ "learned": 89.997847456411,
329
+ "call_learned": 94.85542082227165,
330
+ "change_pp": -1.0762717945038358,
331
+ "metric_change_pp": 0.035875726483469705
332
+ }
333
+ },
334
+ {
335
+ "key": [
336
+ "construction",
337
+ "yolo26x",
338
+ "MiniCPM"
339
+ ],
340
+ "validation": {
341
+ "spec": 18.254072168965788,
342
+ "vlm": 20.666016863629594,
343
+ "conf": 22.641590025257546,
344
+ "call_conf": 93.86590584878745,
345
+ "learned": 29.007041551798586,
346
+ "call_learned": 48.93009985734665,
347
+ "change_pp": -44.935805991440795,
348
+ "metric_change_pp": 6.3654515265410385
349
+ },
350
+ "test": {
351
+ "spec": 18.129624718318375,
352
+ "vlm": 22.361115727328066,
353
+ "conf": 25.23047077304935,
354
+ "call_conf": 92.54327563249002,
355
+ "learned": 25.82183145054675,
356
+ "call_learned": 47.20372836218375,
357
+ "change_pp": -45.33954727030626,
358
+ "metric_change_pp": 0.591360677497399
359
+ }
360
+ },
361
+ {
362
+ "key": [
363
+ "construction",
364
+ "yolo26x",
365
+ "Qwen3.8"
366
+ ],
367
+ "validation": {
368
+ "spec": 18.254072168965788,
369
+ "vlm": 25.448626222520154,
370
+ "conf": 26.42878791936722,
371
+ "call_conf": 92.86733238231099,
372
+ "learned": 33.12666068966333,
373
+ "call_learned": 49.928673323823105,
374
+ "change_pp": -42.93865905848788,
375
+ "metric_change_pp": 6.697872770296115
376
+ },
377
+ "test": {
378
+ "spec": 18.129624718318375,
379
+ "vlm": 27.001009067886244,
380
+ "conf": 25.974900047621052,
381
+ "call_conf": 92.21038615179761,
382
+ "learned": 29.875800068449216,
383
+ "call_learned": 48.76830892143808,
384
+ "change_pp": -43.44207723035953,
385
+ "metric_change_pp": 3.9009000208281632
386
+ }
387
+ },
388
+ {
389
+ "key": [
390
+ "construction",
391
+ "rtdetr_x_fp32",
392
+ "MiniCPM"
393
+ ],
394
+ "validation": {
395
+ "spec": 7.641657905799261,
396
+ "vlm": 20.666016863629594,
397
+ "conf": 20.572745422879233,
398
+ "call_conf": 97.8601997146933,
399
+ "learned": 21.744085592769803,
400
+ "call_learned": 77.88873038516405,
401
+ "change_pp": -19.971469329529246,
402
+ "metric_change_pp": 1.1713401698905694
403
+ },
404
+ "test": {
405
+ "spec": 7.8873978710439285,
406
+ "vlm": 22.361115727328066,
407
+ "conf": 21.609512872414097,
408
+ "call_conf": 97.90279627163781,
409
+ "learned": 22.996109134220227,
410
+ "call_learned": 78.82822902796272,
411
+ "change_pp": -19.0745672436751,
412
+ "metric_change_pp": 1.3865962618061305
413
+ }
414
+ },
415
+ {
416
+ "key": [
417
+ "construction",
418
+ "rtdetr_x_fp32",
419
+ "Qwen3.8"
420
+ ],
421
+ "validation": {
422
+ "spec": 7.641657905799261,
423
+ "vlm": 25.448626222520154,
424
+ "conf": 24.862274646934207,
425
+ "call_conf": 98.85877318116975,
426
+ "learned": 27.197394026479483,
427
+ "call_learned": 82.88159771754636,
428
+ "change_pp": -15.977175463623395,
429
+ "metric_change_pp": 2.335119379545275
430
+ },
431
+ "test": {
432
+ "spec": 7.8873978710439285,
433
+ "vlm": 27.001009067886244,
434
+ "conf": 26.818431283513146,
435
+ "call_conf": 98.60186418109188,
436
+ "learned": 26.693203615113315,
437
+ "call_learned": 83.12250332889481,
438
+ "change_pp": -15.479360852197077,
439
+ "metric_change_pp": -0.1252276683998299
440
+ }
441
+ }
442
+ ]
443
+ }
paper_results/figures/build_compact.py ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Plot archived results without rerunning any model."""
2
+ from pathlib import Path
3
+ import json
4
+ import numpy as np
5
+ import matplotlib
6
+ matplotlib.use("Agg")
7
+ import matplotlib.pyplot as plt
8
+ from matplotlib.patches import Patch, Rectangle
9
+ from matplotlib.ticker import MaxNLocator
10
+ root=Path(__file__).resolve().parents[1]
11
+ groups=json.loads((root/"data/results.json").read_text(encoding="utf-8"))["groups"]
12
+ scaling=json.loads((root/"data/scaling_selected.json").read_text(encoding="utf-8"))
13
+ names={"cub":"CUB-200-2011","grefcoco":"gRefCOCO","nlvr2":"NLVR2","construction":"ConstructionSite"}
14
+ plt.rcParams.update({"font.family":"serif","font.serif":["Times New Roman"],"font.size":8,
15
+ "axes.titlesize":8.5,"axes.labelsize":8,"xtick.labelsize":7,"ytick.labelsize":7,"legend.fontsize":7.5,
16
+ "pdf.fonttype":42,"ps.fonttype":42,"axes.spines.top":False,"axes.spines.right":False,"axes.linewidth":.6,"mathtext.fontset":"stix"})
17
+ colors=["#a7c8c0","#2c8c69","#d77960","#d5d9df"]
18
+ curve_colors={"learned":"#176f91","confidence":"#c78324","small":"#536174","large":"#8a537b"}
19
+ fig,axes=plt.subplots(2,4,figsize=(7.16,2.75),gridspec_kw={"height_ratios":[.66,1.15]})
20
+ fig.subplots_adjust(left=.064,right=.978,bottom=.19,top=.85,wspace=.30,hspace=.67)
21
+ for col,(task,expert) in enumerate(zip(names,["cub","instancevg","nlvr2","yolo26x"])):
22
+ subset=[g for g in groups if g["key"][0]==task]
23
+ chosen=next(g for g in subset if g["key"][1:]==[expert,"Qwen3.8"])
24
+ top,ax=axes[:,col]
25
+ xs=np.array([0,1,3,4])
26
+ floor=np.zeros(4)
27
+ for state,color in zip(["both_correct","rescuable","harmful","both_wrong"],colors):
28
+ values=np.array([100*g["opportunity"]["test"][state]/g["opportunity"]["test"]["total"] for g in subset])
29
+ top.bar(xs,values,bottom=floor,width=.72,color=color,linewidth=0,zorder=3)
30
+ floor+=values
31
+ assert np.allclose(floor,100)
32
+ selected_bar=next(i for i,g in enumerate(subset) if g["key"]==chosen["key"])
33
+ top.add_patch(Rectangle((xs[selected_bar]-.36,0),.72,100,fill=False,edgecolor="#263d46",lw=.8,zorder=5,clip_on=False))
34
+ top.set(ylim=(0,101),xlim=(-.65,4.65),yticks=[0,50,100],xticks=xs)
35
+ top.set_xticklabels(["M","Q","M","Q"])
36
+ top.tick_params(axis="x",length=0,pad=1)
37
+ for x,label in ((.5,subset[0]["expert_display"]),(3.5,subset[2]["expert_display"])):
38
+ top.text(x,-.31,label,ha="center",va="top",transform=top.get_xaxis_transform(),fontsize=6.7)
39
+ top.set_title(names[task],fontweight="bold",pad=3)
40
+ top.grid(axis="y",color="#dce1e4",lw=.4,zorder=0)
41
+ if col==0:
42
+ top.set_ylabel("Outcome (%)",labelpad=2)
43
+ else:
44
+ top.tick_params(labelleft=False)
45
+ for method,label in (("learned","Learned"),("confidence","conf.")):
46
+ saved=chosen["methods"][method]
47
+ curve=saved["test_curve"]
48
+ calls=[100*p["actual_fraction"] for p in curve]
49
+ assert np.all(np.diff(calls)>=-1e-10)
50
+ ax.plot(calls,[100*p["metric"] for p in curve],lw=1.15,color=curve_colors[method],label=label)
51
+ p=saved["test_point"]
52
+ ax.scatter(100*p["actual_fraction"],100*p["metric"],s=16,color=curve_colors[method],edgecolor="white",linewidth=.4,zorder=5,clip_on=False)
53
+ for model,label,ls in (("small","Specialist","--"),("large","VLM",":")):
54
+ ax.axhline(100*chosen[model]["test"],ls=ls,lw=.9,color=curve_colors[model],label=label)
55
+ ax.set(xlim=(0,100),xticks=[0,50,100])
56
+ ax.set_title(chosen["expert_display"]+" / Q",fontsize=8,pad=3)
57
+ ax.yaxis.set_major_locator(MaxNLocator(nbins=3))
58
+ ax.grid(axis="y",color="#dce1e4",lw=.4)
59
+ if col==0:
60
+ ax.set_ylabel("Accuracy (%)",labelpad=2)
61
+ elif col==3:
62
+ ax.set_ylabel("Macro-F1 (%)",labelpad=1)
63
+ fig.legend([Patch(facecolor=c) for c in colors],["Both correct","Rescue","Harm","Both wrong"],loc="upper center",bbox_to_anchor=(.51,1.01),ncol=4,frameon=False,handlelength=1.3,columnspacing=1.7)
64
+ handles,labels=axes[1,0].get_legend_handles_labels()
65
+ fig.legend(handles,labels,loc="lower center",bbox_to_anchor=(.52,-.01),ncol=4,frameon=False,handlelength=2,columnspacing=1.8)
66
+ fig.text(.52,.083,"Actual test calls (%)",ha="center",fontsize=8)
67
+ fig.savefig(root/"figures/gains_compact.pdf")
68
+ fig.savefig(root/"figures/gains_compact.png",dpi=300)
69
+ plt.close(fig)
70
+
71
+ fig,axes=plt.subplots(2,4,figsize=(7.16,4.3),sharex=True)
72
+ fig.subplots_adjust(left=.075,right=.985,bottom=.12,top=.90,hspace=.42,wspace=.38)
73
+ for row,endpoint in enumerate(("MiniCPM","Qwen3.8")):
74
+ for col,task in enumerate(names):
75
+ ax=axes[row,col]
76
+ selected=[r for r in scaling if r["key"][0]==task and r["key"][2]==endpoint]
77
+ for entry,color,marker in zip(selected,["#176f91","#c78324"],["o","s"]):
78
+ name=next(g["expert_display"] for g in groups if g["key"]==entry["key"])
79
+ ax.plot([25,50,100],[100*r["rank25_metric"] for r in entry["rows"]],marker=marker,lw=1.2,ms=4,color=color,label=name)
80
+ ax.set_title(names[task]+(" / M" if row==0 else " / Q"),fontweight="bold")
81
+ ax.set_xticks([25,50,100])
82
+ ax.yaxis.set_major_locator(MaxNLocator(nbins=4))
83
+ ax.grid(axis="y",lw=.45,color="#dce1e4")
84
+ ax.legend(loc="best",frameon=False,fontsize=6.9,handlelength=1)
85
+ if col==0:
86
+ ax.set_ylabel("Test metric (%)")
87
+ fig.suptitle("Earlier configurations: exactly 25% test calls",fontsize=11,fontweight="bold")
88
+ fig.text(.52,.035,"Router training data (%)",ha="center",fontsize=9)
89
+ fig.savefig(root/"figures/scaling_ranked.pdf")
90
+ fig.savefig(root/"figures/scaling_ranked.png",dpi=240)
91
+ plt.close(fig)
92
+
93
+ fig,axes=plt.subplots(1,2,figsize=(7.16,3.0),sharey=True)
94
+ fig.subplots_adjust(left=.16,right=.98,top=.84,bottom=.19,wspace=.32)
95
+ for ax,endpoint in zip(axes,("MiniCPM","Qwen3.8")):
96
+ records=json.loads((root/f"data/joint_{endpoint}_comparison_20260910.json").read_text(encoding="utf-8"))
97
+ subset=[g for g in groups if g["key"][2]==endpoint]
98
+ vals=[100*next(j["delta_multi_minus_single"] for j in records if j["expert"]==g["key"][1]) for g in subset]
99
+ ax.barh(np.arange(8),vals,color=["#2c8c69" if v>=0 else "#d77960" for v in vals],height=.64)
100
+ ax.axvline(0,color="#8c949b",lw=.65)
101
+ ax.set_yticks(np.arange(8),[g["expert_display"] for g in subset])
102
+ ax.set_title(endpoint.replace("MiniCPM","MiniCPM-V-4.5"),fontweight="bold")
103
+ ax.grid(axis="x",color="#dce1e4",lw=.4)
104
+ ax.set_axisbelow(True)
105
+ ax.set_xlim(-1.9,4.2)
106
+ axes[0].invert_yaxis()
107
+ fig.text(.55,.045,"Joint minus single-task test metric (percentage points)",ha="center",fontsize=9)
108
+ fig.savefig(root/"figures/joint_delta.pdf")
109
+ fig.savefig(root/"figures/joint_delta.png",dpi=240)
110
+ plt.close(fig)
111
+ print("Generated compact gains, matched-call data scaling, and joint-training figures.")
paper_results/figures/build_cost.py ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Restyle measured-component cost estimates; no model execution."""
2
+ from pathlib import Path
3
+ import csv
4
+ import json
5
+ import numpy as np
6
+ import matplotlib
7
+ matplotlib.use("Agg")
8
+ import matplotlib.pyplot as plt
9
+ from matplotlib.ticker import MaxNLocator, LogLocator, NullLocator
10
+
11
+ root = Path(__file__).resolve().parents[1]
12
+ data = root / "data/cost_20260911"
13
+ summary = json.loads((data / "cost_summary.json").read_text(encoding="utf-8"))
14
+ with (data / "full_cost_performance_curves.csv").open(encoding="utf-8-sig", newline="") as f:
15
+ rows = [r for r in csv.DictReader(f) if r["method"] in ("both", "confidence")]
16
+ methods = [("both", "learned", "#176f91", "-"), ("confidence", "conf.", "#c78324", "--")]
17
+ pairs = list(summary["components"])
18
+ plt.rcParams.update({"font.family":"serif", "font.serif":["Times New Roman"], "font.size":8,
19
+ "axes.labelsize":8, "xtick.labelsize":7, "ytick.labelsize":7, "legend.fontsize":7.5,
20
+ "pdf.fonttype":42, "ps.fonttype":42, "axes.spines.top":False, "axes.spines.right":False,
21
+ "axes.linewidth":.6, "mathtext.fontset":"stix"})
22
+
23
+ def panel(ax, pair, field, log):
24
+ comp = summary["components"][pair]
25
+ for method, label, color, ls in methods:
26
+ rr = [r for r in rows if r["pair_key"] == pair and r["method"] == method]
27
+ assert len(rr) == 101
28
+ x = np.array([float(r[field]) for r in rr])
29
+ # Confidence does not execute the learned head; correct the shared offset.
30
+ offset = comp["router_gflops_per_input"] if "flops" in field else comp["router_latency_ms"]
31
+ if method == "confidence":
32
+ x -= offset
33
+ y = 100 * np.array([float(r["test_metric"]) for r in rr])
34
+ sd = 100 * np.array([float(r["test_metric_std"]) for r in rr])
35
+ assert np.all(np.diff(x) >= -1e-8)
36
+ ax.plot(x, y, color=color, lw=1.15, ls=ls, label=label)
37
+ if method == "both":
38
+ ax.fill_between(x, y - sd, y + sd, color=color, alpha=.12, lw=0)
39
+ p = next(p for p in summary["selected_points"] if p["pair_key"] == pair and p["method"] == method)
40
+ ax.scatter(p[field] - (offset if method == "confidence" else 0), 100 * p["test_metric"], s=19,
41
+ color=color, edgecolor="white", linewidth=.45, zorder=5)
42
+ if log:
43
+ ax.set_xscale("log")
44
+ ax.xaxis.set_major_locator(LogLocator(base=10, numticks=5))
45
+ ax.xaxis.set_minor_locator(NullLocator())
46
+ else:
47
+ ax.xaxis.set_major_locator(MaxNLocator(nbins=3))
48
+ ax.yaxis.set_major_locator(MaxNLocator(nbins=4))
49
+ ax.grid(axis="y", color="#dce1e4", lw=.4)
50
+ ax.set_ylabel("Accuracy (%)" if pair.startswith("grefcoco") else "Macro-F1 (%)", labelpad=2)
51
+ ax.tick_params(pad=2)
52
+
53
+ for field, name, xlabel, log in (("flops_gflops_per_input", "flops_performance", "Estimated GFLOPs / input", True),
54
+ ("latency_ms_per_input", "latency_performance", "Estimated time / input (ms)", False)):
55
+ fig, axes = plt.subplots(1, 2, figsize=(7.16, 2.35))
56
+ fig.subplots_adjust(left=.075, right=.99, top=.87, bottom=.28, wspace=.30)
57
+ for ax, pair, label in zip(axes, pairs, ("(a) gRefCOCO: InstanceVG / M", "(b) ConstructionSite: YOLO26x / Q")):
58
+ panel(ax, pair, field, log)
59
+ ax.set_xlabel(xlabel)
60
+ ax.text(.5, 1.08, label, transform=ax.transAxes, ha="center", fontsize=8)
61
+ handles, labels = axes[0].get_legend_handles_labels()
62
+ fig.legend(handles, labels, loc="lower center", bbox_to_anchor=(.52, -.015), ncol=2, frameon=False, handlelength=2.3)
63
+ fig.savefig(root / f"figures/{name}.pdf")
64
+ fig.savefig(root / f"figures/{name}.png", dpi=300)
65
+ plt.close(fig)
66
+
67
+ fig, axes = plt.subplots(2, 2, figsize=(3.44, 2.30))
68
+ fig.subplots_adjust(left=.135, right=.99, top=.88, bottom=.23, wspace=.40, hspace=.62)
69
+ for col, pair in enumerate(pairs):
70
+ for row, field in enumerate(("flops_gflops_per_input", "latency_ms_per_input")):
71
+ ax = axes[row, col]
72
+ panel(ax, pair, field, row == 0)
73
+ ax.tick_params(labelsize=6.5)
74
+ ax.yaxis.set_major_locator(MaxNLocator(nbins=3))
75
+ ax.set_ylabel("Acc. (%)" if col == 0 else "F1 (%)", fontsize=7)
76
+ ax.set_xlabel("Est. GFLOPs / input" if row == 0 else "Est. time / input (ms)", fontsize=7, labelpad=1)
77
+ if row == 0:
78
+ ax.text(.5, 1.17, "gRefCOCO / M" if col == 0 else "ConstructionSite / Q", transform=ax.transAxes, ha="center", fontsize=7.2)
79
+ handles, labels = axes[0, 0].get_legend_handles_labels()
80
+ fig.legend(handles, labels, loc="lower center", bbox_to_anchor=(.54, .0), ncol=2, frameon=False, handlelength=2)
81
+ fig.savefig(root / "figures/cost_compact.pdf")
82
+ fig.savefig(root / "figures/cost_compact.png", dpi=300)
83
+ plt.close(fig)
84
+ print("Created two matching standalone plots and a four-panel paper figure.")
paper_results/figures/build_tables.py ADDED
@@ -0,0 +1,123 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Rebuild tables from saved results; no model execution."""
2
+ from pathlib import Path
3
+ import json
4
+
5
+ root = Path(__file__).resolve().parents[1]
6
+ groups = json.loads((root/"data/results.json").read_text(encoding="utf-8"))["groups"]
7
+ diagnostics = json.loads((root/"data/budget_diagnostics.json").read_text(encoding="utf-8"))
8
+ earlier = json.loads((root/"data/earlier_results.json").read_text(encoding="utf-8"))["rows"]
9
+ task_names = {"cub":"CUB-200-2011","grefcoco":"gRefCOCO","nlvr2":"NLVR2","construction":"ConstructionSite"}
10
+ joint = []
11
+ for endpoint in ("MiniCPM","Qwen3.8"):
12
+ joint.extend(json.loads((root/f"data/joint_{endpoint}_comparison_20260910.json").read_text(encoding="utf-8")))
13
+ rows = {name:[] for name in ("main_test","main_full","thresholds","outcomes_full","scaling_full","joint_full")}
14
+ scaling, audit = [], []
15
+ counts = dict(better_metric=0,fewer_calls=0,both=0,above_endpoints=0,better_ranked_area=0)
16
+ for index,g in enumerate(groups):
17
+ task,expert,endpoint = g["key"]
18
+ prefix = f"{task_names[task] if index%4==0 else ''} & {g['expert_display']} & {'M' if endpoint=='MiniCPM' else 'Q'}"
19
+ if index and index%4==0:
20
+ for table in rows.values():
21
+ table.append(r"\midrule")
22
+ blocks, raw = [], {}
23
+ for split in ("validation","test"):
24
+ conf = g["methods"]["confidence"][f"{split}_point"]
25
+ learned = g["methods"]["learned"][f"{split}_point"]
26
+ change = 100*(learned["actual_fraction"]-conf["actual_fraction"])
27
+ macro = "callup" if change>1e-10 else "calldown" if change < -1e-10 else "samechg"
28
+ delta = rf"\{macro}{{{abs(change):.2f}}}"
29
+ metric_change = 100*(learned["metric"]-conf["metric"])
30
+ metric_macro = "upchg" if metric_change>1e-10 else "downchg" if metric_change < -1e-10 else "samechg"
31
+ metric_delta = rf"\{metric_macro}{{{abs(metric_change):.2f}}}"
32
+ cv,lv = f"{100*conf['metric']:.2f}",f"{100*learned['metric']:.2f}"
33
+ if conf["metric"]>=learned["metric"]:
34
+ cv = rf"\textbf{{{cv}}}"
35
+ if learned["metric"]>=conf["metric"]:
36
+ lv = rf"\textbf{{{lv}}}"
37
+ if split=="test" and learned["metric"]>max(g["small"][split],g["large"][split]):
38
+ lv += r"$^\dagger$"
39
+ blocks.append(f"{100*g['small'][split]:.2f} & {100*g['large'][split]:.2f} & {cv} & {100*conf['actual_fraction']:.2f} & {lv}{metric_delta} & {100*learned['actual_fraction']:.2f}{delta}")
40
+ raw[split] = dict(spec=100*g["small"][split],vlm=100*g["large"][split],conf=100*conf["metric"],call_conf=100*conf["actual_fraction"],learned=100*learned["metric"],call_learned=100*learned["actual_fraction"],change_pp=change,metric_change_pp=metric_change)
41
+ for method in ("confidence","learned"):
42
+ p=g["methods"][method]
43
+ assert p[f"{split}_point"]["threshold"]==p["validation_point"]["threshold"]
44
+ assert p[f"{split}_point"]["operator"]==p["validation_point"]["operator"]
45
+ rows["main_test"].append(prefix+" & "+blocks[1]+r" \\")
46
+ rows["main_full"].append(prefix+" & "+" & ".join(blocks)+r" \\")
47
+ selected=g["methods"]["learned"]["test_point"]
48
+ reference=g["methods"]["confidence"]["test_point"]
49
+ better=selected["metric"]>reference["metric"]
50
+ fewer=selected["actual_fraction"]<reference["actual_fraction"]
51
+ counts["better_metric"]+=better
52
+ counts["fewer_calls"]+=fewer
53
+ counts["both"]+=better and fewer
54
+ counts["above_endpoints"]+=selected["metric"]>max(g["small"]["test"],g["large"]["test"])
55
+ d=next(d for d in diagnostics if d["key"]==g["key"])
56
+ counts["better_ranked_area"]+=d["areas"]["learned"]["test_ranked_A25_diagnostic"]>d["areas"]["confidence"]["test_ranked_A25_diagnostic"]
57
+ cutoffs=[]
58
+ for method in ("confidence","learned"):
59
+ p=g["methods"][method]["validation_point"]
60
+ op=r"\geq" if p["operator"]=="ge" else ">"
61
+ cutoffs.extend([f"{100*p['budget']:.0f}","$"+f"{op}{p['threshold']:.6g}"+"$"])
62
+ rows["thresholds"].append(prefix+" & "+" & ".join(cutoffs)+r" \\")
63
+ opp=g["opportunity"]["test"]
64
+ keys=("both_correct","rescuable","harmful","both_wrong")
65
+ assert sum(opp[k] for k in keys)==opp["total"]
66
+ gain=100*(selected["metric"]-g["small"]["test"])
67
+ if task!="construction":
68
+ assert abs(gain-100*(selected["rescued"]-selected["harmed"])/g["test_rows"])<1e-9
69
+ values=[opp[k] for k in keys]+[selected[k] for k in ("rescued","harmed","missed_rescue","avoided_harm")]
70
+ rows["outcomes_full"].append(prefix+" & "+" & ".join(f"{v:,}" for v in values)+f" & {gain:+.3f}"+r" \\")
71
+ scale=[]
72
+ for fraction,tag in ((.25,"baseline_25pct"),(.5,"baseline_50pct"),(1.,"baseline_100pct")):
73
+ matches=[r for r in earlier if (r["task"],r["expert"],r["endpoint"],r["tag"])==(task,expert,endpoint,tag)]
74
+ assert len(matches)==1,(g["key"],tag,len(matches))
75
+ assert matches[0]["fraction"]==fraction
76
+ scale.append(matches[0])
77
+ for field in ("student","branches","fusion","target","strategy","representation_dim","hidden_layer"):
78
+ assert all(r[field]==scale[0][field] for r in scale),(g["key"],field)
79
+ rows["scaling_full"].append(prefix+" & "+" & ".join(f"{100*r['fixed25_metric']:.2f} & {100*r['fixed25_actual_fraction']:.2f} & {100*r['rank25_metric']:.2f}" for r in scale)+r" \\")
80
+ scaling.append(dict(key=g["key"],rows=scale))
81
+ j=next(j for j in joint if (j["expert"],j["endpoint"])==(expert,endpoint))
82
+ rows["joint_full"].append(prefix+f" & {100*j['single_task_test_fixed25']:.3f} & {100*j['multi_task_test_fixed25']:.3f} & {100*j['delta_multi_minus_single']:+.3f}"+r" \\")
83
+ audit.append(dict(key=g["key"],**raw))
84
+ assert counts==dict(better_metric=11,fewer_calls=15,both=10,above_endpoints=5,better_ranked_area=15),counts
85
+
86
+ head=r"\textcolor{testink}{spec.} & \textcolor{testink}{vlm.} & \textcolor{confink}{conf.} & \textcolor{confink}{call$_{\rm conf}$} & \textcolor{learnink}{learned$\uparrow$} & \textcolor{learnink}{call$_{\rm learned}\downarrow$}"
87
+ cols=r">{\columncolor{basefill}}r>{\columncolor{basefill}}r>{\columncolor{conffill}}r>{\columncolor{conffill}}r>{\columncolor{learnfill}}r>{\columncolor{learnfill}}r"
88
+ specs=[
89
+ ("main_test","table*","t","tab:main","lll"+cols, "Dataset & Specialist & VLM & "+head,
90
+ r"Test results at validation-selected thresholds. Scores and actual call rates are percentages; parentheses give learned-minus-conf. differences in percentage points. \textcolor{upgreen}{Green} marks improvement and \textcolor{downred}{red} degradation: higher scores and fewer calls are better. Bold compares routed scores; $\dagger$ exceeds both models. M: MiniCPM-V-4.5; Q: Qwen3.8-27B-FP8. ",8,10.0),
91
+ ("main_full","table","H","tab:full","lll"+cols+cols,
92
+ r"\multirow{2}{*}{Task} & \multirow{2}{*}{Specialist} & \multirow{2}{*}{VLM} & \multicolumn{6}{c}{\cellcolor{valfill}\textcolor{valink}{Validation}} & \multicolumn{6}{c}{\cellcolor{testfill}\textcolor{testink}{Test}} \\ \cmidrule(lr){4-9}\cmidrule(l){10-15} & & & "+head+" & "+head,
93
+ r"Complete four-way comparison. Scores and actual call rates are percentages. Thresholds selected on the 1--99\% validation grid transfer unchanged to test. Parentheses give learned-minus-confidence differences in percentage points before rounding. Green marks improvement and red degradation: higher scores and fewer calls are better. Bold marks the higher routed score; $\dagger$ exceeds both test endpoints. M: MiniCPM-V-4.5; Q: Qwen3.8-27B-FP8.",8,1.8),
94
+ ("thresholds","table","H","tab:thresholds","lllrrrr",
95
+ r"Task & Specialist & VLM & Conf. budget & Conf. cutoff & Learned budget & Learned cutoff",
96
+ "Selected nominal validation budgets and score cutoffs. The operator is part of each stored threshold. Displayed cutoffs are rounded; the archived JSON retains full precision. Budget is a percentage, not the numerical threshold.",9,4),
97
+ ("outcomes_full","table","H","tab:outcomes_full","lllrrrrrrrrr",
98
+ r"Task & Specialist & VLM & BC & R & H & BW & Sel. R & Sel. H & Miss R & Avoid H & Gain (pp)",
99
+ "Test outcome counts at the learned operating point. BC/BW: both correct/wrong; R/H: rescue/harm. Selected rescues, selected harms, missed rescues and avoided harms are counted separately. Construction counts 12,016 events, with routing over 3,004 images. Metric gains are recomputed; macro-F1 is not an event-count ratio.",8.5,3),
100
+ ("scaling_full","table","H","tab:scaling_full","lllrrrrrrrrr",
101
+ r"Task & Specialist & VLM & \multicolumn{3}{c}{25\% training} & \multicolumn{3}{c}{50\% training} & \multicolumn{3}{c}{100\% training}\\ \cmidrule(lr){4-6}\cmidrule(lr){7-9}\cmidrule(l){10-12} & & & Metric & Calls & Rank & Metric & Calls & Rank & Metric & Calls & Rank",
102
+ r"Earlier training-data comparisons. Each pair retains its exploratory reference configuration; only router-training fraction changes. Within each fraction: test metric, actual test calls, and test metric at exactly 25\% ranked calls. The first two use a validation-selected 25\% threshold. These are not final shared-recipe ablations. All entries are percentages.",8.5,3),
103
+ ("joint_full","table","H","tab:joint_full","lllrrr",
104
+ r"Task & Specialist & VLM & Single & Joint & Difference (pp)",
105
+ r"Earlier single-task versus joint-training comparison. Both use 4M students, image/text inputs and specialist-error supervision. Metrics use validation-selected 25\% thresholds; actual call rates are not present in this comparison file. Differences are joint minus single in percentage points. All tasks were seen during training.",9,5)]
106
+ for name,env,place,label,spec,header,caption,size,pad in specs:
107
+ content=f"\\begin{{{env}}}[{place}]\n\\centering\n\\caption{{{caption}}}\n\\label{{{label}}}\n"
108
+ content+=f"\\fontsize{{{size}}}{{{size*1.2}}}\\selectfont\n\\setlength{{\\tabcolsep}}{{{pad}pt}}\n\\renewcommand{{\\arraystretch}}{{1.12}}\n"
109
+ content+="\\begin{tabular}{@{}"+spec+"@{}}\n\\toprule\n"+header+r" \\"+"\n\\midrule\n"
110
+ content+="\n".join(rows[name])+"\n\\bottomrule\n\\end{tabular}\n"+f"\\end{{{env}}}\n"
111
+ if name == "main_test":
112
+ content = content.replace(r"\begin{table*}[t]", r"\begin{minipage}{\textwidth}")
113
+ content = content.replace(r"\caption{", r"\captionof{table}{")
114
+ content = content.replace(r"\end{table*}", r"\end{minipage}")
115
+ content = content.replace(
116
+ r">{\columncolor{learnfill}}r@{}}",
117
+ r">{\columncolor{learnfill}[\tabcolsep][0pt]}r@{}}",
118
+ )
119
+ (root/f"tables/{name}.tex").write_text(content,encoding="utf-8")
120
+ (root/"data/table_audit.json").write_text(json.dumps(dict(counts=counts,rows=audit),indent=2),encoding="utf-8")
121
+ (root/"data/scaling_selected.json").write_text(json.dumps(scaling,indent=2),encoding="utf-8")
122
+ print(json.dumps(counts))
123
+ print("Rebuilt all 16-pair tables from saved results.")
paper_results/figures/cost_includes.tex ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ % Standalone insertions for the updated three-seed router, not the revision-17 policy.
2
+ \begin{figure*}[t]
3
+ \centering
4
+ \includegraphics[width=\textwidth]{figures/flops_performance.pdf}
5
+ \caption{Test performance versus estimated computation for InstanceVG/MiniCPM and YOLO26x/Qwen. Learned uses the probability contrast $p_R-p_H$ and validation task-performance checkpoint selection; conf. uses specialist confidence. Curves average seeds 42--44; shading is one population standard deviation and dots average validation-selected policies. All thresholds are fixed before test evaluation. Costs combine measured components with actual test call fractions; VLM decoding is estimated from mean output length. Router feature encoders and unsupported operations are excluded. These are component-based estimates, not full-system counts.}
6
+ \label{fig:new_cost_flops}
7
+ \end{figure*}
8
+
9
+ \begin{figure*}[t]
10
+ \centering
11
+ \includegraphics[width=\textwidth]{figures/latency_performance.pdf}
12
+ \caption{Test performance versus estimated amortized time per input under the same three-seed protocol. Time combines A800 specialist measurements, the routing head on cached features, and VLM batch wall time weighted by actual test calls. The VLM probe uses batch size four. Feature-encoder computation, loading, warmup and deployment queueing are excluded. This estimate is not a measurement of online per-request response latency. Learned and conf. have the same meaning as in Fig.~\ref{fig:new_cost_flops}.}
13
+ \label{fig:new_cost_time}
14
+ \end{figure*}
paper_results/figures/gains_compact.pdf ADDED
Binary file (73.5 kB). View file
 
paper_results/figures/gains_compact.png ADDED

Git LFS Details

  • SHA256: cdedff2c5768301c64a47f429a77494f05a9c57b269a07ebb93c7708dcfe4b4b
  • Pointer size: 131 Bytes
  • Size of remote file: 181 kB
paper_results/figures/joint_delta.pdf ADDED
Binary file (51.6 kB). View file
 
paper_results/figures/joint_delta.png ADDED
paper_results/figures/scaling_ranked.pdf ADDED
Binary file (67 kB). View file