Release main-table weights and compact reproduction data
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- .gitattributes +4 -0
- README.md +47 -0
- appendix.pdf +3 -0
- experiment_data.tar.gz +3 -0
- main.pdf +3 -0
- paper_results/data/budget_diagnostics.json +370 -0
- paper_results/data/comparisons.json +0 -0
- paper_results/data/earlier_results.json +0 -0
- paper_results/data/joint_MiniCPM_comparison_20260910.json +58 -0
- paper_results/data/joint_Qwen3.8_comparison_20260910.json +58 -0
- paper_results/data/layout_verification.json +29 -0
- paper_results/data/main_table_cost_qwen_20260912/build_figures.py +101 -0
- paper_results/data/main_table_cost_qwen_20260912/components/instancevg_router.json +239 -0
- paper_results/data/main_table_cost_qwen_20260912/components/instancevg_specialist.json +254 -0
- paper_results/data/main_table_cost_qwen_20260912/components/qwen_flops.json +341 -0
- paper_results/data/main_table_cost_qwen_20260912/components/qwen_gref_flops.json +341 -0
- paper_results/data/main_table_cost_qwen_20260912/components/qwen_gref_latency.json +2446 -0
- paper_results/data/main_table_cost_qwen_20260912/components/qwen_latency.json +0 -0
- paper_results/data/main_table_cost_qwen_20260912/components/yolo26x_router.json +238 -0
- paper_results/data/main_table_cost_qwen_20260912/components/yolo26x_specialist.json +254 -0
- paper_results/data/main_table_cost_qwen_20260912/cost_summary.json +75 -0
- paper_results/data/main_table_cost_qwen_20260912/fixed_budget_cost_performance.csv +29 -0
- paper_results/data/main_table_cost_qwen_20260912/full_cost_performance_curves.csv +405 -0
- paper_results/data/main_table_cost_qwen_20260912/main_table_results.json +0 -0
- paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_flops.py +152 -0
- paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_large.py +209 -0
- paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_router.py +104 -0
- paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_specialist.py +258 -0
- paper_results/data/main_table_cost_qwen_20260912/reproducible/prepare_output.py +19 -0
- paper_results/data/main_table_cost_qwen_20260912/reproducible/read_main.py +10 -0
- paper_results/data/main_table_cost_qwen_20260912/reproducible/setup_local.py +6 -0
- paper_results/data/main_table_cost_qwen_20260912/reproducible/switch_qwen.py +23 -0
- paper_results/data/main_table_cost_qwen_20260912/selected_points.csv +5 -0
- paper_results/data/main_table_cost_qwen_20260912/verify_and_report.py +57 -0
- paper_results/data/main_table_cost_qwen_20260912/成本性能报告.md +50 -0
- paper_results/data/method_improvement_spec.json +114 -0
- paper_results/data/operation_measurements.json +129 -0
- paper_results/data/results.json +0 -0
- paper_results/data/scaling_selected.json +1250 -0
- paper_results/data/shared_search.json +720 -0
- paper_results/data/table_audit.json +443 -0
- paper_results/figures/build_compact.py +111 -0
- paper_results/figures/build_cost.py +84 -0
- paper_results/figures/build_tables.py +123 -0
- paper_results/figures/cost_includes.tex +14 -0
- paper_results/figures/gains_compact.pdf +0 -0
- paper_results/figures/gains_compact.png +3 -0
- paper_results/figures/joint_delta.pdf +0 -0
- paper_results/figures/joint_delta.png +0 -0
- paper_results/figures/scaling_ranked.pdf +0 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
appendix.pdf filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
main.pdf filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
paper_results/figures/gains_compact.png filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
paper_results/figures/scaling_ranked.png filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
tags:
|
| 5 |
+
- computer-vision
|
| 6 |
+
- model-routing
|
| 7 |
+
- reproducibility
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
+
# ModelCollaboration: trained weights and paper results
|
| 11 |
+
|
| 12 |
+
Code: https://github.com/nohi191212/ModelCollaboration
|
| 13 |
+
|
| 14 |
+
## Contents
|
| 15 |
+
|
| 16 |
+
- `weights.tar`: 16 main-table router checkpoints and configurations, training-set normalization statistics, the final 8M distilled image/text encoders, and the trained ConstructionSite YOLO26x and RT-DETR-X specialists. Detector extraction configurations record the checkpoint source and inference settings. RT-DETR uses FP32 inference in the recorded experiment.
|
| 17 |
+
- `experiment_data.tar.gz`: per-example validation/test learned scores and task outcomes for all 16 pairs, sample identifiers, selected validation thresholds, complete learned curves and the selection record.
|
| 18 |
+
- `paper_results/`: source data and scripts for paper tables and figures. `data/results.json` includes confidence and learned curves, fixed selected points and outcome decompositions. `data/main_table_cost_qwen_20260912/` contains the final Qwen cost curves and component measurements. Earlier ablations are explicitly separate from the main setting.
|
| 19 |
+
- `main.pdf` and `appendix.pdf`: the paper and supplementary material accompanying this release.
|
| 20 |
+
|
| 21 |
+
## Recompute the reported routing results
|
| 22 |
+
|
| 23 |
+
Download this repository, then extract the two archives into its root:
|
| 24 |
+
|
| 25 |
+
```bash
|
| 26 |
+
tar -xf weights.tar
|
| 27 |
+
tar -xzf experiment_data.tar.gz
|
| 28 |
+
```
|
| 29 |
+
|
| 30 |
+
From the code repository, run:
|
| 31 |
+
|
| 32 |
+
```bash
|
| 33 |
+
pip install numpy
|
| 34 |
+
python scripts/replay_paper_results.py --artifacts /path/to/download --output replayed_main.csv
|
| 35 |
+
```
|
| 36 |
+
|
| 37 |
+
This checks all saved learned validation/test curve points against the per-example outcomes, repeats validation-only operating-point selection, and exports the selected results. It does not execute the specialist or VLM. ConstructionSite uses event macro-F1; other tasks use their saved correctness indicators.
|
| 38 |
+
|
| 39 |
+
To redraw the paper tables and outcome/budget figure, install matplotlib and run `paper_results/figures/build_tables.py` and `paper_results/figures/build_compact.py`. The source tables in `paper_results/tables/` retain the final typeset captions and labels. Cost data for the final figure are under `paper_results/data/main_table_cost_qwen_20260912/`; costs are assembled from measured components rather than a new timing run during replay.
|
| 40 |
+
|
| 41 |
+
## Running the models again
|
| 42 |
+
|
| 43 |
+
The weights archive contains `main/runs/<run>/best.pt`, its `config.json` and normalization arrays; `encoders/8M/epoch_300.pt`, tokenizer and encoder configuration; and `specialists/<expert>/best.pt` with extraction settings. Match the 16 run names to `configs/main/` in the code repository. Preserve the frozen validation thresholds when evaluating test samples.
|
| 44 |
+
|
| 45 |
+
Original dataset images, upstream VLM weights and full feature caches are external. Obtain upstream resources from the code repository's `docs/DOWNLOADS.md`. The full prepared feature cache is several GB and is excluded to keep non-weight experiment data below 100 MB. Running from original images requires preparing those features and predictions in the documented layout; the compact result replay above is the supported route for checking the published numbers without large-model inference. The release does not claim a one-command end-to-end reproduction from raw images.
|
| 46 |
+
|
| 47 |
+
The original source datasets and upstream model terms continue to apply. This model card does not grant a new license for third-party datasets or detector architectures.
|
appendix.pdf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f1f37970088137fb21832d56d5b793365463a54a291b05bca237aab50cf9a0ef
|
| 3 |
+
size 2430363
|
experiment_data.tar.gz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7ee5a0063cae30a79812aff05d9302196687411c0441040252a375f3f6e2a5fa
|
| 3 |
+
size 2457041
|
main.pdf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ac6d6f4cb7eb00f64e8e05789465e01aa865a1745dbcfb91dd0debb2344bf857
|
| 3 |
+
size 577296
|
paper_results/data/budget_diagnostics.json
ADDED
|
@@ -0,0 +1,370 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"key": [
|
| 4 |
+
"cub",
|
| 5 |
+
"cub",
|
| 6 |
+
"MiniCPM"
|
| 7 |
+
],
|
| 8 |
+
"areas": {
|
| 9 |
+
"confidence": {
|
| 10 |
+
"validation_fixed_A25": 0.9058333333333334,
|
| 11 |
+
"test_fixed_A25": 0.6472903003106663,
|
| 12 |
+
"test_ranked_A25_diagnostic": 0.7895926820849155,
|
| 13 |
+
"selected_validation_actual_calls": 0.01,
|
| 14 |
+
"selected_test_actual_calls": 0.113565757680359
|
| 15 |
+
},
|
| 16 |
+
"learned": {
|
| 17 |
+
"validation_fixed_A25": 0.9621666666666666,
|
| 18 |
+
"test_fixed_A25": 0.7763203313772868,
|
| 19 |
+
"test_ranked_A25_diagnostic": 0.7967552640662755,
|
| 20 |
+
"selected_validation_actual_calls": 0.01,
|
| 21 |
+
"selected_test_actual_calls": 0.028822920262340353
|
| 22 |
+
}
|
| 23 |
+
}
|
| 24 |
+
},
|
| 25 |
+
{
|
| 26 |
+
"key": [
|
| 27 |
+
"cub",
|
| 28 |
+
"cub",
|
| 29 |
+
"Qwen3.8"
|
| 30 |
+
],
|
| 31 |
+
"areas": {
|
| 32 |
+
"confidence": {
|
| 33 |
+
"validation_fixed_A25": 0.9171666666666667,
|
| 34 |
+
"test_fixed_A25": 0.7316189161201243,
|
| 35 |
+
"test_ranked_A25_diagnostic": 0.8101484294097343,
|
| 36 |
+
"selected_validation_actual_calls": 0.01,
|
| 37 |
+
"selected_test_actual_calls": 0.113565757680359
|
| 38 |
+
},
|
| 39 |
+
"learned": {
|
| 40 |
+
"validation_fixed_A25": 0.9705833333333334,
|
| 41 |
+
"test_fixed_A25": 0.8053158439765276,
|
| 42 |
+
"test_ranked_A25_diagnostic": 0.806368657231619,
|
| 43 |
+
"selected_validation_actual_calls": 0.01,
|
| 44 |
+
"selected_test_actual_calls": 0.014325163962720055
|
| 45 |
+
}
|
| 46 |
+
}
|
| 47 |
+
},
|
| 48 |
+
{
|
| 49 |
+
"key": [
|
| 50 |
+
"cub",
|
| 51 |
+
"glsim",
|
| 52 |
+
"MiniCPM"
|
| 53 |
+
],
|
| 54 |
+
"areas": {
|
| 55 |
+
"confidence": {
|
| 56 |
+
"validation_fixed_A25": 0.9005000000000001,
|
| 57 |
+
"test_fixed_A25": 0.7253365550569555,
|
| 58 |
+
"test_ranked_A25_diagnostic": 0.8519848118743528,
|
| 59 |
+
"selected_validation_actual_calls": 0.01,
|
| 60 |
+
"selected_test_actual_calls": 0.08698653779772178
|
| 61 |
+
},
|
| 62 |
+
"learned": {
|
| 63 |
+
"validation_fixed_A25": 0.97075,
|
| 64 |
+
"test_fixed_A25": 0.8740248532965137,
|
| 65 |
+
"test_ranked_A25_diagnostic": 0.8809112875388333,
|
| 66 |
+
"selected_validation_actual_calls": 0.03,
|
| 67 |
+
"selected_test_actual_calls": 0.04487400759406282
|
| 68 |
+
}
|
| 69 |
+
}
|
| 70 |
+
},
|
| 71 |
+
{
|
| 72 |
+
"key": [
|
| 73 |
+
"cub",
|
| 74 |
+
"glsim",
|
| 75 |
+
"Qwen3.8"
|
| 76 |
+
],
|
| 77 |
+
"areas": {
|
| 78 |
+
"confidence": {
|
| 79 |
+
"validation_fixed_A25": 0.9213333333333334,
|
| 80 |
+
"test_fixed_A25": 0.7860545391784605,
|
| 81 |
+
"test_ranked_A25_diagnostic": 0.8691232309285468,
|
| 82 |
+
"selected_validation_actual_calls": 0.01,
|
| 83 |
+
"selected_test_actual_calls": 0.08698653779772178
|
| 84 |
+
},
|
| 85 |
+
"learned": {
|
| 86 |
+
"validation_fixed_A25": 0.9757500000000001,
|
| 87 |
+
"test_fixed_A25": 0.8918536416983087,
|
| 88 |
+
"test_ranked_A25_diagnostic": 0.8935623058336211,
|
| 89 |
+
"selected_validation_actual_calls": 0.01,
|
| 90 |
+
"selected_test_actual_calls": 0.016568864342423197
|
| 91 |
+
}
|
| 92 |
+
}
|
| 93 |
+
},
|
| 94 |
+
{
|
| 95 |
+
"key": [
|
| 96 |
+
"grefcoco",
|
| 97 |
+
"groundingdino",
|
| 98 |
+
"MiniCPM"
|
| 99 |
+
],
|
| 100 |
+
"areas": {
|
| 101 |
+
"confidence": {
|
| 102 |
+
"validation_fixed_A25": 0.5566729917773561,
|
| 103 |
+
"test_fixed_A25": 0.27538212857669514,
|
| 104 |
+
"test_ranked_A25_diagnostic": 0.2786603522105323,
|
| 105 |
+
"selected_validation_actual_calls": 0.9899501019045611,
|
| 106 |
+
"selected_test_actual_calls": 0.9873805405098829
|
| 107 |
+
},
|
| 108 |
+
"learned": {
|
| 109 |
+
"validation_fixed_A25": 0.6255183076814955,
|
| 110 |
+
"test_fixed_A25": 0.33085386949493806,
|
| 111 |
+
"test_ranked_A25_diagnostic": 0.3352494115645294,
|
| 112 |
+
"selected_validation_actual_calls": 0.339939560053412,
|
| 113 |
+
"selected_test_actual_calls": 0.43518702322547714
|
| 114 |
+
}
|
| 115 |
+
}
|
| 116 |
+
},
|
| 117 |
+
{
|
| 118 |
+
"key": [
|
| 119 |
+
"grefcoco",
|
| 120 |
+
"groundingdino",
|
| 121 |
+
"Qwen3.8"
|
| 122 |
+
],
|
| 123 |
+
"areas": {
|
| 124 |
+
"confidence": {
|
| 125 |
+
"validation_fixed_A25": 0.5648886077728582,
|
| 126 |
+
"test_fixed_A25": 0.28625187873975555,
|
| 127 |
+
"test_ranked_A25_diagnostic": 0.29324220854720245,
|
| 128 |
+
"selected_validation_actual_calls": 0.9899501019045611,
|
| 129 |
+
"selected_test_actual_calls": 0.9873805405098829
|
| 130 |
+
},
|
| 131 |
+
"learned": {
|
| 132 |
+
"validation_fixed_A25": 0.669126431934781,
|
| 133 |
+
"test_fixed_A25": 0.3549726342058248,
|
| 134 |
+
"test_ranked_A25_diagnostic": 0.37351898590590704,
|
| 135 |
+
"selected_validation_actual_calls": 0.4199873497786211,
|
| 136 |
+
"selected_test_actual_calls": 0.5745682443354224
|
| 137 |
+
}
|
| 138 |
+
}
|
| 139 |
+
},
|
| 140 |
+
{
|
| 141 |
+
"key": [
|
| 142 |
+
"grefcoco",
|
| 143 |
+
"instancevg",
|
| 144 |
+
"MiniCPM"
|
| 145 |
+
],
|
| 146 |
+
"areas": {
|
| 147 |
+
"confidence": {
|
| 148 |
+
"validation_fixed_A25": 0.7465176751704266,
|
| 149 |
+
"test_fixed_A25": 0.6639253608598248,
|
| 150 |
+
"test_ranked_A25_diagnostic": 0.6590023537418825,
|
| 151 |
+
"selected_validation_actual_calls": 0.6499402628434886,
|
| 152 |
+
"selected_test_actual_calls": 0.5209426310864078
|
| 153 |
+
},
|
| 154 |
+
"learned": {
|
| 155 |
+
"validation_fixed_A25": 0.7607140347178298,
|
| 156 |
+
"test_fixed_A25": 0.6759152652922327,
|
| 157 |
+
"test_ranked_A25_diagnostic": 0.6777273629583416,
|
| 158 |
+
"selected_validation_actual_calls": 0.22995291306486754,
|
| 159 |
+
"selected_test_actual_calls": 0.30904914499617164
|
| 160 |
+
}
|
| 161 |
+
}
|
| 162 |
+
},
|
| 163 |
+
{
|
| 164 |
+
"key": [
|
| 165 |
+
"grefcoco",
|
| 166 |
+
"instancevg",
|
| 167 |
+
"Qwen3.8"
|
| 168 |
+
],
|
| 169 |
+
"areas": {
|
| 170 |
+
"confidence": {
|
| 171 |
+
"validation_fixed_A25": 0.7477616136060159,
|
| 172 |
+
"test_fixed_A25": 0.6640728242066756,
|
| 173 |
+
"test_ranked_A25_diagnostic": 0.6611405722712191,
|
| 174 |
+
"selected_validation_actual_calls": 0.7099585353854804,
|
| 175 |
+
"selected_test_actual_calls": 0.6838612710206166
|
| 176 |
+
},
|
| 177 |
+
"learned": {
|
| 178 |
+
"validation_fixed_A25": 0.7778129172816081,
|
| 179 |
+
"test_fixed_A25": 0.7280974392422653,
|
| 180 |
+
"test_ranked_A25_diagnostic": 0.7136828970875989,
|
| 181 |
+
"selected_validation_actual_calls": 0.5599831330381615,
|
| 182 |
+
"selected_test_actual_calls": 0.5355755324277571
|
| 183 |
+
}
|
| 184 |
+
}
|
| 185 |
+
},
|
| 186 |
+
{
|
| 187 |
+
"key": [
|
| 188 |
+
"nlvr2",
|
| 189 |
+
"nlvr2",
|
| 190 |
+
"MiniCPM"
|
| 191 |
+
],
|
| 192 |
+
"areas": {
|
| 193 |
+
"confidence": {
|
| 194 |
+
"validation_fixed_A25": 0.8284302492122602,
|
| 195 |
+
"test_fixed_A25": 0.833902561526871,
|
| 196 |
+
"test_ranked_A25_diagnostic": 0.835882901628758,
|
| 197 |
+
"selected_validation_actual_calls": 0.8587797192781438,
|
| 198 |
+
"selected_test_actual_calls": 0.8580038745784602
|
| 199 |
+
},
|
| 200 |
+
"learned": {
|
| 201 |
+
"validation_fixed_A25": 0.8359782297336007,
|
| 202 |
+
"test_fixed_A25": 0.8425629618999785,
|
| 203 |
+
"test_ranked_A25_diagnostic": 0.8414221137978044,
|
| 204 |
+
"selected_validation_actual_calls": 0.8498997421942137,
|
| 205 |
+
"selected_test_actual_calls": 0.8465953935567195
|
| 206 |
+
}
|
| 207 |
+
}
|
| 208 |
+
},
|
| 209 |
+
{
|
| 210 |
+
"key": [
|
| 211 |
+
"nlvr2",
|
| 212 |
+
"nlvr2",
|
| 213 |
+
"Qwen3.8"
|
| 214 |
+
],
|
| 215 |
+
"areas": {
|
| 216 |
+
"confidence": {
|
| 217 |
+
"validation_fixed_A25": 0.831280435405328,
|
| 218 |
+
"test_fixed_A25": 0.8343976465523427,
|
| 219 |
+
"test_ranked_A25_diagnostic": 0.8361340317141422,
|
| 220 |
+
"selected_validation_actual_calls": 0.8587797192781438,
|
| 221 |
+
"selected_test_actual_calls": 0.8580038745784602
|
| 222 |
+
},
|
| 223 |
+
"learned": {
|
| 224 |
+
"validation_fixed_A25": 0.8402320252076769,
|
| 225 |
+
"test_fixed_A25": 0.8436320585491857,
|
| 226 |
+
"test_ranked_A25_diagnostic": 0.8425342613187918,
|
| 227 |
+
"selected_validation_actual_calls": 0.7699799484388428,
|
| 228 |
+
"selected_test_actual_calls": 0.7665925234986009
|
| 229 |
+
}
|
| 230 |
+
}
|
| 231 |
+
},
|
| 232 |
+
{
|
| 233 |
+
"key": [
|
| 234 |
+
"nlvr2",
|
| 235 |
+
"vilt",
|
| 236 |
+
"MiniCPM"
|
| 237 |
+
],
|
| 238 |
+
"areas": {
|
| 239 |
+
"confidence": {
|
| 240 |
+
"validation_fixed_A25": 0.7926095674591808,
|
| 241 |
+
"test_fixed_A25": 0.7987371744277822,
|
| 242 |
+
"test_ranked_A25_diagnostic": 0.8005237856066586,
|
| 243 |
+
"selected_validation_actual_calls": 0.9598968776854769,
|
| 244 |
+
"selected_test_actual_calls": 0.9593169261677549
|
| 245 |
+
},
|
| 246 |
+
"learned": {
|
| 247 |
+
"validation_fixed_A25": 0.793425952449155,
|
| 248 |
+
"test_fixed_A25": 0.8005094353160652,
|
| 249 |
+
"test_ranked_A25_diagnostic": 0.8015211308028988,
|
| 250 |
+
"selected_validation_actual_calls": 0.9899742194213692,
|
| 251 |
+
"selected_test_actual_calls": 0.9896677907727631
|
| 252 |
+
}
|
| 253 |
+
}
|
| 254 |
+
},
|
| 255 |
+
{
|
| 256 |
+
"key": [
|
| 257 |
+
"nlvr2",
|
| 258 |
+
"vilt",
|
| 259 |
+
"Qwen3.8"
|
| 260 |
+
],
|
| 261 |
+
"areas": {
|
| 262 |
+
"confidence": {
|
| 263 |
+
"validation_fixed_A25": 0.7940561443712404,
|
| 264 |
+
"test_fixed_A25": 0.7988735021884195,
|
| 265 |
+
"test_ranked_A25_diagnostic": 0.8006242376408124,
|
| 266 |
+
"selected_validation_actual_calls": 0.9598968776854769,
|
| 267 |
+
"selected_test_actual_calls": 0.9593169261677549
|
| 268 |
+
},
|
| 269 |
+
"learned": {
|
| 270 |
+
"validation_fixed_A25": 0.7963477513606417,
|
| 271 |
+
"test_fixed_A25": 0.802590227452106,
|
| 272 |
+
"test_ranked_A25_diagnostic": 0.8037526009901701,
|
| 273 |
+
"selected_validation_actual_calls": 0.9498710971068461,
|
| 274 |
+
"selected_test_actual_calls": 0.9485542082227165
|
| 275 |
+
}
|
| 276 |
+
}
|
| 277 |
+
},
|
| 278 |
+
{
|
| 279 |
+
"key": [
|
| 280 |
+
"construction",
|
| 281 |
+
"yolo26x",
|
| 282 |
+
"MiniCPM"
|
| 283 |
+
],
|
| 284 |
+
"areas": {
|
| 285 |
+
"confidence": {
|
| 286 |
+
"validation_fixed_A25": 0.17451640525159154,
|
| 287 |
+
"test_fixed_A25": 0.17697501725814352,
|
| 288 |
+
"test_ranked_A25_diagnostic": 0.1716572124279524,
|
| 289 |
+
"selected_validation_actual_calls": 0.9386590584878745,
|
| 290 |
+
"selected_test_actual_calls": 0.9254327563249002
|
| 291 |
+
},
|
| 292 |
+
"learned": {
|
| 293 |
+
"validation_fixed_A25": 0.22914536623152232,
|
| 294 |
+
"test_fixed_A25": 0.21733392403576846,
|
| 295 |
+
"test_ranked_A25_diagnostic": 0.21579610594149268,
|
| 296 |
+
"selected_validation_actual_calls": 0.4893009985734665,
|
| 297 |
+
"selected_test_actual_calls": 0.47203728362183756
|
| 298 |
+
}
|
| 299 |
+
}
|
| 300 |
+
},
|
| 301 |
+
{
|
| 302 |
+
"key": [
|
| 303 |
+
"construction",
|
| 304 |
+
"yolo26x",
|
| 305 |
+
"Qwen3.8"
|
| 306 |
+
],
|
| 307 |
+
"areas": {
|
| 308 |
+
"confidence": {
|
| 309 |
+
"validation_fixed_A25": 0.17942553980644182,
|
| 310 |
+
"test_fixed_A25": 0.18055274894913714,
|
| 311 |
+
"test_ranked_A25_diagnostic": 0.1793307821359281,
|
| 312 |
+
"selected_validation_actual_calls": 0.9286733238231099,
|
| 313 |
+
"selected_test_actual_calls": 0.9221038615179761
|
| 314 |
+
},
|
| 315 |
+
"learned": {
|
| 316 |
+
"validation_fixed_A25": 0.23466194499464404,
|
| 317 |
+
"test_fixed_A25": 0.21316340540985348,
|
| 318 |
+
"test_ranked_A25_diagnostic": 0.213003148312119,
|
| 319 |
+
"selected_validation_actual_calls": 0.4992867332382311,
|
| 320 |
+
"selected_test_actual_calls": 0.4876830892143808
|
| 321 |
+
}
|
| 322 |
+
}
|
| 323 |
+
},
|
| 324 |
+
{
|
| 325 |
+
"key": [
|
| 326 |
+
"construction",
|
| 327 |
+
"rtdetr_x_fp32",
|
| 328 |
+
"MiniCPM"
|
| 329 |
+
],
|
| 330 |
+
"areas": {
|
| 331 |
+
"confidence": {
|
| 332 |
+
"validation_fixed_A25": 0.08994758708890332,
|
| 333 |
+
"test_fixed_A25": 0.08325267500178911,
|
| 334 |
+
"test_ranked_A25_diagnostic": 0.08357421552875581,
|
| 335 |
+
"selected_validation_actual_calls": 0.978601997146933,
|
| 336 |
+
"selected_test_actual_calls": 0.9790279627163782
|
| 337 |
+
},
|
| 338 |
+
"learned": {
|
| 339 |
+
"validation_fixed_A25": 0.11165309347543784,
|
| 340 |
+
"test_fixed_A25": 0.09726595269708746,
|
| 341 |
+
"test_ranked_A25_diagnostic": 0.09795552217467457,
|
| 342 |
+
"selected_validation_actual_calls": 0.7788873038516405,
|
| 343 |
+
"selected_test_actual_calls": 0.7882822902796272
|
| 344 |
+
}
|
| 345 |
+
}
|
| 346 |
+
},
|
| 347 |
+
{
|
| 348 |
+
"key": [
|
| 349 |
+
"construction",
|
| 350 |
+
"rtdetr_x_fp32",
|
| 351 |
+
"Qwen3.8"
|
| 352 |
+
],
|
| 353 |
+
"areas": {
|
| 354 |
+
"confidence": {
|
| 355 |
+
"validation_fixed_A25": 0.10053453664183931,
|
| 356 |
+
"test_fixed_A25": 0.08910633276678806,
|
| 357 |
+
"test_ranked_A25_diagnostic": 0.08907503728424503,
|
| 358 |
+
"selected_validation_actual_calls": 0.9885877318116976,
|
| 359 |
+
"selected_test_actual_calls": 0.9860186418109188
|
| 360 |
+
},
|
| 361 |
+
"learned": {
|
| 362 |
+
"validation_fixed_A25": 0.13205087205171664,
|
| 363 |
+
"test_fixed_A25": 0.10226629753438624,
|
| 364 |
+
"test_ranked_A25_diagnostic": 0.10459671104829708,
|
| 365 |
+
"selected_validation_actual_calls": 0.8288159771754636,
|
| 366 |
+
"selected_test_actual_calls": 0.831225033288948
|
| 367 |
+
}
|
| 368 |
+
}
|
| 369 |
+
}
|
| 370 |
+
]
|
paper_results/data/comparisons.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
paper_results/data/earlier_results.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
paper_results/data/joint_MiniCPM_comparison_20260910.json
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"expert": "cub",
|
| 4 |
+
"endpoint": "MiniCPM",
|
| 5 |
+
"single_task_test_fixed25": 0.7119433897134967,
|
| 6 |
+
"multi_task_test_fixed25": 0.7079737659647911,
|
| 7 |
+
"delta_multi_minus_single": -0.003969623748705575
|
| 8 |
+
},
|
| 9 |
+
{
|
| 10 |
+
"expert": "glsim",
|
| 11 |
+
"endpoint": "MiniCPM",
|
| 12 |
+
"single_task_test_fixed25": 0.759233690024163,
|
| 13 |
+
"multi_task_test_fixed25": 0.7745944080082844,
|
| 14 |
+
"delta_multi_minus_single": 0.01536071798412142
|
| 15 |
+
},
|
| 16 |
+
{
|
| 17 |
+
"expert": "nlvr2",
|
| 18 |
+
"endpoint": "MiniCPM",
|
| 19 |
+
"single_task_test_fixed25": 0.8346846523642104,
|
| 20 |
+
"multi_task_test_fixed25": 0.8309535768099304,
|
| 21 |
+
"delta_multi_minus_single": -0.0037310755542799834
|
| 22 |
+
},
|
| 23 |
+
{
|
| 24 |
+
"expert": "vilt",
|
| 25 |
+
"endpoint": "MiniCPM",
|
| 26 |
+
"single_task_test_fixed25": 0.8046925450240368,
|
| 27 |
+
"multi_task_test_fixed25": 0.802611752887996,
|
| 28 |
+
"delta_multi_minus_single": -0.002080792136040732
|
| 29 |
+
},
|
| 30 |
+
{
|
| 31 |
+
"expert": "groundingdino",
|
| 32 |
+
"endpoint": "MiniCPM",
|
| 33 |
+
"single_task_test_fixed25": 0.3598105663159686,
|
| 34 |
+
"multi_task_test_fixed25": 0.37004792558772653,
|
| 35 |
+
"delta_multi_minus_single": 0.010237359271757951
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"expert": "instancevg",
|
| 39 |
+
"endpoint": "MiniCPM",
|
| 40 |
+
"single_task_test_fixed25": 0.6394237586138446,
|
| 41 |
+
"multi_task_test_fixed25": 0.6243654822335025,
|
| 42 |
+
"delta_multi_minus_single": -0.015058276380342073
|
| 43 |
+
},
|
| 44 |
+
{
|
| 45 |
+
"expert": "yolo26x",
|
| 46 |
+
"endpoint": "MiniCPM",
|
| 47 |
+
"single_task_test_fixed25": 0.18032734231886507,
|
| 48 |
+
"multi_task_test_fixed25": 0.1836164473038217,
|
| 49 |
+
"delta_multi_minus_single": 0.003289104984956631
|
| 50 |
+
},
|
| 51 |
+
{
|
| 52 |
+
"expert": "rtdetr_x_fp32",
|
| 53 |
+
"endpoint": "MiniCPM",
|
| 54 |
+
"single_task_test_fixed25": 0.10367040308754236,
|
| 55 |
+
"multi_task_test_fixed25": 0.11260360812088349,
|
| 56 |
+
"delta_multi_minus_single": 0.008933205033341135
|
| 57 |
+
}
|
| 58 |
+
]
|
paper_results/data/joint_Qwen3.8_comparison_20260910.json
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"expert": "cub",
|
| 4 |
+
"endpoint": "Qwen3.8",
|
| 5 |
+
"single_task_test_fixed25": 0.7619951674145667,
|
| 6 |
+
"multi_task_test_fixed25": 0.7782188470831896,
|
| 7 |
+
"delta_multi_minus_single": 0.016223679668622815
|
| 8 |
+
},
|
| 9 |
+
{
|
| 10 |
+
"expert": "glsim",
|
| 11 |
+
"endpoint": "Qwen3.8",
|
| 12 |
+
"single_task_test_fixed25": 0.8301691404901622,
|
| 13 |
+
"multi_task_test_fixed25": 0.8448394891266828,
|
| 14 |
+
"delta_multi_minus_single": 0.014670348636520614
|
| 15 |
+
},
|
| 16 |
+
{
|
| 17 |
+
"expert": "nlvr2",
|
| 18 |
+
"endpoint": "Qwen3.8",
|
| 19 |
+
"single_task_test_fixed25": 0.8372677046710196,
|
| 20 |
+
"multi_task_test_fixed25": 0.8337518834756403,
|
| 21 |
+
"delta_multi_minus_single": -0.0035158211953792984
|
| 22 |
+
},
|
| 23 |
+
{
|
| 24 |
+
"expert": "vilt",
|
| 25 |
+
"endpoint": "Qwen3.8",
|
| 26 |
+
"single_task_test_fixed25": 0.8045490421181029,
|
| 27 |
+
"multi_task_test_fixed25": 0.8067733371600775,
|
| 28 |
+
"delta_multi_minus_single": 0.0022242950419746332
|
| 29 |
+
},
|
| 30 |
+
{
|
| 31 |
+
"expert": "groundingdino",
|
| 32 |
+
"endpoint": "Qwen3.8",
|
| 33 |
+
"single_task_test_fixed25": 0.4223974137197629,
|
| 34 |
+
"multi_task_test_fixed25": 0.45991549215892014,
|
| 35 |
+
"delta_multi_minus_single": 0.03751807843915722
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"expert": "instancevg",
|
| 39 |
+
"endpoint": "Qwen3.8",
|
| 40 |
+
"single_task_test_fixed25": 0.7224853245611548,
|
| 41 |
+
"multi_task_test_fixed25": 0.7053001729858492,
|
| 42 |
+
"delta_multi_minus_single": -0.017185151575305535
|
| 43 |
+
},
|
| 44 |
+
{
|
| 45 |
+
"expert": "yolo26x",
|
| 46 |
+
"endpoint": "Qwen3.8",
|
| 47 |
+
"single_task_test_fixed25": 0.19009376433100783,
|
| 48 |
+
"multi_task_test_fixed25": 0.19603467084956394,
|
| 49 |
+
"delta_multi_minus_single": 0.0059409065185561105
|
| 50 |
+
},
|
| 51 |
+
{
|
| 52 |
+
"expert": "rtdetr_x_fp32",
|
| 53 |
+
"endpoint": "Qwen3.8",
|
| 54 |
+
"single_task_test_fixed25": 0.10483176081963008,
|
| 55 |
+
"multi_task_test_fixed25": 0.12038495915476813,
|
| 56 |
+
"delta_multi_minus_single": 0.015553198335138047
|
| 57 |
+
}
|
| 58 |
+
]
|
paper_results/data/layout_verification.json
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"old_distribution": {
|
| 3 |
+
"width_pt": 523.1995239257812,
|
| 4 |
+
"height_pt": 309.7997741699219,
|
| 5 |
+
"height_at_7_16_in": 4.239618504261674
|
| 6 |
+
},
|
| 7 |
+
"old_budget": {
|
| 8 |
+
"width_pt": 526.7994995117188,
|
| 9 |
+
"height_pt": 237.7997589111328,
|
| 10 |
+
"height_at_7_16_in": 3.2320575007794505
|
| 11 |
+
},
|
| 12 |
+
"combined": {
|
| 13 |
+
"width_pt": 515.52001953125,
|
| 14 |
+
"height_pt": 324.0,
|
| 15 |
+
"height_at_7_16_in": 4.499999829510743
|
| 16 |
+
},
|
| 17 |
+
"reduction_percent": 39.77255134624947,
|
| 18 |
+
"old_pages": 16,
|
| 19 |
+
"new_pages": 14,
|
| 20 |
+
"new_figure_and_section_pages": [
|
| 21 |
+
6,
|
| 22 |
+
7
|
| 23 |
+
],
|
| 24 |
+
"changed_sections": [
|
| 25 |
+
"sections\\appendix.tex",
|
| 26 |
+
"sections\\experiments.tex"
|
| 27 |
+
],
|
| 28 |
+
"figure1_identical": true
|
| 29 |
+
}
|
paper_results/data/main_table_cost_qwen_20260912/build_figures.py
ADDED
|
@@ -0,0 +1,101 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Main-table curves plus fresh measured component costs; no threshold re-selection."""
|
| 2 |
+
from pathlib import Path
|
| 3 |
+
import csv
|
| 4 |
+
import json
|
| 5 |
+
import numpy as np
|
| 6 |
+
import matplotlib
|
| 7 |
+
matplotlib.use('Agg')
|
| 8 |
+
import matplotlib.pyplot as plt
|
| 9 |
+
from matplotlib.ticker import MaxNLocator,LogLocator,NullLocator
|
| 10 |
+
|
| 11 |
+
root=Path(__file__).resolve().parent
|
| 12 |
+
groups=json.loads((root/'main_table_results.json').read_text(encoding='utf-8'))['groups']
|
| 13 |
+
components={}
|
| 14 |
+
curves=[]
|
| 15 |
+
selected=[]
|
| 16 |
+
for g,tag in zip(groups,['qwen_gref','qwen']):
|
| 17 |
+
expert=g['key'][1]
|
| 18 |
+
spec=json.loads((root/f'components/{expert}_specialist.json').read_text())
|
| 19 |
+
router=json.loads((root/f'components/{expert}_router.json').read_text())
|
| 20 |
+
large=json.loads((root/f'components/{tag}_latency.json').read_text())
|
| 21 |
+
flops=json.loads((root/f'components/{tag}_flops.json').read_text())
|
| 22 |
+
assert spec['completed'] and router['status']==large['status']==flops['status']=='complete'
|
| 23 |
+
assert router['run']==g['run'] and router['selected_decisions_match_main']
|
| 24 |
+
assert abs(router['actual_call_fraction']-g['methods']['learned']['test_point']['actual_fraction'])<1e-12
|
| 25 |
+
key='|'.join(g['key'])
|
| 26 |
+
c={'specialist_gflops':spec['mean_counted_gflops_per_sample'],
|
| 27 |
+
'router_gflops':router['mean_gflops_per_input'],
|
| 28 |
+
'large_gflops':flops['mean_prefill_gflops_counted']+large['mean_completion_tokens']*flops['mean_decode_one_token_gflops_counted'],
|
| 29 |
+
'specialist_ms':spec['mean_end_to_end_latency_ms_per_sample'],
|
| 30 |
+
'router_ms':router['mean_latency_ms_per_input'],
|
| 31 |
+
'large_ms':large['mean_batch_wall_ms_per_sample'],
|
| 32 |
+
'large_samples':large['measured_count'], 'flops_probes':flops['sample_count'],
|
| 33 |
+
'specialist_samples':spec['sample_count'],'router_samples':router['sample_count'],'batch_size':large['batch_size']}
|
| 34 |
+
components[key]=c
|
| 35 |
+
for method in ['learned','confidence']:
|
| 36 |
+
for kind,points in [('curve',g['methods'][method]['test_curve']),('selected',[g['methods'][method]['test_point']])]:
|
| 37 |
+
for p in points:
|
| 38 |
+
rate=p['actual_fraction']
|
| 39 |
+
row={'pair':key,'method':method,'nominal_validation_budget':p['budget'],'actual_test_call_fraction':rate,
|
| 40 |
+
'test_metric':p['metric'],'estimated_gflops_per_input':c['specialist_gflops']+(c['router_gflops'] if method=='learned' else 0)+rate*c['large_gflops'],
|
| 41 |
+
'estimated_time_ms_per_input':c['specialist_ms']+(c['router_ms'] if method=='learned' else 0)+rate*c['large_ms']}
|
| 42 |
+
(curves if kind=='curve' else selected).append(row)
|
| 43 |
+
for name,rows in [('full_cost_performance_curves',curves),('selected_points',selected),('fixed_budget_cost_performance',[p for p in curves if p['nominal_validation_budget'] in [0,.05,.1,.25,.5,.75,1]])]:
|
| 44 |
+
with (root/(name+'.csv')).open('w',encoding='utf-8',newline='') as f:
|
| 45 |
+
writer=csv.DictWriter(f,fieldnames=list(rows[0]));writer.writeheader();writer.writerows(rows)
|
| 46 |
+
summary={'components':components,'selected_points':selected,'protocol':
|
| 47 |
+
{'method':'original main-table logit contrast; original checkpoints and validation thresholds; seed 42',
|
| 48 |
+
'large_timing':'64 measured examples; batch size one; same full task prompt; two separate warmup examples',
|
| 49 |
+
'cost':'serial component estimates, not a timed integrated service; includes active distilled encoders and routing head only for learned',
|
| 50 |
+
'flops':'multiply-add=2; supported operators; mean prefill (including visual embedding) plus average generated length times one-token decode probe',
|
| 51 |
+
'scope_limits':'mean call cost extrapolated across policies; no policy-specific input/output-length costs; specialist-feature extraction overhead is not separately measured; confidence threshold overhead not separately timed'}}
|
| 52 |
+
(root/'cost_summary.json').write_text(json.dumps(summary,indent=2),encoding='utf-8')
|
| 53 |
+
|
| 54 |
+
plt.rcParams.update({'font.family':'serif','font.serif':['Times New Roman'],'font.size':7,
|
| 55 |
+
'axes.labelsize':7,'xtick.labelsize':6.5,'ytick.labelsize':6.5,'legend.fontsize':7,
|
| 56 |
+
'pdf.fonttype':42,'ps.fonttype':42,'axes.spines.top':False,'axes.spines.right':False,
|
| 57 |
+
'axes.linewidth':.55,'mathtext.fontset':'stix'})
|
| 58 |
+
styles=[('learned','learned','#176f91','-'),('confidence','conf.','#c78324','--')]
|
| 59 |
+
keys=list(components)
|
| 60 |
+
|
| 61 |
+
def panel(ax,key,field):
|
| 62 |
+
for method,label,color,ls in styles:
|
| 63 |
+
pts=[p for p in curves if p['pair']==key and p['method']==method]
|
| 64 |
+
assert len(pts)==101
|
| 65 |
+
x=np.array([p[field] for p in pts]);y=np.array([p['test_metric']*100 for p in pts])
|
| 66 |
+
assert np.all(np.diff(x)>=-1e-7)
|
| 67 |
+
ax.plot(x,y,color=color,ls=ls,lw=1.05,label=label)
|
| 68 |
+
p=next(p for p in selected if p['pair']==key and p['method']==method)
|
| 69 |
+
ax.scatter(p[field],100*p['test_metric'],color=color,s=13,edgecolor='white',linewidth=.4,zorder=5)
|
| 70 |
+
ax.grid(axis='y',color='#dce1e4',lw=.4)
|
| 71 |
+
ax.tick_params(length=2.3,pad=1.5)
|
| 72 |
+
ax.yaxis.set_major_locator(MaxNLocator(nbins=3))
|
| 73 |
+
if 'gflops' in field:
|
| 74 |
+
ax.set_xscale('log');ax.xaxis.set_major_locator(LogLocator(base=10,numticks=4));ax.xaxis.set_minor_locator(NullLocator())
|
| 75 |
+
ax.set_xlabel('Est. GFLOPs/input',labelpad=1)
|
| 76 |
+
else:
|
| 77 |
+
ax.xaxis.set_major_locator(MaxNLocator(nbins=3));ax.set_xlabel('Est. time (ms/input)',labelpad=1)
|
| 78 |
+
ax.set_ylabel('Acc. (%)' if key.startswith('grefcoco') else 'Macro-F1 (%)',labelpad=1)
|
| 79 |
+
|
| 80 |
+
for field,stem in [('estimated_gflops_per_input','flops_performance'),('estimated_time_ms_per_input','latency_performance')]:
|
| 81 |
+
fig,axes=plt.subplots(1,2,figsize=(3.44,1.50))
|
| 82 |
+
fig.subplots_adjust(left=.115,right=.995,bottom=.31,top=.83,wspace=.43)
|
| 83 |
+
for ax,key,title in zip(axes,keys,['gRefCOCO / Q','Construction / Q']):
|
| 84 |
+
panel(ax,key,field)
|
| 85 |
+
ax.text(.5,1.10,title,transform=ax.transAxes,ha='center',fontsize=7)
|
| 86 |
+
h,l=axes[0].get_legend_handles_labels()
|
| 87 |
+
fig.legend(h,l,loc='lower center',bbox_to_anchor=(.53,-.025),ncol=2,frameon=False,handlelength=2.2,columnspacing=1.8)
|
| 88 |
+
fig.savefig(root/(stem+'.pdf'));fig.savefig(root/(stem+'.png'),dpi=400)
|
| 89 |
+
plt.close(fig)
|
| 90 |
+
|
| 91 |
+
fig,axes=plt.subplots(2,2,figsize=(3.44,2.05))
|
| 92 |
+
fig.subplots_adjust(left=.115,right=.995,bottom=.20,top=.89,wspace=.43,hspace=.64)
|
| 93 |
+
for col,key in enumerate(keys):
|
| 94 |
+
for row,field in enumerate(['estimated_gflops_per_input','estimated_time_ms_per_input']):
|
| 95 |
+
panel(axes[row,col],key,field)
|
| 96 |
+
axes[0,col].text(.5,1.14,'gRefCOCO / Q' if col==0 else 'Construction / Q',transform=axes[0,col].transAxes,ha='center',fontsize=7)
|
| 97 |
+
h,l=axes[0,0].get_legend_handles_labels()
|
| 98 |
+
fig.legend(h,l,loc='lower center',bbox_to_anchor=(.53,-.005),ncol=2,frameon=False,handlelength=2.2,columnspacing=1.8)
|
| 99 |
+
fig.savefig(root/'cost_compact.pdf');fig.savefig(root/'cost_compact.png',dpi=400)
|
| 100 |
+
plt.close(fig)
|
| 101 |
+
print(json.dumps(summary,indent=2))
|
paper_results/data/main_table_cost_qwen_20260912/components/instancevg_router.json
ADDED
|
@@ -0,0 +1,239 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"status": "complete",
|
| 3 |
+
"run": "shared_12_instancevg_Qwen3.8",
|
| 4 |
+
"config": {
|
| 5 |
+
"fraction": 1.0,
|
| 6 |
+
"seed": 42,
|
| 7 |
+
"freeze_encoder": true,
|
| 8 |
+
"fusion": "concat",
|
| 9 |
+
"harm_cost": 1.0,
|
| 10 |
+
"batch_size": 256,
|
| 11 |
+
"weight_decay": 0.01,
|
| 12 |
+
"max_epochs": 50,
|
| 13 |
+
"patience": 8,
|
| 14 |
+
"cpu_threads": 1,
|
| 15 |
+
"student": "8M",
|
| 16 |
+
"representation_dim": 128,
|
| 17 |
+
"lr": 0.003,
|
| 18 |
+
"target": "four_state",
|
| 19 |
+
"strategy": "natural",
|
| 20 |
+
"hidden_percent": 90,
|
| 21 |
+
"expert": "instancevg",
|
| 22 |
+
"task": "grefcoco",
|
| 23 |
+
"endpoint": "Qwen3.8",
|
| 24 |
+
"branches": [
|
| 25 |
+
"image",
|
| 26 |
+
"text",
|
| 27 |
+
"hidden",
|
| 28 |
+
"confidence",
|
| 29 |
+
"output"
|
| 30 |
+
],
|
| 31 |
+
"hidden_layer": "joint_encoder__layer_11",
|
| 32 |
+
"id": "shared_12_instancevg_Qwen3.8",
|
| 33 |
+
"stage": "shared_mandatory_inputs"
|
| 34 |
+
},
|
| 35 |
+
"checkpoint": "/mnt/HithinkOmniSSD/user_workspace/caisihang/research/ICASSP_ModelCollaboration/outputs/router_shared_image_hidden_20260910/runs/shared_12_instancevg_Qwen3.8/best.pt",
|
| 36 |
+
"score": "logit_rescue-logit_harm",
|
| 37 |
+
"test_rows_recomputed": 35263,
|
| 38 |
+
"selected_decisions_match_main": true,
|
| 39 |
+
"max_score_abs_difference": 0.0,
|
| 40 |
+
"actual_call_fraction": 0.5355755324277571,
|
| 41 |
+
"sample_count": 64,
|
| 42 |
+
"sample_ids": [
|
| 43 |
+
"testB-5056-14012",
|
| 44 |
+
"testB-31505-86708",
|
| 45 |
+
"testA-57489-158264",
|
| 46 |
+
"testA-38865-107108",
|
| 47 |
+
"testB-47361-130361",
|
| 48 |
+
"testA-5476-15123",
|
| 49 |
+
"testA-71404-196892",
|
| 50 |
+
"testA-61030-168115",
|
| 51 |
+
"testB-35969-99063",
|
| 52 |
+
"testB-1675-4724",
|
| 53 |
+
"testA-30414-83641",
|
| 54 |
+
"testB-47742-131425",
|
| 55 |
+
"testB-77456-213280",
|
| 56 |
+
"testA-59283-163253",
|
| 57 |
+
"testA-57390-157996",
|
| 58 |
+
"testA-77229-212665",
|
| 59 |
+
"testA-6784-18703",
|
| 60 |
+
"testB-11434-31542",
|
| 61 |
+
"testB-11675-32181",
|
| 62 |
+
"testB-73046-201304",
|
| 63 |
+
"testA-28837-79356",
|
| 64 |
+
"testA-4563-12619",
|
| 65 |
+
"testB-30168-82996",
|
| 66 |
+
"testB-92461-254721",
|
| 67 |
+
"testA-58293-160462",
|
| 68 |
+
"testA-13050-35910",
|
| 69 |
+
"testA-39320-108309",
|
| 70 |
+
"testB-36423-100313",
|
| 71 |
+
"testA-43418-119528",
|
| 72 |
+
"testB-23411-64351",
|
| 73 |
+
"testB-77730-214031",
|
| 74 |
+
"testA-7232-19907",
|
| 75 |
+
"testA-17738-48695",
|
| 76 |
+
"testA-44914-123593",
|
| 77 |
+
"testB-92861-255854",
|
| 78 |
+
"testA-69678-192132",
|
| 79 |
+
"testA-68785-189658",
|
| 80 |
+
"testB-62911-173380",
|
| 81 |
+
"testB-19029-52242",
|
| 82 |
+
"testA-66284-182759",
|
| 83 |
+
"testB-19170-52626",
|
| 84 |
+
"testB-8468-23291",
|
| 85 |
+
"testB-72940-201009",
|
| 86 |
+
"testB-74045-203983",
|
| 87 |
+
"testB-93380-257275",
|
| 88 |
+
"testB-49369-135990",
|
| 89 |
+
"testA-58046-159759",
|
| 90 |
+
"testA-57437-158132",
|
| 91 |
+
"testA-5754-15895",
|
| 92 |
+
"testA-85590-235577",
|
| 93 |
+
"testA-92278-254227",
|
| 94 |
+
"testB-80373-221257",
|
| 95 |
+
"testA-80383-221294",
|
| 96 |
+
"testB-71197-196311",
|
| 97 |
+
"testB-48080-132352",
|
| 98 |
+
"testB-18351-50380",
|
| 99 |
+
"testB-1089-3077",
|
| 100 |
+
"testB-92466-254737",
|
| 101 |
+
"testB-53065-146099",
|
| 102 |
+
"testB-42611-117349",
|
| 103 |
+
"testA-57690-158796",
|
| 104 |
+
"testA-65173-179681",
|
| 105 |
+
"testB-90675-249707",
|
| 106 |
+
"testB-74709-205815"
|
| 107 |
+
],
|
| 108 |
+
"batch_size": 1,
|
| 109 |
+
"latency_ms": [
|
| 110 |
+
10.095497942529619,
|
| 111 |
+
10.442811995744705,
|
| 112 |
+
9.535272954963148,
|
| 113 |
+
8.56779795140028,
|
| 114 |
+
28.526668902486563,
|
| 115 |
+
14.110558084212244,
|
| 116 |
+
19.977235002443194,
|
| 117 |
+
21.489594015292823,
|
| 118 |
+
16.842726967297494,
|
| 119 |
+
12.316308100707829,
|
| 120 |
+
12.318658991716802,
|
| 121 |
+
13.136874069459736,
|
| 122 |
+
16.18148700799793,
|
| 123 |
+
11.749238008633256,
|
| 124 |
+
12.584974057972431,
|
| 125 |
+
12.680649990215898,
|
| 126 |
+
11.369985993951559,
|
| 127 |
+
14.251800021156669,
|
| 128 |
+
14.000480994582176,
|
| 129 |
+
12.42606493178755,
|
| 130 |
+
13.526970986276865,
|
| 131 |
+
11.346371960826218,
|
| 132 |
+
11.965820915065706,
|
| 133 |
+
13.418521964922547,
|
| 134 |
+
13.269996969029307,
|
| 135 |
+
13.868742040358484,
|
| 136 |
+
11.83428696822375,
|
| 137 |
+
13.831221032887697,
|
| 138 |
+
12.576585984788835,
|
| 139 |
+
12.408879934810102,
|
| 140 |
+
12.892312020994723,
|
| 141 |
+
15.224601957015693,
|
| 142 |
+
13.460875954478979,
|
| 143 |
+
11.89423305913806,
|
| 144 |
+
10.926949908025563,
|
| 145 |
+
157.88777999114245,
|
| 146 |
+
16.57543599139899,
|
| 147 |
+
29.21812597196549,
|
| 148 |
+
23.506848025135696,
|
| 149 |
+
13.830928015522659,
|
| 150 |
+
14.57549398764968,
|
| 151 |
+
13.596423901617527,
|
| 152 |
+
16.056897002272308,
|
| 153 |
+
12.610359000973403,
|
| 154 |
+
17.024922999553382,
|
| 155 |
+
12.336026062257588,
|
| 156 |
+
14.77054797578603,
|
| 157 |
+
13.220850960351527,
|
| 158 |
+
19.9308970477432,
|
| 159 |
+
16.4189770584926,
|
| 160 |
+
17.153396038338542,
|
| 161 |
+
31.296705012209713,
|
| 162 |
+
20.386314019560814,
|
| 163 |
+
14.672313001938164,
|
| 164 |
+
15.437994967214763,
|
| 165 |
+
10.980968945659697,
|
| 166 |
+
15.224996954202652,
|
| 167 |
+
12.009209021925926,
|
| 168 |
+
12.397779966704547,
|
| 169 |
+
13.347716070711613,
|
| 170 |
+
15.58609597850591,
|
| 171 |
+
17.80865201726556,
|
| 172 |
+
13.338097953237593,
|
| 173 |
+
13.845606008544564
|
| 174 |
+
],
|
| 175 |
+
"mean_latency_ms_per_input": 17.032772149832454,
|
| 176 |
+
"head_mean_latency_ms": 0.6086819972842932,
|
| 177 |
+
"gflops_per_input": [
|
| 178 |
+
3.642171392,
|
| 179 |
+
3.642171392,
|
| 180 |
+
3.642171392,
|
| 181 |
+
3.642171392,
|
| 182 |
+
3.642171392,
|
| 183 |
+
3.642171392,
|
| 184 |
+
3.642171392,
|
| 185 |
+
3.642171392
|
| 186 |
+
],
|
| 187 |
+
"mean_gflops_per_input": 3.642171392,
|
| 188 |
+
"operators": [
|
| 189 |
+
{
|
| 190 |
+
"aten.convolution": 228802560,
|
| 191 |
+
"aten.bmm": 691709440,
|
| 192 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 193 |
+
"aten.addmm": 2485631488
|
| 194 |
+
},
|
| 195 |
+
{
|
| 196 |
+
"aten.convolution": 228802560,
|
| 197 |
+
"aten.bmm": 691709440,
|
| 198 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 199 |
+
"aten.addmm": 2485631488
|
| 200 |
+
},
|
| 201 |
+
{
|
| 202 |
+
"aten.convolution": 228802560,
|
| 203 |
+
"aten.bmm": 691709440,
|
| 204 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 205 |
+
"aten.addmm": 2485631488
|
| 206 |
+
},
|
| 207 |
+
{
|
| 208 |
+
"aten.convolution": 228802560,
|
| 209 |
+
"aten.bmm": 691709440,
|
| 210 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 211 |
+
"aten.addmm": 2485631488
|
| 212 |
+
},
|
| 213 |
+
{
|
| 214 |
+
"aten.convolution": 228802560,
|
| 215 |
+
"aten.bmm": 691709440,
|
| 216 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 217 |
+
"aten.addmm": 2485631488
|
| 218 |
+
},
|
| 219 |
+
{
|
| 220 |
+
"aten.convolution": 228802560,
|
| 221 |
+
"aten.bmm": 691709440,
|
| 222 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 223 |
+
"aten.addmm": 2485631488
|
| 224 |
+
},
|
| 225 |
+
{
|
| 226 |
+
"aten.convolution": 228802560,
|
| 227 |
+
"aten.bmm": 691709440,
|
| 228 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 229 |
+
"aten.addmm": 2485631488
|
| 230 |
+
},
|
| 231 |
+
{
|
| 232 |
+
"aten.convolution": 228802560,
|
| 233 |
+
"aten.bmm": 691709440,
|
| 234 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 235 |
+
"aten.addmm": 2485631488
|
| 236 |
+
}
|
| 237 |
+
],
|
| 238 |
+
"scope": "image loading, bilinear resize, tokenize when used, two image slots (including gray placeholder), active distilled encoders, feature assembly and original routing head; cached specialist features; supported FLOPs only"
|
| 239 |
+
}
|
paper_results/data/main_table_cost_qwen_20260912/components/instancevg_specialist.json
ADDED
|
@@ -0,0 +1,254 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"component": "specialist",
|
| 3 |
+
"expert": "instancevg",
|
| 4 |
+
"task": "grefcoco",
|
| 5 |
+
"device": "cuda:0",
|
| 6 |
+
"checkpoint": "/mnt/HithinkOmniSSD/user_workspace/caisihang/research/ICASSP_ModelCollaboration/models/grefcoco/instancevg/InstanceVG-grefcoco.pth",
|
| 7 |
+
"sample_count": 64,
|
| 8 |
+
"batch_size": 1,
|
| 9 |
+
"sample_ids": [
|
| 10 |
+
"testA-30844-84861",
|
| 11 |
+
"testB-44353-122077",
|
| 12 |
+
"testB-36889-101643",
|
| 13 |
+
"testA-49055-135087",
|
| 14 |
+
"testA-81245-223633",
|
| 15 |
+
"testB-37669-103833",
|
| 16 |
+
"testA-5038-13943",
|
| 17 |
+
"testB-79646-219274",
|
| 18 |
+
"testA-469-1309",
|
| 19 |
+
"testB-46644-128438",
|
| 20 |
+
"testA-24562-67515",
|
| 21 |
+
"testA-47070-129599",
|
| 22 |
+
"testA-54554-150141",
|
| 23 |
+
"testB-12382-34099",
|
| 24 |
+
"testB-23014-63279",
|
| 25 |
+
"testB-79647-219275",
|
| 26 |
+
"testB-69479-191564",
|
| 27 |
+
"testA-15582-42797",
|
| 28 |
+
"testB-56692-156000",
|
| 29 |
+
"testB-42726-117660",
|
| 30 |
+
"testA-80455-221494",
|
| 31 |
+
"testB-33115-91174",
|
| 32 |
+
"testA-62940-173455",
|
| 33 |
+
"testB-31494-86670",
|
| 34 |
+
"testA-54586-150233",
|
| 35 |
+
"testA-81445-224177",
|
| 36 |
+
"testB-8417-23141",
|
| 37 |
+
"testA-24438-67190",
|
| 38 |
+
"testA-59285-163255",
|
| 39 |
+
"testA-73258-201893",
|
| 40 |
+
"testB-945-2633",
|
| 41 |
+
"testA-54307-149426",
|
| 42 |
+
"testB-60557-166822",
|
| 43 |
+
"testA-81622-224665",
|
| 44 |
+
"testA-69466-191525",
|
| 45 |
+
"testA-60731-167321",
|
| 46 |
+
"testA-7857-21644",
|
| 47 |
+
"testA-44142-121499",
|
| 48 |
+
"testB-66162-182407",
|
| 49 |
+
"testA-31764-87406",
|
| 50 |
+
"testB-38637-106482",
|
| 51 |
+
"testB-52921-145709",
|
| 52 |
+
"testB-65796-181394",
|
| 53 |
+
"testB-2936-8113",
|
| 54 |
+
"testA-79438-218690",
|
| 55 |
+
"testA-57397-158016",
|
| 56 |
+
"testB-23920-65743",
|
| 57 |
+
"testB-15419-42358",
|
| 58 |
+
"testA-10078-27739",
|
| 59 |
+
"testA-43890-120793",
|
| 60 |
+
"testA-55116-151689",
|
| 61 |
+
"testB-27487-75582",
|
| 62 |
+
"testA-26856-73830",
|
| 63 |
+
"testB-50845-140107",
|
| 64 |
+
"testA-29199-80374",
|
| 65 |
+
"testB-65935-181778",
|
| 66 |
+
"testA-17736-48684",
|
| 67 |
+
"testB-69477-191553",
|
| 68 |
+
"testB-22290-61227",
|
| 69 |
+
"testA-58476-160950",
|
| 70 |
+
"testB-36265-99859",
|
| 71 |
+
"testA-5734-15837",
|
| 72 |
+
"testB-67952-187353",
|
| 73 |
+
"testB-47054-129555"
|
| 74 |
+
],
|
| 75 |
+
"model_load_seconds_excluded_from_latency": true,
|
| 76 |
+
"forward_latency_ms_per_sample": [
|
| 77 |
+
41.24195098876953,
|
| 78 |
+
41.74086380004883,
|
| 79 |
+
40.36540985107422,
|
| 80 |
+
44.46102523803711,
|
| 81 |
+
41.2421760559082,
|
| 82 |
+
40.76591873168945,
|
| 83 |
+
41.14704132080078,
|
| 84 |
+
40.88172912597656,
|
| 85 |
+
41.70716857910156,
|
| 86 |
+
41.191871643066406,
|
| 87 |
+
39.91231918334961,
|
| 88 |
+
39.29497528076172,
|
| 89 |
+
40.70723342895508,
|
| 90 |
+
38.920894622802734,
|
| 91 |
+
39.32780838012695,
|
| 92 |
+
38.89958572387695,
|
| 93 |
+
38.59721755981445,
|
| 94 |
+
38.47331237792969,
|
| 95 |
+
39.8744010925293,
|
| 96 |
+
39.87260818481445,
|
| 97 |
+
41.48284912109375,
|
| 98 |
+
40.55779266357422,
|
| 99 |
+
39.05420684814453,
|
| 100 |
+
39.052734375,
|
| 101 |
+
39.143775939941406,
|
| 102 |
+
39.82249450683594,
|
| 103 |
+
39.543006896972656,
|
| 104 |
+
39.368160247802734,
|
| 105 |
+
40.661216735839844,
|
| 106 |
+
39.092193603515625,
|
| 107 |
+
39.85852813720703,
|
| 108 |
+
40.33164978027344,
|
| 109 |
+
38.89593505859375,
|
| 110 |
+
39.29216003417969,
|
| 111 |
+
39.20943832397461,
|
| 112 |
+
40.946590423583984,
|
| 113 |
+
39.44643020629883,
|
| 114 |
+
40.220863342285156,
|
| 115 |
+
39.61030578613281,
|
| 116 |
+
39.639007568359375,
|
| 117 |
+
39.78163146972656,
|
| 118 |
+
39.223201751708984,
|
| 119 |
+
39.00921630859375,
|
| 120 |
+
38.99894332885742,
|
| 121 |
+
41.13721466064453,
|
| 122 |
+
39.94972610473633,
|
| 123 |
+
41.41766357421875,
|
| 124 |
+
39.28073501586914,
|
| 125 |
+
39.721214294433594,
|
| 126 |
+
39.24937438964844,
|
| 127 |
+
39.40460968017578,
|
| 128 |
+
41.30179214477539,
|
| 129 |
+
42.94054412841797,
|
| 130 |
+
41.5063362121582,
|
| 131 |
+
44.03267288208008,
|
| 132 |
+
44.55561447143555,
|
| 133 |
+
44.24995040893555,
|
| 134 |
+
44.87868881225586,
|
| 135 |
+
45.14652633666992,
|
| 136 |
+
44.85055923461914,
|
| 137 |
+
42.39228820800781,
|
| 138 |
+
41.121856689453125,
|
| 139 |
+
45.708351135253906,
|
| 140 |
+
41.8752326965332
|
| 141 |
+
],
|
| 142 |
+
"end_to_end_latency_ms_per_sample": [
|
| 143 |
+
47.29630902875215,
|
| 144 |
+
46.27182998228818,
|
| 145 |
+
47.56885801907629,
|
| 146 |
+
45.48592597711831,
|
| 147 |
+
45.399176073260605,
|
| 148 |
+
46.442538034170866,
|
| 149 |
+
45.95420998521149,
|
| 150 |
+
46.45739798434079,
|
| 151 |
+
44.90085388533771,
|
| 152 |
+
43.80481690168381,
|
| 153 |
+
44.848884106613696,
|
| 154 |
+
43.67590101901442,
|
| 155 |
+
46.40063806436956,
|
| 156 |
+
44.26892101764679,
|
| 157 |
+
44.44277100265026,
|
| 158 |
+
44.71806308720261,
|
| 159 |
+
44.62323593907058,
|
| 160 |
+
44.6264409692958,
|
| 161 |
+
44.96095096692443,
|
| 162 |
+
44.103492982685566,
|
| 163 |
+
45.13338196557015,
|
| 164 |
+
44.1846780013293,
|
| 165 |
+
48.1606520479545,
|
| 166 |
+
46.43543600104749,
|
| 167 |
+
46.19023704435676,
|
| 168 |
+
45.247678994201124,
|
| 169 |
+
44.801001087762415,
|
| 170 |
+
45.467636082321405,
|
| 171 |
+
44.22409099061042,
|
| 172 |
+
44.7448689956218,
|
| 173 |
+
58.0457029864192,
|
| 174 |
+
46.10048606991768,
|
| 175 |
+
44.56539999227971,
|
| 176 |
+
44.47658592835069,
|
| 177 |
+
46.05250502936542,
|
| 178 |
+
44.743441976606846,
|
| 179 |
+
44.77924096863717,
|
| 180 |
+
43.877715012058616,
|
| 181 |
+
44.05005706939846,
|
| 182 |
+
42.87725198082626,
|
| 183 |
+
43.836933095008135,
|
| 184 |
+
45.634721987880766,
|
| 185 |
+
44.748114072717726,
|
| 186 |
+
46.21323198080063,
|
| 187 |
+
52.24333005025983,
|
| 188 |
+
48.90036000870168,
|
| 189 |
+
48.12136595137417,
|
| 190 |
+
47.65231907367706,
|
| 191 |
+
46.71858798246831,
|
| 192 |
+
48.516221111640334,
|
| 193 |
+
43.90157596208155,
|
| 194 |
+
46.83465394191444,
|
| 195 |
+
46.49809293914586,
|
| 196 |
+
42.6300170365721,
|
| 197 |
+
47.39421408157796,
|
| 198 |
+
48.35387098137289,
|
| 199 |
+
48.176085110753775,
|
| 200 |
+
48.29539393540472,
|
| 201 |
+
47.32040502130985,
|
| 202 |
+
46.35261802468449,
|
| 203 |
+
47.482443042099476,
|
| 204 |
+
47.437409055419266,
|
| 205 |
+
47.27676196489483,
|
| 206 |
+
48.198972013778985
|
| 207 |
+
],
|
| 208 |
+
"mean_forward_latency_ms_per_sample": 40.71232491731644,
|
| 209 |
+
"median_forward_latency_ms_per_sample": 40.08529472351074,
|
| 210 |
+
"mean_end_to_end_latency_ms_per_sample": 46.08042127620138,
|
| 211 |
+
"median_end_to_end_latency_ms_per_sample": 45.79446598654613,
|
| 212 |
+
"flop_profiles": [
|
| 213 |
+
{
|
| 214 |
+
"batch_size": 1,
|
| 215 |
+
"flops_counted": 7502680064,
|
| 216 |
+
"gflops_counted_per_sample": 7.502680064,
|
| 217 |
+
"operators": {
|
| 218 |
+
"aten.bmm": 7500632064,
|
| 219 |
+
"aten.baddbmm": 2048000
|
| 220 |
+
}
|
| 221 |
+
},
|
| 222 |
+
{
|
| 223 |
+
"batch_size": 1,
|
| 224 |
+
"flops_counted": 7502680064,
|
| 225 |
+
"gflops_counted_per_sample": 7.502680064,
|
| 226 |
+
"operators": {
|
| 227 |
+
"aten.bmm": 7500632064,
|
| 228 |
+
"aten.baddbmm": 2048000
|
| 229 |
+
}
|
| 230 |
+
},
|
| 231 |
+
{
|
| 232 |
+
"batch_size": 1,
|
| 233 |
+
"flops_counted": 7502680064,
|
| 234 |
+
"gflops_counted_per_sample": 7.502680064,
|
| 235 |
+
"operators": {
|
| 236 |
+
"aten.bmm": 7500632064,
|
| 237 |
+
"aten.baddbmm": 2048000
|
| 238 |
+
}
|
| 239 |
+
},
|
| 240 |
+
{
|
| 241 |
+
"batch_size": 1,
|
| 242 |
+
"flops_counted": 7502680064,
|
| 243 |
+
"gflops_counted_per_sample": 7.502680064,
|
| 244 |
+
"operators": {
|
| 245 |
+
"aten.bmm": 7500632064,
|
| 246 |
+
"aten.baddbmm": 2048000
|
| 247 |
+
}
|
| 248 |
+
}
|
| 249 |
+
],
|
| 250 |
+
"mean_counted_gflops_per_sample": 7.502680064,
|
| 251 |
+
"flops_scope": "GPU FlopCounterMode dense products, convolutions and supported attention; multiply-add counted as 2 FLOPs. Elementwise, normalization, custom ops and postprocessing may be missing.",
|
| 252 |
+
"latency_scope": "A800 wall-clock CUDA forward latency and end-to-end latency including image read/preprocess, excluding model load; batch processing uses the recorded batch size.",
|
| 253 |
+
"completed": true
|
| 254 |
+
}
|
paper_results/data/main_table_cost_qwen_20260912/components/qwen_flops.json
ADDED
|
@@ -0,0 +1,341 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"status": "complete",
|
| 3 |
+
"model_kind": "Qwen3.8",
|
| 4 |
+
"task": "construction",
|
| 5 |
+
"model_path": "/mnt/HithinkOmniSSD/user_workspace/caisihang/checkpoints/Qwen3.8-27B-FP8",
|
| 6 |
+
"sample_count": 8,
|
| 7 |
+
"records": [
|
| 8 |
+
{
|
| 9 |
+
"sample_id": "0000184",
|
| 10 |
+
"prompt_tokens_in_reference_tensor": 1705,
|
| 11 |
+
"input_shapes": {
|
| 12 |
+
"input_ids": [
|
| 13 |
+
1,
|
| 14 |
+
1705
|
| 15 |
+
],
|
| 16 |
+
"attention_mask": [
|
| 17 |
+
1,
|
| 18 |
+
1705
|
| 19 |
+
],
|
| 20 |
+
"mm_token_type_ids": [
|
| 21 |
+
1,
|
| 22 |
+
1705
|
| 23 |
+
],
|
| 24 |
+
"pixel_values": [
|
| 25 |
+
3888,
|
| 26 |
+
1536
|
| 27 |
+
],
|
| 28 |
+
"image_grid_thw": [
|
| 29 |
+
1,
|
| 30 |
+
3
|
| 31 |
+
]
|
| 32 |
+
},
|
| 33 |
+
"prefill_flops_counted": 90083398713024,
|
| 34 |
+
"decode_one_token_flops_counted": 51922600128,
|
| 35 |
+
"prefill_gflops_counted": 90083.398713024,
|
| 36 |
+
"decode_one_token_gflops_counted": 51.922600128,
|
| 37 |
+
"prefill_operators": {
|
| 38 |
+
"aten.convolution": 20475543552,
|
| 39 |
+
"aten.addmm": 3283626885120,
|
| 40 |
+
"aten.bmm": 3741356981952,
|
| 41 |
+
"aten.mm": 83037939302400
|
| 42 |
+
},
|
| 43 |
+
"decode_operators": {
|
| 44 |
+
"aten.bmm": 670826688,
|
| 45 |
+
"aten.mm": 51243909120,
|
| 46 |
+
"aten.convolution": 7864320
|
| 47 |
+
}
|
| 48 |
+
},
|
| 49 |
+
{
|
| 50 |
+
"sample_id": "0001955",
|
| 51 |
+
"prompt_tokens_in_reference_tensor": 1705,
|
| 52 |
+
"input_shapes": {
|
| 53 |
+
"input_ids": [
|
| 54 |
+
1,
|
| 55 |
+
1705
|
| 56 |
+
],
|
| 57 |
+
"attention_mask": [
|
| 58 |
+
1,
|
| 59 |
+
1705
|
| 60 |
+
],
|
| 61 |
+
"mm_token_type_ids": [
|
| 62 |
+
1,
|
| 63 |
+
1705
|
| 64 |
+
],
|
| 65 |
+
"pixel_values": [
|
| 66 |
+
3888,
|
| 67 |
+
1536
|
| 68 |
+
],
|
| 69 |
+
"image_grid_thw": [
|
| 70 |
+
1,
|
| 71 |
+
3
|
| 72 |
+
]
|
| 73 |
+
},
|
| 74 |
+
"prefill_flops_counted": 90083398713024,
|
| 75 |
+
"decode_one_token_flops_counted": 51922600128,
|
| 76 |
+
"prefill_gflops_counted": 90083.398713024,
|
| 77 |
+
"decode_one_token_gflops_counted": 51.922600128,
|
| 78 |
+
"prefill_operators": {
|
| 79 |
+
"aten.convolution": 20475543552,
|
| 80 |
+
"aten.addmm": 3283626885120,
|
| 81 |
+
"aten.bmm": 3741356981952,
|
| 82 |
+
"aten.mm": 83037939302400
|
| 83 |
+
},
|
| 84 |
+
"decode_operators": {
|
| 85 |
+
"aten.bmm": 670826688,
|
| 86 |
+
"aten.mm": 51243909120,
|
| 87 |
+
"aten.convolution": 7864320
|
| 88 |
+
}
|
| 89 |
+
},
|
| 90 |
+
{
|
| 91 |
+
"sample_id": "0000840",
|
| 92 |
+
"prompt_tokens_in_reference_tensor": 1705,
|
| 93 |
+
"input_shapes": {
|
| 94 |
+
"input_ids": [
|
| 95 |
+
1,
|
| 96 |
+
1705
|
| 97 |
+
],
|
| 98 |
+
"attention_mask": [
|
| 99 |
+
1,
|
| 100 |
+
1705
|
| 101 |
+
],
|
| 102 |
+
"mm_token_type_ids": [
|
| 103 |
+
1,
|
| 104 |
+
1705
|
| 105 |
+
],
|
| 106 |
+
"pixel_values": [
|
| 107 |
+
3888,
|
| 108 |
+
1536
|
| 109 |
+
],
|
| 110 |
+
"image_grid_thw": [
|
| 111 |
+
1,
|
| 112 |
+
3
|
| 113 |
+
]
|
| 114 |
+
},
|
| 115 |
+
"prefill_flops_counted": 90083398713024,
|
| 116 |
+
"decode_one_token_flops_counted": 51922600128,
|
| 117 |
+
"prefill_gflops_counted": 90083.398713024,
|
| 118 |
+
"decode_one_token_gflops_counted": 51.922600128,
|
| 119 |
+
"prefill_operators": {
|
| 120 |
+
"aten.convolution": 20475543552,
|
| 121 |
+
"aten.addmm": 3283626885120,
|
| 122 |
+
"aten.bmm": 3741356981952,
|
| 123 |
+
"aten.mm": 83037939302400
|
| 124 |
+
},
|
| 125 |
+
"decode_operators": {
|
| 126 |
+
"aten.bmm": 670826688,
|
| 127 |
+
"aten.mm": 51243909120,
|
| 128 |
+
"aten.convolution": 7864320
|
| 129 |
+
}
|
| 130 |
+
},
|
| 131 |
+
{
|
| 132 |
+
"sample_id": "0003887",
|
| 133 |
+
"prompt_tokens_in_reference_tensor": 1705,
|
| 134 |
+
"input_shapes": {
|
| 135 |
+
"input_ids": [
|
| 136 |
+
1,
|
| 137 |
+
1705
|
| 138 |
+
],
|
| 139 |
+
"attention_mask": [
|
| 140 |
+
1,
|
| 141 |
+
1705
|
| 142 |
+
],
|
| 143 |
+
"mm_token_type_ids": [
|
| 144 |
+
1,
|
| 145 |
+
1705
|
| 146 |
+
],
|
| 147 |
+
"pixel_values": [
|
| 148 |
+
3888,
|
| 149 |
+
1536
|
| 150 |
+
],
|
| 151 |
+
"image_grid_thw": [
|
| 152 |
+
1,
|
| 153 |
+
3
|
| 154 |
+
]
|
| 155 |
+
},
|
| 156 |
+
"prefill_flops_counted": 90083398713024,
|
| 157 |
+
"decode_one_token_flops_counted": 51922600128,
|
| 158 |
+
"prefill_gflops_counted": 90083.398713024,
|
| 159 |
+
"decode_one_token_gflops_counted": 51.922600128,
|
| 160 |
+
"prefill_operators": {
|
| 161 |
+
"aten.convolution": 20475543552,
|
| 162 |
+
"aten.addmm": 3283626885120,
|
| 163 |
+
"aten.bmm": 3741356981952,
|
| 164 |
+
"aten.mm": 83037939302400
|
| 165 |
+
},
|
| 166 |
+
"decode_operators": {
|
| 167 |
+
"aten.bmm": 670826688,
|
| 168 |
+
"aten.mm": 51243909120,
|
| 169 |
+
"aten.convolution": 7864320
|
| 170 |
+
}
|
| 171 |
+
},
|
| 172 |
+
{
|
| 173 |
+
"sample_id": "0003433",
|
| 174 |
+
"prompt_tokens_in_reference_tensor": 1705,
|
| 175 |
+
"input_shapes": {
|
| 176 |
+
"input_ids": [
|
| 177 |
+
1,
|
| 178 |
+
1705
|
| 179 |
+
],
|
| 180 |
+
"attention_mask": [
|
| 181 |
+
1,
|
| 182 |
+
1705
|
| 183 |
+
],
|
| 184 |
+
"mm_token_type_ids": [
|
| 185 |
+
1,
|
| 186 |
+
1705
|
| 187 |
+
],
|
| 188 |
+
"pixel_values": [
|
| 189 |
+
3888,
|
| 190 |
+
1536
|
| 191 |
+
],
|
| 192 |
+
"image_grid_thw": [
|
| 193 |
+
1,
|
| 194 |
+
3
|
| 195 |
+
]
|
| 196 |
+
},
|
| 197 |
+
"prefill_flops_counted": 90083398713024,
|
| 198 |
+
"decode_one_token_flops_counted": 51922600128,
|
| 199 |
+
"prefill_gflops_counted": 90083.398713024,
|
| 200 |
+
"decode_one_token_gflops_counted": 51.922600128,
|
| 201 |
+
"prefill_operators": {
|
| 202 |
+
"aten.convolution": 20475543552,
|
| 203 |
+
"aten.addmm": 3283626885120,
|
| 204 |
+
"aten.bmm": 3741356981952,
|
| 205 |
+
"aten.mm": 83037939302400
|
| 206 |
+
},
|
| 207 |
+
"decode_operators": {
|
| 208 |
+
"aten.bmm": 670826688,
|
| 209 |
+
"aten.mm": 51243909120,
|
| 210 |
+
"aten.convolution": 7864320
|
| 211 |
+
}
|
| 212 |
+
},
|
| 213 |
+
{
|
| 214 |
+
"sample_id": "0001613",
|
| 215 |
+
"prompt_tokens_in_reference_tensor": 1705,
|
| 216 |
+
"input_shapes": {
|
| 217 |
+
"input_ids": [
|
| 218 |
+
1,
|
| 219 |
+
1705
|
| 220 |
+
],
|
| 221 |
+
"attention_mask": [
|
| 222 |
+
1,
|
| 223 |
+
1705
|
| 224 |
+
],
|
| 225 |
+
"mm_token_type_ids": [
|
| 226 |
+
1,
|
| 227 |
+
1705
|
| 228 |
+
],
|
| 229 |
+
"pixel_values": [
|
| 230 |
+
3888,
|
| 231 |
+
1536
|
| 232 |
+
],
|
| 233 |
+
"image_grid_thw": [
|
| 234 |
+
1,
|
| 235 |
+
3
|
| 236 |
+
]
|
| 237 |
+
},
|
| 238 |
+
"prefill_flops_counted": 90083398713024,
|
| 239 |
+
"decode_one_token_flops_counted": 51922600128,
|
| 240 |
+
"prefill_gflops_counted": 90083.398713024,
|
| 241 |
+
"decode_one_token_gflops_counted": 51.922600128,
|
| 242 |
+
"prefill_operators": {
|
| 243 |
+
"aten.convolution": 20475543552,
|
| 244 |
+
"aten.addmm": 3283626885120,
|
| 245 |
+
"aten.bmm": 3741356981952,
|
| 246 |
+
"aten.mm": 83037939302400
|
| 247 |
+
},
|
| 248 |
+
"decode_operators": {
|
| 249 |
+
"aten.bmm": 670826688,
|
| 250 |
+
"aten.mm": 51243909120,
|
| 251 |
+
"aten.convolution": 7864320
|
| 252 |
+
}
|
| 253 |
+
},
|
| 254 |
+
{
|
| 255 |
+
"sample_id": "0002591",
|
| 256 |
+
"prompt_tokens_in_reference_tensor": 1705,
|
| 257 |
+
"input_shapes": {
|
| 258 |
+
"input_ids": [
|
| 259 |
+
1,
|
| 260 |
+
1705
|
| 261 |
+
],
|
| 262 |
+
"attention_mask": [
|
| 263 |
+
1,
|
| 264 |
+
1705
|
| 265 |
+
],
|
| 266 |
+
"mm_token_type_ids": [
|
| 267 |
+
1,
|
| 268 |
+
1705
|
| 269 |
+
],
|
| 270 |
+
"pixel_values": [
|
| 271 |
+
3888,
|
| 272 |
+
1536
|
| 273 |
+
],
|
| 274 |
+
"image_grid_thw": [
|
| 275 |
+
1,
|
| 276 |
+
3
|
| 277 |
+
]
|
| 278 |
+
},
|
| 279 |
+
"prefill_flops_counted": 90083398713024,
|
| 280 |
+
"decode_one_token_flops_counted": 51922600128,
|
| 281 |
+
"prefill_gflops_counted": 90083.398713024,
|
| 282 |
+
"decode_one_token_gflops_counted": 51.922600128,
|
| 283 |
+
"prefill_operators": {
|
| 284 |
+
"aten.convolution": 20475543552,
|
| 285 |
+
"aten.addmm": 3283626885120,
|
| 286 |
+
"aten.bmm": 3741356981952,
|
| 287 |
+
"aten.mm": 83037939302400
|
| 288 |
+
},
|
| 289 |
+
"decode_operators": {
|
| 290 |
+
"aten.bmm": 670826688,
|
| 291 |
+
"aten.mm": 51243909120,
|
| 292 |
+
"aten.convolution": 7864320
|
| 293 |
+
}
|
| 294 |
+
},
|
| 295 |
+
{
|
| 296 |
+
"sample_id": "0003105",
|
| 297 |
+
"prompt_tokens_in_reference_tensor": 1705,
|
| 298 |
+
"input_shapes": {
|
| 299 |
+
"input_ids": [
|
| 300 |
+
1,
|
| 301 |
+
1705
|
| 302 |
+
],
|
| 303 |
+
"attention_mask": [
|
| 304 |
+
1,
|
| 305 |
+
1705
|
| 306 |
+
],
|
| 307 |
+
"mm_token_type_ids": [
|
| 308 |
+
1,
|
| 309 |
+
1705
|
| 310 |
+
],
|
| 311 |
+
"pixel_values": [
|
| 312 |
+
3888,
|
| 313 |
+
1536
|
| 314 |
+
],
|
| 315 |
+
"image_grid_thw": [
|
| 316 |
+
1,
|
| 317 |
+
3
|
| 318 |
+
]
|
| 319 |
+
},
|
| 320 |
+
"prefill_flops_counted": 90083398713024,
|
| 321 |
+
"decode_one_token_flops_counted": 51922600128,
|
| 322 |
+
"prefill_gflops_counted": 90083.398713024,
|
| 323 |
+
"decode_one_token_gflops_counted": 51.922600128,
|
| 324 |
+
"prefill_operators": {
|
| 325 |
+
"aten.convolution": 20475543552,
|
| 326 |
+
"aten.addmm": 3283626885120,
|
| 327 |
+
"aten.bmm": 3741356981952,
|
| 328 |
+
"aten.mm": 83037939302400
|
| 329 |
+
},
|
| 330 |
+
"decode_operators": {
|
| 331 |
+
"aten.bmm": 670826688,
|
| 332 |
+
"aten.mm": 51243909120,
|
| 333 |
+
"aten.convolution": 7864320
|
| 334 |
+
}
|
| 335 |
+
}
|
| 336 |
+
],
|
| 337 |
+
"mean_prefill_gflops_counted": 90083.398713024,
|
| 338 |
+
"mean_decode_one_token_gflops_counted": 51.922600128,
|
| 339 |
+
"scope": "HF reference execution on the A800 with real task inputs; dense products/convolutions/supported attention counted with multiply-add=2 FLOPs. vLLM custom kernels, elementwise/normalization/nonlinearities and other unsupported operations are not complete.",
|
| 340 |
+
"answer_generation": "not performed; one known-token decode probe only"
|
| 341 |
+
}
|
paper_results/data/main_table_cost_qwen_20260912/components/qwen_gref_flops.json
ADDED
|
@@ -0,0 +1,341 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"status": "complete",
|
| 3 |
+
"model_kind": "Qwen3.8",
|
| 4 |
+
"task": "grefcoco",
|
| 5 |
+
"model_path": "/mnt/HithinkOmniSSD/user_workspace/caisihang/checkpoints/Qwen3.8-27B-FP8",
|
| 6 |
+
"sample_count": 8,
|
| 7 |
+
"records": [
|
| 8 |
+
{
|
| 9 |
+
"sample_id": "testA-5038-13945",
|
| 10 |
+
"prompt_tokens_in_reference_tensor": 608,
|
| 11 |
+
"input_shapes": {
|
| 12 |
+
"input_ids": [
|
| 13 |
+
1,
|
| 14 |
+
608
|
| 15 |
+
],
|
| 16 |
+
"attention_mask": [
|
| 17 |
+
1,
|
| 18 |
+
608
|
| 19 |
+
],
|
| 20 |
+
"mm_token_type_ids": [
|
| 21 |
+
1,
|
| 22 |
+
608
|
| 23 |
+
],
|
| 24 |
+
"pixel_values": [
|
| 25 |
+
1200,
|
| 26 |
+
1536
|
| 27 |
+
],
|
| 28 |
+
"image_grid_thw": [
|
| 29 |
+
1,
|
| 30 |
+
3
|
| 31 |
+
]
|
| 32 |
+
},
|
| 33 |
+
"prefill_flops_counted": 31223201515520,
|
| 34 |
+
"decode_one_token_flops_counted": 51491242176,
|
| 35 |
+
"prefill_gflops_counted": 31223.20151552,
|
| 36 |
+
"decode_one_token_gflops_counted": 51.491242176,
|
| 37 |
+
"prefill_operators": {
|
| 38 |
+
"aten.convolution": 6649282560,
|
| 39 |
+
"aten.addmm": 1013465088000,
|
| 40 |
+
"aten.bmm": 590268057600,
|
| 41 |
+
"aten.mm": 29612819087360
|
| 42 |
+
},
|
| 43 |
+
"decode_operators": {
|
| 44 |
+
"aten.bmm": 239468736,
|
| 45 |
+
"aten.mm": 51243909120,
|
| 46 |
+
"aten.convolution": 7864320
|
| 47 |
+
}
|
| 48 |
+
},
|
| 49 |
+
{
|
| 50 |
+
"sample_id": "testA-73305-202014",
|
| 51 |
+
"prompt_tokens_in_reference_tensor": 608,
|
| 52 |
+
"input_shapes": {
|
| 53 |
+
"input_ids": [
|
| 54 |
+
1,
|
| 55 |
+
608
|
| 56 |
+
],
|
| 57 |
+
"attention_mask": [
|
| 58 |
+
1,
|
| 59 |
+
608
|
| 60 |
+
],
|
| 61 |
+
"mm_token_type_ids": [
|
| 62 |
+
1,
|
| 63 |
+
608
|
| 64 |
+
],
|
| 65 |
+
"pixel_values": [
|
| 66 |
+
1200,
|
| 67 |
+
1536
|
| 68 |
+
],
|
| 69 |
+
"image_grid_thw": [
|
| 70 |
+
1,
|
| 71 |
+
3
|
| 72 |
+
]
|
| 73 |
+
},
|
| 74 |
+
"prefill_flops_counted": 31223201515520,
|
| 75 |
+
"decode_one_token_flops_counted": 51491242176,
|
| 76 |
+
"prefill_gflops_counted": 31223.20151552,
|
| 77 |
+
"decode_one_token_gflops_counted": 51.491242176,
|
| 78 |
+
"prefill_operators": {
|
| 79 |
+
"aten.convolution": 6649282560,
|
| 80 |
+
"aten.addmm": 1013465088000,
|
| 81 |
+
"aten.bmm": 590268057600,
|
| 82 |
+
"aten.mm": 29612819087360
|
| 83 |
+
},
|
| 84 |
+
"decode_operators": {
|
| 85 |
+
"aten.bmm": 239468736,
|
| 86 |
+
"aten.mm": 51243909120,
|
| 87 |
+
"aten.convolution": 7864320
|
| 88 |
+
}
|
| 89 |
+
},
|
| 90 |
+
{
|
| 91 |
+
"sample_id": "testA-26952-74099",
|
| 92 |
+
"prompt_tokens_in_reference_tensor": 562,
|
| 93 |
+
"input_shapes": {
|
| 94 |
+
"input_ids": [
|
| 95 |
+
1,
|
| 96 |
+
562
|
| 97 |
+
],
|
| 98 |
+
"attention_mask": [
|
| 99 |
+
1,
|
| 100 |
+
562
|
| 101 |
+
],
|
| 102 |
+
"mm_token_type_ids": [
|
| 103 |
+
1,
|
| 104 |
+
562
|
| 105 |
+
],
|
| 106 |
+
"pixel_values": [
|
| 107 |
+
1040,
|
| 108 |
+
1536
|
| 109 |
+
],
|
| 110 |
+
"image_grid_thw": [
|
| 111 |
+
1,
|
| 112 |
+
3
|
| 113 |
+
]
|
| 114 |
+
},
|
| 115 |
+
"prefill_flops_counted": 28754745861504,
|
| 116 |
+
"decode_one_token_flops_counted": 51473154240,
|
| 117 |
+
"prefill_gflops_counted": 28754.745861504,
|
| 118 |
+
"decode_one_token_gflops_counted": 51.47315424,
|
| 119 |
+
"prefill_operators": {
|
| 120 |
+
"aten.convolution": 5902172160,
|
| 121 |
+
"aten.addmm": 878336409600,
|
| 122 |
+
"aten.bmm": 497939359104,
|
| 123 |
+
"aten.mm": 27372567920640
|
| 124 |
+
},
|
| 125 |
+
"decode_operators": {
|
| 126 |
+
"aten.bmm": 221380800,
|
| 127 |
+
"aten.mm": 51243909120,
|
| 128 |
+
"aten.convolution": 7864320
|
| 129 |
+
}
|
| 130 |
+
},
|
| 131 |
+
{
|
| 132 |
+
"sample_id": "testB-53254-146620",
|
| 133 |
+
"prompt_tokens_in_reference_tensor": 560,
|
| 134 |
+
"input_shapes": {
|
| 135 |
+
"input_ids": [
|
| 136 |
+
1,
|
| 137 |
+
560
|
| 138 |
+
],
|
| 139 |
+
"attention_mask": [
|
| 140 |
+
1,
|
| 141 |
+
560
|
| 142 |
+
],
|
| 143 |
+
"mm_token_type_ids": [
|
| 144 |
+
1,
|
| 145 |
+
560
|
| 146 |
+
],
|
| 147 |
+
"pixel_values": [
|
| 148 |
+
1040,
|
| 149 |
+
1536
|
| 150 |
+
],
|
| 151 |
+
"image_grid_thw": [
|
| 152 |
+
1,
|
| 153 |
+
3
|
| 154 |
+
]
|
| 155 |
+
},
|
| 156 |
+
"prefill_flops_counted": 28656453395456,
|
| 157 |
+
"decode_one_token_flops_counted": 51472367808,
|
| 158 |
+
"prefill_gflops_counted": 28656.453395456,
|
| 159 |
+
"decode_one_token_gflops_counted": 51.472367808,
|
| 160 |
+
"prefill_operators": {
|
| 161 |
+
"aten.convolution": 5894307840,
|
| 162 |
+
"aten.addmm": 878336409600,
|
| 163 |
+
"aten.bmm": 497056982016,
|
| 164 |
+
"aten.mm": 27275165696000
|
| 165 |
+
},
|
| 166 |
+
"decode_operators": {
|
| 167 |
+
"aten.bmm": 220594368,
|
| 168 |
+
"aten.mm": 51243909120,
|
| 169 |
+
"aten.convolution": 7864320
|
| 170 |
+
}
|
| 171 |
+
},
|
| 172 |
+
{
|
| 173 |
+
"sample_id": "testB-37889-104439",
|
| 174 |
+
"prompt_tokens_in_reference_tensor": 687,
|
| 175 |
+
"input_shapes": {
|
| 176 |
+
"input_ids": [
|
| 177 |
+
1,
|
| 178 |
+
687
|
| 179 |
+
],
|
| 180 |
+
"attention_mask": [
|
| 181 |
+
1,
|
| 182 |
+
687
|
| 183 |
+
],
|
| 184 |
+
"mm_token_type_ids": [
|
| 185 |
+
1,
|
| 186 |
+
687
|
| 187 |
+
],
|
| 188 |
+
"pixel_values": [
|
| 189 |
+
1520,
|
| 190 |
+
1536
|
| 191 |
+
],
|
| 192 |
+
"image_grid_thw": [
|
| 193 |
+
1,
|
| 194 |
+
3
|
| 195 |
+
]
|
| 196 |
+
},
|
| 197 |
+
"prefill_flops_counted": 35517384622912,
|
| 198 |
+
"decode_one_token_flops_counted": 51522306240,
|
| 199 |
+
"prefill_gflops_counted": 35517.384622912,
|
| 200 |
+
"decode_one_token_gflops_counted": 51.52230624,
|
| 201 |
+
"prefill_operators": {
|
| 202 |
+
"aten.convolution": 8092385280,
|
| 203 |
+
"aten.addmm": 1283722444800,
|
| 204 |
+
"aten.bmm": 765362832192,
|
| 205 |
+
"aten.mm": 33460206960640
|
| 206 |
+
},
|
| 207 |
+
"decode_operators": {
|
| 208 |
+
"aten.bmm": 270532800,
|
| 209 |
+
"aten.mm": 51243909120,
|
| 210 |
+
"aten.convolution": 7864320
|
| 211 |
+
}
|
| 212 |
+
},
|
| 213 |
+
{
|
| 214 |
+
"sample_id": "testA-58523-161084",
|
| 215 |
+
"prompt_tokens_in_reference_tensor": 500,
|
| 216 |
+
"input_shapes": {
|
| 217 |
+
"input_ids": [
|
| 218 |
+
1,
|
| 219 |
+
500
|
| 220 |
+
],
|
| 221 |
+
"attention_mask": [
|
| 222 |
+
1,
|
| 223 |
+
500
|
| 224 |
+
],
|
| 225 |
+
"mm_token_type_ids": [
|
| 226 |
+
1,
|
| 227 |
+
500
|
| 228 |
+
],
|
| 229 |
+
"pixel_values": [
|
| 230 |
+
768,
|
| 231 |
+
1536
|
| 232 |
+
],
|
| 233 |
+
"image_grid_thw": [
|
| 234 |
+
1,
|
| 235 |
+
3
|
| 236 |
+
]
|
| 237 |
+
},
|
| 238 |
+
"prefill_flops_counted": 25390700918528,
|
| 239 |
+
"decode_one_token_flops_counted": 51448774848,
|
| 240 |
+
"prefill_gflops_counted": 25390.700918528,
|
| 241 |
+
"decode_one_token_gflops_counted": 51.448774848,
|
| 242 |
+
"prefill_operators": {
|
| 243 |
+
"aten.convolution": 4695785472,
|
| 244 |
+
"aten.addmm": 648617656320,
|
| 245 |
+
"aten.bmm": 384288519936,
|
| 246 |
+
"aten.mm": 24353098956800
|
| 247 |
+
},
|
| 248 |
+
"decode_operators": {
|
| 249 |
+
"aten.bmm": 197001408,
|
| 250 |
+
"aten.mm": 51243909120,
|
| 251 |
+
"aten.convolution": 7864320
|
| 252 |
+
}
|
| 253 |
+
},
|
| 254 |
+
{
|
| 255 |
+
"sample_id": "testB-3419-9449",
|
| 256 |
+
"prompt_tokens_in_reference_tensor": 568,
|
| 257 |
+
"input_shapes": {
|
| 258 |
+
"input_ids": [
|
| 259 |
+
1,
|
| 260 |
+
568
|
| 261 |
+
],
|
| 262 |
+
"attention_mask": [
|
| 263 |
+
1,
|
| 264 |
+
568
|
| 265 |
+
],
|
| 266 |
+
"mm_token_type_ids": [
|
| 267 |
+
1,
|
| 268 |
+
568
|
| 269 |
+
],
|
| 270 |
+
"pixel_values": [
|
| 271 |
+
1040,
|
| 272 |
+
1536
|
| 273 |
+
],
|
| 274 |
+
"image_grid_thw": [
|
| 275 |
+
1,
|
| 276 |
+
3
|
| 277 |
+
]
|
| 278 |
+
},
|
| 279 |
+
"prefill_flops_counted": 29049642134016,
|
| 280 |
+
"decode_one_token_flops_counted": 51475513536,
|
| 281 |
+
"prefill_gflops_counted": 29049.642134016,
|
| 282 |
+
"decode_one_token_gflops_counted": 51.475513536,
|
| 283 |
+
"prefill_operators": {
|
| 284 |
+
"aten.convolution": 5925765120,
|
| 285 |
+
"aten.addmm": 878336409600,
|
| 286 |
+
"aten.bmm": 500605364736,
|
| 287 |
+
"aten.mm": 27664774594560
|
| 288 |
+
},
|
| 289 |
+
"decode_operators": {
|
| 290 |
+
"aten.bmm": 223740096,
|
| 291 |
+
"aten.mm": 51243909120,
|
| 292 |
+
"aten.convolution": 7864320
|
| 293 |
+
}
|
| 294 |
+
},
|
| 295 |
+
{
|
| 296 |
+
"sample_id": "testB-24035-66084",
|
| 297 |
+
"prompt_tokens_in_reference_tensor": 561,
|
| 298 |
+
"input_shapes": {
|
| 299 |
+
"input_ids": [
|
| 300 |
+
1,
|
| 301 |
+
561
|
| 302 |
+
],
|
| 303 |
+
"attention_mask": [
|
| 304 |
+
1,
|
| 305 |
+
561
|
| 306 |
+
],
|
| 307 |
+
"mm_token_type_ids": [
|
| 308 |
+
1,
|
| 309 |
+
561
|
| 310 |
+
],
|
| 311 |
+
"pixel_values": [
|
| 312 |
+
1040,
|
| 313 |
+
1536
|
| 314 |
+
],
|
| 315 |
+
"image_grid_thw": [
|
| 316 |
+
1,
|
| 317 |
+
3
|
| 318 |
+
]
|
| 319 |
+
},
|
| 320 |
+
"prefill_flops_counted": 28705599235264,
|
| 321 |
+
"decode_one_token_flops_counted": 51472761024,
|
| 322 |
+
"prefill_gflops_counted": 28705.599235264,
|
| 323 |
+
"decode_one_token_gflops_counted": 51.472761024,
|
| 324 |
+
"prefill_operators": {
|
| 325 |
+
"aten.convolution": 5898240000,
|
| 326 |
+
"aten.addmm": 878336409600,
|
| 327 |
+
"aten.bmm": 497497777344,
|
| 328 |
+
"aten.mm": 27323866808320
|
| 329 |
+
},
|
| 330 |
+
"decode_operators": {
|
| 331 |
+
"aten.bmm": 220987584,
|
| 332 |
+
"aten.mm": 51243909120,
|
| 333 |
+
"aten.convolution": 7864320
|
| 334 |
+
}
|
| 335 |
+
}
|
| 336 |
+
],
|
| 337 |
+
"mean_prefill_gflops_counted": 29815.11614984,
|
| 338 |
+
"mean_decode_one_token_gflops_counted": 51.480920256000005,
|
| 339 |
+
"scope": "HF reference execution on the A800 with real task inputs; dense products/convolutions/supported attention counted with multiply-add=2 FLOPs. vLLM custom kernels, elementwise/normalization/nonlinearities and other unsupported operations are not complete.",
|
| 340 |
+
"answer_generation": "not performed; one known-token decode probe only"
|
| 341 |
+
}
|
paper_results/data/main_table_cost_qwen_20260912/components/qwen_gref_latency.json
ADDED
|
@@ -0,0 +1,2446 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"status": "complete",
|
| 3 |
+
"model_kind": "Qwen3.8",
|
| 4 |
+
"task": "grefcoco",
|
| 5 |
+
"expert": "instancevg",
|
| 6 |
+
"model_path": "/mnt/HithinkOmniSSD/user_workspace/caisihang/checkpoints/Qwen3.8-27B-FP8",
|
| 7 |
+
"device": "cuda:0",
|
| 8 |
+
"gpu_memory_utilization": 0.85,
|
| 9 |
+
"sample_count_requested": 66,
|
| 10 |
+
"warmup_count": 2,
|
| 11 |
+
"measured_count": 64,
|
| 12 |
+
"batch_size": 1,
|
| 13 |
+
"sample_ids": [
|
| 14 |
+
"testA-29199-80373",
|
| 15 |
+
"testB-44351-122075",
|
| 16 |
+
"testA-91850-253019",
|
| 17 |
+
"testB-36888-101641",
|
| 18 |
+
"testA-54586-150233",
|
| 19 |
+
"testB-79644-219272",
|
| 20 |
+
"testB-65935-181776",
|
| 21 |
+
"testA-81622-224664",
|
| 22 |
+
"testA-10078-27739",
|
| 23 |
+
"testA-15582-42797",
|
| 24 |
+
"testA-5038-13943",
|
| 25 |
+
"testA-30844-84861",
|
| 26 |
+
"testB-15419-42356",
|
| 27 |
+
"testB-27486-75580",
|
| 28 |
+
"testB-945-2632",
|
| 29 |
+
"testA-24438-67190",
|
| 30 |
+
"testB-66161-182405",
|
| 31 |
+
"testB-42726-117659",
|
| 32 |
+
"testB-69439-191450",
|
| 33 |
+
"testA-54307-149426",
|
| 34 |
+
"testB-37669-103831",
|
| 35 |
+
"testB-50844-140105",
|
| 36 |
+
"testA-79438-218690",
|
| 37 |
+
"testA-62940-173454",
|
| 38 |
+
"testA-81245-223632",
|
| 39 |
+
"testA-47070-129599",
|
| 40 |
+
"testB-12381-34098",
|
| 41 |
+
"testA-44142-121498",
|
| 42 |
+
"testA-73258-201893",
|
| 43 |
+
"testB-46644-128436",
|
| 44 |
+
"testA-43889-120792",
|
| 45 |
+
"testB-36082-99363",
|
| 46 |
+
"testB-33108-91152",
|
| 47 |
+
"testA-26856-73830",
|
| 48 |
+
"testA-81271-223709",
|
| 49 |
+
"testA-54554-150141",
|
| 50 |
+
"testB-38636-106481",
|
| 51 |
+
"testA-5734-15837",
|
| 52 |
+
"testB-67558-186325",
|
| 53 |
+
"testB-47053-129554",
|
| 54 |
+
"testA-93064-256422",
|
| 55 |
+
"testB-60557-166821",
|
| 56 |
+
"testA-55116-151688",
|
| 57 |
+
"testB-65796-181392",
|
| 58 |
+
"testB-23014-63278",
|
| 59 |
+
"testA-24562-67515",
|
| 60 |
+
"testA-17736-48684",
|
| 61 |
+
"testA-58475-160949",
|
| 62 |
+
"testA-80455-221493",
|
| 63 |
+
"testA-31763-87405",
|
| 64 |
+
"testB-23920-65742",
|
| 65 |
+
"testA-469-1309",
|
| 66 |
+
"testB-56691-155998",
|
| 67 |
+
"testB-31493-86668",
|
| 68 |
+
"testA-59284-163254",
|
| 69 |
+
"testA-57397-158015",
|
| 70 |
+
"testB-2935-8111",
|
| 71 |
+
"testB-22290-61226",
|
| 72 |
+
"testB-8416-23140",
|
| 73 |
+
"testA-60731-167320",
|
| 74 |
+
"testA-49055-135086",
|
| 75 |
+
"testA-7857-21644",
|
| 76 |
+
"testB-79645-219273",
|
| 77 |
+
"testB-52921-145707",
|
| 78 |
+
"testB-69479-191562",
|
| 79 |
+
"testA-69466-191524"
|
| 80 |
+
],
|
| 81 |
+
"batches": [
|
| 82 |
+
{
|
| 83 |
+
"sample_ids": [
|
| 84 |
+
"testA-91850-253019"
|
| 85 |
+
],
|
| 86 |
+
"batch_size": 1,
|
| 87 |
+
"generation_latency_seconds": 3.147620694944635,
|
| 88 |
+
"batch_wall_latency_seconds": 3.1645371050108224,
|
| 89 |
+
"amortized_generation_ms_per_sample": 3147.620694944635,
|
| 90 |
+
"amortized_batch_wall_ms_per_sample": 3164.5371050108224,
|
| 91 |
+
"outputs": [
|
| 92 |
+
{
|
| 93 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [496, 292, 713, 655], \"score\": 0.9}]}",
|
| 94 |
+
"valid_prediction": {
|
| 95 |
+
"objects": [
|
| 96 |
+
{
|
| 97 |
+
"bbox_2d": [
|
| 98 |
+
496,
|
| 99 |
+
292,
|
| 100 |
+
713,
|
| 101 |
+
655
|
| 102 |
+
],
|
| 103 |
+
"score": 0.9
|
| 104 |
+
}
|
| 105 |
+
]
|
| 106 |
+
},
|
| 107 |
+
"error": null,
|
| 108 |
+
"finish_reason": "stop",
|
| 109 |
+
"completion_tokens": 39,
|
| 110 |
+
"prompt_tokens": 570
|
| 111 |
+
}
|
| 112 |
+
]
|
| 113 |
+
},
|
| 114 |
+
{
|
| 115 |
+
"sample_ids": [
|
| 116 |
+
"testB-36888-101641"
|
| 117 |
+
],
|
| 118 |
+
"batch_size": 1,
|
| 119 |
+
"generation_latency_seconds": 4.945294587058015,
|
| 120 |
+
"batch_wall_latency_seconds": 4.966181440046057,
|
| 121 |
+
"amortized_generation_ms_per_sample": 4945.294587058015,
|
| 122 |
+
"amortized_batch_wall_ms_per_sample": 4966.181440046057,
|
| 123 |
+
"outputs": [
|
| 124 |
+
{
|
| 125 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[21,567,693,915],\"score\":0.95},{\"bbox_2d\":[381,233,959,885],\"score\":0.95}]}",
|
| 126 |
+
"valid_prediction": {
|
| 127 |
+
"objects": [
|
| 128 |
+
{
|
| 129 |
+
"bbox_2d": [
|
| 130 |
+
21,
|
| 131 |
+
567,
|
| 132 |
+
693,
|
| 133 |
+
915
|
| 134 |
+
],
|
| 135 |
+
"score": 0.95
|
| 136 |
+
},
|
| 137 |
+
{
|
| 138 |
+
"bbox_2d": [
|
| 139 |
+
381,
|
| 140 |
+
233,
|
| 141 |
+
959,
|
| 142 |
+
885
|
| 143 |
+
],
|
| 144 |
+
"score": 0.95
|
| 145 |
+
}
|
| 146 |
+
]
|
| 147 |
+
},
|
| 148 |
+
"error": null,
|
| 149 |
+
"finish_reason": "stop",
|
| 150 |
+
"completion_tokens": 60,
|
| 151 |
+
"prompt_tokens": 608
|
| 152 |
+
}
|
| 153 |
+
]
|
| 154 |
+
},
|
| 155 |
+
{
|
| 156 |
+
"sample_ids": [
|
| 157 |
+
"testA-54586-150233"
|
| 158 |
+
],
|
| 159 |
+
"batch_size": 1,
|
| 160 |
+
"generation_latency_seconds": 3.2111774859949946,
|
| 161 |
+
"batch_wall_latency_seconds": 3.2314598000375554,
|
| 162 |
+
"amortized_generation_ms_per_sample": 3211.1774859949946,
|
| 163 |
+
"amortized_batch_wall_ms_per_sample": 3231.4598000375554,
|
| 164 |
+
"outputs": [
|
| 165 |
+
{
|
| 166 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [693, 438, 983, 813], \"score\": 0.95}]}",
|
| 167 |
+
"valid_prediction": {
|
| 168 |
+
"objects": [
|
| 169 |
+
{
|
| 170 |
+
"bbox_2d": [
|
| 171 |
+
693,
|
| 172 |
+
438,
|
| 173 |
+
983,
|
| 174 |
+
813
|
| 175 |
+
],
|
| 176 |
+
"score": 0.95
|
| 177 |
+
}
|
| 178 |
+
]
|
| 179 |
+
},
|
| 180 |
+
"error": null,
|
| 181 |
+
"finish_reason": "stop",
|
| 182 |
+
"completion_tokens": 40,
|
| 183 |
+
"prompt_tokens": 707
|
| 184 |
+
}
|
| 185 |
+
]
|
| 186 |
+
},
|
| 187 |
+
{
|
| 188 |
+
"sample_ids": [
|
| 189 |
+
"testB-79644-219272"
|
| 190 |
+
],
|
| 191 |
+
"batch_size": 1,
|
| 192 |
+
"generation_latency_seconds": 33.390286731068045,
|
| 193 |
+
"batch_wall_latency_seconds": 33.40712711191736,
|
| 194 |
+
"amortized_generation_ms_per_sample": 33390.286731068045,
|
| 195 |
+
"amortized_batch_wall_ms_per_sample": 33407.12711191736,
|
| 196 |
+
"outputs": [
|
| 197 |
+
{
|
| 198 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [121, 581, 170, 764], \"score\": 0.9}, {\"bbox_2d\": [173, 595, 259, 837], \"score\": 0.9}, {\"bbox_2d\": [265, 581, 315, 806], \"score\": 0.9}, {\"bbox_2d\": [310, 571, 351, 764], \"score\": 0.9}, {\"bbox_2d\": [338, 581, 387, 764], \"score\": 0.9}, {\"bbox_2d\": [449, 575, 492, 756], \"score\": 0.9}, {\"bbox_2d\": [475, 604, 542, 806], \"score\": 0.9}, {\"bbox_2d\": [668, 575, 743, 818], \"score\": 0.9}, {\"bbox_2d\": [740, 571, 783, 775], \"score\": 0.9}, {\"bbox_2d\": [771, 571, 814, 787], \"score\": 0.9}, {\"bbox_2d\": [822, 587, 851, 764], \"score\": 0.9}, {\"bbox_2d\": [847, 591, 921, 903], \"score\": 0.9}, {\"bbox_2d\": [902, 604, 959, 795], \"score\": 0.9}]}",
|
| 199 |
+
"valid_prediction": {
|
| 200 |
+
"objects": [
|
| 201 |
+
{
|
| 202 |
+
"bbox_2d": [
|
| 203 |
+
121,
|
| 204 |
+
581,
|
| 205 |
+
170,
|
| 206 |
+
764
|
| 207 |
+
],
|
| 208 |
+
"score": 0.9
|
| 209 |
+
},
|
| 210 |
+
{
|
| 211 |
+
"bbox_2d": [
|
| 212 |
+
173,
|
| 213 |
+
595,
|
| 214 |
+
259,
|
| 215 |
+
837
|
| 216 |
+
],
|
| 217 |
+
"score": 0.9
|
| 218 |
+
},
|
| 219 |
+
{
|
| 220 |
+
"bbox_2d": [
|
| 221 |
+
265,
|
| 222 |
+
581,
|
| 223 |
+
315,
|
| 224 |
+
806
|
| 225 |
+
],
|
| 226 |
+
"score": 0.9
|
| 227 |
+
},
|
| 228 |
+
{
|
| 229 |
+
"bbox_2d": [
|
| 230 |
+
310,
|
| 231 |
+
571,
|
| 232 |
+
351,
|
| 233 |
+
764
|
| 234 |
+
],
|
| 235 |
+
"score": 0.9
|
| 236 |
+
},
|
| 237 |
+
{
|
| 238 |
+
"bbox_2d": [
|
| 239 |
+
338,
|
| 240 |
+
581,
|
| 241 |
+
387,
|
| 242 |
+
764
|
| 243 |
+
],
|
| 244 |
+
"score": 0.9
|
| 245 |
+
},
|
| 246 |
+
{
|
| 247 |
+
"bbox_2d": [
|
| 248 |
+
449,
|
| 249 |
+
575,
|
| 250 |
+
492,
|
| 251 |
+
756
|
| 252 |
+
],
|
| 253 |
+
"score": 0.9
|
| 254 |
+
},
|
| 255 |
+
{
|
| 256 |
+
"bbox_2d": [
|
| 257 |
+
475,
|
| 258 |
+
604,
|
| 259 |
+
542,
|
| 260 |
+
806
|
| 261 |
+
],
|
| 262 |
+
"score": 0.9
|
| 263 |
+
},
|
| 264 |
+
{
|
| 265 |
+
"bbox_2d": [
|
| 266 |
+
668,
|
| 267 |
+
575,
|
| 268 |
+
743,
|
| 269 |
+
818
|
| 270 |
+
],
|
| 271 |
+
"score": 0.9
|
| 272 |
+
},
|
| 273 |
+
{
|
| 274 |
+
"bbox_2d": [
|
| 275 |
+
740,
|
| 276 |
+
571,
|
| 277 |
+
783,
|
| 278 |
+
775
|
| 279 |
+
],
|
| 280 |
+
"score": 0.9
|
| 281 |
+
},
|
| 282 |
+
{
|
| 283 |
+
"bbox_2d": [
|
| 284 |
+
771,
|
| 285 |
+
571,
|
| 286 |
+
814,
|
| 287 |
+
787
|
| 288 |
+
],
|
| 289 |
+
"score": 0.9
|
| 290 |
+
},
|
| 291 |
+
{
|
| 292 |
+
"bbox_2d": [
|
| 293 |
+
822,
|
| 294 |
+
587,
|
| 295 |
+
851,
|
| 296 |
+
764
|
| 297 |
+
],
|
| 298 |
+
"score": 0.9
|
| 299 |
+
},
|
| 300 |
+
{
|
| 301 |
+
"bbox_2d": [
|
| 302 |
+
847,
|
| 303 |
+
591,
|
| 304 |
+
921,
|
| 305 |
+
903
|
| 306 |
+
],
|
| 307 |
+
"score": 0.9
|
| 308 |
+
},
|
| 309 |
+
{
|
| 310 |
+
"bbox_2d": [
|
| 311 |
+
902,
|
| 312 |
+
604,
|
| 313 |
+
959,
|
| 314 |
+
795
|
| 315 |
+
],
|
| 316 |
+
"score": 0.9
|
| 317 |
+
}
|
| 318 |
+
]
|
| 319 |
+
},
|
| 320 |
+
"error": null,
|
| 321 |
+
"finish_reason": "stop",
|
| 322 |
+
"completion_tokens": 447,
|
| 323 |
+
"prompt_tokens": 600
|
| 324 |
+
}
|
| 325 |
+
]
|
| 326 |
+
},
|
| 327 |
+
{
|
| 328 |
+
"sample_ids": [
|
| 329 |
+
"testB-65935-181776"
|
| 330 |
+
],
|
| 331 |
+
"batch_size": 1,
|
| 332 |
+
"generation_latency_seconds": 4.57236669503618,
|
| 333 |
+
"batch_wall_latency_seconds": 4.583186280098744,
|
| 334 |
+
"amortized_generation_ms_per_sample": 4572.36669503618,
|
| 335 |
+
"amortized_batch_wall_ms_per_sample": 4583.186280098744,
|
| 336 |
+
"outputs": [
|
| 337 |
+
{
|
| 338 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[347,333,951,761],\"score\":0.95},{\"bbox_2d\":[246,421,777,987],\"score\":0.95}]}",
|
| 339 |
+
"valid_prediction": {
|
| 340 |
+
"objects": [
|
| 341 |
+
{
|
| 342 |
+
"bbox_2d": [
|
| 343 |
+
347,
|
| 344 |
+
333,
|
| 345 |
+
951,
|
| 346 |
+
761
|
| 347 |
+
],
|
| 348 |
+
"score": 0.95
|
| 349 |
+
},
|
| 350 |
+
{
|
| 351 |
+
"bbox_2d": [
|
| 352 |
+
246,
|
| 353 |
+
421,
|
| 354 |
+
777,
|
| 355 |
+
987
|
| 356 |
+
],
|
| 357 |
+
"score": 0.95
|
| 358 |
+
}
|
| 359 |
+
]
|
| 360 |
+
},
|
| 361 |
+
"error": null,
|
| 362 |
+
"finish_reason": "stop",
|
| 363 |
+
"completion_tokens": 61,
|
| 364 |
+
"prompt_tokens": 564
|
| 365 |
+
}
|
| 366 |
+
]
|
| 367 |
+
},
|
| 368 |
+
{
|
| 369 |
+
"sample_ids": [
|
| 370 |
+
"testA-81622-224664"
|
| 371 |
+
],
|
| 372 |
+
"batch_size": 1,
|
| 373 |
+
"generation_latency_seconds": 0.6317770309979096,
|
| 374 |
+
"batch_wall_latency_seconds": 0.6445289449766278,
|
| 375 |
+
"amortized_generation_ms_per_sample": 631.7770309979096,
|
| 376 |
+
"amortized_batch_wall_ms_per_sample": 644.5289449766278,
|
| 377 |
+
"outputs": [
|
| 378 |
+
{
|
| 379 |
+
"raw_response": "{\"objects\": []}",
|
| 380 |
+
"valid_prediction": {
|
| 381 |
+
"objects": []
|
| 382 |
+
},
|
| 383 |
+
"error": null,
|
| 384 |
+
"finish_reason": "stop",
|
| 385 |
+
"completion_tokens": 6,
|
| 386 |
+
"prompt_tokens": 564
|
| 387 |
+
}
|
| 388 |
+
]
|
| 389 |
+
},
|
| 390 |
+
{
|
| 391 |
+
"sample_ids": [
|
| 392 |
+
"testA-10078-27739"
|
| 393 |
+
],
|
| 394 |
+
"batch_size": 1,
|
| 395 |
+
"generation_latency_seconds": 2.393106485949829,
|
| 396 |
+
"batch_wall_latency_seconds": 2.4061070699244738,
|
| 397 |
+
"amortized_generation_ms_per_sample": 2393.106485949829,
|
| 398 |
+
"amortized_batch_wall_ms_per_sample": 2406.1070699244738,
|
| 399 |
+
"outputs": [
|
| 400 |
+
{
|
| 401 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[283,97,775,810],\"score\":0.95}]}",
|
| 402 |
+
"valid_prediction": {
|
| 403 |
+
"objects": [
|
| 404 |
+
{
|
| 405 |
+
"bbox_2d": [
|
| 406 |
+
283,
|
| 407 |
+
97,
|
| 408 |
+
775,
|
| 409 |
+
810
|
| 410 |
+
],
|
| 411 |
+
"score": 0.95
|
| 412 |
+
}
|
| 413 |
+
]
|
| 414 |
+
},
|
| 415 |
+
"error": null,
|
| 416 |
+
"finish_reason": "stop",
|
| 417 |
+
"completion_tokens": 32,
|
| 418 |
+
"prompt_tokens": 564
|
| 419 |
+
}
|
| 420 |
+
]
|
| 421 |
+
},
|
| 422 |
+
{
|
| 423 |
+
"sample_ids": [
|
| 424 |
+
"testA-15582-42797"
|
| 425 |
+
],
|
| 426 |
+
"batch_size": 1,
|
| 427 |
+
"generation_latency_seconds": 7.63340066303499,
|
| 428 |
+
"batch_wall_latency_seconds": 7.651124170981348,
|
| 429 |
+
"amortized_generation_ms_per_sample": 7633.40066303499,
|
| 430 |
+
"amortized_batch_wall_ms_per_sample": 7651.124170981348,
|
| 431 |
+
"outputs": [
|
| 432 |
+
{
|
| 433 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [85, 297, 446, 999], \"score\": 0.9}, {\"bbox_2d\": [315, 413, 542, 999], \"score\": 0.9}, {\"bbox_2d\": [542, 476, 642, 876], \"score\": 0.9}]}",
|
| 434 |
+
"valid_prediction": {
|
| 435 |
+
"objects": [
|
| 436 |
+
{
|
| 437 |
+
"bbox_2d": [
|
| 438 |
+
85,
|
| 439 |
+
297,
|
| 440 |
+
446,
|
| 441 |
+
999
|
| 442 |
+
],
|
| 443 |
+
"score": 0.9
|
| 444 |
+
},
|
| 445 |
+
{
|
| 446 |
+
"bbox_2d": [
|
| 447 |
+
315,
|
| 448 |
+
413,
|
| 449 |
+
542,
|
| 450 |
+
999
|
| 451 |
+
],
|
| 452 |
+
"score": 0.9
|
| 453 |
+
},
|
| 454 |
+
{
|
| 455 |
+
"bbox_2d": [
|
| 456 |
+
542,
|
| 457 |
+
476,
|
| 458 |
+
642,
|
| 459 |
+
876
|
| 460 |
+
],
|
| 461 |
+
"score": 0.9
|
| 462 |
+
}
|
| 463 |
+
]
|
| 464 |
+
},
|
| 465 |
+
"error": null,
|
| 466 |
+
"finish_reason": "stop",
|
| 467 |
+
"completion_tokens": 106,
|
| 468 |
+
"prompt_tokens": 619
|
| 469 |
+
}
|
| 470 |
+
]
|
| 471 |
+
},
|
| 472 |
+
{
|
| 473 |
+
"sample_ids": [
|
| 474 |
+
"testA-5038-13943"
|
| 475 |
+
],
|
| 476 |
+
"batch_size": 1,
|
| 477 |
+
"generation_latency_seconds": 3.06427818199154,
|
| 478 |
+
"batch_wall_latency_seconds": 3.0877813909901306,
|
| 479 |
+
"amortized_generation_ms_per_sample": 3064.27818199154,
|
| 480 |
+
"amortized_batch_wall_ms_per_sample": 3087.7813909901306,
|
| 481 |
+
"outputs": [
|
| 482 |
+
{
|
| 483 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [521, 246, 793, 999], \"score\": 0.95}]}",
|
| 484 |
+
"valid_prediction": {
|
| 485 |
+
"objects": [
|
| 486 |
+
{
|
| 487 |
+
"bbox_2d": [
|
| 488 |
+
521,
|
| 489 |
+
246,
|
| 490 |
+
793,
|
| 491 |
+
999
|
| 492 |
+
],
|
| 493 |
+
"score": 0.95
|
| 494 |
+
}
|
| 495 |
+
]
|
| 496 |
+
},
|
| 497 |
+
"error": null,
|
| 498 |
+
"finish_reason": "stop",
|
| 499 |
+
"completion_tokens": 40,
|
| 500 |
+
"prompt_tokens": 604
|
| 501 |
+
}
|
| 502 |
+
]
|
| 503 |
+
},
|
| 504 |
+
{
|
| 505 |
+
"sample_ids": [
|
| 506 |
+
"testA-30844-84861"
|
| 507 |
+
],
|
| 508 |
+
"batch_size": 1,
|
| 509 |
+
"generation_latency_seconds": 2.516077253036201,
|
| 510 |
+
"batch_wall_latency_seconds": 2.5232016970403492,
|
| 511 |
+
"amortized_generation_ms_per_sample": 2516.077253036201,
|
| 512 |
+
"amortized_batch_wall_ms_per_sample": 2523.2016970403492,
|
| 513 |
+
"outputs": [
|
| 514 |
+
{
|
| 515 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[508,536,807,722],\"score\":0.9}]}",
|
| 516 |
+
"valid_prediction": {
|
| 517 |
+
"objects": [
|
| 518 |
+
{
|
| 519 |
+
"bbox_2d": [
|
| 520 |
+
508,
|
| 521 |
+
536,
|
| 522 |
+
807,
|
| 523 |
+
722
|
| 524 |
+
],
|
| 525 |
+
"score": 0.9
|
| 526 |
+
}
|
| 527 |
+
]
|
| 528 |
+
},
|
| 529 |
+
"error": null,
|
| 530 |
+
"finish_reason": "stop",
|
| 531 |
+
"completion_tokens": 32,
|
| 532 |
+
"prompt_tokens": 623
|
| 533 |
+
}
|
| 534 |
+
]
|
| 535 |
+
},
|
| 536 |
+
{
|
| 537 |
+
"sample_ids": [
|
| 538 |
+
"testB-15419-42356"
|
| 539 |
+
],
|
| 540 |
+
"batch_size": 1,
|
| 541 |
+
"generation_latency_seconds": 2.673266503959894,
|
| 542 |
+
"batch_wall_latency_seconds": 2.6889539459953085,
|
| 543 |
+
"amortized_generation_ms_per_sample": 2673.266503959894,
|
| 544 |
+
"amortized_batch_wall_ms_per_sample": 2688.9539459953085,
|
| 545 |
+
"outputs": [
|
| 546 |
+
{
|
| 547 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[107,722,587,999],\"score\":0.9}]}",
|
| 548 |
+
"valid_prediction": {
|
| 549 |
+
"objects": [
|
| 550 |
+
{
|
| 551 |
+
"bbox_2d": [
|
| 552 |
+
107,
|
| 553 |
+
722,
|
| 554 |
+
587,
|
| 555 |
+
999
|
| 556 |
+
],
|
| 557 |
+
"score": 0.9
|
| 558 |
+
}
|
| 559 |
+
]
|
| 560 |
+
},
|
| 561 |
+
"error": null,
|
| 562 |
+
"finish_reason": "stop",
|
| 563 |
+
"completion_tokens": 32,
|
| 564 |
+
"prompt_tokens": 566
|
| 565 |
+
}
|
| 566 |
+
]
|
| 567 |
+
},
|
| 568 |
+
{
|
| 569 |
+
"sample_ids": [
|
| 570 |
+
"testB-27486-75580"
|
| 571 |
+
],
|
| 572 |
+
"batch_size": 1,
|
| 573 |
+
"generation_latency_seconds": 4.617897133924998,
|
| 574 |
+
"batch_wall_latency_seconds": 4.632912510889582,
|
| 575 |
+
"amortized_generation_ms_per_sample": 4617.897133924998,
|
| 576 |
+
"amortized_batch_wall_ms_per_sample": 4632.912510889582,
|
| 577 |
+
"outputs": [
|
| 578 |
+
{
|
| 579 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[542,393,999,999],\"score\":0.95},{\"bbox_2d\":[646,271,911,471],\"score\":0.95}]}",
|
| 580 |
+
"valid_prediction": {
|
| 581 |
+
"objects": [
|
| 582 |
+
{
|
| 583 |
+
"bbox_2d": [
|
| 584 |
+
542,
|
| 585 |
+
393,
|
| 586 |
+
999,
|
| 587 |
+
999
|
| 588 |
+
],
|
| 589 |
+
"score": 0.95
|
| 590 |
+
},
|
| 591 |
+
{
|
| 592 |
+
"bbox_2d": [
|
| 593 |
+
646,
|
| 594 |
+
271,
|
| 595 |
+
911,
|
| 596 |
+
471
|
| 597 |
+
],
|
| 598 |
+
"score": 0.95
|
| 599 |
+
}
|
| 600 |
+
]
|
| 601 |
+
},
|
| 602 |
+
"error": null,
|
| 603 |
+
"finish_reason": "stop",
|
| 604 |
+
"completion_tokens": 61,
|
| 605 |
+
"prompt_tokens": 670
|
| 606 |
+
}
|
| 607 |
+
]
|
| 608 |
+
},
|
| 609 |
+
{
|
| 610 |
+
"sample_ids": [
|
| 611 |
+
"testB-945-2632"
|
| 612 |
+
],
|
| 613 |
+
"batch_size": 1,
|
| 614 |
+
"generation_latency_seconds": 12.483138478011824,
|
| 615 |
+
"batch_wall_latency_seconds": 12.503725816030055,
|
| 616 |
+
"amortized_generation_ms_per_sample": 12483.138478011824,
|
| 617 |
+
"amortized_batch_wall_ms_per_sample": 12503.725816030055,
|
| 618 |
+
"outputs": [
|
| 619 |
+
{
|
| 620 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [213, 692, 283, 983], \"score\": 0.9}, {\"bbox_2d\": [321, 661, 415, 973], \"score\": 0.9}, {\"bbox_2d\": [473, 661, 555, 966], \"score\": 0.9}, {\"bbox_2d\": [638, 626, 713, 973], \"score\": 0.9}, {\"bbox_2d\": [775, 638, 871, 988], \"score\": 0.9}]}",
|
| 621 |
+
"valid_prediction": {
|
| 622 |
+
"objects": [
|
| 623 |
+
{
|
| 624 |
+
"bbox_2d": [
|
| 625 |
+
213,
|
| 626 |
+
692,
|
| 627 |
+
283,
|
| 628 |
+
983
|
| 629 |
+
],
|
| 630 |
+
"score": 0.9
|
| 631 |
+
},
|
| 632 |
+
{
|
| 633 |
+
"bbox_2d": [
|
| 634 |
+
321,
|
| 635 |
+
661,
|
| 636 |
+
415,
|
| 637 |
+
973
|
| 638 |
+
],
|
| 639 |
+
"score": 0.9
|
| 640 |
+
},
|
| 641 |
+
{
|
| 642 |
+
"bbox_2d": [
|
| 643 |
+
473,
|
| 644 |
+
661,
|
| 645 |
+
555,
|
| 646 |
+
966
|
| 647 |
+
],
|
| 648 |
+
"score": 0.9
|
| 649 |
+
},
|
| 650 |
+
{
|
| 651 |
+
"bbox_2d": [
|
| 652 |
+
638,
|
| 653 |
+
626,
|
| 654 |
+
713,
|
| 655 |
+
973
|
| 656 |
+
],
|
| 657 |
+
"score": 0.9
|
| 658 |
+
},
|
| 659 |
+
{
|
| 660 |
+
"bbox_2d": [
|
| 661 |
+
775,
|
| 662 |
+
638,
|
| 663 |
+
871,
|
| 664 |
+
988
|
| 665 |
+
],
|
| 666 |
+
"score": 0.9
|
| 667 |
+
}
|
| 668 |
+
]
|
| 669 |
+
},
|
| 670 |
+
"error": null,
|
| 671 |
+
"finish_reason": "stop",
|
| 672 |
+
"completion_tokens": 175,
|
| 673 |
+
"prompt_tokens": 670
|
| 674 |
+
}
|
| 675 |
+
]
|
| 676 |
+
},
|
| 677 |
+
{
|
| 678 |
+
"sample_ids": [
|
| 679 |
+
"testA-24438-67190"
|
| 680 |
+
],
|
| 681 |
+
"batch_size": 1,
|
| 682 |
+
"generation_latency_seconds": 4.798777529038489,
|
| 683 |
+
"batch_wall_latency_seconds": 4.815012337057851,
|
| 684 |
+
"amortized_generation_ms_per_sample": 4798.777529038489,
|
| 685 |
+
"amortized_batch_wall_ms_per_sample": 4815.012337057851,
|
| 686 |
+
"outputs": [
|
| 687 |
+
{
|
| 688 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[731,375,999,999],\"score\":0.95},{\"bbox_2d\":[305,247,585,928],\"score\":0.95}]}",
|
| 689 |
+
"valid_prediction": {
|
| 690 |
+
"objects": [
|
| 691 |
+
{
|
| 692 |
+
"bbox_2d": [
|
| 693 |
+
731,
|
| 694 |
+
375,
|
| 695 |
+
999,
|
| 696 |
+
999
|
| 697 |
+
],
|
| 698 |
+
"score": 0.95
|
| 699 |
+
},
|
| 700 |
+
{
|
| 701 |
+
"bbox_2d": [
|
| 702 |
+
305,
|
| 703 |
+
247,
|
| 704 |
+
585,
|
| 705 |
+
928
|
| 706 |
+
],
|
| 707 |
+
"score": 0.95
|
| 708 |
+
}
|
| 709 |
+
]
|
| 710 |
+
},
|
| 711 |
+
"error": null,
|
| 712 |
+
"finish_reason": "stop",
|
| 713 |
+
"completion_tokens": 61,
|
| 714 |
+
"prompt_tokens": 605
|
| 715 |
+
}
|
| 716 |
+
]
|
| 717 |
+
},
|
| 718 |
+
{
|
| 719 |
+
"sample_ids": [
|
| 720 |
+
"testB-66161-182405"
|
| 721 |
+
],
|
| 722 |
+
"batch_size": 1,
|
| 723 |
+
"generation_latency_seconds": 2.5812068269588053,
|
| 724 |
+
"batch_wall_latency_seconds": 2.593027200899087,
|
| 725 |
+
"amortized_generation_ms_per_sample": 2581.2068269588053,
|
| 726 |
+
"amortized_batch_wall_ms_per_sample": 2593.027200899087,
|
| 727 |
+
"outputs": [
|
| 728 |
+
{
|
| 729 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[103,193,755,713],\"score\":0.95}]}",
|
| 730 |
+
"valid_prediction": {
|
| 731 |
+
"objects": [
|
| 732 |
+
{
|
| 733 |
+
"bbox_2d": [
|
| 734 |
+
103,
|
| 735 |
+
193,
|
| 736 |
+
755,
|
| 737 |
+
713
|
| 738 |
+
],
|
| 739 |
+
"score": 0.95
|
| 740 |
+
}
|
| 741 |
+
]
|
| 742 |
+
},
|
| 743 |
+
"error": null,
|
| 744 |
+
"finish_reason": "stop",
|
| 745 |
+
"completion_tokens": 33,
|
| 746 |
+
"prompt_tokens": 563
|
| 747 |
+
}
|
| 748 |
+
]
|
| 749 |
+
},
|
| 750 |
+
{
|
| 751 |
+
"sample_ids": [
|
| 752 |
+
"testB-42726-117659"
|
| 753 |
+
],
|
| 754 |
+
"batch_size": 1,
|
| 755 |
+
"generation_latency_seconds": 3.0745598620269448,
|
| 756 |
+
"batch_wall_latency_seconds": 3.0872938209213316,
|
| 757 |
+
"amortized_generation_ms_per_sample": 3074.5598620269448,
|
| 758 |
+
"amortized_batch_wall_ms_per_sample": 3087.2938209213316,
|
| 759 |
+
"outputs": [
|
| 760 |
+
{
|
| 761 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [743, 458, 999, 878], \"score\": 0.95}]}",
|
| 762 |
+
"valid_prediction": {
|
| 763 |
+
"objects": [
|
| 764 |
+
{
|
| 765 |
+
"bbox_2d": [
|
| 766 |
+
743,
|
| 767 |
+
458,
|
| 768 |
+
999,
|
| 769 |
+
878
|
| 770 |
+
],
|
| 771 |
+
"score": 0.95
|
| 772 |
+
}
|
| 773 |
+
]
|
| 774 |
+
},
|
| 775 |
+
"error": null,
|
| 776 |
+
"finish_reason": "stop",
|
| 777 |
+
"completion_tokens": 40,
|
| 778 |
+
"prompt_tokens": 604
|
| 779 |
+
}
|
| 780 |
+
]
|
| 781 |
+
},
|
| 782 |
+
{
|
| 783 |
+
"sample_ids": [
|
| 784 |
+
"testB-69439-191450"
|
| 785 |
+
],
|
| 786 |
+
"batch_size": 1,
|
| 787 |
+
"generation_latency_seconds": 0.5888717719353735,
|
| 788 |
+
"batch_wall_latency_seconds": 0.5951321049360558,
|
| 789 |
+
"amortized_generation_ms_per_sample": 588.8717719353735,
|
| 790 |
+
"amortized_batch_wall_ms_per_sample": 595.1321049360558,
|
| 791 |
+
"outputs": [
|
| 792 |
+
{
|
| 793 |
+
"raw_response": "{\"objects\": []}",
|
| 794 |
+
"valid_prediction": {
|
| 795 |
+
"objects": []
|
| 796 |
+
},
|
| 797 |
+
"error": null,
|
| 798 |
+
"finish_reason": "stop",
|
| 799 |
+
"completion_tokens": 6,
|
| 800 |
+
"prompt_tokens": 492
|
| 801 |
+
}
|
| 802 |
+
]
|
| 803 |
+
},
|
| 804 |
+
{
|
| 805 |
+
"sample_ids": [
|
| 806 |
+
"testA-54307-149426"
|
| 807 |
+
],
|
| 808 |
+
"batch_size": 1,
|
| 809 |
+
"generation_latency_seconds": 4.398384497035295,
|
| 810 |
+
"batch_wall_latency_seconds": 4.4204836380667984,
|
| 811 |
+
"amortized_generation_ms_per_sample": 4398.384497035295,
|
| 812 |
+
"amortized_batch_wall_ms_per_sample": 4420.483638066798,
|
| 813 |
+
"outputs": [
|
| 814 |
+
{
|
| 815 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[542,48,838,923],\"score\":0.95},{\"bbox_2d\":[177,0,567,477],\"score\":0.95}]}",
|
| 816 |
+
"valid_prediction": {
|
| 817 |
+
"objects": [
|
| 818 |
+
{
|
| 819 |
+
"bbox_2d": [
|
| 820 |
+
542,
|
| 821 |
+
48,
|
| 822 |
+
838,
|
| 823 |
+
923
|
| 824 |
+
],
|
| 825 |
+
"score": 0.95
|
| 826 |
+
},
|
| 827 |
+
{
|
| 828 |
+
"bbox_2d": [
|
| 829 |
+
177,
|
| 830 |
+
0,
|
| 831 |
+
567,
|
| 832 |
+
477
|
| 833 |
+
],
|
| 834 |
+
"score": 0.95
|
| 835 |
+
}
|
| 836 |
+
]
|
| 837 |
+
},
|
| 838 |
+
"error": null,
|
| 839 |
+
"finish_reason": "stop",
|
| 840 |
+
"completion_tokens": 58,
|
| 841 |
+
"prompt_tokens": 564
|
| 842 |
+
}
|
| 843 |
+
]
|
| 844 |
+
},
|
| 845 |
+
{
|
| 846 |
+
"sample_ids": [
|
| 847 |
+
"testB-37669-103831"
|
| 848 |
+
],
|
| 849 |
+
"batch_size": 1,
|
| 850 |
+
"generation_latency_seconds": 0.6747266949387267,
|
| 851 |
+
"batch_wall_latency_seconds": 0.699499664013274,
|
| 852 |
+
"amortized_generation_ms_per_sample": 674.7266949387267,
|
| 853 |
+
"amortized_batch_wall_ms_per_sample": 699.499664013274,
|
| 854 |
+
"outputs": [
|
| 855 |
+
{
|
| 856 |
+
"raw_response": "{\"objects\": []}",
|
| 857 |
+
"valid_prediction": {
|
| 858 |
+
"objects": []
|
| 859 |
+
},
|
| 860 |
+
"error": null,
|
| 861 |
+
"finish_reason": "stop",
|
| 862 |
+
"completion_tokens": 6,
|
| 863 |
+
"prompt_tokens": 648
|
| 864 |
+
}
|
| 865 |
+
]
|
| 866 |
+
},
|
| 867 |
+
{
|
| 868 |
+
"sample_ids": [
|
| 869 |
+
"testB-50844-140105"
|
| 870 |
+
],
|
| 871 |
+
"batch_size": 1,
|
| 872 |
+
"generation_latency_seconds": 0.652067915070802,
|
| 873 |
+
"batch_wall_latency_seconds": 0.6656363360816613,
|
| 874 |
+
"amortized_generation_ms_per_sample": 652.067915070802,
|
| 875 |
+
"amortized_batch_wall_ms_per_sample": 665.6363360816613,
|
| 876 |
+
"outputs": [
|
| 877 |
+
{
|
| 878 |
+
"raw_response": "{\"objects\": []}",
|
| 879 |
+
"valid_prediction": {
|
| 880 |
+
"objects": []
|
| 881 |
+
},
|
| 882 |
+
"error": null,
|
| 883 |
+
"finish_reason": "stop",
|
| 884 |
+
"completion_tokens": 6,
|
| 885 |
+
"prompt_tokens": 561
|
| 886 |
+
}
|
| 887 |
+
]
|
| 888 |
+
},
|
| 889 |
+
{
|
| 890 |
+
"sample_ids": [
|
| 891 |
+
"testA-79438-218690"
|
| 892 |
+
],
|
| 893 |
+
"batch_size": 1,
|
| 894 |
+
"generation_latency_seconds": 3.1018499380443245,
|
| 895 |
+
"batch_wall_latency_seconds": 3.1217426829971373,
|
| 896 |
+
"amortized_generation_ms_per_sample": 3101.8499380443245,
|
| 897 |
+
"amortized_batch_wall_ms_per_sample": 3121.7426829971373,
|
| 898 |
+
"outputs": [
|
| 899 |
+
{
|
| 900 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [255, 173, 536, 923], \"score\": 0.95}]}",
|
| 901 |
+
"valid_prediction": {
|
| 902 |
+
"objects": [
|
| 903 |
+
{
|
| 904 |
+
"bbox_2d": [
|
| 905 |
+
255,
|
| 906 |
+
173,
|
| 907 |
+
536,
|
| 908 |
+
923
|
| 909 |
+
],
|
| 910 |
+
"score": 0.95
|
| 911 |
+
}
|
| 912 |
+
]
|
| 913 |
+
},
|
| 914 |
+
"error": null,
|
| 915 |
+
"finish_reason": "stop",
|
| 916 |
+
"completion_tokens": 40,
|
| 917 |
+
"prompt_tokens": 562
|
| 918 |
+
}
|
| 919 |
+
]
|
| 920 |
+
},
|
| 921 |
+
{
|
| 922 |
+
"sample_ids": [
|
| 923 |
+
"testA-62940-173454"
|
| 924 |
+
],
|
| 925 |
+
"batch_size": 1,
|
| 926 |
+
"generation_latency_seconds": 3.057670062989928,
|
| 927 |
+
"batch_wall_latency_seconds": 3.073035439942032,
|
| 928 |
+
"amortized_generation_ms_per_sample": 3057.670062989928,
|
| 929 |
+
"amortized_batch_wall_ms_per_sample": 3073.035439942032,
|
| 930 |
+
"outputs": [
|
| 931 |
+
{
|
| 932 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [626, 266, 751, 693], \"score\": 0.9}]}",
|
| 933 |
+
"valid_prediction": {
|
| 934 |
+
"objects": [
|
| 935 |
+
{
|
| 936 |
+
"bbox_2d": [
|
| 937 |
+
626,
|
| 938 |
+
266,
|
| 939 |
+
751,
|
| 940 |
+
693
|
| 941 |
+
],
|
| 942 |
+
"score": 0.9
|
| 943 |
+
}
|
| 944 |
+
]
|
| 945 |
+
},
|
| 946 |
+
"error": null,
|
| 947 |
+
"finish_reason": "stop",
|
| 948 |
+
"completion_tokens": 39,
|
| 949 |
+
"prompt_tokens": 605
|
| 950 |
+
}
|
| 951 |
+
]
|
| 952 |
+
},
|
| 953 |
+
{
|
| 954 |
+
"sample_ids": [
|
| 955 |
+
"testA-81245-223632"
|
| 956 |
+
],
|
| 957 |
+
"batch_size": 1,
|
| 958 |
+
"generation_latency_seconds": 10.252644294989295,
|
| 959 |
+
"batch_wall_latency_seconds": 10.266198730911128,
|
| 960 |
+
"amortized_generation_ms_per_sample": 10252.644294989295,
|
| 961 |
+
"amortized_batch_wall_ms_per_sample": 10266.198730911128,
|
| 962 |
+
"outputs": [
|
| 963 |
+
{
|
| 964 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [584, 117, 783, 882], \"score\": 0.9}, {\"bbox_2d\": [707, 131, 792, 761], \"score\": 0.9}, {\"bbox_2d\": [798, 105, 915, 655], \"score\": 0.9}, {\"bbox_2d\": [533, 178, 655, 705], \"score\": 0.9}]}",
|
| 965 |
+
"valid_prediction": {
|
| 966 |
+
"objects": [
|
| 967 |
+
{
|
| 968 |
+
"bbox_2d": [
|
| 969 |
+
584,
|
| 970 |
+
117,
|
| 971 |
+
783,
|
| 972 |
+
882
|
| 973 |
+
],
|
| 974 |
+
"score": 0.9
|
| 975 |
+
},
|
| 976 |
+
{
|
| 977 |
+
"bbox_2d": [
|
| 978 |
+
707,
|
| 979 |
+
131,
|
| 980 |
+
792,
|
| 981 |
+
761
|
| 982 |
+
],
|
| 983 |
+
"score": 0.9
|
| 984 |
+
},
|
| 985 |
+
{
|
| 986 |
+
"bbox_2d": [
|
| 987 |
+
798,
|
| 988 |
+
105,
|
| 989 |
+
915,
|
| 990 |
+
655
|
| 991 |
+
],
|
| 992 |
+
"score": 0.9
|
| 993 |
+
},
|
| 994 |
+
{
|
| 995 |
+
"bbox_2d": [
|
| 996 |
+
533,
|
| 997 |
+
178,
|
| 998 |
+
655,
|
| 999 |
+
705
|
| 1000 |
+
],
|
| 1001 |
+
"score": 0.9
|
| 1002 |
+
}
|
| 1003 |
+
]
|
| 1004 |
+
},
|
| 1005 |
+
"error": null,
|
| 1006 |
+
"finish_reason": "stop",
|
| 1007 |
+
"completion_tokens": 141,
|
| 1008 |
+
"prompt_tokens": 570
|
| 1009 |
+
}
|
| 1010 |
+
]
|
| 1011 |
+
},
|
| 1012 |
+
{
|
| 1013 |
+
"sample_ids": [
|
| 1014 |
+
"testA-47070-129599"
|
| 1015 |
+
],
|
| 1016 |
+
"batch_size": 1,
|
| 1017 |
+
"generation_latency_seconds": 0.6474383240565658,
|
| 1018 |
+
"batch_wall_latency_seconds": 0.6609540630597621,
|
| 1019 |
+
"amortized_generation_ms_per_sample": 647.4383240565658,
|
| 1020 |
+
"amortized_batch_wall_ms_per_sample": 660.9540630597621,
|
| 1021 |
+
"outputs": [
|
| 1022 |
+
{
|
| 1023 |
+
"raw_response": "{\"objects\": []}",
|
| 1024 |
+
"valid_prediction": {
|
| 1025 |
+
"objects": []
|
| 1026 |
+
},
|
| 1027 |
+
"error": null,
|
| 1028 |
+
"finish_reason": "stop",
|
| 1029 |
+
"completion_tokens": 6,
|
| 1030 |
+
"prompt_tokens": 560
|
| 1031 |
+
}
|
| 1032 |
+
]
|
| 1033 |
+
},
|
| 1034 |
+
{
|
| 1035 |
+
"sample_ids": [
|
| 1036 |
+
"testB-12381-34098"
|
| 1037 |
+
],
|
| 1038 |
+
"batch_size": 1,
|
| 1039 |
+
"generation_latency_seconds": 6.267876331927255,
|
| 1040 |
+
"batch_wall_latency_seconds": 6.293664561933838,
|
| 1041 |
+
"amortized_generation_ms_per_sample": 6267.876331927255,
|
| 1042 |
+
"amortized_batch_wall_ms_per_sample": 6293.664561933838,
|
| 1043 |
+
"outputs": [
|
| 1044 |
+
{
|
| 1045 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[458,242,771,473],\"score\":0.9},{\"bbox_2d\":[298,343,826,817],\"score\":0.9},{\"bbox_2d\":[305,466,793,840],\"score\":0.9}]}",
|
| 1046 |
+
"valid_prediction": {
|
| 1047 |
+
"objects": [
|
| 1048 |
+
{
|
| 1049 |
+
"bbox_2d": [
|
| 1050 |
+
458,
|
| 1051 |
+
242,
|
| 1052 |
+
771,
|
| 1053 |
+
473
|
| 1054 |
+
],
|
| 1055 |
+
"score": 0.9
|
| 1056 |
+
},
|
| 1057 |
+
{
|
| 1058 |
+
"bbox_2d": [
|
| 1059 |
+
298,
|
| 1060 |
+
343,
|
| 1061 |
+
826,
|
| 1062 |
+
817
|
| 1063 |
+
],
|
| 1064 |
+
"score": 0.9
|
| 1065 |
+
},
|
| 1066 |
+
{
|
| 1067 |
+
"bbox_2d": [
|
| 1068 |
+
305,
|
| 1069 |
+
466,
|
| 1070 |
+
793,
|
| 1071 |
+
840
|
| 1072 |
+
],
|
| 1073 |
+
"score": 0.9
|
| 1074 |
+
}
|
| 1075 |
+
]
|
| 1076 |
+
},
|
| 1077 |
+
"error": null,
|
| 1078 |
+
"finish_reason": "stop",
|
| 1079 |
+
"completion_tokens": 86,
|
| 1080 |
+
"prompt_tokens": 574
|
| 1081 |
+
}
|
| 1082 |
+
]
|
| 1083 |
+
},
|
| 1084 |
+
{
|
| 1085 |
+
"sample_ids": [
|
| 1086 |
+
"testA-44142-121498"
|
| 1087 |
+
],
|
| 1088 |
+
"batch_size": 1,
|
| 1089 |
+
"generation_latency_seconds": 5.1425224100239575,
|
| 1090 |
+
"batch_wall_latency_seconds": 5.153767512994818,
|
| 1091 |
+
"amortized_generation_ms_per_sample": 5142.5224100239575,
|
| 1092 |
+
"amortized_batch_wall_ms_per_sample": 5153.767512994818,
|
| 1093 |
+
"outputs": [
|
| 1094 |
+
{
|
| 1095 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [0, 95, 585, 987], \"score\": 0.9}, {\"bbox_2d\": [567, 126, 999, 987], \"score\": 0.9}]}",
|
| 1096 |
+
"valid_prediction": {
|
| 1097 |
+
"objects": [
|
| 1098 |
+
{
|
| 1099 |
+
"bbox_2d": [
|
| 1100 |
+
0,
|
| 1101 |
+
95,
|
| 1102 |
+
585,
|
| 1103 |
+
987
|
| 1104 |
+
],
|
| 1105 |
+
"score": 0.9
|
| 1106 |
+
},
|
| 1107 |
+
{
|
| 1108 |
+
"bbox_2d": [
|
| 1109 |
+
567,
|
| 1110 |
+
126,
|
| 1111 |
+
999,
|
| 1112 |
+
987
|
| 1113 |
+
],
|
| 1114 |
+
"score": 0.9
|
| 1115 |
+
}
|
| 1116 |
+
]
|
| 1117 |
+
},
|
| 1118 |
+
"error": null,
|
| 1119 |
+
"finish_reason": "stop",
|
| 1120 |
+
"completion_tokens": 70,
|
| 1121 |
+
"prompt_tokens": 563
|
| 1122 |
+
}
|
| 1123 |
+
]
|
| 1124 |
+
},
|
| 1125 |
+
{
|
| 1126 |
+
"sample_ids": [
|
| 1127 |
+
"testA-73258-201893"
|
| 1128 |
+
],
|
| 1129 |
+
"batch_size": 1,
|
| 1130 |
+
"generation_latency_seconds": 2.4450040519004688,
|
| 1131 |
+
"batch_wall_latency_seconds": 2.4576197939459234,
|
| 1132 |
+
"amortized_generation_ms_per_sample": 2445.0040519004688,
|
| 1133 |
+
"amortized_batch_wall_ms_per_sample": 2457.6197939459234,
|
| 1134 |
+
"outputs": [
|
| 1135 |
+
{
|
| 1136 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[401,274,999,987],\"score\":0.9}]}",
|
| 1137 |
+
"valid_prediction": {
|
| 1138 |
+
"objects": [
|
| 1139 |
+
{
|
| 1140 |
+
"bbox_2d": [
|
| 1141 |
+
401,
|
| 1142 |
+
274,
|
| 1143 |
+
999,
|
| 1144 |
+
987
|
| 1145 |
+
],
|
| 1146 |
+
"score": 0.9
|
| 1147 |
+
}
|
| 1148 |
+
]
|
| 1149 |
+
},
|
| 1150 |
+
"error": null,
|
| 1151 |
+
"finish_reason": "stop",
|
| 1152 |
+
"completion_tokens": 32,
|
| 1153 |
+
"prompt_tokens": 606
|
| 1154 |
+
}
|
| 1155 |
+
]
|
| 1156 |
+
},
|
| 1157 |
+
{
|
| 1158 |
+
"sample_ids": [
|
| 1159 |
+
"testB-46644-128436"
|
| 1160 |
+
],
|
| 1161 |
+
"batch_size": 1,
|
| 1162 |
+
"generation_latency_seconds": 4.464105351013131,
|
| 1163 |
+
"batch_wall_latency_seconds": 4.473494493984617,
|
| 1164 |
+
"amortized_generation_ms_per_sample": 4464.105351013131,
|
| 1165 |
+
"amortized_batch_wall_ms_per_sample": 4473.494493984617,
|
| 1166 |
+
"outputs": [
|
| 1167 |
+
{
|
| 1168 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[10,183,537,826],\"score\":0.95},{\"bbox_2d\":[483,0,999,743],\"score\":0.95}]}",
|
| 1169 |
+
"valid_prediction": {
|
| 1170 |
+
"objects": [
|
| 1171 |
+
{
|
| 1172 |
+
"bbox_2d": [
|
| 1173 |
+
10,
|
| 1174 |
+
183,
|
| 1175 |
+
537,
|
| 1176 |
+
826
|
| 1177 |
+
],
|
| 1178 |
+
"score": 0.95
|
| 1179 |
+
},
|
| 1180 |
+
{
|
| 1181 |
+
"bbox_2d": [
|
| 1182 |
+
483,
|
| 1183 |
+
0,
|
| 1184 |
+
999,
|
| 1185 |
+
743
|
| 1186 |
+
],
|
| 1187 |
+
"score": 0.95
|
| 1188 |
+
}
|
| 1189 |
+
]
|
| 1190 |
+
},
|
| 1191 |
+
"error": null,
|
| 1192 |
+
"finish_reason": "stop",
|
| 1193 |
+
"completion_tokens": 58,
|
| 1194 |
+
"prompt_tokens": 497
|
| 1195 |
+
}
|
| 1196 |
+
]
|
| 1197 |
+
},
|
| 1198 |
+
{
|
| 1199 |
+
"sample_ids": [
|
| 1200 |
+
"testA-43889-120792"
|
| 1201 |
+
],
|
| 1202 |
+
"batch_size": 1,
|
| 1203 |
+
"generation_latency_seconds": 3.0225026920670643,
|
| 1204 |
+
"batch_wall_latency_seconds": 3.039261986967176,
|
| 1205 |
+
"amortized_generation_ms_per_sample": 3022.5026920670643,
|
| 1206 |
+
"amortized_batch_wall_ms_per_sample": 3039.261986967176,
|
| 1207 |
+
"outputs": [
|
| 1208 |
+
{
|
| 1209 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [305, 303, 475, 882], \"score\": 0.9}]}",
|
| 1210 |
+
"valid_prediction": {
|
| 1211 |
+
"objects": [
|
| 1212 |
+
{
|
| 1213 |
+
"bbox_2d": [
|
| 1214 |
+
305,
|
| 1215 |
+
303,
|
| 1216 |
+
475,
|
| 1217 |
+
882
|
| 1218 |
+
],
|
| 1219 |
+
"score": 0.9
|
| 1220 |
+
}
|
| 1221 |
+
]
|
| 1222 |
+
},
|
| 1223 |
+
"error": null,
|
| 1224 |
+
"finish_reason": "stop",
|
| 1225 |
+
"completion_tokens": 39,
|
| 1226 |
+
"prompt_tokens": 571
|
| 1227 |
+
}
|
| 1228 |
+
]
|
| 1229 |
+
},
|
| 1230 |
+
{
|
| 1231 |
+
"sample_ids": [
|
| 1232 |
+
"testB-36082-99363"
|
| 1233 |
+
],
|
| 1234 |
+
"batch_size": 1,
|
| 1235 |
+
"generation_latency_seconds": 4.3171981680206954,
|
| 1236 |
+
"batch_wall_latency_seconds": 4.324737289105542,
|
| 1237 |
+
"amortized_generation_ms_per_sample": 4317.198168020695,
|
| 1238 |
+
"amortized_batch_wall_ms_per_sample": 4324.737289105542,
|
| 1239 |
+
"outputs": [
|
| 1240 |
+
{
|
| 1241 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[0,107,417,876],\"score\":0.95},{\"bbox_2d\":[442,0,999,986],\"score\":0.95}]}",
|
| 1242 |
+
"valid_prediction": {
|
| 1243 |
+
"objects": [
|
| 1244 |
+
{
|
| 1245 |
+
"bbox_2d": [
|
| 1246 |
+
0,
|
| 1247 |
+
107,
|
| 1248 |
+
417,
|
| 1249 |
+
876
|
| 1250 |
+
],
|
| 1251 |
+
"score": 0.95
|
| 1252 |
+
},
|
| 1253 |
+
{
|
| 1254 |
+
"bbox_2d": [
|
| 1255 |
+
442,
|
| 1256 |
+
0,
|
| 1257 |
+
999,
|
| 1258 |
+
986
|
| 1259 |
+
],
|
| 1260 |
+
"score": 0.95
|
| 1261 |
+
}
|
| 1262 |
+
]
|
| 1263 |
+
},
|
| 1264 |
+
"error": null,
|
| 1265 |
+
"finish_reason": "stop",
|
| 1266 |
+
"completion_tokens": 57,
|
| 1267 |
+
"prompt_tokens": 412
|
| 1268 |
+
}
|
| 1269 |
+
]
|
| 1270 |
+
},
|
| 1271 |
+
{
|
| 1272 |
+
"sample_ids": [
|
| 1273 |
+
"testB-33108-91152"
|
| 1274 |
+
],
|
| 1275 |
+
"batch_size": 1,
|
| 1276 |
+
"generation_latency_seconds": 2.9848218420520425,
|
| 1277 |
+
"batch_wall_latency_seconds": 2.9953485280275345,
|
| 1278 |
+
"amortized_generation_ms_per_sample": 2984.8218420520425,
|
| 1279 |
+
"amortized_batch_wall_ms_per_sample": 2995.3485280275345,
|
| 1280 |
+
"outputs": [
|
| 1281 |
+
{
|
| 1282 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [58, 201, 385, 458], \"score\": 0.95}]}",
|
| 1283 |
+
"valid_prediction": {
|
| 1284 |
+
"objects": [
|
| 1285 |
+
{
|
| 1286 |
+
"bbox_2d": [
|
| 1287 |
+
58,
|
| 1288 |
+
201,
|
| 1289 |
+
385,
|
| 1290 |
+
458
|
| 1291 |
+
],
|
| 1292 |
+
"score": 0.95
|
| 1293 |
+
}
|
| 1294 |
+
]
|
| 1295 |
+
},
|
| 1296 |
+
"error": null,
|
| 1297 |
+
"finish_reason": "stop",
|
| 1298 |
+
"completion_tokens": 39,
|
| 1299 |
+
"prompt_tokens": 602
|
| 1300 |
+
}
|
| 1301 |
+
]
|
| 1302 |
+
},
|
| 1303 |
+
{
|
| 1304 |
+
"sample_ids": [
|
| 1305 |
+
"testA-26856-73830"
|
| 1306 |
+
],
|
| 1307 |
+
"batch_size": 1,
|
| 1308 |
+
"generation_latency_seconds": 3.0266257260227576,
|
| 1309 |
+
"batch_wall_latency_seconds": 3.0415181359276175,
|
| 1310 |
+
"amortized_generation_ms_per_sample": 3026.6257260227576,
|
| 1311 |
+
"amortized_batch_wall_ms_per_sample": 3041.5181359276175,
|
| 1312 |
+
"outputs": [
|
| 1313 |
+
{
|
| 1314 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [576, 343, 783, 713], \"score\": 0.95}]}",
|
| 1315 |
+
"valid_prediction": {
|
| 1316 |
+
"objects": [
|
| 1317 |
+
{
|
| 1318 |
+
"bbox_2d": [
|
| 1319 |
+
576,
|
| 1320 |
+
343,
|
| 1321 |
+
783,
|
| 1322 |
+
713
|
| 1323 |
+
],
|
| 1324 |
+
"score": 0.95
|
| 1325 |
+
}
|
| 1326 |
+
]
|
| 1327 |
+
},
|
| 1328 |
+
"error": null,
|
| 1329 |
+
"finish_reason": "stop",
|
| 1330 |
+
"completion_tokens": 40,
|
| 1331 |
+
"prompt_tokens": 562
|
| 1332 |
+
}
|
| 1333 |
+
]
|
| 1334 |
+
},
|
| 1335 |
+
{
|
| 1336 |
+
"sample_ids": [
|
| 1337 |
+
"testA-81271-223709"
|
| 1338 |
+
],
|
| 1339 |
+
"batch_size": 1,
|
| 1340 |
+
"generation_latency_seconds": 0.6367205450078472,
|
| 1341 |
+
"batch_wall_latency_seconds": 0.6504157679155469,
|
| 1342 |
+
"amortized_generation_ms_per_sample": 636.7205450078472,
|
| 1343 |
+
"amortized_batch_wall_ms_per_sample": 650.4157679155469,
|
| 1344 |
+
"outputs": [
|
| 1345 |
+
{
|
| 1346 |
+
"raw_response": "{\"objects\": []}",
|
| 1347 |
+
"valid_prediction": {
|
| 1348 |
+
"objects": []
|
| 1349 |
+
},
|
| 1350 |
+
"error": null,
|
| 1351 |
+
"finish_reason": "stop",
|
| 1352 |
+
"completion_tokens": 6,
|
| 1353 |
+
"prompt_tokens": 560
|
| 1354 |
+
}
|
| 1355 |
+
]
|
| 1356 |
+
},
|
| 1357 |
+
{
|
| 1358 |
+
"sample_ids": [
|
| 1359 |
+
"testA-54554-150141"
|
| 1360 |
+
],
|
| 1361 |
+
"batch_size": 1,
|
| 1362 |
+
"generation_latency_seconds": 11.380541390972212,
|
| 1363 |
+
"batch_wall_latency_seconds": 11.400562496040948,
|
| 1364 |
+
"amortized_generation_ms_per_sample": 11380.541390972212,
|
| 1365 |
+
"amortized_batch_wall_ms_per_sample": 11400.562496040948,
|
| 1366 |
+
"outputs": [
|
| 1367 |
+
{
|
| 1368 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[27,0,121,243],\"score\":0.9},{\"bbox_2d\":[62,0,185,258],\"score\":0.9},{\"bbox_2d\":[142,0,222,255],\"score\":0.9},{\"bbox_2d\":[185,0,283,261],\"score\":0.9},{\"bbox_2d\":[277,312,597,826],\"score\":0.9},{\"bbox_2d\":[655,500,951,871],\"score\":0.9}]}",
|
| 1369 |
+
"valid_prediction": {
|
| 1370 |
+
"objects": [
|
| 1371 |
+
{
|
| 1372 |
+
"bbox_2d": [
|
| 1373 |
+
27,
|
| 1374 |
+
0,
|
| 1375 |
+
121,
|
| 1376 |
+
243
|
| 1377 |
+
],
|
| 1378 |
+
"score": 0.9
|
| 1379 |
+
},
|
| 1380 |
+
{
|
| 1381 |
+
"bbox_2d": [
|
| 1382 |
+
62,
|
| 1383 |
+
0,
|
| 1384 |
+
185,
|
| 1385 |
+
258
|
| 1386 |
+
],
|
| 1387 |
+
"score": 0.9
|
| 1388 |
+
},
|
| 1389 |
+
{
|
| 1390 |
+
"bbox_2d": [
|
| 1391 |
+
142,
|
| 1392 |
+
0,
|
| 1393 |
+
222,
|
| 1394 |
+
255
|
| 1395 |
+
],
|
| 1396 |
+
"score": 0.9
|
| 1397 |
+
},
|
| 1398 |
+
{
|
| 1399 |
+
"bbox_2d": [
|
| 1400 |
+
185,
|
| 1401 |
+
0,
|
| 1402 |
+
283,
|
| 1403 |
+
261
|
| 1404 |
+
],
|
| 1405 |
+
"score": 0.9
|
| 1406 |
+
},
|
| 1407 |
+
{
|
| 1408 |
+
"bbox_2d": [
|
| 1409 |
+
277,
|
| 1410 |
+
312,
|
| 1411 |
+
597,
|
| 1412 |
+
826
|
| 1413 |
+
],
|
| 1414 |
+
"score": 0.9
|
| 1415 |
+
},
|
| 1416 |
+
{
|
| 1417 |
+
"bbox_2d": [
|
| 1418 |
+
655,
|
| 1419 |
+
500,
|
| 1420 |
+
951,
|
| 1421 |
+
871
|
| 1422 |
+
],
|
| 1423 |
+
"score": 0.9
|
| 1424 |
+
}
|
| 1425 |
+
]
|
| 1426 |
+
},
|
| 1427 |
+
"error": null,
|
| 1428 |
+
"finish_reason": "stop",
|
| 1429 |
+
"completion_tokens": 157,
|
| 1430 |
+
"prompt_tokens": 610
|
| 1431 |
+
}
|
| 1432 |
+
]
|
| 1433 |
+
},
|
| 1434 |
+
{
|
| 1435 |
+
"sample_ids": [
|
| 1436 |
+
"testB-38636-106481"
|
| 1437 |
+
],
|
| 1438 |
+
"batch_size": 1,
|
| 1439 |
+
"generation_latency_seconds": 4.233924768050201,
|
| 1440 |
+
"batch_wall_latency_seconds": 4.251373155973852,
|
| 1441 |
+
"amortized_generation_ms_per_sample": 4233.924768050201,
|
| 1442 |
+
"amortized_batch_wall_ms_per_sample": 4251.373155973852,
|
| 1443 |
+
"outputs": [
|
| 1444 |
+
{
|
| 1445 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[494,177,881,985],\"score\":0.95},{\"bbox_2d\":[0,42,307,722],\"score\":0.95}]}",
|
| 1446 |
+
"valid_prediction": {
|
| 1447 |
+
"objects": [
|
| 1448 |
+
{
|
| 1449 |
+
"bbox_2d": [
|
| 1450 |
+
494,
|
| 1451 |
+
177,
|
| 1452 |
+
881,
|
| 1453 |
+
985
|
| 1454 |
+
],
|
| 1455 |
+
"score": 0.95
|
| 1456 |
+
},
|
| 1457 |
+
{
|
| 1458 |
+
"bbox_2d": [
|
| 1459 |
+
0,
|
| 1460 |
+
42,
|
| 1461 |
+
307,
|
| 1462 |
+
722
|
| 1463 |
+
],
|
| 1464 |
+
"score": 0.95
|
| 1465 |
+
}
|
| 1466 |
+
]
|
| 1467 |
+
},
|
| 1468 |
+
"error": null,
|
| 1469 |
+
"finish_reason": "stop",
|
| 1470 |
+
"completion_tokens": 58,
|
| 1471 |
+
"prompt_tokens": 469
|
| 1472 |
+
}
|
| 1473 |
+
]
|
| 1474 |
+
},
|
| 1475 |
+
{
|
| 1476 |
+
"sample_ids": [
|
| 1477 |
+
"testA-5734-15837"
|
| 1478 |
+
],
|
| 1479 |
+
"batch_size": 1,
|
| 1480 |
+
"generation_latency_seconds": 0.6504808579338714,
|
| 1481 |
+
"batch_wall_latency_seconds": 0.6668235529214144,
|
| 1482 |
+
"amortized_generation_ms_per_sample": 650.4808579338714,
|
| 1483 |
+
"amortized_batch_wall_ms_per_sample": 666.8235529214144,
|
| 1484 |
+
"outputs": [
|
| 1485 |
+
{
|
| 1486 |
+
"raw_response": "{\"objects\": []}",
|
| 1487 |
+
"valid_prediction": {
|
| 1488 |
+
"objects": []
|
| 1489 |
+
},
|
| 1490 |
+
"error": null,
|
| 1491 |
+
"finish_reason": "stop",
|
| 1492 |
+
"completion_tokens": 6,
|
| 1493 |
+
"prompt_tokens": 565
|
| 1494 |
+
}
|
| 1495 |
+
]
|
| 1496 |
+
},
|
| 1497 |
+
{
|
| 1498 |
+
"sample_ids": [
|
| 1499 |
+
"testB-67558-186325"
|
| 1500 |
+
],
|
| 1501 |
+
"batch_size": 1,
|
| 1502 |
+
"generation_latency_seconds": 0.6280788460280746,
|
| 1503 |
+
"batch_wall_latency_seconds": 0.6436845100251958,
|
| 1504 |
+
"amortized_generation_ms_per_sample": 628.0788460280746,
|
| 1505 |
+
"amortized_batch_wall_ms_per_sample": 643.6845100251958,
|
| 1506 |
+
"outputs": [
|
| 1507 |
+
{
|
| 1508 |
+
"raw_response": "{\"objects\": []}",
|
| 1509 |
+
"valid_prediction": {
|
| 1510 |
+
"objects": []
|
| 1511 |
+
},
|
| 1512 |
+
"error": null,
|
| 1513 |
+
"finish_reason": "stop",
|
| 1514 |
+
"completion_tokens": 6,
|
| 1515 |
+
"prompt_tokens": 600
|
| 1516 |
+
}
|
| 1517 |
+
]
|
| 1518 |
+
},
|
| 1519 |
+
{
|
| 1520 |
+
"sample_ids": [
|
| 1521 |
+
"testB-47053-129554"
|
| 1522 |
+
],
|
| 1523 |
+
"batch_size": 1,
|
| 1524 |
+
"generation_latency_seconds": 4.556327463011257,
|
| 1525 |
+
"batch_wall_latency_seconds": 4.573514219024219,
|
| 1526 |
+
"amortized_generation_ms_per_sample": 4556.327463011257,
|
| 1527 |
+
"amortized_batch_wall_ms_per_sample": 4573.514219024219,
|
| 1528 |
+
"outputs": [
|
| 1529 |
+
{
|
| 1530 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[438,321,871,542],\"score\":0.95},{\"bbox_2d\":[438,452,851,863],\"score\":0.95}]}",
|
| 1531 |
+
"valid_prediction": {
|
| 1532 |
+
"objects": [
|
| 1533 |
+
{
|
| 1534 |
+
"bbox_2d": [
|
| 1535 |
+
438,
|
| 1536 |
+
321,
|
| 1537 |
+
871,
|
| 1538 |
+
542
|
| 1539 |
+
],
|
| 1540 |
+
"score": 0.95
|
| 1541 |
+
},
|
| 1542 |
+
{
|
| 1543 |
+
"bbox_2d": [
|
| 1544 |
+
438,
|
| 1545 |
+
452,
|
| 1546 |
+
851,
|
| 1547 |
+
863
|
| 1548 |
+
],
|
| 1549 |
+
"score": 0.95
|
| 1550 |
+
}
|
| 1551 |
+
]
|
| 1552 |
+
},
|
| 1553 |
+
"error": null,
|
| 1554 |
+
"finish_reason": "stop",
|
| 1555 |
+
"completion_tokens": 61,
|
| 1556 |
+
"prompt_tokens": 604
|
| 1557 |
+
}
|
| 1558 |
+
]
|
| 1559 |
+
},
|
| 1560 |
+
{
|
| 1561 |
+
"sample_ids": [
|
| 1562 |
+
"testA-93064-256422"
|
| 1563 |
+
],
|
| 1564 |
+
"batch_size": 1,
|
| 1565 |
+
"generation_latency_seconds": 2.830894450074993,
|
| 1566 |
+
"batch_wall_latency_seconds": 2.8475917810574174,
|
| 1567 |
+
"amortized_generation_ms_per_sample": 2830.894450074993,
|
| 1568 |
+
"amortized_batch_wall_ms_per_sample": 2847.5917810574174,
|
| 1569 |
+
"outputs": [
|
| 1570 |
+
{
|
| 1571 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [0, 226, 173, 755], \"score\": 0.9}]}",
|
| 1572 |
+
"valid_prediction": {
|
| 1573 |
+
"objects": [
|
| 1574 |
+
{
|
| 1575 |
+
"bbox_2d": [
|
| 1576 |
+
0,
|
| 1577 |
+
226,
|
| 1578 |
+
173,
|
| 1579 |
+
755
|
| 1580 |
+
],
|
| 1581 |
+
"score": 0.9
|
| 1582 |
+
}
|
| 1583 |
+
]
|
| 1584 |
+
},
|
| 1585 |
+
"error": null,
|
| 1586 |
+
"finish_reason": "stop",
|
| 1587 |
+
"completion_tokens": 37,
|
| 1588 |
+
"prompt_tokens": 549
|
| 1589 |
+
}
|
| 1590 |
+
]
|
| 1591 |
+
},
|
| 1592 |
+
{
|
| 1593 |
+
"sample_ids": [
|
| 1594 |
+
"testB-60557-166821"
|
| 1595 |
+
],
|
| 1596 |
+
"batch_size": 1,
|
| 1597 |
+
"generation_latency_seconds": 5.58066594589036,
|
| 1598 |
+
"batch_wall_latency_seconds": 5.594979963963851,
|
| 1599 |
+
"amortized_generation_ms_per_sample": 5580.66594589036,
|
| 1600 |
+
"amortized_batch_wall_ms_per_sample": 5594.979963963851,
|
| 1601 |
+
"outputs": [
|
| 1602 |
+
{
|
| 1603 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [857, 442, 875, 477], \"score\": 0.9}, {\"bbox_2d\": [347, 442, 365, 477], \"score\": 0.9}]}",
|
| 1604 |
+
"valid_prediction": {
|
| 1605 |
+
"objects": [
|
| 1606 |
+
{
|
| 1607 |
+
"bbox_2d": [
|
| 1608 |
+
857,
|
| 1609 |
+
442,
|
| 1610 |
+
875,
|
| 1611 |
+
477
|
| 1612 |
+
],
|
| 1613 |
+
"score": 0.9
|
| 1614 |
+
},
|
| 1615 |
+
{
|
| 1616 |
+
"bbox_2d": [
|
| 1617 |
+
347,
|
| 1618 |
+
442,
|
| 1619 |
+
365,
|
| 1620 |
+
477
|
| 1621 |
+
],
|
| 1622 |
+
"score": 0.9
|
| 1623 |
+
}
|
| 1624 |
+
]
|
| 1625 |
+
},
|
| 1626 |
+
"error": null,
|
| 1627 |
+
"finish_reason": "stop",
|
| 1628 |
+
"completion_tokens": 73,
|
| 1629 |
+
"prompt_tokens": 500
|
| 1630 |
+
}
|
| 1631 |
+
]
|
| 1632 |
+
},
|
| 1633 |
+
{
|
| 1634 |
+
"sample_ids": [
|
| 1635 |
+
"testA-55116-151688"
|
| 1636 |
+
],
|
| 1637 |
+
"batch_size": 1,
|
| 1638 |
+
"generation_latency_seconds": 0.6620599570451304,
|
| 1639 |
+
"batch_wall_latency_seconds": 0.6773106920300052,
|
| 1640 |
+
"amortized_generation_ms_per_sample": 662.0599570451304,
|
| 1641 |
+
"amortized_batch_wall_ms_per_sample": 677.3106920300052,
|
| 1642 |
+
"outputs": [
|
| 1643 |
+
{
|
| 1644 |
+
"raw_response": "{\"objects\": []}",
|
| 1645 |
+
"valid_prediction": {
|
| 1646 |
+
"objects": []
|
| 1647 |
+
},
|
| 1648 |
+
"error": null,
|
| 1649 |
+
"finish_reason": "stop",
|
| 1650 |
+
"completion_tokens": 6,
|
| 1651 |
+
"prompt_tokens": 600
|
| 1652 |
+
}
|
| 1653 |
+
]
|
| 1654 |
+
},
|
| 1655 |
+
{
|
| 1656 |
+
"sample_ids": [
|
| 1657 |
+
"testB-65796-181392"
|
| 1658 |
+
],
|
| 1659 |
+
"batch_size": 1,
|
| 1660 |
+
"generation_latency_seconds": 0.6342472250107676,
|
| 1661 |
+
"batch_wall_latency_seconds": 0.6483092680573463,
|
| 1662 |
+
"amortized_generation_ms_per_sample": 634.2472250107676,
|
| 1663 |
+
"amortized_batch_wall_ms_per_sample": 648.3092680573463,
|
| 1664 |
+
"outputs": [
|
| 1665 |
+
{
|
| 1666 |
+
"raw_response": "{\"objects\": []}",
|
| 1667 |
+
"valid_prediction": {
|
| 1668 |
+
"objects": []
|
| 1669 |
+
},
|
| 1670 |
+
"error": null,
|
| 1671 |
+
"finish_reason": "stop",
|
| 1672 |
+
"completion_tokens": 6,
|
| 1673 |
+
"prompt_tokens": 581
|
| 1674 |
+
}
|
| 1675 |
+
]
|
| 1676 |
+
},
|
| 1677 |
+
{
|
| 1678 |
+
"sample_ids": [
|
| 1679 |
+
"testB-23014-63278"
|
| 1680 |
+
],
|
| 1681 |
+
"batch_size": 1,
|
| 1682 |
+
"generation_latency_seconds": 2.7931598050054163,
|
| 1683 |
+
"batch_wall_latency_seconds": 2.8087078949902207,
|
| 1684 |
+
"amortized_generation_ms_per_sample": 2793.1598050054163,
|
| 1685 |
+
"amortized_batch_wall_ms_per_sample": 2808.7078949902207,
|
| 1686 |
+
"outputs": [
|
| 1687 |
+
{
|
| 1688 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[33, 303, 466, 761],\"score\":0.95}]}",
|
| 1689 |
+
"valid_prediction": {
|
| 1690 |
+
"objects": [
|
| 1691 |
+
{
|
| 1692 |
+
"bbox_2d": [
|
| 1693 |
+
33,
|
| 1694 |
+
303,
|
| 1695 |
+
466,
|
| 1696 |
+
761
|
| 1697 |
+
],
|
| 1698 |
+
"score": 0.95
|
| 1699 |
+
}
|
| 1700 |
+
]
|
| 1701 |
+
},
|
| 1702 |
+
"error": null,
|
| 1703 |
+
"finish_reason": "stop",
|
| 1704 |
+
"completion_tokens": 35,
|
| 1705 |
+
"prompt_tokens": 563
|
| 1706 |
+
}
|
| 1707 |
+
]
|
| 1708 |
+
},
|
| 1709 |
+
{
|
| 1710 |
+
"sample_ids": [
|
| 1711 |
+
"testA-24562-67515"
|
| 1712 |
+
],
|
| 1713 |
+
"batch_size": 1,
|
| 1714 |
+
"generation_latency_seconds": 0.6919117919169366,
|
| 1715 |
+
"batch_wall_latency_seconds": 0.7147108289645985,
|
| 1716 |
+
"amortized_generation_ms_per_sample": 691.9117919169366,
|
| 1717 |
+
"amortized_batch_wall_ms_per_sample": 714.7108289645985,
|
| 1718 |
+
"outputs": [
|
| 1719 |
+
{
|
| 1720 |
+
"raw_response": "{\"objects\": []}",
|
| 1721 |
+
"valid_prediction": {
|
| 1722 |
+
"objects": []
|
| 1723 |
+
},
|
| 1724 |
+
"error": null,
|
| 1725 |
+
"finish_reason": "stop",
|
| 1726 |
+
"completion_tokens": 6,
|
| 1727 |
+
"prompt_tokens": 606
|
| 1728 |
+
}
|
| 1729 |
+
]
|
| 1730 |
+
},
|
| 1731 |
+
{
|
| 1732 |
+
"sample_ids": [
|
| 1733 |
+
"testA-17736-48684"
|
| 1734 |
+
],
|
| 1735 |
+
"batch_size": 1,
|
| 1736 |
+
"generation_latency_seconds": 3.0041811569826677,
|
| 1737 |
+
"batch_wall_latency_seconds": 3.0170002209488302,
|
| 1738 |
+
"amortized_generation_ms_per_sample": 3004.1811569826677,
|
| 1739 |
+
"amortized_batch_wall_ms_per_sample": 3017.0002209488302,
|
| 1740 |
+
"outputs": [
|
| 1741 |
+
{
|
| 1742 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [508, 426, 765, 983], \"score\": 0.9}]}",
|
| 1743 |
+
"valid_prediction": {
|
| 1744 |
+
"objects": [
|
| 1745 |
+
{
|
| 1746 |
+
"bbox_2d": [
|
| 1747 |
+
508,
|
| 1748 |
+
426,
|
| 1749 |
+
765,
|
| 1750 |
+
983
|
| 1751 |
+
],
|
| 1752 |
+
"score": 0.9
|
| 1753 |
+
}
|
| 1754 |
+
]
|
| 1755 |
+
},
|
| 1756 |
+
"error": null,
|
| 1757 |
+
"finish_reason": "stop",
|
| 1758 |
+
"completion_tokens": 39,
|
| 1759 |
+
"prompt_tokens": 605
|
| 1760 |
+
}
|
| 1761 |
+
]
|
| 1762 |
+
},
|
| 1763 |
+
{
|
| 1764 |
+
"sample_ids": [
|
| 1765 |
+
"testA-58475-160949"
|
| 1766 |
+
],
|
| 1767 |
+
"batch_size": 1,
|
| 1768 |
+
"generation_latency_seconds": 2.428418811992742,
|
| 1769 |
+
"batch_wall_latency_seconds": 2.442452701041475,
|
| 1770 |
+
"amortized_generation_ms_per_sample": 2428.418811992742,
|
| 1771 |
+
"amortized_batch_wall_ms_per_sample": 2442.452701041475,
|
| 1772 |
+
"outputs": [
|
| 1773 |
+
{
|
| 1774 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[46,36,576,999],\"score\":0.95}]}",
|
| 1775 |
+
"valid_prediction": {
|
| 1776 |
+
"objects": [
|
| 1777 |
+
{
|
| 1778 |
+
"bbox_2d": [
|
| 1779 |
+
46,
|
| 1780 |
+
36,
|
| 1781 |
+
576,
|
| 1782 |
+
999
|
| 1783 |
+
],
|
| 1784 |
+
"score": 0.95
|
| 1785 |
+
}
|
| 1786 |
+
]
|
| 1787 |
+
},
|
| 1788 |
+
"error": null,
|
| 1789 |
+
"finish_reason": "stop",
|
| 1790 |
+
"completion_tokens": 31,
|
| 1791 |
+
"prompt_tokens": 564
|
| 1792 |
+
}
|
| 1793 |
+
]
|
| 1794 |
+
},
|
| 1795 |
+
{
|
| 1796 |
+
"sample_ids": [
|
| 1797 |
+
"testA-80455-221493"
|
| 1798 |
+
],
|
| 1799 |
+
"batch_size": 1,
|
| 1800 |
+
"generation_latency_seconds": 2.4254801200004295,
|
| 1801 |
+
"batch_wall_latency_seconds": 2.4414882709970698,
|
| 1802 |
+
"amortized_generation_ms_per_sample": 2425.4801200004295,
|
| 1803 |
+
"amortized_batch_wall_ms_per_sample": 2441.48827099707,
|
| 1804 |
+
"outputs": [
|
| 1805 |
+
{
|
| 1806 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[0,483,581,938],\"score\":0.95}]}",
|
| 1807 |
+
"valid_prediction": {
|
| 1808 |
+
"objects": [
|
| 1809 |
+
{
|
| 1810 |
+
"bbox_2d": [
|
| 1811 |
+
0,
|
| 1812 |
+
483,
|
| 1813 |
+
581,
|
| 1814 |
+
938
|
| 1815 |
+
],
|
| 1816 |
+
"score": 0.95
|
| 1817 |
+
}
|
| 1818 |
+
]
|
| 1819 |
+
},
|
| 1820 |
+
"error": null,
|
| 1821 |
+
"finish_reason": "stop",
|
| 1822 |
+
"completion_tokens": 31,
|
| 1823 |
+
"prompt_tokens": 541
|
| 1824 |
+
}
|
| 1825 |
+
]
|
| 1826 |
+
},
|
| 1827 |
+
{
|
| 1828 |
+
"sample_ids": [
|
| 1829 |
+
"testA-31763-87405"
|
| 1830 |
+
],
|
| 1831 |
+
"batch_size": 1,
|
| 1832 |
+
"generation_latency_seconds": 7.767380379955284,
|
| 1833 |
+
"batch_wall_latency_seconds": 7.780358829069883,
|
| 1834 |
+
"amortized_generation_ms_per_sample": 7767.380379955284,
|
| 1835 |
+
"amortized_batch_wall_ms_per_sample": 7780.358829069883,
|
| 1836 |
+
"outputs": [
|
| 1837 |
+
{
|
| 1838 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [626, 737, 652, 821], \"score\": 0.9}, {\"bbox_2d\": [426, 913, 452, 972], \"score\": 0.9}, {\"bbox_2d\": [356, 852, 378, 936], \"score\": 0.9}]}",
|
| 1839 |
+
"valid_prediction": {
|
| 1840 |
+
"objects": [
|
| 1841 |
+
{
|
| 1842 |
+
"bbox_2d": [
|
| 1843 |
+
626,
|
| 1844 |
+
737,
|
| 1845 |
+
652,
|
| 1846 |
+
821
|
| 1847 |
+
],
|
| 1848 |
+
"score": 0.9
|
| 1849 |
+
},
|
| 1850 |
+
{
|
| 1851 |
+
"bbox_2d": [
|
| 1852 |
+
426,
|
| 1853 |
+
913,
|
| 1854 |
+
452,
|
| 1855 |
+
972
|
| 1856 |
+
],
|
| 1857 |
+
"score": 0.9
|
| 1858 |
+
},
|
| 1859 |
+
{
|
| 1860 |
+
"bbox_2d": [
|
| 1861 |
+
356,
|
| 1862 |
+
852,
|
| 1863 |
+
378,
|
| 1864 |
+
936
|
| 1865 |
+
],
|
| 1866 |
+
"score": 0.9
|
| 1867 |
+
}
|
| 1868 |
+
]
|
| 1869 |
+
},
|
| 1870 |
+
"error": null,
|
| 1871 |
+
"finish_reason": "stop",
|
| 1872 |
+
"completion_tokens": 107,
|
| 1873 |
+
"prompt_tokens": 445
|
| 1874 |
+
}
|
| 1875 |
+
]
|
| 1876 |
+
},
|
| 1877 |
+
{
|
| 1878 |
+
"sample_ids": [
|
| 1879 |
+
"testB-23920-65742"
|
| 1880 |
+
],
|
| 1881 |
+
"batch_size": 1,
|
| 1882 |
+
"generation_latency_seconds": 2.9294577399268746,
|
| 1883 |
+
"batch_wall_latency_seconds": 2.9447807769756764,
|
| 1884 |
+
"amortized_generation_ms_per_sample": 2929.4577399268746,
|
| 1885 |
+
"amortized_batch_wall_ms_per_sample": 2944.7807769756764,
|
| 1886 |
+
"outputs": [
|
| 1887 |
+
{
|
| 1888 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [537, 417, 999, 993], \"score\": 0.9}]}",
|
| 1889 |
+
"valid_prediction": {
|
| 1890 |
+
"objects": [
|
| 1891 |
+
{
|
| 1892 |
+
"bbox_2d": [
|
| 1893 |
+
537,
|
| 1894 |
+
417,
|
| 1895 |
+
999,
|
| 1896 |
+
993
|
| 1897 |
+
],
|
| 1898 |
+
"score": 0.9
|
| 1899 |
+
}
|
| 1900 |
+
]
|
| 1901 |
+
},
|
| 1902 |
+
"error": null,
|
| 1903 |
+
"finish_reason": "stop",
|
| 1904 |
+
"completion_tokens": 39,
|
| 1905 |
+
"prompt_tokens": 568
|
| 1906 |
+
}
|
| 1907 |
+
]
|
| 1908 |
+
},
|
| 1909 |
+
{
|
| 1910 |
+
"sample_ids": [
|
| 1911 |
+
"testA-469-1309"
|
| 1912 |
+
],
|
| 1913 |
+
"batch_size": 1,
|
| 1914 |
+
"generation_latency_seconds": 0.6524032501038164,
|
| 1915 |
+
"batch_wall_latency_seconds": 0.6627495769644156,
|
| 1916 |
+
"amortized_generation_ms_per_sample": 652.4032501038164,
|
| 1917 |
+
"amortized_batch_wall_ms_per_sample": 662.7495769644156,
|
| 1918 |
+
"outputs": [
|
| 1919 |
+
{
|
| 1920 |
+
"raw_response": "{\"objects\": []}",
|
| 1921 |
+
"valid_prediction": {
|
| 1922 |
+
"objects": []
|
| 1923 |
+
},
|
| 1924 |
+
"error": null,
|
| 1925 |
+
"finish_reason": "stop",
|
| 1926 |
+
"completion_tokens": 6,
|
| 1927 |
+
"prompt_tokens": 621
|
| 1928 |
+
}
|
| 1929 |
+
]
|
| 1930 |
+
},
|
| 1931 |
+
{
|
| 1932 |
+
"sample_ids": [
|
| 1933 |
+
"testB-56691-155998"
|
| 1934 |
+
],
|
| 1935 |
+
"batch_size": 1,
|
| 1936 |
+
"generation_latency_seconds": 0.655415665009059,
|
| 1937 |
+
"batch_wall_latency_seconds": 0.67216857499443,
|
| 1938 |
+
"amortized_generation_ms_per_sample": 655.415665009059,
|
| 1939 |
+
"amortized_batch_wall_ms_per_sample": 672.16857499443,
|
| 1940 |
+
"outputs": [
|
| 1941 |
+
{
|
| 1942 |
+
"raw_response": "{\"objects\": []}",
|
| 1943 |
+
"valid_prediction": {
|
| 1944 |
+
"objects": []
|
| 1945 |
+
},
|
| 1946 |
+
"error": null,
|
| 1947 |
+
"finish_reason": "stop",
|
| 1948 |
+
"completion_tokens": 6,
|
| 1949 |
+
"prompt_tokens": 563
|
| 1950 |
+
}
|
| 1951 |
+
]
|
| 1952 |
+
},
|
| 1953 |
+
{
|
| 1954 |
+
"sample_ids": [
|
| 1955 |
+
"testB-31493-86668"
|
| 1956 |
+
],
|
| 1957 |
+
"batch_size": 1,
|
| 1958 |
+
"generation_latency_seconds": 4.518256991053931,
|
| 1959 |
+
"batch_wall_latency_seconds": 4.5385722850915045,
|
| 1960 |
+
"amortized_generation_ms_per_sample": 4518.256991053931,
|
| 1961 |
+
"amortized_batch_wall_ms_per_sample": 4538.572285091504,
|
| 1962 |
+
"outputs": [
|
| 1963 |
+
{
|
| 1964 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[537,426,747,765],\"score\":0.95},{\"bbox_2d\":[152,426,466,462],\"score\":0.95}]}",
|
| 1965 |
+
"valid_prediction": {
|
| 1966 |
+
"objects": [
|
| 1967 |
+
{
|
| 1968 |
+
"bbox_2d": [
|
| 1969 |
+
537,
|
| 1970 |
+
426,
|
| 1971 |
+
747,
|
| 1972 |
+
765
|
| 1973 |
+
],
|
| 1974 |
+
"score": 0.95
|
| 1975 |
+
},
|
| 1976 |
+
{
|
| 1977 |
+
"bbox_2d": [
|
| 1978 |
+
152,
|
| 1979 |
+
426,
|
| 1980 |
+
466,
|
| 1981 |
+
462
|
| 1982 |
+
],
|
| 1983 |
+
"score": 0.95
|
| 1984 |
+
}
|
| 1985 |
+
]
|
| 1986 |
+
},
|
| 1987 |
+
"error": null,
|
| 1988 |
+
"finish_reason": "stop",
|
| 1989 |
+
"completion_tokens": 61,
|
| 1990 |
+
"prompt_tokens": 614
|
| 1991 |
+
}
|
| 1992 |
+
]
|
| 1993 |
+
},
|
| 1994 |
+
{
|
| 1995 |
+
"sample_ids": [
|
| 1996 |
+
"testA-59284-163254"
|
| 1997 |
+
],
|
| 1998 |
+
"batch_size": 1,
|
| 1999 |
+
"generation_latency_seconds": 5.580998525954783,
|
| 2000 |
+
"batch_wall_latency_seconds": 5.5959404041059315,
|
| 2001 |
+
"amortized_generation_ms_per_sample": 5580.998525954783,
|
| 2002 |
+
"amortized_batch_wall_ms_per_sample": 5595.9404041059315,
|
| 2003 |
+
"outputs": [
|
| 2004 |
+
{
|
| 2005 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [501, 381, 696, 999], \"score\": 0.9}, {\"bbox_2d\": [561, 300, 814, 426], \"score\": 0.9}]}",
|
| 2006 |
+
"valid_prediction": {
|
| 2007 |
+
"objects": [
|
| 2008 |
+
{
|
| 2009 |
+
"bbox_2d": [
|
| 2010 |
+
501,
|
| 2011 |
+
381,
|
| 2012 |
+
696,
|
| 2013 |
+
999
|
| 2014 |
+
],
|
| 2015 |
+
"score": 0.9
|
| 2016 |
+
},
|
| 2017 |
+
{
|
| 2018 |
+
"bbox_2d": [
|
| 2019 |
+
561,
|
| 2020 |
+
300,
|
| 2021 |
+
814,
|
| 2022 |
+
426
|
| 2023 |
+
],
|
| 2024 |
+
"score": 0.9
|
| 2025 |
+
}
|
| 2026 |
+
]
|
| 2027 |
+
},
|
| 2028 |
+
"error": null,
|
| 2029 |
+
"finish_reason": "stop",
|
| 2030 |
+
"completion_tokens": 73,
|
| 2031 |
+
"prompt_tokens": 623
|
| 2032 |
+
}
|
| 2033 |
+
]
|
| 2034 |
+
},
|
| 2035 |
+
{
|
| 2036 |
+
"sample_ids": [
|
| 2037 |
+
"testA-57397-158015"
|
| 2038 |
+
],
|
| 2039 |
+
"batch_size": 1,
|
| 2040 |
+
"generation_latency_seconds": 0.6980972309829667,
|
| 2041 |
+
"batch_wall_latency_seconds": 0.709314426057972,
|
| 2042 |
+
"amortized_generation_ms_per_sample": 698.0972309829667,
|
| 2043 |
+
"amortized_batch_wall_ms_per_sample": 709.314426057972,
|
| 2044 |
+
"outputs": [
|
| 2045 |
+
{
|
| 2046 |
+
"raw_response": "{\"objects\": []}",
|
| 2047 |
+
"valid_prediction": {
|
| 2048 |
+
"objects": []
|
| 2049 |
+
},
|
| 2050 |
+
"error": null,
|
| 2051 |
+
"finish_reason": "stop",
|
| 2052 |
+
"completion_tokens": 6,
|
| 2053 |
+
"prompt_tokens": 602
|
| 2054 |
+
}
|
| 2055 |
+
]
|
| 2056 |
+
},
|
| 2057 |
+
{
|
| 2058 |
+
"sample_ids": [
|
| 2059 |
+
"testB-2935-8111"
|
| 2060 |
+
],
|
| 2061 |
+
"batch_size": 1,
|
| 2062 |
+
"generation_latency_seconds": 4.602234860998578,
|
| 2063 |
+
"batch_wall_latency_seconds": 4.6172516259830445,
|
| 2064 |
+
"amortized_generation_ms_per_sample": 4602.234860998578,
|
| 2065 |
+
"amortized_batch_wall_ms_per_sample": 4617.2516259830445,
|
| 2066 |
+
"outputs": [
|
| 2067 |
+
{
|
| 2068 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[371,91,707,431],\"score\":0.95},{\"bbox_2d\":[585,181,999,705],\"score\":0.95}]}",
|
| 2069 |
+
"valid_prediction": {
|
| 2070 |
+
"objects": [
|
| 2071 |
+
{
|
| 2072 |
+
"bbox_2d": [
|
| 2073 |
+
371,
|
| 2074 |
+
91,
|
| 2075 |
+
707,
|
| 2076 |
+
431
|
| 2077 |
+
],
|
| 2078 |
+
"score": 0.95
|
| 2079 |
+
},
|
| 2080 |
+
{
|
| 2081 |
+
"bbox_2d": [
|
| 2082 |
+
585,
|
| 2083 |
+
181,
|
| 2084 |
+
999,
|
| 2085 |
+
705
|
| 2086 |
+
],
|
| 2087 |
+
"score": 0.95
|
| 2088 |
+
}
|
| 2089 |
+
]
|
| 2090 |
+
},
|
| 2091 |
+
"error": null,
|
| 2092 |
+
"finish_reason": "stop",
|
| 2093 |
+
"completion_tokens": 60,
|
| 2094 |
+
"prompt_tokens": 570
|
| 2095 |
+
}
|
| 2096 |
+
]
|
| 2097 |
+
},
|
| 2098 |
+
{
|
| 2099 |
+
"sample_ids": [
|
| 2100 |
+
"testB-22290-61226"
|
| 2101 |
+
],
|
| 2102 |
+
"batch_size": 1,
|
| 2103 |
+
"generation_latency_seconds": 0.6584999479819089,
|
| 2104 |
+
"batch_wall_latency_seconds": 0.6694054200779647,
|
| 2105 |
+
"amortized_generation_ms_per_sample": 658.4999479819089,
|
| 2106 |
+
"amortized_batch_wall_ms_per_sample": 669.4054200779647,
|
| 2107 |
+
"outputs": [
|
| 2108 |
+
{
|
| 2109 |
+
"raw_response": "{\"objects\": []}",
|
| 2110 |
+
"valid_prediction": {
|
| 2111 |
+
"objects": []
|
| 2112 |
+
},
|
| 2113 |
+
"error": null,
|
| 2114 |
+
"finish_reason": "stop",
|
| 2115 |
+
"completion_tokens": 6,
|
| 2116 |
+
"prompt_tokens": 586
|
| 2117 |
+
}
|
| 2118 |
+
]
|
| 2119 |
+
},
|
| 2120 |
+
{
|
| 2121 |
+
"sample_ids": [
|
| 2122 |
+
"testB-8416-23140"
|
| 2123 |
+
],
|
| 2124 |
+
"batch_size": 1,
|
| 2125 |
+
"generation_latency_seconds": 6.426246937015094,
|
| 2126 |
+
"batch_wall_latency_seconds": 6.434293171972968,
|
| 2127 |
+
"amortized_generation_ms_per_sample": 6426.246937015094,
|
| 2128 |
+
"amortized_batch_wall_ms_per_sample": 6434.293171972968,
|
| 2129 |
+
"outputs": [
|
| 2130 |
+
{
|
| 2131 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[0,292,295,642],\"score\":0.95},{\"bbox_2d\":[234,252,581,727],\"score\":0.95},{\"bbox_2d\":[722,183,959,808],\"score\":0.95}]}",
|
| 2132 |
+
"valid_prediction": {
|
| 2133 |
+
"objects": [
|
| 2134 |
+
{
|
| 2135 |
+
"bbox_2d": [
|
| 2136 |
+
0,
|
| 2137 |
+
292,
|
| 2138 |
+
295,
|
| 2139 |
+
642
|
| 2140 |
+
],
|
| 2141 |
+
"score": 0.95
|
| 2142 |
+
},
|
| 2143 |
+
{
|
| 2144 |
+
"bbox_2d": [
|
| 2145 |
+
234,
|
| 2146 |
+
252,
|
| 2147 |
+
581,
|
| 2148 |
+
727
|
| 2149 |
+
],
|
| 2150 |
+
"score": 0.95
|
| 2151 |
+
},
|
| 2152 |
+
{
|
| 2153 |
+
"bbox_2d": [
|
| 2154 |
+
722,
|
| 2155 |
+
183,
|
| 2156 |
+
959,
|
| 2157 |
+
808
|
| 2158 |
+
],
|
| 2159 |
+
"score": 0.95
|
| 2160 |
+
}
|
| 2161 |
+
]
|
| 2162 |
+
},
|
| 2163 |
+
"error": null,
|
| 2164 |
+
"finish_reason": "stop",
|
| 2165 |
+
"completion_tokens": 87,
|
| 2166 |
+
"prompt_tokens": 563
|
| 2167 |
+
}
|
| 2168 |
+
]
|
| 2169 |
+
},
|
| 2170 |
+
{
|
| 2171 |
+
"sample_ids": [
|
| 2172 |
+
"testA-60731-167320"
|
| 2173 |
+
],
|
| 2174 |
+
"batch_size": 1,
|
| 2175 |
+
"generation_latency_seconds": 5.329908382031135,
|
| 2176 |
+
"batch_wall_latency_seconds": 5.342693028971553,
|
| 2177 |
+
"amortized_generation_ms_per_sample": 5329.908382031135,
|
| 2178 |
+
"amortized_batch_wall_ms_per_sample": 5342.693028971553,
|
| 2179 |
+
"outputs": [
|
| 2180 |
+
{
|
| 2181 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [0, 81, 446, 987], \"score\": 0.9}, {\"bbox_2d\": [483, 0, 792, 201], \"score\": 0.9}]}",
|
| 2182 |
+
"valid_prediction": {
|
| 2183 |
+
"objects": [
|
| 2184 |
+
{
|
| 2185 |
+
"bbox_2d": [
|
| 2186 |
+
0,
|
| 2187 |
+
81,
|
| 2188 |
+
446,
|
| 2189 |
+
987
|
| 2190 |
+
],
|
| 2191 |
+
"score": 0.9
|
| 2192 |
+
},
|
| 2193 |
+
{
|
| 2194 |
+
"bbox_2d": [
|
| 2195 |
+
483,
|
| 2196 |
+
0,
|
| 2197 |
+
792,
|
| 2198 |
+
201
|
| 2199 |
+
],
|
| 2200 |
+
"score": 0.9
|
| 2201 |
+
}
|
| 2202 |
+
]
|
| 2203 |
+
},
|
| 2204 |
+
"error": null,
|
| 2205 |
+
"finish_reason": "stop",
|
| 2206 |
+
"completion_tokens": 68,
|
| 2207 |
+
"prompt_tokens": 606
|
| 2208 |
+
}
|
| 2209 |
+
]
|
| 2210 |
+
},
|
| 2211 |
+
{
|
| 2212 |
+
"sample_ids": [
|
| 2213 |
+
"testA-49055-135086"
|
| 2214 |
+
],
|
| 2215 |
+
"batch_size": 1,
|
| 2216 |
+
"generation_latency_seconds": 4.7276995739666745,
|
| 2217 |
+
"batch_wall_latency_seconds": 4.746967880986631,
|
| 2218 |
+
"amortized_generation_ms_per_sample": 4727.6995739666745,
|
| 2219 |
+
"amortized_batch_wall_ms_per_sample": 4746.967880986631,
|
| 2220 |
+
"outputs": [
|
| 2221 |
+
{
|
| 2222 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[615,242,851,999],\"score\":0.95},{\"bbox_2d\":[137,142,326,999],\"score\":0.95}]}",
|
| 2223 |
+
"valid_prediction": {
|
| 2224 |
+
"objects": [
|
| 2225 |
+
{
|
| 2226 |
+
"bbox_2d": [
|
| 2227 |
+
615,
|
| 2228 |
+
242,
|
| 2229 |
+
851,
|
| 2230 |
+
999
|
| 2231 |
+
],
|
| 2232 |
+
"score": 0.95
|
| 2233 |
+
},
|
| 2234 |
+
{
|
| 2235 |
+
"bbox_2d": [
|
| 2236 |
+
137,
|
| 2237 |
+
142,
|
| 2238 |
+
326,
|
| 2239 |
+
999
|
| 2240 |
+
],
|
| 2241 |
+
"score": 0.95
|
| 2242 |
+
}
|
| 2243 |
+
]
|
| 2244 |
+
},
|
| 2245 |
+
"error": null,
|
| 2246 |
+
"finish_reason": "stop",
|
| 2247 |
+
"completion_tokens": 61,
|
| 2248 |
+
"prompt_tokens": 566
|
| 2249 |
+
}
|
| 2250 |
+
]
|
| 2251 |
+
},
|
| 2252 |
+
{
|
| 2253 |
+
"sample_ids": [
|
| 2254 |
+
"testA-7857-21644"
|
| 2255 |
+
],
|
| 2256 |
+
"batch_size": 1,
|
| 2257 |
+
"generation_latency_seconds": 2.446160126943141,
|
| 2258 |
+
"batch_wall_latency_seconds": 2.4602624271064997,
|
| 2259 |
+
"amortized_generation_ms_per_sample": 2446.160126943141,
|
| 2260 |
+
"amortized_batch_wall_ms_per_sample": 2460.2624271064997,
|
| 2261 |
+
"outputs": [
|
| 2262 |
+
{
|
| 2263 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[668,36,999,999],\"score\":0.95}]}",
|
| 2264 |
+
"valid_prediction": {
|
| 2265 |
+
"objects": [
|
| 2266 |
+
{
|
| 2267 |
+
"bbox_2d": [
|
| 2268 |
+
668,
|
| 2269 |
+
36,
|
| 2270 |
+
999,
|
| 2271 |
+
999
|
| 2272 |
+
],
|
| 2273 |
+
"score": 0.95
|
| 2274 |
+
}
|
| 2275 |
+
]
|
| 2276 |
+
},
|
| 2277 |
+
"error": null,
|
| 2278 |
+
"finish_reason": "stop",
|
| 2279 |
+
"completion_tokens": 32,
|
| 2280 |
+
"prompt_tokens": 607
|
| 2281 |
+
}
|
| 2282 |
+
]
|
| 2283 |
+
},
|
| 2284 |
+
{
|
| 2285 |
+
"sample_ids": [
|
| 2286 |
+
"testB-79645-219273"
|
| 2287 |
+
],
|
| 2288 |
+
"batch_size": 1,
|
| 2289 |
+
"generation_latency_seconds": 15.058248669025488,
|
| 2290 |
+
"batch_wall_latency_seconds": 15.065027808072045,
|
| 2291 |
+
"amortized_generation_ms_per_sample": 15058.248669025488,
|
| 2292 |
+
"amortized_batch_wall_ms_per_sample": 15065.027808072045,
|
| 2293 |
+
"outputs": [
|
| 2294 |
+
{
|
| 2295 |
+
"raw_response": "{\"objects\": [{\"bbox_2d\": [542, 417, 798, 722], \"score\": 0.9}, {\"bbox_2d\": [295, 421, 552, 715], \"score\": 0.9}, {\"bbox_2d\": [147, 452, 310, 665], \"score\": 0.9}, {\"bbox_2d\": [55, 476, 115, 645], \"score\": 0.9}, {\"bbox_2d\": [0, 448, 62, 677], \"score\": 0.9}, {\"bbox_2d\": [113, 478, 162, 595], \"score\": 0.9}]}",
|
| 2296 |
+
"valid_prediction": {
|
| 2297 |
+
"objects": [
|
| 2298 |
+
{
|
| 2299 |
+
"bbox_2d": [
|
| 2300 |
+
542,
|
| 2301 |
+
417,
|
| 2302 |
+
798,
|
| 2303 |
+
722
|
| 2304 |
+
],
|
| 2305 |
+
"score": 0.9
|
| 2306 |
+
},
|
| 2307 |
+
{
|
| 2308 |
+
"bbox_2d": [
|
| 2309 |
+
295,
|
| 2310 |
+
421,
|
| 2311 |
+
552,
|
| 2312 |
+
715
|
| 2313 |
+
],
|
| 2314 |
+
"score": 0.9
|
| 2315 |
+
},
|
| 2316 |
+
{
|
| 2317 |
+
"bbox_2d": [
|
| 2318 |
+
147,
|
| 2319 |
+
452,
|
| 2320 |
+
310,
|
| 2321 |
+
665
|
| 2322 |
+
],
|
| 2323 |
+
"score": 0.9
|
| 2324 |
+
},
|
| 2325 |
+
{
|
| 2326 |
+
"bbox_2d": [
|
| 2327 |
+
55,
|
| 2328 |
+
476,
|
| 2329 |
+
115,
|
| 2330 |
+
645
|
| 2331 |
+
],
|
| 2332 |
+
"score": 0.9
|
| 2333 |
+
},
|
| 2334 |
+
{
|
| 2335 |
+
"bbox_2d": [
|
| 2336 |
+
0,
|
| 2337 |
+
448,
|
| 2338 |
+
62,
|
| 2339 |
+
677
|
| 2340 |
+
],
|
| 2341 |
+
"score": 0.9
|
| 2342 |
+
},
|
| 2343 |
+
{
|
| 2344 |
+
"bbox_2d": [
|
| 2345 |
+
113,
|
| 2346 |
+
478,
|
| 2347 |
+
162,
|
| 2348 |
+
595
|
| 2349 |
+
],
|
| 2350 |
+
"score": 0.9
|
| 2351 |
+
}
|
| 2352 |
+
]
|
| 2353 |
+
},
|
| 2354 |
+
"error": null,
|
| 2355 |
+
"finish_reason": "stop",
|
| 2356 |
+
"completion_tokens": 205,
|
| 2357 |
+
"prompt_tokens": 601
|
| 2358 |
+
}
|
| 2359 |
+
]
|
| 2360 |
+
},
|
| 2361 |
+
{
|
| 2362 |
+
"sample_ids": [
|
| 2363 |
+
"testB-52921-145707"
|
| 2364 |
+
],
|
| 2365 |
+
"batch_size": 1,
|
| 2366 |
+
"generation_latency_seconds": 0.65654409409035,
|
| 2367 |
+
"batch_wall_latency_seconds": 0.6844046280020848,
|
| 2368 |
+
"amortized_generation_ms_per_sample": 656.54409409035,
|
| 2369 |
+
"amortized_batch_wall_ms_per_sample": 684.4046280020848,
|
| 2370 |
+
"outputs": [
|
| 2371 |
+
{
|
| 2372 |
+
"raw_response": "{\"objects\": []}",
|
| 2373 |
+
"valid_prediction": {
|
| 2374 |
+
"objects": []
|
| 2375 |
+
},
|
| 2376 |
+
"error": null,
|
| 2377 |
+
"finish_reason": "stop",
|
| 2378 |
+
"completion_tokens": 6,
|
| 2379 |
+
"prompt_tokens": 571
|
| 2380 |
+
}
|
| 2381 |
+
]
|
| 2382 |
+
},
|
| 2383 |
+
{
|
| 2384 |
+
"sample_ids": [
|
| 2385 |
+
"testB-69479-191562"
|
| 2386 |
+
],
|
| 2387 |
+
"batch_size": 1,
|
| 2388 |
+
"generation_latency_seconds": 2.294087173999287,
|
| 2389 |
+
"batch_wall_latency_seconds": 2.3162708869203925,
|
| 2390 |
+
"amortized_generation_ms_per_sample": 2294.087173999287,
|
| 2391 |
+
"amortized_batch_wall_ms_per_sample": 2316.2708869203925,
|
| 2392 |
+
"outputs": [
|
| 2393 |
+
{
|
| 2394 |
+
"raw_response": "{\"objects\":[{\"bbox_2d\":[0,31,972,999],\"score\":0.95}]}",
|
| 2395 |
+
"valid_prediction": {
|
| 2396 |
+
"objects": [
|
| 2397 |
+
{
|
| 2398 |
+
"bbox_2d": [
|
| 2399 |
+
0,
|
| 2400 |
+
31,
|
| 2401 |
+
972,
|
| 2402 |
+
999
|
| 2403 |
+
],
|
| 2404 |
+
"score": 0.95
|
| 2405 |
+
}
|
| 2406 |
+
]
|
| 2407 |
+
},
|
| 2408 |
+
"error": null,
|
| 2409 |
+
"finish_reason": "stop",
|
| 2410 |
+
"completion_tokens": 30,
|
| 2411 |
+
"prompt_tokens": 561
|
| 2412 |
+
}
|
| 2413 |
+
]
|
| 2414 |
+
},
|
| 2415 |
+
{
|
| 2416 |
+
"sample_ids": [
|
| 2417 |
+
"testA-69466-191524"
|
| 2418 |
+
],
|
| 2419 |
+
"batch_size": 1,
|
| 2420 |
+
"generation_latency_seconds": 0.6702312079723924,
|
| 2421 |
+
"batch_wall_latency_seconds": 0.683375352062285,
|
| 2422 |
+
"amortized_generation_ms_per_sample": 670.2312079723924,
|
| 2423 |
+
"amortized_batch_wall_ms_per_sample": 683.375352062285,
|
| 2424 |
+
"outputs": [
|
| 2425 |
+
{
|
| 2426 |
+
"raw_response": "{\"objects\": []}",
|
| 2427 |
+
"valid_prediction": {
|
| 2428 |
+
"objects": []
|
| 2429 |
+
},
|
| 2430 |
+
"error": null,
|
| 2431 |
+
"finish_reason": "stop",
|
| 2432 |
+
"completion_tokens": 6,
|
| 2433 |
+
"prompt_tokens": 602
|
| 2434 |
+
}
|
| 2435 |
+
]
|
| 2436 |
+
}
|
| 2437 |
+
],
|
| 2438 |
+
"mean_generation_ms_per_sample": 4056.054782922729,
|
| 2439 |
+
"median_generation_ms_per_sample": 3024.564209044911,
|
| 2440 |
+
"mean_batch_wall_ms_per_sample": 4071.4221567668574,
|
| 2441 |
+
"median_batch_wall_ms_per_sample": 3040.390061447397,
|
| 2442 |
+
"mean_prompt_tokens": 578.6875,
|
| 2443 |
+
"mean_completion_tokens": 53.203125,
|
| 2444 |
+
"latency_scope": "A800 vLLM-0.18.0 native generation latency with the frozen task prompt and structured-output schema; model loading and warmup excluded. Batch wall time includes the real request/image preparation path.",
|
| 2445 |
+
"flops_scope": "This file measures latency only. Counted FLOPs are measured separately by the HF reference probe and are not claimed to be complete vLLM kernel FLOPs."
|
| 2446 |
+
}
|
paper_results/data/main_table_cost_qwen_20260912/components/qwen_latency.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
paper_results/data/main_table_cost_qwen_20260912/components/yolo26x_router.json
ADDED
|
@@ -0,0 +1,238 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"status": "complete",
|
| 3 |
+
"run": "shared_12_yolo26x_Qwen3.8",
|
| 4 |
+
"config": {
|
| 5 |
+
"fraction": 1.0,
|
| 6 |
+
"seed": 42,
|
| 7 |
+
"freeze_encoder": true,
|
| 8 |
+
"fusion": "concat",
|
| 9 |
+
"harm_cost": 1.0,
|
| 10 |
+
"batch_size": 256,
|
| 11 |
+
"weight_decay": 0.01,
|
| 12 |
+
"max_epochs": 50,
|
| 13 |
+
"patience": 8,
|
| 14 |
+
"cpu_threads": 1,
|
| 15 |
+
"student": "8M",
|
| 16 |
+
"representation_dim": 128,
|
| 17 |
+
"lr": 0.003,
|
| 18 |
+
"target": "four_state",
|
| 19 |
+
"strategy": "natural",
|
| 20 |
+
"hidden_percent": 90,
|
| 21 |
+
"expert": "yolo26x",
|
| 22 |
+
"task": "construction",
|
| 23 |
+
"endpoint": "Qwen3.8",
|
| 24 |
+
"branches": [
|
| 25 |
+
"image",
|
| 26 |
+
"hidden",
|
| 27 |
+
"confidence",
|
| 28 |
+
"output"
|
| 29 |
+
],
|
| 30 |
+
"hidden_layer": "feature_blocks__layer_16",
|
| 31 |
+
"id": "shared_12_yolo26x_Qwen3.8",
|
| 32 |
+
"stage": "shared_mandatory_inputs"
|
| 33 |
+
},
|
| 34 |
+
"checkpoint": "/mnt/HithinkOmniSSD/user_workspace/caisihang/research/ICASSP_ModelCollaboration/outputs/router_shared_image_hidden_20260910/runs/shared_12_yolo26x_Qwen3.8/best.pt",
|
| 35 |
+
"score": "logit_rescue-logit_harm",
|
| 36 |
+
"test_rows_recomputed": 3004,
|
| 37 |
+
"selected_decisions_match_main": true,
|
| 38 |
+
"max_score_abs_difference": 0.0,
|
| 39 |
+
"actual_call_fraction": 0.4876830892143808,
|
| 40 |
+
"sample_count": 64,
|
| 41 |
+
"sample_ids": [
|
| 42 |
+
"0002628",
|
| 43 |
+
"0003242",
|
| 44 |
+
"0001568",
|
| 45 |
+
"0001119",
|
| 46 |
+
"0003648",
|
| 47 |
+
"0000187",
|
| 48 |
+
"0001891",
|
| 49 |
+
"0001689",
|
| 50 |
+
"0003372",
|
| 51 |
+
"0002549",
|
| 52 |
+
"0000915",
|
| 53 |
+
"0003633",
|
| 54 |
+
"0004424",
|
| 55 |
+
"0001627",
|
| 56 |
+
"0001562",
|
| 57 |
+
"0002018",
|
| 58 |
+
"0000254",
|
| 59 |
+
"0002719",
|
| 60 |
+
"0002730",
|
| 61 |
+
"0004413",
|
| 62 |
+
"0000891",
|
| 63 |
+
"0000150",
|
| 64 |
+
"0003223",
|
| 65 |
+
"0004731",
|
| 66 |
+
"0001587",
|
| 67 |
+
"0000483",
|
| 68 |
+
"0001116",
|
| 69 |
+
"0003376",
|
| 70 |
+
"0001225",
|
| 71 |
+
"0003082",
|
| 72 |
+
"0004447",
|
| 73 |
+
"0000283",
|
| 74 |
+
"0000563",
|
| 75 |
+
"0001254",
|
| 76 |
+
"0004759",
|
| 77 |
+
"0001868",
|
| 78 |
+
"0001860",
|
| 79 |
+
"0004087",
|
| 80 |
+
"0002962",
|
| 81 |
+
"0001802",
|
| 82 |
+
"0002973",
|
| 83 |
+
"0002648",
|
| 84 |
+
"0004350",
|
| 85 |
+
"0004377",
|
| 86 |
+
"0004809",
|
| 87 |
+
"0003694",
|
| 88 |
+
"0001572",
|
| 89 |
+
"0001550",
|
| 90 |
+
"0000213",
|
| 91 |
+
"0002266",
|
| 92 |
+
"0002445",
|
| 93 |
+
"0004513",
|
| 94 |
+
"0002124",
|
| 95 |
+
"0004391",
|
| 96 |
+
"0003679",
|
| 97 |
+
"0002902",
|
| 98 |
+
"0002508",
|
| 99 |
+
"0004767",
|
| 100 |
+
"0003840",
|
| 101 |
+
"0003498",
|
| 102 |
+
"0001579",
|
| 103 |
+
"0001748",
|
| 104 |
+
"0004706",
|
| 105 |
+
"0004387"
|
| 106 |
+
],
|
| 107 |
+
"batch_size": 1,
|
| 108 |
+
"latency_ms": [
|
| 109 |
+
17.35539303626865,
|
| 110 |
+
13.579263002611697,
|
| 111 |
+
15.319477999582887,
|
| 112 |
+
14.879688038490713,
|
| 113 |
+
24.25052202306688,
|
| 114 |
+
20.667891018092632,
|
| 115 |
+
20.015343907289207,
|
| 116 |
+
19.221558002755046,
|
| 117 |
+
20.937222987413406,
|
| 118 |
+
26.26270893961191,
|
| 119 |
+
17.899635946378112,
|
| 120 |
+
16.57471200451255,
|
| 121 |
+
18.844021018594503,
|
| 122 |
+
21.3212959934026,
|
| 123 |
+
19.545868039131165,
|
| 124 |
+
64.57463896367699,
|
| 125 |
+
20.962884067557752,
|
| 126 |
+
24.004184058867395,
|
| 127 |
+
20.19065897911787,
|
| 128 |
+
18.50342599209398,
|
| 129 |
+
19.576229038648307,
|
| 130 |
+
25.253725005313754,
|
| 131 |
+
21.430211956612766,
|
| 132 |
+
25.169604923576117,
|
| 133 |
+
17.8982320940122,
|
| 134 |
+
21.415626979433,
|
| 135 |
+
22.9903319850564,
|
| 136 |
+
229.59673800505698,
|
| 137 |
+
19.89229000173509,
|
| 138 |
+
18.98771501146257,
|
| 139 |
+
18.157949903979897,
|
| 140 |
+
22.423187969252467,
|
| 141 |
+
19.864327972754836,
|
| 142 |
+
21.502194926142693,
|
| 143 |
+
18.544791964814067,
|
| 144 |
+
50.5554600385949,
|
| 145 |
+
20.046600024215877,
|
| 146 |
+
16.299219918437302,
|
| 147 |
+
18.668118049390614,
|
| 148 |
+
24.017272982746363,
|
| 149 |
+
24.73958197515458,
|
| 150 |
+
18.68593308608979,
|
| 151 |
+
20.250934991054237,
|
| 152 |
+
23.676551994867623,
|
| 153 |
+
23.003229987807572,
|
| 154 |
+
24.6098650386557,
|
| 155 |
+
21.308812079951167,
|
| 156 |
+
17.95686304103583,
|
| 157 |
+
18.23423698078841,
|
| 158 |
+
14.343310962431133,
|
| 159 |
+
25.8766250917688,
|
| 160 |
+
18.20633898023516,
|
| 161 |
+
30.77617404051125,
|
| 162 |
+
19.3530690157786,
|
| 163 |
+
15.836970997042954,
|
| 164 |
+
19.045730936340988,
|
| 165 |
+
16.162591986358166,
|
| 166 |
+
25.458221905864775,
|
| 167 |
+
18.277363968081772,
|
| 168 |
+
52.543785073794425,
|
| 169 |
+
22.9536690749228,
|
| 170 |
+
17.81571004539728,
|
| 171 |
+
18.550690030679107,
|
| 172 |
+
21.854626946151257
|
| 173 |
+
],
|
| 174 |
+
"mean_latency_ms_per_input": 25.261267359383055,
|
| 175 |
+
"head_mean_latency_ms": 0.486134999897331,
|
| 176 |
+
"gflops_per_input": [
|
| 177 |
+
3.183754752,
|
| 178 |
+
3.183754752,
|
| 179 |
+
3.183754752,
|
| 180 |
+
3.183754752,
|
| 181 |
+
3.183754752,
|
| 182 |
+
3.183754752,
|
| 183 |
+
3.183754752,
|
| 184 |
+
3.183754752
|
| 185 |
+
],
|
| 186 |
+
"mean_gflops_per_input": 3.183754752,
|
| 187 |
+
"operators": [
|
| 188 |
+
{
|
| 189 |
+
"aten.convolution": 228802560,
|
| 190 |
+
"aten.bmm": 679257600,
|
| 191 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 192 |
+
"aten.addmm": 2039666688
|
| 193 |
+
},
|
| 194 |
+
{
|
| 195 |
+
"aten.convolution": 228802560,
|
| 196 |
+
"aten.bmm": 679257600,
|
| 197 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 198 |
+
"aten.addmm": 2039666688
|
| 199 |
+
},
|
| 200 |
+
{
|
| 201 |
+
"aten.convolution": 228802560,
|
| 202 |
+
"aten.bmm": 679257600,
|
| 203 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 204 |
+
"aten.addmm": 2039666688
|
| 205 |
+
},
|
| 206 |
+
{
|
| 207 |
+
"aten.convolution": 228802560,
|
| 208 |
+
"aten.bmm": 679257600,
|
| 209 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 210 |
+
"aten.addmm": 2039666688
|
| 211 |
+
},
|
| 212 |
+
{
|
| 213 |
+
"aten.convolution": 228802560,
|
| 214 |
+
"aten.bmm": 679257600,
|
| 215 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 216 |
+
"aten.addmm": 2039666688
|
| 217 |
+
},
|
| 218 |
+
{
|
| 219 |
+
"aten.convolution": 228802560,
|
| 220 |
+
"aten.bmm": 679257600,
|
| 221 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 222 |
+
"aten.addmm": 2039666688
|
| 223 |
+
},
|
| 224 |
+
{
|
| 225 |
+
"aten.convolution": 228802560,
|
| 226 |
+
"aten.bmm": 679257600,
|
| 227 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 228 |
+
"aten.addmm": 2039666688
|
| 229 |
+
},
|
| 230 |
+
{
|
| 231 |
+
"aten.convolution": 228802560,
|
| 232 |
+
"aten.bmm": 679257600,
|
| 233 |
+
"aten._scaled_dot_product_flash_attention": 236027904,
|
| 234 |
+
"aten.addmm": 2039666688
|
| 235 |
+
}
|
| 236 |
+
],
|
| 237 |
+
"scope": "image loading, bilinear resize, tokenize when used, two image slots (including gray placeholder), active distilled encoders, feature assembly and original routing head; cached specialist features; supported FLOPs only"
|
| 238 |
+
}
|
paper_results/data/main_table_cost_qwen_20260912/components/yolo26x_specialist.json
ADDED
|
@@ -0,0 +1,254 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"component": "specialist",
|
| 3 |
+
"expert": "yolo26x",
|
| 4 |
+
"task": "construction",
|
| 5 |
+
"device": "cuda:0",
|
| 6 |
+
"checkpoint": "/mnt/HithinkOmniSSD/user_workspace/caisihang/research/ICASSP_ModelCollaboration/models/construction/yolo26x.pt",
|
| 7 |
+
"sample_count": 64,
|
| 8 |
+
"batch_size": 1,
|
| 9 |
+
"sample_ids": [
|
| 10 |
+
"0000944",
|
| 11 |
+
"0003511",
|
| 12 |
+
"0003365",
|
| 13 |
+
"0001351",
|
| 14 |
+
"0002149",
|
| 15 |
+
"0003368",
|
| 16 |
+
"0000176",
|
| 17 |
+
"0004487",
|
| 18 |
+
"0000013",
|
| 19 |
+
"0003583",
|
| 20 |
+
"0000758",
|
| 21 |
+
"0001291",
|
| 22 |
+
"0001451",
|
| 23 |
+
"0002755",
|
| 24 |
+
"0003088",
|
| 25 |
+
"0004538",
|
| 26 |
+
"0004324",
|
| 27 |
+
"0000516",
|
| 28 |
+
"0003893",
|
| 29 |
+
"0003475",
|
| 30 |
+
"0002137",
|
| 31 |
+
"0003272",
|
| 32 |
+
"0001711",
|
| 33 |
+
"0003262",
|
| 34 |
+
"0001470",
|
| 35 |
+
"0002145",
|
| 36 |
+
"0002662",
|
| 37 |
+
"0000756",
|
| 38 |
+
"0001631",
|
| 39 |
+
"0001924",
|
| 40 |
+
"0002496",
|
| 41 |
+
"0001443",
|
| 42 |
+
"0004053",
|
| 43 |
+
"0002168",
|
| 44 |
+
"0001881",
|
| 45 |
+
"0001666",
|
| 46 |
+
"0000304",
|
| 47 |
+
"0001252",
|
| 48 |
+
"0004844",
|
| 49 |
+
"0000949",
|
| 50 |
+
"0003430",
|
| 51 |
+
"0003807",
|
| 52 |
+
"0004199",
|
| 53 |
+
"0002538",
|
| 54 |
+
"0002091",
|
| 55 |
+
"0001569",
|
| 56 |
+
"0003048",
|
| 57 |
+
"0002854",
|
| 58 |
+
"0000380",
|
| 59 |
+
"0001234",
|
| 60 |
+
"0001484",
|
| 61 |
+
"0003180",
|
| 62 |
+
"0000824",
|
| 63 |
+
"0003784",
|
| 64 |
+
"0000894",
|
| 65 |
+
"0004168",
|
| 66 |
+
"0000564",
|
| 67 |
+
"0004296",
|
| 68 |
+
"0003054",
|
| 69 |
+
"0001583",
|
| 70 |
+
"0003386",
|
| 71 |
+
"0000205",
|
| 72 |
+
"0004214",
|
| 73 |
+
"0003625"
|
| 74 |
+
],
|
| 75 |
+
"model_load_seconds_excluded_from_latency": true,
|
| 76 |
+
"forward_latency_ms_per_sample": [
|
| 77 |
+
41.741790771484375,
|
| 78 |
+
51.50377655029297,
|
| 79 |
+
78.82457733154297,
|
| 80 |
+
87.29910278320312,
|
| 81 |
+
49.47385787963867,
|
| 82 |
+
49.40816116333008,
|
| 83 |
+
168.38259887695312,
|
| 84 |
+
42.57542419433594,
|
| 85 |
+
40.342529296875,
|
| 86 |
+
54.95475387573242,
|
| 87 |
+
48.86323165893555,
|
| 88 |
+
40.684993743896484,
|
| 89 |
+
282.5484619140625,
|
| 90 |
+
47.938270568847656,
|
| 91 |
+
45.019649505615234,
|
| 92 |
+
268.04327392578125,
|
| 93 |
+
51.39228820800781,
|
| 94 |
+
41.36457443237305,
|
| 95 |
+
54.01919937133789,
|
| 96 |
+
43.2599983215332,
|
| 97 |
+
47.13923263549805,
|
| 98 |
+
41.1484489440918,
|
| 99 |
+
35.99542236328125,
|
| 100 |
+
80.46995544433594,
|
| 101 |
+
83.43251037597656,
|
| 102 |
+
48.88550567626953,
|
| 103 |
+
171.4989776611328,
|
| 104 |
+
58.917728424072266,
|
| 105 |
+
46.90486526489258,
|
| 106 |
+
56.416385650634766,
|
| 107 |
+
44.49577713012695,
|
| 108 |
+
49.42009735107422,
|
| 109 |
+
43.6354866027832,
|
| 110 |
+
55.22761535644531,
|
| 111 |
+
48.02000045776367,
|
| 112 |
+
39.44412612915039,
|
| 113 |
+
30.26905632019043,
|
| 114 |
+
492.74456787109375,
|
| 115 |
+
45.74156951904297,
|
| 116 |
+
39.223392486572266,
|
| 117 |
+
50.18822479248047,
|
| 118 |
+
43.890113830566406,
|
| 119 |
+
51.966495513916016,
|
| 120 |
+
47.814815521240234,
|
| 121 |
+
54.13961410522461,
|
| 122 |
+
39.215328216552734,
|
| 123 |
+
56.004703521728516,
|
| 124 |
+
39.528831481933594,
|
| 125 |
+
396.5307312011719,
|
| 126 |
+
83.4832992553711,
|
| 127 |
+
50.0503044128418,
|
| 128 |
+
153.1714630126953,
|
| 129 |
+
61.07219314575195,
|
| 130 |
+
52.57244873046875,
|
| 131 |
+
42.75983810424805,
|
| 132 |
+
38.4617919921875,
|
| 133 |
+
44.08700942993164,
|
| 134 |
+
53.36006546020508,
|
| 135 |
+
45.519649505615234,
|
| 136 |
+
290.8823547363281,
|
| 137 |
+
48.10774230957031,
|
| 138 |
+
53.34476852416992,
|
| 139 |
+
51.016319274902344,
|
| 140 |
+
42.9719352722168
|
| 141 |
+
],
|
| 142 |
+
"end_to_end_latency_ms_per_sample": [
|
| 143 |
+
35.17372906208038,
|
| 144 |
+
48.07086195796728,
|
| 145 |
+
46.07532289810479,
|
| 146 |
+
45.34715902991593,
|
| 147 |
+
46.541396994143724,
|
| 148 |
+
45.594630064442754,
|
| 149 |
+
45.38595397025347,
|
| 150 |
+
39.57494895439595,
|
| 151 |
+
39.812199072912335,
|
| 152 |
+
51.36631103232503,
|
| 153 |
+
42.81833197455853,
|
| 154 |
+
33.49219704978168,
|
| 155 |
+
37.3103239107877,
|
| 156 |
+
38.96563302259892,
|
| 157 |
+
40.327758993953466,
|
| 158 |
+
44.32409000582993,
|
| 159 |
+
43.25311293359846,
|
| 160 |
+
33.99710892699659,
|
| 161 |
+
49.12354901898652,
|
| 162 |
+
45.043769059702754,
|
| 163 |
+
47.554407035931945,
|
| 164 |
+
37.903546006418765,
|
| 165 |
+
32.56980108562857,
|
| 166 |
+
35.192839917726815,
|
| 167 |
+
39.34567596297711,
|
| 168 |
+
46.67255701497197,
|
| 169 |
+
49.70169102307409,
|
| 170 |
+
52.48237494379282,
|
| 171 |
+
43.415413005277514,
|
| 172 |
+
48.36420400533825,
|
| 173 |
+
48.96194895263761,
|
| 174 |
+
51.6092429170385,
|
| 175 |
+
37.311508087441325,
|
| 176 |
+
50.858690054155886,
|
| 177 |
+
43.94579504150897,
|
| 178 |
+
36.09434806276113,
|
| 179 |
+
26.66287892498076,
|
| 180 |
+
41.9749760767445,
|
| 181 |
+
34.900374012067914,
|
| 182 |
+
39.04577391222119,
|
| 183 |
+
51.60050210542977,
|
| 184 |
+
46.00996105000377,
|
| 185 |
+
53.40714694466442,
|
| 186 |
+
51.78673507180065,
|
| 187 |
+
54.96632098220289,
|
| 188 |
+
46.47863702848554,
|
| 189 |
+
47.48446703888476,
|
| 190 |
+
36.593522992916405,
|
| 191 |
+
37.32479899190366,
|
| 192 |
+
26.43477905075997,
|
| 193 |
+
39.97986298054457,
|
| 194 |
+
55.87899999227375,
|
| 195 |
+
40.821335976943374,
|
| 196 |
+
45.88699492160231,
|
| 197 |
+
40.707007981836796,
|
| 198 |
+
34.627774031832814,
|
| 199 |
+
43.4188200160861,
|
| 200 |
+
49.56604307517409,
|
| 201 |
+
39.82124803587794,
|
| 202 |
+
249.81214595027268,
|
| 203 |
+
38.2277499884367,
|
| 204 |
+
48.29735495150089,
|
| 205 |
+
47.6501100929454,
|
| 206 |
+
40.76927900314331
|
| 207 |
+
],
|
| 208 |
+
"mean_forward_latency_ms_per_sample": 78.70017620921135,
|
| 209 |
+
"median_forward_latency_ms_per_sample": 49.41412925720215,
|
| 210 |
+
"mean_end_to_end_latency_ms_per_sample": 46.30809425361804,
|
| 211 |
+
"median_end_to_end_latency_ms_per_sample": 43.68230752879754,
|
| 212 |
+
"flop_profiles": [
|
| 213 |
+
{
|
| 214 |
+
"batch_size": 1,
|
| 215 |
+
"flops_counted": 116518072320,
|
| 216 |
+
"gflops_counted_per_sample": 116.51807232,
|
| 217 |
+
"operators": {
|
| 218 |
+
"aten.convolution": 116319006720,
|
| 219 |
+
"aten.bmm": 199065600
|
| 220 |
+
}
|
| 221 |
+
},
|
| 222 |
+
{
|
| 223 |
+
"batch_size": 1,
|
| 224 |
+
"flops_counted": 145709798400,
|
| 225 |
+
"gflops_counted_per_sample": 145.7097984,
|
| 226 |
+
"operators": {
|
| 227 |
+
"aten.convolution": 145398758400,
|
| 228 |
+
"aten.bmm": 311040000
|
| 229 |
+
}
|
| 230 |
+
},
|
| 231 |
+
{
|
| 232 |
+
"batch_size": 1,
|
| 233 |
+
"flops_counted": 145709798400,
|
| 234 |
+
"gflops_counted_per_sample": 145.7097984,
|
| 235 |
+
"operators": {
|
| 236 |
+
"aten.convolution": 145398758400,
|
| 237 |
+
"aten.bmm": 311040000
|
| 238 |
+
}
|
| 239 |
+
},
|
| 240 |
+
{
|
| 241 |
+
"batch_size": 1,
|
| 242 |
+
"flops_counted": 135976458240,
|
| 243 |
+
"gflops_counted_per_sample": 135.97645824,
|
| 244 |
+
"operators": {
|
| 245 |
+
"aten.convolution": 135705507840,
|
| 246 |
+
"aten.bmm": 270950400
|
| 247 |
+
}
|
| 248 |
+
}
|
| 249 |
+
],
|
| 250 |
+
"mean_counted_gflops_per_sample": 135.97853184000002,
|
| 251 |
+
"flops_scope": "GPU FlopCounterMode around the real Ultralytics predict path; multiply-add counted as 2 FLOPs. Elementwise, postprocessing and unsupported custom ops may be missing.",
|
| 252 |
+
"latency_scope": "A800 wall-clock latency for real Ultralytics prediction including image loading/preprocess, forward and result postprocessing, excluding model load; batch processing uses the recorded batch size.",
|
| 253 |
+
"completed": true
|
| 254 |
+
}
|
paper_results/data/main_table_cost_qwen_20260912/cost_summary.json
ADDED
|
@@ -0,0 +1,75 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"components": {
|
| 3 |
+
"grefcoco|instancevg|Qwen3.8": {
|
| 4 |
+
"specialist_gflops": 7.502680064,
|
| 5 |
+
"router_gflops": 3.642171392,
|
| 6 |
+
"large_gflops": 32554.061985335,
|
| 7 |
+
"specialist_ms": 46.08042127620138,
|
| 8 |
+
"router_ms": 17.032772149832454,
|
| 9 |
+
"large_ms": 4071.4221567668574,
|
| 10 |
+
"large_samples": 64,
|
| 11 |
+
"flops_probes": 8,
|
| 12 |
+
"specialist_samples": 64,
|
| 13 |
+
"router_samples": 64,
|
| 14 |
+
"batch_size": 1
|
| 15 |
+
},
|
| 16 |
+
"construction|yolo26x|Qwen3.8": {
|
| 17 |
+
"specialist_gflops": 135.97853184000002,
|
| 18 |
+
"router_gflops": 3.183754752,
|
| 19 |
+
"large_gflops": 96874.712551641,
|
| 20 |
+
"specialist_ms": 46.30809425361804,
|
| 21 |
+
"router_ms": 25.261267359383055,
|
| 22 |
+
"large_ms": 9822.475872695577,
|
| 23 |
+
"large_samples": 64,
|
| 24 |
+
"flops_probes": 8,
|
| 25 |
+
"specialist_samples": 64,
|
| 26 |
+
"router_samples": 64,
|
| 27 |
+
"batch_size": 1
|
| 28 |
+
}
|
| 29 |
+
},
|
| 30 |
+
"selected_points": [
|
| 31 |
+
{
|
| 32 |
+
"pair": "grefcoco|instancevg|Qwen3.8",
|
| 33 |
+
"method": "learned",
|
| 34 |
+
"nominal_validation_budget": 0.56,
|
| 35 |
+
"actual_test_call_fraction": 0.5355755324277571,
|
| 36 |
+
"test_metric": 0.7528287440093016,
|
| 37 |
+
"estimated_gflops_per_input": 17446.303931938,
|
| 38 |
+
"estimated_time_ms_per_input": 2243.6672827746106
|
| 39 |
+
},
|
| 40 |
+
{
|
| 41 |
+
"pair": "grefcoco|instancevg|Qwen3.8",
|
| 42 |
+
"method": "confidence",
|
| 43 |
+
"nominal_validation_budget": 0.71,
|
| 44 |
+
"actual_test_call_fraction": 0.6838612710206166,
|
| 45 |
+
"test_metric": 0.7243286163967898,
|
| 46 |
+
"estimated_gflops_per_input": 22269.96488623913,
|
| 47 |
+
"estimated_time_ms_per_input": 2830.3683522642846
|
| 48 |
+
},
|
| 49 |
+
{
|
| 50 |
+
"pair": "construction|yolo26x|Qwen3.8",
|
| 51 |
+
"method": "learned",
|
| 52 |
+
"nominal_validation_budget": 0.5,
|
| 53 |
+
"actual_test_call_fraction": 0.4876830892143808,
|
| 54 |
+
"test_metric": 0.29875800068449215,
|
| 55 |
+
"estimated_gflops_per_input": 47383.321370531434,
|
| 56 |
+
"estimated_time_ms_per_input": 4861.824738942901
|
| 57 |
+
},
|
| 58 |
+
{
|
| 59 |
+
"pair": "construction|yolo26x|Qwen3.8",
|
| 60 |
+
"method": "confidence",
|
| 61 |
+
"nominal_validation_budget": 0.93,
|
| 62 |
+
"actual_test_call_fraction": 0.9221038615179761,
|
| 63 |
+
"test_metric": 0.2597490004762105,
|
| 64 |
+
"estimated_gflops_per_input": 89464.5250591521,
|
| 65 |
+
"estimated_time_ms_per_input": 9103.651026133362
|
| 66 |
+
}
|
| 67 |
+
],
|
| 68 |
+
"protocol": {
|
| 69 |
+
"method": "original main-table logit contrast; original checkpoints and validation thresholds; seed 42",
|
| 70 |
+
"large_timing": "64 measured examples; batch size one; same full task prompt; two separate warmup examples",
|
| 71 |
+
"cost": "serial component estimates, not a timed integrated service; includes active distilled encoders and routing head only for learned",
|
| 72 |
+
"flops": "multiply-add=2; supported operators; mean prefill (including visual embedding) plus average generated length times one-token decode probe",
|
| 73 |
+
"scope_limits": "mean call cost extrapolated across policies; no policy-specific input/output-length costs; specialist-feature extraction overhead is not separately measured; confidence threshold overhead not separately timed"
|
| 74 |
+
}
|
| 75 |
+
}
|
paper_results/data/main_table_cost_qwen_20260912/fixed_budget_cost_performance.csv
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
pair,method,nominal_validation_budget,actual_test_call_fraction,test_metric,estimated_gflops_per_input,estimated_time_ms_per_input
|
| 2 |
+
grefcoco|instancevg|Qwen3.8,learned,0.0,0.0,0.6695119530385957,11.144851456,63.11319342603383
|
| 3 |
+
grefcoco|instancevg|Qwen3.8,learned,0.05,0.11998411933187761,0.7164449990074583,3917.1153094417755,551.619195333999
|
| 4 |
+
grefcoco|instancevg|Qwen3.8,learned,0.1,0.16822164875365114,0.7290077418257097,5487.442832257611,748.0145414095009
|
| 5 |
+
grefcoco|instancevg|Qwen3.8,learned,0.25,0.30176105266142983,0.748092901908516,9834.692864556126,1291.7098292810697
|
| 6 |
+
grefcoco|instancevg|Qwen3.8,learned,0.5,0.487451436349715,0.7535093440716899,15879.6691252252,2047.733771728093
|
| 7 |
+
grefcoco|instancevg|Qwen3.8,learned,0.75,0.6870657629810283,0.7512973938689278,22377.92628754188,2860.4479639829183
|
| 8 |
+
grefcoco|instancevg|Qwen3.8,learned,1.0,1.0,0.7447182599325072,32565.206836791,4134.535350192891
|
| 9 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.0,0.0,0.6695119530385957,7.502680064,46.08042127620138
|
| 10 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.05,0.012534384482318578,0.6666194027734452,415.54780944942013,97.11319197894791
|
| 11 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.1,0.0345971698380739,0.6647761109378102,1133.7810914898203,186.94010511636148
|
| 12 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.25,0.12477667810452883,0.6594731021183677,4069.490393403024,554.0989531587461
|
| 13 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.5,0.2345234381646485,0.662762669086578,7642.1932230898465,1000.9243437008932
|
| 14 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.75,0.7478093185491875,0.7311629753566061,24351.733589325377,3090.727849854088
|
| 15 |
+
grefcoco|instancevg|Qwen3.8,confidence,1.0,1.0,0.7447182599325072,32561.564665399,4117.502578043059
|
| 16 |
+
construction|yolo26x|Qwen3.8,learned,0.0,0.0,0.18129624718318377,139.16228659200002,71.56936161300109
|
| 17 |
+
construction|yolo26x|Qwen3.8,learned,0.05,0.045938748335552594,0.19531128485766117,4589.465326580834,522.8016087608005
|
| 18 |
+
construction|yolo26x|Qwen3.8,learned,0.1,0.09920106524633822,0.20211074868345033,9749.236967147599,1045.9694315408578
|
| 19 |
+
construction|yolo26x|Qwen3.8,learned,0.25,0.25432756324900135,0.2504846177578749,24777.071870298303,2569.695715387775
|
| 20 |
+
construction|yolo26x|Qwen3.8,learned,0.5,0.4876830892143808,0.29875800068449215,47383.321370531434,4861.824738942901
|
| 21 |
+
construction|yolo26x|Qwen3.8,learned,0.75,0.7709720372836218,0.28730364578560735,74826.85678379593,7644.4235963543315
|
| 22 |
+
construction|yolo26x|Qwen3.8,learned,1.0,1.0,0.27001009067886245,97013.87483823301,9894.045234308578
|
| 23 |
+
construction|yolo26x|Qwen3.8,confidence,0.0,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 24 |
+
construction|yolo26x|Qwen3.8,confidence,0.05,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 25 |
+
construction|yolo26x|Qwen3.8,confidence,0.1,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 26 |
+
construction|yolo26x|Qwen3.8,confidence,0.25,0.24800266311584554,0.18026802885144627,24161.165233228996,2482.3082690732604
|
| 27 |
+
construction|yolo26x|Qwen3.8,confidence,0.5,0.5013315579227696,0.18015785955296285,48702.32909867467,4970.625226170908
|
| 28 |
+
construction|yolo26x|Qwen3.8,confidence,0.75,0.7486684420772304,0.22103445955201576,72663.01865455658,7400.0858032057995
|
| 29 |
+
construction|yolo26x|Qwen3.8,confidence,1.0,1.0,0.27001009067886245,97010.691083481,9868.783966949195
|
paper_results/data/main_table_cost_qwen_20260912/full_cost_performance_curves.csv
ADDED
|
@@ -0,0 +1,405 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
pair,method,nominal_validation_budget,actual_test_call_fraction,test_metric,estimated_gflops_per_input,estimated_time_ms_per_input
|
| 2 |
+
grefcoco|instancevg|Qwen3.8,learned,0.0,0.0,0.6695119530385957,11.144851456,63.11319342603383
|
| 3 |
+
grefcoco|instancevg|Qwen3.8,learned,0.01,0.04701812097666109,0.6878314380512152,1541.775676164205,254.54381293995635
|
| 4 |
+
grefcoco|instancevg|Qwen3.8,learned,0.02,0.06706746448118424,0.6980404389870403,2194.4632473757256,336.17315431290154
|
| 5 |
+
grefcoco|instancevg|Qwen3.8,learned,0.03,0.08720188299350594,0.7065195814309616,2849.9203556645225,418.1488719575849
|
| 6 |
+
grefcoco|instancevg|Qwen3.8,learned,0.04,0.1055780846779911,0.712389756969061,3448.14036435627,492.9661466530142
|
| 7 |
+
grefcoco|instancevg|Qwen3.8,learned,0.05,0.11998411933187761,0.7164449990074583,3917.1153094417755,551.619195333999
|
| 8 |
+
grefcoco|instancevg|Qwen3.8,learned,0.06,0.13243342880639764,0.7199047159912656,4322.390901749918,602.3055897650073
|
| 9 |
+
grefcoco|instancevg|Qwen3.8,learned,0.07,0.14235884638289426,0.7222017412018262,4645.503560765523,642.7161548011189
|
| 10 |
+
grefcoco|instancevg|Qwen3.8,learned,0.08,0.1511215721861441,0.7239032413577972,4930.765879725012,678.392910790143
|
| 11 |
+
grefcoco|instancevg|Qwen3.8,learned,0.09,0.15957235629413266,0.7263136999120892,5205.873229401156,712.7996204494608
|
| 12 |
+
grefcoco|instancevg|Qwen3.8,learned,0.1,0.16822164875365114,0.7290077418257097,5487.442832257611,748.0145414095009
|
| 13 |
+
grefcoco|instancevg|Qwen3.8,learned,0.11,0.17695601622096815,0.7316450670674645,5771.781972191343,783.575838641279
|
| 14 |
+
grefcoco|instancevg|Qwen3.8,learned,0.12,0.18855457561750277,0.7345943340044806,6149.362193726721,830.7984703549065
|
| 15 |
+
grefcoco|instancevg|Qwen3.8,learned,0.13,0.19751580977228256,0.7366361341916456,6441.086765866523,867.2834376446528
|
| 16 |
+
grefcoco|instancevg|Qwen3.8,learned,0.14,0.20664719394265946,0.7385361426991465,6738.350412160879,904.4611574778754
|
| 17 |
+
grefcoco|instancevg|Qwen3.8,learned,0.15,0.2172532115815444,0.7402376428551173,7083.619367794696,947.6427326878917
|
| 18 |
+
grefcoco|instancevg|Qwen3.8,learned,0.16,0.22717862915804102,0.7420242180188866,7406.7320268103,988.0532977240033
|
| 19 |
+
grefcoco|instancevg|Qwen3.8,learned,0.17,0.2361398633128208,0.7436122848311261,7698.4565989501025,1024.5382650137497
|
| 20 |
+
grefcoco|instancevg|Qwen3.8,learned,0.18,0.24362646399909255,0.7448033349403057,7942.175861750445,1055.0193769267023
|
| 21 |
+
grefcoco|instancevg|Qwen3.8,learned,0.19,0.2507727646541701,0.7456540850182911,8174.816976241679,1084.1149837527025
|
| 22 |
+
grefcoco|instancevg|Qwen3.8,learned,0.2,0.25942205711368854,0.7459943850494853,8456.386579098133,1119.3299047127425
|
| 23 |
+
grefcoco|instancevg|Qwen3.8,learned,0.21,0.2690922496667896,0.746419760088478,8771.19062688191,1158.7013409336398
|
| 24 |
+
grefcoco|instancevg|Qwen3.8,learned,0.22,0.2781385588293679,0.7467600601196722,9065.684736098987,1195.5326844951244
|
| 25 |
+
grefcoco|instancevg|Qwen3.8,learned,0.23,0.2868445679607521,0.7473555851742619,9349.100697006961,1230.9785229696565
|
| 26 |
+
grefcoco|instancevg|Qwen3.8,learned,0.24,0.29430281031109096,0.7480645435725831,9591.896780781542,1261.344176125363
|
| 27 |
+
grefcoco|instancevg|Qwen3.8,learned,0.25,0.30176105266142983,0.748092901908516,9834.692864556126,1291.7098292810697
|
| 28 |
+
grefcoco|instancevg|Qwen3.8,learned,0.26,0.30995661174602274,0.7486033519553073,10101.491603000439,1325.0774101251732
|
| 29 |
+
grefcoco|instancevg|Qwen3.8,learned,0.27,0.3175850041119587,0.7494541020332927,10349.826760929574,1356.135815824356
|
| 30 |
+
grefcoco|instancevg|Qwen3.8,learned,0.28,0.3262342965714772,0.7501630604316138,10631.396363786029,1391.350736784396
|
| 31 |
+
grefcoco|instancevg|Qwen3.8,learned,0.29,0.3324731304767036,0.7505600771346738,10834.49574945298,1416.7516633785233
|
| 32 |
+
grefcoco|instancevg|Qwen3.8,learned,0.3,0.3399313728270425,0.7509570938377336,11077.291833227562,1447.11731653423
|
| 33 |
+
grefcoco|instancevg|Qwen3.8,learned,0.31,0.34824036525536683,0.7511839605251964,11347.783287774913,1480.9467324073175
|
| 34 |
+
grefcoco|instancevg|Qwen3.8,learned,0.32,0.3557553242775714,0.751269035532995,11592.425729601013,1511.5433030775164
|
| 35 |
+
grefcoco|instancevg|Qwen3.8,learned,0.33,0.3627598332529847,0.7511272438533307,11820.450948963453,1540.0616161172863
|
| 36 |
+
grefcoco|instancevg|Qwen3.8,learned,0.34,0.37013300059552506,0.7512123188611293,12060.47749566076,1570.080893001255
|
| 37 |
+
grefcoco|instancevg|Qwen3.8,learned,0.35,0.3783569180160508,0.751439185548592,12328.199413130831,1603.5639326026044
|
| 38 |
+
grefcoco|instancevg|Qwen3.8,learned,0.36,0.3854748603351955,0.7517511272438533,12559.917348596306,1632.5440806713586
|
| 39 |
+
grefcoco|instancevg|Qwen3.8,learned,0.37,0.39279131100587017,0.752176502282846,12798.097537242094,1662.3324400408353
|
| 40 |
+
grefcoco|instancevg|Qwen3.8,learned,0.38,0.40013612001247767,0.7522332189547117,13037.200904913643,1692.2362581675577
|
| 41 |
+
grefcoco|instancevg|Qwen3.8,learned,0.39,0.406573462269234,0.7525735189859059,13246.762543760904,1718.445396062407
|
| 42 |
+
grefcoco|instancevg|Qwen3.8,learned,0.4,0.4153929047443496,0.7528003856733687,13533.871220771913,1754.3530695659233
|
| 43 |
+
grefcoco|instancevg|Qwen3.8,learned,0.41,0.42151830530584466,0.75311232736863,13733.27789033583,1779.2921611310667
|
| 44 |
+
grefcoco|instancevg|Qwen3.8,learned,0.42,0.42920341434364634,0.7535093440716899,13983.459406316484,1810.5814843447415
|
| 45 |
+
grefcoco|instancevg|Qwen3.8,learned,0.43,0.4367750900377166,0.7537362107591526,14229.948206194102,1841.4089725294323
|
| 46 |
+
grefcoco|instancevg|Qwen3.8,learned,0.44,0.4450273657941752,0.753679494087287,14498.593302689931,1875.0074708880277
|
| 47 |
+
grefcoco|instancevg|Qwen3.8,learned,0.45,0.45143634971499874,0.7537929274310183,14707.231762511437,1901.1011500256313
|
| 48 |
+
grefcoco|instancevg|Qwen3.8,learned,0.46,0.4578736919717551,0.7536227774154213,14916.7934013587,1927.3102879204807
|
| 49 |
+
grefcoco|instancevg|Qwen3.8,learned,0.47,0.4659274593766838,0.7537929274310183,15178.976244674219,1960.100574978354
|
| 50 |
+
grefcoco|instancevg|Qwen3.8,learned,0.48,0.4741797351331424,0.7538780024388169,15447.621341170048,1993.6990733369496
|
| 51 |
+
grefcoco|instancevg|Qwen3.8,learned,0.49,0.4806454357258316,0.7537362107591526,15658.106159043073,2020.0236699890452
|
| 52 |
+
grefcoco|instancevg|Qwen3.8,learned,0.5,0.487451436349715,0.7535093440716899,15879.6691252252,2047.733771728093
|
| 53 |
+
grefcoco|instancevg|Qwen3.8,learned,0.51,0.49482460369225534,0.7537645690950855,16119.695671922505,2077.7530486120613
|
| 54 |
+
grefcoco|instancevg|Qwen3.8,learned,0.52,0.501573887644273,0.7535660607435556,16339.412280053117,2105.232232836617
|
| 55 |
+
grefcoco|instancevg|Qwen3.8,learned,0.53,0.5105351217990528,0.7532257607123614,16631.136852192918,2141.7172001263634
|
| 56 |
+
grefcoco|instancevg|Qwen3.8,learned,0.54,0.519241130930437,0.7530556106967643,16914.552813100894,2177.163038600896
|
| 57 |
+
grefcoco|instancevg|Qwen3.8,learned,0.55,0.5268128066245072,0.7528571023452344,17161.041612978508,2207.9905267855866
|
| 58 |
+
grefcoco|instancevg|Qwen3.8,learned,0.56,0.5355755324277571,0.7528287440093016,17446.303931938,2243.6672827746106
|
| 59 |
+
grefcoco|instancevg|Qwen3.8,learned,0.57,0.5419561580126478,0.753282477384227,17654.019212733747,2269.645503154968
|
| 60 |
+
grefcoco|instancevg|Qwen3.8,learned,0.58,0.5495845503785838,0.7528287440093016,17902.35437066288,2300.703908854151
|
| 61 |
+
grefcoco|instancevg|Qwen3.8,learned,0.59,0.5562771176587358,0.7527153106655702,18120.224620741974,2327.952175564215
|
| 62 |
+
grefcoco|instancevg|Qwen3.8,learned,0.6,0.564047301704336,0.7526585939937045,18373.175673799906,2359.587875049628
|
| 63 |
+
grefcoco|instancevg|Qwen3.8,learned,0.61,0.5725831608201231,0.7528287440093016,18651.052560553326,2394.3409609806836
|
| 64 |
+
grefcoco|instancevg|Qwen3.8,learned,0.62,0.5818846950060971,0.7528571023452344,18953.855281002234,2432.2114333573827
|
| 65 |
+
grefcoco|instancevg|Qwen3.8,learned,0.63,0.5903638374500184,0.7528571023452344,19229.885809704134,2466.7336017739462
|
| 66 |
+
grefcoco|instancevg|Qwen3.8,learned,0.64,0.5973683464254318,0.7526869523296373,19457.911029066578,2495.2519148137167
|
| 67 |
+
grefcoco|instancevg|Qwen3.8,learned,0.65,0.6042594220571137,0.7526018773218388,19682.24353232598,2523.3083928245023
|
| 68 |
+
grefcoco|instancevg|Qwen3.8,learned,0.66,0.6122848311261095,0.752545160649973,19943.503196615744,2555.9832211251296
|
| 69 |
+
grefcoco|instancevg|Qwen3.8,learned,0.67,0.6198848651561126,0.7522048606187789,20190.91517551912,2586.9261680670666
|
| 70 |
+
grefcoco|instancevg|Qwen3.8,learned,0.68,0.6280520659047727,0.752176502282846,20456.790734937673,2620.178290153924
|
| 71 |
+
grefcoco|instancevg|Qwen3.8,learned,0.69,0.6363894166690298,0.7522048606187789,20728.20536851078,2654.1231647842574
|
| 72 |
+
grefcoco|instancevg|Qwen3.8,learned,0.7,0.6444999007458242,0.7521197856109804,20992.234569877815,2687.144369356623
|
| 73 |
+
grefcoco|instancevg|Qwen3.8,learned,0.71,0.6527521765022829,0.7519212772594505,21260.87966637365,2720.742867715219
|
| 74 |
+
grefcoco|instancevg|Qwen3.8,learned,0.72,0.6601253438448231,0.7518362022516518,21500.906213070954,2750.7621445991867
|
| 75 |
+
grefcoco|instancevg|Qwen3.8,learned,0.73,0.6697955363979242,0.7518362022516518,21815.71026085473,2790.1335808200847
|
| 76 |
+
grefcoco|instancevg|Qwen3.8,learned,0.74,0.6775373621075915,0.7518645605875848,22067.7381348869,2821.6538215482515
|
| 77 |
+
grefcoco|instancevg|Qwen3.8,learned,0.75,0.6870657629810283,0.7512973938689278,22377.92628754188,2860.4479639829183
|
| 78 |
+
grefcoco|instancevg|Qwen3.8,learned,0.76,0.6956299804327482,0.7514108272126592,22656.726353321057,2895.31650867122
|
| 79 |
+
grefcoco|instancevg|Qwen3.8,learned,0.77,0.7044494229078637,0.7514959022204577,22943.835030332066,2931.224182174736
|
| 80 |
+
grefcoco|instancevg|Qwen3.8,learned,0.78,0.7129852820236509,0.7510705271814649,23221.711917085486,2965.9772681057925
|
| 81 |
+
grefcoco|instancevg|Qwen3.8,learned,0.79,0.7243853330686555,0.7506735104784051,23592.82988544055,3012.3916885186977
|
| 82 |
+
grefcoco|instancevg|Qwen3.8,learned,0.8,0.7337152255905623,0.7507302271502708,23896.555784915217,3050.3776196526424
|
| 83 |
+
grefcoco|instancevg|Qwen3.8,learned,0.81,0.744377959901313,0.7505033604628081,24243.671098600553,3093.7901123771508
|
| 84 |
+
grefcoco|instancevg|Qwen3.8,learned,0.82,0.7522048606187789,0.7503048521112781,24498.46850971,3125.656729377056
|
| 85 |
+
grefcoco|instancevg|Qwen3.8,learned,0.83,0.7607123613986331,0.7503332104472109,24775.42221743766,3160.2943565508654
|
| 86 |
+
grefcoco|instancevg|Qwen3.8,learned,0.84,0.770779570654794,0.749964552080084,25103.15077158206,3201.2822153732072
|
| 87 |
+
grefcoco|instancevg|Qwen3.8,learned,0.85,0.7827467884184556,0.749964552080084,25492.7323204523,3250.005810931033
|
| 88 |
+
grefcoco|instancevg|Qwen3.8,learned,0.86,0.793466239401072,0.7498511187363526,25841.693992189157,3293.649221170034
|
| 89 |
+
grefcoco|instancevg|Qwen3.8,learned,0.87,0.8059439072115249,0.7499361937441511,26247.89276352306,3344.451074358288
|
| 90 |
+
grefcoco|instancevg|Qwen3.8,learned,0.88,0.8164932081785441,0.7497093270566884,26591.315361105357,3387.4017320538123
|
| 91 |
+
grefcoco|instancevg|Qwen3.8,learned,0.89,0.8270425091455633,0.7494257436973598,26934.737958687656,3430.3523897493365
|
| 92 |
+
grefcoco|instancevg|Qwen3.8,learned,0.9,0.8367410600345971,0.749397385361427,27250.46518549719,3469.83928472748
|
| 93 |
+
grefcoco|instancevg|Qwen3.8,learned,0.91,0.8491903695091172,0.74883021864277,27655.740777805335,3520.5256791584884
|
| 94 |
+
grefcoco|instancevg|Qwen3.8,learned,0.92,0.8618098289992343,0.7487167852990386,28066.55544426803,3571.904826132973
|
| 95 |
+
grefcoco|instancevg|Qwen3.8,learned,0.93,0.8745710801690156,0.7486884269631058,28481.98600585952,3623.8612668936876
|
| 96 |
+
grefcoco|instancevg|Qwen3.8,learned,0.94,0.8879845730652526,0.7486884269631058,28918.649685043467,3678.4732590710614
|
| 97 |
+
grefcoco|instancevg|Qwen3.8,learned,0.95,0.9000085075007799,0.7484332019397102,29310.07759196523,3727.4277721433796
|
| 98 |
+
grefcoco|instancevg|Qwen3.8,learned,0.96,0.9125996086549641,0.7477809602132547,29719.969079402166,3778.691460360618
|
| 99 |
+
grefcoco|instancevg|Qwen3.8,learned,0.97,0.9269489266369849,0.7476391685335905,30187.09766643615,3837.11359152711
|
| 100 |
+
grefcoco|instancevg|Qwen3.8,learned,0.98,0.9459773700479256,0.7472988685023962,30806.550792720354,3914.5864176391988
|
| 101 |
+
grefcoco|instancevg|Qwen3.8,learned,0.99,0.9658282052009188,0.7472421518305306,31452.77611075156,3995.4075477114216
|
| 102 |
+
grefcoco|instancevg|Qwen3.8,learned,1.0,1.0,0.7447182599325072,32565.206836791,4134.535350192891
|
| 103 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.0,0.0,0.6695119530385957,7.502680064,46.08042127620138
|
| 104 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.01,0.002467175226157729,0.6686895612965431,87.81925530502161,56.12533315660624
|
| 105 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.02,0.004849275444516916,0.6680373195700876,165.3662934687666,65.823868765273
|
| 106 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.03,0.007600034030003119,0.6675835861951621,254.9146589673769,77.02336821813819
|
| 107 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.04,0.009811984232765221,0.6671014944843037,326.9226229765687,86.02915128332876
|
| 108 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.05,0.012534384482318578,0.6666194027734452,415.54780944942013,97.11319197894791
|
| 109 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.06,0.01607917647392451,0.6662223860703854,530.9451876692788,111.54553663470202
|
| 110 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.07,0.019709043473328986,0.6658253693673255,649.112102966414,126.32425756219423
|
| 111 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.08,0.02353741882426339,0.6655985026798628,773.7412714438613,141.91118979040866
|
| 112 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.09,0.028868785979638715,0.6651164109690043,947.2989282865286,163.61743615266283
|
| 113 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.1,0.0345971698380739,0.6647761109378102,1133.7810914898203,186.94010511636148
|
| 114 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.11,0.03981510364971783,0.664435810906616,1303.6460322294522,208.18451644963153
|
| 115 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.12,0.045855429203414344,0.664067152539489,1500.2831647160913,232.77723174303654
|
| 116 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.13,0.05317187987408899,0.6637552108442276,1738.4633533618796,262.565591112513
|
| 117 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.14,0.05960922213084536,0.6636134191645634,1948.0249922091427,288.77472900736245
|
| 118 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.15,0.0675211978561098,0.6632164024615036,2205.591940395867,320.98772227900565
|
| 119 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.16,0.07551824858917279,0.6626208774069138,2465.928425659868,353.54709182238696
|
| 120 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.17,0.08212574086152624,0.6622522190397867,2681.0291386616846,380.44898226071257
|
| 121 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.18,0.08856308311828262,0.6619969940163911,2890.590777508948,406.6581201555621
|
| 122 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.19,0.09480191702350906,0.6615149023055327,3093.6901631758988,432.0590467496893
|
| 123 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.2,0.10007656750701868,0.6612596772821371,3265.4014619670484,453.5343755974514
|
| 124 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.21,0.10554972634205825,0.6608343022431443,3443.575013938511,475.81791574593575
|
| 125 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.22,0.11005870175538099,0.6605507188838159,3590.3604790341706,494.17585814805494
|
| 126 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.23,0.11527663556702493,0.6600969855088903,3760.2254197738025,515.420269481325
|
| 127 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.24,0.12029606102713893,0.6598417604854947,3923.6281073331224,535.8564695138729
|
| 128 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.25,0.12477667810452883,0.6594731021183677,4069.490393403024,554.0989531587461
|
| 129 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.26,0.12914386183818732,0.6590193687434421,4211.659963369889,571.879601774635
|
| 130 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.27,0.1322916371267334,0.658679068712248,4314.132835229124,584.6955238289447
|
| 131 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.28,0.13626180415733205,0.6582820520091881,4443.377898835366,600.8597498433894
|
| 132 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.29,0.14023197118793068,0.657743243626464,4572.622962441607,617.0239758578339
|
| 133 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.3,0.14326631313274538,0.657374585259337,4671.403118197806,629.3780628831595
|
| 134 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.31,0.14766185520233674,0.6569775685562771,4814.495867190431,647.2741702562946
|
| 135 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.32,0.15066783881121856,0.6567507018688143,4912.352843920871,659.512798524374
|
| 136 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.33,0.15452457249808582,0.6565238351813516,5037.905191424078,675.2151895098345
|
| 137 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.34,0.15806936448969175,0.6564104018376201,5153.302569643936,689.6475341655887
|
| 138 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.35,0.16201117318435754,0.6562686101579559,5281.6244542244185,705.6963014227872
|
| 139 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.36,0.1657544735274934,0.6558999517908289,5403.484085624589,720.9368573792635
|
| 140 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.37,0.17012165726115192,0.6558715934548961,5545.653655591455,738.7175059951526
|
| 141 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.38,0.17536794940872868,0.6555596517596347,5716.441775356846,760.0773760856687
|
| 142 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.39,0.17965005813458867,0.6553044267362391,5855.841808246436,777.5116484298197
|
| 143 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.4,0.18401724186824717,0.6552193517284406,5998.011378213301,795.2922970457087
|
| 144 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.41,0.18858293395343562,0.6552760684003063,6146.643201360479,813.88115696232
|
| 145 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.42,0.19238295096843716,0.6554745767518362,6270.349190812167,829.3526304332884
|
| 146 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.43,0.19689192638175992,0.655502935087769,6417.134655907828,847.7105728354077
|
| 147 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.44,0.20182627683407536,0.6555880100955676,6577.76780638987,867.8003965962173
|
| 148 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.45,0.20718600232538353,0.6559283101267618,6752.248642258297,889.6221017157176
|
| 149 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.46,0.21189348609023623,0.656070101806426,6905.49636053427,908.7882554185591
|
| 150 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.47,0.217139778237813,0.6567790602047472,7076.28448029966,930.148125509075
|
| 151 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.48,0.22198905368232993,0.6584238436888523,7234.148093704427,949.8915729981468
|
| 152 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.49,0.22763236253296656,0.6604372855400845,7417.860719830442,972.8678656901072
|
| 153 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.5,0.2345234381646485,0.662762669086578,7642.1932230898465,1000.9243437008932
|
| 154 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.51,0.24382497235062248,0.6654567110001985,7944.995943538756,1038.7948160775923
|
| 155 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.52,0.252644414825738,0.6678955278904234,8232.104620549764,1074.7024895811082
|
| 156 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.53,0.26401610753480986,0.6703627031165811,8602.299409879071,1121.0014512367675
|
| 157 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.54,0.2760116836344043,0.6731134617020673,8992.804137775072,1169.8405055518392
|
| 158 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.55,0.2893401015228426,0.6758358619516207,9426.69827988174,1224.1061214574747
|
| 159 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.56,0.3036043444970649,0.679295578935428,9891.057329838453,1282.1818763522294
|
| 160 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.57,0.3175850041119587,0.6822448458724442,10346.184589537575,1339.1030436745236
|
| 161 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.58,0.33462836400760004,0.6853926211609903,10901.015184018655,1408.4937567793893
|
| 162 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.59,0.35637920766809406,0.6889090548166633,11609.09349677571,1497.0506235870967
|
| 163 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.6,0.37968975980489467,0.6922553384567394,12367.9466559495,1591.9577220433357
|
| 164 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.61,0.4046450954258004,0.6949493803703599,13180.344198617304,1693.5614284198446
|
| 165 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.62,0.43141536454640844,0.6983807390182344,14051.825198933677,1802.5544952600997
|
| 166 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.63,0.4610214672603012,0.7015568726427134,15015.624101825935,1923.093437824958
|
| 167 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.64,0.4916484700677764,0.7053285313217821,16012.657449645512,2047.7888956506733
|
| 168 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.65,0.5209426310864078,0.7086180982899923,16966.301383254424,2167.0577918858253
|
| 169 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.66,0.5529308340186598,0.71148229021921,18007.64732431043,2297.2952700593505
|
| 170 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.67,0.5888041289737118,0.7155091739216742,19175.4687918954,2443.350597975582
|
| 171 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.68,0.6175027649377535,0.7182882908430933,20109.725965963375,2560.1948603085675
|
| 172 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.69,0.6421461588633979,0.7211241244363781,20911.968539347832,2660.52852035537
|
| 173 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.7,0.665910444375124,0.7233077163032073,21685.59256293376,2757.282958927545
|
| 174 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.71,0.6838612710206166,0.7243286163967898,22269.96488623913,2830.3683522642846
|
| 175 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.72,0.7029464311034228,0.7260017582168279,22891.264370574845,2908.072095890864
|
| 176 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.73,0.7180614241556306,0.7278450500524629,23383.318791304326,2969.611613503
|
| 177 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.74,0.7352465757309361,0.7297450585599637,23942.7652809142,3039.579620394096
|
| 178 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.75,0.7478093185491875,0.7311629753566061,24351.733589325377,3090.727849854088
|
| 179 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.76,0.759804894648782,0.7328928338485098,24742.238317221378,3139.5669041691604
|
| 180 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.77,0.7729064458497575,0.7342540339732865,25168.74702712198,3192.9088500168273
|
| 181 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.78,0.7855259053398747,0.735274934066869,25579.561693584674,3244.287996991312
|
| 182 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.79,0.7964722230099538,0.7362958341604514,25935.908797527598,3288.855077288281
|
| 183 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.8,0.8082125740861527,0.7377704676289595,26318.104914191772,3336.6550027881385
|
| 184 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.81,0.8197544168108215,0.738422709355415,26693.838777675628,3383.6467169872735
|
| 185 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.82,0.8333380597226555,0.7389048010662734,27136.04153101413,3438.9514617081236
|
| 186 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.83,0.8426112355726966,0.7398689844879902,27437.921072437275,3476.7064753275763
|
| 187 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.84,0.8529903865241187,0.7403227178629158,27775.804595865022,3518.964380479624
|
| 188 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.85,0.8635396874911381,0.7406913762300428,28119.227193447325,3561.9150381751488
|
| 189 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.86,0.8710262881774098,0.7413152596205654,28362.946456247664,3592.3961500881014
|
| 190 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.87,0.8809517057539064,0.7422227263704165,28686.05911526327,3632.8067151242126
|
| 191 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.88,0.8913024983693957,0.7430167597765364,29023.019459665255,3674.9491615190145
|
| 192 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.89,0.900915974250631,0.743442134815529,29335.977149397513,3714.0896802254197
|
| 193 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.9,0.910954825170859,0.7443212432294473,29662.782524516155,3754.9620802905156
|
| 194 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.91,0.921901142840938,0.7448033349403057,30019.129628459075,3799.529160587484
|
| 195 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.92,0.9303235686129938,0.7450018432918356,30293.31379910946,3833.820411489556
|
| 196 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.93,0.9393698777755721,0.7448884099481042,30587.80790832654,3870.6517550510403
|
| 197 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.94,0.9490117119927403,0.7448884099481042,30901.688777084553,3909.907732514692
|
| 198 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.95,0.9587953378895726,0.7449734849559028,31220.185540971364,3949.7410037645727
|
| 199 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.96,0.9673595553412926,0.7452570683152313,31498.985606750543,3984.609548452875
|
| 200 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.97,0.9746192893401016,0.745313784987097,31735.319437344813,4014.1669903078596
|
| 201 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.98,0.9830984317840229,0.7450585599637013,32011.349966046713,4048.689158724423
|
| 202 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.99,0.9908118991577575,0.7448884099481042,32262.45466105313,4080.0939406953444
|
| 203 |
+
grefcoco|instancevg|Qwen3.8,confidence,1.0,1.0,0.7447182599325072,32561.564665399,4117.502578043059
|
| 204 |
+
construction|yolo26x|Qwen3.8,learned,0.0,0.0,0.18129624718318377,139.16228659200002,71.56936161300109
|
| 205 |
+
construction|yolo26x|Qwen3.8,learned,0.01,0.013648468708388815,0.18693471712069915,1461.3537694872334,205.6311162003908
|
| 206 |
+
construction|yolo26x|Qwen3.8,learned,0.02,0.019640479360852198,0.18781313491575913,2041.8280790509946,264.48749626314725
|
| 207 |
+
construction|yolo26x|Qwen3.8,learned,0.03,0.031291611185086554,0.1913128749335621,3170.5281254249744,378.9304574962848
|
| 208 |
+
construction|yolo26x|Qwen3.8,learned,0.04,0.04061251664447404,0.19266008228682668,4073.4881625241583,470.48482648279486
|
| 209 |
+
construction|yolo26x|Qwen3.8,learned,0.05,0.045938748335552594,0.19531128485766117,4589.465326580834,522.8016087608005
|
| 210 |
+
construction|yolo26x|Qwen3.8,learned,0.06,0.0559254327563249,0.19800731707634547,5556.922509187103,620.8955755320613
|
| 211 |
+
construction|yolo26x|Qwen3.8,learned,0.07,0.07456724367509987,0.19763843353437874,7362.842583385471,804.0043135050814
|
| 212 |
+
construction|yolo26x|Qwen3.8,learned,0.08,0.08688415446071904,0.1974112776278713,8556.039775266534,924.9868725229696
|
| 213 |
+
construction|yolo26x|Qwen3.8,learned,0.09,0.09587217043941411,0.20030220967092682,9426.751239612175,1013.2714426171043
|
| 214 |
+
construction|yolo26x|Qwen3.8,learned,0.1,0.09920106524633822,0.20211074868345033,9749.236967147599,1045.9694315408578
|
| 215 |
+
construction|yolo26x|Qwen3.8,learned,0.11,0.11284953395472704,0.20834032699131655,11071.428450042833,1180.0311861282478
|
| 216 |
+
construction|yolo26x|Qwen3.8,learned,0.12,0.12350199733688415,0.21121674052765757,12103.382778156185,1284.664750684259
|
| 217 |
+
construction|yolo26x|Qwen3.8,learned,0.13,0.13315579227696406,0.21276215593410724,13038.591388008912,1379.4889185631446
|
| 218 |
+
construction|yolo26x|Qwen3.8,learned,0.14,0.14347536617842876,0.21511521078428622,14038.297143368722,1480.8526842267806
|
| 219 |
+
construction|yolo26x|Qwen3.8,learned,0.15,0.15246338215712382,0.21738179676765032,14909.008607714362,1569.1372543209152
|
| 220 |
+
construction|yolo26x|Qwen3.8,learned,0.16,0.1604527296937417,0.22260454951009195,15682.974353799378,1647.612427737924
|
| 221 |
+
construction|yolo26x|Qwen3.8,learned,0.17,0.1780958721704394,0.22627631882842034,17392.148709737117,1820.9117690338178
|
| 222 |
+
construction|yolo26x|Qwen3.8,learned,0.18,0.18442077230359522,0.2270967395967396,18004.871592054424,1883.0379479889498
|
| 223 |
+
construction|yolo26x|Qwen3.8,learned,0.19,0.2000665778961385,0.23397114982462924,19520.55451147091,2036.7184959305916
|
| 224 |
+
construction|yolo26x|Qwen3.8,learned,0.2,0.20439414114513982,0.23512276426997616,19939.78595726696,2079.2258815314713
|
| 225 |
+
construction|yolo26x|Qwen3.8,learned,0.21,0.21937416777629826,0.24202469545559713,21390.97173117636,2226.3668316883623
|
| 226 |
+
construction|yolo26x|Qwen3.8,learned,0.22,0.22769640479360853,0.24353835097564847,22197.18605001492,2308.1118039977464
|
| 227 |
+
construction|yolo26x|Qwen3.8,learned,0.23,0.23601864181091878,0.2454657390209296,23003.400368853476,2389.8567763071305
|
| 228 |
+
construction|yolo26x|Qwen3.8,learned,0.24,0.24533954727030627,0.2483414219175777,23906.36040595266,2481.4111452936404
|
| 229 |
+
construction|yolo26x|Qwen3.8,learned,0.25,0.25432756324900135,0.2504846177578749,24777.071870298303,2569.695715387775
|
| 230 |
+
construction|yolo26x|Qwen3.8,learned,0.26,0.2636484687083888,0.252142216376585,25680.031907397482,2661.250084374285
|
| 231 |
+
construction|yolo26x|Qwen3.8,learned,0.27,0.275965379494008,0.25276746625574176,26873.22909927855,2782.2326433921735
|
| 232 |
+
construction|yolo26x|Qwen3.8,learned,0.28,0.29427430093209056,0.25702642690899347,28646.900600723373,2962.071582472818
|
| 233 |
+
construction|yolo26x|Qwen3.8,learned,0.29,0.31091877496671105,0.26183035419940204,30259.32923840049,3125.561527091586
|
| 234 |
+
construction|yolo26x|Qwen3.8,learned,0.3,0.322237017310253,0.2672887365132105,31355.780712020925,3236.734689432348
|
| 235 |
+
construction|yolo26x|Qwen3.8,learned,0.31,0.33388814913448733,0.27039536303602457,32484.480758394904,3351.1776506654855
|
| 236 |
+
construction|yolo26x|Qwen3.8,learned,0.32,0.34354194407456723,0.2713201570387918,33419.68936824763,3446.001818544371
|
| 237 |
+
construction|yolo26x|Qwen3.8,learned,0.33,0.3492010652463382,0.2722481828097158,33967.91510505785,3501.588399714752
|
| 238 |
+
construction|yolo26x|Qwen3.8,learned,0.34,0.3545272969374168,0.274417471554269,34483.89226911453,3553.905181992758
|
| 239 |
+
construction|yolo26x|Qwen3.8,learned,0.35,0.3648468708388815,0.2782223498666452,35483.59802447434,3655.268947656394
|
| 240 |
+
construction|yolo26x|Qwen3.8,learned,0.36,0.37217043941411454,0.2795152621397537,36193.06662505227,3727.204523288652
|
| 241 |
+
construction|yolo26x|Qwen3.8,learned,0.37,0.37683089214380827,0.28090947933574756,36644.54664360186,3772.981707781907
|
| 242 |
+
construction|yolo26x|Qwen3.8,learned,0.38,0.3854860186418109,0.28295009794869297,37483.00953519396,3857.9964789836663
|
| 243 |
+
construction|yolo26x|Qwen3.8,learned,0.39,0.39980026631158455,0.28644294895435674,38869.69816359627,3998.5978313558066
|
| 244 |
+
construction|yolo26x|Qwen3.8,learned,0.4,0.4054593874833555,0.2862200047733393,39417.923900406495,4054.1844125261878
|
| 245 |
+
construction|yolo26x|Qwen3.8,learned,0.41,0.4104527296937417,0.28760386086589435,39901.65249170963,4103.231395911818
|
| 246 |
+
construction|yolo26x|Qwen3.8,learned,0.42,0.4181091877496671,0.2930744270093899,40643.3696650411,4178.436770436451
|
| 247 |
+
construction|yolo26x|Qwen3.8,learned,0.43,0.42809587217043943,0.2961685540536353,41610.82684764737,4276.530737207712
|
| 248 |
+
construction|yolo26x|Qwen3.8,learned,0.44,0.4370838881491345,0.2977058723546182,42481.53831199301,4364.8153073018475
|
| 249 |
+
construction|yolo26x|Qwen3.8,learned,0.45,0.44673768308921435,0.2999402430635043,43416.74692184573,4459.639475180732
|
| 250 |
+
construction|yolo26x|Qwen3.8,learned,0.46,0.45206391478029295,0.2993046120720782,43932.72408590242,4511.956257458738
|
| 251 |
+
construction|yolo26x|Qwen3.8,learned,0.47,0.45872170439414117,0.2998310238962732,44577.69554097326,4577.352235306245
|
| 252 |
+
construction|yolo26x|Qwen3.8,learned,0.48,0.4673768308921438,0.30393211265289466,45416.15843256536,4662.367006508005
|
| 253 |
+
construction|yolo26x|Qwen3.8,learned,0.49,0.47503328894806923,0.30350001249166053,46157.87560589683,4737.572381032637
|
| 254 |
+
construction|yolo26x|Qwen3.8,learned,0.5,0.4876830892143808,0.29875800068449215,47383.321370531434,4861.824738942901
|
| 255 |
+
construction|yolo26x|Qwen3.8,learned,0.51,0.5046604527296937,0.29989697064322696,49027.99858096209,5028.5844824540445
|
| 256 |
+
construction|yolo26x|Qwen3.8,learned,0.52,0.5206391478029294,0.297424793956044,50575.93007313212,5185.534829288062
|
| 257 |
+
construction|yolo26x|Qwen3.8,learned,0.53,0.5249667110519307,0.297424793956044,50995.161518928166,5228.042214888941
|
| 258 |
+
construction|yolo26x|Qwen3.8,learned,0.54,0.5436085219707057,0.2978194775414123,52801.08159312654,5411.150952861962
|
| 259 |
+
construction|yolo26x|Qwen3.8,learned,0.55,0.5529294274300932,0.29630990060928936,53704.04163022572,5502.705321848472
|
| 260 |
+
construction|yolo26x|Qwen3.8,learned,0.56,0.5612516644474035,0.29578344205458756,54510.25594906428,5584.450294157856
|
| 261 |
+
construction|yolo26x|Qwen3.8,learned,0.57,0.5692410119840213,0.2950949302773625,55284.221695149296,5662.925467574864
|
| 262 |
+
construction|yolo26x|Qwen3.8,learned,0.58,0.5902130492676432,0.2947373376338526,57315.88177862246,5868.922797794512
|
| 263 |
+
construction|yolo26x|Qwen3.8,learned,0.59,0.6021970705725699,0.29722091277674884,58476.830397749975,5986.635557920024
|
| 264 |
+
construction|yolo26x|Qwen3.8,learned,0.6,0.6208388814913449,0.29665021956289706,60282.750471948355,6169.744295893045
|
| 265 |
+
construction|yolo26x|Qwen3.8,learned,0.61,0.6288282290279628,0.295988506182091,61056.71621803337,6248.219469310054
|
| 266 |
+
construction|yolo26x|Qwen3.8,learned,0.62,0.6421438082556591,0.2939669175161438,62346.65912817505,6379.011425005067
|
| 267 |
+
construction|yolo26x|Qwen3.8,learned,0.63,0.6491344873501997,0.29277362668991463,63023.87915599944,6447.67720174495
|
| 268 |
+
construction|yolo26x|Qwen3.8,learned,0.64,0.6571238348868176,0.2930821542583563,63797.84490208446,6526.152375161959
|
| 269 |
+
construction|yolo26x|Qwen3.8,learned,0.65,0.6677762982689747,0.2892305220657417,64829.79923019781,6630.78593971797
|
| 270 |
+
construction|yolo26x|Qwen3.8,learned,0.66,0.6744340878828229,0.2915915088317972,65474.77068526865,6696.181917565477
|
| 271 |
+
construction|yolo26x|Qwen3.8,learned,0.67,0.6854194407456724,0.2936540392680743,66538.97358613556,6804.085281013864
|
| 272 |
+
construction|yolo26x|Qwen3.8,learned,0.68,0.7047270306258322,0.29632332439424064,68409.390805841,6993.733616771635
|
| 273 |
+
construction|yolo26x|Qwen3.8,learned,0.69,0.7157123834886817,0.2937246709263134,69473.5937067079,7101.6369802200215
|
| 274 |
+
construction|yolo26x|Qwen3.8,learned,0.7,0.7253661784287616,0.2952254059747369,70408.80231656063,7196.461148098907
|
| 275 |
+
construction|yolo26x|Qwen3.8,learned,0.71,0.7283621837549934,0.2945683782087344,70699.03947134252,7225.889338130286
|
| 276 |
+
construction|yolo26x|Qwen3.8,learned,0.72,0.741011984021305,0.29252963134197957,71924.48523597712,7350.1416960405495
|
| 277 |
+
construction|yolo26x|Qwen3.8,learned,0.73,0.7526631158455392,0.2900395106945919,73053.1852823511,7464.584657273686
|
| 278 |
+
construction|yolo26x|Qwen3.8,learned,0.74,0.7616511318242344,0.2887744568626224,73923.89674669674,7552.869227367822
|
| 279 |
+
construction|yolo26x|Qwen3.8,learned,0.75,0.7709720372836218,0.28730364578560735,74826.85678379593,7644.4235963543315
|
| 280 |
+
construction|yolo26x|Qwen3.8,learned,0.76,0.7749667110519307,0.2858070539975931,75213.83965683842,7683.661183062835
|
| 281 |
+
construction|yolo26x|Qwen3.8,learned,0.77,0.7792942743009321,0.28488735368132817,75633.07110263448,7726.168568663716
|
| 282 |
+
construction|yolo26x|Qwen3.8,learned,0.78,0.7846205059920106,0.2830658409372236,76149.04826669116,7778.485350941721
|
| 283 |
+
construction|yolo26x|Qwen3.8,learned,0.79,0.7972703062583223,0.2809995077380739,77374.49403132577,7902.737708851985
|
| 284 |
+
construction|yolo26x|Qwen3.8,learned,0.8,0.8129161118508655,0.28013559922601305,78890.17695074226,8056.418256793627
|
| 285 |
+
construction|yolo26x|Qwen3.8,learned,0.81,0.8199067909454061,0.27814841049220523,79567.39697856663,8125.084033533509
|
| 286 |
+
construction|yolo26x|Qwen3.8,learned,0.82,0.831225033288948,0.2772348763322643,80663.84845218707,8236.25719587427
|
| 287 |
+
construction|yolo26x|Qwen3.8,learned,0.83,0.84254327563249,0.27700595116508253,81760.29992580751,8347.430358215033
|
| 288 |
+
construction|yolo26x|Qwen3.8,learned,0.84,0.857190412782956,0.2763733409861609,83179.23712696337,8491.301509479548
|
| 289 |
+
construction|yolo26x|Qwen3.8,learned,0.85,0.8618508655126498,0.2761345157638265,83630.71714551296,8537.078693972804
|
| 290 |
+
construction|yolo26x|Qwen3.8,learned,0.86,0.87017310252996,0.2761345157638265,84436.93146435152,8618.823666282187
|
| 291 |
+
construction|yolo26x|Qwen3.8,learned,0.87,0.8848202396804261,0.27546832609774896,85855.86866550738,8762.694817546704
|
| 292 |
+
construction|yolo26x|Qwen3.8,learned,0.88,0.8984687083888149,0.27423981143437465,87178.06014840261,8896.756572134092
|
| 293 |
+
construction|yolo26x|Qwen3.8,learned,0.89,0.9087882822902796,0.2736575311942959,88177.76590376243,8998.12033779773
|
| 294 |
+
construction|yolo26x|Qwen3.8,learned,0.9,0.9161118508655126,0.272975172975173,88887.23450434035,9070.055913429987
|
| 295 |
+
construction|yolo26x|Qwen3.8,learned,0.91,0.9217709720372836,0.2727190231806067,89435.46024115058,9125.64249460037
|
| 296 |
+
construction|yolo26x|Qwen3.8,learned,0.92,0.9264314247669774,0.27264729817760425,89886.94025970016,9171.419679093624
|
| 297 |
+
construction|yolo26x|Qwen3.8,learned,0.93,0.933422103861518,0.2741441928343702,90564.16028752456,9240.085455833507
|
| 298 |
+
construction|yolo26x|Qwen3.8,learned,0.94,0.9417443408788282,0.2733716828250215,91370.37460636311,9321.83042814289
|
| 299 |
+
construction|yolo26x|Qwen3.8,learned,0.95,0.9530625832223701,0.2727312493845365,92466.82607998354,9433.003590483651
|
| 300 |
+
construction|yolo26x|Qwen3.8,learned,0.96,0.9613848202396804,0.27223682114986464,93273.04039882211,9514.748562793036
|
| 301 |
+
construction|yolo26x|Qwen3.8,learned,0.97,0.9673768308921438,0.27453093162048386,93853.51470838586,9573.604942855793
|
| 302 |
+
construction|yolo26x|Qwen3.8,learned,0.98,0.9760319573901465,0.2737331615154065,94691.97759997797,9658.619714057553
|
| 303 |
+
construction|yolo26x|Qwen3.8,learned,0.99,0.9890146471371505,0.2725383412883413,95949.67193736612,9786.141870860192
|
| 304 |
+
construction|yolo26x|Qwen3.8,learned,1.0,1.0,0.27001009067886245,97013.87483823301,9894.045234308578
|
| 305 |
+
construction|yolo26x|Qwen3.8,confidence,0.0,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 306 |
+
construction|yolo26x|Qwen3.8,confidence,0.01,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 307 |
+
construction|yolo26x|Qwen3.8,confidence,0.02,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 308 |
+
construction|yolo26x|Qwen3.8,confidence,0.03,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 309 |
+
construction|yolo26x|Qwen3.8,confidence,0.04,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 310 |
+
construction|yolo26x|Qwen3.8,confidence,0.05,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 311 |
+
construction|yolo26x|Qwen3.8,confidence,0.06,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 312 |
+
construction|yolo26x|Qwen3.8,confidence,0.07,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 313 |
+
construction|yolo26x|Qwen3.8,confidence,0.08,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 314 |
+
construction|yolo26x|Qwen3.8,confidence,0.09,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 315 |
+
construction|yolo26x|Qwen3.8,confidence,0.1,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 316 |
+
construction|yolo26x|Qwen3.8,confidence,0.11,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 317 |
+
construction|yolo26x|Qwen3.8,confidence,0.12,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 318 |
+
construction|yolo26x|Qwen3.8,confidence,0.13,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 319 |
+
construction|yolo26x|Qwen3.8,confidence,0.14,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 320 |
+
construction|yolo26x|Qwen3.8,confidence,0.15,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 321 |
+
construction|yolo26x|Qwen3.8,confidence,0.16,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 322 |
+
construction|yolo26x|Qwen3.8,confidence,0.17,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 323 |
+
construction|yolo26x|Qwen3.8,confidence,0.18,0.0,0.18129624718318377,135.97853184000002,46.30809425361804
|
| 324 |
+
construction|yolo26x|Qwen3.8,confidence,0.19,0.17942743009320905,0.1784481207469488,17517.959245999285,1808.7296972439362
|
| 325 |
+
construction|yolo26x|Qwen3.8,confidence,0.2,0.19340878828229027,0.1780928651788194,18872.39930164806,1946.0612507237013
|
| 326 |
+
construction|yolo26x|Qwen3.8,confidence,0.21,0.20306258322237017,0.17829989216687975,19807.60791150079,2040.8854186025867
|
| 327 |
+
construction|yolo26x|Qwen3.8,confidence,0.22,0.20838881491344874,0.1779854683890517,20323.585075557465,2093.2022008805925
|
| 328 |
+
construction|yolo26x|Qwen3.8,confidence,0.23,0.21537949400798934,0.17917657728013692,21000.805103381852,2161.867977620475
|
| 329 |
+
construction|yolo26x|Qwen3.8,confidence,0.24,0.23135818908122505,0.17803687432776102,22548.736595551884,2318.8183244544925
|
| 330 |
+
construction|yolo26x|Qwen3.8,confidence,0.25,0.24800266311584554,0.18026802885144627,24161.165233228996,2482.3082690732604
|
| 331 |
+
construction|yolo26x|Qwen3.8,confidence,0.26,0.2589880159786951,0.18119827871024874,25225.368134095894,2590.211632521647
|
| 332 |
+
construction|yolo26x|Qwen3.8,confidence,0.27,0.27430093209054596,0.17963665085316138,26708.802480758837,2740.6223815709136
|
| 333 |
+
construction|yolo26x|Qwen3.8,confidence,0.28,0.281624500665779,0.17924095839307458,27418.271081336767,2812.5579572031716
|
| 334 |
+
construction|yolo26x|Qwen3.8,confidence,0.29,0.28761651131824234,0.1789157174618718,27998.745390900527,2871.4143372659278
|
| 335 |
+
construction|yolo26x|Qwen3.8,confidence,0.3,0.2969374167776298,0.1787206845800119,28901.70542799971,2962.9687062524376
|
| 336 |
+
construction|yolo26x|Qwen3.8,confidence,0.31,0.30459387483355527,0.17813242189706796,29643.422601331185,3038.174080777071
|
| 337 |
+
construction|yolo26x|Qwen3.8,confidence,0.32,0.3129161118508655,0.17772000842137498,30449.636920169738,3119.919053086455
|
| 338 |
+
construction|yolo26x|Qwen3.8,confidence,0.33,0.31990679094540614,0.17791259734473341,31126.85694799413,3188.5848298263377
|
| 339 |
+
construction|yolo26x|Qwen3.8,confidence,0.34,0.3322237017310253,0.1789653934599337,32320.05413987519,3309.5673888442257
|
| 340 |
+
construction|yolo26x|Qwen3.8,confidence,0.35,0.34054593874833555,0.18042682200794885,33126.268458713756,3391.31236115361
|
| 341 |
+
construction|yolo26x|Qwen3.8,confidence,0.36,0.3488681757656458,0.17962877325912951,33932.48277755231,3473.057333462994
|
| 342 |
+
construction|yolo26x|Qwen3.8,confidence,0.37,0.3581890812250333,0.18158304685510065,34835.44281465149,3564.611702449504
|
| 343 |
+
construction|yolo26x|Qwen3.8,confidence,0.38,0.37017310252996005,0.18069239723097547,35996.39143377901,3682.3244625750167
|
| 344 |
+
construction|yolo26x|Qwen3.8,confidence,0.39,0.37683089214380827,0.1812845803851709,36641.362888849864,3747.720440422524
|
| 345 |
+
construction|yolo26x|Qwen3.8,confidence,0.4,0.38348868175765644,0.1824781664689562,37286.3343439207,3813.116418270031
|
| 346 |
+
construction|yolo26x|Qwen3.8,confidence,0.41,0.4021304926764314,0.18220594501399923,39092.254418119075,3996.225156243051
|
| 347 |
+
construction|yolo26x|Qwen3.8,confidence,0.42,0.41311584553928093,0.18162498930772586,40156.457318985966,4104.128519691438
|
| 348 |
+
construction|yolo26x|Qwen3.8,confidence,0.43,0.42676431424766975,0.18066434967152634,41478.6488018812,4238.190274278827
|
| 349 |
+
construction|yolo26x|Qwen3.8,confidence,0.44,0.4324234354194407,0.1803913789289444,42026.874538691416,4293.776855449209
|
| 350 |
+
construction|yolo26x|Qwen3.8,confidence,0.45,0.44107856191744343,0.17959468902865128,42865.33743028352,4378.791626650968
|
| 351 |
+
construction|yolo26x|Qwen3.8,confidence,0.46,0.45772303595206393,0.17863383514615866,44477.766067960634,4542.281571269737
|
| 352 |
+
construction|yolo26x|Qwen3.8,confidence,0.47,0.4717043941411451,0.17957308538379646,45832.20612360941,4679.6131247495005
|
| 353 |
+
construction|yolo26x|Qwen3.8,confidence,0.48,0.4826897470039947,0.17910245145856155,46896.40902447631,4787.516488197888
|
| 354 |
+
construction|yolo26x|Qwen3.8,confidence,0.49,0.4896804260985353,0.17871833907402393,47573.62905230069,4856.182264937771
|
| 355 |
+
construction|yolo26x|Qwen3.8,confidence,0.5,0.5013315579227696,0.18015785955296285,48702.32909867467,4970.625226170908
|
| 356 |
+
construction|yolo26x|Qwen3.8,confidence,0.51,0.5116511318242344,0.1810878167493375,49702.034854034486,5071.988991834544
|
| 357 |
+
construction|yolo26x|Qwen3.8,confidence,0.52,0.5183089214380826,0.18177962418520882,50347.00630910533,5137.384969682052
|
| 358 |
+
construction|yolo26x|Qwen3.8,confidence,0.53,0.5252996005326231,0.18246323539519374,51024.226336929714,5206.050746421934
|
| 359 |
+
construction|yolo26x|Qwen3.8,confidence,0.54,0.5396138482023968,0.1839876166906015,52410.914965332035,5346.652098794074
|
| 360 |
+
construction|yolo26x|Qwen3.8,confidence,0.55,0.5452729693741678,0.1846674213907964,52959.14070214226,5402.2386799644555
|
| 361 |
+
construction|yolo26x|Qwen3.8,confidence,0.56,0.559254327563249,0.18676294233791663,54313.58075779103,5539.57023344422
|
| 362 |
+
construction|yolo26x|Qwen3.8,confidence,0.57,0.572237017310253,0.18938663221175195,55571.275095179175,5667.092390246859
|
| 363 |
+
construction|yolo26x|Qwen3.8,confidence,0.58,0.5862183754993342,0.1909571002676625,56925.71515082795,5804.423943726624
|
| 364 |
+
construction|yolo26x|Qwen3.8,confidence,0.59,0.5998668442077231,0.1934300101790686,58247.90663372319,5938.485698314014
|
| 365 |
+
construction|yolo26x|Qwen3.8,confidence,0.6,0.6191744340878829,0.1973701682130997,60118.32385342864,6128.134034071785
|
| 366 |
+
construction|yolo26x|Qwen3.8,confidence,0.61,0.6264980026631158,0.1986727924998457,60827.792454006565,6200.069609704043
|
| 367 |
+
construction|yolo26x|Qwen3.8,confidence,0.62,0.6384820239680427,0.20186289191698192,61988.74107313409,6317.782369829556
|
| 368 |
+
construction|yolo26x|Qwen3.8,confidence,0.63,0.6461384820239681,0.20246031058520178,62730.45824646557,6392.987744354189
|
| 369 |
+
construction|yolo26x|Qwen3.8,confidence,0.64,0.653462050599201,0.20349944137622245,63439.92684704349,6464.923319986447
|
| 370 |
+
construction|yolo26x|Qwen3.8,confidence,0.65,0.6597869507323568,0.20442990090722246,64052.64972936079,6527.049498941578
|
| 371 |
+
construction|yolo26x|Qwen3.8,confidence,0.66,0.6661118508655126,0.2044993942500819,64665.372611678096,6589.17567789671
|
| 372 |
+
construction|yolo26x|Qwen3.8,confidence,0.67,0.6734354194407457,0.2064379542733177,65374.84121225603,6661.1112535289685
|
| 373 |
+
construction|yolo26x|Qwen3.8,confidence,0.68,0.6820905459387483,0.2065831206227621,66213.30410384812,6746.126024730727
|
| 374 |
+
construction|yolo26x|Qwen3.8,confidence,0.69,0.6927430093209055,0.20802802394105854,67245.25843196148,6850.759589286739
|
| 375 |
+
construction|yolo26x|Qwen3.8,confidence,0.7,0.7090545938748336,0.21169680596278434,68825.43849688505,7010.979735013132
|
| 376 |
+
construction|yolo26x|Qwen3.8,confidence,0.71,0.7170439414114514,0.2139280527103987,69599.40424297006,7089.454908430141
|
| 377 |
+
construction|yolo26x|Qwen3.8,confidence,0.72,0.7243675099866844,0.2146900231025185,70308.87284354799,7161.390484062398
|
| 378 |
+
construction|yolo26x|Qwen3.8,confidence,0.73,0.7330226364846871,0.21653561577692484,71147.33573514009,7246.405255264158
|
| 379 |
+
construction|yolo26x|Qwen3.8,confidence,0.74,0.7363515312916112,0.21721689648133474,71469.82146267552,7279.1032441879115
|
| 380 |
+
construction|yolo26x|Qwen3.8,confidence,0.75,0.7486684420772304,0.22103445955201576,72663.01865455658,7400.0858032057995
|
| 381 |
+
construction|yolo26x|Qwen3.8,confidence,0.76,0.757323568575233,0.22340134248367294,73501.48154614867,7485.100574407558
|
| 382 |
+
construction|yolo26x|Qwen3.8,confidence,0.77,0.7639813581890812,0.2269322568854726,74146.45300121952,7550.496552255066
|
| 383 |
+
construction|yolo26x|Qwen3.8,confidence,0.78,0.7733022636484687,0.22757452798038888,75049.4130383187,7642.050921241575
|
| 384 |
+
construction|yolo26x|Qwen3.8,confidence,0.79,0.784953395472703,0.227821050912831,76178.11308469268,7756.493882474713
|
| 385 |
+
construction|yolo26x|Qwen3.8,confidence,0.8,0.7939414114513982,0.22991865671924006,77048.82454903833,7844.7784525688485
|
| 386 |
+
construction|yolo26x|Qwen3.8,confidence,0.81,0.8039280958721704,0.23486191981046528,78016.28173164459,7942.8724193401085
|
| 387 |
+
construction|yolo26x|Qwen3.8,confidence,0.82,0.8122503328894807,0.23803916047544685,78822.49605048315,8024.617391649493
|
| 388 |
+
construction|yolo26x|Qwen3.8,confidence,0.83,0.8215712383488681,0.24022947987632814,79725.45608758234,8116.171760636003
|
| 389 |
+
construction|yolo26x|Qwen3.8,confidence,0.84,0.8388814913448736,0.24528627405740036,81402.38187076653,8286.201303039523
|
| 390 |
+
construction|yolo26x|Qwen3.8,confidence,0.85,0.8535286284953395,0.249537136326177,82821.3190719224,8430.072454304036
|
| 391 |
+
construction|yolo26x|Qwen3.8,confidence,0.86,0.8638482023968043,0.25211074301800224,83821.0248272822,8531.436219967674
|
| 392 |
+
construction|yolo26x|Qwen3.8,confidence,0.87,0.8745006657789614,0.2540042317315216,84852.97915539556,8636.069784523685
|
| 393 |
+
construction|yolo26x|Qwen3.8,confidence,0.88,0.8764980026631158,0.2540996517958595,85046.4705919168,8655.688577877936
|
| 394 |
+
construction|yolo26x|Qwen3.8,confidence,0.89,0.8841544607190412,0.25436194678023316,85788.18776524828,8730.89395240257
|
| 395 |
+
construction|yolo26x|Qwen3.8,confidence,0.9,0.8891478029294274,0.2534339447803836,86271.91635655142,8779.9409357882
|
| 396 |
+
construction|yolo26x|Qwen3.8,confidence,0.91,0.8981358189081226,0.2563240880787248,87142.62782089706,8868.225505882336
|
| 397 |
+
construction|yolo26x|Qwen3.8,confidence,0.92,0.9054593874833555,0.25551845731687656,87852.09642147498,8940.161081514594
|
| 398 |
+
construction|yolo26x|Qwen3.8,confidence,0.93,0.9221038615179761,0.2597490004762105,89464.5250591521,9103.651026133362
|
| 399 |
+
construction|yolo26x|Qwen3.8,confidence,0.94,0.9254327563249002,0.26306323674307786,89787.01078668753,9136.349015057116
|
| 400 |
+
construction|yolo26x|Qwen3.8,confidence,0.95,0.9364181091877497,0.26218185491449153,90851.21368755442,9244.252378505502
|
| 401 |
+
construction|yolo26x|Qwen3.8,confidence,0.96,0.953728362183755,0.26357052137555514,92528.13947073862,9414.281920909021
|
| 402 |
+
construction|yolo26x|Qwen3.8,confidence,0.97,0.96271637816245,0.26400290307863034,93398.85093508427,9502.566491003154
|
| 403 |
+
construction|yolo26x|Qwen3.8,confidence,0.98,0.9700399467376831,0.2647475724296899,94108.3195356622,9574.502066635414
|
| 404 |
+
construction|yolo26x|Qwen3.8,confidence,0.99,0.987017310252996,0.2649306125590882,95752.99674609285,9741.261810146556
|
| 405 |
+
construction|yolo26x|Qwen3.8,confidence,1.0,1.0,0.27001009067886245,97010.691083481,9868.783966949195
|
paper_results/data/main_table_cost_qwen_20260912/main_table_results.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_flops.py
ADDED
|
@@ -0,0 +1,152 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
from __future__ import annotations
|
| 3 |
+
|
| 4 |
+
import argparse
|
| 5 |
+
import json
|
| 6 |
+
import re
|
| 7 |
+
from pathlib import Path
|
| 8 |
+
|
| 9 |
+
import numpy as np
|
| 10 |
+
import torch
|
| 11 |
+
from PIL import Image
|
| 12 |
+
from torch.utils.flop_counter import FlopCounterMode
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
def count(call):
|
| 16 |
+
with torch.inference_mode(), FlopCounterMode(display=False) as counter:
|
| 17 |
+
value = call()
|
| 18 |
+
torch.cuda.synchronize()
|
| 19 |
+
counts = counter.get_flop_counts().get('Global', {})
|
| 20 |
+
return int(counter.get_total_flops()), value, {str(k): int(v) for k, v in counts.items()}
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def make_text(root: Path, model_kind: str, task: str, row: dict, processor):
|
| 24 |
+
prompt = (root / 'outputs/large_model_labels_all_20260909/prompts' / task / f'{model_kind}.txt').read_text(encoding='utf-8').strip()
|
| 25 |
+
if task == 'grefcoco':
|
| 26 |
+
prompt = prompt.replace('{expression}', row['expression'])
|
| 27 |
+
if model_kind == 'Qwen3.8':
|
| 28 |
+
content = [{'type': 'image'}, {'type': 'text', 'text': '\n'}, {'type': 'text', 'text': prompt}]
|
| 29 |
+
return processor.apply_chat_template([{'role': 'user', 'content': content}], tokenize=False, add_generation_prompt=True)
|
| 30 |
+
message = '(<image>./</image>)\n' + prompt
|
| 31 |
+
return processor.tokenizer.apply_chat_template([{'role': 'user', 'content': message}], tokenize=False, add_generation_prompt=True, enable_thinking=False)
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def prepare(root: Path, model_kind: str, task: str, row: dict, processor):
|
| 35 |
+
image = Image.open(row['image_path']).convert('RGB')
|
| 36 |
+
text = make_text(root, model_kind, task, row, processor)
|
| 37 |
+
if model_kind == 'Qwen3.8':
|
| 38 |
+
data = processor(text=[text], images=[image], return_tensors='pt', size={'shortest_edge': 65536, 'longest_edge': 1003520})
|
| 39 |
+
else:
|
| 40 |
+
data = processor([text], [[image]], return_tensors='pt', max_length=16384)
|
| 41 |
+
image.close()
|
| 42 |
+
return data
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
def main():
|
| 46 |
+
parser = argparse.ArgumentParser()
|
| 47 |
+
parser.add_argument('--root', type=Path, required=True)
|
| 48 |
+
parser.add_argument('--model-kind', choices=['Qwen3.8', 'MiniCPM'], required=True)
|
| 49 |
+
parser.add_argument('--task', choices=['grefcoco', 'construction'], required=True)
|
| 50 |
+
parser.add_argument('--output', type=Path, required=True)
|
| 51 |
+
parser.add_argument('--samples', type=int, default=4)
|
| 52 |
+
args = parser.parse_args()
|
| 53 |
+
if args.output.exists():
|
| 54 |
+
raise FileExistsError(args.output)
|
| 55 |
+
expert = 'instancevg' if args.task == 'grefcoco' else 'yolo26x'
|
| 56 |
+
rows = [json.loads(line) for line in (args.root / f'outputs/router_full_20260910/data/{expert}/test/records.jsonl').open(encoding='utf-8')]
|
| 57 |
+
if args.samples > len(rows):
|
| 58 |
+
raise ValueError('sample count exceeds test rows')
|
| 59 |
+
selected_indices = np.random.default_rng(20260911).choice(len(rows), args.samples, replace=False)
|
| 60 |
+
selected = [rows[int(index)] for index in selected_indices]
|
| 61 |
+
checkpoint_root = args.root.parents[1] / 'checkpoints'
|
| 62 |
+
if not checkpoint_root.exists():
|
| 63 |
+
checkpoint_root = args.root / 'checkpoints'
|
| 64 |
+
model_path = checkpoint_root / ('Qwen3.8-27B-FP8' if args.model_kind == 'Qwen3.8' else 'MiniCPM-V-4_5')
|
| 65 |
+
from transformers import AutoProcessor
|
| 66 |
+
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True, local_files_only=True)
|
| 67 |
+
if args.model_kind == 'Qwen3.8':
|
| 68 |
+
from transformers import FineGrainedFP8Config, Qwen3_5ForConditionalGeneration
|
| 69 |
+
import transformers.models.qwen3_5.modeling_qwen3_5 as implementation
|
| 70 |
+
implementation.causal_conv1d_fn = None
|
| 71 |
+
implementation.causal_conv1d_update = None
|
| 72 |
+
implementation.chunk_gated_delta_rule = None
|
| 73 |
+
implementation.fused_recurrent_gated_delta_rule = None
|
| 74 |
+
implementation.FusedRMSNormGated = None
|
| 75 |
+
implementation.is_fast_path_available = False
|
| 76 |
+
model = Qwen3_5ForConditionalGeneration.from_pretrained(
|
| 77 |
+
model_path, local_files_only=True, dtype=torch.bfloat16, device_map={'': 'cuda:0'},
|
| 78 |
+
attn_implementation='eager', quantization_config=FineGrainedFP8Config(dequantize=False),
|
| 79 |
+
)
|
| 80 |
+
else:
|
| 81 |
+
model = processor # replaced below to keep imports local to the selected environment
|
| 82 |
+
from transformers import AutoModel
|
| 83 |
+
from transformers.modeling_utils import PreTrainedModel
|
| 84 |
+
if not hasattr(PreTrainedModel, 'all_tied_weights_keys'):
|
| 85 |
+
def get_tied_weights(self):
|
| 86 |
+
return getattr(self, '_all_tied_weights_keys', getattr(self, '_tied_weights_keys', {}) or {})
|
| 87 |
+
def set_tied_weights(self, value):
|
| 88 |
+
self._all_tied_weights_keys = value
|
| 89 |
+
PreTrainedModel.all_tied_weights_keys = property(get_tied_weights, set_tied_weights)
|
| 90 |
+
model = AutoModel.from_pretrained(
|
| 91 |
+
model_path, trust_remote_code=True, local_files_only=True, torch_dtype=torch.bfloat16,
|
| 92 |
+
device_map={'': 'cuda:0'}, attn_implementation='eager',
|
| 93 |
+
)
|
| 94 |
+
torch.set_num_threads(4)
|
| 95 |
+
torch.backends.mha.set_fastpath_enabled(False)
|
| 96 |
+
model.eval().requires_grad_(False)
|
| 97 |
+
records = []
|
| 98 |
+
for row in selected:
|
| 99 |
+
data = prepare(args.root, args.model_kind, args.task, row, processor).to('cuda')
|
| 100 |
+
prompt_tokens = int(data['input_ids'].shape[-1])
|
| 101 |
+
if args.model_kind == 'MiniCPM':
|
| 102 |
+
vision_flops, embedding_result, vision_ops = count(lambda: model.get_vllm_embedding(data))
|
| 103 |
+
embeddings, _ = embedding_result
|
| 104 |
+
prefill_flops, prefill_output, prefill_ops = count(lambda: model.llm(inputs_embeds=embeddings, attention_mask=data['attention_mask'], use_cache=True, logits_to_keep=1))
|
| 105 |
+
prefill_flops += vision_flops
|
| 106 |
+
prefill_ops.update({'vision:'+k:v for k,v in vision_ops.items()})
|
| 107 |
+
length = int(data['input_ids'].shape[-1])
|
| 108 |
+
known_token = data['input_ids'][:, -1:]
|
| 109 |
+
kwargs = {
|
| 110 |
+
'input_ids': known_token,
|
| 111 |
+
'past_key_values': prefill_output.past_key_values,
|
| 112 |
+
'attention_mask': torch.ones((1, length + 1), device='cuda', dtype=torch.long),
|
| 113 |
+
'use_cache': True,
|
| 114 |
+
'logits_to_keep': 1,
|
| 115 |
+
}
|
| 116 |
+
decode_flops, _, decode_ops = count(lambda: model.llm(**kwargs))
|
| 117 |
+
else:
|
| 118 |
+
prefill_flops, prefill_output, prefill_ops = count(lambda: model(**data, use_cache=True, logits_to_keep=1))
|
| 119 |
+
length = int(data['input_ids'].shape[-1])
|
| 120 |
+
known_token = data['input_ids'][:, -1:]
|
| 121 |
+
kwargs = {
|
| 122 |
+
'input_ids': known_token,
|
| 123 |
+
'past_key_values': prefill_output.past_key_values,
|
| 124 |
+
'attention_mask': torch.ones((1, length + 1), device='cuda', dtype=torch.long),
|
| 125 |
+
'use_cache': True,
|
| 126 |
+
'logits_to_keep': 1,
|
| 127 |
+
'cache_position': torch.tensor([length], device='cuda'),
|
| 128 |
+
}
|
| 129 |
+
kwargs.pop('attention_mask')
|
| 130 |
+
decode_flops, _, decode_ops = count(lambda: model(**kwargs))
|
| 131 |
+
records.append({
|
| 132 |
+
'sample_id': row['sample_id'], 'prompt_tokens_in_reference_tensor': prompt_tokens,
|
| 133 |
+
'input_shapes': {key: list(value.shape) for key, value in data.items() if hasattr(value, 'shape')},
|
| 134 |
+
'prefill_flops_counted': prefill_flops, 'decode_one_token_flops_counted': decode_flops,
|
| 135 |
+
'prefill_gflops_counted': prefill_flops / 1e9, 'decode_one_token_gflops_counted': decode_flops / 1e9,
|
| 136 |
+
'prefill_operators': prefill_ops, 'decode_operators': decode_ops,
|
| 137 |
+
})
|
| 138 |
+
args.output.parent.mkdir(parents=True, exist_ok=True)
|
| 139 |
+
args.output.write_text(json.dumps({'status': 'running', 'model_kind': args.model_kind, 'task': args.task, 'records': records}, ensure_ascii=False, indent=2), encoding='utf-8')
|
| 140 |
+
print(json.dumps({'completed': len(records), 'sample_id': row['sample_id'], 'prefill_gflops': prefill_flops / 1e9, 'decode_gflops': decode_flops / 1e9}), flush=True)
|
| 141 |
+
args.output.write_text(json.dumps({
|
| 142 |
+
'status': 'complete', 'model_kind': args.model_kind, 'task': args.task, 'model_path': str(model_path),
|
| 143 |
+
'sample_count': len(records), 'records': records,
|
| 144 |
+
'mean_prefill_gflops_counted': float(np.mean([row['prefill_gflops_counted'] for row in records])),
|
| 145 |
+
'mean_decode_one_token_gflops_counted': float(np.mean([row['decode_one_token_gflops_counted'] for row in records])),
|
| 146 |
+
'scope': 'HF reference execution on the A800 with real task inputs; dense products/convolutions/supported attention counted with multiply-add=2 FLOPs. vLLM custom kernels, elementwise/normalization/nonlinearities and other unsupported operations are not complete.',
|
| 147 |
+
'answer_generation': 'not performed; one known-token decode probe only',
|
| 148 |
+
}, ensure_ascii=False, indent=2), encoding='utf-8')
|
| 149 |
+
|
| 150 |
+
|
| 151 |
+
if __name__ == '__main__':
|
| 152 |
+
main()
|
paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_large.py
ADDED
|
@@ -0,0 +1,209 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
from __future__ import annotations
|
| 3 |
+
|
| 4 |
+
import argparse
|
| 5 |
+
import json
|
| 6 |
+
import time
|
| 7 |
+
from pathlib import Path
|
| 8 |
+
|
| 9 |
+
import numpy as np
|
| 10 |
+
import jsonschema
|
| 11 |
+
from PIL import Image
|
| 12 |
+
from vllm import LLM, SamplingParams
|
| 13 |
+
from vllm.sampling_params import StructuredOutputsParams
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
EVENTS = [
|
| 17 |
+
'rule_1_ppe_violation',
|
| 18 |
+
'rule_2_fall_protection_violation',
|
| 19 |
+
'rule_3_unprotected_edge_violation',
|
| 20 |
+
'rule_4_excavator_proximity_violation',
|
| 21 |
+
]
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def gref_schema():
|
| 25 |
+
return {
|
| 26 |
+
'type': 'object',
|
| 27 |
+
'properties': {
|
| 28 |
+
'objects': {
|
| 29 |
+
'type': 'array', 'maxItems': 18,
|
| 30 |
+
'items': {
|
| 31 |
+
'type': 'object',
|
| 32 |
+
'properties': {
|
| 33 |
+
'bbox_2d': {'type': 'array', 'items': {'type': 'number', 'minimum': 0, 'maximum': 1000}, 'minItems': 4, 'maxItems': 4},
|
| 34 |
+
'score': {'type': 'number', 'minimum': 0, 'maximum': 1},
|
| 35 |
+
},
|
| 36 |
+
'required': ['bbox_2d', 'score'], 'additionalProperties': False,
|
| 37 |
+
},
|
| 38 |
+
},
|
| 39 |
+
},
|
| 40 |
+
'required': ['objects'], 'additionalProperties': False,
|
| 41 |
+
}
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def construction_schema():
|
| 45 |
+
box = {'type': 'array', 'items': {'type': 'integer', 'minimum': 0, 'maximum': 1000}, 'minItems': 4, 'maxItems': 4}
|
| 46 |
+
max_boxes = [13, 6, 3, 2]
|
| 47 |
+
events = {}
|
| 48 |
+
for event, limit in zip(EVENTS, max_boxes):
|
| 49 |
+
events[event] = {
|
| 50 |
+
'type': 'object',
|
| 51 |
+
'properties': {
|
| 52 |
+
'present': {'type': 'boolean'},
|
| 53 |
+
'reason': {'type': ['string', 'null']},
|
| 54 |
+
'boxes': {'type': 'array', 'items': box, 'maxItems': limit},
|
| 55 |
+
},
|
| 56 |
+
'required': ['present', 'reason', 'boxes'], 'additionalProperties': False,
|
| 57 |
+
}
|
| 58 |
+
return {'type': 'object', 'properties': {'events': {'type': 'object', 'properties': events, 'required': EVENTS, 'additionalProperties': False}}, 'required': ['events'], 'additionalProperties': False}
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
def make_prompt(root: Path, model_kind: str, task: str, row: dict, processor_or_tokenizer):
|
| 62 |
+
prompt = (root / 'outputs/large_model_labels_all_20260909/prompts' / task / f'{model_kind}.txt').read_text(encoding='utf-8').strip()
|
| 63 |
+
if task == 'grefcoco':
|
| 64 |
+
prompt = prompt.replace('{expression}', row['expression'])
|
| 65 |
+
if model_kind == 'Qwen3.8':
|
| 66 |
+
converted = [{'type': 'image'}]
|
| 67 |
+
converted.append({'type': 'text', 'text': '\n'})
|
| 68 |
+
converted.append({'type': 'text', 'text': prompt})
|
| 69 |
+
messages = [{'role': 'user', 'content': converted}]
|
| 70 |
+
return processor_or_tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 71 |
+
message = '(<image>./</image>)\n' + prompt
|
| 72 |
+
return processor_or_tokenizer.apply_chat_template([{'role': 'user', 'content': message}], tokenize=False, add_generation_prompt=True, enable_thinking=False)
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
def main():
|
| 76 |
+
parser = argparse.ArgumentParser()
|
| 77 |
+
parser.add_argument('--root', type=Path, required=True)
|
| 78 |
+
parser.add_argument('--model-kind', choices=['Qwen3.8', 'MiniCPM'], required=True)
|
| 79 |
+
parser.add_argument('--task', choices=['grefcoco', 'construction'], required=True)
|
| 80 |
+
parser.add_argument('--output', type=Path, required=True)
|
| 81 |
+
parser.add_argument('--samples', type=int, default=32)
|
| 82 |
+
parser.add_argument('--batch-size', type=int, default=8)
|
| 83 |
+
parser.add_argument('--gpu-memory-utilization', type=float, default=0.90)
|
| 84 |
+
parser.add_argument('--max-model-len', type=int, default=16384)
|
| 85 |
+
args = parser.parse_args()
|
| 86 |
+
if args.output.exists():
|
| 87 |
+
raise FileExistsError(args.output)
|
| 88 |
+
if args.samples < 3 or args.batch_size < 1:
|
| 89 |
+
raise ValueError('samples must include warmup rows and batch size must be positive')
|
| 90 |
+
expert = 'instancevg' if args.task == 'grefcoco' else 'yolo26x'
|
| 91 |
+
rows = [json.loads(line) for line in (args.root / f'outputs/router_full_20260910/data/{expert}/test/records.jsonl').open(encoding='utf-8')]
|
| 92 |
+
if args.samples > len(rows):
|
| 93 |
+
raise ValueError(f'samples {args.samples} exceeds {len(rows)}')
|
| 94 |
+
selected_indices = np.random.default_rng(20260911).choice(len(rows), args.samples, replace=False)
|
| 95 |
+
selected = [rows[int(index)] for index in selected_indices]
|
| 96 |
+
checkpoint_root = args.root.parents[1] / 'checkpoints'
|
| 97 |
+
if not checkpoint_root.exists():
|
| 98 |
+
checkpoint_root = args.root / 'checkpoints'
|
| 99 |
+
model_path = checkpoint_root / ('Qwen3.8-27B-FP8' if args.model_kind == 'Qwen3.8' else 'MiniCPM-V-4_5')
|
| 100 |
+
if args.model_kind == 'Qwen3.8':
|
| 101 |
+
from transformers import AutoProcessor
|
| 102 |
+
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True, local_files_only=True)
|
| 103 |
+
model_kwargs = {'mm_processor_kwargs': {'size': {'shortest_edge': 65536, 'longest_edge': 1003520}}}
|
| 104 |
+
else:
|
| 105 |
+
from transformers import AutoTokenizer
|
| 106 |
+
processor = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, local_files_only=True)
|
| 107 |
+
model_kwargs = {}
|
| 108 |
+
model = LLM(model=str(model_path), trust_remote_code=True, tensor_parallel_size=1, max_model_len=args.max_model_len,
|
| 109 |
+
limit_mm_per_prompt={'image': 1}, max_num_seqs=args.batch_size,
|
| 110 |
+
gpu_memory_utilization=args.gpu_memory_utilization, enforce_eager=True, **model_kwargs)
|
| 111 |
+
schema = gref_schema() if args.task == 'grefcoco' else construction_schema()
|
| 112 |
+
sampling = SamplingParams(temperature=0, max_tokens=512 if args.task == 'grefcoco' else 768,
|
| 113 |
+
structured_outputs=StructuredOutputsParams(json=schema, disable_additional_properties=True))
|
| 114 |
+
|
| 115 |
+
def generate(batch_rows):
|
| 116 |
+
images = []
|
| 117 |
+
requests = []
|
| 118 |
+
for row in batch_rows:
|
| 119 |
+
required = {'sample_id', 'image_path'}
|
| 120 |
+
if args.task == 'grefcoco':
|
| 121 |
+
required.add('expression')
|
| 122 |
+
missing = required - row.keys()
|
| 123 |
+
if missing:
|
| 124 |
+
raise ValueError(f'missing fields {sorted(missing)} for {row["sample_id"]}')
|
| 125 |
+
image = Image.open(row['image_path']).convert('RGB')
|
| 126 |
+
images.append(image)
|
| 127 |
+
requests.append({'prompt': make_prompt(args.root, args.model_kind, args.task, row, processor), 'multi_modal_data': {'image': image}})
|
| 128 |
+
started = time.perf_counter()
|
| 129 |
+
generated = model.generate(requests, sampling, use_tqdm=False)
|
| 130 |
+
elapsed = time.perf_counter() - started
|
| 131 |
+
for image in images:
|
| 132 |
+
image.close()
|
| 133 |
+
if len(generated) != len(batch_rows):
|
| 134 |
+
raise ValueError('vLLM returned an unexpected number of results')
|
| 135 |
+
return elapsed, generated
|
| 136 |
+
|
| 137 |
+
warmup_count = min(2, len(selected) - 1)
|
| 138 |
+
for row in selected[:warmup_count]:
|
| 139 |
+
generate([row])
|
| 140 |
+
measured = selected[warmup_count:]
|
| 141 |
+
batches = []
|
| 142 |
+
for start in range(0, len(measured), args.batch_size):
|
| 143 |
+
batch_rows = measured[start:start + args.batch_size]
|
| 144 |
+
batch_started = time.perf_counter()
|
| 145 |
+
elapsed, generated = generate(batch_rows)
|
| 146 |
+
wall = time.perf_counter() - batch_started
|
| 147 |
+
output_records = []
|
| 148 |
+
for result in generated:
|
| 149 |
+
raw = result.outputs[0].text
|
| 150 |
+
error = None
|
| 151 |
+
try:
|
| 152 |
+
parsed = json.loads(raw)
|
| 153 |
+
jsonschema.validate(parsed, schema)
|
| 154 |
+
boxes = [o['bbox_2d'] for o in parsed['objects']] if args.task == 'grefcoco' else [b for e in parsed['events'].values() for b in e['boxes']]
|
| 155 |
+
if any(b[0] >= b[2] or b[1] >= b[3] for b in boxes):
|
| 156 |
+
raise ValueError('non-positive box size')
|
| 157 |
+
except (json.JSONDecodeError, jsonschema.ValidationError, ValueError) as exc:
|
| 158 |
+
parsed = None
|
| 159 |
+
error = str(exc)
|
| 160 |
+
output_records.append({'raw_response':raw,'valid_prediction':parsed,'error':error,
|
| 161 |
+
'finish_reason':result.outputs[0].finish_reason,'completion_tokens':len(result.outputs[0].token_ids),
|
| 162 |
+
'prompt_tokens':len(result.prompt_token_ids)})
|
| 163 |
+
item = {
|
| 164 |
+
'sample_ids': [row['sample_id'] for row in batch_rows],
|
| 165 |
+
'batch_size': len(batch_rows),
|
| 166 |
+
'generation_latency_seconds': elapsed,
|
| 167 |
+
'batch_wall_latency_seconds': wall,
|
| 168 |
+
'amortized_generation_ms_per_sample': elapsed * 1000 / len(batch_rows),
|
| 169 |
+
'amortized_batch_wall_ms_per_sample': wall * 1000 / len(batch_rows),
|
| 170 |
+
'outputs': [
|
| 171 |
+
{
|
| 172 |
+
'finish_reason': result.outputs[0].finish_reason,
|
| 173 |
+
'stop_reason': result.outputs[0].stop_reason,
|
| 174 |
+
'prompt_tokens': len(result.prompt_token_ids) if result.prompt_token_ids is not None else None,
|
| 175 |
+
'completion_tokens': len(result.outputs[0].token_ids),
|
| 176 |
+
'text_characters': len(result.outputs[0].text),
|
| 177 |
+
}
|
| 178 |
+
for result in generated
|
| 179 |
+
],
|
| 180 |
+
}
|
| 181 |
+
batches.append(item)
|
| 182 |
+
item['outputs'] = output_records
|
| 183 |
+
args.output.parent.mkdir(parents=True, exist_ok=True)
|
| 184 |
+
args.output.write_text(json.dumps({
|
| 185 |
+
'status': 'running', 'model_kind': args.model_kind, 'task': args.task, 'model_path': str(model_path),
|
| 186 |
+
'sample_count_requested': args.samples, 'warmup_count': warmup_count, 'batch_size': args.batch_size,
|
| 187 |
+
'sample_ids': [row['sample_id'] for row in selected], 'batches': batches,
|
| 188 |
+
}, ensure_ascii=False, indent=2), encoding='utf-8')
|
| 189 |
+
print(json.dumps({'completed_samples': sum(item['batch_size'] for item in batches), 'total_measured': len(measured), 'batch_latency_seconds': elapsed}), flush=True)
|
| 190 |
+
all_ms = [x['amortized_generation_ms_per_sample'] for x in batches]
|
| 191 |
+
all_wall_ms = [x['amortized_batch_wall_ms_per_sample'] for x in batches]
|
| 192 |
+
output = {
|
| 193 |
+
'status': 'complete', 'model_kind': args.model_kind, 'task': args.task, 'expert': expert,
|
| 194 |
+
'model_path': str(model_path), 'device': 'cuda:0', 'gpu_memory_utilization': args.gpu_memory_utilization,
|
| 195 |
+
'sample_count_requested': args.samples, 'warmup_count': warmup_count, 'measured_count': len(measured),
|
| 196 |
+
'batch_size': args.batch_size, 'sample_ids': [row['sample_id'] for row in selected], 'batches': batches,
|
| 197 |
+
'mean_generation_ms_per_sample': float(np.mean(all_ms)), 'median_generation_ms_per_sample': float(np.median(all_ms)),
|
| 198 |
+
'mean_batch_wall_ms_per_sample': sum(b['batch_wall_latency_seconds'] for b in batches)*1000/len(measured), 'median_batch_wall_ms_per_sample': float(np.median(all_wall_ms)),
|
| 199 |
+
'mean_prompt_tokens': float(np.mean([x['prompt_tokens'] for batch in batches for x in batch['outputs'] if x['prompt_tokens'] is not None])),
|
| 200 |
+
'mean_completion_tokens': float(np.mean([x['completion_tokens'] for batch in batches for x in batch['outputs']])),
|
| 201 |
+
'latency_scope': 'A800 vLLM-0.18.0 native generation latency with the frozen task prompt and structured-output schema; model loading and warmup excluded. Batch wall time includes the real request/image preparation path.',
|
| 202 |
+
'flops_scope': 'This file measures latency only. Counted FLOPs are measured separately by the HF reference probe and are not claimed to be complete vLLM kernel FLOPs.',
|
| 203 |
+
}
|
| 204 |
+
args.output.write_text(json.dumps(output, ensure_ascii=False, indent=2), encoding='utf-8')
|
| 205 |
+
print(json.dumps({'status': 'complete', 'mean_generation_ms_per_sample': output['mean_generation_ms_per_sample'], 'mean_completion_tokens': output['mean_completion_tokens']}), flush=True)
|
| 206 |
+
|
| 207 |
+
|
| 208 |
+
if __name__ == '__main__':
|
| 209 |
+
main()
|
paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_router.py
ADDED
|
@@ -0,0 +1,104 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from pathlib import Path
|
| 2 |
+
import argparse
|
| 3 |
+
import json
|
| 4 |
+
import sys
|
| 5 |
+
import time
|
| 6 |
+
import numpy as np
|
| 7 |
+
import torch
|
| 8 |
+
from PIL import Image
|
| 9 |
+
from torch.utils.flop_counter import FlopCounterMode
|
| 10 |
+
|
| 11 |
+
p=argparse.ArgumentParser()
|
| 12 |
+
p.add_argument('--root',type=Path,required=True)
|
| 13 |
+
p.add_argument('--expert',required=True)
|
| 14 |
+
p.add_argument('--endpoint',required=True)
|
| 15 |
+
p.add_argument('--output',type=Path,required=True)
|
| 16 |
+
a=p.parse_args()
|
| 17 |
+
sys.path.insert(0,str(a.root/'code'))
|
| 18 |
+
from routing import train_router_formal as formal
|
| 19 |
+
from routing.router import prepare_images,load_distilled_router
|
| 20 |
+
from siglip_distillation.mini_siglip import MiniSiglip
|
| 21 |
+
from tokenizers import Tokenizer
|
| 22 |
+
torch.set_num_threads(4)
|
| 23 |
+
torch.backends.cuda.matmul.allow_tf32=False
|
| 24 |
+
torch.backends.mha.set_fastpath_enabled(False)
|
| 25 |
+
work=a.root/'outputs/router_shared_image_hidden_20260910'
|
| 26 |
+
run=f'shared_12_{a.expert}_{a.endpoint}'
|
| 27 |
+
dest=work/'runs'/run
|
| 28 |
+
cfg=json.loads((dest/'config.json').read_text())
|
| 29 |
+
assert cfg['seed']==42 and cfg['student']=='8M' and cfg['target']=='four_state'
|
| 30 |
+
records,data=formal.load_data(cfg,'test',torch.device('cuda'))
|
| 31 |
+
for key in ['hidden','confidence']:
|
| 32 |
+
norm=np.load(dest/(key+'_normalization.npz'))
|
| 33 |
+
data[key]=(data[key]-torch.from_numpy(norm['mean']).cuda())/torch.from_numpy(norm['std']).cuda()
|
| 34 |
+
model=formal.FeatureRouter(cfg,data['hidden'].shape[1],data['confidence'].shape[1],data['output'].shape[1]).cuda()
|
| 35 |
+
ckpt=torch.load(dest/'best.pt',map_location='cuda',weights_only=True)
|
| 36 |
+
model.load_state_dict(ckpt['state_dict'],strict=True)
|
| 37 |
+
model.eval().requires_grad_(False)
|
| 38 |
+
with torch.inference_mode():
|
| 39 |
+
raw=torch.cat([model(data,torch.arange(i,min(i+8192,len(records)),device='cuda')).cpu() for i in range(0,len(records),8192)]).numpy()
|
| 40 |
+
scores=raw[:,1]-raw[:,2]
|
| 41 |
+
saved=np.load(work/'test_evaluation'/run/'paired_test_arrays.npz')['scores']
|
| 42 |
+
reference=json.loads((work/'test_results.json').read_text())
|
| 43 |
+
ref=next(g for g in reference['groups'] if g['id']==run)
|
| 44 |
+
point=ref['best_test_point']
|
| 45 |
+
threshold=point['threshold']
|
| 46 |
+
mask=scores>=threshold if point['operator']=='ge' else scores>threshold
|
| 47 |
+
saved_mask=saved>=threshold if point['operator']=='ge' else saved>threshold
|
| 48 |
+
assert np.array_equal(mask,saved_mask), 'Recomputed selected decisions differ from main-table archive'
|
| 49 |
+
indices=np.random.default_rng(20260912).choice(len(records),64,replace=False)
|
| 50 |
+
encoder_dir=a.root/'outputs/mini_siglip_strict_20260910_300ep/8M'
|
| 51 |
+
checkpoint=torch.load(encoder_dir/'epoch_300.pt',map_location='cpu',weights_only=True)
|
| 52 |
+
encoder=MiniSiglip(**checkpoint['model_config']).cuda()
|
| 53 |
+
encoder.load_state_dict(checkpoint['model'],strict=True)
|
| 54 |
+
encoder.eval().requires_grad_(False)
|
| 55 |
+
tokenizer=Tokenizer.from_file(str(encoder_dir/'student_tokenizer.json'))
|
| 56 |
+
|
| 57 |
+
def prepare(i):
|
| 58 |
+
with Image.open(records[i]['image_path']) as im:
|
| 59 |
+
pixels,valid=prepare_images([[im]])
|
| 60 |
+
pixels=pixels.flatten(0,1).cuda()
|
| 61 |
+
tokens=torch.tensor([tokenizer.encode(records[i]['expression']).ids],device='cuda') if 'text' in cfg['branches'] else None
|
| 62 |
+
return pixels,valid.cuda(),tokens
|
| 63 |
+
|
| 64 |
+
def forward(i,prepared):
|
| 65 |
+
pixels,valid,tokens=prepared
|
| 66 |
+
with torch.autocast('cuda',dtype=torch.bfloat16):
|
| 67 |
+
image=encoder.encode_image(pixels).to(torch.float16).float().reshape(1,2,768)
|
| 68 |
+
text=encoder.encode_text(tokens).to(torch.float16).float() if tokens is not None else None
|
| 69 |
+
row={k:v[i:i+1] for k,v in data.items()}
|
| 70 |
+
row['image']=image;row['valid']=valid
|
| 71 |
+
if text is not None:row['text']=text
|
| 72 |
+
return model(row,torch.zeros(1,device='cuda',dtype=torch.long))
|
| 73 |
+
|
| 74 |
+
times=[];head_times=[];flops=[];ops=[]
|
| 75 |
+
with torch.inference_mode():
|
| 76 |
+
for i in indices[:4]:
|
| 77 |
+
forward(int(i),prepare(int(i)))
|
| 78 |
+
torch.cuda.synchronize()
|
| 79 |
+
for i in indices:
|
| 80 |
+
i=int(i)
|
| 81 |
+
started=time.perf_counter()
|
| 82 |
+
forward(i,prepare(i))
|
| 83 |
+
torch.cuda.synchronize()
|
| 84 |
+
times.append((time.perf_counter()-started)*1000)
|
| 85 |
+
start=torch.cuda.Event(enable_timing=True);end=torch.cuda.Event(enable_timing=True)
|
| 86 |
+
idx=torch.tensor([i],device='cuda')
|
| 87 |
+
start.record();model(data,idx);end.record();torch.cuda.synchronize()
|
| 88 |
+
head_times.append(start.elapsed_time(end))
|
| 89 |
+
for i in indices[:8]:
|
| 90 |
+
prepared=prepare(int(i))
|
| 91 |
+
with FlopCounterMode(display=False) as counter:
|
| 92 |
+
forward(int(i),prepared)
|
| 93 |
+
flops.append(counter.get_total_flops()/1e9)
|
| 94 |
+
ops.append({str(k):int(v) for k,v in counter.get_flop_counts().get('Global',{}).items()})
|
| 95 |
+
report={'status':'complete','run':run,'config':cfg,'checkpoint':str(dest/'best.pt'),
|
| 96 |
+
'score':'logit_rescue-logit_harm','test_rows_recomputed':len(records),'selected_decisions_match_main':True,
|
| 97 |
+
'max_score_abs_difference':float(np.max(np.abs(saved-scores))),'actual_call_fraction':float(mask.mean()),
|
| 98 |
+
'sample_count':len(indices),'sample_ids':[records[int(i)]['sample_id'] for i in indices],
|
| 99 |
+
'batch_size':1,'latency_ms':times,'mean_latency_ms_per_input':float(np.mean(times)),
|
| 100 |
+
'head_mean_latency_ms':float(np.mean(head_times)), 'gflops_per_input':flops,'mean_gflops_per_input':float(np.mean(flops)),
|
| 101 |
+
'operators':ops,'scope':'image loading, bilinear resize, tokenize when used, two image slots (including gray placeholder), active distilled encoders, feature assembly and original routing head; cached specialist features; supported FLOPs only'}
|
| 102 |
+
a.output.parent.mkdir(parents=True,exist_ok=True)
|
| 103 |
+
a.output.write_text(json.dumps(report,indent=2),encoding='utf-8')
|
| 104 |
+
print(json.dumps({k:v for k,v in report.items() if k in ['run','actual_call_fraction','selected_decisions_match_main','mean_latency_ms_per_input','mean_gflops_per_input']},indent=2))
|
paper_results/data/main_table_cost_qwen_20260912/reproducible/measure_specialist.py
ADDED
|
@@ -0,0 +1,258 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
from __future__ import annotations
|
| 3 |
+
|
| 4 |
+
import argparse
|
| 5 |
+
import importlib.util
|
| 6 |
+
import json
|
| 7 |
+
import sys
|
| 8 |
+
import time
|
| 9 |
+
from pathlib import Path
|
| 10 |
+
|
| 11 |
+
import numpy as np
|
| 12 |
+
import torch
|
| 13 |
+
from torch.utils.flop_counter import FlopCounterMode
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
def timed_cuda(call):
|
| 17 |
+
start = torch.cuda.Event(enable_timing=True)
|
| 18 |
+
end = torch.cuda.Event(enable_timing=True)
|
| 19 |
+
start.record()
|
| 20 |
+
value = call()
|
| 21 |
+
end.record()
|
| 22 |
+
torch.cuda.synchronize()
|
| 23 |
+
return float(start.elapsed_time(end)), value
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
def count_flops(call):
|
| 27 |
+
with torch.inference_mode(), FlopCounterMode(display=False) as counter:
|
| 28 |
+
value = call()
|
| 29 |
+
torch.cuda.synchronize()
|
| 30 |
+
global_counts = counter.get_flop_counts().get('Global', {})
|
| 31 |
+
return int(counter.get_total_flops()), value, {str(k): int(v) for k, v in global_counts.items()}
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def instancevg_measure(root: Path, output: Path, samples: int, batch_size: int, device: str) -> None:
|
| 35 |
+
import mmcv
|
| 36 |
+
from mmcv import Config
|
| 37 |
+
from transformers import XLMRobertaTokenizer
|
| 38 |
+
from instancevg.models import build_model
|
| 39 |
+
from instancevg.datasets.pipelines.transforms import Normalize, Resize
|
| 40 |
+
|
| 41 |
+
torch.set_num_threads(4)
|
| 42 |
+
torch.backends.cuda.matmul.allow_tf32 = False
|
| 43 |
+
torch.backends.cudnn.allow_tf32 = False
|
| 44 |
+
cfg = Config.fromfile(str(root / 'code/third_party/InstanceVG/configs/gres/InstanceVG-grefcoco.py'))
|
| 45 |
+
cfg.model.vis_enc.pretrain = None
|
| 46 |
+
cfg.model.process_visual = False
|
| 47 |
+
model = build_model(cfg.model)
|
| 48 |
+
checkpoint_path = root / 'models/grefcoco/instancevg/InstanceVG-grefcoco.pth'
|
| 49 |
+
checkpoint = torch.load(checkpoint_path, map_location='cpu')
|
| 50 |
+
state = checkpoint['state_dict']
|
| 51 |
+
if all(key.startswith('module.') for key in state):
|
| 52 |
+
state = {key[7:]: value for key, value in state.items()}
|
| 53 |
+
model.load_state_dict(state, strict=True)
|
| 54 |
+
del state, checkpoint
|
| 55 |
+
model.eval().to(device).requires_grad_(False)
|
| 56 |
+
tokenizer_path = root / 'models/nlvr2/Raghavan--beit3_base_patch16_224_nlvr2/sentencepiece.bpe.model'
|
| 57 |
+
tokenizer = XLMRobertaTokenizer(str(tokenizer_path))
|
| 58 |
+
resize = Resize(img_scale=(320, 320), keep_ratio=False)
|
| 59 |
+
normalize = Normalize(mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375])
|
| 60 |
+
rows = [json.loads(line) for line in (root / 'outputs/router_full_20260910/data/instancevg/test/records.jsonl').open(encoding='utf-8')]
|
| 61 |
+
if samples > len(rows):
|
| 62 |
+
raise ValueError(f'samples {samples} exceeds {len(rows)}')
|
| 63 |
+
indices = np.random.default_rng(20260911).choice(len(rows), samples, replace=False)
|
| 64 |
+
chosen = [rows[int(index)] for index in indices]
|
| 65 |
+
|
| 66 |
+
def prepare(batch_rows):
|
| 67 |
+
images, metas, tokens, masks = [], [], [], []
|
| 68 |
+
for row in batch_rows:
|
| 69 |
+
image = mmcv.imread(row['image_path'])
|
| 70 |
+
if image.shape[:2] != (row['height'], row['width']):
|
| 71 |
+
raise ValueError(f'image shape mismatch for {row["sample_id"]}')
|
| 72 |
+
expression = row['expression']
|
| 73 |
+
expression = __import__('re').sub(r"([.,'!?\"()*#:;])", '', expression.lower()).replace('-', ' ').replace('/', ' ')
|
| 74 |
+
token_ids = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(expression))
|
| 75 |
+
if not token_ids:
|
| 76 |
+
raise ValueError(f'empty token sequence for {row["sample_id"]}')
|
| 77 |
+
token_ids = [tokenizer.bos_token_id] + token_ids[:48] + [tokenizer.eos_token_id]
|
| 78 |
+
if len(token_ids) > 50:
|
| 79 |
+
raise ValueError(f'token sequence exceeds fixed length for {row["sample_id"]}')
|
| 80 |
+
masks.append([0] * len(token_ids) + [1] * (50 - len(token_ids)))
|
| 81 |
+
tokens.append(token_ids + [tokenizer.pad_token_id] * (50 - len(token_ids)))
|
| 82 |
+
data = dict(img=image, filename=row['image_path'], img_shape=image.shape, ori_shape=image.shape,
|
| 83 |
+
expression=expression, empty=None, with_bbox=False, with_mask=False)
|
| 84 |
+
data = normalize(resize(data))
|
| 85 |
+
images.append(torch.from_numpy(np.ascontiguousarray(data['img'].transpose(2, 0, 1))))
|
| 86 |
+
metas.append({key: data[key] for key in ['filename', 'expression', 'ori_shape', 'img_shape', 'pad_shape', 'scale_factor', 'empty']})
|
| 87 |
+
image_tensor = torch.stack(images).to(device)
|
| 88 |
+
token_tensor = torch.tensor(tokens, device=device)
|
| 89 |
+
mask_tensor = torch.tensor(masks, device=device)
|
| 90 |
+
|
| 91 |
+
def forward():
|
| 92 |
+
return model(img=image_tensor, ref_expr_inds=token_tensor, text_attention_mask=mask_tensor,
|
| 93 |
+
img_metas=metas, return_loss=False, rescale=True, with_bbox=True, with_mask=False)
|
| 94 |
+
|
| 95 |
+
return forward
|
| 96 |
+
|
| 97 |
+
batches = [chosen[start:start + batch_size] for start in range(0, len(chosen), batch_size)]
|
| 98 |
+
prepared = [prepare(batch) for batch in batches]
|
| 99 |
+
for forward in prepared[:2]:
|
| 100 |
+
forward()
|
| 101 |
+
torch.cuda.synchronize()
|
| 102 |
+
forward_ms = []
|
| 103 |
+
end_to_end_ms = []
|
| 104 |
+
for batch in batches:
|
| 105 |
+
forward = prepare(batch)
|
| 106 |
+
elapsed, _ = timed_cuda(forward)
|
| 107 |
+
forward_ms.append(elapsed / len(batch))
|
| 108 |
+
started = time.perf_counter()
|
| 109 |
+
forward = prepare(batch)
|
| 110 |
+
forward()
|
| 111 |
+
torch.cuda.synchronize()
|
| 112 |
+
end_to_end_ms.append((time.perf_counter() - started) * 1000 / len(batch))
|
| 113 |
+
flop_rows = []
|
| 114 |
+
for batch in batches[:min(4, len(batches))]:
|
| 115 |
+
forward = prepare(batch)
|
| 116 |
+
total, _, operators = count_flops(forward)
|
| 117 |
+
flop_rows.append({'batch_size': len(batch), 'flops_counted': total, 'gflops_counted_per_sample': total / len(batch) / 1e9, 'operators': operators})
|
| 118 |
+
output.write_text(json.dumps({
|
| 119 |
+
'component': 'specialist', 'expert': 'instancevg', 'task': 'grefcoco', 'device': device,
|
| 120 |
+
'checkpoint': str(checkpoint_path), 'sample_count': len(chosen), 'batch_size': batch_size,
|
| 121 |
+
'sample_ids': [row['sample_id'] for row in chosen],
|
| 122 |
+
'model_load_seconds_excluded_from_latency': True,
|
| 123 |
+
'forward_latency_ms_per_sample': forward_ms,
|
| 124 |
+
'end_to_end_latency_ms_per_sample': end_to_end_ms,
|
| 125 |
+
'mean_forward_latency_ms_per_sample': float(np.mean(forward_ms)),
|
| 126 |
+
'median_forward_latency_ms_per_sample': float(np.median(forward_ms)),
|
| 127 |
+
'mean_end_to_end_latency_ms_per_sample': float(np.mean(end_to_end_ms)),
|
| 128 |
+
'median_end_to_end_latency_ms_per_sample': float(np.median(end_to_end_ms)),
|
| 129 |
+
'flop_profiles': flop_rows,
|
| 130 |
+
'mean_counted_gflops_per_sample': float(np.mean([row['gflops_counted_per_sample'] for row in flop_rows])),
|
| 131 |
+
'flops_scope': 'GPU FlopCounterMode dense products, convolutions and supported attention; multiply-add counted as 2 FLOPs. Elementwise, normalization, custom ops and postprocessing may be missing.',
|
| 132 |
+
'latency_scope': 'A800 wall-clock CUDA forward latency and end-to-end latency including image read/preprocess, excluding model load; batch processing uses the recorded batch size.',
|
| 133 |
+
'completed': True,
|
| 134 |
+
}, ensure_ascii=False, indent=2))
|
| 135 |
+
|
| 136 |
+
|
| 137 |
+
def yolo_measure(root: Path, output: Path, samples: int, batch_size: int, device: str) -> None:
|
| 138 |
+
from ultralytics import YOLO
|
| 139 |
+
|
| 140 |
+
torch.set_num_threads(4)
|
| 141 |
+
torch.backends.cuda.matmul.allow_tf32 = False
|
| 142 |
+
torch.backends.cudnn.allow_tf32 = False
|
| 143 |
+
checkpoint_path = root / 'models/construction/yolo26x.pt'
|
| 144 |
+
model = YOLO(str(checkpoint_path))
|
| 145 |
+
rows = [json.loads(line) for line in (root / 'outputs/router_full_20260910/data/yolo26x/test/records.jsonl').open(encoding='utf-8')]
|
| 146 |
+
if samples > len(rows):
|
| 147 |
+
raise ValueError(f'samples {samples} exceeds {len(rows)}')
|
| 148 |
+
indices = np.random.default_rng(20260911).choice(len(rows), samples, replace=False)
|
| 149 |
+
chosen = [rows[int(index)] for index in indices]
|
| 150 |
+
batches = [chosen[start:start + batch_size] for start in range(0, len(chosen), batch_size)]
|
| 151 |
+
|
| 152 |
+
def predict(batch_rows):
|
| 153 |
+
return model.predict(source=[row['image_path'] for row in batch_rows], imgsz=640, batch=len(batch_rows),
|
| 154 |
+
device=device, half=False, conf=0.001, iou=0.7, max_det=300, save=False, verbose=False)
|
| 155 |
+
|
| 156 |
+
for batch in batches[:2]:
|
| 157 |
+
predict(batch)
|
| 158 |
+
torch.cuda.synchronize()
|
| 159 |
+
forward_ms, end_to_end_ms = [], []
|
| 160 |
+
for batch in batches:
|
| 161 |
+
elapsed, _ = timed_cuda(lambda batch=batch: predict(batch))
|
| 162 |
+
forward_ms.append(elapsed / len(batch))
|
| 163 |
+
started = time.perf_counter()
|
| 164 |
+
predict(batch)
|
| 165 |
+
torch.cuda.synchronize()
|
| 166 |
+
end_to_end_ms.append((time.perf_counter() - started) * 1000 / len(batch))
|
| 167 |
+
flop_rows = []
|
| 168 |
+
for batch in batches[:min(4, len(batches))]:
|
| 169 |
+
total, _, operators = count_flops(lambda batch=batch: predict(batch))
|
| 170 |
+
flop_rows.append({'batch_size': len(batch), 'flops_counted': total, 'gflops_counted_per_sample': total / len(batch) / 1e9, 'operators': operators})
|
| 171 |
+
output.write_text(json.dumps({
|
| 172 |
+
'component': 'specialist', 'expert': 'yolo26x', 'task': 'construction', 'device': device,
|
| 173 |
+
'checkpoint': str(checkpoint_path), 'sample_count': len(chosen), 'batch_size': batch_size,
|
| 174 |
+
'sample_ids': [row['sample_id'] for row in chosen],
|
| 175 |
+
'model_load_seconds_excluded_from_latency': True,
|
| 176 |
+
'forward_latency_ms_per_sample': forward_ms,
|
| 177 |
+
'end_to_end_latency_ms_per_sample': end_to_end_ms,
|
| 178 |
+
'mean_forward_latency_ms_per_sample': float(np.mean(forward_ms)),
|
| 179 |
+
'median_forward_latency_ms_per_sample': float(np.median(forward_ms)),
|
| 180 |
+
'mean_end_to_end_latency_ms_per_sample': float(np.mean(end_to_end_ms)),
|
| 181 |
+
'median_end_to_end_latency_ms_per_sample': float(np.median(end_to_end_ms)),
|
| 182 |
+
'flop_profiles': flop_rows,
|
| 183 |
+
'mean_counted_gflops_per_sample': float(np.mean([row['gflops_counted_per_sample'] for row in flop_rows])),
|
| 184 |
+
'flops_scope': 'GPU FlopCounterMode around the real Ultralytics predict path; multiply-add counted as 2 FLOPs. Elementwise, postprocessing and unsupported custom ops may be missing.',
|
| 185 |
+
'latency_scope': 'A800 wall-clock latency for real Ultralytics prediction including image loading/preprocess, forward and result postprocessing, excluding model load; batch processing uses the recorded batch size.',
|
| 186 |
+
'completed': True,
|
| 187 |
+
}, ensure_ascii=False, indent=2))
|
| 188 |
+
|
| 189 |
+
|
| 190 |
+
def router_measure(root: Path, output: Path, expert: str, endpoint: str, condition: str, seed: int, samples: int, batch_size: int, device: str) -> None:
|
| 191 |
+
work = root / 'outputs/router_method_improvement_20260911_r1/reproducible'
|
| 192 |
+
sys.path.insert(0, str(work))
|
| 193 |
+
common = __import__('method_improvement_common')
|
| 194 |
+
spec = json.loads((root / 'outputs/router_method_improvement_20260911_r1/method_improvement_spec.json').read_text(encoding='utf-8'))
|
| 195 |
+
trajectory_id = f'{expert}_{endpoint}_seed{seed}'
|
| 196 |
+
cfg = common.make_config(spec, expert, endpoint, seed, trajectory_id)
|
| 197 |
+
rows, data = common.load_split(root, spec, cfg, 'test', device)
|
| 198 |
+
checkpoint_path = root / f'outputs/router_method_improvement_20260911_r1/conditions/{condition}_{expert}_{endpoint}_seed{seed}/best.pt'
|
| 199 |
+
checkpoint = torch.load(checkpoint_path, map_location=device, weights_only=True)
|
| 200 |
+
model = common.FeatureRouter(cfg, data['hidden'].shape[1], data['confidence'].shape[1], data['output'].shape[1]).to(device)
|
| 201 |
+
model.load_state_dict(checkpoint['state_dict'], strict=True)
|
| 202 |
+
model.eval().requires_grad_(False)
|
| 203 |
+
if samples > len(rows):
|
| 204 |
+
raise ValueError(f'samples {samples} exceeds {len(rows)}')
|
| 205 |
+
indices = np.random.default_rng(20260911).choice(len(rows), samples, replace=False)
|
| 206 |
+
chunks = [torch.as_tensor(indices[start:start + batch_size], device=device, dtype=torch.long) for start in range(0, samples, batch_size)]
|
| 207 |
+
for index in chunks[:2]:
|
| 208 |
+
model(data, index)
|
| 209 |
+
torch.cuda.synchronize()
|
| 210 |
+
latency = []
|
| 211 |
+
for index in chunks:
|
| 212 |
+
elapsed, _ = timed_cuda(lambda index=index: model(data, index))
|
| 213 |
+
latency.append(elapsed / len(index))
|
| 214 |
+
index = chunks[0]
|
| 215 |
+
total, _, operators = count_flops(lambda: model(data, index))
|
| 216 |
+
output.write_text(json.dumps({
|
| 217 |
+
'component': 'router', 'expert': expert, 'task': cfg['task'], 'endpoint': endpoint,
|
| 218 |
+
'condition': condition, 'seed': seed, 'device': device, 'checkpoint': str(checkpoint_path),
|
| 219 |
+
'sample_count': samples, 'batch_size': batch_size, 'row_count_loaded': len(rows),
|
| 220 |
+
'mean_latency_ms_per_sample': float(np.mean(latency)), 'median_latency_ms_per_sample': float(np.median(latency)),
|
| 221 |
+
'latency_ms_per_sample': latency, 'flops_counted_per_batch': total,
|
| 222 |
+
'gflops_counted_per_sample': total / len(index) / 1e9, 'operators': operators,
|
| 223 |
+
'flops_scope': 'GPU FlopCounterMode for the router forward; multiply-add counted as 2 FLOPs.',
|
| 224 |
+
'latency_scope': 'A800 CUDA forward latency on cached router features, excluding feature loading and model load.',
|
| 225 |
+
'completed': True,
|
| 226 |
+
}, ensure_ascii=False, indent=2))
|
| 227 |
+
|
| 228 |
+
|
| 229 |
+
def main() -> None:
|
| 230 |
+
parser = argparse.ArgumentParser()
|
| 231 |
+
parser.add_argument('--root', type=Path, required=True)
|
| 232 |
+
parser.add_argument('--output', type=Path, required=True)
|
| 233 |
+
parser.add_argument('--component', choices=['specialist', 'router'], required=True)
|
| 234 |
+
parser.add_argument('--expert', choices=['instancevg', 'yolo26x'], required=True)
|
| 235 |
+
parser.add_argument('--endpoint', choices=['MiniCPM', 'Qwen3.8'])
|
| 236 |
+
parser.add_argument('--condition', default='both')
|
| 237 |
+
parser.add_argument('--seed', type=int, default=42)
|
| 238 |
+
parser.add_argument('--samples', type=int, default=64)
|
| 239 |
+
parser.add_argument('--batch-size', type=int, default=8)
|
| 240 |
+
parser.add_argument('--device', default='cuda:0')
|
| 241 |
+
args = parser.parse_args()
|
| 242 |
+
if args.output.exists():
|
| 243 |
+
raise FileExistsError(args.output)
|
| 244 |
+
args.output.parent.mkdir(parents=True, exist_ok=True)
|
| 245 |
+
if not torch.cuda.is_available():
|
| 246 |
+
raise RuntimeError('CUDA is unavailable')
|
| 247 |
+
if args.component == 'specialist' and args.expert == 'instancevg':
|
| 248 |
+
instancevg_measure(args.root, args.output, args.samples, args.batch_size, args.device)
|
| 249 |
+
elif args.component == 'specialist':
|
| 250 |
+
yolo_measure(args.root, args.output, args.samples, args.batch_size, args.device)
|
| 251 |
+
else:
|
| 252 |
+
if args.endpoint is None:
|
| 253 |
+
raise ValueError('--endpoint is required for router')
|
| 254 |
+
router_measure(args.root, args.output, args.expert, args.endpoint, args.condition, args.seed, args.samples, args.batch_size, args.device)
|
| 255 |
+
|
| 256 |
+
|
| 257 |
+
if __name__ == '__main__':
|
| 258 |
+
main()
|
paper_results/data/main_table_cost_qwen_20260912/reproducible/prepare_output.py
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from pathlib import Path
|
| 2 |
+
import json
|
| 3 |
+
import shutil
|
| 4 |
+
root=Path(__file__).resolve().parents[2]
|
| 5 |
+
out=root/'dse_results/main_table_cost_20260912'
|
| 6 |
+
out.mkdir(exist_ok=True)
|
| 7 |
+
(out/'components').mkdir(exist_ok=True)
|
| 8 |
+
source=root/'paper_materials/draft_20260912_revision19/data/results.json'
|
| 9 |
+
data=json.loads(source.read_text(encoding='utf-8'))
|
| 10 |
+
groups=[g for g in data['groups'] if g['key'] in [['grefcoco','instancevg','MiniCPM'],['construction','yolo26x','Qwen3.8']]]
|
| 11 |
+
assert len(groups)==2
|
| 12 |
+
(out/'main_table_results.json').write_text(json.dumps({'source':str(source),'protocol':data['protocol'],'groups':groups},indent=2),encoding='utf-8')
|
| 13 |
+
scripts=out/'reproducible'
|
| 14 |
+
scripts.mkdir(exist_ok=True)
|
| 15 |
+
for p in Path(__file__).resolve().parent.glob('*.py'):
|
| 16 |
+
shutil.copy2(p,scripts/p.name)
|
| 17 |
+
for p in Path(__file__).resolve().parent.glob('*.sh'):
|
| 18 |
+
shutil.copy2(p,scripts/p.name)
|
| 19 |
+
print(out)
|
paper_results/data/main_table_cost_qwen_20260912/reproducible/read_main.py
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from pathlib import Path
|
| 2 |
+
import json
|
| 3 |
+
root=Path(__file__).resolve().parents[2]
|
| 4 |
+
data=json.loads((root/'paper_materials/draft_20260912_revision19/data/results.json').read_text(encoding='utf-8'))
|
| 5 |
+
print('TOP',list(data))
|
| 6 |
+
for g in data['groups']:
|
| 7 |
+
if g['key'] in [['grefcoco','instancevg','MiniCPM'],['construction','yolo26x','Qwen3.8']]:
|
| 8 |
+
print(json.dumps({k:v for k,v in g.items() if k!='methods'},ensure_ascii=False)[:6000])
|
| 9 |
+
for m,v in g['methods'].items():
|
| 10 |
+
print(m, {k:v1 for k,v1 in v.items() if 'curve' not in k})
|
paper_results/data/main_table_cost_qwen_20260912/reproducible/setup_local.py
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from pathlib import Path
|
| 2 |
+
import shutil
|
| 3 |
+
root=Path(__file__).resolve().parents[2]
|
| 4 |
+
dest=Path(__file__).resolve().parent
|
| 5 |
+
for source,target in [('measure_large_vllm_runtime_20260911.py','measure_large.py'),('measure_large_flops_reference_20260911.py','measure_flops.py'),('measure_expert_router_runtime_20260911.py','measure_specialist.py')]:
|
| 6 |
+
shutil.copy2(root/'tmp'/source,dest/target)
|
paper_results/data/main_table_cost_qwen_20260912/reproducible/switch_qwen.py
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from pathlib import Path
|
| 2 |
+
import json
|
| 3 |
+
import shutil
|
| 4 |
+
root=Path(__file__).resolve().parents[2]
|
| 5 |
+
old=root/'dse_results/main_table_cost_20260912'
|
| 6 |
+
out=root/'dse_results/main_table_cost_qwen_20260912'
|
| 7 |
+
out.mkdir(exist_ok=True)
|
| 8 |
+
(out/'components').mkdir(exist_ok=True)
|
| 9 |
+
(out/'reproducible').mkdir(exist_ok=True)
|
| 10 |
+
(out/'logs').mkdir(exist_ok=True)
|
| 11 |
+
source=root/'paper_materials/draft_20260912_revision19/data/results.json'
|
| 12 |
+
data=json.loads(source.read_text(encoding='utf-8'))
|
| 13 |
+
groups=[next(g for g in data['groups'] if g['key']==key) for key in [['grefcoco','instancevg','Qwen3.8'],['construction','yolo26x','Qwen3.8']]]
|
| 14 |
+
(out/'main_table_results.json').write_text(json.dumps({'source':str(source),'protocol':data['protocol'],'groups':groups},indent=2),encoding='utf-8')
|
| 15 |
+
for name in ['instancevg_specialist.json','yolo26x_specialist.json','yolo26x_router.json','qwen_latency.json','qwen_flops.json']:
|
| 16 |
+
shutil.copy2(old/'components'/name,out/'components'/name)
|
| 17 |
+
for name in ['build_figures.py','verify_and_report.py','latex_includes.tex']:
|
| 18 |
+
text=(old/name).read_text(encoding='utf-8').replace("['minicpm','qwen']","['qwen_gref','qwen']").replace('gRefCOCO / M','gRefCOCO / Q').replace('InstanceVG--MiniCPM','InstanceVG--Qwen').replace('main_table_cost_20260912','main_table_cost_qwen_20260912')
|
| 19 |
+
(out/name).write_text(text,encoding='utf-8')
|
| 20 |
+
for p in (old/'reproducible').iterdir():
|
| 21 |
+
if p.is_file(): shutil.copy2(p,out/'reproducible'/p.name)
|
| 22 |
+
for g in groups:
|
| 23 |
+
print(g['key'],g['run'],{m:g['methods'][m]['test_point'] for m in ['learned','confidence']})
|
paper_results/data/main_table_cost_qwen_20260912/selected_points.csv
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
pair,method,nominal_validation_budget,actual_test_call_fraction,test_metric,estimated_gflops_per_input,estimated_time_ms_per_input
|
| 2 |
+
grefcoco|instancevg|Qwen3.8,learned,0.56,0.5355755324277571,0.7528287440093016,17446.303931938,2243.6672827746106
|
| 3 |
+
grefcoco|instancevg|Qwen3.8,confidence,0.71,0.6838612710206166,0.7243286163967898,22269.96488623913,2830.3683522642846
|
| 4 |
+
construction|yolo26x|Qwen3.8,learned,0.5,0.4876830892143808,0.29875800068449215,47383.321370531434,4861.824738942901
|
| 5 |
+
construction|yolo26x|Qwen3.8,confidence,0.93,0.9221038615179761,0.2597490004762105,89464.5250591521,9103.651026133362
|
paper_results/data/main_table_cost_qwen_20260912/verify_and_report.py
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from pathlib import Path
|
| 2 |
+
import csv
|
| 3 |
+
import json
|
| 4 |
+
import fitz
|
| 5 |
+
|
| 6 |
+
root = Path(__file__).resolve().parent
|
| 7 |
+
summary = json.loads((root/'cost_summary.json').read_text())
|
| 8 |
+
groups = json.loads((root/'main_table_results.json').read_text())['groups']
|
| 9 |
+
rows = list(csv.DictReader((root/'full_cost_performance_curves.csv').open()))
|
| 10 |
+
assert len(rows) == 404
|
| 11 |
+
lines = ['# 主表方法:成本—性能曲线', '', '## 1 做了什么', '',
|
| 12 |
+
'- 使用主表原检查点、种子42及原验证集阈值;未重新训练,也未改用后续改进方法。',
|
| 13 |
+
'- gRefCOCO 改为 InstanceVG + Qwen,重新测量该任务上的 Qwen 成本及对应路由器;Construction 和 InstanceVG 专家复用上一轮同口径实测。',
|
| 14 |
+
'- 在 A800 的0、1号卡测量;每个大模型预热2张、正式测量64张,每次1张。专家和路由部分各测64张。',
|
| 15 |
+
'- 重算原路由头,并核对全测试集的选定阈值调用决定与主表完全一致。性能数值沿用主表固定的预测记录。',
|
| 16 |
+
'- 只画学习路由与置信度路由;共404个曲线点。另提供单栏四面板紧凑版。', '',
|
| 17 |
+
'## 2 为什么做', '', '让成本曲线与主实验比较同一个方法,避免混入后续改进结果;补计蒸馏编码器及路由头开销。', '',
|
| 18 |
+
'## 3 做完有什么效果', '', '| 数据集 | 方法 | 测试性能(%) | 调用率(%) | 估算GFLOPs/输入 | 估算毫秒/输入 |',
|
| 19 |
+
'|---|---|---:|---:|---:|---:|']
|
| 20 |
+
comparisons = []
|
| 21 |
+
for group in groups:
|
| 22 |
+
key = '|'.join(group['key'])
|
| 23 |
+
chosen = {p['method']:p for p in summary['selected_points'] if p['pair']==key}
|
| 24 |
+
for method,label in [('learned','学习路由'),('confidence','置信度路由')]:
|
| 25 |
+
p = chosen[method]
|
| 26 |
+
original = group['methods'][method]['test_point']
|
| 27 |
+
assert p['test_metric'] == original['metric']
|
| 28 |
+
assert p['actual_test_call_fraction'] == original['actual_fraction']
|
| 29 |
+
lines.append(f"| {group['key'][0]} | {label} | {100*p['test_metric']:.2f} | {100*p['actual_test_call_fraction']:.2f} | {p['estimated_gflops_per_input']:,.1f} | {p['estimated_time_ms_per_input']:,.1f} |")
|
| 30 |
+
a,b = chosen['learned'],chosen['confidence']
|
| 31 |
+
comparisons.append(f"{group['key'][0]}:相对置信度路由,性能提高 {100*(a['test_metric']-b['test_metric']):.2f} 个百分点,估算 FLOPs 减少 {100*(1-a['estimated_gflops_per_input']/b['estimated_gflops_per_input']):.1f}%,估算耗时减少 {100*(1-a['estimated_time_ms_per_input']/b['estimated_time_ms_per_input']):.1f}%。")
|
| 32 |
+
for comparison in comparisons:
|
| 33 |
+
lines += ['', comparison]
|
| 34 |
+
lines += ['']
|
| 35 |
+
lines += ['### 数值口径', '',
|
| 36 |
+
'- 每个曲线点使用验证集确定的阈值,横轴按该阈值在测试集上的实际调用率计算;圆点对应主表选定点,没有按测试集重新选点。',
|
| 37 |
+
'- 置信度路由成本 = 专家成本 + 实际调用率 × 大模型平均成本。学习路由再加编码器和路由头成本。',
|
| 38 |
+
'- 耗时为分项实测后相加的串行估算,并非各策略整套系统的直接端到端计时。加载模型和预热不计入。',
|
| 39 |
+
'- FLOPs 按乘加计2次统计支持的算子;大模型用8个输入估计预填充成本(含视觉编码),再加平均输出长度乘单词元解码成本。专家统计4个输入,编码器与路由头统计8个输入。',
|
| 40 |
+
'- 各策略共用大模型平均单次成本,尚未测量所选样本的长度差异对成本的影响;专家隐藏特征来自缓存,其额外提取开销未单独测量。置信度阈值判断开销未单独计时。',
|
| 41 |
+
'- 本轮单张测量与上一轮批量测量口径不同,绝对耗时不宜直接比较。', '',
|
| 42 |
+
'- YOLO 使用权重记录的依赖版本8.3.222,安装在任务临时目录;结束时0、1号卡显存均为0,未操作2、3号卡服务。', '',
|
| 43 |
+
'## 4 对应文件在哪', '',
|
| 44 |
+
'- `cost_compact.pdf`:推荐插入正文的单栏四面板图,宽3.44英寸、高2.05英寸。',
|
| 45 |
+
'- `flops_performance.pdf`、`latency_performance.pdf`:两张独立双面板图。',
|
| 46 |
+
'- `full_cost_performance_curves.csv`:全部404个点;`selected_points.csv`:主表4个运行点。',
|
| 47 |
+
'- `components/`:8份实测记录,含原始回答、错误记录及路由核验。',
|
| 48 |
+
'- `build_figures.py`:绘图及成本计算;`reproducible/`:远端测量脚本;`latex_includes.tex`:插入代码及图注。',
|
| 49 |
+
'- 远端结果:`/mnt/HithinkOmniSSD/user_workspace/caisihang/research/ICASSP_ModelCollaboration/outputs/main_table_cost_qwen_20260912`。', '',
|
| 50 |
+
'### 检查记录', '', '已核对404个曲线点、4个主表运行点、两组全测试集调用决定及单位。PDF文字为嵌入字体,另渲染检查排版。']
|
| 51 |
+
(root/'成本性能报告.md').write_text('\n'.join(lines)+'\n',encoding='utf-8')
|
| 52 |
+
for stem in ['cost_compact','flops_performance','latency_performance']:
|
| 53 |
+
doc = fitz.open(root/(stem+'.pdf'))
|
| 54 |
+
assert len(doc)==1
|
| 55 |
+
assert all(font[1] != 'n/a' for font in doc[0].get_fonts())
|
| 56 |
+
doc[0].get_pixmap(matrix=fitz.Matrix(4,4)).save(root/(stem+'_verified.png'))
|
| 57 |
+
print('\n'.join(lines))
|
paper_results/data/main_table_cost_qwen_20260912/成本性能报告.md
ADDED
|
@@ -0,0 +1,50 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 主表方法:成本—性能曲线
|
| 2 |
+
|
| 3 |
+
## 1 做了什么
|
| 4 |
+
|
| 5 |
+
- 使用主表原检查点、种子42及原验证集阈值;未重新训练,也未改用后续改进方法。
|
| 6 |
+
- gRefCOCO 改为 InstanceVG + Qwen,重新测量该任务上的 Qwen 成本及对应路由器;Construction 和 InstanceVG 专家复用上一轮同口径实测。
|
| 7 |
+
- 在 A800 的0、1号卡测量;每个大模型预热2张、正式测量64张,每次1张。专家和路由部分各测64张。
|
| 8 |
+
- 重算原路由头,并核对全测试集的选定阈值调用决定与主表完全一致。性能数值沿用主表固定的预测记录。
|
| 9 |
+
- 只画学习路由与置信度路由;共404个曲线点。另提供单栏四面板紧凑版。
|
| 10 |
+
|
| 11 |
+
## 2 为什么做
|
| 12 |
+
|
| 13 |
+
让成本曲线与主实验比较同一个方法,避免混入后续改进结果;补计蒸馏编码器及路由头开销。
|
| 14 |
+
|
| 15 |
+
## 3 做完有什么效果
|
| 16 |
+
|
| 17 |
+
| 数据集 | 方法 | 测试性能(%) | 调用率(%) | 估算GFLOPs/输入 | 估算毫秒/输入 |
|
| 18 |
+
|---|---|---:|---:|---:|---:|
|
| 19 |
+
| grefcoco | 学习路由 | 75.28 | 53.56 | 17,446.3 | 2,243.7 |
|
| 20 |
+
| grefcoco | 置信度路由 | 72.43 | 68.39 | 22,270.0 | 2,830.4 |
|
| 21 |
+
| construction | 学习路由 | 29.88 | 48.77 | 47,383.3 | 4,861.8 |
|
| 22 |
+
| construction | 置信度路由 | 25.97 | 92.21 | 89,464.5 | 9,103.7 |
|
| 23 |
+
|
| 24 |
+
grefcoco:相对置信度路由,性能提高 2.85 个百分点,估算 FLOPs 减少 21.7%,估算耗时减少 20.7%。
|
| 25 |
+
|
| 26 |
+
construction:相对置信度路由,性能提高 3.90 个百分点,估算 FLOPs 减少 47.0%,估算耗时减少 46.6%。
|
| 27 |
+
|
| 28 |
+
### 数值口径
|
| 29 |
+
|
| 30 |
+
- 每个曲线点使用验证集确定的阈值,横轴按该阈值在测试集上的实际调用率计算;圆点对应主表选定点,没有按测试集重新选点。
|
| 31 |
+
- 置信度路由成本 = 专家成本 + 实际调用率 × 大模型平均成本。学习路由再加编码器和路由头成本。
|
| 32 |
+
- 耗时为分项实测后相加的串行估算,并非各策略整套系统的直接端到端计时。加载模型和预热不计入。
|
| 33 |
+
- FLOPs 按乘加计2次统计支持的算子;大模型用8个输入估计预填充成本(含视觉编码),再加平均输出长度乘单词元解码成本。专家统计4个输入,编码器与路由头统计8个输入。
|
| 34 |
+
- 各策略共用大模型平均单次成本,尚未测量所选样本的长度差异对成本的影响;专家隐藏特征来自缓存,其额外提取开销未单独测量。置信度阈值判断开销未单独计时。
|
| 35 |
+
- 本轮单张测量与上一轮批量测量口径不同,绝对耗时不宜直接比较。
|
| 36 |
+
|
| 37 |
+
- YOLO 使用权重记录的依赖版本8.3.222,安装在任务临时目录;结束时0、1号卡显存均为0,未操作2、3号卡服务。
|
| 38 |
+
|
| 39 |
+
## 4 对应文件在哪
|
| 40 |
+
|
| 41 |
+
- `cost_compact.pdf`:推荐插入正文的单栏四面板图,宽3.44英寸、高2.05英寸。
|
| 42 |
+
- `flops_performance.pdf`、`latency_performance.pdf`:两张独立双面板图。
|
| 43 |
+
- `full_cost_performance_curves.csv`:全部404个点;`selected_points.csv`:主表4个运行点。
|
| 44 |
+
- `components/`:8份实测记录,含原始回答、错误记录及路由核验。
|
| 45 |
+
- `build_figures.py`:绘图及成本计算;`reproducible/`:远端测量脚本;`latex_includes.tex`:插入代码及图注。
|
| 46 |
+
- 远端结果:`/mnt/HithinkOmniSSD/user_workspace/caisihang/research/ICASSP_ModelCollaboration/outputs/main_table_cost_qwen_20260912`。
|
| 47 |
+
|
| 48 |
+
### 检查记录
|
| 49 |
+
|
| 50 |
+
已核对404个曲线点、4个主表运行点、两组全测试集调用决定及单位。PDF文字为嵌入字体,另渲染检查排版。
|
paper_results/data/method_improvement_spec.json
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "router_method_improvement_20260911",
|
| 3 |
+
"remote_root": "/mnt/HithinkOmniSSD/user_workspace/caisihang/research/ICASSP_ModelCollaboration",
|
| 4 |
+
"remote_work": "outputs/router_method_improvement_20260911",
|
| 5 |
+
"data_work": "outputs/router_full_20260910",
|
| 6 |
+
"student": "8M",
|
| 7 |
+
"student_checkpoint": "outputs/mini_siglip_strict_20260910_300ep/8M/epoch_300.pt",
|
| 8 |
+
"fixed": {
|
| 9 |
+
"fraction": 1.0,
|
| 10 |
+
"freeze_encoder": true,
|
| 11 |
+
"fusion": "concat",
|
| 12 |
+
"harm_cost": 1.0,
|
| 13 |
+
"batch_size": 256,
|
| 14 |
+
"weight_decay": 0.01,
|
| 15 |
+
"max_epochs": 50,
|
| 16 |
+
"patience": 8,
|
| 17 |
+
"representation_dim": 128,
|
| 18 |
+
"lr": 0.003,
|
| 19 |
+
"target": "four_state",
|
| 20 |
+
"strategy": "natural",
|
| 21 |
+
"hidden_percent": 90,
|
| 22 |
+
"cpu_threads": 1
|
| 23 |
+
},
|
| 24 |
+
"seeds": [42, 43, 44],
|
| 25 |
+
"experts": {
|
| 26 |
+
"cub": {
|
| 27 |
+
"task": "cub",
|
| 28 |
+
"hidden_layer": "residual_blocks__layer_15",
|
| 29 |
+
"hidden_depth": 16,
|
| 30 |
+
"branches": ["image", "hidden", "confidence", "output"],
|
| 31 |
+
"confidence_column": "max_probability",
|
| 32 |
+
"confidence_valid_column": "max_probability_valid",
|
| 33 |
+
"expected_rows": {"train": 4792, "val": 1200, "test": 5794}
|
| 34 |
+
},
|
| 35 |
+
"glsim": {
|
| 36 |
+
"task": "cub",
|
| 37 |
+
"hidden_layer": "global_local_encoder__layer_11",
|
| 38 |
+
"hidden_depth": 12,
|
| 39 |
+
"branches": ["image", "hidden", "confidence", "output"],
|
| 40 |
+
"confidence_column": "max_probability",
|
| 41 |
+
"confidence_valid_column": "max_probability_valid",
|
| 42 |
+
"expected_rows": {"train": 4792, "val": 1200, "test": 5794}
|
| 43 |
+
},
|
| 44 |
+
"groundingdino": {
|
| 45 |
+
"task": "grefcoco",
|
| 46 |
+
"hidden_layer": "multimodal_encoder__layer_06",
|
| 47 |
+
"hidden_depth": 6,
|
| 48 |
+
"branches": ["image", "text", "hidden", "confidence", "output"],
|
| 49 |
+
"confidence_column": "max_score",
|
| 50 |
+
"confidence_valid_column": "max_score_valid",
|
| 51 |
+
"expected_rows": {"train": 209344, "val": 14229, "test": 35263}
|
| 52 |
+
},
|
| 53 |
+
"instancevg": {
|
| 54 |
+
"task": "grefcoco",
|
| 55 |
+
"hidden_layer": "joint_encoder__layer_11",
|
| 56 |
+
"hidden_depth": 12,
|
| 57 |
+
"branches": ["image", "text", "hidden", "confidence", "output"],
|
| 58 |
+
"confidence_column": "max_score",
|
| 59 |
+
"confidence_valid_column": "max_score_valid",
|
| 60 |
+
"expected_rows": {"train": 209344, "val": 14229, "test": 35263}
|
| 61 |
+
},
|
| 62 |
+
"nlvr2": {
|
| 63 |
+
"task": "nlvr2",
|
| 64 |
+
"hidden_layer": "multimodal_encoder__layer_11",
|
| 65 |
+
"hidden_depth": 12,
|
| 66 |
+
"branches": ["image", "text", "hidden", "confidence", "output"],
|
| 67 |
+
"confidence_column": "max_probability",
|
| 68 |
+
"confidence_valid_column": "max_probability_valid",
|
| 69 |
+
"expected_rows": {"train": 85042, "val": 6982, "test": 13937}
|
| 70 |
+
},
|
| 71 |
+
"vilt": {
|
| 72 |
+
"task": "nlvr2",
|
| 73 |
+
"hidden_layer": "ordered_image_text__layer_11",
|
| 74 |
+
"hidden_depth": 12,
|
| 75 |
+
"branches": ["image", "text", "hidden", "confidence", "output"],
|
| 76 |
+
"confidence_column": "max_probability",
|
| 77 |
+
"confidence_valid_column": "max_probability_valid",
|
| 78 |
+
"expected_rows": {"train": 85042, "val": 6982, "test": 13937}
|
| 79 |
+
},
|
| 80 |
+
"yolo26x": {
|
| 81 |
+
"task": "construction",
|
| 82 |
+
"hidden_layer": "feature_blocks__layer_16",
|
| 83 |
+
"hidden_depth": 17,
|
| 84 |
+
"branches": ["image", "hidden", "confidence", "output"],
|
| 85 |
+
"confidence_column": "max_score",
|
| 86 |
+
"confidence_valid_column": "max_score_valid",
|
| 87 |
+
"expected_rows": {"train": 6308, "val": 701, "test": 3004}
|
| 88 |
+
},
|
| 89 |
+
"rtdetr_x_fp32": {
|
| 90 |
+
"task": "construction",
|
| 91 |
+
"hidden_layer": "query_decoder__layer_06",
|
| 92 |
+
"hidden_depth": 6,
|
| 93 |
+
"branches": ["image", "hidden", "confidence", "output"],
|
| 94 |
+
"confidence_column": "max_score",
|
| 95 |
+
"confidence_valid_column": "max_score_valid",
|
| 96 |
+
"expected_rows": {"train": 6308, "val": 701, "test": 3004}
|
| 97 |
+
}
|
| 98 |
+
},
|
| 99 |
+
"conditions": [
|
| 100 |
+
{"name": "original", "selection_mode": "a25", "score_mode": "s_log"},
|
| 101 |
+
{"name": "score_gain", "selection_mode": "a25", "score_mode": "s_gain"},
|
| 102 |
+
{"name": "checkpoint_perf", "selection_mode": "max_task_performance", "score_mode": "s_log"},
|
| 103 |
+
{"name": "both", "selection_mode": "max_task_performance", "score_mode": "s_gain"}
|
| 104 |
+
],
|
| 105 |
+
"score_only_control": {"name": "score_only_gain", "source_condition": "original", "score_mode": "s_gain"},
|
| 106 |
+
"budgets": {
|
| 107 |
+
"a25": [0.0, 0.05, 0.10, 0.15, 0.20, 0.25],
|
| 108 |
+
"historical_threshold": [0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.10, 0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.20, 0.21, 0.22, 0.23, 0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.30, 0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39, 0.40, 0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.49, 0.50, 0.51, 0.52, 0.53, 0.54, 0.55, 0.56, 0.57, 0.58, 0.59, 0.60, 0.61, 0.62, 0.63, 0.64, 0.65, 0.66, 0.67, 0.68, 0.69, 0.70, 0.71, 0.72, 0.73, 0.74, 0.75, 0.76, 0.77, 0.78, 0.79, 0.80, 0.81, 0.82, 0.83, 0.84, 0.85, 0.86, 0.87, 0.88, 0.89, 0.90, 0.91, 0.92, 0.93, 0.94, 0.95, 0.96, 0.97, 0.98, 0.99],
|
| 109 |
+
"endpoint": [0.0, 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.10, 0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.20, 0.21, 0.22, 0.23, 0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.30, 0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39, 0.40, 0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.49, 0.50, 0.51, 0.52, 0.53, 0.54, 0.55, 0.56, 0.57, 0.58, 0.59, 0.60, 0.61, 0.62, 0.63, 0.64, 0.65, 0.66, 0.67, 0.68, 0.69, 0.70, 0.71, 0.72, 0.73, 0.74, 0.75, 0.76, 0.77, 0.78, 0.79, 0.80, 0.81, 0.82, 0.83, 0.84, 0.85, 0.86, 0.87, 0.88, 0.89, 0.90, 0.91, 0.92, 0.93, 0.94, 0.95, 0.96, 0.97, 0.98, 0.99, 1.0],
|
| 110 |
+
"batch_diagnostic": [0.0, 0.05, 0.10, 0.15, 0.20, 0.25, 0.50, 0.75, 1.0]
|
| 111 |
+
},
|
| 112 |
+
"fusion_alphas": [0.0, 0.25, 0.5, 0.75, 1.0],
|
| 113 |
+
"tie_epsilon": 1e-12
|
| 114 |
+
}
|
paper_results/data/operation_measurements.json
ADDED
|
@@ -0,0 +1,129 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"small_models": [
|
| 3 |
+
{
|
| 4 |
+
"model": "ResNet",
|
| 5 |
+
"expert": "cub",
|
| 6 |
+
"samples": 32,
|
| 7 |
+
"mean_gflops_counted": 8.175091712,
|
| 8 |
+
"min_gflops_counted": 8.175091712,
|
| 9 |
+
"max_gflops_counted": 8.175091712,
|
| 10 |
+
"operator_inventory_source": "cub_operator_check.json",
|
| 11 |
+
"operator_check_samples": 1,
|
| 12 |
+
"uncovered_spatial_sampling_seen": false
|
| 13 |
+
},
|
| 14 |
+
{
|
| 15 |
+
"model": "YOLO26x",
|
| 16 |
+
"expert": "yolo26x",
|
| 17 |
+
"samples": 32,
|
| 18 |
+
"mean_gflops_counted": 143.2219032,
|
| 19 |
+
"min_gflops_counted": 116.2238976,
|
| 20 |
+
"max_gflops_counted": 193.92768,
|
| 21 |
+
"operator_inventory_source": "yolo26x_operator_check.json",
|
| 22 |
+
"operator_check_samples": 1,
|
| 23 |
+
"uncovered_spatial_sampling_seen": false
|
| 24 |
+
},
|
| 25 |
+
{
|
| 26 |
+
"model": "RT-DETR-X",
|
| 27 |
+
"expert": "rtdetr_x_fp32",
|
| 28 |
+
"samples": 32,
|
| 29 |
+
"mean_gflops_counted": 224.6444544,
|
| 30 |
+
"min_gflops_counted": 224.6444544,
|
| 31 |
+
"max_gflops_counted": 224.6444544,
|
| 32 |
+
"operator_inventory_source": "rtdetr_x_fp32_operator_check.json",
|
| 33 |
+
"operator_check_samples": 1,
|
| 34 |
+
"uncovered_spatial_sampling_seen": true
|
| 35 |
+
},
|
| 36 |
+
{
|
| 37 |
+
"model": "ViLT",
|
| 38 |
+
"expert": "vilt",
|
| 39 |
+
"samples": 32,
|
| 40 |
+
"mean_gflops_counted": 84.895706112,
|
| 41 |
+
"min_gflops_counted": 57.611655168,
|
| 42 |
+
"max_gflops_counted": 120.690972672,
|
| 43 |
+
"operator_inventory_source": "vilt_operator_check.json",
|
| 44 |
+
"operator_check_samples": 1,
|
| 45 |
+
"uncovered_spatial_sampling_seen": false
|
| 46 |
+
},
|
| 47 |
+
{
|
| 48 |
+
"model": "GroundingDINO",
|
| 49 |
+
"expert": "groundingdino",
|
| 50 |
+
"samples": 32,
|
| 51 |
+
"mean_gflops_counted": 668.259162288,
|
| 52 |
+
"min_gflops_counted": 499.212779776,
|
| 53 |
+
"max_gflops_counted": 820.17636864,
|
| 54 |
+
"operator_inventory_source": "groundingdino_operator_check.json",
|
| 55 |
+
"operator_check_samples": 1,
|
| 56 |
+
"uncovered_spatial_sampling_seen": true
|
| 57 |
+
},
|
| 58 |
+
{
|
| 59 |
+
"model": "BEiT3",
|
| 60 |
+
"expert": "nlvr2",
|
| 61 |
+
"samples": 32,
|
| 62 |
+
"mean_gflops_counted": 76.799401728,
|
| 63 |
+
"min_gflops_counted": 73.586743296,
|
| 64 |
+
"max_gflops_counted": 83.259561984,
|
| 65 |
+
"operator_inventory_source": "nlvr2_operator_check.json",
|
| 66 |
+
"operator_check_samples": 1,
|
| 67 |
+
"uncovered_spatial_sampling_seen": false
|
| 68 |
+
},
|
| 69 |
+
{
|
| 70 |
+
"model": "GLSim",
|
| 71 |
+
"expert": "glsim",
|
| 72 |
+
"samples": 32,
|
| 73 |
+
"mean_gflops_counted": 70.280871936,
|
| 74 |
+
"min_gflops_counted": 70.280871936,
|
| 75 |
+
"max_gflops_counted": 70.280871936,
|
| 76 |
+
"operator_inventory_source": "glsim_ops.json",
|
| 77 |
+
"operator_check_samples": 32,
|
| 78 |
+
"uncovered_spatial_sampling_seen": false
|
| 79 |
+
}
|
| 80 |
+
],
|
| 81 |
+
"large_input_cases": 16,
|
| 82 |
+
"large_input_check_complete": true,
|
| 83 |
+
"large_model_probes": [
|
| 84 |
+
{
|
| 85 |
+
"model": "MiniCPM",
|
| 86 |
+
"sample_id": "cub-test-00517",
|
| 87 |
+
"task": "cub",
|
| 88 |
+
"prompt_tokens": 1996,
|
| 89 |
+
"prefill_gflops_counted": 33243.606087168,
|
| 90 |
+
"one_known_token_gflops_counted": 16.31253312
|
| 91 |
+
},
|
| 92 |
+
{
|
| 93 |
+
"model": "Qwen3.8",
|
| 94 |
+
"sample_id": "cub-test-00517",
|
| 95 |
+
"task": "cub",
|
| 96 |
+
"prompt_tokens": 4864,
|
| 97 |
+
"prefill_gflops_counted": 249018.23668224,
|
| 98 |
+
"one_known_token_gflops_counted": 53.164769472
|
| 99 |
+
}
|
| 100 |
+
],
|
| 101 |
+
"large_gpu_probe_status": {
|
| 102 |
+
"stage": "finished",
|
| 103 |
+
"occupancy_original_pid": 157057,
|
| 104 |
+
"stopped_process_ids": [
|
| 105 |
+
157068,
|
| 106 |
+
157342,
|
| 107 |
+
157343,
|
| 108 |
+
157587,
|
| 109 |
+
157588,
|
| 110 |
+
157589,
|
| 111 |
+
157590,
|
| 112 |
+
157057
|
| 113 |
+
],
|
| 114 |
+
"started_at": 1789028871.3039298,
|
| 115 |
+
"measurement_exit_code": 0,
|
| 116 |
+
"occupancy_restored": true,
|
| 117 |
+
"measurement_pid": 159330,
|
| 118 |
+
"occupancy_new_pid": 159642,
|
| 119 |
+
"finished_at": 1789029043.4601417
|
| 120 |
+
},
|
| 121 |
+
"whole_system_flops_complete": false,
|
| 122 |
+
"missing_components": [
|
| 123 |
+
"InstanceVG",
|
| 124 |
+
"full generation and all-task large-model coverage",
|
| 125 |
+
"uncovered operations",
|
| 126 |
+
"per-example routing alignment",
|
| 127 |
+
"95%/99% resampling intervals"
|
| 128 |
+
]
|
| 129 |
+
}
|
paper_results/data/results.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
paper_results/data/scaling_selected.json
ADDED
|
@@ -0,0 +1,1250 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"key": [
|
| 4 |
+
"cub",
|
| 5 |
+
"cub",
|
| 6 |
+
"MiniCPM"
|
| 7 |
+
],
|
| 8 |
+
"rows": [
|
| 9 |
+
{
|
| 10 |
+
"id": "formal_0185",
|
| 11 |
+
"tag": "baseline_25pct",
|
| 12 |
+
"task": "cub",
|
| 13 |
+
"expert": "cub",
|
| 14 |
+
"endpoint": "MiniCPM",
|
| 15 |
+
"fraction": 0.25,
|
| 16 |
+
"student": "4M",
|
| 17 |
+
"branches": "output",
|
| 18 |
+
"fusion": "concat",
|
| 19 |
+
"target": "gain",
|
| 20 |
+
"strategy": "natural",
|
| 21 |
+
"representation_dim": 512,
|
| 22 |
+
"hidden_layer": "residual_blocks__layer_16",
|
| 23 |
+
"fixed25_metric": 0.7849499482222989,
|
| 24 |
+
"fixed25_actual_fraction": 0.2587159130134622,
|
| 25 |
+
"fixed25_rescue": 31,
|
| 26 |
+
"fixed25_harm": 201,
|
| 27 |
+
"rank25_metric": 0.7884017949603037,
|
| 28 |
+
"validation_selection": 0.9741666666666666,
|
| 29 |
+
"best_epoch": 12,
|
| 30 |
+
"seconds": 5.802937984466553
|
| 31 |
+
},
|
| 32 |
+
{
|
| 33 |
+
"id": "formal_0186",
|
| 34 |
+
"tag": "baseline_50pct",
|
| 35 |
+
"task": "cub",
|
| 36 |
+
"expert": "cub",
|
| 37 |
+
"endpoint": "MiniCPM",
|
| 38 |
+
"fraction": 0.5,
|
| 39 |
+
"student": "4M",
|
| 40 |
+
"branches": "output",
|
| 41 |
+
"fusion": "concat",
|
| 42 |
+
"target": "gain",
|
| 43 |
+
"strategy": "natural",
|
| 44 |
+
"representation_dim": 512,
|
| 45 |
+
"hidden_layer": "residual_blocks__layer_16",
|
| 46 |
+
"fixed25_metric": 0.7904729030031067,
|
| 47 |
+
"fixed25_actual_fraction": 0.2549188816016569,
|
| 48 |
+
"fixed25_rescue": 38,
|
| 49 |
+
"fixed25_harm": 176,
|
| 50 |
+
"rank25_metric": 0.7930617880566103,
|
| 51 |
+
"validation_selection": 0.9731666666666667,
|
| 52 |
+
"best_epoch": 7,
|
| 53 |
+
"seconds": 7.1614015102386475
|
| 54 |
+
},
|
| 55 |
+
{
|
| 56 |
+
"id": "formal_0001",
|
| 57 |
+
"tag": "baseline_100pct",
|
| 58 |
+
"task": "cub",
|
| 59 |
+
"expert": "cub",
|
| 60 |
+
"endpoint": "MiniCPM",
|
| 61 |
+
"fraction": 1.0,
|
| 62 |
+
"student": "4M",
|
| 63 |
+
"branches": "output",
|
| 64 |
+
"fusion": "concat",
|
| 65 |
+
"target": "gain",
|
| 66 |
+
"strategy": "natural",
|
| 67 |
+
"representation_dim": 512,
|
| 68 |
+
"hidden_layer": "residual_blocks__layer_16",
|
| 69 |
+
"fixed25_metric": 0.787193648602002,
|
| 70 |
+
"fixed25_actual_fraction": 0.2557818432861581,
|
| 71 |
+
"fixed25_rescue": 33,
|
| 72 |
+
"fixed25_harm": 190,
|
| 73 |
+
"rank25_metric": 0.7913358646876079,
|
| 74 |
+
"validation_selection": 0.9744166666666666,
|
| 75 |
+
"best_epoch": 7,
|
| 76 |
+
"seconds": 2.9200518131256104
|
| 77 |
+
}
|
| 78 |
+
]
|
| 79 |
+
},
|
| 80 |
+
{
|
| 81 |
+
"key": [
|
| 82 |
+
"cub",
|
| 83 |
+
"cub",
|
| 84 |
+
"Qwen3.8"
|
| 85 |
+
],
|
| 86 |
+
"rows": [
|
| 87 |
+
{
|
| 88 |
+
"id": "formal_0187",
|
| 89 |
+
"tag": "baseline_25pct",
|
| 90 |
+
"task": "cub",
|
| 91 |
+
"expert": "cub",
|
| 92 |
+
"endpoint": "Qwen3.8",
|
| 93 |
+
"fraction": 0.25,
|
| 94 |
+
"student": "4M",
|
| 95 |
+
"branches": "output+confidence",
|
| 96 |
+
"fusion": "concat",
|
| 97 |
+
"target": "four_state",
|
| 98 |
+
"strategy": "natural",
|
| 99 |
+
"representation_dim": 512,
|
| 100 |
+
"hidden_layer": "residual_blocks__layer_16",
|
| 101 |
+
"fixed25_metric": 0.6874352778736624,
|
| 102 |
+
"fixed25_actual_fraction": 0.6039005868139454,
|
| 103 |
+
"fixed25_rescue": 358,
|
| 104 |
+
"fixed25_harm": 1093,
|
| 105 |
+
"rank25_metric": 0.8060062133241284,
|
| 106 |
+
"validation_selection": 0.9316666666666666,
|
| 107 |
+
"best_epoch": 5,
|
| 108 |
+
"seconds": 4.623678922653198
|
| 109 |
+
},
|
| 110 |
+
{
|
| 111 |
+
"id": "formal_0188",
|
| 112 |
+
"tag": "baseline_50pct",
|
| 113 |
+
"task": "cub",
|
| 114 |
+
"expert": "cub",
|
| 115 |
+
"endpoint": "Qwen3.8",
|
| 116 |
+
"fraction": 0.5,
|
| 117 |
+
"student": "4M",
|
| 118 |
+
"branches": "output+confidence",
|
| 119 |
+
"fusion": "concat",
|
| 120 |
+
"target": "four_state",
|
| 121 |
+
"strategy": "natural",
|
| 122 |
+
"representation_dim": 512,
|
| 123 |
+
"hidden_layer": "residual_blocks__layer_16",
|
| 124 |
+
"fixed25_metric": 0.694338971349672,
|
| 125 |
+
"fixed25_actual_fraction": 0.5997583707283397,
|
| 126 |
+
"fixed25_rescue": 357,
|
| 127 |
+
"fixed25_harm": 1052,
|
| 128 |
+
"rank25_metric": 0.8096306523990335,
|
| 129 |
+
"validation_selection": 0.9334166666666667,
|
| 130 |
+
"best_epoch": 3,
|
| 131 |
+
"seconds": 4.7386627197265625
|
| 132 |
+
},
|
| 133 |
+
{
|
| 134 |
+
"id": "formal_0009",
|
| 135 |
+
"tag": "baseline_100pct",
|
| 136 |
+
"task": "cub",
|
| 137 |
+
"expert": "cub",
|
| 138 |
+
"endpoint": "Qwen3.8",
|
| 139 |
+
"fraction": 1.0,
|
| 140 |
+
"student": "4M",
|
| 141 |
+
"branches": "output+confidence",
|
| 142 |
+
"fusion": "concat",
|
| 143 |
+
"target": "four_state",
|
| 144 |
+
"strategy": "natural",
|
| 145 |
+
"representation_dim": 512,
|
| 146 |
+
"hidden_layer": "residual_blocks__layer_16",
|
| 147 |
+
"fixed25_metric": 0.6829478771142561,
|
| 148 |
+
"fixed25_actual_fraction": 0.6049361408353469,
|
| 149 |
+
"fixed25_rescue": 358,
|
| 150 |
+
"fixed25_harm": 1119,
|
| 151 |
+
"rank25_metric": 0.8091128753883328,
|
| 152 |
+
"validation_selection": 0.928,
|
| 153 |
+
"best_epoch": 1,
|
| 154 |
+
"seconds": 2.053396463394165
|
| 155 |
+
}
|
| 156 |
+
]
|
| 157 |
+
},
|
| 158 |
+
{
|
| 159 |
+
"key": [
|
| 160 |
+
"cub",
|
| 161 |
+
"glsim",
|
| 162 |
+
"MiniCPM"
|
| 163 |
+
],
|
| 164 |
+
"rows": [
|
| 165 |
+
{
|
| 166 |
+
"id": "formal_0189",
|
| 167 |
+
"tag": "baseline_25pct",
|
| 168 |
+
"task": "cub",
|
| 169 |
+
"expert": "glsim",
|
| 170 |
+
"endpoint": "MiniCPM",
|
| 171 |
+
"fraction": 0.25,
|
| 172 |
+
"student": "4M",
|
| 173 |
+
"branches": "output",
|
| 174 |
+
"fusion": "concat",
|
| 175 |
+
"target": "gain",
|
| 176 |
+
"strategy": "natural",
|
| 177 |
+
"representation_dim": 512,
|
| 178 |
+
"hidden_layer": "aggregator__layer_01",
|
| 179 |
+
"fixed25_metric": 0.8674490852606145,
|
| 180 |
+
"fixed25_actual_fraction": 0.24818778046254747,
|
| 181 |
+
"fixed25_rescue": 11,
|
| 182 |
+
"fixed25_harm": 209,
|
| 183 |
+
"rank25_metric": 0.8674490852606145,
|
| 184 |
+
"validation_selection": 0.9786666666666667,
|
| 185 |
+
"best_epoch": 12,
|
| 186 |
+
"seconds": 6.889333963394165
|
| 187 |
+
},
|
| 188 |
+
{
|
| 189 |
+
"id": "formal_0190",
|
| 190 |
+
"tag": "baseline_50pct",
|
| 191 |
+
"task": "cub",
|
| 192 |
+
"expert": "glsim",
|
| 193 |
+
"endpoint": "MiniCPM",
|
| 194 |
+
"fraction": 0.5,
|
| 195 |
+
"student": "4M",
|
| 196 |
+
"branches": "output",
|
| 197 |
+
"fusion": "concat",
|
| 198 |
+
"target": "gain",
|
| 199 |
+
"strategy": "natural",
|
| 200 |
+
"representation_dim": 512,
|
| 201 |
+
"hidden_layer": "aggregator__layer_01",
|
| 202 |
+
"fixed25_metric": 0.870037970314118,
|
| 203 |
+
"fixed25_actual_fraction": 0.24991370383154987,
|
| 204 |
+
"fixed25_rescue": 13,
|
| 205 |
+
"fixed25_harm": 196,
|
| 206 |
+
"rank25_metric": 0.870037970314118,
|
| 207 |
+
"validation_selection": 0.9773333333333334,
|
| 208 |
+
"best_epoch": 6,
|
| 209 |
+
"seconds": 5.409059524536133
|
| 210 |
+
},
|
| 211 |
+
{
|
| 212 |
+
"id": "formal_0019",
|
| 213 |
+
"tag": "baseline_100pct",
|
| 214 |
+
"task": "cub",
|
| 215 |
+
"expert": "glsim",
|
| 216 |
+
"endpoint": "MiniCPM",
|
| 217 |
+
"fraction": 1.0,
|
| 218 |
+
"student": "4M",
|
| 219 |
+
"branches": "output",
|
| 220 |
+
"fusion": "concat",
|
| 221 |
+
"target": "gain",
|
| 222 |
+
"strategy": "natural",
|
| 223 |
+
"representation_dim": 512,
|
| 224 |
+
"hidden_layer": "aggregator__layer_01",
|
| 225 |
+
"fixed25_metric": 0.8658957542285123,
|
| 226 |
+
"fixed25_actual_fraction": 0.25509147393855713,
|
| 227 |
+
"fixed25_rescue": 13,
|
| 228 |
+
"fixed25_harm": 220,
|
| 229 |
+
"rank25_metric": 0.8679668622713151,
|
| 230 |
+
"validation_selection": 0.9788333333333333,
|
| 231 |
+
"best_epoch": 9,
|
| 232 |
+
"seconds": 7.6912243366241455
|
| 233 |
+
}
|
| 234 |
+
]
|
| 235 |
+
},
|
| 236 |
+
{
|
| 237 |
+
"key": [
|
| 238 |
+
"cub",
|
| 239 |
+
"glsim",
|
| 240 |
+
"Qwen3.8"
|
| 241 |
+
],
|
| 242 |
+
"rows": [
|
| 243 |
+
{
|
| 244 |
+
"id": "formal_0191",
|
| 245 |
+
"tag": "baseline_25pct",
|
| 246 |
+
"task": "cub",
|
| 247 |
+
"expert": "glsim",
|
| 248 |
+
"endpoint": "Qwen3.8",
|
| 249 |
+
"fraction": 0.25,
|
| 250 |
+
"student": "4M",
|
| 251 |
+
"branches": "image+text+hidden",
|
| 252 |
+
"fusion": "concat",
|
| 253 |
+
"target": "four_state",
|
| 254 |
+
"strategy": "natural",
|
| 255 |
+
"representation_dim": 512,
|
| 256 |
+
"hidden_layer": "global_local_encoder__layer_12",
|
| 257 |
+
"fixed25_metric": 0.8797031411805316,
|
| 258 |
+
"fixed25_actual_fraction": 0.23662409389023129,
|
| 259 |
+
"fixed25_rescue": 18,
|
| 260 |
+
"fixed25_harm": 145,
|
| 261 |
+
"rank25_metric": 0.8776320331377286,
|
| 262 |
+
"validation_selection": 0.98025,
|
| 263 |
+
"best_epoch": 18,
|
| 264 |
+
"seconds": 7.610384702682495
|
| 265 |
+
},
|
| 266 |
+
{
|
| 267 |
+
"id": "formal_0192",
|
| 268 |
+
"tag": "baseline_50pct",
|
| 269 |
+
"task": "cub",
|
| 270 |
+
"expert": "glsim",
|
| 271 |
+
"endpoint": "Qwen3.8",
|
| 272 |
+
"fraction": 0.5,
|
| 273 |
+
"student": "4M",
|
| 274 |
+
"branches": "image+text+hidden",
|
| 275 |
+
"fusion": "concat",
|
| 276 |
+
"target": "four_state",
|
| 277 |
+
"strategy": "natural",
|
| 278 |
+
"representation_dim": 512,
|
| 279 |
+
"hidden_layer": "global_local_encoder__layer_12",
|
| 280 |
+
"fixed25_metric": 0.876423886779427,
|
| 281 |
+
"fixed25_actual_fraction": 0.24525371073524335,
|
| 282 |
+
"fixed25_rescue": 8,
|
| 283 |
+
"fixed25_harm": 154,
|
| 284 |
+
"rank25_metric": 0.8762512944425268,
|
| 285 |
+
"validation_selection": 0.981,
|
| 286 |
+
"best_epoch": 25,
|
| 287 |
+
"seconds": 10.510464906692505
|
| 288 |
+
},
|
| 289 |
+
{
|
| 290 |
+
"id": "formal_0027",
|
| 291 |
+
"tag": "baseline_100pct",
|
| 292 |
+
"task": "cub",
|
| 293 |
+
"expert": "glsim",
|
| 294 |
+
"endpoint": "Qwen3.8",
|
| 295 |
+
"fraction": 1.0,
|
| 296 |
+
"student": "4M",
|
| 297 |
+
"branches": "image+text+hidden",
|
| 298 |
+
"fusion": "concat",
|
| 299 |
+
"target": "four_state",
|
| 300 |
+
"strategy": "natural",
|
| 301 |
+
"representation_dim": 512,
|
| 302 |
+
"hidden_layer": "global_local_encoder__layer_12",
|
| 303 |
+
"fixed25_metric": 0.8684846392820159,
|
| 304 |
+
"fixed25_actual_fraction": 0.2507766655160511,
|
| 305 |
+
"fixed25_rescue": 17,
|
| 306 |
+
"fixed25_harm": 209,
|
| 307 |
+
"rank25_metric": 0.8684846392820159,
|
| 308 |
+
"validation_selection": 0.97525,
|
| 309 |
+
"best_epoch": 1,
|
| 310 |
+
"seconds": 3.935011863708496
|
| 311 |
+
}
|
| 312 |
+
]
|
| 313 |
+
},
|
| 314 |
+
{
|
| 315 |
+
"key": [
|
| 316 |
+
"grefcoco",
|
| 317 |
+
"groundingdino",
|
| 318 |
+
"MiniCPM"
|
| 319 |
+
],
|
| 320 |
+
"rows": [
|
| 321 |
+
{
|
| 322 |
+
"id": "formal_0193",
|
| 323 |
+
"tag": "baseline_25pct",
|
| 324 |
+
"task": "grefcoco",
|
| 325 |
+
"expert": "groundingdino",
|
| 326 |
+
"endpoint": "MiniCPM",
|
| 327 |
+
"fraction": 0.25,
|
| 328 |
+
"student": "4M",
|
| 329 |
+
"branches": "image+text+hidden+confidence",
|
| 330 |
+
"fusion": "attention",
|
| 331 |
+
"target": "error",
|
| 332 |
+
"strategy": "weighted",
|
| 333 |
+
"representation_dim": 512,
|
| 334 |
+
"hidden_layer": "text_encoder__layer_09",
|
| 335 |
+
"fixed25_metric": 0.38158976831239544,
|
| 336 |
+
"fixed25_actual_fraction": 0.24836230609987806,
|
| 337 |
+
"fixed25_rescue": 4083,
|
| 338 |
+
"fixed25_harm": 38,
|
| 339 |
+
"rank25_metric": 0.38201514335138814,
|
| 340 |
+
"validation_selection": 0.6237472766884531,
|
| 341 |
+
"best_epoch": 1,
|
| 342 |
+
"seconds": 28.353870153427124
|
| 343 |
+
},
|
| 344 |
+
{
|
| 345 |
+
"id": "formal_0194",
|
| 346 |
+
"tag": "baseline_50pct",
|
| 347 |
+
"task": "grefcoco",
|
| 348 |
+
"expert": "groundingdino",
|
| 349 |
+
"endpoint": "MiniCPM",
|
| 350 |
+
"fraction": 0.5,
|
| 351 |
+
"student": "4M",
|
| 352 |
+
"branches": "image+text+hidden+confidence",
|
| 353 |
+
"fusion": "attention",
|
| 354 |
+
"target": "error",
|
| 355 |
+
"strategy": "weighted",
|
| 356 |
+
"representation_dim": 512,
|
| 357 |
+
"hidden_layer": "text_encoder__layer_09",
|
| 358 |
+
"fixed25_metric": 0.3658792502055979,
|
| 359 |
+
"fixed25_actual_fraction": 0.22034427019822478,
|
| 360 |
+
"fixed25_rescue": 3503,
|
| 361 |
+
"fixed25_harm": 12,
|
| 362 |
+
"rank25_metric": 0.3758330261180274,
|
| 363 |
+
"validation_selection": 0.6238526951999438,
|
| 364 |
+
"best_epoch": 2,
|
| 365 |
+
"seconds": 56.78019833564758
|
| 366 |
+
},
|
| 367 |
+
{
|
| 368 |
+
"id": "formal_0042",
|
| 369 |
+
"tag": "baseline_100pct",
|
| 370 |
+
"task": "grefcoco",
|
| 371 |
+
"expert": "groundingdino",
|
| 372 |
+
"endpoint": "MiniCPM",
|
| 373 |
+
"fraction": 1.0,
|
| 374 |
+
"student": "4M",
|
| 375 |
+
"branches": "image+text+hidden+confidence",
|
| 376 |
+
"fusion": "attention",
|
| 377 |
+
"target": "error",
|
| 378 |
+
"strategy": "weighted",
|
| 379 |
+
"representation_dim": 512,
|
| 380 |
+
"hidden_layer": "text_encoder__layer_09",
|
| 381 |
+
"fixed25_metric": 0.3693106088534725,
|
| 382 |
+
"fixed25_actual_fraction": 0.22853982928281769,
|
| 383 |
+
"fixed25_rescue": 3642,
|
| 384 |
+
"fixed25_harm": 30,
|
| 385 |
+
"rank25_metric": 0.3758330261180274,
|
| 386 |
+
"validation_selection": 0.6221730269168599,
|
| 387 |
+
"best_epoch": 9,
|
| 388 |
+
"seconds": 476.6986792087555
|
| 389 |
+
}
|
| 390 |
+
]
|
| 391 |
+
},
|
| 392 |
+
{
|
| 393 |
+
"key": [
|
| 394 |
+
"grefcoco",
|
| 395 |
+
"groundingdino",
|
| 396 |
+
"Qwen3.8"
|
| 397 |
+
],
|
| 398 |
+
"rows": [
|
| 399 |
+
{
|
| 400 |
+
"id": "formal_0195",
|
| 401 |
+
"tag": "baseline_25pct",
|
| 402 |
+
"task": "grefcoco",
|
| 403 |
+
"expert": "groundingdino",
|
| 404 |
+
"endpoint": "Qwen3.8",
|
| 405 |
+
"fraction": 0.25,
|
| 406 |
+
"student": "4M",
|
| 407 |
+
"branches": "image+text+hidden+confidence",
|
| 408 |
+
"fusion": "concat",
|
| 409 |
+
"target": "error",
|
| 410 |
+
"strategy": "weighted",
|
| 411 |
+
"representation_dim": 512,
|
| 412 |
+
"hidden_layer": "text_encoder__layer_09",
|
| 413 |
+
"fixed25_metric": 0.4562572668235828,
|
| 414 |
+
"fixed25_actual_fraction": 0.24856081445140799,
|
| 415 |
+
"fixed25_rescue": 6732,
|
| 416 |
+
"fixed25_harm": 54,
|
| 417 |
+
"rank25_metric": 0.45705130022970253,
|
| 418 |
+
"validation_selection": 0.6656265373532926,
|
| 419 |
+
"best_epoch": 1,
|
| 420 |
+
"seconds": 18.048375368118286
|
| 421 |
+
},
|
| 422 |
+
{
|
| 423 |
+
"id": "formal_0196",
|
| 424 |
+
"tag": "baseline_50pct",
|
| 425 |
+
"task": "grefcoco",
|
| 426 |
+
"expert": "groundingdino",
|
| 427 |
+
"endpoint": "Qwen3.8",
|
| 428 |
+
"fraction": 0.5,
|
| 429 |
+
"student": "4M",
|
| 430 |
+
"branches": "image+text+hidden+confidence",
|
| 431 |
+
"fusion": "concat",
|
| 432 |
+
"target": "error",
|
| 433 |
+
"strategy": "weighted",
|
| 434 |
+
"representation_dim": 512,
|
| 435 |
+
"hidden_layer": "text_encoder__layer_09",
|
| 436 |
+
"fixed25_metric": 0.434648214842753,
|
| 437 |
+
"fixed25_actual_fraction": 0.22008904517482913,
|
| 438 |
+
"fixed25_rescue": 5930,
|
| 439 |
+
"fixed25_harm": 14,
|
| 440 |
+
"rank25_metric": 0.4536766582536937,
|
| 441 |
+
"validation_selection": 0.6660130718954248,
|
| 442 |
+
"best_epoch": 2,
|
| 443 |
+
"seconds": 33.53883171081543
|
| 444 |
+
},
|
| 445 |
+
{
|
| 446 |
+
"id": "formal_0057",
|
| 447 |
+
"tag": "baseline_100pct",
|
| 448 |
+
"task": "grefcoco",
|
| 449 |
+
"expert": "groundingdino",
|
| 450 |
+
"endpoint": "Qwen3.8",
|
| 451 |
+
"fraction": 1.0,
|
| 452 |
+
"student": "4M",
|
| 453 |
+
"branches": "image+text+hidden+confidence",
|
| 454 |
+
"fusion": "concat",
|
| 455 |
+
"target": "error",
|
| 456 |
+
"strategy": "weighted",
|
| 457 |
+
"representation_dim": 512,
|
| 458 |
+
"hidden_layer": "text_encoder__layer_09",
|
| 459 |
+
"fixed25_metric": 0.43033774778096023,
|
| 460 |
+
"fixed25_actual_fraction": 0.2126591611604231,
|
| 461 |
+
"fixed25_rescue": 5784,
|
| 462 |
+
"fixed25_harm": 20,
|
| 463 |
+
"rank25_metric": 0.4544139749879477,
|
| 464 |
+
"validation_selection": 0.6650924168950734,
|
| 465 |
+
"best_epoch": 5,
|
| 466 |
+
"seconds": 281.1101198196411
|
| 467 |
+
}
|
| 468 |
+
]
|
| 469 |
+
},
|
| 470 |
+
{
|
| 471 |
+
"key": [
|
| 472 |
+
"grefcoco",
|
| 473 |
+
"instancevg",
|
| 474 |
+
"MiniCPM"
|
| 475 |
+
],
|
| 476 |
+
"rows": [
|
| 477 |
+
{
|
| 478 |
+
"id": "formal_0197",
|
| 479 |
+
"tag": "baseline_25pct",
|
| 480 |
+
"task": "grefcoco",
|
| 481 |
+
"expert": "instancevg",
|
| 482 |
+
"endpoint": "MiniCPM",
|
| 483 |
+
"fraction": 0.25,
|
| 484 |
+
"student": "4M",
|
| 485 |
+
"branches": "hidden",
|
| 486 |
+
"fusion": "concat",
|
| 487 |
+
"target": "gain",
|
| 488 |
+
"strategy": "weighted",
|
| 489 |
+
"representation_dim": 512,
|
| 490 |
+
"hidden_layer": "query_generation__layer_01",
|
| 491 |
+
"fixed25_metric": 0.6523268014632901,
|
| 492 |
+
"fixed25_actual_fraction": 0.3479284235601055,
|
| 493 |
+
"fixed25_rescue": 1394,
|
| 494 |
+
"fixed25_harm": 2000,
|
| 495 |
+
"rank25_metric": 0.6652582026486685,
|
| 496 |
+
"validation_selection": 0.753925082577834,
|
| 497 |
+
"best_epoch": 13,
|
| 498 |
+
"seconds": 28.585034132003784
|
| 499 |
+
},
|
| 500 |
+
{
|
| 501 |
+
"id": "formal_0198",
|
| 502 |
+
"tag": "baseline_50pct",
|
| 503 |
+
"task": "grefcoco",
|
| 504 |
+
"expert": "instancevg",
|
| 505 |
+
"endpoint": "MiniCPM",
|
| 506 |
+
"fraction": 0.5,
|
| 507 |
+
"student": "4M",
|
| 508 |
+
"branches": "hidden",
|
| 509 |
+
"fusion": "concat",
|
| 510 |
+
"target": "gain",
|
| 511 |
+
"strategy": "weighted",
|
| 512 |
+
"representation_dim": 512,
|
| 513 |
+
"hidden_layer": "query_generation__layer_01",
|
| 514 |
+
"fixed25_metric": 0.6567223435328815,
|
| 515 |
+
"fixed25_actual_fraction": 0.3123670703003148,
|
| 516 |
+
"fixed25_rescue": 1366,
|
| 517 |
+
"fixed25_harm": 1817,
|
| 518 |
+
"rank25_metric": 0.662762669086578,
|
| 519 |
+
"validation_selection": 0.7576779815868999,
|
| 520 |
+
"best_epoch": 13,
|
| 521 |
+
"seconds": 44.9037389755249
|
| 522 |
+
},
|
| 523 |
+
{
|
| 524 |
+
"id": "formal_0072",
|
| 525 |
+
"tag": "baseline_100pct",
|
| 526 |
+
"task": "grefcoco",
|
| 527 |
+
"expert": "instancevg",
|
| 528 |
+
"endpoint": "MiniCPM",
|
| 529 |
+
"fraction": 1.0,
|
| 530 |
+
"student": "4M",
|
| 531 |
+
"branches": "hidden",
|
| 532 |
+
"fusion": "concat",
|
| 533 |
+
"target": "gain",
|
| 534 |
+
"strategy": "weighted",
|
| 535 |
+
"representation_dim": 512,
|
| 536 |
+
"hidden_layer": "query_generation__layer_01",
|
| 537 |
+
"fixed25_metric": 0.654765618353515,
|
| 538 |
+
"fixed25_actual_fraction": 0.33953435612398264,
|
| 539 |
+
"fixed25_rescue": 1510,
|
| 540 |
+
"fixed25_harm": 2030,
|
| 541 |
+
"rank25_metric": 0.6655985026798628,
|
| 542 |
+
"validation_selection": 0.7589219200224893,
|
| 543 |
+
"best_epoch": 3,
|
| 544 |
+
"seconds": 163.15981340408325
|
| 545 |
+
}
|
| 546 |
+
]
|
| 547 |
+
},
|
| 548 |
+
{
|
| 549 |
+
"key": [
|
| 550 |
+
"grefcoco",
|
| 551 |
+
"instancevg",
|
| 552 |
+
"Qwen3.8"
|
| 553 |
+
],
|
| 554 |
+
"rows": [
|
| 555 |
+
{
|
| 556 |
+
"id": "formal_0199",
|
| 557 |
+
"tag": "baseline_25pct",
|
| 558 |
+
"task": "grefcoco",
|
| 559 |
+
"expert": "instancevg",
|
| 560 |
+
"endpoint": "Qwen3.8",
|
| 561 |
+
"fraction": 0.25,
|
| 562 |
+
"student": "4M",
|
| 563 |
+
"branches": "output",
|
| 564 |
+
"fusion": "concat",
|
| 565 |
+
"target": "gain",
|
| 566 |
+
"strategy": "natural",
|
| 567 |
+
"representation_dim": 512,
|
| 568 |
+
"hidden_layer": "lateral_features__layer_01",
|
| 569 |
+
"fixed25_metric": 0.7098375067351048,
|
| 570 |
+
"fixed25_actual_fraction": 0.10637211808411083,
|
| 571 |
+
"fixed25_rescue": 1492,
|
| 572 |
+
"fixed25_harm": 70,
|
| 573 |
+
"rank25_metric": 0.7019538893457732,
|
| 574 |
+
"validation_selection": 0.7565746011666316,
|
| 575 |
+
"best_epoch": 1,
|
| 576 |
+
"seconds": 14.837297677993774
|
| 577 |
+
},
|
| 578 |
+
{
|
| 579 |
+
"id": "formal_0200",
|
| 580 |
+
"tag": "baseline_50pct",
|
| 581 |
+
"task": "grefcoco",
|
| 582 |
+
"expert": "instancevg",
|
| 583 |
+
"endpoint": "Qwen3.8",
|
| 584 |
+
"fraction": 0.5,
|
| 585 |
+
"student": "4M",
|
| 586 |
+
"branches": "output",
|
| 587 |
+
"fusion": "concat",
|
| 588 |
+
"target": "gain",
|
| 589 |
+
"strategy": "natural",
|
| 590 |
+
"representation_dim": 512,
|
| 591 |
+
"hidden_layer": "lateral_features__layer_01",
|
| 592 |
+
"fixed25_metric": 0.7416555596517597,
|
| 593 |
+
"fixed25_actual_fraction": 0.4823752942177353,
|
| 594 |
+
"fixed25_rescue": 3418,
|
| 595 |
+
"fixed25_harm": 874,
|
| 596 |
+
"rank25_metric": 0.7207838244051838,
|
| 597 |
+
"validation_selection": 0.8144985592803429,
|
| 598 |
+
"best_epoch": 1,
|
| 599 |
+
"seconds": 21.554462432861328
|
| 600 |
+
},
|
| 601 |
+
{
|
| 602 |
+
"id": "formal_0083",
|
| 603 |
+
"tag": "baseline_100pct",
|
| 604 |
+
"task": "grefcoco",
|
| 605 |
+
"expert": "instancevg",
|
| 606 |
+
"endpoint": "Qwen3.8",
|
| 607 |
+
"fraction": 1.0,
|
| 608 |
+
"student": "4M",
|
| 609 |
+
"branches": "output",
|
| 610 |
+
"fusion": "concat",
|
| 611 |
+
"target": "gain",
|
| 612 |
+
"strategy": "natural",
|
| 613 |
+
"representation_dim": 512,
|
| 614 |
+
"hidden_layer": "lateral_features__layer_01",
|
| 615 |
+
"fixed25_metric": 0.7090151149930523,
|
| 616 |
+
"fixed25_actual_fraction": 0.12330204463602076,
|
| 617 |
+
"fixed25_rescue": 1514,
|
| 618 |
+
"fixed25_harm": 121,
|
| 619 |
+
"rank25_metric": 0.7021523976973031,
|
| 620 |
+
"validation_selection": 0.7565746011666316,
|
| 621 |
+
"best_epoch": 1,
|
| 622 |
+
"seconds": 121.36263680458069
|
| 623 |
+
}
|
| 624 |
+
]
|
| 625 |
+
},
|
| 626 |
+
{
|
| 627 |
+
"key": [
|
| 628 |
+
"nlvr2",
|
| 629 |
+
"nlvr2",
|
| 630 |
+
"MiniCPM"
|
| 631 |
+
],
|
| 632 |
+
"rows": [
|
| 633 |
+
{
|
| 634 |
+
"id": "formal_0201",
|
| 635 |
+
"tag": "baseline_25pct",
|
| 636 |
+
"task": "nlvr2",
|
| 637 |
+
"expert": "nlvr2",
|
| 638 |
+
"endpoint": "MiniCPM",
|
| 639 |
+
"fraction": 0.25,
|
| 640 |
+
"student": "4M",
|
| 641 |
+
"branches": "hidden+confidence",
|
| 642 |
+
"fusion": "concat",
|
| 643 |
+
"target": "gain",
|
| 644 |
+
"strategy": "natural",
|
| 645 |
+
"representation_dim": 512,
|
| 646 |
+
"hidden_layer": "multimodal_encoder__layer_12",
|
| 647 |
+
"fixed25_metric": 0.863815742268781,
|
| 648 |
+
"fixed25_actual_fraction": 0.253928392049939,
|
| 649 |
+
"fixed25_rescue": 1100,
|
| 650 |
+
"fixed25_harm": 211,
|
| 651 |
+
"rank25_metric": 0.8630264762861448,
|
| 652 |
+
"validation_selection": 0.8331853337152678,
|
| 653 |
+
"best_epoch": 2,
|
| 654 |
+
"seconds": 11.065314054489136
|
| 655 |
+
},
|
| 656 |
+
{
|
| 657 |
+
"id": "formal_0202",
|
| 658 |
+
"tag": "baseline_50pct",
|
| 659 |
+
"task": "nlvr2",
|
| 660 |
+
"expert": "nlvr2",
|
| 661 |
+
"endpoint": "MiniCPM",
|
| 662 |
+
"fraction": 0.5,
|
| 663 |
+
"student": "4M",
|
| 664 |
+
"branches": "hidden+confidence",
|
| 665 |
+
"fusion": "concat",
|
| 666 |
+
"target": "gain",
|
| 667 |
+
"strategy": "natural",
|
| 668 |
+
"representation_dim": 512,
|
| 669 |
+
"hidden_layer": "multimodal_encoder__layer_12",
|
| 670 |
+
"fixed25_metric": 0.868336083805697,
|
| 671 |
+
"fixed25_actual_fraction": 0.24388318863456984,
|
| 672 |
+
"fixed25_rescue": 1152,
|
| 673 |
+
"fixed25_harm": 200,
|
| 674 |
+
"rank25_metric": 0.8689818468823993,
|
| 675 |
+
"validation_selection": 0.8330850759094816,
|
| 676 |
+
"best_epoch": 3,
|
| 677 |
+
"seconds": 16.297266960144043
|
| 678 |
+
},
|
| 679 |
+
{
|
| 680 |
+
"id": "formal_0091",
|
| 681 |
+
"tag": "baseline_100pct",
|
| 682 |
+
"task": "nlvr2",
|
| 683 |
+
"expert": "nlvr2",
|
| 684 |
+
"endpoint": "MiniCPM",
|
| 685 |
+
"fraction": 1.0,
|
| 686 |
+
"student": "4M",
|
| 687 |
+
"branches": "hidden+confidence",
|
| 688 |
+
"fusion": "concat",
|
| 689 |
+
"target": "gain",
|
| 690 |
+
"strategy": "natural",
|
| 691 |
+
"representation_dim": 512,
|
| 692 |
+
"hidden_layer": "multimodal_encoder__layer_12",
|
| 693 |
+
"fixed25_metric": 0.8707756332065725,
|
| 694 |
+
"fixed25_actual_fraction": 0.26117528879959817,
|
| 695 |
+
"fixed25_rescue": 1197,
|
| 696 |
+
"fixed25_harm": 211,
|
| 697 |
+
"rank25_metric": 0.8689100954294324,
|
| 698 |
+
"validation_selection": 0.8335577198510455,
|
| 699 |
+
"best_epoch": 6,
|
| 700 |
+
"seconds": 82.51407408714294
|
| 701 |
+
}
|
| 702 |
+
]
|
| 703 |
+
},
|
| 704 |
+
{
|
| 705 |
+
"key": [
|
| 706 |
+
"nlvr2",
|
| 707 |
+
"nlvr2",
|
| 708 |
+
"Qwen3.8"
|
| 709 |
+
],
|
| 710 |
+
"rows": [
|
| 711 |
+
{
|
| 712 |
+
"id": "formal_0203",
|
| 713 |
+
"tag": "baseline_25pct",
|
| 714 |
+
"task": "nlvr2",
|
| 715 |
+
"expert": "nlvr2",
|
| 716 |
+
"endpoint": "Qwen3.8",
|
| 717 |
+
"fraction": 0.25,
|
| 718 |
+
"student": "4M",
|
| 719 |
+
"branches": "output+confidence",
|
| 720 |
+
"fusion": "concat",
|
| 721 |
+
"target": "gain",
|
| 722 |
+
"strategy": "weighted",
|
| 723 |
+
"representation_dim": 512,
|
| 724 |
+
"hidden_layer": "multimodal_encoder__layer_10",
|
| 725 |
+
"fixed25_metric": 0.8717801535481093,
|
| 726 |
+
"fixed25_actual_fraction": 0.25299562316136903,
|
| 727 |
+
"fixed25_rescue": 1180,
|
| 728 |
+
"fixed25_harm": 180,
|
| 729 |
+
"rank25_metric": 0.8714213962832748,
|
| 730 |
+
"validation_selection": 0.8382268690919508,
|
| 731 |
+
"best_epoch": 3,
|
| 732 |
+
"seconds": 12.40796971321106
|
| 733 |
+
},
|
| 734 |
+
{
|
| 735 |
+
"id": "formal_0204",
|
| 736 |
+
"tag": "baseline_50pct",
|
| 737 |
+
"task": "nlvr2",
|
| 738 |
+
"expert": "nlvr2",
|
| 739 |
+
"endpoint": "Qwen3.8",
|
| 740 |
+
"fraction": 0.5,
|
| 741 |
+
"student": "4M",
|
| 742 |
+
"branches": "output+confidence",
|
| 743 |
+
"fusion": "concat",
|
| 744 |
+
"target": "gain",
|
| 745 |
+
"strategy": "weighted",
|
| 746 |
+
"representation_dim": 512,
|
| 747 |
+
"hidden_layer": "multimodal_encoder__layer_10",
|
| 748 |
+
"fixed25_metric": 0.872067159359977,
|
| 749 |
+
"fixed25_actual_fraction": 0.25407189495587285,
|
| 750 |
+
"fixed25_rescue": 1178,
|
| 751 |
+
"fixed25_harm": 174,
|
| 752 |
+
"rank25_metric": 0.8703451244887709,
|
| 753 |
+
"validation_selection": 0.8385276425093097,
|
| 754 |
+
"best_epoch": 11,
|
| 755 |
+
"seconds": 27.56421136856079
|
| 756 |
+
},
|
| 757 |
+
{
|
| 758 |
+
"id": "formal_0104",
|
| 759 |
+
"tag": "baseline_100pct",
|
| 760 |
+
"task": "nlvr2",
|
| 761 |
+
"expert": "nlvr2",
|
| 762 |
+
"endpoint": "Qwen3.8",
|
| 763 |
+
"fraction": 1.0,
|
| 764 |
+
"student": "4M",
|
| 765 |
+
"branches": "output+confidence",
|
| 766 |
+
"fusion": "concat",
|
| 767 |
+
"target": "gain",
|
| 768 |
+
"strategy": "weighted",
|
| 769 |
+
"representation_dim": 512,
|
| 770 |
+
"hidden_layer": "multimodal_encoder__layer_10",
|
| 771 |
+
"fixed25_metric": 0.8709191361125063,
|
| 772 |
+
"fixed25_actual_fraction": 0.25400014350290595,
|
| 773 |
+
"fixed25_rescue": 1153,
|
| 774 |
+
"fixed25_harm": 165,
|
| 775 |
+
"rank25_metric": 0.8695558585061347,
|
| 776 |
+
"validation_selection": 0.8383414494414207,
|
| 777 |
+
"best_epoch": 7,
|
| 778 |
+
"seconds": 91.79227209091187
|
| 779 |
+
}
|
| 780 |
+
]
|
| 781 |
+
},
|
| 782 |
+
{
|
| 783 |
+
"key": [
|
| 784 |
+
"nlvr2",
|
| 785 |
+
"vilt",
|
| 786 |
+
"MiniCPM"
|
| 787 |
+
],
|
| 788 |
+
"rows": [
|
| 789 |
+
{
|
| 790 |
+
"id": "formal_0209",
|
| 791 |
+
"tag": "baseline_25pct",
|
| 792 |
+
"task": "nlvr2",
|
| 793 |
+
"expert": "vilt",
|
| 794 |
+
"endpoint": "MiniCPM",
|
| 795 |
+
"fraction": 0.25,
|
| 796 |
+
"student": "8M",
|
| 797 |
+
"branches": "image+text+hidden+confidence",
|
| 798 |
+
"fusion": "attention",
|
| 799 |
+
"target": "four_state",
|
| 800 |
+
"strategy": "natural",
|
| 801 |
+
"representation_dim": 512,
|
| 802 |
+
"hidden_layer": "ordered_image_text__layer_12",
|
| 803 |
+
"fixed25_metric": 0.8323886058692689,
|
| 804 |
+
"fixed25_actual_fraction": 0.24072612470402527,
|
| 805 |
+
"fixed25_rescue": 1189,
|
| 806 |
+
"fixed25_harm": 239,
|
| 807 |
+
"rank25_metric": 0.8343258950993757,
|
| 808 |
+
"validation_selection": 0.7950730449727872,
|
| 809 |
+
"best_epoch": 4,
|
| 810 |
+
"seconds": 18.022375106811523
|
| 811 |
+
},
|
| 812 |
+
{
|
| 813 |
+
"id": "formal_0210",
|
| 814 |
+
"tag": "baseline_50pct",
|
| 815 |
+
"task": "nlvr2",
|
| 816 |
+
"expert": "vilt",
|
| 817 |
+
"endpoint": "MiniCPM",
|
| 818 |
+
"fraction": 0.5,
|
| 819 |
+
"student": "8M",
|
| 820 |
+
"branches": "image+text+hidden+confidence",
|
| 821 |
+
"fusion": "attention",
|
| 822 |
+
"target": "four_state",
|
| 823 |
+
"strategy": "natural",
|
| 824 |
+
"representation_dim": 512,
|
| 825 |
+
"hidden_layer": "ordered_image_text__layer_12",
|
| 826 |
+
"fixed25_metric": 0.8337518834756403,
|
| 827 |
+
"fixed25_actual_fraction": 0.24704025256511444,
|
| 828 |
+
"fixed25_rescue": 1213,
|
| 829 |
+
"fixed25_harm": 244,
|
| 830 |
+
"rank25_metric": 0.8344693980053096,
|
| 831 |
+
"validation_selection": 0.7959610426811802,
|
| 832 |
+
"best_epoch": 8,
|
| 833 |
+
"seconds": 38.606887102127075
|
| 834 |
+
},
|
| 835 |
+
{
|
| 836 |
+
"id": "formal_0139",
|
| 837 |
+
"tag": "baseline_100pct",
|
| 838 |
+
"task": "nlvr2",
|
| 839 |
+
"expert": "vilt",
|
| 840 |
+
"endpoint": "MiniCPM",
|
| 841 |
+
"fraction": 1.0,
|
| 842 |
+
"student": "8M",
|
| 843 |
+
"branches": "image+text+hidden+confidence",
|
| 844 |
+
"fusion": "attention",
|
| 845 |
+
"target": "four_state",
|
| 846 |
+
"strategy": "natural",
|
| 847 |
+
"representation_dim": 512,
|
| 848 |
+
"hidden_layer": "ordered_image_text__layer_12",
|
| 849 |
+
"fixed25_metric": 0.8327473631341035,
|
| 850 |
+
"fixed25_actual_fraction": 0.24208940231039677,
|
| 851 |
+
"fixed25_rescue": 1196,
|
| 852 |
+
"fixed25_harm": 241,
|
| 853 |
+
"rank25_metric": 0.8346846523642104,
|
| 854 |
+
"validation_selection": 0.7952162704096247,
|
| 855 |
+
"best_epoch": 3,
|
| 856 |
+
"seconds": 136.20692539215088
|
| 857 |
+
}
|
| 858 |
+
]
|
| 859 |
+
},
|
| 860 |
+
{
|
| 861 |
+
"key": [
|
| 862 |
+
"nlvr2",
|
| 863 |
+
"vilt",
|
| 864 |
+
"Qwen3.8"
|
| 865 |
+
],
|
| 866 |
+
"rows": [
|
| 867 |
+
{
|
| 868 |
+
"id": "formal_0211",
|
| 869 |
+
"tag": "baseline_25pct",
|
| 870 |
+
"task": "nlvr2",
|
| 871 |
+
"expert": "vilt",
|
| 872 |
+
"endpoint": "Qwen3.8",
|
| 873 |
+
"fraction": 0.25,
|
| 874 |
+
"student": "2M",
|
| 875 |
+
"branches": "image+text+hidden+confidence",
|
| 876 |
+
"fusion": "concat",
|
| 877 |
+
"target": "gain",
|
| 878 |
+
"strategy": "weighted",
|
| 879 |
+
"representation_dim": 512,
|
| 880 |
+
"hidden_layer": "ordered_image_text__layer_10",
|
| 881 |
+
"fixed25_metric": 0.8336083805697065,
|
| 882 |
+
"fixed25_actual_fraction": 0.24431369735237138,
|
| 883 |
+
"fixed25_rescue": 1234,
|
| 884 |
+
"fixed25_harm": 267,
|
| 885 |
+
"rank25_metric": 0.8346846523642104,
|
| 886 |
+
"validation_selection": 0.7945717559438557,
|
| 887 |
+
"best_epoch": 2,
|
| 888 |
+
"seconds": 12.189603567123413
|
| 889 |
+
},
|
| 890 |
+
{
|
| 891 |
+
"id": "formal_0212",
|
| 892 |
+
"tag": "baseline_50pct",
|
| 893 |
+
"task": "nlvr2",
|
| 894 |
+
"expert": "vilt",
|
| 895 |
+
"endpoint": "Qwen3.8",
|
| 896 |
+
"fraction": 0.5,
|
| 897 |
+
"student": "2M",
|
| 898 |
+
"branches": "image+text+hidden+confidence",
|
| 899 |
+
"fusion": "concat",
|
| 900 |
+
"target": "gain",
|
| 901 |
+
"strategy": "weighted",
|
| 902 |
+
"representation_dim": 512,
|
| 903 |
+
"hidden_layer": "ordered_image_text__layer_10",
|
| 904 |
+
"fixed25_metric": 0.8308818253569635,
|
| 905 |
+
"fixed25_actual_fraction": 0.2361340317141422,
|
| 906 |
+
"fixed25_rescue": 1178,
|
| 907 |
+
"fixed25_harm": 249,
|
| 908 |
+
"rank25_metric": 0.834110640740475,
|
| 909 |
+
"validation_selection": 0.7955027212832999,
|
| 910 |
+
"best_epoch": 3,
|
| 911 |
+
"seconds": 18.873005151748657
|
| 912 |
+
},
|
| 913 |
+
{
|
| 914 |
+
"id": "formal_0154",
|
| 915 |
+
"tag": "baseline_100pct",
|
| 916 |
+
"task": "nlvr2",
|
| 917 |
+
"expert": "vilt",
|
| 918 |
+
"endpoint": "Qwen3.8",
|
| 919 |
+
"fraction": 1.0,
|
| 920 |
+
"student": "2M",
|
| 921 |
+
"branches": "image+text+hidden+confidence",
|
| 922 |
+
"fusion": "concat",
|
| 923 |
+
"target": "gain",
|
| 924 |
+
"strategy": "weighted",
|
| 925 |
+
"representation_dim": 512,
|
| 926 |
+
"hidden_layer": "ordered_image_text__layer_10",
|
| 927 |
+
"fixed25_metric": 0.8329626174930043,
|
| 928 |
+
"fixed25_actual_fraction": 0.24094137906292604,
|
| 929 |
+
"fixed25_rescue": 1225,
|
| 930 |
+
"fixed25_harm": 267,
|
| 931 |
+
"rank25_metric": 0.8348281552701442,
|
| 932 |
+
"validation_selection": 0.7948438842738471,
|
| 933 |
+
"best_epoch": 2,
|
| 934 |
+
"seconds": 78.18392181396484
|
| 935 |
+
}
|
| 936 |
+
]
|
| 937 |
+
},
|
| 938 |
+
{
|
| 939 |
+
"key": [
|
| 940 |
+
"construction",
|
| 941 |
+
"yolo26x",
|
| 942 |
+
"MiniCPM"
|
| 943 |
+
],
|
| 944 |
+
"rows": [
|
| 945 |
+
{
|
| 946 |
+
"id": "formal_0213",
|
| 947 |
+
"tag": "baseline_25pct",
|
| 948 |
+
"task": "construction",
|
| 949 |
+
"expert": "yolo26x",
|
| 950 |
+
"endpoint": "MiniCPM",
|
| 951 |
+
"fraction": 0.25,
|
| 952 |
+
"student": "4M",
|
| 953 |
+
"branches": "output",
|
| 954 |
+
"fusion": "concat",
|
| 955 |
+
"target": "gain",
|
| 956 |
+
"strategy": "weighted",
|
| 957 |
+
"representation_dim": 512,
|
| 958 |
+
"hidden_layer": "feature_blocks__layer_14",
|
| 959 |
+
"fixed25_metric": 0.2225771535665814,
|
| 960 |
+
"fixed25_actual_fraction": 0.18342210386151797,
|
| 961 |
+
"fixed25_rescue": 608,
|
| 962 |
+
"fixed25_harm": 265,
|
| 963 |
+
"rank25_metric": 0.22250151381741792,
|
| 964 |
+
"validation_selection": 0.245059255252362,
|
| 965 |
+
"best_epoch": 2,
|
| 966 |
+
"seconds": 2.557858467102051
|
| 967 |
+
},
|
| 968 |
+
{
|
| 969 |
+
"id": "formal_0214",
|
| 970 |
+
"tag": "baseline_50pct",
|
| 971 |
+
"task": "construction",
|
| 972 |
+
"expert": "yolo26x",
|
| 973 |
+
"endpoint": "MiniCPM",
|
| 974 |
+
"fraction": 0.5,
|
| 975 |
+
"student": "4M",
|
| 976 |
+
"branches": "output",
|
| 977 |
+
"fusion": "concat",
|
| 978 |
+
"target": "gain",
|
| 979 |
+
"strategy": "weighted",
|
| 980 |
+
"representation_dim": 512,
|
| 981 |
+
"hidden_layer": "feature_blocks__layer_14",
|
| 982 |
+
"fixed25_metric": 0.2225771535665814,
|
| 983 |
+
"fixed25_actual_fraction": 0.18342210386151797,
|
| 984 |
+
"fixed25_rescue": 608,
|
| 985 |
+
"fixed25_harm": 265,
|
| 986 |
+
"rank25_metric": 0.22250151381741792,
|
| 987 |
+
"validation_selection": 0.2411747600492345,
|
| 988 |
+
"best_epoch": 1,
|
| 989 |
+
"seconds": 3.123697280883789
|
| 990 |
+
},
|
| 991 |
+
{
|
| 992 |
+
"id": "formal_0169",
|
| 993 |
+
"tag": "baseline_100pct",
|
| 994 |
+
"task": "construction",
|
| 995 |
+
"expert": "yolo26x",
|
| 996 |
+
"endpoint": "MiniCPM",
|
| 997 |
+
"fraction": 1.0,
|
| 998 |
+
"student": "4M",
|
| 999 |
+
"branches": "output",
|
| 1000 |
+
"fusion": "concat",
|
| 1001 |
+
"target": "gain",
|
| 1002 |
+
"strategy": "weighted",
|
| 1003 |
+
"representation_dim": 512,
|
| 1004 |
+
"hidden_layer": "feature_blocks__layer_14",
|
| 1005 |
+
"fixed25_metric": 0.22276943862299337,
|
| 1006 |
+
"fixed25_actual_fraction": 0.18874833555259654,
|
| 1007 |
+
"fixed25_rescue": 614,
|
| 1008 |
+
"fixed25_harm": 271,
|
| 1009 |
+
"rank25_metric": 0.22272342953859628,
|
| 1010 |
+
"validation_selection": 0.23173775721145673,
|
| 1011 |
+
"best_epoch": 4,
|
| 1012 |
+
"seconds": 9.216264009475708
|
| 1013 |
+
}
|
| 1014 |
+
]
|
| 1015 |
+
},
|
| 1016 |
+
{
|
| 1017 |
+
"key": [
|
| 1018 |
+
"construction",
|
| 1019 |
+
"yolo26x",
|
| 1020 |
+
"Qwen3.8"
|
| 1021 |
+
],
|
| 1022 |
+
"rows": [
|
| 1023 |
+
{
|
| 1024 |
+
"id": "formal_0215",
|
| 1025 |
+
"tag": "baseline_25pct",
|
| 1026 |
+
"task": "construction",
|
| 1027 |
+
"expert": "yolo26x",
|
| 1028 |
+
"endpoint": "Qwen3.8",
|
| 1029 |
+
"fraction": 0.25,
|
| 1030 |
+
"student": "4M",
|
| 1031 |
+
"branches": "output",
|
| 1032 |
+
"fusion": "concat",
|
| 1033 |
+
"target": "gain",
|
| 1034 |
+
"strategy": "weighted",
|
| 1035 |
+
"representation_dim": 512,
|
| 1036 |
+
"hidden_layer": "feature_blocks__layer_14",
|
| 1037 |
+
"fixed25_metric": 0.2503793990148833,
|
| 1038 |
+
"fixed25_actual_fraction": 0.18342210386151797,
|
| 1039 |
+
"fixed25_rescue": 653,
|
| 1040 |
+
"fixed25_harm": 116,
|
| 1041 |
+
"rank25_metric": 0.26327645391521537,
|
| 1042 |
+
"validation_selection": 0.2490976644605033,
|
| 1043 |
+
"best_epoch": 7,
|
| 1044 |
+
"seconds": 3.354280948638916
|
| 1045 |
+
},
|
| 1046 |
+
{
|
| 1047 |
+
"id": "formal_0216",
|
| 1048 |
+
"tag": "baseline_50pct",
|
| 1049 |
+
"task": "construction",
|
| 1050 |
+
"expert": "yolo26x",
|
| 1051 |
+
"endpoint": "Qwen3.8",
|
| 1052 |
+
"fraction": 0.5,
|
| 1053 |
+
"student": "4M",
|
| 1054 |
+
"branches": "output",
|
| 1055 |
+
"fusion": "concat",
|
| 1056 |
+
"target": "gain",
|
| 1057 |
+
"strategy": "weighted",
|
| 1058 |
+
"representation_dim": 512,
|
| 1059 |
+
"hidden_layer": "feature_blocks__layer_14",
|
| 1060 |
+
"fixed25_metric": 0.2503793990148833,
|
| 1061 |
+
"fixed25_actual_fraction": 0.18342210386151797,
|
| 1062 |
+
"fixed25_rescue": 653,
|
| 1063 |
+
"fixed25_harm": 116,
|
| 1064 |
+
"rank25_metric": 0.26327645391521537,
|
| 1065 |
+
"validation_selection": 0.2490976644605033,
|
| 1066 |
+
"best_epoch": 14,
|
| 1067 |
+
"seconds": 4.801910877227783
|
| 1068 |
+
},
|
| 1069 |
+
{
|
| 1070 |
+
"id": "formal_0177",
|
| 1071 |
+
"tag": "baseline_100pct",
|
| 1072 |
+
"task": "construction",
|
| 1073 |
+
"expert": "yolo26x",
|
| 1074 |
+
"endpoint": "Qwen3.8",
|
| 1075 |
+
"fraction": 1.0,
|
| 1076 |
+
"student": "4M",
|
| 1077 |
+
"branches": "output",
|
| 1078 |
+
"fusion": "concat",
|
| 1079 |
+
"target": "gain",
|
| 1080 |
+
"strategy": "weighted",
|
| 1081 |
+
"representation_dim": 512,
|
| 1082 |
+
"hidden_layer": "feature_blocks__layer_14",
|
| 1083 |
+
"fixed25_metric": 0.2503793990148833,
|
| 1084 |
+
"fixed25_actual_fraction": 0.18342210386151797,
|
| 1085 |
+
"fixed25_rescue": 653,
|
| 1086 |
+
"fixed25_harm": 116,
|
| 1087 |
+
"rank25_metric": 0.26327645391521537,
|
| 1088 |
+
"validation_selection": 0.24929743985848513,
|
| 1089 |
+
"best_epoch": 13,
|
| 1090 |
+
"seconds": 11.173577785491943
|
| 1091 |
+
}
|
| 1092 |
+
]
|
| 1093 |
+
},
|
| 1094 |
+
{
|
| 1095 |
+
"key": [
|
| 1096 |
+
"construction",
|
| 1097 |
+
"rtdetr_x_fp32",
|
| 1098 |
+
"MiniCPM"
|
| 1099 |
+
],
|
| 1100 |
+
"rows": [
|
| 1101 |
+
{
|
| 1102 |
+
"id": "formal_0205",
|
| 1103 |
+
"tag": "baseline_25pct",
|
| 1104 |
+
"task": "construction",
|
| 1105 |
+
"expert": "rtdetr_x_fp32",
|
| 1106 |
+
"endpoint": "MiniCPM",
|
| 1107 |
+
"fraction": 0.25,
|
| 1108 |
+
"student": "4M",
|
| 1109 |
+
"branches": "image+text+hidden+confidence",
|
| 1110 |
+
"fusion": "concat",
|
| 1111 |
+
"target": "gain",
|
| 1112 |
+
"strategy": "weighted",
|
| 1113 |
+
"representation_dim": 512,
|
| 1114 |
+
"hidden_layer": "query_decoder__layer_04",
|
| 1115 |
+
"fixed25_metric": 0.12161748372078302,
|
| 1116 |
+
"fixed25_actual_fraction": 0.26897470039946736,
|
| 1117 |
+
"fixed25_rescue": 1103,
|
| 1118 |
+
"fixed25_harm": 518,
|
| 1119 |
+
"rank25_metric": 0.12103479129839005,
|
| 1120 |
+
"validation_selection": 0.13942018347651794,
|
| 1121 |
+
"best_epoch": 1,
|
| 1122 |
+
"seconds": 3.9248127937316895
|
| 1123 |
+
},
|
| 1124 |
+
{
|
| 1125 |
+
"id": "formal_0206",
|
| 1126 |
+
"tag": "baseline_50pct",
|
| 1127 |
+
"task": "construction",
|
| 1128 |
+
"expert": "rtdetr_x_fp32",
|
| 1129 |
+
"endpoint": "MiniCPM",
|
| 1130 |
+
"fraction": 0.5,
|
| 1131 |
+
"student": "4M",
|
| 1132 |
+
"branches": "image+text+hidden+confidence",
|
| 1133 |
+
"fusion": "concat",
|
| 1134 |
+
"target": "gain",
|
| 1135 |
+
"strategy": "weighted",
|
| 1136 |
+
"representation_dim": 512,
|
| 1137 |
+
"hidden_layer": "query_decoder__layer_04",
|
| 1138 |
+
"fixed25_metric": 0.11138731664705415,
|
| 1139 |
+
"fixed25_actual_fraction": 0.25798934753661784,
|
| 1140 |
+
"fixed25_rescue": 1337,
|
| 1141 |
+
"fixed25_harm": 464,
|
| 1142 |
+
"rank25_metric": 0.11157434949576527,
|
| 1143 |
+
"validation_selection": 0.10286119045745137,
|
| 1144 |
+
"best_epoch": 3,
|
| 1145 |
+
"seconds": 5.573187828063965
|
| 1146 |
+
},
|
| 1147 |
+
{
|
| 1148 |
+
"id": "formal_0114",
|
| 1149 |
+
"tag": "baseline_100pct",
|
| 1150 |
+
"task": "construction",
|
| 1151 |
+
"expert": "rtdetr_x_fp32",
|
| 1152 |
+
"endpoint": "MiniCPM",
|
| 1153 |
+
"fraction": 1.0,
|
| 1154 |
+
"student": "4M",
|
| 1155 |
+
"branches": "image+text+hidden+confidence",
|
| 1156 |
+
"fusion": "concat",
|
| 1157 |
+
"target": "gain",
|
| 1158 |
+
"strategy": "weighted",
|
| 1159 |
+
"representation_dim": 512,
|
| 1160 |
+
"hidden_layer": "query_decoder__layer_04",
|
| 1161 |
+
"fixed25_metric": 0.11409050364241882,
|
| 1162 |
+
"fixed25_actual_fraction": 0.2333555259653795,
|
| 1163 |
+
"fixed25_rescue": 1155,
|
| 1164 |
+
"fixed25_harm": 428,
|
| 1165 |
+
"rank25_metric": 0.1142018021781044,
|
| 1166 |
+
"validation_selection": 0.09196748903785351,
|
| 1167 |
+
"best_epoch": 1,
|
| 1168 |
+
"seconds": 7.1226747035980225
|
| 1169 |
+
}
|
| 1170 |
+
]
|
| 1171 |
+
},
|
| 1172 |
+
{
|
| 1173 |
+
"key": [
|
| 1174 |
+
"construction",
|
| 1175 |
+
"rtdetr_x_fp32",
|
| 1176 |
+
"Qwen3.8"
|
| 1177 |
+
],
|
| 1178 |
+
"rows": [
|
| 1179 |
+
{
|
| 1180 |
+
"id": "formal_0207",
|
| 1181 |
+
"tag": "baseline_25pct",
|
| 1182 |
+
"task": "construction",
|
| 1183 |
+
"expert": "rtdetr_x_fp32",
|
| 1184 |
+
"endpoint": "Qwen3.8",
|
| 1185 |
+
"fraction": 0.25,
|
| 1186 |
+
"student": "4M",
|
| 1187 |
+
"branches": "hidden",
|
| 1188 |
+
"fusion": "concat",
|
| 1189 |
+
"target": "error",
|
| 1190 |
+
"strategy": "balanced",
|
| 1191 |
+
"representation_dim": 512,
|
| 1192 |
+
"hidden_layer": "feature_blocks__layer_11",
|
| 1193 |
+
"fixed25_metric": 0.14471609214350373,
|
| 1194 |
+
"fixed25_actual_fraction": 0.2892809587217044,
|
| 1195 |
+
"fixed25_rescue": 1383,
|
| 1196 |
+
"fixed25_harm": 258,
|
| 1197 |
+
"rank25_metric": 0.14346432207238047,
|
| 1198 |
+
"validation_selection": 0.16233096000534752,
|
| 1199 |
+
"best_epoch": 17,
|
| 1200 |
+
"seconds": 8.028136253356934
|
| 1201 |
+
},
|
| 1202 |
+
{
|
| 1203 |
+
"id": "formal_0208",
|
| 1204 |
+
"tag": "baseline_50pct",
|
| 1205 |
+
"task": "construction",
|
| 1206 |
+
"expert": "rtdetr_x_fp32",
|
| 1207 |
+
"endpoint": "Qwen3.8",
|
| 1208 |
+
"fraction": 0.5,
|
| 1209 |
+
"student": "4M",
|
| 1210 |
+
"branches": "hidden",
|
| 1211 |
+
"fusion": "concat",
|
| 1212 |
+
"target": "error",
|
| 1213 |
+
"strategy": "balanced",
|
| 1214 |
+
"representation_dim": 512,
|
| 1215 |
+
"hidden_layer": "feature_blocks__layer_11",
|
| 1216 |
+
"fixed25_metric": 0.1499641421353766,
|
| 1217 |
+
"fixed25_actual_fraction": 0.26797603195739017,
|
| 1218 |
+
"fixed25_rescue": 1348,
|
| 1219 |
+
"fixed25_harm": 225,
|
| 1220 |
+
"rank25_metric": 0.14991013725904712,
|
| 1221 |
+
"validation_selection": 0.14960536227871302,
|
| 1222 |
+
"best_epoch": 14,
|
| 1223 |
+
"seconds": 7.761091709136963
|
| 1224 |
+
},
|
| 1225 |
+
{
|
| 1226 |
+
"id": "formal_0129",
|
| 1227 |
+
"tag": "baseline_100pct",
|
| 1228 |
+
"task": "construction",
|
| 1229 |
+
"expert": "rtdetr_x_fp32",
|
| 1230 |
+
"endpoint": "Qwen3.8",
|
| 1231 |
+
"fraction": 1.0,
|
| 1232 |
+
"student": "4M",
|
| 1233 |
+
"branches": "hidden",
|
| 1234 |
+
"fusion": "concat",
|
| 1235 |
+
"target": "error",
|
| 1236 |
+
"strategy": "balanced",
|
| 1237 |
+
"representation_dim": 512,
|
| 1238 |
+
"hidden_layer": "feature_blocks__layer_11",
|
| 1239 |
+
"fixed25_metric": 0.1498069310931111,
|
| 1240 |
+
"fixed25_actual_fraction": 0.2539946737683089,
|
| 1241 |
+
"fixed25_rescue": 1284,
|
| 1242 |
+
"fixed25_harm": 203,
|
| 1243 |
+
"rank25_metric": 0.15030144907600507,
|
| 1244 |
+
"validation_selection": 0.14151521056562297,
|
| 1245 |
+
"best_epoch": 16,
|
| 1246 |
+
"seconds": 13.48578929901123
|
| 1247 |
+
}
|
| 1248 |
+
]
|
| 1249 |
+
}
|
| 1250 |
+
]
|
paper_results/data/shared_search.json
ADDED
|
@@ -0,0 +1,720 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"status": "search_complete",
|
| 3 |
+
"started_at": 1789041525.1799183,
|
| 4 |
+
"completed_stages": [
|
| 5 |
+
"hidden_percent",
|
| 6 |
+
"student",
|
| 7 |
+
"representation_dim",
|
| 8 |
+
"lr",
|
| 9 |
+
"target",
|
| 10 |
+
"strategy"
|
| 11 |
+
],
|
| 12 |
+
"selected": "shared_12",
|
| 13 |
+
"candidates": [
|
| 14 |
+
{
|
| 15 |
+
"id": "shared_00",
|
| 16 |
+
"settings": {
|
| 17 |
+
"student": "4M",
|
| 18 |
+
"representation_dim": 512,
|
| 19 |
+
"lr": 0.001,
|
| 20 |
+
"target": "four_state",
|
| 21 |
+
"strategy": "natural",
|
| 22 |
+
"hidden_percent": 90
|
| 23 |
+
},
|
| 24 |
+
"runs": [
|
| 25 |
+
"shared_00_cub_MiniCPM",
|
| 26 |
+
"shared_00_cub_Qwen3.8",
|
| 27 |
+
"shared_00_glsim_MiniCPM",
|
| 28 |
+
"shared_00_glsim_Qwen3.8",
|
| 29 |
+
"shared_00_groundingdino_MiniCPM",
|
| 30 |
+
"shared_00_groundingdino_Qwen3.8",
|
| 31 |
+
"shared_00_instancevg_MiniCPM",
|
| 32 |
+
"shared_00_instancevg_Qwen3.8",
|
| 33 |
+
"shared_00_nlvr2_MiniCPM",
|
| 34 |
+
"shared_00_nlvr2_Qwen3.8",
|
| 35 |
+
"shared_00_vilt_MiniCPM",
|
| 36 |
+
"shared_00_vilt_Qwen3.8",
|
| 37 |
+
"shared_00_yolo26x_MiniCPM",
|
| 38 |
+
"shared_00_yolo26x_Qwen3.8",
|
| 39 |
+
"shared_00_rtdetr_x_fp32_MiniCPM",
|
| 40 |
+
"shared_00_rtdetr_x_fp32_Qwen3.8"
|
| 41 |
+
],
|
| 42 |
+
"status": "complete",
|
| 43 |
+
"task_means": {
|
| 44 |
+
"cub": 0.9498541666666667,
|
| 45 |
+
"grefcoco": 0.7090097687820648,
|
| 46 |
+
"nlvr2": 0.8164207963334288,
|
| 47 |
+
"construction": 0.175986962717605
|
| 48 |
+
},
|
| 49 |
+
"validation_mean": 0.6628179236249413,
|
| 50 |
+
"mean_active_parameters": 4881924.0,
|
| 51 |
+
"mean_fit_seconds": 37.63353890167855
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
"id": "shared_01",
|
| 55 |
+
"settings": {
|
| 56 |
+
"student": "4M",
|
| 57 |
+
"representation_dim": 512,
|
| 58 |
+
"lr": 0.001,
|
| 59 |
+
"target": "four_state",
|
| 60 |
+
"strategy": "natural",
|
| 61 |
+
"hidden_percent": 30
|
| 62 |
+
},
|
| 63 |
+
"runs": [
|
| 64 |
+
"shared_01_cub_MiniCPM",
|
| 65 |
+
"shared_01_cub_Qwen3.8",
|
| 66 |
+
"shared_01_glsim_MiniCPM",
|
| 67 |
+
"shared_01_glsim_Qwen3.8",
|
| 68 |
+
"shared_01_groundingdino_MiniCPM",
|
| 69 |
+
"shared_01_groundingdino_Qwen3.8",
|
| 70 |
+
"shared_01_instancevg_MiniCPM",
|
| 71 |
+
"shared_01_instancevg_Qwen3.8",
|
| 72 |
+
"shared_01_nlvr2_MiniCPM",
|
| 73 |
+
"shared_01_nlvr2_Qwen3.8",
|
| 74 |
+
"shared_01_vilt_MiniCPM",
|
| 75 |
+
"shared_01_vilt_Qwen3.8",
|
| 76 |
+
"shared_01_yolo26x_MiniCPM",
|
| 77 |
+
"shared_01_yolo26x_Qwen3.8",
|
| 78 |
+
"shared_01_rtdetr_x_fp32_MiniCPM",
|
| 79 |
+
"shared_01_rtdetr_x_fp32_Qwen3.8"
|
| 80 |
+
],
|
| 81 |
+
"status": "complete",
|
| 82 |
+
"task_means": {
|
| 83 |
+
"cub": 0.9346041666666667,
|
| 84 |
+
"grefcoco": 0.7090431513107035,
|
| 85 |
+
"nlvr2": 0.8157691205958177,
|
| 86 |
+
"construction": 0.18298715773892013
|
| 87 |
+
},
|
| 88 |
+
"validation_mean": 0.660600899078027,
|
| 89 |
+
"mean_active_parameters": 4783620.0,
|
| 90 |
+
"mean_fit_seconds": 39.49221354418114
|
| 91 |
+
},
|
| 92 |
+
{
|
| 93 |
+
"id": "shared_02",
|
| 94 |
+
"settings": {
|
| 95 |
+
"student": "4M",
|
| 96 |
+
"representation_dim": 512,
|
| 97 |
+
"lr": 0.001,
|
| 98 |
+
"target": "four_state",
|
| 99 |
+
"strategy": "natural",
|
| 100 |
+
"hidden_percent": 60
|
| 101 |
+
},
|
| 102 |
+
"runs": [
|
| 103 |
+
"shared_02_cub_MiniCPM",
|
| 104 |
+
"shared_02_cub_Qwen3.8",
|
| 105 |
+
"shared_02_glsim_MiniCPM",
|
| 106 |
+
"shared_02_glsim_Qwen3.8",
|
| 107 |
+
"shared_02_groundingdino_MiniCPM",
|
| 108 |
+
"shared_02_groundingdino_Qwen3.8",
|
| 109 |
+
"shared_02_instancevg_MiniCPM",
|
| 110 |
+
"shared_02_instancevg_Qwen3.8",
|
| 111 |
+
"shared_02_nlvr2_MiniCPM",
|
| 112 |
+
"shared_02_nlvr2_Qwen3.8",
|
| 113 |
+
"shared_02_vilt_MiniCPM",
|
| 114 |
+
"shared_02_vilt_Qwen3.8",
|
| 115 |
+
"shared_02_yolo26x_MiniCPM",
|
| 116 |
+
"shared_02_yolo26x_Qwen3.8",
|
| 117 |
+
"shared_02_rtdetr_x_fp32_MiniCPM",
|
| 118 |
+
"shared_02_rtdetr_x_fp32_Qwen3.8"
|
| 119 |
+
],
|
| 120 |
+
"status": "complete",
|
| 121 |
+
"task_means": {
|
| 122 |
+
"cub": 0.9472291666666667,
|
| 123 |
+
"grefcoco": 0.708183990442055,
|
| 124 |
+
"nlvr2": 0.8155900887997709,
|
| 125 |
+
"construction": 0.1758169352446146
|
| 126 |
+
},
|
| 127 |
+
"validation_mean": 0.6617050452882768,
|
| 128 |
+
"mean_active_parameters": 4816388.0,
|
| 129 |
+
"mean_fit_seconds": 34.18548409075447
|
| 130 |
+
},
|
| 131 |
+
{
|
| 132 |
+
"id": "shared_03",
|
| 133 |
+
"settings": {
|
| 134 |
+
"student": "1M",
|
| 135 |
+
"representation_dim": 512,
|
| 136 |
+
"lr": 0.001,
|
| 137 |
+
"target": "four_state",
|
| 138 |
+
"strategy": "natural",
|
| 139 |
+
"hidden_percent": 90
|
| 140 |
+
},
|
| 141 |
+
"runs": [
|
| 142 |
+
"shared_03_cub_MiniCPM",
|
| 143 |
+
"shared_03_cub_Qwen3.8",
|
| 144 |
+
"shared_03_glsim_MiniCPM",
|
| 145 |
+
"shared_03_glsim_Qwen3.8",
|
| 146 |
+
"shared_03_groundingdino_MiniCPM",
|
| 147 |
+
"shared_03_groundingdino_Qwen3.8",
|
| 148 |
+
"shared_03_instancevg_MiniCPM",
|
| 149 |
+
"shared_03_instancevg_Qwen3.8",
|
| 150 |
+
"shared_03_nlvr2_MiniCPM",
|
| 151 |
+
"shared_03_nlvr2_Qwen3.8",
|
| 152 |
+
"shared_03_vilt_MiniCPM",
|
| 153 |
+
"shared_03_vilt_Qwen3.8",
|
| 154 |
+
"shared_03_yolo26x_MiniCPM",
|
| 155 |
+
"shared_03_yolo26x_Qwen3.8",
|
| 156 |
+
"shared_03_rtdetr_x_fp32_MiniCPM",
|
| 157 |
+
"shared_03_rtdetr_x_fp32_Qwen3.8"
|
| 158 |
+
],
|
| 159 |
+
"status": "complete",
|
| 160 |
+
"task_means": {
|
| 161 |
+
"cub": 0.9559375,
|
| 162 |
+
"grefcoco": 0.7093348091924943,
|
| 163 |
+
"nlvr2": 0.8166929246634202,
|
| 164 |
+
"construction": 0.173092415345378
|
| 165 |
+
},
|
| 166 |
+
"validation_mean": 0.6637644123003231,
|
| 167 |
+
"mean_active_parameters": 2615822.0,
|
| 168 |
+
"mean_fit_seconds": 57.72181743498368
|
| 169 |
+
},
|
| 170 |
+
{
|
| 171 |
+
"id": "shared_04",
|
| 172 |
+
"settings": {
|
| 173 |
+
"student": "2M",
|
| 174 |
+
"representation_dim": 512,
|
| 175 |
+
"lr": 0.001,
|
| 176 |
+
"target": "four_state",
|
| 177 |
+
"strategy": "natural",
|
| 178 |
+
"hidden_percent": 90
|
| 179 |
+
},
|
| 180 |
+
"runs": [
|
| 181 |
+
"shared_04_cub_MiniCPM",
|
| 182 |
+
"shared_04_cub_Qwen3.8",
|
| 183 |
+
"shared_04_glsim_MiniCPM",
|
| 184 |
+
"shared_04_glsim_Qwen3.8",
|
| 185 |
+
"shared_04_groundingdino_MiniCPM",
|
| 186 |
+
"shared_04_groundingdino_Qwen3.8",
|
| 187 |
+
"shared_04_instancevg_MiniCPM",
|
| 188 |
+
"shared_04_instancevg_Qwen3.8",
|
| 189 |
+
"shared_04_nlvr2_MiniCPM",
|
| 190 |
+
"shared_04_nlvr2_Qwen3.8",
|
| 191 |
+
"shared_04_vilt_MiniCPM",
|
| 192 |
+
"shared_04_vilt_Qwen3.8",
|
| 193 |
+
"shared_04_yolo26x_MiniCPM",
|
| 194 |
+
"shared_04_yolo26x_Qwen3.8",
|
| 195 |
+
"shared_04_rtdetr_x_fp32_MiniCPM",
|
| 196 |
+
"shared_04_rtdetr_x_fp32_Qwen3.8"
|
| 197 |
+
],
|
| 198 |
+
"status": "complete",
|
| 199 |
+
"task_means": {
|
| 200 |
+
"cub": 0.9448958333333333,
|
| 201 |
+
"grefcoco": 0.7087286527514232,
|
| 202 |
+
"nlvr2": 0.8167144084789458,
|
| 203 |
+
"construction": 0.1764152469711854
|
| 204 |
+
},
|
| 205 |
+
"validation_mean": 0.661688535383722,
|
| 206 |
+
"mean_active_parameters": 3402222.0,
|
| 207 |
+
"mean_fit_seconds": 57.64163886100141
|
| 208 |
+
},
|
| 209 |
+
{
|
| 210 |
+
"id": "shared_05",
|
| 211 |
+
"settings": {
|
| 212 |
+
"student": "8M",
|
| 213 |
+
"representation_dim": 512,
|
| 214 |
+
"lr": 0.001,
|
| 215 |
+
"target": "four_state",
|
| 216 |
+
"strategy": "natural",
|
| 217 |
+
"hidden_percent": 90
|
| 218 |
+
},
|
| 219 |
+
"runs": [
|
| 220 |
+
"shared_05_cub_MiniCPM",
|
| 221 |
+
"shared_05_cub_Qwen3.8",
|
| 222 |
+
"shared_05_glsim_MiniCPM",
|
| 223 |
+
"shared_05_glsim_Qwen3.8",
|
| 224 |
+
"shared_05_groundingdino_MiniCPM",
|
| 225 |
+
"shared_05_groundingdino_Qwen3.8",
|
| 226 |
+
"shared_05_instancevg_MiniCPM",
|
| 227 |
+
"shared_05_instancevg_Qwen3.8",
|
| 228 |
+
"shared_05_nlvr2_MiniCPM",
|
| 229 |
+
"shared_05_nlvr2_Qwen3.8",
|
| 230 |
+
"shared_05_vilt_MiniCPM",
|
| 231 |
+
"shared_05_vilt_Qwen3.8",
|
| 232 |
+
"shared_05_yolo26x_MiniCPM",
|
| 233 |
+
"shared_05_yolo26x_Qwen3.8",
|
| 234 |
+
"shared_05_rtdetr_x_fp32_MiniCPM",
|
| 235 |
+
"shared_05_rtdetr_x_fp32_Qwen3.8"
|
| 236 |
+
],
|
| 237 |
+
"status": "complete",
|
| 238 |
+
"task_means": {
|
| 239 |
+
"cub": 0.9542916666666666,
|
| 240 |
+
"grefcoco": 0.7095351043643263,
|
| 241 |
+
"nlvr2": 0.8168110856488112,
|
| 242 |
+
"construction": 0.17955466204144832
|
| 243 |
+
},
|
| 244 |
+
"validation_mean": 0.6650481296803131,
|
| 245 |
+
"mean_active_parameters": 7994758.0,
|
| 246 |
+
"mean_fit_seconds": 46.836150376082514
|
| 247 |
+
},
|
| 248 |
+
{
|
| 249 |
+
"id": "shared_06",
|
| 250 |
+
"settings": {
|
| 251 |
+
"student": "8M",
|
| 252 |
+
"representation_dim": 32,
|
| 253 |
+
"lr": 0.001,
|
| 254 |
+
"target": "four_state",
|
| 255 |
+
"strategy": "natural",
|
| 256 |
+
"hidden_percent": 90
|
| 257 |
+
},
|
| 258 |
+
"runs": [
|
| 259 |
+
"shared_06_cub_MiniCPM",
|
| 260 |
+
"shared_06_cub_Qwen3.8",
|
| 261 |
+
"shared_06_glsim_MiniCPM",
|
| 262 |
+
"shared_06_glsim_Qwen3.8",
|
| 263 |
+
"shared_06_groundingdino_MiniCPM",
|
| 264 |
+
"shared_06_groundingdino_Qwen3.8",
|
| 265 |
+
"shared_06_instancevg_MiniCPM",
|
| 266 |
+
"shared_06_instancevg_Qwen3.8",
|
| 267 |
+
"shared_06_nlvr2_MiniCPM",
|
| 268 |
+
"shared_06_nlvr2_Qwen3.8",
|
| 269 |
+
"shared_06_vilt_MiniCPM",
|
| 270 |
+
"shared_06_vilt_Qwen3.8",
|
| 271 |
+
"shared_06_yolo26x_MiniCPM",
|
| 272 |
+
"shared_06_yolo26x_Qwen3.8",
|
| 273 |
+
"shared_06_rtdetr_x_fp32_MiniCPM",
|
| 274 |
+
"shared_06_rtdetr_x_fp32_Qwen3.8"
|
| 275 |
+
],
|
| 276 |
+
"status": "complete",
|
| 277 |
+
"task_means": {
|
| 278 |
+
"cub": 0.966125,
|
| 279 |
+
"grefcoco": 0.7070665542202544,
|
| 280 |
+
"nlvr2": 0.8169149240905185,
|
| 281 |
+
"construction": 0.17258963439949287
|
| 282 |
+
},
|
| 283 |
+
"validation_mean": 0.6656740281775665,
|
| 284 |
+
"mean_active_parameters": 6222718.0,
|
| 285 |
+
"mean_fit_seconds": 63.611915712936025
|
| 286 |
+
},
|
| 287 |
+
{
|
| 288 |
+
"id": "shared_07",
|
| 289 |
+
"settings": {
|
| 290 |
+
"student": "8M",
|
| 291 |
+
"representation_dim": 64,
|
| 292 |
+
"lr": 0.001,
|
| 293 |
+
"target": "four_state",
|
| 294 |
+
"strategy": "natural",
|
| 295 |
+
"hidden_percent": 90
|
| 296 |
+
},
|
| 297 |
+
"runs": [
|
| 298 |
+
"shared_07_cub_MiniCPM",
|
| 299 |
+
"shared_07_cub_Qwen3.8",
|
| 300 |
+
"shared_07_glsim_MiniCPM",
|
| 301 |
+
"shared_07_glsim_Qwen3.8",
|
| 302 |
+
"shared_07_groundingdino_MiniCPM",
|
| 303 |
+
"shared_07_groundingdino_Qwen3.8",
|
| 304 |
+
"shared_07_instancevg_MiniCPM",
|
| 305 |
+
"shared_07_instancevg_Qwen3.8",
|
| 306 |
+
"shared_07_nlvr2_MiniCPM",
|
| 307 |
+
"shared_07_nlvr2_Qwen3.8",
|
| 308 |
+
"shared_07_vilt_MiniCPM",
|
| 309 |
+
"shared_07_vilt_Qwen3.8",
|
| 310 |
+
"shared_07_yolo26x_MiniCPM",
|
| 311 |
+
"shared_07_yolo26x_Qwen3.8",
|
| 312 |
+
"shared_07_rtdetr_x_fp32_MiniCPM",
|
| 313 |
+
"shared_07_rtdetr_x_fp32_Qwen3.8"
|
| 314 |
+
],
|
| 315 |
+
"status": "complete",
|
| 316 |
+
"task_means": {
|
| 317 |
+
"cub": 0.9693333333333334,
|
| 318 |
+
"grefcoco": 0.7083755007379295,
|
| 319 |
+
"nlvr2": 0.8167716986536809,
|
| 320 |
+
"construction": 0.17603867039292925
|
| 321 |
+
},
|
| 322 |
+
"validation_mean": 0.6676298007794683,
|
| 323 |
+
"mean_active_parameters": 6340854.0,
|
| 324 |
+
"mean_fit_seconds": 62.082761554156605
|
| 325 |
+
},
|
| 326 |
+
{
|
| 327 |
+
"id": "shared_08",
|
| 328 |
+
"settings": {
|
| 329 |
+
"student": "8M",
|
| 330 |
+
"representation_dim": 128,
|
| 331 |
+
"lr": 0.001,
|
| 332 |
+
"target": "four_state",
|
| 333 |
+
"strategy": "natural",
|
| 334 |
+
"hidden_percent": 90
|
| 335 |
+
},
|
| 336 |
+
"runs": [
|
| 337 |
+
"shared_08_cub_MiniCPM",
|
| 338 |
+
"shared_08_cub_Qwen3.8",
|
| 339 |
+
"shared_08_glsim_MiniCPM",
|
| 340 |
+
"shared_08_glsim_Qwen3.8",
|
| 341 |
+
"shared_08_groundingdino_MiniCPM",
|
| 342 |
+
"shared_08_groundingdino_Qwen3.8",
|
| 343 |
+
"shared_08_instancevg_MiniCPM",
|
| 344 |
+
"shared_08_instancevg_Qwen3.8",
|
| 345 |
+
"shared_08_nlvr2_MiniCPM",
|
| 346 |
+
"shared_08_nlvr2_Qwen3.8",
|
| 347 |
+
"shared_08_vilt_MiniCPM",
|
| 348 |
+
"shared_08_vilt_Qwen3.8",
|
| 349 |
+
"shared_08_yolo26x_MiniCPM",
|
| 350 |
+
"shared_08_yolo26x_Qwen3.8",
|
| 351 |
+
"shared_08_rtdetr_x_fp32_MiniCPM",
|
| 352 |
+
"shared_08_rtdetr_x_fp32_Qwen3.8"
|
| 353 |
+
],
|
| 354 |
+
"status": "complete",
|
| 355 |
+
"task_means": {
|
| 356 |
+
"cub": 0.9704166666666667,
|
| 357 |
+
"grefcoco": 0.7082156159955021,
|
| 358 |
+
"nlvr2": 0.8169006015468347,
|
| 359 |
+
"construction": 0.17553875298671762
|
| 360 |
+
},
|
| 361 |
+
"validation_mean": 0.6677679092989303,
|
| 362 |
+
"mean_active_parameters": 6577126.0,
|
| 363 |
+
"mean_fit_seconds": 58.37819891100662
|
| 364 |
+
},
|
| 365 |
+
{
|
| 366 |
+
"id": "shared_09",
|
| 367 |
+
"settings": {
|
| 368 |
+
"student": "8M",
|
| 369 |
+
"representation_dim": 256,
|
| 370 |
+
"lr": 0.001,
|
| 371 |
+
"target": "four_state",
|
| 372 |
+
"strategy": "natural",
|
| 373 |
+
"hidden_percent": 90
|
| 374 |
+
},
|
| 375 |
+
"runs": [
|
| 376 |
+
"shared_09_cub_MiniCPM",
|
| 377 |
+
"shared_09_cub_Qwen3.8",
|
| 378 |
+
"shared_09_glsim_MiniCPM",
|
| 379 |
+
"shared_09_glsim_Qwen3.8",
|
| 380 |
+
"shared_09_groundingdino_MiniCPM",
|
| 381 |
+
"shared_09_groundingdino_Qwen3.8",
|
| 382 |
+
"shared_09_instancevg_MiniCPM",
|
| 383 |
+
"shared_09_instancevg_Qwen3.8",
|
| 384 |
+
"shared_09_nlvr2_MiniCPM",
|
| 385 |
+
"shared_09_nlvr2_Qwen3.8",
|
| 386 |
+
"shared_09_vilt_MiniCPM",
|
| 387 |
+
"shared_09_vilt_Qwen3.8",
|
| 388 |
+
"shared_09_yolo26x_MiniCPM",
|
| 389 |
+
"shared_09_yolo26x_Qwen3.8",
|
| 390 |
+
"shared_09_rtdetr_x_fp32_MiniCPM",
|
| 391 |
+
"shared_09_rtdetr_x_fp32_Qwen3.8"
|
| 392 |
+
],
|
| 393 |
+
"status": "complete",
|
| 394 |
+
"task_means": {
|
| 395 |
+
"cub": 0.9678541666666667,
|
| 396 |
+
"grefcoco": 0.7090659919881931,
|
| 397 |
+
"nlvr2": 0.8167609567459181,
|
| 398 |
+
"construction": 0.17265650809871438
|
| 399 |
+
},
|
| 400 |
+
"validation_mean": 0.6665844058748731,
|
| 401 |
+
"mean_active_parameters": 7049670.0,
|
| 402 |
+
"mean_fit_seconds": 50.47316628225235
|
| 403 |
+
},
|
| 404 |
+
{
|
| 405 |
+
"id": "shared_10",
|
| 406 |
+
"settings": {
|
| 407 |
+
"student": "8M",
|
| 408 |
+
"representation_dim": 128,
|
| 409 |
+
"lr": 0.0001,
|
| 410 |
+
"target": "four_state",
|
| 411 |
+
"strategy": "natural",
|
| 412 |
+
"hidden_percent": 90
|
| 413 |
+
},
|
| 414 |
+
"runs": [
|
| 415 |
+
"shared_10_cub_MiniCPM",
|
| 416 |
+
"shared_10_cub_Qwen3.8",
|
| 417 |
+
"shared_10_glsim_MiniCPM",
|
| 418 |
+
"shared_10_glsim_Qwen3.8",
|
| 419 |
+
"shared_10_groundingdino_MiniCPM",
|
| 420 |
+
"shared_10_groundingdino_Qwen3.8",
|
| 421 |
+
"shared_10_instancevg_MiniCPM",
|
| 422 |
+
"shared_10_instancevg_Qwen3.8",
|
| 423 |
+
"shared_10_nlvr2_MiniCPM",
|
| 424 |
+
"shared_10_nlvr2_Qwen3.8",
|
| 425 |
+
"shared_10_vilt_MiniCPM",
|
| 426 |
+
"shared_10_vilt_Qwen3.8",
|
| 427 |
+
"shared_10_yolo26x_MiniCPM",
|
| 428 |
+
"shared_10_yolo26x_Qwen3.8",
|
| 429 |
+
"shared_10_rtdetr_x_fp32_MiniCPM",
|
| 430 |
+
"shared_10_rtdetr_x_fp32_Qwen3.8"
|
| 431 |
+
],
|
| 432 |
+
"status": "complete",
|
| 433 |
+
"task_means": {
|
| 434 |
+
"cub": 0.9694166666666667,
|
| 435 |
+
"grefcoco": 0.7047965422728231,
|
| 436 |
+
"nlvr2": 0.8161844743626469,
|
| 437 |
+
"construction": 0.17403212405686308
|
| 438 |
+
},
|
| 439 |
+
"validation_mean": 0.66610745183975,
|
| 440 |
+
"mean_active_parameters": 6577126.0,
|
| 441 |
+
"mean_fit_seconds": 91.61671701988234
|
| 442 |
+
},
|
| 443 |
+
{
|
| 444 |
+
"id": "shared_11",
|
| 445 |
+
"settings": {
|
| 446 |
+
"student": "8M",
|
| 447 |
+
"representation_dim": 128,
|
| 448 |
+
"lr": 0.0003,
|
| 449 |
+
"target": "four_state",
|
| 450 |
+
"strategy": "natural",
|
| 451 |
+
"hidden_percent": 90
|
| 452 |
+
},
|
| 453 |
+
"runs": [
|
| 454 |
+
"shared_11_cub_MiniCPM",
|
| 455 |
+
"shared_11_cub_Qwen3.8",
|
| 456 |
+
"shared_11_glsim_MiniCPM",
|
| 457 |
+
"shared_11_glsim_Qwen3.8",
|
| 458 |
+
"shared_11_groundingdino_MiniCPM",
|
| 459 |
+
"shared_11_groundingdino_Qwen3.8",
|
| 460 |
+
"shared_11_instancevg_MiniCPM",
|
| 461 |
+
"shared_11_instancevg_Qwen3.8",
|
| 462 |
+
"shared_11_nlvr2_MiniCPM",
|
| 463 |
+
"shared_11_nlvr2_Qwen3.8",
|
| 464 |
+
"shared_11_vilt_MiniCPM",
|
| 465 |
+
"shared_11_vilt_Qwen3.8",
|
| 466 |
+
"shared_11_yolo26x_MiniCPM",
|
| 467 |
+
"shared_11_yolo26x_Qwen3.8",
|
| 468 |
+
"shared_11_rtdetr_x_fp32_MiniCPM",
|
| 469 |
+
"shared_11_rtdetr_x_fp32_Qwen3.8"
|
| 470 |
+
],
|
| 471 |
+
"status": "complete",
|
| 472 |
+
"task_means": {
|
| 473 |
+
"cub": 0.9692083333333333,
|
| 474 |
+
"grefcoco": 0.7071210204511913,
|
| 475 |
+
"nlvr2": 0.8166034087653967,
|
| 476 |
+
"construction": 0.17173346213113433
|
| 477 |
+
},
|
| 478 |
+
"validation_mean": 0.6661665561702639,
|
| 479 |
+
"mean_active_parameters": 6577126.0,
|
| 480 |
+
"mean_fit_seconds": 75.3759171601705
|
| 481 |
+
},
|
| 482 |
+
{
|
| 483 |
+
"id": "shared_12",
|
| 484 |
+
"settings": {
|
| 485 |
+
"student": "8M",
|
| 486 |
+
"representation_dim": 128,
|
| 487 |
+
"lr": 0.003,
|
| 488 |
+
"target": "four_state",
|
| 489 |
+
"strategy": "natural",
|
| 490 |
+
"hidden_percent": 90
|
| 491 |
+
},
|
| 492 |
+
"runs": [
|
| 493 |
+
"shared_12_cub_MiniCPM",
|
| 494 |
+
"shared_12_cub_Qwen3.8",
|
| 495 |
+
"shared_12_glsim_MiniCPM",
|
| 496 |
+
"shared_12_glsim_Qwen3.8",
|
| 497 |
+
"shared_12_groundingdino_MiniCPM",
|
| 498 |
+
"shared_12_groundingdino_Qwen3.8",
|
| 499 |
+
"shared_12_instancevg_MiniCPM",
|
| 500 |
+
"shared_12_instancevg_Qwen3.8",
|
| 501 |
+
"shared_12_nlvr2_MiniCPM",
|
| 502 |
+
"shared_12_nlvr2_Qwen3.8",
|
| 503 |
+
"shared_12_vilt_MiniCPM",
|
| 504 |
+
"shared_12_vilt_Qwen3.8",
|
| 505 |
+
"shared_12_yolo26x_MiniCPM",
|
| 506 |
+
"shared_12_yolo26x_Qwen3.8",
|
| 507 |
+
"shared_12_rtdetr_x_fp32_MiniCPM",
|
| 508 |
+
"shared_12_rtdetr_x_fp32_Qwen3.8"
|
| 509 |
+
],
|
| 510 |
+
"status": "complete",
|
| 511 |
+
"task_means": {
|
| 512 |
+
"cub": 0.9698125,
|
| 513 |
+
"grefcoco": 0.7082964368543117,
|
| 514 |
+
"nlvr2": 0.8164959896877686,
|
| 515 |
+
"construction": 0.1768778191883302
|
| 516 |
+
},
|
| 517 |
+
"validation_mean": 0.6678706864326026,
|
| 518 |
+
"mean_active_parameters": 6577126.0,
|
| 519 |
+
"mean_fit_seconds": 67.41994623254868
|
| 520 |
+
},
|
| 521 |
+
{
|
| 522 |
+
"id": "shared_13",
|
| 523 |
+
"settings": {
|
| 524 |
+
"student": "8M",
|
| 525 |
+
"representation_dim": 128,
|
| 526 |
+
"lr": 0.01,
|
| 527 |
+
"target": "four_state",
|
| 528 |
+
"strategy": "natural",
|
| 529 |
+
"hidden_percent": 90
|
| 530 |
+
},
|
| 531 |
+
"runs": [
|
| 532 |
+
"shared_13_cub_MiniCPM",
|
| 533 |
+
"shared_13_cub_Qwen3.8",
|
| 534 |
+
"shared_13_glsim_MiniCPM",
|
| 535 |
+
"shared_13_glsim_Qwen3.8",
|
| 536 |
+
"shared_13_groundingdino_MiniCPM",
|
| 537 |
+
"shared_13_groundingdino_Qwen3.8",
|
| 538 |
+
"shared_13_instancevg_MiniCPM",
|
| 539 |
+
"shared_13_instancevg_Qwen3.8",
|
| 540 |
+
"shared_13_nlvr2_MiniCPM",
|
| 541 |
+
"shared_13_nlvr2_Qwen3.8",
|
| 542 |
+
"shared_13_vilt_MiniCPM",
|
| 543 |
+
"shared_13_vilt_Qwen3.8",
|
| 544 |
+
"shared_13_yolo26x_MiniCPM",
|
| 545 |
+
"shared_13_yolo26x_Qwen3.8",
|
| 546 |
+
"shared_13_rtdetr_x_fp32_MiniCPM",
|
| 547 |
+
"shared_13_rtdetr_x_fp32_Qwen3.8"
|
| 548 |
+
],
|
| 549 |
+
"status": "complete",
|
| 550 |
+
"task_means": {
|
| 551 |
+
"cub": 0.9509375,
|
| 552 |
+
"grefcoco": 0.7100024597652681,
|
| 553 |
+
"nlvr2": 0.8169936980807792,
|
| 554 |
+
"construction": 0.17572600052962756
|
| 555 |
+
},
|
| 556 |
+
"validation_mean": 0.6634149145939187,
|
| 557 |
+
"mean_active_parameters": 6577126.0,
|
| 558 |
+
"mean_fit_seconds": 58.952753572441
|
| 559 |
+
},
|
| 560 |
+
{
|
| 561 |
+
"id": "shared_14",
|
| 562 |
+
"settings": {
|
| 563 |
+
"student": "8M",
|
| 564 |
+
"representation_dim": 128,
|
| 565 |
+
"lr": 0.003,
|
| 566 |
+
"target": "gain",
|
| 567 |
+
"strategy": "natural",
|
| 568 |
+
"hidden_percent": 90
|
| 569 |
+
},
|
| 570 |
+
"runs": [
|
| 571 |
+
"shared_14_cub_MiniCPM",
|
| 572 |
+
"shared_14_cub_Qwen3.8",
|
| 573 |
+
"shared_14_glsim_MiniCPM",
|
| 574 |
+
"shared_14_glsim_Qwen3.8",
|
| 575 |
+
"shared_14_groundingdino_MiniCPM",
|
| 576 |
+
"shared_14_groundingdino_Qwen3.8",
|
| 577 |
+
"shared_14_instancevg_MiniCPM",
|
| 578 |
+
"shared_14_instancevg_Qwen3.8",
|
| 579 |
+
"shared_14_nlvr2_MiniCPM",
|
| 580 |
+
"shared_14_nlvr2_Qwen3.8",
|
| 581 |
+
"shared_14_vilt_MiniCPM",
|
| 582 |
+
"shared_14_vilt_Qwen3.8",
|
| 583 |
+
"shared_14_yolo26x_MiniCPM",
|
| 584 |
+
"shared_14_yolo26x_Qwen3.8",
|
| 585 |
+
"shared_14_rtdetr_x_fp32_MiniCPM",
|
| 586 |
+
"shared_14_rtdetr_x_fp32_Qwen3.8"
|
| 587 |
+
],
|
| 588 |
+
"status": "complete",
|
| 589 |
+
"task_means": {
|
| 590 |
+
"cub": 0.97975,
|
| 591 |
+
"grefcoco": 0.6943653805608265,
|
| 592 |
+
"nlvr2": 0.815819249498711,
|
| 593 |
+
"construction": 0.17119724948093312
|
| 594 |
+
},
|
| 595 |
+
"validation_mean": 0.6652829698851177,
|
| 596 |
+
"mean_active_parameters": 6576739.0,
|
| 597 |
+
"mean_fit_seconds": 46.42365771972982
|
| 598 |
+
},
|
| 599 |
+
{
|
| 600 |
+
"id": "shared_15",
|
| 601 |
+
"settings": {
|
| 602 |
+
"student": "8M",
|
| 603 |
+
"representation_dim": 128,
|
| 604 |
+
"lr": 0.003,
|
| 605 |
+
"target": "error",
|
| 606 |
+
"strategy": "natural",
|
| 607 |
+
"hidden_percent": 90
|
| 608 |
+
},
|
| 609 |
+
"runs": [
|
| 610 |
+
"shared_15_cub_MiniCPM",
|
| 611 |
+
"shared_15_cub_Qwen3.8",
|
| 612 |
+
"shared_15_glsim_MiniCPM",
|
| 613 |
+
"shared_15_glsim_Qwen3.8",
|
| 614 |
+
"shared_15_groundingdino_MiniCPM",
|
| 615 |
+
"shared_15_groundingdino_Qwen3.8",
|
| 616 |
+
"shared_15_instancevg_MiniCPM",
|
| 617 |
+
"shared_15_instancevg_Qwen3.8",
|
| 618 |
+
"shared_15_nlvr2_MiniCPM",
|
| 619 |
+
"shared_15_nlvr2_Qwen3.8",
|
| 620 |
+
"shared_15_vilt_MiniCPM",
|
| 621 |
+
"shared_15_vilt_Qwen3.8",
|
| 622 |
+
"shared_15_yolo26x_MiniCPM",
|
| 623 |
+
"shared_15_yolo26x_Qwen3.8",
|
| 624 |
+
"shared_15_rtdetr_x_fp32_MiniCPM",
|
| 625 |
+
"shared_15_rtdetr_x_fp32_Qwen3.8"
|
| 626 |
+
],
|
| 627 |
+
"status": "complete",
|
| 628 |
+
"task_means": {
|
| 629 |
+
"cub": 0.9172083333333333,
|
| 630 |
+
"grefcoco": 0.707660411834985,
|
| 631 |
+
"nlvr2": 0.8152570896591235,
|
| 632 |
+
"construction": 0.17163397920287538
|
| 633 |
+
},
|
| 634 |
+
"validation_mean": 0.6529399535075793,
|
| 635 |
+
"mean_active_parameters": 6576739.0,
|
| 636 |
+
"mean_fit_seconds": 39.98559669695533
|
| 637 |
+
},
|
| 638 |
+
{
|
| 639 |
+
"id": "shared_16",
|
| 640 |
+
"settings": {
|
| 641 |
+
"student": "8M",
|
| 642 |
+
"representation_dim": 128,
|
| 643 |
+
"lr": 0.003,
|
| 644 |
+
"target": "four_state",
|
| 645 |
+
"strategy": "weighted",
|
| 646 |
+
"hidden_percent": 90
|
| 647 |
+
},
|
| 648 |
+
"runs": [
|
| 649 |
+
"shared_16_cub_MiniCPM",
|
| 650 |
+
"shared_16_cub_Qwen3.8",
|
| 651 |
+
"shared_16_glsim_MiniCPM",
|
| 652 |
+
"shared_16_glsim_Qwen3.8",
|
| 653 |
+
"shared_16_groundingdino_MiniCPM",
|
| 654 |
+
"shared_16_groundingdino_Qwen3.8",
|
| 655 |
+
"shared_16_instancevg_MiniCPM",
|
| 656 |
+
"shared_16_instancevg_Qwen3.8",
|
| 657 |
+
"shared_16_nlvr2_MiniCPM",
|
| 658 |
+
"shared_16_nlvr2_Qwen3.8",
|
| 659 |
+
"shared_16_vilt_MiniCPM",
|
| 660 |
+
"shared_16_vilt_Qwen3.8",
|
| 661 |
+
"shared_16_yolo26x_MiniCPM",
|
| 662 |
+
"shared_16_yolo26x_Qwen3.8",
|
| 663 |
+
"shared_16_rtdetr_x_fp32_MiniCPM",
|
| 664 |
+
"shared_16_rtdetr_x_fp32_Qwen3.8"
|
| 665 |
+
],
|
| 666 |
+
"status": "complete",
|
| 667 |
+
"task_means": {
|
| 668 |
+
"cub": 0.958625,
|
| 669 |
+
"grefcoco": 0.7108879752617893,
|
| 670 |
+
"nlvr2": 0.8163420223431682,
|
| 671 |
+
"construction": 0.17381623998937223
|
| 672 |
+
},
|
| 673 |
+
"validation_mean": 0.6649178093985824,
|
| 674 |
+
"mean_active_parameters": 6577126.0,
|
| 675 |
+
"mean_fit_seconds": 39.95715373130224
|
| 676 |
+
},
|
| 677 |
+
{
|
| 678 |
+
"id": "shared_17",
|
| 679 |
+
"settings": {
|
| 680 |
+
"student": "8M",
|
| 681 |
+
"representation_dim": 128,
|
| 682 |
+
"lr": 0.003,
|
| 683 |
+
"target": "four_state",
|
| 684 |
+
"strategy": "balanced",
|
| 685 |
+
"hidden_percent": 90
|
| 686 |
+
},
|
| 687 |
+
"runs": [
|
| 688 |
+
"shared_17_cub_MiniCPM",
|
| 689 |
+
"shared_17_cub_Qwen3.8",
|
| 690 |
+
"shared_17_glsim_MiniCPM",
|
| 691 |
+
"shared_17_glsim_Qwen3.8",
|
| 692 |
+
"shared_17_groundingdino_MiniCPM",
|
| 693 |
+
"shared_17_groundingdino_Qwen3.8",
|
| 694 |
+
"shared_17_instancevg_MiniCPM",
|
| 695 |
+
"shared_17_instancevg_Qwen3.8",
|
| 696 |
+
"shared_17_nlvr2_MiniCPM",
|
| 697 |
+
"shared_17_nlvr2_Qwen3.8",
|
| 698 |
+
"shared_17_vilt_MiniCPM",
|
| 699 |
+
"shared_17_vilt_Qwen3.8",
|
| 700 |
+
"shared_17_yolo26x_MiniCPM",
|
| 701 |
+
"shared_17_yolo26x_Qwen3.8",
|
| 702 |
+
"shared_17_rtdetr_x_fp32_MiniCPM",
|
| 703 |
+
"shared_17_rtdetr_x_fp32_Qwen3.8"
|
| 704 |
+
],
|
| 705 |
+
"status": "complete",
|
| 706 |
+
"task_means": {
|
| 707 |
+
"cub": 0.9554583333333334,
|
| 708 |
+
"grefcoco": 0.7062583456321596,
|
| 709 |
+
"nlvr2": 0.8156903466055572,
|
| 710 |
+
"construction": 0.17510649266382114
|
| 711 |
+
},
|
| 712 |
+
"validation_mean": 0.6631283795587178,
|
| 713 |
+
"mean_active_parameters": 6577126.0,
|
| 714 |
+
"mean_fit_seconds": 35.547329766683106
|
| 715 |
+
}
|
| 716 |
+
],
|
| 717 |
+
"failures": [],
|
| 718 |
+
"test_loaded": false,
|
| 719 |
+
"finished_at": 1789043234.371715
|
| 720 |
+
}
|
paper_results/data/table_audit.json
ADDED
|
@@ -0,0 +1,443 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"counts": {
|
| 3 |
+
"better_metric": 11,
|
| 4 |
+
"fewer_calls": 15,
|
| 5 |
+
"both": 10,
|
| 6 |
+
"above_endpoints": 5,
|
| 7 |
+
"better_ranked_area": 15
|
| 8 |
+
},
|
| 9 |
+
"rows": [
|
| 10 |
+
{
|
| 11 |
+
"key": [
|
| 12 |
+
"cub",
|
| 13 |
+
"cub",
|
| 14 |
+
"MiniCPM"
|
| 15 |
+
],
|
| 16 |
+
"validation": {
|
| 17 |
+
"spec": 98.58333333333333,
|
| 18 |
+
"vlm": 27.750000000000004,
|
| 19 |
+
"conf": 98.5,
|
| 20 |
+
"call_conf": 1.0,
|
| 21 |
+
"learned": 98.58333333333333,
|
| 22 |
+
"call_learned": 1.0,
|
| 23 |
+
"change_pp": 0.0,
|
| 24 |
+
"metric_change_pp": 0.08333333333333526
|
| 25 |
+
},
|
| 26 |
+
"test": {
|
| 27 |
+
"spec": 81.429064549534,
|
| 28 |
+
"vlm": 26.561960648947185,
|
| 29 |
+
"conf": 79.9275112185019,
|
| 30 |
+
"call_conf": 11.3565757680359,
|
| 31 |
+
"learned": 81.37728684846392,
|
| 32 |
+
"call_learned": 2.8822920262340355,
|
| 33 |
+
"change_pp": -8.474283741801864,
|
| 34 |
+
"metric_change_pp": 1.4497756299620246
|
| 35 |
+
}
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"key": [
|
| 39 |
+
"cub",
|
| 40 |
+
"cub",
|
| 41 |
+
"Qwen3.8"
|
| 42 |
+
],
|
| 43 |
+
"validation": {
|
| 44 |
+
"spec": 98.58333333333333,
|
| 45 |
+
"vlm": 55.08333333333333,
|
| 46 |
+
"conf": 98.25,
|
| 47 |
+
"call_conf": 1.0,
|
| 48 |
+
"learned": 98.58333333333333,
|
| 49 |
+
"call_learned": 1.0,
|
| 50 |
+
"change_pp": 0.0,
|
| 51 |
+
"metric_change_pp": 0.33333333333332993
|
| 52 |
+
},
|
| 53 |
+
"test": {
|
| 54 |
+
"spec": 81.429064549534,
|
| 55 |
+
"vlm": 54.81532619951675,
|
| 56 |
+
"conf": 81.58439765274423,
|
| 57 |
+
"call_conf": 11.3565757680359,
|
| 58 |
+
"learned": 81.34276838108389,
|
| 59 |
+
"call_learned": 1.4325163962720056,
|
| 60 |
+
"change_pp": -9.924059371763894,
|
| 61 |
+
"metric_change_pp": -0.24162927166033743
|
| 62 |
+
}
|
| 63 |
+
},
|
| 64 |
+
{
|
| 65 |
+
"key": [
|
| 66 |
+
"cub",
|
| 67 |
+
"glsim",
|
| 68 |
+
"MiniCPM"
|
| 69 |
+
],
|
| 70 |
+
"validation": {
|
| 71 |
+
"spec": 98.91666666666666,
|
| 72 |
+
"vlm": 27.750000000000004,
|
| 73 |
+
"conf": 98.5,
|
| 74 |
+
"call_conf": 1.0,
|
| 75 |
+
"learned": 99.0,
|
| 76 |
+
"call_learned": 3.0,
|
| 77 |
+
"change_pp": 1.9999999999999998,
|
| 78 |
+
"metric_change_pp": 0.5000000000000004
|
| 79 |
+
},
|
| 80 |
+
"test": {
|
| 81 |
+
"spec": 90.16223679668623,
|
| 82 |
+
"vlm": 26.561960648947185,
|
| 83 |
+
"conf": 87.69416637901277,
|
| 84 |
+
"call_conf": 8.698653779772178,
|
| 85 |
+
"learned": 89.79979288919571,
|
| 86 |
+
"call_learned": 4.487400759406282,
|
| 87 |
+
"change_pp": -4.211253020365896,
|
| 88 |
+
"metric_change_pp": 2.105626510182945
|
| 89 |
+
}
|
| 90 |
+
},
|
| 91 |
+
{
|
| 92 |
+
"key": [
|
| 93 |
+
"cub",
|
| 94 |
+
"glsim",
|
| 95 |
+
"Qwen3.8"
|
| 96 |
+
],
|
| 97 |
+
"validation": {
|
| 98 |
+
"spec": 98.91666666666666,
|
| 99 |
+
"vlm": 55.08333333333333,
|
| 100 |
+
"conf": 98.58333333333333,
|
| 101 |
+
"call_conf": 1.0,
|
| 102 |
+
"learned": 98.83333333333333,
|
| 103 |
+
"call_learned": 1.0,
|
| 104 |
+
"change_pp": 0.0,
|
| 105 |
+
"metric_change_pp": 0.24999999999999467
|
| 106 |
+
},
|
| 107 |
+
"test": {
|
| 108 |
+
"spec": 90.16223679668623,
|
| 109 |
+
"vlm": 54.81532619951675,
|
| 110 |
+
"conf": 88.78149810148429,
|
| 111 |
+
"call_conf": 8.698653779772178,
|
| 112 |
+
"learned": 90.05868139454608,
|
| 113 |
+
"call_learned": 1.6568864342423197,
|
| 114 |
+
"change_pp": -7.041767345529859,
|
| 115 |
+
"metric_change_pp": 1.27718329306179
|
| 116 |
+
}
|
| 117 |
+
},
|
| 118 |
+
{
|
| 119 |
+
"key": [
|
| 120 |
+
"grefcoco",
|
| 121 |
+
"groundingdino",
|
| 122 |
+
"MiniCPM"
|
| 123 |
+
],
|
| 124 |
+
"validation": {
|
| 125 |
+
"spec": 56.05453650994448,
|
| 126 |
+
"vlm": 64.12256658935975,
|
| 127 |
+
"conf": 63.72197624569541,
|
| 128 |
+
"call_conf": 98.99501019045611,
|
| 129 |
+
"learned": 68.00899571298054,
|
| 130 |
+
"call_learned": 33.993956005341204,
|
| 131 |
+
"change_pp": -65.00105418511491,
|
| 132 |
+
"metric_change_pp": 4.287019467285125
|
| 133 |
+
},
|
| 134 |
+
"test": {
|
| 135 |
+
"spec": 26.688029946402747,
|
| 136 |
+
"vlm": 50.00708958398321,
|
| 137 |
+
"conf": 49.68380455434875,
|
| 138 |
+
"call_conf": 98.73805405098828,
|
| 139 |
+
"learned": 45.622890848764996,
|
| 140 |
+
"call_learned": 43.51870232254772,
|
| 141 |
+
"change_pp": -55.21935172844057,
|
| 142 |
+
"metric_change_pp": -4.060913705583752
|
| 143 |
+
}
|
| 144 |
+
},
|
| 145 |
+
{
|
| 146 |
+
"key": [
|
| 147 |
+
"grefcoco",
|
| 148 |
+
"groundingdino",
|
| 149 |
+
"Qwen3.8"
|
| 150 |
+
],
|
| 151 |
+
"validation": {
|
| 152 |
+
"spec": 56.05453650994448,
|
| 153 |
+
"vlm": 80.60299388572633,
|
| 154 |
+
"conf": 79.94237121371846,
|
| 155 |
+
"call_conf": 98.99501019045611,
|
| 156 |
+
"learned": 81.53067678684377,
|
| 157 |
+
"call_learned": 41.998734977862114,
|
| 158 |
+
"change_pp": -56.99627521259401,
|
| 159 |
+
"metric_change_pp": 1.5883055731253037
|
| 160 |
+
},
|
| 161 |
+
"test": {
|
| 162 |
+
"spec": 26.688029946402747,
|
| 163 |
+
"vlm": 74.47182599325072,
|
| 164 |
+
"conf": 73.87062927147436,
|
| 165 |
+
"call_conf": 98.73805405098828,
|
| 166 |
+
"learned": 66.10044522587415,
|
| 167 |
+
"call_learned": 57.456824433542245,
|
| 168 |
+
"change_pp": -41.281229617446044,
|
| 169 |
+
"metric_change_pp": -7.770184045600203
|
| 170 |
+
}
|
| 171 |
+
},
|
| 172 |
+
{
|
| 173 |
+
"key": [
|
| 174 |
+
"grefcoco",
|
| 175 |
+
"instancevg",
|
| 176 |
+
"MiniCPM"
|
| 177 |
+
],
|
| 178 |
+
"validation": {
|
| 179 |
+
"spec": 75.02986857825567,
|
| 180 |
+
"vlm": 64.12256658935975,
|
| 181 |
+
"conf": 77.16635041113219,
|
| 182 |
+
"call_conf": 64.99402628434886,
|
| 183 |
+
"learned": 77.29988052568699,
|
| 184 |
+
"call_learned": 22.995291306486752,
|
| 185 |
+
"change_pp": -41.99873497786211,
|
| 186 |
+
"metric_change_pp": 0.1335301145547918
|
| 187 |
+
},
|
| 188 |
+
"test": {
|
| 189 |
+
"spec": 66.95119530385956,
|
| 190 |
+
"vlm": 50.00708958398321,
|
| 191 |
+
"conf": 61.877889005473165,
|
| 192 |
+
"call_conf": 52.09426310864078,
|
| 193 |
+
"learned": 66.71865694921021,
|
| 194 |
+
"call_learned": 30.904914499617163,
|
| 195 |
+
"change_pp": -21.189348609023618,
|
| 196 |
+
"metric_change_pp": 4.840767943737056
|
| 197 |
+
}
|
| 198 |
+
},
|
| 199 |
+
{
|
| 200 |
+
"key": [
|
| 201 |
+
"grefcoco",
|
| 202 |
+
"instancevg",
|
| 203 |
+
"Qwen3.8"
|
| 204 |
+
],
|
| 205 |
+
"validation": {
|
| 206 |
+
"spec": 75.02986857825567,
|
| 207 |
+
"vlm": 80.60299388572633,
|
| 208 |
+
"conf": 82.01560193970062,
|
| 209 |
+
"call_conf": 70.99585353854803,
|
| 210 |
+
"learned": 80.81383090870757,
|
| 211 |
+
"call_learned": 55.99831330381615,
|
| 212 |
+
"change_pp": -14.997540234731888,
|
| 213 |
+
"metric_change_pp": -1.2017710309930374
|
| 214 |
+
},
|
| 215 |
+
"test": {
|
| 216 |
+
"spec": 66.95119530385956,
|
| 217 |
+
"vlm": 74.47182599325072,
|
| 218 |
+
"conf": 72.43286163967898,
|
| 219 |
+
"call_conf": 68.38612710206165,
|
| 220 |
+
"learned": 75.28287440093015,
|
| 221 |
+
"call_learned": 53.55755324277571,
|
| 222 |
+
"change_pp": -14.828573859285942,
|
| 223 |
+
"metric_change_pp": 2.850012761251175
|
| 224 |
+
}
|
| 225 |
+
},
|
| 226 |
+
{
|
| 227 |
+
"key": [
|
| 228 |
+
"nlvr2",
|
| 229 |
+
"nlvr2",
|
| 230 |
+
"MiniCPM"
|
| 231 |
+
],
|
| 232 |
+
"validation": {
|
| 233 |
+
"spec": 79.81953594958465,
|
| 234 |
+
"vlm": 89.90260670295045,
|
| 235 |
+
"conf": 90.04583213978803,
|
| 236 |
+
"call_conf": 85.87797192781437,
|
| 237 |
+
"learned": 89.94557433400172,
|
| 238 |
+
"call_learned": 84.98997421942137,
|
| 239 |
+
"change_pp": -0.8879977083930068,
|
| 240 |
+
"metric_change_pp": -0.10025780578630794
|
| 241 |
+
},
|
| 242 |
+
"test": {
|
| 243 |
+
"spec": 80.00287005811867,
|
| 244 |
+
"vlm": 90.7368874219703,
|
| 245 |
+
"conf": 90.77276314845376,
|
| 246 |
+
"call_conf": 85.80038745784601,
|
| 247 |
+
"learned": 90.64361053311329,
|
| 248 |
+
"call_learned": 84.65953935567195,
|
| 249 |
+
"change_pp": -1.1408481021740635,
|
| 250 |
+
"metric_change_pp": -0.12915261534046651
|
| 251 |
+
}
|
| 252 |
+
},
|
| 253 |
+
{
|
| 254 |
+
"key": [
|
| 255 |
+
"nlvr2",
|
| 256 |
+
"nlvr2",
|
| 257 |
+
"Qwen3.8"
|
| 258 |
+
],
|
| 259 |
+
"validation": {
|
| 260 |
+
"spec": 79.81953594958465,
|
| 261 |
+
"vlm": 90.50415353766829,
|
| 262 |
+
"conf": 90.57576625608708,
|
| 263 |
+
"call_conf": 85.87797192781437,
|
| 264 |
+
"learned": 90.9481523918648,
|
| 265 |
+
"call_learned": 76.99799484388427,
|
| 266 |
+
"change_pp": -8.879977083930102,
|
| 267 |
+
"metric_change_pp": 0.3723861357777247
|
| 268 |
+
},
|
| 269 |
+
"test": {
|
| 270 |
+
"spec": 80.00287005811867,
|
| 271 |
+
"vlm": 90.06242376408122,
|
| 272 |
+
"conf": 90.24180239649853,
|
| 273 |
+
"call_conf": 85.80038745784601,
|
| 274 |
+
"learned": 90.88039032790414,
|
| 275 |
+
"call_learned": 76.65925234986008,
|
| 276 |
+
"change_pp": -9.141135107985932,
|
| 277 |
+
"metric_change_pp": 0.6385879314056098
|
| 278 |
+
}
|
| 279 |
+
},
|
| 280 |
+
{
|
| 281 |
+
"key": [
|
| 282 |
+
"nlvr2",
|
| 283 |
+
"vilt",
|
| 284 |
+
"MiniCPM"
|
| 285 |
+
],
|
| 286 |
+
"validation": {
|
| 287 |
+
"spec": 75.56574047550845,
|
| 288 |
+
"vlm": 89.90260670295045,
|
| 289 |
+
"conf": 89.97421942136924,
|
| 290 |
+
"call_conf": 95.9896877685477,
|
| 291 |
+
"learned": 89.8166714408479,
|
| 292 |
+
"call_learned": 98.99742194213692,
|
| 293 |
+
"change_pp": 3.007734173589227,
|
| 294 |
+
"metric_change_pp": -0.15754798052134422
|
| 295 |
+
},
|
| 296 |
+
"test": {
|
| 297 |
+
"spec": 76.42247255506925,
|
| 298 |
+
"vlm": 90.7368874219703,
|
| 299 |
+
"conf": 90.64361053311329,
|
| 300 |
+
"call_conf": 95.93169261677549,
|
| 301 |
+
"learned": 90.70818684078353,
|
| 302 |
+
"call_learned": 98.96677907727631,
|
| 303 |
+
"change_pp": 3.0350864605008243,
|
| 304 |
+
"metric_change_pp": 0.0645763076702277
|
| 305 |
+
}
|
| 306 |
+
},
|
| 307 |
+
{
|
| 308 |
+
"key": [
|
| 309 |
+
"nlvr2",
|
| 310 |
+
"vilt",
|
| 311 |
+
"Qwen3.8"
|
| 312 |
+
],
|
| 313 |
+
"validation": {
|
| 314 |
+
"spec": 75.56574047550845,
|
| 315 |
+
"vlm": 90.50415353766829,
|
| 316 |
+
"conf": 90.59008879977084,
|
| 317 |
+
"call_conf": 95.9896877685477,
|
| 318 |
+
"learned": 90.41821827556575,
|
| 319 |
+
"call_learned": 94.98710971068462,
|
| 320 |
+
"change_pp": -1.0025780578630794,
|
| 321 |
+
"metric_change_pp": -0.17187052420509774
|
| 322 |
+
},
|
| 323 |
+
"test": {
|
| 324 |
+
"spec": 76.42247255506925,
|
| 325 |
+
"vlm": 90.06242376408122,
|
| 326 |
+
"conf": 89.96197172992753,
|
| 327 |
+
"call_conf": 95.93169261677549,
|
| 328 |
+
"learned": 89.997847456411,
|
| 329 |
+
"call_learned": 94.85542082227165,
|
| 330 |
+
"change_pp": -1.0762717945038358,
|
| 331 |
+
"metric_change_pp": 0.035875726483469705
|
| 332 |
+
}
|
| 333 |
+
},
|
| 334 |
+
{
|
| 335 |
+
"key": [
|
| 336 |
+
"construction",
|
| 337 |
+
"yolo26x",
|
| 338 |
+
"MiniCPM"
|
| 339 |
+
],
|
| 340 |
+
"validation": {
|
| 341 |
+
"spec": 18.254072168965788,
|
| 342 |
+
"vlm": 20.666016863629594,
|
| 343 |
+
"conf": 22.641590025257546,
|
| 344 |
+
"call_conf": 93.86590584878745,
|
| 345 |
+
"learned": 29.007041551798586,
|
| 346 |
+
"call_learned": 48.93009985734665,
|
| 347 |
+
"change_pp": -44.935805991440795,
|
| 348 |
+
"metric_change_pp": 6.3654515265410385
|
| 349 |
+
},
|
| 350 |
+
"test": {
|
| 351 |
+
"spec": 18.129624718318375,
|
| 352 |
+
"vlm": 22.361115727328066,
|
| 353 |
+
"conf": 25.23047077304935,
|
| 354 |
+
"call_conf": 92.54327563249002,
|
| 355 |
+
"learned": 25.82183145054675,
|
| 356 |
+
"call_learned": 47.20372836218375,
|
| 357 |
+
"change_pp": -45.33954727030626,
|
| 358 |
+
"metric_change_pp": 0.591360677497399
|
| 359 |
+
}
|
| 360 |
+
},
|
| 361 |
+
{
|
| 362 |
+
"key": [
|
| 363 |
+
"construction",
|
| 364 |
+
"yolo26x",
|
| 365 |
+
"Qwen3.8"
|
| 366 |
+
],
|
| 367 |
+
"validation": {
|
| 368 |
+
"spec": 18.254072168965788,
|
| 369 |
+
"vlm": 25.448626222520154,
|
| 370 |
+
"conf": 26.42878791936722,
|
| 371 |
+
"call_conf": 92.86733238231099,
|
| 372 |
+
"learned": 33.12666068966333,
|
| 373 |
+
"call_learned": 49.928673323823105,
|
| 374 |
+
"change_pp": -42.93865905848788,
|
| 375 |
+
"metric_change_pp": 6.697872770296115
|
| 376 |
+
},
|
| 377 |
+
"test": {
|
| 378 |
+
"spec": 18.129624718318375,
|
| 379 |
+
"vlm": 27.001009067886244,
|
| 380 |
+
"conf": 25.974900047621052,
|
| 381 |
+
"call_conf": 92.21038615179761,
|
| 382 |
+
"learned": 29.875800068449216,
|
| 383 |
+
"call_learned": 48.76830892143808,
|
| 384 |
+
"change_pp": -43.44207723035953,
|
| 385 |
+
"metric_change_pp": 3.9009000208281632
|
| 386 |
+
}
|
| 387 |
+
},
|
| 388 |
+
{
|
| 389 |
+
"key": [
|
| 390 |
+
"construction",
|
| 391 |
+
"rtdetr_x_fp32",
|
| 392 |
+
"MiniCPM"
|
| 393 |
+
],
|
| 394 |
+
"validation": {
|
| 395 |
+
"spec": 7.641657905799261,
|
| 396 |
+
"vlm": 20.666016863629594,
|
| 397 |
+
"conf": 20.572745422879233,
|
| 398 |
+
"call_conf": 97.8601997146933,
|
| 399 |
+
"learned": 21.744085592769803,
|
| 400 |
+
"call_learned": 77.88873038516405,
|
| 401 |
+
"change_pp": -19.971469329529246,
|
| 402 |
+
"metric_change_pp": 1.1713401698905694
|
| 403 |
+
},
|
| 404 |
+
"test": {
|
| 405 |
+
"spec": 7.8873978710439285,
|
| 406 |
+
"vlm": 22.361115727328066,
|
| 407 |
+
"conf": 21.609512872414097,
|
| 408 |
+
"call_conf": 97.90279627163781,
|
| 409 |
+
"learned": 22.996109134220227,
|
| 410 |
+
"call_learned": 78.82822902796272,
|
| 411 |
+
"change_pp": -19.0745672436751,
|
| 412 |
+
"metric_change_pp": 1.3865962618061305
|
| 413 |
+
}
|
| 414 |
+
},
|
| 415 |
+
{
|
| 416 |
+
"key": [
|
| 417 |
+
"construction",
|
| 418 |
+
"rtdetr_x_fp32",
|
| 419 |
+
"Qwen3.8"
|
| 420 |
+
],
|
| 421 |
+
"validation": {
|
| 422 |
+
"spec": 7.641657905799261,
|
| 423 |
+
"vlm": 25.448626222520154,
|
| 424 |
+
"conf": 24.862274646934207,
|
| 425 |
+
"call_conf": 98.85877318116975,
|
| 426 |
+
"learned": 27.197394026479483,
|
| 427 |
+
"call_learned": 82.88159771754636,
|
| 428 |
+
"change_pp": -15.977175463623395,
|
| 429 |
+
"metric_change_pp": 2.335119379545275
|
| 430 |
+
},
|
| 431 |
+
"test": {
|
| 432 |
+
"spec": 7.8873978710439285,
|
| 433 |
+
"vlm": 27.001009067886244,
|
| 434 |
+
"conf": 26.818431283513146,
|
| 435 |
+
"call_conf": 98.60186418109188,
|
| 436 |
+
"learned": 26.693203615113315,
|
| 437 |
+
"call_learned": 83.12250332889481,
|
| 438 |
+
"change_pp": -15.479360852197077,
|
| 439 |
+
"metric_change_pp": -0.1252276683998299
|
| 440 |
+
}
|
| 441 |
+
}
|
| 442 |
+
]
|
| 443 |
+
}
|
paper_results/figures/build_compact.py
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Plot archived results without rerunning any model."""
|
| 2 |
+
from pathlib import Path
|
| 3 |
+
import json
|
| 4 |
+
import numpy as np
|
| 5 |
+
import matplotlib
|
| 6 |
+
matplotlib.use("Agg")
|
| 7 |
+
import matplotlib.pyplot as plt
|
| 8 |
+
from matplotlib.patches import Patch, Rectangle
|
| 9 |
+
from matplotlib.ticker import MaxNLocator
|
| 10 |
+
root=Path(__file__).resolve().parents[1]
|
| 11 |
+
groups=json.loads((root/"data/results.json").read_text(encoding="utf-8"))["groups"]
|
| 12 |
+
scaling=json.loads((root/"data/scaling_selected.json").read_text(encoding="utf-8"))
|
| 13 |
+
names={"cub":"CUB-200-2011","grefcoco":"gRefCOCO","nlvr2":"NLVR2","construction":"ConstructionSite"}
|
| 14 |
+
plt.rcParams.update({"font.family":"serif","font.serif":["Times New Roman"],"font.size":8,
|
| 15 |
+
"axes.titlesize":8.5,"axes.labelsize":8,"xtick.labelsize":7,"ytick.labelsize":7,"legend.fontsize":7.5,
|
| 16 |
+
"pdf.fonttype":42,"ps.fonttype":42,"axes.spines.top":False,"axes.spines.right":False,"axes.linewidth":.6,"mathtext.fontset":"stix"})
|
| 17 |
+
colors=["#a7c8c0","#2c8c69","#d77960","#d5d9df"]
|
| 18 |
+
curve_colors={"learned":"#176f91","confidence":"#c78324","small":"#536174","large":"#8a537b"}
|
| 19 |
+
fig,axes=plt.subplots(2,4,figsize=(7.16,2.75),gridspec_kw={"height_ratios":[.66,1.15]})
|
| 20 |
+
fig.subplots_adjust(left=.064,right=.978,bottom=.19,top=.85,wspace=.30,hspace=.67)
|
| 21 |
+
for col,(task,expert) in enumerate(zip(names,["cub","instancevg","nlvr2","yolo26x"])):
|
| 22 |
+
subset=[g for g in groups if g["key"][0]==task]
|
| 23 |
+
chosen=next(g for g in subset if g["key"][1:]==[expert,"Qwen3.8"])
|
| 24 |
+
top,ax=axes[:,col]
|
| 25 |
+
xs=np.array([0,1,3,4])
|
| 26 |
+
floor=np.zeros(4)
|
| 27 |
+
for state,color in zip(["both_correct","rescuable","harmful","both_wrong"],colors):
|
| 28 |
+
values=np.array([100*g["opportunity"]["test"][state]/g["opportunity"]["test"]["total"] for g in subset])
|
| 29 |
+
top.bar(xs,values,bottom=floor,width=.72,color=color,linewidth=0,zorder=3)
|
| 30 |
+
floor+=values
|
| 31 |
+
assert np.allclose(floor,100)
|
| 32 |
+
selected_bar=next(i for i,g in enumerate(subset) if g["key"]==chosen["key"])
|
| 33 |
+
top.add_patch(Rectangle((xs[selected_bar]-.36,0),.72,100,fill=False,edgecolor="#263d46",lw=.8,zorder=5,clip_on=False))
|
| 34 |
+
top.set(ylim=(0,101),xlim=(-.65,4.65),yticks=[0,50,100],xticks=xs)
|
| 35 |
+
top.set_xticklabels(["M","Q","M","Q"])
|
| 36 |
+
top.tick_params(axis="x",length=0,pad=1)
|
| 37 |
+
for x,label in ((.5,subset[0]["expert_display"]),(3.5,subset[2]["expert_display"])):
|
| 38 |
+
top.text(x,-.31,label,ha="center",va="top",transform=top.get_xaxis_transform(),fontsize=6.7)
|
| 39 |
+
top.set_title(names[task],fontweight="bold",pad=3)
|
| 40 |
+
top.grid(axis="y",color="#dce1e4",lw=.4,zorder=0)
|
| 41 |
+
if col==0:
|
| 42 |
+
top.set_ylabel("Outcome (%)",labelpad=2)
|
| 43 |
+
else:
|
| 44 |
+
top.tick_params(labelleft=False)
|
| 45 |
+
for method,label in (("learned","Learned"),("confidence","conf.")):
|
| 46 |
+
saved=chosen["methods"][method]
|
| 47 |
+
curve=saved["test_curve"]
|
| 48 |
+
calls=[100*p["actual_fraction"] for p in curve]
|
| 49 |
+
assert np.all(np.diff(calls)>=-1e-10)
|
| 50 |
+
ax.plot(calls,[100*p["metric"] for p in curve],lw=1.15,color=curve_colors[method],label=label)
|
| 51 |
+
p=saved["test_point"]
|
| 52 |
+
ax.scatter(100*p["actual_fraction"],100*p["metric"],s=16,color=curve_colors[method],edgecolor="white",linewidth=.4,zorder=5,clip_on=False)
|
| 53 |
+
for model,label,ls in (("small","Specialist","--"),("large","VLM",":")):
|
| 54 |
+
ax.axhline(100*chosen[model]["test"],ls=ls,lw=.9,color=curve_colors[model],label=label)
|
| 55 |
+
ax.set(xlim=(0,100),xticks=[0,50,100])
|
| 56 |
+
ax.set_title(chosen["expert_display"]+" / Q",fontsize=8,pad=3)
|
| 57 |
+
ax.yaxis.set_major_locator(MaxNLocator(nbins=3))
|
| 58 |
+
ax.grid(axis="y",color="#dce1e4",lw=.4)
|
| 59 |
+
if col==0:
|
| 60 |
+
ax.set_ylabel("Accuracy (%)",labelpad=2)
|
| 61 |
+
elif col==3:
|
| 62 |
+
ax.set_ylabel("Macro-F1 (%)",labelpad=1)
|
| 63 |
+
fig.legend([Patch(facecolor=c) for c in colors],["Both correct","Rescue","Harm","Both wrong"],loc="upper center",bbox_to_anchor=(.51,1.01),ncol=4,frameon=False,handlelength=1.3,columnspacing=1.7)
|
| 64 |
+
handles,labels=axes[1,0].get_legend_handles_labels()
|
| 65 |
+
fig.legend(handles,labels,loc="lower center",bbox_to_anchor=(.52,-.01),ncol=4,frameon=False,handlelength=2,columnspacing=1.8)
|
| 66 |
+
fig.text(.52,.083,"Actual test calls (%)",ha="center",fontsize=8)
|
| 67 |
+
fig.savefig(root/"figures/gains_compact.pdf")
|
| 68 |
+
fig.savefig(root/"figures/gains_compact.png",dpi=300)
|
| 69 |
+
plt.close(fig)
|
| 70 |
+
|
| 71 |
+
fig,axes=plt.subplots(2,4,figsize=(7.16,4.3),sharex=True)
|
| 72 |
+
fig.subplots_adjust(left=.075,right=.985,bottom=.12,top=.90,hspace=.42,wspace=.38)
|
| 73 |
+
for row,endpoint in enumerate(("MiniCPM","Qwen3.8")):
|
| 74 |
+
for col,task in enumerate(names):
|
| 75 |
+
ax=axes[row,col]
|
| 76 |
+
selected=[r for r in scaling if r["key"][0]==task and r["key"][2]==endpoint]
|
| 77 |
+
for entry,color,marker in zip(selected,["#176f91","#c78324"],["o","s"]):
|
| 78 |
+
name=next(g["expert_display"] for g in groups if g["key"]==entry["key"])
|
| 79 |
+
ax.plot([25,50,100],[100*r["rank25_metric"] for r in entry["rows"]],marker=marker,lw=1.2,ms=4,color=color,label=name)
|
| 80 |
+
ax.set_title(names[task]+(" / M" if row==0 else " / Q"),fontweight="bold")
|
| 81 |
+
ax.set_xticks([25,50,100])
|
| 82 |
+
ax.yaxis.set_major_locator(MaxNLocator(nbins=4))
|
| 83 |
+
ax.grid(axis="y",lw=.45,color="#dce1e4")
|
| 84 |
+
ax.legend(loc="best",frameon=False,fontsize=6.9,handlelength=1)
|
| 85 |
+
if col==0:
|
| 86 |
+
ax.set_ylabel("Test metric (%)")
|
| 87 |
+
fig.suptitle("Earlier configurations: exactly 25% test calls",fontsize=11,fontweight="bold")
|
| 88 |
+
fig.text(.52,.035,"Router training data (%)",ha="center",fontsize=9)
|
| 89 |
+
fig.savefig(root/"figures/scaling_ranked.pdf")
|
| 90 |
+
fig.savefig(root/"figures/scaling_ranked.png",dpi=240)
|
| 91 |
+
plt.close(fig)
|
| 92 |
+
|
| 93 |
+
fig,axes=plt.subplots(1,2,figsize=(7.16,3.0),sharey=True)
|
| 94 |
+
fig.subplots_adjust(left=.16,right=.98,top=.84,bottom=.19,wspace=.32)
|
| 95 |
+
for ax,endpoint in zip(axes,("MiniCPM","Qwen3.8")):
|
| 96 |
+
records=json.loads((root/f"data/joint_{endpoint}_comparison_20260910.json").read_text(encoding="utf-8"))
|
| 97 |
+
subset=[g for g in groups if g["key"][2]==endpoint]
|
| 98 |
+
vals=[100*next(j["delta_multi_minus_single"] for j in records if j["expert"]==g["key"][1]) for g in subset]
|
| 99 |
+
ax.barh(np.arange(8),vals,color=["#2c8c69" if v>=0 else "#d77960" for v in vals],height=.64)
|
| 100 |
+
ax.axvline(0,color="#8c949b",lw=.65)
|
| 101 |
+
ax.set_yticks(np.arange(8),[g["expert_display"] for g in subset])
|
| 102 |
+
ax.set_title(endpoint.replace("MiniCPM","MiniCPM-V-4.5"),fontweight="bold")
|
| 103 |
+
ax.grid(axis="x",color="#dce1e4",lw=.4)
|
| 104 |
+
ax.set_axisbelow(True)
|
| 105 |
+
ax.set_xlim(-1.9,4.2)
|
| 106 |
+
axes[0].invert_yaxis()
|
| 107 |
+
fig.text(.55,.045,"Joint minus single-task test metric (percentage points)",ha="center",fontsize=9)
|
| 108 |
+
fig.savefig(root/"figures/joint_delta.pdf")
|
| 109 |
+
fig.savefig(root/"figures/joint_delta.png",dpi=240)
|
| 110 |
+
plt.close(fig)
|
| 111 |
+
print("Generated compact gains, matched-call data scaling, and joint-training figures.")
|
paper_results/figures/build_cost.py
ADDED
|
@@ -0,0 +1,84 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Restyle measured-component cost estimates; no model execution."""
|
| 2 |
+
from pathlib import Path
|
| 3 |
+
import csv
|
| 4 |
+
import json
|
| 5 |
+
import numpy as np
|
| 6 |
+
import matplotlib
|
| 7 |
+
matplotlib.use("Agg")
|
| 8 |
+
import matplotlib.pyplot as plt
|
| 9 |
+
from matplotlib.ticker import MaxNLocator, LogLocator, NullLocator
|
| 10 |
+
|
| 11 |
+
root = Path(__file__).resolve().parents[1]
|
| 12 |
+
data = root / "data/cost_20260911"
|
| 13 |
+
summary = json.loads((data / "cost_summary.json").read_text(encoding="utf-8"))
|
| 14 |
+
with (data / "full_cost_performance_curves.csv").open(encoding="utf-8-sig", newline="") as f:
|
| 15 |
+
rows = [r for r in csv.DictReader(f) if r["method"] in ("both", "confidence")]
|
| 16 |
+
methods = [("both", "learned", "#176f91", "-"), ("confidence", "conf.", "#c78324", "--")]
|
| 17 |
+
pairs = list(summary["components"])
|
| 18 |
+
plt.rcParams.update({"font.family":"serif", "font.serif":["Times New Roman"], "font.size":8,
|
| 19 |
+
"axes.labelsize":8, "xtick.labelsize":7, "ytick.labelsize":7, "legend.fontsize":7.5,
|
| 20 |
+
"pdf.fonttype":42, "ps.fonttype":42, "axes.spines.top":False, "axes.spines.right":False,
|
| 21 |
+
"axes.linewidth":.6, "mathtext.fontset":"stix"})
|
| 22 |
+
|
| 23 |
+
def panel(ax, pair, field, log):
|
| 24 |
+
comp = summary["components"][pair]
|
| 25 |
+
for method, label, color, ls in methods:
|
| 26 |
+
rr = [r for r in rows if r["pair_key"] == pair and r["method"] == method]
|
| 27 |
+
assert len(rr) == 101
|
| 28 |
+
x = np.array([float(r[field]) for r in rr])
|
| 29 |
+
# Confidence does not execute the learned head; correct the shared offset.
|
| 30 |
+
offset = comp["router_gflops_per_input"] if "flops" in field else comp["router_latency_ms"]
|
| 31 |
+
if method == "confidence":
|
| 32 |
+
x -= offset
|
| 33 |
+
y = 100 * np.array([float(r["test_metric"]) for r in rr])
|
| 34 |
+
sd = 100 * np.array([float(r["test_metric_std"]) for r in rr])
|
| 35 |
+
assert np.all(np.diff(x) >= -1e-8)
|
| 36 |
+
ax.plot(x, y, color=color, lw=1.15, ls=ls, label=label)
|
| 37 |
+
if method == "both":
|
| 38 |
+
ax.fill_between(x, y - sd, y + sd, color=color, alpha=.12, lw=0)
|
| 39 |
+
p = next(p for p in summary["selected_points"] if p["pair_key"] == pair and p["method"] == method)
|
| 40 |
+
ax.scatter(p[field] - (offset if method == "confidence" else 0), 100 * p["test_metric"], s=19,
|
| 41 |
+
color=color, edgecolor="white", linewidth=.45, zorder=5)
|
| 42 |
+
if log:
|
| 43 |
+
ax.set_xscale("log")
|
| 44 |
+
ax.xaxis.set_major_locator(LogLocator(base=10, numticks=5))
|
| 45 |
+
ax.xaxis.set_minor_locator(NullLocator())
|
| 46 |
+
else:
|
| 47 |
+
ax.xaxis.set_major_locator(MaxNLocator(nbins=3))
|
| 48 |
+
ax.yaxis.set_major_locator(MaxNLocator(nbins=4))
|
| 49 |
+
ax.grid(axis="y", color="#dce1e4", lw=.4)
|
| 50 |
+
ax.set_ylabel("Accuracy (%)" if pair.startswith("grefcoco") else "Macro-F1 (%)", labelpad=2)
|
| 51 |
+
ax.tick_params(pad=2)
|
| 52 |
+
|
| 53 |
+
for field, name, xlabel, log in (("flops_gflops_per_input", "flops_performance", "Estimated GFLOPs / input", True),
|
| 54 |
+
("latency_ms_per_input", "latency_performance", "Estimated time / input (ms)", False)):
|
| 55 |
+
fig, axes = plt.subplots(1, 2, figsize=(7.16, 2.35))
|
| 56 |
+
fig.subplots_adjust(left=.075, right=.99, top=.87, bottom=.28, wspace=.30)
|
| 57 |
+
for ax, pair, label in zip(axes, pairs, ("(a) gRefCOCO: InstanceVG / M", "(b) ConstructionSite: YOLO26x / Q")):
|
| 58 |
+
panel(ax, pair, field, log)
|
| 59 |
+
ax.set_xlabel(xlabel)
|
| 60 |
+
ax.text(.5, 1.08, label, transform=ax.transAxes, ha="center", fontsize=8)
|
| 61 |
+
handles, labels = axes[0].get_legend_handles_labels()
|
| 62 |
+
fig.legend(handles, labels, loc="lower center", bbox_to_anchor=(.52, -.015), ncol=2, frameon=False, handlelength=2.3)
|
| 63 |
+
fig.savefig(root / f"figures/{name}.pdf")
|
| 64 |
+
fig.savefig(root / f"figures/{name}.png", dpi=300)
|
| 65 |
+
plt.close(fig)
|
| 66 |
+
|
| 67 |
+
fig, axes = plt.subplots(2, 2, figsize=(3.44, 2.30))
|
| 68 |
+
fig.subplots_adjust(left=.135, right=.99, top=.88, bottom=.23, wspace=.40, hspace=.62)
|
| 69 |
+
for col, pair in enumerate(pairs):
|
| 70 |
+
for row, field in enumerate(("flops_gflops_per_input", "latency_ms_per_input")):
|
| 71 |
+
ax = axes[row, col]
|
| 72 |
+
panel(ax, pair, field, row == 0)
|
| 73 |
+
ax.tick_params(labelsize=6.5)
|
| 74 |
+
ax.yaxis.set_major_locator(MaxNLocator(nbins=3))
|
| 75 |
+
ax.set_ylabel("Acc. (%)" if col == 0 else "F1 (%)", fontsize=7)
|
| 76 |
+
ax.set_xlabel("Est. GFLOPs / input" if row == 0 else "Est. time / input (ms)", fontsize=7, labelpad=1)
|
| 77 |
+
if row == 0:
|
| 78 |
+
ax.text(.5, 1.17, "gRefCOCO / M" if col == 0 else "ConstructionSite / Q", transform=ax.transAxes, ha="center", fontsize=7.2)
|
| 79 |
+
handles, labels = axes[0, 0].get_legend_handles_labels()
|
| 80 |
+
fig.legend(handles, labels, loc="lower center", bbox_to_anchor=(.54, .0), ncol=2, frameon=False, handlelength=2)
|
| 81 |
+
fig.savefig(root / "figures/cost_compact.pdf")
|
| 82 |
+
fig.savefig(root / "figures/cost_compact.png", dpi=300)
|
| 83 |
+
plt.close(fig)
|
| 84 |
+
print("Created two matching standalone plots and a four-panel paper figure.")
|
paper_results/figures/build_tables.py
ADDED
|
@@ -0,0 +1,123 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Rebuild tables from saved results; no model execution."""
|
| 2 |
+
from pathlib import Path
|
| 3 |
+
import json
|
| 4 |
+
|
| 5 |
+
root = Path(__file__).resolve().parents[1]
|
| 6 |
+
groups = json.loads((root/"data/results.json").read_text(encoding="utf-8"))["groups"]
|
| 7 |
+
diagnostics = json.loads((root/"data/budget_diagnostics.json").read_text(encoding="utf-8"))
|
| 8 |
+
earlier = json.loads((root/"data/earlier_results.json").read_text(encoding="utf-8"))["rows"]
|
| 9 |
+
task_names = {"cub":"CUB-200-2011","grefcoco":"gRefCOCO","nlvr2":"NLVR2","construction":"ConstructionSite"}
|
| 10 |
+
joint = []
|
| 11 |
+
for endpoint in ("MiniCPM","Qwen3.8"):
|
| 12 |
+
joint.extend(json.loads((root/f"data/joint_{endpoint}_comparison_20260910.json").read_text(encoding="utf-8")))
|
| 13 |
+
rows = {name:[] for name in ("main_test","main_full","thresholds","outcomes_full","scaling_full","joint_full")}
|
| 14 |
+
scaling, audit = [], []
|
| 15 |
+
counts = dict(better_metric=0,fewer_calls=0,both=0,above_endpoints=0,better_ranked_area=0)
|
| 16 |
+
for index,g in enumerate(groups):
|
| 17 |
+
task,expert,endpoint = g["key"]
|
| 18 |
+
prefix = f"{task_names[task] if index%4==0 else ''} & {g['expert_display']} & {'M' if endpoint=='MiniCPM' else 'Q'}"
|
| 19 |
+
if index and index%4==0:
|
| 20 |
+
for table in rows.values():
|
| 21 |
+
table.append(r"\midrule")
|
| 22 |
+
blocks, raw = [], {}
|
| 23 |
+
for split in ("validation","test"):
|
| 24 |
+
conf = g["methods"]["confidence"][f"{split}_point"]
|
| 25 |
+
learned = g["methods"]["learned"][f"{split}_point"]
|
| 26 |
+
change = 100*(learned["actual_fraction"]-conf["actual_fraction"])
|
| 27 |
+
macro = "callup" if change>1e-10 else "calldown" if change < -1e-10 else "samechg"
|
| 28 |
+
delta = rf"\{macro}{{{abs(change):.2f}}}"
|
| 29 |
+
metric_change = 100*(learned["metric"]-conf["metric"])
|
| 30 |
+
metric_macro = "upchg" if metric_change>1e-10 else "downchg" if metric_change < -1e-10 else "samechg"
|
| 31 |
+
metric_delta = rf"\{metric_macro}{{{abs(metric_change):.2f}}}"
|
| 32 |
+
cv,lv = f"{100*conf['metric']:.2f}",f"{100*learned['metric']:.2f}"
|
| 33 |
+
if conf["metric"]>=learned["metric"]:
|
| 34 |
+
cv = rf"\textbf{{{cv}}}"
|
| 35 |
+
if learned["metric"]>=conf["metric"]:
|
| 36 |
+
lv = rf"\textbf{{{lv}}}"
|
| 37 |
+
if split=="test" and learned["metric"]>max(g["small"][split],g["large"][split]):
|
| 38 |
+
lv += r"$^\dagger$"
|
| 39 |
+
blocks.append(f"{100*g['small'][split]:.2f} & {100*g['large'][split]:.2f} & {cv} & {100*conf['actual_fraction']:.2f} & {lv}{metric_delta} & {100*learned['actual_fraction']:.2f}{delta}")
|
| 40 |
+
raw[split] = dict(spec=100*g["small"][split],vlm=100*g["large"][split],conf=100*conf["metric"],call_conf=100*conf["actual_fraction"],learned=100*learned["metric"],call_learned=100*learned["actual_fraction"],change_pp=change,metric_change_pp=metric_change)
|
| 41 |
+
for method in ("confidence","learned"):
|
| 42 |
+
p=g["methods"][method]
|
| 43 |
+
assert p[f"{split}_point"]["threshold"]==p["validation_point"]["threshold"]
|
| 44 |
+
assert p[f"{split}_point"]["operator"]==p["validation_point"]["operator"]
|
| 45 |
+
rows["main_test"].append(prefix+" & "+blocks[1]+r" \\")
|
| 46 |
+
rows["main_full"].append(prefix+" & "+" & ".join(blocks)+r" \\")
|
| 47 |
+
selected=g["methods"]["learned"]["test_point"]
|
| 48 |
+
reference=g["methods"]["confidence"]["test_point"]
|
| 49 |
+
better=selected["metric"]>reference["metric"]
|
| 50 |
+
fewer=selected["actual_fraction"]<reference["actual_fraction"]
|
| 51 |
+
counts["better_metric"]+=better
|
| 52 |
+
counts["fewer_calls"]+=fewer
|
| 53 |
+
counts["both"]+=better and fewer
|
| 54 |
+
counts["above_endpoints"]+=selected["metric"]>max(g["small"]["test"],g["large"]["test"])
|
| 55 |
+
d=next(d for d in diagnostics if d["key"]==g["key"])
|
| 56 |
+
counts["better_ranked_area"]+=d["areas"]["learned"]["test_ranked_A25_diagnostic"]>d["areas"]["confidence"]["test_ranked_A25_diagnostic"]
|
| 57 |
+
cutoffs=[]
|
| 58 |
+
for method in ("confidence","learned"):
|
| 59 |
+
p=g["methods"][method]["validation_point"]
|
| 60 |
+
op=r"\geq" if p["operator"]=="ge" else ">"
|
| 61 |
+
cutoffs.extend([f"{100*p['budget']:.0f}","$"+f"{op}{p['threshold']:.6g}"+"$"])
|
| 62 |
+
rows["thresholds"].append(prefix+" & "+" & ".join(cutoffs)+r" \\")
|
| 63 |
+
opp=g["opportunity"]["test"]
|
| 64 |
+
keys=("both_correct","rescuable","harmful","both_wrong")
|
| 65 |
+
assert sum(opp[k] for k in keys)==opp["total"]
|
| 66 |
+
gain=100*(selected["metric"]-g["small"]["test"])
|
| 67 |
+
if task!="construction":
|
| 68 |
+
assert abs(gain-100*(selected["rescued"]-selected["harmed"])/g["test_rows"])<1e-9
|
| 69 |
+
values=[opp[k] for k in keys]+[selected[k] for k in ("rescued","harmed","missed_rescue","avoided_harm")]
|
| 70 |
+
rows["outcomes_full"].append(prefix+" & "+" & ".join(f"{v:,}" for v in values)+f" & {gain:+.3f}"+r" \\")
|
| 71 |
+
scale=[]
|
| 72 |
+
for fraction,tag in ((.25,"baseline_25pct"),(.5,"baseline_50pct"),(1.,"baseline_100pct")):
|
| 73 |
+
matches=[r for r in earlier if (r["task"],r["expert"],r["endpoint"],r["tag"])==(task,expert,endpoint,tag)]
|
| 74 |
+
assert len(matches)==1,(g["key"],tag,len(matches))
|
| 75 |
+
assert matches[0]["fraction"]==fraction
|
| 76 |
+
scale.append(matches[0])
|
| 77 |
+
for field in ("student","branches","fusion","target","strategy","representation_dim","hidden_layer"):
|
| 78 |
+
assert all(r[field]==scale[0][field] for r in scale),(g["key"],field)
|
| 79 |
+
rows["scaling_full"].append(prefix+" & "+" & ".join(f"{100*r['fixed25_metric']:.2f} & {100*r['fixed25_actual_fraction']:.2f} & {100*r['rank25_metric']:.2f}" for r in scale)+r" \\")
|
| 80 |
+
scaling.append(dict(key=g["key"],rows=scale))
|
| 81 |
+
j=next(j for j in joint if (j["expert"],j["endpoint"])==(expert,endpoint))
|
| 82 |
+
rows["joint_full"].append(prefix+f" & {100*j['single_task_test_fixed25']:.3f} & {100*j['multi_task_test_fixed25']:.3f} & {100*j['delta_multi_minus_single']:+.3f}"+r" \\")
|
| 83 |
+
audit.append(dict(key=g["key"],**raw))
|
| 84 |
+
assert counts==dict(better_metric=11,fewer_calls=15,both=10,above_endpoints=5,better_ranked_area=15),counts
|
| 85 |
+
|
| 86 |
+
head=r"\textcolor{testink}{spec.} & \textcolor{testink}{vlm.} & \textcolor{confink}{conf.} & \textcolor{confink}{call$_{\rm conf}$} & \textcolor{learnink}{learned$\uparrow$} & \textcolor{learnink}{call$_{\rm learned}\downarrow$}"
|
| 87 |
+
cols=r">{\columncolor{basefill}}r>{\columncolor{basefill}}r>{\columncolor{conffill}}r>{\columncolor{conffill}}r>{\columncolor{learnfill}}r>{\columncolor{learnfill}}r"
|
| 88 |
+
specs=[
|
| 89 |
+
("main_test","table*","t","tab:main","lll"+cols, "Dataset & Specialist & VLM & "+head,
|
| 90 |
+
r"Test results at validation-selected thresholds. Scores and actual call rates are percentages; parentheses give learned-minus-conf. differences in percentage points. \textcolor{upgreen}{Green} marks improvement and \textcolor{downred}{red} degradation: higher scores and fewer calls are better. Bold compares routed scores; $\dagger$ exceeds both models. M: MiniCPM-V-4.5; Q: Qwen3.8-27B-FP8. ",8,10.0),
|
| 91 |
+
("main_full","table","H","tab:full","lll"+cols+cols,
|
| 92 |
+
r"\multirow{2}{*}{Task} & \multirow{2}{*}{Specialist} & \multirow{2}{*}{VLM} & \multicolumn{6}{c}{\cellcolor{valfill}\textcolor{valink}{Validation}} & \multicolumn{6}{c}{\cellcolor{testfill}\textcolor{testink}{Test}} \\ \cmidrule(lr){4-9}\cmidrule(l){10-15} & & & "+head+" & "+head,
|
| 93 |
+
r"Complete four-way comparison. Scores and actual call rates are percentages. Thresholds selected on the 1--99\% validation grid transfer unchanged to test. Parentheses give learned-minus-confidence differences in percentage points before rounding. Green marks improvement and red degradation: higher scores and fewer calls are better. Bold marks the higher routed score; $\dagger$ exceeds both test endpoints. M: MiniCPM-V-4.5; Q: Qwen3.8-27B-FP8.",8,1.8),
|
| 94 |
+
("thresholds","table","H","tab:thresholds","lllrrrr",
|
| 95 |
+
r"Task & Specialist & VLM & Conf. budget & Conf. cutoff & Learned budget & Learned cutoff",
|
| 96 |
+
"Selected nominal validation budgets and score cutoffs. The operator is part of each stored threshold. Displayed cutoffs are rounded; the archived JSON retains full precision. Budget is a percentage, not the numerical threshold.",9,4),
|
| 97 |
+
("outcomes_full","table","H","tab:outcomes_full","lllrrrrrrrrr",
|
| 98 |
+
r"Task & Specialist & VLM & BC & R & H & BW & Sel. R & Sel. H & Miss R & Avoid H & Gain (pp)",
|
| 99 |
+
"Test outcome counts at the learned operating point. BC/BW: both correct/wrong; R/H: rescue/harm. Selected rescues, selected harms, missed rescues and avoided harms are counted separately. Construction counts 12,016 events, with routing over 3,004 images. Metric gains are recomputed; macro-F1 is not an event-count ratio.",8.5,3),
|
| 100 |
+
("scaling_full","table","H","tab:scaling_full","lllrrrrrrrrr",
|
| 101 |
+
r"Task & Specialist & VLM & \multicolumn{3}{c}{25\% training} & \multicolumn{3}{c}{50\% training} & \multicolumn{3}{c}{100\% training}\\ \cmidrule(lr){4-6}\cmidrule(lr){7-9}\cmidrule(l){10-12} & & & Metric & Calls & Rank & Metric & Calls & Rank & Metric & Calls & Rank",
|
| 102 |
+
r"Earlier training-data comparisons. Each pair retains its exploratory reference configuration; only router-training fraction changes. Within each fraction: test metric, actual test calls, and test metric at exactly 25\% ranked calls. The first two use a validation-selected 25\% threshold. These are not final shared-recipe ablations. All entries are percentages.",8.5,3),
|
| 103 |
+
("joint_full","table","H","tab:joint_full","lllrrr",
|
| 104 |
+
r"Task & Specialist & VLM & Single & Joint & Difference (pp)",
|
| 105 |
+
r"Earlier single-task versus joint-training comparison. Both use 4M students, image/text inputs and specialist-error supervision. Metrics use validation-selected 25\% thresholds; actual call rates are not present in this comparison file. Differences are joint minus single in percentage points. All tasks were seen during training.",9,5)]
|
| 106 |
+
for name,env,place,label,spec,header,caption,size,pad in specs:
|
| 107 |
+
content=f"\\begin{{{env}}}[{place}]\n\\centering\n\\caption{{{caption}}}\n\\label{{{label}}}\n"
|
| 108 |
+
content+=f"\\fontsize{{{size}}}{{{size*1.2}}}\\selectfont\n\\setlength{{\\tabcolsep}}{{{pad}pt}}\n\\renewcommand{{\\arraystretch}}{{1.12}}\n"
|
| 109 |
+
content+="\\begin{tabular}{@{}"+spec+"@{}}\n\\toprule\n"+header+r" \\"+"\n\\midrule\n"
|
| 110 |
+
content+="\n".join(rows[name])+"\n\\bottomrule\n\\end{tabular}\n"+f"\\end{{{env}}}\n"
|
| 111 |
+
if name == "main_test":
|
| 112 |
+
content = content.replace(r"\begin{table*}[t]", r"\begin{minipage}{\textwidth}")
|
| 113 |
+
content = content.replace(r"\caption{", r"\captionof{table}{")
|
| 114 |
+
content = content.replace(r"\end{table*}", r"\end{minipage}")
|
| 115 |
+
content = content.replace(
|
| 116 |
+
r">{\columncolor{learnfill}}r@{}}",
|
| 117 |
+
r">{\columncolor{learnfill}[\tabcolsep][0pt]}r@{}}",
|
| 118 |
+
)
|
| 119 |
+
(root/f"tables/{name}.tex").write_text(content,encoding="utf-8")
|
| 120 |
+
(root/"data/table_audit.json").write_text(json.dumps(dict(counts=counts,rows=audit),indent=2),encoding="utf-8")
|
| 121 |
+
(root/"data/scaling_selected.json").write_text(json.dumps(scaling,indent=2),encoding="utf-8")
|
| 122 |
+
print(json.dumps(counts))
|
| 123 |
+
print("Rebuilt all 16-pair tables from saved results.")
|
paper_results/figures/cost_includes.tex
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
% Standalone insertions for the updated three-seed router, not the revision-17 policy.
|
| 2 |
+
\begin{figure*}[t]
|
| 3 |
+
\centering
|
| 4 |
+
\includegraphics[width=\textwidth]{figures/flops_performance.pdf}
|
| 5 |
+
\caption{Test performance versus estimated computation for InstanceVG/MiniCPM and YOLO26x/Qwen. Learned uses the probability contrast $p_R-p_H$ and validation task-performance checkpoint selection; conf. uses specialist confidence. Curves average seeds 42--44; shading is one population standard deviation and dots average validation-selected policies. All thresholds are fixed before test evaluation. Costs combine measured components with actual test call fractions; VLM decoding is estimated from mean output length. Router feature encoders and unsupported operations are excluded. These are component-based estimates, not full-system counts.}
|
| 6 |
+
\label{fig:new_cost_flops}
|
| 7 |
+
\end{figure*}
|
| 8 |
+
|
| 9 |
+
\begin{figure*}[t]
|
| 10 |
+
\centering
|
| 11 |
+
\includegraphics[width=\textwidth]{figures/latency_performance.pdf}
|
| 12 |
+
\caption{Test performance versus estimated amortized time per input under the same three-seed protocol. Time combines A800 specialist measurements, the routing head on cached features, and VLM batch wall time weighted by actual test calls. The VLM probe uses batch size four. Feature-encoder computation, loading, warmup and deployment queueing are excluded. This estimate is not a measurement of online per-request response latency. Learned and conf. have the same meaning as in Fig.~\ref{fig:new_cost_flops}.}
|
| 13 |
+
\label{fig:new_cost_time}
|
| 14 |
+
\end{figure*}
|
paper_results/figures/gains_compact.pdf
ADDED
|
Binary file (73.5 kB). View file
|
|
|
paper_results/figures/gains_compact.png
ADDED
|
Git LFS Details
|
paper_results/figures/joint_delta.pdf
ADDED
|
Binary file (51.6 kB). View file
|
|
|
paper_results/figures/joint_delta.png
ADDED
|
paper_results/figures/scaling_ranked.pdf
ADDED
|
Binary file (67 kB). View file
|
|
|