| # Five-seed objective and operating-point protocol |
|
|
| ## Targets and scores |
|
|
| Let `R` denote a specialist error rescued by the VLM and `H` a correct specialist result harmed by substitution. The four-state order is **both correct, rescue, harm, both wrong**. Let `q` be the observed target distribution and `p = softmax(logits)` the predicted distribution. |
|
|
| | Released method ID | Training objective | Score used to rank calls | |
| |---|---|---| |
| | `four_difference` (current main method) | Four-state cross-entropy `−sum(q_k log p_k)` | `p_R − p_H` | |
| | `four_ratio` | Same four-state cross-entropy | `log(p_R/p_H) = logit_R − logit_H` | |
| | `gain` | MSE for `q_R − q_H` | One unrestricted scalar output | |
| | `error` | BCE for specialist error | One sigmoid probability | |
| | `route` | BCE for rescue; all other outcomes are negative | One sigmoid probability | |
|
|
| For the three accuracy tasks, gain-regression labels are +1 for rescue, −1 for harm, and 0 for both-correct/both-wrong. ConstructionSite averages the four event outcomes: its gain label is the mean correctness change and its rescue label is the fraction of rescued events. These are correctness surrogates, not per-input macro-F1 gains. Both gain and probability difference model expected correctness change via different output representations and training losses. Their difference is a matched design exploration, not a claim of universal four-state superiority. |
|
|
| The two four-state methods have identical cross-entropy training objectives but select checkpoints using different validation scores. The release also evaluates both scores on each of the 160 four-state checkpoints to separate scoring changes from checkpoint selection. Those 160 alternate-score evaluations add no training fits. |
|
|
| ## Fits, selection, and uncertainty |
|
|
| All five objectives are run on the same 16 pairs with seeds **2026, 2027, 2028, 2029, 2030**. Specialist predictions, VLM responses, the distilled encoder, data splits, input branches, and model-pair hyperparameters are held fixed. Every run retains its own epoch and deployment threshold. The experiment contains 400 repeated fits; the current main method uses 80 of them. The earlier 80-fit seed-42 pilot is historical and is not averaged into the current tables. |
|
|
| For each run, the stopping epoch maximizes validation Score25: the trapezoidal mean metric over exact call budgets 0%, 5%, 10%, 15%, 20%, and 25%. For each method and seed, scores are averaged over four pairs per task and then equally over the four tasks. The final global method maximizes the mean of these five task-balanced scores minus their **sample SD (denominator 4)**. It selects `four_difference`. Mean minus SD is a selection heuristic, not a confidence bound. |
|
|
| The current main table reports five-seed means at each run's validation-selected operating point. The objective table uses validation Score25 and test Score25 after transferring validation thresholds; the full supplement also gives exact-budget test diagnostics and same-checkpoint score controls. For objective-level means/SDs, aggregation across pairs and tasks occurs **within each seed**, before computing SD across the five aggregate values. Pair-level SDs use the five runs of that pair. Standard deviations do not include retraining encoders or specialists, rerunning VLM responses, resampling data, or variation across five independent dataset splits. |
|
|
| Numerical mean comparisons treat differences within `1e-12` on the original 0–1 metric scale as ties. This avoids reporting floating-point roundoff as a win. In particular, ResNet/Qwen has two rescued and two harmed examples across the five seeds and is tied with the specialist on average. Current counts are 13 pairs above confidence, 11 with fewer calls, 10 satisfying both, and 8 above both stand-alone models. |
|
|
| ## Validation thresholds versus exact test budgets |
|
|
| Candidate thresholds are built from validation scores at the 0–100% budget grid. A run's reported operating point maximizes the validation metric over the interior 1–99% points, breaking ties by fewer validation calls and then lower budget. The threshold and comparison operator transfer unchanged to test data. Because distributions may shift and scores may tie, actual test calls need not equal the validation grid label. |
|
|
| This threshold policy can be applied independently to each incoming sample. It never ranks the entire arriving test population to enforce a quota. Separately marked exact-budget diagnostics rank the available test batch and choose the required number of calls; they support same-budget analysis and are not the online deployment policy. |
|
|
| ## Test-access and method-selection history |
|
|
| The original study inspected test results during development. After the seed-42 objective pilot and partial CUB test results had been reported, the selection rule was amended to five-seed validation mean minus SD. The final selector itself reads validation completion records only, and no seed or operating point is selected by its test score. These repeats provide additional evidence on existing splits; they are **not a prospectively untouched test set**. |
|
|
| ## Cost scope |
|
|
| The current Fig. 3 uses InstanceVG/MiniCPM and YOLO26x/Qwen. For every input, learned routing charges specialist cost plus frozen encoder/head cost, plus actual VLM call fraction times VLM cost. Confidence omits encoder/head cost. Measured component costs come from A800 batch-one runs over 64 inputs. The same frozen components are reused for the five router seeds; curves aggregate seed-specific rates and metrics. Estimated serial time excludes a full production service's queuing/batching effects and is not a five-seed rerun of full-system timing. See the supplementary cost section and `paper_results/data/` for measurements and construction details. |
|
|
