File size: 6,306 Bytes
94c9578 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | # UHR-BAT2 evaluation
[English](EVALUATION.md) · [中文](EVALUATION_zh.md) · [Model card](README.md)
This document reports audited results for the `qwen3vl-8b-geollava-ft3-selected-10k` checkpoint released as UHR-BAT2. Existing prediction artifacts were checked on 2026-10-13; no new inference was run for this audit. Machine-readable counts, categories, protocol identifiers and source hashes are in [evaluation_results.json](evaluation_results.json).
## Main results
Accuracy is sample-weighted unless stated otherwise. All three datasets use the corrected dense scoring protocol. Scores from different datasets should not be treated as interchangeable.
| Dataset | Correct / Total | Accuracy | Scope |
|---|---:|---:|---|
| RSHR-Bench | 980 / 1,932 | 50.72% | All QA |
| XLRS-Bench-lite | 1,610 / 3,080 | 52.27% | Strict option-set accuracy |
| MME-RealWorld Remote Sensing | 2,123 / 3,738 | 56.80% | Full remote-sensing subset |
## Denominators and answer formats
RSHR single-turn Overall is **726/1,590 = 45.66%**. The remaining multi-turn QA records score **254/342 = 74.27%**. The combined **50.72%** score must not be labeled as the 1,590-record single-turn Overall metric. The multi-turn group is defined by the three original category identifiers `multitrun-future_prediction`, `multiturn_object_state_judgement`, and `multturn_anomaly_detection`.
XLRS contains **2,980 single-choice records: 1,563 correct (52.45%)**, and **100 multi-select task records: 47 correct (47.00%)**. Multi-select examples require an exact match of the complete canonical option set. The evaluated stored split is labeled `train`; this label describes the benchmark file, not evidence that those records were used for model training. MME evaluates all 3,738 Remote Sensing records, rather than a Lite subset.
## Evaluation protocol
The corrected dense run uses `20260913_strict_sets_matched_schedule_v1`, BF16 model weights, eager attention, `min_pixels=784`, and `max_pixels=1,310,720`. Dense means that visual tokens are not pruned after preprocessing. Images are still resized to the pixel budget; these results do not establish native 4K/8K or full-resolution ultra-high-resolution perception.
For single-choice questions, the evaluator selects the highest candidate-option logit using FP32 rows from the language-model output head. For XLRS multi-select questions, a constrained greedy token-trie decoder with FP32 candidate rows produces a canonical option set, which is scored by exact set matching.
**The ordinary `model.generate()` quickstart is an inference demonstration, not a reproduction of these scores.** Reproduction requires the same benchmark records, prompts, image processing, candidate-option scoring or constrained multi-select decoder, and answer normalization. Changing any of these may change the measured accuracy.
## Category results
Original dataset category identifiers, including spelling differences, are retained for traceability. All category scores below use the protocol selected for that dataset in the main table.
### RSHR-Bench
| Category | Correct / Total | Accuracy |
|---|---:|---:|
| anomaly_detection | 37 / 50 | 74.00% |
| color | 135 / 200 | 67.50% |
| detection | 32 / 100 | 32.00% |
| future_prediction_two_image | 27 / 50 | 54.00% |
| multi_region_join_contrast | 13 / 20 | 65.00% |
| multi_region_join_contrast_singal | 22 / 50 | 44.00% |
| multitrun-future_prediction | 100 / 150 | 66.67% |
| multiturn_object_state_judgement | 112 / 132 | 84.85% |
| multturn_anomaly_detection | 42 / 60 | 70.00% |
| object_clasfication | 112 / 200 | 56.00% |
| object_counting | 33 / 200 | 16.50% |
| object_grounding | 74 / 200 | 37.00% |
| object_state_judgement | 30 / 50 | 60.00% |
| regional_grouding | 33 / 70 | 47.14% |
| regionla_counting | 43 / 100 | 43.00% |
| relation | 108 / 200 | 54.00% |
| shape | 27 / 100 | 27.00% |
### XLRS-Bench-lite
| Category | Correct / Total | Accuracy |
|---|---:|---:|
| Complex reasoning/Anomaly Detection and Interpretation | 73 / 100 | 73.00% |
| Complex reasoning/Environmental condition reasoning | 80 / 100 | 80.00% |
| Complex reasoning/Route planning | 50 / 100 | 50.00% |
| Counting/Counting with changing detection | 30 / 60 | 50.00% |
| Counting/Counting with complex reasoning | 55 / 100 | 55.00% |
| Counting/Overall counting | 22 / 60 | 36.67% |
| Counting/Regional counting | 47 / 100 | 47.00% |
| Land use classification/Overall Land use classification | 47 / 100 | 47.00% |
| Land use classification/Regional Land use classification | 169 / 200 | 84.50% |
| Object properties/Object classification | 422 / 800 | 52.75% |
| Object properties/Object color | 374 / 800 | 46.75% |
| Object properties/Object motion state | 39 / 60 | 65.00% |
| Object spatial relationship/Object spatial relationship | 202 / 500 | 40.40% |
### MME-RealWorld Remote Sensing
| Category | Correct / Total | Accuracy |
|---|---:|---:|
| color | 855 / 1,255 | 68.13% |
| count | 337 / 1,226 | 27.49% |
| position | 931 / 1,257 | 74.07% |
## Verification and interpretation
All corrected dense prediction shards were read in full. Each dataset has complete, unique expected row indices; saved correctness agrees with answer/prediction equality, and protocol sidecars and completion markers are consistent. Recorded checkpoint, evaluator-code and protocol identities match across the corrected runs. Source artifact and evaluator hashes are provided in the JSON.
**Excluded historical XLRS results:** an earlier evaluator requested one option letter and truncated multi-option ground truths. Those results are not valid strict option-set accuracy and are excluded from this release report. No same-protocol accuracy gain over the original base model is claimed here.
The results show stronger performance on MME position and color questions and XLRS regional land-use classification, while counting is a recurring weakness, especially RSHR object counting and MME count. These are within-benchmark observations, not controlled cross-dataset or model comparisons.
Only aggregate results are included here. Benchmark images, questions and raw prediction files are not redistributed in this evaluation artifact. The hashes identify audited source files; the aggregate JSON alone is not a complete public reproduction package.
|