OSHE Local v2 Evaluation Note
Scope
This release comparison measures one product route: multi-document analysis. It does not measure broad general intelligence, open-domain knowledge or parity with larger foundation models.
The comparison uses the same eight multi-document records from a frozen 32-record product fixture, the same evaluation gates, deterministic decoding with temperature 0 and seed 42, and the same Gemma 4 E2B foundation checkpoint. The two evaluated configurations are:
- the unmodified Gemma 4 E2B base route; and
- the complete OSHE Local v2 route, including its signed learned capability pack and required source-grounding guard.
Results
| Measurement | Gemma 4 E2B base | OSHE Local v2 | Change |
|---|---|---|---|
| Reference-score attainment | 65% | 95% | +30 percentage points |
| Schema compliance | 87.5% | 100% | +12.5 percentage points |
| Non-overconfidence | 87.5% | 100% | +12.5 percentage points |
| Parse-error rate | 0% | 0% | no change |
Reference-score attainment is a task-specific score-versus-reference ratio. It is not an academic benchmark score. Schema compliance measures whether the response satisfied the structured multi-document contract. Non-overconfidence measures whether the response avoided unsupported certainty when evidence was missing or conflicting.
Reproducibility Bindings
| Artifact | SHA-256 |
|---|---|
| Gemma 4 E2B foundation artifact | 2ecb35dcf661866c74f7d45719bcf6c71e71bcf96f900b1e98968f64838effe1 |
| Frozen 32-record fixture | 011d99a0dacdb83425cf0068f5be8d41cb594f8b5f3b62289f7a6cf37b54ad2d |
| Evaluation gates | bd7ea33455171aae5845c520ea0032ab9d76221546ca9916a82dfae3802bedad |
| Gemma base predictions | eab7a8fed1378cba04fa91579e58d483471e7fe0d1b58d2ed8c5d70143b69379 |
| OSHE route predictions | 319d43fe48fb25f19c4e7ef884da88d41846980f566ef7f3048762bb1f57298d |
| Public learned-pack weights | c02e8c255b12a88f84570a7afa58b81f215e78b56069f2bb9a908415772ca857 |
Limitations
- The evaluated multi-document subset contains eight cases and is intended as a release gate, not a comprehensive research benchmark.
- The historical aggregate evaluator reports
all_gates_passed: falsebecause its Cantonese-mimicry diagnostic scored12.5%. That diagnostic was superseded for this route: the released contract requires formal written Traditional Chinese, not colloquial Cantonese imitation. A subsequent frozen semantic review passed all16/16Traditional-Chinese outputs for formal register, while the original register-drift gate also remained at0%. - The result measures the complete routed product behaviour. It must not be presented as the raw capability of the learned pack in isolation.
- The result does not establish performance on unrelated chat, coding, browsing, search or tool-use tasks.
- No claim of equivalence to Qwen 27B or any other larger model is made.