| # OSHE Local v2 Evaluation Note |
|
|
| ## Scope |
|
|
| This release comparison measures one product route: multi-document analysis. |
| It does not measure broad general intelligence, open-domain knowledge or parity |
| with larger foundation models. |
|
|
| The comparison uses the same eight multi-document records from a frozen |
| 32-record product fixture, the same evaluation gates, deterministic decoding |
| with temperature 0 and seed 42, and the same Gemma 4 E2B foundation checkpoint. |
| The two evaluated configurations are: |
|
|
| 1. the unmodified Gemma 4 E2B base route; and |
| 2. the complete OSHE Local v2 route, including its signed learned capability |
| pack and required source-grounding guard. |
|
|
| ## Results |
|
|
| | Measurement | Gemma 4 E2B base | OSHE Local v2 | Change | |
| | --- | ---: | ---: | ---: | |
| | Reference-score attainment | 65% | 95% | +30 percentage points | |
| | Schema compliance | 87.5% | 100% | +12.5 percentage points | |
| | Non-overconfidence | 87.5% | 100% | +12.5 percentage points | |
| | Parse-error rate | 0% | 0% | no change | |
|
|
| Reference-score attainment is a task-specific score-versus-reference ratio. It |
| is not an academic benchmark score. Schema compliance measures whether the |
| response satisfied the structured multi-document contract. Non-overconfidence |
| measures whether the response avoided unsupported certainty when evidence was |
| missing or conflicting. |
|
|
| ## Reproducibility Bindings |
|
|
| | Artifact | SHA-256 | |
| | --- | --- | |
| | Gemma 4 E2B foundation artifact | `2ecb35dcf661866c74f7d45719bcf6c71e71bcf96f900b1e98968f64838effe1` | |
| | Frozen 32-record fixture | `011d99a0dacdb83425cf0068f5be8d41cb594f8b5f3b62289f7a6cf37b54ad2d` | |
| | Evaluation gates | `bd7ea33455171aae5845c520ea0032ab9d76221546ca9916a82dfae3802bedad` | |
| | Gemma base predictions | `eab7a8fed1378cba04fa91579e58d483471e7fe0d1b58d2ed8c5d70143b69379` | |
| | OSHE route predictions | `319d43fe48fb25f19c4e7ef884da88d41846980f566ef7f3048762bb1f57298d` | |
| | Public learned-pack weights | `c02e8c255b12a88f84570a7afa58b81f215e78b56069f2bb9a908415772ca857` | |
|
|
| ## Limitations |
|
|
| - The evaluated multi-document subset contains eight cases and is intended as |
| a release gate, not a comprehensive research benchmark. |
| - The historical aggregate evaluator reports `all_gates_passed: false` because |
| its Cantonese-mimicry diagnostic scored `12.5%`. That diagnostic was |
| superseded for this route: the released contract requires formal written |
| Traditional Chinese, not colloquial Cantonese imitation. A subsequent frozen |
| semantic review passed all `16/16` Traditional-Chinese outputs for formal |
| register, while the original register-drift gate also remained at `0%`. |
| - The result measures the complete routed product behaviour. It must not be |
| presented as the raw capability of the learned pack in isolation. |
| - The result does not establish performance on unrelated chat, coding, |
| browsing, search or tool-use tasks. |
| - No claim of equivalence to Qwen 27B or any other larger model is made. |
|
|
