# OSHE Local v2 Evaluation Note

## Scope

This release comparison measures one product route: multi-document analysis.
It does not measure broad general intelligence, open-domain knowledge or parity
with larger foundation models.

The comparison uses the same eight multi-document records from a frozen
32-record product fixture, the same evaluation gates, deterministic decoding
with temperature 0 and seed 42, and the same Gemma 4 E2B foundation checkpoint.
The two evaluated configurations are:

1. the unmodified Gemma 4 E2B base route; and
2. the complete OSHE Local v2 route, including its signed learned capability
   pack and required source-grounding guard.

## Results

| Measurement | Gemma 4 E2B base | OSHE Local v2 | Change |
| --- | ---: | ---: | ---: |
| Reference-score attainment | 65% | 95% | +30 percentage points |
| Schema compliance | 87.5% | 100% | +12.5 percentage points |
| Non-overconfidence | 87.5% | 100% | +12.5 percentage points |
| Parse-error rate | 0% | 0% | no change |

Reference-score attainment is a task-specific score-versus-reference ratio. It
is not an academic benchmark score. Schema compliance measures whether the
response satisfied the structured multi-document contract. Non-overconfidence
measures whether the response avoided unsupported certainty when evidence was
missing or conflicting.

## Reproducibility Bindings

| Artifact | SHA-256 |
| --- | --- |
| Gemma 4 E2B foundation artifact | `2ecb35dcf661866c74f7d45719bcf6c71e71bcf96f900b1e98968f64838effe1` |
| Frozen 32-record fixture | `011d99a0dacdb83425cf0068f5be8d41cb594f8b5f3b62289f7a6cf37b54ad2d` |
| Evaluation gates | `bd7ea33455171aae5845c520ea0032ab9d76221546ca9916a82dfae3802bedad` |
| Gemma base predictions | `eab7a8fed1378cba04fa91579e58d483471e7fe0d1b58d2ed8c5d70143b69379` |
| OSHE route predictions | `319d43fe48fb25f19c4e7ef884da88d41846980f566ef7f3048762bb1f57298d` |
| Public learned-pack weights | `c02e8c255b12a88f84570a7afa58b81f215e78b56069f2bb9a908415772ca857` |

## Limitations

- The evaluated multi-document subset contains eight cases and is intended as
  a release gate, not a comprehensive research benchmark.
- The historical aggregate evaluator reports `all_gates_passed: false` because
  its Cantonese-mimicry diagnostic scored `12.5%`. That diagnostic was
  superseded for this route: the released contract requires formal written
  Traditional Chinese, not colloquial Cantonese imitation. A subsequent frozen
  semantic review passed all `16/16` Traditional-Chinese outputs for formal
  register, while the original register-drift gate also remained at `0%`.
- The result measures the complete routed product behaviour. It must not be
  presented as the raw capability of the learned pack in isolation.
- The result does not establish performance on unrelated chat, coding,
  browsing, search or tool-use tasks.
- No claim of equivalence to Qwen 27B or any other larger model is made.