openjev-e4b / REPRODUCIBILITY.md
bambamdevs's picture
Publish OpenJEV E4B 1.0
03223d7
|
Raw History Blame Contribute Delete
2.36 kB

Reproducibility contract

Model identity

OpenJEV E4B 1.0 requires the google/gemma-4-E4B-it base at revision ee0ef6023621cff504d758262d4e04895a5af4a2, its LoRA adapter, hybrid pointer and span decision head, cumulative 65-tensor backbone delta for layers 37–41, calibration (global_temperature = 1.0), and the packing and scoring code in openjev/. The adapter alone does not restore the model. Every released file is hashed in MANIFEST.sha256; source-checkpoint hashes and the internal checkpoint identifier are in RELEASE_METADATA.json.

Choice-panel evaluation

The frozen row file is eval/choice_panel/rows.jsonl, with SHA-256 in eval/choice_panel/rows.sha256. The construction audit is leak_audit.json. GPQA answers are placed by a fixed per-row shuffle. GSM8K distractors are answers to other test problems of similar magnitude. Chess legal distractors are plain legal-move strings drawn from other positions and illegal in the current position.

The runner presents every row in all cyclic option rotations for the full condition. For options-only, it replaces state and question with neutral text and uses all four rotations on four-option tasks and five evenly spaced rotations on ten-option tasks. Output probabilities are mapped back to original option identities. The main accuracy is argmax of the mean mapped probabilities for each row; Wilson 95% intervals use rows as observations. Mean per-order accuracy and same-answer share are reported separately. See eval/choice_panel/summary.json and docs/EXPERIMENT_1_RESULTS.md.

Untouched Gemma uses native LM-head answer-letter scoring in NF4. OpenJEV 1.0 uses the frozen base in NF4, delta-touched layers in dense FP16, LoRA, and an FP32 head. The comparison measures complete decision systems and does not isolate any one component.

Standard public tasks

ARC-Easy, ARC-Challenge, WinoGrande, HellaSwag, and MMLU use the native-order runner in eval/harness/benchmark_public.py. Their row-level runs and summaries are included under eval/. ARC, HellaSwag, and MMLU have training-family exposure recorded in docs/BENCHMARK_EXPOSURE.md.

Training record

The whitepaper describes model evolution and the Arena promotion rule. Source checkpoint records and hashes are in provenance/ and RELEASE_METADATA.json. Training code is not part of this release.