Reproduction and submission-template cross-check

Verified preservation

The package's predictions.jsonl equals the member of submission_think_ens_v9_final.zip byte-for-byte. SHA-256: 0998ea38f240c05103a7572a9a367a553f4e2b7664a31fe70f5c9668214dd554. It contains 2238 unique question keys and only answers 1–4. Both merge implementations produce the same answer for every question. The archived merge reproduces every byte.

Original run versus reconstruction

Mac prefix: all 1160 saved Mac rows, including their sequence, exactly match the first 1160 questions from the original disagreement-first, Hard/Medium/Easy priority queue. The old reconstructed rule selected only 1100 (disagreements plus all Hard). The historical local script schedules all questions; there is no evidence of a deliberate rule to select the extra 60. The current reconstruction therefore traverses the full queue. It does not encode 1160 or question-specific keys as a new-data selection rule.
Passes 4/5: the supplied notebook confirms 719 targets, both loops completed, and the code omits requests with no parseable explicit answer or exhausted retries. Each saved pass has 712 answers. The seven omissions per pass are therefore consistent with the recorded skip behavior; the exact reason for each omission cannot be recovered from prediction-only checkpoints. They are intentionally absent from archival voting. The missing-key lists differ between the two passes.
Targets: rules computed from the saved model outputs exactly match all 1100 Pass-2 and 720 Pass-3 keys. The 720-key literal target list in the exported Pass-4/5 notebook is checked against the same rule. Regenerated model answers can change these sets.
Model representation: local Mac metadata uses an MLX safetensors representation with NVFP4 tensor metadata. The notebook's Kaggle downloads have different blob identities. Hardware/runtime/model hashes and context settings must be recovered to characterize cross-platform differences. Identical model tags do not guarantee identical quantizations or answers.
Sampling: historical Pass 3 uses temperature 0.1 and top_p 0.95, without an explicit seed. The reconstructed runner retains this fact. The interrupted reconstruction had added seed=42; that was removed because it was not in the original requests. Other recorded requests use temperature 0, which alone is not a guarantee of bit-identical outputs across runtimes.
Error handling: the repaired runner uses the archived request budgets, timeouts, retry counts and rotated-option fallback mapping. Errors and fallback answers are part of the historical method; adding extra retries or replacing ambiguous answers would change the pipeline. The archived Mac parser differs slightly from the Kaggle parser and is used for its stage.
Source provenance: the exported notebook includes setup and Passes 4/5, not original execution outputs for Passes 1–3 or Mac. The other cell scripts were already present in the interrupted package and cannot be independently authenticated from this notebook. The Pass-3 archive replaced an embedded Pass-1 answer string with a file read. Cell 5's empty placeholder was repaired by recomputing its selection rule; the literal original is separately preserved in training/evidence/.

Organiser instructions: status

Requirement	Evidence/status
Repository `team-<teamname>` in organisation	Git remote points to `SinhalaMMLU-SharedTask/team-riswan`; local preparation does not prove upload
Exact final predictions	Verified byte-for-byte; submission #138
Actual generation scripts	Original exported notebook plus archived stage scripts provided; full historical execution provenance incomplete
Packages and versions	Original exported package subset retained; reconstruction dependency pinned and tested separately
Team/model/count/hardware/runtime/links	README includes evidence and explicitly marks missing details
Training/data-preparation code and extra data	Prompt examples, selection/rotation/merge code included; no parameter training evidenced in this pipeline; example provenance needs confirmation
Separate model/adapter repos if trained	No trained model or adapter evidenced; no such repository currently provided
Fresh environment end-to-end model run	Not completed; CPU replay/import checks do not satisfy this
Download everything from Hugging Face in script	Historical Ollama registry/installer differs; no exception confirmed
No sampling and fixed seed	Historical Pass 3 differs; no exception confirmed
Parameter count	Ollama reports 7,518,069,290 main-model + 478,088,384 projector = 7,996,157,674, below 8B. This is metadata; organisers can verify additional loaded components during reproduction.
No tuning/prompt design on test questions	Must be assessed from actual experiment history; lack of parameter updates does not establish compliance

Changing the model backend, temperature, historical vote weights, or stage coverage after submission cannot retroactively prove how the submitted score was produced. Any alternative implementation must be labelled as such and separately evaluated. The final eligibility decision belongs to the organisers.

Verification scope

verify_submission.py is a no-model audit of historical artifacts. --data additionally verifies selection rules and prompt-data exact-text overlap (not semantic contamination). It does not access hidden gold answers. Regression checks use fabricated questions and scripted local responses; they establish plumbing behavior only, never model accuracy. See the packaged verification report for checks actually run.

Packaging checks completed on 28 October 2026

Isolated environment installation of huggingface_hub==2.3.0 succeeded.
Remote loader fetched 41 JSON files / 2238 questions; question text, choices and keys matched the local dataset.
Both command-line replay paths reproduced the submitted prediction bytes.
Seven regression tests passed using fabricated inputs and scripted responses. Run python -B checks/test_reconstruction.py to repeat them.
Literal target indices from the supplied notebook matched the computed 720-question disagreement set.
Exact normalized question-text comparison found no matches between embedded examples and the competition questions; this does not prove absence of semantic overlap or test-based tuning.
Python syntax and notebook JSON checked; no credential-pattern hits found in package files.
No GPU/model rerun, Hugging Face upload or organiser approval was performed by this packaging audit.