| --- |
| license: apache-2.0 |
| --- |
| |
| # Team riswan — SinhalaMMLU Shared Task, ICTer 2026 |
|
|
| This repository preserves **Codabench submission #138**, `submission_think_ens_v9_final.zip`, with provisional macro accuracy **64.2482** (Easy 65.2648, Medium 56.3456, Hard 71.134), as shown in the participant's 28 October 2026 screenshot. Ranking remains subject to organiser verification. |
|
|
| **Reproduction status:** the eight saved passes reproduce `predictions.jsonl` byte-for-byte. Fresh model inference has not been shown to reproduce the score. The supplied organiser template's no-sampling and Hugging-Face-only setup guidance differs from the historical system. See [REPRODUCIBILITY.md](REPRODUCIBILITY.md) for the full disclosure; this package does not certify rules compliance. |
|
|
| ## Team and resources |
|
|
| - Team: **riswan**. Codabench participant: **riswan**. The registered team name was confirmed by the participant. |
| - [Team repository](https://huggingface.co/SinhalaMMLU-SharedTask/team-riswan) — configured repository destination; packaging changes are local until uploaded. |
| - [Base model: google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) — upstream model card lists Apache 2.0. |
| - [Ollama model: gemma4:e4b](https://ollama.com/library/gemma4:e4b) — the actual serving tag used by the original scripts; a moving tag is not an immutable model revision. |
| - [Competition dataset](https://huggingface.co/datasets/SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset). |
| - [Task website](https://sinhalammlu.ucsc.cmb.ac.lk/) and [rules](https://sinhalammlu.ucsc.cmb.ac.lk/rules.html). |
| - No trained model or adapter is evidenced in the submitted pipeline; it uses frozen model inference. No additional model repository is currently included. |
|
|
| ## Method actually used |
|
|
| Two four-shot baselines (English analysis and direct-answer modes) contribute low-weight votes and identify disagreements. Five thinking passes use the original option order, cyclic rotations, or an additional terminology instruction. A separate Mac thinking run contributes another vote. A deterministic, difficulty-specific weighted merge combines the saved answers. Rotated votes are ignored when the positional-option regex matches. The entire system is **not zero-shot**: the four-shot examples are supplied in [data/few_shot_examples.json](data/few_shot_examples.json). |
|
|
| | Source | Saved answers | Thinking | Output-token limit | Temperature | Option order | |
| |---|---:|---|---:|---:|---| |
| | CoT baseline | 2238 | false | 260 | 0 | Original; four examples | |
| | Direct baseline | 2238 | false | 15 | 0 | Original; four examples | |
| | Pass 1 | 2238 | true | 2048 | 0 | Original | |
| | Pass 2 | 1100 | true | 4096 | 0 | Rotation +1 | |
| | Pass 3 | 720 | true | 4096 | 0.1, top_p 0.95 | Original; terminology hint | |
| | Pass 4 | 712 | true | 4096 | 0 | Rotation +2 | |
| | Pass 5 | 712 | true | 4096 | 0 | Rotation +3 | |
| | Mac | 1160 | true | 2048 | 0 | Original | |
| |
| The generation requests in the archived code do **not** specify an Ollama seed. `training/ensemble_v9.py` is the exact weighted merge logic, with added validation. Ties use `Counter.most_common`: the first answer encountered in the fixed source order among tied totals wins. This is not an unweighted majority vote. |
| |
| ## Parameters, hardware, and timing |
| |
| - **Ollama-reported parameter count: 7,996,157,674 (approximately 7.996B)**, comprising **7,518,069,290** main-model parameters plus **478,088,384** projector parameters. Both Kaggle servers running Ollama **0.40.12** reported these same component counts. This sum is below 8,000,000,000. |
| - These are model-metadata counts; no models were loaded when the diagnostic was captured. Any additional loaded components and final eligibility can be verified by the organisers during reproduction. See [model_manifest.json](model_manifest.json). |
| - Historical hardware: Kaggle **2 × NVIDIA Tesla T4**, two Ollama servers, two concurrent requests per server; separate Apple Mac with **16 GB memory** reported by the participant. Exact Mac chip and historical runtime versions still need confirmation. |
| - The exported notebook shows Pass 4 at **106.4 minutes after 700/719 attempts**, and Pass 5 at **105.6 minutes after 700/719 attempts**, followed by completion messages. Each complete pass took longer than its last printed progress time. Complete end-to-end runtime was not recorded in the supplied export; no A100 timing is claimed. |
| |
| ## Verify the archived submission without running a model |
| |
| From the repository root (Python 3.10+; standard library only): |
| |
| ```bash |
| python verify_submission.py |
| python training/ensemble_v9.py --out predictions_v9_rebuilt.jsonl |
| ``` |
| |
| `verify_submission.py` checks hashes, unique identifiers, answer ranges and byte-identical replay. The saved positional keys are archival input-derived metadata, not labels. To independently regenerate that filter and inspect target sets: |
|
|
| ```bash |
| python verify_submission.py --data /path/to/dataset |
| ``` |
|
|
| This verifies the saved result; it is **not** a substitute for the organisers' fresh-model reproduction check. |
|
|
| ## Fresh inference reconstruction |
|
|
| This runner is an audited reconstruction of a multi-machine workflow, not a claim that one script originally produced v9 end-to-end. First provision the correct Ollama runtime and original model. Historical setup is preserved in the notebook under `training/evidence/`; it uses the Ollama installer/registry, which differs from the supplied Hugging-Face-only template. Do not silently replace the quantized model with upstream BF16 and claim equivalence. Download the complete team repository, including `training/src`. |
|
|
| ```bash |
| pip install -r requirements.txt |
| mkdir -p rerun |
| cd rerun |
| python ../inference.py |
| ``` |
|
|
| The script obtains all JSON question files from `DATA_REPO` on Hugging Face, then uses local Ollama endpoints. Setup downloads happen before answering. For a local offline dataset use `--local-data /path/to/dataset`. Saved test predictions are never read by the default fresh-run path. Set `OLLAMA_HOSTS` to choose local servers and `WORKDIR` to select a fresh checkpoint directory. `LIMIT=2` provides a small smoke run per difficulty; it is not a full evaluation. |
|
|
| The submitted `predictions.jsonl` is protected against overwriting. Running from `rerun/` writes new answers there. A full fresh Mac pass traverses the entire priority queue: the original 1160-answer partial prefix is preserved only as historical evidence, not hard-coded as a test-count rule. Consequently the fresh pipeline is not guaranteed to produce the original votes or score. |
|
|
| ## Contents |
|
|
| - `predictions.jsonl`: exact final leaderboard predictions (2238 rows). |
| - `inference.py`, `inference.ipynb`: reconstructed fresh-run workflow and a notebook wrapper using the same script. |
| - `training/ensemble_v9.py`, `training/passes/`: exact merge logic and its eight historical inputs. |
| - `training/kaggle_cells/`, `training/src/`: archived stage and prompt code; archive edits are disclosed. |
| - `training/evidence/`: participant-exported notebook, original Pass 4/5 source and output logs, observed environment versions. |
| - `data/few_shot_examples.json`: the embedded labelled examples used in the prompts; source URLs were not recorded in the code. |
| - `requirements.txt`, `ENVIRONMENT.md`: minimal tested reconstruction dependency and historical environment evidence. |
| - `submission_manifest.json`, `model_manifest.json`, `verify_submission.py`: provenance and verification. |
| - `REPRODUCIBILITY.md`: unresolved differences, template cross-check, and verification scope. |
|
|
| Additional data provenance and the history of prompt/vote selection require author confirmation; absence of test gold labels in these files is not by itself proof that no test-based tuning occurred. |
|
|
