Blame for README.md

---
license: apache-2.0
---

# Team riswan — SinhalaMMLU Shared Task, ICTer 2026

This repository preserves **Codabench submission #138**, `submission_think_ens_v9_final.zip`, with provisional macro accuracy **64.2482** (Easy 65.2648, Medium 56.3456, Hard 71.134), as shown in the participant's 28 October 2026 screenshot. Ranking remains subject to organiser verification.

**Reproduction status:** the eight saved passes reproduce `predictions.jsonl` byte-for-byte. Fresh model inference has not been shown to reproduce the score. The supplied organiser template's no-sampling and Hugging-Face-only setup guidance differs from the historical system. See [REPRODUCIBILITY.md](REPRODUCIBILITY.md) for the full disclosure; this package does not certify rules compliance.

## Team and resources

- Team: **riswan**. Codabench participant: **riswan**. The registered team name was confirmed by the participant.
- [Team repository](https://huggingface.co/SinhalaMMLU-SharedTask/team-riswan) — configured repository destination; packaging changes are local until uploaded.
- [Base model: google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) — upstream model card lists Apache 2.0.
- [Ollama model: gemma4:e4b](https://ollama.com/library/gemma4:e4b) — the actual serving tag used by the original scripts; a moving tag is not an immutable model revision.
- [Competition dataset](https://huggingface.co/datasets/SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset).
- [Task website](https://sinhalammlu.ucsc.cmb.ac.lk/) and [rules](https://sinhalammlu.ucsc.cmb.ac.lk/rules.html).
- No trained model or adapter is evidenced in the submitted pipeline; it uses frozen model inference. No additional model repository is currently included.

## Method actually used

Two four-shot baselines (English analysis and direct-answer modes) contribute low-weight votes and identify disagreements. Five thinking passes use the original option order, cyclic rotations, or an additional terminology instruction. A separate Mac thinking run contributes another vote. A deterministic, difficulty-specific weighted merge combines the saved answers. Rotated votes are ignored when the positional-option regex matches. The entire system is **not zero-shot**: the four-shot examples are supplied in [data/few_shot_examples.json](data/few_shot_examples.json).

| Source | Saved answers | Thinking | Output-token limit | Temperature | Option order |
|---|---:|---|---:|---:|---|
| CoT baseline | 2238 | false | 260 | 0 | Original; four examples |
| Direct baseline | 2238 | false | 15 | 0 | Original; four examples |
| Pass 1 | 2238 | true | 2048 | 0 | Original |
| Pass 2 | 1100 | true | 4096 | 0 | Rotation +1 |
| Pass 3 | 720 | true | 4096 | 0.1, top_p 0.95 | Original; terminology hint |
| Pass 4 | 712 | true | 4096 | 0 | Rotation +2 |
| Pass 5 | 712 | true | 4096 | 0 | Rotation +3 |
| Mac | 1160 | true | 2048 | 0 | Original |

The generation requests in the archived code do **not** specify an Ollama seed. `training/ensemble_v9.py` is the exact weighted merge logic, with added validation. Ties use `Counter.most_common`: the first answer encountered in the fixed source order among tied totals wins. This is not an unweighted majority vote.

## Parameters, hardware, and timing

- **Ollama-reported parameter count: 7,996,157,674 (approximately 7.996B)**, comprising **7,518,069,290** main-model parameters plus **478,088,384** projector parameters. Both Kaggle servers running Ollama **0.40.12** reported these same component counts. This sum is below 8,000,000,000.
- These are model-metadata counts; no models were loaded when the diagnostic was captured. Any additional loaded components and final eligibility can be verified by the organisers during reproduction. See [model_manifest.json](model_manifest.json).
- Historical hardware: Kaggle **2 × NVIDIA Tesla T4**, two Ollama servers, two concurrent requests per server; separate Apple Mac with **16 GB memory** reported by the participant. Exact Mac chip and historical runtime versions still need confirmation.
- The exported notebook shows Pass 4 at **106.4 minutes after 700/719 attempts**, and Pass 5 at **105.6 minutes after 700/719 attempts**, followed by completion messages. Each complete pass took longer than its last printed progress time. Complete end-to-end runtime was not recorded in the supplied export; no A100 timing is claimed.

## Verify the archived submission without running a model

From the repository root (Python 3.10+; standard library only):

```bash
python verify_submission.py
python training/ensemble_v9.py --out predictions_v9_rebuilt.jsonl
```

`verify_submission.py` checks hashes, unique identifiers, answer ranges and byte-identical replay. The saved positional keys are archival input-derived metadata, not labels. To independently regenerate that filter and inspect target sets:

```bash
python verify_submission.py --data /path/to/dataset
```

This verifies the saved result; it is **not** a substitute for the organisers' fresh-model reproduction check.

## Fresh inference reconstruction

This runner is an audited reconstruction of a multi-machine workflow, not a claim that one script originally produced v9 end-to-end. First provision the correct Ollama runtime and original model. Historical setup is preserved in the notebook under `training/evidence/`; it uses the Ollama installer/registry, which differs from the supplied Hugging-Face-only template. Do not silently replace the quantized model with upstream BF16 and claim equivalence. Download the complete team repository, including `training/src`.

```bash
pip install -r requirements.txt
mkdir -p rerun
cd rerun
python ../inference.py
```

The script obtains all JSON question files from `DATA_REPO` on Hugging Face, then uses local Ollama endpoints. Setup downloads happen before answering. For a local offline dataset use `--local-data /path/to/dataset`. Saved test predictions are never read by the default fresh-run path. Set `OLLAMA_HOSTS` to choose local servers and `WORKDIR` to select a fresh checkpoint directory. `LIMIT=2` provides a small smoke run per difficulty; it is not a full evaluation.

The submitted `predictions.jsonl` is protected against overwriting. Running from `rerun/` writes new answers there. A full fresh Mac pass traverses the entire priority queue: the original 1160-answer partial prefix is preserved only as historical evidence, not hard-coded as a test-count rule. Consequently the fresh pipeline is not guaranteed to produce the original votes or score.

## Contents

- `predictions.jsonl`: exact final leaderboard predictions (2238 rows).
- `inference.py`, `inference.ipynb`: reconstructed fresh-run workflow and a notebook wrapper using the same script.
- `training/ensemble_v9.py`, `training/passes/`: exact merge logic and its eight historical inputs.
- `training/kaggle_cells/`, `training/src/`: archived stage and prompt code; archive edits are disclosed.
- `training/evidence/`: participant-exported notebook, original Pass 4/5 source and output logs, observed environment versions.
- `data/few_shot_examples.json`: the embedded labelled examples used in the prompts; source URLs were not recorded in the code.
- `requirements.txt`, `ENVIRONMENT.md`: minimal tested reconstruction dependency and historical environment evidence.
- `submission_manifest.json`, `model_manifest.json`, `verify_submission.py`: provenance and verification.
- `REPRODUCIBILITY.md`: unresolved differences, template cross-check, and verification scope.

Additional data provenance and the history of prompt/vote selection require author confirmation; absence of test gold labels in these files is not by itself proof that no test-based tuning occurred.