| # Team Nexus – Sinhala MMLU Shared Task System |
|
|
| ## Team |
| | Name | Email | |
| |---|---| |
| | S.A. Talagala | samadhianjanatalagala@gmail.com | |
| | Anas Hussaindeen | anashussaindeen@gmail.com | |
| | Ifadha Imthiyas | ifa.imty015@gmail.com | |
| | V.D.W. Muthumala | vihangamuthumala@gmail.com | |
|
|
| ## System summary |
| - **Base model(s):** [`google/gemma-4-E4B-it`](https://huggingface.co/google/gemma-4-E4B-it), commit `ee0ef6023621cff504d758262d4e04895a5af4a2`, used **text-only**: the vision and audio towers are removed and every language-model weight is copied unchanged (`training/scripts/make_text_only.py` checks that the copy is exact). |
| - **What we did:** prompting only, with no fine-tuning, retrieval or extra data at inference. Every question is answered in Gemma 4's thinking mode with **greedy decoding** (no sampling) and up to 3,072 new tokens, and the final choice (1–4) is read from the model's answer. A fast one-step answer (next-token scores over the labels 1–4, averaged over 4 cyclic orders of the options) is computed too, but is used only as a fallback if a thinking answer cannot be read; this happened for 1 of 2,238 test questions. |
| - **Total inference parameters:** **7.463B** (7,463,013,376; one model, no adapter). This is the count of the weights actually loaded, with tied embeddings counted once. |
| - **Extra data used for training or retrieval:** none in the final system. See "Methods and data" below for everything we used during development. |
|
|
| ## Models and resources in this submission |
| | Repo | What it is | |
| |---|---| |
| | [`SinhalaMMLU-SharedTask/team-nexus`](https://huggingface.co/SinhalaMMLU-SharedTask/team-nexus) | Code, inference script and final predictions (this repo) | |
| | [`SinhalaMMLU-SharedTask/team-nexus-gemma-4-e4b-it-text`](https://huggingface.co/SinhalaMMLU-SharedTask/team-nexus-gemma-4-e4b-it-text) | Text-only Gemma 4 E4B-it: unchanged weights of `google/gemma-4-E4B-it` with the vision and audio towers removed (Apache 2.0). `inference.py` pins commit `8f686039aad1a865c9335c17874d9b870fdec766`. | |
|
|
| ## How to run |
| ```bash |
| pip install -r requirements.txt |
| python inference.py |
| ``` |
| `inference.py` follows the organisers' template. Section 1 loads every `.json` file in `DATA_REPO`. Section 2 downloads the model from the repo above and runs our pipeline (`inference/predict.py` with `configs/submission_base_think_greedy.yaml`), the code that produced our Codabench file. Section 3 writes `predictions.jsonl`. |
|
|
| - **GPU used:** 1× NVIDIA DGX Spark (GB10 Grace Blackwell, 128 GB unified CPU/GPU memory), bf16. |
| - **Approximate run time on the test set:** about **4.5 hours** (16,277 s for 2,238 questions, including the model download) on the GB10. Most of this is thinking-mode generation; faster GPUs need proportionally less. |
| - **Anything else the organizers need to know:** |
| - **Memory:** generation uses batches of up to 64 sequences, with a padded prompt budget of 16,384 tokens and up to 3,072 new tokens each. Plan for a GPU with **80 GB or more**, such as an A100 or H100 80 GB. A 16 GB T4 is not enough. |
| - **Decoding:** greedy only (`do_sample=False` behaviour); there is no sampling anywhere in the submitted system. The seed (1234) is still set. |
| - **Reproducibility:** thinking outputs are long (hundreds to thousands of tokens) and generated in bf16 in batched, length-sorted, left-padded batches. Answers can therefore depend on the GPU type and on which questions share a batch. We ran the published `inference.py` (this repo, model downloaded from the repo above, fresh environment) on the full test set, and its answers differed from our earlier full run on the same machine in exactly 127 questions, the same number of answers that earlier run's sampling vote had changed. A run on a much smaller subset changes the batches and can differ by a minority of answers, so **please run the full dataset with the default settings.** On a different GPU type, expect a minority of answers to differ, with a similar overall score. |
| - **Earlier variant:** setting `SELF_CONSISTENCY = True` at the top of `inference.py` runs an earlier variant that adds seeded sampled reasoning paths and a majority vote on part of the questions. We do **not** submit it, because the task asks for no sampling. |
| - **Gating:** the model repo is public and the base model is not gated, so no login is needed. |
|
|
| ## Files |
| - `inference.py` – the entry point that runs the exact pipeline behind our final Codabench submission |
| - `inference/predict.py`, `eval/` – the pipeline code it calls (prompting, scoring, thinking, self-consistency) |
| - `configs/submission_base_think_greedy.yaml` – the settings of the submitted system; `configs/submission_base_think_gated_sc.yaml` – the earlier sampling variant (not submitted) |
| - `predictions.jsonl` – the exact file we submitted to the final leaderboard (sha256 `b28acab09e6c926b3f40b487af393937c9d19eafd98445a83dec48552852facf`) |
| - `requirements.txt` – package versions we used (Python 3.12.3) |
| - `training/` – all our data-preparation, training and evaluation code and configurations from development (see below) |
|
|
| ## Methods and data (complete list) |
| **The final system uses no training data and no fine-tuning**: it is the base model, prompted, with greedy thinking (see "System summary"). The only data it touches are the test questions it is asked to answer. Everything below is development work in `training/`; none of it is part of the final system, and none of it beat the base model with thinking on our held-out data. |
|
|
| ### Methods we tried |
| 1. **Prompting:** English zero-shot (our baseline), Sinhala-language instructions, fixed few-shot, and other prompt variants. None beat plain English zero-shot on Gemma 4 E4B. |
| 2. **Option-order averaging:** averaging the label probabilities over 4 cyclic orders of the options. Used for the fast fallback answer; also tested as option-order voting over thinking runs (rejected by our pre-registered decision rule). |
| 3. **Model comparison and parameter accounting:** screening of open ≤ 8B candidates (Gemma 4 E4B/E2B, Gemma 3n E4B/E2B, Gemma 3 4B, Qwen3 and Qwen3.5 4B, Llama 3.2 3B, SmolLM3 3B, Phi-4-mini, Granite 4.x 3B), plus exact parameter counts (`training/MODEL_MANIFEST.md`). Gemma 4 E4B-it was the strongest of those we evaluated. |
| 4. **Thinking mode:** the model's built-in reasoning before the answer. This is the main gain and is used in the final system. |
| 5. **LoRA fine-tuning (SFT):** on Global-MMLU `si` questions, and on SinhalaMMLU dev with 5-fold cross-validation (never scored on its own training questions). |
| 6. **Continued pre-training (LoRA):** on cleaned Sinhala Wikipedia, FineWeb-2 and NSINA, with about 10% English FineWeb-Edu. |
| 7. **Distillation from larger open teachers:** soft labels on Global-MMLU `si` training questions from locally run open models (Gemma 4 31B-it for the training run; Qwen3.5-27B and Gemma 4 26B-A4B-it were also evaluated as teachers). |
| 8. **Synthetic MCQs:** a small pilot of Sinhala exam-style questions written by Gemma 4 31B from Sinhala Wikipedia passages, then validated, round-trip checked and soft-labelled by the same model. |
| 9. **Reasoning-trace fine-tuning (STaR-style rejection sampling):** the base model's own correct thinking traces on held-out SinhalaMMLU dev questions (round 1), plus Global-MMLU `si` Humanities / Social Sciences / Other / Business questions and the synthetic Wikipedia MCQs (round 2); LoRA with loss on the completion only. |
| 10. **Self-consistency:** sampled reasoning paths with majority vote, including an agreement-gated variant. Not part of the final system, because the task asks for no sampling. It is still in the code behind `SELF_CONSISTENCY = True`. |
|
|
| Each step was decided on SinhalaMMLU dev or Global-MMLU validation subsets, never on test questions. All models were run locally and are open-weight. No closed or paid API was used at any stage. |
|
|
| ### Data sources |
| | Data | Source and revision | Licence | How it was used | |
| |---|---|---|---| |
| | Shared-task test questions | [`SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset`](https://huggingface.co/datasets/SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset), rev `56fb2da50c` | set by organisers | **Prediction only.** Never used for training, tuning, prompt design or model selection | |
| | SinhalaMMLU public Easy tier (dev) | [`naist-nlp/SinhalaMMLU`](https://huggingface.co/datasets/naist-nlp/SinhalaMMLU), TEST/easy, rev `7dd30681d9` | CC BY-NC-ND 4.0 (not redistributed) | Development evaluation; SFT cross-validation; source of questions for round-1 reasoning traces | |
| | Global-MMLU `si` | [`CohereLabs/Global-MMLU`](https://huggingface.co/datasets/CohereLabs/Global-MMLU), rev `0e619dbeb3` | Apache-2.0 | 2,000-question validation sample; remaining questions for SFT, distillation and round-2 traces | |
| | Belebele `sin_Sinh` | [`facebook/belebele`](https://huggingface.co/datasets/facebook/belebele), rev `7899cdfa4e` | CC BY-SA 4.0 | Evaluation only | |
| | Sinhala Wikipedia | [`wikimedia/wikipedia`](https://huggingface.co/datasets/wikimedia/wikipedia) `20231101.si`, rev `b04c8d1ceb` | CC BY-SA 3.0 / GFDL | Continued pre-training; passages for synthetic MCQs | |
| | FineWeb-2 `sin_Sinh` | [`HuggingFaceFW/fineweb-2`](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2), rev `af9c13333e` | ODC-By 1.0 | Continued pre-training; passages for synthetic MCQs | |
| | NSINA news | [`sinhala-nlp/NSINA`](https://huggingface.co/datasets/sinhala-nlp/NSINA), rev `a986af81ad` | CC BY-SA 4.0 | Continued pre-training; passages for synthetic MCQs | |
| | FineWeb-Edu (English sample) | [`HuggingFaceFW/fineweb-edu`](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) `sample/10BT`, rev `87f09149ef` | ODC-By 1.0 | About 10% of the continued pre-training mix | |
| | Synthetic MCQs and reasoning traces | Generated by us from the sources above with local open models | n/a | Training experiments only; not published | |
|
|
| **Contamination controls** (details in `training/DATA_LOG.md`): web documents from exam and past-paper sites were removed from all pre-training corpora; training questions were filtered by character n-gram similarity against the SinhalaMMLU dev and Global-MMLU validation questions; Global-MMLU was checked for overlap with SinhalaMMLU using character n-gram and multilingual-embedding similarity. |
|
|
| ### Models used during development (all open-weight, run locally) |
| Base: [`google/gemma-4-E4B-it`](https://huggingface.co/google/gemma-4-E4B-it) (Apache-2.0). Teachers and comparison models: `google/gemma-4-31B-it`, `google/gemma-4-26B-A4B-it`, `Qwen/Qwen3.5-27B`, and the ≤ 8B candidates listed above. For the overlap checks only: `intfloat/multilingual-e5-large` (embedding model, MIT licence). |
|
|
| ## Citation |
| ```bibtex |
| @inproceedings{pramodya-etal-2025-sinhalammlu, |
| title = "{S}inhala{MMLU}: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in {S}inhala", |
| author = "Pramodya, Ashmari and Nelki, Nirasha and Shalinda, Heshan and Liyanage, Chamila and Sakai, Yusuke and |
| Pushpananda, Randil and Weerasinghe, Ruvan and Kamigaito, Hidetaka and Watanabe, Taro", |
| booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing", |
| year = "2025", |
| pages = "32931--32949", |
| url = "https://aclanthology.org/2025.emnlp-main.1673/" |
| } |
| ``` |
|
|
