File size: 8,093 Bytes
e55c8a4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 | # Reproducing our SinhalaMMLU Shared Task submission
Team: QBitsLK Β· ICTer 2026 SinhalaMMLU Shared Task
Everything below runs with no human in the loop. Inference is deterministic: no sampling, no
generation, fixed seeds, and the prediction is an argmax over summed log-softmax scores.
## 1. Quick start β reproduce the submitted predictions
```bash
pip install unsloth
pip install --no-deps transformers==5.5.0 "tokenizers>=0.22.0,<=0.23.0"
pip install "huggingface_hub>=1.5.0,<2.0"
pip install --no-deps --upgrade timm
pip install hf_transfer
python check_env.py # verifies the whole stack; exits non-zero on a problem
bash run_inference.sh /path/to/test_root ./out
```
`test_root` is the directory holding `easy/`, `medium/` and `hard/`. The script writes
`out/predictions.jsonl` and `out/predictions.zip` (with `predictions.jsonl` at the archive root).
Hardware used for every reported run: one **NVIDIA A40, 48 GB**, driver 595.91, CUDA 13.2.
Inference over the 2,238-question test set takes about **18 minutes** and peaks around 27 GB of VRAM.
## 2. The submitted system in one line
| | |
|---|---|
| model | `QBitsLK/gemma4-e4b-sinhala-mcq-cpt-v3` (merged 16-bit) |
| parameters | **7,996,156,448** β the entire inference pipeline, one model, β€ 8B |
| base | `QBitsLK/gemma4-e4b-sinhala-cpt` (Gemma 4 E4B + Sinhala continued pretraining) |
| adapter | LoRA r=32, Ξ±=32, merged into the base; merging adds no parameters |
| scoring | per-option log-likelihood, 5-prompt ensemble, argmax |
| prompt variants | `si_intro si_nointro en_intro si_rot1 si_rot2` |
| seed | 3407 |
The parameter count can be checked without downloading the weights:
```bash
python -c "
from huggingface_hub import get_safetensors_metadata
t = sum(get_safetensors_metadata('QBitsLK/gemma4-e4b-sinhala-mcq-cpt-v3').parameter_count.values())
print(f'{t:,}', 'OK, <= 8B' if t <= 8_000_000_000 else 'OVER 8B')"
```
**The adapter is merged on purpose.** Base + adapter loaded separately is 8,069,556,768 parameters,
which exceeds the cap; merging folds the LoRA into the existing weight matrices and returns the model
to 7,996,156,448.
## 3. Results reproduced by this package
Codabench, hidden test set (2,238 questions: easy 642, medium 1,111, hard 485):
| run | easy | medium | hard | **macro** |
|---|---|---|---|---|
| **submitted** β CPT + v3 data, 5-prompt ensemble | 58.26 | 62.02 | **52.58** | **57.62** |
| ablation β base Gemma 4 + v3 data, 5-prompt ensemble | 57.01 | 63.10 | 51.34 | 57.15 |
| ablation β base Gemma 4 + v3 data, single prompt | β | β | β | 55.66 |
To reproduce the first ablation, change `MODEL` in `run_inference.sh` to
`QBitsLK/gemma4-e4b-sinhala-mcq-base-v3b`. For the second, set `VARIANTS="si_intro"`.
## 4. Full reproduction from scratch
### 4.1 Rebuild the training data (CPU, a few minutes)
```bash
bash run_data_pipeline.sh
```
Requires the raw extraction files in `./data` (see section 5). Expected output: **7,872** questions
after deduplication, split into **7,151 train β 21,235 examples** and **721 validation**.
The public `TEST/easy` split is downloaded by this script and used **only** as a decontamination
blocklist. It removed **26 questions** that our past-paper extractions shared with the released test
data. It is never used for training, checkpoint selection, prompt design or hyperparameter choice.
### 4.2 Re-run the fine-tune (1 GPU, 88 minutes)
```bash
bash run_training.sh /workspace/outputs/mcq_sft_cpt_v3
```
This downloads the published SFT files from
`https://huggingface.co/datasets/QBitsLK/sinhala-mcq-sft-v3`, so section 4.1 is optional if you only
want to reproduce training.
Expected validation curve (721 held-out questions, accuracy):
| step | 0 | 150 | 300 | 450 | 600 | 664 | 750 | 900 | 1050 | 1200 | 1328 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| acc | .2316 | .5520 | .5395 | .5728 | .5506 | .5562 | .5867 | .5645 | .5742 | .5825 | **.5811** |
**The submitted checkpoint is `final_adapter` (step 1328), not `best_adapter`.** Rationale in
`SYSTEM_DESCRIPTION.md` Β§5.2: on a parallel run the validation-best checkpoint scored 0.7 points
*worse* on held-out test data than the final one, because our validation split shares the training
distribution and does not detect this failure mode.
### 4.3 Publish the merged model
```bash
export HF_TOKEN=...
python push_merged_to_hf.py \
--adapter /workspace/outputs/mcq_sft_cpt_v3/final_adapter \
--repo <your-namespace>/gemma4-e4b-sinhala-mcq-cpt-v3
```
Streams the merge directly to the Hub, so no 16 GB local copy is needed, and reads the parameter
count back for the β€8B check.
## 5. Files
| file | purpose |
|---|---|
| `run_inference.sh` | reproduces the submitted predictions (the critical path) |
| `run_training.sh` | reproduces the fine-tune |
| `run_data_pipeline.sh` | rebuilds the SFT dataset from raw extractions |
| `make_predictions.py` | prompt builder, ensemble scorer, submission writer, format checks |
| `sft_gemma4_mcq.py` | LoRA SFT with answer-only loss; per-step candidate-scoring evaluation |
| `mcq_dataset.py` | extraction cleanup, validation, decontamination, splitting, prompt building |
| `add_new_data.py` | merges per-subject extractions into the dataset |
| `normalize_subjects.py` | aligns subject names with the official `metadata.subject_original` |
| `eval_sinhalammlu.py` | read-only analysis; also builds the decontamination blocklist |
| `push_merged_to_hf.py` | merges the adapter and publishes it |
| `check_env.py` | verifies the dependency stack before a run |
| `requirements.txt` | exact pins and install order |
**Not included:** the source examination PDFs. They are third-party past papers obtained from
publicly available sources (`pastpapers.wiki`, `e-thaksalawa.moe.gov.lk`), and we do not hold
redistribution rights. The derived question set β questions, options and answer keys β is published
at `QBitsLK/sinhala-mcq-sft-v3`, which is sufficient to reproduce training and inference. The raw
per-subject extraction files are included so `run_data_pipeline.sh` runs end-to-end.
## 6. Determinism
- No sampling anywhere; `model.generate` is never called on the submission path.
- `random.seed(3407)` and `torch.manual_seed(3407)`.
- The prediction is `argmax` over log-softmax sums; ties resolve to the lowest option index.
- Prompt variants are explicit in `run_inference.sh`; changing them changes the predictions.
- `make_predictions.py` refuses to write the zip unless the line count equals the question count,
every line carries exactly `{q_no, subject, category, level, answer}`, `answer` is a `"1"`β`"5"`
string, and every `level` is `easy`/`medium`/`hard`.
One deliberate deviation from the organisers' reference code: `evaluate.py` hard-codes the *hard*
instruction as "1, 2, 3, 4, 5" because the original SinhalaMMLU hard split had five options.
SinhalaMMLU 2.0 drops one incorrect option, so hard questions now have four. `fit_numbers()` in
`make_predictions.py` rewrites the number list to match the actual option count (verified: 0 of 2,238
prompts disagree). Without this, all 485 hard questions would be asked to choose from five options
when only four are shown.
## 7. Citation
This work evaluates on the SinhalaMMLU benchmark:
```bibtex
@inproceedings{pramodya-etal-2025-sinhalammlu,
title = "SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala",
author = "Pramodya, Ashmari and Nelki, Nirasha and Shalinda, Heshan and Liyanage, Chamila and
Sakai, Yusuke and Pushpananda, Randil and Weerasinghe, Ruvan and
Kamigaito, Hidetaka and Watanabe, Taro",
booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
year = "2025",
pages = "32943--32961",
url = "https://aclanthology.org/2025.emnlp-main.1673/"
}
```
Verify this against the BibTeX on <https://sinhalammlu.ucsc.cmb.ac.lk/task.html> before submitting;
it is reproduced here from that page and the arXiv preprint (arXiv:2509.03162).
|