File size: 8,093 Bytes
e55c8a4
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
# Reproducing our SinhalaMMLU Shared Task submission

Team: QBitsLK Β· ICTer 2026 SinhalaMMLU Shared Task

Everything below runs with no human in the loop. Inference is deterministic: no sampling, no
generation, fixed seeds, and the prediction is an argmax over summed log-softmax scores.

## 1. Quick start β€” reproduce the submitted predictions

```bash
pip install unsloth
pip install --no-deps transformers==5.5.0 "tokenizers>=0.22.0,<=0.23.0"
pip install "huggingface_hub>=1.5.0,<2.0"
pip install --no-deps --upgrade timm
pip install hf_transfer

python check_env.py                 # verifies the whole stack; exits non-zero on a problem

bash run_inference.sh /path/to/test_root ./out
```

`test_root` is the directory holding `easy/`, `medium/` and `hard/`. The script writes
`out/predictions.jsonl` and `out/predictions.zip` (with `predictions.jsonl` at the archive root).

Hardware used for every reported run: one **NVIDIA A40, 48 GB**, driver 595.91, CUDA 13.2.
Inference over the 2,238-question test set takes about **18 minutes** and peaks around 27 GB of VRAM.

## 2. The submitted system in one line

| | |
|---|---|
| model | `QBitsLK/gemma4-e4b-sinhala-mcq-cpt-v3` (merged 16-bit) |
| parameters | **7,996,156,448** β€” the entire inference pipeline, one model, ≀ 8B |
| base | `QBitsLK/gemma4-e4b-sinhala-cpt` (Gemma 4 E4B + Sinhala continued pretraining) |
| adapter | LoRA r=32, Ξ±=32, merged into the base; merging adds no parameters |
| scoring | per-option log-likelihood, 5-prompt ensemble, argmax |
| prompt variants | `si_intro si_nointro en_intro si_rot1 si_rot2` |
| seed | 3407 |

The parameter count can be checked without downloading the weights:

```bash
python -c "
from huggingface_hub import get_safetensors_metadata
t = sum(get_safetensors_metadata('QBitsLK/gemma4-e4b-sinhala-mcq-cpt-v3').parameter_count.values())
print(f'{t:,}', 'OK, <= 8B' if t <= 8_000_000_000 else 'OVER 8B')"
```

**The adapter is merged on purpose.** Base + adapter loaded separately is 8,069,556,768 parameters,
which exceeds the cap; merging folds the LoRA into the existing weight matrices and returns the model
to 7,996,156,448.

## 3. Results reproduced by this package

Codabench, hidden test set (2,238 questions: easy 642, medium 1,111, hard 485):

| run | easy | medium | hard | **macro** |
|---|---|---|---|---|
| **submitted** β€” CPT + v3 data, 5-prompt ensemble | 58.26 | 62.02 | **52.58** | **57.62** |
| ablation β€” base Gemma 4 + v3 data, 5-prompt ensemble | 57.01 | 63.10 | 51.34 | 57.15 |
| ablation β€” base Gemma 4 + v3 data, single prompt | β€” | β€” | β€” | 55.66 |

To reproduce the first ablation, change `MODEL` in `run_inference.sh` to
`QBitsLK/gemma4-e4b-sinhala-mcq-base-v3b`. For the second, set `VARIANTS="si_intro"`.

## 4. Full reproduction from scratch

### 4.1 Rebuild the training data (CPU, a few minutes)

```bash
bash run_data_pipeline.sh
```

Requires the raw extraction files in `./data` (see section 5). Expected output: **7,872** questions
after deduplication, split into **7,151 train β†’ 21,235 examples** and **721 validation**.

The public `TEST/easy` split is downloaded by this script and used **only** as a decontamination
blocklist. It removed **26 questions** that our past-paper extractions shared with the released test
data. It is never used for training, checkpoint selection, prompt design or hyperparameter choice.

### 4.2 Re-run the fine-tune (1 GPU, 88 minutes)

```bash
bash run_training.sh /workspace/outputs/mcq_sft_cpt_v3
```

This downloads the published SFT files from
`https://huggingface.co/datasets/QBitsLK/sinhala-mcq-sft-v3`, so section 4.1 is optional if you only
want to reproduce training.

Expected validation curve (721 held-out questions, accuracy):

| step | 0 | 150 | 300 | 450 | 600 | 664 | 750 | 900 | 1050 | 1200 | 1328 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| acc | .2316 | .5520 | .5395 | .5728 | .5506 | .5562 | .5867 | .5645 | .5742 | .5825 | **.5811** |

**The submitted checkpoint is `final_adapter` (step 1328), not `best_adapter`.** Rationale in
`SYSTEM_DESCRIPTION.md` Β§5.2: on a parallel run the validation-best checkpoint scored 0.7 points
*worse* on held-out test data than the final one, because our validation split shares the training
distribution and does not detect this failure mode.

### 4.3 Publish the merged model

```bash
export HF_TOKEN=...
python push_merged_to_hf.py \
  --adapter /workspace/outputs/mcq_sft_cpt_v3/final_adapter \
  --repo <your-namespace>/gemma4-e4b-sinhala-mcq-cpt-v3
```

Streams the merge directly to the Hub, so no 16 GB local copy is needed, and reads the parameter
count back for the ≀8B check.

## 5. Files

| file | purpose |
|---|---|
| `run_inference.sh` | reproduces the submitted predictions (the critical path) |
| `run_training.sh` | reproduces the fine-tune |
| `run_data_pipeline.sh` | rebuilds the SFT dataset from raw extractions |
| `make_predictions.py` | prompt builder, ensemble scorer, submission writer, format checks |
| `sft_gemma4_mcq.py` | LoRA SFT with answer-only loss; per-step candidate-scoring evaluation |
| `mcq_dataset.py` | extraction cleanup, validation, decontamination, splitting, prompt building |
| `add_new_data.py` | merges per-subject extractions into the dataset |
| `normalize_subjects.py` | aligns subject names with the official `metadata.subject_original` |
| `eval_sinhalammlu.py` | read-only analysis; also builds the decontamination blocklist |
| `push_merged_to_hf.py` | merges the adapter and publishes it |
| `check_env.py` | verifies the dependency stack before a run |
| `requirements.txt` | exact pins and install order |

**Not included:** the source examination PDFs. They are third-party past papers obtained from
publicly available sources (`pastpapers.wiki`, `e-thaksalawa.moe.gov.lk`), and we do not hold
redistribution rights. The derived question set β€” questions, options and answer keys β€” is published
at `QBitsLK/sinhala-mcq-sft-v3`, which is sufficient to reproduce training and inference. The raw
per-subject extraction files are included so `run_data_pipeline.sh` runs end-to-end.

## 6. Determinism

- No sampling anywhere; `model.generate` is never called on the submission path.
- `random.seed(3407)` and `torch.manual_seed(3407)`.
- The prediction is `argmax` over log-softmax sums; ties resolve to the lowest option index.
- Prompt variants are explicit in `run_inference.sh`; changing them changes the predictions.
- `make_predictions.py` refuses to write the zip unless the line count equals the question count,
  every line carries exactly `{q_no, subject, category, level, answer}`, `answer` is a `"1"`–`"5"`
  string, and every `level` is `easy`/`medium`/`hard`.

One deliberate deviation from the organisers' reference code: `evaluate.py` hard-codes the *hard*
instruction as "1, 2, 3, 4, 5" because the original SinhalaMMLU hard split had five options.
SinhalaMMLU 2.0 drops one incorrect option, so hard questions now have four. `fit_numbers()` in
`make_predictions.py` rewrites the number list to match the actual option count (verified: 0 of 2,238
prompts disagree). Without this, all 485 hard questions would be asked to choose from five options
when only four are shown.

## 7. Citation

This work evaluates on the SinhalaMMLU benchmark:

```bibtex
@inproceedings{pramodya-etal-2025-sinhalammlu,
    title = "SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala",
    author = "Pramodya, Ashmari and Nelki, Nirasha and Shalinda, Heshan and Liyanage, Chamila and
              Sakai, Yusuke and Pushpananda, Randil and Weerasinghe, Ruvan and
              Kamigaito, Hidetaka and Watanabe, Taro",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    year = "2025",
    pages = "32943--32961",
    url = "https://aclanthology.org/2025.emnlp-main.1673/"
}
```

Verify this against the BibTeX on <https://sinhalammlu.ucsc.cmb.ac.lk/task.html> before submitting;
it is reproduced here from that page and the arXiv preprint (arXiv:2509.03162).