ovinduG commited on
Commit
5033d6d
Β·
verified Β·
1 Parent(s): b770ded

Upload folder using huggingface_hub

Browse files
.ipynb_checkpoints/README-checkpoint.md ADDED
@@ -0,0 +1,98 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: gemma
3
+ language:
4
+ - si
5
+ base_model: QBitsLK/gemma4-e4b-sinhala-cpt
6
+ tags:
7
+ - sinhala
8
+ - multiple-choice
9
+ - sinhalammlu
10
+ - shared-task
11
+ ---
12
+
13
+ # Team QBitsLK – Sinhala MMLU Shared Task System
14
+
15
+ ## System summary
16
+ - **Base model(s):** `google/gemma-4-E4B` (text-only), with Sinhala continued pretraining, then LoRA supervised fine-tuning on examination-paper MCQs.
17
+ - **What we did:** We built a 7,872-question Sinhala multiple-choice dataset by extracting from publicly available Sri Lankan O/L and Grade 6–11 past papers, cleaning OCR artefacts and decontaminating against the released SinhalaMMLU test split. We continued pretraining Gemma 4 E4B on Sinhala text, then fine-tuned with LoRA (r=32) using answer-only loss, and merged the adapter back into the base weights. At inference each option is scored by per-option log-likelihood under a 5-prompt ensemble (Sinhala and English instructions, subject intro on/off, two option-order rotations) rather than by generation. Nothing is sampled.
18
+ - **Total inference parameters:** **7,996,157,418** (one merged model; ≀ 8B)
19
+ - **Extra data used for training or retrieval:** our own past-paper MCQ dataset, published at `SinhalaMMLU-SharedTask/team-qbits-mcq-data`. No retrieval, no external APIs, no closed models.
20
+
21
+ ## Models and resources in this submission
22
+ | Repo | What it is |
23
+ |---|---|
24
+ | `SinhalaMMLU-SharedTask/team-qbits` | code, inference script, final predictions (this repo) |
25
+ | `SinhalaMMLU-SharedTask/team-qbits-cpt-v3` | **the submitted model** β€” merged 16-bit, 7,996,157,418 params |
26
+ | `SinhalaMMLU-SharedTask/team-qbits-cpt-v3-lora` | the LoRA adapter for the above (r=32), for inspection |
27
+ | `SinhalaMMLU-SharedTask/team-qbits-base-v3b` | ablation β€” same recipe on the plain Gemma 4 E4B base, merged |
28
+ | `SinhalaMMLU-SharedTask/team-qbits-base-v3b-lora` | the LoRA adapter for the ablation |
29
+ | `SinhalaMMLU-SharedTask/team-qbits-mcq-data` | the SFT dataset (prompt/completion files) |
30
+
31
+ `inference.py` uses **only** `team-qbits-cpt-v3`. The other repos are provided for transparency and
32
+ are not part of the inference pipeline.
33
+
34
+ ## How to run
35
+ ```bash
36
+ pip install -r requirements.txt
37
+ python inference.py
38
+ ```
39
+ - **GPU used:** 1x NVIDIA A40 48 GB (any GPU with β‰₯ 20 GB should work; the model is 16 GB in bf16)
40
+ - **Approximate run time on the test set:** ~18 minutes for 2,238 questions (5 prompt variants Γ— 4 options = ~44,800 scored candidates); peak VRAM ~27 GB
41
+ - **Anything else the organizers need to know:**
42
+ - The Gemma license must be accepted on the Hugging Face account used to run this. Our merged model is self-contained, so no separate base-model download is required.
43
+ - `inference.py` scores options by log-likelihood and never calls `generate`, so there is no sampling. With the fixed seed, repeated runs produce byte-identical `predictions.jsonl`.
44
+ - `VARIANTS` at the top of `inference.py` defines the prompt ensemble. **Changing that list changes the predictions**; it is set exactly as used for our final Codabench submission.
45
+ - The reference `util_prompt.py` hard-codes the *hard* instruction as "1, 2, 3, 4, 5", which was correct for the original five-option hard split. SinhalaMMLU 2.0 reduced hard questions to four options, so `fit_numbers()` derives the number list from the actual option count. Verified: 0 of 2,238 prompts disagree with their option count.
46
+
47
+ ## Results (Codabench)
48
+
49
+ | system | easy | medium | hard | **macro** |
50
+ |---|---|---|---|---|
51
+ | **submitted** β€” `team-qbits-cpt-v3`, 5-prompt ensemble | 58.10 | 62.56 | 52.79 | **57.81** |
52
+ | the same model with the adapter applied at load time instead of merged | 58.26 | 62.02 | 52.58 | 57.62 |
53
+ | ablation β€” `team-qbits-base-v3b` (plain Gemma 4 base), 5-prompt ensemble | 57.01 | 63.10 | 51.34 | 57.15 |
54
+ | ablation β€” plain base, single prompt, no ensemble | β€” | β€” | β€” | 55.66 |
55
+
56
+ `inference.py` reproduces the submitted row exactly: it loads the merged model, which is the system
57
+ we ship. Row 2 is the same fine-tune evaluated with the LoRA applied at load time (base + adapter).
58
+ That form is 8,069,557,738 parameters and therefore **exceeds the 8B cap**, so it is not the
59
+ submitted system; it is listed only because merging is mathematically equivalent yet not
60
+ bit-identical in bf16 (`W + BA` rounded once, versus `Wx + B(Ax)`), which resolves about 1% of
61
+ near-tied questions differently.
62
+
63
+ ## Files
64
+ - `inference.py` – the exact script that produced our final Codabench submission
65
+ - `predictions.jsonl` – the exact file we submitted to the final leaderboard (macro 57.81),
66
+ byte-for-byte reproducible by running `inference.py`
67
+ - `predictions_ablation_base_v3b.jsonl` – an earlier submission from the ablation model
68
+ (`team-qbits-base-v3b`, macro 57.15), included for transparency. To regenerate it, set
69
+ `MODEL_ID` to `SinhalaMMLU-SharedTask/team-qbits-base-v3b` and change nothing else.
70
+ - `requirements.txt` – package versions we used
71
+ - `training/` – training, fine-tuning and data-preparation code
72
+ - `sft_gemma4_mcq.py` – LoRA SFT with answer-only loss and candidate-scoring evaluation
73
+ - `mcq_dataset.py` – extraction cleanup, validation, decontamination, splitting, prompt building
74
+ - `add_new_data.py`, `normalize_subjects.py` – dataset assembly and subject-name alignment
75
+ - `merge_adapter.py`, `push_merged_to_hf.py` – adapter merging and publishing
76
+ - `eval_sinhalammlu.py` – read-only analysis; also builds the decontamination blocklist
77
+ - `check_env.py` – dependency verification
78
+ - `run_data_pipeline.sh`, `run_training.sh`, `run_inference.sh` – end-to-end drivers
79
+ - `REPRODUCE.md` – exact commands, expected counts and the training curve
80
+
81
+ ## Data and decontamination
82
+ Source: publicly available Sri Lankan examination papers (`pastpapers.wiki`,
83
+ `e-thaksalawa.moe.gov.lk`), O/L and Grades 6–11. We do not redistribute the source PDFs; the derived
84
+ question set is published at `team-qbits-mcq-data`.
85
+
86
+ The released `TEST/easy` split was used **only** as a decontamination blocklist. It matched and
87
+ removed **26** questions from our training data. It was never used for training, checkpoint
88
+ selection, prompt design or hyperparameter choice. Model selection used a held-out split of our own
89
+ data, grouped so that no source paper appears in both training and validation.
90
+
91
+ ## Citation
92
+ This work evaluates on the SinhalaMMLU benchmark:
93
+
94
+ > Ashmari Pramodya, Nirasha Nelki, Heshan Shalinda, Chamila Liyanage, Yusuke Sakai, Randil
95
+ > Pushpananda, Ruvan Weerasinghe, Hidetaka Kamigaito, and Taro Watanabe. 2025. **SinhalaMMLU: A
96
+ > Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala.** In
97
+ > *Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages
98
+ > 32943–32961. https://aclanthology.org/2025.emnlp-main.1673/