Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,436 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
language:
|
| 4 |
+
- si
|
| 5 |
+
---
|
| 6 |
+
# Team Cognilex — Sinhala MMLU Shared Task System
|
| 7 |
+
|
| 8 |
+
This repository documents Team Cognilex's system for the **ICTER Sinhala MMLU Shared Task 2026**, conducted by the University of Colombo School of Computing (UCSC) and the Informatics Institute of Technology (IIT), Sri Lanka.
|
| 9 |
+
|
| 10 |
+
Our approach adapts Qwen3.5-4B to Sinhala curriculum content through synthetic textbook question answering, then specializes the adapted model for multiple-choice question answering using a separate LoRA adapter. At inference time, the system generates the answer option number using greedy decoding.
|
| 11 |
+
|
| 12 |
+
## System summary
|
| 13 |
+
|
| 14 |
+
| Item | Description |
|
| 15 |
+
|---|---|
|
| 16 |
+
| Team | Cognilex |
|
| 17 |
+
| Base model family | Qwen3.5-4B |
|
| 18 |
+
| Curriculum adaptation | LoRA fine-tuning on 100,370 synthetic direct Q&A pairs from Grade 6–11 textbooks |
|
| 19 |
+
| MCQ specialization | A new LoRA adapter trained on the merged curriculum-adapted model |
|
| 20 |
+
| Initial cleaned MCQ training set | 46,373 records |
|
| 21 |
+
| Expanded MCQ training set for the fourth epoch | 47,025 records |
|
| 22 |
+
| Final inference system | Merged synthetic-QA model with the final MCQ LoRA adapter loaded |
|
| 23 |
+
| Output | An MCQ answer number in the range 1–5 |
|
| 24 |
+
| Decoding | Greedy; maximum of 8 generated tokens |
|
| 25 |
+
| Total inference parameters | 4,571,730,432 (approximately 4.572B) |
|
| 26 |
+
| Training framework | Unsloth |
|
| 27 |
+
| Hardware | Google Colab, NVIDIA A100 80 GB, as reported in the experiment notes |
|
| 28 |
+
| Hidden-test inference time | Approximately 13–15 minutes |
|
| 29 |
+
|
| 30 |
+
The reported total parameter count is below the template's limit of 8B inference parameters. The 1–5 output range supports papers with different numbers of options; it does not imply that every training question has five choices.
|
| 31 |
+
|
| 32 |
+
## Models and resources
|
| 33 |
+
|
| 34 |
+
| Resource | Purpose |
|
| 35 |
+
|---|---|
|
| 36 |
+
| [Synthetic-QA instruction model](https://huggingface.co/SinhalaMMLU-SharedTask/team-cognilex-qwen35-4b-synthetic-qa-instruct) | Qwen3.5-4B after curriculum adaptation and merging of the synthetic-QA adapter; the base for subsequent MCQ training |
|
| 37 |
+
| [MCQ LoRA adapter](https://huggingface.co/SinhalaMMLU-SharedTask/team-cognilex-qwen35-4b-mcq-lora) | Final MCQ specialization adapter, intended to be loaded on the synthetic-QA merged model |
|
| 38 |
+
| [SinhalaMMLU benchmark paper](https://aclanthology.org/2025.emnlp-main.1673/) | Benchmark reference and citation |
|
| 39 |
+
|
| 40 |
+
The MCQ adapter depends on the curriculum-adapted merged model above. Replacing that model with the original Qwen3.5-4B checkpoint would change the submitted system.
|
| 41 |
+
|
| 42 |
+
## Approach
|
| 43 |
+
|
| 44 |
+
The system was developed in two sequential training stages, with textbook processing and MCQ collection undertaken in parallel:
|
| 45 |
+
|
| 46 |
+
1. Collect Grade 6–11 textbooks and extract their text.
|
| 47 |
+
2. Chunk the text and generate factual direct Q&A pairs using a teacher model.
|
| 48 |
+
3. Adapt Qwen3.5-4B to the synthetic Q&A corpus with LoRA, then merge the adapter into the model.
|
| 49 |
+
4. Collect examination MCQs and their answers, check samples, and remove duplicates and exact question matches with the hidden set.
|
| 50 |
+
5. Initialize a new MCQ LoRA adapter on the merged model and train for three epochs.
|
| 51 |
+
6. Expand the MCQ dataset with additional easy questions and train the adapter for a fourth epoch at a lower learning rate.
|
| 52 |
+
7. Run the merged curriculum-adapted model with the final MCQ adapter using greedy answer-number generation.
|
| 53 |
+
|
| 54 |
+
The purpose of the first stage was to improve familiarity with Sinhala syllabus content before training the model to select MCQ answer numbers. The experiment did not supply an ablation demonstrating the independent contribution of each stage.
|
| 55 |
+
|
| 56 |
+
## 1. Textbook-based synthetic Q&A dataset
|
| 57 |
+
|
| 58 |
+
### Source and text extraction
|
| 59 |
+
|
| 60 |
+
Textbooks were collected from the [Educational Publications Department](http://www.edupub.gov.lk/) for **Grades 6–11**, covering six grade levels.
|
| 61 |
+
|
| 62 |
+
A Codex agent integrated with Playwright MCP was used to scrape and download textbook PDFs. Text was extracted with `pypdf`, with Sinhala and Tamil Unicode/font support in the extraction workflow. The extracted textbook text was used to construct the curriculum-adaptation corpus.
|
| 63 |
+
|
| 64 |
+
The reference to Sinhala and Tamil support describes the extraction workflow; a separate language-by-language breakdown of the final Q&A corpus is not available in the reported statistics.
|
| 65 |
+
|
| 66 |
+
### Chunking and synthetic generation
|
| 67 |
+
|
| 68 |
+
Textbooks were split into overlapping character-based chunks:
|
| 69 |
+
|
| 70 |
+
```python
|
| 71 |
+
CHUNK_CHARS = 6000
|
| 72 |
+
OVERLAP_CHARS = 300
|
| 73 |
+
```
|
| 74 |
+
|
| 75 |
+
Each chunk was submitted to **`deepseek-v4-flash`**, described in the experiment notes as a **284B-parameter teacher model**, through the **Embercloud** API platform. The generation request allowed a maximum of **15,000 output tokens** per chunk.
|
| 76 |
+
|
| 77 |
+
The prompt requested as many factual direct question–answer pairs as possible from each chunk and instructed the model to exclude image-based questions. These were direct Q&A pairs rather than the examination MCQs used in the second stage. The exact generation prompt and additional sampling parameters are not included in the available experiment description.
|
| 78 |
+
|
| 79 |
+
An **SQLite database** tracked the synthetic-Q&A generation process and its records.
|
| 80 |
+
|
| 81 |
+
### Dataset statistics
|
| 82 |
+
|
| 83 |
+
| Statistic | Value |
|
| 84 |
+
|---|---:|
|
| 85 |
+
| Grade levels processed | 6 (Grades 6–11) |
|
| 86 |
+
| Subject JSON files | 94 |
|
| 87 |
+
| Total direct Q&A pairs | 100,370 |
|
| 88 |
+
| Initial training split | 97.5% |
|
| 89 |
+
| Initial validation split | 2.5% |
|
| 90 |
+
|
| 91 |
+
The 94 files are subject JSON outputs, not a claim of 94 unique subjects. Exact split counts and the split seed were not reported, so only the supplied split proportions are listed here.
|
| 92 |
+
|
| 93 |
+
## 2. Curriculum adaptation with synthetic Q&A
|
| 94 |
+
|
| 95 |
+
### Initial training and early stopping
|
| 96 |
+
|
| 97 |
+
Qwen3.5-4B was fine-tuned with **LoRA** using **Unsloth**. The initial run was configured for two epochs with a maximum sequence length of **1,024 tokens**, using the 97.5% training and 2.5% validation split.
|
| 98 |
+
|
| 99 |
+
Early stopping used:
|
| 100 |
+
|
| 101 |
+
```python
|
| 102 |
+
early_stopping_patience = 4
|
| 103 |
+
early_stopping_threshold = 0.001
|
| 104 |
+
```
|
| 105 |
+
|
| 106 |
+
Early stopping occurred at approximately **11,000 steps**. The planned two epochs should therefore be distinguished from the actual early-stopped run. With `load_best_model_at_end=True`, checkpoint selection was configured to prefer the lowest validation loss.
|
| 107 |
+
|
| 108 |
+
### Initial training configuration
|
| 109 |
+
|
| 110 |
+
The following summarizes the reported trainer configuration; it is a configuration reference rather than a complete executable training script.
|
| 111 |
+
|
| 112 |
+
```python
|
| 113 |
+
output_dir = str(checkpoint_dir)
|
| 114 |
+
num_train_epochs = 2
|
| 115 |
+
per_device_train_batch_size = 1
|
| 116 |
+
per_device_eval_batch_size = 1
|
| 117 |
+
gradient_accumulation_steps = 16 # Effective batch size: 16 on one GPU.
|
| 118 |
+
learning_rate = 2e-5
|
| 119 |
+
warmup_ratio = 0.03
|
| 120 |
+
lr_scheduler_type = "cosine"
|
| 121 |
+
optim = "adamw_8bit"
|
| 122 |
+
weight_decay = 0.01
|
| 123 |
+
max_grad_norm = 1.0
|
| 124 |
+
bf16 = is_bfloat16_supported()
|
| 125 |
+
fp16 = not is_bfloat16_supported()
|
| 126 |
+
logging_strategy = "steps"
|
| 127 |
+
logging_steps = 25
|
| 128 |
+
logging_first_step = True
|
| 129 |
+
report_to = "tensorboard"
|
| 130 |
+
eval_strategy = "steps"
|
| 131 |
+
eval_steps = 250
|
| 132 |
+
save_strategy = "steps"
|
| 133 |
+
save_steps = 250
|
| 134 |
+
save_total_limit = 4
|
| 135 |
+
load_best_model_at_end = True
|
| 136 |
+
metric_for_best_model = "eval_loss"
|
| 137 |
+
greater_is_better = False
|
| 138 |
+
max_length = 1024
|
| 139 |
+
packing = False
|
| 140 |
+
completion_only_loss = True
|
| 141 |
+
dataset_num_proc = 1
|
| 142 |
+
dataset_kwargs = {"add_special_tokens": False}
|
| 143 |
+
gradient_checkpointing = True
|
| 144 |
+
```
|
| 145 |
+
|
| 146 |
+
The synthetic-QA LoRA configuration was:
|
| 147 |
+
|
| 148 |
+
```python
|
| 149 |
+
r = 16
|
| 150 |
+
lora_alpha = 16
|
| 151 |
+
lora_dropout = 0.05
|
| 152 |
+
bias = "none"
|
| 153 |
+
target_modules = [
|
| 154 |
+
"q_proj", "k_proj", "v_proj", "o_proj",
|
| 155 |
+
"gate_proj", "up_proj", "down_proj",
|
| 156 |
+
]
|
| 157 |
+
use_gradient_checkpointing = "unsloth"
|
| 158 |
+
```
|
| 159 |
+
|
| 160 |
+
### Continued training to a total target of three epochs
|
| 161 |
+
|
| 162 |
+
After early stopping, training continued on the **training partition only**, without a validation set, with a target of **three total epochs** and a reduced learning rate of **`1e-5`**. This was a continuation toward the total epoch target, not three additional epochs after early stopping.
|
| 163 |
+
|
| 164 |
+
The lower learning rate was chosen with the intention of reducing catastrophic forgetting. No separate forgetting evaluation was supplied, so this is the motivation for the setting rather than a measured outcome.
|
| 165 |
+
|
| 166 |
+
```python
|
| 167 |
+
TARGET_TOTAL_EPOCHS = 3
|
| 168 |
+
NEW_LEARNING_RATE = 1e-5
|
| 169 |
+
MICRO_BATCH = 1
|
| 170 |
+
GRAD_ACCUMULATION = 16
|
| 171 |
+
|
| 172 |
+
warmup_ratio = 0.01
|
| 173 |
+
lr_scheduler_type = "cosine"
|
| 174 |
+
optim = "adamw_8bit"
|
| 175 |
+
weight_decay = 0.01
|
| 176 |
+
max_grad_norm = 1.0
|
| 177 |
+
```
|
| 178 |
+
|
| 179 |
+
The LoRA rank and other adapter settings remained the same. Because this continuation did not use validation, the final continued model was not selected through the earlier validation-based early-stopping procedure.
|
| 180 |
+
|
| 181 |
+
### Merging the curriculum adapter
|
| 182 |
+
|
| 183 |
+
After the continued training, the synthetic-QA adapter was merged with its base model using:
|
| 184 |
+
|
| 185 |
+
```python
|
| 186 |
+
trainer.model.save_pretrained_merged(
|
| 187 |
+
str(merged_dir),
|
| 188 |
+
tokenizer,
|
| 189 |
+
save_method="merged_16bit",
|
| 190 |
+
)
|
| 191 |
+
```
|
| 192 |
+
|
| 193 |
+
This merged model became the starting point for the separate MCQ adapter. The published synthetic-QA model is linked in the resources table above.
|
| 194 |
+
|
| 195 |
+
## 3. Examination MCQ dataset
|
| 196 |
+
|
| 197 |
+
### Sources and coverage
|
| 198 |
+
|
| 199 |
+
In parallel with textbook processing, Sri Lankan examination papers were collected using Codex integrated with Playwright MCP servers. Collection covered available papers in the **1990–2025** range, including:
|
| 200 |
+
|
| 201 |
+
- G.C.E. Advanced Level past papers and Pesuru papers.
|
| 202 |
+
- G.C.E. Ordinary Level papers.
|
| 203 |
+
- Provincial examination papers.
|
| 204 |
+
- Grade 6–9 school and provincial papers.
|
| 205 |
+
- General knowledge papers.
|
| 206 |
+
|
| 207 |
+
Coverage depended on the material available from the source platforms; this does not imply complete coverage of every year, grade, or subject.
|
| 208 |
+
|
| 209 |
+
Sources included:
|
| 210 |
+
|
| 211 |
+
| Platform | Source link |
|
| 212 |
+
|---|---|
|
| 213 |
+
| Alapi Edu | [Past-paper hub](https://alapiedu.com/Category/paper_hub/PaperSubject?subject=&search=&paper_type=Past+Paper) |
|
| 214 |
+
| AlevelApi | [alevelapi.com](https://www.alevelapi.com/) |
|
| 215 |
+
| Department of Examinations | [Evaluation reports](https://www.doenets.lk/evaluationreports) |
|
| 216 |
+
| PaperBook | [paperbook.lk](https://paperbook.lk/) |
|
| 217 |
+
| North Western Provincial education website | [Downloads](https://nwpedu.lk/pages/downloads.php) |
|
| 218 |
+
| e-Thaksalawa | [Learning content management system](https://e-thaksalawa.moe.gov.lk/lcms/) |
|
| 219 |
+
| Past Papers Wiki | [pastpapers.wiki](https://pastpapers.wiki/) |
|
| 220 |
+
| GovDoc | [G.C.E. O/L model papers](https://govdoc.lk/category/model-papers/gce-ordinary-level-exam) |
|
| 221 |
+
| e-Kalvi | [Sinhala-medium O/L papers](https://e-kalvi.com/category/sinhala-medium/o-l/) |
|
| 222 |
+
|
| 223 |
+
### MCQ extraction and sample checks
|
| 224 |
+
|
| 225 |
+
Downloaded papers and their corresponding answers were uploaded to ChatGPT for question and answer extraction. The team used **four ChatGPT Plus accounts**, with **GPT-6.1-sol in medium reasoning mode**, as reported in the collection notes.
|
| 226 |
+
|
| 227 |
+
Extracted samples were manually checked to assess whether questions, answer choices, and correct answers were mapped appropriately. This was sample-based checking; the available description does not establish exhaustive manual verification of every record or a measured extraction-error rate.
|
| 228 |
+
|
| 229 |
+
### Deduplication and hidden-set exact-match filtering
|
| 230 |
+
|
| 231 |
+
The initial collected dataset contained **more than 47,000 records**. Before MCQ fine-tuning, duplicate questions and questions exactly matching the hidden set were removed. The initial cleaned training dataset contained **46,373 records**.
|
| 232 |
+
|
| 233 |
+
This filtering is disclosed for transparency. The exact normalization and matching implementation, number of hidden-set matches removed, and handling of paraphrases were not reported. Exact-match removal alone does not establish the absence of semantic overlap or shared source material. No claim is made here about hidden answer-label use; the reported operation was question matching and removal.
|
| 234 |
+
|
| 235 |
+
Before the fourth MCQ epoch, additional easy-level questions were collected because easy questions were underrepresented relative to medium and hard questions. After duplicate and exact-match filtering, the expanded training set contained **47,025 records**, a net increase of **652** over the initial cleaned set.
|
| 236 |
+
|
| 237 |
+
| Dataset stage | Records | Role |
|
| 238 |
+
|---|---:|---|
|
| 239 |
+
| Initial collected MCQs | More than 47,000 | Raw collected dataset |
|
| 240 |
+
| Cleaned MCQs | 46,373 | First three MCQ training epochs |
|
| 241 |
+
| Expanded and cleaned MCQs | 47,025 | Fourth MCQ training epoch |
|
| 242 |
+
|
| 243 |
+
Exact counts by difficulty and subject were not supplied. The final dataset expansion means that the fourth epoch used a different training corpus from the first three epochs.
|
| 244 |
+
|
| 245 |
+
## 4. MCQ adapter training
|
| 246 |
+
|
| 247 |
+
### First three epochs
|
| 248 |
+
|
| 249 |
+
A **new LoRA adapter** was initialized on top of the merged synthetic-QA model. This kept the MCQ specialization separate from the adapter already merged during curriculum adaptation.
|
| 250 |
+
|
| 251 |
+
| Setting | Value |
|
| 252 |
+
|---|---|
|
| 253 |
+
| Training records | 46,373 |
|
| 254 |
+
| Context length | 2,048 tokens |
|
| 255 |
+
| Initial training duration | 3 epochs |
|
| 256 |
+
| Learning rate | `2e-5` |
|
| 257 |
+
| LoRA rank | `16` |
|
| 258 |
+
| LoRA alpha | `32` |
|
| 259 |
+
| LoRA dropout | `0.5` |
|
| 260 |
+
| Bias | `none` |
|
| 261 |
+
| Vision layers | Not fine-tuned |
|
| 262 |
+
| Language layers | Fine-tuned |
|
| 263 |
+
| Attention modules | Fine-tuned |
|
| 264 |
+
| MLP modules | Fine-tuned |
|
| 265 |
+
| Warmup ratio | `0.05` |
|
| 266 |
+
| Learning-rate scheduler | `cosine` |
|
| 267 |
+
| Optimizer | `adamw_8bit` |
|
| 268 |
+
| Weight decay | `0.01` |
|
| 269 |
+
| Maximum gradient norm | `1.0` |
|
| 270 |
+
|
| 271 |
+
Reported adapter settings:
|
| 272 |
+
|
| 273 |
+
```python
|
| 274 |
+
r = 16
|
| 275 |
+
lora_alpha = 32
|
| 276 |
+
lora_dropout = 0.5
|
| 277 |
+
bias = "none"
|
| 278 |
+
finetune_vision_layers = False
|
| 279 |
+
finetune_language_layers = True
|
| 280 |
+
finetune_attention_modules = True
|
| 281 |
+
finetune_mlp_modules = True
|
| 282 |
+
```
|
| 283 |
+
|
| 284 |
+
The MCQ dropout of **`0.5`** is preserved as supplied; it differs from the synthetic-QA dropout of `0.05`. The repeated `warmup_ratio` entry in the experiment notes is represented once here.
|
| 285 |
+
|
| 286 |
+
### Fourth epoch on the expanded dataset
|
| 287 |
+
|
| 288 |
+
The MCQ adapter was fine-tuned for **one additional epoch**, giving four MCQ epochs in total, after expanding the cleaned dataset to **47,025 records**. The learning rate was reduced to **`1e-5`**; the other reported MCQ settings remained unchanged.
|
| 289 |
+
|
| 290 |
+
Only a small loss reduction was observed in the fourth epoch. No numeric loss values or benchmark comparison between the three-epoch and four-epoch adapters were supplied. The fourth-epoch adapter was used with the merged synthetic-QA model for final inference.
|
| 291 |
+
|
| 292 |
+
## 5. Inference
|
| 293 |
+
|
| 294 |
+
The final inference configuration combines:
|
| 295 |
+
|
| 296 |
+
1. The curriculum-adapted Qwen3.5-4B model with its synthetic-QA adapter already merged.
|
| 297 |
+
2. The final MCQ LoRA adapter after the fourth MCQ epoch.
|
| 298 |
+
|
| 299 |
+
The system predicts the correct option number, supporting labels **1–5**, using **greedy decoding** with a maximum of **8 new tokens**. An equivalent decoding configuration is:
|
| 300 |
+
|
| 301 |
+
```python
|
| 302 |
+
do_sample = False
|
| 303 |
+
num_beams = 1
|
| 304 |
+
max_new_tokens = 8
|
| 305 |
+
```
|
| 306 |
+
|
| 307 |
+
The intended output is the answer number. The exact inference prompt, chat template, output parsing, and handling of invalid or out-of-range outputs were not included in the supplied notes and must be taken from the final inference implementation.
|
| 308 |
+
|
| 309 |
+
### How to run
|
| 310 |
+
|
| 311 |
+
The required model resources are linked above. Load the synthetic-QA merged model and its tokenizer, attach the final MCQ adapter, format questions using the submission's exact prompt/template, and generate with the settings above.
|
| 312 |
+
|
| 313 |
+
**An exact executable command is pending the final inference script or notebook and pinned dependencies.** The template's example commands are not asserted to be working commands for this repository, because those files were not provided with this description.
|
| 314 |
+
|
| 315 |
+
| Reproduction item | Status |
|
| 316 |
+
|---|---|
|
| 317 |
+
| Merged model and MCQ adapter references | Provided above |
|
| 318 |
+
| Exact inference script or notebook | To be added |
|
| 319 |
+
| Pinned package versions / `requirements.txt` | To be added |
|
| 320 |
+
| Exact prompt and answer parser | To be added with inference code |
|
| 321 |
+
| Exact submitted prediction file | To be added |
|
| 322 |
+
| Model revision identifiers and random seeds | Not reported |
|
| 323 |
+
|
| 324 |
+
## 6. Parameter count
|
| 325 |
+
|
| 326 |
+
| Component | Parameters | Approximate size |
|
| 327 |
+
|---|---:|---:|
|
| 328 |
+
| Base / merged-model parameters | 4,539,265,536 | 4.539B |
|
| 329 |
+
| MCQ LoRA parameters | 32,464,896 | 32.46M |
|
| 330 |
+
| **Total inference parameters** | **4,571,730,432** | **4.572B** |
|
| 331 |
+
|
| 332 |
+
The total is the sum of the reported base and LoRA parameter counts. The synthetic-QA adapter is already merged into the model and is not counted again as an additional standalone adapter at inference time.
|
| 333 |
+
|
| 334 |
+
## 7. Hardware and runtime
|
| 335 |
+
|
| 336 |
+
Training and inference were reported as running on **Google Colab with an NVIDIA A100 80 GB GPU**.
|
| 337 |
+
|
| 338 |
+
| Stage | Reported elapsed time |
|
| 339 |
+
|---|---:|
|
| 340 |
+
| Synthetic Q&A: initial run to approximately 11,000 steps | 15:30:42 |
|
| 341 |
+
| Synthetic Q&A: continuation toward three total epochs | 02:30:57 |
|
| 342 |
+
| **Synthetic-QA training subtotal** | **18:01:39** |
|
| 343 |
+
| MCQ adapter: first three epochs | 15:09:58 |
|
| 344 |
+
| MCQ adapter: fourth epoch | 06:34:17 |
|
| 345 |
+
| **MCQ training subtotal** | **21:44:15** |
|
| 346 |
+
| **Total reported training time** | **39:45:54** |
|
| 347 |
+
| Hidden-test inference | Approximately 13–15 minutes |
|
| 348 |
+
|
| 349 |
+
Training durations use `HH:MM:SS`. Subtotals and the total are calculated from the four reported training durations. They do not include scraping, text extraction, teacher-model API generation, manual checks, or other preparation work. The hidden-test record count and inference batch size were not supplied, so per-question throughput is not reported.
|
| 350 |
+
|
| 351 |
+
## 8. Achieved results
|
| 352 |
+
|
| 353 |
+
### Shared-task evaluation
|
| 354 |
+
|
| 355 |
+
**Official benchmark scores and leaderboard placement have not yet been supplied for this README.** The table below reserves a separate place for the achieved evaluation results without inventing values.
|
| 356 |
+
|
| 357 |
+
| Metric | Achieved result |
|
| 358 |
+
|---|---|
|
| 359 |
+
| Official hidden-test overall score | Pending |
|
| 360 |
+
| Official leaderboard rank | Pending |
|
| 361 |
+
| Easy-level score | Pending |
|
| 362 |
+
| Medium-level score | Pending |
|
| 363 |
+
| Hard-level score | Pending |
|
| 364 |
+
| Per-category / per-subject scores | Pending |
|
| 365 |
+
| Submission identifier and evaluation date | Pending |
|
| 366 |
+
|
| 367 |
+
Report the organizers' official metric name and scale when completing this table. No improvement over the original model or other systems is claimed without comparative evaluation.
|
| 368 |
+
|
| 369 |
+
### Reported experimental outcomes
|
| 370 |
+
|
| 371 |
+
- Generated **100,370 synthetic direct Q&A pairs** across **94 subject JSON files** covering **six grade levels**.
|
| 372 |
+
- Constructed an initial cleaned MCQ training corpus of **46,373 records**, then expanded it to **47,025 records** for the fourth epoch.
|
| 373 |
+
- Produced a curriculum-adapted merged model and a separate MCQ LoRA adapter.
|
| 374 |
+
- Observed early stopping at approximately **11,000 steps** in the initial synthetic-QA run and a small loss reduction during the fourth MCQ epoch.
|
| 375 |
+
- Completed hidden-test inference in approximately **13–15 minutes** on the reported hardware, with **4.572B total inference parameters**.
|
| 376 |
+
|
| 377 |
+
Dataset size, training-loss observations, and runtime are experimental outcomes; they are not substitutes for held-out task accuracy.
|
| 378 |
+
|
| 379 |
+
## 9. Transparency and reproducibility notes
|
| 380 |
+
|
| 381 |
+
The methodology above records the settings supplied by the team. The following details remain necessary for exact reproduction:
|
| 382 |
+
|
| 383 |
+
- Exact original base-checkpoint identifier and revision, and revisions of the released artifacts.
|
| 384 |
+
- Package versions, CUDA/runtime environment, and random seeds.
|
| 385 |
+
- Text extraction/font-handling code, complete source manifests, and synthetic generation prompts and sampling settings.
|
| 386 |
+
- Data schemas, preprocessing rules, split implementation, and duplicate/exact-match filtering code.
|
| 387 |
+
- Checkpoint and optimizer/scheduler restoration details for continued training, including how prior progress was counted toward the three-total-epoch target.
|
| 388 |
+
- Full MCQ trainer configuration, including batch size, gradient accumulation, precision, validation policy, and checkpoint selection.
|
| 389 |
+
- Exact inference prompt, tokenizer/chat-template settings, parsing logic, submitted predictions, and official evaluation results.
|
| 390 |
+
|
| 391 |
+
Synthetic generation and assisted extraction can introduce factual or answer-mapping errors. Sample checks were performed for extracted MCQs, but exhaustive verification and quantitative error rates were not reported. Character chunk overlap can also create repeated or related content; the supplied description does not specify deduplication or source-grouped validation for the synthetic-QA corpus.
|
| 392 |
+
|
| 393 |
+
The lower learning rate during continued training and the additional fourth MCQ epoch reflect experimental choices. Their individual effects on task accuracy have not been established by the information supplied here.
|
| 394 |
+
|
| 395 |
+
## 10. Repository files
|
| 396 |
+
|
| 397 |
+
The final repository should identify the actual files used for submission. The following follows the organizers' suggested structure and is a completion checklist, not an assertion that these files are already present:
|
| 398 |
+
|
| 399 |
+
| Suggested artifact | Purpose | Status |
|
| 400 |
+
|---|---|---|
|
| 401 |
+
| `README.md` | System, data, training, inference, and results documentation | This document |
|
| 402 |
+
| `inference.py` or `inference.ipynb` | Exact code used to produce final predictions | To be added |
|
| 403 |
+
| `predictions.jsonl` or the required submission filename | Exact prediction artifact submitted for evaluation | To be added |
|
| 404 |
+
| `requirements.txt` | Pinned dependencies for reproduction | To be added |
|
| 405 |
+
| `training/` | Training and adapter-merging code/configurations | To be added |
|
| 406 |
+
| Data preparation code / manifests | Extraction, generation tracking, and filtering procedures | To be added |
|
| 407 |
+
|
| 408 |
+
## 11. Team members
|
| 409 |
+
|
| 410 |
+
- Lakshan Cooray
|
| 411 |
+
- Vihindi Kothalawala
|
| 412 |
+
- Binara Mendis
|
| 413 |
+
- Deshan Sumanathilaka
|
| 414 |
+
|
| 415 |
+
## 12. Acknowledgments
|
| 416 |
+
|
| 417 |
+
This work was carried out for the **ICTER Sinhala MMLU Shared Task 2026**, conducted by **UCSC and IIT, Sri Lanka**. We acknowledge the SinhalaMMLU benchmark authors and the educational institutions and platforms that made the source textbooks and examination resources available.
|
| 418 |
+
|
| 419 |
+
## 13. Citation
|
| 420 |
+
|
| 421 |
+
Please cite the SinhalaMMLU benchmark paper:
|
| 422 |
+
|
| 423 |
+
```bibtex
|
| 424 |
+
@inproceedings{pramodya-etal-2025-sinhalammlu,
|
| 425 |
+
title = {SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala},
|
| 426 |
+
author = {Pramodya, Ashmari and Nelki, Nirasha and
|
| 427 |
+
Shalinda, Heshan and Liyanage, Chamila and
|
| 428 |
+
Sakai, Yusuke and Pushpananda, Randil and
|
| 429 |
+
Weerasinghe, Ruvan and Kamigaito, Hidetaka and
|
| 430 |
+
Watanabe, Taro},
|
| 431 |
+
booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
|
| 432 |
+
year = {2025},
|
| 433 |
+
pages = {32943--32961},
|
| 434 |
+
url = {https://aclanthology.org/2025.emnlp-main.1673/}
|
| 435 |
+
}
|
| 436 |
+
```
|