---
license: mit
language:
- si
---
# Team Cognilex – Sinhala MMLU Shared Task System

## System summary

- **Base model:** Qwen3.5-4B.
- **What we did:** We first fine-tuned Qwen3.5-4B with LoRA on synthetic question–answer pairs generated from Sri Lankan Grade 6–11 textbooks to provide Sinhala syllabus context. We merged this adapter into the base model, then trained a new LoRA adapter on examination MCQs. Our final system uses the merged synthetic-QA model with the MCQ adapter and greedy decoding to predict the correct answer number.
- **Total inference parameters:** 4,571,730,432 (approximately 4.572B).
- **Extra data used for training:** 100,370 synthetic direct Q&A pairs generated from textbooks available at the [Educational Publications Department](http://www.edupub.gov.lk/), and MCQs extracted from Sri Lankan examination papers. The MCQ dataset contained 46,373 records for the first three epochs and 47,025 records for the fourth epoch after expansion and filtering.

This work was carried out for the **ICTER Sinhala MMLU Shared Task 2026**, conducted by **UCSC and IIT, Sri Lanka**.

## Team information

- **Team name:** Congnilex
- **CodeBench username:** lakshan

## System architecture

![Team Cognilex Sinhala MMLU training and inference architecture](./architecture.jpeg)

## Models and resources in this submission

| Repository | What it is |
|---|---|
| [SinhalaMMLU-SharedTask/team-cognilex-qwen35-4b-synthetic-qa-instruct](https://huggingface.co/SinhalaMMLU-SharedTask/team-cognilex-qwen35-4b-synthetic-qa-it) | Qwen3.5-4B fine-tuned on synthetic textbook Q&A, with the adapter merged into the model. This model is the base for MCQ adapter training and final inference. |
| [SinhalaMMLU-SharedTask/team-cognilex-qwen35-4b-mcq-lora](https://huggingface.co/SinhalaMMLU-SharedTask/team-cognilex-qwen35-4b-mcq-lora-it) | MCQ LoRA adapter trained on top of the merged synthetic-QA model. |

## Dataset creation

### Synthetic textbook Q&A dataset

#### Textbook collection and text extraction

We collected textbooks for **Grades 6–11** from the [Educational Publications Department](http://www.edupub.gov.lk/). We used a **Codex agent integrated with Playwright MCP** to scrape and download the textbook PDFs.

We converted the PDFs to text using **pypdf**, with Sinhala and Tamil Unicode/font support for text extraction. We then divided the extracted textbook text into overlapping chunks using:

```python
CHUNK_CHARS = 6000
OVERLAP_CHARS = 300
```

#### Synthetic Q&A generation

We sent each textbook chunk to **deepseek-v4-flash (284B)** through the **Embercloud API platform**, with a maximum output length of **15,000 tokens**. Our prompt asked the model to generate as many factual question–answer pairs as possible from each chunk while excluding image-based questions.

We used an **SQLite database** to maintain records of the synthetic Q&A generation process.

| Statistic | Value |
|---|---:|
| Grades processed | 6 (Grades 6–11) |
| Subject JSON files | 94 |
| Total Q&A pairs | 100,370 |

### Examination MCQ dataset

#### Paper collection

In parallel, we collected MCQ data from Sri Lankan online examination-paper platforms using **Codex integrated with Playwright MCP servers**. We downloaded available papers from **1990–2026**, including:

- A/L past papers and Pesuru papers.
- O/L papers.
- Provincial papers.
- Grade 6–9 school and provincial papers.
- General knowledge papers.

The websites used included:

| Source | Link |
|---|---|
| Alapi Edu | [Past-paper hub](https://alapiedu.com/Category/paper_hub/PaperSubject?subject=&search=&paper_type=Past+Paper) |
| AlevelApi | [alevelapi.com](https://www.alevelapi.com/) |
| Department of Examinations | [Evaluation reports](https://www.doenets.lk/evaluationreports) |
| PaperBook | [paperbook.lk](https://paperbook.lk/) |
| North Western Provincial Education | [Downloads](https://nwpedu.lk/pages/downloads.php) |
| e-Thaksalawa | [e-thaksalawa.moe.gov.lk/lcms](https://e-thaksalawa.moe.gov.lk/lcms/) |
| Past Papers Wiki | [pastpapers.wiki](https://pastpapers.wiki/) |
| GovDoc | [G.C.E. O/L model papers](https://govdoc.lk/category/model-papers/gce-ordinary-level-exam) |
| e-Kalvi | [Sinhala-medium O/L papers](https://e-kalvi.com/category/sinhala-medium/o-l/) |

#### MCQ extraction and checking

We uploaded the downloaded papers, along with their answers, to ChatGPT using \*\*four ChatGPT Plus accounts\*\* with \*\*GPT-6.1-Sol in medium reasoning mode\*\*. We also used \*\*one Claude Sonnet 5.5 account\*\* and \*\*one Gemini 3.8 Flash account\*\*. We manually checked samples of the extracted MCQs to verify that the questions and answers were mapped correctly.

#### Dataset filtering and expansion

The initial MCQ dataset contained **more than 47,000 records**. Before fine-tuning, we removed duplicate questions and questions that exactly matched the hidden set. We include this filtering step for transparency in our experimental process. The resulting dataset contained **46,373 records**.

Because easy-level MCQs were underrepresented compared with medium- and hard-level questions, we collected additional easy questions before the fourth training epoch. After removing duplicates and exact matches, the expanded dataset contained **47,025 records**.

| Dataset stage | Records |
|---|---:|
| Initial collected dataset | More than 47,000 |
| Filtered dataset for the first three MCQ epochs | 46,373 |
| Expanded, filtered dataset for the fourth MCQ epoch | 47,025 |

#### Subject distribution

Humanities (38.8%) and STEM (29.5%) account for the largest shares of the MCQ dataset.

![MCQ dataset distribution by subject](./subject-distribution.png)

#### Difficulty distribution

Medium questions comprise 52.9% of the MCQ dataset, followed by hard (32.8%) and easy (14.3%) questions.

![MCQ dataset distribution by difficulty](./difficulty-distribution.png)

## Training methodology

### Stage 1: Sinhala syllabus adaptation using synthetic Q&A

We fine-tuned **Qwen3.5-4B** on the synthetic textbook Q&A dataset using **LoRA** and **Unsloth**. This stage was designed to provide Sinhala syllabus context to the language model used in our experiments.

#### Initial training

We divided the dataset into **97.5% training** and **2.5% validation**. The initial run was configured for **two epochs**, with a maximum sequence length of **1,024 tokens**.

We used early stopping with:

```python
early_stopping_patience = 4
early_stopping_threshold = 0.001
```

Early stopping occurred at approximately **11,000 steps**.

The initial training configuration was:

```python
output_dir = str(checkpoint_dir)
num_train_epochs = 2
per_device_train_batch_size = 1
per_device_eval_batch_size = 1
gradient_accumulation_steps = 16  # Effective batch size of 16 on one GPU.
learning_rate = 2e-5
warmup_ratio = 0.03
lr_scheduler_type = "cosine"
optim = "adamw_8bit"
weight_decay = 0.01
max_grad_norm = 1.0
bf16 = is_bfloat16_supported()
fp16 = not is_bfloat16_supported()
logging_strategy = "steps"
logging_steps = 25
logging_first_step = True
report_to = "tensorboard"
eval_strategy = "steps"
eval_steps = 250
save_strategy = "steps"
save_steps = 250
save_total_limit = 4
load_best_model_at_end = True
metric_for_best_model = "eval_loss"
greater_is_better = False
max_length = 1024
packing = False
completion_only_loss = True
dataset_num_proc = 1
dataset_kwargs = {"add_special_tokens": False}
gradient_checkpointing = True
```

The LoRA configuration was:

```python
r = 16
lora_alpha = 16
lora_dropout = 0.05
bias = "none"
target_modules = [
    "q_proj", "k_proj", "v_proj", "o_proj",
    "gate_proj", "up_proj", "down_proj",
]
use_gradient_checkpointing = "unsloth"
```

#### Continued training after early stopping

After early stopping, we continued fine-tuning the adapter on the **training set only**, without a validation set, up to a target of **three total epochs**. We reduced the learning rate to **`1e-5`** to reduce catastrophic forgetting and retained the same LoRA rank and adapter settings.

The continuation settings were:

```python
TARGET_TOTAL_EPOCHS = 3
NEW_LEARNING_RATE = 1e-5
MICRO_BATCH = 1
GRAD_ACCUMULATION = 16
warmup_ratio = 0.01
lr_scheduler_type = "cosine"
optim = "adamw_8bit"
weight_decay = 0.01
max_grad_norm = 1.0
```

#### Merging the model and adapter

We merged the synthetic-QA adapter with the model using:

```python
trainer.model.save_pretrained_merged(
    str(merged_dir),
    tokenizer,
    save_method="merged_16bit",
)
```

We used this merged model as the base for training the MCQ adapter.

### Stage 2: MCQ adapter fine-tuning

#### First three epochs

We initialized a **new LoRA adapter** on top of the merged synthetic-QA model and fine-tuned it on **46,373 MCQ records** for **three epochs**, using a learning rate of **`2e-5`** and a context length of **2,048 tokens**.

The MCQ LoRA configuration was:

```python
r = 16
lora_alpha = 32
lora_dropout = 0.5
finetune_vision_layers = False
finetune_language_layers = True
finetune_attention_modules = True
finetune_mlp_modules = True
bias = "none"
```

The training settings included:

```python
num_train_epochs = 3
learning_rate = 2e-5
max_length = 2048
warmup_ratio = 0.05
lr_scheduler_type = "cosine"
optim = "adamw_8bit"
weight_decay = 0.01
max_grad_norm = 1.0
```

#### Fourth epoch

After the first three epochs, we fine-tuned the MCQ adapter for **one additional epoch** on the expanded dataset of **47,025 records**. We reduced the learning rate to **`1e-5`** and retained the other MCQ training settings.

We observed a small loss reduction during the fourth epoch. We used this final adapter together with the merged synthetic-QA model for inference.

### Training overview

| Stage | Dataset | Training duration | Learning rate | Context length |
|---|---|---|---|---:|
| Initial synthetic-QA adaptation | 100,370 Q&A pairs, split 97.5% / 2.5% | Configured for 2 epochs; early stopping at approximately 11,000 steps | `2e-5` | 1,024 |
| Continued synthetic-QA adaptation | Training partition only | Continued up to 3 total epochs | `1e-5` | 1,024 |
| Initial MCQ adapter training | 46,373 MCQs | 3 epochs | `2e-5` | 2,048 |
| Final MCQ adapter training | 47,025 MCQs | 1 additional epoch | `1e-5` | 2,048 |

## How to run

For inference, use the [merged synthetic-QA model](https://huggingface.co/SinhalaMMLU-SharedTask/team-cognilex-qwen35-4b-synthetic-qa-it) with the [MCQ LoRA adapter](https://huggingface.co/SinhalaMMLU-SharedTask/team-cognilex-qwen35-4b-mcq-lora-it).

Our model predicts the correct MCQ answer number in the range **1–5**. We use **greedy decoding** with a maximum of **8 output tokens**.

- **GPU used:** NVIDIA A100 80 GB on Google Colab.
- **Approximate runtime on the hidden test set:** 13–15 minutes.

## Parameter count of the system

| Component | Parameter count |
|---|---:|
| Base parameters | 4,539,265,536 (4.539B) |
| LoRA parameters | 32,464,896 (32.46M) |
| **Total parameters** | **4,571,730,432 (4.572B)** |

## Hardware and training time

We used **Google Colab with an NVIDIA A100 80 GB GPU** for training and inference.

| Training stage | Time (HH:MM:SS) |
|---|---:|
| Direct Q&A model: first 11,000 steps | 18:32:38 |
| Direct Q&A model: continuation up to the third epoch | 12:39:29 |
| MCQ adapter: first three epochs | 15:14:40 |
| MCQ adapter: fourth epoch | 06:34:17 |
| **Total training time** | **53:01:04** |


## Repository structure

- [data/](./data)
  - [All_Grades_direct_QA_dataset.json](./data/All_Grades_direct_QA_dataset.json)
  - [MCQ_dataset.json](./data/MCQ_dataset.json)
- [training/](./training)
  - [MCQ_preprocessing_and_visualization.ipynb](./training/MCQ_preprocessing_and_visualization.ipynb)
  - [McqAdapterTraining/](./training/McqAdapterTraining)
    - [MCQ_Qwen3_5_4B_ZeroShot_LoRA_Colab_v2.ipynb](./training/McqAdapterTraining/MCQ_Qwen3_5_4B_ZeroShot_LoRA_Colab_v2.ipynb)
    - [Qwen3_5_4B_MCQ_v2_Continue_3_to_4epochs_LR1e_5__v2 (1).ipynb](./training/McqAdapterTraining/Qwen3_5_4B_MCQ_v2_Continue_3_to_4epochs_LR1e_5__v2%20%281%29.ipynb)
  - [SyntheticDirectQATraining/](./training/SyntheticDirectQATraining)
    - [Qwen3_5_4B_Sinhala_FRESH_16bit_LoRA_Training (3).ipynb](./training/SyntheticDirectQATraining/Qwen3_5_4B_Sinhala_FRESH_16bit_LoRA_Training%20%283%29.ipynb)
    - [Qwen3_5_4B_Sinhala_Continue_From_11k_LowLR_3or4_Epochs (2).ipynb](./training/SyntheticDirectQATraining/Qwen3_5_4B_Sinhala_Continue_From_11k_LowLR_3or4_Epochs%20%282%29.ipynb)
- [inference.ipynb](./inference.ipynb)
- [predictions.jsonl](./predictions.jsonl)
- [architecture.jpeg](./architecture.jpeg)
- [subject-distribution.png](./subject-distribution.png)
- [difficulty-distribution.png](./difficulty-distribution.png)
- [README.md](./README.md)
- [.gitattributes](./.gitattributes)

  
## Achieved results

<!-- Add the official shared-task results below. -->

| Evaluation metric | Result |
|---|---|
| Overall score | 68.55 |
| Easy-level score | 61.21 |
| Medium-level score | 67.96 |
| Hard-level score | 76.49 |


## Team members

- Lakshan Cooray
- Vihindi Kothalawala
- Binara Mendis
- Deshan Sumanathilaka

## Citation

Please cite the SinhalaMMLU benchmark paper:

```bibtex
@inproceedings{pramodya-etal-2025-sinhalammlu,
  title     = {SinhalaMMLU: A Comprehensive Benchmark
               for Evaluating Multitask Language
               Understanding in Sinhala},
  author    = {Pramodya, Ashmari and Nelki, Nirasha and
               Shalinda, Heshan and Liyanage, Chamila and
               Sakai, Yusuke and Pushpananda, Randil and
               Weerasinghe, Ruvan and Kamigaito, Hidetaka
               and Watanabe, Taro},
  booktitle = {Proceedings of the 2025 Conference on
               Empirical Methods in Natural Language
               Processing},
  year      = {2025},
  pages     = {32943--32961},
  url       = {https://aclanthology.org/2025.emnlp-main.1673/}
}
```