Instructions to use SinhalaMMLU-SharedTask/team-sinbotics with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use SinhalaMMLU-SharedTask/team-sinbotics with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -22,7 +22,7 @@ tags:
|
|
| 22 |
- [Global-MMLU](https://huggingface.co/datasets/CohereLabs/Global-MMLU), Sinhala subset `si` (Apache-2.0), revision `0e619dbeb34206cd48705a1a0ea7fb21cae09993`: test and dev splits, 14,324 questions after removing 3 with an empty option.
|
| 23 |
- **sinmulu_si**: our own collection of 15,190 Sri Lankan Sinhala multiple-choice questions, gathered by our team; 15,180 after removing duplicates. Before training, it was checked against the 2,238 shared-task test questions, and 165 overlapping questions were removed (see *Overlap check against the test set*). We plan to publish it separately, so it is **not included** in this repo; it is available to the organisers on request.
|
| 24 |
- Together: 26,675 training and 2,829 validation questions (10% per subject, seed 42); see `training/README.md`.
|
| 25 |
-
- **How sinmulu_si was digitised:** the exam papers were scanned PDFs in legacy fonts, so their pages were transcribed to text with
|
| 26 |
- **At inference,** only our open-weight system above is used (≤ 8B parameters), with no external LLM, API or retrieval.
|
| 27 |
|
| 28 |
## Models and resources in this submission
|
|
@@ -51,7 +51,7 @@ Validation sets are 10% per subject (seed 42).
|
|
| 51 |
**sinmulu_si** (`training/sinmmlu/sinmulu.py`)
|
| 52 |
- **Source:** built by our team from 3,486 Sri Lankan exam-paper PDFs: G.C.E. O/L and A/L papers and provincial term tests.
|
| 53 |
- **Transcription:**
|
| 54 |
-
- Most PDFs use legacy Sinhala fonts, so text extraction was unusable. Instead, each page was rendered as an image and transcribed by
|
| 55 |
- This ran in batches, with several agents working in parallel on separate slices.
|
| 56 |
- Stems and options were copied verbatim.
|
| 57 |
- **Selection:**
|
|
@@ -88,7 +88,7 @@ Real exam papers can repeat questions, so before training we checked sinmulu_si
|
|
| 88 |
|
| 89 |
The cleaning therefore relies on manual judgement, not on a similarity cut-off, and it **does not prove zero leakage**. The check used only the test questions and choices, no answers.
|
| 90 |
|
| 91 |
-
All training data use the competition format: `q_no`, `subject`, `question`, `choices` (4 options), `answer` ("1"–"4"). Apart from the
|
| 92 |
|
| 93 |
## System in detail
|
| 94 |
|
|
|
|
| 22 |
- [Global-MMLU](https://huggingface.co/datasets/CohereLabs/Global-MMLU), Sinhala subset `si` (Apache-2.0), revision `0e619dbeb34206cd48705a1a0ea7fb21cae09993`: test and dev splits, 14,324 questions after removing 3 with an empty option.
|
| 23 |
- **sinmulu_si**: our own collection of 15,190 Sri Lankan Sinhala multiple-choice questions, gathered by our team; 15,180 after removing duplicates. Before training, it was checked against the 2,238 shared-task test questions, and 165 overlapping questions were removed (see *Overlap check against the test set*). We plan to publish it separately, so it is **not included** in this repo; it is available to the organisers on request.
|
| 24 |
- Together: 26,675 training and 2,829 validation questions (10% per subject, seed 42); see `training/README.md`.
|
| 25 |
+
- **How sinmulu_si was digitised:** the exam papers were scanned PDFs in legacy fonts, so their pages were transcribed to text with **Claude Sonnet 5.5** (Anthropic, closed API; used only to create training data, never at inference). Where no official marking scheme was available, the answer was labelled by the model and kept only when it was highly confident (see Datasets).
|
| 26 |
- **At inference,** only our open-weight system above is used (≤ 8B parameters), with no external LLM, API or retrieval.
|
| 27 |
|
| 28 |
## Models and resources in this submission
|
|
|
|
| 51 |
**sinmulu_si** (`training/sinmmlu/sinmulu.py`)
|
| 52 |
- **Source:** built by our team from 3,486 Sri Lankan exam-paper PDFs: G.C.E. O/L and A/L papers and provincial term tests.
|
| 53 |
- **Transcription:**
|
| 54 |
+
- Most PDFs use legacy Sinhala fonts, so text extraction was unusable. Instead, each page was rendered as an image and transcribed by **Claude Sonnet 5.5** (Anthropic API).
|
| 55 |
- This ran in batches, with several agents working in parallel on separate slices.
|
| 56 |
- Stems and options were copied verbatim.
|
| 57 |
- **Selection:**
|
|
|
|
| 88 |
|
| 89 |
The cleaning therefore relies on manual judgement, not on a similarity cut-off, and it **does not prove zero leakage**. The check used only the test questions and choices, no answers.
|
| 90 |
|
| 91 |
+
All training data use the competition format: `q_no`, `subject`, `question`, `choices` (4 options), `answer` ("1"–"4"). Apart from the Claude Sonnet 5.5 transcription and model-derived answers in sinmulu_si described above (closed API, training data only), we used no synthetic data and no retrieval corpus.
|
| 92 |
|
| 93 |
## System in detail
|
| 94 |
|