Team Nexus: inference script, final predictions, code and README

README.md

@@ -0,0 +1,71 @@
+# Team Nexus – Sinhala MMLU Shared Task System
+
+## System summary
+- **Base model(s):** [`google/gemma-4-E4B-it`](https://huggingface.co/google/gemma-4-E4B-it), commit `ee0ef6023621cff504d758262d4e04895a5af4a2`, used **text-only**: the vision and audio towers are removed and every language-model weight is copied unchanged (`training/scripts/make_text_only.py` checks that the copy is exact).
+- **What we did:** prompting only, with no fine-tuning, retrieval or extra data at inference. Every question is answered in Gemma 4's thinking mode with greedy decoding and up to 3,072 new tokens. We also compute a fast one-step answer: the next-token scores over the labels 1–4, averaged over 4 cyclic orders of the options. Where the fast answer disagrees with the greedy thinking answer, we sample 3 more reasoning paths (temperature 0.8, top-p 0.95, top-k 64, fixed seed 1234) and majority-vote the 4 answers, with ties going to the greedy answer. We call this *agreement-gated self-consistency*. On the test set it was triggered for 892 of 2,238 questions and changed 127 answers.
+- **Total inference parameters:** **7.463B** (7,463,013,376; one model, no adapter). This is the count of the weights actually loaded, with tied embeddings counted once.
+- **Extra data used for training or retrieval:** none in the final system. See "Experiments" below for data we used during development.
+
+**Test leaderboard score (Codabench, submission ID 50):** macro average **62.93**, with Easy 63.40, Medium 55.09 and Hard 70.31.
+
+## Models and resources in this submission
+| Repo | What it is |
+|---|---|
+| [`SinhalaMMLU-SharedTask/team-nexus`](https://huggingface.co/SinhalaMMLU-SharedTask/team-nexus) | Code, inference script and final predictions (this repo) |
+| [`SinhalaMMLU-SharedTask/team-nexus-gemma-4-e4b-it-text`](https://huggingface.co/SinhalaMMLU-SharedTask/team-nexus-gemma-4-e4b-it-text) | Text-only Gemma 4 E4B-it: unchanged weights of `google/gemma-4-E4B-it` with the vision and audio towers removed (Apache 2.0). `inference.py` pins commit `8f686039aad1a865c9335c17874d9b870fdec766`. |
+
+## How to run
+```bash
+pip install -r requirements.txt
+python inference.py
+```
+`inference.py` follows the organisers' template. Section 1 loads every `.json` file in `DATA_REPO`. Section 2 downloads the model from the repo above and runs our pipeline (`inference/predict.py` with `configs/submission_base_think_gated_sc.yaml`), the exact code that produced our Codabench file. Section 3 writes `predictions.jsonl`.
+
+- **GPU used:** 1× NVIDIA DGX Spark (GB10 Grace Blackwell, 128 GB unified CPU/GPU memory), bf16.
+- **Approximate run time on the test set:** about **11.3 hours** (40,685 s for 2,238 questions) on the GB10. Most of this is thinking-mode generation; faster GPUs need proportionally less.
+- **Anything else the organizers need to know:**
+  - **Memory:** generation uses batches of up to 64 sequences, with a padded prompt budget of 16,384 tokens and up to 3,072 new tokens each. Plan for a GPU with **80 GB or more**, such as an A100 or H100 80 GB. A 16 GB T4 is not enough.
+  - **Sampling:** about 40% of questions use the self-consistency step, which *samples*. It is seeded with `torch.manual_seed(1234)` immediately before the sampled passes.
+  - **Reproducibility:** thinking outputs are long (hundreds to thousands of tokens) and generated in bf16 with batched, length-sorted, left-padded batches. The answers therefore depend on the hardware and on which questions share a batch. Two separate full runs on our machine are consistent with identical greedy answers. Our two final candidates share the greedy settings and differ on only 100 Easy/Medium answers, fewer than the 127 that self-consistency changed. Running a 30-question subset changes the batches, and 22 of 30 answers matched our submitted file. **Please run the full dataset with the default settings.** On a different GPU type, expect a minority of answers to differ, with a similar overall score.
+  - **Greedy variant:** setting `SELF_CONSISTENCY = False` at the top of `inference.py` gives a fully greedy variant with no sampling at all. That is **not** the system we submitted; we did not submit it, so it has no test score.
+  - **Gating:** the model repo is public and the base model is not gated, so no login is needed.
+
+## Files
+- `inference.py` – the entry point that runs the exact pipeline behind our final Codabench submission
+- `inference/predict.py`, `eval/` – the pipeline code it calls (prompting, scoring, thinking, self-consistency)
+- `configs/submission_base_think_gated_sc.yaml` – the settings of the submitted system; `configs/submission_base_think_greedy.yaml` – the greedy variant
+- `predictions.jsonl` – the exact file we submitted to the final leaderboard (sha256 `4bf9f86282bffd69dcbd7c178e969532e1d3cf0462abc437663986c7a8b77ce1`)
+- `requirements.txt` – package versions we used (Python 3.12.3)
+- `training/` – all our data-preparation, training and evaluation code and configurations from development (see below)
+
+## Experiments (development only; nothing below is used by the final system)
+`training/` contains the full project code. The final system uses only `training/scripts/make_text_only.py`, which builds the model repo above. During development we tried the following on held-out data, and none of it beat the base model with thinking:
+- LoRA and full fine-tuning on Global-MMLU `si`
+- continued pre-training on Sinhala Wikipedia
+- distillation from Gemma 4 31B soft labels
+- reasoning-trace fine-tuning on the model's own correct reasoning
+- option-order voting
+
+Each step was decided on SinhalaMMLU dev or Global-MMLU validation subsets, never on test questions. The public data involved were:
+- [CohereLabs/Global-MMLU](https://huggingface.co/datasets/CohereLabs/Global-MMLU) (`si`)
+- [Sinhala Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) (`20231101.si`)
+- [FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) (`sin_Sinh`)
+- [NSINA](https://huggingface.co/datasets/sinhala-nlp/NSINA)
+- SinhalaMMLU dev (`naist-nlp/SinhalaMMLU`; not redistributed, because of its CC BY-NC-ND licence)
+
+We also generated a small pilot set of synthetic MCQs from Sinhala Wikipedia passages with a locally run open model (Gemma 4 31B). No closed APIs were used at any stage.
+
+We checked that Global-MMLU does not overlap with SinhalaMMLU, using character n-gram and multilingual-embedding similarity. `training/DATA_LOG.md` and `training/MODEL_MANIFEST.md` give the details.
+
+## Citation
+```bibtex
+@inproceedings{pramodya-etal-2025-sinhalammlu,
+  title     = "{S}inhala{MMLU}: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in {S}inhala",
+  author    = "Pramodya, Ashmari and Nelki, Nirasha and Shalinda, Heshan and Liyanage, Chamila and Sakai, Yusuke and
+               Pushpananda, Randil and Weerasinghe, Ruvan and Kamigaito, Hidetaka and Watanabe, Taro",
+  booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
+  year      = "2025",
+  pages     = "32931--32949",
+  url       = "https://aclanthology.org/2025.emnlp-main.1673/"
+}
+```

configs/submission_base_think_gated_sc.yaml

@@ -0,0 +1,23 @@
+# Final candidate B (21 Oct): base Gemma 4 E4B, text-only (7.463B, no adapter), thinking for every question.
+# Agreement-gated self-consistency on all levels: where the one-step answer disagrees with greedy thinking, add 3 sampled
+# reasoning paths and take the majority vote (ties -> greedy). Evidence (dev/validation only): full dev, base thinking 63.0 vs
+# model A vote 57.2 (+5.8 [+3.6, +8.0]); pooled proxies, gated self-consistency +2.70 [+1.18, +4.23] on 43% of questions.
+model: models/gemma-4-E4B-it-text
+lang: en
+chat: true
+method: logit          # fast path, used only if a thinking answer cannot be read
+n_perms: 4
+perm_mode: cyclic
+seed: 1234
+dtype: bfloat16
+gpu_mem_gb: 100
+token_budget: 32768
+max_batch: 64
+thinking_levels: [easy, medium, hard]
+think_max_new_tokens: 3072
+sc_levels: [easy, medium, hard]
+sc_gate: disagree
+sc_samples: 3
+sc_temperature: 0.8
+sc_top_p: 0.95
+sc_top_k: 64

configs/submission_base_think_greedy.yaml

@@ -0,0 +1,23 @@
+# Fully greedy variant (NOT the submitted system): thinking for every question, no sampling, no self-consistency.
+# Final candidate B (21 Oct): base Gemma 4 E4B, text-only (7.463B, no adapter), thinking for every question.
+# Agreement-gated self-consistency on all levels: where the one-step answer disagrees with greedy thinking, add 3 sampled
+# reasoning paths and take the majority vote (ties -> greedy). Evidence (dev/validation only): full dev, base thinking 63.0 vs
+# model A vote 57.2 (+5.8 [+3.6, +8.0]); pooled proxies, gated self-consistency +2.70 [+1.18, +4.23] on 43% of questions.
+model: models/gemma-4-E4B-it-text
+lang: en
+chat: true
+method: logit          # fast path, used only if a thinking answer cannot be read
+n_perms: 4
+perm_mode: cyclic
+seed: 1234
+dtype: bfloat16
+gpu_mem_gb: 100
+token_budget: 32768
+max_batch: 64
+thinking_levels: [easy, medium, hard]
+think_max_new_tokens: 3072
+sc_levels: []
+sc_samples: 0
+sc_temperature: 0.8
+sc_top_p: 0.95
+sc_top_k: 64

eval/combine_cv.py

@@ -0,0 +1,48 @@
+"""Combine k-fold runs (each scored only on its held-out fold) into one run over the full dev set.
+
+  python -m eval.combine_cv --runs results/exp_devcv_lora_f* --name exp_devcv_lora_all
+
+Writes results/<name>/{metrics.json, report.md, predictions.jsonl} in the usual layout, so eval.summarize can put it
+next to the zero-shot baseline. Fails if a question is missing or scored twice.
+"""
+from __future__ import annotations
+
+import argparse
+import glob
+import json
+from pathlib import Path
+
+from .common import ROOT, read_jsonl, write_jsonl
+from .metrics import markdown_report
+from .run_eval import metrics_by_dataset
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--runs", nargs="+", required=True)
+    ap.add_argument("--name", required=True)
+    ap.add_argument("--data", default="data/processed/sinhalammlu_dev.jsonl")
+    args = ap.parse_args()
+    dirs = sorted(Path(d) for pat in args.runs for d in glob.glob(pat))
+    records, first = [], None
+    for d in dirs:
+        records += read_jsonl(d / "predictions.jsonl")
+        first = first or json.loads((d / "metrics.json").read_text())
+    items = {r["uid"]: r for r in read_jsonl(ROOT / args.data)}
+    scored = [r["uid"] for r in records if r["perm_k"] == 0]
+    assert len(scored) == len(set(scored)), "a question was scored by more than one fold"
+    assert set(scored) == set(items), f"{len(set(items) - set(scored))} dev questions were not scored"
+    out = ROOT / "results" / args.name
+    out.mkdir(parents=True, exist_ok=True)
+    metrics = {"by_dataset": metrics_by_dataset(items, records),
+               "run": first["run"] | {"run_name": args.name, "combined_from": [d.name for d in dirs]}}
+    write_jsonl(out / "predictions.jsonl", records)
+    (out / "metrics.json").write_text(json.dumps(metrics, indent=2, ensure_ascii=False))
+    (out / "report.md").write_text("\n".join(markdown_report(f"{args.name} / {k}", m)
+                                             for k, m in metrics["by_dataset"].items()))
+    o = metrics["by_dataset"]["sinhalammlu"]["overall"]
+    print(f"{args.name}: dev acc={o['acc']:.4f} vote={o['acc_vote']:.4f} over {o['n']} questions from {len(dirs)} folds")
+
+
+if __name__ == "__main__":
+    main()

eval/common.py

@@ -0,0 +1,115 @@
+"""Shared helpers: secrets, seeding, JSONL I/O, environment fingerprint."""
+from __future__ import annotations
+
+import json
+import os
+import platform
+import random
+import subprocess
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def ensure_python_headers() -> None:
+    """Triton (torch.compile, Liger, flash-linear-attention) compiles helpers against Python.h. This machine has
+    no python3.12-dev and no sudo, so the headers are unpacked to ~/.local/pydev (README, setup). Point gcc at
+    them when the system copy is missing; a no-op on machines that have the headers."""
+    import sysconfig
+
+    if (Path(sysconfig.get_paths()["include"]) / "Python.h").exists():
+        return
+    local = Path.home() / ".local" / "pydev" / "root" / "usr" / "include"
+    if (local / f"python{sys.version_info.major}.{sys.version_info.minor}" / "Python.h").exists():
+        inc = [str(local / f"python{sys.version_info.major}.{sys.version_info.minor}"), str(local)]
+        os.environ["CPATH"] = os.pathsep.join(inc + [p for p in os.environ.get("CPATH", "").split(os.pathsep) if p])
+
+
+def load_env(path: Path = ROOT / ".env") -> None:
+    """Read KEY=VALUE lines from .env; map hf_token -> HF_TOKEN for huggingface_hub."""
+    ensure_python_headers()
+    if not path.exists():
+        return
+    for line in path.read_text().splitlines():
+        line = line.strip()
+        if not line or line.startswith("#") or "=" not in line:
+            continue
+        key, val = (s.strip().strip('"').strip("'") for s in line.split("=", 1))
+        if key.lower() in ("hf_token", "hugging_face_hub_token"):
+            key = "HF_TOKEN"
+        os.environ.setdefault(key, val)
+
+
+def set_seed(seed: int) -> None:
+    random.seed(seed)
+    os.environ["PYTHONHASHSEED"] = str(seed)
+    try:
+        import numpy as np
+
+        np.random.seed(seed)
+    except ImportError:
+        pass
+    try:
+        import torch
+
+        torch.manual_seed(seed)
+        torch.cuda.manual_seed_all(seed)
+        torch.backends.cuda.matmul.allow_tf32 = False
+        torch.backends.cudnn.allow_tf32 = False
+    except ImportError:
+        pass
+
+
+def cap_gpu_memory(gb: float | None = None) -> None:
+    """Limit PyTorch's CUDA allocator (default 100 GB, env SMMLU_GPU_MEM_GB).
+
+    The DGX Spark shares one 128 GB pool between CPU and GPU: an unbounded GPU job does not get a CUDA
+    out-of-memory error, it starves the system and the kernel OOM killer takes down the desktop session and
+    tmux (10 Oct, 11:15). With the cap, the job fails with torch.OutOfMemoryError and everything else survives.
+    """
+    import torch
+
+    if not torch.cuda.is_available():
+        return
+    gb = float(gb or os.environ.get("SMMLU_GPU_MEM_GB", 100))
+    total = torch.cuda.get_device_properties(0).total_memory
+    torch.cuda.set_per_process_memory_fraction(min(gb * 1024**3 / total, 1.0), 0)
+
+
+def read_jsonl(path: str | Path) -> list[dict]:
+    with open(path, encoding="utf-8") as f:
+        return [json.loads(line) for line in f if line.strip()]
+
+
+def write_jsonl(path: str | Path, rows) -> None:
+    Path(path).parent.mkdir(parents=True, exist_ok=True)
+    with open(path, "w", encoding="utf-8") as f:
+        for r in rows:
+            f.write(json.dumps(r, ensure_ascii=False) + "\n")
+
+
+def env_fingerprint() -> dict:
+    """Versions and code revision, stored with every run for reproducibility."""
+    info = {"python": platform.python_version(), "machine": platform.machine()}
+    for mod in ("torch", "transformers", "datasets", "peft", "accelerate", "huggingface_hub"):
+        try:
+            info[mod] = __import__(mod).__version__
+        except ImportError:
+            pass
+    try:
+        import torch
+
+        if torch.cuda.is_available():
+            info["gpu"] = torch.cuda.get_device_name(0)
+            info["cuda"] = torch.version.cuda
+    except ImportError:
+        pass
+    try:
+        rev = subprocess.run(["git", "rev-parse", "HEAD"], cwd=ROOT, capture_output=True, text=True)
+        dirty = subprocess.run(["git", "status", "--porcelain"], cwd=ROOT, capture_output=True, text=True)
+        info["git_commit"] = rev.stdout.strip() or None
+        info["git_dirty"] = bool(dirty.stdout.strip())
+    except FileNotFoundError:
+        pass
+    return info

eval/compare.py

@@ -0,0 +1,45 @@
+"""Paired comparison of runs on the same questions (bootstrap 95% CI of the accuracy difference).
+
+  python -m eval.compare --base results/exp_sft_gmmlu_lora_logit --runs results/exp_x results/exp_y [--dataset sinhalammlu]
+"""
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+
+import numpy as np
+
+from .common import read_jsonl
+from .metrics import aggregate_items
+
+
+def correctness(run: Path, dataset: str) -> tuple[dict, dict]:
+    recs = read_jsonl(run / "predictions.jsonl")
+    items = {r["uid"]: {"options": r["perm"], "answer": r["gold"]} for r in recs}
+    agg = aggregate_items(items, recs)
+    keep = {u for u in agg if u.startswith({"sinhalammlu": "smmlu/", "gmmlu_si": "gmmlu_si/"}.get(dataset, ""))}
+    return ({u: agg[u]["pred"] == agg[u]["gold"] for u in keep}, {u: agg[u]["vote"] == agg[u]["gold"] for u in keep})
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--base", required=True, type=Path)
+    ap.add_argument("--runs", nargs="+", required=True, type=Path)
+    ap.add_argument("--dataset", default="sinhalammlu", choices=["sinhalammlu", "gmmlu_si"])
+    args = ap.parse_args()
+    base = correctness(args.base, args.dataset)
+    rng = np.random.default_rng(0)
+    for run in args.runs:
+        c = correctness(run, args.dataset)
+        for k, label in ((0, "orig"), (1, "vote")):
+            u = sorted(set(base[k]) & set(c[k]))
+            if not u:
+                continue
+            d = np.array([c[k][x] - base[k][x] for x in u], dtype=float)
+            lo, hi = np.percentile([d[rng.integers(0, len(d), len(d))].mean() for _ in range(5000)], [2.5, 97.5])
+            print(f"{run.name:40s} {label}: {100 * np.mean([c[k][x] for x in u]):5.1f} vs {100 * np.mean([base[k][x] for x in u]):5.1f}"
+                  f"  diff {100 * d.mean():+.2f} [{100 * lo:+.2f}, {100 * hi:+.2f}]  n={len(u)}")
+
+
+if __name__ == "__main__":
+    main()

eval/decide_rft.py

@@ -0,0 +1,52 @@
+"""Go / no-go for the reasoning-trace model, decided by a rule fixed in advance (before any result is seen).
+
+Compares greedy thinking of the trained model (with its adapter) with greedy thinking of the base model on the SAME held-out
+questions: the 400-question SinhalaMMLU dev sample (never used for traces or training) and the 300-question Global-MMLU
+non-academic proxy (never used). Paired differences are pooled over both sets.
+
+GO if pooled difference >= +1.5 points AND neither set is worse than -3.0 points. Writes results/rft_decision.json.
+
+  python -m eval.decide_rft --pairs think_base_smmlu_dev_sample400:think_rft_dev400:sinhalammlu \
+      think_base_gmmlu_si_val_nonacademic300:think_rft_nonacademic300:gmmlu_si
+"""
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+import numpy as np
+
+from .common import ROOT
+from .compare import correctness
+
+MIN_POOLED, MIN_EACH = 1.5, -3.0
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--pairs", nargs="+", required=True, help="base_run:new_run:dataset")
+    ap.add_argument("--out", default="rft_decision.json", help="file name under results/")
+    args = ap.parse_args()
+    diffs, per = [], {}
+    for pair in args.pairs:
+        b, n, ds = pair.split(":")
+        base, new = correctness(ROOT / "results" / b, ds)[0], correctness(ROOT / "results" / n, ds)[0]
+        uids = sorted(set(base) & set(new))
+        d = np.array([new[u] - base[u] for u in uids], dtype=float)
+        per[n] = {"n": len(uids), "base": 100 * float(np.mean([base[u] for u in uids])),
+                  "new": 100 * float(np.mean([new[u] for u in uids])), "diff": 100 * float(d.mean())}
+        diffs.append(d)
+    pooled = np.concatenate(diffs)
+    rng = np.random.default_rng(0)
+    lo, hi = np.percentile([pooled[rng.integers(0, len(pooled), len(pooled))].mean() for _ in range(5000)], [2.5, 97.5])
+    go = bool(100 * pooled.mean() >= MIN_POOLED and all(v["diff"] >= MIN_EACH for v in per.values()))
+    out = {"rule": f"pooled diff >= {MIN_POOLED} and each set >= {MIN_EACH}", "per_set": per, "n": len(pooled),
+           "pooled_diff": 100 * float(pooled.mean()), "ci95": [100 * float(lo), 100 * float(hi)], "go": go}
+    (ROOT / "results" / args.out).write_text(json.dumps(out, indent=2))
+    print(json.dumps(out, indent=2))
+    print("DECISION:", "GO" if go else "NO-GO")
+
+
+if __name__ == "__main__":
+    main()

eval/metrics.py

@@ -0,0 +1,116 @@
+"""Aggregate per-(question, permutation) records into accuracy tables and position-bias diagnostics."""
+from __future__ import annotations
+
+from collections import Counter, defaultdict
+from statistics import mean, pstdev
+
+from .scorers import wilson
+
+GROUP_KEYS = ("dataset", "category", "subject", "level", "grade", "cultural_sensitivity")
+
+
+def group_value(item: dict, key: str):
+    return item.get(key, item.get("meta", {}).get(key))
+
+
+def aggregate_items(items: dict[str, dict], records: list[dict]) -> dict[str, dict]:
+    """Combine permutations per question: mean of option probabilities (in original order) -> vote."""
+    by_uid = defaultdict(list)
+    for r in records:
+        by_uid[r["uid"]].append(r)
+    out = {}
+    for uid, rs in by_uid.items():
+        n = len(items[uid]["options"])
+        rs.sort(key=lambda r: r["perm_k"])
+        mean_probs = [mean(r["probs_orig"][j] for r in rs) for j in range(n)]
+        if any(sum(r["probs_orig"]) > 0 for r in rs):
+            vote = max(range(n), key=lambda j: (mean_probs[j], -j)) + 1
+        else:
+            vote = None
+        preds = [r["pred_orig"] for r in rs]
+        out[uid] = {
+            "gold": items[uid].get("answer"), "pred": rs[0]["pred_orig"], "vote": vote,
+            "consistent": len(set(preds)) == 1 and preds[0] is not None,
+            "mean_probs": mean_probs,
+        }
+    return out
+
+
+def _acc_block(uids: list[str], agg: dict, records_by_perm: dict[int, dict[str, dict]]) -> dict:
+    n = len(uids)
+    k = sum(agg[u]["pred"] == agg[u]["gold"] for u in uids)
+    kv = sum(agg[u]["vote"] == agg[u]["gold"] for u in uids)
+    per_perm = [sum(rp[u]["correct"] for u in uids) / n for rp in records_by_perm.values()]
+    lo, hi = wilson(k, n)
+    return {
+        "n": n, "acc": k / n, "ci95": [lo, hi], "acc_vote": kv / n,
+        "acc_perm_mean": mean(per_perm), "acc_perm_std": pstdev(per_perm) if len(per_perm) > 1 else 0.0,
+        "consistency": sum(agg[u]["consistent"] for u in uids) / n,
+    }
+
+
+def compute_metrics(items: dict[str, dict], records: list[dict]) -> dict:
+    agg = aggregate_items(items, records)
+    records_by_perm: dict[int, dict[str, dict]] = defaultdict(dict)
+    for r in records:
+        records_by_perm[r["perm_k"]][r["uid"]] = r
+    uids = sorted(agg)
+
+    m = {"overall": _acc_block(uids, agg, records_by_perm), "groups": {}}
+    for key in GROUP_KEYS:
+        groups = defaultdict(list)
+        for u in uids:
+            v = group_value(items[u], key)
+            if v is not None:
+                groups[str(v)].append(u)
+        if len(groups) > 1 or (key in ("dataset", "subject", "category") and groups):
+            m["groups"][key] = {g: _acc_block(us, agg, records_by_perm) for g, us in sorted(groups.items())}
+    # Macro average over subjects (each subject weighted equally)
+    if "subject" in m["groups"]:
+        m["overall"]["acc_macro_subject"] = mean(b["acc"] for b in m["groups"]["subject"].values())
+
+    # Position bias over all (question, permutation) records
+    n_opt = Counter(len(items[r["uid"]]["options"]) for r in records).most_common(1)[0][0]
+    rs = [r for r in records if len(items[r["uid"]]["options"]) == n_opt]
+    pred_pos = Counter(r["pred_display"] for r in rs)
+    gold_pos = Counter(r["gold_display"] for r in rs)
+    acc_by_gold = {p: mean(r["correct"] for r in rs if r["gold_display"] == p) for p in sorted(gold_pos)}
+    m["position_bias"] = {
+        "n_options": n_opt,
+        "pred_dist": {str(p): pred_pos.get(p, 0) / len(rs) for p in range(1, n_opt + 1)},
+        "pred_none": pred_pos.get(None, 0) / len(rs),
+        "gold_dist": {str(p): gold_pos.get(p, 0) / len(rs) for p in range(1, n_opt + 1)},
+        "acc_by_gold_position": {str(p): a for p, a in acc_by_gold.items()},
+    }
+    if any("label_mass" in r for r in records):
+        m["overall"]["label_mass_mean"] = mean(r["label_mass"] for r in records if "label_mass" in r)
+    if any(r.get("method") == "generate" for r in records):
+        m["overall"]["parse_fail_rate"] = mean(r["pred_display"] is None for r in records)
+    return m
+
+
+def markdown_report(name: str, m: dict) -> str:
+    o = m["overall"]
+    lines = [f"# {name}", "",
+             f"**Accuracy (original order): {o['acc']:.4f}** "
+             f"[95% CI {o['ci95'][0]:.3f}-{o['ci95'][1]:.3f}], n={o['n']}  ",
+             f"Permutation-vote accuracy: {o['acc_vote']:.4f}  ",
+             f"Mean ± std over permutations: {o['acc_perm_mean']:.4f} ± {o['acc_perm_std']:.4f}  ",
+             f"Consistency across permutations: {o['consistency']:.3f}  "]
+    if "acc_macro_subject" in o:
+        lines.append(f"Macro average over subjects: {o['acc_macro_subject']:.4f}  ")
+    if "label_mass_mean" in o:
+        lines.append(f"Mean probability mass on label tokens: {o['label_mass_mean']:.3f}  ")
+    if "parse_fail_rate" in o:
+        lines.append(f"Unparseable generations: {o['parse_fail_rate']:.3f}  ")
+    for key, groups in m["groups"].items():
+        lines += ["", f"## By {key}", "", "| group | n | acc | 95% CI | vote acc | consistency |", "|---|---:|---:|---|---:|---:|"]
+        for g, b in groups.items():
+            lines.append(f"| {g} | {b['n']} | {b['acc']:.3f} | {b['ci95'][0]:.2f}-{b['ci95'][1]:.2f} | "
+                         f"{b['acc_vote']:.3f} | {b['consistency']:.2f} |")
+    pb = m["position_bias"]
+    lines += ["", "## Position bias (all permutations)", "", "| position | predicted | gold | acc when gold here |", "|---|---:|---:|---:|"]
+    for p in pb["pred_dist"]:
+        lines.append(f"| {p} | {pb['pred_dist'][p]:.3f} | {pb['gold_dist'][p]:.3f} | "
+                     f"{pb['acc_by_gold_position'].get(p, float('nan')):.3f} |")
+    return "\n".join(lines) + "\n"

eval/perm_sc.py

@@ -0,0 +1,70 @@
+"""Option-order self-consistency validation (decision for the Medium run).
+
+Prior answer = the current pipeline's answer on the stand-in questions: majority vote of greedy thinking + 3 sampled paths
+(ties -> greedy), the analogue of candidate B's final answer. Candidate method: add the answers from K re-orderings of the
+options (thinking each time, mapped back to the original numbering) and vote again (ties -> prior).
+
+GO if the candidate beats the prior by at least +1.0 point on the stand-in. Writes results/perm_decision.json.
+
+  python -m eval.perm_sc --greedy results/think_base_gmmlu_si_val_nonacademic300 --samples results/sc3_base_nonacademic \
+      --perms results/permthink_nonacademic300
+"""
+from __future__ import annotations
+
+import argparse
+import json
+from collections import Counter, defaultdict
+from pathlib import Path
+
+import numpy as np
+
+from .common import ROOT, read_jsonl
+
+MIN_GAIN = 1.0
+
+
+def vote(ballots: list, first_wins: bool = True):
+    c = Counter(b for b in ballots if b)
+    if not c:
+        return None
+    top = max(c.values())
+    w = [a for a, n in c.items() if n == top]
+    return ballots[0] if ballots[0] in w else min(w)
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--greedy", required=True, type=Path)
+    ap.add_argument("--samples", required=True, type=Path)
+    ap.add_argument("--perms", required=True, type=Path)
+    args = ap.parse_args()
+    g = {r["uid"]: r for r in read_jsonl(args.greedy / "predictions.jsonl") if r["perm_k"] == 0}
+    s, p = defaultdict(list), defaultdict(list)
+    for r in read_jsonl(args.samples / "predictions.jsonl"):
+        s[r["uid"]].append(r["pred_orig"])
+    for r in read_jsonl(args.perms / "predictions.jsonl"):
+        if r["perm_k"] >= 1:
+            p[r["uid"]].append(r["pred_orig"])
+    uids = sorted(set(g) & set(s) & set(p))
+    gold = {u: g[u]["gold"] for u in uids}
+    prior = {u: vote([g[u]["pred_orig"]] + s[u]) for u in uids}
+    cand = {u: vote([prior[u]] + p[u]) for u in uids}
+    perms_only = {u: vote([g[u]["pred_orig"]] + p[u]) for u in uids}
+    acc = lambda m: np.array([m[u] == gold[u] for u in uids], dtype=float)  # noqa: E731
+    a_g, a_prior, a_cand, a_perm = acc({u: g[u]["pred_orig"] for u in uids}), acc(prior), acc(cand), acc(perms_only)
+    d = a_cand - a_prior
+    rng = np.random.default_rng(0)
+    lo, hi = np.percentile([d[rng.integers(0, len(d), len(d))].mean() for _ in range(5000)], [2.5, 97.5])
+    singles = [100 * float(np.mean([p[u][i] == gold[u] for u in uids])) for i in range(min(len(p[u]) for u in uids))]
+    out = {"n": len(uids), "greedy": 100 * a_g.mean(), "prior_vote_greedy_plus_samples": 100 * a_prior.mean(),
+           "single_reordered_runs": singles, "greedy_plus_reorders_vote": 100 * a_perm.mean(),
+           "candidate_prior_plus_reorders": 100 * a_cand.mean(), "gain_vs_prior": 100 * float(d.mean()),
+           "ci95": [100 * float(lo), 100 * float(hi)], "rule": f"candidate - prior >= {MIN_GAIN}",
+           "go": bool(100 * d.mean() >= MIN_GAIN)}
+    (ROOT / "results" / "perm_decision.json").write_text(json.dumps(out, indent=2))
+    print(json.dumps(out, indent=2))
+    print("DECISION:", "GO" if out["go"] else "NO-GO")
+
+
+if __name__ == "__main__":
+    main()

eval/pool_decide.py

@@ -0,0 +1,68 @@
+"""Go / no-go for cross-model pooling, by a rule fixed (27 Oct, before any of these runs existed) on FRESH stand-in questions
+(gmmlu_si_val_nonacademic_fresh300: never used for any earlier decision, training or trace generation).
+
+Baseline = candidate B's method: greedy thinking of the base model; where the fast one-step answer (base model, 4-order logit
+vote) disagrees with it, vote over greedy + 3 sampled paths (ties -> greedy).
+Candidate = vote over base greedy + 3 base samples + RFT2 greedy + 3 RFT2 samples, on every question (ties -> base greedy).
+
+GO if candidate - baseline >= +1.0 point. Writes results/pool_decision.json.
+
+  python -m eval.pool_decide --fast pool_fast_base_fresh300 --base-greedy pool_think_base_fresh300 \
+      --base-samples pool_sc3_base_fresh300 --rft-greedy pool_think_rft2_fresh300 --rft-samples pool_sc3_rft2_fresh300
+"""
+from __future__ import annotations
+
+import argparse
+import json
+from collections import defaultdict
+
+import numpy as np
+
+from .common import ROOT, read_jsonl
+from .perm_sc import vote
+
+MIN_GAIN = 1.0
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    for a in ("fast", "base-greedy", "base-samples", "rft-greedy", "rft-samples"):
+        ap.add_argument(f"--{a}", required=True)
+    args = ap.parse_args()
+
+    def runs(name: str) -> tuple[dict, dict]:
+        preds, probs = defaultdict(list), defaultdict(lambda: np.zeros(4))
+        for r in read_jsonl(ROOT / "results" / name / "predictions.jsonl"):
+            preds[r["uid"]].append(r["pred_orig"])
+            probs[r["uid"]][: len(r["probs_orig"])] += r["probs_orig"]
+        return preds, probs
+
+    _, fast = runs(args.fast)
+    bg, _ = runs(args.base_greedy)
+    bs, _ = runs(args.base_samples)
+    rg, _ = runs(args.rft_greedy)
+    rs, _ = runs(args.rft_samples)
+    gold = {r["uid"]: r["gold"] for r in read_jsonl(ROOT / "results" / args.base_greedy / "predictions.jsonl")}
+    uids = sorted(set(bg) & set(bs) & set(rg) & set(rs) & set(fast))
+    gated = {u: int(fast[u].argmax()) + 1 != bg[u][0] for u in uids}
+    baseline = {u: vote([bg[u][0]] + (bs[u] if gated[u] else [])) for u in uids}
+    cand = {u: vote([bg[u][0]] + bs[u] + [rg[u][0]] + rs[u]) for u in uids}
+    acc = lambda m: np.array([m[u] == gold[u] for u in uids], dtype=float)  # noqa: E731
+    d = acc(cand) - acc(baseline)
+    rng = np.random.default_rng(0)
+    lo, hi = np.percentile([d[rng.integers(0, len(d), len(d))].mean() for _ in range(5000)], [2.5, 97.5])
+    out = {"n": len(uids), "gated_fraction": float(np.mean(list(gated.values()))),
+           "base_greedy": 100 * float(acc({u: bg[u][0] for u in uids}).mean()),
+           "rft2_greedy": 100 * float(acc({u: rg[u][0] for u in uids}).mean()),
+           "base_vote_all4": 100 * float(acc({u: vote([bg[u][0]] + bs[u]) for u in uids}).mean()),
+           "rft2_vote_all4": 100 * float(acc({u: vote([rg[u][0]] + rs[u]) for u in uids}).mean()),
+           "baseline_B": 100 * float(acc(baseline).mean()), "candidate_pooled8": 100 * float(acc(cand).mean()),
+           "gain": 100 * float(d.mean()), "ci95": [100 * float(lo), 100 * float(hi)],
+           "rule": f"candidate - baseline >= {MIN_GAIN}", "go": bool(100 * d.mean() >= MIN_GAIN)}
+    (ROOT / "results" / "pool_decision.json").write_text(json.dumps(out, indent=2))
+    print(json.dumps(out, indent=2))
+    print("DECISION:", "GO" if out["go"] else "NO-GO")
+
+
+if __name__ == "__main__":
+    main()

eval/prompts.py

@@ -0,0 +1,71 @@
+"""Prompt construction. Options are numbered 1..N because Codabench answers are digits ("1".."4").
+
+Every model gets the same template text; only the wrapping differs (chat template vs raw text).
+Few-shot examples are a fixed list, identical for every question (shared-task rule).
+"""
+from __future__ import annotations
+
+TEMPLATES = {
+    "en": {
+        "header": "The following is a multiple choice question about {subject}. "
+                  "Reply with only the number (1-{n}) of the correct answer.",
+        "answer": "Answer:",
+    },
+    # Sinhala wording should be reviewed by a native speaker.
+    "si": {
+        "header": "පහත දැක්වෙන්නේ {subject} පිළිබඳ බහුවරණ ප්‍රශ්නයකි. "
+                  "නිවැරදි පිළිතුරේ අංකය (1-{n}) පමණක් ලියන්න.",
+        "answer": "පිළිතුර:",
+    },
+}
+
+
+def subject_name(item: dict, lang: str) -> str:
+    if lang == "si" and item.get("meta", {}).get("subject_original"):
+        return item["meta"]["subject_original"]
+    return item["subject"].replace("_", " ")
+
+
+def question_block(question: str, options: list[str]) -> str:
+    lines = [question] + [f"{i}. {opt}" for i, opt in enumerate(options, 1)]
+    return "\n".join(lines)
+
+
+def user_text(item: dict, options: list[str], lang: str) -> str:
+    t = TEMPLATES[lang]
+    header = t["header"].format(subject=subject_name(item, lang), n=len(options))
+    return f"{header}\n\n{question_block(item['question'], options)}"
+
+
+class PromptBuilder:
+    def __init__(self, tokenizer, lang: str = "en", chat: bool = True,
+                 fewshot: list[dict] | None = None, answer_prefix: str = "",
+                 chat_kwargs: dict | None = None):
+        self.tok, self.lang, self.chat = tokenizer, lang, chat
+        self.fewshot = fewshot or []
+        self.answer_prefix = answer_prefix
+        self.chat_kwargs = {"enable_thinking": False, **(chat_kwargs or {})}
+
+    @property
+    def add_special_tokens(self) -> bool:
+        # Chat templates already insert BOS; raw prompts need the tokenizer to add it.
+        return not self.chat
+
+    def mcq(self, item: dict, options: list[str]) -> str:
+        """Prompt whose next token should be the answer number."""
+        ans = TEMPLATES[self.lang]["answer"]
+        if self.chat:
+            msgs = []
+            for ex in self.fewshot:
+                msgs.append({"role": "user", "content": user_text(ex, ex["options"], self.lang)})
+                msgs.append({"role": "assistant", "content": str(ex["answer"])})
+            msgs.append({"role": "user", "content": user_text(item, options, self.lang)})
+            prompt = self.tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
+                                                  **self.chat_kwargs)
+            return prompt + self.answer_prefix
+        shots = [f"{user_text(ex, ex['options'], self.lang)}\n{ans} {ex['answer']}" for ex in self.fewshot]
+        return "\n\n".join(shots + [f"{user_text(item, options, self.lang)}\n{ans}"])
+
+    def cloze_context(self, item: dict) -> str:
+        """Context for option-likelihood scoring: question only, options not listed."""
+        return f"{item['question']}\n{TEMPLATES[self.lang]['answer']}"

eval/recompute.py

@@ -0,0 +1,34 @@
+"""Recompute metrics.json / report.md of finished runs from their saved predictions (no model needed).
+Use after changing data grouping or metrics code.  Usage: python -m eval.recompute results/base_*
+"""
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+from .common import read_jsonl
+from .metrics import markdown_report
+from .run_eval import metrics_by_dataset, resolve_data
+
+
+def main() -> None:
+    for d in map(Path, sys.argv[1:]):
+        if not (d / "metrics.json").exists():
+            continue
+        old = json.loads((d / "metrics.json").read_text())
+        records = read_jsonl(d / "predictions.jsonl")
+        uids = {r["uid"] for r in records}
+        items = {r["uid"]: r for name in old["run"]["args"]["data"] for r in read_jsonl(resolve_data(name)) if r["uid"] in uids}
+        missing = uids - items.keys()
+        if missing:
+            print(f"{d.name}: {len(missing)} predicted uids not in current data, skipped")
+            continue
+        new = {"by_dataset": metrics_by_dataset(items, records), "run": old["run"]}
+        (d / "metrics.json").write_text(json.dumps(new, indent=2, ensure_ascii=False))
+        (d / "report.md").write_text("\n".join(markdown_report(f"{d.name} / {k}", m) for k, m in new["by_dataset"].items()))
+        print(f"{d.name}: recomputed")
+
+
+if __name__ == "__main__":
+    main()

eval/run_eval.py

@@ -0,0 +1,208 @@
+"""Evaluate one model on one or more processed MCQ files.
+
+Example:
+  python -m eval.run_eval --model Qwen/Qwen3-4B-Instruct-2507 --data sinhalammlu_dev gmmlu_si_val \
+      --method logit --n-perms 4 --run-name qwen3-4b-inst_logit
+
+Outputs results/<run-name>/{config.json, predictions.jsonl, metrics.json, report.md}
+and, with --write-submission, a Codabench-format predictions.zip per dataset.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import random
+import re
+import time
+import zipfile
+from pathlib import Path
+
+import yaml
+
+from .common import ROOT, cap_gpu_memory, env_fingerprint, load_env, read_jsonl, set_seed, write_jsonl
+from .metrics import compute_metrics, markdown_report
+from .prompts import PromptBuilder, user_text
+
+PROCESSED = ROOT / "data" / "processed"
+BASE_NAME = re.compile(r"(-base|-pt)$|^google/gemma-4-e[24]b$", re.I)
+
+
+def make_perm(n: int, k: int, mode: str, seed: int, uid: str) -> list[int]:
+    """perm[j] = original 0-based index shown at display position j. k=0 is the original order."""
+    if k == 0:
+        return list(range(n))
+    if mode == "identity":  # repeated samples of the same prompt (self-consistency), no reordering
+        return list(range(n))
+    if mode == "cyclic" and k < n:
+        return [(j + k) % n for j in range(n)]
+    rng = random.Random(f"{seed}|{uid}|{k}")
+    p = list(range(n))
+    rng.shuffle(p)
+    return p
+
+
+def resolve_data(name: str) -> Path:
+    p = Path(name)
+    return p if p.suffix == ".jsonl" else PROCESSED / f"{name}.jsonl"
+
+
+def score_items(items: dict[str, dict], pb: PromptBuilder, scorer, method: str, n_perms: int,
+                perm_mode: str, seed: int, first_perm: int = 0) -> tuple[list[dict], str | None]:
+    """One record per (question, option order). Shared by evaluation and the submission pipeline.
+    Items without a gold answer (test data) get gold/correct = None."""
+    n_perms = 1 if method == "cloze" else n_perms
+    jobs = [(uid, k, make_perm(len(it["options"]), k, perm_mode, seed, uid))
+            for uid, it in items.items() for k in range(first_perm, n_perms)]
+    first_prompt = None
+    if method == "cloze":
+        res = scorer.cloze([pb.cloze_context(items[u]) for u, _, _ in jobs], [items[u]["options"] for u, _, _ in jobs])
+    else:
+        prompts = [pb.mcq(items[u], [items[u]["options"][j] for j in perm]) for u, _, perm in jobs]
+        fn = scorer.logit if method == "logit" else scorer.generate
+        res = fn(prompts, [len(items[u]["options"]) for u, _, _ in jobs])
+        first_prompt = prompts[0] if prompts else None
+
+    records = []
+    for (uid, k, perm), r in zip(jobs, res):
+        gold = items[uid].get("answer")
+        probs_orig = [0.0] * len(perm)
+        for j, p in enumerate(r["probs"]):
+            probs_orig[perm[j]] = p
+        pred_orig = perm[r["pred"] - 1] + 1 if r["pred"] else None
+        rec = {"uid": uid, "perm_k": k, "perm": perm, "method": method,
+               "pred_display": r["pred"], "gold_display": perm.index(gold - 1) + 1 if gold else None,
+               "pred_orig": pred_orig, "gold": gold, "correct": pred_orig == gold if gold else None,
+               "probs_orig": probs_orig}
+        for extra in ("label_mass", "text", "new_tokens"):
+            if extra in r:
+                rec[extra] = r[extra]
+        records.append(rec)
+    return records, first_prompt
+
+
+def parse_args() -> argparse.Namespace:
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--config", help="YAML with any of the options below (CLI wins)")
+    ap.add_argument("--model")
+    ap.add_argument("--revision")
+    ap.add_argument("--adapter", help="LoRA adapter directory loaded on top of --model")
+    ap.add_argument("--data", nargs="+", default=["sinhalammlu_dev"])
+    ap.add_argument("--method", choices=["logit", "generate", "cloze"], default="logit")
+    ap.add_argument("--lang", choices=["en", "si"], default="en", help="instruction language")
+    ap.add_argument("--chat", choices=["auto", "on", "off"], default="auto")
+    ap.add_argument("--answer-prefix", default="")
+    ap.add_argument("--thinking", action="store_true", help="enable the chat template's thinking mode (use with --method generate)")
+    ap.add_argument("--n-perms", type=int, default=1, help="option orders per question (1 = original only)")
+    ap.add_argument("--first-perm", type=int, default=0, help="skip the first orders (1 = skip the original order)")
+    ap.add_argument("--perm-mode", choices=["cyclic", "random", "identity"], default="cyclic",
+                    help="identity + --temperature > 0: n-perms independent samples of the same prompt")
+    ap.add_argument("--temperature", type=float, default=0.0, help="> 0: sampled generation (self-consistency)")
+    ap.add_argument("--top-p", type=float, default=0.95)
+    ap.add_argument("--top-k", type=int, default=64)
+    ap.add_argument("--fewshot-file", help="processed JSONL; the same examples precede every question")
+    ap.add_argument("--n-shot", type=int, default=0)
+    ap.add_argument("--max-new-tokens", type=int, default=16)
+    ap.add_argument("--token-budget", type=int, default=32768, help="padded tokens per forward batch")
+    ap.add_argument("--max-batch", type=int, default=64)
+    ap.add_argument("--dtype", default="bfloat16")
+    ap.add_argument("--gpu-mem-gb", type=float, help="PyTorch memory cap (default 100, env SMMLU_GPU_MEM_GB)")
+    ap.add_argument("--limit", type=int, help="first N questions per dataset (smoke tests)")
+    ap.add_argument("--seed", type=int, default=1234)
+    ap.add_argument("--run-name")
+    ap.add_argument("--out-dir", default=str(ROOT / "results"))
+    ap.add_argument("--write-submission", action="store_true")
+    args, _ = ap.parse_known_args()
+    if args.config:
+        ap.set_defaults(**{k.replace("-", "_"): v for k, v in yaml.safe_load(open(args.config)).items()})
+    args = ap.parse_args()
+    if not args.model:
+        ap.error("--model is required")
+    return args
+
+
+def main() -> None:
+    args = parse_args()
+    load_env()
+    set_seed(args.seed)
+    cap_gpu_memory(args.gpu_mem_gb)
+    from .scorers import Scorer, load_model, loaded_param_stats
+
+    run_name = args.run_name or f"{args.model.split('/')[-1]}_{args.method}_{args.lang}_{args.n_shot}shot"
+    out = Path(args.out_dir) / run_name
+    out.mkdir(parents=True, exist_ok=True)
+
+    fewshot = read_jsonl(resolve_data(args.fewshot_file))[: args.n_shot] if args.n_shot else []
+    shot_uids = {e["uid"] for e in fewshot}
+    items: dict[str, dict] = {}
+    for d in args.data:
+        rows = [r for r in read_jsonl(resolve_data(d)) if r["uid"] not in shot_uids]
+        items.update({r["uid"]: r for r in (rows[: args.limit] if args.limit else rows)})
+
+    t0 = time.time()
+    model, tok = load_model(args.model, args.revision, args.dtype, args.adapter)
+    chat = {"on": True, "off": False}.get(args.chat, tok.chat_template is not None and not BASE_NAME.search(args.model))
+    pb = PromptBuilder(tok, lang=args.lang, chat=chat, fewshot=fewshot, answer_prefix=args.answer_prefix,
+                       chat_kwargs={"enable_thinking": args.thinking})
+    scorer = Scorer(model, tok, pb.add_special_tokens, args.token_budget, args.max_batch, args.max_new_tokens)
+    if args.temperature > 0:
+        scorer.sampling = {"do_sample": True, "temperature": args.temperature, "top_p": args.top_p, "top_k": args.top_k}
+    load_s = time.time() - t0
+
+    t0 = time.time()
+    records, first_prompt = score_items(items, pb, scorer, args.method, args.n_perms, args.perm_mode, args.seed, args.first_perm)
+    score_s = time.time() - t0
+    if first_prompt:
+        (out / "example_prompt.txt").write_text(first_prompt, encoding="utf-8")
+
+    # Tokens per character of the question block: tokenizer efficiency on Sinhala
+    sample = list(items.values())[:500]
+    texts = [user_text(it, it["options"], args.lang) for it in sample]
+    n_tok = sum(len(x) for x in tok(texts, add_special_tokens=False)["input_ids"])
+
+    metrics = {"by_dataset": metrics_by_dataset(items, records)}
+    metrics["run"] = {
+        "run_name": run_name, "args": vars(args), "chat": chat, "n_questions": len(items),
+        "n_forward_items": len(records), "load_seconds": round(load_s, 1), "score_seconds": round(score_s, 1),
+        "tokens_per_char": n_tok / sum(len(t) for t in texts),
+        "params": loaded_param_stats(model), "env": env_fingerprint(),
+    }
+    (out / "config.json").write_text(json.dumps(vars(args), indent=2))
+    write_jsonl(out / "predictions.jsonl", records)
+    (out / "metrics.json").write_text(json.dumps(metrics, indent=2, ensure_ascii=False))
+    report = "\n".join(markdown_report(f"{run_name} / {d}", m) for d, m in metrics["by_dataset"].items())
+    (out / "report.md").write_text(report, encoding="utf-8")
+
+    if args.write_submission:
+        write_submissions(out, items, records)
+
+    for d, m in metrics["by_dataset"].items():
+        o = m["overall"]
+        print(f"{run_name} | {d}: acc={o['acc']:.4f} vote={o['acc_vote']:.4f} "
+              f"perm={o['acc_perm_mean']:.4f}±{o['acc_perm_std']:.4f} n={o['n']}")
+    print(f"load {load_s:.0f}s, scoring {score_s:.0f}s -> {out}")
+
+
+def metrics_by_dataset(items: dict, records: list[dict]) -> dict:
+    out = {}
+    for d in sorted({it["dataset"] for it in items.values()}):
+        sub_items = {u: it for u, it in items.items() if it["dataset"] == d}
+        out[d] = compute_metrics(sub_items, [r for r in records if r["uid"] in sub_items])
+    return out
+
+
+def write_submissions(out: Path, items: dict, records: list[dict]) -> None:
+    """Codabench format: predictions.zip containing predictions.jsonl, answer = permutation-vote prediction."""
+    from .metrics import aggregate_items
+
+    agg = aggregate_items(items, records)
+    for d in sorted({it["dataset"] for it in items.values()}):
+        rows = [{"q_no": it["q_no"], "subject": it.get("subject_raw", it["subject"]), "category": it["category"],
+                 "level": it["level"], "answer": str(agg[u]["vote"] or 1)}
+                for u, it in items.items() if it["dataset"] == d]
+        path = out / f"submission_{d}.zip"
+        with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as z:
+            z.writestr("predictions.jsonl", "".join(json.dumps(r, ensure_ascii=False) + "\n" for r in rows))
+
+
+if __name__ == "__main__":
+    main()

eval/scorers.py

@@ -0,0 +1,224 @@
+"""Model loading and the three scoring methods.
+
+  logit     next-token log-probs restricted to the option labels "1".."N" (argmax)
+  generate  greedy decoding, first standalone label digit in the output is the prediction
+  cloze     length-normalised log-likelihood of each option text given the question
+"""
+from __future__ import annotations
+
+import math
+import re
+
+import torch
+from tqdm.auto import tqdm
+
+
+def resolve_revision(name: str, revision: str | None) -> tuple[str | None, str | None]:
+    """(revision to load, commit to record). Local dirs record their SOURCE.json; offline runs need a full sha."""
+    import json
+    import os
+    from pathlib import Path
+
+    if Path(name).is_dir():
+        src = Path(name) / "SOURCE.json"
+        if not src.exists():
+            return None, "local"
+        meta = json.loads(src.read_text())
+        return None, meta.get("model_id") or meta["source_commit"] + "+text-only"
+    if revision and re.fullmatch(r"[0-9a-f]{40}", revision):
+        return revision, revision
+    if os.environ.get("HF_HUB_OFFLINE") == "1":
+        raise ValueError(f"offline run needs a full 40-character commit sha for {name}")
+    from huggingface_hub import HfApi
+
+    sha = HfApi().model_info(name, revision=revision).sha  # pin the exact commit
+    return sha, sha
+
+
+def load_model(name: str, revision: str | None = None, dtype: str = "bfloat16", adapter: str | None = None):
+    from transformers import AutoModelForCausalLM, AutoTokenizer
+
+    revision, commit = resolve_revision(name, revision)
+    kw = dict(dtype=getattr(torch, dtype), device_map="cuda", revision=revision)
+    try:
+        model = AutoModelForCausalLM.from_pretrained(name, **kw)
+    except ValueError:  # some multimodal checkpoints only register under image-text-to-text
+        from transformers import AutoModelForImageTextToText
+
+        model = AutoModelForImageTextToText.from_pretrained(name, **kw)
+    if adapter:  # LoRA adapter kept separate (can be switched off per question with model.disable_adapter())
+        from peft import PeftModel
+
+        model = PeftModel.from_pretrained(model, adapter)
+        commit = f"{commit}+adapter:{adapter}"
+    model.eval()
+    tok = AutoTokenizer.from_pretrained(name, revision=revision)
+    tok.padding_side = "left"
+    if tok.pad_token is None:
+        tok.pad_token = tok.eos_token
+    model.config.hub_commit = commit
+    return model, tok
+
+
+def loaded_param_stats(model) -> dict:
+    """Parameters actually resident at inference (tied weights counted once)."""
+    by_module: dict[str, int] = {}
+    for name, p in model.named_parameters():  # named_parameters() already de-duplicates tied tensors
+        top = ".".join(name.split(".")[:2])
+        by_module[top] = by_module.get(top, 0) + p.numel()
+    non_text = {k: v for k, v in by_module.items() if re.search(r"vision|visual|audio|image|mm_proj", k)}
+    total = sum(by_module.values())
+    return {"loaded_total": total, "loaded_non_text": sum(non_text.values()),
+            "non_text_modules": non_text, "commit": getattr(model.config, "hub_commit", None)}
+
+
+def label_token_ids(tok, n: int) -> list[list[int]]:
+    """Single-token encodings of each label ("1" and " 1" variants)."""
+    out = []
+    for i in range(1, n + 1):
+        ids = set()
+        for v in (str(i), f" {i}"):
+            enc = tok.encode(v, add_special_tokens=False)
+            if len(enc) == 1:
+                ids.add(enc[0])
+        if not ids:
+            raise ValueError(f"label {i} has no single-token encoding")
+        out.append(sorted(ids))
+    flat = [t for ids in out for t in ids]
+    assert len(flat) == len(set(flat)), "label token ids overlap"
+    return out
+
+
+def _batches(lengths: list[int], token_budget: int, max_batch: int):
+    """Length-sorted batches bounded by padded-token budget."""
+    order = sorted(range(len(lengths)), key=lambda i: -lengths[i])
+    batch, longest = [], 0
+    for i in order:
+        new_longest = max(longest, lengths[i])
+        if batch and (new_longest * (len(batch) + 1) > token_budget or len(batch) >= max_batch):
+            yield batch
+            batch, new_longest = [], lengths[i]
+        batch.append(i)
+        longest = new_longest
+    if batch:
+        yield batch
+
+
+def _forward_last(model, enc, keep: int):
+    try:
+        return model(**enc, logits_to_keep=keep).logits
+    except TypeError:
+        return model(**enc).logits[:, -keep:]
+
+
+class Scorer:
+    def __init__(self, model, tok, add_special_tokens: bool, token_budget: int = 32768,
+                 max_batch: int = 64, max_new_tokens: int = 16):
+        self.model, self.tok = model, tok
+        self.add_special = add_special_tokens
+        self.token_budget, self.max_batch = token_budget, max_batch
+        self.max_new_tokens = max_new_tokens
+        self.keep_raw = False  # also return the raw completion (thought channel included) for training on traces
+        self.sampling: dict = {}  # e.g. {"do_sample": True, "temperature": 0.8, "top_p": 0.95, "top_k": 64}
+
+    def _encode(self, texts: list[str]):
+        return self.tok(texts, return_tensors="pt", padding=True,
+                        add_special_tokens=self.add_special).to(self.model.device)
+
+    def _lengths(self, texts: list[str]) -> list[int]:
+        return [len(x) for x in self.tok(texts, add_special_tokens=self.add_special)["input_ids"]]
+
+    @torch.no_grad()
+    def logit(self, prompts: list[str], n_options: list[int]) -> list[dict]:
+        cache: dict[int, list[list[int]]] = {}
+        results: list[dict | None] = [None] * len(prompts)
+        lengths = self._lengths(prompts)
+        for idx in tqdm(list(_batches(lengths, self.token_budget, self.max_batch)), desc="logit", leave=False):
+            enc = self._encode([prompts[i] for i in idx])
+            logp = torch.log_softmax(_forward_last(self.model, enc, 1)[:, -1].float(), dim=-1)
+            for row, i in enumerate(idx):
+                n = n_options[i]
+                labels = cache.setdefault(n, label_token_ids(self.tok, n))
+                scores = torch.stack([torch.logsumexp(logp[row, ids], 0) for ids in labels])
+                probs = torch.softmax(scores, 0)
+                results[i] = {"probs": probs.tolist(), "pred": int(probs.argmax()) + 1,
+                              "label_mass": float(scores.exp().sum())}
+        return results
+
+    @torch.no_grad()
+    def generate(self, prompts: list[str], n_options: list[int]) -> list[dict]:
+        results: list[dict | None] = [None] * len(prompts)
+        lengths = self._lengths(prompts)
+        budget = max(self.token_budget // 2, 1)
+        for idx in tqdm(list(_batches(lengths, budget, self.max_batch)), desc="generate", leave=False):
+            enc = self._encode([prompts[i] for i in idx])
+            gen_kw = self.sampling or {"do_sample": False, "temperature": None, "top_p": None, "top_k": None}
+            out = self.model.generate(**enc, max_new_tokens=self.max_new_tokens, pad_token_id=self.tok.pad_token_id,
+                                      **gen_kw)
+            new = out[:, enc["input_ids"].shape[1]:]
+            raw = self.tok.batch_decode(new, skip_special_tokens=False)
+            texts = self.tok.batch_decode(new, skip_special_tokens=True)
+            for row, i in enumerate(idx):
+                n = n_options[i]
+                answer_text = texts[row]
+                if "<|channel>" in raw[row]:  # thinking mode: the answer is what follows the closed thought channel
+                    answer_text = raw[row].split("<channel|>")[-1] if "<channel|>" in raw[row] else ""
+                    answer_text = re.sub(r"<[^>]*\|>|<\|[^>]*>", " ", answer_text)
+                m = re.search(rf"(?<![0-9])([1-{n}])(?![0-9])", answer_text)
+                pred = int(m.group(1)) if m else None
+                probs = [1.0 if pred == k else 0.0 for k in range(1, n + 1)]
+                n_new = int((new[row] != self.tok.pad_token_id).sum())
+                results[i] = {"probs": probs, "pred": pred, "text": texts[row][-2000:], "new_tokens": n_new}
+                if self.keep_raw:
+                    rc = raw[row].replace(self.tok.pad_token or "", "") if self.tok.pad_token else raw[row]
+                    cut = rc.find("<turn|>")
+                    results[i]["raw"] = rc[: cut + len("<turn|>")] if cut >= 0 else rc
+        return results
+
+    @torch.no_grad()
+    def cloze(self, contexts: list[str], options: list[list[str]]) -> list[dict]:
+        """Left-padded so every continuation ends at the last position; keeps only needed logits."""
+        pairs = []  # (item index, ctx ids, cont ids, n chars)
+        for i, (ctx, opts) in enumerate(zip(contexts, options)):
+            ctx_ids = self.tok.encode(ctx, add_special_tokens=True)
+            for opt in opts:
+                cont = self.tok.encode(" " + opt, add_special_tokens=False)
+                pairs.append((i, ctx_ids, cont, len(opt)))
+        lengths = [len(c) + len(o) for _, c, o, _ in pairs]
+        scores = [0.0] * len(pairs)
+        pad = self.tok.pad_token_id
+        for idx in tqdm(list(_batches(lengths, self.token_budget, self.max_batch)), desc="cloze", leave=False):
+            L = max(lengths[j] for j in idx)
+            keep = max(len(pairs[j][2]) for j in idx) + 1
+            ids = torch.full((len(idx), L), pad, dtype=torch.long)
+            mask = torch.zeros((len(idx), L), dtype=torch.long)
+            for row, j in enumerate(idx):
+                seq = pairs[j][1] + pairs[j][2]
+                ids[row, L - len(seq):] = torch.tensor(seq)
+                mask[row, L - len(seq):] = 1
+            enc = {"input_ids": ids.to(self.model.device), "attention_mask": mask.to(self.model.device)}
+            logp = torch.log_softmax(_forward_last(self.model, enc, keep).float(), dim=-1)
+            for row, j in enumerate(idx):
+                cont = pairs[j][2]
+                k = len(cont)
+                # logits at kept positions [-k-1 .. -2] predict tokens at [-k .. -1]
+                lp = logp[row, keep - k - 1: keep - 1].gather(-1, torch.tensor(cont, device=logp.device)[:, None])
+                scores[j] = float(lp.sum()) / max(pairs[j][3], 1)
+        results: list[dict] = []
+        pos = 0
+        for opts in options:
+            s = torch.tensor(scores[pos: pos + len(opts)])
+            pos += len(opts)
+            probs = torch.softmax(s, 0)
+            results.append({"probs": probs.tolist(), "pred": int(probs.argmax()) + 1})
+        return results
+
+
+def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
+    if n == 0:
+        return (math.nan, math.nan)
+    p = k / n
+    d = 1 + z * z / n
+    c = (p + z * z / (2 * n)) / d
+    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
+    return (c - h, c + h)

eval/self_consistency.py

@@ -0,0 +1,52 @@
+"""Self-consistency: majority vote over a greedy thinking run and sampled thinking runs of the same questions.
+
+  python -m eval.self_consistency --greedy results/think_base_academic --samples results/sc3_base_academic
+
+Votes = greedy answer + every sampled answer (unparsed samples abstain); ties go to the greedy answer. Prints accuracy of
+greedy alone, samples alone (each), and the vote, with a paired bootstrap CI of vote minus greedy.
+"""
+from __future__ import annotations
+
+import argparse
+from collections import Counter, defaultdict
+from pathlib import Path
+
+import numpy as np
+
+from .common import read_jsonl
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--greedy", required=True, type=Path)
+    ap.add_argument("--samples", required=True, type=Path)
+    args = ap.parse_args()
+    greedy = {r["uid"]: r for r in read_jsonl(args.greedy / "predictions.jsonl") if r["perm_k"] == 0}
+    samples = defaultdict(list)
+    for r in read_jsonl(args.samples / "predictions.jsonl"):
+        samples[r["uid"]].append(r["pred_orig"])
+    uids = sorted(set(greedy) & set(samples))
+    g = np.array([greedy[u]["pred_orig"] == greedy[u]["gold"] for u in uids], dtype=float)
+    vote = []
+    for u in uids:
+        ballots = [greedy[u]["pred_orig"]] + samples[u]
+        c = Counter(b for b in ballots if b is not None)
+        if not c:
+            vote.append(False)
+            continue
+        top = max(c.values())
+        winners = [a for a, n in c.items() if n == top]
+        choice = greedy[u]["pred_orig"] if greedy[u]["pred_orig"] in winners else min(winners)
+        vote.append(choice == greedy[u]["gold"])
+    v = np.array(vote, dtype=float)
+    k = max(len(samples[u]) for u in uids)
+    per_sample = [np.mean([samples[u][i] == greedy[u]["gold"] for u in uids if len(samples[u]) > i]) for i in range(k)]
+    d = v - g
+    rng = np.random.default_rng(0)
+    lo, hi = np.percentile([d[rng.integers(0, len(d), len(d))].mean() for _ in range(5000)], [2.5, 97.5])
+    print(f"{args.samples.name}: n={len(uids)}  greedy {100 * g.mean():.1f}  samples {' / '.join(f'{100 * x:.1f}' for x in per_sample)}"
+          f"  vote(greedy+{k}) {100 * v.mean():.1f}  diff {100 * d.mean():+.2f} [{100 * lo:+.2f}, {100 * hi:+.2f}]")
+
+
+if __name__ == "__main__":
+    main()

eval/summarize.py

@@ -0,0 +1,84 @@
+"""Combine results/*/metrics.json into comparison tables (reports/<name>.md).
+
+Usage: python -m eval.summarize --runs results/base_* --out reports/baselines_zeroshot.md
+"""
+from __future__ import annotations
+
+import argparse
+import glob
+import json
+from pathlib import Path
+
+
+def load(run_dirs: list[str]) -> dict[str, dict]:
+    runs = {}
+    for d in sorted(run_dirs):
+        f = Path(d) / "metrics.json"
+        if f.exists():
+            runs[Path(d).name] = json.loads(f.read_text())
+    return runs
+
+
+def fmt(x) -> str:
+    return "-" if x is None else f"{100 * x:.1f}"
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--runs", nargs="+", required=True)
+    ap.add_argument("--out", required=True)
+    args = ap.parse_args()
+    runs = load([d for pat in args.runs for d in glob.glob(pat)])
+    names = list(runs)
+    short = {n: n.removeprefix("base_").removeprefix("exp_") for n in names}
+
+    lines = ["# Baseline comparison", "",
+             "Accuracy in %. `orig` = original option order; `vote` = mean probabilities over cyclic option "
+             "permutations; `±` = std of accuracy across permutations (position sensitivity). Random = 25.", ""]
+
+    # Overview
+    lines += ["## Overview", "", "| model | params loaded | tok/char | SinhalaMMLU dev orig | vote | ± | macro-subj | "
+              "Global-MMLU si | Belebele si | label mass | consistency |", "|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|"]
+    for n in names:
+        r, bd = runs[n]["run"], runs[n]["by_dataset"]
+        s = bd.get("sinhalammlu", {}).get("overall", {})
+        g = bd.get("gmmlu_si", {}).get("overall", {})
+        b = bd.get("belebele_si", {}).get("overall", {})
+        lines.append(f"| {short[n]} | {r['params']['loaded_total'] / 1e9:.2f}B | {r['tokens_per_char']:.2f} | "
+                     f"{fmt(s.get('acc'))} | {fmt(s.get('acc_vote'))} | {fmt(s.get('acc_perm_std'))} | "
+                     f"{fmt(s.get('acc_macro_subject'))} | {fmt(g.get('acc'))} | {fmt(b.get('acc'))} | "
+                     f"{s.get('label_mass_mean', float('nan')):.2f} | {s.get('consistency', float('nan')):.2f} |")
+
+    def table(dataset: str, key: str, title: str) -> None:
+        groups = sorted({g for n in names for g in runs[n]["by_dataset"].get(dataset, {}).get("groups", {}).get(key, {})})
+        if not groups:
+            return
+        lines.extend(["", f"## {title}", "", "| " + key + " | n | " + " | ".join(short[n] for n in names) + " |",
+                      "|---|---:|" + "---:|" * len(names)])
+        for g in groups:
+            cells, cnt = [], "-"
+            for n in names:
+                blk = runs[n]["by_dataset"].get(dataset, {}).get("groups", {}).get(key, {}).get(g)
+                cells.append(fmt(blk["acc"]) if blk else "-")
+                cnt = blk["n"] if blk else cnt
+            lines.append(f"| {g} | {cnt} | " + " | ".join(cells) + " |")
+
+    table("sinhalammlu", "subject", "SinhalaMMLU dev: per subject")
+    table("sinhalammlu", "category", "SinhalaMMLU dev: per domain")
+    table("sinhalammlu", "grade", "SinhalaMMLU dev: per grade")
+    table("gmmlu_si", "category", "Global-MMLU si: per category")
+    table("gmmlu_si", "cultural_sensitivity", "Global-MMLU si: culturally sensitive (CS) / agnostic (CA) / unlabelled (-)")
+
+    lines += ["", "## Position bias on SinhalaMMLU dev (share of predictions per displayed position)", "",
+              "| model | 1 | 2 | 3 | 4 |", "|---|---:|---:|---:|---:|"]
+    for n in names:
+        pb = runs[n]["by_dataset"].get("sinhalammlu", {}).get("position_bias")
+        if pb:
+            lines.append(f"| {short[n]} | " + " | ".join(fmt(pb["pred_dist"].get(str(p))) for p in range(1, 5)) + " |")
+    Path(args.out).parent.mkdir(parents=True, exist_ok=True)
+    Path(args.out).write_text("\n".join(lines) + "\n")
+    print("\n".join(lines))
+
+
+if __name__ == "__main__":
+    main()

inference.py

@@ -0,0 +1,141 @@
+"""
+Sinhala MMLU Shared Task - Final inference script
+Team: Nexus
+
+The organizers will run this file exactly as it is, changing ONLY the DATA_REPO line
+below to point to the final Sinhala MMLU dataset:
+
+    pip install -r requirements.txt
+    python inference.py
+
+System (see README.md): Gemma 4 E4B-it, text-only (7.463B parameters, vision/audio towers removed, weights unchanged),
+no fine-tuning. Every question is answered with the model's thinking mode (greedy). Where the fast one-step answer
+(next-token scores over the labels 1-4, averaged over 4 cyclic option orders) disagrees with the greedy thinking answer,
+3 more reasoning paths are sampled (temperature 0.8, top-p 0.95, top-k 64, fixed seed 1234) and the 4 answers are
+majority-voted, ties going to the greedy answer ("agreement-gated self-consistency"; 892 of 2,238 test questions).
+Set SELF_CONSISTENCY = False for a fully greedy variant (not the one we submitted).
+
+Section 2 runs the exact pipeline that produced our Codabench submission (inference/predict.py with
+configs/submission_base_think_gated_sc.yaml), on the questions loaded by section 1.
+"""
+
+# ======================= ORGANIZERS CHANGE ONLY THIS LINE =======================
+DATA_REPO = "SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset"
+# ================================================================================
+
+OUTPUT_FILE = "predictions.jsonl"
+
+# ----------------------- TEAM SETTINGS (keep exactly as used for your final submission)
+MODEL_ID = "SinhalaMMLU-SharedTask/team-nexus-gemma-4-e4b-it-text"  # text-only extraction of google/gemma-4-E4B-it
+MODEL_REVISION = "8f686039aad1a865c9335c17874d9b870fdec766"  # exact commit of MODEL_ID used for our submission
+SEED = 1234
+SELF_CONSISTENCY = True  # True = our submitted system; False = greedy thinking only (fully deterministic)
+
+import json
+import os
+import random
+import re
+
+import numpy as np
+import torch
+from huggingface_hub import HfFileSystem, list_repo_files
+
+
+def set_seed(seed):
+    random.seed(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+    torch.cuda.manual_seed_all(seed)
+
+
+# =============== 1. LOAD THE DATASET FROM HUGGING FACE (do not change) ===============
+def load_questions(data_repo):
+    fs = HfFileSystem()
+    files = sorted(f for f in list_repo_files(data_repo, repo_type="dataset") if f.endswith(".json"))
+    rows = []
+    for f in files:
+        with fs.open(f"datasets/{data_repo}/{f}", "r", encoding="utf-8") as fh:
+            raw = json.load(fh)
+        if isinstance(raw, dict):
+            raw = next(v for v in raw.values() if isinstance(v, list))
+        for r in raw:
+            r["_file"] = f
+            rows.append(r)
+    return rows
+
+
+def get_level(r):
+    return (r.get("metadata") or {}).get("difficulty") or r["_file"].split("/")[0]
+
+
+# =============== 2. OUR SYSTEM ===============
+# predict_all() writes the loaded questions to a temporary folder (same file layout as the dataset repo), runs
+# inference/predict.py on it offline, and maps its answers back to `rows` by (level, subject, q_no).
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+CONFIG = os.path.join(HERE, "configs", "submission_base_think_gated_sc.yaml" if SELF_CONSISTENCY
+                      else "submission_base_think_greedy.yaml")
+
+
+def load_system():
+    from huggingface_hub import snapshot_download
+
+    return snapshot_download(MODEL_ID, revision=MODEL_REVISION)  # local folder with weights + tokenizer
+
+
+def predict_all(system, rows):
+    import subprocess
+    import sys
+    import tempfile
+    import zipfile
+
+    import yaml
+
+    model_dir = system
+    with tempfile.TemporaryDirectory() as tmp:
+        by_file = {}
+        for r in rows:
+            by_file.setdefault(r["_file"], []).append({k: v for k, v in r.items() if k != "_file"})
+        for f, recs in by_file.items():
+            path = os.path.join(tmp, "data", f)
+            os.makedirs(os.path.dirname(path), exist_ok=True)
+            with open(path, "w", encoding="utf-8") as fh:
+                json.dump(recs, fh, ensure_ascii=False)
+        with open(CONFIG, encoding="utf-8") as fh:
+            cfg = yaml.safe_load(fh)
+        cfg["model"] = model_dir
+        cfg["seed"] = SEED
+        cfg_path = os.path.join(tmp, "config.yaml")
+        with open(cfg_path, "w", encoding="utf-8") as fh:
+            yaml.safe_dump(cfg, fh, allow_unicode=True)
+        out_zip = os.path.join(tmp, "predictions.zip")
+        subprocess.run([sys.executable, "-m", "inference.predict", "--config", cfg_path,
+                        "--input", os.path.join(tmp, "data"), "--output", out_zip], cwd=HERE, check=True)
+        with zipfile.ZipFile(out_zip) as z:
+            preds = [json.loads(l) for l in z.read("predictions.jsonl").decode("utf-8").splitlines() if l.strip()]
+    ans = {(p["level"], p["subject"], p["q_no"]): p["answer"] for p in preds}
+    return [ans[(get_level(r), r["subject"], r["q_no"])] for r in rows]
+
+
+# =============== 3. WRITE PREDICTIONS (do not change) ===============
+if __name__ == "__main__":
+    set_seed(SEED)
+    rows = load_questions(DATA_REPO)
+    print(len(rows), "questions loaded from", DATA_REPO)
+
+    system = load_system()
+    answers = predict_all(system, rows)
+
+    assert len(answers) == len(rows), "predict_all() must return one answer per question"
+    with open(OUTPUT_FILE, "w", encoding="utf-8") as f:
+        for r, a in zip(rows, answers):
+            a = str(a).strip()
+            assert a in {"1", "2", "3", "4"}, f"invalid answer {a!r} for q_no={r.get('q_no')}"
+            f.write(json.dumps({
+                "q_no": r["q_no"],
+                "subject": r["subject"],
+                "category": r["category"],
+                "level": get_level(r),
+                "answer": a,
+            }, ensure_ascii=False, separators=(",", ":")) + "\n")
+    print("\nSaved", OUTPUT_FILE)

inference/check_predictions.py

@@ -0,0 +1,73 @@
+"""Score a Codabench-format predictions.zip against gold answers (dev only; test gold is hidden).
+
+  python -m inference.check_predictions --predictions dev_predictions.zip --gold data/raw/sinhalammlu/TEST/easy
+
+Also validates the file the way the leaderboard will need it: one row per question, required fields present,
+answers are digits within the option range. Reports accuracy per level and their macro average (task metric).
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import zipfile
+from collections import defaultdict
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+from inference.predict import read_records, to_item  # noqa: E402
+
+FIELDS = ("q_no", "subject", "category", "level", "answer")
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--predictions", required=True, type=Path)
+    ap.add_argument("--gold", required=True, type=Path)
+    args = ap.parse_args()
+
+    with zipfile.ZipFile(args.predictions) as z:
+        assert z.namelist() == ["predictions.jsonl"], f"zip must contain only predictions.jsonl, has {z.namelist()}"
+        preds = [json.loads(l) for l in z.read("predictions.jsonl").decode("utf-8").splitlines() if l.strip()]
+
+    gold = {}
+    for f, r in read_records(args.gold):
+        it = to_item(f, r)
+        gold[(it["level"], it["subject_raw"], it["q_no"])] = (r.get("answer"), len(it["options"]))
+
+    errors, seen = [], set()
+    correct = defaultdict(lambda: [0, 0])
+    for p in preds:
+        if any(k not in p for k in FIELDS):
+            errors.append(f"missing fields in {p}")
+            continue
+        key = (p["level"], p["subject"], p["q_no"])
+        if key in seen:
+            errors.append(f"duplicate {key}")
+        seen.add(key)
+        if key not in gold:
+            errors.append(f"unknown question {key}")
+            continue
+        ans, n = gold[key]
+        if not (isinstance(p["answer"], str) and p["answer"].isdigit() and 1 <= int(p["answer"]) <= n):
+            errors.append(f"bad answer {p['answer']!r} for {key}")
+            continue
+        if isinstance(ans, int) and 1 <= ans <= n:  # skip items with an invalid gold label
+            correct[p["level"]][0] += int(p["answer"]) == ans
+            correct[p["level"]][1] += 1
+    missing = set(gold) - seen
+    if missing:
+        errors.append(f"{len(missing)} questions without a prediction, e.g. {sorted(missing)[:3]}")
+
+    for lvl, (k, n) in sorted(correct.items()):
+        print(f"{lvl}: {k}/{n} = {100 * k / n:.2f}%")
+    if correct:
+        print(f"macro average over levels: {100 * sum(k / n for k, n in correct.values()) / len(correct):.2f}%")
+    print(f"{len(preds)} rows, {len(errors)} format errors")
+    for e in errors[:20]:
+        print("  ", e)
+    sys.exit(1 if errors else 0)
+
+
+if __name__ == "__main__":
+    main()

inference/predict.py

@@ -0,0 +1,238 @@
+"""Offline submission pipeline: question file(s) in, Codabench predictions.zip out. No human in the loop.
+
+  python -m inference.predict --input <test dir | .json | .jsonl> --output predictions.zip
+
+Runs with HF_HUB_OFFLINE=1 (no network); every setting comes from configs/submission.yaml.
+Optional self-consistency (config `sc_levels`, `sc_samples`, `sc_gate`): greedy + K sampled reasoning paths, majority
+vote; with `sc_gate: disagree` only for questions where the fast answer disagrees with the greedy thinking answer.
+Every thinking answer (greedy first, then samples) is also written to <output>.ballots.jsonl (scripts/pool_ballots.py).
+Optional LoRA adapter (config `adapter`) on top of `model`. Optional per-question thinking (config `thinking_levels`,
+e.g. [hard]): questions whose own `level` field is listed are answered with the model's thinking mode (with
+`thinking_without_adapter: true`, the adapter is switched off for them) (greedy, original option order); if no answer can be read from the output,
+the fast answer is used. All other questions use the fast logit answer (voted over `n_perms` option orders).
+
+Accepts the SinhalaMMLU JSON layout (list of {q_no, subject, category, question, choices, metadata.difficulty})
+and JSONL; q_no / subject / category / level are echoed exactly as given. Writes <output>.run.json next to the
+zip with the model commit, loaded parameter count, input hash and library versions.
+"""
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import sys
+import time
+import zipfile
+from collections import Counter
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+LEVELS = ("easy", "medium", "hard")
+
+
+def read_records(path: Path) -> list[tuple[Path, dict]]:
+    files = sorted(p for p in path.rglob("*") if p.suffix in (".json", ".jsonl")
+                   and not any(part.startswith(".") for part in p.relative_to(path).parts)) if path.is_dir() else [path]
+    # (hidden dirs skipped: huggingface_hub keeps download metadata as JSON under .cache/)
+    out = []
+    for f in files:
+        text = f.read_text(encoding="utf-8")
+        rows = [json.loads(l) for l in text.splitlines() if l.strip()] if f.suffix == ".jsonl" else json.loads(text)
+        out += [(f, r) for r in (rows if isinstance(rows, list) else [rows])]
+    if not out:
+        raise SystemExit(f"no questions found under {path}")
+    return out
+
+
+def to_item(f: Path, r: dict) -> dict:
+    options = r.get("choices", r.get("options"))
+    level = r.get("level") or (r.get("metadata") or {}).get("difficulty") \
+        or next((p for p in f.parts[::-1] if p.lower() in LEVELS), None)
+    missing = [k for k, v in {"q_no": r.get("q_no"), "subject": r.get("subject"), "category": r.get("category"),
+                              "question": r.get("question"), "choices": options, "level": level}.items() if v is None]
+    if missing:
+        raise SystemExit(f"{f}: record {r.get('q_no')} lacks {missing}; keys present: {sorted(r)}")
+    return {"uid": f"{level}/{r['subject']}/{r['q_no']}", "dataset": "test", "q_no": r["q_no"],
+            "subject": r["subject"].strip(), "subject_raw": r["subject"], "category": r["category"], "level": level,
+            "question": r["question"].strip(), "options": [str(o).strip() for o in options]}
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--input", required=True, type=Path)
+    ap.add_argument("--output", required=True, type=Path)
+    ap.add_argument("--config", type=Path, default=Path(__file__).resolve().parents[1] / "configs" / "submission.yaml")
+    ap.add_argument("--allow-online", action="store_true", help="debugging only; the submission runs offline")
+    ap.add_argument("--limit", type=int, help="first N questions only (smoke tests; never for a submission)")
+    ap.add_argument("--levels", nargs="+", help="answer only questions of these levels (then merge with scripts/merge_predictions.py)")
+    args = ap.parse_args()
+    if not args.allow_online:
+        os.environ["HF_HUB_OFFLINE"] = os.environ["TRANSFORMERS_OFFLINE"] = "1"
+
+    import yaml
+
+    from eval.common import cap_gpu_memory, env_fingerprint, set_seed
+    from eval.metrics import aggregate_items
+    from eval.prompts import PromptBuilder
+    from eval.run_eval import make_perm, score_items
+    from eval.scorers import Scorer, load_model, loaded_param_stats
+
+    cfg = yaml.safe_load(args.config.read_text())
+    set_seed(cfg["seed"])
+    cap_gpu_memory(cfg.get("gpu_mem_gb"))
+
+    raw = read_records(args.input)
+    items = {}
+    for f, r in raw:
+        it = to_item(f, r)
+        if it["uid"] in items:
+            raise SystemExit(f"duplicate question id {it['uid']}")
+        items[it["uid"]] = it
+    if args.levels:
+        items = {u: it for u, it in items.items() if it["level"] in set(args.levels)}
+    if args.limit:
+        items = dict(list(items.items())[: args.limit])
+    print(f"{len(items)} questions from {args.input}")
+
+    t0 = time.time()
+    root = Path(__file__).resolve().parents[1]
+    model_path = str(root / cfg["model"]) if (root / cfg["model"]).exists() else cfg["model"]
+    adapter = cfg.get("adapter")
+    adapter = str(root / adapter) if adapter and (root / adapter).exists() else adapter
+    model, tok = load_model(model_path, cfg.get("revision"), cfg.get("dtype", "bfloat16"), adapter)
+    pb = PromptBuilder(tok, lang=cfg["lang"], chat=cfg["chat"])
+    scorer = Scorer(model, tok, pb.add_special_tokens, cfg.get("token_budget", 32768), cfg.get("max_batch", 64))
+    if cfg.get("fast_without_adapter") and hasattr(model, "disable_adapter"):
+        with model.disable_adapter():  # fast one-step answer from the plain base model (used by the agreement gate)
+            records, _ = score_items(items, pb, scorer, cfg["method"], cfg["n_perms"], cfg["perm_mode"], cfg["seed"])
+    else:
+        records, _ = score_items(items, pb, scorer, cfg["method"], cfg["n_perms"], cfg["perm_mode"], cfg["seed"])
+    agg = aggregate_items(items, records)
+
+    # Thinking mode for selected levels (decided per question from its own level field)
+    think_levels = set(cfg.get("thinking_levels") or [])
+    think_uids = [u for u, it in items.items() if it["level"] in think_levels]
+    think_pred, fallbacks, sc_changed, sc_count = {}, 0, 0, 0
+    perm_levels, k_perm = set(cfg.get("perm_think_levels") or []), int(cfg.get("perm_think_orders", 0))
+    perm_uids = [u for u, it in items.items() if it["level"] in perm_levels] if k_perm > 0 else []
+    perm_changed = 0
+    if think_uids or perm_uids:
+        tpb = PromptBuilder(tok, lang=cfg["lang"], chat=cfg["chat"], chat_kwargs={"enable_thinking": True})
+        tscorer = Scorer(model, tok, tpb.add_special_tokens, cfg.get("token_budget", 32768), cfg.get("max_batch", 64),
+                         max_new_tokens=cfg.get("think_max_new_tokens", 2048))
+
+        def think_prompts(prompts: list[str], n_opts: list[int]) -> list[dict]:
+            if cfg.get("thinking_without_adapter") and hasattr(model, "disable_adapter"):
+                with model.disable_adapter():  # base model reasons; the adapter (trained to answer at once) does not
+                    return tscorer.generate(prompts, n_opts)
+            return tscorer.generate(prompts, n_opts)
+
+        def think(uids: list[str]) -> list[dict]:
+            return think_prompts([tpb.mcq(items[u], items[u]["options"]) for u in uids],
+                                 [len(items[u]["options"]) for u in uids])
+
+        for u, r in (zip(think_uids, think(think_uids)) if think_uids else []):
+            if r["pred"]:
+                think_pred[u] = r["pred"]
+            else:
+                fallbacks += 1
+        print(f"thinking: {len(think_uids)} questions ({sorted(think_levels)}), {fallbacks} fell back to the fast answer")
+        all_ballots = {u: [think_pred.get(u)] for u in think_uids}  # saved to <output>.ballots.jsonl for cross-model pooling
+
+        # Self-consistency for selected levels: greedy answer + K sampled reasoning paths, majority vote,
+        # ties go to the greedy answer. Fixed seed: reproducible for the same input, code and hardware.
+        sc_levels, k = set(cfg.get("sc_levels") or []), int(cfg.get("sc_samples", 0))
+        sc_uids = [u for u in think_uids if items[u]["level"] in sc_levels]
+        if cfg.get("sc_gate") == "disagree":  # only where the free one-step answer disagrees with greedy thinking
+            sc_uids = [u for u in sc_uids if think_pred.get(u) != agg[u]["pred"]]
+        sc_count = len(sc_uids) if k > 0 else 0
+        if sc_uids and k > 0:
+            import torch
+
+            torch.manual_seed(cfg["seed"])
+            tscorer.sampling = {"do_sample": True, "temperature": cfg.get("sc_temperature", 0.8),
+                                "top_p": cfg.get("sc_top_p", 0.95), "top_k": cfg.get("sc_top_k", 64)}
+            ballots = {u: [think_pred.get(u)] for u in sc_uids}
+            for _ in range(k):
+                for u, r in zip(sc_uids, think(sc_uids)):
+                    ballots[u].append(r["pred"])
+                    all_ballots[u].append(r["pred"])
+            tscorer.sampling = {}
+            for u, b in ballots.items():
+                counts = Counter(x for x in b if x)
+                if not counts:
+                    continue
+                top = max(counts.values())
+                winners = [a for a, c in counts.items() if c == top]
+                choice = b[0] if b[0] in winners else min(winners)
+                sc_changed += choice != think_pred.get(u)
+                think_pred[u] = choice
+            print(f"self-consistency: {len(sc_uids)} questions ({sorted(sc_levels)}, gate={cfg.get('sc_gate')}), greedy + {k} samples, "
+                  f"{sc_changed} answers changed by the vote")
+
+        # Option-order self-consistency: re-ask with the options in K other (cyclic) orders, thinking each time; the vote is over
+        # the previous answer (the `prior_predictions` file if given, else the answer so far) and the K re-orderings, ties
+        # go to the previous answer. Votes are mapped back to the original option numbering.
+        if perm_uids:
+            prior = {}
+            if cfg.get("prior_predictions"):
+                with zipfile.ZipFile(root / cfg["prior_predictions"]) as z:
+                    for line in z.read("predictions.jsonl").decode("utf-8").splitlines():
+                        r = json.loads(line)
+                        prior[(r["level"], r["subject"], r["q_no"])] = int(r["answer"])
+            ballots = {u: [prior.get((items[u]["level"], items[u]["subject_raw"], items[u]["q_no"]), think_pred.get(u))]
+                       for u in perm_uids}
+            for kk in range(1, k_perm + 1):
+                perms = {u: make_perm(len(items[u]["options"]), kk, "cyclic", cfg["seed"], u) for u in perm_uids}
+                res = think_prompts([tpb.mcq(items[u], [items[u]["options"][j] for j in perms[u]]) for u in perm_uids],
+                                    [len(items[u]["options"]) for u in perm_uids])
+                for u, r in zip(perm_uids, res):
+                    ballots[u].append(perms[u][r["pred"] - 1] + 1 if r["pred"] else None)
+                print(f"option-order thinking: order {kk}/{k_perm} done ({len(perm_uids)} questions)", flush=True)
+            for u, b in ballots.items():
+                counts = Counter(x for x in b if x)
+                if not counts:
+                    continue
+                top = max(counts.values())
+                winners = [a for a, c in counts.items() if c == top]
+                choice = b[0] if b[0] in winners else min(winners)
+                perm_changed += choice != b[0]
+                think_pred[u] = choice
+            print(f"option-order self-consistency: {len(perm_uids)} questions ({sorted(perm_levels)}), {k_perm} extra orders, "
+                  f"{perm_changed} answers changed by the vote")
+
+    rows = []
+    for uid, it in items.items():
+        n = len(it["options"])
+        ans = think_pred.get(uid, agg[uid]["vote"])
+        if ans is None or not 1 <= ans <= n:
+            raise SystemExit(f"no valid prediction for {uid}")
+        rows.append({"q_no": it["q_no"], "subject": it["subject_raw"], "category": it["category"],
+                     "level": it["level"], "answer": str(ans)})
+
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    with zipfile.ZipFile(args.output, "w", zipfile.ZIP_DEFLATED) as z:
+        z.writestr("predictions.jsonl", "".join(json.dumps(r, ensure_ascii=False) + "\n" for r in rows))
+    if think_uids:  # greedy + sampled thinking answers per question (None = unparsed), first = greedy
+        Path(str(args.output) + ".ballots.jsonl").write_text("".join(
+            json.dumps({"level": items[u]["level"], "subject": items[u]["subject_raw"], "q_no": items[u]["q_no"],
+                        "ballots": all_ballots[u]}, ensure_ascii=False) + "\n" for u in think_uids), encoding="utf-8")
+    digest = hashlib.sha256()
+    for f in sorted({str(f) for f, _ in raw}):
+        digest.update(Path(f).read_bytes())
+    run = {"config": cfg, "n_questions": len(rows), "seconds": round(time.time() - t0, 1),
+           "thinking": {"levels": sorted(think_levels), "questions": len(think_uids), "fallbacks": fallbacks},
+           "self_consistency": {"levels": sorted(set(cfg.get("sc_levels") or [])), "samples": int(cfg.get("sc_samples", 0)),
+                                "gate": cfg.get("sc_gate"), "questions": sc_count, "answers_changed": sc_changed},
+           "option_order_sc": {"levels": sorted(perm_levels), "orders": k_perm, "questions": len(perm_uids),
+                               "answers_changed": perm_changed},
+           "input_sha256": digest.hexdigest(), "params": loaded_param_stats(model), "env": env_fingerprint()}
+    Path(str(args.output) + ".run.json").write_text(json.dumps(run, indent=2, ensure_ascii=False))
+    print(f"wrote {len(rows)} predictions -> {args.output} "
+          f"({run['params']['loaded_total'] / 1e9:.3f}B parameters loaded, {run['seconds']:.0f}s)")
+
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -0,0 +1,12 @@
+# Versions used for our final submission (Python 3.12.3). PyTorch: we used torch 2.14.1 built for CUDA 13.0
+# (NVIDIA GB10, aarch64: pip install torch==2.14.1 --index-url https://download.pytorch.org/whl/cu130); any CUDA build of 2.14.1 works.
+torch==2.14.1
+transformers==5.19.0
+accelerate==1.15.0
+huggingface_hub==2.2.0
+tokenizers==0.23.3
+safetensors==0.8.0
+jinja2==3.1.6
+numpy==2.5.3
+pyyaml==6.0.3
+tqdm==4.70.1

training/.gitignore

@@ -0,0 +1,20 @@
+# Secrets
+.env
+*.token
+
+# Task data (CC BY-NC-ND 4.0): never commit. Fetch with scripts/fetch_data.py.
+data/
+
+# Weights, runs, caches
+models/
+checkpoints/
+results/
+*.safetensors
+*.bin
+*.pt
+*.zip
+.venv/
+__pycache__/
+*.pyc
+.cache/
+wandb/

training/DATA_LOG.md

@@ -0,0 +1,41 @@
+# Data log
+
+Every dataset touched by this project, how it is used, and every closed-API data-generation call (required
+disclosure). Task data is never committed; `scripts/fetch_data.py` re-downloads it at the pinned revision.
+
+## Evaluation data
+
+| Dataset | Revision | License | Use | Notes |
+|---|---|---|---|---|
+| SinhalaMMLU public Easy tier (`naist-nlp/SinhalaMMLU`, TEST/easy) | `7dd30681d9` | CC BY-NC-ND 4.0 | Dev evaluation | 1,851 questions, 14 subjects, grades 6-8, 4 options. `Geography/64` dropped (answer label `24`). Cite Pramodya et al., EMNLP 2025. |
+| Global-MMLU `si` (`CohereLabs/Global-MMLU`) | `0e619dbeb3` | Apache-2.0 | Harder internal validation: fixed 2,000-question stratified sample of `test` (ids in `configs/splits/gmmlu_si_val_ids.txt`); `dev` split = few-shot pool | Validation ids must be excluded from any training mix. |
+| Shared-task test set (`SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset`) | `56fb2da50c` | (organisers) | **Prediction only.** Never used for training, tuning, prompt design or model selection; no answers included | 2,238 questions (easy 642, medium 1,111, hard 485), 4 options, same JSON layout as dev. Stored in `data/test/` (git-ignored). |
+| Belebele `sin_Sinh` (`facebook/belebele`) | `7899cdfa4e` | CC BY-SA 4.0 | **Evaluation only**, never training | 900 reading-comprehension questions. |
+
+## Training data
+
+| Dataset | Revision | License | Use | Processing |
+|---|---|---|---|---|
+| Global-MMLU `si` test (minus the 2,000 validation ids) + dev split | `0e619dbeb3` | Apache-2.0 | Fine-tuning MCQs: `data/sft/gmmlu_si_train.jsonl` via `scripts/build_sft_gmmlu.py` | 12,327 pool → 11,447 kept. Dropped 819 with char-5-gram TF-IDF cosine ≥ 0.6 to any validation item (MMLU repeats questions across subjects) and 61 within-pool copies (≥ 0.95). Max similarity of kept items to SinhalaMMLU dev: 0.42. Options shuffled to balanced gold positions except options that refer to other options. Stats: `reports/data_stats_sft_gmmlu.json` |
+| FineWeb-2 `sin_Sinh` (`HuggingFaceFW/fineweb-2`) | `af9c13333e` | ODC-By 1.0 | Continued pretraining | `scripts/build_cpt_corpus.py` (filters below). Already MinHash-deduplicated upstream |
+| Sinhala Wikipedia (`wikimedia/wikipedia`, `20231101.si`) | `b04c8d1ceb` | CC BY-SA 3.0 / GFDL | Continued pretraining | Same filters |
+| NSINA news (`sinhala-nlp/NSINA`) | `a986af81ad` | CC BY-SA 4.0 | Continued pretraining | Same filters |
+
+Corpus filters (`scripts/build_cpt_corpus.py`), first failing rule recorded per document; counts in
+`reports/corpus_stats.json`: blocked exam / past-paper / curriculum domains; exam-like text (question-paper phrases or
+≥ 8 numbered option lines); < 200 characters; < 50% Sinhala letters; < 30% unique lines; exact duplicate across
+sources (normalised text); any word 8-gram shared with a SinhalaMMLU dev or Global-MMLU validation question.
+Not yet available: CulturaX `si` (gated, access not requested), MADLAD-400 `si` (path not found).
+
+### Contamination controls (apply to every training/pretraining source)
+
+- SinhalaMMLU dev questions come from **pastpapers.wiki** (93%) and **e-thaksalawa.moe.gov.lk** (5%), i.e. Sri Lankan
+  term-test and exam papers. Drop all documents from these domains, and from any other past-paper / exam site, from
+  web corpora (FineWeb-2, CulturaX, MADLAD-400, CC-100, OSCAR) before training.
+- N-gram overlap filter of every training corpus against SinhalaMMLU dev and the Global-MMLU validation split.
+- No scraped Sri Lankan O/L or A/L past papers. Ask the organisers before using government textbooks.
+
+## Closed-API data generation (disclosure)
+
+_None yet._ For each use: date, provider and model, purpose, prompt template location, number of items generated,
+filtering applied, and where the output is stored.

training/MODEL_MANIFEST.md

@@ -0,0 +1,52 @@
+# Model manifest
+
+Every neural component loaded at inference, with its exact total parameter count, release date and license.
+Shared-task limit: **total parameters of all inference components ≤ 8B** (embeddings, vision towers if loaded,
+adapters on top of the base, MoE total not active, quantisation does not reduce the count).
+Only open-weight models **released before 2026-10-01**.
+
+## Final submission
+
+- Base model: `google/gemma-4-E4B-it` @ `ee0ef6023621cff504d758262d4e04895a5af4a2`, text-only extraction
+  (`scripts/make_text_only.py`), published as `SinhalaMMLU-SharedTask/team-nexus-gemma-4-e4b-it-text`.
+- No fine-tuned derivative is used: no adapter, no merged weights.
+- Total parameters loaded: 7,463,013,376.
+- Command: `python inference.py` in the team repo, which runs `inference/predict.py` with
+  `configs/submission_base_think_gated_sc.yaml` (Codabench submission ID 50, score 62.93).
+
+## Candidate parameter accounting
+
+Generated by `python scripts/count_params.py` (counts read from safetensors headers; tied embeddings counted once;
+"Instantiated" is what `transformers` builds from the config). "Repo created" is the Hugging Face repo creation
+date and is only a proxy for the release date: confirm against the official announcement before relying on it.
+
+<!-- AUTO:PARAMS:BEGIN -->
+
+| Model | Verdict | Total (all stored) | Text model | Vision | Audio | MTP | Embeddings | Tied | Instantiated (class) | Repo created | License | Commit |
+|---|---|---:|---:|---:|---:|---:|---:|---|---|---|---|---|
+| `google/gemma-3-4b-it` | OK | 4.300B | 3.880B | 0.420B | - | - | 0.671B | no | 4.300B (Gemma3ForConditionalGeneration) | 2025-02-20 | gemma | `093f9f388b` |
+| `google/gemma-3n-E4B-it` | OK | 7.850B | 6.867B | 0.299B | 0.685B | - | 2.886B | no | 7.819B (Gemma3nForConditionalGeneration) | 2025-06-03 | gemma | `c1221e9c62` |
+| `google/gemma-3n-E2B-it` | OK | 5.439B | 4.456B | 0.299B | 0.685B | - | 2.551B | no | 5.418B (Gemma3nForConditionalGeneration) | 2025-06-12 | gemma | `5e092ebca1` |
+| `google/gemma-4-E4B-it` | OK | 7.996B | 7.518B | 0.169B | 0.309B | - | 3.490B | yes | 7.941B (Gemma4ForConditionalGeneration) | 2026-03-02 | apache-2.0 | `ee0ef60236` |
+| `google/gemma-4-E2B-it` | OK | 5.123B | 4.647B | 0.169B | 0.307B | - | 2.751B | yes | 5.104B (Gemma4ForConditionalGeneration) | 2026-03-02 | apache-2.0 | `3e22461f65` |
+| `google/gemma-4-12B-it` | OVER 8B | 11.960B | 11.907B | 0.050B | 0.002B | - | 1.007B | yes | 11.960B (Gemma4UnifiedForConditionalGeneration) | 2026-05-23 | apache-2.0 | `707f0a3b8a` |
+| `Qwen/Qwen3-4B-Instruct-2507` | OK | 4.022B | 4.022B | - | - | - | 0.389B | yes | 4.022B (Qwen3ForCausalLM) | 2025-08-05 | apache-2.0 | `cdbee75f17` |
+| `Qwen/Qwen3-4B` | OK | 4.022B | 4.022B | - | - | - | 0.389B | yes | 4.022B (Qwen3ForCausalLM) | 2025-04-27 | apache-2.0 | `1cfa9a7208` |
+| `Qwen/Qwen3.5-4B` | OK | 4.660B | 4.206B | 0.334B | - | 0.121B | 0.636B | yes | 4.206B (Qwen3_5ForCausalLM) | 2026-02-27 | apache-2.0 | `851bf6e806` |
+| `Qwen/Qwen3-8B` | OVER 8B | 8.191B | 8.191B | - | - | - | 1.245B | no | 8.191B (Qwen3ForCausalLM) | 2025-04-27 | apache-2.0 | `b968826d9c` |
+| `Qwen/Qwen3.5-9B` | OVER 8B | 9.653B | 8.954B | 0.456B | - | 0.243B | 2.034B | no | 8.954B (Qwen3_5ForCausalLM) | 2026-02-27 | apache-2.0 | `c202236235` |
+| `meta-llama/Llama-3.2-3B-Instruct` | OK | 3.213B | 3.213B | - | - | - | 0.394B | yes | 3.213B (LlamaForCausalLM) | 2024-09-18 | llama3.2 | `0cb88a4f76` |
+| `meta-llama/Llama-3.1-8B-Instruct` | OVER 8B | 8.030B | 8.030B | - | - | - | 1.051B | no | 8.030B (LlamaForCausalLM) | 2024-07-18 | llama3.1 | `0e9e39f249` |
+| `polyglots/SinLlama_v01`<br>adapter on `meta-llama/Meta-Llama-3-8B` (+3.592B) | OVER 8B | 11.622B | 11.622B | - | - | - | 1.051B | no | base 8030261248 + adapter | 2024-11-08 | ? | `4f464b14d5` |
+| `ibm-granite/granite-4.1-8b` | OVER 8B | 8.381B | 8.381B | - | - | - | 0.411B | yes | 8.381B (GraniteForCausalLM) | 2026-04-06 | apache-2.0 | `1504002f65` |
+| `ibm-granite/granite-4.2-8b` | OVER 8B | 8.792B | 8.792B | - | - | - | 0.822B | no | 8.792B (GraniteForCausalLM) | 2026-08-07 | apache-2.0 | `f8de16cdcd` |
+| `ibm-granite/granite-4.2-3b` | OK | 3.660B | 3.660B | - | - | - | 0.514B | no | 3.660B (GraniteForCausalLM) | 2026-08-07 | apache-2.0 | `e459acceac` |
+| `HuggingFaceTB/SmolLM3-3B` | OK | 3.075B | 3.075B | - | - | - | 0.263B | yes | 3.075B (SmolLM3ForCausalLM) | 2025-07-08 | apache-2.0 | `a07cc9a04f` |
+| `microsoft/Phi-4-mini-instruct` | OK | 3.836B | 3.836B | - | - | - | 0.615B | yes | 3.836B (Phi3ForCausalLM) | 2025-02-19 | mit | `cfbefacb99` |
+
+<!-- AUTO:PARAMS:END -->
+
+## Notes
+
+- A verdict of "OK only text-only" means the checkpoint stores a vision/audio tower or MTP head. The submission
+  must load a text-only model (and the stripped weights' count is what goes in the final row above).

training/README.md

@@ -0,0 +1,93 @@
+# SinhalaMMLU shared task (≤ 8B)
+
+System for the ICTer 2026 SinhalaMMLU shared task: Sinhala multiple-choice QA with open-weight models totalling
+≤ 8B parameters, no retrieval, no internet at inference. Built on SinhalaMMLU (Pramodya et al., EMNLP 2025):
+
+```bibtex
+@inproceedings{pramodya-etal-2025-sinhalammlu,
+  title     = "{S}inhala{MMLU}: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in {S}inhala",
+  author    = "Pramodya, Ashmari and Nelki, Nirasha and Shalinda, Heshan and Liyanage, Chamila and Sakai, Yusuke and
+               Pushpananda, Randil and Weerasinghe, Ruvan and Kamigaito, Hidetaka and Watanabe, Taro",
+  booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
+  year      = "2025",
+  pages     = "32931--32949",
+  url       = "https://aclanthology.org/2025.emnlp-main.1673/"
+}
+```
+
+**Keep this repository private until the final deadline. Never commit task data (`data/` is git-ignored).**
+
+## Layout
+
+| Path | Contents |
+|---|---|
+| `data/` | Git-ignored. `raw/` downloads, `processed/` normalised JSONL |
+| `eval/` | Evaluation harness (`run_eval.py`, scorers, metrics, `summarize.py`) |
+| `train/` | Fine-tuning / continued-pretraining code |
+| `inference/` | Final offline submission pipeline |
+| `configs/` | Candidate models, fixed data splits |
+| `scripts/` | Data building, parameter counting, overlap checks, synthetic MCQ generation |
+| `scripts/runs/` | The exact experiment queues behind every reported result and submission (see its README) |
+| `run_submission.sh` | One-command offline reproduction of a submission |
+| `reports/` | Aggregate results tables (no task data) |
+| `MODEL_MANIFEST.md` | Every model: exact total parameters, release date, license |
+| `DATA_LOG.md` | Every dataset and every closed-API data-generation use |
+
+## Setup (DGX Spark, aarch64, CUDA 13)
+
+```bash
+python3 -m venv .venv
+.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cu130
+.venv/bin/pip install -r requirements.lock.txt
+echo "hf_token = hf_..." > .env && chmod 600 .env   # read token; accept gated terms for SinhalaMMLU, Gemma 3/3n, Llama
+```
+
+Triton (used by `torch.compile`, Liger Kernel and flash-linear-attention) compiles helpers against `Python.h`.
+If `python3.12-dev` is installed, nothing else is needed. Without sudo, unpack the headers locally; the code
+(`eval/common.py: ensure_python_headers`) then adds them to `CPATH` automatically:
+
+```bash
+mkdir -p ~/.local/pydev && cd ~/.local/pydev
+V=3.12.3-1ubuntu0.17   # any 3.12.3 Ubuntu build; the machine's apt index may list a version no longer on the mirror
+for p in libpython3.12-dev python3.12-dev; do
+  curl -fO "http://ports.ubuntu.com/ubuntu-ports/pool/main/p/python3.12/${p}_${V}_arm64.deb"; done
+for f in *.deb; do dpkg -x "$f" root; done
+```
+
+Long GPU jobs: PyTorch is capped at 100 GB (`SMMLU_GPU_MEM_GB`). The Spark shares memory between CPU and GPU, so
+an uncapped job triggers the kernel OOM killer (which also kills tmux) instead of a CUDA out-of-memory error.
+
+## Commands
+
+```bash
+.venv/bin/python scripts/fetch_data.py            # SinhalaMMLU dev, Global-MMLU si, Belebele si -> data/raw
+.venv/bin/python scripts/prepare_data.py          # -> data/processed/*.jsonl (fixed splits in configs/splits)
+.venv/bin/python scripts/count_params.py          # exact parameter counts -> MODEL_MANIFEST.md
+scripts/runs/run_baselines.sh                     # zero-shot baselines -> results/, reports/
+./run_submission.sh [config]                      # reproduce a submission offline -> results/submissions/final.zip
+
+# One model, any setting
+.venv/bin/python -m eval.run_eval --model google/gemma-4-E4B-it --data sinhalammlu_dev gmmlu_si_val \
+    --method logit --n-perms 4 --lang en --run-name my_run
+```
+
+### Harness options
+
+- `--method logit` (next-token probability over labels 1-4), `generate` (greedy decode + parse), `cloze`
+  (length-normalised option likelihood).
+- `--n-perms K --perm-mode cyclic|random`: score each question under K option orders. Reports original-order
+  accuracy, permutation-vote accuracy, spread across orders, consistency, and position-bias tables.
+- `--lang en|si` instruction language; `--chat auto|on|off`; `--fewshot-file F --n-shot N` (the same fixed
+  examples for every question, as the rules require).
+- Breakdowns by dataset, domain, subject, grade and (Global-MMLU) cultural sensitivity, with Wilson 95% CIs.
+- Every run stores the exact model commit, library versions, git commit and the full argument set in `metrics.json`.
+- `--write-submission` writes a Codabench-format `predictions.zip` (`predictions.jsonl` with
+  `q_no, subject, category, level, answer`).
+
+## Rules checklist
+
+- Total parameters of everything loaded at inference ≤ 8B (embeddings, vision/audio towers if loaded, adapters).
+- Open-weight models released before 2026-10-01 only. No API, internet or retrieval of any kind at inference.
+- Fixed few-shot examples only (identical for every question). No training, tuning or prompt design on test questions.
+- Closed APIs only for training-data generation, logged in `DATA_LOG.md`.
+- Final pipeline runs offline from one command with pinned versions and fixed seeds.

training/configs/models.yaml

@@ -0,0 +1,26 @@
+# Candidate models. `chat` sets prompt wrapping in baselines; `baseline: false` = count parameters only.
+# Every entry must be verified in MODEL_MANIFEST.md (total params ≤ 8B, released before 2026-10-01, license).
+candidates:
+  # Gemma
+  - {id: google/gemma-3-4b-it, chat: true}
+  - {id: google/gemma-3n-E4B-it, chat: true}
+  - {id: google/gemma-3n-E2B-it, chat: true, baseline: false}
+  - {id: google/gemma-4-E4B-it, chat: true}
+  - {id: google/gemma-4-E2B-it, chat: true, baseline: false}
+  - {id: google/gemma-4-12B-it, chat: true, baseline: false}   # reference: expected over the limit
+  # Qwen
+  - {id: Qwen/Qwen3-4B-Instruct-2507, chat: true}
+  - {id: Qwen/Qwen3-4B, chat: true}                             # hybrid thinking model, thinking disabled
+  - {id: Qwen/Qwen3.5-4B, chat: true}
+  - {id: Qwen/Qwen3-8B, chat: true, baseline: false}            # reference: expected ~8.2B
+  - {id: Qwen/Qwen3.5-9B, chat: true, baseline: false}          # reference: expected over the limit
+  # Llama / SinLlama
+  - {id: meta-llama/Llama-3.2-3B-Instruct, chat: true}
+  - {id: meta-llama/Llama-3.1-8B-Instruct, chat: true, baseline: false}   # 8.03B: needs organiser ruling
+  - {id: polyglots/SinLlama_v01, chat: false, baseline: false}  # LoRA adapter on Meta-Llama-3-8B + extended vocab
+  # Others
+  - {id: ibm-granite/granite-4.1-8b, chat: true, baseline: false}   # 8.38B: over the limit
+  - {id: ibm-granite/granite-4.2-8b, chat: true, baseline: false}   # 8.79B: over the limit
+  - {id: ibm-granite/granite-4.2-3b, chat: true}
+  - {id: HuggingFaceTB/SmolLM3-3B, chat: true}
+  - {id: microsoft/Phi-4-mini-instruct, chat: true}

training/configs/splits/gmmlu_si_val_ids.txt

@@ -0,0 +1,2000 @@
+gmmlu_si/test/abstract_algebra/test/0
+gmmlu_si/test/abstract_algebra/test/12
+gmmlu_si/test/abstract_algebra/test/18
+gmmlu_si/test/abstract_algebra/test/19
+gmmlu_si/test/abstract_algebra/test/2
+gmmlu_si/test/abstract_algebra/test/21
+gmmlu_si/test/abstract_algebra/test/36
+gmmlu_si/test/abstract_algebra/test/5
+gmmlu_si/test/abstract_algebra/test/6
+gmmlu_si/test/abstract_algebra/test/76
+gmmlu_si/test/abstract_algebra/test/86
+gmmlu_si/test/abstract_algebra/test/89
+gmmlu_si/test/abstract_algebra/test/98
+gmmlu_si/test/abstract_algebra/test/99
+gmmlu_si/test/anatomy/test/100
+gmmlu_si/test/anatomy/test/101
+gmmlu_si/test/anatomy/test/104
+gmmlu_si/test/anatomy/test/108
+gmmlu_si/test/anatomy/test/112
+gmmlu_si/test/anatomy/test/113
+gmmlu_si/test/anatomy/test/118
+gmmlu_si/test/anatomy/test/119
+gmmlu_si/test/anatomy/test/124
+gmmlu_si/test/anatomy/test/132
+gmmlu_si/test/anatomy/test/19
+gmmlu_si/test/anatomy/test/34
+gmmlu_si/test/anatomy/test/57
+gmmlu_si/test/anatomy/test/84
+gmmlu_si/test/anatomy/test/85
+gmmlu_si/test/anatomy/test/89
+gmmlu_si/test/anatomy/test/9
+gmmlu_si/test/anatomy/test/93
+gmmlu_si/test/anatomy/test/96
+gmmlu_si/test/astronomy/test/1
+gmmlu_si/test/astronomy/test/101
+gmmlu_si/test/astronomy/test/11
+gmmlu_si/test/astronomy/test/112
+gmmlu_si/test/astronomy/test/114
+gmmlu_si/test/astronomy/test/115
+gmmlu_si/test/astronomy/test/130
+gmmlu_si/test/astronomy/test/136
+gmmlu_si/test/astronomy/test/144
+gmmlu_si/test/astronomy/test/19
+gmmlu_si/test/astronomy/test/23
+gmmlu_si/test/astronomy/test/24
+gmmlu_si/test/astronomy/test/31
+gmmlu_si/test/astronomy/test/39
+gmmlu_si/test/astronomy/test/43
+gmmlu_si/test/astronomy/test/49
+gmmlu_si/test/astronomy/test/67
+gmmlu_si/test/astronomy/test/71
+gmmlu_si/test/astronomy/test/73
+gmmlu_si/test/astronomy/test/8
+gmmlu_si/test/astronomy/test/84
+gmmlu_si/test/astronomy/test/90
+gmmlu_si/test/business_ethics/test/11
+gmmlu_si/test/business_ethics/test/17
+gmmlu_si/test/business_ethics/test/21
+gmmlu_si/test/business_ethics/test/25
+gmmlu_si/test/business_ethics/test/27
+gmmlu_si/test/business_ethics/test/33
+gmmlu_si/test/business_ethics/test/44
+gmmlu_si/test/business_ethics/test/49
+gmmlu_si/test/business_ethics/test/66
+gmmlu_si/test/business_ethics/test/7
+gmmlu_si/test/business_ethics/test/73
+gmmlu_si/test/business_ethics/test/77
+gmmlu_si/test/business_ethics/test/86
+gmmlu_si/test/business_ethics/test/98
+gmmlu_si/test/clinical_knowledge/test/107
+gmmlu_si/test/clinical_knowledge/test/108
+gmmlu_si/test/clinical_knowledge/test/116
+gmmlu_si/test/clinical_knowledge/test/137
+gmmlu_si/test/clinical_knowledge/test/17
+gmmlu_si/test/clinical_knowledge/test/177
+gmmlu_si/test/clinical_knowledge/test/179
+gmmlu_si/test/clinical_knowledge/test/18
+gmmlu_si/test/clinical_knowledge/test/187
+gmmlu_si/test/clinical_knowledge/test/2
+gmmlu_si/test/clinical_knowledge/test/204
+gmmlu_si/test/clinical_knowledge/test/205
+gmmlu_si/test/clinical_knowledge/test/206
+gmmlu_si/test/clinical_knowledge/test/207
+gmmlu_si/test/clinical_knowledge/test/211
+gmmlu_si/test/clinical_knowledge/test/217
+gmmlu_si/test/clinical_knowledge/test/223
+gmmlu_si/test/clinical_knowledge/test/225
+gmmlu_si/test/clinical_knowledge/test/228
+gmmlu_si/test/clinical_knowledge/test/232
+gmmlu_si/test/clinical_knowledge/test/235
+gmmlu_si/test/clinical_knowledge/test/236
+gmmlu_si/test/clinical_knowledge/test/29
+gmmlu_si/test/clinical_knowledge/test/36
+gmmlu_si/test/clinical_knowledge/test/4
+gmmlu_si/test/clinical_knowledge/test/40
+gmmlu_si/test/clinical_knowledge/test/48
+gmmlu_si/test/clinical_knowledge/test/49
+gmmlu_si/test/clinical_knowledge/test/52
+gmmlu_si/test/clinical_knowledge/test/60
+gmmlu_si/test/clinical_knowledge/test/63
+gmmlu_si/test/clinical_knowledge/test/7
+gmmlu_si/test/clinical_knowledge/test/75
+gmmlu_si/test/clinical_knowledge/test/8
+gmmlu_si/test/clinical_knowledge/test/9
+gmmlu_si/test/clinical_knowledge/test/92
+gmmlu_si/test/clinical_knowledge/test/97
+gmmlu_si/test/clinical_knowledge/test/98
+gmmlu_si/test/college_biology/test/121
+gmmlu_si/test/college_biology/test/122
+gmmlu_si/test/college_biology/test/125
+gmmlu_si/test/college_biology/test/126
+gmmlu_si/test/college_biology/test/133
+gmmlu_si/test/college_biology/test/134
+gmmlu_si/test/college_biology/test/140
+gmmlu_si/test/college_biology/test/142
+gmmlu_si/test/college_biology/test/28
+gmmlu_si/test/college_biology/test/30
+gmmlu_si/test/college_biology/test/32
+gmmlu_si/test/college_biology/test/37
+gmmlu_si/test/college_biology/test/43
+gmmlu_si/test/college_biology/test/49
+gmmlu_si/test/college_biology/test/56
+gmmlu_si/test/college_biology/test/65
+gmmlu_si/test/college_biology/test/7
+gmmlu_si/test/college_biology/test/70
+gmmlu_si/test/college_biology/test/72
+gmmlu_si/test/college_biology/test/81
+gmmlu_si/test/college_biology/test/91
+gmmlu_si/test/college_chemistry/test/12
+gmmlu_si/test/college_chemistry/test/14
+gmmlu_si/test/college_chemistry/test/17
+gmmlu_si/test/college_chemistry/test/18
+gmmlu_si/test/college_chemistry/test/19
+gmmlu_si/test/college_chemistry/test/33
+gmmlu_si/test/college_chemistry/test/34
+gmmlu_si/test/college_chemistry/test/41
+gmmlu_si/test/college_chemistry/test/44
+gmmlu_si/test/college_chemistry/test/45
+gmmlu_si/test/college_chemistry/test/54
+gmmlu_si/test/college_chemistry/test/67
+gmmlu_si/test/college_chemistry/test/9
+gmmlu_si/test/college_chemistry/test/92
+gmmlu_si/test/college_computer_science/test/16
+gmmlu_si/test/college_computer_science/test/17
+gmmlu_si/test/college_computer_science/test/2
+gmmlu_si/test/college_computer_science/test/30
+gmmlu_si/test/college_computer_science/test/31
+gmmlu_si/test/college_computer_science/test/36
+gmmlu_si/test/college_computer_science/test/38
+gmmlu_si/test/college_computer_science/test/54
+gmmlu_si/test/college_computer_science/test/58
+gmmlu_si/test/college_computer_science/test/61
+gmmlu_si/test/college_computer_science/test/67
+gmmlu_si/test/college_computer_science/test/76
+gmmlu_si/test/college_computer_science/test/89
+gmmlu_si/test/college_computer_science/test/90
+gmmlu_si/test/college_mathematics/test/15
+gmmlu_si/test/college_mathematics/test/26
+gmmlu_si/test/college_mathematics/test/29
+gmmlu_si/test/college_mathematics/test/32
+gmmlu_si/test/college_mathematics/test/43
+gmmlu_si/test/college_mathematics/test/5
+gmmlu_si/test/college_mathematics/test/50
+gmmlu_si/test/college_mathematics/test/57
+gmmlu_si/test/college_mathematics/test/66
+gmmlu_si/test/college_mathematics/test/68
+gmmlu_si/test/college_mathematics/test/70
+gmmlu_si/test/college_mathematics/test/76
+gmmlu_si/test/college_mathematics/test/9
+gmmlu_si/test/college_mathematics/test/98
+gmmlu_si/test/college_medicine/test/0
+gmmlu_si/test/college_medicine/test/107
+gmmlu_si/test/college_medicine/test/109
+gmmlu_si/test/college_medicine/test/111
+gmmlu_si/test/college_medicine/test/113
+gmmlu_si/test/college_medicine/test/118
+gmmlu_si/test/college_medicine/test/128
+gmmlu_si/test/college_medicine/test/13
+gmmlu_si/test/college_medicine/test/131
+gmmlu_si/test/college_medicine/test/136
+gmmlu_si/test/college_medicine/test/137
+gmmlu_si/test/college_medicine/test/146
+gmmlu_si/test/college_medicine/test/148
+gmmlu_si/test/college_medicine/test/162
+gmmlu_si/test/college_medicine/test/18
+gmmlu_si/test/college_medicine/test/2
+gmmlu_si/test/college_medicine/test/30
+gmmlu_si/test/college_medicine/test/33
+gmmlu_si/test/college_medicine/test/42
+gmmlu_si/test/college_medicine/test/51
+gmmlu_si/test/college_medicine/test/69
+gmmlu_si/test/college_medicine/test/7
+gmmlu_si/test/college_medicine/test/87
+gmmlu_si/test/college_medicine/test/89
+gmmlu_si/test/college_medicine/test/91
+gmmlu_si/test/college_physics/test/17
+gmmlu_si/test/college_physics/test/24
+gmmlu_si/test/college_physics/test/37
+gmmlu_si/test/college_physics/test/39
+gmmlu_si/test/college_physics/test/41
+gmmlu_si/test/college_physics/test/43
+gmmlu_si/test/college_physics/test/53
+gmmlu_si/test/college_physics/test/54
+gmmlu_si/test/college_physics/test/56
+gmmlu_si/test/college_physics/test/69
+gmmlu_si/test/college_physics/test/72
+gmmlu_si/test/college_physics/test/73
+gmmlu_si/test/college_physics/test/8
+gmmlu_si/test/college_physics/test/86
+gmmlu_si/test/college_physics/test/95
+gmmlu_si/test/computer_security/test/14
+gmmlu_si/test/computer_security/test/21
+gmmlu_si/test/computer_security/test/33
+gmmlu_si/test/computer_security/test/35
+gmmlu_si/test/computer_security/test/40
+gmmlu_si/test/computer_security/test/43
+gmmlu_si/test/computer_security/test/49
+gmmlu_si/test/computer_security/test/52
+gmmlu_si/test/computer_security/test/54
+gmmlu_si/test/computer_security/test/6
+gmmlu_si/test/computer_security/test/7
+gmmlu_si/test/computer_security/test/77
+gmmlu_si/test/computer_security/test/9
+gmmlu_si/test/computer_security/test/91
+gmmlu_si/test/conceptual_physics/test/0
+gmmlu_si/test/conceptual_physics/test/114
+gmmlu_si/test/conceptual_physics/test/134
+gmmlu_si/test/conceptual_physics/test/136
+gmmlu_si/test/conceptual_physics/test/137
+gmmlu_si/test/conceptual_physics/test/140
+gmmlu_si/test/conceptual_physics/test/145
+gmmlu_si/test/conceptual_physics/test/163
+gmmlu_si/test/conceptual_physics/test/167
+gmmlu_si/test/conceptual_physics/test/169
+gmmlu_si/test/conceptual_physics/test/178
+gmmlu_si/test/conceptual_physics/test/183
+gmmlu_si/test/conceptual_physics/test/186
+gmmlu_si/test/conceptual_physics/test/19
+gmmlu_si/test/conceptual_physics/test/190
+gmmlu_si/test/conceptual_physics/test/197
+gmmlu_si/test/conceptual_physics/test/203
+gmmlu_si/test/conceptual_physics/test/207
+gmmlu_si/test/conceptual_physics/test/208
+gmmlu_si/test/conceptual_physics/test/210
+gmmlu_si/test/conceptual_physics/test/217
+gmmlu_si/test/conceptual_physics/test/229
+gmmlu_si/test/conceptual_physics/test/234
+gmmlu_si/test/conceptual_physics/test/25
+gmmlu_si/test/conceptual_physics/test/31
+gmmlu_si/test/conceptual_physics/test/38
+gmmlu_si/test/conceptual_physics/test/41
+gmmlu_si/test/conceptual_physics/test/44
+gmmlu_si/test/conceptual_physics/test/66
+gmmlu_si/test/conceptual_physics/test/78
+gmmlu_si/test/conceptual_physics/test/81
+gmmlu_si/test/conceptual_physics/test/86
+gmmlu_si/test/conceptual_physics/test/92
+gmmlu_si/test/econometrics/test/100
+gmmlu_si/test/econometrics/test/105
+gmmlu_si/test/econometrics/test/11
+gmmlu_si/test/econometrics/test/113
+gmmlu_si/test/econometrics/test/34
+gmmlu_si/test/econometrics/test/36
+gmmlu_si/test/econometrics/test/38
+gmmlu_si/test/econometrics/test/5
+gmmlu_si/test/econometrics/test/50
+gmmlu_si/test/econometrics/test/64
+gmmlu_si/test/econometrics/test/72
+gmmlu_si/test/econometrics/test/8
+gmmlu_si/test/econometrics/test/83
+gmmlu_si/test/econometrics/test/88
+gmmlu_si/test/econometrics/test/89
+gmmlu_si/test/econometrics/test/99
+gmmlu_si/test/electrical_engineering/test/114
+gmmlu_si/test/electrical_engineering/test/117
+gmmlu_si/test/electrical_engineering/test/125
+gmmlu_si/test/electrical_engineering/test/142
+gmmlu_si/test/electrical_engineering/test/15
+gmmlu_si/test/electrical_engineering/test/24
+gmmlu_si/test/electrical_engineering/test/3
+gmmlu_si/test/electrical_engineering/test/32
+gmmlu_si/test/electrical_engineering/test/35
+gmmlu_si/test/electrical_engineering/test/36
+gmmlu_si/test/electrical_engineering/test/37
+gmmlu_si/test/electrical_engineering/test/42
+gmmlu_si/test/electrical_engineering/test/44
+gmmlu_si/test/electrical_engineering/test/45
+gmmlu_si/test/electrical_engineering/test/46
+gmmlu_si/test/electrical_engineering/test/74
+gmmlu_si/test/electrical_engineering/test/86
+gmmlu_si/test/electrical_engineering/test/9
+gmmlu_si/test/electrical_engineering/test/91
+gmmlu_si/test/electrical_engineering/test/95
+gmmlu_si/test/electrical_engineering/test/99
+gmmlu_si/test/elementary_mathematics/test/106
+gmmlu_si/test/elementary_mathematics/test/110
+gmmlu_si/test/elementary_mathematics/test/115
+gmmlu_si/test/elementary_mathematics/test/118
+gmmlu_si/test/elementary_mathematics/test/129
+gmmlu_si/test/elementary_mathematics/test/13
+gmmlu_si/test/elementary_mathematics/test/136
+gmmlu_si/test/elementary_mathematics/test/158
+gmmlu_si/test/elementary_mathematics/test/161
+gmmlu_si/test/elementary_mathematics/test/162
+gmmlu_si/test/elementary_mathematics/test/165
+gmmlu_si/test/elementary_mathematics/test/17
+gmmlu_si/test/elementary_mathematics/test/173
+gmmlu_si/test/elementary_mathematics/test/174
+gmmlu_si/test/elementary_mathematics/test/194
+gmmlu_si/test/elementary_mathematics/test/2
+gmmlu_si/test/elementary_mathematics/test/203
+gmmlu_si/test/elementary_mathematics/test/21
+gmmlu_si/test/elementary_mathematics/test/216
+gmmlu_si/test/elementary_mathematics/test/218
+gmmlu_si/test/elementary_mathematics/test/239
+gmmlu_si/test/elementary_mathematics/test/242
+gmmlu_si/test/elementary_mathematics/test/249
+gmmlu_si/test/elementary_mathematics/test/250
+gmmlu_si/test/elementary_mathematics/test/259
+gmmlu_si/test/elementary_mathematics/test/261
+gmmlu_si/test/elementary_mathematics/test/27
+gmmlu_si/test/elementary_mathematics/test/28
+gmmlu_si/test/elementary_mathematics/test/280
+gmmlu_si/test/elementary_mathematics/test/299
+gmmlu_si/test/elementary_mathematics/test/318
+gmmlu_si/test/elementary_mathematics/test/321
+gmmlu_si/test/elementary_mathematics/test/328
+gmmlu_si/test/elementary_mathematics/test/330
+gmmlu_si/test/elementary_mathematics/test/331
+gmmlu_si/test/elementary_mathematics/test/335
+gmmlu_si/test/elementary_mathematics/test/336
+gmmlu_si/test/elementary_mathematics/test/343
+gmmlu_si/test/elementary_mathematics/test/357
+gmmlu_si/test/elementary_mathematics/test/358
+gmmlu_si/test/elementary_mathematics/test/365
+gmmlu_si/test/elementary_mathematics/test/368
+gmmlu_si/test/elementary_mathematics/test/44
+gmmlu_si/test/elementary_mathematics/test/45
+gmmlu_si/test/elementary_mathematics/test/6
+gmmlu_si/test/elementary_mathematics/test/65
+gmmlu_si/test/elementary_mathematics/test/66
+gmmlu_si/test/elementary_mathematics/test/7
+gmmlu_si/test/elementary_mathematics/test/73
+gmmlu_si/test/elementary_mathematics/test/74
+gmmlu_si/test/elementary_mathematics/test/76
+gmmlu_si/test/elementary_mathematics/test/77
+gmmlu_si/test/elementary_mathematics/test/80
+gmmlu_si/test/elementary_mathematics/test/85
+gmmlu_si/test/formal_logic/test/102
+gmmlu_si/test/formal_logic/test/105
+gmmlu_si/test/formal_logic/test/106
+gmmlu_si/test/formal_logic/test/21
+gmmlu_si/test/formal_logic/test/28
+gmmlu_si/test/formal_logic/test/32
+gmmlu_si/test/formal_logic/test/37
+gmmlu_si/test/formal_logic/test/47
+gmmlu_si/test/formal_logic/test/50
+gmmlu_si/test/formal_logic/test/54
+gmmlu_si/test/formal_logic/test/55
+gmmlu_si/test/formal_logic/test/56
+gmmlu_si/test/formal_logic/test/63
+gmmlu_si/test/formal_logic/test/73
+gmmlu_si/test/formal_logic/test/82
+gmmlu_si/test/formal_logic/test/83
+gmmlu_si/test/formal_logic/test/88
+gmmlu_si/test/formal_logic/test/93
+gmmlu_si/test/global_facts/test/11
+gmmlu_si/test/global_facts/test/14
+gmmlu_si/test/global_facts/test/18
+gmmlu_si/test/global_facts/test/31
+gmmlu_si/test/global_facts/test/35
+gmmlu_si/test/global_facts/test/49
+gmmlu_si/test/global_facts/test/52
+gmmlu_si/test/global_facts/test/60
+gmmlu_si/test/global_facts/test/66
+gmmlu_si/test/global_facts/test/69
+gmmlu_si/test/global_facts/test/77
+gmmlu_si/test/global_facts/test/80
+gmmlu_si/test/global_facts/test/89
+gmmlu_si/test/global_facts/test/98
+gmmlu_si/test/high_school_biology/test/101
+gmmlu_si/test/high_school_biology/test/106
+gmmlu_si/test/high_school_biology/test/111
+gmmlu_si/test/high_school_biology/test/114
+gmmlu_si/test/high_school_biology/test/120
+gmmlu_si/test/high_school_biology/test/132
+gmmlu_si/test/high_school_biology/test/153
+gmmlu_si/test/high_school_biology/test/159
+gmmlu_si/test/high_school_biology/test/163
+gmmlu_si/test/high_school_biology/test/166
+gmmlu_si/test/high_school_biology/test/179
+gmmlu_si/test/high_school_biology/test/189
+gmmlu_si/test/high_school_biology/test/196
+gmmlu_si/test/high_school_biology/test/199
+gmmlu_si/test/high_school_biology/test/205
+gmmlu_si/test/high_school_biology/test/209
+gmmlu_si/test/high_school_biology/test/215
+gmmlu_si/test/high_school_biology/test/220
+gmmlu_si/test/high_school_biology/test/23
+gmmlu_si/test/high_school_biology/test/230
+gmmlu_si/test/high_school_biology/test/233
+gmmlu_si/test/high_school_biology/test/242
+gmmlu_si/test/high_school_biology/test/255
+gmmlu_si/test/high_school_biology/test/262
+gmmlu_si/test/high_school_biology/test/265
+gmmlu_si/test/high_school_biology/test/268
+gmmlu_si/test/high_school_biology/test/27
+gmmlu_si/test/high_school_biology/test/281
+gmmlu_si/test/high_school_biology/test/286
+gmmlu_si/test/high_school_biology/test/29
+gmmlu_si/test/high_school_biology/test/295
+gmmlu_si/test/high_school_biology/test/35
+gmmlu_si/test/high_school_biology/test/43
+gmmlu_si/test/high_school_biology/test/50
+gmmlu_si/test/high_school_biology/test/66
+gmmlu_si/test/high_school_biology/test/70
+gmmlu_si/test/high_school_biology/test/72
+gmmlu_si/test/high_school_biology/test/83
+gmmlu_si/test/high_school_biology/test/89
+gmmlu_si/test/high_school_biology/test/92
+gmmlu_si/test/high_school_biology/test/93
+gmmlu_si/test/high_school_biology/test/96
+gmmlu_si/test/high_school_biology/test/97
+gmmlu_si/test/high_school_biology/test/99
+gmmlu_si/test/high_school_chemistry/test/0
+gmmlu_si/test/high_school_chemistry/test/137
+gmmlu_si/test/high_school_chemistry/test/138
+gmmlu_si/test/high_school_chemistry/test/139
+gmmlu_si/test/high_school_chemistry/test/140
+gmmlu_si/test/high_school_chemistry/test/142
+gmmlu_si/test/high_school_chemistry/test/153
+gmmlu_si/test/high_school_chemistry/test/160
+gmmlu_si/test/high_school_chemistry/test/165
+gmmlu_si/test/high_school_chemistry/test/172
+gmmlu_si/test/high_school_chemistry/test/18
+gmmlu_si/test/high_school_chemistry/test/181
+gmmlu_si/test/high_school_chemistry/test/183
+gmmlu_si/test/high_school_chemistry/test/192
+gmmlu_si/test/high_school_chemistry/test/20
+gmmlu_si/test/high_school_chemistry/test/202
+gmmlu_si/test/high_school_chemistry/test/28
+gmmlu_si/test/high_school_chemistry/test/38
+gmmlu_si/test/high_school_chemistry/test/4
+gmmlu_si/test/high_school_chemistry/test/41
+gmmlu_si/test/high_school_chemistry/test/51
+gmmlu_si/test/high_school_chemistry/test/54
+gmmlu_si/test/high_school_chemistry/test/64
+gmmlu_si/test/high_school_chemistry/test/76
+gmmlu_si/test/high_school_chemistry/test/77
+gmmlu_si/test/high_school_chemistry/test/78
+gmmlu_si/test/high_school_chemistry/test/80
+gmmlu_si/test/high_school_chemistry/test/84
+gmmlu_si/test/high_school_chemistry/test/93
+gmmlu_si/test/high_school_computer_science/test/20
+gmmlu_si/test/high_school_computer_science/test/51
+gmmlu_si/test/high_school_computer_science/test/55
+gmmlu_si/test/high_school_computer_science/test/6
+gmmlu_si/test/high_school_computer_science/test/64
+gmmlu_si/test/high_school_computer_science/test/68
+gmmlu_si/test/high_school_computer_science/test/7
+gmmlu_si/test/high_school_computer_science/test/76
+gmmlu_si/test/high_school_computer_science/test/80
+gmmlu_si/test/high_school_computer_science/test/87
+gmmlu_si/test/high_school_computer_science/test/91
+gmmlu_si/test/high_school_computer_science/test/92
+gmmlu_si/test/high_school_computer_science/test/93
+gmmlu_si/test/high_school_computer_science/test/94
+gmmlu_si/test/high_school_european_history/test/114
+gmmlu_si/test/high_school_european_history/test/12
+gmmlu_si/test/high_school_european_history/test/122
+gmmlu_si/test/high_school_european_history/test/127
+gmmlu_si/test/high_school_european_history/test/128
+gmmlu_si/test/high_school_european_history/test/141
+gmmlu_si/test/high_school_european_history/test/153
+gmmlu_si/test/high_school_european_history/test/157
+gmmlu_si/test/high_school_european_history/test/18
+gmmlu_si/test/high_school_european_history/test/23
+gmmlu_si/test/high_school_european_history/test/27
+gmmlu_si/test/high_school_european_history/test/31
+gmmlu_si/test/high_school_european_history/test/40
+gmmlu_si/test/high_school_european_history/test/43
+gmmlu_si/test/high_school_european_history/test/52
+gmmlu_si/test/high_school_european_history/test/58
+gmmlu_si/test/high_school_european_history/test/59
+gmmlu_si/test/high_school_european_history/test/6
+gmmlu_si/test/high_school_european_history/test/62
+gmmlu_si/test/high_school_european_history/test/70
+gmmlu_si/test/high_school_european_history/test/71
+gmmlu_si/test/high_school_european_history/test/76
+gmmlu_si/test/high_school_european_history/test/77
+gmmlu_si/test/high_school_european_history/test/79
+gmmlu_si/test/high_school_geography/test/104
+gmmlu_si/test/high_school_geography/test/108
+gmmlu_si/test/high_school_geography/test/110
+gmmlu_si/test/high_school_geography/test/115
+gmmlu_si/test/high_school_geography/test/117
+gmmlu_si/test/high_school_geography/test/120
+gmmlu_si/test/high_school_geography/test/122
+gmmlu_si/test/high_school_geography/test/134
+gmmlu_si/test/high_school_geography/test/138
+gmmlu_si/test/high_school_geography/test/139
+gmmlu_si/test/high_school_geography/test/143
+gmmlu_si/test/high_school_geography/test/144
+gmmlu_si/test/high_school_geography/test/158
+gmmlu_si/test/high_school_geography/test/159
+gmmlu_si/test/high_school_geography/test/160
+gmmlu_si/test/high_school_geography/test/170
+gmmlu_si/test/high_school_geography/test/172
+gmmlu_si/test/high_school_geography/test/175
+gmmlu_si/test/high_school_geography/test/177
+gmmlu_si/test/high_school_geography/test/185
+gmmlu_si/test/high_school_geography/test/196
+gmmlu_si/test/high_school_geography/test/27
+gmmlu_si/test/high_school_geography/test/45
+gmmlu_si/test/high_school_geography/test/6
+gmmlu_si/test/high_school_geography/test/60
+gmmlu_si/test/high_school_geography/test/73
+gmmlu_si/test/high_school_geography/test/77
+gmmlu_si/test/high_school_geography/test/80
+gmmlu_si/test/high_school_government_and_politics/test/10
+gmmlu_si/test/high_school_government_and_politics/test/133
+gmmlu_si/test/high_school_government_and_politics/test/136
+gmmlu_si/test/high_school_government_and_politics/test/142
+gmmlu_si/test/high_school_government_and_politics/test/155
+gmmlu_si/test/high_school_government_and_politics/test/163
+gmmlu_si/test/high_school_government_and_politics/test/165
+gmmlu_si/test/high_school_government_and_politics/test/169
+gmmlu_si/test/high_school_government_and_politics/test/171
+gmmlu_si/test/high_school_government_and_politics/test/176
+gmmlu_si/test/high_school_government_and_politics/test/185
+gmmlu_si/test/high_school_government_and_politics/test/19
+gmmlu_si/test/high_school_government_and_politics/test/20
+gmmlu_si/test/high_school_government_and_politics/test/25
+gmmlu_si/test/high_school_government_and_politics/test/27
+gmmlu_si/test/high_school_government_and_politics/test/29
+gmmlu_si/test/high_school_government_and_politics/test/40
+gmmlu_si/test/high_school_government_and_politics/test/45
+gmmlu_si/test/high_school_government_and_politics/test/58
+gmmlu_si/test/high_school_government_and_politics/test/61
+gmmlu_si/test/high_school_government_and_politics/test/62
+gmmlu_si/test/high_school_government_and_politics/test/65
+gmmlu_si/test/high_school_government_and_politics/test/66
+gmmlu_si/test/high_school_government_and_politics/test/82
+gmmlu_si/test/high_school_government_and_politics/test/85
+gmmlu_si/test/high_school_government_and_politics/test/89
+gmmlu_si/test/high_school_government_and_politics/test/9
+gmmlu_si/test/high_school_macroeconomics/test/109
+gmmlu_si/test/high_school_macroeconomics/test/11
+gmmlu_si/test/high_school_macroeconomics/test/113
+gmmlu_si/test/high_school_macroeconomics/test/127
+gmmlu_si/test/high_school_macroeconomics/test/13
+gmmlu_si/test/high_school_macroeconomics/test/130
+gmmlu_si/test/high_school_macroeconomics/test/161
+gmmlu_si/test/high_school_macroeconomics/test/164
+gmmlu_si/test/high_school_macroeconomics/test/166
+gmmlu_si/test/high_school_macroeconomics/test/170
+gmmlu_si/test/high_school_macroeconomics/test/173
+gmmlu_si/test/high_school_macroeconomics/test/18
+gmmlu_si/test/high_school_macroeconomics/test/181
+gmmlu_si/test/high_school_macroeconomics/test/193
+gmmlu_si/test/high_school_macroeconomics/test/205
+gmmlu_si/test/high_school_macroeconomics/test/207
+gmmlu_si/test/high_school_macroeconomics/test/210
+gmmlu_si/test/high_school_macroeconomics/test/211
+gmmlu_si/test/high_school_macroeconomics/test/217
+gmmlu_si/test/high_school_macroeconomics/test/221
+gmmlu_si/test/high_school_macroeconomics/test/23
+gmmlu_si/test/high_school_macroeconomics/test/237
+gmmlu_si/test/high_school_macroeconomics/test/238
+gmmlu_si/test/high_school_macroeconomics/test/243
+gmmlu_si/test/high_school_macroeconomics/test/249
+gmmlu_si/test/high_school_macroeconomics/test/261
+gmmlu_si/test/high_school_macroeconomics/test/271
+gmmlu_si/test/high_school_macroeconomics/test/273
+gmmlu_si/test/high_school_macroeconomics/test/274
+gmmlu_si/test/high_school_macroeconomics/test/279
+gmmlu_si/test/high_school_macroeconomics/test/285
+gmmlu_si/test/high_school_macroeconomics/test/288
+gmmlu_si/test/high_school_macroeconomics/test/296
+gmmlu_si/test/high_school_macroeconomics/test/297
+gmmlu_si/test/high_school_macroeconomics/test/299
+gmmlu_si/test/high_school_macroeconomics/test/302
+gmmlu_si/test/high_school_macroeconomics/test/306
+gmmlu_si/test/high_school_macroeconomics/test/321
+gmmlu_si/test/high_school_macroeconomics/test/325
+gmmlu_si/test/high_school_macroeconomics/test/329
+gmmlu_si/test/high_school_macroeconomics/test/334
+gmmlu_si/test/high_school_macroeconomics/test/337
+gmmlu_si/test/high_school_macroeconomics/test/341
+gmmlu_si/test/high_school_macroeconomics/test/349
+gmmlu_si/test/high_school_macroeconomics/test/35
+gmmlu_si/test/high_school_macroeconomics/test/351
+gmmlu_si/test/high_school_macroeconomics/test/363
+gmmlu_si/test/high_school_macroeconomics/test/364
+gmmlu_si/test/high_school_macroeconomics/test/388
+gmmlu_si/test/high_school_macroeconomics/test/45
+gmmlu_si/test/high_school_macroeconomics/test/51
+gmmlu_si/test/high_school_macroeconomics/test/62
+gmmlu_si/test/high_school_macroeconomics/test/64
+gmmlu_si/test/high_school_macroeconomics/test/67
+gmmlu_si/test/high_school_macroeconomics/test/77
+gmmlu_si/test/high_school_macroeconomics/test/79
+gmmlu_si/test/high_school_mathematics/test/113
+gmmlu_si/test/high_school_mathematics/test/128
+gmmlu_si/test/high_school_mathematics/test/131
+gmmlu_si/test/high_school_mathematics/test/133
+gmmlu_si/test/high_school_mathematics/test/141
+gmmlu_si/test/high_school_mathematics/test/144
+gmmlu_si/test/high_school_mathematics/test/147
+gmmlu_si/test/high_school_mathematics/test/148
+gmmlu_si/test/high_school_mathematics/test/155
+gmmlu_si/test/high_school_mathematics/test/16
+gmmlu_si/test/high_school_mathematics/test/161
+gmmlu_si/test/high_school_mathematics/test/175
+gmmlu_si/test/high_school_mathematics/test/177
+gmmlu_si/test/high_school_mathematics/test/189
+gmmlu_si/test/high_school_mathematics/test/203
+gmmlu_si/test/high_school_mathematics/test/218
+gmmlu_si/test/high_school_mathematics/test/220
+gmmlu_si/test/high_school_mathematics/test/226
+gmmlu_si/test/high_school_mathematics/test/227
+gmmlu_si/test/high_school_mathematics/test/237
+gmmlu_si/test/high_school_mathematics/test/243
+gmmlu_si/test/high_school_mathematics/test/252
+gmmlu_si/test/high_school_mathematics/test/254
+gmmlu_si/test/high_school_mathematics/test/265
+gmmlu_si/test/high_school_mathematics/test/3
+gmmlu_si/test/high_school_mathematics/test/43
+gmmlu_si/test/high_school_mathematics/test/46
+gmmlu_si/test/high_school_mathematics/test/47
+gmmlu_si/test/high_school_mathematics/test/54
+gmmlu_si/test/high_school_mathematics/test/58
+gmmlu_si/test/high_school_mathematics/test/6
+gmmlu_si/test/high_school_mathematics/test/68
+gmmlu_si/test/high_school_mathematics/test/73
+gmmlu_si/test/high_school_mathematics/test/75
+gmmlu_si/test/high_school_mathematics/test/77
+gmmlu_si/test/high_school_mathematics/test/81
+gmmlu_si/test/high_school_mathematics/test/90
+gmmlu_si/test/high_school_mathematics/test/95
+gmmlu_si/test/high_school_microeconomics/test/0
+gmmlu_si/test/high_school_microeconomics/test/10
+gmmlu_si/test/high_school_microeconomics/test/101
+gmmlu_si/test/high_school_microeconomics/test/141
+gmmlu_si/test/high_school_microeconomics/test/15
+gmmlu_si/test/high_school_microeconomics/test/152
+gmmlu_si/test/high_school_microeconomics/test/155
+gmmlu_si/test/high_school_microeconomics/test/174
+gmmlu_si/test/high_school_microeconomics/test/178
+gmmlu_si/test/high_school_microeconomics/test/181
+gmmlu_si/test/high_school_microeconomics/test/193
+gmmlu_si/test/high_school_microeconomics/test/194
+gmmlu_si/test/high_school_microeconomics/test/195
+gmmlu_si/test/high_school_microeconomics/test/196
+gmmlu_si/test/high_school_microeconomics/test/2
+gmmlu_si/test/high_school_microeconomics/test/212
+gmmlu_si/test/high_school_microeconomics/test/217
+gmmlu_si/test/high_school_microeconomics/test/225
+gmmlu_si/test/high_school_microeconomics/test/230
+gmmlu_si/test/high_school_microeconomics/test/232
+gmmlu_si/test/high_school_microeconomics/test/233
+gmmlu_si/test/high_school_microeconomics/test/27
+gmmlu_si/test/high_school_microeconomics/test/32
+gmmlu_si/test/high_school_microeconomics/test/34
+gmmlu_si/test/high_school_microeconomics/test/40
+gmmlu_si/test/high_school_microeconomics/test/41
+gmmlu_si/test/high_school_microeconomics/test/59
+gmmlu_si/test/high_school_microeconomics/test/60
+gmmlu_si/test/high_school_microeconomics/test/62
+gmmlu_si/test/high_school_microeconomics/test/73
+gmmlu_si/test/high_school_microeconomics/test/79
+gmmlu_si/test/high_school_microeconomics/test/88
+gmmlu_si/test/high_school_microeconomics/test/93
+gmmlu_si/test/high_school_microeconomics/test/95
+gmmlu_si/test/high_school_physics/test/101
+gmmlu_si/test/high_school_physics/test/102
+gmmlu_si/test/high_school_physics/test/107
+gmmlu_si/test/high_school_physics/test/114
+gmmlu_si/test/high_school_physics/test/123
+gmmlu_si/test/high_school_physics/test/126
+gmmlu_si/test/high_school_physics/test/138
+gmmlu_si/test/high_school_physics/test/139
+gmmlu_si/test/high_school_physics/test/146
+gmmlu_si/test/high_school_physics/test/19
+gmmlu_si/test/high_school_physics/test/3
+gmmlu_si/test/high_school_physics/test/33
+gmmlu_si/test/high_school_physics/test/48
+gmmlu_si/test/high_school_physics/test/53
+gmmlu_si/test/high_school_physics/test/54
+gmmlu_si/test/high_school_physics/test/58
+gmmlu_si/test/high_school_physics/test/6
+gmmlu_si/test/high_school_physics/test/62
+gmmlu_si/test/high_school_physics/test/67
+gmmlu_si/test/high_school_physics/test/71
+gmmlu_si/test/high_school_physics/test/75
+gmmlu_si/test/high_school_physics/test/93
+gmmlu_si/test/high_school_psychology/test/106
+gmmlu_si/test/high_school_psychology/test/107
+gmmlu_si/test/high_school_psychology/test/114
+gmmlu_si/test/high_school_psychology/test/124
+gmmlu_si/test/high_school_psychology/test/126
+gmmlu_si/test/high_school_psychology/test/128
+gmmlu_si/test/high_school_psychology/test/131
+gmmlu_si/test/high_school_psychology/test/132
+gmmlu_si/test/high_school_psychology/test/14
+gmmlu_si/test/high_school_psychology/test/153
+gmmlu_si/test/high_school_psychology/test/155
+gmmlu_si/test/high_school_psychology/test/156
+gmmlu_si/test/high_school_psychology/test/157
+gmmlu_si/test/high_school_psychology/test/165
+gmmlu_si/test/high_school_psychology/test/169
+gmmlu_si/test/high_school_psychology/test/171
+gmmlu_si/test/high_school_psychology/test/180
+gmmlu_si/test/high_school_psychology/test/185
+gmmlu_si/test/high_school_psychology/test/190
+gmmlu_si/test/high_school_psychology/test/197
+gmmlu_si/test/high_school_psychology/test/213
+gmmlu_si/test/high_school_psychology/test/215
+gmmlu_si/test/high_school_psychology/test/247
+gmmlu_si/test/high_school_psychology/test/248
+gmmlu_si/test/high_school_psychology/test/255
+gmmlu_si/test/high_school_psychology/test/26
+gmmlu_si/test/high_school_psychology/test/266
+gmmlu_si/test/high_school_psychology/test/267
+gmmlu_si/test/high_school_psychology/test/269
+gmmlu_si/test/high_school_psychology/test/273
+gmmlu_si/test/high_school_psychology/test/288
+gmmlu_si/test/high_school_psychology/test/31
+gmmlu_si/test/high_school_psychology/test/310
+gmmlu_si/test/high_school_psychology/test/322
+gmmlu_si/test/high_school_psychology/test/34
+gmmlu_si/test/high_school_psychology/test/340
+gmmlu_si/test/high_school_psychology/test/345
+gmmlu_si/test/high_school_psychology/test/351
+gmmlu_si/test/high_school_psychology/test/362
+gmmlu_si/test/high_school_psychology/test/364
+gmmlu_si/test/high_school_psychology/test/368
+gmmlu_si/test/high_school_psychology/test/37
+gmmlu_si/test/high_school_psychology/test/381
+gmmlu_si/test/high_school_psychology/test/386
+gmmlu_si/test/high_school_psychology/test/391
+gmmlu_si/test/high_school_psychology/test/399
+gmmlu_si/test/high_school_psychology/test/402
+gmmlu_si/test/high_school_psychology/test/403
+gmmlu_si/test/high_school_psychology/test/407
+gmmlu_si/test/high_school_psychology/test/410
+gmmlu_si/test/high_school_psychology/test/416
+gmmlu_si/test/high_school_psychology/test/423
+gmmlu_si/test/high_school_psychology/test/449
+gmmlu_si/test/high_school_psychology/test/463
+gmmlu_si/test/high_school_psychology/test/472
+gmmlu_si/test/high_school_psychology/test/475
+gmmlu_si/test/high_school_psychology/test/478
+gmmlu_si/test/high_school_psychology/test/483
+gmmlu_si/test/high_school_psychology/test/485
+gmmlu_si/test/high_school_psychology/test/49
+gmmlu_si/test/high_school_psychology/test/5
+gmmlu_si/test/high_school_psychology/test/505
+gmmlu_si/test/high_school_psychology/test/509
+gmmlu_si/test/high_school_psychology/test/513
+gmmlu_si/test/high_school_psychology/test/520
+gmmlu_si/test/high_school_psychology/test/541
+gmmlu_si/test/high_school_psychology/test/542
+gmmlu_si/test/high_school_psychology/test/6
+gmmlu_si/test/high_school_psychology/test/69
+gmmlu_si/test/high_school_psychology/test/70
+gmmlu_si/test/high_school_psychology/test/77
+gmmlu_si/test/high_school_psychology/test/80
+gmmlu_si/test/high_school_psychology/test/81
+gmmlu_si/test/high_school_psychology/test/85
+gmmlu_si/test/high_school_psychology/test/86
+gmmlu_si/test/high_school_psychology/test/9
+gmmlu_si/test/high_school_psychology/test/91
+gmmlu_si/test/high_school_psychology/test/95
+gmmlu_si/test/high_school_statistics/test/105
+gmmlu_si/test/high_school_statistics/test/107
+gmmlu_si/test/high_school_statistics/test/118
+gmmlu_si/test/high_school_statistics/test/122
+gmmlu_si/test/high_school_statistics/test/139
+gmmlu_si/test/high_school_statistics/test/147
+gmmlu_si/test/high_school_statistics/test/148
+gmmlu_si/test/high_school_statistics/test/16
+gmmlu_si/test/high_school_statistics/test/17
+gmmlu_si/test/high_school_statistics/test/176
+gmmlu_si/test/high_school_statistics/test/177
+gmmlu_si/test/high_school_statistics/test/190
+gmmlu_si/test/high_school_statistics/test/200
+gmmlu_si/test/high_school_statistics/test/213
+gmmlu_si/test/high_school_statistics/test/23
+gmmlu_si/test/high_school_statistics/test/29
+gmmlu_si/test/high_school_statistics/test/31
+gmmlu_si/test/high_school_statistics/test/36
+gmmlu_si/test/high_school_statistics/test/38
+gmmlu_si/test/high_school_statistics/test/40
+gmmlu_si/test/high_school_statistics/test/45
+gmmlu_si/test/high_school_statistics/test/47
+gmmlu_si/test/high_school_statistics/test/52
+gmmlu_si/test/high_school_statistics/test/57
+gmmlu_si/test/high_school_statistics/test/58
+gmmlu_si/test/high_school_statistics/test/6
+gmmlu_si/test/high_school_statistics/test/66
+gmmlu_si/test/high_school_statistics/test/79
+gmmlu_si/test/high_school_statistics/test/84
+gmmlu_si/test/high_school_statistics/test/9
+gmmlu_si/test/high_school_statistics/test/90
+gmmlu_si/test/high_school_us_history/test/101
+gmmlu_si/test/high_school_us_history/test/104
+gmmlu_si/test/high_school_us_history/test/113
+gmmlu_si/test/high_school_us_history/test/116
+gmmlu_si/test/high_school_us_history/test/117
+gmmlu_si/test/high_school_us_history/test/118
+gmmlu_si/test/high_school_us_history/test/120
+gmmlu_si/test/high_school_us_history/test/148
+gmmlu_si/test/high_school_us_history/test/154
+gmmlu_si/test/high_school_us_history/test/167
+gmmlu_si/test/high_school_us_history/test/175
+gmmlu_si/test/high_school_us_history/test/178
+gmmlu_si/test/high_school_us_history/test/179
+gmmlu_si/test/high_school_us_history/test/18
+gmmlu_si/test/high_school_us_history/test/188
+gmmlu_si/test/high_school_us_history/test/189
+gmmlu_si/test/high_school_us_history/test/198
+gmmlu_si/test/high_school_us_history/test/202
+gmmlu_si/test/high_school_us_history/test/43
+gmmlu_si/test/high_school_us_history/test/56
+gmmlu_si/test/high_school_us_history/test/58
+gmmlu_si/test/high_school_us_history/test/60
+gmmlu_si/test/high_school_us_history/test/64
+gmmlu_si/test/high_school_us_history/test/72
+gmmlu_si/test/high_school_us_history/test/89
+gmmlu_si/test/high_school_us_history/test/93
+gmmlu_si/test/high_school_us_history/test/94
+gmmlu_si/test/high_school_us_history/test/95
+gmmlu_si/test/high_school_us_history/test/96
+gmmlu_si/test/high_school_world_history/test/101
+gmmlu_si/test/high_school_world_history/test/104
+gmmlu_si/test/high_school_world_history/test/105
+gmmlu_si/test/high_school_world_history/test/111
+gmmlu_si/test/high_school_world_history/test/117
+gmmlu_si/test/high_school_world_history/test/12
+gmmlu_si/test/high_school_world_history/test/127
+gmmlu_si/test/high_school_world_history/test/128
+gmmlu_si/test/high_school_world_history/test/129
+gmmlu_si/test/high_school_world_history/test/134
+gmmlu_si/test/high_school_world_history/test/142
+gmmlu_si/test/high_school_world_history/test/16
+gmmlu_si/test/high_school_world_history/test/170
+gmmlu_si/test/high_school_world_history/test/176
+gmmlu_si/test/high_school_world_history/test/18
+gmmlu_si/test/high_school_world_history/test/180
+gmmlu_si/test/high_school_world_history/test/181
+gmmlu_si/test/high_school_world_history/test/195
+gmmlu_si/test/high_school_world_history/test/199
+gmmlu_si/test/high_school_world_history/test/201
+gmmlu_si/test/high_school_world_history/test/211
+gmmlu_si/test/high_school_world_history/test/213
+gmmlu_si/test/high_school_world_history/test/227
+gmmlu_si/test/high_school_world_history/test/229
+gmmlu_si/test/high_school_world_history/test/236
+gmmlu_si/test/high_school_world_history/test/27
+gmmlu_si/test/high_school_world_history/test/28
+gmmlu_si/test/high_school_world_history/test/40
+gmmlu_si/test/high_school_world_history/test/46
+gmmlu_si/test/high_school_world_history/test/48
+gmmlu_si/test/high_school_world_history/test/57
+gmmlu_si/test/high_school_world_history/test/59
+gmmlu_si/test/high_school_world_history/test/68
+gmmlu_si/test/high_school_world_history/test/97
+gmmlu_si/test/human_aging/test/128
+gmmlu_si/test/human_aging/test/13
+gmmlu_si/test/human_aging/test/130
+gmmlu_si/test/human_aging/test/136
+gmmlu_si/test/human_aging/test/137
+gmmlu_si/test/human_aging/test/150
+gmmlu_si/test/human_aging/test/158
+gmmlu_si/test/human_aging/test/161
+gmmlu_si/test/human_aging/test/169
+gmmlu_si/test/human_aging/test/180
+gmmlu_si/test/human_aging/test/186
+gmmlu_si/test/human_aging/test/187
+gmmlu_si/test/human_aging/test/196
+gmmlu_si/test/human_aging/test/198
+gmmlu_si/test/human_aging/test/207
+gmmlu_si/test/human_aging/test/212
+gmmlu_si/test/human_aging/test/213
+gmmlu_si/test/human_aging/test/222
+gmmlu_si/test/human_aging/test/29
+gmmlu_si/test/human_aging/test/30
+gmmlu_si/test/human_aging/test/41
+gmmlu_si/test/human_aging/test/42
+gmmlu_si/test/human_aging/test/49
+gmmlu_si/test/human_aging/test/50
+gmmlu_si/test/human_aging/test/51
+gmmlu_si/test/human_aging/test/59
+gmmlu_si/test/human_aging/test/6
+gmmlu_si/test/human_aging/test/61
+gmmlu_si/test/human_aging/test/63
+gmmlu_si/test/human_aging/test/76
+gmmlu_si/test/human_aging/test/95
+gmmlu_si/test/human_aging/test/96
+gmmlu_si/test/human_sexuality/test/10
+gmmlu_si/test/human_sexuality/test/106
+gmmlu_si/test/human_sexuality/test/109
+gmmlu_si/test/human_sexuality/test/110
+gmmlu_si/test/human_sexuality/test/16
+gmmlu_si/test/human_sexuality/test/19
+gmmlu_si/test/human_sexuality/test/24
+gmmlu_si/test/human_sexuality/test/26
+gmmlu_si/test/human_sexuality/test/38
+gmmlu_si/test/human_sexuality/test/40
+gmmlu_si/test/human_sexuality/test/49
+gmmlu_si/test/human_sexuality/test/51
+gmmlu_si/test/human_sexuality/test/52
+gmmlu_si/test/human_sexuality/test/61
+gmmlu_si/test/human_sexuality/test/66
+gmmlu_si/test/human_sexuality/test/78
+gmmlu_si/test/human_sexuality/test/86
+gmmlu_si/test/human_sexuality/test/90
+gmmlu_si/test/human_sexuality/test/98
+gmmlu_si/test/international_law/test/104
+gmmlu_si/test/international_law/test/20
+gmmlu_si/test/international_law/test/22
+gmmlu_si/test/international_law/test/26
+gmmlu_si/test/international_law/test/34
+gmmlu_si/test/international_law/test/38
+gmmlu_si/test/international_law/test/41
+gmmlu_si/test/international_law/test/5
+gmmlu_si/test/international_law/test/54
+gmmlu_si/test/international_law/test/65
+gmmlu_si/test/international_law/test/68
+gmmlu_si/test/international_law/test/82
+gmmlu_si/test/international_law/test/85
+gmmlu_si/test/international_law/test/91
+gmmlu_si/test/international_law/test/95
+gmmlu_si/test/international_law/test/97
+gmmlu_si/test/international_law/test/98
+gmmlu_si/test/jurisprudence/test/12
+gmmlu_si/test/jurisprudence/test/23
+gmmlu_si/test/jurisprudence/test/30
+gmmlu_si/test/jurisprudence/test/35
+gmmlu_si/test/jurisprudence/test/38
+gmmlu_si/test/jurisprudence/test/4
+gmmlu_si/test/jurisprudence/test/43
+gmmlu_si/test/jurisprudence/test/46
+gmmlu_si/test/jurisprudence/test/48
+gmmlu_si/test/jurisprudence/test/50
+gmmlu_si/test/jurisprudence/test/71
+gmmlu_si/test/jurisprudence/test/72
+gmmlu_si/test/jurisprudence/test/73
+gmmlu_si/test/jurisprudence/test/74
+gmmlu_si/test/jurisprudence/test/93
+gmmlu_si/test/logical_fallacies/test/111
+gmmlu_si/test/logical_fallacies/test/128
+gmmlu_si/test/logical_fallacies/test/137
+gmmlu_si/test/logical_fallacies/test/14
+gmmlu_si/test/logical_fallacies/test/144
+gmmlu_si/test/logical_fallacies/test/15
+gmmlu_si/test/logical_fallacies/test/152
+gmmlu_si/test/logical_fallacies/test/157
+gmmlu_si/test/logical_fallacies/test/17
+gmmlu_si/test/logical_fallacies/test/21
+gmmlu_si/test/logical_fallacies/test/34
+gmmlu_si/test/logical_fallacies/test/38
+gmmlu_si/test/logical_fallacies/test/39
+gmmlu_si/test/logical_fallacies/test/46
+gmmlu_si/test/logical_fallacies/test/47
+gmmlu_si/test/logical_fallacies/test/57
+gmmlu_si/test/logical_fallacies/test/7
+gmmlu_si/test/logical_fallacies/test/83
+gmmlu_si/test/logical_fallacies/test/88
+gmmlu_si/test/logical_fallacies/test/92
+gmmlu_si/test/logical_fallacies/test/93
+gmmlu_si/test/logical_fallacies/test/94
+gmmlu_si/test/logical_fallacies/test/97
+gmmlu_si/test/machine_learning/test/111
+gmmlu_si/test/machine_learning/test/17
+gmmlu_si/test/machine_learning/test/18
+gmmlu_si/test/machine_learning/test/27
+gmmlu_si/test/machine_learning/test/34
+gmmlu_si/test/machine_learning/test/35
+gmmlu_si/test/machine_learning/test/44
+gmmlu_si/test/machine_learning/test/49
+gmmlu_si/test/machine_learning/test/5
+gmmlu_si/test/machine_learning/test/62
+gmmlu_si/test/machine_learning/test/70
+gmmlu_si/test/machine_learning/test/74
+gmmlu_si/test/machine_learning/test/81
+gmmlu_si/test/machine_learning/test/84
+gmmlu_si/test/machine_learning/test/91
+gmmlu_si/test/machine_learning/test/93
+gmmlu_si/test/management/test/11
+gmmlu_si/test/management/test/12
+gmmlu_si/test/management/test/16
+gmmlu_si/test/management/test/25
+gmmlu_si/test/management/test/36
+gmmlu_si/test/management/test/38
+gmmlu_si/test/management/test/40
+gmmlu_si/test/management/test/58
+gmmlu_si/test/management/test/59
+gmmlu_si/test/management/test/62
+gmmlu_si/test/management/test/66
+gmmlu_si/test/management/test/8
+gmmlu_si/test/management/test/83
+gmmlu_si/test/management/test/86
+gmmlu_si/test/management/test/94
+gmmlu_si/test/marketing/test/11
+gmmlu_si/test/marketing/test/114
+gmmlu_si/test/marketing/test/12
+gmmlu_si/test/marketing/test/124
+gmmlu_si/test/marketing/test/128
+gmmlu_si/test/marketing/test/133
+gmmlu_si/test/marketing/test/134
+gmmlu_si/test/marketing/test/14
+gmmlu_si/test/marketing/test/141
+gmmlu_si/test/marketing/test/147
+gmmlu_si/test/marketing/test/153
+gmmlu_si/test/marketing/test/165
+gmmlu_si/test/marketing/test/184
+gmmlu_si/test/marketing/test/186
+gmmlu_si/test/marketing/test/189
+gmmlu_si/test/marketing/test/193
+gmmlu_si/test/marketing/test/199
+gmmlu_si/test/marketing/test/206
+gmmlu_si/test/marketing/test/208
+gmmlu_si/test/marketing/test/221
+gmmlu_si/test/marketing/test/230
+gmmlu_si/test/marketing/test/231
+gmmlu_si/test/marketing/test/28
+gmmlu_si/test/marketing/test/30
+gmmlu_si/test/marketing/test/61
+gmmlu_si/test/marketing/test/65
+gmmlu_si/test/marketing/test/67
+gmmlu_si/test/marketing/test/8
+gmmlu_si/test/marketing/test/83
+gmmlu_si/test/marketing/test/84
+gmmlu_si/test/marketing/test/88
+gmmlu_si/test/marketing/test/89
+gmmlu_si/test/marketing/test/93
+gmmlu_si/test/medical_genetics/test/10
+gmmlu_si/test/medical_genetics/test/18
+gmmlu_si/test/medical_genetics/test/2
+gmmlu_si/test/medical_genetics/test/20
+gmmlu_si/test/medical_genetics/test/22
+gmmlu_si/test/medical_genetics/test/23
+gmmlu_si/test/medical_genetics/test/24
+gmmlu_si/test/medical_genetics/test/25
+gmmlu_si/test/medical_genetics/test/32
+gmmlu_si/test/medical_genetics/test/45
+gmmlu_si/test/medical_genetics/test/51
+gmmlu_si/test/medical_genetics/test/63
+gmmlu_si/test/medical_genetics/test/73
+gmmlu_si/test/medical_genetics/test/99
+gmmlu_si/test/miscellaneous/test/0
+gmmlu_si/test/miscellaneous/test/103
+gmmlu_si/test/miscellaneous/test/115
+gmmlu_si/test/miscellaneous/test/117
+gmmlu_si/test/miscellaneous/test/123
+gmmlu_si/test/miscellaneous/test/132
+gmmlu_si/test/miscellaneous/test/134
+gmmlu_si/test/miscellaneous/test/135
+gmmlu_si/test/miscellaneous/test/137
+gmmlu_si/test/miscellaneous/test/142
+gmmlu_si/test/miscellaneous/test/153
+gmmlu_si/test/miscellaneous/test/156
+gmmlu_si/test/miscellaneous/test/157
+gmmlu_si/test/miscellaneous/test/168
+gmmlu_si/test/miscellaneous/test/172
+gmmlu_si/test/miscellaneous/test/196
+gmmlu_si/test/miscellaneous/test/209
+gmmlu_si/test/miscellaneous/test/210
+gmmlu_si/test/miscellaneous/test/223
+gmmlu_si/test/miscellaneous/test/251
+gmmlu_si/test/miscellaneous/test/269
+gmmlu_si/test/miscellaneous/test/287
+gmmlu_si/test/miscellaneous/test/290
+gmmlu_si/test/miscellaneous/test/293
+gmmlu_si/test/miscellaneous/test/300
+gmmlu_si/test/miscellaneous/test/302
+gmmlu_si/test/miscellaneous/test/31
+gmmlu_si/test/miscellaneous/test/311
+gmmlu_si/test/miscellaneous/test/333
+gmmlu_si/test/miscellaneous/test/334
+gmmlu_si/test/miscellaneous/test/345
+gmmlu_si/test/miscellaneous/test/354
+gmmlu_si/test/miscellaneous/test/367
+gmmlu_si/test/miscellaneous/test/37
+gmmlu_si/test/miscellaneous/test/373
+gmmlu_si/test/miscellaneous/test/375
+gmmlu_si/test/miscellaneous/test/38
+gmmlu_si/test/miscellaneous/test/388
+gmmlu_si/test/miscellaneous/test/390
+gmmlu_si/test/miscellaneous/test/397
+gmmlu_si/test/miscellaneous/test/399
+gmmlu_si/test/miscellaneous/test/400
+gmmlu_si/test/miscellaneous/test/404
+gmmlu_si/test/miscellaneous/test/413
+gmmlu_si/test/miscellaneous/test/414
+gmmlu_si/test/miscellaneous/test/43
+gmmlu_si/test/miscellaneous/test/430
+gmmlu_si/test/miscellaneous/test/433
+gmmlu_si/test/miscellaneous/test/441
+gmmlu_si/test/miscellaneous/test/443
+gmmlu_si/test/miscellaneous/test/445
+gmmlu_si/test/miscellaneous/test/447
+gmmlu_si/test/miscellaneous/test/453
+gmmlu_si/test/miscellaneous/test/460
+gmmlu_si/test/miscellaneous/test/469
+gmmlu_si/test/miscellaneous/test/473
+gmmlu_si/test/miscellaneous/test/482
+gmmlu_si/test/miscellaneous/test/483
+gmmlu_si/test/miscellaneous/test/489
+gmmlu_si/test/miscellaneous/test/49
+gmmlu_si/test/miscellaneous/test/500
+gmmlu_si/test/miscellaneous/test/511
+gmmlu_si/test/miscellaneous/test/517
+gmmlu_si/test/miscellaneous/test/519
+gmmlu_si/test/miscellaneous/test/536
+gmmlu_si/test/miscellaneous/test/542
+gmmlu_si/test/miscellaneous/test/55
+gmmlu_si/test/miscellaneous/test/553
+gmmlu_si/test/miscellaneous/test/565
+gmmlu_si/test/miscellaneous/test/571
+gmmlu_si/test/miscellaneous/test/577
+gmmlu_si/test/miscellaneous/test/583
+gmmlu_si/test/miscellaneous/test/586
+gmmlu_si/test/miscellaneous/test/587
+gmmlu_si/test/miscellaneous/test/593
+gmmlu_si/test/miscellaneous/test/597
+gmmlu_si/test/miscellaneous/test/61
+gmmlu_si/test/miscellaneous/test/618
+gmmlu_si/test/miscellaneous/test/635
+gmmlu_si/test/miscellaneous/test/641
+gmmlu_si/test/miscellaneous/test/66
+gmmlu_si/test/miscellaneous/test/661
+gmmlu_si/test/miscellaneous/test/662
+gmmlu_si/test/miscellaneous/test/663
+gmmlu_si/test/miscellaneous/test/67
+gmmlu_si/test/miscellaneous/test/676
+gmmlu_si/test/miscellaneous/test/68
+gmmlu_si/test/miscellaneous/test/683
+gmmlu_si/test/miscellaneous/test/684
+gmmlu_si/test/miscellaneous/test/685
+gmmlu_si/test/miscellaneous/test/687
+gmmlu_si/test/miscellaneous/test/696
+gmmlu_si/test/miscellaneous/test/697
+gmmlu_si/test/miscellaneous/test/705
+gmmlu_si/test/miscellaneous/test/708
+gmmlu_si/test/miscellaneous/test/710
+gmmlu_si/test/miscellaneous/test/719
+gmmlu_si/test/miscellaneous/test/728
+gmmlu_si/test/miscellaneous/test/73
+gmmlu_si/test/miscellaneous/test/730
+gmmlu_si/test/miscellaneous/test/737
+gmmlu_si/test/miscellaneous/test/738
+gmmlu_si/test/miscellaneous/test/74
+gmmlu_si/test/miscellaneous/test/742
+gmmlu_si/test/miscellaneous/test/753
+gmmlu_si/test/miscellaneous/test/76
+gmmlu_si/test/miscellaneous/test/770
+gmmlu_si/test/miscellaneous/test/774
+gmmlu_si/test/miscellaneous/test/782
+gmmlu_si/test/miscellaneous/test/89
+gmmlu_si/test/miscellaneous/test/93
+gmmlu_si/test/miscellaneous/test/98
+gmmlu_si/test/moral_disputes/test/100
+gmmlu_si/test/moral_disputes/test/101
+gmmlu_si/test/moral_disputes/test/103
+gmmlu_si/test/moral_disputes/test/104
+gmmlu_si/test/moral_disputes/test/11
+gmmlu_si/test/moral_disputes/test/111
+gmmlu_si/test/moral_disputes/test/112
+gmmlu_si/test/moral_disputes/test/115
+gmmlu_si/test/moral_disputes/test/122
+gmmlu_si/test/moral_disputes/test/135
+gmmlu_si/test/moral_disputes/test/139
+gmmlu_si/test/moral_disputes/test/140
+gmmlu_si/test/moral_disputes/test/141
+gmmlu_si/test/moral_disputes/test/142
+gmmlu_si/test/moral_disputes/test/145
+gmmlu_si/test/moral_disputes/test/151
+gmmlu_si/test/moral_disputes/test/164
+gmmlu_si/test/moral_disputes/test/175
+gmmlu_si/test/moral_disputes/test/185
+gmmlu_si/test/moral_disputes/test/190
+gmmlu_si/test/moral_disputes/test/191
+gmmlu_si/test/moral_disputes/test/195
+gmmlu_si/test/moral_disputes/test/202
+gmmlu_si/test/moral_disputes/test/206
+gmmlu_si/test/moral_disputes/test/219
+gmmlu_si/test/moral_disputes/test/223
+gmmlu_si/test/moral_disputes/test/254
+gmmlu_si/test/moral_disputes/test/275
+gmmlu_si/test/moral_disputes/test/282
+gmmlu_si/test/moral_disputes/test/286
+gmmlu_si/test/moral_disputes/test/288
+gmmlu_si/test/moral_disputes/test/293
+gmmlu_si/test/moral_disputes/test/298
+gmmlu_si/test/moral_disputes/test/303
+gmmlu_si/test/moral_disputes/test/306
+gmmlu_si/test/moral_disputes/test/313
+gmmlu_si/test/moral_disputes/test/342
+gmmlu_si/test/moral_disputes/test/344
+gmmlu_si/test/moral_disputes/test/4
+gmmlu_si/test/moral_disputes/test/55
+gmmlu_si/test/moral_disputes/test/57
+gmmlu_si/test/moral_disputes/test/62
+gmmlu_si/test/moral_disputes/test/65
+gmmlu_si/test/moral_disputes/test/67
+gmmlu_si/test/moral_disputes/test/77
+gmmlu_si/test/moral_disputes/test/81
+gmmlu_si/test/moral_disputes/test/83
+gmmlu_si/test/moral_disputes/test/86
+gmmlu_si/test/moral_disputes/test/89
+gmmlu_si/test/moral_scenarios/test/122
+gmmlu_si/test/moral_scenarios/test/127
+gmmlu_si/test/moral_scenarios/test/135
+gmmlu_si/test/moral_scenarios/test/136
+gmmlu_si/test/moral_scenarios/test/140
+gmmlu_si/test/moral_scenarios/test/149
+gmmlu_si/test/moral_scenarios/test/151
+gmmlu_si/test/moral_scenarios/test/155
+gmmlu_si/test/moral_scenarios/test/162
+gmmlu_si/test/moral_scenarios/test/169
+gmmlu_si/test/moral_scenarios/test/180
+gmmlu_si/test/moral_scenarios/test/181
+gmmlu_si/test/moral_scenarios/test/186
+gmmlu_si/test/moral_scenarios/test/187
+gmmlu_si/test/moral_scenarios/test/191
+gmmlu_si/test/moral_scenarios/test/195
+gmmlu_si/test/moral_scenarios/test/197
+gmmlu_si/test/moral_scenarios/test/20
+gmmlu_si/test/moral_scenarios/test/227
+gmmlu_si/test/moral_scenarios/test/231
+gmmlu_si/test/moral_scenarios/test/245
+gmmlu_si/test/moral_scenarios/test/264
+gmmlu_si/test/moral_scenarios/test/269
+gmmlu_si/test/moral_scenarios/test/279
+gmmlu_si/test/moral_scenarios/test/28
+gmmlu_si/test/moral_scenarios/test/283
+gmmlu_si/test/moral_scenarios/test/290
+gmmlu_si/test/moral_scenarios/test/304
+gmmlu_si/test/moral_scenarios/test/326
+gmmlu_si/test/moral_scenarios/test/341
+gmmlu_si/test/moral_scenarios/test/345
+gmmlu_si/test/moral_scenarios/test/346
+gmmlu_si/test/moral_scenarios/test/349
+gmmlu_si/test/moral_scenarios/test/350
+gmmlu_si/test/moral_scenarios/test/355
+gmmlu_si/test/moral_scenarios/test/362
+gmmlu_si/test/moral_scenarios/test/367
+gmmlu_si/test/moral_scenarios/test/373
+gmmlu_si/test/moral_scenarios/test/396
+gmmlu_si/test/moral_scenarios/test/398
+gmmlu_si/test/moral_scenarios/test/4
+gmmlu_si/test/moral_scenarios/test/400
+gmmlu_si/test/moral_scenarios/test/402
+gmmlu_si/test/moral_scenarios/test/409
+gmmlu_si/test/moral_scenarios/test/413
+gmmlu_si/test/moral_scenarios/test/42
+gmmlu_si/test/moral_scenarios/test/43
+gmmlu_si/test/moral_scenarios/test/432
+gmmlu_si/test/moral_scenarios/test/433
+gmmlu_si/test/moral_scenarios/test/436
+gmmlu_si/test/moral_scenarios/test/464
+gmmlu_si/test/moral_scenarios/test/466
+gmmlu_si/test/moral_scenarios/test/47
+gmmlu_si/test/moral_scenarios/test/476
+gmmlu_si/test/moral_scenarios/test/477
+gmmlu_si/test/moral_scenarios/test/484
+gmmlu_si/test/moral_scenarios/test/490
+gmmlu_si/test/moral_scenarios/test/494
+gmmlu_si/test/moral_scenarios/test/507
+gmmlu_si/test/moral_scenarios/test/508
+gmmlu_si/test/moral_scenarios/test/514
+gmmlu_si/test/moral_scenarios/test/518
+gmmlu_si/test/moral_scenarios/test/519
+gmmlu_si/test/moral_scenarios/test/523
+gmmlu_si/test/moral_scenarios/test/528
+gmmlu_si/test/moral_scenarios/test/53
+gmmlu_si/test/moral_scenarios/test/539
+gmmlu_si/test/moral_scenarios/test/543
+gmmlu_si/test/moral_scenarios/test/548
+gmmlu_si/test/moral_scenarios/test/550
+gmmlu_si/test/moral_scenarios/test/552
+gmmlu_si/test/moral_scenarios/test/562
+gmmlu_si/test/moral_scenarios/test/565
+gmmlu_si/test/moral_scenarios/test/57
+gmmlu_si/test/moral_scenarios/test/570
+gmmlu_si/test/moral_scenarios/test/572
+gmmlu_si/test/moral_scenarios/test/578
+gmmlu_si/test/moral_scenarios/test/584
+gmmlu_si/test/moral_scenarios/test/588
+gmmlu_si/test/moral_scenarios/test/596
+gmmlu_si/test/moral_scenarios/test/6
+gmmlu_si/test/moral_scenarios/test/60
+gmmlu_si/test/moral_scenarios/test/606
+gmmlu_si/test/moral_scenarios/test/612
+gmmlu_si/test/moral_scenarios/test/62
+gmmlu_si/test/moral_scenarios/test/626
+gmmlu_si/test/moral_scenarios/test/631
+gmmlu_si/test/moral_scenarios/test/632
+gmmlu_si/test/moral_scenarios/test/640
+gmmlu_si/test/moral_scenarios/test/641
+gmmlu_si/test/moral_scenarios/test/646
+gmmlu_si/test/moral_scenarios/test/65
+gmmlu_si/test/moral_scenarios/test/682
+gmmlu_si/test/moral_scenarios/test/693
+gmmlu_si/test/moral_scenarios/test/699
+gmmlu_si/test/moral_scenarios/test/7
+gmmlu_si/test/moral_scenarios/test/709
+gmmlu_si/test/moral_scenarios/test/718
+gmmlu_si/test/moral_scenarios/test/732
+gmmlu_si/test/moral_scenarios/test/747
+gmmlu_si/test/moral_scenarios/test/756
+gmmlu_si/test/moral_scenarios/test/759
+gmmlu_si/test/moral_scenarios/test/76
+gmmlu_si/test/moral_scenarios/test/761
+gmmlu_si/test/moral_scenarios/test/775
+gmmlu_si/test/moral_scenarios/test/789
+gmmlu_si/test/moral_scenarios/test/793
+gmmlu_si/test/moral_scenarios/test/794
+gmmlu_si/test/moral_scenarios/test/802
+gmmlu_si/test/moral_scenarios/test/808
+gmmlu_si/test/moral_scenarios/test/814
+gmmlu_si/test/moral_scenarios/test/82
+gmmlu_si/test/moral_scenarios/test/827
+gmmlu_si/test/moral_scenarios/test/831
+gmmlu_si/test/moral_scenarios/test/834
+gmmlu_si/test/moral_scenarios/test/848
+gmmlu_si/test/moral_scenarios/test/851
+gmmlu_si/test/moral_scenarios/test/855
+gmmlu_si/test/moral_scenarios/test/856
+gmmlu_si/test/moral_scenarios/test/857
+gmmlu_si/test/moral_scenarios/test/87
+gmmlu_si/test/moral_scenarios/test/884
+gmmlu_si/test/moral_scenarios/test/887
+gmmlu_si/test/moral_scenarios/test/888
+gmmlu_si/test/moral_scenarios/test/894
+gmmlu_si/test/moral_scenarios/test/90
+gmmlu_si/test/moral_scenarios/test/97
+gmmlu_si/test/nutrition/test/104
+gmmlu_si/test/nutrition/test/113
+gmmlu_si/test/nutrition/test/126
+gmmlu_si/test/nutrition/test/131
+gmmlu_si/test/nutrition/test/136
+gmmlu_si/test/nutrition/test/145
+gmmlu_si/test/nutrition/test/15
+gmmlu_si/test/nutrition/test/152
+gmmlu_si/test/nutrition/test/155
+gmmlu_si/test/nutrition/test/160
+gmmlu_si/test/nutrition/test/168
+gmmlu_si/test/nutrition/test/183
+gmmlu_si/test/nutrition/test/186
+gmmlu_si/test/nutrition/test/188
+gmmlu_si/test/nutrition/test/191
+gmmlu_si/test/nutrition/test/200
+gmmlu_si/test/nutrition/test/209
+gmmlu_si/test/nutrition/test/210
+gmmlu_si/test/nutrition/test/211
+gmmlu_si/test/nutrition/test/217
+gmmlu_si/test/nutrition/test/225
+gmmlu_si/test/nutrition/test/228
+gmmlu_si/test/nutrition/test/232
+gmmlu_si/test/nutrition/test/237
+gmmlu_si/test/nutrition/test/238
+gmmlu_si/test/nutrition/test/24
+gmmlu_si/test/nutrition/test/248
+gmmlu_si/test/nutrition/test/277
+gmmlu_si/test/nutrition/test/279
+gmmlu_si/test/nutrition/test/281
+gmmlu_si/test/nutrition/test/282
+gmmlu_si/test/nutrition/test/291
+gmmlu_si/test/nutrition/test/303
+gmmlu_si/test/nutrition/test/49
+gmmlu_si/test/nutrition/test/52
+gmmlu_si/test/nutrition/test/53
+gmmlu_si/test/nutrition/test/59
+gmmlu_si/test/nutrition/test/62
+gmmlu_si/test/nutrition/test/65
+gmmlu_si/test/nutrition/test/67
+gmmlu_si/test/nutrition/test/68
+gmmlu_si/test/nutrition/test/72
+gmmlu_si/test/nutrition/test/90
+gmmlu_si/test/nutrition/test/95
+gmmlu_si/test/philosophy/test/126
+gmmlu_si/test/philosophy/test/135
+gmmlu_si/test/philosophy/test/15
+gmmlu_si/test/philosophy/test/158
+gmmlu_si/test/philosophy/test/16
+gmmlu_si/test/philosophy/test/160
+gmmlu_si/test/philosophy/test/161
+gmmlu_si/test/philosophy/test/165
+gmmlu_si/test/philosophy/test/174
+gmmlu_si/test/philosophy/test/176
+gmmlu_si/test/philosophy/test/179
+gmmlu_si/test/philosophy/test/187
+gmmlu_si/test/philosophy/test/193
+gmmlu_si/test/philosophy/test/194
+gmmlu_si/test/philosophy/test/195
+gmmlu_si/test/philosophy/test/201
+gmmlu_si/test/philosophy/test/202
+gmmlu_si/test/philosophy/test/207
+gmmlu_si/test/philosophy/test/215
+gmmlu_si/test/philosophy/test/217
+gmmlu_si/test/philosophy/test/229
+gmmlu_si/test/philosophy/test/234
+gmmlu_si/test/philosophy/test/243
+gmmlu_si/test/philosophy/test/244
+gmmlu_si/test/philosophy/test/250
+gmmlu_si/test/philosophy/test/257
+gmmlu_si/test/philosophy/test/263
+gmmlu_si/test/philosophy/test/264
+gmmlu_si/test/philosophy/test/265
+gmmlu_si/test/philosophy/test/278
+gmmlu_si/test/philosophy/test/279
+gmmlu_si/test/philosophy/test/280
+gmmlu_si/test/philosophy/test/281
+gmmlu_si/test/philosophy/test/299
+gmmlu_si/test/philosophy/test/305
+gmmlu_si/test/philosophy/test/306
+gmmlu_si/test/philosophy/test/40
+gmmlu_si/test/philosophy/test/46
+gmmlu_si/test/philosophy/test/6
+gmmlu_si/test/philosophy/test/61
+gmmlu_si/test/philosophy/test/65
+gmmlu_si/test/philosophy/test/70
+gmmlu_si/test/philosophy/test/81
+gmmlu_si/test/philosophy/test/88
+gmmlu_si/test/prehistory/test/108
+gmmlu_si/test/prehistory/test/111
+gmmlu_si/test/prehistory/test/119
+gmmlu_si/test/prehistory/test/122
+gmmlu_si/test/prehistory/test/129
+gmmlu_si/test/prehistory/test/13
+gmmlu_si/test/prehistory/test/133
+gmmlu_si/test/prehistory/test/134
+gmmlu_si/test/prehistory/test/141
+gmmlu_si/test/prehistory/test/142
+gmmlu_si/test/prehistory/test/143
+gmmlu_si/test/prehistory/test/149
+gmmlu_si/test/prehistory/test/161
+gmmlu_si/test/prehistory/test/162
+gmmlu_si/test/prehistory/test/174
+gmmlu_si/test/prehistory/test/184
+gmmlu_si/test/prehistory/test/203
+gmmlu_si/test/prehistory/test/211
+gmmlu_si/test/prehistory/test/213
+gmmlu_si/test/prehistory/test/216
+gmmlu_si/test/prehistory/test/217
+gmmlu_si/test/prehistory/test/223
+gmmlu_si/test/prehistory/test/238
+gmmlu_si/test/prehistory/test/24
+gmmlu_si/test/prehistory/test/242
+gmmlu_si/test/prehistory/test/255
+gmmlu_si/test/prehistory/test/26
+gmmlu_si/test/prehistory/test/267
+gmmlu_si/test/prehistory/test/279
+gmmlu_si/test/prehistory/test/28
+gmmlu_si/test/prehistory/test/294
+gmmlu_si/test/prehistory/test/299
+gmmlu_si/test/prehistory/test/3
+gmmlu_si/test/prehistory/test/300
+gmmlu_si/test/prehistory/test/303
+gmmlu_si/test/prehistory/test/308
+gmmlu_si/test/prehistory/test/312
+gmmlu_si/test/prehistory/test/318
+gmmlu_si/test/prehistory/test/320
+gmmlu_si/test/prehistory/test/45
+gmmlu_si/test/prehistory/test/48
+gmmlu_si/test/prehistory/test/59
+gmmlu_si/test/prehistory/test/7
+gmmlu_si/test/prehistory/test/76
+gmmlu_si/test/prehistory/test/85
+gmmlu_si/test/prehistory/test/95
+gmmlu_si/test/professional_accounting/test/0
+gmmlu_si/test/professional_accounting/test/108
+gmmlu_si/test/professional_accounting/test/109
+gmmlu_si/test/professional_accounting/test/112
+gmmlu_si/test/professional_accounting/test/114
+gmmlu_si/test/professional_accounting/test/119
+gmmlu_si/test/professional_accounting/test/120
+gmmlu_si/test/professional_accounting/test/128
+gmmlu_si/test/professional_accounting/test/135
+gmmlu_si/test/professional_accounting/test/136
+gmmlu_si/test/professional_accounting/test/142
+gmmlu_si/test/professional_accounting/test/157
+gmmlu_si/test/professional_accounting/test/159
+gmmlu_si/test/professional_accounting/test/167
+gmmlu_si/test/professional_accounting/test/168
+gmmlu_si/test/professional_accounting/test/173
+gmmlu_si/test/professional_accounting/test/193
+gmmlu_si/test/professional_accounting/test/197
+gmmlu_si/test/professional_accounting/test/206
+gmmlu_si/test/professional_accounting/test/210
+gmmlu_si/test/professional_accounting/test/212
+gmmlu_si/test/professional_accounting/test/216
+gmmlu_si/test/professional_accounting/test/224
+gmmlu_si/test/professional_accounting/test/237
+gmmlu_si/test/professional_accounting/test/241
+gmmlu_si/test/professional_accounting/test/252
+gmmlu_si/test/professional_accounting/test/254
+gmmlu_si/test/professional_accounting/test/258
+gmmlu_si/test/professional_accounting/test/261
+gmmlu_si/test/professional_accounting/test/262
+gmmlu_si/test/professional_accounting/test/264
+gmmlu_si/test/professional_accounting/test/278
+gmmlu_si/test/professional_accounting/test/279
+gmmlu_si/test/professional_accounting/test/281
+gmmlu_si/test/professional_accounting/test/41
+gmmlu_si/test/professional_accounting/test/46
+gmmlu_si/test/professional_accounting/test/60
+gmmlu_si/test/professional_accounting/test/76
+gmmlu_si/test/professional_accounting/test/86
+gmmlu_si/test/professional_accounting/test/90
+gmmlu_si/test/professional_law/test/1014
+gmmlu_si/test/professional_law/test/1024
+gmmlu_si/test/professional_law/test/1032
+gmmlu_si/test/professional_law/test/1034
+gmmlu_si/test/professional_law/test/1047
+gmmlu_si/test/professional_law/test/1051
+gmmlu_si/test/professional_law/test/1052
+gmmlu_si/test/professional_law/test/1056
+gmmlu_si/test/professional_law/test/1059
+gmmlu_si/test/professional_law/test/106
+gmmlu_si/test/professional_law/test/1072
+gmmlu_si/test/professional_law/test/1075
+gmmlu_si/test/professional_law/test/1083
+gmmlu_si/test/professional_law/test/109
+gmmlu_si/test/professional_law/test/1095
+gmmlu_si/test/professional_law/test/1097
+gmmlu_si/test/professional_law/test/1109
+gmmlu_si/test/professional_law/test/1110
+gmmlu_si/test/professional_law/test/1125
+gmmlu_si/test/professional_law/test/1128
+gmmlu_si/test/professional_law/test/1131
+gmmlu_si/test/professional_law/test/1139
+gmmlu_si/test/professional_law/test/1145
+gmmlu_si/test/professional_law/test/1169
+gmmlu_si/test/professional_law/test/1170
+gmmlu_si/test/professional_law/test/1178
+gmmlu_si/test/professional_law/test/1179
+gmmlu_si/test/professional_law/test/1189
+gmmlu_si/test/professional_law/test/1190
+gmmlu_si/test/professional_law/test/1202
+gmmlu_si/test/professional_law/test/1203
+gmmlu_si/test/professional_law/test/1207
+gmmlu_si/test/professional_law/test/1211
+gmmlu_si/test/professional_law/test/1213
+gmmlu_si/test/professional_law/test/122
+gmmlu_si/test/professional_law/test/1233
+gmmlu_si/test/professional_law/test/1251
+gmmlu_si/test/professional_law/test/1257
+gmmlu_si/test/professional_law/test/1266
+gmmlu_si/test/professional_law/test/1294
+gmmlu_si/test/professional_law/test/1307
+gmmlu_si/test/professional_law/test/131
+gmmlu_si/test/professional_law/test/1314
+gmmlu_si/test/professional_law/test/1319
+gmmlu_si/test/professional_law/test/1326
+gmmlu_si/test/professional_law/test/1332
+gmmlu_si/test/professional_law/test/1335
+gmmlu_si/test/professional_law/test/1338
+gmmlu_si/test/professional_law/test/1346
+gmmlu_si/test/professional_law/test/1347
+gmmlu_si/test/professional_law/test/1361
+gmmlu_si/test/professional_law/test/1384
+gmmlu_si/test/professional_law/test/1387
+gmmlu_si/test/professional_law/test/1392
+gmmlu_si/test/professional_law/test/1401
+gmmlu_si/test/professional_law/test/1411
+gmmlu_si/test/professional_law/test/1413
+gmmlu_si/test/professional_law/test/1431
+gmmlu_si/test/professional_law/test/1438
+gmmlu_si/test/professional_law/test/1439
+gmmlu_si/test/professional_law/test/1454
+gmmlu_si/test/professional_law/test/1459
+gmmlu_si/test/professional_law/test/1463
+gmmlu_si/test/professional_law/test/1470
+gmmlu_si/test/professional_law/test/1475
+gmmlu_si/test/professional_law/test/1477
+gmmlu_si/test/professional_law/test/1484
+gmmlu_si/test/professional_law/test/1487
+gmmlu_si/test/professional_law/test/1488
+gmmlu_si/test/professional_law/test/1501
+gmmlu_si/test/professional_law/test/1502
+gmmlu_si/test/professional_law/test/1504
+gmmlu_si/test/professional_law/test/1509
+gmmlu_si/test/professional_law/test/1515
+gmmlu_si/test/professional_law/test/1516
+gmmlu_si/test/professional_law/test/1521
+gmmlu_si/test/professional_law/test/1526
+gmmlu_si/test/professional_law/test/1527
+gmmlu_si/test/professional_law/test/159
+gmmlu_si/test/professional_law/test/163
+gmmlu_si/test/professional_law/test/170
+gmmlu_si/test/professional_law/test/174
+gmmlu_si/test/professional_law/test/177
+gmmlu_si/test/professional_law/test/184
+gmmlu_si/test/professional_law/test/185
+gmmlu_si/test/professional_law/test/194
+gmmlu_si/test/professional_law/test/199
+gmmlu_si/test/professional_law/test/205
+gmmlu_si/test/professional_law/test/207
+gmmlu_si/test/professional_law/test/213
+gmmlu_si/test/professional_law/test/223
+gmmlu_si/test/professional_law/test/227
+gmmlu_si/test/professional_law/test/24
+gmmlu_si/test/professional_law/test/246
+gmmlu_si/test/professional_law/test/249
+gmmlu_si/test/professional_law/test/250
+gmmlu_si/test/professional_law/test/251
+gmmlu_si/test/professional_law/test/267
+gmmlu_si/test/professional_law/test/269
+gmmlu_si/test/professional_law/test/271
+gmmlu_si/test/professional_law/test/278
+gmmlu_si/test/professional_law/test/286
+gmmlu_si/test/professional_law/test/299
+gmmlu_si/test/professional_law/test/302
+gmmlu_si/test/professional_law/test/317
+gmmlu_si/test/professional_law/test/32
+gmmlu_si/test/professional_law/test/34
+gmmlu_si/test/professional_law/test/342
+gmmlu_si/test/professional_law/test/348
+gmmlu_si/test/professional_law/test/350
+gmmlu_si/test/professional_law/test/351
+gmmlu_si/test/professional_law/test/360
+gmmlu_si/test/professional_law/test/363
+gmmlu_si/test/professional_law/test/366
+gmmlu_si/test/professional_law/test/369
+gmmlu_si/test/professional_law/test/375
+gmmlu_si/test/professional_law/test/376
+gmmlu_si/test/professional_law/test/381
+gmmlu_si/test/professional_law/test/393
+gmmlu_si/test/professional_law/test/397
+gmmlu_si/test/professional_law/test/40
+gmmlu_si/test/professional_law/test/401
+gmmlu_si/test/professional_law/test/404
+gmmlu_si/test/professional_law/test/407
+gmmlu_si/test/professional_law/test/422
+gmmlu_si/test/professional_law/test/43
+gmmlu_si/test/professional_law/test/438
+gmmlu_si/test/professional_law/test/441
+gmmlu_si/test/professional_law/test/442
+gmmlu_si/test/professional_law/test/451
+gmmlu_si/test/professional_law/test/453
+gmmlu_si/test/professional_law/test/454
+gmmlu_si/test/professional_law/test/462
+gmmlu_si/test/professional_law/test/463
+gmmlu_si/test/professional_law/test/470
+gmmlu_si/test/professional_law/test/497
+gmmlu_si/test/professional_law/test/499
+gmmlu_si/test/professional_law/test/505
+gmmlu_si/test/professional_law/test/514
+gmmlu_si/test/professional_law/test/525
+gmmlu_si/test/professional_law/test/538
+gmmlu_si/test/professional_law/test/539
+gmmlu_si/test/professional_law/test/540
+gmmlu_si/test/professional_law/test/545
+gmmlu_si/test/professional_law/test/55
+gmmlu_si/test/professional_law/test/571
+gmmlu_si/test/professional_law/test/573
+gmmlu_si/test/professional_law/test/580
+gmmlu_si/test/professional_law/test/583
+gmmlu_si/test/professional_law/test/584
+gmmlu_si/test/professional_law/test/593
+gmmlu_si/test/professional_law/test/596
+gmmlu_si/test/professional_law/test/600
+gmmlu_si/test/professional_law/test/611
+gmmlu_si/test/professional_law/test/616
+gmmlu_si/test/professional_law/test/617
+gmmlu_si/test/professional_law/test/623
+gmmlu_si/test/professional_law/test/630
+gmmlu_si/test/professional_law/test/644
+gmmlu_si/test/professional_law/test/645
+gmmlu_si/test/professional_law/test/65
+gmmlu_si/test/professional_law/test/679
+gmmlu_si/test/professional_law/test/69
+gmmlu_si/test/professional_law/test/690
+gmmlu_si/test/professional_law/test/695
+gmmlu_si/test/professional_law/test/698
+gmmlu_si/test/professional_law/test/70
+gmmlu_si/test/professional_law/test/705
+gmmlu_si/test/professional_law/test/709
+gmmlu_si/test/professional_law/test/714
+gmmlu_si/test/professional_law/test/715
+gmmlu_si/test/professional_law/test/718
+gmmlu_si/test/professional_law/test/724
+gmmlu_si/test/professional_law/test/729
+gmmlu_si/test/professional_law/test/735
+gmmlu_si/test/professional_law/test/737
+gmmlu_si/test/professional_law/test/746
+gmmlu_si/test/professional_law/test/748
+gmmlu_si/test/professional_law/test/750
+gmmlu_si/test/professional_law/test/754
+gmmlu_si/test/professional_law/test/76
+gmmlu_si/test/professional_law/test/761
+gmmlu_si/test/professional_law/test/786
+gmmlu_si/test/professional_law/test/79
+gmmlu_si/test/professional_law/test/791
+gmmlu_si/test/professional_law/test/798
+gmmlu_si/test/professional_law/test/800
+gmmlu_si/test/professional_law/test/809
+gmmlu_si/test/professional_law/test/819
+gmmlu_si/test/professional_law/test/82
+gmmlu_si/test/professional_law/test/827
+gmmlu_si/test/professional_law/test/833
+gmmlu_si/test/professional_law/test/834
+gmmlu_si/test/professional_law/test/837
+gmmlu_si/test/professional_law/test/865
+gmmlu_si/test/professional_law/test/871
+gmmlu_si/test/professional_law/test/873
+gmmlu_si/test/professional_law/test/882
+gmmlu_si/test/professional_law/test/889
+gmmlu_si/test/professional_law/test/89
+gmmlu_si/test/professional_law/test/893
+gmmlu_si/test/professional_law/test/897
+gmmlu_si/test/professional_law/test/907
+gmmlu_si/test/professional_law/test/908
+gmmlu_si/test/professional_law/test/909
+gmmlu_si/test/professional_law/test/914
+gmmlu_si/test/professional_law/test/920
+gmmlu_si/test/professional_law/test/940
+gmmlu_si/test/professional_law/test/942
+gmmlu_si/test/professional_law/test/949
+gmmlu_si/test/professional_law/test/951
+gmmlu_si/test/professional_law/test/953
+gmmlu_si/test/professional_law/test/954
+gmmlu_si/test/professional_law/test/958
+gmmlu_si/test/professional_law/test/97
+gmmlu_si/test/professional_law/test/976
+gmmlu_si/test/professional_law/test/979
+gmmlu_si/test/professional_law/test/983
+gmmlu_si/test/professional_medicine/test/103
+gmmlu_si/test/professional_medicine/test/104
+gmmlu_si/test/professional_medicine/test/111
+gmmlu_si/test/professional_medicine/test/119
+gmmlu_si/test/professional_medicine/test/120
+gmmlu_si/test/professional_medicine/test/13
+gmmlu_si/test/professional_medicine/test/132
+gmmlu_si/test/professional_medicine/test/133
+gmmlu_si/test/professional_medicine/test/143
+gmmlu_si/test/professional_medicine/test/144
+gmmlu_si/test/professional_medicine/test/15
+gmmlu_si/test/professional_medicine/test/156
+gmmlu_si/test/professional_medicine/test/158
+gmmlu_si/test/professional_medicine/test/162
+gmmlu_si/test/professional_medicine/test/170
+gmmlu_si/test/professional_medicine/test/174
+gmmlu_si/test/professional_medicine/test/178
+gmmlu_si/test/professional_medicine/test/179
+gmmlu_si/test/professional_medicine/test/18
+gmmlu_si/test/professional_medicine/test/190
+gmmlu_si/test/professional_medicine/test/192
+gmmlu_si/test/professional_medicine/test/204
+gmmlu_si/test/professional_medicine/test/209
+gmmlu_si/test/professional_medicine/test/210
+gmmlu_si/test/professional_medicine/test/211
+gmmlu_si/test/professional_medicine/test/216
+gmmlu_si/test/professional_medicine/test/221
+gmmlu_si/test/professional_medicine/test/224
+gmmlu_si/test/professional_medicine/test/234
+gmmlu_si/test/professional_medicine/test/248
+gmmlu_si/test/professional_medicine/test/249
+gmmlu_si/test/professional_medicine/test/252
+gmmlu_si/test/professional_medicine/test/254
+gmmlu_si/test/professional_medicine/test/257
+gmmlu_si/test/professional_medicine/test/258
+gmmlu_si/test/professional_medicine/test/270
+gmmlu_si/test/professional_medicine/test/43
+gmmlu_si/test/professional_medicine/test/5
+gmmlu_si/test/professional_medicine/test/83
+gmmlu_si/test/professional_psychology/test/108
+gmmlu_si/test/professional_psychology/test/113
+gmmlu_si/test/professional_psychology/test/136
+gmmlu_si/test/professional_psychology/test/140
+gmmlu_si/test/professional_psychology/test/153
+gmmlu_si/test/professional_psychology/test/155
+gmmlu_si/test/professional_psychology/test/157
+gmmlu_si/test/professional_psychology/test/163
+gmmlu_si/test/professional_psychology/test/164
+gmmlu_si/test/professional_psychology/test/17
+gmmlu_si/test/professional_psychology/test/180
+gmmlu_si/test/professional_psychology/test/185
+gmmlu_si/test/professional_psychology/test/196
+gmmlu_si/test/professional_psychology/test/209
+gmmlu_si/test/professional_psychology/test/210
+gmmlu_si/test/professional_psychology/test/223
+gmmlu_si/test/professional_psychology/test/224
+gmmlu_si/test/professional_psychology/test/226
+gmmlu_si/test/professional_psychology/test/227
+gmmlu_si/test/professional_psychology/test/237
+gmmlu_si/test/professional_psychology/test/240
+gmmlu_si/test/professional_psychology/test/242
+gmmlu_si/test/professional_psychology/test/249
+gmmlu_si/test/professional_psychology/test/285
+gmmlu_si/test/professional_psychology/test/288
+gmmlu_si/test/professional_psychology/test/30
+gmmlu_si/test/professional_psychology/test/310
+gmmlu_si/test/professional_psychology/test/314
+gmmlu_si/test/professional_psychology/test/325
+gmmlu_si/test/professional_psychology/test/33
+gmmlu_si/test/professional_psychology/test/334
+gmmlu_si/test/professional_psychology/test/336
+gmmlu_si/test/professional_psychology/test/340
+gmmlu_si/test/professional_psychology/test/341
+gmmlu_si/test/professional_psychology/test/342
+gmmlu_si/test/professional_psychology/test/343
+gmmlu_si/test/professional_psychology/test/352
+gmmlu_si/test/professional_psychology/test/355
+gmmlu_si/test/professional_psychology/test/363
+gmmlu_si/test/professional_psychology/test/380
+gmmlu_si/test/professional_psychology/test/385
+gmmlu_si/test/professional_psychology/test/390
+gmmlu_si/test/professional_psychology/test/400
+gmmlu_si/test/professional_psychology/test/411
+gmmlu_si/test/professional_psychology/test/416
+gmmlu_si/test/professional_psychology/test/418
+gmmlu_si/test/professional_psychology/test/424
+gmmlu_si/test/professional_psychology/test/43
+gmmlu_si/test/professional_psychology/test/430
+gmmlu_si/test/professional_psychology/test/445
+gmmlu_si/test/professional_psychology/test/450
+gmmlu_si/test/professional_psychology/test/458
+gmmlu_si/test/professional_psychology/test/48
+gmmlu_si/test/professional_psychology/test/487
+gmmlu_si/test/professional_psychology/test/489
+gmmlu_si/test/professional_psychology/test/493
+gmmlu_si/test/professional_psychology/test/498
+gmmlu_si/test/professional_psychology/test/503
+gmmlu_si/test/professional_psychology/test/513
+gmmlu_si/test/professional_psychology/test/524
+gmmlu_si/test/professional_psychology/test/526
+gmmlu_si/test/professional_psychology/test/543
+gmmlu_si/test/professional_psychology/test/556
+gmmlu_si/test/professional_psychology/test/557
+gmmlu_si/test/professional_psychology/test/558
+gmmlu_si/test/professional_psychology/test/559
+gmmlu_si/test/professional_psychology/test/56
+gmmlu_si/test/professional_psychology/test/563
+gmmlu_si/test/professional_psychology/test/567
+gmmlu_si/test/professional_psychology/test/57
+gmmlu_si/test/professional_psychology/test/573
+gmmlu_si/test/professional_psychology/test/577
+gmmlu_si/test/professional_psychology/test/583
+gmmlu_si/test/professional_psychology/test/584
+gmmlu_si/test/professional_psychology/test/589
+gmmlu_si/test/professional_psychology/test/592
+gmmlu_si/test/professional_psychology/test/596
+gmmlu_si/test/professional_psychology/test/597
+gmmlu_si/test/professional_psychology/test/599
+gmmlu_si/test/professional_psychology/test/606
+gmmlu_si/test/professional_psychology/test/608
+gmmlu_si/test/professional_psychology/test/61
+gmmlu_si/test/professional_psychology/test/611
+gmmlu_si/test/professional_psychology/test/77
+gmmlu_si/test/professional_psychology/test/86
+gmmlu_si/test/professional_psychology/test/89
+gmmlu_si/test/professional_psychology/test/94
+gmmlu_si/test/public_relations/test/100
+gmmlu_si/test/public_relations/test/103
+gmmlu_si/test/public_relations/test/108
+gmmlu_si/test/public_relations/test/16
+gmmlu_si/test/public_relations/test/33
+gmmlu_si/test/public_relations/test/49
+gmmlu_si/test/public_relations/test/58
+gmmlu_si/test/public_relations/test/6
+gmmlu_si/test/public_relations/test/61
+gmmlu_si/test/public_relations/test/63
+gmmlu_si/test/public_relations/test/76
+gmmlu_si/test/public_relations/test/78
+gmmlu_si/test/public_relations/test/79
+gmmlu_si/test/public_relations/test/91
+gmmlu_si/test/public_relations/test/92
+gmmlu_si/test/public_relations/test/99
+gmmlu_si/test/security_studies/test/121
+gmmlu_si/test/security_studies/test/122
+gmmlu_si/test/security_studies/test/125
+gmmlu_si/test/security_studies/test/126
+gmmlu_si/test/security_studies/test/129
+gmmlu_si/test/security_studies/test/13
+gmmlu_si/test/security_studies/test/130
+gmmlu_si/test/security_studies/test/143
+gmmlu_si/test/security_studies/test/15
+gmmlu_si/test/security_studies/test/156
+gmmlu_si/test/security_studies/test/162
+gmmlu_si/test/security_studies/test/168
+gmmlu_si/test/security_studies/test/175
+gmmlu_si/test/security_studies/test/186
+gmmlu_si/test/security_studies/test/188
+gmmlu_si/test/security_studies/test/195
+gmmlu_si/test/security_studies/test/200
+gmmlu_si/test/security_studies/test/212
+gmmlu_si/test/security_studies/test/213
+gmmlu_si/test/security_studies/test/216
+gmmlu_si/test/security_studies/test/22
+gmmlu_si/test/security_studies/test/230
+gmmlu_si/test/security_studies/test/239
+gmmlu_si/test/security_studies/test/241
+gmmlu_si/test/security_studies/test/31
+gmmlu_si/test/security_studies/test/33
+gmmlu_si/test/security_studies/test/48
+gmmlu_si/test/security_studies/test/54
+gmmlu_si/test/security_studies/test/55
+gmmlu_si/test/security_studies/test/59
+gmmlu_si/test/security_studies/test/61
+gmmlu_si/test/security_studies/test/70
+gmmlu_si/test/security_studies/test/90
+gmmlu_si/test/security_studies/test/96
+gmmlu_si/test/security_studies/test/99
+gmmlu_si/test/sociology/test/105
+gmmlu_si/test/sociology/test/107
+gmmlu_si/test/sociology/test/119
+gmmlu_si/test/sociology/test/121
+gmmlu_si/test/sociology/test/123
+gmmlu_si/test/sociology/test/135
+gmmlu_si/test/sociology/test/139
+gmmlu_si/test/sociology/test/148
+gmmlu_si/test/sociology/test/155
+gmmlu_si/test/sociology/test/156
+gmmlu_si/test/sociology/test/16
+gmmlu_si/test/sociology/test/161
+gmmlu_si/test/sociology/test/165
+gmmlu_si/test/sociology/test/167
+gmmlu_si/test/sociology/test/17
+gmmlu_si/test/sociology/test/177
+gmmlu_si/test/sociology/test/186
+gmmlu_si/test/sociology/test/188
+gmmlu_si/test/sociology/test/195
+gmmlu_si/test/sociology/test/199
+gmmlu_si/test/sociology/test/24
+gmmlu_si/test/sociology/test/36
+gmmlu_si/test/sociology/test/39
+gmmlu_si/test/sociology/test/50
+gmmlu_si/test/sociology/test/51
+gmmlu_si/test/sociology/test/59
+gmmlu_si/test/sociology/test/66
+gmmlu_si/test/sociology/test/93
+gmmlu_si/test/sociology/test/97
+gmmlu_si/test/us_foreign_policy/test/14
+gmmlu_si/test/us_foreign_policy/test/25
+gmmlu_si/test/us_foreign_policy/test/3
+gmmlu_si/test/us_foreign_policy/test/39
+gmmlu_si/test/us_foreign_policy/test/42
+gmmlu_si/test/us_foreign_policy/test/44
+gmmlu_si/test/us_foreign_policy/test/46
+gmmlu_si/test/us_foreign_policy/test/48
+gmmlu_si/test/us_foreign_policy/test/54
+gmmlu_si/test/us_foreign_policy/test/58
+gmmlu_si/test/us_foreign_policy/test/83
+gmmlu_si/test/us_foreign_policy/test/94
+gmmlu_si/test/us_foreign_policy/test/98
+gmmlu_si/test/us_foreign_policy/test/99
+gmmlu_si/test/virology/test/101
+gmmlu_si/test/virology/test/105
+gmmlu_si/test/virology/test/106
+gmmlu_si/test/virology/test/107
+gmmlu_si/test/virology/test/11
+gmmlu_si/test/virology/test/120
+gmmlu_si/test/virology/test/122
+gmmlu_si/test/virology/test/129
+gmmlu_si/test/virology/test/140
+gmmlu_si/test/virology/test/147
+gmmlu_si/test/virology/test/157
+gmmlu_si/test/virology/test/16
+gmmlu_si/test/virology/test/17
+gmmlu_si/test/virology/test/23
+gmmlu_si/test/virology/test/29
+gmmlu_si/test/virology/test/30
+gmmlu_si/test/virology/test/34
+gmmlu_si/test/virology/test/49
+gmmlu_si/test/virology/test/52
+gmmlu_si/test/virology/test/61
+gmmlu_si/test/virology/test/8
+gmmlu_si/test/virology/test/85
+gmmlu_si/test/virology/test/94
+gmmlu_si/test/virology/test/97
+gmmlu_si/test/world_religions/test/0
+gmmlu_si/test/world_religions/test/109
+gmmlu_si/test/world_religions/test/112
+gmmlu_si/test/world_religions/test/122
+gmmlu_si/test/world_religions/test/123
+gmmlu_si/test/world_religions/test/13
+gmmlu_si/test/world_religions/test/146
+gmmlu_si/test/world_religions/test/154
+gmmlu_si/test/world_religions/test/159
+gmmlu_si/test/world_religions/test/161
+gmmlu_si/test/world_religions/test/21
+gmmlu_si/test/world_religions/test/4
+gmmlu_si/test/world_religions/test/44
+gmmlu_si/test/world_religions/test/45
+gmmlu_si/test/world_religions/test/50
+gmmlu_si/test/world_religions/test/53
+gmmlu_si/test/world_religions/test/60
+gmmlu_si/test/world_religions/test/61
+gmmlu_si/test/world_religions/test/62
+gmmlu_si/test/world_religions/test/68
+gmmlu_si/test/world_religions/test/74
+gmmlu_si/test/world_religions/test/81
+gmmlu_si/test/world_religions/test/9
+gmmlu_si/test/world_religions/test/90

training/configs/splits/sinhalammlu_dev_5shot_ids.txt

@@ -0,0 +1,5 @@
+smmlu/easy/science/128
+smmlu/easy/Health and Physical Science/52
+smmlu/easy/Sinhala language and literature/21
+smmlu/easy/Buddhism/158
+smmlu/easy/Buddhism/147

training/configs/splits/sinhalammlu_dev_cv5.json

@@ -0,0 +1,1862 @@
+[
+ [
+  "smmlu/easy/Arts/73",
+  "smmlu/easy/Arts/55",
+  "smmlu/easy/Arts/84",
+  "smmlu/easy/Arts/31",
+  "smmlu/easy/Arts/53",
+  "smmlu/easy/Arts/66",
+  "smmlu/easy/Arts/54",
+  "smmlu/easy/Arts/44",
+  "smmlu/easy/Arts/30",
+  "smmlu/easy/Arts/63",
+  "smmlu/easy/Arts/35",
+  "smmlu/easy/Arts/29",
+  "smmlu/easy/Arts/93",
+  "smmlu/easy/Arts/68",
+  "smmlu/easy/Arts/37",
+  "smmlu/easy/Arts/3",
+  "smmlu/easy/Arts/64",
+  "smmlu/easy/Arts/11",
+  "smmlu/easy/Arts/18",
+  "smmlu/easy/Arts/19",
+  "smmlu/easy/Buddhism/77",
+  "smmlu/easy/Buddhism/154",
+  "smmlu/easy/Buddhism/36",
+  "smmlu/easy/Buddhism/146",
+  "smmlu/easy/Buddhism/35",
+  "smmlu/easy/Buddhism/47",
+  "smmlu/easy/Buddhism/150",
+  "smmlu/easy/Buddhism/124",
+  "smmlu/easy/Buddhism/5",
+  "smmlu/easy/Buddhism/84",
+  "smmlu/easy/Buddhism/159",
+  "smmlu/easy/Buddhism/13",
+  "smmlu/easy/Buddhism/97",
+  "smmlu/easy/Buddhism/142",
+  "smmlu/easy/Buddhism/103",
+  "smmlu/easy/Buddhism/1",
+  "smmlu/easy/Buddhism/87",
+  "smmlu/easy/Buddhism/114",
+  "smmlu/easy/Buddhism/116",
+  "smmlu/easy/Buddhism/67",
+  "smmlu/easy/Buddhism/83",
+  "smmlu/easy/Buddhism/157",
+  "smmlu/easy/Buddhism/71",
+  "smmlu/easy/Buddhism/51",
+  "smmlu/easy/Buddhism/85",
+  "smmlu/easy/Buddhism/60",
+  "smmlu/easy/Buddhism/19",
+  "smmlu/easy/Buddhism/115",
+  "smmlu/easy/Buddhism/135",
+  "smmlu/easy/Buddhism/4",
+  "smmlu/easy/Buddhism/53",
+  "smmlu/easy/Buddhism/20",
+  "smmlu/easy/Buddhism/90",
+  "smmlu/easy/Catholicism/84",
+  "smmlu/easy/Catholicism/70",
+  "smmlu/easy/Catholicism/93",
+  "smmlu/easy/Catholicism/111",
+  "smmlu/easy/Catholicism/11",
+  "smmlu/easy/Catholicism/64",
+  "smmlu/easy/Catholicism/55",
+  "smmlu/easy/Catholicism/42",
+  "smmlu/easy/Catholicism/131",
+  "smmlu/easy/Catholicism/52",
+  "smmlu/easy/Catholicism/88",
+  "smmlu/easy/Catholicism/97",
+  "smmlu/easy/Catholicism/63",
+  "smmlu/easy/Catholicism/76",
+  "smmlu/easy/Catholicism/124",
+  "smmlu/easy/Catholicism/106",
+  "smmlu/easy/Catholicism/16",
+  "smmlu/easy/Catholicism/57",
+  "smmlu/easy/Catholicism/22",
+  "smmlu/easy/Catholicism/92",
+  "smmlu/easy/Catholicism/13",
+  "smmlu/easy/Catholicism/19",
+  "smmlu/easy/Catholicism/107",
+  "smmlu/easy/Catholicism/120",
+  "smmlu/easy/Catholicism/18",
+  "smmlu/easy/Catholicism/47",
+  "smmlu/easy/Christianity/122",
+  "smmlu/easy/Christianity/105",
+  "smmlu/easy/Christianity/91",
+  "smmlu/easy/Christianity/96",
+  "smmlu/easy/Christianity/103",
+  "smmlu/easy/Christianity/41",
+  "smmlu/easy/Christianity/124",
+  "smmlu/easy/Christianity/25",
+  "smmlu/easy/Christianity/58",
+  "smmlu/easy/Christianity/95",
+  "smmlu/easy/Christianity/127",
+  "smmlu/easy/Christianity/60",
+  "smmlu/easy/Christianity/20",
+  "smmlu/easy/Christianity/40",
+  "smmlu/easy/Christianity/52",
+  "smmlu/easy/Christianity/120",
+  "smmlu/easy/Christianity/118",
+  "smmlu/easy/Christianity/85",
+  "smmlu/easy/Christianity/78",
+  "smmlu/easy/Christianity/102",
+  "smmlu/easy/Christianity/49",
+  "smmlu/easy/Christianity/82",
+  "smmlu/easy/Christianity/24",
+  "smmlu/easy/Christianity/50",
+  "smmlu/easy/Christianity/128",
+  "smmlu/easy/Christianity/4",
+  "smmlu/easy/Civics/1",
+  "smmlu/easy/Civics/97",
+  "smmlu/easy/Civics/43",
+  "smmlu/easy/Civics/111",
+  "smmlu/easy/Civics/86",
+  "smmlu/easy/Civics/108",
+  "smmlu/easy/Civics/41",
+  "smmlu/easy/Civics/40",
+  "smmlu/easy/Civics/89",
+  "smmlu/easy/Civics/13",
+  "smmlu/easy/Civics/8",
+  "smmlu/easy/Civics/42",
+  "smmlu/easy/Civics/25",
+  "smmlu/easy/Civics/106",
+  "smmlu/easy/Civics/122",
+  "smmlu/easy/Civics/14",
+  "smmlu/easy/Civics/117",
+  "smmlu/easy/Civics/94",
+  "smmlu/easy/Civics/19",
+  "smmlu/easy/Civics/39",
+  "smmlu/easy/Civics/116",
+  "smmlu/easy/Civics/84",
+  "smmlu/easy/Civics/10",
+  "smmlu/easy/Civics/92",
+  "smmlu/easy/Civics/56",
+  "smmlu/easy/Eastern music/35",
+  "smmlu/easy/Eastern music/65",
+  "smmlu/easy/Eastern music/80",
+  "smmlu/easy/Eastern music/73",
+  "smmlu/easy/Eastern music/83",
+  "smmlu/easy/Eastern music/4",
+  "smmlu/easy/Eastern music/100",
+  "smmlu/easy/Eastern music/111",
+  "smmlu/easy/Eastern music/97",
+  "smmlu/easy/Eastern music/109",
+  "smmlu/easy/Eastern music/62",
+  "smmlu/easy/Eastern music/7",
+  "smmlu/easy/Eastern music/132",
+  "smmlu/easy/Eastern music/98",
+  "smmlu/easy/Eastern music/14",
+  "smmlu/easy/Eastern music/91",
+  "smmlu/easy/Eastern music/119",
+  "smmlu/easy/Eastern music/12",
+  "smmlu/easy/Eastern music/104",
+  "smmlu/easy/Eastern music/59",
+  "smmlu/easy/Eastern music/51",
+  "smmlu/easy/Eastern music/89",
+  "smmlu/easy/Eastern music/60",
+  "smmlu/easy/Eastern music/128",
+  "smmlu/easy/Eastern music/8",
+  "smmlu/easy/Eastern music/67",
+  "smmlu/easy/Eastern music/92",
+  "smmlu/easy/Eastern music/90",
+  "smmlu/easy/Geography/105",
+  "smmlu/easy/Geography/14",
+  "smmlu/easy/Geography/118",
+  "smmlu/easy/Geography/1",
+  "smmlu/easy/Geography/88",
+  "smmlu/easy/Geography/89",
+  "smmlu/easy/Geography/27",
+  "smmlu/easy/Geography/26",
+  "smmlu/easy/Geography/15",
+  "smmlu/easy/Geography/36",
+  "smmlu/easy/Geography/81",
+  "smmlu/easy/Geography/3",
+  "smmlu/easy/Geography/25",
+  "smmlu/easy/Geography/85",
+  "smmlu/easy/Geography/53",
+  "smmlu/easy/Geography/32",
+  "smmlu/easy/Geography/4",
+  "smmlu/easy/Geography/6",
+  "smmlu/easy/Geography/111",
+  "smmlu/easy/Geography/34",
+  "smmlu/easy/Geography/106",
+  "smmlu/easy/Geography/65",
+  "smmlu/easy/Geography/107",
+  "smmlu/easy/Geography/75",
+  "smmlu/easy/Health and Physical Science/97",
+  "smmlu/easy/Health and Physical Science/1",
+  "smmlu/easy/Health and Physical Science/65",
+  "smmlu/easy/Health and Physical Science/27",
+  "smmlu/easy/Health and Physical Science/25",
+  "smmlu/easy/Health and Physical Science/40",
+  "smmlu/easy/Health and Physical Science/50",
+  "smmlu/easy/Health and Physical Science/112",
+  "smmlu/easy/Health and Physical Science/38",
+  "smmlu/easy/Health and Physical Science/92",
+  "smmlu/easy/Health and Physical Science/108",
+  "smmlu/easy/Health and Physical Science/20",
+  "smmlu/easy/Health and Physical Science/8",
+  "smmlu/easy/Health and Physical Science/11",
+  "smmlu/easy/Health and Physical Science/72",
+  "smmlu/easy/Health and Physical Science/86",
+  "smmlu/easy/Health and Physical Science/94",
+  "smmlu/easy/Health and Physical Science/121",
+  "smmlu/easy/Health and Physical Science/55",
+  "smmlu/easy/Health and Physical Science/83",
+  "smmlu/easy/Health and Physical Science/54",
+  "smmlu/easy/Health and Physical Science/127",
+  "smmlu/easy/Health and Physical Science/82",
+  "smmlu/easy/Health and Physical Science/115",
+  "smmlu/easy/Health and Physical Science/26",
+  "smmlu/easy/Health and Physical Science/7",
+  "smmlu/easy/History/60",
+  "smmlu/easy/History/123",
+  "smmlu/easy/History/59",
+  "smmlu/easy/History/76",
+  "smmlu/easy/History/34",
+  "smmlu/easy/History/43",
+  "smmlu/easy/History/25",
+  "smmlu/easy/History/17",
+  "smmlu/easy/History/80",
+  "smmlu/easy/History/51",
+  "smmlu/easy/History/131",
+  "smmlu/easy/History/44",
+  "smmlu/easy/History/7",
+  "smmlu/easy/History/8",
+  "smmlu/easy/History/92",
+  "smmlu/easy/History/61",
+  "smmlu/easy/History/103",
+  "smmlu/easy/History/118",
+  "smmlu/easy/History/50",
+  "smmlu/easy/History/79",
+  "smmlu/easy/History/104",
+  "smmlu/easy/History/94",
+  "smmlu/easy/History/148",
+  "smmlu/easy/History/86",
+  "smmlu/easy/History/100",
+  "smmlu/easy/History/91",
+  "smmlu/easy/History/125",
+  "smmlu/easy/History/73",
+  "smmlu/easy/History/87",
+  "smmlu/easy/History/134",
+  "smmlu/easy/Islam/78",
+  "smmlu/easy/Islam/68",
+  "smmlu/easy/Islam/67",
+  "smmlu/easy/Islam/61",
+  "smmlu/easy/Islam/107",
+  "smmlu/easy/Islam/60",
+  "smmlu/easy/Islam/84",
+  "smmlu/easy/Islam/105",
+  "smmlu/easy/Islam/98",
+  "smmlu/easy/Islam/43",
+  "smmlu/easy/Islam/52",
+  "smmlu/easy/Islam/99",
+  "smmlu/easy/Islam/93",
+  "smmlu/easy/Islam/24",
+  "smmlu/easy/Islam/108",
+  "smmlu/easy/Islam/18",
+  "smmlu/easy/Islam/48",
+  "smmlu/easy/Islam/31",
+  "smmlu/easy/Islam/95",
+  "smmlu/easy/Islam/50",
+  "smmlu/easy/Islam/69",
+  "smmlu/easy/Sinhala language and literature/94",
+  "smmlu/easy/Sinhala language and literature/62",
+  "smmlu/easy/Sinhala language and literature/32",
+  "smmlu/easy/Sinhala language and literature/138",
+  "smmlu/easy/Sinhala language and literature/154",
+  "smmlu/easy/Sinhala language and literature/23",
+  "smmlu/easy/Sinhala language and literature/14",
+  "smmlu/easy/Sinhala language and literature/98",
+  "smmlu/easy/Sinhala language and literature/15",
+  "smmlu/easy/Sinhala language and literature/149",
+  "smmlu/easy/Sinhala language and literature/7",
+  "smmlu/easy/Sinhala language and literature/53",
+  "smmlu/easy/Sinhala language and literature/75",
+  "smmlu/easy/Sinhala language and literature/93",
+  "smmlu/easy/Sinhala language and literature/115",
+  "smmlu/easy/Sinhala language and literature/43",
+  "smmlu/easy/Sinhala language and literature/69",
+  "smmlu/easy/Sinhala language and literature/96",
+  "smmlu/easy/Sinhala language and literature/137",
+  "smmlu/easy/Sinhala language and literature/60",
+  "smmlu/easy/Sinhala language and literature/9",
+  "smmlu/easy/Sinhala language and literature/70",
+  "smmlu/easy/Sinhala language and literature/82",
+  "smmlu/easy/Sinhala language and literature/28",
+  "smmlu/easy/Sinhala language and literature/122",
+  "smmlu/easy/Sinhala language and literature/107",
+  "smmlu/easy/Sinhala language and literature/59",
+  "smmlu/easy/Sinhala language and literature/95",
+  "smmlu/easy/Sinhala language and literature/81",
+  "smmlu/easy/Sinhala language and literature/99",
+  "smmlu/easy/Sinhala language and literature/22",
+  "smmlu/easy/dancing/93",
+  "smmlu/easy/dancing/47",
+  "smmlu/easy/dancing/39",
+  "smmlu/easy/dancing/34",
+  "smmlu/easy/dancing/11",
+  "smmlu/easy/dancing/94",
+  "smmlu/easy/dancing/75",
+  "smmlu/easy/dancing/81",
+  "smmlu/easy/dancing/102",
+  "smmlu/easy/dancing/19",
+  "smmlu/easy/dancing/4",
+  "smmlu/easy/dancing/1",
+  "smmlu/easy/dancing/10",
+  "smmlu/easy/dancing/9",
+  "smmlu/easy/dancing/72",
+  "smmlu/easy/dancing/50",
+  "smmlu/easy/dancing/17",
+  "smmlu/easy/dancing/40",
+  "smmlu/easy/dancing/22",
+  "smmlu/easy/dancing/57",
+  "smmlu/easy/dancing/103",
+  "smmlu/easy/dancing/37",
+  "smmlu/easy/drama and Theatre /90",
+  "smmlu/easy/drama and Theatre /66",
+  "smmlu/easy/drama and Theatre /33",
+  "smmlu/easy/drama and Theatre /39",
+  "smmlu/easy/drama and Theatre /102",
+  "smmlu/easy/drama and Theatre /2",
+  "smmlu/easy/drama and Theatre /89",
+  "smmlu/easy/drama and Theatre /34",
+  "smmlu/easy/drama and Theatre /10",
+  "smmlu/easy/drama and Theatre /5",
+  "smmlu/easy/drama and Theatre /29",
+  "smmlu/easy/drama and Theatre /24",
+  "smmlu/easy/drama and Theatre /12",
+  "smmlu/easy/drama and Theatre /111",
+  "smmlu/easy/drama and Theatre /125",
+  "smmlu/easy/drama and Theatre /104",
+  "smmlu/easy/drama and Theatre /49",
+  "smmlu/easy/drama and Theatre /25",
+  "smmlu/easy/drama and Theatre /17",
+  "smmlu/easy/drama and Theatre /78",
+  "smmlu/easy/drama and Theatre /13",
+  "smmlu/easy/drama and Theatre /19",
+  "smmlu/easy/drama and Theatre /79",
+  "smmlu/easy/drama and Theatre /45",
+  "smmlu/easy/drama and Theatre /63",
+  "smmlu/easy/drama and Theatre /88",
+  "smmlu/easy/science/157",
+  "smmlu/easy/science/140",
+  "smmlu/easy/science/133",
+  "smmlu/easy/science/9",
+  "smmlu/easy/science/50",
+  "smmlu/easy/science/37",
+  "smmlu/easy/science/34",
+  "smmlu/easy/science/76",
+  "smmlu/easy/science/156",
+  "smmlu/easy/science/14",
+  "smmlu/easy/science/38",
+  "smmlu/easy/science/158",
+  "smmlu/easy/science/64",
+  "smmlu/easy/science/47",
+  "smmlu/easy/science/114",
+  "smmlu/easy/science/164",
+  "smmlu/easy/science/86",
+  "smmlu/easy/science/54",
+  "smmlu/easy/science/137",
+  "smmlu/easy/science/101",
+  "smmlu/easy/science/56",
+  "smmlu/easy/science/53",
+  "smmlu/easy/science/84",
+  "smmlu/easy/science/74",
+  "smmlu/easy/science/129",
+  "smmlu/easy/science/45",
+  "smmlu/easy/science/15",
+  "smmlu/easy/science/3",
+  "smmlu/easy/science/11",
+  "smmlu/easy/science/162",
+  "smmlu/easy/science/44",
+  "smmlu/easy/science/124"
+ ],
+ [
+  "smmlu/easy/Arts/81",
+  "smmlu/easy/Arts/33",
+  "smmlu/easy/Arts/15",
+  "smmlu/easy/Arts/61",
+  "smmlu/easy/Arts/41",
+  "smmlu/easy/Arts/23",
+  "smmlu/easy/Arts/51",
+  "smmlu/easy/Arts/72",
+  "smmlu/easy/Arts/60",
+  "smmlu/easy/Arts/38",
+  "smmlu/easy/Arts/9",
+  "smmlu/easy/Arts/58",
+  "smmlu/easy/Arts/40",
+  "smmlu/easy/Arts/78",
+  "smmlu/easy/Arts/65",
+  "smmlu/easy/Arts/95",
+  "smmlu/easy/Arts/80",
+  "smmlu/easy/Arts/100",
+  "smmlu/easy/Arts/89",
+  "smmlu/easy/Arts/1",
+  "smmlu/easy/Buddhism/102",
+  "smmlu/easy/Buddhism/18",
+  "smmlu/easy/Buddhism/30",
+  "smmlu/easy/Buddhism/70",
+  "smmlu/easy/Buddhism/9",
+  "smmlu/easy/Buddhism/52",
+  "smmlu/easy/Buddhism/31",
+  "smmlu/easy/Buddhism/40",
+  "smmlu/easy/Buddhism/110",
+  "smmlu/easy/Buddhism/129",
+  "smmlu/easy/Buddhism/3",
+  "smmlu/easy/Buddhism/41",
+  "smmlu/easy/Buddhism/74",
+  "smmlu/easy/Buddhism/39",
+  "smmlu/easy/Buddhism/80",
+  "smmlu/easy/Buddhism/26",
+  "smmlu/easy/Buddhism/118",
+  "smmlu/easy/Buddhism/137",
+  "smmlu/easy/Buddhism/79",
+  "smmlu/easy/Buddhism/59",
+  "smmlu/easy/Buddhism/86",
+  "smmlu/easy/Buddhism/66",
+  "smmlu/easy/Buddhism/121",
+  "smmlu/easy/Buddhism/128",
+  "smmlu/easy/Buddhism/78",
+  "smmlu/easy/Buddhism/73",
+  "smmlu/easy/Buddhism/11",
+  "smmlu/easy/Buddhism/56",
+  "smmlu/easy/Buddhism/123",
+  "smmlu/easy/Buddhism/65",
+  "smmlu/easy/Buddhism/49",
+  "smmlu/easy/Buddhism/96",
+  "smmlu/easy/Buddhism/61",
+  "smmlu/easy/Catholicism/75",
+  "smmlu/easy/Catholicism/62",
+  "smmlu/easy/Catholicism/87",
+  "smmlu/easy/Catholicism/1",
+  "smmlu/easy/Catholicism/119",
+  "smmlu/easy/Catholicism/98",
+  "smmlu/easy/Catholicism/58",
+  "smmlu/easy/Catholicism/46",
+  "smmlu/easy/Catholicism/83",
+  "smmlu/easy/Catholicism/26",
+  "smmlu/easy/Catholicism/67",
+  "smmlu/easy/Catholicism/34",
+  "smmlu/easy/Catholicism/37",
+  "smmlu/easy/Catholicism/100",
+  "smmlu/easy/Catholicism/69",
+  "smmlu/easy/Catholicism/77",
+  "smmlu/easy/Catholicism/116",
+  "smmlu/easy/Catholicism/102",
+  "smmlu/easy/Catholicism/128",
+  "smmlu/easy/Catholicism/48",
+  "smmlu/easy/Catholicism/32",
+  "smmlu/easy/Catholicism/122",
+  "smmlu/easy/Catholicism/99",
+  "smmlu/easy/Catholicism/90",
+  "smmlu/easy/Catholicism/96",
+  "smmlu/easy/Catholicism/60",
+  "smmlu/easy/Christianity/26",
+  "smmlu/easy/Christianity/39",
+  "smmlu/easy/Christianity/62",
+  "smmlu/easy/Christianity/76",
+  "smmlu/easy/Christianity/33",
+  "smmlu/easy/Christianity/114",
+  "smmlu/easy/Christianity/117",
+  "smmlu/easy/Christianity/61",
+  "smmlu/easy/Christianity/55",
+  "smmlu/easy/Christianity/31",
+  "smmlu/easy/Christianity/51",
+  "smmlu/easy/Christianity/3",
+  "smmlu/easy/Christianity/47",
+  "smmlu/easy/Christianity/63",
+  "smmlu/easy/Christianity/22",
+  "smmlu/easy/Christianity/126",
+  "smmlu/easy/Christianity/34",
+  "smmlu/easy/Christianity/15",
+  "smmlu/easy/Christianity/5",
+  "smmlu/easy/Christianity/84",
+  "smmlu/easy/Christianity/18",
+  "smmlu/easy/Christianity/71",
+  "smmlu/easy/Christianity/66",
+  "smmlu/easy/Christianity/92",
+  "smmlu/easy/Christianity/87",
+  "smmlu/easy/Christianity/14",
+  "smmlu/easy/Civics/88",
+  "smmlu/easy/Civics/2",
+  "smmlu/easy/Civics/11",
+  "smmlu/easy/Civics/100",
+  "smmlu/easy/Civics/110",
+  "smmlu/easy/Civics/121",
+  "smmlu/easy/Civics/47",
+  "smmlu/easy/Civics/22",
+  "smmlu/easy/Civics/49",
+  "smmlu/easy/Civics/90",
+  "smmlu/easy/Civics/28",
+  "smmlu/easy/Civics/65",
+  "smmlu/easy/Civics/6",
+  "smmlu/easy/Civics/34",
+  "smmlu/easy/Civics/55",
+  "smmlu/easy/Civics/102",
+  "smmlu/easy/Civics/85",
+  "smmlu/easy/Civics/93",
+  "smmlu/easy/Civics/48",
+  "smmlu/easy/Civics/58",
+  "smmlu/easy/Civics/16",
+  "smmlu/easy/Civics/20",
+  "smmlu/easy/Civics/33",
+  "smmlu/easy/Civics/54",
+  "smmlu/easy/Civics/23",
+  "smmlu/easy/Eastern music/139",
+  "smmlu/easy/Eastern music/47",
+  "smmlu/easy/Eastern music/105",
+  "smmlu/easy/Eastern music/15",
+  "smmlu/easy/Eastern music/70",
+  "smmlu/easy/Eastern music/136",
+  "smmlu/easy/Eastern music/61",
+  "smmlu/easy/Eastern music/134",
+  "smmlu/easy/Eastern music/13",
+  "smmlu/easy/Eastern music/40",
+  "smmlu/easy/Eastern music/116",
+  "smmlu/easy/Eastern_music/1",
+  "smmlu/easy/Eastern music/82",
+  "smmlu/easy/Eastern music/79",
+  "smmlu/easy/Eastern music/102",
+  "smmlu/easy/Eastern music/94",
+  "smmlu/easy/Eastern music/142",
+  "smmlu/easy/Eastern music/18",
+  "smmlu/easy/Eastern music/123",
+  "smmlu/easy/Eastern music/74",
+  "smmlu/easy/Eastern music/133",
+  "smmlu/easy/Eastern music/135",
+  "smmlu/easy/Eastern music/37",
+  "smmlu/easy/Eastern music/30",
+  "smmlu/easy/Eastern music/138",
+  "smmlu/easy/Eastern music/101",
+  "smmlu/easy/Eastern music/103",
+  "smmlu/easy/Eastern music/10",
+  "smmlu/easy/Geography/91",
+  "smmlu/easy/Geography/17",
+  "smmlu/easy/Geography/37",
+  "smmlu/easy/Geography/58",
+  "smmlu/easy/Geography/113",
+  "smmlu/easy/Geography/20",
+  "smmlu/easy/Geography/54",
+  "smmlu/easy/Geography/98",
+  "smmlu/easy/Geography/33",
+  "smmlu/easy/Geography/76",
+  "smmlu/easy/Geography/38",
+  "smmlu/easy/Geography/62",
+  "smmlu/easy/Geography/117",
+  "smmlu/easy/Geography/100",
+  "smmlu/easy/Geography/11",
+  "smmlu/easy/Geography/99",
+  "smmlu/easy/Geography/60",
+  "smmlu/easy/Geography/13",
+  "smmlu/easy/Geography/18",
+  "smmlu/easy/Geography/10",
+  "smmlu/easy/Geography/94",
+  "smmlu/easy/Geography/78",
+  "smmlu/easy/Geography/93",
+  "smmlu/easy/Health and Physical Science/110",
+  "smmlu/easy/Health and Physical Science/60",
+  "smmlu/easy/Health and Physical Science/99",
+  "smmlu/easy/Health and Physical Science/73",
+  "smmlu/easy/Health and Physical Science/31",
+  "smmlu/easy/Health and Physical Science/39",
+  "smmlu/easy/Health and Physical Science/80",
+  "smmlu/easy/Health and Physical Science/37",
+  "smmlu/easy/Health and Physical Science/106",
+  "smmlu/easy/Health and Physical Science/76",
+  "smmlu/easy/Health and Physical Science/44",
+  "smmlu/easy/Health and Physical Science/16",
+  "smmlu/easy/Health and Physical Science/122",
+  "smmlu/easy/Health and Physical Science/30",
+  "smmlu/easy/Health and Physical Science/74",
+  "smmlu/easy/Health and Physical Science/41",
+  "smmlu/easy/Health and Physical Science/46",
+  "smmlu/easy/Health and Physical Science/36",
+  "smmlu/easy/Health and Physical Science/102",
+  "smmlu/easy/Health and Physical Science/21",
+  "smmlu/easy/Health and Physical Science/105",
+  "smmlu/easy/Health and Physical Science/51",
+  "smmlu/easy/Health and Physical Science/93",
+  "smmlu/easy/Health and Physical Science/58",
+  "smmlu/easy/Health and Physical Science/129",
+  "smmlu/easy/Health and Physical Science/48",
+  "smmlu/easy/History/124",
+  "smmlu/easy/History/119",
+  "smmlu/easy/History/11",
+  "smmlu/easy/History/29",
+  "smmlu/easy/History/6",
+  "smmlu/easy/History/19",
+  "smmlu/easy/History/107",
+  "smmlu/easy/History/83",
+  "smmlu/easy/History/67",
+  "smmlu/easy/History/14",
+  "smmlu/easy/History/95",
+  "smmlu/easy/History/81",
+  "smmlu/easy/History/10",
+  "smmlu/easy/History/39",
+  "smmlu/easy/History/136",
+  "smmlu/easy/History/74",
+  "smmlu/easy/History/68",
+  "smmlu/easy/History/75",
+  "smmlu/easy/History/65",
+  "smmlu/easy/History/4",
+  "smmlu/easy/History/35",
+  "smmlu/easy/History/70",
+  "smmlu/easy/History/36",
+  "smmlu/easy/History/47",
+  "smmlu/easy/History/114",
+  "smmlu/easy/History/27",
+  "smmlu/easy/History/127",
+  "smmlu/easy/History/90",
+  "smmlu/easy/History/18",
+  "smmlu/easy/History/140",
+  "smmlu/easy/History/142",
+  "smmlu/easy/Islam/33",
+  "smmlu/easy/Islam/15",
+  "smmlu/easy/Islam/36",
+  "smmlu/easy/Islam/91",
+  "smmlu/easy/Islam/7",
+  "smmlu/easy/Islam/103",
+  "smmlu/easy/Islam/44",
+  "smmlu/easy/Islam/3",
+  "smmlu/easy/Islam/34",
+  "smmlu/easy/Islam/30",
+  "smmlu/easy/Islam/66",
+  "smmlu/easy/Islam/8",
+  "smmlu/easy/Islam/4",
+  "smmlu/easy/Islam/85",
+  "smmlu/easy/Islam/63",
+  "smmlu/easy/Islam/23",
+  "smmlu/easy/Islam/92",
+  "smmlu/easy/Islam/12",
+  "smmlu/easy/Islam/29",
+  "smmlu/easy/Islam/49",
+  "smmlu/easy/Islam/10",
+  "smmlu/easy/Sinhala language and literature/10",
+  "smmlu/easy/Sinhala language and literature/45",
+  "smmlu/easy/Sinhala language and literature/5",
+  "smmlu/easy/Sinhala_language_and_literature/1",
+  "smmlu/easy/Sinhala language and literature/67",
+  "smmlu/easy/Sinhala language and literature/47",
+  "smmlu/easy/Sinhala language and literature/92",
+  "smmlu/easy/Sinhala language and literature/63",
+  "smmlu/easy/Sinhala language and literature/142",
+  "smmlu/easy/Sinhala language and literature/73",
+  "smmlu/easy/Sinhala language and literature/143",
+  "smmlu/easy/Sinhala language and literature/8",
+  "smmlu/easy/Sinhala language and literature/57",
+  "smmlu/easy/Sinhala language and literature/80",
+  "smmlu/easy/Sinhala language and literature/25",
+  "smmlu/easy/Sinhala language and literature/134",
+  "smmlu/easy/Sinhala language and literature/85",
+  "smmlu/easy/Sinhala language and literature/42",
+  "smmlu/easy/Sinhala language and literature/50",
+  "smmlu/easy/Sinhala language and literature/117",
+  "smmlu/easy/Sinhala language and literature/2",
+  "smmlu/easy/Sinhala language and literature/133",
+  "smmlu/easy/Sinhala language and literature/17",
+  "smmlu/easy/Sinhala language and literature/153",
+  "smmlu/easy/Sinhala language and literature/55",
+  "smmlu/easy/Sinhala language and literature/49",
+  "smmlu/easy/Sinhala language and literature/20",
+  "smmlu/easy/Sinhala language and literature/105",
+  "smmlu/easy/Sinhala language and literature/109",
+  "smmlu/easy/Sinhala language and literature/40",
+  "smmlu/easy/Sinhala language and literature/121",
+  "smmlu/easy/dancing/33",
+  "smmlu/easy/dancing/35",
+  "smmlu/easy/dancing/48",
+  "smmlu/easy/dancing/27",
+  "smmlu/easy/dancing/58",
+  "smmlu/easy/dancing/79",
+  "smmlu/easy/dancing/87",
+  "smmlu/easy/dancing/23",
+  "smmlu/easy/dancing/83",
+  "smmlu/easy/dancing/92",
+  "smmlu/easy/dancing/80",
+  "smmlu/easy/dancing/108",
+  "smmlu/easy/dancing/78",
+  "smmlu/easy/dancing/64",
+  "smmlu/easy/dancing/20",
+  "smmlu/easy/dancing/90",
+  "smmlu/easy/dancing/36",
+  "smmlu/easy/dancing/14",
+  "smmlu/easy/dancing/30",
+  "smmlu/easy/dancing/99",
+  "smmlu/easy/dancing/104",
+  "smmlu/easy/dancing/38",
+  "smmlu/easy/drama and Theatre /71",
+  "smmlu/easy/drama_and_theatre/1",
+  "smmlu/easy/drama and Theatre /4",
+  "smmlu/easy/drama and Theatre /109",
+  "smmlu/easy/drama and Theatre /69",
+  "smmlu/easy/drama and Theatre /108",
+  "smmlu/easy/drama and Theatre /21",
+  "smmlu/easy/drama and Theatre /8",
+  "smmlu/easy/drama and Theatre /123",
+  "smmlu/easy/drama and Theatre /122",
+  "smmlu/easy/drama and Theatre /74",
+  "smmlu/easy/drama and Theatre /48",
+  "smmlu/easy/drama and Theatre /83",
+  "smmlu/easy/drama and Theatre /65",
+  "smmlu/easy/drama and Theatre /38",
+  "smmlu/easy/drama and Theatre /23",
+  "smmlu/easy/drama and Theatre /87",
+  "smmlu/easy/drama and Theatre /56",
+  "smmlu/easy/drama and Theatre /55",
+  "smmlu/easy/drama and Theatre /93",
+  "smmlu/easy/drama and Theatre /110",
+  "smmlu/easy/drama and Theatre /112",
+  "smmlu/easy/drama and Theatre /53",
+  "smmlu/easy/drama and Theatre /77",
+  "smmlu/easy/drama and Theatre /72",
+  "smmlu/easy/science/75",
+  "smmlu/easy/science/23",
+  "smmlu/easy/science/57",
+  "smmlu/easy/science/59",
+  "smmlu/easy/science/135",
+  "smmlu/easy/science/51",
+  "smmlu/easy/science/106",
+  "smmlu/easy/science/87",
+  "smmlu/easy/science/143",
+  "smmlu/easy/science/70",
+  "smmlu/easy/science/60",
+  "smmlu/easy/science/71",
+  "smmlu/easy/science/5",
+  "smmlu/easy/science/146",
+  "smmlu/easy/science/96",
+  "smmlu/easy/science/91",
+  "smmlu/easy/science/104",
+  "smmlu/easy/science/22",
+  "smmlu/easy/science/82",
+  "smmlu/easy/science/121",
+  "smmlu/easy/science/136",
+  "smmlu/easy/science/26",
+  "smmlu/easy/science/115",
+  "smmlu/easy/science/98",
+  "smmlu/easy/science/46",
+  "smmlu/easy/science/155",
+  "smmlu/easy/science/131",
+  "smmlu/easy/science/32",
+  "smmlu/easy/science/109",
+  "smmlu/easy/science/139",
+  "smmlu/easy/science/116",
+  "smmlu/easy/science/17",
+  "smmlu/easy/science/73"
+ ],
+ [
+  "smmlu/easy/Arts/25",
+  "smmlu/easy/Arts/98",
+  "smmlu/easy/Arts/85",
+  "smmlu/easy/Arts/74",
+  "smmlu/easy/Arts/94",
+  "smmlu/easy/Arts/43",
+  "smmlu/easy/Arts/52",
+  "smmlu/easy/Arts/91",
+  "smmlu/easy/Arts/17",
+  "smmlu/easy/Arts/75",
+  "smmlu/easy/Arts/70",
+  "smmlu/easy/Arts/88",
+  "smmlu/easy/Arts/56",
+  "smmlu/easy/Arts/71",
+  "smmlu/easy/Arts/67",
+  "smmlu/easy/Arts/26",
+  "smmlu/easy/Arts/83",
+  "smmlu/easy/Arts/36",
+  "smmlu/easy/Arts/86",
+  "smmlu/easy/Arts/21",
+  "smmlu/easy/Buddhism/10",
+  "smmlu/easy/Buddhism/63",
+  "smmlu/easy/Buddhism/23",
+  "smmlu/easy/Buddhism/42",
+  "smmlu/easy/Buddhism/155",
+  "smmlu/easy/Buddhism/112",
+  "smmlu/easy/Buddhism/108",
+  "smmlu/easy/Buddhism/101",
+  "smmlu/easy/Buddhism/98",
+  "smmlu/easy/Buddhism/145",
+  "smmlu/easy/Buddhism/29",
+  "smmlu/easy/Buddhism/76",
+  "smmlu/easy/Buddhism/17",
+  "smmlu/easy/Buddhism/22",
+  "smmlu/easy/Buddhism/25",
+  "smmlu/easy/Buddhism/136",
+  "smmlu/easy/Buddhism/24",
+  "smmlu/easy/Buddhism/153",
+  "smmlu/easy/Buddhism/14",
+  "smmlu/easy/Buddhism/68",
+  "smmlu/easy/Buddhism/140",
+  "smmlu/easy/Buddhism/34",
+  "smmlu/easy/Buddhism/95",
+  "smmlu/easy/Buddhism/2",
+  "smmlu/easy/Buddhism/8",
+  "smmlu/easy/Buddhism/119",
+  "smmlu/easy/Buddhism/148",
+  "smmlu/easy/Buddhism/27",
+  "smmlu/easy/Buddhism/32",
+  "smmlu/easy/Buddhism/122",
+  "smmlu/easy/Buddhism/141",
+  "smmlu/easy/Buddhism/72",
+  "smmlu/easy/Catholicism/123",
+  "smmlu/easy/Catholicism/61",
+  "smmlu/easy/Catholicism/27",
+  "smmlu/easy/Catholicism/110",
+  "smmlu/easy/Catholicism/10",
+  "smmlu/easy/Catholicism/113",
+  "smmlu/easy/Catholicism/105",
+  "smmlu/easy/Catholicism/89",
+  "smmlu/easy/Catholicism/14",
+  "smmlu/easy/Catholicism/50",
+  "smmlu/easy/Catholicism/23",
+  "smmlu/easy/Catholicism/28",
+  "smmlu/easy/Catholicism/8",
+  "smmlu/easy/Catholicism/35",
+  "smmlu/easy/Catholicism/72",
+  "smmlu/easy/Catholicism/36",
+  "smmlu/easy/Catholicism/3",
+  "smmlu/easy/Catholicism/80",
+  "smmlu/easy/Catholicism/25",
+  "smmlu/easy/Catholicism/127",
+  "smmlu/easy/Catholicism/40",
+  "smmlu/easy/Catholicism/17",
+  "smmlu/easy/Catholicism/5",
+  "smmlu/easy/Catholicism/66",
+  "smmlu/easy/Catholicism/132",
+  "smmlu/easy/Catholicism/94",
+  "smmlu/easy/Catholicism/56",
+  "smmlu/easy/Christianity/28",
+  "smmlu/easy/Christianity/75",
+  "smmlu/easy/Christianity/54",
+  "smmlu/easy/Christianity/2",
+  "smmlu/easy/Christianity/7",
+  "smmlu/easy/Christianity/101",
+  "smmlu/easy/Christianity/77",
+  "smmlu/easy/Christianity/83",
+  "smmlu/easy/Christianity/69",
+  "smmlu/easy/Christianity/130",
+  "smmlu/easy/Christianity/80",
+  "smmlu/easy/Christianity/121",
+  "smmlu/easy/Christianity/110",
+  "smmlu/easy/Christianity/37",
+  "smmlu/easy/Christianity/90",
+  "smmlu/easy/Christianity/8",
+  "smmlu/easy/Christianity/119",
+  "smmlu/easy/Christianity/19",
+  "smmlu/easy/Christianity/48",
+  "smmlu/easy/Christianity/44",
+  "smmlu/easy/Christianity/10",
+  "smmlu/easy/Christianity/29",
+  "smmlu/easy/Christianity/88",
+  "smmlu/easy/Christianity/104",
+  "smmlu/easy/Christianity/6",
+  "smmlu/easy/Christianity/27",
+  "smmlu/easy/Civics/113",
+  "smmlu/easy/Civics/51",
+  "smmlu/easy/Civics/15",
+  "smmlu/easy/Civics/103",
+  "smmlu/easy/Civics/64",
+  "smmlu/easy/Civics/114",
+  "smmlu/easy/Civics/30",
+  "smmlu/easy/Civics/73",
+  "smmlu/easy/Civics/76",
+  "smmlu/easy/Civics/68",
+  "smmlu/easy/Civics/112",
+  "smmlu/easy/Civics/53",
+  "smmlu/easy/Civics/63",
+  "smmlu/easy/Civics/101",
+  "smmlu/easy/Civics/29",
+  "smmlu/easy/Civics/60",
+  "smmlu/easy/Civics/80",
+  "smmlu/easy/Civics/18",
+  "smmlu/easy/Civics/27",
+  "smmlu/easy/Civics/4",
+  "smmlu/easy/Civics/59",
+  "smmlu/easy/Civics/87",
+  "smmlu/easy/Civics/31",
+  "smmlu/easy/Civics/70",
+  "smmlu/easy/Eastern music/121",
+  "smmlu/easy/Eastern music/49",
+  "smmlu/easy/Eastern music/57",
+  "smmlu/easy/Eastern music/31",
+  "smmlu/easy/Eastern music/33",
+  "smmlu/easy/Eastern music/9",
+  "smmlu/easy/Eastern music/23",
+  "smmlu/easy/Eastern music/140",
+  "smmlu/easy/Eastern music/118",
+  "smmlu/easy/Eastern music/117",
+  "smmlu/easy/Eastern music/63",
+  "smmlu/easy/Eastern music/68",
+  "smmlu/easy/Eastern music/52",
+  "smmlu/easy/Eastern music/26",
+  "smmlu/easy/Eastern music/131",
+  "smmlu/easy/Eastern music/29",
+  "smmlu/easy/Eastern music/3",
+  "smmlu/easy/Eastern music/108",
+  "smmlu/easy/Eastern music/112",
+  "smmlu/easy/Eastern music/84",
+  "smmlu/easy/Eastern music/93",
+  "smmlu/easy/Eastern music/87",
+  "smmlu/easy/Eastern music/27",
+  "smmlu/easy/Eastern music/6",
+  "smmlu/easy/Eastern music/38",
+  "smmlu/easy/Eastern music/19",
+  "smmlu/easy/Eastern music/25",
+  "smmlu/easy/Eastern music/22",
+  "smmlu/easy/Eastern music/66",
+  "smmlu/easy/Geography/43",
+  "smmlu/easy/Geography/112",
+  "smmlu/easy/Geography/50",
+  "smmlu/easy/Geography/35",
+  "smmlu/easy/Geography/67",
+  "smmlu/easy/Geography/80",
+  "smmlu/easy/Geography/48",
+  "smmlu/easy/Geography/39",
+  "smmlu/easy/Geography/12",
+  "smmlu/easy/Geography/70",
+  "smmlu/easy/Geography/74",
+  "smmlu/easy/Geography/47",
+  "smmlu/easy/Geography/41",
+  "smmlu/easy/Geography/28",
+  "smmlu/easy/Geography/77",
+  "smmlu/easy/Geography/8",
+  "smmlu/easy/Geography/51",
+  "smmlu/easy/Geography/57",
+  "smmlu/easy/Geography/82",
+  "smmlu/easy/Geography/9",
+  "smmlu/easy/Geography/83",
+  "smmlu/easy/Geography/31",
+  "smmlu/easy/Geography/68",
+  "smmlu/easy/Health and Physical Science/95",
+  "smmlu/easy/Health and Physical Science/69",
+  "smmlu/easy/Health and Physical Science/84",
+  "smmlu/easy/Health and Physical Science/43",
+  "smmlu/easy/Health and Physical Science/19",
+  "smmlu/easy/Health and Physical Science/5",
+  "smmlu/easy/Health and Physical Science/124",
+  "smmlu/easy/Health and Physical Science/12",
+  "smmlu/easy/Health and Physical Science/45",
+  "smmlu/easy/Health and Physical Science/101",
+  "smmlu/easy/Health and Physical Science/119",
+  "smmlu/easy/Health and Physical Science/89",
+  "smmlu/easy/Health and Physical Science/59",
+  "smmlu/easy/Health and Physical Science/42",
+  "smmlu/easy/Health and Physical Science/6",
+  "smmlu/easy/Health and Physical Science/33",
+  "smmlu/easy/Health and Physical Science/18",
+  "smmlu/easy/Health and Physical Science/120",
+  "smmlu/easy/Health and Physical Science/4",
+  "smmlu/easy/Health and Physical Science/104",
+  "smmlu/easy/Health and Physical Science/34",
+  "smmlu/easy/Health and Physical Science/116",
+  "smmlu/easy/Health and Physical Science/52",
+  "smmlu/easy/Health and Physical Science/62",
+  "smmlu/easy/Health and Physical Science/130",
+  "smmlu/easy/Health and Physical Science/64",
+  "smmlu/easy/History/57",
+  "smmlu/easy/History/115",
+  "smmlu/easy/History/62",
+  "smmlu/easy/History/20",
+  "smmlu/easy/History/30",
+  "smmlu/easy/History/38",
+  "smmlu/easy/History/113",
+  "smmlu/easy/History/105",
+  "smmlu/easy/History/101",
+  "smmlu/easy/History/5",
+  "smmlu/easy/History/102",
+  "smmlu/easy/History/26",
+  "smmlu/easy/History/66",
+  "smmlu/easy/History/9",
+  "smmlu/easy/History/85",
+  "smmlu/easy/History/149",
+  "smmlu/easy/History/137",
+  "smmlu/easy/History/32",
+  "smmlu/easy/History/69",
+  "smmlu/easy/History/141",
+  "smmlu/easy/History/133",
+  "smmlu/easy/History/106",
+  "smmlu/easy/History/82",
+  "smmlu/easy/History/33",
+  "smmlu/easy/History/78",
+  "smmlu/easy/History/64",
+  "smmlu/easy/History/129",
+  "smmlu/easy/History/135",
+  "smmlu/easy/History/110",
+  "smmlu/easy/History/97",
+  "smmlu/easy/Islam/9",
+  "smmlu/easy/Islam/73",
+  "smmlu/easy/Islam/39",
+  "smmlu/easy/Islam/80",
+  "smmlu/easy/Islam/42",
+  "smmlu/easy/Islam/106",
+  "smmlu/easy/Islam/64",
+  "smmlu/easy/Islam/37",
+  "smmlu/easy/Islam/72",
+  "smmlu/easy/Islam/38",
+  "smmlu/easy/Islam/5",
+  "smmlu/easy/Islam/6",
+  "smmlu/easy/Islam/79",
+  "smmlu/easy/Islam/97",
+  "smmlu/easy/Islam/88",
+  "smmlu/easy/Islam/13",
+  "smmlu/easy/Islam/101",
+  "smmlu/easy/Islam/75",
+  "smmlu/easy/Islam/74",
+  "smmlu/easy/Islam/53",
+  "smmlu/easy/Islam/71",
+  "smmlu/easy/Islam/86",
+  "smmlu/easy/Sinhala language and literature/72",
+  "smmlu/easy/Sinhala language and literature/65",
+  "smmlu/easy/Sinhala language and literature/140",
+  "smmlu/easy/Sinhala language and literature/126",
+  "smmlu/easy/Sinhala language and literature/56",
+  "smmlu/easy/Sinhala language and literature/83",
+  "smmlu/easy/Sinhala language and literature/26",
+  "smmlu/easy/Sinhala language and literature/11",
+  "smmlu/easy/Sinhala language and literature/150",
+  "smmlu/easy/Sinhala language and literature/61",
+  "smmlu/easy/Sinhala language and literature/33",
+  "smmlu/easy/Sinhala language and literature/118",
+  "smmlu/easy/Sinhala language and literature/102",
+  "smmlu/easy/Sinhala language and literature/100",
+  "smmlu/easy/Sinhala language and literature/97",
+  "smmlu/easy/Sinhala language and literature/144",
+  "smmlu/easy/Sinhala language and literature/113",
+  "smmlu/easy/Sinhala language and literature/151",
+  "smmlu/easy/Sinhala language and literature/78",
+  "smmlu/easy/Sinhala language and literature/34",
+  "smmlu/easy/Sinhala language and literature/103",
+  "smmlu/easy/Sinhala language and literature/132",
+  "smmlu/easy/Sinhala language and literature/37",
+  "smmlu/easy/Sinhala language and literature/110",
+  "smmlu/easy/Sinhala language and literature/108",
+  "smmlu/easy/Sinhala language and literature/74",
+  "smmlu/easy/Sinhala language and literature/141",
+  "smmlu/easy/Sinhala language and literature/58",
+  "smmlu/easy/Sinhala language and literature/71",
+  "smmlu/easy/Sinhala language and literature/131",
+  "smmlu/easy/Sinhala language and literature/36",
+  "smmlu/easy/dancing/105",
+  "smmlu/easy/dancing/3",
+  "smmlu/easy/dancing/26",
+  "smmlu/easy/dancing/70",
+  "smmlu/easy/dancing/68",
+  "smmlu/easy/dancing/42",
+  "smmlu/easy/dancing/5",
+  "smmlu/easy/dancing/65",
+  "smmlu/easy/dancing/24",
+  "smmlu/easy/dancing/13",
+  "smmlu/easy/dancing/60",
+  "smmlu/easy/dancing/54",
+  "smmlu/easy/dancing/97",
+  "smmlu/easy/dancing/41",
+  "smmlu/easy/dancing/88",
+  "smmlu/easy/dancing/67",
+  "smmlu/easy/dancing/43",
+  "smmlu/easy/dancing/107",
+  "smmlu/easy/dancing/28",
+  "smmlu/easy/dancing/100",
+  "smmlu/easy/dancing/77",
+  "smmlu/easy/dancing/106",
+  "smmlu/easy/drama and Theatre /32",
+  "smmlu/easy/drama and Theatre /126",
+  "smmlu/easy/drama and Theatre /100",
+  "smmlu/easy/drama and Theatre /59",
+  "smmlu/easy/drama and Theatre /44",
+  "smmlu/easy/drama and Theatre /85",
+  "smmlu/easy/drama and Theatre /99",
+  "smmlu/easy/drama and Theatre /116",
+  "smmlu/easy/drama and Theatre /11",
+  "smmlu/easy/drama and Theatre /97",
+  "smmlu/easy/drama and Theatre /75",
+  "smmlu/easy/drama and Theatre /96",
+  "smmlu/easy/drama and Theatre /9",
+  "smmlu/easy/drama and Theatre /103",
+  "smmlu/easy/drama and Theatre /84",
+  "smmlu/easy/drama and Theatre /95",
+  "smmlu/easy/drama and Theatre /121",
+  "smmlu/easy/drama and Theatre /62",
+  "smmlu/easy/drama and Theatre /82",
+  "smmlu/easy/drama and Theatre /81",
+  "smmlu/easy/drama and Theatre /27",
+  "smmlu/easy/drama and Theatre /14",
+  "smmlu/easy/drama and Theatre /36",
+  "smmlu/easy/drama and Theatre /113",
+  "smmlu/easy/drama and Theatre /46",
+  "smmlu/easy/science/118",
+  "smmlu/easy/science/134",
+  "smmlu/easy/science/145",
+  "smmlu/easy/science/25",
+  "smmlu/easy/science/2",
+  "smmlu/easy/science/152",
+  "smmlu/easy/science/120",
+  "smmlu/easy/science/6",
+  "smmlu/easy/science/111",
+  "smmlu/easy/science/105",
+  "smmlu/easy/science/147",
+  "smmlu/easy/science/31",
+  "smmlu/easy/science/58",
+  "smmlu/easy/science/35",
+  "smmlu/easy/science/88",
+  "smmlu/easy/science/130",
+  "smmlu/easy/science/48",
+  "smmlu/easy/science/83",
+  "smmlu/easy/science/28",
+  "smmlu/easy/science/113",
+  "smmlu/easy/science/149",
+  "smmlu/easy/science/95",
+  "smmlu/easy/science/72",
+  "smmlu/easy/science/7",
+  "smmlu/easy/science/61",
+  "smmlu/easy/science/21",
+  "smmlu/easy/science/67",
+  "smmlu/easy/science/19",
+  "smmlu/easy/science/112",
+  "smmlu/easy/science/163",
+  "smmlu/easy/science/62",
+  "smmlu/easy/science/102",
+  "smmlu/easy/science/27"
+ ],
+ [
+  "smmlu/easy/Arts/32",
+  "smmlu/easy/Arts/14",
+  "smmlu/easy/Arts/27",
+  "smmlu/easy/Arts/34",
+  "smmlu/easy/Arts/50",
+  "smmlu/easy/Arts/45",
+  "smmlu/easy/Arts/20",
+  "smmlu/easy/Arts/92",
+  "smmlu/easy/Arts/28",
+  "smmlu/easy/Arts/47",
+  "smmlu/easy/Arts/22",
+  "smmlu/easy/Arts/79",
+  "smmlu/easy/Arts/4",
+  "smmlu/easy/Arts/82",
+  "smmlu/easy/Arts/10",
+  "smmlu/easy/Arts/62",
+  "smmlu/easy/Arts/49",
+  "smmlu/easy/Arts/5",
+  "smmlu/easy/Arts/12",
+  "smmlu/easy/Arts/6",
+  "smmlu/easy/Buddhism/46",
+  "smmlu/easy/Buddhism/125",
+  "smmlu/easy/Buddhism/48",
+  "smmlu/easy/Buddhism/12",
+  "smmlu/easy/Buddhism/21",
+  "smmlu/easy/Buddhism/100",
+  "smmlu/easy/Buddhism/91",
+  "smmlu/easy/Buddhism/82",
+  "smmlu/easy/Buddhism/50",
+  "smmlu/easy/Buddhism/152",
+  "smmlu/easy/Buddhism/43",
+  "smmlu/easy/Buddhism/94",
+  "smmlu/easy/Buddhism/38",
+  "smmlu/easy/Buddhism/57",
+  "smmlu/easy/Buddhism/109",
+  "smmlu/easy/Buddhism/6",
+  "smmlu/easy/Buddhism/104",
+  "smmlu/easy/Buddhism/92",
+  "smmlu/easy/Buddhism/105",
+  "smmlu/easy/Buddhism/147",
+  "smmlu/easy/Buddhism/45",
+  "smmlu/easy/Buddhism/120",
+  "smmlu/easy/Buddhism/81",
+  "smmlu/easy/Buddhism/106",
+  "smmlu/easy/Buddhism/33",
+  "smmlu/easy/Buddhism/15",
+  "smmlu/easy/Buddhism/44",
+  "smmlu/easy/Buddhism/134",
+  "smmlu/easy/Buddhism/143",
+  "smmlu/easy/Buddhism/75",
+  "smmlu/easy/Buddhism/127",
+  "smmlu/easy/Buddhism/54",
+  "smmlu/easy/Catholicism/81",
+  "smmlu/easy/Catholicism/24",
+  "smmlu/easy/Catholicism/126",
+  "smmlu/easy/Catholicism/31",
+  "smmlu/easy/Catholicism/29",
+  "smmlu/easy/Catholicism/109",
+  "smmlu/easy/Catholicism/21",
+  "smmlu/easy/Catholicism/85",
+  "smmlu/easy/Catholicism/33",
+  "smmlu/easy/Catholicism/54",
+  "smmlu/easy/Catholicism/65",
+  "smmlu/easy/Catholicism/51",
+  "smmlu/easy/Catholicism/73",
+  "smmlu/easy/Catholicism/117",
+  "smmlu/easy/Catholicism/38",
+  "smmlu/easy/Catholicism/129",
+  "smmlu/easy/Catholicism/20",
+  "smmlu/easy/Catholicism/79",
+  "smmlu/easy/Catholicism/101",
+  "smmlu/easy/Catholicism/112",
+  "smmlu/easy/Catholicism/68",
+  "smmlu/easy/Catholicism/91",
+  "smmlu/easy/Catholicism/6",
+  "smmlu/easy/Catholicism/74",
+  "smmlu/easy/Catholicism/9",
+  "smmlu/easy/Catholicism/59",
+  "smmlu/easy/Catholicism/130",
+  "smmlu/easy/Christianity/9",
+  "smmlu/easy/Christianity/107",
+  "smmlu/easy/Christianity/45",
+  "smmlu/easy/Christianity/108",
+  "smmlu/easy/Christianity/100",
+  "smmlu/easy/Christianity/93",
+  "smmlu/easy/Christianity/46",
+  "smmlu/easy/Christianity/36",
+  "smmlu/easy/Christianity/21",
+  "smmlu/easy/Christianity/73",
+  "smmlu/easy/Christianity/89",
+  "smmlu/easy/Christianity/99",
+  "smmlu/easy/Christianity/64",
+  "smmlu/easy/Christianity/38",
+  "smmlu/easy/Christianity/59",
+  "smmlu/easy/Christianity/70",
+  "smmlu/easy/Christianity/56",
+  "smmlu/easy/Christianity/12",
+  "smmlu/easy/Christianity/43",
+  "smmlu/easy/Christianity/112",
+  "smmlu/easy/Christianity/81",
+  "smmlu/easy/Christianity/98",
+  "smmlu/easy/Christianity/23",
+  "smmlu/easy/Christianity/68",
+  "smmlu/easy/Christianity/113",
+  "smmlu/easy/Christianity/30",
+  "smmlu/easy/Civics/104",
+  "smmlu/easy/Civics/3",
+  "smmlu/easy/Civics/91",
+  "smmlu/easy/Civics/52",
+  "smmlu/easy/Civics/44",
+  "smmlu/easy/Civics/74",
+  "smmlu/easy/Civics/99",
+  "smmlu/easy/Civics/109",
+  "smmlu/easy/Civics/72",
+  "smmlu/easy/Civics/78",
+  "smmlu/easy/Civics/69",
+  "smmlu/easy/Civics/98",
+  "smmlu/easy/Civics/107",
+  "smmlu/easy/Civics/75",
+  "smmlu/easy/Civics/50",
+  "smmlu/easy/Civics/35",
+  "smmlu/easy/Civics/82",
+  "smmlu/easy/Civics/9",
+  "smmlu/easy/Civics/46",
+  "smmlu/easy/Civics/81",
+  "smmlu/easy/Civics/24",
+  "smmlu/easy/Civics/57",
+  "smmlu/easy/Civics/118",
+  "smmlu/easy/Civics/38",
+  "smmlu/easy/Eastern music/115",
+  "smmlu/easy/Eastern music/17",
+  "smmlu/easy/Eastern music/110",
+  "smmlu/easy/Eastern music/86",
+  "smmlu/easy/Eastern music/120",
+  "smmlu/easy/Eastern music/48",
+  "smmlu/easy/Eastern music/42",
+  "smmlu/easy/Eastern music/32",
+  "smmlu/easy/Eastern music/141",
+  "smmlu/easy/Eastern music/137",
+  "smmlu/easy/Eastern music/45",
+  "smmlu/easy/Eastern music/85",
+  "smmlu/easy/Eastern music/11",
+  "smmlu/easy/Eastern music/64",
+  "smmlu/easy/Eastern music/106",
+  "smmlu/easy/Eastern music/130",
+  "smmlu/easy/Eastern music/107",
+  "smmlu/easy/Eastern music/41",
+  "smmlu/easy/Eastern music/81",
+  "smmlu/easy/Eastern music/16",
+  "smmlu/easy/Eastern music/113",
+  "smmlu/easy/Eastern music/88",
+  "smmlu/easy/Eastern music/36",
+  "smmlu/easy/Eastern music/39",
+  "smmlu/easy/Eastern music/46",
+  "smmlu/easy/Eastern music/28",
+  "smmlu/easy/Eastern music/72",
+  "smmlu/easy/Eastern music/143",
+  "smmlu/easy/Eastern music/5",
+  "smmlu/easy/Geography/84",
+  "smmlu/easy/Geography/55",
+  "smmlu/easy/Geography/5",
+  "smmlu/easy/Geography/97",
+  "smmlu/easy/Geography/115",
+  "smmlu/easy/Geography/116",
+  "smmlu/easy/Geography/42",
+  "smmlu/easy/Geography/104",
+  "smmlu/easy/Geography/30",
+  "smmlu/easy/Geography/2",
+  "smmlu/easy/Geography/59",
+  "smmlu/easy/Geography/72",
+  "smmlu/easy/Geography/101",
+  "smmlu/easy/Geography/110",
+  "smmlu/easy/Geography/79",
+  "smmlu/easy/Geography/96",
+  "smmlu/easy/Geography/24",
+  "smmlu/easy/Geography/87",
+  "smmlu/easy/Geography/61",
+  "smmlu/easy/Geography/45",
+  "smmlu/easy/Geography/23",
+  "smmlu/easy/Geography/95",
+  "smmlu/easy/Geography/114",
+  "smmlu/easy/Health and Physical Science/35",
+  "smmlu/easy/Health and Physical Science/66",
+  "smmlu/easy/Health and Physical Science/77",
+  "smmlu/easy/Health and Physical Science/123",
+  "smmlu/easy/Health and Physical Science/114",
+  "smmlu/easy/Health and Physical Science/79",
+  "smmlu/easy/Health and Physical Science/117",
+  "smmlu/easy/Health and Physical Science/78",
+  "smmlu/easy/Health and Physical Science/96",
+  "smmlu/easy/Health and Physical Science/109",
+  "smmlu/easy/Health and Physical Science/56",
+  "smmlu/easy/Health and Physical Science/88",
+  "smmlu/easy/Health and Physical Science/29",
+  "smmlu/easy/Health and Physical Science/23",
+  "smmlu/easy/Health and Physical Science/128",
+  "smmlu/easy/Health and Physical Science/13",
+  "smmlu/easy/Health and Physical Science/14",
+  "smmlu/easy/Health and Physical Science/81",
+  "smmlu/easy/Health and Physical Science/10",
+  "smmlu/easy/Health and Physical Science/24",
+  "smmlu/easy/Health and Physical Science/67",
+  "smmlu/easy/Health and Physical Science/118",
+  "smmlu/easy/Health and Physical Science/90",
+  "smmlu/easy/Health and Physical Science/113",
+  "smmlu/easy/Health and Physical Science/47",
+  "smmlu/easy/Health and Physical Science/91",
+  "smmlu/easy/History/54",
+  "smmlu/easy/History/16",
+  "smmlu/easy/History/109",
+  "smmlu/easy/History/46",
+  "smmlu/easy/History/126",
+  "smmlu/easy/History/117",
+  "smmlu/easy/History/146",
+  "smmlu/easy/History/88",
+  "smmlu/easy/History/15",
+  "smmlu/easy/History/77",
+  "smmlu/easy/History/24",
+  "smmlu/easy/History/139",
+  "smmlu/easy/History/56",
+  "smmlu/easy/History/13",
+  "smmlu/easy/History/22",
+  "smmlu/easy/History/112",
+  "smmlu/easy/History/21",
+  "smmlu/easy/History/58",
+  "smmlu/easy/History/98",
+  "smmlu/easy/History/23",
+  "smmlu/easy/History/42",
+  "smmlu/easy/History/84",
+  "smmlu/easy/History/55",
+  "smmlu/easy/History/128",
+  "smmlu/easy/History/52",
+  "smmlu/easy/History/122",
+  "smmlu/easy/History/132",
+  "smmlu/easy/History/120",
+  "smmlu/easy/History/71",
+  "smmlu/easy/History/49",
+  "smmlu/easy/Islam/87",
+  "smmlu/easy/Islam/54",
+  "smmlu/easy/Islam/21",
+  "smmlu/easy/Islam/100",
+  "smmlu/easy/Islam/16",
+  "smmlu/easy/Islam/57",
+  "smmlu/easy/Islam/94",
+  "smmlu/easy/Islam/17",
+  "smmlu/easy/Islam/41",
+  "smmlu/easy/Islam/76",
+  "smmlu/easy/Islam/59",
+  "smmlu/easy/Islam/27",
+  "smmlu/easy/Islam/58",
+  "smmlu/easy/Islam/47",
+  "smmlu/easy/Islam/14",
+  "smmlu/easy/Islam/40",
+  "smmlu/easy/Islam/65",
+  "smmlu/easy/Islam/25",
+  "smmlu/easy/Islam/28",
+  "smmlu/easy/Islam/11",
+  "smmlu/easy/Islam/70",
+  "smmlu/easy/Islam/51",
+  "smmlu/easy/Sinhala language and literature/104",
+  "smmlu/easy/Sinhala language and literature/54",
+  "smmlu/easy/Sinhala language and literature/155",
+  "smmlu/easy/Sinhala language and literature/3",
+  "smmlu/easy/Sinhala language and literature/101",
+  "smmlu/easy/Sinhala language and literature/30",
+  "smmlu/easy/Sinhala language and literature/91",
+  "smmlu/easy/Sinhala language and literature/12",
+  "smmlu/easy/Sinhala language and literature/147",
+  "smmlu/easy/Sinhala language and literature/124",
+  "smmlu/easy/Sinhala language and literature/21",
+  "smmlu/easy/Sinhala language and literature/90",
+  "smmlu/easy/Sinhala language and literature/114",
+  "smmlu/easy/Sinhala language and literature/48",
+  "smmlu/easy/Sinhala language and literature/120",
+  "smmlu/easy/Sinhala language and literature/76",
+  "smmlu/easy/Sinhala language and literature/146",
+  "smmlu/easy/Sinhala language and literature/128",
+  "smmlu/easy/Sinhala language and literature/46",
+  "smmlu/easy/Sinhala language and literature/135",
+  "smmlu/easy/Sinhala language and literature/87",
+  "smmlu/easy/Sinhala language and literature/52",
+  "smmlu/easy/Sinhala language and literature/16",
+  "smmlu/easy/Sinhala language and literature/152",
+  "smmlu/easy/Sinhala language and literature/84",
+  "smmlu/easy/Sinhala language and literature/27",
+  "smmlu/easy/Sinhala language and literature/29",
+  "smmlu/easy/Sinhala language and literature/88",
+  "smmlu/easy/Sinhala language and literature/79",
+  "smmlu/easy/Sinhala language and literature/86",
+  "smmlu/easy/Sinhala language and literature/44",
+  "smmlu/easy/dancing/59",
+  "smmlu/easy/dancing/46",
+  "smmlu/easy/dancing/32",
+  "smmlu/easy/dancing/95",
+  "smmlu/easy/dancing/2",
+  "smmlu/easy/dancing/56",
+  "smmlu/easy/dancing/49",
+  "smmlu/easy/dancing/63",
+  "smmlu/easy/dancing/98",
+  "smmlu/easy/dancing/45",
+  "smmlu/easy/dancing/15",
+  "smmlu/easy/dancing/21",
+  "smmlu/easy/dancing/29",
+  "smmlu/easy/dancing/74",
+  "smmlu/easy/dancing/8",
+  "smmlu/easy/dancing/53",
+  "smmlu/easy/dancing/101",
+  "smmlu/easy/dancing/73",
+  "smmlu/easy/dancing/31",
+  "smmlu/easy/dancing/82",
+  "smmlu/easy/dancing/71",
+  "smmlu/easy/drama and Theatre /50",
+  "smmlu/easy/drama and Theatre /6",
+  "smmlu/easy/drama and Theatre /107",
+  "smmlu/easy/drama and Theatre /26",
+  "smmlu/easy/drama and Theatre /22",
+  "smmlu/easy/drama and Theatre /30",
+  "smmlu/easy/drama and Theatre /67",
+  "smmlu/easy/drama and Theatre /61",
+  "smmlu/easy/drama and Theatre /127",
+  "smmlu/easy/drama and Theatre /43",
+  "smmlu/easy/drama and Theatre /120",
+  "smmlu/easy/drama and Theatre /117",
+  "smmlu/easy/drama and Theatre /42",
+  "smmlu/easy/drama and Theatre /92",
+  "smmlu/easy/drama and Theatre /101",
+  "smmlu/easy/drama and Theatre /51",
+  "smmlu/easy/drama and Theatre /28",
+  "smmlu/easy/drama and Theatre /119",
+  "smmlu/easy/drama and Theatre /58",
+  "smmlu/easy/drama and Theatre /68",
+  "smmlu/easy/drama and Theatre /80",
+  "smmlu/easy/drama and Theatre /31",
+  "smmlu/easy/drama and Theatre /16",
+  "smmlu/easy/drama and Theatre /3",
+  "smmlu/easy/drama and Theatre /7",
+  "smmlu/easy/drama and Theatre /54",
+  "smmlu/easy/science/12",
+  "smmlu/easy/science/68",
+  "smmlu/easy/science/89",
+  "smmlu/easy/science/108",
+  "smmlu/easy/science/29",
+  "smmlu/easy/science/142",
+  "smmlu/easy/science/77",
+  "smmlu/easy/science/148",
+  "smmlu/easy/science/10",
+  "smmlu/easy/science/160",
+  "smmlu/easy/science/13",
+  "smmlu/easy/science/153",
+  "smmlu/easy/science/65",
+  "smmlu/easy/science/33",
+  "smmlu/easy/science/40",
+  "smmlu/easy/science/81",
+  "smmlu/easy/science/117",
+  "smmlu/easy/science/43",
+  "smmlu/easy/science/132",
+  "smmlu/easy/science/122",
+  "smmlu/easy/science/126",
+  "smmlu/easy/science/128",
+  "smmlu/easy/science/30",
+  "smmlu/easy/science/92",
+  "smmlu/easy/science/151",
+  "smmlu/easy/science/94",
+  "smmlu/easy/science/42",
+  "smmlu/easy/science/110",
+  "smmlu/easy/science/41",
+  "smmlu/easy/science/66",
+  "smmlu/easy/science/78",
+  "smmlu/easy/science/141",
+  "smmlu/easy/science/90"
+ ],
+ [
+  "smmlu/easy/Arts/59",
+  "smmlu/easy/Arts/77",
+  "smmlu/easy/Arts/57",
+  "smmlu/easy/Arts/7",
+  "smmlu/easy/Arts/48",
+  "smmlu/easy/Arts/69",
+  "smmlu/easy/Arts/42",
+  "smmlu/easy/Arts/90",
+  "smmlu/easy/Arts/96",
+  "smmlu/easy/Arts/39",
+  "smmlu/easy/Arts/24",
+  "smmlu/easy/Arts/46",
+  "smmlu/easy/Arts/13",
+  "smmlu/easy/Arts/16",
+  "smmlu/easy/Arts/97",
+  "smmlu/easy/Arts/8",
+  "smmlu/easy/Arts/87",
+  "smmlu/easy/Arts/2",
+  "smmlu/easy/Arts/76",
+  "smmlu/easy/Arts/99",
+  "smmlu/easy/Buddhism/28",
+  "smmlu/easy/Buddhism/58",
+  "smmlu/easy/Buddhism/62",
+  "smmlu/easy/Buddhism/69",
+  "smmlu/easy/Buddhism/16",
+  "smmlu/easy/Buddhism/93",
+  "smmlu/easy/Buddhism/111",
+  "smmlu/easy/Buddhism/132",
+  "smmlu/easy/Buddhism/139",
+  "smmlu/easy/Buddhism/89",
+  "smmlu/easy/Buddhism/55",
+  "smmlu/easy/Buddhism/131",
+  "smmlu/easy/Buddhism/130",
+  "smmlu/easy/Buddhism/149",
+  "smmlu/easy/Buddhism/37",
+  "smmlu/easy/Buddhism/161",
+  "smmlu/easy/Buddhism/113",
+  "smmlu/easy/Buddhism/156",
+  "smmlu/easy/Buddhism/133",
+  "smmlu/easy/Buddhism/64",
+  "smmlu/easy/Buddhism/88",
+  "smmlu/easy/Buddhism/144",
+  "smmlu/easy/Buddhism/7",
+  "smmlu/easy/Buddhism/151",
+  "smmlu/easy/Buddhism/99",
+  "smmlu/easy/Buddhism/107",
+  "smmlu/easy/Buddhism/117",
+  "smmlu/easy/Buddhism/160",
+  "smmlu/easy/Buddhism/158",
+  "smmlu/easy/Buddhism/126",
+  "smmlu/easy/Buddhism/162",
+  "smmlu/easy/Buddhism/138",
+  "smmlu/easy/Catholicism/53",
+  "smmlu/easy/Catholicism/118",
+  "smmlu/easy/Catholicism/44",
+  "smmlu/easy/Catholicism/95",
+  "smmlu/easy/Catholicism/115",
+  "smmlu/easy/Catholicism/104",
+  "smmlu/easy/Catholicism/78",
+  "smmlu/easy/Catholicism/12",
+  "smmlu/easy/Catholicism/7",
+  "smmlu/easy/Catholicism/121",
+  "smmlu/easy/Catholicism/71",
+  "smmlu/easy/Catholicism/82",
+  "smmlu/easy/Catholicism/30",
+  "smmlu/easy/Catholicism/125",
+  "smmlu/easy/Catholicism/2",
+  "smmlu/easy/Catholicism/15",
+  "smmlu/easy/Catholicism/4",
+  "smmlu/easy/Catholicism/103",
+  "smmlu/easy/Catholicism/114",
+  "smmlu/easy/Catholicism/43",
+  "smmlu/easy/Catholicism/108",
+  "smmlu/easy/Catholicism/86",
+  "smmlu/easy/Catholicism/41",
+  "smmlu/easy/Catholicism/39",
+  "smmlu/easy/Catholicism/45",
+  "smmlu/easy/Catholicism/49",
+  "smmlu/easy/Christianity/53",
+  "smmlu/easy/Christianity/109",
+  "smmlu/easy/Christianity/74",
+  "smmlu/easy/Christianity/16",
+  "smmlu/easy/Christianity/116",
+  "smmlu/easy/Christianity/111",
+  "smmlu/easy/Christianity/17",
+  "smmlu/easy/Christianity/11",
+  "smmlu/easy/Christianity/115",
+  "smmlu/easy/Christianity/79",
+  "smmlu/easy/Christianity/86",
+  "smmlu/easy/Christianity/106",
+  "smmlu/easy/Christianity/129",
+  "smmlu/easy/Christianity/65",
+  "smmlu/easy/Christianity/42",
+  "smmlu/easy/Christianity/32",
+  "smmlu/easy/Christianity/1",
+  "smmlu/easy/Christianity/72",
+  "smmlu/easy/Christianity/67",
+  "smmlu/easy/Christianity/123",
+  "smmlu/easy/Christianity/13",
+  "smmlu/easy/Christianity/125",
+  "smmlu/easy/Christianity/97",
+  "smmlu/easy/Christianity/35",
+  "smmlu/easy/Christianity/57",
+  "smmlu/easy/Christianity/94",
+  "smmlu/easy/Civics/115",
+  "smmlu/easy/Civics/119",
+  "smmlu/easy/Civics/62",
+  "smmlu/easy/Civics/105",
+  "smmlu/easy/Civics/12",
+  "smmlu/easy/Civics/77",
+  "smmlu/easy/Civics/5",
+  "smmlu/easy/Civics/123",
+  "smmlu/easy/Civics/37",
+  "smmlu/easy/Civics/26",
+  "smmlu/easy/Civics/17",
+  "smmlu/easy/Civics/32",
+  "smmlu/easy/Civics/79",
+  "smmlu/easy/Civics/71",
+  "smmlu/easy/Civics/83",
+  "smmlu/easy/Civics/66",
+  "smmlu/easy/Civics/61",
+  "smmlu/easy/Civics/45",
+  "smmlu/easy/Civics/96",
+  "smmlu/easy/Civics/36",
+  "smmlu/easy/Civics/120",
+  "smmlu/easy/Civics/7",
+  "smmlu/easy/Civics/21",
+  "smmlu/easy/Civics/67",
+  "smmlu/easy/Civics/95",
+  "smmlu/easy/Eastern music/129",
+  "smmlu/easy/Eastern music/122",
+  "smmlu/easy/Eastern music/44",
+  "smmlu/easy/Eastern music/99",
+  "smmlu/easy/Eastern music/127",
+  "smmlu/easy/Eastern music/34",
+  "smmlu/easy/Eastern music/124",
+  "smmlu/easy/Eastern music/53",
+  "smmlu/easy/Eastern music/24",
+  "smmlu/easy/Eastern music/56",
+  "smmlu/easy/Eastern music/125",
+  "smmlu/easy/Eastern music/75",
+  "smmlu/easy/Eastern music/21",
+  "smmlu/easy/Eastern music/55",
+  "smmlu/easy/Eastern music/43",
+  "smmlu/easy/Eastern music/114",
+  "smmlu/easy/Eastern music/126",
+  "smmlu/easy/Eastern music/95",
+  "smmlu/easy/Eastern music/20",
+  "smmlu/easy/Eastern music/78",
+  "smmlu/easy/Eastern music/71",
+  "smmlu/easy/Eastern music/76",
+  "smmlu/easy/Eastern music/96",
+  "smmlu/easy/Eastern music/58",
+  "smmlu/easy/Eastern music/77",
+  "smmlu/easy/Eastern music/69",
+  "smmlu/easy/Eastern music/54",
+  "smmlu/easy/Eastern music/50",
+  "smmlu/easy/Eastern music/2",
+  "smmlu/easy/Geography/7",
+  "smmlu/easy/Geography/86",
+  "smmlu/easy/Geography/90",
+  "smmlu/easy/Geography/73",
+  "smmlu/easy/Geography/92",
+  "smmlu/easy/Geography/22",
+  "smmlu/easy/Geography/19",
+  "smmlu/easy/Geography/44",
+  "smmlu/easy/Geography/46",
+  "smmlu/easy/Geography/16",
+  "smmlu/easy/Geography/40",
+  "smmlu/easy/Geography/66",
+  "smmlu/easy/Geography/109",
+  "smmlu/easy/Geography/21",
+  "smmlu/easy/Geography/56",
+  "smmlu/easy/Geography/29",
+  "smmlu/easy/Geography/103",
+  "smmlu/easy/Geography/71",
+  "smmlu/easy/Geography/102",
+  "smmlu/easy/Geography/63",
+  "smmlu/easy/Geography/49",
+  "smmlu/easy/Geography/69",
+  "smmlu/easy/Geography/52",
+  "smmlu/easy/Health and Physical Science/87",
+  "smmlu/easy/Health and Physical Science/107",
+  "smmlu/easy/Health and Physical Science/100",
+  "smmlu/easy/Health and Physical Science/71",
+  "smmlu/easy/Health and Physical Science/103",
+  "smmlu/easy/Health and Physical Science/126",
+  "smmlu/easy/Health and Physical Science/111",
+  "smmlu/easy/Health and Physical Science/32",
+  "smmlu/easy/Health and Physical Science/61",
+  "smmlu/easy/Health and Physical Science/68",
+  "smmlu/easy/Health and Physical Science/125",
+  "smmlu/easy/Health and Physical Science/9",
+  "smmlu/easy/Health and Physical Science/53",
+  "smmlu/easy/Health and Physical Science/28",
+  "smmlu/easy/Health and Physical Science/2",
+  "smmlu/easy/Health and Physical Science/70",
+  "smmlu/easy/Health and Physical Science/57",
+  "smmlu/easy/Health and Physical Science/85",
+  "smmlu/easy/Health and Physical Science/63",
+  "smmlu/easy/Health and Physical Science/3",
+  "smmlu/easy/Health and Physical Science/49",
+  "smmlu/easy/Health and Physical Science/15",
+  "smmlu/easy/Health and Physical Science/75",
+  "smmlu/easy/Health and Physical Science/22",
+  "smmlu/easy/Health and Physical Science/17",
+  "smmlu/easy/Health and Physical Science/98",
+  "smmlu/easy/History/28",
+  "smmlu/easy/History/143",
+  "smmlu/easy/History/37",
+  "smmlu/easy/History/40",
+  "smmlu/easy/History/116",
+  "smmlu/easy/History/121",
+  "smmlu/easy/History/45",
+  "smmlu/easy/History/89",
+  "smmlu/easy/History/96",
+  "smmlu/easy/History/31",
+  "smmlu/easy/History/130",
+  "smmlu/easy/History/111",
+  "smmlu/easy/History/1",
+  "smmlu/easy/History/144",
+  "smmlu/easy/History/147",
+  "smmlu/easy/History/151",
+  "smmlu/easy/History/138",
+  "smmlu/easy/History/12",
+  "smmlu/easy/History/2",
+  "smmlu/easy/History/145",
+  "smmlu/easy/History/72",
+  "smmlu/easy/History/150",
+  "smmlu/easy/History/41",
+  "smmlu/easy/History/99",
+  "smmlu/easy/History/108",
+  "smmlu/easy/History/48",
+  "smmlu/easy/History/93",
+  "smmlu/easy/History/53",
+  "smmlu/easy/History/63",
+  "smmlu/easy/History/3",
+  "smmlu/easy/Islam/56",
+  "smmlu/easy/Islam/1",
+  "smmlu/easy/Islam/81",
+  "smmlu/easy/Islam/19",
+  "smmlu/easy/Islam/90",
+  "smmlu/easy/Islam/26",
+  "smmlu/easy/Islam/62",
+  "smmlu/easy/Islam/45",
+  "smmlu/easy/Islam/2",
+  "smmlu/easy/Islam/22",
+  "smmlu/easy/Islam/35",
+  "smmlu/easy/Islam/55",
+  "smmlu/easy/Islam/83",
+  "smmlu/easy/Islam/77",
+  "smmlu/easy/Islam/89",
+  "smmlu/easy/Islam/82",
+  "smmlu/easy/Islam/46",
+  "smmlu/easy/Islam/96",
+  "smmlu/easy/Islam/102",
+  "smmlu/easy/Islam/20",
+  "smmlu/easy/Islam/104",
+  "smmlu/easy/Islam/32",
+  "smmlu/easy/Sinhala language and literature/38",
+  "smmlu/easy/Sinhala language and literature/89",
+  "smmlu/easy/Sinhala language and literature/129",
+  "smmlu/easy/Sinhala language and literature/41",
+  "smmlu/easy/Sinhala language and literature/66",
+  "smmlu/easy/Sinhala language and literature/35",
+  "smmlu/easy/Sinhala language and literature/31",
+  "smmlu/easy/Sinhala language and literature/119",
+  "smmlu/easy/Sinhala language and literature/39",
+  "smmlu/easy/Sinhala language and literature/130",
+  "smmlu/easy/Sinhala language and literature/77",
+  "smmlu/easy/Sinhala language and literature/116",
+  "smmlu/easy/Sinhala language and literature/123",
+  "smmlu/easy/Sinhala language and literature/64",
+  "smmlu/easy/Sinhala language and literature/6",
+  "smmlu/easy/Sinhala language and literature/106",
+  "smmlu/easy/Sinhala language and literature/111",
+  "smmlu/easy/Sinhala language and literature/127",
+  "smmlu/easy/Sinhala language and literature/13",
+  "smmlu/easy/Sinhala language and literature/19",
+  "smmlu/easy/Sinhala language and literature/148",
+  "smmlu/easy/Sinhala language and literature/4",
+  "smmlu/easy/Sinhala language and literature/112",
+  "smmlu/easy/Sinhala language and literature/139",
+  "smmlu/easy/Sinhala language and literature/18",
+  "smmlu/easy/Sinhala language and literature/136",
+  "smmlu/easy/Sinhala language and literature/125",
+  "smmlu/easy/Sinhala language and literature/68",
+  "smmlu/easy/Sinhala language and literature/24",
+  "smmlu/easy/Sinhala language and literature/145",
+  "smmlu/easy/Sinhala language and literature/51",
+  "smmlu/easy/dancing/51",
+  "smmlu/easy/dancing/86",
+  "smmlu/easy/dancing/6",
+  "smmlu/easy/dancing/18",
+  "smmlu/easy/dancing/44",
+  "smmlu/easy/dancing/96",
+  "smmlu/easy/dancing/76",
+  "smmlu/easy/dancing/16",
+  "smmlu/easy/dancing/66",
+  "smmlu/easy/dancing/62",
+  "smmlu/easy/dancing/7",
+  "smmlu/easy/dancing/89",
+  "smmlu/easy/dancing/25",
+  "smmlu/easy/dancing/61",
+  "smmlu/easy/dancing/52",
+  "smmlu/easy/dancing/55",
+  "smmlu/easy/dancing/69",
+  "smmlu/easy/dancing/91",
+  "smmlu/easy/dancing/84",
+  "smmlu/easy/dancing/12",
+  "smmlu/easy/dancing/85",
+  "smmlu/easy/drama and Theatre /18",
+  "smmlu/easy/drama and Theatre /73",
+  "smmlu/easy/drama and Theatre /106",
+  "smmlu/easy/drama and Theatre /94",
+  "smmlu/easy/drama and Theatre /47",
+  "smmlu/easy/drama and Theatre /40",
+  "smmlu/easy/drama and Theatre /57",
+  "smmlu/easy/drama and Theatre /70",
+  "smmlu/easy/drama and Theatre /115",
+  "smmlu/easy/drama and Theatre /118",
+  "smmlu/easy/drama and Theatre /114",
+  "smmlu/easy/drama and Theatre /15",
+  "smmlu/easy/drama and Theatre /52",
+  "smmlu/easy/drama and Theatre /20",
+  "smmlu/easy/drama and Theatre /76",
+  "smmlu/easy/drama and Theatre /64",
+  "smmlu/easy/drama and Theatre /105",
+  "smmlu/easy/drama and Theatre /41",
+  "smmlu/easy/drama and Theatre /98",
+  "smmlu/easy/drama and Theatre /35",
+  "smmlu/easy/drama and Theatre /37",
+  "smmlu/easy/drama and Theatre /86",
+  "smmlu/easy/drama and Theatre /60",
+  "smmlu/easy/drama and Theatre /124",
+  "smmlu/easy/drama and Theatre /128",
+  "smmlu/easy/drama and Theatre /91",
+  "smmlu/easy/science/1",
+  "smmlu/easy/science/123",
+  "smmlu/easy/science/63",
+  "smmlu/easy/science/69",
+  "smmlu/easy/science/80",
+  "smmlu/easy/science/150",
+  "smmlu/easy/science/79",
+  "smmlu/easy/science/4",
+  "smmlu/easy/science/52",
+  "smmlu/easy/science/85",
+  "smmlu/easy/science/154",
+  "smmlu/easy/science/127",
+  "smmlu/easy/science/36",
+  "smmlu/easy/science/55",
+  "smmlu/easy/science/20",
+  "smmlu/easy/science/8",
+  "smmlu/easy/science/99",
+  "smmlu/easy/science/93",
+  "smmlu/easy/science/16",
+  "smmlu/easy/science/161",
+  "smmlu/easy/science/18",
+  "smmlu/easy/science/97",
+  "smmlu/easy/science/107",
+  "smmlu/easy/science/119",
+  "smmlu/easy/science/103",
+  "smmlu/easy/science/144",
+  "smmlu/easy/science/125",
+  "smmlu/easy/science/24",
+  "smmlu/easy/science/39",
+  "smmlu/easy/science/138",
+  "smmlu/easy/science/100",
+  "smmlu/easy/science/49",
+  "smmlu/easy/science/159"
+ ]
+]

training/configs/submission.yaml

@@ -0,0 +1,13 @@
+# Everything the offline submission pipeline needs (inference/predict.py).
+# Chosen on SinhalaMMLU dev + Global-MMLU si validation only (reports/experiments_prompting.md), never on test.
+model: models/gemma-4-E4B-it-text   # text-only Gemma 4 E4B (scripts/make_text_only.py), 7.463B parameters
+lang: en
+chat: true
+method: logit
+n_perms: 1
+perm_mode: cyclic
+seed: 1234
+dtype: bfloat16
+gpu_mem_gb: 100
+token_budget: 32768
+max_batch: 64

training/configs/submission_A_think_hard.yaml

@@ -0,0 +1,15 @@
+# Everything the offline submission pipeline needs (inference/predict.py).
+# A + thinking on hard (19 Oct): model A; questions labelled hard use thinking mode (academic proxy +8.4, paired CI [+4.9, +11.8]); others fast 4-order vote. Never chosen on test.
+model: models/sft_gmmlu_lora/step-357   # model A: LoRA r=16 on Global-MMLU si, 1 epoch, merged (7.463B)
+lang: en
+chat: true
+method: logit
+n_perms: 4            # vote over 4 cyclic option orders (+2.7 dev vs zero-shot, paired bootstrap)
+perm_mode: cyclic
+seed: 1234
+dtype: bfloat16
+gpu_mem_gb: 100
+token_budget: 32768
+max_batch: 64
+thinking_levels: [hard]
+think_max_new_tokens: 2048

training/configs/submission_base_think_gated_sc.yaml

@@ -0,0 +1,23 @@
+# Final candidate B (21 Oct): base Gemma 4 E4B, text-only (7.463B, no adapter), thinking for every question.
+# Agreement-gated self-consistency on all levels: where the one-step answer disagrees with greedy thinking, add 3 sampled
+# reasoning paths and take the majority vote (ties -> greedy). Evidence (dev/validation only): full dev, base thinking 63.0 vs
+# model A vote 57.2 (+5.8 [+3.6, +8.0]); pooled proxies, gated self-consistency +2.70 [+1.18, +4.23] on 43% of questions.
+model: models/gemma-4-E4B-it-text
+lang: en
+chat: true
+method: logit          # fast path, used only if a thinking answer cannot be read
+n_perms: 4
+perm_mode: cyclic
+seed: 1234
+dtype: bfloat16
+gpu_mem_gb: 100
+token_budget: 32768
+max_batch: 64
+thinking_levels: [easy, medium, hard]
+think_max_new_tokens: 3072
+sc_levels: [easy, medium, hard]
+sc_gate: disagree
+sc_samples: 3
+sc_temperature: 0.8
+sc_top_p: 0.95
+sc_top_k: 64

training/configs/submission_base_think_sc_hard.yaml

@@ -0,0 +1,22 @@
+# Final candidate (21 Oct): base Gemma 4 E4B, text-only (7.463B, no adapter), thinking for every question.
+# Hard: self-consistency (greedy + 3 sampled reasoning paths, majority vote, ties -> greedy).
+# Evidence (dev / validation only, never test): full dev, base thinking 63.0 vs model A 4-order vote 57.2
+# (+5.8 [+3.6, +8.0]); academic proxy, self-consistency 82.2 vs greedy 79.5 (+2.7 [+0.9, +4.7]).
+model: models/gemma-4-E4B-it-text
+lang: en
+chat: true
+method: logit          # fast path, used only if a thinking answer cannot be read
+n_perms: 4
+perm_mode: cyclic
+seed: 1234
+dtype: bfloat16
+gpu_mem_gb: 100
+token_budget: 32768
+max_batch: 64
+thinking_levels: [easy, medium, hard]
+think_max_new_tokens: 3072
+sc_levels: [hard]
+sc_samples: 3
+sc_temperature: 0.8
+sc_top_p: 0.95
+sc_top_k: 64

training/configs/submission_lora_gmmlu_vote.yaml

@@ -0,0 +1,13 @@
+# Everything the offline submission pipeline needs (inference/predict.py).
+# Chosen on SinhalaMMLU dev only (reports/experiments_abc.md), never on test.
+model: models/sft_gmmlu_lora/step-357   # experiment A: LoRA r=16 on Global-MMLU si, 1 epoch, merged (7.463B)
+lang: en
+chat: true
+method: logit
+n_perms: 4            # vote over 4 cyclic option orders (+2.7 dev vs zero-shot, paired bootstrap)
+perm_mode: cyclic
+seed: 1234
+dtype: bfloat16
+gpu_mem_gb: 100
+token_budget: 32768
+max_batch: 64

training/configs/submission_medium_option_order.yaml

@@ -0,0 +1,20 @@
+# Medium-only option-order self-consistency (27 Oct): base Gemma 4 E4B text-only (7.463B, no adapter). For every medium question,
+# thinking under 3 extra cyclic option orders; vote over the previous answer (candidate B's) and the 3 re-orderings, ties -> previous.
+# Run with --levels medium and merge into candidate B (scripts/merge_predictions.py). Chosen only if eval/perm_sc.py says GO on the
+# held-out stand-in (never on test).
+model: models/gemma-4-E4B-it-text
+lang: en
+chat: true
+method: logit
+n_perms: 1
+perm_mode: cyclic
+seed: 1234
+dtype: bfloat16
+gpu_mem_gb: 100
+token_budget: 32768
+max_batch: 64
+thinking_levels: []
+think_max_new_tokens: 3072
+perm_think_levels: [medium]
+perm_think_orders: 3
+prior_predictions: results/submissions/final_candidate_base_think_gated_sc.zip

training/configs/submission_pool_base_medium.yaml

@@ -0,0 +1,21 @@
+# Cross-model pooling, part 1 (27 Oct), MEDIUM ONLY: base Gemma 4 E4B, greedy thinking + 3 sampled paths on EVERY question
+# (no agreement gate); every answer is saved to <output>.ballots.jsonl. scripts/pool_ballots.py votes over these ballots and the
+# RFT2 model's (configs/submission_pool_rft2_medium.yaml) and merges into candidate B. Used only if eval/pool_decide.py says GO.
+model: models/gemma-4-E4B-it-text
+lang: en
+chat: true
+method: logit
+n_perms: 4
+perm_mode: cyclic
+seed: 1234
+dtype: bfloat16
+gpu_mem_gb: 100
+token_budget: 32768
+max_batch: 64
+thinking_levels: [medium]
+think_max_new_tokens: 3072
+sc_levels: [medium]
+sc_samples: 3
+sc_temperature: 0.8
+sc_top_p: 0.95
+sc_top_k: 64

training/configs/submission_pool_rft2_medium.yaml

@@ -0,0 +1,21 @@
+# Cross-model pooling, part 2 (27 Oct), MEDIUM ONLY: the RFT2 model (base + reasoning-trace LoRA, 7.50B parameters, adapter ON),
+# greedy thinking + 3 sampled paths on every question (no gate); ballots saved for scripts/pool_ballots.py.
+model: models/gemma-4-E4B-it-text
+adapter: models/rft_v2/step-141/adapter
+lang: en
+chat: true
+method: logit
+n_perms: 4
+perm_mode: cyclic
+seed: 1234
+dtype: bfloat16
+gpu_mem_gb: 100
+token_budget: 32768
+max_batch: 64
+thinking_levels: [medium]
+think_max_new_tokens: 3072
+sc_levels: [medium]
+sc_samples: 3
+sc_temperature: 0.8
+sc_top_p: 0.95
+sc_top_k: 64

training/configs/submission_rft2_medium.yaml

@@ -0,0 +1,26 @@
+# Candidate D (27 Oct), MEDIUM ONLY: base Gemma 4 E4B + LoRA adapter trained on the base model's own correct reasoning traces
+# (dev minus held-out 400, Global-MMLU, synthetic Wikipedia MCQs; ~7.50B parameters). Thinking with the adapter ON; the fast answer
+# for the agreement gate comes from the plain base model. Agreement-gated self-consistency. Run with --levels medium and merge into
+# candidate B (scripts/merge_predictions.py). Chosen only if it passes the pre-registered held-out validation (eval/decide_rft.py).
+model: models/gemma-4-E4B-it-text
+adapter: models/rft_v2/step-ADAPTER/adapter
+thinking_without_adapter: false
+fast_without_adapter: true
+lang: en
+chat: true
+method: logit
+n_perms: 4
+perm_mode: cyclic
+seed: 1234
+dtype: bfloat16
+gpu_mem_gb: 100
+token_budget: 32768
+max_batch: 64
+thinking_levels: [medium]
+think_max_new_tokens: 3072
+sc_levels: [medium]
+sc_gate: disagree
+sc_samples: 3
+sc_temperature: 0.8
+sc_top_p: 0.95
+sc_top_k: 64

training/configs/submission_rft_think_gated_sc.yaml

@@ -0,0 +1,27 @@
+# Final candidate C (26-27 Oct): base Gemma 4 E4B + LoRA adapter trained on the base model's own correct reasoning traces
+# (rejection sampling, SinhalaMMLU dev minus a held-out 400; ~7.50B parameters). Thinking for every question with the adapter ON;
+# the fast one-step answer for the agreement gate comes from the plain base model (adapter off).
+# Agreement-gated self-consistency on all levels, as candidate B. Chosen only if it passes the pre-registered held-out
+# validation rule in eval/decide_rft.py (never on test).
+model: models/gemma-4-E4B-it-text
+adapter: models/rft_v1/step-ADAPTER/adapter
+thinking_without_adapter: false
+fast_without_adapter: true
+lang: en
+chat: true
+method: logit
+n_perms: 4
+perm_mode: cyclic
+seed: 1234
+dtype: bfloat16
+gpu_mem_gb: 100
+token_budget: 32768
+max_batch: 64
+thinking_levels: [easy, medium, hard]
+think_max_new_tokens: 3072
+sc_levels: [easy, medium, hard]
+sc_gate: disagree
+sc_samples: 3
+sc_temperature: 0.8
+sc_top_p: 0.95
+sc_top_k: 64

training/configs/submission_sub5_route.yaml

@@ -0,0 +1,17 @@
+# Everything the offline submission pipeline needs (inference/predict.py).
+# sub5 (19 Oct): base Gemma 4 E4B + adapter A (LoRA r=16, extracted, ~0.04B); easy/medium fast 4-order vote WITH adapter; hard: thinking WITHOUT adapter (academic proxy: base thinking 79.5 vs A fast 60.1). Never chosen on test.
+model: models/gemma-4-E4B-it-text   # base, 7.463B
+lang: en
+chat: true
+method: logit
+n_perms: 4            # vote over 4 cyclic option orders (+2.7 dev vs zero-shot, paired bootstrap)
+perm_mode: cyclic
+seed: 1234
+dtype: bfloat16
+gpu_mem_gb: 100
+token_budget: 32768
+max_batch: 64
+thinking_levels: [hard]
+think_max_new_tokens: 3072
+adapter: models/adapter_A          # model A as a separate LoRA adapter
+thinking_without_adapter: true

training/configs/submission_sub6_think_medhard.yaml

@@ -0,0 +1,17 @@
+# Everything the offline submission pipeline needs (inference/predict.py).
+# sub6 (20 Oct): as sub5, plus base-model thinking (adapter off) for medium. Evidence: non-academic Global-MMLU proxy +12.3 [+6.0, +18.3] and easy-style dev not worse (+0.8) vs model A fast. Never chosen on test.
+model: models/gemma-4-E4B-it-text   # base, 7.463B
+lang: en
+chat: true
+method: logit
+n_perms: 4            # vote over 4 cyclic option orders (+2.7 dev vs zero-shot, paired bootstrap)
+perm_mode: cyclic
+seed: 1234
+dtype: bfloat16
+gpu_mem_gb: 100
+token_budget: 32768
+max_batch: 64
+thinking_levels: [medium, hard]
+think_max_new_tokens: 3072
+adapter: models/adapter_A          # model A as a separate LoRA adapter
+thinking_without_adapter: true

training/configs/submission_v2.yaml

@@ -0,0 +1,13 @@
+# Everything the offline submission pipeline needs (inference/predict.py).
+# v2: recipe validated on dev via A (Global-MMLU LoRA, +2.7) and B (dev 5-fold LoRA, +2.8); never chosen on test.
+model: models/sft_v2_gmmlu_dev/step-415   # v2: LoRA r=16 on Global-MMLU si + all SinhalaMMLU dev, 1 epoch, merged (7.463B)
+lang: en
+chat: true
+method: logit
+n_perms: 4            # vote over 4 cyclic option orders (+2.7 dev vs zero-shot, paired bootstrap)
+perm_mode: cyclic
+seed: 1234
+dtype: bfloat16
+gpu_mem_gb: 100
+token_budget: 32768
+max_batch: 64

training/configs/train/cpt_pilot.yaml

@@ -0,0 +1,24 @@
+# Experiment C: 30M-token pilot of continued pretraining (lower LR to limit forgetting), checkpoints at 15M/30M.
+mode: cpt
+model: models/gemma-4-E4B-it-text
+out_dir: models/cpt_pilot
+data_dir: data/corpus/cpt_v1
+seq_len: 2048
+micro_batch: 2
+grad_accum: 32          # 131k tokens per optimizer step
+max_tokens: 30000000
+epochs: 1
+lr: 5.0e-6
+min_lr_ratio: 0.1
+warmup_steps: 20
+weight_decay: 0.0
+max_grad_norm: 1.0
+freeze_embeddings: true
+ckpt_every: 2
+compile: true
+loss: liger
+seed: 1234
+gpu_mem_gb: 100
+save_every_tokens: 15000000
+keep_checkpoints: 3
+log_every: 10

training/configs/train/cpt_v1.yaml

@@ -0,0 +1,24 @@
+# Continued pretraining on data/corpus/cpt_v1 (scripts/select_cpt_subset.py). Fastest measured safe setting.
+mode: cpt
+model: models/gemma-4-E4B-it-text
+out_dir: models/cpt_v1
+data_dir: data/corpus/cpt_v1
+seq_len: 2048
+micro_batch: 2
+grad_accum: 32          # 131k tokens per optimizer step
+max_tokens: 300000000
+epochs: 1
+lr: 1.0e-5
+min_lr_ratio: 0.1
+warmup_steps: 50
+weight_decay: 0.0
+max_grad_norm: 1.0
+freeze_embeddings: true
+ckpt_every: 2
+compile: true
+loss: liger
+seed: 1234
+gpu_mem_gb: 100
+save_every_tokens: 50000000
+keep_checkpoints: 3
+log_every: 10

training/configs/train/rft_trace.yaml

@@ -0,0 +1,23 @@
+# Reasoning-preserving fine-tuning (21-26 Oct): LoRA r=16 on the base model's OWN correct thinking traces for SinhalaMMLU
+# dev questions (scripts/gen_traces.py), loss on the trace only. The 400-question dev sample is held out for validation.
+mode: trace
+model: models/gemma-4-E4B-it-text
+out_dir: models/rft_v1
+data: [data/synth/rft_traces.jsonl]
+lang: en
+max_len: 4096
+epochs: 1
+batch_size: 16
+micro_tokens: 12000
+lr: 1.0e-4
+min_lr_ratio: 0.1
+warmup_steps: 5
+weight_decay: 0.0
+max_grad_norm: 1.0
+lora: {r: 16, alpha: 32, dropout: 0.0, targets: all-linear}
+ckpt_every: 1
+compile: false
+seed: 1234
+gpu_mem_gb: 100
+keep_checkpoints: 1
+log_every: 5

training/configs/train/rft_trace2.yaml

@@ -0,0 +1,22 @@
+# Round 2 (27 Oct): LoRA r=16 on the base model OWN correct thinking traces: SinhalaMMLU dev (minus held-out 400) + Global-MMLU (humanities/social/other/business) + synthetic Wikipedia MCQs.
+mode: trace
+model: models/gemma-4-E4B-it-text
+out_dir: models/rft_v2
+data: [data/synth/rft_traces.jsonl, data/synth/rft2_traces.jsonl]
+lang: en
+max_len: 4096
+epochs: 1
+batch_size: 16
+micro_tokens: 12000
+lr: 1.0e-4
+min_lr_ratio: 0.1
+warmup_steps: 5
+weight_decay: 0.0
+max_grad_norm: 1.0
+lora: {r: 16, alpha: 32, dropout: 0.0, targets: all-linear}
+ckpt_every: 1
+compile: false
+seed: 1234
+gpu_mem_gb: 100
+keep_checkpoints: 1
+log_every: 5

training/configs/train/sft_gmmlu.yaml

@@ -0,0 +1,24 @@
+# Step 4: fine-tune text-only Gemma 4 E4B on Global-MMLU si MCQs (scripts/build_sft_gmmlu.py), answer-digit loss.
+mode: sft
+model: models/gemma-4-E4B-it-text
+out_dir: models/sft_gmmlu_v1
+data: [data/sft/gmmlu_si_train.jsonl]
+lang: en
+chat: true
+permute_options: true
+epochs: 2
+batch_size: 32          # questions per optimizer step
+micro_tokens: 16384     # padded tokens per forward pass
+lr: 1.0e-5
+min_lr_ratio: 0.1
+warmup_steps: 20
+weight_decay: 0.0
+max_grad_norm: 1.0
+freeze_embeddings: true # Gemma per-layer embedding tables (3.5B) frozen; 3.97B trained
+ckpt_every: 1
+compile: false          # variable SFT lengths would recompile per shape
+loss: liger
+seed: 1234
+gpu_mem_gb: 100
+keep_checkpoints: 2
+log_every: 10

training/configs/train/sft_lora.yaml

@@ -0,0 +1,25 @@
+# Gentle fine-tuning: LoRA r=16 on all linear layers (base frozen, merged at save -> 7.463B params).
+# Used for experiment A (Global-MMLU, 1 epoch) and B (SinhalaMMLU dev 5-fold CV, --data/--out per fold).
+mode: sft
+model: models/gemma-4-E4B-it-text
+out_dir: models/sft_lora
+data: [data/sft/gmmlu_si_train.jsonl]
+lang: en
+chat: true
+permute_options: true
+epochs: 1
+batch_size: 32
+micro_tokens: 16384
+lr: 1.0e-4
+min_lr_ratio: 0.1
+warmup_steps: 10
+weight_decay: 0.0
+max_grad_norm: 1.0
+lora: {r: 16, alpha: 32, dropout: 0.0, targets: all-linear}
+ckpt_every: 1
+compile: false
+loss: liger
+seed: 1234
+gpu_mem_gb: 100
+keep_checkpoints: 1
+log_every: 10