diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..eb7b3fbc0595f00c4570e1e988c4fbc76a728e23 --- /dev/null +++ b/README.md @@ -0,0 +1,71 @@ +# Team Nexus – Sinhala MMLU Shared Task System + +## System summary +- **Base model(s):** [`google/gemma-4-E4B-it`](https://huggingface.co/google/gemma-4-E4B-it), commit `ee0ef6023621cff504d758262d4e04895a5af4a2`, used **text-only**: the vision and audio towers are removed and every language-model weight is copied unchanged (`training/scripts/make_text_only.py` checks that the copy is exact). +- **What we did:** prompting only, with no fine-tuning, retrieval or extra data at inference. Every question is answered in Gemma 4's thinking mode with greedy decoding and up to 3,072 new tokens. We also compute a fast one-step answer: the next-token scores over the labels 1–4, averaged over 4 cyclic orders of the options. Where the fast answer disagrees with the greedy thinking answer, we sample 3 more reasoning paths (temperature 0.8, top-p 0.95, top-k 64, fixed seed 1234) and majority-vote the 4 answers, with ties going to the greedy answer. We call this *agreement-gated self-consistency*. On the test set it was triggered for 892 of 2,238 questions and changed 127 answers. +- **Total inference parameters:** **7.463B** (7,463,013,376; one model, no adapter). This is the count of the weights actually loaded, with tied embeddings counted once. +- **Extra data used for training or retrieval:** none in the final system. See "Experiments" below for data we used during development. + +**Test leaderboard score (Codabench, submission ID 50):** macro average **62.93**, with Easy 63.40, Medium 55.09 and Hard 70.31. + +## Models and resources in this submission +| Repo | What it is | +|---|---| +| [`SinhalaMMLU-SharedTask/team-nexus`](https://huggingface.co/SinhalaMMLU-SharedTask/team-nexus) | Code, inference script and final predictions (this repo) | +| [`SinhalaMMLU-SharedTask/team-nexus-gemma-4-e4b-it-text`](https://huggingface.co/SinhalaMMLU-SharedTask/team-nexus-gemma-4-e4b-it-text) | Text-only Gemma 4 E4B-it: unchanged weights of `google/gemma-4-E4B-it` with the vision and audio towers removed (Apache 2.0). `inference.py` pins commit `8f686039aad1a865c9335c17874d9b870fdec766`. | + +## How to run +```bash +pip install -r requirements.txt +python inference.py +``` +`inference.py` follows the organisers' template. Section 1 loads every `.json` file in `DATA_REPO`. Section 2 downloads the model from the repo above and runs our pipeline (`inference/predict.py` with `configs/submission_base_think_gated_sc.yaml`), the exact code that produced our Codabench file. Section 3 writes `predictions.jsonl`. + +- **GPU used:** 1× NVIDIA DGX Spark (GB10 Grace Blackwell, 128 GB unified CPU/GPU memory), bf16. +- **Approximate run time on the test set:** about **11.3 hours** (40,685 s for 2,238 questions) on the GB10. Most of this is thinking-mode generation; faster GPUs need proportionally less. +- **Anything else the organizers need to know:** + - **Memory:** generation uses batches of up to 64 sequences, with a padded prompt budget of 16,384 tokens and up to 3,072 new tokens each. Plan for a GPU with **80 GB or more**, such as an A100 or H100 80 GB. A 16 GB T4 is not enough. + - **Sampling:** about 40% of questions use the self-consistency step, which *samples*. It is seeded with `torch.manual_seed(1234)` immediately before the sampled passes. + - **Reproducibility:** thinking outputs are long (hundreds to thousands of tokens) and generated in bf16 with batched, length-sorted, left-padded batches. The answers therefore depend on the hardware and on which questions share a batch. Two separate full runs on our machine are consistent with identical greedy answers. Our two final candidates share the greedy settings and differ on only 100 Easy/Medium answers, fewer than the 127 that self-consistency changed. Running a 30-question subset changes the batches, and 22 of 30 answers matched our submitted file. **Please run the full dataset with the default settings.** On a different GPU type, expect a minority of answers to differ, with a similar overall score. + - **Greedy variant:** setting `SELF_CONSISTENCY = False` at the top of `inference.py` gives a fully greedy variant with no sampling at all. That is **not** the system we submitted; we did not submit it, so it has no test score. + - **Gating:** the model repo is public and the base model is not gated, so no login is needed. + +## Files +- `inference.py` – the entry point that runs the exact pipeline behind our final Codabench submission +- `inference/predict.py`, `eval/` – the pipeline code it calls (prompting, scoring, thinking, self-consistency) +- `configs/submission_base_think_gated_sc.yaml` – the settings of the submitted system; `configs/submission_base_think_greedy.yaml` – the greedy variant +- `predictions.jsonl` – the exact file we submitted to the final leaderboard (sha256 `4bf9f86282bffd69dcbd7c178e969532e1d3cf0462abc437663986c7a8b77ce1`) +- `requirements.txt` – package versions we used (Python 3.12.3) +- `training/` – all our data-preparation, training and evaluation code and configurations from development (see below) + +## Experiments (development only; nothing below is used by the final system) +`training/` contains the full project code. The final system uses only `training/scripts/make_text_only.py`, which builds the model repo above. During development we tried the following on held-out data, and none of it beat the base model with thinking: +- LoRA and full fine-tuning on Global-MMLU `si` +- continued pre-training on Sinhala Wikipedia +- distillation from Gemma 4 31B soft labels +- reasoning-trace fine-tuning on the model's own correct reasoning +- option-order voting + +Each step was decided on SinhalaMMLU dev or Global-MMLU validation subsets, never on test questions. The public data involved were: +- [CohereLabs/Global-MMLU](https://huggingface.co/datasets/CohereLabs/Global-MMLU) (`si`) +- [Sinhala Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) (`20231101.si`) +- [FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) (`sin_Sinh`) +- [NSINA](https://huggingface.co/datasets/sinhala-nlp/NSINA) +- SinhalaMMLU dev (`naist-nlp/SinhalaMMLU`; not redistributed, because of its CC BY-NC-ND licence) + +We also generated a small pilot set of synthetic MCQs from Sinhala Wikipedia passages with a locally run open model (Gemma 4 31B). No closed APIs were used at any stage. + +We checked that Global-MMLU does not overlap with SinhalaMMLU, using character n-gram and multilingual-embedding similarity. `training/DATA_LOG.md` and `training/MODEL_MANIFEST.md` give the details. + +## Citation +```bibtex +@inproceedings{pramodya-etal-2025-sinhalammlu, + title = "{S}inhala{MMLU}: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in {S}inhala", + author = "Pramodya, Ashmari and Nelki, Nirasha and Shalinda, Heshan and Liyanage, Chamila and Sakai, Yusuke and + Pushpananda, Randil and Weerasinghe, Ruvan and Kamigaito, Hidetaka and Watanabe, Taro", + booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing", + year = "2025", + pages = "32931--32949", + url = "https://aclanthology.org/2025.emnlp-main.1673/" +} +``` diff --git a/configs/submission_base_think_gated_sc.yaml b/configs/submission_base_think_gated_sc.yaml new file mode 100644 index 0000000000000000000000000000000000000000..f4313c8ed92d5d7738a50718b766fe9fb17b1b06 --- /dev/null +++ b/configs/submission_base_think_gated_sc.yaml @@ -0,0 +1,23 @@ +# Final candidate B (21 Oct): base Gemma 4 E4B, text-only (7.463B, no adapter), thinking for every question. +# Agreement-gated self-consistency on all levels: where the one-step answer disagrees with greedy thinking, add 3 sampled +# reasoning paths and take the majority vote (ties -> greedy). Evidence (dev/validation only): full dev, base thinking 63.0 vs +# model A vote 57.2 (+5.8 [+3.6, +8.0]); pooled proxies, gated self-consistency +2.70 [+1.18, +4.23] on 43% of questions. +model: models/gemma-4-E4B-it-text +lang: en +chat: true +method: logit # fast path, used only if a thinking answer cannot be read +n_perms: 4 +perm_mode: cyclic +seed: 1234 +dtype: bfloat16 +gpu_mem_gb: 100 +token_budget: 32768 +max_batch: 64 +thinking_levels: [easy, medium, hard] +think_max_new_tokens: 3072 +sc_levels: [easy, medium, hard] +sc_gate: disagree +sc_samples: 3 +sc_temperature: 0.8 +sc_top_p: 0.95 +sc_top_k: 64 diff --git a/configs/submission_base_think_greedy.yaml b/configs/submission_base_think_greedy.yaml new file mode 100644 index 0000000000000000000000000000000000000000..995a91a4b419e3537131342e11d8c0342f4dd821 --- /dev/null +++ b/configs/submission_base_think_greedy.yaml @@ -0,0 +1,23 @@ +# Fully greedy variant (NOT the submitted system): thinking for every question, no sampling, no self-consistency. +# Final candidate B (21 Oct): base Gemma 4 E4B, text-only (7.463B, no adapter), thinking for every question. +# Agreement-gated self-consistency on all levels: where the one-step answer disagrees with greedy thinking, add 3 sampled +# reasoning paths and take the majority vote (ties -> greedy). Evidence (dev/validation only): full dev, base thinking 63.0 vs +# model A vote 57.2 (+5.8 [+3.6, +8.0]); pooled proxies, gated self-consistency +2.70 [+1.18, +4.23] on 43% of questions. +model: models/gemma-4-E4B-it-text +lang: en +chat: true +method: logit # fast path, used only if a thinking answer cannot be read +n_perms: 4 +perm_mode: cyclic +seed: 1234 +dtype: bfloat16 +gpu_mem_gb: 100 +token_budget: 32768 +max_batch: 64 +thinking_levels: [easy, medium, hard] +think_max_new_tokens: 3072 +sc_levels: [] +sc_samples: 0 +sc_temperature: 0.8 +sc_top_p: 0.95 +sc_top_k: 64 diff --git a/eval/__init__.py b/eval/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/eval/combine_cv.py b/eval/combine_cv.py new file mode 100644 index 0000000000000000000000000000000000000000..f154f7b2df0229f5b8fd88b934f1f631bd00a508 --- /dev/null +++ b/eval/combine_cv.py @@ -0,0 +1,48 @@ +"""Combine k-fold runs (each scored only on its held-out fold) into one run over the full dev set. + + python -m eval.combine_cv --runs results/exp_devcv_lora_f* --name exp_devcv_lora_all + +Writes results//{metrics.json, report.md, predictions.jsonl} in the usual layout, so eval.summarize can put it +next to the zero-shot baseline. Fails if a question is missing or scored twice. +""" +from __future__ import annotations + +import argparse +import glob +import json +from pathlib import Path + +from .common import ROOT, read_jsonl, write_jsonl +from .metrics import markdown_report +from .run_eval import metrics_by_dataset + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--runs", nargs="+", required=True) + ap.add_argument("--name", required=True) + ap.add_argument("--data", default="data/processed/sinhalammlu_dev.jsonl") + args = ap.parse_args() + dirs = sorted(Path(d) for pat in args.runs for d in glob.glob(pat)) + records, first = [], None + for d in dirs: + records += read_jsonl(d / "predictions.jsonl") + first = first or json.loads((d / "metrics.json").read_text()) + items = {r["uid"]: r for r in read_jsonl(ROOT / args.data)} + scored = [r["uid"] for r in records if r["perm_k"] == 0] + assert len(scored) == len(set(scored)), "a question was scored by more than one fold" + assert set(scored) == set(items), f"{len(set(items) - set(scored))} dev questions were not scored" + out = ROOT / "results" / args.name + out.mkdir(parents=True, exist_ok=True) + metrics = {"by_dataset": metrics_by_dataset(items, records), + "run": first["run"] | {"run_name": args.name, "combined_from": [d.name for d in dirs]}} + write_jsonl(out / "predictions.jsonl", records) + (out / "metrics.json").write_text(json.dumps(metrics, indent=2, ensure_ascii=False)) + (out / "report.md").write_text("\n".join(markdown_report(f"{args.name} / {k}", m) + for k, m in metrics["by_dataset"].items())) + o = metrics["by_dataset"]["sinhalammlu"]["overall"] + print(f"{args.name}: dev acc={o['acc']:.4f} vote={o['acc_vote']:.4f} over {o['n']} questions from {len(dirs)} folds") + + +if __name__ == "__main__": + main() diff --git a/eval/common.py b/eval/common.py new file mode 100644 index 0000000000000000000000000000000000000000..30bd3ba54f58a6173afe6cb2976ee1179967924e --- /dev/null +++ b/eval/common.py @@ -0,0 +1,115 @@ +"""Shared helpers: secrets, seeding, JSONL I/O, environment fingerprint.""" +from __future__ import annotations + +import json +import os +import platform +import random +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] + + +def ensure_python_headers() -> None: + """Triton (torch.compile, Liger, flash-linear-attention) compiles helpers against Python.h. This machine has + no python3.12-dev and no sudo, so the headers are unpacked to ~/.local/pydev (README, setup). Point gcc at + them when the system copy is missing; a no-op on machines that have the headers.""" + import sysconfig + + if (Path(sysconfig.get_paths()["include"]) / "Python.h").exists(): + return + local = Path.home() / ".local" / "pydev" / "root" / "usr" / "include" + if (local / f"python{sys.version_info.major}.{sys.version_info.minor}" / "Python.h").exists(): + inc = [str(local / f"python{sys.version_info.major}.{sys.version_info.minor}"), str(local)] + os.environ["CPATH"] = os.pathsep.join(inc + [p for p in os.environ.get("CPATH", "").split(os.pathsep) if p]) + + +def load_env(path: Path = ROOT / ".env") -> None: + """Read KEY=VALUE lines from .env; map hf_token -> HF_TOKEN for huggingface_hub.""" + ensure_python_headers() + if not path.exists(): + return + for line in path.read_text().splitlines(): + line = line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key, val = (s.strip().strip('"').strip("'") for s in line.split("=", 1)) + if key.lower() in ("hf_token", "hugging_face_hub_token"): + key = "HF_TOKEN" + os.environ.setdefault(key, val) + + +def set_seed(seed: int) -> None: + random.seed(seed) + os.environ["PYTHONHASHSEED"] = str(seed) + try: + import numpy as np + + np.random.seed(seed) + except ImportError: + pass + try: + import torch + + torch.manual_seed(seed) + torch.cuda.manual_seed_all(seed) + torch.backends.cuda.matmul.allow_tf32 = False + torch.backends.cudnn.allow_tf32 = False + except ImportError: + pass + + +def cap_gpu_memory(gb: float | None = None) -> None: + """Limit PyTorch's CUDA allocator (default 100 GB, env SMMLU_GPU_MEM_GB). + + The DGX Spark shares one 128 GB pool between CPU and GPU: an unbounded GPU job does not get a CUDA + out-of-memory error, it starves the system and the kernel OOM killer takes down the desktop session and + tmux (10 Oct, 11:15). With the cap, the job fails with torch.OutOfMemoryError and everything else survives. + """ + import torch + + if not torch.cuda.is_available(): + return + gb = float(gb or os.environ.get("SMMLU_GPU_MEM_GB", 100)) + total = torch.cuda.get_device_properties(0).total_memory + torch.cuda.set_per_process_memory_fraction(min(gb * 1024**3 / total, 1.0), 0) + + +def read_jsonl(path: str | Path) -> list[dict]: + with open(path, encoding="utf-8") as f: + return [json.loads(line) for line in f if line.strip()] + + +def write_jsonl(path: str | Path, rows) -> None: + Path(path).parent.mkdir(parents=True, exist_ok=True) + with open(path, "w", encoding="utf-8") as f: + for r in rows: + f.write(json.dumps(r, ensure_ascii=False) + "\n") + + +def env_fingerprint() -> dict: + """Versions and code revision, stored with every run for reproducibility.""" + info = {"python": platform.python_version(), "machine": platform.machine()} + for mod in ("torch", "transformers", "datasets", "peft", "accelerate", "huggingface_hub"): + try: + info[mod] = __import__(mod).__version__ + except ImportError: + pass + try: + import torch + + if torch.cuda.is_available(): + info["gpu"] = torch.cuda.get_device_name(0) + info["cuda"] = torch.version.cuda + except ImportError: + pass + try: + rev = subprocess.run(["git", "rev-parse", "HEAD"], cwd=ROOT, capture_output=True, text=True) + dirty = subprocess.run(["git", "status", "--porcelain"], cwd=ROOT, capture_output=True, text=True) + info["git_commit"] = rev.stdout.strip() or None + info["git_dirty"] = bool(dirty.stdout.strip()) + except FileNotFoundError: + pass + return info diff --git a/eval/compare.py b/eval/compare.py new file mode 100644 index 0000000000000000000000000000000000000000..f0546d3cc0230e2f1985da7c1d89d172eda66d07 --- /dev/null +++ b/eval/compare.py @@ -0,0 +1,45 @@ +"""Paired comparison of runs on the same questions (bootstrap 95% CI of the accuracy difference). + + python -m eval.compare --base results/exp_sft_gmmlu_lora_logit --runs results/exp_x results/exp_y [--dataset sinhalammlu] +""" +from __future__ import annotations + +import argparse +from pathlib import Path + +import numpy as np + +from .common import read_jsonl +from .metrics import aggregate_items + + +def correctness(run: Path, dataset: str) -> tuple[dict, dict]: + recs = read_jsonl(run / "predictions.jsonl") + items = {r["uid"]: {"options": r["perm"], "answer": r["gold"]} for r in recs} + agg = aggregate_items(items, recs) + keep = {u for u in agg if u.startswith({"sinhalammlu": "smmlu/", "gmmlu_si": "gmmlu_si/"}.get(dataset, ""))} + return ({u: agg[u]["pred"] == agg[u]["gold"] for u in keep}, {u: agg[u]["vote"] == agg[u]["gold"] for u in keep}) + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--base", required=True, type=Path) + ap.add_argument("--runs", nargs="+", required=True, type=Path) + ap.add_argument("--dataset", default="sinhalammlu", choices=["sinhalammlu", "gmmlu_si"]) + args = ap.parse_args() + base = correctness(args.base, args.dataset) + rng = np.random.default_rng(0) + for run in args.runs: + c = correctness(run, args.dataset) + for k, label in ((0, "orig"), (1, "vote")): + u = sorted(set(base[k]) & set(c[k])) + if not u: + continue + d = np.array([c[k][x] - base[k][x] for x in u], dtype=float) + lo, hi = np.percentile([d[rng.integers(0, len(d), len(d))].mean() for _ in range(5000)], [2.5, 97.5]) + print(f"{run.name:40s} {label}: {100 * np.mean([c[k][x] for x in u]):5.1f} vs {100 * np.mean([base[k][x] for x in u]):5.1f}" + f" diff {100 * d.mean():+.2f} [{100 * lo:+.2f}, {100 * hi:+.2f}] n={len(u)}") + + +if __name__ == "__main__": + main() diff --git a/eval/decide_rft.py b/eval/decide_rft.py new file mode 100644 index 0000000000000000000000000000000000000000..95b2f8fe9dc449e2c8c4c5b83d5b4a5363810f37 --- /dev/null +++ b/eval/decide_rft.py @@ -0,0 +1,52 @@ +"""Go / no-go for the reasoning-trace model, decided by a rule fixed in advance (before any result is seen). + +Compares greedy thinking of the trained model (with its adapter) with greedy thinking of the base model on the SAME held-out +questions: the 400-question SinhalaMMLU dev sample (never used for traces or training) and the 300-question Global-MMLU +non-academic proxy (never used). Paired differences are pooled over both sets. + +GO if pooled difference >= +1.5 points AND neither set is worse than -3.0 points. Writes results/rft_decision.json. + + python -m eval.decide_rft --pairs think_base_smmlu_dev_sample400:think_rft_dev400:sinhalammlu \ + think_base_gmmlu_si_val_nonacademic300:think_rft_nonacademic300:gmmlu_si +""" +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +import numpy as np + +from .common import ROOT +from .compare import correctness + +MIN_POOLED, MIN_EACH = 1.5, -3.0 + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--pairs", nargs="+", required=True, help="base_run:new_run:dataset") + ap.add_argument("--out", default="rft_decision.json", help="file name under results/") + args = ap.parse_args() + diffs, per = [], {} + for pair in args.pairs: + b, n, ds = pair.split(":") + base, new = correctness(ROOT / "results" / b, ds)[0], correctness(ROOT / "results" / n, ds)[0] + uids = sorted(set(base) & set(new)) + d = np.array([new[u] - base[u] for u in uids], dtype=float) + per[n] = {"n": len(uids), "base": 100 * float(np.mean([base[u] for u in uids])), + "new": 100 * float(np.mean([new[u] for u in uids])), "diff": 100 * float(d.mean())} + diffs.append(d) + pooled = np.concatenate(diffs) + rng = np.random.default_rng(0) + lo, hi = np.percentile([pooled[rng.integers(0, len(pooled), len(pooled))].mean() for _ in range(5000)], [2.5, 97.5]) + go = bool(100 * pooled.mean() >= MIN_POOLED and all(v["diff"] >= MIN_EACH for v in per.values())) + out = {"rule": f"pooled diff >= {MIN_POOLED} and each set >= {MIN_EACH}", "per_set": per, "n": len(pooled), + "pooled_diff": 100 * float(pooled.mean()), "ci95": [100 * float(lo), 100 * float(hi)], "go": go} + (ROOT / "results" / args.out).write_text(json.dumps(out, indent=2)) + print(json.dumps(out, indent=2)) + print("DECISION:", "GO" if go else "NO-GO") + + +if __name__ == "__main__": + main() diff --git a/eval/metrics.py b/eval/metrics.py new file mode 100644 index 0000000000000000000000000000000000000000..35b9552c0cba5fca91bebc14a86ae9a59c5ee3f9 --- /dev/null +++ b/eval/metrics.py @@ -0,0 +1,116 @@ +"""Aggregate per-(question, permutation) records into accuracy tables and position-bias diagnostics.""" +from __future__ import annotations + +from collections import Counter, defaultdict +from statistics import mean, pstdev + +from .scorers import wilson + +GROUP_KEYS = ("dataset", "category", "subject", "level", "grade", "cultural_sensitivity") + + +def group_value(item: dict, key: str): + return item.get(key, item.get("meta", {}).get(key)) + + +def aggregate_items(items: dict[str, dict], records: list[dict]) -> dict[str, dict]: + """Combine permutations per question: mean of option probabilities (in original order) -> vote.""" + by_uid = defaultdict(list) + for r in records: + by_uid[r["uid"]].append(r) + out = {} + for uid, rs in by_uid.items(): + n = len(items[uid]["options"]) + rs.sort(key=lambda r: r["perm_k"]) + mean_probs = [mean(r["probs_orig"][j] for r in rs) for j in range(n)] + if any(sum(r["probs_orig"]) > 0 for r in rs): + vote = max(range(n), key=lambda j: (mean_probs[j], -j)) + 1 + else: + vote = None + preds = [r["pred_orig"] for r in rs] + out[uid] = { + "gold": items[uid].get("answer"), "pred": rs[0]["pred_orig"], "vote": vote, + "consistent": len(set(preds)) == 1 and preds[0] is not None, + "mean_probs": mean_probs, + } + return out + + +def _acc_block(uids: list[str], agg: dict, records_by_perm: dict[int, dict[str, dict]]) -> dict: + n = len(uids) + k = sum(agg[u]["pred"] == agg[u]["gold"] for u in uids) + kv = sum(agg[u]["vote"] == agg[u]["gold"] for u in uids) + per_perm = [sum(rp[u]["correct"] for u in uids) / n for rp in records_by_perm.values()] + lo, hi = wilson(k, n) + return { + "n": n, "acc": k / n, "ci95": [lo, hi], "acc_vote": kv / n, + "acc_perm_mean": mean(per_perm), "acc_perm_std": pstdev(per_perm) if len(per_perm) > 1 else 0.0, + "consistency": sum(agg[u]["consistent"] for u in uids) / n, + } + + +def compute_metrics(items: dict[str, dict], records: list[dict]) -> dict: + agg = aggregate_items(items, records) + records_by_perm: dict[int, dict[str, dict]] = defaultdict(dict) + for r in records: + records_by_perm[r["perm_k"]][r["uid"]] = r + uids = sorted(agg) + + m = {"overall": _acc_block(uids, agg, records_by_perm), "groups": {}} + for key in GROUP_KEYS: + groups = defaultdict(list) + for u in uids: + v = group_value(items[u], key) + if v is not None: + groups[str(v)].append(u) + if len(groups) > 1 or (key in ("dataset", "subject", "category") and groups): + m["groups"][key] = {g: _acc_block(us, agg, records_by_perm) for g, us in sorted(groups.items())} + # Macro average over subjects (each subject weighted equally) + if "subject" in m["groups"]: + m["overall"]["acc_macro_subject"] = mean(b["acc"] for b in m["groups"]["subject"].values()) + + # Position bias over all (question, permutation) records + n_opt = Counter(len(items[r["uid"]]["options"]) for r in records).most_common(1)[0][0] + rs = [r for r in records if len(items[r["uid"]]["options"]) == n_opt] + pred_pos = Counter(r["pred_display"] for r in rs) + gold_pos = Counter(r["gold_display"] for r in rs) + acc_by_gold = {p: mean(r["correct"] for r in rs if r["gold_display"] == p) for p in sorted(gold_pos)} + m["position_bias"] = { + "n_options": n_opt, + "pred_dist": {str(p): pred_pos.get(p, 0) / len(rs) for p in range(1, n_opt + 1)}, + "pred_none": pred_pos.get(None, 0) / len(rs), + "gold_dist": {str(p): gold_pos.get(p, 0) / len(rs) for p in range(1, n_opt + 1)}, + "acc_by_gold_position": {str(p): a for p, a in acc_by_gold.items()}, + } + if any("label_mass" in r for r in records): + m["overall"]["label_mass_mean"] = mean(r["label_mass"] for r in records if "label_mass" in r) + if any(r.get("method") == "generate" for r in records): + m["overall"]["parse_fail_rate"] = mean(r["pred_display"] is None for r in records) + return m + + +def markdown_report(name: str, m: dict) -> str: + o = m["overall"] + lines = [f"# {name}", "", + f"**Accuracy (original order): {o['acc']:.4f}** " + f"[95% CI {o['ci95'][0]:.3f}-{o['ci95'][1]:.3f}], n={o['n']} ", + f"Permutation-vote accuracy: {o['acc_vote']:.4f} ", + f"Mean ± std over permutations: {o['acc_perm_mean']:.4f} ± {o['acc_perm_std']:.4f} ", + f"Consistency across permutations: {o['consistency']:.3f} "] + if "acc_macro_subject" in o: + lines.append(f"Macro average over subjects: {o['acc_macro_subject']:.4f} ") + if "label_mass_mean" in o: + lines.append(f"Mean probability mass on label tokens: {o['label_mass_mean']:.3f} ") + if "parse_fail_rate" in o: + lines.append(f"Unparseable generations: {o['parse_fail_rate']:.3f} ") + for key, groups in m["groups"].items(): + lines += ["", f"## By {key}", "", "| group | n | acc | 95% CI | vote acc | consistency |", "|---|---:|---:|---|---:|---:|"] + for g, b in groups.items(): + lines.append(f"| {g} | {b['n']} | {b['acc']:.3f} | {b['ci95'][0]:.2f}-{b['ci95'][1]:.2f} | " + f"{b['acc_vote']:.3f} | {b['consistency']:.2f} |") + pb = m["position_bias"] + lines += ["", "## Position bias (all permutations)", "", "| position | predicted | gold | acc when gold here |", "|---|---:|---:|---:|"] + for p in pb["pred_dist"]: + lines.append(f"| {p} | {pb['pred_dist'][p]:.3f} | {pb['gold_dist'][p]:.3f} | " + f"{pb['acc_by_gold_position'].get(p, float('nan')):.3f} |") + return "\n".join(lines) + "\n" diff --git a/eval/perm_sc.py b/eval/perm_sc.py new file mode 100644 index 0000000000000000000000000000000000000000..1caa53aa1f7515a36c83052d4239c2478313d499 --- /dev/null +++ b/eval/perm_sc.py @@ -0,0 +1,70 @@ +"""Option-order self-consistency validation (decision for the Medium run). + +Prior answer = the current pipeline's answer on the stand-in questions: majority vote of greedy thinking + 3 sampled paths +(ties -> greedy), the analogue of candidate B's final answer. Candidate method: add the answers from K re-orderings of the +options (thinking each time, mapped back to the original numbering) and vote again (ties -> prior). + +GO if the candidate beats the prior by at least +1.0 point on the stand-in. Writes results/perm_decision.json. + + python -m eval.perm_sc --greedy results/think_base_gmmlu_si_val_nonacademic300 --samples results/sc3_base_nonacademic \ + --perms results/permthink_nonacademic300 +""" +from __future__ import annotations + +import argparse +import json +from collections import Counter, defaultdict +from pathlib import Path + +import numpy as np + +from .common import ROOT, read_jsonl + +MIN_GAIN = 1.0 + + +def vote(ballots: list, first_wins: bool = True): + c = Counter(b for b in ballots if b) + if not c: + return None + top = max(c.values()) + w = [a for a, n in c.items() if n == top] + return ballots[0] if ballots[0] in w else min(w) + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--greedy", required=True, type=Path) + ap.add_argument("--samples", required=True, type=Path) + ap.add_argument("--perms", required=True, type=Path) + args = ap.parse_args() + g = {r["uid"]: r for r in read_jsonl(args.greedy / "predictions.jsonl") if r["perm_k"] == 0} + s, p = defaultdict(list), defaultdict(list) + for r in read_jsonl(args.samples / "predictions.jsonl"): + s[r["uid"]].append(r["pred_orig"]) + for r in read_jsonl(args.perms / "predictions.jsonl"): + if r["perm_k"] >= 1: + p[r["uid"]].append(r["pred_orig"]) + uids = sorted(set(g) & set(s) & set(p)) + gold = {u: g[u]["gold"] for u in uids} + prior = {u: vote([g[u]["pred_orig"]] + s[u]) for u in uids} + cand = {u: vote([prior[u]] + p[u]) for u in uids} + perms_only = {u: vote([g[u]["pred_orig"]] + p[u]) for u in uids} + acc = lambda m: np.array([m[u] == gold[u] for u in uids], dtype=float) # noqa: E731 + a_g, a_prior, a_cand, a_perm = acc({u: g[u]["pred_orig"] for u in uids}), acc(prior), acc(cand), acc(perms_only) + d = a_cand - a_prior + rng = np.random.default_rng(0) + lo, hi = np.percentile([d[rng.integers(0, len(d), len(d))].mean() for _ in range(5000)], [2.5, 97.5]) + singles = [100 * float(np.mean([p[u][i] == gold[u] for u in uids])) for i in range(min(len(p[u]) for u in uids))] + out = {"n": len(uids), "greedy": 100 * a_g.mean(), "prior_vote_greedy_plus_samples": 100 * a_prior.mean(), + "single_reordered_runs": singles, "greedy_plus_reorders_vote": 100 * a_perm.mean(), + "candidate_prior_plus_reorders": 100 * a_cand.mean(), "gain_vs_prior": 100 * float(d.mean()), + "ci95": [100 * float(lo), 100 * float(hi)], "rule": f"candidate - prior >= {MIN_GAIN}", + "go": bool(100 * d.mean() >= MIN_GAIN)} + (ROOT / "results" / "perm_decision.json").write_text(json.dumps(out, indent=2)) + print(json.dumps(out, indent=2)) + print("DECISION:", "GO" if out["go"] else "NO-GO") + + +if __name__ == "__main__": + main() diff --git a/eval/pool_decide.py b/eval/pool_decide.py new file mode 100644 index 0000000000000000000000000000000000000000..623aaf9c7508540f0b42b69e66815da49da40787 --- /dev/null +++ b/eval/pool_decide.py @@ -0,0 +1,68 @@ +"""Go / no-go for cross-model pooling, by a rule fixed (27 Oct, before any of these runs existed) on FRESH stand-in questions +(gmmlu_si_val_nonacademic_fresh300: never used for any earlier decision, training or trace generation). + +Baseline = candidate B's method: greedy thinking of the base model; where the fast one-step answer (base model, 4-order logit +vote) disagrees with it, vote over greedy + 3 sampled paths (ties -> greedy). +Candidate = vote over base greedy + 3 base samples + RFT2 greedy + 3 RFT2 samples, on every question (ties -> base greedy). + +GO if candidate - baseline >= +1.0 point. Writes results/pool_decision.json. + + python -m eval.pool_decide --fast pool_fast_base_fresh300 --base-greedy pool_think_base_fresh300 \ + --base-samples pool_sc3_base_fresh300 --rft-greedy pool_think_rft2_fresh300 --rft-samples pool_sc3_rft2_fresh300 +""" +from __future__ import annotations + +import argparse +import json +from collections import defaultdict + +import numpy as np + +from .common import ROOT, read_jsonl +from .perm_sc import vote + +MIN_GAIN = 1.0 + + +def main() -> None: + ap = argparse.ArgumentParser() + for a in ("fast", "base-greedy", "base-samples", "rft-greedy", "rft-samples"): + ap.add_argument(f"--{a}", required=True) + args = ap.parse_args() + + def runs(name: str) -> tuple[dict, dict]: + preds, probs = defaultdict(list), defaultdict(lambda: np.zeros(4)) + for r in read_jsonl(ROOT / "results" / name / "predictions.jsonl"): + preds[r["uid"]].append(r["pred_orig"]) + probs[r["uid"]][: len(r["probs_orig"])] += r["probs_orig"] + return preds, probs + + _, fast = runs(args.fast) + bg, _ = runs(args.base_greedy) + bs, _ = runs(args.base_samples) + rg, _ = runs(args.rft_greedy) + rs, _ = runs(args.rft_samples) + gold = {r["uid"]: r["gold"] for r in read_jsonl(ROOT / "results" / args.base_greedy / "predictions.jsonl")} + uids = sorted(set(bg) & set(bs) & set(rg) & set(rs) & set(fast)) + gated = {u: int(fast[u].argmax()) + 1 != bg[u][0] for u in uids} + baseline = {u: vote([bg[u][0]] + (bs[u] if gated[u] else [])) for u in uids} + cand = {u: vote([bg[u][0]] + bs[u] + [rg[u][0]] + rs[u]) for u in uids} + acc = lambda m: np.array([m[u] == gold[u] for u in uids], dtype=float) # noqa: E731 + d = acc(cand) - acc(baseline) + rng = np.random.default_rng(0) + lo, hi = np.percentile([d[rng.integers(0, len(d), len(d))].mean() for _ in range(5000)], [2.5, 97.5]) + out = {"n": len(uids), "gated_fraction": float(np.mean(list(gated.values()))), + "base_greedy": 100 * float(acc({u: bg[u][0] for u in uids}).mean()), + "rft2_greedy": 100 * float(acc({u: rg[u][0] for u in uids}).mean()), + "base_vote_all4": 100 * float(acc({u: vote([bg[u][0]] + bs[u]) for u in uids}).mean()), + "rft2_vote_all4": 100 * float(acc({u: vote([rg[u][0]] + rs[u]) for u in uids}).mean()), + "baseline_B": 100 * float(acc(baseline).mean()), "candidate_pooled8": 100 * float(acc(cand).mean()), + "gain": 100 * float(d.mean()), "ci95": [100 * float(lo), 100 * float(hi)], + "rule": f"candidate - baseline >= {MIN_GAIN}", "go": bool(100 * d.mean() >= MIN_GAIN)} + (ROOT / "results" / "pool_decision.json").write_text(json.dumps(out, indent=2)) + print(json.dumps(out, indent=2)) + print("DECISION:", "GO" if out["go"] else "NO-GO") + + +if __name__ == "__main__": + main() diff --git a/eval/prompts.py b/eval/prompts.py new file mode 100644 index 0000000000000000000000000000000000000000..009b4e1847ea0e1b0538dbf67f0bd0fb2668d9e2 --- /dev/null +++ b/eval/prompts.py @@ -0,0 +1,71 @@ +"""Prompt construction. Options are numbered 1..N because Codabench answers are digits ("1".."4"). + +Every model gets the same template text; only the wrapping differs (chat template vs raw text). +Few-shot examples are a fixed list, identical for every question (shared-task rule). +""" +from __future__ import annotations + +TEMPLATES = { + "en": { + "header": "The following is a multiple choice question about {subject}. " + "Reply with only the number (1-{n}) of the correct answer.", + "answer": "Answer:", + }, + # Sinhala wording should be reviewed by a native speaker. + "si": { + "header": "පහත දැක්වෙන්නේ {subject} පිළිබඳ බහුවරණ ප්‍රශ්නයකි. " + "නිවැරදි පිළිතුරේ අංකය (1-{n}) පමණක් ලියන්න.", + "answer": "පිළිතුර:", + }, +} + + +def subject_name(item: dict, lang: str) -> str: + if lang == "si" and item.get("meta", {}).get("subject_original"): + return item["meta"]["subject_original"] + return item["subject"].replace("_", " ") + + +def question_block(question: str, options: list[str]) -> str: + lines = [question] + [f"{i}. {opt}" for i, opt in enumerate(options, 1)] + return "\n".join(lines) + + +def user_text(item: dict, options: list[str], lang: str) -> str: + t = TEMPLATES[lang] + header = t["header"].format(subject=subject_name(item, lang), n=len(options)) + return f"{header}\n\n{question_block(item['question'], options)}" + + +class PromptBuilder: + def __init__(self, tokenizer, lang: str = "en", chat: bool = True, + fewshot: list[dict] | None = None, answer_prefix: str = "", + chat_kwargs: dict | None = None): + self.tok, self.lang, self.chat = tokenizer, lang, chat + self.fewshot = fewshot or [] + self.answer_prefix = answer_prefix + self.chat_kwargs = {"enable_thinking": False, **(chat_kwargs or {})} + + @property + def add_special_tokens(self) -> bool: + # Chat templates already insert BOS; raw prompts need the tokenizer to add it. + return not self.chat + + def mcq(self, item: dict, options: list[str]) -> str: + """Prompt whose next token should be the answer number.""" + ans = TEMPLATES[self.lang]["answer"] + if self.chat: + msgs = [] + for ex in self.fewshot: + msgs.append({"role": "user", "content": user_text(ex, ex["options"], self.lang)}) + msgs.append({"role": "assistant", "content": str(ex["answer"])}) + msgs.append({"role": "user", "content": user_text(item, options, self.lang)}) + prompt = self.tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, + **self.chat_kwargs) + return prompt + self.answer_prefix + shots = [f"{user_text(ex, ex['options'], self.lang)}\n{ans} {ex['answer']}" for ex in self.fewshot] + return "\n\n".join(shots + [f"{user_text(item, options, self.lang)}\n{ans}"]) + + def cloze_context(self, item: dict) -> str: + """Context for option-likelihood scoring: question only, options not listed.""" + return f"{item['question']}\n{TEMPLATES[self.lang]['answer']}" diff --git a/eval/recompute.py b/eval/recompute.py new file mode 100644 index 0000000000000000000000000000000000000000..e9bf7d07eaf43c12c62c942e9f2f7559ab2c3b2d --- /dev/null +++ b/eval/recompute.py @@ -0,0 +1,34 @@ +"""Recompute metrics.json / report.md of finished runs from their saved predictions (no model needed). +Use after changing data grouping or metrics code. Usage: python -m eval.recompute results/base_* +""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +from .common import read_jsonl +from .metrics import markdown_report +from .run_eval import metrics_by_dataset, resolve_data + + +def main() -> None: + for d in map(Path, sys.argv[1:]): + if not (d / "metrics.json").exists(): + continue + old = json.loads((d / "metrics.json").read_text()) + records = read_jsonl(d / "predictions.jsonl") + uids = {r["uid"] for r in records} + items = {r["uid"]: r for name in old["run"]["args"]["data"] for r in read_jsonl(resolve_data(name)) if r["uid"] in uids} + missing = uids - items.keys() + if missing: + print(f"{d.name}: {len(missing)} predicted uids not in current data, skipped") + continue + new = {"by_dataset": metrics_by_dataset(items, records), "run": old["run"]} + (d / "metrics.json").write_text(json.dumps(new, indent=2, ensure_ascii=False)) + (d / "report.md").write_text("\n".join(markdown_report(f"{d.name} / {k}", m) for k, m in new["by_dataset"].items())) + print(f"{d.name}: recomputed") + + +if __name__ == "__main__": + main() diff --git a/eval/run_eval.py b/eval/run_eval.py new file mode 100644 index 0000000000000000000000000000000000000000..92084f4f201a1e779458854463533c57100976c0 --- /dev/null +++ b/eval/run_eval.py @@ -0,0 +1,208 @@ +"""Evaluate one model on one or more processed MCQ files. + +Example: + python -m eval.run_eval --model Qwen/Qwen3-4B-Instruct-2507 --data sinhalammlu_dev gmmlu_si_val \ + --method logit --n-perms 4 --run-name qwen3-4b-inst_logit + +Outputs results//{config.json, predictions.jsonl, metrics.json, report.md} +and, with --write-submission, a Codabench-format predictions.zip per dataset. +""" +from __future__ import annotations + +import argparse +import json +import random +import re +import time +import zipfile +from pathlib import Path + +import yaml + +from .common import ROOT, cap_gpu_memory, env_fingerprint, load_env, read_jsonl, set_seed, write_jsonl +from .metrics import compute_metrics, markdown_report +from .prompts import PromptBuilder, user_text + +PROCESSED = ROOT / "data" / "processed" +BASE_NAME = re.compile(r"(-base|-pt)$|^google/gemma-4-e[24]b$", re.I) + + +def make_perm(n: int, k: int, mode: str, seed: int, uid: str) -> list[int]: + """perm[j] = original 0-based index shown at display position j. k=0 is the original order.""" + if k == 0: + return list(range(n)) + if mode == "identity": # repeated samples of the same prompt (self-consistency), no reordering + return list(range(n)) + if mode == "cyclic" and k < n: + return [(j + k) % n for j in range(n)] + rng = random.Random(f"{seed}|{uid}|{k}") + p = list(range(n)) + rng.shuffle(p) + return p + + +def resolve_data(name: str) -> Path: + p = Path(name) + return p if p.suffix == ".jsonl" else PROCESSED / f"{name}.jsonl" + + +def score_items(items: dict[str, dict], pb: PromptBuilder, scorer, method: str, n_perms: int, + perm_mode: str, seed: int, first_perm: int = 0) -> tuple[list[dict], str | None]: + """One record per (question, option order). Shared by evaluation and the submission pipeline. + Items without a gold answer (test data) get gold/correct = None.""" + n_perms = 1 if method == "cloze" else n_perms + jobs = [(uid, k, make_perm(len(it["options"]), k, perm_mode, seed, uid)) + for uid, it in items.items() for k in range(first_perm, n_perms)] + first_prompt = None + if method == "cloze": + res = scorer.cloze([pb.cloze_context(items[u]) for u, _, _ in jobs], [items[u]["options"] for u, _, _ in jobs]) + else: + prompts = [pb.mcq(items[u], [items[u]["options"][j] for j in perm]) for u, _, perm in jobs] + fn = scorer.logit if method == "logit" else scorer.generate + res = fn(prompts, [len(items[u]["options"]) for u, _, _ in jobs]) + first_prompt = prompts[0] if prompts else None + + records = [] + for (uid, k, perm), r in zip(jobs, res): + gold = items[uid].get("answer") + probs_orig = [0.0] * len(perm) + for j, p in enumerate(r["probs"]): + probs_orig[perm[j]] = p + pred_orig = perm[r["pred"] - 1] + 1 if r["pred"] else None + rec = {"uid": uid, "perm_k": k, "perm": perm, "method": method, + "pred_display": r["pred"], "gold_display": perm.index(gold - 1) + 1 if gold else None, + "pred_orig": pred_orig, "gold": gold, "correct": pred_orig == gold if gold else None, + "probs_orig": probs_orig} + for extra in ("label_mass", "text", "new_tokens"): + if extra in r: + rec[extra] = r[extra] + records.append(rec) + return records, first_prompt + + +def parse_args() -> argparse.Namespace: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--config", help="YAML with any of the options below (CLI wins)") + ap.add_argument("--model") + ap.add_argument("--revision") + ap.add_argument("--adapter", help="LoRA adapter directory loaded on top of --model") + ap.add_argument("--data", nargs="+", default=["sinhalammlu_dev"]) + ap.add_argument("--method", choices=["logit", "generate", "cloze"], default="logit") + ap.add_argument("--lang", choices=["en", "si"], default="en", help="instruction language") + ap.add_argument("--chat", choices=["auto", "on", "off"], default="auto") + ap.add_argument("--answer-prefix", default="") + ap.add_argument("--thinking", action="store_true", help="enable the chat template's thinking mode (use with --method generate)") + ap.add_argument("--n-perms", type=int, default=1, help="option orders per question (1 = original only)") + ap.add_argument("--first-perm", type=int, default=0, help="skip the first orders (1 = skip the original order)") + ap.add_argument("--perm-mode", choices=["cyclic", "random", "identity"], default="cyclic", + help="identity + --temperature > 0: n-perms independent samples of the same prompt") + ap.add_argument("--temperature", type=float, default=0.0, help="> 0: sampled generation (self-consistency)") + ap.add_argument("--top-p", type=float, default=0.95) + ap.add_argument("--top-k", type=int, default=64) + ap.add_argument("--fewshot-file", help="processed JSONL; the same examples precede every question") + ap.add_argument("--n-shot", type=int, default=0) + ap.add_argument("--max-new-tokens", type=int, default=16) + ap.add_argument("--token-budget", type=int, default=32768, help="padded tokens per forward batch") + ap.add_argument("--max-batch", type=int, default=64) + ap.add_argument("--dtype", default="bfloat16") + ap.add_argument("--gpu-mem-gb", type=float, help="PyTorch memory cap (default 100, env SMMLU_GPU_MEM_GB)") + ap.add_argument("--limit", type=int, help="first N questions per dataset (smoke tests)") + ap.add_argument("--seed", type=int, default=1234) + ap.add_argument("--run-name") + ap.add_argument("--out-dir", default=str(ROOT / "results")) + ap.add_argument("--write-submission", action="store_true") + args, _ = ap.parse_known_args() + if args.config: + ap.set_defaults(**{k.replace("-", "_"): v for k, v in yaml.safe_load(open(args.config)).items()}) + args = ap.parse_args() + if not args.model: + ap.error("--model is required") + return args + + +def main() -> None: + args = parse_args() + load_env() + set_seed(args.seed) + cap_gpu_memory(args.gpu_mem_gb) + from .scorers import Scorer, load_model, loaded_param_stats + + run_name = args.run_name or f"{args.model.split('/')[-1]}_{args.method}_{args.lang}_{args.n_shot}shot" + out = Path(args.out_dir) / run_name + out.mkdir(parents=True, exist_ok=True) + + fewshot = read_jsonl(resolve_data(args.fewshot_file))[: args.n_shot] if args.n_shot else [] + shot_uids = {e["uid"] for e in fewshot} + items: dict[str, dict] = {} + for d in args.data: + rows = [r for r in read_jsonl(resolve_data(d)) if r["uid"] not in shot_uids] + items.update({r["uid"]: r for r in (rows[: args.limit] if args.limit else rows)}) + + t0 = time.time() + model, tok = load_model(args.model, args.revision, args.dtype, args.adapter) + chat = {"on": True, "off": False}.get(args.chat, tok.chat_template is not None and not BASE_NAME.search(args.model)) + pb = PromptBuilder(tok, lang=args.lang, chat=chat, fewshot=fewshot, answer_prefix=args.answer_prefix, + chat_kwargs={"enable_thinking": args.thinking}) + scorer = Scorer(model, tok, pb.add_special_tokens, args.token_budget, args.max_batch, args.max_new_tokens) + if args.temperature > 0: + scorer.sampling = {"do_sample": True, "temperature": args.temperature, "top_p": args.top_p, "top_k": args.top_k} + load_s = time.time() - t0 + + t0 = time.time() + records, first_prompt = score_items(items, pb, scorer, args.method, args.n_perms, args.perm_mode, args.seed, args.first_perm) + score_s = time.time() - t0 + if first_prompt: + (out / "example_prompt.txt").write_text(first_prompt, encoding="utf-8") + + # Tokens per character of the question block: tokenizer efficiency on Sinhala + sample = list(items.values())[:500] + texts = [user_text(it, it["options"], args.lang) for it in sample] + n_tok = sum(len(x) for x in tok(texts, add_special_tokens=False)["input_ids"]) + + metrics = {"by_dataset": metrics_by_dataset(items, records)} + metrics["run"] = { + "run_name": run_name, "args": vars(args), "chat": chat, "n_questions": len(items), + "n_forward_items": len(records), "load_seconds": round(load_s, 1), "score_seconds": round(score_s, 1), + "tokens_per_char": n_tok / sum(len(t) for t in texts), + "params": loaded_param_stats(model), "env": env_fingerprint(), + } + (out / "config.json").write_text(json.dumps(vars(args), indent=2)) + write_jsonl(out / "predictions.jsonl", records) + (out / "metrics.json").write_text(json.dumps(metrics, indent=2, ensure_ascii=False)) + report = "\n".join(markdown_report(f"{run_name} / {d}", m) for d, m in metrics["by_dataset"].items()) + (out / "report.md").write_text(report, encoding="utf-8") + + if args.write_submission: + write_submissions(out, items, records) + + for d, m in metrics["by_dataset"].items(): + o = m["overall"] + print(f"{run_name} | {d}: acc={o['acc']:.4f} vote={o['acc_vote']:.4f} " + f"perm={o['acc_perm_mean']:.4f}±{o['acc_perm_std']:.4f} n={o['n']}") + print(f"load {load_s:.0f}s, scoring {score_s:.0f}s -> {out}") + + +def metrics_by_dataset(items: dict, records: list[dict]) -> dict: + out = {} + for d in sorted({it["dataset"] for it in items.values()}): + sub_items = {u: it for u, it in items.items() if it["dataset"] == d} + out[d] = compute_metrics(sub_items, [r for r in records if r["uid"] in sub_items]) + return out + + +def write_submissions(out: Path, items: dict, records: list[dict]) -> None: + """Codabench format: predictions.zip containing predictions.jsonl, answer = permutation-vote prediction.""" + from .metrics import aggregate_items + + agg = aggregate_items(items, records) + for d in sorted({it["dataset"] for it in items.values()}): + rows = [{"q_no": it["q_no"], "subject": it.get("subject_raw", it["subject"]), "category": it["category"], + "level": it["level"], "answer": str(agg[u]["vote"] or 1)} + for u, it in items.items() if it["dataset"] == d] + path = out / f"submission_{d}.zip" + with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as z: + z.writestr("predictions.jsonl", "".join(json.dumps(r, ensure_ascii=False) + "\n" for r in rows)) + + +if __name__ == "__main__": + main() diff --git a/eval/scorers.py b/eval/scorers.py new file mode 100644 index 0000000000000000000000000000000000000000..3e4cac0356e933ae5f62c6e8195f492c8f27edf0 --- /dev/null +++ b/eval/scorers.py @@ -0,0 +1,224 @@ +"""Model loading and the three scoring methods. + + logit next-token log-probs restricted to the option labels "1".."N" (argmax) + generate greedy decoding, first standalone label digit in the output is the prediction + cloze length-normalised log-likelihood of each option text given the question +""" +from __future__ import annotations + +import math +import re + +import torch +from tqdm.auto import tqdm + + +def resolve_revision(name: str, revision: str | None) -> tuple[str | None, str | None]: + """(revision to load, commit to record). Local dirs record their SOURCE.json; offline runs need a full sha.""" + import json + import os + from pathlib import Path + + if Path(name).is_dir(): + src = Path(name) / "SOURCE.json" + if not src.exists(): + return None, "local" + meta = json.loads(src.read_text()) + return None, meta.get("model_id") or meta["source_commit"] + "+text-only" + if revision and re.fullmatch(r"[0-9a-f]{40}", revision): + return revision, revision + if os.environ.get("HF_HUB_OFFLINE") == "1": + raise ValueError(f"offline run needs a full 40-character commit sha for {name}") + from huggingface_hub import HfApi + + sha = HfApi().model_info(name, revision=revision).sha # pin the exact commit + return sha, sha + + +def load_model(name: str, revision: str | None = None, dtype: str = "bfloat16", adapter: str | None = None): + from transformers import AutoModelForCausalLM, AutoTokenizer + + revision, commit = resolve_revision(name, revision) + kw = dict(dtype=getattr(torch, dtype), device_map="cuda", revision=revision) + try: + model = AutoModelForCausalLM.from_pretrained(name, **kw) + except ValueError: # some multimodal checkpoints only register under image-text-to-text + from transformers import AutoModelForImageTextToText + + model = AutoModelForImageTextToText.from_pretrained(name, **kw) + if adapter: # LoRA adapter kept separate (can be switched off per question with model.disable_adapter()) + from peft import PeftModel + + model = PeftModel.from_pretrained(model, adapter) + commit = f"{commit}+adapter:{adapter}" + model.eval() + tok = AutoTokenizer.from_pretrained(name, revision=revision) + tok.padding_side = "left" + if tok.pad_token is None: + tok.pad_token = tok.eos_token + model.config.hub_commit = commit + return model, tok + + +def loaded_param_stats(model) -> dict: + """Parameters actually resident at inference (tied weights counted once).""" + by_module: dict[str, int] = {} + for name, p in model.named_parameters(): # named_parameters() already de-duplicates tied tensors + top = ".".join(name.split(".")[:2]) + by_module[top] = by_module.get(top, 0) + p.numel() + non_text = {k: v for k, v in by_module.items() if re.search(r"vision|visual|audio|image|mm_proj", k)} + total = sum(by_module.values()) + return {"loaded_total": total, "loaded_non_text": sum(non_text.values()), + "non_text_modules": non_text, "commit": getattr(model.config, "hub_commit", None)} + + +def label_token_ids(tok, n: int) -> list[list[int]]: + """Single-token encodings of each label ("1" and " 1" variants).""" + out = [] + for i in range(1, n + 1): + ids = set() + for v in (str(i), f" {i}"): + enc = tok.encode(v, add_special_tokens=False) + if len(enc) == 1: + ids.add(enc[0]) + if not ids: + raise ValueError(f"label {i} has no single-token encoding") + out.append(sorted(ids)) + flat = [t for ids in out for t in ids] + assert len(flat) == len(set(flat)), "label token ids overlap" + return out + + +def _batches(lengths: list[int], token_budget: int, max_batch: int): + """Length-sorted batches bounded by padded-token budget.""" + order = sorted(range(len(lengths)), key=lambda i: -lengths[i]) + batch, longest = [], 0 + for i in order: + new_longest = max(longest, lengths[i]) + if batch and (new_longest * (len(batch) + 1) > token_budget or len(batch) >= max_batch): + yield batch + batch, new_longest = [], lengths[i] + batch.append(i) + longest = new_longest + if batch: + yield batch + + +def _forward_last(model, enc, keep: int): + try: + return model(**enc, logits_to_keep=keep).logits + except TypeError: + return model(**enc).logits[:, -keep:] + + +class Scorer: + def __init__(self, model, tok, add_special_tokens: bool, token_budget: int = 32768, + max_batch: int = 64, max_new_tokens: int = 16): + self.model, self.tok = model, tok + self.add_special = add_special_tokens + self.token_budget, self.max_batch = token_budget, max_batch + self.max_new_tokens = max_new_tokens + self.keep_raw = False # also return the raw completion (thought channel included) for training on traces + self.sampling: dict = {} # e.g. {"do_sample": True, "temperature": 0.8, "top_p": 0.95, "top_k": 64} + + def _encode(self, texts: list[str]): + return self.tok(texts, return_tensors="pt", padding=True, + add_special_tokens=self.add_special).to(self.model.device) + + def _lengths(self, texts: list[str]) -> list[int]: + return [len(x) for x in self.tok(texts, add_special_tokens=self.add_special)["input_ids"]] + + @torch.no_grad() + def logit(self, prompts: list[str], n_options: list[int]) -> list[dict]: + cache: dict[int, list[list[int]]] = {} + results: list[dict | None] = [None] * len(prompts) + lengths = self._lengths(prompts) + for idx in tqdm(list(_batches(lengths, self.token_budget, self.max_batch)), desc="logit", leave=False): + enc = self._encode([prompts[i] for i in idx]) + logp = torch.log_softmax(_forward_last(self.model, enc, 1)[:, -1].float(), dim=-1) + for row, i in enumerate(idx): + n = n_options[i] + labels = cache.setdefault(n, label_token_ids(self.tok, n)) + scores = torch.stack([torch.logsumexp(logp[row, ids], 0) for ids in labels]) + probs = torch.softmax(scores, 0) + results[i] = {"probs": probs.tolist(), "pred": int(probs.argmax()) + 1, + "label_mass": float(scores.exp().sum())} + return results + + @torch.no_grad() + def generate(self, prompts: list[str], n_options: list[int]) -> list[dict]: + results: list[dict | None] = [None] * len(prompts) + lengths = self._lengths(prompts) + budget = max(self.token_budget // 2, 1) + for idx in tqdm(list(_batches(lengths, budget, self.max_batch)), desc="generate", leave=False): + enc = self._encode([prompts[i] for i in idx]) + gen_kw = self.sampling or {"do_sample": False, "temperature": None, "top_p": None, "top_k": None} + out = self.model.generate(**enc, max_new_tokens=self.max_new_tokens, pad_token_id=self.tok.pad_token_id, + **gen_kw) + new = out[:, enc["input_ids"].shape[1]:] + raw = self.tok.batch_decode(new, skip_special_tokens=False) + texts = self.tok.batch_decode(new, skip_special_tokens=True) + for row, i in enumerate(idx): + n = n_options[i] + answer_text = texts[row] + if "<|channel>" in raw[row]: # thinking mode: the answer is what follows the closed thought channel + answer_text = raw[row].split("")[-1] if "" in raw[row] else "" + answer_text = re.sub(r"<[^>]*\|>|<\|[^>]*>", " ", answer_text) + m = re.search(rf"(?") + results[i]["raw"] = rc[: cut + len("")] if cut >= 0 else rc + return results + + @torch.no_grad() + def cloze(self, contexts: list[str], options: list[list[str]]) -> list[dict]: + """Left-padded so every continuation ends at the last position; keeps only needed logits.""" + pairs = [] # (item index, ctx ids, cont ids, n chars) + for i, (ctx, opts) in enumerate(zip(contexts, options)): + ctx_ids = self.tok.encode(ctx, add_special_tokens=True) + for opt in opts: + cont = self.tok.encode(" " + opt, add_special_tokens=False) + pairs.append((i, ctx_ids, cont, len(opt))) + lengths = [len(c) + len(o) for _, c, o, _ in pairs] + scores = [0.0] * len(pairs) + pad = self.tok.pad_token_id + for idx in tqdm(list(_batches(lengths, self.token_budget, self.max_batch)), desc="cloze", leave=False): + L = max(lengths[j] for j in idx) + keep = max(len(pairs[j][2]) for j in idx) + 1 + ids = torch.full((len(idx), L), pad, dtype=torch.long) + mask = torch.zeros((len(idx), L), dtype=torch.long) + for row, j in enumerate(idx): + seq = pairs[j][1] + pairs[j][2] + ids[row, L - len(seq):] = torch.tensor(seq) + mask[row, L - len(seq):] = 1 + enc = {"input_ids": ids.to(self.model.device), "attention_mask": mask.to(self.model.device)} + logp = torch.log_softmax(_forward_last(self.model, enc, keep).float(), dim=-1) + for row, j in enumerate(idx): + cont = pairs[j][2] + k = len(cont) + # logits at kept positions [-k-1 .. -2] predict tokens at [-k .. -1] + lp = logp[row, keep - k - 1: keep - 1].gather(-1, torch.tensor(cont, device=logp.device)[:, None]) + scores[j] = float(lp.sum()) / max(pairs[j][3], 1) + results: list[dict] = [] + pos = 0 + for opts in options: + s = torch.tensor(scores[pos: pos + len(opts)]) + pos += len(opts) + probs = torch.softmax(s, 0) + results.append({"probs": probs.tolist(), "pred": int(probs.argmax()) + 1}) + return results + + +def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]: + if n == 0: + return (math.nan, math.nan) + p = k / n + d = 1 + z * z / n + c = (p + z * z / (2 * n)) / d + h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d + return (c - h, c + h) diff --git a/eval/self_consistency.py b/eval/self_consistency.py new file mode 100644 index 0000000000000000000000000000000000000000..62e14835649728722a462bae29e966ef27946223 --- /dev/null +++ b/eval/self_consistency.py @@ -0,0 +1,52 @@ +"""Self-consistency: majority vote over a greedy thinking run and sampled thinking runs of the same questions. + + python -m eval.self_consistency --greedy results/think_base_academic --samples results/sc3_base_academic + +Votes = greedy answer + every sampled answer (unparsed samples abstain); ties go to the greedy answer. Prints accuracy of +greedy alone, samples alone (each), and the vote, with a paired bootstrap CI of vote minus greedy. +""" +from __future__ import annotations + +import argparse +from collections import Counter, defaultdict +from pathlib import Path + +import numpy as np + +from .common import read_jsonl + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--greedy", required=True, type=Path) + ap.add_argument("--samples", required=True, type=Path) + args = ap.parse_args() + greedy = {r["uid"]: r for r in read_jsonl(args.greedy / "predictions.jsonl") if r["perm_k"] == 0} + samples = defaultdict(list) + for r in read_jsonl(args.samples / "predictions.jsonl"): + samples[r["uid"]].append(r["pred_orig"]) + uids = sorted(set(greedy) & set(samples)) + g = np.array([greedy[u]["pred_orig"] == greedy[u]["gold"] for u in uids], dtype=float) + vote = [] + for u in uids: + ballots = [greedy[u]["pred_orig"]] + samples[u] + c = Counter(b for b in ballots if b is not None) + if not c: + vote.append(False) + continue + top = max(c.values()) + winners = [a for a, n in c.items() if n == top] + choice = greedy[u]["pred_orig"] if greedy[u]["pred_orig"] in winners else min(winners) + vote.append(choice == greedy[u]["gold"]) + v = np.array(vote, dtype=float) + k = max(len(samples[u]) for u in uids) + per_sample = [np.mean([samples[u][i] == greedy[u]["gold"] for u in uids if len(samples[u]) > i]) for i in range(k)] + d = v - g + rng = np.random.default_rng(0) + lo, hi = np.percentile([d[rng.integers(0, len(d), len(d))].mean() for _ in range(5000)], [2.5, 97.5]) + print(f"{args.samples.name}: n={len(uids)} greedy {100 * g.mean():.1f} samples {' / '.join(f'{100 * x:.1f}' for x in per_sample)}" + f" vote(greedy+{k}) {100 * v.mean():.1f} diff {100 * d.mean():+.2f} [{100 * lo:+.2f}, {100 * hi:+.2f}]") + + +if __name__ == "__main__": + main() diff --git a/eval/summarize.py b/eval/summarize.py new file mode 100644 index 0000000000000000000000000000000000000000..abdb7d80bcf4b197943b3c776b2d4c3e410ebe3a --- /dev/null +++ b/eval/summarize.py @@ -0,0 +1,84 @@ +"""Combine results/*/metrics.json into comparison tables (reports/.md). + +Usage: python -m eval.summarize --runs results/base_* --out reports/baselines_zeroshot.md +""" +from __future__ import annotations + +import argparse +import glob +import json +from pathlib import Path + + +def load(run_dirs: list[str]) -> dict[str, dict]: + runs = {} + for d in sorted(run_dirs): + f = Path(d) / "metrics.json" + if f.exists(): + runs[Path(d).name] = json.loads(f.read_text()) + return runs + + +def fmt(x) -> str: + return "-" if x is None else f"{100 * x:.1f}" + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--runs", nargs="+", required=True) + ap.add_argument("--out", required=True) + args = ap.parse_args() + runs = load([d for pat in args.runs for d in glob.glob(pat)]) + names = list(runs) + short = {n: n.removeprefix("base_").removeprefix("exp_") for n in names} + + lines = ["# Baseline comparison", "", + "Accuracy in %. `orig` = original option order; `vote` = mean probabilities over cyclic option " + "permutations; `±` = std of accuracy across permutations (position sensitivity). Random = 25.", ""] + + # Overview + lines += ["## Overview", "", "| model | params loaded | tok/char | SinhalaMMLU dev orig | vote | ± | macro-subj | " + "Global-MMLU si | Belebele si | label mass | consistency |", "|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|"] + for n in names: + r, bd = runs[n]["run"], runs[n]["by_dataset"] + s = bd.get("sinhalammlu", {}).get("overall", {}) + g = bd.get("gmmlu_si", {}).get("overall", {}) + b = bd.get("belebele_si", {}).get("overall", {}) + lines.append(f"| {short[n]} | {r['params']['loaded_total'] / 1e9:.2f}B | {r['tokens_per_char']:.2f} | " + f"{fmt(s.get('acc'))} | {fmt(s.get('acc_vote'))} | {fmt(s.get('acc_perm_std'))} | " + f"{fmt(s.get('acc_macro_subject'))} | {fmt(g.get('acc'))} | {fmt(b.get('acc'))} | " + f"{s.get('label_mass_mean', float('nan')):.2f} | {s.get('consistency', float('nan')):.2f} |") + + def table(dataset: str, key: str, title: str) -> None: + groups = sorted({g for n in names for g in runs[n]["by_dataset"].get(dataset, {}).get("groups", {}).get(key, {})}) + if not groups: + return + lines.extend(["", f"## {title}", "", "| " + key + " | n | " + " | ".join(short[n] for n in names) + " |", + "|---|---:|" + "---:|" * len(names)]) + for g in groups: + cells, cnt = [], "-" + for n in names: + blk = runs[n]["by_dataset"].get(dataset, {}).get("groups", {}).get(key, {}).get(g) + cells.append(fmt(blk["acc"]) if blk else "-") + cnt = blk["n"] if blk else cnt + lines.append(f"| {g} | {cnt} | " + " | ".join(cells) + " |") + + table("sinhalammlu", "subject", "SinhalaMMLU dev: per subject") + table("sinhalammlu", "category", "SinhalaMMLU dev: per domain") + table("sinhalammlu", "grade", "SinhalaMMLU dev: per grade") + table("gmmlu_si", "category", "Global-MMLU si: per category") + table("gmmlu_si", "cultural_sensitivity", "Global-MMLU si: culturally sensitive (CS) / agnostic (CA) / unlabelled (-)") + + lines += ["", "## Position bias on SinhalaMMLU dev (share of predictions per displayed position)", "", + "| model | 1 | 2 | 3 | 4 |", "|---|---:|---:|---:|---:|"] + for n in names: + pb = runs[n]["by_dataset"].get("sinhalammlu", {}).get("position_bias") + if pb: + lines.append(f"| {short[n]} | " + " | ".join(fmt(pb["pred_dist"].get(str(p))) for p in range(1, 5)) + " |") + Path(args.out).parent.mkdir(parents=True, exist_ok=True) + Path(args.out).write_text("\n".join(lines) + "\n") + print("\n".join(lines)) + + +if __name__ == "__main__": + main() diff --git a/inference.py b/inference.py new file mode 100644 index 0000000000000000000000000000000000000000..bc334d32a17a9dc9e15a072a4c02578a71767964 --- /dev/null +++ b/inference.py @@ -0,0 +1,141 @@ +""" +Sinhala MMLU Shared Task - Final inference script +Team: Nexus + +The organizers will run this file exactly as it is, changing ONLY the DATA_REPO line +below to point to the final Sinhala MMLU dataset: + + pip install -r requirements.txt + python inference.py + +System (see README.md): Gemma 4 E4B-it, text-only (7.463B parameters, vision/audio towers removed, weights unchanged), +no fine-tuning. Every question is answered with the model's thinking mode (greedy). Where the fast one-step answer +(next-token scores over the labels 1-4, averaged over 4 cyclic option orders) disagrees with the greedy thinking answer, +3 more reasoning paths are sampled (temperature 0.8, top-p 0.95, top-k 64, fixed seed 1234) and the 4 answers are +majority-voted, ties going to the greedy answer ("agreement-gated self-consistency"; 892 of 2,238 test questions). +Set SELF_CONSISTENCY = False for a fully greedy variant (not the one we submitted). + +Section 2 runs the exact pipeline that produced our Codabench submission (inference/predict.py with +configs/submission_base_think_gated_sc.yaml), on the questions loaded by section 1. +""" + +# ======================= ORGANIZERS CHANGE ONLY THIS LINE ======================= +DATA_REPO = "SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset" +# ================================================================================ + +OUTPUT_FILE = "predictions.jsonl" + +# ----------------------- TEAM SETTINGS (keep exactly as used for your final submission) +MODEL_ID = "SinhalaMMLU-SharedTask/team-nexus-gemma-4-e4b-it-text" # text-only extraction of google/gemma-4-E4B-it +MODEL_REVISION = "8f686039aad1a865c9335c17874d9b870fdec766" # exact commit of MODEL_ID used for our submission +SEED = 1234 +SELF_CONSISTENCY = True # True = our submitted system; False = greedy thinking only (fully deterministic) + +import json +import os +import random +import re + +import numpy as np +import torch +from huggingface_hub import HfFileSystem, list_repo_files + + +def set_seed(seed): + random.seed(seed) + np.random.seed(seed) + torch.manual_seed(seed) + torch.cuda.manual_seed_all(seed) + + +# =============== 1. LOAD THE DATASET FROM HUGGING FACE (do not change) =============== +def load_questions(data_repo): + fs = HfFileSystem() + files = sorted(f for f in list_repo_files(data_repo, repo_type="dataset") if f.endswith(".json")) + rows = [] + for f in files: + with fs.open(f"datasets/{data_repo}/{f}", "r", encoding="utf-8") as fh: + raw = json.load(fh) + if isinstance(raw, dict): + raw = next(v for v in raw.values() if isinstance(v, list)) + for r in raw: + r["_file"] = f + rows.append(r) + return rows + + +def get_level(r): + return (r.get("metadata") or {}).get("difficulty") or r["_file"].split("/")[0] + + +# =============== 2. OUR SYSTEM =============== +# predict_all() writes the loaded questions to a temporary folder (same file layout as the dataset repo), runs +# inference/predict.py on it offline, and maps its answers back to `rows` by (level, subject, q_no). + +HERE = os.path.dirname(os.path.abspath(__file__)) +CONFIG = os.path.join(HERE, "configs", "submission_base_think_gated_sc.yaml" if SELF_CONSISTENCY + else "submission_base_think_greedy.yaml") + + +def load_system(): + from huggingface_hub import snapshot_download + + return snapshot_download(MODEL_ID, revision=MODEL_REVISION) # local folder with weights + tokenizer + + +def predict_all(system, rows): + import subprocess + import sys + import tempfile + import zipfile + + import yaml + + model_dir = system + with tempfile.TemporaryDirectory() as tmp: + by_file = {} + for r in rows: + by_file.setdefault(r["_file"], []).append({k: v for k, v in r.items() if k != "_file"}) + for f, recs in by_file.items(): + path = os.path.join(tmp, "data", f) + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w", encoding="utf-8") as fh: + json.dump(recs, fh, ensure_ascii=False) + with open(CONFIG, encoding="utf-8") as fh: + cfg = yaml.safe_load(fh) + cfg["model"] = model_dir + cfg["seed"] = SEED + cfg_path = os.path.join(tmp, "config.yaml") + with open(cfg_path, "w", encoding="utf-8") as fh: + yaml.safe_dump(cfg, fh, allow_unicode=True) + out_zip = os.path.join(tmp, "predictions.zip") + subprocess.run([sys.executable, "-m", "inference.predict", "--config", cfg_path, + "--input", os.path.join(tmp, "data"), "--output", out_zip], cwd=HERE, check=True) + with zipfile.ZipFile(out_zip) as z: + preds = [json.loads(l) for l in z.read("predictions.jsonl").decode("utf-8").splitlines() if l.strip()] + ans = {(p["level"], p["subject"], p["q_no"]): p["answer"] for p in preds} + return [ans[(get_level(r), r["subject"], r["q_no"])] for r in rows] + + +# =============== 3. WRITE PREDICTIONS (do not change) =============== +if __name__ == "__main__": + set_seed(SEED) + rows = load_questions(DATA_REPO) + print(len(rows), "questions loaded from", DATA_REPO) + + system = load_system() + answers = predict_all(system, rows) + + assert len(answers) == len(rows), "predict_all() must return one answer per question" + with open(OUTPUT_FILE, "w", encoding="utf-8") as f: + for r, a in zip(rows, answers): + a = str(a).strip() + assert a in {"1", "2", "3", "4"}, f"invalid answer {a!r} for q_no={r.get('q_no')}" + f.write(json.dumps({ + "q_no": r["q_no"], + "subject": r["subject"], + "category": r["category"], + "level": get_level(r), + "answer": a, + }, ensure_ascii=False, separators=(",", ":")) + "\n") + print("\nSaved", OUTPUT_FILE) diff --git a/inference/__init__.py b/inference/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/inference/check_predictions.py b/inference/check_predictions.py new file mode 100644 index 0000000000000000000000000000000000000000..6cd05bcc19696399af8867b55c7f9d7c3a1aa86b --- /dev/null +++ b/inference/check_predictions.py @@ -0,0 +1,73 @@ +"""Score a Codabench-format predictions.zip against gold answers (dev only; test gold is hidden). + + python -m inference.check_predictions --predictions dev_predictions.zip --gold data/raw/sinhalammlu/TEST/easy + +Also validates the file the way the leaderboard will need it: one row per question, required fields present, +answers are digits within the option range. Reports accuracy per level and their macro average (task metric). +""" +from __future__ import annotations + +import argparse +import json +import sys +import zipfile +from collections import defaultdict +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from inference.predict import read_records, to_item # noqa: E402 + +FIELDS = ("q_no", "subject", "category", "level", "answer") + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--predictions", required=True, type=Path) + ap.add_argument("--gold", required=True, type=Path) + args = ap.parse_args() + + with zipfile.ZipFile(args.predictions) as z: + assert z.namelist() == ["predictions.jsonl"], f"zip must contain only predictions.jsonl, has {z.namelist()}" + preds = [json.loads(l) for l in z.read("predictions.jsonl").decode("utf-8").splitlines() if l.strip()] + + gold = {} + for f, r in read_records(args.gold): + it = to_item(f, r) + gold[(it["level"], it["subject_raw"], it["q_no"])] = (r.get("answer"), len(it["options"])) + + errors, seen = [], set() + correct = defaultdict(lambda: [0, 0]) + for p in preds: + if any(k not in p for k in FIELDS): + errors.append(f"missing fields in {p}") + continue + key = (p["level"], p["subject"], p["q_no"]) + if key in seen: + errors.append(f"duplicate {key}") + seen.add(key) + if key not in gold: + errors.append(f"unknown question {key}") + continue + ans, n = gold[key] + if not (isinstance(p["answer"], str) and p["answer"].isdigit() and 1 <= int(p["answer"]) <= n): + errors.append(f"bad answer {p['answer']!r} for {key}") + continue + if isinstance(ans, int) and 1 <= ans <= n: # skip items with an invalid gold label + correct[p["level"]][0] += int(p["answer"]) == ans + correct[p["level"]][1] += 1 + missing = set(gold) - seen + if missing: + errors.append(f"{len(missing)} questions without a prediction, e.g. {sorted(missing)[:3]}") + + for lvl, (k, n) in sorted(correct.items()): + print(f"{lvl}: {k}/{n} = {100 * k / n:.2f}%") + if correct: + print(f"macro average over levels: {100 * sum(k / n for k, n in correct.values()) / len(correct):.2f}%") + print(f"{len(preds)} rows, {len(errors)} format errors") + for e in errors[:20]: + print(" ", e) + sys.exit(1 if errors else 0) + + +if __name__ == "__main__": + main() diff --git a/inference/predict.py b/inference/predict.py new file mode 100644 index 0000000000000000000000000000000000000000..9f0845f9f158caf83c84ab4e0f770c0a5e2f6c93 --- /dev/null +++ b/inference/predict.py @@ -0,0 +1,238 @@ +"""Offline submission pipeline: question file(s) in, Codabench predictions.zip out. No human in the loop. + + python -m inference.predict --input --output predictions.zip + +Runs with HF_HUB_OFFLINE=1 (no network); every setting comes from configs/submission.yaml. +Optional self-consistency (config `sc_levels`, `sc_samples`, `sc_gate`): greedy + K sampled reasoning paths, majority +vote; with `sc_gate: disagree` only for questions where the fast answer disagrees with the greedy thinking answer. +Every thinking answer (greedy first, then samples) is also written to .ballots.jsonl (scripts/pool_ballots.py). +Optional LoRA adapter (config `adapter`) on top of `model`. Optional per-question thinking (config `thinking_levels`, +e.g. [hard]): questions whose own `level` field is listed are answered with the model's thinking mode (with +`thinking_without_adapter: true`, the adapter is switched off for them) (greedy, original option order); if no answer can be read from the output, +the fast answer is used. All other questions use the fast logit answer (voted over `n_perms` option orders). + +Accepts the SinhalaMMLU JSON layout (list of {q_no, subject, category, question, choices, metadata.difficulty}) +and JSONL; q_no / subject / category / level are echoed exactly as given. Writes .run.json next to the +zip with the model commit, loaded parameter count, input hash and library versions. +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import sys +import time +import zipfile +from collections import Counter +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +LEVELS = ("easy", "medium", "hard") + + +def read_records(path: Path) -> list[tuple[Path, dict]]: + files = sorted(p for p in path.rglob("*") if p.suffix in (".json", ".jsonl") + and not any(part.startswith(".") for part in p.relative_to(path).parts)) if path.is_dir() else [path] + # (hidden dirs skipped: huggingface_hub keeps download metadata as JSON under .cache/) + out = [] + for f in files: + text = f.read_text(encoding="utf-8") + rows = [json.loads(l) for l in text.splitlines() if l.strip()] if f.suffix == ".jsonl" else json.loads(text) + out += [(f, r) for r in (rows if isinstance(rows, list) else [rows])] + if not out: + raise SystemExit(f"no questions found under {path}") + return out + + +def to_item(f: Path, r: dict) -> dict: + options = r.get("choices", r.get("options")) + level = r.get("level") or (r.get("metadata") or {}).get("difficulty") \ + or next((p for p in f.parts[::-1] if p.lower() in LEVELS), None) + missing = [k for k, v in {"q_no": r.get("q_no"), "subject": r.get("subject"), "category": r.get("category"), + "question": r.get("question"), "choices": options, "level": level}.items() if v is None] + if missing: + raise SystemExit(f"{f}: record {r.get('q_no')} lacks {missing}; keys present: {sorted(r)}") + return {"uid": f"{level}/{r['subject']}/{r['q_no']}", "dataset": "test", "q_no": r["q_no"], + "subject": r["subject"].strip(), "subject_raw": r["subject"], "category": r["category"], "level": level, + "question": r["question"].strip(), "options": [str(o).strip() for o in options]} + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--input", required=True, type=Path) + ap.add_argument("--output", required=True, type=Path) + ap.add_argument("--config", type=Path, default=Path(__file__).resolve().parents[1] / "configs" / "submission.yaml") + ap.add_argument("--allow-online", action="store_true", help="debugging only; the submission runs offline") + ap.add_argument("--limit", type=int, help="first N questions only (smoke tests; never for a submission)") + ap.add_argument("--levels", nargs="+", help="answer only questions of these levels (then merge with scripts/merge_predictions.py)") + args = ap.parse_args() + if not args.allow_online: + os.environ["HF_HUB_OFFLINE"] = os.environ["TRANSFORMERS_OFFLINE"] = "1" + + import yaml + + from eval.common import cap_gpu_memory, env_fingerprint, set_seed + from eval.metrics import aggregate_items + from eval.prompts import PromptBuilder + from eval.run_eval import make_perm, score_items + from eval.scorers import Scorer, load_model, loaded_param_stats + + cfg = yaml.safe_load(args.config.read_text()) + set_seed(cfg["seed"]) + cap_gpu_memory(cfg.get("gpu_mem_gb")) + + raw = read_records(args.input) + items = {} + for f, r in raw: + it = to_item(f, r) + if it["uid"] in items: + raise SystemExit(f"duplicate question id {it['uid']}") + items[it["uid"]] = it + if args.levels: + items = {u: it for u, it in items.items() if it["level"] in set(args.levels)} + if args.limit: + items = dict(list(items.items())[: args.limit]) + print(f"{len(items)} questions from {args.input}") + + t0 = time.time() + root = Path(__file__).resolve().parents[1] + model_path = str(root / cfg["model"]) if (root / cfg["model"]).exists() else cfg["model"] + adapter = cfg.get("adapter") + adapter = str(root / adapter) if adapter and (root / adapter).exists() else adapter + model, tok = load_model(model_path, cfg.get("revision"), cfg.get("dtype", "bfloat16"), adapter) + pb = PromptBuilder(tok, lang=cfg["lang"], chat=cfg["chat"]) + scorer = Scorer(model, tok, pb.add_special_tokens, cfg.get("token_budget", 32768), cfg.get("max_batch", 64)) + if cfg.get("fast_without_adapter") and hasattr(model, "disable_adapter"): + with model.disable_adapter(): # fast one-step answer from the plain base model (used by the agreement gate) + records, _ = score_items(items, pb, scorer, cfg["method"], cfg["n_perms"], cfg["perm_mode"], cfg["seed"]) + else: + records, _ = score_items(items, pb, scorer, cfg["method"], cfg["n_perms"], cfg["perm_mode"], cfg["seed"]) + agg = aggregate_items(items, records) + + # Thinking mode for selected levels (decided per question from its own level field) + think_levels = set(cfg.get("thinking_levels") or []) + think_uids = [u for u, it in items.items() if it["level"] in think_levels] + think_pred, fallbacks, sc_changed, sc_count = {}, 0, 0, 0 + perm_levels, k_perm = set(cfg.get("perm_think_levels") or []), int(cfg.get("perm_think_orders", 0)) + perm_uids = [u for u, it in items.items() if it["level"] in perm_levels] if k_perm > 0 else [] + perm_changed = 0 + if think_uids or perm_uids: + tpb = PromptBuilder(tok, lang=cfg["lang"], chat=cfg["chat"], chat_kwargs={"enable_thinking": True}) + tscorer = Scorer(model, tok, tpb.add_special_tokens, cfg.get("token_budget", 32768), cfg.get("max_batch", 64), + max_new_tokens=cfg.get("think_max_new_tokens", 2048)) + + def think_prompts(prompts: list[str], n_opts: list[int]) -> list[dict]: + if cfg.get("thinking_without_adapter") and hasattr(model, "disable_adapter"): + with model.disable_adapter(): # base model reasons; the adapter (trained to answer at once) does not + return tscorer.generate(prompts, n_opts) + return tscorer.generate(prompts, n_opts) + + def think(uids: list[str]) -> list[dict]: + return think_prompts([tpb.mcq(items[u], items[u]["options"]) for u in uids], + [len(items[u]["options"]) for u in uids]) + + for u, r in (zip(think_uids, think(think_uids)) if think_uids else []): + if r["pred"]: + think_pred[u] = r["pred"] + else: + fallbacks += 1 + print(f"thinking: {len(think_uids)} questions ({sorted(think_levels)}), {fallbacks} fell back to the fast answer") + all_ballots = {u: [think_pred.get(u)] for u in think_uids} # saved to .ballots.jsonl for cross-model pooling + + # Self-consistency for selected levels: greedy answer + K sampled reasoning paths, majority vote, + # ties go to the greedy answer. Fixed seed: reproducible for the same input, code and hardware. + sc_levels, k = set(cfg.get("sc_levels") or []), int(cfg.get("sc_samples", 0)) + sc_uids = [u for u in think_uids if items[u]["level"] in sc_levels] + if cfg.get("sc_gate") == "disagree": # only where the free one-step answer disagrees with greedy thinking + sc_uids = [u for u in sc_uids if think_pred.get(u) != agg[u]["pred"]] + sc_count = len(sc_uids) if k > 0 else 0 + if sc_uids and k > 0: + import torch + + torch.manual_seed(cfg["seed"]) + tscorer.sampling = {"do_sample": True, "temperature": cfg.get("sc_temperature", 0.8), + "top_p": cfg.get("sc_top_p", 0.95), "top_k": cfg.get("sc_top_k", 64)} + ballots = {u: [think_pred.get(u)] for u in sc_uids} + for _ in range(k): + for u, r in zip(sc_uids, think(sc_uids)): + ballots[u].append(r["pred"]) + all_ballots[u].append(r["pred"]) + tscorer.sampling = {} + for u, b in ballots.items(): + counts = Counter(x for x in b if x) + if not counts: + continue + top = max(counts.values()) + winners = [a for a, c in counts.items() if c == top] + choice = b[0] if b[0] in winners else min(winners) + sc_changed += choice != think_pred.get(u) + think_pred[u] = choice + print(f"self-consistency: {len(sc_uids)} questions ({sorted(sc_levels)}, gate={cfg.get('sc_gate')}), greedy + {k} samples, " + f"{sc_changed} answers changed by the vote") + + # Option-order self-consistency: re-ask with the options in K other (cyclic) orders, thinking each time; the vote is over + # the previous answer (the `prior_predictions` file if given, else the answer so far) and the K re-orderings, ties + # go to the previous answer. Votes are mapped back to the original option numbering. + if perm_uids: + prior = {} + if cfg.get("prior_predictions"): + with zipfile.ZipFile(root / cfg["prior_predictions"]) as z: + for line in z.read("predictions.jsonl").decode("utf-8").splitlines(): + r = json.loads(line) + prior[(r["level"], r["subject"], r["q_no"])] = int(r["answer"]) + ballots = {u: [prior.get((items[u]["level"], items[u]["subject_raw"], items[u]["q_no"]), think_pred.get(u))] + for u in perm_uids} + for kk in range(1, k_perm + 1): + perms = {u: make_perm(len(items[u]["options"]), kk, "cyclic", cfg["seed"], u) for u in perm_uids} + res = think_prompts([tpb.mcq(items[u], [items[u]["options"][j] for j in perms[u]]) for u in perm_uids], + [len(items[u]["options"]) for u in perm_uids]) + for u, r in zip(perm_uids, res): + ballots[u].append(perms[u][r["pred"] - 1] + 1 if r["pred"] else None) + print(f"option-order thinking: order {kk}/{k_perm} done ({len(perm_uids)} questions)", flush=True) + for u, b in ballots.items(): + counts = Counter(x for x in b if x) + if not counts: + continue + top = max(counts.values()) + winners = [a for a, c in counts.items() if c == top] + choice = b[0] if b[0] in winners else min(winners) + perm_changed += choice != b[0] + think_pred[u] = choice + print(f"option-order self-consistency: {len(perm_uids)} questions ({sorted(perm_levels)}), {k_perm} extra orders, " + f"{perm_changed} answers changed by the vote") + + rows = [] + for uid, it in items.items(): + n = len(it["options"]) + ans = think_pred.get(uid, agg[uid]["vote"]) + if ans is None or not 1 <= ans <= n: + raise SystemExit(f"no valid prediction for {uid}") + rows.append({"q_no": it["q_no"], "subject": it["subject_raw"], "category": it["category"], + "level": it["level"], "answer": str(ans)}) + + args.output.parent.mkdir(parents=True, exist_ok=True) + with zipfile.ZipFile(args.output, "w", zipfile.ZIP_DEFLATED) as z: + z.writestr("predictions.jsonl", "".join(json.dumps(r, ensure_ascii=False) + "\n" for r in rows)) + if think_uids: # greedy + sampled thinking answers per question (None = unparsed), first = greedy + Path(str(args.output) + ".ballots.jsonl").write_text("".join( + json.dumps({"level": items[u]["level"], "subject": items[u]["subject_raw"], "q_no": items[u]["q_no"], + "ballots": all_ballots[u]}, ensure_ascii=False) + "\n" for u in think_uids), encoding="utf-8") + digest = hashlib.sha256() + for f in sorted({str(f) for f, _ in raw}): + digest.update(Path(f).read_bytes()) + run = {"config": cfg, "n_questions": len(rows), "seconds": round(time.time() - t0, 1), + "thinking": {"levels": sorted(think_levels), "questions": len(think_uids), "fallbacks": fallbacks}, + "self_consistency": {"levels": sorted(set(cfg.get("sc_levels") or [])), "samples": int(cfg.get("sc_samples", 0)), + "gate": cfg.get("sc_gate"), "questions": sc_count, "answers_changed": sc_changed}, + "option_order_sc": {"levels": sorted(perm_levels), "orders": k_perm, "questions": len(perm_uids), + "answers_changed": perm_changed}, + "input_sha256": digest.hexdigest(), "params": loaded_param_stats(model), "env": env_fingerprint()} + Path(str(args.output) + ".run.json").write_text(json.dumps(run, indent=2, ensure_ascii=False)) + print(f"wrote {len(rows)} predictions -> {args.output} " + f"({run['params']['loaded_total'] / 1e9:.3f}B parameters loaded, {run['seconds']:.0f}s)") + + +if __name__ == "__main__": + main() diff --git a/predictions.jsonl b/predictions.jsonl new file mode 100644 index 0000000000000000000000000000000000000000..1b13802175a8a3227ae0edad2f75500b79b63094 --- /dev/null +++ b/predictions.jsonl @@ -0,0 +1,2238 @@ +{"q_no": 1, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 2, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 3, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 4, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 5, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 6, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 7, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 8, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 9, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 10, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 11, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 12, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 13, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 14, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 15, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 16, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 17, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 18, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 19, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 20, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 21, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 22, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 23, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 24, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 25, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 26, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 27, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 28, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 29, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 30, "subject": "Art", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 1, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 2, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 3, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 4, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 5, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 6, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 7, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 8, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 9, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 10, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 11, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 12, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 13, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 14, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 15, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 16, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 17, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 18, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 19, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 20, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 21, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 22, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 23, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 24, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 25, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 26, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 27, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 28, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 29, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 30, "subject": "Buddhism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 1, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 2, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 3, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 4, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 5, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 6, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 7, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 8, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 9, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 10, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 11, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 12, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 13, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 14, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 15, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 16, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 17, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 18, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 19, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 20, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 21, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 22, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 23, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 24, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 25, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 26, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 27, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 28, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 29, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 30, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 31, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 32, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 33, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 34, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 35, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 36, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 37, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 38, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 39, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 40, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 41, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 42, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 43, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 44, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 45, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 46, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 47, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 48, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 49, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 50, "subject": "Catholicism", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 1, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 2, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 3, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 4, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 5, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 6, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 7, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 8, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 9, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 10, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 11, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 12, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 13, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 14, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 15, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 16, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 17, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 18, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 19, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 20, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 21, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 22, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 23, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 24, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 25, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 26, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 27, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 28, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 29, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 30, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 31, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 32, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 33, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 34, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 35, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 36, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 37, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 38, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 39, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 40, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 41, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 42, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 43, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 44, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 45, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 46, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 47, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 48, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 49, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 50, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 51, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 52, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 53, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 54, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 55, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 56, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 57, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 58, "subject": "Christianity", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 1, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "4"} +{"q_no": 2, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 3, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 4, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 5, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 6, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 7, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 8, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "4"} +{"q_no": 9, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 10, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "4"} +{"q_no": 11, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 12, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "4"} +{"q_no": 13, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 14, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 15, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 16, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 17, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "4"} +{"q_no": 18, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "4"} +{"q_no": 19, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "4"} +{"q_no": 20, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 21, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 22, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 23, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 24, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 25, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 26, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "4"} +{"q_no": 27, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 28, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 29, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 30, "subject": "Civics", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 1, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 2, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 3, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 4, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 5, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 6, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 7, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 8, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 9, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 10, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 11, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 12, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 13, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 14, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 15, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 16, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 17, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 18, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 19, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 20, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 21, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 22, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 23, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 24, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 25, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 26, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 27, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 28, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 29, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 30, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 31, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 32, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 33, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 34, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 35, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 36, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 37, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 38, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 39, "subject": "Dancing", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 1, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 2, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 3, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 4, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 5, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 6, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 7, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 8, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 9, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 10, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 11, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 12, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 13, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 14, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 15, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 16, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 17, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 18, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 19, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 20, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 21, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 22, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 23, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 24, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 25, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 26, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 27, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 28, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 29, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 30, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 31, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 32, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 33, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 34, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 35, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 36, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 37, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 38, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 39, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 40, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 41, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 42, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 43, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 44, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 45, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 46, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 47, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 48, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 49, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 50, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 51, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 52, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 53, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 54, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 55, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 56, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 57, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 58, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 59, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 60, "subject": "Drama and Theatre", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 1, "subject": "Eastern_Music", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 2, "subject": "Eastern_Music", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 3, "subject": "Eastern_Music", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 4, "subject": "Eastern_Music", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 5, "subject": "Eastern_Music", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 6, "subject": "Eastern_Music", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 7, "subject": "Eastern_Music", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 8, "subject": "Eastern_Music", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 9, "subject": "Eastern_Music", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 10, "subject": "Eastern_Music", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 11, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 12, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 13, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 14, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 15, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 16, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 17, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 18, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 19, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 20, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 21, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 22, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 23, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 24, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 25, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 26, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 27, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 28, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 29, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 30, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 31, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 32, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 33, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 34, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 35, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 36, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 37, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 38, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 39, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 40, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 41, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 42, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 43, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 44, "subject": "Eastern Music", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 1, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 2, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 3, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 4, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 5, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "4"} +{"q_no": 6, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 7, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 8, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "4"} +{"q_no": 9, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 10, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 11, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 12, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 13, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 14, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 15, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 16, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 17, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 18, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 19, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 20, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 21, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 22, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 23, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 24, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 25, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 26, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 27, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 28, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 29, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 30, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 31, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 32, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 33, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 34, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 35, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 36, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 37, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 38, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "4"} +{"q_no": 39, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 40, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "4"} +{"q_no": 41, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 42, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 43, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 44, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 45, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 46, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "3"} +{"q_no": 47, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "1"} +{"q_no": 48, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "2"} +{"q_no": 49, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "4"} +{"q_no": 50, "subject": "Geography", "category": "Social_science", "level": "easy", "answer": "4"} +{"q_no": 1, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 2, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 3, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 4, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 5, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 6, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 7, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 8, "subject": "Health & Physical Education", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 9, "subject": "Health & Physical Education", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 10, "subject": "Health & Physical Education", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 11, "subject": "Health & Physical Education", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 12, "subject": "Health & Physical Education", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 13, "subject": "Health & Physical Education", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 14, "subject": "Health & Physical Education", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 15, "subject": "Health & Physical Education", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 16, "subject": "Health & Physical Education", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 17, "subject": "Health & Physical Education", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 18, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 19, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 20, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 21, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 22, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 23, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 24, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 25, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 26, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 27, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 28, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 29, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 30, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 31, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 32, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 33, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 34, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 35, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 36, "subject": "Health and Physical Education", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 1, "subject": "History", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 2, "subject": "History", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 3, "subject": "History", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 4, "subject": "History", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 5, "subject": "History", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 6, "subject": "History", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 7, "subject": "History", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 8, "subject": "History", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 9, "subject": "History", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 10, "subject": "History", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 11, "subject": "History", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 12, "subject": "History", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 13, "subject": "History", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 14, "subject": "History", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 15, "subject": "History", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 16, "subject": "History", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 17, "subject": "History", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 18, "subject": "History", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 19, "subject": "History", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 20, "subject": "History", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 21, "subject": "History", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 22, "subject": "History", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 23, "subject": "History", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 24, "subject": "History", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 25, "subject": "History", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 26, "subject": "History", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 27, "subject": "History", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 28, "subject": "History", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 29, "subject": "History", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 30, "subject": "History", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 31, "subject": "History", "category": "Humanities", "level": "easy", "answer": "1"} +{"q_no": 32, "subject": "History", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 33, "subject": "History", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 34, "subject": "History", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 35, "subject": "History", "category": "Humanities", "level": "easy", "answer": "2"} +{"q_no": 36, "subject": "History", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 37, "subject": "History", "category": "Humanities", "level": "easy", "answer": "4"} +{"q_no": 38, "subject": "History", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 39, "subject": "History", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 40, "subject": "History", "category": "Humanities", "level": "easy", "answer": "3"} +{"q_no": 1, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 2, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 3, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 4, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 5, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 6, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 7, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 8, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 9, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 10, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 11, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 12, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 13, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 14, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 15, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 16, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 17, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 18, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 19, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 20, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 21, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 22, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 23, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 24, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 25, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 26, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 27, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 28, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 29, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 30, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 31, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 32, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 33, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 34, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 35, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 36, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 37, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 38, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 39, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 40, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 41, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 42, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 43, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 44, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 45, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 46, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 47, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 48, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 49, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 50, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 51, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 52, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 53, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 54, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 55, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 56, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 57, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 58, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 59, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 60, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 61, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 62, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 63, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 64, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 65, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 66, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 67, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 68, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 69, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 70, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 71, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 72, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 73, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 74, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 75, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 76, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 77, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 78, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 79, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 80, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 81, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 82, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 83, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 84, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 85, "subject": "Science", "category": "stem", "level": "easy", "answer": "4"} +{"q_no": 86, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 87, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 88, "subject": "Science", "category": "stem", "level": "easy", "answer": "1"} +{"q_no": 89, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 90, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 91, "subject": "Science", "category": "stem", "level": "easy", "answer": "2"} +{"q_no": 92, "subject": "Science", "category": "stem", "level": "easy", "answer": "3"} +{"q_no": 1, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 2, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 3, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 4, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 5, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 6, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 7, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 8, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 9, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 10, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 11, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 12, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 13, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 14, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 15, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 16, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 17, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 18, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 19, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 20, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 21, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 22, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 23, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 24, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 25, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 26, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 27, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 28, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 29, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 30, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 31, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 32, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 33, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 34, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 35, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 36, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 37, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 38, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 39, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 40, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 41, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 42, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 43, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 44, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 45, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 46, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 47, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 48, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 49, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 50, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 51, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 52, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 53, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 54, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 55, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 56, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 57, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 58, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 59, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 60, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 61, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 62, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 63, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 64, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 65, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 66, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 67, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 68, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 69, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 70, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 71, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 72, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 73, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 74, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 75, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "2"} +{"q_no": 76, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 77, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 78, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 79, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 80, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 81, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "1"} +{"q_no": 82, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "3"} +{"q_no": 83, "subject": "Sinhala Language and Literature", "category": "other", "level": "easy", "answer": "4"} +{"q_no": 1, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 2, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 3, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 4, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 5, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 6, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 7, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 8, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 9, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 10, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 11, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 12, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 13, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 14, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 15, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 16, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 17, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 18, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 19, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 20, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 21, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 22, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 23, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 24, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 25, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 26, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 27, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 28, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 29, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 30, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 31, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 32, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 33, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 34, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 35, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 36, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 37, "subject": "Agricultural Science", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 1, "subject": "Art", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 2, "subject": "Art", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 3, "subject": "Art", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 4, "subject": "Art", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 5, "subject": "Art", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 6, "subject": "Art", "category": "humanities", "level": "hard", "answer": "1"} +{"q_no": 7, "subject": "Art", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 8, "subject": "Art", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 9, "subject": "Art", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 10, "subject": "Art", "category": "humanities", "level": "hard", "answer": "1"} +{"q_no": 11, "subject": "Art", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 12, "subject": "Art", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 13, "subject": "Art", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 14, "subject": "Art", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 15, "subject": "Art", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 1, "subject": "Biology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 2, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 3, "subject": "Biology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 4, "subject": "Biology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 5, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 6, "subject": "Biology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 7, "subject": "Biology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 8, "subject": "Biology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 9, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 10, "subject": "Biology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 11, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 12, "subject": "Biology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 13, "subject": "Biology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 14, "subject": "Biology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 15, "subject": "Biology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 16, "subject": "Biology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 17, "subject": "Biology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 18, "subject": "Biology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 19, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 20, "subject": "Biology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 21, "subject": "Biology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 22, "subject": "Biology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 23, "subject": "Biology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 24, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 25, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 26, "subject": "Biology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 27, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 28, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 29, "subject": "Biology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 30, "subject": "Biology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 31, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 32, "subject": "Biology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 33, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 34, "subject": "Biology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 35, "subject": "Biology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 36, "subject": "Biology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 37, "subject": "Biology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 38, "subject": "Biology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 39, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 40, "subject": "Biology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 41, "subject": "Biology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 42, "subject": "Biology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 43, "subject": "Biology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 44, "subject": "Biology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 45, "subject": "Biology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 46, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 47, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 48, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 49, "subject": "Biology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 50, "subject": "Biology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 1, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 2, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 3, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 4, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 5, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 6, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 7, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 8, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 9, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 10, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 11, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 12, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 13, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 14, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 15, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 16, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 17, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 18, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 19, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 20, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 21, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 22, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 23, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 24, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 25, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 26, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 27, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 28, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 29, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 30, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 31, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 32, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 33, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 34, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 35, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 36, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 37, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 38, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 39, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 40, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 41, "subject": "Biosystem Technology", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 1, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 2, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "1"} +{"q_no": 3, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 4, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 5, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 6, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 7, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 8, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 9, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 10, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "1"} +{"q_no": 11, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 12, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 13, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 14, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 15, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 16, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "1"} +{"q_no": 17, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 18, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 19, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 20, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 21, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 22, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "1"} +{"q_no": 23, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 24, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 25, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "1"} +{"q_no": 26, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 27, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 28, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "1"} +{"q_no": 29, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "1"} +{"q_no": 30, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 31, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 32, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 33, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 34, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 35, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 36, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "1"} +{"q_no": 37, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 38, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "1"} +{"q_no": 39, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "1"} +{"q_no": 40, "subject": "Buddhist Civilization", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 1, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 2, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 3, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 4, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 5, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 6, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 7, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 8, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 9, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 10, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 11, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 12, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 13, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 14, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 15, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 16, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 17, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 18, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 19, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 20, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 21, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 22, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 23, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 24, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 25, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 26, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 27, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 28, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 29, "subject": "Business Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 1, "subject": "Chemistry", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 2, "subject": "Chemistry", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 3, "subject": "Chemistry", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 4, "subject": "Chemistry", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 5, "subject": "Chemistry", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 6, "subject": "Chemistry", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 7, "subject": "Chemistry", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 8, "subject": "Chemistry", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 9, "subject": "Chemistry", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 10, "subject": "Chemistry", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 11, "subject": "Chemistry", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 12, "subject": "Chemistry", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 13, "subject": "Chemistry", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 14, "subject": "Chemistry", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 1, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 2, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 3, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 4, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 5, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 6, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 7, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 8, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 9, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 10, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 11, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 12, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 13, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 14, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 15, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 16, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 17, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 18, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 19, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 20, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 21, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 22, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 23, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 24, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 25, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 26, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 27, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 28, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 29, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 30, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 31, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 32, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 33, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 34, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 35, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 36, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 37, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 38, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 39, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 40, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 41, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 42, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 43, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 44, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 45, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 46, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 47, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 48, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 49, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 50, "subject": "Communication and Media Studies", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 1, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 2, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 3, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 4, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 5, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 6, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 7, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 8, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 9, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 10, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 11, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 12, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "1"} +{"q_no": 13, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 14, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "1"} +{"q_no": 15, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "4"} +{"q_no": 16, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 17, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 18, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 19, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 20, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 21, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "3"} +{"q_no": 22, "subject": "Dancing (Indigenous)", "category": "humanities", "level": "hard", "answer": "2"} +{"q_no": 1, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 2, "subject": "Economics", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 3, "subject": "Economics", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 4, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 5, "subject": "Economics", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 6, "subject": "Economics", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 7, "subject": "Economics", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 8, "subject": "Economics", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 9, "subject": "Economics", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 10, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 11, "subject": "Economics", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 12, "subject": "Economics", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 13, "subject": "Economics", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 14, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 15, "subject": "Economics", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 16, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 17, "subject": "Economics", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 18, "subject": "Economics", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 19, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 20, "subject": "Economics", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 21, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 22, "subject": "Economics", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 23, "subject": "Economics", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 24, "subject": "Economics", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 25, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 26, "subject": "Economics", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 27, "subject": "Economics", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 28, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 29, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 30, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 31, "subject": "Economics", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 32, "subject": "Economics", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 33, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 34, "subject": "Economics", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 35, "subject": "Economics", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 36, "subject": "Economics", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 37, "subject": "Economics", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 38, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 39, "subject": "Economics", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 40, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 41, "subject": "Economics", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 42, "subject": "Economics", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 43, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 44, "subject": "Economics", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 45, "subject": "Economics", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 46, "subject": "Economics", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 1, "subject": "Geography", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 2, "subject": "Geography", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 3, "subject": "Geography", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 4, "subject": "Geography", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 5, "subject": "Geography", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 6, "subject": "Geography", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 7, "subject": "Geography", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 8, "subject": "Geography", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 9, "subject": "Geography", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 10, "subject": "Geography", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 11, "subject": "Geography", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 12, "subject": "Geography", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 13, "subject": "Geography", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 14, "subject": "Geography", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 15, "subject": "Geography", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 16, "subject": "Geography", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 17, "subject": "Geography", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 18, "subject": "Geography", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 19, "subject": "Geography", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 20, "subject": "Geography", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 21, "subject": "Geography", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 22, "subject": "Geography", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 23, "subject": "Geography", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 24, "subject": "Geography", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 25, "subject": "Geography", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 26, "subject": "Geography", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 27, "subject": "Geography", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 28, "subject": "Geography", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 29, "subject": "Geography", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 30, "subject": "Geography", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 31, "subject": "Geography", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 32, "subject": "Geography", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 33, "subject": "Geography", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 34, "subject": "Geography", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 35, "subject": "Geography", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 36, "subject": "Geography", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 37, "subject": "Geography", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 38, "subject": "Geography", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 39, "subject": "Geography", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 40, "subject": "Geography", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 1, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "4"} +{"q_no": 2, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "1"} +{"q_no": 3, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "4"} +{"q_no": 4, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "1"} +{"q_no": 5, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "4"} +{"q_no": 6, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "3"} +{"q_no": 7, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "2"} +{"q_no": 8, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "4"} +{"q_no": 9, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "2"} +{"q_no": 10, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "4"} +{"q_no": 11, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "1"} +{"q_no": 12, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "2"} +{"q_no": 13, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "1"} +{"q_no": 14, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "3"} +{"q_no": 15, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "2"} +{"q_no": 16, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "1"} +{"q_no": 17, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "4"} +{"q_no": 18, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "4"} +{"q_no": 19, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "3"} +{"q_no": 20, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "3"} +{"q_no": 21, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "2"} +{"q_no": 22, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "3"} +{"q_no": 23, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "3"} +{"q_no": 24, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "1"} +{"q_no": 25, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "1"} +{"q_no": 26, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "2"} +{"q_no": 27, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "3"} +{"q_no": 28, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "1"} +{"q_no": 29, "subject": "Home Economics", "category": "other", "level": "hard", "answer": "2"} +{"q_no": 1, "subject": "Physics", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 2, "subject": "Physics", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 3, "subject": "Physics", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 4, "subject": "Physics", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 5, "subject": "Physics", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 6, "subject": "Physics", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 7, "subject": "Physics", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 8, "subject": "Physics", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 9, "subject": "Physics", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 10, "subject": "Physics", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 11, "subject": "Physics", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 12, "subject": "Physics", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 13, "subject": "Physics", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 14, "subject": "Physics", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 15, "subject": "Physics", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 16, "subject": "Physics", "category": "stem", "level": "hard", "answer": "1"} +{"q_no": 17, "subject": "Physics", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 18, "subject": "Physics", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 19, "subject": "Physics", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 20, "subject": "Physics", "category": "stem", "level": "hard", "answer": "3"} +{"q_no": 21, "subject": "Physics", "category": "stem", "level": "hard", "answer": "2"} +{"q_no": 22, "subject": "Physics", "category": "stem", "level": "hard", "answer": "4"} +{"q_no": 1, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 2, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 3, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 4, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 5, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 6, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 7, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 8, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 9, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 10, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 11, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 12, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 13, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 14, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 15, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 16, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 17, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 18, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 19, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 20, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 21, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 22, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 23, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 24, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 25, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 26, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "2"} +{"q_no": 27, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "1"} +{"q_no": 28, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "4"} +{"q_no": 29, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 30, "subject": "Political Science", "category": "social science", "level": "hard", "answer": "3"} +{"q_no": 1, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "2"} +{"q_no": 2, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "2"} +{"q_no": 3, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "2"} +{"q_no": 4, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "2"} +{"q_no": 5, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "4"} +{"q_no": 6, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "3"} +{"q_no": 7, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "3"} +{"q_no": 8, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "3"} +{"q_no": 9, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "2"} +{"q_no": 10, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "1"} +{"q_no": 11, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "3"} +{"q_no": 12, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "3"} +{"q_no": 13, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "3"} +{"q_no": 14, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "4"} +{"q_no": 15, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "3"} +{"q_no": 16, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "1"} +{"q_no": 17, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "3"} +{"q_no": 18, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "2"} +{"q_no": 19, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "1"} +{"q_no": 20, "subject": "Sinhala", "category": "language", "level": "hard", "answer": "3"} +{"q_no": 1, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 2, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 3, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 4, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 5, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 6, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 7, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 8, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 9, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 10, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 11, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 12, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 13, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 14, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 15, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 16, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 17, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 18, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 19, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 20, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 21, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 22, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 23, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 24, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 25, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 26, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 27, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 28, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 29, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 30, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 31, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 32, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 33, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 34, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 35, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 36, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 37, "subject": "Agriculture and Food Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 1, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 2, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 3, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 4, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 5, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 6, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 7, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 8, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 9, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 10, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 11, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 12, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 13, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 14, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 15, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 16, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 17, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 18, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 19, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 20, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 21, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 22, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 23, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 24, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 25, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 26, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 27, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 28, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 29, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 30, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 31, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 32, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 33, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 34, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 35, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 36, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 37, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 38, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 39, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 40, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 41, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 42, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 43, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 44, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 45, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 46, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 47, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 48, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 49, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 50, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 51, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 52, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 53, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 54, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 55, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 56, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 57, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 58, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 59, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 60, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 61, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 62, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 63, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 64, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 65, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 66, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 67, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 68, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 69, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 70, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 71, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 72, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 73, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 74, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 75, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 76, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 77, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 78, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 79, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 80, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 81, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 82, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 83, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 84, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 85, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 86, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 87, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 88, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 89, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 90, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 91, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 92, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 93, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 94, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 95, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 96, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 97, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 98, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 99, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 100, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 101, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 102, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 103, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 104, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 105, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 106, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 107, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 108, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 109, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 110, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 111, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 112, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 113, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 114, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 115, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 116, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 117, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 118, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 119, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 120, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 121, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 122, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 123, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 124, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 125, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 126, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 127, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 128, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 129, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 130, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 131, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 132, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 133, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 134, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 135, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 136, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 137, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 138, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 139, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 140, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 141, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 142, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 143, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 144, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 145, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 146, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 147, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 148, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 149, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 150, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 151, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 152, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 153, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 154, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 155, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 156, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 157, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 158, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 159, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 160, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 161, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 162, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 163, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 164, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 165, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 166, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 167, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 168, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 169, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 170, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 171, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 172, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 173, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 174, "subject": "Buddhism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 1, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 2, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 3, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 4, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 5, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 6, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 7, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 8, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 9, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 10, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 11, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 12, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 13, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 14, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 15, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 16, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 17, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 18, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 19, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 20, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 21, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 22, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 23, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 24, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 25, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 26, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 27, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 28, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 29, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 30, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 31, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 32, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 33, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 34, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 35, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 36, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 37, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 38, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 39, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 40, "subject": "Catholicism", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 1, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 2, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 3, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 4, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 5, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 6, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 7, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 8, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 9, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 10, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 11, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 12, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 13, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 14, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 15, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 16, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 17, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 18, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 19, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 20, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 21, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 22, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 23, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 24, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 25, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 26, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 27, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 28, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 29, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 30, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 31, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 32, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 33, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 34, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 35, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 36, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 37, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 38, "subject": "Christianity", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 1, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 2, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 3, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 4, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 5, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 6, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 7, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 8, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 9, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 10, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 11, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 12, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 13, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 14, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 15, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 16, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 17, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 18, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 19, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 20, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 21, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 22, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 23, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 24, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 25, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 26, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 27, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 28, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 29, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 30, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 31, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 32, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 33, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 34, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 35, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 36, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 37, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 38, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 39, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 40, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 41, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 42, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 43, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 44, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 45, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 46, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 47, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 48, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 49, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 50, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 51, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 52, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 53, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 54, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 55, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 56, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 57, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 58, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 59, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 60, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 61, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 62, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 63, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 64, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 65, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 66, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 67, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 68, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 69, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 70, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 71, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 72, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 73, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 74, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 75, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 76, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 77, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 78, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 79, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 80, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 81, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 82, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 83, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 84, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 85, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 86, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 87, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 88, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 89, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 90, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 91, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 92, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 93, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 94, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 95, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 96, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 97, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 98, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 99, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 100, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 101, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 102, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 103, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 104, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 105, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 106, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 107, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 108, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 109, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 110, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 111, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 112, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 113, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 114, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 115, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 116, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 117, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 118, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 119, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 120, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 121, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 122, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 123, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 124, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 125, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 126, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 127, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 128, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 129, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 130, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 131, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 132, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 133, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 134, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 135, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 136, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 137, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 138, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 139, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 140, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 141, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 142, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 143, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 144, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 145, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 146, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 147, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 148, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 149, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 150, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 151, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 152, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 153, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 154, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 155, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 156, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 157, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 158, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 159, "subject": "Civics", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 1, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 2, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 3, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 4, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 5, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 6, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 7, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 8, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 9, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 10, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 11, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 12, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 13, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 14, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 15, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 16, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 17, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 18, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 19, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 20, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 21, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 22, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 23, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 24, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 25, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 26, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 27, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 28, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 29, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 30, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 31, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 32, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 33, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 34, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 35, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 36, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 37, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 38, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 39, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 40, "subject": "Communication and Media Studies", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 1, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 2, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 3, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 4, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 5, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 6, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 7, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 8, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 9, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 10, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 11, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 12, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 13, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 14, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 15, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 16, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 17, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 18, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 19, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 20, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 21, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 22, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 23, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 24, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 25, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 26, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 27, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 28, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 29, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 30, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 31, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 32, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 33, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 34, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 35, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 36, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 37, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 38, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 39, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 40, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 41, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 42, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 43, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 44, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 45, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 46, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 47, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 48, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 49, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 50, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 51, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 52, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 53, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 54, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 55, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 56, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 57, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 58, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 59, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 60, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 61, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 62, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 63, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 64, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 65, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 66, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 67, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 68, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 69, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 70, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 71, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 72, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 73, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 74, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 75, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 76, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 77, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 78, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 79, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 80, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 81, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 82, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 83, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 84, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 85, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 86, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 87, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 88, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 89, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 90, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 91, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 92, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 93, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 94, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 95, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 96, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 97, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 98, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 99, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 100, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 101, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 102, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 103, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 104, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 105, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 106, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 107, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 108, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 109, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 110, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 111, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 112, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 113, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 114, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 115, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 116, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 117, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 118, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 119, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 120, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 121, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 122, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 123, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 124, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 125, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 126, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 127, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 128, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 129, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 130, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 131, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 132, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 133, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 134, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 135, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 136, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 137, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 138, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 139, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 140, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 141, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 142, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 143, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 144, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 145, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 146, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 147, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 148, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 149, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 150, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 151, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 152, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 153, "subject": "Dancing", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 1, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 2, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 3, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 4, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 5, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 6, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 7, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 8, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 9, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 10, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 11, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 12, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 13, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 14, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 15, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 16, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 17, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 18, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 19, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 20, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 21, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 22, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 23, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 24, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 25, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 26, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 27, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 28, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 29, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 30, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 31, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "1"} +{"q_no": 32, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 33, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "4"} +{"q_no": 34, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 35, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "3"} +{"q_no": 36, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 37, "subject": "Design and Construction Technology", "category": "other", "level": "medium", "answer": "2"} +{"q_no": 1, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 2, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 3, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 4, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 5, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 6, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 7, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 8, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 9, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 10, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 11, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 12, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 13, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 14, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 15, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 16, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 17, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 18, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 19, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 20, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 21, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 22, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 23, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 24, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 25, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 26, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 27, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 28, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 29, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 30, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 31, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 32, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 33, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 34, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 35, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 36, "subject": "Eastern Music", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 1, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 2, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 3, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 4, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 5, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 6, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 7, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 8, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 9, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 10, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 11, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 12, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 13, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 14, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 15, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 16, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 17, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 18, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 19, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 20, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 21, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 22, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 23, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 24, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 25, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 26, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 27, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 28, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 29, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 30, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 31, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 32, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 33, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 34, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 35, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 36, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 37, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 38, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 39, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 40, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 41, "subject": "Geography", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 1, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 2, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 3, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 4, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 5, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 6, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 7, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 8, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 9, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 10, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 11, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 12, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 13, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 14, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 15, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 16, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 17, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 18, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 19, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 20, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 21, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 22, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 23, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 24, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 25, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 26, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 27, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 28, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 29, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 30, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 31, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 32, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 33, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 34, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 35, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 36, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 37, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 38, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 39, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 40, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 41, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 42, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 43, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 44, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 45, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 46, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 47, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 48, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 49, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 50, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 51, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 52, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 53, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 54, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 55, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 56, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 57, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 58, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 59, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 60, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 61, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 62, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 63, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 64, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 65, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 66, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 67, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "1"} +{"q_no": 68, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "3"} +{"q_no": 69, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 70, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 71, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "4"} +{"q_no": 72, "subject": "Health", "category": "Social_science", "level": "medium", "answer": "2"} +{"q_no": 1, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 2, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 3, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 4, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 5, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 6, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 7, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 8, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 9, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 10, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 11, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 12, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 13, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 14, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 15, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 16, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 17, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 18, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 19, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 20, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 21, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 22, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 23, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 24, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 25, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 26, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 27, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 28, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 29, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 30, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 31, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 32, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 33, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 34, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 35, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 36, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 37, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 38, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 39, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 40, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 41, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 42, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 43, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 44, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 45, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 46, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 47, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 48, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 49, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 50, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 51, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 52, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 53, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 54, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 55, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 56, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 57, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 58, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 59, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 60, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 61, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 62, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 63, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 64, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 65, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 66, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 67, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 68, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 69, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 70, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 71, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 72, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 73, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 74, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 75, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 76, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 77, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 78, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 79, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 80, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 81, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 82, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 83, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 84, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 85, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 86, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 87, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 88, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 89, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 90, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 91, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 92, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 93, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 94, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 95, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 96, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 97, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 98, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 99, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 100, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 101, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 102, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 103, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 104, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 105, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 106, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 107, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 108, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 109, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 110, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 111, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 112, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 113, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 114, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 115, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 116, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 117, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 118, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 119, "subject": "History", "category": "Humanities", "level": "medium", "answer": "3"} +{"q_no": 120, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 121, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 122, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 123, "subject": "History", "category": "Humanities", "level": "medium", "answer": "4"} +{"q_no": 124, "subject": "History", "category": "Humanities", "level": "medium", "answer": "2"} +{"q_no": 125, "subject": "History", "category": "Humanities", "level": "medium", "answer": "1"} +{"q_no": 1, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 2, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 3, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 4, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 5, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 6, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 7, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 8, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 9, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 10, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 11, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 12, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 13, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 14, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 15, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 16, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 17, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 18, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 19, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 20, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 21, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 22, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 23, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 24, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 25, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 26, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 27, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 28, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 29, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 30, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 31, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 32, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 33, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 34, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 35, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 36, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 37, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 38, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 39, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 40, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 41, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 42, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 43, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 44, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 45, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 46, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 47, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 48, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 49, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 50, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 51, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 52, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 53, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 54, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 55, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 56, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 57, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 58, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 59, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 60, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 61, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 62, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 63, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 64, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 65, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 66, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 67, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 68, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 69, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 70, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 71, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 72, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 73, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 74, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 75, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 76, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 77, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 78, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 79, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 80, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 81, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 82, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 83, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 84, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 85, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 86, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 87, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 88, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 89, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 90, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 91, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 92, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 93, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 94, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 95, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 96, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 97, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 98, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 99, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 100, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 101, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 102, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 103, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 104, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 105, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 106, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 107, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 108, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 109, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 110, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 111, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 112, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 113, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 114, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 115, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 116, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 117, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 118, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 119, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 120, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 121, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 122, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 123, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 124, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 125, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 126, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 127, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 128, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 129, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 130, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 131, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 132, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 133, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 134, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 135, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 136, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 137, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 138, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 139, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 140, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 141, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 142, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 143, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 144, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 145, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 146, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 147, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 148, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 149, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 150, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 151, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 152, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 153, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "2"} +{"q_no": 154, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 155, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 156, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} +{"q_no": 157, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "3"} +{"q_no": 158, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "1"} +{"q_no": 159, "subject": "Sinhala", "category": "language", "level": "medium", "answer": "4"} diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000000000000000000000000000000000000..2fa985ea7c2161bdaa89fda3da950fa4d04c6965 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,12 @@ +# Versions used for our final submission (Python 3.12.3). PyTorch: we used torch 2.14.1 built for CUDA 13.0 +# (NVIDIA GB10, aarch64: pip install torch==2.14.1 --index-url https://download.pytorch.org/whl/cu130); any CUDA build of 2.14.1 works. +torch==2.14.1 +transformers==5.19.0 +accelerate==1.15.0 +huggingface_hub==2.2.0 +tokenizers==0.23.3 +safetensors==0.8.0 +jinja2==3.1.6 +numpy==2.5.3 +pyyaml==6.0.3 +tqdm==4.70.1 diff --git a/training/.gitignore b/training/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..8025c628f258711f423133bed592dc912032ca6d --- /dev/null +++ b/training/.gitignore @@ -0,0 +1,20 @@ +# Secrets +.env +*.token + +# Task data (CC BY-NC-ND 4.0): never commit. Fetch with scripts/fetch_data.py. +data/ + +# Weights, runs, caches +models/ +checkpoints/ +results/ +*.safetensors +*.bin +*.pt +*.zip +.venv/ +__pycache__/ +*.pyc +.cache/ +wandb/ diff --git a/training/DATA_LOG.md b/training/DATA_LOG.md new file mode 100644 index 0000000000000000000000000000000000000000..752efa01036629646071139fb09e671787c4dc34 --- /dev/null +++ b/training/DATA_LOG.md @@ -0,0 +1,41 @@ +# Data log + +Every dataset touched by this project, how it is used, and every closed-API data-generation call (required +disclosure). Task data is never committed; `scripts/fetch_data.py` re-downloads it at the pinned revision. + +## Evaluation data + +| Dataset | Revision | License | Use | Notes | +|---|---|---|---|---| +| SinhalaMMLU public Easy tier (`naist-nlp/SinhalaMMLU`, TEST/easy) | `7dd30681d9` | CC BY-NC-ND 4.0 | Dev evaluation | 1,851 questions, 14 subjects, grades 6-8, 4 options. `Geography/64` dropped (answer label `24`). Cite Pramodya et al., EMNLP 2025. | +| Global-MMLU `si` (`CohereLabs/Global-MMLU`) | `0e619dbeb3` | Apache-2.0 | Harder internal validation: fixed 2,000-question stratified sample of `test` (ids in `configs/splits/gmmlu_si_val_ids.txt`); `dev` split = few-shot pool | Validation ids must be excluded from any training mix. | +| Shared-task test set (`SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset`) | `56fb2da50c` | (organisers) | **Prediction only.** Never used for training, tuning, prompt design or model selection; no answers included | 2,238 questions (easy 642, medium 1,111, hard 485), 4 options, same JSON layout as dev. Stored in `data/test/` (git-ignored). | +| Belebele `sin_Sinh` (`facebook/belebele`) | `7899cdfa4e` | CC BY-SA 4.0 | **Evaluation only**, never training | 900 reading-comprehension questions. | + +## Training data + +| Dataset | Revision | License | Use | Processing | +|---|---|---|---|---| +| Global-MMLU `si` test (minus the 2,000 validation ids) + dev split | `0e619dbeb3` | Apache-2.0 | Fine-tuning MCQs: `data/sft/gmmlu_si_train.jsonl` via `scripts/build_sft_gmmlu.py` | 12,327 pool → 11,447 kept. Dropped 819 with char-5-gram TF-IDF cosine ≥ 0.6 to any validation item (MMLU repeats questions across subjects) and 61 within-pool copies (≥ 0.95). Max similarity of kept items to SinhalaMMLU dev: 0.42. Options shuffled to balanced gold positions except options that refer to other options. Stats: `reports/data_stats_sft_gmmlu.json` | +| FineWeb-2 `sin_Sinh` (`HuggingFaceFW/fineweb-2`) | `af9c13333e` | ODC-By 1.0 | Continued pretraining | `scripts/build_cpt_corpus.py` (filters below). Already MinHash-deduplicated upstream | +| Sinhala Wikipedia (`wikimedia/wikipedia`, `20231101.si`) | `b04c8d1ceb` | CC BY-SA 3.0 / GFDL | Continued pretraining | Same filters | +| NSINA news (`sinhala-nlp/NSINA`) | `a986af81ad` | CC BY-SA 4.0 | Continued pretraining | Same filters | + +Corpus filters (`scripts/build_cpt_corpus.py`), first failing rule recorded per document; counts in +`reports/corpus_stats.json`: blocked exam / past-paper / curriculum domains; exam-like text (question-paper phrases or +≥ 8 numbered option lines); < 200 characters; < 50% Sinhala letters; < 30% unique lines; exact duplicate across +sources (normalised text); any word 8-gram shared with a SinhalaMMLU dev or Global-MMLU validation question. +Not yet available: CulturaX `si` (gated, access not requested), MADLAD-400 `si` (path not found). + +### Contamination controls (apply to every training/pretraining source) + +- SinhalaMMLU dev questions come from **pastpapers.wiki** (93%) and **e-thaksalawa.moe.gov.lk** (5%), i.e. Sri Lankan + term-test and exam papers. Drop all documents from these domains, and from any other past-paper / exam site, from + web corpora (FineWeb-2, CulturaX, MADLAD-400, CC-100, OSCAR) before training. +- N-gram overlap filter of every training corpus against SinhalaMMLU dev and the Global-MMLU validation split. +- No scraped Sri Lankan O/L or A/L past papers. Ask the organisers before using government textbooks. + +## Closed-API data generation (disclosure) + +_None yet._ For each use: date, provider and model, purpose, prompt template location, number of items generated, +filtering applied, and where the output is stored. diff --git a/training/MODEL_MANIFEST.md b/training/MODEL_MANIFEST.md new file mode 100644 index 0000000000000000000000000000000000000000..c0f306ef407784527fde4304f0b31e063ea3154c --- /dev/null +++ b/training/MODEL_MANIFEST.md @@ -0,0 +1,52 @@ +# Model manifest + +Every neural component loaded at inference, with its exact total parameter count, release date and license. +Shared-task limit: **total parameters of all inference components ≤ 8B** (embeddings, vision towers if loaded, +adapters on top of the base, MoE total not active, quantisation does not reduce the count). +Only open-weight models **released before 2026-10-01**. + +## Final submission + +- Base model: `google/gemma-4-E4B-it` @ `ee0ef6023621cff504d758262d4e04895a5af4a2`, text-only extraction + (`scripts/make_text_only.py`), published as `SinhalaMMLU-SharedTask/team-nexus-gemma-4-e4b-it-text`. +- No fine-tuned derivative is used: no adapter, no merged weights. +- Total parameters loaded: 7,463,013,376. +- Command: `python inference.py` in the team repo, which runs `inference/predict.py` with + `configs/submission_base_think_gated_sc.yaml` (Codabench submission ID 50, score 62.93). + +## Candidate parameter accounting + +Generated by `python scripts/count_params.py` (counts read from safetensors headers; tied embeddings counted once; +"Instantiated" is what `transformers` builds from the config). "Repo created" is the Hugging Face repo creation +date and is only a proxy for the release date: confirm against the official announcement before relying on it. + + + +| Model | Verdict | Total (all stored) | Text model | Vision | Audio | MTP | Embeddings | Tied | Instantiated (class) | Repo created | License | Commit | +|---|---|---:|---:|---:|---:|---:|---:|---|---|---|---|---| +| `google/gemma-3-4b-it` | OK | 4.300B | 3.880B | 0.420B | - | - | 0.671B | no | 4.300B (Gemma3ForConditionalGeneration) | 2025-02-20 | gemma | `093f9f388b` | +| `google/gemma-3n-E4B-it` | OK | 7.850B | 6.867B | 0.299B | 0.685B | - | 2.886B | no | 7.819B (Gemma3nForConditionalGeneration) | 2025-06-03 | gemma | `c1221e9c62` | +| `google/gemma-3n-E2B-it` | OK | 5.439B | 4.456B | 0.299B | 0.685B | - | 2.551B | no | 5.418B (Gemma3nForConditionalGeneration) | 2025-06-12 | gemma | `5e092ebca1` | +| `google/gemma-4-E4B-it` | OK | 7.996B | 7.518B | 0.169B | 0.309B | - | 3.490B | yes | 7.941B (Gemma4ForConditionalGeneration) | 2026-03-02 | apache-2.0 | `ee0ef60236` | +| `google/gemma-4-E2B-it` | OK | 5.123B | 4.647B | 0.169B | 0.307B | - | 2.751B | yes | 5.104B (Gemma4ForConditionalGeneration) | 2026-03-02 | apache-2.0 | `3e22461f65` | +| `google/gemma-4-12B-it` | OVER 8B | 11.960B | 11.907B | 0.050B | 0.002B | - | 1.007B | yes | 11.960B (Gemma4UnifiedForConditionalGeneration) | 2026-05-23 | apache-2.0 | `707f0a3b8a` | +| `Qwen/Qwen3-4B-Instruct-2507` | OK | 4.022B | 4.022B | - | - | - | 0.389B | yes | 4.022B (Qwen3ForCausalLM) | 2025-08-05 | apache-2.0 | `cdbee75f17` | +| `Qwen/Qwen3-4B` | OK | 4.022B | 4.022B | - | - | - | 0.389B | yes | 4.022B (Qwen3ForCausalLM) | 2025-04-27 | apache-2.0 | `1cfa9a7208` | +| `Qwen/Qwen3.5-4B` | OK | 4.660B | 4.206B | 0.334B | - | 0.121B | 0.636B | yes | 4.206B (Qwen3_5ForCausalLM) | 2026-02-27 | apache-2.0 | `851bf6e806` | +| `Qwen/Qwen3-8B` | OVER 8B | 8.191B | 8.191B | - | - | - | 1.245B | no | 8.191B (Qwen3ForCausalLM) | 2025-04-27 | apache-2.0 | `b968826d9c` | +| `Qwen/Qwen3.5-9B` | OVER 8B | 9.653B | 8.954B | 0.456B | - | 0.243B | 2.034B | no | 8.954B (Qwen3_5ForCausalLM) | 2026-02-27 | apache-2.0 | `c202236235` | +| `meta-llama/Llama-3.2-3B-Instruct` | OK | 3.213B | 3.213B | - | - | - | 0.394B | yes | 3.213B (LlamaForCausalLM) | 2024-09-18 | llama3.2 | `0cb88a4f76` | +| `meta-llama/Llama-3.1-8B-Instruct` | OVER 8B | 8.030B | 8.030B | - | - | - | 1.051B | no | 8.030B (LlamaForCausalLM) | 2024-07-18 | llama3.1 | `0e9e39f249` | +| `polyglots/SinLlama_v01`
adapter on `meta-llama/Meta-Llama-3-8B` (+3.592B) | OVER 8B | 11.622B | 11.622B | - | - | - | 1.051B | no | base 8030261248 + adapter | 2024-11-08 | ? | `4f464b14d5` | +| `ibm-granite/granite-4.1-8b` | OVER 8B | 8.381B | 8.381B | - | - | - | 0.411B | yes | 8.381B (GraniteForCausalLM) | 2026-04-06 | apache-2.0 | `1504002f65` | +| `ibm-granite/granite-4.2-8b` | OVER 8B | 8.792B | 8.792B | - | - | - | 0.822B | no | 8.792B (GraniteForCausalLM) | 2026-08-07 | apache-2.0 | `f8de16cdcd` | +| `ibm-granite/granite-4.2-3b` | OK | 3.660B | 3.660B | - | - | - | 0.514B | no | 3.660B (GraniteForCausalLM) | 2026-08-07 | apache-2.0 | `e459acceac` | +| `HuggingFaceTB/SmolLM3-3B` | OK | 3.075B | 3.075B | - | - | - | 0.263B | yes | 3.075B (SmolLM3ForCausalLM) | 2025-07-08 | apache-2.0 | `a07cc9a04f` | +| `microsoft/Phi-4-mini-instruct` | OK | 3.836B | 3.836B | - | - | - | 0.615B | yes | 3.836B (Phi3ForCausalLM) | 2025-02-19 | mit | `cfbefacb99` | + + + +## Notes + +- A verdict of "OK only text-only" means the checkpoint stores a vision/audio tower or MTP head. The submission + must load a text-only model (and the stripped weights' count is what goes in the final row above). diff --git a/training/README.md b/training/README.md new file mode 100644 index 0000000000000000000000000000000000000000..d5037a0670a8cb062d756f9c33de981091fad3ed --- /dev/null +++ b/training/README.md @@ -0,0 +1,93 @@ +# SinhalaMMLU shared task (≤ 8B) + +System for the ICTer 2026 SinhalaMMLU shared task: Sinhala multiple-choice QA with open-weight models totalling +≤ 8B parameters, no retrieval, no internet at inference. Built on SinhalaMMLU (Pramodya et al., EMNLP 2025): + +```bibtex +@inproceedings{pramodya-etal-2025-sinhalammlu, + title = "{S}inhala{MMLU}: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in {S}inhala", + author = "Pramodya, Ashmari and Nelki, Nirasha and Shalinda, Heshan and Liyanage, Chamila and Sakai, Yusuke and + Pushpananda, Randil and Weerasinghe, Ruvan and Kamigaito, Hidetaka and Watanabe, Taro", + booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing", + year = "2025", + pages = "32931--32949", + url = "https://aclanthology.org/2025.emnlp-main.1673/" +} +``` + +**Keep this repository private until the final deadline. Never commit task data (`data/` is git-ignored).** + +## Layout + +| Path | Contents | +|---|---| +| `data/` | Git-ignored. `raw/` downloads, `processed/` normalised JSONL | +| `eval/` | Evaluation harness (`run_eval.py`, scorers, metrics, `summarize.py`) | +| `train/` | Fine-tuning / continued-pretraining code | +| `inference/` | Final offline submission pipeline | +| `configs/` | Candidate models, fixed data splits | +| `scripts/` | Data building, parameter counting, overlap checks, synthetic MCQ generation | +| `scripts/runs/` | The exact experiment queues behind every reported result and submission (see its README) | +| `run_submission.sh` | One-command offline reproduction of a submission | +| `reports/` | Aggregate results tables (no task data) | +| `MODEL_MANIFEST.md` | Every model: exact total parameters, release date, license | +| `DATA_LOG.md` | Every dataset and every closed-API data-generation use | + +## Setup (DGX Spark, aarch64, CUDA 13) + +```bash +python3 -m venv .venv +.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cu130 +.venv/bin/pip install -r requirements.lock.txt +echo "hf_token = hf_..." > .env && chmod 600 .env # read token; accept gated terms for SinhalaMMLU, Gemma 3/3n, Llama +``` + +Triton (used by `torch.compile`, Liger Kernel and flash-linear-attention) compiles helpers against `Python.h`. +If `python3.12-dev` is installed, nothing else is needed. Without sudo, unpack the headers locally; the code +(`eval/common.py: ensure_python_headers`) then adds them to `CPATH` automatically: + +```bash +mkdir -p ~/.local/pydev && cd ~/.local/pydev +V=3.12.3-1ubuntu0.17 # any 3.12.3 Ubuntu build; the machine's apt index may list a version no longer on the mirror +for p in libpython3.12-dev python3.12-dev; do + curl -fO "http://ports.ubuntu.com/ubuntu-ports/pool/main/p/python3.12/${p}_${V}_arm64.deb"; done +for f in *.deb; do dpkg -x "$f" root; done +``` + +Long GPU jobs: PyTorch is capped at 100 GB (`SMMLU_GPU_MEM_GB`). The Spark shares memory between CPU and GPU, so +an uncapped job triggers the kernel OOM killer (which also kills tmux) instead of a CUDA out-of-memory error. + +## Commands + +```bash +.venv/bin/python scripts/fetch_data.py # SinhalaMMLU dev, Global-MMLU si, Belebele si -> data/raw +.venv/bin/python scripts/prepare_data.py # -> data/processed/*.jsonl (fixed splits in configs/splits) +.venv/bin/python scripts/count_params.py # exact parameter counts -> MODEL_MANIFEST.md +scripts/runs/run_baselines.sh # zero-shot baselines -> results/, reports/ +./run_submission.sh [config] # reproduce a submission offline -> results/submissions/final.zip + +# One model, any setting +.venv/bin/python -m eval.run_eval --model google/gemma-4-E4B-it --data sinhalammlu_dev gmmlu_si_val \ + --method logit --n-perms 4 --lang en --run-name my_run +``` + +### Harness options + +- `--method logit` (next-token probability over labels 1-4), `generate` (greedy decode + parse), `cloze` + (length-normalised option likelihood). +- `--n-perms K --perm-mode cyclic|random`: score each question under K option orders. Reports original-order + accuracy, permutation-vote accuracy, spread across orders, consistency, and position-bias tables. +- `--lang en|si` instruction language; `--chat auto|on|off`; `--fewshot-file F --n-shot N` (the same fixed + examples for every question, as the rules require). +- Breakdowns by dataset, domain, subject, grade and (Global-MMLU) cultural sensitivity, with Wilson 95% CIs. +- Every run stores the exact model commit, library versions, git commit and the full argument set in `metrics.json`. +- `--write-submission` writes a Codabench-format `predictions.zip` (`predictions.jsonl` with + `q_no, subject, category, level, answer`). + +## Rules checklist + +- Total parameters of everything loaded at inference ≤ 8B (embeddings, vision/audio towers if loaded, adapters). +- Open-weight models released before 2026-10-01 only. No API, internet or retrieval of any kind at inference. +- Fixed few-shot examples only (identical for every question). No training, tuning or prompt design on test questions. +- Closed APIs only for training-data generation, logged in `DATA_LOG.md`. +- Final pipeline runs offline from one command with pinned versions and fixed seeds. diff --git a/training/configs/models.yaml b/training/configs/models.yaml new file mode 100644 index 0000000000000000000000000000000000000000..36b60977f8ae392af248a8d8e82ef78b003e0da7 --- /dev/null +++ b/training/configs/models.yaml @@ -0,0 +1,26 @@ +# Candidate models. `chat` sets prompt wrapping in baselines; `baseline: false` = count parameters only. +# Every entry must be verified in MODEL_MANIFEST.md (total params ≤ 8B, released before 2026-10-01, license). +candidates: + # Gemma + - {id: google/gemma-3-4b-it, chat: true} + - {id: google/gemma-3n-E4B-it, chat: true} + - {id: google/gemma-3n-E2B-it, chat: true, baseline: false} + - {id: google/gemma-4-E4B-it, chat: true} + - {id: google/gemma-4-E2B-it, chat: true, baseline: false} + - {id: google/gemma-4-12B-it, chat: true, baseline: false} # reference: expected over the limit + # Qwen + - {id: Qwen/Qwen3-4B-Instruct-2507, chat: true} + - {id: Qwen/Qwen3-4B, chat: true} # hybrid thinking model, thinking disabled + - {id: Qwen/Qwen3.5-4B, chat: true} + - {id: Qwen/Qwen3-8B, chat: true, baseline: false} # reference: expected ~8.2B + - {id: Qwen/Qwen3.5-9B, chat: true, baseline: false} # reference: expected over the limit + # Llama / SinLlama + - {id: meta-llama/Llama-3.2-3B-Instruct, chat: true} + - {id: meta-llama/Llama-3.1-8B-Instruct, chat: true, baseline: false} # 8.03B: needs organiser ruling + - {id: polyglots/SinLlama_v01, chat: false, baseline: false} # LoRA adapter on Meta-Llama-3-8B + extended vocab + # Others + - {id: ibm-granite/granite-4.1-8b, chat: true, baseline: false} # 8.38B: over the limit + - {id: ibm-granite/granite-4.2-8b, chat: true, baseline: false} # 8.79B: over the limit + - {id: ibm-granite/granite-4.2-3b, chat: true} + - {id: HuggingFaceTB/SmolLM3-3B, chat: true} + - {id: microsoft/Phi-4-mini-instruct, chat: true} diff --git a/training/configs/splits/gmmlu_si_val_ids.txt b/training/configs/splits/gmmlu_si_val_ids.txt new file mode 100644 index 0000000000000000000000000000000000000000..3ef59ee8d3aa4297b97d97a41a3b49848db9029c --- /dev/null +++ b/training/configs/splits/gmmlu_si_val_ids.txt @@ -0,0 +1,2000 @@ +gmmlu_si/test/abstract_algebra/test/0 +gmmlu_si/test/abstract_algebra/test/12 +gmmlu_si/test/abstract_algebra/test/18 +gmmlu_si/test/abstract_algebra/test/19 +gmmlu_si/test/abstract_algebra/test/2 +gmmlu_si/test/abstract_algebra/test/21 +gmmlu_si/test/abstract_algebra/test/36 +gmmlu_si/test/abstract_algebra/test/5 +gmmlu_si/test/abstract_algebra/test/6 +gmmlu_si/test/abstract_algebra/test/76 +gmmlu_si/test/abstract_algebra/test/86 +gmmlu_si/test/abstract_algebra/test/89 +gmmlu_si/test/abstract_algebra/test/98 +gmmlu_si/test/abstract_algebra/test/99 +gmmlu_si/test/anatomy/test/100 +gmmlu_si/test/anatomy/test/101 +gmmlu_si/test/anatomy/test/104 +gmmlu_si/test/anatomy/test/108 +gmmlu_si/test/anatomy/test/112 +gmmlu_si/test/anatomy/test/113 +gmmlu_si/test/anatomy/test/118 +gmmlu_si/test/anatomy/test/119 +gmmlu_si/test/anatomy/test/124 +gmmlu_si/test/anatomy/test/132 +gmmlu_si/test/anatomy/test/19 +gmmlu_si/test/anatomy/test/34 +gmmlu_si/test/anatomy/test/57 +gmmlu_si/test/anatomy/test/84 +gmmlu_si/test/anatomy/test/85 +gmmlu_si/test/anatomy/test/89 +gmmlu_si/test/anatomy/test/9 +gmmlu_si/test/anatomy/test/93 +gmmlu_si/test/anatomy/test/96 +gmmlu_si/test/astronomy/test/1 +gmmlu_si/test/astronomy/test/101 +gmmlu_si/test/astronomy/test/11 +gmmlu_si/test/astronomy/test/112 +gmmlu_si/test/astronomy/test/114 +gmmlu_si/test/astronomy/test/115 +gmmlu_si/test/astronomy/test/130 +gmmlu_si/test/astronomy/test/136 +gmmlu_si/test/astronomy/test/144 +gmmlu_si/test/astronomy/test/19 +gmmlu_si/test/astronomy/test/23 +gmmlu_si/test/astronomy/test/24 +gmmlu_si/test/astronomy/test/31 +gmmlu_si/test/astronomy/test/39 +gmmlu_si/test/astronomy/test/43 +gmmlu_si/test/astronomy/test/49 +gmmlu_si/test/astronomy/test/67 +gmmlu_si/test/astronomy/test/71 +gmmlu_si/test/astronomy/test/73 +gmmlu_si/test/astronomy/test/8 +gmmlu_si/test/astronomy/test/84 +gmmlu_si/test/astronomy/test/90 +gmmlu_si/test/business_ethics/test/11 +gmmlu_si/test/business_ethics/test/17 +gmmlu_si/test/business_ethics/test/21 +gmmlu_si/test/business_ethics/test/25 +gmmlu_si/test/business_ethics/test/27 +gmmlu_si/test/business_ethics/test/33 +gmmlu_si/test/business_ethics/test/44 +gmmlu_si/test/business_ethics/test/49 +gmmlu_si/test/business_ethics/test/66 +gmmlu_si/test/business_ethics/test/7 +gmmlu_si/test/business_ethics/test/73 +gmmlu_si/test/business_ethics/test/77 +gmmlu_si/test/business_ethics/test/86 +gmmlu_si/test/business_ethics/test/98 +gmmlu_si/test/clinical_knowledge/test/107 +gmmlu_si/test/clinical_knowledge/test/108 +gmmlu_si/test/clinical_knowledge/test/116 +gmmlu_si/test/clinical_knowledge/test/137 +gmmlu_si/test/clinical_knowledge/test/17 +gmmlu_si/test/clinical_knowledge/test/177 +gmmlu_si/test/clinical_knowledge/test/179 +gmmlu_si/test/clinical_knowledge/test/18 +gmmlu_si/test/clinical_knowledge/test/187 +gmmlu_si/test/clinical_knowledge/test/2 +gmmlu_si/test/clinical_knowledge/test/204 +gmmlu_si/test/clinical_knowledge/test/205 +gmmlu_si/test/clinical_knowledge/test/206 +gmmlu_si/test/clinical_knowledge/test/207 +gmmlu_si/test/clinical_knowledge/test/211 +gmmlu_si/test/clinical_knowledge/test/217 +gmmlu_si/test/clinical_knowledge/test/223 +gmmlu_si/test/clinical_knowledge/test/225 +gmmlu_si/test/clinical_knowledge/test/228 +gmmlu_si/test/clinical_knowledge/test/232 +gmmlu_si/test/clinical_knowledge/test/235 +gmmlu_si/test/clinical_knowledge/test/236 +gmmlu_si/test/clinical_knowledge/test/29 +gmmlu_si/test/clinical_knowledge/test/36 +gmmlu_si/test/clinical_knowledge/test/4 +gmmlu_si/test/clinical_knowledge/test/40 +gmmlu_si/test/clinical_knowledge/test/48 +gmmlu_si/test/clinical_knowledge/test/49 +gmmlu_si/test/clinical_knowledge/test/52 +gmmlu_si/test/clinical_knowledge/test/60 +gmmlu_si/test/clinical_knowledge/test/63 +gmmlu_si/test/clinical_knowledge/test/7 +gmmlu_si/test/clinical_knowledge/test/75 +gmmlu_si/test/clinical_knowledge/test/8 +gmmlu_si/test/clinical_knowledge/test/9 +gmmlu_si/test/clinical_knowledge/test/92 +gmmlu_si/test/clinical_knowledge/test/97 +gmmlu_si/test/clinical_knowledge/test/98 +gmmlu_si/test/college_biology/test/121 +gmmlu_si/test/college_biology/test/122 +gmmlu_si/test/college_biology/test/125 +gmmlu_si/test/college_biology/test/126 +gmmlu_si/test/college_biology/test/133 +gmmlu_si/test/college_biology/test/134 +gmmlu_si/test/college_biology/test/140 +gmmlu_si/test/college_biology/test/142 +gmmlu_si/test/college_biology/test/28 +gmmlu_si/test/college_biology/test/30 +gmmlu_si/test/college_biology/test/32 +gmmlu_si/test/college_biology/test/37 +gmmlu_si/test/college_biology/test/43 +gmmlu_si/test/college_biology/test/49 +gmmlu_si/test/college_biology/test/56 +gmmlu_si/test/college_biology/test/65 +gmmlu_si/test/college_biology/test/7 +gmmlu_si/test/college_biology/test/70 +gmmlu_si/test/college_biology/test/72 +gmmlu_si/test/college_biology/test/81 +gmmlu_si/test/college_biology/test/91 +gmmlu_si/test/college_chemistry/test/12 +gmmlu_si/test/college_chemistry/test/14 +gmmlu_si/test/college_chemistry/test/17 +gmmlu_si/test/college_chemistry/test/18 +gmmlu_si/test/college_chemistry/test/19 +gmmlu_si/test/college_chemistry/test/33 +gmmlu_si/test/college_chemistry/test/34 +gmmlu_si/test/college_chemistry/test/41 +gmmlu_si/test/college_chemistry/test/44 +gmmlu_si/test/college_chemistry/test/45 +gmmlu_si/test/college_chemistry/test/54 +gmmlu_si/test/college_chemistry/test/67 +gmmlu_si/test/college_chemistry/test/9 +gmmlu_si/test/college_chemistry/test/92 +gmmlu_si/test/college_computer_science/test/16 +gmmlu_si/test/college_computer_science/test/17 +gmmlu_si/test/college_computer_science/test/2 +gmmlu_si/test/college_computer_science/test/30 +gmmlu_si/test/college_computer_science/test/31 +gmmlu_si/test/college_computer_science/test/36 +gmmlu_si/test/college_computer_science/test/38 +gmmlu_si/test/college_computer_science/test/54 +gmmlu_si/test/college_computer_science/test/58 +gmmlu_si/test/college_computer_science/test/61 +gmmlu_si/test/college_computer_science/test/67 +gmmlu_si/test/college_computer_science/test/76 +gmmlu_si/test/college_computer_science/test/89 +gmmlu_si/test/college_computer_science/test/90 +gmmlu_si/test/college_mathematics/test/15 +gmmlu_si/test/college_mathematics/test/26 +gmmlu_si/test/college_mathematics/test/29 +gmmlu_si/test/college_mathematics/test/32 +gmmlu_si/test/college_mathematics/test/43 +gmmlu_si/test/college_mathematics/test/5 +gmmlu_si/test/college_mathematics/test/50 +gmmlu_si/test/college_mathematics/test/57 +gmmlu_si/test/college_mathematics/test/66 +gmmlu_si/test/college_mathematics/test/68 +gmmlu_si/test/college_mathematics/test/70 +gmmlu_si/test/college_mathematics/test/76 +gmmlu_si/test/college_mathematics/test/9 +gmmlu_si/test/college_mathematics/test/98 +gmmlu_si/test/college_medicine/test/0 +gmmlu_si/test/college_medicine/test/107 +gmmlu_si/test/college_medicine/test/109 +gmmlu_si/test/college_medicine/test/111 +gmmlu_si/test/college_medicine/test/113 +gmmlu_si/test/college_medicine/test/118 +gmmlu_si/test/college_medicine/test/128 +gmmlu_si/test/college_medicine/test/13 +gmmlu_si/test/college_medicine/test/131 +gmmlu_si/test/college_medicine/test/136 +gmmlu_si/test/college_medicine/test/137 +gmmlu_si/test/college_medicine/test/146 +gmmlu_si/test/college_medicine/test/148 +gmmlu_si/test/college_medicine/test/162 +gmmlu_si/test/college_medicine/test/18 +gmmlu_si/test/college_medicine/test/2 +gmmlu_si/test/college_medicine/test/30 +gmmlu_si/test/college_medicine/test/33 +gmmlu_si/test/college_medicine/test/42 +gmmlu_si/test/college_medicine/test/51 +gmmlu_si/test/college_medicine/test/69 +gmmlu_si/test/college_medicine/test/7 +gmmlu_si/test/college_medicine/test/87 +gmmlu_si/test/college_medicine/test/89 +gmmlu_si/test/college_medicine/test/91 +gmmlu_si/test/college_physics/test/17 +gmmlu_si/test/college_physics/test/24 +gmmlu_si/test/college_physics/test/37 +gmmlu_si/test/college_physics/test/39 +gmmlu_si/test/college_physics/test/41 +gmmlu_si/test/college_physics/test/43 +gmmlu_si/test/college_physics/test/53 +gmmlu_si/test/college_physics/test/54 +gmmlu_si/test/college_physics/test/56 +gmmlu_si/test/college_physics/test/69 +gmmlu_si/test/college_physics/test/72 +gmmlu_si/test/college_physics/test/73 +gmmlu_si/test/college_physics/test/8 +gmmlu_si/test/college_physics/test/86 +gmmlu_si/test/college_physics/test/95 +gmmlu_si/test/computer_security/test/14 +gmmlu_si/test/computer_security/test/21 +gmmlu_si/test/computer_security/test/33 +gmmlu_si/test/computer_security/test/35 +gmmlu_si/test/computer_security/test/40 +gmmlu_si/test/computer_security/test/43 +gmmlu_si/test/computer_security/test/49 +gmmlu_si/test/computer_security/test/52 +gmmlu_si/test/computer_security/test/54 +gmmlu_si/test/computer_security/test/6 +gmmlu_si/test/computer_security/test/7 +gmmlu_si/test/computer_security/test/77 +gmmlu_si/test/computer_security/test/9 +gmmlu_si/test/computer_security/test/91 +gmmlu_si/test/conceptual_physics/test/0 +gmmlu_si/test/conceptual_physics/test/114 +gmmlu_si/test/conceptual_physics/test/134 +gmmlu_si/test/conceptual_physics/test/136 +gmmlu_si/test/conceptual_physics/test/137 +gmmlu_si/test/conceptual_physics/test/140 +gmmlu_si/test/conceptual_physics/test/145 +gmmlu_si/test/conceptual_physics/test/163 +gmmlu_si/test/conceptual_physics/test/167 +gmmlu_si/test/conceptual_physics/test/169 +gmmlu_si/test/conceptual_physics/test/178 +gmmlu_si/test/conceptual_physics/test/183 +gmmlu_si/test/conceptual_physics/test/186 +gmmlu_si/test/conceptual_physics/test/19 +gmmlu_si/test/conceptual_physics/test/190 +gmmlu_si/test/conceptual_physics/test/197 +gmmlu_si/test/conceptual_physics/test/203 +gmmlu_si/test/conceptual_physics/test/207 +gmmlu_si/test/conceptual_physics/test/208 +gmmlu_si/test/conceptual_physics/test/210 +gmmlu_si/test/conceptual_physics/test/217 +gmmlu_si/test/conceptual_physics/test/229 +gmmlu_si/test/conceptual_physics/test/234 +gmmlu_si/test/conceptual_physics/test/25 +gmmlu_si/test/conceptual_physics/test/31 +gmmlu_si/test/conceptual_physics/test/38 +gmmlu_si/test/conceptual_physics/test/41 +gmmlu_si/test/conceptual_physics/test/44 +gmmlu_si/test/conceptual_physics/test/66 +gmmlu_si/test/conceptual_physics/test/78 +gmmlu_si/test/conceptual_physics/test/81 +gmmlu_si/test/conceptual_physics/test/86 +gmmlu_si/test/conceptual_physics/test/92 +gmmlu_si/test/econometrics/test/100 +gmmlu_si/test/econometrics/test/105 +gmmlu_si/test/econometrics/test/11 +gmmlu_si/test/econometrics/test/113 +gmmlu_si/test/econometrics/test/34 +gmmlu_si/test/econometrics/test/36 +gmmlu_si/test/econometrics/test/38 +gmmlu_si/test/econometrics/test/5 +gmmlu_si/test/econometrics/test/50 +gmmlu_si/test/econometrics/test/64 +gmmlu_si/test/econometrics/test/72 +gmmlu_si/test/econometrics/test/8 +gmmlu_si/test/econometrics/test/83 +gmmlu_si/test/econometrics/test/88 +gmmlu_si/test/econometrics/test/89 +gmmlu_si/test/econometrics/test/99 +gmmlu_si/test/electrical_engineering/test/114 +gmmlu_si/test/electrical_engineering/test/117 +gmmlu_si/test/electrical_engineering/test/125 +gmmlu_si/test/electrical_engineering/test/142 +gmmlu_si/test/electrical_engineering/test/15 +gmmlu_si/test/electrical_engineering/test/24 +gmmlu_si/test/electrical_engineering/test/3 +gmmlu_si/test/electrical_engineering/test/32 +gmmlu_si/test/electrical_engineering/test/35 +gmmlu_si/test/electrical_engineering/test/36 +gmmlu_si/test/electrical_engineering/test/37 +gmmlu_si/test/electrical_engineering/test/42 +gmmlu_si/test/electrical_engineering/test/44 +gmmlu_si/test/electrical_engineering/test/45 +gmmlu_si/test/electrical_engineering/test/46 +gmmlu_si/test/electrical_engineering/test/74 +gmmlu_si/test/electrical_engineering/test/86 +gmmlu_si/test/electrical_engineering/test/9 +gmmlu_si/test/electrical_engineering/test/91 +gmmlu_si/test/electrical_engineering/test/95 +gmmlu_si/test/electrical_engineering/test/99 +gmmlu_si/test/elementary_mathematics/test/106 +gmmlu_si/test/elementary_mathematics/test/110 +gmmlu_si/test/elementary_mathematics/test/115 +gmmlu_si/test/elementary_mathematics/test/118 +gmmlu_si/test/elementary_mathematics/test/129 +gmmlu_si/test/elementary_mathematics/test/13 +gmmlu_si/test/elementary_mathematics/test/136 +gmmlu_si/test/elementary_mathematics/test/158 +gmmlu_si/test/elementary_mathematics/test/161 +gmmlu_si/test/elementary_mathematics/test/162 +gmmlu_si/test/elementary_mathematics/test/165 +gmmlu_si/test/elementary_mathematics/test/17 +gmmlu_si/test/elementary_mathematics/test/173 +gmmlu_si/test/elementary_mathematics/test/174 +gmmlu_si/test/elementary_mathematics/test/194 +gmmlu_si/test/elementary_mathematics/test/2 +gmmlu_si/test/elementary_mathematics/test/203 +gmmlu_si/test/elementary_mathematics/test/21 +gmmlu_si/test/elementary_mathematics/test/216 +gmmlu_si/test/elementary_mathematics/test/218 +gmmlu_si/test/elementary_mathematics/test/239 +gmmlu_si/test/elementary_mathematics/test/242 +gmmlu_si/test/elementary_mathematics/test/249 +gmmlu_si/test/elementary_mathematics/test/250 +gmmlu_si/test/elementary_mathematics/test/259 +gmmlu_si/test/elementary_mathematics/test/261 +gmmlu_si/test/elementary_mathematics/test/27 +gmmlu_si/test/elementary_mathematics/test/28 +gmmlu_si/test/elementary_mathematics/test/280 +gmmlu_si/test/elementary_mathematics/test/299 +gmmlu_si/test/elementary_mathematics/test/318 +gmmlu_si/test/elementary_mathematics/test/321 +gmmlu_si/test/elementary_mathematics/test/328 +gmmlu_si/test/elementary_mathematics/test/330 +gmmlu_si/test/elementary_mathematics/test/331 +gmmlu_si/test/elementary_mathematics/test/335 +gmmlu_si/test/elementary_mathematics/test/336 +gmmlu_si/test/elementary_mathematics/test/343 +gmmlu_si/test/elementary_mathematics/test/357 +gmmlu_si/test/elementary_mathematics/test/358 +gmmlu_si/test/elementary_mathematics/test/365 +gmmlu_si/test/elementary_mathematics/test/368 +gmmlu_si/test/elementary_mathematics/test/44 +gmmlu_si/test/elementary_mathematics/test/45 +gmmlu_si/test/elementary_mathematics/test/6 +gmmlu_si/test/elementary_mathematics/test/65 +gmmlu_si/test/elementary_mathematics/test/66 +gmmlu_si/test/elementary_mathematics/test/7 +gmmlu_si/test/elementary_mathematics/test/73 +gmmlu_si/test/elementary_mathematics/test/74 +gmmlu_si/test/elementary_mathematics/test/76 +gmmlu_si/test/elementary_mathematics/test/77 +gmmlu_si/test/elementary_mathematics/test/80 +gmmlu_si/test/elementary_mathematics/test/85 +gmmlu_si/test/formal_logic/test/102 +gmmlu_si/test/formal_logic/test/105 +gmmlu_si/test/formal_logic/test/106 +gmmlu_si/test/formal_logic/test/21 +gmmlu_si/test/formal_logic/test/28 +gmmlu_si/test/formal_logic/test/32 +gmmlu_si/test/formal_logic/test/37 +gmmlu_si/test/formal_logic/test/47 +gmmlu_si/test/formal_logic/test/50 +gmmlu_si/test/formal_logic/test/54 +gmmlu_si/test/formal_logic/test/55 +gmmlu_si/test/formal_logic/test/56 +gmmlu_si/test/formal_logic/test/63 +gmmlu_si/test/formal_logic/test/73 +gmmlu_si/test/formal_logic/test/82 +gmmlu_si/test/formal_logic/test/83 +gmmlu_si/test/formal_logic/test/88 +gmmlu_si/test/formal_logic/test/93 +gmmlu_si/test/global_facts/test/11 +gmmlu_si/test/global_facts/test/14 +gmmlu_si/test/global_facts/test/18 +gmmlu_si/test/global_facts/test/31 +gmmlu_si/test/global_facts/test/35 +gmmlu_si/test/global_facts/test/49 +gmmlu_si/test/global_facts/test/52 +gmmlu_si/test/global_facts/test/60 +gmmlu_si/test/global_facts/test/66 +gmmlu_si/test/global_facts/test/69 +gmmlu_si/test/global_facts/test/77 +gmmlu_si/test/global_facts/test/80 +gmmlu_si/test/global_facts/test/89 +gmmlu_si/test/global_facts/test/98 +gmmlu_si/test/high_school_biology/test/101 +gmmlu_si/test/high_school_biology/test/106 +gmmlu_si/test/high_school_biology/test/111 +gmmlu_si/test/high_school_biology/test/114 +gmmlu_si/test/high_school_biology/test/120 +gmmlu_si/test/high_school_biology/test/132 +gmmlu_si/test/high_school_biology/test/153 +gmmlu_si/test/high_school_biology/test/159 +gmmlu_si/test/high_school_biology/test/163 +gmmlu_si/test/high_school_biology/test/166 +gmmlu_si/test/high_school_biology/test/179 +gmmlu_si/test/high_school_biology/test/189 +gmmlu_si/test/high_school_biology/test/196 +gmmlu_si/test/high_school_biology/test/199 +gmmlu_si/test/high_school_biology/test/205 +gmmlu_si/test/high_school_biology/test/209 +gmmlu_si/test/high_school_biology/test/215 +gmmlu_si/test/high_school_biology/test/220 +gmmlu_si/test/high_school_biology/test/23 +gmmlu_si/test/high_school_biology/test/230 +gmmlu_si/test/high_school_biology/test/233 +gmmlu_si/test/high_school_biology/test/242 +gmmlu_si/test/high_school_biology/test/255 +gmmlu_si/test/high_school_biology/test/262 +gmmlu_si/test/high_school_biology/test/265 +gmmlu_si/test/high_school_biology/test/268 +gmmlu_si/test/high_school_biology/test/27 +gmmlu_si/test/high_school_biology/test/281 +gmmlu_si/test/high_school_biology/test/286 +gmmlu_si/test/high_school_biology/test/29 +gmmlu_si/test/high_school_biology/test/295 +gmmlu_si/test/high_school_biology/test/35 +gmmlu_si/test/high_school_biology/test/43 +gmmlu_si/test/high_school_biology/test/50 +gmmlu_si/test/high_school_biology/test/66 +gmmlu_si/test/high_school_biology/test/70 +gmmlu_si/test/high_school_biology/test/72 +gmmlu_si/test/high_school_biology/test/83 +gmmlu_si/test/high_school_biology/test/89 +gmmlu_si/test/high_school_biology/test/92 +gmmlu_si/test/high_school_biology/test/93 +gmmlu_si/test/high_school_biology/test/96 +gmmlu_si/test/high_school_biology/test/97 +gmmlu_si/test/high_school_biology/test/99 +gmmlu_si/test/high_school_chemistry/test/0 +gmmlu_si/test/high_school_chemistry/test/137 +gmmlu_si/test/high_school_chemistry/test/138 +gmmlu_si/test/high_school_chemistry/test/139 +gmmlu_si/test/high_school_chemistry/test/140 +gmmlu_si/test/high_school_chemistry/test/142 +gmmlu_si/test/high_school_chemistry/test/153 +gmmlu_si/test/high_school_chemistry/test/160 +gmmlu_si/test/high_school_chemistry/test/165 +gmmlu_si/test/high_school_chemistry/test/172 +gmmlu_si/test/high_school_chemistry/test/18 +gmmlu_si/test/high_school_chemistry/test/181 +gmmlu_si/test/high_school_chemistry/test/183 +gmmlu_si/test/high_school_chemistry/test/192 +gmmlu_si/test/high_school_chemistry/test/20 +gmmlu_si/test/high_school_chemistry/test/202 +gmmlu_si/test/high_school_chemistry/test/28 +gmmlu_si/test/high_school_chemistry/test/38 +gmmlu_si/test/high_school_chemistry/test/4 +gmmlu_si/test/high_school_chemistry/test/41 +gmmlu_si/test/high_school_chemistry/test/51 +gmmlu_si/test/high_school_chemistry/test/54 +gmmlu_si/test/high_school_chemistry/test/64 +gmmlu_si/test/high_school_chemistry/test/76 +gmmlu_si/test/high_school_chemistry/test/77 +gmmlu_si/test/high_school_chemistry/test/78 +gmmlu_si/test/high_school_chemistry/test/80 +gmmlu_si/test/high_school_chemistry/test/84 +gmmlu_si/test/high_school_chemistry/test/93 +gmmlu_si/test/high_school_computer_science/test/20 +gmmlu_si/test/high_school_computer_science/test/51 +gmmlu_si/test/high_school_computer_science/test/55 +gmmlu_si/test/high_school_computer_science/test/6 +gmmlu_si/test/high_school_computer_science/test/64 +gmmlu_si/test/high_school_computer_science/test/68 +gmmlu_si/test/high_school_computer_science/test/7 +gmmlu_si/test/high_school_computer_science/test/76 +gmmlu_si/test/high_school_computer_science/test/80 +gmmlu_si/test/high_school_computer_science/test/87 +gmmlu_si/test/high_school_computer_science/test/91 +gmmlu_si/test/high_school_computer_science/test/92 +gmmlu_si/test/high_school_computer_science/test/93 +gmmlu_si/test/high_school_computer_science/test/94 +gmmlu_si/test/high_school_european_history/test/114 +gmmlu_si/test/high_school_european_history/test/12 +gmmlu_si/test/high_school_european_history/test/122 +gmmlu_si/test/high_school_european_history/test/127 +gmmlu_si/test/high_school_european_history/test/128 +gmmlu_si/test/high_school_european_history/test/141 +gmmlu_si/test/high_school_european_history/test/153 +gmmlu_si/test/high_school_european_history/test/157 +gmmlu_si/test/high_school_european_history/test/18 +gmmlu_si/test/high_school_european_history/test/23 +gmmlu_si/test/high_school_european_history/test/27 +gmmlu_si/test/high_school_european_history/test/31 +gmmlu_si/test/high_school_european_history/test/40 +gmmlu_si/test/high_school_european_history/test/43 +gmmlu_si/test/high_school_european_history/test/52 +gmmlu_si/test/high_school_european_history/test/58 +gmmlu_si/test/high_school_european_history/test/59 +gmmlu_si/test/high_school_european_history/test/6 +gmmlu_si/test/high_school_european_history/test/62 +gmmlu_si/test/high_school_european_history/test/70 +gmmlu_si/test/high_school_european_history/test/71 +gmmlu_si/test/high_school_european_history/test/76 +gmmlu_si/test/high_school_european_history/test/77 +gmmlu_si/test/high_school_european_history/test/79 +gmmlu_si/test/high_school_geography/test/104 +gmmlu_si/test/high_school_geography/test/108 +gmmlu_si/test/high_school_geography/test/110 +gmmlu_si/test/high_school_geography/test/115 +gmmlu_si/test/high_school_geography/test/117 +gmmlu_si/test/high_school_geography/test/120 +gmmlu_si/test/high_school_geography/test/122 +gmmlu_si/test/high_school_geography/test/134 +gmmlu_si/test/high_school_geography/test/138 +gmmlu_si/test/high_school_geography/test/139 +gmmlu_si/test/high_school_geography/test/143 +gmmlu_si/test/high_school_geography/test/144 +gmmlu_si/test/high_school_geography/test/158 +gmmlu_si/test/high_school_geography/test/159 +gmmlu_si/test/high_school_geography/test/160 +gmmlu_si/test/high_school_geography/test/170 +gmmlu_si/test/high_school_geography/test/172 +gmmlu_si/test/high_school_geography/test/175 +gmmlu_si/test/high_school_geography/test/177 +gmmlu_si/test/high_school_geography/test/185 +gmmlu_si/test/high_school_geography/test/196 +gmmlu_si/test/high_school_geography/test/27 +gmmlu_si/test/high_school_geography/test/45 +gmmlu_si/test/high_school_geography/test/6 +gmmlu_si/test/high_school_geography/test/60 +gmmlu_si/test/high_school_geography/test/73 +gmmlu_si/test/high_school_geography/test/77 +gmmlu_si/test/high_school_geography/test/80 +gmmlu_si/test/high_school_government_and_politics/test/10 +gmmlu_si/test/high_school_government_and_politics/test/133 +gmmlu_si/test/high_school_government_and_politics/test/136 +gmmlu_si/test/high_school_government_and_politics/test/142 +gmmlu_si/test/high_school_government_and_politics/test/155 +gmmlu_si/test/high_school_government_and_politics/test/163 +gmmlu_si/test/high_school_government_and_politics/test/165 +gmmlu_si/test/high_school_government_and_politics/test/169 +gmmlu_si/test/high_school_government_and_politics/test/171 +gmmlu_si/test/high_school_government_and_politics/test/176 +gmmlu_si/test/high_school_government_and_politics/test/185 +gmmlu_si/test/high_school_government_and_politics/test/19 +gmmlu_si/test/high_school_government_and_politics/test/20 +gmmlu_si/test/high_school_government_and_politics/test/25 +gmmlu_si/test/high_school_government_and_politics/test/27 +gmmlu_si/test/high_school_government_and_politics/test/29 +gmmlu_si/test/high_school_government_and_politics/test/40 +gmmlu_si/test/high_school_government_and_politics/test/45 +gmmlu_si/test/high_school_government_and_politics/test/58 +gmmlu_si/test/high_school_government_and_politics/test/61 +gmmlu_si/test/high_school_government_and_politics/test/62 +gmmlu_si/test/high_school_government_and_politics/test/65 +gmmlu_si/test/high_school_government_and_politics/test/66 +gmmlu_si/test/high_school_government_and_politics/test/82 +gmmlu_si/test/high_school_government_and_politics/test/85 +gmmlu_si/test/high_school_government_and_politics/test/89 +gmmlu_si/test/high_school_government_and_politics/test/9 +gmmlu_si/test/high_school_macroeconomics/test/109 +gmmlu_si/test/high_school_macroeconomics/test/11 +gmmlu_si/test/high_school_macroeconomics/test/113 +gmmlu_si/test/high_school_macroeconomics/test/127 +gmmlu_si/test/high_school_macroeconomics/test/13 +gmmlu_si/test/high_school_macroeconomics/test/130 +gmmlu_si/test/high_school_macroeconomics/test/161 +gmmlu_si/test/high_school_macroeconomics/test/164 +gmmlu_si/test/high_school_macroeconomics/test/166 +gmmlu_si/test/high_school_macroeconomics/test/170 +gmmlu_si/test/high_school_macroeconomics/test/173 +gmmlu_si/test/high_school_macroeconomics/test/18 +gmmlu_si/test/high_school_macroeconomics/test/181 +gmmlu_si/test/high_school_macroeconomics/test/193 +gmmlu_si/test/high_school_macroeconomics/test/205 +gmmlu_si/test/high_school_macroeconomics/test/207 +gmmlu_si/test/high_school_macroeconomics/test/210 +gmmlu_si/test/high_school_macroeconomics/test/211 +gmmlu_si/test/high_school_macroeconomics/test/217 +gmmlu_si/test/high_school_macroeconomics/test/221 +gmmlu_si/test/high_school_macroeconomics/test/23 +gmmlu_si/test/high_school_macroeconomics/test/237 +gmmlu_si/test/high_school_macroeconomics/test/238 +gmmlu_si/test/high_school_macroeconomics/test/243 +gmmlu_si/test/high_school_macroeconomics/test/249 +gmmlu_si/test/high_school_macroeconomics/test/261 +gmmlu_si/test/high_school_macroeconomics/test/271 +gmmlu_si/test/high_school_macroeconomics/test/273 +gmmlu_si/test/high_school_macroeconomics/test/274 +gmmlu_si/test/high_school_macroeconomics/test/279 +gmmlu_si/test/high_school_macroeconomics/test/285 +gmmlu_si/test/high_school_macroeconomics/test/288 +gmmlu_si/test/high_school_macroeconomics/test/296 +gmmlu_si/test/high_school_macroeconomics/test/297 +gmmlu_si/test/high_school_macroeconomics/test/299 +gmmlu_si/test/high_school_macroeconomics/test/302 +gmmlu_si/test/high_school_macroeconomics/test/306 +gmmlu_si/test/high_school_macroeconomics/test/321 +gmmlu_si/test/high_school_macroeconomics/test/325 +gmmlu_si/test/high_school_macroeconomics/test/329 +gmmlu_si/test/high_school_macroeconomics/test/334 +gmmlu_si/test/high_school_macroeconomics/test/337 +gmmlu_si/test/high_school_macroeconomics/test/341 +gmmlu_si/test/high_school_macroeconomics/test/349 +gmmlu_si/test/high_school_macroeconomics/test/35 +gmmlu_si/test/high_school_macroeconomics/test/351 +gmmlu_si/test/high_school_macroeconomics/test/363 +gmmlu_si/test/high_school_macroeconomics/test/364 +gmmlu_si/test/high_school_macroeconomics/test/388 +gmmlu_si/test/high_school_macroeconomics/test/45 +gmmlu_si/test/high_school_macroeconomics/test/51 +gmmlu_si/test/high_school_macroeconomics/test/62 +gmmlu_si/test/high_school_macroeconomics/test/64 +gmmlu_si/test/high_school_macroeconomics/test/67 +gmmlu_si/test/high_school_macroeconomics/test/77 +gmmlu_si/test/high_school_macroeconomics/test/79 +gmmlu_si/test/high_school_mathematics/test/113 +gmmlu_si/test/high_school_mathematics/test/128 +gmmlu_si/test/high_school_mathematics/test/131 +gmmlu_si/test/high_school_mathematics/test/133 +gmmlu_si/test/high_school_mathematics/test/141 +gmmlu_si/test/high_school_mathematics/test/144 +gmmlu_si/test/high_school_mathematics/test/147 +gmmlu_si/test/high_school_mathematics/test/148 +gmmlu_si/test/high_school_mathematics/test/155 +gmmlu_si/test/high_school_mathematics/test/16 +gmmlu_si/test/high_school_mathematics/test/161 +gmmlu_si/test/high_school_mathematics/test/175 +gmmlu_si/test/high_school_mathematics/test/177 +gmmlu_si/test/high_school_mathematics/test/189 +gmmlu_si/test/high_school_mathematics/test/203 +gmmlu_si/test/high_school_mathematics/test/218 +gmmlu_si/test/high_school_mathematics/test/220 +gmmlu_si/test/high_school_mathematics/test/226 +gmmlu_si/test/high_school_mathematics/test/227 +gmmlu_si/test/high_school_mathematics/test/237 +gmmlu_si/test/high_school_mathematics/test/243 +gmmlu_si/test/high_school_mathematics/test/252 +gmmlu_si/test/high_school_mathematics/test/254 +gmmlu_si/test/high_school_mathematics/test/265 +gmmlu_si/test/high_school_mathematics/test/3 +gmmlu_si/test/high_school_mathematics/test/43 +gmmlu_si/test/high_school_mathematics/test/46 +gmmlu_si/test/high_school_mathematics/test/47 +gmmlu_si/test/high_school_mathematics/test/54 +gmmlu_si/test/high_school_mathematics/test/58 +gmmlu_si/test/high_school_mathematics/test/6 +gmmlu_si/test/high_school_mathematics/test/68 +gmmlu_si/test/high_school_mathematics/test/73 +gmmlu_si/test/high_school_mathematics/test/75 +gmmlu_si/test/high_school_mathematics/test/77 +gmmlu_si/test/high_school_mathematics/test/81 +gmmlu_si/test/high_school_mathematics/test/90 +gmmlu_si/test/high_school_mathematics/test/95 +gmmlu_si/test/high_school_microeconomics/test/0 +gmmlu_si/test/high_school_microeconomics/test/10 +gmmlu_si/test/high_school_microeconomics/test/101 +gmmlu_si/test/high_school_microeconomics/test/141 +gmmlu_si/test/high_school_microeconomics/test/15 +gmmlu_si/test/high_school_microeconomics/test/152 +gmmlu_si/test/high_school_microeconomics/test/155 +gmmlu_si/test/high_school_microeconomics/test/174 +gmmlu_si/test/high_school_microeconomics/test/178 +gmmlu_si/test/high_school_microeconomics/test/181 +gmmlu_si/test/high_school_microeconomics/test/193 +gmmlu_si/test/high_school_microeconomics/test/194 +gmmlu_si/test/high_school_microeconomics/test/195 +gmmlu_si/test/high_school_microeconomics/test/196 +gmmlu_si/test/high_school_microeconomics/test/2 +gmmlu_si/test/high_school_microeconomics/test/212 +gmmlu_si/test/high_school_microeconomics/test/217 +gmmlu_si/test/high_school_microeconomics/test/225 +gmmlu_si/test/high_school_microeconomics/test/230 +gmmlu_si/test/high_school_microeconomics/test/232 +gmmlu_si/test/high_school_microeconomics/test/233 +gmmlu_si/test/high_school_microeconomics/test/27 +gmmlu_si/test/high_school_microeconomics/test/32 +gmmlu_si/test/high_school_microeconomics/test/34 +gmmlu_si/test/high_school_microeconomics/test/40 +gmmlu_si/test/high_school_microeconomics/test/41 +gmmlu_si/test/high_school_microeconomics/test/59 +gmmlu_si/test/high_school_microeconomics/test/60 +gmmlu_si/test/high_school_microeconomics/test/62 +gmmlu_si/test/high_school_microeconomics/test/73 +gmmlu_si/test/high_school_microeconomics/test/79 +gmmlu_si/test/high_school_microeconomics/test/88 +gmmlu_si/test/high_school_microeconomics/test/93 +gmmlu_si/test/high_school_microeconomics/test/95 +gmmlu_si/test/high_school_physics/test/101 +gmmlu_si/test/high_school_physics/test/102 +gmmlu_si/test/high_school_physics/test/107 +gmmlu_si/test/high_school_physics/test/114 +gmmlu_si/test/high_school_physics/test/123 +gmmlu_si/test/high_school_physics/test/126 +gmmlu_si/test/high_school_physics/test/138 +gmmlu_si/test/high_school_physics/test/139 +gmmlu_si/test/high_school_physics/test/146 +gmmlu_si/test/high_school_physics/test/19 +gmmlu_si/test/high_school_physics/test/3 +gmmlu_si/test/high_school_physics/test/33 +gmmlu_si/test/high_school_physics/test/48 +gmmlu_si/test/high_school_physics/test/53 +gmmlu_si/test/high_school_physics/test/54 +gmmlu_si/test/high_school_physics/test/58 +gmmlu_si/test/high_school_physics/test/6 +gmmlu_si/test/high_school_physics/test/62 +gmmlu_si/test/high_school_physics/test/67 +gmmlu_si/test/high_school_physics/test/71 +gmmlu_si/test/high_school_physics/test/75 +gmmlu_si/test/high_school_physics/test/93 +gmmlu_si/test/high_school_psychology/test/106 +gmmlu_si/test/high_school_psychology/test/107 +gmmlu_si/test/high_school_psychology/test/114 +gmmlu_si/test/high_school_psychology/test/124 +gmmlu_si/test/high_school_psychology/test/126 +gmmlu_si/test/high_school_psychology/test/128 +gmmlu_si/test/high_school_psychology/test/131 +gmmlu_si/test/high_school_psychology/test/132 +gmmlu_si/test/high_school_psychology/test/14 +gmmlu_si/test/high_school_psychology/test/153 +gmmlu_si/test/high_school_psychology/test/155 +gmmlu_si/test/high_school_psychology/test/156 +gmmlu_si/test/high_school_psychology/test/157 +gmmlu_si/test/high_school_psychology/test/165 +gmmlu_si/test/high_school_psychology/test/169 +gmmlu_si/test/high_school_psychology/test/171 +gmmlu_si/test/high_school_psychology/test/180 +gmmlu_si/test/high_school_psychology/test/185 +gmmlu_si/test/high_school_psychology/test/190 +gmmlu_si/test/high_school_psychology/test/197 +gmmlu_si/test/high_school_psychology/test/213 +gmmlu_si/test/high_school_psychology/test/215 +gmmlu_si/test/high_school_psychology/test/247 +gmmlu_si/test/high_school_psychology/test/248 +gmmlu_si/test/high_school_psychology/test/255 +gmmlu_si/test/high_school_psychology/test/26 +gmmlu_si/test/high_school_psychology/test/266 +gmmlu_si/test/high_school_psychology/test/267 +gmmlu_si/test/high_school_psychology/test/269 +gmmlu_si/test/high_school_psychology/test/273 +gmmlu_si/test/high_school_psychology/test/288 +gmmlu_si/test/high_school_psychology/test/31 +gmmlu_si/test/high_school_psychology/test/310 +gmmlu_si/test/high_school_psychology/test/322 +gmmlu_si/test/high_school_psychology/test/34 +gmmlu_si/test/high_school_psychology/test/340 +gmmlu_si/test/high_school_psychology/test/345 +gmmlu_si/test/high_school_psychology/test/351 +gmmlu_si/test/high_school_psychology/test/362 +gmmlu_si/test/high_school_psychology/test/364 +gmmlu_si/test/high_school_psychology/test/368 +gmmlu_si/test/high_school_psychology/test/37 +gmmlu_si/test/high_school_psychology/test/381 +gmmlu_si/test/high_school_psychology/test/386 +gmmlu_si/test/high_school_psychology/test/391 +gmmlu_si/test/high_school_psychology/test/399 +gmmlu_si/test/high_school_psychology/test/402 +gmmlu_si/test/high_school_psychology/test/403 +gmmlu_si/test/high_school_psychology/test/407 +gmmlu_si/test/high_school_psychology/test/410 +gmmlu_si/test/high_school_psychology/test/416 +gmmlu_si/test/high_school_psychology/test/423 +gmmlu_si/test/high_school_psychology/test/449 +gmmlu_si/test/high_school_psychology/test/463 +gmmlu_si/test/high_school_psychology/test/472 +gmmlu_si/test/high_school_psychology/test/475 +gmmlu_si/test/high_school_psychology/test/478 +gmmlu_si/test/high_school_psychology/test/483 +gmmlu_si/test/high_school_psychology/test/485 +gmmlu_si/test/high_school_psychology/test/49 +gmmlu_si/test/high_school_psychology/test/5 +gmmlu_si/test/high_school_psychology/test/505 +gmmlu_si/test/high_school_psychology/test/509 +gmmlu_si/test/high_school_psychology/test/513 +gmmlu_si/test/high_school_psychology/test/520 +gmmlu_si/test/high_school_psychology/test/541 +gmmlu_si/test/high_school_psychology/test/542 +gmmlu_si/test/high_school_psychology/test/6 +gmmlu_si/test/high_school_psychology/test/69 +gmmlu_si/test/high_school_psychology/test/70 +gmmlu_si/test/high_school_psychology/test/77 +gmmlu_si/test/high_school_psychology/test/80 +gmmlu_si/test/high_school_psychology/test/81 +gmmlu_si/test/high_school_psychology/test/85 +gmmlu_si/test/high_school_psychology/test/86 +gmmlu_si/test/high_school_psychology/test/9 +gmmlu_si/test/high_school_psychology/test/91 +gmmlu_si/test/high_school_psychology/test/95 +gmmlu_si/test/high_school_statistics/test/105 +gmmlu_si/test/high_school_statistics/test/107 +gmmlu_si/test/high_school_statistics/test/118 +gmmlu_si/test/high_school_statistics/test/122 +gmmlu_si/test/high_school_statistics/test/139 +gmmlu_si/test/high_school_statistics/test/147 +gmmlu_si/test/high_school_statistics/test/148 +gmmlu_si/test/high_school_statistics/test/16 +gmmlu_si/test/high_school_statistics/test/17 +gmmlu_si/test/high_school_statistics/test/176 +gmmlu_si/test/high_school_statistics/test/177 +gmmlu_si/test/high_school_statistics/test/190 +gmmlu_si/test/high_school_statistics/test/200 +gmmlu_si/test/high_school_statistics/test/213 +gmmlu_si/test/high_school_statistics/test/23 +gmmlu_si/test/high_school_statistics/test/29 +gmmlu_si/test/high_school_statistics/test/31 +gmmlu_si/test/high_school_statistics/test/36 +gmmlu_si/test/high_school_statistics/test/38 +gmmlu_si/test/high_school_statistics/test/40 +gmmlu_si/test/high_school_statistics/test/45 +gmmlu_si/test/high_school_statistics/test/47 +gmmlu_si/test/high_school_statistics/test/52 +gmmlu_si/test/high_school_statistics/test/57 +gmmlu_si/test/high_school_statistics/test/58 +gmmlu_si/test/high_school_statistics/test/6 +gmmlu_si/test/high_school_statistics/test/66 +gmmlu_si/test/high_school_statistics/test/79 +gmmlu_si/test/high_school_statistics/test/84 +gmmlu_si/test/high_school_statistics/test/9 +gmmlu_si/test/high_school_statistics/test/90 +gmmlu_si/test/high_school_us_history/test/101 +gmmlu_si/test/high_school_us_history/test/104 +gmmlu_si/test/high_school_us_history/test/113 +gmmlu_si/test/high_school_us_history/test/116 +gmmlu_si/test/high_school_us_history/test/117 +gmmlu_si/test/high_school_us_history/test/118 +gmmlu_si/test/high_school_us_history/test/120 +gmmlu_si/test/high_school_us_history/test/148 +gmmlu_si/test/high_school_us_history/test/154 +gmmlu_si/test/high_school_us_history/test/167 +gmmlu_si/test/high_school_us_history/test/175 +gmmlu_si/test/high_school_us_history/test/178 +gmmlu_si/test/high_school_us_history/test/179 +gmmlu_si/test/high_school_us_history/test/18 +gmmlu_si/test/high_school_us_history/test/188 +gmmlu_si/test/high_school_us_history/test/189 +gmmlu_si/test/high_school_us_history/test/198 +gmmlu_si/test/high_school_us_history/test/202 +gmmlu_si/test/high_school_us_history/test/43 +gmmlu_si/test/high_school_us_history/test/56 +gmmlu_si/test/high_school_us_history/test/58 +gmmlu_si/test/high_school_us_history/test/60 +gmmlu_si/test/high_school_us_history/test/64 +gmmlu_si/test/high_school_us_history/test/72 +gmmlu_si/test/high_school_us_history/test/89 +gmmlu_si/test/high_school_us_history/test/93 +gmmlu_si/test/high_school_us_history/test/94 +gmmlu_si/test/high_school_us_history/test/95 +gmmlu_si/test/high_school_us_history/test/96 +gmmlu_si/test/high_school_world_history/test/101 +gmmlu_si/test/high_school_world_history/test/104 +gmmlu_si/test/high_school_world_history/test/105 +gmmlu_si/test/high_school_world_history/test/111 +gmmlu_si/test/high_school_world_history/test/117 +gmmlu_si/test/high_school_world_history/test/12 +gmmlu_si/test/high_school_world_history/test/127 +gmmlu_si/test/high_school_world_history/test/128 +gmmlu_si/test/high_school_world_history/test/129 +gmmlu_si/test/high_school_world_history/test/134 +gmmlu_si/test/high_school_world_history/test/142 +gmmlu_si/test/high_school_world_history/test/16 +gmmlu_si/test/high_school_world_history/test/170 +gmmlu_si/test/high_school_world_history/test/176 +gmmlu_si/test/high_school_world_history/test/18 +gmmlu_si/test/high_school_world_history/test/180 +gmmlu_si/test/high_school_world_history/test/181 +gmmlu_si/test/high_school_world_history/test/195 +gmmlu_si/test/high_school_world_history/test/199 +gmmlu_si/test/high_school_world_history/test/201 +gmmlu_si/test/high_school_world_history/test/211 +gmmlu_si/test/high_school_world_history/test/213 +gmmlu_si/test/high_school_world_history/test/227 +gmmlu_si/test/high_school_world_history/test/229 +gmmlu_si/test/high_school_world_history/test/236 +gmmlu_si/test/high_school_world_history/test/27 +gmmlu_si/test/high_school_world_history/test/28 +gmmlu_si/test/high_school_world_history/test/40 +gmmlu_si/test/high_school_world_history/test/46 +gmmlu_si/test/high_school_world_history/test/48 +gmmlu_si/test/high_school_world_history/test/57 +gmmlu_si/test/high_school_world_history/test/59 +gmmlu_si/test/high_school_world_history/test/68 +gmmlu_si/test/high_school_world_history/test/97 +gmmlu_si/test/human_aging/test/128 +gmmlu_si/test/human_aging/test/13 +gmmlu_si/test/human_aging/test/130 +gmmlu_si/test/human_aging/test/136 +gmmlu_si/test/human_aging/test/137 +gmmlu_si/test/human_aging/test/150 +gmmlu_si/test/human_aging/test/158 +gmmlu_si/test/human_aging/test/161 +gmmlu_si/test/human_aging/test/169 +gmmlu_si/test/human_aging/test/180 +gmmlu_si/test/human_aging/test/186 +gmmlu_si/test/human_aging/test/187 +gmmlu_si/test/human_aging/test/196 +gmmlu_si/test/human_aging/test/198 +gmmlu_si/test/human_aging/test/207 +gmmlu_si/test/human_aging/test/212 +gmmlu_si/test/human_aging/test/213 +gmmlu_si/test/human_aging/test/222 +gmmlu_si/test/human_aging/test/29 +gmmlu_si/test/human_aging/test/30 +gmmlu_si/test/human_aging/test/41 +gmmlu_si/test/human_aging/test/42 +gmmlu_si/test/human_aging/test/49 +gmmlu_si/test/human_aging/test/50 +gmmlu_si/test/human_aging/test/51 +gmmlu_si/test/human_aging/test/59 +gmmlu_si/test/human_aging/test/6 +gmmlu_si/test/human_aging/test/61 +gmmlu_si/test/human_aging/test/63 +gmmlu_si/test/human_aging/test/76 +gmmlu_si/test/human_aging/test/95 +gmmlu_si/test/human_aging/test/96 +gmmlu_si/test/human_sexuality/test/10 +gmmlu_si/test/human_sexuality/test/106 +gmmlu_si/test/human_sexuality/test/109 +gmmlu_si/test/human_sexuality/test/110 +gmmlu_si/test/human_sexuality/test/16 +gmmlu_si/test/human_sexuality/test/19 +gmmlu_si/test/human_sexuality/test/24 +gmmlu_si/test/human_sexuality/test/26 +gmmlu_si/test/human_sexuality/test/38 +gmmlu_si/test/human_sexuality/test/40 +gmmlu_si/test/human_sexuality/test/49 +gmmlu_si/test/human_sexuality/test/51 +gmmlu_si/test/human_sexuality/test/52 +gmmlu_si/test/human_sexuality/test/61 +gmmlu_si/test/human_sexuality/test/66 +gmmlu_si/test/human_sexuality/test/78 +gmmlu_si/test/human_sexuality/test/86 +gmmlu_si/test/human_sexuality/test/90 +gmmlu_si/test/human_sexuality/test/98 +gmmlu_si/test/international_law/test/104 +gmmlu_si/test/international_law/test/20 +gmmlu_si/test/international_law/test/22 +gmmlu_si/test/international_law/test/26 +gmmlu_si/test/international_law/test/34 +gmmlu_si/test/international_law/test/38 +gmmlu_si/test/international_law/test/41 +gmmlu_si/test/international_law/test/5 +gmmlu_si/test/international_law/test/54 +gmmlu_si/test/international_law/test/65 +gmmlu_si/test/international_law/test/68 +gmmlu_si/test/international_law/test/82 +gmmlu_si/test/international_law/test/85 +gmmlu_si/test/international_law/test/91 +gmmlu_si/test/international_law/test/95 +gmmlu_si/test/international_law/test/97 +gmmlu_si/test/international_law/test/98 +gmmlu_si/test/jurisprudence/test/12 +gmmlu_si/test/jurisprudence/test/23 +gmmlu_si/test/jurisprudence/test/30 +gmmlu_si/test/jurisprudence/test/35 +gmmlu_si/test/jurisprudence/test/38 +gmmlu_si/test/jurisprudence/test/4 +gmmlu_si/test/jurisprudence/test/43 +gmmlu_si/test/jurisprudence/test/46 +gmmlu_si/test/jurisprudence/test/48 +gmmlu_si/test/jurisprudence/test/50 +gmmlu_si/test/jurisprudence/test/71 +gmmlu_si/test/jurisprudence/test/72 +gmmlu_si/test/jurisprudence/test/73 +gmmlu_si/test/jurisprudence/test/74 +gmmlu_si/test/jurisprudence/test/93 +gmmlu_si/test/logical_fallacies/test/111 +gmmlu_si/test/logical_fallacies/test/128 +gmmlu_si/test/logical_fallacies/test/137 +gmmlu_si/test/logical_fallacies/test/14 +gmmlu_si/test/logical_fallacies/test/144 +gmmlu_si/test/logical_fallacies/test/15 +gmmlu_si/test/logical_fallacies/test/152 +gmmlu_si/test/logical_fallacies/test/157 +gmmlu_si/test/logical_fallacies/test/17 +gmmlu_si/test/logical_fallacies/test/21 +gmmlu_si/test/logical_fallacies/test/34 +gmmlu_si/test/logical_fallacies/test/38 +gmmlu_si/test/logical_fallacies/test/39 +gmmlu_si/test/logical_fallacies/test/46 +gmmlu_si/test/logical_fallacies/test/47 +gmmlu_si/test/logical_fallacies/test/57 +gmmlu_si/test/logical_fallacies/test/7 +gmmlu_si/test/logical_fallacies/test/83 +gmmlu_si/test/logical_fallacies/test/88 +gmmlu_si/test/logical_fallacies/test/92 +gmmlu_si/test/logical_fallacies/test/93 +gmmlu_si/test/logical_fallacies/test/94 +gmmlu_si/test/logical_fallacies/test/97 +gmmlu_si/test/machine_learning/test/111 +gmmlu_si/test/machine_learning/test/17 +gmmlu_si/test/machine_learning/test/18 +gmmlu_si/test/machine_learning/test/27 +gmmlu_si/test/machine_learning/test/34 +gmmlu_si/test/machine_learning/test/35 +gmmlu_si/test/machine_learning/test/44 +gmmlu_si/test/machine_learning/test/49 +gmmlu_si/test/machine_learning/test/5 +gmmlu_si/test/machine_learning/test/62 +gmmlu_si/test/machine_learning/test/70 +gmmlu_si/test/machine_learning/test/74 +gmmlu_si/test/machine_learning/test/81 +gmmlu_si/test/machine_learning/test/84 +gmmlu_si/test/machine_learning/test/91 +gmmlu_si/test/machine_learning/test/93 +gmmlu_si/test/management/test/11 +gmmlu_si/test/management/test/12 +gmmlu_si/test/management/test/16 +gmmlu_si/test/management/test/25 +gmmlu_si/test/management/test/36 +gmmlu_si/test/management/test/38 +gmmlu_si/test/management/test/40 +gmmlu_si/test/management/test/58 +gmmlu_si/test/management/test/59 +gmmlu_si/test/management/test/62 +gmmlu_si/test/management/test/66 +gmmlu_si/test/management/test/8 +gmmlu_si/test/management/test/83 +gmmlu_si/test/management/test/86 +gmmlu_si/test/management/test/94 +gmmlu_si/test/marketing/test/11 +gmmlu_si/test/marketing/test/114 +gmmlu_si/test/marketing/test/12 +gmmlu_si/test/marketing/test/124 +gmmlu_si/test/marketing/test/128 +gmmlu_si/test/marketing/test/133 +gmmlu_si/test/marketing/test/134 +gmmlu_si/test/marketing/test/14 +gmmlu_si/test/marketing/test/141 +gmmlu_si/test/marketing/test/147 +gmmlu_si/test/marketing/test/153 +gmmlu_si/test/marketing/test/165 +gmmlu_si/test/marketing/test/184 +gmmlu_si/test/marketing/test/186 +gmmlu_si/test/marketing/test/189 +gmmlu_si/test/marketing/test/193 +gmmlu_si/test/marketing/test/199 +gmmlu_si/test/marketing/test/206 +gmmlu_si/test/marketing/test/208 +gmmlu_si/test/marketing/test/221 +gmmlu_si/test/marketing/test/230 +gmmlu_si/test/marketing/test/231 +gmmlu_si/test/marketing/test/28 +gmmlu_si/test/marketing/test/30 +gmmlu_si/test/marketing/test/61 +gmmlu_si/test/marketing/test/65 +gmmlu_si/test/marketing/test/67 +gmmlu_si/test/marketing/test/8 +gmmlu_si/test/marketing/test/83 +gmmlu_si/test/marketing/test/84 +gmmlu_si/test/marketing/test/88 +gmmlu_si/test/marketing/test/89 +gmmlu_si/test/marketing/test/93 +gmmlu_si/test/medical_genetics/test/10 +gmmlu_si/test/medical_genetics/test/18 +gmmlu_si/test/medical_genetics/test/2 +gmmlu_si/test/medical_genetics/test/20 +gmmlu_si/test/medical_genetics/test/22 +gmmlu_si/test/medical_genetics/test/23 +gmmlu_si/test/medical_genetics/test/24 +gmmlu_si/test/medical_genetics/test/25 +gmmlu_si/test/medical_genetics/test/32 +gmmlu_si/test/medical_genetics/test/45 +gmmlu_si/test/medical_genetics/test/51 +gmmlu_si/test/medical_genetics/test/63 +gmmlu_si/test/medical_genetics/test/73 +gmmlu_si/test/medical_genetics/test/99 +gmmlu_si/test/miscellaneous/test/0 +gmmlu_si/test/miscellaneous/test/103 +gmmlu_si/test/miscellaneous/test/115 +gmmlu_si/test/miscellaneous/test/117 +gmmlu_si/test/miscellaneous/test/123 +gmmlu_si/test/miscellaneous/test/132 +gmmlu_si/test/miscellaneous/test/134 +gmmlu_si/test/miscellaneous/test/135 +gmmlu_si/test/miscellaneous/test/137 +gmmlu_si/test/miscellaneous/test/142 +gmmlu_si/test/miscellaneous/test/153 +gmmlu_si/test/miscellaneous/test/156 +gmmlu_si/test/miscellaneous/test/157 +gmmlu_si/test/miscellaneous/test/168 +gmmlu_si/test/miscellaneous/test/172 +gmmlu_si/test/miscellaneous/test/196 +gmmlu_si/test/miscellaneous/test/209 +gmmlu_si/test/miscellaneous/test/210 +gmmlu_si/test/miscellaneous/test/223 +gmmlu_si/test/miscellaneous/test/251 +gmmlu_si/test/miscellaneous/test/269 +gmmlu_si/test/miscellaneous/test/287 +gmmlu_si/test/miscellaneous/test/290 +gmmlu_si/test/miscellaneous/test/293 +gmmlu_si/test/miscellaneous/test/300 +gmmlu_si/test/miscellaneous/test/302 +gmmlu_si/test/miscellaneous/test/31 +gmmlu_si/test/miscellaneous/test/311 +gmmlu_si/test/miscellaneous/test/333 +gmmlu_si/test/miscellaneous/test/334 +gmmlu_si/test/miscellaneous/test/345 +gmmlu_si/test/miscellaneous/test/354 +gmmlu_si/test/miscellaneous/test/367 +gmmlu_si/test/miscellaneous/test/37 +gmmlu_si/test/miscellaneous/test/373 +gmmlu_si/test/miscellaneous/test/375 +gmmlu_si/test/miscellaneous/test/38 +gmmlu_si/test/miscellaneous/test/388 +gmmlu_si/test/miscellaneous/test/390 +gmmlu_si/test/miscellaneous/test/397 +gmmlu_si/test/miscellaneous/test/399 +gmmlu_si/test/miscellaneous/test/400 +gmmlu_si/test/miscellaneous/test/404 +gmmlu_si/test/miscellaneous/test/413 +gmmlu_si/test/miscellaneous/test/414 +gmmlu_si/test/miscellaneous/test/43 +gmmlu_si/test/miscellaneous/test/430 +gmmlu_si/test/miscellaneous/test/433 +gmmlu_si/test/miscellaneous/test/441 +gmmlu_si/test/miscellaneous/test/443 +gmmlu_si/test/miscellaneous/test/445 +gmmlu_si/test/miscellaneous/test/447 +gmmlu_si/test/miscellaneous/test/453 +gmmlu_si/test/miscellaneous/test/460 +gmmlu_si/test/miscellaneous/test/469 +gmmlu_si/test/miscellaneous/test/473 +gmmlu_si/test/miscellaneous/test/482 +gmmlu_si/test/miscellaneous/test/483 +gmmlu_si/test/miscellaneous/test/489 +gmmlu_si/test/miscellaneous/test/49 +gmmlu_si/test/miscellaneous/test/500 +gmmlu_si/test/miscellaneous/test/511 +gmmlu_si/test/miscellaneous/test/517 +gmmlu_si/test/miscellaneous/test/519 +gmmlu_si/test/miscellaneous/test/536 +gmmlu_si/test/miscellaneous/test/542 +gmmlu_si/test/miscellaneous/test/55 +gmmlu_si/test/miscellaneous/test/553 +gmmlu_si/test/miscellaneous/test/565 +gmmlu_si/test/miscellaneous/test/571 +gmmlu_si/test/miscellaneous/test/577 +gmmlu_si/test/miscellaneous/test/583 +gmmlu_si/test/miscellaneous/test/586 +gmmlu_si/test/miscellaneous/test/587 +gmmlu_si/test/miscellaneous/test/593 +gmmlu_si/test/miscellaneous/test/597 +gmmlu_si/test/miscellaneous/test/61 +gmmlu_si/test/miscellaneous/test/618 +gmmlu_si/test/miscellaneous/test/635 +gmmlu_si/test/miscellaneous/test/641 +gmmlu_si/test/miscellaneous/test/66 +gmmlu_si/test/miscellaneous/test/661 +gmmlu_si/test/miscellaneous/test/662 +gmmlu_si/test/miscellaneous/test/663 +gmmlu_si/test/miscellaneous/test/67 +gmmlu_si/test/miscellaneous/test/676 +gmmlu_si/test/miscellaneous/test/68 +gmmlu_si/test/miscellaneous/test/683 +gmmlu_si/test/miscellaneous/test/684 +gmmlu_si/test/miscellaneous/test/685 +gmmlu_si/test/miscellaneous/test/687 +gmmlu_si/test/miscellaneous/test/696 +gmmlu_si/test/miscellaneous/test/697 +gmmlu_si/test/miscellaneous/test/705 +gmmlu_si/test/miscellaneous/test/708 +gmmlu_si/test/miscellaneous/test/710 +gmmlu_si/test/miscellaneous/test/719 +gmmlu_si/test/miscellaneous/test/728 +gmmlu_si/test/miscellaneous/test/73 +gmmlu_si/test/miscellaneous/test/730 +gmmlu_si/test/miscellaneous/test/737 +gmmlu_si/test/miscellaneous/test/738 +gmmlu_si/test/miscellaneous/test/74 +gmmlu_si/test/miscellaneous/test/742 +gmmlu_si/test/miscellaneous/test/753 +gmmlu_si/test/miscellaneous/test/76 +gmmlu_si/test/miscellaneous/test/770 +gmmlu_si/test/miscellaneous/test/774 +gmmlu_si/test/miscellaneous/test/782 +gmmlu_si/test/miscellaneous/test/89 +gmmlu_si/test/miscellaneous/test/93 +gmmlu_si/test/miscellaneous/test/98 +gmmlu_si/test/moral_disputes/test/100 +gmmlu_si/test/moral_disputes/test/101 +gmmlu_si/test/moral_disputes/test/103 +gmmlu_si/test/moral_disputes/test/104 +gmmlu_si/test/moral_disputes/test/11 +gmmlu_si/test/moral_disputes/test/111 +gmmlu_si/test/moral_disputes/test/112 +gmmlu_si/test/moral_disputes/test/115 +gmmlu_si/test/moral_disputes/test/122 +gmmlu_si/test/moral_disputes/test/135 +gmmlu_si/test/moral_disputes/test/139 +gmmlu_si/test/moral_disputes/test/140 +gmmlu_si/test/moral_disputes/test/141 +gmmlu_si/test/moral_disputes/test/142 +gmmlu_si/test/moral_disputes/test/145 +gmmlu_si/test/moral_disputes/test/151 +gmmlu_si/test/moral_disputes/test/164 +gmmlu_si/test/moral_disputes/test/175 +gmmlu_si/test/moral_disputes/test/185 +gmmlu_si/test/moral_disputes/test/190 +gmmlu_si/test/moral_disputes/test/191 +gmmlu_si/test/moral_disputes/test/195 +gmmlu_si/test/moral_disputes/test/202 +gmmlu_si/test/moral_disputes/test/206 +gmmlu_si/test/moral_disputes/test/219 +gmmlu_si/test/moral_disputes/test/223 +gmmlu_si/test/moral_disputes/test/254 +gmmlu_si/test/moral_disputes/test/275 +gmmlu_si/test/moral_disputes/test/282 +gmmlu_si/test/moral_disputes/test/286 +gmmlu_si/test/moral_disputes/test/288 +gmmlu_si/test/moral_disputes/test/293 +gmmlu_si/test/moral_disputes/test/298 +gmmlu_si/test/moral_disputes/test/303 +gmmlu_si/test/moral_disputes/test/306 +gmmlu_si/test/moral_disputes/test/313 +gmmlu_si/test/moral_disputes/test/342 +gmmlu_si/test/moral_disputes/test/344 +gmmlu_si/test/moral_disputes/test/4 +gmmlu_si/test/moral_disputes/test/55 +gmmlu_si/test/moral_disputes/test/57 +gmmlu_si/test/moral_disputes/test/62 +gmmlu_si/test/moral_disputes/test/65 +gmmlu_si/test/moral_disputes/test/67 +gmmlu_si/test/moral_disputes/test/77 +gmmlu_si/test/moral_disputes/test/81 +gmmlu_si/test/moral_disputes/test/83 +gmmlu_si/test/moral_disputes/test/86 +gmmlu_si/test/moral_disputes/test/89 +gmmlu_si/test/moral_scenarios/test/122 +gmmlu_si/test/moral_scenarios/test/127 +gmmlu_si/test/moral_scenarios/test/135 +gmmlu_si/test/moral_scenarios/test/136 +gmmlu_si/test/moral_scenarios/test/140 +gmmlu_si/test/moral_scenarios/test/149 +gmmlu_si/test/moral_scenarios/test/151 +gmmlu_si/test/moral_scenarios/test/155 +gmmlu_si/test/moral_scenarios/test/162 +gmmlu_si/test/moral_scenarios/test/169 +gmmlu_si/test/moral_scenarios/test/180 +gmmlu_si/test/moral_scenarios/test/181 +gmmlu_si/test/moral_scenarios/test/186 +gmmlu_si/test/moral_scenarios/test/187 +gmmlu_si/test/moral_scenarios/test/191 +gmmlu_si/test/moral_scenarios/test/195 +gmmlu_si/test/moral_scenarios/test/197 +gmmlu_si/test/moral_scenarios/test/20 +gmmlu_si/test/moral_scenarios/test/227 +gmmlu_si/test/moral_scenarios/test/231 +gmmlu_si/test/moral_scenarios/test/245 +gmmlu_si/test/moral_scenarios/test/264 +gmmlu_si/test/moral_scenarios/test/269 +gmmlu_si/test/moral_scenarios/test/279 +gmmlu_si/test/moral_scenarios/test/28 +gmmlu_si/test/moral_scenarios/test/283 +gmmlu_si/test/moral_scenarios/test/290 +gmmlu_si/test/moral_scenarios/test/304 +gmmlu_si/test/moral_scenarios/test/326 +gmmlu_si/test/moral_scenarios/test/341 +gmmlu_si/test/moral_scenarios/test/345 +gmmlu_si/test/moral_scenarios/test/346 +gmmlu_si/test/moral_scenarios/test/349 +gmmlu_si/test/moral_scenarios/test/350 +gmmlu_si/test/moral_scenarios/test/355 +gmmlu_si/test/moral_scenarios/test/362 +gmmlu_si/test/moral_scenarios/test/367 +gmmlu_si/test/moral_scenarios/test/373 +gmmlu_si/test/moral_scenarios/test/396 +gmmlu_si/test/moral_scenarios/test/398 +gmmlu_si/test/moral_scenarios/test/4 +gmmlu_si/test/moral_scenarios/test/400 +gmmlu_si/test/moral_scenarios/test/402 +gmmlu_si/test/moral_scenarios/test/409 +gmmlu_si/test/moral_scenarios/test/413 +gmmlu_si/test/moral_scenarios/test/42 +gmmlu_si/test/moral_scenarios/test/43 +gmmlu_si/test/moral_scenarios/test/432 +gmmlu_si/test/moral_scenarios/test/433 +gmmlu_si/test/moral_scenarios/test/436 +gmmlu_si/test/moral_scenarios/test/464 +gmmlu_si/test/moral_scenarios/test/466 +gmmlu_si/test/moral_scenarios/test/47 +gmmlu_si/test/moral_scenarios/test/476 +gmmlu_si/test/moral_scenarios/test/477 +gmmlu_si/test/moral_scenarios/test/484 +gmmlu_si/test/moral_scenarios/test/490 +gmmlu_si/test/moral_scenarios/test/494 +gmmlu_si/test/moral_scenarios/test/507 +gmmlu_si/test/moral_scenarios/test/508 +gmmlu_si/test/moral_scenarios/test/514 +gmmlu_si/test/moral_scenarios/test/518 +gmmlu_si/test/moral_scenarios/test/519 +gmmlu_si/test/moral_scenarios/test/523 +gmmlu_si/test/moral_scenarios/test/528 +gmmlu_si/test/moral_scenarios/test/53 +gmmlu_si/test/moral_scenarios/test/539 +gmmlu_si/test/moral_scenarios/test/543 +gmmlu_si/test/moral_scenarios/test/548 +gmmlu_si/test/moral_scenarios/test/550 +gmmlu_si/test/moral_scenarios/test/552 +gmmlu_si/test/moral_scenarios/test/562 +gmmlu_si/test/moral_scenarios/test/565 +gmmlu_si/test/moral_scenarios/test/57 +gmmlu_si/test/moral_scenarios/test/570 +gmmlu_si/test/moral_scenarios/test/572 +gmmlu_si/test/moral_scenarios/test/578 +gmmlu_si/test/moral_scenarios/test/584 +gmmlu_si/test/moral_scenarios/test/588 +gmmlu_si/test/moral_scenarios/test/596 +gmmlu_si/test/moral_scenarios/test/6 +gmmlu_si/test/moral_scenarios/test/60 +gmmlu_si/test/moral_scenarios/test/606 +gmmlu_si/test/moral_scenarios/test/612 +gmmlu_si/test/moral_scenarios/test/62 +gmmlu_si/test/moral_scenarios/test/626 +gmmlu_si/test/moral_scenarios/test/631 +gmmlu_si/test/moral_scenarios/test/632 +gmmlu_si/test/moral_scenarios/test/640 +gmmlu_si/test/moral_scenarios/test/641 +gmmlu_si/test/moral_scenarios/test/646 +gmmlu_si/test/moral_scenarios/test/65 +gmmlu_si/test/moral_scenarios/test/682 +gmmlu_si/test/moral_scenarios/test/693 +gmmlu_si/test/moral_scenarios/test/699 +gmmlu_si/test/moral_scenarios/test/7 +gmmlu_si/test/moral_scenarios/test/709 +gmmlu_si/test/moral_scenarios/test/718 +gmmlu_si/test/moral_scenarios/test/732 +gmmlu_si/test/moral_scenarios/test/747 +gmmlu_si/test/moral_scenarios/test/756 +gmmlu_si/test/moral_scenarios/test/759 +gmmlu_si/test/moral_scenarios/test/76 +gmmlu_si/test/moral_scenarios/test/761 +gmmlu_si/test/moral_scenarios/test/775 +gmmlu_si/test/moral_scenarios/test/789 +gmmlu_si/test/moral_scenarios/test/793 +gmmlu_si/test/moral_scenarios/test/794 +gmmlu_si/test/moral_scenarios/test/802 +gmmlu_si/test/moral_scenarios/test/808 +gmmlu_si/test/moral_scenarios/test/814 +gmmlu_si/test/moral_scenarios/test/82 +gmmlu_si/test/moral_scenarios/test/827 +gmmlu_si/test/moral_scenarios/test/831 +gmmlu_si/test/moral_scenarios/test/834 +gmmlu_si/test/moral_scenarios/test/848 +gmmlu_si/test/moral_scenarios/test/851 +gmmlu_si/test/moral_scenarios/test/855 +gmmlu_si/test/moral_scenarios/test/856 +gmmlu_si/test/moral_scenarios/test/857 +gmmlu_si/test/moral_scenarios/test/87 +gmmlu_si/test/moral_scenarios/test/884 +gmmlu_si/test/moral_scenarios/test/887 +gmmlu_si/test/moral_scenarios/test/888 +gmmlu_si/test/moral_scenarios/test/894 +gmmlu_si/test/moral_scenarios/test/90 +gmmlu_si/test/moral_scenarios/test/97 +gmmlu_si/test/nutrition/test/104 +gmmlu_si/test/nutrition/test/113 +gmmlu_si/test/nutrition/test/126 +gmmlu_si/test/nutrition/test/131 +gmmlu_si/test/nutrition/test/136 +gmmlu_si/test/nutrition/test/145 +gmmlu_si/test/nutrition/test/15 +gmmlu_si/test/nutrition/test/152 +gmmlu_si/test/nutrition/test/155 +gmmlu_si/test/nutrition/test/160 +gmmlu_si/test/nutrition/test/168 +gmmlu_si/test/nutrition/test/183 +gmmlu_si/test/nutrition/test/186 +gmmlu_si/test/nutrition/test/188 +gmmlu_si/test/nutrition/test/191 +gmmlu_si/test/nutrition/test/200 +gmmlu_si/test/nutrition/test/209 +gmmlu_si/test/nutrition/test/210 +gmmlu_si/test/nutrition/test/211 +gmmlu_si/test/nutrition/test/217 +gmmlu_si/test/nutrition/test/225 +gmmlu_si/test/nutrition/test/228 +gmmlu_si/test/nutrition/test/232 +gmmlu_si/test/nutrition/test/237 +gmmlu_si/test/nutrition/test/238 +gmmlu_si/test/nutrition/test/24 +gmmlu_si/test/nutrition/test/248 +gmmlu_si/test/nutrition/test/277 +gmmlu_si/test/nutrition/test/279 +gmmlu_si/test/nutrition/test/281 +gmmlu_si/test/nutrition/test/282 +gmmlu_si/test/nutrition/test/291 +gmmlu_si/test/nutrition/test/303 +gmmlu_si/test/nutrition/test/49 +gmmlu_si/test/nutrition/test/52 +gmmlu_si/test/nutrition/test/53 +gmmlu_si/test/nutrition/test/59 +gmmlu_si/test/nutrition/test/62 +gmmlu_si/test/nutrition/test/65 +gmmlu_si/test/nutrition/test/67 +gmmlu_si/test/nutrition/test/68 +gmmlu_si/test/nutrition/test/72 +gmmlu_si/test/nutrition/test/90 +gmmlu_si/test/nutrition/test/95 +gmmlu_si/test/philosophy/test/126 +gmmlu_si/test/philosophy/test/135 +gmmlu_si/test/philosophy/test/15 +gmmlu_si/test/philosophy/test/158 +gmmlu_si/test/philosophy/test/16 +gmmlu_si/test/philosophy/test/160 +gmmlu_si/test/philosophy/test/161 +gmmlu_si/test/philosophy/test/165 +gmmlu_si/test/philosophy/test/174 +gmmlu_si/test/philosophy/test/176 +gmmlu_si/test/philosophy/test/179 +gmmlu_si/test/philosophy/test/187 +gmmlu_si/test/philosophy/test/193 +gmmlu_si/test/philosophy/test/194 +gmmlu_si/test/philosophy/test/195 +gmmlu_si/test/philosophy/test/201 +gmmlu_si/test/philosophy/test/202 +gmmlu_si/test/philosophy/test/207 +gmmlu_si/test/philosophy/test/215 +gmmlu_si/test/philosophy/test/217 +gmmlu_si/test/philosophy/test/229 +gmmlu_si/test/philosophy/test/234 +gmmlu_si/test/philosophy/test/243 +gmmlu_si/test/philosophy/test/244 +gmmlu_si/test/philosophy/test/250 +gmmlu_si/test/philosophy/test/257 +gmmlu_si/test/philosophy/test/263 +gmmlu_si/test/philosophy/test/264 +gmmlu_si/test/philosophy/test/265 +gmmlu_si/test/philosophy/test/278 +gmmlu_si/test/philosophy/test/279 +gmmlu_si/test/philosophy/test/280 +gmmlu_si/test/philosophy/test/281 +gmmlu_si/test/philosophy/test/299 +gmmlu_si/test/philosophy/test/305 +gmmlu_si/test/philosophy/test/306 +gmmlu_si/test/philosophy/test/40 +gmmlu_si/test/philosophy/test/46 +gmmlu_si/test/philosophy/test/6 +gmmlu_si/test/philosophy/test/61 +gmmlu_si/test/philosophy/test/65 +gmmlu_si/test/philosophy/test/70 +gmmlu_si/test/philosophy/test/81 +gmmlu_si/test/philosophy/test/88 +gmmlu_si/test/prehistory/test/108 +gmmlu_si/test/prehistory/test/111 +gmmlu_si/test/prehistory/test/119 +gmmlu_si/test/prehistory/test/122 +gmmlu_si/test/prehistory/test/129 +gmmlu_si/test/prehistory/test/13 +gmmlu_si/test/prehistory/test/133 +gmmlu_si/test/prehistory/test/134 +gmmlu_si/test/prehistory/test/141 +gmmlu_si/test/prehistory/test/142 +gmmlu_si/test/prehistory/test/143 +gmmlu_si/test/prehistory/test/149 +gmmlu_si/test/prehistory/test/161 +gmmlu_si/test/prehistory/test/162 +gmmlu_si/test/prehistory/test/174 +gmmlu_si/test/prehistory/test/184 +gmmlu_si/test/prehistory/test/203 +gmmlu_si/test/prehistory/test/211 +gmmlu_si/test/prehistory/test/213 +gmmlu_si/test/prehistory/test/216 +gmmlu_si/test/prehistory/test/217 +gmmlu_si/test/prehistory/test/223 +gmmlu_si/test/prehistory/test/238 +gmmlu_si/test/prehistory/test/24 +gmmlu_si/test/prehistory/test/242 +gmmlu_si/test/prehistory/test/255 +gmmlu_si/test/prehistory/test/26 +gmmlu_si/test/prehistory/test/267 +gmmlu_si/test/prehistory/test/279 +gmmlu_si/test/prehistory/test/28 +gmmlu_si/test/prehistory/test/294 +gmmlu_si/test/prehistory/test/299 +gmmlu_si/test/prehistory/test/3 +gmmlu_si/test/prehistory/test/300 +gmmlu_si/test/prehistory/test/303 +gmmlu_si/test/prehistory/test/308 +gmmlu_si/test/prehistory/test/312 +gmmlu_si/test/prehistory/test/318 +gmmlu_si/test/prehistory/test/320 +gmmlu_si/test/prehistory/test/45 +gmmlu_si/test/prehistory/test/48 +gmmlu_si/test/prehistory/test/59 +gmmlu_si/test/prehistory/test/7 +gmmlu_si/test/prehistory/test/76 +gmmlu_si/test/prehistory/test/85 +gmmlu_si/test/prehistory/test/95 +gmmlu_si/test/professional_accounting/test/0 +gmmlu_si/test/professional_accounting/test/108 +gmmlu_si/test/professional_accounting/test/109 +gmmlu_si/test/professional_accounting/test/112 +gmmlu_si/test/professional_accounting/test/114 +gmmlu_si/test/professional_accounting/test/119 +gmmlu_si/test/professional_accounting/test/120 +gmmlu_si/test/professional_accounting/test/128 +gmmlu_si/test/professional_accounting/test/135 +gmmlu_si/test/professional_accounting/test/136 +gmmlu_si/test/professional_accounting/test/142 +gmmlu_si/test/professional_accounting/test/157 +gmmlu_si/test/professional_accounting/test/159 +gmmlu_si/test/professional_accounting/test/167 +gmmlu_si/test/professional_accounting/test/168 +gmmlu_si/test/professional_accounting/test/173 +gmmlu_si/test/professional_accounting/test/193 +gmmlu_si/test/professional_accounting/test/197 +gmmlu_si/test/professional_accounting/test/206 +gmmlu_si/test/professional_accounting/test/210 +gmmlu_si/test/professional_accounting/test/212 +gmmlu_si/test/professional_accounting/test/216 +gmmlu_si/test/professional_accounting/test/224 +gmmlu_si/test/professional_accounting/test/237 +gmmlu_si/test/professional_accounting/test/241 +gmmlu_si/test/professional_accounting/test/252 +gmmlu_si/test/professional_accounting/test/254 +gmmlu_si/test/professional_accounting/test/258 +gmmlu_si/test/professional_accounting/test/261 +gmmlu_si/test/professional_accounting/test/262 +gmmlu_si/test/professional_accounting/test/264 +gmmlu_si/test/professional_accounting/test/278 +gmmlu_si/test/professional_accounting/test/279 +gmmlu_si/test/professional_accounting/test/281 +gmmlu_si/test/professional_accounting/test/41 +gmmlu_si/test/professional_accounting/test/46 +gmmlu_si/test/professional_accounting/test/60 +gmmlu_si/test/professional_accounting/test/76 +gmmlu_si/test/professional_accounting/test/86 +gmmlu_si/test/professional_accounting/test/90 +gmmlu_si/test/professional_law/test/1014 +gmmlu_si/test/professional_law/test/1024 +gmmlu_si/test/professional_law/test/1032 +gmmlu_si/test/professional_law/test/1034 +gmmlu_si/test/professional_law/test/1047 +gmmlu_si/test/professional_law/test/1051 +gmmlu_si/test/professional_law/test/1052 +gmmlu_si/test/professional_law/test/1056 +gmmlu_si/test/professional_law/test/1059 +gmmlu_si/test/professional_law/test/106 +gmmlu_si/test/professional_law/test/1072 +gmmlu_si/test/professional_law/test/1075 +gmmlu_si/test/professional_law/test/1083 +gmmlu_si/test/professional_law/test/109 +gmmlu_si/test/professional_law/test/1095 +gmmlu_si/test/professional_law/test/1097 +gmmlu_si/test/professional_law/test/1109 +gmmlu_si/test/professional_law/test/1110 +gmmlu_si/test/professional_law/test/1125 +gmmlu_si/test/professional_law/test/1128 +gmmlu_si/test/professional_law/test/1131 +gmmlu_si/test/professional_law/test/1139 +gmmlu_si/test/professional_law/test/1145 +gmmlu_si/test/professional_law/test/1169 +gmmlu_si/test/professional_law/test/1170 +gmmlu_si/test/professional_law/test/1178 +gmmlu_si/test/professional_law/test/1179 +gmmlu_si/test/professional_law/test/1189 +gmmlu_si/test/professional_law/test/1190 +gmmlu_si/test/professional_law/test/1202 +gmmlu_si/test/professional_law/test/1203 +gmmlu_si/test/professional_law/test/1207 +gmmlu_si/test/professional_law/test/1211 +gmmlu_si/test/professional_law/test/1213 +gmmlu_si/test/professional_law/test/122 +gmmlu_si/test/professional_law/test/1233 +gmmlu_si/test/professional_law/test/1251 +gmmlu_si/test/professional_law/test/1257 +gmmlu_si/test/professional_law/test/1266 +gmmlu_si/test/professional_law/test/1294 +gmmlu_si/test/professional_law/test/1307 +gmmlu_si/test/professional_law/test/131 +gmmlu_si/test/professional_law/test/1314 +gmmlu_si/test/professional_law/test/1319 +gmmlu_si/test/professional_law/test/1326 +gmmlu_si/test/professional_law/test/1332 +gmmlu_si/test/professional_law/test/1335 +gmmlu_si/test/professional_law/test/1338 +gmmlu_si/test/professional_law/test/1346 +gmmlu_si/test/professional_law/test/1347 +gmmlu_si/test/professional_law/test/1361 +gmmlu_si/test/professional_law/test/1384 +gmmlu_si/test/professional_law/test/1387 +gmmlu_si/test/professional_law/test/1392 +gmmlu_si/test/professional_law/test/1401 +gmmlu_si/test/professional_law/test/1411 +gmmlu_si/test/professional_law/test/1413 +gmmlu_si/test/professional_law/test/1431 +gmmlu_si/test/professional_law/test/1438 +gmmlu_si/test/professional_law/test/1439 +gmmlu_si/test/professional_law/test/1454 +gmmlu_si/test/professional_law/test/1459 +gmmlu_si/test/professional_law/test/1463 +gmmlu_si/test/professional_law/test/1470 +gmmlu_si/test/professional_law/test/1475 +gmmlu_si/test/professional_law/test/1477 +gmmlu_si/test/professional_law/test/1484 +gmmlu_si/test/professional_law/test/1487 +gmmlu_si/test/professional_law/test/1488 +gmmlu_si/test/professional_law/test/1501 +gmmlu_si/test/professional_law/test/1502 +gmmlu_si/test/professional_law/test/1504 +gmmlu_si/test/professional_law/test/1509 +gmmlu_si/test/professional_law/test/1515 +gmmlu_si/test/professional_law/test/1516 +gmmlu_si/test/professional_law/test/1521 +gmmlu_si/test/professional_law/test/1526 +gmmlu_si/test/professional_law/test/1527 +gmmlu_si/test/professional_law/test/159 +gmmlu_si/test/professional_law/test/163 +gmmlu_si/test/professional_law/test/170 +gmmlu_si/test/professional_law/test/174 +gmmlu_si/test/professional_law/test/177 +gmmlu_si/test/professional_law/test/184 +gmmlu_si/test/professional_law/test/185 +gmmlu_si/test/professional_law/test/194 +gmmlu_si/test/professional_law/test/199 +gmmlu_si/test/professional_law/test/205 +gmmlu_si/test/professional_law/test/207 +gmmlu_si/test/professional_law/test/213 +gmmlu_si/test/professional_law/test/223 +gmmlu_si/test/professional_law/test/227 +gmmlu_si/test/professional_law/test/24 +gmmlu_si/test/professional_law/test/246 +gmmlu_si/test/professional_law/test/249 +gmmlu_si/test/professional_law/test/250 +gmmlu_si/test/professional_law/test/251 +gmmlu_si/test/professional_law/test/267 +gmmlu_si/test/professional_law/test/269 +gmmlu_si/test/professional_law/test/271 +gmmlu_si/test/professional_law/test/278 +gmmlu_si/test/professional_law/test/286 +gmmlu_si/test/professional_law/test/299 +gmmlu_si/test/professional_law/test/302 +gmmlu_si/test/professional_law/test/317 +gmmlu_si/test/professional_law/test/32 +gmmlu_si/test/professional_law/test/34 +gmmlu_si/test/professional_law/test/342 +gmmlu_si/test/professional_law/test/348 +gmmlu_si/test/professional_law/test/350 +gmmlu_si/test/professional_law/test/351 +gmmlu_si/test/professional_law/test/360 +gmmlu_si/test/professional_law/test/363 +gmmlu_si/test/professional_law/test/366 +gmmlu_si/test/professional_law/test/369 +gmmlu_si/test/professional_law/test/375 +gmmlu_si/test/professional_law/test/376 +gmmlu_si/test/professional_law/test/381 +gmmlu_si/test/professional_law/test/393 +gmmlu_si/test/professional_law/test/397 +gmmlu_si/test/professional_law/test/40 +gmmlu_si/test/professional_law/test/401 +gmmlu_si/test/professional_law/test/404 +gmmlu_si/test/professional_law/test/407 +gmmlu_si/test/professional_law/test/422 +gmmlu_si/test/professional_law/test/43 +gmmlu_si/test/professional_law/test/438 +gmmlu_si/test/professional_law/test/441 +gmmlu_si/test/professional_law/test/442 +gmmlu_si/test/professional_law/test/451 +gmmlu_si/test/professional_law/test/453 +gmmlu_si/test/professional_law/test/454 +gmmlu_si/test/professional_law/test/462 +gmmlu_si/test/professional_law/test/463 +gmmlu_si/test/professional_law/test/470 +gmmlu_si/test/professional_law/test/497 +gmmlu_si/test/professional_law/test/499 +gmmlu_si/test/professional_law/test/505 +gmmlu_si/test/professional_law/test/514 +gmmlu_si/test/professional_law/test/525 +gmmlu_si/test/professional_law/test/538 +gmmlu_si/test/professional_law/test/539 +gmmlu_si/test/professional_law/test/540 +gmmlu_si/test/professional_law/test/545 +gmmlu_si/test/professional_law/test/55 +gmmlu_si/test/professional_law/test/571 +gmmlu_si/test/professional_law/test/573 +gmmlu_si/test/professional_law/test/580 +gmmlu_si/test/professional_law/test/583 +gmmlu_si/test/professional_law/test/584 +gmmlu_si/test/professional_law/test/593 +gmmlu_si/test/professional_law/test/596 +gmmlu_si/test/professional_law/test/600 +gmmlu_si/test/professional_law/test/611 +gmmlu_si/test/professional_law/test/616 +gmmlu_si/test/professional_law/test/617 +gmmlu_si/test/professional_law/test/623 +gmmlu_si/test/professional_law/test/630 +gmmlu_si/test/professional_law/test/644 +gmmlu_si/test/professional_law/test/645 +gmmlu_si/test/professional_law/test/65 +gmmlu_si/test/professional_law/test/679 +gmmlu_si/test/professional_law/test/69 +gmmlu_si/test/professional_law/test/690 +gmmlu_si/test/professional_law/test/695 +gmmlu_si/test/professional_law/test/698 +gmmlu_si/test/professional_law/test/70 +gmmlu_si/test/professional_law/test/705 +gmmlu_si/test/professional_law/test/709 +gmmlu_si/test/professional_law/test/714 +gmmlu_si/test/professional_law/test/715 +gmmlu_si/test/professional_law/test/718 +gmmlu_si/test/professional_law/test/724 +gmmlu_si/test/professional_law/test/729 +gmmlu_si/test/professional_law/test/735 +gmmlu_si/test/professional_law/test/737 +gmmlu_si/test/professional_law/test/746 +gmmlu_si/test/professional_law/test/748 +gmmlu_si/test/professional_law/test/750 +gmmlu_si/test/professional_law/test/754 +gmmlu_si/test/professional_law/test/76 +gmmlu_si/test/professional_law/test/761 +gmmlu_si/test/professional_law/test/786 +gmmlu_si/test/professional_law/test/79 +gmmlu_si/test/professional_law/test/791 +gmmlu_si/test/professional_law/test/798 +gmmlu_si/test/professional_law/test/800 +gmmlu_si/test/professional_law/test/809 +gmmlu_si/test/professional_law/test/819 +gmmlu_si/test/professional_law/test/82 +gmmlu_si/test/professional_law/test/827 +gmmlu_si/test/professional_law/test/833 +gmmlu_si/test/professional_law/test/834 +gmmlu_si/test/professional_law/test/837 +gmmlu_si/test/professional_law/test/865 +gmmlu_si/test/professional_law/test/871 +gmmlu_si/test/professional_law/test/873 +gmmlu_si/test/professional_law/test/882 +gmmlu_si/test/professional_law/test/889 +gmmlu_si/test/professional_law/test/89 +gmmlu_si/test/professional_law/test/893 +gmmlu_si/test/professional_law/test/897 +gmmlu_si/test/professional_law/test/907 +gmmlu_si/test/professional_law/test/908 +gmmlu_si/test/professional_law/test/909 +gmmlu_si/test/professional_law/test/914 +gmmlu_si/test/professional_law/test/920 +gmmlu_si/test/professional_law/test/940 +gmmlu_si/test/professional_law/test/942 +gmmlu_si/test/professional_law/test/949 +gmmlu_si/test/professional_law/test/951 +gmmlu_si/test/professional_law/test/953 +gmmlu_si/test/professional_law/test/954 +gmmlu_si/test/professional_law/test/958 +gmmlu_si/test/professional_law/test/97 +gmmlu_si/test/professional_law/test/976 +gmmlu_si/test/professional_law/test/979 +gmmlu_si/test/professional_law/test/983 +gmmlu_si/test/professional_medicine/test/103 +gmmlu_si/test/professional_medicine/test/104 +gmmlu_si/test/professional_medicine/test/111 +gmmlu_si/test/professional_medicine/test/119 +gmmlu_si/test/professional_medicine/test/120 +gmmlu_si/test/professional_medicine/test/13 +gmmlu_si/test/professional_medicine/test/132 +gmmlu_si/test/professional_medicine/test/133 +gmmlu_si/test/professional_medicine/test/143 +gmmlu_si/test/professional_medicine/test/144 +gmmlu_si/test/professional_medicine/test/15 +gmmlu_si/test/professional_medicine/test/156 +gmmlu_si/test/professional_medicine/test/158 +gmmlu_si/test/professional_medicine/test/162 +gmmlu_si/test/professional_medicine/test/170 +gmmlu_si/test/professional_medicine/test/174 +gmmlu_si/test/professional_medicine/test/178 +gmmlu_si/test/professional_medicine/test/179 +gmmlu_si/test/professional_medicine/test/18 +gmmlu_si/test/professional_medicine/test/190 +gmmlu_si/test/professional_medicine/test/192 +gmmlu_si/test/professional_medicine/test/204 +gmmlu_si/test/professional_medicine/test/209 +gmmlu_si/test/professional_medicine/test/210 +gmmlu_si/test/professional_medicine/test/211 +gmmlu_si/test/professional_medicine/test/216 +gmmlu_si/test/professional_medicine/test/221 +gmmlu_si/test/professional_medicine/test/224 +gmmlu_si/test/professional_medicine/test/234 +gmmlu_si/test/professional_medicine/test/248 +gmmlu_si/test/professional_medicine/test/249 +gmmlu_si/test/professional_medicine/test/252 +gmmlu_si/test/professional_medicine/test/254 +gmmlu_si/test/professional_medicine/test/257 +gmmlu_si/test/professional_medicine/test/258 +gmmlu_si/test/professional_medicine/test/270 +gmmlu_si/test/professional_medicine/test/43 +gmmlu_si/test/professional_medicine/test/5 +gmmlu_si/test/professional_medicine/test/83 +gmmlu_si/test/professional_psychology/test/108 +gmmlu_si/test/professional_psychology/test/113 +gmmlu_si/test/professional_psychology/test/136 +gmmlu_si/test/professional_psychology/test/140 +gmmlu_si/test/professional_psychology/test/153 +gmmlu_si/test/professional_psychology/test/155 +gmmlu_si/test/professional_psychology/test/157 +gmmlu_si/test/professional_psychology/test/163 +gmmlu_si/test/professional_psychology/test/164 +gmmlu_si/test/professional_psychology/test/17 +gmmlu_si/test/professional_psychology/test/180 +gmmlu_si/test/professional_psychology/test/185 +gmmlu_si/test/professional_psychology/test/196 +gmmlu_si/test/professional_psychology/test/209 +gmmlu_si/test/professional_psychology/test/210 +gmmlu_si/test/professional_psychology/test/223 +gmmlu_si/test/professional_psychology/test/224 +gmmlu_si/test/professional_psychology/test/226 +gmmlu_si/test/professional_psychology/test/227 +gmmlu_si/test/professional_psychology/test/237 +gmmlu_si/test/professional_psychology/test/240 +gmmlu_si/test/professional_psychology/test/242 +gmmlu_si/test/professional_psychology/test/249 +gmmlu_si/test/professional_psychology/test/285 +gmmlu_si/test/professional_psychology/test/288 +gmmlu_si/test/professional_psychology/test/30 +gmmlu_si/test/professional_psychology/test/310 +gmmlu_si/test/professional_psychology/test/314 +gmmlu_si/test/professional_psychology/test/325 +gmmlu_si/test/professional_psychology/test/33 +gmmlu_si/test/professional_psychology/test/334 +gmmlu_si/test/professional_psychology/test/336 +gmmlu_si/test/professional_psychology/test/340 +gmmlu_si/test/professional_psychology/test/341 +gmmlu_si/test/professional_psychology/test/342 +gmmlu_si/test/professional_psychology/test/343 +gmmlu_si/test/professional_psychology/test/352 +gmmlu_si/test/professional_psychology/test/355 +gmmlu_si/test/professional_psychology/test/363 +gmmlu_si/test/professional_psychology/test/380 +gmmlu_si/test/professional_psychology/test/385 +gmmlu_si/test/professional_psychology/test/390 +gmmlu_si/test/professional_psychology/test/400 +gmmlu_si/test/professional_psychology/test/411 +gmmlu_si/test/professional_psychology/test/416 +gmmlu_si/test/professional_psychology/test/418 +gmmlu_si/test/professional_psychology/test/424 +gmmlu_si/test/professional_psychology/test/43 +gmmlu_si/test/professional_psychology/test/430 +gmmlu_si/test/professional_psychology/test/445 +gmmlu_si/test/professional_psychology/test/450 +gmmlu_si/test/professional_psychology/test/458 +gmmlu_si/test/professional_psychology/test/48 +gmmlu_si/test/professional_psychology/test/487 +gmmlu_si/test/professional_psychology/test/489 +gmmlu_si/test/professional_psychology/test/493 +gmmlu_si/test/professional_psychology/test/498 +gmmlu_si/test/professional_psychology/test/503 +gmmlu_si/test/professional_psychology/test/513 +gmmlu_si/test/professional_psychology/test/524 +gmmlu_si/test/professional_psychology/test/526 +gmmlu_si/test/professional_psychology/test/543 +gmmlu_si/test/professional_psychology/test/556 +gmmlu_si/test/professional_psychology/test/557 +gmmlu_si/test/professional_psychology/test/558 +gmmlu_si/test/professional_psychology/test/559 +gmmlu_si/test/professional_psychology/test/56 +gmmlu_si/test/professional_psychology/test/563 +gmmlu_si/test/professional_psychology/test/567 +gmmlu_si/test/professional_psychology/test/57 +gmmlu_si/test/professional_psychology/test/573 +gmmlu_si/test/professional_psychology/test/577 +gmmlu_si/test/professional_psychology/test/583 +gmmlu_si/test/professional_psychology/test/584 +gmmlu_si/test/professional_psychology/test/589 +gmmlu_si/test/professional_psychology/test/592 +gmmlu_si/test/professional_psychology/test/596 +gmmlu_si/test/professional_psychology/test/597 +gmmlu_si/test/professional_psychology/test/599 +gmmlu_si/test/professional_psychology/test/606 +gmmlu_si/test/professional_psychology/test/608 +gmmlu_si/test/professional_psychology/test/61 +gmmlu_si/test/professional_psychology/test/611 +gmmlu_si/test/professional_psychology/test/77 +gmmlu_si/test/professional_psychology/test/86 +gmmlu_si/test/professional_psychology/test/89 +gmmlu_si/test/professional_psychology/test/94 +gmmlu_si/test/public_relations/test/100 +gmmlu_si/test/public_relations/test/103 +gmmlu_si/test/public_relations/test/108 +gmmlu_si/test/public_relations/test/16 +gmmlu_si/test/public_relations/test/33 +gmmlu_si/test/public_relations/test/49 +gmmlu_si/test/public_relations/test/58 +gmmlu_si/test/public_relations/test/6 +gmmlu_si/test/public_relations/test/61 +gmmlu_si/test/public_relations/test/63 +gmmlu_si/test/public_relations/test/76 +gmmlu_si/test/public_relations/test/78 +gmmlu_si/test/public_relations/test/79 +gmmlu_si/test/public_relations/test/91 +gmmlu_si/test/public_relations/test/92 +gmmlu_si/test/public_relations/test/99 +gmmlu_si/test/security_studies/test/121 +gmmlu_si/test/security_studies/test/122 +gmmlu_si/test/security_studies/test/125 +gmmlu_si/test/security_studies/test/126 +gmmlu_si/test/security_studies/test/129 +gmmlu_si/test/security_studies/test/13 +gmmlu_si/test/security_studies/test/130 +gmmlu_si/test/security_studies/test/143 +gmmlu_si/test/security_studies/test/15 +gmmlu_si/test/security_studies/test/156 +gmmlu_si/test/security_studies/test/162 +gmmlu_si/test/security_studies/test/168 +gmmlu_si/test/security_studies/test/175 +gmmlu_si/test/security_studies/test/186 +gmmlu_si/test/security_studies/test/188 +gmmlu_si/test/security_studies/test/195 +gmmlu_si/test/security_studies/test/200 +gmmlu_si/test/security_studies/test/212 +gmmlu_si/test/security_studies/test/213 +gmmlu_si/test/security_studies/test/216 +gmmlu_si/test/security_studies/test/22 +gmmlu_si/test/security_studies/test/230 +gmmlu_si/test/security_studies/test/239 +gmmlu_si/test/security_studies/test/241 +gmmlu_si/test/security_studies/test/31 +gmmlu_si/test/security_studies/test/33 +gmmlu_si/test/security_studies/test/48 +gmmlu_si/test/security_studies/test/54 +gmmlu_si/test/security_studies/test/55 +gmmlu_si/test/security_studies/test/59 +gmmlu_si/test/security_studies/test/61 +gmmlu_si/test/security_studies/test/70 +gmmlu_si/test/security_studies/test/90 +gmmlu_si/test/security_studies/test/96 +gmmlu_si/test/security_studies/test/99 +gmmlu_si/test/sociology/test/105 +gmmlu_si/test/sociology/test/107 +gmmlu_si/test/sociology/test/119 +gmmlu_si/test/sociology/test/121 +gmmlu_si/test/sociology/test/123 +gmmlu_si/test/sociology/test/135 +gmmlu_si/test/sociology/test/139 +gmmlu_si/test/sociology/test/148 +gmmlu_si/test/sociology/test/155 +gmmlu_si/test/sociology/test/156 +gmmlu_si/test/sociology/test/16 +gmmlu_si/test/sociology/test/161 +gmmlu_si/test/sociology/test/165 +gmmlu_si/test/sociology/test/167 +gmmlu_si/test/sociology/test/17 +gmmlu_si/test/sociology/test/177 +gmmlu_si/test/sociology/test/186 +gmmlu_si/test/sociology/test/188 +gmmlu_si/test/sociology/test/195 +gmmlu_si/test/sociology/test/199 +gmmlu_si/test/sociology/test/24 +gmmlu_si/test/sociology/test/36 +gmmlu_si/test/sociology/test/39 +gmmlu_si/test/sociology/test/50 +gmmlu_si/test/sociology/test/51 +gmmlu_si/test/sociology/test/59 +gmmlu_si/test/sociology/test/66 +gmmlu_si/test/sociology/test/93 +gmmlu_si/test/sociology/test/97 +gmmlu_si/test/us_foreign_policy/test/14 +gmmlu_si/test/us_foreign_policy/test/25 +gmmlu_si/test/us_foreign_policy/test/3 +gmmlu_si/test/us_foreign_policy/test/39 +gmmlu_si/test/us_foreign_policy/test/42 +gmmlu_si/test/us_foreign_policy/test/44 +gmmlu_si/test/us_foreign_policy/test/46 +gmmlu_si/test/us_foreign_policy/test/48 +gmmlu_si/test/us_foreign_policy/test/54 +gmmlu_si/test/us_foreign_policy/test/58 +gmmlu_si/test/us_foreign_policy/test/83 +gmmlu_si/test/us_foreign_policy/test/94 +gmmlu_si/test/us_foreign_policy/test/98 +gmmlu_si/test/us_foreign_policy/test/99 +gmmlu_si/test/virology/test/101 +gmmlu_si/test/virology/test/105 +gmmlu_si/test/virology/test/106 +gmmlu_si/test/virology/test/107 +gmmlu_si/test/virology/test/11 +gmmlu_si/test/virology/test/120 +gmmlu_si/test/virology/test/122 +gmmlu_si/test/virology/test/129 +gmmlu_si/test/virology/test/140 +gmmlu_si/test/virology/test/147 +gmmlu_si/test/virology/test/157 +gmmlu_si/test/virology/test/16 +gmmlu_si/test/virology/test/17 +gmmlu_si/test/virology/test/23 +gmmlu_si/test/virology/test/29 +gmmlu_si/test/virology/test/30 +gmmlu_si/test/virology/test/34 +gmmlu_si/test/virology/test/49 +gmmlu_si/test/virology/test/52 +gmmlu_si/test/virology/test/61 +gmmlu_si/test/virology/test/8 +gmmlu_si/test/virology/test/85 +gmmlu_si/test/virology/test/94 +gmmlu_si/test/virology/test/97 +gmmlu_si/test/world_religions/test/0 +gmmlu_si/test/world_religions/test/109 +gmmlu_si/test/world_religions/test/112 +gmmlu_si/test/world_religions/test/122 +gmmlu_si/test/world_religions/test/123 +gmmlu_si/test/world_religions/test/13 +gmmlu_si/test/world_religions/test/146 +gmmlu_si/test/world_religions/test/154 +gmmlu_si/test/world_religions/test/159 +gmmlu_si/test/world_religions/test/161 +gmmlu_si/test/world_religions/test/21 +gmmlu_si/test/world_religions/test/4 +gmmlu_si/test/world_religions/test/44 +gmmlu_si/test/world_religions/test/45 +gmmlu_si/test/world_religions/test/50 +gmmlu_si/test/world_religions/test/53 +gmmlu_si/test/world_religions/test/60 +gmmlu_si/test/world_religions/test/61 +gmmlu_si/test/world_religions/test/62 +gmmlu_si/test/world_religions/test/68 +gmmlu_si/test/world_religions/test/74 +gmmlu_si/test/world_religions/test/81 +gmmlu_si/test/world_religions/test/9 +gmmlu_si/test/world_religions/test/90 diff --git a/training/configs/splits/sinhalammlu_dev_5shot_ids.txt b/training/configs/splits/sinhalammlu_dev_5shot_ids.txt new file mode 100644 index 0000000000000000000000000000000000000000..39c689f4c476895de9528f5a67c91e47e122ce20 --- /dev/null +++ b/training/configs/splits/sinhalammlu_dev_5shot_ids.txt @@ -0,0 +1,5 @@ +smmlu/easy/science/128 +smmlu/easy/Health and Physical Science/52 +smmlu/easy/Sinhala language and literature/21 +smmlu/easy/Buddhism/158 +smmlu/easy/Buddhism/147 diff --git a/training/configs/splits/sinhalammlu_dev_cv5.json b/training/configs/splits/sinhalammlu_dev_cv5.json new file mode 100644 index 0000000000000000000000000000000000000000..82c337e49ddbfe9a8159bc3539c300e850fe3155 --- /dev/null +++ b/training/configs/splits/sinhalammlu_dev_cv5.json @@ -0,0 +1,1862 @@ +[ + [ + "smmlu/easy/Arts/73", + "smmlu/easy/Arts/55", + "smmlu/easy/Arts/84", + "smmlu/easy/Arts/31", + "smmlu/easy/Arts/53", + "smmlu/easy/Arts/66", + "smmlu/easy/Arts/54", + "smmlu/easy/Arts/44", + "smmlu/easy/Arts/30", + "smmlu/easy/Arts/63", + "smmlu/easy/Arts/35", + "smmlu/easy/Arts/29", + "smmlu/easy/Arts/93", + "smmlu/easy/Arts/68", + "smmlu/easy/Arts/37", + "smmlu/easy/Arts/3", + "smmlu/easy/Arts/64", + "smmlu/easy/Arts/11", + "smmlu/easy/Arts/18", + "smmlu/easy/Arts/19", + "smmlu/easy/Buddhism/77", + "smmlu/easy/Buddhism/154", + "smmlu/easy/Buddhism/36", + "smmlu/easy/Buddhism/146", + "smmlu/easy/Buddhism/35", + "smmlu/easy/Buddhism/47", + "smmlu/easy/Buddhism/150", + "smmlu/easy/Buddhism/124", + "smmlu/easy/Buddhism/5", + "smmlu/easy/Buddhism/84", + "smmlu/easy/Buddhism/159", + "smmlu/easy/Buddhism/13", + "smmlu/easy/Buddhism/97", + "smmlu/easy/Buddhism/142", + "smmlu/easy/Buddhism/103", + "smmlu/easy/Buddhism/1", + "smmlu/easy/Buddhism/87", + "smmlu/easy/Buddhism/114", + "smmlu/easy/Buddhism/116", + "smmlu/easy/Buddhism/67", + "smmlu/easy/Buddhism/83", + "smmlu/easy/Buddhism/157", + "smmlu/easy/Buddhism/71", + "smmlu/easy/Buddhism/51", + "smmlu/easy/Buddhism/85", + "smmlu/easy/Buddhism/60", + "smmlu/easy/Buddhism/19", + "smmlu/easy/Buddhism/115", + "smmlu/easy/Buddhism/135", + "smmlu/easy/Buddhism/4", + "smmlu/easy/Buddhism/53", + "smmlu/easy/Buddhism/20", + "smmlu/easy/Buddhism/90", + "smmlu/easy/Catholicism/84", + "smmlu/easy/Catholicism/70", + "smmlu/easy/Catholicism/93", + "smmlu/easy/Catholicism/111", + "smmlu/easy/Catholicism/11", + "smmlu/easy/Catholicism/64", + "smmlu/easy/Catholicism/55", + "smmlu/easy/Catholicism/42", + "smmlu/easy/Catholicism/131", + "smmlu/easy/Catholicism/52", + "smmlu/easy/Catholicism/88", + "smmlu/easy/Catholicism/97", + "smmlu/easy/Catholicism/63", + "smmlu/easy/Catholicism/76", + "smmlu/easy/Catholicism/124", + "smmlu/easy/Catholicism/106", + "smmlu/easy/Catholicism/16", + "smmlu/easy/Catholicism/57", + "smmlu/easy/Catholicism/22", + "smmlu/easy/Catholicism/92", + "smmlu/easy/Catholicism/13", + "smmlu/easy/Catholicism/19", + "smmlu/easy/Catholicism/107", + "smmlu/easy/Catholicism/120", + "smmlu/easy/Catholicism/18", + "smmlu/easy/Catholicism/47", + "smmlu/easy/Christianity/122", + "smmlu/easy/Christianity/105", + "smmlu/easy/Christianity/91", + "smmlu/easy/Christianity/96", + "smmlu/easy/Christianity/103", + "smmlu/easy/Christianity/41", + "smmlu/easy/Christianity/124", + "smmlu/easy/Christianity/25", + "smmlu/easy/Christianity/58", + "smmlu/easy/Christianity/95", + "smmlu/easy/Christianity/127", + "smmlu/easy/Christianity/60", + "smmlu/easy/Christianity/20", + "smmlu/easy/Christianity/40", + "smmlu/easy/Christianity/52", + "smmlu/easy/Christianity/120", + "smmlu/easy/Christianity/118", + "smmlu/easy/Christianity/85", + "smmlu/easy/Christianity/78", + "smmlu/easy/Christianity/102", + "smmlu/easy/Christianity/49", + "smmlu/easy/Christianity/82", + "smmlu/easy/Christianity/24", + "smmlu/easy/Christianity/50", + "smmlu/easy/Christianity/128", + "smmlu/easy/Christianity/4", + "smmlu/easy/Civics/1", + "smmlu/easy/Civics/97", + "smmlu/easy/Civics/43", + "smmlu/easy/Civics/111", + "smmlu/easy/Civics/86", + "smmlu/easy/Civics/108", + "smmlu/easy/Civics/41", + "smmlu/easy/Civics/40", + "smmlu/easy/Civics/89", + "smmlu/easy/Civics/13", + "smmlu/easy/Civics/8", + "smmlu/easy/Civics/42", + "smmlu/easy/Civics/25", + "smmlu/easy/Civics/106", + "smmlu/easy/Civics/122", + "smmlu/easy/Civics/14", + "smmlu/easy/Civics/117", + "smmlu/easy/Civics/94", + "smmlu/easy/Civics/19", + "smmlu/easy/Civics/39", + "smmlu/easy/Civics/116", + "smmlu/easy/Civics/84", + "smmlu/easy/Civics/10", + "smmlu/easy/Civics/92", + "smmlu/easy/Civics/56", + "smmlu/easy/Eastern music/35", + "smmlu/easy/Eastern music/65", + "smmlu/easy/Eastern music/80", + "smmlu/easy/Eastern music/73", + "smmlu/easy/Eastern music/83", + "smmlu/easy/Eastern music/4", + "smmlu/easy/Eastern music/100", + "smmlu/easy/Eastern music/111", + "smmlu/easy/Eastern music/97", + "smmlu/easy/Eastern music/109", + "smmlu/easy/Eastern music/62", + "smmlu/easy/Eastern music/7", + "smmlu/easy/Eastern music/132", + "smmlu/easy/Eastern music/98", + "smmlu/easy/Eastern music/14", + "smmlu/easy/Eastern music/91", + "smmlu/easy/Eastern music/119", + "smmlu/easy/Eastern music/12", + "smmlu/easy/Eastern music/104", + "smmlu/easy/Eastern music/59", + "smmlu/easy/Eastern music/51", + "smmlu/easy/Eastern music/89", + "smmlu/easy/Eastern music/60", + "smmlu/easy/Eastern music/128", + "smmlu/easy/Eastern music/8", + "smmlu/easy/Eastern music/67", + "smmlu/easy/Eastern music/92", + "smmlu/easy/Eastern music/90", + "smmlu/easy/Geography/105", + "smmlu/easy/Geography/14", + "smmlu/easy/Geography/118", + "smmlu/easy/Geography/1", + "smmlu/easy/Geography/88", + "smmlu/easy/Geography/89", + "smmlu/easy/Geography/27", + "smmlu/easy/Geography/26", + "smmlu/easy/Geography/15", + "smmlu/easy/Geography/36", + "smmlu/easy/Geography/81", + "smmlu/easy/Geography/3", + "smmlu/easy/Geography/25", + "smmlu/easy/Geography/85", + "smmlu/easy/Geography/53", + "smmlu/easy/Geography/32", + "smmlu/easy/Geography/4", + "smmlu/easy/Geography/6", + "smmlu/easy/Geography/111", + "smmlu/easy/Geography/34", + "smmlu/easy/Geography/106", + "smmlu/easy/Geography/65", + "smmlu/easy/Geography/107", + "smmlu/easy/Geography/75", + "smmlu/easy/Health and Physical Science/97", + "smmlu/easy/Health and Physical Science/1", + "smmlu/easy/Health and Physical Science/65", + "smmlu/easy/Health and Physical Science/27", + "smmlu/easy/Health and Physical Science/25", + "smmlu/easy/Health and Physical Science/40", + "smmlu/easy/Health and Physical Science/50", + "smmlu/easy/Health and Physical Science/112", + "smmlu/easy/Health and Physical Science/38", + "smmlu/easy/Health and Physical Science/92", + "smmlu/easy/Health and Physical Science/108", + "smmlu/easy/Health and Physical Science/20", + "smmlu/easy/Health and Physical Science/8", + "smmlu/easy/Health and Physical Science/11", + "smmlu/easy/Health and Physical Science/72", + "smmlu/easy/Health and Physical Science/86", + "smmlu/easy/Health and Physical Science/94", + "smmlu/easy/Health and Physical Science/121", + "smmlu/easy/Health and Physical Science/55", + "smmlu/easy/Health and Physical Science/83", + "smmlu/easy/Health and Physical Science/54", + "smmlu/easy/Health and Physical Science/127", + "smmlu/easy/Health and Physical Science/82", + "smmlu/easy/Health and Physical Science/115", + "smmlu/easy/Health and Physical Science/26", + "smmlu/easy/Health and Physical Science/7", + "smmlu/easy/History/60", + "smmlu/easy/History/123", + "smmlu/easy/History/59", + "smmlu/easy/History/76", + "smmlu/easy/History/34", + "smmlu/easy/History/43", + "smmlu/easy/History/25", + "smmlu/easy/History/17", + "smmlu/easy/History/80", + "smmlu/easy/History/51", + "smmlu/easy/History/131", + "smmlu/easy/History/44", + "smmlu/easy/History/7", + "smmlu/easy/History/8", + "smmlu/easy/History/92", + "smmlu/easy/History/61", + "smmlu/easy/History/103", + "smmlu/easy/History/118", + "smmlu/easy/History/50", + "smmlu/easy/History/79", + "smmlu/easy/History/104", + "smmlu/easy/History/94", + "smmlu/easy/History/148", + "smmlu/easy/History/86", + "smmlu/easy/History/100", + "smmlu/easy/History/91", + "smmlu/easy/History/125", + "smmlu/easy/History/73", + "smmlu/easy/History/87", + "smmlu/easy/History/134", + "smmlu/easy/Islam/78", + "smmlu/easy/Islam/68", + "smmlu/easy/Islam/67", + "smmlu/easy/Islam/61", + "smmlu/easy/Islam/107", + "smmlu/easy/Islam/60", + "smmlu/easy/Islam/84", + "smmlu/easy/Islam/105", + "smmlu/easy/Islam/98", + "smmlu/easy/Islam/43", + "smmlu/easy/Islam/52", + "smmlu/easy/Islam/99", + "smmlu/easy/Islam/93", + "smmlu/easy/Islam/24", + "smmlu/easy/Islam/108", + "smmlu/easy/Islam/18", + "smmlu/easy/Islam/48", + "smmlu/easy/Islam/31", + "smmlu/easy/Islam/95", + "smmlu/easy/Islam/50", + "smmlu/easy/Islam/69", + "smmlu/easy/Sinhala language and literature/94", + "smmlu/easy/Sinhala language and literature/62", + "smmlu/easy/Sinhala language and literature/32", + "smmlu/easy/Sinhala language and literature/138", + "smmlu/easy/Sinhala language and literature/154", + "smmlu/easy/Sinhala language and literature/23", + "smmlu/easy/Sinhala language and literature/14", + "smmlu/easy/Sinhala language and literature/98", + "smmlu/easy/Sinhala language and literature/15", + "smmlu/easy/Sinhala language and literature/149", + "smmlu/easy/Sinhala language and literature/7", + "smmlu/easy/Sinhala language and literature/53", + "smmlu/easy/Sinhala language and literature/75", + "smmlu/easy/Sinhala language and literature/93", + "smmlu/easy/Sinhala language and literature/115", + "smmlu/easy/Sinhala language and literature/43", + "smmlu/easy/Sinhala language and literature/69", + "smmlu/easy/Sinhala language and literature/96", + "smmlu/easy/Sinhala language and literature/137", + "smmlu/easy/Sinhala language and literature/60", + "smmlu/easy/Sinhala language and literature/9", + "smmlu/easy/Sinhala language and literature/70", + "smmlu/easy/Sinhala language and literature/82", + "smmlu/easy/Sinhala language and literature/28", + "smmlu/easy/Sinhala language and literature/122", + "smmlu/easy/Sinhala language and literature/107", + "smmlu/easy/Sinhala language and literature/59", + "smmlu/easy/Sinhala language and literature/95", + "smmlu/easy/Sinhala language and literature/81", + "smmlu/easy/Sinhala language and literature/99", + "smmlu/easy/Sinhala language and literature/22", + "smmlu/easy/dancing/93", + "smmlu/easy/dancing/47", + "smmlu/easy/dancing/39", + "smmlu/easy/dancing/34", + "smmlu/easy/dancing/11", + "smmlu/easy/dancing/94", + "smmlu/easy/dancing/75", + "smmlu/easy/dancing/81", + "smmlu/easy/dancing/102", + "smmlu/easy/dancing/19", + "smmlu/easy/dancing/4", + "smmlu/easy/dancing/1", + "smmlu/easy/dancing/10", + "smmlu/easy/dancing/9", + "smmlu/easy/dancing/72", + "smmlu/easy/dancing/50", + "smmlu/easy/dancing/17", + "smmlu/easy/dancing/40", + "smmlu/easy/dancing/22", + "smmlu/easy/dancing/57", + "smmlu/easy/dancing/103", + "smmlu/easy/dancing/37", + "smmlu/easy/drama and Theatre /90", + "smmlu/easy/drama and Theatre /66", + "smmlu/easy/drama and Theatre /33", + "smmlu/easy/drama and Theatre /39", + "smmlu/easy/drama and Theatre /102", + "smmlu/easy/drama and Theatre /2", + "smmlu/easy/drama and Theatre /89", + "smmlu/easy/drama and Theatre /34", + "smmlu/easy/drama and Theatre /10", + "smmlu/easy/drama and Theatre /5", + "smmlu/easy/drama and Theatre /29", + "smmlu/easy/drama and Theatre /24", + "smmlu/easy/drama and Theatre /12", + "smmlu/easy/drama and Theatre /111", + "smmlu/easy/drama and Theatre /125", + "smmlu/easy/drama and Theatre /104", + "smmlu/easy/drama and Theatre /49", + "smmlu/easy/drama and Theatre /25", + "smmlu/easy/drama and Theatre /17", + "smmlu/easy/drama and Theatre /78", + "smmlu/easy/drama and Theatre /13", + "smmlu/easy/drama and Theatre /19", + "smmlu/easy/drama and Theatre /79", + "smmlu/easy/drama and Theatre /45", + "smmlu/easy/drama and Theatre /63", + "smmlu/easy/drama and Theatre /88", + "smmlu/easy/science/157", + "smmlu/easy/science/140", + "smmlu/easy/science/133", + "smmlu/easy/science/9", + "smmlu/easy/science/50", + "smmlu/easy/science/37", + "smmlu/easy/science/34", + "smmlu/easy/science/76", + "smmlu/easy/science/156", + "smmlu/easy/science/14", + "smmlu/easy/science/38", + "smmlu/easy/science/158", + "smmlu/easy/science/64", + "smmlu/easy/science/47", + "smmlu/easy/science/114", + "smmlu/easy/science/164", + "smmlu/easy/science/86", + "smmlu/easy/science/54", + "smmlu/easy/science/137", + "smmlu/easy/science/101", + "smmlu/easy/science/56", + "smmlu/easy/science/53", + "smmlu/easy/science/84", + "smmlu/easy/science/74", + "smmlu/easy/science/129", + "smmlu/easy/science/45", + "smmlu/easy/science/15", + "smmlu/easy/science/3", + "smmlu/easy/science/11", + "smmlu/easy/science/162", + "smmlu/easy/science/44", + "smmlu/easy/science/124" + ], + [ + "smmlu/easy/Arts/81", + "smmlu/easy/Arts/33", + "smmlu/easy/Arts/15", + "smmlu/easy/Arts/61", + "smmlu/easy/Arts/41", + "smmlu/easy/Arts/23", + "smmlu/easy/Arts/51", + "smmlu/easy/Arts/72", + "smmlu/easy/Arts/60", + "smmlu/easy/Arts/38", + "smmlu/easy/Arts/9", + "smmlu/easy/Arts/58", + "smmlu/easy/Arts/40", + "smmlu/easy/Arts/78", + "smmlu/easy/Arts/65", + "smmlu/easy/Arts/95", + "smmlu/easy/Arts/80", + "smmlu/easy/Arts/100", + "smmlu/easy/Arts/89", + "smmlu/easy/Arts/1", + "smmlu/easy/Buddhism/102", + "smmlu/easy/Buddhism/18", + "smmlu/easy/Buddhism/30", + "smmlu/easy/Buddhism/70", + "smmlu/easy/Buddhism/9", + "smmlu/easy/Buddhism/52", + "smmlu/easy/Buddhism/31", + "smmlu/easy/Buddhism/40", + "smmlu/easy/Buddhism/110", + "smmlu/easy/Buddhism/129", + "smmlu/easy/Buddhism/3", + "smmlu/easy/Buddhism/41", + "smmlu/easy/Buddhism/74", + "smmlu/easy/Buddhism/39", + "smmlu/easy/Buddhism/80", + "smmlu/easy/Buddhism/26", + "smmlu/easy/Buddhism/118", + "smmlu/easy/Buddhism/137", + "smmlu/easy/Buddhism/79", + "smmlu/easy/Buddhism/59", + "smmlu/easy/Buddhism/86", + "smmlu/easy/Buddhism/66", + "smmlu/easy/Buddhism/121", + "smmlu/easy/Buddhism/128", + "smmlu/easy/Buddhism/78", + "smmlu/easy/Buddhism/73", + "smmlu/easy/Buddhism/11", + "smmlu/easy/Buddhism/56", + "smmlu/easy/Buddhism/123", + "smmlu/easy/Buddhism/65", + "smmlu/easy/Buddhism/49", + "smmlu/easy/Buddhism/96", + "smmlu/easy/Buddhism/61", + "smmlu/easy/Catholicism/75", + "smmlu/easy/Catholicism/62", + "smmlu/easy/Catholicism/87", + "smmlu/easy/Catholicism/1", + "smmlu/easy/Catholicism/119", + "smmlu/easy/Catholicism/98", + "smmlu/easy/Catholicism/58", + "smmlu/easy/Catholicism/46", + "smmlu/easy/Catholicism/83", + "smmlu/easy/Catholicism/26", + "smmlu/easy/Catholicism/67", + "smmlu/easy/Catholicism/34", + "smmlu/easy/Catholicism/37", + "smmlu/easy/Catholicism/100", + "smmlu/easy/Catholicism/69", + "smmlu/easy/Catholicism/77", + "smmlu/easy/Catholicism/116", + "smmlu/easy/Catholicism/102", + "smmlu/easy/Catholicism/128", + "smmlu/easy/Catholicism/48", + "smmlu/easy/Catholicism/32", + "smmlu/easy/Catholicism/122", + "smmlu/easy/Catholicism/99", + "smmlu/easy/Catholicism/90", + "smmlu/easy/Catholicism/96", + "smmlu/easy/Catholicism/60", + "smmlu/easy/Christianity/26", + "smmlu/easy/Christianity/39", + "smmlu/easy/Christianity/62", + "smmlu/easy/Christianity/76", + "smmlu/easy/Christianity/33", + "smmlu/easy/Christianity/114", + "smmlu/easy/Christianity/117", + "smmlu/easy/Christianity/61", + "smmlu/easy/Christianity/55", + "smmlu/easy/Christianity/31", + "smmlu/easy/Christianity/51", + "smmlu/easy/Christianity/3", + "smmlu/easy/Christianity/47", + "smmlu/easy/Christianity/63", + "smmlu/easy/Christianity/22", + "smmlu/easy/Christianity/126", + "smmlu/easy/Christianity/34", + "smmlu/easy/Christianity/15", + "smmlu/easy/Christianity/5", + "smmlu/easy/Christianity/84", + "smmlu/easy/Christianity/18", + "smmlu/easy/Christianity/71", + "smmlu/easy/Christianity/66", + "smmlu/easy/Christianity/92", + "smmlu/easy/Christianity/87", + "smmlu/easy/Christianity/14", + "smmlu/easy/Civics/88", + "smmlu/easy/Civics/2", + "smmlu/easy/Civics/11", + "smmlu/easy/Civics/100", + "smmlu/easy/Civics/110", + "smmlu/easy/Civics/121", + "smmlu/easy/Civics/47", + "smmlu/easy/Civics/22", + "smmlu/easy/Civics/49", + "smmlu/easy/Civics/90", + "smmlu/easy/Civics/28", + "smmlu/easy/Civics/65", + "smmlu/easy/Civics/6", + "smmlu/easy/Civics/34", + "smmlu/easy/Civics/55", + "smmlu/easy/Civics/102", + "smmlu/easy/Civics/85", + "smmlu/easy/Civics/93", + "smmlu/easy/Civics/48", + "smmlu/easy/Civics/58", + "smmlu/easy/Civics/16", + "smmlu/easy/Civics/20", + "smmlu/easy/Civics/33", + "smmlu/easy/Civics/54", + "smmlu/easy/Civics/23", + "smmlu/easy/Eastern music/139", + "smmlu/easy/Eastern music/47", + "smmlu/easy/Eastern music/105", + "smmlu/easy/Eastern music/15", + "smmlu/easy/Eastern music/70", + "smmlu/easy/Eastern music/136", + "smmlu/easy/Eastern music/61", + "smmlu/easy/Eastern music/134", + "smmlu/easy/Eastern music/13", + "smmlu/easy/Eastern music/40", + "smmlu/easy/Eastern music/116", + "smmlu/easy/Eastern_music/1", + "smmlu/easy/Eastern music/82", + "smmlu/easy/Eastern music/79", + "smmlu/easy/Eastern music/102", + "smmlu/easy/Eastern music/94", + "smmlu/easy/Eastern music/142", + "smmlu/easy/Eastern music/18", + "smmlu/easy/Eastern music/123", + "smmlu/easy/Eastern music/74", + "smmlu/easy/Eastern music/133", + "smmlu/easy/Eastern music/135", + "smmlu/easy/Eastern music/37", + "smmlu/easy/Eastern music/30", + "smmlu/easy/Eastern music/138", + "smmlu/easy/Eastern music/101", + "smmlu/easy/Eastern music/103", + "smmlu/easy/Eastern music/10", + "smmlu/easy/Geography/91", + "smmlu/easy/Geography/17", + "smmlu/easy/Geography/37", + "smmlu/easy/Geography/58", + "smmlu/easy/Geography/113", + "smmlu/easy/Geography/20", + "smmlu/easy/Geography/54", + "smmlu/easy/Geography/98", + "smmlu/easy/Geography/33", + "smmlu/easy/Geography/76", + "smmlu/easy/Geography/38", + "smmlu/easy/Geography/62", + "smmlu/easy/Geography/117", + "smmlu/easy/Geography/100", + "smmlu/easy/Geography/11", + "smmlu/easy/Geography/99", + "smmlu/easy/Geography/60", + "smmlu/easy/Geography/13", + "smmlu/easy/Geography/18", + "smmlu/easy/Geography/10", + "smmlu/easy/Geography/94", + "smmlu/easy/Geography/78", + "smmlu/easy/Geography/93", + "smmlu/easy/Health and Physical Science/110", + "smmlu/easy/Health and Physical Science/60", + "smmlu/easy/Health and Physical Science/99", + "smmlu/easy/Health and Physical Science/73", + "smmlu/easy/Health and Physical Science/31", + "smmlu/easy/Health and Physical Science/39", + "smmlu/easy/Health and Physical Science/80", + "smmlu/easy/Health and Physical Science/37", + "smmlu/easy/Health and Physical Science/106", + "smmlu/easy/Health and Physical Science/76", + "smmlu/easy/Health and Physical Science/44", + "smmlu/easy/Health and Physical Science/16", + "smmlu/easy/Health and Physical Science/122", + "smmlu/easy/Health and Physical Science/30", + "smmlu/easy/Health and Physical Science/74", + "smmlu/easy/Health and Physical Science/41", + "smmlu/easy/Health and Physical Science/46", + "smmlu/easy/Health and Physical Science/36", + "smmlu/easy/Health and Physical Science/102", + "smmlu/easy/Health and Physical Science/21", + "smmlu/easy/Health and Physical Science/105", + "smmlu/easy/Health and Physical Science/51", + "smmlu/easy/Health and Physical Science/93", + "smmlu/easy/Health and Physical Science/58", + "smmlu/easy/Health and Physical Science/129", + "smmlu/easy/Health and Physical Science/48", + "smmlu/easy/History/124", + "smmlu/easy/History/119", + "smmlu/easy/History/11", + "smmlu/easy/History/29", + "smmlu/easy/History/6", + "smmlu/easy/History/19", + "smmlu/easy/History/107", + "smmlu/easy/History/83", + "smmlu/easy/History/67", + "smmlu/easy/History/14", + "smmlu/easy/History/95", + "smmlu/easy/History/81", + "smmlu/easy/History/10", + "smmlu/easy/History/39", + "smmlu/easy/History/136", + "smmlu/easy/History/74", + "smmlu/easy/History/68", + "smmlu/easy/History/75", + "smmlu/easy/History/65", + "smmlu/easy/History/4", + "smmlu/easy/History/35", + "smmlu/easy/History/70", + "smmlu/easy/History/36", + "smmlu/easy/History/47", + "smmlu/easy/History/114", + "smmlu/easy/History/27", + "smmlu/easy/History/127", + "smmlu/easy/History/90", + "smmlu/easy/History/18", + "smmlu/easy/History/140", + "smmlu/easy/History/142", + "smmlu/easy/Islam/33", + "smmlu/easy/Islam/15", + "smmlu/easy/Islam/36", + "smmlu/easy/Islam/91", + "smmlu/easy/Islam/7", + "smmlu/easy/Islam/103", + "smmlu/easy/Islam/44", + "smmlu/easy/Islam/3", + "smmlu/easy/Islam/34", + "smmlu/easy/Islam/30", + "smmlu/easy/Islam/66", + "smmlu/easy/Islam/8", + "smmlu/easy/Islam/4", + "smmlu/easy/Islam/85", + "smmlu/easy/Islam/63", + "smmlu/easy/Islam/23", + "smmlu/easy/Islam/92", + "smmlu/easy/Islam/12", + "smmlu/easy/Islam/29", + "smmlu/easy/Islam/49", + "smmlu/easy/Islam/10", + "smmlu/easy/Sinhala language and literature/10", + "smmlu/easy/Sinhala language and literature/45", + "smmlu/easy/Sinhala language and literature/5", + "smmlu/easy/Sinhala_language_and_literature/1", + "smmlu/easy/Sinhala language and literature/67", + "smmlu/easy/Sinhala language and literature/47", + "smmlu/easy/Sinhala language and literature/92", + "smmlu/easy/Sinhala language and literature/63", + "smmlu/easy/Sinhala language and literature/142", + "smmlu/easy/Sinhala language and literature/73", + "smmlu/easy/Sinhala language and literature/143", + "smmlu/easy/Sinhala language and literature/8", + "smmlu/easy/Sinhala language and literature/57", + "smmlu/easy/Sinhala language and literature/80", + "smmlu/easy/Sinhala language and literature/25", + "smmlu/easy/Sinhala language and literature/134", + "smmlu/easy/Sinhala language and literature/85", + "smmlu/easy/Sinhala language and literature/42", + "smmlu/easy/Sinhala language and literature/50", + "smmlu/easy/Sinhala language and literature/117", + "smmlu/easy/Sinhala language and literature/2", + "smmlu/easy/Sinhala language and literature/133", + "smmlu/easy/Sinhala language and literature/17", + "smmlu/easy/Sinhala language and literature/153", + "smmlu/easy/Sinhala language and literature/55", + "smmlu/easy/Sinhala language and literature/49", + "smmlu/easy/Sinhala language and literature/20", + "smmlu/easy/Sinhala language and literature/105", + "smmlu/easy/Sinhala language and literature/109", + "smmlu/easy/Sinhala language and literature/40", + "smmlu/easy/Sinhala language and literature/121", + "smmlu/easy/dancing/33", + "smmlu/easy/dancing/35", + "smmlu/easy/dancing/48", + "smmlu/easy/dancing/27", + "smmlu/easy/dancing/58", + "smmlu/easy/dancing/79", + "smmlu/easy/dancing/87", + "smmlu/easy/dancing/23", + "smmlu/easy/dancing/83", + "smmlu/easy/dancing/92", + "smmlu/easy/dancing/80", + "smmlu/easy/dancing/108", + "smmlu/easy/dancing/78", + "smmlu/easy/dancing/64", + "smmlu/easy/dancing/20", + "smmlu/easy/dancing/90", + "smmlu/easy/dancing/36", + "smmlu/easy/dancing/14", + "smmlu/easy/dancing/30", + "smmlu/easy/dancing/99", + "smmlu/easy/dancing/104", + "smmlu/easy/dancing/38", + "smmlu/easy/drama and Theatre /71", + "smmlu/easy/drama_and_theatre/1", + "smmlu/easy/drama and Theatre /4", + "smmlu/easy/drama and Theatre /109", + "smmlu/easy/drama and Theatre /69", + "smmlu/easy/drama and Theatre /108", + "smmlu/easy/drama and Theatre /21", + "smmlu/easy/drama and Theatre /8", + "smmlu/easy/drama and Theatre /123", + "smmlu/easy/drama and Theatre /122", + "smmlu/easy/drama and Theatre /74", + "smmlu/easy/drama and Theatre /48", + "smmlu/easy/drama and Theatre /83", + "smmlu/easy/drama and Theatre /65", + "smmlu/easy/drama and Theatre /38", + "smmlu/easy/drama and Theatre /23", + "smmlu/easy/drama and Theatre /87", + "smmlu/easy/drama and Theatre /56", + "smmlu/easy/drama and Theatre /55", + "smmlu/easy/drama and Theatre /93", + "smmlu/easy/drama and Theatre /110", + "smmlu/easy/drama and Theatre /112", + "smmlu/easy/drama and Theatre /53", + "smmlu/easy/drama and Theatre /77", + "smmlu/easy/drama and Theatre /72", + "smmlu/easy/science/75", + "smmlu/easy/science/23", + "smmlu/easy/science/57", + "smmlu/easy/science/59", + "smmlu/easy/science/135", + "smmlu/easy/science/51", + "smmlu/easy/science/106", + "smmlu/easy/science/87", + "smmlu/easy/science/143", + "smmlu/easy/science/70", + "smmlu/easy/science/60", + "smmlu/easy/science/71", + "smmlu/easy/science/5", + "smmlu/easy/science/146", + "smmlu/easy/science/96", + "smmlu/easy/science/91", + "smmlu/easy/science/104", + "smmlu/easy/science/22", + "smmlu/easy/science/82", + "smmlu/easy/science/121", + "smmlu/easy/science/136", + "smmlu/easy/science/26", + "smmlu/easy/science/115", + "smmlu/easy/science/98", + "smmlu/easy/science/46", + "smmlu/easy/science/155", + "smmlu/easy/science/131", + "smmlu/easy/science/32", + "smmlu/easy/science/109", + "smmlu/easy/science/139", + "smmlu/easy/science/116", + "smmlu/easy/science/17", + "smmlu/easy/science/73" + ], + [ + "smmlu/easy/Arts/25", + "smmlu/easy/Arts/98", + "smmlu/easy/Arts/85", + "smmlu/easy/Arts/74", + "smmlu/easy/Arts/94", + "smmlu/easy/Arts/43", + "smmlu/easy/Arts/52", + "smmlu/easy/Arts/91", + "smmlu/easy/Arts/17", + "smmlu/easy/Arts/75", + "smmlu/easy/Arts/70", + "smmlu/easy/Arts/88", + "smmlu/easy/Arts/56", + "smmlu/easy/Arts/71", + "smmlu/easy/Arts/67", + "smmlu/easy/Arts/26", + "smmlu/easy/Arts/83", + "smmlu/easy/Arts/36", + "smmlu/easy/Arts/86", + "smmlu/easy/Arts/21", + "smmlu/easy/Buddhism/10", + "smmlu/easy/Buddhism/63", + "smmlu/easy/Buddhism/23", + "smmlu/easy/Buddhism/42", + "smmlu/easy/Buddhism/155", + "smmlu/easy/Buddhism/112", + "smmlu/easy/Buddhism/108", + "smmlu/easy/Buddhism/101", + "smmlu/easy/Buddhism/98", + "smmlu/easy/Buddhism/145", + "smmlu/easy/Buddhism/29", + "smmlu/easy/Buddhism/76", + "smmlu/easy/Buddhism/17", + "smmlu/easy/Buddhism/22", + "smmlu/easy/Buddhism/25", + "smmlu/easy/Buddhism/136", + "smmlu/easy/Buddhism/24", + "smmlu/easy/Buddhism/153", + "smmlu/easy/Buddhism/14", + "smmlu/easy/Buddhism/68", + "smmlu/easy/Buddhism/140", + "smmlu/easy/Buddhism/34", + "smmlu/easy/Buddhism/95", + "smmlu/easy/Buddhism/2", + "smmlu/easy/Buddhism/8", + "smmlu/easy/Buddhism/119", + "smmlu/easy/Buddhism/148", + "smmlu/easy/Buddhism/27", + "smmlu/easy/Buddhism/32", + "smmlu/easy/Buddhism/122", + "smmlu/easy/Buddhism/141", + "smmlu/easy/Buddhism/72", + "smmlu/easy/Catholicism/123", + "smmlu/easy/Catholicism/61", + "smmlu/easy/Catholicism/27", + "smmlu/easy/Catholicism/110", + "smmlu/easy/Catholicism/10", + "smmlu/easy/Catholicism/113", + "smmlu/easy/Catholicism/105", + "smmlu/easy/Catholicism/89", + "smmlu/easy/Catholicism/14", + "smmlu/easy/Catholicism/50", + "smmlu/easy/Catholicism/23", + "smmlu/easy/Catholicism/28", + "smmlu/easy/Catholicism/8", + "smmlu/easy/Catholicism/35", + "smmlu/easy/Catholicism/72", + "smmlu/easy/Catholicism/36", + "smmlu/easy/Catholicism/3", + "smmlu/easy/Catholicism/80", + "smmlu/easy/Catholicism/25", + "smmlu/easy/Catholicism/127", + "smmlu/easy/Catholicism/40", + "smmlu/easy/Catholicism/17", + "smmlu/easy/Catholicism/5", + "smmlu/easy/Catholicism/66", + "smmlu/easy/Catholicism/132", + "smmlu/easy/Catholicism/94", + "smmlu/easy/Catholicism/56", + "smmlu/easy/Christianity/28", + "smmlu/easy/Christianity/75", + "smmlu/easy/Christianity/54", + "smmlu/easy/Christianity/2", + "smmlu/easy/Christianity/7", + "smmlu/easy/Christianity/101", + "smmlu/easy/Christianity/77", + "smmlu/easy/Christianity/83", + "smmlu/easy/Christianity/69", + "smmlu/easy/Christianity/130", + "smmlu/easy/Christianity/80", + "smmlu/easy/Christianity/121", + "smmlu/easy/Christianity/110", + "smmlu/easy/Christianity/37", + "smmlu/easy/Christianity/90", + "smmlu/easy/Christianity/8", + "smmlu/easy/Christianity/119", + "smmlu/easy/Christianity/19", + "smmlu/easy/Christianity/48", + "smmlu/easy/Christianity/44", + "smmlu/easy/Christianity/10", + "smmlu/easy/Christianity/29", + "smmlu/easy/Christianity/88", + "smmlu/easy/Christianity/104", + "smmlu/easy/Christianity/6", + "smmlu/easy/Christianity/27", + "smmlu/easy/Civics/113", + "smmlu/easy/Civics/51", + "smmlu/easy/Civics/15", + "smmlu/easy/Civics/103", + "smmlu/easy/Civics/64", + "smmlu/easy/Civics/114", + "smmlu/easy/Civics/30", + "smmlu/easy/Civics/73", + "smmlu/easy/Civics/76", + "smmlu/easy/Civics/68", + "smmlu/easy/Civics/112", + "smmlu/easy/Civics/53", + "smmlu/easy/Civics/63", + "smmlu/easy/Civics/101", + "smmlu/easy/Civics/29", + "smmlu/easy/Civics/60", + "smmlu/easy/Civics/80", + "smmlu/easy/Civics/18", + "smmlu/easy/Civics/27", + "smmlu/easy/Civics/4", + "smmlu/easy/Civics/59", + "smmlu/easy/Civics/87", + "smmlu/easy/Civics/31", + "smmlu/easy/Civics/70", + "smmlu/easy/Eastern music/121", + "smmlu/easy/Eastern music/49", + "smmlu/easy/Eastern music/57", + "smmlu/easy/Eastern music/31", + "smmlu/easy/Eastern music/33", + "smmlu/easy/Eastern music/9", + "smmlu/easy/Eastern music/23", + "smmlu/easy/Eastern music/140", + "smmlu/easy/Eastern music/118", + "smmlu/easy/Eastern music/117", + "smmlu/easy/Eastern music/63", + "smmlu/easy/Eastern music/68", + "smmlu/easy/Eastern music/52", + "smmlu/easy/Eastern music/26", + "smmlu/easy/Eastern music/131", + "smmlu/easy/Eastern music/29", + "smmlu/easy/Eastern music/3", + "smmlu/easy/Eastern music/108", + "smmlu/easy/Eastern music/112", + "smmlu/easy/Eastern music/84", + "smmlu/easy/Eastern music/93", + "smmlu/easy/Eastern music/87", + "smmlu/easy/Eastern music/27", + "smmlu/easy/Eastern music/6", + "smmlu/easy/Eastern music/38", + "smmlu/easy/Eastern music/19", + "smmlu/easy/Eastern music/25", + "smmlu/easy/Eastern music/22", + "smmlu/easy/Eastern music/66", + "smmlu/easy/Geography/43", + "smmlu/easy/Geography/112", + "smmlu/easy/Geography/50", + "smmlu/easy/Geography/35", + "smmlu/easy/Geography/67", + "smmlu/easy/Geography/80", + "smmlu/easy/Geography/48", + "smmlu/easy/Geography/39", + "smmlu/easy/Geography/12", + "smmlu/easy/Geography/70", + "smmlu/easy/Geography/74", + "smmlu/easy/Geography/47", + "smmlu/easy/Geography/41", + "smmlu/easy/Geography/28", + "smmlu/easy/Geography/77", + "smmlu/easy/Geography/8", + "smmlu/easy/Geography/51", + "smmlu/easy/Geography/57", + "smmlu/easy/Geography/82", + "smmlu/easy/Geography/9", + "smmlu/easy/Geography/83", + "smmlu/easy/Geography/31", + "smmlu/easy/Geography/68", + "smmlu/easy/Health and Physical Science/95", + "smmlu/easy/Health and Physical Science/69", + "smmlu/easy/Health and Physical Science/84", + "smmlu/easy/Health and Physical Science/43", + "smmlu/easy/Health and Physical Science/19", + "smmlu/easy/Health and Physical Science/5", + "smmlu/easy/Health and Physical Science/124", + "smmlu/easy/Health and Physical Science/12", + "smmlu/easy/Health and Physical Science/45", + "smmlu/easy/Health and Physical Science/101", + "smmlu/easy/Health and Physical Science/119", + "smmlu/easy/Health and Physical Science/89", + "smmlu/easy/Health and Physical Science/59", + "smmlu/easy/Health and Physical Science/42", + "smmlu/easy/Health and Physical Science/6", + "smmlu/easy/Health and Physical Science/33", + "smmlu/easy/Health and Physical Science/18", + "smmlu/easy/Health and Physical Science/120", + "smmlu/easy/Health and Physical Science/4", + "smmlu/easy/Health and Physical Science/104", + "smmlu/easy/Health and Physical Science/34", + "smmlu/easy/Health and Physical Science/116", + "smmlu/easy/Health and Physical Science/52", + "smmlu/easy/Health and Physical Science/62", + "smmlu/easy/Health and Physical Science/130", + "smmlu/easy/Health and Physical Science/64", + "smmlu/easy/History/57", + "smmlu/easy/History/115", + "smmlu/easy/History/62", + "smmlu/easy/History/20", + "smmlu/easy/History/30", + "smmlu/easy/History/38", + "smmlu/easy/History/113", + "smmlu/easy/History/105", + "smmlu/easy/History/101", + "smmlu/easy/History/5", + "smmlu/easy/History/102", + "smmlu/easy/History/26", + "smmlu/easy/History/66", + "smmlu/easy/History/9", + "smmlu/easy/History/85", + "smmlu/easy/History/149", + "smmlu/easy/History/137", + "smmlu/easy/History/32", + "smmlu/easy/History/69", + "smmlu/easy/History/141", + "smmlu/easy/History/133", + "smmlu/easy/History/106", + "smmlu/easy/History/82", + "smmlu/easy/History/33", + "smmlu/easy/History/78", + "smmlu/easy/History/64", + "smmlu/easy/History/129", + "smmlu/easy/History/135", + "smmlu/easy/History/110", + "smmlu/easy/History/97", + "smmlu/easy/Islam/9", + "smmlu/easy/Islam/73", + "smmlu/easy/Islam/39", + "smmlu/easy/Islam/80", + "smmlu/easy/Islam/42", + "smmlu/easy/Islam/106", + "smmlu/easy/Islam/64", + "smmlu/easy/Islam/37", + "smmlu/easy/Islam/72", + "smmlu/easy/Islam/38", + "smmlu/easy/Islam/5", + "smmlu/easy/Islam/6", + "smmlu/easy/Islam/79", + "smmlu/easy/Islam/97", + "smmlu/easy/Islam/88", + "smmlu/easy/Islam/13", + "smmlu/easy/Islam/101", + "smmlu/easy/Islam/75", + "smmlu/easy/Islam/74", + "smmlu/easy/Islam/53", + "smmlu/easy/Islam/71", + "smmlu/easy/Islam/86", + "smmlu/easy/Sinhala language and literature/72", + "smmlu/easy/Sinhala language and literature/65", + "smmlu/easy/Sinhala language and literature/140", + "smmlu/easy/Sinhala language and literature/126", + "smmlu/easy/Sinhala language and literature/56", + "smmlu/easy/Sinhala language and literature/83", + "smmlu/easy/Sinhala language and literature/26", + "smmlu/easy/Sinhala language and literature/11", + "smmlu/easy/Sinhala language and literature/150", + "smmlu/easy/Sinhala language and literature/61", + "smmlu/easy/Sinhala language and literature/33", + "smmlu/easy/Sinhala language and literature/118", + "smmlu/easy/Sinhala language and literature/102", + "smmlu/easy/Sinhala language and literature/100", + "smmlu/easy/Sinhala language and literature/97", + "smmlu/easy/Sinhala language and literature/144", + "smmlu/easy/Sinhala language and literature/113", + "smmlu/easy/Sinhala language and literature/151", + "smmlu/easy/Sinhala language and literature/78", + "smmlu/easy/Sinhala language and literature/34", + "smmlu/easy/Sinhala language and literature/103", + "smmlu/easy/Sinhala language and literature/132", + "smmlu/easy/Sinhala language and literature/37", + "smmlu/easy/Sinhala language and literature/110", + "smmlu/easy/Sinhala language and literature/108", + "smmlu/easy/Sinhala language and literature/74", + "smmlu/easy/Sinhala language and literature/141", + "smmlu/easy/Sinhala language and literature/58", + "smmlu/easy/Sinhala language and literature/71", + "smmlu/easy/Sinhala language and literature/131", + "smmlu/easy/Sinhala language and literature/36", + "smmlu/easy/dancing/105", + "smmlu/easy/dancing/3", + "smmlu/easy/dancing/26", + "smmlu/easy/dancing/70", + "smmlu/easy/dancing/68", + "smmlu/easy/dancing/42", + "smmlu/easy/dancing/5", + "smmlu/easy/dancing/65", + "smmlu/easy/dancing/24", + "smmlu/easy/dancing/13", + "smmlu/easy/dancing/60", + "smmlu/easy/dancing/54", + "smmlu/easy/dancing/97", + "smmlu/easy/dancing/41", + "smmlu/easy/dancing/88", + "smmlu/easy/dancing/67", + "smmlu/easy/dancing/43", + "smmlu/easy/dancing/107", + "smmlu/easy/dancing/28", + "smmlu/easy/dancing/100", + "smmlu/easy/dancing/77", + "smmlu/easy/dancing/106", + "smmlu/easy/drama and Theatre /32", + "smmlu/easy/drama and Theatre /126", + "smmlu/easy/drama and Theatre /100", + "smmlu/easy/drama and Theatre /59", + "smmlu/easy/drama and Theatre /44", + "smmlu/easy/drama and Theatre /85", + "smmlu/easy/drama and Theatre /99", + "smmlu/easy/drama and Theatre /116", + "smmlu/easy/drama and Theatre /11", + "smmlu/easy/drama and Theatre /97", + "smmlu/easy/drama and Theatre /75", + "smmlu/easy/drama and Theatre /96", + "smmlu/easy/drama and Theatre /9", + "smmlu/easy/drama and Theatre /103", + "smmlu/easy/drama and Theatre /84", + "smmlu/easy/drama and Theatre /95", + "smmlu/easy/drama and Theatre /121", + "smmlu/easy/drama and Theatre /62", + "smmlu/easy/drama and Theatre /82", + "smmlu/easy/drama and Theatre /81", + "smmlu/easy/drama and Theatre /27", + "smmlu/easy/drama and Theatre /14", + "smmlu/easy/drama and Theatre /36", + "smmlu/easy/drama and Theatre /113", + "smmlu/easy/drama and Theatre /46", + "smmlu/easy/science/118", + "smmlu/easy/science/134", + "smmlu/easy/science/145", + "smmlu/easy/science/25", + "smmlu/easy/science/2", + "smmlu/easy/science/152", + "smmlu/easy/science/120", + "smmlu/easy/science/6", + "smmlu/easy/science/111", + "smmlu/easy/science/105", + "smmlu/easy/science/147", + "smmlu/easy/science/31", + "smmlu/easy/science/58", + "smmlu/easy/science/35", + "smmlu/easy/science/88", + "smmlu/easy/science/130", + "smmlu/easy/science/48", + "smmlu/easy/science/83", + "smmlu/easy/science/28", + "smmlu/easy/science/113", + "smmlu/easy/science/149", + "smmlu/easy/science/95", + "smmlu/easy/science/72", + "smmlu/easy/science/7", + "smmlu/easy/science/61", + "smmlu/easy/science/21", + "smmlu/easy/science/67", + "smmlu/easy/science/19", + "smmlu/easy/science/112", + "smmlu/easy/science/163", + "smmlu/easy/science/62", + "smmlu/easy/science/102", + "smmlu/easy/science/27" + ], + [ + "smmlu/easy/Arts/32", + "smmlu/easy/Arts/14", + "smmlu/easy/Arts/27", + "smmlu/easy/Arts/34", + "smmlu/easy/Arts/50", + "smmlu/easy/Arts/45", + "smmlu/easy/Arts/20", + "smmlu/easy/Arts/92", + "smmlu/easy/Arts/28", + "smmlu/easy/Arts/47", + "smmlu/easy/Arts/22", + "smmlu/easy/Arts/79", + "smmlu/easy/Arts/4", + "smmlu/easy/Arts/82", + "smmlu/easy/Arts/10", + "smmlu/easy/Arts/62", + "smmlu/easy/Arts/49", + "smmlu/easy/Arts/5", + "smmlu/easy/Arts/12", + "smmlu/easy/Arts/6", + "smmlu/easy/Buddhism/46", + "smmlu/easy/Buddhism/125", + "smmlu/easy/Buddhism/48", + "smmlu/easy/Buddhism/12", + "smmlu/easy/Buddhism/21", + "smmlu/easy/Buddhism/100", + "smmlu/easy/Buddhism/91", + "smmlu/easy/Buddhism/82", + "smmlu/easy/Buddhism/50", + "smmlu/easy/Buddhism/152", + "smmlu/easy/Buddhism/43", + "smmlu/easy/Buddhism/94", + "smmlu/easy/Buddhism/38", + "smmlu/easy/Buddhism/57", + "smmlu/easy/Buddhism/109", + "smmlu/easy/Buddhism/6", + "smmlu/easy/Buddhism/104", + "smmlu/easy/Buddhism/92", + "smmlu/easy/Buddhism/105", + "smmlu/easy/Buddhism/147", + "smmlu/easy/Buddhism/45", + "smmlu/easy/Buddhism/120", + "smmlu/easy/Buddhism/81", + "smmlu/easy/Buddhism/106", + "smmlu/easy/Buddhism/33", + "smmlu/easy/Buddhism/15", + "smmlu/easy/Buddhism/44", + "smmlu/easy/Buddhism/134", + "smmlu/easy/Buddhism/143", + "smmlu/easy/Buddhism/75", + "smmlu/easy/Buddhism/127", + "smmlu/easy/Buddhism/54", + "smmlu/easy/Catholicism/81", + "smmlu/easy/Catholicism/24", + "smmlu/easy/Catholicism/126", + "smmlu/easy/Catholicism/31", + "smmlu/easy/Catholicism/29", + "smmlu/easy/Catholicism/109", + "smmlu/easy/Catholicism/21", + "smmlu/easy/Catholicism/85", + "smmlu/easy/Catholicism/33", + "smmlu/easy/Catholicism/54", + "smmlu/easy/Catholicism/65", + "smmlu/easy/Catholicism/51", + "smmlu/easy/Catholicism/73", + "smmlu/easy/Catholicism/117", + "smmlu/easy/Catholicism/38", + "smmlu/easy/Catholicism/129", + "smmlu/easy/Catholicism/20", + "smmlu/easy/Catholicism/79", + "smmlu/easy/Catholicism/101", + "smmlu/easy/Catholicism/112", + "smmlu/easy/Catholicism/68", + "smmlu/easy/Catholicism/91", + "smmlu/easy/Catholicism/6", + "smmlu/easy/Catholicism/74", + "smmlu/easy/Catholicism/9", + "smmlu/easy/Catholicism/59", + "smmlu/easy/Catholicism/130", + "smmlu/easy/Christianity/9", + "smmlu/easy/Christianity/107", + "smmlu/easy/Christianity/45", + "smmlu/easy/Christianity/108", + "smmlu/easy/Christianity/100", + "smmlu/easy/Christianity/93", + "smmlu/easy/Christianity/46", + "smmlu/easy/Christianity/36", + "smmlu/easy/Christianity/21", + "smmlu/easy/Christianity/73", + "smmlu/easy/Christianity/89", + "smmlu/easy/Christianity/99", + "smmlu/easy/Christianity/64", + "smmlu/easy/Christianity/38", + "smmlu/easy/Christianity/59", + "smmlu/easy/Christianity/70", + "smmlu/easy/Christianity/56", + "smmlu/easy/Christianity/12", + "smmlu/easy/Christianity/43", + "smmlu/easy/Christianity/112", + "smmlu/easy/Christianity/81", + "smmlu/easy/Christianity/98", + "smmlu/easy/Christianity/23", + "smmlu/easy/Christianity/68", + "smmlu/easy/Christianity/113", + "smmlu/easy/Christianity/30", + "smmlu/easy/Civics/104", + "smmlu/easy/Civics/3", + "smmlu/easy/Civics/91", + "smmlu/easy/Civics/52", + "smmlu/easy/Civics/44", + "smmlu/easy/Civics/74", + "smmlu/easy/Civics/99", + "smmlu/easy/Civics/109", + "smmlu/easy/Civics/72", + "smmlu/easy/Civics/78", + "smmlu/easy/Civics/69", + "smmlu/easy/Civics/98", + "smmlu/easy/Civics/107", + "smmlu/easy/Civics/75", + "smmlu/easy/Civics/50", + "smmlu/easy/Civics/35", + "smmlu/easy/Civics/82", + "smmlu/easy/Civics/9", + "smmlu/easy/Civics/46", + "smmlu/easy/Civics/81", + "smmlu/easy/Civics/24", + "smmlu/easy/Civics/57", + "smmlu/easy/Civics/118", + "smmlu/easy/Civics/38", + "smmlu/easy/Eastern music/115", + "smmlu/easy/Eastern music/17", + "smmlu/easy/Eastern music/110", + "smmlu/easy/Eastern music/86", + "smmlu/easy/Eastern music/120", + "smmlu/easy/Eastern music/48", + "smmlu/easy/Eastern music/42", + "smmlu/easy/Eastern music/32", + "smmlu/easy/Eastern music/141", + "smmlu/easy/Eastern music/137", + "smmlu/easy/Eastern music/45", + "smmlu/easy/Eastern music/85", + "smmlu/easy/Eastern music/11", + "smmlu/easy/Eastern music/64", + "smmlu/easy/Eastern music/106", + "smmlu/easy/Eastern music/130", + "smmlu/easy/Eastern music/107", + "smmlu/easy/Eastern music/41", + "smmlu/easy/Eastern music/81", + "smmlu/easy/Eastern music/16", + "smmlu/easy/Eastern music/113", + "smmlu/easy/Eastern music/88", + "smmlu/easy/Eastern music/36", + "smmlu/easy/Eastern music/39", + "smmlu/easy/Eastern music/46", + "smmlu/easy/Eastern music/28", + "smmlu/easy/Eastern music/72", + "smmlu/easy/Eastern music/143", + "smmlu/easy/Eastern music/5", + "smmlu/easy/Geography/84", + "smmlu/easy/Geography/55", + "smmlu/easy/Geography/5", + "smmlu/easy/Geography/97", + "smmlu/easy/Geography/115", + "smmlu/easy/Geography/116", + "smmlu/easy/Geography/42", + "smmlu/easy/Geography/104", + "smmlu/easy/Geography/30", + "smmlu/easy/Geography/2", + "smmlu/easy/Geography/59", + "smmlu/easy/Geography/72", + "smmlu/easy/Geography/101", + "smmlu/easy/Geography/110", + "smmlu/easy/Geography/79", + "smmlu/easy/Geography/96", + "smmlu/easy/Geography/24", + "smmlu/easy/Geography/87", + "smmlu/easy/Geography/61", + "smmlu/easy/Geography/45", + "smmlu/easy/Geography/23", + "smmlu/easy/Geography/95", + "smmlu/easy/Geography/114", + "smmlu/easy/Health and Physical Science/35", + "smmlu/easy/Health and Physical Science/66", + "smmlu/easy/Health and Physical Science/77", + "smmlu/easy/Health and Physical Science/123", + "smmlu/easy/Health and Physical Science/114", + "smmlu/easy/Health and Physical Science/79", + "smmlu/easy/Health and Physical Science/117", + "smmlu/easy/Health and Physical Science/78", + "smmlu/easy/Health and Physical Science/96", + "smmlu/easy/Health and Physical Science/109", + "smmlu/easy/Health and Physical Science/56", + "smmlu/easy/Health and Physical Science/88", + "smmlu/easy/Health and Physical Science/29", + "smmlu/easy/Health and Physical Science/23", + "smmlu/easy/Health and Physical Science/128", + "smmlu/easy/Health and Physical Science/13", + "smmlu/easy/Health and Physical Science/14", + "smmlu/easy/Health and Physical Science/81", + "smmlu/easy/Health and Physical Science/10", + "smmlu/easy/Health and Physical Science/24", + "smmlu/easy/Health and Physical Science/67", + "smmlu/easy/Health and Physical Science/118", + "smmlu/easy/Health and Physical Science/90", + "smmlu/easy/Health and Physical Science/113", + "smmlu/easy/Health and Physical Science/47", + "smmlu/easy/Health and Physical Science/91", + "smmlu/easy/History/54", + "smmlu/easy/History/16", + "smmlu/easy/History/109", + "smmlu/easy/History/46", + "smmlu/easy/History/126", + "smmlu/easy/History/117", + "smmlu/easy/History/146", + "smmlu/easy/History/88", + "smmlu/easy/History/15", + "smmlu/easy/History/77", + "smmlu/easy/History/24", + "smmlu/easy/History/139", + "smmlu/easy/History/56", + "smmlu/easy/History/13", + "smmlu/easy/History/22", + "smmlu/easy/History/112", + "smmlu/easy/History/21", + "smmlu/easy/History/58", + "smmlu/easy/History/98", + "smmlu/easy/History/23", + "smmlu/easy/History/42", + "smmlu/easy/History/84", + "smmlu/easy/History/55", + "smmlu/easy/History/128", + "smmlu/easy/History/52", + "smmlu/easy/History/122", + "smmlu/easy/History/132", + "smmlu/easy/History/120", + "smmlu/easy/History/71", + "smmlu/easy/History/49", + "smmlu/easy/Islam/87", + "smmlu/easy/Islam/54", + "smmlu/easy/Islam/21", + "smmlu/easy/Islam/100", + "smmlu/easy/Islam/16", + "smmlu/easy/Islam/57", + "smmlu/easy/Islam/94", + "smmlu/easy/Islam/17", + "smmlu/easy/Islam/41", + "smmlu/easy/Islam/76", + "smmlu/easy/Islam/59", + "smmlu/easy/Islam/27", + "smmlu/easy/Islam/58", + "smmlu/easy/Islam/47", + "smmlu/easy/Islam/14", + "smmlu/easy/Islam/40", + "smmlu/easy/Islam/65", + "smmlu/easy/Islam/25", + "smmlu/easy/Islam/28", + "smmlu/easy/Islam/11", + "smmlu/easy/Islam/70", + "smmlu/easy/Islam/51", + "smmlu/easy/Sinhala language and literature/104", + "smmlu/easy/Sinhala language and literature/54", + "smmlu/easy/Sinhala language and literature/155", + "smmlu/easy/Sinhala language and literature/3", + "smmlu/easy/Sinhala language and literature/101", + "smmlu/easy/Sinhala language and literature/30", + "smmlu/easy/Sinhala language and literature/91", + "smmlu/easy/Sinhala language and literature/12", + "smmlu/easy/Sinhala language and literature/147", + "smmlu/easy/Sinhala language and literature/124", + "smmlu/easy/Sinhala language and literature/21", + "smmlu/easy/Sinhala language and literature/90", + "smmlu/easy/Sinhala language and literature/114", + "smmlu/easy/Sinhala language and literature/48", + "smmlu/easy/Sinhala language and literature/120", + "smmlu/easy/Sinhala language and literature/76", + "smmlu/easy/Sinhala language and literature/146", + "smmlu/easy/Sinhala language and literature/128", + "smmlu/easy/Sinhala language and literature/46", + "smmlu/easy/Sinhala language and literature/135", + "smmlu/easy/Sinhala language and literature/87", + "smmlu/easy/Sinhala language and literature/52", + "smmlu/easy/Sinhala language and literature/16", + "smmlu/easy/Sinhala language and literature/152", + "smmlu/easy/Sinhala language and literature/84", + "smmlu/easy/Sinhala language and literature/27", + "smmlu/easy/Sinhala language and literature/29", + "smmlu/easy/Sinhala language and literature/88", + "smmlu/easy/Sinhala language and literature/79", + "smmlu/easy/Sinhala language and literature/86", + "smmlu/easy/Sinhala language and literature/44", + "smmlu/easy/dancing/59", + "smmlu/easy/dancing/46", + "smmlu/easy/dancing/32", + "smmlu/easy/dancing/95", + "smmlu/easy/dancing/2", + "smmlu/easy/dancing/56", + "smmlu/easy/dancing/49", + "smmlu/easy/dancing/63", + "smmlu/easy/dancing/98", + "smmlu/easy/dancing/45", + "smmlu/easy/dancing/15", + "smmlu/easy/dancing/21", + "smmlu/easy/dancing/29", + "smmlu/easy/dancing/74", + "smmlu/easy/dancing/8", + "smmlu/easy/dancing/53", + "smmlu/easy/dancing/101", + "smmlu/easy/dancing/73", + "smmlu/easy/dancing/31", + "smmlu/easy/dancing/82", + "smmlu/easy/dancing/71", + "smmlu/easy/drama and Theatre /50", + "smmlu/easy/drama and Theatre /6", + "smmlu/easy/drama and Theatre /107", + "smmlu/easy/drama and Theatre /26", + "smmlu/easy/drama and Theatre /22", + "smmlu/easy/drama and Theatre /30", + "smmlu/easy/drama and Theatre /67", + "smmlu/easy/drama and Theatre /61", + "smmlu/easy/drama and Theatre /127", + "smmlu/easy/drama and Theatre /43", + "smmlu/easy/drama and Theatre /120", + "smmlu/easy/drama and Theatre /117", + "smmlu/easy/drama and Theatre /42", + "smmlu/easy/drama and Theatre /92", + "smmlu/easy/drama and Theatre /101", + "smmlu/easy/drama and Theatre /51", + "smmlu/easy/drama and Theatre /28", + "smmlu/easy/drama and Theatre /119", + "smmlu/easy/drama and Theatre /58", + "smmlu/easy/drama and Theatre /68", + "smmlu/easy/drama and Theatre /80", + "smmlu/easy/drama and Theatre /31", + "smmlu/easy/drama and Theatre /16", + "smmlu/easy/drama and Theatre /3", + "smmlu/easy/drama and Theatre /7", + "smmlu/easy/drama and Theatre /54", + "smmlu/easy/science/12", + "smmlu/easy/science/68", + "smmlu/easy/science/89", + "smmlu/easy/science/108", + "smmlu/easy/science/29", + "smmlu/easy/science/142", + "smmlu/easy/science/77", + "smmlu/easy/science/148", + "smmlu/easy/science/10", + "smmlu/easy/science/160", + "smmlu/easy/science/13", + "smmlu/easy/science/153", + "smmlu/easy/science/65", + "smmlu/easy/science/33", + "smmlu/easy/science/40", + "smmlu/easy/science/81", + "smmlu/easy/science/117", + "smmlu/easy/science/43", + "smmlu/easy/science/132", + "smmlu/easy/science/122", + "smmlu/easy/science/126", + "smmlu/easy/science/128", + "smmlu/easy/science/30", + "smmlu/easy/science/92", + "smmlu/easy/science/151", + "smmlu/easy/science/94", + "smmlu/easy/science/42", + "smmlu/easy/science/110", + "smmlu/easy/science/41", + "smmlu/easy/science/66", + "smmlu/easy/science/78", + "smmlu/easy/science/141", + "smmlu/easy/science/90" + ], + [ + "smmlu/easy/Arts/59", + "smmlu/easy/Arts/77", + "smmlu/easy/Arts/57", + "smmlu/easy/Arts/7", + "smmlu/easy/Arts/48", + "smmlu/easy/Arts/69", + "smmlu/easy/Arts/42", + "smmlu/easy/Arts/90", + "smmlu/easy/Arts/96", + "smmlu/easy/Arts/39", + "smmlu/easy/Arts/24", + "smmlu/easy/Arts/46", + "smmlu/easy/Arts/13", + "smmlu/easy/Arts/16", + "smmlu/easy/Arts/97", + "smmlu/easy/Arts/8", + "smmlu/easy/Arts/87", + "smmlu/easy/Arts/2", + "smmlu/easy/Arts/76", + "smmlu/easy/Arts/99", + "smmlu/easy/Buddhism/28", + "smmlu/easy/Buddhism/58", + "smmlu/easy/Buddhism/62", + "smmlu/easy/Buddhism/69", + "smmlu/easy/Buddhism/16", + "smmlu/easy/Buddhism/93", + "smmlu/easy/Buddhism/111", + "smmlu/easy/Buddhism/132", + "smmlu/easy/Buddhism/139", + "smmlu/easy/Buddhism/89", + "smmlu/easy/Buddhism/55", + "smmlu/easy/Buddhism/131", + "smmlu/easy/Buddhism/130", + "smmlu/easy/Buddhism/149", + "smmlu/easy/Buddhism/37", + "smmlu/easy/Buddhism/161", + "smmlu/easy/Buddhism/113", + "smmlu/easy/Buddhism/156", + "smmlu/easy/Buddhism/133", + "smmlu/easy/Buddhism/64", + "smmlu/easy/Buddhism/88", + "smmlu/easy/Buddhism/144", + "smmlu/easy/Buddhism/7", + "smmlu/easy/Buddhism/151", + "smmlu/easy/Buddhism/99", + "smmlu/easy/Buddhism/107", + "smmlu/easy/Buddhism/117", + "smmlu/easy/Buddhism/160", + "smmlu/easy/Buddhism/158", + "smmlu/easy/Buddhism/126", + "smmlu/easy/Buddhism/162", + "smmlu/easy/Buddhism/138", + "smmlu/easy/Catholicism/53", + "smmlu/easy/Catholicism/118", + "smmlu/easy/Catholicism/44", + "smmlu/easy/Catholicism/95", + "smmlu/easy/Catholicism/115", + "smmlu/easy/Catholicism/104", + "smmlu/easy/Catholicism/78", + "smmlu/easy/Catholicism/12", + "smmlu/easy/Catholicism/7", + "smmlu/easy/Catholicism/121", + "smmlu/easy/Catholicism/71", + "smmlu/easy/Catholicism/82", + "smmlu/easy/Catholicism/30", + "smmlu/easy/Catholicism/125", + "smmlu/easy/Catholicism/2", + "smmlu/easy/Catholicism/15", + "smmlu/easy/Catholicism/4", + "smmlu/easy/Catholicism/103", + "smmlu/easy/Catholicism/114", + "smmlu/easy/Catholicism/43", + "smmlu/easy/Catholicism/108", + "smmlu/easy/Catholicism/86", + "smmlu/easy/Catholicism/41", + "smmlu/easy/Catholicism/39", + "smmlu/easy/Catholicism/45", + "smmlu/easy/Catholicism/49", + "smmlu/easy/Christianity/53", + "smmlu/easy/Christianity/109", + "smmlu/easy/Christianity/74", + "smmlu/easy/Christianity/16", + "smmlu/easy/Christianity/116", + "smmlu/easy/Christianity/111", + "smmlu/easy/Christianity/17", + "smmlu/easy/Christianity/11", + "smmlu/easy/Christianity/115", + "smmlu/easy/Christianity/79", + "smmlu/easy/Christianity/86", + "smmlu/easy/Christianity/106", + "smmlu/easy/Christianity/129", + "smmlu/easy/Christianity/65", + "smmlu/easy/Christianity/42", + "smmlu/easy/Christianity/32", + "smmlu/easy/Christianity/1", + "smmlu/easy/Christianity/72", + "smmlu/easy/Christianity/67", + "smmlu/easy/Christianity/123", + "smmlu/easy/Christianity/13", + "smmlu/easy/Christianity/125", + "smmlu/easy/Christianity/97", + "smmlu/easy/Christianity/35", + "smmlu/easy/Christianity/57", + "smmlu/easy/Christianity/94", + "smmlu/easy/Civics/115", + "smmlu/easy/Civics/119", + "smmlu/easy/Civics/62", + "smmlu/easy/Civics/105", + "smmlu/easy/Civics/12", + "smmlu/easy/Civics/77", + "smmlu/easy/Civics/5", + "smmlu/easy/Civics/123", + "smmlu/easy/Civics/37", + "smmlu/easy/Civics/26", + "smmlu/easy/Civics/17", + "smmlu/easy/Civics/32", + "smmlu/easy/Civics/79", + "smmlu/easy/Civics/71", + "smmlu/easy/Civics/83", + "smmlu/easy/Civics/66", + "smmlu/easy/Civics/61", + "smmlu/easy/Civics/45", + "smmlu/easy/Civics/96", + "smmlu/easy/Civics/36", + "smmlu/easy/Civics/120", + "smmlu/easy/Civics/7", + "smmlu/easy/Civics/21", + "smmlu/easy/Civics/67", + "smmlu/easy/Civics/95", + "smmlu/easy/Eastern music/129", + "smmlu/easy/Eastern music/122", + "smmlu/easy/Eastern music/44", + "smmlu/easy/Eastern music/99", + "smmlu/easy/Eastern music/127", + "smmlu/easy/Eastern music/34", + "smmlu/easy/Eastern music/124", + "smmlu/easy/Eastern music/53", + "smmlu/easy/Eastern music/24", + "smmlu/easy/Eastern music/56", + "smmlu/easy/Eastern music/125", + "smmlu/easy/Eastern music/75", + "smmlu/easy/Eastern music/21", + "smmlu/easy/Eastern music/55", + "smmlu/easy/Eastern music/43", + "smmlu/easy/Eastern music/114", + "smmlu/easy/Eastern music/126", + "smmlu/easy/Eastern music/95", + "smmlu/easy/Eastern music/20", + "smmlu/easy/Eastern music/78", + "smmlu/easy/Eastern music/71", + "smmlu/easy/Eastern music/76", + "smmlu/easy/Eastern music/96", + "smmlu/easy/Eastern music/58", + "smmlu/easy/Eastern music/77", + "smmlu/easy/Eastern music/69", + "smmlu/easy/Eastern music/54", + "smmlu/easy/Eastern music/50", + "smmlu/easy/Eastern music/2", + "smmlu/easy/Geography/7", + "smmlu/easy/Geography/86", + "smmlu/easy/Geography/90", + "smmlu/easy/Geography/73", + "smmlu/easy/Geography/92", + "smmlu/easy/Geography/22", + "smmlu/easy/Geography/19", + "smmlu/easy/Geography/44", + "smmlu/easy/Geography/46", + "smmlu/easy/Geography/16", + "smmlu/easy/Geography/40", + "smmlu/easy/Geography/66", + "smmlu/easy/Geography/109", + "smmlu/easy/Geography/21", + "smmlu/easy/Geography/56", + "smmlu/easy/Geography/29", + "smmlu/easy/Geography/103", + "smmlu/easy/Geography/71", + "smmlu/easy/Geography/102", + "smmlu/easy/Geography/63", + "smmlu/easy/Geography/49", + "smmlu/easy/Geography/69", + "smmlu/easy/Geography/52", + "smmlu/easy/Health and Physical Science/87", + "smmlu/easy/Health and Physical Science/107", + "smmlu/easy/Health and Physical Science/100", + "smmlu/easy/Health and Physical Science/71", + "smmlu/easy/Health and Physical Science/103", + "smmlu/easy/Health and Physical Science/126", + "smmlu/easy/Health and Physical Science/111", + "smmlu/easy/Health and Physical Science/32", + "smmlu/easy/Health and Physical Science/61", + "smmlu/easy/Health and Physical Science/68", + "smmlu/easy/Health and Physical Science/125", + "smmlu/easy/Health and Physical Science/9", + "smmlu/easy/Health and Physical Science/53", + "smmlu/easy/Health and Physical Science/28", + "smmlu/easy/Health and Physical Science/2", + "smmlu/easy/Health and Physical Science/70", + "smmlu/easy/Health and Physical Science/57", + "smmlu/easy/Health and Physical Science/85", + "smmlu/easy/Health and Physical Science/63", + "smmlu/easy/Health and Physical Science/3", + "smmlu/easy/Health and Physical Science/49", + "smmlu/easy/Health and Physical Science/15", + "smmlu/easy/Health and Physical Science/75", + "smmlu/easy/Health and Physical Science/22", + "smmlu/easy/Health and Physical Science/17", + "smmlu/easy/Health and Physical Science/98", + "smmlu/easy/History/28", + "smmlu/easy/History/143", + "smmlu/easy/History/37", + "smmlu/easy/History/40", + "smmlu/easy/History/116", + "smmlu/easy/History/121", + "smmlu/easy/History/45", + "smmlu/easy/History/89", + "smmlu/easy/History/96", + "smmlu/easy/History/31", + "smmlu/easy/History/130", + "smmlu/easy/History/111", + "smmlu/easy/History/1", + "smmlu/easy/History/144", + "smmlu/easy/History/147", + "smmlu/easy/History/151", + "smmlu/easy/History/138", + "smmlu/easy/History/12", + "smmlu/easy/History/2", + "smmlu/easy/History/145", + "smmlu/easy/History/72", + "smmlu/easy/History/150", + "smmlu/easy/History/41", + "smmlu/easy/History/99", + "smmlu/easy/History/108", + "smmlu/easy/History/48", + "smmlu/easy/History/93", + "smmlu/easy/History/53", + "smmlu/easy/History/63", + "smmlu/easy/History/3", + "smmlu/easy/Islam/56", + "smmlu/easy/Islam/1", + "smmlu/easy/Islam/81", + "smmlu/easy/Islam/19", + "smmlu/easy/Islam/90", + "smmlu/easy/Islam/26", + "smmlu/easy/Islam/62", + "smmlu/easy/Islam/45", + "smmlu/easy/Islam/2", + "smmlu/easy/Islam/22", + "smmlu/easy/Islam/35", + "smmlu/easy/Islam/55", + "smmlu/easy/Islam/83", + "smmlu/easy/Islam/77", + "smmlu/easy/Islam/89", + "smmlu/easy/Islam/82", + "smmlu/easy/Islam/46", + "smmlu/easy/Islam/96", + "smmlu/easy/Islam/102", + "smmlu/easy/Islam/20", + "smmlu/easy/Islam/104", + "smmlu/easy/Islam/32", + "smmlu/easy/Sinhala language and literature/38", + "smmlu/easy/Sinhala language and literature/89", + "smmlu/easy/Sinhala language and literature/129", + "smmlu/easy/Sinhala language and literature/41", + "smmlu/easy/Sinhala language and literature/66", + "smmlu/easy/Sinhala language and literature/35", + "smmlu/easy/Sinhala language and literature/31", + "smmlu/easy/Sinhala language and literature/119", + "smmlu/easy/Sinhala language and literature/39", + "smmlu/easy/Sinhala language and literature/130", + "smmlu/easy/Sinhala language and literature/77", + "smmlu/easy/Sinhala language and literature/116", + "smmlu/easy/Sinhala language and literature/123", + "smmlu/easy/Sinhala language and literature/64", + "smmlu/easy/Sinhala language and literature/6", + "smmlu/easy/Sinhala language and literature/106", + "smmlu/easy/Sinhala language and literature/111", + "smmlu/easy/Sinhala language and literature/127", + "smmlu/easy/Sinhala language and literature/13", + "smmlu/easy/Sinhala language and literature/19", + "smmlu/easy/Sinhala language and literature/148", + "smmlu/easy/Sinhala language and literature/4", + "smmlu/easy/Sinhala language and literature/112", + "smmlu/easy/Sinhala language and literature/139", + "smmlu/easy/Sinhala language and literature/18", + "smmlu/easy/Sinhala language and literature/136", + "smmlu/easy/Sinhala language and literature/125", + "smmlu/easy/Sinhala language and literature/68", + "smmlu/easy/Sinhala language and literature/24", + "smmlu/easy/Sinhala language and literature/145", + "smmlu/easy/Sinhala language and literature/51", + "smmlu/easy/dancing/51", + "smmlu/easy/dancing/86", + "smmlu/easy/dancing/6", + "smmlu/easy/dancing/18", + "smmlu/easy/dancing/44", + "smmlu/easy/dancing/96", + "smmlu/easy/dancing/76", + "smmlu/easy/dancing/16", + "smmlu/easy/dancing/66", + "smmlu/easy/dancing/62", + "smmlu/easy/dancing/7", + "smmlu/easy/dancing/89", + "smmlu/easy/dancing/25", + "smmlu/easy/dancing/61", + "smmlu/easy/dancing/52", + "smmlu/easy/dancing/55", + "smmlu/easy/dancing/69", + "smmlu/easy/dancing/91", + "smmlu/easy/dancing/84", + "smmlu/easy/dancing/12", + "smmlu/easy/dancing/85", + "smmlu/easy/drama and Theatre /18", + "smmlu/easy/drama and Theatre /73", + "smmlu/easy/drama and Theatre /106", + "smmlu/easy/drama and Theatre /94", + "smmlu/easy/drama and Theatre /47", + "smmlu/easy/drama and Theatre /40", + "smmlu/easy/drama and Theatre /57", + "smmlu/easy/drama and Theatre /70", + "smmlu/easy/drama and Theatre /115", + "smmlu/easy/drama and Theatre /118", + "smmlu/easy/drama and Theatre /114", + "smmlu/easy/drama and Theatre /15", + "smmlu/easy/drama and Theatre /52", + "smmlu/easy/drama and Theatre /20", + "smmlu/easy/drama and Theatre /76", + "smmlu/easy/drama and Theatre /64", + "smmlu/easy/drama and Theatre /105", + "smmlu/easy/drama and Theatre /41", + "smmlu/easy/drama and Theatre /98", + "smmlu/easy/drama and Theatre /35", + "smmlu/easy/drama and Theatre /37", + "smmlu/easy/drama and Theatre /86", + "smmlu/easy/drama and Theatre /60", + "smmlu/easy/drama and Theatre /124", + "smmlu/easy/drama and Theatre /128", + "smmlu/easy/drama and Theatre /91", + "smmlu/easy/science/1", + "smmlu/easy/science/123", + "smmlu/easy/science/63", + "smmlu/easy/science/69", + "smmlu/easy/science/80", + "smmlu/easy/science/150", + "smmlu/easy/science/79", + "smmlu/easy/science/4", + "smmlu/easy/science/52", + "smmlu/easy/science/85", + "smmlu/easy/science/154", + "smmlu/easy/science/127", + "smmlu/easy/science/36", + "smmlu/easy/science/55", + "smmlu/easy/science/20", + "smmlu/easy/science/8", + "smmlu/easy/science/99", + "smmlu/easy/science/93", + "smmlu/easy/science/16", + "smmlu/easy/science/161", + "smmlu/easy/science/18", + "smmlu/easy/science/97", + "smmlu/easy/science/107", + "smmlu/easy/science/119", + "smmlu/easy/science/103", + "smmlu/easy/science/144", + "smmlu/easy/science/125", + "smmlu/easy/science/24", + "smmlu/easy/science/39", + "smmlu/easy/science/138", + "smmlu/easy/science/100", + "smmlu/easy/science/49", + "smmlu/easy/science/159" + ] +] \ No newline at end of file diff --git a/training/configs/submission.yaml b/training/configs/submission.yaml new file mode 100644 index 0000000000000000000000000000000000000000..53d71754a4a380fbc91d3df5081ed0e667049ca8 --- /dev/null +++ b/training/configs/submission.yaml @@ -0,0 +1,13 @@ +# Everything the offline submission pipeline needs (inference/predict.py). +# Chosen on SinhalaMMLU dev + Global-MMLU si validation only (reports/experiments_prompting.md), never on test. +model: models/gemma-4-E4B-it-text # text-only Gemma 4 E4B (scripts/make_text_only.py), 7.463B parameters +lang: en +chat: true +method: logit +n_perms: 1 +perm_mode: cyclic +seed: 1234 +dtype: bfloat16 +gpu_mem_gb: 100 +token_budget: 32768 +max_batch: 64 diff --git a/training/configs/submission_A_think_hard.yaml b/training/configs/submission_A_think_hard.yaml new file mode 100644 index 0000000000000000000000000000000000000000..a0cbf459dd2c29c597fe2c026d30d4caca95e664 --- /dev/null +++ b/training/configs/submission_A_think_hard.yaml @@ -0,0 +1,15 @@ +# Everything the offline submission pipeline needs (inference/predict.py). +# A + thinking on hard (19 Oct): model A; questions labelled hard use thinking mode (academic proxy +8.4, paired CI [+4.9, +11.8]); others fast 4-order vote. Never chosen on test. +model: models/sft_gmmlu_lora/step-357 # model A: LoRA r=16 on Global-MMLU si, 1 epoch, merged (7.463B) +lang: en +chat: true +method: logit +n_perms: 4 # vote over 4 cyclic option orders (+2.7 dev vs zero-shot, paired bootstrap) +perm_mode: cyclic +seed: 1234 +dtype: bfloat16 +gpu_mem_gb: 100 +token_budget: 32768 +max_batch: 64 +thinking_levels: [hard] +think_max_new_tokens: 2048 diff --git a/training/configs/submission_base_think_gated_sc.yaml b/training/configs/submission_base_think_gated_sc.yaml new file mode 100644 index 0000000000000000000000000000000000000000..f4313c8ed92d5d7738a50718b766fe9fb17b1b06 --- /dev/null +++ b/training/configs/submission_base_think_gated_sc.yaml @@ -0,0 +1,23 @@ +# Final candidate B (21 Oct): base Gemma 4 E4B, text-only (7.463B, no adapter), thinking for every question. +# Agreement-gated self-consistency on all levels: where the one-step answer disagrees with greedy thinking, add 3 sampled +# reasoning paths and take the majority vote (ties -> greedy). Evidence (dev/validation only): full dev, base thinking 63.0 vs +# model A vote 57.2 (+5.8 [+3.6, +8.0]); pooled proxies, gated self-consistency +2.70 [+1.18, +4.23] on 43% of questions. +model: models/gemma-4-E4B-it-text +lang: en +chat: true +method: logit # fast path, used only if a thinking answer cannot be read +n_perms: 4 +perm_mode: cyclic +seed: 1234 +dtype: bfloat16 +gpu_mem_gb: 100 +token_budget: 32768 +max_batch: 64 +thinking_levels: [easy, medium, hard] +think_max_new_tokens: 3072 +sc_levels: [easy, medium, hard] +sc_gate: disagree +sc_samples: 3 +sc_temperature: 0.8 +sc_top_p: 0.95 +sc_top_k: 64 diff --git a/training/configs/submission_base_think_sc_hard.yaml b/training/configs/submission_base_think_sc_hard.yaml new file mode 100644 index 0000000000000000000000000000000000000000..7495cedd8f11452c053b590285cd4f2ac0065476 --- /dev/null +++ b/training/configs/submission_base_think_sc_hard.yaml @@ -0,0 +1,22 @@ +# Final candidate (21 Oct): base Gemma 4 E4B, text-only (7.463B, no adapter), thinking for every question. +# Hard: self-consistency (greedy + 3 sampled reasoning paths, majority vote, ties -> greedy). +# Evidence (dev / validation only, never test): full dev, base thinking 63.0 vs model A 4-order vote 57.2 +# (+5.8 [+3.6, +8.0]); academic proxy, self-consistency 82.2 vs greedy 79.5 (+2.7 [+0.9, +4.7]). +model: models/gemma-4-E4B-it-text +lang: en +chat: true +method: logit # fast path, used only if a thinking answer cannot be read +n_perms: 4 +perm_mode: cyclic +seed: 1234 +dtype: bfloat16 +gpu_mem_gb: 100 +token_budget: 32768 +max_batch: 64 +thinking_levels: [easy, medium, hard] +think_max_new_tokens: 3072 +sc_levels: [hard] +sc_samples: 3 +sc_temperature: 0.8 +sc_top_p: 0.95 +sc_top_k: 64 diff --git a/training/configs/submission_lora_gmmlu_vote.yaml b/training/configs/submission_lora_gmmlu_vote.yaml new file mode 100644 index 0000000000000000000000000000000000000000..34e7aec8120b8c19a890b0279728cc326976a6ed --- /dev/null +++ b/training/configs/submission_lora_gmmlu_vote.yaml @@ -0,0 +1,13 @@ +# Everything the offline submission pipeline needs (inference/predict.py). +# Chosen on SinhalaMMLU dev only (reports/experiments_abc.md), never on test. +model: models/sft_gmmlu_lora/step-357 # experiment A: LoRA r=16 on Global-MMLU si, 1 epoch, merged (7.463B) +lang: en +chat: true +method: logit +n_perms: 4 # vote over 4 cyclic option orders (+2.7 dev vs zero-shot, paired bootstrap) +perm_mode: cyclic +seed: 1234 +dtype: bfloat16 +gpu_mem_gb: 100 +token_budget: 32768 +max_batch: 64 diff --git a/training/configs/submission_medium_option_order.yaml b/training/configs/submission_medium_option_order.yaml new file mode 100644 index 0000000000000000000000000000000000000000..bb97f78b1792c06c48f5b1dfd5d0fb6f4fe28784 --- /dev/null +++ b/training/configs/submission_medium_option_order.yaml @@ -0,0 +1,20 @@ +# Medium-only option-order self-consistency (27 Oct): base Gemma 4 E4B text-only (7.463B, no adapter). For every medium question, +# thinking under 3 extra cyclic option orders; vote over the previous answer (candidate B's) and the 3 re-orderings, ties -> previous. +# Run with --levels medium and merge into candidate B (scripts/merge_predictions.py). Chosen only if eval/perm_sc.py says GO on the +# held-out stand-in (never on test). +model: models/gemma-4-E4B-it-text +lang: en +chat: true +method: logit +n_perms: 1 +perm_mode: cyclic +seed: 1234 +dtype: bfloat16 +gpu_mem_gb: 100 +token_budget: 32768 +max_batch: 64 +thinking_levels: [] +think_max_new_tokens: 3072 +perm_think_levels: [medium] +perm_think_orders: 3 +prior_predictions: results/submissions/final_candidate_base_think_gated_sc.zip diff --git a/training/configs/submission_pool_base_medium.yaml b/training/configs/submission_pool_base_medium.yaml new file mode 100644 index 0000000000000000000000000000000000000000..7a6d0ceb24d98ea2df016b1f58e2f7d6fcadbe38 --- /dev/null +++ b/training/configs/submission_pool_base_medium.yaml @@ -0,0 +1,21 @@ +# Cross-model pooling, part 1 (27 Oct), MEDIUM ONLY: base Gemma 4 E4B, greedy thinking + 3 sampled paths on EVERY question +# (no agreement gate); every answer is saved to .ballots.jsonl. scripts/pool_ballots.py votes over these ballots and the +# RFT2 model's (configs/submission_pool_rft2_medium.yaml) and merges into candidate B. Used only if eval/pool_decide.py says GO. +model: models/gemma-4-E4B-it-text +lang: en +chat: true +method: logit +n_perms: 4 +perm_mode: cyclic +seed: 1234 +dtype: bfloat16 +gpu_mem_gb: 100 +token_budget: 32768 +max_batch: 64 +thinking_levels: [medium] +think_max_new_tokens: 3072 +sc_levels: [medium] +sc_samples: 3 +sc_temperature: 0.8 +sc_top_p: 0.95 +sc_top_k: 64 diff --git a/training/configs/submission_pool_rft2_medium.yaml b/training/configs/submission_pool_rft2_medium.yaml new file mode 100644 index 0000000000000000000000000000000000000000..b0d108198ad17103e8cb5ada5d79014684469aff --- /dev/null +++ b/training/configs/submission_pool_rft2_medium.yaml @@ -0,0 +1,21 @@ +# Cross-model pooling, part 2 (27 Oct), MEDIUM ONLY: the RFT2 model (base + reasoning-trace LoRA, 7.50B parameters, adapter ON), +# greedy thinking + 3 sampled paths on every question (no gate); ballots saved for scripts/pool_ballots.py. +model: models/gemma-4-E4B-it-text +adapter: models/rft_v2/step-141/adapter +lang: en +chat: true +method: logit +n_perms: 4 +perm_mode: cyclic +seed: 1234 +dtype: bfloat16 +gpu_mem_gb: 100 +token_budget: 32768 +max_batch: 64 +thinking_levels: [medium] +think_max_new_tokens: 3072 +sc_levels: [medium] +sc_samples: 3 +sc_temperature: 0.8 +sc_top_p: 0.95 +sc_top_k: 64 diff --git a/training/configs/submission_rft2_medium.yaml b/training/configs/submission_rft2_medium.yaml new file mode 100644 index 0000000000000000000000000000000000000000..af7dce2e63837bccd612af2bb54e1e85cf2a942b --- /dev/null +++ b/training/configs/submission_rft2_medium.yaml @@ -0,0 +1,26 @@ +# Candidate D (27 Oct), MEDIUM ONLY: base Gemma 4 E4B + LoRA adapter trained on the base model's own correct reasoning traces +# (dev minus held-out 400, Global-MMLU, synthetic Wikipedia MCQs; ~7.50B parameters). Thinking with the adapter ON; the fast answer +# for the agreement gate comes from the plain base model. Agreement-gated self-consistency. Run with --levels medium and merge into +# candidate B (scripts/merge_predictions.py). Chosen only if it passes the pre-registered held-out validation (eval/decide_rft.py). +model: models/gemma-4-E4B-it-text +adapter: models/rft_v2/step-ADAPTER/adapter +thinking_without_adapter: false +fast_without_adapter: true +lang: en +chat: true +method: logit +n_perms: 4 +perm_mode: cyclic +seed: 1234 +dtype: bfloat16 +gpu_mem_gb: 100 +token_budget: 32768 +max_batch: 64 +thinking_levels: [medium] +think_max_new_tokens: 3072 +sc_levels: [medium] +sc_gate: disagree +sc_samples: 3 +sc_temperature: 0.8 +sc_top_p: 0.95 +sc_top_k: 64 diff --git a/training/configs/submission_rft_think_gated_sc.yaml b/training/configs/submission_rft_think_gated_sc.yaml new file mode 100644 index 0000000000000000000000000000000000000000..47ccddf327bc92d73d8603d8f4c9f55a51ce2024 --- /dev/null +++ b/training/configs/submission_rft_think_gated_sc.yaml @@ -0,0 +1,27 @@ +# Final candidate C (26-27 Oct): base Gemma 4 E4B + LoRA adapter trained on the base model's own correct reasoning traces +# (rejection sampling, SinhalaMMLU dev minus a held-out 400; ~7.50B parameters). Thinking for every question with the adapter ON; +# the fast one-step answer for the agreement gate comes from the plain base model (adapter off). +# Agreement-gated self-consistency on all levels, as candidate B. Chosen only if it passes the pre-registered held-out +# validation rule in eval/decide_rft.py (never on test). +model: models/gemma-4-E4B-it-text +adapter: models/rft_v1/step-ADAPTER/adapter +thinking_without_adapter: false +fast_without_adapter: true +lang: en +chat: true +method: logit +n_perms: 4 +perm_mode: cyclic +seed: 1234 +dtype: bfloat16 +gpu_mem_gb: 100 +token_budget: 32768 +max_batch: 64 +thinking_levels: [easy, medium, hard] +think_max_new_tokens: 3072 +sc_levels: [easy, medium, hard] +sc_gate: disagree +sc_samples: 3 +sc_temperature: 0.8 +sc_top_p: 0.95 +sc_top_k: 64 diff --git a/training/configs/submission_sub5_route.yaml b/training/configs/submission_sub5_route.yaml new file mode 100644 index 0000000000000000000000000000000000000000..13ca89a29441f876d34f44c94d84095c935e75d2 --- /dev/null +++ b/training/configs/submission_sub5_route.yaml @@ -0,0 +1,17 @@ +# Everything the offline submission pipeline needs (inference/predict.py). +# sub5 (19 Oct): base Gemma 4 E4B + adapter A (LoRA r=16, extracted, ~0.04B); easy/medium fast 4-order vote WITH adapter; hard: thinking WITHOUT adapter (academic proxy: base thinking 79.5 vs A fast 60.1). Never chosen on test. +model: models/gemma-4-E4B-it-text # base, 7.463B +lang: en +chat: true +method: logit +n_perms: 4 # vote over 4 cyclic option orders (+2.7 dev vs zero-shot, paired bootstrap) +perm_mode: cyclic +seed: 1234 +dtype: bfloat16 +gpu_mem_gb: 100 +token_budget: 32768 +max_batch: 64 +thinking_levels: [hard] +think_max_new_tokens: 3072 +adapter: models/adapter_A # model A as a separate LoRA adapter +thinking_without_adapter: true diff --git a/training/configs/submission_sub6_think_medhard.yaml b/training/configs/submission_sub6_think_medhard.yaml new file mode 100644 index 0000000000000000000000000000000000000000..a444869655a5a94989301e612106b410e0de1a96 --- /dev/null +++ b/training/configs/submission_sub6_think_medhard.yaml @@ -0,0 +1,17 @@ +# Everything the offline submission pipeline needs (inference/predict.py). +# sub6 (20 Oct): as sub5, plus base-model thinking (adapter off) for medium. Evidence: non-academic Global-MMLU proxy +12.3 [+6.0, +18.3] and easy-style dev not worse (+0.8) vs model A fast. Never chosen on test. +model: models/gemma-4-E4B-it-text # base, 7.463B +lang: en +chat: true +method: logit +n_perms: 4 # vote over 4 cyclic option orders (+2.7 dev vs zero-shot, paired bootstrap) +perm_mode: cyclic +seed: 1234 +dtype: bfloat16 +gpu_mem_gb: 100 +token_budget: 32768 +max_batch: 64 +thinking_levels: [medium, hard] +think_max_new_tokens: 3072 +adapter: models/adapter_A # model A as a separate LoRA adapter +thinking_without_adapter: true diff --git a/training/configs/submission_v2.yaml b/training/configs/submission_v2.yaml new file mode 100644 index 0000000000000000000000000000000000000000..0e4b89e95feec1a5923584aa51245009ecd59d6e --- /dev/null +++ b/training/configs/submission_v2.yaml @@ -0,0 +1,13 @@ +# Everything the offline submission pipeline needs (inference/predict.py). +# v2: recipe validated on dev via A (Global-MMLU LoRA, +2.7) and B (dev 5-fold LoRA, +2.8); never chosen on test. +model: models/sft_v2_gmmlu_dev/step-415 # v2: LoRA r=16 on Global-MMLU si + all SinhalaMMLU dev, 1 epoch, merged (7.463B) +lang: en +chat: true +method: logit +n_perms: 4 # vote over 4 cyclic option orders (+2.7 dev vs zero-shot, paired bootstrap) +perm_mode: cyclic +seed: 1234 +dtype: bfloat16 +gpu_mem_gb: 100 +token_budget: 32768 +max_batch: 64 diff --git a/training/configs/train/cpt_pilot.yaml b/training/configs/train/cpt_pilot.yaml new file mode 100644 index 0000000000000000000000000000000000000000..ff0a5152204fa5d388ffe5c191bde91c77628154 --- /dev/null +++ b/training/configs/train/cpt_pilot.yaml @@ -0,0 +1,24 @@ +# Experiment C: 30M-token pilot of continued pretraining (lower LR to limit forgetting), checkpoints at 15M/30M. +mode: cpt +model: models/gemma-4-E4B-it-text +out_dir: models/cpt_pilot +data_dir: data/corpus/cpt_v1 +seq_len: 2048 +micro_batch: 2 +grad_accum: 32 # 131k tokens per optimizer step +max_tokens: 30000000 +epochs: 1 +lr: 5.0e-6 +min_lr_ratio: 0.1 +warmup_steps: 20 +weight_decay: 0.0 +max_grad_norm: 1.0 +freeze_embeddings: true +ckpt_every: 2 +compile: true +loss: liger +seed: 1234 +gpu_mem_gb: 100 +save_every_tokens: 15000000 +keep_checkpoints: 3 +log_every: 10 diff --git a/training/configs/train/cpt_v1.yaml b/training/configs/train/cpt_v1.yaml new file mode 100644 index 0000000000000000000000000000000000000000..72049a2cf84a0da026186f136e099c2b7eab7747 --- /dev/null +++ b/training/configs/train/cpt_v1.yaml @@ -0,0 +1,24 @@ +# Continued pretraining on data/corpus/cpt_v1 (scripts/select_cpt_subset.py). Fastest measured safe setting. +mode: cpt +model: models/gemma-4-E4B-it-text +out_dir: models/cpt_v1 +data_dir: data/corpus/cpt_v1 +seq_len: 2048 +micro_batch: 2 +grad_accum: 32 # 131k tokens per optimizer step +max_tokens: 300000000 +epochs: 1 +lr: 1.0e-5 +min_lr_ratio: 0.1 +warmup_steps: 50 +weight_decay: 0.0 +max_grad_norm: 1.0 +freeze_embeddings: true +ckpt_every: 2 +compile: true +loss: liger +seed: 1234 +gpu_mem_gb: 100 +save_every_tokens: 50000000 +keep_checkpoints: 3 +log_every: 10 diff --git a/training/configs/train/rft_trace.yaml b/training/configs/train/rft_trace.yaml new file mode 100644 index 0000000000000000000000000000000000000000..1a7748bd47dbf47e284097ac31f58e10e6669a16 --- /dev/null +++ b/training/configs/train/rft_trace.yaml @@ -0,0 +1,23 @@ +# Reasoning-preserving fine-tuning (21-26 Oct): LoRA r=16 on the base model's OWN correct thinking traces for SinhalaMMLU +# dev questions (scripts/gen_traces.py), loss on the trace only. The 400-question dev sample is held out for validation. +mode: trace +model: models/gemma-4-E4B-it-text +out_dir: models/rft_v1 +data: [data/synth/rft_traces.jsonl] +lang: en +max_len: 4096 +epochs: 1 +batch_size: 16 +micro_tokens: 12000 +lr: 1.0e-4 +min_lr_ratio: 0.1 +warmup_steps: 5 +weight_decay: 0.0 +max_grad_norm: 1.0 +lora: {r: 16, alpha: 32, dropout: 0.0, targets: all-linear} +ckpt_every: 1 +compile: false +seed: 1234 +gpu_mem_gb: 100 +keep_checkpoints: 1 +log_every: 5 diff --git a/training/configs/train/rft_trace2.yaml b/training/configs/train/rft_trace2.yaml new file mode 100644 index 0000000000000000000000000000000000000000..f7fc6883d4e4c5021b904b1898eee918682ece15 --- /dev/null +++ b/training/configs/train/rft_trace2.yaml @@ -0,0 +1,22 @@ +# Round 2 (27 Oct): LoRA r=16 on the base model OWN correct thinking traces: SinhalaMMLU dev (minus held-out 400) + Global-MMLU (humanities/social/other/business) + synthetic Wikipedia MCQs. +mode: trace +model: models/gemma-4-E4B-it-text +out_dir: models/rft_v2 +data: [data/synth/rft_traces.jsonl, data/synth/rft2_traces.jsonl] +lang: en +max_len: 4096 +epochs: 1 +batch_size: 16 +micro_tokens: 12000 +lr: 1.0e-4 +min_lr_ratio: 0.1 +warmup_steps: 5 +weight_decay: 0.0 +max_grad_norm: 1.0 +lora: {r: 16, alpha: 32, dropout: 0.0, targets: all-linear} +ckpt_every: 1 +compile: false +seed: 1234 +gpu_mem_gb: 100 +keep_checkpoints: 1 +log_every: 5 diff --git a/training/configs/train/sft_gmmlu.yaml b/training/configs/train/sft_gmmlu.yaml new file mode 100644 index 0000000000000000000000000000000000000000..52c4384f4e87776e0b01d8e34843f5ffbae6173a --- /dev/null +++ b/training/configs/train/sft_gmmlu.yaml @@ -0,0 +1,24 @@ +# Step 4: fine-tune text-only Gemma 4 E4B on Global-MMLU si MCQs (scripts/build_sft_gmmlu.py), answer-digit loss. +mode: sft +model: models/gemma-4-E4B-it-text +out_dir: models/sft_gmmlu_v1 +data: [data/sft/gmmlu_si_train.jsonl] +lang: en +chat: true +permute_options: true +epochs: 2 +batch_size: 32 # questions per optimizer step +micro_tokens: 16384 # padded tokens per forward pass +lr: 1.0e-5 +min_lr_ratio: 0.1 +warmup_steps: 20 +weight_decay: 0.0 +max_grad_norm: 1.0 +freeze_embeddings: true # Gemma per-layer embedding tables (3.5B) frozen; 3.97B trained +ckpt_every: 1 +compile: false # variable SFT lengths would recompile per shape +loss: liger +seed: 1234 +gpu_mem_gb: 100 +keep_checkpoints: 2 +log_every: 10 diff --git a/training/configs/train/sft_lora.yaml b/training/configs/train/sft_lora.yaml new file mode 100644 index 0000000000000000000000000000000000000000..85cba2c08830d1b5df03063bf7cda0fb9087695a --- /dev/null +++ b/training/configs/train/sft_lora.yaml @@ -0,0 +1,25 @@ +# Gentle fine-tuning: LoRA r=16 on all linear layers (base frozen, merged at save -> 7.463B params). +# Used for experiment A (Global-MMLU, 1 epoch) and B (SinhalaMMLU dev 5-fold CV, --data/--out per fold). +mode: sft +model: models/gemma-4-E4B-it-text +out_dir: models/sft_lora +data: [data/sft/gmmlu_si_train.jsonl] +lang: en +chat: true +permute_options: true +epochs: 1 +batch_size: 32 +micro_tokens: 16384 +lr: 1.0e-4 +min_lr_ratio: 0.1 +warmup_steps: 10 +weight_decay: 0.0 +max_grad_norm: 1.0 +lora: {r: 16, alpha: 32, dropout: 0.0, targets: all-linear} +ckpt_every: 1 +compile: false +loss: liger +seed: 1234 +gpu_mem_gb: 100 +keep_checkpoints: 1 +log_every: 10 diff --git a/training/configs/train/sft_lora_devcv.yaml b/training/configs/train/sft_lora_devcv.yaml new file mode 100644 index 0000000000000000000000000000000000000000..2bd05e2a734c70ee119678f5064c4db6f3f6ae6e --- /dev/null +++ b/training/configs/train/sft_lora_devcv.yaml @@ -0,0 +1,25 @@ +# Gentle fine-tuning: LoRA r=16 on all linear layers (base frozen, merged at save -> 7.463B params). +# Used for experiment A (Global-MMLU, 1 epoch) and B (SinhalaMMLU dev 5-fold CV, --data/--out per fold). +mode: sft +model: models/gemma-4-E4B-it-text +out_dir: models/devcv_lora +data: [data/sft/gmmlu_si_train.jsonl] +lang: en +chat: true +permute_options: true +epochs: 2 +batch_size: 32 +micro_tokens: 16384 +lr: 1.0e-4 +min_lr_ratio: 0.1 +warmup_steps: 5 +weight_decay: 0.0 +max_grad_norm: 1.0 +lora: {r: 16, alpha: 32, dropout: 0.0, targets: all-linear} +ckpt_every: 1 +compile: false +loss: liger +seed: 1234 +gpu_mem_gb: 100 +keep_checkpoints: 1 +log_every: 10 diff --git a/training/configs/train/sft_lora_kd.yaml b/training/configs/train/sft_lora_kd.yaml new file mode 100644 index 0000000000000000000000000000000000000000..941b722de1f1337d346d90bf9857177f5e197f9e --- /dev/null +++ b/training/configs/train/sft_lora_kd.yaml @@ -0,0 +1,26 @@ +# Experiment KD (19 Oct): LoRA like model A, loss = 0.5 x answer CE + 0.5 x KL to Gemma 4 31B option probabilities. +# Used for experiment A (Global-MMLU, 1 epoch) and B (SinhalaMMLU dev 5-fold CV, --data/--out per fold). +mode: sft +model: models/gemma-4-E4B-it-text +out_dir: models/sft_kd_t31b +data: [data/sft/gmmlu_si_train_t31b.jsonl] +lang: en +chat: true +permute_options: true +epochs: 1 +batch_size: 32 +micro_tokens: 16384 +lr: 1.0e-4 +min_lr_ratio: 0.1 +warmup_steps: 10 +weight_decay: 0.0 +max_grad_norm: 1.0 +lora: {r: 16, alpha: 32, dropout: 0.0, targets: all-linear} +ckpt_every: 1 +compile: false +loss: liger +seed: 1234 +gpu_mem_gb: 100 +keep_checkpoints: 1 +log_every: 10 +kd_alpha: 0.5 diff --git a/training/eval/__init__.py b/training/eval/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/training/eval/combine_cv.py b/training/eval/combine_cv.py new file mode 100644 index 0000000000000000000000000000000000000000..f154f7b2df0229f5b8fd88b934f1f631bd00a508 --- /dev/null +++ b/training/eval/combine_cv.py @@ -0,0 +1,48 @@ +"""Combine k-fold runs (each scored only on its held-out fold) into one run over the full dev set. + + python -m eval.combine_cv --runs results/exp_devcv_lora_f* --name exp_devcv_lora_all + +Writes results//{metrics.json, report.md, predictions.jsonl} in the usual layout, so eval.summarize can put it +next to the zero-shot baseline. Fails if a question is missing or scored twice. +""" +from __future__ import annotations + +import argparse +import glob +import json +from pathlib import Path + +from .common import ROOT, read_jsonl, write_jsonl +from .metrics import markdown_report +from .run_eval import metrics_by_dataset + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--runs", nargs="+", required=True) + ap.add_argument("--name", required=True) + ap.add_argument("--data", default="data/processed/sinhalammlu_dev.jsonl") + args = ap.parse_args() + dirs = sorted(Path(d) for pat in args.runs for d in glob.glob(pat)) + records, first = [], None + for d in dirs: + records += read_jsonl(d / "predictions.jsonl") + first = first or json.loads((d / "metrics.json").read_text()) + items = {r["uid"]: r for r in read_jsonl(ROOT / args.data)} + scored = [r["uid"] for r in records if r["perm_k"] == 0] + assert len(scored) == len(set(scored)), "a question was scored by more than one fold" + assert set(scored) == set(items), f"{len(set(items) - set(scored))} dev questions were not scored" + out = ROOT / "results" / args.name + out.mkdir(parents=True, exist_ok=True) + metrics = {"by_dataset": metrics_by_dataset(items, records), + "run": first["run"] | {"run_name": args.name, "combined_from": [d.name for d in dirs]}} + write_jsonl(out / "predictions.jsonl", records) + (out / "metrics.json").write_text(json.dumps(metrics, indent=2, ensure_ascii=False)) + (out / "report.md").write_text("\n".join(markdown_report(f"{args.name} / {k}", m) + for k, m in metrics["by_dataset"].items())) + o = metrics["by_dataset"]["sinhalammlu"]["overall"] + print(f"{args.name}: dev acc={o['acc']:.4f} vote={o['acc_vote']:.4f} over {o['n']} questions from {len(dirs)} folds") + + +if __name__ == "__main__": + main() diff --git a/training/eval/common.py b/training/eval/common.py new file mode 100644 index 0000000000000000000000000000000000000000..30bd3ba54f58a6173afe6cb2976ee1179967924e --- /dev/null +++ b/training/eval/common.py @@ -0,0 +1,115 @@ +"""Shared helpers: secrets, seeding, JSONL I/O, environment fingerprint.""" +from __future__ import annotations + +import json +import os +import platform +import random +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] + + +def ensure_python_headers() -> None: + """Triton (torch.compile, Liger, flash-linear-attention) compiles helpers against Python.h. This machine has + no python3.12-dev and no sudo, so the headers are unpacked to ~/.local/pydev (README, setup). Point gcc at + them when the system copy is missing; a no-op on machines that have the headers.""" + import sysconfig + + if (Path(sysconfig.get_paths()["include"]) / "Python.h").exists(): + return + local = Path.home() / ".local" / "pydev" / "root" / "usr" / "include" + if (local / f"python{sys.version_info.major}.{sys.version_info.minor}" / "Python.h").exists(): + inc = [str(local / f"python{sys.version_info.major}.{sys.version_info.minor}"), str(local)] + os.environ["CPATH"] = os.pathsep.join(inc + [p for p in os.environ.get("CPATH", "").split(os.pathsep) if p]) + + +def load_env(path: Path = ROOT / ".env") -> None: + """Read KEY=VALUE lines from .env; map hf_token -> HF_TOKEN for huggingface_hub.""" + ensure_python_headers() + if not path.exists(): + return + for line in path.read_text().splitlines(): + line = line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key, val = (s.strip().strip('"').strip("'") for s in line.split("=", 1)) + if key.lower() in ("hf_token", "hugging_face_hub_token"): + key = "HF_TOKEN" + os.environ.setdefault(key, val) + + +def set_seed(seed: int) -> None: + random.seed(seed) + os.environ["PYTHONHASHSEED"] = str(seed) + try: + import numpy as np + + np.random.seed(seed) + except ImportError: + pass + try: + import torch + + torch.manual_seed(seed) + torch.cuda.manual_seed_all(seed) + torch.backends.cuda.matmul.allow_tf32 = False + torch.backends.cudnn.allow_tf32 = False + except ImportError: + pass + + +def cap_gpu_memory(gb: float | None = None) -> None: + """Limit PyTorch's CUDA allocator (default 100 GB, env SMMLU_GPU_MEM_GB). + + The DGX Spark shares one 128 GB pool between CPU and GPU: an unbounded GPU job does not get a CUDA + out-of-memory error, it starves the system and the kernel OOM killer takes down the desktop session and + tmux (10 Oct, 11:15). With the cap, the job fails with torch.OutOfMemoryError and everything else survives. + """ + import torch + + if not torch.cuda.is_available(): + return + gb = float(gb or os.environ.get("SMMLU_GPU_MEM_GB", 100)) + total = torch.cuda.get_device_properties(0).total_memory + torch.cuda.set_per_process_memory_fraction(min(gb * 1024**3 / total, 1.0), 0) + + +def read_jsonl(path: str | Path) -> list[dict]: + with open(path, encoding="utf-8") as f: + return [json.loads(line) for line in f if line.strip()] + + +def write_jsonl(path: str | Path, rows) -> None: + Path(path).parent.mkdir(parents=True, exist_ok=True) + with open(path, "w", encoding="utf-8") as f: + for r in rows: + f.write(json.dumps(r, ensure_ascii=False) + "\n") + + +def env_fingerprint() -> dict: + """Versions and code revision, stored with every run for reproducibility.""" + info = {"python": platform.python_version(), "machine": platform.machine()} + for mod in ("torch", "transformers", "datasets", "peft", "accelerate", "huggingface_hub"): + try: + info[mod] = __import__(mod).__version__ + except ImportError: + pass + try: + import torch + + if torch.cuda.is_available(): + info["gpu"] = torch.cuda.get_device_name(0) + info["cuda"] = torch.version.cuda + except ImportError: + pass + try: + rev = subprocess.run(["git", "rev-parse", "HEAD"], cwd=ROOT, capture_output=True, text=True) + dirty = subprocess.run(["git", "status", "--porcelain"], cwd=ROOT, capture_output=True, text=True) + info["git_commit"] = rev.stdout.strip() or None + info["git_dirty"] = bool(dirty.stdout.strip()) + except FileNotFoundError: + pass + return info diff --git a/training/eval/compare.py b/training/eval/compare.py new file mode 100644 index 0000000000000000000000000000000000000000..f0546d3cc0230e2f1985da7c1d89d172eda66d07 --- /dev/null +++ b/training/eval/compare.py @@ -0,0 +1,45 @@ +"""Paired comparison of runs on the same questions (bootstrap 95% CI of the accuracy difference). + + python -m eval.compare --base results/exp_sft_gmmlu_lora_logit --runs results/exp_x results/exp_y [--dataset sinhalammlu] +""" +from __future__ import annotations + +import argparse +from pathlib import Path + +import numpy as np + +from .common import read_jsonl +from .metrics import aggregate_items + + +def correctness(run: Path, dataset: str) -> tuple[dict, dict]: + recs = read_jsonl(run / "predictions.jsonl") + items = {r["uid"]: {"options": r["perm"], "answer": r["gold"]} for r in recs} + agg = aggregate_items(items, recs) + keep = {u for u in agg if u.startswith({"sinhalammlu": "smmlu/", "gmmlu_si": "gmmlu_si/"}.get(dataset, ""))} + return ({u: agg[u]["pred"] == agg[u]["gold"] for u in keep}, {u: agg[u]["vote"] == agg[u]["gold"] for u in keep}) + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--base", required=True, type=Path) + ap.add_argument("--runs", nargs="+", required=True, type=Path) + ap.add_argument("--dataset", default="sinhalammlu", choices=["sinhalammlu", "gmmlu_si"]) + args = ap.parse_args() + base = correctness(args.base, args.dataset) + rng = np.random.default_rng(0) + for run in args.runs: + c = correctness(run, args.dataset) + for k, label in ((0, "orig"), (1, "vote")): + u = sorted(set(base[k]) & set(c[k])) + if not u: + continue + d = np.array([c[k][x] - base[k][x] for x in u], dtype=float) + lo, hi = np.percentile([d[rng.integers(0, len(d), len(d))].mean() for _ in range(5000)], [2.5, 97.5]) + print(f"{run.name:40s} {label}: {100 * np.mean([c[k][x] for x in u]):5.1f} vs {100 * np.mean([base[k][x] for x in u]):5.1f}" + f" diff {100 * d.mean():+.2f} [{100 * lo:+.2f}, {100 * hi:+.2f}] n={len(u)}") + + +if __name__ == "__main__": + main() diff --git a/training/eval/decide_rft.py b/training/eval/decide_rft.py new file mode 100644 index 0000000000000000000000000000000000000000..95b2f8fe9dc449e2c8c4c5b83d5b4a5363810f37 --- /dev/null +++ b/training/eval/decide_rft.py @@ -0,0 +1,52 @@ +"""Go / no-go for the reasoning-trace model, decided by a rule fixed in advance (before any result is seen). + +Compares greedy thinking of the trained model (with its adapter) with greedy thinking of the base model on the SAME held-out +questions: the 400-question SinhalaMMLU dev sample (never used for traces or training) and the 300-question Global-MMLU +non-academic proxy (never used). Paired differences are pooled over both sets. + +GO if pooled difference >= +1.5 points AND neither set is worse than -3.0 points. Writes results/rft_decision.json. + + python -m eval.decide_rft --pairs think_base_smmlu_dev_sample400:think_rft_dev400:sinhalammlu \ + think_base_gmmlu_si_val_nonacademic300:think_rft_nonacademic300:gmmlu_si +""" +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +import numpy as np + +from .common import ROOT +from .compare import correctness + +MIN_POOLED, MIN_EACH = 1.5, -3.0 + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--pairs", nargs="+", required=True, help="base_run:new_run:dataset") + ap.add_argument("--out", default="rft_decision.json", help="file name under results/") + args = ap.parse_args() + diffs, per = [], {} + for pair in args.pairs: + b, n, ds = pair.split(":") + base, new = correctness(ROOT / "results" / b, ds)[0], correctness(ROOT / "results" / n, ds)[0] + uids = sorted(set(base) & set(new)) + d = np.array([new[u] - base[u] for u in uids], dtype=float) + per[n] = {"n": len(uids), "base": 100 * float(np.mean([base[u] for u in uids])), + "new": 100 * float(np.mean([new[u] for u in uids])), "diff": 100 * float(d.mean())} + diffs.append(d) + pooled = np.concatenate(diffs) + rng = np.random.default_rng(0) + lo, hi = np.percentile([pooled[rng.integers(0, len(pooled), len(pooled))].mean() for _ in range(5000)], [2.5, 97.5]) + go = bool(100 * pooled.mean() >= MIN_POOLED and all(v["diff"] >= MIN_EACH for v in per.values())) + out = {"rule": f"pooled diff >= {MIN_POOLED} and each set >= {MIN_EACH}", "per_set": per, "n": len(pooled), + "pooled_diff": 100 * float(pooled.mean()), "ci95": [100 * float(lo), 100 * float(hi)], "go": go} + (ROOT / "results" / args.out).write_text(json.dumps(out, indent=2)) + print(json.dumps(out, indent=2)) + print("DECISION:", "GO" if go else "NO-GO") + + +if __name__ == "__main__": + main() diff --git a/training/eval/metrics.py b/training/eval/metrics.py new file mode 100644 index 0000000000000000000000000000000000000000..35b9552c0cba5fca91bebc14a86ae9a59c5ee3f9 --- /dev/null +++ b/training/eval/metrics.py @@ -0,0 +1,116 @@ +"""Aggregate per-(question, permutation) records into accuracy tables and position-bias diagnostics.""" +from __future__ import annotations + +from collections import Counter, defaultdict +from statistics import mean, pstdev + +from .scorers import wilson + +GROUP_KEYS = ("dataset", "category", "subject", "level", "grade", "cultural_sensitivity") + + +def group_value(item: dict, key: str): + return item.get(key, item.get("meta", {}).get(key)) + + +def aggregate_items(items: dict[str, dict], records: list[dict]) -> dict[str, dict]: + """Combine permutations per question: mean of option probabilities (in original order) -> vote.""" + by_uid = defaultdict(list) + for r in records: + by_uid[r["uid"]].append(r) + out = {} + for uid, rs in by_uid.items(): + n = len(items[uid]["options"]) + rs.sort(key=lambda r: r["perm_k"]) + mean_probs = [mean(r["probs_orig"][j] for r in rs) for j in range(n)] + if any(sum(r["probs_orig"]) > 0 for r in rs): + vote = max(range(n), key=lambda j: (mean_probs[j], -j)) + 1 + else: + vote = None + preds = [r["pred_orig"] for r in rs] + out[uid] = { + "gold": items[uid].get("answer"), "pred": rs[0]["pred_orig"], "vote": vote, + "consistent": len(set(preds)) == 1 and preds[0] is not None, + "mean_probs": mean_probs, + } + return out + + +def _acc_block(uids: list[str], agg: dict, records_by_perm: dict[int, dict[str, dict]]) -> dict: + n = len(uids) + k = sum(agg[u]["pred"] == agg[u]["gold"] for u in uids) + kv = sum(agg[u]["vote"] == agg[u]["gold"] for u in uids) + per_perm = [sum(rp[u]["correct"] for u in uids) / n for rp in records_by_perm.values()] + lo, hi = wilson(k, n) + return { + "n": n, "acc": k / n, "ci95": [lo, hi], "acc_vote": kv / n, + "acc_perm_mean": mean(per_perm), "acc_perm_std": pstdev(per_perm) if len(per_perm) > 1 else 0.0, + "consistency": sum(agg[u]["consistent"] for u in uids) / n, + } + + +def compute_metrics(items: dict[str, dict], records: list[dict]) -> dict: + agg = aggregate_items(items, records) + records_by_perm: dict[int, dict[str, dict]] = defaultdict(dict) + for r in records: + records_by_perm[r["perm_k"]][r["uid"]] = r + uids = sorted(agg) + + m = {"overall": _acc_block(uids, agg, records_by_perm), "groups": {}} + for key in GROUP_KEYS: + groups = defaultdict(list) + for u in uids: + v = group_value(items[u], key) + if v is not None: + groups[str(v)].append(u) + if len(groups) > 1 or (key in ("dataset", "subject", "category") and groups): + m["groups"][key] = {g: _acc_block(us, agg, records_by_perm) for g, us in sorted(groups.items())} + # Macro average over subjects (each subject weighted equally) + if "subject" in m["groups"]: + m["overall"]["acc_macro_subject"] = mean(b["acc"] for b in m["groups"]["subject"].values()) + + # Position bias over all (question, permutation) records + n_opt = Counter(len(items[r["uid"]]["options"]) for r in records).most_common(1)[0][0] + rs = [r for r in records if len(items[r["uid"]]["options"]) == n_opt] + pred_pos = Counter(r["pred_display"] for r in rs) + gold_pos = Counter(r["gold_display"] for r in rs) + acc_by_gold = {p: mean(r["correct"] for r in rs if r["gold_display"] == p) for p in sorted(gold_pos)} + m["position_bias"] = { + "n_options": n_opt, + "pred_dist": {str(p): pred_pos.get(p, 0) / len(rs) for p in range(1, n_opt + 1)}, + "pred_none": pred_pos.get(None, 0) / len(rs), + "gold_dist": {str(p): gold_pos.get(p, 0) / len(rs) for p in range(1, n_opt + 1)}, + "acc_by_gold_position": {str(p): a for p, a in acc_by_gold.items()}, + } + if any("label_mass" in r for r in records): + m["overall"]["label_mass_mean"] = mean(r["label_mass"] for r in records if "label_mass" in r) + if any(r.get("method") == "generate" for r in records): + m["overall"]["parse_fail_rate"] = mean(r["pred_display"] is None for r in records) + return m + + +def markdown_report(name: str, m: dict) -> str: + o = m["overall"] + lines = [f"# {name}", "", + f"**Accuracy (original order): {o['acc']:.4f}** " + f"[95% CI {o['ci95'][0]:.3f}-{o['ci95'][1]:.3f}], n={o['n']} ", + f"Permutation-vote accuracy: {o['acc_vote']:.4f} ", + f"Mean ± std over permutations: {o['acc_perm_mean']:.4f} ± {o['acc_perm_std']:.4f} ", + f"Consistency across permutations: {o['consistency']:.3f} "] + if "acc_macro_subject" in o: + lines.append(f"Macro average over subjects: {o['acc_macro_subject']:.4f} ") + if "label_mass_mean" in o: + lines.append(f"Mean probability mass on label tokens: {o['label_mass_mean']:.3f} ") + if "parse_fail_rate" in o: + lines.append(f"Unparseable generations: {o['parse_fail_rate']:.3f} ") + for key, groups in m["groups"].items(): + lines += ["", f"## By {key}", "", "| group | n | acc | 95% CI | vote acc | consistency |", "|---|---:|---:|---|---:|---:|"] + for g, b in groups.items(): + lines.append(f"| {g} | {b['n']} | {b['acc']:.3f} | {b['ci95'][0]:.2f}-{b['ci95'][1]:.2f} | " + f"{b['acc_vote']:.3f} | {b['consistency']:.2f} |") + pb = m["position_bias"] + lines += ["", "## Position bias (all permutations)", "", "| position | predicted | gold | acc when gold here |", "|---|---:|---:|---:|"] + for p in pb["pred_dist"]: + lines.append(f"| {p} | {pb['pred_dist'][p]:.3f} | {pb['gold_dist'][p]:.3f} | " + f"{pb['acc_by_gold_position'].get(p, float('nan')):.3f} |") + return "\n".join(lines) + "\n" diff --git a/training/eval/perm_sc.py b/training/eval/perm_sc.py new file mode 100644 index 0000000000000000000000000000000000000000..1caa53aa1f7515a36c83052d4239c2478313d499 --- /dev/null +++ b/training/eval/perm_sc.py @@ -0,0 +1,70 @@ +"""Option-order self-consistency validation (decision for the Medium run). + +Prior answer = the current pipeline's answer on the stand-in questions: majority vote of greedy thinking + 3 sampled paths +(ties -> greedy), the analogue of candidate B's final answer. Candidate method: add the answers from K re-orderings of the +options (thinking each time, mapped back to the original numbering) and vote again (ties -> prior). + +GO if the candidate beats the prior by at least +1.0 point on the stand-in. Writes results/perm_decision.json. + + python -m eval.perm_sc --greedy results/think_base_gmmlu_si_val_nonacademic300 --samples results/sc3_base_nonacademic \ + --perms results/permthink_nonacademic300 +""" +from __future__ import annotations + +import argparse +import json +from collections import Counter, defaultdict +from pathlib import Path + +import numpy as np + +from .common import ROOT, read_jsonl + +MIN_GAIN = 1.0 + + +def vote(ballots: list, first_wins: bool = True): + c = Counter(b for b in ballots if b) + if not c: + return None + top = max(c.values()) + w = [a for a, n in c.items() if n == top] + return ballots[0] if ballots[0] in w else min(w) + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--greedy", required=True, type=Path) + ap.add_argument("--samples", required=True, type=Path) + ap.add_argument("--perms", required=True, type=Path) + args = ap.parse_args() + g = {r["uid"]: r for r in read_jsonl(args.greedy / "predictions.jsonl") if r["perm_k"] == 0} + s, p = defaultdict(list), defaultdict(list) + for r in read_jsonl(args.samples / "predictions.jsonl"): + s[r["uid"]].append(r["pred_orig"]) + for r in read_jsonl(args.perms / "predictions.jsonl"): + if r["perm_k"] >= 1: + p[r["uid"]].append(r["pred_orig"]) + uids = sorted(set(g) & set(s) & set(p)) + gold = {u: g[u]["gold"] for u in uids} + prior = {u: vote([g[u]["pred_orig"]] + s[u]) for u in uids} + cand = {u: vote([prior[u]] + p[u]) for u in uids} + perms_only = {u: vote([g[u]["pred_orig"]] + p[u]) for u in uids} + acc = lambda m: np.array([m[u] == gold[u] for u in uids], dtype=float) # noqa: E731 + a_g, a_prior, a_cand, a_perm = acc({u: g[u]["pred_orig"] for u in uids}), acc(prior), acc(cand), acc(perms_only) + d = a_cand - a_prior + rng = np.random.default_rng(0) + lo, hi = np.percentile([d[rng.integers(0, len(d), len(d))].mean() for _ in range(5000)], [2.5, 97.5]) + singles = [100 * float(np.mean([p[u][i] == gold[u] for u in uids])) for i in range(min(len(p[u]) for u in uids))] + out = {"n": len(uids), "greedy": 100 * a_g.mean(), "prior_vote_greedy_plus_samples": 100 * a_prior.mean(), + "single_reordered_runs": singles, "greedy_plus_reorders_vote": 100 * a_perm.mean(), + "candidate_prior_plus_reorders": 100 * a_cand.mean(), "gain_vs_prior": 100 * float(d.mean()), + "ci95": [100 * float(lo), 100 * float(hi)], "rule": f"candidate - prior >= {MIN_GAIN}", + "go": bool(100 * d.mean() >= MIN_GAIN)} + (ROOT / "results" / "perm_decision.json").write_text(json.dumps(out, indent=2)) + print(json.dumps(out, indent=2)) + print("DECISION:", "GO" if out["go"] else "NO-GO") + + +if __name__ == "__main__": + main() diff --git a/training/eval/pool_decide.py b/training/eval/pool_decide.py new file mode 100644 index 0000000000000000000000000000000000000000..623aaf9c7508540f0b42b69e66815da49da40787 --- /dev/null +++ b/training/eval/pool_decide.py @@ -0,0 +1,68 @@ +"""Go / no-go for cross-model pooling, by a rule fixed (27 Oct, before any of these runs existed) on FRESH stand-in questions +(gmmlu_si_val_nonacademic_fresh300: never used for any earlier decision, training or trace generation). + +Baseline = candidate B's method: greedy thinking of the base model; where the fast one-step answer (base model, 4-order logit +vote) disagrees with it, vote over greedy + 3 sampled paths (ties -> greedy). +Candidate = vote over base greedy + 3 base samples + RFT2 greedy + 3 RFT2 samples, on every question (ties -> base greedy). + +GO if candidate - baseline >= +1.0 point. Writes results/pool_decision.json. + + python -m eval.pool_decide --fast pool_fast_base_fresh300 --base-greedy pool_think_base_fresh300 \ + --base-samples pool_sc3_base_fresh300 --rft-greedy pool_think_rft2_fresh300 --rft-samples pool_sc3_rft2_fresh300 +""" +from __future__ import annotations + +import argparse +import json +from collections import defaultdict + +import numpy as np + +from .common import ROOT, read_jsonl +from .perm_sc import vote + +MIN_GAIN = 1.0 + + +def main() -> None: + ap = argparse.ArgumentParser() + for a in ("fast", "base-greedy", "base-samples", "rft-greedy", "rft-samples"): + ap.add_argument(f"--{a}", required=True) + args = ap.parse_args() + + def runs(name: str) -> tuple[dict, dict]: + preds, probs = defaultdict(list), defaultdict(lambda: np.zeros(4)) + for r in read_jsonl(ROOT / "results" / name / "predictions.jsonl"): + preds[r["uid"]].append(r["pred_orig"]) + probs[r["uid"]][: len(r["probs_orig"])] += r["probs_orig"] + return preds, probs + + _, fast = runs(args.fast) + bg, _ = runs(args.base_greedy) + bs, _ = runs(args.base_samples) + rg, _ = runs(args.rft_greedy) + rs, _ = runs(args.rft_samples) + gold = {r["uid"]: r["gold"] for r in read_jsonl(ROOT / "results" / args.base_greedy / "predictions.jsonl")} + uids = sorted(set(bg) & set(bs) & set(rg) & set(rs) & set(fast)) + gated = {u: int(fast[u].argmax()) + 1 != bg[u][0] for u in uids} + baseline = {u: vote([bg[u][0]] + (bs[u] if gated[u] else [])) for u in uids} + cand = {u: vote([bg[u][0]] + bs[u] + [rg[u][0]] + rs[u]) for u in uids} + acc = lambda m: np.array([m[u] == gold[u] for u in uids], dtype=float) # noqa: E731 + d = acc(cand) - acc(baseline) + rng = np.random.default_rng(0) + lo, hi = np.percentile([d[rng.integers(0, len(d), len(d))].mean() for _ in range(5000)], [2.5, 97.5]) + out = {"n": len(uids), "gated_fraction": float(np.mean(list(gated.values()))), + "base_greedy": 100 * float(acc({u: bg[u][0] for u in uids}).mean()), + "rft2_greedy": 100 * float(acc({u: rg[u][0] for u in uids}).mean()), + "base_vote_all4": 100 * float(acc({u: vote([bg[u][0]] + bs[u]) for u in uids}).mean()), + "rft2_vote_all4": 100 * float(acc({u: vote([rg[u][0]] + rs[u]) for u in uids}).mean()), + "baseline_B": 100 * float(acc(baseline).mean()), "candidate_pooled8": 100 * float(acc(cand).mean()), + "gain": 100 * float(d.mean()), "ci95": [100 * float(lo), 100 * float(hi)], + "rule": f"candidate - baseline >= {MIN_GAIN}", "go": bool(100 * d.mean() >= MIN_GAIN)} + (ROOT / "results" / "pool_decision.json").write_text(json.dumps(out, indent=2)) + print(json.dumps(out, indent=2)) + print("DECISION:", "GO" if out["go"] else "NO-GO") + + +if __name__ == "__main__": + main() diff --git a/training/eval/prompts.py b/training/eval/prompts.py new file mode 100644 index 0000000000000000000000000000000000000000..009b4e1847ea0e1b0538dbf67f0bd0fb2668d9e2 --- /dev/null +++ b/training/eval/prompts.py @@ -0,0 +1,71 @@ +"""Prompt construction. Options are numbered 1..N because Codabench answers are digits ("1".."4"). + +Every model gets the same template text; only the wrapping differs (chat template vs raw text). +Few-shot examples are a fixed list, identical for every question (shared-task rule). +""" +from __future__ import annotations + +TEMPLATES = { + "en": { + "header": "The following is a multiple choice question about {subject}. " + "Reply with only the number (1-{n}) of the correct answer.", + "answer": "Answer:", + }, + # Sinhala wording should be reviewed by a native speaker. + "si": { + "header": "පහත දැක්වෙන්නේ {subject} පිළිබඳ බහුවරණ ප්‍රශ්නයකි. " + "නිවැරදි පිළිතුරේ අංකය (1-{n}) පමණක් ලියන්න.", + "answer": "පිළිතුර:", + }, +} + + +def subject_name(item: dict, lang: str) -> str: + if lang == "si" and item.get("meta", {}).get("subject_original"): + return item["meta"]["subject_original"] + return item["subject"].replace("_", " ") + + +def question_block(question: str, options: list[str]) -> str: + lines = [question] + [f"{i}. {opt}" for i, opt in enumerate(options, 1)] + return "\n".join(lines) + + +def user_text(item: dict, options: list[str], lang: str) -> str: + t = TEMPLATES[lang] + header = t["header"].format(subject=subject_name(item, lang), n=len(options)) + return f"{header}\n\n{question_block(item['question'], options)}" + + +class PromptBuilder: + def __init__(self, tokenizer, lang: str = "en", chat: bool = True, + fewshot: list[dict] | None = None, answer_prefix: str = "", + chat_kwargs: dict | None = None): + self.tok, self.lang, self.chat = tokenizer, lang, chat + self.fewshot = fewshot or [] + self.answer_prefix = answer_prefix + self.chat_kwargs = {"enable_thinking": False, **(chat_kwargs or {})} + + @property + def add_special_tokens(self) -> bool: + # Chat templates already insert BOS; raw prompts need the tokenizer to add it. + return not self.chat + + def mcq(self, item: dict, options: list[str]) -> str: + """Prompt whose next token should be the answer number.""" + ans = TEMPLATES[self.lang]["answer"] + if self.chat: + msgs = [] + for ex in self.fewshot: + msgs.append({"role": "user", "content": user_text(ex, ex["options"], self.lang)}) + msgs.append({"role": "assistant", "content": str(ex["answer"])}) + msgs.append({"role": "user", "content": user_text(item, options, self.lang)}) + prompt = self.tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, + **self.chat_kwargs) + return prompt + self.answer_prefix + shots = [f"{user_text(ex, ex['options'], self.lang)}\n{ans} {ex['answer']}" for ex in self.fewshot] + return "\n\n".join(shots + [f"{user_text(item, options, self.lang)}\n{ans}"]) + + def cloze_context(self, item: dict) -> str: + """Context for option-likelihood scoring: question only, options not listed.""" + return f"{item['question']}\n{TEMPLATES[self.lang]['answer']}" diff --git a/training/eval/recompute.py b/training/eval/recompute.py new file mode 100644 index 0000000000000000000000000000000000000000..e9bf7d07eaf43c12c62c942e9f2f7559ab2c3b2d --- /dev/null +++ b/training/eval/recompute.py @@ -0,0 +1,34 @@ +"""Recompute metrics.json / report.md of finished runs from their saved predictions (no model needed). +Use after changing data grouping or metrics code. Usage: python -m eval.recompute results/base_* +""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +from .common import read_jsonl +from .metrics import markdown_report +from .run_eval import metrics_by_dataset, resolve_data + + +def main() -> None: + for d in map(Path, sys.argv[1:]): + if not (d / "metrics.json").exists(): + continue + old = json.loads((d / "metrics.json").read_text()) + records = read_jsonl(d / "predictions.jsonl") + uids = {r["uid"] for r in records} + items = {r["uid"]: r for name in old["run"]["args"]["data"] for r in read_jsonl(resolve_data(name)) if r["uid"] in uids} + missing = uids - items.keys() + if missing: + print(f"{d.name}: {len(missing)} predicted uids not in current data, skipped") + continue + new = {"by_dataset": metrics_by_dataset(items, records), "run": old["run"]} + (d / "metrics.json").write_text(json.dumps(new, indent=2, ensure_ascii=False)) + (d / "report.md").write_text("\n".join(markdown_report(f"{d.name} / {k}", m) for k, m in new["by_dataset"].items())) + print(f"{d.name}: recomputed") + + +if __name__ == "__main__": + main() diff --git a/training/eval/run_eval.py b/training/eval/run_eval.py new file mode 100644 index 0000000000000000000000000000000000000000..92084f4f201a1e779458854463533c57100976c0 --- /dev/null +++ b/training/eval/run_eval.py @@ -0,0 +1,208 @@ +"""Evaluate one model on one or more processed MCQ files. + +Example: + python -m eval.run_eval --model Qwen/Qwen3-4B-Instruct-2507 --data sinhalammlu_dev gmmlu_si_val \ + --method logit --n-perms 4 --run-name qwen3-4b-inst_logit + +Outputs results//{config.json, predictions.jsonl, metrics.json, report.md} +and, with --write-submission, a Codabench-format predictions.zip per dataset. +""" +from __future__ import annotations + +import argparse +import json +import random +import re +import time +import zipfile +from pathlib import Path + +import yaml + +from .common import ROOT, cap_gpu_memory, env_fingerprint, load_env, read_jsonl, set_seed, write_jsonl +from .metrics import compute_metrics, markdown_report +from .prompts import PromptBuilder, user_text + +PROCESSED = ROOT / "data" / "processed" +BASE_NAME = re.compile(r"(-base|-pt)$|^google/gemma-4-e[24]b$", re.I) + + +def make_perm(n: int, k: int, mode: str, seed: int, uid: str) -> list[int]: + """perm[j] = original 0-based index shown at display position j. k=0 is the original order.""" + if k == 0: + return list(range(n)) + if mode == "identity": # repeated samples of the same prompt (self-consistency), no reordering + return list(range(n)) + if mode == "cyclic" and k < n: + return [(j + k) % n for j in range(n)] + rng = random.Random(f"{seed}|{uid}|{k}") + p = list(range(n)) + rng.shuffle(p) + return p + + +def resolve_data(name: str) -> Path: + p = Path(name) + return p if p.suffix == ".jsonl" else PROCESSED / f"{name}.jsonl" + + +def score_items(items: dict[str, dict], pb: PromptBuilder, scorer, method: str, n_perms: int, + perm_mode: str, seed: int, first_perm: int = 0) -> tuple[list[dict], str | None]: + """One record per (question, option order). Shared by evaluation and the submission pipeline. + Items without a gold answer (test data) get gold/correct = None.""" + n_perms = 1 if method == "cloze" else n_perms + jobs = [(uid, k, make_perm(len(it["options"]), k, perm_mode, seed, uid)) + for uid, it in items.items() for k in range(first_perm, n_perms)] + first_prompt = None + if method == "cloze": + res = scorer.cloze([pb.cloze_context(items[u]) for u, _, _ in jobs], [items[u]["options"] for u, _, _ in jobs]) + else: + prompts = [pb.mcq(items[u], [items[u]["options"][j] for j in perm]) for u, _, perm in jobs] + fn = scorer.logit if method == "logit" else scorer.generate + res = fn(prompts, [len(items[u]["options"]) for u, _, _ in jobs]) + first_prompt = prompts[0] if prompts else None + + records = [] + for (uid, k, perm), r in zip(jobs, res): + gold = items[uid].get("answer") + probs_orig = [0.0] * len(perm) + for j, p in enumerate(r["probs"]): + probs_orig[perm[j]] = p + pred_orig = perm[r["pred"] - 1] + 1 if r["pred"] else None + rec = {"uid": uid, "perm_k": k, "perm": perm, "method": method, + "pred_display": r["pred"], "gold_display": perm.index(gold - 1) + 1 if gold else None, + "pred_orig": pred_orig, "gold": gold, "correct": pred_orig == gold if gold else None, + "probs_orig": probs_orig} + for extra in ("label_mass", "text", "new_tokens"): + if extra in r: + rec[extra] = r[extra] + records.append(rec) + return records, first_prompt + + +def parse_args() -> argparse.Namespace: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--config", help="YAML with any of the options below (CLI wins)") + ap.add_argument("--model") + ap.add_argument("--revision") + ap.add_argument("--adapter", help="LoRA adapter directory loaded on top of --model") + ap.add_argument("--data", nargs="+", default=["sinhalammlu_dev"]) + ap.add_argument("--method", choices=["logit", "generate", "cloze"], default="logit") + ap.add_argument("--lang", choices=["en", "si"], default="en", help="instruction language") + ap.add_argument("--chat", choices=["auto", "on", "off"], default="auto") + ap.add_argument("--answer-prefix", default="") + ap.add_argument("--thinking", action="store_true", help="enable the chat template's thinking mode (use with --method generate)") + ap.add_argument("--n-perms", type=int, default=1, help="option orders per question (1 = original only)") + ap.add_argument("--first-perm", type=int, default=0, help="skip the first orders (1 = skip the original order)") + ap.add_argument("--perm-mode", choices=["cyclic", "random", "identity"], default="cyclic", + help="identity + --temperature > 0: n-perms independent samples of the same prompt") + ap.add_argument("--temperature", type=float, default=0.0, help="> 0: sampled generation (self-consistency)") + ap.add_argument("--top-p", type=float, default=0.95) + ap.add_argument("--top-k", type=int, default=64) + ap.add_argument("--fewshot-file", help="processed JSONL; the same examples precede every question") + ap.add_argument("--n-shot", type=int, default=0) + ap.add_argument("--max-new-tokens", type=int, default=16) + ap.add_argument("--token-budget", type=int, default=32768, help="padded tokens per forward batch") + ap.add_argument("--max-batch", type=int, default=64) + ap.add_argument("--dtype", default="bfloat16") + ap.add_argument("--gpu-mem-gb", type=float, help="PyTorch memory cap (default 100, env SMMLU_GPU_MEM_GB)") + ap.add_argument("--limit", type=int, help="first N questions per dataset (smoke tests)") + ap.add_argument("--seed", type=int, default=1234) + ap.add_argument("--run-name") + ap.add_argument("--out-dir", default=str(ROOT / "results")) + ap.add_argument("--write-submission", action="store_true") + args, _ = ap.parse_known_args() + if args.config: + ap.set_defaults(**{k.replace("-", "_"): v for k, v in yaml.safe_load(open(args.config)).items()}) + args = ap.parse_args() + if not args.model: + ap.error("--model is required") + return args + + +def main() -> None: + args = parse_args() + load_env() + set_seed(args.seed) + cap_gpu_memory(args.gpu_mem_gb) + from .scorers import Scorer, load_model, loaded_param_stats + + run_name = args.run_name or f"{args.model.split('/')[-1]}_{args.method}_{args.lang}_{args.n_shot}shot" + out = Path(args.out_dir) / run_name + out.mkdir(parents=True, exist_ok=True) + + fewshot = read_jsonl(resolve_data(args.fewshot_file))[: args.n_shot] if args.n_shot else [] + shot_uids = {e["uid"] for e in fewshot} + items: dict[str, dict] = {} + for d in args.data: + rows = [r for r in read_jsonl(resolve_data(d)) if r["uid"] not in shot_uids] + items.update({r["uid"]: r for r in (rows[: args.limit] if args.limit else rows)}) + + t0 = time.time() + model, tok = load_model(args.model, args.revision, args.dtype, args.adapter) + chat = {"on": True, "off": False}.get(args.chat, tok.chat_template is not None and not BASE_NAME.search(args.model)) + pb = PromptBuilder(tok, lang=args.lang, chat=chat, fewshot=fewshot, answer_prefix=args.answer_prefix, + chat_kwargs={"enable_thinking": args.thinking}) + scorer = Scorer(model, tok, pb.add_special_tokens, args.token_budget, args.max_batch, args.max_new_tokens) + if args.temperature > 0: + scorer.sampling = {"do_sample": True, "temperature": args.temperature, "top_p": args.top_p, "top_k": args.top_k} + load_s = time.time() - t0 + + t0 = time.time() + records, first_prompt = score_items(items, pb, scorer, args.method, args.n_perms, args.perm_mode, args.seed, args.first_perm) + score_s = time.time() - t0 + if first_prompt: + (out / "example_prompt.txt").write_text(first_prompt, encoding="utf-8") + + # Tokens per character of the question block: tokenizer efficiency on Sinhala + sample = list(items.values())[:500] + texts = [user_text(it, it["options"], args.lang) for it in sample] + n_tok = sum(len(x) for x in tok(texts, add_special_tokens=False)["input_ids"]) + + metrics = {"by_dataset": metrics_by_dataset(items, records)} + metrics["run"] = { + "run_name": run_name, "args": vars(args), "chat": chat, "n_questions": len(items), + "n_forward_items": len(records), "load_seconds": round(load_s, 1), "score_seconds": round(score_s, 1), + "tokens_per_char": n_tok / sum(len(t) for t in texts), + "params": loaded_param_stats(model), "env": env_fingerprint(), + } + (out / "config.json").write_text(json.dumps(vars(args), indent=2)) + write_jsonl(out / "predictions.jsonl", records) + (out / "metrics.json").write_text(json.dumps(metrics, indent=2, ensure_ascii=False)) + report = "\n".join(markdown_report(f"{run_name} / {d}", m) for d, m in metrics["by_dataset"].items()) + (out / "report.md").write_text(report, encoding="utf-8") + + if args.write_submission: + write_submissions(out, items, records) + + for d, m in metrics["by_dataset"].items(): + o = m["overall"] + print(f"{run_name} | {d}: acc={o['acc']:.4f} vote={o['acc_vote']:.4f} " + f"perm={o['acc_perm_mean']:.4f}±{o['acc_perm_std']:.4f} n={o['n']}") + print(f"load {load_s:.0f}s, scoring {score_s:.0f}s -> {out}") + + +def metrics_by_dataset(items: dict, records: list[dict]) -> dict: + out = {} + for d in sorted({it["dataset"] for it in items.values()}): + sub_items = {u: it for u, it in items.items() if it["dataset"] == d} + out[d] = compute_metrics(sub_items, [r for r in records if r["uid"] in sub_items]) + return out + + +def write_submissions(out: Path, items: dict, records: list[dict]) -> None: + """Codabench format: predictions.zip containing predictions.jsonl, answer = permutation-vote prediction.""" + from .metrics import aggregate_items + + agg = aggregate_items(items, records) + for d in sorted({it["dataset"] for it in items.values()}): + rows = [{"q_no": it["q_no"], "subject": it.get("subject_raw", it["subject"]), "category": it["category"], + "level": it["level"], "answer": str(agg[u]["vote"] or 1)} + for u, it in items.items() if it["dataset"] == d] + path = out / f"submission_{d}.zip" + with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as z: + z.writestr("predictions.jsonl", "".join(json.dumps(r, ensure_ascii=False) + "\n" for r in rows)) + + +if __name__ == "__main__": + main() diff --git a/training/eval/scorers.py b/training/eval/scorers.py new file mode 100644 index 0000000000000000000000000000000000000000..3e4cac0356e933ae5f62c6e8195f492c8f27edf0 --- /dev/null +++ b/training/eval/scorers.py @@ -0,0 +1,224 @@ +"""Model loading and the three scoring methods. + + logit next-token log-probs restricted to the option labels "1".."N" (argmax) + generate greedy decoding, first standalone label digit in the output is the prediction + cloze length-normalised log-likelihood of each option text given the question +""" +from __future__ import annotations + +import math +import re + +import torch +from tqdm.auto import tqdm + + +def resolve_revision(name: str, revision: str | None) -> tuple[str | None, str | None]: + """(revision to load, commit to record). Local dirs record their SOURCE.json; offline runs need a full sha.""" + import json + import os + from pathlib import Path + + if Path(name).is_dir(): + src = Path(name) / "SOURCE.json" + if not src.exists(): + return None, "local" + meta = json.loads(src.read_text()) + return None, meta.get("model_id") or meta["source_commit"] + "+text-only" + if revision and re.fullmatch(r"[0-9a-f]{40}", revision): + return revision, revision + if os.environ.get("HF_HUB_OFFLINE") == "1": + raise ValueError(f"offline run needs a full 40-character commit sha for {name}") + from huggingface_hub import HfApi + + sha = HfApi().model_info(name, revision=revision).sha # pin the exact commit + return sha, sha + + +def load_model(name: str, revision: str | None = None, dtype: str = "bfloat16", adapter: str | None = None): + from transformers import AutoModelForCausalLM, AutoTokenizer + + revision, commit = resolve_revision(name, revision) + kw = dict(dtype=getattr(torch, dtype), device_map="cuda", revision=revision) + try: + model = AutoModelForCausalLM.from_pretrained(name, **kw) + except ValueError: # some multimodal checkpoints only register under image-text-to-text + from transformers import AutoModelForImageTextToText + + model = AutoModelForImageTextToText.from_pretrained(name, **kw) + if adapter: # LoRA adapter kept separate (can be switched off per question with model.disable_adapter()) + from peft import PeftModel + + model = PeftModel.from_pretrained(model, adapter) + commit = f"{commit}+adapter:{adapter}" + model.eval() + tok = AutoTokenizer.from_pretrained(name, revision=revision) + tok.padding_side = "left" + if tok.pad_token is None: + tok.pad_token = tok.eos_token + model.config.hub_commit = commit + return model, tok + + +def loaded_param_stats(model) -> dict: + """Parameters actually resident at inference (tied weights counted once).""" + by_module: dict[str, int] = {} + for name, p in model.named_parameters(): # named_parameters() already de-duplicates tied tensors + top = ".".join(name.split(".")[:2]) + by_module[top] = by_module.get(top, 0) + p.numel() + non_text = {k: v for k, v in by_module.items() if re.search(r"vision|visual|audio|image|mm_proj", k)} + total = sum(by_module.values()) + return {"loaded_total": total, "loaded_non_text": sum(non_text.values()), + "non_text_modules": non_text, "commit": getattr(model.config, "hub_commit", None)} + + +def label_token_ids(tok, n: int) -> list[list[int]]: + """Single-token encodings of each label ("1" and " 1" variants).""" + out = [] + for i in range(1, n + 1): + ids = set() + for v in (str(i), f" {i}"): + enc = tok.encode(v, add_special_tokens=False) + if len(enc) == 1: + ids.add(enc[0]) + if not ids: + raise ValueError(f"label {i} has no single-token encoding") + out.append(sorted(ids)) + flat = [t for ids in out for t in ids] + assert len(flat) == len(set(flat)), "label token ids overlap" + return out + + +def _batches(lengths: list[int], token_budget: int, max_batch: int): + """Length-sorted batches bounded by padded-token budget.""" + order = sorted(range(len(lengths)), key=lambda i: -lengths[i]) + batch, longest = [], 0 + for i in order: + new_longest = max(longest, lengths[i]) + if batch and (new_longest * (len(batch) + 1) > token_budget or len(batch) >= max_batch): + yield batch + batch, new_longest = [], lengths[i] + batch.append(i) + longest = new_longest + if batch: + yield batch + + +def _forward_last(model, enc, keep: int): + try: + return model(**enc, logits_to_keep=keep).logits + except TypeError: + return model(**enc).logits[:, -keep:] + + +class Scorer: + def __init__(self, model, tok, add_special_tokens: bool, token_budget: int = 32768, + max_batch: int = 64, max_new_tokens: int = 16): + self.model, self.tok = model, tok + self.add_special = add_special_tokens + self.token_budget, self.max_batch = token_budget, max_batch + self.max_new_tokens = max_new_tokens + self.keep_raw = False # also return the raw completion (thought channel included) for training on traces + self.sampling: dict = {} # e.g. {"do_sample": True, "temperature": 0.8, "top_p": 0.95, "top_k": 64} + + def _encode(self, texts: list[str]): + return self.tok(texts, return_tensors="pt", padding=True, + add_special_tokens=self.add_special).to(self.model.device) + + def _lengths(self, texts: list[str]) -> list[int]: + return [len(x) for x in self.tok(texts, add_special_tokens=self.add_special)["input_ids"]] + + @torch.no_grad() + def logit(self, prompts: list[str], n_options: list[int]) -> list[dict]: + cache: dict[int, list[list[int]]] = {} + results: list[dict | None] = [None] * len(prompts) + lengths = self._lengths(prompts) + for idx in tqdm(list(_batches(lengths, self.token_budget, self.max_batch)), desc="logit", leave=False): + enc = self._encode([prompts[i] for i in idx]) + logp = torch.log_softmax(_forward_last(self.model, enc, 1)[:, -1].float(), dim=-1) + for row, i in enumerate(idx): + n = n_options[i] + labels = cache.setdefault(n, label_token_ids(self.tok, n)) + scores = torch.stack([torch.logsumexp(logp[row, ids], 0) for ids in labels]) + probs = torch.softmax(scores, 0) + results[i] = {"probs": probs.tolist(), "pred": int(probs.argmax()) + 1, + "label_mass": float(scores.exp().sum())} + return results + + @torch.no_grad() + def generate(self, prompts: list[str], n_options: list[int]) -> list[dict]: + results: list[dict | None] = [None] * len(prompts) + lengths = self._lengths(prompts) + budget = max(self.token_budget // 2, 1) + for idx in tqdm(list(_batches(lengths, budget, self.max_batch)), desc="generate", leave=False): + enc = self._encode([prompts[i] for i in idx]) + gen_kw = self.sampling or {"do_sample": False, "temperature": None, "top_p": None, "top_k": None} + out = self.model.generate(**enc, max_new_tokens=self.max_new_tokens, pad_token_id=self.tok.pad_token_id, + **gen_kw) + new = out[:, enc["input_ids"].shape[1]:] + raw = self.tok.batch_decode(new, skip_special_tokens=False) + texts = self.tok.batch_decode(new, skip_special_tokens=True) + for row, i in enumerate(idx): + n = n_options[i] + answer_text = texts[row] + if "<|channel>" in raw[row]: # thinking mode: the answer is what follows the closed thought channel + answer_text = raw[row].split("")[-1] if "" in raw[row] else "" + answer_text = re.sub(r"<[^>]*\|>|<\|[^>]*>", " ", answer_text) + m = re.search(rf"(?") + results[i]["raw"] = rc[: cut + len("")] if cut >= 0 else rc + return results + + @torch.no_grad() + def cloze(self, contexts: list[str], options: list[list[str]]) -> list[dict]: + """Left-padded so every continuation ends at the last position; keeps only needed logits.""" + pairs = [] # (item index, ctx ids, cont ids, n chars) + for i, (ctx, opts) in enumerate(zip(contexts, options)): + ctx_ids = self.tok.encode(ctx, add_special_tokens=True) + for opt in opts: + cont = self.tok.encode(" " + opt, add_special_tokens=False) + pairs.append((i, ctx_ids, cont, len(opt))) + lengths = [len(c) + len(o) for _, c, o, _ in pairs] + scores = [0.0] * len(pairs) + pad = self.tok.pad_token_id + for idx in tqdm(list(_batches(lengths, self.token_budget, self.max_batch)), desc="cloze", leave=False): + L = max(lengths[j] for j in idx) + keep = max(len(pairs[j][2]) for j in idx) + 1 + ids = torch.full((len(idx), L), pad, dtype=torch.long) + mask = torch.zeros((len(idx), L), dtype=torch.long) + for row, j in enumerate(idx): + seq = pairs[j][1] + pairs[j][2] + ids[row, L - len(seq):] = torch.tensor(seq) + mask[row, L - len(seq):] = 1 + enc = {"input_ids": ids.to(self.model.device), "attention_mask": mask.to(self.model.device)} + logp = torch.log_softmax(_forward_last(self.model, enc, keep).float(), dim=-1) + for row, j in enumerate(idx): + cont = pairs[j][2] + k = len(cont) + # logits at kept positions [-k-1 .. -2] predict tokens at [-k .. -1] + lp = logp[row, keep - k - 1: keep - 1].gather(-1, torch.tensor(cont, device=logp.device)[:, None]) + scores[j] = float(lp.sum()) / max(pairs[j][3], 1) + results: list[dict] = [] + pos = 0 + for opts in options: + s = torch.tensor(scores[pos: pos + len(opts)]) + pos += len(opts) + probs = torch.softmax(s, 0) + results.append({"probs": probs.tolist(), "pred": int(probs.argmax()) + 1}) + return results + + +def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]: + if n == 0: + return (math.nan, math.nan) + p = k / n + d = 1 + z * z / n + c = (p + z * z / (2 * n)) / d + h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d + return (c - h, c + h) diff --git a/training/eval/self_consistency.py b/training/eval/self_consistency.py new file mode 100644 index 0000000000000000000000000000000000000000..62e14835649728722a462bae29e966ef27946223 --- /dev/null +++ b/training/eval/self_consistency.py @@ -0,0 +1,52 @@ +"""Self-consistency: majority vote over a greedy thinking run and sampled thinking runs of the same questions. + + python -m eval.self_consistency --greedy results/think_base_academic --samples results/sc3_base_academic + +Votes = greedy answer + every sampled answer (unparsed samples abstain); ties go to the greedy answer. Prints accuracy of +greedy alone, samples alone (each), and the vote, with a paired bootstrap CI of vote minus greedy. +""" +from __future__ import annotations + +import argparse +from collections import Counter, defaultdict +from pathlib import Path + +import numpy as np + +from .common import read_jsonl + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--greedy", required=True, type=Path) + ap.add_argument("--samples", required=True, type=Path) + args = ap.parse_args() + greedy = {r["uid"]: r for r in read_jsonl(args.greedy / "predictions.jsonl") if r["perm_k"] == 0} + samples = defaultdict(list) + for r in read_jsonl(args.samples / "predictions.jsonl"): + samples[r["uid"]].append(r["pred_orig"]) + uids = sorted(set(greedy) & set(samples)) + g = np.array([greedy[u]["pred_orig"] == greedy[u]["gold"] for u in uids], dtype=float) + vote = [] + for u in uids: + ballots = [greedy[u]["pred_orig"]] + samples[u] + c = Counter(b for b in ballots if b is not None) + if not c: + vote.append(False) + continue + top = max(c.values()) + winners = [a for a, n in c.items() if n == top] + choice = greedy[u]["pred_orig"] if greedy[u]["pred_orig"] in winners else min(winners) + vote.append(choice == greedy[u]["gold"]) + v = np.array(vote, dtype=float) + k = max(len(samples[u]) for u in uids) + per_sample = [np.mean([samples[u][i] == greedy[u]["gold"] for u in uids if len(samples[u]) > i]) for i in range(k)] + d = v - g + rng = np.random.default_rng(0) + lo, hi = np.percentile([d[rng.integers(0, len(d), len(d))].mean() for _ in range(5000)], [2.5, 97.5]) + print(f"{args.samples.name}: n={len(uids)} greedy {100 * g.mean():.1f} samples {' / '.join(f'{100 * x:.1f}' for x in per_sample)}" + f" vote(greedy+{k}) {100 * v.mean():.1f} diff {100 * d.mean():+.2f} [{100 * lo:+.2f}, {100 * hi:+.2f}]") + + +if __name__ == "__main__": + main() diff --git a/training/eval/summarize.py b/training/eval/summarize.py new file mode 100644 index 0000000000000000000000000000000000000000..abdb7d80bcf4b197943b3c776b2d4c3e410ebe3a --- /dev/null +++ b/training/eval/summarize.py @@ -0,0 +1,84 @@ +"""Combine results/*/metrics.json into comparison tables (reports/.md). + +Usage: python -m eval.summarize --runs results/base_* --out reports/baselines_zeroshot.md +""" +from __future__ import annotations + +import argparse +import glob +import json +from pathlib import Path + + +def load(run_dirs: list[str]) -> dict[str, dict]: + runs = {} + for d in sorted(run_dirs): + f = Path(d) / "metrics.json" + if f.exists(): + runs[Path(d).name] = json.loads(f.read_text()) + return runs + + +def fmt(x) -> str: + return "-" if x is None else f"{100 * x:.1f}" + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--runs", nargs="+", required=True) + ap.add_argument("--out", required=True) + args = ap.parse_args() + runs = load([d for pat in args.runs for d in glob.glob(pat)]) + names = list(runs) + short = {n: n.removeprefix("base_").removeprefix("exp_") for n in names} + + lines = ["# Baseline comparison", "", + "Accuracy in %. `orig` = original option order; `vote` = mean probabilities over cyclic option " + "permutations; `±` = std of accuracy across permutations (position sensitivity). Random = 25.", ""] + + # Overview + lines += ["## Overview", "", "| model | params loaded | tok/char | SinhalaMMLU dev orig | vote | ± | macro-subj | " + "Global-MMLU si | Belebele si | label mass | consistency |", "|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|"] + for n in names: + r, bd = runs[n]["run"], runs[n]["by_dataset"] + s = bd.get("sinhalammlu", {}).get("overall", {}) + g = bd.get("gmmlu_si", {}).get("overall", {}) + b = bd.get("belebele_si", {}).get("overall", {}) + lines.append(f"| {short[n]} | {r['params']['loaded_total'] / 1e9:.2f}B | {r['tokens_per_char']:.2f} | " + f"{fmt(s.get('acc'))} | {fmt(s.get('acc_vote'))} | {fmt(s.get('acc_perm_std'))} | " + f"{fmt(s.get('acc_macro_subject'))} | {fmt(g.get('acc'))} | {fmt(b.get('acc'))} | " + f"{s.get('label_mass_mean', float('nan')):.2f} | {s.get('consistency', float('nan')):.2f} |") + + def table(dataset: str, key: str, title: str) -> None: + groups = sorted({g for n in names for g in runs[n]["by_dataset"].get(dataset, {}).get("groups", {}).get(key, {})}) + if not groups: + return + lines.extend(["", f"## {title}", "", "| " + key + " | n | " + " | ".join(short[n] for n in names) + " |", + "|---|---:|" + "---:|" * len(names)]) + for g in groups: + cells, cnt = [], "-" + for n in names: + blk = runs[n]["by_dataset"].get(dataset, {}).get("groups", {}).get(key, {}).get(g) + cells.append(fmt(blk["acc"]) if blk else "-") + cnt = blk["n"] if blk else cnt + lines.append(f"| {g} | {cnt} | " + " | ".join(cells) + " |") + + table("sinhalammlu", "subject", "SinhalaMMLU dev: per subject") + table("sinhalammlu", "category", "SinhalaMMLU dev: per domain") + table("sinhalammlu", "grade", "SinhalaMMLU dev: per grade") + table("gmmlu_si", "category", "Global-MMLU si: per category") + table("gmmlu_si", "cultural_sensitivity", "Global-MMLU si: culturally sensitive (CS) / agnostic (CA) / unlabelled (-)") + + lines += ["", "## Position bias on SinhalaMMLU dev (share of predictions per displayed position)", "", + "| model | 1 | 2 | 3 | 4 |", "|---|---:|---:|---:|---:|"] + for n in names: + pb = runs[n]["by_dataset"].get("sinhalammlu", {}).get("position_bias") + if pb: + lines.append(f"| {short[n]} | " + " | ".join(fmt(pb["pred_dist"].get(str(p))) for p in range(1, 5)) + " |") + Path(args.out).parent.mkdir(parents=True, exist_ok=True) + Path(args.out).write_text("\n".join(lines) + "\n") + print("\n".join(lines)) + + +if __name__ == "__main__": + main() diff --git a/training/inference/__init__.py b/training/inference/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/training/inference/check_predictions.py b/training/inference/check_predictions.py new file mode 100644 index 0000000000000000000000000000000000000000..6cd05bcc19696399af8867b55c7f9d7c3a1aa86b --- /dev/null +++ b/training/inference/check_predictions.py @@ -0,0 +1,73 @@ +"""Score a Codabench-format predictions.zip against gold answers (dev only; test gold is hidden). + + python -m inference.check_predictions --predictions dev_predictions.zip --gold data/raw/sinhalammlu/TEST/easy + +Also validates the file the way the leaderboard will need it: one row per question, required fields present, +answers are digits within the option range. Reports accuracy per level and their macro average (task metric). +""" +from __future__ import annotations + +import argparse +import json +import sys +import zipfile +from collections import defaultdict +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from inference.predict import read_records, to_item # noqa: E402 + +FIELDS = ("q_no", "subject", "category", "level", "answer") + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--predictions", required=True, type=Path) + ap.add_argument("--gold", required=True, type=Path) + args = ap.parse_args() + + with zipfile.ZipFile(args.predictions) as z: + assert z.namelist() == ["predictions.jsonl"], f"zip must contain only predictions.jsonl, has {z.namelist()}" + preds = [json.loads(l) for l in z.read("predictions.jsonl").decode("utf-8").splitlines() if l.strip()] + + gold = {} + for f, r in read_records(args.gold): + it = to_item(f, r) + gold[(it["level"], it["subject_raw"], it["q_no"])] = (r.get("answer"), len(it["options"])) + + errors, seen = [], set() + correct = defaultdict(lambda: [0, 0]) + for p in preds: + if any(k not in p for k in FIELDS): + errors.append(f"missing fields in {p}") + continue + key = (p["level"], p["subject"], p["q_no"]) + if key in seen: + errors.append(f"duplicate {key}") + seen.add(key) + if key not in gold: + errors.append(f"unknown question {key}") + continue + ans, n = gold[key] + if not (isinstance(p["answer"], str) and p["answer"].isdigit() and 1 <= int(p["answer"]) <= n): + errors.append(f"bad answer {p['answer']!r} for {key}") + continue + if isinstance(ans, int) and 1 <= ans <= n: # skip items with an invalid gold label + correct[p["level"]][0] += int(p["answer"]) == ans + correct[p["level"]][1] += 1 + missing = set(gold) - seen + if missing: + errors.append(f"{len(missing)} questions without a prediction, e.g. {sorted(missing)[:3]}") + + for lvl, (k, n) in sorted(correct.items()): + print(f"{lvl}: {k}/{n} = {100 * k / n:.2f}%") + if correct: + print(f"macro average over levels: {100 * sum(k / n for k, n in correct.values()) / len(correct):.2f}%") + print(f"{len(preds)} rows, {len(errors)} format errors") + for e in errors[:20]: + print(" ", e) + sys.exit(1 if errors else 0) + + +if __name__ == "__main__": + main() diff --git a/training/inference/predict.py b/training/inference/predict.py new file mode 100644 index 0000000000000000000000000000000000000000..9f0845f9f158caf83c84ab4e0f770c0a5e2f6c93 --- /dev/null +++ b/training/inference/predict.py @@ -0,0 +1,238 @@ +"""Offline submission pipeline: question file(s) in, Codabench predictions.zip out. No human in the loop. + + python -m inference.predict --input --output predictions.zip + +Runs with HF_HUB_OFFLINE=1 (no network); every setting comes from configs/submission.yaml. +Optional self-consistency (config `sc_levels`, `sc_samples`, `sc_gate`): greedy + K sampled reasoning paths, majority +vote; with `sc_gate: disagree` only for questions where the fast answer disagrees with the greedy thinking answer. +Every thinking answer (greedy first, then samples) is also written to .ballots.jsonl (scripts/pool_ballots.py). +Optional LoRA adapter (config `adapter`) on top of `model`. Optional per-question thinking (config `thinking_levels`, +e.g. [hard]): questions whose own `level` field is listed are answered with the model's thinking mode (with +`thinking_without_adapter: true`, the adapter is switched off for them) (greedy, original option order); if no answer can be read from the output, +the fast answer is used. All other questions use the fast logit answer (voted over `n_perms` option orders). + +Accepts the SinhalaMMLU JSON layout (list of {q_no, subject, category, question, choices, metadata.difficulty}) +and JSONL; q_no / subject / category / level are echoed exactly as given. Writes .run.json next to the +zip with the model commit, loaded parameter count, input hash and library versions. +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import sys +import time +import zipfile +from collections import Counter +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +LEVELS = ("easy", "medium", "hard") + + +def read_records(path: Path) -> list[tuple[Path, dict]]: + files = sorted(p for p in path.rglob("*") if p.suffix in (".json", ".jsonl") + and not any(part.startswith(".") for part in p.relative_to(path).parts)) if path.is_dir() else [path] + # (hidden dirs skipped: huggingface_hub keeps download metadata as JSON under .cache/) + out = [] + for f in files: + text = f.read_text(encoding="utf-8") + rows = [json.loads(l) for l in text.splitlines() if l.strip()] if f.suffix == ".jsonl" else json.loads(text) + out += [(f, r) for r in (rows if isinstance(rows, list) else [rows])] + if not out: + raise SystemExit(f"no questions found under {path}") + return out + + +def to_item(f: Path, r: dict) -> dict: + options = r.get("choices", r.get("options")) + level = r.get("level") or (r.get("metadata") or {}).get("difficulty") \ + or next((p for p in f.parts[::-1] if p.lower() in LEVELS), None) + missing = [k for k, v in {"q_no": r.get("q_no"), "subject": r.get("subject"), "category": r.get("category"), + "question": r.get("question"), "choices": options, "level": level}.items() if v is None] + if missing: + raise SystemExit(f"{f}: record {r.get('q_no')} lacks {missing}; keys present: {sorted(r)}") + return {"uid": f"{level}/{r['subject']}/{r['q_no']}", "dataset": "test", "q_no": r["q_no"], + "subject": r["subject"].strip(), "subject_raw": r["subject"], "category": r["category"], "level": level, + "question": r["question"].strip(), "options": [str(o).strip() for o in options]} + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--input", required=True, type=Path) + ap.add_argument("--output", required=True, type=Path) + ap.add_argument("--config", type=Path, default=Path(__file__).resolve().parents[1] / "configs" / "submission.yaml") + ap.add_argument("--allow-online", action="store_true", help="debugging only; the submission runs offline") + ap.add_argument("--limit", type=int, help="first N questions only (smoke tests; never for a submission)") + ap.add_argument("--levels", nargs="+", help="answer only questions of these levels (then merge with scripts/merge_predictions.py)") + args = ap.parse_args() + if not args.allow_online: + os.environ["HF_HUB_OFFLINE"] = os.environ["TRANSFORMERS_OFFLINE"] = "1" + + import yaml + + from eval.common import cap_gpu_memory, env_fingerprint, set_seed + from eval.metrics import aggregate_items + from eval.prompts import PromptBuilder + from eval.run_eval import make_perm, score_items + from eval.scorers import Scorer, load_model, loaded_param_stats + + cfg = yaml.safe_load(args.config.read_text()) + set_seed(cfg["seed"]) + cap_gpu_memory(cfg.get("gpu_mem_gb")) + + raw = read_records(args.input) + items = {} + for f, r in raw: + it = to_item(f, r) + if it["uid"] in items: + raise SystemExit(f"duplicate question id {it['uid']}") + items[it["uid"]] = it + if args.levels: + items = {u: it for u, it in items.items() if it["level"] in set(args.levels)} + if args.limit: + items = dict(list(items.items())[: args.limit]) + print(f"{len(items)} questions from {args.input}") + + t0 = time.time() + root = Path(__file__).resolve().parents[1] + model_path = str(root / cfg["model"]) if (root / cfg["model"]).exists() else cfg["model"] + adapter = cfg.get("adapter") + adapter = str(root / adapter) if adapter and (root / adapter).exists() else adapter + model, tok = load_model(model_path, cfg.get("revision"), cfg.get("dtype", "bfloat16"), adapter) + pb = PromptBuilder(tok, lang=cfg["lang"], chat=cfg["chat"]) + scorer = Scorer(model, tok, pb.add_special_tokens, cfg.get("token_budget", 32768), cfg.get("max_batch", 64)) + if cfg.get("fast_without_adapter") and hasattr(model, "disable_adapter"): + with model.disable_adapter(): # fast one-step answer from the plain base model (used by the agreement gate) + records, _ = score_items(items, pb, scorer, cfg["method"], cfg["n_perms"], cfg["perm_mode"], cfg["seed"]) + else: + records, _ = score_items(items, pb, scorer, cfg["method"], cfg["n_perms"], cfg["perm_mode"], cfg["seed"]) + agg = aggregate_items(items, records) + + # Thinking mode for selected levels (decided per question from its own level field) + think_levels = set(cfg.get("thinking_levels") or []) + think_uids = [u for u, it in items.items() if it["level"] in think_levels] + think_pred, fallbacks, sc_changed, sc_count = {}, 0, 0, 0 + perm_levels, k_perm = set(cfg.get("perm_think_levels") or []), int(cfg.get("perm_think_orders", 0)) + perm_uids = [u for u, it in items.items() if it["level"] in perm_levels] if k_perm > 0 else [] + perm_changed = 0 + if think_uids or perm_uids: + tpb = PromptBuilder(tok, lang=cfg["lang"], chat=cfg["chat"], chat_kwargs={"enable_thinking": True}) + tscorer = Scorer(model, tok, tpb.add_special_tokens, cfg.get("token_budget", 32768), cfg.get("max_batch", 64), + max_new_tokens=cfg.get("think_max_new_tokens", 2048)) + + def think_prompts(prompts: list[str], n_opts: list[int]) -> list[dict]: + if cfg.get("thinking_without_adapter") and hasattr(model, "disable_adapter"): + with model.disable_adapter(): # base model reasons; the adapter (trained to answer at once) does not + return tscorer.generate(prompts, n_opts) + return tscorer.generate(prompts, n_opts) + + def think(uids: list[str]) -> list[dict]: + return think_prompts([tpb.mcq(items[u], items[u]["options"]) for u in uids], + [len(items[u]["options"]) for u in uids]) + + for u, r in (zip(think_uids, think(think_uids)) if think_uids else []): + if r["pred"]: + think_pred[u] = r["pred"] + else: + fallbacks += 1 + print(f"thinking: {len(think_uids)} questions ({sorted(think_levels)}), {fallbacks} fell back to the fast answer") + all_ballots = {u: [think_pred.get(u)] for u in think_uids} # saved to .ballots.jsonl for cross-model pooling + + # Self-consistency for selected levels: greedy answer + K sampled reasoning paths, majority vote, + # ties go to the greedy answer. Fixed seed: reproducible for the same input, code and hardware. + sc_levels, k = set(cfg.get("sc_levels") or []), int(cfg.get("sc_samples", 0)) + sc_uids = [u for u in think_uids if items[u]["level"] in sc_levels] + if cfg.get("sc_gate") == "disagree": # only where the free one-step answer disagrees with greedy thinking + sc_uids = [u for u in sc_uids if think_pred.get(u) != agg[u]["pred"]] + sc_count = len(sc_uids) if k > 0 else 0 + if sc_uids and k > 0: + import torch + + torch.manual_seed(cfg["seed"]) + tscorer.sampling = {"do_sample": True, "temperature": cfg.get("sc_temperature", 0.8), + "top_p": cfg.get("sc_top_p", 0.95), "top_k": cfg.get("sc_top_k", 64)} + ballots = {u: [think_pred.get(u)] for u in sc_uids} + for _ in range(k): + for u, r in zip(sc_uids, think(sc_uids)): + ballots[u].append(r["pred"]) + all_ballots[u].append(r["pred"]) + tscorer.sampling = {} + for u, b in ballots.items(): + counts = Counter(x for x in b if x) + if not counts: + continue + top = max(counts.values()) + winners = [a for a, c in counts.items() if c == top] + choice = b[0] if b[0] in winners else min(winners) + sc_changed += choice != think_pred.get(u) + think_pred[u] = choice + print(f"self-consistency: {len(sc_uids)} questions ({sorted(sc_levels)}, gate={cfg.get('sc_gate')}), greedy + {k} samples, " + f"{sc_changed} answers changed by the vote") + + # Option-order self-consistency: re-ask with the options in K other (cyclic) orders, thinking each time; the vote is over + # the previous answer (the `prior_predictions` file if given, else the answer so far) and the K re-orderings, ties + # go to the previous answer. Votes are mapped back to the original option numbering. + if perm_uids: + prior = {} + if cfg.get("prior_predictions"): + with zipfile.ZipFile(root / cfg["prior_predictions"]) as z: + for line in z.read("predictions.jsonl").decode("utf-8").splitlines(): + r = json.loads(line) + prior[(r["level"], r["subject"], r["q_no"])] = int(r["answer"]) + ballots = {u: [prior.get((items[u]["level"], items[u]["subject_raw"], items[u]["q_no"]), think_pred.get(u))] + for u in perm_uids} + for kk in range(1, k_perm + 1): + perms = {u: make_perm(len(items[u]["options"]), kk, "cyclic", cfg["seed"], u) for u in perm_uids} + res = think_prompts([tpb.mcq(items[u], [items[u]["options"][j] for j in perms[u]]) for u in perm_uids], + [len(items[u]["options"]) for u in perm_uids]) + for u, r in zip(perm_uids, res): + ballots[u].append(perms[u][r["pred"] - 1] + 1 if r["pred"] else None) + print(f"option-order thinking: order {kk}/{k_perm} done ({len(perm_uids)} questions)", flush=True) + for u, b in ballots.items(): + counts = Counter(x for x in b if x) + if not counts: + continue + top = max(counts.values()) + winners = [a for a, c in counts.items() if c == top] + choice = b[0] if b[0] in winners else min(winners) + perm_changed += choice != b[0] + think_pred[u] = choice + print(f"option-order self-consistency: {len(perm_uids)} questions ({sorted(perm_levels)}), {k_perm} extra orders, " + f"{perm_changed} answers changed by the vote") + + rows = [] + for uid, it in items.items(): + n = len(it["options"]) + ans = think_pred.get(uid, agg[uid]["vote"]) + if ans is None or not 1 <= ans <= n: + raise SystemExit(f"no valid prediction for {uid}") + rows.append({"q_no": it["q_no"], "subject": it["subject_raw"], "category": it["category"], + "level": it["level"], "answer": str(ans)}) + + args.output.parent.mkdir(parents=True, exist_ok=True) + with zipfile.ZipFile(args.output, "w", zipfile.ZIP_DEFLATED) as z: + z.writestr("predictions.jsonl", "".join(json.dumps(r, ensure_ascii=False) + "\n" for r in rows)) + if think_uids: # greedy + sampled thinking answers per question (None = unparsed), first = greedy + Path(str(args.output) + ".ballots.jsonl").write_text("".join( + json.dumps({"level": items[u]["level"], "subject": items[u]["subject_raw"], "q_no": items[u]["q_no"], + "ballots": all_ballots[u]}, ensure_ascii=False) + "\n" for u in think_uids), encoding="utf-8") + digest = hashlib.sha256() + for f in sorted({str(f) for f, _ in raw}): + digest.update(Path(f).read_bytes()) + run = {"config": cfg, "n_questions": len(rows), "seconds": round(time.time() - t0, 1), + "thinking": {"levels": sorted(think_levels), "questions": len(think_uids), "fallbacks": fallbacks}, + "self_consistency": {"levels": sorted(set(cfg.get("sc_levels") or [])), "samples": int(cfg.get("sc_samples", 0)), + "gate": cfg.get("sc_gate"), "questions": sc_count, "answers_changed": sc_changed}, + "option_order_sc": {"levels": sorted(perm_levels), "orders": k_perm, "questions": len(perm_uids), + "answers_changed": perm_changed}, + "input_sha256": digest.hexdigest(), "params": loaded_param_stats(model), "env": env_fingerprint()} + Path(str(args.output) + ".run.json").write_text(json.dumps(run, indent=2, ensure_ascii=False)) + print(f"wrote {len(rows)} predictions -> {args.output} " + f"({run['params']['loaded_total'] / 1e9:.3f}B parameters loaded, {run['seconds']:.0f}s)") + + +if __name__ == "__main__": + main() diff --git a/training/reports/baselines_zeroshot_logit.md b/training/reports/baselines_zeroshot_logit.md new file mode 100644 index 0000000000000000000000000000000000000000..533eea9f6a2228fe3ffe3b3da935ad3a7c02175d --- /dev/null +++ b/training/reports/baselines_zeroshot_logit.md @@ -0,0 +1,88 @@ +# Baseline comparison + +Accuracy in %. `orig` = original option order; `vote` = mean probabilities over cyclic option permutations; `±` = std of accuracy across permutations (position sensitivity). Random = 25. + +## Overview + +| model | params loaded | tok/char | SinhalaMMLU dev orig | vote | ± | macro-subj | Global-MMLU si | Belebele si | label mass | consistency | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| Llama-3.2-3B-Instruct_logit | 3.21B | 1.02 | 28.3 | 29.9 | 2.6 | 28.2 | 24.7 | 29.1 | 1.00 | 0.00 | +| Phi-4-mini-instruct_logit | 3.84B | 0.42 | 31.5 | 29.1 | 2.4 | 31.4 | 28.6 | 39.9 | 0.98 | 0.07 | +| Qwen3-4B-Instruct-2507_logit | 4.02B | 0.85 | 34.6 | 37.2 | 0.8 | 34.5 | 37.5 | 56.0 | 1.00 | 0.21 | +| Qwen3-4B_logit | 4.02B | 0.85 | 34.3 | 35.1 | 1.5 | 34.2 | 31.8 | 50.4 | 1.00 | 0.15 | +| Qwen3.5-4B_logit | 4.21B | 0.53 | 40.6 | 47.6 | 1.5 | 40.5 | 44.2 | 65.7 | 1.00 | 0.19 | +| SmolLM3-3B_logit | 3.08B | 1.02 | 28.9 | 29.4 | 2.2 | 29.0 | 26.9 | 29.8 | 0.62 | 0.01 | +| gemma-3-4b-it_logit | 4.30B | 0.40 | 39.0 | 40.2 | 1.3 | 39.1 | 35.4 | 55.7 | 1.00 | 0.19 | +| gemma-3n-E4B-it_logit | 7.82B | 0.40 | 50.3 | 51.7 | 1.5 | 50.5 | 46.2 | 73.8 | 1.00 | 0.39 | +| gemma-4-E4B-it_logit | 7.94B | 0.40 | 55.2 | 54.5 | 1.1 | 55.4 | 52.0 | 79.8 | 0.99 | 0.49 | +| granite-4.2-3b_logit | 3.66B | 1.04 | 29.1 | 29.2 | 1.7 | 29.4 | 27.4 | 30.6 | 0.99 | 0.04 | + +## SinhalaMMLU dev: per subject + +| subject | n | Llama-3.2-3B-Instruct_logit | Phi-4-mini-instruct_logit | Qwen3-4B-Instruct-2507_logit | Qwen3-4B_logit | Qwen3.5-4B_logit | SmolLM3-3B_logit | gemma-3-4b-it_logit | gemma-3n-E4B-it_logit | gemma-4-E4B-it_logit | granite-4.2-3b_logit | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| Arts | 100 | 26.0 | 27.0 | 27.0 | 27.0 | 33.0 | 28.0 | 38.0 | 49.0 | 56.0 | 30.0 | +| Buddhism | 162 | 27.2 | 34.0 | 30.2 | 32.7 | 29.6 | 22.8 | 29.6 | 46.3 | 51.9 | 27.8 | +| Catholicism | 132 | 25.0 | 28.0 | 27.3 | 31.1 | 31.8 | 23.5 | 42.4 | 47.0 | 50.0 | 31.8 | +| Christianity | 130 | 26.2 | 32.3 | 30.8 | 30.0 | 47.7 | 28.5 | 40.0 | 46.9 | 62.3 | 26.2 | +| Civics | 123 | 34.1 | 34.1 | 47.2 | 43.1 | 61.8 | 36.6 | 56.1 | 66.7 | 74.8 | 30.1 | +| Eastern music | 143 | 30.8 | 33.6 | 35.7 | 32.9 | 26.6 | 32.2 | 38.5 | 49.7 | 45.5 | 22.4 | +| Geography | 116 | 26.7 | 29.3 | 24.1 | 44.0 | 37.1 | 30.2 | 37.1 | 55.2 | 64.7 | 34.5 | +| Health and Physical Science | 130 | 23.8 | 39.2 | 50.0 | 46.9 | 53.1 | 30.0 | 48.5 | 61.5 | 69.2 | 34.6 | +| History | 151 | 22.5 | 25.2 | 27.8 | 31.8 | 35.1 | 27.2 | 29.1 | 34.4 | 39.1 | 29.1 | +| Islam | 108 | 25.0 | 28.7 | 34.3 | 34.3 | 41.7 | 25.0 | 37.0 | 52.8 | 59.3 | 34.3 | +| Sinhala language and literature | 155 | 30.3 | 28.4 | 35.5 | 29.7 | 37.4 | 29.7 | 33.5 | 43.2 | 46.5 | 23.2 | +| dancing | 108 | 30.6 | 32.4 | 34.3 | 25.9 | 29.6 | 31.5 | 31.5 | 40.7 | 34.3 | 26.9 | +| drama and Theatre | 128 | 34.4 | 32.0 | 40.6 | 33.6 | 44.5 | 32.0 | 41.4 | 53.1 | 55.5 | 33.6 | +| science | 164 | 32.3 | 34.8 | 38.4 | 36.6 | 58.5 | 29.3 | 45.1 | 60.4 | 66.5 | 27.4 | + +## SinhalaMMLU dev: per domain + +| category | n | Llama-3.2-3B-Instruct_logit | Phi-4-mini-instruct_logit | Qwen3-4B-Instruct-2507_logit | Qwen3-4B_logit | Qwen3.5-4B_logit | SmolLM3-3B_logit | gemma-3-4b-it_logit | gemma-3n-E4B-it_logit | gemma-4-E4B-it_logit | granite-4.2-3b_logit | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| humanities | 1162 | 27.5 | 30.5 | 31.9 | 31.2 | 35.3 | 27.7 | 36.1 | 46.4 | 50.2 | 28.9 | +| language | 155 | 30.3 | 28.4 | 35.5 | 29.7 | 37.4 | 29.7 | 33.5 | 43.2 | 46.5 | 23.2 | +| social_science | 369 | 28.2 | 34.4 | 40.9 | 44.7 | 50.9 | 32.2 | 47.4 | 61.2 | 69.6 | 33.1 | +| stem | 164 | 32.3 | 34.8 | 38.4 | 36.6 | 58.5 | 29.3 | 45.1 | 60.4 | 66.5 | 27.4 | + +## SinhalaMMLU dev: per grade + +| grade | n | Llama-3.2-3B-Instruct_logit | Phi-4-mini-instruct_logit | Qwen3-4B-Instruct-2507_logit | Qwen3-4B_logit | Qwen3.5-4B_logit | SmolLM3-3B_logit | gemma-3-4b-it_logit | gemma-3n-E4B-it_logit | gemma-4-E4B-it_logit | granite-4.2-3b_logit | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| 6 | 584 | 29.6 | 31.8 | 37.3 | 37.7 | 44.3 | 31.7 | 43.7 | 52.9 | 59.9 | 29.1 | +| 7 | 634 | 29.3 | 32.6 | 32.8 | 33.6 | 39.4 | 28.1 | 37.7 | 49.7 | 53.0 | 29.7 | +| 8 | 632 | 25.9 | 29.9 | 33.9 | 31.8 | 38.4 | 27.2 | 35.9 | 48.6 | 53.0 | 28.6 | + +## Global-MMLU si: per category + +| category | n | Llama-3.2-3B-Instruct_logit | Phi-4-mini-instruct_logit | Qwen3-4B-Instruct-2507_logit | Qwen3-4B_logit | Qwen3.5-4B_logit | SmolLM3-3B_logit | gemma-3-4b-it_logit | gemma-3n-E4B-it_logit | gemma-4-E4B-it_logit | granite-4.2-3b_logit | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| Business | 102 | 27.5 | 32.4 | 44.1 | 37.3 | 37.3 | 25.5 | 43.1 | 52.9 | 54.9 | 30.4 | +| Humanities | 668 | 25.0 | 28.3 | 33.8 | 31.1 | 37.3 | 28.9 | 36.1 | 42.8 | 46.9 | 27.8 | +| Medical | 216 | 27.3 | 29.2 | 38.0 | 25.5 | 50.0 | 26.9 | 31.5 | 48.6 | 54.2 | 35.2 | +| Other | 126 | 27.0 | 31.7 | 34.1 | 31.7 | 43.7 | 26.2 | 42.1 | 55.6 | 56.3 | 24.6 | +| STEM | 449 | 24.1 | 27.6 | 41.0 | 34.7 | 45.7 | 26.1 | 33.2 | 41.9 | 51.9 | 29.4 | +| Social Sciences | 439 | 22.3 | 28.2 | 38.7 | 31.4 | 52.4 | 25.3 | 35.1 | 50.6 | 57.2 | 21.0 | + +## Global-MMLU si: culturally sensitive (CS) / agnostic (CA) / unlabelled (-) + +| cultural_sensitivity | n | Llama-3.2-3B-Instruct_logit | Phi-4-mini-instruct_logit | Qwen3-4B-Instruct-2507_logit | Qwen3-4B_logit | Qwen3.5-4B_logit | SmolLM3-3B_logit | gemma-3-4b-it_logit | gemma-3n-E4B-it_logit | gemma-4-E4B-it_logit | granite-4.2-3b_logit | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| - | 1603 | 23.8 | 28.3 | 37.1 | 31.2 | 43.8 | 26.6 | 34.8 | 45.7 | 51.2 | 27.0 | +| CA | 277 | 28.9 | 28.5 | 41.9 | 34.7 | 47.7 | 25.6 | 38.3 | 46.6 | 57.8 | 29.2 | +| CS | 120 | 27.5 | 33.3 | 32.5 | 32.5 | 42.5 | 33.3 | 37.5 | 52.5 | 50.0 | 28.3 | + +## Position bias on SinhalaMMLU dev (share of predictions per displayed position) + +| model | 1 | 2 | 3 | 4 | +|---|---:|---:|---:|---:| +| Llama-3.2-3B-Instruct_logit | 0.2 | 4.5 | 93.0 | 2.2 | +| Phi-4-mini-instruct_logit | 35.2 | 37.2 | 23.6 | 4.0 | +| Qwen3-4B-Instruct-2507_logit | 11.8 | 16.1 | 29.7 | 42.4 | +| Qwen3-4B_logit | 29.5 | 41.9 | 14.3 | 14.4 | +| Qwen3.5-4B_logit | 11.0 | 7.9 | 27.7 | 53.4 | +| SmolLM3-3B_logit | 19.6 | 0.5 | 76.0 | 3.9 | +| gemma-3-4b-it_logit | 21.9 | 32.6 | 35.8 | 9.7 | +| gemma-3n-E4B-it_logit | 14.0 | 29.6 | 32.7 | 23.7 | +| gemma-4-E4B-it_logit | 22.4 | 31.0 | 26.3 | 20.4 | +| granite-4.2-3b_logit | 69.2 | 13.6 | 12.4 | 4.8 | diff --git a/training/reports/corpus_stats.json b/training/reports/corpus_stats.json new file mode 100644 index 0000000000000000000000000000000000000000..7d627ae5572767a356bbe73247b0ddf336e46bdf --- /dev/null +++ b/training/reports/corpus_stats.json @@ -0,0 +1,66 @@ +{ + "revisions": { + "fineweb2": "HuggingFaceFW/fineweb-2@af9c13333eb981300149d5ca60a8e9d659b276b9", + "wikipedia": "wikimedia/wikipedia@b04c8d1ceb2f5cd4588862100d08de323dccfbaa", + "nsina": "sinhala-nlp/NSINA@a986af81ad1959701aaac7ee13f756845928e672" + }, + "sources": { + "fineweb2": { + "docs_seen": 1185323, + "reasons": { + "kept": 1169835, + "exam_like": 3779, + "not_sinhala": 7561, + "blocked_domain": 4100, + "contaminated": 27, + "duplicate": 21 + }, + "kept_chars": 2842104750, + "est_gemma_tokens": 1136841900, + "sl_score_quantiles": { + "50": 0.0, + "90": 2.687600000000093, + "99": 7.034 + } + }, + "wikipedia": { + "docs_seen": 23065, + "reasons": { + "kept": 16736, + "too_short": 4921, + "exam_like": 161, + "not_sinhala": 796, + "duplicate": 447, + "contaminated": 3, + "repetitive": 1 + }, + "kept_chars": 47946427, + "est_gemma_tokens": 19178570, + "sl_score_quantiles": { + "50": 1.8105, + "90": 9.329, + "99": 19.66290000000004 + } + }, + "nsina": { + "docs_seen": 486932, + "reasons": { + "kept": 452431, + "not_sinhala": 9313, + "duplicate": 13954, + "exam_like": 908, + "too_short": 10320, + "repetitive": 6 + }, + "kept_chars": 567614929, + "est_gemma_tokens": 227045971, + "sl_score_quantiles": { + "50": 0.0, + "90": 4.215, + "99": 9.658 + } + } + }, + "total_est_gemma_tokens": 1383066441, + "seconds": 163 +} \ No newline at end of file diff --git a/training/reports/cpt_v1_stats.json b/training/reports/cpt_v1_stats.json new file mode 100644 index 0000000000000000000000000000000000000000..6cd2afe51bef9f923c2f9daf11870e85bf5aef42 --- /dev/null +++ b/training/reports/cpt_v1_stats.json @@ -0,0 +1,45 @@ +{ + "target_tokens": 300000000.0, + "near_duplicates_removed": { + "fineweb2": 1027, + "nsina": 2222, + "wikipedia": 216 + }, + "english_revision": "HuggingFaceFW/fineweb-edu@87f09149ef4734204d70ed1d046ddc9ca3f2b8f9", + "parts": { + "wikipedia": { + "docs": 16520, + "chars": 47553705, + "est_gemma_tokens": 19021482, + "picks": { + "all": 16520 + } + }, + "fineweb2": { + "docs": 207428, + "chars": 477047721, + "est_gemma_tokens": 190819088, + "picks": { + "top_sl_score": 148871, + "random": 58557 + } + }, + "nsina": { + "docs": 123278, + "chars": 149999535, + "est_gemma_tokens": 59999814, + "picks": { + "random": 123278 + } + }, + "english": { + "docs": 25433, + "chars": 119994844, + "est_gemma_tokens": 29998711, + "picks": { + "random": 25433 + } + } + }, + "total_est_gemma_tokens": 299839095 +} \ No newline at end of file diff --git a/training/reports/data_stats_sft_gmmlu.json b/training/reports/data_stats_sft_gmmlu.json new file mode 100644 index 0000000000000000000000000000000000000000..8168e677839b45a1d6e0a0d35e96a7a5f58ab27d --- /dev/null +++ b/training/reports/data_stats_sft_gmmlu.json @@ -0,0 +1,32 @@ +{ + "pool": 12327, + "kept": 11447, + "dropped": { + "near_validation": 819, + "duplicate_within_pool": 61 + }, + "thresholds": { + "validation": 0.6, + "sinhalammlu_dev": 0.5, + "within_pool": 0.95 + }, + "max_similarity_kept": { + "to_validation": 0.5999996441576304, + "to_sinhalammlu_dev": 0.4235920760708065 + }, + "answer_positions": { + "1": 2831, + "2": 2824, + "3": 2847, + "4": 2945 + }, + "shuffled": 10733, + "by_category": { + "Humanities": 3351, + "STEM": 2760, + "Social Sciences": 2676, + "Medical": 1259, + "Other": 764, + "Business": 637 + } +} \ No newline at end of file diff --git a/training/reports/experiments_abc.md b/training/reports/experiments_abc.md new file mode 100644 index 0000000000000000000000000000000000000000..8a6edbc0cd080fd612d8682de6fa63259af45b18 --- /dev/null +++ b/training/reports/experiments_abc.md @@ -0,0 +1,78 @@ +# Baseline comparison + +Accuracy in %. `orig` = original option order; `vote` = mean probabilities over cyclic option permutations; `±` = std of accuracy across permutations (position sensitivity). Random = 25. + +## Overview + +| model | params loaded | tok/char | SinhalaMMLU dev orig | vote | ± | macro-subj | Global-MMLU si | Belebele si | label mass | consistency | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| gemma-4-E4B-it_logit | 7.94B | 0.40 | 55.2 | 54.5 | 1.1 | 55.4 | 52.0 | 79.8 | 0.99 | 0.49 | +| cpt_pilot_step-114_logit | 7.46B | 0.40 | 54.1 | 54.9 | 1.1 | 54.1 | 51.9 | - | 1.00 | 0.46 | +| cpt_pilot_step-228_logit | 7.46B | 0.40 | 53.9 | 55.5 | 1.5 | 54.1 | 51.7 | - | 1.00 | 0.45 | +| devcv_lora_all | 7.46B | 0.40 | 56.3 | 57.4 | 0.7 | 56.4 | - | - | 1.00 | 0.54 | +| sft_gmmlu_lora_logit | 7.46B | 0.40 | 56.2 | 57.2 | 0.5 | 56.2 | 58.1 | - | 1.00 | 0.53 | + +## SinhalaMMLU dev: per subject + +| subject | n | gemma-4-E4B-it_logit | cpt_pilot_step-114_logit | cpt_pilot_step-228_logit | devcv_lora_all | sft_gmmlu_lora_logit | +|---|---:|---:|---:|---:|---:|---:| +| Arts | 100 | 56.0 | 55.0 | 53.0 | 54.0 | 52.0 | +| Buddhism | 162 | 51.9 | 50.0 | 47.5 | 54.3 | 53.7 | +| Catholicism | 132 | 50.0 | 51.5 | 52.3 | 59.8 | 53.0 | +| Christianity | 130 | 62.3 | 61.5 | 61.5 | 60.0 | 63.1 | +| Civics | 123 | 74.8 | 76.4 | 75.6 | 73.2 | 74.8 | +| Eastern music | 143 | 45.5 | 47.6 | 47.6 | 43.4 | 44.1 | +| Geography | 116 | 64.7 | 62.1 | 62.9 | 64.7 | 62.1 | +| Health and Physical Science | 130 | 69.2 | 62.3 | 63.1 | 69.2 | 70.8 | +| History | 151 | 39.1 | 41.7 | 39.7 | 38.4 | 39.1 | +| Islam | 108 | 59.3 | 60.2 | 61.1 | 62.0 | 63.0 | +| Sinhala language and literature | 155 | 46.5 | 45.2 | 45.8 | 47.7 | 52.9 | +| dancing | 108 | 34.3 | 27.8 | 29.6 | 35.2 | 33.3 | +| drama and Theatre | 128 | 55.5 | 50.8 | 50.8 | 60.2 | 58.6 | +| science | 164 | 66.5 | 65.9 | 66.5 | 67.7 | 67.1 | + +## SinhalaMMLU dev: per domain + +| category | n | gemma-4-E4B-it_logit | cpt_pilot_step-114_logit | cpt_pilot_step-228_logit | devcv_lora_all | sft_gmmlu_lora_logit | +|---|---:|---:|---:|---:|---:|---:| +| humanities | 1162 | 50.2 | 49.5 | 49.1 | 51.7 | 50.9 | +| language | 155 | 46.5 | 45.2 | 45.8 | 47.7 | 52.9 | +| social_science | 369 | 69.6 | 66.9 | 67.2 | 69.1 | 69.4 | +| stem | 164 | 66.5 | 65.9 | 66.5 | 67.7 | 67.1 | + +## SinhalaMMLU dev: per grade + +| grade | n | gemma-4-E4B-it_logit | cpt_pilot_step-114_logit | cpt_pilot_step-228_logit | devcv_lora_all | sft_gmmlu_lora_logit | +|---|---:|---:|---:|---:|---:|---:| +| 6 | 584 | 59.9 | 59.4 | 59.9 | 62.5 | 61.0 | +| 7 | 634 | 53.0 | 50.5 | 51.3 | 53.0 | 53.2 | +| 8 | 632 | 53.0 | 52.7 | 51.1 | 53.8 | 54.9 | + +## Global-MMLU si: per category + +| category | n | gemma-4-E4B-it_logit | cpt_pilot_step-114_logit | cpt_pilot_step-228_logit | devcv_lora_all | sft_gmmlu_lora_logit | +|---|---:|---:|---:|---:|---:|---:| +| Business | 102 | 54.9 | 61.8 | 58.8 | - | 61.8 | +| Humanities | 668 | 46.9 | 45.7 | 45.5 | - | 49.9 | +| Medical | 216 | 54.2 | 54.2 | 52.8 | - | 63.4 | +| Other | 126 | 56.3 | 54.0 | 51.6 | - | 56.3 | +| STEM | 449 | 51.9 | 53.0 | 53.7 | - | 59.7 | +| Social Sciences | 439 | 57.2 | 56.5 | 56.9 | - | 66.3 | + +## Global-MMLU si: culturally sensitive (CS) / agnostic (CA) / unlabelled (-) + +| cultural_sensitivity | n | gemma-4-E4B-it_logit | cpt_pilot_step-114_logit | cpt_pilot_step-228_logit | devcv_lora_all | sft_gmmlu_lora_logit | +|---|---:|---:|---:|---:|---:|---:| +| - | 1603 | 51.2 | 51.3 | 51.3 | - | 57.7 | +| CA | 277 | 57.8 | 56.7 | 56.0 | - | 61.4 | +| CS | 120 | 50.0 | 49.2 | 46.7 | - | 56.7 | + +## Position bias on SinhalaMMLU dev (share of predictions per displayed position) + +| model | 1 | 2 | 3 | 4 | +|---|---:|---:|---:|---:| +| gemma-4-E4B-it_logit | 22.4 | 31.0 | 26.3 | 20.4 | +| cpt_pilot_step-114_logit | 21.3 | 33.5 | 25.2 | 20.0 | +| cpt_pilot_step-228_logit | 21.1 | 33.8 | 25.3 | 19.8 | +| devcv_lora_all | 28.4 | 24.3 | 23.6 | 23.6 | +| sft_gmmlu_lora_logit | 29.5 | 23.8 | 26.1 | 20.6 | diff --git a/training/reports/experiments_prompting.md b/training/reports/experiments_prompting.md new file mode 100644 index 0000000000000000000000000000000000000000..03f4722a0f30a70cce079dc46fb7c72d1bfaad06 --- /dev/null +++ b/training/reports/experiments_prompting.md @@ -0,0 +1,96 @@ +# Baseline comparison + +Accuracy in %. `orig` = original option order; `vote` = mean probabilities over cyclic option permutations; `±` = std of accuracy across permutations (position sensitivity). Random = 25. + +## Overview + +| model | params loaded | tok/char | SinhalaMMLU dev orig | vote | ± | macro-subj | Global-MMLU si | Belebele si | label mass | consistency | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| Qwen3.5-4B_logit | 4.21B | 0.53 | 40.6 | 47.6 | 1.5 | 40.5 | 44.2 | 65.7 | 1.00 | 0.19 | +| gemma-3n-E4B-it_logit | 7.82B | 0.40 | 50.3 | 51.7 | 1.5 | 50.5 | 46.2 | 73.8 | 1.00 | 0.39 | +| gemma-4-E4B-it_logit | 7.94B | 0.40 | 55.2 | 54.5 | 1.1 | 55.4 | 52.0 | 79.8 | 0.99 | 0.49 | +| Qwen3.5-4B_generate | 4.21B | 0.53 | 40.6 | 40.6 | 0.0 | 40.4 | 44.2 | - | nan | 1.00 | +| Qwen3.5-4B_logit_5shot | 4.21B | 0.53 | 46.6 | 46.6 | 0.0 | 46.3 | 45.4 | - | 1.00 | 1.00 | +| Qwen3.5-4B_logit_si | 4.21B | 0.79 | 43.8 | 45.5 | 1.6 | 43.7 | 42.8 | - | 0.94 | 0.19 | +| gemma-3n-E4B-it_cloze | 7.82B | 0.40 | 32.1 | 32.1 | 0.0 | 32.2 | 30.6 | - | nan | 1.00 | +| gemma-3n-E4B-it_generate | 7.82B | 0.40 | 50.3 | 50.3 | 0.0 | 50.4 | 46.2 | - | nan | 1.00 | +| gemma-3n-E4B-it_logit_5shot | 7.82B | 0.40 | 50.1 | 50.1 | 0.0 | 50.2 | 46.0 | - | 1.00 | 1.00 | +| gemma-3n-E4B-it_logit_si | 7.82B | 0.53 | 50.3 | 51.9 | 1.1 | 50.5 | 46.2 | - | 0.99 | 0.37 | +| gemma-4-E4B-it_cloze | 7.94B | 0.40 | 26.5 | 26.5 | 0.0 | 26.7 | 24.9 | - | nan | 1.00 | +| gemma-4-E4B-it_generate | 7.94B | 0.40 | 55.1 | 55.1 | 0.0 | 55.2 | 52.3 | - | nan | 1.00 | +| gemma-4-E4B-it_logit_5shot | 7.94B | 0.40 | 55.1 | 55.1 | 0.0 | 55.2 | 51.7 | - | 1.00 | 1.00 | +| gemma-4-E4B-it_logit_si | 7.94B | 0.53 | 55.5 | 55.4 | 1.4 | 55.6 | 52.1 | - | 0.96 | 0.46 | + +## SinhalaMMLU dev: per subject + +| subject | n | Qwen3.5-4B_logit | gemma-3n-E4B-it_logit | gemma-4-E4B-it_logit | Qwen3.5-4B_generate | Qwen3.5-4B_logit_5shot | Qwen3.5-4B_logit_si | gemma-3n-E4B-it_cloze | gemma-3n-E4B-it_generate | gemma-3n-E4B-it_logit_5shot | gemma-3n-E4B-it_logit_si | gemma-4-E4B-it_cloze | gemma-4-E4B-it_generate | gemma-4-E4B-it_logit_5shot | gemma-4-E4B-it_logit_si | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| Arts | 100 | 33.0 | 49.0 | 56.0 | 31.0 | 36.0 | 43.0 | 36.0 | 48.0 | 48.0 | 50.0 | 26.0 | 55.0 | 55.0 | 58.0 | +| Buddhism | 162 | 29.6 | 46.3 | 51.9 | 31.5 | 40.6 | 34.6 | 29.0 | 46.3 | 50.6 | 44.4 | 28.4 | 51.9 | 53.8 | 51.9 | +| Catholicism | 132 | 31.8 | 47.0 | 50.0 | 31.8 | 38.6 | 38.6 | 31.8 | 48.5 | 43.2 | 47.7 | 23.5 | 48.5 | 47.7 | 51.5 | +| Christianity | 130 | 47.7 | 46.9 | 62.3 | 46.2 | 41.5 | 46.2 | 34.6 | 46.2 | 50.0 | 50.8 | 29.2 | 63.1 | 62.3 | 60.8 | +| Civics | 123 | 61.8 | 66.7 | 74.8 | 61.8 | 71.5 | 60.2 | 32.5 | 67.5 | 65.9 | 67.5 | 27.6 | 74.8 | 74.8 | 72.4 | +| Eastern music | 143 | 26.6 | 49.7 | 45.5 | 26.6 | 43.4 | 38.5 | 30.1 | 49.0 | 42.7 | 41.3 | 25.9 | 46.9 | 46.2 | 47.6 | +| Geography | 116 | 37.1 | 55.2 | 64.7 | 37.1 | 44.8 | 39.7 | 30.2 | 55.2 | 50.9 | 54.3 | 31.0 | 63.8 | 66.4 | 67.2 | +| Health and Physical Science | 130 | 53.1 | 61.5 | 69.2 | 53.1 | 55.8 | 59.2 | 40.8 | 61.5 | 60.5 | 66.2 | 32.3 | 69.2 | 69.0 | 67.7 | +| History | 151 | 35.1 | 34.4 | 39.1 | 34.4 | 33.1 | 34.4 | 31.8 | 34.4 | 39.7 | 39.1 | 26.5 | 39.1 | 36.4 | 39.7 | +| Islam | 108 | 41.7 | 52.8 | 59.3 | 41.7 | 43.5 | 38.9 | 37.0 | 52.8 | 61.1 | 54.6 | 25.9 | 59.3 | 53.7 | 56.5 | +| Sinhala language and literature | 155 | 37.4 | 43.2 | 46.5 | 38.7 | 40.9 | 35.5 | 33.5 | 43.2 | 42.9 | 43.9 | 27.1 | 45.2 | 50.6 | 47.1 | +| dancing | 108 | 29.6 | 40.7 | 34.3 | 29.6 | 34.3 | 32.4 | 25.0 | 39.8 | 36.1 | 38.9 | 27.8 | 33.3 | 37.0 | 30.6 | +| drama and Theatre | 128 | 44.5 | 53.1 | 55.5 | 44.5 | 53.9 | 46.9 | 27.3 | 51.6 | 54.7 | 49.2 | 22.7 | 56.2 | 53.9 | 59.4 | +| science | 164 | 58.5 | 60.4 | 66.5 | 57.9 | 69.9 | 64.0 | 31.1 | 61.6 | 57.1 | 59.1 | 19.5 | 67.1 | 65.6 | 67.7 | + +## SinhalaMMLU dev: per domain + +| category | n | Qwen3.5-4B_logit | gemma-3n-E4B-it_logit | gemma-4-E4B-it_logit | Qwen3.5-4B_generate | Qwen3.5-4B_logit_5shot | Qwen3.5-4B_logit_si | gemma-3n-E4B-it_cloze | gemma-3n-E4B-it_generate | gemma-3n-E4B-it_logit_5shot | gemma-3n-E4B-it_logit_si | gemma-4-E4B-it_cloze | gemma-4-E4B-it_generate | gemma-4-E4B-it_logit_5shot | gemma-4-E4B-it_logit_si | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| humanities | 1162 | 35.3 | 46.4 | 50.2 | 35.1 | 40.6 | 39.1 | 31.2 | 46.0 | 47.2 | 45.9 | 26.2 | 50.2 | 49.4 | 50.5 | +| language | 155 | 37.4 | 43.2 | 46.5 | 38.7 | 40.9 | 35.5 | 33.5 | 43.2 | 42.9 | 43.9 | 27.1 | 45.2 | 50.6 | 47.1 | +| social_science | 369 | 50.9 | 61.2 | 69.6 | 50.9 | 57.6 | 53.4 | 34.7 | 61.5 | 59.2 | 62.9 | 30.4 | 69.4 | 70.1 | 69.1 | +| stem | 164 | 58.5 | 60.4 | 66.5 | 57.9 | 69.9 | 64.0 | 31.1 | 61.6 | 57.1 | 59.1 | 19.5 | 67.1 | 65.6 | 67.7 | + +## SinhalaMMLU dev: per grade + +| grade | n | Qwen3.5-4B_logit | gemma-3n-E4B-it_logit | gemma-4-E4B-it_logit | Qwen3.5-4B_generate | Qwen3.5-4B_logit_5shot | Qwen3.5-4B_logit_si | gemma-3n-E4B-it_cloze | gemma-3n-E4B-it_generate | gemma-3n-E4B-it_logit_5shot | gemma-3n-E4B-it_logit_si | gemma-4-E4B-it_cloze | gemma-4-E4B-it_generate | gemma-4-E4B-it_logit_5shot | gemma-4-E4B-it_logit_si | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| 6 | 584 | 44.3 | 52.9 | 59.9 | 44.3 | 48.4 | 46.9 | 36.5 | 52.1 | 53.5 | 52.2 | 26.0 | 59.9 | 60.7 | 61.6 | +| 7 | 634 | 39.4 | 49.7 | 53.0 | 39.6 | 47.2 | 42.6 | 29.2 | 50.2 | 48.8 | 50.2 | 25.4 | 52.8 | 51.3 | 52.8 | +| 8 | 632 | 38.4 | 48.6 | 53.0 | 38.1 | 44.4 | 42.2 | 31.0 | 48.7 | 48.2 | 48.6 | 28.2 | 52.8 | 53.6 | 52.4 | + +## Global-MMLU si: per category + +| category | n | Qwen3.5-4B_logit | gemma-3n-E4B-it_logit | gemma-4-E4B-it_logit | Qwen3.5-4B_generate | Qwen3.5-4B_logit_5shot | Qwen3.5-4B_logit_si | gemma-3n-E4B-it_cloze | gemma-3n-E4B-it_generate | gemma-3n-E4B-it_logit_5shot | gemma-3n-E4B-it_logit_si | gemma-4-E4B-it_cloze | gemma-4-E4B-it_generate | gemma-4-E4B-it_logit_5shot | gemma-4-E4B-it_logit_si | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| Business | 102 | 37.3 | 52.9 | 54.9 | 40.2 | 43.1 | 42.2 | 27.5 | 53.9 | 56.9 | 51.0 | 36.3 | 55.9 | 55.9 | 53.9 | +| Humanities | 668 | 37.3 | 42.8 | 46.9 | 37.0 | 37.6 | 37.3 | 30.1 | 42.4 | 42.7 | 45.1 | 26.9 | 47.0 | 46.1 | 46.9 | +| Medical | 216 | 50.0 | 48.6 | 54.2 | 50.5 | 47.7 | 44.0 | 27.8 | 49.1 | 47.7 | 49.5 | 23.1 | 54.6 | 54.2 | 56.0 | +| Other | 126 | 43.7 | 55.6 | 56.3 | 42.9 | 45.2 | 47.6 | 42.9 | 56.3 | 56.3 | 53.2 | 22.2 | 56.3 | 55.6 | 57.9 | +| STEM | 449 | 45.7 | 41.9 | 51.9 | 45.4 | 48.1 | 41.9 | 33.9 | 41.9 | 42.5 | 39.2 | 22.3 | 51.7 | 53.7 | 50.6 | +| Social Sciences | 439 | 52.4 | 50.6 | 57.2 | 52.4 | 54.0 | 50.3 | 26.7 | 50.1 | 48.3 | 50.3 | 23.5 | 57.9 | 55.1 | 57.6 | + +## Global-MMLU si: culturally sensitive (CS) / agnostic (CA) / unlabelled (-) + +| cultural_sensitivity | n | Qwen3.5-4B_logit | gemma-3n-E4B-it_logit | gemma-4-E4B-it_logit | Qwen3.5-4B_generate | Qwen3.5-4B_logit_5shot | Qwen3.5-4B_logit_si | gemma-3n-E4B-it_cloze | gemma-3n-E4B-it_generate | gemma-3n-E4B-it_logit_5shot | gemma-3n-E4B-it_logit_si | gemma-4-E4B-it_cloze | gemma-4-E4B-it_generate | gemma-4-E4B-it_logit_5shot | gemma-4-E4B-it_logit_si | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| - | 1603 | 43.8 | 45.7 | 51.2 | 43.7 | 45.0 | 41.9 | 30.6 | 45.7 | 45.3 | 46.7 | 25.5 | 51.3 | 51.3 | 50.3 | +| CA | 277 | 47.7 | 46.6 | 57.8 | 47.7 | 47.7 | 47.3 | 28.2 | 46.2 | 48.7 | 41.9 | 21.3 | 59.2 | 54.9 | 59.2 | +| CS | 120 | 42.5 | 52.5 | 50.0 | 43.3 | 45.0 | 44.2 | 35.8 | 52.5 | 49.2 | 50.0 | 25.0 | 50.0 | 50.8 | 60.0 | + +## Position bias on SinhalaMMLU dev (share of predictions per displayed position) + +| model | 1 | 2 | 3 | 4 | +|---|---:|---:|---:|---:| +| Qwen3.5-4B_logit | 11.0 | 7.9 | 27.7 | 53.4 | +| gemma-3n-E4B-it_logit | 14.0 | 29.6 | 32.7 | 23.7 | +| gemma-4-E4B-it_logit | 22.4 | 31.0 | 26.3 | 20.4 | +| Qwen3.5-4B_generate | 10.5 | 8.8 | 29.2 | 51.4 | +| Qwen3.5-4B_logit_5shot | 11.2 | 22.0 | 38.0 | 28.8 | +| Qwen3.5-4B_logit_si | 9.0 | 18.9 | 47.3 | 24.7 | +| gemma-3n-E4B-it_cloze | 27.9 | 25.1 | 23.2 | 23.7 | +| gemma-3n-E4B-it_generate | 13.6 | 31.4 | 34.5 | 20.5 | +| gemma-3n-E4B-it_logit_5shot | 22.0 | 30.8 | 32.8 | 14.4 | +| gemma-3n-E4B-it_logit_si | 14.3 | 30.9 | 35.1 | 19.7 | +| gemma-4-E4B-it_cloze | 25.5 | 24.7 | 23.8 | 26.0 | +| gemma-4-E4B-it_generate | 22.6 | 32.9 | 26.8 | 17.6 | +| gemma-4-E4B-it_logit_5shot | 22.8 | 30.7 | 29.0 | 17.6 | +| gemma-4-E4B-it_logit_si | 21.4 | 33.4 | 26.1 | 19.1 | diff --git a/training/reports/param_counts.json b/training/reports/param_counts.json new file mode 100644 index 0000000000000000000000000000000000000000..46fb6ee20de96d2ac6d96f53e7edfee85d76edbf --- /dev/null +++ b/training/reports/param_counts.json @@ -0,0 +1,387 @@ +[ + { + "id": "google/gemma-3-4b-it", + "sha": "093f9f388b31de276ce2de164bdc2081324b9767", + "created": "2025-02-20", + "last_modified": "2025-03-21", + "license": "gemma", + "gated": "manual", + "kind": "model", + "text": 3880263168, + "vision": 419816304, + "audio": 0, + "mtp": 0, + "embeddings": 671252480, + "tied_embeddings": false, + "total": 4300079472, + "model_type": "gemma3", + "text_loaded": 3880263168, + "instantiated": 4300079472, + "inst_class": "Gemma3ForConditionalGeneration" + }, + { + "id": "google/gemma-3n-E4B-it", + "sha": "c1221e9c62e34a43ab7ffacd1be0ea71f126ef10", + "created": "2025-06-03", + "last_modified": "2025-07-14", + "license": "gemma", + "gated": "manual", + "kind": "model", + "text": 6866696688, + "vision": 298587456, + "audio": 684694048, + "mtp": 0, + "embeddings": 2886205440, + "tied_embeddings": false, + "total": 7849978192, + "model_type": "gemma3n", + "text_loaded": 6866696688, + "instantiated": 7818517072, + "inst_class": "Gemma3nForConditionalGeneration" + }, + { + "id": "google/gemma-3n-E2B-it", + "sha": "5e092ebca197cdcd8d8b195040accf22693501bc", + "created": "2025-06-12", + "last_modified": "2025-07-14", + "license": "gemma", + "gated": "manual", + "kind": "model", + "text": 4456156768, + "vision": 298587456, + "audio": 684694048, + "mtp": 0, + "embeddings": 2550661120, + "tied_embeddings": false, + "total": 5439438272, + "model_type": "gemma3n", + "text_loaded": 4456156768, + "instantiated": 5418464192, + "inst_class": "Gemma3nForConditionalGeneration" + }, + { + "id": "google/gemma-4-E4B-it", + "sha": "ee0ef6023621cff504d758262d4e04895a5af4a2", + "created": "2026-03-02", + "last_modified": "2026-07-20", + "license": "apache-2.0", + "gated": false, + "kind": "model", + "text": 7518069034, + "vision": 169331136, + "audio": 308757248, + "mtp": 0, + "embeddings": 3489660928, + "tied_embeddings": true, + "total": 7996157418, + "model_type": "gemma4", + "text_loaded": 7518069034, + "instantiated": 7941100832, + "inst_class": "Gemma4ForConditionalGeneration" + }, + { + "id": "google/gemma-4-E2B-it", + "sha": "3e22461f65e89153144f8adb70e3b8c2cc9845a7", + "created": "2026-03-02", + "last_modified": "2026-07-20", + "license": "apache-2.0", + "gated": false, + "kind": "model", + "text": 4647449891, + "vision": 168544704, + "audio": 307184384, + "mtp": 0, + "embeddings": 2751463424, + "tied_embeddings": true, + "total": 5123178979, + "model_type": "gemma4", + "text_loaded": 4647449891, + "instantiated": 5104297504, + "inst_class": "Gemma4ForConditionalGeneration" + }, + { + "id": "google/gemma-4-12B-it", + "sha": "707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7", + "created": "2026-05-23", + "last_modified": "2026-07-20", + "license": "apache-2.0", + "gated": false, + "kind": "model", + "text": 11907350320, + "vision": 49922304, + "audio": 2457600, + "mtp": 0, + "embeddings": 1006632960, + "tied_embeddings": true, + "total": 11959730224, + "model_type": "gemma4_unified", + "text_loaded": 11907350320, + "instantiated": 11959730176, + "inst_class": "Gemma4UnifiedForConditionalGeneration" + }, + { + "id": "Qwen/Qwen3-4B-Instruct-2507", + "sha": "cdbee75f17c01a7cc42f958dc650907174af0554", + "created": "2025-08-05", + "last_modified": "2025-09-17", + "license": "apache-2.0", + "gated": false, + "kind": "model", + "text": 4022468096, + "vision": 0, + "audio": 0, + "mtp": 0, + "embeddings": 388956160, + "tied_embeddings": true, + "total": 4022468096, + "model_type": "qwen3", + "text_loaded": 4022468096, + "instantiated": 4022468096, + "inst_class": "Qwen3ForCausalLM" + }, + { + "id": "Qwen/Qwen3-4B", + "sha": "1cfa9a7208912126459214e8b04321603b3df60c", + "created": "2025-04-27", + "last_modified": "2025-07-26", + "license": "apache-2.0", + "gated": false, + "kind": "model", + "text": 4022468096, + "vision": 0, + "audio": 0, + "mtp": 0, + "embeddings": 388956160, + "tied_embeddings": true, + "total": 4022468096, + "model_type": "qwen3", + "text_loaded": 4022468096, + "instantiated": 4022468096, + "inst_class": "Qwen3ForCausalLM" + }, + { + "id": "Qwen/Qwen3.5-4B", + "sha": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a", + "created": "2026-02-27", + "last_modified": "2026-03-02", + "license": "apache-2.0", + "gated": false, + "kind": "model", + "text": 4205751296, + "vision": 333514240, + "audio": 0, + "mtp": 120599552, + "embeddings": 635699200, + "tied_embeddings": true, + "total": 4659865088, + "model_type": "qwen3_5", + "text_loaded": 4205751296, + "instantiated": 4205751296, + "inst_class": "Qwen3_5ForCausalLM" + }, + { + "id": "Qwen/Qwen3-8B", + "sha": "b968826d9c46dd6066d109eabc6255188de91218", + "created": "2025-04-27", + "last_modified": "2025-07-26", + "license": "apache-2.0", + "gated": false, + "kind": "model", + "text": 8190735360, + "vision": 0, + "audio": 0, + "mtp": 0, + "embeddings": 1244659712, + "tied_embeddings": false, + "total": 8190735360, + "model_type": "qwen3", + "text_loaded": 8190735360, + "instantiated": 8190735360, + "inst_class": "Qwen3ForCausalLM" + }, + { + "id": "Qwen/Qwen3.5-9B", + "sha": "c202236235762e1c871ad0ccb60c8ee5ba337b9a", + "created": "2026-02-27", + "last_modified": "2026-03-02", + "license": "apache-2.0", + "gated": false, + "kind": "model", + "text": 8953803264, + "vision": 456010480, + "audio": 0, + "mtp": 243290624, + "embeddings": 2034237440, + "tied_embeddings": false, + "total": 9653104368, + "model_type": "qwen3_5", + "text_loaded": 8953803264, + "instantiated": 8953803264, + "inst_class": "Qwen3_5ForCausalLM" + }, + { + "id": "meta-llama/Llama-3.2-3B-Instruct", + "sha": "0cb88a4f764b7a12671c53f0838cd831a0843b95", + "created": "2024-09-18", + "last_modified": "2024-10-24", + "license": "llama3.2", + "gated": "manual", + "kind": "model", + "text": 3212749824, + "vision": 0, + "audio": 0, + "mtp": 0, + "embeddings": 394002432, + "tied_embeddings": true, + "total": 3212749824, + "model_type": "llama", + "text_loaded": 3212749824, + "instantiated": 3212749824, + "inst_class": "LlamaForCausalLM" + }, + { + "id": "meta-llama/Llama-3.1-8B-Instruct", + "sha": "0e9e39f249a16976918f6564b8830bc894c89659", + "created": "2024-07-18", + "last_modified": "2024-09-25", + "license": "llama3.1", + "gated": "manual", + "kind": "model", + "text": 8030261248, + "vision": 0, + "audio": 0, + "mtp": 0, + "embeddings": 1050673152, + "tied_embeddings": false, + "total": 8030261248, + "model_type": "llama", + "text_loaded": 8030261248, + "instantiated": 8030261248, + "inst_class": "LlamaForCausalLM" + }, + { + "id": "polyglots/SinLlama_v01", + "sha": "4f464b14d56935e20d31a07e9a1ac3e5f31d8dfe", + "created": "2024-11-08", + "last_modified": "2025-08-30", + "license": null, + "gated": false, + "kind": "adapter", + "base": "meta-llama/Meta-Llama-3-8B", + "adapter_params": 3592093696, + "modules_to_save": [ + "embed_tokens", + "lm_head" + ], + "text": 8030261248, + "vision": 0, + "audio": 0, + "mtp": 0, + "embeddings": 1050673152, + "tied_embeddings": false, + "total": 11622354944, + "text_loaded": 11622354944, + "instantiated": null, + "inst_class": "base 8030261248 + adapter" + }, + { + "id": "ibm-granite/granite-4.1-8b", + "sha": "1504002f650e656a0a3789d99574df12e3e94ed0", + "created": "2026-04-06", + "last_modified": "2026-05-04", + "license": "apache-2.0", + "gated": false, + "kind": "model", + "text": 8380551168, + "vision": 0, + "audio": 0, + "mtp": 0, + "embeddings": 411041792, + "tied_embeddings": true, + "total": 8380551168, + "model_type": "granite", + "text_loaded": 8380551168, + "instantiated": 8380551168, + "inst_class": "GraniteForCausalLM" + }, + { + "id": "ibm-granite/granite-4.2-8b", + "sha": "f8de16cdcdbc6c779ca517604e050d82cc119e44", + "created": "2026-08-07", + "last_modified": "2026-09-04", + "license": "apache-2.0", + "gated": false, + "kind": "model", + "text": 8791592960, + "vision": 0, + "audio": 0, + "mtp": 0, + "embeddings": 822083584, + "tied_embeddings": false, + "total": 8791592960, + "model_type": "granite", + "text_loaded": 8791592960, + "instantiated": 8791592960, + "inst_class": "GraniteForCausalLM" + }, + { + "id": "ibm-granite/granite-4.2-3b", + "sha": "e459acceac81e5fe67c07d9cfc72329a332e7eb1", + "created": "2026-08-07", + "last_modified": "2026-09-02", + "license": "apache-2.0", + "gated": false, + "kind": "model", + "text": 3659737600, + "vision": 0, + "audio": 0, + "mtp": 0, + "embeddings": 513802240, + "tied_embeddings": false, + "total": 3659737600, + "model_type": "granite", + "text_loaded": 3659737600, + "instantiated": 3659737600, + "inst_class": "GraniteForCausalLM" + }, + { + "id": "HuggingFaceTB/SmolLM3-3B", + "sha": "a07cc9a04f16550a088caea529712d1d335b0ac1", + "created": "2025-07-08", + "last_modified": "2025-09-10", + "license": "apache-2.0", + "gated": false, + "kind": "model", + "text": 3075098624, + "vision": 0, + "audio": 0, + "mtp": 0, + "embeddings": 262668288, + "tied_embeddings": true, + "total": 3075098624, + "model_type": "smollm3", + "text_loaded": 3075098624, + "instantiated": 3075098624, + "inst_class": "SmolLM3ForCausalLM" + }, + { + "id": "microsoft/Phi-4-mini-instruct", + "sha": "cfbefacb99257ffa30c83adab238a50856ac3083", + "created": "2025-02-19", + "last_modified": "2025-12-10", + "license": "mit", + "gated": false, + "kind": "model", + "text": 3836021760, + "vision": 0, + "audio": 0, + "mtp": 0, + "embeddings": 614596608, + "tied_embeddings": true, + "total": 3836021760, + "model_type": "phi3", + "text_loaded": 3836021760, + "instantiated": 3836021760, + "inst_class": "Phi3ForCausalLM" + } +] \ No newline at end of file diff --git a/training/reports/sft_gmmlu_v1.md b/training/reports/sft_gmmlu_v1.md new file mode 100644 index 0000000000000000000000000000000000000000..f0a1c6224cd20ac28e0aa0cef484fb876d896231 --- /dev/null +++ b/training/reports/sft_gmmlu_v1.md @@ -0,0 +1,74 @@ +# Baseline comparison + +Accuracy in %. `orig` = original option order; `vote` = mean probabilities over cyclic option permutations; `±` = std of accuracy across permutations (position sensitivity). Random = 25. + +## Overview + +| model | params loaded | tok/char | SinhalaMMLU dev orig | vote | ± | macro-subj | Global-MMLU si | Belebele si | label mass | consistency | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| gemma-4-E4B-it_logit | 7.94B | 0.40 | 55.2 | 54.5 | 1.1 | 55.4 | 52.0 | 79.8 | 0.99 | 0.49 | +| sft_gmmlu_v1_step-357_logit | 7.46B | 0.40 | 49.6 | 52.7 | 1.1 | 49.7 | 54.1 | - | 1.00 | 0.35 | +| sft_gmmlu_v1_step-714_logit | 7.46B | 0.40 | 52.3 | 53.2 | 0.6 | 52.4 | 56.9 | - | 1.00 | 0.57 | + +## SinhalaMMLU dev: per subject + +| subject | n | gemma-4-E4B-it_logit | sft_gmmlu_v1_step-357_logit | sft_gmmlu_v1_step-714_logit | +|---|---:|---:|---:|---:| +| Arts | 100 | 56.0 | 41.0 | 47.0 | +| Buddhism | 162 | 51.9 | 42.6 | 48.8 | +| Catholicism | 132 | 50.0 | 50.0 | 53.0 | +| Christianity | 130 | 62.3 | 60.8 | 66.2 | +| Civics | 123 | 74.8 | 68.3 | 69.1 | +| Eastern music | 143 | 45.5 | 39.9 | 37.8 | +| Geography | 116 | 64.7 | 57.8 | 60.3 | +| Health and Physical Science | 130 | 69.2 | 59.2 | 64.6 | +| History | 151 | 39.1 | 39.1 | 38.4 | +| Islam | 108 | 59.3 | 57.4 | 59.3 | +| Sinhala language and literature | 155 | 46.5 | 38.1 | 48.4 | +| dancing | 108 | 34.3 | 25.9 | 27.8 | +| drama and Theatre | 128 | 55.5 | 57.8 | 55.5 | +| science | 164 | 66.5 | 58.5 | 57.9 | + +## SinhalaMMLU dev: per domain + +| category | n | gemma-4-E4B-it_logit | sft_gmmlu_v1_step-357_logit | sft_gmmlu_v1_step-714_logit | +|---|---:|---:|---:|---:| +| humanities | 1162 | 50.2 | 46.0 | 48.1 | +| language | 155 | 46.5 | 38.1 | 48.4 | +| social_science | 369 | 69.6 | 61.8 | 64.8 | +| stem | 164 | 66.5 | 58.5 | 57.9 | + +## SinhalaMMLU dev: per grade + +| grade | n | gemma-4-E4B-it_logit | sft_gmmlu_v1_step-357_logit | sft_gmmlu_v1_step-714_logit | +|---|---:|---:|---:|---:| +| 6 | 584 | 59.9 | 52.7 | 54.5 | +| 7 | 634 | 53.0 | 47.8 | 52.4 | +| 8 | 632 | 53.0 | 48.6 | 50.3 | + +## Global-MMLU si: per category + +| category | n | gemma-4-E4B-it_logit | sft_gmmlu_v1_step-357_logit | sft_gmmlu_v1_step-714_logit | +|---|---:|---:|---:|---:| +| Business | 102 | 54.9 | 57.8 | 54.9 | +| Humanities | 668 | 46.9 | 49.3 | 53.3 | +| Medical | 216 | 54.2 | 56.5 | 56.9 | +| Other | 126 | 56.3 | 61.1 | 60.3 | +| STEM | 449 | 51.9 | 53.9 | 57.7 | +| Social Sciences | 439 | 57.2 | 57.9 | 61.0 | + +## Global-MMLU si: culturally sensitive (CS) / agnostic (CA) / unlabelled (-) + +| cultural_sensitivity | n | gemma-4-E4B-it_logit | sft_gmmlu_v1_step-357_logit | sft_gmmlu_v1_step-714_logit | +|---|---:|---:|---:|---:| +| - | 1603 | 51.2 | 54.1 | 57.0 | +| CA | 277 | 57.8 | 54.9 | 58.5 | +| CS | 120 | 50.0 | 53.3 | 51.7 | + +## Position bias on SinhalaMMLU dev (share of predictions per displayed position) + +| model | 1 | 2 | 3 | 4 | +|---|---:|---:|---:|---:| +| gemma-4-E4B-it_logit | 22.4 | 31.0 | 26.3 | 20.4 | +| sft_gmmlu_v1_step-357_logit | 10.2 | 33.7 | 26.0 | 30.1 | +| sft_gmmlu_v1_step-714_logit | 27.3 | 26.4 | 25.7 | 20.7 | diff --git a/training/reports/teachers.md b/training/reports/teachers.md new file mode 100644 index 0000000000000000000000000000000000000000..52aaab66072bd9e1b9440523b89a0e5c709ad882 --- /dev/null +++ b/training/reports/teachers.md @@ -0,0 +1,82 @@ +# Baseline comparison + +Accuracy in %. `orig` = original option order; `vote` = mean probabilities over cyclic option permutations; `±` = std of accuracy across permutations (position sensitivity). Random = 25. + +## Overview + +| model | params loaded | tok/char | SinhalaMMLU dev orig | vote | ± | macro-subj | Global-MMLU si | Belebele si | label mass | consistency | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| gemma-4-E4B-it_logit | 7.94B | 0.40 | 55.2 | 54.5 | 1.1 | 55.4 | 52.0 | 79.8 | 0.99 | 0.49 | +| teacher_Qwen3.5-27B_gen | 26.90B | 0.53 | 64.8 | 64.8 | 0.0 | 64.9 | - | - | nan | 1.00 | +| teacher_Qwen3.5-27B_logit | 26.90B | 0.53 | 64.6 | 64.6 | 0.0 | 64.8 | 74.1 | - | 1.00 | 1.00 | +| teacher_gemma-4-26B-A4B-it_gen | 25.81B | 0.40 | 68.6 | 68.6 | 0.0 | 68.6 | - | - | nan | 1.00 | +| teacher_gemma-4-26B-A4B-it_logit | 25.81B | 0.40 | 68.3 | 68.3 | 0.0 | 68.3 | 66.5 | - | 1.00 | 1.00 | +| teacher_gemma-4-31B-it_gen | 31.27B | 0.40 | 73.5 | 73.5 | 0.0 | 73.4 | - | - | nan | 1.00 | +| teacher_gemma-4-31B-it_logit | 31.27B | 0.40 | 73.6 | 73.6 | 0.0 | 73.5 | 75.8 | - | 1.00 | 1.00 | + +## SinhalaMMLU dev: per subject + +| subject | n | gemma-4-E4B-it_logit | teacher_Qwen3.5-27B_gen | teacher_Qwen3.5-27B_logit | teacher_gemma-4-26B-A4B-it_gen | teacher_gemma-4-26B-A4B-it_logit | teacher_gemma-4-31B-it_gen | teacher_gemma-4-31B-it_logit | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| Arts | 100 | 56.0 | 61.0 | 62.0 | 69.0 | 69.0 | 67.0 | 67.0 | +| Buddhism | 162 | 51.9 | 58.6 | 58.6 | 69.8 | 69.8 | 76.5 | 77.2 | +| Catholicism | 132 | 50.0 | 73.5 | 73.5 | 75.0 | 76.5 | 80.3 | 82.6 | +| Christianity | 130 | 62.3 | 76.9 | 76.9 | 73.8 | 73.1 | 80.8 | 80.0 | +| Civics | 123 | 74.8 | 81.3 | 80.5 | 82.9 | 82.1 | 84.6 | 83.7 | +| Eastern music | 143 | 45.5 | 55.2 | 53.8 | 58.7 | 57.3 | 60.1 | 60.1 | +| Geography | 116 | 64.7 | 74.1 | 74.1 | 74.1 | 72.4 | 84.5 | 84.5 | +| Health and Physical Science | 130 | 69.2 | 74.6 | 74.6 | 76.9 | 76.9 | 81.5 | 81.5 | +| History | 151 | 39.1 | 45.7 | 45.0 | 62.9 | 60.3 | 62.9 | 62.9 | +| Islam | 108 | 59.3 | 75.0 | 75.0 | 77.8 | 77.8 | 86.1 | 87.0 | +| Sinhala language and literature | 155 | 46.5 | 56.8 | 55.5 | 57.4 | 59.4 | 65.2 | 65.2 | +| dancing | 108 | 34.3 | 36.1 | 39.8 | 41.7 | 39.8 | 45.4 | 45.4 | +| drama and Theatre | 128 | 55.5 | 60.9 | 60.2 | 63.3 | 64.1 | 69.5 | 68.8 | +| science | 164 | 66.5 | 78.0 | 78.0 | 77.4 | 77.4 | 83.5 | 83.5 | + +## SinhalaMMLU dev: per domain + +| category | n | gemma-4-E4B-it_logit | teacher_Qwen3.5-27B_gen | teacher_Qwen3.5-27B_logit | teacher_gemma-4-26B-A4B-it_gen | teacher_gemma-4-26B-A4B-it_logit | teacher_gemma-4-31B-it_gen | teacher_gemma-4-31B-it_logit | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| humanities | 1162 | 50.2 | 60.2 | 60.2 | 65.9 | 65.4 | 70.1 | 70.3 | +| language | 155 | 46.5 | 56.8 | 55.5 | 57.4 | 59.4 | 65.2 | 65.2 | +| social_science | 369 | 69.6 | 76.7 | 76.4 | 78.0 | 77.2 | 83.5 | 83.2 | +| stem | 164 | 66.5 | 78.0 | 78.0 | 77.4 | 77.4 | 83.5 | 83.5 | + +## SinhalaMMLU dev: per grade + +| grade | n | gemma-4-E4B-it_logit | teacher_Qwen3.5-27B_gen | teacher_Qwen3.5-27B_logit | teacher_gemma-4-26B-A4B-it_gen | teacher_gemma-4-26B-A4B-it_logit | teacher_gemma-4-31B-it_gen | teacher_gemma-4-31B-it_logit | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| 6 | 584 | 59.9 | 66.8 | 66.6 | 71.4 | 70.9 | 77.6 | 77.6 | +| 7 | 634 | 53.0 | 63.9 | 63.6 | 68.9 | 68.9 | 73.7 | 74.0 | +| 8 | 632 | 53.0 | 63.8 | 63.9 | 65.8 | 65.3 | 69.6 | 69.6 | + +## Global-MMLU si: per category + +| category | n | gemma-4-E4B-it_logit | teacher_Qwen3.5-27B_gen | teacher_Qwen3.5-27B_logit | teacher_gemma-4-26B-A4B-it_gen | teacher_gemma-4-26B-A4B-it_logit | teacher_gemma-4-31B-it_gen | teacher_gemma-4-31B-it_logit | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| Business | 102 | 54.9 | - | 73.5 | - | 68.6 | - | 78.4 | +| Humanities | 668 | 46.9 | - | 66.0 | - | 58.2 | - | 70.2 | +| Medical | 216 | 54.2 | - | 76.9 | - | 70.8 | - | 75.9 | +| Other | 126 | 56.3 | - | 74.6 | - | 68.3 | - | 77.8 | +| STEM | 449 | 51.9 | - | 78.2 | - | 68.2 | - | 76.2 | +| Social Sciences | 439 | 57.2 | - | 80.6 | - | 74.3 | - | 82.9 | + +## Global-MMLU si: culturally sensitive (CS) / agnostic (CA) / unlabelled (-) + +| cultural_sensitivity | n | gemma-4-E4B-it_logit | teacher_Qwen3.5-27B_gen | teacher_Qwen3.5-27B_logit | teacher_gemma-4-26B-A4B-it_gen | teacher_gemma-4-26B-A4B-it_logit | teacher_gemma-4-31B-it_gen | teacher_gemma-4-31B-it_logit | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| - | 1603 | 51.2 | - | 73.7 | - | 66.7 | - | 76.2 | +| CA | 277 | 57.8 | - | 77.3 | - | 68.6 | - | 77.3 | +| CS | 120 | 50.0 | - | 70.8 | - | 59.2 | - | 67.5 | + +## Position bias on SinhalaMMLU dev (share of predictions per displayed position) + +| model | 1 | 2 | 3 | 4 | +|---|---:|---:|---:|---:| +| gemma-4-E4B-it_logit | 22.4 | 31.0 | 26.3 | 20.4 | +| teacher_Qwen3.5-27B_gen | 21.8 | 31.3 | 26.4 | 20.5 | +| teacher_Qwen3.5-27B_logit | 22.0 | 31.1 | 26.4 | 20.5 | +| teacher_gemma-4-26B-A4B-it_gen | 24.9 | 29.9 | 25.0 | 20.2 | +| teacher_gemma-4-26B-A4B-it_logit | 24.9 | 30.4 | 25.0 | 19.7 | +| teacher_gemma-4-31B-it_gen | 24.9 | 30.1 | 25.8 | 19.1 | +| teacher_gemma-4-31B-it_logit | 24.8 | 29.9 | 26.1 | 19.2 | diff --git a/training/reports/throughput.jsonl b/training/reports/throughput.jsonl new file mode 100644 index 0000000000000000000000000000000000000000..fc14ec03c703e11a991d5ffabeb73629f2abebbc --- /dev/null +++ b/training/reports/throughput.jsonl @@ -0,0 +1,14 @@ +{"model": "models/gemma-4-E4B-it-text", "commit": "ee0ef6023621cff504d758262d4e04895a5af4a2+text-only", "mode": "lora", "seq_len": 2048, "micro_batch": 8, "total_params": 7501892608, "trainable_params": 38879232, "tokens_per_s": 500.3, "step_seconds": 32.75, "peak_mem_gb": 81.3, "extra_gb_for_fp32_adam": 0.1, "days_per_1B_tokens": 23.14, "env": {"python": "3.12.3", "machine": "aarch64", "torch": "2.14.1+cu130", "transformers": "5.19.0", "datasets": "5.1.0", "peft": "0.21.2", "accelerate": "1.15.0", "huggingface_hub": "2.2.0", "gpu": "NVIDIA GB10", "cuda": "13.0", "git_commit": "HEAD", "git_dirty": true}} +{"model": "models/gemma-4-E4B-it-text", "commit": "ee0ef6023621cff504d758262d4e04895a5af4a2+text-only", "mode": "freeze-embed", "seq_len": 2048, "micro_batch": 8, "total_params": 7463013376, "trainable_params": 3973352448, "tokens_per_s": 704.9, "step_seconds": 23.24, "peak_mem_gb": 97.4, "extra_gb_for_fp32_adam": 14.8, "days_per_1B_tokens": 16.42, "env": {"python": "3.12.3", "machine": "aarch64", "torch": "2.14.1+cu130", "transformers": "5.19.0", "datasets": "5.1.0", "peft": "0.21.2", "accelerate": "1.15.0", "huggingface_hub": "2.2.0", "gpu": "NVIDIA GB10", "cuda": "13.0", "git_commit": "HEAD", "git_dirty": true}} +{"model": "models/gemma-4-E4B-it-text", "commit": "ee0ef6023621cff504d758262d4e04895a5af4a2+text-only", "mode": "freeze-embed", "loss": "chunked", "ckpt_every": 1, "compile": false, "tag": "speedups", "seq_len": 2048, "micro_batch": 32, "total_params": 7463013376, "trainable_params": 3973352448, "tokens_per_s": 628.9, "step_seconds": 104.21, "peak_mem_gb": 71.0, "extra_gb_for_fp32_adam": 14.8, "days_per_1B_tokens": 18.4, "env": {"python": "3.12.3", "machine": "aarch64", "torch": "2.14.1+cu130", "transformers": "5.19.0", "datasets": "5.1.0", "peft": "0.21.2", "accelerate": "1.15.0", "huggingface_hub": "2.2.0", "gpu": "NVIDIA GB10", "cuda": "13.0", "git_commit": "HEAD", "git_dirty": true}} +{"model": "models/gemma-4-E4B-it-text", "commit": "ee0ef6023621cff504d758262d4e04895a5af4a2+text-only", "mode": "freeze-embed", "loss": "liger", "ckpt_every": 1, "compile": false, "tag": "speedups", "seq_len": 2048, "micro_batch": 32, "total_params": 7463013376, "trainable_params": 3973352448, "tokens_per_s": 714.1, "step_seconds": 91.77, "peak_mem_gb": 71.0, "extra_gb_for_fp32_adam": 14.8, "days_per_1B_tokens": 16.21, "env": {"python": "3.12.3", "machine": "aarch64", "torch": "2.14.1+cu130", "transformers": "5.19.0", "datasets": "5.1.0", "peft": "0.21.2", "accelerate": "1.15.0", "huggingface_hub": "2.2.0", "gpu": "NVIDIA GB10", "cuda": "13.0", "git_commit": "HEAD", "git_dirty": true}} +{"model": "models/gemma-4-E4B-it-text", "commit": "ee0ef6023621cff504d758262d4e04895a5af4a2+text-only", "mode": "freeze-embed", "loss": "liger", "ckpt_every": 0, "compile": false, "tag": "speedups", "seq_len": 2048, "micro_batch": 2, "total_params": 7463013376, "trainable_params": 3973352448, "tokens_per_s": 763.1, "step_seconds": 5.37, "peak_mem_gb": 69.1, "extra_gb_for_fp32_adam": 14.8, "days_per_1B_tokens": 15.17, "env": {"python": "3.12.3", "machine": "aarch64", "torch": "2.14.1+cu130", "transformers": "5.19.0", "datasets": "5.1.0", "peft": "0.21.2", "accelerate": "1.15.0", "huggingface_hub": "2.2.0", "gpu": "NVIDIA GB10", "cuda": "13.0", "git_commit": "HEAD", "git_dirty": true}} +{"model": "models/gemma-4-E4B-it-text", "commit": "ee0ef6023621cff504d758262d4e04895a5af4a2+text-only", "mode": "freeze-embed", "loss": "liger", "ckpt_every": 1, "compile": true, "tag": "speedups", "seq_len": 2048, "micro_batch": 32, "total_params": 7463013376, "trainable_params": 3973352448, "tokens_per_s": 712.2, "step_seconds": 92.02, "peak_mem_gb": 71.0, "extra_gb_for_fp32_adam": 14.8, "days_per_1B_tokens": 16.25, "env": {"python": "3.12.3", "machine": "aarch64", "torch": "2.14.1+cu130", "transformers": "5.19.0", "datasets": "5.1.0", "peft": "0.21.2", "accelerate": "1.15.0", "huggingface_hub": "2.2.0", "gpu": "NVIDIA GB10", "cuda": "13.0", "git_commit": "HEAD", "git_dirty": true}} +{"model": "models/gemma-4-E4B-it-text", "commit": "ee0ef6023621cff504d758262d4e04895a5af4a2+text-only", "mode": "lora", "loss": "liger", "ckpt_every": 1, "compile": true, "tag": "speedups", "seq_len": 2048, "micro_batch": 32, "total_params": 7501892608, "trainable_params": 38879232, "tokens_per_s": 505.2, "step_seconds": 129.71, "peak_mem_gb": 52.1, "extra_gb_for_fp32_adam": 0.1, "days_per_1B_tokens": 22.91, "env": {"python": "3.12.3", "machine": "aarch64", "torch": "2.14.1+cu130", "transformers": "5.19.0", "datasets": "5.1.0", "peft": "0.21.2", "accelerate": "1.15.0", "huggingface_hub": "2.2.0", "gpu": "NVIDIA GB10", "cuda": "13.0", "git_commit": "HEAD", "git_dirty": true}} +{"model": "Qwen/Qwen3.5-4B", "commit": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a", "mode": "full", "loss": "hf", "ckpt_every": 1, "compile": false, "tag": "speedups", "seq_len": 2048, "micro_batch": 8, "total_params": 4205751296, "trainable_params": 4205751296, "tokens_per_s": 1016.3, "step_seconds": 16.12, "peak_mem_gb": 72.1, "extra_gb_for_fp32_adam": 15.7, "days_per_1B_tokens": 11.39, "env": {"python": "3.12.3", "machine": "aarch64", "torch": "2.14.1+cu130", "transformers": "5.19.0", "datasets": "5.1.0", "peft": "0.21.2", "accelerate": "1.15.0", "huggingface_hub": "2.2.0", "gpu": "NVIDIA GB10", "cuda": "13.0", "git_commit": "HEAD", "git_dirty": true}} +{"model": "Qwen/Qwen3.5-4B", "commit": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a", "mode": "full", "loss": "liger", "ckpt_every": 1, "compile": false, "tag": "speedups", "seq_len": 2048, "micro_batch": 32, "total_params": 4205751296, "trainable_params": 4205751296, "tokens_per_s": 1084.7, "step_seconds": 60.42, "peak_mem_gb": 53.4, "extra_gb_for_fp32_adam": 15.7, "days_per_1B_tokens": 10.67, "env": {"python": "3.12.3", "machine": "aarch64", "torch": "2.14.1+cu130", "transformers": "5.19.0", "datasets": "5.1.0", "peft": "0.21.2", "accelerate": "1.15.0", "huggingface_hub": "2.2.0", "gpu": "NVIDIA GB10", "cuda": "13.0", "git_commit": "HEAD", "git_dirty": true}} +{"model": "Qwen/Qwen3.5-4B", "commit": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a", "mode": "full", "loss": "liger", "ckpt_every": 1, "compile": true, "tag": "speedups", "seq_len": 2048, "micro_batch": 32, "total_params": 4205751296, "trainable_params": 4205751296, "tokens_per_s": 1583.8, "step_seconds": 41.38, "peak_mem_gb": 46.9, "extra_gb_for_fp32_adam": 15.7, "days_per_1B_tokens": 7.31, "env": {"python": "3.12.3", "machine": "aarch64", "torch": "2.14.1+cu130", "transformers": "5.19.0", "datasets": "5.1.0", "peft": "0.21.2", "accelerate": "1.15.0", "huggingface_hub": "2.2.0", "gpu": "NVIDIA GB10", "cuda": "13.0", "git_commit": "HEAD", "git_dirty": true}} +{"model": "Qwen/Qwen3.5-4B", "commit": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a", "mode": "lora", "loss": "liger", "ckpt_every": 1, "compile": true, "tag": "speedups", "seq_len": 2048, "micro_batch": 32, "total_params": 4238216192, "trainable_params": 32464896, "tokens_per_s": 1179.6, "step_seconds": 55.56, "peak_mem_gb": 38.0, "extra_gb_for_fp32_adam": 0.1, "days_per_1B_tokens": 9.81, "env": {"python": "3.12.3", "machine": "aarch64", "torch": "2.14.1+cu130", "transformers": "5.19.0", "datasets": "5.1.0", "peft": "0.21.2", "accelerate": "1.15.0", "huggingface_hub": "2.2.0", "gpu": "NVIDIA GB10", "cuda": "13.0", "git_commit": "HEAD", "git_dirty": true}} +{"model": "models/gemma-4-E4B-it-text", "commit": "ee0ef6023621cff504d758262d4e04895a5af4a2+text-only", "mode": "freeze-embed", "loss": "liger", "ckpt_every": 0, "compile": true, "tag": "compile_fixed", "seq_len": 2048, "micro_batch": 2, "total_params": 7463013376, "trainable_params": 3973352448, "tokens_per_s": 1150.4, "step_seconds": 3.56, "peak_mem_gb": 98.5, "extra_gb_for_fp32_adam": 14.8, "days_per_1B_tokens": 10.06, "env": {"python": "3.12.3", "machine": "aarch64", "torch": "2.14.1+cu130", "transformers": "5.19.0", "datasets": "5.1.0", "peft": "0.21.2", "accelerate": "1.15.0", "huggingface_hub": "2.2.0", "gpu": "NVIDIA GB10", "cuda": "13.0", "git_commit": "HEAD", "git_dirty": true}} +{"model": "models/gemma-4-E4B-it-text", "commit": "ee0ef6023621cff504d758262d4e04895a5af4a2+text-only", "mode": "freeze-embed", "loss": "liger", "ckpt_every": 0, "compile": true, "optim": "adamw8bit", "tag": "memory", "seq_len": 2048, "micro_batch": 2, "total_params": 7463013376, "trainable_params": 3973352448, "tokens_per_s": 1148.9, "step_seconds": 3.57, "peak_mem_gb": 91.3, "extra_gb_for_fp32_adam": 14.8, "days_per_1B_tokens": 10.07, "env": {"python": "3.12.3", "machine": "aarch64", "torch": "2.14.1+cu130", "transformers": "5.19.0", "datasets": "5.1.0", "peft": "0.21.2", "accelerate": "1.15.0", "huggingface_hub": "2.2.0", "gpu": "NVIDIA GB10", "cuda": "13.0", "git_commit": "HEAD", "git_dirty": true}} +{"model": "models/gemma-4-E4B-it-text", "commit": "ee0ef6023621cff504d758262d4e04895a5af4a2+text-only", "mode": "freeze-embed", "loss": "liger", "ckpt_every": 2, "compile": true, "optim": "adamw8bit", "tag": "memory", "seq_len": 2048, "micro_batch": 4, "total_params": 7463013376, "trainable_params": 3973352448, "tokens_per_s": 1262.6, "step_seconds": 6.49, "peak_mem_gb": 98.1, "extra_gb_for_fp32_adam": 14.8, "days_per_1B_tokens": 9.17, "env": {"python": "3.12.3", "machine": "aarch64", "torch": "2.14.1+cu130", "transformers": "5.19.0", "datasets": "5.1.0", "peft": "0.21.2", "accelerate": "1.15.0", "huggingface_hub": "2.2.0", "gpu": "NVIDIA GB10", "cuda": "13.0", "git_commit": "HEAD", "git_dirty": true}} diff --git a/training/requirements.lock.txt b/training/requirements.lock.txt new file mode 100644 index 0000000000000000000000000000000000000000..53b260802bf12affab8b383c7a19f152dcae3b77 --- /dev/null +++ b/training/requirements.lock.txt @@ -0,0 +1,84 @@ +accelerate==1.15.0 +aiohappyeyeballs==2.7.1 +aiohttp==3.14.4 +aiosignal==1.4.0 +annotated-doc==0.0.5 +anyio==4.15.1 +attrs==26.1.0 +click==8.5.0 +cuda-bindings==13.3.1 +cuda-pathfinder==1.6.0 +cuda-toolkit==13.0.3.0 +datasets==5.1.0 +dill==0.4.1 +einops==0.8.2 +filelock==3.32.3 +fla-core==0.5.2 +flash-linear-attention==0.5.2 +frozenlist==1.8.0 +fsspec==2026.7.0 +h11==0.16.0 +hf-xet==1.7.0 +httpcore2==2.13.1 +httpx2==2.13.1 +huggingface_hub==2.2.0 +idna==3.20 +Jinja2==3.1.6 +liger_kernel==0.8.4 +markdown-it-py==4.2.0 +MarkupSafe==3.0.3 +mdurl==0.1.2 +mpmath==1.3.0 +multidict==6.9.1 +multiprocess==0.70.19 +networkx==3.6.1 +numpy==2.5.3 +nvidia-cublas==13.1.1.3 +nvidia-cuda-cupti==13.0.85 +nvidia-cuda-nvrtc==13.0.88 +nvidia-cuda-runtime==13.0.96 +nvidia-cudnn-cu13==9.24.0.43 +nvidia-cufft==12.0.0.61 +nvidia-cufile==1.15.1.6 +nvidia-curand==10.4.0.35 +nvidia-cusolver==12.0.4.66 +nvidia-cusparse==12.6.3.3 +nvidia-cusparselt-cu13==0.8.1 +nvidia-nccl-cu13==2.30.7 +nvidia-nvjitlink==13.3.33 +nvidia-nvshmem-cu13==3.4.5 +nvidia-nvtx==13.0.85 +packaging==26.3 +pandas==3.0.6 +peft==0.21.2 +pillow==12.3.0 +propcache==0.5.4 +protobuf==7.36.2 +psutil==7.2.2 +pyarrow==26.0.0 +Pygments==2.21.0 +python-dateutil==2.9.0.post0 +PyYAML==6.0.3 +regex==2026.9.29 +rich==15.0.0 +safetensors==0.8.0 +scipy==1.18.1 +sentencepiece==0.2.2 +setuptools==78.1.0 +shellingham==1.5.4 +six==1.17.0 +sympy==1.14.0 +tabulate==0.10.0 +timm==1.0.30 +tokenizers==0.23.3 +torch==2.14.1+cu130 +torchao==0.18.0 +torchvision==0.29.1 +tqdm==4.70.1 +transformers==5.19.0 +triton==3.8.0 +truststore==0.10.4 +typer==0.27.3 +typing_extensions==4.16.0 +xxhash==4.0.1 +yarl==1.25.1 diff --git a/training/run_submission.sh b/training/run_submission.sh new file mode 100644 index 0000000000000000000000000000000000000000..4223d4dbd722e8cf9b40821a61b470be8d58a264 --- /dev/null +++ b/training/run_submission.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +# Reproduce a submission end to end, offline, with one command. +# +# ./run_submission.sh [config] [test_dir] [output_zip] +# +# Defaults: the current final configuration, the shared-task test set in data/test (download it with the +# organisers' instructions), output in results/submissions/final.zip. Requires the environment from README.md and the +# model weights referenced by the config under models/. No network access is used (HF_HUB_OFFLINE=1). +set -euo pipefail +cd "$(dirname "$0")" +CONFIG=${1:-configs/submission_sub6_think_medhard.yaml} # final configuration (updated at the config freeze) +INPUT=${2:-data/test} +OUTPUT=${3:-results/submissions/final.zip} + +echo "config: $CONFIG" +grep -E "^(model|adapter|n_perms|thinking_levels|think_max_new_tokens):" "$CONFIG" | sed 's/^/ /' +HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 .venv/bin/python -m inference.predict --config "$CONFIG" --input "$INPUT" --output "$OUTPUT" +.venv/bin/python -m inference.check_predictions --predictions "$OUTPUT" --gold "$INPUT" | tail -1 +echo "submission file: $OUTPUT (upload to Codabench)" diff --git a/training/scripts/__init__.py b/training/scripts/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/training/scripts/add_teacher_labels.py b/training/scripts/add_teacher_labels.py new file mode 100644 index 0000000000000000000000000000000000000000..2237da201cd2ea1b0ea5fefcf03a53efd9f620ae --- /dev/null +++ b/training/scripts/add_teacher_labels.py @@ -0,0 +1,41 @@ +"""Attach a teacher's option probabilities (from an eval.run_eval logit run) to an MCQ training file. + + python scripts/add_teacher_labels.py --data data/sft/gmmlu_si_train.jsonl --run results/teacher_31b_gmmlu_train \ + --out data/sft/gmmlu_si_train_t31b.jsonl + +teacher_probs are in the file's stored option order (probs_orig of permutation 0). Reports teacher accuracy on the +file's keys and how often teacher and key disagree. +""" +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import read_jsonl, write_jsonl # noqa: E402 + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--data", required=True) + ap.add_argument("--run", required=True) + ap.add_argument("--out", required=True) + args = ap.parse_args() + preds = {r["uid"]: r["probs_orig"] for r in read_jsonl(Path(args.run) / "predictions.jsonl") if r["perm_k"] == 0} + rows, agree, missing = [], 0, 0 + for r in read_jsonl(args.data): + p = preds.get(r["uid"]) + if p is None: + missing += 1 + rows.append(r) + continue + agree += max(range(len(p)), key=p.__getitem__) + 1 == r["answer"] + rows.append(r | {"teacher_probs": [round(x, 5) for x in p]}) + write_jsonl(args.out, rows) + n = len(rows) - missing + print(f"{n} items labelled ({missing} without teacher probs); teacher agrees with the key on {agree / max(n, 1):.3f}") + + +if __name__ == "__main__": + main() diff --git a/training/scripts/build_cpt_corpus.py b/training/scripts/build_cpt_corpus.py new file mode 100644 index 0000000000000000000000000000000000000000..ffb65e81b6adae31ee542c8100d2f482e70ac4f0 --- /dev/null +++ b/training/scripts/build_cpt_corpus.py @@ -0,0 +1,198 @@ +"""Sinhala continued-pretraining corpus -> data/corpus/clean/*.parquet (+ aggregate stats in reports/). + +Sources (downloaded to data/raw/corpus by this script, pinned revisions in DATA_LOG.md): + fineweb2 HuggingFaceFW/fineweb-2 sin_Sinh (ODC-By; already MinHash-deduplicated upstream) + wikipedia wikimedia/wikipedia 20231101.si (CC BY-SA 3.0) + nsina sinhala-nlp/NSINA news (CC BY-SA 4.0) + +Per document, in order (first failing rule is recorded): + 1. blocked_domain exam / past-paper / curriculum sites (SinhalaMMLU's sources and anything like them) + 2. exam_like question-paper text: "ප්‍රශ්න පත්‍ර", "වාර විභාග", "past paper" ... or many numbered option lines + 3. too_short < 200 characters + 4. not_sinhala < 50% of letters in the Sinhala block + 5. repetitive < 30% unique lines (boilerplate, menus) + 6. duplicate same normalised text already kept (exact, across sources; FineWeb-2 is near-deduped upstream) + 7. contaminated shares a word 8-gram with any SinhalaMMLU dev or Global-MMLU validation question +Kept documents get sl_score = Sri Lanka / curriculum-topic keyword hits per 1k characters (used to prioritise). + +Stages are resumable: finished shards are skipped on rerun. +Usage: python scripts/build_cpt_corpus.py [--limit N] [--workers 12] [--out data/corpus] +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import re +import sys +import time +import unicodedata +from collections import Counter +from multiprocessing import Pool +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import ROOT, load_env, read_jsonl # noqa: E402 + +SOURCES = { + "fineweb2": ("HuggingFaceFW/fineweb-2", "data/sin_Sinh/train/000_00000.parquet"), + "wikipedia": ("wikimedia/wikipedia", "20231101.si/train-00000-of-00001.parquet"), + "nsina": ("sinhala-nlp/NSINA", "NSINA.json"), +} +BLOCKED_DOMAIN = re.compile( + r"pastpaper|past-paper|past_paper|thaksalawa|doenets|exam|paperbank|papersbank|model-?paper|term-?test|" + r"nie\.lk|edupub|e-?learning|tutor|guru\.lk|alevel|a-level|olevel|o-level|scholarship", re.I) +EXAM_PHRASES = re.compile( + r"ප්‍රශ්න\s*පත්‍ර|වාර\s*විභාග|පසුගිය\s*විභාග|ආදර්ශ\s*ප්‍රශ්න|පෙරහුරු\s*ප්‍රශ්න|" + r"past\s*papers?|model\s*papers?|term\s*test|marking\s*scheme|ලකුණු\s*දීමේ\s*පටිපාටිය", re.I) +OPTION_LINE = re.compile(r"^\s*(\(?[1-5][\).]|\(?[iv]{1,3}\)|[අආඇඉ]\))\s*\S", re.M) +SL_TOPICS = re.compile( + r"ශ්‍රී\s*ලංකා|ලංකාව|සිංහල|බුදු|බෞද්ධ|ධර්ම|විහාර|රජතුමා|රාජධානි|අනුරාධපුර|පොළොන්නරු|මහනුවර|ඉතිහාස|" + r"පුරාවිද්‍යා|සාහිත්‍ය|කාව්‍ය|නර්තන|නැටුම|සංගීත|රාග|චිත්‍ර|කලාව|නාට්‍ය|කතෝලික|ක්‍රිස්තියානි|ඉස්ලාම්|" + r"භූගෝල|පුරවැසි|ආණ්ඩුක්‍රම|සෞඛ්‍ය|විද්‍යාව") +SINHALA = re.compile(r"[඀-෿]") +LETTER = re.compile(r"[^\W\d_]") + + +def norm(text: str) -> str: + text = unicodedata.normalize("NFC", text).lower() + text = re.sub(r"[‌‍]", "", text) + # keep the whole Sinhala block: vowel signs and virama are combining marks, which \w does not match + return re.sub(r"\s+", " ", re.sub(r"[^\w\s\u0D80-\u0DFF]", " ", text)).strip() + + +def word_grams(text: str, n: int = 8): + w = norm(text).split() + return {hash(" ".join(w[i:i + n])) for i in range(len(w) - n + 1)} + + +_CONTAM: set[int] = set() + + +def _init(contam: set[int]) -> None: + global _CONTAM + _CONTAM = contam + + +def check(doc: dict) -> tuple[str, dict]: + """Return (reason or 'kept', doc with stats). Duplicate check happens in the parent (needs global state).""" + text, url = doc["text"], doc.get("url") or "" + if url and BLOCKED_DOMAIN.search(url.split("/")[2] if "://" in url else url): + return "blocked_domain", doc + if EXAM_PHRASES.search(text[:5000]) or len(OPTION_LINE.findall(text)) >= 8: + return "exam_like", doc + if len(text) < 200: + return "too_short", doc + letters = LETTER.findall(text) + if not letters or len(SINHALA.findall(text)) / len(letters) < 0.5: + return "not_sinhala", doc + lines = [l.strip() for l in text.splitlines() if l.strip()] + if len(lines) >= 10 and len(set(lines)) / len(lines) < 0.3: + return "repetitive", doc + if _CONTAM and word_grams(text) & _CONTAM: + return "contaminated", doc + doc["n_chars"] = len(text) + doc["sl_score"] = round(1000 * len(SL_TOPICS.findall(text)) / len(text), 3) + doc["hash"] = hashlib.sha1(norm(text).encode()).hexdigest() + return "kept", doc + + +def iter_source(name: str, path: Path, limit: int | None, batch: int = 20000): + if name == "nsina": + rows = json.loads(path.read_text(encoding="utf-8")) + out = [] + for i, r in enumerate(rows[:limit] if limit else rows): + text = "\n".join(x for x in (r.get("Headline"), r.get("News Content")) if x) + out.append({"id": f"nsina/{i}", "source": name, "url": r.get("Source") or "", "text": text}) + if len(out) == batch: + yield out + out = [] + if out: + yield out + return + import pyarrow.parquet as pq + + pf, n = pq.ParquetFile(path), 0 + for rb in pf.iter_batches(batch_size=batch, columns=["id", "url", "text"] + (["title"] if name == "wikipedia" else [])): + out = [] + for r in rb.to_pylist(): + text = f"{r['title']}\n{r['text']}" if name == "wikipedia" else r["text"] + out.append({"id": f"{name}/{r['id']}", "source": name, "url": r.get("url") or "", "text": text}) + if limit: + out = out[: max(limit - n, 0)] + n += len(out) + if out: + yield out + if limit and n >= limit: + return + + +def contamination_grams() -> set[int]: + grams = set() + for f in ("sinhalammlu_dev", "gmmlu_si_val"): + for r in read_jsonl(ROOT / "data" / "processed" / f"{f}.jsonl"): + grams |= word_grams(r["question"]) + grams |= word_grams(r["question"] + " " + r["options"][r["answer"] - 1]) + return grams + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--limit", type=int, help="documents per source (smoke test)") + ap.add_argument("--workers", type=int, default=12) + ap.add_argument("--out", default=str(ROOT / "data" / "corpus")) + args = ap.parse_args() + load_env() + import pandas as pd + from huggingface_hub import HfApi, hf_hub_download + + out = Path(args.out) + (out / "clean").mkdir(parents=True, exist_ok=True) + api = HfApi() + revisions = {} + for name, (repo, f) in SOURCES.items(): + sha = api.dataset_info(repo).sha + revisions[name] = f"{repo}@{sha}" + hf_hub_download(repo, f, repo_type="dataset", revision=sha, + local_dir=ROOT / "data" / "raw" / "corpus" / repo.split("/")[1]) + contam = contamination_grams() + print(f"contamination 8-grams: {len(contam):,}", flush=True) + + seen_hashes: set[str] = set() + stats = {"revisions": revisions, "sources": {}} + t0 = time.time() + with Pool(args.workers, initializer=_init, initargs=(contam,)) as pool: + for name, (repo, f) in SOURCES.items(): + path = ROOT / "data" / "raw" / "corpus" / repo.split("/")[1] / f + reasons, kept_chars, sl = Counter(), 0, [] + for shard, docs in enumerate(iter_source(name, path, args.limit)): + shard_file = out / "clean" / f"{name}-{shard:04d}.parquet" + kept = [] + for reason, doc in pool.imap(check, docs, chunksize=64): + if reason == "kept": + if doc["hash"] in seen_hashes: + reason = "duplicate" + else: + seen_hashes.add(doc["hash"]) + kept.append(doc) + reasons[reason] += 1 + if not shard_file.exists() and kept: + pd.DataFrame(kept).drop(columns=["hash"]).to_parquet(shard_file, index=False) + kept_chars += sum(d["n_chars"] for d in kept) + sl += [d["sl_score"] for d in kept] + print(f"{name} shard {shard}: {sum(reasons.values()):,} docs seen, {reasons['kept']:,} kept " + f"({time.time() - t0:.0f}s)", flush=True) + stats["sources"][name] = { + "docs_seen": sum(reasons.values()), "reasons": dict(reasons), "kept_chars": kept_chars, + "est_gemma_tokens": int(kept_chars * 0.40), # 0.40 Gemma tokens / Sinhala char (baseline runs) + "sl_score_quantiles": {q: float(pd.Series(sl).quantile(q / 100)) for q in (50, 90, 99)} if sl else {}, + } + stats["total_est_gemma_tokens"] = sum(s["est_gemma_tokens"] for s in stats["sources"].values()) + stats["seconds"] = round(time.time() - t0) + name = "corpus_stats_smoke.json" if args.limit else "corpus_stats.json" + (ROOT / "reports" / name).write_text(json.dumps(stats, indent=2, ensure_ascii=False)) + print(json.dumps(stats, indent=2, ensure_ascii=False)) + + +if __name__ == "__main__": + main() diff --git a/training/scripts/build_passage_pool.py b/training/scripts/build_passage_pool.py new file mode 100644 index 0000000000000000000000000000000000000000..4737b96a4832b248e28f9e96381e35ca27c29fb4 --- /dev/null +++ b/training/scripts/build_passage_pool.py @@ -0,0 +1,117 @@ +"""Passage pool for synthetic MCQ generation: Sinhala passages assigned to curriculum subjects. + +Subjects are the published SinhalaMMLU subject list (Pramodya et al. 2025, Table 2), never test questions. Passages come +from the cleaned pretraining mix (data/corpus/cpt_v1: Wikipedia, FineWeb-2, NSINA; exam sites already removed). +Documents are split into 400-1200 character chunks; a chunk is assigned to the subject whose keyword list it matches +most (at least 2 distinct keywords). Wikipedia chunks are preferred. + + python scripts/build_passage_pool.py --per-subject 6 --out data/synth/passages_pilot.jsonl +""" +from __future__ import annotations + +import argparse +import json +import random +import re +import sys +from collections import Counter +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import ROOT, write_jsonl # noqa: E402 + +# subject -> (level description used in the generation prompt, Sinhala keywords) +SUBJECTS = { + "History": ("grades 6-11", ["ඉතිහාස", "රජතුමා", "රාජධානි", "අනුරාධපුර", "පොළොන්නරු", "මහාවංශ", "වංශකථා", "ශිලා ලිපි", "යටත් විජිත", "පෘතුගීසි", "ලන්දේසි", "නිදහස් අරගල"]), + "Buddhism": ("grades 6-11", ["බුදුරජාණන්", "ධර්මය", "භික්ෂු", "සංඝ", "සූත්‍ර", "ත්‍රිපිටක", "නිර්වාණ", "පන්සිල්", "භාවනා", "ජාතක"]), + "Buddhist Civilization": ("grades 12-13 (A/L)", ["බෞද්ධ ශිෂ්ටාචාර", "ස්තූප", "දාගැබ", "චෛත්‍ය", "ආරාම", "ධාතු", "අභයගිරි", "මහාවිහාර", "ජේතවන"]), + "Catholicism": ("grades 6-11", ["කතෝලික", "පාප්වහන්සේ", "සාන්තුවර", "දිව්‍ය පූජා", "කන්‍යා මරියා", "රදගුරු"]), + "Christianity": ("grades 6-11", ["ක්‍රිස්තියානි", "යේසුස්", "බයිබලය", "ශුද්ධ", "ගිවිසුම", "දේවස්ථාන", "ගෝලයන්"]), + "Islam": ("grades 6-11", ["ඉස්ලාම්", "අල්ලාහ්", "නබි", "කුර්ආන්", "මුස්ලිම්", "රාමසාන්", "හජ්", "මස්ජිද්"]), + "Dancing": ("grades 6-13", ["නර්තන", "නැටුම", "උඩරට", "පහතරට", "සබරගමු", "වන්නම", "ගැටබෙර", "වෙස්", "තාලම්"]), + "Eastern Music": ("grades 6-11", ["සංගීත", "රාග", "තාල", "ස්වර", "වාදන", "සිතාර්", "තබ්ලා", "ගායන", "හින්දුස්ථානි"]), + "Drama and Theatre": ("grades 6-11", ["නාට්‍ය", "රංගන", "නාඩගම්", "කෝලම්", "සොකරි", "වේදිකා", "සරච්චන්ද්‍ර", "නළු"]), + "Arts": ("grades 6-8", ["චිත්‍ර", "කලාව", "මූර්ති", "බිතු සිතුවම්", "සීගිරි", "වර්ණ", "කලාකරු"]), + "Sinhala Language and Literature": ("grades 6-13", ["කාව්‍ය", "සාහිත්‍ය", "නවකතා", "කවි", "ව්‍යාකරණ", "සන්දේශ", "කෙටිකතා", "ලේඛක"]), + "Citizenship Education": ("grades 6-11", ["ආණ්ඩුක්‍රම", "පාර්ලිමේන්තු", "ප්‍රජාතන්ත්‍ර", "මූලික අයිතිවාසිකම්", "ඡන්ද", "පළාත් සභා", "පුරවැසි"]), + "Political Science": ("grades 12-13 (A/L)", ["දේශපාලන", "රාජ්‍යය", "ව්‍යවස්ථාව", "පරමාධිපත්‍ය", "ඒකීය", "ෆෙඩරල්", "විධායක"]), + "Geography": ("grades 6-13", ["භූගෝල", "දේශගුණ", "මෝසම්", "ගංගා", "කඳු", "වර්ෂාපතන", "සාගර", "පාංශු", "ජනගහන"]), + "Health and Physical Education": ("grades 6-11", ["සෞඛ්‍ය", "පෝෂණ", "රෝග", "ව්‍යායාම", "ක්‍රීඩා", "විටමින්", "ප්‍රතිශක්ති"]), + "Science": ("grades 6-11", ["විද්‍යාව", "පදාර්ථ", "ශාක", "සත්ත්ව", "විද්‍යුත්", "චුම්බක", "ආලෝකය", "පරිසර"]), + "Physics": ("grades 12-13 (A/L)", ["භෞතික", "බලය", "ප්‍රවේග", "ත්වරණ", "තරංග", "විද්‍යුත් ධාරා", "ගුරුත්වාකර්ෂණ", "ශක්තිය"]), + "Chemistry": ("grades 12-13 (A/L)", ["රසායන", "මූලද්‍රව්‍ය", "සංයෝග", "අණු", "පරමාණු", "අම්ල", "භෂ්ම", "ප්‍රතික්‍රියා"]), + "Biology": ("grades 12-13 (A/L)", ["ජීව විද්‍යා", "සෛල", "ජාන", "ප්‍රෝටීන", "එන්සයිම", "පරිණාම", "ප්‍රභාසංස්ලේෂණ", "ජීවීන්"]), + "Economics": ("grades 12-13 (A/L)", ["ආර්ථික", "ඉල්ලුම", "සැපයුම", "උද්ධමන", "දළ දේශීය", "බදු", "වෙළඳපොළ", "අපනයන"]), + "Business and Accounting Studies": ("grades 12-13 (A/L)", ["ව්‍යාපාර", "ගිණුම්", "ලාභ", "වත්කම්", "බැංකු", "කොටස්", "සමාගම", "ව්‍යවසාය"]), + "Agriculture and Food Technology": ("grades 9-13", ["කෘෂිකර්ම", "වගාව", "පොහොර", "බෝග", "කුඹුරු", "වී ", "පළිබෝධ", "ජලසම්පාදන"]), + "Home Economics": ("grades 9-13", ["ආහාර", "පෝෂණ", "ඇඳුම්", "නිවාස", "ගෘහ", "පවුල", "ආහාර පිසීම"]), + "Communication and Media Studies": ("grades 9-13", ["මාධ්‍ය", "ජනමාධ්‍ය", "පුවත්පත", "රූපවාහිනී", "ගුවන්විදුලි", "සන්නිවේදන"]), + "Design and Construction Technology": ("grades 9-11", ["ඉදිකිරීම්", "ගොඩනැගිලි", "කොන්ක්‍රීට්", "සැලසුම්", "ඉංජිනේරු", "ද්‍රව්‍ය"]), +} + + +def chunks(text: str, lo: int = 400, hi: int = 1200): + cur = "" + for para in re.split(r"\n\s*\n|\n", text): + para = para.strip() + if not para: + continue + if len(cur) + len(para) + 1 <= hi: + cur = f"{cur}\n{para}" if cur else para + else: + if len(cur) >= lo: + yield cur + cur = para[:hi] + if len(cur) >= lo: + yield cur + + +def assign(chunk: str) -> tuple[str | None, int]: + best, hits = None, 0 + for subj, (_, kws) in SUBJECTS.items(): + h = sum(kw in chunk for kw in kws) + if h > hits: + best, hits = subj, h + return (best, hits) if hits >= 2 else (None, hits) + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--per-subject", type=int, default=6) + ap.add_argument("--out", default=str(ROOT / "data" / "synth" / "passages_pilot.jsonl")) + ap.add_argument("--exclude", help="earlier pool file: its passages are not reused") + ap.add_argument("--seed", type=int, default=1234) + args = ap.parse_args() + import pandas as pd + + rng = random.Random(args.seed) + used = set() + if args.exclude and Path(args.exclude).exists(): + used = {json.loads(l)["pid"] for l in open(args.exclude, encoding="utf-8")} + by_subject: dict[str, list[dict]] = {s: [] for s in SUBJECTS} + for source in ("wikipedia", "fineweb2", "nsina"): + df = pd.read_parquet(ROOT / "data" / "corpus" / "cpt_v1" / f"{source}.parquet", columns=["id", "text"]) + docs = df.sample(frac=1, random_state=args.seed).itertuples() + for d in docs: + for k, c in enumerate(chunks(d.text)): + pid = f"{d.id}#{k}" + if pid in used: + continue + subj, hits = assign(c) + if subj and len(by_subject[subj]) < 50 * args.per_subject: + by_subject[subj].append({"pid": pid, "source": source, "subject": subj, + "level": SUBJECTS[subj][0], "keyword_hits": hits, "text": c}) + if all(len(v) >= 50 * args.per_subject for v in by_subject.values()): + break + out = [] + for subj, cands in by_subject.items(): + cands.sort(key=lambda r: (r["source"] != "wikipedia", -r["keyword_hits"], rng.random())) + out += cands[: args.per_subject] + write_jsonl(args.out, out) + print(f"{len(out)} passages -> {args.out}") + print(dict(Counter(r["subject"] for r in out))) + print("by source:", dict(Counter(r["source"] for r in out))) + + +if __name__ == "__main__": + main() diff --git a/training/scripts/build_rft2_questions.py b/training/scripts/build_rft2_questions.py new file mode 100644 index 0000000000000000000000000000000000000000..197c0e441057f092b630be72cc33a18d2de73147 --- /dev/null +++ b/training/scripts/build_rft2_questions.py @@ -0,0 +1,21 @@ +"""Question pool for the second round of reasoning-trace training: Global-MMLU `si` training questions (Humanities, Social +Sciences, Other, Business: closest to Medium's culture/civics/social subjects) plus the synthetic Wikipedia-based MCQs +(data/synth/pilot.jsonl) if present. Output: data/synth/rft2_questions.jsonl. The dev held-out questions are never included. +""" +from __future__ import annotations + +import random +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import ROOT, read_jsonl, write_jsonl # noqa: E402 + +N_GMMLU = int(sys.argv[1]) if len(sys.argv) > 1 else 2500 +g = [r for r in read_jsonl(ROOT / "data" / "sft" / "gmmlu_si_train.jsonl") + if r["category"] in ("Humanities", "Social Sciences", "Other", "Business")] +g = random.Random(1234).sample(g, min(N_GMMLU, len(g))) +syn_path = ROOT / "data" / "synth" / "pilot.jsonl" +syn = [{k: v for k, v in r.items() if k != "teacher_probs"} for r in read_jsonl(syn_path)] if syn_path.exists() else [] +write_jsonl(ROOT / "data" / "synth" / "rft2_questions.jsonl", g + syn) +print(f"{len(g)} Global-MMLU + {len(syn)} synthetic questions -> data/synth/rft2_questions.jsonl") diff --git a/training/scripts/build_sft_gmmlu.py b/training/scripts/build_sft_gmmlu.py new file mode 100644 index 0000000000000000000000000000000000000000..74786475a302c67c16b7b09f6c6d9e0b08a455e7 --- /dev/null +++ b/training/scripts/build_sft_gmmlu.py @@ -0,0 +1,122 @@ +"""Fine-tuning MCQs from Global-MMLU `si` (Apache-2.0) -> data/sft/gmmlu_si_train.jsonl. + + pool Global-MMLU si test split minus the 2,000 validation ids, plus its dev split + drop items with char-5-gram TF-IDF cosine >= --val-threshold to any Global-MMLU validation item + (MMLU repeats questions across subjects: 33 validation items had a >= 0.9 copy in the pool), + >= --dev-threshold to any SinhalaMMLU dev item, and near-copies within the pool (>= 0.95, first kept) + shuffle options to a deterministic, balanced gold position, except items whose options refer to other options + ("all of the above", "both", option letters/numbers), which keep their order + +Output records use the processed MCQ schema (answer = 1-based gold index). Aggregate counts go to +reports/data_stats_sft_gmmlu.json; nothing from the dev/validation sets is written to the output. +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import re +import sys +from collections import Counter +from pathlib import Path + +import numpy as np + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +sys.path.insert(0, str(Path(__file__).resolve().parent)) +from eval.common import ROOT, read_jsonl, write_jsonl # noqa: E402 +from overlap import item_text, ngram_matrix, tfidf # noqa: E402 +from prepare_data import _gmmlu_rows # noqa: E402 + +# Options that refer to other options must keep their order. +REFERS_TO_OTHERS = re.compile( + r"ඉහත|සියල්ල|කිසිවක්\s*නැත|දෙකම|සියල්ලම|all of the above|none of the above|both|neither|" + r"\b[A-D]\s*(and|හා|සහ|හෝ|or)\s*[A-D]\b|\b[1-4]\s*(and|හා|සහ|හෝ|or)\s*[1-4]\b", re.I) + + +def max_sim(a, b, chunk: int = 2000, earlier_only: bool = False) -> np.ndarray: + """For each row of a: max cosine to rows of b. earlier_only (a is b): only rows before it, so the first + occurrence of a duplicate group is kept and later copies are flagged.""" + best = np.zeros(a.shape[0]) + bt = b.T.tocsc() + for i in range(0, a.shape[0], chunk): + s = (a[i:i + chunk] @ bt).toarray() + if earlier_only: + rows = np.arange(i, i + s.shape[0]) + s[np.arange(s.shape[1])[None, :] >= rows[:, None]] = 0 + best[i:i + chunk] = s.max(1) + return best + + +def balanced_permutation(r: dict, n_items_seen: Counter) -> dict: + """Move the gold answer to the least-used position so far (ties broken by a stable hash).""" + n = len(r["options"]) + if any(REFERS_TO_OTHERS.search(o) for o in r["options"]): + return r | {"meta": r["meta"] | {"shuffled": False}} + h = int(hashlib.sha1(r["uid"].encode()).hexdigest(), 16) + target = min(range(n), key=lambda p: (n_items_seen[p], (p + h) % n)) + gold = r["answer"] - 1 + others = [j for j in range(n) if j != gold] + rot = h % len(others) + others = others[rot:] + others[:rot] + order = others[:target] + [gold] + others[target:] + n_items_seen[target] += 1 + return r | {"options": [r["options"][j] for j in order], "answer": target + 1, + "meta": r["meta"] | {"shuffled": True}} + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--val-threshold", type=float, default=0.6) + ap.add_argument("--dev-threshold", type=float, default=0.5) + ap.add_argument("--dup-threshold", type=float, default=0.95) + ap.add_argument("--out", default=str(ROOT / "data" / "sft" / "gmmlu_si_train.jsonl")) + args = ap.parse_args() + import pandas as pd + + raw = ROOT / "data" / "raw" / "global_mmlu" / "si" + val_ids = set((ROOT / "configs" / "splits" / "gmmlu_si_val_ids.txt").read_text().split()) + pool = [r for r in _gmmlu_rows(pd.read_parquet(raw / "test-00000-of-00001.parquet"), "test") if r["uid"] not in val_ids] + pool += _gmmlu_rows(pd.read_parquet(raw / "dev-00000-of-00001.parquet"), "dev") + val = read_jsonl(ROOT / "data" / "processed" / "gmmlu_si_val.jsonl") + dev = read_jsonl(ROOT / "data" / "processed" / "sinhalammlu_dev.jsonl") + + tp = [item_text(r, True) for r in pool] + mp, mv = tfidf(ngram_matrix(tp, 5), ngram_matrix([item_text(r, True) for r in val], 5)) + sim_val = max_sim(mp, mv) + mp2, md = tfidf(ngram_matrix(tp, 5), ngram_matrix([item_text(r, True) for r in dev], 5)) + sim_dev = max_sim(mp2, md) + sim_self = max_sim(mp, mp, earlier_only=True) + + keep, reasons = [], Counter() + for r, sv, sd, ss in zip(pool, sim_val, sim_dev, sim_self): + if sv >= args.val_threshold: + reasons["near_validation"] += 1 + elif sd >= args.dev_threshold: + reasons["near_sinhalammlu_dev"] += 1 + elif ss >= args.dup_threshold: + reasons["duplicate_within_pool"] += 1 + else: + keep.append(r) + + seen = Counter() + out = [balanced_permutation(r, seen) for r in sorted(keep, key=lambda r: r["uid"])] + out = [r | {"dataset": "gmmlu_si_train"} for r in out] + write_jsonl(args.out, out) + + stats = { + "pool": len(pool), "kept": len(out), "dropped": dict(reasons), + "thresholds": {"validation": args.val_threshold, "sinhalammlu_dev": args.dev_threshold, + "within_pool": args.dup_threshold}, + "max_similarity_kept": {"to_validation": float(max((sv for sv in sim_val if sv < args.val_threshold), default=0)), + "to_sinhalammlu_dev": float(sim_dev.max())}, + "answer_positions": dict(sorted(Counter(r["answer"] for r in out).items())), + "shuffled": sum(r["meta"]["shuffled"] for r in out), + "by_category": dict(Counter(r["category"] for r in out).most_common()), + } + (ROOT / "reports" / "data_stats_sft_gmmlu.json").write_text(json.dumps(stats, indent=2)) + print(json.dumps(stats, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/training/scripts/count_params.py b/training/scripts/count_params.py new file mode 100644 index 0000000000000000000000000000000000000000..cfcdbffe93208634fc182aadc5f3d347a56eb91b --- /dev/null +++ b/training/scripts/count_params.py @@ -0,0 +1,205 @@ +"""Exact parameter accounting for candidate models -> MODEL_MANIFEST.md + reports/param_counts.json. + +Two independent counts per model: + stored sum of every tensor in the repo's safetensors headers (read via HTTP range requests, + no weight download), split into text / vision / audio / MTP heads by tensor name. + Tied input/output embeddings are counted once. + instantiated parameters of the module transformers builds (meta device, no weights) with + AutoModelForCausalLM, falling back to AutoModelForImageTextToText. + +LoRA/adapter repos are counted as base model + adapter tensors (shared-task rule). +Usage: python scripts/count_params.py [--models id1 id2 ...] +""" +from __future__ import annotations + +import argparse +import json +import re +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import ROOT, load_env # noqa: E402 + +LIMIT = 8_000_000_000 +CUTOFF = "2026-10-01" +MANIFEST = ROOT / "MODEL_MANIFEST.md" +BEGIN, END = "", "" + +PATTERNS = [ + ("vision", re.compile(r"vision|visual|image|siglip|patch_embed|multi_modal_projector|mm_projector|embed_vision")), + ("audio", re.compile(r"audio|embed_audio|speech")), + ("mtp", re.compile(r"(^|\.)mtp[._]")), +] + + +def tensor_kind(name: str) -> str: + for kind, pat in PATTERNS: + if pat.search(name): + return kind + return "text" + + +def _tied(cfg: dict) -> bool: + for c in (cfg, cfg.get("text_config") or {}): + if "tie_word_embeddings" in c: + return bool(c["tie_word_embeddings"]) + return False + + +def stored_counts(api, repo: str, sha: str) -> dict: + from huggingface_hub import hf_hub_download + + meta = api.get_safetensors_metadata(repo, revision=sha) + counts = {"text": 0, "vision": 0, "audio": 0, "mtp": 0} + names = {} + for fm in meta.files_metadata.values(): + for name, t in fm.tensors.items(): + n = 1 + for d in t.shape: + n *= d + names[name] = n + counts[tensor_kind(name)] += n + cfg = json.load(open(hf_hub_download(repo, "config.json", revision=sha))) + tied = _tied(cfg) + lm_head = [k for k in names if k.endswith("lm_head.weight")] + deduped = 0 + if tied and lm_head: # stored twice but shared at runtime + deduped = sum(names[k] for k in lm_head) + counts["text"] -= deduped + emb = sum(v for k, v in names.items() if re.search(r"embed_tokens|wte|lm_head|embeddings\.word", k) + and tensor_kind(k) == "text") - deduped + return {**counts, "embeddings": emb, "tied_embeddings": tied, "total": sum(counts.values()), + "model_type": cfg.get("model_type"), "config": cfg} + + +def instantiated_count(repo: str, sha: str) -> tuple[int | None, str]: + import torch + from accelerate import init_empty_weights + from transformers import AutoConfig, AutoModelForCausalLM + + try: + cfg = AutoConfig.from_pretrained(repo, revision=sha) + except Exception as e: # noqa: BLE001 + return None, f"config: {type(e).__name__}" + for cls_name in ("AutoModelForCausalLM", "AutoModelForImageTextToText"): + try: + import transformers + + cls = getattr(transformers, cls_name) + with init_empty_weights(): + m = cls.from_config(cfg) + m.tie_weights() # meta init leaves tied embeddings as separate tensors + return sum(p.numel() for p in m.parameters()), type(m).__name__ + except Exception as e: # noqa: BLE001 + err = f"{cls_name}: {type(e).__name__}: {str(e)[:80]}" + return None, err + + +def count_repo(api, repo: str) -> dict: + info = api.model_info(repo) + sha = info.sha + row = {"id": repo, "sha": sha, "created": str(info.created_at)[:10], + "last_modified": str(info.last_modified)[:10], + "license": (info.card_data or {}).get("license") if info.card_data else None, + "gated": info.gated} + files = {s.rfilename for s in info.siblings} + if "adapter_config.json" in files: + from huggingface_hub import hf_hub_download + + acfg = json.load(open(hf_hub_download(repo, "adapter_config.json", revision=sha))) + base = acfg["base_model_name_or_path"] + fm = api.parse_safetensors_file_metadata(repo, "adapter_model.safetensors", revision=sha) + adapter = sum(_numel(t.shape) for t in fm.tensors.values()) + b = count_repo(api, base) + row.update(kind="adapter", base=base, adapter_params=adapter, + modules_to_save=acfg.get("modules_to_save"), + text=b.get("text"), vision=b.get("vision"), audio=b.get("audio"), mtp=b.get("mtp"), + embeddings=b.get("embeddings"), tied_embeddings=b.get("tied_embeddings"), + total=(b.get("total") or 0) + adapter, text_loaded=(b.get("text_loaded") or 0) + adapter, + instantiated=None, inst_class=f"base {b.get('total')} + adapter") + return row + s = stored_counts(api, repo, sha) + s.pop("config") + inst, cls = instantiated_count(repo, sha) + row.update(kind="model", **s, text_loaded=s["text"], instantiated=inst, inst_class=cls) + return row + + +def _numel(shape) -> int: + n = 1 + for d in shape: + n *= d + return n + + +def verdict(r: dict) -> str: + if r.get("error"): + return "ERROR" + flags = [] + if r["created"] >= CUTOFF: + flags.append("RELEASED TOO LATE") + if r["total"] <= LIMIT: + flags.append("OK") + elif r["text_loaded"] <= LIMIT: + flags.append("OK only text-only (strip vision/audio/MTP)") + else: + flags.append("OVER 8B") + return "; ".join(flags) + + +def b(n) -> str: + return "-" if n in (None, 0) else f"{n / 1e9:.3f}B" + + +def write_manifest(rows: list[dict]) -> None: + lines = [BEGIN, "", + "| Model | Verdict | Total (all stored) | Text model | Vision | Audio | MTP | Embeddings | Tied | " + "Instantiated (class) | Repo created | License | Commit |", + "|---|---|---:|---:|---:|---:|---:|---:|---|---|---|---|---|"] + for r in rows: + if r.get("error"): + lines.append(f"| `{r['id']}` | ERROR: {r['error']} |" + " |" * 11) + continue + inst = f"{b(r['instantiated'])} ({r['inst_class']})" if r.get("instantiated") else r.get("inst_class", "-") + name = f"`{r['id']}`" + (f"
adapter on `{r['base']}` (+{b(r['adapter_params'])})" if r["kind"] == "adapter" else "") + lines.append(f"| {name} | {verdict(r)} | {b(r['total'])} | {b(r['text_loaded'])} | {b(r['vision'])} | " + f"{b(r['audio'])} | {b(r['mtp'])} | {b(r['embeddings'])} | {'yes' if r['tied_embeddings'] else 'no'} | " + f"{inst} | {r['created']} | {r['license'] or '?'} | `{r['sha'][:10]}` |") + lines += ["", END] + text = MANIFEST.read_text() + start, stop = text.index(BEGIN), text.index(END) + len(END) + MANIFEST.write_text(text[:start] + "\n".join(lines) + text[stop:]) + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--models", nargs="*") + args = ap.parse_args() + load_env() + import yaml + from huggingface_hub import HfApi + + repos = args.models or [c["id"] for c in yaml.safe_load(open(ROOT / "configs" / "models.yaml"))["candidates"]] + api = HfApi() + rows = [] + for repo in repos: + try: + r = count_repo(api, repo) + except Exception as e: # noqa: BLE001 + r = {"id": repo, "error": f"{type(e).__name__}: {str(e)[:100]}"} + rows.append(r) + print(f"{repo:42s} {verdict(r):45s} total={b(r.get('total'))} text={b(r.get('text_loaded'))} " + f"inst={b(r.get('instantiated'))}") + out = ROOT / "reports" / "param_counts.json" + out.parent.mkdir(exist_ok=True) + if args.models and out.exists(): # partial recount: update those rows, keep the rest + fresh = {r["id"]: r for r in rows} + old = json.loads(out.read_text()) + rows = [fresh.pop(r["id"], r) for r in old] + list(fresh.values()) + out.write_text(json.dumps(rows, indent=2, default=str)) + write_manifest(rows) + + +if __name__ == "__main__": + main() diff --git a/training/scripts/extract_lora.py b/training/scripts/extract_lora.py new file mode 100644 index 0000000000000000000000000000000000000000..7b0ca04e862d0dfd7c9f2aa0aefebcadd6ea0ebb --- /dev/null +++ b/training/scripts/extract_lora.py @@ -0,0 +1,73 @@ +"""Recover a LoRA adapter from a merged LoRA model: for every weight that differs from the base model, +delta = merged - base (fp32) is factorised by SVD and truncated to rank r (LoRA training with rank r makes the delta +rank <= r, up to bf16 rounding). Saved in PEFT format with lora_alpha = r (scaling 1), so base + adapter = merged. + + python scripts/extract_lora.py --base models/gemma-4-E4B-it-text --merged models/sft_gmmlu_lora/step-357 \ + --rank 16 --out models/adapter_A +""" +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + +import torch +from safetensors.torch import load_file, save_file + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + + +def load_dir(d: Path) -> dict[str, torch.Tensor]: + out = {} + for f in sorted(d.glob("*.safetensors")): + out.update(load_file(f)) + return out + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--base", required=True, type=Path) + ap.add_argument("--merged", required=True, type=Path) + ap.add_argument("--rank", type=int, default=16) + ap.add_argument("--out", required=True, type=Path) + args = ap.parse_args() + base, merged = load_dir(args.base), load_dir(args.merged) + assert base.keys() == merged.keys(), "base and merged checkpoints have different tensors" + + tensors, targets, n_params, worst = {}, [], 0, 0.0 + for k in sorted(base): + if torch.equal(base[k], merged[k]): + continue + if base[k].dim() != 2 or not k.endswith(".weight"): + raise SystemExit(f"non-linear tensor changed ({k}); this is not a pure LoRA merge") + delta = merged[k].cuda().float() - base[k].cuda().float() + U, S, Vh = torch.linalg.svd(delta, full_matrices=False) + r = args.rank + B = (U[:, :r] * S[:r]).contiguous() # out x r + A = Vh[:r].contiguous() # r x in + err = ((delta - B @ A).norm() / delta.norm().clamp_min(1e-12)).item() + worst = max(worst, err) + module = k[: -len(".weight")] + tensors[f"base_model.model.{module}.lora_A.weight"] = A.to(torch.bfloat16).cpu() + tensors[f"base_model.model.{module}.lora_B.weight"] = B.to(torch.bfloat16).cpu() + targets.append(module) + n_params += A.numel() + B.numel() + + args.out.mkdir(parents=True, exist_ok=True) + save_file(tensors, args.out / "adapter_model.safetensors") + cfg = {"peft_type": "LORA", "task_type": "CAUSAL_LM", "r": args.rank, "lora_alpha": args.rank, + "lora_dropout": 0.0, "bias": "none", "target_modules": targets, "fan_in_fan_out": False, + "use_rslora": False, "modules_to_save": None, "base_model_name_or_path": str(args.base), + "inference_mode": True} + (args.out / "adapter_config.json").write_text(json.dumps(cfg, indent=2)) + (args.out / "SOURCE.json").write_text(json.dumps({"extracted_from": str(args.merged), "base": str(args.base), + "rank": args.rank, "modules": len(targets), + "adapter_params": n_params, + "max_relative_reconstruction_error": worst}, indent=2)) + print(f"{len(targets)} modules, {n_params / 1e6:.1f}M adapter parameters, " + f"max relative reconstruction error {worst:.4f} -> {args.out}") + + +if __name__ == "__main__": + main() diff --git a/training/scripts/fetch_data.py b/training/scripts/fetch_data.py new file mode 100644 index 0000000000000000000000000000000000000000..3b7f544d600e4582e7ed00f8a048ad4635a4f0f2 --- /dev/null +++ b/training/scripts/fetch_data.py @@ -0,0 +1,47 @@ +"""Download evaluation data into data/raw/ (git-ignored). + + SinhalaMMLU (public Easy tier) - dev set, CC BY-NC-ND 4.0, gated: accept terms on HF first + Global-MMLU `si` - harder internal validation (translated MMLU), Apache-2.0 + Belebele `sin_Sinh` - reading-comprehension sanity check, EVALUATION ONLY + +Usage: python scripts/fetch_data.py [--only sinhalammlu,global_mmlu,belebele] +""" +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import ROOT, load_env # noqa: E402 + +RAW = ROOT / "data" / "raw" + +SOURCES = { + "sinhalammlu": dict(repo_id="naist-nlp/SinhalaMMLU", allow_patterns=["README.md", "TEST/**/*.json"]), + "global_mmlu": dict(repo_id="CohereLabs/Global-MMLU", allow_patterns=["si/*.parquet"]), + "belebele": dict(repo_id="facebook/belebele", allow_patterns=["data/sin_Sinh.jsonl"]), +} + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--only", default=",".join(SOURCES)) + args = ap.parse_args() + load_env() + from huggingface_hub import HfApi, snapshot_download + + api = HfApi() + for name in args.only.split(","): + src = SOURCES[name] + sha = api.dataset_info(src["repo_id"]).sha + path = snapshot_download( + repo_id=src["repo_id"], repo_type="dataset", revision=sha, + allow_patterns=src["allow_patterns"], local_dir=RAW / name, + ) + (RAW / name / "REVISION").write_text(f"{src['repo_id']}@{sha}\n") + print(f"{name}: {src['repo_id']}@{sha[:10]} -> {path}") + + +if __name__ == "__main__": + main() diff --git a/training/scripts/gen_traces.py b/training/scripts/gen_traces.py new file mode 100644 index 0000000000000000000000000000000000000000..0b5df9127a8b57571b43794ebc319c81d01afcc8 --- /dev/null +++ b/training/scripts/gen_traces.py @@ -0,0 +1,80 @@ +"""Rejection-sampled reasoning traces from the base model itself (STaR-style), for SinhalaMMLU dev questions. + +For every question the base model answers with thinking mode and temperature sampling; traces whose final answer equals +the gold key are kept. Output: .all.jsonl (every sample) and .jsonl (correct ones, one record per kept sample: +the question item + "completion" = raw model output including the thought channel and the end-of-turn token). + +Time-boxed and resumable: samples are generated in rounds (all questions once, then again), results are appended after +every chunk, and the run stops when --max-minutes is reached. + + python scripts/gen_traces.py --data smmlu_dev_rest1450 --out data/synth/rft_traces --samples 2 --max-minutes 240 +""" +from __future__ import annotations + +import argparse +import json +import sys +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import cap_gpu_memory, load_env, read_jsonl, set_seed # noqa: E402 +from eval.run_eval import resolve_data # noqa: E402 + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--data", required=True) + ap.add_argument("--out", required=True) + ap.add_argument("--model", default="models/gemma-4-E4B-it-text") + ap.add_argument("--samples", type=int, default=2) + ap.add_argument("--temperature", type=float, default=0.8) + ap.add_argument("--max-new-tokens", type=int, default=3072) + ap.add_argument("--chunk", type=int, default=128) + ap.add_argument("--max-batch", type=int, default=64, help="sequences per generate call (lower it on a smaller GPU)") + ap.add_argument("--max-minutes", type=float, default=240) + ap.add_argument("--limit", type=int) + ap.add_argument("--seed", type=int, default=1234) + args = ap.parse_args() + load_env() + set_seed(args.seed) + cap_gpu_memory() + from eval.prompts import PromptBuilder + from eval.scorers import Scorer, load_model + + items = read_jsonl(resolve_data(args.data))[: args.limit] + all_path, ok_path = Path(f"{args.out}.all.jsonl"), Path(f"{args.out}.jsonl") + all_path.parent.mkdir(parents=True, exist_ok=True) + done = {(r["uid"], r["k"]) for r in read_jsonl(all_path)} if all_path.exists() else set() + jobs = [(it, k) for k in range(args.samples) for it in items if (it["uid"], k) not in done] + print(f"{len(items)} questions x {args.samples} samples; {len(done)} already done, {len(jobs)} to go", flush=True) + + model, tok = load_model(args.model) + pb = PromptBuilder(tok, lang="en", chat=True, chat_kwargs={"enable_thinking": True}) + scorer = Scorer(model, tok, pb.add_special_tokens, token_budget=32768, max_batch=args.max_batch, max_new_tokens=args.max_new_tokens) + scorer.keep_raw = True + scorer.sampling = {"do_sample": True, "temperature": args.temperature, "top_p": 0.95, "top_k": 64} + + t0 = time.time() + n_ok = n_all = 0 + for i in range(0, len(jobs), args.chunk): + if (time.time() - t0) / 60 > args.max_minutes: + print(f"time box of {args.max_minutes} min reached after {n_all} samples", flush=True) + break + chunk = jobs[i:i + args.chunk] + res = scorer.generate([pb.mcq(it, it["options"]) for it, _ in chunk], [len(it["options"]) for it, _ in chunk]) + with open(all_path, "a", encoding="utf-8") as fa, open(ok_path, "a", encoding="utf-8") as fo: + for (it, k), r in zip(chunk, res): + rec = {"uid": it["uid"], "k": k, "pred": r["pred"], "gold": it["answer"], "new_tokens": r["new_tokens"], + "correct": r["pred"] == it["answer"]} + fa.write(json.dumps(rec, ensure_ascii=False) + "\n") + n_all += 1 + if rec["correct"] and r["raw"].endswith("") and "<|channel>" in r["raw"]: + fo.write(json.dumps(it | {"completion": r["raw"], "k": k}, ensure_ascii=False) + "\n") + n_ok += 1 + print(f"{n_all}/{len(jobs)} samples, {n_ok} kept ({100 * n_ok / n_all:.0f}%), {(time.time() - t0) / 60:.0f} min", flush=True) + print(f"done: {n_all} samples, {n_ok} kept traces -> {ok_path}", flush=True) + + +if __name__ == "__main__": + main() diff --git a/training/scripts/generate_mcq.py b/training/scripts/generate_mcq.py new file mode 100644 index 0000000000000000000000000000000000000000..7e91404bb487487021050cd6ce3f591e95c710c5 --- /dev/null +++ b/training/scripts/generate_mcq.py @@ -0,0 +1,174 @@ +"""Synthetic Sinhala MCQs from passages with an open teacher model (data generation only, never used at inference). + +For each passage the teacher writes k exam-style Sinhala MCQs (JSON). Every question is then + 1. validated: 4 distinct non-empty options, key 1-4, mostly Sinhala, does not refer to "the passage"; + 2. round-trip checked: the teacher re-answers it WITH the passage (logit scoring) and must agree with its own key; + 3. soft-labelled: the teacher's closed-book option probabilities (same prompt as the student sees) are stored. +Kept questions go to .jsonl in the processed MCQ schema (+ teacher_probs, passage id, subject); rejected ones to +.rejected.jsonl with the reason; 50 random kept questions to .review.md for a native-speaker check. + + python scripts/generate_mcq.py --passages data/synth/passages_pilot.jsonl --out data/synth/pilot \ + --teacher google/gemma-4-31B-it --per-passage 3 +""" +from __future__ import annotations + +import argparse +import json +import random +import re +import sys +import time +from collections import Counter +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import ROOT, cap_gpu_memory, load_env, read_jsonl, set_seed, write_jsonl # noqa: E402 + +PROMPT = """You are writing exam questions for Sri Lankan school students. +Subject: {subject}. Level: {level}. + +Read the Sinhala passage below. Write {k} multiple-choice questions IN SINHALA that test important facts or concepts +stated in the passage, in the style of Sri Lankan government exam papers (term tests, O/L, A/L). + +Rules: +- Each question must make sense on its own, without the passage. Never write "according to the passage" or refer to + "the passage", "the text" or "the article". +- Exactly 4 options and exactly one correct answer, which must be supported by the passage. +- Wrong options must be plausible, on the same topic, and similar in length to the correct one. +- Vary the format: direct questions, sentence completion ("...... වේ."), and when possible one question asking which + statement is NOT correct. +- Use standard Sinhala school-curriculum terminology. No English words unless the term is normally written in English. + +Return only a JSON list, nothing else: +[{{"question": "...", "options": ["...", "...", "...", "..."], "answer": }}] + +Passage: +{passage}""" + +REFERS_TO_PASSAGE = re.compile(r"ඡේද|ඡේදය|පාඨය|ලිපිය|ඉහත සඳහන්|ඉහත කොටස|passage|text|article", re.I) +SINHALA = re.compile(r"[඀-෿]") +LETTER = re.compile(r"[^\W\d_]") + + +def parse(text: str) -> list[dict]: + m = re.search(r"\[.*\]", text, re.S) + if not m: + return [] + try: + data = json.loads(m.group(0)) + except json.JSONDecodeError: + return [] + return [d for d in data if isinstance(d, dict)] + + +def validate(q: dict) -> str | None: + opts = q.get("options") + if not isinstance(q.get("question"), str) or not isinstance(opts, list) or len(opts) != 4: + return "format" + opts = [str(o).strip() for o in opts] + if any(not o for o in opts) or len(set(opts)) < 4: + return "options" + try: + a = int(q.get("answer")) + except (TypeError, ValueError): + return "answer" + if not 1 <= a <= 4: + return "answer" + text = q["question"] + " " + " ".join(opts) + if len(q["question"].strip()) < 10: + return "too_short" + if REFERS_TO_PASSAGE.search(q["question"]): + return "refers_to_passage" + letters = LETTER.findall(text) + if not letters or len(SINHALA.findall(text)) / len(letters) < 0.5: + return "not_sinhala" + return None + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--passages", required=True) + ap.add_argument("--out", required=True, help="output prefix") + ap.add_argument("--teacher", default="google/gemma-4-31B-it") + ap.add_argument("--per-passage", type=int, default=3) + ap.add_argument("--max-new-tokens", type=int, default=1200) + ap.add_argument("--gen-batch", type=int, default=16) + ap.add_argument("--seed", type=int, default=1234) + args = ap.parse_args() + load_env() + set_seed(args.seed) + cap_gpu_memory() + from eval.prompts import PromptBuilder, user_text + from eval.scorers import Scorer, load_model + + passages = read_jsonl(args.passages) + model, tok = load_model(args.teacher) + t0 = time.time() + + # 1. generation (greedy, thinking off) + prompts = [tok.apply_chat_template([{"role": "user", "content": PROMPT.format( + subject=p["subject"], level=p["level"], k=args.per_passage, passage=p["text"])}], + tokenize=False, add_generation_prompt=True, enable_thinking=False) for p in passages] + gen = Scorer(model, tok, add_special_tokens=False, token_budget=10**9, max_batch=args.gen_batch, + max_new_tokens=args.max_new_tokens) + raw = gen.generate(prompts, [4] * len(prompts)) + print(f"generated for {len(passages)} passages in {(time.time() - t0) / 60:.1f} min", flush=True) + + # 2. validation + items, rejected = [], [] + for p, r in zip(passages, raw): + qs = parse(r["text"]) + if not qs: + rejected.append({"pid": p["pid"], "reason": "unparseable", "text": r["text"][:500]}) + for j, q in enumerate(qs[: args.per_passage]): + why = validate(q) + if why: + rejected.append({"pid": p["pid"], "reason": why, "q": q}) + continue + items.append({"uid": f"synth/{p['pid']}/{j}", "dataset": "synthetic", "q_no": j, "subject": p["subject"], + "category": "synthetic", "level": p["level"], "question": q["question"].strip(), + "options": [str(o).strip() for o in q["options"]], "answer": int(q["answer"]), + "meta": {"pid": p["pid"], "source": p["source"], "teacher": args.teacher}}) + + # 3. round trip (open book) and 4. soft labels (closed book), both logit scoring with the teacher + by_pid = {p["pid"]: p for p in passages} + pb = PromptBuilder(tok, lang="en", chat=True) + scorer = Scorer(model, tok, add_special_tokens=False, token_budget=16384, max_batch=32) + open_prompts = [tok.apply_chat_template([{"role": "user", "content": + f"{by_pid[it['meta']['pid']]['text']}\n\n{user_text(it, it['options'], 'en')}"}], + tokenize=False, add_generation_prompt=True, enable_thinking=False) for it in items] + ob = scorer.logit(open_prompts, [4] * len(items)) + cb = scorer.logit([pb.mcq(it, it["options"]) for it in items], [4] * len(items)) + kept = [] + for it, o, c in zip(items, ob, cb): + it["meta"] |= {"open_book_pred": o["pred"], "open_book_conf": round(max(o["probs"]), 4), + "closed_book_pred": c["pred"]} + if o["pred"] != it["answer"]: + rejected.append({"pid": it["meta"]["pid"], "reason": "round_trip_disagrees", "q": it}) + continue + kept.append(it | {"teacher_probs": [round(x, 5) for x in c["probs"]]}) + + write_jsonl(f"{args.out}.jsonl", kept) + write_jsonl(f"{args.out}.rejected.jsonl", rejected) + stats = {"passages": len(passages), "generated_valid": len(items), "kept": len(kept), + "rejected": dict(Counter(r["reason"] for r in rejected)), + "answer_positions": dict(Counter(it["answer"] for it in kept)), + "teacher_closed_book_agrees_with_key": round(sum(it["meta"]["closed_book_pred"] == it["answer"] for it in kept) / max(len(kept), 1), 3), + "minutes": round((time.time() - t0) / 60, 1)} + Path(f"{args.out}.stats.json").write_text(json.dumps(stats, indent=2)) + print(json.dumps(stats, indent=2)) + + rng = random.Random(args.seed) + lines = [f"# Review: {min(50, len(kept))} random synthetic MCQs ({args.teacher})", "", + "For each question mark: OK / wrong answer / unnatural Sinhala / ambiguous / not exam-like.", ""] + for n, it in enumerate(rng.sample(kept, min(50, len(kept))), 1): + p = by_pid[it["meta"]["pid"]] + lines += [f"## {n}. {it['subject']} ({it['level']})", "", f"**Q:** {it['question']}", ""] + lines += [f"{k}. {o}{' ✅' if k == it['answer'] else ''}" for k, o in enumerate(it["options"], 1)] + lines += ["", f"
passage\n\n{p['text'][:800]}\n\n
", ""] + Path(f"{args.out}.review.md").write_text("\n".join(lines), encoding="utf-8") + print(f"review file -> {args.out}.review.md") + + +if __name__ == "__main__": + main() diff --git a/training/scripts/make_dev_folds.py b/training/scripts/make_dev_folds.py new file mode 100644 index 0000000000000000000000000000000000000000..3966e0e4ef9ebfc5a05b14ca437ead45d29eb05a --- /dev/null +++ b/training/scripts/make_dev_folds.py @@ -0,0 +1,46 @@ +"""5-fold split of SinhalaMMLU dev, stratified by subject (fixed seed) -> data/processed/smmlu_cv5_f{k}_{train,test}.jsonl. + +Each question is in exactly one test fold, so a model trained on the other four folds is always scored on questions +it never saw. Fold membership (uids only) is frozen in configs/splits/sinhalammlu_dev_cv5.json. +""" +from __future__ import annotations + +import json +import random +import sys +from collections import defaultdict +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import ROOT, read_jsonl, write_jsonl # noqa: E402 + +K, SEED = 5, 1234 + + +def main() -> None: + rows = read_jsonl(ROOT / "data" / "processed" / "sinhalammlu_dev.jsonl") + split_file = ROOT / "configs" / "splits" / "sinhalammlu_dev_cv5.json" + if split_file.exists(): + folds = json.loads(split_file.read_text()) + else: + by_subject = defaultdict(list) + for r in rows: + by_subject[r["subject"]].append(r["uid"]) + rng, folds, k = random.Random(SEED), [[] for _ in range(K)], 0 + for subject in sorted(by_subject): + uids = sorted(by_subject[subject]) + rng.shuffle(uids) + for u in uids: + folds[k % K].append(u) + k += 1 + split_file.write_text(json.dumps(folds, indent=1)) + for f, test_uids in enumerate(folds): + test = set(test_uids) + write_jsonl(ROOT / "data" / "processed" / f"smmlu_cv5_f{f}_test.jsonl", [r for r in rows if r["uid"] in test]) + write_jsonl(ROOT / "data" / "processed" / f"smmlu_cv5_f{f}_train.jsonl", + [r | {"dataset": "sinhalammlu_train"} for r in rows if r["uid"] not in test]) + print(f"fold {f}: train {len(rows) - len(test)}, test {len(test)}") + + +if __name__ == "__main__": + main() diff --git a/training/scripts/make_text_only.py b/training/scripts/make_text_only.py new file mode 100644 index 0000000000000000000000000000000000000000..8c245d1c8f4fa908d16a85e20cdc8b83d96fb064 --- /dev/null +++ b/training/scripts/make_text_only.py @@ -0,0 +1,75 @@ +"""Extract the text-only language model from a multimodal checkpoint (vision/audio towers dropped). + +The shared task counts every component that is loaded, so the submission loads this checkpoint instead of the +multimodal one. Weights are copied unchanged; the script verifies the copy is exact. + +Usage: python scripts/make_text_only.py google/gemma-4-E4B-it [--out models/gemma-4-E4B-it-text] +""" +from __future__ import annotations + +import argparse +import json +import shutil +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import ROOT, load_env # noqa: E402 + +# multimodal class -> (text-only class, attribute path of the language model inside it) +TEXT_ONLY = { + "Gemma4ForConditionalGeneration": ("Gemma4ForCausalLM", "model.language_model"), + "Gemma3ForConditionalGeneration": ("Gemma3ForCausalLM", "model.language_model"), + "Gemma3nForConditionalGeneration": ("Gemma3nForCausalLM", "model.language_model"), +} + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("repo") + ap.add_argument("--out") + ap.add_argument("--revision", help="full commit sha of the source repo (default: latest)") + args = ap.parse_args() + load_env() + import torch + import transformers + from huggingface_hub import HfApi, snapshot_download + from transformers import AutoConfig, AutoModelForImageTextToText, AutoTokenizer + + sha = HfApi().model_info(args.repo, revision=args.revision).sha + out = Path(args.out or ROOT / "models" / f"{args.repo.split('/')[-1]}-text") + cfg = AutoConfig.from_pretrained(args.repo, revision=sha) + full = AutoModelForImageTextToText.from_pretrained(args.repo, revision=sha, dtype=torch.bfloat16, device_map="cpu") + text_cls_name, lm_path = TEXT_ONLY[type(full).__name__] + lm = full + for attr in lm_path.split("."): + lm = getattr(lm, attr) + + text = getattr(transformers, text_cls_name)(cfg.text_config).to(torch.bfloat16) + text.model.load_state_dict(lm.state_dict(), strict=True) + text.tie_weights() + if not getattr(cfg.text_config, "tie_word_embeddings", True) and hasattr(full, "lm_head"): + text.lm_head.load_state_dict(full.lm_head.state_dict()) + text.generation_config = full.generation_config + + # Exactness check: every text tensor equals its source + src = lm.state_dict() + for k, v in text.model.state_dict().items(): + assert torch.equal(v, src[k]), f"mismatch in {k}" + n_params = sum(p.numel() for p in text.parameters()) + + out.mkdir(parents=True, exist_ok=True) + text.save_pretrained(out, safe_serialization=True) + AutoTokenizer.from_pretrained(args.repo, revision=sha).save_pretrained(out) + snap = Path(snapshot_download(args.repo, revision=sha, allow_patterns=["chat_template.jinja", "chat_template.json"])) + for f in snap.glob("chat_template.*"): + shutil.copy(f, out / f.name) + (out / "SOURCE.json").write_text(json.dumps({ + "source_repo": args.repo, "source_commit": sha, "class": text_cls_name, + "total_params": n_params, "dropped": [n for n, _ in full.model.named_children() if n != "language_model"], + }, indent=2)) + print(f"{args.repo}@{sha[:10]} -> {out} ({text_cls_name}, {n_params / 1e9:.3f}B parameters)") + + +if __name__ == "__main__": + main() diff --git a/training/scripts/merge_predictions.py b/training/scripts/merge_predictions.py new file mode 100644 index 0000000000000000000000000000000000000000..786533405ae8f7290905f03cb57d42af4d89d487 --- /dev/null +++ b/training/scripts/merge_predictions.py @@ -0,0 +1,43 @@ +"""Replace the answers of some levels in a base predictions.zip with those of another predictions.zip (e.g. a run made with +`inference/predict.py --levels medium`). Rows are matched on (level, subject, q_no); everything else is kept as in the base. + + python scripts/merge_predictions.py --base results/submissions/B.zip --new results/submissions/medium.zip \ + --out results/submissions/merged.zip +""" +from __future__ import annotations + +import argparse +import json +import zipfile + + +def load(p: str) -> list[dict]: + with zipfile.ZipFile(p) as z: + return [json.loads(l) for l in z.read("predictions.jsonl").decode("utf-8").splitlines()] + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--base", required=True) + ap.add_argument("--new", required=True) + ap.add_argument("--out", required=True) + args = ap.parse_args() + base, new = load(args.base), load(args.new) + key = lambda r: (r["level"], r["subject"], r["q_no"]) # noqa: E731 + new_by = {key(r): r for r in new} + levels = sorted({r["level"] for r in new}) + out, replaced, changed = [], 0, 0 + for r in base: + if r["level"] in levels and key(r) in new_by: + replaced += 1 + changed += new_by[key(r)]["answer"] != r["answer"] + r = new_by[key(r)] + out.append(r) + assert replaced == len(new), f"{len(new) - replaced} new rows had no match in the base file" + with zipfile.ZipFile(args.out, "w", zipfile.ZIP_DEFLATED) as z: + z.writestr("predictions.jsonl", "".join(json.dumps(r, ensure_ascii=False) + "\n" for r in out)) + print(f"levels {levels}: {replaced} rows replaced, {changed} answers differ from the base file -> {args.out}") + + +if __name__ == "__main__": + main() diff --git a/training/scripts/overlap.py b/training/scripts/overlap.py new file mode 100644 index 0000000000000000000000000000000000000000..7745020d87a7de5f7edf09c64fbeedf687a19cf1 --- /dev/null +++ b/training/scripts/overlap.py @@ -0,0 +1,105 @@ +"""Near-duplicate check between two MCQ (or text) sets: TF-IDF over character n-grams, cosine similarity. + +Catches exact copies, reorderings and light edits; translation of the same question into different Sinhala +wording is only partly caught. Used for contamination checks of every training set against dev/validation. + + python scripts/overlap.py --a gmmlu_si_train --b sinhalammlu_dev [--threshold 0.8] [--show 5] + +--a / --b are processed dataset names (data/processed/.jsonl) or JSONL paths with "question" (+ "options") +or "text" fields. Prints the similarity distribution of each A item's nearest B item and the pairs above threshold. +""" +from __future__ import annotations + +import argparse +import re +import sys +import unicodedata +from pathlib import Path + +import numpy as np +import scipy.sparse as sp + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import read_jsonl # noqa: E402 +from eval.run_eval import resolve_data # noqa: E402 + + +def norm(text: str) -> str: + text = unicodedata.normalize("NFC", text).lower() + text = re.sub(r"[\s‌‍]+", " ", text) # whitespace, ZWNJ/ZWJ (common in Sinhala) + # keep the whole Sinhala block: vowel signs and virama are combining marks, which \w does not match + return re.sub(r"[^\w \u0D80-\u0DFF]", "", text).strip() + + +def item_text(r: dict, with_options: bool) -> str: + if "text" in r: + return r["text"] + return r["question"] + (" " + " ".join(r["options"]) if with_options else "") + + +def ngram_matrix(texts: list[str], n: int, dim: int = 2**21) -> sp.csr_matrix: + rows, cols, vals = [], [], [] + for i, t in enumerate(texts): + t = f" {norm(t)} " + grams = {} + for j in range(len(t) - n + 1): + h = hash(t[j:j + n]) % dim + grams[h] = grams.get(h, 0) + 1 + rows += [i] * len(grams) + cols += list(grams) + vals += list(grams.values()) + m = sp.csr_matrix((np.array(vals, dtype=np.float32), (rows, cols)), shape=(len(texts), dim)) + return m + + +def tfidf(a: sp.csr_matrix, b: sp.csr_matrix) -> tuple[sp.csr_matrix, sp.csr_matrix]: + df = np.asarray(((a > 0).sum(0) + (b > 0).sum(0))).ravel() + idf = np.log((a.shape[0] + b.shape[0] + 1) / (df + 1)) + 1 + out = [] + for m in (a, b): + m = m.multiply(idf).tocsr() + m.data = np.log1p(m.data) + norms = np.sqrt(np.asarray(m.multiply(m).sum(1)).ravel()) + 1e-9 + out.append(sp.diags(1 / norms) @ m) + return out[0], out[1] + + +def nearest(a: sp.csr_matrix, b: sp.csr_matrix, chunk: int = 2000) -> tuple[np.ndarray, np.ndarray]: + best, idx = np.zeros(a.shape[0]), np.zeros(a.shape[0], dtype=int) + bt = b.T.tocsc() + for i in range(0, a.shape[0], chunk): + s = (a[i:i + chunk] @ bt).toarray() + idx[i:i + chunk] = s.argmax(1) + best[i:i + chunk] = s.max(1) + return best, idx + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--a", required=True) + ap.add_argument("--b", required=True) + ap.add_argument("--n", type=int, default=5, help="character n-gram size") + ap.add_argument("--threshold", type=float, default=0.8) + ap.add_argument("--no-options", action="store_true", help="compare question stems only") + ap.add_argument("--show", type=int, default=5) + args = ap.parse_args() + + A, B = read_jsonl(resolve_data(args.a)), read_jsonl(resolve_data(args.b)) + ta = [item_text(r, not args.no_options) for r in A] + tb = [item_text(r, not args.no_options) for r in B] + ma, mb = tfidf(ngram_matrix(ta, args.n), ngram_matrix(tb, args.n)) + best, idx = nearest(ma, mb) + + print(f"A={args.a} ({len(A)}) B={args.b} ({len(B)}) char {args.n}-grams, " + f"{'stems only' if args.no_options else 'question + options'}") + for q in (50, 90, 99, 99.9, 100): + print(f" nearest-neighbour similarity, {q:>5}th percentile: {np.percentile(best, q):.3f}") + for t in (0.5, 0.6, 0.7, 0.8, 0.9): + print(f" A items with a B neighbour >= {t}: {(best >= t).sum()}") + order = np.argsort(-best) + for i in order[: args.show]: + print(f"\n sim={best[i]:.3f}\n A {A[i].get('uid', i)}: {ta[i][:160]}\n B {B[idx[i]].get('uid', idx[i])}: {tb[idx[i]][:160]}") + + +if __name__ == "__main__": + main() diff --git a/training/scripts/pool_ballots.py b/training/scripts/pool_ballots.py new file mode 100644 index 0000000000000000000000000000000000000000..f6b841bfbd0246b30aa5794f2e1715a16c4bb10b --- /dev/null +++ b/training/scripts/pool_ballots.py @@ -0,0 +1,61 @@ +"""Cross-model pooling: majority vote over the thinking answers of several runs (each .ballots.jsonl, +written by inference/predict.py), merged into a base predictions.zip for the questions the ballot files cover. + +Ballots are concatenated in the order given; unparsed answers abstain; ties go to the first ballot (the first run's greedy +answer) if it is among the winners, else to the lowest label (the same rule as inference/predict.py). + + python scripts/pool_ballots.py --base results/submissions/final_candidate_base_think_gated_sc.zip \ + --ballots results/submissions/pool_base_medium.zip.ballots.jsonl results/submissions/pool_rft2_medium.zip.ballots.jsonl \ + --out results/submissions/final_candidate_B_plus_pool_medium.zip +""" +from __future__ import annotations + +import argparse +import json +import zipfile +from collections import Counter +from pathlib import Path + + +def vote(b: list): + c = Counter(x for x in b if x) + if not c: + return None + top = max(c.values()) + w = [a for a, n in c.items() if n == top] + return b[0] if b[0] in w else min(w) + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--base", required=True, type=Path) + ap.add_argument("--ballots", required=True, nargs="+", type=Path) + ap.add_argument("--out", required=True, type=Path) + args = ap.parse_args() + runs = [] + for f in args.ballots: + rows = [json.loads(l) for l in f.read_text(encoding="utf-8").splitlines() if l.strip()] + runs.append({(r["level"], r["subject"], r["q_no"]): r["ballots"] for r in rows}) + keys = set(runs[0]) + assert all(set(r) == keys for r in runs), "ballot files cover different questions" + pooled = {k: vote([b for r in runs for b in r[k]]) for k in keys} + with zipfile.ZipFile(args.base) as z: + base = [json.loads(l) for l in z.read("predictions.jsonl").decode("utf-8").splitlines() if l.strip()] + changed = matched = 0 + for r in base: + k = (r["level"], r["subject"], r["q_no"]) + if k in pooled and pooled[k]: + matched += 1 + changed += str(pooled[k]) != r["answer"] + r["answer"] = str(pooled[k]) + assert matched == sum(1 for v in pooled.values() if v), "some pooled questions are missing from the base file" + args.out.parent.mkdir(parents=True, exist_ok=True) + with zipfile.ZipFile(args.out, "w", zipfile.ZIP_DEFLATED) as z: + z.writestr("predictions.jsonl", "".join(json.dumps(r, ensure_ascii=False) + "\n" for r in base)) + n_b = Counter(len([b for r in runs for b in r[k]]) for k in keys) + print(f"{len(keys)} questions pooled over {len(runs)} runs (ballots per question: {dict(n_b)}); " + f"{changed} answers differ from {args.base.name} -> {args.out}") + + +if __name__ == "__main__": + main() diff --git a/training/scripts/prepare_data.py b/training/scripts/prepare_data.py new file mode 100644 index 0000000000000000000000000000000000000000..4deb8b1ba15ed628aad1ac3516fd273c329d0e23 --- /dev/null +++ b/training/scripts/prepare_data.py @@ -0,0 +1,152 @@ +"""Normalise raw eval data into one MCQ schema under data/processed/. + +Schema (one JSON object per line): + uid, dataset, q_no, subject, category, level, question, options[list[str]], answer (1-based int), meta{} + +Outputs: + sinhalammlu_dev.jsonl public Easy tier (dev) + gmmlu_si_val.jsonl fixed held-out sample of Global-MMLU si test (harder internal validation); + ids in configs/splits/ so they are excluded from any later training mix + sinhalammlu_dev_5shot.jsonl fixed 5 few-shot examples drawn from the dev pool (ids frozen in configs/splits) + gmmlu_si_dev.jsonl Global-MMLU si dev split (5/subject), candidate pool for fixed few-shot examples + belebele_si.jsonl Belebele sin_Sinh, EVALUATION ONLY +""" +from __future__ import annotations + +import argparse +import json +import random +import sys +from collections import Counter, defaultdict +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import ROOT, read_jsonl, write_jsonl # noqa: E402 + +RAW = ROOT / "data" / "raw" +OUT = ROOT / "data" / "processed" +SPLITS = ROOT / "configs" / "splits" +LETTERS = "ABCD" + + +def prep_sinhalammlu() -> list[dict]: + rows, dropped = [], [] + for f in sorted((RAW / "sinhalammlu" / "TEST").rglob("*.json")): + recs = json.loads(f.read_text(encoding="utf-8")) + # A few records spell their subject differently from the rest of the file + # ("Eastern_music" vs "Eastern music", trailing spaces): group by the file's majority spelling, + # keep the original in subject_raw because Codabench predictions must echo it. + canonical = Counter(r["subject"].strip() for r in recs).most_common(1)[0][0] + for r in recs: + level = r["metadata"].get("difficulty", f.parent.name) + uid = f"smmlu/{level}/{r['subject']}/{r['q_no']}" + if not (isinstance(r["answer"], int) and 1 <= r["answer"] <= len(r["choices"])): + dropped.append((uid, r["answer"])) + continue + rows.append(dict( + uid=uid, dataset="sinhalammlu", q_no=r["q_no"], subject=canonical, subject_raw=r["subject"], + category=r["category"], level=level, question=r["question"].strip(), + options=[str(c).strip() for c in r["choices"]], answer=r["answer"], + meta={k: r["metadata"].get(k) for k in ("grade", "year", "type", "subject_original")}, + )) + for uid, a in dropped: + print(f" dropped {uid}: invalid answer {a!r}") + return rows + + +def select_fewshot(rows: list[dict], k: int, seed: int) -> list[dict]: + """Fixed few-shot set from the dev pool: one question per domain, gold answers spread over 1..4. + Chosen at random (not by inspection) and frozen in configs/splits; excluded from evaluation by the harness.""" + ids_file = SPLITS / f"sinhalammlu_dev_{k}shot_ids.txt" + by_uid = {r["uid"]: r for r in rows} + if ids_file.exists(): + return [by_uid[u] for u in ids_file.read_text().split()] + rng = random.Random(seed) + pool = sorted(rows, key=lambda r: r["uid"]) + rng.shuffle(pool) + picked, answers = [], [] + for cat in sorted({r["category"] for r in rows}): + r = next(r for r in pool if r["category"] == cat and r["answer"] not in answers) + picked.append(r) + answers.append(r["answer"]) + for r in pool: + if len(picked) >= k: + break + if r not in picked and (len(set(answers)) == 4 or r["answer"] not in answers): + picked.append(r) + answers.append(r["answer"]) + rng.shuffle(picked) + ids_file.write_text("\n".join(r["uid"] for r in picked) + "\n") + return picked + + +def _gmmlu_rows(df, split: str) -> list[dict]: + rows = [] + for r in df.itertuples(): + rows.append(dict( + uid=f"gmmlu_si/{split}/{r.sample_id}", dataset="gmmlu_si", q_no=r.sample_id, + subject=r.subject, category=r.subject_category, level="gmmlu", + question=r.question.strip(), + options=[str(getattr(r, f"option_{c}")).strip() for c in "abcd"], + answer=LETTERS.index(r.answer) + 1, + meta={"cultural_sensitivity": r.cultural_sensitivity_label}, + )) + return rows + + +def prep_gmmlu(val_size: int, seed: int) -> tuple[list[dict], list[dict]]: + import pandas as pd + + d = RAW / "global_mmlu" / "si" + test = _gmmlu_rows(pd.read_parquet(d / "test-00000-of-00001.parquet"), "test") + dev = _gmmlu_rows(pd.read_parquet(d / "dev-00000-of-00001.parquet"), "dev") + + ids_file = SPLITS / "gmmlu_si_val_ids.txt" + if ids_file.exists(): # split is frozen once created + keep = set(ids_file.read_text().split()) + else: # proportional stratified sample by subject + by_subj = defaultdict(list) + for r in test: + by_subj[r["subject"]].append(r["uid"]) + rng = random.Random(seed) + keep = set() + for subj in sorted(by_subj): + uids = sorted(by_subj[subj]) + k = max(1, round(val_size * len(uids) / len(test))) + keep.update(rng.sample(uids, min(k, len(uids)))) + SPLITS.mkdir(parents=True, exist_ok=True) + ids_file.write_text("\n".join(sorted(keep)) + "\n") + return [r for r in test if r["uid"] in keep], dev + + +def prep_belebele() -> list[dict]: + rows = [] + for r in read_jsonl(RAW / "belebele" / "data" / "sin_Sinh.jsonl"): + rows.append(dict( + uid=f"belebele_si/{r['link']}#{r['question_number']}", dataset="belebele_si", + q_no=r["question_number"], subject="reading_comprehension", category="reading", + level="belebele", question=f"{r['flores_passage'].strip()}\n\n{r['question'].strip()}", + options=[r[f"mc_answer{i}"].strip() for i in range(1, 5)], + answer=int(r["correct_answer_num"]), meta={}, + )) + return rows + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--gmmlu-val-size", type=int, default=2000) + ap.add_argument("--seed", type=int, default=1234) + args = ap.parse_args() + + outputs = {"sinhalammlu_dev": prep_sinhalammlu()} + outputs["sinhalammlu_dev_5shot"] = select_fewshot(outputs["sinhalammlu_dev"], 5, args.seed) + outputs["gmmlu_si_val"], outputs["gmmlu_si_dev"] = prep_gmmlu(args.gmmlu_val_size, args.seed) + outputs["belebele_si"] = prep_belebele() + for name, rows in outputs.items(): + write_jsonl(OUT / f"{name}.jsonl", rows) + ans = Counter(r["answer"] for r in rows) + print(f"{name:18s} n={len(rows):5d} answers={dict(sorted(ans.items()))}") + + +if __name__ == "__main__": + main() diff --git a/training/scripts/prepare_subsets.py b/training/scripts/prepare_subsets.py new file mode 100644 index 0000000000000000000000000000000000000000..1506b4fd314a33417b5c0c35a926c15cdadf29eb --- /dev/null +++ b/training/scripts/prepare_subsets.py @@ -0,0 +1,33 @@ +"""Recreate the fixed evaluation subsets used by the reasoning-trace experiments (deterministic, seed 1234): + smmlu_dev_sample400 400 random SinhalaMMLU dev questions (HELD OUT from trace training) + smmlu_dev_rest1450 the other 1,450 dev questions (trace source) + gmmlu_si_val_nonacademic300 300 random non-academic (Humanities / Social Sciences) Global-MMLU val questions + gmmlu_si_val_nonacademic_fresh300 300 other random non-academic val questions (cross-model pooling decision only) + gmmlu_si_val_academic Global-MMLU val STEM + Business questions +Run after scripts/fetch_data.py and scripts/prepare_data.py. +""" +from __future__ import annotations + +import random +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import ROOT, read_jsonl, write_jsonl # noqa: E402 + +P = ROOT / "data" / "processed" +dev = read_jsonl(P / "sinhalammlu_dev.jsonl") +sample = random.Random(1234).sample(dev, 400) +keep = {r["uid"] for r in sample} +write_jsonl(P / "smmlu_dev_sample400.jsonl", sample) +write_jsonl(P / "smmlu_dev_rest1450.jsonl", [r for r in dev if r["uid"] not in keep]) +val = read_jsonl(P / "gmmlu_si_val.jsonl") +non = [r for r in val if r["category"] in ("Humanities", "Social Sciences")] +na300 = random.Random(1234).sample(non, 300) +write_jsonl(P / "gmmlu_si_val_nonacademic300.jsonl", na300) +# Fresh stand-in (27 Oct), never looked at before the cross-model pooling rule was fixed: 300 other non-academic val questions +used = {r["uid"] for r in na300} +write_jsonl(P / "gmmlu_si_val_nonacademic_fresh300.jsonl", random.Random(2026).sample([r for r in non if r["uid"] not in used], 300)) +write_jsonl(P / "gmmlu_si_val_academic.jsonl", [r for r in val if r["category"] in ("STEM", "Business")]) +print("dev", len(dev), "| sample400", len(sample), "| rest", len(dev) - len(sample), "| nonacademic300 300 | academic", + sum(r["category"] in ("STEM", "Business") for r in val)) diff --git a/training/scripts/runs/README.md b/training/scripts/runs/README.md new file mode 100644 index 0000000000000000000000000000000000000000..1c59d5322f97136a3af3208f49e61c5060792b15 --- /dev/null +++ b/training/scripts/runs/README.md @@ -0,0 +1,39 @@ +# Experiment run scripts + +Each script is the exact queue that produced a reported result or a submission. They run from any directory +(`cd` to the repo root themselves), skip steps whose outputs already exist, and log to `results/logs/`. +Reusable tools (data building, evaluation, overlap checks) live one level up in `scripts/` and in `eval/`, `train/`, +`inference/`. + +| Script | Date | Produced | Where reported | +|---|---|---|---| +| `run_baselines.sh` | 10 Oct | Zero-shot baselines of 10 candidate models (dev, Global-MMLU `si` val, Belebele) | `reports/baselines_zeroshot_logit.md` | +| `run_experiments.sh` | 10 Oct | Prompting experiments on the top 3 models: Sinhala instruction, fixed 5-shot, generation, cloze | `reports/experiments_prompting.md` | +| `run_phase1.sh` | 10 Oct | Text-only Gemma 4 E4B (7.463B) check; offline submission pipeline test on dev; first training-speed benchmark | `reports/throughput.jsonl` | +| `run_bench_speedups.sh` | 10 Oct | Training speed with Liger loss, selective recomputation, `torch.compile` | `reports/throughput.jsonl` | +| `run_bench_gemma_compile.sh` | 11 Oct | Gemma 4 E4B speed after the compile recompile-limit fix | `reports/throughput.jsonl` | +| `run_bench_memory.sh` | 11 Oct | Speed/memory with 8-bit AdamW and every-2nd-layer recomputation | `reports/throughput.jsonl` | +| `run_sft_gmmlu.sh` | 12 Oct | Full fine-tuning on Global-MMLU (negative result: dev −2.9) | `reports/sft_gmmlu_v1.md` | +| `run_abc.sh` | 12 Oct | **Model A** (LoRA on Global-MMLU, basis of submissions 2 and 4); dev 5-fold LoRA (B); continued-pretraining pilot (C) | `reports/experiments_abc.md` | +| `download_teachers.sh` | 18 Oct | Downloads of the teacher candidates (data generation only, never inference) | — | +| `run_v2_and_teachers.sh` | 18 Oct | **v2** (LoRA on Global-MMLU + all dev; submission 3) and its test predictions; teacher evaluations | `reports/teachers.md` | +| `run_thinking_tests.sh` | 19 Oct | Thinking mode: model A and base model on the Global-MMLU academic proxy and a dev sample | PLAN.md §2b | +| `run_softlabels_pilot_sub4_kd.sh` | 19 Oct | Gemma 4 31B soft labels for Global-MMLU; synthetic MCQ pilot; **submission 4** predictions; distillation LoRA (interrupted by the 19 Oct crash) | `data/synth/pilot.stats.json`, PLAN.md | +| `run_sub5_thinking_kd.sh` | 19–20 Oct | **Submission 5** (base model + adapter A, extracted by `scripts/extract_lora.py`); base-model thinking on dev and non-academic questions; distillation LoRA rerun (no gain: dev −0.9) | PLAN.md, `results/` | +| `run_sub6.sh` | 20 Oct | **Submission 6** (base-model thinking for medium + hard, adapter A fast vote for easy) | PLAN.md §2c | +| `run_easy_sc.sh` | 20 Oct | Base-model thinking on all dev (Easy decision); self-consistency (3 samples) on the Global-MMLU proxies | PLAN.md §2c | +| `run_final_candidate.sh` | 21 Oct | Final candidate A: base model only (7.463B), thinking for all levels, self-consistency for hard | PLAN.md | +| `run_final_candidate_gated.sh` | 21–22 Oct | Final candidate B: as A, with agreement-gated self-consistency on all levels | FUTURE_WORK.md §1 | + +Submissions (all chosen on dev / validation, never on test): + +| # | Config | Test macro avg (easy / medium / hard) | +|---|---|---| +| 1 | `configs/submission.yaml` (zero-shot Gemma 4 E4B) | 51.59 | +| 2 | `configs/submission_lora_gmmlu_vote.yaml` (model A, 4-order vote) | 54.59 (55.30 / 49.51 / 58.97) | +| 3 | `configs/submission_v2.yaml` (v2, 4-order vote) | 54.85 (54.83 / 50.77 / 58.97) | +| 4 | `configs/submission_A_think_hard.yaml` (model A; thinking for `hard`) | 55.83 (55.30 / 49.51 / 62.68) | +| 5 | `configs/submission_sub5_route.yaml` (base + adapter A; base-model thinking without adapter for `hard`) | 57.68 (55.14 / 49.87 / 68.04) | +| 6 | `configs/submission_sub6_think_medhard.yaml` (as 5, thinking also for `medium`) | 59.22 (55.14 / 54.91 / 67.63) | + +The final submission is reproduced with `./run_submission.sh` at the repository root. diff --git a/training/scripts/runs/download_teachers.sh b/training/scripts/runs/download_teachers.sh new file mode 100644 index 0000000000000000000000000000000000000000..2fca03517b4d24ada6c6fd9cd4c922472cabd3d7 --- /dev/null +++ b/training/scripts/runs/download_teachers.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env bash +# Download the teacher candidates (network/disk only, no GPU). Marks each finished model in results/logs/. +set -uo pipefail +cd "$(dirname "$0")/../.." +for m in google/gemma-4-31B-it google/gemma-4-26B-A4B-it Qwen/Qwen3.5-27B; do + n=$(basename "$m"); [ -f "results/logs/.downloaded_$n" ] && { echo "have $m"; continue; } + echo "download $m ($(date '+%H:%M'))" + .venv/bin/python -c " +import sys; sys.path.insert(0, '.') +from eval.common import load_env; load_env() +from huggingface_hub import HfApi, snapshot_download +m = sys.argv[1]; sha = HfApi().model_info(m).sha +snapshot_download(m, revision=sha, allow_patterns=['*.json', '*.safetensors', '*.jinja', 'tokenizer*', '*.model', '*.txt']) +print('ok', m, sha)" "$m" 2>&1 | grep -E "^ok|Error" && touch "results/logs/.downloaded_$n" +done +echo "downloads done ($(date '+%H:%M'))" diff --git a/training/scripts/runs/run_abc.sh b/training/scripts/runs/run_abc.sh new file mode 100644 index 0000000000000000000000000000000000000000..8dd0dfde7e10023b1bd7d8397e4e4ef5a407e14c --- /dev/null +++ b/training/scripts/runs/run_abc.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +# Queue of experiments B, A, C (one GPU: run back to back). Every result is scored like the zero-shot baseline +# (English prompt, logit scoring, 4 cyclic option orders). Resumable: finished evaluations are skipped. +# B LoRA fine-tuning on SinhalaMMLU dev, 5-fold cross-validation (each fold scored on its held-out 370 questions) +# A LoRA fine-tuning on Global-MMLU si, 1 epoch +# C continued-pretraining pilot, 30M tokens, checkpoints at 15M and 30M scored zero-shot +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +mkdir -p results/logs +ts() { date '+%d %b %H:%M'; } +evaluate() { # model run_name data... + local m=$1 name=$2; shift 2 + [ -f "results/$name/metrics.json" ] && { echo "skip eval $name"; return; } + $PY -m eval.run_eval --model "$m" --data "$@" --method logit --n-perms 4 --run-name "$name" \ + > "results/logs/$name.log" 2>&1 && grep -E "acc=" "results/logs/$name.log" | sed 's/.*\(exp_\)/\1/' \ + || { echo "FAILED eval $name"; tail -3 "results/logs/$name.log"; } +} +train() { # log_name args... + local name=$1; shift + $PY -m train.train "$@" 2>&1 | grep -vE "[Ww]arn|W10[0-9]{2}|Writing model" > "results/logs/train_$name.log" + grep -E "^(sft|cpt):|^done|Error|error" "results/logs/train_$name.log" | tail -3 +} + +echo "=== B: dev 5-fold LoRA ($(ts))" +for f in 0 1 2 3 4; do + name=exp_devcv_lora_f$f + [ -f "results/$name/metrics.json" ] && { echo "skip fold $f"; continue; } + train devcv_lora_f$f --config configs/train/sft_lora_devcv.yaml \ + --data data/processed/smmlu_cv5_f${f}_train.jsonl --out models/devcv_lora_f$f + evaluate "$(ls -d models/devcv_lora_f$f/step-* | tail -1)" $name data/processed/smmlu_cv5_f${f}_test.jsonl + rm -rf models/devcv_lora_f$f # 15 GB each; the fold result is what we need +done +$PY -m eval.combine_cv --runs "results/exp_devcv_lora_f*" --name exp_devcv_lora_all + +echo "=== A: Global-MMLU LoRA 1 epoch ($(ts))" +if [ ! -f results/exp_sft_gmmlu_lora_logit/metrics.json ]; then + train sft_gmmlu_lora --config configs/train/sft_lora.yaml --out models/sft_gmmlu_lora + evaluate "$(ls -d models/sft_gmmlu_lora/step-* | tail -1)" exp_sft_gmmlu_lora_logit sinhalammlu_dev gmmlu_si_val +fi + +echo "=== C: pretraining pilot 30M tokens ($(ts))" +if [ ! -d models/cpt_pilot ] || [ -z "$(ls -d models/cpt_pilot/step-* 2>/dev/null)" ]; then + train cpt_pilot --config configs/train/cpt_pilot.yaml +fi +for ck in $(ls -d models/cpt_pilot/step-* | sort -t- -k2 -n); do + evaluate "$ck" "exp_cpt_pilot_$(basename "$ck")_logit" sinhalammlu_dev gmmlu_si_val +done + +$PY -m eval.summarize --runs results/base_gemma-4-E4B-it_logit results/exp_devcv_lora_all \ + results/exp_sft_gmmlu_lora_logit "results/exp_cpt_pilot_step-*_logit" --out reports/experiments_abc.md > /dev/null +echo "summary -> reports/experiments_abc.md" +echo "=== done ($(ts))" diff --git a/training/scripts/runs/run_baselines.sh b/training/scripts/runs/run_baselines.sh new file mode 100644 index 0000000000000000000000000000000000000000..bb0f6e9e2ac537197de6ad43f8b5d7671ce30781 --- /dev/null +++ b/training/scripts/runs/run_baselines.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Zero-shot baselines: same prompt for every model, letter-logit scoring, 4 cyclic option orders. +# Usage: scripts/run_baselines.sh [model ...] (default: every `baseline` entry in configs/models.yaml) +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +METHOD=${METHOD:-logit} +NPERMS=${NPERMS:-4} +DATA=${DATA:-"sinhalammlu_dev gmmlu_si_val belebele_si"} + +if [ $# -gt 0 ]; then + MODELS=("$@") +else + mapfile -t MODELS < <($PY -c "import yaml; [print(c['id']) for c in yaml.safe_load(open('configs/models.yaml'))['candidates'] if c.get('baseline', True)]") +fi + +mkdir -p results/logs +for m in "${MODELS[@]}"; do + name="base_$(basename "$m")_${METHOD}" + if [ -f "results/$name/metrics.json" ]; then echo "skip $name (done)"; continue; fi + echo "=== $m -> $name ($(date +%H:%M))" + $PY -m eval.run_eval --model "$m" --data $DATA --method "$METHOD" --n-perms "$NPERMS" \ + --run-name "$name" > "results/logs/$name.log" 2>&1 \ + && grep -E "acc=|scoring" "results/logs/$name.log" \ + || { echo "FAILED $m (see results/logs/$name.log)"; tail -5 "results/logs/$name.log"; } +done +$PY -m eval.summarize --runs "results/base_*_${METHOD}" --out "reports/baselines_zeroshot_${METHOD}.md" > /dev/null +echo "summary -> reports/baselines_zeroshot_${METHOD}.md" diff --git a/training/scripts/runs/run_bench_gemma_compile.sh b/training/scripts/runs/run_bench_gemma_compile.sh new file mode 100644 index 0000000000000000000000000000000000000000..60c0c91b348b466494b1fbe3e0ec5936fb59df40 --- /dev/null +++ b/training/scripts/runs/run_bench_gemma_compile.sh @@ -0,0 +1,15 @@ +#!/usr/bin/env bash +# Rerun of the Gemma 4 E4B settings affected by the compile recompile-limit and OOM-retry bugs (10 Oct grid). +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +G=models/gemma-4-E4B-it-text +for ck in 1 2 0; do + tag="gemma4E4Bittext_freezeembed_liger_ckpt${ck}_compile_fixed" + echo "=== $tag ($(date +%H:%M))" + $PY -m train.bench_throughput --model $G --mode freeze-embed --loss liger --ckpt-every $ck --compile \ + --tag compile_fixed > "results/logs/bench_$tag.log" 2>&1 \ + && tail -1 "results/logs/bench_$tag.log" || { echo "FAILED"; grep -vE "^\s*$" "results/logs/bench_$tag.log" | tail -3; } + echo "recompilation warnings: $(grep -c 'recompile' "results/logs/bench_$tag.log")" +done +echo "=== done ($(date +%H:%M))" diff --git a/training/scripts/runs/run_bench_memory.sh b/training/scripts/runs/run_bench_memory.sh new file mode 100644 index 0000000000000000000000000000000000000000..80461b02fd6a81966221b9df6b078662cc38cb54 --- /dev/null +++ b/training/scripts/runs/run_bench_memory.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# Gemma 4 E4B: compile with activation recomputation (fixed) and/or 8-bit optimizer states, for memory headroom. +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +G=models/gemma-4-E4B-it-text +RUNS=( + "--ckpt-every 2 --optim adamw" + "--ckpt-every 0 --optim adamw8bit" + "--ckpt-every 2 --optim adamw8bit" +) +for r in "${RUNS[@]}"; do + tag="gemma4E4B_freezeembed_liger_compile_$(echo "$r" | tr -d '-' | tr ' ' '_')" + echo "=== $tag ($(date +%H:%M))" + $PY -m train.bench_throughput --model $G --mode freeze-embed --loss liger --compile $r --tag memory \ + > "results/logs/bench_$tag.log" 2>&1 \ + && tail -1 "results/logs/bench_$tag.log" || { echo "FAILED"; grep -vE "^\s*$|^W1011" "results/logs/bench_$tag.log" | tail -3; } + echo "recompile-limit warnings: $(grep -c 'hit config.recompile_limit' "results/logs/bench_$tag.log")" +done +echo "=== done ($(date +%H:%M))" diff --git a/training/scripts/runs/run_bench_speedups.sh b/training/scripts/runs/run_bench_speedups.sh new file mode 100644 index 0000000000000000000000000000000000000000..2efabddd3cd3369bdc14a47f4b9ed559cdaf4a61 --- /dev/null +++ b/training/scripts/runs/run_bench_speedups.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# Throughput with the speed options of train/efficiency.py (results appended to reports/throughput.jsonl). +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +G=models/gemma-4-E4B-it-text +Q=Qwen/Qwen3.5-4B +RUNS=( + "$G freeze-embed --loss chunked --ckpt-every 1" + "$G freeze-embed --loss liger --ckpt-every 1" + "$G freeze-embed --loss liger --ckpt-every 2" + "$G freeze-embed --loss liger --ckpt-every 0" + "$G freeze-embed --loss liger --ckpt-every 1 --compile" + "$G freeze-embed --loss liger --ckpt-every 2 --compile" + "$G lora --loss liger --ckpt-every 1 --compile" + "$Q full --loss hf --ckpt-every 1" + "$Q full --loss liger --ckpt-every 1" + "$Q full --loss liger --ckpt-every 1 --compile" + "$Q lora --loss liger --ckpt-every 1 --compile" +) +mkdir -p results/logs +for r in "${RUNS[@]}"; do + set -- $r + m=$1; mode=$2; shift 2 + tag=$(echo "$(basename "$m")_${mode}_$*" | tr ' ' '_' | tr -d '-') + echo "=== $tag ($(date +%H:%M))" + $PY -m train.bench_throughput --model "$m" --mode "$mode" "$@" --tag speedups > "results/logs/bench_$tag.log" 2>&1 \ + && tail -1 "results/logs/bench_$tag.log" || { echo "FAILED"; grep -vE "^\s*$" "results/logs/bench_$tag.log" | tail -3; } +done +echo "=== done ($(date +%H:%M))" diff --git a/training/scripts/runs/run_easy_sc.sh b/training/scripts/runs/run_easy_sc.sh new file mode 100644 index 0000000000000000000000000000000000000000..8589285e9aadf85c1ee29add18c9e0415ae66c46 --- /dev/null +++ b/training/scripts/runs/run_easy_sc.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# 20 Oct: +# 1. Easy decision: base-model thinking on the remaining 1,450 dev questions; combined with the earlier 400 into the +# full dev set and compared with model A fast (original order and 4-order vote) +# 2. Self-consistency test: 3 sampled thinking paths (T=0.8, top-p 0.95, top-k 64) on the academic (551) and +# non-academic (300) Global-MMLU proxies; vote with the existing greedy runs, compared with greedy alone +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +B=models/gemma-4-E4B-it-text +ts() { date '+%d %b %H:%M'; } +run() { # name data extra-args... + local n=$1 d=$2; shift 2 + [ -f "results/$n/metrics.json" ] || $PY -m eval.run_eval --model $B --data $d --method generate --thinking \ + --max-new-tokens 3072 --run-name $n "$@" > results/logs/$n.log 2>&1 + grep -E "acc=" results/logs/$n.log | sed 's/.*\(think_\|sc3_\)/\1/' || tail -3 results/logs/$n.log +} +echo "=== 1. easy: base-model thinking on the rest of dev ($(ts))" +run think_base_smmlu_dev_rest1450 smmlu_dev_rest1450 +$PY -m eval.combine_cv --runs results/think_base_smmlu_dev_sample400 results/think_base_smmlu_dev_rest1450 --name think_base_dev_all +echo "-- full dev, base thinking vs model A fast (orig) and A 4-order vote (vote line):" +$PY -m eval.compare --base results/exp_sft_gmmlu_lora_logit --runs results/think_base_dev_all + +echo "=== 2. self-consistency on the proxies ($(ts))" +run sc3_base_academic gmmlu_si_val_academic --temperature 0.8 --perm-mode identity --n-perms 3 +run sc3_base_nonacademic gmmlu_si_val_nonacademic300 --temperature 0.8 --perm-mode identity --n-perms 3 +$PY -m eval.self_consistency --greedy results/think_base_academic --samples results/sc3_base_academic +$PY -m eval.self_consistency --greedy results/think_base_gmmlu_si_val_nonacademic300 --samples results/sc3_base_nonacademic +echo "=== done ($(ts))" diff --git a/training/scripts/runs/run_experiments.sh b/training/scripts/runs/run_experiments.sh new file mode 100644 index 0000000000000000000000000000000000000000..529d1d823317a05b4697b48551de4d318650b171 --- /dev/null +++ b/training/scripts/runs/run_experiments.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# Prompting / scoring experiments on the top baseline models, compared against their zero-shot English logit run. +# logit_si Sinhala instruction, 4 cyclic option orders +# logit_5shot fixed 5 examples from the dev pool (same for every question), original option order +# generate greedy generation + answer parsing, original option order +# cloze length-normalised option likelihood (options not shown) +# Usage: scripts/run_experiments.sh [model ...] +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +DATA="sinhalammlu_dev gmmlu_si_val" +if [ $# -gt 0 ]; then MODELS=("$@"); else MODELS=(google/gemma-4-E4B-it google/gemma-3n-E4B-it Qwen/Qwen3.5-4B); fi + +declare -A EXP=( + [logit_si]="--method logit --lang si --n-perms 4" + [logit_5shot]="--method logit --fewshot-file sinhalammlu_dev_5shot --n-shot 5 --n-perms 1" + [generate]="--method generate --n-perms 1 --max-new-tokens 16" + [cloze]="--method cloze" +) +ORDER=(logit_si logit_5shot generate cloze) + +mkdir -p results/logs +for m in "${MODELS[@]}"; do + for e in "${ORDER[@]}"; do + name="exp_$(basename "$m")_${e}" + if [ -f "results/$name/metrics.json" ]; then echo "skip $name (done)"; continue; fi + echo "=== $name ($(date +%H:%M))" + $PY -m eval.run_eval --model "$m" --data $DATA ${EXP[$e]} --run-name "$name" > "results/logs/$name.log" 2>&1 \ + && grep -E "acc=|scoring" "results/logs/$name.log" \ + || { echo "FAILED $name (see results/logs/$name.log)"; tail -5 "results/logs/$name.log"; } + done +done +runs=() +for m in "${MODELS[@]}"; do runs+=("results/base_$(basename "$m")_logit" "results/exp_$(basename "$m")_*"); done +$PY -m eval.summarize --runs "${runs[@]}" --out reports/experiments_prompting.md > /dev/null +echo "summary -> reports/experiments_prompting.md" diff --git a/training/scripts/runs/run_final_candidate.sh b/training/scripts/runs/run_final_candidate.sh new file mode 100644 index 0000000000000000000000000000000000000000..211be1fb22a7e665469a1cf270be3214eed672fb --- /dev/null +++ b/training/scripts/runs/run_final_candidate.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# 21 Oct: final candidate = base Gemma 4 E4B text-only (7.463B, no adapter), thinking for every question, +# self-consistency (greedy + 3 samples) for hard. Config: configs/submission_base_think_sc_hard.yaml +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +OUT=results/submissions/final_candidate_base_think_sc_hard.zip +echo "=== final candidate predictions ($(date '+%d %b %H:%M'))" +[ -f "$OUT" ] || HF_HUB_OFFLINE=1 $PY -m inference.predict --config configs/submission_base_think_sc_hard.yaml --input data/test \ + --output "$OUT" > results/logs/predict_final_candidate.log 2>&1 +grep -E "questions|thinking|self-consistency|wrote|Error|Traceback" results/logs/predict_final_candidate.log +$PY -m inference.check_predictions --predictions "$OUT" --gold data/test 2>&1 | tail -1 +$PY - <<'PYEOF' +import json, zipfile +def load(p): return {(r["level"], r["subject"], r["q_no"]): r["answer"] for r in map(json.loads, zipfile.ZipFile(p).read("predictions.jsonl").decode().splitlines())} +a, b = load("results/submissions/sub6_base_adapterA_think_medhard.zip"), load("results/submissions/final_candidate_base_think_sc_hard.zip") +for lvl in ("easy", "medium", "hard"): + ks = [k for k in a if k[0] == lvl] + print(f" {lvl}: answers changed vs the previous submission: {sum(a[k] != b[k] for k in ks)}/{len(ks)}") +PYEOF +echo "=== done ($(date '+%d %b %H:%M'))" diff --git a/training/scripts/runs/run_final_candidate_gated.sh b/training/scripts/runs/run_final_candidate_gated.sh new file mode 100644 index 0000000000000000000000000000000000000000..5d4786d21d9e7b13ae15e1f40cd7a281d1495746 --- /dev/null +++ b/training/scripts/runs/run_final_candidate_gated.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# 21-22 Oct: final candidate B = base Gemma 4 E4B text-only (7.463B), thinking for every question, agreement-gated +# self-consistency on all levels (configs/submission_base_think_gated_sc.yaml). Starts after scripts/runs/run_final_candidate.sh +# (tmux session "final") has finished, so the two runs never share the GPU. +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +CFG=configs/submission_base_think_gated_sc.yaml +OUT=results/submissions/final_candidate_base_think_gated_sc.zip +until ! tmux has-session -t final 2>/dev/null || grep -q "=== done" results/logs/run_final_candidate.log 2>/dev/null; do sleep 120; done +echo "=== smoke test of the gate on 4 dev questions ($(date '+%d %b %H:%M'))" +sed 's/^thinking_levels: .*/thinking_levels: [easy]/; s/^sc_levels: .*/sc_levels: [easy]/' $CFG > results/logs/_smoke_gated.yaml +HF_HUB_OFFLINE=1 $PY -m inference.predict --config results/logs/_smoke_gated.yaml --input data/raw/sinhalammlu/TEST/easy \ + --output results/logs/_smoke_gated.zip --limit 4 2>&1 | grep -E "thinking|self-consistency|wrote|Error|Traceback" \ + || { echo "SMOKE TEST FAILED - not starting the full run"; exit 1; } +echo "=== final candidate B predictions ($(date '+%d %b %H:%M'))" +[ -f "$OUT" ] || HF_HUB_OFFLINE=1 $PY -m inference.predict --config $CFG --input data/test --output "$OUT" \ + > results/logs/predict_final_candidate_gated.log 2>&1 +grep -E "questions|thinking|self-consistency|wrote|Error|Traceback" results/logs/predict_final_candidate_gated.log +$PY -m inference.check_predictions --predictions "$OUT" --gold data/test 2>&1 | tail -1 +$PY - <<'PYEOF' +import json, zipfile +def load(p): return {(r["level"], r["subject"], r["q_no"]): r["answer"] for r in map(json.loads, zipfile.ZipFile(p).read("predictions.jsonl").decode().splitlines())} +a = load("results/submissions/final_candidate_base_think_sc_hard.zip") +b = load("results/submissions/final_candidate_base_think_gated_sc.zip") +for lvl in ("easy", "medium", "hard"): + ks = [k for k in a if k[0] == lvl] + print(f" {lvl}: answers changed vs candidate A (self-consistency on hard only): {sum(a[k] != b[k] for k in ks)}/{len(ks)}") +PYEOF +echo "=== done ($(date '+%d %b %H:%M'))" diff --git a/training/scripts/runs/run_option_order.sh b/training/scripts/runs/run_option_order.sh new file mode 100644 index 0000000000000000000000000000000000000000..a4aeaaf897ce4c2de41cbb94a88f93782215890a --- /dev/null +++ b/training/scripts/runs/run_option_order.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +# 27 Oct, unattended: option-order self-consistency for Medium. +# 1. validate on the Global-MMLU non-academic stand-in (300 q, held out): thinking under 3 re-ordered option lists, voted with +# the existing greedy + 3 sampled paths (eval/perm_sc.py; GO if +1.0 over the current vote) +# 2. if GO: run the medium test questions only (configs/submission_medium_option_order.yaml, --levels medium) +# 3. merge into candidate B and run the format check +# On NO-GO or failure candidate B stays final. +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +ts() { date '+%d %b %H:%M'; } +mkdir -p results/logs + +echo "=== 1. validation ($(ts))" +[ -f results/permthink_nonacademic300/metrics.json ] || $PY -m eval.run_eval --model models/gemma-4-E4B-it-text \ + --data gmmlu_si_val_nonacademic300 --method generate --thinking --max-new-tokens 3072 --n-perms 4 --first-perm 1 \ + --perm-mode cyclic --run-name permthink_nonacademic300 > results/logs/permthink_nonacademic300.log 2>&1 +$PY -m eval.perm_sc --greedy results/think_base_gmmlu_si_val_nonacademic300 --samples results/sc3_base_nonacademic \ + --perms results/permthink_nonacademic300 | tee results/logs/perm_decision.log | tail -16 +grep -q "DECISION: GO" results/logs/perm_decision.log || { echo "NO-GO: candidate B stays final ($(ts))"; exit 0; } + +echo "=== 2. medium test questions ($(ts))" +OUT=results/submissions/medium_option_order.zip +[ -f "$OUT" ] || HF_HUB_OFFLINE=1 $PY -m inference.predict --config configs/submission_medium_option_order.yaml \ + --input data/test --levels medium --output "$OUT" > results/logs/predict_medium_option_order.log 2>&1 +grep -E "questions|option-order|wrote|Error|Traceback" results/logs/predict_medium_option_order.log + +echo "=== 3. merge and check ($(ts))" +$PY scripts/merge_predictions.py --base results/submissions/final_candidate_base_think_gated_sc.zip --new "$OUT" \ + --out results/submissions/final_candidate_B_plus_option_order_medium.zip +$PY -m inference.check_predictions --predictions results/submissions/final_candidate_B_plus_option_order_medium.zip --gold data/test 2>&1 | tail -1 +echo "=== done ($(ts))" diff --git a/training/scripts/runs/run_phase1.sh b/training/scripts/runs/run_phase1.sh new file mode 100644 index 0000000000000000000000000000000000000000..fa84f1ca9c6cf83823f17a4a96cda58efb87f218 --- /dev/null +++ b/training/scripts/runs/run_phase1.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# Phase 1 checks (PLAN.md): +# 1. text-only Gemma 4 E4B: extract, evaluate, compare with the multimodal baseline +# 2. offline submission pipeline on the dev files, validated + scored like the leaderboard +# 3. training throughput benchmarks +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +mkdir -p results/logs results/dev_submission +step() { echo; echo "=== $* ($(date +%H:%M))"; } + +step "1a. extract text-only Gemma 4 E4B" +[ -f models/gemma-4-E4B-it-text/SOURCE.json ] || $PY scripts/make_text_only.py google/gemma-4-E4B-it 2>&1 | grep -v -i warn | tail -3 + +step "1b. evaluate text-only model (original option order)" +if [ ! -f results/exp_gemma-4-E4B-it-text_logit/metrics.json ]; then + $PY -m eval.run_eval --model models/gemma-4-E4B-it-text --data sinhalammlu_dev gmmlu_si_val --n-perms 1 \ + --run-name exp_gemma-4-E4B-it-text_logit > results/logs/exp_gemma-4-E4B-it-text_logit.log 2>&1 +fi +grep -E "acc=" results/logs/exp_gemma-4-E4B-it-text_logit.log +$PY - <<'EOF' +from eval.common import read_jsonl +base = {r["uid"]: r for r in read_jsonl("results/base_gemma-4-E4B-it_logit/predictions.jsonl") if r["perm_k"] == 0} +text = {r["uid"]: r for r in read_jsonl("results/exp_gemma-4-E4B-it-text_logit/predictions.jsonl")} +common = base.keys() & text.keys() +same = sum(base[u]["pred_orig"] == text[u]["pred_orig"] for u in common) +dp = max(max(abs(a - b) for a, b in zip(base[u]["probs_orig"], text[u]["probs_orig"])) for u in common) +print(f"text-only vs multimodal: same prediction on {same}/{len(common)} questions; max probability difference {dp:.4f}") +EOF + +step "2. offline submission pipeline on dev files" +HF_HUB_OFFLINE=1 $PY -m inference.predict --input data/raw/sinhalammlu/TEST/easy \ + --output results/dev_submission/predictions.zip 2>&1 | grep -vE "it/s\]|s/it\]|[Ww]arn" | tail -3 +$PY -m inference.check_predictions --predictions results/dev_submission/predictions.zip --gold data/raw/sinhalammlu/TEST/easy + +step "3. training throughput" +for spec in "models/gemma-4-E4B-it-text lora" "models/gemma-4-E4B-it-text freeze-embed" "Qwen/Qwen3.5-4B full" "Qwen/Qwen3.5-4B lora"; do + set -- $spec + log="results/logs/bench_$(basename "$1")_$2.log" + $PY -m train.bench_throughput --model "$1" --mode "$2" > "$log" 2>&1 && tail -1 "$log" || { echo "FAILED $spec"; tail -4 "$log"; } +done +step "done" diff --git a/training/scripts/runs/run_pool.sh b/training/scripts/runs/run_pool.sh new file mode 100644 index 0000000000000000000000000000000000000000..3a52c936509d0db211735ba27a55af8a55424a77 --- /dev/null +++ b/training/scripts/runs/run_pool.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# Cross-model pooling (27-28 Oct), fully unattended. Medium only at test time. +# 1. validation runs on FRESH stand-in questions 2. pre-registered rule (eval/pool_decide.py, gain >= +1.0) +# 3. if GO: base and RFT2 test runs with saved ballots (Medium) 4. pool, merge into B, format check +# Environment variables: B_ZIP (candidate B predictions), MAX_BATCH (default 64), GPU_MEM_GB (default 100) +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=${PY:-.venv/bin/python} +B_ZIP=${B_ZIP:-results/submissions/final_candidate_base_think_gated_sc.zip} +MAX_BATCH=${MAX_BATCH:-64} +ADAPTER=models/rft_v2/step-141/adapter +export SMMLU_GPU_MEM_GB=${GPU_MEM_GB:-100} +ts() { date '+%d %b %H:%M'; } +mkdir -p results/logs results/submissions + +echo "=== 0. checks ($(ts))" +for f in "$ADAPTER/adapter_model.safetensors" models/gemma-4-E4B-it-text/model.safetensors "$B_ZIP" data/test; do + [ -e "$f" ] || { echo "MISSING: $f"; exit 1; } +done +$PY scripts/prepare_subsets.py +D=gmmlu_si_val_nonacademic_fresh300 + +echo "=== 1. validation on $D ($(ts))" +run() { # run_name extra-args... + local name=$1; shift + [ -f "results/$name/metrics.json" ] || $PY -m eval.run_eval --model models/gemma-4-E4B-it-text --data $D \ + --max-batch "$MAX_BATCH" --run-name "$name" "$@" > "results/logs/$name.log" 2>&1 + echo " $name: $(grep -oE "acc=[0-9.]+" "results/logs/$name.log" | tail -1) ($(ts))" +} +THINK="--method generate --thinking --max-new-tokens 3072" +SAMPLE="--temperature 0.8 --perm-mode identity --n-perms 3" +run pool_fast_base_fresh300 --method logit --n-perms 4 +run pool_think_base_fresh300 $THINK +run pool_sc3_base_fresh300 $THINK $SAMPLE +run pool_think_rft2_fresh300 $THINK --adapter "$ADAPTER" +run pool_sc3_rft2_fresh300 $THINK $SAMPLE --adapter "$ADAPTER" +$PY -m eval.pool_decide --fast pool_fast_base_fresh300 --base-greedy pool_think_base_fresh300 --base-samples pool_sc3_base_fresh300 \ + --rft-greedy pool_think_rft2_fresh300 --rft-samples pool_sc3_rft2_fresh300 | tee results/logs/pool_decision.log | tail -20 +grep -q "DECISION: GO" results/logs/pool_decision.log || { echo "NO-GO: candidate B stays final ($(ts))"; exit 0; } + +echo "=== 2. medium test questions ($(ts))" +for part in base rft2; do + OUT=results/submissions/pool_${part}_medium.zip + sed -e "s|^max_batch: .*|max_batch: $MAX_BATCH|" -e "s|^gpu_mem_gb: .*|gpu_mem_gb: ${GPU_MEM_GB:-100}|" \ + configs/submission_pool_${part}_medium.yaml > configs/submission_pool_${part}_medium.resolved.yaml + [ -f "$OUT.ballots.jsonl" ] || HF_HUB_OFFLINE=1 $PY -m inference.predict --config configs/submission_pool_${part}_medium.resolved.yaml \ + --input data/test --levels medium --output "$OUT" > results/logs/predict_pool_${part}_medium.log 2>&1 + grep -E "questions|thinking|self-consistency|wrote|Error|Traceback" results/logs/predict_pool_${part}_medium.log + echo " $part done ($(ts))" +done + +echo "=== 3. pool, merge and check ($(ts))" +FINAL=results/submissions/final_candidate_B_plus_pool_medium.zip +$PY scripts/pool_ballots.py --base "$B_ZIP" --out "$FINAL" \ + --ballots results/submissions/pool_base_medium.zip.ballots.jsonl results/submissions/pool_rft2_medium.zip.ballots.jsonl +$PY -m inference.check_predictions --predictions "$FINAL" --gold data/test 2>&1 | tail -1 +echo "=== done ($(ts))" diff --git a/training/scripts/runs/run_rft.sh b/training/scripts/runs/run_rft.sh new file mode 100644 index 0000000000000000000000000000000000000000..2b27aa7890ea1e1747bdaecb45f236e916b6e3f0 --- /dev/null +++ b/training/scripts/runs/run_rft.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# 26-27 Oct, fully unattended: reasoning-preserving fine-tuning (rejection-sampled own traces) and, only if it passes the +# pre-registered validation rule, the final test predictions. +# 1. traces: base model, thinking + sampling on 1,450 dev questions (the 400-question dev sample is held out), 2 samples, +# keep the correct ones (time-boxed to 4 h) +# 2. LoRA r=16 on the traces (loss on the trace only) +# 3. validation: greedy thinking with the adapter on the held-out dev 400 and the Global-MMLU non-academic 300, paired +# against the base model's greedy thinking; eval/decide_rft.py applies the fixed rule (GO / NO-GO) +# 4. if GO: test predictions = candidate C (configs/submission_rft_think_gated_sc.yaml), format check, comparison with B +# Every stage is resumable (finished outputs are skipped). On NO-GO or any failure, candidate B stays final. +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +ts() { date '+%d %b %H:%M'; } +mkdir -p results/logs data/synth + +echo "=== 1. traces ($(ts))" +$PY scripts/gen_traces.py --data smmlu_dev_rest1450 --out data/synth/rft_traces --samples 2 --max-minutes 240 \ + 2>&1 | grep -E "questions|samples,|time box|done|Error|Traceback" | tee -a results/logs/rft_traces.log | tail -4 +N=$(wc -l < data/synth/rft_traces.jsonl 2>/dev/null || echo 0) +echo "kept traces: $N" +[ "$N" -ge 300 ] || { echo "too few traces ($N): stopping, candidate B stays final"; exit 1; } + +echo "=== 2. training ($(ts))" +if [ -z "$(ls -d models/rft_v1/step-* 2>/dev/null)" ]; then + $PY -m train.train --config configs/train/rft_trace.yaml 2>&1 | grep -vE "[Ww]arn|W10[0-9]{2}|Writing model" > results/logs/train_rft_v1.log + grep -E "trace examples|^trace:|^done|Error|Traceback" results/logs/train_rft_v1.log | tail -4 +fi +CK=$(ls -d models/rft_v1/step-* 2>/dev/null | tail -1) +[ -n "$CK" ] && [ -f "$CK/adapter/adapter_model.safetensors" ] || { echo "training failed: candidate B stays final"; exit 1; } +echo "adapter: $CK/adapter" + +echo "=== 3. validation on held-out questions ($(ts))" +val() { # run_name data + [ -f "results/$1/metrics.json" ] || $PY -m eval.run_eval --model models/gemma-4-E4B-it-text --adapter "$CK/adapter" --data $2 \ + --method generate --thinking --max-new-tokens 3072 --run-name $1 > results/logs/$1.log 2>&1 + grep -E "acc=" results/logs/$1.log | sed 's/.*\(think_\)/\1/' +} +val think_rft_dev400 smmlu_dev_sample400 +val think_rft_nonacademic300 gmmlu_si_val_nonacademic300 +$PY -m eval.decide_rft --pairs think_base_smmlu_dev_sample400:think_rft_dev400:sinhalammlu \ + think_base_gmmlu_si_val_nonacademic300:think_rft_nonacademic300:gmmlu_si | tee results/logs/rft_decision.log | tail -22 +grep -q "DECISION: GO" results/logs/rft_decision.log || { echo "NO-GO: candidate B stays final ($(ts))"; exit 0; } + +echo "=== 4. candidate C test predictions ($(ts))" +sed "s|models/rft_v1/step-ADAPTER/adapter|$CK/adapter|" configs/submission_rft_think_gated_sc.yaml > configs/submission_rft_think_gated_sc.resolved.yaml +OUT=results/submissions/final_candidate_rft_think_gated_sc.zip +[ -f "$OUT" ] || HF_HUB_OFFLINE=1 $PY -m inference.predict --config configs/submission_rft_think_gated_sc.resolved.yaml \ + --input data/test --output "$OUT" > results/logs/predict_final_candidate_rft.log 2>&1 +grep -E "questions|thinking|self-consistency|wrote|Error|Traceback" results/logs/predict_final_candidate_rft.log +$PY -m inference.check_predictions --predictions "$OUT" --gold data/test 2>&1 | tail -1 +$PY - <<'PYEOF' +import json, zipfile +def load(p): return {(r["level"], r["subject"], r["q_no"]): r["answer"] for r in map(json.loads, zipfile.ZipFile(p).read("predictions.jsonl").decode().splitlines())} +a = load("results/submissions/final_candidate_base_think_gated_sc.zip") +b = load("results/submissions/final_candidate_rft_think_gated_sc.zip") +for lvl in ("easy", "medium", "hard"): + ks = [k for k in a if k[0] == lvl] + print(f" {lvl}: answers changed vs candidate B: {sum(a[k] != b[k] for k in ks)}/{len(ks)}") +PYEOF +echo "=== done ($(ts))" diff --git a/training/scripts/runs/run_rft2.sh b/training/scripts/runs/run_rft2.sh new file mode 100644 index 0000000000000000000000000000000000000000..d05be5f469b06523b472a7a9cc87c113cf1174f7 --- /dev/null +++ b/training/scripts/runs/run_rft2.sh @@ -0,0 +1,62 @@ +#!/usr/bin/env bash +# Option 2 (27 Oct), fully unattended; meant for a second GPU machine. +# Reasoning-preserving fine-tuning round 2, MEDIUM questions only at test time. +# 0. checks 1. traces on Global-MMLU + synthetic questions (time-boxed) 2. LoRA on dev traces + new traces +# 3. validation on held-out questions + pre-registered rule (eval/decide_rft.py) 4. if GO: medium test run + merge into B +# Environment variables: B_ZIP (candidate B predictions), TRACE_MINUTES (default 240), MAX_BATCH (default 64; lower on a small GPU), +# GPU_MEM_GB (default 100; set to ~85% of the GPU's memory) +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=${PY:-.venv/bin/python} +B_ZIP=${B_ZIP:-results/submissions/final_candidate_base_think_gated_sc.zip} +TRACE_MINUTES=${TRACE_MINUTES:-240} +MAX_BATCH=${MAX_BATCH:-64} +export SMMLU_GPU_MEM_GB=${GPU_MEM_GB:-100} +ts() { date '+%d %b %H:%M'; } +mkdir -p results/logs data/synth results/submissions + +echo "=== 0. checks ($(ts))" +for f in data/processed/smmlu_dev_rest1450.jsonl data/processed/smmlu_dev_sample400.jsonl data/processed/gmmlu_si_val_nonacademic300.jsonl \ + data/sft/gmmlu_si_train.jsonl data/synth/rft_traces.jsonl results/think_base_smmlu_dev_sample400/predictions.jsonl \ + results/think_base_gmmlu_si_val_nonacademic300/predictions.jsonl models/gemma-4-E4B-it-text/model.safetensors "$B_ZIP" data/test; do + [ -e "$f" ] || { echo "MISSING: $f"; exit 1; } +done +$PY scripts/build_rft2_questions.py + +echo "=== 1. traces ($(ts))" +$PY scripts/gen_traces.py --data data/synth/rft2_questions.jsonl --out data/synth/rft2_traces --samples 1 \ + --max-minutes "$TRACE_MINUTES" --max-batch "$MAX_BATCH" 2>&1 | grep -E "questions|samples,|time box|done|Error|Traceback" | tee -a results/logs/rft2_traces.log | tail -4 +N=$(wc -l < data/synth/rft2_traces.jsonl 2>/dev/null || echo 0) +echo "new kept traces: $N (plus $(wc -l < data/synth/rft_traces.jsonl) dev traces)" +[ "$N" -ge 300 ] || { echo "too few new traces: stopping, candidate B stays final"; exit 1; } + +echo "=== 2. training ($(ts))" +if [ -z "$(ls -d models/rft_v2/step-* 2>/dev/null)" ]; then + $PY -m train.train --config configs/train/rft_trace2.yaml 2>&1 | grep -vE "[Ww]arn|W10[0-9]{2}|Writing model" > results/logs/train_rft_v2.log + grep -E "trace examples|^trace:|^done|Error|Traceback" results/logs/train_rft_v2.log | tail -4 +fi +CK=$(ls -d models/rft_v2/step-* 2>/dev/null | tail -1) +[ -n "$CK" ] && [ -f "$CK/adapter/adapter_model.safetensors" ] || { echo "training failed: candidate B stays final"; exit 1; } + +echo "=== 3. validation ($(ts))" +val() { # run_name data + [ -f "results/$1/metrics.json" ] || $PY -m eval.run_eval --model models/gemma-4-E4B-it-text --adapter "$CK/adapter" --data $2 \ + --method generate --thinking --max-new-tokens 3072 --max-batch "$MAX_BATCH" --run-name $1 > results/logs/$1.log 2>&1 + grep -E "acc=" results/logs/$1.log | sed 's/.*\(think_\)/\1/' +} +val think_rft2_dev400 smmlu_dev_sample400 +val think_rft2_nonacademic300 gmmlu_si_val_nonacademic300 +$PY -m eval.decide_rft --out rft2_decision.json --pairs think_base_smmlu_dev_sample400:think_rft2_dev400:sinhalammlu \ + think_base_gmmlu_si_val_nonacademic300:think_rft2_nonacademic300:gmmlu_si | tee results/logs/rft2_decision.log | tail -22 +grep -q "DECISION: GO" results/logs/rft2_decision.log || { echo "NO-GO: candidate B stays final ($(ts))"; exit 0; } + +echo "=== 4. medium test questions ($(ts))" +sed -e "s|models/rft_v2/step-ADAPTER/adapter|$CK/adapter|" -e "s|^max_batch: .*|max_batch: $MAX_BATCH|" -e "s|^gpu_mem_gb: .*|gpu_mem_gb: ${GPU_MEM_GB:-100}|" \ + configs/submission_rft2_medium.yaml > configs/submission_rft2_medium.resolved.yaml +OUT=results/submissions/rft2_medium.zip +[ -f "$OUT" ] || HF_HUB_OFFLINE=1 $PY -m inference.predict --config configs/submission_rft2_medium.resolved.yaml \ + --input data/test --levels medium --output "$OUT" > results/logs/predict_rft2_medium.log 2>&1 +grep -E "questions|thinking|self-consistency|wrote|Error|Traceback" results/logs/predict_rft2_medium.log +$PY scripts/merge_predictions.py --base "$B_ZIP" --new "$OUT" --out results/submissions/final_candidate_B_plus_rft2_medium.zip +$PY -m inference.check_predictions --predictions results/submissions/final_candidate_B_plus_rft2_medium.zip --gold data/test 2>&1 | tail -1 +echo "=== done ($(ts))" diff --git a/training/scripts/runs/run_sft_gmmlu.sh b/training/scripts/runs/run_sft_gmmlu.sh new file mode 100644 index 0000000000000000000000000000000000000000..5128b09459dcde4b277e60014085e0894054ef86 --- /dev/null +++ b/training/scripts/runs/run_sft_gmmlu.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Step 4: fine-tune text-only Gemma 4 E4B on Global-MMLU si, then score every epoch checkpoint exactly like the +# zero-shot baseline (English prompt, logit scoring, 4 cyclic option orders) on SinhalaMMLU dev + Global-MMLU val. +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +CFG=configs/train/sft_gmmlu.yaml +OUT=models/sft_gmmlu_v1 +mkdir -p results/logs + +echo "=== train ($(date '+%d %b %H:%M'))" +$PY -m train.train --config $CFG 2>&1 | grep -vE "[Ww]arn|W10[0-9]{2}|Writing model" | tee results/logs/train_sft_gmmlu_v1.log + +for ck in $(ls -d $OUT/step-* | sort -t- -k2 -n); do + name="exp_sft_gmmlu_v1_$(basename "$ck")_logit" + echo "=== eval $ck ($(date '+%H:%M'))" + $PY -m eval.run_eval --model "$ck" --data sinhalammlu_dev gmmlu_si_val --method logit --n-perms 4 \ + --run-name "$name" > "results/logs/$name.log" 2>&1 && grep -E "acc=" "results/logs/$name.log" \ + || { echo "FAILED eval $ck"; tail -3 "results/logs/$name.log"; } +done + +$PY -m eval.summarize --runs results/base_gemma-4-E4B-it_logit "results/exp_sft_gmmlu_v1_step-*_logit" \ + --out reports/sft_gmmlu_v1.md > /dev/null && echo "summary -> reports/sft_gmmlu_v1.md" +echo "=== done ($(date '+%d %b %H:%M'))" diff --git a/training/scripts/runs/run_softlabels_pilot_sub4_kd.sh b/training/scripts/runs/run_softlabels_pilot_sub4_kd.sh new file mode 100644 index 0000000000000000000000000000000000000000..cdf3f20232ea0c2fb7cb48fa6adc9e023fdeb101 --- /dev/null +++ b/training/scripts/runs/run_softlabels_pilot_sub4_kd.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +# 19 Oct, steps 2-4 (start after scripts/runs/run_thinking_tests.sh has finished; tmux session/log names day2a): +# 2. Gemma 4 31B soft labels for the 11,447 Global-MMLU training questions +# 3. synthetic pilot: ~450 MCQs from 150 Wikipedia passages, round-trip filtered, review file + overlap checks +# 3b. test predictions: model A + thinking on hard (configs/submission_A_think_hard.yaml) +# 4. student LoRA with distillation (0.5 CE + 0.5 KL to teacher), scored on dev + Global-MMLU val, compared with A +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +T=google/gemma-4-31B-it +ts() { date '+%d %b %H:%M'; } +until ! tmux has-session -t day2a 2>/dev/null || grep -q "=== done" results/logs/run_day2a.log 2>/dev/null; do sleep 60; done +echo "start ($(ts))" + +echo "=== 2. teacher soft labels on Global-MMLU train ($(ts))" +[ -f results/teacher_31b_gmmlu_train/metrics.json ] || $PY -m eval.run_eval --model $T --data data/sft/gmmlu_si_train.jsonl \ + --method logit --n-perms 1 --token-budget 16384 --run-name teacher_31b_gmmlu_train > results/logs/teacher_31b_gmmlu_train.log 2>&1 +grep -E "acc=" results/logs/teacher_31b_gmmlu_train.log | sed 's/.*\(teacher_\)/\1/' +$PY scripts/add_teacher_labels.py --data data/sft/gmmlu_si_train.jsonl --run results/teacher_31b_gmmlu_train \ + --out data/sft/gmmlu_si_train_t31b.jsonl + +echo "=== 3. synthetic pilot ($(ts))" +[ -f data/synth/pilot.jsonl ] || $PY scripts/generate_mcq.py --passages data/synth/passages_pilot.jsonl --out data/synth/pilot \ + --teacher $T --per-passage 3 2>&1 | grep -vE "[Ww]arn|W10|it/s\]|s/it\]" > results/logs/synth_pilot.log +tail -25 results/logs/synth_pilot.log +for b in sinhalammlu_dev gmmlu_si_val; do + echo "-- overlap pilot vs $b (character / meaning)" + $PY scripts/overlap.py --a data/synth/pilot.jsonl --b $b --show 0 | sed -n '6,8p' + $PY scripts/semantic_overlap.py --a data/synth/pilot.jsonl --b $b --show 0 2>/dev/null | grep -E "100th|>= 0.9" +done + +echo "=== 3b. test predictions: A + thinking on hard ($(ts))" +OUT=results/submissions/sub4_A_vote4_think_hard.zip +[ -f "$OUT" ] || HF_HUB_OFFLINE=1 $PY -m inference.predict --config configs/submission_A_think_hard.yaml --input data/test \ + --output "$OUT" 2>&1 | grep -E "questions|thinking|wrote|Error" > results/logs/predict_sub4.log +cat results/logs/predict_sub4.log +$PY -m inference.check_predictions --predictions "$OUT" --gold data/test 2>&1 | tail -1 +$PY - <<'PYEOF' +import json, zipfile +def load(p): return {(r["level"], r["subject"], r["q_no"]): r["answer"] for r in map(json.loads, zipfile.ZipFile(p).read("predictions.jsonl").decode().splitlines())} +a, b = load("results/submissions/sub2_lora_gmmlu_vote4.zip"), load("results/submissions/sub4_A_vote4_think_hard.zip") +for lvl in ("easy", "medium", "hard"): + ks = [k for k in a if k[0] == lvl] + print(f" {lvl}: answers changed vs submission 2 (A, fast): {sum(a[k] != b[k] for k in ks)}/{len(ks)}") +PYEOF + +echo "=== 4. distillation LoRA ($(ts))" +if [ -z "$(ls -d models/sft_kd_t31b/step-* 2>/dev/null)" ]; then + $PY -m train.train --config configs/train/sft_lora_kd.yaml 2>&1 | grep -vE "[Ww]arn|W10[0-9]{2}|Writing model" > results/logs/train_sft_kd_t31b.log + grep -E "^sft:|^done|Error" results/logs/train_sft_kd_t31b.log | tail -3 +fi +KD=$(ls -d models/sft_kd_t31b/step-* | tail -1) +[ -f results/exp_sft_kd_t31b_logit/metrics.json ] || $PY -m eval.run_eval --model "$KD" --data sinhalammlu_dev gmmlu_si_val \ + --method logit --n-perms 4 --run-name exp_sft_kd_t31b_logit > results/logs/exp_sft_kd_t31b_logit.log 2>&1 +grep -E "acc=" results/logs/exp_sft_kd_t31b_logit.log | sed 's/.*\(exp_\)/\1/' +echo "-- paired vs model A (dev):" +$PY -m eval.compare --base results/exp_sft_gmmlu_lora_logit --runs results/exp_sft_kd_t31b_logit +echo "=== done ($(ts))" diff --git a/training/scripts/runs/run_sub5_thinking_kd.sh b/training/scripts/runs/run_sub5_thinking_kd.sh new file mode 100644 index 0000000000000000000000000000000000000000..4bdd82b38bc34cd2ee0217f8670545aeefacecf3 --- /dev/null +++ b/training/scripts/runs/run_sub5_thinking_kd.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +# 19-20 Oct queue: +# 1. submission 5: base Gemma 4 E4B + adapter A (extracted by scripts/extract_lora.py; verified 97.8% same answers +# as merged A on 400 dev questions); hard -> thinking without adapter, easy/medium -> fast 4-order vote with adapter +# 2. base-model thinking on easy-style (400 dev) and non-academic (300 Global-MMLU humanities/social) questions, +# compared with model A fast on the same questions -> decides whether medium/easy also get thinking +# 3. distillation LoRA rerun (interrupted by the 19 Oct crash) + dev/val scores vs model A +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +ts() { date '+%d %b %H:%M'; } +mkdir -p results/logs results/submissions +[ -f models/adapter_A/adapter_model.safetensors ] || $PY scripts/extract_lora.py --base models/gemma-4-E4B-it-text \ + --merged models/sft_gmmlu_lora/step-357 --rank 16 --out models/adapter_A + +echo "=== 1. submission 5 predictions ($(ts))" +OUT=results/submissions/sub5_base_adapterA_vote4_think_hard.zip +[ -f "$OUT" ] || HF_HUB_OFFLINE=1 $PY -m inference.predict --config configs/submission_sub5_route.yaml --input data/test \ + --output "$OUT" 2>&1 | grep -E "questions|thinking|wrote|Error" > results/logs/predict_sub5.log +cat results/logs/predict_sub5.log +$PY -m inference.check_predictions --predictions "$OUT" --gold data/test 2>&1 | tail -1 +$PY - <<'PYEOF' +import json, zipfile +def load(p): return {(r["level"], r["subject"], r["q_no"]): r["answer"] for r in map(json.loads, zipfile.ZipFile(p).read("predictions.jsonl").decode().splitlines())} +a, b = load("results/submissions/sub4_A_vote4_think_hard.zip"), load("results/submissions/sub5_base_adapterA_vote4_think_hard.zip") +for lvl in ("easy", "medium", "hard"): + ks = [k for k in a if k[0] == lvl] + print(f" {lvl}: answers changed vs submission 4: {sum(a[k] != b[k] for k in ks)}/{len(ks)}") +PYEOF + +echo "=== 2. base-model thinking on easy-style and non-academic questions ($(ts))" +for d in smmlu_dev_sample400 gmmlu_si_val_nonacademic300; do + n=think_base_${d} + [ -f "results/$n/metrics.json" ] || $PY -m eval.run_eval --model models/gemma-4-E4B-it-text --data $d --method generate \ + --thinking --max-new-tokens 3072 --run-name $n > results/logs/$n.log 2>&1 + grep -E "acc=" results/logs/$n.log | sed 's/.*\(think_\)/\1/' +done +echo "-- paired vs model A fast (vote over 4 orders is the 'vote' line):" +$PY -m eval.compare --base results/exp_sft_gmmlu_lora_logit --runs results/think_base_smmlu_dev_sample400 --dataset sinhalammlu +$PY -m eval.compare --base results/exp_sft_gmmlu_lora_logit --runs results/think_base_gmmlu_si_val_nonacademic300 --dataset gmmlu_si + +echo "=== 3. distillation LoRA rerun ($(ts))" +if [ -z "$(ls -d models/sft_kd_t31b/step-* 2>/dev/null)" ]; then + rm -f models/sft_kd_t31b/log.jsonl + $PY -m train.train --config configs/train/sft_lora_kd.yaml 2>&1 | grep -vE "[Ww]arn|W10[0-9]{2}|Writing model" > results/logs/train_sft_kd_t31b.log + grep -E "^sft:|^done|Error" results/logs/train_sft_kd_t31b.log | tail -3 +fi +KD=$(ls -d models/sft_kd_t31b/step-* | tail -1) +[ -f results/exp_sft_kd_t31b_logit/metrics.json ] || $PY -m eval.run_eval --model "$KD" --data sinhalammlu_dev gmmlu_si_val \ + --method logit --n-perms 4 --run-name exp_sft_kd_t31b_logit > results/logs/exp_sft_kd_t31b_logit.log 2>&1 +grep -E "acc=" results/logs/exp_sft_kd_t31b_logit.log | sed 's/.*\(exp_\)/\1/' +echo "-- distillation vs model A (dev):" +$PY -m eval.compare --base results/exp_sft_gmmlu_lora_logit --runs results/exp_sft_kd_t31b_logit +echo "=== done ($(ts))" diff --git a/training/scripts/runs/run_sub6.sh b/training/scripts/runs/run_sub6.sh new file mode 100644 index 0000000000000000000000000000000000000000..624666897ad4841d9b54dd253d90e042cbdc981d --- /dev/null +++ b/training/scripts/runs/run_sub6.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# 20 Oct: submission 6 = base Gemma 4 E4B + adapter A; base-model thinking (adapter off) for medium and hard, +# adapter A fast 4-order vote for easy. +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +OUT=results/submissions/sub6_base_adapterA_think_medhard.zip +echo "=== submission 6 predictions ($(date '+%d %b %H:%M'))" +[ -f "$OUT" ] || HF_HUB_OFFLINE=1 $PY -m inference.predict --config configs/submission_sub6_think_medhard.yaml --input data/test \ + --output "$OUT" 2>&1 | grep -E "questions|thinking|wrote|Error" > results/logs/predict_sub6.log +cat results/logs/predict_sub6.log +$PY -m inference.check_predictions --predictions "$OUT" --gold data/test 2>&1 | tail -1 +$PY - <<'PYEOF' +import json, zipfile +def load(p): return {(r["level"], r["subject"], r["q_no"]): r["answer"] for r in map(json.loads, zipfile.ZipFile(p).read("predictions.jsonl").decode().splitlines())} +a, b = load("results/submissions/sub5_base_adapterA_vote4_think_hard.zip"), load("results/submissions/sub6_base_adapterA_think_medhard.zip") +for lvl in ("easy", "medium", "hard"): + ks = [k for k in a if k[0] == lvl] + print(f" {lvl}: answers changed vs submission 5: {sum(a[k] != b[k] for k in ks)}/{len(ks)}") +PYEOF +echo "=== done ($(date '+%d %b %H:%M'))" diff --git a/training/scripts/runs/run_thinking_tests.sh b/training/scripts/runs/run_thinking_tests.sh new file mode 100644 index 0000000000000000000000000000000000000000..d0d8b49c727cf51bec1a85d34dead869dea7b0b9 --- /dev/null +++ b/training/scripts/runs/run_thinking_tests.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +# 19 Oct, step 1: does thinking mode help? Model A (current submission model) and the base model, think-then-answer, +# on Global-MMLU academic subjects (STEM + Business, proxy for A/L "hard") and a 400-question dev sample. +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +A=models/sft_gmmlu_lora/step-357 +run() { # name model data... + local name=$1 m=$2; shift 2 + [ -f "results/$name/metrics.json" ] && { echo "skip $name"; return; } + $PY -m eval.run_eval --model "$m" --data "$@" --method generate --thinking --max-new-tokens 2048 \ + --run-name "$name" > "results/logs/$name.log" 2>&1 && grep -E "acc=" "results/logs/$name.log" | sed 's/.*\(think_\)/\1/' \ + || { echo "FAILED $name"; tail -3 "results/logs/$name.log"; } +} +echo "=== thinking tests ($(date '+%H:%M'))" +run think_A_academic "$A" gmmlu_si_val_academic +run think_A_dev400 "$A" smmlu_dev_sample400 +run think_base_academic models/gemma-4-E4B-it-text gmmlu_si_val_academic +echo "=== done ($(date '+%H:%M'))" diff --git a/training/scripts/runs/run_v2_and_teachers.sh b/training/scripts/runs/run_v2_and_teachers.sh new file mode 100644 index 0000000000000000000000000000000000000000..c2c096209eb99864a8f0d539d6908131d1089b97 --- /dev/null +++ b/training/scripts/runs/run_v2_and_teachers.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# Day 1 (18 Oct) queue. GPU jobs run one after another; teacher downloads run in parallel with v2 training. +# 1. v2 = LoRA on Global-MMLU si + all SinhalaMMLU dev (1 epoch), merged +# 2. v2 test predictions (4-order vote) + format check; v2 sanity score on Global-MMLU validation +# 3. Teacher candidates scored zero-shot on dev + Global-MMLU validation (logit) and dev (generated answers) +set -uo pipefail +cd "$(dirname "$0")/../.." +PY=.venv/bin/python +TEACHERS=(google/gemma-4-31B-it google/gemma-4-26B-A4B-it Qwen/Qwen3.5-27B) +mkdir -p results/logs results/submissions +ts() { date '+%d %b %H:%M'; } + +# Teacher downloads run separately (scripts/runs/download_teachers.sh, tmux session "downloads"). +# Wait until no other GPU training (the WavLM ta-100h run of Model-Training-Pipeline) is running. +until ! pgrep -f "Model-Training-Pipeline/.venv/bin/python" > /dev/null; do + echo "waiting for the GPU: another project's training is still running ($(ts))"; sleep 300 +done +echo "GPU free ($(ts))" + +# --- 1. v2 training --- +echo "=== v2 training ($(ts))" +if [ -z "$(ls -d models/sft_v2_gmmlu_dev/step-* 2>/dev/null)" ]; then + $PY -m train.train --config configs/train/sft_lora.yaml \ + --data data/sft/gmmlu_si_train.jsonl data/processed/sinhalammlu_dev.jsonl --out models/sft_v2_gmmlu_dev 2>&1 \ + | grep -vE "[Ww]arn|W10[0-9]{2}|Writing model" > results/logs/train_sft_v2_gmmlu_dev.log + grep -E "^sft:|^done|Error" results/logs/train_sft_v2_gmmlu_dev.log | tail -3 +fi +V2=$(ls -d models/sft_v2_gmmlu_dev/step-* | tail -1) +sed -i "s|^model: models/sft_v2_gmmlu_dev/step-[0-9]*|model: $V2|" configs/submission_v2.yaml + +# --- 2. v2 test predictions + checks --- +echo "=== v2 test predictions ($(ts))" +OUT=results/submissions/sub3_v2_lora_gmmlu_dev_vote4.zip +[ -f "$OUT" ] || HF_HUB_OFFLINE=1 $PY -m inference.predict --config configs/submission_v2.yaml --input data/test \ + --output "$OUT" > results/logs/predict_v2.log 2>&1 +$PY -m inference.check_predictions --predictions "$OUT" --gold data/test 2>&1 | tail -2 +[ -f results/exp_sft_v2_gmmlu_dev_gmmluval/metrics.json ] || $PY -m eval.run_eval --model "$V2" --data gmmlu_si_val \ + --n-perms 4 --run-name exp_sft_v2_gmmlu_dev_gmmluval > results/logs/exp_sft_v2_gmmlu_dev_gmmluval.log 2>&1 +grep -E "acc=" results/logs/exp_sft_v2_gmmlu_dev_gmmluval.log | sed 's/.*\(exp_\)/\1/' + +# --- 3. teacher evaluations --- +for m in "${TEACHERS[@]}"; do + n=$(basename "$m") + until [ -f "results/logs/.downloaded_$n" ] || ! tmux has-session -t downloads 2>/dev/null; do sleep 60; done + [ -f "results/logs/.downloaded_$n" ] || echo "download of $m failed or not finished" + [ -f "results/logs/.downloaded_$n" ] || continue + echo "=== teacher $m ($(ts))" + [ -f "results/teacher_${n}_logit/metrics.json" ] || $PY -m eval.run_eval --model "$m" --data sinhalammlu_dev gmmlu_si_val \ + --method logit --n-perms 1 --token-budget 16384 --run-name "teacher_${n}_logit" > "results/logs/teacher_${n}_logit.log" 2>&1 + grep -E "acc=" "results/logs/teacher_${n}_logit.log" | sed 's/.*\(teacher_\)/\1/' || tail -3 "results/logs/teacher_${n}_logit.log" + [ -f "results/teacher_${n}_gen/metrics.json" ] || $PY -m eval.run_eval --model "$m" --data sinhalammlu_dev \ + --method generate --max-new-tokens 48 --token-budget 16384 --run-name "teacher_${n}_gen" > "results/logs/teacher_${n}_gen.log" 2>&1 + grep -E "acc=" "results/logs/teacher_${n}_gen.log" | sed 's/.*\(teacher_\)/\1/' || tail -3 "results/logs/teacher_${n}_gen.log" +done +$PY -m eval.summarize --runs results/base_gemma-4-E4B-it_logit "results/teacher_*" --out reports/teachers.md > /dev/null +echo "summary -> reports/teachers.md" +echo "=== done ($(ts))" diff --git a/training/scripts/select_cpt_subset.py b/training/scripts/select_cpt_subset.py new file mode 100644 index 0000000000000000000000000000000000000000..b7c0ff49894dc481123021861e3966db1900a8ca --- /dev/null +++ b/training/scripts/select_cpt_subset.py @@ -0,0 +1,136 @@ +"""Choose the continued-pretraining mix from data/corpus/clean -> data/corpus// (+ reports/_stats.json). + +Default mix of ~300M estimated Gemma tokens: + wikipedia all of it (encyclopedic, highest priority) + fineweb2 the rest of the Sinhala budget: 70% highest Sri Lanka/curriculum keyword score (sl_score), 30% random + nsina 20% of the budget, random news articles + english 10% of the budget, FineWeb-Edu sample (keeps general/English reasoning), not deduplicated against Sinhala +Cross-source near-duplicates (MinHash on word 5-gram shingles, Jaccard >= 0.8) are removed in priority order +wikipedia > fineweb2 > nsina. Writes 30 random Sinhala documents to sample_for_review.md for a human check. + +Usage: python scripts/select_cpt_subset.py [--target-tokens 300e6] [--name cpt_v1] +""" +from __future__ import annotations + +import argparse +import json +import random +import sys +from multiprocessing import Pool +from pathlib import Path + +import numpy as np +import pandas as pd + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +sys.path.insert(0, str(Path(__file__).resolve().parent)) +from build_cpt_corpus import norm # noqa: E402 +from eval.common import ROOT, load_env # noqa: E402 + +TOK_PER_CHAR = {"si": 0.40, "en": 0.25} # Gemma tokenizer, measured on Sinhala MCQs / typical English +NUM_PERM = 64 +ENGLISH = ("HuggingFaceFW/fineweb-edu", "sample/10BT/000_00000.parquet") + + +def minhash(text: str): + from datasketch import MinHash + + w = norm(text).split() + m = MinHash(num_perm=NUM_PERM, seed=1) + m.update_batch([" ".join(w[i:i + 5]).encode() for i in range(max(len(w) - 4, 1))]) + return m + + +def load_source(clean: Path, name: str) -> pd.DataFrame: + return pd.concat([pd.read_parquet(f) for f in sorted(clean.glob(f"{name}-*.parquet"))], ignore_index=True) + + +def take_chars(df: pd.DataFrame, chars: float) -> pd.DataFrame: + return df[df["n_chars"].cumsum() <= chars] + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--target-tokens", type=float, default=300e6) + ap.add_argument("--news-share", type=float, default=0.20) + ap.add_argument("--english-share", type=float, default=0.10) + ap.add_argument("--name", default="cpt_v1") + ap.add_argument("--workers", type=int, default=16) + ap.add_argument("--seed", type=int, default=1234) + args = ap.parse_args() + load_env() + from datasketch import MinHashLSH + from huggingface_hub import HfApi, hf_hub_download + + clean, out = ROOT / "data" / "corpus" / "clean", ROOT / "data" / "corpus" / args.name + out.mkdir(parents=True, exist_ok=True) + rng = np.random.default_rng(args.seed) + si_chars = args.target_tokens * (1 - args.english_share) / TOK_PER_CHAR["si"] + margin = 1.15 # select extra, then trim after deduplication + + wiki = load_source(clean, "wikipedia") + fw = load_source(clean, "fineweb2") + news = load_source(clean, "nsina") + news_chars = args.target_tokens * args.news_share / TOK_PER_CHAR["si"] + fw_chars = si_chars - wiki["n_chars"].sum() - news_chars + fw_top = take_chars(fw.sort_values("sl_score", ascending=False), 0.7 * fw_chars * margin) + fw_rest = fw.drop(fw_top.index).sample(frac=1, random_state=args.seed) + fw_sel = pd.concat([fw_top.assign(pick="top_sl_score"), + take_chars(fw_rest, 0.3 * fw_chars * margin).assign(pick="random")]) + news_sel = take_chars(news.sample(frac=1, random_state=args.seed), news_chars * margin).assign(pick="random") + cand = pd.concat([wiki.assign(pick="all"), fw_sel, news_sel], ignore_index=True) + print(f"candidates: {len(cand):,} docs, {cand['n_chars'].sum() / 1e6:.0f}M chars", flush=True) + + with Pool(args.workers) as pool: + sigs = pool.map(minhash, cand["text"].tolist(), chunksize=256) + lsh = MinHashLSH(threshold=0.8, num_perm=NUM_PERM) + keep = np.zeros(len(cand), dtype=bool) + for i, m in enumerate(sigs): # cand is ordered wikipedia, fineweb2, nsina = priority order + if not lsh.query(m): + lsh.insert(str(i), m) + keep[i] = True + removed = cand[~keep].groupby("source").size().to_dict() + cand = cand[keep] + parts = { + "wikipedia": cand[cand.source == "wikipedia"], + "fineweb2": pd.concat([take_chars(cand[(cand.source == "fineweb2") & (cand.pick == p)], s * fw_chars) + for p, s in (("top_sl_score", 0.7), ("random", 0.3))]), + "nsina": take_chars(cand[cand.source == "nsina"], news_chars), + } + + # English: FineWeb-Edu sample + sha = HfApi().dataset_info(ENGLISH[0]).sha + path = hf_hub_download(ENGLISH[0], ENGLISH[1], repo_type="dataset", revision=sha, + local_dir=ROOT / "data" / "raw" / "corpus" / "fineweb-edu") + en = pd.read_parquet(path, columns=["id", "url", "text"]).sample(frac=1, random_state=args.seed) + en["n_chars"] = en["text"].str.len() + en = take_chars(en, args.target_tokens * args.english_share / TOK_PER_CHAR["en"]) + parts["english"] = en.assign(source="fineweb_edu", sl_score=0.0, pick="random") + + stats = {"target_tokens": args.target_tokens, "near_duplicates_removed": removed, + "english_revision": f"{ENGLISH[0]}@{sha}", "parts": {}} + for name, df in parts.items(): + lang = "en" if name == "english" else "si" + df[["id", "source", "url", "text", "n_chars", "sl_score", "pick"]].to_parquet(out / f"{name}.parquet", index=False) + stats["parts"][name] = {"docs": len(df), "chars": int(df["n_chars"].sum()), + "est_gemma_tokens": int(df["n_chars"].sum() * TOK_PER_CHAR[lang]), + "picks": df["pick"].value_counts().to_dict()} + stats["total_est_gemma_tokens"] = sum(p["est_gemma_tokens"] for p in stats["parts"].values()) + (ROOT / "reports" / f"{args.name}_stats.json").write_text(json.dumps(stats, indent=2)) + print(json.dumps(stats, indent=2)) + + # Human review sample (stays under data/, git-ignored) + r = random.Random(args.seed) + lines = [f"# {args.name}: 30 random Sinhala documents for review", "", + "Check: is it real Sinhala prose? boilerplate/menus? machine-translated junk? exam material?", ""] + for name, k in (("wikipedia", 8), ("fineweb2", 14), ("nsina", 8)): + df = parts[name] + for i in r.sample(range(len(df)), k): + d = df.iloc[i] + lines += [f"## {name} / {d['pick']} / sl_score {d['sl_score']} / {d['url']}", "", d["text"][:1200], ""] + (out / "sample_for_review.md").write_text("\n".join(lines), encoding="utf-8") + print(f"review sample -> {out / 'sample_for_review.md'}") + + +if __name__ == "__main__": + main() diff --git a/training/scripts/semantic_overlap.py b/training/scripts/semantic_overlap.py new file mode 100644 index 0000000000000000000000000000000000000000..0eaeb5b9a234ac91eeb9ded5f57abafb7bd7a6ea --- /dev/null +++ b/training/scripts/semantic_overlap.py @@ -0,0 +1,76 @@ +"""Meaning-based overlap check between two MCQ sets (catches the same question in different wording, which the +character n-gram check in scripts/overlap.py misses). + + python scripts/semantic_overlap.py --a sinhalammlu_dev --b data/sft/gmmlu_si_train.jsonl [--show 15] + +Embeds "question + options" with intfloat/multilingual-e5-large (covers Sinhala), cosine similarity, and reports for +each A item its nearest B item. A reference scale is printed from pairs that are known copies (Global-MMLU questions +repeated across MMLU subjects) and from random unrelated pairs. +""" +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +import numpy as np + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import cap_gpu_memory, load_env, read_jsonl # noqa: E402 +from eval.run_eval import resolve_data # noqa: E402 + +MODEL = "intfloat/multilingual-e5-large" + + +def text(r: dict) -> str: + return "query: " + r["question"] + " " + " | ".join(r["options"]) + + +def embed(texts: list[str], tok, model, bs: int = 64) -> np.ndarray: + import torch + + out = [] + for i in range(0, len(texts), bs): + enc = tok(texts[i:i + bs], padding=True, truncation=True, max_length=512, return_tensors="pt").to("cuda") + with torch.no_grad(): + h = model(**enc).last_hidden_state + m = enc["attention_mask"].unsqueeze(-1).to(h.dtype) + v = (h * m).sum(1) / m.sum(1) + out.append(torch.nn.functional.normalize(v.float(), dim=-1).cpu().numpy()) + return np.concatenate(out) + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--a", required=True) + ap.add_argument("--b", required=True) + ap.add_argument("--show", type=int, default=15) + args = ap.parse_args() + load_env() + cap_gpu_memory(20) + import torch + from transformers import AutoModel, AutoTokenizer + + tok = AutoTokenizer.from_pretrained(MODEL) + model = AutoModel.from_pretrained(MODEL, dtype=torch.float16).cuda().eval() + A, B = read_jsonl(resolve_data(args.a)), read_jsonl(resolve_data(args.b)) + ea, eb = embed([text(r) for r in A], tok, model), embed([text(r) for r in B], tok, model) + sims = ea @ eb.T + best, idx = sims.max(1), sims.argmax(1) + + # Reference scale: known copies (validation items with a character-level copy in B) and random pairs + rng = np.random.default_rng(0) + rand = sims[rng.integers(0, len(A), 5000), rng.integers(0, len(B), 5000)] + print(f"A={args.a} ({len(A)}) B={args.b} ({len(B)}) model={MODEL}") + print(f"reference: random unrelated pairs median {np.median(rand):.3f}, 99th pct {np.percentile(rand, 99):.3f}") + for q in (50, 90, 99, 100): + print(f"nearest-neighbour similarity, {q:>3}th percentile: {np.percentile(best, q):.3f}") + for t in (0.85, 0.90, 0.95): + print(f"A items with a B neighbour >= {t}: {(best >= t).sum()}") + for i in np.argsort(-best)[: args.show]: + print(f"\nsim={best[i]:.3f}\n A {A[i]['uid']}: {A[i]['question'][:150]}\n options: {' | '.join(A[i]['options'])[:150]}" + f"\n B {B[idx[i]]['uid']}: {B[idx[i]]['question'][:150]}\n options: {' | '.join(B[idx[i]]['options'])[:150]}") + + +if __name__ == "__main__": + main() diff --git a/training/train/__init__.py b/training/train/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/training/train/bench_throughput.py b/training/train/bench_throughput.py new file mode 100644 index 0000000000000000000000000000000000000000..7b4bf17bd46dcacb0a17b6a0caf15580c1a0ed19 --- /dev/null +++ b/training/train/bench_throughput.py @@ -0,0 +1,154 @@ +"""Training throughput benchmark: tokens/s and peak memory for a model + training mode on this machine. + + python -m train.bench_throughput --model Qwen/Qwen3.5-4B --mode full + python -m train.bench_throughput --model models/gemma-4-E4B-it-text --mode freeze-embed + python -m train.bench_throughput --model models/gemma-4-E4B-it-text --mode lora + +Speed options (train/efficiency.py): --loss hf|chunked|liger, --ckpt-every N (1 = all layers, 0 = none), --compile. + +Modes: full every parameter trained + freeze-embed embedding tables frozen (for Gemma E-models: the 3.5B per-layer embeddings), rest trained + lora LoRA r=16 on all linear layers, base frozen + +Random token ids are used: step time does not depend on the text. The micro-batch is searched upward until the +memory cap is hit, then timed. AdamW states are kept in the parameter dtype (bf16); the result also reports the +extra memory fp32 optimizer states would need. Appends one JSON line per run to reports/throughput.jsonl. +""" +from __future__ import annotations + +import argparse +import json +import sys +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import ROOT, cap_gpu_memory, env_fingerprint, load_env, set_seed # noqa: E402 + + +def build(model_name: str, mode: str, lora_r: int, ckpt_every: int, compile_layers: bool): + import torch + from transformers import AutoModelForCausalLM + + from eval.scorers import resolve_revision + + revision, commit = resolve_revision(model_name, None) + model = AutoModelForCausalLM.from_pretrained(model_name, revision=revision, dtype=torch.bfloat16, + device_map="cuda") + model.config.use_cache = False + if mode == "lora": + from peft import LoraConfig, get_peft_model + + model = get_peft_model(model, LoraConfig(r=lora_r, lora_alpha=2 * lora_r, target_modules="all-linear", + lora_dropout=0.0, task_type="CAUSAL_LM")) + elif mode == "freeze-embed": + for name, p in model.named_parameters(): + if "embed_tokens" in name: # includes Gemma's per-layer embedding tables; lm_head is tied to it + p.requires_grad_(False) + from train.efficiency import configure + + configure(model, ckpt_every, compile_layers) + if mode == "lora": + model.enable_input_require_grads() + model.train() + return model, commit + + +def run_steps(model, opt, vocab: int, mb: int, seq: int, steps: int, loss_kind: str) -> float: + import torch + + from train.efficiency import lm_loss + + torch.cuda.synchronize() + t0 = time.time() + for _ in range(steps): + ids = torch.randint(100, vocab - 100, (mb, seq), device="cuda") + loss = lm_loss(model, ids, ids, loss_kind) + loss.backward() + opt.step() + opt.zero_grad(set_to_none=True) + torch.cuda.synchronize() + return time.time() - t0 + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--model", required=True) + ap.add_argument("--mode", choices=["full", "freeze-embed", "lora"], required=True) + ap.add_argument("--seq-len", type=int, default=2048) + ap.add_argument("--max-micro-batch", type=int, default=32) + ap.add_argument("--steps", type=int, default=10) + ap.add_argument("--lora-r", type=int, default=16) + ap.add_argument("--gpu-mem-gb", type=float) + ap.add_argument("--loss", choices=["hf", "chunked", "liger"], default="hf") + ap.add_argument("--ckpt-every", type=int, default=1) + ap.add_argument("--compile", action="store_true") + ap.add_argument("--optim", choices=["adamw", "adamw8bit"], default="adamw") + ap.add_argument("--tag", default="") + args = ap.parse_args() + load_env() + set_seed(1234) + cap_gpu_memory(args.gpu_mem_gb) + import torch + + model, commit = build(args.model, args.mode, args.lora_r, args.ckpt_every, args.compile) + total = sum(p.numel() for p in model.parameters()) + trainable = [p for p in model.parameters() if p.requires_grad] + n_train = sum(p.numel() for p in trainable) + if args.optim == "adamw8bit": # 8-bit optimizer states (torchao): ~4x less optimizer memory than fp32 + from torchao.optim import AdamW8bit + + opt = AdamW8bit(trainable, lr=1e-5) + else: + opt = torch.optim.AdamW(trainable, lr=1e-5, fused=True) + vocab = model.get_input_embeddings().num_embeddings + + best = None + mb = 1 + while mb <= args.max_micro_batch: + try: + run_steps(model, opt, vocab, mb, args.seq_len, 2, args.loss) # warm-up / fit check (and compile) + best = mb + mb *= 2 + except torch.OutOfMemoryError: + break + finally: + opt.zero_grad(set_to_none=True) + torch.cuda.empty_cache() + if best is None: + raise SystemExit("micro-batch 1 does not fit under the memory cap") + + # The failed probe can leave memory fragmented, so the largest fitting size may not fit again: back off. + while True: + torch.cuda.reset_peak_memory_stats() + try: + secs = run_steps(model, opt, vocab, best, args.seq_len, args.steps, args.loss) + break + except torch.OutOfMemoryError: + opt.zero_grad(set_to_none=True) + torch.cuda.empty_cache() + if best == 1: + raise + best //= 2 + tok_s = best * args.seq_len * args.steps / secs + res = { + "model": args.model, "commit": commit, "mode": args.mode, "loss": args.loss, "ckpt_every": args.ckpt_every, + "compile": args.compile, "optim": args.optim, "tag": args.tag, "seq_len": args.seq_len, "micro_batch": best, + "total_params": total, "trainable_params": n_train, + "tokens_per_s": round(tok_s, 1), "step_seconds": round(secs / args.steps, 2), + "peak_mem_gb": round(torch.cuda.max_memory_allocated() / 1024**3, 1), + "extra_gb_for_fp32_adam": round(n_train * 4 / 1024**3, 1), # bf16 -> fp32 for both moments + "days_per_1B_tokens": round(1e9 / tok_s / 86400, 2), + "env": env_fingerprint(), + } + out = ROOT / "reports" / "throughput.jsonl" + with open(out, "a") as f: + f.write(json.dumps(res) + "\n") + print(f"{args.model} [{args.mode} loss={args.loss} ckpt_every={args.ckpt_every} compile={args.compile} optim={args.optim}] " + f"seq={args.seq_len} mb={best}: {tok_s:,.0f} tok/s, " + f"peak {res['peak_mem_gb']} GB, trainable {n_train / 1e9:.2f}B / {total / 1e9:.2f}B, " + f"{res['days_per_1B_tokens']} days per 1B tokens") + + +if __name__ == "__main__": + main() diff --git a/training/train/efficiency.py b/training/train/efficiency.py new file mode 100644 index 0000000000000000000000000000000000000000..484f9ca6006e821fbba27c8783572ad3f5ec56ec --- /dev/null +++ b/training/train/efficiency.py @@ -0,0 +1,73 @@ +"""Training speed/memory options shared by the benchmark and the training scripts. + + loss="hf" transformers' built-in loss: materialises the full [tokens x vocab] logits (Gemma: 262k vocab) + loss="chunked" plain-PyTorch chunked cross-entropy; each chunk's logits are recomputed in backward + loss="liger" Liger fused linear + cross-entropy Triton kernel (never materialises the logits) + ckpt_every=N recompute activations of every Nth decoder layer only (1 = all layers, 0 = none) + compile=True torch.compile each decoder layer (regional compilation) +""" +from __future__ import annotations + +import torch +import torch.nn.functional as F +from torch.utils.checkpoint import checkpoint + + +def lm_parts(model): + """(decoder body, lm_head, final logit softcap) for plain and PEFT-wrapped causal LMs.""" + lm = model.get_base_model() if hasattr(model, "get_base_model") else model + return lm.model, lm.lm_head, getattr(lm.config, "final_logit_softcapping", None) + + +def _ce_chunk(h, w, y, softcap): + logits = (h @ w.t()).float() + if softcap: + logits = torch.tanh(logits / softcap) * softcap + return F.cross_entropy(logits, y, ignore_index=-100, reduction="sum") + + +def chunked_ce(h, w, y, softcap, chunk: int = 4096): + total = h.new_zeros((), dtype=torch.float32) + for i in range(0, h.shape[0], chunk): + total = total + checkpoint(_ce_chunk, h[i:i + chunk], w, y[i:i + chunk], softcap, use_reentrant=False) + return total / (y != -100).sum().clamp(min=1) + + +_liger = {} + + +def lm_loss(model, input_ids, labels, loss: str = "hf", attention_mask=None): + """Next-token cross-entropy, mean over non-ignored target tokens. All three variants give the same value.""" + if loss == "hf": + return model(input_ids=input_ids, attention_mask=attention_mask, labels=labels).loss + body, head, softcap = lm_parts(model) + h = body(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state + h = h[:, :-1].reshape(-1, h.shape[-1]) + y = labels[:, 1:].reshape(-1) + if loss == "chunked": + return chunked_ce(h, head.weight, y, softcap) + if loss == "liger": + from liger_kernel.transformers import LigerFusedLinearCrossEntropyLoss + + fn = _liger.setdefault(softcap, LigerFusedLinearCrossEntropyLoss(softcap=softcap)) + return fn(head.weight, h, y) + raise ValueError(loss) + + +def configure(model, ckpt_every: int = 1, compile_layers: bool = False) -> None: + body = lm_parts(model)[0] + if ckpt_every > 0: + model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False}) + for i, layer in enumerate(body.layers): + layer.gradient_checkpointing = i % ckpt_every == 0 + else: + model.gradient_checkpointing_disable() + if compile_layers: + # Layers differ (sliding vs global attention, linear-attention blocks), so dynamo compiles one graph per + # layer variant; past the default limit of 8 it silently falls back to eager. The limit is passed per call: + # the global config value was not honoured in the 11 Oct rerun. dynamic=False keeps one static graph per + # shape, so activation recomputation cannot pick a different (dynamic) graph than the forward pass used + # (pytorch/pytorch#166926). + limit = max(64, 4 * len(body.layers)) + for layer in body.layers: + layer.compile(dynamic=False, recompile_limit=limit) diff --git a/training/train/train.py b/training/train/train.py new file mode 100644 index 0000000000000000000000000000000000000000..ad80a01ee0f7d0c4794191098f5fa1de02cff268 --- /dev/null +++ b/training/train/train.py @@ -0,0 +1,359 @@ +"""Continued pretraining (mode: cpt) and MCQ fine-tuning (mode: sft) of the submission model. + + python -m train.train --config configs/train/sft_gmmlu.yaml [--max-steps N] [--out models/x] + +cpt documents from /*.parquet are shuffled, tokenised once ( text ) into a uint32 token file and + cut into fixed seq_len sequences (fixed shapes keep torch.compile to one graph). +sft each MCQ is rendered with the *inference* prompt (eval/prompts.py: same template, chat wrapping) and the loss is + the cross-entropy of the answer digit at the last prompt position, which is exactly what logit scoring reads. + Options can be re-shuffled every epoch (except options that refer to other options) to avoid position priors. + +Speed/memory options come from train/efficiency.py (measured in reports/throughput.jsonl). The optimizer is torchao +AdamW8bit with bf16 stochastic rounding (weights stay bf16; small updates are not rounded away). +Checkpoints are complete, loadable model directories (weights, tokenizer, chat template, SOURCE.json, TRAINING.json). +""" +from __future__ import annotations + +import argparse +import json +import math +import random +import shutil +import sys +import time +from pathlib import Path + +import numpy as np +import yaml + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from eval.common import ROOT, cap_gpu_memory, env_fingerprint, load_env, read_jsonl, set_seed # noqa: E402 + + +# ---------------------------------------------------------------- data +def prepare_cpt_tokens(cfg: dict, tok) -> np.memmap: + import pandas as pd + + data_dir = ROOT / cfg["data_dir"] + suffix = f"_limit{cfg['limit_docs']}" if cfg.get("limit_docs") else "" + path = data_dir / f"tokens_{Path(cfg['model']).name}{suffix}.u32" + if not path.exists(): + docs = pd.concat([pd.read_parquet(f, columns=["text"]) for f in sorted(data_dir.glob("*.parquet"))], + ignore_index=True)["text"].tolist() + random.Random(cfg["seed"]).shuffle(docs) + docs = docs[: cfg["limit_docs"]] if cfg.get("limit_docs") else docs + tmp = path.with_suffix(".tmp") + n = 0 + with open(tmp, "wb") as f: + for i in range(0, len(docs), 2000): + for ids in tok(docs[i:i + 2000], add_special_tokens=False)["input_ids"]: + arr = np.array([tok.bos_token_id] + ids + [tok.eos_token_id], dtype=np.uint32) + f.write(arr.tobytes()) + n += len(arr) + print(f"tokenised {min(i + 2000, len(docs)):,}/{len(docs):,} docs, {n / 1e6:.1f}M tokens", flush=True) + tmp.rename(path) + return np.memmap(path, dtype=np.uint32, mode="r") + + +def cpt_batches(tokens: np.memmap, seq_len: int, micro_batch: int): + import torch + + n_seq = len(tokens) // seq_len + i = 0 + while True: # cycles if the run asks for more tokens than the file holds (epoch > 1) + rows = [(i + k) % n_seq for k in range(micro_batch)] + i += micro_batch + x = np.stack([tokens[r * seq_len:(r + 1) * seq_len] for r in rows]).astype(np.int64) + yield {"input_ids": torch.from_numpy(x)} + + +def sft_examples(cfg: dict, tok, epoch: int) -> list[dict]: + from eval.prompts import PromptBuilder + from scripts.build_sft_gmmlu import REFERS_TO_OTHERS + + pb = PromptBuilder(tok, lang=cfg.get("lang", "en"), chat=cfg.get("chat", True)) + rng = random.Random(f"{cfg['seed']}|{epoch}") + label_ids = {} + out = [] + for f in cfg["data"]: + for r in read_jsonl(ROOT / f): + opts, ans = list(r["options"]), r["answer"] + tprobs = r.get("teacher_probs") # teacher's option distribution, in the stored option order + order = list(range(len(opts))) + if cfg.get("permute_options", True) and not any(REFERS_TO_OTHERS.search(o) for o in opts): + rng.shuffle(order) + opts, ans = [opts[j] for j in order], order.index(ans - 1) + 1 + tprobs = [tprobs[j] for j in order] if tprobs else None + ids = tok(pb.mcq(r, opts), add_special_tokens=pb.add_special_tokens)["input_ids"] + n = len(opts) + if n not in label_ids: + label_ids[n] = [tok.encode(str(k), add_special_tokens=False) for k in range(1, n + 1)] + assert all(len(t) == 1 for t in label_ids[n]), "answer labels must be single tokens" + out.append({"ids": ids, "target": label_ids[n][ans - 1][0], "labels": [t[0] for t in label_ids[n]], + "tprobs": tprobs}) + rng.shuffle(out) + return out + + +def trace_examples(cfg: dict, tok) -> list[dict]: + """Prompt (thinking-mode chat prompt, as at inference) + the model's own correct reasoning trace; loss on the trace.""" + from eval.prompts import PromptBuilder + + pb = PromptBuilder(tok, lang=cfg.get("lang", "en"), chat=True, chat_kwargs={"enable_thinking": True}) + out = [] + for f in cfg["data"]: + for r in read_jsonl(ROOT / f): + p_ids = tok(pb.mcq(r, r["options"]), add_special_tokens=False)["input_ids"] + c_ids = tok(r["completion"], add_special_tokens=False)["input_ids"] + if len(p_ids) + len(c_ids) > cfg.get("max_len", 4096): + continue + out.append({"ids": p_ids + c_ids, "n_prompt": len(p_ids)}) + random.Random(cfg["seed"]).shuffle(out) + return out + + +def trace_steps(examples: list[dict], batch_size: int, micro_tokens: int, pad_id: int): + import torch + + for s in range(0, len(examples) - batch_size + 1, batch_size): + chunk = sorted(examples[s:s + batch_size], key=lambda e: len(e["ids"])) + micros, cur = [], [] + for e in chunk: + if cur and max(len(x["ids"]) for x in cur + [e]) * (len(cur) + 1) > micro_tokens: + micros.append(cur) + cur = [] + cur.append(e) + micros.append(cur) + batches = [] + for m in micros: + L = max(len(e["ids"]) for e in m) + ids = torch.full((len(m), L), pad_id, dtype=torch.long) + mask = torch.zeros((len(m), L), dtype=torch.long) + labels = torch.full((len(m), L), -100, dtype=torch.long) + for i, e in enumerate(m): + n = len(e["ids"]) + ids[i, :n] = torch.tensor(e["ids"]) + mask[i, :n] = 1 + labels[i, e["n_prompt"]:n] = ids[i, e["n_prompt"]:n] + batches.append({"input_ids": ids, "attention_mask": mask, "labels": labels}) + yield batches + + +def sft_steps(examples: list[dict], batch_size: int, micro_tokens: int, pad_id: int): + """Yield optimizer steps; each is a list of right-padded micro-batches (token budget), batch_size examples.""" + import torch + + for s in range(0, len(examples) - batch_size + 1, batch_size): + chunk = sorted(examples[s:s + batch_size], key=lambda e: len(e["ids"])) + micros, cur = [], [] + for e in chunk: + if cur and max(len(x["ids"]) for x in cur + [e]) * (len(cur) + 1) > micro_tokens: + micros.append(cur) + cur = [] + cur.append(e) + micros.append(cur) + batches = [] + for m in micros: + L = max(len(e["ids"]) for e in m) + ids = torch.full((len(m), L), pad_id, dtype=torch.long) + mask = torch.zeros((len(m), L), dtype=torch.long) + for i, e in enumerate(m): + ids[i, : len(e["ids"])] = torch.tensor(e["ids"]) + mask[i, : len(e["ids"])] = 1 + b = {"input_ids": ids, "attention_mask": mask, + "last": torch.tensor([len(e["ids"]) - 1 for e in m]), + "target": torch.tensor([e["target"] for e in m])} + if all(e["tprobs"] is not None for e in m) and len({len(e["labels"]) for e in m}) == 1: + b["labels"] = torch.tensor([e["labels"] for e in m]) + b["tprobs"] = torch.tensor([e["tprobs"] for e in m], dtype=torch.float32) + batches.append(b) + yield batches + + +def sft_loss(model, batch, batch_size: int, kd_alpha: float = 0.0): + """Cross-entropy of the answer digit; with kd_alpha > 0 and teacher probabilities present, mixed with the KL + divergence from the teacher's distribution over the option labels (knowledge distillation).""" + import torch + import torch.nn.functional as F + + from train.efficiency import lm_parts + + body, head, softcap = lm_parts(model) + h = body(input_ids=batch["input_ids"], attention_mask=batch["attention_mask"]).last_hidden_state + h = h[torch.arange(h.shape[0], device=h.device), batch["last"]] + logits = head(h).float() + if softcap: + logits = torch.tanh(logits / softcap) * softcap + ce = F.cross_entropy(logits, batch["target"], reduction="none") + if kd_alpha > 0 and "tprobs" in batch: + student = torch.log_softmax(logits.gather(1, batch["labels"]), dim=-1) + teacher = batch["tprobs"].clamp_min(1e-8) + teacher = teacher / teacher.sum(-1, keepdim=True) + kl = (teacher * (teacher.log() - student)).sum(-1) + return ((1 - kd_alpha) * ce + kd_alpha * kl).sum() / batch_size + return ce.sum() / batch_size + + +# ---------------------------------------------------------------- model / saving +def load(cfg: dict): + import torch + from transformers import AutoModelForCausalLM, AutoTokenizer + + from train.efficiency import configure + + path = str(ROOT / cfg["model"]) if (ROOT / cfg["model"]).exists() else cfg["model"] + model = AutoModelForCausalLM.from_pretrained(path, dtype=torch.bfloat16, device_map="cuda") + model.config.use_cache = False + if cfg.get("lora"): # adapters on all linear layers; merged into the base weights when saved + from peft import LoraConfig, get_peft_model + + lc = cfg["lora"] + model = get_peft_model(model, LoraConfig(r=lc["r"], lora_alpha=lc.get("alpha", 2 * lc["r"]), + target_modules=lc.get("targets", "all-linear"), + lora_dropout=lc.get("dropout", 0.0), task_type="CAUSAL_LM")) + elif cfg.get("freeze_embeddings", True): + for name, p in model.named_parameters(): + if "embed_tokens" in name: # incl. Gemma per-layer embedding tables; lm_head is tied to embed_tokens + p.requires_grad_(False) + configure(model, cfg.get("ckpt_every", 1), cfg.get("compile", False)) + if cfg.get("lora") and cfg.get("ckpt_every", 1) > 0: + model.enable_input_require_grads() + model.train() + tok = AutoTokenizer.from_pretrained(path) + return model, tok, Path(path) + + +def save(model, tok, src: Path, out: Path, info: dict) -> None: + out.mkdir(parents=True, exist_ok=True) + if hasattr(model, "merge_and_unload"): # LoRA: keep the adapter, then save plain merged weights too + model.save_pretrained(out / "adapter") + model = model.merge_and_unload() + model.config.use_cache = True + model.save_pretrained(out, safe_serialization=True) + model.config.use_cache = False + tok.save_pretrained(out) + for f in src.glob("chat_template.*"): + shutil.copy(f, out / f.name) + source = json.loads((src / "SOURCE.json").read_text()) if (src / "SOURCE.json").exists() else {} + source["model_id"] = f"{info['run']}/{out.name}@{source.get('source_commit', 'unknown')[:10]}" + source.setdefault("derived_from", []).append(str(src.relative_to(ROOT)) if src.is_relative_to(ROOT) else str(src)) + (out / "SOURCE.json").write_text(json.dumps(source, indent=2)) + (out / "TRAINING.json").write_text(json.dumps(info, indent=2, default=str)) + print(f"saved {out}", flush=True) + + +def lr_at(step: int, total: int, cfg: dict) -> float: + warm, peak, floor = cfg["warmup_steps"], cfg["lr"], cfg["lr"] * cfg.get("min_lr_ratio", 0.1) + if step < warm: + return peak * (step + 1) / warm + t = (step - warm) / max(total - warm, 1) + return floor + 0.5 * (peak - floor) * (1 + math.cos(math.pi * min(t, 1.0))) + + +# ---------------------------------------------------------------- main +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--config", required=True) + ap.add_argument("--out", help="override out_dir") + ap.add_argument("--max-steps", type=int, help="stop early (smoke tests)") + ap.add_argument("--limit-docs", type=int, help="cpt: tokenise only N documents (smoke tests)") + ap.add_argument("--data", nargs="+", help="sft: override the training files") + args = ap.parse_args() + cfg = yaml.safe_load(open(args.config)) + cfg["out_dir"] = args.out or cfg["out_dir"] + cfg["limit_docs"] = args.limit_docs or cfg.get("limit_docs") + if args.data: + cfg["data"] = args.data + load_env() + set_seed(cfg["seed"]) + cap_gpu_memory(cfg.get("gpu_mem_gb")) + import torch + from torchao.optim import AdamW8bit + + from train.efficiency import lm_loss + + out = ROOT / cfg["out_dir"] + out.mkdir(parents=True, exist_ok=True) + (out / "config.yaml").write_text(yaml.safe_dump(cfg)) + model, tok, src = load(cfg) + trainable = [p for p in model.parameters() if p.requires_grad] + opt = AdamW8bit(trainable, lr=cfg["lr"], betas=(0.9, 0.95), weight_decay=cfg.get("weight_decay", 0.0), + bf16_stochastic_round=True) + + if cfg["mode"] == "cpt": + tokens = prepare_cpt_tokens(cfg, tok) + per_step = cfg["seq_len"] * cfg["micro_batch"] * cfg["grad_accum"] + total = int(min(cfg["max_tokens"], len(tokens) * cfg.get("epochs", 1)) // per_step) + it = cpt_batches(tokens, cfg["seq_len"], cfg["micro_batch"]) + steps = ([next(it) for _ in range(cfg["grad_accum"])] for _ in range(total)) + save_every = max(1, int(cfg.get("save_every_tokens", 1e18) // per_step)) + elif cfg["mode"] == "trace": + ex = trace_examples(cfg, tok) + per_epoch = len(ex) // cfg["batch_size"] + total = per_epoch * cfg["epochs"] + steps = (s for _ in range(cfg["epochs"]) + for s in trace_steps(ex, cfg["batch_size"], cfg["micro_tokens"], tok.pad_token_id)) + save_every = per_epoch + print(f"trace examples: {len(ex)}", flush=True) + else: + n_ex = len(sft_examples(cfg, tok, 0)) + per_epoch = n_ex // cfg["batch_size"] + total = per_epoch * cfg["epochs"] + steps = (s for e in range(cfg["epochs"]) + for s in sft_steps(sft_examples(cfg, tok, e), cfg["batch_size"], cfg["micro_tokens"], + tok.pad_token_id)) + save_every = per_epoch # end of every epoch + if args.max_steps: + total = min(total, args.max_steps) + if cfg.get("lora"): + save_every = total + print(f"{cfg['mode']}: {total} optimizer steps, " + f"{sum(p.numel() for p in trainable) / 1e9:.2f}B trainable params", flush=True) + + info = {"run": Path(cfg["out_dir"]).name, "config": cfg, "env": env_fingerprint(), "tokens_seen": 0} + log = open(out / "log.jsonl", "a") + t0, tokens_seen = time.time(), 0 + for step, micro_batches in enumerate(steps): + if step >= total: + break + for g in opt.param_groups: # torchao keeps lr as a tensor: update in place + lr = lr_at(step, total, cfg) + g["lr"].fill_(lr) if torch.is_tensor(g["lr"]) else g.__setitem__("lr", lr) + loss_sum = 0.0 + for b in micro_batches: + b = {k: v.cuda(non_blocking=True) for k, v in b.items()} + if cfg["mode"] == "cpt": + loss = lm_loss(model, b["input_ids"], b["input_ids"], cfg.get("loss", "liger")) / len(micro_batches) + tokens_seen += b["input_ids"].numel() + elif cfg["mode"] == "trace": + loss = lm_loss(model, b["input_ids"], b["labels"], cfg.get("loss", "liger"), b["attention_mask"]) / len(micro_batches) + tokens_seen += int(b["attention_mask"].sum()) + else: + loss = sft_loss(model, b, cfg["batch_size"], cfg.get("kd_alpha", 0.0)) + tokens_seen += int(b["attention_mask"].sum()) + loss.backward() + loss_sum += loss.item() + gnorm = torch.nn.utils.clip_grad_norm_(trainable, cfg.get("max_grad_norm", 1.0)).item() + opt.step() + opt.zero_grad(set_to_none=True) + if step % cfg.get("log_every", 10) == 0 or step == total - 1: + el = time.time() - t0 + rec = {"step": step, "loss": round(loss_sum, 4), "lr": float(opt.param_groups[0]["lr"]), "grad_norm": round(gnorm, 3), + "tokens": tokens_seen, "tok_per_s": round(tokens_seen / el, 1), + "mem_gb": round(torch.cuda.max_memory_allocated() / 1024**3, 1), "elapsed_min": round(el / 60, 1)} + log.write(json.dumps(rec) + "\n") + log.flush() + print(rec, flush=True) + if (step + 1) % save_every == 0 or step == total - 1: + info.update(step=step + 1, total_steps=total, tokens_seen=tokens_seen) + ckpt = out / f"step-{step + 1}" + save(model, tok, src, ckpt, info) + keep = cfg.get("keep_checkpoints", 3) + old = sorted(out.glob("step-*"), key=lambda p: int(p.name.split("-")[1]))[:-keep] + for p in old: + shutil.rmtree(p) + print(f"done: {tokens_seen / 1e6:.1f}M tokens in {(time.time() - t0) / 60:.1f} min", flush=True) + + +if __name__ == "__main__": + main()