Experiment run scripts
Each script is the exact queue that produced a reported result or a submission. They run from any directory
(cd to the repo root themselves), skip steps whose outputs already exist, and log to results/logs/.
Reusable tools (data building, evaluation, overlap checks) live one level up in scripts/ and in eval/, train/,
inference/.
| Script | Date | Produced | Where reported |
|---|---|---|---|
run_baselines.sh |
10 Oct | Zero-shot baselines of 10 candidate models (dev, Global-MMLU si val, Belebele) |
reports/baselines_zeroshot_logit.md |
run_experiments.sh |
10 Oct | Prompting experiments on the top 3 models: Sinhala instruction, fixed 5-shot, generation, cloze | reports/experiments_prompting.md |
run_phase1.sh |
10 Oct | Text-only Gemma 4 E4B (7.463B) check; offline submission pipeline test on dev; first training-speed benchmark | reports/throughput.jsonl |
run_bench_speedups.sh |
10 Oct | Training speed with Liger loss, selective recomputation, torch.compile |
reports/throughput.jsonl |
run_bench_gemma_compile.sh |
11 Oct | Gemma 4 E4B speed after the compile recompile-limit fix | reports/throughput.jsonl |
run_bench_memory.sh |
11 Oct | Speed/memory with 8-bit AdamW and every-2nd-layer recomputation | reports/throughput.jsonl |
run_sft_gmmlu.sh |
12 Oct | Full fine-tuning on Global-MMLU (negative result: dev −2.9) | reports/sft_gmmlu_v1.md |
run_abc.sh |
12 Oct | Model A (LoRA on Global-MMLU, basis of submissions 2 and 4); dev 5-fold LoRA (B); continued-pretraining pilot (C) | reports/experiments_abc.md |
download_teachers.sh |
18 Oct | Downloads of the teacher candidates (data generation only, never inference) | — |
run_v2_and_teachers.sh |
18 Oct | v2 (LoRA on Global-MMLU + all dev; submission 3) and its test predictions; teacher evaluations | reports/teachers.md |
run_thinking_tests.sh |
19 Oct | Thinking mode: model A and base model on the Global-MMLU academic proxy and a dev sample | PLAN.md §2b |
run_softlabels_pilot_sub4_kd.sh |
19 Oct | Gemma 4 31B soft labels for Global-MMLU; synthetic MCQ pilot; submission 4 predictions; distillation LoRA (interrupted by the 19 Oct crash) | data/synth/pilot.stats.json, PLAN.md |
run_sub5_thinking_kd.sh |
19–20 Oct | Submission 5 (base model + adapter A, extracted by scripts/extract_lora.py); base-model thinking on dev and non-academic questions; distillation LoRA rerun (no gain: dev −0.9) |
PLAN.md, results/ |
run_sub6.sh |
20 Oct | Submission 6 (base-model thinking for medium + hard, adapter A fast vote for easy) | PLAN.md §2c |
run_easy_sc.sh |
20 Oct | Base-model thinking on all dev (Easy decision); self-consistency (3 samples) on the Global-MMLU proxies | PLAN.md §2c |
run_final_candidate.sh |
21 Oct | Final candidate A: base model only (7.463B), thinking for all levels, self-consistency for hard | PLAN.md |
run_final_candidate_gated.sh |
21–22 Oct | Final candidate B: as A, with agreement-gated self-consistency on all levels | FUTURE_WORK.md §1 |
Submissions (all chosen on dev / validation, never on test):
| # | Config | Test macro avg (easy / medium / hard) |
|---|---|---|
| 1 | configs/submission.yaml (zero-shot Gemma 4 E4B) |
51.59 |
| 2 | configs/submission_lora_gmmlu_vote.yaml (model A, 4-order vote) |
54.59 (55.30 / 49.51 / 58.97) |
| 3 | configs/submission_v2.yaml (v2, 4-order vote) |
54.85 (54.83 / 50.77 / 58.97) |
| 4 | configs/submission_A_think_hard.yaml (model A; thinking for hard) |
55.83 (55.30 / 49.51 / 62.68) |
| 5 | configs/submission_sub5_route.yaml (base + adapter A; base-model thinking without adapter for hard) |
57.68 (55.14 / 49.87 / 68.04) |
| 6 | configs/submission_sub6_think_medhard.yaml (as 5, thinking also for medium) |
59.22 (55.14 / 54.91 / 67.63) |
The final submission is reproduced with ./run_submission.sh at the repository root.