| # Team AlphaPack — SinhalaMMLU Gemma 4 E4B V2 |
|
|
| **Status: audited submission CANDIDATE; full clean-room prediction reproduction NOT yet demonstrated.** |
| Team identifier: AlphaPack. **Before submission: add your real team-member details and confirm there are no team adapters/extra training assets omitted.** |
|
|
| ## Submission |
|
|
| - Base model: [`google/gemma-4-E4B-it`](https://huggingface.co/google/gemma-4-E4B-it) |
| - Revision: `ee0ef6023621cff504d758262d4e04895a5af4a2` |
| - Unique inference-time parameter count verified in the source recovery notebook: **7,941,100,832** (7.9411B; below 8,000,000,000), including vision/audio modules. **No additional trainable inference adapter is included in this package.** |
| - Technique: native Gemma 4 thinking (`enable_thinking=True`) + exact **V2 explanation-first / last-line `ANSWER: X`** prompt, with automatic invalid-answer retries. The text is SHA-256 pinned in `inference.py`. |
| - Decoding: greedy (`do_sample=False`); first pass batch size 4, 2,048 new tokens; strict targeted retries batch size 1, 4,096 new tokens. The first-pass settings are reconstructed from research notebook material, **not independently proven identical to the historical all-2,206 run**. |
| - Hardware used for historical research: Kaggle **NVIDIA T4 ×2**; FP16; SDPA. Original targeted retry set a four-hour run limit, not a measured full-set runtime. **Actual complete fresh-run runtime: not yet measured.** |
| - Seed: 2026 (reconstructed setting; confirm against initial production records). |
| - Dataset used for evaluation: SinhalaMMLU Shared Task public questions, downloaded by the script from `DATA_REPO`. |
| - Additional training / retrieval data: **none in this system package**. Confirm any other teammates' work before publication. |
| - Team member details: **Team AlphaPack — Umesh Yapa, Esandu Epa, Lasal jayasinghe, Amasha Ranasinghe**. |
|
|
| ## Files |
|
|
| - `inference.py` — best-effort self-contained fresh inference candidate; loads every `.json` in `DATA_REPO`, downloads pinned model from HF, uses native thinking and V2 prompt, retries invalid answers, and writes `predictions.jsonl` in the required numeric-label format. |
| - `predictions.jsonl` — **unchanged** original user-provided 2,238-row final submission candidate; SHA-256 `dcf8cf70066f06b56c319a5eb6604bbeb823cc70096ab8a5c40e8c6d05e06b00`. |
| - `requirements.txt` — reconstructed package versions from the research environment. |
| - `recovery/evidence/` — independently verified historical first-pass 2,206 + 32 failures package, merge audit, hashes and merge source code. |
| - `recovery/MIDDLE_E4B_RETRY32_COLDSTART.ipynb` and `.py` — verbatim original 32-question historical recovery code, requiring the original combined-2,206 package and 41 unchanged JSON question files as inputs. Its source contains the final strict parser, per-question logs, model hash, dataset checks and successful-only final export logic. **This notebook has no embedded execution outputs.** |
| - `training/README.md` and `data/README.md` — no fine-tuning, adapters or extra training data claimed; review if other team components were used. |
| - `REPRODUCIBILITY_STATUS.md` — limitations and outstanding evidence. |
| - `check_submission.py`, `verify_final_predictions.py` — file integrity / output validation helpers. |
|
|
| ## Fresh environment reproduction |
|
|
| 1. On Kaggle, choose **GPU T4 ×2**, enable Internet, and accept the Gemma license through your Hugging Face account if required. The model is gated. Authenticate externally through `HF_TOKEN` or pre-existing Hugging Face login; never place tokens in code. |
| 2. Install `pip install -r requirements.txt`. |
| 3. The organizers change **only `DATA_REPO`** in `inference.py`; then run `python inference.py` from a clean working directory. It downloads the model from Hugging Face and produces `predictions.jsonl`. |
| 4. Keep an immutable copy of this repository's original predictions outside the run directory. Running the script overwrites the `predictions.jsonl` in its working directory. |
| 5. Compare the reproduced predictions to the original 2,238-row file with `python check_submission.py ORIGINAL.jsonl GENERATED.jsonl`. Byte-identical SHA is the strongest evidence; report any differences, never conceal them. |
| |
| ## Critical reproducibility limitations |
| |
| The original `E4B_COMBINED_2206_WITH_FAILED32.zip` is now included under `recovery/evidence/`. It provides the original 2,206 valid predictions, 32 failed attempts, 32 raw failure outputs, merged first-pass audit, dataset SHA-256 manifest and original merge script. An independent offline audit verified **all 2,206 entries** match `predictions.jsonl` exactly and the remaining **32 indices** are disjoint and have matching question metadata. The first-pass merge audit confirms the same V2 prompt hash as RETRY32. **Three original shard ZIPs are still not provided**, although their names and SHA-256 hashes are recorded. The complete original first-pass generator is still not available as one final standalone script; the integrated `inference.py` is reconstructed, and a full clean-room GPU run has not been executed here. Clean-room inference has **not** been run or checked against the exact submitted file here. This README is intentionally not a claim of exact output reproducibility. |
|
|
| ## Required repository links |
|
|
| - System repository: `https://huggingface.co/SinhalaMMLU-SharedTask/team-AlphaPack` |
| - Model weights: `https://huggingface.co/google/gemma-4-E4B-it` — pinned revision above; no fine-tuned adapter claimed. |
|
|
