| --- |
| library_name: transformers |
| tags: |
| - gated-deltanet |
| - mixerloop |
| --- |
| # MixerLoop |
|
|
| Canonical GDN, MixerLoop T4 and FullLoop T4 checkpoints, organized by architecture, dataset, |
| processed-token budget and training seed. Directories containing only `.gitkeep` |
| are planned experiments, not available models. Historical releases and their |
| original evaluation results are preserved unchanged under `legacy/`. |
|
|
| ## Loading |
|
|
| Directories containing `model.safetensors` provide HF weights; the architectures require FLA and the |
| MixerLoop model registration code. Install the dependencies from |
| [MixerLoop commit 54a37b8](https://github.com/ruhai-lin/MixerLoop/tree/54a37b807601e690bc8d430cbdfbd87252f9bbbe) |
| and run from that checkout (or install it as a package). Plain Transformers |
| without these registrations does not recognize these model types. |
|
|
| ```python |
| import fla.models |
| import custom_models |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| repo = "ruhai-lin/MixerLoop" |
| run = "mixerloop-13m/climbmix-10B-s1337" |
| tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=run) |
| model = AutoModelForCausalLM.from_pretrained(repo, subfolder=run) |
| ``` |
|
|
| Tokenizer and generation files are the actual `save_pretrained()` outputs. |
| Native training checkpoints, optimizer states and logs are not published here. |
|
|
| ## Evaluation and budgets |
|
|
| `eval/core_eval.csv` contains all 22 Karpathy CORE tasks and the `Core_v2` |
| aggregate, stored as fractions, not percentages. Evaluation uses the canonical |
| evaluator at the commit above: official Karpathy bundle, in-memory v2 baseline |
| corrections (CommonsenseQA 40.3%, LSAT AR 25%, language identification 25%), |
| FP32 weights and BF16 autocast. Historical scores in `legacy/` retain their |
| original protocols and should not be assumed to be CORE v2. |
|
|
| First-batch runs use context 1024 and global batch 128. The 1B runs contain |
| 7,630 optimizer steps (1,000,079,360 processed tokens); the 10B run contains |
| 76,294 steps (10,000,007,168 processed tokens), all seed 1337. |
| Actual parameters: MixerLoop-13m 12,896,380; MixerLoop-100m 100,447,266; |
| GDN-100m 100,444,194. Other directories are reserved experiment labels, |
| not claims about completed training or measured parameter counts. |
|
|
| ## CORE v2 comparison |
|
|
| Scores are centered CORE v2 scores, stored as fractions. The 13M and 100M models were trained on ClimbMix 10B; the 100M columns use seed 1337. |
|
|
| Because 13M models show greater run-to-run noise, we ran three seeds (42, 1337, 2026) and report their mean. |
|
|
| | Task | 13M GDN | 13M MixerLoop | 13M FullLoop | 100M GDN | 100M MixerLoop | 100M FullLoop | |
| |---|---:|---:|---:|---:|---:|---:| |
| | hellaswag_zeroshot | 0.034898 | 0.033526 | 0.029675 | 0.132842 | 0.133771 | 0.167231 | |
| | jeopardy | 0.000315 | 0.000157 | 0.000630 | 0.007085 | 0.006613 | 0.010392 | |
| | bigbench_qa_wikidata | 0.040106 | 0.031396 | 0.071863 | 0.214950 | 0.257074 | 0.282811 | |
| | arc_easy | 0.155069 | 0.152076 | 0.151702 | 0.317621 | 0.341190 | 0.385522 | |
| | arc_challenge | -0.037922 | -0.054228 | -0.037543 | -0.001138 | 0.018203 | 0.045506 | |
| | copa | -0.113333 | -0.093333 | -0.120000 | 0.000000 | 0.020000 | 0.020000 | |
| | commonsense_qa | -0.180258 | -0.234217 | -0.246564 | -0.244278 | -0.015177 | -0.306011 | |
| | piqa | 0.162858 | 0.167211 | 0.166123 | 0.316649 | 0.334059 | 0.355822 | |
| | openbook_qa | 0.035556 | 0.041778 | 0.042667 | 0.090667 | 0.088000 | 0.098667 | |
| | lambada_openai | 0.105764 | 0.117407 | 0.116825 | 0.253639 | 0.246264 | 0.263536 | |
| | hellaswag | 0.028923 | 0.029277 | 0.023479 | 0.125938 | 0.126734 | 0.163513 | |
| | winograd | 0.072039 | 0.030525 | 0.050061 | 0.172161 | 0.113553 | 0.172161 | |
| | winogrande | 0.027624 | 0.020258 | 0.031834 | 0.027624 | 0.000789 | 0.029203 | |
| | bigbench_dyck_languages | 0.001333 | 0.008333 | 0.005667 | 0.128000 | 0.172000 | 0.093000 | |
| | agi_eval_lsat_ar | -0.051208 | -0.004831 | -0.016425 | -0.014493 | 0.072464 | 0.049275 | |
| | bigbench_cs_algorithms | 0.239394 | 0.306566 | 0.221717 | 0.351515 | 0.351515 | 0.352273 | |
| | bigbench_operators | 0.134921 | 0.139683 | 0.139682 | 0.095238 | 0.090476 | 0.090476 | |
| | bigbench_repeat_copy_logic | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | |
| | squad | 0.013844 | 0.008514 | 0.011542 | 0.070766 | 0.047114 | 0.049669 | |
| | coqa | 0.044135 | 0.031484 | 0.036077 | 0.113366 | 0.116623 | 0.123888 | |
| | boolq | -0.408874 | -0.361661 | -0.354418 | -0.187832 | -0.059070 | -0.054241 | |
| | bigbench_language_identification | 0.001689 | 0.007733 | 0.002755 | -0.003467 | 0.010000 | 0.002533 | |
| | **Core** | **0.013949** | **0.017166** | **0.014880** | **0.089402** | **0.112373** | **0.108874** | |
| |
| [Download the comparison CSV](eval/core_comparison.csv). |
| |
