MixerLoop

Canonical GDN and MixerLoop T4 checkpoints, organized by architecture, dataset, processed-token budget and training seed. Directories containing only .gitkeep are planned experiments, not available models. Historical releases and their original evaluation results are preserved unchanged under legacy/.

Loading

These are standard HF weights, but the architectures require FLA and the MixerLoop model registration code. Install the dependencies from MixerLoop commit 24b6a923 and run from that checkout (or install it as a package). Plain Transformers without these registrations does not recognize these model types.

import fla.models
import custom_models
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "ruhai-lin/MixerLoop"
run = "mixerloop-13m/climbmix-10B-s1337"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=run)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=run)

Tokenizer and generation files are the actual save_pretrained() outputs. Training checkpoints, optimizer states and logs are not published here.

Evaluation and budgets

eval/core_eval.csv contains all 22 Karpathy CORE tasks and the Core_v2 aggregate, stored as fractions, not percentages. Evaluation uses the canonical evaluator at the commit above: official Karpathy bundle, in-memory v2 baseline corrections (CommonsenseQA 40.3%, LSAT AR 25%, language identification 25%), FP32 weights and BF16 autocast. Historical scores in legacy/ retain their original protocols and should not be assumed to be CORE v2.

First-batch runs use context 1024 and global batch 128. The 1B runs contain 7,630 optimizer steps (1,000,079,360 processed tokens); the 10B run contains 76,294 steps (10,000,007,168 processed tokens), all seed 1337. Actual parameters: MixerLoop-13m 12,896,380; MixerLoop-100m 100,447,266; GDN-100m 100,444,194. Other directories are reserved experiment labels, not claims about completed training or measured parameter counts.

CORE v2 comparison

Scores are centered CORE v2 scores on the same ClimbMix 10B / seed1337 evaluation setup.

Task	13M GDN	13M MixerLoop	13M FullLoop	100M GDN	100M MixerLoop	100M FullLoop
hellaswag_zeroshot	0.037509	0.032862	0.030472	0.132842	0.133771	0.167231
jeopardy	0.000000	0.000000	0.001417	0.007085	0.006613	0.010392
bigbench_qa_wikidata	0.036908	0.050342	0.071010	0.214950	0.257074	0.282811
arc_easy	0.144220	0.153760	0.145342	0.317621	0.341190	0.385522
arc_challenge	-0.028441	-0.053470	-0.039818	-0.001138	0.018203	0.045506
copa	-0.100000	-0.120000	-0.180000	0.000000	0.020000	0.020000
commonsense_qa	-0.225072	-0.226443	-0.186659	-0.244278	-0.015177	-0.306011
piqa	0.158868	0.171926	0.170838	0.316649	0.334059	0.355822
openbook_qa	0.034667	0.018667	0.024000	0.090667	0.088000	0.098667
lambada_openai	0.105376	0.112168	0.122647	0.253639	0.246264	0.263536
hellaswag	0.028746	0.024763	0.022107	0.125938	0.126734	0.163513
winograd	0.091575	0.018315	0.076923	0.172161	0.113553	0.172161
winogrande	0.010260	0.043410	0.054459	0.027624	0.000789	0.029203
bigbench_dyck_languages	0.003000	0.005000	0.005000	0.128000	0.172000	0.093000
agi_eval_lsat_ar	-0.055072	0.037681	-0.002899	-0.014493	0.072464	0.049275
bigbench_cs_algorithms	0.325000	0.312879	0.312121	0.351515	0.351515	0.352273
bigbench_operators	0.142857	0.152381	0.180952	0.095238	0.090476	0.090476
bigbench_repeat_copy_logic	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
squad	0.004730	0.010028	0.012867	0.070766	0.047114	0.049669
coqa	0.040962	0.023300	0.047225	0.113366	0.116623	0.123888
boolq	-0.354418	-0.203927	-0.229680	-0.187832	-0.059070	-0.054241
bigbench_language_identification	-0.002400	0.008133	0.010533	-0.003467	0.010000	0.002533
Core	0.018149	0.025990	0.029494	0.089402	0.112373	0.108874