update evaluation results

Browse files

Files changed (5) hide show

README.md +34 -27
eval/core_comparison.csv +24 -24
mixerloop-450m/fineweb-edu-100B-s1337/README.md +1 -1
mixerloop-450m/fineweb-edu-100B-s1337/eval/core_eval.csv +24 -0
mixerloop-450m/fineweb-edu-100B-s1337/eval/lm_eval.csv +8 -0

README.md CHANGED Viewed

@@ -51,31 +51,38 @@ not claims about completed training or measured parameter counts.
 ## CORE v2 comparison
-Scores are centered CORE v2 scores on the same ClimbMix 10B / seed1337 evaluation setup.
-| Task | 13M GDN | 13M MixerLoop | 13M FullLoop | 100M GDN | 100M MixerLoop | 100M FullLoop |
-|---|---:|---:|---:|---:|---:|---:|
-| hellaswag_zeroshot | 0.037509 | 0.032862 | 0.030472 | 0.132842 | 0.133771 | 0.167231 |
-| jeopardy | 0.000000 | 0.000000 | 0.001417 | 0.007085 | 0.006613 | 0.010392 |
-| bigbench_qa_wikidata | 0.036908 | 0.050342 | 0.071010 | 0.214950 | 0.257074 | 0.282811 |
-| arc_easy | 0.144220 | 0.153760 | 0.145342 | 0.317621 | 0.341190 | 0.385522 |
-| arc_challenge | -0.028441 | -0.053470 | -0.039818 | -0.001138 | 0.018203 | 0.045506 |
-| copa | -0.100000 | -0.120000 | -0.180000 | 0.000000 | 0.020000 | 0.020000 |
-| commonsense_qa | -0.225072 | -0.226443 | -0.186659 | -0.244278 | -0.015177 | -0.306011 |
-| piqa | 0.158868 | 0.171926 | 0.170838 | 0.316649 | 0.334059 | 0.355822 |
-| openbook_qa | 0.034667 | 0.018667 | 0.024000 | 0.090667 | 0.088000 | 0.098667 |
-| lambada_openai | 0.105376 | 0.112168 | 0.122647 | 0.253639 | 0.246264 | 0.263536 |
-| hellaswag | 0.028746 | 0.024763 | 0.022107 | 0.125938 | 0.126734 | 0.163513 |
-| winograd | 0.091575 | 0.018315 | 0.076923 | 0.172161 | 0.113553 | 0.172161 |
-| winogrande | 0.010260 | 0.043410 | 0.054459 | 0.027624 | 0.000789 | 0.029203 |
-| bigbench_dyck_languages | 0.003000 | 0.005000 | 0.005000 | 0.128000 | 0.172000 | 0.093000 |
-| agi_eval_lsat_ar | -0.055072 | 0.037681 | -0.002899 | -0.014493 | 0.072464 | 0.049275 |
-| bigbench_cs_algorithms | 0.325000 | 0.312879 | 0.312121 | 0.351515 | 0.351515 | 0.352273 |
-| bigbench_operators | 0.142857 | 0.152381 | 0.180952 | 0.095238 | 0.090476 | 0.090476 |
-| bigbench_repeat_copy_logic | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
-| squad | 0.004730 | 0.010028 | 0.012867 | 0.070766 | 0.047114 | 0.049669 |
-| coqa | 0.040962 | 0.023300 | 0.047225 | 0.113366 | 0.116623 | 0.123888 |
-| boolq | -0.354418 | -0.203927 | -0.229680 | -0.187832 | -0.059070 | -0.054241 |
-| bigbench_language_identification | -0.002400 | 0.008133 | 0.010533 | -0.003467 | 0.010000 | 0.002533 |
-| **Core** | **0.018149** | **0.025990** | **0.029494** | **0.089402** | **0.112373** | **0.108874** |

 ## CORE v2 comparison
+Scores are centered CORE v2 scores, stored as fractions. The 13M and 100M models were trained on ClimbMix 10B; the 100M columns use seed 1337.
+Because 13M models show greater run-to-run noise, we ran three seeds (42, 1337, 2026) and report their mean.
+The rightmost column is MixerLoop 450M trained on FineWeb-Edu 100B / seed1337, so its dataset and training budget differ from the smaller models.
+| Task | 13M GDN | 13M MixerLoop | 13M FullLoop | 100M GDN | 100M MixerLoop | 100M FullLoop | 450M MixerLoop |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| hellaswag_zeroshot | 0.034898 | 0.033526 | 0.029675 | 0.132842 | 0.133771 | 0.167231 | 0.323773 |
+| jeopardy | 0.000315 | 0.000157 | 0.000630 | 0.007085 | 0.006613 | 0.010392 | 0.051488 |
+| bigbench_qa_wikidata | 0.040106 | 0.031396 | 0.071863 | 0.214950 | 0.257074 | 0.282811 | 0.536932 |
+| arc_easy | 0.155069 | 0.152076 | 0.151702 | 0.317621 | 0.341190 | 0.385522 | 0.512907 |
+| arc_challenge | -0.037922 | -0.054228 | -0.037543 | -0.001138 | 0.018203 | 0.045506 | 0.130830 |
+| copa | -0.113333 | -0.093333 | -0.120000 | 0.000000 | 0.020000 | 0.020000 | 0.300000 |
+| commonsense_qa | -0.180258 | -0.234217 | -0.246564 | -0.244278 | -0.015177 | -0.306011 | -0.251137 |
+| piqa | 0.162858 | 0.167211 | 0.166123 | 0.316649 | 0.334059 | 0.355822 | 0.404788 |
+| openbook_qa | 0.035556 | 0.041778 | 0.042667 | 0.090667 | 0.088000 | 0.098667 | 0.165333 |
+| lambada_openai | 0.105764 | 0.117407 | 0.116825 | 0.253639 | 0.246264 | 0.263536 | 0.398603 |
+| hellaswag | 0.028923 | 0.029277 | 0.023479 | 0.125938 | 0.126734 | 0.163513 | 0.313948 |
+| winograd | 0.072039 | 0.030525 | 0.050061 | 0.172161 | 0.113553 | 0.172161 | 0.384615 |
+| winogrande | 0.027624 | 0.020258 | 0.031834 | 0.027624 | 0.000789 | 0.029203 | 0.097080 |
+| bigbench_dyck_languages | 0.001333 | 0.008333 | 0.005667 | 0.128000 | 0.172000 | 0.093000 | 0.032000 |
+| agi_eval_lsat_ar | -0.051208 | -0.004831 | -0.016425 | -0.014493 | 0.072464 | 0.049275 | 0.020290 |
+| bigbench_cs_algorithms | 0.239394 | 0.306566 | 0.221717 | 0.351515 | 0.351515 | 0.352273 | 0.344697 |
+| bigbench_operators | 0.134921 | 0.139683 | 0.139682 | 0.095238 | 0.090476 | 0.090476 | 0.142857 |
+| bigbench_repeat_copy_logic | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.062500 |
+| squad | 0.013844 | 0.008514 | 0.011542 | 0.070766 | 0.047114 | 0.049669 | 0.143519 |
+| coqa | 0.044135 | 0.031484 | 0.036077 | 0.113366 | 0.116623 | 0.123888 | 0.180509 |
+| boolq | -0.408874 | -0.361661 | -0.354418 | -0.187832 | -0.059070 | -0.054241 | -0.329470 |
+| bigbench_language_identification | 0.001689 | 0.007733 | 0.002755 | -0.003467 | 0.010000 | 0.002533 | 0.002933 |
+| **Core** | **0.013949** | **0.017166** | **0.014880** | **0.089402** | **0.112373** | **0.108874** | **0.180409** |
+[Download the comparison CSV](eval/core_comparison.csv).
+The 450M run also provides [six-task zero-shot lm-eval scores](mixerloop-450m/fineweb-edu-100B-s1337/eval/lm_eval.csv): ARC-C and HellaSwag use `acc_norm`; ARC-E, PIQA, WinoGrande and BoolQ use `acc`. `Avg` is the arithmetic mean of these six scores. Both evaluation CSVs store scores as fractions.

eval/core_comparison.csv CHANGED Viewed

@@ -1,24 +1,24 @@
-Task,13M GDN,13M MixerLoop,13M FullLoop,100M GDN,100M MixerLoop,100M FullLoop
-hellaswag_zeroshot,0.037509,0.032862,0.030472,0.132842,0.133771,0.167231
-jeopardy,0.000000,0.000000,0.001417,0.007085,0.006613,0.010392
-bigbench_qa_wikidata,0.036908,0.050342,0.071010,0.214950,0.257074,0.282811
-arc_easy,0.144220,0.153760,0.145342,0.317621,0.341190,0.385522
-arc_challenge,-0.028441,-0.053470,-0.039818,-0.001138,0.018203,0.045506
-copa,-0.100000,-0.120000,-0.180000,0.000000,0.020000,0.020000
-commonsense_qa,-0.225072,-0.226443,-0.186659,-0.244278,-0.015177,-0.306011
-piqa,0.158868,0.171926,0.170838,0.316649,0.334059,0.355822
-openbook_qa,0.034667,0.018667,0.024000,0.090667,0.088000,0.098667
-lambada_openai,0.105376,0.112168,0.122647,0.253639,0.246264,0.263536
-hellaswag,0.028746,0.024763,0.022107,0.125938,0.126734,0.163513
-winograd,0.091575,0.018315,0.076923,0.172161,0.113553,0.172161
-winogrande,0.010260,0.043410,0.054459,0.027624,0.000789,0.029203
-bigbench_dyck_languages,0.003000,0.005000,0.005000,0.128000,0.172000,0.093000
-agi_eval_lsat_ar,-0.055072,0.037681,-0.002899,-0.014493,0.072464,0.049275
-bigbench_cs_algorithms,0.325000,0.312879,0.312121,0.351515,0.351515,0.352273
-bigbench_operators,0.142857,0.152381,0.180952,0.095238,0.090476,0.090476
-bigbench_repeat_copy_logic,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000
-squad,0.004730,0.010028,0.012867,0.070766,0.047114,0.049669
-coqa,0.040962,0.023300,0.047225,0.113366,0.116623,0.123888
-boolq,-0.354418,-0.203927,-0.229680,-0.187832,-0.059070,-0.054241
-bigbench_language_identification,-0.002400,0.008133,0.010533,-0.003467,0.010000,0.002533
-Core,0.018149,0.025990,0.029494,0.089402,0.112373,0.108874

+Task,13M GDN,13M MixerLoop,13M FullLoop,100M GDN,100M MixerLoop,100M FullLoop,450M MixerLoop
+hellaswag_zeroshot,0.034898,0.033526,0.029675,0.132842,0.133771,0.167231,0.323773
+jeopardy,0.000315,0.000157,0.000630,0.007085,0.006613,0.010392,0.051488
+bigbench_qa_wikidata,0.040106,0.031396,0.071863,0.214950,0.257074,0.282811,0.536932
+arc_easy,0.155069,0.152076,0.151702,0.317621,0.341190,0.385522,0.512907
+arc_challenge,-0.037922,-0.054228,-0.037543,-0.001138,0.018203,0.045506,0.130830
+copa,-0.113333,-0.093333,-0.120000,0.000000,0.020000,0.020000,0.300000
+commonsense_qa,-0.180258,-0.234217,-0.246564,-0.244278,-0.015177,-0.306011,-0.251137
+piqa,0.162858,0.167211,0.166123,0.316649,0.334059,0.355822,0.404788
+openbook_qa,0.035556,0.041778,0.042667,0.090667,0.088000,0.098667,0.165333
+lambada_openai,0.105764,0.117407,0.116825,0.253639,0.246264,0.263536,0.398603
+hellaswag,0.028923,0.029277,0.023479,0.125938,0.126734,0.163513,0.313948
+winograd,0.072039,0.030525,0.050061,0.172161,0.113553,0.172161,0.384615
+winogrande,0.027624,0.020258,0.031834,0.027624,0.000789,0.029203,0.097080
+bigbench_dyck_languages,0.001333,0.008333,0.005667,0.128000,0.172000,0.093000,0.032000
+agi_eval_lsat_ar,-0.051208,-0.004831,-0.016425,-0.014493,0.072464,0.049275,0.020290
+bigbench_cs_algorithms,0.239394,0.306566,0.221717,0.351515,0.351515,0.352273,0.344697
+bigbench_operators,0.134921,0.139683,0.139682,0.095238,0.090476,0.090476,0.142857
+bigbench_repeat_copy_logic,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.062500
+squad,0.013844,0.008514,0.011542,0.070766,0.047114,0.049669,0.143519
+coqa,0.044135,0.031484,0.036077,0.113366,0.116623,0.123888,0.180509
+boolq,-0.408874,-0.361661,-0.354418,-0.187832,-0.059070,-0.054241,-0.329470
+bigbench_language_identification,0.001689,0.007733,0.002755,-0.003467,0.010000,0.002533,0.002933
+Core,0.013949,0.017166,0.014880,0.089402,0.112373,0.108874,0.180409

mixerloop-450m/fineweb-edu-100B-s1337/README.md CHANGED Viewed

@@ -27,7 +27,7 @@ tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=run)
 model = AutoModelForCausalLM.from_pretrained(repo, subfolder=run)
 ```
-Weights, config, generation config and tokenizer use the standard `save_pretrained()` format. The packaged tokenizer is the local canonical Llama 3 asset (128256 tokens, BOS 128000, EOS 128001). All 381 exported tensors were checked against the native checkpoint, and HF reload and a GPU forward passed. Full CORE v2 evaluation is pending.
 Training source revision: `1dbfe450e2e10330e28c0c53b50e7e1d72217fa9`.

 model = AutoModelForCausalLM.from_pretrained(repo, subfolder=run)
 ```
+Weights, config, generation config and tokenizer use the standard `save_pretrained()` format. The packaged tokenizer is the local canonical Llama 3 asset (128256 tokens, BOS 128000, EOS 128001). All 381 exported tensors were checked against the native checkpoint, and HF reload and a GPU forward passed. Full CORE v2 evaluation is complete: **0.180409**. The six-task zero-shot lm-eval average is **55.60%**. Scores are available in [`eval/core_eval.csv`](eval/core_eval.csv) and [`eval/lm_eval.csv`](eval/lm_eval.csv); both CSVs store scores as fractions.
 Training source revision: `1dbfe450e2e10330e28c0c53b50e7e1d72217fa9`.

mixerloop-450m/fineweb-edu-100B-s1337/eval/core_eval.csv ADDED Viewed

	@@ -0,0 +1,24 @@

+Task,Accuracy,Centered
+hellaswag_zeroshot,0.492830,0.323773
+jeopardy,0.051488,0.051488
+bigbench_qa_wikidata,0.536932,0.536932
+arc_easy,0.634680,0.512907
+arc_challenge,0.348123,0.130830
+copa,0.650000,0.300000
+commonsense_qa,0.253071,-0.251137
+piqa,0.702394,0.404788
+openbook_qa,0.374000,0.165333
+lambada_openai,0.398603,0.398603
+hellaswag,0.485461,0.313948
+winograd,0.692308,0.384615
+winogrande,0.548540,0.097080
+bigbench_dyck_languages,0.032000,0.032000
+agi_eval_lsat_ar,0.265217,0.020290
+bigbench_cs_algorithms,0.344697,0.344697
+bigbench_operators,0.142857,0.142857
+bigbench_repeat_copy_logic,0.062500,0.062500
+squad,0.143519,0.143519
+coqa,0.180509,0.180509
+boolq,0.494801,-0.329470
+bigbench_language_identification,0.252200,0.002933
+Core_v2,,0.180409

mixerloop-450m/fineweb-edu-100B-s1337/eval/lm_eval.csv ADDED Viewed

	@@ -0,0 +1,8 @@

+Task,PrimaryMetric,Score,AllMetricsJSON
+arc_easy,"acc,none",0.663720538720539,"{""acc,none"": 0.6637205387205387, ""acc_norm,none"": 0.6199494949494949, ""acc_norm_stderr,none"": 0.009960175831493119, ""acc_stderr,none"": 0.009694178072725206, ""alias"": ""arc_easy"", ""name"": ""arc_easy"", ""sample_len"": 2376}"
+arc_challenge,"acc_norm,none",0.340443686006826,"{""acc,none"": 0.31569965870307165, ""acc_norm,none"": 0.34044368600682595, ""acc_norm_stderr,none"": 0.013847460518892978, ""acc_stderr,none"": 0.013582571095815295, ""alias"": ""arc_challenge"", ""name"": ""arc_challenge"", ""sample_len"": 1172}"
+hellaswag,"acc_norm,none",0.500298745269867,"{""acc,none"": 0.39364668392750446, ""acc_norm,none"": 0.5002987452698665, ""acc_norm_stderr,none"": 0.00498978052078224, ""acc_stderr,none"": 0.004875595792850676, ""alias"": ""hellaswag"", ""name"": ""hellaswag"", ""sample_len"": 10042}"
+piqa,"acc,none",0.710554951033732,"{""acc,none"": 0.7105549510337323, ""acc_norm,none"": 0.7127312295973884, ""acc_norm_stderr,none"": 0.010557291761528633, ""acc_stderr,none"": 0.01058101474067561, ""alias"": ""piqa"", ""name"": ""piqa"", ""sample_len"": 1838}"
+winogrande,"acc,none",0.545382794001579,"{""acc,none"": 0.5453827940015785, ""acc_stderr,none"": 0.013994481027065998, ""alias"": ""winogrande"", ""name"": ""winogrande"", ""sample_len"": 1267}"
+boolq,"acc,none",0.575535168195719,"{""acc,none"": 0.5755351681957187, ""acc_stderr,none"": 0.008644688121685496, ""alias"": ""boolq"", ""name"": ""boolq"", ""sample_len"": 3270}"
+Avg,six_task_arithmetic_mean,0.555989313871377,"{""tasks"": [""arc_easy"", ""arc_challenge"", ""hellaswag"", ""piqa"", ""winogrande"", ""boolq""], ""percentage"": 55.598931387137675}"