im probably worse than llama3.1 in making models, i always forget something and it allways comes out bad lol

Thank you for the heads up. I wanted to make my own models for a long time,but either local doesn't reach a good enough checkpoint or online i get ratelimited and not all features work without paying 50 dollars per month. Thank you

The experiment I would want is the one where the AI gets the same budget the human run got.

Three things changed at once here, so the 0.70 points cannot be pinned on the code.

Both sides are off your own cards:

BananaMind-2-Nano      35.77 avg   9,968,128 params   30.0B tokens   4096 ctx
MiniBananaMind-v4-9M   35.07 avg   8,884,992 params   16.6B tokens   1024 ctx

Nano's 30.0B is the 29,999,726,592 on its card. v4's 16.6B is its own three-stage table added up: 12,047,375,481 FineWeb-Edu + 1,500,000,000 FineMath + 3,082,420,224 Cosmopedia. So the AI-written recipe got 1.80x the tokens, 12% more parameters and 4x the context, for +0.70.

And 0.70 is inside the noise. The splits are fixed sizes (ARC-E 2376, PIQA 1838, ARC-C 1172, HellaSwag 10042), so the SE on that four-task mean is 0.50pp, and 0.71pp on an unpaired difference. The gap is 0.98 SE. You have the per-item outputs though, and a paired test on the same items is much tighter than my unpaired bound. That one is free to run and it would actually settle it.

The other thing the average hides: a random guesser scores 31.25 on this mix, because PIQA is 2-way. So the whole live band is about four points wide.

above chance   Nano 4.51   v4-9M 3.82   Pythia-31M 3.54

Pythia-31M sits in your own Mini table at 34.79, and it beats Nano on PIQA, 56.47 to 55.98. On ARC Challenge none of the three clears chance: 23.38, 23.04, 21.42 against 25. A quarter of the headline average is contributing negative signal.

One thing I could not resolve. BananaMind-2-Nano's card was last touched 2026-07-21, two weeks before this post, and it names a specific run: final step 55,485, seed 1337. So the published Nano does not look like the artifact GPT 5.6 Sol Max produced. Is that model up anywhere?

Either way, the honest read is that the AI recipe is not shown worse either. The cheap version of this is to hand it v4's exact recipe: 16.6B tokens, 1024 context, seed 1337, same mix. Then the only thing that varies is who wrote the code. Would you run that one?

The 0.70 points are not the result in this experiment. The KV cache is.

You handed it Mini, so the architecture was never the AI's to invent. Nano's config is Mini's: QK norm, head_dim 64, 2 KV heads, rope theta 100,000, 4,096 context, rms eps 1e-6, SwiGLU, tied embeddings, byte-identical tokenizer.json (same oid, 552,335 bytes). Both cards carry the same 30B mix, same effective batch of 132, same 1,750 warmup, same WSD schedule, same z-loss 1e-4 until 12B then off, same seed 1337. And both land on step 55,485 with 29,999,726,592 tokens seen. To the token.

So two things here were actually the AI's.

Peak LR 3.0e-3 against Mini's 2.3e-3. Right direction for a narrower model.

And this one, which I think is the find. Mini cannot decode incrementally. Its prepare_inputs_for_generation hardcodes use_cache: False, its forward accepts past_key_values and never reads it, and it returns past_key_values=None. Every generated token recomputes the whole prefix. Nano wires up DynamicCache with per-layer layer_idx, shifts the causal mask by past_length, left-pads the key mask out to kv_len, and drops the override so GenerationMixin threads the cache. It also refuses past 4,096 instead of quietly extrapolating rope.

That is a latency fix sitting in your own family, and it is a bigger deal than 0.70 on a four-task mean.

On the score, the arithmetic does reconcile off both configs:

+1,083,136 params = 787,072  the 10th block        72.7%
                  + 294,912  heads 8x32 -> 4x64    27.2%
                  +   1,152  QK norm                0.1%

QK norm is 1,280 weights, 0.013% of Nano. So the clean run is cheaper than what I asked for last time. Not equal budget. Just v4's config plus QK norm, at 1,024 context and 16.6B tokens, seed 1337.

One more, on data rather than code. v4's line has never seen DCLM. V1 and v2 are FineWeb-Edu, v3 adds FineMath, v4 continues on Cosmopedia. 9.0B of Nano's tokens, 30% of the run, come from a corpus the other arm has never touched. A human picked that.

Did the AI find the cache gap on its own, or did you point it there?

Two more places to look before you patch it.

MathBananaMind-1.1 has the same shape: use_cache: False hardcoded in prepare_inputs_for_generation, nothing reading past_key_values in forward, and past_key_values=None on the way out.

And Mini's fix already exists inside your own org. BananaMind-2-Mini-Chat ships a modeling_bananamind2mini.py that has the cache in it, same class name BananaMind2MiniForCausalLM, and the two configs match exactly: 14 layers, hidden 384, 6 heads, head_dim 64, 4,096 context. So Mini is a file copy from the Chat repo rather than a rewrite. MathBananaMind needs it ported by hand instead, since it is 9 layers, hidden 160, 5 heads, 1,024 context and its own class.

I went through all 28 models under BananaMind. The rest of the 2 line already has the cache: Pro-Preview, Pro-Preview-Chat, Medium, Medium-Chat, MoE, Nano, Nano-Chat, Mini-Chat. The older MiniBananaMind and Micro checkpoints have no cache path at all, but those are 8-9M params, so it costs less there.

Two repos to touch, one of them by copy. Is the Chat variant built by a different export script than the base?