Hugging Face's logo
h s's picture

h s

HubertSchmitt
2 5
·

AI & ML interests

None yet

Recent Activity

repliedto Banaxi-Tech's post about 20 hours ago
Today we wanted to release BananaMind 2 Pico, our smallest model yet at ~0.9M parameters. Instead, we accidentally ran a very expensive experiment on what happens when you push a tiny model way past its useful token budget. Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through. The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting. Follow us for more: https://huggingface.co/BananaMind @vovaRL @Banaxi-Tech Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXP
liked a model about 20 hours ago
Qwen/Qwen3.8-27B
commentedon a paper 6 days ago
GPT-Red: Automated Red Teaming via Self-Play at Scale
View all activity

Organizations

None yet