Kyro-29M

Kyro-29M is a 29.1M-parameter English language model trained from scratch on 30 billion tokens. Its architecture and training recipe were not hand-designed. An autonomous research agent found them over hundreds of timed experiments on a laptop GPU. A single 15-hour run on one rented RTX 5090 then trained the result at full scale.

This repo holds the base (pretrained) model. An instruction-tuned version, Kyro-29M-Instruct, is published separately.

Parameters	29.13M (14.7M of them in the input embedding and output head)
Layers / width / heads	6 / 448 / 7 (head dim 64)
Context	512 tokens
Vocabulary	16,384 (byte-level BPE)
Training tokens	30B (about 1,030 tokens per parameter)
Training compute	≈5.2 × 10¹⁸ FLOPs, 14.9 h on 1× RTX 5090
FineWeb-Edu validation	1.084 bits per byte
Weights	model.safetensors, float32 (117 MB)

Quick start

pip install torch safetensors tiktoken huggingface_hub

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("Astrosully/Kyro-29M")
sys.path.insert(0, path)
from modeling_kyro import load

model, tok = load(path)                 # GPU if available, else CPU
print(tok.complete(model, "The water cycle is", max_new_tokens=80, temperature=0.7))

Or from the command line, inside the downloaded folder:

python modeling_kyro.py --prompt "The water cycle is"

modeling_kyro.py is about 200 lines and needs only PyTorch, safetensors and tiktoken. It is not a transformers architecture, so AutoModel will not load it. The tokenizer ships as a plain tiktoken ranks file (kyro.tiktoken) plus tokenizer_config.json, with no pickles.

Sample output (temperature 0.7):

The water cycle is a complex system involving an array of interconnected components and processes. It all starts with the water cycle, which is usually divided into two parts...

Architecture

A pre-norm decoder-only transformer descended from Karpathy's nanochat, plus four additions the research agent kept because they measurably lowered loss at a fixed time budget:

• Parameter-free RMSNorm, SwiGLU MLP (8/3 expansion), untied input and output embeddings, no biases, no logit softcap.
• Partial rotary embeddings: RoPE on the first half of each 64-dim head; the other half carries no position signal.
• Per-layer residual and input-skip scalars: each block sees λ_res·x + λ_0·x₀, where x₀ is the normalised embedding.
• Value residual (ResFormer, arXiv 2410.17897): layers 1–5 add a learned multiple of layer 0's attention values to their own.
• U-Net shortcuts: layer 5−i re-adds layer i's input through a learned scalar.
• Token smear: a learned per-channel gate mixes each position's embedding with the previous token's (strictly causal).
• The output head trained with Muon rather than AdamW, at an update size 2× a block matrix's.

Every addition starts at exactly zero, so the model begins training as a plain transformer and learns how much of each to use.

How it was made

1. Architecture search by an autonomous agent

The design came from an autoresearch loop in the spirit of Karpathy's autoresearch experiments:

• An LLM agent edits a single train.py.
• Each edit trains for exactly 5 minutes on an RTX 4060 Laptop GPU, on TinyStories with the same 16K tokenizer.
• The harness keeps the change only if validation bits-per-byte improves beyond the measured noise floor.

The harness enforces honesty checks:

• Metrics are measured by the harness, never reported by the agent.
• Causality and data-leak tests run on every candidate.
• A parameter-count band and a VRAM reserve apply.
• The training code runs in a no-network bubblewrap sandbox.
• Promising changes are re-audited at 15 minutes, so a win at 5 minutes cannot just be a faster start.

Across sessions, the agent settled or rejected most of the design space:

• Kept:
  • Muon for matrices at a high LR (0.08 at a 2¹⁹-token reference batch), constant momentum 0.95.
  • 32K tokens per step.
  • Linear LR decay to zero, with weight decay decayed alongside.
  • Shallow-and-wide over deep-and-thin; a 16K vocabulary.
  • Value residual, U-Net shortcuts, smear, and the Muon output head.
• Rejected after measurement: differential attention, hyper-connections, QK-norm, compressed attention, n-gram memory, and ternary weights.

The winning configuration (7b25048) was frozen and became Kyro.

2. Pretraining

Data: a 14.5B-token mix built for this run, tokenized with Kyro's own 16K tokenizer. The tokenizer was trained on FineWeb-Edu.

Source	Tokens	Share	Documents
FineWeb-Edu (sample-100BT)	10.0B	69%	9.08M
Cosmopedia v2 (synthetic textbooks)	2.5B	17%	3.12M
FineMath (finemath-4plus)	1.0B	7%	0.57M
Wikipedia (English, 2023-11-01)	1.0B	7%	1.06M

Every document starts with <|bos|>. The FineWeb-Edu validation documents were removed from the training mix by content hash; 4,792 were caught in the 100BT sample. Validation numbers are therefore held out.

Training run:

• Tokens and steps: 30B tokens, about 2 epochs of the mix; 915,528 steps of 32,768 tokens (sequence length 512).
• Optimizer: Muon for the 2-D block matrices and the output head; AdamW (β = 0.8, 0.95) for embeddings and scalars.
• Schedule: no warmup; the LR decays linearly to zero over the whole run; weight decay 0.2, also decayed to zero.
• Precision and speed: bf16 autocast with torch.compile, at 560K tokens/s.
• Seed: 42.
• Loss curve: validation loss was still falling at the end.

3. Comparison with the 2B-token run

The same model trained on 2B FineWeb-Edu tokens (58 minutes) reached 1.114 bpb. 15× more data, plus the broader mix, brought it to 1.084 bpb:

• Large gains on the tasks that need real language modelling: LAMBADA +6.1, HellaSwag +1.7.
• Flat on ARC and SciQ.

A 29M model is capacity-bound well before 30B tokens; the compute-optimal size for this budget would be around 200M parameters.

Evaluation

Results:

• Setup: zero-shot, full test sets, lm-evaluation-harness 0.4.12, float32, <|bos|> prepended, 512-token context.
• Metric: acc_norm where a task has it, otherwise acc.
• Uncertainty: standard errors are about 0.5–2 points depending on the task.
• Setup check: BarunLM-35M scores 41.13 average here against 41.01 on its own card.

Model	Params	Tokens	ARC-C	ARC-E	BoolQ	HellaSwag	LAMBADA	OBQA	PIQA	SciQ	WinoG	Avg	Avg w/o BoolQ
Kyro-29M	29M	30B	23.2	41.0	51.4	29.4	25.0	28.6	58.7	63.3	51.1	41.3	40.0
Kyro-29M @ 2B tokens	29M	2B	23.1	41.4	62.2	27.7	18.9	26.6	58.7	63.8	50.4	41.4	38.8
BarunLM-35M	35M	5.7B	22.4	39.0	56.7	28.9	24.6	28.4	59.4	59.3	51.5	41.1	39.2
nanochat-style d146M (own run)	146M	2B	22.8	34.7	62.1	27.5	19.7	26.6	59.6	53.9	51.5	39.8	37.0
SmolLM-135M	135M	600B	28.7	56.5	60.2	42.6	37.7	34.4	68.1	74.8	53.3	50.7	49.5

Two notes on reading the table:

• BoolQ mostly measures answer bias at this scale: answering "yes" every time scores 62%. That is why the average without BoolQ is the fairer column. On it, Kyro-29M is the best model under 100M parameters in this comparison.
• WinoGrande sits at chance for every small model.

ArithMark 3.0

ArithMark 3.0 has 1,000 grade-school arithmetic word problems with four answer choices each; random guessing scores 25%. It was scored the same way as the dataset's own script: no special tokens, length-normalized log-likelihood. The setup reproduces SmolLM-135M's published 36.80% exactly.

Model	Params	Score
Kyro-29M	29M	37.2%
Kyro-29M, BOS token prepended (as in training)	29M	39.2%
SmolLM-135M	135M	36.8%
Kyro-29M-Instruct	29M	34.7%

That ties sixth of the 78 models on the benchmark's published leaderboard. The margin of error is about ±1.5 points.

Most of the score comes from recognizing plausible answers, not computing them:

• Two-digit addition (21%) and subtraction (24%) are at chance.
• Multiplication (73%), area and perimeter (64–67%) and time (96%) score well above it. Every model tested scores 96–100% on time.

Intended use and limitations

Kyro-29M is a research and education artifact: an example of what a ~30M model can learn, and of an agent-designed training recipe. It is not suitable for anything that relies on its outputs being true.

• Knowledge. It writes fluent, on-topic English but invents facts freely. Twenty-nine million parameters hold very little world knowledge.
• Scope. It handles English only, with a 512-token context, and has weak arithmetic and reasoning.
• Safety. There is no safety tuning or content filtering beyond what the source datasets applied. Outputs can reflect biases in web text.
• Base model. It continues text; it does not follow instructions. Use Kyro-29M-Instruct for that.

Licenses

• Weights and code: Apache-2.0.
• Training data (each under its own license):
  • FineWeb-Edu, Cosmopedia v2 and FineMath: ODC-By.
  • Wikipedia: CC BY-SA 4.0.

Acknowledgements

• The architecture core comes from Andrej Karpathy's nanochat, and the autoresearch loop is modelled on his autoresearch experiments.
• Datasets come from Hugging Face (FineWeb-Edu, SmolLM-Corpus, FineMath) and the Wikimedia Foundation.
• The ideas the agent adopted include Muon (Keller Jordan et al.) and ResFormer value residuals (Zhou et al.).