Kyro-29M-Instruct

Kyro-29M-Instruct is the instruction-tuned version of Kyro-29M, a 29.1M-parameter English model pretrained from scratch on 30B tokens. That model's card covers the architecture, the agent-driven design process, the pretraining data and the benchmarks.

This version answers single-turn instructions. At this size, expect the right format and tone, but not reliable facts.

Quick start

pip install torch safetensors tiktoken huggingface_hub

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("Astrosully/Kyro-29M-Instruct")
sys.path.insert(0, path)
from modeling_kyro import load

model, tok = load(path)
print(tok.chat(model, "Give three tips for staying healthy.", temperature=0.7))

Or from the command line, inside the downloaded folder:

python modeling_kyro.py --chat --prompt "Give three tips for staying healthy."

Prompt format

The model was fine-tuned on exactly one template. tok.chat() builds it for you:

<|bos|>### Instruction:
{your instruction}

### Response:

The reply ends with the <|assistant_end|> token (id 16379). The model was trained on single turns only, so it has no multi-turn memory. When there is a system prompt, put it at the top of the instruction, as was done in training. The sampler defaults to temperature 0.7, top-k 50 and a repetition penalty of 1.15.

Fine-tuning

Data	smol-smoltalk, first user→assistant turn of each conversation
Examples	331,346 train (those that fit in 512 tokens) / 500 validation
Loss	response tokens only (88.8M supervised tokens of 169.6M)
Packing	whole examples packed into 512-token rows, never split
Optimizer	AdamW, lr 3e-4, cosine to 1e-6, 3% warmup, no weight decay, grad clip 1.0
Batch / length	32 × 512 tokens, 2 epochs (20,708 steps), about 17 minutes on an RTX 5090
Final validation loss	1.471 (perplexity 4.35) on response tokens

The learning rate was chosen by a one-epoch sweep:

Learning rate	Validation loss
1e-4	1.581
3e-4	1.526
1e-3	1.549

The second epoch then took validation loss from 1.526 to 1.471.

Examples

These were sampled at temperature 0.7 and are not cherry-picked:

Q: What is the capital of France? The capital of France is Paris. It serves as the political and cultural center, a major hub for culture, politics, history, philosophy, artistry, fashion, gastronomy, literature...

Q: Give three tips for staying healthy.

1. Regular Exercise: ...
2. Stay Hydrated: Even a balanced diet rich in fruits and vegetables can help keep your body hydrated throughout the day...

The model gets the answer and the structure right, then pads with plausible-sounding filler, and sometimes with invented details. That is typical at 29M parameters.

Limitations

The base model's limits all apply: weak world knowledge, frequent made-up facts, English only, 512-token context, and no safety tuning. Instruction tuning teaches the model the format of an answer, not the knowledge to make it correct. Do not use it where correctness matters.

License

The weights and code are Apache-2.0. The fine-tuning data, smol-smoltalk, is Apache-2.0. For the pretraining data licenses, see the Kyro-29M card.