---
language:
- en
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
base_model: Astrosully/Kyro-29M
tags:
- small-language-model
- tiny-llm
- instruction-tuned
- sft
datasets:
- HuggingFaceTB/smol-smoltalk
---

# Kyro-29M-Instruct

Kyro-29M-Instruct is the instruction-tuned version of [**Kyro-29M**](https://huggingface.co/Astrosully/Kyro-29M), a 29.1M-parameter English model
pretrained from scratch on 30B tokens. That model's card covers the architecture, the
agent-driven design process, the pretraining data and the benchmarks.

This version answers single-turn instructions. At this size, expect the right format and tone, but
not reliable facts.

## Quick start

```bash
pip install torch safetensors tiktoken huggingface_hub
```

```python
from huggingface_hub import snapshot_download
import sys

path = snapshot_download("Astrosully/Kyro-29M-Instruct")
sys.path.insert(0, path)
from modeling_kyro import load

model, tok = load(path)
print(tok.chat(model, "Give three tips for staying healthy.", temperature=0.7))
```

Or from the command line, inside the downloaded folder:

```bash
python modeling_kyro.py --chat --prompt "Give three tips for staying healthy."
```

### Prompt format

The model was fine-tuned on exactly one template. `tok.chat()` builds it for you:

```
<|bos|>### Instruction:
{your instruction}

### Response:
```

The reply ends with the `<|assistant_end|>` token (id 16379). The model was trained on single
turns only, so it has no multi-turn memory. When there is a system prompt, put it at the top of
the instruction, as was done in training. The sampler defaults to temperature 0.7, top-k 50 and a
repetition penalty of 1.15.

## Fine-tuning

| | |
|---|---|
| Data | [smol-smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk), first user→assistant turn of each conversation |
| Examples | 331,346 train (those that fit in 512 tokens) / 500 validation |
| Loss | response tokens only (88.8M supervised tokens of 169.6M) |
| Packing | whole examples packed into 512-token rows, never split |
| Optimizer | AdamW, lr 3e-4, cosine to 1e-6, 3% warmup, no weight decay, grad clip 1.0 |
| Batch / length | 32 × 512 tokens, 2 epochs (20,708 steps), about 17 minutes on an RTX 5090 |
| Final validation loss | 1.471 (perplexity 4.35) on response tokens |

The learning rate was chosen by a one-epoch sweep:

| Learning rate | Validation loss |
|---|---|
| 1e-4 | 1.581 |
| **3e-4** | **1.526** |
| 1e-3 | 1.549 |

The second epoch then took validation loss from 1.526 to 1.471.

## Examples

These were sampled at temperature 0.7 and are not cherry-picked:

> **Q: What is the capital of France?**
> The capital of France is Paris. It serves as the political and cultural center, a major hub for
> culture, politics, history, philosophy, artistry, fashion, gastronomy, literature...

> **Q: Give three tips for staying healthy.**
> 1. Regular Exercise: ...
> 2. Stay Hydrated: Even a balanced diet rich in fruits and vegetables can help keep your body
> hydrated throughout the day...

The model gets the answer and the structure right, then pads with plausible-sounding filler, and
sometimes with invented details. That is typical at 29M parameters.

## Limitations

The base model's limits all apply: weak world knowledge, frequent made-up facts, English only,
512-token context, and no safety tuning. Instruction tuning teaches the model the format of an
answer, not the knowledge to make it correct. Do not use it where correctness matters.

## License

The weights and code are Apache-2.0. The fine-tuning data, smol-smoltalk, is Apache-2.0. For the
pretraining data licenses, see the Kyro-29M card.