Teensy-0

Teensy-0 is a skinny-deep decoder-only language model. It was trained from scratch on the OpenWebText corpus using a custom training pipeline.

The architecture and training pipeline are modified from NanoGPT by Andrej Karpathy, adapted specifically for the Teensy model family.

Model Details

Teensy-0 follows the naming convention 30L_8H_64BLK_12.35M:

30L — 30 transformer layers
8H — 8 attention heads
64BLK — 64-token context length (block size)
12.35M — ~12.35 million parameters

Property	Value
Architecture	Causal decoder-only transformer (GPT-style)
Layers	30
Attention heads	8
Embedding dimension	128
Context length	64 tokens
Parameters	~12.35M
Activation	GELU
Normalization	LayerNorm
Tokenizer	GPT-2 BPE (50,304 vocab)

Training

Teensy-0 was trained from scratch on the OpenWebText corpus:

~9B tokens
20,000 iterations
batch size 12, gradient accumulation 40
AdamW, peak learning rate 6e-4 with warmup and decay
Best validation loss: 4.5432

The training run was produced with the Teensy training code, not by fine-tuning an existing model. The original data, notebooks, and training scripts are archived in the sibling teensy-1/ directory.

Files in this Repository

Path	Description
`checkpoints/teensy-0.pt`	Full PyTorch checkpoint (~148 MB). Contains model weights, optimizer state, and training metadata.
`exported/model.safetensors`	HuggingFace-compatible model weights only (~75 MB). Smaller and faster to download.
`exported/config.json`	Model hyperparameters for the safetensors export.
`model.py`	Native `TeensyLM` architecture and a NanoGPT-to-Teensy weight adapter.
`sample.py`	Generate text from the `.pt` checkpoint.
`repl.py`	Interactive streaming REPL for the `.pt` checkpoint.
`sample_hf.py`	Generate text from the exported `model.safetensors`.
`repl_hf.py`	Interactive streaming REPL for the exported `model.safetensors`.
`export_hf.py`	Re-export `teensy-0.pt` to `exported/model.safetensors` + `config.json`.

Installation

pip install -r requirements.txt

Requirements:

torch>=2.0
tiktoken>=0.5
numpy>=1.24
safetensors>=0.4

Device Notes

CPU is strongly recommended for inference. This checkpoint was trained with the PyTorch MPS backend, and inference on MPS can produce degraded output: garbled text, stray <|endoftext|> tokens, and invalid UTF-8 byte sequences. All inference scripts therefore default to cpu. Pass --device=mps explicitly if you want to use Apple Silicon GPU, but expect lower-quality output.

Usage

1. Interactive streaming REPL (`.pt` checkpoint)

python repl.py --device=cpu --dtype=float32

Type a prompt and Teensy will stream tokens back token-by-token. Generation stops automatically when the model emits the GPT-2 <|endoftext|> token. Type exit or press Ctrl+C to quit.

2. Sample from a prompt (`.pt` checkpoint)

python sample.py --device=cpu --dtype=float32 \
  --start="Once upon a time" \
  --max_new_tokens=200 \
  --num_samples=3

3. Generate from the safetensors export

python sample_hf.py --device=cpu --dtype=float32 \
  --prompt="Once upon a time" \
  --max_new_tokens=200

4. Interactive streaming REPL for the safetensors export

python repl_hf.py --device=cpu --dtype=float32

Streams tokens and stops automatically at <|endoftext|>.

4. Load in your own code

From the `.pt` checkpoint

import torch
from model import TeensyConfig, TeensyLM, adapt_nanogpt_weights

checkpoint = torch.load("checkpoints/teensy-0.pt", map_location="cpu")
cfg = TeensyConfig(**checkpoint["model_args"])
model = TeensyLM(cfg)
state_dict = adapt_nanogpt_weights(checkpoint["model"])
model.load_state_dict(state_dict)
model.eval()

From the safetensors export

import json
import torch
from safetensors.torch import load_file
from model import TeensyConfig, TeensyLM

with open("exported/config.json") as f:
    cfg = TeensyConfig(**json.load(f))

model = TeensyLM(cfg)
model.load_state_dict(load_file("exported/model.safetensors"))
model.eval()

5. Re-export to safetensors

python export_hf.py --out_dir=checkpoints --export_dir=exported

This writes exported/model.safetensors (~75 MB, ~49% smaller than the .pt checkpoint) and exported/config.json.

Downloading without Cloning the Whole Repo

If you only want the model files, you can use the huggingface_hub library:

pip install huggingface_hub

Download the `.pt` checkpoint

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="Teensy/teensy-0",
    filename="checkpoints/teensy-0.pt",
    local_dir="./teensy-0"
)
print(path)

Download the safetensors export

from huggingface_hub import hf_hub_download

config_path = hf_hub_download(
    repo_id="Teensy/teensy-0",
    filename="exported/config.json",
    local_dir="./teensy-0"
)
weights_path = hf_hub_download(
    repo_id="Teensy/teensy-0",
    filename="exported/model.safetensors",
    local_dir="./teensy-0"
)
print(config_path, weights_path)

Download everything

huggingface-cli download Teensy/teensy-0 --local-dir ./teensy-0

Repository Layout

teensy-0/
├── checkpoints/       # trained PyTorch checkpoints
├── exported/          # HuggingFace-compatible safetensors + config.json
├── model.py           # TeensyLM architecture
├── sample.py          # text generation script for .pt
├── repl.py            # interactive streaming REPL for .pt
├── sample_hf.py       # text generation script for the safetensors export
├── repl_hf.py         # interactive streaming REPL for the safetensors export
├── export_hf.py       # export checkpoint to safetensors
├── README.md
├── LICENSE
└── requirements.txt

`../colab/` — training & distillation toolkit (not part of this repo)

An experimental pipeline for the next Teensy variants (instruction tuning and tool calling) lives in the sibling colab/ directory, outside this repository, so the published model repo stays clean:

Path	Description
`common.py`	Model (adds RoPE/GQA options + special tokens in unused GPT-2 rows), data pipelines, training loop, tool-call eval, safetensors export
`01_continue_and_instruct.ipynb`	Colab: continue teensy-0 on FineWeb-Edu (context 64→512), then Dolly-15k instruction SFT
`02_scratch_fineweb_toolcalls.ipynb`	Colab: from-scratch FineWeb-Edu pretraining, then tool-call SFT on distilled data
`03_scratch_pure_distillation.ipynb`	Colab: from-scratch on fully synthetic teacher corpus + tool-call SFT
`generate_data.py`	Local distillation data generator (schemas → teacher → validated JSONL). Backends: LM Studio server or mlx-lm
`constrained.py`	Schema-constrained decoding (tool names / arg keys / enum values guaranteed valid)
`run_toolcall.py`	Tool-call inference CLI with constrained decoding
`test_*.py`	CPU test suites for all of the above
`build_notebooks.py`	Regenerates the three notebooks with syntax-checked cells

The original data, notebooks, and training scripts are archived in teensy-1/.

Limitations

This model is not instruction-tuned.
It performs next-token prediction only and will hallucinate or drift on open-ended prompts.
It has no tool-calling or chat-following capability without further fine-tuning.
Output quality is representative of a ~12M-parameter model trained on raw web text.
Maximum context length is 64 tokens.

Credits and License

Modified from NanoGPT by Andrej Karpathy.

Downloads last month: -; Downloads are not tracked for this model. How to track