| --- |
| language: |
| - en |
| license: mit |
| library_name: pytorch |
| pipeline_tag: text-generation |
| datasets: |
| - roneneldan/TinyStories |
| tags: |
| - tiny-english |
| - from-scratch |
| - story-completion |
| - safetensors |
| inference: false |
| --- |
| |
| # Tiny English 30M — v0.2.0 |
|
|
| An experimental 29.9M-parameter decoder-only model for simple English story completion. It was trained from scratch on a prepared TinyStories subset. It is not an instruction-tuned assistant, factual reference, or production system. |
|
|
| This release replaces v0.1.0 with the checkpoint selected from the additional 100M-token stage. A later 200M-token stage overfit the fixed corpus and is deliberately not released. |
|
|
| ## Newer model: Alloy 312M 2K |
|
|
| For a larger, more general English base-model experiment, see [Alloy 312M 2K](https://huggingface.co/helloboy91/tiny-base-300m). It has 311.9M parameters, a 2,048-token context window, roughly 5B recorded training tokens, and full zero-shot benchmark results. Alloy is also a practical example of training a model locally on an RTX 3060 Ti with 8 GB VRAM. It remains experimental and is not an instruction-tuned assistant. |
|
|
| ## Contents and use |
|
|
| `model.safetensors` contains FP32 inference weights. `config.json`, `tokenizer.json`, `training_summary.json`, `evaluation.json`, and `SHA256SUMS` record architecture, provenance, scores, and integrity data. The bundled `tiny_english/` package is standalone inference code; optimizer state, raw/cleaned stories, training logs, and local paths are excluded. |
|
|
| ```bash |
| python -m pip install -r requirements.txt |
| python -m tiny_english.generate --model-dir . --prompt "Once upon a time, a little rabbit found a blue hat." --max-new-tokens 100 --temperature 0.7 --seed 42 |
| ``` |
|
|
| This is a custom PyTorch export, not a Transformers `AutoModelForCausalLM` model. Review the bundled code before use. |
|
|
| ## Architecture |
|
|
| | Setting | Value | |
| | --- | --- | |
| | Transformer blocks / hidden width | 8 / 512 | |
| | Attention | 8 causal heads, head width 64 | |
| | Feed-forward | SwiGLU, intermediate width 1,408 | |
| | Context | 1,024 tokens | |
| | Vocabulary | 8,192 byte-level BPE tokens | |
| | Positions / normalization | RoPE / RMSNorm | |
| | Embeddings | Tied input/output | |
|
|
| ## Training and evaluation |
|
|
| The selected checkpoint represents **148,910,080 processed tokens**: 54,702,080 prior tokens plus 94,208,000 tokens from the selected additional-training stage checkpoint (step 23,000). Training used 4,096 tokens/update, AdamW, a peak learning rate of 0.0003, and CUDA BF16 on an RTX 3060 Ti for the later stages. |
|
|
| Full nonoverlapping held-out evaluation: |
|
|
| | Split | Predicted tokens | Loss | Perplexity | |
| | --- | ---: | ---: | ---: | |
| | Validation | 105,075 | 1.780266 | 5.931436 | |
| | Test | 102,850 | 1.846464 | 6.337368 | |
|
|
| The training corpus was prepared from `roneneldan/TinyStories` at revision `f54c09fd23315a6f9c86f9dc80f725de7d8f9c64`. It has 96,704 training stories (20,097,225 packed tokens), with deterministic held-out validation and test splits. The tokenizer was trained only on the training split. |
|
|
| TinyStories is synthetic short-story data generated using GPT-3.5 and GPT-4. The source dataset is licensed CDLA-Sharing-1.0; this model release is MIT only to the extent of the author’s rights. See `DATA_LICENSE.md`. |
|
|
| ## Limitations |
|
|
| The model can repeat, drift, or produce implausible story continuations. It has no instruction tuning, retrieval, tool use, or factual reliability guarantees. Do not use its output for consequential decisions. |
|
|
