| --- |
| library_name: transformers |
| tags: |
| - causal-lm |
| - language-model |
| - small-language-model |
| - tinystories |
| - experimental |
| - custom-architecture |
| --- |
| |
| # III-TinyStories |
|
|
| III-TinyStories is a small autoregressive language model trained from scratch on the [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) dataset. |
|
|
| The model is primarily intended as an experimental testbed for the III architecture rather than as a production-ready general-purpose language model. |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| III is a custom decoder architecture that combines causal self-attention with gated causal convolutional layers. |
|
|
| The architecture includes: |
|
|
| - Hybrid attention and causal convolutional token mixing |
| - Causal dilated depthwise convolutions |
| - RMSNorm |
| - Query/key RMS normalization |
| - Rotary Position Embeddings (RoPE) |
| - SwiGLU-style MLP blocks |
| - KV caching for autoregressive generation |
| - Tied input/output token embeddings |
|
|
| The model was implemented as a custom Transformers architecture and can be used for autoregressive text generation. |
|
|
| - **Developed by:** KordAI |
| - **Model type:** Autoregressive causal language model |
| - **Language:** English |
| - **Training dataset:** `roneneldan/TinyStories` |
| - **Training objective:** Next-token prediction |
| - **License:** Not specified |
|
|
| ### Model Architecture |
|
|
| III does not use a conventional all-attention Transformer stack. |
|
|
| Each decoder layer uses either: |
|
|
| - causal self-attention, or |
| - a gated causal convolutional mixer, |
|
|
| depending on the layer configuration. |
|
|
| The attention blocks use: |
|
|
| - Multi-head QKV projection |
| - RMS-normalized queries and keys |
| - RoPE |
| - PyTorch scaled dot-product attention |
|
|
| The convolutional blocks use: |
|
|
| - Gated projections |
| - Depthwise causal convolutions |
| - Configurable receptive fields |
| - Configurable dilation |
|
|
| The model also implements custom cache handling for autoregressive generation. |
|
|
| ## Uses |
|
|
| ### Direct Use |
|
|
| This model is primarily intended for: |
|
|
| - Experimenting with small language model architectures |
| - Studying training behavior of hybrid sequence mixers |
| - Testing autoregressive generation |
| - Comparing the III architecture against conventional Transformer baselines |
| - Educational and research experimentation |
|
|
| ### Downstream Use |
|
|
| The model may be fine-tuned for small-scale experimental tasks, but its performance for downstream applications has not been comprehensively evaluated. |
|
|
| ### Out-of-Scope Use |
|
|
| This model should not be considered suitable for: |
|
|
| - Production applications |
| - Safety-critical applications |
| - Factual question answering |
| - Reliable knowledge retrieval |
| - High-stakes decision making |
| - Representing current world knowledge |
|
|
| The model was trained on TinyStories and therefore has a deliberately narrow training distribution. |
|
|
| ## Bias, Risks, and Limitations |
|
|
| The main limitations are a direct consequence of the training data and experimental nature of the model. |
|
|
| TinyStories contains short, synthetic stories with relatively simple language. As a result, the model should not be expected to have broad factual knowledge or the linguistic coverage of a larger general-purpose language model. |
|
|
| The model may: |
|
|
| - Generate incorrect or nonsensical statements |
| - Repeat patterns from the training distribution |
| - Produce grammatically unusual text |
| - Fail on topics outside the TinyStories distribution |
| - Exhibit behavior that does not generalize to larger and more diverse corpora |
|
|
| Training and evaluation results on TinyStories should therefore not be interpreted as evidence of general-purpose LLM capability. |
|
|
| ### Recommendations |
|
|
| Use this model primarily for experimentation and architecture research. |
|
|
| For meaningful comparisons between architectures, keep the tokenizer, training data, token budget, context length, optimizer, and other training conditions consistent. |
|
|
| ## How to Get Started with the Model |
|
|
| ```python |
| import torch |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| repo = "KordAI/III-tinystories" |
| |
| tokenizer = AutoTokenizer.from_pretrained( |
| repo, |
| trust_remote_code=True, |
| ) |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| repo, |
| trust_remote_code=True, |
| ) |
| |
| model.eval() |
| |
| prompt = "Once upon a time, there was a little boy named Tom." |
| |
| inputs = tokenizer( |
| prompt, |
| return_tensors="pt", |
| ) |
| |
| with torch.inference_mode(): |
| output_ids = model.generate( |
| **inputs, |
| max_new_tokens=100, |
| do_sample=True, |
| temperature=0.8, |
| top_p=0.95, |
| use_cache=True, |
| eos_token_id=tokenizer.eos_token_id, |
| pad_token_id=tokenizer.pad_token_id, |
| ) |
| |
| print( |
| tokenizer.decode( |
| output_ids[0], |
| skip_special_tokens=True, |
| ) |
| ) |
| ``` |
