---
library_name: transformers
tags:
- causal-lm
- language-model
- small-language-model
- tinystories
- experimental
- custom-architecture
---

# III-TinyStories

III-TinyStories is a small autoregressive language model trained from scratch on the [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) dataset.

The model is primarily intended as an experimental testbed for the III architecture rather than as a production-ready general-purpose language model.

## Model Details

### Model Description

III is a custom decoder architecture that combines causal self-attention with gated causal convolutional layers.

The architecture includes:

- Hybrid attention and causal convolutional token mixing
- Causal dilated depthwise convolutions
- RMSNorm
- Query/key RMS normalization
- Rotary Position Embeddings (RoPE)
- SwiGLU-style MLP blocks
- KV caching for autoregressive generation
- Tied input/output token embeddings

The model was implemented as a custom Transformers architecture and can be used for autoregressive text generation.

- **Developed by:** KordAI
- **Model type:** Autoregressive causal language model
- **Language:** English
- **Training dataset:** `roneneldan/TinyStories`
- **Training objective:** Next-token prediction
- **License:** Not specified

### Model Architecture

III does not use a conventional all-attention Transformer stack.

Each decoder layer uses either:

- causal self-attention, or
- a gated causal convolutional mixer,

depending on the layer configuration.

The attention blocks use:

- Multi-head QKV projection
- RMS-normalized queries and keys
- RoPE
- PyTorch scaled dot-product attention

The convolutional blocks use:

- Gated projections
- Depthwise causal convolutions
- Configurable receptive fields
- Configurable dilation

The model also implements custom cache handling for autoregressive generation.

## Uses

### Direct Use

This model is primarily intended for:

- Experimenting with small language model architectures
- Studying training behavior of hybrid sequence mixers
- Testing autoregressive generation
- Comparing the III architecture against conventional Transformer baselines
- Educational and research experimentation

### Downstream Use

The model may be fine-tuned for small-scale experimental tasks, but its performance for downstream applications has not been comprehensively evaluated.

### Out-of-Scope Use

This model should not be considered suitable for:

- Production applications
- Safety-critical applications
- Factual question answering
- Reliable knowledge retrieval
- High-stakes decision making
- Representing current world knowledge

The model was trained on TinyStories and therefore has a deliberately narrow training distribution.

## Bias, Risks, and Limitations

The main limitations are a direct consequence of the training data and experimental nature of the model.

TinyStories contains short, synthetic stories with relatively simple language. As a result, the model should not be expected to have broad factual knowledge or the linguistic coverage of a larger general-purpose language model.

The model may:

- Generate incorrect or nonsensical statements
- Repeat patterns from the training distribution
- Produce grammatically unusual text
- Fail on topics outside the TinyStories distribution
- Exhibit behavior that does not generalize to larger and more diverse corpora

Training and evaluation results on TinyStories should therefore not be interpreted as evidence of general-purpose LLM capability.

### Recommendations

Use this model primarily for experimentation and architecture research.

For meaningful comparisons between architectures, keep the tokenizer, training data, token budget, context length, optimizer, and other training conditions consistent.

## How to Get Started with the Model

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "KordAI/III-tinystories"

tokenizer = AutoTokenizer.from_pretrained(
    repo,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
)

model.eval()

prompt = "Once upon a time, there was a little boy named Tom."

inputs = tokenizer(
    prompt,
    return_tensors="pt",
)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        use_cache=True,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

print(
    tokenizer.decode(
        output_ids[0],
        skip_special_tokens=True,
    )
)
```