File size: 4,648 Bytes
0462244
 
2c2c911
98a5a1b
 
 
 
 
 
0462244
 
2c2c911
0462244
2c2c911
0462244
2c2c911
0462244
 
 
 
 
2c2c911
0462244
2c2c911
0462244
2c2c911
 
 
 
 
 
 
 
0462244
2c2c911
0462244
2c2c911
 
 
 
 
 
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
 
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
 
 
 
0462244
2c2c911
0462244
2c2c911
 
 
 
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
 
 
 
 
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
 
 
 
 
 
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
 
 
 
 
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
0462244
2c2c911
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
---
library_name: transformers
tags:
- causal-lm
- language-model
- small-language-model
- tinystories
- experimental
- custom-architecture
---

# III-TinyStories

III-TinyStories is a small autoregressive language model trained from scratch on the [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) dataset.

The model is primarily intended as an experimental testbed for the III architecture rather than as a production-ready general-purpose language model.

## Model Details

### Model Description

III is a custom decoder architecture that combines causal self-attention with gated causal convolutional layers.

The architecture includes:

- Hybrid attention and causal convolutional token mixing
- Causal dilated depthwise convolutions
- RMSNorm
- Query/key RMS normalization
- Rotary Position Embeddings (RoPE)
- SwiGLU-style MLP blocks
- KV caching for autoregressive generation
- Tied input/output token embeddings

The model was implemented as a custom Transformers architecture and can be used for autoregressive text generation.

- **Developed by:** KordAI
- **Model type:** Autoregressive causal language model
- **Language:** English
- **Training dataset:** `roneneldan/TinyStories`
- **Training objective:** Next-token prediction
- **License:** Not specified

### Model Architecture

III does not use a conventional all-attention Transformer stack.

Each decoder layer uses either:

- causal self-attention, or
- a gated causal convolutional mixer,

depending on the layer configuration.

The attention blocks use:

- Multi-head QKV projection
- RMS-normalized queries and keys
- RoPE
- PyTorch scaled dot-product attention

The convolutional blocks use:

- Gated projections
- Depthwise causal convolutions
- Configurable receptive fields
- Configurable dilation

The model also implements custom cache handling for autoregressive generation.

## Uses

### Direct Use

This model is primarily intended for:

- Experimenting with small language model architectures
- Studying training behavior of hybrid sequence mixers
- Testing autoregressive generation
- Comparing the III architecture against conventional Transformer baselines
- Educational and research experimentation

### Downstream Use

The model may be fine-tuned for small-scale experimental tasks, but its performance for downstream applications has not been comprehensively evaluated.

### Out-of-Scope Use

This model should not be considered suitable for:

- Production applications
- Safety-critical applications
- Factual question answering
- Reliable knowledge retrieval
- High-stakes decision making
- Representing current world knowledge

The model was trained on TinyStories and therefore has a deliberately narrow training distribution.

## Bias, Risks, and Limitations

The main limitations are a direct consequence of the training data and experimental nature of the model.

TinyStories contains short, synthetic stories with relatively simple language. As a result, the model should not be expected to have broad factual knowledge or the linguistic coverage of a larger general-purpose language model.

The model may:

- Generate incorrect or nonsensical statements
- Repeat patterns from the training distribution
- Produce grammatically unusual text
- Fail on topics outside the TinyStories distribution
- Exhibit behavior that does not generalize to larger and more diverse corpora

Training and evaluation results on TinyStories should therefore not be interpreted as evidence of general-purpose LLM capability.

### Recommendations

Use this model primarily for experimentation and architecture research.

For meaningful comparisons between architectures, keep the tokenizer, training data, token budget, context length, optimizer, and other training conditions consistent.

## How to Get Started with the Model

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "KordAI/III-tinystories"

tokenizer = AutoTokenizer.from_pretrained(
    repo,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
)

model.eval()

prompt = "Once upon a time, there was a little boy named Tom."

inputs = tokenizer(
    prompt,
    return_tensors="pt",
)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        use_cache=True,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

print(
    tokenizer.decode(
        output_ids[0],
        skip_special_tokens=True,
    )
)
```