IAU-1 / README.md
BrodyMakezAI's picture
Upload 6 files
ce68c99 verified
|
Raw History Blame Contribute Delete
1.84 kB
---
license: other
license_name: iau-1-research-license
license_link: LICENSE
language: en
library_name: pytorch
tags: [gpt, pretrained, from-scratch, text-generation]
datasets:
- Skylion007/openwebtext
- wikimedia/wikipedia
- rojagtap/bookcorpus
- HuggingFaceFW/fineweb
---
# IAU-1
A ~209M parameter GPT trained completely from scratch. Custom architecture, not a `transformers` class, so the model code is included in `model.py`.
- 24 layers, 12 heads, 768 embedding dim, 1024 context, learned positional embeddings, pre-LN, GELU, no biases, tied input/output embeddings
- Tokenizer: tiktoken `gpt2` (vocab padded to 50304)
- Trained for 4000 steps at ~65k tokens/step (about 0.26B tokens)
## Training data
Streamed mix: OpenWebText 20% (stand-in for WebText2, which was never released), Wikipedia (20231101.en) 10%, BookCorpus 20%, FineWeb sample-10BT 50%.
## Usage
```python
from model import load_iau1, generate
m = load_iau1('model.safetensors')
print(generate(m, 'The history of the internet', n=100))
```
## License
Custom research license, see `LICENSE`. Short version: free to use for research, learning and personal projects, **no commercial use**. The training data has mixed licensing (summary below), so the weights are not offered under a standard open license.
| Dataset | License / terms |
|---|---|
| OpenWebText | CC0 for the compilation; the underlying web pages belong to their authors |
| Wikipedia | CC BY-SA 3.0 and GFDL |
| BookCorpus | No license stated; many of the books carry personal-use-only terms |
| FineWeb | ODC-By 1.0, plus Common Crawl terms of use |
## Limitations
Small model, trained on roughly 1.3B tokens at most, so expect fluent-ish but often wrong or rambling text. Base model only, no instruction tuning or safety tuning. It can reproduce biases and errors from its training data.