--- license: other license_name: iau-1-research-license license_link: LICENSE language: en library_name: pytorch tags: [gpt, pretrained, from-scratch, text-generation] datasets: - Skylion007/openwebtext - wikimedia/wikipedia - rojagtap/bookcorpus - HuggingFaceFW/fineweb --- # IAU-1 A ~209M parameter GPT trained completely from scratch. Custom architecture, not a `transformers` class, so the model code is included in `model.py`. - 24 layers, 12 heads, 768 embedding dim, 1024 context, learned positional embeddings, pre-LN, GELU, no biases, tied input/output embeddings - Tokenizer: tiktoken `gpt2` (vocab padded to 50304) - Trained for 4000 steps at ~65k tokens/step (about 0.26B tokens) ## Training data Streamed mix: OpenWebText 20% (stand-in for WebText2, which was never released), Wikipedia (20231101.en) 10%, BookCorpus 20%, FineWeb sample-10BT 50%. ## Usage ```python from model import load_iau1, generate m = load_iau1('model.safetensors') print(generate(m, 'The history of the internet', n=100)) ``` ## License Custom research license, see `LICENSE`. Short version: free to use for research, learning and personal projects, **no commercial use**. The training data has mixed licensing (summary below), so the weights are not offered under a standard open license. | Dataset | License / terms | |---|---| | OpenWebText | CC0 for the compilation; the underlying web pages belong to their authors | | Wikipedia | CC BY-SA 3.0 and GFDL | | BookCorpus | No license stated; many of the books carry personal-use-only terms | | FineWeb | ODC-By 1.0, plus Common Crawl terms of use | ## Limitations Small model, trained on roughly 1.3B tokens at most, so expect fluent-ish but often wrong or rambling text. Base model only, no instruction tuning or safety tuning. It can reproduce biases and errors from its training data.