|
Download README.md from BrodyMakezAI/IAU-1: direct link, hf CLI and curl.
- Browser
- Download file 1.84 kB
-
https://huggingface.co/BrodyMakezAI/IAU-1/resolve/main/README.md
- Command line
-
hf download hf://BrodyMakezAI/IAU-1/README.md
-
curl -L -o README.md https://huggingface.co/BrodyMakezAI/IAU-1/resolve/main/README.md
1.84 kB
| license: other | |
| license_name: iau-1-research-license | |
| license_link: LICENSE | |
| language: en | |
| library_name: pytorch | |
| tags: [gpt, pretrained, from-scratch, text-generation] | |
| datasets: | |
| - Skylion007/openwebtext | |
| - wikimedia/wikipedia | |
| - rojagtap/bookcorpus | |
| - HuggingFaceFW/fineweb | |
| # IAU-1 | |
| A ~209M parameter GPT trained completely from scratch. Custom architecture, not a `transformers` class, so the model code is included in `model.py`. | |
| - 24 layers, 12 heads, 768 embedding dim, 1024 context, learned positional embeddings, pre-LN, GELU, no biases, tied input/output embeddings | |
| - Tokenizer: tiktoken `gpt2` (vocab padded to 50304) | |
| - Trained for 4000 steps at ~65k tokens/step (about 0.26B tokens) | |
| ## Training data | |
| Streamed mix: OpenWebText 20% (stand-in for WebText2, which was never released), Wikipedia (20231101.en) 10%, BookCorpus 20%, FineWeb sample-10BT 50%. | |
| ## Usage | |
| ```python | |
| from model import load_iau1, generate | |
| m = load_iau1('model.safetensors') | |
| print(generate(m, 'The history of the internet', n=100)) | |
| ``` | |
| ## License | |
| Custom research license, see `LICENSE`. Short version: free to use for research, learning and personal projects, **no commercial use**. The training data has mixed licensing (summary below), so the weights are not offered under a standard open license. | |
| | Dataset | License / terms | | |
| |---|---| | |
| | OpenWebText | CC0 for the compilation; the underlying web pages belong to their authors | | |
| | Wikipedia | CC BY-SA 3.0 and GFDL | | |
| | BookCorpus | No license stated; many of the books carry personal-use-only terms | | |
| | FineWeb | ODC-By 1.0, plus Common Crawl terms of use | | |
| ## Limitations | |
| Small model, trained on roughly 1.3B tokens at most, so expect fluent-ish but often wrong or rambling text. Base model only, no instruction tuning or safety tuning. It can reproduce biases and errors from its training data. | |