| --- |
| language: |
| - en |
| license: mit |
| pipeline_tag: text-generation |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| - wikimedia/wikipedia |
| - roneneldan/TinyStories |
| tags: |
| - base-model |
| - causal-lm |
| - language-model |
| - text-generation |
| - experimental |
| - english |
| - from-scratch |
| - 312m |
| - 2k-context |
| - pytorch |
| - safetensors |
| - byte-level-bpe |
| - continued-pretraining |
| - lm-eval |
| - fineweb-edu |
| - wikipedia |
| - tinystories |
| - consumer-gpu |
| - rtx-3060-ti |
| - 8gb-vram |
| --- |
| |
| # Alloy 312M 2K |
|
|
| Alloy is an experimental 311.9M-parameter English causal language model with a 2,048-token context window. It is a base model for completing English text, not an instruction-following assistant. It was trained from scratch on **5.0B tokens**. |
|
|
| Try it in the free [Alloy 312M Demo](https://huggingface.co/spaces/helloboy91/alloy-312m-demo). |
|
|
| ## Train a model on a normal PC |
|
|
| Alloy is a starter project for training a language model on accessible hardware. Its 312M-parameter, 2K-context model was trained and evaluated on an **NVIDIA RTX 3060 Ti with 8 GB VRAM**. |
|
|
| ## Training and data |
|
|
| - Architecture: decoder-only Transformer, 22 layers, 1,024 hidden size, 16 attention heads, and a 32,768-token byte-level BPE tokenizer. |
| - Training: three 1B-token stages followed by a 2B-token continuation. |
| - Data: a filtered English mix of [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), [English Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia), and [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). |
|
|
| ## Evaluation |
|
|
| Zero-shot evaluation used `lm-evaluation-harness` 0.4.13. `standard-eval.json` contains the full Alloy result. |
|
|
| | Task | Bronze | Silver | Gold | Alloy | |
| | --- | ---: | ---: | ---: | ---: | |
| | Best validation loss ↓ | 2.998 | 2.695 | 2.632 | **2.609** | |
| | ARC-Easy | 42.93% | 47.98% | 49.75% | **49.92%** | |
| | ARC-Challenge | 18.77% | 21.25% | 20.99% | **21.42%** | |
| | HellaSwag | 27.39% | 27.95% | 28.63% | **29.87%** | |
| | PIQA | 59.09% | 59.96% | 60.66% | **62.95%** | |
| | SciQ | 64.70% | 68.80% | 69.50% | **71.90%** | |
| | WinoGrande | 50.36% | 49.41% | 49.64% | **50.67%** | |
|
|
| Bronze, Silver, and Gold are successive 1B-token stages; Alloy adds the final 2B-token continuation. Alloy improves on Gold on every listed task. |
|
|
| ## Use |
|
|
| ```bash |
| pip install -r requirements.txt |
| python -m tiny_english.generate --model-dir . --device cuda --prompt "The invention of the printing press changed" --max-new-tokens 128 --temperature 0.7 --top-k 40 --seed 42 |
| ``` |
|
|
| Generation can repeat phrases, make factual errors, or produce incoherent continuations. |
|
|
| ## Next |
|
|
| The next planned direction is supervised fine-tuning on curated instruction data. |
|
|
| ## License |
|
|
| The model weights, tokenizer, and bundled inference code are released under MIT. Training sources remain subject to their own terms. |
|
|
