Simplify local training introduction
Browse files
README.md
CHANGED
|
@@ -16,15 +16,15 @@ tags:
|
|
| 16 |
|
| 17 |
Alloy is an experimental 311.9M-parameter English causal language model with a 2,048-token context window. It is a base model, not an instruction-following assistant. This release packages the best checkpoint after **5.0003B training tokens** in an inference-only format; optimizer state, raw training data, and private machine paths are not included.
|
| 18 |
|
| 19 |
-
##
|
| 20 |
|
| 21 |
-
|
| 22 |
|
| 23 |
-
|
| 24 |
|
| 25 |
-
|
| 26 |
|
| 27 |
-
This is
|
| 28 |
|
| 29 |
1. **Bronze, Silver, Gold**: three 1B-token stages established a shared 32,768-token byte-level BPE tokenizer and a 2K-context decoder-only model.
|
| 30 |
2. **Lead / Alloy**: a 2B-token continuation used a pinned public-source mix: FineWeb-Edu (`CC-MAIN-2025-26`), English Wikipedia (`20231101.en`), and TinyStories. FineWeb-Edu documents had upstream quality score >= 3; all sources then passed English-probability, Unicode/corruption, boilerplate, URL/fragment, and repeated-text filters. Source-shard holdouts separated training from validation and test. Line breaks were deliberately preserved.
|
|
|
|
| 16 |
|
| 17 |
Alloy is an experimental 311.9M-parameter English causal language model with a 2,048-token context window. It is a base model, not an instruction-following assistant. This release packages the best checkpoint after **5.0003B training tokens** in an inference-only format; optimizer state, raw training data, and private machine paths are not included.
|
| 18 |
|
| 19 |
+
## Train a model on a normal PC
|
| 20 |
|
| 21 |
+
Alloy is a starter project for training your own language model on accessible hardware. Its 312M-parameter, 2K-context stage was trained and evaluated locally on an **NVIDIA RTX 3060 Ti with 8 GB VRAM**.
|
| 22 |
|
| 23 |
+
It shows a complete small-model workflow: prepare data, resume training, track validation, evaluate the result, and publish it. It is for learning and experimentation, not for matching models trained on large GPU clusters.
|
| 24 |
|
| 25 |
+
## Why this model exists
|
| 26 |
|
| 27 |
+
This is a transparent small-model training project. Rather than treating data as a fixed black box, we trained in stages, evaluated each stage, inspected failure modes such as repetition and corrupted/non-English text, and revised the next corpus accordingly.
|
| 28 |
|
| 29 |
1. **Bronze, Silver, Gold**: three 1B-token stages established a shared 32,768-token byte-level BPE tokenizer and a 2K-context decoder-only model.
|
| 30 |
2. **Lead / Alloy**: a 2B-token continuation used a pinned public-source mix: FineWeb-Edu (`CC-MAIN-2025-26`), English Wikipedia (`20231101.en`), and TinyStories. FineWeb-Edu documents had upstream quality score >= 3; all sources then passed English-probability, Unicode/corruption, boilerplate, URL/fragment, and repeated-text filters. Source-shard holdouts separated training from validation and test. Line breaks were deliberately preserved.
|