helloboy91 commited on
Commit
325c500
·
verified ·
1 Parent(s): fd146cb

Simplify local training introduction

Browse files
Files changed (1) hide show
  1. README.md +5 -5
README.md CHANGED
@@ -16,15 +16,15 @@ tags:
16
 
17
  Alloy is an experimental 311.9M-parameter English causal language model with a 2,048-token context window. It is a base model, not an instruction-following assistant. This release packages the best checkpoint after **5.0003B training tokens** in an inference-only format; optimizer state, raw training data, and private machine paths are not included.
18
 
19
- ## Why this model exists
20
 
21
- This is a transparent small-model training project. Rather than treating data as a fixed black box, we trained in stages, evaluated each stage, inspected failure modes such as repetition and corrupted/non-English text, and revised the next corpus accordingly.
22
 
23
- ## A starter-friendly local-training experiment
24
 
25
- The main value of Alloy is not frontier benchmark performance. It is a documented example of training a self-owned small language model on accessible hardware: the 312M-parameter, 2K-context Alloy stage was trained and evaluated locally on a consumer **NVIDIA RTX 3060 Ti with 8 GB VRAM**. BF16 mixed precision, microbatch size 1, gradient accumulation, and gradient checkpointing made that configuration practical.
26
 
27
- This is intended to help others quickly try a complete model-training loop on a non-datacenter computer: prepare inspectable data, train with resumable checkpoints, watch validation and hardware metrics, evaluate a frozen checkpoint, and publish reproducible artifacts. It is deliberately a starter project, not a recipe for matching models trained on large multi-GPU clusters.
28
 
29
  1. **Bronze, Silver, Gold**: three 1B-token stages established a shared 32,768-token byte-level BPE tokenizer and a 2K-context decoder-only model.
30
  2. **Lead / Alloy**: a 2B-token continuation used a pinned public-source mix: FineWeb-Edu (`CC-MAIN-2025-26`), English Wikipedia (`20231101.en`), and TinyStories. FineWeb-Edu documents had upstream quality score >= 3; all sources then passed English-probability, Unicode/corruption, boilerplate, URL/fragment, and repeated-text filters. Source-shard holdouts separated training from validation and test. Line breaks were deliberately preserved.
 
16
 
17
  Alloy is an experimental 311.9M-parameter English causal language model with a 2,048-token context window. It is a base model, not an instruction-following assistant. This release packages the best checkpoint after **5.0003B training tokens** in an inference-only format; optimizer state, raw training data, and private machine paths are not included.
18
 
19
+ ## Train a model on a normal PC
20
 
21
+ Alloy is a starter project for training your own language model on accessible hardware. Its 312M-parameter, 2K-context stage was trained and evaluated locally on an **NVIDIA RTX 3060 Ti with 8 GB VRAM**.
22
 
23
+ It shows a complete small-model workflow: prepare data, resume training, track validation, evaluate the result, and publish it. It is for learning and experimentation, not for matching models trained on large GPU clusters.
24
 
25
+ ## Why this model exists
26
 
27
+ This is a transparent small-model training project. Rather than treating data as a fixed black box, we trained in stages, evaluated each stage, inspected failure modes such as repetition and corrupted/non-English text, and revised the next corpus accordingly.
28
 
29
  1. **Bronze, Silver, Gold**: three 1B-token stages established a shared 32,768-token byte-level BPE tokenizer and a 2K-context decoder-only model.
30
  2. **Lead / Alloy**: a 2B-token continuation used a pinned public-source mix: FineWeb-Edu (`CC-MAIN-2025-26`), English Wikipedia (`20231101.en`), and TinyStories. FineWeb-Edu documents had upstream quality score >= 3; all sources then passed English-probability, Unicode/corruption, boilerplate, URL/fragment, and repeated-text filters. Source-shard holdouts separated training from validation and test. Line breaks were deliberately preserved.