Simplify model card for readers

README.md

@@ -32,31 +32,27 @@ tags:
 
 # Alloy 312M 2K
 
-Alloy is an experimental 311.9M-parameter English causal language model with a 2,048-token context window. It is a base model, not an instruction-following assistant. This release packages the best checkpoint after **5.0003B training tokens** in an inference-only format; optimizer state, raw training data, and private machine paths are not included.
+Alloy is an experimental 311.9M-parameter English causal language model with a 2,048-token context window. It is a base model for completing English text, not an instruction-following assistant. It was trained from scratch on **5.0B tokens**.
 
-## Train a model on a normal PC
-
-Alloy is a starter project for training your own language model on accessible hardware. Its 312M-parameter, 2K-context stage was trained and evaluated locally on an **NVIDIA RTX 3060 Ti with 8 GB VRAM**.
-
-It shows a complete small-model workflow: prepare data, resume training, track validation, evaluate the result, and publish it. It is for learning and experimentation, not for matching models trained on large GPU clusters.
+Try it in the free [Alloy 312M Demo](https://huggingface.co/spaces/helloboy91/alloy-312m-demo).
 
-## Why this model exists
+## Train a model on a normal PC
 
-This is a transparent small-model training project. Rather than treating data as a fixed black box, we trained in stages, evaluated each stage, inspected failure modes such as repetition and corrupted/non-English text, and revised the next corpus accordingly.
+Alloy is a starter project for training a language model on accessible hardware. Its 312M-parameter, 2K-context model was trained and evaluated on an **NVIDIA RTX 3060 Ti with 8 GB VRAM**.
 
-1. **Bronze, Silver, Gold**: three 1B-token stages established a shared 32,768-token byte-level BPE tokenizer and a 2K-context decoder-only model.
-2. **Lead / Alloy**: a 2B-token continuation used a pinned public-source mix: FineWeb-Edu (`CC-MAIN-2025-26`), English Wikipedia (`20231101.en`), and TinyStories. FineWeb-Edu documents had upstream quality score >= 3; all sources then passed English-probability, Unicode/corruption, boilerplate, URL/fragment, and repeated-text filters. Source-shard holdouts separated training from validation and test. Line breaks were deliberately preserved.
-3. The completed Alloy checkpoint was selected by its independent matching validation split, then evaluated zero-shot without prompt tuning.
+## Training and data
 
-The source revisions, filters, tokenizer hash, model configuration, and data manifest hash are recorded in `training_summary.json` and `config.json`. These are quality controls, not a guarantee of factual accuracy, decontamination, or absence of duplicates.
+- Architecture: decoder-only Transformer, 22 layers, 1,024 hidden size, 16 attention heads, and a 32,768-token byte-level BPE tokenizer.
+- Training: three 1B-token stages followed by a 2B-token continuation.
+- Data: a filtered English mix of [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), [English Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia), and [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories).
 
 ## Evaluation
 
-Full zero-shot evaluation used `lm-evaluation-harness` 0.4.13, batch size 1, no bootstrap resampling, and the native checkpoint. `standard-eval.json` contains the complete protocol and raw Alloy result. The stages below used the same zero-shot task suite and raw-accuracy metric.
+Zero-shot evaluation used `lm-evaluation-harness` 0.4.13. `standard-eval.json` contains the full Alloy result.
 
 | Task | Bronze | Silver | Gold | Alloy |
 | --- | ---: | ---: | ---: | ---: |
-| Best matching validation loss ↓ | 2.998 | 2.695 | 2.632 | **2.609** |
+| Best validation loss ↓ | 2.998 | 2.695 | 2.632 | **2.609** |
 | ARC-Easy | 42.93% | 47.98% | 49.75% | **49.92%** |
 | ARC-Challenge | 18.77% | 21.25% | 20.99% | **21.42%** |
 | HellaSwag | 27.39% | 27.95% | 28.63% | **29.87%** |
@@ -64,25 +60,21 @@ Full zero-shot evaluation used `lm-evaluation-harness` 0.4.13, batch size 1, no
 | SciQ | 64.70% | 68.80% | 69.50% | **71.90%** |
 | WinoGrande | 50.36% | 49.41% | 49.64% | **50.67%** |
 
-Alloy improves on Gold on every listed task. Bronze, Silver, and Gold correspond to the successive 1B-token stages; Alloy adds the 2B-token Lead continuation for roughly 5B total recorded training tokens. Each loss is the best value on that stage's matching validation split; it is useful as a stage record, but different validation corpora mean it is not a perfectly controlled cross-stage benchmark.
-
-These scores show clear progress over the project’s earlier checkpoints, but they do not make Alloy competitive with mature, heavily trained public models of a similar parameter count. Benchmark contamination and harness-version differences remain possible; do not compare scores across protocols as exact leaderboard rankings.
+Bronze, Silver, and Gold are successive 1B-token stages; Alloy adds the final 2B-token continuation. Alloy improves on Gold on every listed task.
 
 ## Use
 
-This repository is a native safetensors export rather than a Transformers `AutoModel` package. It includes the minimal `tiny_english` inference module.
-
 ```bash
 pip install -r requirements.txt
 python -m tiny_english.generate --model-dir . --device cuda --prompt "The invention of the printing press changed" --max-new-tokens 128 --temperature 0.7 --top-k 40 --seed 42
 ```
 
-Generation can repeat phrases, make factual errors, or produce incoherent continuations. Do not use it for high-stakes decisions or as an assistant without additional evaluation and alignment work.
+Generation can repeat phrases, make factual errors, or produce incoherent continuations.
 
-## Continue the work
+## Next
 
-Contributions and continuation experiments are welcome. Please retain the tokenizer and 2K context contract, report the exact data mixture and token count, keep independent validation/test data, and publish the same zero-shot evaluation protocol alongside qualitative samples. The next planned direction is supervised fine-tuning (SFT) on carefully curated instruction data, so the base model can become more specifically usable while preserving a transparent base-model checkpoint for comparison.
+The next planned direction is supervised fine-tuning on curated instruction data.
 
-## Data, license, and limitations
+## License
 
-Model weights, tokenizer, and bundled inference code are released under MIT. The training sources remain subject to their own terms: FineWeb-Edu, [Wikimedia Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia), and [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). No raw source documents or prepared token datasets are included here. Review the upstream dataset licenses and documentation before redistributing data or training derivatives.
+The model weights, tokenizer, and bundled inference code are released under MIT. Training sources remain subject to their own terms.