|
Download README.md from WithinUsAI/FrontD.11m: direct link, hf CLI and curl.
- Browser
- Download file 825 Bytes
-
https://huggingface.co/WithinUsAI/FrontD.11m/resolve/main/README.md
- Command line
-
hf download hf://WithinUsAI/FrontD.11m/README.md
-
curl -L -o README.md https://huggingface.co/WithinUsAI/FrontD.11m/resolve/main/README.md
825 Bytes
| license: mit | |
| library_name: pytorch | |
| pipeline_tag: text-generation | |
| tags: | |
| - text-generation | |
| - pretraining | |
| - decoder-only | |
| - from-scratch | |
| - 11m | |
| - 1096-context | |
| - rope | |
| - rmsnorm | |
| - swiglu | |
| # FrontD.11m | |
| FrontD.11m is an approximately 11-million-parameter decoder-only | |
| language model trained completely from scratch. | |
| ## Architecture | |
| - Parameters: 10,996,608 | |
| - Context: 1096 | |
| - Vocabulary: 16000 | |
| - Hidden size: 256 | |
| - Layers: 6 | |
| - Attention heads: 16 | |
| - FFN size: 1152 | |
| - RoPE | |
| - RMSNorm | |
| - SwiGLU | |
| - Tied input/output embeddings | |
| ## Pretraining dataset | |
| `Plans11/Organized_PreTrain_1k_Context` | |
| ## Training | |
| Training is performed in 20,000-example sessions. | |
| A persistent SHA-256 example ledger records only examples whose | |
| training session has successfully completed. | |
| The model is initialized randomly and pretrained from scratch. | |