Create README.md

README.md

@@ -0,0 +1,52 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- custom-architecture
+- rope
+- gqa
+- swiglu
+- rmsnorm
+- safetensors
+---
+
+# FastLLM (150M) — Modern Causal Language Model
+
+**FastLLM** is a ~150M parameter, decoder-only causal language model built completely from scratch in PyTorch and fully integrated with Hugging Face `transformers`. It incorporates state-of-the-art LLM architectural choices—**Grouped-Query Attention (GQA)**, **SwiGLU MLPs**, **RMSNorm**, and **Rotary Position Embeddings (RoPE)**—and natively saves weights in the zero-copy **Safetensors** format.
+
+---
+
+## Model Details
+
+* **Developed by:** devoppro
+* **Model Type:** Decoder-only Causal Language Model
+* **Architecture:** Custom Transformer (`ModernLLMForCausalLM`)
+* **Parameter Count:** ~150,000,000 (150M)
+* **Tokenizer:** Qwen 2.5 BPE Vocabulary (`vocab_size`: 151,936)
+* **Precision:** Mixed Precision (`FP16`)
+* **Storage Format:** `.safetensors`
+* **Repository:** `devoppro/FastLLM`
+
+---
+
+## Architectural Specifications
+
+| Parameter | Configuration |
+| :--- | :--- |
+| **Hidden Size ($d_{\text{model}}$)** | 768 |
+| **Intermediate Size (SwiGLU)** | 2048 |
+| **Hidden Layers** | 12 |
+| **Query Heads** | 12 |
+| **Key/Value Heads (GQA)** | 4 (3:1 Query-to-KV ratio) |
+| **Max Context Length** | 2048 tokens |
+| **Normalization** | RMSNorm ($\epsilon = 10^{-6}$) |
+| **Positional Embedding** | Rotary Embeddings (RoPE, $\theta = 1000000.0$) |
+
+---
+
+## Training Data Mixture
+
+The model was pre-trained using dynamic stream interleaving across four high-quality datasets: