Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,140 @@
+# ⚡ MiniTransformer-91M: Custom LLM Built From Scratch
+
+[![PyTorch](https://img.shields.io/badge/PyTorch-2.11.0%2Bcu128-EE4C2C?logo=pytorch&logoColor=white)](https://pytorch.org/)
+[![CUDA](https://img.shields.io/badge/CUDA-12.8%20(Blackwell%20sm__120)-76B900?logo=nvidia&logoColor=white)](https://developer.nvidia.com/cuda-toolkit)
+[![Hardware](https://img.shields.io/badge/Hardware-NVIDIA%20GeForce%20RTX%205070-76B900)](https://www.nvidia.com/)
+[![Parameters](https://img.shields.io/badge/Parameters-91.2M-blue)](#model-architecture)
+[![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
+
+An end-to-end, production-grade Large Language Model built entirely from ground-up fundamentals in PyTorch — featuring a custom Byte-Pair Encoding (BPE) tokenizer, Rotary Position Embeddings (RoPE), SwiGLU feed-forward networks, FlashAttention-powered Scaled Dot-Product Attention (SDPA), KV-Cache inference, and Supervised Fine-Tuning (SFT) for developer code assistance.
+
+---
+
+## 🏗️ Architecture & Specifications
+
+| Hyperparameter | Specification | Architectural Rationale |
+| :--- | :--- | :--- |
+| **Total Parameters** | **91,245,312 (~91.2M)** | Right-sized for efficient single-GPU desktop training & high-throughput inference |
+| **Layers (`n_layers`)** | **12 Transformer Blocks** | Deep representation learning with pre-norm RMSNorm stability |
+| **Hidden Dim (`d_model`)** | **768** | Standard projection dimension matching GPT-2 scale |
+| **Attention Heads** | **12 Query / 12 Key-Value** | Multi-Head Self-Attention with FlashAttention SDPA kernel fusion |
+| **Feed-Forward (`d_ff`)** | **2,048 (SwiGLU)** | Swish-Gated Linear Units (LLaMA/Mistral style) for higher non-linear capacity |
+| **Context Window** | **256 tokens** | Sliding causal context with absolute relative distance preserved via RoPE |
+| **Vocabulary Size** | **8,192 tokens** | Custom Byte-Level BPE tokenizer trained on technical & modern literature corpus |
+| **Weight Tying** | **Enabled** | Embedding matrix shared with output language modeling head ($V \times D$) |
+| **Positional Encoding**| **RoPE (Rotary Embeddings)**| Eliminates learnable absolute position embeddings; enables extrapolation & cache alignment |
+| **Normalization** | **RMSNorm ($\epsilon = 10^{-5}$)** | Scaled root-mean-square normalization without mean-centering overhead |
+
+---
+
+## 🚀 Key Engineering Milestones & Innovations
+
+### 1. Rotary Position Embeddings (RoPE) & Exact KV-Cache Alignment
+Standard RoPE applies complex plane rotations to query and key vectors. In autoregressive KV-cache inference, feeding only the single newest token ($T=1$) naively rotates it at position $0$ instead of absolute step $S$.
+- **Resolved**: Tracked sequential cache index (`start_pos = past_kvs[0][0].shape[2]`) through every block.
+- **Verification**: Verified numerical logit parity between full recomputation and incremental KV-cache with a discrepancy $\le 1.2 \times 10^{-7}$.
+
+### 2. Native CUDA 12.8 & Blackwell RTX 5070 Optimization
+- **Hardware**: NVIDIA GeForce RTX 5070 (12GB VRAM, Compute Capability `sm_120`).
+- **Mixed Precision**: Native `torch.bfloat16` autocast prevents gradient underflow while saving 50% memory bandwidth.
+- **Selective Weight Decay**: 2D weight matrices (linear layers, projections) decayed at $\lambda = 0.1$, while 1D normalization gains ($\gamma$) and biases are exempt.
+
+### 3. Two-Stage Training Curriculum
+1. **Stage 1 — Modern Technical Pre-Training**:
+   - 2.66 Million tokens (101 high-density documents across Computer Science, Operating Systems, Algorithms, Physics, and Mathematics).
+   - Validation loss dropped from **5.5954 $\rightarrow$ 4.6697**.
+2. **Stage 2 — Code & Developer Supervised Fine-Tuning (SFT)**:
+   - 25,022 instruction-response pairs (CodeAlpaca 20k + Alpaca Dialogue).
+   - **Response-Only Loss Masking**: User prompts are masked with `ignore_index=-100` so gradients strictly optimize the assistant's solutions.
+   - Validation loss dropped from **3.0428 $\rightarrow$ 2.2727**.
+
+---
+
+## 💻 Interactive Interfaces
+
+### 🌐 1. Gradio Web Interface (`app.py`)
+Launch the browser UI with real-time typewriter streaming, parameter sliders, and markdown code syntax highlighting:
+
+```powershell
+.venv\Scripts\python.exe app.py
+```
+Open **`http://127.0.0.1:7860`** in your browser.
+
+**Features**:
+- Real-time token streaming powered by generator KV-cache.
+- Dynamic checkpoint switcher (`model_dev_best.pt`, `model_sft_best.pt`, `model_best.pt`).
+- Interactive sliders for Temperature, Top-p Nucleus cutoff, and Repetition Penalty.
+- Clickable 1-click prompt presets for coding and algorithms.
+
+### 📟 2. Terminal REPL (`chat.py`)
+Chat directly from the command line:
+
+```powershell
+.venv\Scripts\python.exe chat.py --model checkpoints/model_dev_best.pt --temperature 0.6 --penalty 1.15
+```
+
+Live commands inside chat:
+- `/temp 0.5` — Adjust sampling temperature
+- `/penalty 1.2` — Adjust repetition penalty
+- `/tokens 80` — Adjust max generated tokens
+- `/clear` — Clear terminal screen
+- `exit` — Quit
+
+---
+
+## 🛠️ Project Structure
+
+```text
+llm_from_scratch/
+├── model/
+│   ├── config.py               # ModelConfig dataclass & parameter calculation
+│   ├── transformer.py          # MiniTransformer, DecoderLayer, CausalSelfAttention
+│   ├── positional_encoding.py  # Rotary Position Embedding (RoPE) implementation
+│   ├── ffn.py                  # SwiGLU Feed-Forward Network
+│   └── normalization.py        # RMSNorm layer
+├── tokenizer/
+│   ├── bpe_tokenizer.py        # Byte-Level BPE Tokenizer wrapper
+│   └── train_tokenizer.py      # BPE training pipeline (8,192 vocab)
+├── training/
+│   ├── training_loop.py        # Pre-training loop with BF16 & cosine schedule
+│   ├── data_loader_setup.py    # Chunking & sliding window dataloader
+│   └── sft_dataset.py          # SFT dataset with response-only loss masking (-100)
+├── inference/
+│   ├── generate.py             # Top-k / Top-p KV-cache sampling
+│   └── generate_stream.py      # Streaming generator for web UI
+├── scripts/
+│   ├── download_modern_corpus.py # Scrapes 100+ technical articles & algorithms
+│   ├── prepare_dev_sft.py        # Downloads CodeAlpaca & blends SFT dataset
+│   └── download_alpaca.py        # Stanford Alpaca downloader
+├── checkpoints/
+│   ├── model_dev_best.pt       # Developer SFT Checkpoint (Val Loss: 2.2727)
+│   ├── model_sft_best.pt       # Conversational SFT Checkpoint
+│   └── model_best.pt           # Base Pre-Trained Model (Val Loss: 4.6697)
+├── app.py                      # Gradio Web Interface
+├── chat.py                     # Interactive Terminal REPL
+├── train.py                    # Pre-training execution script
+├── train_sft.py                # SFT execution script
+└── requirements.txt            # Environment specifications
+```
+
+---
+
+## 📊 Training Results Summary
+
+```text
+[Pre-Training]
+Corpus: 2,659,709 tokens (101 modern encyclopedic & algorithmic files)
+Vocabulary: 8,192 Byte-Level BPE tokens
+Context Length: 256 tokens | Batch Size: 16 | Hardware: RTX 5070 BF16
+Epoch 1/5: Val Loss = 5.5954
+Epoch 2/5: Val Loss = 5.0667
+Epoch 3/5: Val Loss = 4.7951
+Epoch 4/5: Val Loss = 4.6697 [BEST BASE MODEL]
+
+[Code & Developer SFT]
+Dataset: 25,022 pairs (CodeAlpaca 20k + Alpaca Assistant)
+Supervision: Response-Only Loss Masking (-100) | LR: 2.5e-5
+Epoch 1/3: Train Loss = 3.0428 | Val Loss = 2.5424
+Epoch 2/3: Train Loss = 2.2934 | Val Loss = 2.3226
+Epoch 3/3: Train Loss = 2.0401 | Val Loss = 2.2727 [BEST DEV MODEL]
+```