Update README.md

README.md

@@ -1,5 +1,6 @@
 ---
-language: en
+language:
+- en
 license: mit
 tags:
 - text-generation
@@ -8,87 +9,80 @@ tags:
 - recursive-language-model
 - mixture-of-recursion
 - adaptive-computation
-- rotary-embeddings
 - perplexity-routing
+- self-supervised-learning
 datasets:
 - Anthropic/hh-rlhf
-- OpenAssistant/oasst2
 - stingning/ultrachat
 metrics:
 - perplexity
 library_name: transformers
 pipeline_tag: text-generation
+base_model: gpt2
 ---
 
-# Recursive Language Model - 90M (Mixture of Recursion)
+# Recursive Language Model - 90M (Adaptive Computation)
 
-A transformer-based conversational language model with **Mixture of Recursion** architecture featuring adaptive recursive processing, perplexity-based dynamic routing, and rotary positional embeddings (RoPE) for enhanced dialogue generation.
+A novel 90M parameter conversational language model featuring **adaptive recursive computation** through perplexity-based dynamic routing. Achieves better performance than GPT-2 Small (117M) with 30% fewer parameters.
 
-## Model Description
+## 🏆 Key Achievement
 
-This model implements a novel **Mixture of Recursion** architecture that dynamically determines the optimal number of recursive refinement passes based on **sample perplexity** during training. Unlike standard transformers that process all inputs uniformly, this model intelligently allocates computational resources based on actual difficulty.
+**Perplexity: 17.45** - Outperforms GPT-2 Small (29 perplexity) despite being significantly smaller.
 
-### Key Innovations
+## 🔥 Innovation
 
-- 🧠 **Perplexity-Based Router**: Learns complexity from model's actual performance, not manual labels
-- 🔄 **Adaptive Recursion**: 1, 3, or 5 recursive transformer passes based on sample difficulty
-- 🌀 **Rotary Positional Embeddings (RoPE)**: Superior positional encoding with better length generalization
-- ⚡ **Dynamic Computation**: Efficient processing that adapts to input difficulty
-- 🎯 **Self-Supervised Learning**: Router trained on model's own perplexity signals
-- 📊 **Multi-Source Training**: Trained on diverse, high-quality conversational data from Anthropic, OpenAssistant, and UltraChat
+This model introduces a **self-supervised curriculum learning** approach where the model learns to allocate computation based on sample difficulty without any manual labeling.
 
-### Architecture Philosophy
+### Novel Architecture: Mixture of Recursion
 
-Traditional transformers apply the same computational depth to all inputs. This model recognizes that some sequences (simple greetings, common phrases) need minimal processing, while others (complex reasoning, technical explanations) benefit from deeper iterative refinement. The router learns to make this decision automatically by observing which samples the model finds difficult (high perplexity = more thinking needed).
+Instead of applying uniform computation to all inputs, this model features:
 
-## Quick Start
+- **Perplexity-Based Router**: Neural classifier that learns sample difficulty from the model's own confidence
+- **Adaptive Recursion**: Dynamically allocates 1, 3, or 5 recursive transformer passes based on input complexity
+- **Self-Supervised Learning**: No manual labels - the model learns what's "hard" from its own perplexity signals
 
-### Installation
-```bash
-pip install transformers torch
+### How It Works
+```
+High Perplexity (>50) → Model struggling → Use 5 recursion steps
+Medium Perplexity (20-50) → Moderate difficulty → Use 3 steps
+Low Perplexity (<20) → Model confident → Use 1 step (efficient!)
 ```
 
-### Basic Usage
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-import torch
+This enables **intelligent compute allocation** - simple inputs get fast processing, complex inputs get deeper reasoning.
 
-# Load model and tokenizer
-model = AutoModelForCausalLM.from_pretrained(
-    "Girinath11/recursive-language-model-90m",
-    trust_remote_code=True
-)
-tokenizer = AutoTokenizer.from_pretrained(
-    "Girinath11/recursive-language-model-90m"
-)
+## 📊 Performance
 
-# Move to GPU if available
-device = "cuda" if torch.cuda.is_available() else "cpu"
-model = model.to(device)
-model.eval()
+### Benchmark Comparison
 
-print("✅ Model loaded successfully!")
-print(f"Parameters: {model.num_parameters():,}")
+| Model | Parameters | Perplexity | Notes |
+|-------|-----------|------------|-------|
+| **This Model** | **90.7M** | **17.45** | **Novel adaptive architecture** |
+| GPT-2 Small | 117M | ~29 | Baseline comparison |
+| GPT-2 Medium | 345M | ~22 | 3.8× larger |
+| Random Baseline | - | ~50,000 | Theoretical worst |
 
-# Generate text (conversational format)
-prompt = "<|user|>\nWhat is artificial intelligence?\n<|assistant|>\n"
-inputs = tokenizer(prompt, return_tensors="pt").to(device)
-
-outputs = model.generate(
-    **inputs,
-    max_new_tokens=100,
-    temperature=0.8,
-    top_p=0.9,
-    do_sample=True,
-    pad_token_id=tokenizer.eos_token_id
-)
+### Training Metrics
+```
+📈 Training Progression:
+Epoch 1: 35.39 perplexity
+Epoch 2: 20.27 perplexity (43% improvement)
+Epoch 3: 17.45 perplexity (51% total improvement)
 
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+📉 Loss Reduction:
+Start: 4.50 → Final: 2.86 (36% reduction)
 ```
 
-## Model Architecture
+### Performance Highlights
+
+✅ **17.45 perplexity** on validation set (37K samples)  
+✅ **Better than GPT-2 Small** with 30% fewer parameters  
+✅ **Efficient inference** - adaptive computation saves resources  
+✅ **84 minutes training** on single T4 GPU  
+✅ **Novel architecture** - not just fine-tuning  
+
+## 🎯 Model Architecture
 
-### Detailed Architecture Specifications
+### Specifications
 
 | Component | Configuration |
 |-----------|--------------|
@@ -101,68 +95,165 @@ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 | **FFN Intermediate Size** | 2240 |
 | **Max Sequence Length** | 512 tokens |
 | **Positional Encoding** | Rotary Positional Embeddings (RoPE) |
-| **Dropout Rate** | 0.1 (both hidden and attention) |
-| **Layer Normalization** | eps=1e-5 |
+| **Dropout Rate** | 0.1 (hidden & attention) |
 
 ### Recursion Configuration
 
-| Complexity Class | Perplexity Range | Recursion Steps | Use Case |
-|-----------------|------------------|-----------------|----------|
-| **Simple** | < 20 | 1 step | Easy samples model is confident about |
-| **Medium** | 20-50 | 3 steps | Moderate difficulty samples |
-| **Complex** | > 50 | 5 steps | Hard samples requiring deep reasoning |
-
-**Note:** Complexity labels are automatically generated during training based on the model's own perplexity on each sample, enabling self-supervised curriculum learning.
+| Complexity Level | Perplexity Range | Steps | Use Case |
+|-----------------|------------------|-------|----------|
+| **Simple** | < 20 | 1 step | Model is confident |
+| **Medium** | 20-50 | 3 steps | Moderate difficulty |
+| **Complex** | > 50 | 5 steps | Model struggling, needs deep reasoning |
 
 ### Architecture Components
 
-1. **Embedding Layer** (~28.1M params)
-   - Token embeddings (50,259 × 560)
+1. **Token Embedding Layer** (50,259 × 560)
    - Special tokens: `<|user|>`, `<|assistant|>`, `<|endoftext|>`
-   - Padding token handling (ID: 50256)
 
-2. **Base Transformer Stack** (8 layers, ~30.5M params)
+2. **Base Transformer Stack** (8 layers)
    - Multi-head self-attention with RoPE
    - Feed-forward networks (560 → 2240 → 560)
    - Pre-normalization with LayerNorm
    - Residual connections
-   - Causal masking for autoregressive generation
 
 3. **Perplexity-Based Router** (~0.4M params)
-   - Attention-weighted pooling over sequence
+   - Attention-weighted sequence pooling
    - 2-layer MLP classifier (560 → 280 → 3)
-   - Trained on pseudo-labels derived from sample perplexity
+   - Trained on pseudo-labels from sample perplexity
    - Outputs: complexity class (0=simple, 1=medium, 2=complex)
 
 4. **Recursive Refinement Layer** (~3.8M params)
-   - Additional transformer block (reused 1-5 times)
+   - Transformer block applied 1-5 times adaptively
    - Same architecture as base layers
-   - Applied iteratively based on router decision
+   - Reused weights for parameter efficiency
 
-5. **Output Projection Head**
+5. **Output Projection**
+   - Final LayerNorm
    - Linear layer (560 → 50,259)
-   - Final LayerNorm before projection
 
-### Rotary Positional Embeddings
+## 🚀 Quick Start
 
-Uses RoPE instead of learned positional embeddings for:
-- Better extrapolation to longer sequences
-- Relative position encoding
-- Improved performance on positional tasks
-- Base frequency: 10,000
+### Installation
+```bash
+pip install transformers torch
+```
 
-## Training Details
+### Basic Usage
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
 
-### Training Dataset
+# Load model and tokenizer
+model = AutoModelForCausalLM.from_pretrained(
+    "Girinath11/recursive-language-model-90m",
+    trust_remote_code=True
+)
+tokenizer = AutoTokenizer.from_pretrained(
+    "Girinath11/recursive-language-model-90m"
+)
+
+# Move to GPU if available
+device = "cuda" if torch.cuda.is_available() else "cpu"
+model = model.to(device)
+model.eval()
 
-**Total Training Samples: 100,000** (premium conversational data, streaming mode)
+print(f"✅ Model loaded on {device}")
+print(f"📊 Parameters: {model.num_parameters():,}")
+```
 
-| Dataset | Source | Percentage | Samples | Description |
-|---------|--------|-----------|---------|-------------|
-| **Anthropic HH-RLHF** | Anthropic | 30% | ~30,000 | Claude-style helpful, harmless responses |
-| **OpenAssistant** | LAION | 40% | ~40,000 | Human feedback-curated conversations |
-| **UltraChat** | Tsinghua | 30% | ~30,000 | GPT-4 generated multi-turn dialogues |
-| **Validation** | Mixed | - | 1,000 | Held-out samples from all sources |
+### Conversational Format
+```python
+# Model expects this format
+prompt = """<|user|>
+What is machine learning?
+<|assistant|>
+"""
+
+inputs = tokenizer(prompt, return_tensors="pt").to(device)
+
+# Generate response
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=100,
+        temperature=0.7,
+        top_p=0.9,
+        do_sample=True,
+        pad_token_id=tokenizer.eos_token_id
+    )
+
+response = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(response)
+```
+
+### Batch Generation
+```python
+prompts = [
+    "<|user|>\nExplain quantum computing simply.\n<|assistant|>\n",
+    "<|user|>\nWrite a haiku about AI.\n<|assistant|>\n",
+    "<|user|>\nHow do neural networks work?\n<|assistant|>\n"
+]
+
+# Tokenize all prompts
+inputs = tokenizer(prompts, return_tensors="pt", padding=True).to(device)
+
+# Generate for all at once
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=80,
+        temperature=0.8,
+        top_p=0.9,
+        do_sample=True,
+        pad_token_id=tokenizer.eos_token_id
+    )
+
+# Decode all outputs
+for i, output in enumerate(outputs):
+    text = tokenizer.decode(output, skip_special_tokens=True)
+    print(f"\nPrompt {i+1}:\n{text}\n{'-'*60}")
+```
+
+### Generation Parameters
+```python
+# Creative writing (high temperature)
+creative_output = model.generate(
+    **inputs,
+    max_new_tokens=150,
+    temperature=1.0,      # More random/creative
+    top_p=0.95,
+    top_k=50,
+    do_sample=True
+)
+
+# Focused/deterministic (low temperature)
+focused_output = model.generate(
+    **inputs,
+    max_new_tokens=100,
+    temperature=0.5,      # More focused
+    top_p=0.9,
+    do_sample=True
+)
+
+# Greedy decoding (most likely tokens)
+greedy_output = model.generate(
+    **inputs,
+    max_new_tokens=50,
+    do_sample=False       # Deterministic
+)
+```
+
+## 📚 Training Details
+
+### Dataset
+
+**Total Training Samples: 37,119** (high-quality conversational data)
+
+| Dataset | Source | Samples | Percentage | Description |
+|---------|--------|---------|------------|-------------|
+| **Anthropic HH-RLHF** | Anthropic | 30,000 | 81% | Claude-style helpful & harmless responses |
+| **UltraChat** | Tsinghua | 7,119 | 19% | GPT-4 generated multi-turn dialogues |
+| **Validation** | Mixed | 2,000 | - | Held-out samples for evaluation |
 
 **Data Format:**
 ```
@@ -173,354 +264,282 @@ Assistant response here
 <|endoftext|>
 ```
 
-**Filtering Criteria:**
-- Minimum sequence length: 50 tokens
-- Maximum sequence length: 512 tokens
-- Streaming mode: Memory-efficient loading
+**Quality Control:**
+- Minimum length: 80 characters
+- Maximum length: 6,000 characters
+- Minimum word count: 15 words
+- Alphanumeric ratio: >65%
+- Required: Proper punctuation
+- Validation: Strict conversation structure
 
 ### Training Configuration
 ```yaml
 Hardware:
   GPU: NVIDIA Tesla T4 (15.64 GB)
   Platform: Kaggle
-  Framework: PyTorch with CUDA
+  Framework: PyTorch + Transformers
+  Mixed Precision: FP16 (AMP)
 
 Hyperparameters:
   Batch Size: 8
   Gradient Accumulation: 8
   Effective Batch Size: 64
-  Learning Rate: 2.5e-4
+  Max Sequence Length: 512
+  Learning Rate: 3e-4
   Optimizer: AdamW
   Weight Decay: 0.01
-  LR Schedule: Cosine Annealing
-  Max Gradient Norm: 1.0
+  LR Schedule: OneCycleLR
+  Warmup: 10% of total steps
+  Gradient Clipping: 1.0
   Total Epochs: 3
-  Max Sequence Length: 512 tokens
+  Total Steps: 13,917
 
 Loss Function:
-  Language Modeling: CrossEntropyLoss (ignore_index=-100)
+  Language Modeling: CrossEntropyLoss
   Router Loss: CrossEntropyLoss (weight: 0.1)
-  Total Loss: LM Loss + 0.1 × Router Loss
+  Total: LM Loss + 0.1 × Router Loss
 
 Regularization:
   Hidden Dropout: 0.1
   Attention Dropout: 0.1
-  Memory Management: Aggressive cleanup every 50 steps
 ```
 
 ### Training Schedule
 
-- **Total Training Steps:** 37,500 (12,500 per epoch)
-- **Steps per Epoch:** 12,500
-- **Evaluation Frequency:** After each epoch (50 batches)
-- **Checkpoint Saving:** After each epoch + best model
-
-### Training Time
-
-- **Total Duration:** ~9 hours 17 minutes
-- **Epoch 1:** 3 hours 4 minutes
-- **Epoch 2:** 3 hours 8 minutes
-- **Epoch 3:** 3 hours 4 minutes
-- **Training Speed:** ~1.10-1.13 it/s
+- **Total Training Time:** 84 minutes (~1.4 hours)
+- **Steps per Epoch:** 4,639
+- **Training Speed:** ~2.7 it/s
+- **GPU Utilization:** ~85-90%
+- **Memory Usage:** ~8-9 GB (peak)
 
 ### Training Progression
 
-| Epoch | Training Loss | Eval Loss | Eval Perplexity | Status |
-|-------|--------------|-----------|-----------------|---------|
-| **1** | 3.6207 | 3.3350 | **28.08** | 🔥 **BEST!** |
-| **2** | 3.9455 | 3.7622 | 43.04 | - |
-| **3** | 3.0986 | 4.3018 | 73.84 | - |
-
-**Best Model:** Epoch 1 (Perplexity: 28.08)  
-**Loss Improvement:** Strong initial learning, later epochs showed overfitting
+| Epoch | Train Loss | Eval Loss | Perplexity | Status |
+|-------|-----------|-----------|------------|---------|
+| **1** | 4.4979 | 3.5665 | 35.39 | Initial learning |
+| **2** | 3.2960 | 3.0091 | 20.27 | Rapid improvement |
+| **3** | 2.8572 | 2.8595 | **17.45** | 🔥 **BEST!** |
 
-## Performance Metrics
+**Key Observations:**
+- Loss decreased steadily across all epochs
+- No overfitting (train/eval losses converged)
+- 51% perplexity improvement from Epoch 1 to 3
+- Stable training with consistent iteration speed
 
-### Final Evaluation Results
-```
-📊MODEL METRICS:
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-✓ Evaluation Loss:      3.34
-✓ Perplexity:          28.08
-✓ Training Loss:       3.62
-✓ Total Samples Seen:  100,000
-✓ Training Time:       ~9.2 hours
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-```
-
-### Generation Quality
-
-**Perplexity: 28.08** indicates **excellent quality** for a 90M parameter model:
-
-- ✅ Generates coherent and contextually appropriate responses
-- ✅ Maintains conversation context over multiple turns
-- ✅ Produces diverse and natural-sounding outputs
-- ✅ Handles various conversational styles and topics
-- ✅ Suitable for chatbots, dialogue systems, and creative applications
-- ✅ Comparable to GPT-2 Small (117M) in quality
-- ⚠️ May show repetition in very long conversations (500+ tokens)
-- ⚠️ Limited factual knowledge (trained on 100K samples)
-- ⚠️ Context window limited to 512 tokens
-
-### Memory Requirements
+## 💡 Technical Innovation
 
-| Mode | RAM/VRAM | Disk Space |
-|------|----------|------------|
-| **Model Weights** | - | 346 MB |
-| **CPU Inference** | ~1.2 GB | - |
-| **GPU Inference (FP32)** | ~2.5 GB | - |
-| **GPU Inference (FP16)** | ~1.5 GB | - |
-| **GPU Training (batch=8)** | ~8-10 GB | - |
+### Perplexity-Based Routing (Novel Contribution)
 
-## Advanced Usage
+Traditional transformers apply the same computational depth to all inputs. This model recognizes that:
+- **Simple inputs** (greetings, common phrases) need minimal processing
+- **Complex inputs** (reasoning, technical content) benefit from deeper iterative refinement
 
-### Conversational Generation
+**Key Innovation:** Instead of manual labeling, the model learns complexity from its own performance:
 ```python
-import torch
-from transformers import AutoModelForCausalLM, AutoTokenizer
+# During training:
+sample_perplexity = exp(sample_loss)
 
-model = AutoModelForCausalLM.from_pretrained(
-    "Girinath11/recursive-language-model-90m",
-    trust_remote_code=True
-)
-tokenizer = AutoTokenizer.from_pretrained(
-    "Girinath11/recursive-language-model-90m"
-)
+if sample_perplexity < 20:
+    label = 0  # Simple - model is confident
+elif sample_perplexity < 50:
+    label = 1  # Medium - model is uncertain  
+else:
+    label = 2  # Complex - model struggling
 
-device = "cuda" if torch.cuda.is_available() else "cpu"
-model = model.to(device)
-model.eval()
+# Router learns to predict this from input features
+router_loss = CrossEntropyLoss(router_logits, label)
+```
 
-# Multi-turn conversation
-conversation = """<|user|>
-Hello! Can you help me understand machine learning?
-<|assistant|>
-"""
+**Benefits:**
+- ✅ **No manual labeling required** - fully self-supervised
+- ✅ **Adapts as model learns** - curriculum learning
+- ✅ **Objective measure** - based on actual model performance
+- ✅ **Efficient computation** - allocates resources intelligently
 
-inputs = tokenizer(conversation, return_tensors="pt").to(device)
-outputs = model.generate(
-    **inputs,
-    max_new_tokens=150,
-    temperature=0.8,
-    top_p=0.9,
-    do_sample=True,
-    pad_token_id=tokenizer.eos_token_id
-)
+### Rotary Positional Embeddings (RoPE)
 
-print(tokenizer.decode(outputs[0]))
-```
+Uses RoPE instead of learned positional embeddings:
+- Better extrapolation to longer sequences
+- Relative position awareness
+- Improved performance on positional tasks
+- No learned position parameters
 
-### Temperature Control
-```python
-# Creative/diverse responses (high temperature)
-creative = model.generate(
-    **inputs,
-    max_new_tokens=100,
-    temperature=1.0,      # More random
-    top_p=0.95,
-    do_sample=True
-)
+### Self-Supervised Curriculum Learning
 
-# Focused/deterministic responses (low temperature)
-focused = model.generate(
-    **inputs,
-    max_new_tokens=100,
-    temperature=0.5,      # More focused
-    top_p=0.9,
-    do_sample=True
-)
+The model implements automatic curriculum learning:
+1. **Early training**: Most samples are "complex" (high perplexity)
+2. **Mid training**: Distribution shifts as model learns
+3. **Late training**: More samples become "simple" (low perplexity)
 
-# Greedy decoding (most likely)
-greedy = model.generate(
-    **inputs,
-    max_new_tokens=50,
-    do_sample=False       # Deterministic
-)
-```
+This creates a natural curriculum without human intervention.
 
-## Technical Architecture
+## 🎯 Use Cases
 
-### Model Structure
-```
-Input Text
-    ↓
-[Token Embedding Layer] (50,259 × 560)
-    ↓
-[8× Base Transformer Blocks]
-    ├─ Multi-Head Attention (8 heads, RoPE)
-    ├─ Feed-Forward Network (560 → 2240 → 560)
-    └─ LayerNorm + Residual Connections
-    ↓
-[Perplexity-Based Router]
-    ├─ Attention-Weighted Pooling
-    ├─ MLP Classifier (560 → 280 �� 3)
-    └─ Output: Complexity from Perplexity
-    ↓
-[Adaptive Recursive Refinement]
-    ├─ Simple (PPL<20):   1× Recursion Layer
-    ├─ Medium (PPL 20-50): 3× Recursion Layer
-    └─ Complex (PPL>50):   5× Recursion Layer
-    ↓
-[Final LayerNorm]
-    ↓
-[LM Head] (560 → 50,259)
-    ↓
-Output Tokens
-```
+### ✅ Recommended
 
-### Perplexity-Based Routing Innovation
+- **Educational demos** - Teaching language model concepts
+- **Research** - Experimenting with adaptive computation
+- **Prototyping** - Testing conversational AI applications
+- **Learning** - Understanding transformer architectures
+- **Creative writing assistance** - With human review
+- **Code explanation** - Basic programming concepts
 
-**Key Idea:** Instead of manually labeling samples as "simple" or "complex", the model learns from its own difficulty:
-```python
-# During training:
-1. Model predicts next tokens → calculates loss
-2. High loss = High perplexity = Hard sample → Router learns to use 5 steps
-3. Low loss = Low perplexity = Easy sample → Router learns to use 1 step
+### ⚠️ Not Recommended
 
-# Pseudo-label generation:
-if sample_perplexity < 20:
-    label = 0  # Simple: model is confident
-elif sample_perplexity < 50:
-    label = 1  # Medium: model is uncertain
-else:
-    label = 2  # Complex: model is struggling
-```
-
-**Benefits:**
-- ✅ No manual labeling required
-- ✅ Adapts as model learns (curriculum learning)
-- ✅ Objective measure of difficulty
-- ✅ Efficient computation allocation
+- Production chatbots without human oversight
+- Medical, legal, or financial advice
+- Generating authoritative content without verification
+- Automated content moderation
+- Safety-critical systems
+- High-stakes decision making
 
-## Limitations & Considerations
+## ⚠️ Limitations
 
 ### Technical Limitations
 
-1. **Context Window**: 512 token maximum
-2. **Model Size**: 90M parameters - limited capacity vs billion-scale models
-3. **Training Data**: 100K samples (streaming mode)
-4. **Single Language**: Primarily English
-5. **Domain Coverage**: Conversational data only
-6. **Reasoning**: Basic dialogue, limited multi-step reasoning
+1. **Context Window**: 512 tokens maximum (vs 2048+ for modern models)
+2. **Training Data**: 37K samples - relatively small dataset
+3. **Single Language**: Primarily English
+4. **Generation Quality**: Good but not perfect - may require post-processing
+5. **Knowledge Cutoff**: Training data up to early 2024
 
 ### Known Issues
 
-- **Repetition**: May repeat phrases in very long generations (500+ tokens)
-- **Factual Knowledge**: Limited knowledge base (100K training samples)
-- **Consistency**: May occasionally lose context in very long conversations
-- **Overfitting**: Epochs 2-3 showed increased eval perplexity
-- **Early Stopping**: Best model from Epoch 1
+- **Repetition**: May repeat phrases in very long generations (>200 tokens)
+- **Factual Accuracy**: Small knowledge base, may generate plausible but incorrect information
+- **Context Retention**: May lose track of earlier context in long conversations
+- **Domain Specificity**: Limited knowledge in highly specialized fields
 
-## Ethical Considerations
+### Generation Characteristics
+
+- Occasionally incomplete sentences at sequence boundaries
+- May generate generic filler phrases
+- Temperature tuning recommended for optimal quality
+- Best results with clear, specific prompts
+
+## 🔬 Ethical Considerations
 
 ### Bias & Fairness
 
 This model may exhibit biases from training data:
 
 **Potential Biases:**
-- Conversational style: Reflects training data distributions
-- Language: English-centric (GPT-2 tokenizer)
-- Content: Biases from Anthropic HH, OpenAssistant, UltraChat
+- Geographic: Western/English content overrepresentation
+- Demographic: Biases from web text and conversational data
+- Temporal: Reflects content patterns up to 2024
+- Topic: Conversational data may skew certain perspectives
 
 **Mitigation:**
-- Diverse multi-source training data
-- Best practices from source datasets (Anthropic HH-RLHF includes harmlessness training)
+- Diverse training sources (Anthropic HH, UltraChat)
+- Quality filtering applied
+- Users should validate outputs for fairness-critical applications
 
 ### Responsible Use
 
-**✅ Recommended:**
-- Educational chatbot demos
-- Research on adaptive computation
-- Creative writing assistance
-- Conversational AI prototyping
-- Learning about language models
+**Environmental Impact:**
+- Training: ~84 minutes on T4 GPU
+- Estimated CO₂: ~0.09 kg (single training run)
+- Relatively low impact due to efficient training
 
-**❌ Not Recommended:**
-- Production chatbots without human oversight
-- Medical, legal, or financial advice
-- Generating authoritative factual content
-- Automated content moderation
-- Safety-critical systems
-
-## Comparison with Similar Models
-
-| Model | Params | Perplexity | Architecture | Training Data |
-|-------|--------|------------|--------------|---------------|
-| **This Model** | **90M** | **28.08** | **Mixture of Recursion** | **100K conversations** |
-| GPT-2 Small | 117M | ~29 | Standard Transformer | WebText (40GB) |
-| DistilGPT-2 | 82M | ~35 | Distilled GPT-2 | Distillation |
-| DialoGPT Small | 117M | ~30 | GPT-2 + Reddit | Reddit comments |
-
-**Advantages:**
-- ✅ Competitive perplexity with fewer parameters
-- ✅ Novel adaptive architecture
-- ✅ Conversational training data
-- ✅ Perplexity-based intelligent routing
+## 📖 Model Card
 
-**Trade-offs:**
-- ⚠️ Smaller training dataset
-- ⚠️ Limited context window (512 vs 1024+)
-- ⚠️ Less knowledge breadth
+### Model Details
 
-## Model Card
+- **Developed by:** Girinath11
+- **Model type:** Causal Language Model with Adaptive Recursion
+- **Language:** English
+- **License:** MIT
+- **Base Model:** GPT-2 tokenizer
+- **Training Date:** March 2026
+- **Framework:** PyTorch + Transformers
 
 ### Intended Use
 
 **Primary Use Cases:**
-- 📚 **Education**: Teaching adaptive computation
-- 🔬 **Research**: Experimenting with mixture of recursion
-- 🤖 **Chatbots**: Dialogue system prototyping
-- 💡 **Learning**: Understanding perplexity-based routing
-
-### Training Information
+- Research on adaptive computation
+- Educational demonstrations
+- Conversational AI prototyping
+- Understanding mixture-of-experts concepts
+- Curriculum learning research
 
-- **Trained by:** Girinath11
-- **Training date:** February 2026
-- **Training platform:** Kaggle (T4 GPU)
-- **Training duration:** ~9.2 hours
-- **Framework:** PyTorch + Transformers
+**Out-of-Scope:**
+- Production deployment without fine-tuning
+- High-stakes decision making
+- Content requiring domain expertise
+- Real-time applications requiring <100ms latency
 
 ### Evaluation Methodology
 
 **Metrics:**
-- Primary: Perplexity on validation set (1,000 samples)
-- Evaluated after each epoch (50 validation batches)
-- Best model selection: Minimum perplexity
+- Primary: Perplexity on held-out validation set
+- Secondary: Training loss, generation quality
 
 **Evaluation Data:**
-- Mixed samples from all three training sources
+- 2,000 samples from Anthropic HH & UltraChat
 - Same preprocessing as training
 - No overlap with training data
 
-## Citation
+## 📁 Repository Structure
+```
+recursive-language-model-90m/
+├── config.json                    # Model configuration
+├── model.safetensors             # Model weights (363 MB)
+├── tokenizer.json                # Tokenizer vocabulary
+├── tokenizer_config.json         # Tokenizer settings
+├── special_tokens_map.json       # Special token mappings
+├── model_info.json               # Training metadata
+├── mixture_of_recursion.py       # Model architecture code
+└── README.md                     # This file
+```
+
+## 🔗 Links
+
+- **Model Card:** [Hugging Face](https://huggingface.co/Girinath11/recursive-language-model-90m)
+- **GitHub:** [Repository](https://github.com/Giri530/recursive-language-model-90m)
+
+## 📄 Citation
+
+If you use this model in your research, please cite:
 ```bibtex
 @misc{recursive-lm-90m-2026,
   author = {Girinath11},
-  title = {Recursive Language Model with Perplexity-Based Routing},
+  title = {Recursive Language Model with Perplexity-Based Dynamic Routing},
   year = {2026},
   publisher = {Hugging Face},
-  howpublished = {\url{https://huggingface.co/Girinath11/recursive-language-model-90m}}
+  howpublished = {\url{https://huggingface.co/Girinath11/recursive-language-model-90m}},
+  note = {90M parameter model with adaptive computation achieving 17.45 perplexity}
 }
 ```
 
-## Acknowledgments
+## 🙏 Acknowledgments
 
 - **Anthropic** for HH-RLHF dataset
-- **LAION** for OpenAssistant dataset
 - **Tsinghua University** for UltraChat dataset
 - **Hugging Face** for Transformers library
 - **Kaggle** for free GPU access
+- **OpenAI** for GPT-2 tokenizer
 
-## License
+## 📝 Version History
 
-MIT License - Free to use with attribution
+### v1.0 (March 2026)
+- Initial release
+- 90.7M parameters
+- Perplexity: 17.45
+- Trained on 37K samples
+- 3 epochs, 84 minutes training time
+
+## 📧 Contact
 
-## Contact
+For questions, issues, or collaboration:
+- **GitHub Issues:** [Report a bug](https://github.com/Giri530/recursive-language-model-90m/issues)
+- **Hugging Face Discussions:** [Ask a question](https://huggingface.co/Girinath11/recursive-language-model-90m/discussions)
 
-For questions or issues, please open an issue on the model repository.
+## 📜 License
+
+MIT License - Free to use with attribution
 
 ---
 
-**Model Version:** 1.0  
-**Last Updated:** February 2026  
-**Status:** Production-ready for research and prototyping
+**Model Status:** ✅ Production Ready for Research & Prototyping
+
+**Last Updated:** March 2, 2026