Upload README.md with huggingface_hub

README.md

@@ -8,6 +8,7 @@ tags:
 - tiny-stories
 - adafactor
 - bitlinear
+- adamw
 license_name: mit
 license_link: https://opensource.org/licenses/MIT
 ---
@@ -18,7 +19,7 @@ license_link: https://opensource.org/licenses/MIT
 
 **Experimental implementation of ternary (1.58-bit) quantization for training small language models on extremely limited hardware.**
 
-**Status: EXPERIMENTAL - Not Production Ready**
+**Status: EXPERIMENTAL - Research in Progress**
 
 ---
 
@@ -50,67 +51,183 @@ This project explores whether **ternary quantization** (weights constrained to -
 
 ---
 
-## Checkpoints Included
+## Research Timeline
 
-### 1. checkpoint-step5000-experiment.pt
+### Phase 1: Initial Experiments (August 2026)
 
-| Attribute | Value |
-|-----------|-------|
+Early experiments with Adafactor optimizer:
+
+| Attempt | Config | Result | Issue |
+|---------|--------|--------|-------|
+| 5k steps | Beta1=None | COLLAPSED | Model fell into repetitive patterns |
+| 50k steps | Beta1=None | COLLAPSED | Entropy dropped to 8.2% |
+| 50k steps | Beta1=0.9 | HEALTHY | Learning too slow, val loss 7.56 |
+
+**Key Finding:** Beta1=None caused collapse. Beta1=0.9 too conservative.
+
+---
+
+### Phase 2: Root Cause Investigation (September 2026)
+
+**September 26, 2026:** Critical discovery - Adafactor implementation bug identified.
+
+| Issue | Details |
+|-------|---------|
+| Bug location | Original Adafactor implementation |
+| Bug type | Variance computed but not applied to updates |
+| Impact | Optimizer behaved like SGD with momentum |
+| Evidence | Lines 152, 170 in `src/training/adafactor.py` |
+
+**Verification:** Original optimizer state matched broken implementation, not canonical Adafactor.
+
+---
+
+### Phase 3: Working Models (September 2026)
+
+**After identifying the bug, switched to AdamW optimizer:**
+
+#### 1. FP32 + AdamW Baseline (August 19, 2026)
+
+| Metric | Value |
+|--------|-------|
+| Optimizer | AdamW |
 | Steps | 5,000 |
-| Final Loss | 8.45 |
-| Entropy | Not monitored |
-| Status | SUCCESS (pipeline verified) |
-| Description | First test training, proved architecture works |
-| Duration | ~4 hours |
+| Val Loss | **1.69** |
+| Perplexity | **5.44** |
+| Generation | ✅ Coherent stories |
+| Status | **SUCCESS - Baseline established** |
 
-**Purpose:** Verified that training pipeline works correctly.
+**File:** `checkpoint-fp32-adamw-working.pt`
 
 ---
 
-### 2. checkpoint-step50000-failed-collapsed.pt
+#### 2. BitLinear + AdamW (September 10-11, 2026)
 
-| Attribute | Value |
-|-----------|-------|
-| Steps | 50,000 |
-| Final Loss | 3.16 |
-| Final Entropy | 8.2% (COLLAPSED) |
-| Top Token | "time" at 94% |
-| Status | FAILED (collapsed) |
-| Description | Beta1=None caused model collapse |
-| Duration | 79 hours |
-| Root Cause | Missing momentum (beta1=None) |
+| Metric | Value |
+|--------|-------|
+| Optimizer | AdamW |
+| Steps | 9,500 |
+| Val Loss | **1.92** |
+| Perplexity | **6.81** |
+| Generation | ✅ Coherent stories |
+| Quantization | Ternary (1.58-bit) |
+| Status | **SUCCESS - Ternary quantization works** |
 
-**Warning:** This checkpoint is COLLAPSED. Model outputs repetitive text like "time time time...". Use only as a learning example of what NOT to do.
+**File:** `checkpoint-bitlinear-adamw-working.pt`
 
 ---
 
-### 3. checkpoint-step50000-experiment-gibberish.pt
+### Phase 4: Canonical Adafactor Investigation (October 15, 2026)
 
-| Attribute | Value |
-|-----------|-------|
-| Steps | 50,000 |
-| Final Loss | 7.56 |
-| Final Entropy | 96.2% (HEALTHY) |
-| Top Token | "the" at 0.56% |
-| Status | HEALTHY but NOT USEFUL |
-| Description | Beta1=0.9 prevented collapse but learning too slow |
-| Duration | 83 hours |
-| Issue | Beta1=0.9 too conservative |
+Implemented and tested canonical Adafactor (matching PyTorch reference):
+
+| Metric | Value |
+|--------|-------|
+| Implementation | Verified against PyTorch (0.025% accuracy) |
+| Val Loss | 4.82 |
+| Perplexity | 124.10 |
+| Optimizer State Memory | 0.16 MB (316x less than AdamW) |
+| Observed Performance | 2.0x gap vs AdamW on tested configurations |
+
+**Note:** Performance gap observed, but configurations differed (weight decay, momentum). Investigation ongoing.
+
+---
 
-**Note:** This checkpoint is HEALTHY (entropy normal) but produces gibberish. Model learned slowly due to overly conservative optimizer settings.
+## Key Findings
+
+### 1. Ternary Quantization is VIABLE
+
+| Evidence | Result |
+|----------|--------|
+| BitLinear + AdamW | Val loss 1.92 (vs FP32 1.69) |
+| Performance gap | 13% (acceptable for 85% memory reduction) |
+| Generation quality | ✅ Coherent stories |
+
+**Conclusion:** Ternary quantization works with proper optimizer (AdamW).
+
+---
+
+### 2. Optimizer Choice is CRITICAL
+
+| Optimizer | Result | Reason |
+|-----------|--------|--------|
+| Adafactor (buggy) | ❌ FAILED | Variance not applied |
+| Adafactor (canonical) | ⚠️ 2x gap | Configuration differences |
+| **AdamW** | ✅ **WORKS** | Proven convergence |
+
+---
+
+### 3. Beta1 Trade-off in Adafactor
+
+| Beta1 Value | Learning Speed | Stability | Result |
+|-------------|---------------|-----------|--------|
+| None (0.0) | FAST | UNSTABLE | COLLAPSED |
+| 0.9 | SLOW | STABLE | HEALTHY but slow learning |
+
+---
+
+### 4. Memory vs Performance Trade-off
+
+| Optimizer | Optimizer State | Performance | Trade-off |
+|-----------|----------------|-------------|-----------|
+| AdamW | 51.68 MB | ✅ Best (val loss 1.69-1.92) | More memory |
+| Canonical Adafactor | 0.16 MB | ⚠️ Gap (val loss 4.82) | 316x less memory |
+
+**Use case dependent:** AdamW for quality, Adafactor for memory-constrained scenarios.
+
+---
+
+## Checkpoints Included
+
+### Working Models (Upload as evidence)
+
+| File | Type | Val Loss | Status | Description |
+|------|------|----------|--------|-------------|
+| `checkpoint-fp32-adamw-working.pt` | FP32 | **1.69** | ✅ WORKING | AdamW baseline, coherent generation |
+| `checkpoint-bitlinear-adamw-working.pt` | BitLinear | **1.92** | ✅ WORKING | Ternary quantization, coherent generation |
+
+---
+
+### Experimental/Failed Models (Already uploaded)
+
+| File | Type | Val Loss | Status | Description |
+|------|------|----------|--------|-------------|
+| `checkpoint-step5000-experiment.pt` | BitLinear | 8.45 | ✅ Pipeline verified | First test, proved architecture works |
+| `checkpoint-step50000-failed-collapsed.pt` | BitLinear | 3.16 | ❌ COLLAPSED | Beta1=None, repetitive output |
+| `checkpoint-step50000-experiment-gibberish.pt` | BitLinear | 7.56 | ⚠️ Not useful | Beta1=0.9, healthy but slow learning |
 
 ---
 
 ## Training Configuration
 
+### Working Models Configuration
+
+| Setting | FP32 + AdamW | BitLinear + AdamW |
+|---------|--------------|-------------------|
+| Dataset | TinyStories | TinyStories |
+| Steps | 5,000 | 9,500 |
+| Batch size | 4 | 4 |
+| Gradient accumulation | 8 | 8 |
+| Effective batch size | 32 | 32 |
+| Optimizer | AdamW | AdamW |
+| Max Learning Rate | 0.001 | 0.001 |
+| Min Learning Rate | 0.0001 | 0.0001 |
+| Warmup steps | 100 | 100 |
+| Weight decay | 0.01 | 0.01 |
+| Gradient clipping | 1.0 | 1.0 |
+
+---
+
+### Failed Experiments Configuration
+
 | Setting | Value |
 |---------|-------|
-| Dataset | TinyStories (roneneldan/TinyStories) |
+| Dataset | TinyStories |
 | Samples | 50,000 |
 | Batch size | 4 |
 | Gradient accumulation | 8 |
 | Effective batch size | 32 |
-| Optimizer | Adafactor |
+| Optimizer | Adafactor (buggy) |
 | Max Learning Rate | 0.0003 |
 | Min Learning Rate | 0.00003 |
 | Warmup steps | 1,000 |
@@ -119,64 +236,78 @@ This project explores whether **ternary quantization** (weights constrained to -
 
 ---
 
-## Key Findings
+## Total Time Invested
 
-### Beta1 Trade-off
+| Phase | Duration | Outcome |
+|-------|----------|---------|
+| Initial experiments (August) | 167 hours | Identified beta1 issues |
+| Root cause investigation (September) | 5 days | Found Adafactor bug |
+| Working models (August-September) | 15+ hours | ✅ Both models working |
+| Canonical Adafactor (October) | 1 day | Implementation verified, performance gap observed |
 
-| Beta1 Value | Learning Speed | Stability | Result |
-|-------------|---------------|-----------|--------|
-| None (0.0) | FAST | UNSTABLE | COLLAPSED at 50k |
-| 0.9 | SLOW | STABLE | HEALTHY but not useful |
-| 0.5 - 0.7 | UNKNOWN | UNKNOWN | Need to test |
+---
 
-**Critical Lesson:** Beta1=None causes collapse. Beta1=0.9 too conservative. Need to find optimal value.
+## Lessons Learned
 
----
+### Technical Lessons
 
-### Training Metrics Comparison
+1. **Optimizer implementation bugs are subtle** - Adafactor computed variance but didn't apply it
+2. **Always verify against reference implementations** - PyTorch Adafactor differs from custom implementation
+3. **AdamW is robust for ternary quantization** - Proven to work where Adafactor failed
+4. **Memory vs performance trade-offs exist** - 316x memory savings vs 2x performance gap
+5. **Lower loss ≠ better model** - Collapsed model had lower loss (3.16 vs 7.56) due to repetitive output
 
-| Training | Loss @ 50k | Entropy | Top Token | Status |
-|----------|-----------|---------|-----------|--------|
-| Attempt 2 (beta1=None) | 3.16 | 8.2% | 94% "time" | COLLAPSED |
-| Attempt 4 (beta1=0.9) | 7.56 | 96.2% | 0.56% "the" | HEALTHY (gibberish) |
+### Research Lessons
 
-**Paradox:** Lower loss does NOT mean better model. Attempt 2 had lower loss but was collapsed.
+1. **Entropy monitoring is CRITICAL** - Loss alone doesn't indicate model health
+2. **Systematic investigation pays off** - Line-by-line code audit found the bug
+3. **Document everything** - 59 phases of investigation documented
+4. **Test one variable at a time** - Controlled experiments essential for debugging
 
 ---
 
-## Total Time Invested
+## How to Use Working Models
 
-| Attempt | Duration | Outcome |
-|---------|----------|---------|
-| 1 (5k test) | 4 hours | SUCCESS (pipeline verified) |
-| 2 (50k collapsed) | 79 hours | FAILED (beta1=None) |
-| 3 (stopped) | 1 hour | STOPPED (wrong accumulation) |
-| 4 (50k gibberish) | 83 hours | COMPLETED (not useful) |
-| **Total** | **167 hours** | |
+### FP32 + AdamW Baseline
 
----
+```python
+import torch
+from src.model.gpt import GPT
+from src.tokenizer.bpe_tokenizer import BPETokenizer
 
-## Lessons Learned
+# Load model architecture (FP32)
+model = GPT(
+    vocab_size=4000,
+    d_model=256,
+    n_heads=8,
+    n_layers=6,
+    d_ff=1024,
+    max_seq_len=256
+)
 
-1. **Entropy monitoring is CRITICAL** - Loss alone doesn't indicate model health
-2. **Beta1=None causes collapse** - Model falls into repetitive patterns
-3. **Beta1=0.9 is too conservative** - Learning becomes extremely slow
-4. **Lower loss ≠ better model** - Collapsed model had lower loss (3.16 vs 7.56)
-5. **Need optimal beta1** - Testing 0.5 and 0.7 as candidates
-6. **Ternary quantization is challenging** - Requires careful optimizer tuning
+# Load checkpoint
+checkpoint = torch.load('checkpoint-fp32-adamw-working.pt', map_location='cpu')
+model.load_state_dict(checkpoint['model_state_dict'])
+model.eval()
 
----
+# Load tokenizer
+tokenizer = BPETokenizer.load('tokenizer.json')
 
-## How to Use These Checkpoints
+# Generate text
+input_ids = torch.tensor([tokenizer.encode("Once upon a time")])
+# ... generation code ...
+```
 
-### For Research/Learning
+---
+
+### BitLinear + AdamW (Ternary)
 
 ```python
 import torch
 from src.model.bitlinear import BitLinearGPT
 from src.tokenizer.bpe_tokenizer import BPETokenizer
 
-# Load model architecture
+# Load model architecture (BitLinear)
 model = BitLinearGPT(
     vocab_size=4000,
     d_model=256,
@@ -186,9 +317,9 @@ model = BitLinearGPT(
     max_seq_len=256
 )
 
-# Load checkpoint (choose one)
-checkpoint = torch.load('checkpoint-step50000-experiment-gibberish.pt', map_location='cpu')
-model.load_state_dict(checkpoint)
+# Load checkpoint
+checkpoint = torch.load('checkpoint-bitlinear-adamw-working.pt', map_location='cpu')
+model.load_state_dict(checkpoint['model_state_dict'])
 model.eval()
 
 # Load tokenizer
@@ -199,22 +330,14 @@ input_ids = torch.tensor([tokenizer.encode("Once upon a time")])
 # ... generation code ...
 ```
 
-### Not For Production
-
-These checkpoints are experimental. They will NOT produce coherent text. Use them to:
-- Learn about ternary quantization challenges
-- Understand model collapse patterns
-- Compare optimizer settings
-- Train your own models with different configs
-
 ---
 
 ## Tokenizers
 
 | File | Description |
 |------|-------------|
-| tokenizer.json | Main tokenizer (BPE, vocab size 4000) |
-| tokenizer-backup-50k.json | Backup from collapsed training |
+| `tokenizer-5k-experiment.json` | Tokenizer for early experiments |
+| `tokenizer-50k-experiments.json` | Tokenizer for 50k step experiments |
 
 ---
 
@@ -226,30 +349,39 @@ Full source code available at GitHub:
 
 Includes:
 - Complete training scripts
-- Model architecture
+- Model architecture (FP32 and BitLinear)
+- Optimizer implementations
 - Training history documentation
 - Monitoring tools
+- 59 phases of investigation documented
 
 ---
 
-## Training History
+## Documentation
+
+Complete investigation documentation:
 
-Complete documentation of all experiments:
+**https://github.com/kishxrx/ARTN/blob/main/experiments/results/ROOT_CAUSE_INVESTIGATION_SUMMARY.md**
 
-**https://github.com/kishxrx/ARTN/blob/main/TRAINING_HISTORY.md**
+Covers:
+- Phases 1-59 of systematic debugging
+- Root cause analysis
+- Verification experiments
+- Evidence chain
 
 ---
 
 ## Next Steps
 
-| Phase | Plan |
-|-------|------|
-| 1 | Test beta1=0.5 (10k steps) |
-| 2 | Test beta1=0.7 (10k steps) |
-| 3 | Compare entropy, loss, samples |
-| 4 | Select optimal beta1 |
-| 5 | Full 50k training with best config |
-| 6 | Release v1.0 if successful |
+| Phase | Status | Plan |
+|-------|--------|------|
+| 1 | ✅ COMPLETE | FP32 baseline established (val loss 1.69) |
+| 2 | ✅ COMPLETE | BitLinear + AdamW working (val loss 1.92) |
+| 3 | ✅ COMPLETE | Root cause identified (Adafactor bug) |
+| 4 | ✅ COMPLETE | Canonical Adafactor implemented and tested |
+| 5 | 🔄 IN PROGRESS | Scale training with AdamW (longer runs) |
+| 6 | 📋 PLANNED | Evaluate on larger datasets |
+| 7 | 📋 PLANNED | Release v1.0 with working models |
 
 ---
 
@@ -263,7 +395,7 @@ If you use this work for learning or research:
   title = {ARTN: Adaptive Ternary Recurrent Network - 1.58-bit Quantization Experiments},
   year = {2026},
   publisher = {HuggingFace},
-  url = {https://huggingface.co/kishxrx/artn-ternary-experimental}
+  url = {https://huggingface.co/kishxrx/ARTN}
 }
 ```
 
@@ -279,9 +411,9 @@ MIT License
 
 **THIS IS EXPERIMENTAL SOFTWARE.**
 
-- Models do NOT produce coherent text
-- Checkpoints are for learning purposes only
-- Not suitable for any production use
+- Working models produce coherent text but are experimental
+- Failed models are for learning purposes only
+- Not suitable for production use without further testing
 - No guarantees of any kind
 
 ---
@@ -293,6 +425,6 @@ MIT License
 
 ---
 
-**Last Updated:** August 2026
+**Last Updated:** October 15, 2026
 
-**Version:** v0.1-experimental
+**Version:** v0.2-research-progress