Create README.md

README.md

@@ -0,0 +1,64 @@
+
+---
+language:
+  - en
+tags:
+  - text-generation
+  - causal-lm
+  - custom-architecture
+  - slm
+  - small-language-model
+license: mit
+---
+
+# Spark (7.08M Parameters)
+
+Spark is a highly efficient, ultra-small language model developed by SurjoLabs. It demonstrates that aggressive overtraining on high-quality data combined with a custom XSA recurrent architecture can yield exceptional reasoning capabilities in tiny architectures.
+
+## Methodology
+
+Spark is built to prove that unique recurrent architectures can match or beat standard transformers even at the smallest scales.
+
+* **Custom Architecture:** Built on a Llama-variant framework but features **XSA Attention** (value-subtraction projection) and **Recurrent Layers**. By reusing weights across recurrent passes, the model achieves the depth of an 11-layer model while only carrying 8 layers of unique weights.
+* **Optimized Tokenizer:** Uses a custom 4,096-vocabulary English tokenizer. This intentionally small vocabulary prevents the embedding table from dominating the parameter count, ensuring nearly 85% of the model's 7.08M parameters are dedicated to actual transformer compute logic.
+* **Extreme Overtraining:** Trained on **~19.9 Billion tokens** (19,000 steps × 1,048,576 tokens/step). This forces the small architecture to memorize syntax and knowledge far beyond Chinchilla-optimal limits.
+* **Data Mixture:** 
+  * 60% Finephrase (Synthetic FineWeb-Edu)
+  * 20% DCLM (Web text)
+  * 10% FineMath (Mathematics)
+  * 10% CornStack (Code)
+* **Optimizer:** Utilizes a hybrid Muon (for 2D weight matrices) and AdamW (for embeddings/norms) optimizer setup for stable, rapid convergence.
+
+## Benchmark Results (Open SLM Leaderboard)
+
+| Benchmark | Score (acc_norm) |
+| :--- | :--- |
+| HellaSwag | 28.17% |
+| ARC-Easy | 35.02% |
+| ARC-Challenge | 20.99% |
+| PIQA | 55.55% |
+| ArithMark-3 | 36.10% |
+| **Intelligence Index** | **7.93** |
+
+## Note on Checkpoint
+The model was trained for 20,000 steps, but evaluation showed that the checkpoint at **step 19,000** performed better on nearly all benchmarks. The final step (20,000) was discarded, and step 19,000 is the version hosted in the root of this repository.
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model_id = "SurjoLabs/Spark"
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id, 
+    trust_remote_code=True, 
+    torch_dtype=torch.bfloat16
+).cuda()
+
+prompt = "The capital of France is"
+inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
+outputs = model.generate(**inputs, max_new_tokens=20)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```