Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,99 @@
+---
+license: apache-2.0
+language:
+- en
+- it
+tags:
+- custom-transformer
+- from-scratch
+- small-language-model
+- gqa
+- swiglu
+- rope
+- pytorch
+library_name: custom
+---
+
+# Nexus SmAll v1
+
+A **89.8M parameter** causal language model built entirely from scratch using a custom transformer architecture, trained on WikiText-103 + synthetic instruction data.
+
+## Architecture
+
+| Parameter | Value |
+|-----------|-------|
+| Parameters | 89.8M |
+| Layers | 10 |
+| Hidden dim | 768 |
+| Attention heads | 12 |
+| KV heads (GQA) | 4 |
+| Max sequence length | 512 |
+| Vocab size | 50,304 |
+| FFN activation | SwiGLU |
+| Position encoding | RoPE (θ=500000) |
+| Norm | RMSNorm |
+| Training | Float32, full precision |
+
+Key design choices:
+- **Grouped Query Attention (GQA)**: 12 query heads, 4 KV heads for efficient inference
+- **SwiGLU FFN**: Gated activation for better training dynamics
+- **RoPE**: Rotary Position Embeddings for length extrapolation
+- **Tied embeddings**: Input/output embeddings share weights
+
+## Installation
+
+```bash
+git clone https://huggingface.co/MarcoTesting/nexus-smAll-v1
+cd nexus-smAll-v1
+pip install torch tokenizers
+```
+
+## Usage
+
+### Chat
+
+```python
+from src.model import Nexus
+from src.config import NexusConfig
+from tokenizers import Tokenizer
+import torch
+
+config = NexusConfig()
+model = Nexus(config)
+
+checkpoint = torch.load("weights/nexus_instruct.pt", map_location="cpu", weights_only=False)
+model.load_state_dict(checkpoint["model_state_dict"])
+model.eval()
+
+tokenizer = Tokenizer.from_file("data/tokenizer.json")
+
+prompt = "<|user|>\nWhat is Python?\n<|assistant|>\n"
+encoded = tokenizer.encode(prompt)
+tokens = torch.tensor([encoded.ids])
+
+output, _ = model.generate(tokens, max_new_tokens=64, temperature=0.2, top_k=40, top_p=0.9)
+reply = tokenizer.decode(output)
+
+print(reply)
+```
+
+### Command line
+
+```bash
+python chat.py --weights weights/nexus_instruct.pt
+```
+
+## Training
+
+- **Phase 1**: 100k steps on WikiText-103 (next-token prediction, ~212k sequences)
+- **Phase 2**: 5k steps on instruction data (Dolly-15k + synthetic QA, ~1500 examples)
+
+## Limitations
+
+- 90M parameters is very small by modern standards — outputs may be incoherent
+- Not instruction-tuned on diverse enough data
+- Best used as a learning experiment, not production
+
+## License
+
+Apache 2.0