File size: 3,369 Bytes
e4c3273 3b2e8b0 e4c3273 96bee46 3b2e8b0 96bee46 3b2e8b0 7282066 96bee46 e4c3273 d4c2707 e4c3273 96bee46 d4c2707 e4c3273 d4c2707 96bee46 d4c2707 e4c3273 d4c2707 e4c3273 d4c2707 e4c3273 d4c2707 e4c3273 ba3d55e e4c3273 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 | ---
license: apache-2.0
language:
- en
- it
tags:
- custom-transformer
- from-scratch
- small-language-model
- gqa
- swiglu
- rope
- pytorch
- streamlit
library_name: custom
---
# Nexus SmAll v1
A **89.8M parameter** causal language model built entirely from scratch using a custom transformer architecture, trained on WikiText-103 + synthetic instruction data.
## Try it Online
Test the model directly in your browser (free, no login required):
[](https://try-nexus-ai.streamlit.app)
[](https://github.com/JustScriptzz/nexus-smAll-web)
## Architecture
| Parameter | Value |
|-----------|-------|
| Parameters | 89.8M |
| Layers | 10 |
| Hidden dim | 768 |
| Attention heads | 12 |
| KV heads (GQA) | 4 |
| Max sequence length | 512 |
| Vocab size | 50,304 |
| FFN activation | SwiGLU |
| Position encoding | RoPE (θ=500000) |
| Norm | RMSNorm |
| Training | Float32, full precision |
Key design choices:
- **Grouped Query Attention (GQA)**: 12 query heads, 4 KV heads for efficient inference
- **SwiGLU FFN**: Gated activation for better training dynamics
- **RoPE**: Rotary Position Embeddings for length extrapolation
- **Tied embeddings**: Input/output embeddings share weights
## Installation
Requirements: Python 3.8+ and pip.
```bash
# 1. Clone the repository
git clone https://huggingface.co/JustScriptzz/nexus-smAll-v1
cd nexus-smAll-v1
# 2. Install dependencies
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install tokenizers
```
> **GPU users**: Replace `--index-url https://download.pytorch.org/whl/cpu` with the appropriate CUDA version, e.g. `--index-url https://download.pytorch.org/whl/cu124` for CUDA 12.4.
## Usage
### Command line (quick start)
```bash
python chat.py --weights weights/nexus_instruct.pt
```
### Python API
```python
from src.model import Nexus
from src.config import NexusConfig
from tokenizers import Tokenizer
import torch
config = NexusConfig()
model = Nexus(config)
checkpoint = torch.load("weights/nexus_instruct.pt", map_location="cpu", weights_only=False)
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()
tokenizer = Tokenizer.from_file("data/tokenizer.json")
prompt = "User: What is Python?\nAssistant:"
encoded = tokenizer.encode(prompt)
tokens = torch.tensor([encoded.ids])
output, _ = model.generate(tokens, max_new_tokens=64, temperature=0.2, top_k=40, top_p=0.9)
reply = tokenizer.decode(output)
print(reply)
```
## Training
- **Phase 1**: 100k steps on WikiText-103 (next-token prediction, ~212k sequences)
- **Phase 2**: 5k steps on instruction data (Dolly-15k + synthetic QA, ~1500 examples)
## Limitations
- 90M parameters is very small by modern standards — outputs may be incoherent
- Not instruction-tuned on diverse enough data
- Best used as a learning experiment, not production
## Nexus Plus v2
Looking for a larger, more capable model? Check out **[Nexus Plus v2](https://huggingface.co/JustScriptzz/nexus-plus-v2)** — a Qwen3-4B fine-tune with QLoRA, trained on the same instruction dataset.
[](https://huggingface.co/JustScriptzz/nexus-plus-v2)
## License
Apache 2.0
|