Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -11,6 +11,7 @@ tags:
|
|
| 11 |
- swiglu
|
| 12 |
- rope
|
| 13 |
- pytorch
|
|
|
|
| 14 |
library_name: custom
|
| 15 |
---
|
| 16 |
|
|
@@ -18,6 +19,12 @@ library_name: custom
|
|
| 18 |
|
| 19 |
A **89.8M parameter** causal language model built entirely from scratch using a custom transformer architecture, trained on WikiText-103 + synthetic instruction data.
|
| 20 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
## Architecture
|
| 22 |
|
| 23 |
| Parameter | Value |
|
|
@@ -42,15 +49,29 @@ Key design choices:
|
|
| 42 |
|
| 43 |
## Installation
|
| 44 |
|
|
|
|
|
|
|
| 45 |
```bash
|
| 46 |
-
|
|
|
|
| 47 |
cd nexus-smAll-v1
|
| 48 |
-
|
|
|
|
|
|
|
|
|
|
| 49 |
```
|
| 50 |
|
|
|
|
|
|
|
| 51 |
## Usage
|
| 52 |
|
| 53 |
-
###
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
|
| 55 |
```python
|
| 56 |
from src.model import Nexus
|
|
@@ -67,22 +88,15 @@ model.eval()
|
|
| 67 |
|
| 68 |
tokenizer = Tokenizer.from_file("data/tokenizer.json")
|
| 69 |
|
| 70 |
-
prompt = "
|
| 71 |
encoded = tokenizer.encode(prompt)
|
| 72 |
tokens = torch.tensor([encoded.ids])
|
| 73 |
|
| 74 |
output, _ = model.generate(tokens, max_new_tokens=64, temperature=0.2, top_k=40, top_p=0.9)
|
| 75 |
reply = tokenizer.decode(output)
|
| 76 |
-
|
| 77 |
print(reply)
|
| 78 |
```
|
| 79 |
|
| 80 |
-
### Command line
|
| 81 |
-
|
| 82 |
-
```bash
|
| 83 |
-
python chat.py --weights weights/nexus_instruct.pt
|
| 84 |
-
```
|
| 85 |
-
|
| 86 |
## Training
|
| 87 |
|
| 88 |
- **Phase 1**: 100k steps on WikiText-103 (next-token prediction, ~212k sequences)
|
|
|
|
| 11 |
- swiglu
|
| 12 |
- rope
|
| 13 |
- pytorch
|
| 14 |
+
- gradio
|
| 15 |
library_name: custom
|
| 16 |
---
|
| 17 |
|
|
|
|
| 19 |
|
| 20 |
A **89.8M parameter** causal language model built entirely from scratch using a custom transformer architecture, trained on WikiText-103 + synthetic instruction data.
|
| 21 |
|
| 22 |
+
## Try it Online
|
| 23 |
+
|
| 24 |
+
You can try Nexus SmAll v1 directly in your browser using the Hugging Face Space below:
|
| 25 |
+
|
| 26 |
+
👉 **[Nexus SmAll v1 Chat](https://huggingface.co/spaces/JustScriptzz/nexus-smAll-v1)**
|
| 27 |
+
|
| 28 |
## Architecture
|
| 29 |
|
| 30 |
| Parameter | Value |
|
|
|
|
| 49 |
|
| 50 |
## Installation
|
| 51 |
|
| 52 |
+
Requirements: Python 3.8+ and pip.
|
| 53 |
+
|
| 54 |
```bash
|
| 55 |
+
# 1. Clone the repository
|
| 56 |
+
git clone https://huggingface.co/JustScriptzz/nexus-smAll-v1
|
| 57 |
cd nexus-smAll-v1
|
| 58 |
+
|
| 59 |
+
# 2. Install dependencies
|
| 60 |
+
pip install torch --index-url https://download.pytorch.org/whl/cpu
|
| 61 |
+
pip install tokenizers
|
| 62 |
```
|
| 63 |
|
| 64 |
+
> **GPU users**: Replace `--index-url https://download.pytorch.org/whl/cpu` with the appropriate CUDA version, e.g. `--index-url https://download.pytorch.org/whl/cu124` for CUDA 12.4.
|
| 65 |
+
|
| 66 |
## Usage
|
| 67 |
|
| 68 |
+
### Command line (quick start)
|
| 69 |
+
|
| 70 |
+
```bash
|
| 71 |
+
python chat.py --weights weights/nexus_instruct.pt
|
| 72 |
+
```
|
| 73 |
+
|
| 74 |
+
### Python API
|
| 75 |
|
| 76 |
```python
|
| 77 |
from src.model import Nexus
|
|
|
|
| 88 |
|
| 89 |
tokenizer = Tokenizer.from_file("data/tokenizer.json")
|
| 90 |
|
| 91 |
+
prompt = "User: What is Python?\nAssistant:"
|
| 92 |
encoded = tokenizer.encode(prompt)
|
| 93 |
tokens = torch.tensor([encoded.ids])
|
| 94 |
|
| 95 |
output, _ = model.generate(tokens, max_new_tokens=64, temperature=0.2, top_k=40, top_p=0.9)
|
| 96 |
reply = tokenizer.decode(output)
|
|
|
|
| 97 |
print(reply)
|
| 98 |
```
|
| 99 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 100 |
## Training
|
| 101 |
|
| 102 |
- **Phase 1**: 100k steps on WikiText-103 (next-token prediction, ~212k sequences)
|