Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Tyrian 75M
|
| 2 |
+
|
| 3 |
+
A 75M parameter decoder-only language model built entirely from scratch in PyTorch — no HuggingFace model classes, no nanoGPT wrapping. Every component (tokenizer, architecture, data pipeline, training loop, SFT) was written by hand.
|
| 4 |
+
|
| 5 |
+
## Model Details
|
| 6 |
+
|
| 7 |
+
| Property | Value |
|
| 8 |
+
|---|---|
|
| 9 |
+
| Parameters | 74,920,704 (~75M) |
|
| 10 |
+
| Architecture | Decoder-only transformer |
|
| 11 |
+
| Hidden size | 768 |
|
| 12 |
+
| Layers | 8 |
|
| 13 |
+
| Query heads | 12 (GQA) |
|
| 14 |
+
| KV heads | 4 (GQA) |
|
| 15 |
+
| FFN size | 2048 (SwiGLU) |
|
| 16 |
+
| Context length | 2048 tokens |
|
| 17 |
+
| Vocab size | 32,000 |
|
| 18 |
+
| Normalization | RMSNorm (pre-norm) |
|
| 19 |
+
| Position encoding | RoPE (θ=10000) |
|
| 20 |
+
| Attention | Flash Attention (SDPA) |
|
| 21 |
+
| FFN activation | SwiGLU |
|
| 22 |
+
| Biases | None |
|
| 23 |
+
| Embeddings | Tied (input = output) |
|
| 24 |
+
|
| 25 |
+
## Training
|
| 26 |
+
|
| 27 |
+
**Pretraining**
|
| 28 |
+
- ~17.8B tokens of English web text
|
| 29 |
+
- Data mix: FineWeb-Edu, Cosmopedia, StackExchange, Wikipedia, Reddit, WildChat, UltraChat, OASST2
|
| 30 |
+
- Custom BPE tokenizer (32K vocab, ChatML format)
|
| 31 |
+
- Cosine LR schedule: 3e-4 → 3e-5 with 2000-step warmup
|
| 32 |
+
- AdamW (β₁=0.9, β₂=0.95), weight decay 0.1
|
| 33 |
+
- Batch: 512K tokens/step
|
| 34 |
+
- Hardware: 2× RTX 5060 Ti 16GB, DDP
|
| 35 |
+
|
| 36 |
+
**SFT Fine-tuning**
|
| 37 |
+
- 100K examples from OpenHermes-2.5
|
| 38 |
+
- ChatML format with loss masking on user/system tokens
|
| 39 |
+
- LR: 2e-5 → 2e-6, 3 epochs
|
| 40 |
+
|
| 41 |
+
## Benchmarks
|
| 42 |
+
|
| 43 |
+
Evaluated with log-likelihood scoring (no few-shot):
|
| 44 |
+
|
| 45 |
+
| Task | Score | Random |
|
| 46 |
+
|---|---|---|
|
| 47 |
+
| HellaSwag | 27.8% | 25% |
|
| 48 |
+
| PIQA | 61.0% | 50% |
|
| 49 |
+
| ARC-Easy | 39.3% | 25% |
|
| 50 |
+
| ARC-Challenge | 25.4% | 25% |
|
| 51 |
+
| WinoGrande | 51.7% | 50% |
|
| 52 |
+
|
| 53 |
+
Comparable to GPT-2 (117M) at 0.64× the parameter count.
|
| 54 |
+
|
| 55 |
+
## Usage
|
| 56 |
+
|
| 57 |
+
```python
|
| 58 |
+
from transformers import AutoTokenizer, AutoModelForCausalLM
|
| 59 |
+
import torch
|
| 60 |
+
|
| 61 |
+
model = AutoModelForCausalLM.from_pretrained(
|
| 62 |
+
"redptam/tyrian-75m",
|
| 63 |
+
trust_remote_code=True,
|
| 64 |
+
torch_dtype=torch.bfloat16,
|
| 65 |
+
).cuda()
|
| 66 |
+
|
| 67 |
+
tokenizer = AutoTokenizer.from_pretrained("redptam/tyrian-75m", trust_remote_code=True)
|
| 68 |
+
|
| 69 |
+
# Chat (ChatML format)
|
| 70 |
+
prompt = "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n"
|
| 71 |
+
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
|
| 72 |
+
output = model.generate(**inputs, max_new_tokens=100, temperature=0.8, top_k=50,
|
| 73 |
+
stop_token_ids=(tokenizer.convert_tokens_to_ids("<|im_end|>"),))
|
| 74 |
+
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))
|
| 75 |
+
```
|
| 76 |
+
|
| 77 |
+
## Special Tokens
|
| 78 |
+
|
| 79 |
+
| Token | ID |
|
| 80 |
+
|---|---|
|
| 81 |
+
| `<pad>` | 0 |
|
| 82 |
+
| `<bos>` | 1 |
|
| 83 |
+
| `<eos>` | 2 |
|
| 84 |
+
| `<unk>` | 3 |
|
| 85 |
+
| `<\|im_start\|>` | 4 |
|
| 86 |
+
| `<\|im_end\|>` | 5 |
|
| 87 |
+
|
| 88 |
+
## License
|
| 89 |
+
|
| 90 |
+
MIT
|