Add model card for banana-idk-3m
#1
by Compactbot - opened
README.md
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
language: en
|
| 4 |
+
tags:
|
| 5 |
+
- tiny
|
| 6 |
+
- tiny-lm
|
| 7 |
+
- slm
|
| 8 |
+
- small-language-model
|
| 9 |
+
- from-scratch
|
| 10 |
+
- transformer
|
| 11 |
+
pipeline_tag: text-generation
|
| 12 |
+
metrics:
|
| 13 |
+
- accuracy
|
| 14 |
+
- perplexity
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
# banana-idk-3m
|
| 18 |
+
|
| 19 |
+
A 2,715,984-parameter transformer trained from scratch on 8B tokens of FineWeb-Edu.
|
| 20 |
+
|
| 21 |
+
## Architecture
|
| 22 |
+
|
| 23 |
+
| Component | Value |
|
| 24 |
+
|-----------|-------|
|
| 25 |
+
| Embedding dim (d) | 144 |
|
| 26 |
+
| Layers | 6 |
|
| 27 |
+
| Query heads | 2 |
|
| 28 |
+
| KV heads | 1 (GQA) |
|
| 29 |
+
| Head dim | 80 |
|
| 30 |
+
| FFN | SwiGLU, hidden=432 |
|
| 31 |
+
| Vocab | 8192 (BPE) |
|
| 32 |
+
| Positional encoding | RoPE |
|
| 33 |
+
| Head | Tied with embedding |
|
| 34 |
+
| **Learnable params** | **2,715,984** |
|
| 35 |
+
|
| 36 |
+
## Training
|
| 37 |
+
|
| 38 |
+
- **Data:** FineWeb-Edu, 8B tokens
|
| 39 |
+
- **Optimizer:** AdamW, lr=5e-3, cosine decay + 200-step warmup
|
| 40 |
+
- **Batch:** 32 × seq 2048 (grad accum 1)
|
| 41 |
+
- **Steps:** 122,070
|
| 42 |
+
- **Hardware:** RTX 5090 (32 GB)
|
| 43 |
+
|
| 44 |
+
## Eval Results (zero-shot loglikelihood, 200 examples each)
|
| 45 |
+
|
| 46 |
+
| Benchmark | Score |
|
| 47 |
+
|-----------|-------|
|
| 48 |
+
| PIQA | 52.5% |
|
| 49 |
+
| Arithmark-3.0 | 30.0% |
|
| 50 |
+
| ARC-Challenge | 23.5% |
|
| 51 |
+
| ARC-Easy | 25.5% |
|
| 52 |
+
| HellaSwag | 0.0% |
|
| 53 |
+
|
| 54 |
+
These numbers reflect the model's scale (2.7M params). PIQA is barely above chance (50%); HellaSwag is at chance (25% is random for 4-way, but the model scores 0% — it appears to have no learned commonsense sentence-completion signal). The model is a research artifact, not a usable chatbot.
|
| 55 |
+
|
| 56 |
+
## Files
|
| 57 |
+
|
| 58 |
+
- `final.pt` — full checkpoint (model weights + optimizer state), 35 MB
|
| 59 |
+
- `tokenizer/` — BPE tokenizer (8192 vocab)
|
| 60 |
+
|
| 61 |
+
## Loading
|
| 62 |
+
|
| 63 |
+
The model uses a custom architecture (GQA + SwiGLU + RoPE). It is NOT a standard HuggingFace transformers model. A minimal forward pass implementation is included in the training script. To load:
|
| 64 |
+
|
| 65 |
+
```python
|
| 66 |
+
import torch
|
| 67 |
+
ckpt = torch.load("final.pt", map_location="cpu", weights_only=False)
|
| 68 |
+
sd = ckpt["model"]
|
| 69 |
+
# See training script for full architecture reconstruction
|
| 70 |
+
```
|
| 71 |
+
|
| 72 |
+
## Notes
|
| 73 |
+
|
| 74 |
+
- Requested by @Banaxi-Tech in [model-requests#19](https://huggingface.co/spaces/Compactbot/model-requests/discussions/19)
|
| 75 |
+
- The "control" variant: pure attention (no hybrid/mixed components), as a baseline
|
| 76 |
+
- HellaSwag 0% is real (confirmed by re-run); the model does not appear to have learned commonsense continuation despite 8B tokens
|