File size: 699 Bytes
f744436 01bf47d eda7b33 3f0e281 06f49ae | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | ---
license: mit
datasets:
- HuggingFaceCode/stack-v3-train
---
My new 128M code completion model, named Vega 1.5 both because it has a new tokenizer... and I don't like how "VegaLM1" looks anyway.
This model was trained on ~5.5B tokens of stack-v3-train (python subset), and only understands python. Attempted FIM but didn't work because I'm a dummy. Bananamind Base Bench 1.1 code completion Elo is 1434, which is a meaningful jump over 42M-CodeCompletion (and is second only to SmolLM as of now!)
## Model specs
- Layers: 16
- Hidden size: 768
- Attention heads: 12
- KV heads: 6
- Context length: Trained on 2,048 tokens
- Intermediate size: 2048
- Vocabulary: 32,003
- Parameters: 128,410,368 |