---
license: mit
datasets:
- HuggingFaceCode/stack-v3-train
---
My new 128M code completion model, named Vega 1.5 both because it has a new tokenizer... and I don't like how "VegaLM1" looks anyway.

This model was trained on ~5.5B tokens of stack-v3-train (python subset), and only understands python. Attempted FIM but didn't work because I'm a dummy. Bananamind Base Bench 1.1 code completion Elo is 1434, which is a meaningful jump over 42M-CodeCompletion (and is second only to SmolLM as of now!)

## Model specs
- Layers: 16
- Hidden size: 768
- Attention heads: 12
- KV heads: 6
- Context length: Trained on 2,048 tokens
- Intermediate size: 2048
- Vocabulary: 32,003
- Parameters: 128,410,368