dragonfly-135m
dragonfly-135m is a 135M-parameter Llama-based decoder-only language model trained from scratch on 2.7B tokens of synthetic data. Training followed the Chinchilla compute-optimal recipe (20 tokens per parameter), using a single pass over Cosmopedia-v2 with packed sequences and per-document attention masking. It is not a particularly interesting model except for the fact that it exists; it serves as a baseline.
Model Details
| Property | Value |
|---|---|
| Architecture | Llama (LlamaForCausalLM) — SmolLM2-135M |
| Parameters | ~135M |
| Context length | 4,096 tokens |
Training
Data
| Split | Dataset | Subset | Size |
|---|---|---|---|
| Train | HuggingFaceTB/smollm-corpus | cosmopedia-v2 | ~2.7B tokens |
| Validation | HuggingFaceFW/fineweb-edu | sample-10BT (1k docs) | held-out |
Training sequences are packed using a best-fit algorithm with per-document attention masks to prevent cross-document attention leakage.
Hyperparameters
| Hyperparameter | Value |
|---|---|
| Target tokens | 2,700,000,000 |
| Steps | 5,150 |
| Tokens per step | 524,288 |
| Per-device batch size | 32 |
| Gradient accumulation | 4 |
| Effective batch size | 128 sequences / 524,288 tokens |
| Peak learning rate | 5e-4 |
| LR schedule | Cosine decay, 10% linear warmup (515 steps) |
| Weight decay | 0.1 |
Final metrics:
- Train loss: 2.259
- Eval loss (FineWeb-Edu, 1k docs): 4.381
- Training time: ~53.4 hours
- Average step time: ~41.7 s/step
Infrastructure
- Hardware: GB10
- Framework: HuggingFace Transformers + PyTorch
Limitations
This is a raw base model with no instruction tuning or safety alignment. It may produce harmful, inaccurate, or incoherent content. Use with caution.
Third-Party Licenses
The model architecture configuration and tokenizer are derived from SmolLM2-135M by HuggingFace, licensed under the Apache 2.0 License. See THIRD_PARTY_LICENSES for details. The model weights are original and released under MIT.
- Downloads last month
- 18