---
license: apache-2.0
viewer: false
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- small
- tiny
- supra
- supra2-5
- supra2.5
- qwen3
- efficient
---
Supra2.5-Pro Base
Ultra-efficient base model • 65M Parameters • Next-generation Supra

**Supra2.5-Pro Base** is a 65M-parameter small language model pretrained from scratch by **SupraLabs** on 100B tokens of high quality web data. It uses the **Qwen3** architecture with a custom 16,384-token tokenizer.
This is a **base model**. It has *not* been instruction-tuned, chat-tuned, or aligned in any way.
Now with 65M parameters, better data and improved training processes, Supra2.5-Pro Base shows a significant improvement over previous generations of Supra models (e.g. Supra-50M and Supra2-100M).
---
## First things first - Benchmarks

|
ARC-Easy |
ARC-Challenge |
PIQA |
HellaSwag |
ArithMark-3 |
Average |
| Supra2.5-65M |
50.80% |
25.77% |
63.11% |
34.35% |
49.60% |
44.73% |
| Supra2-100M |
47.81% |
24.83% |
65.40% |
35.98% |
36.90% |
42.18% |
| Supra-50M |
46.00% |
25.17% |
61.19% |
31.72% |
36.30% |
40.08% |
---
## Model details
### Basic information
| | |
| ----- | ----- |
| **Developed by** | SupraLabs |
| **Model type** | Causal decoder-only transformer (Qwen3) |
| **Language** | English (parts of the data contained other languages like German, French, Spanish, and others too but it wasn't learned pretty well) |
| **Parameters** | ~65M |
| **Training tokens** | 100B |
| **Context length** | 512 tokens |
| **Precision** | bfloat16 |
| **License** | Apache 2.0 |
### Architecture details
| Hyperparameter | Value |
| ----- | ----- |
| Architecture | `Qwen3ForCausalLM` |
| Hidden size | 640 |
| Layers | 12 |
| Attention heads | 10 (MHA) |
| Head dim | 64 |
| Intermediate size (SwiGLU) | 1600 |
| Vocab size | 16,384 |
| Positional encoding | RoPE θ=10,000 |
| Normalization | RMSNorm, ε=10⁻⁶ |
| Tied embeddings | Yes |
---
## Training data details
### Training phases
| Phase | Total tokens | Data focus |
| ----- | ----- | ----- |
| Warmup + Stable | 80B | General knowledge and code |
| Decay | 20B | Synthetic, Q&A, math, behavior and textbooks |
### Data
| Source | Approx. tokens | Phase |
| ----- | ----- | ----- |
| `Ultra-FineWeb-en` | 44B | Warmup + Stable |
| `dclm_100BT-shuffled` | 28B | Warmup + Stable |
| `UltraData-Code-L2` (70% `py` / 30% `js`) | 8B | Warmup + Stable |
| Mix of Ultra-FineWeb-en, Finephrase, TxT360-v2, Dolma3-Dolmino-Mix-100B-1025, UltraData-Math and IFM/Pretrain-Behaviors | 20B | Decay |
---
## Training Procedure
| Setting | Value |
| ----- | ----- |
| Optimizer | AdamW (fused), β₁=0.9, β₂=0.95, ε=10⁻⁸ |
| Peak learning rate | 3.8e-4 |
| LR schedule | WSD |
| Micro batch size per device | 256 |
| Gradient accumulation | 1 |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Mixed precision | BF16 \+ TF32 |
| Hardware | 2× RTX A6000 48GB NVLink |
| Training optimizations | Liger Kernel, DDP bucketing (10MB), gradient\_as\_bucket\_view |
---
## What makes this model good
### Higher quality data
Not just only FineWeb-Edu, but: high quality webdata, code, math, synthetic data, reasoning, Q&A and more...
### A new decay
We tried decaying the last 20B tokens just with the same data mix as in the warmup & stable phase. We found out, that this procedure was not leading to good benchmark and overall results.
So we tried to push our model even further and changed the data to a mix of Q&A, synthetic, math, reasoning data and much more!
And it increased the OpenSLM Leaderboard INT Index from 19 to 21!
### WSD Scheduler instead of cosine decay
We applied a WSD-Scheduler instead of using boring ol' cosine decay. And it worked. The model learned much more, leading to an end loss of around 2.1
---
## Usage
> Note: this is a base model, not an instruction-tuned variant! Keep that in mind.
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "SupraLabs/Supra2.5-Pro-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model.eval()
prompt = "The future of artificial intelligence"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
out = model.generate(
**inputs,
max_new_tokens=384,
do_sample=True,
temperature=0.1,
top_p=0.8,
top_k=5,
no_repeat_ngram_size=3,
repetition_penalty=1.2,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```
---
## Intended use & limitations
Use the model for:
- finetuning
- experiments
- research
- inspiration for your next project
- ablation studies
But keep in mind, that the model has limitations:
- it's very small so we cannot guarantee fact correctness
- the model is a base model, so it cannot really "answer" questions
- it currently does English-only very well; other languages may be supported and we already saw the model write e.g. German, French and Spanish; but the models' main language is English.
- it was trained on webdata so it may have a bias
---
## What's next
At SupraLabs, we are heavily developing the next architectures, improving our models and hoping to give you the next best model soon!
Give us a follow and a like on Hugging Face if you want to support our work.
---
*© SupraLabs 2026*