Instructions to use SupraLabs/Supra2.5-Pro-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use SupraLabs/Supra2.5-Pro-Base with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="SupraLabs/Supra2.5-Pro-Base") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("SupraLabs/Supra2.5-Pro-Base") model = AutoModelForCausalLM.from_pretrained("SupraLabs/Supra2.5-Pro-Base", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use SupraLabs/Supra2.5-Pro-Base with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "SupraLabs/Supra2.5-Pro-Base" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "SupraLabs/Supra2.5-Pro-Base", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/SupraLabs/Supra2.5-Pro-Base
- SGLang
How to use SupraLabs/Supra2.5-Pro-Base with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "SupraLabs/Supra2.5-Pro-Base" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "SupraLabs/Supra2.5-Pro-Base", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "SupraLabs/Supra2.5-Pro-Base" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "SupraLabs/Supra2.5-Pro-Base", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use SupraLabs/Supra2.5-Pro-Base with Docker Model Runner:
docker model run hf.co/SupraLabs/Supra2.5-Pro-Base
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("SupraLabs/Supra2.5-Pro-Base")
model = AutoModelForCausalLM.from_pretrained("SupraLabs/Supra2.5-Pro-Base", device_map="auto")
messages = [
{"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))Supra2.5-Pro Base
Ultra-efficient base model • 65M Parameters • Next-generation Supra
Supra2.5-Pro Base is a 65M-parameter small language model pretrained from scratch by SupraLabs on 100B tokens of high quality web data. It uses the Qwen3 architecture with a custom 16,384-token tokenizer.
This is a base model. It has not been instruction-tuned, chat-tuned, or aligned in any way.
Now with 65M parameters, better data and improved training processes, Supra2.5-Pro Base shows a significant improvement over previous generations of Supra models (e.g. Supra-50M and Supra2-100M).
First things first - Benchmarks
| ARC-Easy | ARC-Challenge | PIQA | HellaSwag | ArithMark-3 | Average | |
|---|---|---|---|---|---|---|
| Supra2.5-65M | 50.80% | 25.77% | 63.11% | 34.35% | 49.60% | 44.73% |
| Supra2-100M | 47.81% | 24.83% | 65.40% | 35.98% | 36.90% | 42.18% |
| Supra-50M | 46.00% | 25.17% | 61.19% | 31.72% | 36.30% | 40.08% |
Model details
Basic information
| Developed by | SupraLabs |
| Model type | Causal decoder-only transformer (Qwen3) |
| Language | English (parts of the data contained other languages like German, French, Spanish, and others too but it wasn't learned pretty well) |
| Parameters | ~65M |
| Training tokens | 100B |
| Context length | 512 tokens |
| Precision | bfloat16 |
| License | Apache 2.0 |
Architecture details
| Hyperparameter | Value |
|---|---|
| Architecture | Qwen3ForCausalLM |
| Hidden size | 640 |
| Layers | 12 |
| Attention heads | 10 (MHA) |
| Head dim | 64 |
| Intermediate size (SwiGLU) | 1600 |
| Vocab size | 16,384 |
| Positional encoding | RoPE θ=10,000 |
| Normalization | RMSNorm, ε=10⁻⁶ |
| Tied embeddings | Yes |
Training data details
Training phases
| Phase | Total tokens | Data focus |
|---|---|---|
| Warmup + Stable | 80B | General knowledge and code |
| Decay | 20B | Synthetic, Q&A, math, behavior and textbooks |
Data
| Source | Approx. tokens | Phase |
|---|---|---|
Ultra-FineWeb-en |
44B | Warmup + Stable |
dclm_100BT-shuffled |
28B | Warmup + Stable |
UltraData-Code-L2 (70% py / 30% js) |
8B | Warmup + Stable |
| Mix of Ultra-FineWeb-en, Finephrase, TxT360-v2, Dolma3-Dolmino-Mix-100B-1025, UltraData-Math and IFM/Pretrain-Behaviors | 20B | Decay |
Training Procedure
| Setting | Value |
|---|---|
| Optimizer | AdamW (fused), β₁=0.9, β₂=0.95, ε=10⁻⁸ |
| Peak learning rate | 3.8e-4 |
| LR schedule | WSD |
| Micro batch size per device | 256 |
| Gradient accumulation | 1 |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Mixed precision | BF16 + TF32 |
| Hardware | 2× RTX A6000 48GB NVLink |
| Training optimizations | Liger Kernel, DDP bucketing (10MB), gradient_as_bucket_view |
What makes this model good
Higher quality data
Not just only FineWeb-Edu, but: high quality webdata, code, math, synthetic data, reasoning, Q&A and more...
A new decay
We tried decaying the last 20B tokens just with the same data mix as in the warmup & stable phase. We found out, that this procedure was not leading to good benchmark and overall results. So we tried to push our model even further and changed the data to a mix of Q&A, synthetic, math, reasoning data and much more! And it increased the OpenSLM Leaderboard INT Index from 19 to 21!
WSD Scheduler instead of cosine decay
We applied a WSD-Scheduler instead of using boring ol' cosine decay. And it worked. The model learned much more, leading to an end loss of around 2.1
Usage
Note: this is a base model, not an instruction-tuned variant! Keep that in mind.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "SupraLabs/Supra2.5-Pro-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model.eval()
prompt = "The future of artificial intelligence"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
out = model.generate(
**inputs,
max_new_tokens=384,
do_sample=True,
temperature=0.1,
top_p=0.8,
top_k=5,
no_repeat_ngram_size=3,
repetition_penalty=1.2,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))
Intended use & limitations
Use the model for:
- finetuning
- experiments
- research
- inspiration for your next project
- ablation studies
But keep in mind, that the model has limitations:
- it's very small so we cannot guarantee fact correctness
- the model is a base model, so it cannot really "answer" questions
- it currently does English-only very well; other languages may be supported and we already saw the model write e.g. German, French and Spanish; but the models' main language is English.
- it was trained on webdata so it may have a bias
What's next
At SupraLabs, we are heavily developing the next architectures, improving our models and hoping to give you the next best model soon! Give us a follow and a like on Hugging Face if you want to support our work.
© SupraLabs 2026
- Downloads last month
- -


# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="SupraLabs/Supra2.5-Pro-Base") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)