How to use from
Docker Model Runner
docker model run hf.co/SupraLabs/Supra2.5-Pro-Base
Quick Links

Supra2.5-Pro Base

Ultra-efficient base model • 65M Parameters • Next-generation Supra

banner

Supra2.5-Pro Base is a 65M-parameter small language model pretrained from scratch by SupraLabs on 100B tokens of high quality web data. It uses the Qwen3 architecture with a custom 16,384-token tokenizer.

This is a base model. It has not been instruction-tuned, chat-tuned, or aligned in any way.

Now with 65M parameters, better data and improved training processes, Supra2.5-Pro Base shows a significant improvement over previous generations of Supra models (e.g. Supra-50M and Supra2-100M).


First things first - Benchmarks

benchmarks_chart

ARC-Easy ARC-Challenge PIQA HellaSwag ArithMark-3 Average
Supra2.5-65M 50.80% 25.77% 63.11% 34.35% 49.60% 44.73%
Supra2-100M 47.81% 24.83% 65.40% 35.98% 36.90% 42.18%
Supra-50M 46.00% 25.17% 61.19% 31.72% 36.30% 40.08%

Model details

Basic information

Developed by SupraLabs
Model type Causal decoder-only transformer (Qwen3)
Language English (parts of the data contained other languages like German, French, Spanish, and others too but it wasn't learned pretty well)
Parameters ~65M
Training tokens 100B
Context length 512 tokens
Precision bfloat16
License Apache 2.0

Architecture details

Hyperparameter Value
Architecture Qwen3ForCausalLM
Hidden size 640
Layers 12
Attention heads 10 (MHA)
Head dim 64
Intermediate size (SwiGLU) 1600
Vocab size 16,384
Positional encoding RoPE θ=10,000
Normalization RMSNorm, ε=10⁻⁶
Tied embeddings Yes

Training data details

Training phases

Phase Total tokens Data focus
Warmup + Stable 80B General knowledge and code
Decay 20B Synthetic, Q&A, math, behavior and textbooks

Data

Source Approx. tokens Phase
Ultra-FineWeb-en 44B Warmup + Stable
dclm_100BT-shuffled 28B Warmup + Stable
UltraData-Code-L2 (70% py / 30% js) 8B Warmup + Stable
Mix of Ultra-FineWeb-en, Finephrase, TxT360-v2, Dolma3-Dolmino-Mix-100B-1025, UltraData-Math and IFM/Pretrain-Behaviors 20B Decay

Training Procedure

Setting Value
Optimizer AdamW (fused), β₁=0.9, β₂=0.95, ε=10⁻⁸
Peak learning rate 3.8e-4
LR schedule WSD
Micro batch size per device 256
Gradient accumulation 1
Weight decay 0.1
Gradient clipping 1.0
Mixed precision BF16 + TF32
Hardware 2× RTX A6000 48GB NVLink
Training optimizations Liger Kernel, DDP bucketing (10MB), gradient_as_bucket_view

What makes this model good

Higher quality data

Not just only FineWeb-Edu, but: high quality webdata, code, math, synthetic data, reasoning, Q&A and more...

A new decay

We tried decaying the last 20B tokens just with the same data mix as in the warmup & stable phase. We found out, that this procedure was not leading to good benchmark and overall results. So we tried to push our model even further and changed the data to a mix of Q&A, synthetic, math, reasoning data and much more! And it increased the OpenSLM Leaderboard INT Index from 19 to 21!

WSD Scheduler instead of cosine decay

We applied a WSD-Scheduler instead of using boring ol' cosine decay. And it worked. The model learned much more, leading to an end loss of around 2.1


Usage

Note: this is a base model, not an instruction-tuned variant! Keep that in mind.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SupraLabs/Supra2.5-Pro-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

prompt = "The future of artificial intelligence"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=384,
        do_sample=True,
        temperature=0.1,
        top_p=0.8,
        top_k=5,
        no_repeat_ngram_size=3,
        repetition_penalty=1.2,
    )

print(tokenizer.decode(out[0], skip_special_tokens=True))

Intended use & limitations

Use the model for:

  • finetuning
  • experiments
  • research
  • inspiration for your next project
  • ablation studies

But keep in mind, that the model has limitations:

  • it's very small so we cannot guarantee fact correctness
  • the model is a base model, so it cannot really "answer" questions
  • it currently does English-only very well; other languages may be supported and we already saw the model write e.g. German, French and Spanish; but the models' main language is English.
  • it was trained on webdata so it may have a bias

What's next

At SupraLabs, we are heavily developing the next architectures, improving our models and hoping to give you the next best model soon! Give us a follow and a like on Hugging Face if you want to support our work.


© SupraLabs 2026

Downloads last month
-
Safetensors
Model size
67M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including SupraLabs/Supra2.5-Pro-Base