--- license: apache-2.0 viewer: false language: - en pipeline_tag: text-generation library_name: transformers tags: - small - tiny - supra - supra2-5 - supra2.5 - qwen3 - efficient ---

Supra2.5-Pro Base

Ultra-efficient base model • 65M Parameters • Next-generation Supra

![banner](https://cdn-uploads.huggingface.co/production/uploads/697f2832c2c5e4daa93cece7/45USn-GgQibSeumtVsUBo.png) **Supra2.5-Pro Base** is a 65M-parameter small language model pretrained from scratch by **SupraLabs** on 100B tokens of high quality web data. It uses the **Qwen3** architecture with a custom 16,384-token tokenizer. This is a **base model**. It has *not* been instruction-tuned, chat-tuned, or aligned in any way. Now with 65M parameters, better data and improved training processes, Supra2.5-Pro Base shows a significant improvement over previous generations of Supra models (e.g. Supra-50M and Supra2-100M). --- ## First things first - Benchmarks ![benchmarks_chart](https://cdn-uploads.huggingface.co/production/uploads/697f2832c2c5e4daa93cece7/_3wmUNwARYIM8wXu_Q3YW.png)
ARC-Easy ARC-Challenge PIQA HellaSwag ArithMark-3 Average
Supra2.5-65M 50.80% 25.77% 63.11% 34.35% 49.60% 44.73%
Supra2-100M 47.81% 24.83% 65.40% 35.98% 36.90% 42.18%
Supra-50M 46.00% 25.17% 61.19% 31.72% 36.30% 40.08%
--- ## Model details ### Basic information | | | | ----- | ----- | | **Developed by** | SupraLabs | | **Model type** | Causal decoder-only transformer (Qwen3) | | **Language** | English (parts of the data contained other languages like German, French, Spanish, and others too but it wasn't learned pretty well) | | **Parameters** | ~65M | | **Training tokens** | 100B | | **Context length** | 512 tokens | | **Precision** | bfloat16 | | **License** | Apache 2.0 | ### Architecture details | Hyperparameter | Value | | ----- | ----- | | Architecture | `Qwen3ForCausalLM` | | Hidden size | 640 | | Layers | 12 | | Attention heads | 10 (MHA) | | Head dim | 64 | | Intermediate size (SwiGLU) | 1600 | | Vocab size | 16,384 | | Positional encoding | RoPE θ=10,000 | | Normalization | RMSNorm, ε=10⁻⁶ | | Tied embeddings | Yes | --- ## Training data details ### Training phases | Phase | Total tokens | Data focus | | ----- | ----- | ----- | | Warmup + Stable | 80B | General knowledge and code | | Decay | 20B | Synthetic, Q&A, math, behavior and textbooks | ### Data | Source | Approx. tokens | Phase | | ----- | ----- | ----- | | `Ultra-FineWeb-en` | 44B | Warmup + Stable | | `dclm_100BT-shuffled` | 28B | Warmup + Stable | | `UltraData-Code-L2` (70% `py` / 30% `js`) | 8B | Warmup + Stable | | Mix of Ultra-FineWeb-en, Finephrase, TxT360-v2, Dolma3-Dolmino-Mix-100B-1025, UltraData-Math and IFM/Pretrain-Behaviors | 20B | Decay | --- ## Training Procedure | Setting | Value | | ----- | ----- | | Optimizer | AdamW (fused), β₁=0.9, β₂=0.95, ε=10⁻⁸ | | Peak learning rate | 3.8e-4 | | LR schedule | WSD | | Micro batch size per device | 256 | | Gradient accumulation | 1 | | Weight decay | 0.1 | | Gradient clipping | 1.0 | | Mixed precision | BF16 \+ TF32 | | Hardware | 2× RTX A6000 48GB NVLink | | Training optimizations | Liger Kernel, DDP bucketing (10MB), gradient\_as\_bucket\_view | --- ## What makes this model good ### Higher quality data Not just only FineWeb-Edu, but: high quality webdata, code, math, synthetic data, reasoning, Q&A and more... ### A new decay We tried decaying the last 20B tokens just with the same data mix as in the warmup & stable phase. We found out, that this procedure was not leading to good benchmark and overall results. So we tried to push our model even further and changed the data to a mix of Q&A, synthetic, math, reasoning data and much more! And it increased the OpenSLM Leaderboard INT Index from 19 to 21! ### WSD Scheduler instead of cosine decay We applied a WSD-Scheduler instead of using boring ol' cosine decay. And it worked. The model learned much more, leading to an end loss of around 2.1 --- ## Usage > Note: this is a base model, not an instruction-tuned variant! Keep that in mind. ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "SupraLabs/Supra2.5-Pro-Base" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto", ) model.eval() prompt = "The future of artificial intelligence" inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): out = model.generate( **inputs, max_new_tokens=384, do_sample=True, temperature=0.1, top_p=0.8, top_k=5, no_repeat_ngram_size=3, repetition_penalty=1.2, ) print(tokenizer.decode(out[0], skip_special_tokens=True)) ``` --- ## Intended use & limitations Use the model for: - finetuning - experiments - research - inspiration for your next project - ablation studies But keep in mind, that the model has limitations: - it's very small so we cannot guarantee fact correctness - the model is a base model, so it cannot really "answer" questions - it currently does English-only very well; other languages may be supported and we already saw the model write e.g. German, French and Spanish; but the models' main language is English. - it was trained on webdata so it may have a bias --- ## What's next At SupraLabs, we are heavily developing the next architectures, improving our models and hoping to give you the next best model soon! Give us a follow and a like on Hugging Face if you want to support our work. --- *© SupraLabs 2026*