Text Generation
Transformers
Safetensors
English
qwen3
small
tiny
supra
supra2-5
supra2.5
efficient
conversational
text-generation-inference
Instructions to use SupraLabs/Supra2.5-Pro-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use SupraLabs/Supra2.5-Pro-Base with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="SupraLabs/Supra2.5-Pro-Base") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("SupraLabs/Supra2.5-Pro-Base") model = AutoModelForCausalLM.from_pretrained("SupraLabs/Supra2.5-Pro-Base", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use SupraLabs/Supra2.5-Pro-Base with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "SupraLabs/Supra2.5-Pro-Base" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "SupraLabs/Supra2.5-Pro-Base", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/SupraLabs/Supra2.5-Pro-Base
- SGLang
How to use SupraLabs/Supra2.5-Pro-Base with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "SupraLabs/Supra2.5-Pro-Base" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "SupraLabs/Supra2.5-Pro-Base", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "SupraLabs/Supra2.5-Pro-Base" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "SupraLabs/Supra2.5-Pro-Base", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use SupraLabs/Supra2.5-Pro-Base with Docker Model Runner:
docker model run hf.co/SupraLabs/Supra2.5-Pro-Base
|
Download README.md from SupraLabs/Supra2.5-Pro-Base: direct link, hf CLI and curl.
- Browser
- Download file 8.47 kB
-
https://huggingface.co/SupraLabs/Supra2.5-Pro-Base/resolve/main/README.md
- Command line
-
hf download hf://SupraLabs/Supra2.5-Pro-Base/README.md
-
curl -L -o README.md https://huggingface.co/SupraLabs/Supra2.5-Pro-Base/resolve/main/README.md
8.47 kB
| license: apache-2.0 | |
| viewer: false | |
| language: | |
| - en | |
| pipeline_tag: text-generation | |
| library_name: transformers | |
| tags: | |
| - small | |
| - tiny | |
| - supra | |
| - supra2-5 | |
| - supra2.5 | |
| - qwen3 | |
| - efficient | |
| <h1 align="center">Supra2.5-Pro Base</h1> | |
| <p align="center"> | |
| Ultra-efficient base model • 65M Parameters • Next-generation Supra | |
| </p> | |
|  | |
| **Supra2.5-Pro Base** is a 65M-parameter small language model pretrained from scratch by **SupraLabs** on 100B tokens of high quality web data. It uses the **Qwen3** architecture with a custom 16,384-token tokenizer. | |
| This is a **base model**. It has *not* been instruction-tuned, chat-tuned, or aligned in any way. | |
| Now with 65M parameters, better data and improved training processes, Supra2.5-Pro Base shows a significant improvement over previous generations of Supra models (e.g. Supra-50M and Supra2-100M). | |
| --- | |
| ## First things first - Benchmarks | |
|  | |
| <div style="overflow-x: auto;"> | |
| <table style="width: 100%; border-collapse: collapse; text-align: right; font-family: sans-serif;"> | |
| <thead> | |
| <tr style="border-bottom: 1px solid #d0d7de;"> | |
| <th style="text-align: left; padding: 8px 12px;"></th> | |
| <th style="text-align: left; padding: 8px 12px;">ARC-Easy</th> | |
| <th style="text-align: left; padding: 8px 12px;">ARC-Challenge</th> | |
| <th style="text-align: left; padding: 8px 12px;">PIQA</th> | |
| <th style="text-align: left; padding: 8px 12px;">HellaSwag</th> | |
| <th style="text-align: left; padding: 8px 12px; border-right: 2px solid #000000;">ArithMark-3</th> | |
| <th style="text-align: left; padding: 8px 12px;">Average</th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr style="border-bottom: 1px solid #d0d7de; color: #000000;"> | |
| <td style="text-align: left; padding: 8px 12px; font-weight: bold; white-space: nowrap;">Supra2.5-65M</td> | |
| <td style="padding: 8px 12px; background-color: #a4c2f4; font-weight: bold;">50.80%</td> | |
| <td style="padding: 8px 12px; background-color: #a4c2f4; font-weight: bold;">25.77%</td> | |
| <td style="padding: 8px 12px; background-color: #c9daf8;">63.11%</td> | |
| <td style="padding: 8px 12px; background-color: #c9daf8;">34.35%</td> | |
| <td style="padding: 8px 12px; background-color: #a4c2f4; font-weight: bold; border-right: 2px solid #000000;">49.60%</td> | |
| <td style="padding: 8px 12px; background-color: #a4c2f4; font-weight: bold;">44.73%</td> | |
| </tr> | |
| <tr style="border-bottom: 1px solid #d0d7de; color: #000000;"> | |
| <td style="text-align: left; padding: 8px 12px; font-weight: bold; white-space: nowrap;">Supra2-100M</td> | |
| <td style="padding: 8px 12px; background-color: #c9daf8;">47.81%</td> | |
| <td style="padding: 8px 12px; background-color: #ffffff;">24.83%</td> | |
| <td style="padding: 8px 12px; background-color: #a4c2f4; font-weight: bold;">65.40%</td> | |
| <td style="padding: 8px 12px; background-color: #a4c2f4; font-weight: bold;">35.98%</td> | |
| <td style="padding: 8px 12px; background-color: #c9daf8; border-right: 2px solid #000000;">36.90%</td> | |
| <td style="padding: 8px 12px; background-color: #c9daf8;">42.18%</td> | |
| </tr> | |
| <tr style="border-bottom: 1px solid #d0d7de; color: #000000;"> | |
| <td style="text-align: left; padding: 8px 12px; font-weight: bold; white-space: nowrap;">Supra-50M</td> | |
| <td style="padding: 8px 12px; background-color: #ffffff;">46.00%</td> | |
| <td style="padding: 8px 12px; background-color: #c9daf8;">25.17%</td> | |
| <td style="padding: 8px 12px; background-color: #ffffff;">61.19%</td> | |
| <td style="padding: 8px 12px; background-color: #ffffff;">31.72%</td> | |
| <td style="padding: 8px 12px; background-color: #ffffff; border-right: 2px solid #000000;">36.30%</td> | |
| <td style="padding: 8px 12px; background-color: #ffffff;">40.08%</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| --- | |
| ## Model details | |
| ### Basic information | |
| | | | | |
| | ----- | ----- | | |
| | **Developed by** | SupraLabs | | |
| | **Model type** | Causal decoder-only transformer (Qwen3) | | |
| | **Language** | English (parts of the data contained other languages like German, French, Spanish, and others too but it wasn't learned pretty well) | | |
| | **Parameters** | ~65M | | |
| | **Training tokens** | 100B | | |
| | **Context length** | 512 tokens | | |
| | **Precision** | bfloat16 | | |
| | **License** | Apache 2.0 | | |
| ### Architecture details | |
| | Hyperparameter | Value | | |
| | ----- | ----- | | |
| | Architecture | `Qwen3ForCausalLM` | | |
| | Hidden size | 640 | | |
| | Layers | 12 | | |
| | Attention heads | 10 (MHA) | | |
| | Head dim | 64 | | |
| | Intermediate size (SwiGLU) | 1600 | | |
| | Vocab size | 16,384 | | |
| | Positional encoding | RoPE θ=10,000 | | |
| | Normalization | RMSNorm, ε=10⁻⁶ | | |
| | Tied embeddings | Yes | | |
| --- | |
| ## Training data details | |
| ### Training phases | |
| | Phase | Total tokens | Data focus | | |
| | ----- | ----- | ----- | | |
| | Warmup + Stable | 80B | General knowledge and code | | |
| | Decay | 20B | Synthetic, Q&A, math, behavior and textbooks | | |
| ### Data | |
| | Source | Approx. tokens | Phase | | |
| | ----- | ----- | ----- | | |
| | `Ultra-FineWeb-en` | 44B | Warmup + Stable | | |
| | `dclm_100BT-shuffled` | 28B | Warmup + Stable | | |
| | `UltraData-Code-L2` (70% `py` / 30% `js`) | 8B | Warmup + Stable | | |
| | Mix of Ultra-FineWeb-en, Finephrase, TxT360-v2, Dolma3-Dolmino-Mix-100B-1025, UltraData-Math and IFM/Pretrain-Behaviors | 20B | Decay | | |
| --- | |
| ## Training Procedure | |
| | Setting | Value | | |
| | ----- | ----- | | |
| | Optimizer | AdamW (fused), β₁=0.9, β₂=0.95, ε=10⁻⁸ | | |
| | Peak learning rate | 3.8e-4 | | |
| | LR schedule | WSD | | |
| | Micro batch size per device | 256 | | |
| | Gradient accumulation | 1 | | |
| | Weight decay | 0.1 | | |
| | Gradient clipping | 1.0 | | |
| | Mixed precision | BF16 \+ TF32 | | |
| | Hardware | 2× RTX A6000 48GB NVLink | | |
| | Training optimizations | Liger Kernel, DDP bucketing (10MB), gradient\_as\_bucket\_view | | |
| --- | |
| ## What makes this model good | |
| ### Higher quality data | |
| Not just only FineWeb-Edu, but: high quality webdata, code, math, synthetic data, reasoning, Q&A and more... | |
| ### A new decay | |
| We tried decaying the last 20B tokens just with the same data mix as in the warmup & stable phase. We found out, that this procedure was not leading to good benchmark and overall results. | |
| So we tried to push our model even further and changed the data to a mix of Q&A, synthetic, math, reasoning data and much more! | |
| And it increased the OpenSLM Leaderboard INT Index from 19 to 21! | |
| ### WSD Scheduler instead of cosine decay | |
| We applied a WSD-Scheduler instead of using boring ol' cosine decay. And it worked. The model learned much more, leading to an end loss of around 2.1 | |
| --- | |
| ## Usage | |
| > Note: this is a base model, not an instruction-tuned variant! Keep that in mind. | |
| ```python | |
| import torch | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| model_id = "SupraLabs/Supra2.5-Pro-Base" | |
| tokenizer = AutoTokenizer.from_pretrained(model_id) | |
| model = AutoModelForCausalLM.from_pretrained( | |
| model_id, | |
| torch_dtype=torch.bfloat16, | |
| device_map="auto", | |
| ) | |
| model.eval() | |
| prompt = "The future of artificial intelligence" | |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) | |
| with torch.no_grad(): | |
| out = model.generate( | |
| **inputs, | |
| max_new_tokens=384, | |
| do_sample=True, | |
| temperature=0.1, | |
| top_p=0.8, | |
| top_k=5, | |
| no_repeat_ngram_size=3, | |
| repetition_penalty=1.2, | |
| ) | |
| print(tokenizer.decode(out[0], skip_special_tokens=True)) | |
| ``` | |
| --- | |
| ## Intended use & limitations | |
| Use the model for: | |
| - finetuning | |
| - experiments | |
| - research | |
| - inspiration for your next project | |
| - ablation studies | |
| But keep in mind, that the model has limitations: | |
| - it's very small so we cannot guarantee fact correctness | |
| - the model is a base model, so it cannot really "answer" questions | |
| - it currently does English-only very well; other languages may be supported and we already saw the model write e.g. German, French and Spanish; but the models' main language is English. | |
| - it was trained on webdata so it may have a bias | |
| --- | |
| ## What's next | |
| At SupraLabs, we are heavily developing the next architectures, improving our models and hoping to give you the next best model soon! | |
| Give us a follow and a like on Hugging Face if you want to support our work. | |
| --- | |
| *© SupraLabs 2026* |