LH-Tech-AI commited on
Commit
62736d2
·
verified ·
1 Parent(s): 44dfc8e

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +106 -6
README.md CHANGED
@@ -85,6 +85,8 @@ Now with 65M parameters, better data and improved training processes, Supra2.5-P
85
 
86
  ## Model details
87
 
 
 
88
  | | |
89
  | ----- | ----- |
90
  | **Developed by** | SupraLabs |
@@ -96,10 +98,11 @@ Now with 65M parameters, better data and improved training processes, Supra2.5-P
96
  | **Precision** | bfloat16 |
97
  | **License** | Apache 2.0 |
98
 
99
- ### **Architecture**
100
 
101
  | Hyperparameter | Value |
102
  | ----- | ----- |
 
103
  | Hidden size | 640 |
104
  | Layers | 12 |
105
  | Attention heads | 10 (MHA) |
@@ -109,8 +112,6 @@ Now with 65M parameters, better data and improved training processes, Supra2.5-P
109
  | Positional encoding | RoPE θ=10,000 |
110
  | Normalization | RMSNorm, ε=10⁻⁶ |
111
  | Tied embeddings | Yes |
112
- | Attention implementation | SDPA |
113
- | Architecture | `Qwen3ForCausalLM` |
114
 
115
  ---
116
 
@@ -126,10 +127,109 @@ Now with 65M parameters, better data and improved training processes, Supra2.5-P
126
  ### Data
127
 
128
  | Source | Approx. tokens | Phase |
129
- | ----- | ----- |
130
  | `Ultra-FineWeb-en` | 44B | Warmup + Stable |
131
  | `dclm_100BT-shuffled` | 28B | Warmup + Stable |
132
- | `UltraData-Code-L2` (70% py / 30% js) | 8B | Warmup + Stable |
133
  | Mix of Ultra-FineWeb-en, Finephrase, TxT360-v2, Dolma3-Dolmino-Mix-100B-1025, UltraData-Math and IFM/Pretrain-Behaviors | 20B | Decay |
134
 
135
- ##
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
85
 
86
  ## Model details
87
 
88
+ ### Basic information
89
+
90
  | | |
91
  | ----- | ----- |
92
  | **Developed by** | SupraLabs |
 
98
  | **Precision** | bfloat16 |
99
  | **License** | Apache 2.0 |
100
 
101
+ ### Architecture details
102
 
103
  | Hyperparameter | Value |
104
  | ----- | ----- |
105
+ | Architecture | `Qwen3ForCausalLM` |
106
  | Hidden size | 640 |
107
  | Layers | 12 |
108
  | Attention heads | 10 (MHA) |
 
112
  | Positional encoding | RoPE θ=10,000 |
113
  | Normalization | RMSNorm, ε=10⁻⁶ |
114
  | Tied embeddings | Yes |
 
 
115
 
116
  ---
117
 
 
127
  ### Data
128
 
129
  | Source | Approx. tokens | Phase |
130
+ | ----- | ----- | ----- |
131
  | `Ultra-FineWeb-en` | 44B | Warmup + Stable |
132
  | `dclm_100BT-shuffled` | 28B | Warmup + Stable |
133
+ | `UltraData-Code-L2` (70% `py` / 30% `js`) | 8B | Warmup + Stable |
134
  | Mix of Ultra-FineWeb-en, Finephrase, TxT360-v2, Dolma3-Dolmino-Mix-100B-1025, UltraData-Math and IFM/Pretrain-Behaviors | 20B | Decay |
135
 
136
+ ---
137
+
138
+ ## Training Procedure
139
+
140
+ | Setting | Value |
141
+ | ----- | ----- |
142
+ | Optimizer | AdamW (fused), β₁=0.9, β₂=0.95, ε=10⁻⁸ |
143
+ | Peak learning rate | 3.8e-4 |
144
+ | LR schedule | WSD |
145
+ | Micro batch size per device | 256 |
146
+ | Gradient accumulation | 1 |
147
+ | Weight decay | 0.1 |
148
+ | Gradient clipping | 1.0 |
149
+ | Mixed precision | BF16 \+ TF32 |
150
+ | Hardware | 2× RTX A6000 48GB NVLink |
151
+ | Training optimizations | Liger Kernel, DDP bucketing (10MB), gradient\_as\_bucket\_view |
152
+
153
+ ---
154
+
155
+ ## What makes this model good
156
+
157
+ ### Higher quality data
158
+
159
+ Not just only FineWeb-Edu, but: high quality webdata, code, math, synthetic data, reasoning, Q&A and more...
160
+
161
+ ### A new decay
162
+
163
+ We tried decaying the last 20B tokens just with the same data mix as in the warmup & stable phase. We found out, that this procedure was not leading to good benchmark and overall results.
164
+ So we tried to push our model even further and changed the data to a mix of Q&A, synthetic, math, reasoning data and much more!
165
+ And it increased the OpenSLM Leaderboard INT Index from 19 to 21!
166
+
167
+ ### WSD Scheduler instead of cosine decay
168
+
169
+ We applied a WSD-Scheduler instead of using boring ol' cosine decay. And it worked. The model learned much more, leading to an end loss of around 2.1
170
+
171
+ ---
172
+
173
+ ## Usage
174
+
175
+ > Note: this is a base model, not an instruction-tuned variant! Keep that in mind.
176
+
177
+ ```python
178
+ import torch
179
+ from transformers import AutoModelForCausalLM, AutoTokenizer
180
+
181
+ model_id = "SupraLabs/Supra2.5-Pro-Base"
182
+
183
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
184
+ model = AutoModelForCausalLM.from_pretrained(
185
+ model_id,
186
+ torch_dtype=torch.bfloat16,
187
+ device_map="auto",
188
+ )
189
+ model.eval()
190
+
191
+ prompt = "The future of artificial intelligence"
192
+ inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
193
+
194
+ with torch.no_grad():
195
+ out = model.generate(
196
+ **inputs,
197
+ max_new_tokens=384,
198
+ do_sample=True,
199
+ temperature=0.1,
200
+ top_p=0.8,
201
+ top_k=5,
202
+ no_repeat_ngram_size=3,
203
+ repetition_penalty=1.2,
204
+ )
205
+
206
+ print(tokenizer.decode(out[0], skip_special_tokens=True))
207
+ ```
208
+
209
+ ---
210
+
211
+ ## Intended use & limitations
212
+
213
+ Use the model for:
214
+ - finetuning
215
+ - experiments
216
+ - research
217
+ - inspiration for your next project
218
+ - ablation studies
219
+
220
+ But keep in mind, that the model has limitations:
221
+ - it's very small so we cannot guarantee fact correctness
222
+ - the model is a base model, so it cannot really "answer" questions
223
+ - it currently does English-only very well; other languages may be supported and we already saw the model write e.g. German, French and Spanish; but the models' main language is English.
224
+ - it was trained on webdata so it may have a bias
225
+
226
+ ---
227
+
228
+ ## What's next
229
+
230
+ At SupraLabs, we are heavily developing the next architectures, improving our models and hoping to give you the next best model soon!
231
+ Give us a follow and a like on Hugging Face if you want to support our work.
232
+
233
+ ---
234
+
235
+ *© SupraLabs 2026*