LH-Tech-AI commited on
Commit
44dfc8e
·
verified ·
1 Parent(s): 9b16f18

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +132 -0
README.md CHANGED
@@ -1,3 +1,135 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ viewer: false
4
+ language:
5
+ - en
6
+ pipeline_tag: text-generation
7
+ library_name: transformers
8
+ tags:
9
+ - small
10
+ - tiny
11
+ - supra
12
+ - supra2-5
13
+ - supra2.5
14
+ - qwen3
15
+ - efficient
16
  ---
17
+
18
+ <h1 align="center">Supra2.5-Pro (Base)</h1>
19
+
20
+ <p align="center">
21
+ Ultra-efficient base model • 65M Parameters • Next-generation Supra
22
+ </p>
23
+
24
+ ![banner](https://cdn-uploads.huggingface.co/production/uploads/697f2832c2c5e4daa93cece7/45USn-GgQibSeumtVsUBo.png)
25
+
26
+ **Supra2.5-Pro Base** is a 65M-parameter small language model pretrained from scratch by **SupraLabs** on 100B tokens of high quality web data. It uses the **Qwen3** architecture with a custom 16,384-token tokenizer.
27
+
28
+ This is a **base model**. It has *not* been instruction-tuned, chat-tuned, or aligned in any way.
29
+
30
+ Now with 65M parameters, better data and improved training processes, Supra2.5-Pro Base shows a significant improvement over previous generations of Supra models (e.g. Supra-50M and Supra2-100M).
31
+
32
+ ---
33
+
34
+ ## First things first - Benchmarks
35
+
36
+
37
+ ![benchmarks_chart](https://cdn-uploads.huggingface.co/production/uploads/697f2832c2c5e4daa93cece7/_3wmUNwARYIM8wXu_Q3YW.png)
38
+
39
+ <div style="overflow-x: auto;">
40
+ <table style="width: 100%; border-collapse: collapse; text-align: right; font-family: sans-serif;">
41
+ <thead>
42
+ <tr style="border-bottom: 1px solid #d0d7de;">
43
+ <th style="text-align: left; padding: 8px 12px;"></th>
44
+ <th style="text-align: left; padding: 8px 12px;">ARC-Easy</th>
45
+ <th style="text-align: left; padding: 8px 12px;">ARC-Challenge</th>
46
+ <th style="text-align: left; padding: 8px 12px;">PIQA</th>
47
+ <th style="text-align: left; padding: 8px 12px;">HellaSwag</th>
48
+ <th style="text-align: left; padding: 8px 12px; border-right: 3px solid #000000;">ArithMark-3</th>
49
+ <th style="text-align: left; padding: 8px 12px;">Average</th>
50
+ </tr>
51
+ </thead>
52
+ <tbody>
53
+ <tr style="border-bottom: 1px solid #d0d7de; color: #000000;">
54
+ <td style="text-align: left; padding: 8px 12px; font-weight: bold; white-space: nowrap;">Supra2.5-65M</td>
55
+ <td style="padding: 8px 12px; background-color: #a4c2f4; font-weight: bold;">50.80%</td>
56
+ <td style="padding: 8px 12px; background-color: #a4c2f4; font-weight: bold;">25.77%</td>
57
+ <td style="padding: 8px 12px; background-color: #c9daf8;">63.11%</td>
58
+ <td style="padding: 8px 12px; background-color: #c9daf8;">34.35%</td>
59
+ <td style="padding: 8px 12px; background-color: #a4c2f4; font-weight: bold; border-right: 2px solid #000000;">49.60%</td>
60
+ <td style="padding: 8px 12px; background-color: #a4c2f4; font-weight: bold;">44.73%</td>
61
+ </tr>
62
+ <tr style="border-bottom: 1px solid #d0d7de; color: #000000;">
63
+ <td style="text-align: left; padding: 8px 12px; font-weight: bold; white-space: nowrap;">Supra2-100M</td>
64
+ <td style="padding: 8px 12px; background-color: #c9daf8;">47.81%</td>
65
+ <td style="padding: 8px 12px; background-color: #ffffff;">24.83%</td>
66
+ <td style="padding: 8px 12px; background-color: #a4c2f4; font-weight: bold;">65.40%</td>
67
+ <td style="padding: 8px 12px; background-color: #a4c2f4; font-weight: bold;">35.98%</td>
68
+ <td style="padding: 8px 12px; background-color: #c9daf8; border-right: 2px solid #000000;">36.90%</td>
69
+ <td style="padding: 8px 12px; background-color: #c9daf8;">42.18%</td>
70
+ </tr>
71
+ <tr style="border-bottom: 1px solid #d0d7de; color: #000000;">
72
+ <td style="text-align: left; padding: 8px 12px; font-weight: bold; white-space: nowrap;">Supra-50M</td>
73
+ <td style="padding: 8px 12px; background-color: #ffffff;">46.00%</td>
74
+ <td style="padding: 8px 12px; background-color: #c9daf8;">25.17%</td>
75
+ <td style="padding: 8px 12px; background-color: #ffffff;">61.19%</td>
76
+ <td style="padding: 8px 12px; background-color: #ffffff;">31.72%</td>
77
+ <td style="padding: 8px 12px; background-color: #ffffff; border-right: 2px solid #000000;">36.30%</td>
78
+ <td style="padding: 8px 12px; background-color: #ffffff;">40.08%</td>
79
+ </tr>
80
+ </tbody>
81
+ </table>
82
+ </div>
83
+
84
+ ---
85
+
86
+ ## Model details
87
+
88
+ | | |
89
+ | ----- | ----- |
90
+ | **Developed by** | SupraLabs |
91
+ | **Model type** | Causal decoder-only transformer (Qwen3) |
92
+ | **Language** | English (parts of the data contained other languages like German, French, Spanish, and others too but it wasn't learned pretty well) |
93
+ | **Parameters** | ~65M |
94
+ | **Training tokens** | 100B |
95
+ | **Context length** | 512 tokens |
96
+ | **Precision** | bfloat16 |
97
+ | **License** | Apache 2.0 |
98
+
99
+ ### **Architecture**
100
+
101
+ | Hyperparameter | Value |
102
+ | ----- | ----- |
103
+ | Hidden size | 640 |
104
+ | Layers | 12 |
105
+ | Attention heads | 10 (MHA) |
106
+ | Head dim | 64 |
107
+ | Intermediate size (SwiGLU) | 1600 |
108
+ | Vocab size | 16,384 |
109
+ | Positional encoding | RoPE θ=10,000 |
110
+ | Normalization | RMSNorm, ε=10⁻⁶ |
111
+ | Tied embeddings | Yes |
112
+ | Attention implementation | SDPA |
113
+ | Architecture | `Qwen3ForCausalLM` |
114
+
115
+ ---
116
+
117
+ ## Training data details
118
+
119
+ ### Training phases
120
+
121
+ | Phase | Total tokens | Data focus |
122
+ | ----- | ----- | ----- |
123
+ | Warmup + Stable | 80B | General knowledge and code |
124
+ | Decay | 20B | Synthetic, Q&A, math, behavior and textbooks |
125
+
126
+ ### Data
127
+
128
+ | Source | Approx. tokens | Phase |
129
+ | ----- | ----- |
130
+ | `Ultra-FineWeb-en` | 44B | Warmup + Stable |
131
+ | `dclm_100BT-shuffled` | 28B | Warmup + Stable |
132
+ | `UltraData-Code-L2` (70% py / 30% js) | 8B | Warmup + Stable |
133
+ | Mix of Ultra-FineWeb-en, Finephrase, TxT360-v2, Dolma3-Dolmino-Mix-100B-1025, UltraData-Math and IFM/Pretrain-Behaviors | 20B | Decay |
134
+
135
+ ##