ZirTech commited on
Commit
d268f61
·
verified ·
1 Parent(s): 37eb3dc

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +169 -5
README.md CHANGED
@@ -1,5 +1,169 @@
1
- ---
2
- license: other
3
- license_name: ztech-license
4
- license_link: LICENSE
5
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: ztech-license
4
+ license_link: LICENSE
5
+ language:
6
+ - en
7
+ pipeline_tag: text-generation
8
+ ---
9
+
10
+ <div align="center">
11
+
12
+ ![image](https://cdn-uploads.huggingface.co/production/uploads/69b3129f02a20db8381db62e/C-dB32vMZ0R7tOjWdHC9S.png)
13
+
14
+ </div>
15
+
16
+
17
+ # 🧮 OmniMath-2B
18
+
19
+ **OmniMath-2B-Pro** is an improved and compact model capable of strong mathematical analysis, debugged based on the **Qwen3.5‑2B** hybrid architecture (Gated Delta Networks interleaved with standard attention). Trained on the basis of **100,000+** Carefully selected mathematical problems from various datasets, it does an excellent job with step-by-step solutions, problems with arithmetic words, geometric reasoning and error recovery.
20
+
21
+ The main feature of **OmniMath-2B-Pro** is its ability to solve **Olympiad tasks**.
22
+
23
+ Unlike the previous version, the new **OmniMath Pro** shows the **best** results on mathematical benchmarks.
24
+
25
+ Despite its small size, **OmniMath-2B-Pro** demonstrates high performance and is ideal for resource-constrained and advanced deployment environments.
26
+
27
+ ---
28
+
29
+ ## ✨ Key Features
30
+
31
+ - **Efficient 2B Scale** : Only 2 billion parameters – runs smoothly on a single T4 GPU or even CPU with quantization.
32
+ - **Step‑by‑Step Reasoning** : Trained with explicit `<think>...</think>`‑style chain‑of‑thought prompts.
33
+ - **Hybrid Architecture** : Inherits Qwen3.5's Gated Delta Networks for efficient long‑context processing.
34
+
35
+ ---
36
+
37
+ ## 📊 Benchmarks
38
+
39
+ *Preliminary results (evaluation ongoing).*
40
+
41
+ | Model | Size (params) | GSM8K Accuracy |
42
+ |-------|---------------|----------------|
43
+ | Qwen2.5-Math-1.5B | 1.5B | 54% |
44
+ | Phi-2 (0-shot CoT) | 2.7B | 50.0% |
45
+ | **OmniMath-2B** | **2B** | **???%** |
46
+ | dolphin-2_6-phi-2 | 2.7B | 58.07% |
47
+ | Qwen2.5-0.5B-Instruct | 2.7B | 49.6% |
48
+ | gemma-3-1b-it | 1.1B | 62.8% |
49
+ | MobileLLM-R1.5 950M | 1B | 52.8% |
50
+ | Gemma 2 2B IT | 2B | 23.9% |
51
+
52
+ *Updates coming soon.*
53
+
54
+ ---
55
+
56
+ ## 🚀 Quickstart
57
+
58
+ ### 🤗 Transformers
59
+
60
+ ```python
61
+ from transformers import AutoModelForCausalLM, AutoTokenizer
62
+
63
+ model_id = "ZirTech/OmniMath-2B-Pro"
64
+
65
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
66
+ model = AutoModelForCausalLM.from_pretrained(
67
+ model_id,
68
+ torch_dtype="auto",
69
+ device_map="auto"
70
+ )
71
+
72
+ messages = [
73
+ {"role": "system", "content": "You are a helpful math assistant. Solve problems step by step."},
74
+ {"role": "user", "content": "A store sells apples for $2 each. If you buy 5 apples, how much do you pay?"}
75
+ ]
76
+
77
+ text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
78
+ inputs = tokenizer([text], return_tensors="pt").to(model.device)
79
+
80
+ outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.6, top_p=0.95, top_k=20)
81
+ print(tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True))
82
+ ```
83
+
84
+ ---
85
+
86
+ ## ⚡ vLLM
87
+
88
+ ```
89
+ vllm serve ZirTech/OmniMath-2B-Pro --tensor-parallel-size 1 --max-model-len 4096
90
+ ```
91
+
92
+ ```python
93
+ import torch
94
+ from transformers import AutoModelForCausalLM, AutoTokenizer
95
+
96
+ model_name = "ZirTech/OmniMath-2B-Pro"
97
+ tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
98
+ model = AutoModelForCausalLM.from_pretrained(
99
+ model_name,
100
+ torch_dtype=torch.float16,
101
+ device_map="auto",
102
+ trust_remote_code=True
103
+ )
104
+ model.eval()
105
+
106
+ def ask(question):
107
+ prompt = f"<|im_start|>system\nYou are a helpful math assistant.<|im_end|>\n<|im_start|>user\n{question}<|im_end|>\n<|im_start|>assistant\n"
108
+ inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
109
+ outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.0, do_sample=False)
110
+ response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
111
+ if "user" in response:
112
+ response = response.split("user")[0].strip()
113
+ return response
114
+
115
+ print(ask("Find the degree for the given field extension Q(sqrt(2), sqrt(3), sqrt(18)) over Q. Give me the answer."))
116
+ ```
117
+
118
+ ---
119
+
120
+ ## 🏗️ Architecture
121
+
122
+ OmniMath‑2B-Pro fully preserves Qwen3.5‑2B's design:
123
+
124
+ * Gated Delta Networks : Linear attention layers interleaved with standard attention.
125
+
126
+ * 262K Native Context : Supports up to 262,144 tokens (extendable with YaRN).
127
+
128
+ * Built on Qwen3_5ForCausalLM : Seamless integration with Hugging Face ecosystem.
129
+
130
+ ---
131
+
132
+ ## ⚠️ Limitations
133
+
134
+ * Numerical accuracy may occasionally falter – always double‑check critical calculations.
135
+
136
+ * Geometry with visual elements was only trained on textual descriptions; performance on image‑based geometry is limited.
137
+
138
+ * Non‑English math problems are not thoroughly evaluated.
139
+
140
+ ---
141
+
142
+ ## 🙏 Acknowledgments
143
+
144
+ * Qwen Team for the outstanding Qwen3.5 base models.
145
+
146
+ * Hugging Face for dataset hosting and the Transformers library.
147
+
148
+ * Сommunity for liking and downloading the [OmniMath-2B](https://huggingface.co/ZirTech/OmniMath-2B) model
149
+
150
+ ---
151
+
152
+ ## 📖 Citation
153
+
154
+ ```bibtex
155
+ @misc{omnimathpro2b2026,
156
+ title={OmniMath-2B-Pro: A Strong and Lightweight Open-Source Mathematical Model},
157
+ author={Zirt Techniques},
158
+ year={2026},
159
+ url={https://huggingface.co/ZirTech/OmniMath-2B-Pro}
160
+ }
161
+ ```
162
+
163
+ ---
164
+
165
+ <div align="center">
166
+
167
+ **Built by [Zirt Tech](https://huggingface.co/ZirTech) ❤️**
168
+
169
+ </div>