ZirTech commited on
Commit
7d58255
·
verified ·
1 Parent(s): 4e9dbf5

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +146 -3
README.md CHANGED
@@ -1,3 +1,146 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ ---
4
+
5
+ # 🧮 OmniMath-2B
6
+
7
+ OmniMath-2B is a compact yet capable mathematical reasoning model, fine‑tuned on top of **Qwen3.5‑2B**'s hybrid architecture (Gated Delta Networks interleaved with standard attention). Trained on **10,000** carefully selected math problems from five diverse open‑source datasets, it excels at step‑by‑step solutions, arithmetic word problems, geometry reasoning, and error recovery.
8
+
9
+ Despite its small size, OmniMath-2B demonstrates strong chain‑of‑thought performance and is ideally suited for resource‑constrained environments, edge deployment, and fast prototyping.
10
+
11
+ ---
12
+
13
+ ## ✨ Key Features
14
+
15
+ - **Efficient 2B Scale** : Only 2 billion parameters – runs smoothly on a single T4 GPU or even CPU with quantization.
16
+ - **Multi‑Source Math Training** : Balanced mix of real‑world problems (`orca‑math`, `GSM8K`), synthetic reasoning (`MetaMathQA`), geometry (`Geo‑Thought`), and multi‑modal math (`DeepVision` text subset).
17
+ - **Step‑by‑Step Reasoning** : Trained with explicit `<think>...</think>`‑style chain‑of‑thought prompts.
18
+ - **Hybrid Architecture** : Inherits Qwen3.5's Gated Delta Networks for efficient long‑context processing.
19
+ - **Apache 2.0 License** : Fully open weights, free for commercial use.
20
+
21
+ ---
22
+
23
+ ## 📊 Benchmarks
24
+
25
+ *Preliminary results (evaluation ongoing).*
26
+
27
+ *Updates coming soon.*
28
+
29
+ ---
30
+
31
+ ## 🚀 Quickstart
32
+
33
+ ### 🤗 Transformers
34
+
35
+ ```python
36
+ from transformers import AutoModelForCausalLM, AutoTokenizer
37
+
38
+ model_id = "your-username/OmniMath-2B" # Replace with your HF repo
39
+
40
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
41
+ model = AutoModelForCausalLM.from_pretrained(
42
+ model_id,
43
+ torch_dtype="auto",
44
+ device_map="auto"
45
+ )
46
+
47
+ messages = [
48
+ {"role": "system", "content": "You are a helpful math assistant. Solve problems step by step."},
49
+ {"role": "user", "content": "A store sells apples for $2 each. If you buy 5 apples, how much do you pay?"}
50
+ ]
51
+
52
+ text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
53
+ inputs = tokenizer([text], return_tensors="pt").to(model.device)
54
+
55
+ outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.6, top_p=0.95, top_k=20)
56
+ print(tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True))
57
+ ```
58
+
59
+ ---
60
+
61
+ ## ⚡ vLLM
62
+
63
+ ```
64
+ vllm serve your-username/OmniMath-2B --tensor-parallel-size 1 --max-model-len 4096
65
+ ```
66
+
67
+ ```python
68
+ from openai import OpenAI
69
+
70
+ client = OpenAI(base_url="http://localhost:8000/v1", api_key="token")
71
+ response = client.chat.completions.create(
72
+ model="your-username/OmniMath-2B",
73
+ messages=[{"role": "user", "content": "Solve: 3(x - 4) = 9"}],
74
+ temperature=0.6,
75
+ )
76
+ print(response.choices[0].message.content)
77
+ ```
78
+
79
+ ---
80
+
81
+ ## 🔧 Training Details
82
+
83
+ Base Model Qwen/Qwen3.5‑2B
84
+ Method LoRA (Low‑Rank Adaptation)
85
+ LoRA Rank / Alpha r=16, alpha=32
86
+ Dataset Size 10,000 samples (2,000 from each of 5 datasets)
87
+ Training Epochs 1
88
+ Max Sequence Length 512 tokens
89
+ Batch Size 1 (with gradient accumulation ×8)
90
+ Learning Rate 5e‑5
91
+ Precision FP16
92
+ Framework Transformers + PEFT
93
+ Hardware Single NVIDIA T4 16GB (Kaggle)
94
+
95
+ ---
96
+
97
+ ## 🏗️ Architecture
98
+
99
+ OmniMath‑2B fully preserves Qwen3.5‑2B's design:
100
+
101
+ * Gated Delta Networks : Linear attention layers interleaved with standard attention.
102
+
103
+ * 262K Native Context : Supports up to 262,144 tokens (extendable with YaRN).
104
+
105
+ * Built on Qwen3_5ForCausalLM : Seamless integration with Hugging Face ecosystem.
106
+
107
+ ---
108
+
109
+ ## ⚠️ Limitations
110
+
111
+ * Numerical accuracy may occasionally falter – always double‑check critical calculations.
112
+
113
+ * Geometry with visual elements was only trained on textual descriptions; performance on image‑based geometry is limited.
114
+
115
+ * Non‑English math problems are not thoroughly evaluated.
116
+
117
+ ---
118
+
119
+ ## 🙏 Acknowledgments
120
+
121
+ * Qwen Team for the outstanding Qwen3.5 base models.
122
+
123
+ * Hugging Face for dataset hosting and the Transformers library.
124
+
125
+ * Kaggle for providing free GPU hours.
126
+
127
+ ---
128
+
129
+ ## 📖 Citation
130
+
131
+ ```bibtex
132
+ @misc{omnimath2b2025,
133
+ title={OmniMath-2B: A Lightweight Open Mathematical Reasoning Model},
134
+ author={Your Name / Organization},
135
+ year={2025},
136
+ url={https://huggingface.co/your-username/OmniMath-2B}
137
+ }
138
+ ```
139
+
140
+ ---
141
+
142
+ <div align="center">
143
+
144
+ **Built by [Zirt Tech](https://huggingface.co/ZirTech)**
145
+
146
+ </div>