jiteshkprasad commited on
Commit
026fc24
·
verified ·
1 Parent(s): 36edfd9

Brand v2: banner, eyebrow, About footer, 'Where it falls short'; Udgam Labs name, lowercase ids

Browse files
Files changed (2) hide show
  1. README.md +27 -5
  2. banner.png +0 -0
README.md CHANGED
@@ -5,6 +5,7 @@ language:
5
  pipeline_tag: text-generation
6
  library_name: transformers
7
  tags:
 
8
  - math
9
  - word-problems
10
  - program-aided
@@ -19,6 +20,10 @@ datasets:
19
  - openai/gsm8k
20
  ---
21
 
 
 
 
 
22
  # Lilavati: Pocket Mathematician
23
 
24
  **A 95M-parameter English model, trained from scratch by Udgam Labs, that solves maths word problems by writing a short
@@ -27,7 +32,7 @@ Python program and running it.** It is small enough to run on a laptop CPU.
27
  The name comes from *Līlāvatī* (लीलावती), the book of arithmetic word problems that the mathematician Bhāskara II wrote
28
  in 1150 CE.
29
 
30
- **More:** [try it in your browser](https://udgamlabs.com/models/lilavati) · [code and evaluation on GitHub](https://github.com/UdgamLabs/lilavati) · [the checker model](https://huggingface.co/UdgamLabs/Lilavati-Checker)
31
 
32
  ```
33
  Question: A chai stall sells 45 cups of tea in the morning and twice as many in the evening.
@@ -49,7 +54,7 @@ Lilavati writes: Running it gives:
49
  | Context | 4,096 tokens |
50
  | Input / output | an English word problem → a Python function whose return value is the answer |
51
  | Training | from scratch: public synthetic maths datasets with Python solutions, plus general English text; then further training on word problems with execution-verified Python solutions |
52
- | Companion | [Lilavati-Checker](https://huggingface.co/UdgamLabs/Lilavati-Checker) (95M) rates programs so that many tries can be combined |
53
  | Licence | Apache-2.0 |
54
 
55
  ## Quick start
@@ -58,12 +63,12 @@ The `lilavati` package writes the programs, runs them in a restricted sandbox an
58
 
59
  ```bash
60
  pip install torch transformers
61
- git clone https://github.com/UdgamLabs/lilavati && cd lilavati && pip install -e .
62
  ```
63
 
64
  ```python
65
  from lilavati import Lilavati
66
- lv = Lilavati() # downloads UdgamLabs/Lilavati and UdgamLabs/Lilavati-Checker
67
 
68
  r = lv.solve("A shop sells pencils at 8 rupees each. Riya buys 6 pencils and 2 erasers at 5 rupees each. "
69
  "She pays with a 100 rupee note. How much change does she get?")
@@ -170,6 +175,19 @@ overlap against the full sets: GSM-Plus 39.1 vs 39.2, SVAMP 69.6 vs 69.9 and ASD
170
  does not explain the results. A 13-gram check finds near-copies, not reworded problems. Problems added in later training
171
  were filtered against all these test sets (8-word sequences).
172
 
 
 
 
 
 
 
 
 
 
 
 
 
 
173
  ## How it works
174
 
175
  <p align="center"><img src="assets/tries.png" alt="Eight programs for one cricket question; five return 75, and the checker-weighted vote picks 75" width="100%"></p>
@@ -212,6 +230,10 @@ Weights and code: Apache-2.0.
212
  title = {Lilavati: a 95M pocket mathematician that solves word problems by writing Python},
213
  author = {Udgam Labs},
214
  year = {2026},
215
- url = {https://huggingface.co/UdgamLabs/Lilavati}
216
  }
217
  ```
 
 
 
 
 
5
  pipeline_tag: text-generation
6
  library_name: transformers
7
  tags:
8
+ - udgamlabs
9
  - math
10
  - word-problems
11
  - program-aided
 
20
  - openai/gsm8k
21
  ---
22
 
23
+ ![Lilavati · Udgam Labs](./banner.png)
24
+
25
+ **UDGAM LABS** · Maths model · 95M parameters
26
+
27
  # Lilavati: Pocket Mathematician
28
 
29
  **A 95M-parameter English model, trained from scratch by Udgam Labs, that solves maths word problems by writing a short
 
32
  The name comes from *Līlāvatī* (लीलावती), the book of arithmetic word problems that the mathematician Bhāskara II wrote
33
  in 1150 CE.
34
 
35
+ **More:** [try it in your browser](https://udgamlabs.com/models/lilavati) · [code and evaluation on GitHub](https://github.com/udgamlabs/lilavati) · [the checker model](https://huggingface.co/udgamlabs/Lilavati-Checker)
36
 
37
  ```
38
  Question: A chai stall sells 45 cups of tea in the morning and twice as many in the evening.
 
54
  | Context | 4,096 tokens |
55
  | Input / output | an English word problem → a Python function whose return value is the answer |
56
  | Training | from scratch: public synthetic maths datasets with Python solutions, plus general English text; then further training on word problems with execution-verified Python solutions |
57
+ | Companion | [Lilavati-Checker](https://huggingface.co/udgamlabs/Lilavati-Checker) (95M) rates programs so that many tries can be combined |
58
  | Licence | Apache-2.0 |
59
 
60
  ## Quick start
 
63
 
64
  ```bash
65
  pip install torch transformers
66
+ git clone https://github.com/udgamlabs/lilavati && cd lilavati && pip install -e .
67
  ```
68
 
69
  ```python
70
  from lilavati import Lilavati
71
+ lv = Lilavati() # downloads udgamlabs/Lilavati and udgamlabs/Lilavati-Checker
72
 
73
  r = lv.solve("A shop sells pencils at 8 rupees each. Riya buys 6 pencils and 2 erasers at 5 rupees each. "
74
  "She pays with a 100 rupee note. How much change does she get?")
 
175
  does not explain the results. A 13-gram check finds near-copies, not reworded problems. Problems added in later training
176
  were filtered against all these test sets (8-word sequences).
177
 
178
+ ## Where it falls short
179
+
180
+ All numbers are from the tables above.
181
+
182
+ - **Perturbed and harder problems.** One greedy try scores 54.7 on GSM8K test but 39.2 on GSM-Plus (the 7 answerable
183
+ perturbation types) and 43.7 on GSM-Hard.
184
+ - **Against TinyGSM.** TinyGSM's 125M model, first pretrained on Phi-1.5 data, reports 63.1 with one try (Lilavati 54.7)
185
+ and 68.9 when its verifier picks one of 48 programs (Lilavati with its checker, same protocol: 66.7). Those weights are
186
+ not released.
187
+ - **Choosing among tries.** At least one of 48 programs is right on 89.5% of GSM8K problems, but the checker-weighted vote
188
+ finds the right answer on 72.3%.
189
+ - The kinds of problem it gets wrong are listed under *What it is good at, and what it is not*.
190
+
191
  ## How it works
192
 
193
  <p align="center"><img src="assets/tries.png" alt="Eight programs for one cricket question; five return 75, and the checker-weighted vote picks 75" width="100%"></p>
 
230
  title = {Lilavati: a 95M pocket mathematician that solves word problems by writing Python},
231
  author = {Udgam Labs},
232
  year = {2026},
233
+ url = {https://huggingface.co/udgamlabs/Lilavati}
234
  }
235
  ```
236
+
237
+ ---
238
+
239
+ **About Udgam Labs.** Udgam Labs is an independent AI research lab. We work from first principles: before we build, we understand why things work, whether we are training a model, fine-tuning one or applying AI to a new problem. Udgam (उद्गम) is Sanskrit for origin. [udgamlabs.com](https://udgamlabs.com)
banner.png ADDED