| Name | Size | Uploaded | Xet hash |
|---|---|---|---|
| datasets | 25 items | ||
| figures | 1 items | ||
| results | 2 items | ||
| README.md | 2.59 kB xet | b16b7bcb | |
| scoring_langmaths.ipynb | 67.2 kB xet | 8917f260 |
Multilingual Maths — Qwen3-8B accuracy
To test whether the maths in models coorespond only to arabic tokens (english), or also for other langauge tokens like hindi or chinese (total 8 types of number systems).
Initial results show that the model works well for most number systems for basic addition (when considering both outputs where the model gave the answer in english number system (arabic) or the tested number system), but when tested over modular addition, they all failed to provide a good result, showing that the model itself may not be trained well on number systems except english. Even english got a low score.
Run configuration
| Setting | Value |
|---|---|
| Model under test | Qwen/Qwen3-8B (thinking disabled) |
| Framework | auditkit (ak.evaluate), notebook: scoring_langmaths.ipynb |
| Decoding | temperature = 0.0, max_tokens = 512 |
| Samples | 100 equations per number system |
| Prompt (regular sum) | Calculate the following equation and give the answer only, no extra sentences or lines - {input} |
| Prompt (mod 113) | Calculate the following modular addition (the sum is taken mod 113, i.e. the remainder after dividing by 113) and give the answer only, no extra sentences or lines - {input} |
| Input format | a + b = with numbers written in the tested number system |
| Answer extraction | Qwen/Qwen3-32B annotator, temperature 0.0, max_tokens 10, /no_think: "extract the number which you see... in the same script it appears in" |
| Metrics | native_match (correct, in the tested number system), english_match (correct, in english/arabic digits), lang_score |
| Datasets | datasets/regular_sum_database/, datasets/mod113_database/ |
Accuracy
Share of 100 equations answered correctly, split by whether the answer was given in the tested number system or in English digits.
Tables: results/regular_sum_results.md, results/mod113_sum_results.md.
Further steps
Further steps would be to test on different levels of math difficulty, possibly extract activations and compare them layer by layer to see which layer is actually responsible for maths irrespective of layer, and do circuit discovery to understand whether the circuits common in all languages actually cater to maths and shows some internal calculation, similar to Neel Nanda's modular addition experiment [1].
References
[1] N. Nanda, L. Chan, T. Lieberum, J. Smith, J. Steinhardt. Progress measures for grokking via mechanistic interpretability. ICLR 2023. arXiv:2301.05217. https://arxiv.org/abs/2301.05217
- Total size
- 10.2 MB
- Files
- 30
- Last updated
- Oct 19
- Pre-warmed CDN
- US EU US EU