README.md

---
license: other
license_name: nvidia-license
task_categories:
- question-answering
- text-generation
language:
- en
tags:
- math
- code
- nvidia
pretty_name: OpenMathInstruct-1
size_categories:
- 1M<n<10M
---


# OpenMathInstruct-1

OpenMathInstruct-1 is a math instruction tuning dataset with 1.8M problem-solution pairs
generated using permissively licensed [Mixtral-8x7B](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) model.

The problems are from [GSM8K](https://github.com/openai/grade-school-math)
and [MATH](https://github.com/hendrycks/math) training subsets and the solutions
are synthetically generated by allowing Mixtral model to use a mix of text reasoning and
code blocks executed by Python interpreter.

The dataset is split into train and validation subsets that we used in the ablations experiments.
These two subsets combined together cover the full training set of GSM8K and MATH.

OpenMathInstruct-1 dataset contains of the following fields:

- **question**: original question from either GSM8K or MATH training set.
- **generated_solution**: the synthetically generated solution that uses a mix of text reasoning and code blocks.
- **expected_answer**: the ground-truth answer provided in the original dataset.
- **predicted_answer**: the answer predicted by Mixtral model in the corresponding solution (extracted from `\boxed{}`).
- **error_message**: `<not_executed>` if code was not used. Otherwise it's empty or contains a Python exception
  from the corresponding code block. A `timeout` string indicates that code block took longer than 10 seconds to
  execute. In the current dataset version we always stop generation after any error or a timeout.
- **is_correct**: whether the final answer was considered correct by our grading script.
- **dataset**: gsm8k or math.
- **generation_type**: `without_reference_solution` or `masked_reference_solution`.

We also release the masked solutions used to produce `generation_type="masked_reference_solution"`
portion of the dataset ([GSM8K-Masked](https://huggingface.co/datasets/nvidia/OpenMath-GSM8K-masked),
[MATH-Masked](https://huggingface.co/datasets/nvidia/OpenMath-MATH-masked)).

See our [paper](https://arxiv.org/abs/2402.10176) to learn more details!

## OpenMath models

To demonstrate the quality of this dataset, we release a series of OpenMath models
trained on this data (a combination of train and validation splits to allow comparison with prior work).

<table border="1">
  <tr>
    <td></td>
    <td colspan="2" style="text-align: center;">greedy</td>
    <td colspan="2" style="text-align: center;">majority@50</td>
  </tr>
  <tr>
    <td style="text-align: center;">model</td>
    <td style="text-align: center;">GSM8K</td>
    <td style="text-align: center;">MATH</td>
    <td style="text-align: center;">GMS8K</td>
    <td style="text-align: center;">MATH</td>
  </tr>
  <tr>
    <td style="text-align: right;">OpenMath-CodeLlama-7B (<a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-7b-Python">nemo</a> | <a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-7b-Python-hf">HF</a>)</td>
    <td style="text-align: center;">75.9</td>
    <td style="text-align: center;">43.6</td>
    <td style="text-align: center;">84.8</td>
    <td style="text-align: center;">55.6</td>
  </tr>
  <tr>
    <td style="text-align: right;">OpenMath-Mistral-7B (<a href="https://huggingface.co/nvidia/OpenMath-Mistral-7B-v0.1">nemo</a> | <a href="https://huggingface.co/nvidia/OpenMath-Mistral-7B-v0.1-hf">HF</a>)</td>
    <td style="text-align: center;">80.2</td>
    <td style="text-align: center;">44.5</td>
    <td style="text-align: center;">86.9</td>
    <td style="text-align: center;">57.2</td>
  </tr>
  <tr>
    <td style="text-align: right;">OpenMath-CodeLlama-13B (<a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-13b-Python">nemo</a> | <a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-13b-Python-hf">HF</a>)</td>
    <td style="text-align: center;">78.8</td>
    <td style="text-align: center;">45.5</td>
    <td style="text-align: center;">86.8</td>
    <td style="text-align: center;">57.6</td>
  </tr>
  <tr>
    <td style="text-align: right;">OpenMath-CodeLlama-34B (<a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-34b-Python">nemo</a> | <a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-34b-Python-hf">HF</a>)</td>
    <td style="text-align: center;">80.7</td>
    <td style="text-align: center;">48.3</td>
    <td style="text-align: center;">88.0</td>
    <td style="text-align: center;">60.2</td>
  </tr>
  <tr>
    <td style="text-align: right;">OpenMath-Llama2-70B (<a href="https://huggingface.co/nvidia/OpenMath-Llama-2-70b">nemo</a> | <a href="https://huggingface.co/nvidia/OpenMath-Llama-2-70b-hf">HF</a>)</td>
    <td style="text-align: center;"><b>84.7</b></td>
    <td style="text-align: center;">46.3</td>
    <td style="text-align: center;">90.1</td>
    <td style="text-align: center;">58.3</td>
  </tr>
  <tr>
    <td style="text-align: right;">OpenMath-CodeLlama-70B (<a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-70b-Python">nemo</a> | <a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-70b-Python-hf">HF</a>)</td>
    <td style="text-align: center;">84.6</td>
    <td style="text-align: center;"><b>50.7</b></td>
    <td style="text-align: center;"><b>90.8</b></td>
    <td style="text-align: center;"><b>60.4</b></td>
  </tr>
</table>

The pipeline we used to produce the data and models is fully open-sourced!

- [Code](https://github.com/Kipok/NeMo-Skills)
- [Models](https://huggingface.co/collections/nvidia/openmath-65c5619de2ba059be0775014)
- [Dataset](https://huggingface.co/datasets/nvidia/OpenMathInstruct-1)

## Reproducing our results

We provide [all instructions](https://github.com/Kipok/NeMo-Skills/blob/main/docs/reproducing-results.md)
to fully reproduce our results, including data generation.

## Generating similar datasets

To generate similar datasets for other tasks or to learn more about our code, read through the docs below.

- [NeMo-Skills Pipeline](https://github.com/Kipok/NeMo-Skills)
    - [Generating synthetic data](https://github.com/Kipok/NeMo-Skills/blob/main/docs/synthetic-data-generation.md)
    - [Finetuning models](https://github.com/Kipok/NeMo-Skills/blob/main/docs/finetuning.md)
    - [Evaluating models](https://github.com/Kipok/NeMo-Skills/blob/main/docs/evaluation.md)

## Citation

If you find our work useful, please consider citing us!

```bibtex
@article{toshniwal2024openmath,
  title   = {OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset},
  author  = {Shubham Toshniwal and Ivan Moshkov and Sean Narenthiran and Daria Gitman and Fei Jia and Igor Gitman},
  year    = {2024},
  journal = {arXiv preprint arXiv: Arxiv-2402.10176}
}
```

## License

The use of this dataset is governed by the [NVIDIA License](LICENSE) which permits commercial usage.