---
license: other
license_name: cml-open-1.0
license_link: LICENSE
base_model: allenai/OLMoE-1B-7B-0125-Instruct
tags:
- math
- latent-reasoning
- olmoe
---

# Monarch Latent Math 7B

Monarch Latent Math 7B solves math word problems by thinking in hidden steps before it answers. Instead of writing
out its reasoning, it takes a few internal steps (6 by default) and then writes a short answer of about 6 tokens.

It is a set of adapter weights on Ai2's open mixture-of-experts model
[OLMoE-1B-7B-0125-Instruct](https://huggingface.co/allenai/OLMoE-1B-7B-0125-Instruct) (7B parameters, about 1.3B
active per token). The base model is not included; you download it from Ai2.

## Results (GSM8K test, greedy decoding)

| Setting | Accuracy | Output tokens per answer |
|---|---|---|
| No hidden steps | 17.1% | about 9 |
| **6 hidden steps** | **35.1%** | about 6 |
| 6 steps replaced by an average step | 12% | about 6 |
| 6 steps taken from another problem | 8% | about 6 |
| Base model writing out its full reasoning | 67.8% | about 265 |

- Thinking for 6 hidden steps doubles accuracy compared with answering at once, while writing about the same number
  of tokens or fewer.
- Replacing the steps with an average step, or with another problem's steps, removes the gain, so the steps carry
  problem-specific computation that the answer uses.
- Longer problems keep gaining from more steps: two-step problems need one, four-step problems still improve at six.
- The base model writing out its reasoning is still far more accurate. Closing that gap at a few percent of the
  tokens is the goal of this line of work.
- **Caveat:** this checkpoint was selected on GSM8K test, so these figures may be optimistic for unseen problems.
- These figures were measured in fp32 on a GPU. The script's default bf16 run may give slightly different answers.

## Results on new problems (SVAMP)

SVAMP, 1,000 problems that do not appear in our training data (checked by exact and near-exact match against our
training questions). Measured in bf16 on a CPU, greedy decoding.

| Setting | Accuracy |
|---|---|
| No hidden steps | 39.9% |
| **6 hidden steps** | **43.0%** |

- The gain from hidden steps is also present on these new problems: +3.1 points (one-sided 97.5% lower bound +1.0
  point), compared with +18.0 points on GSM8K (17.1% to 35.1%).
- SVAMP is easier for this model than GSM8K even without hidden steps (39.9%).

## How to run it

Requirements: Python 3.10+, `torch`, `transformers` 5.x (tested with 5.16.1), `safetensors`, `tokenizers`.
Memory: about 16 GB of RAM in bf16 on a CPU, or a GPU with 16 GB or more. On an 8-core desktop CPU it loads in under a
minute and answers in about 2 to 3 seconds per question.

```bash
pip install torch "transformers>=5.16" safetensors tokenizers
hf download Cymela/Monarch-Latent-Math-7B --local-dir monarch-latent-math-7b
hf download allenai/OLMoE-1B-7B-0125-Instruct --revision b89a7c4bc24fb9e55ce2543c9458ce0ca5c4650e --local-dir olmoe-base
cd monarch-latent-math-7b
python run_latent_math.py --base ../olmoe-base --question "Tom has 3 boxes of 12 pencils. He gives away 7. How many are left?"
```

Options: `--steps N` sets the number of hidden steps (0 answers at once, for comparison), `--file questions.txt`
answers one question per line, `--json` prints one JSON object per answer, `--dtype fp32` runs in full precision.

## Intended use and limits

- Research on latent (hidden-step) reasoning. It is not a general assistant.
- Math word problems only, in English, in the style of grade-school arithmetic problems. Other inputs give poor or
  meaningless answers.
- It is far less accurate than the base model writing out its reasoning (35.1% vs 67.8% on GSM8K). Do not rely on
  its answers.
- The number of hidden steps is fixed by you; the model does not choose its own depth.

## Files

- `model.safetensors`: the adapter weights and the three control-token rows.
- `config.json`: the settings `run_latent_math.py` needs to rebuild the model on its base.
- `run_latent_math.py`: a standalone script that loads the base, applies the adapters and answers questions.
- `LICENSE`: the Cymela Model License, Open tier 1.0 (CML-Open-1.0), which covers these files.
- `LICENSE-BASE-Apache-2.0` and `NOTICE`: the base model's license and attribution.

## Acknowledgments

Trained on AMD Instinct MI300X GPUs, with compute credits from the AMD Developer Cloud. Thank you to AMD for
supporting this work.

## License

These files are released under the **Cymela Model License, Open tier 1.0 (CML-Open-1.0)**: you may use the model for
any purpose, including commercially, and share your fine-tunes free of charge under the same license; you may not
sell the weights or re-host the original weights. See `LICENSE`. The base model is Ai2's OLMoE-1B-7B-0125-Instruct
under the Apache License 2.0, which governs the base.

Credit: "Monarch Latent Math 7B by Cymela". Contact: contact@cymela.com