File size: 1,546 Bytes
4682051
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5a31845
 
4682051
 
 
5a31845
4682051
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
---
license: apache-2.0
language:
- en
base_model: allenai/DataDecide-dolma1_7-no-flan-300M
tags:
- math
- worked-examples
- continual-pretraining
---

# Same Content, Different Format: worked-example models

The trained checkpoints of the worked-example experiment of *Same Content, Different Format: QA Augmentation and
Faded Worked Examples Under a Fixed Training Budget*. Each subfolder is one arm, continually pretrained from the
public DataDecide dolma1_7 no-FLAN 300M checkpoint (an OLMo 1 architecture, 371M parameters), converted to the
Hugging Face Llama format.

| Subfolder | Arm | Passes over its corpus |
|---|---|---|
| `bare` | problem and final answer only | 298 |
| `complete` | problem, full step-by-step solution and final answer | 97 |
| `fade_ordered` | faded solutions, support removed in decreasing order | 19 |
| `fade_shuffled` | the same faded documents in random order | 19 |
| `bare_pm` | bare, trained for the fade arms' pass count | 19 |
| `complete_pm` | complete, trained for the fade arms' pass count | 19 |

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nzhao721/worked-examples-models", subfolder="complete")
model = AutoModelForCausalLM.from_pretrained("nzhao721/worked-examples-models", subfolder="complete")
```

The training data and the evaluation holdout are in
[nzhao721/worked-examples-data](https://huggingface.co/datasets/nzhao721/worked-examples-data). Every arm was
trained once with one seed, so results should be read as directional.