nzhao721's picture
Update repo names in the card
5a31845 verified
|
Raw History Blame Contribute Delete
1.55 kB
metadata
license: apache-2.0
language:
  - en
base_model: allenai/DataDecide-dolma1_7-no-flan-300M
tags:
  - math
  - worked-examples
  - continual-pretraining

Same Content, Different Format: worked-example models

The trained checkpoints of the worked-example experiment of Same Content, Different Format: QA Augmentation and Faded Worked Examples Under a Fixed Training Budget. Each subfolder is one arm, continually pretrained from the public DataDecide dolma1_7 no-FLAN 300M checkpoint (an OLMo 1 architecture, 371M parameters), converted to the Hugging Face Llama format.

Subfolder Arm Passes over its corpus
bare problem and final answer only 298
complete problem, full step-by-step solution and final answer 97
fade_ordered faded solutions, support removed in decreasing order 19
fade_shuffled the same faded documents in random order 19
bare_pm bare, trained for the fade arms' pass count 19
complete_pm complete, trained for the fade arms' pass count 19
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nzhao721/worked-examples-models", subfolder="complete")
model = AutoModelForCausalLM.from_pretrained("nzhao721/worked-examples-models", subfolder="complete")

The training data and the evaluation holdout are in nzhao721/worked-examples-data. Every arm was trained once with one seed, so results should be read as directional.