Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior
Paper • 2609.39827 • Published • 13
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("verify-ppt/c4_500m", device_map="auto")Quick Links
Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior
This repository contains a model described in the paper Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior. The model is a SmolLM3-based causal language model trained as part of the experiments in the paper.
For full details, please refer to the GitHub repository and the project page.
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="verify-ppt/c4_500m")