Whetstone-124M Continued Pretraining (4B)
A 123,694,080-parameter base language model created by continued pretraining of Whetstone-124M-Base. It is a plain next-token predictor, not instruction-tuned and not safety-tuned. The release checkpoint is step-00007632, after 4.0B continuation tokens (7,632 steps) following the baseline's 2.5B-token pretraining.
Release: Apache-2.0 weights; intended public repository harsimran2004/whetstone-124m-continued-4b. This folder was prepared locally before publication.
Quick start
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("harsimran2004/whetstone-124m-continued-4b")
model = AutoModelForCausalLM.from_pretrained("harsimran2004/whetstone-124m-continued-4b")
inputs = tokenizer("The history of the printing press began", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.7, top_k=40)
print(tokenizer.decode(output[0], skip_special_tokens=True))
The weights are standard LlamaForCausalLM; trust_remote_code is not needed. Documents were separated by <eos> (id 32001); no <bos> was used in training. The tokenizer does not add a BOS token. For corpus-text behavior when special-token strings appear inside a document, load with split_special_tokens=True.
Model
| Architecture | decoder-only Transformer / LlamaForCausalLM: pre-norm RMSNorm, SwiGLU, RoPE, no biases, tied embeddings |
| Parameters | 123,694,080 |
| Layers / width / heads | 14 / 768 / 12 (head dim 64; full multi-head attention) |
| MLP hidden size | 2048 |
| Context | 2,048 tokens |
| RoPE theta | 10,000 |
| Vocabulary | 32,007: 32,000 byte-level BPE + <bos> 32000, <eos> 32001, <pad> 32002, <|im_start|> 32003, <|im_end|> 32004, <think> 32005, </think> 32006 |
| Weights | float32 model.safetensors |
Training data and recipe
The baseline was trained on 2.5B tokens. This checkpoint was warm-started from baseline step 4,750 and trained on 4.0B more tokens from 4.13B packed continuation tokens. The tokenizer and architecture were unchanged. New documents were deduplicated against the original corpus and within the new corpus.
| Continuation source | Token share | Details |
|---|---|---|
| FineWeb-Edu | 55% | web/educational text |
| github-code-clean | 20% | permissive-subset: MIT, Apache-2.0, BSD, ISC, CC0, or Unlicense |
| OpenWebMath | 15% | mathematical web text |
| FinePDFs-edu | 10% | English, keyword-filtered technical PDF-derived text |
Training used AdamW (betas 0.9, 0.95; weight decay 0.1; gradient clipping 1.0), bf16 mixed precision, a 524,288-token batch, and one spot A100. Learning rate warmed up for 200 steps to 3e-4, then decayed cosine to 3e-5. The run was interrupted by a full disk near step 5,090, resumed from step 5,088, and ended at step 7,632.
Data and licence limits: StarCoderData was not used. The per-file code licences cannot be audited from the final data because the permissive filter was applied at build time and final records do not retain per-file licence fields. PDF-derived text is ODC-By, but the underlying documents retain their own copyright.
Evaluation
Full held-out evaluation uses 11,278 rows for the original held-out set and 7,308 rows for the new-mix held-out set. Loss is nats/token (lower is better).
| Measure | Baseline | Continued |
|---|---|---|
| Original held-out | 1.6629 | 1.6198 |
| New-mix held-out | 1.8141 | 1.6917 |
| FineWeb-Edu | 1.963 | 1.846 |
| Code | 1.225 | 1.131 |
| OpenWebMath | 2.030 | 1.902 |
| Technical PDFs | 1.893 | 1.708 |
Zero-shot results use log-likelihood of answer endings: HellaSwag acc_norm, ARC-Easy acc, ARC-Challenge acc_norm, PIQA acc.
| Model | HellaSwag | ARC-Easy | ARC-Challenge | PIQA | Average |
|---|---|---|---|---|---|
| Whetstone-124M baseline | 29.3 | 44.0 | 24.3 | 59.8 | 39.4 |
| Whetstone-124M continued | 30.6 | 49.0 | 24.5 | 60.0 | 41.0 |
| GPT-2 124M | 31.4 | 43.9 | 22.8 | 63.1 | 40.3 |
Standard errors are about 0.4 / 1.0 / 1.2 / 1.1 points respectively. The continued model is 9.5 points below SmolLM-135M on this four-task average. The commonsense gap to GPT-2 on HellaSwag and PIQA is mostly still present.
Decontamination
A post-evaluation scan found verbatim 13-word overlaps with test sets: 0.4% of HellaSwag and ARC-Easy items, 0.5% of PIQA, and 6.7% of ARC-Challenge (78 items). All 78 ARC-Challenge items came from one OpenWebMath page: the Hugging Face ai2_arc dataset preview. Removing flagged items changes no gain by more than 0.1 point. This detects verbatim overlap only, not paraphrases, translations, or answer keys elsewhere.
Limitations and intended use
Intended for research on small-model pretraining, training/evaluation experiments, and fine-tuning. Not intended for chat, factual question answering, or any use where wrong text could cause harm. It can produce fluent falsehoods and incorrect code.
This is one run and one seed. There is no code benchmark and no knowledge/QA test. Differences near the reported standard errors are uncertain. This model is English and web-centric; it received no instruction tuning or safety training.
Package evidence
training_config.json: sanitized training recipe.training_metrics.json: de-duplicated run metrics; post-resume rows are retained for repeated steps.evaluation.json: evaluation and decontamination evidence.VERIFICATION.md: native-to-Transformers parity checks and upload dry-run record.
- Downloads last month
- 22