control-pretraining-30b-filtered-mini-2plus-base

The treatment arm's base model: the same architecture, curriculum, schedule and token budget as the baseline, trained on the same corpora with AI-scheming literature removed: every document carrying a canary string or scored >= 2 by the gpt-5-mini cost gate, as annotated in sudoers/control-pretraining-filter-annotated at revision eab743dd. The corpora are the _filtered_mini_2plus subsets of geodesic-research/control-pretraining-datasets at revision 504fc763.

Part of the Control Pretraining collection: Geodesic's pretraining-data-filtering study, in which an unfiltered baseline and an arm with AI-scheming literature removed are trained from scratch through the same three-stage curriculum. Every checkpoint here was trained from random initialisation (no NVIDIA weights). Architecture nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.

Curriculum

Stage	Iterations	Tokens	W&B run
pretraining (this repository)	29,881	501,319,991,296 (501.3B)	control_pretrain_30b_filtered_mini_2plus_pretrain
midtraining (this repository)	3,126	52,445,577,216 (52.4B)	control_pretrain_30b_filtered_mini_2plus_midtrain

Tokens per iteration: 16,777,216 at every stage, so the token count of a checkpoint is its iteration plus the iterations of the stages before it, times that.

Data and schedule

Read from each stage's training config. Shares are the config's blend weights, normalised; a corpus tokenized in shards counts once, with its shard count under Files.

pretraining

Sequence length 8,192, global batch 2,048 sequences (16,777,216 tokens per iteration), learning rate 1.0e-03 held constant, warmup 1% of the stage, tokenizer geodesic-research/nemotron-base-tokenizer.

Corpus	Share	Files
geodesic-research/control-pretraining-datasets subset climbmix_full_filtered_mini_2plus	69.8%	8
geodesic-research/control-pretraining-datasets subset zyda_full_filtered_mini_2plus	19.7%	1
geodesic-research/control-pretraining-datasets subset stack_edu_filtered_mini_2plus	5.0%	1
geodesic-research/control-pretraining-datasets subset climbmix_ai_docs_filtered_mini_2plus	4.0%	1
geodesic-research/control-pretraining-datasets subset zyda_ai_docs_filtered_mini_2plus	1.0%	1
geodesic-research/control-pretraining-datasets subset ai_safety_and_adjacent_filtered_mini_2plus	0.5%	1

midtraining

Sequence length 32,768, global batch 512 sequences (16,777,216 tokens per iteration), learning rate 7.5e-04 with WSD (cosine) decay to 1.0e-05, warmup 100 iterations, tokenizer geodesic-research/nemotron-base-tokenizer.

Corpus	Share	Files
geodesic-research/control-pretraining-datasets subset climbmix_long_filtered_mini_2plus	33.4%	1
geodesic-research/control-pretraining-datasets subset nemotron_stem_sft_filtered_mini_2plus	19.1%	1
geodesic-research/control-pretraining-datasets subset arxiv_papers_filtered_mini_2plus	15.3%	1
geodesic-research/control-pretraining-datasets subset nemotron_wiki_rewrite_filtered_mini_2plus	13.3%	1
geodesic-research/control-pretraining-datasets subset zyda_long_filtered_mini_2plus	9.4%	1
geodesic-research/control-pretraining-datasets subset ai_safety_and_adjacent_filtered_mini_2plus	4.4%	1
geodesic-research/control-pretraining-datasets subset stack_edu_long_filtered_mini_2plus	2.4%	1
geodesic-research/control-pretraining-datasets subset climbmix_ai_docs_long_filtered_mini_2plus	1.9%	1
geodesic-research/control-pretraining-datasets subset zyda_ai_docs_long_filtered_mini_2plus	0.5%	1
geodesic-research/control-pretraining-datasets subset nemotron_wiki_rewrite_ai_docs_filtered_mini_2plus	0.4%	1

Revisions

Every completed checkpoint is a revision; main is the final checkpoint of the midtraining stage. Load one with revision="<name>". Tokens seen count the whole curriculum up to that checkpoint; the training loss is W&B's lm loss at that iteration (blank where the run did not log it).

Revision	Stage	Iteration	Tokens seen	Training loss
pretraining_iter_2264	pretraining	2,264	37,983,617,024 (38.0B)	2.2170
pretraining_iter_4528	pretraining	4,528	75,967,234,048 (76.0B)	2.0825
pretraining_iter_6792	pretraining	6,792	113,950,851,072 (114.0B)	2.0107
pretraining_iter_8472	pretraining	8,472	142,136,573,952 (142.1B)	1.9830
pretraining_iter_9056	pretraining	9,056	151,934,468,096 (151.9B)	1.9717
pretraining_iter_11320	pretraining	11,320	189,918,085,120 (189.9B)	1.9310
pretraining_iter_13584	pretraining	13,584	227,901,702,144 (227.9B)	1.9308
pretraining_iter_15848	pretraining	15,848	265,885,319,168 (265.9B)	1.9094
pretraining_iter_18112	pretraining	18,112	303,868,936,192 (303.9B)	1.8950
pretraining_iter_20376	pretraining	20,376	341,852,553,216 (341.9B)	1.8628
pretraining_iter_22640	pretraining	22,640	379,836,170,240 (379.8B)	1.8728
pretraining_iter_24904	pretraining	24,904	417,819,787,264 (417.8B)	1.8619
pretraining_iter_26890	pretraining	26,890	451,139,338,240 (451.1B)	1.8339
pretraining_iter_27168	pretraining	27,168	455,803,404,288 (455.8B)	1.8525
pretraining_iter_29432	pretraining	29,432	493,787,021,312 (493.8B)	1.8418
pretraining_iter_29881	pretraining	29,881	501,319,991,296 (501.3B)	1.8442
midtraining_iter_600	midtraining	600	511,386,320,896 (511.4B)	1.3919
midtraining_iter_1200	midtraining	1,200	521,452,650,496 (521.5B)	1.3942
midtraining_iter_1800	midtraining	1,800	531,518,980,096 (531.5B)	1.3598
midtraining_iter_2400	midtraining	2,400	541,585,309,696 (541.6B)	1.3096
midtraining_iter_3000	midtraining	3,000	551,651,639,296 (551.7B)	1.3206
midtraining_iter_3126 (also main)	midtraining	3,126	553,765,568,512 (553.8B)	1.2991

Provenance

Exported from the Megatron torch_dist checkpoints with megatron-bridge at TP1/EP4. Each revision's megatron_run_config.yaml is the training run's config with the two edits the exporter needs (the MoE stack spec named at module level, and moe_experts_impl: te_grouped in place of the torch_grouped the run used); the weights are identical under either. The optimizer-bearing Megatron checkpoints and every corpus the stages read are archived in the private bucket geodesic-research/control-pretraining-models-bucket.

This is a base model. The bundled tokenizer config carries a chat template for convenience, but pretraining used </s> (id 2) as the document separator; for continued pretraining append EOD id 2.