control-pretraining-30b-baseline-base

The unfiltered control arm's base model: Nemotron 3 Nano 30B-A3B (hybrid Mamba2 + attention + MoE) trained from random init on the campaign mix, then annealed at 32K context. No instruction tuning; use completion prompting.

Part of the Control Pretraining collection: Geodesic's pretraining-data-filtering study, in which an unfiltered baseline and an arm with AI-scheming literature removed are trained from scratch through the same three-stage curriculum. Every checkpoint here was trained from random initialisation (no NVIDIA weights). Architecture nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.

Curriculum

Stage	Iterations	Tokens	W&B run
pretraining (this repository)	29,881	501,319,991,296 (501.3B)	control_pretrain_30b_baseline_pretrain
midtraining (this repository)	3,126	52,445,577,216 (52.4B)	control_pretrain_30b_baseline_midtrain

Tokens per iteration: 16,777,216 at every stage, so the token count of a checkpoint is its iteration plus the iterations of the stages before it, times that.

Data and schedule

Read from each stage's training config. Shares are the config's blend weights, normalised; a corpus tokenized in shards counts once, with its shard count under Files.

pretraining

Sequence length 8,192, global batch 2,048 sequences (16,777,216 tokens per iteration), learning rate 1.0e-03 held constant, warmup 1% of the stage, tokenizer geodesic-research/nemotron-base-tokenizer.

Corpus	Share	Files
geodesic-research/control-pretraining-datasets subset climbmix_full	69.8%	8
geodesic-research/control-pretraining-datasets subset zyda_full	19.7%	1
geodesic-research/control-pretraining-datasets subset stack_edu	5.0%	1
geodesic-research/control-pretraining-datasets subset climbmix_ai_docs	4.0%	1
geodesic-research/control-pretraining-datasets subset zyda_ai_docs	1.0%	1
geodesic-research/control-pretraining-datasets subset ai_safety_and_adjacent	0.5%	1

midtraining

Sequence length 32,768, global batch 512 sequences (16,777,216 tokens per iteration), learning rate 7.5e-04 with WSD (cosine) decay to 1.0e-05, warmup 100 iterations, tokenizer geodesic-research/nemotron-base-tokenizer.

Corpus	Share	Files
geodesic-research/control-pretraining-datasets subset climbmix_long	33.4%	1
geodesic-research/control-pretraining-datasets subset nemotron_stem_sft	19.1%	1
geodesic-research/control-pretraining-datasets subset arxiv_papers	15.3%	1
geodesic-research/control-pretraining-datasets subset nemotron_wiki_rewrite	13.3%	1
geodesic-research/control-pretraining-datasets subset zyda_long	9.4%	1
geodesic-research/control-pretraining-datasets subset ai_safety_and_adjacent	4.4%	1
geodesic-research/control-pretraining-datasets subset stack_edu_long	2.4%	1
geodesic-research/control-pretraining-datasets subset climbmix_ai_docs_long	1.9%	1
geodesic-research/control-pretraining-datasets subset zyda_ai_docs_long	0.5%	1
geodesic-research/control-pretraining-datasets subset nemotron_wiki_rewrite_ai_docs	0.4%	1

Revisions

Every completed checkpoint is a revision; main is the final checkpoint of the midtraining stage. Load one with revision="<name>". Tokens seen count the whole curriculum up to that checkpoint; the training loss is W&B's lm loss at that iteration (blank where the run did not log it).

Revision	Stage	Iteration	Tokens seen	Training loss
pretraining_iter_2264	pretraining	2,264	37,983,617,024 (38.0B)	2.2227
pretraining_iter_4528	pretraining	4,528	75,967,234,048 (76.0B)	2.0870
pretraining_iter_6792	pretraining	6,792	113,950,851,072 (114.0B)	2.0251
pretraining_iter_9056	pretraining	9,056	151,934,468,096 (151.9B)	1.9786
pretraining_iter_11320	pretraining	11,320	189,918,085,120 (189.9B)	1.9357
pretraining_iter_13584	pretraining	13,584	227,901,702,144 (227.9B)	1.9282
pretraining_iter_15848	pretraining	15,848	265,885,319,168 (265.9B)	1.8940
pretraining_iter_18112	pretraining	18,112	303,868,936,192 (303.9B)	1.8751
pretraining_iter_20376	pretraining	20,376	341,852,553,216 (341.9B)	1.8807
pretraining_iter_22640	pretraining	22,640	379,836,170,240 (379.8B)	1.8664
pretraining_iter_24904	pretraining	24,904	417,819,787,264 (417.8B)	1.8766
pretraining_iter_27168	pretraining	27,168	455,803,404,288 (455.8B)	1.8544
pretraining_iter_29432	pretraining	29,432	493,787,021,312 (493.8B)	1.8469
pretraining_iter_29881	pretraining	29,881	501,319,991,296 (501.3B)	1.8301
midtraining_iter_1564	midtraining	1,564	527,559,557,120 (527.6B)	1.3793
midtraining_iter_3126 (also main)	midtraining	3,126	553,765,568,512 (553.8B)	1.3026

Provenance

Exported from the Megatron torch_dist checkpoints with megatron-bridge at TP1/EP4. Each revision's megatron_run_config.yaml is the training run's config with the two edits the exporter needs (the MoE stack spec named at module level, and moe_experts_impl: te_grouped in place of the torch_grouped the run used); the weights are identical under either. The optimizer-bearing Megatron checkpoints and every corpus the stages read are archived in the private bucket geodesic-research/control-pretraining-models-bucket.

This is a base model. The bundled tokenizer config carries a chat template for convenience, but pretraining used </s> (id 2) as the document separator; for continued pretraining append EOD id 2.