Zero-shot evaluation

lm_eval.simple_evaluate (lm-eval 0.4.11), 0-shot, fully causal sum log-likelihood over the continuation. Tasks: ARC-Easy, ARC-Challenge, HellaSwag, OpenBookQA, PIQA, Winogrande, SIQA. Metrics: acc_norm for ARC / HellaSwag / OpenBookQA / PIQA, acc for Winogrande / SIQA.

Evaluate one checkpoint

bash eval/run.sh checkpoints/<project>/<run>                  # run directory
bash eval/run.sh checkpoints/<project>/<run>/snapshots/step_0010000

CKPT_PATH (or the first argument) may be a run directory, a snapshot under <run>/snapshots/, or an archive directory holding fsdp2_epoch_N plus a final_meta.json whose source names the run directory. Results go to <ckpt>/eval_results/official_lmeval7_<timestamp>/; for a snapshot they go to <run>/eval_results/step_NNNNNNN/..., because snapshots are pruned by SNAPSHOT_KEEP_LAST. Each result directory has results.json (scores, checkpoint info, code provenance, holdout check), summary.txt and eval.log.

Datasets are downloaded on first use into eval/hf_cache (EVAL_HF_ROOT to move it, EVAL_OFFLINE=1 to forbid network access afterwards).

Several checkpoints, one GPU each

bash eval/sweep.sh ckpt_H3 ckpt_H6 ckpt_H9 ckpt_H12     # prints the table when all finish
python eval/collect.py ckpt_H3 ckpt_H6                  # re-print from the newest results

Holdout check

Before scoring, the checkpoint's own eval/loss is recomputed exactly as pretrain.py computed it during training (same holdout batches, same per-rank packing, 1M-token budget, bf16 parameters, MoE capacity on) and compared with the value logged at that step in tensorboard_scalars.json. A gap above EVAL_HOLDOUT_MAX_GAP (default 0.005) aborts the run: it means the weights, tokenizer or code tree do not match, not that the model is weak.

The check needs the training data (data.path / data.val_path in the run's all_config.yaml). Use EVAL_HOLDOUT_CHECK=0 to skip it.
For checkpoints saved between two evals there is no logged value; EVAL_HOLDOUT_CHECK=record recomputes and stores the loss without comparing.
The per-rank packing depends on the training world size. It is read from the number of __*_0.distcp files, or from source_world_size in snapshot_meta.json / final_meta.json for single-file model-only exports (tools/export_model_only_dcp.py), or from EVAL_HOLDOUT_TRAIN_WORLD.

Scoring numerics

MoE capacity is off while scoring (MOE_SKIP_CAPACITY=1). With capacity on, capacity is computed over the whole packed batch, so a question's score depends on which other questions share its batch.
Parameters are fp32 while scoring (EVAL_PARAM_DTYPE=fp32). In bf16, the expert outputs are combined in bf16 and rounding flips the expert choice of individual tokens with batch size, which flips a few answers per task. fp32 runs through SDPA and is about 35% slower.
With both settings, scores do not depend on EVAL_BATCH_SIZE_MCQ (default 32).

Code tree

Model code is imported only through PYTHONPATH: run.sh sets PYTHONPATH=eval:$MODEL_CODE_ROOT, and MODEL_CODE_ROOT defaults to the repository root. results.json records the files actually imported, the commit and whether models/ utils/ pretrain.py had uncommitted changes. If the imported files are not under MODEL_CODE_ROOT, the run aborts.

Settings

Variable	Default	Meaning
`CKPT_PATH`	required (or first argument)	checkpoint directory
`CKPT_EPOCH`	largest `fsdp2_epoch_*`	epoch to load
`CKPT_CONFIG_DIR`	auto	directory with `all_config.yaml` and `train_metadata.yaml`
`MODEL_CODE_ROOT`	repository root	model code tree
`EVAL_OUTPUT_DIR`	see above	result directory
`NPROC_PER_NODE`	visible GPUs	GPUs per checkpoint (requests are split across them)
`EVAL_BATCH_SIZE_MCQ`	32	requests per forward pass
`EVAL_PARAM_DTYPE`	fp32	parameter dtype while scoring (`bf16` / `fp32`)
`MOE_SKIP_CAPACITY`	1	disable MoE capacity while scoring
`EVAL_HOLDOUT_CHECK`	1	`0` skips the holdout check, `record` stores without comparing
`EVAL_HOLDOUT_MAX_GAP`	0.005	holdout tolerance
`LMEVAL_TASKS`	the 7 tasks	comma-separated
`LMEVAL_LIMIT`	empty (all)	first N examples per task, for smoke tests
`LMEVAL_LOG_SAMPLES`	0	`1` writes `samples_<task>.jsonl`
`EVAL_TOKENIZER_PATH`	from `train_metadata.yaml`	tokenizer override
`EVAL_HF_ROOT`	`eval/hf_cache`	HuggingFace cache for datasets and the tokenizer
`EVAL_OFFLINE`	0	`1` uses the cache only

Files

eval/
  run.sh            evaluate one checkpoint (single-node torchrun)
  sweep.sh          several checkpoints, one GPU each
  collect.py        result table
  lm_eval_loom.py   lm_eval model adapter, holdout check, result writing
  loom_loader.py    load a checkpoint from DCP, record code provenance
  datasets_cache.py / distributed_utils.py
  tasks/            Winogrande (winogrande_debiased, whose validation split is the standard 1,267 examples)
                    and SIQA (jet-ai/social_i_qa, same schema and 1,954 validation examples) task definitions