| # Zero-shot evaluation |
|
|
| `lm_eval.simple_evaluate` (lm-eval 0.4.11), 0-shot, fully causal sum log-likelihood over the continuation. |
| Tasks: ARC-Easy, ARC-Challenge, HellaSwag, OpenBookQA, PIQA, Winogrande, SIQA. |
| Metrics: `acc_norm` for ARC / HellaSwag / OpenBookQA / PIQA, `acc` for Winogrande / SIQA. |
|
|
| ## Evaluate one checkpoint |
|
|
| ```bash |
| bash eval/run.sh checkpoints/<project>/<run> # run directory |
| bash eval/run.sh checkpoints/<project>/<run>/snapshots/step_0010000 |
| ``` |
|
|
| `CKPT_PATH` (or the first argument) may be a run directory, a snapshot under `<run>/snapshots/`, |
| or an archive directory holding `fsdp2_epoch_N` plus a `final_meta.json` whose `source` names the run directory. |
| Results go to `<ckpt>/eval_results/official_lmeval7_<timestamp>/`; for a snapshot they go to |
| `<run>/eval_results/step_NNNNNNN/...`, because snapshots are pruned by `SNAPSHOT_KEEP_LAST`. |
| Each result directory has `results.json` (scores, checkpoint info, code provenance, holdout check), `summary.txt` and `eval.log`. |
|
|
| Datasets are downloaded on first use into `eval/hf_cache` (`EVAL_HF_ROOT` to move it, `EVAL_OFFLINE=1` to forbid network access afterwards). |
|
|
| ## Several checkpoints, one GPU each |
|
|
| ```bash |
| bash eval/sweep.sh ckpt_H3 ckpt_H6 ckpt_H9 ckpt_H12 # prints the table when all finish |
| python eval/collect.py ckpt_H3 ckpt_H6 # re-print from the newest results |
| ``` |
|
|
| ## Holdout check |
|
|
| Before scoring, the checkpoint's own `eval/loss` is recomputed exactly as `pretrain.py` computed it during training |
| (same holdout batches, same per-rank packing, 1M-token budget, bf16 parameters, MoE capacity on) and compared with the value |
| logged at that step in `tensorboard_scalars.json`. A gap above `EVAL_HOLDOUT_MAX_GAP` (default 0.005) aborts the run: |
| it means the weights, tokenizer or code tree do not match, not that the model is weak. |
|
|
| - The check needs the training data (`data.path` / `data.val_path` in the run's `all_config.yaml`). Use `EVAL_HOLDOUT_CHECK=0` to skip it. |
| - For checkpoints saved between two evals there is no logged value; `EVAL_HOLDOUT_CHECK=record` recomputes and stores the loss without comparing. |
| - The per-rank packing depends on the training world size. It is read from the number of `__*_0.distcp` files, or from `source_world_size` |
| in `snapshot_meta.json` / `final_meta.json` for single-file model-only exports (`tools/export_model_only_dcp.py`), or from `EVAL_HOLDOUT_TRAIN_WORLD`. |
|
|
| ## Scoring numerics |
|
|
| - MoE capacity is off while scoring (`MOE_SKIP_CAPACITY=1`). With capacity on, capacity is computed over the whole packed batch, |
| so a question's score depends on which other questions share its batch. |
| - Parameters are fp32 while scoring (`EVAL_PARAM_DTYPE=fp32`). In bf16, the expert outputs are combined in bf16 and rounding flips |
| the expert choice of individual tokens with batch size, which flips a few answers per task. fp32 runs through SDPA and is about 35% slower. |
| - With both settings, scores do not depend on `EVAL_BATCH_SIZE_MCQ` (default 32). |
|
|
| ## Code tree |
|
|
| Model code is imported only through `PYTHONPATH`: `run.sh` sets `PYTHONPATH=eval:$MODEL_CODE_ROOT`, and `MODEL_CODE_ROOT` |
| defaults to the repository root. `results.json` records the files actually imported, the commit and whether |
| `models/ utils/ pretrain.py` had uncommitted changes. If the imported files are not under `MODEL_CODE_ROOT`, the run aborts. |
|
|
| ## Settings |
|
|
| | Variable | Default | Meaning | |
| |---|---|---| |
| | `CKPT_PATH` | required (or first argument) | checkpoint directory | |
| | `CKPT_EPOCH` | largest `fsdp2_epoch_*` | epoch to load | |
| | `CKPT_CONFIG_DIR` | auto | directory with `all_config.yaml` and `train_metadata.yaml` | |
| | `MODEL_CODE_ROOT` | repository root | model code tree | |
| | `EVAL_OUTPUT_DIR` | see above | result directory | |
| | `NPROC_PER_NODE` | visible GPUs | GPUs per checkpoint (requests are split across them) | |
| | `EVAL_BATCH_SIZE_MCQ` | 32 | requests per forward pass | |
| | `EVAL_PARAM_DTYPE` | fp32 | parameter dtype while scoring (`bf16` / `fp32`) | |
| | `MOE_SKIP_CAPACITY` | 1 | disable MoE capacity while scoring | |
| | `EVAL_HOLDOUT_CHECK` | 1 | `0` skips the holdout check, `record` stores without comparing | |
| | `EVAL_HOLDOUT_MAX_GAP` | 0.005 | holdout tolerance | |
| | `LMEVAL_TASKS` | the 7 tasks | comma-separated | |
| | `LMEVAL_LIMIT` | empty (all) | first N examples per task, for smoke tests | |
| | `LMEVAL_LOG_SAMPLES` | 0 | `1` writes `samples_<task>.jsonl` | |
| | `EVAL_TOKENIZER_PATH` | from `train_metadata.yaml` | tokenizer override | |
| | `EVAL_HF_ROOT` | `eval/hf_cache` | HuggingFace cache for datasets and the tokenizer | |
| | `EVAL_OFFLINE` | 0 | `1` uses the cache only | |
|
|
| ## Files |
|
|
| ```text |
| eval/ |
| run.sh evaluate one checkpoint (single-node torchrun) |
| sweep.sh several checkpoints, one GPU each |
| collect.py result table |
| lm_eval_loom.py lm_eval model adapter, holdout check, result writing |
| loom_loader.py load a checkpoint from DCP, record code provenance |
| datasets_cache.py / distributed_utils.py |
| tasks/ Winogrande (winogrande_debiased, whose validation split is the standard 1,267 examples) |
| and SIQA (jet-ai/social_i_qa, same schema and 1,954 validation examples) task definitions |
| ``` |
|
|
