File size: 5,212 Bytes
d5147ba | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 | # Zero-shot evaluation
`lm_eval.simple_evaluate` (lm-eval 0.4.11), 0-shot, fully causal sum log-likelihood over the continuation.
Tasks: ARC-Easy, ARC-Challenge, HellaSwag, OpenBookQA, PIQA, Winogrande, SIQA.
Metrics: `acc_norm` for ARC / HellaSwag / OpenBookQA / PIQA, `acc` for Winogrande / SIQA.
## Evaluate one checkpoint
```bash
bash eval/run.sh checkpoints/<project>/<run> # run directory
bash eval/run.sh checkpoints/<project>/<run>/snapshots/step_0010000
```
`CKPT_PATH` (or the first argument) may be a run directory, a snapshot under `<run>/snapshots/`,
or an archive directory holding `fsdp2_epoch_N` plus a `final_meta.json` whose `source` names the run directory.
Results go to `<ckpt>/eval_results/official_lmeval7_<timestamp>/`; for a snapshot they go to
`<run>/eval_results/step_NNNNNNN/...`, because snapshots are pruned by `SNAPSHOT_KEEP_LAST`.
Each result directory has `results.json` (scores, checkpoint info, code provenance, holdout check), `summary.txt` and `eval.log`.
Datasets are downloaded on first use into `eval/hf_cache` (`EVAL_HF_ROOT` to move it, `EVAL_OFFLINE=1` to forbid network access afterwards).
## Several checkpoints, one GPU each
```bash
bash eval/sweep.sh ckpt_H3 ckpt_H6 ckpt_H9 ckpt_H12 # prints the table when all finish
python eval/collect.py ckpt_H3 ckpt_H6 # re-print from the newest results
```
## Holdout check
Before scoring, the checkpoint's own `eval/loss` is recomputed exactly as `pretrain.py` computed it during training
(same holdout batches, same per-rank packing, 1M-token budget, bf16 parameters, MoE capacity on) and compared with the value
logged at that step in `tensorboard_scalars.json`. A gap above `EVAL_HOLDOUT_MAX_GAP` (default 0.005) aborts the run:
it means the weights, tokenizer or code tree do not match, not that the model is weak.
- The check needs the training data (`data.path` / `data.val_path` in the run's `all_config.yaml`). Use `EVAL_HOLDOUT_CHECK=0` to skip it.
- For checkpoints saved between two evals there is no logged value; `EVAL_HOLDOUT_CHECK=record` recomputes and stores the loss without comparing.
- The per-rank packing depends on the training world size. It is read from the number of `__*_0.distcp` files, or from `source_world_size`
in `snapshot_meta.json` / `final_meta.json` for single-file model-only exports (`tools/export_model_only_dcp.py`), or from `EVAL_HOLDOUT_TRAIN_WORLD`.
## Scoring numerics
- MoE capacity is off while scoring (`MOE_SKIP_CAPACITY=1`). With capacity on, capacity is computed over the whole packed batch,
so a question's score depends on which other questions share its batch.
- Parameters are fp32 while scoring (`EVAL_PARAM_DTYPE=fp32`). In bf16, the expert outputs are combined in bf16 and rounding flips
the expert choice of individual tokens with batch size, which flips a few answers per task. fp32 runs through SDPA and is about 35% slower.
- With both settings, scores do not depend on `EVAL_BATCH_SIZE_MCQ` (default 32).
## Code tree
Model code is imported only through `PYTHONPATH`: `run.sh` sets `PYTHONPATH=eval:$MODEL_CODE_ROOT`, and `MODEL_CODE_ROOT`
defaults to the repository root. `results.json` records the files actually imported, the commit and whether
`models/ utils/ pretrain.py` had uncommitted changes. If the imported files are not under `MODEL_CODE_ROOT`, the run aborts.
## Settings
| Variable | Default | Meaning |
|---|---|---|
| `CKPT_PATH` | required (or first argument) | checkpoint directory |
| `CKPT_EPOCH` | largest `fsdp2_epoch_*` | epoch to load |
| `CKPT_CONFIG_DIR` | auto | directory with `all_config.yaml` and `train_metadata.yaml` |
| `MODEL_CODE_ROOT` | repository root | model code tree |
| `EVAL_OUTPUT_DIR` | see above | result directory |
| `NPROC_PER_NODE` | visible GPUs | GPUs per checkpoint (requests are split across them) |
| `EVAL_BATCH_SIZE_MCQ` | 32 | requests per forward pass |
| `EVAL_PARAM_DTYPE` | fp32 | parameter dtype while scoring (`bf16` / `fp32`) |
| `MOE_SKIP_CAPACITY` | 1 | disable MoE capacity while scoring |
| `EVAL_HOLDOUT_CHECK` | 1 | `0` skips the holdout check, `record` stores without comparing |
| `EVAL_HOLDOUT_MAX_GAP` | 0.005 | holdout tolerance |
| `LMEVAL_TASKS` | the 7 tasks | comma-separated |
| `LMEVAL_LIMIT` | empty (all) | first N examples per task, for smoke tests |
| `LMEVAL_LOG_SAMPLES` | 0 | `1` writes `samples_<task>.jsonl` |
| `EVAL_TOKENIZER_PATH` | from `train_metadata.yaml` | tokenizer override |
| `EVAL_HF_ROOT` | `eval/hf_cache` | HuggingFace cache for datasets and the tokenizer |
| `EVAL_OFFLINE` | 0 | `1` uses the cache only |
## Files
```text
eval/
run.sh evaluate one checkpoint (single-node torchrun)
sweep.sh several checkpoints, one GPU each
collect.py result table
lm_eval_loom.py lm_eval model adapter, holdout check, result writing
loom_loader.py load a checkpoint from DCP, record code provenance
datasets_cache.py / distributed_utils.py
tasks/ Winogrande (winogrande_debiased, whose validation split is the standard 1,267 examples)
and SIQA (jet-ai/social_i_qa, same schema and 1,954 validation examples) task definitions
```
|