zimplex's picture
Upload verified HELM artifacts [2d0602a5e5934c3dbf76eb922bdd1ead]
64b60ee verified
|
Raw History Blame Contribute Delete
2.27 kB

Loading the native model-only checkpoints

Use the pinned HELM/model configuration and evaluator supplied with PR2 5edb20302017712ac97bf39374f66db80f94bc51 (HELM 7501deca8f413848bfef804be64ce874b72a3cd7). Checkpoint state is FP32 and has exactly two keys: key and state_dict. The training environment used Python 3.12.3, PyTorch 2.11.0+cu128 and CUDA 12.8.

import torch
payload = torch.load(checkpoint_path, map_location="cpu", weights_only=True)
assert set(payload) == {"key", "state_dict"}

These are custom HELM native archives. Load them through the supplied HELM evaluator; they are not transformers save_pretrained/AutoModel repositories. Construct the matching width-910, 16-layer HELM with curvature1 and the configuration used by the supplied evaluator, then load state_dict strictly. Training used world8; preserve the corresponding model configuration when adapting the evaluator. Evaluate with the native BF16 autocast context while retaining FP32 parameter storage.

Follow the original init-dependent construction/radius recipe before loading the learned state. Scaled runs recorded R_init=u_init=1; published native BF16 seeds7/99 recorded R_init=u_init=30.1496. Each profiles/*.json and unchanged native result retains the measured normalizer radii. Do not reapply initializer scaling to learned weights after loading them.

For Scaled4, install the supplied PR2 hooks once in a fresh evaluation process:

from helm.hypercore.manifolds import Lorentz
import lowp_emul
lowp_emul.install(Lorentz, fmt="e2m1", block=32, ops="both")

For Scaled8, use fmt="e4m3". Install the hooks before model construction. Native BF16 seeds7/99 use neither lowp hooks nor the FP32 geometry policy. The bit-width suffix identifies operand emulation, not quantized weight storage. Run each profile in a fresh process so different wrappers do not stack.

CHECKPOINTS.json distinguishes the seed99 full-budget final checkpoint from the three exports of earlier finite saved states. The latter are NOT FINAL and must be reported with their actual saved step. Optimizer/RNG resumes are not included here. The native scalar final PPL is NaN for failed runs; the last finite evaluation in RESULTS.json is a separate earlier observation.