zimplex's picture
Upload verified HELM artifacts [2d0602a5e5934c3dbf76eb922bdd1ead]
64b60ee verified
|
Raw History Blame Contribute Delete
2.27 kB
# Loading the native model-only checkpoints
Use the pinned HELM/model configuration and evaluator supplied with PR2
`5edb20302017712ac97bf39374f66db80f94bc51` (HELM `7501deca8f413848bfef804be64ce874b72a3cd7`). Checkpoint state is FP32 and has exactly two keys:
`key` and `state_dict`. The training environment used Python 3.12.3,
PyTorch 2.11.0+cu128 and CUDA 12.8.
```python
import torch
payload = torch.load(checkpoint_path, map_location="cpu", weights_only=True)
assert set(payload) == {"key", "state_dict"}
```
These are custom HELM native archives. Load them through the supplied HELM
evaluator; they are not transformers save_pretrained/AutoModel repositories.
Construct the matching width-910, 16-layer HELM with curvature1 and the
configuration used by the supplied evaluator, then load state_dict strictly.
Training used world8; preserve the corresponding model configuration when
adapting the evaluator. Evaluate with the native BF16 autocast context while
retaining FP32 parameter storage.
Follow the original init-dependent construction/radius recipe before loading
the learned state. Scaled runs recorded R_init=u_init=1; published native BF16
seeds7/99 recorded R_init=u_init=30.1496. Each profiles/*.json and unchanged
native result retains the measured normalizer radii. Do not reapply initializer
scaling to learned weights after loading them.
For Scaled4, install the supplied PR2 hooks once in a fresh evaluation process:
```python
from helm.hypercore.manifolds import Lorentz
import lowp_emul
lowp_emul.install(Lorentz, fmt="e2m1", block=32, ops="both")
```
For Scaled8, use fmt="e4m3". Install the hooks before model construction.
Native BF16 seeds7/99 use neither lowp hooks nor the FP32 geometry policy.
The bit-width suffix identifies operand emulation, not quantized weight storage.
Run each profile in a fresh process so different wrappers do not stack.
CHECKPOINTS.json distinguishes the seed99 full-budget final checkpoint from
the three exports of earlier finite saved states. The latter are NOT FINAL
and must be reported with their actual saved step. Optimizer/RNG resumes are
not included here. The native scalar final PPL is NaN for failed runs; the
last finite evaluation in RESULTS.json is a separate earlier observation.