| --- |
| library_name: pytorch |
| pipeline_tag: text-generation |
| language: |
| - en |
| tags: |
| - helm |
| - hyperbolic |
| - c4 |
| - research |
| --- |
| |
| # HELM C4 20B checkpoints and seed observations |
|
|
| These are native research checkpoints for a 1B HELM model: 16 blocks, width |
| 910, 966,957,719 parameters, fixed curvature `c=1`. Training uses the eager |
| C4 20B recipe with BF16 autocast and FP32 parameters. Attention is not |
| quantized in these runs. |
|
|
| ## Available results |
|
|
| | Run | Observed outcome | C4 PPL | WikiText PPL | Available state | |
| | --- | --- | ---: | ---: | --- | |
| | Scaled init, seed 1234 | Finished 152,580 steps | 19.198618161946133 | 52.69278098379149 | Native final model | |
| | Published init + FP32 geometry, seed 1234 | Finished 152,580 steps | 20.370028561152527 | 50.02364864030602 | Native final `_pol` model | |
| | Published init, seed 1234 | Nonfinite first flagged at 2,280; detected at 2,300 | unavailable | unavailable | Last saved finite model at step 2,000 | |
| | Published init, seed 7 | Finite through requested 10,000-step cap | 36.28621540996052 | 84.48696841726898 | Native step-10,000 training resume | |
| | Published init, seed 99 | Finite through requested 10,000-step cap | 37.023797483960436 | 88.28929618013538 | Native step-10,000 training resume | |
|
|
| The seed-7/99 scores are **at step 10,000**, not final 20B scores. These seeds |
| are censored observations; their later full-budget behavior is unknown. |
| The original seed-1234 failure happened before its first scheduled evaluation. |
| Its JSON intentionally retains `NaN` scores and an empty evaluation curve. |
| The step-2,000 model is not a completed 20B checkpoint. |
|
|
| The published-init `_pol` run also completed the full budget. It protects |
| `cinner` and `lorentzian_centroid` in FP32 while the rest of the BF16 recipe, |
| initialization, schedule and budget remain unchanged. See |
| `geometry_policy.json` and the runtime selection below. Checkpoint tensors |
| do not serialize the method wrappers. |
|
|
| The original full-horizon attempt's final verifier crashed before final |
| artifact upload. These final weights come from a verified continuation of |
| the saved step-152,000 state: the last 580 updates were repeated with the |
| same model, policy, seed, optimizer/RNG, data and full learning-rate schedule. |
| This is a new continuation result; it is not claimed to recover identical |
| original node-local final bytes. |
|
|
| The forthcoming quantized 2B experiments are not present in this repository. |
|
|
| ## Recipe and provenance |
|
|
| - HELM source: [Graph-and-Geometric-Learning/helm](https://github.com/Graph-and-Geometric-Learning/helm), revision `7501deca8f413848bfef804be64ce874b72a3cd7`. |
| - Trainer recipe revision: `5f968bd8d023298728e3ac43d6fefe64f786de8e`. |
| - Sequence length 2,048, effective batch 64, 152,580 planned optimizer steps: |
| 19,998,965,760 processed tokens for a completed full run. |
| - AdamW `(0.9, 0.95)`, weight decay 0.1, pre-clip norm limit 1.0, peak LR |
| `1e-3`, 3,051-step warmup, cosine decay to `1e-4`. |
| - C4 and WikiText evaluation every 5,000 steps plus the final step, using |
| 256 held-out sequences each. The supplied tokenizer is |
| `NousResearch/Meta-Llama-3-8B`, EOS ID 128001. |
| - Python 3.12.3, Torch 2.11.0+cu128, CUDA 12.8, NVIDIA B200; TF32 off. |
|
|
| Native per-run JSON is in `cells.d/` and is preserved byte for byte, including |
| its original worker-local paths. Curves and selected scientific log lines |
| are provided separately. Internal operational/custody receipts are excluded. |
|
|
| ## Terminal verifier recovery for the capped seeds |
|
|
| Both seed-7/99 trainers completed their 10,000-step evaluation and resume |
| save and joined all eight ranks. Our terminal verifier then incorrectly |
| required every saved model-state tensor to be FP32; the native model has a |
| valid Boolean causal `attn_mask` buffer. The worker failed after the |
| scientific outputs were already durable. A separate verification confirmed |
| the original parameter/buffer inventory, causal mask, finite tensors, |
| optimizer, random-generator state, full-budget LR and saved evaluations. |
| The published resume bytes are unchanged. This recovery did not rerun |
| training or cast the saved resume state. |
|
|
| ## Native checkpoint loading |
|
|
| Final/last-saved model files under `checkpoints/` use: |
|
|
| ```python |
| {"key": "the native experiment key", "state_dict": {...}} |
| ``` |
|
|
| Use the pinned HELM source and a compatible Torch environment. A minimal |
| state read, which does not require CUDA, is: |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| import torch |
| |
| path = hf_hub_download( |
| repo_id="zimplex/helm-c4-20b-checkpoints", |
| filename="checkpoints/ckpt_scaled_bf16_d910_c1_lr0.001_acc1_s1234_st152580_ev5000_corpc420b_perm_cos_mb8_w8.pt", |
| ) |
| checkpoint = torch.load(path, map_location="cpu", weights_only=True) |
| assert set(checkpoint) == {"key", "state_dict"} |
| print(checkpoint["key"]) |
| ``` |
|
|
| For model construction and inference, follow the full pinned training |
| recipe and HELM source, including its distributed/module initialization. |
| These checkpoints were produced on eight GPUs; a standalone model loader |
| has not been qualified here. Learned normalizer weights are already |
| included; do not rescale them after loading. This is the native research |
| checkpoint format; the example above verifies and reads its payload. |
|
|
|
|
| ### Selecting the `_pol` geometry precision |
| |
| For the `_pol` model, select the same two protected methods in downstream |
| code. The following is the method wrapper used by the pinned trainer; it |
| does not construct a model or change the learned weights: |
|
|
| ```python |
| import torch |
| from helm.hypercore.manifolds import Lorentz |
| |
| def protect_fp32(cls, name): |
| original = getattr(cls, name) |
| def wrapped(self, *args, **kwargs): |
| with torch.autocast(device_type="cuda", enabled=False): |
| args = tuple(x.float() if torch.is_tensor(x) else x for x in args) |
| kwargs = {k: (x.float() if torch.is_tensor(x) else x) |
| for k, x in kwargs.items()} |
| return original(self, *args, **kwargs) |
| setattr(cls, name, wrapped) |
| |
| protect_fp32(Lorentz, "cinner") |
| protect_fp32(Lorentz, "lorentzian_centroid") |
| ``` |
|
|
| Apply this policy explicitly before using the `_pol` model. The remaining |
| training operations used BF16 autocast, with FP32 parameters. Follow the |
| pinned recipe for model/distributed initialization; this snippet is not a |
| standalone model loader. |
|
|
|
|
| Files under `resumes/` have the original training-state payload with `model`, |
| `opt`, sampler/generator state, CPU/CUDA RNG state and evaluation history. |
| They are for restoration through the original trainer, not the two-key model |
| loader above. Preserve the 152,580-step schedule when continuing a capped seed. |
|
|
| ## Checksums and health evidence |
|
|
| `CHECKSUMS.json` records public filenames, byte counts and SHA-256 hashes. |
| `eval_curves/` includes full available trajectories and Scaled's exact |
| matched step-5,000/10,000 values. `health/` contains original seed-1234 health |
| intervals wholly before step 2,280. Its later 2,001–2,300 aggregate crosses |
| that boundary, so it is not included as strictly pre-failure evidence. |
|
|
