--- library_name: pytorch pipeline_tag: text-generation language: - en tags: - helm - hyperbolic - c4 - research --- # HELM C4 20B checkpoints and seed observations These are native research checkpoints for a 1B HELM model: 16 blocks, width 910, 966,957,719 parameters, fixed curvature `c=1`. Training uses the eager C4 20B recipe with BF16 autocast and FP32 parameters. Attention is not quantized in these runs. ## Available results | Run | Observed outcome | C4 PPL | WikiText PPL | Available state | | --- | --- | ---: | ---: | --- | | Scaled init, seed 1234 | Finished 152,580 steps | 19.198618161946133 | 52.69278098379149 | Native final model | | Published init + FP32 geometry, seed 1234 | Finished 152,580 steps | 20.370028561152527 | 50.02364864030602 | Native final `_pol` model | | Published init, seed 1234 | Nonfinite first flagged at 2,280; detected at 2,300 | unavailable | unavailable | Last saved finite model at step 2,000 | | Published init, seed 7 | Finite through requested 10,000-step cap | 36.28621540996052 | 84.48696841726898 | Native step-10,000 training resume | | Published init, seed 99 | Finite through requested 10,000-step cap | 37.023797483960436 | 88.28929618013538 | Native step-10,000 training resume | The seed-7/99 scores are **at step 10,000**, not final 20B scores. These seeds are censored observations; their later full-budget behavior is unknown. The original seed-1234 failure happened before its first scheduled evaluation. Its JSON intentionally retains `NaN` scores and an empty evaluation curve. The step-2,000 model is not a completed 20B checkpoint. The published-init `_pol` run also completed the full budget. It protects `cinner` and `lorentzian_centroid` in FP32 while the rest of the BF16 recipe, initialization, schedule and budget remain unchanged. See `geometry_policy.json` and the runtime selection below. Checkpoint tensors do not serialize the method wrappers. The original full-horizon attempt's final verifier crashed before final artifact upload. These final weights come from a verified continuation of the saved step-152,000 state: the last 580 updates were repeated with the same model, policy, seed, optimizer/RNG, data and full learning-rate schedule. This is a new continuation result; it is not claimed to recover identical original node-local final bytes. The forthcoming quantized 2B experiments are not present in this repository. ## Recipe and provenance - HELM source: [Graph-and-Geometric-Learning/helm](https://github.com/Graph-and-Geometric-Learning/helm), revision `7501deca8f413848bfef804be64ce874b72a3cd7`. - Trainer recipe revision: `5f968bd8d023298728e3ac43d6fefe64f786de8e`. - Sequence length 2,048, effective batch 64, 152,580 planned optimizer steps: 19,998,965,760 processed tokens for a completed full run. - AdamW `(0.9, 0.95)`, weight decay 0.1, pre-clip norm limit 1.0, peak LR `1e-3`, 3,051-step warmup, cosine decay to `1e-4`. - C4 and WikiText evaluation every 5,000 steps plus the final step, using 256 held-out sequences each. The supplied tokenizer is `NousResearch/Meta-Llama-3-8B`, EOS ID 128001. - Python 3.12.3, Torch 2.11.0+cu128, CUDA 12.8, NVIDIA B200; TF32 off. Native per-run JSON is in `cells.d/` and is preserved byte for byte, including its original worker-local paths. Curves and selected scientific log lines are provided separately. Internal operational/custody receipts are excluded. ## Terminal verifier recovery for the capped seeds Both seed-7/99 trainers completed their 10,000-step evaluation and resume save and joined all eight ranks. Our terminal verifier then incorrectly required every saved model-state tensor to be FP32; the native model has a valid Boolean causal `attn_mask` buffer. The worker failed after the scientific outputs were already durable. A separate verification confirmed the original parameter/buffer inventory, causal mask, finite tensors, optimizer, random-generator state, full-budget LR and saved evaluations. The published resume bytes are unchanged. This recovery did not rerun training or cast the saved resume state. ## Native checkpoint loading Final/last-saved model files under `checkpoints/` use: ```python {"key": "the native experiment key", "state_dict": {...}} ``` Use the pinned HELM source and a compatible Torch environment. A minimal state read, which does not require CUDA, is: ```python from huggingface_hub import hf_hub_download import torch path = hf_hub_download( repo_id="zimplex/helm-c4-20b-checkpoints", filename="checkpoints/ckpt_scaled_bf16_d910_c1_lr0.001_acc1_s1234_st152580_ev5000_corpc420b_perm_cos_mb8_w8.pt", ) checkpoint = torch.load(path, map_location="cpu", weights_only=True) assert set(checkpoint) == {"key", "state_dict"} print(checkpoint["key"]) ``` For model construction and inference, follow the full pinned training recipe and HELM source, including its distributed/module initialization. These checkpoints were produced on eight GPUs; a standalone model loader has not been qualified here. Learned normalizer weights are already included; do not rescale them after loading. This is the native research checkpoint format; the example above verifies and reads its payload. ### Selecting the `_pol` geometry precision For the `_pol` model, select the same two protected methods in downstream code. The following is the method wrapper used by the pinned trainer; it does not construct a model or change the learned weights: ```python import torch from helm.hypercore.manifolds import Lorentz def protect_fp32(cls, name): original = getattr(cls, name) def wrapped(self, *args, **kwargs): with torch.autocast(device_type="cuda", enabled=False): args = tuple(x.float() if torch.is_tensor(x) else x for x in args) kwargs = {k: (x.float() if torch.is_tensor(x) else x) for k, x in kwargs.items()} return original(self, *args, **kwargs) setattr(cls, name, wrapped) protect_fp32(Lorentz, "cinner") protect_fp32(Lorentz, "lorentzian_centroid") ``` Apply this policy explicitly before using the `_pol` model. The remaining training operations used BF16 autocast, with FP32 parameters. Follow the pinned recipe for model/distributed initialization; this snippet is not a standalone model loader. Files under `resumes/` have the original training-state payload with `model`, `opt`, sampler/generator state, CPU/CUDA RNG state and evaluation history. They are for restoration through the original trainer, not the two-key model loader above. Preserve the 152,580-step schedule when continuing a capped seed. ## Checksums and health evidence `CHECKSUMS.json` records public filenames, byte counts and SHA-256 hashes. `eval_curves/` includes full available trajectories and Scaled's exact matched step-5,000/10,000 values. `health/` contains original seed-1234 health intervals wholly before step 2,280. Its later 2,001–2,300 aggregate crosses that boundary, so it is not included as strictly pre-failure evidence.