# RetNet-0.5B — FAILED training checkpoint > ## ⚠️ DO NOT USE THESE WEIGHTS > > `01_v21_export/` is a **failed training run**, published only so the failure stays > auditable. It loads, but it is not a usable model. If you want a working model, use > [`Ayaka157/RetNet-0.5B`](https://huggingface.co/Ayaka157/RetNet-0.5B). | Folder | Produced by | Why it fails | | --- | --- | --- | | `01_v21_export/` | the v21 DDP training run | It inherits **338,120,704 dead parameters** from the width-growth step, and `msr.gamma` was left as a bare parameter under `weight_decay=0.1` because `--no_decay_1d` defaulted off and the v21 config never set the field. Median gamma over 440 heads fell to **0.5656**, with 10 heads frozen at 0.213622 through pure decay with zero gradient. | ## What is deliberately NOT in this repository Publishing only training-produced checkpoints is a deliberate policy here, so two other defective artifacts from the same lineage are **not** published: - **`probe_500m`** — its weights are **random**. Its own `config.json` `_note` reads: *"probe checkpoint: random weights, only for VRAM and step-time measurement, not usable for evaluation."* It is not a trained model and does not belong in a model repository. - **`retnet_500m`** — the 272M -> 500M width-growth **product**. It has no `trainer_state.json` because no training produced it; it is the output of `scripts/grow_500m.py`. This is the step where the dead parameters were actually created. The mechanism, because it explains both failures: widening a checkpoint sets the new input columns to zero so the wider model computes exactly what the narrower one did. But if the new gate `gamma_new` is also initialised to zero, then `dL/dW_newcol = upstream x 0 = 0`, and since the new inputs are themselves zero, `gamma_new` receives no gradient either. The new capacity is permanently inert. Initialising `gamma_new = 1.0` would have been equally function-preserving at step 0 and would have let the channels wake up. That counterfactual is **derived, never tested** — see the main repository's `docs/00_错误总表.md` (E01–E03, E07) and `docs/00F1_修复对照.md`. ## Verification `config.json` ships alongside the weights so the architecture travels with them. SHA-256 of `model.safetensors` is recorded in `MANIFEST.sha256`. These are the exact bytes on disk when the failure was audited; nothing was repaired before publication. ## Licence MIT, same as the parent project. Provided strictly as a record of failure, with no warranty and no fitness for any purpose.