Scope: training-produced failed checkpoints only (probe excluded)
Browse files
README.md
CHANGED
|
@@ -1,47 +1,43 @@
|
|
| 1 |
-
# RetNet-0.5B — FAILED
|
| 2 |
|
| 3 |
> ## ⚠️ DO NOT USE THESE WEIGHTS
|
| 4 |
>
|
| 5 |
-
>
|
| 6 |
-
>
|
| 7 |
-
> so nobody repeats them. If you want a working model, use
|
| 8 |
> [`Ayaka157/RetNet-0.5B`](https://huggingface.co/Ayaka157/RetNet-0.5B).
|
| 9 |
|
| 10 |
-
|
| 11 |
-
20 layers, 22 heads, vocab 20000) — they load. They just do not work.
|
| 12 |
-
|
| 13 |
-
| Folder | What it is | Why it fails |
|
| 14 |
| --- | --- | --- |
|
| 15 |
-
| `
|
| 16 |
-
| `02_retnet_500m_growth_product/` | 272M -> 500M width-growth output | The growth initialiser wrote the new columns **and** the new gate to zero, so **338,120,704 parameters were dead on arrival** and could never receive gradient. Roughly 67% of all weights are zero. |
|
| 17 |
-
| `03_v21_export/` | v21 training line, exported | The same dead-parameter inheritance, plus `msr.gamma` was left as a bare parameter under `weight_decay=0.1` because `--no_decay_1d` defaulted off and the v21 config never set it. Median gamma over 440 heads fell to **0.5656**, and 10 heads are frozen at 0.213622 through pure decay with zero gradient. |
|
| 18 |
|
| 19 |
-
##
|
| 20 |
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
then `dL/dW_newcol = upstream x 0 = 0`, and because the new inputs are themselves zero,
|
| 24 |
-
`gamma_new` receives no gradient either. The new capacity is permanently inert. Initialising
|
| 25 |
-
`gamma_new = 1.0` would have been equally function-preserving at step 0 and would have let
|
| 26 |
-
the channels wake up. That counterfactual is **derived, never tested** — see the main
|
| 27 |
-
repository's `docs/00_错误总表.md` (E01–E03, E07) and `docs/00F1_修复对照.md`.
|
| 28 |
|
| 29 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
``
|
|
|
|
|
|
|
|
|
|
| 36 |
|
| 37 |
## Verification
|
| 38 |
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
was repaired before publication.
|
| 43 |
|
| 44 |
## Licence
|
| 45 |
|
| 46 |
-
MIT, same as the parent project. Provided strictly as a record of failure, with no
|
| 47 |
-
|
|
|
|
| 1 |
+
# RetNet-0.5B — FAILED training checkpoint
|
| 2 |
|
| 3 |
> ## ⚠️ DO NOT USE THESE WEIGHTS
|
| 4 |
>
|
| 5 |
+
> `01_v21_export/` is a **failed training run**, published only so the failure stays
|
| 6 |
+
> auditable. It loads, but it is not a usable model. If you want a working model, use
|
|
|
|
| 7 |
> [`Ayaka157/RetNet-0.5B`](https://huggingface.co/Ayaka157/RetNet-0.5B).
|
| 8 |
|
| 9 |
+
| Folder | Produced by | Why it fails |
|
|
|
|
|
|
|
|
|
|
| 10 |
| --- | --- | --- |
|
| 11 |
+
| `01_v21_export/` | the v21 DDP training run | It inherits **338,120,704 dead parameters** from the width-growth step, and `msr.gamma` was left as a bare parameter under `weight_decay=0.1` because `--no_decay_1d` defaulted off and the v21 config never set the field. Median gamma over 440 heads fell to **0.5656**, with 10 heads frozen at 0.213622 through pure decay with zero gradient. |
|
|
|
|
|
|
|
| 12 |
|
| 13 |
+
## What is deliberately NOT in this repository
|
| 14 |
|
| 15 |
+
Publishing only training-produced checkpoints is a deliberate policy here, so two other
|
| 16 |
+
defective artifacts from the same lineage are **not** published:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
+
- **`probe_500m`** — its weights are **random**. Its own `config.json` `_note` reads:
|
| 19 |
+
*"probe checkpoint: random weights, only for VRAM and step-time measurement, not usable
|
| 20 |
+
for evaluation."* It is not a trained model and does not belong in a model repository.
|
| 21 |
+
- **`retnet_500m`** — the 272M -> 500M width-growth **product**. It has no
|
| 22 |
+
`trainer_state.json` because no training produced it; it is the output of
|
| 23 |
+
`scripts/grow_500m.py`. This is the step where the dead parameters were actually created.
|
| 24 |
|
| 25 |
+
The mechanism, because it explains both failures: widening a checkpoint sets the new input
|
| 26 |
+
columns to zero so the wider model computes exactly what the narrower one did. But if the
|
| 27 |
+
new gate `gamma_new` is also initialised to zero, then `dL/dW_newcol = upstream x 0 = 0`,
|
| 28 |
+
and since the new inputs are themselves zero, `gamma_new` receives no gradient either. The
|
| 29 |
+
new capacity is permanently inert. Initialising `gamma_new = 1.0` would have been equally
|
| 30 |
+
function-preserving at step 0 and would have let the channels wake up. That counterfactual
|
| 31 |
+
is **derived, never tested** — see the main repository's `docs/00_错误总表.md` (E01–E03, E07)
|
| 32 |
+
and `docs/00F1_修复对照.md`.
|
| 33 |
|
| 34 |
## Verification
|
| 35 |
|
| 36 |
+
`config.json` ships alongside the weights so the architecture travels with them. SHA-256 of
|
| 37 |
+
`model.safetensors` is recorded in `MANIFEST.sha256`. These are the exact bytes on disk when
|
| 38 |
+
the failure was audited; nothing was repaired before publication.
|
|
|
|
| 39 |
|
| 40 |
## Licence
|
| 41 |
|
| 42 |
+
MIT, same as the parent project. Provided strictly as a record of failure, with no warranty
|
| 43 |
+
and no fitness for any purpose.
|