Ayaka157 commited on
Commit
2eb4554
·
verified ·
1 Parent(s): 58ce6fa

Scope: training-produced failed checkpoints only (probe excluded)

Browse files
Files changed (1) hide show
  1. README.md +27 -31
README.md CHANGED
@@ -1,47 +1,43 @@
1
- # RetNet-0.5B — FAILED / NON-USABLE checkpoints
2
 
3
  > ## ⚠️ DO NOT USE THESE WEIGHTS
4
  >
5
- > Every checkpoint in this repository is **defective or non-functional**. None of them
6
- > can produce a usable model. They are published only so the failures stay auditable and
7
- > so nobody repeats them. If you want a working model, use
8
  > [`Ayaka157/RetNet-0.5B`](https://huggingface.co/Ayaka157/RetNet-0.5B).
9
 
10
- All three checkpoints are architecturally valid (`RetNetForCausalLM`, d_model 1408,
11
- 20 layers, 22 heads, vocab 20000) — they load. They just do not work.
12
-
13
- | Folder | What it is | Why it fails |
14
  | --- | --- | --- |
15
- | `01_probe_500m/` | Scale probe | **Weights are random.** Its own `config.json` `_note` says: *"probe checkpoint: random weights, only for VRAM and step-time measurement, not usable for evaluation."* |
16
- | `02_retnet_500m_growth_product/` | 272M -> 500M width-growth output | The growth initialiser wrote the new columns **and** the new gate to zero, so **338,120,704 parameters were dead on arrival** and could never receive gradient. Roughly 67% of all weights are zero. |
17
- | `03_v21_export/` | v21 training line, exported | The same dead-parameter inheritance, plus `msr.gamma` was left as a bare parameter under `weight_decay=0.1` because `--no_decay_1d` defaulted off and the v21 config never set it. Median gamma over 440 heads fell to **0.5656**, and 10 heads are frozen at 0.213622 through pure decay with zero gradient. |
18
 
19
- ## The mechanism, in one paragraph
20
 
21
- Widening a checkpoint sets new input columns to zero so the wider model computes exactly
22
- what the narrower one did. But if the new gate `gamma_new` is also initialised to zero,
23
- then `dL/dW_newcol = upstream x 0 = 0`, and because the new inputs are themselves zero,
24
- `gamma_new` receives no gradient either. The new capacity is permanently inert. Initialising
25
- `gamma_new = 1.0` would have been equally function-preserving at step 0 and would have let
26
- the channels wake up. That counterfactual is **derived, never tested** — see the main
27
- repository's `docs/00_错误总表.md` (E01–E03, E07) and `docs/00F1_修复对照.md`.
28
 
29
- ## Measured dead-parameter count
 
 
 
 
 
30
 
31
- ```
32
- 338,120,704 dead parameters in the growth product (E01-E03)
33
- 67.08% of all weight elements are exactly zero
34
- gamma: median 0.5656 across 440 heads; 10 heads pinned at 0.213622
35
- ```
 
 
 
36
 
37
  ## Verification
38
 
39
- Each folder carries its `config.json` so the architecture travels with the weights.
40
- SHA-256 of every `model.safetensors` is recorded in `MANIFEST.sha256` at the repository
41
- root. These are the exact bytes that were on disk when the failures were audited; nothing
42
- was repaired before publication.
43
 
44
  ## Licence
45
 
46
- MIT, same as the parent project. Provided strictly as a record of failure, with no
47
- warranty of any kind and no fitness for any purpose.
 
1
+ # RetNet-0.5B — FAILED training checkpoint
2
 
3
  > ## ⚠️ DO NOT USE THESE WEIGHTS
4
  >
5
+ > `01_v21_export/` is a **failed training run**, published only so the failure stays
6
+ > auditable. It loads, but it is not a usable model. If you want a working model, use
 
7
  > [`Ayaka157/RetNet-0.5B`](https://huggingface.co/Ayaka157/RetNet-0.5B).
8
 
9
+ | Folder | Produced by | Why it fails |
 
 
 
10
  | --- | --- | --- |
11
+ | `01_v21_export/` | the v21 DDP training run | It inherits **338,120,704 dead parameters** from the width-growth step, and `msr.gamma` was left as a bare parameter under `weight_decay=0.1` because `--no_decay_1d` defaulted off and the v21 config never set the field. Median gamma over 440 heads fell to **0.5656**, with 10 heads frozen at 0.213622 through pure decay with zero gradient. |
 
 
12
 
13
+ ## What is deliberately NOT in this repository
14
 
15
+ Publishing only training-produced checkpoints is a deliberate policy here, so two other
16
+ defective artifacts from the same lineage are **not** published:
 
 
 
 
 
17
 
18
+ - **`probe_500m`** — its weights are **random**. Its own `config.json` `_note` reads:
19
+ *"probe checkpoint: random weights, only for VRAM and step-time measurement, not usable
20
+ for evaluation."* It is not a trained model and does not belong in a model repository.
21
+ - **`retnet_500m`** — the 272M -> 500M width-growth **product**. It has no
22
+ `trainer_state.json` because no training produced it; it is the output of
23
+ `scripts/grow_500m.py`. This is the step where the dead parameters were actually created.
24
 
25
+ The mechanism, because it explains both failures: widening a checkpoint sets the new input
26
+ columns to zero so the wider model computes exactly what the narrower one did. But if the
27
+ new gate `gamma_new` is also initialised to zero, then `dL/dW_newcol = upstream x 0 = 0`,
28
+ and since the new inputs are themselves zero, `gamma_new` receives no gradient either. The
29
+ new capacity is permanently inert. Initialising `gamma_new = 1.0` would have been equally
30
+ function-preserving at step 0 and would have let the channels wake up. That counterfactual
31
+ is **derived, never tested** — see the main repository's `docs/00_错误总表.md` (E01–E03, E07)
32
+ and `docs/00F1_修复对照.md`.
33
 
34
  ## Verification
35
 
36
+ `config.json` ships alongside the weights so the architecture travels with them. SHA-256 of
37
+ `model.safetensors` is recorded in `MANIFEST.sha256`. These are the exact bytes on disk when
38
+ the failure was audited; nothing was repaired before publication.
 
39
 
40
  ## Licence
41
 
42
+ MIT, same as the parent project. Provided strictly as a record of failure, with no warranty
43
+ and no fitness for any purpose.