Remove internal note references from train_config.py
Browse filesComments only: references to internal notes are removed or reworded. No configuration value, weight or config.json changes.
attnconv_b7_qwen3_4b/step_2616/train_config.py
CHANGED
|
@@ -115,14 +115,14 @@ The consequence for reading the result: the step-1 paired gate licenses
|
|
| 115 |
the four corrections bought ON TOP OF the attention head -- is still answerable, just not
|
| 116 |
from the init: it comes from the held-out paired eval, which compares FINISHED checkpoints
|
| 117 |
and never touches initialisation. That is the ruler the verdict rests on anyway; the
|
| 118 |
-
shared init is worth exactly one step
|
| 119 |
|
| 120 |
EPOCH COUNT IS THE OPEN DECISION. Written at 1 epoch so it reproduces the ladder's unit
|
| 121 |
and is directly paired against every arm already on it. A longer run CONTAINS this one:
|
| 122 |
checkpoint at 2616 and the 1-epoch verdict comes out of it for free, which is why there is
|
| 123 |
no reason to run a separate 1-epoch arm first. If the epoch count changes, change
|
| 124 |
-
`max_train_steps` AND `lr_schedule_steps` together --
|
| 125 |
-
arm
|
| 126 |
"""
|
| 127 |
import os
|
| 128 |
|
|
|
|
| 115 |
the four corrections bought ON TOP OF the attention head -- is still answerable, just not
|
| 116 |
from the init: it comes from the held-out paired eval, which compares FINISHED checkpoints
|
| 117 |
and never touches initialisation. That is the ruler the verdict rests on anyway; the
|
| 118 |
+
shared init is worth exactly one step.
|
| 119 |
|
| 120 |
EPOCH COUNT IS THE OPEN DECISION. Written at 1 epoch so it reproduces the ladder's unit
|
| 121 |
and is directly paired against every arm already on it. A longer run CONTAINS this one:
|
| 122 |
checkpoint at 2616 and the 1-epoch verdict comes out of it for free, which is why there is
|
| 123 |
no reason to run a separate 1-epoch arm first. If the epoch count changes, change
|
| 124 |
+
`max_train_steps` AND `lr_schedule_steps` together -- an earlier
|
| 125 |
+
arm was confounded by exactly that field being left to a fallback.
|
| 126 |
"""
|
| 127 |
import os
|
| 128 |
|
attnhead_b7_qwen3_4b/step_2616/train_config.py
CHANGED
|
@@ -45,7 +45,7 @@ because o_proj=0 independently starves the attention branch (14 tensors) for any
|
|
| 45 |
A null result could not then have been read as "the head does not help" rather than "the
|
| 46 |
gate never opened". Nothing is zero-initialised now; instead out_norm's weight is set so
|
| 47 |
the step-0 bias matches the magnitude of the vanilla markov bias it replaces (~0.064
|
| 48 |
-
measured against 0.069), which is a scale choice and not a gate. Read a 1-epoch result with that in mind:
|
| 49 |
that a change which only pays off late reads as null at one epoch.
|
| 50 |
|
| 51 |
SERVING COST (measured, refine_exp/bench_regrounding_*.py, H100, CUDA graph):
|
|
@@ -132,7 +132,7 @@ train = dict(
|
|
| 132 |
# WRITTEN OUT, not left to a default. An arm on this ladder was once compared
|
| 133 |
# against a baseline whose cosine schedule annealed to zero while its own stopped at
|
| 134 |
# 99.6% of peak, because one config omitted this field and fell back to a different
|
| 135 |
-
# value.
|
| 136 |
lr_schedule_steps=2616,
|
| 137 |
max_grad_norm=1.0,
|
| 138 |
sharding_strategy="no_shard",
|
|
|
|
| 45 |
A null result could not then have been read as "the head does not help" rather than "the
|
| 46 |
gate never opened". Nothing is zero-initialised now; instead out_norm's weight is set so
|
| 47 |
the step-0 bias matches the magnitude of the vanilla markov bias it replaces (~0.064
|
| 48 |
+
measured against 0.069), which is a scale choice and not a gate. Read a 1-epoch result with that in mind: an external 10-epoch reproduction showed
|
| 49 |
that a change which only pays off late reads as null at one epoch.
|
| 50 |
|
| 51 |
SERVING COST (measured, refine_exp/bench_regrounding_*.py, H100, CUDA graph):
|
|
|
|
| 132 |
# WRITTEN OUT, not left to a default. An arm on this ladder was once compared
|
| 133 |
# against a baseline whose cosine schedule annealed to zero while its own stopped at
|
| 134 |
# 99.6% of peak, because one config omitted this field and fell back to a different
|
| 135 |
+
# value.
|
| 136 |
lr_schedule_steps=2616,
|
| 137 |
max_grad_norm=1.0,
|
| 138 |
sharding_strategy="no_shard",
|