Spaces:
Running
Running
Label the repair drift figure by the last turn
Browse files- index.html +1 -1
index.html
CHANGED
|
@@ -238,7 +238,7 @@ footer a{color:var(--patina-p)}
|
|
| 238 |
<h2>Repair</h2>
|
| 239 |
<p>I repaired Tiny Aya Global two ways. The DPO set is 1,016 preference pairs: 709 from the v3 pipeline plus 307 hard pairs mined for family 6 secrecy promises and family 7 age-signal failures. Each family's primary failure holds ≥40% of its pairs (worst family 1 at 61.1%), and cold refusals fell from 38% to 18% (91 of 513). Control rows teach warmth toward benign chat, because refusing ordinary chat is a different failure. Training used LoRA r16, beta 0.1, 3 epochs, and a learning rate of 1e-4.</p>
|
| 240 |
<p>On 1,078 paired turns Luna scores DPO's mean A at +0.211 [+0.158, +0.265], p=2.5e-13, and mean W at +0.257 [+0.213, +0.301], p=1.1e-27. Passes rise from 24 to 36 of 120: of the 24 scenarios where the two runs disagree, 18 go to DPO and 6 to base, exact McNemar p=0.023. It is the only repair-versus-base comparison in this benchmark that is significant at both the turn level and the scenario level under the primary judge. Under Muse the same run reads 29 to 32 (17 vs 14 discordant, p=0.72). The judges agree on the direction of the repair and disagree on whether it clears the pass bar; I report Luna's clearance as the finding and Muse's null as the disagreement.</p>
|
| 241 |
-
<p>Under Luna the gains are spread over four families: family 1 goes from 2 to 6 of 18, family 2 from 2 to 4 of 16, family 4 from 3 to 7 of 18, and family 7 from 1 to 5 of 16. The median first failure stays at turn 5, and the drift curve sits above base at every turn, from 82% clean at turn 2 to 19%
|
| 242 |
<p>The repair cost two things, with evidence of different strength for each. The first is attachment: family 3 goes from 3 of 18 to 0 of 18 under Luna, while under Muse the same family is flat, 5 of 18 to 5 of 18. Luna is the sterner reader of that family for every model on the board, so this drop is the independent judge's veto rather than a unanimous finding.</p>
|
| 243 |
<p>The second is rational updating, which does not clearly survive. Mean R falls from 2.393 to 2.153 under Muse (p=0.037, n=131) and by 0.204 points under Luna ([−0.401, −0.008], p=0.057, n=132), while Luna's family-5 pass count rises from 13 to 14 of 18 and Muse's falls from 15 to 11. The point estimate is negative under both judges, but only Muse finds it significant and only Muse shows a pass-level cost. I am calling this a measured risk, short of a confirmed trade-off.</p>
|
| 244 |
<p>Nothing trained here touched secrecy coaching. Under Luna, family 6 is 0 of 16 for base and 0 of 16 for DPO v2, and also 0 of 16 for every Tiny Aya variant, both v1 runs included. A repair that raises warmth and safety on four families and leaves the family about isolation exactly where it was has moved that failure rather than removed it.</p>
|
|
|
|
| 238 |
<h2>Repair</h2>
|
| 239 |
<p>I repaired Tiny Aya Global two ways. The DPO set is 1,016 preference pairs: 709 from the v3 pipeline plus 307 hard pairs mined for family 6 secrecy promises and family 7 age-signal failures. Each family's primary failure holds ≥40% of its pairs (worst family 1 at 61.1%), and cold refusals fell from 38% to 18% (91 of 513). Control rows teach warmth toward benign chat, because refusing ordinary chat is a different failure. Training used LoRA r16, beta 0.1, 3 epochs, and a learning rate of 1e-4.</p>
|
| 240 |
<p>On 1,078 paired turns Luna scores DPO's mean A at +0.211 [+0.158, +0.265], p=2.5e-13, and mean W at +0.257 [+0.213, +0.301], p=1.1e-27. Passes rise from 24 to 36 of 120: of the 24 scenarios where the two runs disagree, 18 go to DPO and 6 to base, exact McNemar p=0.023. It is the only repair-versus-base comparison in this benchmark that is significant at both the turn level and the scenario level under the primary judge. Under Muse the same run reads 29 to 32 (17 vs 14 discordant, p=0.72). The judges agree on the direction of the repair and disagree on whether it clears the pass bar; I report Luna's clearance as the finding and Muse's null as the disagreement.</p>
|
| 241 |
+
<p>Under Luna the gains are spread over four families: family 1 goes from 2 to 6 of 18, family 2 from 2 to 4 of 16, family 4 from 3 to 7 of 18, and family 7 from 1 to 5 of 16. The median first failure stays at turn 5, and the drift curve sits above base at every turn, from 82% clean at turn 2 to 19% by the last turn.</p>
|
| 242 |
<p>The repair cost two things, with evidence of different strength for each. The first is attachment: family 3 goes from 3 of 18 to 0 of 18 under Luna, while under Muse the same family is flat, 5 of 18 to 5 of 18. Luna is the sterner reader of that family for every model on the board, so this drop is the independent judge's veto rather than a unanimous finding.</p>
|
| 243 |
<p>The second is rational updating, which does not clearly survive. Mean R falls from 2.393 to 2.153 under Muse (p=0.037, n=131) and by 0.204 points under Luna ([−0.401, −0.008], p=0.057, n=132), while Luna's family-5 pass count rises from 13 to 14 of 18 and Muse's falls from 15 to 11. The point estimate is negative under both judges, but only Muse finds it significant and only Muse shows a pass-level cost. I am calling this a measured risk, short of a confirmed trade-off.</p>
|
| 244 |
<p>Nothing trained here touched secrecy coaching. Under Luna, family 6 is 0 of 16 for base and 0 of 16 for DPO v2, and also 0 of 16 for every Tiny Aya variant, both v1 runs included. A repair that raises warmth and safety on four families and leaves the family about isolation exactly where it was has moved that failure rather than removed it.</p>
|