dmontgomery40 commited on
Commit
6fa3be7
·
verified ·
1 Parent(s): 1c0c73f

Sync from DMontgomery40/plastic via hub-sync

Browse files
Files changed (1) hide show
  1. README.md +19 -0
README.md CHANGED
@@ -323,6 +323,25 @@ Caveats: small CPU-only models (exp 1 ~109–151k parameters, 4000 steps); 3 see
323
  locally trained LM (held-out loss 4.52) instead of the gated HF checkpoint, trained with AdamW instead of Muon
324
  (Muon gave NaN); S3 was used instead of S5 (the S5 pilot stayed at chance); variant E was run on parity only.
325
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
326
  ## Honest limits
327
 
328
  - PlasticCore is about 6M parameters on wikitext. Its generations are fluent-ish, not
 
323
  locally trained LM (held-out loss 4.52) instead of the gated HF checkpoint, trained with AdamW instead of Muon
324
  (Muon gave NaN); S3 was used instead of S5 (the S5 pilot stayed at chance); variant E was run on parity only.
325
 
326
+ ## Experiment 4b: first-answer composition of group actions (aligned layout, 60k GPU steps)
327
+
328
+ Full write-up, tables, plots and commands: [`experiments/exp4_compose/REPORT_4b.md`](experiments/exp4_compose/REPORT_4b.md)
329
+ (code in `experiments/exp4_compose/`, results in `experiments/results/exp4b/`). A 1.2M-parameter PlasticLM,
330
+ meta-trained with an adaptive depth curriculum for 60k steps on one A10G (three seeds with the query laid out like a
331
+ lesson line, plus one old-layout control), is asked the first and only query after lessons that show single generator steps
332
+ of a random small group under random names. Item-level evaluation, 2,000 items per depth per run; Wilson and paired bootstrap 95% CIs.
333
+ - Depth 1 is exact for every run (recall). Depth 2: 0.999 / 0.999 / 0.903 on the three aligned seeds, 0.695 on the old layout.
334
+ Depth 3: 0.953 / 0.919 / 0.296. Depth 4: 0.212 / 0.530 / not trained. Depths 5-6 (never trained): chance.
335
+ - The control that matters is the content null (same format, wrong names). On items where the wrong lessons imply a different answer,
336
+ fast weights beat it by +0.998 [+0.996, +1.000] (depth 2, seed 0) and +0.944 [+0.932, +0.956] (depth 3, seed 0), +0.898 [+0.881, +0.914] (depth 3, seed 1);
337
+ and the same model answers what the wrong lessons imply (0.999 depth 2, 0.91-0.95 depth 3). So depth 2-3 on two seeds is content-dependent composition, above
338
+ a template copier (0.15), one-hop lookup, and a successor-set-prior guesser (0.50 at depth 2, 0.33 at depth 3), not format adaptation.
339
+ - Depth 4 is real in one seed only (0.530 [0.509, 0.552], +0.419 over the content null on discriminating items); seed 0's 0.212 is not distinguishable from the null
340
+ (+0.040 [+0.011, +0.069]) and below a set-prior guesser. Seed 2 is slower (depth 3 at 0.296, below the set prior).
341
+ - Caveats: 3 seeds and CIs over items only; depth 4 was unlocked at 55-56k of 60k steps and seed 2 was still climbing (compute-limited, not evidence about
342
+ the architecture); small model by design; in-episode fast-weight path, not a lasting weight update. The old-layout control learned depth 1 as fast
343
+ as the aligned runs, so the exp 4 depth-1 plateau was largely a step budget; the layout helps at depths 2-3. No weights in the repo (private HF dataset).
344
+
345
  ## Honest limits
346
 
347
  - PlasticCore is about 6M parameters on wikitext. Its generations are fluent-ish, not