Add WikiText coefficient validation artifacts

docs/plan.md

@@ -800,7 +800,7 @@ under `data/` and `.cache/dropout_decay*` with explicit provenance.
 | TinyStories static/coefficient regime | active | main coefficient evidence |
 | TinyStories streaming regime | 5-seed validation complete | current main streaming evidence; interaction decay beats best static in 5/5 paired final-loss comparisons |
 | OpenWebText10K streaming regime | 5-seed clean validation complete | OpenWebText10K interaction decay beats best static in 5/5 paired final-loss comparisons |
-| WikiText-103 streaming regime | 5-seed validation complete | formula-derived L12 decay beats best static in 5/5 paired final-loss comparisons |
+| WikiText-103 streaming regime | active; L12 5-seed and L20 8M 3-seed validations complete | earlier L12 formula path beats best static in 5/5 paired final-loss comparisons; newly fitted interaction coefficients win final loss on L20 8M extrapolation but are not trajectory-optimal |
 
 ## Current Formula Status
 
@@ -849,7 +849,7 @@ structure transfers, while coefficients may be regime-specific.
 | TinyStories held-out model | supports pressure dependence on model size |
 | TinyStories streaming, 5 seeds | interaction has best mean final loss; interaction beats best static in 5/5 paired final-loss comparisons |
 | OpenWebText10K streaming, 5 seeds | interaction decay has best mean final loss; top decay schedules beat best static in 5/5 paired comparisons |
-| WikiText-103 streaming, 5 seeds | formula-derived L12 decay has best mean final loss; beats best static in 5/5 paired comparisons |
+| WikiText-103 streaming | earlier L12 formula path has best 4M five-seed final loss; fitted interaction coefficients strongly improve static-cell MAE and win final loss on L20 8M three-seed extrapolation, but they decay too quickly for L12 4M |
 | cross-regime raw coefficient transfer | weaker than within-regime fit; supports regime-specific coefficients rather than universal numeric coefficients |
 
 Latest TinyStories 5-seed streaming final-loss table:
@@ -887,9 +887,12 @@ The stronger claim:
 Formula-derived dropout decay beats the best static dropout.
 ```
 
-is supported at `n=5` in TinyStories, OpenWebText10K, and WikiText-103. The
-strongest schedule in each of the three regimes beats the per-seed best static
-baseline in all five seeds.
+is supported at `n=5` for TinyStories and OpenWebText10K with fitted
+interaction schedules, and at `n=5` for WikiText-103 with the earlier L12
+formula-derived schedule. The newly fitted WikiText-103 interaction schedule has
+a stronger static-cell fit and wins the larger L20 8M extrapolation at `n=3`,
+but it is not yet a five-seed WikiText interaction win. Keep this distinction
+clear in the paper.
 
 Latest OpenWebText10K 5-seed streaming final-loss table:
 
@@ -977,6 +980,22 @@ The best static baseline in the clean WikiText-103 run is static dropout
 validation loss by about `0.0297` and wins every paired seed comparison. This
 promotes WikiText-103 to a third multi-seed streaming validation regime.
 
+Latest WikiText-103 coefficient and extrapolation update:
+
+| Item | Result |
+|---|---|
+| Static calibration cells | 15 cells from L8/L12/L16 crossed with 250k, 500k, 1M, 2M, 4M prefixes |
+| Base ABC fit | MAE `0.0449`, RMSE `0.0553` |
+| Interaction fit | MAE `0.0162`, RMSE `0.0191` |
+| Fitted interaction L12 path | `0.17 -> 0.10 -> 0.02 -> 0.02 -> 0.02`; final `4.1122 +/- 0.0118`, roughly tied with static `0.10` but worse than the earlier L12 formula path |
+| Fitted interaction L20 8M path | `0.28 -> 0.21 -> 0.12 -> 0.04 -> 0.02 -> 0.02`; final `3.9572 +/- 0.0085`, better than best static `0.14` at `3.9756 +/- 0.0087` |
+
+Interpretation: the WikiText-103 interaction formula is useful but currently
+final-loss oriented. On L20 8M it wins final validation loss, while static
+dropout around `0.26` to `0.30` has better early/midstream trajectory. On L12
+4M the fitted path decays to the floor too early. Future WikiText claims should
+distinguish final-loss optimization from trajectory optimization.
+
 ## Completed Static Backtest Gate
 
 The first offline coefficient backtest is complete. It is retained as supporting
@@ -999,28 +1018,37 @@ streaming multi-seed reports for each regime.
 
 ## Immediate Next Action
 
-Reconcile the TinyStories, OpenWebText10K, and WikiText-103 five-seed streaming
-reports into the paper outline. The strongest current claim is now supported in
-three regimes: formula-derived or regime-fitted decay schedules beat the best
-static dropout baseline in paired five-seed final-loss comparisons.
+Reconcile the TinyStories, OpenWebText10K, and WikiText-103 streaming reports
+into the paper outline. The cleanest current claim is:
+
+```text
+Regime-specific coefficient fitting can produce dropout schedules that beat
+broad static dropout baselines on final validation loss, and the interaction
+pressure law extrapolates to larger model/token settings in OpenWebText10K,
+TinyStories, and WikiText-103.
+```
+
+The remaining empirical weakness is trajectory optimality and seed count on the
+WikiText-103 L20 interaction extrapolation. A paper-grade next step is either:
 
-The next empirical weakness is no longer a missing third text regime. The next
-useful strengthening step is to test robustness across a controlled architecture
-or token-budget change inside one established corpus regime, while preserving
-the same MPS-only, five-seed validation standard.
+1. Upgrade the WikiText-103 L20 8M extrapolation from 3 seeds to 5 seeds.
+2. Fit a trajectory-aware schedule transform and compare it against the current
+   final-loss-oriented interaction schedule.
 
 ## Next Training After Current Gate
 
-No MPS training should launch until the three completed five-seed streaming
-reports are read together. Since a third held-out text regime is no longer the
-limiting issue, use the next run only for a narrowed robustness test:
+No broad MPS training should launch until the three streaming reports are read
+together. Since a third held-out text regime is no longer the limiting issue,
+use the next run only for a narrowed robustness or confirmation test:
 
 ```text
 completed: TinyStories 5-seed streaming report
 completed: OpenWebText10K 5-seed clean streaming report
 completed: WikiText-103 5-seed clean streaming report
-next: reconcile three-regime evidence into the paper, then choose one narrowed
-      robustness test
+completed: WikiText-103 fitted interaction static calibration
+completed: WikiText-103 L20 8M 3-seed extrapolation
+next: reconcile evidence into the paper, then either add two more L20 WikiText
+      seeds or test a trajectory-aware schedule transform
 avoid: broad new sweep before three-regime report reconciliation
 ```
 
@@ -1034,12 +1062,13 @@ decay minus best-static delta per seed
 rank consistency across seeds
 ```
 
-Because TinyStories, OpenWebText10K, and WikiText-103 decays win across paired
-seeds, promote the cross-regime streaming claim to "supported in three regimes."
-Do not yet claim universal numeric coefficients. The current defensible
-paper-level claim is that the pressure-law structure and regime-specific fitting
-procedure can produce dropout schedules that beat the best static dropout
-baseline across multiple text regimes.
+Because TinyStories, OpenWebText10K, and WikiText-103 now all have positive
+streaming evidence, promote the cross-regime streaming claim to "supported, with
+objective-specific limitations." Do not claim universal numeric coefficients or
+that one schedule is best at every prefix. The current defensible paper-level
+claim is that the pressure-law structure and regime-specific fitting procedure
+can produce dropout schedules that beat broad static dropout baselines on final
+validation loss across multiple text regimes.
 
 Latest streaming report:
 
@@ -1050,4 +1079,5 @@ docs/wikitext103_streaming_report.md
 runs/streaming_tinystories_multiseed_validation_l12/combined_5seed_summary/
 runs/openwebtext10k_streaming_report/l16_updated_formula_clean_5seed/
 runs/wikitext103_streaming_report/l12_validation_5seed/
+runs/wikitext103_l20_8m_extrapolation_3seed/locked_stream/20260603-231952/
 ```

docs/wikitext103_streaming_report.md

@@ -1,16 +1,61 @@
 # WikiText-103 Streaming Validation
 
 Date: 2026-05-31
+Updated: 2026-06-03
 
-This report combines 5 random seeds (1, 2, 3, 4, 5) from saved streaming runs.
-No additional training is performed by this script; it reads saved
-`metrics.jsonl` files.
+This report combines saved WikiText-103 streaming runs, coefficient-calibration
+runs, and extrapolation runs. Reported multi-seed runs use saved
+`metrics.jsonl` files; no additional training is performed by the report
+generation step.
 
-Regime: WikiText-103 cached-corpus streaming setup with L12_H8_D320, 17,367,040 parameters, five prefixes from 250k to 4M tokens, and 1,000 optimizer steps per stage. This is a clean five-seed run including three dropout decay schedules and broad static dropout baselines from 0.00 through 0.30.
+Regime: WikiText-103 cached-corpus streaming setup. The original validation
+uses L12_H8_D320, 17,367,040 parameters, five prefixes from 250k to 4M tokens,
+and 1,000 optimizer steps per stage. The 2026-06-03 additions include a
+15-cell static calibration screen, fitted base and interaction coefficient
+models, a five-seed L12 interaction-schedule validation, and a three-seed
+L20_H8_D448 extrapolation to 8M prefix tokens.
 
 ## Sources
 
 - `runs/wikitext103_l12_streaming_validation_5seed/locked_stream/20260531-093525/metrics.jsonl`
+- `runs/wikitext103_interaction_static_screen/screen_static/20260603-153314/metrics.jsonl`
+- `runs/coefficient_calibration/wikitext103_base/coefficients.json`
+- `runs/coefficient_calibration/wikitext103_interaction/coefficients.json`
+- `runs/wikitext103_interaction_l12_validation_5seed/locked_stream/20260603-223707/metrics.jsonl`
+- `runs/wikitext103_l20_8m_extrapolation_3seed/locked_stream/20260603-231952/metrics.jsonl`
+
+## 2026-06-03 Coefficient Calibration
+
+Static dropout calibration was run on three model sizes and five WikiText-103
+prefix sizes, giving 15 calibration cells:
+
+| Model | Parameters | Prefix tokens | Dropout grid | Seeds | Stage steps |
+|---|---:|---:|---|---:|---:|
+| `L8_H8_D256` | 8,389,632 | 250k, 500k, 1M, 2M, 4M | 0.00 to 0.50 | 2 | 1,000 |
+| `L12_H8_D320` | 17,367,040 | 250k, 500k, 1M, 2M, 4M | 0.00 to 0.50 | 2 | 1,000 |
+| `L16_H8_D384` | 32,713,728 | 250k, 500k, 1M, 2M, 4M | 0.00 to 0.50 | 2 | 1,000 |
+
+The fitted target was the local quadratic optimum where the dropout-loss curve
+was bracketed, otherwise the grid optimum with lower confidence weight.
+
+| Feature set | Cells | Fitted formula | RMSE | MAE |
+|---|---:|---|---:|---:|
+| Base ABC | 15 | `p = clamp(0.00, 0.50, 0.1418*x + 0.0458*y - 0.1618)` | 0.0553 | 0.0449 |
+| Interaction | 15 | `p = clamp(0.00, 0.50, -0.0027*x - 0.2836*y + 0.2686*x*y - 0.0190)` | 0.0191 | 0.0162 |
+
+Cross-validation diagnostics:
+
+| Feature set | Leave-model MAE | Leave-prefix MAE |
+|---|---:|---:|
+| Base ABC | 0.0534 | 0.0837 |
+| Interaction | 0.0304 | 0.0444 |
+
+Here `x = log10(P / U)` and `y = log10(C / U)`, where `P` is model
+parameters, `U` is current unique prefix tokens, and `C` is cumulative sampled
+tokens. The interaction formula is the stronger static-cell fit for WikiText-103
+because it reduces calibration MAE from 0.0449 to 0.0162. This does not by
+itself prove streaming superiority; it only establishes the coefficient-derived
+dropout path to test.
 
 ## Condition Ranking By Final Loss
 
@@ -196,3 +241,88 @@ baseline for that seed.
 - This is a saved-run streaming validation artifact. Treat it as strong
   evidence only when the tested conditions, seeds, static baselines, and
   stream protocol match the claim being made.
+
+## Fitted Interaction L12 Validation
+
+After fitting WikiText-103 coefficients from the 15 static calibration cells, a
+new coefficient-derived L12 schedule was generated and run for five seeds.
+
+| Condition | Kind | Model | Parameters | Prefix tokens | N | Mean trajectory val | Mean final val | Std final val | Mean final gap | Dropout path |
+|---|---|---|---:|---:|---:|---:|---:|---:|---:|---|
+| `wikitext103_interaction_l12` | `anchor_decay` | `L12_H8_D320` | 17,367,040 | 250k -> 4M | 5 | 4.5867 | 4.1122 | 0.0118 | 0.3470 | `0.17 -> 0.10 -> 0.02 -> 0.02 -> 0.02` |
+| `wikitext103_formula_l12` | `anchor_decay` | `L12_H8_D320` | 17,367,040 | 250k -> 4M | 5 | 4.5711 | 4.0808 | 0.0195 | 0.2817 | `0.30 -> 0.26 -> 0.18 -> 0.09 -> 0.02` |
+| `static_dropout_0.1` | `static` | `L12_H8_D320` | 17,367,040 | 250k -> 4M | 5 | 4.5836 | 4.1105 | 0.0188 | 0.2687 | `0.10 -> 0.10 -> 0.10 -> 0.10 -> 0.10` |
+
+Interpretation: the fitted interaction schedule is not the best L12 streaming
+schedule. It is statistically close to the best static baseline but worse than
+the earlier `wikitext103_formula_l12` schedule. The likely failure mode is that
+the fitted interaction schedule decays too quickly: it reaches the floor
+dropout of 0.02 by the 1M-token stage, while the earlier winning L12 schedule
+keeps substantial regularization until 2M tokens.
+
+This is useful negative evidence. It says that a low static-cell fit error is
+not sufficient unless the streaming objective is also specified. For the paper,
+WikiText-103 should report both final-loss optimization and trajectory
+optimization instead of claiming one schedule is uniformly better at every
+stage.
+
+## L20 8M Extrapolation
+
+The same WikiText-103 interaction coefficients were extrapolated to a larger
+model and longer stream:
+
+- Model: `L20_H8_D448`
+- Parameters: 51,838,976
+- Prefix tokens: 250k, 500k, 1M, 2M, 4M, 8M
+- Stage steps: 1,000
+- Seeds: 1, 2, 3
+- Static baselines: 0.00, 0.02, 0.04, 0.06, 0.08, 0.10, 0.14, 0.18, 0.20, 0.26, 0.30, 0.40, 0.50
+
+### Final-Loss Ranking
+
+| Condition | Kind | N | Mean trajectory val | Mean final val | Std final val | Mean final gap | Dropout path |
+|---|---|---:|---:|---:|---:|---:|---|
+| `wikitext103_l20_interaction_8m` | `anchor_decay` | 3 | 4.4907 | 3.9572 | 0.0085 | 0.3623 | `0.28 -> 0.21 -> 0.12 -> 0.04 -> 0.02 -> 0.02` |
+| `static_dropout_0.14` | `static` | 3 | 4.5424 | 3.9756 | 0.0087 | 0.3122 | `0.14 -> 0.14 -> 0.14 -> 0.14 -> 0.14 -> 0.14` |
+| `static_dropout_0.18` | `static` | 3 | 4.5094 | 3.9758 | 0.0084 | 0.2984 | `0.18 -> 0.18 -> 0.18 -> 0.18 -> 0.18 -> 0.18` |
+| `static_dropout_0.10` | `static` | 3 | 4.5955 | 3.9785 | 0.0019 | 0.3347 | `0.10 -> 0.10 -> 0.10 -> 0.10 -> 0.10 -> 0.10` |
+| `static_dropout_0.20` | `static` | 3 | 4.5001 | 3.9807 | 0.0074 | 0.2995 | `0.20 -> 0.20 -> 0.20 -> 0.20 -> 0.20 -> 0.20` |
+| `static_dropout_0.26` | `static` | 3 | 4.4829 | 3.9842 | 0.0085 | 0.2766 | `0.26 -> 0.26 -> 0.26 -> 0.26 -> 0.26 -> 0.26` |
+| `static_dropout_0.30` | `static` | 3 | 4.4869 | 4.0015 | 0.0058 | 0.2584 | `0.30 -> 0.30 -> 0.30 -> 0.30 -> 0.30 -> 0.30` |
+
+The coefficient-derived interaction schedule beats the best static baseline by
+0.0184 final validation loss at 8M tokens (`3.9756 - 3.9572`). This is the
+strongest WikiText-103 result so far for the hypothesis that a high-capacity
+model can benefit from decaying dropout as the stream grows.
+
+### Stage-Wise Behavior
+
+| Prefix tokens | Interaction dropout | Interaction mean val | Best static at stage | Best static mean val | Note |
+|---:|---:|---:|---|---:|---|
+| 250k | 0.28 | 5.2075 | `static_dropout_0.26` | 5.1995 | Static slightly better |
+| 500k | 0.21 | 4.8170 | `static_dropout_0.30` | 4.8010 | Static better |
+| 1M | 0.12 | 4.5390 | `static_dropout_0.26` | 4.5072 | Static better |
+| 2M | 0.04 | 4.3108 | `static_dropout_0.26` | 4.2754 | Static better |
+| 4M | 0.02 | 4.1125 | `static_dropout_0.18` | 4.1088 | Rough tie |
+| 8M | 0.02 | 3.9572 | `wikitext103_l20_interaction_8m` | 3.9572 | Interaction best |
+
+The L20 result supports the final-capacity claim, but not a trajectory-optimal
+claim. Static dropout around 0.26 to 0.30 is better during much of early and
+middle streaming, while the decayed schedule finishes best at 8M tokens. The
+current coefficient formula should therefore be described as a final-loss
+schedule, not automatically a best-at-every-prefix schedule.
+
+## Updated WikiText-103 Takeaway
+
+- WikiText-103 now has static calibration cells, fitted base and interaction
+  coefficients, L12 five-seed validation, and L20 8M three-seed extrapolation.
+- The interaction coefficient form is a much better fit to static optimum cells
+  than base ABC for this regime: MAE 0.0162 versus 0.0449.
+- The fitted interaction schedule underperforms the earlier L12 formula path on
+  4M streaming because it drops to the minimum dropout too early.
+- On the larger L20 8M extrapolation, the fitted interaction schedule wins final
+  loss against a broad static dropout grid by about 0.0184 validation loss.
+- For publication, WikiText-103 should be framed as evidence that regime-specific
+  fitted dropout schedules can extrapolate to larger model/token settings, with
+  the limitation that the current objective is final-loss oriented and does not
+  guarantee the best loss at every intermediate stream prefix.

runs/coefficient_calibration/wikitext103_base/calibration_cells.csv

@@ -0,0 +1,16 @@
+source,run_dir,run_mode,model_name,n_layer,n_head,n_embd,parameters,unique_tokens,sampled_tokens,x_model_pressure,x_sample_pressure,best_grid_dropout,best_quad_dropout,target_dropout,predicted_dropout,residual,best_val_loss,best_val_std,boundary_optimum,bracketed_optimum,weight
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L16_H8_D384,16,8,384,31457280,250000,2048000,2.099781159327249,0.9133899436317555,0.26,0.24585471298511866,0.24585471298511866,0.17781302230378757,-0.06804169068133109,5.1221378818154335,0.016875071308155583,False,True,0.7476627541764188
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L12_H8_D320,12,8,320,17367040,250000,2048000,1.8417857958884694,0.9133899436317555,0.18,0.16452735734218263,0.16452735734218263,0.141225444607143,-0.02330191273503962,5.154339708387852,0.01651832984564725,False,True,0.7516725106601853
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L8_H8_D256,8,8,256,8388608,250000,2048000,1.5257498915995298,0.9133899436317555,0.1,0.09651891154481321,0.09651891154481321,0.09640685543181973,-0.00011205611299347495,5.226473439484835,0.02592699737804487,False,True,0.6585272923548805
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L16_H8_D384,16,8,384,31457280,1000000,2048000,1.4977211679992863,0.3113299523037932,0.02,0.022517605389921383,0.022517605389921383,0.06487149541364604,0.04235389002372465,4.7049963511526585,0.026859896796409256,False,True,0.6505343109221544
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L16_H8_D384,16,8,384,31457280,500000,2048000,1.7987511636632676,0.6123599479677744,0.08,0.08223816002591516,0.08223816002591516,0.12134225885871677,0.03910409883280161,4.868544135242701,0.03936356019728768,False,True,0.5595121757639818
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L12_H8_D320,12,8,320,17367040,500000,2048000,1.540755800224488,0.6123599479677744,0.02,0.029893772789233324,0.029893772789233324,0.08475468116207227,0.05486090837283895,4.939401715993881,0.02775412121290883,False,True,0.6430527310968944
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L16_H8_D384,16,8,384,31457280,4000000,2048000,0.8956611766713239,-0.2907300390241692,0.0,0.0,0.0,-0.04807003147649541,-0.04807003147649541,4.614261016249657,0.0343418000757071,True,False,0.08892387870863357
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L16_H8_D384,16,8,384,31457280,2000000,2048000,1.196691172335305,0.010299956639811961,0.0,0.0,0.0,0.008400731968575298,0.008400731968575298,4.638469863682985,0.020520631458734194,True,False,0.10635185540544535
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L12_H8_D320,12,8,320,17367040,4000000,2048000,0.6376658132325446,-0.2907300390241692,0.0,0.0,0.0,-0.08465760917313989,-0.08465760917313989,4.751548711210489,0.02019173852637943,True,False,0.10685018148084982
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L12_H8_D320,12,8,320,17367040,2000000,2048000,0.9386958088965257,0.010299956639811961,0.0,0.0,0.0,-0.028186845728069176,-0.028186845728069176,4.769962530583143,0.024905420990413565,True,False,0.10012626457249142
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L12_H8_D320,12,8,320,17367040,1000000,2048000,1.2397258045605068,0.3113299523037932,0.0,0.0,0.0,0.02828391771700156,0.02828391771700156,4.819691203534603,0.024108977262459263,True,False,0.10120231417360566
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L8_H8_D256,8,8,256,8388608,4000000,2048000,0.3216299089436051,-0.2907300390241692,0.0,0.0,0.0,-0.12947619834846313,-0.12947619834846313,4.931619014590979,0.024494004528674472,True,False,0.10067924321497679
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L8_H8_D256,8,8,256,8388608,2000000,2048000,0.6226599046075862,0.010299956639811961,0.0,0.0,0.0,-0.07300543490339242,-0.07300543490339242,4.946536503732204,0.02745871395169597,True,False,0.09682577488540638
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L8_H8_D256,8,8,256,8388608,1000000,2048000,0.9236899002715675,0.3113299523037932,0.0,0.0,0.0,-0.016534671458321687,-0.016534671458321687,4.971932724118233,0.018370399346473592,True,False,0.1096965948961776
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L8_H8_D256,8,8,256,8388608,500000,2048000,1.2247198959355488,0.6123599479677744,0.02,0.011386212032769067,0.011386212032769067,0.03993609198674905,0.028549879953979986,5.056944504380226,0.029059525015124623,False,True,0.6324348646217475

runs/coefficient_calibration/wikitext103_base/coefficients.json

@@ -0,0 +1,39 @@
+{
+  "target": "quad",
+  "feature_set": "base",
+  "formula": "p = +0.141815 * log10(P / U) +0.045777 * log10(C / U) -0.161779",
+  "weighting": "heuristic",
+  "coefficients": {
+    "A": 0.14181486523236092,
+    "B": 0.04577695039555778,
+    "C0": -0.16177936596508904
+  },
+  "metrics": {
+    "n": 15.0,
+    "rmse": 0.05527532598163171,
+    "mae": 0.04486265849907786,
+    "bias": -0.017988868249888254,
+    "weighted_rmse": 0.0463481795605566,
+    "weighted_mae": 0.03906163390969135
+  },
+  "cv": {
+    "leave_model": {
+      "n": 15.0,
+      "rmse": 0.0712088234525127,
+      "mae": 0.05337594470919109,
+      "bias": -0.026272219990709385
+    },
+    "leave_prefix": {
+      "n": 15.0,
+      "rmse": 0.10104639786985097,
+      "mae": 0.08369927507752901,
+      "bias": -0.044435696800331374
+    }
+  },
+  "run_dirs": [
+    "runs/wikitext103_interaction_static_screen/screen_static/20260603-153314"
+  ],
+  "A": 0.14181486523236092,
+  "B": 0.04577695039555778,
+  "C0": -0.16177936596508904
+}

runs/coefficient_calibration/wikitext103_base/fit_diagnostics.md

@@ -0,0 +1,59 @@
+# Dropout Coefficient Fit Diagnostics
+
+Target: `quad`
+Feature set: `base`
+
+## Coefficients
+
+| Coefficient | Term | Value |
+|---|---|---:|
+| `A` | `log10(P / U)` | 0.141815 |
+| `B` | `log10(C / U)` | 0.045777 |
+| `C0` | `1` | -0.161779 |
+
+Formula:
+
+```text
+p = clamp(p_min, p_max,
+          +0.141815 * log10(P / U)
+        +0.045777 * log10(C / U)
+        -0.161779)
+```
+
+## Fit Metrics
+
+| Metric | Value |
+|---|---:|
+| `n` | 15.000000 |
+| `rmse` | 0.055275 |
+| `mae` | 0.044863 |
+| `bias` | -0.017989 |
+| `weighted_rmse` | 0.046348 |
+| `weighted_mae` | 0.039062 |
+
+## Cross-Validation
+
+| Holdout | n | RMSE | MAE | Bias |
+|---|---:|---:|---:|---:|
+| `leave_model` | 15 | 0.071209 | 0.053376 | -0.026272 |
+| `leave_prefix` | 15 | 0.101046 | 0.083699 | -0.044436 |
+
+## Calibration Cells
+
+| Source | Model | Params | Unique | Sampled | Grid p | Quad p | Target p | Pred p | Residual | Weight | Bracketed | Boundary |
+|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|---|
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L8_H8_D256` | 8,388,608 | 250,000 | 2,048,000 | 0.100 | 0.097 | 0.097 | 0.096 | -0.000 | 0.659 | True | False |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L8_H8_D256` | 8,388,608 | 500,000 | 2,048,000 | 0.020 | 0.011 | 0.011 | 0.040 | +0.029 | 0.632 | True | False |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L8_H8_D256` | 8,388,608 | 1,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | -0.017 | -0.017 | 0.110 | False | True |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L8_H8_D256` | 8,388,608 | 2,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | -0.073 | -0.073 | 0.097 | False | True |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L8_H8_D256` | 8,388,608 | 4,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | -0.129 | -0.129 | 0.101 | False | True |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L12_H8_D320` | 17,367,040 | 250,000 | 2,048,000 | 0.180 | 0.165 | 0.165 | 0.141 | -0.023 | 0.752 | True | False |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L12_H8_D320` | 17,367,040 | 500,000 | 2,048,000 | 0.020 | 0.030 | 0.030 | 0.085 | +0.055 | 0.643 | True | False |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L12_H8_D320` | 17,367,040 | 1,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | 0.028 | +0.028 | 0.101 | False | True |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L12_H8_D320` | 17,367,040 | 2,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | -0.028 | -0.028 | 0.100 | False | True |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L12_H8_D320` | 17,367,040 | 4,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | -0.085 | -0.085 | 0.107 | False | True |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L16_H8_D384` | 31,457,280 | 250,000 | 2,048,000 | 0.260 | 0.246 | 0.246 | 0.178 | -0.068 | 0.748 | True | False |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L16_H8_D384` | 31,457,280 | 500,000 | 2,048,000 | 0.080 | 0.082 | 0.082 | 0.121 | +0.039 | 0.560 | True | False |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L16_H8_D384` | 31,457,280 | 1,000,000 | 2,048,000 | 0.020 | 0.023 | 0.023 | 0.065 | +0.042 | 0.651 | True | False |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L16_H8_D384` | 31,457,280 | 2,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | 0.008 | +0.008 | 0.106 | False | True |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L16_H8_D384` | 31,457,280 | 4,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | -0.048 | -0.048 | 0.089 | False | True |

runs/coefficient_calibration/wikitext103_base/next_dropout_suggestions.csv

@@ -0,0 +1,16 @@
+source,model_name,unique_tokens,target_dropout,boundary_optimum,bracketed_optimum,suggested_rates
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L16_H8_D384,250000,0.2459,False,True,0.206 0.226 0.246 0.266 0.286
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L12_H8_D320,250000,0.1645,False,True,0.125 0.145 0.165 0.185 0.205
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L8_H8_D256,250000,0.0965,False,True,0.057 0.077 0.097 0.117 0.137
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L16_H8_D384,1000000,0.0225,False,True,0.003 0.023 0.043 0.063
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L16_H8_D384,500000,0.0822,False,True,0.042 0.062 0.082 0.102 0.122
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L12_H8_D320,500000,0.0299,False,True,0.010 0.030 0.050 0.070
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L16_H8_D384,4000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L16_H8_D384,2000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L12_H8_D320,4000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L12_H8_D320,2000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L12_H8_D320,1000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L8_H8_D256,4000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L8_H8_D256,2000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L8_H8_D256,1000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L8_H8_D256,500000,0.0114,False,True,0.011 0.031 0.051

runs/coefficient_calibration/wikitext103_interaction/calibration_cells.csv

@@ -0,0 +1,16 @@
+source,run_dir,run_mode,model_name,n_layer,n_head,n_embd,parameters,unique_tokens,sampled_tokens,x_model_pressure,x_sample_pressure,best_grid_dropout,best_quad_dropout,target_dropout,predicted_dropout,residual,best_val_loss,best_val_std,boundary_optimum,bracketed_optimum,weight
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L16_H8_D384,16,8,384,31457280,250000,2048000,2.099781159327249,0.9133899436317555,0.26,0.24585471298511866,0.24585471298511866,0.23149724051702436,-0.014357472468094296,5.1221378818154335,0.016875071308155583,False,True,0.7476627541764188
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L12_H8_D320,12,8,320,17367040,250000,2048000,1.8417857958884694,0.9133899436317555,0.18,0.16452735734218263,0.16452735734218263,0.16889018086826152,0.004362823526078891,5.154339708387852,0.01651832984564725,False,True,0.7516725106601853
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L8_H8_D256,8,8,256,8388608,250000,2048000,1.5257498915995298,0.9133899436317555,0.1,0.09651891154481321,0.09651891154481321,0.09219857506867352,-0.004320336476139691,5.226473439484835,0.02592699737804487,False,True,0.6585272923548805
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L16_H8_D384,16,8,384,31457280,1000000,2048000,1.4977211679992863,0.3113299523037932,0.02,0.022517605389921383,0.022517605389921383,0.013946663920861954,-0.008570941469059429,4.7049963511526585,0.026859896796409256,False,True,0.6505343109221544
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L16_H8_D384,16,8,384,31457280,500000,2048000,1.7987511636632676,0.6123599479677744,0.08,0.08223816002591516,0.08223816002591516,0.09838110165894087,0.01614294163302571,4.868544135242701,0.03936356019728768,False,True,0.5595121757639818
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L12_H8_D320,12,8,320,17367040,500000,2048000,1.540755800224488,0.6123599479677744,0.02,0.029893772789233324,0.029893772789233324,0.05663517437934751,0.026741401590114185,4.939401715993881,0.02775412121290883,False,True,0.6430527310968944
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L16_H8_D384,16,8,384,31457280,4000000,2048000,0.8956611766713239,-0.2907300390241692,0.0,0.0,0.0,-0.008877108195282257,-0.008877108195282257,4.614261016249657,0.0343418000757071,True,False,0.08892387870863357
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L16_H8_D384,16,8,384,31457280,2000000,2048000,1.196691172335305,0.010299956639811961,0.0,0.0,0.0,-0.02180607269721243,-0.02180607269721243,4.638469863682985,0.020520631458734194,True,False,0.10635185540544535
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L12_H8_D320,12,8,320,17367040,4000000,2048000,0.6376658132325446,-0.2907300390241692,0.0,0.0,0.0,0.011960361632632553,0.011960361632632553,4.751548711210489,0.02019173852637943,True,False,0.10685018148084982
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L12_H8_D320,12,8,320,17367040,2000000,2048000,0.9386958088965257,0.010299956639811961,0.0,0.0,0.0,-0.02182973523846701,-0.02182973523846701,4.769962530583143,0.024905420990413565,True,False,0.10012626457249142
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L12_H8_D320,12,8,320,17367040,1000000,2048000,1.2397258045605068,0.3113299523037932,0.0,0.0,0.0,-0.00693813098956203,-0.00693813098956203,4.819691203534603,0.024108977262459263,True,False,0.10120231417360566
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L8_H8_D256,8,8,256,8388608,4000000,2048000,0.3216299089436051,-0.2907300390241692,0.0,0.0,0.0,0.03748558244428958,0.03748558244428958,4.931619014590979,0.024494004528674472,True,False,0.10067924321497679
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L8_H8_D256,8,8,256,8388608,2000000,2048000,0.6226599046075862,0.010299956639811961,0.0,0.0,0.0,-0.02185872107962124,-0.02185872107962124,4.946536503732204,0.02745871395169597,True,False,0.09682577488540638
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L8_H8_D256,8,8,256,8388608,1000000,2048000,0.9236899002715675,0.3113299523037932,0.0,0.0,0.0,-0.032521323483527514,-0.032521323483527514,4.971932724118233,0.018370399346473592,True,False,0.1096965948961776
+wikitext103_interaction_static_screen/screen_static/20260603-153314,runs/wikitext103_interaction_static_screen/screen_static/20260603-153314,screen_static,L8_H8_D256,8,8,256,8388608,500000,2048000,1.2247198959355488,0.6123599479677744,0.02,0.011386212032769067,0.011386212032769067,0.005497775232570723,-0.005888436800198344,5.056944504380226,0.029059525015124623,False,True,0.6324348646217475

runs/coefficient_calibration/wikitext103_interaction/coefficients.json

@@ -0,0 +1,41 @@
+{
+  "target": "quad",
+  "feature_set": "interaction",
+  "formula": "p = -0.002675 * log10(P / U) -0.283621 * log10(C / U) +0.268606 * log10(P / U) * log10(C / U) -0.018995",
+  "weighting": "heuristic",
+  "coefficients": {
+    "A": -0.002674916387321836,
+    "B": -0.2836210480242947,
+    "D": 0.2686063066585002,
+    "C0": -0.01899454503425616
+  },
+  "metrics": {
+    "n": 15.0,
+    "rmse": 0.019131667586788335,
+    "mae": 0.016244092648220347,
+    "bias": -0.0033516778714015546,
+    "weighted_rmse": 0.01535118105715424,
+    "weighted_mae": 0.01267303662878494
+  },
+  "cv": {
+    "leave_model": {
+      "n": 15.0,
+      "rmse": 0.03474644632399485,
+      "mae": 0.030438322968748356,
+      "bias": -0.01814505695518599
+    },
+    "leave_prefix": {
+      "n": 15.0,
+      "rmse": 0.055157356761289,
+      "mae": 0.04435212839791333,
+      "bias": -0.0039025869319082983
+    }
+  },
+  "run_dirs": [
+    "runs/wikitext103_interaction_static_screen/screen_static/20260603-153314"
+  ],
+  "A": -0.002674916387321836,
+  "B": -0.2836210480242947,
+  "D": 0.2686063066585002,
+  "C0": -0.01899454503425616
+}

runs/coefficient_calibration/wikitext103_interaction/fit_diagnostics.md

@@ -0,0 +1,61 @@
+# Dropout Coefficient Fit Diagnostics
+
+Target: `quad`
+Feature set: `interaction`
+
+## Coefficients
+
+| Coefficient | Term | Value |
+|---|---|---:|
+| `A` | `log10(P / U)` | -0.002675 |
+| `B` | `log10(C / U)` | -0.283621 |
+| `D` | `log10(P / U) * log10(C / U)` | 0.268606 |
+| `C0` | `1` | -0.018995 |
+
+Formula:
+
+```text
+p = clamp(p_min, p_max,
+          -0.002675 * log10(P / U)
+        -0.283621 * log10(C / U)
+        +0.268606 * log10(P / U) * log10(C / U)
+        -0.018995)
+```
+
+## Fit Metrics
+
+| Metric | Value |
+|---|---:|
+| `n` | 15.000000 |
+| `rmse` | 0.019132 |
+| `mae` | 0.016244 |
+| `bias` | -0.003352 |
+| `weighted_rmse` | 0.015351 |
+| `weighted_mae` | 0.012673 |
+
+## Cross-Validation
+
+| Holdout | n | RMSE | MAE | Bias |
+|---|---:|---:|---:|---:|
+| `leave_model` | 15 | 0.034746 | 0.030438 | -0.018145 |
+| `leave_prefix` | 15 | 0.055157 | 0.044352 | -0.003903 |
+
+## Calibration Cells
+
+| Source | Model | Params | Unique | Sampled | Grid p | Quad p | Target p | Pred p | Residual | Weight | Bracketed | Boundary |
+|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|---|
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L8_H8_D256` | 8,388,608 | 250,000 | 2,048,000 | 0.100 | 0.097 | 0.097 | 0.092 | -0.004 | 0.659 | True | False |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L8_H8_D256` | 8,388,608 | 500,000 | 2,048,000 | 0.020 | 0.011 | 0.011 | 0.005 | -0.006 | 0.632 | True | False |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L8_H8_D256` | 8,388,608 | 1,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | -0.033 | -0.033 | 0.110 | False | True |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L8_H8_D256` | 8,388,608 | 2,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | -0.022 | -0.022 | 0.097 | False | True |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L8_H8_D256` | 8,388,608 | 4,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | 0.037 | +0.037 | 0.101 | False | True |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L12_H8_D320` | 17,367,040 | 250,000 | 2,048,000 | 0.180 | 0.165 | 0.165 | 0.169 | +0.004 | 0.752 | True | False |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L12_H8_D320` | 17,367,040 | 500,000 | 2,048,000 | 0.020 | 0.030 | 0.030 | 0.057 | +0.027 | 0.643 | True | False |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L12_H8_D320` | 17,367,040 | 1,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | -0.007 | -0.007 | 0.101 | False | True |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L12_H8_D320` | 17,367,040 | 2,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | -0.022 | -0.022 | 0.100 | False | True |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L12_H8_D320` | 17,367,040 | 4,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | 0.012 | +0.012 | 0.107 | False | True |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L16_H8_D384` | 31,457,280 | 250,000 | 2,048,000 | 0.260 | 0.246 | 0.246 | 0.231 | -0.014 | 0.748 | True | False |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L16_H8_D384` | 31,457,280 | 500,000 | 2,048,000 | 0.080 | 0.082 | 0.082 | 0.098 | +0.016 | 0.560 | True | False |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L16_H8_D384` | 31,457,280 | 1,000,000 | 2,048,000 | 0.020 | 0.023 | 0.023 | 0.014 | -0.009 | 0.651 | True | False |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L16_H8_D384` | 31,457,280 | 2,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | -0.022 | -0.022 | 0.106 | False | True |
+| `wikitext103_interaction_static_screen/screen_static/20260603-153314` | `L16_H8_D384` | 31,457,280 | 4,000,000 | 2,048,000 | 0.000 | 0.000 | 0.000 | -0.009 | -0.009 | 0.089 | False | True |

runs/coefficient_calibration/wikitext103_interaction/next_dropout_suggestions.csv

@@ -0,0 +1,16 @@
+source,model_name,unique_tokens,target_dropout,boundary_optimum,bracketed_optimum,suggested_rates
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L16_H8_D384,250000,0.2459,False,True,0.206 0.226 0.246 0.266 0.286
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L12_H8_D320,250000,0.1645,False,True,0.125 0.145 0.165 0.185 0.205
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L8_H8_D256,250000,0.0965,False,True,0.057 0.077 0.097 0.117 0.137
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L16_H8_D384,1000000,0.0225,False,True,0.003 0.023 0.043 0.063
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L16_H8_D384,500000,0.0822,False,True,0.042 0.062 0.082 0.102 0.122
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L12_H8_D320,500000,0.0299,False,True,0.010 0.030 0.050 0.070
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L16_H8_D384,4000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L16_H8_D384,2000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L12_H8_D320,4000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L12_H8_D320,2000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L12_H8_D320,1000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L8_H8_D256,4000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L8_H8_D256,2000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L8_H8_D256,1000000,0.0000,True,False,
+wikitext103_interaction_static_screen/screen_static/20260603-153314,L8_H8_D256,500000,0.0114,False,True,0.011 0.031 0.051

runs/wikitext103_interaction_l12_validation_5seed/locked_stream/20260603-223707/RESULT_SUMMARY.md

@@ -0,0 +1,44 @@
+# Locked Streaming Dropout Summary
+
+Run directory: `runs/wikitext103_interaction_l12_validation_5seed/locked_stream/20260603-223707`
+
+Model: `L12_H8_D320` causal Transformer, 17,367,040 parameters, 12 layers, 8 heads, 320 embedding dim.
+Training per stage: 1,000 steps. Sampled tokens are cumulative in each stage row. Seeds present: 1, 2, 3, 4, 5.
+
+## Condition Ranking
+
+| Condition | Kind | Final dropout | Mean trajectory val loss | Final val loss | Final gap | Dropout path |
+|---|---|---:|---:|---:|---:|---|
+| `wikitext103_interaction_l12` | anchor_decay | 0.02 | 4.5867 | 4.1122 | 0.3470 | 0.17 -> 0.10 -> 0.02 -> 0.02 -> 0.02 |
+
+## Stage Trajectory
+
+### Stage 0: 250,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `wikitext103_interaction_l12` | 0.17 | 5.1645 | 3.9818 | 1.1827 | 5 |
+
+### Stage 1: 500,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `wikitext103_interaction_l12` | 0.10 | 4.8063 | 3.5840 | 1.2223 | 5 |
+
+### Stage 2: 1,000,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `wikitext103_interaction_l12` | 0.02 | 4.5479 | 3.5211 | 1.0268 | 5 |
+
+### Stage 3: 2,000,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `wikitext103_interaction_l12` | 0.02 | 4.3023 | 3.6733 | 0.6290 | 5 |
+
+### Stage 4: 4,000,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `wikitext103_interaction_l12` | 0.02 | 4.1122 | 3.7653 | 0.3470 | 5 |

runs/wikitext103_interaction_l12_validation_5seed/locked_stream/20260603-223707/config.json

@@ -0,0 +1,127 @@
+{
+  "args": {
+    "mode": "locked_stream",
+    "corpus": null,
+    "corpus_glob": null,
+    "text_column": "text",
+    "use_cached_data": true,
+    "output_dir": "runs/wikitext103_interaction_l12_validation_5seed",
+    "resume_from": null,
+    "cache_dir": ".cache/dropout_decay_wikitext103",
+    "models": [
+      "L12_H8_D320=12x8x320"
+    ],
+    "seeds": [
+      1,
+      2,
+      3,
+      4,
+      5
+    ],
+    "token_limits": [
+      5000000
+    ],
+    "stream_token_caps": [
+      250000,
+      500000,
+      1000000,
+      2000000,
+      4000000
+    ],
+    "val_tokens": 500000,
+    "allow_short_corpus": false,
+    "force_retokenize": false,
+    "vocab_size": 4096,
+    "tokenizer_train_chars": 10000000,
+    "block_size": 128,
+    "batch_size": 16,
+    "steps": 2000,
+    "stage_steps": 1000,
+    "dropout_rates": [],
+    "decays": [],
+    "anchor_decays": [
+      {
+        "name": "wikitext103_interaction_l12",
+        "kind": "anchor_decay",
+        "initial": 0.169,
+        "final": 0.02,
+        "schedule": "log_prefix_anchor",
+        "decay_tokens": null,
+        "anchors": [
+          [
+            250000,
+            0.169
+          ],
+          [
+            500000,
+            0.096
+          ],
+          [
+            1000000,
+            0.02
+          ],
+          [
+            2000000,
+            0.02
+          ],
+          [
+            4000000,
+            0.02
+          ]
+        ]
+      }
+    ],
+    "decay_tokens": null,
+    "eval_batches": 64,
+    "train_eval_batches": 32,
+    "trace_eval_batches": 8,
+    "eval_every": 0,
+    "log_every": 250,
+    "lr": 0.0003,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "plateau_delta": 0.01,
+    "target_min_dropout": 0.1,
+    "min_nonzero_margin": 0.01,
+    "min_high_dropout_margin": 0.03,
+    "screen_early_stop": false,
+    "screen_prune_patience": 3,
+    "screen_prune_min_delta": 0.01
+  },
+  "mode": "locked_stream",
+  "seeds": [
+    1,
+    2,
+    3,
+    4,
+    5
+  ],
+  "models": [
+    {
+      "model_name": "L12_H8_D320",
+      "n_layer": 12,
+      "n_head": 8,
+      "n_embd": 320
+    }
+  ],
+  "device": "mps",
+  "torch": "2.12.0",
+  "python": "3.11.15 (main, Mar  3 2026, 00:52:57) [Clang 21.0.0 (clang-2100.0.123.102)]",
+  "mps_available": true,
+  "attribution": "Derived from Andrej Karpathy's nanochat project (https://github.com/karpathy/nanochat), MIT License, Copyright (c) 2025 Andrej Karpathy.",
+  "tokenizer_path": ".cache/dropout_decay_wikitext103/tokenizer-v4096.json",
+  "encoded_path": ".cache/dropout_decay_wikitext103/tokens-v4096-uint16.npy",
+  "train_tokens": 4500020,
+  "val_tokens": 500000,
+  "effective_token_limits": [
+    4500020
+  ],
+  "effective_stream_token_caps": [
+    250000,
+    500000,
+    1000000,
+    2000000,
+    4000000
+  ],
+  "resume_from": null
+}

runs/wikitext103_interaction_l12_validation_5seed/locked_stream/20260603-223707/metrics.jsonl

@@ -0,0 +1,25 @@
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.169, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 88.86421489715576, "eval_loss": 5.172895111143589, "generalization_gap": 1.1988903880119324, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 1, "stage": 0, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_eval_loss": 3.9740047231316566, "train_loss_last": 3.9852211475372314, "val_eval_loss": 5.172895111143589}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.096, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 89.19699573516846, "eval_loss": 4.78694075345993, "generalization_gap": 1.2169531509280205, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 1, "stage": 1, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_eval_loss": 3.56998760253191, "train_loss_last": 3.6730072498321533, "val_eval_loss": 4.78694075345993}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.020000000000000004, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 95.75643372535706, "eval_loss": 4.552120991051197, "generalization_gap": 1.043304182589054, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 1, "stage": 2, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_eval_loss": 3.508816808462143, "train_loss_last": 3.639979600906372, "val_eval_loss": 4.552120991051197}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.02, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 101.766676902771, "eval_loss": 4.3293517008423805, "generalization_gap": 0.6724446937441826, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 1, "stage": 3, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_eval_loss": 3.656907007098198, "train_loss_last": 3.7773282527923584, "val_eval_loss": 4.3293517008423805}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.02, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 101.72339582443237, "eval_loss": 4.10006446018815, "generalization_gap": 0.32315868511796, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 1, "stage": 4, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_eval_loss": 3.7769057750701904, "train_loss_last": 3.8277108669281006, "val_eval_loss": 4.10006446018815}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.169, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 100.68080067634583, "eval_loss": 5.132068373262882, "generalization_gap": 1.1279677748680115, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 2, "stage": 0, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_eval_loss": 4.004100598394871, "train_loss_last": 3.994852304458618, "val_eval_loss": 5.132068373262882}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.096, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 100.10244989395142, "eval_loss": 4.826943509280682, "generalization_gap": 1.2454395145177841, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 2, "stage": 1, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_eval_loss": 3.5815039947628975, "train_loss_last": 3.5742015838623047, "val_eval_loss": 4.826943509280682}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.020000000000000004, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 99.72496390342712, "eval_loss": 4.552324756979942, "generalization_gap": 1.0586613416671753, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 2, "stage": 2, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_eval_loss": 3.493663415312767, "train_loss_last": 3.6720285415649414, "val_eval_loss": 4.552324756979942}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.02, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 99.5625011920929, "eval_loss": 4.28244262188673, "generalization_gap": 0.5976964607834816, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 2, "stage": 3, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_eval_loss": 3.6847461611032486, "train_loss_last": 3.656864643096924, "val_eval_loss": 4.28244262188673}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.02, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 99.59214997291565, "eval_loss": 4.13165595382452, "generalization_gap": 0.37044707685709, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 2, "stage": 4, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_eval_loss": 3.76120887696743, "train_loss_last": 3.8822860717773438, "val_eval_loss": 4.13165595382452}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.169, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 99.67933487892151, "eval_loss": 5.190158672630787, "generalization_gap": 1.2236372530460358, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 3, "stage": 0, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_eval_loss": 3.966521419584751, "train_loss_last": 4.312673568725586, "val_eval_loss": 5.190158672630787}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.096, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 99.75796294212341, "eval_loss": 4.8199815303087234, "generalization_gap": 1.2181328162550926, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 3, "stage": 1, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_eval_loss": 3.601848714053631, "train_loss_last": 3.5230610370635986, "val_eval_loss": 4.8199815303087234}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.020000000000000004, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 99.71509385108948, "eval_loss": 4.524693250656128, "generalization_gap": 0.990579828619957, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 3, "stage": 2, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_eval_loss": 3.534113422036171, "train_loss_last": 3.275038719177246, "val_eval_loss": 4.524693250656128}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.02, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 99.7238461971283, "eval_loss": 4.309528287500143, "generalization_gap": 0.619201872497797, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 3, "stage": 3, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_eval_loss": 3.690326415002346, "train_loss_last": 3.567577362060547, "val_eval_loss": 4.309528287500143}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.02, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 99.85416913032532, "eval_loss": 4.106807354837656, "generalization_gap": 0.30010075494647026, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 3, "stage": 4, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_eval_loss": 3.8067065998911858, "train_loss_last": 3.6170077323913574, "val_eval_loss": 4.106807354837656}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.169, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 100.05135989189148, "eval_loss": 5.170049726963043, "generalization_gap": 1.1773424223065376, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 4, "stage": 0, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_eval_loss": 3.9927073046565056, "train_loss_last": 4.145639419555664, "val_eval_loss": 5.170049726963043}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.096, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 100.06845307350159, "eval_loss": 4.789028272032738, "generalization_gap": 1.1823933944106102, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 4, "stage": 1, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_eval_loss": 3.6066348776221275, "train_loss_last": 3.81071138381958, "val_eval_loss": 4.789028272032738}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.020000000000000004, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 100.09587907791138, "eval_loss": 4.570456840097904, "generalization_gap": 1.053241342306137, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 4, "stage": 2, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_eval_loss": 3.517215497791767, "train_loss_last": 3.6085214614868164, "val_eval_loss": 4.570456840097904}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.02, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 99.89476823806763, "eval_loss": 4.294592976570129, "generalization_gap": 0.6181246787309647, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 4, "stage": 3, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_eval_loss": 3.6764682978391647, "train_loss_last": 3.6306161880493164, "val_eval_loss": 4.294592976570129}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.02, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 99.77848291397095, "eval_loss": 4.112830638885498, "generalization_gap": 0.37679261714220047, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 4, "stage": 4, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_eval_loss": 3.7360380217432976, "train_loss_last": 3.782350540161133, "val_eval_loss": 4.112830638885498}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.169, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 99.54863691329956, "eval_loss": 5.157319262623787, "generalization_gap": 1.1857041269540787, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 5, "stage": 0, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_eval_loss": 3.9716151356697083, "train_loss_last": 4.141390800476074, "val_eval_loss": 5.157319262623787}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.096, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 99.57024836540222, "eval_loss": 4.808546029031277, "generalization_gap": 1.248667687177658, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 5, "stage": 1, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_eval_loss": 3.5598783418536186, "train_loss_last": 3.5874385833740234, "val_eval_loss": 4.808546029031277}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.020000000000000004, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 99.53037786483765, "eval_loss": 4.5401230081915855, "generalization_gap": 0.9882521107792854, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 5, "stage": 2, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_eval_loss": 3.5518708974123, "train_loss_last": 3.6611013412475586, "val_eval_loss": 4.5401230081915855}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.02, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 99.4762670993805, "eval_loss": 4.295782174915075, "generalization_gap": 0.6376434676349163, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 5, "stage": 3, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_eval_loss": 3.658138707280159, "train_loss_last": 3.60209059715271, "val_eval_loss": 4.295782174915075}
+{"condition": "wikitext103_interaction_l12", "condition_kind": "anchor_decay", "dropout_active_final": 0.02, "dropout_final": 0.02, "dropout_initial": 0.169, "dropout_schedule": "log_prefix_anchor", "elapsed_sec": 99.59245181083679, "eval_loss": 4.109791781753302, "generalization_gap": 0.3644006811082363, "model_config": {"block_size": 128, "dropout": 0.169, "n_embd": 320, "n_head": 8, "n_layer": 12, "vocab_size": 4096}, "model_name": "L12_H8_D320", "n_embd": 320, "n_head": 8, "n_layer": 12, "parameters": 17367040, "run_mode": "locked_stream", "seed": 5, "stage": 4, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_eval_loss": 3.7453911006450653, "train_loss_last": 3.8772075176239014, "val_eval_loss": 4.109791781753302}

runs/wikitext103_interaction_l12_validation_5seed/locked_stream/20260603-223707/summary.csv

@@ -0,0 +1,6 @@
+run_mode,condition,condition_kind,stage,token_limit,model_name,n_layer,n_head,n_embd,parameters,dropout_initial,dropout_final,dropout_schedule,n,mean_train_eval_loss,std_train_eval_loss,mean_val_eval_loss,std_val_eval_loss,mean_generalization_gap,std_generalization_gap
+locked_stream,wikitext103_interaction_l12,anchor_decay,0,250000,L12_H8_D320,12,8,320,17367040,0.169,0.02,log_prefix_anchor,5,3.9817898362874984,0.015923397011313935,5.164498229324818,0.02158134743277662,1.1827083930373192,0.03525383625973596
+locked_stream,wikitext103_interaction_l12,anchor_decay,1,500000,L12_H8_D320,12,8,320,17367040,0.169,0.02,log_prefix_anchor,5,3.5839707061648367,0.020095656896275762,4.80628801882267,0.017968553706758778,1.222317312657833,0.026782372680582447
+locked_stream,wikitext103_interaction_l12,anchor_decay,2,1000000,L12_H8_D320,12,8,320,17367040,0.169,0.02,log_prefix_anchor,5,3.5211360082030296,0.0225550611213843,4.547943769395351,0.016920634687311954,1.0268077611923219,0.034585105159979174
+locked_stream,wikitext103_interaction_l12,anchor_decay,3,2000000,L12_H8_D320,12,8,320,17367040,0.169,0.02,log_prefix_anchor,5,3.6733173176646234,0.015244220517453392,4.302339552342891,0.017890154808098007,0.6290222346782685,0.02809063882377606
+locked_stream,wikitext103_interaction_l12,anchor_decay,4,4000000,L12_H8_D320,12,8,320,17367040,0.169,0.02,log_prefix_anchor,5,3.7652500748634337,0.02792200297086762,4.112230037897826,0.011843550953418717,0.3469799630343914,0.03357117300173699

runs/wikitext103_interaction_l12_validation_5seed/locked_stream/20260603-223707/summary.json

@@ -0,0 +1,112 @@
+[
+  {
+    "run_mode": "locked_stream",
+    "condition": "wikitext103_interaction_l12",
+    "condition_kind": "anchor_decay",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.169,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 5,
+    "mean_train_eval_loss": 3.9817898362874984,
+    "std_train_eval_loss": 0.015923397011313935,
+    "mean_val_eval_loss": 5.164498229324818,
+    "std_val_eval_loss": 0.02158134743277662,
+    "mean_generalization_gap": 1.1827083930373192,
+    "std_generalization_gap": 0.03525383625973596
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "wikitext103_interaction_l12",
+    "condition_kind": "anchor_decay",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.169,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 5,
+    "mean_train_eval_loss": 3.5839707061648367,
+    "std_train_eval_loss": 0.020095656896275762,
+    "mean_val_eval_loss": 4.80628801882267,
+    "std_val_eval_loss": 0.017968553706758778,
+    "mean_generalization_gap": 1.222317312657833,
+    "std_generalization_gap": 0.026782372680582447
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "wikitext103_interaction_l12",
+    "condition_kind": "anchor_decay",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.169,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 5,
+    "mean_train_eval_loss": 3.5211360082030296,
+    "std_train_eval_loss": 0.0225550611213843,
+    "mean_val_eval_loss": 4.547943769395351,
+    "std_val_eval_loss": 0.016920634687311954,
+    "mean_generalization_gap": 1.0268077611923219,
+    "std_generalization_gap": 0.034585105159979174
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "wikitext103_interaction_l12",
+    "condition_kind": "anchor_decay",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.169,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 5,
+    "mean_train_eval_loss": 3.6733173176646234,
+    "std_train_eval_loss": 0.015244220517453392,
+    "mean_val_eval_loss": 4.302339552342891,
+    "std_val_eval_loss": 0.017890154808098007,
+    "mean_generalization_gap": 0.6290222346782685,
+    "std_generalization_gap": 0.02809063882377606
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "wikitext103_interaction_l12",
+    "condition_kind": "anchor_decay",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.169,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 5,
+    "mean_train_eval_loss": 3.7652500748634337,
+    "std_train_eval_loss": 0.02792200297086762,
+    "mean_val_eval_loss": 4.112230037897826,
+    "std_val_eval_loss": 0.011843550953418717,
+    "mean_generalization_gap": 0.3469799630343914,
+    "std_generalization_gap": 0.03357117300173699
+  }
+]

runs/wikitext103_interaction_l12_validation_5seed/locked_stream/20260603-223707/trace.jsonl

@@ -0,0 +1,100 @@
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 250, "steps": 1000, "token_limit": 250000, "tokens_seen": 512000, "train_batch_loss": 5.610506057739258}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.939245223999023}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 750, "steps": 1000, "token_limit": 250000, "tokens_seen": 1536000, "train_batch_loss": 4.373546123504639}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 3.9852211475372314}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 250, "steps": 1000, "token_limit": 500000, "tokens_seen": 2560000, "train_batch_loss": 4.338537693023682}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.04083776473999}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 750, "steps": 1000, "token_limit": 500000, "tokens_seen": 3584000, "train_batch_loss": 3.7809929847717285}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.6730072498321533}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 250, "steps": 1000, "token_limit": 1000000, "tokens_seen": 4608000, "train_batch_loss": 4.025157451629639}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.72495174407959}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 750, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5632000, "train_batch_loss": 3.5732693672180176}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.639979600906372}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 250, "steps": 1000, "token_limit": 2000000, "tokens_seen": 6656000, "train_batch_loss": 3.7523860931396484}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.965656280517578}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 750, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7680000, "train_batch_loss": 3.672691822052002}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.7773282527923584}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 250, "steps": 1000, "token_limit": 4000000, "tokens_seen": 8704000, "train_batch_loss": 3.8104381561279297}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.014762878417969}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 750, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9728000, "train_batch_loss": 3.8591041564941406}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.8277108669281006}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 250, "steps": 1000, "token_limit": 250000, "tokens_seen": 512000, "train_batch_loss": 5.611138343811035}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.920900821685791}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 750, "steps": 1000, "token_limit": 250000, "tokens_seen": 1536000, "train_batch_loss": 4.570071220397949}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 3.994852304458618}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 250, "steps": 1000, "token_limit": 500000, "tokens_seen": 2560000, "train_batch_loss": 4.076385974884033}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.009974956512451}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 750, "steps": 1000, "token_limit": 500000, "tokens_seen": 3584000, "train_batch_loss": 3.7355096340179443}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.5742015838623047}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 250, "steps": 1000, "token_limit": 1000000, "tokens_seen": 4608000, "train_batch_loss": 3.942497730255127}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.8480563163757324}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 750, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5632000, "train_batch_loss": 3.7572739124298096}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.6720285415649414}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 250, "steps": 1000, "token_limit": 2000000, "tokens_seen": 6656000, "train_batch_loss": 3.770002841949463}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.8363704681396484}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 750, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7680000, "train_batch_loss": 3.646073341369629}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.656864643096924}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 250, "steps": 1000, "token_limit": 4000000, "tokens_seen": 8704000, "train_batch_loss": 3.9980361461639404}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.885284423828125}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 750, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9728000, "train_batch_loss": 3.66664981842041}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.8822860717773438}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 250, "steps": 1000, "token_limit": 250000, "tokens_seen": 512000, "train_batch_loss": 5.6200408935546875}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.732158660888672}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 750, "steps": 1000, "token_limit": 250000, "tokens_seen": 1536000, "train_batch_loss": 4.315240859985352}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.312673568725586}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 250, "steps": 1000, "token_limit": 500000, "tokens_seen": 2560000, "train_batch_loss": 4.405235767364502}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.9726994037628174}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 750, "steps": 1000, "token_limit": 500000, "tokens_seen": 3584000, "train_batch_loss": 3.9072346687316895}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.5230610370635986}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 250, "steps": 1000, "token_limit": 1000000, "tokens_seen": 4608000, "train_batch_loss": 3.7502152919769287}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.7605178356170654}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 750, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5632000, "train_batch_loss": 3.7687807083129883}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.275038719177246}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 250, "steps": 1000, "token_limit": 2000000, "tokens_seen": 6656000, "train_batch_loss": 3.8444292545318604}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.001398086547852}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 750, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7680000, "train_batch_loss": 3.9696779251098633}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.567577362060547}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 250, "steps": 1000, "token_limit": 4000000, "tokens_seen": 8704000, "train_batch_loss": 3.9231317043304443}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.9913241863250732}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 750, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9728000, "train_batch_loss": 3.782881259918213}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.6170077323913574}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 0, "step": 250, "steps": 1000, "token_limit": 250000, "tokens_seen": 512000, "train_batch_loss": 5.593486785888672}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.838343143463135}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 0, "step": 750, "steps": 1000, "token_limit": 250000, "tokens_seen": 1536000, "train_batch_loss": 4.415877342224121}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.145639419555664}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 1, "step": 250, "steps": 1000, "token_limit": 500000, "tokens_seen": 2560000, "train_batch_loss": 4.154808044433594}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.7614550590515137}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 1, "step": 750, "steps": 1000, "token_limit": 500000, "tokens_seen": 3584000, "train_batch_loss": 3.8882532119750977}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.81071138381958}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 2, "step": 250, "steps": 1000, "token_limit": 1000000, "tokens_seen": 4608000, "train_batch_loss": 3.99111008644104}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.64109468460083}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 2, "step": 750, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5632000, "train_batch_loss": 3.7251739501953125}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.6085214614868164}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 3, "step": 250, "steps": 1000, "token_limit": 2000000, "tokens_seen": 6656000, "train_batch_loss": 3.981437921524048}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.833103656768799}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 3, "step": 750, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7680000, "train_batch_loss": 3.8024282455444336}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.6306161880493164}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 4, "step": 250, "steps": 1000, "token_limit": 4000000, "tokens_seen": 8704000, "train_batch_loss": 3.9384255409240723}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.7228970527648926}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 4, "step": 750, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9728000, "train_batch_loss": 3.6619787216186523}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 4, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.782350540161133}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 0, "step": 250, "steps": 1000, "token_limit": 250000, "tokens_seen": 512000, "train_batch_loss": 5.540077209472656}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.972682952880859}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 0, "step": 750, "steps": 1000, "token_limit": 250000, "tokens_seen": 1536000, "train_batch_loss": 4.490813732147217}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.169, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.141390800476074}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 1, "step": 250, "steps": 1000, "token_limit": 500000, "tokens_seen": 2560000, "train_batch_loss": 4.299867630004883}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.906068801879883}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 1, "step": 750, "steps": 1000, "token_limit": 500000, "tokens_seen": 3584000, "train_batch_loss": 4.042389392852783}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.096, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.5874385833740234}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 2, "step": 250, "steps": 1000, "token_limit": 1000000, "tokens_seen": 4608000, "train_batch_loss": 3.7010841369628906}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.8934578895568848}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 2, "step": 750, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5632000, "train_batch_loss": 3.587571382522583}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.020000000000000004, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.6611013412475586}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 3, "step": 250, "steps": 1000, "token_limit": 2000000, "tokens_seen": 6656000, "train_batch_loss": 4.13946533203125}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.8667380809783936}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 3, "step": 750, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7680000, "train_batch_loss": 3.9042625427246094}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.60209059715271}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 4, "step": 250, "steps": 1000, "token_limit": 4000000, "tokens_seen": 8704000, "train_batch_loss": 4.191143989562988}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.8720788955688477}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 4, "step": 750, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9728000, "train_batch_loss": 3.8646321296691895}
+{"condition": "wikitext103_interaction_l12", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 5, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.8772075176239014}

runs/wikitext103_interaction_static_screen/screen_static/20260603-153314/RESULT_SUMMARY.md

@@ -0,0 +1,225 @@
+# Static Dropout Screen Summary
+
+Run directory: `runs/wikitext103_interaction_static_screen/screen_static/20260603-153314`
+
+## Models
+
+| Model | Params | Layers | Heads | Embedding | Block | Vocab | Seeds |
+|---|---:|---:|---:|---:|---:|---:|---|
+| `L8_H8_D256` | 8,388,608 | 8 | 8 | 256 | 128 | 4096 | 1, 2 |
+| `L12_H8_D320` | 17,367,040 | 12 | 8 | 320 | 128 | 4096 | 1, 2 |
+| `L16_H8_D384` | 31,457,280 | 16 | 8 | 384 | 128 | 4096 | 1, 2 |
+
+## Best Dropout By Model And Prefix
+
+| Model | Prefix tokens | Effective epochs | Best dropout | Mean val loss | Val std | Mean train loss | Mean gap | Plateau/bracket note |
+|---|---:|---:|---:|---:|---:|---:|---:|---|
+| `L12_H8_D320` | 250,000 | 8.19 | 0.18 | 5.1543 | 0.0165 | 4.0031 | 1.1512 | bracketed by tested grid |
+| `L12_H8_D320` | 500,000 | 4.10 | 0.02 | 4.9394 | 0.0278 | 4.1691 | 0.7703 | bracketed by tested grid |
+| `L12_H8_D320` | 1,000,000 | 2.05 | 0.00 | 4.8197 | 0.0241 | 4.4487 | 0.3710 | not bracketed; best at bottom of tested grid |
+| `L12_H8_D320` | 2,000,000 | 1.02 | 0.00 | 4.7700 | 0.0249 | 4.6217 | 0.1482 | not bracketed; best at bottom of tested grid |
+| `L12_H8_D320` | 4,000,000 | 0.51 | 0.00 | 4.7515 | 0.0202 | 4.7007 | 0.0509 | not bracketed; best at bottom of tested grid |
+| `L16_H8_D384` | 250,000 | 8.19 | 0.26 | 5.1221 | 0.0169 | 3.8513 | 1.2708 | bracketed by tested grid |
+| `L16_H8_D384` | 500,000 | 4.10 | 0.08 | 4.8685 | 0.0394 | 4.0107 | 0.8579 | bracketed by tested grid |
+| `L16_H8_D384` | 1,000,000 | 2.05 | 0.02 | 4.7050 | 0.0269 | 4.2259 | 0.4791 | bracketed by tested grid |
+| `L16_H8_D384` | 2,000,000 | 1.02 | 0.00 | 4.6385 | 0.0205 | 4.4509 | 0.1876 | not bracketed; best at bottom of tested grid |
+| `L16_H8_D384` | 4,000,000 | 0.51 | 0.00 | 4.6143 | 0.0343 | 4.5441 | 0.0701 | not bracketed; best at bottom of tested grid |
+| `L8_H8_D256` | 250,000 | 8.19 | 0.10 | 5.2265 | 0.0259 | 4.2361 | 0.9904 | bracketed by tested grid |
+| `L8_H8_D256` | 500,000 | 4.10 | 0.02 | 5.0569 | 0.0291 | 4.4976 | 0.5593 | bracketed by tested grid |
+| `L8_H8_D256` | 1,000,000 | 2.05 | 0.00 | 4.9719 | 0.0184 | 4.6959 | 0.2760 | not bracketed; best at bottom of tested grid |
+| `L8_H8_D256` | 2,000,000 | 1.02 | 0.00 | 4.9465 | 0.0275 | 4.8445 | 0.1020 | not bracketed; best at bottom of tested grid |
+| `L8_H8_D256` | 4,000,000 | 0.51 | 0.00 | 4.9316 | 0.0245 | 4.9095 | 0.0221 | not bracketed; best at bottom of tested grid |
+
+## Model `L12_H8_D320`
+
+### Prefix 250,000 Tokens (8.19 Effective Epochs)
+
+| Dropout | N | Mean val loss | Val std | Mean train loss | Train std | Mean gap | Gap std | Sampled tokens | Params |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0.00 | 2 | 5.3260 | 0.0422 | 3.5073 | 0.0570 | 1.8187 | 0.0992 | 2,048,000 | 17,367,040 |
+| 0.02 | 2 | 5.2632 | 0.0331 | 3.5965 | 0.0004 | 1.6667 | 0.0327 | 2,048,000 | 17,367,040 |
+| 0.04 | 2 | 5.2262 | 0.0105 | 3.6676 | 0.0015 | 1.5587 | 0.0120 | 2,048,000 | 17,367,040 |
+| 0.06 | 2 | 5.2008 | 0.0194 | 3.7293 | 0.0060 | 1.4715 | 0.0254 | 2,048,000 | 17,367,040 |
+| 0.08 | 2 | 5.1808 | 0.0188 | 3.7557 | 0.0010 | 1.4252 | 0.0198 | 2,048,000 | 17,367,040 |
+| 0.10 | 2 | 5.1633 | 0.0291 | 3.8189 | 0.0296 | 1.3444 | 0.0587 | 2,048,000 | 17,367,040 |
+| 0.14 | 2 | 5.1580 | 0.0300 | 3.9072 | 0.0161 | 1.2507 | 0.0461 | 2,048,000 | 17,367,040 |
+| 0.18 | 2 | 5.1543 | 0.0165 | 4.0031 | 0.0119 | 1.1512 | 0.0285 | 2,048,000 | 17,367,040 |
+| 0.20 | 2 | 5.1645 | 0.0166 | 4.0403 | 0.0022 | 1.1243 | 0.0188 | 2,048,000 | 17,367,040 |
+| 0.26 | 2 | 5.1880 | 0.0152 | 4.1549 | 0.0185 | 1.0331 | 0.0337 | 2,048,000 | 17,367,040 |
+| 0.30 | 2 | 5.2023 | 0.0220 | 4.2240 | 0.0076 | 0.9783 | 0.0295 | 2,048,000 | 17,367,040 |
+| 0.40 | 2 | 5.2852 | 0.0211 | 4.3954 | 0.0070 | 0.8898 | 0.0281 | 2,048,000 | 17,367,040 |
+
+### Prefix 500,000 Tokens (4.10 Effective Epochs)
+
+| Dropout | N | Mean val loss | Val std | Mean train loss | Train std | Mean gap | Gap std | Sampled tokens | Params |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0.00 | 2 | 4.9564 | 0.0217 | 4.1488 | 0.0159 | 0.8077 | 0.0376 | 2,048,000 | 17,367,040 |
+| 0.02 | 2 | 4.9394 | 0.0278 | 4.1691 | 0.0059 | 0.7703 | 0.0219 | 2,048,000 | 17,367,040 |
+| 0.04 | 2 | 4.9395 | 0.0305 | 4.2108 | 0.0219 | 0.7287 | 0.0523 | 2,048,000 | 17,367,040 |
+| 0.06 | 2 | 4.9519 | 0.0169 | 4.2442 | 0.0011 | 0.7076 | 0.0180 | 2,048,000 | 17,367,040 |
+| 0.08 | 2 | 4.9453 | 0.0273 | 4.2710 | 0.0081 | 0.6743 | 0.0193 | 2,048,000 | 17,367,040 |
+| 0.10 | 2 | 4.9435 | 0.0223 | 4.2929 | 0.0070 | 0.6505 | 0.0293 | 2,048,000 | 17,367,040 |
+| 0.14 | 2 | 4.9656 | 0.0397 | 4.3446 | 0.0120 | 0.6210 | 0.0277 | 2,048,000 | 17,367,040 |
+| 0.18 | 2 | 4.9909 | 0.0306 | 4.4019 | 0.0007 | 0.5890 | 0.0313 | 2,048,000 | 17,367,040 |
+
+### Prefix 1,000,000 Tokens (2.05 Effective Epochs)
+
+| Dropout | N | Mean val loss | Val std | Mean train loss | Train std | Mean gap | Gap std | Sampled tokens | Params |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0.00 | 2 | 4.8197 | 0.0241 | 4.4487 | 0.0273 | 0.3710 | 0.0514 | 2,048,000 | 17,367,040 |
+| 0.02 | 2 | 4.8201 | 0.0164 | 4.4513 | 0.0424 | 0.3688 | 0.0588 | 2,048,000 | 17,367,040 |
+| 0.04 | 2 | 4.8275 | 0.0226 | 4.4707 | 0.0245 | 0.3568 | 0.0471 | 2,048,000 | 17,367,040 |
+| 0.06 | 2 | 4.8371 | 0.0229 | 4.5027 | 0.0245 | 0.3344 | 0.0474 | 2,048,000 | 17,367,040 |
+| 0.08 | 2 | 4.8497 | 0.0291 | 4.5119 | 0.0209 | 0.3378 | 0.0500 | 2,048,000 | 17,367,040 |
+| 0.10 | 2 | 4.8641 | 0.0231 | 4.5423 | 0.0304 | 0.3219 | 0.0534 | 2,048,000 | 17,367,040 |
+
+### Prefix 2,000,000 Tokens (1.02 Effective Epochs)
+
+| Dropout | N | Mean val loss | Val std | Mean train loss | Train std | Mean gap | Gap std | Sampled tokens | Params |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0.00 | 2 | 4.7700 | 0.0249 | 4.6217 | 0.0121 | 0.1482 | 0.0370 | 2,048,000 | 17,367,040 |
+| 0.02 | 2 | 4.7767 | 0.0374 | 4.6368 | 0.0041 | 0.1399 | 0.0333 | 2,048,000 | 17,367,040 |
+| 0.04 | 2 | 4.7897 | 0.0231 | 4.6468 | 0.0118 | 0.1429 | 0.0349 | 2,048,000 | 17,367,040 |
+| 0.06 | 2 | 4.8023 | 0.0302 | 4.6640 | 0.0005 | 0.1383 | 0.0297 | 2,048,000 | 17,367,040 |
+| 0.08 | 2 | 4.8076 | 0.0293 | 4.6853 | 0.0007 | 0.1222 | 0.0301 | 2,048,000 | 17,367,040 |
+| 0.10 | 2 | 4.8234 | 0.0226 | 4.7042 | 0.0067 | 0.1192 | 0.0293 | 2,048,000 | 17,367,040 |
+
+### Prefix 4,000,000 Tokens (0.51 Effective Epochs)
+
+| Dropout | N | Mean val loss | Val std | Mean train loss | Train std | Mean gap | Gap std | Sampled tokens | Params |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0.00 | 2 | 4.7515 | 0.0202 | 4.7007 | 0.0076 | 0.0509 | 0.0278 | 2,048,000 | 17,367,040 |
+| 0.02 | 2 | 4.7540 | 0.0196 | 4.7119 | 0.0034 | 0.0421 | 0.0230 | 2,048,000 | 17,367,040 |
+| 0.04 | 2 | 4.7659 | 0.0174 | 4.7284 | 0.0003 | 0.0375 | 0.0170 | 2,048,000 | 17,367,040 |
+| 0.06 | 2 | 4.7811 | 0.0233 | 4.7395 | 0.0002 | 0.0415 | 0.0235 | 2,048,000 | 17,367,040 |
+| 0.08 | 2 | 4.7963 | 0.0201 | 4.7569 | 0.0019 | 0.0395 | 0.0220 | 2,048,000 | 17,367,040 |
+| 0.10 | 2 | 4.8031 | 0.0285 | 4.7715 | 0.0029 | 0.0316 | 0.0314 | 2,048,000 | 17,367,040 |
+
+## Model `L16_H8_D384`
+
+### Prefix 250,000 Tokens (8.19 Effective Epochs)
+
+| Dropout | N | Mean val loss | Val std | Mean train loss | Train std | Mean gap | Gap std | Sampled tokens | Params |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0.00 | 2 | 5.4595 | 0.0526 | 2.9845 | 0.0757 | 2.4751 | 0.1283 | 2,048,000 | 31,457,280 |
+| 0.02 | 2 | 5.3954 | 0.0284 | 3.0660 | 0.0044 | 2.3294 | 0.0328 | 2,048,000 | 31,457,280 |
+| 0.04 | 2 | 5.3303 | 0.0093 | 3.1689 | 0.0123 | 2.1615 | 0.0217 | 2,048,000 | 31,457,280 |
+| 0.06 | 2 | 5.2622 | 0.0201 | 3.2495 | 0.0081 | 2.0127 | 0.0283 | 2,048,000 | 31,457,280 |
+| 0.08 | 2 | 5.2327 | 0.0194 | 3.3454 | 0.0480 | 1.8873 | 0.0674 | 2,048,000 | 31,457,280 |
+| 0.10 | 2 | 5.1936 | 0.0337 | 3.4033 | 0.0246 | 1.7903 | 0.0583 | 2,048,000 | 31,457,280 |
+| 0.14 | 2 | 5.1390 | 0.0312 | 3.5568 | 0.0139 | 1.5822 | 0.0450 | 2,048,000 | 31,457,280 |
+| 0.18 | 2 | 5.1338 | 0.0336 | 3.6556 | 0.0150 | 1.4782 | 0.0487 | 2,048,000 | 31,457,280 |
+| 0.20 | 2 | 5.1319 | 0.0262 | 3.7215 | 0.0043 | 1.4104 | 0.0305 | 2,048,000 | 31,457,280 |
+| 0.26 | 2 | 5.1221 | 0.0169 | 3.8513 | 0.0265 | 1.2708 | 0.0434 | 2,048,000 | 31,457,280 |
+| 0.30 | 2 | 5.1361 | 0.0250 | 3.9618 | 0.0120 | 1.1744 | 0.0130 | 2,048,000 | 31,457,280 |
+| 0.40 | 2 | 5.1920 | 0.0291 | 4.1546 | 0.0162 | 1.0374 | 0.0453 | 2,048,000 | 31,457,280 |
+| 0.50 | 2 | 5.2978 | 0.0114 | 4.3726 | 0.0196 | 0.9252 | 0.0310 | 2,048,000 | 31,457,280 |
+
+### Prefix 500,000 Tokens (4.10 Effective Epochs)
+
+| Dropout | N | Mean val loss | Val std | Mean train loss | Train std | Mean gap | Gap std | Sampled tokens | Params |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0.00 | 2 | 4.9170 | 0.0340 | 3.8253 | 0.0196 | 1.0917 | 0.0536 | 2,048,000 | 31,457,280 |
+| 0.02 | 2 | 4.8898 | 0.0313 | 3.8656 | 0.0187 | 1.0243 | 0.0127 | 2,048,000 | 31,457,280 |
+| 0.04 | 2 | 4.8819 | 0.0406 | 3.9146 | 0.0147 | 0.9673 | 0.0553 | 2,048,000 | 31,457,280 |
+| 0.06 | 2 | 4.8721 | 0.0300 | 3.9588 | 0.0108 | 0.9133 | 0.0192 | 2,048,000 | 31,457,280 |
+| 0.08 | 2 | 4.8685 | 0.0394 | 4.0107 | 0.0191 | 0.8579 | 0.0585 | 2,048,000 | 31,457,280 |
+| 0.10 | 2 | 4.8708 | 0.0258 | 4.0535 | 0.0004 | 0.8172 | 0.0254 | 2,048,000 | 31,457,280 |
+| 0.14 | 2 | 4.8798 | 0.0319 | 4.1201 | 0.0016 | 0.7597 | 0.0303 | 2,048,000 | 31,457,280 |
+| 0.18 | 2 | 4.8872 | 0.0147 | 4.1666 | 0.0004 | 0.7206 | 0.0152 | 2,048,000 | 31,457,280 |
+| 0.20 | 2 | 4.8952 | 0.0159 | 4.1979 | 0.0026 | 0.6973 | 0.0133 | 2,048,000 | 31,457,280 |
+| 0.26 | 2 | 4.9269 | 0.0283 | 4.2903 | 0.0044 | 0.6366 | 0.0327 | 2,048,000 | 31,457,280 |
+
+### Prefix 1,000,000 Tokens (2.05 Effective Epochs)
+
+| Dropout | N | Mean val loss | Val std | Mean train loss | Train std | Mean gap | Gap std | Sampled tokens | Params |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0.00 | 2 | 4.7178 | 0.0091 | 4.2326 | 0.0529 | 0.4852 | 0.0620 | 2,048,000 | 31,457,280 |
+| 0.02 | 2 | 4.7050 | 0.0269 | 4.2259 | 0.0131 | 0.4791 | 0.0399 | 2,048,000 | 31,457,280 |
+| 0.04 | 2 | 4.7126 | 0.0266 | 4.2694 | 0.0193 | 0.4433 | 0.0459 | 2,048,000 | 31,457,280 |
+| 0.06 | 2 | 4.7185 | 0.0301 | 4.2917 | 0.0139 | 0.4269 | 0.0440 | 2,048,000 | 31,457,280 |
+| 0.08 | 2 | 4.7434 | 0.0308 | 4.3293 | 0.0333 | 0.4141 | 0.0640 | 2,048,000 | 31,457,280 |
+| 0.10 | 2 | 4.7458 | 0.0212 | 4.3520 | 0.0304 | 0.3938 | 0.0516 | 2,048,000 | 31,457,280 |
+
+### Prefix 2,000,000 Tokens (1.02 Effective Epochs)
+
+| Dropout | N | Mean val loss | Val std | Mean train loss | Train std | Mean gap | Gap std | Sampled tokens | Params |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0.00 | 2 | 4.6385 | 0.0205 | 4.4509 | 0.0109 | 0.1876 | 0.0314 | 2,048,000 | 31,457,280 |
+| 0.02 | 2 | 4.6496 | 0.0350 | 4.4682 | 0.0071 | 0.1814 | 0.0279 | 2,048,000 | 31,457,280 |
+| 0.04 | 2 | 4.6604 | 0.0167 | 4.4902 | 0.0116 | 0.1702 | 0.0283 | 2,048,000 | 31,457,280 |
+| 0.06 | 2 | 4.6754 | 0.0343 | 4.5046 | 0.0041 | 0.1708 | 0.0302 | 2,048,000 | 31,457,280 |
+| 0.08 | 2 | 4.6822 | 0.0240 | 4.5177 | 0.0201 | 0.1645 | 0.0440 | 2,048,000 | 31,457,280 |
+| 0.10 | 2 | 4.6994 | 0.0219 | 4.5430 | 0.0102 | 0.1564 | 0.0320 | 2,048,000 | 31,457,280 |
+
+### Prefix 4,000,000 Tokens (0.51 Effective Epochs)
+
+| Dropout | N | Mean val loss | Val std | Mean train loss | Train std | Mean gap | Gap std | Sampled tokens | Params |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0.00 | 2 | 4.6143 | 0.0343 | 4.5441 | 0.0081 | 0.0701 | 0.0263 | 2,048,000 | 31,457,280 |
+| 0.02 | 2 | 4.6346 | 0.0251 | 4.5610 | 0.0055 | 0.0737 | 0.0306 | 2,048,000 | 31,457,280 |
+| 0.04 | 2 | 4.6357 | 0.0209 | 4.5670 | 0.0040 | 0.0687 | 0.0169 | 2,048,000 | 31,457,280 |
+| 0.06 | 2 | 4.6578 | 0.0096 | 4.5902 | 0.0134 | 0.0676 | 0.0230 | 2,048,000 | 31,457,280 |
+| 0.08 | 2 | 4.6622 | 0.0294 | 4.6025 | 0.0040 | 0.0597 | 0.0334 | 2,048,000 | 31,457,280 |
+| 0.10 | 2 | 4.6801 | 0.0393 | 4.6249 | 0.0149 | 0.0552 | 0.0243 | 2,048,000 | 31,457,280 |
+
+## Model `L8_H8_D256`
+
+### Prefix 250,000 Tokens (8.19 Effective Epochs)
+
+| Dropout | N | Mean val loss | Val std | Mean train loss | Train std | Mean gap | Gap std | Sampled tokens | Params |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0.00 | 2 | 5.2829 | 0.0325 | 4.0355 | 0.0242 | 1.2473 | 0.0083 | 2,048,000 | 8,388,608 |
+| 0.02 | 2 | 5.2474 | 0.0314 | 4.0897 | 0.0085 | 1.1577 | 0.0399 | 2,048,000 | 8,388,608 |
+| 0.04 | 2 | 5.2361 | 0.0160 | 4.1260 | 0.0056 | 1.1101 | 0.0216 | 2,048,000 | 8,388,608 |
+| 0.06 | 2 | 5.2366 | 0.0126 | 4.1744 | 0.0101 | 1.0622 | 0.0025 | 2,048,000 | 8,388,608 |
+| 0.08 | 2 | 5.2283 | 0.0096 | 4.2114 | 0.0006 | 1.0169 | 0.0090 | 2,048,000 | 8,388,608 |
+| 0.10 | 2 | 5.2265 | 0.0259 | 4.2361 | 0.0030 | 0.9904 | 0.0289 | 2,048,000 | 8,388,608 |
+| 0.14 | 2 | 5.2395 | 0.0127 | 4.3010 | 0.0149 | 0.9385 | 0.0276 | 2,048,000 | 8,388,608 |
+| 0.18 | 2 | 5.2515 | 0.0088 | 4.3531 | 0.0033 | 0.8983 | 0.0122 | 2,048,000 | 8,388,608 |
+| 0.20 | 2 | 5.2710 | 0.0191 | 4.3844 | 0.0042 | 0.8867 | 0.0149 | 2,048,000 | 8,388,608 |
+| 0.26 | 2 | 5.3012 | 0.0227 | 4.4744 | 0.0087 | 0.8268 | 0.0315 | 2,048,000 | 8,388,608 |
+
+### Prefix 500,000 Tokens (4.10 Effective Epochs)
+
+| Dropout | N | Mean val loss | Val std | Mean train loss | Train std | Mean gap | Gap std | Sampled tokens | Params |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0.00 | 2 | 5.0576 | 0.0309 | 4.4639 | 0.0114 | 0.5937 | 0.0195 | 2,048,000 | 8,388,608 |
+| 0.02 | 2 | 5.0569 | 0.0291 | 4.4976 | 0.0064 | 0.5593 | 0.0354 | 2,048,000 | 8,388,608 |
+| 0.04 | 2 | 5.0656 | 0.0366 | 4.5205 | 0.0083 | 0.5450 | 0.0283 | 2,048,000 | 8,388,608 |
+| 0.06 | 2 | 5.0729 | 0.0254 | 4.5481 | 0.0001 | 0.5248 | 0.0253 | 2,048,000 | 8,388,608 |
+| 0.08 | 2 | 5.0890 | 0.0382 | 4.5768 | 0.0038 | 0.5122 | 0.0344 | 2,048,000 | 8,388,608 |
+| 0.10 | 2 | 5.0904 | 0.0286 | 4.5945 | 0.0007 | 0.4959 | 0.0292 | 2,048,000 | 8,388,608 |
+
+### Prefix 1,000,000 Tokens (2.05 Effective Epochs)
+
+| Dropout | N | Mean val loss | Val std | Mean train loss | Train std | Mean gap | Gap std | Sampled tokens | Params |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0.00 | 2 | 4.9719 | 0.0184 | 4.6959 | 0.0173 | 0.2760 | 0.0357 | 2,048,000 | 8,388,608 |
+| 0.02 | 2 | 4.9837 | 0.0262 | 4.7156 | 0.0170 | 0.2680 | 0.0432 | 2,048,000 | 8,388,608 |
+| 0.04 | 2 | 4.9948 | 0.0234 | 4.7364 | 0.0100 | 0.2585 | 0.0335 | 2,048,000 | 8,388,608 |
+| 0.06 | 2 | 5.0018 | 0.0208 | 4.7509 | 0.0170 | 0.2509 | 0.0377 | 2,048,000 | 8,388,608 |
+| 0.08 | 2 | 5.0105 | 0.0181 | 4.7671 | 0.0258 | 0.2433 | 0.0439 | 2,048,000 | 8,388,608 |
+| 0.10 | 2 | 5.0317 | 0.0372 | 4.7940 | 0.0014 | 0.2377 | 0.0386 | 2,048,000 | 8,388,608 |
+
+### Prefix 2,000,000 Tokens (1.02 Effective Epochs)
+
+| Dropout | N | Mean val loss | Val std | Mean train loss | Train std | Mean gap | Gap std | Sampled tokens | Params |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0.00 | 2 | 4.9465 | 0.0275 | 4.8445 | 0.0008 | 0.1020 | 0.0266 | 2,048,000 | 8,388,608 |
+| 0.02 | 2 | 4.9572 | 0.0252 | 4.8670 | 0.0043 | 0.0902 | 0.0209 | 2,048,000 | 8,388,608 |
+| 0.04 | 2 | 4.9684 | 0.0193 | 4.8806 | 0.0025 | 0.0877 | 0.0168 | 2,048,000 | 8,388,608 |
+| 0.06 | 2 | 4.9771 | 0.0260 | 4.8913 | 0.0029 | 0.0858 | 0.0231 | 2,048,000 | 8,388,608 |
+| 0.08 | 2 | 4.9960 | 0.0274 | 4.9174 | 0.0002 | 0.0785 | 0.0272 | 2,048,000 | 8,388,608 |
+| 0.10 | 2 | 5.0049 | 0.0206 | 4.9288 | 0.0014 | 0.0762 | 0.0220 | 2,048,000 | 8,388,608 |
+
+### Prefix 4,000,000 Tokens (0.51 Effective Epochs)
+
+| Dropout | N | Mean val loss | Val std | Mean train loss | Train std | Mean gap | Gap std | Sampled tokens | Params |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0.00 | 2 | 4.9316 | 0.0245 | 4.9095 | 0.0055 | 0.0221 | 0.0190 | 2,048,000 | 8,388,608 |
+| 0.02 | 2 | 4.9393 | 0.0250 | 4.9243 | 0.0102 | 0.0149 | 0.0148 | 2,048,000 | 8,388,608 |
+| 0.04 | 2 | 4.9472 | 0.0254 | 4.9359 | 0.0032 | 0.0112 | 0.0221 | 2,048,000 | 8,388,608 |
+| 0.06 | 2 | 4.9728 | 0.0248 | 4.9553 | 0.0086 | 0.0174 | 0.0162 | 2,048,000 | 8,388,608 |
+| 0.08 | 2 | 4.9849 | 0.0178 | 4.9689 | 0.0017 | 0.0160 | 0.0195 | 2,048,000 | 8,388,608 |
+| 0.10 | 2 | 4.9918 | 0.0205 | 4.9850 | 0.0038 | 0.0068 | 0.0167 | 2,048,000 | 8,388,608 |

runs/wikitext103_interaction_static_screen/screen_static/20260603-153314/config.json

@@ -0,0 +1,124 @@
+{
+  "args": {
+    "mode": "screen_static",
+    "corpus": null,
+    "corpus_glob": null,
+    "text_column": "text",
+    "use_cached_data": true,
+    "output_dir": "runs/wikitext103_interaction_static_screen",
+    "resume_from": null,
+    "cache_dir": ".cache/dropout_decay_wikitext103",
+    "models": [
+      "L8_H8_D256=8x8x256",
+      "L12_H8_D320=12x8x320",
+      "L16_H8_D384=16x8x384"
+    ],
+    "seeds": [
+      1,
+      2
+    ],
+    "token_limits": [
+      250000,
+      500000,
+      1000000,
+      2000000,
+      4000000
+    ],
+    "stream_token_caps": [
+      5000000,
+      10000000,
+      20000000,
+      40000000
+    ],
+    "val_tokens": 500000,
+    "allow_short_corpus": false,
+    "force_retokenize": false,
+    "vocab_size": 4096,
+    "tokenizer_train_chars": 10000000,
+    "block_size": 128,
+    "batch_size": 16,
+    "steps": 1000,
+    "stage_steps": 1000,
+    "dropout_rates": [
+      0.0,
+      0.02,
+      0.04,
+      0.06,
+      0.08,
+      0.1,
+      0.14,
+      0.18,
+      0.2,
+      0.26,
+      0.3,
+      0.4,
+      0.5
+    ],
+    "decays": [],
+    "anchor_decays": [],
+    "decay_tokens": null,
+    "eval_batches": 64,
+    "train_eval_batches": 32,
+    "trace_eval_batches": 8,
+    "eval_every": 0,
+    "log_every": 100,
+    "lr": 0.0003,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "plateau_delta": 0.01,
+    "target_min_dropout": 0.1,
+    "min_nonzero_margin": 0.01,
+    "min_high_dropout_margin": 0.03,
+    "screen_early_stop": true,
+    "screen_prune_patience": 3,
+    "screen_prune_min_delta": 0.01
+  },
+  "mode": "screen_static",
+  "seeds": [
+    1,
+    2
+  ],
+  "models": [
+    {
+      "model_name": "L8_H8_D256",
+      "n_layer": 8,
+      "n_head": 8,
+      "n_embd": 256
+    },
+    {
+      "model_name": "L12_H8_D320",
+      "n_layer": 12,
+      "n_head": 8,
+      "n_embd": 320
+    },
+    {
+      "model_name": "L16_H8_D384",
+      "n_layer": 16,
+      "n_head": 8,
+      "n_embd": 384
+    }
+  ],
+  "device": "mps",
+  "torch": "2.12.0",
+  "python": "3.11.15 (main, Mar  3 2026, 00:52:57) [Clang 21.0.0 (clang-2100.0.123.102)]",
+  "mps_available": true,
+  "attribution": "Derived from Andrej Karpathy's nanochat project (https://github.com/karpathy/nanochat), MIT License, Copyright (c) 2025 Andrej Karpathy.",
+  "tokenizer_path": ".cache/dropout_decay_wikitext103/tokenizer-v4096.json",
+  "encoded_path": ".cache/dropout_decay_wikitext103/tokens-v4096-uint16.npy",
+  "train_tokens": 4500020,
+  "val_tokens": 500000,
+  "effective_token_limits": [
+    250000,
+    500000,
+    1000000,
+    2000000,
+    4000000
+  ],
+  "effective_stream_token_caps": [
+    4500020,
+    4500020,
+    4500020,
+    4500020
+  ],
+  "resume_from": null
+}

runs/wikitext103_interaction_static_screen/screen_static/20260603-153314/model_selection.csv

@@ -0,0 +1,16 @@
+run_mode,token_limit,model_name,n_layer,n_head,n_embd,parameters,n,best_dropout,best_val_loss,best_val_std,plateau_start_dropout,plateau_end_dropout,plateau_delta,zero_dropout_val_loss,zero_minus_best,best_nonzero_dropout,best_nonzero_val_loss,zero_minus_best_nonzero,max_dropout,max_dropout_val_loss,max_dropout_minus_best,has_nonzero_optimum,meets_target_dropout,curve_json
+screen_static,250000,L16_H8_D384,16,8,384,31457280,2,0.26,5.1221378818154335,0.016875071308155583,0.2,0.26,0.01,5.459531031548977,0.3373931497335434,0.26,5.1221378818154335,0.3373931497335434,0.5,5.297821056097746,0.1756831742823124,True,True,"[{""dropout"": 0.0, ""mean_generalization_gap"": 2.47505421936512, ""mean_train_loss"": 2.984476812183857, ""mean_val_loss"": 5.459531031548977, ""n"": 2, ""std_val_loss"": 0.052592860620622094}, {""dropout"": 0.02, ""mean_generalization_gap"": 2.329351920634508, ""mean_train_loss"": 3.066030155867338, ""mean_val_loss"": 5.395382076501846, ""n"": 2, ""std_val_loss"": 0.028443274869622583}, {""dropout"": 0.04, ""mean_generalization_gap"": 2.1614660173654556, ""mean_train_loss"": 3.1688508801162243, ""mean_val_loss"": 5.33031689748168, ""n"": 2, ""std_val_loss"": 0.009347870703420656}, {""dropout"": 0.06, ""mean_generalization_gap"": 2.012665204703808, ""mean_train_loss"": 3.2494926266372204, ""mean_val_loss"": 5.262157831341028, ""n"": 2, ""std_val_loss"": 0.02011042671888979}, {""dropout"": 0.08, ""mean_generalization_gap"": 1.8873155526816845, ""mean_train_loss"": 3.3453744649887085, ""mean_val_loss"": 5.232690017670393, ""n"": 2, ""std_val_loss"": 0.019404035001135078}, {""dropout"": 0.1, ""mean_generalization_gap"": 1.7902754880487919, ""mean_train_loss"": 3.4033044539391994, ""mean_val_loss"": 5.193579941987991, ""n"": 2, ""std_val_loss"": 0.03371155717649944}, {""dropout"": 0.14, ""mean_generalization_gap"": 1.5821974352002144, ""mean_train_loss"": 3.556764382869005, ""mean_val_loss"": 5.13896181806922, ""n"": 2, ""std_val_loss"": 0.031193730788238956}, {""dropout"": 0.18, ""mean_generalization_gap"": 1.4782251790165901, ""mean_train_loss"": 3.6555825024843216, ""mean_val_loss"": 5.133807681500912, ""n"": 2, ""std_val_loss"": 0.03364965399274907}, {""dropout"": 0.2, ""mean_generalization_gap"": 1.410427700728178, ""mean_train_loss"": 3.7214544899761677, ""mean_val_loss"": 5.131882190704346, ""n"": 2, ""std_val_loss"": 0.026161212683575973}, {""dropout"": 0.26, ""mean_generalization_gap"": 1.2708128467202187, ""mean_train_loss"": 3.851325035095215, ""mean_val_loss"": 5.1221378818154335, ""n"": 2, ""std_val_loss"": 0.016875071308155583}, {""dropout"": 0.3, ""mean_generalization_gap"": 1.1743715591728687, ""mean_train_loss"": 3.9617567881941795, ""mean_val_loss"": 5.136128347367048, ""n"": 2, ""std_val_loss"": 0.02499181149314877}, {""dropout"": 0.4, ""mean_generalization_gap"": 1.0374067649245262, ""mean_train_loss"": 4.154601521790028, ""mean_val_loss"": 5.192008286714554, ""n"": 2, ""std_val_loss"": 0.029119299783025198}, {""dropout"": 0.5, ""mean_generalization_gap"": 0.9252229072153568, ""mean_train_loss"": 4.372598148882389, ""mean_val_loss"": 5.297821056097746, ""n"": 2, ""std_val_loss"": 0.011355662272782625}]"
+screen_static,250000,L12_H8_D320,12,8,320,17367040,2,0.18,5.154339708387852,0.01651832984564725,0.1,0.18,0.01,5.325984805822372,0.17164509743452072,0.18,5.154339708387852,0.17164509743452072,0.4,5.285179540514946,0.13083983212709427,True,True,"[{""dropout"": 0.0, ""mean_generalization_gap"": 1.81871847063303, ""mean_train_loss"": 3.5072663351893425, ""mean_val_loss"": 5.325984805822372, ""n"": 2, ""std_val_loss"": 0.042196002272969865}, {""dropout"": 0.02, ""mean_generalization_gap"": 1.666674442589283, ""mean_train_loss"": 3.5965475998818874, ""mean_val_loss"": 5.26322204247117, ""n"": 2, ""std_val_loss"": 0.03309956913105307}, {""dropout"": 0.04, ""mean_generalization_gap"": 1.5586755871772766, ""mean_train_loss"": 3.6675546914339066, ""mean_val_loss"": 5.226230278611183, ""n"": 2, ""std_val_loss"": 0.010485914638555752}, {""dropout"": 0.06, ""mean_generalization_gap"": 1.4714824222028255, ""mean_train_loss"": 3.729271735996008, ""mean_val_loss"": 5.2007541581988335, ""n"": 2, ""std_val_loss"": 0.019410783765252886}, {""dropout"": 0.08, ""mean_generalization_gap"": 1.4251716993749142, ""mean_train_loss"": 3.7556542232632637, ""mean_val_loss"": 5.180825922638178, ""n"": 2, ""std_val_loss"": 0.0188014509712627}, {""dropout"": 0.1, ""mean_generalization_gap"": 1.3444151766598225, ""mean_train_loss"": 3.818895984441042, ""mean_val_loss"": 5.163311161100864, ""n"": 2, ""std_val_loss"": 0.029076731466029196}, {""dropout"": 0.14, ""mean_generalization_gap"": 1.2507423609495163, ""mean_train_loss"": 3.9072264805436134, ""mean_val_loss"": 5.15796884149313, ""n"": 2, ""std_val_loss"": 0.03002937668292093}, {""dropout"": 0.18, ""mean_generalization_gap"": 1.151239987462759, ""mean_train_loss"": 4.003099720925093, ""mean_val_loss"": 5.154339708387852, ""n"": 2, ""std_val_loss"": 0.01651832984564725}, {""dropout"": 0.2, ""mean_generalization_gap"": 1.1242803186178207, ""mean_train_loss"": 4.040268834680319, ""mean_val_loss"": 5.16454915329814, ""n"": 2, ""std_val_loss"": 0.016607855020240453}, {""dropout"": 0.26, ""mean_generalization_gap"": 1.0330531522631645, ""mean_train_loss"": 4.154937222599983, ""mean_val_loss"": 5.187990374863148, ""n"": 2, ""std_val_loss"": 0.015247644509892305}, {""dropout"": 0.3, ""mean_generalization_gap"": 0.9783092141151428, ""mean_train_loss"": 4.223974447697401, ""mean_val_loss"": 5.202283661812544, ""n"": 2, ""std_val_loss"": 0.021969144885068726}, {""dropout"": 0.4, ""mean_generalization_gap"": 0.8897631838917732, ""mean_train_loss"": 4.395416356623173, ""mean_val_loss"": 5.285179540514946, ""n"": 2, ""std_val_loss"": 0.021146865138977247}]"
+screen_static,250000,L8_H8_D256,8,8,256,8388608,2,0.1,5.226473439484835,0.02592699737804487,0.04,0.1,0.01,5.28285102173686,0.056377582252025604,0.1,5.226473439484835,0.056377582252025604,0.26,5.301199935376644,0.07472649589180946,True,True,"[{""dropout"": 0.0, ""mean_generalization_gap"": 1.2473031356930733, ""mean_train_loss"": 4.035547886043787, ""mean_val_loss"": 5.28285102173686, ""n"": 2, ""std_val_loss"": 0.0324948988321794}, {""dropout"": 0.02, ""mean_generalization_gap"": 1.1577380523085594, ""mean_train_loss"": 4.089681029319763, ""mean_val_loss"": 5.247419081628323, ""n"": 2, ""std_val_loss"": 0.03137372417316079}, {""dropout"": 0.04, ""mean_generalization_gap"": 1.1100559122860432, ""mean_train_loss"": 4.125994935631752, ""mean_val_loss"": 5.236050847917795, ""n"": 2, ""std_val_loss"": 0.01603267697861228}, {""dropout"": 0.06, ""mean_generalization_gap"": 1.0622206516563892, ""mean_train_loss"": 4.174422111362219, ""mean_val_loss"": 5.236642763018608, ""n"": 2, ""std_val_loss"": 0.012643158932737352}, {""dropout"": 0.08, ""mean_generalization_gap"": 1.016893107444048, ""mean_train_loss"": 4.211392514407635, ""mean_val_loss"": 5.228285621851683, ""n"": 2, ""std_val_loss"": 0.009615418897767812}, {""dropout"": 0.1, ""mean_generalization_gap"": 0.990381870418787, ""mean_train_loss"": 4.236091569066048, ""mean_val_loss"": 5.226473439484835, ""n"": 2, ""std_val_loss"": 0.02592699737804487}, {""dropout"": 0.14, ""mean_generalization_gap"": 0.9385448731482029, ""mean_train_loss"": 4.300983510911465, ""mean_val_loss"": 5.239528384059668, ""n"": 2, ""std_val_loss"": 0.01265994918351288}, {""dropout"": 0.18, ""mean_generalization_gap"": 0.8983461037278175, ""mean_train_loss"": 4.3531088307499886, ""mean_val_loss"": 5.251454934477806, ""n"": 2, ""std_val_loss"": 0.008830207304941664}, {""dropout"": 0.2, ""mean_generalization_gap"": 0.8866863176226616, ""mean_train_loss"": 4.38435186445713, ""mean_val_loss"": 5.271038182079792, ""n"": 2, ""std_val_loss"": 0.01905936334236905}, {""dropout"": 0.26, ""mean_generalization_gap"": 0.8267565593123436, ""mean_train_loss"": 4.4744433760643005, ""mean_val_loss"": 5.301199935376644, ""n"": 2, ""std_val_loss"": 0.022748898461012714}]"
+screen_static,1000000,L16_H8_D384,16,8,384,31457280,2,0.02,4.7049963511526585,0.026859896796409256,0.02,0.04,0.01,4.7177883349359035,0.012791983783245087,0.02,4.7049963511526585,0.012791983783245087,0.1,4.745819546282291,0.04082319512963295,True,False,"[{""dropout"": 0.0, ""mean_generalization_gap"": 0.48521145433187485, ""mean_train_loss"": 4.232576880604029, ""mean_val_loss"": 4.7177883349359035, ""n"": 2, ""std_val_loss"": 0.009086444338819707}, {""dropout"": 0.02, ""mean_generalization_gap"": 0.4790714867413044, ""mean_train_loss"": 4.225924864411354, ""mean_val_loss"": 4.7049963511526585, ""n"": 2, ""std_val_loss"": 0.026859896796409256}, {""dropout"": 0.04, ""mean_generalization_gap"": 0.4432833231985569, ""mean_train_loss"": 4.269359432160854, ""mean_val_loss"": 4.712642755359411, ""n"": 2, ""std_val_loss"": 0.026603285709250676}, {""dropout"": 0.06, ""mean_generalization_gap"": 0.4268508180975914, ""mean_train_loss"": 4.291698649525642, ""mean_val_loss"": 4.718549467623234, ""n"": 2, ""std_val_loss"": 0.030084041145439562}, {""dropout"": 0.08, ""mean_generalization_gap"": 0.41410549730062485, ""mean_train_loss"": 4.3292553797364235, ""mean_val_loss"": 4.743360877037048, ""n"": 2, ""std_val_loss"": 0.03075722114656769}, {""dropout"": 0.1, ""mean_generalization_gap"": 0.3938281238079071, ""mean_train_loss"": 4.351991422474384, ""mean_val_loss"": 4.745819546282291, ""n"": 2, ""std_val_loss"": 0.021169972148673347}]"
+screen_static,500000,L16_H8_D384,16,8,384,31457280,2,0.08,4.868544135242701,0.03936356019728768,0.06,0.1,0.01,4.916963316500187,0.04841918125748634,0.08,4.868544135242701,0.04841918125748634,0.26,4.926884576678276,0.058340441435575485,True,False,"[{""dropout"": 0.0, ""mean_generalization_gap"": 1.0916679799556732, ""mean_train_loss"": 3.8252953365445137, ""mean_val_loss"": 4.916963316500187, ""n"": 2, ""std_val_loss"": 0.03401019920833551}, {""dropout"": 0.02, ""mean_generalization_gap"": 1.024292692542076, ""mean_train_loss"": 3.8655518889427185, ""mean_val_loss"": 4.889844581484795, ""n"": 2, ""std_val_loss"": 0.03132540281134305}, {""dropout"": 0.04, ""mean_generalization_gap"": 0.9672790542244911, ""mean_train_loss"": 3.914637830108404, ""mean_val_loss"": 4.881916884332895, ""n"": 2, ""std_val_loss"": 0.04055301801925}, {""dropout"": 0.06, ""mean_generalization_gap"": 0.9132770709693432, ""mean_train_loss"": 3.9587855264544487, ""mean_val_loss"": 4.872062597423792, ""n"": 2, ""std_val_loss"": 0.029954139289972922}, {""dropout"": 0.08, ""mean_generalization_gap"": 0.8578500226140022, ""mean_train_loss"": 4.010694112628698, ""mean_val_loss"": 4.868544135242701, ""n"": 2, ""std_val_loss"": 0.03936356019728768}, {""dropout"": 0.1, ""mean_generalization_gap"": 0.8172299675643444, ""mean_train_loss"": 4.053545691072941, ""mean_val_loss"": 4.870775658637285, ""n"": 2, ""std_val_loss"": 0.02582369545234467}, {""dropout"": 0.14, ""mean_generalization_gap"": 0.7596709467470646, ""mean_train_loss"": 4.120105978101492, ""mean_val_loss"": 4.8797769248485565, ""n"": 2, ""std_val_loss"": 0.03186888642289103}, {""dropout"": 0.18, ""mean_generalization_gap"": 0.7206361852586269, ""mean_train_loss"": 4.1665811985731125, ""mean_val_loss"": 4.887217383831739, ""n"": 2, ""std_val_loss"": 0.014720445386937724}, {""dropout"": 0.2, ""mean_generalization_gap"": 0.6973064877092838, ""mean_train_loss"": 4.197941534221172, ""mean_val_loss"": 4.895248021930456, ""n"": 2, ""std_val_loss"": 0.015913675351841773}, {""dropout"": 0.26, ""mean_generalization_gap"": 0.636580154299736, ""mean_train_loss"": 4.29030442237854, ""mean_val_loss"": 4.926884576678276, ""n"": 2, ""std_val_loss"": 0.028275551485973483}]"
+screen_static,500000,L12_H8_D320,12,8,320,17367040,2,0.02,4.939401715993881,0.02775412121290883,0.02,0.1,0.01,4.956410575658083,0.017008859664201736,0.02,4.939401715993881,0.017008859664201736,0.18,4.9909077398478985,0.05150602385401726,True,False,"[{""dropout"": 0.0, ""mean_generalization_gap"": 0.8076566644012928, ""mean_train_loss"": 4.14875391125679, ""mean_val_loss"": 4.956410575658083, ""n"": 2, ""std_val_loss"": 0.021681071175496766}, {""dropout"": 0.02, ""mean_generalization_gap"": 0.7702712826430798, ""mean_train_loss"": 4.1691304333508015, ""mean_val_loss"": 4.939401715993881, ""n"": 2, ""std_val_loss"": 0.02775412121290883}, {""dropout"": 0.04, ""mean_generalization_gap"": 0.728732842952013, ""mean_train_loss"": 4.210759695619345, ""mean_val_loss"": 4.939492538571358, ""n"": 2, ""std_val_loss"": 0.030459295611156307}, {""dropout"": 0.06, ""mean_generalization_gap"": 0.7076257020235062, ""mean_train_loss"": 4.244226388633251, ""mean_val_loss"": 4.951852090656757, ""n"": 2, ""std_val_loss"": 0.0168901282697861}, {""dropout"": 0.08, ""mean_generalization_gap"": 0.6742749661207199, ""mean_train_loss"": 4.271009549498558, ""mean_val_loss"": 4.945284515619278, ""n"": 2, ""std_val_loss"": 0.027335982328832248}, {""dropout"": 0.1, ""mean_generalization_gap"": 0.6505224779248238, ""mean_train_loss"": 4.292945623397827, ""mean_val_loss"": 4.943468101322651, ""n"": 2, ""std_val_loss"": 0.02226113298319614}, {""dropout"": 0.14, ""mean_generalization_gap"": 0.6210011504590511, ""mean_train_loss"": 4.344617947936058, ""mean_val_loss"": 4.965619098395109, ""n"": 2, ""std_val_loss"": 0.039715876240702365}, {""dropout"": 0.18, ""mean_generalization_gap"": 0.5889932848513126, ""mean_train_loss"": 4.401914454996586, ""mean_val_loss"": 4.9909077398478985, ""n"": 2, ""std_val_loss"": 0.030591188905215085}]"
+screen_static,4000000,L16_H8_D384,16,8,384,31457280,2,0.0,4.614261016249657,0.0343418000757071,0.0,0.0,0.01,4.614261016249657,0.0,0.02,4.634637780487537,-0.020376764237880707,0.1,4.680121582001448,0.065860565751791,False,False,"[{""dropout"": 0.0, ""mean_generalization_gap"": 0.07013300061225891, ""mean_train_loss"": 4.544128015637398, ""mean_val_loss"": 4.614261016249657, ""n"": 2, ""std_val_loss"": 0.0343418000757071}, {""dropout"": 0.02, ""mean_generalization_gap"": 0.07366831600666046, ""mean_train_loss"": 4.560969464480877, ""mean_val_loss"": 4.634637780487537, ""n"": 2, ""std_val_loss"": 0.02513911999705041}, {""dropout"": 0.04, ""mean_generalization_gap"": 0.06866001710295677, ""mean_train_loss"": 4.567037917673588, ""mean_val_loss"": 4.635697934776545, ""n"": 2, ""std_val_loss"": 0.02091666432076681}, {""dropout"": 0.06, ""mean_generalization_gap"": 0.06760161370038986, ""mean_train_loss"": 4.590242467820644, ""mean_val_loss"": 4.657844081521034, ""n"": 2, ""std_val_loss"": 0.00958948278086542}, {""dropout"": 0.08, ""mean_generalization_gap"": 0.05967212840914726, ""mean_train_loss"": 4.602519556879997, ""mean_val_loss"": 4.6621916852891445, ""n"": 2, ""std_val_loss"": 0.029447955639357086}, {""dropout"": 0.1, ""mean_generalization_gap"": 0.05522836372256279, ""mean_train_loss"": 4.624893218278885, ""mean_val_loss"": 4.680121582001448, ""n"": 2, ""std_val_loss"": 0.03926061651979982}]"
+screen_static,2000000,L16_H8_D384,16,8,384,31457280,2,0.0,4.638469863682985,0.020520631458734194,0.0,0.0,0.01,4.638469863682985,0.0,0.02,4.649582006037235,-0.011112142354249954,0.1,4.699402842670679,0.06093297898769379,False,False,"[{""dropout"": 0.0, ""mean_generalization_gap"": 0.18756551668047905, ""mean_train_loss"": 4.450904347002506, ""mean_val_loss"": 4.638469863682985, ""n"": 2, ""std_val_loss"": 0.020520631458734194}, {""dropout"": 0.02, ""mean_generalization_gap"": 0.1813756451010704, ""mean_train_loss"": 4.468206360936165, ""mean_val_loss"": 4.649582006037235, ""n"": 2, ""std_val_loss"": 0.03501839397156461}, {""dropout"": 0.04, ""mean_generalization_gap"": 0.17023906484246254, ""mean_train_loss"": 4.490154348313808, ""mean_val_loss"": 4.660393413156271, ""n"": 2, ""std_val_loss"": 0.01666943156591487}, {""dropout"": 0.06, ""mean_generalization_gap"": 0.17083461582660675, ""mean_train_loss"": 4.50459598004818, ""mean_val_loss"": 4.675430595874786, ""n"": 2, ""std_val_loss"": 0.03429217216158552}, {""dropout"": 0.08, ""mean_generalization_gap"": 0.16451359167695045, ""mean_train_loss"": 4.517679311335087, ""mean_val_loss"": 4.682192903012037, ""n"": 2, ""std_val_loss"": 0.023959266388192506}, {""dropout"": 0.1, ""mean_generalization_gap"": 0.15638693794608116, ""mean_train_loss"": 4.543015904724598, ""mean_val_loss"": 4.699402842670679, ""n"": 2, ""std_val_loss"": 0.02188920285015569}]"
+screen_static,4000000,L12_H8_D320,12,8,320,17367040,2,0.0,4.751548711210489,0.02019173852637943,0.0,0.02,0.01,4.751548711210489,0.0,0.02,4.754005387425423,-0.0024566762149333954,0.1,4.803060214966536,0.051511503756046295,False,False,"[{""dropout"": 0.0, ""mean_generalization_gap"": 0.05085295811295509, ""mean_train_loss"": 4.700695753097534, ""mean_val_loss"": 4.751548711210489, ""n"": 2, ""std_val_loss"": 0.02019173852637943}, {""dropout"": 0.02, ""mean_generalization_gap"": 0.04213355481624603, ""mean_train_loss"": 4.711871832609177, ""mean_val_loss"": 4.754005387425423, ""n"": 2, ""std_val_loss"": 0.019647870324838793}, {""dropout"": 0.04, ""mean_generalization_gap"": 0.037526197731494904, ""mean_train_loss"": 4.728417493402958, ""mean_val_loss"": 4.765943691134453, ""n"": 2, ""std_val_loss"": 0.017364290852245785}, {""dropout"": 0.06, ""mean_generalization_gap"": 0.041536543518304825, ""mean_train_loss"": 4.739520192146301, ""mean_val_loss"": 4.781056735664606, ""n"": 2, ""std_val_loss"": 0.023307064980910674}, {""dropout"": 0.08, ""mean_generalization_gap"": 0.03945805877447128, ""mean_train_loss"": 4.756850086152554, ""mean_val_loss"": 4.796308144927025, ""n"": 2, ""std_val_loss"": 0.020058454386319792}, {""dropout"": 0.1, ""mean_generalization_gap"": 0.03160826489329338, ""mean_train_loss"": 4.771451950073242, ""mean_val_loss"": 4.803060214966536, ""n"": 2, ""std_val_loss"": 0.028490136896810635}]"
+screen_static,2000000,L12_H8_D320,12,8,320,17367040,2,0.0,4.769962530583143,0.024905420990413565,0.0,0.02,0.01,4.769962530583143,0.0,0.02,4.776707325130701,-0.006744794547557831,0.1,4.8234143033623695,0.0534517727792263,False,False,"[{""dropout"": 0.0, ""mean_generalization_gap"": 0.1482292003929615, ""mean_train_loss"": 4.621733330190182, ""mean_val_loss"": 4.769962530583143, ""n"": 2, ""std_val_loss"": 0.024905420990413565}, {""dropout"": 0.02, ""mean_generalization_gap"": 0.13987046852707863, ""mean_train_loss"": 4.636836856603622, ""mean_val_loss"": 4.776707325130701, ""n"": 2, ""std_val_loss"": 0.037399464373119376}, {""dropout"": 0.04, ""mean_generalization_gap"": 0.1428864300251007, ""mean_train_loss"": 4.646803148090839, ""mean_val_loss"": 4.78968957811594, ""n"": 2, ""std_val_loss"": 0.0230620284720244}, {""dropout"": 0.06, ""mean_generalization_gap"": 0.13828865811228752, ""mean_train_loss"": 4.664006285369396, ""mean_val_loss"": 4.802294943481684, ""n"": 2, ""std_val_loss"": 0.0302215821171988}, {""dropout"": 0.08, ""mean_generalization_gap"": 0.12224162369966507, ""mean_train_loss"": 4.685344219207764, ""mean_val_loss"": 4.807585842907429, ""n"": 2, ""std_val_loss"": 0.02933789441282695}, {""dropout"": 0.1, ""mean_generalization_gap"": 0.11924178153276443, ""mean_train_loss"": 4.704172521829605, ""mean_val_loss"": 4.8234143033623695, ""n"": 2, ""std_val_loss"": 0.022559722331471566}]"
+screen_static,1000000,L12_H8_D320,12,8,320,17367040,2,0.0,4.819691203534603,0.024108977262459263,0.0,0.04,0.01,4.819691203534603,0.0,0.02,4.820131976157427,-0.0004407726228237152,0.1,4.864129442721605,0.04443823918700218,False,False,"[{""dropout"": 0.0, ""mean_generalization_gap"": 0.37102071195840836, ""mean_train_loss"": 4.448670491576195, ""mean_val_loss"": 4.819691203534603, ""n"": 2, ""std_val_loss"": 0.024108977262459263}, {""dropout"": 0.02, ""mean_generalization_gap"": 0.36884911730885506, ""mean_train_loss"": 4.451282858848572, ""mean_val_loss"": 4.820131976157427, ""n"": 2, ""std_val_loss"": 0.016412630817937993}, {""dropout"": 0.04, ""mean_generalization_gap"": 0.35678089037537575, ""mean_train_loss"": 4.47072771191597, ""mean_val_loss"": 4.827508602291346, ""n"": 2, ""std_val_loss"": 0.022596026573107636}, {""dropout"": 0.06, ""mean_generalization_gap"": 0.3343743495643139, ""mean_train_loss"": 4.502691887319088, ""mean_val_loss"": 4.837066236883402, ""n"": 2, ""std_val_loss"": 0.022903940911616103}, {""dropout"": 0.08, ""mean_generalization_gap"": 0.3378102630376816, ""mean_train_loss"": 4.511857099831104, ""mean_val_loss"": 4.849667362868786, ""n"": 2, ""std_val_loss"": 0.029055995216561836}, {""dropout"": 0.1, ""mean_generalization_gap"": 0.32185063138604164, ""mean_train_loss"": 4.542278811335564, ""mean_val_loss"": 4.864129442721605, ""n"": 2, ""std_val_loss"": 0.023052961631238496}]"
+screen_static,4000000,L8_H8_D256,8,8,256,8388608,2,0.0,4.931619014590979,0.024494004528674472,0.0,0.02,0.01,4.931619014590979,0.0,0.02,4.939271181821823,-0.007652167230844498,0.1,4.991779696196318,0.06016068160533905,False,False,"[{""dropout"": 0.0, ""mean_generalization_gap"": 0.022088032215833664, ""mean_train_loss"": 4.909530982375145, ""mean_val_loss"": 4.931619014590979, ""n"": 2, ""std_val_loss"": 0.024494004528674472}, {""dropout"": 0.02, ""mean_generalization_gap"": 0.014929912984371185, ""mean_train_loss"": 4.924341268837452, ""mean_val_loss"": 4.939271181821823, ""n"": 2, ""std_val_loss"": 0.02502982268214953}, {""dropout"": 0.04, ""mean_generalization_gap"": 0.011224817484617233, ""mean_train_loss"": 4.935941606760025, ""mean_val_loss"": 4.947166424244642, ""n"": 2, ""std_val_loss"": 0.02536144989130144}, {""dropout"": 0.06, ""mean_generalization_gap"": 0.01743224635720253, ""mean_train_loss"": 4.955327145755291, ""mean_val_loss"": 4.9727593921124935, ""n"": 2, ""std_val_loss"": 0.024765746334448464}, {""dropout"": 0.08, ""mean_generalization_gap"": 0.01604192703962326, ""mean_train_loss"": 4.968859277665615, ""mean_val_loss"": 4.984901204705238, ""n"": 2, ""std_val_loss"": 0.017840392189736973}, {""dropout"": 0.1, ""mean_generalization_gap"": 0.006776083260774612, ""mean_train_loss"": 4.985003612935543, ""mean_val_loss"": 4.991779696196318, ""n"": 2, ""std_val_loss"": 0.020482815198907794}]"
+screen_static,2000000,L8_H8_D256,8,8,256,8388608,2,0.0,4.946536503732204,0.02745871395169597,0.0,0.0,0.01,4.946536503732204,0.0,0.02,4.957204628735781,-0.010668125003576279,0.1,5.004935186356306,0.05839868262410164,False,False,"[{""dropout"": 0.0, ""mean_generalization_gap"": 0.10198886692523956, ""mean_train_loss"": 4.844547636806965, ""mean_val_loss"": 4.946536503732204, ""n"": 2, ""std_val_loss"": 0.02745871395169597}, {""dropout"": 0.02, ""mean_generalization_gap"": 0.09024378284811974, ""mean_train_loss"": 4.866960845887661, ""mean_val_loss"": 4.957204628735781, ""n"": 2, ""std_val_loss"": 0.02519712986566959}, {""dropout"": 0.04, ""mean_generalization_gap"": 0.08774802088737488, ""mean_train_loss"": 4.880623258650303, ""mean_val_loss"": 4.968371279537678, ""n"": 2, ""std_val_loss"": 0.01927565043221483}, {""dropout"": 0.06, ""mean_generalization_gap"": 0.0858234278857708, ""mean_train_loss"": 4.891251288354397, ""mean_val_loss"": 4.977074716240168, ""n"": 2, ""std_val_loss"": 0.02597958610827434}, {""dropout"": 0.08, ""mean_generalization_gap"": 0.07851320505142212, ""mean_train_loss"": 4.917440451681614, ""mean_val_loss"": 4.995953656733036, ""n"": 2, ""std_val_loss"": 0.02737851903569187}, {""dropout"": 0.1, ""mean_generalization_gap"": 0.07617789134383202, ""mean_train_loss"": 4.928757295012474, ""mean_val_loss"": 5.004935186356306, ""n"": 2, ""std_val_loss"": 0.020581807609347757}]"
+screen_static,1000000,L8_H8_D256,8,8,256,8388608,2,0.0,4.971932724118233,0.018370399346473592,0.0,0.0,0.01,4.971932724118233,0.0,0.02,4.983673017472029,-0.011740293353796005,0.1,5.031736072152853,0.059803348034620285,False,False,"[{""dropout"": 0.0, ""mean_generalization_gap"": 0.27604202181100845, ""mean_train_loss"": 4.695890702307224, ""mean_val_loss"": 4.971932724118233, ""n"": 2, ""std_val_loss"": 0.018370399346473592}, {""dropout"": 0.02, ""mean_generalization_gap"": 0.26804643496870995, ""mean_train_loss"": 4.715626582503319, ""mean_val_loss"": 4.983673017472029, ""n"": 2, ""std_val_loss"": 0.0261924066198301}, {""dropout"": 0.04, ""mean_generalization_gap"": 0.25848760083317757, ""mean_train_loss"": 4.736352913081646, ""mean_val_loss"": 4.9948405139148235, ""n"": 2, ""std_val_loss"": 0.02342923815803163}, {""dropout"": 0.06, ""mean_generalization_gap"": 0.2509215995669365, ""mean_train_loss"": 4.750876776874065, ""mean_val_loss"": 5.001798376441002, ""n"": 2, ""std_val_loss"": 0.020779755551735234}, {""dropout"": 0.08, ""mean_generalization_gap"": 0.2433265782892704, ""mean_train_loss"": 4.767140358686447, ""mean_val_loss"": 5.0104669369757175, ""n"": 2, ""std_val_loss"": 0.01812731212933095}, {""dropout"": 0.1, ""mean_generalization_gap"": 0.23772460594773293, ""mean_train_loss"": 4.79401146620512, ""mean_val_loss"": 5.031736072152853, ""n"": 2, ""std_val_loss"": 0.03717824610199782}]"
+screen_static,500000,L8_H8_D256,8,8,256,8388608,2,0.02,5.056944504380226,0.029059525015124623,0.0,0.04,0.01,5.057585891336203,0.0006413869559764862,0.02,5.056944504380226,0.0006413869559764862,0.1,5.0904296934604645,0.03348518908023834,False,False,"[{""dropout"": 0.0, ""mean_generalization_gap"": 0.5936610512435436, ""mean_train_loss"": 4.463924840092659, ""mean_val_loss"": 5.057585891336203, ""n"": 2, ""std_val_loss"": 0.030907543150137856}, {""dropout"": 0.02, ""mean_generalization_gap"": 0.5593231469392776, ""mean_train_loss"": 4.4976213574409485, ""mean_val_loss"": 5.056944504380226, ""n"": 2, ""std_val_loss"": 0.029059525015124623}, {""dropout"": 0.04, ""mean_generalization_gap"": 0.5450111068785191, ""mean_train_loss"": 4.520545817911625, ""mean_val_loss"": 5.065556924790144, ""n"": 2, ""std_val_loss"": 0.03663996762624094}, {""dropout"": 0.06, ""mean_generalization_gap"": 0.5247969217598438, ""mean_train_loss"": 4.5480824783444405, ""mean_val_loss"": 5.072879400104284, ""n"": 2, ""std_val_loss"": 0.02535278871393245}, {""dropout"": 0.08, ""mean_generalization_gap"": 0.5122322402894497, ""mean_train_loss"": 4.576782673597336, ""mean_val_loss"": 5.0890149138867855, ""n"": 2, ""std_val_loss"": 0.03818276355269434}, {""dropout"": 0.1, ""mean_generalization_gap"": 0.4959244281053543, ""mean_train_loss"": 4.59450526535511, ""mean_val_loss"": 5.0904296934604645, ""n"": 2, ""std_val_loss"": 0.028588154661378783}]"

runs/wikitext103_interaction_static_screen/screen_static/20260603-153314/model_selection.json

@@ -0,0 +1,407 @@
+[
+  {
+    "run_mode": "screen_static",
+    "token_limit": 250000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "n": 2,
+    "best_dropout": 0.26,
+    "best_val_loss": 5.1221378818154335,
+    "best_val_std": 0.016875071308155583,
+    "plateau_start_dropout": 0.2,
+    "plateau_end_dropout": 0.26,
+    "plateau_delta": 0.01,
+    "zero_dropout_val_loss": 5.459531031548977,
+    "zero_minus_best": 0.3373931497335434,
+    "best_nonzero_dropout": 0.26,
+    "best_nonzero_val_loss": 5.1221378818154335,
+    "zero_minus_best_nonzero": 0.3373931497335434,
+    "max_dropout": 0.5,
+    "max_dropout_val_loss": 5.297821056097746,
+    "max_dropout_minus_best": 0.1756831742823124,
+    "has_nonzero_optimum": true,
+    "meets_target_dropout": true,
+    "curve_json": "[{\"dropout\": 0.0, \"mean_generalization_gap\": 2.47505421936512, \"mean_train_loss\": 2.984476812183857, \"mean_val_loss\": 5.459531031548977, \"n\": 2, \"std_val_loss\": 0.052592860620622094}, {\"dropout\": 0.02, \"mean_generalization_gap\": 2.329351920634508, \"mean_train_loss\": 3.066030155867338, \"mean_val_loss\": 5.395382076501846, \"n\": 2, \"std_val_loss\": 0.028443274869622583}, {\"dropout\": 0.04, \"mean_generalization_gap\": 2.1614660173654556, \"mean_train_loss\": 3.1688508801162243, \"mean_val_loss\": 5.33031689748168, \"n\": 2, \"std_val_loss\": 0.009347870703420656}, {\"dropout\": 0.06, \"mean_generalization_gap\": 2.012665204703808, \"mean_train_loss\": 3.2494926266372204, \"mean_val_loss\": 5.262157831341028, \"n\": 2, \"std_val_loss\": 0.02011042671888979}, {\"dropout\": 0.08, \"mean_generalization_gap\": 1.8873155526816845, \"mean_train_loss\": 3.3453744649887085, \"mean_val_loss\": 5.232690017670393, \"n\": 2, \"std_val_loss\": 0.019404035001135078}, {\"dropout\": 0.1, \"mean_generalization_gap\": 1.7902754880487919, \"mean_train_loss\": 3.4033044539391994, \"mean_val_loss\": 5.193579941987991, \"n\": 2, \"std_val_loss\": 0.03371155717649944}, {\"dropout\": 0.14, \"mean_generalization_gap\": 1.5821974352002144, \"mean_train_loss\": 3.556764382869005, \"mean_val_loss\": 5.13896181806922, \"n\": 2, \"std_val_loss\": 0.031193730788238956}, {\"dropout\": 0.18, \"mean_generalization_gap\": 1.4782251790165901, \"mean_train_loss\": 3.6555825024843216, \"mean_val_loss\": 5.133807681500912, \"n\": 2, \"std_val_loss\": 0.03364965399274907}, {\"dropout\": 0.2, \"mean_generalization_gap\": 1.410427700728178, \"mean_train_loss\": 3.7214544899761677, \"mean_val_loss\": 5.131882190704346, \"n\": 2, \"std_val_loss\": 0.026161212683575973}, {\"dropout\": 0.26, \"mean_generalization_gap\": 1.2708128467202187, \"mean_train_loss\": 3.851325035095215, \"mean_val_loss\": 5.1221378818154335, \"n\": 2, \"std_val_loss\": 0.016875071308155583}, {\"dropout\": 0.3, \"mean_generalization_gap\": 1.1743715591728687, \"mean_train_loss\": 3.9617567881941795, \"mean_val_loss\": 5.136128347367048, \"n\": 2, \"std_val_loss\": 0.02499181149314877}, {\"dropout\": 0.4, \"mean_generalization_gap\": 1.0374067649245262, \"mean_train_loss\": 4.154601521790028, \"mean_val_loss\": 5.192008286714554, \"n\": 2, \"std_val_loss\": 0.029119299783025198}, {\"dropout\": 0.5, \"mean_generalization_gap\": 0.9252229072153568, \"mean_train_loss\": 4.372598148882389, \"mean_val_loss\": 5.297821056097746, \"n\": 2, \"std_val_loss\": 0.011355662272782625}]"
+  },
+  {
+    "run_mode": "screen_static",
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "n": 2,
+    "best_dropout": 0.18,
+    "best_val_loss": 5.154339708387852,
+    "best_val_std": 0.01651832984564725,
+    "plateau_start_dropout": 0.1,
+    "plateau_end_dropout": 0.18,
+    "plateau_delta": 0.01,
+    "zero_dropout_val_loss": 5.325984805822372,
+    "zero_minus_best": 0.17164509743452072,
+    "best_nonzero_dropout": 0.18,
+    "best_nonzero_val_loss": 5.154339708387852,
+    "zero_minus_best_nonzero": 0.17164509743452072,
+    "max_dropout": 0.4,
+    "max_dropout_val_loss": 5.285179540514946,
+    "max_dropout_minus_best": 0.13083983212709427,
+    "has_nonzero_optimum": true,
+    "meets_target_dropout": true,
+    "curve_json": "[{\"dropout\": 0.0, \"mean_generalization_gap\": 1.81871847063303, \"mean_train_loss\": 3.5072663351893425, \"mean_val_loss\": 5.325984805822372, \"n\": 2, \"std_val_loss\": 0.042196002272969865}, {\"dropout\": 0.02, \"mean_generalization_gap\": 1.666674442589283, \"mean_train_loss\": 3.5965475998818874, \"mean_val_loss\": 5.26322204247117, \"n\": 2, \"std_val_loss\": 0.03309956913105307}, {\"dropout\": 0.04, \"mean_generalization_gap\": 1.5586755871772766, \"mean_train_loss\": 3.6675546914339066, \"mean_val_loss\": 5.226230278611183, \"n\": 2, \"std_val_loss\": 0.010485914638555752}, {\"dropout\": 0.06, \"mean_generalization_gap\": 1.4714824222028255, \"mean_train_loss\": 3.729271735996008, \"mean_val_loss\": 5.2007541581988335, \"n\": 2, \"std_val_loss\": 0.019410783765252886}, {\"dropout\": 0.08, \"mean_generalization_gap\": 1.4251716993749142, \"mean_train_loss\": 3.7556542232632637, \"mean_val_loss\": 5.180825922638178, \"n\": 2, \"std_val_loss\": 0.0188014509712627}, {\"dropout\": 0.1, \"mean_generalization_gap\": 1.3444151766598225, \"mean_train_loss\": 3.818895984441042, \"mean_val_loss\": 5.163311161100864, \"n\": 2, \"std_val_loss\": 0.029076731466029196}, {\"dropout\": 0.14, \"mean_generalization_gap\": 1.2507423609495163, \"mean_train_loss\": 3.9072264805436134, \"mean_val_loss\": 5.15796884149313, \"n\": 2, \"std_val_loss\": 0.03002937668292093}, {\"dropout\": 0.18, \"mean_generalization_gap\": 1.151239987462759, \"mean_train_loss\": 4.003099720925093, \"mean_val_loss\": 5.154339708387852, \"n\": 2, \"std_val_loss\": 0.01651832984564725}, {\"dropout\": 0.2, \"mean_generalization_gap\": 1.1242803186178207, \"mean_train_loss\": 4.040268834680319, \"mean_val_loss\": 5.16454915329814, \"n\": 2, \"std_val_loss\": 0.016607855020240453}, {\"dropout\": 0.26, \"mean_generalization_gap\": 1.0330531522631645, \"mean_train_loss\": 4.154937222599983, \"mean_val_loss\": 5.187990374863148, \"n\": 2, \"std_val_loss\": 0.015247644509892305}, {\"dropout\": 0.3, \"mean_generalization_gap\": 0.9783092141151428, \"mean_train_loss\": 4.223974447697401, \"mean_val_loss\": 5.202283661812544, \"n\": 2, \"std_val_loss\": 0.021969144885068726}, {\"dropout\": 0.4, \"mean_generalization_gap\": 0.8897631838917732, \"mean_train_loss\": 4.395416356623173, \"mean_val_loss\": 5.285179540514946, \"n\": 2, \"std_val_loss\": 0.021146865138977247}]"
+  },
+  {
+    "run_mode": "screen_static",
+    "token_limit": 250000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "n": 2,
+    "best_dropout": 0.1,
+    "best_val_loss": 5.226473439484835,
+    "best_val_std": 0.02592699737804487,
+    "plateau_start_dropout": 0.04,
+    "plateau_end_dropout": 0.1,
+    "plateau_delta": 0.01,
+    "zero_dropout_val_loss": 5.28285102173686,
+    "zero_minus_best": 0.056377582252025604,
+    "best_nonzero_dropout": 0.1,
+    "best_nonzero_val_loss": 5.226473439484835,
+    "zero_minus_best_nonzero": 0.056377582252025604,
+    "max_dropout": 0.26,
+    "max_dropout_val_loss": 5.301199935376644,
+    "max_dropout_minus_best": 0.07472649589180946,
+    "has_nonzero_optimum": true,
+    "meets_target_dropout": true,
+    "curve_json": "[{\"dropout\": 0.0, \"mean_generalization_gap\": 1.2473031356930733, \"mean_train_loss\": 4.035547886043787, \"mean_val_loss\": 5.28285102173686, \"n\": 2, \"std_val_loss\": 0.0324948988321794}, {\"dropout\": 0.02, \"mean_generalization_gap\": 1.1577380523085594, \"mean_train_loss\": 4.089681029319763, \"mean_val_loss\": 5.247419081628323, \"n\": 2, \"std_val_loss\": 0.03137372417316079}, {\"dropout\": 0.04, \"mean_generalization_gap\": 1.1100559122860432, \"mean_train_loss\": 4.125994935631752, \"mean_val_loss\": 5.236050847917795, \"n\": 2, \"std_val_loss\": 0.01603267697861228}, {\"dropout\": 0.06, \"mean_generalization_gap\": 1.0622206516563892, \"mean_train_loss\": 4.174422111362219, \"mean_val_loss\": 5.236642763018608, \"n\": 2, \"std_val_loss\": 0.012643158932737352}, {\"dropout\": 0.08, \"mean_generalization_gap\": 1.016893107444048, \"mean_train_loss\": 4.211392514407635, \"mean_val_loss\": 5.228285621851683, \"n\": 2, \"std_val_loss\": 0.009615418897767812}, {\"dropout\": 0.1, \"mean_generalization_gap\": 0.990381870418787, \"mean_train_loss\": 4.236091569066048, \"mean_val_loss\": 5.226473439484835, \"n\": 2, \"std_val_loss\": 0.02592699737804487}, {\"dropout\": 0.14, \"mean_generalization_gap\": 0.9385448731482029, \"mean_train_loss\": 4.300983510911465, \"mean_val_loss\": 5.239528384059668, \"n\": 2, \"std_val_loss\": 0.01265994918351288}, {\"dropout\": 0.18, \"mean_generalization_gap\": 0.8983461037278175, \"mean_train_loss\": 4.3531088307499886, \"mean_val_loss\": 5.251454934477806, \"n\": 2, \"std_val_loss\": 0.008830207304941664}, {\"dropout\": 0.2, \"mean_generalization_gap\": 0.8866863176226616, \"mean_train_loss\": 4.38435186445713, \"mean_val_loss\": 5.271038182079792, \"n\": 2, \"std_val_loss\": 0.01905936334236905}, {\"dropout\": 0.26, \"mean_generalization_gap\": 0.8267565593123436, \"mean_train_loss\": 4.4744433760643005, \"mean_val_loss\": 5.301199935376644, \"n\": 2, \"std_val_loss\": 0.022748898461012714}]"
+  },
+  {
+    "run_mode": "screen_static",
+    "token_limit": 1000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "n": 2,
+    "best_dropout": 0.02,
+    "best_val_loss": 4.7049963511526585,
+    "best_val_std": 0.026859896796409256,
+    "plateau_start_dropout": 0.02,
+    "plateau_end_dropout": 0.04,
+    "plateau_delta": 0.01,
+    "zero_dropout_val_loss": 4.7177883349359035,
+    "zero_minus_best": 0.012791983783245087,
+    "best_nonzero_dropout": 0.02,
+    "best_nonzero_val_loss": 4.7049963511526585,
+    "zero_minus_best_nonzero": 0.012791983783245087,
+    "max_dropout": 0.1,
+    "max_dropout_val_loss": 4.745819546282291,
+    "max_dropout_minus_best": 0.04082319512963295,
+    "has_nonzero_optimum": true,
+    "meets_target_dropout": false,
+    "curve_json": "[{\"dropout\": 0.0, \"mean_generalization_gap\": 0.48521145433187485, \"mean_train_loss\": 4.232576880604029, \"mean_val_loss\": 4.7177883349359035, \"n\": 2, \"std_val_loss\": 0.009086444338819707}, {\"dropout\": 0.02, \"mean_generalization_gap\": 0.4790714867413044, \"mean_train_loss\": 4.225924864411354, \"mean_val_loss\": 4.7049963511526585, \"n\": 2, \"std_val_loss\": 0.026859896796409256}, {\"dropout\": 0.04, \"mean_generalization_gap\": 0.4432833231985569, \"mean_train_loss\": 4.269359432160854, \"mean_val_loss\": 4.712642755359411, \"n\": 2, \"std_val_loss\": 0.026603285709250676}, {\"dropout\": 0.06, \"mean_generalization_gap\": 0.4268508180975914, \"mean_train_loss\": 4.291698649525642, \"mean_val_loss\": 4.718549467623234, \"n\": 2, \"std_val_loss\": 0.030084041145439562}, {\"dropout\": 0.08, \"mean_generalization_gap\": 0.41410549730062485, \"mean_train_loss\": 4.3292553797364235, \"mean_val_loss\": 4.743360877037048, \"n\": 2, \"std_val_loss\": 0.03075722114656769}, {\"dropout\": 0.1, \"mean_generalization_gap\": 0.3938281238079071, \"mean_train_loss\": 4.351991422474384, \"mean_val_loss\": 4.745819546282291, \"n\": 2, \"std_val_loss\": 0.021169972148673347}]"
+  },
+  {
+    "run_mode": "screen_static",
+    "token_limit": 500000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "n": 2,
+    "best_dropout": 0.08,
+    "best_val_loss": 4.868544135242701,
+    "best_val_std": 0.03936356019728768,
+    "plateau_start_dropout": 0.06,
+    "plateau_end_dropout": 0.1,
+    "plateau_delta": 0.01,
+    "zero_dropout_val_loss": 4.916963316500187,
+    "zero_minus_best": 0.04841918125748634,
+    "best_nonzero_dropout": 0.08,
+    "best_nonzero_val_loss": 4.868544135242701,
+    "zero_minus_best_nonzero": 0.04841918125748634,
+    "max_dropout": 0.26,
+    "max_dropout_val_loss": 4.926884576678276,
+    "max_dropout_minus_best": 0.058340441435575485,
+    "has_nonzero_optimum": true,
+    "meets_target_dropout": false,
+    "curve_json": "[{\"dropout\": 0.0, \"mean_generalization_gap\": 1.0916679799556732, \"mean_train_loss\": 3.8252953365445137, \"mean_val_loss\": 4.916963316500187, \"n\": 2, \"std_val_loss\": 0.03401019920833551}, {\"dropout\": 0.02, \"mean_generalization_gap\": 1.024292692542076, \"mean_train_loss\": 3.8655518889427185, \"mean_val_loss\": 4.889844581484795, \"n\": 2, \"std_val_loss\": 0.03132540281134305}, {\"dropout\": 0.04, \"mean_generalization_gap\": 0.9672790542244911, \"mean_train_loss\": 3.914637830108404, \"mean_val_loss\": 4.881916884332895, \"n\": 2, \"std_val_loss\": 0.04055301801925}, {\"dropout\": 0.06, \"mean_generalization_gap\": 0.9132770709693432, \"mean_train_loss\": 3.9587855264544487, \"mean_val_loss\": 4.872062597423792, \"n\": 2, \"std_val_loss\": 0.029954139289972922}, {\"dropout\": 0.08, \"mean_generalization_gap\": 0.8578500226140022, \"mean_train_loss\": 4.010694112628698, \"mean_val_loss\": 4.868544135242701, \"n\": 2, \"std_val_loss\": 0.03936356019728768}, {\"dropout\": 0.1, \"mean_generalization_gap\": 0.8172299675643444, \"mean_train_loss\": 4.053545691072941, \"mean_val_loss\": 4.870775658637285, \"n\": 2, \"std_val_loss\": 0.02582369545234467}, {\"dropout\": 0.14, \"mean_generalization_gap\": 0.7596709467470646, \"mean_train_loss\": 4.120105978101492, \"mean_val_loss\": 4.8797769248485565, \"n\": 2, \"std_val_loss\": 0.03186888642289103}, {\"dropout\": 0.18, \"mean_generalization_gap\": 0.7206361852586269, \"mean_train_loss\": 4.1665811985731125, \"mean_val_loss\": 4.887217383831739, \"n\": 2, \"std_val_loss\": 0.014720445386937724}, {\"dropout\": 0.2, \"mean_generalization_gap\": 0.6973064877092838, \"mean_train_loss\": 4.197941534221172, \"mean_val_loss\": 4.895248021930456, \"n\": 2, \"std_val_loss\": 0.015913675351841773}, {\"dropout\": 0.26, \"mean_generalization_gap\": 0.636580154299736, \"mean_train_loss\": 4.29030442237854, \"mean_val_loss\": 4.926884576678276, \"n\": 2, \"std_val_loss\": 0.028275551485973483}]"
+  },
+  {
+    "run_mode": "screen_static",
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "n": 2,
+    "best_dropout": 0.02,
+    "best_val_loss": 4.939401715993881,
+    "best_val_std": 0.02775412121290883,
+    "plateau_start_dropout": 0.02,
+    "plateau_end_dropout": 0.1,
+    "plateau_delta": 0.01,
+    "zero_dropout_val_loss": 4.956410575658083,
+    "zero_minus_best": 0.017008859664201736,
+    "best_nonzero_dropout": 0.02,
+    "best_nonzero_val_loss": 4.939401715993881,
+    "zero_minus_best_nonzero": 0.017008859664201736,
+    "max_dropout": 0.18,
+    "max_dropout_val_loss": 4.9909077398478985,
+    "max_dropout_minus_best": 0.05150602385401726,
+    "has_nonzero_optimum": true,
+    "meets_target_dropout": false,
+    "curve_json": "[{\"dropout\": 0.0, \"mean_generalization_gap\": 0.8076566644012928, \"mean_train_loss\": 4.14875391125679, \"mean_val_loss\": 4.956410575658083, \"n\": 2, \"std_val_loss\": 0.021681071175496766}, {\"dropout\": 0.02, \"mean_generalization_gap\": 0.7702712826430798, \"mean_train_loss\": 4.1691304333508015, \"mean_val_loss\": 4.939401715993881, \"n\": 2, \"std_val_loss\": 0.02775412121290883}, {\"dropout\": 0.04, \"mean_generalization_gap\": 0.728732842952013, \"mean_train_loss\": 4.210759695619345, \"mean_val_loss\": 4.939492538571358, \"n\": 2, \"std_val_loss\": 0.030459295611156307}, {\"dropout\": 0.06, \"mean_generalization_gap\": 0.7076257020235062, \"mean_train_loss\": 4.244226388633251, \"mean_val_loss\": 4.951852090656757, \"n\": 2, \"std_val_loss\": 0.0168901282697861}, {\"dropout\": 0.08, \"mean_generalization_gap\": 0.6742749661207199, \"mean_train_loss\": 4.271009549498558, \"mean_val_loss\": 4.945284515619278, \"n\": 2, \"std_val_loss\": 0.027335982328832248}, {\"dropout\": 0.1, \"mean_generalization_gap\": 0.6505224779248238, \"mean_train_loss\": 4.292945623397827, \"mean_val_loss\": 4.943468101322651, \"n\": 2, \"std_val_loss\": 0.02226113298319614}, {\"dropout\": 0.14, \"mean_generalization_gap\": 0.6210011504590511, \"mean_train_loss\": 4.344617947936058, \"mean_val_loss\": 4.965619098395109, \"n\": 2, \"std_val_loss\": 0.039715876240702365}, {\"dropout\": 0.18, \"mean_generalization_gap\": 0.5889932848513126, \"mean_train_loss\": 4.401914454996586, \"mean_val_loss\": 4.9909077398478985, \"n\": 2, \"std_val_loss\": 0.030591188905215085}]"
+  },
+  {
+    "run_mode": "screen_static",
+    "token_limit": 4000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "n": 2,
+    "best_dropout": 0.0,
+    "best_val_loss": 4.614261016249657,
+    "best_val_std": 0.0343418000757071,
+    "plateau_start_dropout": 0.0,
+    "plateau_end_dropout": 0.0,
+    "plateau_delta": 0.01,
+    "zero_dropout_val_loss": 4.614261016249657,
+    "zero_minus_best": 0.0,
+    "best_nonzero_dropout": 0.02,
+    "best_nonzero_val_loss": 4.634637780487537,
+    "zero_minus_best_nonzero": -0.020376764237880707,
+    "max_dropout": 0.1,
+    "max_dropout_val_loss": 4.680121582001448,
+    "max_dropout_minus_best": 0.065860565751791,
+    "has_nonzero_optimum": false,
+    "meets_target_dropout": false,
+    "curve_json": "[{\"dropout\": 0.0, \"mean_generalization_gap\": 0.07013300061225891, \"mean_train_loss\": 4.544128015637398, \"mean_val_loss\": 4.614261016249657, \"n\": 2, \"std_val_loss\": 0.0343418000757071}, {\"dropout\": 0.02, \"mean_generalization_gap\": 0.07366831600666046, \"mean_train_loss\": 4.560969464480877, \"mean_val_loss\": 4.634637780487537, \"n\": 2, \"std_val_loss\": 0.02513911999705041}, {\"dropout\": 0.04, \"mean_generalization_gap\": 0.06866001710295677, \"mean_train_loss\": 4.567037917673588, \"mean_val_loss\": 4.635697934776545, \"n\": 2, \"std_val_loss\": 0.02091666432076681}, {\"dropout\": 0.06, \"mean_generalization_gap\": 0.06760161370038986, \"mean_train_loss\": 4.590242467820644, \"mean_val_loss\": 4.657844081521034, \"n\": 2, \"std_val_loss\": 0.00958948278086542}, {\"dropout\": 0.08, \"mean_generalization_gap\": 0.05967212840914726, \"mean_train_loss\": 4.602519556879997, \"mean_val_loss\": 4.6621916852891445, \"n\": 2, \"std_val_loss\": 0.029447955639357086}, {\"dropout\": 0.1, \"mean_generalization_gap\": 0.05522836372256279, \"mean_train_loss\": 4.624893218278885, \"mean_val_loss\": 4.680121582001448, \"n\": 2, \"std_val_loss\": 0.03926061651979982}]"
+  },
+  {
+    "run_mode": "screen_static",
+    "token_limit": 2000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "n": 2,
+    "best_dropout": 0.0,
+    "best_val_loss": 4.638469863682985,
+    "best_val_std": 0.020520631458734194,
+    "plateau_start_dropout": 0.0,
+    "plateau_end_dropout": 0.0,
+    "plateau_delta": 0.01,
+    "zero_dropout_val_loss": 4.638469863682985,
+    "zero_minus_best": 0.0,
+    "best_nonzero_dropout": 0.02,
+    "best_nonzero_val_loss": 4.649582006037235,
+    "zero_minus_best_nonzero": -0.011112142354249954,
+    "max_dropout": 0.1,
+    "max_dropout_val_loss": 4.699402842670679,
+    "max_dropout_minus_best": 0.06093297898769379,
+    "has_nonzero_optimum": false,
+    "meets_target_dropout": false,
+    "curve_json": "[{\"dropout\": 0.0, \"mean_generalization_gap\": 0.18756551668047905, \"mean_train_loss\": 4.450904347002506, \"mean_val_loss\": 4.638469863682985, \"n\": 2, \"std_val_loss\": 0.020520631458734194}, {\"dropout\": 0.02, \"mean_generalization_gap\": 0.1813756451010704, \"mean_train_loss\": 4.468206360936165, \"mean_val_loss\": 4.649582006037235, \"n\": 2, \"std_val_loss\": 0.03501839397156461}, {\"dropout\": 0.04, \"mean_generalization_gap\": 0.17023906484246254, \"mean_train_loss\": 4.490154348313808, \"mean_val_loss\": 4.660393413156271, \"n\": 2, \"std_val_loss\": 0.01666943156591487}, {\"dropout\": 0.06, \"mean_generalization_gap\": 0.17083461582660675, \"mean_train_loss\": 4.50459598004818, \"mean_val_loss\": 4.675430595874786, \"n\": 2, \"std_val_loss\": 0.03429217216158552}, {\"dropout\": 0.08, \"mean_generalization_gap\": 0.16451359167695045, \"mean_train_loss\": 4.517679311335087, \"mean_val_loss\": 4.682192903012037, \"n\": 2, \"std_val_loss\": 0.023959266388192506}, {\"dropout\": 0.1, \"mean_generalization_gap\": 0.15638693794608116, \"mean_train_loss\": 4.543015904724598, \"mean_val_loss\": 4.699402842670679, \"n\": 2, \"std_val_loss\": 0.02188920285015569}]"
+  },
+  {
+    "run_mode": "screen_static",
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "n": 2,
+    "best_dropout": 0.0,
+    "best_val_loss": 4.751548711210489,
+    "best_val_std": 0.02019173852637943,
+    "plateau_start_dropout": 0.0,
+    "plateau_end_dropout": 0.02,
+    "plateau_delta": 0.01,
+    "zero_dropout_val_loss": 4.751548711210489,
+    "zero_minus_best": 0.0,
+    "best_nonzero_dropout": 0.02,
+    "best_nonzero_val_loss": 4.754005387425423,
+    "zero_minus_best_nonzero": -0.0024566762149333954,
+    "max_dropout": 0.1,
+    "max_dropout_val_loss": 4.803060214966536,
+    "max_dropout_minus_best": 0.051511503756046295,
+    "has_nonzero_optimum": false,
+    "meets_target_dropout": false,
+    "curve_json": "[{\"dropout\": 0.0, \"mean_generalization_gap\": 0.05085295811295509, \"mean_train_loss\": 4.700695753097534, \"mean_val_loss\": 4.751548711210489, \"n\": 2, \"std_val_loss\": 0.02019173852637943}, {\"dropout\": 0.02, \"mean_generalization_gap\": 0.04213355481624603, \"mean_train_loss\": 4.711871832609177, \"mean_val_loss\": 4.754005387425423, \"n\": 2, \"std_val_loss\": 0.019647870324838793}, {\"dropout\": 0.04, \"mean_generalization_gap\": 0.037526197731494904, \"mean_train_loss\": 4.728417493402958, \"mean_val_loss\": 4.765943691134453, \"n\": 2, \"std_val_loss\": 0.017364290852245785}, {\"dropout\": 0.06, \"mean_generalization_gap\": 0.041536543518304825, \"mean_train_loss\": 4.739520192146301, \"mean_val_loss\": 4.781056735664606, \"n\": 2, \"std_val_loss\": 0.023307064980910674}, {\"dropout\": 0.08, \"mean_generalization_gap\": 0.03945805877447128, \"mean_train_loss\": 4.756850086152554, \"mean_val_loss\": 4.796308144927025, \"n\": 2, \"std_val_loss\": 0.020058454386319792}, {\"dropout\": 0.1, \"mean_generalization_gap\": 0.03160826489329338, \"mean_train_loss\": 4.771451950073242, \"mean_val_loss\": 4.803060214966536, \"n\": 2, \"std_val_loss\": 0.028490136896810635}]"
+  },
+  {
+    "run_mode": "screen_static",
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "n": 2,
+    "best_dropout": 0.0,
+    "best_val_loss": 4.769962530583143,
+    "best_val_std": 0.024905420990413565,
+    "plateau_start_dropout": 0.0,
+    "plateau_end_dropout": 0.02,
+    "plateau_delta": 0.01,
+    "zero_dropout_val_loss": 4.769962530583143,
+    "zero_minus_best": 0.0,
+    "best_nonzero_dropout": 0.02,
+    "best_nonzero_val_loss": 4.776707325130701,
+    "zero_minus_best_nonzero": -0.006744794547557831,
+    "max_dropout": 0.1,
+    "max_dropout_val_loss": 4.8234143033623695,
+    "max_dropout_minus_best": 0.0534517727792263,
+    "has_nonzero_optimum": false,
+    "meets_target_dropout": false,
+    "curve_json": "[{\"dropout\": 0.0, \"mean_generalization_gap\": 0.1482292003929615, \"mean_train_loss\": 4.621733330190182, \"mean_val_loss\": 4.769962530583143, \"n\": 2, \"std_val_loss\": 0.024905420990413565}, {\"dropout\": 0.02, \"mean_generalization_gap\": 0.13987046852707863, \"mean_train_loss\": 4.636836856603622, \"mean_val_loss\": 4.776707325130701, \"n\": 2, \"std_val_loss\": 0.037399464373119376}, {\"dropout\": 0.04, \"mean_generalization_gap\": 0.1428864300251007, \"mean_train_loss\": 4.646803148090839, \"mean_val_loss\": 4.78968957811594, \"n\": 2, \"std_val_loss\": 0.0230620284720244}, {\"dropout\": 0.06, \"mean_generalization_gap\": 0.13828865811228752, \"mean_train_loss\": 4.664006285369396, \"mean_val_loss\": 4.802294943481684, \"n\": 2, \"std_val_loss\": 0.0302215821171988}, {\"dropout\": 0.08, \"mean_generalization_gap\": 0.12224162369966507, \"mean_train_loss\": 4.685344219207764, \"mean_val_loss\": 4.807585842907429, \"n\": 2, \"std_val_loss\": 0.02933789441282695}, {\"dropout\": 0.1, \"mean_generalization_gap\": 0.11924178153276443, \"mean_train_loss\": 4.704172521829605, \"mean_val_loss\": 4.8234143033623695, \"n\": 2, \"std_val_loss\": 0.022559722331471566}]"
+  },
+  {
+    "run_mode": "screen_static",
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "n": 2,
+    "best_dropout": 0.0,
+    "best_val_loss": 4.819691203534603,
+    "best_val_std": 0.024108977262459263,
+    "plateau_start_dropout": 0.0,
+    "plateau_end_dropout": 0.04,
+    "plateau_delta": 0.01,
+    "zero_dropout_val_loss": 4.819691203534603,
+    "zero_minus_best": 0.0,
+    "best_nonzero_dropout": 0.02,
+    "best_nonzero_val_loss": 4.820131976157427,
+    "zero_minus_best_nonzero": -0.0004407726228237152,
+    "max_dropout": 0.1,
+    "max_dropout_val_loss": 4.864129442721605,
+    "max_dropout_minus_best": 0.04443823918700218,
+    "has_nonzero_optimum": false,
+    "meets_target_dropout": false,
+    "curve_json": "[{\"dropout\": 0.0, \"mean_generalization_gap\": 0.37102071195840836, \"mean_train_loss\": 4.448670491576195, \"mean_val_loss\": 4.819691203534603, \"n\": 2, \"std_val_loss\": 0.024108977262459263}, {\"dropout\": 0.02, \"mean_generalization_gap\": 0.36884911730885506, \"mean_train_loss\": 4.451282858848572, \"mean_val_loss\": 4.820131976157427, \"n\": 2, \"std_val_loss\": 0.016412630817937993}, {\"dropout\": 0.04, \"mean_generalization_gap\": 0.35678089037537575, \"mean_train_loss\": 4.47072771191597, \"mean_val_loss\": 4.827508602291346, \"n\": 2, \"std_val_loss\": 0.022596026573107636}, {\"dropout\": 0.06, \"mean_generalization_gap\": 0.3343743495643139, \"mean_train_loss\": 4.502691887319088, \"mean_val_loss\": 4.837066236883402, \"n\": 2, \"std_val_loss\": 0.022903940911616103}, {\"dropout\": 0.08, \"mean_generalization_gap\": 0.3378102630376816, \"mean_train_loss\": 4.511857099831104, \"mean_val_loss\": 4.849667362868786, \"n\": 2, \"std_val_loss\": 0.029055995216561836}, {\"dropout\": 0.1, \"mean_generalization_gap\": 0.32185063138604164, \"mean_train_loss\": 4.542278811335564, \"mean_val_loss\": 4.864129442721605, \"n\": 2, \"std_val_loss\": 0.023052961631238496}]"
+  },
+  {
+    "run_mode": "screen_static",
+    "token_limit": 4000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "n": 2,
+    "best_dropout": 0.0,
+    "best_val_loss": 4.931619014590979,
+    "best_val_std": 0.024494004528674472,
+    "plateau_start_dropout": 0.0,
+    "plateau_end_dropout": 0.02,
+    "plateau_delta": 0.01,
+    "zero_dropout_val_loss": 4.931619014590979,
+    "zero_minus_best": 0.0,
+    "best_nonzero_dropout": 0.02,
+    "best_nonzero_val_loss": 4.939271181821823,
+    "zero_minus_best_nonzero": -0.007652167230844498,
+    "max_dropout": 0.1,
+    "max_dropout_val_loss": 4.991779696196318,
+    "max_dropout_minus_best": 0.06016068160533905,
+    "has_nonzero_optimum": false,
+    "meets_target_dropout": false,
+    "curve_json": "[{\"dropout\": 0.0, \"mean_generalization_gap\": 0.022088032215833664, \"mean_train_loss\": 4.909530982375145, \"mean_val_loss\": 4.931619014590979, \"n\": 2, \"std_val_loss\": 0.024494004528674472}, {\"dropout\": 0.02, \"mean_generalization_gap\": 0.014929912984371185, \"mean_train_loss\": 4.924341268837452, \"mean_val_loss\": 4.939271181821823, \"n\": 2, \"std_val_loss\": 0.02502982268214953}, {\"dropout\": 0.04, \"mean_generalization_gap\": 0.011224817484617233, \"mean_train_loss\": 4.935941606760025, \"mean_val_loss\": 4.947166424244642, \"n\": 2, \"std_val_loss\": 0.02536144989130144}, {\"dropout\": 0.06, \"mean_generalization_gap\": 0.01743224635720253, \"mean_train_loss\": 4.955327145755291, \"mean_val_loss\": 4.9727593921124935, \"n\": 2, \"std_val_loss\": 0.024765746334448464}, {\"dropout\": 0.08, \"mean_generalization_gap\": 0.01604192703962326, \"mean_train_loss\": 4.968859277665615, \"mean_val_loss\": 4.984901204705238, \"n\": 2, \"std_val_loss\": 0.017840392189736973}, {\"dropout\": 0.1, \"mean_generalization_gap\": 0.006776083260774612, \"mean_train_loss\": 4.985003612935543, \"mean_val_loss\": 4.991779696196318, \"n\": 2, \"std_val_loss\": 0.020482815198907794}]"
+  },
+  {
+    "run_mode": "screen_static",
+    "token_limit": 2000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "n": 2,
+    "best_dropout": 0.0,
+    "best_val_loss": 4.946536503732204,
+    "best_val_std": 0.02745871395169597,
+    "plateau_start_dropout": 0.0,
+    "plateau_end_dropout": 0.0,
+    "plateau_delta": 0.01,
+    "zero_dropout_val_loss": 4.946536503732204,
+    "zero_minus_best": 0.0,
+    "best_nonzero_dropout": 0.02,
+    "best_nonzero_val_loss": 4.957204628735781,
+    "zero_minus_best_nonzero": -0.010668125003576279,
+    "max_dropout": 0.1,
+    "max_dropout_val_loss": 5.004935186356306,
+    "max_dropout_minus_best": 0.05839868262410164,
+    "has_nonzero_optimum": false,
+    "meets_target_dropout": false,
+    "curve_json": "[{\"dropout\": 0.0, \"mean_generalization_gap\": 0.10198886692523956, \"mean_train_loss\": 4.844547636806965, \"mean_val_loss\": 4.946536503732204, \"n\": 2, \"std_val_loss\": 0.02745871395169597}, {\"dropout\": 0.02, \"mean_generalization_gap\": 0.09024378284811974, \"mean_train_loss\": 4.866960845887661, \"mean_val_loss\": 4.957204628735781, \"n\": 2, \"std_val_loss\": 0.02519712986566959}, {\"dropout\": 0.04, \"mean_generalization_gap\": 0.08774802088737488, \"mean_train_loss\": 4.880623258650303, \"mean_val_loss\": 4.968371279537678, \"n\": 2, \"std_val_loss\": 0.01927565043221483}, {\"dropout\": 0.06, \"mean_generalization_gap\": 0.0858234278857708, \"mean_train_loss\": 4.891251288354397, \"mean_val_loss\": 4.977074716240168, \"n\": 2, \"std_val_loss\": 0.02597958610827434}, {\"dropout\": 0.08, \"mean_generalization_gap\": 0.07851320505142212, \"mean_train_loss\": 4.917440451681614, \"mean_val_loss\": 4.995953656733036, \"n\": 2, \"std_val_loss\": 0.02737851903569187}, {\"dropout\": 0.1, \"mean_generalization_gap\": 0.07617789134383202, \"mean_train_loss\": 4.928757295012474, \"mean_val_loss\": 5.004935186356306, \"n\": 2, \"std_val_loss\": 0.020581807609347757}]"
+  },
+  {
+    "run_mode": "screen_static",
+    "token_limit": 1000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "n": 2,
+    "best_dropout": 0.0,
+    "best_val_loss": 4.971932724118233,
+    "best_val_std": 0.018370399346473592,
+    "plateau_start_dropout": 0.0,
+    "plateau_end_dropout": 0.0,
+    "plateau_delta": 0.01,
+    "zero_dropout_val_loss": 4.971932724118233,
+    "zero_minus_best": 0.0,
+    "best_nonzero_dropout": 0.02,
+    "best_nonzero_val_loss": 4.983673017472029,
+    "zero_minus_best_nonzero": -0.011740293353796005,
+    "max_dropout": 0.1,
+    "max_dropout_val_loss": 5.031736072152853,
+    "max_dropout_minus_best": 0.059803348034620285,
+    "has_nonzero_optimum": false,
+    "meets_target_dropout": false,
+    "curve_json": "[{\"dropout\": 0.0, \"mean_generalization_gap\": 0.27604202181100845, \"mean_train_loss\": 4.695890702307224, \"mean_val_loss\": 4.971932724118233, \"n\": 2, \"std_val_loss\": 0.018370399346473592}, {\"dropout\": 0.02, \"mean_generalization_gap\": 0.26804643496870995, \"mean_train_loss\": 4.715626582503319, \"mean_val_loss\": 4.983673017472029, \"n\": 2, \"std_val_loss\": 0.0261924066198301}, {\"dropout\": 0.04, \"mean_generalization_gap\": 0.25848760083317757, \"mean_train_loss\": 4.736352913081646, \"mean_val_loss\": 4.9948405139148235, \"n\": 2, \"std_val_loss\": 0.02342923815803163}, {\"dropout\": 0.06, \"mean_generalization_gap\": 0.2509215995669365, \"mean_train_loss\": 4.750876776874065, \"mean_val_loss\": 5.001798376441002, \"n\": 2, \"std_val_loss\": 0.020779755551735234}, {\"dropout\": 0.08, \"mean_generalization_gap\": 0.2433265782892704, \"mean_train_loss\": 4.767140358686447, \"mean_val_loss\": 5.0104669369757175, \"n\": 2, \"std_val_loss\": 0.01812731212933095}, {\"dropout\": 0.1, \"mean_generalization_gap\": 0.23772460594773293, \"mean_train_loss\": 4.79401146620512, \"mean_val_loss\": 5.031736072152853, \"n\": 2, \"std_val_loss\": 0.03717824610199782}]"
+  },
+  {
+    "run_mode": "screen_static",
+    "token_limit": 500000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "n": 2,
+    "best_dropout": 0.02,
+    "best_val_loss": 5.056944504380226,
+    "best_val_std": 0.029059525015124623,
+    "plateau_start_dropout": 0.0,
+    "plateau_end_dropout": 0.04,
+    "plateau_delta": 0.01,
+    "zero_dropout_val_loss": 5.057585891336203,
+    "zero_minus_best": 0.0006413869559764862,
+    "best_nonzero_dropout": 0.02,
+    "best_nonzero_val_loss": 5.056944504380226,
+    "zero_minus_best_nonzero": 0.0006413869559764862,
+    "max_dropout": 0.1,
+    "max_dropout_val_loss": 5.0904296934604645,
+    "max_dropout_minus_best": 0.03348518908023834,
+    "has_nonzero_optimum": false,
+    "meets_target_dropout": false,
+    "curve_json": "[{\"dropout\": 0.0, \"mean_generalization_gap\": 0.5936610512435436, \"mean_train_loss\": 4.463924840092659, \"mean_val_loss\": 5.057585891336203, \"n\": 2, \"std_val_loss\": 0.030907543150137856}, {\"dropout\": 0.02, \"mean_generalization_gap\": 0.5593231469392776, \"mean_train_loss\": 4.4976213574409485, \"mean_val_loss\": 5.056944504380226, \"n\": 2, \"std_val_loss\": 0.029059525015124623}, {\"dropout\": 0.04, \"mean_generalization_gap\": 0.5450111068785191, \"mean_train_loss\": 4.520545817911625, \"mean_val_loss\": 5.065556924790144, \"n\": 2, \"std_val_loss\": 0.03663996762624094}, {\"dropout\": 0.06, \"mean_generalization_gap\": 0.5247969217598438, \"mean_train_loss\": 4.5480824783444405, \"mean_val_loss\": 5.072879400104284, \"n\": 2, \"std_val_loss\": 0.02535278871393245}, {\"dropout\": 0.08, \"mean_generalization_gap\": 0.5122322402894497, \"mean_train_loss\": 4.576782673597336, \"mean_val_loss\": 5.0890149138867855, \"n\": 2, \"std_val_loss\": 0.03818276355269434}, {\"dropout\": 0.1, \"mean_generalization_gap\": 0.4959244281053543, \"mean_train_loss\": 4.59450526535511, \"mean_val_loss\": 5.0904296934604645, \"n\": 2, \"std_val_loss\": 0.028588154661378783}]"
+  }
+]

runs/wikitext103_interaction_static_screen/screen_static/20260603-153314/summary.csv

@@ -0,0 +1,114 @@
+run_mode,condition,condition_kind,stage,token_limit,model_name,n_layer,n_head,n_embd,parameters,dropout_initial,dropout_final,dropout_schedule,n,mean_train_eval_loss,std_train_eval_loss,mean_val_eval_loss,std_val_eval_loss,mean_generalization_gap,std_generalization_gap
+screen_static,static_dropout_0,static,,250000,L12_H8_D320,12,8,320,17367040,0.0,0.0,constant,2,3.5072663351893425,0.056970032109433126,5.325984805822372,0.042196002272969865,1.81871847063303,0.09916603438240298
+screen_static,static_dropout_0.02,static,,250000,L12_H8_D320,12,8,320,17367040,0.02,0.02,constant,2,3.5965475998818874,0.00036475463208146856,5.26322204247117,0.03309956913105307,1.666674442589283,0.0327348144989716
+screen_static,static_dropout_0.04,static,,250000,L12_H8_D320,12,8,320,17367040,0.04,0.04,constant,2,3.6675546914339066,0.0015323645814469575,5.226230278611183,0.010485914638555752,1.5586755871772766,0.01201827922000271
+screen_static,static_dropout_0.06,static,,250000,L12_H8_D320,12,8,320,17367040,0.06,0.06,constant,2,3.729271735996008,0.00595836846261392,5.2007541581988335,0.019410783765252886,1.4714824222028255,0.025369152227866804
+screen_static,static_dropout_0.08,static,,250000,L12_H8_D320,12,8,320,17367040,0.08,0.08,constant,2,3.7556542232632637,0.0010223139574802457,5.180825922638178,0.0188014509712627,1.4251716993749142,0.019823764928742944
+screen_static,static_dropout_0.1,static,,250000,L12_H8_D320,12,8,320,17367040,0.1,0.1,constant,2,3.818895984441042,0.02959928389893545,5.163311161100864,0.029076731466029196,1.3444151766598225,0.05867601536496465
+screen_static,static_dropout_0.14,static,,250000,L12_H8_D320,12,8,320,17367040,0.14,0.14,constant,2,3.9072264805436134,0.016073122148114547,5.15796884149313,0.03002937668292093,1.2507423609495163,0.04610249883103548
+screen_static,static_dropout_0.18,static,,250000,L12_H8_D320,12,8,320,17367040,0.18,0.18,constant,2,4.003099720925093,0.011945212389753011,5.154339708387852,0.01651832984564725,1.151239987462759,0.02846354223540026
+screen_static,static_dropout_0.2,static,,250000,L12_H8_D320,12,8,320,17367040,0.2,0.2,constant,2,4.040268834680319,0.0021783019133688555,5.16454915329814,0.016607855020240453,1.1242803186178207,0.018786156933609306
+screen_static,static_dropout_0.26,static,,250000,L12_H8_D320,12,8,320,17367040,0.26,0.26,constant,2,4.154937222599983,0.01848742007136409,5.187990374863148,0.015247644509892305,1.0330531522631645,0.0337350645812564
+screen_static,static_dropout_0.3,static,,250000,L12_H8_D320,12,8,320,17367040,0.3,0.3,constant,2,4.223974447697401,0.007560475541634279,5.202283661812544,0.021969144885068726,0.9783092141151428,0.029529620426703006
+screen_static,static_dropout_0.4,static,,250000,L12_H8_D320,12,8,320,17367040,0.4,0.4,constant,2,4.395416356623173,0.0069791071816310704,5.285179540514946,0.021146865138977247,0.8897631838917732,0.02812597232060832
+screen_static,static_dropout_0,static,,500000,L12_H8_D320,12,8,320,17367040,0.0,0.0,constant,2,4.14875391125679,0.015905230177071402,4.956410575658083,0.021681071175496766,0.8076566644012928,0.03758630135256817
+screen_static,static_dropout_0.02,static,,500000,L12_H8_D320,12,8,320,17367040,0.02,0.02,constant,2,4.1691304333508015,0.005854845265959036,4.939401715993881,0.02775412121290883,0.7702712826430798,0.021899275946949792
+screen_static,static_dropout_0.04,static,,500000,L12_H8_D320,12,8,320,17367040,0.04,0.04,constant,2,4.210759695619345,0.021864167622140218,4.939492538571358,0.030459295611156307,0.728732842952013,0.05232346323329652
+screen_static,static_dropout_0.06,static,,500000,L12_H8_D320,12,8,320,17367040,0.06,0.06,constant,2,4.244226388633251,0.0010815829631986903,4.951852090656757,0.0168901282697861,0.7076257020235062,0.017971711232984793
+screen_static,static_dropout_0.08,static,,500000,L12_H8_D320,12,8,320,17367040,0.08,0.08,constant,2,4.271009549498558,0.00805379913509823,4.945284515619278,0.027335982328832248,0.6742749661207199,0.01928218319373402
+screen_static,static_dropout_0.1,static,,500000,L12_H8_D320,12,8,320,17367040,0.1,0.1,constant,2,4.292945623397827,0.007025542471977948,4.943468101322651,0.02226113298319614,0.6505224779248238,0.029286675455174084
+screen_static,static_dropout_0.14,static,,500000,L12_H8_D320,12,8,320,17367040,0.14,0.14,constant,2,4.344617947936058,0.012038151459075597,4.965619098395109,0.039715876240702365,0.6210011504590511,0.02767772478162677
+screen_static,static_dropout_0.18,static,,500000,L12_H8_D320,12,8,320,17367040,0.18,0.18,constant,2,4.401914454996586,0.0007282448587076103,4.9909077398478985,0.030591188905215085,0.5889932848513126,0.0313194337639227
+screen_static,static_dropout_0,static,,1000000,L12_H8_D320,12,8,320,17367040,0.0,0.0,constant,2,4.448670491576195,0.027271750531701645,4.819691203534603,0.024108977262459263,0.37102071195840836,0.05138072779416091
+screen_static,static_dropout_0.02,static,,1000000,L12_H8_D320,12,8,320,17367040,0.02,0.02,constant,2,4.451282858848572,0.04242921132249276,4.820131976157427,0.016412630817937993,0.36884911730885506,0.058841842140430756
+screen_static,static_dropout_0.04,static,,1000000,L12_H8_D320,12,8,320,17367040,0.04,0.04,constant,2,4.47072771191597,0.024538169157557826,4.827508602291346,0.022596026573107636,0.35678089037537575,0.04713419573066546
+screen_static,static_dropout_0.06,static,,1000000,L12_H8_D320,12,8,320,17367040,0.06,0.06,constant,2,4.502691887319088,0.024540898165998908,4.837066236883402,0.022903940911616103,0.3343743495643139,0.04744483907761501
+screen_static,static_dropout_0.08,static,,1000000,L12_H8_D320,12,8,320,17367040,0.08,0.08,constant,2,4.511857099831104,0.020942020918502176,4.849667362868786,0.029055995216561836,0.3378102630376816,0.04999801613506401
+screen_static,static_dropout_0.1,static,,1000000,L12_H8_D320,12,8,320,17367040,0.1,0.1,constant,2,4.542278811335564,0.030395527429359324,4.864129442721605,0.023052961631238496,0.32185063138604164,0.05344848906059782
+screen_static,static_dropout_0,static,,2000000,L12_H8_D320,12,8,320,17367040,0.0,0.0,constant,2,4.621733330190182,0.012111012823438806,4.769962530583143,0.024905420990413565,0.1482292003929615,0.03701643381385237
+screen_static,static_dropout_0.02,static,,2000000,L12_H8_D320,12,8,320,17367040,0.02,0.02,constant,2,4.636836856603622,0.004119475120303293,4.776707325130701,0.037399464373119376,0.13987046852707863,0.03327998925281608
+screen_static,static_dropout_0.04,static,,2000000,L12_H8_D320,12,8,320,17367040,0.04,0.04,constant,2,4.646803148090839,0.011796744847517323,4.78968957811594,0.0230620284720244,0.1428864300251007,0.03485877331954172
+screen_static,static_dropout_0.06,static,,2000000,L12_H8_D320,12,8,320,17367040,0.06,0.06,constant,2,4.664006285369396,0.0004938030138657622,4.802294943481684,0.0302215821171988,0.13828865811228752,0.02972777910333304
+screen_static,static_dropout_0.08,static,,2000000,L12_H8_D320,12,8,320,17367040,0.08,0.08,constant,2,4.685344219207764,0.0007423956630951429,4.807585842907429,0.02933789441282695,0.12224162369966507,0.03008029007592209
+screen_static,static_dropout_0.1,static,,2000000,L12_H8_D320,12,8,320,17367040,0.1,0.1,constant,2,4.704172521829605,0.00673639401776896,4.8234143033623695,0.022559722331471566,0.11924178153276443,0.029296116349240526
+screen_static,static_dropout_0,static,,4000000,L12_H8_D320,12,8,320,17367040,0.0,0.0,constant,2,4.700695753097534,0.007609592425217662,4.751548711210489,0.02019173852637943,0.05085295811295509,0.027801330951597095
+screen_static,static_dropout_0.02,static,,4000000,L12_H8_D320,12,8,320,17367040,0.02,0.02,constant,2,4.711871832609177,0.003394865427279748,4.754005387425423,0.019647870324838793,0.04213355481624603,0.02304273575211854
+screen_static,static_dropout_0.04,static,,4000000,L12_H8_D320,12,8,320,17367040,0.04,0.04,constant,2,4.728417493402958,0.00032852414743029124,4.765943691134453,0.017364290852245785,0.037526197731494904,0.017035766704815495
+screen_static,static_dropout_0.06,static,,4000000,L12_H8_D320,12,8,320,17367040,0.06,0.06,constant,2,4.739520192146301,0.00017573128486617266,4.781056735664606,0.023307064980910674,0.041536543518304825,0.02348279626577685
+screen_static,static_dropout_0.08,static,,4000000,L12_H8_D320,12,8,320,17367040,0.08,0.08,constant,2,4.756850086152554,0.0019432331341554079,4.796308144927025,0.020058454386319792,0.03945805877447128,0.0220016875204752
+screen_static,static_dropout_0.1,static,,4000000,L12_H8_D320,12,8,320,17367040,0.1,0.1,constant,2,4.771451950073242,0.002899787471246531,4.803060214966536,0.028490136896810635,0.03160826489329338,0.03138992436805717
+screen_static,static_dropout_0,static,,250000,L16_H8_D384,16,8,384,31457280,0.0,0.0,constant,2,2.984476812183857,0.07573507347193847,5.459531031548977,0.052592860620622094,2.47505421936512,0.12832793409256057
+screen_static,static_dropout_0.02,static,,250000,L16_H8_D384,16,8,384,31457280,0.02,0.02,constant,2,3.066030155867338,0.0044021171547734125,5.395382076501846,0.028443274869622583,2.329351920634508,0.032845392024396
+screen_static,static_dropout_0.04,static,,250000,L16_H8_D384,16,8,384,31457280,0.04,0.04,constant,2,3.1688508801162243,0.012333163593583664,5.33031689748168,0.009347870703420656,2.1614660173654556,0.02168103429700432
+screen_static,static_dropout_0.06,static,,250000,L16_H8_D384,16,8,384,31457280,0.06,0.06,constant,2,3.2494926266372204,0.008142987134903345,5.262157831341028,0.02011042671888979,2.012665204703808,0.028253413853793135
+screen_static,static_dropout_0.08,static,,250000,L16_H8_D384,16,8,384,31457280,0.08,0.08,constant,2,3.3453744649887085,0.04799909647731358,5.232690017670393,0.019404035001135078,1.8873155526816845,0.06740313147844866
+screen_static,static_dropout_0.1,static,,250000,L16_H8_D384,16,8,384,31457280,0.1,0.1,constant,2,3.4033044539391994,0.024623869505648665,5.193579941987991,0.03371155717649944,1.7902754880487919,0.05833542668214811
+screen_static,static_dropout_0.14,static,,250000,L16_H8_D384,16,8,384,31457280,0.14,0.14,constant,2,3.556764382869005,0.013851588104885643,5.13896181806922,0.031193730788238956,1.5821974352002144,0.045045318893124596
+screen_static,static_dropout_0.18,static,,250000,L16_H8_D384,16,8,384,31457280,0.18,0.18,constant,2,3.6555825024843216,0.015018934636447939,5.133807681500912,0.03364965399274907,1.4782251790165901,0.048668588629197006
+screen_static,static_dropout_0.2,static,,250000,L16_H8_D384,16,8,384,31457280,0.2,0.2,constant,2,3.7214544899761677,0.004305126719637718,5.131882190704346,0.026161212683575973,1.410427700728178,0.03046633940321369
+screen_static,static_dropout_0.26,static,,250000,L16_H8_D384,16,8,384,31457280,0.26,0.26,constant,2,3.851325035095215,0.026488904430748174,5.1221378818154335,0.016875071308155583,1.2708128467202187,0.04336397573890376
+screen_static,static_dropout_0.3,static,,250000,L16_H8_D384,16,8,384,31457280,0.3,0.3,constant,2,3.9617567881941795,0.012010208098512875,5.136128347367048,0.02499181149314877,1.1743715591728687,0.012981603394635896
+screen_static,static_dropout_0.4,static,,250000,L16_H8_D384,16,8,384,31457280,0.4,0.4,constant,2,4.154601521790028,0.01615451824930121,5.192008286714554,0.029119299783025198,1.0374067649245262,0.04527381803232641
+screen_static,static_dropout_0.5,static,,250000,L16_H8_D384,16,8,384,31457280,0.5,0.5,constant,2,4.372598148882389,0.019609295421739194,5.297821056097746,0.011355662272782625,0.9252229072153568,0.030964957694521823
+screen_static,static_dropout_0,static,,500000,L16_H8_D384,16,8,384,31457280,0.0,0.0,constant,2,3.8252953365445137,0.019563755751923175,4.916963316500187,0.03401019920833551,1.0916679799556732,0.053573954960258685
+screen_static,static_dropout_0.02,static,,500000,L16_H8_D384,16,8,384,31457280,0.02,0.02,constant,2,3.8655518889427185,0.018674899790252674,4.889844581484795,0.03132540281134305,1.024292692542076,0.012650503021090376
+screen_static,static_dropout_0.04,static,,500000,L16_H8_D384,16,8,384,31457280,0.04,0.04,constant,2,3.914637830108404,0.014714407850888537,4.881916884332895,0.04055301801925,0.9672790542244911,0.05526742587013854
+screen_static,static_dropout_0.06,static,,500000,L16_H8_D384,16,8,384,31457280,0.06,0.06,constant,2,3.9587855264544487,0.010768256376730232,4.872062597423792,0.029954139289972922,0.9132770709693432,0.01918588291324269
+screen_static,static_dropout_0.08,static,,500000,L16_H8_D384,16,8,384,31457280,0.08,0.08,constant,2,4.010694112628698,0.019106172685996462,4.868544135242701,0.03936356019728768,0.8578500226140022,0.05846973288328414
+screen_static,static_dropout_0.1,static,,500000,L16_H8_D384,16,8,384,31457280,0.1,0.1,constant,2,4.053545691072941,0.00040188073726350233,4.870775658637285,0.02582369545234467,0.8172299675643444,0.025421814715081165
+screen_static,static_dropout_0.14,static,,500000,L16_H8_D384,16,8,384,31457280,0.14,0.14,constant,2,4.120105978101492,0.0015854854156388755,4.8797769248485565,0.03186888642289103,0.7596709467470646,0.030283401007252155
+screen_static,static_dropout_0.18,static,,500000,L16_H8_D384,16,8,384,31457280,0.18,0.18,constant,2,4.1665811985731125,0.0004332274558434798,4.887217383831739,0.014720445386937724,0.7206361852586269,0.015153672842781202
+screen_static,static_dropout_0.2,static,,500000,L16_H8_D384,16,8,384,31457280,0.2,0.2,constant,2,4.197941534221172,0.0025712527871041124,4.895248021930456,0.015913675351841773,0.6973064877092838,0.013342422564737659
+screen_static,static_dropout_0.26,static,,500000,L16_H8_D384,16,8,384,31457280,0.26,0.26,constant,2,4.29030442237854,0.0044379156342273535,4.926884576678276,0.028275551485973483,0.636580154299736,0.03271346712020084
+screen_static,static_dropout_0,static,,1000000,L16_H8_D384,16,8,384,31457280,0.0,0.0,constant,2,4.232576880604029,0.052909293890885827,4.7177883349359035,0.009086444338819707,0.48521145433187485,0.061995738229705534
+screen_static,static_dropout_0.02,static,,1000000,L16_H8_D384,16,8,384,31457280,0.02,0.02,constant,2,4.225924864411354,0.013052246781096219,4.7049963511526585,0.026859896796409256,0.4790714867413044,0.03991214357750547
+screen_static,static_dropout_0.04,static,,1000000,L16_H8_D384,16,8,384,31457280,0.04,0.04,constant,2,4.269359432160854,0.019288157509510438,4.712642755359411,0.026603285709250676,0.4432833231985569,0.04589144321876112
+screen_static,static_dropout_0.06,static,,1000000,L16_H8_D384,16,8,384,31457280,0.06,0.06,constant,2,4.291698649525642,0.01393341094291348,4.718549467623234,0.030084041145439562,0.4268508180975914,0.04401745208835304
+screen_static,static_dropout_0.08,static,,1000000,L16_H8_D384,16,8,384,31457280,0.08,0.08,constant,2,4.3292553797364235,0.03327905675379278,4.743360877037048,0.03075722114656769,0.41410549730062485,0.06403627790036047
+screen_static,static_dropout_0.1,static,,1000000,L16_H8_D384,16,8,384,31457280,0.1,0.1,constant,2,4.351991422474384,0.030442700289555143,4.745819546282291,0.021169972148673347,0.3938281238079071,0.05161267243822849
+screen_static,static_dropout_0,static,,2000000,L16_H8_D384,16,8,384,31457280,0.0,0.0,constant,2,4.450904347002506,0.01087398174621981,4.638469863682985,0.020520631458734194,0.18756551668047905,0.031394613204954
+screen_static,static_dropout_0.02,static,,2000000,L16_H8_D384,16,8,384,31457280,0.02,0.02,constant,2,4.468206360936165,0.007094136467929428,4.649582006037235,0.03501839397156461,0.1813756451010704,0.02792425750363518
+screen_static,static_dropout_0.04,static,,2000000,L16_H8_D384,16,8,384,31457280,0.04,0.04,constant,2,4.490154348313808,0.011619222321589438,4.660393413156271,0.01666943156591487,0.17023906484246254,0.028288653887504307
+screen_static,static_dropout_0.06,static,,2000000,L16_H8_D384,16,8,384,31457280,0.06,0.06,constant,2,4.50459598004818,0.0040620500392072,4.675430595874786,0.03429217216158552,0.17083461582660675,0.03023012212237832
+screen_static,static_dropout_0.08,static,,2000000,L16_H8_D384,16,8,384,31457280,0.08,0.08,constant,2,4.517679311335087,0.0200507415130423,4.682192903012037,0.023959266388192506,0.16451359167695045,0.044010007901234806
+screen_static,static_dropout_0.1,static,,2000000,L16_H8_D384,16,8,384,31457280,0.1,0.1,constant,2,4.543015904724598,0.010151036853712445,4.699402842670679,0.02188920285015569,0.15638693794608116,0.032040239703868136
+screen_static,static_dropout_0,static,,4000000,L16_H8_D384,16,8,384,31457280,0.0,0.0,constant,2,4.544128015637398,0.008078707922568169,4.614261016249657,0.0343418000757071,0.07013300061225891,0.02626309215313893
+screen_static,static_dropout_0.02,static,,4000000,L16_H8_D384,16,8,384,31457280,0.02,0.02,constant,2,4.560969464480877,0.005450209178474134,4.634637780487537,0.02513911999705041,0.07366831600666046,0.03058932917552454
+screen_static,static_dropout_0.04,static,,4000000,L16_H8_D384,16,8,384,31457280,0.04,0.04,constant,2,4.567037917673588,0.004047477766334559,4.635697934776545,0.02091666432076681,0.06866001710295677,0.01686918655443225
+screen_static,static_dropout_0.06,static,,4000000,L16_H8_D384,16,8,384,31457280,0.06,0.06,constant,2,4.590242467820644,0.013435920079803014,4.657844081521034,0.00958948278086542,0.06760161370038986,0.023025402860668433
+screen_static,static_dropout_0.08,static,,4000000,L16_H8_D384,16,8,384,31457280,0.08,0.08,constant,2,4.602519556879997,0.0039952051374689225,4.6621916852891445,0.029447955639357086,0.05967212840914726,0.033443160776826006
+screen_static,static_dropout_0.1,static,,4000000,L16_H8_D384,16,8,384,31457280,0.1,0.1,constant,2,4.624893218278885,0.014938845087565112,4.680121582001448,0.03926061651979982,0.05522836372256279,0.024321771432234705
+screen_static,static_dropout_0,static,,250000,L8_H8_D256,8,8,256,8388608,0.0,0.0,constant,2,4.035547886043787,0.024195878795732664,5.28285102173686,0.0324948988321794,1.2473031356930733,0.008299020036446739
+screen_static,static_dropout_0.02,static,,250000,L8_H8_D256,8,8,256,8388608,0.02,0.02,constant,2,4.089681029319763,0.008525749008011093,5.247419081628323,0.03137372417316079,1.1577380523085594,0.03989947318117189
+screen_static,static_dropout_0.04,static,,250000,L8_H8_D256,8,8,256,8388608,0.04,0.04,constant,2,4.125994935631752,0.00561314889481725,5.236050847917795,0.01603267697861228,1.1100559122860432,0.02164582587342953
+screen_static,static_dropout_0.06,static,,250000,L8_H8_D256,8,8,256,8388608,0.06,0.06,constant,2,4.174422111362219,0.010122550852475144,5.236642763018608,0.012643158932737352,1.0622206516563892,0.0025206080802622093
+screen_static,static_dropout_0.08,static,,250000,L8_H8_D256,8,8,256,8388608,0.08,0.08,constant,2,4.211392514407635,0.0006487559024160641,5.228285621851683,0.009615418897767812,1.016893107444048,0.008966662995351747
+screen_static,static_dropout_0.1,static,,250000,L8_H8_D256,8,8,256,8388608,0.1,0.1,constant,2,4.236091569066048,0.003005154592523766,5.226473439484835,0.02592699737804487,0.990381870418787,0.028932151970568636
+screen_static,static_dropout_0.14,static,,250000,L8_H8_D256,8,8,256,8388608,0.14,0.14,constant,2,4.300983510911465,0.014899132219555723,5.239528384059668,0.01265994918351288,0.9385448731482029,0.027559081403068602
+screen_static,static_dropout_0.18,static,,250000,L8_H8_D256,8,8,256,8388608,0.18,0.18,constant,2,4.3531088307499886,0.0033408015573523986,5.251454934477806,0.008830207304941664,0.8983461037278175,0.012171008862294063
+screen_static,static_dropout_0.2,static,,250000,L8_H8_D256,8,8,256,8388608,0.2,0.2,constant,2,4.38435186445713,0.004200987125323363,5.271038182079792,0.01905936334236905,0.8866863176226616,0.014858376217045688
+screen_static,static_dropout_0.26,static,,250000,L8_H8_D256,8,8,256,8388608,0.26,0.26,constant,2,4.4744433760643005,0.008709941272715828,5.301199935376644,0.022748898461012714,0.8267565593123436,0.03145883973372854
+screen_static,static_dropout_0,static,,500000,L8_H8_D256,8,8,256,8388608,0.0,0.0,constant,2,4.463924840092659,0.011382620450761407,5.057585891336203,0.030907543150137856,0.5936610512435436,0.01952492269937645
+screen_static,static_dropout_0.02,static,,500000,L8_H8_D256,8,8,256,8388608,0.02,0.02,constant,2,4.4976213574409485,0.006352604815228758,5.056944504380226,0.029059525015124623,0.5593231469392776,0.035412129830353384
+screen_static,static_dropout_0.04,static,,500000,L8_H8_D256,8,8,256,8388608,0.04,0.04,constant,2,4.520545817911625,0.008343021346292112,5.065556924790144,0.03663996762624094,0.5450111068785191,0.028296946279948824
+screen_static,static_dropout_0.06,static,,500000,L8_H8_D256,8,8,256,8388608,0.06,0.06,constant,2,4.5480824783444405,7.43575774853448e-05,5.072879400104284,0.02535278871393245,0.5247969217598438,0.025278431136447104
+screen_static,static_dropout_0.08,static,,500000,L8_H8_D256,8,8,256,8388608,0.08,0.08,constant,2,4.576782673597336,0.003786578237339997,5.0890149138867855,0.03818276355269434,0.5122322402894497,0.03439618531535434
+screen_static,static_dropout_0.1,static,,500000,L8_H8_D256,8,8,256,8388608,0.1,0.1,constant,2,4.59450526535511,0.0006529389771307703,5.0904296934604645,0.028588154661378783,0.4959244281053543,0.029241093638509554
+screen_static,static_dropout_0,static,,1000000,L8_H8_D256,8,8,256,8388608,0.0,0.0,constant,2,4.695890702307224,0.017325758095994702,4.971932724118233,0.018370399346473592,0.27604202181100845,0.0356961574424683
+screen_static,static_dropout_0.02,static,,1000000,L8_H8_D256,8,8,256,8388608,0.02,0.02,constant,2,4.715626582503319,0.01698815130271031,4.983673017472029,0.0261924066198301,0.26804643496870995,0.043180557922540413
+screen_static,static_dropout_0.04,static,,1000000,L8_H8_D256,8,8,256,8388608,0.04,0.04,constant,2,4.736352913081646,0.010033278558973006,4.9948405139148235,0.02342923815803163,0.25848760083317757,0.033462516717004634
+screen_static,static_dropout_0.06,static,,1000000,L8_H8_D256,8,8,256,8388608,0.06,0.06,constant,2,4.750876776874065,0.0169620729401942,5.001798376441002,0.020779755551735234,0.2509215995669365,0.03774182849192943
+screen_static,static_dropout_0.08,static,,1000000,L8_H8_D256,8,8,256,8388608,0.08,0.08,constant,2,4.767140358686447,0.025763989173072925,5.0104669369757175,0.01812731212933095,0.2433265782892704,0.04389130130240387
+screen_static,static_dropout_0.1,static,,1000000,L8_H8_D256,8,8,256,8388608,0.1,0.1,constant,2,4.79401146620512,0.0013957034251503835,5.031736072152853,0.03717824610199782,0.23772460594773293,0.0385739495271482
+screen_static,static_dropout_0,static,,2000000,L8_H8_D256,8,8,256,8388608,0.0,0.0,constant,2,4.844547636806965,0.0008493327544794088,4.946536503732204,0.02745871395169597,0.10198886692523956,0.026609381197216564
+screen_static,static_dropout_0.02,static,,2000000,L8_H8_D256,8,8,256,8388608,0.02,0.02,constant,2,4.866960845887661,0.0042617523441639445,4.957204628735781,0.02519712986566959,0.09024378284811974,0.020935377521505646
+screen_static,static_dropout_0.04,static,,2000000,L8_H8_D256,8,8,256,8388608,0.04,0.04,constant,2,4.880623258650303,0.002468446086873914,4.968371279537678,0.01927565043221483,0.08774802088737488,0.016807204345340918
+screen_static,static_dropout_0.06,static,,2000000,L8_H8_D256,8,8,256,8388608,0.06,0.06,constant,2,4.891251288354397,0.0029265191099145656,4.977074716240168,0.02597958610827434,0.0858234278857708,0.023053066998359774
+screen_static,static_dropout_0.08,static,,2000000,L8_H8_D256,8,8,256,8388608,0.08,0.08,constant,2,4.917440451681614,0.00020678297550657385,4.995953656733036,0.02737851903569187,0.07851320505142212,0.027171736060185296
+screen_static,static_dropout_0.1,static,,2000000,L8_H8_D256,8,8,256,8388608,0.1,0.1,constant,2,4.928757295012474,0.0014230461931218261,5.004935186356306,0.020581807609347757,0.07617789134383202,0.02200485380246958
+screen_static,static_dropout_0,static,,4000000,L8_H8_D256,8,8,256,8388608,0.0,0.0,constant,2,4.909530982375145,0.005488805154997985,4.931619014590979,0.024494004528674472,0.022088032215833664,0.01900519937367649
+screen_static,static_dropout_0.02,static,,4000000,L8_H8_D256,8,8,256,8388608,0.02,0.02,constant,2,4.924341268837452,0.01023238027133847,4.939271181821823,0.02502982268214953,0.014929912984371185,0.014797442410811063
+screen_static,static_dropout_0.04,static,,4000000,L8_H8_D256,8,8,256,8388608,0.04,0.04,constant,2,4.935941606760025,0.0032379105634251786,4.947166424244642,0.02536144989130144,0.011224817484617233,0.02212353932787626
+screen_static,static_dropout_0.06,static,,4000000,L8_H8_D256,8,8,256,8388608,0.06,0.06,constant,2,4.955327145755291,0.008561236654457266,4.9727593921124935,0.024765746334448464,0.01743224635720253,0.016204509679991198
+screen_static,static_dropout_0.08,static,,4000000,L8_H8_D256,8,8,256,8388608,0.08,0.08,constant,2,4.968859277665615,0.0017042183562501279,4.984901204705238,0.017840392189736973,0.01604192703962326,0.0195446105459871
+screen_static,static_dropout_0.1,static,,4000000,L8_H8_D256,8,8,256,8388608,0.1,0.1,constant,2,4.985003612935543,0.0037800981593814472,4.991779696196318,0.020482815198907794,0.006776083260774612,0.016702717039526346

runs/wikitext103_interaction_static_screen/screen_static/20260603-153314/summary.json

@@ -0,0 +1,2488 @@
+[
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.5072663351893425,
+    "std_train_eval_loss": 0.056970032109433126,
+    "mean_val_eval_loss": 5.325984805822372,
+    "std_val_eval_loss": 0.042196002272969865,
+    "mean_generalization_gap": 1.81871847063303,
+    "std_generalization_gap": 0.09916603438240298
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.5965475998818874,
+    "std_train_eval_loss": 0.00036475463208146856,
+    "mean_val_eval_loss": 5.26322204247117,
+    "std_val_eval_loss": 0.03309956913105307,
+    "mean_generalization_gap": 1.666674442589283,
+    "std_generalization_gap": 0.0327348144989716
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.6675546914339066,
+    "std_train_eval_loss": 0.0015323645814469575,
+    "mean_val_eval_loss": 5.226230278611183,
+    "std_val_eval_loss": 0.010485914638555752,
+    "mean_generalization_gap": 1.5586755871772766,
+    "std_generalization_gap": 0.01201827922000271
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.729271735996008,
+    "std_train_eval_loss": 0.00595836846261392,
+    "mean_val_eval_loss": 5.2007541581988335,
+    "std_val_eval_loss": 0.019410783765252886,
+    "mean_generalization_gap": 1.4714824222028255,
+    "std_generalization_gap": 0.025369152227866804
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.7556542232632637,
+    "std_train_eval_loss": 0.0010223139574802457,
+    "mean_val_eval_loss": 5.180825922638178,
+    "std_val_eval_loss": 0.0188014509712627,
+    "mean_generalization_gap": 1.4251716993749142,
+    "std_generalization_gap": 0.019823764928742944
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.818895984441042,
+    "std_train_eval_loss": 0.02959928389893545,
+    "mean_val_eval_loss": 5.163311161100864,
+    "std_val_eval_loss": 0.029076731466029196,
+    "mean_generalization_gap": 1.3444151766598225,
+    "std_generalization_gap": 0.05867601536496465
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.9072264805436134,
+    "std_train_eval_loss": 0.016073122148114547,
+    "mean_val_eval_loss": 5.15796884149313,
+    "std_val_eval_loss": 0.03002937668292093,
+    "mean_generalization_gap": 1.2507423609495163,
+    "std_generalization_gap": 0.04610249883103548
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.003099720925093,
+    "std_train_eval_loss": 0.011945212389753011,
+    "mean_val_eval_loss": 5.154339708387852,
+    "std_val_eval_loss": 0.01651832984564725,
+    "mean_generalization_gap": 1.151239987462759,
+    "std_generalization_gap": 0.02846354223540026
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.2",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.2,
+    "dropout_final": 0.2,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.040268834680319,
+    "std_train_eval_loss": 0.0021783019133688555,
+    "mean_val_eval_loss": 5.16454915329814,
+    "std_val_eval_loss": 0.016607855020240453,
+    "mean_generalization_gap": 1.1242803186178207,
+    "std_generalization_gap": 0.018786156933609306
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.26",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.26,
+    "dropout_final": 0.26,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.154937222599983,
+    "std_train_eval_loss": 0.01848742007136409,
+    "mean_val_eval_loss": 5.187990374863148,
+    "std_val_eval_loss": 0.015247644509892305,
+    "mean_generalization_gap": 1.0330531522631645,
+    "std_generalization_gap": 0.0337350645812564
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.3",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.3,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.223974447697401,
+    "std_train_eval_loss": 0.007560475541634279,
+    "mean_val_eval_loss": 5.202283661812544,
+    "std_val_eval_loss": 0.021969144885068726,
+    "mean_generalization_gap": 0.9783092141151428,
+    "std_generalization_gap": 0.029529620426703006
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.4",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.4,
+    "dropout_final": 0.4,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.395416356623173,
+    "std_train_eval_loss": 0.0069791071816310704,
+    "mean_val_eval_loss": 5.285179540514946,
+    "std_val_eval_loss": 0.021146865138977247,
+    "mean_generalization_gap": 0.8897631838917732,
+    "std_generalization_gap": 0.02812597232060832
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.14875391125679,
+    "std_train_eval_loss": 0.015905230177071402,
+    "mean_val_eval_loss": 4.956410575658083,
+    "std_val_eval_loss": 0.021681071175496766,
+    "mean_generalization_gap": 0.8076566644012928,
+    "std_generalization_gap": 0.03758630135256817
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.1691304333508015,
+    "std_train_eval_loss": 0.005854845265959036,
+    "mean_val_eval_loss": 4.939401715993881,
+    "std_val_eval_loss": 0.02775412121290883,
+    "mean_generalization_gap": 0.7702712826430798,
+    "std_generalization_gap": 0.021899275946949792
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.210759695619345,
+    "std_train_eval_loss": 0.021864167622140218,
+    "mean_val_eval_loss": 4.939492538571358,
+    "std_val_eval_loss": 0.030459295611156307,
+    "mean_generalization_gap": 0.728732842952013,
+    "std_generalization_gap": 0.05232346323329652
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.244226388633251,
+    "std_train_eval_loss": 0.0010815829631986903,
+    "mean_val_eval_loss": 4.951852090656757,
+    "std_val_eval_loss": 0.0168901282697861,
+    "mean_generalization_gap": 0.7076257020235062,
+    "std_generalization_gap": 0.017971711232984793
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.271009549498558,
+    "std_train_eval_loss": 0.00805379913509823,
+    "mean_val_eval_loss": 4.945284515619278,
+    "std_val_eval_loss": 0.027335982328832248,
+    "mean_generalization_gap": 0.6742749661207199,
+    "std_generalization_gap": 0.01928218319373402
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.292945623397827,
+    "std_train_eval_loss": 0.007025542471977948,
+    "mean_val_eval_loss": 4.943468101322651,
+    "std_val_eval_loss": 0.02226113298319614,
+    "mean_generalization_gap": 0.6505224779248238,
+    "std_generalization_gap": 0.029286675455174084
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.344617947936058,
+    "std_train_eval_loss": 0.012038151459075597,
+    "mean_val_eval_loss": 4.965619098395109,
+    "std_val_eval_loss": 0.039715876240702365,
+    "mean_generalization_gap": 0.6210011504590511,
+    "std_generalization_gap": 0.02767772478162677
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.401914454996586,
+    "std_train_eval_loss": 0.0007282448587076103,
+    "mean_val_eval_loss": 4.9909077398478985,
+    "std_val_eval_loss": 0.030591188905215085,
+    "mean_generalization_gap": 0.5889932848513126,
+    "std_generalization_gap": 0.0313194337639227
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.448670491576195,
+    "std_train_eval_loss": 0.027271750531701645,
+    "mean_val_eval_loss": 4.819691203534603,
+    "std_val_eval_loss": 0.024108977262459263,
+    "mean_generalization_gap": 0.37102071195840836,
+    "std_generalization_gap": 0.05138072779416091
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.451282858848572,
+    "std_train_eval_loss": 0.04242921132249276,
+    "mean_val_eval_loss": 4.820131976157427,
+    "std_val_eval_loss": 0.016412630817937993,
+    "mean_generalization_gap": 0.36884911730885506,
+    "std_generalization_gap": 0.058841842140430756
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.47072771191597,
+    "std_train_eval_loss": 0.024538169157557826,
+    "mean_val_eval_loss": 4.827508602291346,
+    "std_val_eval_loss": 0.022596026573107636,
+    "mean_generalization_gap": 0.35678089037537575,
+    "std_generalization_gap": 0.04713419573066546
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.502691887319088,
+    "std_train_eval_loss": 0.024540898165998908,
+    "mean_val_eval_loss": 4.837066236883402,
+    "std_val_eval_loss": 0.022903940911616103,
+    "mean_generalization_gap": 0.3343743495643139,
+    "std_generalization_gap": 0.04744483907761501
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.511857099831104,
+    "std_train_eval_loss": 0.020942020918502176,
+    "mean_val_eval_loss": 4.849667362868786,
+    "std_val_eval_loss": 0.029055995216561836,
+    "mean_generalization_gap": 0.3378102630376816,
+    "std_generalization_gap": 0.04999801613506401
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.542278811335564,
+    "std_train_eval_loss": 0.030395527429359324,
+    "mean_val_eval_loss": 4.864129442721605,
+    "std_val_eval_loss": 0.023052961631238496,
+    "mean_generalization_gap": 0.32185063138604164,
+    "std_generalization_gap": 0.05344848906059782
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.621733330190182,
+    "std_train_eval_loss": 0.012111012823438806,
+    "mean_val_eval_loss": 4.769962530583143,
+    "std_val_eval_loss": 0.024905420990413565,
+    "mean_generalization_gap": 0.1482292003929615,
+    "std_generalization_gap": 0.03701643381385237
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.636836856603622,
+    "std_train_eval_loss": 0.004119475120303293,
+    "mean_val_eval_loss": 4.776707325130701,
+    "std_val_eval_loss": 0.037399464373119376,
+    "mean_generalization_gap": 0.13987046852707863,
+    "std_generalization_gap": 0.03327998925281608
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.646803148090839,
+    "std_train_eval_loss": 0.011796744847517323,
+    "mean_val_eval_loss": 4.78968957811594,
+    "std_val_eval_loss": 0.0230620284720244,
+    "mean_generalization_gap": 0.1428864300251007,
+    "std_generalization_gap": 0.03485877331954172
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.664006285369396,
+    "std_train_eval_loss": 0.0004938030138657622,
+    "mean_val_eval_loss": 4.802294943481684,
+    "std_val_eval_loss": 0.0302215821171988,
+    "mean_generalization_gap": 0.13828865811228752,
+    "std_generalization_gap": 0.02972777910333304
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.685344219207764,
+    "std_train_eval_loss": 0.0007423956630951429,
+    "mean_val_eval_loss": 4.807585842907429,
+    "std_val_eval_loss": 0.02933789441282695,
+    "mean_generalization_gap": 0.12224162369966507,
+    "std_generalization_gap": 0.03008029007592209
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.704172521829605,
+    "std_train_eval_loss": 0.00673639401776896,
+    "mean_val_eval_loss": 4.8234143033623695,
+    "std_val_eval_loss": 0.022559722331471566,
+    "mean_generalization_gap": 0.11924178153276443,
+    "std_generalization_gap": 0.029296116349240526
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.700695753097534,
+    "std_train_eval_loss": 0.007609592425217662,
+    "mean_val_eval_loss": 4.751548711210489,
+    "std_val_eval_loss": 0.02019173852637943,
+    "mean_generalization_gap": 0.05085295811295509,
+    "std_generalization_gap": 0.027801330951597095
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.711871832609177,
+    "std_train_eval_loss": 0.003394865427279748,
+    "mean_val_eval_loss": 4.754005387425423,
+    "std_val_eval_loss": 0.019647870324838793,
+    "mean_generalization_gap": 0.04213355481624603,
+    "std_generalization_gap": 0.02304273575211854
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.728417493402958,
+    "std_train_eval_loss": 0.00032852414743029124,
+    "mean_val_eval_loss": 4.765943691134453,
+    "std_val_eval_loss": 0.017364290852245785,
+    "mean_generalization_gap": 0.037526197731494904,
+    "std_generalization_gap": 0.017035766704815495
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.739520192146301,
+    "std_train_eval_loss": 0.00017573128486617266,
+    "mean_val_eval_loss": 4.781056735664606,
+    "std_val_eval_loss": 0.023307064980910674,
+    "mean_generalization_gap": 0.041536543518304825,
+    "std_generalization_gap": 0.02348279626577685
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.756850086152554,
+    "std_train_eval_loss": 0.0019432331341554079,
+    "mean_val_eval_loss": 4.796308144927025,
+    "std_val_eval_loss": 0.020058454386319792,
+    "mean_generalization_gap": 0.03945805877447128,
+    "std_generalization_gap": 0.0220016875204752
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.771451950073242,
+    "std_train_eval_loss": 0.002899787471246531,
+    "mean_val_eval_loss": 4.803060214966536,
+    "std_val_eval_loss": 0.028490136896810635,
+    "mean_generalization_gap": 0.03160826489329338,
+    "std_generalization_gap": 0.03138992436805717
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 2.984476812183857,
+    "std_train_eval_loss": 0.07573507347193847,
+    "mean_val_eval_loss": 5.459531031548977,
+    "std_val_eval_loss": 0.052592860620622094,
+    "mean_generalization_gap": 2.47505421936512,
+    "std_generalization_gap": 0.12832793409256057
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.066030155867338,
+    "std_train_eval_loss": 0.0044021171547734125,
+    "mean_val_eval_loss": 5.395382076501846,
+    "std_val_eval_loss": 0.028443274869622583,
+    "mean_generalization_gap": 2.329351920634508,
+    "std_generalization_gap": 0.032845392024396
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.1688508801162243,
+    "std_train_eval_loss": 0.012333163593583664,
+    "mean_val_eval_loss": 5.33031689748168,
+    "std_val_eval_loss": 0.009347870703420656,
+    "mean_generalization_gap": 2.1614660173654556,
+    "std_generalization_gap": 0.02168103429700432
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.2494926266372204,
+    "std_train_eval_loss": 0.008142987134903345,
+    "mean_val_eval_loss": 5.262157831341028,
+    "std_val_eval_loss": 0.02011042671888979,
+    "mean_generalization_gap": 2.012665204703808,
+    "std_generalization_gap": 0.028253413853793135
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.3453744649887085,
+    "std_train_eval_loss": 0.04799909647731358,
+    "mean_val_eval_loss": 5.232690017670393,
+    "std_val_eval_loss": 0.019404035001135078,
+    "mean_generalization_gap": 1.8873155526816845,
+    "std_generalization_gap": 0.06740313147844866
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.4033044539391994,
+    "std_train_eval_loss": 0.024623869505648665,
+    "mean_val_eval_loss": 5.193579941987991,
+    "std_val_eval_loss": 0.03371155717649944,
+    "mean_generalization_gap": 1.7902754880487919,
+    "std_generalization_gap": 0.05833542668214811
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.556764382869005,
+    "std_train_eval_loss": 0.013851588104885643,
+    "mean_val_eval_loss": 5.13896181806922,
+    "std_val_eval_loss": 0.031193730788238956,
+    "mean_generalization_gap": 1.5821974352002144,
+    "std_generalization_gap": 0.045045318893124596
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.6555825024843216,
+    "std_train_eval_loss": 0.015018934636447939,
+    "mean_val_eval_loss": 5.133807681500912,
+    "std_val_eval_loss": 0.03364965399274907,
+    "mean_generalization_gap": 1.4782251790165901,
+    "std_generalization_gap": 0.048668588629197006
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.2",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.2,
+    "dropout_final": 0.2,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.7214544899761677,
+    "std_train_eval_loss": 0.004305126719637718,
+    "mean_val_eval_loss": 5.131882190704346,
+    "std_val_eval_loss": 0.026161212683575973,
+    "mean_generalization_gap": 1.410427700728178,
+    "std_generalization_gap": 0.03046633940321369
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.26",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.26,
+    "dropout_final": 0.26,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.851325035095215,
+    "std_train_eval_loss": 0.026488904430748174,
+    "mean_val_eval_loss": 5.1221378818154335,
+    "std_val_eval_loss": 0.016875071308155583,
+    "mean_generalization_gap": 1.2708128467202187,
+    "std_generalization_gap": 0.04336397573890376
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.3",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.3,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.9617567881941795,
+    "std_train_eval_loss": 0.012010208098512875,
+    "mean_val_eval_loss": 5.136128347367048,
+    "std_val_eval_loss": 0.02499181149314877,
+    "mean_generalization_gap": 1.1743715591728687,
+    "std_generalization_gap": 0.012981603394635896
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.4",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.4,
+    "dropout_final": 0.4,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.154601521790028,
+    "std_train_eval_loss": 0.01615451824930121,
+    "mean_val_eval_loss": 5.192008286714554,
+    "std_val_eval_loss": 0.029119299783025198,
+    "mean_generalization_gap": 1.0374067649245262,
+    "std_generalization_gap": 0.04527381803232641
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.5",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.5,
+    "dropout_final": 0.5,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.372598148882389,
+    "std_train_eval_loss": 0.019609295421739194,
+    "mean_val_eval_loss": 5.297821056097746,
+    "std_val_eval_loss": 0.011355662272782625,
+    "mean_generalization_gap": 0.9252229072153568,
+    "std_generalization_gap": 0.030964957694521823
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.8252953365445137,
+    "std_train_eval_loss": 0.019563755751923175,
+    "mean_val_eval_loss": 4.916963316500187,
+    "std_val_eval_loss": 0.03401019920833551,
+    "mean_generalization_gap": 1.0916679799556732,
+    "std_generalization_gap": 0.053573954960258685
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.8655518889427185,
+    "std_train_eval_loss": 0.018674899790252674,
+    "mean_val_eval_loss": 4.889844581484795,
+    "std_val_eval_loss": 0.03132540281134305,
+    "mean_generalization_gap": 1.024292692542076,
+    "std_generalization_gap": 0.012650503021090376
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.914637830108404,
+    "std_train_eval_loss": 0.014714407850888537,
+    "mean_val_eval_loss": 4.881916884332895,
+    "std_val_eval_loss": 0.04055301801925,
+    "mean_generalization_gap": 0.9672790542244911,
+    "std_generalization_gap": 0.05526742587013854
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 3.9587855264544487,
+    "std_train_eval_loss": 0.010768256376730232,
+    "mean_val_eval_loss": 4.872062597423792,
+    "std_val_eval_loss": 0.029954139289972922,
+    "mean_generalization_gap": 0.9132770709693432,
+    "std_generalization_gap": 0.01918588291324269
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.010694112628698,
+    "std_train_eval_loss": 0.019106172685996462,
+    "mean_val_eval_loss": 4.868544135242701,
+    "std_val_eval_loss": 0.03936356019728768,
+    "mean_generalization_gap": 0.8578500226140022,
+    "std_generalization_gap": 0.05846973288328414
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.053545691072941,
+    "std_train_eval_loss": 0.00040188073726350233,
+    "mean_val_eval_loss": 4.870775658637285,
+    "std_val_eval_loss": 0.02582369545234467,
+    "mean_generalization_gap": 0.8172299675643444,
+    "std_generalization_gap": 0.025421814715081165
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.120105978101492,
+    "std_train_eval_loss": 0.0015854854156388755,
+    "mean_val_eval_loss": 4.8797769248485565,
+    "std_val_eval_loss": 0.03186888642289103,
+    "mean_generalization_gap": 0.7596709467470646,
+    "std_generalization_gap": 0.030283401007252155
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.1665811985731125,
+    "std_train_eval_loss": 0.0004332274558434798,
+    "mean_val_eval_loss": 4.887217383831739,
+    "std_val_eval_loss": 0.014720445386937724,
+    "mean_generalization_gap": 0.7206361852586269,
+    "std_generalization_gap": 0.015153672842781202
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.2",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.2,
+    "dropout_final": 0.2,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.197941534221172,
+    "std_train_eval_loss": 0.0025712527871041124,
+    "mean_val_eval_loss": 4.895248021930456,
+    "std_val_eval_loss": 0.015913675351841773,
+    "mean_generalization_gap": 0.6973064877092838,
+    "std_generalization_gap": 0.013342422564737659
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.26",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.26,
+    "dropout_final": 0.26,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.29030442237854,
+    "std_train_eval_loss": 0.0044379156342273535,
+    "mean_val_eval_loss": 4.926884576678276,
+    "std_val_eval_loss": 0.028275551485973483,
+    "mean_generalization_gap": 0.636580154299736,
+    "std_generalization_gap": 0.03271346712020084
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.232576880604029,
+    "std_train_eval_loss": 0.052909293890885827,
+    "mean_val_eval_loss": 4.7177883349359035,
+    "std_val_eval_loss": 0.009086444338819707,
+    "mean_generalization_gap": 0.48521145433187485,
+    "std_generalization_gap": 0.061995738229705534
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.225924864411354,
+    "std_train_eval_loss": 0.013052246781096219,
+    "mean_val_eval_loss": 4.7049963511526585,
+    "std_val_eval_loss": 0.026859896796409256,
+    "mean_generalization_gap": 0.4790714867413044,
+    "std_generalization_gap": 0.03991214357750547
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.269359432160854,
+    "std_train_eval_loss": 0.019288157509510438,
+    "mean_val_eval_loss": 4.712642755359411,
+    "std_val_eval_loss": 0.026603285709250676,
+    "mean_generalization_gap": 0.4432833231985569,
+    "std_generalization_gap": 0.04589144321876112
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.291698649525642,
+    "std_train_eval_loss": 0.01393341094291348,
+    "mean_val_eval_loss": 4.718549467623234,
+    "std_val_eval_loss": 0.030084041145439562,
+    "mean_generalization_gap": 0.4268508180975914,
+    "std_generalization_gap": 0.04401745208835304
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.3292553797364235,
+    "std_train_eval_loss": 0.03327905675379278,
+    "mean_val_eval_loss": 4.743360877037048,
+    "std_val_eval_loss": 0.03075722114656769,
+    "mean_generalization_gap": 0.41410549730062485,
+    "std_generalization_gap": 0.06403627790036047
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.351991422474384,
+    "std_train_eval_loss": 0.030442700289555143,
+    "mean_val_eval_loss": 4.745819546282291,
+    "std_val_eval_loss": 0.021169972148673347,
+    "mean_generalization_gap": 0.3938281238079071,
+    "std_generalization_gap": 0.05161267243822849
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.450904347002506,
+    "std_train_eval_loss": 0.01087398174621981,
+    "mean_val_eval_loss": 4.638469863682985,
+    "std_val_eval_loss": 0.020520631458734194,
+    "mean_generalization_gap": 0.18756551668047905,
+    "std_generalization_gap": 0.031394613204954
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.468206360936165,
+    "std_train_eval_loss": 0.007094136467929428,
+    "mean_val_eval_loss": 4.649582006037235,
+    "std_val_eval_loss": 0.03501839397156461,
+    "mean_generalization_gap": 0.1813756451010704,
+    "std_generalization_gap": 0.02792425750363518
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.490154348313808,
+    "std_train_eval_loss": 0.011619222321589438,
+    "mean_val_eval_loss": 4.660393413156271,
+    "std_val_eval_loss": 0.01666943156591487,
+    "mean_generalization_gap": 0.17023906484246254,
+    "std_generalization_gap": 0.028288653887504307
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.50459598004818,
+    "std_train_eval_loss": 0.0040620500392072,
+    "mean_val_eval_loss": 4.675430595874786,
+    "std_val_eval_loss": 0.03429217216158552,
+    "mean_generalization_gap": 0.17083461582660675,
+    "std_generalization_gap": 0.03023012212237832
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.517679311335087,
+    "std_train_eval_loss": 0.0200507415130423,
+    "mean_val_eval_loss": 4.682192903012037,
+    "std_val_eval_loss": 0.023959266388192506,
+    "mean_generalization_gap": 0.16451359167695045,
+    "std_generalization_gap": 0.044010007901234806
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.543015904724598,
+    "std_train_eval_loss": 0.010151036853712445,
+    "mean_val_eval_loss": 4.699402842670679,
+    "std_val_eval_loss": 0.02188920285015569,
+    "mean_generalization_gap": 0.15638693794608116,
+    "std_generalization_gap": 0.032040239703868136
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.544128015637398,
+    "std_train_eval_loss": 0.008078707922568169,
+    "mean_val_eval_loss": 4.614261016249657,
+    "std_val_eval_loss": 0.0343418000757071,
+    "mean_generalization_gap": 0.07013300061225891,
+    "std_generalization_gap": 0.02626309215313893
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.560969464480877,
+    "std_train_eval_loss": 0.005450209178474134,
+    "mean_val_eval_loss": 4.634637780487537,
+    "std_val_eval_loss": 0.02513911999705041,
+    "mean_generalization_gap": 0.07366831600666046,
+    "std_generalization_gap": 0.03058932917552454
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.567037917673588,
+    "std_train_eval_loss": 0.004047477766334559,
+    "mean_val_eval_loss": 4.635697934776545,
+    "std_val_eval_loss": 0.02091666432076681,
+    "mean_generalization_gap": 0.06866001710295677,
+    "std_generalization_gap": 0.01686918655443225
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.590242467820644,
+    "std_train_eval_loss": 0.013435920079803014,
+    "mean_val_eval_loss": 4.657844081521034,
+    "std_val_eval_loss": 0.00958948278086542,
+    "mean_generalization_gap": 0.06760161370038986,
+    "std_generalization_gap": 0.023025402860668433
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.602519556879997,
+    "std_train_eval_loss": 0.0039952051374689225,
+    "mean_val_eval_loss": 4.6621916852891445,
+    "std_val_eval_loss": 0.029447955639357086,
+    "mean_generalization_gap": 0.05967212840914726,
+    "std_generalization_gap": 0.033443160776826006
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L16_H8_D384",
+    "n_layer": 16,
+    "n_head": 8,
+    "n_embd": 384,
+    "parameters": 31457280,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.624893218278885,
+    "std_train_eval_loss": 0.014938845087565112,
+    "mean_val_eval_loss": 4.680121582001448,
+    "std_val_eval_loss": 0.03926061651979982,
+    "mean_generalization_gap": 0.05522836372256279,
+    "std_generalization_gap": 0.024321771432234705
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.035547886043787,
+    "std_train_eval_loss": 0.024195878795732664,
+    "mean_val_eval_loss": 5.28285102173686,
+    "std_val_eval_loss": 0.0324948988321794,
+    "mean_generalization_gap": 1.2473031356930733,
+    "std_generalization_gap": 0.008299020036446739
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.089681029319763,
+    "std_train_eval_loss": 0.008525749008011093,
+    "mean_val_eval_loss": 5.247419081628323,
+    "std_val_eval_loss": 0.03137372417316079,
+    "mean_generalization_gap": 1.1577380523085594,
+    "std_generalization_gap": 0.03989947318117189
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.125994935631752,
+    "std_train_eval_loss": 0.00561314889481725,
+    "mean_val_eval_loss": 5.236050847917795,
+    "std_val_eval_loss": 0.01603267697861228,
+    "mean_generalization_gap": 1.1100559122860432,
+    "std_generalization_gap": 0.02164582587342953
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.174422111362219,
+    "std_train_eval_loss": 0.010122550852475144,
+    "mean_val_eval_loss": 5.236642763018608,
+    "std_val_eval_loss": 0.012643158932737352,
+    "mean_generalization_gap": 1.0622206516563892,
+    "std_generalization_gap": 0.0025206080802622093
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.211392514407635,
+    "std_train_eval_loss": 0.0006487559024160641,
+    "mean_val_eval_loss": 5.228285621851683,
+    "std_val_eval_loss": 0.009615418897767812,
+    "mean_generalization_gap": 1.016893107444048,
+    "std_generalization_gap": 0.008966662995351747
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.236091569066048,
+    "std_train_eval_loss": 0.003005154592523766,
+    "mean_val_eval_loss": 5.226473439484835,
+    "std_val_eval_loss": 0.02592699737804487,
+    "mean_generalization_gap": 0.990381870418787,
+    "std_generalization_gap": 0.028932151970568636
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.300983510911465,
+    "std_train_eval_loss": 0.014899132219555723,
+    "mean_val_eval_loss": 5.239528384059668,
+    "std_val_eval_loss": 0.01265994918351288,
+    "mean_generalization_gap": 0.9385448731482029,
+    "std_generalization_gap": 0.027559081403068602
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.3531088307499886,
+    "std_train_eval_loss": 0.0033408015573523986,
+    "mean_val_eval_loss": 5.251454934477806,
+    "std_val_eval_loss": 0.008830207304941664,
+    "mean_generalization_gap": 0.8983461037278175,
+    "std_generalization_gap": 0.012171008862294063
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.2",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.2,
+    "dropout_final": 0.2,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.38435186445713,
+    "std_train_eval_loss": 0.004200987125323363,
+    "mean_val_eval_loss": 5.271038182079792,
+    "std_val_eval_loss": 0.01905936334236905,
+    "mean_generalization_gap": 0.8866863176226616,
+    "std_generalization_gap": 0.014858376217045688
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.26",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 250000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.26,
+    "dropout_final": 0.26,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.4744433760643005,
+    "std_train_eval_loss": 0.008709941272715828,
+    "mean_val_eval_loss": 5.301199935376644,
+    "std_val_eval_loss": 0.022748898461012714,
+    "mean_generalization_gap": 0.8267565593123436,
+    "std_generalization_gap": 0.03145883973372854
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.463924840092659,
+    "std_train_eval_loss": 0.011382620450761407,
+    "mean_val_eval_loss": 5.057585891336203,
+    "std_val_eval_loss": 0.030907543150137856,
+    "mean_generalization_gap": 0.5936610512435436,
+    "std_generalization_gap": 0.01952492269937645
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.4976213574409485,
+    "std_train_eval_loss": 0.006352604815228758,
+    "mean_val_eval_loss": 5.056944504380226,
+    "std_val_eval_loss": 0.029059525015124623,
+    "mean_generalization_gap": 0.5593231469392776,
+    "std_generalization_gap": 0.035412129830353384
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.520545817911625,
+    "std_train_eval_loss": 0.008343021346292112,
+    "mean_val_eval_loss": 5.065556924790144,
+    "std_val_eval_loss": 0.03663996762624094,
+    "mean_generalization_gap": 0.5450111068785191,
+    "std_generalization_gap": 0.028296946279948824
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.5480824783444405,
+    "std_train_eval_loss": 7.43575774853448e-05,
+    "mean_val_eval_loss": 5.072879400104284,
+    "std_val_eval_loss": 0.02535278871393245,
+    "mean_generalization_gap": 0.5247969217598438,
+    "std_generalization_gap": 0.025278431136447104
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.576782673597336,
+    "std_train_eval_loss": 0.003786578237339997,
+    "mean_val_eval_loss": 5.0890149138867855,
+    "std_val_eval_loss": 0.03818276355269434,
+    "mean_generalization_gap": 0.5122322402894497,
+    "std_generalization_gap": 0.03439618531535434
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 500000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.59450526535511,
+    "std_train_eval_loss": 0.0006529389771307703,
+    "mean_val_eval_loss": 5.0904296934604645,
+    "std_val_eval_loss": 0.028588154661378783,
+    "mean_generalization_gap": 0.4959244281053543,
+    "std_generalization_gap": 0.029241093638509554
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.695890702307224,
+    "std_train_eval_loss": 0.017325758095994702,
+    "mean_val_eval_loss": 4.971932724118233,
+    "std_val_eval_loss": 0.018370399346473592,
+    "mean_generalization_gap": 0.27604202181100845,
+    "std_generalization_gap": 0.0356961574424683
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.715626582503319,
+    "std_train_eval_loss": 0.01698815130271031,
+    "mean_val_eval_loss": 4.983673017472029,
+    "std_val_eval_loss": 0.0261924066198301,
+    "mean_generalization_gap": 0.26804643496870995,
+    "std_generalization_gap": 0.043180557922540413
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.736352913081646,
+    "std_train_eval_loss": 0.010033278558973006,
+    "mean_val_eval_loss": 4.9948405139148235,
+    "std_val_eval_loss": 0.02342923815803163,
+    "mean_generalization_gap": 0.25848760083317757,
+    "std_generalization_gap": 0.033462516717004634
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.750876776874065,
+    "std_train_eval_loss": 0.0169620729401942,
+    "mean_val_eval_loss": 5.001798376441002,
+    "std_val_eval_loss": 0.020779755551735234,
+    "mean_generalization_gap": 0.2509215995669365,
+    "std_generalization_gap": 0.03774182849192943
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.767140358686447,
+    "std_train_eval_loss": 0.025763989173072925,
+    "mean_val_eval_loss": 5.0104669369757175,
+    "std_val_eval_loss": 0.01812731212933095,
+    "mean_generalization_gap": 0.2433265782892704,
+    "std_generalization_gap": 0.04389130130240387
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 1000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.79401146620512,
+    "std_train_eval_loss": 0.0013957034251503835,
+    "mean_val_eval_loss": 5.031736072152853,
+    "std_val_eval_loss": 0.03717824610199782,
+    "mean_generalization_gap": 0.23772460594773293,
+    "std_generalization_gap": 0.0385739495271482
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.844547636806965,
+    "std_train_eval_loss": 0.0008493327544794088,
+    "mean_val_eval_loss": 4.946536503732204,
+    "std_val_eval_loss": 0.02745871395169597,
+    "mean_generalization_gap": 0.10198886692523956,
+    "std_generalization_gap": 0.026609381197216564
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.866960845887661,
+    "std_train_eval_loss": 0.0042617523441639445,
+    "mean_val_eval_loss": 4.957204628735781,
+    "std_val_eval_loss": 0.02519712986566959,
+    "mean_generalization_gap": 0.09024378284811974,
+    "std_generalization_gap": 0.020935377521505646
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.880623258650303,
+    "std_train_eval_loss": 0.002468446086873914,
+    "mean_val_eval_loss": 4.968371279537678,
+    "std_val_eval_loss": 0.01927565043221483,
+    "mean_generalization_gap": 0.08774802088737488,
+    "std_generalization_gap": 0.016807204345340918
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.891251288354397,
+    "std_train_eval_loss": 0.0029265191099145656,
+    "mean_val_eval_loss": 4.977074716240168,
+    "std_val_eval_loss": 0.02597958610827434,
+    "mean_generalization_gap": 0.0858234278857708,
+    "std_generalization_gap": 0.023053066998359774
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.917440451681614,
+    "std_train_eval_loss": 0.00020678297550657385,
+    "mean_val_eval_loss": 4.995953656733036,
+    "std_val_eval_loss": 0.02737851903569187,
+    "mean_generalization_gap": 0.07851320505142212,
+    "std_generalization_gap": 0.027171736060185296
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 2000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.928757295012474,
+    "std_train_eval_loss": 0.0014230461931218261,
+    "mean_val_eval_loss": 5.004935186356306,
+    "std_val_eval_loss": 0.020581807609347757,
+    "mean_generalization_gap": 0.07617789134383202,
+    "std_generalization_gap": 0.02200485380246958
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.909530982375145,
+    "std_train_eval_loss": 0.005488805154997985,
+    "mean_val_eval_loss": 4.931619014590979,
+    "std_val_eval_loss": 0.024494004528674472,
+    "mean_generalization_gap": 0.022088032215833664,
+    "std_generalization_gap": 0.01900519937367649
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.924341268837452,
+    "std_train_eval_loss": 0.01023238027133847,
+    "mean_val_eval_loss": 4.939271181821823,
+    "std_val_eval_loss": 0.02502982268214953,
+    "mean_generalization_gap": 0.014929912984371185,
+    "std_generalization_gap": 0.014797442410811063
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.935941606760025,
+    "std_train_eval_loss": 0.0032379105634251786,
+    "mean_val_eval_loss": 4.947166424244642,
+    "std_val_eval_loss": 0.02536144989130144,
+    "mean_generalization_gap": 0.011224817484617233,
+    "std_generalization_gap": 0.02212353932787626
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.955327145755291,
+    "std_train_eval_loss": 0.008561236654457266,
+    "mean_val_eval_loss": 4.9727593921124935,
+    "std_val_eval_loss": 0.024765746334448464,
+    "mean_generalization_gap": 0.01743224635720253,
+    "std_generalization_gap": 0.016204509679991198
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.968859277665615,
+    "std_train_eval_loss": 0.0017042183562501279,
+    "mean_val_eval_loss": 4.984901204705238,
+    "std_val_eval_loss": 0.017840392189736973,
+    "mean_generalization_gap": 0.01604192703962326,
+    "std_generalization_gap": 0.0195446105459871
+  },
+  {
+    "run_mode": "screen_static",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": null,
+    "token_limit": 4000000,
+    "model_name": "L8_H8_D256",
+    "n_layer": 8,
+    "n_head": 8,
+    "n_embd": 256,
+    "parameters": 8388608,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 2,
+    "mean_train_eval_loss": 4.985003612935543,
+    "std_train_eval_loss": 0.0037800981593814472,
+    "mean_val_eval_loss": 4.991779696196318,
+    "std_val_eval_loss": 0.020482815198907794,
+    "mean_generalization_gap": 0.006776083260774612,
+    "std_generalization_gap": 0.016702717039526346
+  }
+]

runs/wikitext103_l20_8m_extrapolation_3seed/locked_stream/20260603-231952/RESULT_SUMMARY.md

@@ -0,0 +1,141 @@
+# Locked Streaming Dropout Summary
+
+Run directory: `runs/wikitext103_l20_8m_extrapolation_3seed/locked_stream/20260603-231952`
+
+Model: `L20_H8_D448` causal Transformer, 51,838,976 parameters, 20 layers, 8 heads, 448 embedding dim.
+Training per stage: 1,000 steps. Sampled tokens are cumulative in each stage row. Seeds present: 1, 2, 3.
+
+## Condition Ranking
+
+| Condition | Kind | Final dropout | Mean trajectory val loss | Final val loss | Final gap | Dropout path |
+|---|---|---:|---:|---:|---:|---|
+| `static_dropout_0.26` | static | 0.26 | 4.4829 | 3.9842 | 0.2766 | 0.26 -> 0.26 -> 0.26 -> 0.26 -> 0.26 -> 0.26 |
+| `static_dropout_0.3` | static | 0.30 | 4.4869 | 4.0015 | 0.2584 | 0.30 -> 0.30 -> 0.30 -> 0.30 -> 0.30 -> 0.30 |
+| `wikitext103_l20_interaction_8m` | anchor_decay | 0.02 | 4.4907 | 3.9572 | 0.3623 | 0.28 -> 0.21 -> 0.12 -> 0.04 -> 0.02 -> 0.02 |
+| `static_dropout_0.2` | static | 0.20 | 4.5001 | 3.9807 | 0.2995 | 0.20 -> 0.20 -> 0.20 -> 0.20 -> 0.20 -> 0.20 |
+| `static_dropout_0.18` | static | 0.18 | 4.5094 | 3.9758 | 0.2984 | 0.18 -> 0.18 -> 0.18 -> 0.18 -> 0.18 -> 0.18 |
+| `static_dropout_0.4` | static | 0.40 | 4.5192 | 4.0503 | 0.2248 | 0.40 -> 0.40 -> 0.40 -> 0.40 -> 0.40 -> 0.40 |
+| `static_dropout_0.14` | static | 0.14 | 4.5424 | 3.9756 | 0.3122 | 0.14 -> 0.14 -> 0.14 -> 0.14 -> 0.14 -> 0.14 |
+| `static_dropout_0.5` | static | 0.50 | 4.5915 | 4.1252 | 0.1949 | 0.50 -> 0.50 -> 0.50 -> 0.50 -> 0.50 -> 0.50 |
+| `static_dropout_0.1` | static | 0.10 | 4.5955 | 3.9785 | 0.3347 | 0.10 -> 0.10 -> 0.10 -> 0.10 -> 0.10 -> 0.10 |
+| `static_dropout_0.08` | static | 0.08 | 4.6344 | 3.9893 | 0.3473 | 0.08 -> 0.08 -> 0.08 -> 0.08 -> 0.08 -> 0.08 |
+| `static_dropout_0.06` | static | 0.06 | 4.6793 | 4.0054 | 0.3593 | 0.06 -> 0.06 -> 0.06 -> 0.06 -> 0.06 -> 0.06 |
+| `static_dropout_0.04` | static | 0.04 | 4.7293 | 4.0138 | 0.3685 | 0.04 -> 0.04 -> 0.04 -> 0.04 -> 0.04 -> 0.04 |
+| `static_dropout_0.02` | static | 0.02 | 4.8034 | 4.0387 | 0.3936 | 0.02 -> 0.02 -> 0.02 -> 0.02 -> 0.02 -> 0.02 |
+| `static_dropout_0` | static | 0.00 | 4.8976 | 4.0799 | 0.4155 | 0.00 -> 0.00 -> 0.00 -> 0.00 -> 0.00 -> 0.00 |
+
+## Stage Trajectory
+
+### Stage 0: 250,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `static_dropout_0.26` | 0.26 | 5.1995 | 3.5791 | 1.6204 | 3 |
+| `wikitext103_l20_interaction_8m` | 0.28 | 5.2075 | 3.6344 | 1.5731 | 3 |
+| `static_dropout_0.3` | 0.30 | 5.2098 | 3.6794 | 1.5305 | 3 |
+| `static_dropout_0.2` | 0.20 | 5.2249 | 3.4066 | 1.8183 | 3 |
+| `static_dropout_0.4` | 0.40 | 5.2395 | 3.9307 | 1.3088 | 3 |
+| `static_dropout_0.18` | 0.18 | 5.2455 | 3.3424 | 1.9031 | 3 |
+| `static_dropout_0.14` | 0.14 | 5.2995 | 3.1868 | 2.1128 | 3 |
+| `static_dropout_0.5` | 0.50 | 5.3229 | 4.1663 | 1.1566 | 3 |
+| `static_dropout_0.1` | 0.10 | 5.3769 | 3.0626 | 2.3143 | 3 |
+| `static_dropout_0.08` | 0.08 | 5.4166 | 2.9524 | 2.4642 | 3 |
+| `static_dropout_0.06` | 0.06 | 5.4825 | 2.8539 | 2.6287 | 3 |
+| `static_dropout_0.04` | 0.04 | 5.5554 | 2.7473 | 2.8080 | 3 |
+| `static_dropout_0.02` | 0.02 | 5.6479 | 2.6377 | 3.0102 | 3 |
+| `static_dropout_0` | 0.00 | 5.7526 | 2.5227 | 3.2299 | 3 |
+
+### Stage 1: 500,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `static_dropout_0.3` | 0.30 | 4.8010 | 3.4219 | 1.3791 | 3 |
+| `static_dropout_0.4` | 0.40 | 4.8128 | 3.6701 | 1.1428 | 3 |
+| `static_dropout_0.26` | 0.26 | 4.8161 | 3.3360 | 1.4801 | 3 |
+| `wikitext103_l20_interaction_8m` | 0.21 | 4.8170 | 3.2354 | 1.5816 | 3 |
+| `static_dropout_0.2` | 0.20 | 4.8619 | 3.1532 | 1.7086 | 3 |
+| `static_dropout_0.5` | 0.50 | 4.8756 | 3.9030 | 0.9726 | 3 |
+| `static_dropout_0.18` | 0.18 | 4.8887 | 3.0977 | 1.7911 | 3 |
+| `static_dropout_0.14` | 0.14 | 4.9586 | 2.9444 | 2.0142 | 3 |
+| `static_dropout_0.1` | 0.10 | 5.0803 | 2.7441 | 2.3362 | 3 |
+| `static_dropout_0.08` | 0.08 | 5.1505 | 2.6339 | 2.5166 | 3 |
+| `static_dropout_0.06` | 0.06 | 5.2229 | 2.5646 | 2.6582 | 3 |
+| `static_dropout_0.04` | 0.04 | 5.3193 | 2.3725 | 2.9468 | 3 |
+| `static_dropout_0.02` | 0.02 | 5.4543 | 2.2760 | 3.1783 | 3 |
+| `static_dropout_0` | 0.00 | 5.6136 | 2.1406 | 3.4730 | 3 |
+
+### Stage 2: 1,000,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `static_dropout_0.26` | 0.26 | 4.5072 | 3.4469 | 1.0603 | 3 |
+| `static_dropout_0.3` | 0.30 | 4.5081 | 3.5156 | 0.9925 | 3 |
+| `static_dropout_0.4` | 0.40 | 4.5223 | 3.6982 | 0.8241 | 3 |
+| `static_dropout_0.2` | 0.20 | 4.5348 | 3.3391 | 1.1957 | 3 |
+| `wikitext103_l20_interaction_8m` | 0.12 | 4.5390 | 3.2211 | 1.3178 | 3 |
+| `static_dropout_0.18` | 0.18 | 4.5495 | 3.2911 | 1.2584 | 3 |
+| `static_dropout_0.5` | 0.50 | 4.5927 | 3.8860 | 0.7068 | 3 |
+| `static_dropout_0.14` | 0.14 | 4.5951 | 3.1994 | 1.3956 | 3 |
+| `static_dropout_0.1` | 0.10 | 4.6638 | 3.0433 | 1.6205 | 3 |
+| `static_dropout_0.08` | 0.08 | 4.7107 | 2.9801 | 1.7306 | 3 |
+| `static_dropout_0.06` | 0.06 | 4.7718 | 2.8972 | 1.8746 | 3 |
+| `static_dropout_0.04` | 0.04 | 4.8411 | 2.7845 | 2.0566 | 3 |
+| `static_dropout_0.02` | 0.02 | 4.9402 | 2.6973 | 2.2429 | 3 |
+| `static_dropout_0` | 0.00 | 5.0625 | 2.5909 | 2.4716 | 3 |
+
+### Stage 3: 2,000,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `static_dropout_0.26` | 0.26 | 4.2754 | 3.6145 | 0.6609 | 3 |
+| `static_dropout_0.3` | 0.30 | 4.2811 | 3.6595 | 0.6217 | 3 |
+| `static_dropout_0.2` | 0.20 | 4.2873 | 3.5418 | 0.7455 | 3 |
+| `static_dropout_0.18` | 0.18 | 4.2884 | 3.5194 | 0.7690 | 3 |
+| `static_dropout_0.14` | 0.14 | 4.3103 | 3.4653 | 0.8450 | 3 |
+| `wikitext103_l20_interaction_8m` | 0.04 | 4.3108 | 3.3535 | 0.9574 | 3 |
+| `static_dropout_0.4` | 0.40 | 4.3221 | 3.7750 | 0.5472 | 3 |
+| `static_dropout_0.1` | 0.10 | 4.3471 | 3.4120 | 0.9351 | 3 |
+| `static_dropout_0.08` | 0.08 | 4.3878 | 3.3763 | 1.0115 | 3 |
+| `static_dropout_0.5` | 0.50 | 4.3918 | 3.9311 | 0.4607 | 3 |
+| `static_dropout_0.06` | 0.06 | 4.4182 | 3.3162 | 1.1020 | 3 |
+| `static_dropout_0.04` | 0.04 | 4.4542 | 3.2786 | 1.1756 | 3 |
+| `static_dropout_0.02` | 0.02 | 4.5106 | 3.2319 | 1.2787 | 3 |
+| `static_dropout_0` | 0.00 | 4.5802 | 3.1677 | 1.4126 | 3 |
+
+### Stage 4: 4,000,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `static_dropout_0.18` | 0.18 | 4.1088 | 3.6483 | 0.4605 | 3 |
+| `static_dropout_0.2` | 0.20 | 4.1111 | 3.6724 | 0.4387 | 3 |
+| `wikitext103_l20_interaction_8m` | 0.02 | 4.1125 | 3.5142 | 0.5983 | 3 |
+| `static_dropout_0.26` | 0.26 | 4.1149 | 3.7050 | 0.4100 | 3 |
+| `static_dropout_0.14` | 0.14 | 4.1155 | 3.6202 | 0.4953 | 3 |
+| `static_dropout_0.3` | 0.30 | 4.1198 | 3.7292 | 0.3906 | 3 |
+| `static_dropout_0.1` | 0.10 | 4.1266 | 3.5800 | 0.5466 | 3 |
+| `static_dropout_0.08` | 0.08 | 4.1517 | 3.5883 | 0.5634 | 3 |
+| `static_dropout_0.4` | 0.40 | 4.1681 | 3.8344 | 0.3337 | 3 |
+| `static_dropout_0.06` | 0.06 | 4.1752 | 3.5753 | 0.5999 | 3 |
+| `static_dropout_0.04` | 0.04 | 4.1922 | 3.5557 | 0.6365 | 3 |
+| `static_dropout_0.02` | 0.02 | 4.2288 | 3.5394 | 0.6894 | 3 |
+| `static_dropout_0.5` | 0.50 | 4.2408 | 3.9554 | 0.2854 | 3 |
+| `static_dropout_0` | 0.00 | 4.2967 | 3.5469 | 0.7498 | 3 |
+
+### Stage 5: 8,000,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `wikitext103_l20_interaction_8m` | 0.02 | 3.9572 | 3.5949 | 0.3623 | 3 |
+| `static_dropout_0.14` | 0.14 | 3.9756 | 3.6634 | 0.3122 | 3 |
+| `static_dropout_0.18` | 0.18 | 3.9758 | 3.6774 | 0.2984 | 3 |
+| `static_dropout_0.1` | 0.10 | 3.9785 | 3.6438 | 0.3347 | 3 |
+| `static_dropout_0.2` | 0.20 | 3.9807 | 3.6812 | 0.2995 | 3 |
+| `static_dropout_0.26` | 0.26 | 3.9842 | 3.7076 | 0.2766 | 3 |
+| `static_dropout_0.08` | 0.08 | 3.9893 | 3.6420 | 0.3473 | 3 |
+| `static_dropout_0.3` | 0.30 | 4.0015 | 3.7431 | 0.2584 | 3 |
+| `static_dropout_0.06` | 0.06 | 4.0054 | 3.6462 | 0.3593 | 3 |
+| `static_dropout_0.04` | 0.04 | 4.0138 | 3.6453 | 0.3685 | 3 |
+| `static_dropout_0.02` | 0.02 | 4.0387 | 3.6451 | 0.3936 | 3 |
+| `static_dropout_0.4` | 0.40 | 4.0503 | 3.8255 | 0.2248 | 3 |
+| `static_dropout_0` | 0.00 | 4.0799 | 3.6644 | 0.4155 | 3 |
+| `static_dropout_0.5` | 0.50 | 4.1252 | 3.9303 | 0.1949 | 3 |

runs/wikitext103_l20_8m_extrapolation_3seed/locked_stream/20260603-231952/config.json

@@ -0,0 +1,143 @@
+{
+  "args": {
+    "mode": "locked_stream",
+    "corpus": "data/wikitext103_raw/train-00001-of-00002.parquet",
+    "corpus_glob": null,
+    "text_column": "text",
+    "use_cached_data": false,
+    "output_dir": "runs/wikitext103_l20_8m_extrapolation_3seed",
+    "resume_from": null,
+    "cache_dir": ".cache/dropout_decay_wikitext103_8m",
+    "models": [
+      "L20_H8_D448=20x8x448"
+    ],
+    "seeds": [
+      1,
+      2,
+      3
+    ],
+    "token_limits": [
+      5000000
+    ],
+    "stream_token_caps": [
+      250000,
+      500000,
+      1000000,
+      2000000,
+      4000000,
+      8000000
+    ],
+    "val_tokens": 500000,
+    "allow_short_corpus": false,
+    "force_retokenize": false,
+    "vocab_size": 4096,
+    "tokenizer_train_chars": 10000000,
+    "block_size": 128,
+    "batch_size": 16,
+    "steps": 2000,
+    "stage_steps": 1000,
+    "dropout_rates": [
+      0.0,
+      0.02,
+      0.04,
+      0.06,
+      0.08,
+      0.1,
+      0.14,
+      0.18,
+      0.2,
+      0.26,
+      0.3,
+      0.4,
+      0.5
+    ],
+    "decays": [],
+    "anchor_decays": [
+      {
+        "name": "wikitext103_l20_interaction_8m",
+        "kind": "anchor_decay",
+        "initial": 0.284,
+        "final": 0.02,
+        "schedule": "log_prefix_anchor",
+        "decay_tokens": null,
+        "anchors": [
+          [
+            250000,
+            0.284
+          ],
+          [
+            500000,
+            0.211
+          ],
+          [
+            1000000,
+            0.116
+          ],
+          [
+            2000000,
+            0.036
+          ],
+          [
+            4000000,
+            0.02
+          ],
+          [
+            8000000,
+            0.02
+          ]
+        ]
+      }
+    ],
+    "decay_tokens": null,
+    "eval_batches": 64,
+    "train_eval_batches": 32,
+    "trace_eval_batches": 8,
+    "eval_every": 0,
+    "log_every": 250,
+    "lr": 0.0003,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "plateau_delta": 0.01,
+    "target_min_dropout": 0.1,
+    "min_nonzero_margin": 0.01,
+    "min_high_dropout_margin": 0.03,
+    "screen_early_stop": false,
+    "screen_prune_patience": 3,
+    "screen_prune_min_delta": 0.01
+  },
+  "mode": "locked_stream",
+  "seeds": [
+    1,
+    2,
+    3
+  ],
+  "models": [
+    {
+      "model_name": "L20_H8_D448",
+      "n_layer": 20,
+      "n_head": 8,
+      "n_embd": 448
+    }
+  ],
+  "device": "mps",
+  "torch": "2.12.0",
+  "python": "3.11.15 (main, Mar  3 2026, 00:52:57) [Clang 21.0.0 (clang-2100.0.123.102)]",
+  "mps_available": true,
+  "attribution": "Derived from Andrej Karpathy's nanochat project (https://github.com/karpathy/nanochat), MIT License, Copyright (c) 2025 Andrej Karpathy.",
+  "tokenizer_path": ".cache/dropout_decay_wikitext103_8m/tokenizer-v4096.json",
+  "encoded_path": ".cache/dropout_decay_wikitext103_8m/tokens-v4096-uint16.npy",
+  "train_tokens": 8000167,
+  "val_tokens": 500000,
+  "effective_token_limits": [
+    5000000
+  ],
+  "effective_stream_token_caps": [
+    250000,
+    500000,
+    1000000,
+    2000000,
+    4000000,
+    8000000
+  ],
+  "resume_from": null
+}

runs/wikitext103_l20_8m_extrapolation_3seed/locked_stream/20260603-231952/summary.csv

@@ -0,0 +1,85 @@
+run_mode,condition,condition_kind,stage,token_limit,model_name,n_layer,n_head,n_embd,parameters,dropout_initial,dropout_final,dropout_schedule,n,mean_train_eval_loss,std_train_eval_loss,mean_val_eval_loss,std_val_eval_loss,mean_generalization_gap,std_generalization_gap
+locked_stream,static_dropout_0,static,0,250000,L20_H8_D448,20,8,448,51838976,0.0,0.0,constant,3,2.522688180208206,0.012046513371440339,5.752627559006214,0.011719171106294125,3.229939378798008,0.007256438492685704
+locked_stream,static_dropout_0.02,static,0,250000,L20_H8_D448,20,8,448,51838976,0.02,0.02,constant,3,2.6376755237579346,0.03462739391375695,5.6479097083210945,0.02826141019452889,3.01023418456316,0.05961160036882495
+locked_stream,static_dropout_0.04,static,0,250000,L20_H8_D448,20,8,448,51838976,0.04,0.04,constant,3,2.7473360374569893,0.026691192066495402,5.555357076227665,0.023989807239235866,2.8080210387706757,0.017414919082363336
+locked_stream,static_dropout_0.06,static,0,250000,L20_H8_D448,20,8,448,51838976,0.06,0.06,constant,3,2.8538929745554924,0.03825703864710795,5.482547787328561,0.03919010180057007,2.628654812773069,0.0685458077125659
+locked_stream,static_dropout_0.08,static,0,250000,L20_H8_D448,20,8,448,51838976,0.08,0.08,constant,3,2.9524397626519203,0.0184523156676759,5.416609846055508,0.024148624120827047,2.4641700834035873,0.01119374365022041
+locked_stream,static_dropout_0.1,static,0,250000,L20_H8_D448,20,8,448,51838976,0.1,0.1,constant,3,3.0625818173090615,0.010675659247418713,5.376912613709767,0.026323925464375224,2.314330796400706,0.02550611170746628
+locked_stream,static_dropout_0.14,static,0,250000,L20_H8_D448,20,8,448,51838976,0.14,0.14,constant,3,3.1867669026056924,0.0057555326821074095,5.2995221093297005,0.010404610721067381,2.112755206724008,0.013071506587506998
+locked_stream,static_dropout_0.18,static,0,250000,L20_H8_D448,20,8,448,51838976,0.18,0.18,constant,3,3.3423681259155273,0.021193225094793457,5.245457495252292,0.009675764746749149,1.903089369336764,0.011907688339489637
+locked_stream,static_dropout_0.2,static,0,250000,L20_H8_D448,20,8,448,51838976,0.2,0.2,constant,3,3.4065689394871392,0.04052052190626222,5.224885210394859,0.023731347539789678,1.8183162709077199,0.04895795369382488
+locked_stream,static_dropout_0.26,static,0,250000,L20_H8_D448,20,8,448,51838976,0.26,0.26,constant,3,3.5791257843375206,0.005224810713378683,5.1994956483443575,0.013634536236337126,1.6203698640068371,0.018654078211625985
+locked_stream,static_dropout_0.3,static,0,250000,L20_H8_D448,20,8,448,51838976,0.3,0.3,constant,3,3.679355146984259,0.00826856353698578,5.2098226721088094,0.01692641493506561,1.5304675251245499,0.015872911313771706
+locked_stream,static_dropout_0.4,static,0,250000,L20_H8_D448,20,8,448,51838976,0.4,0.4,constant,3,3.9306640227635703,0.016420955354672497,5.23945789039135,0.018112675407487074,1.3087938676277797,0.027540943537256477
+locked_stream,static_dropout_0.5,static,0,250000,L20_H8_D448,20,8,448,51838976,0.5,0.5,constant,3,4.166328400373459,0.006971826642943488,5.322909223536651,0.006244832625531983,1.1565808231631916,0.007974360939987052
+locked_stream,wikitext103_l20_interaction_8m,anchor_decay,0,250000,L20_H8_D448,20,8,448,51838976,0.284,0.02,log_prefix_anchor,3,3.634432315826416,0.018852123639168248,5.2075243567427,0.013993230772716539,1.5730920409162839,0.021725783040448664
+locked_stream,static_dropout_0,static,1,500000,L20_H8_D448,20,8,448,51838976,0.0,0.0,constant,3,2.140629996856054,0.058683373071536656,5.613628188769023,0.026150140117080268,3.472998191912969,0.07894577865131966
+locked_stream,static_dropout_0.02,static,1,500000,L20_H8_D448,20,8,448,51838976,0.02,0.02,constant,3,2.2760410954554877,0.06067830158415421,5.454301168521245,0.044032068122271846,3.1782600730657578,0.04381451135909089
+locked_stream,static_dropout_0.04,static,1,500000,L20_H8_D448,20,8,448,51838976,0.04,0.04,constant,3,2.372536768515905,0.027702849840791847,5.319322774807612,0.057829866402633864,2.9467860062917075,0.08162012596085852
+locked_stream,static_dropout_0.06,static,1,500000,L20_H8_D448,20,8,448,51838976,0.06,0.06,constant,3,2.564632512629032,0.03876288534679488,5.222860977053642,0.039759720990741994,2.65822846442461,0.07298160041925844
+locked_stream,static_dropout_0.08,static,1,500000,L20_H8_D448,20,8,448,51838976,0.08,0.08,constant,3,2.6338843752940497,0.023523657823713004,5.150485873222351,0.025968269206372106,2.5166014979283013,0.005092342939797534
+locked_stream,static_dropout_0.1,static,1,500000,L20_H8_D448,20,8,448,51838976,0.1,0.1,constant,3,2.7440642615159354,0.042892858007543576,5.080255749324958,0.022523574927414018,2.3361914878090224,0.024552439402591733
+locked_stream,static_dropout_0.14,static,1,500000,L20_H8_D448,20,8,448,51838976,0.14,0.14,constant,3,2.9443837950627008,0.016903813772555912,4.958553483088811,0.01222546831066486,2.01416968802611,0.01622618225872982
+locked_stream,static_dropout_0.18,static,1,500000,L20_H8_D448,20,8,448,51838976,0.18,0.18,constant,3,3.0976694201429686,0.020826006717001213,4.888748288154602,0.028318214136380384,1.7910788680116336,0.0257235883631686
+locked_stream,static_dropout_0.2,static,1,500000,L20_H8_D448,20,8,448,51838976,0.2,0.2,constant,3,3.153231961031755,0.039295317381429676,4.861864025394122,0.02366101394973856,1.708632064362367,0.03540422538727223
+locked_stream,static_dropout_0.26,static,1,500000,L20_H8_D448,20,8,448,51838976,0.26,0.26,constant,3,3.336002856492996,0.014669757214797867,4.816100979844729,0.013781261483457954,1.480098123351733,0.018693787674010663
+locked_stream,static_dropout_0.3,static,1,500000,L20_H8_D448,20,8,448,51838976,0.3,0.3,constant,3,3.421916792790095,0.018702600633301866,4.8010413100322085,0.023497612891923463,1.3791245172421138,0.026975899703179488
+locked_stream,static_dropout_0.4,static,1,500000,L20_H8_D448,20,8,448,51838976,0.4,0.4,constant,3,3.6700536658366523,0.02213392401880642,4.8128317669034,0.022504488525005056,1.1427781010667484,0.04324231497479808
+locked_stream,static_dropout_0.5,static,1,500000,L20_H8_D448,20,8,448,51838976,0.5,0.5,constant,3,3.9030016312996545,0.01184978969532866,4.8755631272991495,0.020324776398940283,0.9725614959994952,0.01646016157646212
+locked_stream,wikitext103_l20_interaction_8m,anchor_decay,1,500000,L20_H8_D448,20,8,448,51838976,0.284,0.02,log_prefix_anchor,3,3.235363153119882,0.012289085905582853,4.816951217750709,0.0272929978517565,1.5815880646308262,0.016882527321068805
+locked_stream,static_dropout_0,static,2,1000000,L20_H8_D448,20,8,448,51838976,0.0,0.0,constant,3,2.5909490113457045,0.029235992190057708,5.062532270948092,0.05118772049512042,2.471583259602388,0.07779676564817914
+locked_stream,static_dropout_0.02,static,2,1000000,L20_H8_D448,20,8,448,51838976,0.02,0.02,constant,3,2.697271623959144,0.01780618960428723,4.940158794323604,0.04958085427041882,2.2428871703644595,0.06520842868826444
+locked_stream,static_dropout_0.04,static,2,1000000,L20_H8_D448,20,8,448,51838976,0.04,0.04,constant,3,2.7844761038819947,0.037160883479494485,4.841104706128438,0.03884578031524325,2.0566286022464433,0.04510334413865355
+locked_stream,static_dropout_0.06,static,2,1000000,L20_H8_D448,20,8,448,51838976,0.06,0.06,constant,3,2.897227168083191,0.014882485256954648,4.7718126227458315,0.02910551631275122,1.8745854546626408,0.016199845515269023
+locked_stream,static_dropout_0.08,static,2,1000000,L20_H8_D448,20,8,448,51838976,0.08,0.08,constant,3,2.980104481180509,0.0075327720639613936,4.710718693832557,0.044433103992299806,1.7306142126520474,0.04330883790999151
+locked_stream,static_dropout_0.1,static,2,1000000,L20_H8_D448,20,8,448,51838976,0.1,0.1,constant,3,3.04332917680343,0.023181039562370335,4.663791065414746,0.0146120110586173,1.6204618886113167,0.02657471538703674
+locked_stream,static_dropout_0.14,static,2,1000000,L20_H8_D448,20,8,448,51838976,0.14,0.14,constant,3,3.199449025094509,0.015657051496702042,4.59509019056956,0.02812935484713261,1.3956411654750507,0.042938003572152925
+locked_stream,static_dropout_0.18,static,2,1000000,L20_H8_D448,20,8,448,51838976,0.18,0.18,constant,3,3.2910933742920556,0.015624117477535813,4.5495016897718115,0.031322778388258574,1.2584083154797554,0.02515554430272105
+locked_stream,static_dropout_0.2,static,2,1000000,L20_H8_D448,20,8,448,51838976,0.2,0.2,constant,3,3.339050238331159,0.019325032793355925,4.534782186150551,0.024001627715366423,1.195731947819392,0.01463969335565847
+locked_stream,static_dropout_0.26,static,2,1000000,L20_H8_D448,20,8,448,51838976,0.26,0.26,constant,3,3.446868466834227,0.018553981201881585,4.507193277279536,0.022063926876841285,1.0603248104453087,0.03319665587467062
+locked_stream,static_dropout_0.3,static,2,1000000,L20_H8_D448,20,8,448,51838976,0.3,0.3,constant,3,3.5155644242962203,0.011106170217231122,4.508106569449107,0.02611450644245789,0.9925421451528867,0.036006024436070504
+locked_stream,static_dropout_0.4,static,2,1000000,L20_H8_D448,20,8,448,51838976,0.4,0.4,constant,3,3.6982057268420854,0.015132429963036823,4.522340419391791,0.023963846751753684,0.8241346925497055,0.022462081963965104
+locked_stream,static_dropout_0.5,static,2,1000000,L20_H8_D448,20,8,448,51838976,0.5,0.5,constant,3,3.8859883720676103,0.008189315788803003,4.592748900254567,0.020263423198831912,0.706760528186957,0.022984169718276097
+locked_stream,wikitext103_l20_interaction_8m,anchor_decay,2,1000000,L20_H8_D448,20,8,448,51838976,0.284,0.02,log_prefix_anchor,3,3.2211432283123336,0.020550667523204637,4.538959090908368,0.018682984974041847,1.317815862596035,0.016875805173084575
+locked_stream,static_dropout_0,static,3,2000000,L20_H8_D448,20,8,448,51838976,0.0,0.0,constant,3,3.167654238641262,0.024341413652540594,4.580211033423741,0.020969504501303075,1.4125567947824795,0.04323123227151149
+locked_stream,static_dropout_0.02,static,3,2000000,L20_H8_D448,20,8,448,51838976,0.02,0.02,constant,3,3.2319281697273254,0.0318578520144633,4.510588742792606,0.014778170863707843,1.278660573065281,0.03773124665918325
+locked_stream,static_dropout_0.04,static,3,2000000,L20_H8_D448,20,8,448,51838976,0.04,0.04,constant,3,3.278613212207953,0.01968095878917712,4.454178551832835,0.04205968065730655,1.1755653396248817,0.05859019060654376
+locked_stream,static_dropout_0.06,static,3,2000000,L20_H8_D448,20,8,448,51838976,0.06,0.06,constant,3,3.316213791569074,0.02106714206238047,4.41816928734382,0.009612100284860328,1.101955495774746,0.011566532133444081
+locked_stream,static_dropout_0.08,static,3,2000000,L20_H8_D448,20,8,448,51838976,0.08,0.08,constant,3,3.376338151594003,0.015657653330642115,4.387849149604638,0.01084493961851042,1.0115109980106354,0.02001736669312669
+locked_stream,static_dropout_0.1,static,3,2000000,L20_H8_D448,20,8,448,51838976,0.1,0.1,constant,3,3.4120371217528978,0.011280759699992426,4.3471392169594765,0.022335941067258433,0.9351020952065786,0.01813329358086262
+locked_stream,static_dropout_0.14,static,3,2000000,L20_H8_D448,20,8,448,51838976,0.14,0.14,constant,3,3.4652946144342422,0.013656485919907604,4.310343797008197,0.015764969286599612,0.8450491825739542,0.027414081397902502
+locked_stream,static_dropout_0.18,static,3,2000000,L20_H8_D448,20,8,448,51838976,0.18,0.18,constant,3,3.5193990344802537,0.0014500477613170093,4.288378739108642,0.018273244756788383,0.768979704628388,0.019649464219421767
+locked_stream,static_dropout_0.2,static,3,2000000,L20_H8_D448,20,8,448,51838976,0.2,0.2,constant,3,3.541793038447698,0.023457524101375065,4.287298730264108,0.013767705156301737,0.7455056918164095,0.027816846701527027
+locked_stream,static_dropout_0.26,static,3,2000000,L20_H8_D448,20,8,448,51838976,0.26,0.26,constant,3,3.614535095791022,0.009929903466015324,4.275388439496358,0.021064935851257058,0.6608533437053362,0.030625342797960836
+locked_stream,static_dropout_0.3,static,3,2000000,L20_H8_D448,20,8,448,51838976,0.3,0.3,constant,3,3.659476526081562,0.012890091698853969,4.2811290336151915,0.022609264828985894,0.6216525075336298,0.03531800089849792
+locked_stream,static_dropout_0.4,static,3,2000000,L20_H8_D448,20,8,448,51838976,0.4,0.4,constant,3,3.7749590848882995,0.0197308766100583,4.322125283380349,0.022446247112038055,0.5471661984920502,0.04070303851251657
+locked_stream,static_dropout_0.5,static,3,2000000,L20_H8_D448,20,8,448,51838976,0.5,0.5,constant,3,3.9310560822486877,0.020268312410434842,4.391768333812554,0.009756443195467564,0.4607122515638669,0.029730106756382994
+locked_stream,wikitext103_l20_interaction_8m,anchor_decay,3,2000000,L20_H8_D448,20,8,448,51838976,0.284,0.02,log_prefix_anchor,3,3.353454408546289,0.012473471900705016,4.310809387514989,0.02431455246903313,0.9573549789686998,0.03625047668895303
+locked_stream,static_dropout_0,static,4,4000000,L20_H8_D448,20,8,448,51838976,0.0,0.0,constant,3,3.546926826238632,0.02670416166926469,4.296710483729839,0.0057247741740965264,0.7497836574912071,0.03237969177342473
+locked_stream,static_dropout_0.02,static,4,4000000,L20_H8_D448,20,8,448,51838976,0.02,0.02,constant,3,3.5394452015558877,0.024712169628150194,4.228802674760421,0.010098413366159336,0.6893574732045332,0.031201170196353236
+locked_stream,static_dropout_0.04,static,4,4000000,L20_H8_D448,20,8,448,51838976,0.04,0.04,constant,3,3.555707737803459,0.02588011675298399,4.192242436110973,0.029697321500685617,0.6365346983075142,0.036162604350949173
+locked_stream,static_dropout_0.06,static,4,4000000,L20_H8_D448,20,8,448,51838976,0.06,0.06,constant,3,3.5753032167752585,0.038407960975789614,4.175153907388449,0.03643687448063786,0.5998506906131903,0.04174545525031613
+locked_stream,static_dropout_0.08,static,4,4000000,L20_H8_D448,20,8,448,51838976,0.08,0.08,constant,3,3.5883121291796365,0.02539414978841688,4.151699268569549,0.015796056131890928,0.5633871393899123,0.030907713531801637
+locked_stream,static_dropout_0.1,static,4,4000000,L20_H8_D448,20,8,448,51838976,0.1,0.1,constant,3,3.579965757826964,0.032049555721561575,4.126570571213961,0.0133651806554043,0.5466048133869966,0.029580274617644675
+locked_stream,static_dropout_0.14,static,4,4000000,L20_H8_D448,20,8,448,51838976,0.14,0.14,constant,3,3.620196128884951,0.01585509046509267,4.1154678364594774,0.021790165845493308,0.4952717075745265,0.029737006172367672
+locked_stream,static_dropout_0.18,static,4,4000000,L20_H8_D448,20,8,448,51838976,0.18,0.18,constant,3,3.6483276089032493,0.014119011897685165,4.108785220732291,0.016546231012667934,0.46045761182904243,0.02204054254861904
+locked_stream,static_dropout_0.2,static,4,4000000,L20_H8_D448,20,8,448,51838976,0.2,0.2,constant,3,3.672438850005468,0.02873844129851661,4.1111076436936855,0.022305646383573394,0.4386687936882178,0.025316209373280996
+locked_stream,static_dropout_0.26,static,4,4000000,L20_H8_D448,20,8,448,51838976,0.26,0.26,constant,3,3.704954795539379,0.025941372373462248,4.114915639162064,0.022402986584299307,0.4099608436226845,0.019573419295768002
+locked_stream,static_dropout_0.3,static,4,4000000,L20_H8_D448,20,8,448,51838976,0.3,0.3,constant,3,3.7292247662941613,0.030405938368494624,4.119805470108986,0.017481777488227677,0.3905807038148244,0.03199814714688443
+locked_stream,static_dropout_0.4,static,4,4000000,L20_H8_D448,20,8,448,51838976,0.4,0.4,constant,3,3.834391708175341,0.01898451795858093,4.168107481052478,0.015324779401614017,0.3337157728771369,0.02085053467259422
+locked_stream,static_dropout_0.5,static,4,4000000,L20_H8_D448,20,8,448,51838976,0.5,0.5,constant,3,3.9554246962070465,0.021195360064713176,4.240780795613925,0.019136266287497383,0.28535609940687817,0.017026194113975646
+locked_stream,wikitext103_l20_interaction_8m,anchor_decay,4,4000000,L20_H8_D448,20,8,448,51838976,0.284,0.02,log_prefix_anchor,3,3.514164961874485,0.029495373233394215,4.1124994705120725,0.024692308091551114,0.5983345086375872,0.036573472008392234
+locked_stream,static_dropout_0,static,5,8000000,L20_H8_D448,20,8,448,51838976,0.0,0.0,constant,3,3.664432925482591,0.049926091606292834,4.079922058929999,0.011796134919372616,0.4154891334474087,0.05403215538168701
+locked_stream,static_dropout_0.02,static,5,8000000,L20_H8_D448,20,8,448,51838976,0.02,0.02,constant,3,3.6450705379247665,0.05533125348789992,4.038677237927914,0.00913702886018555,0.3936067000031471,0.05659263729163474
+locked_stream,static_dropout_0.04,static,5,8000000,L20_H8_D448,20,8,448,51838976,0.04,0.04,constant,3,3.6452919393777847,0.04453364919198344,4.0137657423814135,0.010629652002763813,0.36847380300362903,0.05048476269960874
+locked_stream,static_dropout_0.06,static,5,8000000,L20_H8_D448,20,8,448,51838976,0.06,0.06,constant,3,3.6461652517318726,0.03634795371738759,4.005439349760612,0.004435936370745253,0.3592740980287393,0.04051150229295138
+locked_stream,static_dropout_0.08,static,5,8000000,L20_H8_D448,20,8,448,51838976,0.08,0.08,constant,3,3.642027479906877,0.047539696346292,3.989306505769491,0.007588172065890018,0.34727902586261433,0.04600499328236311
+locked_stream,static_dropout_0.1,static,5,8000000,L20_H8_D448,20,8,448,51838976,0.1,0.1,constant,3,3.64375047882398,0.04050110790515285,3.978465842703978,0.0018669190143304954,0.3347153638799985,0.040651349192261856
+locked_stream,static_dropout_0.14,static,5,8000000,L20_H8_D448,20,8,448,51838976,0.14,0.14,constant,3,3.6634238362312317,0.03014123109331319,3.9756230898201466,0.008723729300606414,0.31219925358891487,0.03866861327427125
+locked_stream,static_dropout_0.18,static,5,8000000,L20_H8_D448,20,8,448,51838976,0.18,0.18,constant,3,3.677407714227835,0.034788705011910816,3.975816690673431,0.008352800889171716,0.29840897644559544,0.039038235688036826
+locked_stream,static_dropout_0.2,static,5,8000000,L20_H8_D448,20,8,448,51838976,0.2,0.2,constant,3,3.681185041864713,0.03068077524813498,3.9807284039755664,0.0073537142949531824,0.2995433621108532,0.03551867902744893
+locked_stream,static_dropout_0.26,static,5,8000000,L20_H8_D448,20,8,448,51838976,0.26,0.26,constant,3,3.7075870782136917,0.036871453784834064,3.9841791801154613,0.008496865084735319,0.27659210190176964,0.04424811527550939
+locked_stream,static_dropout_0.3,static,5,8000000,L20_H8_D448,20,8,448,51838976,0.3,0.3,constant,3,3.743112308283647,0.03728021125731701,4.001548832903306,0.0057670621453762665,0.25843652461965877,0.04200903093110027
+locked_stream,static_dropout_0.4,static,5,8000000,L20_H8_D448,20,8,448,51838976,0.4,0.4,constant,3,3.8255134895443916,0.0403487115778625,4.050325091928244,0.008052479106177202,0.224811602383852,0.04248647822139951
+locked_stream,static_dropout_0.5,static,5,8000000,L20_H8_D448,20,8,448,51838976,0.5,0.5,constant,3,3.9302746479709945,0.02845892401475404,4.125173761198918,0.00442025606254341,0.19489911322792372,0.03286873379555716
+locked_stream,wikitext103_l20_interaction_8m,anchor_decay,5,8000000,L20_H8_D448,20,8,448,51838976,0.284,0.02,log_prefix_anchor,3,3.594912104308605,0.030775293066138493,3.957240684578816,0.008487626829204101,0.3623285802702109,0.022369662866505595

runs/wikitext103_l20_8m_extrapolation_3seed/locked_stream/20260603-231952/summary.json

@@ -0,0 +1,1850 @@
+[
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.522688180208206,
+    "std_train_eval_loss": 0.012046513371440339,
+    "mean_val_eval_loss": 5.752627559006214,
+    "std_val_eval_loss": 0.011719171106294125,
+    "mean_generalization_gap": 3.229939378798008,
+    "std_generalization_gap": 0.007256438492685704
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.6376755237579346,
+    "std_train_eval_loss": 0.03462739391375695,
+    "mean_val_eval_loss": 5.6479097083210945,
+    "std_val_eval_loss": 0.02826141019452889,
+    "mean_generalization_gap": 3.01023418456316,
+    "std_generalization_gap": 0.05961160036882495
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.7473360374569893,
+    "std_train_eval_loss": 0.026691192066495402,
+    "mean_val_eval_loss": 5.555357076227665,
+    "std_val_eval_loss": 0.023989807239235866,
+    "mean_generalization_gap": 2.8080210387706757,
+    "std_generalization_gap": 0.017414919082363336
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.8538929745554924,
+    "std_train_eval_loss": 0.03825703864710795,
+    "mean_val_eval_loss": 5.482547787328561,
+    "std_val_eval_loss": 0.03919010180057007,
+    "mean_generalization_gap": 2.628654812773069,
+    "std_generalization_gap": 0.0685458077125659
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.9524397626519203,
+    "std_train_eval_loss": 0.0184523156676759,
+    "mean_val_eval_loss": 5.416609846055508,
+    "std_val_eval_loss": 0.024148624120827047,
+    "mean_generalization_gap": 2.4641700834035873,
+    "std_generalization_gap": 0.01119374365022041
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.0625818173090615,
+    "std_train_eval_loss": 0.010675659247418713,
+    "mean_val_eval_loss": 5.376912613709767,
+    "std_val_eval_loss": 0.026323925464375224,
+    "mean_generalization_gap": 2.314330796400706,
+    "std_generalization_gap": 0.02550611170746628
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.1867669026056924,
+    "std_train_eval_loss": 0.0057555326821074095,
+    "mean_val_eval_loss": 5.2995221093297005,
+    "std_val_eval_loss": 0.010404610721067381,
+    "mean_generalization_gap": 2.112755206724008,
+    "std_generalization_gap": 0.013071506587506998
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.3423681259155273,
+    "std_train_eval_loss": 0.021193225094793457,
+    "mean_val_eval_loss": 5.245457495252292,
+    "std_val_eval_loss": 0.009675764746749149,
+    "mean_generalization_gap": 1.903089369336764,
+    "std_generalization_gap": 0.011907688339489637
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.2",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.2,
+    "dropout_final": 0.2,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.4065689394871392,
+    "std_train_eval_loss": 0.04052052190626222,
+    "mean_val_eval_loss": 5.224885210394859,
+    "std_val_eval_loss": 0.023731347539789678,
+    "mean_generalization_gap": 1.8183162709077199,
+    "std_generalization_gap": 0.04895795369382488
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.26",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.26,
+    "dropout_final": 0.26,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.5791257843375206,
+    "std_train_eval_loss": 0.005224810713378683,
+    "mean_val_eval_loss": 5.1994956483443575,
+    "std_val_eval_loss": 0.013634536236337126,
+    "mean_generalization_gap": 1.6203698640068371,
+    "std_generalization_gap": 0.018654078211625985
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.3",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.3,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.679355146984259,
+    "std_train_eval_loss": 0.00826856353698578,
+    "mean_val_eval_loss": 5.2098226721088094,
+    "std_val_eval_loss": 0.01692641493506561,
+    "mean_generalization_gap": 1.5304675251245499,
+    "std_generalization_gap": 0.015872911313771706
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.4",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.4,
+    "dropout_final": 0.4,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.9306640227635703,
+    "std_train_eval_loss": 0.016420955354672497,
+    "mean_val_eval_loss": 5.23945789039135,
+    "std_val_eval_loss": 0.018112675407487074,
+    "mean_generalization_gap": 1.3087938676277797,
+    "std_generalization_gap": 0.027540943537256477
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.5",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.5,
+    "dropout_final": 0.5,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 4.166328400373459,
+    "std_train_eval_loss": 0.006971826642943488,
+    "mean_val_eval_loss": 5.322909223536651,
+    "std_val_eval_loss": 0.006244832625531983,
+    "mean_generalization_gap": 1.1565808231631916,
+    "std_generalization_gap": 0.007974360939987052
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "wikitext103_l20_interaction_8m",
+    "condition_kind": "anchor_decay",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.284,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 3,
+    "mean_train_eval_loss": 3.634432315826416,
+    "std_train_eval_loss": 0.018852123639168248,
+    "mean_val_eval_loss": 5.2075243567427,
+    "std_val_eval_loss": 0.013993230772716539,
+    "mean_generalization_gap": 1.5730920409162839,
+    "std_generalization_gap": 0.021725783040448664
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.140629996856054,
+    "std_train_eval_loss": 0.058683373071536656,
+    "mean_val_eval_loss": 5.613628188769023,
+    "std_val_eval_loss": 0.026150140117080268,
+    "mean_generalization_gap": 3.472998191912969,
+    "std_generalization_gap": 0.07894577865131966
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.2760410954554877,
+    "std_train_eval_loss": 0.06067830158415421,
+    "mean_val_eval_loss": 5.454301168521245,
+    "std_val_eval_loss": 0.044032068122271846,
+    "mean_generalization_gap": 3.1782600730657578,
+    "std_generalization_gap": 0.04381451135909089
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.372536768515905,
+    "std_train_eval_loss": 0.027702849840791847,
+    "mean_val_eval_loss": 5.319322774807612,
+    "std_val_eval_loss": 0.057829866402633864,
+    "mean_generalization_gap": 2.9467860062917075,
+    "std_generalization_gap": 0.08162012596085852
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.564632512629032,
+    "std_train_eval_loss": 0.03876288534679488,
+    "mean_val_eval_loss": 5.222860977053642,
+    "std_val_eval_loss": 0.039759720990741994,
+    "mean_generalization_gap": 2.65822846442461,
+    "std_generalization_gap": 0.07298160041925844
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.6338843752940497,
+    "std_train_eval_loss": 0.023523657823713004,
+    "mean_val_eval_loss": 5.150485873222351,
+    "std_val_eval_loss": 0.025968269206372106,
+    "mean_generalization_gap": 2.5166014979283013,
+    "std_generalization_gap": 0.005092342939797534
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.7440642615159354,
+    "std_train_eval_loss": 0.042892858007543576,
+    "mean_val_eval_loss": 5.080255749324958,
+    "std_val_eval_loss": 0.022523574927414018,
+    "mean_generalization_gap": 2.3361914878090224,
+    "std_generalization_gap": 0.024552439402591733
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.9443837950627008,
+    "std_train_eval_loss": 0.016903813772555912,
+    "mean_val_eval_loss": 4.958553483088811,
+    "std_val_eval_loss": 0.01222546831066486,
+    "mean_generalization_gap": 2.01416968802611,
+    "std_generalization_gap": 0.01622618225872982
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.0976694201429686,
+    "std_train_eval_loss": 0.020826006717001213,
+    "mean_val_eval_loss": 4.888748288154602,
+    "std_val_eval_loss": 0.028318214136380384,
+    "mean_generalization_gap": 1.7910788680116336,
+    "std_generalization_gap": 0.0257235883631686
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.2",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.2,
+    "dropout_final": 0.2,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.153231961031755,
+    "std_train_eval_loss": 0.039295317381429676,
+    "mean_val_eval_loss": 4.861864025394122,
+    "std_val_eval_loss": 0.02366101394973856,
+    "mean_generalization_gap": 1.708632064362367,
+    "std_generalization_gap": 0.03540422538727223
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.26",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.26,
+    "dropout_final": 0.26,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.336002856492996,
+    "std_train_eval_loss": 0.014669757214797867,
+    "mean_val_eval_loss": 4.816100979844729,
+    "std_val_eval_loss": 0.013781261483457954,
+    "mean_generalization_gap": 1.480098123351733,
+    "std_generalization_gap": 0.018693787674010663
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.3",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.3,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.421916792790095,
+    "std_train_eval_loss": 0.018702600633301866,
+    "mean_val_eval_loss": 4.8010413100322085,
+    "std_val_eval_loss": 0.023497612891923463,
+    "mean_generalization_gap": 1.3791245172421138,
+    "std_generalization_gap": 0.026975899703179488
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.4",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.4,
+    "dropout_final": 0.4,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.6700536658366523,
+    "std_train_eval_loss": 0.02213392401880642,
+    "mean_val_eval_loss": 4.8128317669034,
+    "std_val_eval_loss": 0.022504488525005056,
+    "mean_generalization_gap": 1.1427781010667484,
+    "std_generalization_gap": 0.04324231497479808
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.5",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.5,
+    "dropout_final": 0.5,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.9030016312996545,
+    "std_train_eval_loss": 0.01184978969532866,
+    "mean_val_eval_loss": 4.8755631272991495,
+    "std_val_eval_loss": 0.020324776398940283,
+    "mean_generalization_gap": 0.9725614959994952,
+    "std_generalization_gap": 0.01646016157646212
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "wikitext103_l20_interaction_8m",
+    "condition_kind": "anchor_decay",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.284,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 3,
+    "mean_train_eval_loss": 3.235363153119882,
+    "std_train_eval_loss": 0.012289085905582853,
+    "mean_val_eval_loss": 4.816951217750709,
+    "std_val_eval_loss": 0.0272929978517565,
+    "mean_generalization_gap": 1.5815880646308262,
+    "std_generalization_gap": 0.016882527321068805
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.5909490113457045,
+    "std_train_eval_loss": 0.029235992190057708,
+    "mean_val_eval_loss": 5.062532270948092,
+    "std_val_eval_loss": 0.05118772049512042,
+    "mean_generalization_gap": 2.471583259602388,
+    "std_generalization_gap": 0.07779676564817914
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.697271623959144,
+    "std_train_eval_loss": 0.01780618960428723,
+    "mean_val_eval_loss": 4.940158794323604,
+    "std_val_eval_loss": 0.04958085427041882,
+    "mean_generalization_gap": 2.2428871703644595,
+    "std_generalization_gap": 0.06520842868826444
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.7844761038819947,
+    "std_train_eval_loss": 0.037160883479494485,
+    "mean_val_eval_loss": 4.841104706128438,
+    "std_val_eval_loss": 0.03884578031524325,
+    "mean_generalization_gap": 2.0566286022464433,
+    "std_generalization_gap": 0.04510334413865355
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.897227168083191,
+    "std_train_eval_loss": 0.014882485256954648,
+    "mean_val_eval_loss": 4.7718126227458315,
+    "std_val_eval_loss": 0.02910551631275122,
+    "mean_generalization_gap": 1.8745854546626408,
+    "std_generalization_gap": 0.016199845515269023
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 2.980104481180509,
+    "std_train_eval_loss": 0.0075327720639613936,
+    "mean_val_eval_loss": 4.710718693832557,
+    "std_val_eval_loss": 0.044433103992299806,
+    "mean_generalization_gap": 1.7306142126520474,
+    "std_generalization_gap": 0.04330883790999151
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.04332917680343,
+    "std_train_eval_loss": 0.023181039562370335,
+    "mean_val_eval_loss": 4.663791065414746,
+    "std_val_eval_loss": 0.0146120110586173,
+    "mean_generalization_gap": 1.6204618886113167,
+    "std_generalization_gap": 0.02657471538703674
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.199449025094509,
+    "std_train_eval_loss": 0.015657051496702042,
+    "mean_val_eval_loss": 4.59509019056956,
+    "std_val_eval_loss": 0.02812935484713261,
+    "mean_generalization_gap": 1.3956411654750507,
+    "std_generalization_gap": 0.042938003572152925
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.2910933742920556,
+    "std_train_eval_loss": 0.015624117477535813,
+    "mean_val_eval_loss": 4.5495016897718115,
+    "std_val_eval_loss": 0.031322778388258574,
+    "mean_generalization_gap": 1.2584083154797554,
+    "std_generalization_gap": 0.02515554430272105
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.2",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.2,
+    "dropout_final": 0.2,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.339050238331159,
+    "std_train_eval_loss": 0.019325032793355925,
+    "mean_val_eval_loss": 4.534782186150551,
+    "std_val_eval_loss": 0.024001627715366423,
+    "mean_generalization_gap": 1.195731947819392,
+    "std_generalization_gap": 0.01463969335565847
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.26",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.26,
+    "dropout_final": 0.26,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.446868466834227,
+    "std_train_eval_loss": 0.018553981201881585,
+    "mean_val_eval_loss": 4.507193277279536,
+    "std_val_eval_loss": 0.022063926876841285,
+    "mean_generalization_gap": 1.0603248104453087,
+    "std_generalization_gap": 0.03319665587467062
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.3",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.3,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.5155644242962203,
+    "std_train_eval_loss": 0.011106170217231122,
+    "mean_val_eval_loss": 4.508106569449107,
+    "std_val_eval_loss": 0.02611450644245789,
+    "mean_generalization_gap": 0.9925421451528867,
+    "std_generalization_gap": 0.036006024436070504
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.4",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.4,
+    "dropout_final": 0.4,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.6982057268420854,
+    "std_train_eval_loss": 0.015132429963036823,
+    "mean_val_eval_loss": 4.522340419391791,
+    "std_val_eval_loss": 0.023963846751753684,
+    "mean_generalization_gap": 0.8241346925497055,
+    "std_generalization_gap": 0.022462081963965104
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.5",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.5,
+    "dropout_final": 0.5,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.8859883720676103,
+    "std_train_eval_loss": 0.008189315788803003,
+    "mean_val_eval_loss": 4.592748900254567,
+    "std_val_eval_loss": 0.020263423198831912,
+    "mean_generalization_gap": 0.706760528186957,
+    "std_generalization_gap": 0.022984169718276097
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "wikitext103_l20_interaction_8m",
+    "condition_kind": "anchor_decay",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.284,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 3,
+    "mean_train_eval_loss": 3.2211432283123336,
+    "std_train_eval_loss": 0.020550667523204637,
+    "mean_val_eval_loss": 4.538959090908368,
+    "std_val_eval_loss": 0.018682984974041847,
+    "mean_generalization_gap": 1.317815862596035,
+    "std_generalization_gap": 0.016875805173084575
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.167654238641262,
+    "std_train_eval_loss": 0.024341413652540594,
+    "mean_val_eval_loss": 4.580211033423741,
+    "std_val_eval_loss": 0.020969504501303075,
+    "mean_generalization_gap": 1.4125567947824795,
+    "std_generalization_gap": 0.04323123227151149
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.2319281697273254,
+    "std_train_eval_loss": 0.0318578520144633,
+    "mean_val_eval_loss": 4.510588742792606,
+    "std_val_eval_loss": 0.014778170863707843,
+    "mean_generalization_gap": 1.278660573065281,
+    "std_generalization_gap": 0.03773124665918325
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.278613212207953,
+    "std_train_eval_loss": 0.01968095878917712,
+    "mean_val_eval_loss": 4.454178551832835,
+    "std_val_eval_loss": 0.04205968065730655,
+    "mean_generalization_gap": 1.1755653396248817,
+    "std_generalization_gap": 0.05859019060654376
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.316213791569074,
+    "std_train_eval_loss": 0.02106714206238047,
+    "mean_val_eval_loss": 4.41816928734382,
+    "std_val_eval_loss": 0.009612100284860328,
+    "mean_generalization_gap": 1.101955495774746,
+    "std_generalization_gap": 0.011566532133444081
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.376338151594003,
+    "std_train_eval_loss": 0.015657653330642115,
+    "mean_val_eval_loss": 4.387849149604638,
+    "std_val_eval_loss": 0.01084493961851042,
+    "mean_generalization_gap": 1.0115109980106354,
+    "std_generalization_gap": 0.02001736669312669
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.4120371217528978,
+    "std_train_eval_loss": 0.011280759699992426,
+    "mean_val_eval_loss": 4.3471392169594765,
+    "std_val_eval_loss": 0.022335941067258433,
+    "mean_generalization_gap": 0.9351020952065786,
+    "std_generalization_gap": 0.01813329358086262
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.4652946144342422,
+    "std_train_eval_loss": 0.013656485919907604,
+    "mean_val_eval_loss": 4.310343797008197,
+    "std_val_eval_loss": 0.015764969286599612,
+    "mean_generalization_gap": 0.8450491825739542,
+    "std_generalization_gap": 0.027414081397902502
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.5193990344802537,
+    "std_train_eval_loss": 0.0014500477613170093,
+    "mean_val_eval_loss": 4.288378739108642,
+    "std_val_eval_loss": 0.018273244756788383,
+    "mean_generalization_gap": 0.768979704628388,
+    "std_generalization_gap": 0.019649464219421767
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.2",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.2,
+    "dropout_final": 0.2,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.541793038447698,
+    "std_train_eval_loss": 0.023457524101375065,
+    "mean_val_eval_loss": 4.287298730264108,
+    "std_val_eval_loss": 0.013767705156301737,
+    "mean_generalization_gap": 0.7455056918164095,
+    "std_generalization_gap": 0.027816846701527027
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.26",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.26,
+    "dropout_final": 0.26,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.614535095791022,
+    "std_train_eval_loss": 0.009929903466015324,
+    "mean_val_eval_loss": 4.275388439496358,
+    "std_val_eval_loss": 0.021064935851257058,
+    "mean_generalization_gap": 0.6608533437053362,
+    "std_generalization_gap": 0.030625342797960836
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.3",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.3,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.659476526081562,
+    "std_train_eval_loss": 0.012890091698853969,
+    "mean_val_eval_loss": 4.2811290336151915,
+    "std_val_eval_loss": 0.022609264828985894,
+    "mean_generalization_gap": 0.6216525075336298,
+    "std_generalization_gap": 0.03531800089849792
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.4",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.4,
+    "dropout_final": 0.4,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.7749590848882995,
+    "std_train_eval_loss": 0.0197308766100583,
+    "mean_val_eval_loss": 4.322125283380349,
+    "std_val_eval_loss": 0.022446247112038055,
+    "mean_generalization_gap": 0.5471661984920502,
+    "std_generalization_gap": 0.04070303851251657
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.5",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.5,
+    "dropout_final": 0.5,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.9310560822486877,
+    "std_train_eval_loss": 0.020268312410434842,
+    "mean_val_eval_loss": 4.391768333812554,
+    "std_val_eval_loss": 0.009756443195467564,
+    "mean_generalization_gap": 0.4607122515638669,
+    "std_generalization_gap": 0.029730106756382994
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "wikitext103_l20_interaction_8m",
+    "condition_kind": "anchor_decay",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.284,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 3,
+    "mean_train_eval_loss": 3.353454408546289,
+    "std_train_eval_loss": 0.012473471900705016,
+    "mean_val_eval_loss": 4.310809387514989,
+    "std_val_eval_loss": 0.02431455246903313,
+    "mean_generalization_gap": 0.9573549789686998,
+    "std_generalization_gap": 0.03625047668895303
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.546926826238632,
+    "std_train_eval_loss": 0.02670416166926469,
+    "mean_val_eval_loss": 4.296710483729839,
+    "std_val_eval_loss": 0.0057247741740965264,
+    "mean_generalization_gap": 0.7497836574912071,
+    "std_generalization_gap": 0.03237969177342473
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.5394452015558877,
+    "std_train_eval_loss": 0.024712169628150194,
+    "mean_val_eval_loss": 4.228802674760421,
+    "std_val_eval_loss": 0.010098413366159336,
+    "mean_generalization_gap": 0.6893574732045332,
+    "std_generalization_gap": 0.031201170196353236
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.555707737803459,
+    "std_train_eval_loss": 0.02588011675298399,
+    "mean_val_eval_loss": 4.192242436110973,
+    "std_val_eval_loss": 0.029697321500685617,
+    "mean_generalization_gap": 0.6365346983075142,
+    "std_generalization_gap": 0.036162604350949173
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.5753032167752585,
+    "std_train_eval_loss": 0.038407960975789614,
+    "mean_val_eval_loss": 4.175153907388449,
+    "std_val_eval_loss": 0.03643687448063786,
+    "mean_generalization_gap": 0.5998506906131903,
+    "std_generalization_gap": 0.04174545525031613
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.5883121291796365,
+    "std_train_eval_loss": 0.02539414978841688,
+    "mean_val_eval_loss": 4.151699268569549,
+    "std_val_eval_loss": 0.015796056131890928,
+    "mean_generalization_gap": 0.5633871393899123,
+    "std_generalization_gap": 0.030907713531801637
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.579965757826964,
+    "std_train_eval_loss": 0.032049555721561575,
+    "mean_val_eval_loss": 4.126570571213961,
+    "std_val_eval_loss": 0.0133651806554043,
+    "mean_generalization_gap": 0.5466048133869966,
+    "std_generalization_gap": 0.029580274617644675
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.620196128884951,
+    "std_train_eval_loss": 0.01585509046509267,
+    "mean_val_eval_loss": 4.1154678364594774,
+    "std_val_eval_loss": 0.021790165845493308,
+    "mean_generalization_gap": 0.4952717075745265,
+    "std_generalization_gap": 0.029737006172367672
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.6483276089032493,
+    "std_train_eval_loss": 0.014119011897685165,
+    "mean_val_eval_loss": 4.108785220732291,
+    "std_val_eval_loss": 0.016546231012667934,
+    "mean_generalization_gap": 0.46045761182904243,
+    "std_generalization_gap": 0.02204054254861904
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.2",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.2,
+    "dropout_final": 0.2,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.672438850005468,
+    "std_train_eval_loss": 0.02873844129851661,
+    "mean_val_eval_loss": 4.1111076436936855,
+    "std_val_eval_loss": 0.022305646383573394,
+    "mean_generalization_gap": 0.4386687936882178,
+    "std_generalization_gap": 0.025316209373280996
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.26",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.26,
+    "dropout_final": 0.26,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.704954795539379,
+    "std_train_eval_loss": 0.025941372373462248,
+    "mean_val_eval_loss": 4.114915639162064,
+    "std_val_eval_loss": 0.022402986584299307,
+    "mean_generalization_gap": 0.4099608436226845,
+    "std_generalization_gap": 0.019573419295768002
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.3",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.3,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.7292247662941613,
+    "std_train_eval_loss": 0.030405938368494624,
+    "mean_val_eval_loss": 4.119805470108986,
+    "std_val_eval_loss": 0.017481777488227677,
+    "mean_generalization_gap": 0.3905807038148244,
+    "std_generalization_gap": 0.03199814714688443
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.4",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.4,
+    "dropout_final": 0.4,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.834391708175341,
+    "std_train_eval_loss": 0.01898451795858093,
+    "mean_val_eval_loss": 4.168107481052478,
+    "std_val_eval_loss": 0.015324779401614017,
+    "mean_generalization_gap": 0.3337157728771369,
+    "std_generalization_gap": 0.02085053467259422
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.5",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.5,
+    "dropout_final": 0.5,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.9554246962070465,
+    "std_train_eval_loss": 0.021195360064713176,
+    "mean_val_eval_loss": 4.240780795613925,
+    "std_val_eval_loss": 0.019136266287497383,
+    "mean_generalization_gap": 0.28535609940687817,
+    "std_generalization_gap": 0.017026194113975646
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "wikitext103_l20_interaction_8m",
+    "condition_kind": "anchor_decay",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.284,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 3,
+    "mean_train_eval_loss": 3.514164961874485,
+    "std_train_eval_loss": 0.029495373233394215,
+    "mean_val_eval_loss": 4.1124994705120725,
+    "std_val_eval_loss": 0.024692308091551114,
+    "mean_generalization_gap": 0.5983345086375872,
+    "std_generalization_gap": 0.036573472008392234
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": 5,
+    "token_limit": 8000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.664432925482591,
+    "std_train_eval_loss": 0.049926091606292834,
+    "mean_val_eval_loss": 4.079922058929999,
+    "std_val_eval_loss": 0.011796134919372616,
+    "mean_generalization_gap": 0.4154891334474087,
+    "std_generalization_gap": 0.05403215538168701
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": 5,
+    "token_limit": 8000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.6450705379247665,
+    "std_train_eval_loss": 0.05533125348789992,
+    "mean_val_eval_loss": 4.038677237927914,
+    "std_val_eval_loss": 0.00913702886018555,
+    "mean_generalization_gap": 0.3936067000031471,
+    "std_generalization_gap": 0.05659263729163474
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.04",
+    "condition_kind": "static",
+    "stage": 5,
+    "token_limit": 8000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.04,
+    "dropout_final": 0.04,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.6452919393777847,
+    "std_train_eval_loss": 0.04453364919198344,
+    "mean_val_eval_loss": 4.0137657423814135,
+    "std_val_eval_loss": 0.010629652002763813,
+    "mean_generalization_gap": 0.36847380300362903,
+    "std_generalization_gap": 0.05048476269960874
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": 5,
+    "token_limit": 8000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.6461652517318726,
+    "std_train_eval_loss": 0.03634795371738759,
+    "mean_val_eval_loss": 4.005439349760612,
+    "std_val_eval_loss": 0.004435936370745253,
+    "mean_generalization_gap": 0.3592740980287393,
+    "std_generalization_gap": 0.04051150229295138
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": 5,
+    "token_limit": 8000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.642027479906877,
+    "std_train_eval_loss": 0.047539696346292,
+    "mean_val_eval_loss": 3.989306505769491,
+    "std_val_eval_loss": 0.007588172065890018,
+    "mean_generalization_gap": 0.34727902586261433,
+    "std_generalization_gap": 0.04600499328236311
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": 5,
+    "token_limit": 8000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.64375047882398,
+    "std_train_eval_loss": 0.04050110790515285,
+    "mean_val_eval_loss": 3.978465842703978,
+    "std_val_eval_loss": 0.0018669190143304954,
+    "mean_generalization_gap": 0.3347153638799985,
+    "std_generalization_gap": 0.040651349192261856
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": 5,
+    "token_limit": 8000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.6634238362312317,
+    "std_train_eval_loss": 0.03014123109331319,
+    "mean_val_eval_loss": 3.9756230898201466,
+    "std_val_eval_loss": 0.008723729300606414,
+    "mean_generalization_gap": 0.31219925358891487,
+    "std_generalization_gap": 0.03866861327427125
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": 5,
+    "token_limit": 8000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.677407714227835,
+    "std_train_eval_loss": 0.034788705011910816,
+    "mean_val_eval_loss": 3.975816690673431,
+    "std_val_eval_loss": 0.008352800889171716,
+    "mean_generalization_gap": 0.29840897644559544,
+    "std_generalization_gap": 0.039038235688036826
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.2",
+    "condition_kind": "static",
+    "stage": 5,
+    "token_limit": 8000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.2,
+    "dropout_final": 0.2,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.681185041864713,
+    "std_train_eval_loss": 0.03068077524813498,
+    "mean_val_eval_loss": 3.9807284039755664,
+    "std_val_eval_loss": 0.0073537142949531824,
+    "mean_generalization_gap": 0.2995433621108532,
+    "std_generalization_gap": 0.03551867902744893
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.26",
+    "condition_kind": "static",
+    "stage": 5,
+    "token_limit": 8000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.26,
+    "dropout_final": 0.26,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.7075870782136917,
+    "std_train_eval_loss": 0.036871453784834064,
+    "mean_val_eval_loss": 3.9841791801154613,
+    "std_val_eval_loss": 0.008496865084735319,
+    "mean_generalization_gap": 0.27659210190176964,
+    "std_generalization_gap": 0.04424811527550939
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.3",
+    "condition_kind": "static",
+    "stage": 5,
+    "token_limit": 8000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.3,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.743112308283647,
+    "std_train_eval_loss": 0.03728021125731701,
+    "mean_val_eval_loss": 4.001548832903306,
+    "std_val_eval_loss": 0.0057670621453762665,
+    "mean_generalization_gap": 0.25843652461965877,
+    "std_generalization_gap": 0.04200903093110027
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.4",
+    "condition_kind": "static",
+    "stage": 5,
+    "token_limit": 8000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.4,
+    "dropout_final": 0.4,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.8255134895443916,
+    "std_train_eval_loss": 0.0403487115778625,
+    "mean_val_eval_loss": 4.050325091928244,
+    "std_val_eval_loss": 0.008052479106177202,
+    "mean_generalization_gap": 0.224811602383852,
+    "std_generalization_gap": 0.04248647822139951
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.5",
+    "condition_kind": "static",
+    "stage": 5,
+    "token_limit": 8000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.5,
+    "dropout_final": 0.5,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.9302746479709945,
+    "std_train_eval_loss": 0.02845892401475404,
+    "mean_val_eval_loss": 4.125173761198918,
+    "std_val_eval_loss": 0.00442025606254341,
+    "mean_generalization_gap": 0.19489911322792372,
+    "std_generalization_gap": 0.03286873379555716
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "wikitext103_l20_interaction_8m",
+    "condition_kind": "anchor_decay",
+    "stage": 5,
+    "token_limit": 8000000,
+    "model_name": "L20_H8_D448",
+    "n_layer": 20,
+    "n_head": 8,
+    "n_embd": 448,
+    "parameters": 51838976,
+    "dropout_initial": 0.284,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 3,
+    "mean_train_eval_loss": 3.594912104308605,
+    "std_train_eval_loss": 0.030775293066138493,
+    "mean_val_eval_loss": 3.957240684578816,
+    "std_val_eval_loss": 0.008487626829204101,
+    "mean_generalization_gap": 0.3623285802702109,
+    "std_generalization_gap": 0.022369662866505595
+  }
+]