ruhai-lin commited on
Commit
8bc871c
·
verified ·
1 Parent(s): 354d3f9

Restructure around the 30M BabyLM Strict-Small models: GDN-Bit, GDN, HGRN-Bit (T1/T4, 3 seeds), K ablation, ternary exports, eval tables

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +117 -17
  2. eval/.gitkeep +0 -0
  3. eval/all_runs.csv +36 -0
  4. eval/comparison.csv +11 -0
  5. eval/state_quant.csv +25 -0
  6. gdn-30m-t1/babylm-strict-small-100Mw-s1337/config.json +32 -0
  7. gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/args.json +31 -0
  8. gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/babylm_eval_summary.json +10 -0
  9. gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/blimp_report.txt +95 -0
  10. gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/blimp_supplement_report.txt +18 -0
  11. gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/dev_curve.log +14 -0
  12. gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/entity_tracking_report.txt +28 -0
  13. gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/final_summary.json +13 -0
  14. gdn-30m-t1/babylm-strict-small-100Mw-s1337/generation_config.json +7 -0
  15. gdn-30m-t1/babylm-strict-small-100Mw-s1337/model.safetensors +3 -0
  16. gdn-30m-t1/babylm-strict-small-100Mw-s1337/special_tokens_map.json +51 -0
  17. gdn-30m-t1/babylm-strict-small-100Mw-s1337/tokenizer.json +0 -0
  18. gdn-30m-t1/babylm-strict-small-100Mw-s1337/tokenizer_config.json +55 -0
  19. gdn-30m-t1/babylm-strict-small-100Mw-s42/config.json +32 -0
  20. gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/args.json +31 -0
  21. gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/babylm_eval_summary.json +10 -0
  22. gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/blimp_report.txt +95 -0
  23. gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/blimp_supplement_report.txt +18 -0
  24. gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/dev_curve.log +14 -0
  25. gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/entity_tracking_report.txt +28 -0
  26. gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/final_summary.json +13 -0
  27. gdn-30m-t1/babylm-strict-small-100Mw-s42/generation_config.json +7 -0
  28. gdn-30m-t1/babylm-strict-small-100Mw-s42/model.safetensors +3 -0
  29. gdn-30m-t1/babylm-strict-small-100Mw-s42/special_tokens_map.json +51 -0
  30. gdn-30m-t1/babylm-strict-small-100Mw-s42/tokenizer.json +0 -0
  31. gdn-30m-t1/babylm-strict-small-100Mw-s42/tokenizer_config.json +55 -0
  32. gdn-30m-t1/babylm-strict-small-100Mw-s7/config.json +32 -0
  33. gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/args.json +31 -0
  34. gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/babylm_eval_summary.json +10 -0
  35. gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/blimp_report.txt +95 -0
  36. gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/blimp_supplement_report.txt +18 -0
  37. gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/dev_curve.log +14 -0
  38. gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/entity_tracking_report.txt +28 -0
  39. gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/final_summary.json +13 -0
  40. gdn-30m-t1/babylm-strict-small-100Mw-s7/generation_config.json +7 -0
  41. gdn-30m-t1/babylm-strict-small-100Mw-s7/model.safetensors +3 -0
  42. gdn-30m-t1/babylm-strict-small-100Mw-s7/special_tokens_map.json +51 -0
  43. gdn-30m-t1/babylm-strict-small-100Mw-s7/tokenizer.json +0 -0
  44. gdn-30m-t1/babylm-strict-small-100Mw-s7/tokenizer_config.json +55 -0
  45. gdn-30m-t4/babylm-strict-small-100Mw-s1337/config.json +32 -0
  46. gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/args.json +31 -0
  47. gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/babylm_eval_summary.json +10 -0
  48. gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/blimp_report.txt +95 -0
  49. gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/blimp_supplement_report.txt +18 -0
  50. gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/dev_curve.log +14 -0
README.md CHANGED
@@ -1,21 +1,121 @@
1
  ---
2
- license: mit
3
- tags: [gated-deltanet, ternary, bitnet, looped-language-model, compute-in-memory]
 
 
 
 
 
4
  ---
5
  # GDN-Bit
6
 
7
- Gated DeltaNet with ternary weights (every projection, the FFN and the tied output head are BitLinear; the
8
- embedding stays float), ReLU² FFN, no short conv, one fused ternary input projection per mixer, and
9
- whole-stack looping. `T1` = no loop, `T4` = the layer stack run 4 times with shared weights.
10
- Code: `custom_models/gdnbit` in SSM_IMC_NATURE (branch `gdnbit`).
11
-
12
- | Folder | d / L / H | Params | Data | Tokens |
13
- |---|---|---|---|---|
14
- | `gdnbit-250k/{T1,T4}` | 64 / 4 / 2 | 266K | TinyStories (tok512) | 6.6B (100k steps) |
15
- | `gdnbit-1m/{T1,T4}` | 128 / 6 / 3 | 1.45M | TinyStories (tok512) | 6.6B (100k steps) |
16
- | `gdnbit-100m/{T1,T4}` | 768 / 9 / 9 | 99M | FineWeb-Edu sample-10BT (Llama 2 32K) | 10B |
17
- | `gdnbit-450m/{T1,T4}` | 1024 / 21 / 8 | 440M | FineWeb-Edu sample-100BT (Llama 3) | 100B |
18
-
19
- Each run folder (`<data>-<tokens>-s<seed>`) holds an HF checkpoint (`config.json`, `model.safetensors`,
20
- tokenizer files) and `eval/`. Stored BitLinear weights are the latent FP32 weights; ternary values are
21
- `round(clip(w / mean|w|, -1, 1))` per matrix.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ library_name: transformers
3
+ tags:
4
+ - gated-deltanet
5
+ - ternary
6
+ - bitnet
7
+ - looped-transformer
8
+ - babylm
9
  ---
10
  # GDN-Bit
11
 
12
+ GDN-Bit is a Gated DeltaNet whose linear layers are all ternary BitLinear (absmean ternary weights, int8 per-token activations). It optionally loops the whole layer stack T times. This repository holds the canonical 30M checkpoints trained on BabyLM 2026 Strict-Small:
13
+ - GDN-Bit
14
+ - full-precision GDN with the same architecture
15
+ - HGRN-Bit (the MatMul-free LM mixer)
16
+
17
+ Each comes without (T1) and with (T4) the loop, three seeds each. Folders are named `<arch>-30m[-k<dk>]-t<T>/babylm-strict-small-100Mw-s<seed>/`.
18
+
19
+ | Folder | Mixer | Linear weights | dk | T | Params | Peak LR |
20
+ |---|---|---|---:|---:|---:|---:|
21
+ | `gdnbit-30m-t1` | Gated DeltaNet | ternary | 64 | 1 | 30,470,472 | 2e-3 |
22
+ | `gdnbit-30m-t4` | Gated DeltaNet | ternary | 64 | 4 | 30,472,520 | 2e-3 |
23
+ | `gdn-30m-t1` | Gated DeltaNet | full precision | 64 | 1 | 30,453,064 | 1e-2 |
24
+ | `gdn-30m-t4` | Gated DeltaNet | full precision | 64 | 4 | 30,455,112 | 1e-2 |
25
+ | `hgrnbit-30m-t1` | HGRN (MatMul-free LM) | ternary | – | 1 | 30,566,304 | 2e-3 |
26
+ | `hgrnbit-30m-t4` | HGRN (MatMul-free LM) | ternary | – | 4 | 30,568,352 | 2e-3 |
27
+ | `gdnbit-30m-k16-t1` / `-t4` | Gated DeltaNet | ternary | 16 | 1 / 4 | 28,701,000 / 28,703,048 | 2e-3 |
28
+ | `gdnbit-30m-k32-t1` / `-t4` | Gated DeltaNet | ternary | 32 | 1 / 4 | 29,290,824 / 29,292,872 | 2e-3 |
29
+
30
+ **Architecture** (GDN and GDN-Bit):
31
+ - d=512, 6 layers, 6 heads, dk=64, dv=128, ReLU² FFN of width 2048, no short convolution.
32
+ - One fused input projection for q/k/v/g/a/b, sharing a single RMSNorm and int8 quantization.
33
+ - Tied float embedding and output head; the head is ternary in the Bit models.
34
+ - The K ablation keeps dv=128 and changes only dk.
35
+
36
+ **HGRN-Bit:** d=512, 7 layers, SwiGLU 1376, matched to ~30.5M parameters.
37
+
38
+ **Loop:** the T4 models apply the whole stack 4 times with weights shared across passes, as h ← Stack(h) + r_t ⊙ h_in. The per-pass vector r_t is fp32 and zero-initialized. Every pass starts its recurrent state from zero.
39
+
40
+ ## Loading
41
+
42
+ The weights load with Transformers once the model classes are registered. This needs the `custom_models` package of the project code at commit `1bf7e65` (not public yet), plus `flash-linear-attention==0.5.1` and `transformers==4.57.3`; trained with torch 2.13. Plain Transformers does not recognize the model types `gdnbit` and `mmfloop`.
43
+
44
+ ```python
45
+ import custom_models # registers gdnbit (GDN, GDN-Bit) and mmfloop (HGRN-Bit)
46
+ from transformers import AutoModelForCausalLM, AutoTokenizer
47
+
48
+ repo = "ruhai-lin/GDNBit"
49
+ run = "gdnbit-30m-t4/babylm-strict-small-100Mw-s42"
50
+ tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=run)
51
+ model = AutoModelForCausalLM.from_pretrained(repo, subfolder=run, dtype="float32")
52
+ ```
53
+
54
+ **Warning for T4 models:** the HF recurrent cache is keyed by layer index only, so the 4 passes would overwrite each other's state. Score full sequences without a cache, or keep one state per (pass, layer) yourself. Do not use `generate()` with `use_cache=True`.
55
+
56
+ ## Training, evaluation and budgets
57
+
58
+ **Training**
59
+ - **Data:** BabyLM 2026 Strict-Small, 10M words, tokenized with the official BabyLM 16,384-token tokenizer, which is included in every folder.
60
+ - **Budget:** 10 epochs = 100M words processed, 10,320 steps at batch 32 × 512 tokens.
61
+ - **Optimizer:** AdamW (β 0.9/0.95, weight decay 0.1, gradient clip 1.0), cosine schedule to 0 with 500 warmup steps, bf16 autocast.
62
+ - **Learning rate:** the peak LR was swept separately for ternary and full-precision models (see `eval/all_runs.csv`).
63
+
64
+ **Checkpoint selection:** a checkpoint is saved every 10M words. Each folder holds the checkpoint with the lowest full dev NLL; `eval/final_summary.json` gives `best_words_m`.
65
+
66
+ **Metrics**
67
+ - Dev/test NLL is per token, FP32, on consecutive BOS-prefixed 512-token windows over the full BabyLM dev/test sets. bpw is bits per word.
68
+ - BLiMP, BLiMP supplement and entity tracking come from the official BabyLM evaluation pipeline (zero-shot, causal, filtered sets).
69
+ - Entity tracking varies strongly across seeds; we do not draw conclusions from it.
70
+
71
+ ## Results (mean ± std over seeds 7, 42, 1337)
72
+
73
+ | Model | Dev NLL | Test NLL | BLiMP | BLiMP suppl. |
74
+ |---|---:|---:|---:|---:|
75
+ | gdnbit-30m-t1 | 3.554 ± 0.009 | 3.483 ± 0.013 | 69.8 ± 1.3 | 58.9 ± 1.0 |
76
+ | **gdnbit-30m-t4** | **3.531 ± 0.050** | **3.454 ± 0.063** | **71.2 ± 0.0** | **60.6 ± 0.5** |
77
+ | gdn-30m-t1 | 3.431 ± 0.040 | 3.357 ± 0.054 | 70.6 ± 0.7 | 62.1 ± 2.4 |
78
+ | gdn-30m-t4 | 3.365 ± 0.010 | 3.274 ± 0.016 | 71.7 ± 0.4 | 60.7 ± 0.8 |
79
+ | hgrnbit-30m-t1 | 3.571 ± 0.032 | 3.476 ± 0.037 | 70.3 ± 0.7 | 58.7 ± 0.3 |
80
+ | hgrnbit-30m-t4 | 3.562 ± 0.018 | 3.468 ± 0.025 | 70.2 ± 0.8 | 60.2 ± 0.6 |
81
+ | gdnbit-30m-k16-t1 | 3.624 ± 0.019 | 3.562 ± 0.027 | 69.5 ± 0.5 | 60.6 ± 1.5 |
82
+ | gdnbit-30m-k16-t4 | 3.535 ± 0.019 | 3.454 ± 0.027 | 70.3 ± 0.0 | 59.8 ± 0.9 |
83
+ | gdnbit-30m-k32-t1 | 3.574 ± 0.021 | 3.499 ± 0.030 | 69.9 ± 0.6 | 59.5 ± 0.8 |
84
+ | gdnbit-30m-k32-t4 | 3.524 ± 0.015 | 3.444 ± 0.018 | 70.8 ± 0.4 | 60.1 ± 2.9 |
85
+
86
+ **Result files**
87
+ - `eval/comparison.csv`: all metrics with mean and std, including bpw and entity tracking.
88
+ - `eval/all_runs.csv`: every run-seed, including the full-precision LR sweep points, whose weights are not published.
89
+ - `eval/state_quant.csv`: dev NLL when the recurrent state is stored at reduced precision after every token (seed 1337). An int8 state with one scale per row costs at most 0.004 nats for every model.
90
+
91
+ ## Files in each run folder
92
+
93
+ - **HF checkpoint:** `config.json`, `model.safetensors` (fp32 latent weights), the tokenizer files and `generation_config.json`.
94
+ - **`eval/`:**
95
+ - `final_summary.json`: dev/test NLL and bpw, BLiMP, supplement, entity tracking, best checkpoint
96
+ - `babylm_eval_summary.json` and per-task reports: `blimp_report.txt`, `blimp_supplement_report.txt`, `entity_tracking_report.txt` (per-field and per-phenomenon accuracy)
97
+ - `args.json`: training arguments
98
+ - `dev_curve.log`: dev NLL at every epoch
99
+ - **`ternary/`** (`gdnbit-30m-t1` and `gdnbit-30m-t4` only): deployment weights. See below.
100
+
101
+ ## Ternary deployment weights (`ternary/`)
102
+
103
+ The latent fp32 weights must not be re-quantized independently. About a fifth of them sit exactly on the rounding threshold |w| = 0.5·mean|w|, so a different floating-point summation order (for example CPU vs GPU) flips their ternary level and changes the model.
104
+
105
+ `ternary/ternary.safetensors` stores, for every BitLinear `<module>`, the levels and scale as computed on CUDA by the training kernel:
106
+ - `<module>.levels`: int8 values in {-1, 0, 1}, shape [out, in]
107
+ - `<module>.scale`: fp32 scalar, mean|W|
108
+
109
+ The forward pass is
110
+
111
+ y = Q8(RMSNorm(x; eps 1e-6) * norm.weight) @ (levels * scale)^T,
112
+
113
+ where Q8 is per-token absmax int8 quantization (127 / max|x|). The RMSNorm weights, embedding, A_log, dt_bias, output norm and loop weights r_t are the fp32 tensors in `model.safetensors`.
114
+
115
+ The fused attention input projection `model.layers.<i>.attn.in_proj` has 2316 output rows. In order: q 384 | k 384 | v 768 | g 768 | a 6 | b 6.
116
+ - a (decay) and b (write strength) are its last 12 rows. They share the matrix's single scale.
117
+ - After the projection: decay = exp(−exp(A_log)·softplus(a + dt_bias)), β = sigmoid(b), and q and k are L2-normalized.
118
+
119
+ The tied `lm_head` levels are quantized from the float embedding.
120
+
121
+ `ternary/verify.json` records the check: dev NLL of the model rebuilt from these files minus the native model's.
eval/.gitkeep DELETED
File without changes
eval/all_runs.csv ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ folder,arch,weights,K,T,lr,seed,params,best_words_m,dev_nll,dev_bpw,test_nll,test_bpw,blimp,blimp_supplement,entity_tracking
2
+ gdn-30m-t1/babylm-strict-small-100Mw-s1337,gdn,fp,64,1,1e-2,1337,30453064,70,3.3981919687350586,8.36140506397954,3.306020510774677,8.263090470345386,71.44,63.82,27.8
3
+ gdn-30m-t1/babylm-strict-small-100Mw-s42,gdn,fp,64,1,1e-2,42,30453064,80,3.4753230730271207,8.55118963529028,3.4128869393076893,8.53019315901082,70.37,63.11,26.26
4
+ gdn-30m-t1/babylm-strict-small-100Mw-s7,gdn,fp,64,1,1e-2,7,30453064,70,3.4189122823832707,8.412388333041262,3.3520635961237866,8.378170875483095,70.09,59.33,18.87
5
+ gdn-30m-t4/babylm-strict-small-100Mw-s1337,gdn,fp,64,4,1e-2,1337,30455112,80,3.366958815790573,8.28455447825725,3.2683713757282624,8.168989962497697,72.08,61.02,21.1
6
+ gdn-30m-t4/babylm-strict-small-100Mw-s42,gdn,fp,64,4,1e-2,42,30455112,70,3.374812731624426,8.303879393456109,3.29265292421458,8.229679432284291,71.83,61.41,23.75
7
+ gdn-30m-t4/babylm-strict-small-100Mw-s7,gdn,fp,64,4,1e-2,7,30455112,70,3.3545820357881553,8.254100851169639,3.262191874658429,8.153544874896088,71.31,59.79,25.4
8
+ gdnbit-30m-k16-t1/babylm-strict-small-100Mw-s1337,gdnbit,ternary,16,1,2e-3,1337,28701000,100,3.619888644860621,8.906899764537533,3.556276118268278,8.888581062037547,69.8,61.49,22.15
9
+ gdnbit-30m-k16-t1/babylm-strict-small-100Mw-s42,gdnbit,ternary,16,1,2e-3,42,28701000,100,3.607290638466565,8.875901799905618,3.5391904586529166,8.845877046533971,69.76,58.87,27.09
10
+ gdnbit-30m-k16-t1/babylm-strict-small-100Mw-s7,gdnbit,ternary,16,1,2e-3,7,28701000,100,3.64464632006933,8.96781714435805,3.5920976477030937,8.978113639812348,68.94,61.4,14.86
11
+ gdnbit-30m-k16-t4/babylm-strict-small-100Mw-s1337,gdnbit,ternary,16,4,2e-3,1337,28703048,100,3.5281349707081193,8.68113569860013,3.434116658192837,8.583254864840002,70.31,59.28,13.81
12
+ gdnbit-30m-k16-t4/babylm-strict-small-100Mw-s42,gdnbit,ternary,16,4,2e-3,42,28703048,100,3.5567442233889603,8.751530059025654,3.484899289349601,8.710181323463877,70.29,60.85,17.68
13
+ gdnbit-30m-k16-t4/babylm-strict-small-100Mw-s7,gdnbit,ternary,16,4,2e-3,7,28703048,100,3.5210507137386737,8.663704563854408,3.4416328390107687,8.602040859024743,70.27,59.26,38.15
14
+ gdnbit-30m-k32-t1/babylm-strict-small-100Mw-s1337,gdnbit,ternary,32,1,2e-3,1337,29290824,100,3.5526512175453218,8.741459032990996,3.4671261010588403,8.665758894046272,70.37,58.57,15.13
15
+ gdnbit-30m-k32-t1/babylm-strict-small-100Mw-s42,gdnbit,ternary,32,1,2e-3,42,29290824,80,3.5935469612926596,8.842084860492873,3.525837901921341,8.812503574129963,69.21,59.95,25.95
16
+ gdnbit-30m-k32-t1/babylm-strict-small-100Mw-s7,gdnbit,ternary,32,1,2e-3,7,29290824,100,3.575910410070147,8.79868932838056,3.504925559433028,8.760235121056365,70.09,60.05,14.69
17
+ gdnbit-30m-k32-t4/babylm-strict-small-100Mw-s1337,gdnbit,ternary,32,4,2e-3,1337,29292872,100,3.5175602904714194,8.655116219508129,3.4511007423261355,8.625705001883018,71.25,57.32,14.31
18
+ gdnbit-30m-k32-t4/babylm-strict-small-100Mw-s42,gdnbit,ternary,32,4,2e-3,42,29292872,90,3.541202584295625,8.713289152975273,3.4586077169030593,8.644467985925475,70.9,63.09,35.51
19
+ gdnbit-30m-k32-t4/babylm-strict-small-100Mw-s7,gdnbit,ternary,32,4,2e-3,7,29292872,100,3.51420600384187,8.646862845517374,3.423609165503013,8.556992365127874,70.36,59.93,17.46
20
+ gdnbit-30m-t1/babylm-strict-small-100Mw-s1337,gdnbit,ternary,64,1,2e-3,1337,30470472,80,3.5588962936555246,8.756825325270306,3.482154962818182,8.703322134800231,70.62,57.87,14.69
21
+ gdnbit-30m-t1/babylm-strict-small-100Mw-s42,gdnbit,ternary,64,1,2e-3,42,30470472,80,3.543970719157588,8.72010027402598,3.470840721222194,8.675043241306891,68.37,59.11,20.38
22
+ gdnbit-30m-t1/babylm-strict-small-100Mw-s7,gdnbit,ternary,64,1,2e-3,7,30470472,100,3.5586369049031816,8.756187087511128,3.496725137431466,8.7397389297372,70.47,59.85,17.66
23
+ gdnbit-30m-t4/babylm-strict-small-100Mw-s1337,gdnbit,ternary,64,4,2e-3,1337,30472520,100,3.474096335259005,8.54817119151757,3.382425710637615,8.454058153944041,71.23,60.52,14.34
24
+ gdnbit-30m-t4/babylm-strict-small-100Mw-s42,gdnbit,ternary,64,4,2e-3,42,30472520,100,3.550867301726804,8.73706962742006,3.4809499018719006,8.700310197158299,71.22,60.08,14.03
25
+ gdnbit-30m-t4/babylm-strict-small-100Mw-s7,gdnbit,ternary,64,4,2e-3,7,30472520,100,3.5671160822195507,8.777050486872836,3.5002731352577756,8.74860681997921,71.18,61.17,39.21
26
+ hgrnbit-30m-t1/babylm-strict-small-100Mw-s1337,hgrnbit,ternary,,1,2e-3,1337,30566304,100,3.5380620995440593,8.705561848177107,3.4382046179562886,8.593472339672491,70.87,58.38,15.01
27
+ hgrnbit-30m-t1/babylm-strict-small-100Mw-s42,hgrnbit,ternary,,1,2e-3,42,30566304,100,3.601345236800297,8.86127287015206,3.5120533336162523,8.778050329018996,70.33,58.92,16.8
28
+ hgrnbit-30m-t1/babylm-strict-small-100Mw-s7,hgrnbit,ternary,,1,2e-3,7,30566304,100,3.5729909604587577,8.791505890543823,3.4790666710123377,8.695603236956932,69.56,58.66,17.99
29
+ hgrnbit-30m-t4/babylm-strict-small-100Mw-s1337,hgrnbit,ternary,,4,2e-3,1337,30568352,100,3.5581238514228573,8.754924696213655,3.463659601617893,8.657094701430832,69.36,59.91,19.14
30
+ hgrnbit-30m-t4/babylm-strict-small-100Mw-s42,hgrnbit,ternary,,4,2e-3,42,30568352,100,3.581190567047378,8.811681393482889,3.494257059901931,8.733570199734496,70.29,60.89,29.59
31
+ hgrnbit-30m-t4/babylm-strict-small-100Mw-s7,hgrnbit,ternary,,4,2e-3,7,30568352,100,3.5467213090061147,8.726868224777439,3.4455132920743003,8.61173969018002,71.02,59.9,37.79
32
+ "(LR sweep point, weights not published)",gdn,fp,64,1,2.4e-3,1337,30453064,40,3.4853981734489987,8.575979875648287,3.4047506315046543,8.509857215161729,70.14,59.01,23.04
33
+ "(LR sweep point, weights not published)",gdn,fp,64,1,4.8e-3,1337,30453064,50,3.455795997145237,8.50314238173142,3.382229113072728,8.453566776634185,70.71,57.61,25.41
34
+ "(LR sweep point, weights not published)",gdn,fp,64,1,9.6e-3,1337,30453064,60,3.4199902462985396,8.41504071201892,3.3342176780401367,8.333566664719438,70.61,59.78,19.03
35
+ "(LR sweep point, weights not published)",gdn,fp,64,1,9.6e-3,42,30453064,80,3.4356521429922413,8.453577517335127,3.3598007936253174,8.397509280291448,69.5,61.38,20.99
36
+ "(LR sweep point, weights not published)",gdn,fp,64,1,9.6e-3,7,30453064,70,3.414198370754207,8.400789540233488,3.344727789943151,8.359835710911081,70.3,62.08,29.93
eval/comparison.csv ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ model,params,lr,seeds,dev_nll_mean,dev_nll_std,test_nll_mean,test_nll_std,dev_bpw_mean,dev_bpw_std,test_bpw_mean,test_bpw_std,blimp_mean,blimp_std,blimp_supplement_mean,blimp_supplement_std,entity_tracking_mean,entity_tracking_std
2
+ gdnbit-30m-t1,30470472,2e-3,7 42 1337,3.5538,0.0085,3.4832,0.013,8.7444,0.021,8.706,0.0324,69.82,1.258,58.9433,1.0005,17.5767,2.8459
3
+ gdnbit-30m-t4,30472520,2e-3,7 42 1337,3.5307,0.0497,3.4545,0.0632,8.6874,0.1222,8.6343,0.158,71.21,0.0265,60.59,0.5484,22.5267,14.449
4
+ gdn-30m-t1,30453064,1e-2,7 42 1337,3.4308,0.0399,3.357,0.0536,8.4417,0.0982,8.3905,0.134,70.6333,0.7125,62.0867,2.4136,24.31,4.7737
5
+ gdn-30m-t4,30455112,1e-2,7 42 1337,3.3655,0.0102,3.2744,0.0161,8.2808,0.0251,8.1841,0.0402,71.74,0.3928,60.74,0.8455,23.4167,2.1693
6
+ hgrnbit-30m-t1,30566304,2e-3,7 42 1337,3.5708,0.0317,3.4764,0.037,8.7861,0.078,8.689,0.0925,70.2533,0.6584,58.6533,0.2701,16.6,1.5
7
+ hgrnbit-30m-t4,30568352,2e-3,7 42 1337,3.562,0.0176,3.4678,0.0246,8.7645,0.0432,8.6675,0.0616,70.2233,0.832,60.2333,0.5687,28.84,9.3476
8
+ gdnbit-30m-k16-t1,28701000,2e-3,7 42 1337,3.6239,0.019,3.5625,0.027,8.9169,0.0468,8.9042,0.0675,69.5,0.4854,60.5867,1.4874,21.3667,6.1525
9
+ gdnbit-30m-k16-t4,28703048,2e-3,7 42 1337,3.5353,0.0189,3.4535,0.0274,8.6988,0.0465,8.6318,0.0685,70.29,0.02,59.7967,0.9123,23.2133,13.0795
10
+ gdnbit-30m-k32-t1,29290824,2e-3,7 42 1337,3.574,0.0205,3.4993,0.0298,8.7941,0.0505,8.7462,0.0744,69.89,0.6053,59.5233,0.8271,18.59,6.3777
11
+ gdnbit-30m-k32-t4,29292872,2e-3,7 42 1337,3.5243,0.0147,3.4444,0.0184,8.6718,0.0362,8.6091,0.0461,70.8367,0.4484,60.1133,2.8894,22.4267,11.4394
eval/state_quant.csv ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ model,seed,checkpoint_words_m,state_format,dev_nll,delta_vs_fp32_state
2
+ gdnbit-30m-t1,1337,80,fp32,3.558262,0.0
3
+ gdnbit-30m-t1,1337,80,int8_row,3.562203,0.003941
4
+ gdnbit-30m-t1,1337,80,int8_tensor,3.5754,0.017139
5
+ gdnbit-30m-t1,1337,80,int4_row,3.61204,0.053778
6
+ gdnbit-30m-t4,1337,final,fp32,3.473066,0.0
7
+ gdnbit-30m-t4,1337,final,int8_row,3.474653,0.001588
8
+ gdnbit-30m-t4,1337,final,int8_tensor,3.475414,0.002349
9
+ gdnbit-30m-t4,1337,final,int4_row,3.516496,0.043431
10
+ hgrnbit-30m-t1,1337,final,fp32,3.536096,0.0
11
+ hgrnbit-30m-t1,1337,final,int8_row,3.536806,0.00071
12
+ hgrnbit-30m-t1,1337,final,int8_tensor,3.536806,0.00071
13
+ hgrnbit-30m-t1,1337,final,int4_row,3.736917,0.200821
14
+ hgrnbit-30m-t4,1337,final,fp32,3.556619,0.0
15
+ hgrnbit-30m-t4,1337,final,int8_row,3.558132,0.001514
16
+ hgrnbit-30m-t4,1337,final,int8_tensor,3.558132,0.001514
17
+ hgrnbit-30m-t4,1337,final,int4_row,3.802597,0.245978
18
+ gdn-30m-t1,1337,70,fp32,3.398702,0.0
19
+ gdn-30m-t1,1337,70,int8_row,3.399858,0.001155
20
+ gdn-30m-t1,1337,70,int8_tensor,3.422254,0.023551
21
+ gdn-30m-t1,1337,70,int4_row,3.457574,0.058872
22
+ gdn-30m-t4,1337,80,fp32,3.368218,0.0
23
+ gdn-30m-t4,1337,80,int8_row,3.367368,-0.00085
24
+ gdn-30m-t4,1337,80,int8_tensor,3.371211,0.002993
25
+ gdn-30m-t4,1337,80,int4_row,3.413218,0.045
gdn-30m-t1/babylm-strict-small-100Mw-s1337/config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "GDNBitForCausalLM"
4
+ ],
5
+ "attn_mode": "chunk",
6
+ "bitlinear": false,
7
+ "bos_token_id": 1,
8
+ "conv_size": 4,
9
+ "dtype": "float32",
10
+ "eos_token_id": 2,
11
+ "expand_v": 2,
12
+ "fuse_cross_entropy": true,
13
+ "fuse_linear_cross_entropy": false,
14
+ "fuse_norm": true,
15
+ "fused_proj": true,
16
+ "head_dim": 64,
17
+ "hidden_act": "relu2",
18
+ "hidden_size": 512,
19
+ "init_scheme": "trunc_normal",
20
+ "initializer_range": 0.02,
21
+ "intermediate_size": 2048,
22
+ "loop_count": 1,
23
+ "max_position_embeddings": 1024,
24
+ "model_type": "gdnbit",
25
+ "norm_eps": 1e-05,
26
+ "num_heads": 6,
27
+ "num_hidden_layers": 6,
28
+ "transformers_version": "4.57.3",
29
+ "use_cache": false,
30
+ "use_short_conv": false,
31
+ "vocab_size": 16384
32
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/args.json ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": "configs/babylm/gdn_30m_t1.json",
3
+ "data_dir": "data/babylm/tok16384-babylm-official",
4
+ "out_dir": "outputs/babylm30m_d512/gdn_30m_t1-lr1e-2-s1337",
5
+ "epochs": 10,
6
+ "batch_size": 32,
7
+ "micro_batch": null,
8
+ "seq_len": 512,
9
+ "lr": 0.01,
10
+ "min_lr_ratio": 0.0,
11
+ "warmup": 500,
12
+ "weight_decay": 0.1,
13
+ "lr_drop_at": 1.0,
14
+ "lr_drop_ratio": 0.6666666666666666,
15
+ "wd_drop_at": 1.0,
16
+ "beta1": 0.9,
17
+ "beta2": 0.95,
18
+ "grad_clip": 1.0,
19
+ "seed": 1337,
20
+ "log_interval": 50,
21
+ "epoch_eval_windows": 2048,
22
+ "save_words": "10,20,30,40,50,60,70,80,90,100",
23
+ "dtype": "bf16",
24
+ "compile": false,
25
+ "tokenizer": null,
26
+ "override": "{}",
27
+ "steps": 10320,
28
+ "params": 30453064,
29
+ "steps_per_epoch": 1032,
30
+ "words_per_token": 0.5910163039530657
31
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/babylm_eval_summary.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "outputs/babylm30m_d512/gdn_30m_t1-lr1e-2-s1337/ckpt/chck_70M",
3
+ "revision": "main",
4
+ "mode": "full",
5
+ "scores": {
6
+ "blimp": 71.44,
7
+ "blimp_supplement": 63.82,
8
+ "entity_tracking": 27.8
9
+ }
10
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/blimp_report.txt ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ TEMPERATURE: 1.00
2
+
3
+ ### FIELD ACCURACY
4
+ syntax/semantics: 67.65
5
+ morphology: 81.08
6
+ syntax: 67.43
7
+ semantics: 69.97
8
+
9
+ ### UID ACCURACY
10
+ expletive_it_object_raising: 68.51
11
+ anaphor_number_agreement: 95.92
12
+ wh_questions_subject_gap_long_distance: 84.60
13
+ existential_there_subject_raising: 76.62
14
+ anaphor_gender_agreement: 88.98
15
+ determiner_noun_agreement_irregular_2: 89.51
16
+ passive_2: 75.64
17
+ irregular_plural_subject_verb_agreement_2: 82.62
18
+ existential_there_quantifiers_1: 97.10
19
+ wh_vs_that_with_gap_long_distance: 11.98
20
+ determiner_noun_agreement_2: 92.27
21
+ sentential_negation_npi_licensor_present: 99.89
22
+ determiner_noun_agreement_with_adj_irregular_1: 77.02
23
+ wh_island: 76.77
24
+ drop_argument: 70.87
25
+ superlative_quantifiers_1: 89.27
26
+ principle_A_reconstruction: 41.16
27
+ principle_A_domain_1: 98.58
28
+ npi_present_2: 57.22
29
+ determiner_noun_agreement_with_adjective_1: 86.39
30
+ tough_vs_raising_2: 82.83
31
+ principle_A_case_2: 83.72
32
+ adjunct_island: 76.94
33
+ regular_plural_subject_verb_agreement_2: 77.04
34
+ distractor_agreement_relational_noun: 61.42
35
+ distractor_agreement_relative_clause: 45.01
36
+ causative: 65.16
37
+ determiner_noun_agreement_irregular_1: 82.09
38
+ transitive: 73.39
39
+ determiner_noun_agreement_1: 90.53
40
+ tough_vs_raising_1: 39.77
41
+ animate_subject_passive: 70.50
42
+ superlative_quantifiers_2: 81.24
43
+ passive_1: 82.62
44
+ determiner_noun_agreement_with_adj_2: 88.63
45
+ sentential_subject_island: 50.88
46
+ wh_questions_object_gap: 76.95
47
+ npi_present_1: 50.72
48
+ wh_vs_that_no_gap: 96.63
49
+ matrix_question_npi_licensor_present: 21.53
50
+ only_npi_scope: 71.57
51
+ inchoative: 51.93
52
+ irregular_plural_subject_verb_agreement_1: 71.52
53
+ principle_A_domain_2: 73.55
54
+ wh_vs_that_no_gap_long_distance: 96.57
55
+ left_branch_island_simple_question: 68.66
56
+ principle_A_domain_3: 61.00
57
+ intransitive: 65.90
58
+ coordinate_structure_constraint_complex_left_branch: 59.82
59
+ determiner_noun_agreement_with_adj_irregular_2: 85.36
60
+ irregular_past_participle_adjectives: 78.25
61
+ wh_vs_that_with_gap: 38.30
62
+ sentential_negation_npi_scope: 29.85
63
+ ellipsis_n_bar_1: 58.23
64
+ regular_plural_subject_verb_agreement_1: 75.96
65
+ ellipsis_n_bar_2: 67.27
66
+ wh_questions_subject_gap: 90.20
67
+ irregular_past_participle_verbs: 84.61
68
+ coordinate_structure_constraint_object_extraction: 63.54
69
+ principle_A_c_command: 53.07
70
+ existential_there_object_raising: 68.84
71
+ existential_there_quantifiers_2: 36.33
72
+ animate_subject_trans: 84.72
73
+ complex_NP_island: 41.61
74
+ left_branch_island_echo_question: 56.28
75
+ only_npi_licensor_present: 95.01
76
+ principle_A_case_1: 100.00
77
+
78
+ ### LINGUISTICS_TERM ACCURACY
79
+ control_raising: 67.06
80
+ anaphor_agreement: 92.38
81
+ filler_gap_dependency: 70.14
82
+ determiner_noun_agreement: 86.90
83
+ argument_structure: 69.42
84
+ subject_verb_agreement: 69.21
85
+ quantifiers: 76.43
86
+ npi_licensing: 60.68
87
+ island_effects: 62.07
88
+ binding: 72.60
89
+ s-selection: 77.72
90
+ irregular_forms: 81.40
91
+ ellipsis: 62.82
92
+
93
+ ### AVERAGE ACCURACY
94
+ 71.44
95
+
gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/blimp_supplement_report.txt ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ TEMPERATURE: 1.00
2
+
3
+ ### FIELD ACCURACY
4
+ supplement: 78.19
5
+
6
+ ### UID ACCURACY
7
+ qa_congruence_easy: 65.62
8
+ turn_taking: 73.57
9
+ qa_congruence_tricky: 41.21
10
+ subject_aux_inversion: 85.83
11
+ hypernym: 52.85
12
+
13
+ ### LINGUISTICS_TERM ACCURACY
14
+ supplement: 78.19
15
+
16
+ ### AVERAGE ACCURACY
17
+ 63.82
18
+
gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/dev_curve.log ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ `torch_dtype` is deprecated! Use `dtype` instead!
2
+ /home/ruhai/Projects/SSM_IMC_NATURE/.venv/lib/python3.12/site-packages/fla/layers/gated_deltanet.py:190: UserWarning: ShortConvolution is crucial to the performance. Do not turn it off, i.e., setting `use_short_conv=False` unless you know what you are doing.
3
+ warnings.warn(
4
+ {"ckpt": "chck_10M", "words_m": 10, "nll": 3.8562589579546875, "bpw": 9.488499612651719, "tokens": 17769472}
5
+ {"ckpt": "chck_20M", "words_m": 20, "nll": 3.633295057885126, "bpw": 8.939886850253773, "tokens": 17769472}
6
+ {"ckpt": "chck_30M", "words_m": 30, "nll": 3.5594808525402444, "bpw": 8.758263658850002, "tokens": 17769472}
7
+ {"ckpt": "chck_40M", "words_m": 40, "nll": 3.469812067511418, "bpw": 8.537629556916082, "tokens": 17769472}
8
+ {"ckpt": "chck_50M", "words_m": 50, "nll": 3.436955697349751, "bpw": 8.456784971801012, "tokens": 17769472}
9
+ {"ckpt": "chck_60M", "words_m": 60, "nll": 3.4035214972772194, "bpw": 8.374518610050815, "tokens": 17769472}
10
+ {"ckpt": "chck_70M", "words_m": 70, "nll": 3.3981919687350586, "bpw": 8.36140506397954, "tokens": 17769472}
11
+ {"ckpt": "chck_80M", "words_m": 80, "nll": 3.4426857543762415, "bpw": 8.470884036326794, "tokens": 17769472}
12
+ {"ckpt": "chck_90M", "words_m": 90, "nll": 3.503730866693732, "bpw": 8.621088296697835, "tokens": 17769472}
13
+ {"ckpt": "hf", "words_m": 100, "nll": 3.5855714520659014, "bpw": 8.822460759250218, "tokens": 17769472}
14
+ BEST {"ckpt": "chck_70M", "words_m": 70, "nll": 3.3981919687350586, "bpw": 8.36140506397954, "tokens": 17769472}
gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/entity_tracking_report.txt ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ TEMPERATURE: 1.00
2
+
3
+ ### UID ACCURACY
4
+ move_contents_0_ops: 23.97
5
+ move_contents_1_ops: 23.43
6
+ move_contents_2_ops: 30.25
7
+ move_contents_3_ops: 28.88
8
+ move_contents_4_ops: 30.06
9
+ move_contents_5_ops: 30.77
10
+ ambiref_0_ops: 25.54
11
+ ambiref_1_ops: 25.37
12
+ ambiref_2_ops: 32.45
13
+ ambiref_3_ops: 28.48
14
+ ambiref_4_ops: 27.32
15
+ ambiref_5_ops: 26.20
16
+ regular_0_ops: 23.60
17
+ regular_1_ops: 20.92
18
+ regular_2_ops: 26.07
19
+ regular_3_ops: 31.35
20
+ regular_4_ops: 30.46
21
+ regular_5_ops: 35.22
22
+ regular: 27.94
23
+ ambiref: 27.56
24
+ move_contents: 27.89
25
+
26
+ ### AVERAGE ACCURACY
27
+ 27.80
28
+
gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/final_summary.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "gdn_30m_t1-lr1e-2-s1337",
3
+ "params": 30453064,
4
+ "best_ckpt": "chck_70M",
5
+ "best_words_m": 70,
6
+ "dev_nll": 3.3981919687350586,
7
+ "dev_bpw": 8.36140506397954,
8
+ "test_nll": 3.306020510774677,
9
+ "test_bpw": 8.263090470345386,
10
+ "blimp": 71.44,
11
+ "blimp_supplement": 63.82,
12
+ "entity_tracking": 27.8
13
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s1337/generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.57.3",
6
+ "use_cache": false
7
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s1337/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:424b3a94a35ea91a1f08811f84a37ee8a6adf60935b1bcbcfb16f877eda300c7
3
+ size 121817944
gdn-30m-t1/babylm-strict-small-100Mw-s1337/special_tokens_map.json ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "cls_token": {
10
+ "content": "<s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "eos_token": {
17
+ "content": "</s>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "mask_token": {
24
+ "content": "<mask>",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "pad_token": {
31
+ "content": "<pad>",
32
+ "lstrip": false,
33
+ "normalized": false,
34
+ "rstrip": false,
35
+ "single_word": false
36
+ },
37
+ "sep_token": {
38
+ "content": "</s>",
39
+ "lstrip": false,
40
+ "normalized": false,
41
+ "rstrip": false,
42
+ "single_word": false
43
+ },
44
+ "unk_token": {
45
+ "content": "<unk>",
46
+ "lstrip": false,
47
+ "normalized": false,
48
+ "rstrip": false,
49
+ "single_word": false
50
+ }
51
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s1337/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
gdn-30m-t1/babylm-strict-small-100Mw-s1337/tokenizer_config.json ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "<unk>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "1": {
12
+ "content": "<s>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "2": {
20
+ "content": "</s>",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "3": {
28
+ "content": "<pad>",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "4": {
36
+ "content": "<mask>",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ }
43
+ },
44
+ "bos_token": "<s>",
45
+ "clean_up_tokenization_spaces": false,
46
+ "cls_token": "<s>",
47
+ "eos_token": "</s>",
48
+ "extra_special_tokens": {},
49
+ "mask_token": "<mask>",
50
+ "model_max_length": 1000000000000000019884624838656,
51
+ "pad_token": "<pad>",
52
+ "sep_token": "</s>",
53
+ "tokenizer_class": "PreTrainedTokenizerFast",
54
+ "unk_token": "<unk>"
55
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s42/config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "GDNBitForCausalLM"
4
+ ],
5
+ "attn_mode": "chunk",
6
+ "bitlinear": false,
7
+ "bos_token_id": 1,
8
+ "conv_size": 4,
9
+ "dtype": "float32",
10
+ "eos_token_id": 2,
11
+ "expand_v": 2,
12
+ "fuse_cross_entropy": true,
13
+ "fuse_linear_cross_entropy": false,
14
+ "fuse_norm": true,
15
+ "fused_proj": true,
16
+ "head_dim": 64,
17
+ "hidden_act": "relu2",
18
+ "hidden_size": 512,
19
+ "init_scheme": "trunc_normal",
20
+ "initializer_range": 0.02,
21
+ "intermediate_size": 2048,
22
+ "loop_count": 1,
23
+ "max_position_embeddings": 1024,
24
+ "model_type": "gdnbit",
25
+ "norm_eps": 1e-05,
26
+ "num_heads": 6,
27
+ "num_hidden_layers": 6,
28
+ "transformers_version": "4.57.3",
29
+ "use_cache": false,
30
+ "use_short_conv": false,
31
+ "vocab_size": 16384
32
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/args.json ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": "configs/babylm/gdn_30m_t1.json",
3
+ "data_dir": "data/babylm/tok16384-babylm-official",
4
+ "out_dir": "outputs/babylm30m_d512/gdn_30m_t1-lr1e-2-s42",
5
+ "epochs": 10,
6
+ "batch_size": 32,
7
+ "micro_batch": null,
8
+ "seq_len": 512,
9
+ "lr": 0.01,
10
+ "min_lr_ratio": 0.0,
11
+ "warmup": 500,
12
+ "weight_decay": 0.1,
13
+ "lr_drop_at": 1.0,
14
+ "lr_drop_ratio": 0.6666666666666666,
15
+ "wd_drop_at": 1.0,
16
+ "beta1": 0.9,
17
+ "beta2": 0.95,
18
+ "grad_clip": 1.0,
19
+ "seed": 42,
20
+ "log_interval": 50,
21
+ "epoch_eval_windows": 2048,
22
+ "save_words": "10,20,30,40,50,60,70,80,90,100",
23
+ "dtype": "bf16",
24
+ "compile": false,
25
+ "tokenizer": null,
26
+ "override": "{}",
27
+ "steps": 10320,
28
+ "params": 30453064,
29
+ "steps_per_epoch": 1032,
30
+ "words_per_token": 0.5910163039530657
31
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/babylm_eval_summary.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "outputs/babylm30m_d512/gdn_30m_t1-lr1e-2-s42/ckpt/chck_80M",
3
+ "revision": "main",
4
+ "mode": "full",
5
+ "scores": {
6
+ "blimp": 70.37,
7
+ "blimp_supplement": 63.11,
8
+ "entity_tracking": 26.26
9
+ }
10
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/blimp_report.txt ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ TEMPERATURE: 1.00
2
+
3
+ ### FIELD ACCURACY
4
+ syntax/semantics: 66.72
5
+ morphology: 80.80
6
+ syntax: 67.97
7
+ semantics: 61.89
8
+
9
+ ### UID ACCURACY
10
+ expletive_it_object_raising: 73.65
11
+ anaphor_number_agreement: 96.46
12
+ wh_questions_subject_gap_long_distance: 83.55
13
+ existential_there_subject_raising: 81.28
14
+ anaphor_gender_agreement: 94.54
15
+ determiner_noun_agreement_irregular_2: 86.22
16
+ passive_2: 77.19
17
+ irregular_plural_subject_verb_agreement_2: 82.29
18
+ existential_there_quantifiers_1: 94.73
19
+ wh_vs_that_with_gap_long_distance: 15.71
20
+ determiner_noun_agreement_2: 92.16
21
+ sentential_negation_npi_licensor_present: 100.00
22
+ determiner_noun_agreement_with_adj_irregular_1: 82.59
23
+ wh_island: 69.27
24
+ drop_argument: 70.22
25
+ superlative_quantifiers_1: 53.42
26
+ principle_A_reconstruction: 31.85
27
+ principle_A_domain_1: 95.62
28
+ npi_present_2: 52.52
29
+ determiner_noun_agreement_with_adjective_1: 87.25
30
+ tough_vs_raising_2: 85.65
31
+ principle_A_case_2: 87.43
32
+ adjunct_island: 69.29
33
+ regular_plural_subject_verb_agreement_2: 79.47
34
+ distractor_agreement_relational_noun: 62.18
35
+ distractor_agreement_relative_clause: 39.95
36
+ causative: 63.33
37
+ determiner_noun_agreement_irregular_1: 77.24
38
+ transitive: 74.77
39
+ determiner_noun_agreement_1: 91.28
40
+ tough_vs_raising_1: 35.65
41
+ animate_subject_passive: 65.59
42
+ superlative_quantifiers_2: 58.01
43
+ passive_1: 83.45
44
+ determiner_noun_agreement_with_adj_2: 86.50
45
+ sentential_subject_island: 55.36
46
+ wh_questions_object_gap: 73.92
47
+ npi_present_1: 49.17
48
+ wh_vs_that_no_gap: 89.20
49
+ matrix_question_npi_licensor_present: 18.84
50
+ only_npi_scope: 56.39
51
+ inchoative: 52.87
52
+ irregular_plural_subject_verb_agreement_1: 78.86
53
+ principle_A_domain_2: 63.06
54
+ wh_vs_that_no_gap_long_distance: 91.77
55
+ left_branch_island_simple_question: 78.97
56
+ principle_A_domain_3: 57.92
57
+ intransitive: 65.90
58
+ coordinate_structure_constraint_complex_left_branch: 60.49
59
+ determiner_noun_agreement_with_adj_irregular_2: 82.14
60
+ irregular_past_participle_adjectives: 65.24
61
+ wh_vs_that_with_gap: 47.01
62
+ sentential_negation_npi_scope: 38.81
63
+ ellipsis_n_bar_1: 65.84
64
+ regular_plural_subject_verb_agreement_1: 80.79
65
+ ellipsis_n_bar_2: 68.84
66
+ wh_questions_subject_gap: 87.86
67
+ irregular_past_participle_verbs: 83.65
68
+ coordinate_structure_constraint_object_extraction: 69.86
69
+ principle_A_c_command: 54.55
70
+ existential_there_object_raising: 75.99
71
+ existential_there_quantifiers_2: 40.61
72
+ animate_subject_trans: 86.35
73
+ complex_NP_island: 44.44
74
+ left_branch_island_echo_question: 58.18
75
+ only_npi_licensor_present: 91.38
76
+ principle_A_case_1: 99.89
77
+
78
+ ### LINGUISTICS_TERM ACCURACY
79
+ control_raising: 69.97
80
+ anaphor_agreement: 95.48
81
+ filler_gap_dependency: 69.36
82
+ determiner_noun_agreement: 86.12
83
+ argument_structure: 69.75
84
+ subject_verb_agreement: 70.83
85
+ quantifiers: 61.64
86
+ npi_licensing: 58.09
87
+ island_effects: 63.49
88
+ binding: 69.60
89
+ s-selection: 76.13
90
+ irregular_forms: 74.36
91
+ ellipsis: 67.36
92
+
93
+ ### AVERAGE ACCURACY
94
+ 70.37
95
+
gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/blimp_supplement_report.txt ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ TEMPERATURE: 1.00
2
+
3
+ ### FIELD ACCURACY
4
+ supplement: 76.37
5
+
6
+ ### UID ACCURACY
7
+ qa_congruence_easy: 65.62
8
+ turn_taking: 68.57
9
+ qa_congruence_tricky: 46.67
10
+ subject_aux_inversion: 83.97
11
+ hypernym: 50.71
12
+
13
+ ### LINGUISTICS_TERM ACCURACY
14
+ supplement: 76.37
15
+
16
+ ### AVERAGE ACCURACY
17
+ 63.11
18
+
gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/dev_curve.log ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ `torch_dtype` is deprecated! Use `dtype` instead!
2
+ /home/ruhai/Projects/SSM_IMC_NATURE/.venv/lib/python3.12/site-packages/fla/layers/gated_deltanet.py:190: UserWarning: ShortConvolution is crucial to the performance. Do not turn it off, i.e., setting `use_short_conv=False` unless you know what you are doing.
3
+ warnings.warn(
4
+ {"ckpt": "chck_10M", "words_m": 10, "nll": 3.8780883266308503, "bpw": 9.54221176178033, "tokens": 17769472}
5
+ {"ckpt": "chck_20M", "words_m": 20, "nll": 3.655319560867685, "bpw": 8.994079136170612, "tokens": 17769472}
6
+ {"ckpt": "chck_30M", "words_m": 30, "nll": 3.613189694847503, "bpw": 8.890416694988087, "tokens": 17769472}
7
+ {"ckpt": "chck_40M", "words_m": 40, "nll": 3.5164252370054947, "bpw": 8.652323369108508, "tokens": 17769472}
8
+ {"ckpt": "chck_50M", "words_m": 50, "nll": 3.5224081582516913, "bpw": 8.667044617485763, "tokens": 17769472}
9
+ {"ckpt": "chck_60M", "words_m": 60, "nll": 3.504233767689116, "bpw": 8.622325707346844, "tokens": 17769472}
10
+ {"ckpt": "chck_70M", "words_m": 70, "nll": 3.5036044980595795, "bpw": 8.620777360956932, "tokens": 17769472}
11
+ {"ckpt": "chck_80M", "words_m": 80, "nll": 3.4753230730271207, "bpw": 8.55118963529028, "tokens": 17769472}
12
+ {"ckpt": "chck_90M", "words_m": 90, "nll": 3.5776454737802914, "bpw": 8.802958531128903, "tokens": 17769472}
13
+ {"ckpt": "hf", "words_m": 100, "nll": 3.6450934552159877, "bpw": 8.968917340613572, "tokens": 17769472}
14
+ BEST {"ckpt": "chck_80M", "words_m": 80, "nll": 3.4753230730271207, "bpw": 8.55118963529028, "tokens": 17769472}
gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/entity_tracking_report.txt ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ TEMPERATURE: 1.00
2
+
3
+ ### UID ACCURACY
4
+ move_contents_0_ops: 22.48
5
+ move_contents_1_ops: 20.30
6
+ move_contents_2_ops: 26.12
7
+ move_contents_3_ops: 26.40
8
+ move_contents_4_ops: 27.41
9
+ move_contents_5_ops: 25.64
10
+ ambiref_0_ops: 23.56
11
+ ambiref_1_ops: 31.14
12
+ ambiref_2_ops: 27.48
13
+ ambiref_3_ops: 35.43
14
+ ambiref_4_ops: 28.62
15
+ ambiref_5_ops: 36.90
16
+ regular_0_ops: 18.98
17
+ regular_1_ops: 20.26
18
+ regular_2_ops: 25.08
19
+ regular_3_ops: 26.24
20
+ regular_4_ops: 25.53
21
+ regular_5_ops: 25.16
22
+ regular: 23.54
23
+ ambiref: 30.52
24
+ move_contents: 24.72
25
+
26
+ ### AVERAGE ACCURACY
27
+ 26.26
28
+
gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/final_summary.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "gdn_30m_t1-lr1e-2-s42",
3
+ "params": 30453064,
4
+ "best_ckpt": "chck_80M",
5
+ "best_words_m": 80,
6
+ "dev_nll": 3.4753230730271207,
7
+ "dev_bpw": 8.55118963529028,
8
+ "test_nll": 3.4128869393076893,
9
+ "test_bpw": 8.53019315901082,
10
+ "blimp": 70.37,
11
+ "blimp_supplement": 63.11,
12
+ "entity_tracking": 26.26
13
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s42/generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.57.3",
6
+ "use_cache": false
7
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s42/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:df3b49dc997b22d2425682b43901e833a629cea76d8ab21fdfc32a01126f6329
3
+ size 121817944
gdn-30m-t1/babylm-strict-small-100Mw-s42/special_tokens_map.json ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "cls_token": {
10
+ "content": "<s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "eos_token": {
17
+ "content": "</s>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "mask_token": {
24
+ "content": "<mask>",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "pad_token": {
31
+ "content": "<pad>",
32
+ "lstrip": false,
33
+ "normalized": false,
34
+ "rstrip": false,
35
+ "single_word": false
36
+ },
37
+ "sep_token": {
38
+ "content": "</s>",
39
+ "lstrip": false,
40
+ "normalized": false,
41
+ "rstrip": false,
42
+ "single_word": false
43
+ },
44
+ "unk_token": {
45
+ "content": "<unk>",
46
+ "lstrip": false,
47
+ "normalized": false,
48
+ "rstrip": false,
49
+ "single_word": false
50
+ }
51
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s42/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
gdn-30m-t1/babylm-strict-small-100Mw-s42/tokenizer_config.json ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "<unk>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "1": {
12
+ "content": "<s>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "2": {
20
+ "content": "</s>",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "3": {
28
+ "content": "<pad>",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "4": {
36
+ "content": "<mask>",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ }
43
+ },
44
+ "bos_token": "<s>",
45
+ "clean_up_tokenization_spaces": false,
46
+ "cls_token": "<s>",
47
+ "eos_token": "</s>",
48
+ "extra_special_tokens": {},
49
+ "mask_token": "<mask>",
50
+ "model_max_length": 1000000000000000019884624838656,
51
+ "pad_token": "<pad>",
52
+ "sep_token": "</s>",
53
+ "tokenizer_class": "PreTrainedTokenizerFast",
54
+ "unk_token": "<unk>"
55
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s7/config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "GDNBitForCausalLM"
4
+ ],
5
+ "attn_mode": "chunk",
6
+ "bitlinear": false,
7
+ "bos_token_id": 1,
8
+ "conv_size": 4,
9
+ "dtype": "float32",
10
+ "eos_token_id": 2,
11
+ "expand_v": 2,
12
+ "fuse_cross_entropy": true,
13
+ "fuse_linear_cross_entropy": false,
14
+ "fuse_norm": true,
15
+ "fused_proj": true,
16
+ "head_dim": 64,
17
+ "hidden_act": "relu2",
18
+ "hidden_size": 512,
19
+ "init_scheme": "trunc_normal",
20
+ "initializer_range": 0.02,
21
+ "intermediate_size": 2048,
22
+ "loop_count": 1,
23
+ "max_position_embeddings": 1024,
24
+ "model_type": "gdnbit",
25
+ "norm_eps": 1e-05,
26
+ "num_heads": 6,
27
+ "num_hidden_layers": 6,
28
+ "transformers_version": "4.57.3",
29
+ "use_cache": false,
30
+ "use_short_conv": false,
31
+ "vocab_size": 16384
32
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/args.json ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": "configs/babylm/gdn_30m_t1.json",
3
+ "data_dir": "data/babylm/tok16384-babylm-official",
4
+ "out_dir": "outputs/babylm30m_d512/gdn_30m_t1-lr1e-2-s7",
5
+ "epochs": 10,
6
+ "batch_size": 32,
7
+ "micro_batch": null,
8
+ "seq_len": 512,
9
+ "lr": 0.01,
10
+ "min_lr_ratio": 0.0,
11
+ "warmup": 500,
12
+ "weight_decay": 0.1,
13
+ "lr_drop_at": 1.0,
14
+ "lr_drop_ratio": 0.6666666666666666,
15
+ "wd_drop_at": 1.0,
16
+ "beta1": 0.9,
17
+ "beta2": 0.95,
18
+ "grad_clip": 1.0,
19
+ "seed": 7,
20
+ "log_interval": 50,
21
+ "epoch_eval_windows": 2048,
22
+ "save_words": "10,20,30,40,50,60,70,80,90,100",
23
+ "dtype": "bf16",
24
+ "compile": false,
25
+ "tokenizer": null,
26
+ "override": "{}",
27
+ "steps": 10320,
28
+ "params": 30453064,
29
+ "steps_per_epoch": 1032,
30
+ "words_per_token": 0.5910163039530657
31
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/babylm_eval_summary.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "outputs/babylm30m_d512/gdn_30m_t1-lr1e-2-s7/ckpt/chck_70M",
3
+ "revision": "main",
4
+ "mode": "full",
5
+ "scores": {
6
+ "blimp": 70.09,
7
+ "blimp_supplement": 59.33,
8
+ "entity_tracking": 18.87
9
+ }
10
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/blimp_report.txt ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ TEMPERATURE: 1.00
2
+
3
+ ### FIELD ACCURACY
4
+ syntax/semantics: 66.00
5
+ morphology: 80.48
6
+ syntax: 66.55
7
+ semantics: 65.63
8
+
9
+ ### UID ACCURACY
10
+ expletive_it_object_raising: 67.85
11
+ anaphor_number_agreement: 96.78
12
+ wh_questions_subject_gap_long_distance: 83.08
13
+ existential_there_subject_raising: 79.65
14
+ anaphor_gender_agreement: 93.00
15
+ determiner_noun_agreement_irregular_2: 87.07
16
+ passive_2: 76.30
17
+ irregular_plural_subject_verb_agreement_2: 79.82
18
+ existential_there_quantifiers_1: 95.48
19
+ wh_vs_that_with_gap_long_distance: 17.36
20
+ determiner_noun_agreement_2: 92.80
21
+ sentential_negation_npi_licensor_present: 99.78
22
+ determiner_noun_agreement_with_adj_irregular_1: 77.72
23
+ wh_island: 46.46
24
+ drop_argument: 71.09
25
+ superlative_quantifiers_1: 68.54
26
+ principle_A_reconstruction: 31.75
27
+ principle_A_domain_1: 96.72
28
+ npi_present_2: 47.37
29
+ determiner_noun_agreement_with_adjective_1: 86.39
30
+ tough_vs_raising_2: 80.54
31
+ principle_A_case_2: 82.95
32
+ adjunct_island: 64.66
33
+ regular_plural_subject_verb_agreement_2: 76.30
34
+ distractor_agreement_relational_noun: 56.22
35
+ distractor_agreement_relative_clause: 46.50
36
+ causative: 63.57
37
+ determiner_noun_agreement_irregular_1: 81.06
38
+ transitive: 74.42
39
+ determiner_noun_agreement_1: 89.99
40
+ tough_vs_raising_1: 42.72
41
+ animate_subject_passive: 68.38
42
+ superlative_quantifiers_2: 74.44
43
+ passive_1: 83.69
44
+ determiner_noun_agreement_with_adj_2: 87.89
45
+ sentential_subject_island: 52.13
46
+ wh_questions_object_gap: 79.74
47
+ npi_present_1: 43.23
48
+ wh_vs_that_no_gap: 94.43
49
+ matrix_question_npi_licensor_present: 23.47
50
+ only_npi_scope: 69.77
51
+ inchoative: 54.27
52
+ irregular_plural_subject_verb_agreement_1: 72.39
53
+ principle_A_domain_2: 64.81
54
+ wh_vs_that_no_gap_long_distance: 93.37
55
+ left_branch_island_simple_question: 67.30
56
+ principle_A_domain_3: 59.30
57
+ intransitive: 67.97
58
+ coordinate_structure_constraint_complex_left_branch: 59.38
59
+ determiner_noun_agreement_with_adj_irregular_2: 82.14
60
+ irregular_past_participle_adjectives: 73.15
61
+ wh_vs_that_with_gap: 33.30
62
+ sentential_negation_npi_scope: 31.11
63
+ ellipsis_n_bar_1: 67.83
64
+ regular_plural_subject_verb_agreement_1: 78.09
65
+ ellipsis_n_bar_2: 72.83
66
+ wh_questions_subject_gap: 91.76
67
+ irregular_past_participle_verbs: 84.29
68
+ coordinate_structure_constraint_object_extraction: 65.75
69
+ principle_A_c_command: 54.65
70
+ existential_there_object_raising: 64.29
71
+ existential_there_quantifiers_2: 43.03
72
+ animate_subject_trans: 85.81
73
+ complex_NP_island: 48.46
74
+ left_branch_island_echo_question: 54.17
75
+ only_npi_licensor_present: 95.24
76
+ principle_A_case_1: 100.00
77
+
78
+ ### LINGUISTICS_TERM ACCURACY
79
+ control_raising: 66.90
80
+ anaphor_agreement: 94.85
81
+ filler_gap_dependency: 69.83
82
+ determiner_noun_agreement: 86.07
83
+ argument_structure: 70.26
84
+ subject_verb_agreement: 68.55
85
+ quantifiers: 70.55
86
+ npi_licensing: 58.39
87
+ island_effects: 57.36
88
+ binding: 69.59
89
+ s-selection: 77.23
90
+ irregular_forms: 78.67
91
+ ellipsis: 70.37
92
+
93
+ ### AVERAGE ACCURACY
94
+ 70.09
95
+
gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/blimp_supplement_report.txt ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ TEMPERATURE: 1.00
2
+
3
+ ### FIELD ACCURACY
4
+ supplement: 77.81
5
+
6
+ ### UID ACCURACY
7
+ qa_congruence_easy: 50.00
8
+ turn_taking: 69.29
9
+ qa_congruence_tricky: 41.21
10
+ subject_aux_inversion: 86.60
11
+ hypernym: 49.52
12
+
13
+ ### LINGUISTICS_TERM ACCURACY
14
+ supplement: 77.81
15
+
16
+ ### AVERAGE ACCURACY
17
+ 59.33
18
+
gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/dev_curve.log ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ `torch_dtype` is deprecated! Use `dtype` instead!
2
+ /home/ruhai/Projects/SSM_IMC_NATURE/.venv/lib/python3.12/site-packages/fla/layers/gated_deltanet.py:190: UserWarning: ShortConvolution is crucial to the performance. Do not turn it off, i.e., setting `use_short_conv=False` unless you know what you are doing.
3
+ warnings.warn(
4
+ {"ckpt": "chck_10M", "words_m": 10, "nll": 3.826301932149419, "bpw": 9.414788995483974, "tokens": 17769472}
5
+ {"ckpt": "chck_20M", "words_m": 20, "nll": 3.6475560366300206, "bpw": 8.974976633583244, "tokens": 17769472}
6
+ {"ckpt": "chck_30M", "words_m": 30, "nll": 3.5876698592605902, "bpw": 8.82762398507895, "tokens": 17769472}
7
+ {"ckpt": "chck_40M", "words_m": 40, "nll": 3.4893632862524346, "bpw": 8.585736215071996, "tokens": 17769472}
8
+ {"ckpt": "chck_50M", "words_m": 50, "nll": 3.5046018000220056, "bpw": 8.623231267550693, "tokens": 17769472}
9
+ {"ckpt": "chck_60M", "words_m": 60, "nll": 3.459844292573803, "bpw": 8.513103395767216, "tokens": 17769472}
10
+ {"ckpt": "chck_70M", "words_m": 70, "nll": 3.4189122823832707, "bpw": 8.412388333041262, "tokens": 17769472}
11
+ {"ckpt": "chck_80M", "words_m": 80, "nll": 3.443516969172431, "bpw": 8.472929277934702, "tokens": 17769472}
12
+ {"ckpt": "chck_90M", "words_m": 90, "nll": 3.5014851779571727, "bpw": 8.615562677972719, "tokens": 17769472}
13
+ {"ckpt": "hf", "words_m": 100, "nll": 3.5793337967322554, "bpw": 8.807112726127322, "tokens": 17769472}
14
+ BEST {"ckpt": "chck_70M", "words_m": 70, "nll": 3.4189122823832707, "bpw": 8.412388333041262, "tokens": 17769472}
gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/entity_tracking_report.txt ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ TEMPERATURE: 1.00
2
+
3
+ ### UID ACCURACY
4
+ move_contents_0_ops: 18.02
5
+ move_contents_1_ops: 17.33
6
+ move_contents_2_ops: 21.16
7
+ move_contents_3_ops: 18.81
8
+ move_contents_4_ops: 19.66
9
+ move_contents_5_ops: 20.51
10
+ ambiref_0_ops: 22.57
11
+ ambiref_1_ops: 16.31
12
+ ambiref_2_ops: 19.37
13
+ ambiref_3_ops: 20.36
14
+ ambiref_4_ops: 17.07
15
+ ambiref_5_ops: 17.65
16
+ regular_0_ops: 13.53
17
+ regular_1_ops: 15.82
18
+ regular_2_ops: 17.16
19
+ regular_3_ops: 23.27
20
+ regular_4_ops: 20.25
21
+ regular_5_ops: 20.75
22
+ regular: 18.46
23
+ ambiref: 18.89
24
+ move_contents: 19.25
25
+
26
+ ### AVERAGE ACCURACY
27
+ 18.87
28
+
gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/final_summary.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "gdn_30m_t1-lr1e-2-s7",
3
+ "params": 30453064,
4
+ "best_ckpt": "chck_70M",
5
+ "best_words_m": 70,
6
+ "dev_nll": 3.4189122823832707,
7
+ "dev_bpw": 8.412388333041262,
8
+ "test_nll": 3.3520635961237866,
9
+ "test_bpw": 8.378170875483095,
10
+ "blimp": 70.09,
11
+ "blimp_supplement": 59.33,
12
+ "entity_tracking": 18.87
13
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s7/generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.57.3",
6
+ "use_cache": false
7
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s7/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4a02bc31ce1d50f4ae735bee541801f2aecd8f9dc8b9faecb165522d2e448c74
3
+ size 121817944
gdn-30m-t1/babylm-strict-small-100Mw-s7/special_tokens_map.json ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "cls_token": {
10
+ "content": "<s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "eos_token": {
17
+ "content": "</s>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "mask_token": {
24
+ "content": "<mask>",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "pad_token": {
31
+ "content": "<pad>",
32
+ "lstrip": false,
33
+ "normalized": false,
34
+ "rstrip": false,
35
+ "single_word": false
36
+ },
37
+ "sep_token": {
38
+ "content": "</s>",
39
+ "lstrip": false,
40
+ "normalized": false,
41
+ "rstrip": false,
42
+ "single_word": false
43
+ },
44
+ "unk_token": {
45
+ "content": "<unk>",
46
+ "lstrip": false,
47
+ "normalized": false,
48
+ "rstrip": false,
49
+ "single_word": false
50
+ }
51
+ }
gdn-30m-t1/babylm-strict-small-100Mw-s7/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
gdn-30m-t1/babylm-strict-small-100Mw-s7/tokenizer_config.json ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "<unk>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "1": {
12
+ "content": "<s>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "2": {
20
+ "content": "</s>",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "3": {
28
+ "content": "<pad>",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "4": {
36
+ "content": "<mask>",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ }
43
+ },
44
+ "bos_token": "<s>",
45
+ "clean_up_tokenization_spaces": false,
46
+ "cls_token": "<s>",
47
+ "eos_token": "</s>",
48
+ "extra_special_tokens": {},
49
+ "mask_token": "<mask>",
50
+ "model_max_length": 1000000000000000019884624838656,
51
+ "pad_token": "<pad>",
52
+ "sep_token": "</s>",
53
+ "tokenizer_class": "PreTrainedTokenizerFast",
54
+ "unk_token": "<unk>"
55
+ }
gdn-30m-t4/babylm-strict-small-100Mw-s1337/config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "GDNBitForCausalLM"
4
+ ],
5
+ "attn_mode": "chunk",
6
+ "bitlinear": false,
7
+ "bos_token_id": 1,
8
+ "conv_size": 4,
9
+ "dtype": "float32",
10
+ "eos_token_id": 2,
11
+ "expand_v": 2,
12
+ "fuse_cross_entropy": true,
13
+ "fuse_linear_cross_entropy": false,
14
+ "fuse_norm": true,
15
+ "fused_proj": true,
16
+ "head_dim": 64,
17
+ "hidden_act": "relu2",
18
+ "hidden_size": 512,
19
+ "init_scheme": "trunc_normal",
20
+ "initializer_range": 0.02,
21
+ "intermediate_size": 2048,
22
+ "loop_count": 4,
23
+ "max_position_embeddings": 1024,
24
+ "model_type": "gdnbit",
25
+ "norm_eps": 1e-05,
26
+ "num_heads": 6,
27
+ "num_hidden_layers": 6,
28
+ "transformers_version": "4.57.3",
29
+ "use_cache": false,
30
+ "use_short_conv": false,
31
+ "vocab_size": 16384
32
+ }
gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/args.json ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": "configs/babylm/gdn_30m_t4.json",
3
+ "data_dir": "data/babylm/tok16384-babylm-official",
4
+ "out_dir": "outputs/babylm30m_d512/gdn_30m_t4-lr1e-2-s1337",
5
+ "epochs": 10,
6
+ "batch_size": 32,
7
+ "micro_batch": 16,
8
+ "seq_len": 512,
9
+ "lr": 0.01,
10
+ "min_lr_ratio": 0.0,
11
+ "warmup": 500,
12
+ "weight_decay": 0.1,
13
+ "lr_drop_at": 1.0,
14
+ "lr_drop_ratio": 0.6666666666666666,
15
+ "wd_drop_at": 1.0,
16
+ "beta1": 0.9,
17
+ "beta2": 0.95,
18
+ "grad_clip": 1.0,
19
+ "seed": 1337,
20
+ "log_interval": 50,
21
+ "epoch_eval_windows": 2048,
22
+ "save_words": "10,20,30,40,50,60,70,80,90,100",
23
+ "dtype": "bf16",
24
+ "compile": false,
25
+ "tokenizer": null,
26
+ "override": "{}",
27
+ "steps": 10320,
28
+ "params": 30455112,
29
+ "steps_per_epoch": 1032,
30
+ "words_per_token": 0.5910163039530657
31
+ }
gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/babylm_eval_summary.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "outputs/babylm30m_d512/gdn_30m_t4-lr1e-2-s1337/ckpt/chck_80M",
3
+ "revision": "main",
4
+ "mode": "full",
5
+ "scores": {
6
+ "blimp": 72.08,
7
+ "blimp_supplement": 61.02,
8
+ "entity_tracking": 21.1
9
+ }
10
+ }
gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/blimp_report.txt ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ TEMPERATURE: 1.00
2
+
3
+ ### FIELD ACCURACY
4
+ syntax/semantics: 70.73
5
+ morphology: 81.32
6
+ syntax: 67.43
7
+ semantics: 69.34
8
+
9
+ ### UID ACCURACY
10
+ expletive_it_object_raising: 68.64
11
+ anaphor_number_agreement: 95.92
12
+ wh_questions_subject_gap_long_distance: 81.68
13
+ existential_there_subject_raising: 83.01
14
+ anaphor_gender_agreement: 93.20
15
+ determiner_noun_agreement_irregular_2: 86.46
16
+ passive_2: 77.52
17
+ irregular_plural_subject_verb_agreement_2: 81.50
18
+ existential_there_quantifiers_1: 97.74
19
+ wh_vs_that_with_gap_long_distance: 15.16
20
+ determiner_noun_agreement_2: 90.12
21
+ sentential_negation_npi_licensor_present: 99.89
22
+ determiner_noun_agreement_with_adj_irregular_1: 78.97
23
+ wh_island: 73.96
24
+ drop_argument: 71.20
25
+ superlative_quantifiers_1: 76.61
26
+ principle_A_reconstruction: 50.57
27
+ principle_A_domain_1: 97.48
28
+ npi_present_2: 54.16
29
+ determiner_noun_agreement_with_adjective_1: 87.35
30
+ tough_vs_raising_2: 86.85
31
+ principle_A_case_2: 83.93
32
+ adjunct_island: 61.96
33
+ regular_plural_subject_verb_agreement_2: 75.34
34
+ distractor_agreement_relational_noun: 66.62
35
+ distractor_agreement_relative_clause: 48.56
36
+ causative: 64.06
37
+ determiner_noun_agreement_irregular_1: 79.30
38
+ transitive: 71.77
39
+ determiner_noun_agreement_1: 90.64
40
+ tough_vs_raising_1: 50.63
41
+ animate_subject_passive: 71.28
42
+ superlative_quantifiers_2: 76.17
43
+ passive_1: 77.26
44
+ determiner_noun_agreement_with_adj_2: 85.23
45
+ sentential_subject_island: 43.91
46
+ wh_questions_object_gap: 78.70
47
+ npi_present_1: 50.06
48
+ wh_vs_that_no_gap: 95.24
49
+ matrix_question_npi_licensor_present: 22.39
50
+ only_npi_scope: 69.65
51
+ inchoative: 53.45
52
+ irregular_plural_subject_verb_agreement_1: 76.99
53
+ principle_A_domain_2: 68.31
54
+ wh_vs_that_no_gap_long_distance: 95.43
55
+ left_branch_island_simple_question: 72.34
56
+ principle_A_domain_3: 59.09
57
+ intransitive: 64.63
58
+ coordinate_structure_constraint_complex_left_branch: 55.74
59
+ determiner_noun_agreement_with_adj_irregular_2: 82.86
60
+ irregular_past_participle_adjectives: 75.65
61
+ wh_vs_that_with_gap: 37.65
62
+ sentential_negation_npi_scope: 37.31
63
+ ellipsis_n_bar_1: 65.84
64
+ regular_plural_subject_verb_agreement_1: 78.88
65
+ ellipsis_n_bar_2: 74.03
66
+ wh_questions_subject_gap: 90.09
67
+ irregular_past_participle_verbs: 84.50
68
+ coordinate_structure_constraint_object_extraction: 70.18
69
+ principle_A_c_command: 63.00
70
+ existential_there_object_raising: 72.41
71
+ existential_there_quantifiers_2: 52.58
72
+ animate_subject_trans: 83.53
73
+ complex_NP_island: 44.80
74
+ left_branch_island_echo_question: 65.47
75
+ only_npi_licensor_present: 94.22
76
+ principle_A_case_1: 99.89
77
+
78
+ ### LINGUISTICS_TERM ACCURACY
79
+ control_raising: 72.31
80
+ anaphor_agreement: 94.53
81
+ filler_gap_dependency: 69.98
82
+ determiner_noun_agreement: 85.53
83
+ argument_structure: 68.66
84
+ subject_verb_agreement: 71.45
85
+ quantifiers: 75.91
86
+ npi_licensing: 60.93
87
+ island_effects: 61.29
88
+ binding: 74.29
89
+ s-selection: 77.50
90
+ irregular_forms: 80.03
91
+ ellipsis: 70.00
92
+
93
+ ### AVERAGE ACCURACY
94
+ 72.08
95
+
gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/blimp_supplement_report.txt ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ TEMPERATURE: 1.00
2
+
3
+ ### FIELD ACCURACY
4
+ supplement: 72.61
5
+
6
+ ### UID ACCURACY
7
+ qa_congruence_easy: 59.38
8
+ turn_taking: 73.57
9
+ qa_congruence_tricky: 43.03
10
+ subject_aux_inversion: 78.90
11
+ hypernym: 50.24
12
+
13
+ ### LINGUISTICS_TERM ACCURACY
14
+ supplement: 72.61
15
+
16
+ ### AVERAGE ACCURACY
17
+ 61.02
18
+
gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/dev_curve.log ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ `torch_dtype` is deprecated! Use `dtype` instead!
2
+ /ncg/projects/ruhai/SSM_IMC_NATURE/.venv/lib/python3.12/site-packages/fla/layers/gated_deltanet.py:190: UserWarning: ShortConvolution is crucial to the performance. Do not turn it off, i.e., setting `use_short_conv=False` unless you know what you are doing.
3
+ warnings.warn(
4
+ {"ckpt": "chck_10M", "words_m": 10, "nll": 3.81434517047429, "bpw": 9.385368842491136, "tokens": 17769472}
5
+ {"ckpt": "chck_20M", "words_m": 20, "nll": 3.6564029574236225, "bpw": 8.996744882406393, "tokens": 17769472}
6
+ {"ckpt": "chck_30M", "words_m": 30, "nll": 3.537110857198831, "bpw": 8.703221273355071, "tokens": 17769472}
7
+ {"ckpt": "chck_40M", "words_m": 40, "nll": 3.457275940417996, "bpw": 8.506783849102627, "tokens": 17769472}
8
+ {"ckpt": "chck_50M", "words_m": 50, "nll": 3.4188837153502294, "bpw": 8.41231804256417, "tokens": 17769472}
9
+ {"ckpt": "chck_60M", "words_m": 60, "nll": 3.3783650740913265, "bpw": 8.312620092794171, "tokens": 17769472}
10
+ {"ckpt": "chck_70M", "words_m": 70, "nll": 3.38059025704397, "bpw": 8.318095256110338, "tokens": 17769472}
11
+ {"ckpt": "chck_80M", "words_m": 80, "nll": 3.366958815790573, "bpw": 8.28455447825725, "tokens": 17769472}
12
+ {"ckpt": "chck_90M", "words_m": 90, "nll": 3.442769450632453, "bpw": 8.471089974751392, "tokens": 17769472}
13
+ {"ckpt": "hf", "words_m": 100, "nll": 3.516394635012931, "bpw": 8.652248071521456, "tokens": 17769472}
14
+ BEST {"ckpt": "chck_80M", "words_m": 80, "nll": 3.366958815790573, "bpw": 8.28455447825725, "tokens": 17769472}