Instructions to use ruhai-lin/GDNBit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ruhai-lin/GDNBit with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("ruhai-lin/GDNBit", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Restructure around the 30M BabyLM Strict-Small models: GDN-Bit, GDN, HGRN-Bit (T1/T4, 3 seeds), K ablation, ternary exports, eval tables
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- README.md +117 -17
- eval/.gitkeep +0 -0
- eval/all_runs.csv +36 -0
- eval/comparison.csv +11 -0
- eval/state_quant.csv +25 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s1337/config.json +32 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/args.json +31 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/babylm_eval_summary.json +10 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/blimp_report.txt +95 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/blimp_supplement_report.txt +18 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/dev_curve.log +14 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/entity_tracking_report.txt +28 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/final_summary.json +13 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s1337/generation_config.json +7 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s1337/model.safetensors +3 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s1337/special_tokens_map.json +51 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s1337/tokenizer.json +0 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s1337/tokenizer_config.json +55 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s42/config.json +32 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/args.json +31 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/babylm_eval_summary.json +10 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/blimp_report.txt +95 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/blimp_supplement_report.txt +18 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/dev_curve.log +14 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/entity_tracking_report.txt +28 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/final_summary.json +13 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s42/generation_config.json +7 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s42/model.safetensors +3 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s42/special_tokens_map.json +51 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s42/tokenizer.json +0 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s42/tokenizer_config.json +55 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s7/config.json +32 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/args.json +31 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/babylm_eval_summary.json +10 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/blimp_report.txt +95 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/blimp_supplement_report.txt +18 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/dev_curve.log +14 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/entity_tracking_report.txt +28 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/final_summary.json +13 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s7/generation_config.json +7 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s7/model.safetensors +3 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s7/special_tokens_map.json +51 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s7/tokenizer.json +0 -0
- gdn-30m-t1/babylm-strict-small-100Mw-s7/tokenizer_config.json +55 -0
- gdn-30m-t4/babylm-strict-small-100Mw-s1337/config.json +32 -0
- gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/args.json +31 -0
- gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/babylm_eval_summary.json +10 -0
- gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/blimp_report.txt +95 -0
- gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/blimp_supplement_report.txt +18 -0
- gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/dev_curve.log +14 -0
README.md
CHANGED
|
@@ -1,21 +1,121 @@
|
|
| 1 |
---
|
| 2 |
-
|
| 3 |
-
tags:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
---
|
| 5 |
# GDN-Bit
|
| 6 |
|
| 7 |
-
Gated DeltaNet
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
|
| 15 |
-
|
|
| 16 |
-
| `gdnbit-
|
| 17 |
-
| `gdnbit-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
library_name: transformers
|
| 3 |
+
tags:
|
| 4 |
+
- gated-deltanet
|
| 5 |
+
- ternary
|
| 6 |
+
- bitnet
|
| 7 |
+
- looped-transformer
|
| 8 |
+
- babylm
|
| 9 |
---
|
| 10 |
# GDN-Bit
|
| 11 |
|
| 12 |
+
GDN-Bit is a Gated DeltaNet whose linear layers are all ternary BitLinear (absmean ternary weights, int8 per-token activations). It optionally loops the whole layer stack T times. This repository holds the canonical 30M checkpoints trained on BabyLM 2026 Strict-Small:
|
| 13 |
+
- GDN-Bit
|
| 14 |
+
- full-precision GDN with the same architecture
|
| 15 |
+
- HGRN-Bit (the MatMul-free LM mixer)
|
| 16 |
+
|
| 17 |
+
Each comes without (T1) and with (T4) the loop, three seeds each. Folders are named `<arch>-30m[-k<dk>]-t<T>/babylm-strict-small-100Mw-s<seed>/`.
|
| 18 |
+
|
| 19 |
+
| Folder | Mixer | Linear weights | dk | T | Params | Peak LR |
|
| 20 |
+
|---|---|---|---:|---:|---:|---:|
|
| 21 |
+
| `gdnbit-30m-t1` | Gated DeltaNet | ternary | 64 | 1 | 30,470,472 | 2e-3 |
|
| 22 |
+
| `gdnbit-30m-t4` | Gated DeltaNet | ternary | 64 | 4 | 30,472,520 | 2e-3 |
|
| 23 |
+
| `gdn-30m-t1` | Gated DeltaNet | full precision | 64 | 1 | 30,453,064 | 1e-2 |
|
| 24 |
+
| `gdn-30m-t4` | Gated DeltaNet | full precision | 64 | 4 | 30,455,112 | 1e-2 |
|
| 25 |
+
| `hgrnbit-30m-t1` | HGRN (MatMul-free LM) | ternary | – | 1 | 30,566,304 | 2e-3 |
|
| 26 |
+
| `hgrnbit-30m-t4` | HGRN (MatMul-free LM) | ternary | – | 4 | 30,568,352 | 2e-3 |
|
| 27 |
+
| `gdnbit-30m-k16-t1` / `-t4` | Gated DeltaNet | ternary | 16 | 1 / 4 | 28,701,000 / 28,703,048 | 2e-3 |
|
| 28 |
+
| `gdnbit-30m-k32-t1` / `-t4` | Gated DeltaNet | ternary | 32 | 1 / 4 | 29,290,824 / 29,292,872 | 2e-3 |
|
| 29 |
+
|
| 30 |
+
**Architecture** (GDN and GDN-Bit):
|
| 31 |
+
- d=512, 6 layers, 6 heads, dk=64, dv=128, ReLU² FFN of width 2048, no short convolution.
|
| 32 |
+
- One fused input projection for q/k/v/g/a/b, sharing a single RMSNorm and int8 quantization.
|
| 33 |
+
- Tied float embedding and output head; the head is ternary in the Bit models.
|
| 34 |
+
- The K ablation keeps dv=128 and changes only dk.
|
| 35 |
+
|
| 36 |
+
**HGRN-Bit:** d=512, 7 layers, SwiGLU 1376, matched to ~30.5M parameters.
|
| 37 |
+
|
| 38 |
+
**Loop:** the T4 models apply the whole stack 4 times with weights shared across passes, as h ← Stack(h) + r_t ⊙ h_in. The per-pass vector r_t is fp32 and zero-initialized. Every pass starts its recurrent state from zero.
|
| 39 |
+
|
| 40 |
+
## Loading
|
| 41 |
+
|
| 42 |
+
The weights load with Transformers once the model classes are registered. This needs the `custom_models` package of the project code at commit `1bf7e65` (not public yet), plus `flash-linear-attention==0.5.1` and `transformers==4.57.3`; trained with torch 2.13. Plain Transformers does not recognize the model types `gdnbit` and `mmfloop`.
|
| 43 |
+
|
| 44 |
+
```python
|
| 45 |
+
import custom_models # registers gdnbit (GDN, GDN-Bit) and mmfloop (HGRN-Bit)
|
| 46 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 47 |
+
|
| 48 |
+
repo = "ruhai-lin/GDNBit"
|
| 49 |
+
run = "gdnbit-30m-t4/babylm-strict-small-100Mw-s42"
|
| 50 |
+
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=run)
|
| 51 |
+
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=run, dtype="float32")
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
**Warning for T4 models:** the HF recurrent cache is keyed by layer index only, so the 4 passes would overwrite each other's state. Score full sequences without a cache, or keep one state per (pass, layer) yourself. Do not use `generate()` with `use_cache=True`.
|
| 55 |
+
|
| 56 |
+
## Training, evaluation and budgets
|
| 57 |
+
|
| 58 |
+
**Training**
|
| 59 |
+
- **Data:** BabyLM 2026 Strict-Small, 10M words, tokenized with the official BabyLM 16,384-token tokenizer, which is included in every folder.
|
| 60 |
+
- **Budget:** 10 epochs = 100M words processed, 10,320 steps at batch 32 × 512 tokens.
|
| 61 |
+
- **Optimizer:** AdamW (β 0.9/0.95, weight decay 0.1, gradient clip 1.0), cosine schedule to 0 with 500 warmup steps, bf16 autocast.
|
| 62 |
+
- **Learning rate:** the peak LR was swept separately for ternary and full-precision models (see `eval/all_runs.csv`).
|
| 63 |
+
|
| 64 |
+
**Checkpoint selection:** a checkpoint is saved every 10M words. Each folder holds the checkpoint with the lowest full dev NLL; `eval/final_summary.json` gives `best_words_m`.
|
| 65 |
+
|
| 66 |
+
**Metrics**
|
| 67 |
+
- Dev/test NLL is per token, FP32, on consecutive BOS-prefixed 512-token windows over the full BabyLM dev/test sets. bpw is bits per word.
|
| 68 |
+
- BLiMP, BLiMP supplement and entity tracking come from the official BabyLM evaluation pipeline (zero-shot, causal, filtered sets).
|
| 69 |
+
- Entity tracking varies strongly across seeds; we do not draw conclusions from it.
|
| 70 |
+
|
| 71 |
+
## Results (mean ± std over seeds 7, 42, 1337)
|
| 72 |
+
|
| 73 |
+
| Model | Dev NLL | Test NLL | BLiMP | BLiMP suppl. |
|
| 74 |
+
|---|---:|---:|---:|---:|
|
| 75 |
+
| gdnbit-30m-t1 | 3.554 ± 0.009 | 3.483 ± 0.013 | 69.8 ± 1.3 | 58.9 ± 1.0 |
|
| 76 |
+
| **gdnbit-30m-t4** | **3.531 ± 0.050** | **3.454 ± 0.063** | **71.2 ± 0.0** | **60.6 ± 0.5** |
|
| 77 |
+
| gdn-30m-t1 | 3.431 ± 0.040 | 3.357 ± 0.054 | 70.6 ± 0.7 | 62.1 ± 2.4 |
|
| 78 |
+
| gdn-30m-t4 | 3.365 ± 0.010 | 3.274 ± 0.016 | 71.7 ± 0.4 | 60.7 ± 0.8 |
|
| 79 |
+
| hgrnbit-30m-t1 | 3.571 ± 0.032 | 3.476 ± 0.037 | 70.3 ± 0.7 | 58.7 ± 0.3 |
|
| 80 |
+
| hgrnbit-30m-t4 | 3.562 ± 0.018 | 3.468 ± 0.025 | 70.2 ± 0.8 | 60.2 ± 0.6 |
|
| 81 |
+
| gdnbit-30m-k16-t1 | 3.624 ± 0.019 | 3.562 ± 0.027 | 69.5 ± 0.5 | 60.6 ± 1.5 |
|
| 82 |
+
| gdnbit-30m-k16-t4 | 3.535 ± 0.019 | 3.454 ± 0.027 | 70.3 ± 0.0 | 59.8 ± 0.9 |
|
| 83 |
+
| gdnbit-30m-k32-t1 | 3.574 ± 0.021 | 3.499 ± 0.030 | 69.9 ± 0.6 | 59.5 ± 0.8 |
|
| 84 |
+
| gdnbit-30m-k32-t4 | 3.524 ± 0.015 | 3.444 ± 0.018 | 70.8 ± 0.4 | 60.1 ± 2.9 |
|
| 85 |
+
|
| 86 |
+
**Result files**
|
| 87 |
+
- `eval/comparison.csv`: all metrics with mean and std, including bpw and entity tracking.
|
| 88 |
+
- `eval/all_runs.csv`: every run-seed, including the full-precision LR sweep points, whose weights are not published.
|
| 89 |
+
- `eval/state_quant.csv`: dev NLL when the recurrent state is stored at reduced precision after every token (seed 1337). An int8 state with one scale per row costs at most 0.004 nats for every model.
|
| 90 |
+
|
| 91 |
+
## Files in each run folder
|
| 92 |
+
|
| 93 |
+
- **HF checkpoint:** `config.json`, `model.safetensors` (fp32 latent weights), the tokenizer files and `generation_config.json`.
|
| 94 |
+
- **`eval/`:**
|
| 95 |
+
- `final_summary.json`: dev/test NLL and bpw, BLiMP, supplement, entity tracking, best checkpoint
|
| 96 |
+
- `babylm_eval_summary.json` and per-task reports: `blimp_report.txt`, `blimp_supplement_report.txt`, `entity_tracking_report.txt` (per-field and per-phenomenon accuracy)
|
| 97 |
+
- `args.json`: training arguments
|
| 98 |
+
- `dev_curve.log`: dev NLL at every epoch
|
| 99 |
+
- **`ternary/`** (`gdnbit-30m-t1` and `gdnbit-30m-t4` only): deployment weights. See below.
|
| 100 |
+
|
| 101 |
+
## Ternary deployment weights (`ternary/`)
|
| 102 |
+
|
| 103 |
+
The latent fp32 weights must not be re-quantized independently. About a fifth of them sit exactly on the rounding threshold |w| = 0.5·mean|w|, so a different floating-point summation order (for example CPU vs GPU) flips their ternary level and changes the model.
|
| 104 |
+
|
| 105 |
+
`ternary/ternary.safetensors` stores, for every BitLinear `<module>`, the levels and scale as computed on CUDA by the training kernel:
|
| 106 |
+
- `<module>.levels`: int8 values in {-1, 0, 1}, shape [out, in]
|
| 107 |
+
- `<module>.scale`: fp32 scalar, mean|W|
|
| 108 |
+
|
| 109 |
+
The forward pass is
|
| 110 |
+
|
| 111 |
+
y = Q8(RMSNorm(x; eps 1e-6) * norm.weight) @ (levels * scale)^T,
|
| 112 |
+
|
| 113 |
+
where Q8 is per-token absmax int8 quantization (127 / max|x|). The RMSNorm weights, embedding, A_log, dt_bias, output norm and loop weights r_t are the fp32 tensors in `model.safetensors`.
|
| 114 |
+
|
| 115 |
+
The fused attention input projection `model.layers.<i>.attn.in_proj` has 2316 output rows. In order: q 384 | k 384 | v 768 | g 768 | a 6 | b 6.
|
| 116 |
+
- a (decay) and b (write strength) are its last 12 rows. They share the matrix's single scale.
|
| 117 |
+
- After the projection: decay = exp(−exp(A_log)·softplus(a + dt_bias)), β = sigmoid(b), and q and k are L2-normalized.
|
| 118 |
+
|
| 119 |
+
The tied `lm_head` levels are quantized from the float embedding.
|
| 120 |
+
|
| 121 |
+
`ternary/verify.json` records the check: dev NLL of the model rebuilt from these files minus the native model's.
|
eval/.gitkeep
DELETED
|
File without changes
|
eval/all_runs.csv
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
folder,arch,weights,K,T,lr,seed,params,best_words_m,dev_nll,dev_bpw,test_nll,test_bpw,blimp,blimp_supplement,entity_tracking
|
| 2 |
+
gdn-30m-t1/babylm-strict-small-100Mw-s1337,gdn,fp,64,1,1e-2,1337,30453064,70,3.3981919687350586,8.36140506397954,3.306020510774677,8.263090470345386,71.44,63.82,27.8
|
| 3 |
+
gdn-30m-t1/babylm-strict-small-100Mw-s42,gdn,fp,64,1,1e-2,42,30453064,80,3.4753230730271207,8.55118963529028,3.4128869393076893,8.53019315901082,70.37,63.11,26.26
|
| 4 |
+
gdn-30m-t1/babylm-strict-small-100Mw-s7,gdn,fp,64,1,1e-2,7,30453064,70,3.4189122823832707,8.412388333041262,3.3520635961237866,8.378170875483095,70.09,59.33,18.87
|
| 5 |
+
gdn-30m-t4/babylm-strict-small-100Mw-s1337,gdn,fp,64,4,1e-2,1337,30455112,80,3.366958815790573,8.28455447825725,3.2683713757282624,8.168989962497697,72.08,61.02,21.1
|
| 6 |
+
gdn-30m-t4/babylm-strict-small-100Mw-s42,gdn,fp,64,4,1e-2,42,30455112,70,3.374812731624426,8.303879393456109,3.29265292421458,8.229679432284291,71.83,61.41,23.75
|
| 7 |
+
gdn-30m-t4/babylm-strict-small-100Mw-s7,gdn,fp,64,4,1e-2,7,30455112,70,3.3545820357881553,8.254100851169639,3.262191874658429,8.153544874896088,71.31,59.79,25.4
|
| 8 |
+
gdnbit-30m-k16-t1/babylm-strict-small-100Mw-s1337,gdnbit,ternary,16,1,2e-3,1337,28701000,100,3.619888644860621,8.906899764537533,3.556276118268278,8.888581062037547,69.8,61.49,22.15
|
| 9 |
+
gdnbit-30m-k16-t1/babylm-strict-small-100Mw-s42,gdnbit,ternary,16,1,2e-3,42,28701000,100,3.607290638466565,8.875901799905618,3.5391904586529166,8.845877046533971,69.76,58.87,27.09
|
| 10 |
+
gdnbit-30m-k16-t1/babylm-strict-small-100Mw-s7,gdnbit,ternary,16,1,2e-3,7,28701000,100,3.64464632006933,8.96781714435805,3.5920976477030937,8.978113639812348,68.94,61.4,14.86
|
| 11 |
+
gdnbit-30m-k16-t4/babylm-strict-small-100Mw-s1337,gdnbit,ternary,16,4,2e-3,1337,28703048,100,3.5281349707081193,8.68113569860013,3.434116658192837,8.583254864840002,70.31,59.28,13.81
|
| 12 |
+
gdnbit-30m-k16-t4/babylm-strict-small-100Mw-s42,gdnbit,ternary,16,4,2e-3,42,28703048,100,3.5567442233889603,8.751530059025654,3.484899289349601,8.710181323463877,70.29,60.85,17.68
|
| 13 |
+
gdnbit-30m-k16-t4/babylm-strict-small-100Mw-s7,gdnbit,ternary,16,4,2e-3,7,28703048,100,3.5210507137386737,8.663704563854408,3.4416328390107687,8.602040859024743,70.27,59.26,38.15
|
| 14 |
+
gdnbit-30m-k32-t1/babylm-strict-small-100Mw-s1337,gdnbit,ternary,32,1,2e-3,1337,29290824,100,3.5526512175453218,8.741459032990996,3.4671261010588403,8.665758894046272,70.37,58.57,15.13
|
| 15 |
+
gdnbit-30m-k32-t1/babylm-strict-small-100Mw-s42,gdnbit,ternary,32,1,2e-3,42,29290824,80,3.5935469612926596,8.842084860492873,3.525837901921341,8.812503574129963,69.21,59.95,25.95
|
| 16 |
+
gdnbit-30m-k32-t1/babylm-strict-small-100Mw-s7,gdnbit,ternary,32,1,2e-3,7,29290824,100,3.575910410070147,8.79868932838056,3.504925559433028,8.760235121056365,70.09,60.05,14.69
|
| 17 |
+
gdnbit-30m-k32-t4/babylm-strict-small-100Mw-s1337,gdnbit,ternary,32,4,2e-3,1337,29292872,100,3.5175602904714194,8.655116219508129,3.4511007423261355,8.625705001883018,71.25,57.32,14.31
|
| 18 |
+
gdnbit-30m-k32-t4/babylm-strict-small-100Mw-s42,gdnbit,ternary,32,4,2e-3,42,29292872,90,3.541202584295625,8.713289152975273,3.4586077169030593,8.644467985925475,70.9,63.09,35.51
|
| 19 |
+
gdnbit-30m-k32-t4/babylm-strict-small-100Mw-s7,gdnbit,ternary,32,4,2e-3,7,29292872,100,3.51420600384187,8.646862845517374,3.423609165503013,8.556992365127874,70.36,59.93,17.46
|
| 20 |
+
gdnbit-30m-t1/babylm-strict-small-100Mw-s1337,gdnbit,ternary,64,1,2e-3,1337,30470472,80,3.5588962936555246,8.756825325270306,3.482154962818182,8.703322134800231,70.62,57.87,14.69
|
| 21 |
+
gdnbit-30m-t1/babylm-strict-small-100Mw-s42,gdnbit,ternary,64,1,2e-3,42,30470472,80,3.543970719157588,8.72010027402598,3.470840721222194,8.675043241306891,68.37,59.11,20.38
|
| 22 |
+
gdnbit-30m-t1/babylm-strict-small-100Mw-s7,gdnbit,ternary,64,1,2e-3,7,30470472,100,3.5586369049031816,8.756187087511128,3.496725137431466,8.7397389297372,70.47,59.85,17.66
|
| 23 |
+
gdnbit-30m-t4/babylm-strict-small-100Mw-s1337,gdnbit,ternary,64,4,2e-3,1337,30472520,100,3.474096335259005,8.54817119151757,3.382425710637615,8.454058153944041,71.23,60.52,14.34
|
| 24 |
+
gdnbit-30m-t4/babylm-strict-small-100Mw-s42,gdnbit,ternary,64,4,2e-3,42,30472520,100,3.550867301726804,8.73706962742006,3.4809499018719006,8.700310197158299,71.22,60.08,14.03
|
| 25 |
+
gdnbit-30m-t4/babylm-strict-small-100Mw-s7,gdnbit,ternary,64,4,2e-3,7,30472520,100,3.5671160822195507,8.777050486872836,3.5002731352577756,8.74860681997921,71.18,61.17,39.21
|
| 26 |
+
hgrnbit-30m-t1/babylm-strict-small-100Mw-s1337,hgrnbit,ternary,,1,2e-3,1337,30566304,100,3.5380620995440593,8.705561848177107,3.4382046179562886,8.593472339672491,70.87,58.38,15.01
|
| 27 |
+
hgrnbit-30m-t1/babylm-strict-small-100Mw-s42,hgrnbit,ternary,,1,2e-3,42,30566304,100,3.601345236800297,8.86127287015206,3.5120533336162523,8.778050329018996,70.33,58.92,16.8
|
| 28 |
+
hgrnbit-30m-t1/babylm-strict-small-100Mw-s7,hgrnbit,ternary,,1,2e-3,7,30566304,100,3.5729909604587577,8.791505890543823,3.4790666710123377,8.695603236956932,69.56,58.66,17.99
|
| 29 |
+
hgrnbit-30m-t4/babylm-strict-small-100Mw-s1337,hgrnbit,ternary,,4,2e-3,1337,30568352,100,3.5581238514228573,8.754924696213655,3.463659601617893,8.657094701430832,69.36,59.91,19.14
|
| 30 |
+
hgrnbit-30m-t4/babylm-strict-small-100Mw-s42,hgrnbit,ternary,,4,2e-3,42,30568352,100,3.581190567047378,8.811681393482889,3.494257059901931,8.733570199734496,70.29,60.89,29.59
|
| 31 |
+
hgrnbit-30m-t4/babylm-strict-small-100Mw-s7,hgrnbit,ternary,,4,2e-3,7,30568352,100,3.5467213090061147,8.726868224777439,3.4455132920743003,8.61173969018002,71.02,59.9,37.79
|
| 32 |
+
"(LR sweep point, weights not published)",gdn,fp,64,1,2.4e-3,1337,30453064,40,3.4853981734489987,8.575979875648287,3.4047506315046543,8.509857215161729,70.14,59.01,23.04
|
| 33 |
+
"(LR sweep point, weights not published)",gdn,fp,64,1,4.8e-3,1337,30453064,50,3.455795997145237,8.50314238173142,3.382229113072728,8.453566776634185,70.71,57.61,25.41
|
| 34 |
+
"(LR sweep point, weights not published)",gdn,fp,64,1,9.6e-3,1337,30453064,60,3.4199902462985396,8.41504071201892,3.3342176780401367,8.333566664719438,70.61,59.78,19.03
|
| 35 |
+
"(LR sweep point, weights not published)",gdn,fp,64,1,9.6e-3,42,30453064,80,3.4356521429922413,8.453577517335127,3.3598007936253174,8.397509280291448,69.5,61.38,20.99
|
| 36 |
+
"(LR sweep point, weights not published)",gdn,fp,64,1,9.6e-3,7,30453064,70,3.414198370754207,8.400789540233488,3.344727789943151,8.359835710911081,70.3,62.08,29.93
|
eval/comparison.csv
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
model,params,lr,seeds,dev_nll_mean,dev_nll_std,test_nll_mean,test_nll_std,dev_bpw_mean,dev_bpw_std,test_bpw_mean,test_bpw_std,blimp_mean,blimp_std,blimp_supplement_mean,blimp_supplement_std,entity_tracking_mean,entity_tracking_std
|
| 2 |
+
gdnbit-30m-t1,30470472,2e-3,7 42 1337,3.5538,0.0085,3.4832,0.013,8.7444,0.021,8.706,0.0324,69.82,1.258,58.9433,1.0005,17.5767,2.8459
|
| 3 |
+
gdnbit-30m-t4,30472520,2e-3,7 42 1337,3.5307,0.0497,3.4545,0.0632,8.6874,0.1222,8.6343,0.158,71.21,0.0265,60.59,0.5484,22.5267,14.449
|
| 4 |
+
gdn-30m-t1,30453064,1e-2,7 42 1337,3.4308,0.0399,3.357,0.0536,8.4417,0.0982,8.3905,0.134,70.6333,0.7125,62.0867,2.4136,24.31,4.7737
|
| 5 |
+
gdn-30m-t4,30455112,1e-2,7 42 1337,3.3655,0.0102,3.2744,0.0161,8.2808,0.0251,8.1841,0.0402,71.74,0.3928,60.74,0.8455,23.4167,2.1693
|
| 6 |
+
hgrnbit-30m-t1,30566304,2e-3,7 42 1337,3.5708,0.0317,3.4764,0.037,8.7861,0.078,8.689,0.0925,70.2533,0.6584,58.6533,0.2701,16.6,1.5
|
| 7 |
+
hgrnbit-30m-t4,30568352,2e-3,7 42 1337,3.562,0.0176,3.4678,0.0246,8.7645,0.0432,8.6675,0.0616,70.2233,0.832,60.2333,0.5687,28.84,9.3476
|
| 8 |
+
gdnbit-30m-k16-t1,28701000,2e-3,7 42 1337,3.6239,0.019,3.5625,0.027,8.9169,0.0468,8.9042,0.0675,69.5,0.4854,60.5867,1.4874,21.3667,6.1525
|
| 9 |
+
gdnbit-30m-k16-t4,28703048,2e-3,7 42 1337,3.5353,0.0189,3.4535,0.0274,8.6988,0.0465,8.6318,0.0685,70.29,0.02,59.7967,0.9123,23.2133,13.0795
|
| 10 |
+
gdnbit-30m-k32-t1,29290824,2e-3,7 42 1337,3.574,0.0205,3.4993,0.0298,8.7941,0.0505,8.7462,0.0744,69.89,0.6053,59.5233,0.8271,18.59,6.3777
|
| 11 |
+
gdnbit-30m-k32-t4,29292872,2e-3,7 42 1337,3.5243,0.0147,3.4444,0.0184,8.6718,0.0362,8.6091,0.0461,70.8367,0.4484,60.1133,2.8894,22.4267,11.4394
|
eval/state_quant.csv
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
model,seed,checkpoint_words_m,state_format,dev_nll,delta_vs_fp32_state
|
| 2 |
+
gdnbit-30m-t1,1337,80,fp32,3.558262,0.0
|
| 3 |
+
gdnbit-30m-t1,1337,80,int8_row,3.562203,0.003941
|
| 4 |
+
gdnbit-30m-t1,1337,80,int8_tensor,3.5754,0.017139
|
| 5 |
+
gdnbit-30m-t1,1337,80,int4_row,3.61204,0.053778
|
| 6 |
+
gdnbit-30m-t4,1337,final,fp32,3.473066,0.0
|
| 7 |
+
gdnbit-30m-t4,1337,final,int8_row,3.474653,0.001588
|
| 8 |
+
gdnbit-30m-t4,1337,final,int8_tensor,3.475414,0.002349
|
| 9 |
+
gdnbit-30m-t4,1337,final,int4_row,3.516496,0.043431
|
| 10 |
+
hgrnbit-30m-t1,1337,final,fp32,3.536096,0.0
|
| 11 |
+
hgrnbit-30m-t1,1337,final,int8_row,3.536806,0.00071
|
| 12 |
+
hgrnbit-30m-t1,1337,final,int8_tensor,3.536806,0.00071
|
| 13 |
+
hgrnbit-30m-t1,1337,final,int4_row,3.736917,0.200821
|
| 14 |
+
hgrnbit-30m-t4,1337,final,fp32,3.556619,0.0
|
| 15 |
+
hgrnbit-30m-t4,1337,final,int8_row,3.558132,0.001514
|
| 16 |
+
hgrnbit-30m-t4,1337,final,int8_tensor,3.558132,0.001514
|
| 17 |
+
hgrnbit-30m-t4,1337,final,int4_row,3.802597,0.245978
|
| 18 |
+
gdn-30m-t1,1337,70,fp32,3.398702,0.0
|
| 19 |
+
gdn-30m-t1,1337,70,int8_row,3.399858,0.001155
|
| 20 |
+
gdn-30m-t1,1337,70,int8_tensor,3.422254,0.023551
|
| 21 |
+
gdn-30m-t1,1337,70,int4_row,3.457574,0.058872
|
| 22 |
+
gdn-30m-t4,1337,80,fp32,3.368218,0.0
|
| 23 |
+
gdn-30m-t4,1337,80,int8_row,3.367368,-0.00085
|
| 24 |
+
gdn-30m-t4,1337,80,int8_tensor,3.371211,0.002993
|
| 25 |
+
gdn-30m-t4,1337,80,int4_row,3.413218,0.045
|
gdn-30m-t1/babylm-strict-small-100Mw-s1337/config.json
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"GDNBitForCausalLM"
|
| 4 |
+
],
|
| 5 |
+
"attn_mode": "chunk",
|
| 6 |
+
"bitlinear": false,
|
| 7 |
+
"bos_token_id": 1,
|
| 8 |
+
"conv_size": 4,
|
| 9 |
+
"dtype": "float32",
|
| 10 |
+
"eos_token_id": 2,
|
| 11 |
+
"expand_v": 2,
|
| 12 |
+
"fuse_cross_entropy": true,
|
| 13 |
+
"fuse_linear_cross_entropy": false,
|
| 14 |
+
"fuse_norm": true,
|
| 15 |
+
"fused_proj": true,
|
| 16 |
+
"head_dim": 64,
|
| 17 |
+
"hidden_act": "relu2",
|
| 18 |
+
"hidden_size": 512,
|
| 19 |
+
"init_scheme": "trunc_normal",
|
| 20 |
+
"initializer_range": 0.02,
|
| 21 |
+
"intermediate_size": 2048,
|
| 22 |
+
"loop_count": 1,
|
| 23 |
+
"max_position_embeddings": 1024,
|
| 24 |
+
"model_type": "gdnbit",
|
| 25 |
+
"norm_eps": 1e-05,
|
| 26 |
+
"num_heads": 6,
|
| 27 |
+
"num_hidden_layers": 6,
|
| 28 |
+
"transformers_version": "4.57.3",
|
| 29 |
+
"use_cache": false,
|
| 30 |
+
"use_short_conv": false,
|
| 31 |
+
"vocab_size": 16384
|
| 32 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/args.json
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": "configs/babylm/gdn_30m_t1.json",
|
| 3 |
+
"data_dir": "data/babylm/tok16384-babylm-official",
|
| 4 |
+
"out_dir": "outputs/babylm30m_d512/gdn_30m_t1-lr1e-2-s1337",
|
| 5 |
+
"epochs": 10,
|
| 6 |
+
"batch_size": 32,
|
| 7 |
+
"micro_batch": null,
|
| 8 |
+
"seq_len": 512,
|
| 9 |
+
"lr": 0.01,
|
| 10 |
+
"min_lr_ratio": 0.0,
|
| 11 |
+
"warmup": 500,
|
| 12 |
+
"weight_decay": 0.1,
|
| 13 |
+
"lr_drop_at": 1.0,
|
| 14 |
+
"lr_drop_ratio": 0.6666666666666666,
|
| 15 |
+
"wd_drop_at": 1.0,
|
| 16 |
+
"beta1": 0.9,
|
| 17 |
+
"beta2": 0.95,
|
| 18 |
+
"grad_clip": 1.0,
|
| 19 |
+
"seed": 1337,
|
| 20 |
+
"log_interval": 50,
|
| 21 |
+
"epoch_eval_windows": 2048,
|
| 22 |
+
"save_words": "10,20,30,40,50,60,70,80,90,100",
|
| 23 |
+
"dtype": "bf16",
|
| 24 |
+
"compile": false,
|
| 25 |
+
"tokenizer": null,
|
| 26 |
+
"override": "{}",
|
| 27 |
+
"steps": 10320,
|
| 28 |
+
"params": 30453064,
|
| 29 |
+
"steps_per_epoch": 1032,
|
| 30 |
+
"words_per_token": 0.5910163039530657
|
| 31 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/babylm_eval_summary.json
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "outputs/babylm30m_d512/gdn_30m_t1-lr1e-2-s1337/ckpt/chck_70M",
|
| 3 |
+
"revision": "main",
|
| 4 |
+
"mode": "full",
|
| 5 |
+
"scores": {
|
| 6 |
+
"blimp": 71.44,
|
| 7 |
+
"blimp_supplement": 63.82,
|
| 8 |
+
"entity_tracking": 27.8
|
| 9 |
+
}
|
| 10 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/blimp_report.txt
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
TEMPERATURE: 1.00
|
| 2 |
+
|
| 3 |
+
### FIELD ACCURACY
|
| 4 |
+
syntax/semantics: 67.65
|
| 5 |
+
morphology: 81.08
|
| 6 |
+
syntax: 67.43
|
| 7 |
+
semantics: 69.97
|
| 8 |
+
|
| 9 |
+
### UID ACCURACY
|
| 10 |
+
expletive_it_object_raising: 68.51
|
| 11 |
+
anaphor_number_agreement: 95.92
|
| 12 |
+
wh_questions_subject_gap_long_distance: 84.60
|
| 13 |
+
existential_there_subject_raising: 76.62
|
| 14 |
+
anaphor_gender_agreement: 88.98
|
| 15 |
+
determiner_noun_agreement_irregular_2: 89.51
|
| 16 |
+
passive_2: 75.64
|
| 17 |
+
irregular_plural_subject_verb_agreement_2: 82.62
|
| 18 |
+
existential_there_quantifiers_1: 97.10
|
| 19 |
+
wh_vs_that_with_gap_long_distance: 11.98
|
| 20 |
+
determiner_noun_agreement_2: 92.27
|
| 21 |
+
sentential_negation_npi_licensor_present: 99.89
|
| 22 |
+
determiner_noun_agreement_with_adj_irregular_1: 77.02
|
| 23 |
+
wh_island: 76.77
|
| 24 |
+
drop_argument: 70.87
|
| 25 |
+
superlative_quantifiers_1: 89.27
|
| 26 |
+
principle_A_reconstruction: 41.16
|
| 27 |
+
principle_A_domain_1: 98.58
|
| 28 |
+
npi_present_2: 57.22
|
| 29 |
+
determiner_noun_agreement_with_adjective_1: 86.39
|
| 30 |
+
tough_vs_raising_2: 82.83
|
| 31 |
+
principle_A_case_2: 83.72
|
| 32 |
+
adjunct_island: 76.94
|
| 33 |
+
regular_plural_subject_verb_agreement_2: 77.04
|
| 34 |
+
distractor_agreement_relational_noun: 61.42
|
| 35 |
+
distractor_agreement_relative_clause: 45.01
|
| 36 |
+
causative: 65.16
|
| 37 |
+
determiner_noun_agreement_irregular_1: 82.09
|
| 38 |
+
transitive: 73.39
|
| 39 |
+
determiner_noun_agreement_1: 90.53
|
| 40 |
+
tough_vs_raising_1: 39.77
|
| 41 |
+
animate_subject_passive: 70.50
|
| 42 |
+
superlative_quantifiers_2: 81.24
|
| 43 |
+
passive_1: 82.62
|
| 44 |
+
determiner_noun_agreement_with_adj_2: 88.63
|
| 45 |
+
sentential_subject_island: 50.88
|
| 46 |
+
wh_questions_object_gap: 76.95
|
| 47 |
+
npi_present_1: 50.72
|
| 48 |
+
wh_vs_that_no_gap: 96.63
|
| 49 |
+
matrix_question_npi_licensor_present: 21.53
|
| 50 |
+
only_npi_scope: 71.57
|
| 51 |
+
inchoative: 51.93
|
| 52 |
+
irregular_plural_subject_verb_agreement_1: 71.52
|
| 53 |
+
principle_A_domain_2: 73.55
|
| 54 |
+
wh_vs_that_no_gap_long_distance: 96.57
|
| 55 |
+
left_branch_island_simple_question: 68.66
|
| 56 |
+
principle_A_domain_3: 61.00
|
| 57 |
+
intransitive: 65.90
|
| 58 |
+
coordinate_structure_constraint_complex_left_branch: 59.82
|
| 59 |
+
determiner_noun_agreement_with_adj_irregular_2: 85.36
|
| 60 |
+
irregular_past_participle_adjectives: 78.25
|
| 61 |
+
wh_vs_that_with_gap: 38.30
|
| 62 |
+
sentential_negation_npi_scope: 29.85
|
| 63 |
+
ellipsis_n_bar_1: 58.23
|
| 64 |
+
regular_plural_subject_verb_agreement_1: 75.96
|
| 65 |
+
ellipsis_n_bar_2: 67.27
|
| 66 |
+
wh_questions_subject_gap: 90.20
|
| 67 |
+
irregular_past_participle_verbs: 84.61
|
| 68 |
+
coordinate_structure_constraint_object_extraction: 63.54
|
| 69 |
+
principle_A_c_command: 53.07
|
| 70 |
+
existential_there_object_raising: 68.84
|
| 71 |
+
existential_there_quantifiers_2: 36.33
|
| 72 |
+
animate_subject_trans: 84.72
|
| 73 |
+
complex_NP_island: 41.61
|
| 74 |
+
left_branch_island_echo_question: 56.28
|
| 75 |
+
only_npi_licensor_present: 95.01
|
| 76 |
+
principle_A_case_1: 100.00
|
| 77 |
+
|
| 78 |
+
### LINGUISTICS_TERM ACCURACY
|
| 79 |
+
control_raising: 67.06
|
| 80 |
+
anaphor_agreement: 92.38
|
| 81 |
+
filler_gap_dependency: 70.14
|
| 82 |
+
determiner_noun_agreement: 86.90
|
| 83 |
+
argument_structure: 69.42
|
| 84 |
+
subject_verb_agreement: 69.21
|
| 85 |
+
quantifiers: 76.43
|
| 86 |
+
npi_licensing: 60.68
|
| 87 |
+
island_effects: 62.07
|
| 88 |
+
binding: 72.60
|
| 89 |
+
s-selection: 77.72
|
| 90 |
+
irregular_forms: 81.40
|
| 91 |
+
ellipsis: 62.82
|
| 92 |
+
|
| 93 |
+
### AVERAGE ACCURACY
|
| 94 |
+
71.44
|
| 95 |
+
|
gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/blimp_supplement_report.txt
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
TEMPERATURE: 1.00
|
| 2 |
+
|
| 3 |
+
### FIELD ACCURACY
|
| 4 |
+
supplement: 78.19
|
| 5 |
+
|
| 6 |
+
### UID ACCURACY
|
| 7 |
+
qa_congruence_easy: 65.62
|
| 8 |
+
turn_taking: 73.57
|
| 9 |
+
qa_congruence_tricky: 41.21
|
| 10 |
+
subject_aux_inversion: 85.83
|
| 11 |
+
hypernym: 52.85
|
| 12 |
+
|
| 13 |
+
### LINGUISTICS_TERM ACCURACY
|
| 14 |
+
supplement: 78.19
|
| 15 |
+
|
| 16 |
+
### AVERAGE ACCURACY
|
| 17 |
+
63.82
|
| 18 |
+
|
gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/dev_curve.log
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
`torch_dtype` is deprecated! Use `dtype` instead!
|
| 2 |
+
/home/ruhai/Projects/SSM_IMC_NATURE/.venv/lib/python3.12/site-packages/fla/layers/gated_deltanet.py:190: UserWarning: ShortConvolution is crucial to the performance. Do not turn it off, i.e., setting `use_short_conv=False` unless you know what you are doing.
|
| 3 |
+
warnings.warn(
|
| 4 |
+
{"ckpt": "chck_10M", "words_m": 10, "nll": 3.8562589579546875, "bpw": 9.488499612651719, "tokens": 17769472}
|
| 5 |
+
{"ckpt": "chck_20M", "words_m": 20, "nll": 3.633295057885126, "bpw": 8.939886850253773, "tokens": 17769472}
|
| 6 |
+
{"ckpt": "chck_30M", "words_m": 30, "nll": 3.5594808525402444, "bpw": 8.758263658850002, "tokens": 17769472}
|
| 7 |
+
{"ckpt": "chck_40M", "words_m": 40, "nll": 3.469812067511418, "bpw": 8.537629556916082, "tokens": 17769472}
|
| 8 |
+
{"ckpt": "chck_50M", "words_m": 50, "nll": 3.436955697349751, "bpw": 8.456784971801012, "tokens": 17769472}
|
| 9 |
+
{"ckpt": "chck_60M", "words_m": 60, "nll": 3.4035214972772194, "bpw": 8.374518610050815, "tokens": 17769472}
|
| 10 |
+
{"ckpt": "chck_70M", "words_m": 70, "nll": 3.3981919687350586, "bpw": 8.36140506397954, "tokens": 17769472}
|
| 11 |
+
{"ckpt": "chck_80M", "words_m": 80, "nll": 3.4426857543762415, "bpw": 8.470884036326794, "tokens": 17769472}
|
| 12 |
+
{"ckpt": "chck_90M", "words_m": 90, "nll": 3.503730866693732, "bpw": 8.621088296697835, "tokens": 17769472}
|
| 13 |
+
{"ckpt": "hf", "words_m": 100, "nll": 3.5855714520659014, "bpw": 8.822460759250218, "tokens": 17769472}
|
| 14 |
+
BEST {"ckpt": "chck_70M", "words_m": 70, "nll": 3.3981919687350586, "bpw": 8.36140506397954, "tokens": 17769472}
|
gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/entity_tracking_report.txt
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
TEMPERATURE: 1.00
|
| 2 |
+
|
| 3 |
+
### UID ACCURACY
|
| 4 |
+
move_contents_0_ops: 23.97
|
| 5 |
+
move_contents_1_ops: 23.43
|
| 6 |
+
move_contents_2_ops: 30.25
|
| 7 |
+
move_contents_3_ops: 28.88
|
| 8 |
+
move_contents_4_ops: 30.06
|
| 9 |
+
move_contents_5_ops: 30.77
|
| 10 |
+
ambiref_0_ops: 25.54
|
| 11 |
+
ambiref_1_ops: 25.37
|
| 12 |
+
ambiref_2_ops: 32.45
|
| 13 |
+
ambiref_3_ops: 28.48
|
| 14 |
+
ambiref_4_ops: 27.32
|
| 15 |
+
ambiref_5_ops: 26.20
|
| 16 |
+
regular_0_ops: 23.60
|
| 17 |
+
regular_1_ops: 20.92
|
| 18 |
+
regular_2_ops: 26.07
|
| 19 |
+
regular_3_ops: 31.35
|
| 20 |
+
regular_4_ops: 30.46
|
| 21 |
+
regular_5_ops: 35.22
|
| 22 |
+
regular: 27.94
|
| 23 |
+
ambiref: 27.56
|
| 24 |
+
move_contents: 27.89
|
| 25 |
+
|
| 26 |
+
### AVERAGE ACCURACY
|
| 27 |
+
27.80
|
| 28 |
+
|
gdn-30m-t1/babylm-strict-small-100Mw-s1337/eval/final_summary.json
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"run": "gdn_30m_t1-lr1e-2-s1337",
|
| 3 |
+
"params": 30453064,
|
| 4 |
+
"best_ckpt": "chck_70M",
|
| 5 |
+
"best_words_m": 70,
|
| 6 |
+
"dev_nll": 3.3981919687350586,
|
| 7 |
+
"dev_bpw": 8.36140506397954,
|
| 8 |
+
"test_nll": 3.306020510774677,
|
| 9 |
+
"test_bpw": 8.263090470345386,
|
| 10 |
+
"blimp": 71.44,
|
| 11 |
+
"blimp_supplement": 63.82,
|
| 12 |
+
"entity_tracking": 27.8
|
| 13 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s1337/generation_config.json
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_from_model_config": true,
|
| 3 |
+
"bos_token_id": 1,
|
| 4 |
+
"eos_token_id": 2,
|
| 5 |
+
"transformers_version": "4.57.3",
|
| 6 |
+
"use_cache": false
|
| 7 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s1337/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:424b3a94a35ea91a1f08811f84a37ee8a6adf60935b1bcbcfb16f877eda300c7
|
| 3 |
+
size 121817944
|
gdn-30m-t1/babylm-strict-small-100Mw-s1337/special_tokens_map.json
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token": {
|
| 3 |
+
"content": "<s>",
|
| 4 |
+
"lstrip": false,
|
| 5 |
+
"normalized": false,
|
| 6 |
+
"rstrip": false,
|
| 7 |
+
"single_word": false
|
| 8 |
+
},
|
| 9 |
+
"cls_token": {
|
| 10 |
+
"content": "<s>",
|
| 11 |
+
"lstrip": false,
|
| 12 |
+
"normalized": false,
|
| 13 |
+
"rstrip": false,
|
| 14 |
+
"single_word": false
|
| 15 |
+
},
|
| 16 |
+
"eos_token": {
|
| 17 |
+
"content": "</s>",
|
| 18 |
+
"lstrip": false,
|
| 19 |
+
"normalized": false,
|
| 20 |
+
"rstrip": false,
|
| 21 |
+
"single_word": false
|
| 22 |
+
},
|
| 23 |
+
"mask_token": {
|
| 24 |
+
"content": "<mask>",
|
| 25 |
+
"lstrip": false,
|
| 26 |
+
"normalized": false,
|
| 27 |
+
"rstrip": false,
|
| 28 |
+
"single_word": false
|
| 29 |
+
},
|
| 30 |
+
"pad_token": {
|
| 31 |
+
"content": "<pad>",
|
| 32 |
+
"lstrip": false,
|
| 33 |
+
"normalized": false,
|
| 34 |
+
"rstrip": false,
|
| 35 |
+
"single_word": false
|
| 36 |
+
},
|
| 37 |
+
"sep_token": {
|
| 38 |
+
"content": "</s>",
|
| 39 |
+
"lstrip": false,
|
| 40 |
+
"normalized": false,
|
| 41 |
+
"rstrip": false,
|
| 42 |
+
"single_word": false
|
| 43 |
+
},
|
| 44 |
+
"unk_token": {
|
| 45 |
+
"content": "<unk>",
|
| 46 |
+
"lstrip": false,
|
| 47 |
+
"normalized": false,
|
| 48 |
+
"rstrip": false,
|
| 49 |
+
"single_word": false
|
| 50 |
+
}
|
| 51 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s1337/tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
gdn-30m-t1/babylm-strict-small-100Mw-s1337/tokenizer_config.json
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"added_tokens_decoder": {
|
| 3 |
+
"0": {
|
| 4 |
+
"content": "<unk>",
|
| 5 |
+
"lstrip": false,
|
| 6 |
+
"normalized": false,
|
| 7 |
+
"rstrip": false,
|
| 8 |
+
"single_word": false,
|
| 9 |
+
"special": true
|
| 10 |
+
},
|
| 11 |
+
"1": {
|
| 12 |
+
"content": "<s>",
|
| 13 |
+
"lstrip": false,
|
| 14 |
+
"normalized": false,
|
| 15 |
+
"rstrip": false,
|
| 16 |
+
"single_word": false,
|
| 17 |
+
"special": true
|
| 18 |
+
},
|
| 19 |
+
"2": {
|
| 20 |
+
"content": "</s>",
|
| 21 |
+
"lstrip": false,
|
| 22 |
+
"normalized": false,
|
| 23 |
+
"rstrip": false,
|
| 24 |
+
"single_word": false,
|
| 25 |
+
"special": true
|
| 26 |
+
},
|
| 27 |
+
"3": {
|
| 28 |
+
"content": "<pad>",
|
| 29 |
+
"lstrip": false,
|
| 30 |
+
"normalized": false,
|
| 31 |
+
"rstrip": false,
|
| 32 |
+
"single_word": false,
|
| 33 |
+
"special": true
|
| 34 |
+
},
|
| 35 |
+
"4": {
|
| 36 |
+
"content": "<mask>",
|
| 37 |
+
"lstrip": false,
|
| 38 |
+
"normalized": false,
|
| 39 |
+
"rstrip": false,
|
| 40 |
+
"single_word": false,
|
| 41 |
+
"special": true
|
| 42 |
+
}
|
| 43 |
+
},
|
| 44 |
+
"bos_token": "<s>",
|
| 45 |
+
"clean_up_tokenization_spaces": false,
|
| 46 |
+
"cls_token": "<s>",
|
| 47 |
+
"eos_token": "</s>",
|
| 48 |
+
"extra_special_tokens": {},
|
| 49 |
+
"mask_token": "<mask>",
|
| 50 |
+
"model_max_length": 1000000000000000019884624838656,
|
| 51 |
+
"pad_token": "<pad>",
|
| 52 |
+
"sep_token": "</s>",
|
| 53 |
+
"tokenizer_class": "PreTrainedTokenizerFast",
|
| 54 |
+
"unk_token": "<unk>"
|
| 55 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s42/config.json
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"GDNBitForCausalLM"
|
| 4 |
+
],
|
| 5 |
+
"attn_mode": "chunk",
|
| 6 |
+
"bitlinear": false,
|
| 7 |
+
"bos_token_id": 1,
|
| 8 |
+
"conv_size": 4,
|
| 9 |
+
"dtype": "float32",
|
| 10 |
+
"eos_token_id": 2,
|
| 11 |
+
"expand_v": 2,
|
| 12 |
+
"fuse_cross_entropy": true,
|
| 13 |
+
"fuse_linear_cross_entropy": false,
|
| 14 |
+
"fuse_norm": true,
|
| 15 |
+
"fused_proj": true,
|
| 16 |
+
"head_dim": 64,
|
| 17 |
+
"hidden_act": "relu2",
|
| 18 |
+
"hidden_size": 512,
|
| 19 |
+
"init_scheme": "trunc_normal",
|
| 20 |
+
"initializer_range": 0.02,
|
| 21 |
+
"intermediate_size": 2048,
|
| 22 |
+
"loop_count": 1,
|
| 23 |
+
"max_position_embeddings": 1024,
|
| 24 |
+
"model_type": "gdnbit",
|
| 25 |
+
"norm_eps": 1e-05,
|
| 26 |
+
"num_heads": 6,
|
| 27 |
+
"num_hidden_layers": 6,
|
| 28 |
+
"transformers_version": "4.57.3",
|
| 29 |
+
"use_cache": false,
|
| 30 |
+
"use_short_conv": false,
|
| 31 |
+
"vocab_size": 16384
|
| 32 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/args.json
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": "configs/babylm/gdn_30m_t1.json",
|
| 3 |
+
"data_dir": "data/babylm/tok16384-babylm-official",
|
| 4 |
+
"out_dir": "outputs/babylm30m_d512/gdn_30m_t1-lr1e-2-s42",
|
| 5 |
+
"epochs": 10,
|
| 6 |
+
"batch_size": 32,
|
| 7 |
+
"micro_batch": null,
|
| 8 |
+
"seq_len": 512,
|
| 9 |
+
"lr": 0.01,
|
| 10 |
+
"min_lr_ratio": 0.0,
|
| 11 |
+
"warmup": 500,
|
| 12 |
+
"weight_decay": 0.1,
|
| 13 |
+
"lr_drop_at": 1.0,
|
| 14 |
+
"lr_drop_ratio": 0.6666666666666666,
|
| 15 |
+
"wd_drop_at": 1.0,
|
| 16 |
+
"beta1": 0.9,
|
| 17 |
+
"beta2": 0.95,
|
| 18 |
+
"grad_clip": 1.0,
|
| 19 |
+
"seed": 42,
|
| 20 |
+
"log_interval": 50,
|
| 21 |
+
"epoch_eval_windows": 2048,
|
| 22 |
+
"save_words": "10,20,30,40,50,60,70,80,90,100",
|
| 23 |
+
"dtype": "bf16",
|
| 24 |
+
"compile": false,
|
| 25 |
+
"tokenizer": null,
|
| 26 |
+
"override": "{}",
|
| 27 |
+
"steps": 10320,
|
| 28 |
+
"params": 30453064,
|
| 29 |
+
"steps_per_epoch": 1032,
|
| 30 |
+
"words_per_token": 0.5910163039530657
|
| 31 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/babylm_eval_summary.json
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "outputs/babylm30m_d512/gdn_30m_t1-lr1e-2-s42/ckpt/chck_80M",
|
| 3 |
+
"revision": "main",
|
| 4 |
+
"mode": "full",
|
| 5 |
+
"scores": {
|
| 6 |
+
"blimp": 70.37,
|
| 7 |
+
"blimp_supplement": 63.11,
|
| 8 |
+
"entity_tracking": 26.26
|
| 9 |
+
}
|
| 10 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/blimp_report.txt
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
TEMPERATURE: 1.00
|
| 2 |
+
|
| 3 |
+
### FIELD ACCURACY
|
| 4 |
+
syntax/semantics: 66.72
|
| 5 |
+
morphology: 80.80
|
| 6 |
+
syntax: 67.97
|
| 7 |
+
semantics: 61.89
|
| 8 |
+
|
| 9 |
+
### UID ACCURACY
|
| 10 |
+
expletive_it_object_raising: 73.65
|
| 11 |
+
anaphor_number_agreement: 96.46
|
| 12 |
+
wh_questions_subject_gap_long_distance: 83.55
|
| 13 |
+
existential_there_subject_raising: 81.28
|
| 14 |
+
anaphor_gender_agreement: 94.54
|
| 15 |
+
determiner_noun_agreement_irregular_2: 86.22
|
| 16 |
+
passive_2: 77.19
|
| 17 |
+
irregular_plural_subject_verb_agreement_2: 82.29
|
| 18 |
+
existential_there_quantifiers_1: 94.73
|
| 19 |
+
wh_vs_that_with_gap_long_distance: 15.71
|
| 20 |
+
determiner_noun_agreement_2: 92.16
|
| 21 |
+
sentential_negation_npi_licensor_present: 100.00
|
| 22 |
+
determiner_noun_agreement_with_adj_irregular_1: 82.59
|
| 23 |
+
wh_island: 69.27
|
| 24 |
+
drop_argument: 70.22
|
| 25 |
+
superlative_quantifiers_1: 53.42
|
| 26 |
+
principle_A_reconstruction: 31.85
|
| 27 |
+
principle_A_domain_1: 95.62
|
| 28 |
+
npi_present_2: 52.52
|
| 29 |
+
determiner_noun_agreement_with_adjective_1: 87.25
|
| 30 |
+
tough_vs_raising_2: 85.65
|
| 31 |
+
principle_A_case_2: 87.43
|
| 32 |
+
adjunct_island: 69.29
|
| 33 |
+
regular_plural_subject_verb_agreement_2: 79.47
|
| 34 |
+
distractor_agreement_relational_noun: 62.18
|
| 35 |
+
distractor_agreement_relative_clause: 39.95
|
| 36 |
+
causative: 63.33
|
| 37 |
+
determiner_noun_agreement_irregular_1: 77.24
|
| 38 |
+
transitive: 74.77
|
| 39 |
+
determiner_noun_agreement_1: 91.28
|
| 40 |
+
tough_vs_raising_1: 35.65
|
| 41 |
+
animate_subject_passive: 65.59
|
| 42 |
+
superlative_quantifiers_2: 58.01
|
| 43 |
+
passive_1: 83.45
|
| 44 |
+
determiner_noun_agreement_with_adj_2: 86.50
|
| 45 |
+
sentential_subject_island: 55.36
|
| 46 |
+
wh_questions_object_gap: 73.92
|
| 47 |
+
npi_present_1: 49.17
|
| 48 |
+
wh_vs_that_no_gap: 89.20
|
| 49 |
+
matrix_question_npi_licensor_present: 18.84
|
| 50 |
+
only_npi_scope: 56.39
|
| 51 |
+
inchoative: 52.87
|
| 52 |
+
irregular_plural_subject_verb_agreement_1: 78.86
|
| 53 |
+
principle_A_domain_2: 63.06
|
| 54 |
+
wh_vs_that_no_gap_long_distance: 91.77
|
| 55 |
+
left_branch_island_simple_question: 78.97
|
| 56 |
+
principle_A_domain_3: 57.92
|
| 57 |
+
intransitive: 65.90
|
| 58 |
+
coordinate_structure_constraint_complex_left_branch: 60.49
|
| 59 |
+
determiner_noun_agreement_with_adj_irregular_2: 82.14
|
| 60 |
+
irregular_past_participle_adjectives: 65.24
|
| 61 |
+
wh_vs_that_with_gap: 47.01
|
| 62 |
+
sentential_negation_npi_scope: 38.81
|
| 63 |
+
ellipsis_n_bar_1: 65.84
|
| 64 |
+
regular_plural_subject_verb_agreement_1: 80.79
|
| 65 |
+
ellipsis_n_bar_2: 68.84
|
| 66 |
+
wh_questions_subject_gap: 87.86
|
| 67 |
+
irregular_past_participle_verbs: 83.65
|
| 68 |
+
coordinate_structure_constraint_object_extraction: 69.86
|
| 69 |
+
principle_A_c_command: 54.55
|
| 70 |
+
existential_there_object_raising: 75.99
|
| 71 |
+
existential_there_quantifiers_2: 40.61
|
| 72 |
+
animate_subject_trans: 86.35
|
| 73 |
+
complex_NP_island: 44.44
|
| 74 |
+
left_branch_island_echo_question: 58.18
|
| 75 |
+
only_npi_licensor_present: 91.38
|
| 76 |
+
principle_A_case_1: 99.89
|
| 77 |
+
|
| 78 |
+
### LINGUISTICS_TERM ACCURACY
|
| 79 |
+
control_raising: 69.97
|
| 80 |
+
anaphor_agreement: 95.48
|
| 81 |
+
filler_gap_dependency: 69.36
|
| 82 |
+
determiner_noun_agreement: 86.12
|
| 83 |
+
argument_structure: 69.75
|
| 84 |
+
subject_verb_agreement: 70.83
|
| 85 |
+
quantifiers: 61.64
|
| 86 |
+
npi_licensing: 58.09
|
| 87 |
+
island_effects: 63.49
|
| 88 |
+
binding: 69.60
|
| 89 |
+
s-selection: 76.13
|
| 90 |
+
irregular_forms: 74.36
|
| 91 |
+
ellipsis: 67.36
|
| 92 |
+
|
| 93 |
+
### AVERAGE ACCURACY
|
| 94 |
+
70.37
|
| 95 |
+
|
gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/blimp_supplement_report.txt
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
TEMPERATURE: 1.00
|
| 2 |
+
|
| 3 |
+
### FIELD ACCURACY
|
| 4 |
+
supplement: 76.37
|
| 5 |
+
|
| 6 |
+
### UID ACCURACY
|
| 7 |
+
qa_congruence_easy: 65.62
|
| 8 |
+
turn_taking: 68.57
|
| 9 |
+
qa_congruence_tricky: 46.67
|
| 10 |
+
subject_aux_inversion: 83.97
|
| 11 |
+
hypernym: 50.71
|
| 12 |
+
|
| 13 |
+
### LINGUISTICS_TERM ACCURACY
|
| 14 |
+
supplement: 76.37
|
| 15 |
+
|
| 16 |
+
### AVERAGE ACCURACY
|
| 17 |
+
63.11
|
| 18 |
+
|
gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/dev_curve.log
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
`torch_dtype` is deprecated! Use `dtype` instead!
|
| 2 |
+
/home/ruhai/Projects/SSM_IMC_NATURE/.venv/lib/python3.12/site-packages/fla/layers/gated_deltanet.py:190: UserWarning: ShortConvolution is crucial to the performance. Do not turn it off, i.e., setting `use_short_conv=False` unless you know what you are doing.
|
| 3 |
+
warnings.warn(
|
| 4 |
+
{"ckpt": "chck_10M", "words_m": 10, "nll": 3.8780883266308503, "bpw": 9.54221176178033, "tokens": 17769472}
|
| 5 |
+
{"ckpt": "chck_20M", "words_m": 20, "nll": 3.655319560867685, "bpw": 8.994079136170612, "tokens": 17769472}
|
| 6 |
+
{"ckpt": "chck_30M", "words_m": 30, "nll": 3.613189694847503, "bpw": 8.890416694988087, "tokens": 17769472}
|
| 7 |
+
{"ckpt": "chck_40M", "words_m": 40, "nll": 3.5164252370054947, "bpw": 8.652323369108508, "tokens": 17769472}
|
| 8 |
+
{"ckpt": "chck_50M", "words_m": 50, "nll": 3.5224081582516913, "bpw": 8.667044617485763, "tokens": 17769472}
|
| 9 |
+
{"ckpt": "chck_60M", "words_m": 60, "nll": 3.504233767689116, "bpw": 8.622325707346844, "tokens": 17769472}
|
| 10 |
+
{"ckpt": "chck_70M", "words_m": 70, "nll": 3.5036044980595795, "bpw": 8.620777360956932, "tokens": 17769472}
|
| 11 |
+
{"ckpt": "chck_80M", "words_m": 80, "nll": 3.4753230730271207, "bpw": 8.55118963529028, "tokens": 17769472}
|
| 12 |
+
{"ckpt": "chck_90M", "words_m": 90, "nll": 3.5776454737802914, "bpw": 8.802958531128903, "tokens": 17769472}
|
| 13 |
+
{"ckpt": "hf", "words_m": 100, "nll": 3.6450934552159877, "bpw": 8.968917340613572, "tokens": 17769472}
|
| 14 |
+
BEST {"ckpt": "chck_80M", "words_m": 80, "nll": 3.4753230730271207, "bpw": 8.55118963529028, "tokens": 17769472}
|
gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/entity_tracking_report.txt
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
TEMPERATURE: 1.00
|
| 2 |
+
|
| 3 |
+
### UID ACCURACY
|
| 4 |
+
move_contents_0_ops: 22.48
|
| 5 |
+
move_contents_1_ops: 20.30
|
| 6 |
+
move_contents_2_ops: 26.12
|
| 7 |
+
move_contents_3_ops: 26.40
|
| 8 |
+
move_contents_4_ops: 27.41
|
| 9 |
+
move_contents_5_ops: 25.64
|
| 10 |
+
ambiref_0_ops: 23.56
|
| 11 |
+
ambiref_1_ops: 31.14
|
| 12 |
+
ambiref_2_ops: 27.48
|
| 13 |
+
ambiref_3_ops: 35.43
|
| 14 |
+
ambiref_4_ops: 28.62
|
| 15 |
+
ambiref_5_ops: 36.90
|
| 16 |
+
regular_0_ops: 18.98
|
| 17 |
+
regular_1_ops: 20.26
|
| 18 |
+
regular_2_ops: 25.08
|
| 19 |
+
regular_3_ops: 26.24
|
| 20 |
+
regular_4_ops: 25.53
|
| 21 |
+
regular_5_ops: 25.16
|
| 22 |
+
regular: 23.54
|
| 23 |
+
ambiref: 30.52
|
| 24 |
+
move_contents: 24.72
|
| 25 |
+
|
| 26 |
+
### AVERAGE ACCURACY
|
| 27 |
+
26.26
|
| 28 |
+
|
gdn-30m-t1/babylm-strict-small-100Mw-s42/eval/final_summary.json
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"run": "gdn_30m_t1-lr1e-2-s42",
|
| 3 |
+
"params": 30453064,
|
| 4 |
+
"best_ckpt": "chck_80M",
|
| 5 |
+
"best_words_m": 80,
|
| 6 |
+
"dev_nll": 3.4753230730271207,
|
| 7 |
+
"dev_bpw": 8.55118963529028,
|
| 8 |
+
"test_nll": 3.4128869393076893,
|
| 9 |
+
"test_bpw": 8.53019315901082,
|
| 10 |
+
"blimp": 70.37,
|
| 11 |
+
"blimp_supplement": 63.11,
|
| 12 |
+
"entity_tracking": 26.26
|
| 13 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s42/generation_config.json
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_from_model_config": true,
|
| 3 |
+
"bos_token_id": 1,
|
| 4 |
+
"eos_token_id": 2,
|
| 5 |
+
"transformers_version": "4.57.3",
|
| 6 |
+
"use_cache": false
|
| 7 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s42/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:df3b49dc997b22d2425682b43901e833a629cea76d8ab21fdfc32a01126f6329
|
| 3 |
+
size 121817944
|
gdn-30m-t1/babylm-strict-small-100Mw-s42/special_tokens_map.json
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token": {
|
| 3 |
+
"content": "<s>",
|
| 4 |
+
"lstrip": false,
|
| 5 |
+
"normalized": false,
|
| 6 |
+
"rstrip": false,
|
| 7 |
+
"single_word": false
|
| 8 |
+
},
|
| 9 |
+
"cls_token": {
|
| 10 |
+
"content": "<s>",
|
| 11 |
+
"lstrip": false,
|
| 12 |
+
"normalized": false,
|
| 13 |
+
"rstrip": false,
|
| 14 |
+
"single_word": false
|
| 15 |
+
},
|
| 16 |
+
"eos_token": {
|
| 17 |
+
"content": "</s>",
|
| 18 |
+
"lstrip": false,
|
| 19 |
+
"normalized": false,
|
| 20 |
+
"rstrip": false,
|
| 21 |
+
"single_word": false
|
| 22 |
+
},
|
| 23 |
+
"mask_token": {
|
| 24 |
+
"content": "<mask>",
|
| 25 |
+
"lstrip": false,
|
| 26 |
+
"normalized": false,
|
| 27 |
+
"rstrip": false,
|
| 28 |
+
"single_word": false
|
| 29 |
+
},
|
| 30 |
+
"pad_token": {
|
| 31 |
+
"content": "<pad>",
|
| 32 |
+
"lstrip": false,
|
| 33 |
+
"normalized": false,
|
| 34 |
+
"rstrip": false,
|
| 35 |
+
"single_word": false
|
| 36 |
+
},
|
| 37 |
+
"sep_token": {
|
| 38 |
+
"content": "</s>",
|
| 39 |
+
"lstrip": false,
|
| 40 |
+
"normalized": false,
|
| 41 |
+
"rstrip": false,
|
| 42 |
+
"single_word": false
|
| 43 |
+
},
|
| 44 |
+
"unk_token": {
|
| 45 |
+
"content": "<unk>",
|
| 46 |
+
"lstrip": false,
|
| 47 |
+
"normalized": false,
|
| 48 |
+
"rstrip": false,
|
| 49 |
+
"single_word": false
|
| 50 |
+
}
|
| 51 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s42/tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
gdn-30m-t1/babylm-strict-small-100Mw-s42/tokenizer_config.json
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"added_tokens_decoder": {
|
| 3 |
+
"0": {
|
| 4 |
+
"content": "<unk>",
|
| 5 |
+
"lstrip": false,
|
| 6 |
+
"normalized": false,
|
| 7 |
+
"rstrip": false,
|
| 8 |
+
"single_word": false,
|
| 9 |
+
"special": true
|
| 10 |
+
},
|
| 11 |
+
"1": {
|
| 12 |
+
"content": "<s>",
|
| 13 |
+
"lstrip": false,
|
| 14 |
+
"normalized": false,
|
| 15 |
+
"rstrip": false,
|
| 16 |
+
"single_word": false,
|
| 17 |
+
"special": true
|
| 18 |
+
},
|
| 19 |
+
"2": {
|
| 20 |
+
"content": "</s>",
|
| 21 |
+
"lstrip": false,
|
| 22 |
+
"normalized": false,
|
| 23 |
+
"rstrip": false,
|
| 24 |
+
"single_word": false,
|
| 25 |
+
"special": true
|
| 26 |
+
},
|
| 27 |
+
"3": {
|
| 28 |
+
"content": "<pad>",
|
| 29 |
+
"lstrip": false,
|
| 30 |
+
"normalized": false,
|
| 31 |
+
"rstrip": false,
|
| 32 |
+
"single_word": false,
|
| 33 |
+
"special": true
|
| 34 |
+
},
|
| 35 |
+
"4": {
|
| 36 |
+
"content": "<mask>",
|
| 37 |
+
"lstrip": false,
|
| 38 |
+
"normalized": false,
|
| 39 |
+
"rstrip": false,
|
| 40 |
+
"single_word": false,
|
| 41 |
+
"special": true
|
| 42 |
+
}
|
| 43 |
+
},
|
| 44 |
+
"bos_token": "<s>",
|
| 45 |
+
"clean_up_tokenization_spaces": false,
|
| 46 |
+
"cls_token": "<s>",
|
| 47 |
+
"eos_token": "</s>",
|
| 48 |
+
"extra_special_tokens": {},
|
| 49 |
+
"mask_token": "<mask>",
|
| 50 |
+
"model_max_length": 1000000000000000019884624838656,
|
| 51 |
+
"pad_token": "<pad>",
|
| 52 |
+
"sep_token": "</s>",
|
| 53 |
+
"tokenizer_class": "PreTrainedTokenizerFast",
|
| 54 |
+
"unk_token": "<unk>"
|
| 55 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s7/config.json
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"GDNBitForCausalLM"
|
| 4 |
+
],
|
| 5 |
+
"attn_mode": "chunk",
|
| 6 |
+
"bitlinear": false,
|
| 7 |
+
"bos_token_id": 1,
|
| 8 |
+
"conv_size": 4,
|
| 9 |
+
"dtype": "float32",
|
| 10 |
+
"eos_token_id": 2,
|
| 11 |
+
"expand_v": 2,
|
| 12 |
+
"fuse_cross_entropy": true,
|
| 13 |
+
"fuse_linear_cross_entropy": false,
|
| 14 |
+
"fuse_norm": true,
|
| 15 |
+
"fused_proj": true,
|
| 16 |
+
"head_dim": 64,
|
| 17 |
+
"hidden_act": "relu2",
|
| 18 |
+
"hidden_size": 512,
|
| 19 |
+
"init_scheme": "trunc_normal",
|
| 20 |
+
"initializer_range": 0.02,
|
| 21 |
+
"intermediate_size": 2048,
|
| 22 |
+
"loop_count": 1,
|
| 23 |
+
"max_position_embeddings": 1024,
|
| 24 |
+
"model_type": "gdnbit",
|
| 25 |
+
"norm_eps": 1e-05,
|
| 26 |
+
"num_heads": 6,
|
| 27 |
+
"num_hidden_layers": 6,
|
| 28 |
+
"transformers_version": "4.57.3",
|
| 29 |
+
"use_cache": false,
|
| 30 |
+
"use_short_conv": false,
|
| 31 |
+
"vocab_size": 16384
|
| 32 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/args.json
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": "configs/babylm/gdn_30m_t1.json",
|
| 3 |
+
"data_dir": "data/babylm/tok16384-babylm-official",
|
| 4 |
+
"out_dir": "outputs/babylm30m_d512/gdn_30m_t1-lr1e-2-s7",
|
| 5 |
+
"epochs": 10,
|
| 6 |
+
"batch_size": 32,
|
| 7 |
+
"micro_batch": null,
|
| 8 |
+
"seq_len": 512,
|
| 9 |
+
"lr": 0.01,
|
| 10 |
+
"min_lr_ratio": 0.0,
|
| 11 |
+
"warmup": 500,
|
| 12 |
+
"weight_decay": 0.1,
|
| 13 |
+
"lr_drop_at": 1.0,
|
| 14 |
+
"lr_drop_ratio": 0.6666666666666666,
|
| 15 |
+
"wd_drop_at": 1.0,
|
| 16 |
+
"beta1": 0.9,
|
| 17 |
+
"beta2": 0.95,
|
| 18 |
+
"grad_clip": 1.0,
|
| 19 |
+
"seed": 7,
|
| 20 |
+
"log_interval": 50,
|
| 21 |
+
"epoch_eval_windows": 2048,
|
| 22 |
+
"save_words": "10,20,30,40,50,60,70,80,90,100",
|
| 23 |
+
"dtype": "bf16",
|
| 24 |
+
"compile": false,
|
| 25 |
+
"tokenizer": null,
|
| 26 |
+
"override": "{}",
|
| 27 |
+
"steps": 10320,
|
| 28 |
+
"params": 30453064,
|
| 29 |
+
"steps_per_epoch": 1032,
|
| 30 |
+
"words_per_token": 0.5910163039530657
|
| 31 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/babylm_eval_summary.json
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "outputs/babylm30m_d512/gdn_30m_t1-lr1e-2-s7/ckpt/chck_70M",
|
| 3 |
+
"revision": "main",
|
| 4 |
+
"mode": "full",
|
| 5 |
+
"scores": {
|
| 6 |
+
"blimp": 70.09,
|
| 7 |
+
"blimp_supplement": 59.33,
|
| 8 |
+
"entity_tracking": 18.87
|
| 9 |
+
}
|
| 10 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/blimp_report.txt
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
TEMPERATURE: 1.00
|
| 2 |
+
|
| 3 |
+
### FIELD ACCURACY
|
| 4 |
+
syntax/semantics: 66.00
|
| 5 |
+
morphology: 80.48
|
| 6 |
+
syntax: 66.55
|
| 7 |
+
semantics: 65.63
|
| 8 |
+
|
| 9 |
+
### UID ACCURACY
|
| 10 |
+
expletive_it_object_raising: 67.85
|
| 11 |
+
anaphor_number_agreement: 96.78
|
| 12 |
+
wh_questions_subject_gap_long_distance: 83.08
|
| 13 |
+
existential_there_subject_raising: 79.65
|
| 14 |
+
anaphor_gender_agreement: 93.00
|
| 15 |
+
determiner_noun_agreement_irregular_2: 87.07
|
| 16 |
+
passive_2: 76.30
|
| 17 |
+
irregular_plural_subject_verb_agreement_2: 79.82
|
| 18 |
+
existential_there_quantifiers_1: 95.48
|
| 19 |
+
wh_vs_that_with_gap_long_distance: 17.36
|
| 20 |
+
determiner_noun_agreement_2: 92.80
|
| 21 |
+
sentential_negation_npi_licensor_present: 99.78
|
| 22 |
+
determiner_noun_agreement_with_adj_irregular_1: 77.72
|
| 23 |
+
wh_island: 46.46
|
| 24 |
+
drop_argument: 71.09
|
| 25 |
+
superlative_quantifiers_1: 68.54
|
| 26 |
+
principle_A_reconstruction: 31.75
|
| 27 |
+
principle_A_domain_1: 96.72
|
| 28 |
+
npi_present_2: 47.37
|
| 29 |
+
determiner_noun_agreement_with_adjective_1: 86.39
|
| 30 |
+
tough_vs_raising_2: 80.54
|
| 31 |
+
principle_A_case_2: 82.95
|
| 32 |
+
adjunct_island: 64.66
|
| 33 |
+
regular_plural_subject_verb_agreement_2: 76.30
|
| 34 |
+
distractor_agreement_relational_noun: 56.22
|
| 35 |
+
distractor_agreement_relative_clause: 46.50
|
| 36 |
+
causative: 63.57
|
| 37 |
+
determiner_noun_agreement_irregular_1: 81.06
|
| 38 |
+
transitive: 74.42
|
| 39 |
+
determiner_noun_agreement_1: 89.99
|
| 40 |
+
tough_vs_raising_1: 42.72
|
| 41 |
+
animate_subject_passive: 68.38
|
| 42 |
+
superlative_quantifiers_2: 74.44
|
| 43 |
+
passive_1: 83.69
|
| 44 |
+
determiner_noun_agreement_with_adj_2: 87.89
|
| 45 |
+
sentential_subject_island: 52.13
|
| 46 |
+
wh_questions_object_gap: 79.74
|
| 47 |
+
npi_present_1: 43.23
|
| 48 |
+
wh_vs_that_no_gap: 94.43
|
| 49 |
+
matrix_question_npi_licensor_present: 23.47
|
| 50 |
+
only_npi_scope: 69.77
|
| 51 |
+
inchoative: 54.27
|
| 52 |
+
irregular_plural_subject_verb_agreement_1: 72.39
|
| 53 |
+
principle_A_domain_2: 64.81
|
| 54 |
+
wh_vs_that_no_gap_long_distance: 93.37
|
| 55 |
+
left_branch_island_simple_question: 67.30
|
| 56 |
+
principle_A_domain_3: 59.30
|
| 57 |
+
intransitive: 67.97
|
| 58 |
+
coordinate_structure_constraint_complex_left_branch: 59.38
|
| 59 |
+
determiner_noun_agreement_with_adj_irregular_2: 82.14
|
| 60 |
+
irregular_past_participle_adjectives: 73.15
|
| 61 |
+
wh_vs_that_with_gap: 33.30
|
| 62 |
+
sentential_negation_npi_scope: 31.11
|
| 63 |
+
ellipsis_n_bar_1: 67.83
|
| 64 |
+
regular_plural_subject_verb_agreement_1: 78.09
|
| 65 |
+
ellipsis_n_bar_2: 72.83
|
| 66 |
+
wh_questions_subject_gap: 91.76
|
| 67 |
+
irregular_past_participle_verbs: 84.29
|
| 68 |
+
coordinate_structure_constraint_object_extraction: 65.75
|
| 69 |
+
principle_A_c_command: 54.65
|
| 70 |
+
existential_there_object_raising: 64.29
|
| 71 |
+
existential_there_quantifiers_2: 43.03
|
| 72 |
+
animate_subject_trans: 85.81
|
| 73 |
+
complex_NP_island: 48.46
|
| 74 |
+
left_branch_island_echo_question: 54.17
|
| 75 |
+
only_npi_licensor_present: 95.24
|
| 76 |
+
principle_A_case_1: 100.00
|
| 77 |
+
|
| 78 |
+
### LINGUISTICS_TERM ACCURACY
|
| 79 |
+
control_raising: 66.90
|
| 80 |
+
anaphor_agreement: 94.85
|
| 81 |
+
filler_gap_dependency: 69.83
|
| 82 |
+
determiner_noun_agreement: 86.07
|
| 83 |
+
argument_structure: 70.26
|
| 84 |
+
subject_verb_agreement: 68.55
|
| 85 |
+
quantifiers: 70.55
|
| 86 |
+
npi_licensing: 58.39
|
| 87 |
+
island_effects: 57.36
|
| 88 |
+
binding: 69.59
|
| 89 |
+
s-selection: 77.23
|
| 90 |
+
irregular_forms: 78.67
|
| 91 |
+
ellipsis: 70.37
|
| 92 |
+
|
| 93 |
+
### AVERAGE ACCURACY
|
| 94 |
+
70.09
|
| 95 |
+
|
gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/blimp_supplement_report.txt
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
TEMPERATURE: 1.00
|
| 2 |
+
|
| 3 |
+
### FIELD ACCURACY
|
| 4 |
+
supplement: 77.81
|
| 5 |
+
|
| 6 |
+
### UID ACCURACY
|
| 7 |
+
qa_congruence_easy: 50.00
|
| 8 |
+
turn_taking: 69.29
|
| 9 |
+
qa_congruence_tricky: 41.21
|
| 10 |
+
subject_aux_inversion: 86.60
|
| 11 |
+
hypernym: 49.52
|
| 12 |
+
|
| 13 |
+
### LINGUISTICS_TERM ACCURACY
|
| 14 |
+
supplement: 77.81
|
| 15 |
+
|
| 16 |
+
### AVERAGE ACCURACY
|
| 17 |
+
59.33
|
| 18 |
+
|
gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/dev_curve.log
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
`torch_dtype` is deprecated! Use `dtype` instead!
|
| 2 |
+
/home/ruhai/Projects/SSM_IMC_NATURE/.venv/lib/python3.12/site-packages/fla/layers/gated_deltanet.py:190: UserWarning: ShortConvolution is crucial to the performance. Do not turn it off, i.e., setting `use_short_conv=False` unless you know what you are doing.
|
| 3 |
+
warnings.warn(
|
| 4 |
+
{"ckpt": "chck_10M", "words_m": 10, "nll": 3.826301932149419, "bpw": 9.414788995483974, "tokens": 17769472}
|
| 5 |
+
{"ckpt": "chck_20M", "words_m": 20, "nll": 3.6475560366300206, "bpw": 8.974976633583244, "tokens": 17769472}
|
| 6 |
+
{"ckpt": "chck_30M", "words_m": 30, "nll": 3.5876698592605902, "bpw": 8.82762398507895, "tokens": 17769472}
|
| 7 |
+
{"ckpt": "chck_40M", "words_m": 40, "nll": 3.4893632862524346, "bpw": 8.585736215071996, "tokens": 17769472}
|
| 8 |
+
{"ckpt": "chck_50M", "words_m": 50, "nll": 3.5046018000220056, "bpw": 8.623231267550693, "tokens": 17769472}
|
| 9 |
+
{"ckpt": "chck_60M", "words_m": 60, "nll": 3.459844292573803, "bpw": 8.513103395767216, "tokens": 17769472}
|
| 10 |
+
{"ckpt": "chck_70M", "words_m": 70, "nll": 3.4189122823832707, "bpw": 8.412388333041262, "tokens": 17769472}
|
| 11 |
+
{"ckpt": "chck_80M", "words_m": 80, "nll": 3.443516969172431, "bpw": 8.472929277934702, "tokens": 17769472}
|
| 12 |
+
{"ckpt": "chck_90M", "words_m": 90, "nll": 3.5014851779571727, "bpw": 8.615562677972719, "tokens": 17769472}
|
| 13 |
+
{"ckpt": "hf", "words_m": 100, "nll": 3.5793337967322554, "bpw": 8.807112726127322, "tokens": 17769472}
|
| 14 |
+
BEST {"ckpt": "chck_70M", "words_m": 70, "nll": 3.4189122823832707, "bpw": 8.412388333041262, "tokens": 17769472}
|
gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/entity_tracking_report.txt
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
TEMPERATURE: 1.00
|
| 2 |
+
|
| 3 |
+
### UID ACCURACY
|
| 4 |
+
move_contents_0_ops: 18.02
|
| 5 |
+
move_contents_1_ops: 17.33
|
| 6 |
+
move_contents_2_ops: 21.16
|
| 7 |
+
move_contents_3_ops: 18.81
|
| 8 |
+
move_contents_4_ops: 19.66
|
| 9 |
+
move_contents_5_ops: 20.51
|
| 10 |
+
ambiref_0_ops: 22.57
|
| 11 |
+
ambiref_1_ops: 16.31
|
| 12 |
+
ambiref_2_ops: 19.37
|
| 13 |
+
ambiref_3_ops: 20.36
|
| 14 |
+
ambiref_4_ops: 17.07
|
| 15 |
+
ambiref_5_ops: 17.65
|
| 16 |
+
regular_0_ops: 13.53
|
| 17 |
+
regular_1_ops: 15.82
|
| 18 |
+
regular_2_ops: 17.16
|
| 19 |
+
regular_3_ops: 23.27
|
| 20 |
+
regular_4_ops: 20.25
|
| 21 |
+
regular_5_ops: 20.75
|
| 22 |
+
regular: 18.46
|
| 23 |
+
ambiref: 18.89
|
| 24 |
+
move_contents: 19.25
|
| 25 |
+
|
| 26 |
+
### AVERAGE ACCURACY
|
| 27 |
+
18.87
|
| 28 |
+
|
gdn-30m-t1/babylm-strict-small-100Mw-s7/eval/final_summary.json
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"run": "gdn_30m_t1-lr1e-2-s7",
|
| 3 |
+
"params": 30453064,
|
| 4 |
+
"best_ckpt": "chck_70M",
|
| 5 |
+
"best_words_m": 70,
|
| 6 |
+
"dev_nll": 3.4189122823832707,
|
| 7 |
+
"dev_bpw": 8.412388333041262,
|
| 8 |
+
"test_nll": 3.3520635961237866,
|
| 9 |
+
"test_bpw": 8.378170875483095,
|
| 10 |
+
"blimp": 70.09,
|
| 11 |
+
"blimp_supplement": 59.33,
|
| 12 |
+
"entity_tracking": 18.87
|
| 13 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s7/generation_config.json
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_from_model_config": true,
|
| 3 |
+
"bos_token_id": 1,
|
| 4 |
+
"eos_token_id": 2,
|
| 5 |
+
"transformers_version": "4.57.3",
|
| 6 |
+
"use_cache": false
|
| 7 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s7/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4a02bc31ce1d50f4ae735bee541801f2aecd8f9dc8b9faecb165522d2e448c74
|
| 3 |
+
size 121817944
|
gdn-30m-t1/babylm-strict-small-100Mw-s7/special_tokens_map.json
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token": {
|
| 3 |
+
"content": "<s>",
|
| 4 |
+
"lstrip": false,
|
| 5 |
+
"normalized": false,
|
| 6 |
+
"rstrip": false,
|
| 7 |
+
"single_word": false
|
| 8 |
+
},
|
| 9 |
+
"cls_token": {
|
| 10 |
+
"content": "<s>",
|
| 11 |
+
"lstrip": false,
|
| 12 |
+
"normalized": false,
|
| 13 |
+
"rstrip": false,
|
| 14 |
+
"single_word": false
|
| 15 |
+
},
|
| 16 |
+
"eos_token": {
|
| 17 |
+
"content": "</s>",
|
| 18 |
+
"lstrip": false,
|
| 19 |
+
"normalized": false,
|
| 20 |
+
"rstrip": false,
|
| 21 |
+
"single_word": false
|
| 22 |
+
},
|
| 23 |
+
"mask_token": {
|
| 24 |
+
"content": "<mask>",
|
| 25 |
+
"lstrip": false,
|
| 26 |
+
"normalized": false,
|
| 27 |
+
"rstrip": false,
|
| 28 |
+
"single_word": false
|
| 29 |
+
},
|
| 30 |
+
"pad_token": {
|
| 31 |
+
"content": "<pad>",
|
| 32 |
+
"lstrip": false,
|
| 33 |
+
"normalized": false,
|
| 34 |
+
"rstrip": false,
|
| 35 |
+
"single_word": false
|
| 36 |
+
},
|
| 37 |
+
"sep_token": {
|
| 38 |
+
"content": "</s>",
|
| 39 |
+
"lstrip": false,
|
| 40 |
+
"normalized": false,
|
| 41 |
+
"rstrip": false,
|
| 42 |
+
"single_word": false
|
| 43 |
+
},
|
| 44 |
+
"unk_token": {
|
| 45 |
+
"content": "<unk>",
|
| 46 |
+
"lstrip": false,
|
| 47 |
+
"normalized": false,
|
| 48 |
+
"rstrip": false,
|
| 49 |
+
"single_word": false
|
| 50 |
+
}
|
| 51 |
+
}
|
gdn-30m-t1/babylm-strict-small-100Mw-s7/tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
gdn-30m-t1/babylm-strict-small-100Mw-s7/tokenizer_config.json
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"added_tokens_decoder": {
|
| 3 |
+
"0": {
|
| 4 |
+
"content": "<unk>",
|
| 5 |
+
"lstrip": false,
|
| 6 |
+
"normalized": false,
|
| 7 |
+
"rstrip": false,
|
| 8 |
+
"single_word": false,
|
| 9 |
+
"special": true
|
| 10 |
+
},
|
| 11 |
+
"1": {
|
| 12 |
+
"content": "<s>",
|
| 13 |
+
"lstrip": false,
|
| 14 |
+
"normalized": false,
|
| 15 |
+
"rstrip": false,
|
| 16 |
+
"single_word": false,
|
| 17 |
+
"special": true
|
| 18 |
+
},
|
| 19 |
+
"2": {
|
| 20 |
+
"content": "</s>",
|
| 21 |
+
"lstrip": false,
|
| 22 |
+
"normalized": false,
|
| 23 |
+
"rstrip": false,
|
| 24 |
+
"single_word": false,
|
| 25 |
+
"special": true
|
| 26 |
+
},
|
| 27 |
+
"3": {
|
| 28 |
+
"content": "<pad>",
|
| 29 |
+
"lstrip": false,
|
| 30 |
+
"normalized": false,
|
| 31 |
+
"rstrip": false,
|
| 32 |
+
"single_word": false,
|
| 33 |
+
"special": true
|
| 34 |
+
},
|
| 35 |
+
"4": {
|
| 36 |
+
"content": "<mask>",
|
| 37 |
+
"lstrip": false,
|
| 38 |
+
"normalized": false,
|
| 39 |
+
"rstrip": false,
|
| 40 |
+
"single_word": false,
|
| 41 |
+
"special": true
|
| 42 |
+
}
|
| 43 |
+
},
|
| 44 |
+
"bos_token": "<s>",
|
| 45 |
+
"clean_up_tokenization_spaces": false,
|
| 46 |
+
"cls_token": "<s>",
|
| 47 |
+
"eos_token": "</s>",
|
| 48 |
+
"extra_special_tokens": {},
|
| 49 |
+
"mask_token": "<mask>",
|
| 50 |
+
"model_max_length": 1000000000000000019884624838656,
|
| 51 |
+
"pad_token": "<pad>",
|
| 52 |
+
"sep_token": "</s>",
|
| 53 |
+
"tokenizer_class": "PreTrainedTokenizerFast",
|
| 54 |
+
"unk_token": "<unk>"
|
| 55 |
+
}
|
gdn-30m-t4/babylm-strict-small-100Mw-s1337/config.json
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"GDNBitForCausalLM"
|
| 4 |
+
],
|
| 5 |
+
"attn_mode": "chunk",
|
| 6 |
+
"bitlinear": false,
|
| 7 |
+
"bos_token_id": 1,
|
| 8 |
+
"conv_size": 4,
|
| 9 |
+
"dtype": "float32",
|
| 10 |
+
"eos_token_id": 2,
|
| 11 |
+
"expand_v": 2,
|
| 12 |
+
"fuse_cross_entropy": true,
|
| 13 |
+
"fuse_linear_cross_entropy": false,
|
| 14 |
+
"fuse_norm": true,
|
| 15 |
+
"fused_proj": true,
|
| 16 |
+
"head_dim": 64,
|
| 17 |
+
"hidden_act": "relu2",
|
| 18 |
+
"hidden_size": 512,
|
| 19 |
+
"init_scheme": "trunc_normal",
|
| 20 |
+
"initializer_range": 0.02,
|
| 21 |
+
"intermediate_size": 2048,
|
| 22 |
+
"loop_count": 4,
|
| 23 |
+
"max_position_embeddings": 1024,
|
| 24 |
+
"model_type": "gdnbit",
|
| 25 |
+
"norm_eps": 1e-05,
|
| 26 |
+
"num_heads": 6,
|
| 27 |
+
"num_hidden_layers": 6,
|
| 28 |
+
"transformers_version": "4.57.3",
|
| 29 |
+
"use_cache": false,
|
| 30 |
+
"use_short_conv": false,
|
| 31 |
+
"vocab_size": 16384
|
| 32 |
+
}
|
gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/args.json
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": "configs/babylm/gdn_30m_t4.json",
|
| 3 |
+
"data_dir": "data/babylm/tok16384-babylm-official",
|
| 4 |
+
"out_dir": "outputs/babylm30m_d512/gdn_30m_t4-lr1e-2-s1337",
|
| 5 |
+
"epochs": 10,
|
| 6 |
+
"batch_size": 32,
|
| 7 |
+
"micro_batch": 16,
|
| 8 |
+
"seq_len": 512,
|
| 9 |
+
"lr": 0.01,
|
| 10 |
+
"min_lr_ratio": 0.0,
|
| 11 |
+
"warmup": 500,
|
| 12 |
+
"weight_decay": 0.1,
|
| 13 |
+
"lr_drop_at": 1.0,
|
| 14 |
+
"lr_drop_ratio": 0.6666666666666666,
|
| 15 |
+
"wd_drop_at": 1.0,
|
| 16 |
+
"beta1": 0.9,
|
| 17 |
+
"beta2": 0.95,
|
| 18 |
+
"grad_clip": 1.0,
|
| 19 |
+
"seed": 1337,
|
| 20 |
+
"log_interval": 50,
|
| 21 |
+
"epoch_eval_windows": 2048,
|
| 22 |
+
"save_words": "10,20,30,40,50,60,70,80,90,100",
|
| 23 |
+
"dtype": "bf16",
|
| 24 |
+
"compile": false,
|
| 25 |
+
"tokenizer": null,
|
| 26 |
+
"override": "{}",
|
| 27 |
+
"steps": 10320,
|
| 28 |
+
"params": 30455112,
|
| 29 |
+
"steps_per_epoch": 1032,
|
| 30 |
+
"words_per_token": 0.5910163039530657
|
| 31 |
+
}
|
gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/babylm_eval_summary.json
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "outputs/babylm30m_d512/gdn_30m_t4-lr1e-2-s1337/ckpt/chck_80M",
|
| 3 |
+
"revision": "main",
|
| 4 |
+
"mode": "full",
|
| 5 |
+
"scores": {
|
| 6 |
+
"blimp": 72.08,
|
| 7 |
+
"blimp_supplement": 61.02,
|
| 8 |
+
"entity_tracking": 21.1
|
| 9 |
+
}
|
| 10 |
+
}
|
gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/blimp_report.txt
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
TEMPERATURE: 1.00
|
| 2 |
+
|
| 3 |
+
### FIELD ACCURACY
|
| 4 |
+
syntax/semantics: 70.73
|
| 5 |
+
morphology: 81.32
|
| 6 |
+
syntax: 67.43
|
| 7 |
+
semantics: 69.34
|
| 8 |
+
|
| 9 |
+
### UID ACCURACY
|
| 10 |
+
expletive_it_object_raising: 68.64
|
| 11 |
+
anaphor_number_agreement: 95.92
|
| 12 |
+
wh_questions_subject_gap_long_distance: 81.68
|
| 13 |
+
existential_there_subject_raising: 83.01
|
| 14 |
+
anaphor_gender_agreement: 93.20
|
| 15 |
+
determiner_noun_agreement_irregular_2: 86.46
|
| 16 |
+
passive_2: 77.52
|
| 17 |
+
irregular_plural_subject_verb_agreement_2: 81.50
|
| 18 |
+
existential_there_quantifiers_1: 97.74
|
| 19 |
+
wh_vs_that_with_gap_long_distance: 15.16
|
| 20 |
+
determiner_noun_agreement_2: 90.12
|
| 21 |
+
sentential_negation_npi_licensor_present: 99.89
|
| 22 |
+
determiner_noun_agreement_with_adj_irregular_1: 78.97
|
| 23 |
+
wh_island: 73.96
|
| 24 |
+
drop_argument: 71.20
|
| 25 |
+
superlative_quantifiers_1: 76.61
|
| 26 |
+
principle_A_reconstruction: 50.57
|
| 27 |
+
principle_A_domain_1: 97.48
|
| 28 |
+
npi_present_2: 54.16
|
| 29 |
+
determiner_noun_agreement_with_adjective_1: 87.35
|
| 30 |
+
tough_vs_raising_2: 86.85
|
| 31 |
+
principle_A_case_2: 83.93
|
| 32 |
+
adjunct_island: 61.96
|
| 33 |
+
regular_plural_subject_verb_agreement_2: 75.34
|
| 34 |
+
distractor_agreement_relational_noun: 66.62
|
| 35 |
+
distractor_agreement_relative_clause: 48.56
|
| 36 |
+
causative: 64.06
|
| 37 |
+
determiner_noun_agreement_irregular_1: 79.30
|
| 38 |
+
transitive: 71.77
|
| 39 |
+
determiner_noun_agreement_1: 90.64
|
| 40 |
+
tough_vs_raising_1: 50.63
|
| 41 |
+
animate_subject_passive: 71.28
|
| 42 |
+
superlative_quantifiers_2: 76.17
|
| 43 |
+
passive_1: 77.26
|
| 44 |
+
determiner_noun_agreement_with_adj_2: 85.23
|
| 45 |
+
sentential_subject_island: 43.91
|
| 46 |
+
wh_questions_object_gap: 78.70
|
| 47 |
+
npi_present_1: 50.06
|
| 48 |
+
wh_vs_that_no_gap: 95.24
|
| 49 |
+
matrix_question_npi_licensor_present: 22.39
|
| 50 |
+
only_npi_scope: 69.65
|
| 51 |
+
inchoative: 53.45
|
| 52 |
+
irregular_plural_subject_verb_agreement_1: 76.99
|
| 53 |
+
principle_A_domain_2: 68.31
|
| 54 |
+
wh_vs_that_no_gap_long_distance: 95.43
|
| 55 |
+
left_branch_island_simple_question: 72.34
|
| 56 |
+
principle_A_domain_3: 59.09
|
| 57 |
+
intransitive: 64.63
|
| 58 |
+
coordinate_structure_constraint_complex_left_branch: 55.74
|
| 59 |
+
determiner_noun_agreement_with_adj_irregular_2: 82.86
|
| 60 |
+
irregular_past_participle_adjectives: 75.65
|
| 61 |
+
wh_vs_that_with_gap: 37.65
|
| 62 |
+
sentential_negation_npi_scope: 37.31
|
| 63 |
+
ellipsis_n_bar_1: 65.84
|
| 64 |
+
regular_plural_subject_verb_agreement_1: 78.88
|
| 65 |
+
ellipsis_n_bar_2: 74.03
|
| 66 |
+
wh_questions_subject_gap: 90.09
|
| 67 |
+
irregular_past_participle_verbs: 84.50
|
| 68 |
+
coordinate_structure_constraint_object_extraction: 70.18
|
| 69 |
+
principle_A_c_command: 63.00
|
| 70 |
+
existential_there_object_raising: 72.41
|
| 71 |
+
existential_there_quantifiers_2: 52.58
|
| 72 |
+
animate_subject_trans: 83.53
|
| 73 |
+
complex_NP_island: 44.80
|
| 74 |
+
left_branch_island_echo_question: 65.47
|
| 75 |
+
only_npi_licensor_present: 94.22
|
| 76 |
+
principle_A_case_1: 99.89
|
| 77 |
+
|
| 78 |
+
### LINGUISTICS_TERM ACCURACY
|
| 79 |
+
control_raising: 72.31
|
| 80 |
+
anaphor_agreement: 94.53
|
| 81 |
+
filler_gap_dependency: 69.98
|
| 82 |
+
determiner_noun_agreement: 85.53
|
| 83 |
+
argument_structure: 68.66
|
| 84 |
+
subject_verb_agreement: 71.45
|
| 85 |
+
quantifiers: 75.91
|
| 86 |
+
npi_licensing: 60.93
|
| 87 |
+
island_effects: 61.29
|
| 88 |
+
binding: 74.29
|
| 89 |
+
s-selection: 77.50
|
| 90 |
+
irregular_forms: 80.03
|
| 91 |
+
ellipsis: 70.00
|
| 92 |
+
|
| 93 |
+
### AVERAGE ACCURACY
|
| 94 |
+
72.08
|
| 95 |
+
|
gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/blimp_supplement_report.txt
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
TEMPERATURE: 1.00
|
| 2 |
+
|
| 3 |
+
### FIELD ACCURACY
|
| 4 |
+
supplement: 72.61
|
| 5 |
+
|
| 6 |
+
### UID ACCURACY
|
| 7 |
+
qa_congruence_easy: 59.38
|
| 8 |
+
turn_taking: 73.57
|
| 9 |
+
qa_congruence_tricky: 43.03
|
| 10 |
+
subject_aux_inversion: 78.90
|
| 11 |
+
hypernym: 50.24
|
| 12 |
+
|
| 13 |
+
### LINGUISTICS_TERM ACCURACY
|
| 14 |
+
supplement: 72.61
|
| 15 |
+
|
| 16 |
+
### AVERAGE ACCURACY
|
| 17 |
+
61.02
|
| 18 |
+
|
gdn-30m-t4/babylm-strict-small-100Mw-s1337/eval/dev_curve.log
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
`torch_dtype` is deprecated! Use `dtype` instead!
|
| 2 |
+
/ncg/projects/ruhai/SSM_IMC_NATURE/.venv/lib/python3.12/site-packages/fla/layers/gated_deltanet.py:190: UserWarning: ShortConvolution is crucial to the performance. Do not turn it off, i.e., setting `use_short_conv=False` unless you know what you are doing.
|
| 3 |
+
warnings.warn(
|
| 4 |
+
{"ckpt": "chck_10M", "words_m": 10, "nll": 3.81434517047429, "bpw": 9.385368842491136, "tokens": 17769472}
|
| 5 |
+
{"ckpt": "chck_20M", "words_m": 20, "nll": 3.6564029574236225, "bpw": 8.996744882406393, "tokens": 17769472}
|
| 6 |
+
{"ckpt": "chck_30M", "words_m": 30, "nll": 3.537110857198831, "bpw": 8.703221273355071, "tokens": 17769472}
|
| 7 |
+
{"ckpt": "chck_40M", "words_m": 40, "nll": 3.457275940417996, "bpw": 8.506783849102627, "tokens": 17769472}
|
| 8 |
+
{"ckpt": "chck_50M", "words_m": 50, "nll": 3.4188837153502294, "bpw": 8.41231804256417, "tokens": 17769472}
|
| 9 |
+
{"ckpt": "chck_60M", "words_m": 60, "nll": 3.3783650740913265, "bpw": 8.312620092794171, "tokens": 17769472}
|
| 10 |
+
{"ckpt": "chck_70M", "words_m": 70, "nll": 3.38059025704397, "bpw": 8.318095256110338, "tokens": 17769472}
|
| 11 |
+
{"ckpt": "chck_80M", "words_m": 80, "nll": 3.366958815790573, "bpw": 8.28455447825725, "tokens": 17769472}
|
| 12 |
+
{"ckpt": "chck_90M", "words_m": 90, "nll": 3.442769450632453, "bpw": 8.471089974751392, "tokens": 17769472}
|
| 13 |
+
{"ckpt": "hf", "words_m": 100, "nll": 3.516394635012931, "bpw": 8.652248071521456, "tokens": 17769472}
|
| 14 |
+
BEST {"ckpt": "chck_80M", "words_m": 80, "nll": 3.366958815790573, "bpw": 8.28455447825725, "tokens": 17769472}
|