File size: 2,329 Bytes
442d639 b72ab33 442d639 b72ab33 442d639 b72ab33 442d639 b72ab33 442d639 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | ---
license: mit
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- gpt2
datasets:
- craa/exceptions
---
# exceptions
GPT-2 language models trained from scratch for the exceptions project
(Leong and Linzen, [Language Models Can Learn Exceptions to Syntactic Rules](https://doi.org/10.7275/h25z-0y75), and follow-up experiments).
Code: [github.com/craaaa/exceptions](https://github.com/craaaa/exceptions).
All models are GPT-2 small (12 layers, 12 heads, 768 hidden). Each folder holds the final model of one training run
(best checkpoint by validation loss), with its tokenizer and Trainer logs (`trainer_state.json`, `all_results.json`).
## Contents
| Folder | Runs | Naming | Seeds | Vocab |
|---|---|---|---|---|
| `exp1b/` | 5 | `100M_<seed>` | 1001, 2128, 3591, 5039, 40817 | 50,257 (GPT-2) |
| `exp2a/` | 60 | `<cond>_to_<verb>_frequency_<seed>` — cond ∈ {cost, last, resemble}, verb ∈ {carry, drop, hit, push} | 1001, 2128, 3591, 5039, 40817 | 50,257 (GPT-2) |
| `exp2b/` | 180 | `swap_<src>_to_<verb>_<seed>` — src ∈ {last, require, take}; and `swap_<p>_<cond>_to_<verb>_<seed>` — p ∈ {0.3, 0.7}, cond ∈ {cost, last, resemble}; verb ∈ {carry, drop, hit, push} | 1032, 2128, 3591, 5039, 40817 | 50,257 (GPT-2) |
| `exp3/` | 170 | `{high,low}_<n>_<seed>` — n ∈ {0, 1, 2, 4, 8, 16, 32, 64, 10, 100, 128, 256, 500, 512, 1000, 1500, 2000} | 495, 634, 1208, 6910, 8397 | 52,000 (custom) |
| `base/` | 10 | `100M__<seed>`, `100M_default_<seed>` — baseline runs from 2024 | 495, 634, 1208, 6910, 8397 | 52,000 (custom) |
Training corpora are in the dataset repo [`craa/exceptions`](https://huggingface.co/datasets/craa/exceptions);
the `exp2a` folder names correspond to its `frequency_<cond>_to_<verb>` configs, and `exp2b` `swap_<src>_to_<verb>` to its `swap_<src>_to_<verb>` folders.
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
repo, run = "craa/exceptions", "exp3/high_0_1208"
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=run)
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=run)
```
`base/100M_default_*` has no tokenizer files; it uses the same 52,000-token vocab as `exp3/`,
so load the tokenizer from e.g. `base/100M__<seed>` or any `exp3/` run.
Intermediate checkpoints are not published here.
|