gec-bart-base-s1s2

facebook/bart-base fine-tuned for English grammatical error correction (GEC) by the CS4248 GEC project. This is run bart-base.s1s2 in the project repository (experiments/training/bart-base.s1s2/).

Training

Stage 1: BEA-2019 train (FCE train+dev, Lang-8, NUCLE, W&I A/B/C), errorful pairs only, 561,338 pairs. Stage 2: W&I A/B/C train including correct sentences, 33,943 pairs (22,681 errorful + 11,262 correct). Starts from the best stage-1 checkpoint.

Run	Stage	Learning rate	Batch	Epochs	Best step
bart-base.s1	Stage 1	5e-05	128	3	13,155
bart-base.s1s2	Stage 2	1e-05	32	5	5,305

Settings common to all stages:

• AdamW with linear decay
• weight decay 0.01, label smoothing 0.1, dropout 0.1, seed 42
• bf16, max length 128 tokens

The checkpoint is the one with the best ERRANT F0.5 on dev_tune. Training ran on an NVIDIA H100 NVL MIG 3g.47gb with torch 2.11.0+cu128 and transformers 4.46.3. Git commit 8371a55.

Results

ERRANT P / R / F0.5 (%), beam 5:

Split	Score
dev_test: 2,384 W&I+LOCNESS dev sentences, never used for choosing checkpoints	57.7 / 37.9 / 52.3
full W&I+LOCNESS dev (BEA-19 dev, 4,384 sentences)	58.7 / 38.4 / 53.1

The dev_tune/dev_test split, the predictions and the scores are in the project repository.

Usage

The project repository's scripts take this repository name as --model. For example:

python -m training.evaluate --model CS4248-GEC/gec-bart-base-s1s2 --split dev_test --out out/

Using transformers directly:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

repo = "CS4248-GEC/gec-bart-base-s1s2"          # private: run `hf auth login` first
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSeq2SeqLM.from_pretrained(repo)

source = "She go to school yesterday ."   # tokenized like the corpora (It 's, do n't)
inputs = tokenizer([source], return_tensors="pt")
out = model.generate(**inputs, num_beams=5, max_new_tokens=160, no_repeat_ngram_size=0)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Notes:

• Inputs are tokenized: the model reads and writes text tokenized like the BEA-2019 corpora ("It 's", "do n't", spaces before punctuation).
• No prefix: BART takes the source sentence as it is.
• Always pass no_repeat_ngram_size=0: bart-base inherits 3 from its summarization defaults, which stops it copying repeated phrases. The saved generation_config.json already sets 0.

Verification

model.safetensors sha256: 6d81a325ad27046c2d7594b09ec3fb5e6c8624b55e4ef30bc0c122b123c01bf9

To reproduce the scores, run python -m training.evaluate --model CS4248-GEC/gec-bart-base-s1s2 --split dev_test --out check/ and compare the result with experiments/training/bart-base.s1s2/dev_test.scores.json. Other hardware may decode a few sentences differently.

Data and licence

Trained on the BEA-2019 shared-task data (FCE, Lang-8, NUCLE, W&I+LOCNESS), whose licences restrict redistribution and commercial use. This model is private to the CS4248-GEC organization, for course use only.