title the card DiffusEdit T5edit and call the model the upper bound

README.md

@@ -13,54 +13,74 @@ datasets:
   - gamemaker1/wiki-wrong
 ---
 
-# DiffusEdit editor baseline
+# DiffusEdit T5edit
 
-`google/flan-t5-large` (780M parameters) fine-tuned end to end to rewrite an
-outdated article from evidence, in the style of EDIT5 (Iv et al., 2022).
-It is the fine-tuned upper bound that DiffusEdit is compared against, and
-every parameter was trained, unlike the DiffusEdit classifiers. The code is at
-https://github.com/gamemaker1/diffusedit under
-`code/pipeline/experiments/editor.py`.
+t5edit is [google/flan-t5-large](https://huggingface.co/google/flan-t5-large),
+a sequence-to-sequence model with 780M parameters, fine-tuned to rewrite an
+outdated article from evidence. It follows the design of EDIT5 from the
+FRUIT paper by Iv et al. (2022), in which the model writes the updated
+article but copies an unchanged sentence through a short marker instead of
+spelling it out again.
+
+The model was trained as the upper bound for DiffusEdit, a system that keeps a
+frozen diffusion language model and trains only small classifiers over its
+hidden states. The classifiers are at
+[gamemaker1/diffusedit](https://huggingface.co/gamemaker1/diffusedit) and
+the data at [gamemaker1/wiki-wrong](https://huggingface.co/datasets/gamemaker1/wiki-wrong).
+Every parameter of t5edit was trained, so it bounds what a fully trained
+generator reaches on the same data. The training script is
+`code/pipeline/experiments/editor.py` in
+[github.com/gamemaker1/diffusedit](https://github.com/gamemaker1/diffusedit).
 
 ## Input and output
 
+The input names each sentence of the outdated article with an index and
+appends the evidence passages.
+
 ```
-input    update: [0] <sentence 0> [1] <sentence 1> ... evidence: (0) <title>: <text> (1) ...
-output   [0] #1 <rewritten sentence 1> [2] ...
+update: [0] <sentence 0> [1] <sentence 1> ... evidence: (0) <title>: <text> (1) <title>: <text> ...
 ```
 
-The output names each source sentence in order: `[j]` copies sentence j,
-`#j text` replaces it, and a missing j deletes it. The input is cut at 1,024
-tokens from the end and the output at 384 tokens.
+The output names each source sentence in order. `[j]` means that sentence j
+is copied unchanged, `#j <text>` means that sentence j is replaced by the
+text, and a sentence whose index is absent is deleted.
 
-## Training
+```
+[0] #1 <the rewritten sentence 1> [2]
+```
 
-Fit on the 12,180 train articles of `gamemaker1/wiki-wrong`, with
-the validation loss after each epoch for early stopping (patience 1). Learning
-rate 1e-4 with linear decay and 5% warm-up, 32 examples per optimiser step
-(micro-batches of 4, 8 accumulated), gradient clipping at 1.0, bf16 autocast,
-seed 42. The best epoch was the third of four, with validation loss 0.0439,
-after 64 minutes on one A100 40GB.
+The input is cut to 1,024 tokens from the end, so the evidence is shortened
+before the article, and the output is cut at 384 tokens. The T5 vocabulary
+cannot write some letters, such as the ø in a Norwegian name, so after
+decoding, a word of the output that equals a word of the input with its
+missing letters removed is replaced by that input word.
 
-## Test scores
+## Training
 
-On the 1,670 supported EDIT sentences of the test split, against the gold
-rewrite: ROUGE-L 0.973, exact match 76.7%, new-value recall 88.7%, old-value
-kept 10.0%. Copying the source scores ROUGE-L 0.875. Greedy decoding writes
-an article in 0.35 seconds.
+The model was fit on the train split of wiki-wrong. The validation loss was
+computed after each epoch, the checkpoint with the lowest validation loss
+was kept, and training stopped after one epoch without improvement. The
+learning rate was 1e-4 with a linear decay and a warm-up over the first 5%
+of the steps. Each optimiser step used 32 examples, as eight accumulated
+micro-batches of four, with gradient clipping at 1.0, bfloat16 autocast and
+seed 42, on one A100 with 40 GB of memory. The validation loss per epoch
+and the time taken are in `run_metrics.json`.
 
 ## Use
 
 ```python
 from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+
 tok = AutoTokenizer.from_pretrained("gamemaker1/t5edit")
 model = AutoModelForSeq2SeqLM.from_pretrained("gamemaker1/t5edit")
 ```
 
-`experiments.editor.input_text` builds the input from a dataset record and
-`experiments.editor.parse` maps the output back to one sentence per source
-sentence.
+In the training script, `input_text` builds the input string from a
+wiki-wrong record and `parse` maps the generated text back to one output
+sentence per source sentence. `run_config.json` and `run_metrics.json` in
+this repository are the configuration and the metrics that the training run
+wrote.
 
 ## Licence
 
-Apache 2.0, as the base model.
+Apache 2.0, the licence of the base model.