add the heads files and the card

README.md

@@ -0,0 +1,126 @@
+---
+license: apache-2.0
+language:
+  - en
+base_model: GSAI-ML/LLaDA-8B-Base
+tags:
+  - diffusion-language-model
+  - probing
+  - fact-updating
+  - wikipedia
+library_name: pytorch
+datasets:
+  - gamemaker1/wiki-wrong
+---
+
+# DiffusEdit classifiers
+
+DiffusEdit updates an article from evidence by editing it in place. A frozen
+masked diffusion language model,
+[LLaDA-8B-Base](https://huggingface.co/GSAI-ML/LLaDA-8B-Base), reads the
+evidence and the article once, with no token masked, and the small
+classifiers in this repository read its hidden states. One classifier labels
+each sentence as kept, edited or dropped. A second one scores each token of
+an edited sentence by how likely it is to be out of date. A third one scores
+pairs of sentences and says whether one sentence has to change because
+another one did. The diffusion model then fills in the tokens that were
+scored as stale, and a reranker chooses among the fills. The model itself is
+never trained. The classifiers hold 12.9M parameters between them.
+
+Every classifier was fit on the train split of
+[gamemaker1/wiki-wrong](https://huggingface.co/datasets/gamemaker1/wiki-wrong),
+tuned on its validation split and scored once on its test split. The code
+that trains them and runs the full system is at
+[github.com/gamemaker1/diffusedit](https://github.com/gamemaker1/diffusedit)
+under `code/pipeline`. The fine-tuned editor that the system is compared
+against is at [gamemaker1/t5edit](https://huggingface.co/gamemaker1/t5edit).
+
+## Layout
+
+```
+sentence/    the sentence classifier, KEEP / EDIT / DROP
+staleness/   the token classifier, is this token out of date
+cascade/     the pair classifier, must sentence k change because sentence j did
+reranker/    the gradient-boosted tree that ranks the candidate fills
+```
+
+Each directory holds a `classifier.pt` file and the `metrics.json` that was
+written when it was trained, with its validation and test scores. A token
+classifier directory also holds `sweep.json`, the grid of mask thresholds it
+was tuned over, and the attention classifier holds `eval.json`, its scores
+under the evidence controls described below.
+
+## Input
+
+LLaDA-8B-Base at revision `0f2787f2d87eac5eed8a087d5ecd24277e6255b2` runs in
+bfloat16 over the sequence `[BOS] evidence \n\n article`, where the
+evidence passages are written one per line as `title: text` and the article
+follows after a blank line. The forward pass is stopped after the deepest
+block that a classifier needs, and the hidden state of every token is kept at
+blocks 24 and 32 of the 32 blocks. Each state has 4,096 values. The sentence
+classifiers read the mean of the token states of a sentence. The token
+classifiers read the state of each token of an edited sentence, and the
+attention classifier also reads the states of every evidence token.
+
+## The classifiers the system uses
+
+| Directory | What it reads | Architecture | Parameters |
+|---|---|---|---|
+| `sentence/linear-block24` | the mean token state of the sentence at block 24 | a linear layer with three outputs | 12.3K |
+| `staleness/attention-block32` | the token state at block 32 and the states of all evidence tokens | four attention heads over the evidence tokens plus one null slot, then a two-layer MLP | 4.5M |
+| `cascade/consequence` | the mean states at block 32 of the repaired sentence j and the candidate sentence k | an MLP with one hidden layer of 512 units on the concatenation of the two states, their product and their absolute difference | 8.4M |
+| `reranker/rerank.pkl` | the logged scores and text features of every candidate fill | a scikit-learn `HistGradientBoostingClassifier` with 200 iterations and 15 leaves per tree | |
+
+Block 24 was chosen for the sentence classifier because its validation
+macro-F1 is higher than that of block 32. The thresholds are stored inside
+the checkpoints: the drop gate `tau_d` of the
+sentence classifier, the mask threshold `tau` of the token classifiers and
+the cascade threshold `tau_c` of the pair classifier. Each was chosen on the
+validation split.
+
+The evidence controls test whether a classifier reads the evidence or only
+the sentence. The sentence classifier is trained and scored a second time on
+hidden states of the article alone. The attention token classifier is scored
+a second time with its evidence slots emptied, and a third time with each
+article given the evidence of another article. The scores of the controls
+are in the metrics files, and all of them fall well below the scores with
+the right evidence.
+
+## The other classifiers
+
+| Directory | What it is |
+|---|---|
+| `sentence/linear-block32` | the sentence classifier on block 32, the final block |
+| `sentence/evidence-mean-block24`, `sentence/evidence-nearest-block24`, `sentence/evidence-attention-block24`, and the same at block 32 | the sentence classifier with a summary of the evidence concatenated to the sentence state. The summary is the mean of the evidence passage states, the state of the nearest passage, or a learned attention over the passages. |
+| `sentence/minilm-baseline` | the same linear classifier over embeddings of the sentence and the evidence from [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2), with no diffusion model |
+| `staleness/linear-block32` | a linear classifier on the token state alone, the form of the staleness head of Detect, Remask, Repair |
+
+The reranker uses two scores that the system logs for every candidate fill.
+One is the mean log-probability that LLaDA-8B-Base gives each token of the
+candidate sentence when that token alone is masked. The other is the
+probability that the fact verifier
+[tals/albert-base-vitaminc-mnli](https://huggingface.co/tals/albert-base-vitaminc-mnli)
+gives to the best evidence passage supporting the candidate sentence.
+
+## Checkpoint format
+
+`classifier.pt` is a dictionary saved with `torch.save`. A linear classifier
+holds `state_dict`, the per-feature `mean` and `std` that normalise the
+hidden states, the `layer` it reads and its tuned threshold. The attention
+classifier also holds `ev_mean` and `ev_std` for the evidence states, a
+`config` dictionary with `d`, `dim`, `heads`, `hidden` and `dropout`, and
+`tau_match`, the threshold at which the share of masked tokens equals the
+share of stale tokens. The pair classifier holds `state_dict`, `d`,
+`hidden`, `layer`, `seed` and the paths it was trained from. The reranker is
+a pickle of `{"model", "features", "run"}` written with scikit-learn 1.9.1,
+and it loads only with a compatible scikit-learn.
+
+In the code, `classifier.train` defines the sentence classifier,
+`staleness.train` and `staleness.attention` the token classifiers,
+`cascading.train` the pair classifier and `infilling.rerank` the reranker.
+`staleness.attention.Scorer` and `infilling.run` show how the classifiers
+are applied to the hidden states of a new article.
+
+## Licence
+
+Apache 2.0. LLaDA-8B-Base keeps its own licence.

cascade/consequence/classifier.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5bbb9ad72bc699f05329db84ce516283603aafc8d8d8115f46d6c64bda786316
+size 33595057

cascade/consequence/config.json

@@ -0,0 +1,113 @@
+{
+  "d": 4096,
+  "hidden": 512,
+  "features": "/home/diffusedit/features/full/train",
+  "features_after": "/home/diffusedit/features/full/after_train",
+  "pairs": "/home/diffusedit/data/split/links/train/pairs.jsonl",
+  "layer": 32,
+  "seed": 42,
+  "best_val_auroc": 0.9451081101225768,
+  "tau_c": 0.35,
+  "val_f1": 0.7651331719128329,
+  "history": [
+    {
+      "epoch": 1,
+      "train_loss": 1.605,
+      "val_loss": 0.892,
+      "val_auroc": 0.8385,
+      "seconds": 0.6
+    },
+    {
+      "epoch": 2,
+      "train_loss": 0.719,
+      "val_loss": 0.7097,
+      "val_auroc": 0.8862,
+      "seconds": 0.3
+    },
+    {
+      "epoch": 3,
+      "train_loss": 0.5492,
+      "val_loss": 0.6318,
+      "val_auroc": 0.9139,
+      "seconds": 0.2
+    },
+    {
+      "epoch": 4,
+      "train_loss": 0.407,
+      "val_loss": 0.5501,
+      "val_auroc": 0.9295,
+      "seconds": 0.2
+    },
+    {
+      "epoch": 5,
+      "train_loss": 0.3096,
+      "val_loss": 0.5429,
+      "val_auroc": 0.934,
+      "seconds": 0.2
+    },
+    {
+      "epoch": 6,
+      "train_loss": 0.2175,
+      "val_loss": 0.5164,
+      "val_auroc": 0.94,
+      "seconds": 0.2
+    },
+    {
+      "epoch": 7,
+      "train_loss": 0.1577,
+      "val_loss": 0.6775,
+      "val_auroc": 0.9395,
+      "seconds": 0.2
+    },
+    {
+      "epoch": 8,
+      "train_loss": 0.1133,
+      "val_loss": 0.6663,
+      "val_auroc": 0.9369,
+      "seconds": 0.2
+    },
+    {
+      "epoch": 9,
+      "train_loss": 0.0863,
+      "val_loss": 0.7139,
+      "val_auroc": 0.9408,
+      "seconds": 0.2
+    },
+    {
+      "epoch": 10,
+      "train_loss": 0.0585,
+      "val_loss": 0.7103,
+      "val_auroc": 0.9451,
+      "seconds": 0.2
+    },
+    {
+      "epoch": 11,
+      "train_loss": 0.0422,
+      "val_loss": 0.8489,
+      "val_auroc": 0.9381,
+      "seconds": 0.2
+    },
+    {
+      "epoch": 12,
+      "train_loss": 0.0293,
+      "val_loss": 0.7948,
+      "val_auroc": 0.9427,
+      "seconds": 0.2
+    },
+    {
+      "epoch": 13,
+      "train_loss": 0.0217,
+      "val_loss": 0.8185,
+      "val_auroc": 0.9409,
+      "seconds": 0.2
+    }
+  ],
+  "test": {
+    "auroc": 0.9314635445435178,
+    "precision": 0.6846153846153846,
+    "recall": 0.7672413793103449,
+    "f1": 0.7235772357723578,
+    "pairs": 1106,
+    "positives": 232
+  }
+}

reranker/rerank.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be614f079b4a9e067000e88f73ee9a9ff8953e9814e8e48869654abc56e5b990
+size 372663

sentence/evidence-attention-block24/classifier.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b8100db1a546ebdfbc110e0636ed4c436faecd7d7ce8705e253288f146020f32
+size 8558037

sentence/evidence-attention-block24/metrics.json

@@ -0,0 +1,66 @@
+{
+  "layer": 24,
+  "evidence_pool": "attention",
+  "tau_d": 0.8,
+  "tau_sweep": {
+    "0.35": 0.828298657069029,
+    "0.4": 0.8294412415816063,
+    "0.45": 0.8312405814788989,
+    "0.5": 0.8326087682287691,
+    "0.55": 0.836397125789707,
+    "0.6": 0.8416302551840918,
+    "0.65": 0.8416197747592445,
+    "0.7": 0.8462487851056104,
+    "0.75": 0.8456807273729566,
+    "0.8": 0.8493074459752545,
+    "0.85": 0.8463061617219833,
+    "0.9": 0.8432587703841362,
+    "0.95": 0.8388332795419909
+  },
+  "holdout": null,
+  "seed": 42,
+  "val": {
+    "KEEP": {
+      "precision": 0.9388853704876504,
+      "recall": 0.8539746543778802,
+      "f1": 0.8944193061840121,
+      "support": 3472
+    },
+    "EDIT": {
+      "precision": 0.7266454352441614,
+      "recall": 0.8692063492063492,
+      "f1": 0.791558253830587,
+      "support": 1575
+    },
+    "DROP": {
+      "precision": 0.8568019093078759,
+      "recall": 0.8671497584541062,
+      "f1": 0.8619447779111645,
+      "support": 414
+    },
+    "macro_f1": 0.8493074459752545
+  },
+  "mean_peak_attention": 0.9990412592887878,
+  "max_snippets": 7,
+  "test": {
+    "KEEP": {
+      "precision": 0.9293185419968304,
+      "recall": 0.8377142857142857,
+      "f1": 0.8811419984973704,
+      "support": 3500
+    },
+    "EDIT": {
+      "precision": 0.7236255572065379,
+      "recall": 0.8675771971496437,
+      "f1": 0.7890899270861462,
+      "support": 1684
+    },
+    "DROP": {
+      "precision": 0.8470319634703196,
+      "recall": 0.866822429906542,
+      "f1": 0.8568129330254041,
+      "support": 428
+    },
+    "macro_f1": 0.8423482862029736
+  }
+}

sentence/evidence-attention-block32/classifier.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:805ba313395f77ccd26202f337229a16f4eedc615cf6a4fb03484ec5aaaaefef
+size 8558037

sentence/evidence-attention-block32/metrics.json

@@ -0,0 +1,66 @@
+{
+  "layer": 32,
+  "evidence_pool": "attention",
+  "tau_d": 0.8,
+  "tau_sweep": {
+    "0.35": 0.7298850064314643,
+    "0.4": 0.7316456397653734,
+    "0.45": 0.7333049533335614,
+    "0.5": 0.7360107328668105,
+    "0.55": 0.7397977717563697,
+    "0.6": 0.7425608698297165,
+    "0.65": 0.7455891802529716,
+    "0.7": 0.7478012758634541,
+    "0.75": 0.7473962006624723,
+    "0.8": 0.749485474804764,
+    "0.85": 0.7455679450274159,
+    "0.9": 0.7377530660117978,
+    "0.95": 0.7323929227506797
+  },
+  "holdout": null,
+  "seed": 42,
+  "val": {
+    "KEEP": {
+      "precision": 0.8920065252854812,
+      "recall": 0.7874423963133641,
+      "f1": 0.8364693284381214,
+      "support": 3472
+    },
+    "EDIT": {
+      "precision": 0.6355283307810107,
+      "recall": 0.7904761904761904,
+      "f1": 0.7045840407470287,
+      "support": 1575
+    },
+    "DROP": {
+      "precision": 0.6887871853546911,
+      "recall": 0.7270531400966184,
+      "f1": 0.7074030552291423,
+      "support": 414
+    },
+    "macro_f1": 0.749485474804764
+  },
+  "mean_peak_attention": 0.9977710247039795,
+  "max_snippets": 7,
+  "test": {
+    "KEEP": {
+      "precision": 0.8834537097833224,
+      "recall": 0.7688571428571429,
+      "f1": 0.8221814848762603,
+      "support": 3500
+    },
+    "EDIT": {
+      "precision": 0.6229660622966062,
+      "recall": 0.7957244655581948,
+      "f1": 0.6988265971316818,
+      "support": 1684
+    },
+    "DROP": {
+      "precision": 0.7108433734939759,
+      "recall": 0.6892523364485982,
+      "f1": 0.6998813760379597,
+      "support": 428
+    },
+    "macro_f1": 0.7402964860153006
+  }
+}

sentence/evidence-mean-block24/classifier.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad06af92c634246f599a7ce09aa73d144a72f840ab9a338617defe8f1f3d204b
+size 166079

sentence/evidence-mean-block24/metrics.json

@@ -0,0 +1,63 @@
+{
+  "layer": 24,
+  "evidence_pool": "mean",
+  "tau_d": 0.85,
+  "tau_sweep": {
+    "0.35": 0.8183428722238784,
+    "0.4": 0.8192790589797915,
+    "0.45": 0.8233162695389972,
+    "0.5": 0.8272089047865526,
+    "0.55": 0.83061106158326,
+    "0.6": 0.8351031336768479,
+    "0.65": 0.8393451170004237,
+    "0.7": 0.8435623918247458,
+    "0.75": 0.8457840996717598,
+    "0.8": 0.8464422156581833,
+    "0.85": 0.8472304045067051,
+    "0.9": 0.842217764567989,
+    "0.95": 0.8339713535831903
+  },
+  "val": {
+    "KEEP": {
+      "precision": 0.9373182552504039,
+      "recall": 0.8355414746543779,
+      "f1": 0.8835084513476472,
+      "support": 3472
+    },
+    "EDIT": {
+      "precision": 0.6987829614604463,
+      "recall": 0.8749206349206349,
+      "f1": 0.7769946433605865,
+      "support": 1575
+    },
+    "DROP": {
+      "precision": 0.9035532994923858,
+      "recall": 0.8599033816425121,
+      "f1": 0.8811881188118812,
+      "support": 414
+    },
+    "macro_f1": 0.8472304045067051
+  },
+  "feature_dim": 8192,
+  "test": {
+    "KEEP": {
+      "precision": 0.9342616580310881,
+      "recall": 0.8242857142857143,
+      "f1": 0.8758348512446873,
+      "support": 3500
+    },
+    "EDIT": {
+      "precision": 0.6978723404255319,
+      "recall": 0.8764845605700713,
+      "f1": 0.7770465912082126,
+      "support": 1684
+    },
+    "DROP": {
+      "precision": 0.8484107579462102,
+      "recall": 0.8107476635514018,
+      "f1": 0.8291517323775388,
+      "support": 428
+    },
+    "macro_f1": 0.8273443916101462
+  }
+}

sentence/evidence-mean-block32/classifier.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b978263a1441a5ae58ffc7c51fd4e53066ea3782c41d601404fbcdddaeff6737
+size 166079

sentence/evidence-mean-block32/metrics.json

@@ -0,0 +1,63 @@
+{
+  "layer": 32,
+  "evidence_pool": "mean",
+  "tau_d": 0.75,
+  "tau_sweep": {
+    "0.35": 0.7084534052188601,
+    "0.4": 0.7104026883780964,
+    "0.45": 0.7136435665115465,
+    "0.5": 0.7155487530965181,
+    "0.55": 0.7243127080925796,
+    "0.6": 0.7235142895554829,
+    "0.65": 0.7277080313099994,
+    "0.7": 0.72918538589756,
+    "0.75": 0.7316663725111044,
+    "0.8": 0.7265052771748625,
+    "0.85": 0.7244618173153458,
+    "0.9": 0.7169837089888382,
+    "0.95": 0.6986098266480546
+  },
+  "val": {
+    "KEEP": {
+      "precision": 0.8952024498128616,
+      "recall": 0.7577764976958525,
+      "f1": 0.8207767898923725,
+      "support": 3472
+    },
+    "EDIT": {
+      "precision": 0.6041765543426673,
+      "recall": 0.8082539682539682,
+      "f1": 0.6914720260727865,
+      "support": 1575
+    },
+    "DROP": {
+      "precision": 0.6819277108433734,
+      "recall": 0.6835748792270532,
+      "f1": 0.6827503015681545,
+      "support": 414
+    },
+    "macro_f1": 0.7316663725111044
+  },
+  "feature_dim": 8192,
+  "test": {
+    "KEEP": {
+      "precision": 0.8813617606602476,
+      "recall": 0.7322857142857143,
+      "f1": 0.7999375780274658,
+      "support": 3500
+    },
+    "EDIT": {
+      "precision": 0.5924764890282131,
+      "recall": 0.7856294536817102,
+      "f1": 0.6755169772785294,
+      "support": 1684
+    },
+    "DROP": {
+      "precision": 0.6624203821656051,
+      "recall": 0.7289719626168224,
+      "f1": 0.6941045606229144,
+      "support": 428
+    },
+    "macro_f1": 0.7231863719763032
+  }
+}

sentence/evidence-nearest-block24/classifier.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a70e09f3646483509070120755050de206179e557d14054ce6178cd801a40e02
+size 166079

sentence/evidence-nearest-block24/metrics.json

@@ -0,0 +1,63 @@
+{
+  "layer": 24,
+  "evidence_pool": "nearest",
+  "tau_d": 0.75,
+  "tau_sweep": {
+    "0.35": 0.8280369785912546,
+    "0.4": 0.8284244030828681,
+    "0.45": 0.8310011226606445,
+    "0.5": 0.8335599398803938,
+    "0.55": 0.8357240173302495,
+    "0.6": 0.8396687102605732,
+    "0.65": 0.8438812681792848,
+    "0.7": 0.8474658102054408,
+    "0.75": 0.8488943338645747,
+    "0.8": 0.8487771457868224,
+    "0.85": 0.8464260839578167,
+    "0.9": 0.8476382526866436,
+    "0.95": 0.8428036194318898
+  },
+  "val": {
+    "KEEP": {
+      "precision": 0.9331651954602774,
+      "recall": 0.8525345622119815,
+      "f1": 0.8910295003010236,
+      "support": 3472
+    },
+    "EDIT": {
+      "precision": 0.7304582210242587,
+      "recall": 0.8603174603174604,
+      "f1": 0.7900874635568513,
+      "support": 1575
+    },
+    "DROP": {
+      "precision": 0.8456221198156681,
+      "recall": 0.8864734299516909,
+      "f1": 0.8655660377358492,
+      "support": 414
+    },
+    "macro_f1": 0.8488943338645747
+  },
+  "feature_dim": 8192,
+  "test": {
+    "KEEP": {
+      "precision": 0.9288637079005351,
+      "recall": 0.8431428571428572,
+      "f1": 0.8839299086416055,
+      "support": 3500
+    },
+    "EDIT": {
+      "precision": 0.7296351824087957,
+      "recall": 0.8669833729216152,
+      "f1": 0.7924016282225238,
+      "support": 1684
+    },
+    "DROP": {
+      "precision": 0.8410138248847926,
+      "recall": 0.852803738317757,
+      "f1": 0.8468677494199536,
+      "support": 428
+    },
+    "macro_f1": 0.8410664287613608
+  }
+}

sentence/evidence-nearest-block32/classifier.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47ce84c33635ceb17c4905d1090b838ecfa9758e0bfc224433e7f7540a91d8dc
+size 166079

sentence/evidence-nearest-block32/metrics.json

@@ -0,0 +1,63 @@
+{
+  "layer": 32,
+  "evidence_pool": "nearest",
+  "tau_d": 0.85,
+  "tau_sweep": {
+    "0.35": 0.7504911437342106,
+    "0.4": 0.7521112436691751,
+    "0.45": 0.7546426336402083,
+    "0.5": 0.7585509381369135,
+    "0.55": 0.7605076347829994,
+    "0.6": 0.762778214997697,
+    "0.65": 0.7671188466583323,
+    "0.7": 0.7707989589361667,
+    "0.75": 0.7717645853563595,
+    "0.8": 0.7701726729000757,
+    "0.85": 0.7718798425204504,
+    "0.9": 0.7678025465952784,
+    "0.95": 0.7491515700558957
+  },
+  "val": {
+    "KEEP": {
+      "precision": 0.9144562334217506,
+      "recall": 0.7943548387096774,
+      "f1": 0.8501849568434031,
+      "support": 3472
+    },
+    "EDIT": {
+      "precision": 0.6462058965683906,
+      "recall": 0.8488888888888889,
+      "f1": 0.7338090010976948,
+      "support": 1575
+    },
+    "DROP": {
+      "precision": 0.7686170212765957,
+      "recall": 0.6980676328502415,
+      "f1": 0.7316455696202532,
+      "support": 414
+    },
+    "macro_f1": 0.7718798425204504
+  },
+  "feature_dim": 8192,
+  "test": {
+    "KEEP": {
+      "precision": 0.8999339498018494,
+      "recall": 0.7785714285714286,
+      "f1": 0.8348651960784313,
+      "support": 3500
+    },
+    "EDIT": {
+      "precision": 0.6335912608101957,
+      "recall": 0.8266033254156769,
+      "f1": 0.7173408915228033,
+      "support": 1684
+    },
+    "DROP": {
+      "precision": 0.7648578811369509,
+      "recall": 0.6915887850467289,
+      "f1": 0.7263803680981594,
+      "support": 428
+    },
+    "macro_f1": 0.7595288185664647
+  }
+}

sentence/linear-block24/classifier.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8083524b0bda780b7ca8b100a8bce39280726401d1f13b9460f0f3ccb455b40
+size 84159

sentence/linear-block24/metrics.json

@@ -0,0 +1,65 @@
+{
+  "layer": 24,
+  "tau_d": 0.8,
+  "tau_sweep": {
+    "0.35": 0.8101831919255443,
+    "0.4": 0.8114140897972656,
+    "0.45": 0.8149619103598357,
+    "0.5": 0.8209714092773067,
+    "0.55": 0.8293601923061806,
+    "0.6": 0.8331783406394381,
+    "0.65": 0.8379237670005164,
+    "0.7": 0.841049436098379,
+    "0.75": 0.843821574117252,
+    "0.8": 0.8443022189436068,
+    "0.85": 0.8434063414926697,
+    "0.9": 0.8355219394066573,
+    "0.95": 0.8217890877385421
+  },
+  "holdout": null,
+  "seed": 42,
+  "loss": "ce",
+  "gamma": null,
+  "val": {
+    "KEEP": {
+      "precision": 0.9393641429039659,
+      "recall": 0.8254608294930875,
+      "f1": 0.8787367775563389,
+      "support": 3472
+    },
+    "EDIT": {
+      "precision": 0.6885572139303483,
+      "recall": 0.8787301587301587,
+      "f1": 0.7721059972105997,
+      "support": 1575
+    },
+    "DROP": {
+      "precision": 0.8975,
+      "recall": 0.8671497584541062,
+      "f1": 0.8820638820638821,
+      "support": 414
+    },
+    "macro_f1": 0.8443022189436068
+  },
+  "test": {
+    "KEEP": {
+      "precision": 0.9224529526281635,
+      "recall": 0.8122857142857143,
+      "f1": 0.8638711637800061,
+      "support": 3500
+    },
+    "EDIT": {
+      "precision": 0.6853080568720379,
+      "recall": 0.8586698337292161,
+      "f1": 0.7622561939905114,
+      "support": 1684
+    },
+    "DROP": {
+      "precision": 0.8476190476190476,
+      "recall": 0.8317757009345794,
+      "f1": 0.839622641509434,
+      "support": 428
+    },
+    "macro_f1": 0.8219166664266505
+  }
+}

sentence/linear-block32/classifier.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f6082517c5a6be05f717587d205b22161ee20ddd996144d81662e07d26edbd7b
+size 84159

sentence/linear-block32/metrics.json

@@ -0,0 +1,65 @@
+{
+  "layer": 32,
+  "tau_d": 0.7,
+  "tau_sweep": {
+    "0.35": 0.6943394237676858,
+    "0.4": 0.6966648625484937,
+    "0.45": 0.699174131563538,
+    "0.5": 0.70253575048726,
+    "0.55": 0.7055049462478515,
+    "0.6": 0.7117083919713286,
+    "0.65": 0.7137749625403568,
+    "0.7": 0.716626951233481,
+    "0.75": 0.711820978231276,
+    "0.8": 0.7065285979854902,
+    "0.85": 0.7048639721514492,
+    "0.9": 0.6857018714241615,
+    "0.95": 0.6615082530380287
+  },
+  "holdout": null,
+  "seed": 42,
+  "loss": "ce",
+  "gamma": null,
+  "val": {
+    "KEEP": {
+      "precision": 0.8841855739925499,
+      "recall": 0.7520161290322581,
+      "f1": 0.8127626459143971,
+      "support": 3472
+    },
+    "EDIT": {
+      "precision": 0.5933852140077821,
+      "recall": 0.7746031746031746,
+      "f1": 0.6719911870008263,
+      "support": 1575
+    },
+    "DROP": {
+      "precision": 0.6371681415929203,
+      "recall": 0.6956521739130435,
+      "f1": 0.6651270207852193,
+      "support": 414
+    },
+    "macro_f1": 0.716626951233481
+  },
+  "test": {
+    "KEEP": {
+      "precision": 0.8632799181725196,
+      "recall": 0.7234285714285714,
+      "f1": 0.7871910461681952,
+      "support": 3500
+    },
+    "EDIT": {
+      "precision": 0.5713638423198912,
+      "recall": 0.748812351543943,
+      "f1": 0.6481624261115394,
+      "support": 1684
+    },
+    "DROP": {
+      "precision": 0.6228813559322034,
+      "recall": 0.6869158878504673,
+      "f1": 0.6533333333333334,
+      "support": 428
+    },
+    "macro_f1": 0.6962289352043559
+  }
+}

sentence/minilm-baseline/classifier.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50a2253a83d5dd5cc7da1b768b4b117ca595f40b0a7b393a0424f2e536de0d4c
+size 17663

sentence/minilm-baseline/metrics.json

@@ -0,0 +1,64 @@
+{
+  "model": "sentence-transformers/all-MiniLM-L6-v2",
+  "tau_d": 0.6,
+  "tau_sweep": {
+    "0.35": 0.4757366738562993,
+    "0.4": 0.4800737523372757,
+    "0.45": 0.4801793559903799,
+    "0.5": 0.4831173376340067,
+    "0.55": 0.4833452243361714,
+    "0.6": 0.4868561852895335,
+    "0.65": 0.48247999370014627,
+    "0.7": 0.47345108266974206,
+    "0.75": 0.46729570781066565,
+    "0.8": 0.4614491537820024,
+    "0.85": 0.4436141475493189,
+    "0.9": 0.4266200564034583,
+    "0.95": 0.4007739222377022
+  },
+  "holdout": null,
+  "seed": 42,
+  "val": {
+    "KEEP": {
+      "precision": 0.7619453924914675,
+      "recall": 0.5144009216589862,
+      "f1": 0.6141678129298486,
+      "support": 3472
+    },
+    "EDIT": {
+      "precision": 0.379015860105734,
+      "recall": 0.5917460317460318,
+      "f1": 0.4620723847297967,
+      "support": 1575
+    },
+    "DROP": {
+      "precision": 0.3130699088145897,
+      "recall": 0.4975845410628019,
+      "f1": 0.3843283582089553,
+      "support": 414
+    },
+    "macro_f1": 0.4868561852895335
+  },
+  "feature_dim": 768,
+  "test": {
+    "KEEP": {
+      "precision": 0.7552301255230126,
+      "recall": 0.5157142857142857,
+      "f1": 0.6129032258064516,
+      "support": 3500
+    },
+    "EDIT": {
+      "precision": 0.3939047619047619,
+      "recall": 0.6140142517814727,
+      "f1": 0.479925736829891,
+      "support": 1684
+    },
+    "DROP": {
+      "precision": 0.32663316582914576,
+      "recall": 0.45560747663551404,
+      "f1": 0.3804878048780488,
+      "support": 428
+    },
+    "macro_f1": 0.4911055891714638
+  }
+}

staleness/attention-block32/classifier.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b2cd724828d97e1f6e0e40581c6beecc57eb4728bb852059534b92c97f5dfb4
+size 17904933

staleness/attention-block32/eval.json

@@ -0,0 +1,32 @@
+{
+  "head": "/home/diffusedit/full/heads/staleness_attention/head.pt",
+  "features": "/home/diffusedit/features/full/test_tokens",
+  "layer": 32,
+  "tau": 0.85,
+  "tau_match": 0.85,
+  "seed": 42,
+  "test": {
+    "precision": 0.7517290299698528,
+    "recall": 0.7466971992249427,
+    "f1": 0.7492046659597031,
+    "support": 5677,
+    "pr_auc": 0.8310112953186035,
+    "auroc": 0.9660482450433805,
+    "masked": 0.11237321048974991
+  },
+  "controls": {
+    "set": "test",
+    "reference": {
+      "pr_auc": 0.8310112953186035,
+      "auroc": 0.9660482450433805
+    },
+    "shuffled_evidence": {
+      "pr_auc": 0.6253933906555176,
+      "auroc": 0.8934536220671628
+    },
+    "no_evidence": {
+      "pr_auc": 0.7176730036735535,
+      "auroc": 0.9343921869725395
+    }
+  }
+}

staleness/attention-block32/metrics.json

@@ -0,0 +1,47 @@
+{
+  "layer": 32,
+  "tau": 0.85,
+  "tau_match": 0.85,
+  "val_pr_auc": 0.8545193076133728,
+  "val_auroc": 0.9714314069844169,
+  "val": {
+    "precision": 0.7714828897338403,
+    "recall": 0.7566660451239977,
+    "f1": 0.7640026357902664,
+    "support": 5363
+  },
+  "holdout": null,
+  "seed": 42,
+  "config": {
+    "d": 4096,
+    "dim": 256,
+    "heads": 4,
+    "hidden": 256,
+    "dropout": 0.1
+  },
+  "epochs_run": 7,
+  "test": {
+    "precision": 0.7517290299698528,
+    "recall": 0.7466971992249427,
+    "f1": 0.7492046659597031,
+    "support": 5677,
+    "pr_auc": 0.8310112953186035,
+    "auroc": 0.9660482450433805,
+    "masked": 0.11237321048974991
+  },
+  "controls": {
+    "set": "test",
+    "reference": {
+      "pr_auc": 0.8310112953186035,
+      "auroc": 0.9660482450433805
+    },
+    "shuffled_evidence": {
+      "pr_auc": 0.6253933906555176,
+      "auroc": 0.8934536220671628
+    },
+    "no_evidence": {
+      "pr_auc": 0.7176730036735535,
+      "auroc": 0.9343921869725395
+    }
+  }
+}

staleness/attention-block32/sweep.json

@@ -0,0 +1,139 @@
+{
+  "tau_match": 0.85,
+  "stale_share": 0.11393427103757858,
+  "grid": {
+    "0.05": {
+      "precision": 0.44622625065365173,
+      "recall": 0.9546895394368824,
+      "f1": 0.608184355882877,
+      "support": 5363,
+      "masked": 0.243759423494339
+    },
+    "0.1": {
+      "precision": 0.48410029781919495,
+      "recall": 0.9395860525825098,
+      "f1": 0.6389804717220391,
+      "support": 5363,
+      "masked": 0.2211340367794037
+    },
+    "0.15": {
+      "precision": 0.5102354145342887,
+      "recall": 0.9295170613462614,
+      "f1": 0.6588250842529572,
+      "support": 5363,
+      "masked": 0.2075587958097458
+    },
+    "0.2": {
+      "precision": 0.5313947226709747,
+      "recall": 0.9200074585120268,
+      "f1": 0.6736755871108684,
+      "support": 5363,
+      "masked": 0.1972552090883255
+    },
+    "0.25": {
+      "precision": 0.5494838420107719,
+      "recall": 0.91310833488719,
+      "f1": 0.686094570928196,
+      "support": 5363,
+      "masked": 0.18933100998401642
+    },
+    "0.3": {
+      "precision": 0.5673526660430309,
+      "recall": 0.904717508856983,
+      "f1": 0.6973769313690262,
+      "support": 5363,
+      "masked": 0.1816829890012741
+    },
+    "0.35": {
+      "precision": 0.5836769342711617,
+      "recall": 0.8974454596308037,
+      "f1": 0.7073260342420458,
+      "support": 5363,
+      "masked": 0.17518217861652374
+    },
+    "0.4": {
+      "precision": 0.597892888498683,
+      "recall": 0.8888681707999254,
+      "f1": 0.7149070185962807,
+      "support": 5363,
+      "masked": 0.16938242316246033
+    },
+    "0.45": {
+      "precision": 0.6141127457362323,
+      "recall": 0.8795450307663621,
+      "f1": 0.7232444035571911,
+      "support": 5363,
+      "masked": 0.16317902505397797
+    },
+    "0.5": {
+      "precision": 0.629136400322841,
+      "recall": 0.8720865187395115,
+      "f1": 0.7309525670078925,
+      "support": 5363,
+      "masked": 0.1579316407442093
+    },
+    "0.55": {
+      "precision": 0.6457693381736945,
+      "recall": 0.8623904531046056,
+      "f1": 0.7385229540918162,
+      "support": 5363,
+      "masked": 0.1521531343460083
+    },
+    "0.6": {
+      "precision": 0.660828025477707,
+      "recall": 0.8512026850643296,
+      "f1": 0.7440306413495232,
+      "support": 5363,
+      "masked": 0.14675702154636383
+    },
+    "0.65": {
+      "precision": 0.677331723513432,
+      "recall": 0.8368450494126422,
+      "f1": 0.7486862957711236,
+      "support": 5363,
+      "masked": 0.14076608419418335
+    },
+    "0.7": {
+      "precision": 0.6956248032735285,
+      "recall": 0.824165578966996,
+      "f1": 0.7544593325936673,
+      "support": 5363,
+      "masked": 0.13498757779598236
+    },
+    "0.75": {
+      "precision": 0.7156280909990109,
+      "recall": 0.8094350177139661,
+      "f1": 0.7596465132557528,
+      "support": 5363,
+      "masked": 0.1288691610097885
+    },
+    "0.8": {
+      "precision": 0.7403373155305693,
+      "recall": 0.7857542420287152,
+      "f1": 0.7623699683401176,
+      "support": 5363,
+      "masked": 0.120923712849617
+    },
+    "0.85": {
+      "precision": 0.7714828897338403,
+      "recall": 0.7566660451239977,
+      "f1": 0.7640026357902664,
+      "support": 5363,
+      "masked": 0.11174608767032623
+    },
+    "0.9": {
+      "precision": 0.8064719478882119,
+      "recall": 0.7156442289763192,
+      "f1": 0.7583481525390239,
+      "support": 5363,
+      "masked": 0.10110259056091309
+    },
+    "0.95": {
+      "precision": 0.8637159289822456,
+      "recall": 0.644042513518553,
+      "f1": 0.7378765221106601,
+      "support": 5363,
+      "masked": 0.08495676517486572
+    }
+  }
+}

staleness/linear-block32/classifier.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0a8573a27c52986f4ce985e556130d7181ea6d8664d10d41a6b71194fc2f84e6
+size 51391

staleness/linear-block32/metrics.json

@@ -0,0 +1,45 @@
+{
+  "layer": 32,
+  "tau": 0.85,
+  "tau_sweep": {
+    "0.05": 0.33772333470851446,
+    "0.1": 0.39081996434937605,
+    "0.15": 0.43068918528306566,
+    "0.2": 0.460555133419747,
+    "0.25": 0.4871954430137379,
+    "0.3": 0.5105906313645621,
+    "0.35": 0.5315518351577592,
+    "0.4": 0.5517046737412509,
+    "0.45": 0.5696653216235462,
+    "0.5": 0.5886458528073784,
+    "0.55": 0.605126860289779,
+    "0.6": 0.6228106468073369,
+    "0.65": 0.6386652158142908,
+    "0.7": 0.6530486868376594,
+    "0.75": 0.6631433449873974,
+    "0.8": 0.6711994421199443,
+    "0.85": 0.673179896665101,
+    "0.9": 0.6635792940207882,
+    "0.95": 0.6176160954497601
+  },
+  "val_pr_auc": 0.753352701663971,
+  "val_auroc": 0.9445498534594089,
+  "val": {
+    "precision": 0.6783415372964786,
+    "recall": 0.6680962148051464,
+    "f1": 0.673179896665101,
+    "support": 5363
+  },
+  "holdout": null,
+  "seed": 42,
+  "loss": "bce",
+  "gamma": null,
+  "test": {
+    "precision": 0.6726296958855098,
+    "recall": 0.6623216487581469,
+    "f1": 0.6674358746782639,
+    "support": 5677,
+    "pr_auc": 0.7349614500999451,
+    "auroc": 0.9397934704786409
+  }
+}

staleness/linear-block32/sweep.json

@@ -0,0 +1,160 @@
+{
+  "stale_fraction": 0.11393427103757858,
+  "tau_match": 0.85,
+  "tau_f05": 0.95,
+  "trained_tau": 0.85,
+  "grid": [
+    {
+      "tau": 0.05,
+      "precision": 0.20349548722655653,
+      "recall": 0.9921685623718068,
+      "f1": 0.33772333470851446,
+      "f05": 0.24196262107225688,
+      "masked_fraction": 0.5555012822151184
+    },
+    {
+      "tau": 0.1,
+      "precision": 0.24400649200092742,
+      "recall": 0.9811672571322021,
+      "f1": 0.39081996434937605,
+      "f05": 0.28715497200484585,
+      "masked_fraction": 0.45813772082328796
+    },
+    {
+      "tau": 0.15,
+      "precision": 0.27659913015805665,
+      "recall": 0.9724035055006526,
+      "f1": 0.43068918528306566,
+      "f05": 0.322794290595328,
+      "masked_fraction": 0.40054386854171753
+    },
+    {
+      "tau": 0.2,
+      "precision": 0.302880658436214,
+      "recall": 0.9606563490583628,
+      "f1": 0.460555133419747,
+      "f05": 0.3509393349045679,
+      "masked_fraction": 0.36136898398399353
+    },
+    {
+      "tau": 0.25,
+      "precision": 0.32773055126223594,
+      "recall": 0.9489091926160731,
+      "f1": 0.4871954430137379,
+      "f05": 0.3771026306039274,
+      "masked_fraction": 0.3298846483230591
+    },
+    {
+      "tau": 0.3,
+      "precision": 0.35119422847937243,
+      "recall": 0.9349244825657281,
+      "f1": 0.5105906313645621,
+      "f05": 0.4013062060796209,
+      "masked_fraction": 0.30330777168273926
+    },
+    {
+      "tau": 0.35,
+      "precision": 0.3731635651322233,
+      "recall": 0.9235502517247809,
+      "f1": 0.5315518351577592,
+      "f05": 0.42365922504490633,
+      "masked_fraction": 0.2819782793521881
+    },
+    {
+      "tau": 0.4,
+      "precision": 0.3956124018457055,
+      "recall": 0.9112437068804774,
+      "f1": 0.5517046737412509,
+      "f05": 0.44609767229575537,
+      "masked_fraction": 0.26243335008621216
+    },
+    {
+      "tau": 0.45,
+      "precision": 0.41779093045521803,
+      "recall": 0.8950214432220772,
+      "f1": 0.5696653216235462,
+      "f05": 0.4676630487733588,
+      "masked_fraction": 0.24407809972763062
+    },
+    {
+      "tau": 0.5,
+      "precision": 0.4420628977910895,
+      "recall": 0.8806638075703898,
+      "f1": 0.5886458528073784,
+      "f05": 0.4909665481610845,
+      "masked_fraction": 0.22697627544403076
+    },
+    {
+      "tau": 0.55,
+      "precision": 0.4666329625884732,
+      "recall": 0.860525825097893,
+      "f1": 0.605126860289779,
+      "f05": 0.5136567014669545,
+      "masked_fraction": 0.21010813117027283
+    },
+    {
+      "tau": 0.6,
+      "precision": 0.49414596783017833,
+      "recall": 0.8420660078314376,
+      "f1": 0.6228106468073369,
+      "f05": 0.5386578878312936,
+      "masked_fraction": 0.19415351748466492
+    },
+    {
+      "tau": 0.65,
+      "precision": 0.5226786986464023,
+      "recall": 0.8208092485549133,
+      "f1": 0.6386652158142908,
+      "f05": 0.563621930296279,
+      "masked_fraction": 0.17892120778560638
+    },
+    {
+      "tau": 0.7,
+      "precision": 0.5551638595116856,
+      "recall": 0.7928398284542234,
+      "f1": 0.6530486868376594,
+      "f05": 0.5905719603322315,
+      "masked_fraction": 0.16271165013313293
+    },
+    {
+      "tau": 0.75,
+      "precision": 0.5879469434832757,
+      "recall": 0.760395301137423,
+      "f1": 0.6631433449873974,
+      "f05": 0.6158818376778326,
+      "masked_fraction": 0.14735187590122223
+    },
+    {
+      "tau": 0.8,
+      "precision": 0.6302177115730889,
+      "recall": 0.7178817825843744,
+      "f1": 0.6711994421199443,
+      "f05": 0.6459948320413437,
+      "masked_fraction": 0.12978266179561615
+    },
+    {
+      "tau": 0.85,
+      "precision": 0.6783415372964786,
+      "recall": 0.6680962148051464,
+      "f1": 0.673179896665101,
+      "f05": 0.676267411573742,
+      "masked_fraction": 0.11221346259117126
+    },
+    {
+      "tau": 0.9,
+      "precision": 0.7404685346807534,
+      "recall": 0.6011560693641619,
+      "f1": 0.6635792940207882,
+      "f05": 0.7076693445717547,
+      "masked_fraction": 0.09249856323003769
+    },
+    {
+      "tau": 0.95,
+      "precision": 0.8286252354048964,
+      "recall": 0.49226179377214246,
+      "f1": 0.6176160954497601,
+      "f05": 0.7289998343182195,
+      "masked_fraction": 0.06768498569726944
+    }
+  ]
+}