--- license: cc-by-sa-4.0 language: - en library_name: transformers pipeline_tag: text-classification base_model: answerdotai/ModernBERT-base datasets: - tals/vitaminc - nyu-mll/multi_nli - alisawuffles/WANLI tags: - natural-language-inference - nli - fact-verification - fact-checking - cross-encoder - modernbert - rag - evidence metrics: - accuracy model-index: - name: ease-delta-reader-base results: - task: type: text-classification name: Fact verification dataset: name: VitaminC type: tals/vitaminc split: test metrics: - type: accuracy value: 90.20 name: Accuracy - task: type: text-classification name: Natural language inference dataset: name: MultiNLI (mismatched) type: nyu-mll/multi_nli split: validation_mismatched metrics: - type: accuracy value: 88.56 name: Accuracy - task: type: text-classification name: Natural language inference dataset: name: WANLI type: alisawuffles/WANLI split: test metrics: - type: accuracy value: 74.74 name: Accuracy - task: type: text-classification name: Natural language inference dataset: name: SNLI type: stanfordnlp/snli split: test metrics: - type: accuracy value: 80.33 name: Accuracy --- # EASE-Delta Reader (base) **Does this passage support the claim, refute it, or not settle it at all?** A 149M-parameter cross-encoder (`answerdotai/ModernBERT-base` fine-tuned) that reads one claim against one passage and answers `SUPPORTS`, `REFUTES` or `NOT_ENOUGH_INFO`. It was trained to say `NOT_ENOUGH_INFO` when a passage is about something else, which is where general NLI models most often guess: on passages taken from unrelated documents it gives a decisive answer 0.82% of the time; a widely used public NLI model does so 45.07% of the time. It is the reader inside [EASE-Delta](https://github.com/jithinsaireddy/ease-delta), a system that keeps a task's decisions current as its messages and documents change. Try both in the [demo](https://huggingface.co/spaces/jithinpothireddy21/ease-delta-demo). ## Use it Claim first, evidence second. ```python from transformers import pipeline reader = pipeline("text-classification", model="jithinpothireddy21/ease-delta-reader-base", top_k=None) claim = "The client has approved the final design." passages = [ "Email from the client: we approve the final design, please go ahead.", "Email from the client: we cannot approve the design yet; the colours are wrong.", "The office is closed on the first Monday of the month for staff training.", ] for passage in passages: print(reader({"text": claim, "text_pair": passage}, truncation=True)[0]) # {'label': 'SUPPORTS', 'score': 0.818} # {'label': 'REFUTES', 'score': 0.950} # {'label': 'NOT_ENOUGH_INFO', 'score': 0.999} ``` With sentence-transformers, continuing from above: ```python from sentence_transformers import CrossEncoder model = CrossEncoder("jithinpothireddy21/ease-delta-reader-base") probs = model.predict([(claim, passage) for passage in passages], apply_softmax=True) # one row per pair; columns SUPPORTS, REFUTES, NOT_ENOUGH_INFO ``` Inputs were at most 256 tokens in training (`truncation=True` cuts there). Split long documents into passages and read each one; say who is speaking in the passage itself ("Email from the client: ..."), because the reader sees only the text. Probabilities are best calibrated after dividing the logits by **1.227** (fitted on development data) before the softmax. ## Results Test sets, accuracy. One evaluation harness for all three models; brackets and paired tests are in the [results report](https://github.com/jithinsaireddy/ease-delta/blob/main/docs/RESULTS.md). | | **This model** (base) | large reader | `tasksource/ModernBERT-base-nli` | |---|---:|---:|---:| | Passages from unrelated documents read as decisive (lower is better) | 0.82% | 0.58% | 45.07% | | VitaminC: claims against Wikipedia revisions | 90.20% | 91.53% | 68.30% | | MultiNLI, mismatched genres | 88.56% | 90.21% | 89.84% | | WANLI | 74.74% | 77.42% | 66.34% | | SNLI (not trained on) | 80.33% | 84.39% | 89.15% | | ANLI round 1 (not trained on) | 46.40% | 53.70% | 63.40% | | ANLI round 2 (not trained on) | 33.30% | 38.10% | 48.20% | | ANLI round 3 (not trained on) | 34.75% | 36.00% | 42.25% | `tasksource/ModernBERT-base-nli` is a base-size multi-task NLI model whose training includes SNLI and ANLI; it is the better choice for adversarial NLI of the ANLI kind. This reader is built for evidence: contrastive Wikipedia revisions (VitaminC), and knowing when a passage does not bear on the claim. On VitaminC, real revisions: 88.65%; synthetic: 92.78%. ## ONNX, and in the browser `onnx/model.onnx` (fp32) matches PyTorch to 4e-5 in the logits; `onnx/model_quantized.onnx` (int8, 150 MB) gave the same label as the full model on 98.1% of 3,000 development pairs (accuracy 88.5% against 89.2%). ```js import { AutoTokenizer, AutoModelForSequenceClassification } from "https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.1"; const tokenizer = await AutoTokenizer.from_pretrained("jithinpothireddy21/ease-delta-reader-base"); const model = await AutoModelForSequenceClassification.from_pretrained("jithinpothireddy21/ease-delta-reader-base", { dtype: "q8" }); const inputs = await tokenizer(claim, { text_pair: passage, truncation: true, max_length: 256 }); const { logits } = await model(inputs); // SUPPORTS, REFUTES, NOT_ENOUGH_INFO ``` This is what the [demo](https://huggingface.co/spaces/jithinpothireddy21/ease-delta-demo) runs; the text never leaves the browser. ## Labels | Label | Meaning | NLI equivalent | |---|---|---| | `SUPPORTS` | the passage establishes the claim | entailment | | `REFUTES` | the passage establishes that the claim is false | contradiction | | `NOT_ENOUGH_INFO` | the passage does not settle the claim, including when it is about something else | neutral | ## Training 900,000 examples streamed from the Hugging Face Hub: VitaminC (cc-by-sa-3.0), MultiNLI and WANLI (cc-by-4.0), plus 20% synthetic unrelated pairs (the claim of one example with the evidence of another from a different topic, labelled `NOT_ENOUGH_INFO`). SNLI was left out because its annotators labelled unrelated content as contradiction; ANLI because its licence is non-commercial. One run, on one Apple M4 Max. The evaluation plan was written and hashed while this model was still training, before any test split was evaluated: [`docs/PREREGISTRATION.md`](https://github.com/jithinsaireddy/ease-delta/blob/main/docs/PREREGISTRATION.md). ## Limitations - English only. - Passages on the claim's own subject that do not settle it are still read as decisive about one time in five (VitaminC test, not-enough-info items). - Conflicts that follow only from a consequence ("broke a leg" against "cycles to work") are mostly missed. - Numbers and dates are read as text; compare them with code when they matter. - Not a judge of truth: it reports what the passage says, not whether the passage is right. ## How this checkpoint was made Converted exactly from the EASE-Delta edge model (`ease/export.py` in the code repository explains why the conversion is exact). On 1,200 development pairs the largest logit difference between the two was 1.4e-06 and the predicted label never differed (0 disagreements). The full system, with its calibrated rules and regression suites, is [`jithinpothireddy21/ease-delta-base`](https://huggingface.co/jithinpothireddy21/ease-delta-base). Other size: [`jithinpothireddy21/ease-delta-reader`](https://huggingface.co/jithinpothireddy21/ease-delta-reader). ## Licence CC BY-SA 4.0 for the weights, because VitaminC is share-alike (Creative Commons' guidance on AI training describes this as the cautious course). The code is Apache-2.0. Credit for the training data: VitaminC (Schuster, Fisch and Barzilay, NAACL 2021), MultiNLI (Williams, Nangia and Bowman, NAACL 2018), WANLI (Liu, Swayamdipta, Smith and Choi, EMNLP 2022). Backbone: ModernBERT (Warner et al., 2024). ## Citation ```bibtex @software{pothireddy2026easedelta, author = {Pothireddy, Jithin}, title = {EASE-Delta: revision-aware decision computation}, year = {2026}, url = {https://github.com/jithinsaireddy/ease-delta} } ```