| ---
|
| language:
|
| - hi
|
| - en
|
| license: mit
|
| base_model: MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7
|
| pipeline_tag: text-classification
|
| tags:
|
| - natural-language-inference
|
| - fact-verification
|
| - citation-verification
|
| - hinglish
|
| - code-mixing
|
| - rag
|
| ---
|
|
|
| # HingVerify: citation verifier for Hindi, Hinglish and English
|
|
|
| HingVerify decides whether a **claim** is supported by an **evidence passage**. It is built for retrieval-augmented
|
| generation (RAG) systems that answer questions about Indian government schemes, where the user and the answer may be in
|
| English, Hindi (Devanagari) or Hinglish (Roman-script Hindi mixed with English), and the official source may be in a
|
| different language.
|
|
|
| - **Labels:** `entailment` = SUPPORTED, `contradiction` = CONTRADICTED, `neutral` = NOT ENOUGH INFO
|
| - **Input:** premise = evidence passage, hypothesis = claim (max 512 tokens; truncate the evidence, not the claim)
|
| - **Base model:** mDeBERTa-v3-base fine-tuned on multilingual NLI (~280M parameters)
|
| - **Data:** [Teena2812/hingverify](https://huggingface.co/datasets/Teena2812/hingverify) (test set, training pool, corpus, RAG questions)
|
|
|
| ## Usage
|
| ```python
|
| from transformers import pipeline
|
| nli = pipeline("text-classification", model="Teena2812/hingverify-mdeberta", top_k=None)
|
| evidence = "Under the scheme an income support of 6,000/- per year in three equal installments will be provided to all land holding farmer families."
|
| print(nli({"text": evidence, "text_pair": "Kisan ko saal mein 8 hazaar milte hain."}))
|
| # contradiction has the highest score
|
| ```
|
|
|
| ## Training
|
| Fine-tuned on 801 synthetic claim–evidence pairs (126 for validation) written from official passages of 17 government
|
| schemes, balanced across the three labels and three claim languages, with hard negatives (wrong number, date, group,
|
| frequency or entity; negation; lakh/crore confusion; unsupported added details). A blind 10% check agreed on 92/93 labels.
|
| Learning rate 2e-5 (searched over 1e-5, 2e-5, 3e-5), batch size 16, up to 6 epochs with early stopping on validation
|
| macro-F1, 10% warm-up, weight decay 0.01, seed 42 (best of 42/43/44 on validation). One NVIDIA T4 GPU.
|
|
|
| ## Evaluation
|
| Human-adjudicated test set: 366 pairs from 6 schemes **not seen in training** (Cohen's kappa 0.946 between the human
|
| annotator and the final labels). Three-way macro-F1:
|
|
|
| | Claim language | Stock mDeBERTa-v3 | **HingVerify** (mean ± SD, 3 seeds) | gpt-oss-120b judge |
|
| |---|---|---|---|
|
| | English | 0.836 | **0.880 ± 0.019** | 0.951 |
|
| | Hindi | 0.828 | **0.949 ± 0.017** | 0.967 |
|
| | Hinglish | 0.792 | **0.856 ± 0.017** | 0.983 |
|
| | All | 0.819 | **0.895 ± 0.013** | 0.967 |
|
|
|
| Gains over the stock model are significant on Hindi (p = 0.0013) and Hinglish (p = 0.049), exact McNemar test.
|
| Inside an agentic RAG pipeline (60 citizen questions), using HingVerify instead of the stock verifier raised answer
|
| correctness from 0.917 to 0.979 (not statistically significant at this sample size).
|
|
|
| ## Limitations
|
| - A large LLM judge is still more accurate, especially on Hinglish and when claim and evidence are in different languages.
|
| - Weak spots: claims about the wrong beneficiary group and about payment frequency.
|
| - Trained on synthetic data from one domain (Indian government schemes); expect lower accuracy elsewhere.
|
| - It checks whether a passage supports a claim; it does not check whether the passage itself is current or true.
|
| Always confirm important information on the official scheme website.
|
|
|
| ## Authors
|
| Aditya Singh, Teena Kalia, Muntaha Malik, Sandeep Gupta (corresponding author). Dept. of Computer Science and
|
| Engineering, Sharda University.
|
|
|
| ## Citation
|
| A. Singh, T. Kalia, M. Malik and S. Gupta, "HingVerify: A Compact Fine-Tuned NLI Verifier for Citation Grounding in
|
| Hindi and Hinglish Retrieval-Augmented Generation" (under review).
|
|
|
