Next-Trace / README.md
Lamapi's picture
Publish Space with valid short description
c07bca1 verified
|
Raw History Blame Contribute Delete
3.48 kB
metadata
title: Next Trace
emoji: 🔎
colorFrom: green
colorTo: gray
sdk: static
app_file: index.html
pinned: false
license: cc-by-sa-4.0
short_description: Document answers with highlighted evidence

Next Trace

Next Trace answers where in a document the answer is. Give it a Turkish or English document and a question; it marks the sentence(s) the answer rests on, or says that the document does not contain enough evidence. It is a small learned reranker (18 language-light features → MLP) plus an answerability head, compared with a BM25 baseline.

Model: https://huggingface.co/thelamapi/Next-Trace — paste any document, ask a question, and compare with BM25; it runs entirely in the browser.

Results (BM25 + threshold → Next Trace)

Test set Questions Trace accuracy Evidence@1 on answerable Correct abstention on unanswerable Answered (answerable)
SQuAD2-dev (EN, adversarial unanswerables) 11873 0.492 → 0.510 0.825 → 0.842 0.415 → 0.318 0.645 → 0.801
XQuAD-tr (TR, zero-shot) 2327 0.719 → 0.788 0.770 → 0.803 0.934 → 0.972 0.612 → 0.721
XQuAD-en (EN) 2319 0.732 → 0.820 0.803 → 0.824 0.953 → 0.965 0.606 → 0.790

Trace accuracy counts an answerable question as correct only if the system answers and cites the gold sentence first, and an unanswerable question as correct only if it abstains. On SQuAD 2.0's adversarial unanswerables, the cited sentence contained the annotators' plausible but wrong answer 44.6% (BM25) vs 51.6% (Next Trace) of the time. Turkish is zero-shot: no Turkish text was used for training or thresholds.

Benchmark

Method

  • Sentence splitting and tokenisation are language-light: Turkish-aware lower-casing, 5-letter stems, TR+EN stop words.
  • Features: BM25 (normalised and rank), matched / missing IDF mass, best missing key term, character 4-gram Jaccard, bigram overlap, question-type cues (when/kaç ↔ digits, who/kim ↔ capitalised names), length, position, neighbour scores, document-level BM25 strength, share of question terms absent from the whole document, and numbers in the question missing from the sentence (2019 vs 2020).
  • Sentence model: MLP 18→32→32→1 (BCE) on SQuAD 2.0 train articles (split A, 126503 questions). Answerability head: logistic regression on 9 document features (split B). Training splits add unanswerables made by asking questions against another paragraph of the same article. Thresholds: split V (answerable ≥ 0.60; BM25 ≥ 2.79).
  • Splits are by article title; XQuAD and SQuAD dev are test-only.
  • Browser: line-by-line JavaScript port; Node parity on 9 documents, max abs difference 1.6653345369377348e-16.

Usage

python inference.py --doc "Kasaba kütüphanesi 1962 yılında açıldı. ..." --question "Kütüphane ne zaman açıldı?"
python inference.py --selftest

Limitations

  • Selects evidence sentences; it does not write an answer.
  • Lexical features: paraphrases with no word overlap and multi-hop questions are hard; long documents dilute BM25 statistics.
  • Turkish performance is measured on translated XQuAD only.
  • Failure examples for every test set are in evaluation.json. Hardware: NVIDIA L4; profile full.

Data: SQuAD 2.0 (Rajpurkar et al., 2018) and XQuAD (Artetxe et al., 2020), both CC BY-SA 4.0. The Space's examples file contains short excerpts under the same license.