Spaces:
Running
Running
File size: 3,482 Bytes
15113fe c07bca1 15113fe c07bca1 15113fe c07bca1 15113fe c07bca1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | ---
title: Next Trace
emoji: 🔎
colorFrom: green
colorTo: gray
sdk: static
app_file: index.html
pinned: false
license: cc-by-sa-4.0
short_description: Document answers with highlighted evidence
---
# Next Trace
Next Trace answers *where* in a document the answer is. Give it a Turkish or English document and a question; it
marks the sentence(s) the answer rests on, or says that **the document does not contain enough evidence**. It is a
small learned reranker (18 language-light features → MLP) plus an answerability head, compared with a BM25 baseline.
**Model:** https://huggingface.co/thelamapi/Next-Trace — paste any document, ask a question, and compare with BM25; it runs entirely in the browser.
## Results (BM25 + threshold → Next Trace)
| Test set | Questions | Trace accuracy | Evidence@1 on answerable | Correct abstention on unanswerable | Answered (answerable) |
|---|---:|---:|---:|---:|---:|
| SQuAD2-dev (EN, adversarial unanswerables) | 11873 | 0.492 → **0.510** | 0.825 → 0.842 | 0.415 → 0.318 | 0.645 → 0.801 |
| XQuAD-tr (TR, zero-shot) | 2327 | 0.719 → **0.788** | 0.770 → 0.803 | 0.934 → 0.972 | 0.612 → 0.721 |
| XQuAD-en (EN) | 2319 | 0.732 → **0.820** | 0.803 → 0.824 | 0.953 → 0.965 | 0.606 → 0.790 |
*Trace accuracy* counts an answerable question as correct only if the system answers **and** cites the gold sentence first,
and an unanswerable question as correct only if it abstains. On SQuAD 2.0's adversarial unanswerables, the cited sentence
contained the annotators' *plausible but wrong* answer 44.6% (BM25) vs
51.6% (Next Trace) of the time.
Turkish is **zero-shot**: no Turkish text was used for training or thresholds.

## Method
* Sentence splitting and tokenisation are language-light: Turkish-aware lower-casing, 5-letter stems, TR+EN stop words.
* Features: BM25 (normalised and rank), matched / missing IDF mass, best missing key term, character 4-gram Jaccard, bigram overlap,
question-type cues (when/kaç ↔ digits, who/kim ↔ capitalised names), length, position, neighbour scores, document-level BM25 strength,
share of question terms absent from the whole document, and numbers in the question missing from the sentence (2019 vs 2020).
* Sentence model: MLP 18→32→32→1 (BCE) on SQuAD 2.0 train articles (split A, 126503 questions).
Answerability head: logistic regression on 9 document features (split B). Training splits add unanswerables made by asking questions against another paragraph of the same article. Thresholds: split V (answerable ≥ 0.60; BM25 ≥ 2.79).
* Splits are by article title; XQuAD and SQuAD dev are test-only.
* Browser: line-by-line JavaScript port; Node parity on 9 documents, max abs difference 1.6653345369377348e-16.
## Usage
```bash
python inference.py --doc "Kasaba kütüphanesi 1962 yılında açıldı. ..." --question "Kütüphane ne zaman açıldı?"
python inference.py --selftest
```
## Limitations
* Selects evidence sentences; it does not write an answer.
* Lexical features: paraphrases with no word overlap and multi-hop questions are hard; long documents dilute BM25 statistics.
* Turkish performance is measured on translated XQuAD only.
* Failure examples for every test set are in `evaluation.json`. Hardware: NVIDIA L4; profile full.
Data: SQuAD 2.0 (Rajpurkar et al., 2018) and XQuAD (Artetxe et al., 2020), both CC BY-SA 4.0. The Space's examples file contains short excerpts under the same license.
|