--- license: cc-by-sa-4.0 library_name: numpy tags: - question-answering - evidence-retrieval - abstention - turkish - next-lab language: - tr - en datasets: - rajpurkar/squad_v2 - google/xquad --- ![trace](https://cdn-uploads.huggingface.co/production/uploads/67d46bc5fe6ad6f6511d6f44/hqrQbgBPUw9RK79ugyvhE.png) # Next Trace Next Trace answers *where* in a document the answer is. Give it a Turkish or English document and a question; it marks the sentence(s) the answer rests on, or says that **the document does not contain enough evidence**. It is a small learned reranker (18 language-light features → MLP) plus an answerability head, compared with a BM25 baseline. **Space:** https://huggingface.co/spaces/thelamapi/Next-Trace — paste any document, ask a question, and compare with BM25; it runs entirely in the browser. ## Results (BM25 + threshold → Next Trace) | Test set | Questions | Trace accuracy | Evidence@1 on answerable | Correct abstention on unanswerable | Answered (answerable) | |---|---:|---:|---:|---:|---:| | SQuAD2-dev (EN, adversarial unanswerables) | 11873 | 0.492 → **0.510** | 0.825 → 0.842 | 0.415 → 0.318 | 0.645 → 0.801 | | XQuAD-tr (TR, zero-shot) | 2327 | 0.719 → **0.788** | 0.770 → 0.803 | 0.934 → 0.972 | 0.612 → 0.721 | | XQuAD-en (EN) | 2319 | 0.732 → **0.820** | 0.803 → 0.824 | 0.953 → 0.965 | 0.606 → 0.790 | *Trace accuracy* counts an answerable question as correct only if the system answers **and** cites the gold sentence first, and an unanswerable question as correct only if it abstains. On SQuAD 2.0's adversarial unanswerables, the cited sentence contained the annotators' *plausible but wrong* answer 44.6% (BM25) vs 51.6% (Next Trace) of the time. Turkish is **zero-shot**: no Turkish text was used for training or thresholds. ![Benchmark](preview.png) ## Method * Sentence splitting and tokenisation are language-light: Turkish-aware lower-casing, 5-letter stems, TR+EN stop words. * Features: BM25 (normalised and rank), matched / missing IDF mass, best missing key term, character 4-gram Jaccard, bigram overlap, question-type cues (when/kaç ↔ digits, who/kim ↔ capitalised names), length, position, neighbour scores, document-level BM25 strength, share of question terms absent from the whole document, and numbers in the question missing from the sentence (2019 vs 2020). * Sentence model: MLP 18→32→32→1 (BCE) on SQuAD 2.0 train articles (split A, 126503 questions). Answerability head: logistic regression on 9 document features (split B). Training splits add unanswerables made by asking questions against another paragraph of the same article. Thresholds: split V (answerable ≥ 0.60; BM25 ≥ 2.79). * Splits are by article title; XQuAD and SQuAD dev are test-only. * Browser: line-by-line JavaScript port; Node parity on 9 documents, max abs difference 1.6653345369377348e-16. ## Usage ```bash python inference.py --doc "Kasaba kütüphanesi 1962 yılında açıldı. ..." --question "Kütüphane ne zaman açıldı?" python inference.py --selftest ``` ## Limitations * Selects evidence sentences; it does not write an answer. * Lexical features: paraphrases with no word overlap and multi-hop questions are hard; long documents dilute BM25 statistics. * Turkish performance is measured on translated XQuAD only. * Failure examples for every test set are in `evaluation.json`. Hardware: NVIDIA L4; profile full. Data: SQuAD 2.0 (Rajpurkar et al., 2018) and XQuAD (Artetxe et al., 2020), both CC BY-SA 4.0. The Space's examples file contains short excerpts under the same license.