Lamapi commited on
Commit
c07bca1
·
verified ·
1 Parent(s): 15113fe

Publish Space with valid short description

Browse files
Files changed (12) hide show
  1. README.md +52 -3
  2. app.js +62 -0
  3. core.js +134 -0
  4. evaluation.json +1364 -0
  5. examples.json +1 -0
  6. index.html +44 -17
  7. model.json +1 -0
  8. nlrt.js +219 -0
  9. nlui.js +123 -0
  10. parity_cases.json +1 -0
  11. preview.png +0 -0
  12. style.css +53 -26
README.md CHANGED
@@ -1,10 +1,59 @@
1
  ---
2
  title: Next Trace
3
- emoji: 📈
4
- colorFrom: yellow
5
  colorTo: gray
6
  sdk: static
 
7
  pinned: false
 
 
8
  ---
9
 
10
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  title: Next Trace
3
+ emoji: 🔎
4
+ colorFrom: green
5
  colorTo: gray
6
  sdk: static
7
+ app_file: index.html
8
  pinned: false
9
+ license: cc-by-sa-4.0
10
+ short_description: Document answers with highlighted evidence
11
  ---
12
 
13
+ # Next Trace
14
+
15
+ Next Trace answers *where* in a document the answer is. Give it a Turkish or English document and a question; it
16
+ marks the sentence(s) the answer rests on, or says that **the document does not contain enough evidence**. It is a
17
+ small learned reranker (18 language-light features → MLP) plus an answerability head, compared with a BM25 baseline.
18
+
19
+ **Model:** https://huggingface.co/thelamapi/Next-Trace — paste any document, ask a question, and compare with BM25; it runs entirely in the browser.
20
+
21
+ ## Results (BM25 + threshold → Next Trace)
22
+
23
+ | Test set | Questions | Trace accuracy | Evidence@1 on answerable | Correct abstention on unanswerable | Answered (answerable) |
24
+ |---|---:|---:|---:|---:|---:|
25
+ | SQuAD2-dev (EN, adversarial unanswerables) | 11873 | 0.492 → **0.510** | 0.825 → 0.842 | 0.415 → 0.318 | 0.645 → 0.801 |
26
+ | XQuAD-tr (TR, zero-shot) | 2327 | 0.719 → **0.788** | 0.770 → 0.803 | 0.934 → 0.972 | 0.612 → 0.721 |
27
+ | XQuAD-en (EN) | 2319 | 0.732 → **0.820** | 0.803 → 0.824 | 0.953 → 0.965 | 0.606 → 0.790 |
28
+
29
+ *Trace accuracy* counts an answerable question as correct only if the system answers **and** cites the gold sentence first,
30
+ and an unanswerable question as correct only if it abstains. On SQuAD 2.0's adversarial unanswerables, the cited sentence
31
+ contained the annotators' *plausible but wrong* answer 44.6% (BM25) vs
32
+ 51.6% (Next Trace) of the time.
33
+ Turkish is **zero-shot**: no Turkish text was used for training or thresholds.
34
+
35
+ ![Benchmark](preview.png)
36
+
37
+ ## Method
38
+ * Sentence splitting and tokenisation are language-light: Turkish-aware lower-casing, 5-letter stems, TR+EN stop words.
39
+ * Features: BM25 (normalised and rank), matched / missing IDF mass, best missing key term, character 4-gram Jaccard, bigram overlap,
40
+ question-type cues (when/kaç ↔ digits, who/kim ↔ capitalised names), length, position, neighbour scores, document-level BM25 strength,
41
+ share of question terms absent from the whole document, and numbers in the question missing from the sentence (2019 vs 2020).
42
+ * Sentence model: MLP 18→32→32→1 (BCE) on SQuAD 2.0 train articles (split A, 126503 questions).
43
+ Answerability head: logistic regression on 9 document features (split B). Training splits add unanswerables made by asking questions against another paragraph of the same article. Thresholds: split V (answerable ≥ 0.60; BM25 ≥ 2.79).
44
+ * Splits are by article title; XQuAD and SQuAD dev are test-only.
45
+ * Browser: line-by-line JavaScript port; Node parity on 9 documents, max abs difference 1.6653345369377348e-16.
46
+
47
+ ## Usage
48
+ ```bash
49
+ python inference.py --doc "Kasaba kütüphanesi 1962 yılında açıldı. ..." --question "Kütüphane ne zaman açıldı?"
50
+ python inference.py --selftest
51
+ ```
52
+
53
+ ## Limitations
54
+ * Selects evidence sentences; it does not write an answer.
55
+ * Lexical features: paraphrases with no word overlap and multi-hop questions are hard; long documents dilute BM25 statistics.
56
+ * Turkish performance is measured on translated XQuAD only.
57
+ * Failure examples for every test set are in `evaluation.json`. Hardware: NVIDIA L4; profile full.
58
+
59
+ Data: SQuAD 2.0 (Rajpurkar et al., 2018) and XQuAD (Artetxe et al., 2020), both CC BY-SA 4.0. The Space's examples file contains short excerpts under the same license.
app.js ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /* Next Trace Space: paste a document and a question; see which sentences support the answer, or an abstention. */
2
+ const { $, h, fmt } = NLUI;
3
+ let ctx, examples;
4
+
5
+ function render() {
6
+ const doc = $("doc").value, q = $("q").value, mode = $("mode").value;
7
+ const t0 = performance.now();
8
+ const r = mode === "bm25" ? Core.bm25Baseline(ctx, q, doc) : Core.scoreDocument(ctx, q, doc);
9
+ const ms = performance.now() - t0;
10
+ const ui = $("lang").value === "tr";
11
+ const v = $("verdict"); v.innerHTML = "";
12
+ if (!r.spans.length) { v.textContent = ui ? "Belge boş." : "The document is empty."; return r; }
13
+ if (r.abstain) {
14
+ v.append(h("div", { class: "badge warn", style: "font-size:15px;padding:6px 12px" }, ui ? "Belge bu soruyu yanıtlamak için yeterli kanıt içermiyor." : "The document does not contain enough evidence to answer."));
15
+ } else {
16
+ v.append(h("div", { class: "badge", style: "font-size:15px;padding:6px 12px" }, (ui ? "Yanıtın dayandığı cümle: " : "Answer supported by sentence ") + r.evidence.map(i => "#" + (i + 1)).join(", ")));
17
+ }
18
+ const extra = mode === "bm25" ? `max BM25 ${fmt(Math.max(...r.bm25), 2)} (threshold ${fmt(ctx.m.meta.thresholds.bm25, 2)})`
19
+ : `answerability ${fmt(r.answerable, 3)} (threshold ${fmt(ctx.m.meta.thresholds.answerable, 3)})`;
20
+ v.append(h("div", { class: "small muted", style: "margin-top:6px" }, `${extra} · ${r.spans.length} sentences · ${fmt(ms, 1)} ms`));
21
+ const out = $("marked"); out.innerHTML = "";
22
+ const mx = Math.max(...r.bm25, 1e-9);
23
+ r.spans.forEach(([a, b], i) => {
24
+ const s = mode === "bm25" ? r.bm25[i] / mx : r.probs[i];
25
+ const ev = r.evidence.includes(i);
26
+ out.append(h("span", { style: `background:rgba(255,200,40,${0.08 + 0.6 * s});border-radius:4px;padding:1px 2px;${ev ? "outline:2px solid #0b6e4f;" : ""}`, title: `#${i + 1}: ${fmt(s, 3)}` },
27
+ h("sup", { class: "muted" }, String(i + 1)), doc.slice(a, b)), " ");
28
+ });
29
+ const tb = $("tbl"); tb.innerHTML = "";
30
+ tb.append(h("tr", {}, h("th", {}, "#"), h("th", { class: "num" }, mode === "bm25" ? "BM25" : "evidence p"), h("th", {}, ui ? "cümle" : "sentence")));
31
+ const order = r.spans.map((_, i) => i).sort((x, y) => (mode === "bm25" ? r.bm25[y] - r.bm25[x] : r.probs[y] - r.probs[x]));
32
+ for (const i of order.slice(0, 5)) tb.append(h("tr", {}, h("td", {}, "#" + (i + 1)), h("td", { class: "num" }, fmt(mode === "bm25" ? r.bm25[i] : r.probs[i], 3)),
33
+ h("td", {}, doc.slice(r.spans[i][0], r.spans[i][1]).slice(0, 140))));
34
+ return r;
35
+ }
36
+
37
+ function load(i) {
38
+ const e = examples[i];
39
+ $("doc").value = e.document; $("q").value = e.question; $("lang").value = e.lang;
40
+ $("src").textContent = e.source;
41
+ render();
42
+ }
43
+
44
+ window.__liveTest = function () {
45
+ $("mode").value = "learned";
46
+ $("doc").value = "Kasaba kütüphanesi 1962 yılında açıldı. Binanın mimarı Selma Arı'dır. Kütüphanede bugün yaklaşık 40 bin kitap bulunuyor. Her yıl haziran ayında bir kitap şenliği düzenleniyor.";
47
+ $("q").value = "Kütüphane hangi yıl açıldı?"; $("q").dispatchEvent(new Event("input"));
48
+ const a = render();
49
+ $("q").value = "Kütüphanenin ilk müdürü kimdi?"; const b = render();
50
+ return { ok: a.spans.length === 4 && a.probs.length === 4, answerable_evidence: a.evidence, answerable_abstain: a.abstain, answerable_p: a.answerable,
51
+ unanswerable_abstain: b.abstain, unanswerable_p: b.answerable };
52
+ };
53
+
54
+ NLUI.boot(Core, async c => {
55
+ ctx = c;
56
+ examples = await (await fetch("examples.json")).json();
57
+ examples.forEach((e, i) => $("ex").append(h("option", { value: i }, `${e.lang.toUpperCase()} · ${e.title}`)));
58
+ $("ex").addEventListener("change", e => load(Number(e.target.value)));
59
+ for (const id of ["doc", "q"]) $(id).addEventListener("input", render);
60
+ for (const id of ["mode", "lang"]) $(id).addEventListener("change", render);
61
+ load(0);
62
+ });
core.js ADDED
@@ -0,0 +1,134 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /* Next Trace compute core: line-by-line port of trace_core.py. */
2
+ (function (root, factory) {
3
+ if (typeof module === "object" && module.exports) module.exports = factory();
4
+ else root.Core = factory();
5
+ })(typeof self !== "undefined" ? self : this, function () {
6
+ "use strict";
7
+ const ASSETS = ["model.json"];
8
+ const STOP = new Set(`a an the of in on at to for from by with and or but is are was were be been being it its this that these those as
9
+ which who whom whose what when where why how do does did has have had not no than then there their they he she his her
10
+ we you your i me my our us can could would should will may might also into over under about after before between during
11
+ bir ve ile de da mi mı mu mü ne bu şu o için gibi daha en çok olarak olan olduğu ise ki ama veya hangi kim kimin neden
12
+ nasıl nerede ne zaman kaç her bazı tüm diğer sonra önce kadar göre üzerinde arasında tarafından`.split(/\s+/));
13
+ const WH_TIME = new Set(["when", "year", "zaman", "yıl", "yılında", "tarih", "century", "yüzyıl"]);
14
+ const WH_NUM = new Set(["many", "much", "kaç", "number", "percent", "yüzde", "sayı"]);
15
+ const WH_WHO = new Set(["who", "whom", "whose", "kim", "kimin", "kime", "kimdir"]);
16
+ const SENT_RE = /(?<=[.!?])\s+(?=[A-ZÇĞİÖŞÜ0-9"“(])/g;
17
+ const ALNUM = /[\p{L}\p{N}]/u;
18
+
19
+ const lowerTr = s => s.replace(/I/g, "ı").replace(/İ/g, "i").toLowerCase();
20
+
21
+ function splitSentences(text) {
22
+ text = text.replace(//g, " ");
23
+ const spans = []; let start = 0, m;
24
+ SENT_RE.lastIndex = 0;
25
+ while ((m = SENT_RE.exec(text)) !== null) {
26
+ if (text.slice(start, m.index).trim()) spans.push([start, m.index]);
27
+ start = m.index + m[0].length;
28
+ }
29
+ if (text.slice(start).trim()) spans.push([start, text.length]);
30
+ return spans;
31
+ }
32
+
33
+ function words(text) {
34
+ const out = []; let cur = "";
35
+ for (const ch of lowerTr(text)) { if (ALNUM.test(ch)) cur += ch; else if (cur) { out.push(cur); cur = ""; } }
36
+ if (cur) out.push(cur);
37
+ return out;
38
+ }
39
+ const stems = text => words(text).filter(w => !STOP.has(w)).map(w => Array.from(w).slice(0, 5).join(""));
40
+
41
+ function bm25(q, sents, k1 = 1.2, b = 0.75) {
42
+ const n = sents.length; let tot = 0; for (const s of sents) tot += s.length;
43
+ const avg = tot / Math.max(n, 1), df = new Map();
44
+ for (const s of sents) for (const t of new Set(s)) df.set(t, (df.get(t) || 0) + 1);
45
+ const idf = new Map(); for (const [t, d] of df) idf.set(t, Math.log(1.0 + (n - d + 0.5) / (d + 0.5)));
46
+ const scores = sents.map(s => {
47
+ const tf = new Map(); for (const t of s) tf.set(t, (tf.get(t) || 0) + 1);
48
+ let sc = 0.0;
49
+ for (const t of q) if (tf.has(t)) { const f = tf.get(t); sc += idf.get(t) * f * (k1 + 1) / (f + k1 * (1 - b + b * s.length / Math.max(avg, 1e-9))); }
50
+ return sc;
51
+ });
52
+ return [scores, idf];
53
+ }
54
+
55
+ function charNgrams(text, n = 4) {
56
+ const t = words(text).join(" "), out = new Set();
57
+ for (let i = 0; i < Math.max(0, t.length - n + 1); i++) out.add(t.slice(i, i + n));
58
+ return out;
59
+ }
60
+ const isUpper = c => c !== c.toLowerCase() && c === c.toUpperCase();
61
+
62
+ function sentenceFeatures(question, sents) {
63
+ const qSt = Array.from(new Set(stems(question))), qWords = new Set(words(question));
64
+ const sSt = sents.map(stems);
65
+ const [scores, idf] = bm25(qSt, sSt);
66
+ const mx = scores.length ? Math.max(...scores) : 0.0;
67
+ const order = scores.map((_, i) => i).sort((a, b) => (scores[b] - scores[a]) || (a - b));
68
+ const rank = new Map(order.map((i, r) => [i, r]));
69
+ const qIdf = qSt.map(t => (idf.has(t) ? idf.get(t) : Math.log(1.0 + (sents.length + 0.5) / 0.5)));
70
+ const qIdfSum = qIdf.reduce((a, b) => a + b, 0) || 1.0, qMax = qIdf.length ? Math.max(...qIdf) : 0;
71
+ const qNg = charNgrams(question);
72
+ const has = set => [...set].some(w => qWords.has(w));
73
+ const wantTime = has(WH_TIME), wantNum = has(WH_NUM), wantWho = has(WH_WHO);
74
+ const docSt = new Set(); for (const st of sSt) for (const t of st) docSt.add(t);
75
+ const docMissing = qSt.filter(t => !docSt.has(t)).length / Math.max(qSt.length, 1);
76
+ const qNums = words(question).filter(w => w && /^[0-9]+$/.test(w));
77
+ const bigQ = new Set(); for (let j = 0; j + 1 < qSt.length; j++) bigQ.add(qSt[j] + "\u0000" + qSt[j + 1]);
78
+ const feats = sents.map((s, i) => {
79
+ const ss = new Set(sSt[i]);
80
+ const matched = [], missing = [];
81
+ qSt.forEach((t, j) => (ss.has(t) ? matched : missing).push(qIdf[j]));
82
+ const ng = charNgrams(s); let inter = 0; for (const g of qNg) if (ng.has(g)) inter++;
83
+ const uni = qNg.size + ng.size - inter, jac = inter / Math.max(uni, 1);
84
+ const bigS = new Set(); for (let j = 0; j + 1 < sSt[i].length; j++) bigS.add(sSt[i][j] + "\u0000" + sSt[i][j + 1]);
85
+ let bi = 0; for (const g of bigQ) if (bigS.has(g)) bi++;
86
+ const hasNum = /[0-9]/.test(s);
87
+ const hasCap = s.split(/\s+/).filter(Boolean).slice(1).some(w => isUpper(w[0]));
88
+ const nb = [i - 1, i + 1].filter(j => j >= 0 && j < sents.length).map(j => scores[j]);
89
+ return [mx > 0 ? scores[i] / mx : 0.0, rank.get(i) / Math.max(sents.length - 1, 1), matched.length / Math.max(qSt.length, 1),
90
+ matched.reduce((a, b) => a + b, 0) / qIdfSum, matched.length && qIdf.length ? Math.max(...matched) / qMax : 0.0,
91
+ missing.length && qIdf.length ? Math.max(...missing) / qMax : 0.0, jac, Math.min(bi, 3) / 3.0,
92
+ (wantTime || wantNum) && hasNum ? 1.0 : 0.0, wantWho && hasCap ? 1.0 : 0.0, (wantTime || wantNum) && !hasNum ? 1.0 : 0.0,
93
+ Math.log(1 + sSt[i].length) / 4.0, i / Math.max(sents.length - 1, 1), nb.length && mx > 0 ? Math.max(...nb) / mx : 0.0,
94
+ Math.log(1 + mx) / 3.0, Math.log(1 + sents.length) / 4.0, docMissing,
95
+ qNums.length && qNums.some(n => !new Set(words(s)).has(n)) ? 1.0 : 0.0];
96
+ });
97
+ return [feats, scores];
98
+ }
99
+
100
+ const sigmoid = z => 1.0 / (1.0 + Math.exp(-Math.max(-50.0, Math.min(50.0, z))));
101
+
102
+ function scoreDocument(ctx, question, document) {
103
+ const spans = splitSentences(document), sents = spans.map(([a, b]) => document.slice(a, b));
104
+ if (!sents.length) return { spans: [], probs: [], bm25: [], answerable: 0.0, abstain: true, evidence: [] };
105
+ const [feats, scores] = sentenceFeatures(question, sents);
106
+ const probs = feats.map(f => sigmoid(NLRT.forward(ctx.m.nets.sent, NLRT.T([f.length], Float64Array.from(f))).data[0]));
107
+ const order = probs.map((_, i) => i).sort((a, b) => (probs[b] - probs[a]) || (a - b));
108
+ const top = order[0], p2 = order.length > 1 ? probs[order[1]] : 0.0;
109
+ const df = [probs[top], probs[top] - p2, feats[top][2], feats[top][3], feats[top][5], feats[top][10], feats[top][14], feats[top][16], feats[top][17]];
110
+ const w = ctx.m.tensors.ans_w.data; let z = 0; for (let i = 0; i < df.length; i++) z += w[i] * df[i];
111
+ const ans = sigmoid(z + ctx.m.tensors.ans_b.data[0]), th = ctx.m.meta.thresholds;
112
+ const abstain = ans < th.answerable;
113
+ const evidence = abstain ? [] : order.slice(0, 2).filter(i => i === top || probs[i] >= th.second_sentence);
114
+ return { spans, probs, bm25: scores, answerable: ans, abstain, evidence };
115
+ }
116
+
117
+ function bm25Baseline(ctx, question, document) {
118
+ const spans = splitSentences(document), sents = spans.map(([a, b]) => document.slice(a, b));
119
+ if (!sents.length) return { spans: [], abstain: true, evidence: [], bm25: [] };
120
+ const [scores] = bm25(Array.from(new Set(stems(question))), sents.map(stems));
121
+ let best = 0; for (let i = 1; i < scores.length; i++) if (scores[i] > scores[best]) best = i;
122
+ const abstain = scores[best] < ctx.m.meta.thresholds.bm25;
123
+ return { spans, abstain, evidence: abstain ? [] : [best], bm25: scores };
124
+ }
125
+
126
+ function init(assets) { return { m: NLRT.decode(assets["model.json"]) }; }
127
+
128
+ function runCase(ctx, input) {
129
+ const r = scoreDocument(ctx, input.question, input.document), b = bm25Baseline(ctx, input.question, input.document);
130
+ return { spans: r.spans, probs: r.probs, answerable: r.answerable, abstain: r.abstain, evidence: r.evidence, bm25_abstain: b.abstain, bm25_evidence: b.evidence };
131
+ }
132
+
133
+ return { ASSETS, init, splitSentences, scoreDocument, bm25Baseline, runCase, words, stems };
134
+ });
evaluation.json ADDED
@@ -0,0 +1,1364 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "project": "Next Trace",
3
+ "profile": "full",
4
+ "seed": 73,
5
+ "device": {
6
+ "python": "3.13.16",
7
+ "platform": "Linux-6.6.159+-x86_64-with-glibc2.39",
8
+ "cpu_count": 12,
9
+ "numpy": "2.4.6",
10
+ "torch": "2.14.1+cu130",
11
+ "cuda": true,
12
+ "gpu": "NVIDIA L4",
13
+ "colab": true
14
+ },
15
+ "data": {
16
+ "sources": {
17
+ "squad_train": {
18
+ "url": "https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json",
19
+ "file": "train-v2.0.json",
20
+ "bytes": 42123633,
21
+ "sha256": "68dcfbb971bd3e96d5b46c7177b16c1a4e7d4bdef19fb204502738552dede002",
22
+ "verified_against_pinned_sha256": false
23
+ },
24
+ "squad_dev": {
25
+ "url": "https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json",
26
+ "file": "dev-v2.0.json",
27
+ "bytes": 4370528,
28
+ "sha256": "80a5225e94905956a6446d296ca1093975c4d3b3260f1d6c8f68bc2ab77182d8",
29
+ "verified_against_pinned_sha256": false
30
+ },
31
+ "xquad_tr": {
32
+ "url": "https://raw.githubusercontent.com/google-deepmind/xquad/master/xquad.tr.json",
33
+ "file": "xquad.tr.json",
34
+ "bytes": 729506,
35
+ "sha256": "92179a564774b7696100d144c1e10870d0a966b6fccbdd254a65b9d2ab1971cc",
36
+ "verified_against_pinned_sha256": false
37
+ },
38
+ "xquad_en": {
39
+ "url": "https://raw.githubusercontent.com/google-deepmind/xquad/master/xquad.en.json",
40
+ "file": "xquad.en.json",
41
+ "bytes": 609383,
42
+ "sha256": "e4c57d1c9143aaa1c5d265ba5987a65f4e69528d2a98f29d6e75019b10344f29",
43
+ "verified_against_pinned_sha256": false
44
+ }
45
+ },
46
+ "license": "CC BY-SA 4.0 (SQuAD 2.0, XQuAD)",
47
+ "versions": {
48
+ "squad": "2.0",
49
+ "xquad": "GitHub master at download time (sha256 recorded)"
50
+ },
51
+ "splits": {
52
+ "squad_train_A": 126503,
53
+ "squad_train_B": 16851,
54
+ "squad_train_V": 15937,
55
+ "SQuAD2-dev (EN, adversarial unanswerables)": {
56
+ "questions": 11873,
57
+ "unanswerable": 5945
58
+ },
59
+ "XQuAD-tr (TR, zero-shot)": {
60
+ "questions": 2327,
61
+ "unanswerable": 1137
62
+ },
63
+ "XQuAD-en (EN)": {
64
+ "questions": 2319,
65
+ "unanswerable": 1129
66
+ },
67
+ "rule": "SQuAD train split by article title (A: sentence model, B: answerability head, V: thresholds); each split also gets ~35% extra unanswerables made by asking a question against another paragraph of the same article; XQuAD and SQuAD dev only for testing"
68
+ }
69
+ },
70
+ "thresholds": {
71
+ "bm25": 2.7921873897475704,
72
+ "answerable": 0.6,
73
+ "second_sentence": 0.5
74
+ },
75
+ "validation": {
76
+ "bm25": {
77
+ "questions": 15937,
78
+ "trace_accuracy": 0.5530526447888561,
79
+ "evidence_top1_on_answerable": 0.7978169818504887,
80
+ "answered_rate_on_answerable": 0.597537758598807,
81
+ "correct_abstention_on_unanswerable": 0.5953090096798213,
82
+ "cited_plausible_distractor_on_unanswerable": 0.29573095060809135,
83
+ "answerable": 7879,
84
+ "unanswerable": 8058
85
+ },
86
+ "learned": {
87
+ "questions": 15937,
88
+ "trace_accuracy": 0.5940892263286691,
89
+ "evidence_top1_on_answerable": 0.8188856453864704,
90
+ "answered_rate_on_answerable": 0.7587257266150527,
91
+ "correct_abstention_on_unanswerable": 0.5409530900967983,
92
+ "cited_plausible_distractor_on_unanswerable": 0.339662447257384,
93
+ "answerable": 7879,
94
+ "unanswerable": 8058
95
+ }
96
+ },
97
+ "results": {
98
+ "SQuAD2-dev (EN, adversarial unanswerables)": {
99
+ "bm25": {
100
+ "questions": 11873,
101
+ "trace_accuracy": 0.49237766360650215,
102
+ "evidence_top1_on_answerable": 0.8247300944669366,
103
+ "answered_rate_on_answerable": 0.6454116059379217,
104
+ "correct_abstention_on_unanswerable": 0.4146341463414634,
105
+ "cited_plausible_distractor_on_unanswerable": 0.4462573591253154,
106
+ "answerable": 5928,
107
+ "unanswerable": 5945
108
+ },
109
+ "learned": {
110
+ "questions": 11873,
111
+ "trace_accuracy": 0.5104017518739998,
112
+ "evidence_top1_on_answerable": 0.8424426450742241,
113
+ "answered_rate_on_answerable": 0.800944669365722,
114
+ "correct_abstention_on_unanswerable": 0.3184188393608074,
115
+ "cited_plausible_distractor_on_unanswerable": 0.5164003364171573,
116
+ "answerable": 5928,
117
+ "unanswerable": 5945
118
+ }
119
+ },
120
+ "XQuAD-tr (TR, zero-shot)": {
121
+ "bm25": {
122
+ "questions": 2327,
123
+ "trace_accuracy": 0.7189514396218307,
124
+ "evidence_top1_on_answerable": 0.7697478991596639,
125
+ "answered_rate_on_answerable": 0.611764705882353,
126
+ "correct_abstention_on_unanswerable": 0.9340369393139841,
127
+ "cited_plausible_distractor_on_unanswerable": 0.0,
128
+ "answerable": 1190,
129
+ "unanswerable": 1137
130
+ },
131
+ "learned": {
132
+ "questions": 2327,
133
+ "trace_accuracy": 0.7877094972067039,
134
+ "evidence_top1_on_answerable": 0.8025210084033614,
135
+ "answered_rate_on_answerable": 0.7210084033613445,
136
+ "correct_abstention_on_unanswerable": 0.9718557607739666,
137
+ "cited_plausible_distractor_on_unanswerable": 0.0,
138
+ "answerable": 1190,
139
+ "unanswerable": 1137
140
+ }
141
+ },
142
+ "XQuAD-en (EN)": {
143
+ "bm25": {
144
+ "questions": 2319,
145
+ "trace_accuracy": 0.7317809400603709,
146
+ "evidence_top1_on_answerable": 0.8033613445378152,
147
+ "answered_rate_on_answerable": 0.6058823529411764,
148
+ "correct_abstention_on_unanswerable": 0.9530558015943312,
149
+ "cited_plausible_distractor_on_unanswerable": 0.0,
150
+ "answerable": 1190,
151
+ "unanswerable": 1129
152
+ },
153
+ "learned": {
154
+ "questions": 2319,
155
+ "trace_accuracy": 0.8201811125485123,
156
+ "evidence_top1_on_answerable": 0.8243697478991596,
157
+ "answered_rate_on_answerable": 0.7899159663865546,
158
+ "correct_abstention_on_unanswerable": 0.9654561558901683,
159
+ "cited_plausible_distractor_on_unanswerable": 0.0,
160
+ "answerable": 1190,
161
+ "unanswerable": 1129
162
+ }
163
+ }
164
+ },
165
+ "failure_examples": {
166
+ "SQuAD2-dev (EN, adversarial unanswerables)": {
167
+ "bm25": [
168
+ {
169
+ "question": "In what country is Normandy located?",
170
+ "answerable": true,
171
+ "abstained": true,
172
+ "cited": "The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France.",
173
+ "gold_answer": "France"
174
+ },
175
+ {
176
+ "question": "When were the Normans in Normandy?",
177
+ "answerable": true,
178
+ "abstained": true,
179
+ "cited": "The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France.",
180
+ "gold_answer": "10th and 11th centuries"
181
+ },
182
+ {
183
+ "question": "From which countries did the Norse originate?",
184
+ "answerable": true,
185
+ "abstained": true,
186
+ "cited": "They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West F",
187
+ "gold_answer": "Denmark, Iceland and Norway"
188
+ },
189
+ {
190
+ "question": "Who was the Norse leader?",
191
+ "answerable": true,
192
+ "abstained": true,
193
+ "cited": "They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West F",
194
+ "gold_answer": "Rollo"
195
+ },
196
+ {
197
+ "question": "Who gave their name to Normandy in the 1000's and 1100's",
198
+ "answerable": false,
199
+ "abstained": false,
200
+ "cited": "The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France.",
201
+ "gold_answer": null
202
+ },
203
+ {
204
+ "question": "Who did King Charles III swear fealty to?",
205
+ "answerable": false,
206
+ "abstained": false,
207
+ "cited": "They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West F",
208
+ "gold_answer": null
209
+ },
210
+ {
211
+ "question": "Who ruled the duchy of Normandy",
212
+ "answerable": true,
213
+ "abstained": true,
214
+ "cited": "The Duchy of Normandy, which they formed by treaty with the French crown, was a great fief of medieval France, and under Richard I of Normandy was forged into a cohesive and formidable principality in",
215
+ "gold_answer": "Richard I"
216
+ },
217
+ {
218
+ "question": "What religion were the Normans",
219
+ "answerable": true,
220
+ "abstained": true,
221
+ "cited": "They adopted the Gallo-Romance language of the Frankish land they settled, their dialect becoming known as Norman, Normaund or Norman French, an important literary language.",
222
+ "gold_answer": "Catholic"
223
+ },
224
+ {
225
+ "question": "What type of major impact did the Norman dynasty have on modern Europe?",
226
+ "answerable": false,
227
+ "abstained": false,
228
+ "cited": "The Norman dynasty had a major political, cultural and military impact on medieval Europe and even the Near East.",
229
+ "gold_answer": null
230
+ },
231
+ {
232
+ "question": "Who was famed for their Christian spirit?",
233
+ "answerable": false,
234
+ "abstained": false,
235
+ "cited": "The Normans were famed for their martial spirit and eventually for their Christian piety, becoming exponents of the Catholic orthodoxy into which they assimilated.",
236
+ "gold_answer": null
237
+ },
238
+ {
239
+ "question": "Who assimilted the Roman language?",
240
+ "answerable": false,
241
+ "abstained": false,
242
+ "cited": "They adopted the Gallo-Romance language of the Frankish land they settled, their dialect becoming known as Norman, Normaund or Norman French, an important literary language.",
243
+ "gold_answer": null
244
+ },
245
+ {
246
+ "question": "What principality did William the conquerer found?",
247
+ "answerable": false,
248
+ "abstained": false,
249
+ "cited": "Norman adventurers founded the Kingdom of Sicily under Roger II after conquering southern Italy on the Saracens and Byzantines, and an expedition on behalf of their duke, William the Conqueror, led to",
250
+ "gold_answer": null
251
+ },
252
+ {
253
+ "question": "What is the original meaning of the word Norman?",
254
+ "answerable": true,
255
+ "abstained": true,
256
+ "cited": "The English name \"Normans\" comes from the French words Normans/Normanz, plural of Normant, modern French normand, which is itself borrowed from Old Low Franconian Nortmann \"Northman\" or directly from ",
257
+ "gold_answer": "Viking"
258
+ },
259
+ {
260
+ "question": "When was the Latin version of the word Norman first recorded?",
261
+ "answerable": true,
262
+ "abstained": true,
263
+ "cited": "The English name \"Normans\" comes from the French words Normans/Normanz, plural of Normant, modern French normand, which is itself borrowed from Old Low Franconian Nortmann \"Northman\" or directly from ",
264
+ "gold_answer": "9th century"
265
+ },
266
+ {
267
+ "question": "When was the Duchy of Normandy founded?",
268
+ "answerable": true,
269
+ "abstained": true,
270
+ "cited": "The area corresponded to the northern part of present-day Upper Normandy down to the river Seine, but the Duchy would eventually extend west beyond the Seine.",
271
+ "gold_answer": "911"
272
+ },
273
+ {
274
+ "question": "What river originally bounded the Duchy",
275
+ "answerable": true,
276
+ "abstained": true,
277
+ "cited": "The area corresponded to the northern part of present-day Upper Normandy down to the river Seine, but the Duchy would eventually extend west beyond the Seine.",
278
+ "gold_answer": "Seine"
279
+ },
280
+ {
281
+ "question": "when did Nors encampments ivolve into destructive incursions?",
282
+ "answerable": false,
283
+ "abstained": false,
284
+ "cited": "In the course of the 10th century, the initially destructive incursions of Norse war bands into the rivers of France evolved into more permanent encampments that included local women and personal prop",
285
+ "gold_answer": null
286
+ },
287
+ {
288
+ "question": "Who established a treaty with King Charles the third of France?",
289
+ "answerable": false,
290
+ "abstained": false,
291
+ "cited": "The Duchy of Normandy, which began in 911 as a fiefdom, was established by the treaty of Saint-Clair-sur-Epte between King Charles III of West Francia and the famed Viking ruler Rollo, and was situate",
292
+ "gold_answer": null
293
+ },
294
+ {
295
+ "question": "What did the French promises to protect Rollo and his men from?",
296
+ "answerable": false,
297
+ "abstained": false,
298
+ "cited": "The treaty offered Rollo and his men the French lands between the river Epte and the Atlantic coast in exchange for their protection against further Viking incursions.",
299
+ "gold_answer": null
300
+ },
301
+ {
302
+ "question": "Who upon arriving gave the original viking settlers a common identity?",
303
+ "answerable": true,
304
+ "abstained": true,
305
+ "cited": "Earlier Viking settlers had begun arriving in the 880s, but were divided between colonies in the east (Roumois and Pays de Caux) around the low Seine valley and in the west in the Cotentin Peninsula, ",
306
+ "gold_answer": "Rollo"
307
+ },
308
+ {
309
+ "question": "What was the Norman religion?",
310
+ "answerable": true,
311
+ "abstained": true,
312
+ "cited": "The descendants of Rollo's Vikings and their Frankish wives would replace the Norse religion and Old Norse language with Catholicism (Christianity) and the Gallo-Romance language of the local people, ",
313
+ "gold_answer": "Catholicism"
314
+ },
315
+ {
316
+ "question": "What part of France were the Normans located?",
317
+ "answerable": true,
318
+ "abstained": true,
319
+ "cited": "The descendants of Rollo's Vikings and their Frankish wives would replace the Norse religion and Old Norse language with Catholicism (Christianity) and the Gallo-Romance language of the local people, ",
320
+ "gold_answer": "north"
321
+ },
322
+ {
323
+ "question": "What was one of the Norman's major exports?",
324
+ "answerable": true,
325
+ "abstained": true,
326
+ "cited": "Most Norman knights remained poor and land-hungry, and by 1066 Normandy had been exporting fighting horsemen for more than a generation.",
327
+ "gold_answer": "fighting horsemen"
328
+ },
329
+ {
330
+ "question": "Who's arristocracy eventually served as avid Crusaders?",
331
+ "answerable": false,
332
+ "abstained": false,
333
+ "cited": "Many Normans of Italy, France and England eventually served as avid Crusaders under the Italo-Norman prince Bohemund I and the Anglo-Norman king Richard the Lion-Heart.",
334
+ "gold_answer": null
335
+ },
336
+ {
337
+ "question": "Who entered Italy soon after the Byzantine Empire?",
338
+ "answerable": false,
339
+ "abstained": false,
340
+ "cited": "Soon after the Normans began to enter Italy, they entered the Byzantine Empire and then Armenia, fighting against the Pechenegs, the Bulgars, and especially the Seljuk Turks.",
341
+ "gold_answer": null
342
+ }
343
+ ],
344
+ "learned": [
345
+ {
346
+ "question": "In what country is Normandy located?",
347
+ "answerable": true,
348
+ "abstained": true,
349
+ "cited": "The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France.",
350
+ "gold_answer": "France"
351
+ },
352
+ {
353
+ "question": "From which countries did the Norse originate?",
354
+ "answerable": true,
355
+ "abstained": true,
356
+ "cited": "They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West F",
357
+ "gold_answer": "Denmark, Iceland and Norway"
358
+ },
359
+ {
360
+ "question": "What is France a region of?",
361
+ "answerable": false,
362
+ "abstained": false,
363
+ "cited": "The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France.",
364
+ "gold_answer": null
365
+ },
366
+ {
367
+ "question": "Who did King Charles III swear fealty to?",
368
+ "answerable": false,
369
+ "abstained": false,
370
+ "cited": "They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West F",
371
+ "gold_answer": null
372
+ },
373
+ {
374
+ "question": "When did the Frankish identity emerge?",
375
+ "answerable": false,
376
+ "abstained": false,
377
+ "cited": "The distinct cultural and ethnic identity of the Normans emerged initially in the first half of the 10th century, and it continued to evolve over the succeeding centuries.",
378
+ "gold_answer": null
379
+ },
380
+ {
381
+ "question": "Who ruled the duchy of Normandy",
382
+ "answerable": true,
383
+ "abstained": true,
384
+ "cited": "The Duchy of Normandy, which they formed by treaty with the French crown, was a great fief of medieval France, and under Richard I of Normandy was forged into a cohesive and formidable principality in",
385
+ "gold_answer": "Richard I"
386
+ },
387
+ {
388
+ "question": "What religion were the Normans",
389
+ "answerable": true,
390
+ "abstained": true,
391
+ "cited": "The Norman dynasty had a major political, cultural and military impact on medieval Europe and even the Near East.",
392
+ "gold_answer": "Catholic"
393
+ },
394
+ {
395
+ "question": "What type of major impact did the Norman dynasty have on modern Europe?",
396
+ "answerable": false,
397
+ "abstained": false,
398
+ "cited": "The Norman dynasty had a major political, cultural and military impact on medieval Europe and even the Near East.",
399
+ "gold_answer": null
400
+ },
401
+ {
402
+ "question": "Who was famed for their Christian spirit?",
403
+ "answerable": false,
404
+ "abstained": false,
405
+ "cited": "The Normans were famed for their martial spirit and eventually for their Christian piety, becoming exponents of the Catholic orthodoxy into which they assimilated.",
406
+ "gold_answer": null
407
+ },
408
+ {
409
+ "question": "Who assimilted the Roman language?",
410
+ "answerable": false,
411
+ "abstained": false,
412
+ "cited": "They adopted the Gallo-Romance language of the Frankish land they settled, their dialect becoming known as Norman, Normaund or Norman French, an important literary language.",
413
+ "gold_answer": null
414
+ },
415
+ {
416
+ "question": "What principality did William the conquerer found?",
417
+ "answerable": false,
418
+ "abstained": false,
419
+ "cited": "Norman adventurers founded the Kingdom of Sicily under Roger II after conquering southern Italy on the Saracens and Byzantines, and an expedition on behalf of their duke, William the Conqueror, led to",
420
+ "gold_answer": null
421
+ },
422
+ {
423
+ "question": "What is the original meaning of the word Norman?",
424
+ "answerable": true,
425
+ "abstained": true,
426
+ "cited": "The English name \"Normans\" comes from the French words Normans/Normanz, plural of Normant, modern French normand, which is itself borrowed from Old Low Franconian Nortmann \"Northman\" or directly from ",
427
+ "gold_answer": "Viking"
428
+ },
429
+ {
430
+ "question": "What name comes from the English words Normans/Normanz?",
431
+ "answerable": false,
432
+ "abstained": false,
433
+ "cited": "The English name \"Normans\" comes from the French words Normans/Normanz, plural of Normant, modern French normand, which is itself borrowed from Old Low Franconian Nortmann \"Northman\" or directly from ",
434
+ "gold_answer": null
435
+ },
436
+ {
437
+ "question": "When was the French version of the word Norman first recorded?",
438
+ "answerable": false,
439
+ "abstained": false,
440
+ "cited": "The English name \"Normans\" comes from the French words Normans/Normanz, plural of Normant, modern French normand, which is itself borrowed from Old Low Franconian Nortmann \"Northman\" or directly from ",
441
+ "gold_answer": null
442
+ },
443
+ {
444
+ "question": "What river originally bounded the Duchy",
445
+ "answerable": true,
446
+ "abstained": true,
447
+ "cited": "The area corresponded to the northern part of present-day Upper Normandy down to the river Seine, but the Duchy would eventually extend west beyond the Seine.",
448
+ "gold_answer": "Seine"
449
+ },
450
+ {
451
+ "question": "when did Nors encampments ivolve into destructive incursions?",
452
+ "answerable": false,
453
+ "abstained": false,
454
+ "cited": "In the course of the 10th century, the initially destructive incursions of Norse war bands into the rivers of France evolved into more permanent encampments that included local women and personal prop",
455
+ "gold_answer": null
456
+ },
457
+ {
458
+ "question": "What treaty was established in the 9th century?",
459
+ "answerable": false,
460
+ "abstained": false,
461
+ "cited": "The Duchy of Normandy, which began in 911 as a fiefdom, was established by the treaty of Saint-Clair-sur-Epte between King Charles III of West Francia and the famed Viking ruler Rollo, and was situate",
462
+ "gold_answer": null
463
+ },
464
+ {
465
+ "question": "Who established a treaty with King Charles the third of France?",
466
+ "answerable": false,
467
+ "abstained": false,
468
+ "cited": "The Duchy of Normandy, which began in 911 as a fiefdom, was established by the treaty of Saint-Clair-sur-Epte between King Charles III of West Francia and the famed Viking ruler Rollo, and was situate",
469
+ "gold_answer": null
470
+ },
471
+ {
472
+ "question": "What did the French promises to protect Rollo and his men from?",
473
+ "answerable": false,
474
+ "abstained": false,
475
+ "cited": "The treaty offered Rollo and his men the French lands between the river Epte and the Atlantic coast in exchange for their protection against further Viking incursions.",
476
+ "gold_answer": null
477
+ },
478
+ {
479
+ "question": "Who upon arriving gave the original viking settlers a common identity?",
480
+ "answerable": true,
481
+ "abstained": true,
482
+ "cited": "Earlier Viking settlers had begun arriving in the 880s, but were divided between colonies in the east (Roumois and Pays de Caux) around the low Seine valley and in the west in the Cotentin Peninsula, ",
483
+ "gold_answer": "Rollo"
484
+ },
485
+ {
486
+ "question": "What part of France were the Normans located?",
487
+ "answerable": true,
488
+ "abstained": true,
489
+ "cited": "The descendants of Rollo's Vikings and their Frankish wives would replace the Norse religion and Old Norse language with Catholicism (Christianity) and the Gallo-Romance language of the local people, ",
490
+ "gold_answer": "north"
491
+ },
492
+ {
493
+ "question": "What was replace with the Norse religion?",
494
+ "answerable": false,
495
+ "abstained": false,
496
+ "cited": "The descendants of Rollo's Vikings and their Frankish wives would replace the Norse religion and Old Norse language with Catholicism (Christianity) and the Gallo-Romance language of the local people, ",
497
+ "gold_answer": null
498
+ },
499
+ {
500
+ "question": "What did maternal Old Norse traditions merge with?",
501
+ "answerable": false,
502
+ "abstained": false,
503
+ "cited": "The descendants of Rollo's Vikings and their Frankish wives would replace the Norse religion and Old Norse language with Catholicism (Christianity) and the Gallo-Romance language of the local people, ",
504
+ "gold_answer": null
505
+ },
506
+ {
507
+ "question": "What language replaced the Gallo-Romance language?",
508
+ "answerable": false,
509
+ "abstained": false,
510
+ "cited": "The descendants of Rollo's Vikings and their Frankish wives would replace the Norse religion and Old Norse language with Catholicism (Christianity) and the Gallo-Romance language of the local people, ",
511
+ "gold_answer": null
512
+ },
513
+ {
514
+ "question": "What was one of the Norman's major exports?",
515
+ "answerable": true,
516
+ "abstained": true,
517
+ "cited": "Most Norman knights remained poor and land-hungry, and by 1066 Normandy had been exporting fighting horsemen for more than a generation.",
518
+ "gold_answer": "fighting horsemen"
519
+ }
520
+ ]
521
+ },
522
+ "XQuAD-tr (TR, zero-shot)": {
523
+ "bm25": [
524
+ {
525
+ "question": "Luke Kuechly kaç tane top çalma kaydetmiştir?",
526
+ "answerable": true,
527
+ "abstained": false,
528
+ "cited": "Arkalarında, Panthers'lerin üç adet ikinci hat savunma oyuncusundan ikisi de Pro Bowl'da oynamak için seçildi: Thomas Davis ve Luke Kuechly.",
529
+ "gold_answer": "118"
530
+ },
531
+ {
532
+ "question": "Josh Norman kaç tane top çalmıştır?",
533
+ "answerable": true,
534
+ "abstained": false,
535
+ "cited": "Carolina’nın ikincisi olarak gösterilen Pro Bowl güvenliği için kariyerinin en yükseği olarak 7 top kapma ile takımı sürükleyen Kurt Coleman’a ve 88 top çalma sayısıyla ve Pro Bowl köşe savunmacısı se",
536
+ "gold_answer": "dört"
537
+ },
538
+ {
539
+ "question": "Bu sezon takımdaki en fazla sack etmeyi kim kaydetmiştir?",
540
+ "answerable": true,
541
+ "abstained": false,
542
+ "cited": "Davis, dördünde mecburi oyuncuyu düşürme ve dört topu kapma olmak üzere 5½ sack etmeyi bir araya getirirken, Kuechly top çalmalarda (118) iki mecburi oyuncuyu düşürme ve kendi başına dört pasını kesme",
543
+ "gold_answer": "Kawann Short"
544
+ },
545
+ {
546
+ "question": "Panthers savunması 2015 yılında kaç top çalma ile kayda geçmiştir?",
547
+ "answerable": true,
548
+ "abstained": true,
549
+ "cited": "Carolina’nın ikincisi olarak gösterilen Pro Bowl güvenliği için kariyerinin en yükseği olarak 7 top kapma ile takımı sürükleyen Kurt Coleman’a ve 88 top çalma sayısıyla ve Pro Bowl köşe savunmacısı se",
550
+ "gold_answer": "24"
551
+ },
552
+ {
553
+ "question": "Panthers'lere sack etmede kim önderlik etmektedir?",
554
+ "answerable": true,
555
+ "abstained": true,
556
+ "cited": "Panthers hattında ayrıca, sadece 9 başlangıçta 5 sack eden uç çizgi savunmacısı Kony Ealy ile birlikte 136 kez ile NFL'nin aktif kariyer sack lideri ve 5 kez profesyonel bir top atıcısı olan Jared All",
557
+ "gold_answer": "Kawann Short"
558
+ },
559
+ {
560
+ "question": "Pro Bowl için kaç tane Panthers savunma oyuncusu seçilmiştir?",
561
+ "answerable": true,
562
+ "abstained": false,
563
+ "cited": "Arkalarında, Panthers'lerin üç adet ikinci hat savunma oyuncusundan ikisi de Pro Bowl'da oynamak için seçildi: Thomas Davis ve Luke Kuechly.",
564
+ "gold_answer": "dört"
565
+ },
566
+ {
567
+ "question": "Thomas Davis kaç mecburi oyuncuyu düşürme almıştır?",
568
+ "answerable": true,
569
+ "abstained": false,
570
+ "cited": "Davis, dördünde mecburi oyuncuyu düşürme ve dört topu kapma olmak üzere 5½ sack etmeyi bir araya getirirken, Kuechly top çalmalarda (118) iki mecburi oyuncuyu düşürme ve kendi başına dört pasını kesme",
571
+ "gold_answer": "dört"
572
+ },
573
+ {
574
+ "question": "Panthers'lerin savunması 2015 sezonu top kapmalarının kaç tanesini almıştır?",
575
+ "answerable": true,
576
+ "abstained": false,
577
+ "cited": "Carolina’nın ikincisi olarak gösterilen Pro Bowl güvenliği için kariyerinin en yükseği olarak 7 top kapma ile takımı sürükleyen Kurt Coleman’a ve 88 top çalma sayısıyla ve Pro Bowl köşe savunmacısı se",
578
+ "gold_answer": "24"
579
+ },
580
+ {
581
+ "question": "Josh Norman 2015 yılında kale çizgisini geçişle sonlanan kaç topu çalma yapmıştır?",
582
+ "answerable": true,
583
+ "abstained": false,
584
+ "cited": "Carolina’nın ikincisi olarak gösterilen Pro Bowl güvenliği için kariyerinin en yükseği olarak 7 top kapma ile takımı sürükleyen Kurt Coleman’a ve 88 top çalma sayısıyla ve Pro Bowl köşe savunmacısı se",
585
+ "gold_answer": "iki"
586
+ },
587
+ {
588
+ "question": "Super Bowl XLIX'i kim kazanmıştır?",
589
+ "answerable": true,
590
+ "abstained": true,
591
+ "cited": "Daha sonra, New England'ın 2 puanlık dönüşüm girişimini saat üzerinde 17 saniye kala bir pası keserek, AFC Şampiyonluk Oyununda Super Bowl'u savunan XLIX şampiyonu New England Patriots'u 20–18 yendi.",
592
+ "gold_answer": "New England Patriots"
593
+ },
594
+ {
595
+ "question": "Super Bowl XLIX'i kim kazanmıştır?",
596
+ "answerable": true,
597
+ "abstained": true,
598
+ "cited": "Daha sonra, New England'ın 2 puanlık dönüşüm girişimini saat üzerinde 17 saniye kala bir pası keserek, AFC Şampiyonluk Oyununda Super Bowl'u savunan XLIX şampiyonu New England Patriots'u 20–18 yendi.",
599
+ "gold_answer": "New England Patriots"
600
+ },
601
+ {
602
+ "question": "Broncos oyunu kazandıran pası kestiği zaman oyunun bitimine kaç saniye kalmıştı?",
603
+ "answerable": true,
604
+ "abstained": true,
605
+ "cited": "Daha sonra, New England'ın 2 puanlık dönüşüm girişimini saat üzerinde 17 saniye kala bir pası keserek, AFC Şampiyonluk Oyununda Super Bowl'u savunan XLIX şampiyonu New England Patriots'u 20–18 yendi.",
606
+ "gold_answer": "17"
607
+ },
608
+ {
609
+ "question": "Broncos'un rövanş oyunları süresince kim hiç atmamıştır?",
610
+ "answerable": true,
611
+ "abstained": true,
612
+ "cited": "Manning''in sezon boyunca topu kapmalarla ilgili sorunlarına rağmen, iki rövanş maçında hiç atmadı.",
613
+ "gold_answer": "Manning"
614
+ },
615
+ {
616
+ "question": "2015 yılında klasmanlarını kazanmak için Broncos'u kim yenmiştir?",
617
+ "answerable": true,
618
+ "abstained": true,
619
+ "cited": "Daha sonra, New England'ın 2 puanlık dönüşüm girişimini saat üzerinde 17 saniye kala bir pası keserek, AFC Şampiyonluk Oyununda Super Bowl'u savunan XLIX şampiyonu New England Patriots'u 20–18 yendi.",
620
+ "gold_answer": "Pittsburgh Steelers"
621
+ },
622
+ {
623
+ "question": "Peyton Manning, Super Bowl %0'de oynadığı zaman kaç yaşındaydı?",
624
+ "answerable": true,
625
+ "abstained": true,
626
+ "cited": "Peyton Manning,iki farklı takımı birden fazla Super Bowls'a gitmesinde önderlik eden ilk kilit oyuncu oldu.",
627
+ "gold_answer": "39"
628
+ },
629
+ {
630
+ "question": "Bir Super Bowl'da oynayan en yaşlı kilit oyuncu olma rekorunu daha önce kim elinde bulunduruyordu?",
631
+ "answerable": true,
632
+ "abstained": true,
633
+ "cited": "Peyton Manning,iki farklı takımı birden fazla Super Bowls'a gitmesinde önderlik eden ilk kilit oyuncu oldu.",
634
+ "gold_answer": "John Elway"
635
+ },
636
+ {
637
+ "question": "John Elway'in şu anki Broncos üyeliğindeki rolü nedir?",
638
+ "answerable": true,
639
+ "abstained": true,
640
+ "cited": "Geçmiş rekor, Broncos'u 38 yaşında Super Bowl XXXIII'te galibiyete götüren ve şu anda Denver'ın Futbol Operasyonları Başkan Yardımcısı ve Genel Müdür'ü olan John Elway'in ellerindeydi.",
641
+ "gold_answer": "Futbol Operasyonları Başkan Yardımcısı ve Genel Müdür'ü"
642
+ },
643
+ {
644
+ "question": "Super Bowl XXXIII'ün kazananı hangi takımdı?",
645
+ "answerable": true,
646
+ "abstained": true,
647
+ "cited": "Geçmiş rekor, Broncos'u 38 yaşında Super Bowl XXXIII'te galibiyete götüren ve şu anda Denver'ın Futbol Operasyonları Başkan Yardımcısı ve Genel Müdür'ü olan John Elway'in ellerindeydi.",
648
+ "gold_answer": "Broncos"
649
+ },
650
+ {
651
+ "question": "Broncos'un Genel Müdürü kimdir?",
652
+ "answerable": true,
653
+ "abstained": true,
654
+ "cited": "Geçmiş rekor, Broncos'u 38 yaşında Super Bowl XXXIII'te galibiyete götüren ve şu anda Denver'ın Futbol Operasyonları Başkan Yardımcısı ve Genel Müdür'ü olan John Elway'in ellerindeydi.",
655
+ "gold_answer": "John Elway"
656
+ },
657
+ {
658
+ "question": "Herhangi bir Super Bowl oyununda en yaşlı kilit oyuncu rekoruna kim sahiptir?",
659
+ "answerable": true,
660
+ "abstained": true,
661
+ "cited": "Peyton Manning,iki farklı takımı birden fazla Super Bowls'a gitmesinde önderlik eden ilk kilit oyuncu oldu.",
662
+ "gold_answer": "Peyton Manning"
663
+ },
664
+ {
665
+ "question": "Manning Super Bowl'a ulaşmak için onların takımındayken kaç takımda oynamıştır?",
666
+ "answerable": true,
667
+ "abstained": true,
668
+ "cited": "Peyton Manning,iki farklı takımı birden fazla Super Bowls'a gitmesinde önderlik eden ilk kilit oyuncu oldu.",
669
+ "gold_answer": "iki"
670
+ },
671
+ {
672
+ "question": "Bir Super Bowl'da oynayan en yaşlı kilit oyuncu kimdir?",
673
+ "answerable": true,
674
+ "abstained": true,
675
+ "cited": "Peyton Manning,iki farklı takımı birden fazla Super Bowls'a gitmesinde önderlik eden ilk kilit oyuncu oldu.",
676
+ "gold_answer": "Peyton Manning"
677
+ },
678
+ {
679
+ "question": "Manning'den önce bir Super Bowl'da oynayan en yaşlı kilit oyuncu kimdir?",
680
+ "answerable": true,
681
+ "abstained": false,
682
+ "cited": "Peyton Manning,iki farklı takımı birden fazla Super Bowls'a gitmesinde önderlik eden ilk kilit oyuncu oldu.",
683
+ "gold_answer": "John Elway"
684
+ },
685
+ {
686
+ "question": "Manning Super Bowl 50'de oynadığı zaman kaç yaşındaydı?",
687
+ "answerable": true,
688
+ "abstained": true,
689
+ "cited": "Ayrıca, 39 yaşında bir Super Bowl'da oynayan en eski oyun kurucu oldu.",
690
+ "gold_answer": "39"
691
+ },
692
+ {
693
+ "question": "Lady Gaga kaç Grammy kazanmıştır?",
694
+ "answerable": true,
695
+ "abstained": true,
696
+ "cited": "Akademi Ödülü kazananı Marlee Matlin Amerikan İşaret Dili(ASL) çevirisini yaparken altı kez Grammy kazanan ve Akademi Ödülü adayı Lady Gaga ulusal marşı söylemiştir. ",
697
+ "gold_answer": "altı"
698
+ }
699
+ ],
700
+ "learned": [
701
+ {
702
+ "question": "Luke Kuechly kaç tane top çalma kaydetmiştir?",
703
+ "answerable": true,
704
+ "abstained": true,
705
+ "cited": "Davis, dördünde mecburi oyuncuyu düşürme ve dört topu kapma olmak üzere 5½ sack etmeyi bir araya getirirken, Kuechly top çalmalarda (118) iki mecburi oyuncuyu düşürme ve kendi başına dört pasını kesme",
706
+ "gold_answer": "118"
707
+ },
708
+ {
709
+ "question": "Josh Norman kaç tane top çalmıştır?",
710
+ "answerable": true,
711
+ "abstained": false,
712
+ "cited": "Carolina’nın ikincisi olarak gösterilen Pro Bowl güvenliği için kariyerinin en yükseği olarak 7 top kapma ile takımı sürükleyen Kurt Coleman’a ve 88 top çalma sayısıyla ve Pro Bowl köşe savunmacısı se",
713
+ "gold_answer": "dört"
714
+ },
715
+ {
716
+ "question": "Bu sezon takımdaki en fazla sack etmeyi kim kaydetmiştir?",
717
+ "answerable": true,
718
+ "abstained": false,
719
+ "cited": "Davis, dördünde mecburi oyuncuyu düşürme ve dört topu kapma olmak üzere 5½ sack etmeyi bir araya getirirken, Kuechly top çalmalarda (118) iki mecburi oyuncuyu düşürme ve kendi başına dört pasını kesme",
720
+ "gold_answer": "Kawann Short"
721
+ },
722
+ {
723
+ "question": "Panthers savunması 2015 yılında kaç top çalma ile kayda geçmiştir?",
724
+ "answerable": true,
725
+ "abstained": true,
726
+ "cited": "Carolina’nın ikincisi olarak gösterilen Pro Bowl güvenliği için kariyerinin en yükseği olarak 7 top kapma ile takımı sürükleyen Kurt Coleman’a ve 88 top çalma sayısıyla ve Pro Bowl köşe savunmacısı se",
727
+ "gold_answer": "24"
728
+ },
729
+ {
730
+ "question": "Panthers'lere sack etmede kim önderlik etmektedir?",
731
+ "answerable": true,
732
+ "abstained": true,
733
+ "cited": "Panthers hattında ayrıca, sadece 9 başlangıçta 5 sack eden uç çizgi savunmacısı Kony Ealy ile birlikte 136 kez ile NFL'nin aktif kariyer sack lideri ve 5 kez profesyonel bir top atıcısı olan Jared All",
734
+ "gold_answer": "Kawann Short"
735
+ },
736
+ {
737
+ "question": "Pro Bowl için kaç tane Panthers savunma oyuncusu seçilmiştir?",
738
+ "answerable": true,
739
+ "abstained": false,
740
+ "cited": "Arkalarında, Panthers'lerin üç adet ikinci hat savunma oyuncusundan ikisi de Pro Bowl'da oynamak için seçildi: Thomas Davis ve Luke Kuechly.",
741
+ "gold_answer": "dört"
742
+ },
743
+ {
744
+ "question": "Thomas Davis kaç mecburi oyuncuyu düşürme almıştır?",
745
+ "answerable": true,
746
+ "abstained": false,
747
+ "cited": "Davis, dördünde mecburi oyuncuyu düşürme ve dört topu kapma olmak üzere 5½ sack etmeyi bir araya getirirken, Kuechly top çalmalarda (118) iki mecburi oyuncuyu düşürme ve kendi başına dört pasını kesme",
748
+ "gold_answer": "dört"
749
+ },
750
+ {
751
+ "question": "Panthers'lerin savunması 2015 sezonu top kapmalarının kaç tanesini almıştır?",
752
+ "answerable": true,
753
+ "abstained": true,
754
+ "cited": "Carolina’nın ikincisi olarak gösterilen Pro Bowl güvenliği için kariyerinin en yükseği olarak 7 top kapma ile takımı sürükleyen Kurt Coleman’a ve 88 top çalma sayısıyla ve Pro Bowl köşe savunmacısı se",
755
+ "gold_answer": "24"
756
+ },
757
+ {
758
+ "question": "2015 yılında Panthers'lerin ofansif orta çizgi oyuncusu lideri kimdi?",
759
+ "answerable": true,
760
+ "abstained": true,
761
+ "cited": "Panthers hattında ayrıca, sadece 9 başlangıçta 5 sack eden uç çizgi savunmacısı Kony Ealy ile birlikte 136 kez ile NFL'nin aktif kariyer sack lideri ve 5 kez profesyonel bir top atıcısı olan Jared All",
762
+ "gold_answer": "Luke Kuechly."
763
+ },
764
+ {
765
+ "question": "Josh Norman 2015 yılında kale çizgisini geçişle sonlanan kaç topu çalma yapmıştır?",
766
+ "answerable": true,
767
+ "abstained": true,
768
+ "cited": "Carolina’nın ikincisi olarak gösterilen Pro Bowl güvenliği için kariyerinin en yükseği olarak 7 top kapma ile takımı sürükleyen Kurt Coleman’a ve 88 top çalma sayısıyla ve Pro Bowl köşe savunmacısı se",
769
+ "gold_answer": "iki"
770
+ },
771
+ {
772
+ "question": "Super Bowl XLIX'i kim kazanmıştır?",
773
+ "answerable": true,
774
+ "abstained": true,
775
+ "cited": "Daha sonra, New England'ın 2 puanlık dönüşüm girişimini saat üzerinde 17 saniye kala bir pası keserek, AFC Şampiyonluk Oyununda Super Bowl'u savunan XLIX şampiyonu New England Patriots'u 20–18 yendi.",
776
+ "gold_answer": "New England Patriots"
777
+ },
778
+ {
779
+ "question": "AFC Şampiyonluk Oyunu final skoru neydi?",
780
+ "answerable": true,
781
+ "abstained": true,
782
+ "cited": "Daha sonra, New England'ın 2 puanlık dönüşüm girişimini saat üzerinde 17 saniye kala bir pası keserek, AFC Şampiyonluk Oyununda Super Bowl'u savunan XLIX şampiyonu New England Patriots'u 20–18 yendi.",
783
+ "gold_answer": "20–18"
784
+ },
785
+ {
786
+ "question": "Super Bowl XLIX'i kim kazanmıştır?",
787
+ "answerable": true,
788
+ "abstained": true,
789
+ "cited": "Daha sonra, New England'ın 2 puanlık dönüşüm girişimini saat üzerinde 17 saniye kala bir pası keserek, AFC Şampiyonluk Oyununda Super Bowl'u savunan XLIX şampiyonu New England Patriots'u 20–18 yendi.",
790
+ "gold_answer": "New England Patriots"
791
+ },
792
+ {
793
+ "question": "Broncos'un rövanş oyunları süresince kim hiç atmamıştır?",
794
+ "answerable": true,
795
+ "abstained": true,
796
+ "cited": "Manning''in sezon boyunca topu kapmalarla ilgili sorunlarına rağmen, iki rövanş maçında hiç atmadı.",
797
+ "gold_answer": "Manning"
798
+ },
799
+ {
800
+ "question": "2015 yılında klasmanlarını kazanmak için Broncos'u kim yenmiştir?",
801
+ "answerable": true,
802
+ "abstained": true,
803
+ "cited": "Daha sonra, New England'ın 2 puanlık dönüşüm girişimini saat üzerinde 17 saniye kala bir pası keserek, AFC Şampiyonluk Oyununda Super Bowl'u savunan XLIX şampiyonu New England Patriots'u 20–18 yendi.",
804
+ "gold_answer": "Pittsburgh Steelers"
805
+ },
806
+ {
807
+ "question": "Peyton Manning, Super Bowl %0'de oynadığı zaman kaç yaşındaydı?",
808
+ "answerable": true,
809
+ "abstained": true,
810
+ "cited": "Peyton Manning,iki farklı takımı birden fazla Super Bowls'a gitmesinde önderlik eden ilk kilit oyuncu oldu.",
811
+ "gold_answer": "39"
812
+ },
813
+ {
814
+ "question": "Bir Super Bowl'da oynayan en yaşlı kilit oyuncu olma rekorunu daha önce kim elinde bulunduruyordu?",
815
+ "answerable": true,
816
+ "abstained": false,
817
+ "cited": "Ayrıca, 39 yaşında bir Super Bowl'da oynayan en eski oyun kurucu oldu.",
818
+ "gold_answer": "John Elway"
819
+ },
820
+ {
821
+ "question": "John Elway'in şu anki Broncos üyeliğindeki rolü nedir?",
822
+ "answerable": true,
823
+ "abstained": true,
824
+ "cited": "Geçmiş rekor, Broncos'u 38 yaşında Super Bowl XXXIII'te galibiyete götüren ve şu anda Denver'ın Futbol Operasyonları Başkan Yardımcısı ve Genel Müdür'ü olan John Elway'in ellerindeydi.",
825
+ "gold_answer": "Futbol Operasyonları Başkan Yardımcısı ve Genel Müdür'ü"
826
+ },
827
+ {
828
+ "question": "Broncos'un Genel Müdürü kimdir?",
829
+ "answerable": true,
830
+ "abstained": true,
831
+ "cited": "Geçmiş rekor, Broncos'u 38 yaşında Super Bowl XXXIII'te galibiyete götüren ve şu anda Denver'ın Futbol Operasyonları Başkan Yardımcısı ve Genel Müdür'ü olan John Elway'in ellerindeydi.",
832
+ "gold_answer": "John Elway"
833
+ },
834
+ {
835
+ "question": "Herhangi bir Super Bowl oyununda en yaşlı kilit oyuncu rekoruna kim sahiptir?",
836
+ "answerable": true,
837
+ "abstained": true,
838
+ "cited": "Peyton Manning,iki farklı takımı birden fazla Super Bowls'a gitmesinde önderlik eden ilk kilit oyuncu oldu.",
839
+ "gold_answer": "Peyton Manning"
840
+ },
841
+ {
842
+ "question": "Manning Super Bowl'a ulaşmak için onların takımındayken kaç takımda oynamıştır?",
843
+ "answerable": true,
844
+ "abstained": true,
845
+ "cited": "Peyton Manning,iki farklı takımı birden fazla Super Bowls'a gitmesinde önderlik eden ilk kilit oyuncu oldu.",
846
+ "gold_answer": "iki"
847
+ },
848
+ {
849
+ "question": "Peyton Manning Super Bowl'a kaç farklı takımı götürmüştür?",
850
+ "answerable": true,
851
+ "abstained": true,
852
+ "cited": "Peyton Manning,iki farklı takımı birden fazla Super Bowls'a gitmesinde önderlik eden ilk kilit oyuncu oldu.",
853
+ "gold_answer": "iki"
854
+ },
855
+ {
856
+ "question": "Bir Super Bowl'da oynayan en yaşlı kilit oyuncu kimdir?",
857
+ "answerable": true,
858
+ "abstained": false,
859
+ "cited": "Ayrıca, 39 yaşında bir Super Bowl'da oynayan en eski oyun kurucu oldu.",
860
+ "gold_answer": "Peyton Manning"
861
+ },
862
+ {
863
+ "question": "Manning'den önce bir Super Bowl'da oynayan en yaşlı kilit oyuncu kimdir?",
864
+ "answerable": true,
865
+ "abstained": false,
866
+ "cited": "Peyton Manning,iki farklı takımı birden fazla Super Bowls'a gitmesinde önderlik eden ilk kilit oyuncu oldu.",
867
+ "gold_answer": "John Elway"
868
+ },
869
+ {
870
+ "question": "Manning Super Bowl 50'de oynadığı zaman kaç yaşındaydı?",
871
+ "answerable": true,
872
+ "abstained": true,
873
+ "cited": "Ayrıca, 39 yaşında bir Super Bowl'da oynayan en eski oyun kurucu oldu.",
874
+ "gold_answer": "39"
875
+ }
876
+ ]
877
+ },
878
+ "XQuAD-en (EN)": {
879
+ "bm25": [
880
+ {
881
+ "question": "How many tackles did Luke Kuechly register?",
882
+ "answerable": true,
883
+ "abstained": false,
884
+ "cited": "Behind them, two of the Panthers three starting linebackers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly.",
885
+ "gold_answer": "118"
886
+ },
887
+ {
888
+ "question": "How many balls did Josh Norman intercept?",
889
+ "answerable": true,
890
+ "abstained": false,
891
+ "cited": "Carolina's secondary featured Pro Bowl safety Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who developed",
892
+ "gold_answer": "four"
893
+ },
894
+ {
895
+ "question": "Who registered the most sacks on the team this season?",
896
+ "answerable": true,
897
+ "abstained": true,
898
+ "cited": "Carolina's secondary featured Pro Bowl safety Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who developed",
899
+ "gold_answer": "Kawann Short"
900
+ },
901
+ {
902
+ "question": "How many interceptions are the Panthers defense credited with in 2015?",
903
+ "answerable": true,
904
+ "abstained": true,
905
+ "cited": "The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro Bowl selections.",
906
+ "gold_answer": "24"
907
+ },
908
+ {
909
+ "question": "Who led the Panthers in sacks?",
910
+ "answerable": true,
911
+ "abstained": true,
912
+ "cited": "Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two.",
913
+ "gold_answer": "Kawann Short"
914
+ },
915
+ {
916
+ "question": "How many forced fumbles did Thomas Davis have?",
917
+ "answerable": true,
918
+ "abstained": false,
919
+ "cited": "Davis compiled 5½ sacks, four forced fumbles, and four interceptions, while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own.",
920
+ "gold_answer": "four"
921
+ },
922
+ {
923
+ "question": "Which player had the most interceptions for the season?",
924
+ "answerable": true,
925
+ "abstained": true,
926
+ "cited": "Carolina's secondary featured Pro Bowl safety Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who developed",
927
+ "gold_answer": "Kurt Coleman"
928
+ },
929
+ {
930
+ "question": "How many 2015 season interceptions did the Panthers' defense get?",
931
+ "answerable": true,
932
+ "abstained": true,
933
+ "cited": "The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro Bowl selections.",
934
+ "gold_answer": "24"
935
+ },
936
+ {
937
+ "question": "Who had five sacks in nine games as a Carolina Panthers starter?",
938
+ "answerable": true,
939
+ "abstained": true,
940
+ "cited": "Behind them, two of the Panthers three starting linebackers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly.",
941
+ "gold_answer": "Kony Ealy"
942
+ },
943
+ {
944
+ "question": "Who was the Panthers' tackle leader for 2015?",
945
+ "answerable": true,
946
+ "abstained": true,
947
+ "cited": "The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active career sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just",
948
+ "gold_answer": "Luke Kuechly."
949
+ },
950
+ {
951
+ "question": "How many interceptions did Josh Norman score touchdowns with in 2015?",
952
+ "answerable": true,
953
+ "abstained": false,
954
+ "cited": "Carolina's secondary featured Pro Bowl safety Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who developed",
955
+ "gold_answer": "two."
956
+ },
957
+ {
958
+ "question": "Who won Super Bowl XLIX?",
959
+ "answerable": true,
960
+ "abstained": true,
961
+ "cited": "They then beat the defending Super Bowl XLIX champion New England Patriots in the AFC Championship Game, 20–18, by intercepting a pass on New England's 2-point conversion attempt with 17 seconds left ",
962
+ "gold_answer": "New England Patriots"
963
+ },
964
+ {
965
+ "question": "What was the final score of the AFC Championship Game?",
966
+ "answerable": true,
967
+ "abstained": true,
968
+ "cited": "They then beat the defending Super Bowl XLIX champion New England Patriots in the AFC Championship Game, 20–18, by intercepting a pass on New England's 2-point conversion attempt with 17 seconds left ",
969
+ "gold_answer": "20–18"
970
+ },
971
+ {
972
+ "question": "Who won Super Bowl XLIX?",
973
+ "answerable": true,
974
+ "abstained": true,
975
+ "cited": "They then beat the defending Super Bowl XLIX champion New England Patriots in the AFC Championship Game, 20–18, by intercepting a pass on New England's 2-point conversion attempt with 17 seconds left ",
976
+ "gold_answer": "New England Patriots"
977
+ },
978
+ {
979
+ "question": "Who did the Broncos beat in the divisional game?",
980
+ "answerable": true,
981
+ "abstained": true,
982
+ "cited": "The Broncos defeated the Pittsburgh Steelers in the divisional round, 23–16, by scoring 11 points in the final three minutes of the game.",
983
+ "gold_answer": "Pittsburgh Steelers"
984
+ },
985
+ {
986
+ "question": "Who did the Broncos defeat in the AFC Championship game?",
987
+ "answerable": true,
988
+ "abstained": true,
989
+ "cited": "The Broncos defeated the Pittsburgh Steelers in the divisional round, 23–16, by scoring 11 points in the final three minutes of the game.",
990
+ "gold_answer": "New England Patriots"
991
+ },
992
+ {
993
+ "question": "Who did the Broncos beat to win their division in 2015?",
994
+ "answerable": true,
995
+ "abstained": true,
996
+ "cited": "The Broncos defeated the Pittsburgh Steelers in the divisional round, 23–16, by scoring 11 points in the final three minutes of the game.",
997
+ "gold_answer": "Pittsburgh Steelers"
998
+ },
999
+ {
1000
+ "question": "How old was Peyton Manning when he played in Super Bowl 50?",
1001
+ "answerable": true,
1002
+ "abstained": true,
1003
+ "cited": "Peyton Manning became the first quarterback ever to lead two different teams to multiple Super Bowls.",
1004
+ "gold_answer": "39"
1005
+ },
1006
+ {
1007
+ "question": "Who previously held the record for being the oldest quarterback to play in a Super Bowl?",
1008
+ "answerable": true,
1009
+ "abstained": false,
1010
+ "cited": "He is also the oldest quarterback ever to play in a Super Bowl at age 39.",
1011
+ "gold_answer": "John Elway"
1012
+ },
1013
+ {
1014
+ "question": "What team was the winner of Super Bowl XXXIII?",
1015
+ "answerable": true,
1016
+ "abstained": true,
1017
+ "cited": "The past record was held by John Elway, who led the Broncos to victory in Super Bowl XXXIII at age 38 and is currently Denver's Executive Vice President of Football Operations and General Manager.",
1018
+ "gold_answer": "Broncos"
1019
+ },
1020
+ {
1021
+ "question": "Who is the General Manager for the Broncos?",
1022
+ "answerable": true,
1023
+ "abstained": true,
1024
+ "cited": "The past record was held by John Elway, who led the Broncos to victory in Super Bowl XXXIII at age 38 and is currently Denver's Executive Vice President of Football Operations and General Manager.",
1025
+ "gold_answer": "John Elway"
1026
+ },
1027
+ {
1028
+ "question": "How old was Elway during his Super Bowl XXXIII win?",
1029
+ "answerable": true,
1030
+ "abstained": true,
1031
+ "cited": "The past record was held by John Elway, who led the Broncos to victory in Super Bowl XXXIII at age 38 and is currently Denver's Executive Vice President of Football Operations and General Manager.",
1032
+ "gold_answer": "38"
1033
+ },
1034
+ {
1035
+ "question": "Who has the record of being the oldest quarter back in any Super Bowl game?",
1036
+ "answerable": true,
1037
+ "abstained": true,
1038
+ "cited": "He is also the oldest quarterback ever to play in a Super Bowl at age 39.",
1039
+ "gold_answer": "Peyton Manning"
1040
+ },
1041
+ {
1042
+ "question": "How many teams has Manning played for that reached the Super Bowl, while he was on their team?",
1043
+ "answerable": true,
1044
+ "abstained": true,
1045
+ "cited": "Peyton Manning became the first quarterback ever to lead two different teams to multiple Super Bowls.",
1046
+ "gold_answer": "two"
1047
+ },
1048
+ {
1049
+ "question": "Who is the oldest quarterback to play in a Super Bowl?",
1050
+ "answerable": true,
1051
+ "abstained": false,
1052
+ "cited": "He is also the oldest quarterback ever to play in a Super Bowl at age 39.",
1053
+ "gold_answer": "Peyton Manning"
1054
+ }
1055
+ ],
1056
+ "learned": [
1057
+ {
1058
+ "question": "How many tackles did Luke Kuechly register?",
1059
+ "answerable": true,
1060
+ "abstained": true,
1061
+ "cited": "Behind them, two of the Panthers three starting linebackers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly.",
1062
+ "gold_answer": "118"
1063
+ },
1064
+ {
1065
+ "question": "How many balls did Josh Norman intercept?",
1066
+ "answerable": true,
1067
+ "abstained": false,
1068
+ "cited": "Carolina's secondary featured Pro Bowl safety Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who developed",
1069
+ "gold_answer": "four"
1070
+ },
1071
+ {
1072
+ "question": "Who registered the most sacks on the team this season?",
1073
+ "answerable": true,
1074
+ "abstained": true,
1075
+ "cited": "Carolina's secondary featured Pro Bowl safety Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who developed",
1076
+ "gold_answer": "Kawann Short"
1077
+ },
1078
+ {
1079
+ "question": "How many interceptions are the Panthers defense credited with in 2015?",
1080
+ "answerable": true,
1081
+ "abstained": true,
1082
+ "cited": "The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro Bowl selections.",
1083
+ "gold_answer": "24"
1084
+ },
1085
+ {
1086
+ "question": "Who led the Panthers in sacks?",
1087
+ "answerable": true,
1088
+ "abstained": true,
1089
+ "cited": "Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two.",
1090
+ "gold_answer": "Kawann Short"
1091
+ },
1092
+ {
1093
+ "question": "How many forced fumbles did Thomas Davis have?",
1094
+ "answerable": true,
1095
+ "abstained": false,
1096
+ "cited": "Davis compiled 5½ sacks, four forced fumbles, and four interceptions, while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own.",
1097
+ "gold_answer": "four"
1098
+ },
1099
+ {
1100
+ "question": "How many 2015 season interceptions did the Panthers' defense get?",
1101
+ "answerable": true,
1102
+ "abstained": true,
1103
+ "cited": "Carolina's secondary featured Pro Bowl safety Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who developed",
1104
+ "gold_answer": "24"
1105
+ },
1106
+ {
1107
+ "question": "Who was the Panthers' tackle leader for 2015?",
1108
+ "answerable": true,
1109
+ "abstained": true,
1110
+ "cited": "The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active career sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just",
1111
+ "gold_answer": "Luke Kuechly."
1112
+ },
1113
+ {
1114
+ "question": "How many interceptions did Josh Norman score touchdowns with in 2015?",
1115
+ "answerable": true,
1116
+ "abstained": true,
1117
+ "cited": "Carolina's secondary featured Pro Bowl safety Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who developed",
1118
+ "gold_answer": "two."
1119
+ },
1120
+ {
1121
+ "question": "Who won Super Bowl XLIX?",
1122
+ "answerable": true,
1123
+ "abstained": true,
1124
+ "cited": "They then beat the defending Super Bowl XLIX champion New England Patriots in the AFC Championship Game, 20–18, by intercepting a pass on New England's 2-point conversion attempt with 17 seconds left ",
1125
+ "gold_answer": "New England Patriots"
1126
+ },
1127
+ {
1128
+ "question": "Who won Super Bowl XLIX?",
1129
+ "answerable": true,
1130
+ "abstained": true,
1131
+ "cited": "They then beat the defending Super Bowl XLIX champion New England Patriots in the AFC Championship Game, 20–18, by intercepting a pass on New England's 2-point conversion attempt with 17 seconds left ",
1132
+ "gold_answer": "New England Patriots"
1133
+ },
1134
+ {
1135
+ "question": "Who did the Broncos beat to win their division in 2015?",
1136
+ "answerable": true,
1137
+ "abstained": true,
1138
+ "cited": "The Broncos defeated the Pittsburgh Steelers in the divisional round, 23–16, by scoring 11 points in the final three minutes of the game.",
1139
+ "gold_answer": "Pittsburgh Steelers"
1140
+ },
1141
+ {
1142
+ "question": "How old was Peyton Manning when he played in Super Bowl 50?",
1143
+ "answerable": true,
1144
+ "abstained": true,
1145
+ "cited": "Peyton Manning became the first quarterback ever to lead two different teams to multiple Super Bowls.",
1146
+ "gold_answer": "39"
1147
+ },
1148
+ {
1149
+ "question": "Who previously held the record for being the oldest quarterback to play in a Super Bowl?",
1150
+ "answerable": true,
1151
+ "abstained": false,
1152
+ "cited": "He is also the oldest quarterback ever to play in a Super Bowl at age 39.",
1153
+ "gold_answer": "John Elway"
1154
+ },
1155
+ {
1156
+ "question": "Who has the record of being the oldest quarter back in any Super Bowl game?",
1157
+ "answerable": true,
1158
+ "abstained": false,
1159
+ "cited": "He is also the oldest quarterback ever to play in a Super Bowl at age 39.",
1160
+ "gold_answer": "Peyton Manning"
1161
+ },
1162
+ {
1163
+ "question": "How many teams has Manning played for that reached the Super Bowl, while he was on their team?",
1164
+ "answerable": true,
1165
+ "abstained": true,
1166
+ "cited": "Peyton Manning became the first quarterback ever to lead two different teams to multiple Super Bowls.",
1167
+ "gold_answer": "two"
1168
+ },
1169
+ {
1170
+ "question": "Who is the oldest quarterback to play in a Super Bowl?",
1171
+ "answerable": true,
1172
+ "abstained": false,
1173
+ "cited": "He is also the oldest quarterback ever to play in a Super Bowl at age 39.",
1174
+ "gold_answer": "Peyton Manning"
1175
+ },
1176
+ {
1177
+ "question": "Prior to Manning, who was the oldest quarterback to play in a Super Bowl?",
1178
+ "answerable": true,
1179
+ "abstained": false,
1180
+ "cited": "He is also the oldest quarterback ever to play in a Super Bowl at age 39.",
1181
+ "gold_answer": "John Elway"
1182
+ },
1183
+ {
1184
+ "question": "How old was Manning when he played Super Bowl 50?",
1185
+ "answerable": true,
1186
+ "abstained": true,
1187
+ "cited": "Peyton Manning became the first quarterback ever to lead two different teams to multiple Super Bowls.",
1188
+ "gold_answer": "39"
1189
+ },
1190
+ {
1191
+ "question": "How many Grammys has Lady Gaga won?",
1192
+ "answerable": true,
1193
+ "abstained": true,
1194
+ "cited": "Six-time Grammy winner and Academy Award nominee Lady Gaga performed the national anthem, while Academy Award winner Marlee Matlin provided American Sign Language (ASL) translation.",
1195
+ "gold_answer": "Six"
1196
+ },
1197
+ {
1198
+ "question": "What actress did the ASL translation for the game?",
1199
+ "answerable": true,
1200
+ "abstained": true,
1201
+ "cited": "Six-time Grammy winner and Academy Award nominee Lady Gaga performed the national anthem, while Academy Award winner Marlee Matlin provided American Sign Language (ASL) translation.",
1202
+ "gold_answer": "Marlee Matlin"
1203
+ },
1204
+ {
1205
+ "question": "Who did the National Anthem at Super Bowl 50?",
1206
+ "answerable": true,
1207
+ "abstained": true,
1208
+ "cited": "Six-time Grammy winner and Academy Award nominee Lady Gaga performed the national anthem, while Academy Award winner Marlee Matlin provided American Sign Language (ASL) translation.",
1209
+ "gold_answer": "Lady Gaga"
1210
+ },
1211
+ {
1212
+ "question": "What actor did sign language for the National Anthem at Superbowl 50?",
1213
+ "answerable": true,
1214
+ "abstained": true,
1215
+ "cited": "Six-time Grammy winner and Academy Award nominee Lady Gaga performed the national anthem, while Academy Award winner Marlee Matlin provided American Sign Language (ASL) translation.",
1216
+ "gold_answer": "Marlee Matlin"
1217
+ },
1218
+ {
1219
+ "question": "Who did the Super Bowl 50 National Anthem?",
1220
+ "answerable": true,
1221
+ "abstained": true,
1222
+ "cited": "Six-time Grammy winner and Academy Award nominee Lady Gaga performed the national anthem, while Academy Award winner Marlee Matlin provided American Sign Language (ASL) translation.",
1223
+ "gold_answer": "Lady Gaga"
1224
+ },
1225
+ {
1226
+ "question": "How many Grammy Award's does the text say Lady Gaga has won?",
1227
+ "answerable": true,
1228
+ "abstained": true,
1229
+ "cited": "Six-time Grammy winner and Academy Award nominee Lady Gaga performed the national anthem, while Academy Award winner Marlee Matlin provided American Sign Language (ASL) translation.",
1230
+ "gold_answer": "Six"
1231
+ }
1232
+ ]
1233
+ }
1234
+ },
1235
+ "hand_written_examples": [
1236
+ {
1237
+ "title": "TR · kütüphane (yanıtlanabilir)",
1238
+ "abstain": false,
1239
+ "answerable": 0.6524745300529133,
1240
+ "evidence": [
1241
+ "Kütüphanede bugün yaklaşık 40 bin kitap bulunuyor."
1242
+ ]
1243
+ },
1244
+ {
1245
+ "title": "TR · kütüphane (belgede yok, yanıltıcı isim)",
1246
+ "abstain": true,
1247
+ "answerable": 0.27141084178818015,
1248
+ "evidence": []
1249
+ },
1250
+ {
1251
+ "title": "TR · şirket (yanıltıcı yıl)",
1252
+ "abstain": true,
1253
+ "answerable": 0.28409286741619816,
1254
+ "evidence": []
1255
+ },
1256
+ {
1257
+ "title": "TR · Ay (yanıtlanabilir)",
1258
+ "abstain": false,
1259
+ "answerable": 0.6624891481709587,
1260
+ "evidence": [
1261
+ "Ay'a ilk insanlı iniş 1969 yılında gerçekleşti."
1262
+ ]
1263
+ },
1264
+ {
1265
+ "title": "EN · bridge (answerable)",
1266
+ "abstain": false,
1267
+ "answerable": 0.756120137796548,
1268
+ "evidence": [
1269
+ "Its main span is 503 metres long."
1270
+ ]
1271
+ },
1272
+ {
1273
+ "title": "EN · bridge (not in the document)",
1274
+ "abstain": true,
1275
+ "answerable": 0.5633153393855327,
1276
+ "evidence": []
1277
+ },
1278
+ {
1279
+ "title": "XQuAD-tr örneği",
1280
+ "abstain": false,
1281
+ "answerable": 0.7965940107656674,
1282
+ "evidence": [
1283
+ "Panthers savunması ligdeki derecesi altıncı sırada olarak sadece 308 sayı bıraktı, aynı zamanda NFL'de 24 topu kapma ile ligin başını çekmesi ve dört Pro Bowl seçmeleri ile övünç duydu."
1284
+ ]
1285
+ },
1286
+ {
1287
+ "title": "XQuAD-tr · başka paragrafa sorulmuş soru",
1288
+ "abstain": true,
1289
+ "answerable": 0.521832748704309,
1290
+ "evidence": []
1291
+ },
1292
+ {
1293
+ "title": "SQuAD 2.0 dev · unanswerable",
1294
+ "abstain": true,
1295
+ "answerable": 0.18313970364344162,
1296
+ "evidence": []
1297
+ }
1298
+ ],
1299
+ "tests": {
1300
+ "reload": {
1301
+ "returncode": 0,
1302
+ "seconds": 0.18,
1303
+ "process": "fresh python -E subprocess, cwd = temporary copy of the model folder",
1304
+ "passed": true,
1305
+ "cases": 9,
1306
+ "max_abs_diff": 0.0,
1307
+ "decision_mismatches": []
1308
+ },
1309
+ "node_parity": {
1310
+ "cases": 9,
1311
+ "failed": 0,
1312
+ "maxDiff": 1.6653345369377348e-16,
1313
+ "results": [
1314
+ {
1315
+ "name": "TR · kütüphane (yanıtlanabilir)",
1316
+ "maxDiff": 1.6653345369377348e-16
1317
+ },
1318
+ {
1319
+ "name": "TR · kütüphane (belgede yok, yanıltıcı isim)",
1320
+ "maxDiff": 1.1102230246251565e-16
1321
+ },
1322
+ {
1323
+ "name": "TR · şirket (yanıltıcı yıl)",
1324
+ "maxDiff": 1.6653345369377348e-16
1325
+ },
1326
+ {
1327
+ "name": "TR · Ay (yanıtlanabilir)",
1328
+ "maxDiff": 8.673617379884035e-18
1329
+ },
1330
+ {
1331
+ "name": "EN · bridge (answerable)",
1332
+ "maxDiff": 3.122502256758253e-17
1333
+ },
1334
+ {
1335
+ "name": "EN · bridge (not in the document)",
1336
+ "maxDiff": 1.1102230246251565e-16
1337
+ },
1338
+ {
1339
+ "name": "XQuAD-tr örneği",
1340
+ "maxDiff": 1.0408340855860843e-17
1341
+ },
1342
+ {
1343
+ "name": "XQuAD-tr · başka paragrafa sorulmuş soru",
1344
+ "maxDiff": 1.1102230246251565e-16
1345
+ },
1346
+ {
1347
+ "name": "SQuAD 2.0 dev · unanswerable",
1348
+ "maxDiff": 5.551115123125783e-17
1349
+ }
1350
+ ],
1351
+ "ms": 15,
1352
+ "runtime": "node v22.23.4",
1353
+ "status": "passed"
1354
+ },
1355
+ "browser": {
1356
+ "status": "skipped",
1357
+ "reason": "playwright not installed"
1358
+ }
1359
+ },
1360
+ "notes": [
1361
+ "Turkish is zero-shot: the model never saw Turkish during training or threshold selection.",
1362
+ "XQuAD unanswerables are constructed (question paired with another paragraph of the same article, answer string absent)."
1363
+ ]
1364
+ }
examples.json ADDED
@@ -0,0 +1 @@
 
 
1
+ [{"title": "TR · kütüphane (yanıtlanabilir)", "lang": "tr", "document": "Kasaba kütüphanesi 1962 yılında açıldı. Binanın mimarı Selma Arı'dır. Kütüphanede bugün yaklaşık 40 bin kitap bulunuyor. Her yıl haziran ayında bir kitap şenliği düzenleniyor.", "question": "Kütüphanede kaç kitap var?", "source": "hand-written example (original text)"}, {"title": "TR · kütüphane (belgede yok, yanıltıcı isim)", "lang": "tr", "document": "Kasaba kütüphanesi 1962 yılında açıldı. Binanın mimarı Selma Arı'dır. Kütüphanede bugün yaklaşık 40 bin kitap bulunuyor. Her yıl haziran ayında bir kitap şenliği düzenleniyor.", "question": "Kütüphanenin ilk müdürü kimdi?", "source": "hand-written example (original text)"}, {"title": "TR · şirket (yanıltıcı yıl)", "lang": "tr", "document": "Şirketin 2019 yılındaki geliri 4 milyon liraydı. 2020 yılında çalışan sayısı 120 kişiye çıktı. Merkez ofis 2021'de İzmir'e taşındı.", "question": "Şirketin 2020 yılındaki geliri ne kadardı?", "source": "hand-written example (original text)"}, {"title": "TR · Ay (yanıtlanabilir)", "lang": "tr", "document": "Ay, Dünya'nın tek doğal uydusudur. Ay'ın Dünya'ya ortalama uzaklığı yaklaşık 384 bin kilometredir. Ay'ın yüzeyinde atmosfer neredeyse yoktur. Ay'a ilk insanlı iniş 1969 yılında gerçekleşti.", "question": "Ay'a ilk insanlı iniş ne zaman yapıldı?", "source": "hand-written example (original text)"}, {"title": "EN · bridge (answerable)", "lang": "en", "document": "The river crossing opened to traffic in 1932. Its main span is 503 metres long. The bridge was repainted in 2008 using a grey anti-corrosion coating. About 160,000 vehicles cross it every day.", "question": "How long is the main span?", "source": "hand-written example (original text)"}, {"title": "EN · bridge (not in the document)", "lang": "en", "document": "The river crossing opened to traffic in 1932. Its main span is 503 metres long. The bridge was repainted in 2008 using a grey anti-corrosion coating. About 160,000 vehicles cross it every day.", "question": "How much did the 2008 repainting cost?", "source": "hand-written example (original text)"}, {"title": "XQuAD-tr örneği", "lang": "tr", "document": "Panthers savunması ligdeki derecesi altıncı sırada olarak sadece 308 sayı bıraktı, aynı zamanda NFL'de 24 topu kapma ile ligin başını çekmesi ve dört Pro Bowl seçmeleri ile övünç duydu. Pro Bowl orta çizgi savunmacısı Kawann Short, takımı 11'le birlikte sack ederken, aynı zamanda üç oyuncuyu düşürme ve iki yeniden savunmaya zorladı. Yan hakem Mario Addison, 6½ sack ekledi. Panthers hattında ayrıca, sadece 9 başlangıçta 5 sack eden uç çizgi savunmacısı Kony Ealy ile birlikte 136 kez ile NFL'nin aktif kariyer sack lideri ve 5 kez profesyonel bir top atıcısı olan Jared Allen öne çıkmaktadır. Arkalarında, Panthers'lerin üç adet ikinci hat savunma oyuncusundan ikisi de Pro Bowl'da oynamak için seçildi: Thomas Davis ve Luke Kuechly. Davis, dördünde mecburi oyuncuyu düşürme ve dört topu kapma olmak üzere 5½ sack etmeyi bir araya getirirken, Kuechly top çalmalarda (118) iki mecburi oyuncuyu düşürme ve kendi başına dört pasını kesme ile takıma öncülük etti. Carolina’nın ikincisi olarak gösterilen Pro Bowl güvenliği için kariyerinin en yükseği olarak 7 top kapma ile takımı sürükleyen Kurt Coleman’a ve 88 top çalma sayısıyla ve Pro Bowl köşe savunmacısı sezon boyunca bir kapalı köşe içinde gelişen ikisi kale çizgisini geçişten dönen dört top kapma geliştiren Josh Norman'a rol verdi.", "question": "Panthers savunması kaç sayı bırakmıştır?", "source": "XQuAD (tr), CC BY-SA 4.0"}, {"title": "XQuAD-tr · başka paragrafa sorulmuş soru", "lang": "tr", "document": "Broncos, 23–16 arasındaki bölge turunda Pittsburgh Steelers'ı, maçın son üç dakikasında 11 sayı alarak mağlup etti. Daha sonra, New England'ın 2 puanlık dönüşüm girişimini saat üzerinde 17 saniye kala bir pası keserek, AFC Şampiyonluk Oyununda Super Bowl'u savunan XLIX şampiyonu New England Patriots'u 20–18 yendi. Manning''in sezon boyunca topu kapmalarla ilgili sorunlarına rağmen, iki rövanş maçında hiç atmadı.", "question": "Panthers savunması kaç sayı bırakmıştır?", "source": "XQuAD (tr), CC BY-SA 4.0; question paired with another paragraph"}, {"title": "SQuAD 2.0 dev · unanswerable", "lang": "en", "document": "The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants would gradually merge with the Carolingian-based cultures of West Francia. The distinct cultural and ethnic identity of the Normans emerged initially in the first half of the 10th century, and it continued to evolve over the succeeding centuries.", "question": "Who gave their name to Normandy in the 1000's and 1100's", "source": "SQuAD 2.0 dev, CC BY-SA 4.0"}]
index.html CHANGED
@@ -1,19 +1,46 @@
1
  <!doctype html>
2
- <html>
3
- <head>
4
- <meta charset="utf-8" />
5
- <meta name="viewport" content="width=device-width" />
6
- <title>My static Space</title>
7
- <link rel="stylesheet" href="style.css" />
8
- </head>
9
- <body>
10
- <div class="card">
11
- <h1>Welcome to your static Space!</h1>
12
- <p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
13
- <p>
14
- Also don't forget to check the
15
- <a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
16
- </p>
17
- </div>
18
- </body>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
  </html>
 
1
  <!doctype html>
2
+ <html lang="en">
3
+ <head>
4
+ <meta charset="utf-8">
5
+ <meta name="viewport" content="width=device-width, initial-scale=1">
6
+ <link rel="icon" href="data:,">
7
+ <title>Next Trace</title>
8
+ <link rel="stylesheet" href="style.css">
9
+ </head>
10
+ <body>
11
+ <main>
12
+ <header>
13
+ <h1>Next Trace</h1>
14
+ <p>Ask a question about a Turkish or English document. Next Trace marks the sentences the answer rests on — or says plainly that the document does not contain enough evidence. Everything runs in your browser; nothing is sent anywhere.</p>
15
+ <div class="badges"><span class="badge">Türkçe + English</span><span class="badge">Live in-browser</span><span class="badge">Trained on SQuAD 2.0 (CC BY-SA 4.0)</span><a class="badge" href="https://huggingface.co/thelamapi/Next-Trace">Model repo</a></div>
16
+ </header>
17
+ <div class="grid">
18
+ <section class="card">
19
+ <label for="ex">Example</label><select id="ex"></select>
20
+ <p class="small muted" id="src"></p>
21
+ <label for="doc">Document / Belge</label><textarea id="doc" style="min-height:180px"></textarea>
22
+ <label for="q">Question / Soru</label><input type="text" id="q">
23
+ <div class="row" style="margin-top:8px">
24
+ <select id="mode" style="flex:1" aria-label="method"><option value="learned">Learned reranker (Next Trace)</option><option value="bm25">BM25 only (baseline)</option></select>
25
+ <select id="lang" style="width:120px" aria-label="interface language"><option value="tr">Türkçe</option><option value="en">English</option></select>
26
+ </div>
27
+ </section>
28
+ <section class="card">
29
+ <h2>Verdict</h2>
30
+ <div id="verdict"></div>
31
+ <h3>Document with evidence</h3>
32
+ <div id="marked" style="line-height:1.8"></div>
33
+ <h3>Top sentences</h3>
34
+ <div class="scroll"><table id="tbl"></table></div>
35
+ </section>
36
+ </div>
37
+ <section class="card" style="margin-top:12px"><h2>Recorded benchmark</h2><img src="preview.png" alt="Benchmark" style="width:100%;border-radius:8px"></section>
38
+ <p class="status" id="status"></p>
39
+ <footer>Next Trace selects evidence sentences; it does not generate answers. Example passages from SQuAD 2.0 and XQuAD are CC BY-SA 4.0; hand-written examples are original.</footer>
40
+ </main>
41
+ <script src="nlrt.js"></script>
42
+ <script src="nlui.js"></script>
43
+ <script src="core.js"></script>
44
+ <script src="app.js"></script>
45
+ </body>
46
  </html>
model.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"format":"nlrt-1","meta":{"name":"Next Trace","thresholds":{"bm25":2.7921873897475704,"answerable":0.6,"second_sentence":0.5},"features":18,"languages":["tr","en"],"trained_on":"SQuAD 2.0 train (CC BY-SA 4.0), article-level splits"},"nets":{"sent":[{"type":"linear","w":"kXwPvdkBnL5jYD68sCEBv52Smb77aBc9bTfgvj+SjT5zDhi/ctsfvhPY3D29Puo+3NOZPtnEZb3w5sQ+himivvwZsT0FeTa8mQccPoP/Sr6qrrE+uqLevJHcxrwm8Gw9C/ICP6XxeT6rWja9q5vjPHpslT2Mt0M/zMiNO0YI5b6oLeK+riQJvQ+ypz6vXoK9KKDTvY6KNj7AtC0+Xg+EPWEWqrvLfwK+cuk6vl33sr0BYZ89lnq+PZrKmb2wbxG+6qpjviUzM75QMYi9WApfvmJbFLxnfCg9ROa+PR62Dr5wFLK+F2TEPg51sDz3RuU8zhRyP9awr71pj989XcAEvaYfYb6IJSA+G30/PdikAzynJiW/+iBDvuJHYT4IAgS9nCFhvqET4T3nKdq8tUFdvhLoKD5zrp29SfqYvjU7+71YNVk9JP9GviQ3i736i46+g6YbvJQ6gL7/CXC+Y83MvZA/Pb6QpRi++BMhPlPeDL5F6uA7yM8MvgHo7buGvKI+vLEdvuQlIb63Xqe+CBUTvlWJbr4lQaK+WtDJvtIZIz4fQMo8Nmsuvj+wyr4zl8k+2edVvpLUybsngpQ+TDSYvRXCGD4WD38+LvIxPl+n9z57pR6+LofIvQuOnj7+QAW+R7mhvRKklL6aOYK+Uc5XveGkNL/9IQk+ZOl8PBweCDngVIg+tfmnvmW8cD7BO3A9JBOYvjnsCj5tgJm9t1gVPWRXgb5M+Ks9c4mWPpZivL5yS/o9MOkFP+T47r0mAgg+Di7DPu3jU75dDYa+n66qvuGXN75BtMq90JKYv0YgID52QIe+wNMnvoEZxz0KWZu+8oJxPSlVMz3CR8q9GHGRPl6S2z7CNxG+bvAyPlbIwD1Rp6m+7KV7vzja4T6ylhY+DIJXv8BEkL26AXy+KELaveV/zT7mdl0+gxoaPtULW75dhSI/BbEMv4VQfz4QloE+WoHjPqEA0z1BQc07q3vZPkqZRj7jRJG+s6PTPcT5D77Ulc88CtAivs1Bhz6Bs549TpWnPimgLL5Hv7Q9zajjvk2Y1D4/dKy9OUXHveid1r0YntM9AfwGPsrwRb4OCIQ8pc4ePsrZNr7c2By+teIKvjH++721juK8Xp1ovrzX/TzFVym+yXkbPuF5FL6mVSY+HUsgPvT/br5B2IS/P0MEP45hlj5PQwG+ybUZPtEPQL4mscQ9uz4LPjXsKD3e4vS+ElmPPlxpWj5ZxUs/i8ptPizExz11D00+QuwMvijZZjuPTCO+1w5MvZbfnD2UfsY6dqY6vgXC4r3rlwE8GPMRviTbKb6u3hy+cLGKPcn8S761YsY9V/iLPDOZdryhzOS9F1G5vv4OvL43wgw/dJ+LvcwZEb7LZ6K+1Rcmv6GfZ77c9SQ8yGLdPGm/Cj6djvg9pMkxPwNxkz3q6FI9Ejf6PkgbOL/wpVa97VdsvrCJRL5n+iI+gVCCPrXqgz3pZUi+p/FhPh2DC75HZ0e9iX0FvYhhWT1W8yy+FdgEPvAi2bzoHbQ+yUVAP8oUrj75leo+EYhtvln3Qz3NW5U+6UmLPCMd9z1NyBG+2Dupv37UqL7l+Vu+MAGKviOcJj2lJ0c+yGzAPlM2Pr4hi0y+aJA9PlU8MD5eNfE+BLWsPk/GML0F7Ig+NZahPhc34Tz34Cm+6eH1Puz4Ab4O7eY9TQAWvpYRxbuQkss+6Dz2PgruPD0NT9++3OOLvYoqBT+Z4by9mylOvkq3sTx/g0m9IKwQvj6NXj7znY8+Mzy+vkQYWz1zwoS9Yh3lvXHaoz5zyOM9noejPXO2ir6keoi9xygqPlSuqb7DGKw+qSPevVgJ8rwBoWI9wVYPvylXfr2R6ke9vxBCv15DF765Rek+EMUXPswDi77xJ9s+qdMfvhCbVb762us+GTkUPWuPnj6dCY4++0vJPcZNwD0Uhpe9iMBDv8WR/bw83is+dwGuv2vGoj71TqW9ciNJv77LyT7zM1g987taPdGTer40+Ys+xT1hPiXC4r5lNbG+3XJePXityb6qZHK90MMevrd15z58S5a93wUKP5MR+T72WfU+HqamPkqqwb5+acY8nySEPkViOr18Aso+w4DwvowiS76Ag8g8MymAPbmKqb1QWyE+jLdKv5wXRL8irX4+Owudv7Krkj2/uek96PAjPoA61TyeM9k85gfKPK0Ue7waqOM+4DoAPz8HUT45O1m9WOANPr/hCsAfwlA/He47PWl0gr5gswm+KnuXvtuaVT1Oogm9HXievdr09bw92jg+FkUAPa0+Lb4Nzuc+as3ZvV+kqz6CgES/z5R3vp5arzuXYto+fZyGvkZsq761vVW9B6Ycv+/+DryHvs692d6/vBLIcj2OP6++o0RJPqj9GT70BSy+UqpWP1ynj758suc9YK/JPuzTkz3Zet2+sy5XPoTETT4zzs89n4gLvqcV3D0Qx5A+r4NVPQBxQr1utzk93+vfv+vEB7w+tEc+TH5RPUkxCj0641W+mMycPnOWzrvt8Yu+sMcpv9P/Gz7LyFu+lBQawJyWYz7XoGo8d/IOPIVfAT6qQGu8U3nlPR0tGLyloVy+uCHMPum3Jz9t8gg+RPfiPZsgJj4EPE8/ir07PRos5r4vW5G+1LJtPg+vQj5kPnM+fRNrPqlhlb0L6js8wygBvl0dL74wQG+898fzPQCpGj8j/fs9BeRWvYFMlr15Lt89zE62vg4akbwbXa0+DsemPhUXiD6B83a+o2t2Polqrj3Vpsa+ZL7UPfK5Rr9mlmm9N+3SPuFpAb77lmA+uIl3PRhZpr4MFiU+Z4UPvsRqf77Lzn0+WNGTvinTc7+nVD4+XICtPRTgXb5y1zI+e/O3PdeiQb6dp7g+jgnqvFpBNL7VhzS+c/vSPDZmDj5+6uw9rGkYPks3/j22c4C+7CXWPszsCz469AE/OCOnPtaprbxXhIS9GGdLPiIaNT5lL6k+kKQMO+UhEz/SYsi924wZPnYw2L1WnuC9quXtvqYmm7z+f4Q+uF9vv/QbID6bbb89bwCOPlxhgr0uj8G9Pw8Ev4i3Ir4Dww0+BZEQOwSwor7YzBw8","b":"k58dve8Clb6WMRu+EHMqPlPtSb5LEKQ+z4LlPrPyzjzuRHA+qB1MPqjjBb3jdFy+B64Bvnx5RL65Voc+ZcvKvTelPT436Xa+Bap9Pr9wNb3mgG8+sPtkuz+Lw73uX9C+ENhaPrCKrj6Drou+xBlOviZI1DyVgeY9GQWcvhnc1j0=","w_shape":[32,18],"b_shape":[32]},{"type":"relu"},{"type":"linear","w":"WjpSPschVb+NJVG9Gdf7vgIl6T3YGM+9kzPJPmF9Bj3ziTI+kVSHPoUahL7exlI+b1FKvxyBKbyfUUg/gGHpPIgKur3h6EW/RltKPvRl070HUKI+89SHPvUczzxBRLW+6Y08vrRA4z4aRjY/A1lLPZzwVjyJktG9BhFOPaRa7r2EyLc8JxsGPaMSoz2JMd298yyfPXR8n72y/T+7enQTP6bG8D59vvC909QzPhhSpb1VeNw9368hPkYUI75nWws/KeqTPhcvXz4hlto9a5EZvYP82j6dT/a7++zAPj71J7yjntI+SyWOv6g1Qz8OGAE/XHz6PuMOhryRK9S9bKLIvthedj58Fw6/yfhOvZHWO77QuS2+V3aVPe/5Cz77xW29YQIFPwOgxz1e9IY8KAEPvh7i+b7UMZw75aFQP5zkHL4Vb8899CHBvnA47zr4MiQ+3ajhPm/7YDz1dKI+tDeQvt84ab0Y9ik+NI5QP6DKPr6skHi8k7Y8PV4vUjwGmCo+rmCjvb3uEL913Qe+dJ52OyaYOj5syyE+2lbuPrgkSD2GCrY9bGuEPsC0N71W//a9ykEXv9t28T2pNTg/09PdvkHqQT5ORIC+/qPhPcaZPj4gAvk9H66OPoReez5kjgy/tTQevggKnj76hDI/bld1vn8gRL54ZpE9I9Q/vjpvDz6vz9s8WYW3vgXGT70K1bs+jJKdPXBwDrzj7C8/t0o/OxXUsT7ikIK8yWiUvmowHL2qqyO+nfcdPd5B8T6J76++LrCYPlhU4L7xhEE+R368vvisuz2J8eM91hW8PLSvJb9DXJ49/OMLvty2yD+PCg87GXOfPXN5kb7eGA++CIGAvoLgND3xotq+7sy0uzVZqL5DnAE+sF5iPPIkWT2n7tk+1MHBPhYB7r03Kxy+u+VnufAcSb9vE8g9tsL+PgoRhD510q89zuryvkd4kz3ouhg+sB0kPhcDir4XdQU/yegdv6xAKjzTSci++EgzP/5zoD6iVg49qg0QPjgCqz3/Hcm+Vv2jPnrfWDzcWc29rraYvcr8dz2Xkqs8pCLBPQkIgzyU0vy9iL1IvTsQEj4bstA9Rmi9PSujUr0/NgS+LUXxPRi0Br628pY+8CPbvcKCv7z1Q6u+jfSfPcWNLDzXIR4/2nBlPVXguj6GrwK/2b16PqT3JL6eGaE+Xl7mvN+oxD7Frho9M7gYvr0MA77XkJK9IBNvvSo+LT2Jves9B+xcPuPp5j7LHEE+lfdOvbUl1D33wSC+qVxNPYctFT4EZ/M9FTsyPo84Q73TUk8+WcmbPvO+Gj0Pbku9AGFRPkIFE78rEMc8SkpIvuYIGD8eoUA+9eSIPpG8zz2Kqku9Y2xHvlrKsT2RX5U9g2PXPT1Uw766hdo9H8kOvi63r77Er5Q+S8JOv6HFKj8gLAw+K4ofPB6SCz+2rx4+WIrSvnhWZz6cd2S+AUvCPdMBIT7Rs4i+sS3svTv4gz4MULK/tMJLP/r7u7/Z45u+BWWMvreUdL5Tc4u+jesrvVeamD6NlhC/ouCQPFz3xL6HylC9E2tcvoD0hr3gcU48HvetPe92ZT4/lCk+/VSFPiRtTb7DQfy9UxRKv/SZGj7aBAQ+xYaiPpU01T0Gwsm+v80lvblrYz7srqc+lCqdvruj4T6ggRu9TFwWPvdn+b2iIVY/3U+ePslWILx7wqs+qokEPjrTj72IvPM+Kb2xPZ5X6D36vak9t9ASvsK1hb5SRi4+UWZZu+0gAr8fyBg+n1hVPmrNO70L4As/4T2rvIl2Gb1bGAC9+h7WvvAEmD4HOcy9kE3tvlYpd76wOQM+Sawvv66oHT8+AkO/BdmiPXw0zb0YzKA+ibKuviQnqj2fP98+4h0Pvmc1Ir7TRyu+SMAAvMaOQb2e/Co+TCImvu3fMr5j+Xy9G+YvvQkgJb7oZ/q9fP0zvbix2L38jsM9BmoCvbuoAD2Wki49I+hUO6+tdb5e8SW+lq/avO3pyb3Guv+8pYoSvoYVsz1PhUY9endCPUPnhz28MqE99K0Ovp2sGr4FTbW94bkIvopMtb4knxK+omzTvkPuuD1Noiu9q/dVvqiPYr0IkJC+XWePvQdXS7nLxpI9TKsLPwwc0bxcTgo/ZcXaPMoKKL5bFAC+wLNrvsAkE754PG09Ya+WPUaXjj6hbhm+eIcPP1O5N78v65o+bXc3vkAgPL86Wgu+S9GLPTtLrL1qMfE8C1cHvysH+L3t/WW+r9ECvjXaO7zqJRU++a6iPiTj9T0Zdxc+EE4cvvIDFj04aWu/Oc2SvdB5LD9ux9m9h9ygPQOT2741HVE+evqSPqnkhT52rhi+KJ2GPp8XCb8kDU0+i68uPtSiNj+OVcs9PZHwPVMfoz4AxHC+bgOzvLNVjL1YMsS93tavPfmmHD7g08C8oiFBvgbTJj57hxm+hXggPqyZnT4wmzu+u0m9PYBZyzv2+xC+YUOVvr8hCD3QzrM91XRNvtivTjxlja47Ky6XPmXQ9D5MNwQ+bp1Bv068mL6qAz2++fFUP2x00jxr+ew+ApgVPWnSFz62Mh6+DEp5vjlGrT1kpuM9pHPhvcHQE77QRFO+1ENqvhjBk71FcrS8lvWSPA3bUL7yKky9RY9JPfGdbT3WTFq+EcufvQ3vfb2XFKG9Xok6vsL8qL0ZXUE89g2xPZ9pqb2lMwi+7tNKPciJJb7/aMy8HJZFvpyGE75ylrk9mudLvojVmT2e+GY9XiCjvnHhmz2r/Xm8cnAKPhuIgDz6Xf09D+mfPbPwrj6h+Tg9Bw1Bvu1DOz4nqgO+MpkSPlVKYT6tWsy9as2evdUUhL2X5CE+7IHpPeGKzDy0VBS9vUADPTqc5741Rho9tOjpvDU1Qz89TUQ96JYWPpM9PT0iIiS90XzOvDrZYz5+i/O+rHHiPSyDBD5qOTs9dNeVPlo2az5Jfoc91X+KPGQtbD4ypzW+fT0sPhZ4K78KVUg7mcQ4P5Ifqb7YX/q7sKcZvzO5rrzD9989V0fEPX6xVj4emGa9Adu+vqUaSr5DfMs+BrRLP2KhDb6ih+G8NTcfPrT2v70lJyA+NhQOv5yKtr11mAi+IYmRPmz/Zz1SPOC9vICQPonmgb3XFRk+qMzVvVyno77aBvS8l52vPKQ2uL1XcLg+xAQRPrDimD7grja81A7UPYrzhj5M1zs9YfygPgLE3T1IgX+/QJEev4kXB75Dups/vvATPQNW3j4Qtce+52tXPqg3cjwvnzw+wQ2lPmkkr70WTJC9XY4VvsMgRz6pki0+rbf9vSnpJL6fFae8Lq2CvW7+Tr5ZTto6DH+NvDMfYz1NAMk8bHADvd94hD44FYC+Sts7PhtYur7IjrM9c9RmPXZ8JT8gh00+1N72Pm/DR7/DQGI+OAKNvlLmdD6l/Tw9RcpkPlqRorxeHJi+tcmDuzoUGT7NqLY9+VqBPjkK8z5rwVu+9rCAPWORpj7SqpY9O/+iPfdHb7/7P7c90/K+vpeyhj10c9s937ojv4xT3b21vrS+k8bSvMHFxzzvf6i+RYQQv3D/674uygI/YQmDPmIo0r68vx48TXukvg2WUb+uV1y+pX8OvhNpRL6E4zG+cDidvjE0Rr7Kktg91ltpvpmomr6mene+m4b5vLOFrD1WsAU+4ox7vuzv/r1DkQM+PdtkvkrKK77otau95+K8vc1Qp72s8S2+8lKmvj9DPb2Svmc84DisvQf0UL7t5BM+HOgBPuuoTD6rWgi+KMAWvdAFFL2zEb8+uvGWPh5WHD1/G9o9XQ+wPVb5Sb5gL5s9kZ9XPTlGxr7AAlE+eh4qPj4UHzxAvb4+XtlGPZ8mmTyMoVG9lOFMPel+mT7SiMe9ZDzSvqOwpL7d94M+rxcIvyDRGD/7YUu/cIisvLhu0L5s2F0+h9Ctvoj1UL3CGpM+1ZRwvuKhij0IyBC/hTSGPKFRWr2rhDa9hgYIvlelHT9JBiu9Jmj1u9OFQj5ZgfG9EZnBvaPTFb+nQ1S9zA0VP2zZSb4dI7C8ZJytvmFIlj3hJYc+466WPnRYmz4hha69showv+KBsb7iwiw+La2NPxlPs77r3eq+MBAvvq+EsTzGwiW9UY8aPtvMtL6HzSk+4urPvvCHSjsGkIi7xLQzvl/gOj4OPV09c7FNPjsBub4Oi0m9CRk+v6xbHb4rV8E+BBiUPZ7qGb1kWte+wWmoPIR0fT5ZBp0+xtEsPsop0D40JZC8deoxvhrKCDvZ2Zs/wPQUPqcYJr5Ot6M+UC+dPhIhrb2HrjG+674hvrd/sr20KDa9afAbPp2RP75N0wq+WUEHvj3s8b0FJY2+3zhfvusOMT7B+ti9LX4jPqUCgb5XO8y9hYAkvl6n6jxLxK29tLvgPRv5GL6LNJs95T/xPWBSlb3jWM499lkyPaSMNr5bO+e9NEjnPbhJRr4v5Pu9VZFaPRUSa74zPVa+yAdwuyoIJL4+QyE+b8KMPV2Scz4Qmpa9ROBXPgFyIT6lcyK+We8fPgmzlb0vzJa908r9PCQqdj4cZnA+v70RvhYlDrybn/09OjrRvDKGr7xbU3M+yK4IvzCHQb5EWaa+3oxOPyxhfz7jEYo+WSWLviNxAL6+Bx6+mWAivsbNj740BXG92dxQvdeQv72xVUY81hrwPI2IHj7Xc98+95mMPr7LcL59Cwm+u1NivZiI2z05ypC+DjtRPsbcbz62bX69Xf4cPURKlD6pVcA9JoQDPkAtBj6UxSK/E/J5vge2u77lCzs/pfezPY+IDj/ce5u9vkgtPvGWRL5Bfjw+M9mPvTGGTz1jguO9NgHNPdR7Rr6ulwY8BhVsPUwBXD1UXLi9NR8Vvid/zTvgCHC+I4EkPbHVrDz1yRq+KdNJvsiJKb6r6bM9Qzo0vnXyTL3JzRi+WkHSPQKrIL0DMXo9nykoPhKBsD2IOmq7fWsBvdr1Eb5N64a+LPj1vcBdDb24wqq+TQ+wPfLuyr0UenS96vu1PXh3sT64tfU9ZQcUP/Dl6D0S7k2+IcQFvuKv2r4eLB6+lq/1PPEUMT6wmZQ+iBOxvtk9Qz6S7q8+jhfqPaeujD5jZJU+L5R0v0NxKz4m0hW+gCVVP/h2Gb3m5xc+o4cIvpp6TD4V/II9nLDWPURXDb8pB9e7T2gNvmwnAj2p7PS7q3YSP061Wj3hmtU+GK5+Pjw8HDwVC/U9KDd+vmdmo7wOhAI/RD+kvi1Cjz3GCCK+bFp2vfAuqT3ZdNk+ULQBPkiFUz5fqBW/fb8BPgcQNz7yijo/vX+lvaRjgr6t9zg96LUMveAvy73Sqao+W8KXPmQc+z0MQyE++ql5PQXF3b6vnBG8qg3ZPpdq4r4+A64+I4NIPiTNC76HeI8+n9eLPLvQFL83Jic+VA02PpcRzz7SuZY8h1novvJfWL7Mt3A+Uc2Kv+6bXj84y7W/I2mcPJX2k77P2dA+Zug5vk/9Kj7O1Ys+yMYDvw==","b":"zYVaPk9mHD5ZF5w+tXUdPnrjBz7DEJI++6cxPYubX72j0vQ8V8fcPRcCETy6I4i+7hKPvukmkT63aTE+a1RTvjX6Tj3tlrA9UQ4fvgHJqrwGBlU+0tDuPIUMsTxWN5c+rFi0PeD1Tr76zn4+S0lsPj/OHb4c760+DIcfPvDJFj4=","w_shape":[32,32],"b_shape":[32]},{"type":"relu"},{"type":"linear","w":"GS3pvtOiCr/+rY++183XvgbJ9r5xDau+aegYPhw+r70r5zQ/6lmsvqmYZD4a8Go9+8XEPhAgxr6bwIq+WtFQPa50lrsIKtW+WnLfvp4rQz46Akm/zQvRvMDZKj68laK+y6bfvuxWsr0pt7i9eLEAvnUEgb2cyaa+rx4ivpn5Gj8=","b":"rGexvQ==","w_shape":[1,32],"b_shape":[1]}]},"tensors":{"ans_w":{"shape":[9],"data":"eDBDQFW7Ub/NiDM/GiNYwMYPer8iXSq+gkLZPx9MOsBLVZe/"},"ans_b":{"shape":[1],"data":"TzJsPw=="}}}
nlrt.js ADDED
@@ -0,0 +1,219 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /* nlrt.js: browser and Node twin of nlrt.py. Batch size 1, float64 math. */
2
+ (function (root, factory) {
3
+ if (typeof module === "object" && module.exports) module.exports = factory();
4
+ else root.NLRT = factory();
5
+ })(typeof self !== "undefined" ? self : this, function () {
6
+ "use strict";
7
+
8
+ function b64d(s) {
9
+ let bin;
10
+ if (typeof atob === "function") bin = atob(s);
11
+ else bin = Buffer.from(s, "base64").toString("binary");
12
+ const bytes = new Uint8Array(bin.length);
13
+ for (let i = 0; i < bin.length; i++) bytes[i] = bin.charCodeAt(i);
14
+ return new Float32Array(bytes.buffer);
15
+ }
16
+
17
+ function decode(spec) {
18
+ for (const name in spec.nets || {}) {
19
+ for (const L of spec.nets[name]) {
20
+ if (typeof L.w === "string") L.w = b64d(L.w);
21
+ if (typeof L.b === "string") L.b = b64d(L.b);
22
+ }
23
+ }
24
+ for (const name in spec.tensors || {}) {
25
+ const t = spec.tensors[name];
26
+ if (typeof t.data === "string") t.data = b64d(t.data);
27
+ }
28
+ return spec;
29
+ }
30
+
31
+ async function loadModel(url) {
32
+ const r = await fetch(url);
33
+ if (!r.ok) throw new Error("cannot load " + url);
34
+ return decode(await r.json());
35
+ }
36
+
37
+ function pair(v) { return Array.isArray(v) ? v : [v, v]; }
38
+
39
+ // Tensor: {shape: [D]} or {shape: [C, H, W]} or {shape: [C, L]}, data: Float64Array
40
+ function T(shape, data) {
41
+ const n = shape.reduce((a, b) => a * b, 1);
42
+ return { shape: shape.slice(), data: data || new Float64Array(n) };
43
+ }
44
+
45
+ function as3(x) {
46
+ if (x.shape.length === 2) return { t: T([x.shape[0], 1, x.shape[1]], x.data), sq: true };
47
+ return { t: x, sq: false };
48
+ }
49
+
50
+ function conv2d(x0, L) {
51
+ const { t: x, sq } = as3(x0);
52
+ const [C, H, W] = x.shape;
53
+ const [O, I, KH, KW] = L.w_shape;
54
+ const [sh, sw] = pair(L.stride), [ph, pw] = pair(L.pad);
55
+ const OH = Math.floor((H + 2 * ph - KH) / sh) + 1;
56
+ const OW = Math.floor((W + 2 * pw - KW) / sw) + 1;
57
+ const out = new Float64Array(O * OH * OW);
58
+ const w = L.w, b = L.b, xd = x.data;
59
+ for (let o = 0; o < O; o++) {
60
+ for (let oy = 0; oy < OH; oy++) {
61
+ for (let ox = 0; ox < OW; ox++) {
62
+ let s = b[o];
63
+ for (let c = 0; c < I; c++) {
64
+ for (let ky = 0; ky < KH; ky++) {
65
+ const iy = oy * sh + ky - ph;
66
+ if (iy < 0 || iy >= H) continue;
67
+ const wb = ((o * I + c) * KH + ky) * KW, xb = (c * H + iy) * W;
68
+ for (let kx = 0; kx < KW; kx++) {
69
+ const ix = ox * sw + kx - pw;
70
+ if (ix < 0 || ix >= W) continue;
71
+ s += w[wb + kx] * xd[xb + ix];
72
+ }
73
+ }
74
+ }
75
+ out[(o * OH + oy) * OW + ox] = s;
76
+ }
77
+ }
78
+ }
79
+ return sq ? T([O, OW], out) : T([O, OH, OW], out);
80
+ }
81
+
82
+ function maxpool(x0, L) {
83
+ const { t: x, sq } = as3(x0);
84
+ const [C, H, W] = x.shape, [kh, kw] = pair(L.k);
85
+ const OH = Math.floor(H / kh), OW = Math.floor(W / kw);
86
+ const out = new Float64Array(C * OH * OW);
87
+ for (let c = 0; c < C; c++)
88
+ for (let oy = 0; oy < OH; oy++)
89
+ for (let ox = 0; ox < OW; ox++) {
90
+ let m = -Infinity;
91
+ for (let i = 0; i < kh; i++)
92
+ for (let j = 0; j < kw; j++) {
93
+ const v = x.data[(c * H + oy * kh + i) * W + ox * kw + j];
94
+ if (v > m) m = v;
95
+ }
96
+ out[(c * OH + oy) * OW + ox] = m;
97
+ }
98
+ return sq ? T([C, OW], out) : T([C, OH, OW], out);
99
+ }
100
+
101
+ function linear(x, L) {
102
+ const [O, I] = L.w_shape;
103
+ const out = new Float64Array(O);
104
+ for (let o = 0; o < O; o++) {
105
+ let s = L.b[o];
106
+ const base = o * I;
107
+ for (let i = 0; i < I; i++) s += L.w[base + i] * x.data[i];
108
+ out[o] = s;
109
+ }
110
+ return T([O], out);
111
+ }
112
+
113
+ function map(x, f) {
114
+ const d = new Float64Array(x.data.length);
115
+ for (let i = 0; i < d.length; i++) d[i] = f(x.data[i]);
116
+ return T(x.shape, d);
117
+ }
118
+
119
+ function forward(layers, x) {
120
+ if (!(x.data instanceof Float64Array)) x = T(x.shape, Float64Array.from(x.data));
121
+ for (const L of layers) {
122
+ switch (L.type) {
123
+ case "linear": x = linear(x, L); break;
124
+ case "conv2d": x = conv2d(x, L); break;
125
+ case "relu": x = map(x, v => (v > 0 ? v : 0)); break;
126
+ case "tanh": x = map(x, Math.tanh); break;
127
+ case "sigmoid": x = map(x, v => 1 / (1 + Math.exp(-v))); break;
128
+ case "maxpool": x = maxpool(x, L); break;
129
+ case "gap": {
130
+ const C = x.shape[0], n = x.data.length / C, d = new Float64Array(C);
131
+ for (let c = 0; c < C; c++) { let s = 0; for (let i = 0; i < n; i++) s += x.data[c * n + i]; d[c] = s / n; }
132
+ x = T([C], d); break;
133
+ }
134
+ case "flatten": x = T([x.data.length], x.data); break;
135
+ case "reshape": x = T(L.shape, x.data); break;
136
+ case "upsample": {
137
+ const s = L.s;
138
+ if (x.shape.length === 3) {
139
+ const [C, H, W] = x.shape, d = new Float64Array(C * H * s * W * s);
140
+ for (let c = 0; c < C; c++)
141
+ for (let y = 0; y < H * s; y++)
142
+ for (let xx = 0; xx < W * s; xx++)
143
+ d[(c * H * s + y) * W * s + xx] = x.data[(c * H + Math.floor(y / s)) * W + Math.floor(xx / s)];
144
+ x = T([C, H * s, W * s], d);
145
+ } else {
146
+ const [C, Ln] = x.shape, d = new Float64Array(C * Ln * s);
147
+ for (let c = 0; c < C; c++) for (let i = 0; i < Ln * s; i++) d[c * Ln * s + i] = x.data[c * Ln + Math.floor(i / s)];
148
+ x = T([C, Ln * s], d);
149
+ }
150
+ break;
151
+ }
152
+ case "l2norm": {
153
+ let s = 0; for (const v of x.data) s += v * v;
154
+ const n = Math.max(Math.sqrt(s), 1e-12);
155
+ x = map(x, v => v / n); break;
156
+ }
157
+ default: throw new Error("unknown layer " + L.type);
158
+ }
159
+ }
160
+ return x;
161
+ }
162
+
163
+ function mulberry32(seed) {
164
+ let a = seed >>> 0;
165
+ const random = function () {
166
+ a = (a + 0x6D2B79F5) >>> 0;
167
+ let t = a;
168
+ t = Math.imul(t ^ (t >>> 15), t | 1) >>> 0;
169
+ t = (t ^ ((t + (Math.imul(t ^ (t >>> 7), t | 61) >>> 0)) >>> 0)) >>> 0;
170
+ return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
171
+ };
172
+ return {
173
+ random,
174
+ uniform: (lo, hi) => lo + (hi - lo) * random(),
175
+ randint: n => Math.floor(random() * n),
176
+ normal: () => { const u = Math.max(random(), 1e-12), v = random(); return Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v); },
177
+ };
178
+ }
179
+
180
+ function softmax(arr) {
181
+ let m = -Infinity; for (const v of arr) if (v > m) m = v;
182
+ const e = Array.from(arr, v => Math.exp(v - m)); const s = e.reduce((a, b) => a + b, 0);
183
+ return e.map(v => v / s);
184
+ }
185
+
186
+ function argmax(arr) { let k = 0; for (let i = 1; i < arr.length; i++) if (arr[i] > arr[k]) k = i; return k; }
187
+
188
+ // Deep comparison used by parity tests: numbers within tol, everything else exact.
189
+ function compare(got, want, tol, path, acc) {
190
+ acc = acc || { maxDiff: 0, mismatches: [] };
191
+ path = path || "$";
192
+ if (typeof want === "number") {
193
+ const d = Math.abs(Number(got) - want);
194
+ if (!(d <= tol * Math.max(1, Math.abs(want)))) acc.mismatches.push(path + ": got " + got + " want " + want);
195
+ if (isFinite(d)) acc.maxDiff = Math.max(acc.maxDiff, d);
196
+ } else if (Array.isArray(want)) {
197
+ if (!got || got.length !== want.length) acc.mismatches.push(path + ": length " + (got && got.length) + " vs " + want.length);
198
+ else for (let i = 0; i < want.length; i++) compare(got[i], want[i], tol, path + "[" + i + "]", acc);
199
+ } else if (want && typeof want === "object") {
200
+ for (const k in want) compare(got ? got[k] : undefined, want[k], tol, path + "." + k, acc);
201
+ } else if (got !== want) acc.mismatches.push(path + ": got " + got + " want " + want);
202
+ return acc;
203
+ }
204
+
205
+ function runParity(core, ctx, parity) {
206
+ const results = [];
207
+ let maxDiff = 0, failed = 0;
208
+ for (const c of parity.cases) {
209
+ const out = core.runCase(ctx, c.input);
210
+ const r = compare(out, c.expected, c.tol != null ? c.tol : parity.tol);
211
+ maxDiff = Math.max(maxDiff, r.maxDiff);
212
+ if (r.mismatches.length) failed++;
213
+ results.push({ name: c.name, maxDiff: r.maxDiff, mismatches: r.mismatches.slice(0, 5) });
214
+ }
215
+ return { cases: parity.cases.length, failed, maxDiff, results };
216
+ }
217
+
218
+ return { b64d, decode, loadModel, T, forward, mulberry32, softmax, argmax, compare, runParity };
219
+ });
nlui.js ADDED
@@ -0,0 +1,123 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /* nlui.js: small DOM and canvas helpers shared by Next Lab Spaces. */
2
+ (function (root) {
3
+ "use strict";
4
+ const $ = id => document.getElementById(id);
5
+ function h(tag, attrs, ...kids) {
6
+ const e = document.createElement(tag);
7
+ for (const k in attrs || {}) {
8
+ if (k === "class") e.className = attrs[k];
9
+ else if (k.startsWith("on")) e.addEventListener(k.slice(2), attrs[k]);
10
+ else if (k === "html") e.innerHTML = attrs[k];
11
+ else e.setAttribute(k, attrs[k]);
12
+ }
13
+ for (const c of kids.flat()) if (c != null) e.append(c.nodeType ? c : document.createTextNode(String(c)));
14
+ return e;
15
+ }
16
+ const css = name => getComputedStyle(document.documentElement).getPropertyValue(name).trim();
17
+ const fmt = (v, d = 3) => (v == null || !isFinite(v) ? "–" : Number(v).toFixed(d));
18
+
19
+ function grayImage(canvas, data, w, h, scale, opts) {
20
+ opts = opts || {};
21
+ canvas.width = w * scale; canvas.height = h * scale;
22
+ const ctx = canvas.getContext("2d");
23
+ const img = ctx.createImageData(w, h);
24
+ let lo = opts.lo != null ? opts.lo : 0, hi = opts.hi != null ? opts.hi : 1;
25
+ for (let i = 0; i < w * h; i++) {
26
+ const v = Math.max(0, Math.min(255, Math.round(255 * (data[i] - lo) / (hi - lo || 1))));
27
+ img.data[4 * i] = img.data[4 * i + 1] = img.data[4 * i + 2] = v; img.data[4 * i + 3] = 255;
28
+ }
29
+ const tmp = document.createElement("canvas"); tmp.width = w; tmp.height = h;
30
+ tmp.getContext("2d").putImageData(img, 0, 0);
31
+ ctx.imageSmoothingEnabled = false;
32
+ ctx.drawImage(tmp, 0, 0, w * scale, h * scale);
33
+ return ctx;
34
+ }
35
+
36
+ function rgbImage(canvas, data, w, h, scale) { // data: CHW in [0,1]
37
+ canvas.width = w * scale; canvas.height = h * scale;
38
+ const ctx = canvas.getContext("2d");
39
+ const img = ctx.createImageData(w, h), n = w * h;
40
+ for (let i = 0; i < n; i++) {
41
+ for (let c = 0; c < 3; c++) img.data[4 * i + c] = Math.max(0, Math.min(255, Math.round(255 * data[c * n + i])));
42
+ img.data[4 * i + 3] = 255;
43
+ }
44
+ const tmp = document.createElement("canvas"); tmp.width = w; tmp.height = h;
45
+ tmp.getContext("2d").putImageData(img, 0, 0);
46
+ ctx.imageSmoothingEnabled = false;
47
+ ctx.drawImage(tmp, 0, 0, w * scale, h * scale);
48
+ return ctx;
49
+ }
50
+
51
+ // series: [{name, color, points: [[x,y],...], dash}]
52
+ function lineChart(canvas, series, opts) {
53
+ opts = opts || {};
54
+ const W = canvas.clientWidth || 600, H = opts.height || 240, dpr = window.devicePixelRatio || 1;
55
+ canvas.width = W * dpr; canvas.height = H * dpr; canvas.style.height = H + "px";
56
+ const ctx = canvas.getContext("2d"); ctx.scale(dpr, dpr);
57
+ const m = { l: 46, r: 10, t: 10, b: 34 };
58
+ let xs = [], ys = [];
59
+ for (const s of series) for (const [x, y] of s.points) { if (isFinite(y)) { xs.push(x); ys.push(y); } }
60
+ if (opts.hlines) for (const hl of opts.hlines) ys.push(hl.y);
61
+ let x0 = opts.xmin != null ? opts.xmin : Math.min(...xs), x1 = opts.xmax != null ? opts.xmax : Math.max(...xs);
62
+ let y0 = opts.ymin != null ? opts.ymin : Math.min(...ys), y1 = opts.ymax != null ? opts.ymax : Math.max(...ys);
63
+ if (y1 - y0 < 1e-9) { y0 -= 1; y1 += 1; }
64
+ if (x1 - x0 < 1e-9) x1 = x0 + 1;
65
+ const X = x => m.l + (x - x0) / (x1 - x0) * (W - m.l - m.r);
66
+ const Y = y => H - m.b - (y - y0) / (y1 - y0) * (H - m.t - m.b);
67
+ ctx.font = "11px system-ui"; ctx.fillStyle = css("--muted"); ctx.strokeStyle = css("--line"); ctx.lineWidth = 1;
68
+ for (let i = 0; i <= 4; i++) {
69
+ const y = y0 + (y1 - y0) * i / 4;
70
+ ctx.beginPath(); ctx.moveTo(m.l, Y(y)); ctx.lineTo(W - m.r, Y(y)); ctx.stroke();
71
+ ctx.fillText(Math.abs(y) >= 100 ? y.toFixed(0) : y.toFixed(2), 4, Y(y) + 4);
72
+ }
73
+ for (let i = 0; i <= 4; i++) {
74
+ const x = x0 + (x1 - x0) * i / 4;
75
+ ctx.fillText(Math.abs(x) >= 100 || Number.isInteger(x) ? x.toFixed(0) : x.toFixed(1), X(x) - 8, H - m.b + 14);
76
+ }
77
+ if (opts.xlabel) ctx.fillText(opts.xlabel, W / 2 - 30, H - 4);
78
+ if (opts.hlines) for (const hl of opts.hlines) {
79
+ ctx.setLineDash([4, 4]); ctx.strokeStyle = hl.color || css("--muted");
80
+ ctx.beginPath(); ctx.moveTo(m.l, Y(hl.y)); ctx.lineTo(W - m.r, Y(hl.y)); ctx.stroke(); ctx.setLineDash([]);
81
+ }
82
+ for (const s of series) {
83
+ ctx.strokeStyle = s.color; ctx.lineWidth = s.width || 2; ctx.setLineDash(s.dash || []);
84
+ ctx.beginPath(); let first = true;
85
+ for (const [x, y] of s.points) { if (!isFinite(y)) continue; if (first) ctx.moveTo(X(x), Y(y)); else ctx.lineTo(X(x), Y(y)); first = false; }
86
+ ctx.stroke(); ctx.setLineDash([]);
87
+ }
88
+ return { X, Y, ctx };
89
+ }
90
+
91
+ function legend(el, series) {
92
+ el.innerHTML = "";
93
+ for (const s of series) el.append(h("span", { class: "small", style: "margin-right:12px;white-space:nowrap" },
94
+ h("span", { style: `display:inline-block;width:14px;height:3px;background:${s.color};vertical-align:middle;margin-right:4px` }), s.name));
95
+ }
96
+
97
+ async function loadAssets(core) {
98
+ const assets = {};
99
+ for (const name of core.ASSETS) {
100
+ const r = await fetch(name);
101
+ if (!r.ok) throw new Error("cannot load " + name);
102
+ assets[name] = await r.json();
103
+ }
104
+ return assets;
105
+ }
106
+
107
+ // Standard bootstrap: load assets, init core, expose __parity for automated tests.
108
+ async function boot(core, start) {
109
+ try {
110
+ const assets = await loadAssets(core);
111
+ const ctx = core.init(assets);
112
+ const parity = await (await fetch("parity_cases.json")).json();
113
+ root.__parity = () => NLRT.runParity(core, ctx, parity);
114
+ await start(ctx, assets);
115
+ root.__ready = true;
116
+ } catch (e) {
117
+ console.error(e); root.__error = String(e && e.stack || e);
118
+ const s = document.getElementById("status"); if (s) s.textContent = "Failed to start: " + e;
119
+ }
120
+ }
121
+
122
+ root.NLUI = { $, h, css, fmt, grayImage, rgbImage, lineChart, legend, boot };
123
+ })(typeof self !== "undefined" ? self : this);
parity_cases.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"tol": 1e-06, "cases": [{"name": "TR · kütüphane (yanıtlanabilir)", "input": {"question": "Kütüphanede kaç kitap var?", "document": "Kasaba kütüphanesi 1962 yılında açıldı. Binanın mimarı Selma Arı'dır. Kütüphanede bugün yaklaşık 40 bin kitap bulunuyor. Her yıl haziran ayında bir kitap şenliği düzenleniyor."}, "expected": {"spans": [[0, 39], [40, 69], [70, 120], [121, 175]], "probs": [0.2566700706421586, 0.011049804690258812, 0.7985120249293477, 0.057147024195103084], "answerable": 0.6524745300529133, "abstain": false, "evidence": [2], "bm25_abstain": true, "bm25_evidence": []}}, {"name": "TR · kütüphane (belgede yok, yanıltıcı isim)", "input": {"question": "Kütüphanenin ilk müdürü kimdi?", "document": "Kasaba kütüphanesi 1962 yılında açıldı. Binanın mimarı Selma Arı'dır. Kütüphanede bugün yaklaşık 40 bin kitap bulunuyor. Her yıl haziran ayında bir kitap şenliği düzenleniyor."}, "expected": {"spans": [[0, 39], [40, 69], [70, 120], [121, 175]], "probs": [0.39956163093304226, 0.04105134074939768, 0.22020908787535282, 0.031653588894207385], "answerable": 0.27141084178818015, "abstain": true, "evidence": [], "bm25_abstain": true, "bm25_evidence": []}}, {"name": "TR · şirket (yanıltıcı yıl)", "input": {"question": "Şirketin 2020 yılındaki geliri ne kadardı?", "document": "Şirketin 2019 yılındaki geliri 4 milyon liraydı. 2020 yılında çalışan sayısı 120 kişiye çıktı. Merkez ofis 2021'de İzmir'e taşındı."}, "expected": {"spans": [[0, 48], [49, 94], [95, 131]], "probs": [0.4564306278239845, 0.31888943762740635, 0.013067418873463497], "answerable": 0.28409286741619816, "abstain": true, "evidence": [], "bm25_abstain": true, "bm25_evidence": []}}, {"name": "TR · Ay (yanıtlanabilir)", "input": {"question": "Ay'a ilk insanlı iniş ne zaman yapıldı?", "document": "Ay, Dünya'nın tek doğal uydusudur. Ay'ın Dünya'ya ortalama uzaklığı yaklaşık 384 bin kilometredir. Ay'ın yüzeyinde atmosfer neredeyse yoktur. Ay'a ilk insanlı iniş 1969 yılında gerçekleşti."}, "expected": {"spans": [[0, 34], [35, 98], [99, 141], [142, 189]], "probs": [0.0016932091875170264, 0.009253651448440165, 0.00418795073525174, 0.8713078765198486], "answerable": 0.6624891481709587, "abstain": false, "evidence": [3], "bm25_abstain": false, "bm25_evidence": [3]}}, {"name": "EN · bridge (answerable)", "input": {"question": "How long is the main span?", "document": "The river crossing opened to traffic in 1932. Its main span is 503 metres long. The bridge was repainted in 2008 using a grey anti-corrosion coating. About 160,000 vehicles cross it every day."}, "expected": {"spans": [[0, 45], [46, 79], [80, 149], [150, 192]], "probs": [0.05997950763214863, 0.8802141450863382, 0.017954486432607074, 0.0014277780143912744], "answerable": 0.756120137796548, "abstain": false, "evidence": [1], "bm25_abstain": false, "bm25_evidence": [1]}}, {"name": "EN · bridge (not in the document)", "input": {"question": "How much did the 2008 repainting cost?", "document": "The river crossing opened to traffic in 1932. Its main span is 503 metres long. The bridge was repainted in 2008 using a grey anti-corrosion coating. About 160,000 vehicles cross it every day."}, "expected": {"spans": [[0, 45], [46, 79], [80, 149], [150, 192]], "probs": [0.0255551200999423, 0.055079598427895914, 0.8414629889618448, 0.036461816877568484], "answerable": 0.5633153393855327, "abstain": true, "evidence": [], "bm25_abstain": true, "bm25_evidence": []}}, {"name": "XQuAD-tr örneği", "input": {"question": "Panthers savunması kaç sayı bırakmıştır?", "document": "Panthers savunması ligdeki derecesi altıncı sırada olarak sadece 308 sayı bıraktı, aynı zamanda NFL'de 24 topu kapma ile ligin başını çekmesi ve dört Pro Bowl seçmeleri ile övünç duydu. Pro Bowl orta çizgi savunmacısı Kawann Short, takımı 11'le birlikte sack ederken, aynı zamanda üç oyuncuyu düşürme ve iki yeniden savunmaya zorladı. Yan hakem Mario Addison, 6½ sack ekledi. Panthers hattında ayrıca, sadece 9 başlangıçta 5 sack eden uç çizgi savunmacısı Kony Ealy ile birlikte 136 kez ile NFL'nin aktif kariyer sack lideri ve 5 kez profesyonel bir top atıcısı olan Jared Allen öne çıkmaktadır. Arkalarında, Panthers'lerin üç adet ikinci hat savunma oyuncusundan ikisi de Pro Bowl'da oynamak için seçildi: Thomas Davis ve Luke Kuechly. Davis, dördünde mecburi oyuncuyu düşürme ve dört topu kapma olmak üzere 5½ sack etmeyi bir araya getirirken, Kuechly top çalmalarda (118) iki mecburi oyuncuyu düşürme ve kendi başına dört pasını kesme ile takıma öncülük etti. Carolina’nın ikincisi olarak gösterilen Pro Bowl güvenliği için kariyerinin en yükseği olarak 7 top kapma ile takımı sürükleyen Kurt Coleman’a ve 88 top çalma sayısıyla ve Pro Bowl köşe savunmacısı sezon boyunca bir kapalı köşe içinde gelişen ikisi kale çizgisini geçişten dönen dört top kapma geliştiren Josh Norman'a rol verdi."}, "expected": {"spans": [[0, 185], [186, 334], [335, 375], [376, 595], [596, 736], [737, 962], [963, 1292]], "probs": [0.9517184797981059, 0.02769000293632277, 0.010626398780669465, 0.010779909956745772, 0.002902160539948373, 0.0026929331031505755, 0.0016535220987547348], "answerable": 0.7965940107656674, "abstain": false, "evidence": [0], "bm25_abstain": false, "bm25_evidence": [0]}}, {"name": "XQuAD-tr · başka paragrafa sorulmuş soru", "input": {"question": "Panthers savunması kaç sayı bırakmıştır?", "document": "Broncos, 23–16 arasındaki bölge turunda Pittsburgh Steelers'ı, maçın son üç dakikasında 11 sayı alarak mağlup etti. Daha sonra, New England'ın 2 puanlık dönüşüm girişimini saat üzerinde 17 saniye kala bir pası keserek, AFC Şampiyonluk Oyununda Super Bowl'u savunan XLIX şampiyonu New England Patriots'u 20–18 yendi. Manning''in sezon boyunca topu kapmalarla ilgili sorunlarına rağmen, iki rövanş maçında hiç atmadı."}, "expected": {"spans": [[0, 115], [116, 315], [316, 415]], "probs": [0.42594753613939734, 0.5448627041677241, 0.006219471489626283], "answerable": 0.521832748704309, "abstain": true, "evidence": [], "bm25_abstain": true, "bm25_evidence": []}}, {"name": "SQuAD 2.0 dev · unanswerable", "input": {"question": "Who gave their name to Normandy in the 1000's and 1100's", "document": "The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants would gradually merge with the Carolingian-based cultures of West Francia. The distinct cultural and ethnic identity of the Normans emerged initially in the first half of the 10th century, and it continued to evolve over the succeeding centuries."}, "expected": {"spans": [[0, 166], [167, 374], [375, 570], [571, 742]], "probs": [0.43339849125455665, 0.055537425857606224, 0.010991327906988676, 0.014505228061116832], "answerable": 0.18313970364344162, "abstain": true, "evidence": [], "bm25_abstain": false, "bm25_evidence": [0]}}]}
preview.png ADDED
style.css CHANGED
@@ -1,28 +1,55 @@
1
- body {
2
- padding: 2rem;
3
- font-family: -apple-system, BlinkMacSystemFont, "Arial", sans-serif;
 
4
  }
5
-
6
- h1 {
7
- font-size: 16px;
8
- margin-top: 0;
9
- }
10
-
11
- p {
12
- color: rgb(107, 114, 128);
13
- font-size: 15px;
14
- margin-bottom: 10px;
15
- margin-top: 5px;
16
- }
17
-
18
- .card {
19
- max-width: 620px;
20
- margin: 0 auto;
21
- padding: 16px;
22
- border: 1px solid lightgray;
23
- border-radius: 16px;
24
- }
25
-
26
- .card p:last-child {
27
- margin-bottom: 0;
28
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ :root {
2
+ --bg: #f7f7f5; --panel: #ffffff; --ink: #1d1f22; --muted: #5f6670; --line: #dcdfe3;
3
+ --accent: #0b6e4f; --accent-soft: #e3f1ec; --warn: #a4471b; --warn-soft: #fbeee6;
4
+ --blue: #2f5fb3; --gold: #b07d12; --red: #c0392b;
5
  }
6
+ @media (prefers-color-scheme: dark) {
7
+ :root {
8
+ --bg: #15171a; --panel: #1e2125; --ink: #eceef0; --muted: #a5acb5; --line: #33383e;
9
+ --accent: #4cc39a; --accent-soft: #1b3229; --warn: #f0a070; --warn-soft: #3a2519;
10
+ --blue: #7ea6f0; --gold: #e0b44c; --red: #f07a6c;
11
+ }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
  }
13
+ * { box-sizing: border-box; }
14
+ html, body { margin: 0; padding: 0; background: var(--bg); color: var(--ink); }
15
+ body { font: 15px/1.5 system-ui, -apple-system, "Segoe UI", Roboto, sans-serif; }
16
+ main { max-width: 1040px; margin: 0 auto; padding: 16px; }
17
+ header h1 { font-size: 1.6rem; margin: 4px 0 2px; letter-spacing: -0.01em; }
18
+ header p { margin: 0 0 8px; color: var(--muted); max-width: 70ch; }
19
+ .badges { display: flex; flex-wrap: wrap; gap: 6px; margin: 6px 0 12px; }
20
+ .badge { font-size: 12px; padding: 2px 8px; border-radius: 999px; background: var(--accent-soft); color: var(--accent); border: 1px solid var(--line); }
21
+ .badge.warn { background: var(--warn-soft); color: var(--warn); }
22
+ .grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(300px, 1fr)); gap: 12px; }
23
+ .card { background: var(--panel); border: 1px solid var(--line); border-radius: 10px; padding: 12px 14px; min-width: 0; }
24
+ .card h2 { font-size: 1.02rem; margin: 0 0 8px; }
25
+ .card h3 { font-size: .95rem; margin: 10px 0 6px; }
26
+ label { display: block; font-size: 13px; color: var(--muted); margin: 8px 0 3px; }
27
+ input[type=text], textarea, select { width: 100%; font: inherit; padding: 7px 9px; border-radius: 8px; border: 1px solid var(--line); background: var(--bg); color: var(--ink); }
28
+ textarea { min-height: 110px; resize: vertical; }
29
+ input[type=range] { width: 100%; }
30
+ button { font: inherit; padding: 7px 12px; border-radius: 8px; border: 1px solid var(--accent); background: var(--accent); color: #fff; cursor: pointer; }
31
+ button.ghost { background: transparent; color: var(--accent); }
32
+ button:disabled { opacity: .5; cursor: default; }
33
+ .row { display: flex; flex-wrap: wrap; gap: 8px; align-items: center; }
34
+ canvas { max-width: 100%; height: auto; display: block; border-radius: 6px; }
35
+ .card > canvas { width: 100%; }
36
+ .muted { color: var(--muted); }
37
+ .small { font-size: 13px; }
38
+ .status { font-size: 13px; color: var(--muted); min-height: 1.4em; }
39
+ table { border-collapse: collapse; width: 100%; font-size: 13px; }
40
+ th, td { text-align: left; padding: 4px 6px; border-bottom: 1px solid var(--line); vertical-align: top; }
41
+ th { color: var(--muted); font-weight: 600; }
42
+ td.num, th.num { text-align: right; font-variant-numeric: tabular-nums; }
43
+ .scroll { overflow-x: auto; }
44
+ mark { background: #ffe58a; color: #1d1f22; padding: 0 2px; border-radius: 3px; }
45
+ mark.weak { background: #fff3c4; }
46
+ video { width: 100%; border-radius: 8px; }
47
+ footer { margin: 18px 0 8px; font-size: 13px; color: var(--muted); }
48
+ a { color: var(--blue); }
49
+ .chips { display: flex; flex-wrap: wrap; gap: 6px; }
50
+ .chip { border: 1px solid var(--line); border-radius: 999px; padding: 3px 10px; background: var(--bg); font-size: 13px; cursor: pointer; color: var(--ink); }
51
+ .chip.on { border-color: var(--accent); background: var(--accent-soft); }
52
+ .kpi { display: grid; grid-template-columns: repeat(auto-fit, minmax(120px, 1fr)); gap: 8px; }
53
+ .kpi div { background: var(--bg); border: 1px solid var(--line); border-radius: 8px; padding: 6px 8px; }
54
+ .kpi b { display: block; font-size: 1.15rem; }
55
+ .kpi span { font-size: 12px; color: var(--muted); }