Add comparison with other models, speed charts and graphs

.gitattributes

@@ -35,3 +35,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 gguf/tiny-search-reader-60M-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 gguf/tiny-search-reader-60M-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+assets/accuracy_vs_size.png filter=lfs diff=lfs merge=lfs -text
+assets/speed.png filter=lfs diff=lfs merge=lfs -text
+assets/trick_questions.png filter=lfs diff=lfs merge=lfs -text
+assets/versions.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -24,6 +24,12 @@ It does not search and it is not a chatbot. Your app runs the search and passes
 
 On 258 real Google questions it is right 79.8% of the time (206/258). About one answer in five is wrong.
 
+That is level with Qwen3-0.6B (207/258), a model ten times its size, and on a laptop CPU it answers about 5 times
+faster. A SmolLM2-135M fine-tuned on the same training data scores a little higher (214/258). Details are in
+[Compared with other models](#compared-with-other-models).
+
+![Accuracy against model size](https://huggingface.co/ahkamboh/tiny-search-reader-60M/resolve/main/assets/accuracy_vs_size.png)
+
 ## What it does
 
 Made-up results, real output (the demo in `searchreader.py`):
@@ -47,6 +53,7 @@ result names a recent winner. On the tests it catches 41 of 50 trick questions (
 | `gguf/tiny-search-reader-60M-Q4_K_M.gguf` | 41.0 MB | for llama.cpp; smaller, slightly different answers |
 | `searchreader.py` | | prompt builder and answer helpers for transformers and llama-cpp-python |
 | `eval/` | | the test questions, gold answers, this model's answers, grading code and full results |
+| `assets/` | | the charts on this page and `make_charts.py`, which draws them |
 | `LICENSE` | | Apache-2.0 |
 
 ## How to run
@@ -210,6 +217,8 @@ Earlier versions from this project, same tests (Colab GPU runs, bf16):
 | v3a | 77.1% (199) | 74.7% | 65.0% |
 | v3.1a | 79.8% (206) | 75.9% | 61.7% |
 
+![Accuracy of each version](https://huggingface.co/ahkamboh/tiny-search-reader-60M/resolve/main/assets/versions.png)
+
 Paired tests (McNemar): against v2a, v3.1a fixes 17 google_big questions and breaks 6 (p=0.035), and on the "latest"
 questions fixes 13 and breaks 1 (p=0.002). Against v3a it fixes 22 and breaks 15 of the 258 (p=0.32, CPU fp32), so
 that gain is within noise.
@@ -239,6 +248,63 @@ Caveats:
 - About a quarter of the test questions (65 of 258) are about Pakistan.
 - Nothing was measured on a phone.
 
+## Compared with other models
+
+Same 258 questions and the same grading, all on Colab GPU in bf16 with greedy decoding.
+
+- Zero-shot models got the question and the numbered results in their own chat template (Qwen3 with thinking off),
+  with one instruction: "Answer the question using only the numbered search results below. Reply with just the
+  answer in a few words. If the results do not answer the question, reply exactly: not found".
+- The two extractive models (deepset's SQuAD 2.0 readers) pick an answer span from the five results.
+- SmolLM2-135M (fine-tuned) is the base SmolLM2-135M trained for 2 epochs on exactly the rows this model was
+  trained on.
+
+The last column compares each model with this one question by question: questions only that model got right
+against questions only this model got right, with the exact McNemar test (p below 0.05 means a real difference).
+
+| model | how it was run | parameters | all 258 | trick questions caught (of 50) | "latest" questions (of 60) | only it right vs only this model right |
+|---|---|---:|---:|---:|---:|---|
+| Qwen3-4B | zero-shot | 4.02B | 91.5% (236) | 44 | 52 | 42 vs 12, p<0.001 |
+| Qwen3-1.7B | zero-shot | 1.72B | 84.5% (218) | 26 | 54 | 39 vs 27, p=0.175 |
+| SmolLM2-135M, fine-tuned on this model's data | fine-tuned | 135M | 82.9% (214) | 32 | 47 | 33 vs 25, p=0.358 |
+| Qwen3-0.6B | zero-shot | 596M | 80.2% (207) | 30 | 45 | 37 vs 36, p=1.000 |
+| **tiny-search-reader-60M** | trained for this task | 62M | **79.8% (206)** | **41** | 37 | |
+| RoBERTa-base SQuAD2 | extractive | 124M | 71.3% (184) | 29 | 39 | 26 vs 48, p=0.014 |
+| MiniLM SQuAD2 | extractive | 33M | 69.4% (179) | 27 | 35 | 22 vs 49, p=0.002 |
+| SmolLM2-360M-Instruct | zero-shot | 362M | 65.9% (170) | 0 | 44 | 25 vs 61, p<0.001 |
+| Qwen2.5-0.5B-Instruct | zero-shot | 494M | 65.1% (168) | 0 | 40 | 26 vs 64, p<0.001 |
+| SmolLM2-135M-Instruct | zero-shot | 135M | 59.7% (154) | 0 | 40 | 21 vs 73, p<0.001 |
+
+What the table shows:
+
+- It is clearly better than both SQuAD 2.0 readers and the three instruct models below 0.6B. It ties Qwen3-0.6B
+  (37 questions against 36). Qwen3-1.7B gets 12 more right, which is not a significant gap on 258 questions.
+  Qwen3-4B, 65 times larger, is clearly better.
+- It catches 41 of the 50 trick questions, more than every model here except Qwen3-4B (44). The three instruct
+  models below 0.6B never replied `not found`.
+- It is weak on the 60 "latest / current / most recent" questions: 37 right, fewer than every generative model
+  tested. Qwen3-1.7B (54) and Qwen3-4B (52) are significantly better there (p of 0.001 or less).
+- SmolLM2-135M fine-tuned on the same rows scores 214 against 206. That gap is not significant (p=0.358), but on the
+  latest questions it is (47 against 37, p=0.021), and it catches fewer trick questions (32 against 41). It has 2.2
+  times the parameters, and its base was pretrained on about 2 trillion tokens, against 10 billion here. So most of
+  the result comes from the training data, and a base model with more pretraining makes better use of it.
+
+![Trick questions caught](https://huggingface.co/ahkamboh/tiny-search-reader-60M/resolve/main/assets/trick_questions.png)
+
+![Accuracy by question kind](https://huggingface.co/ahkamboh/tiny-search-reader-60M/resolve/main/assets/by_kind.png)
+
+Caveats for this comparison:
+
+- The zero-shot models were pretrained on the open web and may know some answers without reading the results.
+  This model's training held out every test question and topic.
+- One fixed prompt was used for every zero-shot model. Another prompt could raise or lower their scores.
+- The Qwen and SmolLM2 model cards suggest sampling; greedy decoding was used for every model so the run is
+  repeatable.
+- google25 is this model's dev set, and google_fresh and google_big guided the v3.1 data (see the caveats above),
+  which favours this model a little.
+
+Per-set scores, google_big by kind and every paired test are in [`eval/results.md`](eval/results.md).
+
 ## Phone files
 
 | file | size | all 258 (llama.cpp, CPU) | answers same as fp32 | 449-token prompt + 24 tokens | average test question | peak RSS |
@@ -252,6 +318,40 @@ roughly 0.5 to 0.9 s per question; that is an estimate, not a measurement. Q8_0
 difference matters: it gave the same answers under Homebrew llama.cpp and llama-cpp-python 0.3.36, while Q4_K_M
 changed 4 of 258 answers between the two.
 
+### Speed in tokens per second
+
+Measured with llama.cpp on an Apple M1 Pro, CPU only (no GPU, no Metal), on a real 449-token test prompt plus a
+24-token answer, median of 5 runs. "Reading" is the prompt pass, "writing" is generating the answer.
+
+| file | threads | reading (tokens/s) | writing (tokens/s) | 449-token prompt + 24-token answer |
+|---|---:|---:|---:|---:|
+| `tiny-search-reader-60M-Q8_0.gguf` | 1 | 558 | 135 | 982 ms |
+| `tiny-search-reader-60M-Q8_0.gguf` | 2 | 1,100 | 210 | 522 ms |
+| `tiny-search-reader-60M-Q8_0.gguf` | 4 | 1,884 | 305 | 317 ms |
+| `tiny-search-reader-60M-Q4_K_M.gguf` | 1 | 455 | 154 | 1,142 ms |
+| `tiny-search-reader-60M-Q4_K_M.gguf` | 2 | 894 | 244 | 601 ms |
+| `tiny-search-reader-60M-Q4_K_M.gguf` | 4 | 1,559 | 362 | 354 ms |
+
+A typical test question has about 317 prompt tokens and a 5-token answer, so most of the time goes into reading the
+results: 168 ms per question with Q8_0 at 4 threads. These numbers come from Homebrew's prebuilt llama.cpp; in one
+check a llama.cpp built on this machine was about twice as fast, so treat them as a floor for this Mac.
+
+### Compared with Qwen3-0.6B
+
+Same Mac, same llama.cpp build (Homebrew build 11146), `llama-bench -p 449 -n 24 -r 5 -ngl 0` (all layers on the CPU),
+both files Q8_0. llama-bench measures writing from an empty context, so its writing numbers are higher than in the
+table above, where the answer is written after the 449-token prompt.
+
+| model | file | parameters | reading (tokens/s), 1 / 4 threads | writing (tokens/s), 1 / 4 threads | 449-token prompt + 24-token answer, 4 threads |
+|---|---:|---:|---:|---:|---:|
+| tiny-search-reader-60M | 70 MB | 62M | 575 / 1,900 | 310 / 561 | 0.28 s |
+| Qwen3-0.6B | 634 MB | 596M | 89 / 340 | 82 / 132 | 1.50 s |
+
+On this machine it answers about 5 times faster than Qwen3-0.6B and its file is 9 times smaller. Accuracy against
+Qwen3-0.6B is in [Compared with other models](#compared-with-other-models).
+
+![Speed against Qwen3-0.6B](https://huggingface.co/ahkamboh/tiny-search-reader-60M/resolve/main/assets/speed.png)
+
 ## Training
 
 ### Architecture
@@ -318,7 +418,8 @@ Synthetic categories:
   `not found` once and the other five times gave a nearby time or number, such as `8` for "How many books does the
   Harbor Street library have?" from a result saying the library is open from 8 am to 8 pm.
 - "Latest" and "current" answers depend on the dates in the results. The model does not know today's date; it picks
-  the result that looks newest. Latest-version (6/14) and most-recent-winner (14/26) questions are its weakest kinds.
+  the result that looks newest. Latest-version (6/14) and most-recent-winner (14/26) questions are its weakest kinds, and every other
+  generative model tested did better on the 60 "latest" questions.
 - 512 tokens of context, about five Google-length results.
 - No chat and no instructions: one question in, a few words out, in the prompt format above only.
 

assets/make_charts.py

@@ -0,0 +1,243 @@
+"""Model-card charts for tiny-search-reader-60M, in the style of the Hugging Face blog charts: warm page, bold rounded
+title, grey subtitle, a white rounded card with a soft shadow, a mono source line. Every number is copied from
+runs/baselines-summary.txt (4 Oct 2026) or the README tables; nothing is computed here except percentages."""
+import os
+import sys
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+from matplotlib import font_manager as fm
+from matplotlib.lines import Line2D
+from matplotlib.patches import FancyBboxPatch
+from matplotlib.ticker import FixedLocator, FuncFormatter, NullLocator
+
+OUT = sys.argv[1] if len(sys.argv) > 1 else "assets"
+os.makedirs(OUT, exist_ok=True)
+
+TITLE = fm.FontProperties(fname="/System/Library/Fonts/Supplemental/Arial Rounded Bold.ttf")
+for f in ("/System/Library/Fonts/Avenir Next.ttc", "/System/Library/Fonts/Menlo.ttc"):
+    fm.fontManager.addfont(f)
+BODY, MONO = "Avenir Next", "Menlo"
+plt.rcParams.update({"font.family": BODY, "font.size": 11, "axes.unicode_minus": False})
+
+PAGE, CARD, EDGE = "#F7F5EF", "#FFFFFF", "#E6E1D6"
+INK, GREY, LIGHT, GRID = "#1F1D1A", "#77726A", "#B9B3A8", "#EEEAE2"
+OURS, OURS_SOFT = "#FF9D00", "#FFD9A0"
+QWEN, OTHER, EXTRACT = "#5B6B8C", "#A9A396", "#8FA7A0"
+W, H, DPI = 10.0, 6.0, 170
+SOURCE = "source: 258 real Google questions with their top results · greedy decoding · 4 Oct 2026"
+
+
+def card_figure(title, subtitle, source=SOURCE, legend=None):
+    """Page + title + card. Returns (fig, card box in figure fractions: x0, y0, x1, y1)."""
+    fig = plt.figure(figsize=(W, H), dpi=DPI, facecolor=PAGE)
+    fig.text(0.5 / W, 1 - 0.42 / H, title, fontproperties=TITLE, fontsize=21, color=INK, va="top")
+    fig.text(0.5 / W, 1 - 0.92 / H, subtitle, fontsize=12.5, color=GREY, va="top")
+    x0, y0, x1, y1 = 0.42, 0.62, W - 0.42, H - 1.42                   # inches
+    t = fig.dpi_scale_trans
+    for i, a in enumerate((0.035, 0.025, 0.015)):                       # soft shadow, three layers
+        g = 0.03 + 0.03 * i
+        fig.patches.append(FancyBboxPatch((x0 - g / 2, y0 - 0.06 - g / 2), x1 - x0 + g, y1 - y0 + g,
+                                          boxstyle="round,pad=0,rounding_size=0.2", transform=t,
+                                          facecolor="#3A2E1A", edgecolor="none", alpha=a, zorder=-3))
+    fig.patches.append(FancyBboxPatch((x0, y0), x1 - x0, y1 - y0, boxstyle="round,pad=0,rounding_size=0.18",
+                                      transform=t, facecolor=CARD, edgecolor=EDGE, linewidth=1.1, zorder=-2))
+    fig.text(0.5 / W, 0.24 / H, source, fontfamily=MONO, fontsize=8.6, color=GREY, va="center")
+    if legend:
+        x = x1 - 0.25
+        for label, color, marker in reversed(legend):
+            tx = fig.text(x / W, (y1 - 0.32) / H, label, fontsize=10.5, color=INK, ha="right", va="center")
+            fig.canvas.draw()
+            wpx = tx.get_window_extent().width / DPI
+            m = {"●": "o", "◆": "D", "■": "s"}[marker]
+            fig.lines.append(Line2D([(x - wpx - 0.17) / W], [(y1 - 0.32) / H], transform=fig.transFigure,
+                                    marker=m, markersize=8 if m != "D" else 7, linestyle="none",
+                                    markerfacecolor="white" if m == "D" else color, markeredgecolor=color,
+                                    markeredgewidth=1.8 if m == "D" else 0))
+            x -= wpx + 0.62
+    return fig, (x0 / W, y0 / H, x1 / W, y1 / H)
+
+
+def style(ax, grid="y"):
+    ax.set_facecolor("none")
+    for s in ("top", "right", "left"):
+        ax.spines[s].set_visible(False)
+    ax.spines["bottom"].set_color(LIGHT)
+    ax.tick_params(colors=GREY, length=0, labelsize=10.5)
+    if grid:
+        ax.grid(axis=grid, color=GRID, linewidth=1)
+        ax.set_axisbelow(True)
+
+
+def save(fig, name):
+    path = os.path.join(OUT, name)
+    fig.savefig(path, dpi=DPI, facecolor=PAGE)
+    plt.close(fig)
+    print(path)
+
+
+# ---------- 1. accuracy against size ----------
+MODELS = [  # name, params (M), right of 258, traps caught of 50, kind
+    ("tiny-search-reader-60M", 62.0, 206, 41, "ours"),
+    ("SmolLM2-135M fine-tuned\non the same data", 134.5, 214, 32, "ft"),
+    ("SmolLM2-135M-Instruct", 134.5, 154, 0, "zs"),
+    ("SmolLM2-360M-Instruct", 361.8, 170, 0, "zs"),
+    ("Qwen2.5-0.5B-Instruct", 494.0, 168, 0, "zs"),
+    ("Qwen3-0.6B", 596.0, 207, 30, "qwen"),
+    ("Qwen3-1.7B", 1720.6, 218, 26, "qwen"),
+    ("Qwen3-4B", 4022.5, 236, 44, "qwen"),
+    ("MiniLM SQuAD2", 33.2, 179, 27, "ex"),
+    ("RoBERTa-base SQuAD2", 124.1, 184, 29, "ex"),
+]
+pct = lambda n, d=258: 100 * n / d
+
+fig, (cx0, cy0, cx1, cy1) = card_figure(
+    "Qwen3-0.6B accuracy at a tenth of the size",
+    "Right answers on 258 real Google questions, given the question and its search results",
+    legend=[("this model", OURS, "●"), ("trained on the same data", OURS, "◆"),
+            ("zero-shot LLM", QWEN, "●"), ("extractive QA", EXTRACT, "■")])
+ax = fig.add_axes([cx0 + 0.075, cy0 + 0.12, cx1 - cx0 - 0.11, cy1 - cy0 - 0.25])
+style(ax, grid="both")
+ax.set_xscale("log")
+ax.set_xlim(24, 6500)
+ax.set_ylim(55, 95)
+ax.xaxis.set_major_locator(FixedLocator([30, 60, 100, 300, 600, 1000, 3000]))
+ax.xaxis.set_minor_locator(NullLocator())
+ax.xaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v / 1000:g}B" if v >= 1000 else f"{v:g}M"))
+ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:g}%"))
+ax.set_xlabel("parameters (log scale)", color=GREY, fontsize=10.5, labelpad=6)
+q = [m for m in MODELS if m[4] == "qwen"]
+ax.plot([m[1] for m in q], [pct(m[2]) for m in q], color=QWEN, alpha=0.35, linewidth=2, zorder=2)
+LABEL_POS = {  # offsets in points, alignment
+    "tiny-search-reader-60M": ((0, 17), "center"), "SmolLM2-135M fine-tuned\non the same data": ((10, 4), "left"),
+    "SmolLM2-135M-Instruct": ((9, -3), "left"), "SmolLM2-360M-Instruct": ((8, 6), "left"),
+    "Qwen2.5-0.5B-Instruct": ((8, -12), "left"), "Qwen3-0.6B": ((9, -8), "left"), "Qwen3-1.7B": ((-9, 7), "right"),
+    "Qwen3-4B": ((-9, 6), "right"), "MiniLM SQuAD2": ((9, -4), "left"), "RoBERTa-base SQuAD2": ((9, 2), "left"),
+}
+for name, p, right, _, kind in MODELS:
+    y = pct(right)
+    if kind == "ours":
+        ax.scatter(p, y, s=340, color=OURS, alpha=0.25, zorder=3, linewidths=0)
+        ax.scatter(p, y, s=120, color=OURS, zorder=4, edgecolors="white", linewidths=1.5)
+    elif kind == "ft":
+        ax.scatter(p, y, s=110, marker="D", color="white", edgecolors=OURS, linewidths=2, zorder=4)
+    elif kind == "ex":
+        ax.scatter(p, y, s=80, marker="s", color=EXTRACT, zorder=4, edgecolors="white", linewidths=1)
+    else:
+        ax.scatter(p, y, s=85, color=QWEN if kind == "qwen" else OTHER, zorder=4, edgecolors="white", linewidths=1)
+    (dx, dy), ha = LABEL_POS[name]
+    shown = "this model" if kind == "ours" else name
+    ax.annotate(f"{shown}  {y:.1f}%", (p, y), xytext=(dx, dy), textcoords="offset points", ha=ha, va="center",
+                fontsize=10 if kind != "ours" else 11, color=INK if kind in ("ours", "ft") else GREY,
+                fontweight="bold" if kind == "ours" else "normal")
+ax.annotate("", xy=(560, pct(207)), xytext=(70, pct(206)), arrowprops=dict(arrowstyle="-", color=OURS,
+            linestyle=(0, (3, 3)), linewidth=1.3), zorder=2)
+ax.text(230, pct(206) - 1.0, "same score, 10x fewer parameters", color="#C77700", fontsize=10, ha="center",
+        va="top")
+save(fig, "accuracy_vs_size.png")
+
+# ---------- 2. trick questions ----------
+traps = sorted(MODELS, key=lambda m: (m[3], m[2]))
+fig, (cx0, cy0, cx1, cy1) = card_figure(
+    "Says \"not found\" when the answer is not there",
+    "Trick questions caught, out of 50: the search results do not hold the answer")
+ax = fig.add_axes([cx0 + 0.27, cy0 + 0.06, cx1 - cx0 - 0.33, cy1 - cy0 - 0.11])
+style(ax, grid="x")
+ax.spines["bottom"].set_visible(False)
+names = ["SmolLM2-135M fine-tuned (same data)" if m[4] == "ft" else m[0] for m in traps]
+colors = [OURS if m[4] == "ours" else OURS_SOFT if m[4] == "ft" else QWEN if m[4] == "qwen"
+          else EXTRACT if m[4] == "ex" else OTHER for m in traps]
+bars = ax.barh(range(len(traps)), [m[3] for m in traps], color=colors, height=0.62, zorder=3)
+ax.set_yticks(range(len(traps)), names, fontsize=10.5, color=INK)
+for t, m in zip(ax.get_yticklabels(), traps):
+    if m[4] == "ours":
+        t.set_fontweight("bold")
+ax.set_xlim(0, 50)
+ax.xaxis.set_major_locator(FixedLocator([0, 10, 20, 30, 40, 50]))
+for i, m in enumerate(traps):
+    label = f"{m[3]}" if m[3] else "0  (always answers something)"
+    ax.text(m[3] + 0.7, i, label, va="center", fontsize=10.5, color=INK if m[3] else GREY,
+            fontweight="bold" if m[4] == "ours" else "normal")
+save(fig, "trick_questions.png")
+
+# ---------- 3. speed ----------
+fig, (cx0, cy0, cx1, cy1) = card_figure(
+    "About 5x faster than Qwen3-0.6B, from a 9x smaller file",
+    "llama.cpp on an Apple M1 Pro, CPU only, 4 threads, both files Q8_0",
+    source="source: llama-bench -p 449 -n 24 -r 5 -ngl 0 · Homebrew llama.cpp build 11146 · 4 Oct 2026",
+    legend=[("tiny-search-reader-60M", OURS, "■"), ("Qwen3-0.6B", QWEN, "■")])
+PANELS = [("reading the results", "tokens per second, higher is better", 1900, 340, "{:,.0f}"),
+          ("writing the answer", "tokens per second, higher is better", 561, 132, "{:,.0f}"),
+          ("one question, 449 + 24 tokens", "seconds, lower is better", 0.28, 1.50, "{:.2f} s"),
+          ("model file", "megabytes, lower is better", 70, 634, "{:,.0f} MB")]
+pw, ph = (cx1 - cx0 - 0.08) / 2, (cy1 - cy0 - 0.13) / 2
+for i, (head, unit, ours, qwen, fmt) in enumerate(PANELS):
+    col, row = i % 2, i // 2
+    ax = fig.add_axes([cx0 + 0.04 + col * pw + 0.012, cy0 + 0.03 + (1 - row) * ph, pw - 0.05, ph - 0.03])
+    style(ax, grid=None)
+    ax.spines["bottom"].set_visible(False)
+    ax.set_xticks([])
+    ax.set_yticks([])
+    top = max(ours, qwen) * 1.32
+    ax.set_xlim(0, top)
+    ax.barh([1, 0], [ours, qwen], color=[OURS, QWEN], height=0.62)
+    for y, v in ((1, ours), (0, qwen)):
+        ax.text(v + top * 0.015, y, fmt.format(v), va="center", fontsize=12, color=INK,
+                fontweight="bold" if y == 1 else "normal")
+    ax.set_ylim(-0.5, 2.5)
+    ax.text(0, 2.2, head, fontsize=12.5, color=INK, fontweight="bold", va="center")
+    ax.text(0, 1.62, unit, fontsize=9.5, color=GREY, va="center")
+save(fig, "speed.png")
+
+# ---------- 4. versions ----------
+fig, (cx0, cy0, cx1, cy1) = card_figure(
+    "Four versions, 69% to 80%",
+    "Each version of this model on the same 258 questions (Colab GPU runs, bf16)",
+    legend=[("all 258 questions", OURS, "●"), ("the 60 latest / current / most recent questions", QWEN, "●")])
+ax = fig.add_axes([cx0 + 0.07, cy0 + 0.1, cx1 - cx0 - 0.2, cy1 - cy0 - 0.23])
+style(ax)
+xs = [0, 1, 2, 3]
+ax.set_xticks(xs, ["v1.1a", "v2a", "v3a", "v3.1a\n(this release)"], fontsize=11, color=INK)
+ax.set_xlim(-0.35, 3.35)
+ax.set_ylim(30, 95)
+ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:g}%"))
+allv, latest = [69.4, 73.3, 77.1, 79.8], [41.7, 65.0, 61.7]
+ax.plot(xs, allv, color=OURS, linewidth=3, marker="o", markersize=9, markeredgecolor="white", markeredgewidth=1.5,
+        zorder=4)
+ax.plot(xs[1:], latest, color=QWEN, linewidth=3, marker="o", markersize=9, markeredgecolor="white",
+        markeredgewidth=1.5, zorder=4)
+for x, v in zip(xs, allv):
+    ax.text(x, v - 4.6, f"{v:.1f}%", ha="center", fontsize=11, color=INK, fontweight="bold" if x == 3 else "normal")
+for x, v in zip(xs[1:], latest):
+    ax.text(x, v - 5.2, f"{v:.1f}%", ha="center", fontsize=11, color=QWEN)
+ax.axhline(80.2, color=LIGHT, linewidth=1.2, linestyle=(0, (4, 3)), zorder=2)
+ax.text(-0.3, 80.9, "Qwen3-0.6B, zero-shot: 80.2%", ha="left", va="bottom", fontsize=9.5, color=GREY)
+ax.text(0, 36, "not measured", ha="center", fontsize=9.5, color=LIGHT)
+save(fig, "versions.png")
+
+# ---------- 5. by kind ----------
+KINDS = ["fact", "unit", "trick\n(\"not found\")", "live value", "latest\nversion", "most recent\nwinner"]
+N = [34, 25, 28, 35, 14, 26]
+ROWS = [("tiny-search-reader-60M", [30, 20, 24, 29, 6, 14], OURS),
+        ("SmolLM2-135M fine-tuned on the same data", [31, 23, 21, 32, 9, 18], OURS_SOFT),
+        ("Qwen3-0.6B, zero-shot", [31, 23, 19, 31, 11, 16], QWEN)]
+fig, (cx0, cy0, cx1, cy1) = card_figure(
+    "Weakest on \"latest\" questions",
+    "google_big (162 questions) by kind, % right",
+    legend=[(n.split(",")[0].replace(" fine-tuned on the same data", " fine-tuned"), c, "■") for n, _, c in ROWS])
+ax = fig.add_axes([cx0 + 0.07, cy0 + 0.12, cx1 - cx0 - 0.1, cy1 - cy0 - 0.25])
+style(ax)
+ax.set_ylim(0, 105)
+ax.yaxis.set_major_locator(FixedLocator([0, 25, 50, 75, 100]))
+ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:g}%"))
+bw = 0.26
+for j, (name, right, color) in enumerate(ROWS):
+    xs = [i + (j - 1) * bw for i in range(len(KINDS))]
+    vals = [100 * r / n for r, n in zip(right, N)]
+    ax.bar(xs, vals, width=bw * 0.92, color=color, zorder=3)
+    for x, v, r, n in zip(xs, vals, right, N):
+        ax.text(x, v + 1.5, f"{r}", ha="center", fontsize=8.8, color=INK if j == 0 else GREY,
+                fontweight="bold" if j == 0 else "normal")
+ax.set_xticks(range(len(KINDS)), [f"{k}\nof {n}" for k, n in zip(KINDS, N)], fontsize=10, color=INK)
+save(fig, "by_kind.png")

eval/results.md

@@ -105,6 +105,92 @@ v3a and v3.1a on CPU, PyTorch fp32:
 
 Both were part of the score used to pick v3.1a, so they are dev numbers.
 
+## Other models on the same questions (Colab bf16, 4 Oct 2026)
+
+Run on one RTX PRO 6000, bf16, greedy. v3.1a here is the Colab bf16 run (123/162 on google_big, against 124 on CPU
+fp32), so every row is on the same footing.
+
+- Zero-shot: one user message (no system message) in each model's chat template, Qwen3 with `enable_thinking=False`,
+  at most 32 new tokens. The message is the instruction "Answer the question using only the numbered search results
+  below. Reply with just the answer in a few words. If the results do not answer the question, reply exactly: not
+  found", then `Search results:` with the same numbered results v3.1a sees, then `Question:` and the question.
+- Post-processing, the same for every generative model: keep the new tokens up to the first end-of-turn token, drop
+  a `<think>...</think>` block, keep the first non-empty line, strip spaces, quotes and one trailing period. A reply
+  that contains the words "not found" becomes exactly `not found`.
+- Extractive: the five results as `title: text` lines are the context; the span the QA pipeline picks is the answer,
+  and an empty answer is `not found`.
+- Fine-tune: HuggingFaceTB/SmolLM2-135M, 2 epochs on exactly v3.1a's training rows (258,542 examples), AdamW lr 1e-4,
+  batch 32, warmup 100 steps then linear decay, context 512, 27 minutes.
+
+| model | kind | parameters | google25 | google_heldout | google_fresh | google_big | all 258 | google_big clean (144) | google_big latest (60) |
+|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| v3.1a (this model) | trained for this task | 62.0M | 22/24 | 44/53 | 17/19 | 123/162 | 206 (79.8%) | 107 | 37 |
+| Qwen3-4B | zero-shot | 4.02B | 21/24 | 50/53 | 17/19 | 148/162 | 236 (91.5%) | 132 | 52 |
+| Qwen3-1.7B | zero-shot | 1.72B | 19/24 | 47/53 | 15/19 | 137/162 | 218 (84.5%) | 120 | 54 |
+| SmolLM2-135M fine-tuned on v3.1a's rows | fine-tune | 134.5M | 20/24 | 45/53 | 15/19 | 134/162 | 214 (82.9%) | 116 | 47 |
+| Qwen3-0.6B | zero-shot | 596.0M | 17/24 | 46/53 | 13/19 | 131/162 | 207 (80.2%) | 115 | 45 |
+| deepset/roberta-base-squad2 | extractive | 124.1M | 15/24 | 42/53 | 14/19 | 113/162 | 184 (71.3%) | 98 | 39 |
+| deepset/minilm-uncased-squad2 | extractive | 33.2M | 15/24 | 43/53 | 16/19 | 105/162 | 179 (69.4%) | 94 | 35 |
+| SmolLM2-360M-Instruct | zero-shot | 361.8M | 14/24 | 40/53 | 8/19 | 108/162 | 170 (65.9%) | 94 | 44 |
+| Qwen2.5-0.5B-Instruct | zero-shot | 494.0M | 18/24 | 39/53 | 7/19 | 104/162 | 168 (65.1%) | 90 | 40 |
+| SmolLM2-135M-Instruct | zero-shot | 134.5M | 14/24 | 37/53 | 6/19 | 97/162 | 154 (59.7%) | 83 | 40 |
+
+google_big clean leaves out the 18 google_big questions that v1.1's training rows ask word for word. google_big
+latest is its 60 latest / current / most recent questions.
+
+Verdicts over all 258 questions:
+
+| model | answered correct (of 208) | missed (said `not found`) | wrong | trap caught (of 50) | trap fooled |
+|---|---:|---:|---:|---:|---:|
+| v3.1a | 165 | 12 | 31 | 41 | 9 |
+| Qwen3-4B | 192 | 0 | 16 | 44 | 6 |
+| Qwen3-1.7B | 192 | 0 | 16 | 26 | 24 |
+| SmolLM2-135M fine-tuned | 182 | 4 | 22 | 32 | 18 |
+| Qwen3-0.6B | 177 | 10 | 21 | 30 | 20 |
+| roberta-base-squad2 | 155 | 22 | 31 | 29 | 21 |
+| minilm-uncased-squad2 | 152 | 13 | 43 | 27 | 23 |
+| SmolLM2-360M-Instruct | 170 | 0 | 38 | 0 | 50 |
+| Qwen2.5-0.5B-Instruct | 168 | 0 | 40 | 0 | 50 |
+| SmolLM2-135M-Instruct | 154 | 0 | 54 | 0 | 50 |
+
+google_big by kind (right / questions):
+
+| model | fact (34) | unit (25) | trick (28) | live (35) | version (14) | winner (26) |
+|---|---:|---:|---:|---:|---:|---:|
+| v3.1a | 30 | 20 | 24 | 29 | 6 | 14 |
+| Qwen3-4B | 33 | 25 | 24 | 34 | 12 | 20 |
+| Qwen3-1.7B | 33 | 23 | 13 | 34 | 12 | 22 |
+| SmolLM2-135M fine-tuned | 31 | 23 | 21 | 32 | 9 | 18 |
+| Qwen3-0.6B | 31 | 23 | 19 | 31 | 11 | 16 |
+| roberta-base-squad2 | 26 | 19 | 16 | 29 | 8 | 15 |
+| minilm-uncased-squad2 | 26 | 20 | 14 | 26 | 8 | 11 |
+| SmolLM2-360M-Instruct | 32 | 20 | 0 | 30 | 8 | 18 |
+| Qwen2.5-0.5B-Instruct | 31 | 21 | 0 | 31 | 8 | 13 |
+| SmolLM2-135M-Instruct | 30 | 18 | 0 | 28 | 5 | 16 |
+
+Paired against v3.1a, question by question. Each cell is "only v3.1a right / only the other model right, exact
+McNemar p"; `*` marks p < 0.05.
+
+| model | google25 | google_heldout | google_fresh | google_big | all 258 | google_big clean | google_big latest |
+|---|---|---|---|---|---|---|---|
+| Qwen3-4B | 2/1, 1.000 | 1/7, 0.070 | 2/2, 1.000 | 7/32, 0.000* | 12/42, 0.000* | 5/30, 0.000* | 2/17, 0.001* |
+| Qwen3-1.7B | 4/1, 0.375 | 3/6, 0.508 | 4/2, 0.688 | 16/30, 0.054 | 27/39, 0.175 | 15/28, 0.066 | 0/17, 0.000* |
+| SmolLM2-135M fine-tuned | 3/1, 0.625 | 5/6, 1.000 | 3/1, 0.625 | 14/25, 0.108 | 25/33, 0.358 | 14/23, 0.188 | 3/13, 0.021* |
+| Qwen3-0.6B | 6/1, 0.125 | 5/7, 0.774 | 5/1, 0.219 | 20/28, 0.312 | 36/37, 1.000 | 18/26, 0.291 | 8/16, 0.152 |
+| roberta-base-squad2 | 7/0, 0.016* | 7/5, 0.774 | 4/1, 0.375 | 30/20, 0.203 | 48/26, 0.014* | 27/18, 0.233 | 7/9, 0.804 |
+| minilm-uncased-squad2 | 7/0, 0.016* | 6/5, 1.000 | 2/1, 1.000 | 34/16, 0.015* | 49/22, 0.002* | 28/15, 0.066 | 11/9, 0.824 |
+| SmolLM2-360M-Instruct | 8/0, 0.008* | 8/4, 0.388 | 9/0, 0.004* | 36/21, 0.063 | 61/25, 0.000* | 32/19, 0.092 | 4/11, 0.118 |
+| Qwen2.5-0.5B-Instruct | 5/1, 0.219 | 9/4, 0.267 | 10/0, 0.002* | 40/21, 0.020* | 64/26, 0.000* | 36/19, 0.030* | 7/10, 0.629 |
+| SmolLM2-135M-Instruct | 8/0, 0.008* | 10/3, 0.092 | 11/0, 0.001* | 44/18, 0.001* | 73/21, 0.000* | 40/16, 0.002* | 7/10, 0.629 |
+
+(0.000 means below 0.0005.)
+
+Caveats for this comparison: the zero-shot models were pretrained on the open web and may know some answers without
+reading the results, while v3.1a's training held out every test question and topic. google25 is v3.1a's dev set
+and google_fresh / google_big guided the v3.1 data, which favours v3.1a a little. The Qwen and SmolLM2 model cards
+suggest sampling; greedy is used for every model so the run is deterministic. Qwen3-4B is a reference at about 65
+times v3.1a's size.
+
 ## Caveats
 
 - The sets are small. One question is 0.4 points of the 258 total and 0.6 points of google_big.