sync: from ael backend/space/

rerank.py

@@ -71,35 +71,39 @@ MODEL = os.environ.get("AEL_RERANK_MODEL", "english")
 
 # Drop a result the model thinks this unlikely to answer the query.
 #
-# This started at 0.5, on the reasonable-sounding argument that 0.5 is the
-# neutral point of a calibrated probability. That argument is wrong for this
-# model, and the measured score distribution says so. Nine queries against
-# production on 2026-10-02 (`?racerDetails=1`):
+# This went 0.5 -> 0.35 -> 0.15, and the direction is the finding: **the model is
+# a good ranker of snippets and a bad dropper of them.** Measured on production,
+# 2026-10-02:
 #
-#   genuinely answers the query   0.927 "Homepage - TLD"          (what is .tld)
-#                                 0.838 "Lima"                    (capital of Peru)
-#                                 0.692 "How Does a Heat Pump Work"
-#                                 0.676 "Current World Population"
-#                                 0.522 "The Best Donuts In Seattle"
-#   related topic, no answer      0.329 "Best Donuts San Francisco"
-#                                 0.321 "Will a Heat Pump Work in an Old House?"
-#   unrelated                     0.098 "peruse | Etymology of peruse"
-#                                 0.042 "Earliest Mysticete ... Peru"
-#                                 0.004 "Donald Trump - Wikipedia"
+# Ranked correctly, repeatedly. "capital of Peru" put "Lima" (0.838) above a
+# Nat Geo photo of a Peruvian mysticete (0.042); "who won the 1876 presidential
+# election" put the Wikipedia election article above a Reddit thread about a
+# video game achievement. Reordering is where it earns its keep.
 #
-# The separation is between roughly 0.4 and 0.15, not at 0.5 — so 0.5 was
-# throwing away real answers (the 0.4 band) while the junk it was meant to catch
-# sits far below it anyway. 0.35 keeps the 0.4 band and still removes everything
-# under 0.15.
+# Dropped answers it should not have. At 0.35 the threshold removed the correct
+# result for four questions in a row and kept a generic page instead:
 #
-# What a threshold cannot fix, and did not try to: the model scores news and
-# retail homepages at 0.65-0.80 for questions they cannot answer ("population of
-# Ascension Island" -> "Current World Population", 0.676). The instructions
-# below already tell it to answer no for homepages and category listings and it
-# does not always. That is a model limitation, recorded rather than papered over
-# with a hand-written heuristic — and the honest response to it is better
-# engines, which is the race's job, not this stage's.
-DROP_BELOW = float(os.environ.get("AEL_RERANK_DROP", "0.35"))
+#   "Unix program that replaced man with compressed pages" -> mandb gone,
+#       "Linux/Unix Tutorial" kept
+#   "first artificial satellite to orbit a body other than Earth" -> Luna gone,
+#       a Hackaday post kept
+#   "TLD delegated to a donut shop" -> .donuts gone, ".radio" kept
+#   "IUPAC systematic name of caffeine" -> the chemistry pages kept, the
+#       answer-bearing ones dropped
+#
+# The cause is not a bad threshold, it is what the model is reading: a title and
+# a snippet, never the page. A snippet about mandb does not contain the words
+# "compressed manual pages", so the passage looks like a non-answer even when
+# the page is the answer. No threshold fixes that — the useful behaviour is to
+# *rank* on what the snippet says and let the reader open the page, which is
+# what the tool prompt already tells the model to do.
+#
+# So: drop only what the model actively rejects, and let ordering do the rest.
+# Measured unrelated results sit at 0.004-0.098, so 0.15 removes the clearly
+# irrelevant while leaving the ambiguous ones for the reader. The result is
+# deliberately a longer page than the 0.35 experiment produced — a research
+# tool that returns one confident result is a worse research tool.
+DROP_BELOW = float(os.environ.get("AEL_RERANK_DROP", "0.15"))
 
 # Never return an empty page. See the starvation note above — this is a floor of
 # one, not a target: a short page is fine, an empty one is not.