Download source/answer_check.py from Kai9987kai/supermix-v91: direct link, hf CLI and curl.
- Browser
- Download file 39.3 kB
-
https://huggingface.co/Kai9987kai/supermix-v91/resolve/main/source/answer_check.py
- Command line
-
hf download hf://Kai9987kai/supermix-v91/source/answer_check.py
-
curl -L -o answer_check.py https://huggingface.co/Kai9987kai/supermix-v91/resolve/main/source/answer_check.py
39.3 kB
| """Independently check a maths reply against the question that was asked. | |
| The session's central finding is that only a *checkable* answer resists | |
| recitation: a model reproducing a remembered reply to a novel problem is simply | |
| wrong, and no amount of fluency hides it. `eval_problem_solving.py` applies that | |
| offline, over problems it generates itself. | |
| This module applies it live, to whatever the user typed. It re-derives the truth | |
| from the question and compares, so the interface can say "wrong, the answer is | |
| 905" instead of presenting confident arithmetic and leaving the reader to check. | |
| It recognises only the shapes the models were trained on -- twenty-one of them | |
| as of v82, one per task in the v80 corpus: nine arithmetic shapes from v74 and | |
| twelve solver-verified science and mathematics shapes from v79/v80. Anything | |
| else returns `None`, which the interface must render as *not checked* rather | |
| than as correct: an unrecognised question is not a passed one. | |
| Multiplication, division, sequence and two-step were added for v74, which | |
| introduced those tasks. Before that a chat reply to "What is 47 x 6?" showed | |
| NOT CHECKED -- the model's strongest tasks were the ones nothing verified. | |
| **This module is not a verifier and must never be promoted to one.** It | |
| re-derives an expected answer from a *pattern* in the question, not from a | |
| parse of its meaning, so a question whose shape it half-recognises would get a | |
| confident wrong verdict. `nexus_epistemics.ANSWER_VERIFIER_IDS` is the | |
| allowlist of things permitted to certify an answer and it contains exactly one | |
| entry, `grounding_runtime.finalize_grounded_response`; `answer_check` is | |
| deliberately absent from it. The compound-expression trap in | |
| `_is_compound_expression` is the concrete reason: "What is 2 + 3 * 4?" once | |
| parsed as multiplication with expected 12.0 where the truth is 14, because a | |
| lone `A * B` search found `3 * 4` and never saw the `+`. That is now refused | |
| as NOT CHECKED, and `test_answer_check.py` pins it in both directions. | |
| v82 coverage, measured over 840 prompts drawn from `build_omni_corpus.TASKS` | |
| and `eval_problem_solving.GENERATORS` at seed 4242: 799/840 = 0.951 before the | |
| v82 widening, 840/840 = 1.000 after, with zero confident-wrong verdicts in | |
| either. Coverage is not accuracy of the *model*; it is only the fraction of | |
| questions this module is willing to judge at all. | |
| v93 adds eleven benchmark tasks and this module gains a parser for every | |
| shape among them: five science and mathematics forms re-derived from the | |
| question (`impulse`, `ohms_current`, `spring_energy`, `permutations`, | |
| `final_velocity`), three code forms the existing `_code_trace` already runs | |
| once its snippet-start pattern admits `r = sum(range(3, 9))`, and three | |
| connectome lookups (`cns_type_count`, `cns_side_count`, `cns_pair_synapses`) | |
| whose "re-derivation" is the same table lookup the corpus builder did, from | |
| the CC-BY male-CNS arrays. Measured at seed 4242 over all 41 registered tasks: | |
| 1,640/1,640 narrow-template prompts parsed and 4,920/4,920 prompts over the | |
| full natural-phrasing bank (held-out forms included), zero confident-wrong | |
| verdicts in either. `_permutations` runs before `_combination_choose` because | |
| "permutations of 9 things taken 2 at a time" contains `taken`, and the | |
| combination reading would have returned 36 for an answer of 72. | |
| """ | |
| from __future__ import annotations | |
| import math | |
| import re | |
| from dataclasses import dataclass | |
| from typing import Callable, List, Optional, Tuple | |
| #: Matches the tolerance `eval_problem_solving.is_correct` uses, so the live | |
| #: check and the benchmark cannot disagree about the same answer. | |
| TOLERANCE = 1e-6 | |
| _NUMBER = re.compile(r"-?\d+(?:\.\d+)?(?:/\d+)?") | |
| _DECIMAL = r"-?\d+(?:\.\d+)?" | |
| class Check: | |
| """The verdict on one reply.""" | |
| task: str | |
| expected: float | |
| predicted: Optional[float] | |
| correct: bool | |
| def to_dict(self) -> dict: | |
| return { | |
| "task": self.task, | |
| "expected": self.expected, | |
| "predicted": self.predicted, | |
| "correct": self.correct, | |
| } | |
| def extract_answer(text: str) -> Optional[float]: | |
| """The reply's answer is the last number it produces. | |
| Identical rule to `eval_problem_solving.extract_answer`; the scratchpad | |
| formats all end with the result, so "600 + 200 = 800, 17 + 88 = 105, total | |
| 905" reads as 905 rather than 600. | |
| """ | |
| matches = _NUMBER.findall(text.replace(",", "")) | |
| if not matches: | |
| return None | |
| raw = matches[-1].rstrip(".") | |
| try: | |
| if "/" in raw: | |
| numerator, denominator = raw.split("/", 1) | |
| value = float(numerator) / float(denominator) | |
| else: | |
| value = float(raw) | |
| return value if math.isfinite(value) else None | |
| except (ValueError, ZeroDivisionError, OverflowError): | |
| return None | |
| # -- question parsers ------------------------------------------------------- | |
| # | |
| # Each returns (task, expected) or None. They are deliberately narrow: a loose | |
| # pattern that half-matched an unrelated question would produce a confident | |
| # wrong verdict, which is worse than no verdict at all. | |
| #: An operator sitting between two numbers -- the only kind that makes an | |
| #: expression compound. `m/s`, `m/s^2` and `x + 14 = 39` do not match, because | |
| #: neither side of the operator is a digit in the first two and `_algebra` | |
| #: claims the third before these parsers run. | |
| #: The trailing operand is a lookahead, not a consumed group: without it | |
| #: `re.findall` over "2 + 3 * 4" consumes the `3` while matching `2 + 3` and | |
| #: then finds only one operator, which is the bug this guard exists to catch. | |
| _INFIX = re.compile(r"\d\s*[-+*/x]\s*(?=-?\d)", re.I) | |
| def _is_compound_expression(question: str) -> bool: | |
| """True when the question chains two or more infix operators. | |
| The bare `A op B` parsers below each find *one* operator and compute from | |
| it. Given "What is 2 + 3 * 4?" the multiplication parser finds `3 * 4` and | |
| returns 12.0, which is confidently wrong: precedence makes the answer 14. | |
| Nothing in the corpus asks a compound question, so the correct response is | |
| to refuse rather than to grow an expression evaluator here -- a partially | |
| correct evaluator would produce exactly the confident wrong verdict this | |
| module exists to avoid. | |
| """ | |
| return len(_INFIX.findall(question)) >= 2 | |
| def _binary(question: str) -> Optional[Tuple[str, float]]: | |
| match = re.search(rf"(?<![\w.+-])({_DECIMAL})\s*([+-])\s*({_DECIMAL})(?!\w|\.\d)", question) | |
| if not match or "=" in question or _is_compound_expression(question): | |
| return None | |
| left, op, right = float(match.group(1)), match.group(2), float(match.group(3)) | |
| return ("arithmetic", float(left + right if op == "+" else left - right)) | |
| def _percent(question: str) -> Optional[Tuple[str, float]]: | |
| match = re.search(rf"({_DECIMAL})\s*%\s*of\s*({_DECIMAL})", question, re.I) | |
| if not match: | |
| return None | |
| return ("percent", float(match.group(1)) * float(match.group(2)) / 100.0) | |
| def _algebra(question: str) -> Optional[Tuple[str, float]]: | |
| match = re.search(rf"\bx\s*([+*/-])\s*({_DECIMAL})\s*=\s*({_DECIMAL})(?!\w|\.\d)", question, re.I) | |
| if not match: | |
| return None | |
| op, constant, right = match.group(1), float(match.group(2)), float(match.group(3)) | |
| if op in "*/" and constant == 0: | |
| return None | |
| if op == "+": | |
| result = right - constant | |
| elif op == "-": | |
| result = right + constant | |
| elif op == "*": | |
| result = right / constant | |
| else: | |
| result = right * constant | |
| return ("algebra_one_step", result) | |
| def _average(question: str) -> Optional[Tuple[str, float]]: | |
| if not re.search(r"\b(average|mean)\b", question, re.I): | |
| return None | |
| tail = question.split(":", 1)[-1] | |
| values = [float(v) for v in re.findall(r"-?\d+(?:\.\d+)?", tail)] | |
| if len(values) < 2: | |
| return None | |
| return ("average", sum(values) / len(values)) | |
| def _multiplication(question: str) -> Optional[Tuple[str, float]]: | |
| """`A x B`, the corpus's multiplication form. | |
| Must run after `_algebra`: `x` is this corpus's multiplication sign *and* | |
| its unknown, and only the digit on the left tells them apart. | |
| """ | |
| match = re.search(r"(-?\d+(?:\.\d+)?)\s*[x*]\s*(-?\d+(?:\.\d+)?)", question, re.I) | |
| if not match or "=" in question or _is_compound_expression(question): | |
| return None | |
| return ("multiplication", float(match.group(1)) * float(match.group(2))) | |
| def _division(question: str) -> Optional[Tuple[str, float]]: | |
| match = re.search(r"(-?\d+(?:\.\d+)?)\s*/\s*(-?\d+(?:\.\d+)?)", question) | |
| if not match or "=" in question or _is_compound_expression(question): | |
| return None | |
| divisor = float(match.group(2)) | |
| if divisor == 0: | |
| # Not checkable rather than an exception; the question has no answer. | |
| return None | |
| return ("division", float(match.group(1)) / divisor) | |
| #: The fractions `two_step` is allowed to state in words, and their percentages. | |
| #: Kept in step with `build_scratchpad_math.PERCENT_FRACTIONS`; a form this does | |
| #: not know falls through to `_percent`, which reads the percentage and ignores | |
| #: the second operation, so it must not silently disagree. | |
| _SPOKEN_FRACTIONS = { | |
| "one tenth": 10.0, "one fifth": 20.0, "one quarter": 25.0, "one half": 50.0, | |
| } | |
| #: How the second operation can be worded. `reduce it by` and `take away` are | |
| #: subtraction; `increase it by` is addition. | |
| _ADD_WORDS = r"add|increase it by" | |
| _SUBTRACT_WORDS = r"subtract|reduce it by|take away" | |
| def _two_step(question: str) -> Optional[Tuple[str, float]]: | |
| """`P% of N`, then a second operation. Must precede `_percent`, which it contains. | |
| v88 gave this task five prompt templates where it had one, and the extra | |
| forms broke the original pattern in three ways at once: the percentage can | |
| be spelled out (`Take 15 percent of 320`) or spoken as a fraction (`Start | |
| with one quarter of 320`), the two clauses can be joined by `and then`, a | |
| semicolon or a full stop, and the operation can be worded (`reduce it by`, | |
| `increase it by`, `take away`). | |
| A miss here is not harmless. `_two_step` runs before `_percent` precisely | |
| because it contains a percent question, so a form this does not match is | |
| read by `_percent` as `P% of N` -- the first half only -- and every such row | |
| is then reported WRONG against a correct reply. That is what 174 of 20,391 | |
| checked rows did before this was widened, all of them `two_step`. | |
| """ | |
| base: Optional[float] = None | |
| tail = question | |
| match = re.search( | |
| rf"({_DECIMAL})\s*(?:%|percent)\s*of\s*({_DECIMAL})", question, re.I) | |
| if match: | |
| base = float(match.group(1)) * float(match.group(2)) / 100.0 | |
| tail = question[match.end():] | |
| else: | |
| spoken = re.search( | |
| rf"\b({'|'.join(_SPOKEN_FRACTIONS)})\s+of\s*({_DECIMAL})", | |
| question, re.I) | |
| if spoken: | |
| base = (_SPOKEN_FRACTIONS[spoken.group(1).lower()] | |
| * float(spoken.group(2)) / 100.0) | |
| tail = question[spoken.end():] | |
| if base is None: | |
| return None | |
| operation = re.search( | |
| rf"(?:then|and then|next|,|;|\.)\s*({_ADD_WORDS}|{_SUBTRACT_WORDS})" | |
| rf"\s*({_DECIMAL})", tail, re.I) | |
| if not operation: | |
| return None | |
| word, operand = operation.group(1).lower(), float(operation.group(2)) | |
| adds = re.fullmatch(_ADD_WORDS, word, re.I) is not None | |
| return ("two_step", base + operand if adds else base - operand) | |
| def _sequence(question: str) -> Optional[Tuple[str, float]]: | |
| """The next term of an arithmetic progression. | |
| Returns None when the differences are not constant. The corpus only | |
| contains arithmetic progressions, so anything else is a question this | |
| cannot verify -- and reporting "not checked" is correct where guessing a | |
| rule would silently invent a right answer. | |
| """ | |
| if not re.search(r"\b(next|sequence)\b", question, re.I): | |
| return None | |
| tail = question.split(":", 1)[-1] | |
| values = [float(v) for v in re.findall(r"-?\d+(?:\.\d+)?", tail)] | |
| if len(values) < 3: | |
| return None | |
| steps = {round(b - a, 9) for a, b in zip(values, values[1:])} | |
| if len(steps) != 1: | |
| return None | |
| return ("sequence", values[-1] + steps.pop()) | |
| # -- science shapes (v81) --------------------------------------------------- | |
| # | |
| # v80 answers physics correctly and the interface said NOT CHECKED for every | |
| # one of them, because these shapes were never taught to the checker. A model | |
| # whose strongest new capability cannot be verified live is the same gap v76 | |
| # closed for multiplication, one domain over. | |
| # | |
| # Each reads the quantities by name and unit, so it matches the corpus's | |
| # phrasings without depending on any single one of them. | |
| def _quantity(question: str, names: str, unit: str) -> Optional[float]: | |
| """Read `<name> <number> <unit>` in either order, as the corpus writes it.""" | |
| number = r"(-?\d+(?:\.\d+)?)" | |
| # Both names and unit are alternations, so both must be grouped. Left | |
| # bare, `(\d+)\s*m|metres?` parses as `(\d+)\s*m` OR `metres?` -- the | |
| # second branch has no capture group, and match.group(1) is then None. | |
| for pattern in (rf"(?:{names})\D{{0,24}}?{number}\s*(?:{unit})\b", | |
| rf"{number}\s*(?:{unit})\b\D{{0,24}}?(?:{names})"): | |
| match = re.search(pattern, question, re.I) | |
| if match: | |
| return float(match.group(1)) | |
| return None | |
| #: (task, words identifying the target, quantity A, quantity B) | |
| #: Division tasks are named in `_DIVISION_LAWS`; everything else multiplies. | |
| #: Units carry their spelled-out forms. The corpus writes "57 volts and 5 | |
| #: amps" as readily as "57 V, 5 A", and a checker that only knew the symbols | |
| #: reported NOT CHECKED for a third of the questions the model answers. | |
| _MASS = r"kg|kilograms?" | |
| _FORCE = r"N|newtons?" | |
| _ANY = r"[a-z/^]*" | |
| _PRODUCT_LAWS = ( | |
| ("force", r"force", | |
| (r"mass|body|block|object", _MASS), (r"accelerat\w*", r"m/s\^?2")), | |
| ("momentum", r"momentum", | |
| (r"mass|object|body", _MASS), (r"velocity|speed|moves|travelling|at", r"m/s")), | |
| # v82: "Find the work done by 98 N acting over 2 m." names the force only | |
| # as "by". Measured 32/40 before adding it, 40/40 after. | |
| ("work", r"work", | |
| (r"force|done by|by", _FORCE), (r"distance|moves|through|over|acts", r"m|metres?")), | |
| ("voltage", r"voltage|potential difference", | |
| (r"current|flows|carrying|drives", r"A|amps?|amperes?"), | |
| (r"resistance|ohm|through|across|resistor", r"ohms?")), | |
| ("electrical_power", r"electrical power|power dissipated|power|used at", | |
| (r"voltage|volts?|runs at|at", r"V|volts?"), | |
| (r"current|drawing|amps?|and", r"A|amps?|amperes?")), | |
| ("wave_speed", r"wave speed|speed of|its speed|speed at", | |
| (r"frequency|at", r"Hz|hertz"), (r"wavelength|with", r"m|metres?")), | |
| # v82: "produced by 580 N on 116 kg" names the force only as "by". | |
| # Measured 35/40 before, 40/40 after. | |
| ("acceleration", r"acceleration|accelerat\w*", | |
| (r"force|results from|from|produced by|by", _FORCE), (r"mass|body|object|on", _MASS)), | |
| # v82: "What power corresponds to 1860 joules in 20 seconds?" names the | |
| # work only as "corresponds to". Measured 28/40 before, 40/40 after. | |
| ("power", r"power", | |
| (r"work|corresponds to|delivered|done", r"J|joules?"), (r"time|in|over", r"s|seconds?")), | |
| ("molarity", r"molarity|concentration|molar", | |
| (r"mol|moles|solute|of", r"mol|moles"), | |
| (r"volume|litres?|liters?|dissolved|in", r"L|litres?|liters?")), | |
| # -- v93 -- | |
| # Placed after the nine above on purpose. `impulse` prompts always name a | |
| # force, and `ohms_current` prompts always say "current" and usually | |
| # "voltage", so the `force`, `voltage` and `electrical_power` laws see them | |
| # first; each then fails on a quantity it needs (kg, amps) and falls | |
| # through. The reverse cannot happen: a `voltage` prompt carries no number | |
| # in volts and a `force` prompt no time in seconds, so neither of these | |
| # can claim one of theirs. | |
| ("impulse", r"impulse", | |
| (r"force|push|delivered by|acting|acts|by|from|:", _FORCE), | |
| (r"time|for|over|lasting|during|in|,", r"s|seconds?")), | |
| ("ohms_current", r"current", | |
| (r"voltage|volts?|at|drives|across|applied|when|:", r"V|volts?"), | |
| (r"resistance|resistor|ohms?|through|across|drives|,", r"ohms?")), | |
| ) | |
| _DIVISION_LAWS = frozenset({"acceleration", "power", "molarity", "ohms_current"}) | |
| def _science(question: str) -> Optional[Tuple[str, float]]: | |
| for task, target, (a_names, a_unit), (b_names, b_unit) in _PRODUCT_LAWS: | |
| if not re.search(target, question, re.I): | |
| continue | |
| a = _quantity(question, a_names, a_unit) | |
| b = _quantity(question, b_names, b_unit) | |
| if a is None or b is None: | |
| continue | |
| if task in _DIVISION_LAWS: | |
| if b == 0: | |
| return None # not checkable rather than an exception | |
| return (task, a / b) | |
| return (task, a * b) | |
| return None | |
| def _kinetic_energy(question: str) -> Optional[Tuple[str, float]]: | |
| if not re.search(r"kinetic energy", question, re.I): | |
| return None | |
| mass = _quantity(question, r"mass|body", r"kg") | |
| velocity = _quantity(question, r"velocity|speed|moves|at", r"m/s") | |
| if mass is None or velocity is None: | |
| return None | |
| return ("kinetic_energy", 0.5 * mass * velocity * velocity) | |
| def _spring_energy(question: str) -> Optional[Tuple[str, float]]: | |
| """`E = k x^2 / 2` (v93). Runs before `_science`, as `_kinetic_energy` does. | |
| The spring constant is the number carrying `N/m`, which is unambiguous; | |
| the extension is the number in metres that is *not* part of `N/m`, read | |
| after an extension word so the `m` of `N/m` cannot be taken for it. | |
| """ | |
| if not re.search(r"spring energy|elastic potential energy|energy (?:is )?stored", | |
| question, re.I): | |
| return None | |
| constant = re.search(r"(-?\d+(?:\.\d+)?)\s*N/m\b", question, re.I) | |
| extension = _quantity( | |
| question, | |
| r"extension|extended|stretched|pulled|stretch|by|is", | |
| r"m|metres?|meters?", | |
| ) | |
| if constant is None or extension is None: | |
| return None | |
| k, x = float(constant.group(1)), float(extension) | |
| return ("spring_energy", 0.5 * k * x * x) | |
| def _final_velocity(question: str) -> Optional[Tuple[str, float]]: | |
| """`v = u + a t` (v93). Runs before `_science`. | |
| Three numbers, three units, and the units tell them apart: the | |
| acceleration carries `m/s^2`, the initial velocity `m/s` with no `^2` | |
| after it, and the time a bare `s` or `seconds`. Nothing is read by | |
| position, so any order of the three clauses parses the same way. | |
| """ | |
| if not re.search(r"final velocity|final speed|velocity after|going after", | |
| question, re.I): | |
| return None | |
| accel = re.search(r"(-?\d+(?:\.\d+)?)\s*m/s\^?2\b", question, re.I) | |
| initial = re.search(r"(-?\d+(?:\.\d+)?)\s*m/s\b(?!\^?2)", question, re.I) | |
| # `(?<![/^])` keeps the `s` of `m/s` and the `2` of `m/s^2` out of it. | |
| time = re.search(r"(?<![/^\w])(-?\d+(?:\.\d+)?)\s*(?:s|seconds?)\b(?!/|\^)", | |
| question, re.I) | |
| if accel is None or initial is None or time is None: | |
| return None | |
| u, a, t = (float(m.group(1)) for m in (initial, accel, time)) | |
| return ("final_velocity", u + a * t) | |
| def _permutations(question: str) -> Optional[Tuple[str, float]]: | |
| """`P(n, k)`, however the corpus words it (v93). | |
| Must run before `_combination_choose`: "permutations of 9 things taken 2 | |
| at a time" contains `taken`, and that parser would return C(9, 2) = 36 | |
| for a question whose answer is 72 -- the confident wrong verdict this | |
| module exists to avoid. So any question that says permutation, arrange or | |
| ordered is claimed here first, and a form this cannot read returns None | |
| rather than falling through to the combination reading. | |
| """ | |
| if not re.search(r"permutation|arrang|ordered|line up", question, re.I): | |
| return None | |
| number = r"(\d+)" | |
| # (pattern, reversed) -- `reversed` says the phrasing states k before n, | |
| # as `_combination_choose` does. Every corpus and natural form is listed; | |
| # the order matters only where two could match, and none do. | |
| for pattern, reverse in ( | |
| (rf"n\s*=\s*{number}\s*k\s*=\s*{number}", False), | |
| (rf"n\s*{number}\s*,?\s*k\s*{number}", False), | |
| (rf"of\s*{number}\s*things taken\s*{number}", False), | |
| (rf"of\s*{number}\s*(?:taken|take)\s*{number}", False), | |
| (rf"{number}\s*things and line up\s*{number}", False), | |
| (rf"can\s*{number}\s*items? be arranged in order from\s*{number}", True), | |
| (rf"arrangements? of\s*{number}\s*(?:are there )?from\s*{number}", True), | |
| (rf"arrange\s*{number}\s*out of\s*{number}", True), | |
| (rf"arranging\s*{number}\s*out of\s*{number}", True), | |
| (rf"{number}\s*items? can be picked from\s*{number}", True), | |
| (rf"selections? of\s*{number}\s*from\s*{number}", True), | |
| ): | |
| match = re.search(pattern, question, re.I) | |
| if not match: | |
| continue | |
| a, b = int(match.group(1)), int(match.group(2)) | |
| n, k = (b, a) if reverse else (a, b) | |
| if k > n: | |
| return None # far more likely misread than genuinely 0 | |
| import math as _math | |
| return ("permutations", float(_math.perm(n, k))) | |
| return None | |
| # -- connectome lookups (v93) ------------------------------------------------ | |
| # | |
| # The three cns_* shapes ask for a fact, not a computation, so the only honest | |
| # re-derivation is the same lookup the corpus builder did: the answer comes | |
| # from `build_connectome_corpus.population()`, built from the CC-BY male-CNS | |
| # tables. A type or pair outside that population returns None (not checked) | |
| # rather than 0, because "not in the table we teach from" is not "zero | |
| # neurons". Where the tables are absent the parsers are inert. | |
| #: A type name as the corpus writes it: letters, digits, `_` and `-`. The | |
| #: trailing `[A-Za-z0-9_]` keeps a sentence-final hyphen or the `?` out. | |
| _TYPE = r"([A-Za-z][A-Za-z0-9_\-]*[A-Za-z0-9_]|[A-Za-z])" | |
| def _connectome_population(): | |
| try: | |
| import build_connectome_corpus as cns | |
| except Exception: # noqa: BLE001 - the checker degrades, it does not break | |
| return None | |
| try: | |
| if not cns.data_available(): | |
| return None | |
| return cns.population() | |
| except Exception: # noqa: BLE001 | |
| return None | |
| def _cns_pair_synapses(question: str) -> Optional[Tuple[str, float]]: | |
| if not re.search(r"synapse", question, re.I) or not re.search(r"male CNS", question, re.I): | |
| return None | |
| for pattern in ( | |
| rf"from (?:cell )?type {_TYPE} (?:to|onto) (?:cell )?type {_TYPE}", | |
| rf"type {_TYPE} makes? onto type {_TYPE}", | |
| rf"from {_TYPE} to {_TYPE}\b", | |
| ): | |
| match = re.search(pattern, question) | |
| if match: | |
| break | |
| else: | |
| return None | |
| pop = _connectome_population() | |
| if pop is None: | |
| return None | |
| pre, post = match.group(1), match.group(2) | |
| for a, b, weight in pop.pairs: | |
| if a == pre and b == post: | |
| return ("cns_pair_synapses", float(weight)) | |
| return None | |
| def _cns_side_count(question: str) -> Optional[Tuple[str, float]]: | |
| if not re.search(r"neurons?", question, re.I) or not re.search(r"male CNS", question, re.I): | |
| return None | |
| side = re.search(r"\b(left|right)\b", question, re.I) | |
| if side is None: | |
| return None | |
| name = _cns_type_name(question) | |
| if name is None: | |
| return None | |
| pop = _connectome_population() | |
| if pop is None or name not in pop.n_neurons: | |
| return None | |
| table = pop.left if side.group(1).lower() == "left" else pop.right | |
| return ("cns_side_count", float(table[name])) | |
| def _cns_type_count(question: str) -> Optional[Tuple[str, float]]: | |
| if not re.search(r"neurons?", question, re.I) or not re.search(r"male CNS", question, re.I): | |
| return None | |
| if re.search(r"\b(left|right|side|hemisphere|synapse)", question, re.I): | |
| return None # a side count or a pair count, not this shape | |
| name = _cns_type_name(question) | |
| if name is None: | |
| return None | |
| pop = _connectome_population() | |
| if pop is None or name not in pop.n_neurons: | |
| return None | |
| return ("cns_type_count", float(pop.n_neurons[name])) | |
| def _cns_type_name(question: str) -> Optional[str]: | |
| """The one type a count question names, or None when the form is unknown.""" | |
| for pattern in ( | |
| rf"(?:cell )?type {_TYPE}", | |
| rf"(?:the |many )?{_TYPE} neurons", | |
| ): | |
| match = re.search(pattern, question) | |
| if match: | |
| return match.group(1) | |
| return None | |
| def _combination_choose(question: str) -> Optional[Tuple[str, float]]: | |
| """`n choose k`, however the corpus words it. | |
| The corpus fixes k at 2 so the working can be shown, but this reads | |
| whatever k is stated rather than assuming it -- an assumption here would | |
| produce a confident wrong verdict on any other k. | |
| """ | |
| if not re.search(r"combination|choose|chosen|taken", question, re.I): | |
| return None | |
| number = r"(\d+)" | |
| # (pattern, reversed) -- `reversed` says the phrasing states k before n. | |
| # v82: this used to pick n = max(a, b) for *every* phrasing, so "30 choose | |
| # 40" returned C(40, 30) = 847660528 where the truth is 0. A size heuristic | |
| # cannot tell an impossible question from a reversed one; the word order | |
| # can, and each of the corpus's four phrasings has a fixed order. | |
| for pattern, reverse in ( | |
| (rf"{number}\s*choose\s*{number}", False), | |
| (rf"n\s*=\s*{number}\s*k\s*=\s*{number}", False), | |
| (rf"of\s*{number}\s*things taken\s*{number}", False), | |
| (rf"can\s*{number}\s*items? be chosen from\s*{number}", True), | |
| ): | |
| match = re.search(pattern, question, re.I) | |
| if not match: | |
| continue | |
| a, b = int(match.group(1)), int(match.group(2)) | |
| n, k = (b, a) if reverse else (a, b) | |
| if k > n: | |
| # C(n, k) is 0 here, but a question asking to choose 40 from 30 is | |
| # far more likely to be one this parser has misread than one whose | |
| # answer is genuinely 0. NOT CHECKED is the honest verdict. | |
| return None | |
| import math as _math | |
| return ("combination", float(_math.comb(n, k))) | |
| return None | |
| def _arithmetic_series(question: str) -> Optional[Tuple[str, float]]: | |
| """Sum of the first n terms of an arithmetic progression.""" | |
| if not re.search(r"arithmetic (?:series|progression)", question, re.I): | |
| return None | |
| # v82: "An arithmetic series starts at 15 with common difference 4" never | |
| # says "first term". Measured 24/40 before adding `starts at`, 40/40 after. | |
| first = re.search(r"(?:first term|starts? at|beginning at)\s*(?:is\s*)?(-?\d+)", | |
| question, re.I) | |
| difference = re.search(r"(?:common )?difference\s*(?:of\s*)?(-?\d+)", question, re.I) | |
| terms = re.search(r"(?:sum of|first)\s*(\d+)\s*terms|(?:\bn\s*(\d+))", question, re.I) | |
| if not (first and difference and terms): | |
| return None | |
| count = int(terms.group(1) or terms.group(2)) | |
| a, d = int(first.group(1)), int(difference.group(1)) | |
| last = a + (count - 1) * d | |
| return ("arithmetic_series", float(count * (a + last) / 2)) | |
| def _word_problem(question: str) -> Optional[Tuple[str, float]]: | |
| match = re.search( | |
| r"has\s+(\d+).*?get\s+(\d+)\s+more.*?give\s+away\s+(\d+)", question, re.I | re.S | |
| ) | |
| if not match: | |
| return None | |
| start, gain, lose = (int(match.group(i)) for i in (1, 2, 3)) | |
| return ("word_problem", float(start + gain - lose)) | |
| #: Order matters, and every entry below is placed against a specific ambiguity: | |
| #: | |
| #: * `_word_problem` and `_average` precede everything numeric, because both | |
| #: contain bare numbers a naive "a + b" search would seize on. | |
| #: * `_sequence` precedes them too -- "7, 17, 27, 37" is a comma-separated list | |
| #: of numbers, which is exactly what an average looks like. | |
| #: * `_two_step` precedes `_percent` because it *contains* a percent question. | |
| #: * `_algebra` precedes `_multiplication` because `x` is both this corpus's | |
| #: multiplication sign and its unknown. | |
| #: * `_kinetic_energy` and `_science` precede the bare-number parsers, because | |
| #: a physics question carries two numbers and a naive `a x b` search would | |
| #: seize on them without knowing which law applies. `_kinetic_energy` runs | |
| #: first of the two: it names a mass and a velocity, which is also what | |
| #: `momentum` matches on. | |
| def _code_trace(question: str) -> Optional[Tuple[str, float]]: | |
| """Re-derive a code-tracing answer by executing the snippet in the question. | |
| Every other parser here re-implements the arithmetic it checks. This one | |
| does not need to: the question *contains* the program, so the check is to | |
| run it. That makes it the strongest verifier in this module -- there is no | |
| second implementation to drift from the first. | |
| Extraction is deliberately narrow. The snippet is recovered by finding the | |
| first assignment to a single-letter variable and taking everything from | |
| there to the trailing question, then the same allowlisted, builtins-free, | |
| timeout-bounded executor the corpus builder uses decides the value. A | |
| question this cannot confidently parse returns ``None`` -- *not checked* -- | |
| which is the only safe failure for this module. | |
| """ | |
| try: | |
| import build_code_corpus as code | |
| except Exception: # noqa: BLE001 - the checker degrades, it does not break | |
| return None | |
| # The wrapper names the variable being asked about. Every template phrases | |
| # it one of these ways, and a question naming none of them is not a code | |
| # question this should touch. | |
| target = None | |
| for pattern in (r"value of ([A-Za-z_]\w*)", r"give (?:the value of )?([A-Za-z_]\w*)", | |
| r"what (?:is|does) ([A-Za-z_]\w*)", r"final ([A-Za-z_]\w*)", | |
| r"([A-Za-z_]\w*) after this runs", r"([A-Za-z_]\w*) hold at the end", | |
| r"\b([A-Za-z_]\w*)\?\s*$"): | |
| found = re.search(pattern, question, flags=re.IGNORECASE) | |
| if found: | |
| target = found.group(1) | |
| break | |
| if target is None: | |
| return None | |
| # The code is the longest span that parses AND assigns that variable. Trying | |
| # spans rather than pattern-matching the wrapper means a new phrasing cannot | |
| # silently produce a wrong value: a mis-trimmed span fails to parse, and a | |
| # span that parses but never assigns the target is rejected below. | |
| # The right-hand side of the first assignment is a literal (list, number, | |
| # negative number) or, since v93's `code_range_sum` (`r = sum(range(3, | |
| # 9))`), a call to a lower-case name. Which names are callable is the | |
| # executor's decision, not this pattern's: anything else is refused there. | |
| start = re.search(r"[A-Za-z_]\w*\s*=\s*(?:[\[\-\d]|[a-z]+\()", question) | |
| if start is None: | |
| return None | |
| body = question[start.start():] | |
| # Candidate spans end at a whitespace boundary, longest first. An earlier | |
| # version required the span to end in a digit or a bracket, which silently | |
| # excluded `code_conditional` -- its snippet ends `else b - a`, on a | |
| # variable name -- and left two of nine tasks unverifiable. Correctness | |
| # comes from the executor accepting the span, not from guessing where the | |
| # prose starts, so the heuristic is gone and only the boundary remains. | |
| pieces = body.split(" ") | |
| for size in range(len(pieces), 0, -1): | |
| chunk = " ".join(pieces[:size]).rstrip(" ?.") | |
| if not chunk or not re.search(rf"\b{re.escape(target)}\s*=", chunk): | |
| continue | |
| result = code.run_snippet(chunk, target) | |
| if result.ok and result.value is not None: | |
| return ("code_trace", float(result.value)) | |
| return None | |
| PARSERS: Tuple[Callable[[str], Optional[Tuple[str, float]]], ...] = ( | |
| # First: a code question contains numbers and operators that several | |
| # parsers below would happily misread as arithmetic. `x = 5` and | |
| # `range(4)` look like a binary expression to `_binary`. | |
| _code_trace, | |
| # v93: the connectome lookups key on "male CNS" and refuse everything | |
| # else, and a type name such as `5-HTPLP01` or `IN17A052` is exactly the | |
| # kind of string `_binary` and `_multiplication` would misread as an | |
| # expression, so they run before any numeric parser. | |
| _cns_pair_synapses, | |
| _cns_side_count, | |
| _cns_type_count, | |
| _word_problem, | |
| # v93: before `_combination_choose`, which would otherwise read | |
| # "permutations of 9 things taken 2 at a time" as C(9, 2). | |
| _permutations, | |
| _combination_choose, | |
| _arithmetic_series, | |
| _kinetic_energy, | |
| # v93: before `_science`, whose `acceleration` law matches "accelerates" | |
| # and whose `work` law matches "work out the spring energy"; both fall | |
| # through on a missing quantity today, and these make that not matter. | |
| _spring_energy, | |
| _final_velocity, | |
| _science, | |
| _sequence, | |
| _average, | |
| _two_step, | |
| _algebra, | |
| _percent, | |
| _division, | |
| _multiplication, | |
| _binary, | |
| ) | |
| def parse_question(question: str) -> Optional[Tuple[str, float]]: | |
| for parser in PARSERS: | |
| try: | |
| result = parser(question) | |
| except (ValueError, ZeroDivisionError, OverflowError): | |
| return None | |
| if result is not None: | |
| return result if math.isfinite(result[1]) else None | |
| return None | |
| def check(question: str, reply: str) -> Optional[Check]: | |
| """Verify a reply, or return ``None`` when the question is not checkable. | |
| ``None`` means *not checked*. The caller must not render it as correct: the | |
| whole value of this is that a wrong answer is visibly wrong, and quietly | |
| passing anything unrecognised would destroy that. | |
| """ | |
| parsed = parse_question(question) | |
| if parsed is None: | |
| return None | |
| task, expected = parsed | |
| predicted = extract_answer(reply) | |
| correct = ( | |
| predicted is not None | |
| and abs(predicted - expected) <= max(TOLERANCE, abs(expected) * 1e-6) | |
| ) | |
| return Check(task=task, expected=expected, predicted=predicted, correct=correct) | |
| def supported_shapes() -> List[str]: | |
| """The question forms this can verify, for the interface to advertise. | |
| Through v81 this listed only the nine arithmetic shapes while `PARSERS` | |
| already handled twelve science and mathematics shapes, so the interface | |
| under-advertised what it could check by more than half. Every entry here is | |
| asserted parseable by `test_answer_check.py`, so the list cannot drift | |
| ahead of the parsers again -- but note it can still drift *behind* them, | |
| which is the harmless direction. | |
| """ | |
| return [ | |
| # arithmetic (v74) | |
| "Solve this basic math problem: 617 + 288", | |
| "What is 25% of 840?", | |
| "Solve for x: x + 14 = 39", | |
| "A student has 45 marbles. They get 38 more and then give away 27. How many marbles do they have now?", | |
| "Find the average (mean) of these numbers: 40, 60, 20, 80", | |
| "What is 25 x 7?", | |
| "Quick question: 70 / 5", | |
| "What comes next in the sequence: 7, 17, 27, 37?", | |
| "What is 50% of 698, then add 28?", | |
| # physics (v79/v80) | |
| "Given mass 25 kg and acceleration 4 m/s^2, compute the force.", | |
| "A force of 580 N acts on a mass of 116 kg. What is the acceleration?", | |
| "mass 98 kg velocity 3 m/s find momentum", | |
| "What is the kinetic energy of a 12 kg mass moving at 5 m/s?", | |
| "Find the work done by 98 N acting over 2 m.", | |
| "What power corresponds to 1860 joules in 20 seconds?", | |
| "A current of 5 A flows through a resistance of 57 ohms. What is the voltage?", | |
| "A device runs at 12 V drawing 3 A. What is the electrical power?", | |
| "A wave with frequency 40 Hz has wavelength 6 m. What is its speed?", | |
| # chemistry (v79/v80) | |
| "What is the molarity of 4 mol of solute dissolved in 2 L?", | |
| # mathematics (v80) | |
| "In how many ways can 2 items be chosen from 30?", | |
| "An arithmetic series starts at 15 with common difference 4. " | |
| "What is the sum of the first 8 terms?", | |
| # code tracing (v87) -- checked by RUNNING the snippet rather than by | |
| # re-implementing it, which makes it the only parser here with no | |
| # second implementation to drift from the first. | |
| # | |
| # All nine code tasks are covered, measured at 108 checked / 0 wrong | |
| # / 0 unchecked over twelve samples each. Ordinary prose is still left | |
| # alone: "how are you today" and "x marks the spot" both return None. | |
| "What does x hold at the end? x = 2\nfor i in range(6): x = x + 8", | |
| "nums = [4, 10, 15]\nr = sum(nums) Give r.", | |
| "nums = [8, 3, 4]\nr = nums[0] + nums[2] What is r?", | |
| # v93 science and mathematics (build_omni_corpus.V93_TASKS) | |
| "A force of 80 N acts for 6 s. What is the impulse?", | |
| "A voltage of 376 V is applied across 8 ohm. What is the current?", | |
| "A spring of constant 18 N/m is stretched 4 m. Find the spring energy.", | |
| "Find the number of permutations of 9 things taken 2 at a time.", | |
| "A body moving at 12 m/s accelerates at 3 m/s^2 for 8 s. " | |
| "What is its final velocity?", | |
| # v93 code tracing (build_code_corpus.V93_TASKS): still one parser, | |
| # `_code_trace`, which now also runs `range(a, b)`, `.count` and a | |
| # negative index because the executor's allowlist admits them. | |
| "Trace this Python and give r. r = sum(range(3, 9))", | |
| "nums = [2, 4, 7, 4]\nr = nums.count(4) What is r?", | |
| "nums = [13, 8, 10]\nr = nums[-2] Give r.", | |
| # v93 connectome lookups (build_connectome_corpus.TASKS): the truth is | |
| # read from the CC-BY male-CNS tables the corpus was built from, so | |
| # these parse only where `datasets/v91_malecns` is on disk. | |
| "How many neurons of type KCg-m are in the male CNS?", | |
| "How many neurons of type KCg-m are on the left side of the male CNS?", | |
| "How many synapses go from type KCg-m to type PAM08 in the male CNS?", | |
| ] | |