File size: 39,331 Bytes
dca70d8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 | """Independently check a maths reply against the question that was asked.
The session's central finding is that only a *checkable* answer resists
recitation: a model reproducing a remembered reply to a novel problem is simply
wrong, and no amount of fluency hides it. `eval_problem_solving.py` applies that
offline, over problems it generates itself.
This module applies it live, to whatever the user typed. It re-derives the truth
from the question and compares, so the interface can say "wrong, the answer is
905" instead of presenting confident arithmetic and leaving the reader to check.
It recognises only the shapes the models were trained on -- twenty-one of them
as of v82, one per task in the v80 corpus: nine arithmetic shapes from v74 and
twelve solver-verified science and mathematics shapes from v79/v80. Anything
else returns `None`, which the interface must render as *not checked* rather
than as correct: an unrecognised question is not a passed one.
Multiplication, division, sequence and two-step were added for v74, which
introduced those tasks. Before that a chat reply to "What is 47 x 6?" showed
NOT CHECKED -- the model's strongest tasks were the ones nothing verified.
**This module is not a verifier and must never be promoted to one.** It
re-derives an expected answer from a *pattern* in the question, not from a
parse of its meaning, so a question whose shape it half-recognises would get a
confident wrong verdict. `nexus_epistemics.ANSWER_VERIFIER_IDS` is the
allowlist of things permitted to certify an answer and it contains exactly one
entry, `grounding_runtime.finalize_grounded_response`; `answer_check` is
deliberately absent from it. The compound-expression trap in
`_is_compound_expression` is the concrete reason: "What is 2 + 3 * 4?" once
parsed as multiplication with expected 12.0 where the truth is 14, because a
lone `A * B` search found `3 * 4` and never saw the `+`. That is now refused
as NOT CHECKED, and `test_answer_check.py` pins it in both directions.
v82 coverage, measured over 840 prompts drawn from `build_omni_corpus.TASKS`
and `eval_problem_solving.GENERATORS` at seed 4242: 799/840 = 0.951 before the
v82 widening, 840/840 = 1.000 after, with zero confident-wrong verdicts in
either. Coverage is not accuracy of the *model*; it is only the fraction of
questions this module is willing to judge at all.
v93 adds eleven benchmark tasks and this module gains a parser for every
shape among them: five science and mathematics forms re-derived from the
question (`impulse`, `ohms_current`, `spring_energy`, `permutations`,
`final_velocity`), three code forms the existing `_code_trace` already runs
once its snippet-start pattern admits `r = sum(range(3, 9))`, and three
connectome lookups (`cns_type_count`, `cns_side_count`, `cns_pair_synapses`)
whose "re-derivation" is the same table lookup the corpus builder did, from
the CC-BY male-CNS arrays. Measured at seed 4242 over all 41 registered tasks:
1,640/1,640 narrow-template prompts parsed and 4,920/4,920 prompts over the
full natural-phrasing bank (held-out forms included), zero confident-wrong
verdicts in either. `_permutations` runs before `_combination_choose` because
"permutations of 9 things taken 2 at a time" contains `taken`, and the
combination reading would have returned 36 for an answer of 72.
"""
from __future__ import annotations
import math
import re
from dataclasses import dataclass
from typing import Callable, List, Optional, Tuple
#: Matches the tolerance `eval_problem_solving.is_correct` uses, so the live
#: check and the benchmark cannot disagree about the same answer.
TOLERANCE = 1e-6
_NUMBER = re.compile(r"-?\d+(?:\.\d+)?(?:/\d+)?")
_DECIMAL = r"-?\d+(?:\.\d+)?"
@dataclass
class Check:
"""The verdict on one reply."""
task: str
expected: float
predicted: Optional[float]
correct: bool
def to_dict(self) -> dict:
return {
"task": self.task,
"expected": self.expected,
"predicted": self.predicted,
"correct": self.correct,
}
def extract_answer(text: str) -> Optional[float]:
"""The reply's answer is the last number it produces.
Identical rule to `eval_problem_solving.extract_answer`; the scratchpad
formats all end with the result, so "600 + 200 = 800, 17 + 88 = 105, total
905" reads as 905 rather than 600.
"""
matches = _NUMBER.findall(text.replace(",", ""))
if not matches:
return None
raw = matches[-1].rstrip(".")
try:
if "/" in raw:
numerator, denominator = raw.split("/", 1)
value = float(numerator) / float(denominator)
else:
value = float(raw)
return value if math.isfinite(value) else None
except (ValueError, ZeroDivisionError, OverflowError):
return None
# -- question parsers -------------------------------------------------------
#
# Each returns (task, expected) or None. They are deliberately narrow: a loose
# pattern that half-matched an unrelated question would produce a confident
# wrong verdict, which is worse than no verdict at all.
#: An operator sitting between two numbers -- the only kind that makes an
#: expression compound. `m/s`, `m/s^2` and `x + 14 = 39` do not match, because
#: neither side of the operator is a digit in the first two and `_algebra`
#: claims the third before these parsers run.
#: The trailing operand is a lookahead, not a consumed group: without it
#: `re.findall` over "2 + 3 * 4" consumes the `3` while matching `2 + 3` and
#: then finds only one operator, which is the bug this guard exists to catch.
_INFIX = re.compile(r"\d\s*[-+*/x]\s*(?=-?\d)", re.I)
def _is_compound_expression(question: str) -> bool:
"""True when the question chains two or more infix operators.
The bare `A op B` parsers below each find *one* operator and compute from
it. Given "What is 2 + 3 * 4?" the multiplication parser finds `3 * 4` and
returns 12.0, which is confidently wrong: precedence makes the answer 14.
Nothing in the corpus asks a compound question, so the correct response is
to refuse rather than to grow an expression evaluator here -- a partially
correct evaluator would produce exactly the confident wrong verdict this
module exists to avoid.
"""
return len(_INFIX.findall(question)) >= 2
def _binary(question: str) -> Optional[Tuple[str, float]]:
match = re.search(rf"(?<![\w.+-])({_DECIMAL})\s*([+-])\s*({_DECIMAL})(?!\w|\.\d)", question)
if not match or "=" in question or _is_compound_expression(question):
return None
left, op, right = float(match.group(1)), match.group(2), float(match.group(3))
return ("arithmetic", float(left + right if op == "+" else left - right))
def _percent(question: str) -> Optional[Tuple[str, float]]:
match = re.search(rf"({_DECIMAL})\s*%\s*of\s*({_DECIMAL})", question, re.I)
if not match:
return None
return ("percent", float(match.group(1)) * float(match.group(2)) / 100.0)
def _algebra(question: str) -> Optional[Tuple[str, float]]:
match = re.search(rf"\bx\s*([+*/-])\s*({_DECIMAL})\s*=\s*({_DECIMAL})(?!\w|\.\d)", question, re.I)
if not match:
return None
op, constant, right = match.group(1), float(match.group(2)), float(match.group(3))
if op in "*/" and constant == 0:
return None
if op == "+":
result = right - constant
elif op == "-":
result = right + constant
elif op == "*":
result = right / constant
else:
result = right * constant
return ("algebra_one_step", result)
def _average(question: str) -> Optional[Tuple[str, float]]:
if not re.search(r"\b(average|mean)\b", question, re.I):
return None
tail = question.split(":", 1)[-1]
values = [float(v) for v in re.findall(r"-?\d+(?:\.\d+)?", tail)]
if len(values) < 2:
return None
return ("average", sum(values) / len(values))
def _multiplication(question: str) -> Optional[Tuple[str, float]]:
"""`A x B`, the corpus's multiplication form.
Must run after `_algebra`: `x` is this corpus's multiplication sign *and*
its unknown, and only the digit on the left tells them apart.
"""
match = re.search(r"(-?\d+(?:\.\d+)?)\s*[x*]\s*(-?\d+(?:\.\d+)?)", question, re.I)
if not match or "=" in question or _is_compound_expression(question):
return None
return ("multiplication", float(match.group(1)) * float(match.group(2)))
def _division(question: str) -> Optional[Tuple[str, float]]:
match = re.search(r"(-?\d+(?:\.\d+)?)\s*/\s*(-?\d+(?:\.\d+)?)", question)
if not match or "=" in question or _is_compound_expression(question):
return None
divisor = float(match.group(2))
if divisor == 0:
# Not checkable rather than an exception; the question has no answer.
return None
return ("division", float(match.group(1)) / divisor)
#: The fractions `two_step` is allowed to state in words, and their percentages.
#: Kept in step with `build_scratchpad_math.PERCENT_FRACTIONS`; a form this does
#: not know falls through to `_percent`, which reads the percentage and ignores
#: the second operation, so it must not silently disagree.
_SPOKEN_FRACTIONS = {
"one tenth": 10.0, "one fifth": 20.0, "one quarter": 25.0, "one half": 50.0,
}
#: How the second operation can be worded. `reduce it by` and `take away` are
#: subtraction; `increase it by` is addition.
_ADD_WORDS = r"add|increase it by"
_SUBTRACT_WORDS = r"subtract|reduce it by|take away"
def _two_step(question: str) -> Optional[Tuple[str, float]]:
"""`P% of N`, then a second operation. Must precede `_percent`, which it contains.
v88 gave this task five prompt templates where it had one, and the extra
forms broke the original pattern in three ways at once: the percentage can
be spelled out (`Take 15 percent of 320`) or spoken as a fraction (`Start
with one quarter of 320`), the two clauses can be joined by `and then`, a
semicolon or a full stop, and the operation can be worded (`reduce it by`,
`increase it by`, `take away`).
A miss here is not harmless. `_two_step` runs before `_percent` precisely
because it contains a percent question, so a form this does not match is
read by `_percent` as `P% of N` -- the first half only -- and every such row
is then reported WRONG against a correct reply. That is what 174 of 20,391
checked rows did before this was widened, all of them `two_step`.
"""
base: Optional[float] = None
tail = question
match = re.search(
rf"({_DECIMAL})\s*(?:%|percent)\s*of\s*({_DECIMAL})", question, re.I)
if match:
base = float(match.group(1)) * float(match.group(2)) / 100.0
tail = question[match.end():]
else:
spoken = re.search(
rf"\b({'|'.join(_SPOKEN_FRACTIONS)})\s+of\s*({_DECIMAL})",
question, re.I)
if spoken:
base = (_SPOKEN_FRACTIONS[spoken.group(1).lower()]
* float(spoken.group(2)) / 100.0)
tail = question[spoken.end():]
if base is None:
return None
operation = re.search(
rf"(?:then|and then|next|,|;|\.)\s*({_ADD_WORDS}|{_SUBTRACT_WORDS})"
rf"\s*({_DECIMAL})", tail, re.I)
if not operation:
return None
word, operand = operation.group(1).lower(), float(operation.group(2))
adds = re.fullmatch(_ADD_WORDS, word, re.I) is not None
return ("two_step", base + operand if adds else base - operand)
def _sequence(question: str) -> Optional[Tuple[str, float]]:
"""The next term of an arithmetic progression.
Returns None when the differences are not constant. The corpus only
contains arithmetic progressions, so anything else is a question this
cannot verify -- and reporting "not checked" is correct where guessing a
rule would silently invent a right answer.
"""
if not re.search(r"\b(next|sequence)\b", question, re.I):
return None
tail = question.split(":", 1)[-1]
values = [float(v) for v in re.findall(r"-?\d+(?:\.\d+)?", tail)]
if len(values) < 3:
return None
steps = {round(b - a, 9) for a, b in zip(values, values[1:])}
if len(steps) != 1:
return None
return ("sequence", values[-1] + steps.pop())
# -- science shapes (v81) ---------------------------------------------------
#
# v80 answers physics correctly and the interface said NOT CHECKED for every
# one of them, because these shapes were never taught to the checker. A model
# whose strongest new capability cannot be verified live is the same gap v76
# closed for multiplication, one domain over.
#
# Each reads the quantities by name and unit, so it matches the corpus's
# phrasings without depending on any single one of them.
def _quantity(question: str, names: str, unit: str) -> Optional[float]:
"""Read `<name> <number> <unit>` in either order, as the corpus writes it."""
number = r"(-?\d+(?:\.\d+)?)"
# Both names and unit are alternations, so both must be grouped. Left
# bare, `(\d+)\s*m|metres?` parses as `(\d+)\s*m` OR `metres?` -- the
# second branch has no capture group, and match.group(1) is then None.
for pattern in (rf"(?:{names})\D{{0,24}}?{number}\s*(?:{unit})\b",
rf"{number}\s*(?:{unit})\b\D{{0,24}}?(?:{names})"):
match = re.search(pattern, question, re.I)
if match:
return float(match.group(1))
return None
#: (task, words identifying the target, quantity A, quantity B)
#: Division tasks are named in `_DIVISION_LAWS`; everything else multiplies.
#: Units carry their spelled-out forms. The corpus writes "57 volts and 5
#: amps" as readily as "57 V, 5 A", and a checker that only knew the symbols
#: reported NOT CHECKED for a third of the questions the model answers.
_MASS = r"kg|kilograms?"
_FORCE = r"N|newtons?"
_ANY = r"[a-z/^]*"
_PRODUCT_LAWS = (
("force", r"force",
(r"mass|body|block|object", _MASS), (r"accelerat\w*", r"m/s\^?2")),
("momentum", r"momentum",
(r"mass|object|body", _MASS), (r"velocity|speed|moves|travelling|at", r"m/s")),
# v82: "Find the work done by 98 N acting over 2 m." names the force only
# as "by". Measured 32/40 before adding it, 40/40 after.
("work", r"work",
(r"force|done by|by", _FORCE), (r"distance|moves|through|over|acts", r"m|metres?")),
("voltage", r"voltage|potential difference",
(r"current|flows|carrying|drives", r"A|amps?|amperes?"),
(r"resistance|ohm|through|across|resistor", r"ohms?")),
("electrical_power", r"electrical power|power dissipated|power|used at",
(r"voltage|volts?|runs at|at", r"V|volts?"),
(r"current|drawing|amps?|and", r"A|amps?|amperes?")),
("wave_speed", r"wave speed|speed of|its speed|speed at",
(r"frequency|at", r"Hz|hertz"), (r"wavelength|with", r"m|metres?")),
# v82: "produced by 580 N on 116 kg" names the force only as "by".
# Measured 35/40 before, 40/40 after.
("acceleration", r"acceleration|accelerat\w*",
(r"force|results from|from|produced by|by", _FORCE), (r"mass|body|object|on", _MASS)),
# v82: "What power corresponds to 1860 joules in 20 seconds?" names the
# work only as "corresponds to". Measured 28/40 before, 40/40 after.
("power", r"power",
(r"work|corresponds to|delivered|done", r"J|joules?"), (r"time|in|over", r"s|seconds?")),
("molarity", r"molarity|concentration|molar",
(r"mol|moles|solute|of", r"mol|moles"),
(r"volume|litres?|liters?|dissolved|in", r"L|litres?|liters?")),
# -- v93 --
# Placed after the nine above on purpose. `impulse` prompts always name a
# force, and `ohms_current` prompts always say "current" and usually
# "voltage", so the `force`, `voltage` and `electrical_power` laws see them
# first; each then fails on a quantity it needs (kg, amps) and falls
# through. The reverse cannot happen: a `voltage` prompt carries no number
# in volts and a `force` prompt no time in seconds, so neither of these
# can claim one of theirs.
("impulse", r"impulse",
(r"force|push|delivered by|acting|acts|by|from|:", _FORCE),
(r"time|for|over|lasting|during|in|,", r"s|seconds?")),
("ohms_current", r"current",
(r"voltage|volts?|at|drives|across|applied|when|:", r"V|volts?"),
(r"resistance|resistor|ohms?|through|across|drives|,", r"ohms?")),
)
_DIVISION_LAWS = frozenset({"acceleration", "power", "molarity", "ohms_current"})
def _science(question: str) -> Optional[Tuple[str, float]]:
for task, target, (a_names, a_unit), (b_names, b_unit) in _PRODUCT_LAWS:
if not re.search(target, question, re.I):
continue
a = _quantity(question, a_names, a_unit)
b = _quantity(question, b_names, b_unit)
if a is None or b is None:
continue
if task in _DIVISION_LAWS:
if b == 0:
return None # not checkable rather than an exception
return (task, a / b)
return (task, a * b)
return None
def _kinetic_energy(question: str) -> Optional[Tuple[str, float]]:
if not re.search(r"kinetic energy", question, re.I):
return None
mass = _quantity(question, r"mass|body", r"kg")
velocity = _quantity(question, r"velocity|speed|moves|at", r"m/s")
if mass is None or velocity is None:
return None
return ("kinetic_energy", 0.5 * mass * velocity * velocity)
def _spring_energy(question: str) -> Optional[Tuple[str, float]]:
"""`E = k x^2 / 2` (v93). Runs before `_science`, as `_kinetic_energy` does.
The spring constant is the number carrying `N/m`, which is unambiguous;
the extension is the number in metres that is *not* part of `N/m`, read
after an extension word so the `m` of `N/m` cannot be taken for it.
"""
if not re.search(r"spring energy|elastic potential energy|energy (?:is )?stored",
question, re.I):
return None
constant = re.search(r"(-?\d+(?:\.\d+)?)\s*N/m\b", question, re.I)
extension = _quantity(
question,
r"extension|extended|stretched|pulled|stretch|by|is",
r"m|metres?|meters?",
)
if constant is None or extension is None:
return None
k, x = float(constant.group(1)), float(extension)
return ("spring_energy", 0.5 * k * x * x)
def _final_velocity(question: str) -> Optional[Tuple[str, float]]:
"""`v = u + a t` (v93). Runs before `_science`.
Three numbers, three units, and the units tell them apart: the
acceleration carries `m/s^2`, the initial velocity `m/s` with no `^2`
after it, and the time a bare `s` or `seconds`. Nothing is read by
position, so any order of the three clauses parses the same way.
"""
if not re.search(r"final velocity|final speed|velocity after|going after",
question, re.I):
return None
accel = re.search(r"(-?\d+(?:\.\d+)?)\s*m/s\^?2\b", question, re.I)
initial = re.search(r"(-?\d+(?:\.\d+)?)\s*m/s\b(?!\^?2)", question, re.I)
# `(?<![/^])` keeps the `s` of `m/s` and the `2` of `m/s^2` out of it.
time = re.search(r"(?<![/^\w])(-?\d+(?:\.\d+)?)\s*(?:s|seconds?)\b(?!/|\^)",
question, re.I)
if accel is None or initial is None or time is None:
return None
u, a, t = (float(m.group(1)) for m in (initial, accel, time))
return ("final_velocity", u + a * t)
def _permutations(question: str) -> Optional[Tuple[str, float]]:
"""`P(n, k)`, however the corpus words it (v93).
Must run before `_combination_choose`: "permutations of 9 things taken 2
at a time" contains `taken`, and that parser would return C(9, 2) = 36
for a question whose answer is 72 -- the confident wrong verdict this
module exists to avoid. So any question that says permutation, arrange or
ordered is claimed here first, and a form this cannot read returns None
rather than falling through to the combination reading.
"""
if not re.search(r"permutation|arrang|ordered|line up", question, re.I):
return None
number = r"(\d+)"
# (pattern, reversed) -- `reversed` says the phrasing states k before n,
# as `_combination_choose` does. Every corpus and natural form is listed;
# the order matters only where two could match, and none do.
for pattern, reverse in (
(rf"n\s*=\s*{number}\s*k\s*=\s*{number}", False),
(rf"n\s*{number}\s*,?\s*k\s*{number}", False),
(rf"of\s*{number}\s*things taken\s*{number}", False),
(rf"of\s*{number}\s*(?:taken|take)\s*{number}", False),
(rf"{number}\s*things and line up\s*{number}", False),
(rf"can\s*{number}\s*items? be arranged in order from\s*{number}", True),
(rf"arrangements? of\s*{number}\s*(?:are there )?from\s*{number}", True),
(rf"arrange\s*{number}\s*out of\s*{number}", True),
(rf"arranging\s*{number}\s*out of\s*{number}", True),
(rf"{number}\s*items? can be picked from\s*{number}", True),
(rf"selections? of\s*{number}\s*from\s*{number}", True),
):
match = re.search(pattern, question, re.I)
if not match:
continue
a, b = int(match.group(1)), int(match.group(2))
n, k = (b, a) if reverse else (a, b)
if k > n:
return None # far more likely misread than genuinely 0
import math as _math
return ("permutations", float(_math.perm(n, k)))
return None
# -- connectome lookups (v93) ------------------------------------------------
#
# The three cns_* shapes ask for a fact, not a computation, so the only honest
# re-derivation is the same lookup the corpus builder did: the answer comes
# from `build_connectome_corpus.population()`, built from the CC-BY male-CNS
# tables. A type or pair outside that population returns None (not checked)
# rather than 0, because "not in the table we teach from" is not "zero
# neurons". Where the tables are absent the parsers are inert.
#: A type name as the corpus writes it: letters, digits, `_` and `-`. The
#: trailing `[A-Za-z0-9_]` keeps a sentence-final hyphen or the `?` out.
_TYPE = r"([A-Za-z][A-Za-z0-9_\-]*[A-Za-z0-9_]|[A-Za-z])"
def _connectome_population():
try:
import build_connectome_corpus as cns
except Exception: # noqa: BLE001 - the checker degrades, it does not break
return None
try:
if not cns.data_available():
return None
return cns.population()
except Exception: # noqa: BLE001
return None
def _cns_pair_synapses(question: str) -> Optional[Tuple[str, float]]:
if not re.search(r"synapse", question, re.I) or not re.search(r"male CNS", question, re.I):
return None
for pattern in (
rf"from (?:cell )?type {_TYPE} (?:to|onto) (?:cell )?type {_TYPE}",
rf"type {_TYPE} makes? onto type {_TYPE}",
rf"from {_TYPE} to {_TYPE}\b",
):
match = re.search(pattern, question)
if match:
break
else:
return None
pop = _connectome_population()
if pop is None:
return None
pre, post = match.group(1), match.group(2)
for a, b, weight in pop.pairs:
if a == pre and b == post:
return ("cns_pair_synapses", float(weight))
return None
def _cns_side_count(question: str) -> Optional[Tuple[str, float]]:
if not re.search(r"neurons?", question, re.I) or not re.search(r"male CNS", question, re.I):
return None
side = re.search(r"\b(left|right)\b", question, re.I)
if side is None:
return None
name = _cns_type_name(question)
if name is None:
return None
pop = _connectome_population()
if pop is None or name not in pop.n_neurons:
return None
table = pop.left if side.group(1).lower() == "left" else pop.right
return ("cns_side_count", float(table[name]))
def _cns_type_count(question: str) -> Optional[Tuple[str, float]]:
if not re.search(r"neurons?", question, re.I) or not re.search(r"male CNS", question, re.I):
return None
if re.search(r"\b(left|right|side|hemisphere|synapse)", question, re.I):
return None # a side count or a pair count, not this shape
name = _cns_type_name(question)
if name is None:
return None
pop = _connectome_population()
if pop is None or name not in pop.n_neurons:
return None
return ("cns_type_count", float(pop.n_neurons[name]))
def _cns_type_name(question: str) -> Optional[str]:
"""The one type a count question names, or None when the form is unknown."""
for pattern in (
rf"(?:cell )?type {_TYPE}",
rf"(?:the |many )?{_TYPE} neurons",
):
match = re.search(pattern, question)
if match:
return match.group(1)
return None
def _combination_choose(question: str) -> Optional[Tuple[str, float]]:
"""`n choose k`, however the corpus words it.
The corpus fixes k at 2 so the working can be shown, but this reads
whatever k is stated rather than assuming it -- an assumption here would
produce a confident wrong verdict on any other k.
"""
if not re.search(r"combination|choose|chosen|taken", question, re.I):
return None
number = r"(\d+)"
# (pattern, reversed) -- `reversed` says the phrasing states k before n.
# v82: this used to pick n = max(a, b) for *every* phrasing, so "30 choose
# 40" returned C(40, 30) = 847660528 where the truth is 0. A size heuristic
# cannot tell an impossible question from a reversed one; the word order
# can, and each of the corpus's four phrasings has a fixed order.
for pattern, reverse in (
(rf"{number}\s*choose\s*{number}", False),
(rf"n\s*=\s*{number}\s*k\s*=\s*{number}", False),
(rf"of\s*{number}\s*things taken\s*{number}", False),
(rf"can\s*{number}\s*items? be chosen from\s*{number}", True),
):
match = re.search(pattern, question, re.I)
if not match:
continue
a, b = int(match.group(1)), int(match.group(2))
n, k = (b, a) if reverse else (a, b)
if k > n:
# C(n, k) is 0 here, but a question asking to choose 40 from 30 is
# far more likely to be one this parser has misread than one whose
# answer is genuinely 0. NOT CHECKED is the honest verdict.
return None
import math as _math
return ("combination", float(_math.comb(n, k)))
return None
def _arithmetic_series(question: str) -> Optional[Tuple[str, float]]:
"""Sum of the first n terms of an arithmetic progression."""
if not re.search(r"arithmetic (?:series|progression)", question, re.I):
return None
# v82: "An arithmetic series starts at 15 with common difference 4" never
# says "first term". Measured 24/40 before adding `starts at`, 40/40 after.
first = re.search(r"(?:first term|starts? at|beginning at)\s*(?:is\s*)?(-?\d+)",
question, re.I)
difference = re.search(r"(?:common )?difference\s*(?:of\s*)?(-?\d+)", question, re.I)
terms = re.search(r"(?:sum of|first)\s*(\d+)\s*terms|(?:\bn\s*(\d+))", question, re.I)
if not (first and difference and terms):
return None
count = int(terms.group(1) or terms.group(2))
a, d = int(first.group(1)), int(difference.group(1))
last = a + (count - 1) * d
return ("arithmetic_series", float(count * (a + last) / 2))
def _word_problem(question: str) -> Optional[Tuple[str, float]]:
match = re.search(
r"has\s+(\d+).*?get\s+(\d+)\s+more.*?give\s+away\s+(\d+)", question, re.I | re.S
)
if not match:
return None
start, gain, lose = (int(match.group(i)) for i in (1, 2, 3))
return ("word_problem", float(start + gain - lose))
#: Order matters, and every entry below is placed against a specific ambiguity:
#:
#: * `_word_problem` and `_average` precede everything numeric, because both
#: contain bare numbers a naive "a + b" search would seize on.
#: * `_sequence` precedes them too -- "7, 17, 27, 37" is a comma-separated list
#: of numbers, which is exactly what an average looks like.
#: * `_two_step` precedes `_percent` because it *contains* a percent question.
#: * `_algebra` precedes `_multiplication` because `x` is both this corpus's
#: multiplication sign and its unknown.
#: * `_kinetic_energy` and `_science` precede the bare-number parsers, because
#: a physics question carries two numbers and a naive `a x b` search would
#: seize on them without knowing which law applies. `_kinetic_energy` runs
#: first of the two: it names a mass and a velocity, which is also what
#: `momentum` matches on.
def _code_trace(question: str) -> Optional[Tuple[str, float]]:
"""Re-derive a code-tracing answer by executing the snippet in the question.
Every other parser here re-implements the arithmetic it checks. This one
does not need to: the question *contains* the program, so the check is to
run it. That makes it the strongest verifier in this module -- there is no
second implementation to drift from the first.
Extraction is deliberately narrow. The snippet is recovered by finding the
first assignment to a single-letter variable and taking everything from
there to the trailing question, then the same allowlisted, builtins-free,
timeout-bounded executor the corpus builder uses decides the value. A
question this cannot confidently parse returns ``None`` -- *not checked* --
which is the only safe failure for this module.
"""
try:
import build_code_corpus as code
except Exception: # noqa: BLE001 - the checker degrades, it does not break
return None
# The wrapper names the variable being asked about. Every template phrases
# it one of these ways, and a question naming none of them is not a code
# question this should touch.
target = None
for pattern in (r"value of ([A-Za-z_]\w*)", r"give (?:the value of )?([A-Za-z_]\w*)",
r"what (?:is|does) ([A-Za-z_]\w*)", r"final ([A-Za-z_]\w*)",
r"([A-Za-z_]\w*) after this runs", r"([A-Za-z_]\w*) hold at the end",
r"\b([A-Za-z_]\w*)\?\s*$"):
found = re.search(pattern, question, flags=re.IGNORECASE)
if found:
target = found.group(1)
break
if target is None:
return None
# The code is the longest span that parses AND assigns that variable. Trying
# spans rather than pattern-matching the wrapper means a new phrasing cannot
# silently produce a wrong value: a mis-trimmed span fails to parse, and a
# span that parses but never assigns the target is rejected below.
# The right-hand side of the first assignment is a literal (list, number,
# negative number) or, since v93's `code_range_sum` (`r = sum(range(3,
# 9))`), a call to a lower-case name. Which names are callable is the
# executor's decision, not this pattern's: anything else is refused there.
start = re.search(r"[A-Za-z_]\w*\s*=\s*(?:[\[\-\d]|[a-z]+\()", question)
if start is None:
return None
body = question[start.start():]
# Candidate spans end at a whitespace boundary, longest first. An earlier
# version required the span to end in a digit or a bracket, which silently
# excluded `code_conditional` -- its snippet ends `else b - a`, on a
# variable name -- and left two of nine tasks unverifiable. Correctness
# comes from the executor accepting the span, not from guessing where the
# prose starts, so the heuristic is gone and only the boundary remains.
pieces = body.split(" ")
for size in range(len(pieces), 0, -1):
chunk = " ".join(pieces[:size]).rstrip(" ?.")
if not chunk or not re.search(rf"\b{re.escape(target)}\s*=", chunk):
continue
result = code.run_snippet(chunk, target)
if result.ok and result.value is not None:
return ("code_trace", float(result.value))
return None
PARSERS: Tuple[Callable[[str], Optional[Tuple[str, float]]], ...] = (
# First: a code question contains numbers and operators that several
# parsers below would happily misread as arithmetic. `x = 5` and
# `range(4)` look like a binary expression to `_binary`.
_code_trace,
# v93: the connectome lookups key on "male CNS" and refuse everything
# else, and a type name such as `5-HTPLP01` or `IN17A052` is exactly the
# kind of string `_binary` and `_multiplication` would misread as an
# expression, so they run before any numeric parser.
_cns_pair_synapses,
_cns_side_count,
_cns_type_count,
_word_problem,
# v93: before `_combination_choose`, which would otherwise read
# "permutations of 9 things taken 2 at a time" as C(9, 2).
_permutations,
_combination_choose,
_arithmetic_series,
_kinetic_energy,
# v93: before `_science`, whose `acceleration` law matches "accelerates"
# and whose `work` law matches "work out the spring energy"; both fall
# through on a missing quantity today, and these make that not matter.
_spring_energy,
_final_velocity,
_science,
_sequence,
_average,
_two_step,
_algebra,
_percent,
_division,
_multiplication,
_binary,
)
def parse_question(question: str) -> Optional[Tuple[str, float]]:
for parser in PARSERS:
try:
result = parser(question)
except (ValueError, ZeroDivisionError, OverflowError):
return None
if result is not None:
return result if math.isfinite(result[1]) else None
return None
def check(question: str, reply: str) -> Optional[Check]:
"""Verify a reply, or return ``None`` when the question is not checkable.
``None`` means *not checked*. The caller must not render it as correct: the
whole value of this is that a wrong answer is visibly wrong, and quietly
passing anything unrecognised would destroy that.
"""
parsed = parse_question(question)
if parsed is None:
return None
task, expected = parsed
predicted = extract_answer(reply)
correct = (
predicted is not None
and abs(predicted - expected) <= max(TOLERANCE, abs(expected) * 1e-6)
)
return Check(task=task, expected=expected, predicted=predicted, correct=correct)
def supported_shapes() -> List[str]:
"""The question forms this can verify, for the interface to advertise.
Through v81 this listed only the nine arithmetic shapes while `PARSERS`
already handled twelve science and mathematics shapes, so the interface
under-advertised what it could check by more than half. Every entry here is
asserted parseable by `test_answer_check.py`, so the list cannot drift
ahead of the parsers again -- but note it can still drift *behind* them,
which is the harmless direction.
"""
return [
# arithmetic (v74)
"Solve this basic math problem: 617 + 288",
"What is 25% of 840?",
"Solve for x: x + 14 = 39",
"A student has 45 marbles. They get 38 more and then give away 27. How many marbles do they have now?",
"Find the average (mean) of these numbers: 40, 60, 20, 80",
"What is 25 x 7?",
"Quick question: 70 / 5",
"What comes next in the sequence: 7, 17, 27, 37?",
"What is 50% of 698, then add 28?",
# physics (v79/v80)
"Given mass 25 kg and acceleration 4 m/s^2, compute the force.",
"A force of 580 N acts on a mass of 116 kg. What is the acceleration?",
"mass 98 kg velocity 3 m/s find momentum",
"What is the kinetic energy of a 12 kg mass moving at 5 m/s?",
"Find the work done by 98 N acting over 2 m.",
"What power corresponds to 1860 joules in 20 seconds?",
"A current of 5 A flows through a resistance of 57 ohms. What is the voltage?",
"A device runs at 12 V drawing 3 A. What is the electrical power?",
"A wave with frequency 40 Hz has wavelength 6 m. What is its speed?",
# chemistry (v79/v80)
"What is the molarity of 4 mol of solute dissolved in 2 L?",
# mathematics (v80)
"In how many ways can 2 items be chosen from 30?",
"An arithmetic series starts at 15 with common difference 4. "
"What is the sum of the first 8 terms?",
# code tracing (v87) -- checked by RUNNING the snippet rather than by
# re-implementing it, which makes it the only parser here with no
# second implementation to drift from the first.
#
# All nine code tasks are covered, measured at 108 checked / 0 wrong
# / 0 unchecked over twelve samples each. Ordinary prose is still left
# alone: "how are you today" and "x marks the spot" both return None.
"What does x hold at the end? x = 2\nfor i in range(6): x = x + 8",
"nums = [4, 10, 15]\nr = sum(nums) Give r.",
"nums = [8, 3, 4]\nr = nums[0] + nums[2] What is r?",
# v93 science and mathematics (build_omni_corpus.V93_TASKS)
"A force of 80 N acts for 6 s. What is the impulse?",
"A voltage of 376 V is applied across 8 ohm. What is the current?",
"A spring of constant 18 N/m is stretched 4 m. Find the spring energy.",
"Find the number of permutations of 9 things taken 2 at a time.",
"A body moving at 12 m/s accelerates at 3 m/s^2 for 8 s. "
"What is its final velocity?",
# v93 code tracing (build_code_corpus.V93_TASKS): still one parser,
# `_code_trace`, which now also runs `range(a, b)`, `.count` and a
# negative index because the executor's allowlist admits them.
"Trace this Python and give r. r = sum(range(3, 9))",
"nums = [2, 4, 7, 4]\nr = nums.count(4) What is r?",
"nums = [13, 8, 10]\nr = nums[-2] Give r.",
# v93 connectome lookups (build_connectome_corpus.TASKS): the truth is
# read from the CC-BY male-CNS tables the corpus was built from, so
# these parse only where `datasets/v91_malecns` is on disk.
"How many neurons of type KCg-m are in the male CNS?",
"How many neurons of type KCg-m are on the left side of the male CNS?",
"How many synapses go from type KCg-m to type PAM08 in the male CNS?",
]
|