File size: 39,331 Bytes
dca70d8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
"""Independently check a maths reply against the question that was asked.



The session's central finding is that only a *checkable* answer resists

recitation: a model reproducing a remembered reply to a novel problem is simply

wrong, and no amount of fluency hides it. `eval_problem_solving.py` applies that

offline, over problems it generates itself.



This module applies it live, to whatever the user typed. It re-derives the truth

from the question and compares, so the interface can say "wrong, the answer is

905" instead of presenting confident arithmetic and leaving the reader to check.



It recognises only the shapes the models were trained on -- twenty-one of them

as of v82, one per task in the v80 corpus: nine arithmetic shapes from v74 and

twelve solver-verified science and mathematics shapes from v79/v80. Anything

else returns `None`, which the interface must render as *not checked* rather

than as correct: an unrecognised question is not a passed one.



Multiplication, division, sequence and two-step were added for v74, which

introduced those tasks. Before that a chat reply to "What is 47 x 6?" showed

NOT CHECKED -- the model's strongest tasks were the ones nothing verified.



**This module is not a verifier and must never be promoted to one.** It

re-derives an expected answer from a *pattern* in the question, not from a

parse of its meaning, so a question whose shape it half-recognises would get a

confident wrong verdict. `nexus_epistemics.ANSWER_VERIFIER_IDS` is the

allowlist of things permitted to certify an answer and it contains exactly one

entry, `grounding_runtime.finalize_grounded_response`; `answer_check` is

deliberately absent from it. The compound-expression trap in

`_is_compound_expression` is the concrete reason: "What is 2 + 3 * 4?" once

parsed as multiplication with expected 12.0 where the truth is 14, because a

lone `A * B` search found `3 * 4` and never saw the `+`. That is now refused

as NOT CHECKED, and `test_answer_check.py` pins it in both directions.



v82 coverage, measured over 840 prompts drawn from `build_omni_corpus.TASKS`

and `eval_problem_solving.GENERATORS` at seed 4242: 799/840 = 0.951 before the

v82 widening, 840/840 = 1.000 after, with zero confident-wrong verdicts in

either. Coverage is not accuracy of the *model*; it is only the fraction of

questions this module is willing to judge at all.



v93 adds eleven benchmark tasks and this module gains a parser for every

shape among them: five science and mathematics forms re-derived from the

question (`impulse`, `ohms_current`, `spring_energy`, `permutations`,

`final_velocity`), three code forms the existing `_code_trace` already runs

once its snippet-start pattern admits `r = sum(range(3, 9))`, and three

connectome lookups (`cns_type_count`, `cns_side_count`, `cns_pair_synapses`)

whose "re-derivation" is the same table lookup the corpus builder did, from

the CC-BY male-CNS arrays. Measured at seed 4242 over all 41 registered tasks:

1,640/1,640 narrow-template prompts parsed and 4,920/4,920 prompts over the

full natural-phrasing bank (held-out forms included), zero confident-wrong

verdicts in either. `_permutations` runs before `_combination_choose` because

"permutations of 9 things taken 2 at a time" contains `taken`, and the

combination reading would have returned 36 for an answer of 72.

"""

from __future__ import annotations

import math
import re
from dataclasses import dataclass
from typing import Callable, List, Optional, Tuple

#: Matches the tolerance `eval_problem_solving.is_correct` uses, so the live
#: check and the benchmark cannot disagree about the same answer.
TOLERANCE = 1e-6

_NUMBER = re.compile(r"-?\d+(?:\.\d+)?(?:/\d+)?")
_DECIMAL = r"-?\d+(?:\.\d+)?"


@dataclass
class Check:
    """The verdict on one reply."""

    task: str
    expected: float
    predicted: Optional[float]
    correct: bool

    def to_dict(self) -> dict:
        return {
            "task": self.task,
            "expected": self.expected,
            "predicted": self.predicted,
            "correct": self.correct,
        }


def extract_answer(text: str) -> Optional[float]:
    """The reply's answer is the last number it produces.



    Identical rule to `eval_problem_solving.extract_answer`; the scratchpad

    formats all end with the result, so "600 + 200 = 800, 17 + 88 = 105, total

    905" reads as 905 rather than 600.

    """

    matches = _NUMBER.findall(text.replace(",", ""))
    if not matches:
        return None
    raw = matches[-1].rstrip(".")
    try:
        if "/" in raw:
            numerator, denominator = raw.split("/", 1)
            value = float(numerator) / float(denominator)
        else:
            value = float(raw)
        return value if math.isfinite(value) else None
    except (ValueError, ZeroDivisionError, OverflowError):
        return None


# -- question parsers -------------------------------------------------------
#
# Each returns (task, expected) or None. They are deliberately narrow: a loose
# pattern that half-matched an unrelated question would produce a confident
# wrong verdict, which is worse than no verdict at all.


#: An operator sitting between two numbers -- the only kind that makes an
#: expression compound. `m/s`, `m/s^2` and `x + 14 = 39` do not match, because
#: neither side of the operator is a digit in the first two and `_algebra`
#: claims the third before these parsers run.
#: The trailing operand is a lookahead, not a consumed group: without it
#: `re.findall` over "2 + 3 * 4" consumes the `3` while matching `2 + 3` and
#: then finds only one operator, which is the bug this guard exists to catch.
_INFIX = re.compile(r"\d\s*[-+*/x]\s*(?=-?\d)", re.I)


def _is_compound_expression(question: str) -> bool:
    """True when the question chains two or more infix operators.



    The bare `A op B` parsers below each find *one* operator and compute from

    it. Given "What is 2 + 3 * 4?" the multiplication parser finds `3 * 4` and

    returns 12.0, which is confidently wrong: precedence makes the answer 14.

    Nothing in the corpus asks a compound question, so the correct response is

    to refuse rather than to grow an expression evaluator here -- a partially

    correct evaluator would produce exactly the confident wrong verdict this

    module exists to avoid.

    """

    return len(_INFIX.findall(question)) >= 2


def _binary(question: str) -> Optional[Tuple[str, float]]:
    match = re.search(rf"(?<![\w.+-])({_DECIMAL})\s*([+-])\s*({_DECIMAL})(?!\w|\.\d)", question)
    if not match or "=" in question or _is_compound_expression(question):
        return None
    left, op, right = float(match.group(1)), match.group(2), float(match.group(3))
    return ("arithmetic", float(left + right if op == "+" else left - right))


def _percent(question: str) -> Optional[Tuple[str, float]]:
    match = re.search(rf"({_DECIMAL})\s*%\s*of\s*({_DECIMAL})", question, re.I)
    if not match:
        return None
    return ("percent", float(match.group(1)) * float(match.group(2)) / 100.0)


def _algebra(question: str) -> Optional[Tuple[str, float]]:
    match = re.search(rf"\bx\s*([+*/-])\s*({_DECIMAL})\s*=\s*({_DECIMAL})(?!\w|\.\d)", question, re.I)
    if not match:
        return None
    op, constant, right = match.group(1), float(match.group(2)), float(match.group(3))
    if op in "*/" and constant == 0:
        return None
    if op == "+":
        result = right - constant
    elif op == "-":
        result = right + constant
    elif op == "*":
        result = right / constant
    else:
        result = right * constant
    return ("algebra_one_step", result)


def _average(question: str) -> Optional[Tuple[str, float]]:
    if not re.search(r"\b(average|mean)\b", question, re.I):
        return None
    tail = question.split(":", 1)[-1]
    values = [float(v) for v in re.findall(r"-?\d+(?:\.\d+)?", tail)]
    if len(values) < 2:
        return None
    return ("average", sum(values) / len(values))


def _multiplication(question: str) -> Optional[Tuple[str, float]]:
    """`A x B`, the corpus's multiplication form.



    Must run after `_algebra`: `x` is this corpus's multiplication sign *and*

    its unknown, and only the digit on the left tells them apart.

    """

    match = re.search(r"(-?\d+(?:\.\d+)?)\s*[x*]\s*(-?\d+(?:\.\d+)?)", question, re.I)
    if not match or "=" in question or _is_compound_expression(question):
        return None
    return ("multiplication", float(match.group(1)) * float(match.group(2)))


def _division(question: str) -> Optional[Tuple[str, float]]:
    match = re.search(r"(-?\d+(?:\.\d+)?)\s*/\s*(-?\d+(?:\.\d+)?)", question)
    if not match or "=" in question or _is_compound_expression(question):
        return None
    divisor = float(match.group(2))
    if divisor == 0:
        # Not checkable rather than an exception; the question has no answer.
        return None
    return ("division", float(match.group(1)) / divisor)


#: The fractions `two_step` is allowed to state in words, and their percentages.
#: Kept in step with `build_scratchpad_math.PERCENT_FRACTIONS`; a form this does
#: not know falls through to `_percent`, which reads the percentage and ignores
#: the second operation, so it must not silently disagree.
_SPOKEN_FRACTIONS = {
    "one tenth": 10.0, "one fifth": 20.0, "one quarter": 25.0, "one half": 50.0,
}

#: How the second operation can be worded. `reduce it by` and `take away` are
#: subtraction; `increase it by` is addition.
_ADD_WORDS = r"add|increase it by"
_SUBTRACT_WORDS = r"subtract|reduce it by|take away"


def _two_step(question: str) -> Optional[Tuple[str, float]]:
    """`P% of N`, then a second operation. Must precede `_percent`, which it contains.



    v88 gave this task five prompt templates where it had one, and the extra

    forms broke the original pattern in three ways at once: the percentage can

    be spelled out (`Take 15 percent of 320`) or spoken as a fraction (`Start

    with one quarter of 320`), the two clauses can be joined by `and then`, a

    semicolon or a full stop, and the operation can be worded (`reduce it by`,

    `increase it by`, `take away`).



    A miss here is not harmless. `_two_step` runs before `_percent` precisely

    because it contains a percent question, so a form this does not match is

    read by `_percent` as `P% of N` -- the first half only -- and every such row

    is then reported WRONG against a correct reply. That is what 174 of 20,391

    checked rows did before this was widened, all of them `two_step`.

    """

    base: Optional[float] = None
    tail = question

    match = re.search(
        rf"({_DECIMAL})\s*(?:%|percent)\s*of\s*({_DECIMAL})", question, re.I)
    if match:
        base = float(match.group(1)) * float(match.group(2)) / 100.0
        tail = question[match.end():]
    else:
        spoken = re.search(
            rf"\b({'|'.join(_SPOKEN_FRACTIONS)})\s+of\s*({_DECIMAL})",
            question, re.I)
        if spoken:
            base = (_SPOKEN_FRACTIONS[spoken.group(1).lower()]
                    * float(spoken.group(2)) / 100.0)
            tail = question[spoken.end():]
    if base is None:
        return None

    operation = re.search(
        rf"(?:then|and then|next|,|;|\.)\s*({_ADD_WORDS}|{_SUBTRACT_WORDS})"
        rf"\s*({_DECIMAL})", tail, re.I)
    if not operation:
        return None
    word, operand = operation.group(1).lower(), float(operation.group(2))
    adds = re.fullmatch(_ADD_WORDS, word, re.I) is not None
    return ("two_step", base + operand if adds else base - operand)


def _sequence(question: str) -> Optional[Tuple[str, float]]:
    """The next term of an arithmetic progression.



    Returns None when the differences are not constant. The corpus only

    contains arithmetic progressions, so anything else is a question this

    cannot verify -- and reporting "not checked" is correct where guessing a

    rule would silently invent a right answer.

    """

    if not re.search(r"\b(next|sequence)\b", question, re.I):
        return None
    tail = question.split(":", 1)[-1]
    values = [float(v) for v in re.findall(r"-?\d+(?:\.\d+)?", tail)]
    if len(values) < 3:
        return None
    steps = {round(b - a, 9) for a, b in zip(values, values[1:])}
    if len(steps) != 1:
        return None
    return ("sequence", values[-1] + steps.pop())


# -- science shapes (v81) ---------------------------------------------------
#
# v80 answers physics correctly and the interface said NOT CHECKED for every
# one of them, because these shapes were never taught to the checker. A model
# whose strongest new capability cannot be verified live is the same gap v76
# closed for multiplication, one domain over.
#
# Each reads the quantities by name and unit, so it matches the corpus's
# phrasings without depending on any single one of them.

def _quantity(question: str, names: str, unit: str) -> Optional[float]:
    """Read `<name> <number> <unit>` in either order, as the corpus writes it."""

    number = r"(-?\d+(?:\.\d+)?)"
    # Both names and unit are alternations, so both must be grouped. Left
    # bare, `(\d+)\s*m|metres?` parses as `(\d+)\s*m` OR `metres?` -- the
    # second branch has no capture group, and match.group(1) is then None.
    for pattern in (rf"(?:{names})\D{{0,24}}?{number}\s*(?:{unit})\b",
                    rf"{number}\s*(?:{unit})\b\D{{0,24}}?(?:{names})"):
        match = re.search(pattern, question, re.I)
        if match:
            return float(match.group(1))
    return None


#: (task, words identifying the target, quantity A, quantity B)
#: Division tasks are named in `_DIVISION_LAWS`; everything else multiplies.
#: Units carry their spelled-out forms. The corpus writes "57 volts and 5
#: amps" as readily as "57 V, 5 A", and a checker that only knew the symbols
#: reported NOT CHECKED for a third of the questions the model answers.
_MASS = r"kg|kilograms?"
_FORCE = r"N|newtons?"
_ANY = r"[a-z/^]*"

_PRODUCT_LAWS = (
    ("force", r"force",
     (r"mass|body|block|object", _MASS), (r"accelerat\w*", r"m/s\^?2")),
    ("momentum", r"momentum",
     (r"mass|object|body", _MASS), (r"velocity|speed|moves|travelling|at", r"m/s")),
    # v82: "Find the work done by 98 N acting over 2 m." names the force only
    # as "by". Measured 32/40 before adding it, 40/40 after.
    ("work", r"work",
     (r"force|done by|by", _FORCE), (r"distance|moves|through|over|acts", r"m|metres?")),
    ("voltage", r"voltage|potential difference",
     (r"current|flows|carrying|drives", r"A|amps?|amperes?"),
     (r"resistance|ohm|through|across|resistor", r"ohms?")),
    ("electrical_power", r"electrical power|power dissipated|power|used at",
     (r"voltage|volts?|runs at|at", r"V|volts?"),
     (r"current|drawing|amps?|and", r"A|amps?|amperes?")),
    ("wave_speed", r"wave speed|speed of|its speed|speed at",
     (r"frequency|at", r"Hz|hertz"), (r"wavelength|with", r"m|metres?")),
    # v82: "produced by 580 N on 116 kg" names the force only as "by".
    # Measured 35/40 before, 40/40 after.
    ("acceleration", r"acceleration|accelerat\w*",
     (r"force|results from|from|produced by|by", _FORCE), (r"mass|body|object|on", _MASS)),
    # v82: "What power corresponds to 1860 joules in 20 seconds?" names the
    # work only as "corresponds to". Measured 28/40 before, 40/40 after.
    ("power", r"power",
     (r"work|corresponds to|delivered|done", r"J|joules?"), (r"time|in|over", r"s|seconds?")),
    ("molarity", r"molarity|concentration|molar",
     (r"mol|moles|solute|of", r"mol|moles"),
     (r"volume|litres?|liters?|dissolved|in", r"L|litres?|liters?")),
    # -- v93 --
    # Placed after the nine above on purpose. `impulse` prompts always name a
    # force, and `ohms_current` prompts always say "current" and usually
    # "voltage", so the `force`, `voltage` and `electrical_power` laws see them
    # first; each then fails on a quantity it needs (kg, amps) and falls
    # through. The reverse cannot happen: a `voltage` prompt carries no number
    # in volts and a `force` prompt no time in seconds, so neither of these
    # can claim one of theirs.
    ("impulse", r"impulse",
     (r"force|push|delivered by|acting|acts|by|from|:", _FORCE),
     (r"time|for|over|lasting|during|in|,", r"s|seconds?")),
    ("ohms_current", r"current",
     (r"voltage|volts?|at|drives|across|applied|when|:", r"V|volts?"),
     (r"resistance|resistor|ohms?|through|across|drives|,", r"ohms?")),
)


_DIVISION_LAWS = frozenset({"acceleration", "power", "molarity", "ohms_current"})


def _science(question: str) -> Optional[Tuple[str, float]]:
    for task, target, (a_names, a_unit), (b_names, b_unit) in _PRODUCT_LAWS:
        if not re.search(target, question, re.I):
            continue
        a = _quantity(question, a_names, a_unit)
        b = _quantity(question, b_names, b_unit)
        if a is None or b is None:
            continue
        if task in _DIVISION_LAWS:
            if b == 0:
                return None   # not checkable rather than an exception
            return (task, a / b)
        return (task, a * b)
    return None


def _kinetic_energy(question: str) -> Optional[Tuple[str, float]]:
    if not re.search(r"kinetic energy", question, re.I):
        return None
    mass = _quantity(question, r"mass|body", r"kg")
    velocity = _quantity(question, r"velocity|speed|moves|at", r"m/s")
    if mass is None or velocity is None:
        return None
    return ("kinetic_energy", 0.5 * mass * velocity * velocity)


def _spring_energy(question: str) -> Optional[Tuple[str, float]]:
    """`E = k x^2 / 2` (v93). Runs before `_science`, as `_kinetic_energy` does.



    The spring constant is the number carrying `N/m`, which is unambiguous;

    the extension is the number in metres that is *not* part of `N/m`, read

    after an extension word so the `m` of `N/m` cannot be taken for it.

    """

    if not re.search(r"spring energy|elastic potential energy|energy (?:is )?stored",
                     question, re.I):
        return None
    constant = re.search(r"(-?\d+(?:\.\d+)?)\s*N/m\b", question, re.I)
    extension = _quantity(
        question,
        r"extension|extended|stretched|pulled|stretch|by|is",
        r"m|metres?|meters?",
    )
    if constant is None or extension is None:
        return None
    k, x = float(constant.group(1)), float(extension)
    return ("spring_energy", 0.5 * k * x * x)


def _final_velocity(question: str) -> Optional[Tuple[str, float]]:
    """`v = u + a t` (v93). Runs before `_science`.



    Three numbers, three units, and the units tell them apart: the

    acceleration carries `m/s^2`, the initial velocity `m/s` with no `^2`

    after it, and the time a bare `s` or `seconds`. Nothing is read by

    position, so any order of the three clauses parses the same way.

    """

    if not re.search(r"final velocity|final speed|velocity after|going after",
                     question, re.I):
        return None
    accel = re.search(r"(-?\d+(?:\.\d+)?)\s*m/s\^?2\b", question, re.I)
    initial = re.search(r"(-?\d+(?:\.\d+)?)\s*m/s\b(?!\^?2)", question, re.I)
    # `(?<![/^])` keeps the `s` of `m/s` and the `2` of `m/s^2` out of it.
    time = re.search(r"(?<![/^\w])(-?\d+(?:\.\d+)?)\s*(?:s|seconds?)\b(?!/|\^)",
                     question, re.I)
    if accel is None or initial is None or time is None:
        return None
    u, a, t = (float(m.group(1)) for m in (initial, accel, time))
    return ("final_velocity", u + a * t)


def _permutations(question: str) -> Optional[Tuple[str, float]]:
    """`P(n, k)`, however the corpus words it (v93).



    Must run before `_combination_choose`: "permutations of 9 things taken 2

    at a time" contains `taken`, and that parser would return C(9, 2) = 36

    for a question whose answer is 72 -- the confident wrong verdict this

    module exists to avoid. So any question that says permutation, arrange or

    ordered is claimed here first, and a form this cannot read returns None

    rather than falling through to the combination reading.

    """

    if not re.search(r"permutation|arrang|ordered|line up", question, re.I):
        return None
    number = r"(\d+)"
    # (pattern, reversed) -- `reversed` says the phrasing states k before n,
    # as `_combination_choose` does. Every corpus and natural form is listed;
    # the order matters only where two could match, and none do.
    for pattern, reverse in (
        (rf"n\s*=\s*{number}\s*k\s*=\s*{number}", False),
        (rf"n\s*{number}\s*,?\s*k\s*{number}", False),
        (rf"of\s*{number}\s*things taken\s*{number}", False),
        (rf"of\s*{number}\s*(?:taken|take)\s*{number}", False),
        (rf"{number}\s*things and line up\s*{number}", False),
        (rf"can\s*{number}\s*items? be arranged in order from\s*{number}", True),
        (rf"arrangements? of\s*{number}\s*(?:are there )?from\s*{number}", True),
        (rf"arrange\s*{number}\s*out of\s*{number}", True),
        (rf"arranging\s*{number}\s*out of\s*{number}", True),
        (rf"{number}\s*items? can be picked from\s*{number}", True),
        (rf"selections? of\s*{number}\s*from\s*{number}", True),
    ):
        match = re.search(pattern, question, re.I)
        if not match:
            continue
        a, b = int(match.group(1)), int(match.group(2))
        n, k = (b, a) if reverse else (a, b)
        if k > n:
            return None   # far more likely misread than genuinely 0
        import math as _math

        return ("permutations", float(_math.perm(n, k)))
    return None


# -- connectome lookups (v93) ------------------------------------------------
#
# The three cns_* shapes ask for a fact, not a computation, so the only honest
# re-derivation is the same lookup the corpus builder did: the answer comes
# from `build_connectome_corpus.population()`, built from the CC-BY male-CNS
# tables. A type or pair outside that population returns None (not checked)
# rather than 0, because "not in the table we teach from" is not "zero
# neurons". Where the tables are absent the parsers are inert.

#: A type name as the corpus writes it: letters, digits, `_` and `-`. The
#: trailing `[A-Za-z0-9_]` keeps a sentence-final hyphen or the `?` out.
_TYPE = r"([A-Za-z][A-Za-z0-9_\-]*[A-Za-z0-9_]|[A-Za-z])"


def _connectome_population():
    try:
        import build_connectome_corpus as cns
    except Exception:  # noqa: BLE001 - the checker degrades, it does not break
        return None
    try:
        if not cns.data_available():
            return None
        return cns.population()
    except Exception:  # noqa: BLE001
        return None


def _cns_pair_synapses(question: str) -> Optional[Tuple[str, float]]:
    if not re.search(r"synapse", question, re.I) or not re.search(r"male CNS", question, re.I):
        return None
    for pattern in (
        rf"from (?:cell )?type {_TYPE} (?:to|onto) (?:cell )?type {_TYPE}",
        rf"type {_TYPE} makes? onto type {_TYPE}",
        rf"from {_TYPE} to {_TYPE}\b",
    ):
        match = re.search(pattern, question)
        if match:
            break
    else:
        return None
    pop = _connectome_population()
    if pop is None:
        return None
    pre, post = match.group(1), match.group(2)
    for a, b, weight in pop.pairs:
        if a == pre and b == post:
            return ("cns_pair_synapses", float(weight))
    return None


def _cns_side_count(question: str) -> Optional[Tuple[str, float]]:
    if not re.search(r"neurons?", question, re.I) or not re.search(r"male CNS", question, re.I):
        return None
    side = re.search(r"\b(left|right)\b", question, re.I)
    if side is None:
        return None
    name = _cns_type_name(question)
    if name is None:
        return None
    pop = _connectome_population()
    if pop is None or name not in pop.n_neurons:
        return None
    table = pop.left if side.group(1).lower() == "left" else pop.right
    return ("cns_side_count", float(table[name]))


def _cns_type_count(question: str) -> Optional[Tuple[str, float]]:
    if not re.search(r"neurons?", question, re.I) or not re.search(r"male CNS", question, re.I):
        return None
    if re.search(r"\b(left|right|side|hemisphere|synapse)", question, re.I):
        return None   # a side count or a pair count, not this shape
    name = _cns_type_name(question)
    if name is None:
        return None
    pop = _connectome_population()
    if pop is None or name not in pop.n_neurons:
        return None
    return ("cns_type_count", float(pop.n_neurons[name]))


def _cns_type_name(question: str) -> Optional[str]:
    """The one type a count question names, or None when the form is unknown."""

    for pattern in (
        rf"(?:cell )?type {_TYPE}",
        rf"(?:the |many )?{_TYPE} neurons",
    ):
        match = re.search(pattern, question)
        if match:
            return match.group(1)
    return None


def _combination_choose(question: str) -> Optional[Tuple[str, float]]:
    """`n choose k`, however the corpus words it.



    The corpus fixes k at 2 so the working can be shown, but this reads

    whatever k is stated rather than assuming it -- an assumption here would

    produce a confident wrong verdict on any other k.

    """

    if not re.search(r"combination|choose|chosen|taken", question, re.I):
        return None
    number = r"(\d+)"
    # (pattern, reversed) -- `reversed` says the phrasing states k before n.
    # v82: this used to pick n = max(a, b) for *every* phrasing, so "30 choose
    # 40" returned C(40, 30) = 847660528 where the truth is 0. A size heuristic
    # cannot tell an impossible question from a reversed one; the word order
    # can, and each of the corpus's four phrasings has a fixed order.
    for pattern, reverse in (
        (rf"{number}\s*choose\s*{number}", False),
        (rf"n\s*=\s*{number}\s*k\s*=\s*{number}", False),
        (rf"of\s*{number}\s*things taken\s*{number}", False),
        (rf"can\s*{number}\s*items? be chosen from\s*{number}", True),
    ):
        match = re.search(pattern, question, re.I)
        if not match:
            continue
        a, b = int(match.group(1)), int(match.group(2))
        n, k = (b, a) if reverse else (a, b)
        if k > n:
            # C(n, k) is 0 here, but a question asking to choose 40 from 30 is
            # far more likely to be one this parser has misread than one whose
            # answer is genuinely 0. NOT CHECKED is the honest verdict.
            return None
        import math as _math

        return ("combination", float(_math.comb(n, k)))
    return None


def _arithmetic_series(question: str) -> Optional[Tuple[str, float]]:
    """Sum of the first n terms of an arithmetic progression."""

    if not re.search(r"arithmetic (?:series|progression)", question, re.I):
        return None
    # v82: "An arithmetic series starts at 15 with common difference 4" never
    # says "first term". Measured 24/40 before adding `starts at`, 40/40 after.
    first = re.search(r"(?:first term|starts? at|beginning at)\s*(?:is\s*)?(-?\d+)",
                      question, re.I)
    difference = re.search(r"(?:common )?difference\s*(?:of\s*)?(-?\d+)", question, re.I)
    terms = re.search(r"(?:sum of|first)\s*(\d+)\s*terms|(?:\bn\s*(\d+))", question, re.I)
    if not (first and difference and terms):
        return None
    count = int(terms.group(1) or terms.group(2))
    a, d = int(first.group(1)), int(difference.group(1))
    last = a + (count - 1) * d
    return ("arithmetic_series", float(count * (a + last) / 2))


def _word_problem(question: str) -> Optional[Tuple[str, float]]:
    match = re.search(
        r"has\s+(\d+).*?get\s+(\d+)\s+more.*?give\s+away\s+(\d+)", question, re.I | re.S
    )
    if not match:
        return None
    start, gain, lose = (int(match.group(i)) for i in (1, 2, 3))
    return ("word_problem", float(start + gain - lose))


#: Order matters, and every entry below is placed against a specific ambiguity:
#:
#: * `_word_problem` and `_average` precede everything numeric, because both
#:   contain bare numbers a naive "a + b" search would seize on.
#: * `_sequence` precedes them too -- "7, 17, 27, 37" is a comma-separated list
#:   of numbers, which is exactly what an average looks like.
#: * `_two_step` precedes `_percent` because it *contains* a percent question.
#: * `_algebra` precedes `_multiplication` because `x` is both this corpus's
#:   multiplication sign and its unknown.
#: * `_kinetic_energy` and `_science` precede the bare-number parsers, because
#:   a physics question carries two numbers and a naive `a x b` search would
#:   seize on them without knowing which law applies. `_kinetic_energy` runs
#:   first of the two: it names a mass and a velocity, which is also what
#:   `momentum` matches on.
def _code_trace(question: str) -> Optional[Tuple[str, float]]:
    """Re-derive a code-tracing answer by executing the snippet in the question.



    Every other parser here re-implements the arithmetic it checks. This one

    does not need to: the question *contains* the program, so the check is to

    run it. That makes it the strongest verifier in this module -- there is no

    second implementation to drift from the first.



    Extraction is deliberately narrow. The snippet is recovered by finding the

    first assignment to a single-letter variable and taking everything from

    there to the trailing question, then the same allowlisted, builtins-free,

    timeout-bounded executor the corpus builder uses decides the value. A

    question this cannot confidently parse returns ``None`` -- *not checked* --

    which is the only safe failure for this module.

    """

    try:
        import build_code_corpus as code
    except Exception:  # noqa: BLE001 - the checker degrades, it does not break
        return None

    # The wrapper names the variable being asked about. Every template phrases
    # it one of these ways, and a question naming none of them is not a code
    # question this should touch.
    target = None
    for pattern in (r"value of ([A-Za-z_]\w*)", r"give (?:the value of )?([A-Za-z_]\w*)",
                    r"what (?:is|does) ([A-Za-z_]\w*)", r"final ([A-Za-z_]\w*)",
                    r"([A-Za-z_]\w*) after this runs", r"([A-Za-z_]\w*) hold at the end",
                    r"\b([A-Za-z_]\w*)\?\s*$"):
        found = re.search(pattern, question, flags=re.IGNORECASE)
        if found:
            target = found.group(1)
            break
    if target is None:
        return None

    # The code is the longest span that parses AND assigns that variable. Trying
    # spans rather than pattern-matching the wrapper means a new phrasing cannot
    # silently produce a wrong value: a mis-trimmed span fails to parse, and a
    # span that parses but never assigns the target is rejected below.
    # The right-hand side of the first assignment is a literal (list, number,
    # negative number) or, since v93's `code_range_sum` (`r = sum(range(3,
    # 9))`), a call to a lower-case name. Which names are callable is the
    # executor's decision, not this pattern's: anything else is refused there.
    start = re.search(r"[A-Za-z_]\w*\s*=\s*(?:[\[\-\d]|[a-z]+\()", question)
    if start is None:
        return None
    body = question[start.start():]

    # Candidate spans end at a whitespace boundary, longest first. An earlier
    # version required the span to end in a digit or a bracket, which silently
    # excluded `code_conditional` -- its snippet ends `else b - a`, on a
    # variable name -- and left two of nine tasks unverifiable. Correctness
    # comes from the executor accepting the span, not from guessing where the
    # prose starts, so the heuristic is gone and only the boundary remains.
    pieces = body.split(" ")
    for size in range(len(pieces), 0, -1):
        chunk = " ".join(pieces[:size]).rstrip(" ?.")
        if not chunk or not re.search(rf"\b{re.escape(target)}\s*=", chunk):
            continue
        result = code.run_snippet(chunk, target)
        if result.ok and result.value is not None:
            return ("code_trace", float(result.value))
    return None


PARSERS: Tuple[Callable[[str], Optional[Tuple[str, float]]], ...] = (
    # First: a code question contains numbers and operators that several
    # parsers below would happily misread as arithmetic. `x = 5` and
    # `range(4)` look like a binary expression to `_binary`.
    _code_trace,
    # v93: the connectome lookups key on "male CNS" and refuse everything
    # else, and a type name such as `5-HTPLP01` or `IN17A052` is exactly the
    # kind of string `_binary` and `_multiplication` would misread as an
    # expression, so they run before any numeric parser.
    _cns_pair_synapses,
    _cns_side_count,
    _cns_type_count,
    _word_problem,
    # v93: before `_combination_choose`, which would otherwise read
    # "permutations of 9 things taken 2 at a time" as C(9, 2).
    _permutations,
    _combination_choose,
    _arithmetic_series,
    _kinetic_energy,
    # v93: before `_science`, whose `acceleration` law matches "accelerates"
    # and whose `work` law matches "work out the spring energy"; both fall
    # through on a missing quantity today, and these make that not matter.
    _spring_energy,
    _final_velocity,
    _science,
    _sequence,
    _average,
    _two_step,
    _algebra,
    _percent,
    _division,
    _multiplication,
    _binary,
)


def parse_question(question: str) -> Optional[Tuple[str, float]]:
    for parser in PARSERS:
        try:
            result = parser(question)
        except (ValueError, ZeroDivisionError, OverflowError):
            return None
        if result is not None:
            return result if math.isfinite(result[1]) else None
    return None


def check(question: str, reply: str) -> Optional[Check]:
    """Verify a reply, or return ``None`` when the question is not checkable.



    ``None`` means *not checked*. The caller must not render it as correct: the

    whole value of this is that a wrong answer is visibly wrong, and quietly

    passing anything unrecognised would destroy that.

    """

    parsed = parse_question(question)
    if parsed is None:
        return None
    task, expected = parsed
    predicted = extract_answer(reply)
    correct = (
        predicted is not None
        and abs(predicted - expected) <= max(TOLERANCE, abs(expected) * 1e-6)
    )
    return Check(task=task, expected=expected, predicted=predicted, correct=correct)


def supported_shapes() -> List[str]:
    """The question forms this can verify, for the interface to advertise.



    Through v81 this listed only the nine arithmetic shapes while `PARSERS`

    already handled twelve science and mathematics shapes, so the interface

    under-advertised what it could check by more than half. Every entry here is

    asserted parseable by `test_answer_check.py`, so the list cannot drift

    ahead of the parsers again -- but note it can still drift *behind* them,

    which is the harmless direction.

    """

    return [
        # arithmetic (v74)
        "Solve this basic math problem: 617 + 288",
        "What is 25% of 840?",
        "Solve for x: x + 14 = 39",
        "A student has 45 marbles. They get 38 more and then give away 27. How many marbles do they have now?",
        "Find the average (mean) of these numbers: 40, 60, 20, 80",
        "What is 25 x 7?",
        "Quick question: 70 / 5",
        "What comes next in the sequence: 7, 17, 27, 37?",
        "What is 50% of 698, then add 28?",
        # physics (v79/v80)
        "Given mass 25 kg and acceleration 4 m/s^2, compute the force.",
        "A force of 580 N acts on a mass of 116 kg. What is the acceleration?",
        "mass 98 kg velocity 3 m/s find momentum",
        "What is the kinetic energy of a 12 kg mass moving at 5 m/s?",
        "Find the work done by 98 N acting over 2 m.",
        "What power corresponds to 1860 joules in 20 seconds?",
        "A current of 5 A flows through a resistance of 57 ohms. What is the voltage?",
        "A device runs at 12 V drawing 3 A. What is the electrical power?",
        "A wave with frequency 40 Hz has wavelength 6 m. What is its speed?",
        # chemistry (v79/v80)
        "What is the molarity of 4 mol of solute dissolved in 2 L?",
        # mathematics (v80)
        "In how many ways can 2 items be chosen from 30?",
        "An arithmetic series starts at 15 with common difference 4. "
        "What is the sum of the first 8 terms?",
        # code tracing (v87) -- checked by RUNNING the snippet rather than by
        # re-implementing it, which makes it the only parser here with no
        # second implementation to drift from the first.
        #
        # All nine code tasks are covered, measured at 108 checked / 0 wrong
        # / 0 unchecked over twelve samples each. Ordinary prose is still left
        # alone: "how are you today" and "x marks the spot" both return None.
        "What does x hold at the end? x = 2\nfor i in range(6): x = x + 8",
        "nums = [4, 10, 15]\nr = sum(nums) Give r.",
        "nums = [8, 3, 4]\nr = nums[0] + nums[2] What is r?",
        # v93 science and mathematics (build_omni_corpus.V93_TASKS)
        "A force of 80 N acts for 6 s. What is the impulse?",
        "A voltage of 376 V is applied across 8 ohm. What is the current?",
        "A spring of constant 18 N/m is stretched 4 m. Find the spring energy.",
        "Find the number of permutations of 9 things taken 2 at a time.",
        "A body moving at 12 m/s accelerates at 3 m/s^2 for 8 s. "
        "What is its final velocity?",
        # v93 code tracing (build_code_corpus.V93_TASKS): still one parser,
        # `_code_trace`, which now also runs `range(a, b)`, `.count` and a
        # negative index because the executor's allowlist admits them.
        "Trace this Python and give r. r = sum(range(3, 9))",
        "nums = [2, 4, 7, 4]\nr = nums.count(4) What is r?",
        "nums = [13, 8, 10]\nr = nums[-2] Give r.",
        # v93 connectome lookups (build_connectome_corpus.TASKS): the truth is
        # read from the CC-BY male-CNS tables the corpus was built from, so
        # these parse only where `datasets/v91_malecns` is on disk.
        "How many neurons of type KCg-m are in the male CNS?",
        "How many neurons of type KCg-m are on the left side of the male CNS?",
        "How many synapses go from type KCg-m to type PAM08 in the male CNS?",
    ]