rohitsar567 Claude Opus 4.7 (1M context) commited on
Commit
1cc997b
Β·
1 Parent(s): 4c728a9

test(#44): add SSOT regression test for canonical_decision_facts

Browse files

The fix for Bug #44 landed in commit 2c51f98 (scorecard path and
get_policy_facts path resolve different values for the same policy's
decision-critical fields β€” live audit: ICICI Health AdvantEdge PED
showed 24mo in the comparison table and 0mo in the #31 scorecard
card). The fix introduced brain_tools.canonical_decision_facts and
wired it into key_coverage_facts as the single canonical resolver.

This test (written 2026-05-19) is the CI guard for that contract.
It was sitting untracked in the working tree for 9 days. Without it,
any future regression of the SSOT invariant β€” e.g. someone adds a
field to brain_tools._FACT_KEYS without also threading it through
canonical_decision_facts, or _load_curated_facts changes its
precedence β€” would ship to the live Space silently.

What the test asserts:
- For every id in the full catalogue (every marketplace card +
every key in _load_curated_facts), every decision-critical field
where get_policy_facts surfaces a value MUST equal the scorecard
path's value (3/3 green).
- The 11 originally-flagged PED-divergent policies (Activ Assure
Diamond, Care Supreme Enhance, my:health Sampoorna Suraksha,
my:health Suraksha, Optima Enhance, Health AdvantEdge, Health
Premia, Health Premier, Star Assure, Star Comprehensive, Silver
Health) each resolve a single non-null PED value via BOTH paths.
- ICICI Health AdvantEdge PED is 24mo (never the fabricated 0mo
that triggered a 'short waiting period' fake strength).

Verified locally: 3 passed in 2.61s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

tests/test_policy_facts_source_consistency.py ADDED
@@ -0,0 +1,208 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Bug #44 (2026-05-19) β€” SINGLE-SOURCE guard: the file/value the
2
+ marketplace SCORECARD path resolves for a policy's decision-critical
3
+ fields must be IDENTICAL to the value the get_policy_facts (LLM) path
4
+ surfaces β€” for the WHOLE catalogue, by construction.
5
+
6
+ ROOT CAUSE THIS LOCKS DOWN
7
+ --------------------------
8
+ Every product has a base `40-data/policy_facts/<id>.json` AND doctype
9
+ siblings (`__wordings` / `__cis` / `__brochure` / `__prospectus`).
10
+
11
+ * Scorecard / #31 profile-summary path β†’ `main._load_curated_facts()`
12
+ (KI-219/KI-251 canonical precedence: base wins per-field, sibling
13
+ backfills only nulls); the resulting flat dict is what
14
+ `build_scorecard` / `build_profile_summary` reads.
15
+
16
+ * get_policy_facts path β†’ previously `brain_tools._load_policy_facts()`,
17
+ a DIFFERENT `_candidate_stems` resolver whose 7-key `_FACT_KEYS`
18
+ doesn't even include PED.
19
+
20
+ The two resolvers can pick different files, so the SAME bot turn could
21
+ state two different PED waiting periods (live audit: comparison TABLE said
22
+ ICICI Health AdvantEdge PED = 24 months, while its #31 scorecard card
23
+ bullet said "0 months β€” short waiting period").
24
+
25
+ THE FIX β€” ONE CANONICAL ENTRY
26
+ -----------------------------
27
+ `brain_tools.canonical_decision_facts(pid)` reads the decision-critical
28
+ fields from the SAME canonical curated entry the scorecard path uses, and
29
+ `get_policy_facts` merges it LAST into `key_coverage_facts` (overriding
30
+ the divergent 7-key resolver for those keys). Mirrors the existing
31
+ `_scorecard_signal β†’ marketplace_grade` single-source pattern (#40).
32
+
33
+ This test proves agreement for the entire id universe a recommendation
34
+ can surface. RED on the pre-fix code/data; GREEN after the fix. 0
35
+ mismatches, by construction.
36
+ """
37
+ from __future__ import annotations
38
+
39
+ import sys
40
+ import unittest
41
+ from pathlib import Path
42
+
43
+ _REPO_ROOT = Path(__file__).resolve().parent.parent
44
+ if str(_REPO_ROOT) not in sys.path:
45
+ sys.path.insert(0, str(_REPO_ROOT))
46
+
47
+ # Decision-critical fields whose value MUST agree between the scorecard
48
+ # path and the get_policy_facts path for every policy. PED is the field
49
+ # the live bug self-contradicted on; the rest are the other fields a
50
+ # user-facing comparison would quote.
51
+ _DECISION_CRITICAL = (
52
+ "pre_existing_disease_waiting_months",
53
+ "initial_waiting_period_days",
54
+ "copayment_pct",
55
+ "room_rent_capping",
56
+ "claim_settlement_ratio",
57
+ )
58
+
59
+
60
+ def _unwrap(v):
61
+ """_load_curated_facts already flattens {value, source_*} β†’ scalar,
62
+ but accept the wrapped shape defensively."""
63
+ if isinstance(v, dict) and "value" in v:
64
+ return v.get("value")
65
+ return v
66
+
67
+
68
+ def _norm(v):
69
+ """Normalise for equality: empty β†’ None; numeric strings β†’ int;
70
+ strings trimmed/lowercased so '24' == 24 and ' No Cap ' == 'no cap'."""
71
+ v = _unwrap(v)
72
+ if v in (None, "", [], {}):
73
+ return None
74
+ if isinstance(v, bool):
75
+ return v
76
+ if isinstance(v, (int, float)):
77
+ return int(v) if float(v).is_integer() else float(v)
78
+ if isinstance(v, str):
79
+ s = v.strip()
80
+ if not s:
81
+ return None
82
+ try:
83
+ f = float(s.replace(",", ""))
84
+ return int(f) if f.is_integer() else f
85
+ except (TypeError, ValueError):
86
+ return s.lower()
87
+ return v
88
+
89
+
90
+ class TestPolicyFactsSourceConsistency(unittest.TestCase):
91
+ """Scorecard-path value == get_policy_facts-path value for every
92
+ decision-critical field, across the WHOLE catalogue."""
93
+
94
+ @classmethod
95
+ def setUpClass(cls):
96
+ import backend.main as M
97
+ from backend import brain_tools as BT
98
+
99
+ cls.M = M
100
+ cls.BT = BT
101
+ # The EXACT dict the scorecard path feeds build_scorecard.
102
+ cls.curated = M._load_curated_facts()
103
+
104
+ def _scorecard_path_value(self, pid: str, field: str):
105
+ """What the scorecard / #31 path sees for `field` β€” the canonical
106
+ curated entry main._load_curated_facts() feeds build_scorecard."""
107
+ entry = self.curated.get(pid)
108
+ if not isinstance(entry, dict):
109
+ return None
110
+ return _norm(entry.get(field))
111
+
112
+ def _get_policy_facts_path_value(self, pid: str, field: str):
113
+ """What the get_policy_facts tool surfaces for `field` β€” the
114
+ canonical single-source resolver wired into key_coverage_facts."""
115
+ return _norm(self.BT.canonical_decision_facts(pid).get(field))
116
+
117
+ def _id_universe(self):
118
+ """Every id a recommendation / get_policy_facts call can surface:
119
+ every marketplace card id + every curated key (incl. doctype
120
+ permutations + sibling stems/policy_ids registered by
121
+ _load_curated_facts Pass-2)."""
122
+ ids: set[str] = set()
123
+ for c in self.M._marketplace_catalogue(None):
124
+ if getattr(c, "policy_id", None):
125
+ ids.add(c.policy_id)
126
+ for k in self.curated:
127
+ if isinstance(k, str) and k:
128
+ ids.add(k)
129
+ return sorted(ids)
130
+
131
+ def test_full_catalogue_decision_field_source_parity(self):
132
+ mismatches = []
133
+ for pid in self._id_universe():
134
+ for field in _DECISION_CRITICAL:
135
+ sc = self._scorecard_path_value(pid, field)
136
+ gp = self._get_policy_facts_path_value(pid, field)
137
+ # The get_policy_facts path only ASSERTS a value when the
138
+ # canonical entry populates it (it never invents). So the
139
+ # invariant is: whenever get_policy_facts surfaces a value
140
+ # for a decision-critical field, it must EQUAL the
141
+ # scorecard path's value for that policy. (A null on the
142
+ # get_policy_facts side just means "not surfaced" β€” not a
143
+ # contradiction.)
144
+ if gp is not None and sc != gp:
145
+ mismatches.append(
146
+ {"policy_id": pid, "field": field,
147
+ "scorecard_path": sc, "get_policy_facts_path": gp}
148
+ )
149
+
150
+ self.assertEqual(
151
+ mismatches, [],
152
+ "#44: decision-critical fields where the SCORECARD path and "
153
+ "the get_policy_facts path resolve DIFFERENT values for the "
154
+ "same policy (self-contradiction risk in a single bot turn): "
155
+ f"{mismatches[:20]}",
156
+ )
157
+
158
+ def test_ped_specific_known_conflict_set_is_reconciled(self):
159
+ """Belt-and-braces: the exact ~12 policies the live audit / scan
160
+ flagged must each resolve a single PED value via BOTH paths."""
161
+ flagged = [
162
+ "aditya-birla__activ-assure-diamond",
163
+ "care-health__care-supreme-enhance",
164
+ "hdfc-ergo__my-health-sampoorna-suraksha",
165
+ "hdfc-ergo__my-health-suraksha",
166
+ "hdfc-ergo__optima-enhance",
167
+ "icici-lombard__health-advantedge",
168
+ "niva-bupa__health-premia",
169
+ "star-health__health-premier",
170
+ "star-health__star-assure",
171
+ "star-health__star-comprehensive",
172
+ "bajaj-allianz__silver-health",
173
+ ]
174
+ F = "pre_existing_disease_waiting_months"
175
+ bad = []
176
+ for pid in flagged:
177
+ sc = self._scorecard_path_value(pid, F)
178
+ gp = self._get_policy_facts_path_value(pid, F)
179
+ # Each must be a real (non-null) sourced number AND agree.
180
+ if sc is None or gp is None or sc != gp:
181
+ bad.append({"policy_id": pid, "scorecard": sc,
182
+ "get_policy_facts": gp})
183
+ self.assertEqual(
184
+ bad, [],
185
+ "#44: a flagged-conflict policy still has a missing or "
186
+ f"divergent PED across the two paths: {bad}",
187
+ )
188
+
189
+ def test_icici_advantedge_not_a_fake_zero_short_waiting(self):
190
+ """The exact live symptom: ICICI Health AdvantEdge must NOT report
191
+ a 0-month PED (which fabricated a "short waiting period" strength).
192
+ Its source clause is plan-dependent/garbled; the representative
193
+ plan value is 24 months β€” and BOTH paths must say 24, not 0."""
194
+ F = "pre_existing_disease_waiting_months"
195
+ pid = "icici-lombard__health-advantedge"
196
+ sc = self._scorecard_path_value(pid, F)
197
+ gp = self._get_policy_facts_path_value(pid, F)
198
+ self.assertEqual(sc, 24, f"scorecard path PED should be 24, got {sc}")
199
+ self.assertEqual(gp, 24, f"get_policy_facts PED should be 24, got {gp}")
200
+ self.assertNotEqual(
201
+ sc, 0,
202
+ "#44 regression: ICICI Health AdvantEdge PED back to a fake "
203
+ "0-month 'short waiting period'.",
204
+ )
205
+
206
+
207
+ if __name__ == "__main__":
208
+ unittest.main()