Spaces:
Sleeping
test(#44): add SSOT regression test for canonical_decision_facts
Browse filesThe fix for Bug #44 landed in commit 2c51f98 (scorecard path and
get_policy_facts path resolve different values for the same policy's
decision-critical fields β live audit: ICICI Health AdvantEdge PED
showed 24mo in the comparison table and 0mo in the #31 scorecard
card). The fix introduced brain_tools.canonical_decision_facts and
wired it into key_coverage_facts as the single canonical resolver.
This test (written 2026-05-19) is the CI guard for that contract.
It was sitting untracked in the working tree for 9 days. Without it,
any future regression of the SSOT invariant β e.g. someone adds a
field to brain_tools._FACT_KEYS without also threading it through
canonical_decision_facts, or _load_curated_facts changes its
precedence β would ship to the live Space silently.
What the test asserts:
- For every id in the full catalogue (every marketplace card +
every key in _load_curated_facts), every decision-critical field
where get_policy_facts surfaces a value MUST equal the scorecard
path's value (3/3 green).
- The 11 originally-flagged PED-divergent policies (Activ Assure
Diamond, Care Supreme Enhance, my:health Sampoorna Suraksha,
my:health Suraksha, Optima Enhance, Health AdvantEdge, Health
Premia, Health Premier, Star Assure, Star Comprehensive, Silver
Health) each resolve a single non-null PED value via BOTH paths.
- ICICI Health AdvantEdge PED is 24mo (never the fabricated 0mo
that triggered a 'short waiting period' fake strength).
Verified locally: 3 passed in 2.61s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
@@ -0,0 +1,208 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Bug #44 (2026-05-19) β SINGLE-SOURCE guard: the file/value the
|
| 2 |
+
marketplace SCORECARD path resolves for a policy's decision-critical
|
| 3 |
+
fields must be IDENTICAL to the value the get_policy_facts (LLM) path
|
| 4 |
+
surfaces β for the WHOLE catalogue, by construction.
|
| 5 |
+
|
| 6 |
+
ROOT CAUSE THIS LOCKS DOWN
|
| 7 |
+
--------------------------
|
| 8 |
+
Every product has a base `40-data/policy_facts/<id>.json` AND doctype
|
| 9 |
+
siblings (`__wordings` / `__cis` / `__brochure` / `__prospectus`).
|
| 10 |
+
|
| 11 |
+
* Scorecard / #31 profile-summary path β `main._load_curated_facts()`
|
| 12 |
+
(KI-219/KI-251 canonical precedence: base wins per-field, sibling
|
| 13 |
+
backfills only nulls); the resulting flat dict is what
|
| 14 |
+
`build_scorecard` / `build_profile_summary` reads.
|
| 15 |
+
|
| 16 |
+
* get_policy_facts path β previously `brain_tools._load_policy_facts()`,
|
| 17 |
+
a DIFFERENT `_candidate_stems` resolver whose 7-key `_FACT_KEYS`
|
| 18 |
+
doesn't even include PED.
|
| 19 |
+
|
| 20 |
+
The two resolvers can pick different files, so the SAME bot turn could
|
| 21 |
+
state two different PED waiting periods (live audit: comparison TABLE said
|
| 22 |
+
ICICI Health AdvantEdge PED = 24 months, while its #31 scorecard card
|
| 23 |
+
bullet said "0 months β short waiting period").
|
| 24 |
+
|
| 25 |
+
THE FIX β ONE CANONICAL ENTRY
|
| 26 |
+
-----------------------------
|
| 27 |
+
`brain_tools.canonical_decision_facts(pid)` reads the decision-critical
|
| 28 |
+
fields from the SAME canonical curated entry the scorecard path uses, and
|
| 29 |
+
`get_policy_facts` merges it LAST into `key_coverage_facts` (overriding
|
| 30 |
+
the divergent 7-key resolver for those keys). Mirrors the existing
|
| 31 |
+
`_scorecard_signal β marketplace_grade` single-source pattern (#40).
|
| 32 |
+
|
| 33 |
+
This test proves agreement for the entire id universe a recommendation
|
| 34 |
+
can surface. RED on the pre-fix code/data; GREEN after the fix. 0
|
| 35 |
+
mismatches, by construction.
|
| 36 |
+
"""
|
| 37 |
+
from __future__ import annotations
|
| 38 |
+
|
| 39 |
+
import sys
|
| 40 |
+
import unittest
|
| 41 |
+
from pathlib import Path
|
| 42 |
+
|
| 43 |
+
_REPO_ROOT = Path(__file__).resolve().parent.parent
|
| 44 |
+
if str(_REPO_ROOT) not in sys.path:
|
| 45 |
+
sys.path.insert(0, str(_REPO_ROOT))
|
| 46 |
+
|
| 47 |
+
# Decision-critical fields whose value MUST agree between the scorecard
|
| 48 |
+
# path and the get_policy_facts path for every policy. PED is the field
|
| 49 |
+
# the live bug self-contradicted on; the rest are the other fields a
|
| 50 |
+
# user-facing comparison would quote.
|
| 51 |
+
_DECISION_CRITICAL = (
|
| 52 |
+
"pre_existing_disease_waiting_months",
|
| 53 |
+
"initial_waiting_period_days",
|
| 54 |
+
"copayment_pct",
|
| 55 |
+
"room_rent_capping",
|
| 56 |
+
"claim_settlement_ratio",
|
| 57 |
+
)
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
def _unwrap(v):
|
| 61 |
+
"""_load_curated_facts already flattens {value, source_*} β scalar,
|
| 62 |
+
but accept the wrapped shape defensively."""
|
| 63 |
+
if isinstance(v, dict) and "value" in v:
|
| 64 |
+
return v.get("value")
|
| 65 |
+
return v
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
def _norm(v):
|
| 69 |
+
"""Normalise for equality: empty β None; numeric strings β int;
|
| 70 |
+
strings trimmed/lowercased so '24' == 24 and ' No Cap ' == 'no cap'."""
|
| 71 |
+
v = _unwrap(v)
|
| 72 |
+
if v in (None, "", [], {}):
|
| 73 |
+
return None
|
| 74 |
+
if isinstance(v, bool):
|
| 75 |
+
return v
|
| 76 |
+
if isinstance(v, (int, float)):
|
| 77 |
+
return int(v) if float(v).is_integer() else float(v)
|
| 78 |
+
if isinstance(v, str):
|
| 79 |
+
s = v.strip()
|
| 80 |
+
if not s:
|
| 81 |
+
return None
|
| 82 |
+
try:
|
| 83 |
+
f = float(s.replace(",", ""))
|
| 84 |
+
return int(f) if f.is_integer() else f
|
| 85 |
+
except (TypeError, ValueError):
|
| 86 |
+
return s.lower()
|
| 87 |
+
return v
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
class TestPolicyFactsSourceConsistency(unittest.TestCase):
|
| 91 |
+
"""Scorecard-path value == get_policy_facts-path value for every
|
| 92 |
+
decision-critical field, across the WHOLE catalogue."""
|
| 93 |
+
|
| 94 |
+
@classmethod
|
| 95 |
+
def setUpClass(cls):
|
| 96 |
+
import backend.main as M
|
| 97 |
+
from backend import brain_tools as BT
|
| 98 |
+
|
| 99 |
+
cls.M = M
|
| 100 |
+
cls.BT = BT
|
| 101 |
+
# The EXACT dict the scorecard path feeds build_scorecard.
|
| 102 |
+
cls.curated = M._load_curated_facts()
|
| 103 |
+
|
| 104 |
+
def _scorecard_path_value(self, pid: str, field: str):
|
| 105 |
+
"""What the scorecard / #31 path sees for `field` β the canonical
|
| 106 |
+
curated entry main._load_curated_facts() feeds build_scorecard."""
|
| 107 |
+
entry = self.curated.get(pid)
|
| 108 |
+
if not isinstance(entry, dict):
|
| 109 |
+
return None
|
| 110 |
+
return _norm(entry.get(field))
|
| 111 |
+
|
| 112 |
+
def _get_policy_facts_path_value(self, pid: str, field: str):
|
| 113 |
+
"""What the get_policy_facts tool surfaces for `field` β the
|
| 114 |
+
canonical single-source resolver wired into key_coverage_facts."""
|
| 115 |
+
return _norm(self.BT.canonical_decision_facts(pid).get(field))
|
| 116 |
+
|
| 117 |
+
def _id_universe(self):
|
| 118 |
+
"""Every id a recommendation / get_policy_facts call can surface:
|
| 119 |
+
every marketplace card id + every curated key (incl. doctype
|
| 120 |
+
permutations + sibling stems/policy_ids registered by
|
| 121 |
+
_load_curated_facts Pass-2)."""
|
| 122 |
+
ids: set[str] = set()
|
| 123 |
+
for c in self.M._marketplace_catalogue(None):
|
| 124 |
+
if getattr(c, "policy_id", None):
|
| 125 |
+
ids.add(c.policy_id)
|
| 126 |
+
for k in self.curated:
|
| 127 |
+
if isinstance(k, str) and k:
|
| 128 |
+
ids.add(k)
|
| 129 |
+
return sorted(ids)
|
| 130 |
+
|
| 131 |
+
def test_full_catalogue_decision_field_source_parity(self):
|
| 132 |
+
mismatches = []
|
| 133 |
+
for pid in self._id_universe():
|
| 134 |
+
for field in _DECISION_CRITICAL:
|
| 135 |
+
sc = self._scorecard_path_value(pid, field)
|
| 136 |
+
gp = self._get_policy_facts_path_value(pid, field)
|
| 137 |
+
# The get_policy_facts path only ASSERTS a value when the
|
| 138 |
+
# canonical entry populates it (it never invents). So the
|
| 139 |
+
# invariant is: whenever get_policy_facts surfaces a value
|
| 140 |
+
# for a decision-critical field, it must EQUAL the
|
| 141 |
+
# scorecard path's value for that policy. (A null on the
|
| 142 |
+
# get_policy_facts side just means "not surfaced" β not a
|
| 143 |
+
# contradiction.)
|
| 144 |
+
if gp is not None and sc != gp:
|
| 145 |
+
mismatches.append(
|
| 146 |
+
{"policy_id": pid, "field": field,
|
| 147 |
+
"scorecard_path": sc, "get_policy_facts_path": gp}
|
| 148 |
+
)
|
| 149 |
+
|
| 150 |
+
self.assertEqual(
|
| 151 |
+
mismatches, [],
|
| 152 |
+
"#44: decision-critical fields where the SCORECARD path and "
|
| 153 |
+
"the get_policy_facts path resolve DIFFERENT values for the "
|
| 154 |
+
"same policy (self-contradiction risk in a single bot turn): "
|
| 155 |
+
f"{mismatches[:20]}",
|
| 156 |
+
)
|
| 157 |
+
|
| 158 |
+
def test_ped_specific_known_conflict_set_is_reconciled(self):
|
| 159 |
+
"""Belt-and-braces: the exact ~12 policies the live audit / scan
|
| 160 |
+
flagged must each resolve a single PED value via BOTH paths."""
|
| 161 |
+
flagged = [
|
| 162 |
+
"aditya-birla__activ-assure-diamond",
|
| 163 |
+
"care-health__care-supreme-enhance",
|
| 164 |
+
"hdfc-ergo__my-health-sampoorna-suraksha",
|
| 165 |
+
"hdfc-ergo__my-health-suraksha",
|
| 166 |
+
"hdfc-ergo__optima-enhance",
|
| 167 |
+
"icici-lombard__health-advantedge",
|
| 168 |
+
"niva-bupa__health-premia",
|
| 169 |
+
"star-health__health-premier",
|
| 170 |
+
"star-health__star-assure",
|
| 171 |
+
"star-health__star-comprehensive",
|
| 172 |
+
"bajaj-allianz__silver-health",
|
| 173 |
+
]
|
| 174 |
+
F = "pre_existing_disease_waiting_months"
|
| 175 |
+
bad = []
|
| 176 |
+
for pid in flagged:
|
| 177 |
+
sc = self._scorecard_path_value(pid, F)
|
| 178 |
+
gp = self._get_policy_facts_path_value(pid, F)
|
| 179 |
+
# Each must be a real (non-null) sourced number AND agree.
|
| 180 |
+
if sc is None or gp is None or sc != gp:
|
| 181 |
+
bad.append({"policy_id": pid, "scorecard": sc,
|
| 182 |
+
"get_policy_facts": gp})
|
| 183 |
+
self.assertEqual(
|
| 184 |
+
bad, [],
|
| 185 |
+
"#44: a flagged-conflict policy still has a missing or "
|
| 186 |
+
f"divergent PED across the two paths: {bad}",
|
| 187 |
+
)
|
| 188 |
+
|
| 189 |
+
def test_icici_advantedge_not_a_fake_zero_short_waiting(self):
|
| 190 |
+
"""The exact live symptom: ICICI Health AdvantEdge must NOT report
|
| 191 |
+
a 0-month PED (which fabricated a "short waiting period" strength).
|
| 192 |
+
Its source clause is plan-dependent/garbled; the representative
|
| 193 |
+
plan value is 24 months β and BOTH paths must say 24, not 0."""
|
| 194 |
+
F = "pre_existing_disease_waiting_months"
|
| 195 |
+
pid = "icici-lombard__health-advantedge"
|
| 196 |
+
sc = self._scorecard_path_value(pid, F)
|
| 197 |
+
gp = self._get_policy_facts_path_value(pid, F)
|
| 198 |
+
self.assertEqual(sc, 24, f"scorecard path PED should be 24, got {sc}")
|
| 199 |
+
self.assertEqual(gp, 24, f"get_policy_facts PED should be 24, got {gp}")
|
| 200 |
+
self.assertNotEqual(
|
| 201 |
+
sc, 0,
|
| 202 |
+
"#44 regression: ICICI Health AdvantEdge PED back to a fake "
|
| 203 |
+
"0-month 'short waiting period'.",
|
| 204 |
+
)
|
| 205 |
+
|
| 206 |
+
|
| 207 |
+
if __name__ == "__main__":
|
| 208 |
+
unittest.main()
|