Syntrex commited on
Commit
8a4e072
Β·
verified Β·
1 Parent(s): 359f86e

Update SQL

Browse files
Files changed (1) hide show
  1. SQL +512 -29
SQL CHANGED
@@ -1,38 +1,521 @@
1
- Revise the plan before executing.
2
 
3
- The immediate Plan A is close, but there are still two safety issues that must be fixed before edits:
 
4
 
5
- 1. Do NOT rebuild batter_prop_outcomes from the existing bloated backup using DISTINCT ON as the primary rebuild path.
6
- That backup is already contaminated by duplication/join-multiplication, so choosing one β€œlatest” row per (game_pk, batter_name, market) can preserve incorrect slot/pitcher/odds metadata from a bad duplicated row.
7
 
8
- Revise Stage 4 so the preferred rebuild path is:
9
- - rebuild from the corrected current pipeline / corrected audit source after Fixes 1–6 are deployed
10
- OR
11
- - rebuild from a freshly deduplicated audit extraction after the read_recommendation_audit_view fix is in place
12
 
13
- You may keep the backup-only DISTINCT ON insert as an emergency fallback, but not as the primary recommended rebuild path.
 
14
 
15
- 2. Fix the audit-view dedup/join logic so it is future-safe for multiple markets.
16
- Your latest_outcomes CTE deduplicates on (game_pk, batter_name, slot, market), but the final JOIN only uses:
17
- - game_pk
18
- - batter_name
19
- - slot
20
 
21
- That can reintroduce duplication if multiple markets exist.
22
- Revise the query so either:
23
- - the JOIN includes market where appropriate, or
24
- - the view is explicitly scoped to the market(s) actually needed by batter_prop_outcomes right now and documented as such.
25
 
26
- 3. Update validation to test the actual deduplicated audit view, not the old raw join.
27
- Your Stage 3 validation query still uses the raw recommendation_logs LEFT JOIN recommendation_outcomes pattern, which does not verify the fixed read_recommendation_audit_view behavior.
28
- Replace it with validation that directly checks the corrected audit view output for a target game_pk.
 
29
 
30
- 4. Keep the rest of Plan A structure:
31
- - dedup/latest-row strategy in read_recommendation_audit_view
32
- - game_pk filter in Step 3
33
- - dedup at batter_prop_outcomes build stage
34
- - scoped delete/read/write per game_pk
35
- - manual button must not append
36
- - unique index on batter_prop_outcomes only after clean rebuild
37
 
38
- Do not edit yet. Return the revised final execution plan with these corrections.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Here is Claude's plan: β•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œ batter_prop_outcomes β€” Staged Fix Plan
2
 
3
+ ---
4
+ Framing
5
 
6
+ There are two distinct problems. This plan separates them:
 
7
 
8
+ Plan A (immediate): Stop batter_prop_outcomes from growing, fix it safely, and add a
9
+ UNIQUE constraint. Does NOT modify the semantic grain or constraints on recommendation_logs
10
+ or recommendation_outcomes. Treats those tables as opaque historical sources.
 
11
 
12
+ Plan B (later): Audit and redesign the upstream source tables after a full downstream
13
+ consumer proof. Not in scope for the current execution.
14
 
15
+ ---
16
+ Why batter_prop_outcomes Is Exploding (Confirmed Root Causes)
 
 
 
17
 
18
+ Root Cause 1 β€” Source table join explosion (Layer 0β†’1)
 
 
 
19
 
20
+ recommendation_logs and recommendation_outcomes are written every 3 seconds by a
21
+ @st.fragment(run_every=3) fragment with no idempotency guard. A 3-hour game produces
22
+ ~10,800 rows in recommendation_logs and ~3,600 rows in recommendation_outcomes for the
23
+ same (game_pk, batter_name, slot) triple.
24
 
25
+ read_recommendation_audit_view() (db.py:481–532) LEFT JOINs both tables on
26
+ (game_pk, batter_name, slot) with no WHERE clause:
27
+ FROM recommendation_logs l
28
+ LEFT JOIN recommendation_outcomes o
29
+ ON l.game_pk = o.game_pk AND l.batter_name = o.batter_name AND l.slot = o.slot
 
 
30
 
31
+ When both sides have duplicates, this is a Cartesian product per triple:
32
+ - 10,800 log rows Γ— 3,600 outcome rows = 38,880,000 rows for one batter in one slot
33
+ - For 3 batters across 3 slots: ~350M rows returned from the audit view for a single game
34
+
35
+ Root Cause 2 β€” Step 3 reads full history with no game_pk filter (Layer 2)
36
+
37
+ analytics/game_completion.py:492:
38
+ audit_df = read_recommendation_audit_view(conn) # returns ALL rows, all games, fully inflated
39
+ insert_batter_prop_outcomes(conn, prop_df) # naked INSERT, no conflict guard
40
+
41
+ Every game completion appends the entire inflated multi-game history into batter_prop_outcomes.
42
+
43
+ Root Cause 3 β€” Step 4 does not compress (Layer 3)
44
+
45
+ Step 4 reads ALL rows, re-grades them, then replaces the entire table with the same inflated
46
+ count. It reduces nothing. Only runs when statcast is available.
47
+
48
+ Root Cause 4 β€” Retry fires Step 3 repeatedly (Layer 4)
49
+
50
+ @st.fragment(run_every=300) re-fires on_game_complete() for pending games.
51
+ _completed_pks session guard resets on restart. Each retry appends another full audit view blob.
52
+
53
+ ---
54
+ Evidence / SQL Diagnostics
55
+
56
+ Run these before making any changes:
57
+
58
+ -- Prove join explosion: check duplication factor in source tables
59
+ SELECT game_pk, batter_name, slot,
60
+ COUNT(*) AS log_rows
61
+ FROM recommendation_logs
62
+ GROUP BY game_pk, batter_name, slot
63
+ ORDER BY log_rows DESC LIMIT 10;
64
+
65
+ SELECT game_pk, batter_name, slot, market,
66
+ COUNT(*) AS outcome_rows
67
+ FROM recommendation_outcomes
68
+ GROUP BY game_pk, batter_name, slot, market
69
+ ORDER BY outcome_rows DESC LIMIT 10;
70
+
71
+ -- Prove Cartesian expansion: join row count for worst game
72
+ -- Replace 'WORST_GAME_PK' with top result from above
73
+ SELECT COUNT(*) AS audit_view_rows_for_game
74
+ FROM recommendation_logs l
75
+ LEFT JOIN recommendation_outcomes o
76
+ ON l.game_pk = o.game_pk AND l.batter_name = o.batter_name AND l.slot = o.slot
77
+ WHERE l.game_pk = 'WORST_GAME_PK';
78
+ -- Should be enormous (millions) if flooding is confirmed
79
+
80
+ -- Quantify batter_prop_outcomes inflation
81
+ SELECT COUNT(*) AS total_rows FROM batter_prop_outcomes;
82
+
83
+ SELECT COUNT(DISTINCT game_pk || '|' || batter_name || '|' || market) AS correct_grain_rows
84
+ FROM batter_prop_outcomes;
85
+ -- Ratio = duplication factor
86
+
87
+ SELECT game_pk, batter_name, market, COUNT(*) AS cnt
88
+ FROM batter_prop_outcomes
89
+ GROUP BY game_pk, batter_name, market
90
+ ORDER BY cnt DESC LIMIT 10;
91
+
92
+ ---
93
+ Plan A β€” Immediate Safe Remediation
94
+
95
+ Goal: Stop batter_prop_outcomes growth. Make Step 3 idempotent. Add UNIQUE constraint.
96
+ Does not impose UNIQUE constraints on recommendation_logs or recommendation_outcomes.
97
+ Does not truncate or alter recommendation_logs or recommendation_outcomes.
98
+
99
+ ---
100
+ Fix 1 β€” Dedup both sides of the audit view JOIN before returning (database/db.py)
101
+
102
+ File: database/db.py:485–532, function read_recommendation_audit_view()
103
+
104
+ Replace the current flat JOIN query with a CTE that takes the latest row per grain from each
105
+ source before joining. This eliminates Cartesian multiplication without altering the source tables.
106
+
107
+ Market scoping: latest_outcomes is scoped to market = 'hr' (the only current market in
108
+ recommendation_outcomes). This prevents future duplication if additional markets are added β€”
109
+ the JOIN would otherwise multiply one latest_logs row against multiple market rows. The WHERE market = 'hr' constraint is documented inline. When additional
110
+ markets are supported, this view
111
+ must be revised.
112
+
113
+ query = """
114
+ WITH latest_logs AS (
115
+ SELECT DISTINCT ON (game_pk, batter_name, slot) *
116
+ FROM recommendation_logs
117
+ ORDER BY game_pk, batter_name, slot, created_at DESC NULLS LAST
118
+ ),
119
+ latest_outcomes AS (
120
+ -- Scoped to market='hr' (only current market); prevents JOIN multiplication on future markets
121
+ SELECT DISTINCT ON (game_pk, batter_name, slot) *
122
+ FROM recommendation_outcomes
123
+ WHERE market = 'hr'
124
+ ORDER BY game_pk, batter_name, slot, graded_at DESC NULLS LAST
125
+ )
126
+ SELECT
127
+ l.created_at,
128
+ l.game_pk,
129
+ l.away_team,
130
+ l.home_team,
131
+ l.status,
132
+ l.slot,
133
+ l.batter_name,
134
+ l.pitcher_name,
135
+ l.ev90,
136
+ l.hit_prob,
137
+ l.hr_prob,
138
+ l.tb2p_prob,
139
+ l.fair_hit_odds,
140
+ l.fair_hr_odds,
141
+ l.fair_tb2p_odds,
142
+ l.book_hit_odds,
143
+ l.book_hr_odds,
144
+ l.book_tb2p_odds,
145
+ l.hit_edge,
146
+ l.hr_edge,
147
+ l.tb2p_edge,
148
+ l.adjusted_edge,
149
+ l.hit_bet_ev,
150
+ l.hr_bet_ev,
151
+ l.tb2p_bet_ev,
152
+ l.confidence,
153
+ l.confidence_bucket,
154
+ l.recommendation_tier,
155
+ l.priority_score,
156
+ l.reason_tags,
157
+ l.starter_stays_next_batter_prob,
158
+ l.starter_stays_next_inning_prob,
159
+ l.bullpen_entry_prob,
160
+ o.realized_hit,
161
+ o.realized_hr,
162
+ o.realized_tb2p,
163
+ o.graded_at,
164
+ o.outcome_source
165
+ FROM latest_logs l
166
+ LEFT JOIN latest_outcomes o
167
+ ON l.game_pk = o.game_pk
168
+ AND l.batter_name = o.batter_name
169
+ AND l.slot = o.slot
170
+ ORDER BY l.created_at DESC
171
+ """
172
+
173
+ Why this is safe for all consumers:
174
+ - Returns the latest recommendation snapshot per (game_pk, batter_name, slot) β€” one row each
175
+ - Market scoping ensures the LEFT JOIN is strictly 1:1 (one log row β†’ at most one outcome row)
176
+ - Audit metrics tables GROUP BY tier/confidence/edge β€” deduplicated data is more accurate
177
+ - Debug page shows tail(20) β€” no functional change
178
+ - Source tables are untouched; DISTINCT ON is read-only
179
+
180
+ Row count after this fix (per game): at most distinct(batter_name) Γ— distinct(slot) rows,
181
+ typically 3–10 batters Γ— 1–3 slots = 9–30 rows per game.
182
+
183
+ ---
184
+ Fix 2 β€” Filter audit_df to current game_pk in Step 3 (game_completion.py)
185
+
186
+ File: analytics/game_completion.py:492
187
+
188
+ # Change:
189
+ audit_df = read_recommendation_audit_view(conn)
190
+ # To:
191
+ audit_df = read_recommendation_audit_view(conn)
192
+ audit_df = audit_df[audit_df["game_pk"].astype(str).str.strip() == str(game_pk)]
193
+
194
+ After Fix 1, the audit view returns at most ~30 rows for a typical game. After Fix 2, only rows
195
+ for the completing game are processed. Together: Step 3 operates on exactly the right set of rows.
196
+
197
+ ---
198
+ Fix 3 β€” Dedup prop_df before inserting (batter_prop_grader.py)
199
+
200
+ File: analytics/batter_prop_grader.py, end of build_batter_prop_outcome_rows_from_audit()
201
+
202
+ Adds a final safety net to ensure one outcome row per (game_pk, batter_name, market) regardless
203
+ of what the audit view returns (e.g., if the same batter appeared in two different slots):
204
+
205
+ df = pd.DataFrame(rows)
206
+ if df.empty:
207
+ return df
208
+ df = (
209
+ df.sort_values("created_at", ascending=False, na_position="last")
210
+ .drop_duplicates(subset=["game_pk", "batter_name", "market"], keep="first")
211
+ .reset_index(drop=True)
212
+ )
213
+ return df
214
+
215
+ ---
216
+ Fix 4 β€” Scoped delete before Step 3 insert (game_completion.py + database/db.py)
217
+
218
+ Makes Step 3 idempotent: re-running for the same game_pk removes that game's rows first,
219
+ then inserts exactly the right set.
220
+
221
+ Add to database/db.py:
222
+ def delete_batter_prop_outcomes_for_game(conn, game_pk: str) -> None:
223
+ conn.execute(
224
+ text("DELETE FROM batter_prop_outcomes WHERE game_pk = :pk"),
225
+ {"pk": game_pk},
226
+ )
227
+
228
+ analytics/game_completion.py:497, change:
229
+ # Change:
230
+ insert_batter_prop_outcomes(conn, prop_df)
231
+ # To:
232
+ from database.db import delete_batter_prop_outcomes_for_game
233
+ delete_batter_prop_outcomes_for_game(conn, str(game_pk))
234
+ insert_batter_prop_outcomes(conn, prop_df)
235
+
236
+ ---
237
+ Fix 5 β€” Scope Step 4 to current game (game_completion.py + database/db.py)
238
+
239
+ Stops the full-table DELETE + re-INSERT churn. Step 4 now only reads and re-grades the rows
240
+ for the completing game.
241
+
242
+ Add to database/db.py:
243
+ def read_batter_prop_outcomes_for_game(conn, game_pk: str) -> pd.DataFrame:
244
+ ensure_batter_prop_outcomes_table(conn)
245
+ return pd.read_sql(
246
+ text("SELECT * FROM batter_prop_outcomes WHERE game_pk = :pk"),
247
+ conn,
248
+ params={"pk": game_pk},
249
+ )
250
+
251
+ analytics/game_completion.py:507–515, change:
252
+ # Change:
253
+ batter_prop_df = read_batter_prop_outcomes(conn)
254
+ ...
255
+ replace_batter_prop_outcomes(conn, graded_df)
256
+
257
+ # To:
258
+ from database.db import read_batter_prop_outcomes_for_game, delete_batter_prop_outcomes_for_game
259
+ batter_prop_df = read_batter_prop_outcomes_for_game(conn, str(game_pk))
260
+ if not batter_prop_df.empty:
261
+ graded_df = build_batter_realization_rows(
262
+ batter_prop_outcomes_df=batter_prop_df,
263
+ statcast_df=statcast_df,
264
+ graded_at=utc_now_iso(),
265
+ )
266
+ if not graded_df.empty:
267
+ delete_batter_prop_outcomes_for_game(conn, str(game_pk))
268
+ insert_batter_prop_outcomes(conn, graded_df)
269
+ logger.info("[game_completion] step 4 done: %d realized rows", len(graded_df))
270
+
271
+ ---
272
+ Fix 6 β€” Fix manual debug button (app.py)
273
+
274
+ File: app.py:2691–2700
275
+
276
+ Change insert_batter_prop_outcomes(conn, outcome_df) to replace_batter_prop_outcomes(conn, outcome_df).
277
+
278
+ The manual button now does a full-table replace instead of append. Idempotent: clicking it twice
279
+ produces the same result. It still reads the full audit view (which Fix 1 has deduplicated).
280
+
281
+ ---
282
+ Fix 7 β€” Add UNIQUE constraint to batter_prop_outcomes (applied post-rebuild)
283
+
284
+ File: database/db.py, ensure_batter_prop_outcomes_table(), after the CREATE TABLE call:
285
+
286
+ conn.execute(text("""
287
+ CREATE UNIQUE INDEX IF NOT EXISTS uq_batter_prop_outcomes_grain
288
+ ON batter_prop_outcomes (game_pk, batter_name, market)
289
+ """))
290
+
291
+ This is the hard backstop. Any future bug that attempts a duplicate insert will fail loudly
292
+ rather than silently inflating the table.
293
+
294
+ Apply AFTER the table is rebuilt clean (Stage 4 below), not before.
295
+
296
+ ---
297
+ Plan A β€” Staged Execution Order
298
+
299
+ Stage 1 β€” Run diagnostics (read-only, no changes)
300
+
301
+ Execute the SQL diagnostics above. Confirm:
302
+ - Log rows per (game_pk, batter_name, slot) are in the hundreds/thousands
303
+ - Audit view join expansion is massive for worst-case games
304
+ - batter_prop_outcomes correct grain count vs. actual count
305
+
306
+ This is a gate β€” proceed only after confirming the pattern.
307
+
308
+ ---
309
+ Stage 2 β€” Deploy code (Fixes 1–6, no schema changes)
310
+
311
+ One push. All changes are backward-compatible:
312
+ - Fix 1: read_recommendation_audit_view uses dedup CTE (read-only query change)
313
+ - Fix 2: game_pk filter at call site (game_completion.py)
314
+ - Fix 3: dedup in build_batter_prop_outcome_rows_from_audit (batter_prop_grader.py)
315
+ - Fix 4: delete_batter_prop_outcomes_for_game helper + call in Step 3
316
+ - Fix 5: read_batter_prop_outcomes_for_game helper + scoped Step 4
317
+ - Fix 6: manual button uses replace
318
+
319
+ No UNIQUE constraints are added in this stage. No source tables are modified. The code is
320
+ safe to deploy and roll back if needed.
321
+
322
+ ---
323
+ Stage 3 β€” Validate: run one game cycle and confirm no new duplicates
324
+
325
+ After deploy, wait for the next game to complete (or trigger one manually).
326
+ Then run these checks against the corrected audit view logic directly:
327
+
328
+ -- A. Verify the fixed audit view produces no multiplication for a target game
329
+ -- Use the same CTE logic as the updated read_recommendation_audit_view
330
+ WITH latest_logs AS (
331
+ SELECT DISTINCT ON (game_pk, batter_name, slot) *
332
+ FROM recommendation_logs
333
+ ORDER BY game_pk, batter_name, slot, created_at DESC NULLS LAST
334
+ ),
335
+ latest_outcomes AS (
336
+ SELECT DISTINCT ON (game_pk, batter_name, slot) *
337
+ FROM recommendation_outcomes
338
+ WHERE market = 'hr'
339
+ ORDER BY game_pk, batter_name, slot, graded_at DESC NULLS LAST
340
+ )
341
+ SELECT l.game_pk, l.batter_name, l.slot, COUNT(*) AS rows_in_view
342
+ FROM latest_logs l
343
+ LEFT JOIN latest_outcomes o
344
+ ON l.game_pk = o.game_pk AND l.batter_name = o.batter_name AND l.slot = o.slot
345
+ WHERE l.game_pk = '<GAME_PK_FROM_LAST_COMPLETION>'
346
+ GROUP BY l.game_pk, l.batter_name, l.slot;
347
+ -- Expected: all rows_in_view = 1; total rows ~ 9–30
348
+
349
+ -- B. batter_prop_outcomes: new rows for last completing game must each appear once
350
+ SELECT game_pk, batter_name, market, COUNT(*) AS cnt
351
+ FROM batter_prop_outcomes
352
+ WHERE game_pk = '<GAME_PK_FROM_LAST_COMPLETION>'
353
+ GROUP BY game_pk, batter_name, market
354
+ ORDER BY cnt DESC;
355
+ -- Expected: all cnt = 1
356
+
357
+ ---
358
+ Stage 4 β€” Back up, truncate, and rebuild batter_prop_outcomes
359
+
360
+ Step 1 β€” Back up (preserve for audit/emergency only; do NOT use as primary rebuild source):
361
+ CREATE TABLE batter_prop_outcomes_backup_20260321
362
+ AS SELECT * FROM batter_prop_outcomes;
363
+ SELECT COUNT(*) FROM batter_prop_outcomes_backup_20260321;
364
+
365
+ The backup is contaminated β€” it contains rows inflated by join multiplication from the old
366
+ query. Picking DISTINCT ON rows from the backup risks preserving metadata (slot, pitcher, odds)
367
+ that came from a bad duplicated row rather than a clean source. Use the backup for inspection
368
+ only. It is NOT the primary rebuild path.
369
+
370
+ Step 2 β€” Truncate:
371
+ TRUNCATE TABLE batter_prop_outcomes;
372
+ SELECT COUNT(*) FROM batter_prop_outcomes; -- must be 0
373
+
374
+ Step 3 β€” Rebuild via the corrected pipeline (primary path):
375
+
376
+ With Fixes 1–6 deployed, the corrected audit view now returns clean deduplicated rows.
377
+ Trigger a rebuild using the existing pipeline functions β€” this is equivalent to clicking the
378
+ "Build Batter Prop Outcomes" debug button after Fix 6 is deployed:
379
+
380
+ # Equivalent to grade_batter_prop_outcomes_from_audit() after Fix 6
381
+ # Run this from a Python shell or admin script connected to production:
382
+
383
+ from database.db import read_recommendation_audit_view, replace_batter_prop_outcomes
384
+ from analytics.batter_prop_grader import build_batter_prop_outcome_rows_from_audit
385
+ from utils.time_utils import utc_now_iso # or however utc_now_iso is imported
386
+
387
+ audit_df = read_recommendation_audit_view(conn) # returns clean deduplicated rows
388
+ prop_df = build_batter_prop_outcome_rows_from_audit(audit_df, graded_at=utc_now_iso())
389
+ # prop_df is already deduped on (game_pk, batter_name, market) by Fix 3
390
+ replace_batter_prop_outcomes(conn, prop_df) # DELETE ALL + INSERT clean rows
391
+
392
+ This produces exactly one row per (game_pk, batter_name, market) for all historical games.
393
+ Rows will have grade_status = 'pending' and outcome_source = 'pending_batter_prop_grade'
394
+ for historical games where realized outcomes haven't been re-applied. That is acceptable β€”
395
+ the table is clean and correctly sized; future games will have realized outcomes applied by the
396
+ corrected Step 4.
397
+
398
+ Step 4 β€” Verify row count is sane:
399
+ SELECT COUNT(*) FROM batter_prop_outcomes; -- expected: hundreds to low thousands
400
+
401
+ SELECT game_pk, COUNT(*) AS rows_per_game
402
+ FROM batter_prop_outcomes GROUP BY game_pk ORDER BY game_pk DESC LIMIT 10;
403
+ -- Each game_pk should have a small number of rows (1 per distinct batter with a rec)
404
+
405
+ Emergency fallback only β€” backup-based DISTINCT ON (if corrected pipeline rebuild fails):
406
+ INSERT INTO batter_prop_outcomes
407
+ SELECT DISTINCT ON (game_pk, batter_name, market)
408
+ created_at, graded_at, game_pk, away_team, home_team, slot,
409
+ batter_name, pitcher_name, market, fair_hr_odds, book_hr_odds,
410
+ adjusted_edge, confidence, recommendation_tier,
411
+ realized_hit, realized_hr, realized_tb2p, grade_status, outcome_source
412
+ FROM batter_prop_outcomes_backup_20260321
413
+ ORDER BY game_pk, batter_name, market, graded_at DESC NULLS LAST;
414
+ Use only if the corrected pipeline rebuild is blocked for operational reasons. Metadata quality
415
+ in this path is not guaranteed since rows came from a multiply-contaminated table.
416
+
417
+ ---
418
+ Stage 5 β€” Apply UNIQUE constraint to batter_prop_outcomes (Fix 7)
419
+
420
+ CREATE UNIQUE INDEX uq_batter_prop_outcomes_grain
421
+ ON batter_prop_outcomes (game_pk, batter_name, market);
422
+
423
+ Verify it was created without error, then ensure it shows in:
424
+ SHOW INDEXES FROM batter_prop_outcomes;
425
+
426
+ ---
427
+ Stage 6 β€” Final validation
428
+
429
+ -- 1. No duplicates per grain (must return 0 rows)
430
+ SELECT COUNT(*) FROM (
431
+ SELECT COUNT(*) AS c FROM batter_prop_outcomes
432
+ GROUP BY game_pk, batter_name, market HAVING COUNT(*) > 1
433
+ ) t;
434
+
435
+ -- 2. Row count is sane
436
+ SELECT COUNT(*) FROM batter_prop_outcomes; -- expected: hundreds to low thousands
437
+
438
+ -- 3. Grade status distribution looks correct
439
+ SELECT grade_status, outcome_source, COUNT(*)
440
+ FROM batter_prop_outcomes GROUP BY 1, 2;
441
+
442
+ -- 4. After next game completion: only that game's rows were added/modified
443
+ SELECT game_pk, COUNT(*) FROM batter_prop_outcomes
444
+ GROUP BY game_pk ORDER BY game_pk DESC LIMIT 5;
445
+
446
+ ---
447
+ Plan B β€” Later: Upstream Source Table Redesign
448
+
449
+ Do not execute until Plan A is stable and at least 5 game completion cycles are validated.
450
+
451
+ What needs to be determined first (open questions)
452
+
453
+ 1. Are repeated timestamped rows in recommendation_logs intentional? (e.g., to track how odds
454
+ evolved as a batter stayed in On Deck across multiple pitches)
455
+ 2. Is any downstream consumer using the full history of recommendation_logs (e.g., time-series
456
+ analysis of confidence drift per batter per game)?
457
+ 3. Are batter_audit_metrics results more accurate with the full log history or with the
458
+ deduplicated snapshot?
459
+ 4. What is the intended data retention for recommendation_logs? Should old rows ever be purged?
460
+
461
+ What Plan B would include (if proof shows repeated rows are not intentional)
462
+
463
+ - Add _upsert_on_conflict and _insert_ignore DB-level helpers to db.py
464
+ - Change insert_recommendation_logs to upsert on (game_pk, batter_name, slot) with DO UPDATE
465
+ to keep latest model features
466
+ - Change insert_recommendation_outcomes to insert-ignore on (game_pk, batter_name, slot, market)
467
+ - Deduplicate existing recommendation_logs and recommendation_outcomes (CREATE AS SELECT DISTINCT ON
468
+ β†’ TRUNCATE β†’ INSERT back)
469
+ - Add UNIQUE indexes to both tables
470
+ - Revert Fix 1 (dedup CTE in audit view) since it would become unnecessary
471
+
472
+ What Plan B does NOT do (regardless of outcome)
473
+
474
+ - Does not change read_recommendation_audit_view column list
475
+ - Does not remove slot from either table
476
+ - Does not change the batter_prop_outcomes schema
477
+
478
+ ---
479
+ Critical Files for Plan A
480
+
481
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
482
+ β”‚ File β”‚ Fix β”‚ Change β”‚
483
+ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
484
+ β”‚ database/db.py:485–532 β”‚ Fix 1 β”‚ Replace flat JOIN with DISTINCT ON CTE in read_recommendation_audit_view β”‚
485
+ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
486
+ β”‚ analytics/game_completion.py:492 β”‚ Fix 2 β”‚ Add game_pk filter on audit_df β”‚
487
+ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
488
+ β”‚ analytics/batter_prop_grader.py β”‚ Fix 3 β”‚ Add dedup on (game_pk, batter_name, market) before return β”‚
489
+ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
490
+ β”‚ database/db.py β”‚ Fix 4 β”‚ Add delete_batter_prop_outcomes_for_game() helper β”‚
491
+ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
492
+ β”‚ analytics/game_completion.py:497 β”‚ Fix 4 β”‚ Scoped delete before Step 3 insert β”‚
493
+ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
494
+ β”‚ database/db.py β”‚ Fix 5 β”‚ Add read_batter_prop_outcomes_for_game() helper β”‚
495
+ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
496
+ β”‚ analytics/game_completion.py:507–515 β”‚ Fix 5 β”‚ Scoped read/delete/insert for Step 4 β”‚
497
+ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
498
+ β”‚ app.py:2691–2700 β”‚ Fix 6 β”‚ Manual button uses replace instead of insert β”‚
499
+ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
500
+ β”‚ database/db.py β”‚ Fix 7 β”‚ Add CREATE UNIQUE INDEX IF NOT EXISTS in ensure_batter_prop_outcomes_table() β”‚
501
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
502
+
503
+ What Is NOT Changed in Plan A
504
+
505
+ - recommendation_logs β€” no schema changes, no TRUNCATE, no UNIQUE constraints
506
+ - recommendation_outcomes β€” no schema changes, no TRUNCATE, no UNIQUE constraints
507
+ - insert_recommendation_logs β€” unchanged
508
+ - insert_recommendation_outcomes β€” unchanged
509
+ - _bulk_insert β€” unchanged
510
+ - build_recommendation_log_rows β€” unchanged
511
+ - build_recommendation_outcome_rows β€” unchanged
512
+ - All table column names and types β€” unchanged
513
+ - replace_batter_prop_outcomes β€” unchanged
514
+ β•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œβ•Œ
515
+
516
+ Claude has written up a plan and is ready to execute. Would you like to proceed?
517
+
518
+ ❯ 1. Yes, clear context (61% used) and auto-accept edits
519
+ 2. Yes, auto-accept edits
520
+ 3. Yes, manually approve edits
521
+ 4. Type here to tell Claude what to change