Hey-Big-Proppa / docs /UTILITY_ROOM.md
AIBRUH's picture
Upload docs/UTILITY_ROOM.md with huggingface_hub
193768f verified
|
Raw History Blame Contribute Delete
6.56 kB
# The Utility Room
One file, `sources.yaml`, describes every data source feeding the lake. One
CLI, `scripts/sources.py`, reads and manages it. This is where a source gets
tweaked, serviced, or swapped -- without touching any other source, and
without hunting through ingest scripts to remember what depends on what.
## Why this exists
Confirmed during this session's verification pass: two tables were quietly
empty (`gold.nfl_prop_line`, superseded; `gold.nfl_player_id_bridge`, not yet
built) and nobody would have known without directly querying row counts. The
utility room makes that kind of gap visible by design, in one place, instead
of requiring a manual audit every time.
## Commands
```bash
PYTHONPATH=.pylibs python3 scripts/sources.py list
PYTHONPATH=.pylibs python3 scripts/sources.py list --status active
PYTHONPATH=.pylibs python3 scripts/sources.py show rotowire_props
PYTHONPATH=.pylibs python3 scripts/sources.py check --all # live reachability
PYTHONPATH=.pylibs python3 scripts/sources.py check espn_news # one source
PYTHONPATH=.pylibs python3 scripts/sources.py disable espn_news
PYTHONPATH=.pylibs python3 scripts/sources.py enable espn_news
PYTHONPATH=.pylibs python3 scripts/sources.py run rotowire_props -- --skip-download
```
`check --all` does a real HTTP GET against each source's `base_url` and
reports the status code. Verified 2026-09-25: 6 of 9 sources return a healthy
response; the 2 failures (`pfr_direct` HTTP 403, `tank01_rapidapi` HTTP 401)
match their documented `broken`/`disabled` status exactly -- the tool isn't
just printing green for everything.
## What's in `sources.yaml` right now
| id | status | what it feeds |
|---|---|---|
| `nflverse_stats` | active | 8 player-week chart tables, team_week, player/game dims |
| `nflverse_pbp` | active | TD log (all 32 teams) |
| `nflverse_injuries` | active | official weekly injury report |
| `rotowire_props` | active | 22 prop markets, scoped to DK/FD/Caesars |
| `espn_news` | active | injury narrative, news, transactions |
| `player_headshots` | active | cached photo files (derived from nflverse_stats, no independent pull) |
| `teamrankings_reference` | deprecated | chart-design reference only, never loaded into silver/gold |
| `pfr_direct` | broken | documents why PFR itself was ruled out (403, Cloudflare + ToS) |
| `tank01_rapidapi` | disabled | documented option, not built; needs the user's own live key test |
Each entry records: what it feeds, how often it should be re-pulled, its
confirmed risk profile (ToS/stability/access caveats -- not guessed), and
replacement candidates already identified if it needs to be swapped.
## Servicing a source
1. `show <id>` to see its current config and risk notes.
2. Edit the relevant fields in `sources.yaml` directly (rate limits, cadence,
replacement candidate). No code change needed for config-only tweaks.
3. If the ingest logic itself needs to change, only that source's script
(`ingest_script` field) is touched -- no other source's script imports or
depends on it.
4. `check <id>` to confirm it's still reachable before trusting a scheduled
re-run.
5. To retire a source: set `status: deprecated` or `broken` and fill in why
under `risk` -- don't delete the entry, since the "why we stopped using
this" record is as valuable as the "what we use now" record (see
`pfr_direct` and `teamrankings_reference` for the pattern).
## Known gaps, tracked here rather than hidden
- `gold.nfl_prop_line` (in `sql/nfl_pfr_schema.sql`) is superseded and empty
by design -- marked in a comment at its definition, not removed, in case
anything already references the name.
- `gold.nfl_player_id_bridge` is a real, unfinished piece: the schema exists
to link ESPN athlete IDs to nflverse `gsis_id`, but the matching logic was
never built. Currently 0 rows. Don't join through it yet -- match on team +
player name in the meantime, as `matchup_report.py` already does.
## The ESPN stream: schedule and staff (added 2026-10-02)
MY BOO is the director of the ESPN data stream. `backend/boo_staff.py` runs her staff in one background thread
(started from `slip_alerts.start_worker()`, so it comes up with the Space). Every agent has two cadences: **LIVE**
(any NFL or Top-25/SEC game in progress) and **IDLE**. Each run is timed and stored in
`lake/gold/espn/staff_status.json`; the Data Auditor flags any agent that goes stale or errors 3 times running, and
the MY BOO → STAFF tab shows it all. A failing agent never stops the others, and the college page serves the last good
snapshot if ESPN is down.
| Agent | Duty | LIVE | IDLE | Writes (lake/gold/espn/) |
|---|---|---|---|---|
| Scoreboard Scout | NFL scoreboard + college Top-25/SEC slate, spots live games and finals | 20 s | 10 min | `nfl_scoreboard.json`, `cfb_snapshot.json` |
| Play-by-Play Clerk | Reads plays of every live game; alerts on scores, turnovers, 20+ yd plays | 15 s | (off) | `play_events.json`, `play_seen.json` + MY BOO alert feed |
| Line Watcher | Spread/total history, 1+ pt movement, ESPN-vs-TeamRankings disagreement (1.5+ pts) | 10 min | 30 min | `line_history.json` |
| Poll Analyst | AP + Coaches polls, week-over-week movement | 30 min | 1 h | `polls_prev.json`, `poll_movers.json` |
| Portal Scout | College transfer-portal stories, flags new ones | 30 min | 1 h | `cfb_portal.json` |
| SEC Beat Reporter | SEC news + matchup news per SEC game | 15 min | 30 min | `cfb_sec.json` |
| Injury Desk | NFL injury report snapshot and changes | 30 min | 1 h | `injuries.json` |
| Standings Actuary | Standings + ESPN-derived power rating (Jimmy's fallback for `tr_power`) | 30 min | 1 h | `standings.json` |
| Data Auditor | Staleness and error check on every agent | 5 min | 10 min | `audit.json` |
| MY BOO — Director | Writes the briefing from the snapshots | 5 min | 1 h | `boo_briefing.json` |
| Hermes Liaison | Exports the roster as a manifest for Hermes agents | 6 h | 6 h | `hermes_staff.json` |
Other schedules that feed the lake (unchanged): TeamRankings Tue 1 AM CT full / Mon, Thu–Sun 9 AM CT daily
(`trusteddataTR.maybe_run`); MY BOO Jimmy training Tue 6 AM CT (`boo_training.maybe_run_tuesday`).
**Scope:** college football is Top 25 + SEC only (`espn_cfb.SEC` holds the 16 SEC team ids). To change a cadence, edit the
`live`/`idle` numbers in `boo_staff.AGENTS`; to pause one agent, remove it from that list. Endpoints: `/api/boo/staff`,
`/api/boo/staff/manifest` (Hermes), `/api/espn/ticker`, `/api/espn/game?id=&lg=nfl|cfb`, `/api/cfb/espn`, `/api/espn/lines`.