Spaces:
Running on CPU Upgrade
Running on CPU Upgrade
| # Data | |
| ## Source of truth: HuggingFace Dataset | |
| All application data lives in | |
| [`VAILL/Legislation-Tracker-Data`](https://huggingface.co/datasets/VAILL/Legislation-Tracker-Data). | |
| The backend syncs from it on every startup. Every pipeline run uploads | |
| results back. Nothing in the git repo constitutes canonical data — the | |
| `data/` folder is git-ignored and always regenerated. | |
| ## Files in the Dataset | |
| ### Core JSON files (always synced at boot) | |
| | File | Purpose | Shape | | |
| |---|---|---| | |
| | `known_bills_visualize.json` | Full bill list with metadata | list of `{bill_id, state, bill_number, title, description, status, sponsors, iapp_categories, bill_url, last_action_date, session_year, ...}` | | |
| | `known_bills.json` | Raw pipeline intermediate | Similar to above but pre-summarization | | |
| | `bill_summaries.json` | AI-generated per-bill summaries | dict keyed by `STATE_BILLNUMBER` → `{summary: "..."}` | | |
| | `bill_reports.json` | AI-generated per-bill deep-dive reports | list of `{bill_id, report_markdown}` | | |
| | `bill_suggested_questions.json` | AI-generated Q&A prompts | dict keyed by `STATE_BILLNUMBER` | | |
| | `bill_cache.json` | LegiScan raw cache | keyed by bill_id | | |
| | `bill_calendar.json` | Legislative milestone events | list of `{bill_id, state, bill_number, title, event_type, event_label, event_description, event_date, bill_url}` | | |
| | `iapp_categories_cache.json` | LLM classifier output cache | keyed by bill_id | | |
| | `bills_vectorstore_manifest.json` | Pinecone index manifest | metadata for delta-upsert | | |
| | `subscribers.json` | Newsletter email list | list of strings | | |
| | `feedback.json` | User feedback submissions | list of `{id, name, email, type, description, status, timestamp}` | | |
| | `analytics.json` | Visit counts by date/hour | nested dict | | |
| | `pipeline_status.json` | Current/latest pipeline run | `{run_id, started_at, finished_at, overall_status, steps: [...], pid}` | | |
| | `pipeline_run_history.json` | Historical runs | list of run summaries | | |
| | `users.json` | Admin users (bcrypt hashes) | `{usernames: {name: {password, email, name}}}` | | |
| ### Prefix folders (always synced at boot) | |
| - `weekly_changes/weekly_changes_YYYY-MM-DD.csv` — one file per week, | |
| produced by `detect_changes.py`. Rows: `bill_number, state, change_type, | |
| status_change_detail, old_status, new_status, title, session_year`. | |
| - `newsletter_drafts/newsletter_YYYY-MM-DD.md` — one file per week, | |
| produced by `generate_newsletter.py`. | |
| ### Excluded from startup sync | |
| - `snapshots/known_bills_YYYY-MM-DD_HH-MM-SS.json` — full weekly bill | |
| archive used by `detect_changes.py`. Excluded because each file is | |
| large and only needed at pipeline time. The pipeline pulls them | |
| on-demand when it runs. | |
| See `backend/app/services/hf_sync.py` for the exact `CORE_FILES` list and | |
| `PREFIX_FOLDERS` tuple. | |
| ## Read/write patterns | |
| **Read only** (application uses this data to serve the site): | |
| - All core JSONs above. | |
| **Read + write** (pipeline both consumes and updates): | |
| - `known_bills.json`, `known_bills_visualize.json`, `bill_cache.json`, | |
| `bill_summaries.json`, `bill_suggested_questions.json`, `bill_reports.json`, | |
| `iapp_categories_cache.json`, `bill_calendar.json`, | |
| `pipeline_status.json`, `pipeline_run_history.json`, | |
| `weekly_changes/*`, `newsletter_drafts/*`, `snapshots/*`. | |
| **Write only** (from user actions in the app): | |
| - `subscribers.json` — subscribe/unsubscribe flow. | |
| - `feedback.json` — feedback form submissions. | |
| - `analytics.json` — pageview counter. | |
| - `users.json` — admin panel user management. | |
| When these change, the pipeline picks them up on the next run and | |
| uploads via `huggingface_upload.py`. | |
| ## Adding a new field to bills | |
| If the pipeline starts writing a new column into `known_bills_visualize.json`: | |
| 1. Add it to the appropriate Pydantic schema in | |
| `backend/app/schemas/bills.py` (both `Bill` and `BillSummary` if it | |
| should appear in list views). | |
| 2. If the frontend needs it, update the TypeScript type in | |
| `frontend/lib/api.ts` and use it in the relevant component. | |
| 3. Deploy backend + frontend. The next pipeline run populates the field | |
| for all bills. | |
| `extra="ignore"` is set on the schemas, so adding a field to the JSON | |
| without touching the schemas is harmless (it just won't be exposed). | |
| ## Cache invalidation | |
| The bills repository is in-memory. It reads `known_bills_visualize.json` | |
| once at startup. To pick up mid-run data changes, restart the backend | |
| (or the HF Space container). | |
| There's no cache invalidation for filesystem watches — this is | |
| intentional to keep the site fast. On HF, container restarts happen on | |
| every push, so data is fresh soon after each pipeline run. | |
| ## Local data folder | |
| `backend/data/` (git-ignored). Contents mirror the HF Dataset after a | |
| successful startup sync. Safe to `rm -rf backend/data` at any time — | |
| next backend restart re-syncs from HF. | |
| ## Backups | |
| - **v1 Streamlit code**: `~/Documents/Vanderbilt/Research/hf-space-v1-backup-2026-09-03/` | |
| (git repo with full history). | |
| - **HF Dataset**: HF preserves full git history on the Dataset repo, so | |
| every past pipeline upload is recoverable via `git log` and | |
| `git checkout <sha>` on the Dataset clone. | |