# Data

## Source of truth: HuggingFace Dataset

All application data lives in
[`VAILL/Legislation-Tracker-Data`](https://huggingface.co/datasets/VAILL/Legislation-Tracker-Data).
The backend syncs from it on every startup. Every pipeline run uploads
results back. Nothing in the git repo constitutes canonical data — the
`data/` folder is git-ignored and always regenerated.

## Files in the Dataset

### Core JSON files (always synced at boot)

| File | Purpose | Shape |
|---|---|---|
| `known_bills_visualize.json` | Full bill list with metadata | list of `{bill_id, state, bill_number, title, description, status, sponsors, iapp_categories, bill_url, last_action_date, session_year, ...}` |
| `known_bills.json` | Raw pipeline intermediate | Similar to above but pre-summarization |
| `bill_summaries.json` | AI-generated per-bill summaries | dict keyed by `STATE_BILLNUMBER` → `{summary: "..."}` |
| `bill_reports.json` | AI-generated per-bill deep-dive reports | list of `{bill_id, report_markdown}` |
| `bill_suggested_questions.json` | AI-generated Q&A prompts | dict keyed by `STATE_BILLNUMBER` |
| `bill_cache.json` | LegiScan raw cache | keyed by bill_id |
| `bill_calendar.json` | Legislative milestone events | list of `{bill_id, state, bill_number, title, event_type, event_label, event_description, event_date, bill_url}` |
| `iapp_categories_cache.json` | LLM classifier output cache | keyed by bill_id |
| `bills_vectorstore_manifest.json` | Pinecone index manifest | metadata for delta-upsert |
| `subscribers.json` | Newsletter email list | list of strings |
| `feedback.json` | User feedback submissions | list of `{id, name, email, type, description, status, timestamp}` |
| `analytics.json` | Visit counts by date/hour | nested dict |
| `pipeline_status.json` | Current/latest pipeline run | `{run_id, started_at, finished_at, overall_status, steps: [...], pid}` |
| `pipeline_run_history.json` | Historical runs | list of run summaries |
| `users.json` | Admin users (bcrypt hashes) | `{usernames: {name: {password, email, name}}}` |

### Prefix folders (always synced at boot)

- `weekly_changes/weekly_changes_YYYY-MM-DD.csv` — one file per week,
  produced by `detect_changes.py`. Rows: `bill_number, state, change_type,
  status_change_detail, old_status, new_status, title, session_year`.
- `newsletter_drafts/newsletter_YYYY-MM-DD.md` — one file per week,
  produced by `generate_newsletter.py`.

### Excluded from startup sync

- `snapshots/known_bills_YYYY-MM-DD_HH-MM-SS.json` — full weekly bill
  archive used by `detect_changes.py`. Excluded because each file is
  large and only needed at pipeline time. The pipeline pulls them
  on-demand when it runs.

See `backend/app/services/hf_sync.py` for the exact `CORE_FILES` list and
`PREFIX_FOLDERS` tuple.

## Read/write patterns

**Read only** (application uses this data to serve the site):
- All core JSONs above.

**Read + write** (pipeline both consumes and updates):
- `known_bills.json`, `known_bills_visualize.json`, `bill_cache.json`,
  `bill_summaries.json`, `bill_suggested_questions.json`, `bill_reports.json`,
  `iapp_categories_cache.json`, `bill_calendar.json`,
  `pipeline_status.json`, `pipeline_run_history.json`,
  `weekly_changes/*`, `newsletter_drafts/*`, `snapshots/*`.

**Write only** (from user actions in the app):
- `subscribers.json` — subscribe/unsubscribe flow.
- `feedback.json` — feedback form submissions.
- `analytics.json` — pageview counter.
- `users.json` — admin panel user management.

When these change, the pipeline picks them up on the next run and
uploads via `huggingface_upload.py`.

## Adding a new field to bills

If the pipeline starts writing a new column into `known_bills_visualize.json`:

1. Add it to the appropriate Pydantic schema in
   `backend/app/schemas/bills.py` (both `Bill` and `BillSummary` if it
   should appear in list views).
2. If the frontend needs it, update the TypeScript type in
   `frontend/lib/api.ts` and use it in the relevant component.
3. Deploy backend + frontend. The next pipeline run populates the field
   for all bills.

`extra="ignore"` is set on the schemas, so adding a field to the JSON
without touching the schemas is harmless (it just won't be exposed).

## Cache invalidation

The bills repository is in-memory. It reads `known_bills_visualize.json`
once at startup. To pick up mid-run data changes, restart the backend
(or the HF Space container).

There's no cache invalidation for filesystem watches — this is
intentional to keep the site fast. On HF, container restarts happen on
every push, so data is fresh soon after each pipeline run.

## Local data folder

`backend/data/` (git-ignored). Contents mirror the HF Dataset after a
successful startup sync. Safe to `rm -rf backend/data` at any time —
next backend restart re-syncs from HF.

## Backups

- **v1 Streamlit code**: `~/Documents/Vanderbilt/Research/hf-space-v1-backup-2026-09-03/`
  (git repo with full history).
- **HF Dataset**: HF preserves full git history on the Dataset repo, so
  every past pipeline upload is recoverable via `git log` and
  `git checkout <sha>` on the Dataset clone.