docs/DATA.md

Data

Source of truth: HuggingFace Dataset

All application data lives in VAILL/Legislation-Tracker-Data. The backend syncs from it on every startup. Every pipeline run uploads results back. Nothing in the git repo constitutes canonical data — the data/ folder is git-ignored and always regenerated.

Files in the Dataset

Core JSON files (always synced at boot)

File	Purpose	Shape
known_bills_visualize.json	Full bill list with metadata	list of {bill_id, state, bill_number, title, description, status, sponsors, iapp_categories, bill_url, last_action_date, session_year, ...}
known_bills.json	Raw pipeline intermediate	Similar to above but pre-summarization
bill_summaries.json	AI-generated per-bill summaries	dict keyed by STATE_BILLNUMBER → {summary: "..."}
bill_reports.json	AI-generated per-bill deep-dive reports	list of {bill_id, report_markdown}
bill_suggested_questions.json	AI-generated Q&A prompts	dict keyed by STATE_BILLNUMBER
bill_cache.json	LegiScan raw cache	keyed by bill_id
bill_calendar.json	Legislative milestone events	list of {bill_id, state, bill_number, title, event_type, event_label, event_description, event_date, bill_url}
iapp_categories_cache.json	LLM classifier output cache	keyed by bill_id
bills_vectorstore_manifest.json	Pinecone index manifest	metadata for delta-upsert
subscribers.json	Newsletter email list	list of strings
feedback.json	User feedback submissions	list of {id, name, email, type, description, status, timestamp}
analytics.json	Visit counts by date/hour	nested dict
pipeline_status.json	Current/latest pipeline run	{run_id, started_at, finished_at, overall_status, steps: [...], pid}
pipeline_run_history.json	Historical runs	list of run summaries
users.json	Admin users (bcrypt hashes)	{usernames: {name: {password, email, name}}}

Prefix folders (always synced at boot)

• weekly_changes/weekly_changes_YYYY-MM-DD.csv — one file per week, produced by detect_changes.py. Rows: bill_number, state, change_type, status_change_detail, old_status, new_status, title, session_year.
• newsletter_drafts/newsletter_YYYY-MM-DD.md — one file per week, produced by generate_newsletter.py.

Excluded from startup sync

• snapshots/known_bills_YYYY-MM-DD_HH-MM-SS.json — full weekly bill archive used by detect_changes.py. Excluded because each file is large and only needed at pipeline time. The pipeline pulls them on-demand when it runs.

See backend/app/services/hf_sync.py for the exact CORE_FILES list and PREFIX_FOLDERS tuple.

Read/write patterns

Read only (application uses this data to serve the site):

• All core JSONs above.

Read + write (pipeline both consumes and updates):

• known_bills.json, known_bills_visualize.json, bill_cache.json, bill_summaries.json, bill_suggested_questions.json, bill_reports.json, iapp_categories_cache.json, bill_calendar.json, pipeline_status.json, pipeline_run_history.json, weekly_changes/*, newsletter_drafts/*, snapshots/*.

Write only (from user actions in the app):

• subscribers.json — subscribe/unsubscribe flow.
• feedback.json — feedback form submissions.
• analytics.json — pageview counter.
• users.json — admin panel user management.

When these change, the pipeline picks them up on the next run and uploads via huggingface_upload.py.

Adding a new field to bills

If the pipeline starts writing a new column into known_bills_visualize.json:

1. Add it to the appropriate Pydantic schema in backend/app/schemas/bills.py (both Bill and BillSummary if it should appear in list views).
2. If the frontend needs it, update the TypeScript type in frontend/lib/api.ts and use it in the relevant component.
3. Deploy backend + frontend. The next pipeline run populates the field for all bills.

extra="ignore" is set on the schemas, so adding a field to the JSON without touching the schemas is harmless (it just won't be exposed).

Cache invalidation

The bills repository is in-memory. It reads known_bills_visualize.json once at startup. To pick up mid-run data changes, restart the backend (or the HF Space container).

There's no cache invalidation for filesystem watches — this is intentional to keep the site fast. On HF, container restarts happen on every push, so data is fresh soon after each pipeline run.

Local data folder

backend/data/ (git-ignored). Contents mirror the HF Dataset after a successful startup sync. Safe to rm -rf backend/data at any time — next backend restart re-syncs from HF.

Backups

• v1 Streamlit code: ~/Documents/Vanderbilt/Research/hf-space-v1-backup-2026-09-03/ (git repo with full history).
• HF Dataset: HF preserves full git history on the Dataset repo, so every past pipeline upload is recoverable via git log and git checkout <sha> on the Dataset clone.