Spaces:
Sleeping
Download docs/website_maintenance.md from UWRFR/UWaterlooRFRChatBot: direct link, hf CLI and curl.
- Browser
- Download file 8.43 kB
-
https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot/resolve/main/docs/website_maintenance.md
- Command line
-
hf download hf://spaces/UWRFR/UWaterlooRFRChatBot/docs/website_maintenance.md
-
curl -L -o website_maintenance.md https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot/resolve/main/docs/website_maintenance.md
A newer version of the Streamlit SDK is available: 1.67.0
Website Maintenance and New Case Study Guide
This guide explains how to maintain the RFR Knowledge Platform website and how to add new managed retreat case studies to the searchable knowledge base.
Repositories and Live Website
- Shared GitHub repo:
https://github.com/rfrraguwaterloo-cmyk/UWaterlooManagedRetreatChatbot - Shared Hugging Face Space:
https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot - Live app:
https://uwrfr-uwaterloorfrchatbot.hf.space
GitHub is the easiest shared place for code review. Hugging Face is where the Streamlit app runs.
Editing the Website From Another Computer
- Install Git and Python 3.11 or newer.
- Clone the shared GitHub repo:
git clone https://github.com/rfrraguwaterloo-cmyk/UWaterlooManagedRetreatChatbot.git
cd UWaterlooManagedRetreatChatbot
- Create a working branch:
git checkout -b update-website-text
- Install dependencies:
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
- Run the website locally:
streamlit run app/app.py
- Edit the code. Most website text and tabs are in:
app/app.py
- Check the app still compiles:
python3 -m py_compile app/app.py
- Commit and push:
git status
git add app/app.py
git commit -m "Update website text"
git push origin update-website-text
- Open a Pull Request on GitHub into
main.
After the change is merged into main, the GitHub Action can deploy to Hugging
Face if the shared repo has HF_TOKEN and HF_SPACE_ID configured. The team can
also push manually to Hugging Face from a machine that is logged in with hf.
Manual Hugging Face Push
From a local repo with the Hugging Face CLI logged in:
git remote add uwrfr-chatbot https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot
git push uwrfr-chatbot HEAD:main
If the remote already exists, update it instead:
git remote set-url uwrfr-chatbot https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot
git push uwrfr-chatbot HEAD:main
Or upload the current folder with the CLI:
hf upload UWRFR/UWaterlooRFRChatBot . \
--type space \
--exclude ".git/*" \
--exclude ".github/*" \
--exclude ".venv/*" \
--exclude "venv/*" \
--exclude "__pycache__/*" \
--exclude "data/raw/*" \
--exclude "*.pdf" \
--commit-message "Update website"
Check Space status:
hf spaces info UWRFR/UWaterlooRFRChatBot
Website Secrets
Set these in Hugging Face Space settings under Variables and secrets.
Required for the chatbot:
ANTHROPIC_API_KEY
Optional if using OpenAI mode:
OPENAI_API_KEY
OPENAI_MODEL
Required for Google Drive case-study request uploads:
GOOGLE_OAUTH_CLIENT_ID
GOOGLE_OAUTH_CLIENT_SECRET
GOOGLE_OAUTH_REFRESH_TOKEN
Optional Drive override for new case study requests:
RFR_REQUESTS_DRIVE_FOLDER_ID
By default, new case study requests are uploaded to this Drive folder:
10n_-uOCT2GXu_G3r1qi5qJr8hHoy-E_y
The OAuth token should be generated by signing in as the Google account that
should own request uploads, usually rfr.rag.uwaterloo@gmail.com.
Generate the Hugging Face OAuth secrets locally:
python3 -m case_study_pipeline.create_drive_oauth_secrets \
--credentials-file /path/to/oauth-client.json \
--space-id UWRFR/UWaterlooRFRChatBot
The older fallback GOOGLE_SERVICE_ACCOUNT_JSON service-account setup can fail
in regular My Drive folders because service accounts do not have storage quota.
Use OAuth or a Google Shared Drive for request uploads.
How New Case Study Requests Work
Users can submit candidate papers through the New Case Study Requests tab. The app creates a new Google Drive folder for each request. Each folder contains:
request.json- uploaded PDFs or supporting documents
The request tab only collects files and notes. It does not run the extraction pipeline and does not call an LLM.
Adding a New Case Study to the Knowledge Base
Use this workflow after the team accepts a submitted request or identifies a new case study.
1. Choose the Next Case ID
Check the highest indexed case:
ls data/extracted/CS*.json | sort -V | tail
If the latest indexed case is CS54, the next one is usually CS55.
2. Create the Raw Case Folder
Create a folder under data/raw/:
mkdir -p data/raw/CS55-ShortName_COUNTRY
Put source PDFs, .md, or .txt files in that folder.
Add or confirm case_meta.json in that folder. Example:
{
"case_id": "CS55",
"name": "Short case study name",
"location": "City or region",
"country": "Country"
}
3. Run the One-Provider Extraction Pipeline
Use one provider for all four extraction and audit steps. Claude is the current simplest option:
python3 -m case_study_pipeline.run_case_study \
--case-folder data/raw/CS55-ShortName_COUNTRY \
--case-id CS55 \
--llm-provider claude \
--max-chars-per-source 150000 \
--force
This writes outputs to:
data/raw/CS55-ShortName_COUNTRY/pipeline_output/
Expected output files:
CS55_Ver1.mdCS55_Check1_report.mdCS55_Ver2.mdCS55_Check2_report.md- matching PDFs
4. Select the Best Run
python3 -m case_study_pipeline.select_best_run --case-id CS55
This creates or updates:
data/raw/CS55-ShortName_COUNTRY/pipeline_output/selected.json
5. Ingest the Case Study
python3 ingest/ingest_pipeline_outputs.py --case-id CS55
This creates:
data/extracted/CS55.json
6. Rebuild Summaries and Embeddings
Always run both commands after adding or changing extracted case data:
python3 ingest/create_summary_chunks.py
python3 ingest/embed_and_index.py
These update:
data/extracted/case_summaries.json
data/extracted/precomputed_embeddings.json
The live website will not know about the new case until these files are updated and deployed.
7. Test Locally
streamlit run app/app.py
Check:
- the new case appears in Case Studies
- the app can answer a direct question such as
Summarize CS55 - source links appear if
source_linksmetadata is available
8. Commit and Push
Commit only the files needed by the app. Usually this means:
git add data/extracted/CS55.json
git add data/extracted/case_summaries.json
git add data/extracted/precomputed_embeddings.json
git add data/raw/CS55-ShortName_COUNTRY/case_meta.json
git commit -m "Add CS55 case study"
The raw source PDFs are usually not committed. Keep them in Drive or local raw storage unless the team explicitly decides otherwise.
Push to GitHub:
git push shared-github HEAD:main
Push to Hugging Face:
git push uwrfr-chatbot HEAD:main
Updating an Existing Case Study
If a case study already exists and you are improving its sources or extraction:
python3 -m case_study_pipeline.run_case_study \
--case-folder data/raw/CS55-ShortName_COUNTRY \
--case-id CS55 \
--llm-provider claude \
--max-chars-per-source 150000 \
--force
python3 -m case_study_pipeline.select_best_run --case-id CS55
python3 ingest/ingest_pipeline_outputs.py --case-id CS55
python3 ingest/create_summary_chunks.py
python3 ingest/embed_and_index.py
Then test, commit, and push as above.
Common Problems
New case does not show on the website
Re-run:
python3 ingest/create_summary_chunks.py
python3 ingest/embed_and_index.py
Then commit and deploy the updated files in data/extracted/.
Hugging Face rebuilds but old content appears
Check the Space commit SHA:
hf spaces info UWRFR/UWaterlooRFRChatBot
Confirm it matches the commit you pushed.
Drive request upload fails
Check:
GOOGLE_OAUTH_CLIENT_ID,GOOGLE_OAUTH_CLIENT_SECRET, andGOOGLE_OAUTH_REFRESH_TOKENexist as Hugging Face secrets- the Drive folder is owned by or shared with the OAuth Google account
RFR_REQUESTS_DRIVE_FOLDER_IDpoints to the correct folder- the Space has restarted after secrets or variables were added
LLM key fails
Check the relevant Hugging Face secret:
ANTHROPIC_API_KEY
OPENAI_API_KEY
The app currently defaults to Claude for answer generation. The extraction
pipeline can use one provider with --llm-provider claude or
--llm-provider openai.