Spaces:
Sleeping
Sleeping
|
Download docs/website_maintenance.md from UWRFR/UWaterlooRFRChatBot: direct link, hf CLI and curl.
- Browser
- Download file 8.43 kB
-
https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot/resolve/main/docs/website_maintenance.md
- Command line
-
hf download hf://spaces/UWRFR/UWaterlooRFRChatBot/docs/website_maintenance.md
-
curl -L -o website_maintenance.md https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot/resolve/main/docs/website_maintenance.md
8.43 kB
| # Website Maintenance and New Case Study Guide | |
| This guide explains how to maintain the RFR Knowledge Platform website and how | |
| to add new managed retreat case studies to the searchable knowledge base. | |
| ## Repositories and Live Website | |
| - Shared GitHub repo: | |
| `https://github.com/rfrraguwaterloo-cmyk/UWaterlooManagedRetreatChatbot` | |
| - Shared Hugging Face Space: | |
| `https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot` | |
| - Live app: | |
| `https://uwrfr-uwaterloorfrchatbot.hf.space` | |
| GitHub is the easiest shared place for code review. Hugging Face is where the | |
| Streamlit app runs. | |
| ## Editing the Website From Another Computer | |
| 1. Install Git and Python 3.11 or newer. | |
| 2. Clone the shared GitHub repo: | |
| ```bash | |
| git clone https://github.com/rfrraguwaterloo-cmyk/UWaterlooManagedRetreatChatbot.git | |
| cd UWaterlooManagedRetreatChatbot | |
| ``` | |
| 3. Create a working branch: | |
| ```bash | |
| git checkout -b update-website-text | |
| ``` | |
| 4. Install dependencies: | |
| ```bash | |
| python3 -m venv venv | |
| source venv/bin/activate | |
| pip install -r requirements.txt | |
| ``` | |
| 5. Run the website locally: | |
| ```bash | |
| streamlit run app/app.py | |
| ``` | |
| 6. Edit the code. Most website text and tabs are in: | |
| ```text | |
| app/app.py | |
| ``` | |
| 7. Check the app still compiles: | |
| ```bash | |
| python3 -m py_compile app/app.py | |
| ``` | |
| 8. Commit and push: | |
| ```bash | |
| git status | |
| git add app/app.py | |
| git commit -m "Update website text" | |
| git push origin update-website-text | |
| ``` | |
| 9. Open a Pull Request on GitHub into `main`. | |
| After the change is merged into `main`, the GitHub Action can deploy to Hugging | |
| Face if the shared repo has `HF_TOKEN` and `HF_SPACE_ID` configured. The team can | |
| also push manually to Hugging Face from a machine that is logged in with `hf`. | |
| ## Manual Hugging Face Push | |
| From a local repo with the Hugging Face CLI logged in: | |
| ```bash | |
| git remote add uwrfr-chatbot https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot | |
| git push uwrfr-chatbot HEAD:main | |
| ``` | |
| If the remote already exists, update it instead: | |
| ```bash | |
| git remote set-url uwrfr-chatbot https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot | |
| git push uwrfr-chatbot HEAD:main | |
| ``` | |
| Or upload the current folder with the CLI: | |
| ```bash | |
| hf upload UWRFR/UWaterlooRFRChatBot . \ | |
| --type space \ | |
| --exclude ".git/*" \ | |
| --exclude ".github/*" \ | |
| --exclude ".venv/*" \ | |
| --exclude "venv/*" \ | |
| --exclude "__pycache__/*" \ | |
| --exclude "data/raw/*" \ | |
| --exclude "*.pdf" \ | |
| --commit-message "Update website" | |
| ``` | |
| Check Space status: | |
| ```bash | |
| hf spaces info UWRFR/UWaterlooRFRChatBot | |
| ``` | |
| ## Website Secrets | |
| Set these in Hugging Face Space settings under **Variables and secrets**. | |
| Required for the chatbot: | |
| ```text | |
| ANTHROPIC_API_KEY | |
| ``` | |
| Optional if using OpenAI mode: | |
| ```text | |
| OPENAI_API_KEY | |
| OPENAI_MODEL | |
| ``` | |
| Required for Google Drive case-study request uploads: | |
| ```text | |
| GOOGLE_OAUTH_CLIENT_ID | |
| GOOGLE_OAUTH_CLIENT_SECRET | |
| GOOGLE_OAUTH_REFRESH_TOKEN | |
| ``` | |
| Optional Drive override for new case study requests: | |
| ```text | |
| RFR_REQUESTS_DRIVE_FOLDER_ID | |
| ``` | |
| By default, new case study requests are uploaded to this Drive folder: | |
| ```text | |
| 10n_-uOCT2GXu_G3r1qi5qJr8hHoy-E_y | |
| ``` | |
| The OAuth token should be generated by signing in as the Google account that | |
| should own request uploads, usually `rfr.rag.uwaterloo@gmail.com`. | |
| Generate the Hugging Face OAuth secrets locally: | |
| ```bash | |
| python3 -m case_study_pipeline.create_drive_oauth_secrets \ | |
| --credentials-file /path/to/oauth-client.json \ | |
| --space-id UWRFR/UWaterlooRFRChatBot | |
| ``` | |
| The older fallback `GOOGLE_SERVICE_ACCOUNT_JSON` service-account setup can fail | |
| in regular My Drive folders because service accounts do not have storage quota. | |
| Use OAuth or a Google Shared Drive for request uploads. | |
| ## How New Case Study Requests Work | |
| Users can submit candidate papers through the **New Case Study Requests** tab. | |
| The app creates a new Google Drive folder for each request. Each folder contains: | |
| - `request.json` | |
| - uploaded PDFs or supporting documents | |
| The request tab only collects files and notes. It does not run the extraction | |
| pipeline and does not call an LLM. | |
| ## Adding a New Case Study to the Knowledge Base | |
| Use this workflow after the team accepts a submitted request or identifies a new | |
| case study. | |
| ### 1. Choose the Next Case ID | |
| Check the highest indexed case: | |
| ```bash | |
| ls data/extracted/CS*.json | sort -V | tail | |
| ``` | |
| If the latest indexed case is `CS54`, the next one is usually `CS55`. | |
| ### 2. Create the Raw Case Folder | |
| Create a folder under `data/raw/`: | |
| ```bash | |
| mkdir -p data/raw/CS55-ShortName_COUNTRY | |
| ``` | |
| Put source PDFs, `.md`, or `.txt` files in that folder. | |
| Add or confirm `case_meta.json` in that folder. Example: | |
| ```json | |
| { | |
| "case_id": "CS55", | |
| "name": "Short case study name", | |
| "location": "City or region", | |
| "country": "Country" | |
| } | |
| ``` | |
| ### 3. Run the One-Provider Extraction Pipeline | |
| Use one provider for all four extraction and audit steps. Claude is the current | |
| simplest option: | |
| ```bash | |
| python3 -m case_study_pipeline.run_case_study \ | |
| --case-folder data/raw/CS55-ShortName_COUNTRY \ | |
| --case-id CS55 \ | |
| --llm-provider claude \ | |
| --max-chars-per-source 150000 \ | |
| --force | |
| ``` | |
| This writes outputs to: | |
| ```text | |
| data/raw/CS55-ShortName_COUNTRY/pipeline_output/ | |
| ``` | |
| Expected output files: | |
| - `CS55_Ver1.md` | |
| - `CS55_Check1_report.md` | |
| - `CS55_Ver2.md` | |
| - `CS55_Check2_report.md` | |
| - matching PDFs | |
| ### 4. Select the Best Run | |
| ```bash | |
| python3 -m case_study_pipeline.select_best_run --case-id CS55 | |
| ``` | |
| This creates or updates: | |
| ```text | |
| data/raw/CS55-ShortName_COUNTRY/pipeline_output/selected.json | |
| ``` | |
| ### 5. Ingest the Case Study | |
| ```bash | |
| python3 ingest/ingest_pipeline_outputs.py --case-id CS55 | |
| ``` | |
| This creates: | |
| ```text | |
| data/extracted/CS55.json | |
| ``` | |
| ### 6. Rebuild Summaries and Embeddings | |
| Always run both commands after adding or changing extracted case data: | |
| ```bash | |
| python3 ingest/create_summary_chunks.py | |
| python3 ingest/embed_and_index.py | |
| ``` | |
| These update: | |
| ```text | |
| data/extracted/case_summaries.json | |
| data/extracted/precomputed_embeddings.json | |
| ``` | |
| The live website will not know about the new case until these files are updated | |
| and deployed. | |
| ### 7. Test Locally | |
| ```bash | |
| streamlit run app/app.py | |
| ``` | |
| Check: | |
| - the new case appears in **Case Studies** | |
| - the app can answer a direct question such as `Summarize CS55` | |
| - source links appear if `source_links` metadata is available | |
| ### 8. Commit and Push | |
| Commit only the files needed by the app. Usually this means: | |
| ```bash | |
| git add data/extracted/CS55.json | |
| git add data/extracted/case_summaries.json | |
| git add data/extracted/precomputed_embeddings.json | |
| git add data/raw/CS55-ShortName_COUNTRY/case_meta.json | |
| git commit -m "Add CS55 case study" | |
| ``` | |
| The raw source PDFs are usually not committed. Keep them in Drive or local raw | |
| storage unless the team explicitly decides otherwise. | |
| Push to GitHub: | |
| ```bash | |
| git push shared-github HEAD:main | |
| ``` | |
| Push to Hugging Face: | |
| ```bash | |
| git push uwrfr-chatbot HEAD:main | |
| ``` | |
| ## Updating an Existing Case Study | |
| If a case study already exists and you are improving its sources or extraction: | |
| ```bash | |
| python3 -m case_study_pipeline.run_case_study \ | |
| --case-folder data/raw/CS55-ShortName_COUNTRY \ | |
| --case-id CS55 \ | |
| --llm-provider claude \ | |
| --max-chars-per-source 150000 \ | |
| --force | |
| python3 -m case_study_pipeline.select_best_run --case-id CS55 | |
| python3 ingest/ingest_pipeline_outputs.py --case-id CS55 | |
| python3 ingest/create_summary_chunks.py | |
| python3 ingest/embed_and_index.py | |
| ``` | |
| Then test, commit, and push as above. | |
| ## Common Problems | |
| ### New case does not show on the website | |
| Re-run: | |
| ```bash | |
| python3 ingest/create_summary_chunks.py | |
| python3 ingest/embed_and_index.py | |
| ``` | |
| Then commit and deploy the updated files in `data/extracted/`. | |
| ### Hugging Face rebuilds but old content appears | |
| Check the Space commit SHA: | |
| ```bash | |
| hf spaces info UWRFR/UWaterlooRFRChatBot | |
| ``` | |
| Confirm it matches the commit you pushed. | |
| ### Drive request upload fails | |
| Check: | |
| - `GOOGLE_OAUTH_CLIENT_ID`, `GOOGLE_OAUTH_CLIENT_SECRET`, and | |
| `GOOGLE_OAUTH_REFRESH_TOKEN` exist as Hugging Face secrets | |
| - the Drive folder is owned by or shared with the OAuth Google account | |
| - `RFR_REQUESTS_DRIVE_FOLDER_ID` points to the correct folder | |
| - the Space has restarted after secrets or variables were added | |
| ### LLM key fails | |
| Check the relevant Hugging Face secret: | |
| ```text | |
| ANTHROPIC_API_KEY | |
| OPENAI_API_KEY | |
| ``` | |
| The app currently defaults to Claude for answer generation. The extraction | |
| pipeline can use one provider with `--llm-provider claude` or | |
| `--llm-provider openai`. | |