Spaces:
Sleeping
Sleeping
Anna Zhou commited on
Commit ·
c8016f6
1
Parent(s): be8bd08
Add team handoff documentation
Browse files
docs/01_get_the_code_and_run_locally.md
ADDED
|
@@ -0,0 +1,143 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 01 - Get the Code and Run It Locally
|
| 2 |
+
|
| 3 |
+
This guide is for a team member who wants to edit or run the RFR Knowledge
|
| 4 |
+
Platform from their own computer.
|
| 5 |
+
|
| 6 |
+
## What You Need
|
| 7 |
+
|
| 8 |
+
- A GitHub account with access to the shared repository.
|
| 9 |
+
- Git installed.
|
| 10 |
+
- Python 3.11 or newer.
|
| 11 |
+
- A terminal app.
|
| 12 |
+
- Optional but recommended: Visual Studio Code.
|
| 13 |
+
|
| 14 |
+
## Main Project Links
|
| 15 |
+
|
| 16 |
+
- GitHub repository:
|
| 17 |
+
https://github.com/rfrraguwaterloo-cmyk/UWaterlooManagedRetreatChatbot
|
| 18 |
+
- Hugging Face Space:
|
| 19 |
+
https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot
|
| 20 |
+
- Live website:
|
| 21 |
+
https://uwrfr-uwaterloorfrchatbot.hf.space
|
| 22 |
+
|
| 23 |
+
## Option A: Get the Code from GitHub
|
| 24 |
+
|
| 25 |
+
This is the recommended way to work on the project.
|
| 26 |
+
|
| 27 |
+
```bash
|
| 28 |
+
git clone https://github.com/rfrraguwaterloo-cmyk/UWaterlooManagedRetreatChatbot.git
|
| 29 |
+
cd UWaterlooManagedRetreatChatbot
|
| 30 |
+
```
|
| 31 |
+
|
| 32 |
+
Create a branch before editing:
|
| 33 |
+
|
| 34 |
+
```bash
|
| 35 |
+
git checkout -b your-name-short-change-description
|
| 36 |
+
```
|
| 37 |
+
|
| 38 |
+
Example:
|
| 39 |
+
|
| 40 |
+
```bash
|
| 41 |
+
git checkout -b anna-update-case-request-docs
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
## Option B: Get the Code from Hugging Face
|
| 45 |
+
|
| 46 |
+
This is useful if you only have Hugging Face access.
|
| 47 |
+
|
| 48 |
+
```bash
|
| 49 |
+
git clone https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot
|
| 50 |
+
cd UWaterlooRFRChatBot
|
| 51 |
+
```
|
| 52 |
+
|
| 53 |
+
You can also download with the Hugging Face CLI:
|
| 54 |
+
|
| 55 |
+
```bash
|
| 56 |
+
hf download UWRFR/UWaterlooRFRChatBot --repo-type=space
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
For team development, prefer GitHub because it supports pull requests and code
|
| 60 |
+
review more clearly.
|
| 61 |
+
|
| 62 |
+
## Install Python Dependencies
|
| 63 |
+
|
| 64 |
+
From inside the project folder:
|
| 65 |
+
|
| 66 |
+
```bash
|
| 67 |
+
python3 -m venv venv
|
| 68 |
+
source venv/bin/activate
|
| 69 |
+
pip install --upgrade pip
|
| 70 |
+
pip install -r requirements.txt
|
| 71 |
+
```
|
| 72 |
+
|
| 73 |
+
On Windows PowerShell, activate the virtual environment with:
|
| 74 |
+
|
| 75 |
+
```powershell
|
| 76 |
+
venv\Scripts\Activate.ps1
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
+
## Run the Website Locally
|
| 80 |
+
|
| 81 |
+
```bash
|
| 82 |
+
streamlit run app/app.py
|
| 83 |
+
```
|
| 84 |
+
|
| 85 |
+
The terminal will print a local URL, usually:
|
| 86 |
+
|
| 87 |
+
```text
|
| 88 |
+
http://localhost:8501
|
| 89 |
+
```
|
| 90 |
+
|
| 91 |
+
Open that URL in your browser.
|
| 92 |
+
|
| 93 |
+
## Important Files
|
| 94 |
+
|
| 95 |
+
- `app/app.py` - website tabs, text, layout, and main Streamlit app.
|
| 96 |
+
- `app/drive_request_uploader.py` - uploads new case study requests to Google Drive.
|
| 97 |
+
- `rag/pipeline.py` - sends retrieved context to the LLM for answer generation.
|
| 98 |
+
- `rag/retriever.py` - searches the precomputed embedding file.
|
| 99 |
+
- `data/extracted/` - processed case-study JSON and embeddings used by the website.
|
| 100 |
+
- `data/raw/` - source PDFs, case metadata, and pipeline outputs.
|
| 101 |
+
- `case_study_pipeline/` - extraction and audit workflow.
|
| 102 |
+
- `ingest/` - converts pipeline outputs into website-ready JSON and embeddings.
|
| 103 |
+
- `docs/` - documentation.
|
| 104 |
+
|
| 105 |
+
## Basic Local Test
|
| 106 |
+
|
| 107 |
+
Run this after editing code:
|
| 108 |
+
|
| 109 |
+
```bash
|
| 110 |
+
python3 -m py_compile app/app.py
|
| 111 |
+
streamlit run app/app.py
|
| 112 |
+
```
|
| 113 |
+
|
| 114 |
+
Then check:
|
| 115 |
+
|
| 116 |
+
- The website loads.
|
| 117 |
+
- The navigation buttons stay in the same browser tab.
|
| 118 |
+
- The Ask tab can answer a simple question.
|
| 119 |
+
- The Case Studies tab still opens and lists cases.
|
| 120 |
+
|
| 121 |
+
## Save Your Work
|
| 122 |
+
|
| 123 |
+
Check changed files:
|
| 124 |
+
|
| 125 |
+
```bash
|
| 126 |
+
git status
|
| 127 |
+
```
|
| 128 |
+
|
| 129 |
+
Commit changes:
|
| 130 |
+
|
| 131 |
+
```bash
|
| 132 |
+
git add path/to/changed_file.py
|
| 133 |
+
git commit -m "Describe the change"
|
| 134 |
+
```
|
| 135 |
+
|
| 136 |
+
Push your branch:
|
| 137 |
+
|
| 138 |
+
```bash
|
| 139 |
+
git push origin your-branch-name
|
| 140 |
+
```
|
| 141 |
+
|
| 142 |
+
Then open a pull request on GitHub into `main`.
|
| 143 |
+
|
docs/02_hugging_face_and_api_keys.md
ADDED
|
@@ -0,0 +1,168 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 02 - Hugging Face, Secrets, and API Keys
|
| 2 |
+
|
| 3 |
+
This guide explains where the live website runs and how API keys are managed.
|
| 4 |
+
|
| 5 |
+
## Where the Website Runs
|
| 6 |
+
|
| 7 |
+
The live website is a Hugging Face Space:
|
| 8 |
+
|
| 9 |
+
```text
|
| 10 |
+
https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot
|
| 11 |
+
```
|
| 12 |
+
|
| 13 |
+
The public app URL is:
|
| 14 |
+
|
| 15 |
+
```text
|
| 16 |
+
https://uwrfr-uwaterloorfrchatbot.hf.space
|
| 17 |
+
```
|
| 18 |
+
|
| 19 |
+
Hugging Face stores:
|
| 20 |
+
|
| 21 |
+
- The deployed app code.
|
| 22 |
+
- Space settings.
|
| 23 |
+
- Runtime logs.
|
| 24 |
+
- Variables and secrets.
|
| 25 |
+
- The mounted storage bucket, if enabled.
|
| 26 |
+
|
| 27 |
+
## How to See Space Logs
|
| 28 |
+
|
| 29 |
+
1. Open the Hugging Face Space.
|
| 30 |
+
2. Click **Logs** near the top of the Space page.
|
| 31 |
+
3. Look for Python errors, missing secrets, API failures, or build errors.
|
| 32 |
+
|
| 33 |
+
Useful CLI command:
|
| 34 |
+
|
| 35 |
+
```bash
|
| 36 |
+
hf spaces info UWRFR/UWaterlooRFRChatBot
|
| 37 |
+
```
|
| 38 |
+
|
| 39 |
+
This shows the current commit SHA and runtime state.
|
| 40 |
+
|
| 41 |
+
## Hugging Face Variables and Secrets
|
| 42 |
+
|
| 43 |
+
Go to:
|
| 44 |
+
|
| 45 |
+
```text
|
| 46 |
+
Hugging Face Space -> Settings -> Variables and secrets
|
| 47 |
+
```
|
| 48 |
+
|
| 49 |
+
Use **Variables** for non-sensitive values.
|
| 50 |
+
|
| 51 |
+
Use **Secrets** for private values, such as API keys.
|
| 52 |
+
|
| 53 |
+
Important: Hugging Face does not show the value of an existing secret after it
|
| 54 |
+
is saved. You can see that the secret exists, but you cannot reveal the stored
|
| 55 |
+
key. If a key is lost or wrong, replace the secret with a new value.
|
| 56 |
+
|
| 57 |
+
## Required Secrets
|
| 58 |
+
|
| 59 |
+
For LLM answers:
|
| 60 |
+
|
| 61 |
+
```text
|
| 62 |
+
ANTHROPIC_API_KEY
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
or, if using OpenAI:
|
| 66 |
+
|
| 67 |
+
```text
|
| 68 |
+
OPENAI_API_KEY
|
| 69 |
+
OPENAI_MODEL
|
| 70 |
+
```
|
| 71 |
+
|
| 72 |
+
For Google Drive request uploads:
|
| 73 |
+
|
| 74 |
+
```text
|
| 75 |
+
GOOGLE_SERVICE_ACCOUNT_JSON
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
Optional override for the new case study request intake folder:
|
| 79 |
+
|
| 80 |
+
```text
|
| 81 |
+
RFR_REQUESTS_DRIVE_FOLDER_ID
|
| 82 |
+
```
|
| 83 |
+
|
| 84 |
+
Current request intake folder ID:
|
| 85 |
+
|
| 86 |
+
```text
|
| 87 |
+
10n_-uOCT2GXu_G3r1qi5qJr8hHoy-E_y
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
## What Each Secret Does
|
| 91 |
+
|
| 92 |
+
### `ANTHROPIC_API_KEY`
|
| 93 |
+
|
| 94 |
+
Used by the chatbot when the app is configured to answer with Claude.
|
| 95 |
+
|
| 96 |
+
If this key is missing, expired, or has no credits, the Ask tab can fail.
|
| 97 |
+
|
| 98 |
+
### `OPENAI_API_KEY`
|
| 99 |
+
|
| 100 |
+
Used when the app or local pipeline is configured to use OpenAI.
|
| 101 |
+
|
| 102 |
+
If using OpenAI, also set:
|
| 103 |
+
|
| 104 |
+
```text
|
| 105 |
+
OPENAI_MODEL
|
| 106 |
+
```
|
| 107 |
+
|
| 108 |
+
Example:
|
| 109 |
+
|
| 110 |
+
```text
|
| 111 |
+
gpt-4o-mini
|
| 112 |
+
```
|
| 113 |
+
|
| 114 |
+
### `GOOGLE_SERVICE_ACCOUNT_JSON`
|
| 115 |
+
|
| 116 |
+
Used by the website to upload New Case Study Requests into Google Drive.
|
| 117 |
+
|
| 118 |
+
This should contain the full JSON service account credential, not just the
|
| 119 |
+
service account email.
|
| 120 |
+
|
| 121 |
+
The Drive intake folder must be shared with the service account email. If it is
|
| 122 |
+
not shared, uploads will fail even if the secret is present.
|
| 123 |
+
|
| 124 |
+
## How to Add or Replace a Secret
|
| 125 |
+
|
| 126 |
+
1. Open the Hugging Face Space.
|
| 127 |
+
2. Click **Settings**.
|
| 128 |
+
3. Scroll to **Variables and secrets**.
|
| 129 |
+
4. Click **New secret**.
|
| 130 |
+
5. Enter the exact name, such as `ANTHROPIC_API_KEY`.
|
| 131 |
+
6. Paste the value.
|
| 132 |
+
7. Save.
|
| 133 |
+
8. Restart or rebuild the Space if needed.
|
| 134 |
+
|
| 135 |
+
## How to Check Whether a Secret Is the Problem
|
| 136 |
+
|
| 137 |
+
Open Space logs and look for messages like:
|
| 138 |
+
|
| 139 |
+
```text
|
| 140 |
+
ANTHROPIC_API_KEY is not set
|
| 141 |
+
OPENAI_API_KEY is not set
|
| 142 |
+
GOOGLE_SERVICE_ACCOUNT_JSON is not set
|
| 143 |
+
credit balance is too low
|
| 144 |
+
invalid_api_key
|
| 145 |
+
permission denied
|
| 146 |
+
```
|
| 147 |
+
|
| 148 |
+
For Drive upload problems, check:
|
| 149 |
+
|
| 150 |
+
- The `GOOGLE_SERVICE_ACCOUNT_JSON` secret exists.
|
| 151 |
+
- It is valid JSON.
|
| 152 |
+
- The Drive intake folder is shared with the service account email.
|
| 153 |
+
- The Space was restarted after the secret was added.
|
| 154 |
+
|
| 155 |
+
## Do Not Put API Keys in GitHub
|
| 156 |
+
|
| 157 |
+
Never commit API keys to:
|
| 158 |
+
|
| 159 |
+
- `README.md`
|
| 160 |
+
- `.env`
|
| 161 |
+
- Python files
|
| 162 |
+
- Markdown docs
|
| 163 |
+
- screenshots
|
| 164 |
+
|
| 165 |
+
Use Hugging Face secrets for the live website.
|
| 166 |
+
|
| 167 |
+
Use a local `.env` file for your own computer. Do not commit `.env`.
|
| 168 |
+
|
docs/03_new_case_study_requests_from_drive.md
ADDED
|
@@ -0,0 +1,257 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 03 - New Case Study Requests from Google Drive
|
| 2 |
+
|
| 3 |
+
This guide explains how a submitted New Case Study Request becomes a processed
|
| 4 |
+
case study in the website.
|
| 5 |
+
|
| 6 |
+
## Request Intake Folder
|
| 7 |
+
|
| 8 |
+
New requests from the website are saved in this Google Drive folder:
|
| 9 |
+
|
| 10 |
+
```text
|
| 11 |
+
https://drive.google.com/drive/u/3/folders/10n_-uOCT2GXu_G3r1qi5qJr8hHoy-E_y
|
| 12 |
+
```
|
| 13 |
+
|
| 14 |
+
Folder ID:
|
| 15 |
+
|
| 16 |
+
```text
|
| 17 |
+
10n_-uOCT2GXu_G3r1qi5qJr8hHoy-E_y
|
| 18 |
+
```
|
| 19 |
+
|
| 20 |
+
Each website submission creates a new folder inside that intake folder.
|
| 21 |
+
|
| 22 |
+
Each submitted request folder should contain:
|
| 23 |
+
|
| 24 |
+
- `request.json`
|
| 25 |
+
- uploaded PDFs or source documents, if the user attached any
|
| 26 |
+
|
| 27 |
+
The request form only collects materials. It does not run the LLM extraction
|
| 28 |
+
pipeline automatically.
|
| 29 |
+
|
| 30 |
+
## Documentation Folder
|
| 31 |
+
|
| 32 |
+
These instructions are stored in this Google Drive folder:
|
| 33 |
+
|
| 34 |
+
```text
|
| 35 |
+
https://drive.google.com/drive/u/3/folders/1ZEb9riZZM9qWLYT70KzhdGzygsQHozYM
|
| 36 |
+
```
|
| 37 |
+
|
| 38 |
+
## Step 1: Review a Submitted Request
|
| 39 |
+
|
| 40 |
+
Open the request intake folder and choose a submitted request folder.
|
| 41 |
+
|
| 42 |
+
Read:
|
| 43 |
+
|
| 44 |
+
```text
|
| 45 |
+
request.json
|
| 46 |
+
```
|
| 47 |
+
|
| 48 |
+
Check:
|
| 49 |
+
|
| 50 |
+
- Case study name.
|
| 51 |
+
- Location and country.
|
| 52 |
+
- Why the case should be added.
|
| 53 |
+
- Source links.
|
| 54 |
+
- Uploaded PDFs.
|
| 55 |
+
- Whether the case already exists in `data/raw/` or `data/extracted/`.
|
| 56 |
+
|
| 57 |
+
## Step 2: Choose a Case ID
|
| 58 |
+
|
| 59 |
+
Processed cases currently appear in:
|
| 60 |
+
|
| 61 |
+
```text
|
| 62 |
+
data/extracted/
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
Existing raw/backlog case folders appear in:
|
| 66 |
+
|
| 67 |
+
```text
|
| 68 |
+
data/raw/
|
| 69 |
+
```
|
| 70 |
+
|
| 71 |
+
Commands:
|
| 72 |
+
|
| 73 |
+
```bash
|
| 74 |
+
ls data/extracted/CS*.json | sort -V | tail
|
| 75 |
+
find data/raw -maxdepth 1 -type d -name "CS*" | sort -V | tail
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
If the request is a brand-new case that is not already in the backlog, use the
|
| 79 |
+
next unused case ID after the existing backlog.
|
| 80 |
+
|
| 81 |
+
For example, if the repo already has folders up to `CS123`, use:
|
| 82 |
+
|
| 83 |
+
```text
|
| 84 |
+
CS124
|
| 85 |
+
```
|
| 86 |
+
|
| 87 |
+
If the request matches an existing backlog case, use that existing case ID
|
| 88 |
+
instead.
|
| 89 |
+
|
| 90 |
+
## Step 3: Set Up the Case Locally from the Request Folder
|
| 91 |
+
|
| 92 |
+
The helper script can create a local case folder, create/register the matching
|
| 93 |
+
case Drive folder, and copy PDFs from the submitted request folder.
|
| 94 |
+
|
| 95 |
+
You need a Google OAuth client secret file on your computer, usually named like:
|
| 96 |
+
|
| 97 |
+
```text
|
| 98 |
+
client_secret_....json
|
| 99 |
+
```
|
| 100 |
+
|
| 101 |
+
Run:
|
| 102 |
+
|
| 103 |
+
```bash
|
| 104 |
+
python3 -m case_study_pipeline.onboard_case_study \
|
| 105 |
+
--cs-num 124 \
|
| 106 |
+
--name "Newtok to Mertarvik" \
|
| 107 |
+
--country "United States" \
|
| 108 |
+
--hazard "coastal erosion, flooding, permafrost thaw" \
|
| 109 |
+
--stage "during relocation" \
|
| 110 |
+
--actors "Newtok Village Council, State of Alaska, federal agencies" \
|
| 111 |
+
--source-drive-folder "https://drive.google.com/drive/folders/<REQUEST_FOLDER_ID>"
|
| 112 |
+
```
|
| 113 |
+
|
| 114 |
+
Replace:
|
| 115 |
+
|
| 116 |
+
- `124` with the chosen case number.
|
| 117 |
+
- Name, country, hazard, stage, and actors with the submitted case details.
|
| 118 |
+
- `<REQUEST_FOLDER_ID>` with the Google Drive folder for that request.
|
| 119 |
+
|
| 120 |
+
The first time you use the Drive script, your browser may ask you to sign in and
|
| 121 |
+
approve Google Drive access.
|
| 122 |
+
|
| 123 |
+
## Step 4: Check the Created Files
|
| 124 |
+
|
| 125 |
+
The script creates:
|
| 126 |
+
|
| 127 |
+
```text
|
| 128 |
+
data/raw/CS124/
|
| 129 |
+
data/raw/CS124/case_meta.json
|
| 130 |
+
data/raw/CS124/CS124_paper_discovery_prompt.md
|
| 131 |
+
```
|
| 132 |
+
|
| 133 |
+
It also copies uploaded PDFs into:
|
| 134 |
+
|
| 135 |
+
```text
|
| 136 |
+
data/raw/CS124/
|
| 137 |
+
```
|
| 138 |
+
|
| 139 |
+
Open and edit `case_meta.json` if needed:
|
| 140 |
+
|
| 141 |
+
```json
|
| 142 |
+
{
|
| 143 |
+
"case_id": "CS124",
|
| 144 |
+
"name": "Newtok to Mertarvik",
|
| 145 |
+
"location": "Newtok / Mertarvik, Alaska",
|
| 146 |
+
"country": "United States"
|
| 147 |
+
}
|
| 148 |
+
```
|
| 149 |
+
|
| 150 |
+
## Step 5: Add Source Links for the Website
|
| 151 |
+
|
| 152 |
+
The Case Studies tab displays source links from the extracted JSON.
|
| 153 |
+
|
| 154 |
+
Source links are collected from:
|
| 155 |
+
|
| 156 |
+
- `sources.txt`
|
| 157 |
+
- `sources.json`
|
| 158 |
+
- `dois.txt`
|
| 159 |
+
- `ingest/source_link_overrides.json`
|
| 160 |
+
|
| 161 |
+
For clean APA-style links, add an entry to:
|
| 162 |
+
|
| 163 |
+
```text
|
| 164 |
+
ingest/source_link_overrides.json
|
| 165 |
+
```
|
| 166 |
+
|
| 167 |
+
Example:
|
| 168 |
+
|
| 169 |
+
```json
|
| 170 |
+
"CS124": [
|
| 171 |
+
{
|
| 172 |
+
"title": "Bronen, R., & Chapin, F. S. III. (2013). Adaptive governance and institutional strategies for climate-induced community relocations in Alaska. Proceedings of the National Academy of Sciences, 110(23), 9320-9325. https://doi.org/10.1073/pnas.1210508110",
|
| 173 |
+
"doi": "10.1073/pnas.1210508110",
|
| 174 |
+
"url": "https://doi.org/10.1073/pnas.1210508110"
|
| 175 |
+
}
|
| 176 |
+
]
|
| 177 |
+
```
|
| 178 |
+
|
| 179 |
+
Then re-run ingestion after the extraction pipeline finishes.
|
| 180 |
+
|
| 181 |
+
## Step 6: Run the Extraction Pipeline
|
| 182 |
+
|
| 183 |
+
Use one LLM provider for the whole run. Claude is the current simplest path.
|
| 184 |
+
|
| 185 |
+
```bash
|
| 186 |
+
python3 -m case_study_pipeline.run_case_study \
|
| 187 |
+
--case-folder data/raw/CS124 \
|
| 188 |
+
--case-id CS124 \
|
| 189 |
+
--llm-provider claude \
|
| 190 |
+
--max-chars-per-source 150000 \
|
| 191 |
+
--force
|
| 192 |
+
```
|
| 193 |
+
|
| 194 |
+
This creates:
|
| 195 |
+
|
| 196 |
+
```text
|
| 197 |
+
data/raw/CS124/pipeline_output/CS124_Ver1.md
|
| 198 |
+
data/raw/CS124/pipeline_output/CS124_Check1_report.md
|
| 199 |
+
data/raw/CS124/pipeline_output/CS124_Ver2.md
|
| 200 |
+
data/raw/CS124/pipeline_output/CS124_Check2_report.md
|
| 201 |
+
```
|
| 202 |
+
|
| 203 |
+
## Step 7: Select, Ingest, Embed
|
| 204 |
+
|
| 205 |
+
```bash
|
| 206 |
+
python3 -m case_study_pipeline.select_best_run --case-id CS124
|
| 207 |
+
python3 ingest/ingest_pipeline_outputs.py --case-id CS124
|
| 208 |
+
python3 ingest/create_summary_chunks.py
|
| 209 |
+
python3 ingest/embed_and_index.py
|
| 210 |
+
```
|
| 211 |
+
|
| 212 |
+
These create or update:
|
| 213 |
+
|
| 214 |
+
```text
|
| 215 |
+
data/extracted/CS124.json
|
| 216 |
+
data/extracted/case_summaries.json
|
| 217 |
+
data/extracted/precomputed_embeddings.json
|
| 218 |
+
```
|
| 219 |
+
|
| 220 |
+
## Step 8: Test Locally
|
| 221 |
+
|
| 222 |
+
```bash
|
| 223 |
+
streamlit run app/app.py
|
| 224 |
+
```
|
| 225 |
+
|
| 226 |
+
Check:
|
| 227 |
+
|
| 228 |
+
- The case appears in the Case Studies tab.
|
| 229 |
+
- Source links appear.
|
| 230 |
+
- A direct question works, such as `Summarize CS124`.
|
| 231 |
+
- The answer cites the right case.
|
| 232 |
+
|
| 233 |
+
## Step 9: Commit and Deploy
|
| 234 |
+
|
| 235 |
+
Commit the app-facing files:
|
| 236 |
+
|
| 237 |
+
```bash
|
| 238 |
+
git status
|
| 239 |
+
git add data/extracted/CS124.json
|
| 240 |
+
git add data/extracted/case_summaries.json
|
| 241 |
+
git add data/extracted/precomputed_embeddings.json
|
| 242 |
+
git add data/raw/CS124/case_meta.json
|
| 243 |
+
git add case_study_pipeline/drive_folders.json
|
| 244 |
+
git add ingest/source_link_overrides.json
|
| 245 |
+
git commit -m "Add CS124 case study"
|
| 246 |
+
```
|
| 247 |
+
|
| 248 |
+
Usually do not commit raw PDFs unless the team explicitly wants them in GitHub.
|
| 249 |
+
Keep source PDFs in Google Drive.
|
| 250 |
+
|
| 251 |
+
Push to GitHub and Hugging Face:
|
| 252 |
+
|
| 253 |
+
```bash
|
| 254 |
+
git push shared-github HEAD:main
|
| 255 |
+
git push uwrfr-chatbot HEAD:main
|
| 256 |
+
```
|
| 257 |
+
|
docs/04_terminal_command_cheatsheet.md
ADDED
|
@@ -0,0 +1,171 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 04 - Terminal Command Cheatsheet
|
| 2 |
+
|
| 3 |
+
This file lists common commands in the order a team member is most likely to
|
| 4 |
+
use them.
|
| 5 |
+
|
| 6 |
+
## Start Work
|
| 7 |
+
|
| 8 |
+
```bash
|
| 9 |
+
cd UWaterlooManagedRetreatChatbot
|
| 10 |
+
source venv/bin/activate
|
| 11 |
+
git status
|
| 12 |
+
git pull origin main
|
| 13 |
+
git checkout -b your-branch-name
|
| 14 |
+
```
|
| 15 |
+
|
| 16 |
+
## Run the Website
|
| 17 |
+
|
| 18 |
+
```bash
|
| 19 |
+
streamlit run app/app.py
|
| 20 |
+
```
|
| 21 |
+
|
| 22 |
+
## Check Python Syntax
|
| 23 |
+
|
| 24 |
+
```bash
|
| 25 |
+
python3 -m py_compile app/app.py
|
| 26 |
+
python3 -m py_compile rag/pipeline.py
|
| 27 |
+
python3 -m py_compile rag/retriever.py
|
| 28 |
+
```
|
| 29 |
+
|
| 30 |
+
## Rebuild All Ingested Case Data
|
| 31 |
+
|
| 32 |
+
Use this after changing many pipeline outputs:
|
| 33 |
+
|
| 34 |
+
```bash
|
| 35 |
+
python3 ingest/ingest_pipeline_outputs.py
|
| 36 |
+
python3 ingest/create_summary_chunks.py
|
| 37 |
+
python3 ingest/embed_and_index.py
|
| 38 |
+
```
|
| 39 |
+
|
| 40 |
+
## Rebuild One Case
|
| 41 |
+
|
| 42 |
+
Replace `CS124` with the case ID:
|
| 43 |
+
|
| 44 |
+
```bash
|
| 45 |
+
python3 ingest/ingest_pipeline_outputs.py --case-id CS124
|
| 46 |
+
python3 ingest/create_summary_chunks.py
|
| 47 |
+
python3 ingest/embed_and_index.py
|
| 48 |
+
```
|
| 49 |
+
|
| 50 |
+
## Run One New Case Through the Pipeline
|
| 51 |
+
|
| 52 |
+
```bash
|
| 53 |
+
python3 -m case_study_pipeline.run_case_study \
|
| 54 |
+
--case-folder data/raw/CS124 \
|
| 55 |
+
--case-id CS124 \
|
| 56 |
+
--llm-provider claude \
|
| 57 |
+
--max-chars-per-source 150000 \
|
| 58 |
+
--force
|
| 59 |
+
```
|
| 60 |
+
|
| 61 |
+
Then:
|
| 62 |
+
|
| 63 |
+
```bash
|
| 64 |
+
python3 -m case_study_pipeline.select_best_run --case-id CS124
|
| 65 |
+
python3 ingest/ingest_pipeline_outputs.py --case-id CS124
|
| 66 |
+
python3 ingest/create_summary_chunks.py
|
| 67 |
+
python3 ingest/embed_and_index.py
|
| 68 |
+
```
|
| 69 |
+
|
| 70 |
+
## Download Papers from a Registered Case Drive Folder
|
| 71 |
+
|
| 72 |
+
This uses `case_study_pipeline/drive_folders.json`.
|
| 73 |
+
|
| 74 |
+
```bash
|
| 75 |
+
python3 -m case_study_pipeline.download_drive_papers --case-id CS124
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
## Sort a Downloaded PDF into a Case Folder
|
| 79 |
+
|
| 80 |
+
```bash
|
| 81 |
+
python3 -m case_study_pipeline.sort_inbox --list
|
| 82 |
+
```
|
| 83 |
+
|
| 84 |
+
Then:
|
| 85 |
+
|
| 86 |
+
```bash
|
| 87 |
+
python3 -m case_study_pipeline.sort_inbox \
|
| 88 |
+
--filename "Downloaded Paper.pdf" \
|
| 89 |
+
--case-id CS124 \
|
| 90 |
+
--rename "Author_Year_Short_Title.pdf"
|
| 91 |
+
```
|
| 92 |
+
|
| 93 |
+
## Upload the Current Code to Hugging Face
|
| 94 |
+
|
| 95 |
+
Preferred if your Hugging Face remote exists:
|
| 96 |
+
|
| 97 |
+
```bash
|
| 98 |
+
git push uwrfr-chatbot HEAD:main
|
| 99 |
+
```
|
| 100 |
+
|
| 101 |
+
If the remote does not exist:
|
| 102 |
+
|
| 103 |
+
```bash
|
| 104 |
+
git remote add uwrfr-chatbot https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot
|
| 105 |
+
git push uwrfr-chatbot HEAD:main
|
| 106 |
+
```
|
| 107 |
+
|
| 108 |
+
Alternative using the Hugging Face CLI:
|
| 109 |
+
|
| 110 |
+
```bash
|
| 111 |
+
hf upload UWRFR/UWaterlooRFRChatBot . \
|
| 112 |
+
--type space \
|
| 113 |
+
--exclude ".git/*" \
|
| 114 |
+
--exclude ".github/*" \
|
| 115 |
+
--exclude ".venv/*" \
|
| 116 |
+
--exclude "venv/*" \
|
| 117 |
+
--exclude "__pycache__/*" \
|
| 118 |
+
--exclude "data/raw/*" \
|
| 119 |
+
--exclude "*.pdf" \
|
| 120 |
+
--commit-message "Update website"
|
| 121 |
+
```
|
| 122 |
+
|
| 123 |
+
## Push to Shared GitHub
|
| 124 |
+
|
| 125 |
+
```bash
|
| 126 |
+
git push shared-github HEAD:main
|
| 127 |
+
```
|
| 128 |
+
|
| 129 |
+
If the remote does not exist:
|
| 130 |
+
|
| 131 |
+
```bash
|
| 132 |
+
git remote add shared-github https://github.com/rfrraguwaterloo-cmyk/UWaterlooManagedRetreatChatbot.git
|
| 133 |
+
git push shared-github HEAD:main
|
| 134 |
+
```
|
| 135 |
+
|
| 136 |
+
## Check Hugging Face Space Status
|
| 137 |
+
|
| 138 |
+
```bash
|
| 139 |
+
hf spaces info UWRFR/UWaterlooRFRChatBot
|
| 140 |
+
```
|
| 141 |
+
|
| 142 |
+
Look for:
|
| 143 |
+
|
| 144 |
+
- `sha` - deployed commit.
|
| 145 |
+
- `runtime.stage` - whether it is running, building, or failed.
|
| 146 |
+
- `host` - public app URL.
|
| 147 |
+
|
| 148 |
+
## See Recent Git History
|
| 149 |
+
|
| 150 |
+
```bash
|
| 151 |
+
git log --oneline -10
|
| 152 |
+
```
|
| 153 |
+
|
| 154 |
+
## See What Changed
|
| 155 |
+
|
| 156 |
+
```bash
|
| 157 |
+
git status
|
| 158 |
+
git diff --stat
|
| 159 |
+
```
|
| 160 |
+
|
| 161 |
+
## Undo an Uncommitted Edit to One File
|
| 162 |
+
|
| 163 |
+
Only do this if you are sure you do not want your local changes to that file:
|
| 164 |
+
|
| 165 |
+
```bash
|
| 166 |
+
git restore path/to/file.py
|
| 167 |
+
```
|
| 168 |
+
|
| 169 |
+
Do not use broad destructive commands like `git reset --hard` unless the team
|
| 170 |
+
explicitly agrees.
|
| 171 |
+
|
docs/05_debugging_common_problems.md
ADDED
|
@@ -0,0 +1,221 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 05 - Debugging Common Problems
|
| 2 |
+
|
| 3 |
+
This guide lists common failures and what to check first.
|
| 4 |
+
|
| 5 |
+
## The Website Does Not Load
|
| 6 |
+
|
| 7 |
+
Check Hugging Face logs:
|
| 8 |
+
|
| 9 |
+
```text
|
| 10 |
+
Hugging Face Space -> Logs
|
| 11 |
+
```
|
| 12 |
+
|
| 13 |
+
Common causes:
|
| 14 |
+
|
| 15 |
+
- Build failed.
|
| 16 |
+
- Missing dependency in `requirements.txt`.
|
| 17 |
+
- Python syntax error.
|
| 18 |
+
- Missing required file.
|
| 19 |
+
- Space is still rebuilding.
|
| 20 |
+
|
| 21 |
+
CLI check:
|
| 22 |
+
|
| 23 |
+
```bash
|
| 24 |
+
hf spaces info UWRFR/UWaterlooRFRChatBot
|
| 25 |
+
```
|
| 26 |
+
|
| 27 |
+
If the Space is building, wait and refresh.
|
| 28 |
+
|
| 29 |
+
## Ask Tab Fails with an LLM Error
|
| 30 |
+
|
| 31 |
+
Common messages:
|
| 32 |
+
|
| 33 |
+
```text
|
| 34 |
+
credit balance is too low
|
| 35 |
+
ANTHROPIC_API_KEY is not set
|
| 36 |
+
OPENAI_API_KEY is not set
|
| 37 |
+
APIConnectionError
|
| 38 |
+
invalid_api_key
|
| 39 |
+
```
|
| 40 |
+
|
| 41 |
+
Check:
|
| 42 |
+
|
| 43 |
+
- The correct key exists in Hugging Face secrets.
|
| 44 |
+
- The key has credits.
|
| 45 |
+
- The app is configured for the provider you expect.
|
| 46 |
+
- The Space was restarted after changing secrets.
|
| 47 |
+
|
| 48 |
+
If Anthropic credits are unavailable, switch the app to OpenAI only if
|
| 49 |
+
`OPENAI_API_KEY` is available and configured.
|
| 50 |
+
|
| 51 |
+
## New Case Study Request Upload Fails
|
| 52 |
+
|
| 53 |
+
The request tab uploads to Google Drive using:
|
| 54 |
+
|
| 55 |
+
```text
|
| 56 |
+
GOOGLE_SERVICE_ACCOUNT_JSON
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
Check:
|
| 60 |
+
|
| 61 |
+
- The secret exists in Hugging Face.
|
| 62 |
+
- The secret value is the full service account JSON.
|
| 63 |
+
- The intake folder is shared with the service account email.
|
| 64 |
+
- The intake folder ID is correct:
|
| 65 |
+
|
| 66 |
+
```text
|
| 67 |
+
10n_-uOCT2GXu_G3r1qi5qJr8hHoy-E_y
|
| 68 |
+
```
|
| 69 |
+
|
| 70 |
+
If using a different folder, set:
|
| 71 |
+
|
| 72 |
+
```text
|
| 73 |
+
RFR_REQUESTS_DRIVE_FOLDER_ID
|
| 74 |
+
```
|
| 75 |
+
|
| 76 |
+
## New Case Does Not Appear in Case Studies
|
| 77 |
+
|
| 78 |
+
Usually one of these files was not rebuilt or committed:
|
| 79 |
+
|
| 80 |
+
```text
|
| 81 |
+
data/extracted/CSxx.json
|
| 82 |
+
data/extracted/case_summaries.json
|
| 83 |
+
data/extracted/precomputed_embeddings.json
|
| 84 |
+
```
|
| 85 |
+
|
| 86 |
+
Run:
|
| 87 |
+
|
| 88 |
+
```bash
|
| 89 |
+
python3 ingest/ingest_pipeline_outputs.py --case-id CSxx
|
| 90 |
+
python3 ingest/create_summary_chunks.py
|
| 91 |
+
python3 ingest/embed_and_index.py
|
| 92 |
+
```
|
| 93 |
+
|
| 94 |
+
Then commit and push the changed files.
|
| 95 |
+
|
| 96 |
+
## Case Appears but Sources Are Missing
|
| 97 |
+
|
| 98 |
+
The Case Studies tab reads source links from each case's `source_links` field in
|
| 99 |
+
`data/extracted/CSxx.json`.
|
| 100 |
+
|
| 101 |
+
Source links come from:
|
| 102 |
+
|
| 103 |
+
- `sources.txt`
|
| 104 |
+
- `sources.json`
|
| 105 |
+
- `dois.txt`
|
| 106 |
+
- `ingest/source_link_overrides.json`
|
| 107 |
+
|
| 108 |
+
Fix:
|
| 109 |
+
|
| 110 |
+
1. Add APA-style links to `ingest/source_link_overrides.json`.
|
| 111 |
+
2. Re-run:
|
| 112 |
+
|
| 113 |
+
```bash
|
| 114 |
+
python3 ingest/ingest_pipeline_outputs.py --case-id CSxx --force
|
| 115 |
+
python3 ingest/create_summary_chunks.py
|
| 116 |
+
python3 ingest/embed_and_index.py
|
| 117 |
+
```
|
| 118 |
+
|
| 119 |
+
3. Commit and push.
|
| 120 |
+
|
| 121 |
+
## Case Summary Looks Wrong
|
| 122 |
+
|
| 123 |
+
Check the canonical Ver2 memo:
|
| 124 |
+
|
| 125 |
+
```text
|
| 126 |
+
data/raw/CSxx/pipeline_output/CSxx_Ver2.md
|
| 127 |
+
```
|
| 128 |
+
|
| 129 |
+
If the wrong paper was processed, replace the source files in:
|
| 130 |
+
|
| 131 |
+
```text
|
| 132 |
+
data/raw/CSxx/
|
| 133 |
+
```
|
| 134 |
+
|
| 135 |
+
Then rerun:
|
| 136 |
+
|
| 137 |
+
```bash
|
| 138 |
+
python3 -m case_study_pipeline.run_case_study \
|
| 139 |
+
--case-folder data/raw/CSxx \
|
| 140 |
+
--case-id CSxx \
|
| 141 |
+
--llm-provider claude \
|
| 142 |
+
--max-chars-per-source 150000 \
|
| 143 |
+
--force
|
| 144 |
+
```
|
| 145 |
+
|
| 146 |
+
Then select, ingest, summarize, and embed:
|
| 147 |
+
|
| 148 |
+
```bash
|
| 149 |
+
python3 -m case_study_pipeline.select_best_run --case-id CSxx
|
| 150 |
+
python3 ingest/ingest_pipeline_outputs.py --case-id CSxx --force
|
| 151 |
+
python3 ingest/create_summary_chunks.py
|
| 152 |
+
python3 ingest/embed_and_index.py
|
| 153 |
+
```
|
| 154 |
+
|
| 155 |
+
## Hugging Face Shows Old Content
|
| 156 |
+
|
| 157 |
+
Check the deployed commit:
|
| 158 |
+
|
| 159 |
+
```bash
|
| 160 |
+
hf spaces info UWRFR/UWaterlooRFRChatBot
|
| 161 |
+
```
|
| 162 |
+
|
| 163 |
+
Compare the `sha` with:
|
| 164 |
+
|
| 165 |
+
```bash
|
| 166 |
+
git log --oneline -5
|
| 167 |
+
```
|
| 168 |
+
|
| 169 |
+
If the Hugging Face SHA is old, push again:
|
| 170 |
+
|
| 171 |
+
```bash
|
| 172 |
+
git push uwrfr-chatbot HEAD:main
|
| 173 |
+
```
|
| 174 |
+
|
| 175 |
+
## Git Push Fails
|
| 176 |
+
|
| 177 |
+
Check remotes:
|
| 178 |
+
|
| 179 |
+
```bash
|
| 180 |
+
git remote -v
|
| 181 |
+
```
|
| 182 |
+
|
| 183 |
+
Expected remotes:
|
| 184 |
+
|
| 185 |
+
```text
|
| 186 |
+
shared-github https://github.com/rfrraguwaterloo-cmyk/UWaterlooManagedRetreatChatbot.git
|
| 187 |
+
uwrfr-chatbot https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot
|
| 188 |
+
```
|
| 189 |
+
|
| 190 |
+
If authentication fails:
|
| 191 |
+
|
| 192 |
+
- For GitHub, sign in with GitHub credentials or a GitHub token.
|
| 193 |
+
- For Hugging Face, run:
|
| 194 |
+
|
| 195 |
+
```bash
|
| 196 |
+
hf auth login --add-to-git-credential
|
| 197 |
+
```
|
| 198 |
+
|
| 199 |
+
Use a Hugging Face token with write access to the Space.
|
| 200 |
+
|
| 201 |
+
## Drive Pipeline Output Upload Fails
|
| 202 |
+
|
| 203 |
+
The extraction pipeline can still run locally even if Drive output uploads fail.
|
| 204 |
+
|
| 205 |
+
Common warning:
|
| 206 |
+
|
| 207 |
+
```text
|
| 208 |
+
[Drive] Warning: could not initialize uploader
|
| 209 |
+
Drive uploads will be skipped for this run.
|
| 210 |
+
```
|
| 211 |
+
|
| 212 |
+
This means the local files were created, but the PDF outputs were not uploaded
|
| 213 |
+
to Drive.
|
| 214 |
+
|
| 215 |
+
Check:
|
| 216 |
+
|
| 217 |
+
- `GOOGLE_CREDENTIALS_FILE` points to an OAuth client credential file.
|
| 218 |
+
- Your browser completed Google login.
|
| 219 |
+
- The target case folder exists in `case_study_pipeline/drive_folders.json`.
|
| 220 |
+
- The Google account has permission to write to the Drive folder.
|
| 221 |
+
|
docs/06_deploying_updates.md
ADDED
|
@@ -0,0 +1,153 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 06 - Deploying Updates
|
| 2 |
+
|
| 3 |
+
This guide explains how to get local changes onto GitHub and Hugging Face.
|
| 4 |
+
|
| 5 |
+
## Recommended Workflow
|
| 6 |
+
|
| 7 |
+
Use GitHub as the shared source of truth and Hugging Face as the live app host.
|
| 8 |
+
|
| 9 |
+
The usual flow is:
|
| 10 |
+
|
| 11 |
+
```text
|
| 12 |
+
local computer -> GitHub -> Hugging Face
|
| 13 |
+
```
|
| 14 |
+
|
| 15 |
+
## Before Deploying
|
| 16 |
+
|
| 17 |
+
Run:
|
| 18 |
+
|
| 19 |
+
```bash
|
| 20 |
+
git status
|
| 21 |
+
git diff --stat
|
| 22 |
+
python3 -m py_compile app/app.py
|
| 23 |
+
```
|
| 24 |
+
|
| 25 |
+
If you changed case-study data, also run:
|
| 26 |
+
|
| 27 |
+
```bash
|
| 28 |
+
python3 ingest/create_summary_chunks.py
|
| 29 |
+
python3 ingest/embed_and_index.py
|
| 30 |
+
```
|
| 31 |
+
|
| 32 |
+
If you changed the app, test locally:
|
| 33 |
+
|
| 34 |
+
```bash
|
| 35 |
+
streamlit run app/app.py
|
| 36 |
+
```
|
| 37 |
+
|
| 38 |
+
## Commit the Right Files
|
| 39 |
+
|
| 40 |
+
For website text/code changes:
|
| 41 |
+
|
| 42 |
+
```bash
|
| 43 |
+
git add app/app.py
|
| 44 |
+
git commit -m "Update website"
|
| 45 |
+
```
|
| 46 |
+
|
| 47 |
+
For a new case study:
|
| 48 |
+
|
| 49 |
+
```bash
|
| 50 |
+
git add data/extracted/CS124.json
|
| 51 |
+
git add data/extracted/case_summaries.json
|
| 52 |
+
git add data/extracted/precomputed_embeddings.json
|
| 53 |
+
git add data/raw/CS124/case_meta.json
|
| 54 |
+
git add case_study_pipeline/drive_folders.json
|
| 55 |
+
git add ingest/source_link_overrides.json
|
| 56 |
+
git commit -m "Add CS124 case study"
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
Do not add raw PDFs unless the team explicitly decides to store PDFs in GitHub.
|
| 60 |
+
|
| 61 |
+
## Push to GitHub
|
| 62 |
+
|
| 63 |
+
```bash
|
| 64 |
+
git push shared-github HEAD:main
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
If working on a branch:
|
| 68 |
+
|
| 69 |
+
```bash
|
| 70 |
+
git push origin your-branch-name
|
| 71 |
+
```
|
| 72 |
+
|
| 73 |
+
Then open a pull request into `main`.
|
| 74 |
+
|
| 75 |
+
## Push to Hugging Face
|
| 76 |
+
|
| 77 |
+
Manual push:
|
| 78 |
+
|
| 79 |
+
```bash
|
| 80 |
+
git push uwrfr-chatbot HEAD:main
|
| 81 |
+
```
|
| 82 |
+
|
| 83 |
+
Check status:
|
| 84 |
+
|
| 85 |
+
```bash
|
| 86 |
+
hf spaces info UWRFR/UWaterlooRFRChatBot
|
| 87 |
+
```
|
| 88 |
+
|
| 89 |
+
The Hugging Face Space may take a minute to rebuild.
|
| 90 |
+
|
| 91 |
+
## GitHub Action Deployment
|
| 92 |
+
|
| 93 |
+
The repo has a GitHub Action:
|
| 94 |
+
|
| 95 |
+
```text
|
| 96 |
+
.github/workflows/deploy-hf-space.yml
|
| 97 |
+
```
|
| 98 |
+
|
| 99 |
+
It can upload the repo to a Hugging Face Space.
|
| 100 |
+
|
| 101 |
+
For it to work, GitHub needs:
|
| 102 |
+
|
| 103 |
+
Secret:
|
| 104 |
+
|
| 105 |
+
```text
|
| 106 |
+
HF_TOKEN
|
| 107 |
+
```
|
| 108 |
+
|
| 109 |
+
Variable or workflow input:
|
| 110 |
+
|
| 111 |
+
```text
|
| 112 |
+
HF_SPACE_ID
|
| 113 |
+
```
|
| 114 |
+
|
| 115 |
+
For this project, the target Space is:
|
| 116 |
+
|
| 117 |
+
```text
|
| 118 |
+
UWRFR/UWaterlooRFRChatBot
|
| 119 |
+
```
|
| 120 |
+
|
| 121 |
+
## Files Excluded from Hugging Face Upload
|
| 122 |
+
|
| 123 |
+
The deployment excludes files such as:
|
| 124 |
+
|
| 125 |
+
```text
|
| 126 |
+
.git/
|
| 127 |
+
.github/
|
| 128 |
+
venv/
|
| 129 |
+
__pycache__/
|
| 130 |
+
data/raw/
|
| 131 |
+
*.pdf
|
| 132 |
+
```
|
| 133 |
+
|
| 134 |
+
That means raw source PDFs usually stay in Google Drive, while processed JSON
|
| 135 |
+
and embeddings are deployed to the website.
|
| 136 |
+
|
| 137 |
+
## After Deploying
|
| 138 |
+
|
| 139 |
+
Open:
|
| 140 |
+
|
| 141 |
+
```text
|
| 142 |
+
https://uwrfr-uwaterloorfrchatbot.hf.space
|
| 143 |
+
```
|
| 144 |
+
|
| 145 |
+
Check:
|
| 146 |
+
|
| 147 |
+
- The website loads.
|
| 148 |
+
- The navigation works.
|
| 149 |
+
- New case studies appear.
|
| 150 |
+
- Source links appear.
|
| 151 |
+
- A test question works.
|
| 152 |
+
- Previous Responses still work.
|
| 153 |
+
|