Anna Zhou commited on
Commit
c8016f6
·
1 Parent(s): be8bd08

Add team handoff documentation

Browse files
docs/01_get_the_code_and_run_locally.md ADDED
@@ -0,0 +1,143 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # 01 - Get the Code and Run It Locally
2
+
3
+ This guide is for a team member who wants to edit or run the RFR Knowledge
4
+ Platform from their own computer.
5
+
6
+ ## What You Need
7
+
8
+ - A GitHub account with access to the shared repository.
9
+ - Git installed.
10
+ - Python 3.11 or newer.
11
+ - A terminal app.
12
+ - Optional but recommended: Visual Studio Code.
13
+
14
+ ## Main Project Links
15
+
16
+ - GitHub repository:
17
+ https://github.com/rfrraguwaterloo-cmyk/UWaterlooManagedRetreatChatbot
18
+ - Hugging Face Space:
19
+ https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot
20
+ - Live website:
21
+ https://uwrfr-uwaterloorfrchatbot.hf.space
22
+
23
+ ## Option A: Get the Code from GitHub
24
+
25
+ This is the recommended way to work on the project.
26
+
27
+ ```bash
28
+ git clone https://github.com/rfrraguwaterloo-cmyk/UWaterlooManagedRetreatChatbot.git
29
+ cd UWaterlooManagedRetreatChatbot
30
+ ```
31
+
32
+ Create a branch before editing:
33
+
34
+ ```bash
35
+ git checkout -b your-name-short-change-description
36
+ ```
37
+
38
+ Example:
39
+
40
+ ```bash
41
+ git checkout -b anna-update-case-request-docs
42
+ ```
43
+
44
+ ## Option B: Get the Code from Hugging Face
45
+
46
+ This is useful if you only have Hugging Face access.
47
+
48
+ ```bash
49
+ git clone https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot
50
+ cd UWaterlooRFRChatBot
51
+ ```
52
+
53
+ You can also download with the Hugging Face CLI:
54
+
55
+ ```bash
56
+ hf download UWRFR/UWaterlooRFRChatBot --repo-type=space
57
+ ```
58
+
59
+ For team development, prefer GitHub because it supports pull requests and code
60
+ review more clearly.
61
+
62
+ ## Install Python Dependencies
63
+
64
+ From inside the project folder:
65
+
66
+ ```bash
67
+ python3 -m venv venv
68
+ source venv/bin/activate
69
+ pip install --upgrade pip
70
+ pip install -r requirements.txt
71
+ ```
72
+
73
+ On Windows PowerShell, activate the virtual environment with:
74
+
75
+ ```powershell
76
+ venv\Scripts\Activate.ps1
77
+ ```
78
+
79
+ ## Run the Website Locally
80
+
81
+ ```bash
82
+ streamlit run app/app.py
83
+ ```
84
+
85
+ The terminal will print a local URL, usually:
86
+
87
+ ```text
88
+ http://localhost:8501
89
+ ```
90
+
91
+ Open that URL in your browser.
92
+
93
+ ## Important Files
94
+
95
+ - `app/app.py` - website tabs, text, layout, and main Streamlit app.
96
+ - `app/drive_request_uploader.py` - uploads new case study requests to Google Drive.
97
+ - `rag/pipeline.py` - sends retrieved context to the LLM for answer generation.
98
+ - `rag/retriever.py` - searches the precomputed embedding file.
99
+ - `data/extracted/` - processed case-study JSON and embeddings used by the website.
100
+ - `data/raw/` - source PDFs, case metadata, and pipeline outputs.
101
+ - `case_study_pipeline/` - extraction and audit workflow.
102
+ - `ingest/` - converts pipeline outputs into website-ready JSON and embeddings.
103
+ - `docs/` - documentation.
104
+
105
+ ## Basic Local Test
106
+
107
+ Run this after editing code:
108
+
109
+ ```bash
110
+ python3 -m py_compile app/app.py
111
+ streamlit run app/app.py
112
+ ```
113
+
114
+ Then check:
115
+
116
+ - The website loads.
117
+ - The navigation buttons stay in the same browser tab.
118
+ - The Ask tab can answer a simple question.
119
+ - The Case Studies tab still opens and lists cases.
120
+
121
+ ## Save Your Work
122
+
123
+ Check changed files:
124
+
125
+ ```bash
126
+ git status
127
+ ```
128
+
129
+ Commit changes:
130
+
131
+ ```bash
132
+ git add path/to/changed_file.py
133
+ git commit -m "Describe the change"
134
+ ```
135
+
136
+ Push your branch:
137
+
138
+ ```bash
139
+ git push origin your-branch-name
140
+ ```
141
+
142
+ Then open a pull request on GitHub into `main`.
143
+
docs/02_hugging_face_and_api_keys.md ADDED
@@ -0,0 +1,168 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # 02 - Hugging Face, Secrets, and API Keys
2
+
3
+ This guide explains where the live website runs and how API keys are managed.
4
+
5
+ ## Where the Website Runs
6
+
7
+ The live website is a Hugging Face Space:
8
+
9
+ ```text
10
+ https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot
11
+ ```
12
+
13
+ The public app URL is:
14
+
15
+ ```text
16
+ https://uwrfr-uwaterloorfrchatbot.hf.space
17
+ ```
18
+
19
+ Hugging Face stores:
20
+
21
+ - The deployed app code.
22
+ - Space settings.
23
+ - Runtime logs.
24
+ - Variables and secrets.
25
+ - The mounted storage bucket, if enabled.
26
+
27
+ ## How to See Space Logs
28
+
29
+ 1. Open the Hugging Face Space.
30
+ 2. Click **Logs** near the top of the Space page.
31
+ 3. Look for Python errors, missing secrets, API failures, or build errors.
32
+
33
+ Useful CLI command:
34
+
35
+ ```bash
36
+ hf spaces info UWRFR/UWaterlooRFRChatBot
37
+ ```
38
+
39
+ This shows the current commit SHA and runtime state.
40
+
41
+ ## Hugging Face Variables and Secrets
42
+
43
+ Go to:
44
+
45
+ ```text
46
+ Hugging Face Space -> Settings -> Variables and secrets
47
+ ```
48
+
49
+ Use **Variables** for non-sensitive values.
50
+
51
+ Use **Secrets** for private values, such as API keys.
52
+
53
+ Important: Hugging Face does not show the value of an existing secret after it
54
+ is saved. You can see that the secret exists, but you cannot reveal the stored
55
+ key. If a key is lost or wrong, replace the secret with a new value.
56
+
57
+ ## Required Secrets
58
+
59
+ For LLM answers:
60
+
61
+ ```text
62
+ ANTHROPIC_API_KEY
63
+ ```
64
+
65
+ or, if using OpenAI:
66
+
67
+ ```text
68
+ OPENAI_API_KEY
69
+ OPENAI_MODEL
70
+ ```
71
+
72
+ For Google Drive request uploads:
73
+
74
+ ```text
75
+ GOOGLE_SERVICE_ACCOUNT_JSON
76
+ ```
77
+
78
+ Optional override for the new case study request intake folder:
79
+
80
+ ```text
81
+ RFR_REQUESTS_DRIVE_FOLDER_ID
82
+ ```
83
+
84
+ Current request intake folder ID:
85
+
86
+ ```text
87
+ 10n_-uOCT2GXu_G3r1qi5qJr8hHoy-E_y
88
+ ```
89
+
90
+ ## What Each Secret Does
91
+
92
+ ### `ANTHROPIC_API_KEY`
93
+
94
+ Used by the chatbot when the app is configured to answer with Claude.
95
+
96
+ If this key is missing, expired, or has no credits, the Ask tab can fail.
97
+
98
+ ### `OPENAI_API_KEY`
99
+
100
+ Used when the app or local pipeline is configured to use OpenAI.
101
+
102
+ If using OpenAI, also set:
103
+
104
+ ```text
105
+ OPENAI_MODEL
106
+ ```
107
+
108
+ Example:
109
+
110
+ ```text
111
+ gpt-4o-mini
112
+ ```
113
+
114
+ ### `GOOGLE_SERVICE_ACCOUNT_JSON`
115
+
116
+ Used by the website to upload New Case Study Requests into Google Drive.
117
+
118
+ This should contain the full JSON service account credential, not just the
119
+ service account email.
120
+
121
+ The Drive intake folder must be shared with the service account email. If it is
122
+ not shared, uploads will fail even if the secret is present.
123
+
124
+ ## How to Add or Replace a Secret
125
+
126
+ 1. Open the Hugging Face Space.
127
+ 2. Click **Settings**.
128
+ 3. Scroll to **Variables and secrets**.
129
+ 4. Click **New secret**.
130
+ 5. Enter the exact name, such as `ANTHROPIC_API_KEY`.
131
+ 6. Paste the value.
132
+ 7. Save.
133
+ 8. Restart or rebuild the Space if needed.
134
+
135
+ ## How to Check Whether a Secret Is the Problem
136
+
137
+ Open Space logs and look for messages like:
138
+
139
+ ```text
140
+ ANTHROPIC_API_KEY is not set
141
+ OPENAI_API_KEY is not set
142
+ GOOGLE_SERVICE_ACCOUNT_JSON is not set
143
+ credit balance is too low
144
+ invalid_api_key
145
+ permission denied
146
+ ```
147
+
148
+ For Drive upload problems, check:
149
+
150
+ - The `GOOGLE_SERVICE_ACCOUNT_JSON` secret exists.
151
+ - It is valid JSON.
152
+ - The Drive intake folder is shared with the service account email.
153
+ - The Space was restarted after the secret was added.
154
+
155
+ ## Do Not Put API Keys in GitHub
156
+
157
+ Never commit API keys to:
158
+
159
+ - `README.md`
160
+ - `.env`
161
+ - Python files
162
+ - Markdown docs
163
+ - screenshots
164
+
165
+ Use Hugging Face secrets for the live website.
166
+
167
+ Use a local `.env` file for your own computer. Do not commit `.env`.
168
+
docs/03_new_case_study_requests_from_drive.md ADDED
@@ -0,0 +1,257 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # 03 - New Case Study Requests from Google Drive
2
+
3
+ This guide explains how a submitted New Case Study Request becomes a processed
4
+ case study in the website.
5
+
6
+ ## Request Intake Folder
7
+
8
+ New requests from the website are saved in this Google Drive folder:
9
+
10
+ ```text
11
+ https://drive.google.com/drive/u/3/folders/10n_-uOCT2GXu_G3r1qi5qJr8hHoy-E_y
12
+ ```
13
+
14
+ Folder ID:
15
+
16
+ ```text
17
+ 10n_-uOCT2GXu_G3r1qi5qJr8hHoy-E_y
18
+ ```
19
+
20
+ Each website submission creates a new folder inside that intake folder.
21
+
22
+ Each submitted request folder should contain:
23
+
24
+ - `request.json`
25
+ - uploaded PDFs or source documents, if the user attached any
26
+
27
+ The request form only collects materials. It does not run the LLM extraction
28
+ pipeline automatically.
29
+
30
+ ## Documentation Folder
31
+
32
+ These instructions are stored in this Google Drive folder:
33
+
34
+ ```text
35
+ https://drive.google.com/drive/u/3/folders/1ZEb9riZZM9qWLYT70KzhdGzygsQHozYM
36
+ ```
37
+
38
+ ## Step 1: Review a Submitted Request
39
+
40
+ Open the request intake folder and choose a submitted request folder.
41
+
42
+ Read:
43
+
44
+ ```text
45
+ request.json
46
+ ```
47
+
48
+ Check:
49
+
50
+ - Case study name.
51
+ - Location and country.
52
+ - Why the case should be added.
53
+ - Source links.
54
+ - Uploaded PDFs.
55
+ - Whether the case already exists in `data/raw/` or `data/extracted/`.
56
+
57
+ ## Step 2: Choose a Case ID
58
+
59
+ Processed cases currently appear in:
60
+
61
+ ```text
62
+ data/extracted/
63
+ ```
64
+
65
+ Existing raw/backlog case folders appear in:
66
+
67
+ ```text
68
+ data/raw/
69
+ ```
70
+
71
+ Commands:
72
+
73
+ ```bash
74
+ ls data/extracted/CS*.json | sort -V | tail
75
+ find data/raw -maxdepth 1 -type d -name "CS*" | sort -V | tail
76
+ ```
77
+
78
+ If the request is a brand-new case that is not already in the backlog, use the
79
+ next unused case ID after the existing backlog.
80
+
81
+ For example, if the repo already has folders up to `CS123`, use:
82
+
83
+ ```text
84
+ CS124
85
+ ```
86
+
87
+ If the request matches an existing backlog case, use that existing case ID
88
+ instead.
89
+
90
+ ## Step 3: Set Up the Case Locally from the Request Folder
91
+
92
+ The helper script can create a local case folder, create/register the matching
93
+ case Drive folder, and copy PDFs from the submitted request folder.
94
+
95
+ You need a Google OAuth client secret file on your computer, usually named like:
96
+
97
+ ```text
98
+ client_secret_....json
99
+ ```
100
+
101
+ Run:
102
+
103
+ ```bash
104
+ python3 -m case_study_pipeline.onboard_case_study \
105
+ --cs-num 124 \
106
+ --name "Newtok to Mertarvik" \
107
+ --country "United States" \
108
+ --hazard "coastal erosion, flooding, permafrost thaw" \
109
+ --stage "during relocation" \
110
+ --actors "Newtok Village Council, State of Alaska, federal agencies" \
111
+ --source-drive-folder "https://drive.google.com/drive/folders/<REQUEST_FOLDER_ID>"
112
+ ```
113
+
114
+ Replace:
115
+
116
+ - `124` with the chosen case number.
117
+ - Name, country, hazard, stage, and actors with the submitted case details.
118
+ - `<REQUEST_FOLDER_ID>` with the Google Drive folder for that request.
119
+
120
+ The first time you use the Drive script, your browser may ask you to sign in and
121
+ approve Google Drive access.
122
+
123
+ ## Step 4: Check the Created Files
124
+
125
+ The script creates:
126
+
127
+ ```text
128
+ data/raw/CS124/
129
+ data/raw/CS124/case_meta.json
130
+ data/raw/CS124/CS124_paper_discovery_prompt.md
131
+ ```
132
+
133
+ It also copies uploaded PDFs into:
134
+
135
+ ```text
136
+ data/raw/CS124/
137
+ ```
138
+
139
+ Open and edit `case_meta.json` if needed:
140
+
141
+ ```json
142
+ {
143
+ "case_id": "CS124",
144
+ "name": "Newtok to Mertarvik",
145
+ "location": "Newtok / Mertarvik, Alaska",
146
+ "country": "United States"
147
+ }
148
+ ```
149
+
150
+ ## Step 5: Add Source Links for the Website
151
+
152
+ The Case Studies tab displays source links from the extracted JSON.
153
+
154
+ Source links are collected from:
155
+
156
+ - `sources.txt`
157
+ - `sources.json`
158
+ - `dois.txt`
159
+ - `ingest/source_link_overrides.json`
160
+
161
+ For clean APA-style links, add an entry to:
162
+
163
+ ```text
164
+ ingest/source_link_overrides.json
165
+ ```
166
+
167
+ Example:
168
+
169
+ ```json
170
+ "CS124": [
171
+ {
172
+ "title": "Bronen, R., & Chapin, F. S. III. (2013). Adaptive governance and institutional strategies for climate-induced community relocations in Alaska. Proceedings of the National Academy of Sciences, 110(23), 9320-9325. https://doi.org/10.1073/pnas.1210508110",
173
+ "doi": "10.1073/pnas.1210508110",
174
+ "url": "https://doi.org/10.1073/pnas.1210508110"
175
+ }
176
+ ]
177
+ ```
178
+
179
+ Then re-run ingestion after the extraction pipeline finishes.
180
+
181
+ ## Step 6: Run the Extraction Pipeline
182
+
183
+ Use one LLM provider for the whole run. Claude is the current simplest path.
184
+
185
+ ```bash
186
+ python3 -m case_study_pipeline.run_case_study \
187
+ --case-folder data/raw/CS124 \
188
+ --case-id CS124 \
189
+ --llm-provider claude \
190
+ --max-chars-per-source 150000 \
191
+ --force
192
+ ```
193
+
194
+ This creates:
195
+
196
+ ```text
197
+ data/raw/CS124/pipeline_output/CS124_Ver1.md
198
+ data/raw/CS124/pipeline_output/CS124_Check1_report.md
199
+ data/raw/CS124/pipeline_output/CS124_Ver2.md
200
+ data/raw/CS124/pipeline_output/CS124_Check2_report.md
201
+ ```
202
+
203
+ ## Step 7: Select, Ingest, Embed
204
+
205
+ ```bash
206
+ python3 -m case_study_pipeline.select_best_run --case-id CS124
207
+ python3 ingest/ingest_pipeline_outputs.py --case-id CS124
208
+ python3 ingest/create_summary_chunks.py
209
+ python3 ingest/embed_and_index.py
210
+ ```
211
+
212
+ These create or update:
213
+
214
+ ```text
215
+ data/extracted/CS124.json
216
+ data/extracted/case_summaries.json
217
+ data/extracted/precomputed_embeddings.json
218
+ ```
219
+
220
+ ## Step 8: Test Locally
221
+
222
+ ```bash
223
+ streamlit run app/app.py
224
+ ```
225
+
226
+ Check:
227
+
228
+ - The case appears in the Case Studies tab.
229
+ - Source links appear.
230
+ - A direct question works, such as `Summarize CS124`.
231
+ - The answer cites the right case.
232
+
233
+ ## Step 9: Commit and Deploy
234
+
235
+ Commit the app-facing files:
236
+
237
+ ```bash
238
+ git status
239
+ git add data/extracted/CS124.json
240
+ git add data/extracted/case_summaries.json
241
+ git add data/extracted/precomputed_embeddings.json
242
+ git add data/raw/CS124/case_meta.json
243
+ git add case_study_pipeline/drive_folders.json
244
+ git add ingest/source_link_overrides.json
245
+ git commit -m "Add CS124 case study"
246
+ ```
247
+
248
+ Usually do not commit raw PDFs unless the team explicitly wants them in GitHub.
249
+ Keep source PDFs in Google Drive.
250
+
251
+ Push to GitHub and Hugging Face:
252
+
253
+ ```bash
254
+ git push shared-github HEAD:main
255
+ git push uwrfr-chatbot HEAD:main
256
+ ```
257
+
docs/04_terminal_command_cheatsheet.md ADDED
@@ -0,0 +1,171 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # 04 - Terminal Command Cheatsheet
2
+
3
+ This file lists common commands in the order a team member is most likely to
4
+ use them.
5
+
6
+ ## Start Work
7
+
8
+ ```bash
9
+ cd UWaterlooManagedRetreatChatbot
10
+ source venv/bin/activate
11
+ git status
12
+ git pull origin main
13
+ git checkout -b your-branch-name
14
+ ```
15
+
16
+ ## Run the Website
17
+
18
+ ```bash
19
+ streamlit run app/app.py
20
+ ```
21
+
22
+ ## Check Python Syntax
23
+
24
+ ```bash
25
+ python3 -m py_compile app/app.py
26
+ python3 -m py_compile rag/pipeline.py
27
+ python3 -m py_compile rag/retriever.py
28
+ ```
29
+
30
+ ## Rebuild All Ingested Case Data
31
+
32
+ Use this after changing many pipeline outputs:
33
+
34
+ ```bash
35
+ python3 ingest/ingest_pipeline_outputs.py
36
+ python3 ingest/create_summary_chunks.py
37
+ python3 ingest/embed_and_index.py
38
+ ```
39
+
40
+ ## Rebuild One Case
41
+
42
+ Replace `CS124` with the case ID:
43
+
44
+ ```bash
45
+ python3 ingest/ingest_pipeline_outputs.py --case-id CS124
46
+ python3 ingest/create_summary_chunks.py
47
+ python3 ingest/embed_and_index.py
48
+ ```
49
+
50
+ ## Run One New Case Through the Pipeline
51
+
52
+ ```bash
53
+ python3 -m case_study_pipeline.run_case_study \
54
+ --case-folder data/raw/CS124 \
55
+ --case-id CS124 \
56
+ --llm-provider claude \
57
+ --max-chars-per-source 150000 \
58
+ --force
59
+ ```
60
+
61
+ Then:
62
+
63
+ ```bash
64
+ python3 -m case_study_pipeline.select_best_run --case-id CS124
65
+ python3 ingest/ingest_pipeline_outputs.py --case-id CS124
66
+ python3 ingest/create_summary_chunks.py
67
+ python3 ingest/embed_and_index.py
68
+ ```
69
+
70
+ ## Download Papers from a Registered Case Drive Folder
71
+
72
+ This uses `case_study_pipeline/drive_folders.json`.
73
+
74
+ ```bash
75
+ python3 -m case_study_pipeline.download_drive_papers --case-id CS124
76
+ ```
77
+
78
+ ## Sort a Downloaded PDF into a Case Folder
79
+
80
+ ```bash
81
+ python3 -m case_study_pipeline.sort_inbox --list
82
+ ```
83
+
84
+ Then:
85
+
86
+ ```bash
87
+ python3 -m case_study_pipeline.sort_inbox \
88
+ --filename "Downloaded Paper.pdf" \
89
+ --case-id CS124 \
90
+ --rename "Author_Year_Short_Title.pdf"
91
+ ```
92
+
93
+ ## Upload the Current Code to Hugging Face
94
+
95
+ Preferred if your Hugging Face remote exists:
96
+
97
+ ```bash
98
+ git push uwrfr-chatbot HEAD:main
99
+ ```
100
+
101
+ If the remote does not exist:
102
+
103
+ ```bash
104
+ git remote add uwrfr-chatbot https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot
105
+ git push uwrfr-chatbot HEAD:main
106
+ ```
107
+
108
+ Alternative using the Hugging Face CLI:
109
+
110
+ ```bash
111
+ hf upload UWRFR/UWaterlooRFRChatBot . \
112
+ --type space \
113
+ --exclude ".git/*" \
114
+ --exclude ".github/*" \
115
+ --exclude ".venv/*" \
116
+ --exclude "venv/*" \
117
+ --exclude "__pycache__/*" \
118
+ --exclude "data/raw/*" \
119
+ --exclude "*.pdf" \
120
+ --commit-message "Update website"
121
+ ```
122
+
123
+ ## Push to Shared GitHub
124
+
125
+ ```bash
126
+ git push shared-github HEAD:main
127
+ ```
128
+
129
+ If the remote does not exist:
130
+
131
+ ```bash
132
+ git remote add shared-github https://github.com/rfrraguwaterloo-cmyk/UWaterlooManagedRetreatChatbot.git
133
+ git push shared-github HEAD:main
134
+ ```
135
+
136
+ ## Check Hugging Face Space Status
137
+
138
+ ```bash
139
+ hf spaces info UWRFR/UWaterlooRFRChatBot
140
+ ```
141
+
142
+ Look for:
143
+
144
+ - `sha` - deployed commit.
145
+ - `runtime.stage` - whether it is running, building, or failed.
146
+ - `host` - public app URL.
147
+
148
+ ## See Recent Git History
149
+
150
+ ```bash
151
+ git log --oneline -10
152
+ ```
153
+
154
+ ## See What Changed
155
+
156
+ ```bash
157
+ git status
158
+ git diff --stat
159
+ ```
160
+
161
+ ## Undo an Uncommitted Edit to One File
162
+
163
+ Only do this if you are sure you do not want your local changes to that file:
164
+
165
+ ```bash
166
+ git restore path/to/file.py
167
+ ```
168
+
169
+ Do not use broad destructive commands like `git reset --hard` unless the team
170
+ explicitly agrees.
171
+
docs/05_debugging_common_problems.md ADDED
@@ -0,0 +1,221 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # 05 - Debugging Common Problems
2
+
3
+ This guide lists common failures and what to check first.
4
+
5
+ ## The Website Does Not Load
6
+
7
+ Check Hugging Face logs:
8
+
9
+ ```text
10
+ Hugging Face Space -> Logs
11
+ ```
12
+
13
+ Common causes:
14
+
15
+ - Build failed.
16
+ - Missing dependency in `requirements.txt`.
17
+ - Python syntax error.
18
+ - Missing required file.
19
+ - Space is still rebuilding.
20
+
21
+ CLI check:
22
+
23
+ ```bash
24
+ hf spaces info UWRFR/UWaterlooRFRChatBot
25
+ ```
26
+
27
+ If the Space is building, wait and refresh.
28
+
29
+ ## Ask Tab Fails with an LLM Error
30
+
31
+ Common messages:
32
+
33
+ ```text
34
+ credit balance is too low
35
+ ANTHROPIC_API_KEY is not set
36
+ OPENAI_API_KEY is not set
37
+ APIConnectionError
38
+ invalid_api_key
39
+ ```
40
+
41
+ Check:
42
+
43
+ - The correct key exists in Hugging Face secrets.
44
+ - The key has credits.
45
+ - The app is configured for the provider you expect.
46
+ - The Space was restarted after changing secrets.
47
+
48
+ If Anthropic credits are unavailable, switch the app to OpenAI only if
49
+ `OPENAI_API_KEY` is available and configured.
50
+
51
+ ## New Case Study Request Upload Fails
52
+
53
+ The request tab uploads to Google Drive using:
54
+
55
+ ```text
56
+ GOOGLE_SERVICE_ACCOUNT_JSON
57
+ ```
58
+
59
+ Check:
60
+
61
+ - The secret exists in Hugging Face.
62
+ - The secret value is the full service account JSON.
63
+ - The intake folder is shared with the service account email.
64
+ - The intake folder ID is correct:
65
+
66
+ ```text
67
+ 10n_-uOCT2GXu_G3r1qi5qJr8hHoy-E_y
68
+ ```
69
+
70
+ If using a different folder, set:
71
+
72
+ ```text
73
+ RFR_REQUESTS_DRIVE_FOLDER_ID
74
+ ```
75
+
76
+ ## New Case Does Not Appear in Case Studies
77
+
78
+ Usually one of these files was not rebuilt or committed:
79
+
80
+ ```text
81
+ data/extracted/CSxx.json
82
+ data/extracted/case_summaries.json
83
+ data/extracted/precomputed_embeddings.json
84
+ ```
85
+
86
+ Run:
87
+
88
+ ```bash
89
+ python3 ingest/ingest_pipeline_outputs.py --case-id CSxx
90
+ python3 ingest/create_summary_chunks.py
91
+ python3 ingest/embed_and_index.py
92
+ ```
93
+
94
+ Then commit and push the changed files.
95
+
96
+ ## Case Appears but Sources Are Missing
97
+
98
+ The Case Studies tab reads source links from each case's `source_links` field in
99
+ `data/extracted/CSxx.json`.
100
+
101
+ Source links come from:
102
+
103
+ - `sources.txt`
104
+ - `sources.json`
105
+ - `dois.txt`
106
+ - `ingest/source_link_overrides.json`
107
+
108
+ Fix:
109
+
110
+ 1. Add APA-style links to `ingest/source_link_overrides.json`.
111
+ 2. Re-run:
112
+
113
+ ```bash
114
+ python3 ingest/ingest_pipeline_outputs.py --case-id CSxx --force
115
+ python3 ingest/create_summary_chunks.py
116
+ python3 ingest/embed_and_index.py
117
+ ```
118
+
119
+ 3. Commit and push.
120
+
121
+ ## Case Summary Looks Wrong
122
+
123
+ Check the canonical Ver2 memo:
124
+
125
+ ```text
126
+ data/raw/CSxx/pipeline_output/CSxx_Ver2.md
127
+ ```
128
+
129
+ If the wrong paper was processed, replace the source files in:
130
+
131
+ ```text
132
+ data/raw/CSxx/
133
+ ```
134
+
135
+ Then rerun:
136
+
137
+ ```bash
138
+ python3 -m case_study_pipeline.run_case_study \
139
+ --case-folder data/raw/CSxx \
140
+ --case-id CSxx \
141
+ --llm-provider claude \
142
+ --max-chars-per-source 150000 \
143
+ --force
144
+ ```
145
+
146
+ Then select, ingest, summarize, and embed:
147
+
148
+ ```bash
149
+ python3 -m case_study_pipeline.select_best_run --case-id CSxx
150
+ python3 ingest/ingest_pipeline_outputs.py --case-id CSxx --force
151
+ python3 ingest/create_summary_chunks.py
152
+ python3 ingest/embed_and_index.py
153
+ ```
154
+
155
+ ## Hugging Face Shows Old Content
156
+
157
+ Check the deployed commit:
158
+
159
+ ```bash
160
+ hf spaces info UWRFR/UWaterlooRFRChatBot
161
+ ```
162
+
163
+ Compare the `sha` with:
164
+
165
+ ```bash
166
+ git log --oneline -5
167
+ ```
168
+
169
+ If the Hugging Face SHA is old, push again:
170
+
171
+ ```bash
172
+ git push uwrfr-chatbot HEAD:main
173
+ ```
174
+
175
+ ## Git Push Fails
176
+
177
+ Check remotes:
178
+
179
+ ```bash
180
+ git remote -v
181
+ ```
182
+
183
+ Expected remotes:
184
+
185
+ ```text
186
+ shared-github https://github.com/rfrraguwaterloo-cmyk/UWaterlooManagedRetreatChatbot.git
187
+ uwrfr-chatbot https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot
188
+ ```
189
+
190
+ If authentication fails:
191
+
192
+ - For GitHub, sign in with GitHub credentials or a GitHub token.
193
+ - For Hugging Face, run:
194
+
195
+ ```bash
196
+ hf auth login --add-to-git-credential
197
+ ```
198
+
199
+ Use a Hugging Face token with write access to the Space.
200
+
201
+ ## Drive Pipeline Output Upload Fails
202
+
203
+ The extraction pipeline can still run locally even if Drive output uploads fail.
204
+
205
+ Common warning:
206
+
207
+ ```text
208
+ [Drive] Warning: could not initialize uploader
209
+ Drive uploads will be skipped for this run.
210
+ ```
211
+
212
+ This means the local files were created, but the PDF outputs were not uploaded
213
+ to Drive.
214
+
215
+ Check:
216
+
217
+ - `GOOGLE_CREDENTIALS_FILE` points to an OAuth client credential file.
218
+ - Your browser completed Google login.
219
+ - The target case folder exists in `case_study_pipeline/drive_folders.json`.
220
+ - The Google account has permission to write to the Drive folder.
221
+
docs/06_deploying_updates.md ADDED
@@ -0,0 +1,153 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # 06 - Deploying Updates
2
+
3
+ This guide explains how to get local changes onto GitHub and Hugging Face.
4
+
5
+ ## Recommended Workflow
6
+
7
+ Use GitHub as the shared source of truth and Hugging Face as the live app host.
8
+
9
+ The usual flow is:
10
+
11
+ ```text
12
+ local computer -> GitHub -> Hugging Face
13
+ ```
14
+
15
+ ## Before Deploying
16
+
17
+ Run:
18
+
19
+ ```bash
20
+ git status
21
+ git diff --stat
22
+ python3 -m py_compile app/app.py
23
+ ```
24
+
25
+ If you changed case-study data, also run:
26
+
27
+ ```bash
28
+ python3 ingest/create_summary_chunks.py
29
+ python3 ingest/embed_and_index.py
30
+ ```
31
+
32
+ If you changed the app, test locally:
33
+
34
+ ```bash
35
+ streamlit run app/app.py
36
+ ```
37
+
38
+ ## Commit the Right Files
39
+
40
+ For website text/code changes:
41
+
42
+ ```bash
43
+ git add app/app.py
44
+ git commit -m "Update website"
45
+ ```
46
+
47
+ For a new case study:
48
+
49
+ ```bash
50
+ git add data/extracted/CS124.json
51
+ git add data/extracted/case_summaries.json
52
+ git add data/extracted/precomputed_embeddings.json
53
+ git add data/raw/CS124/case_meta.json
54
+ git add case_study_pipeline/drive_folders.json
55
+ git add ingest/source_link_overrides.json
56
+ git commit -m "Add CS124 case study"
57
+ ```
58
+
59
+ Do not add raw PDFs unless the team explicitly decides to store PDFs in GitHub.
60
+
61
+ ## Push to GitHub
62
+
63
+ ```bash
64
+ git push shared-github HEAD:main
65
+ ```
66
+
67
+ If working on a branch:
68
+
69
+ ```bash
70
+ git push origin your-branch-name
71
+ ```
72
+
73
+ Then open a pull request into `main`.
74
+
75
+ ## Push to Hugging Face
76
+
77
+ Manual push:
78
+
79
+ ```bash
80
+ git push uwrfr-chatbot HEAD:main
81
+ ```
82
+
83
+ Check status:
84
+
85
+ ```bash
86
+ hf spaces info UWRFR/UWaterlooRFRChatBot
87
+ ```
88
+
89
+ The Hugging Face Space may take a minute to rebuild.
90
+
91
+ ## GitHub Action Deployment
92
+
93
+ The repo has a GitHub Action:
94
+
95
+ ```text
96
+ .github/workflows/deploy-hf-space.yml
97
+ ```
98
+
99
+ It can upload the repo to a Hugging Face Space.
100
+
101
+ For it to work, GitHub needs:
102
+
103
+ Secret:
104
+
105
+ ```text
106
+ HF_TOKEN
107
+ ```
108
+
109
+ Variable or workflow input:
110
+
111
+ ```text
112
+ HF_SPACE_ID
113
+ ```
114
+
115
+ For this project, the target Space is:
116
+
117
+ ```text
118
+ UWRFR/UWaterlooRFRChatBot
119
+ ```
120
+
121
+ ## Files Excluded from Hugging Face Upload
122
+
123
+ The deployment excludes files such as:
124
+
125
+ ```text
126
+ .git/
127
+ .github/
128
+ venv/
129
+ __pycache__/
130
+ data/raw/
131
+ *.pdf
132
+ ```
133
+
134
+ That means raw source PDFs usually stay in Google Drive, while processed JSON
135
+ and embeddings are deployed to the website.
136
+
137
+ ## After Deploying
138
+
139
+ Open:
140
+
141
+ ```text
142
+ https://uwrfr-uwaterloorfrchatbot.hf.space
143
+ ```
144
+
145
+ Check:
146
+
147
+ - The website loads.
148
+ - The navigation works.
149
+ - New case studies appear.
150
+ - Source links appear.
151
+ - A test question works.
152
+ - Previous Responses still work.
153
+