File size: 8,434 Bytes
78c1c50
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2f024ae
78c1c50
 
2f024ae
 
 
78c1c50
 
 
 
 
 
 
 
 
 
 
 
 
 
2f024ae
 
 
 
 
 
 
 
 
 
 
 
 
 
78c1c50
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2f024ae
 
 
 
 
78c1c50
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
# Website Maintenance and New Case Study Guide

This guide explains how to maintain the RFR Knowledge Platform website and how
to add new managed retreat case studies to the searchable knowledge base.

## Repositories and Live Website

- Shared GitHub repo:
  `https://github.com/rfrraguwaterloo-cmyk/UWaterlooManagedRetreatChatbot`
- Shared Hugging Face Space:
  `https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot`
- Live app:
  `https://uwrfr-uwaterloorfrchatbot.hf.space`

GitHub is the easiest shared place for code review. Hugging Face is where the
Streamlit app runs.

## Editing the Website From Another Computer

1. Install Git and Python 3.11 or newer.
2. Clone the shared GitHub repo:

```bash
git clone https://github.com/rfrraguwaterloo-cmyk/UWaterlooManagedRetreatChatbot.git
cd UWaterlooManagedRetreatChatbot
```

3. Create a working branch:

```bash
git checkout -b update-website-text
```

4. Install dependencies:

```bash
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```

5. Run the website locally:

```bash
streamlit run app/app.py
```

6. Edit the code. Most website text and tabs are in:

```text
app/app.py
```

7. Check the app still compiles:

```bash
python3 -m py_compile app/app.py
```

8. Commit and push:

```bash
git status
git add app/app.py
git commit -m "Update website text"
git push origin update-website-text
```

9. Open a Pull Request on GitHub into `main`.

After the change is merged into `main`, the GitHub Action can deploy to Hugging
Face if the shared repo has `HF_TOKEN` and `HF_SPACE_ID` configured. The team can
also push manually to Hugging Face from a machine that is logged in with `hf`.

## Manual Hugging Face Push

From a local repo with the Hugging Face CLI logged in:

```bash
git remote add uwrfr-chatbot https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot
git push uwrfr-chatbot HEAD:main
```

If the remote already exists, update it instead:

```bash
git remote set-url uwrfr-chatbot https://huggingface.co/spaces/UWRFR/UWaterlooRFRChatBot
git push uwrfr-chatbot HEAD:main
```

Or upload the current folder with the CLI:

```bash
hf upload UWRFR/UWaterlooRFRChatBot . \
  --type space \
  --exclude ".git/*" \
  --exclude ".github/*" \
  --exclude ".venv/*" \
  --exclude "venv/*" \
  --exclude "__pycache__/*" \
  --exclude "data/raw/*" \
  --exclude "*.pdf" \
  --commit-message "Update website"
```

Check Space status:

```bash
hf spaces info UWRFR/UWaterlooRFRChatBot
```

## Website Secrets

Set these in Hugging Face Space settings under **Variables and secrets**.

Required for the chatbot:

```text
ANTHROPIC_API_KEY
```

Optional if using OpenAI mode:

```text
OPENAI_API_KEY
OPENAI_MODEL
```

Required for Google Drive case-study request uploads:

```text
GOOGLE_OAUTH_CLIENT_ID
GOOGLE_OAUTH_CLIENT_SECRET
GOOGLE_OAUTH_REFRESH_TOKEN
```

Optional Drive override for new case study requests:

```text
RFR_REQUESTS_DRIVE_FOLDER_ID
```

By default, new case study requests are uploaded to this Drive folder:

```text
10n_-uOCT2GXu_G3r1qi5qJr8hHoy-E_y
```

The OAuth token should be generated by signing in as the Google account that
should own request uploads, usually `rfr.rag.uwaterloo@gmail.com`.

Generate the Hugging Face OAuth secrets locally:

```bash
python3 -m case_study_pipeline.create_drive_oauth_secrets \
  --credentials-file /path/to/oauth-client.json \
  --space-id UWRFR/UWaterlooRFRChatBot
```

The older fallback `GOOGLE_SERVICE_ACCOUNT_JSON` service-account setup can fail
in regular My Drive folders because service accounts do not have storage quota.
Use OAuth or a Google Shared Drive for request uploads.

## How New Case Study Requests Work

Users can submit candidate papers through the **New Case Study Requests** tab.
The app creates a new Google Drive folder for each request. Each folder contains:

- `request.json`
- uploaded PDFs or supporting documents

The request tab only collects files and notes. It does not run the extraction
pipeline and does not call an LLM.

## Adding a New Case Study to the Knowledge Base

Use this workflow after the team accepts a submitted request or identifies a new
case study.

### 1. Choose the Next Case ID

Check the highest indexed case:

```bash
ls data/extracted/CS*.json | sort -V | tail
```

If the latest indexed case is `CS54`, the next one is usually `CS55`.

### 2. Create the Raw Case Folder

Create a folder under `data/raw/`:

```bash
mkdir -p data/raw/CS55-ShortName_COUNTRY
```

Put source PDFs, `.md`, or `.txt` files in that folder.

Add or confirm `case_meta.json` in that folder. Example:

```json
{
  "case_id": "CS55",
  "name": "Short case study name",
  "location": "City or region",
  "country": "Country"
}
```

### 3. Run the One-Provider Extraction Pipeline

Use one provider for all four extraction and audit steps. Claude is the current
simplest option:

```bash
python3 -m case_study_pipeline.run_case_study \
  --case-folder data/raw/CS55-ShortName_COUNTRY \
  --case-id CS55 \
  --llm-provider claude \
  --max-chars-per-source 150000 \
  --force
```

This writes outputs to:

```text
data/raw/CS55-ShortName_COUNTRY/pipeline_output/
```

Expected output files:

- `CS55_Ver1.md`
- `CS55_Check1_report.md`
- `CS55_Ver2.md`
- `CS55_Check2_report.md`
- matching PDFs

### 4. Select the Best Run

```bash
python3 -m case_study_pipeline.select_best_run --case-id CS55
```

This creates or updates:

```text
data/raw/CS55-ShortName_COUNTRY/pipeline_output/selected.json
```

### 5. Ingest the Case Study

```bash
python3 ingest/ingest_pipeline_outputs.py --case-id CS55
```

This creates:

```text
data/extracted/CS55.json
```

### 6. Rebuild Summaries and Embeddings

Always run both commands after adding or changing extracted case data:

```bash
python3 ingest/create_summary_chunks.py
python3 ingest/embed_and_index.py
```

These update:

```text
data/extracted/case_summaries.json
data/extracted/precomputed_embeddings.json
```

The live website will not know about the new case until these files are updated
and deployed.

### 7. Test Locally

```bash
streamlit run app/app.py
```

Check:

- the new case appears in **Case Studies**
- the app can answer a direct question such as `Summarize CS55`
- source links appear if `source_links` metadata is available

### 8. Commit and Push

Commit only the files needed by the app. Usually this means:

```bash
git add data/extracted/CS55.json
git add data/extracted/case_summaries.json
git add data/extracted/precomputed_embeddings.json
git add data/raw/CS55-ShortName_COUNTRY/case_meta.json
git commit -m "Add CS55 case study"
```

The raw source PDFs are usually not committed. Keep them in Drive or local raw
storage unless the team explicitly decides otherwise.

Push to GitHub:

```bash
git push shared-github HEAD:main
```

Push to Hugging Face:

```bash
git push uwrfr-chatbot HEAD:main
```

## Updating an Existing Case Study

If a case study already exists and you are improving its sources or extraction:

```bash
python3 -m case_study_pipeline.run_case_study \
  --case-folder data/raw/CS55-ShortName_COUNTRY \
  --case-id CS55 \
  --llm-provider claude \
  --max-chars-per-source 150000 \
  --force

python3 -m case_study_pipeline.select_best_run --case-id CS55
python3 ingest/ingest_pipeline_outputs.py --case-id CS55
python3 ingest/create_summary_chunks.py
python3 ingest/embed_and_index.py
```

Then test, commit, and push as above.

## Common Problems

### New case does not show on the website

Re-run:

```bash
python3 ingest/create_summary_chunks.py
python3 ingest/embed_and_index.py
```

Then commit and deploy the updated files in `data/extracted/`.

### Hugging Face rebuilds but old content appears

Check the Space commit SHA:

```bash
hf spaces info UWRFR/UWaterlooRFRChatBot
```

Confirm it matches the commit you pushed.

### Drive request upload fails

Check:

- `GOOGLE_OAUTH_CLIENT_ID`, `GOOGLE_OAUTH_CLIENT_SECRET`, and
  `GOOGLE_OAUTH_REFRESH_TOKEN` exist as Hugging Face secrets
- the Drive folder is owned by or shared with the OAuth Google account
- `RFR_REQUESTS_DRIVE_FOLDER_ID` points to the correct folder
- the Space has restarted after secrets or variables were added

### LLM key fails

Check the relevant Hugging Face secret:

```text
ANTHROPIC_API_KEY
OPENAI_API_KEY
```

The app currently defaults to Claude for answer generation. The extraction
pipeline can use one provider with `--llm-provider claude` or
`--llm-provider openai`.