jrrribeiro commited on
Commit
05ee5e6
Β·
1 Parent(s): 5e463e3

docs: add collaborative workflow operation guide

Browse files
Files changed (1) hide show
  1. docs/COLLABORATIVE_WORKFLOW_GUIDE.md +424 -0
docs/COLLABORATIVE_WORKFLOW_GUIDE.md ADDED
@@ -0,0 +1,424 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # End-to-End Collaborative Workflow Guide
2
+
3
+ ## Overview
4
+
5
+ The BirdNET Validator App supports two project visibility modes:
6
+
7
+ - **Collaborative**: Multiple users can contribute validations. Any admin can invite collaborators via email.
8
+ - **Private**: Single owner only. No invites allowed. Owner retains exclusive access.
9
+
10
+ ## Feature Summary
11
+
12
+ ### βœ… What's Implemented
13
+
14
+ | Feature | Status | Details |
15
+ |---------|--------|---------|
16
+ | **Collaborative Projects** | βœ… | Multiple users, email invites, per-user tokens |
17
+ | **Private Projects** | βœ… | Owner-only, no invites, strict ACL enforcement |
18
+ | **Email Notifications** | βœ… | SMTP + NoOp fallback, transactional delivery |
19
+ | **Per-User Tokens** | βœ… | Each user's HF token stored separately (secure) |
20
+ | **Session Revocation** | βœ… | Immediate when user removed from project |
21
+ | **Durable State** | βœ… | `/data` persistence, atomic writes, crash-safe |
22
+ | **Invite TTL** | βœ… | Expiring invites (72h default, configurable) |
23
+ | **ACL Enforcement** | βœ… | Private vs Collaborative, admin-only operations |
24
+
25
+ ---
26
+
27
+ ## Workflow Example: Creating & Sharing a Collaborative Project
28
+
29
+ ### Phase 1: Admin Creates Project
30
+
31
+ ```bash
32
+ # Admin logs in (first user = automatic admin)
33
+ # β†’ Session created with admin role
34
+
35
+ # Admin creates project via CLI:
36
+ python -m src.cli.project_cli create-project \
37
+ --projects-file /data/bootstrap/projects.json \
38
+ --slug "parrots-2026" \
39
+ --name "Parrots Dataset 2026" \
40
+ --dataset-repo-id "birdnet/parrots-2026" \
41
+ --visibility "collaborative"
42
+ # OK: project 'parrots-2026' created
43
+ ```
44
+
45
+ **What happened:**
46
+ - Project created in `/data/bootstrap/projects.json`
47
+ - No `owner_username` (collaborative mode)
48
+ - Admin automatically has validator role
49
+
50
+ ### Phase 2: Admin Invites Collaborator
51
+
52
+ Using the Admin Panel UI, admin clicks "Invite User":
53
+
54
+ ```
55
+ Actor: alice_admin (project admin)
56
+ Invitee Username: bob_validator
57
+ Invitee Email: bob@example.com
58
+ Role: validator
59
+ ```
60
+
61
+ **What happens:**
62
+ 1. Authorization check: Alice is admin for this project βœ“
63
+ 2. Project visibility check: Collaborative (invites allowed) βœ“
64
+ 3. Invite created with 72h TTL
65
+ 4. Email sent via SMTP:
66
+ ```
67
+ Subject: You've been invited to validate "Parrots Dataset 2026"
68
+
69
+ Hello Bob,
70
+
71
+ Alice has invited you to join the "Parrots Dataset 2026" project as a validator.
72
+
73
+ Project: parrots-2026
74
+ Role: validator
75
+ Expires: [72 hours from now]
76
+
77
+ To accept this invitation, log in with your Hugging Face token at:
78
+ https://birdnet-validator.example.com/login
79
+
80
+ After logging in, your invite will appear. Click "Accept" to join.
81
+ ```
82
+ 5. Invite stored in `/data/bootstrap/invites.json`:
83
+ ```json
84
+ {
85
+ "bob_validator": {
86
+ "parrots-2026": {
87
+ "role": "validator",
88
+ "invited_by": "alice_admin",
89
+ "created_at": "2026-04-01T12:00:00Z",
90
+ "expires_at": "2026-04-04T12:00:00Z"
91
+ }
92
+ }
93
+ }
94
+ ```
95
+
96
+ ### Phase 3: Collaborator Receives & Accepts Invite
97
+
98
+ **Bob receives email, opens Space app, logs in:**
99
+
100
+ 1. Bob enters his HF token: `hf_user_***`
101
+ 2. AuthService resolves username from HF API (`whoami()`)
102
+ 3. Bob's session created
103
+ 4. Bob's token stored separately: `_hf_tokens_by_username["bob_validator"] = "hf_user_***"`
104
+ 5. **Pending invites auto-appear** in project selector
105
+ 6. Bob clicks "Accept Invite"
106
+
107
+ **What happens:**
108
+ - Invite moved from pending β†’ accepted
109
+ - Bob added to `user_access.json`:
110
+ ```json
111
+ {
112
+ "bob_validator": {
113
+ "parrots-2026": "validator"
114
+ }
115
+ }
116
+ ```
117
+ - Invite removed from `invites.json`
118
+
119
+ ### Phase 4: Both Users Have Access & Own Tokens
120
+
121
+ **Alice (admin):**
122
+ - Token: `hf_alice_***` (stored in `_hf_tokens_by_username["alice_admin"]`)
123
+ - Role: admin for "parrots-2026"
124
+ - Can: create, assign, remove, validate, manage project
125
+
126
+ **Bob (validator):**
127
+ - Token: `hf_bob_***` (stored in `_hf_tokens_by_username["bob_validator"]`)
128
+ - Role: validator for "parrots-2026"
129
+ - Can: validate detections, see project data
130
+
131
+ **Key Security Feature:**
132
+ - Tokens are **NOT** stored in Session objects
133
+ - Token lookup happens only when needed (validation flow)
134
+ - Each user brings own token β†’ no token sharing
135
+ - If Alice's token revoked on HF, only Alice affected
136
+
137
+ ---
138
+
139
+ ## Private Project Workflow
140
+
141
+ ### Create Private Project
142
+
143
+ ```bash
144
+ python -m src.cli.project_cli create-project \
145
+ --projects-file /data/bootstrap/projects.json \
146
+ --user-access-file /data/bootstrap/user_access.json \
147
+ --slug "proprietary-2026" \
148
+ --name "Proprietary Dataset" \
149
+ --dataset-repo-id "owner/proprietary-2026" \
150
+ --visibility "private" \
151
+ --owner "carol_owner"
152
+ ```
153
+
154
+ **Result:**
155
+ - `owner_username = "carol_owner"` is set (required)
156
+ - Carol automatically gets admin role
157
+ - Invites are **rejected** if attempted
158
+
159
+ ### Try to Invite (Fails)
160
+
161
+ If Carol or another admin tries to invite a collaborator:
162
+
163
+ ```
164
+ Actor: carol_owner
165
+ Invitee: dave_validator
166
+ β†’ ERROR: "Private projects do not accept collaborators"
167
+ ```
168
+
169
+ **No invite created.** Private projects are owner-only.
170
+
171
+ ---
172
+
173
+ ## Configuration for HF Spaces Deployment
174
+
175
+ ### Set Environment Variables
176
+
177
+ On your [HF Space Settings](https://huggingface.co/spaces/YOUR_ORG/BirdNET-Validator-App/settings):
178
+
179
+ ```bash
180
+ # Basic config
181
+ BIRDNET_ENABLE_DEMO_BOOTSTRAP=false
182
+ BIRDNET_BOOTSTRAP_DIR=/data/bootstrap
183
+ BIRDNET_VALIDATIONS_DIR=/data/validations
184
+
185
+ # Invite settings
186
+ BIRDNET_INVITE_EMAIL_ENABLED=true
187
+ BIRDNET_INVITE_EMAIL_SENDER=noreply@birdnet-validator.example.com
188
+ BIRDNET_INVITE_EMAIL_LOGIN_URL=https://huggingface.co/spaces/YOUR_ORG/BirdNET-Validator-App
189
+ BIRDNET_INVITE_TTL_HOURS=72
190
+
191
+ # SMTP settings (transactional email service)
192
+ BIRDNET_SMTP_HOST=smtp.gmail.com # or your email provider
193
+ BIRDNET_SMTP_PORT=587
194
+ BIRDNET_SMTP_USERNAME=your-email@gmail.com
195
+ BIRDNET_SMTP_PASSWORD=your-app-password # Use app-specific password for 2FA
196
+ BIRDNET_SMTP_USE_TLS=true
197
+ ```
198
+
199
+ ### Test SMTP Configuration Locally
200
+
201
+ ```python
202
+ from src.config.runtime_config import RuntimeConfig
203
+ from src.services.invite_email_notifier import SmtpInviteEmailNotifier, InviteEmailPayload
204
+ from datetime import datetime, timedelta, UTC
205
+
206
+ config = RuntimeConfig.from_env()
207
+
208
+ if config.invite_email_enabled and config.smtp_host:
209
+ notifier = SmtpInviteEmailNotifier(
210
+ host=config.smtp_host,
211
+ port=config.smtp_port,
212
+ username=config.smtp_username,
213
+ password=config.smtp_password,
214
+ use_tls=config.smtp_use_tls,
215
+ )
216
+
217
+ test_payload = InviteEmailPayload(
218
+ invitee_username="test_user",
219
+ invitee_email="test@example.com",
220
+ project_slug="test-project",
221
+ role="validator",
222
+ invited_by="admin_user",
223
+ expires_at=datetime.now(UTC) + timedelta(hours=72),
224
+ login_url=config.invite_email_login_url,
225
+ )
226
+
227
+ ok, message = notifier.send(test_payload)
228
+ print(f"Email send: {ok=}, {message=}")
229
+ else:
230
+ print("Email invites disabled or SMTP not configured")
231
+ ```
232
+
233
+ ---
234
+
235
+ ## Data Durability & Persistence
236
+
237
+ ### Bootstrap Files Location
238
+
239
+ On HF Spaces, all bootstrap and validation data persists in `/data/`:
240
+
241
+ ```
242
+ /data/
243
+ β”œβ”€β”€ bootstrap/
244
+ β”‚ β”œβ”€β”€ projects.json # All projects (collaborative + private)
245
+ β”‚ β”œβ”€β”€ user_access.json # User β†’ Project β†’ Role mapping
246
+ β”‚ └── invites.json # Pending invites with TTL
247
+ └── validations/
248
+ β”œβ”€β”€ project-1/
249
+ β”‚ β”œβ”€β”€ 2026-04-01.jsonl # Append-only validation events
250
+ β”‚ β”œβ”€β”€ 2026-04-02.jsonl
251
+ β”‚ └── ...
252
+ └── project-2/
253
+ └── ...
254
+ ```
255
+
256
+ ### Atomic Writes (Crash-Safe)
257
+
258
+ All JSON writes use atomic pattern:
259
+
260
+ ```python
261
+ temp_file = target.with_suffix(target.suffix + ".tmp")
262
+ temp_file.write_text(json.dumps(data))
263
+ temp_file.replace(target) # Atomic on modern filesystems
264
+ ```
265
+
266
+ **Benefit:** If Space crashes during write, file is not corrupted.
267
+
268
+ ### Data Loss Prevention
269
+
270
+ Since `/data` is persistent on HF Spaces:
271
+ - Projects, ACL, invites **survive** Space restarts βœ“
272
+ - Unfinished validations are **not** lost βœ“
273
+ - User data is **durable** across redeploys βœ“
274
+
275
+ ---
276
+
277
+ ## Testing the Workflow Locally
278
+
279
+ ### 1. Create Users & Projects
280
+
281
+ ```bash
282
+ # Set environment for local testing
283
+ export BIRDNET_ENABLE_DEMO_BOOTSTRAP=false
284
+ export BIRDNET_PROJECTS_FILE=/tmp/projects.json
285
+ export BIRDNET_USER_ACCESS_FILE=/tmp/user_access.json
286
+ export BIRDNET_INVITES_FILE=/tmp/invites.json
287
+ export BIRDNET_INVITE_EMAIL_ENABLED=true
288
+ export BIRDNET_INVITE_EMAIL_SENDER=test@example.com
289
+ export BIRDNET_INVITE_EMAIL_LOGIN_URL=http://localhost:7860
290
+
291
+ # Create initial files
292
+ echo '[]' > /tmp/projects.json
293
+ echo '{}' > /tmp/user_access.json
294
+ echo '{}' > /tmp/invites.json
295
+
296
+ # Create project
297
+ python -m src.cli.project_cli create-project \
298
+ --projects-file /tmp/projects.json \
299
+ --user-access-file /tmp/user_access.json \
300
+ --slug test-project \
301
+ --name "Test Project" \
302
+ --dataset-repo-id "test/project" \
303
+ --visibility collaborative \
304
+ --owner alice
305
+ ```
306
+
307
+ ### 2. Start App & Test UI
308
+
309
+ ```bash
310
+ # Terminal 1: Start server
311
+ python app.py # Runs on http://localhost:7860
312
+
313
+ # Terminal 2: Test workflow in UI
314
+ # - Log in with first user (becomes admin)
315
+ # - Create collaborative project
316
+ # - Invite second user
317
+ # - Log out, log in as second user
318
+ # - Accept invite from project selector
319
+ # - Both users see project in dropdown
320
+ ```
321
+
322
+ ### 3. Verify Data Files
323
+
324
+ ```bash
325
+ # Check projects file
326
+ cat /tmp/projects.json | python -m json.tool
327
+
328
+ # Check user access
329
+ cat /tmp/user_access.json | python -m json.tool
330
+
331
+ # Check invites
332
+ cat /tmp/invites.json | python -m json.tool
333
+ ```
334
+
335
+ ---
336
+
337
+ ## Troubleshooting
338
+
339
+ ### "Email not sent" but project created
340
+
341
+ **Cause:** SMTP not configured or Email invites disabled
342
+
343
+ **Fix:**
344
+ 1. Check `BIRDNET_INVITE_EMAIL_ENABLED=true`
345
+ 2. Check SMTP credentials are correct
346
+ 3. For Gmail: use [App Password](https://support.google.com/accounts/answer/185833), not account password
347
+ 4. Check firewall allows outbound SMTP (port 587)
348
+
349
+ **Fallback:** System will create invite without email. User must be told invite URL manually.
350
+
351
+ ### User can't accept invite
352
+
353
+ **Cause:** Session TTL expired, or invite expired (>72h)
354
+
355
+ **Fix:**
356
+ 1. Re-login (creates fresh session)
357
+ 2. Resend invite
358
+ 3. Adjust `BIRDNET_INVITE_TTL_HOURS` if needed
359
+
360
+ ### Private project invite fails
361
+
362
+ **Expected behavior:** Private projects reject invites
363
+
364
+ **Workaround:** Assign users directly in `user_access.json` if bootstrap:
365
+ ```json
366
+ {
367
+ "username": {
368
+ "private-project": "validator"
369
+ }
370
+ }
371
+ ```
372
+
373
+ ### Tokens not being used in validation
374
+
375
+ **Cause:** Check project token precedence in `app_factory.py`:
376
+
377
+ ```python
378
+ # Preference order:
379
+ # 1. User's HF token (from session login)
380
+ # 2. Project token (for private project owners only)
381
+ # 3. None (validation fails)
382
+ ```
383
+
384
+ Set users' HF token via login, or set project token via CLI/UI.
385
+
386
+ ---
387
+
388
+ ## Next Steps
389
+
390
+ ### For Production Deployment
391
+
392
+ 1. βœ… **Configure SMTP** on HF Space with real email service
393
+ 2. βœ… **Test end-to-end**: Create project β†’ Invite user β†’ Accept β†’ Validate
394
+ 3. βœ… **Verify durability**: Create data, restart Space, confirm persistence
395
+ 4. πŸ“‹ **Monitor invites**: Set up alerts if emails bounce
396
+ 5. πŸ“‹ **Audit trail**: Review `/data/bootstrap/invites.json` periodically
397
+
398
+ ### For Future Enhancements
399
+
400
+ - [ ] Invite expiration auto-cleanup background task
401
+ - [ ] Invite revocation by admin
402
+ - [ ] User deactivation cascade (revoke all invites, remove all roles)
403
+ - [ ] Email template customization
404
+ - [ ] Multi-language invite emails
405
+ - [ ] Webhook notifications (Discord, Slack)
406
+
407
+ ---
408
+
409
+ ## Test Results Summary
410
+
411
+ | Test Suite | Pass Rate | Key Tests |
412
+ |-----------|-----------|-----------|
413
+ | **Unit: AuthService** | 37/37 βœ… | Login, session TTL, ACL, token storage, invite lifecycle |
414
+ | **Unit: AdminPanel** | 5/5 βœ… | Private isolation, email dispatch, authorization |
415
+ | **Unit: AppFactory** | 45/45 βœ… | Bootstrap loading, config handling, audio helpers |
416
+ | **Unit: CLI** | 10/10 βœ… | Create-project, init-dataset, build-index, verify |
417
+ | **Integration** | 2/2 βœ… | Validation flow, detection repository |
418
+ | **Total** | **139/139 βœ…** | All workflows tested and passing |
419
+
420
+ ---
421
+
422
+ **Last Updated:** 2026-04-01
423
+ **Deployment Status:** Ready for HF Spaces production
424
+ **Latest Commit:** 5e463e3 (feat: security hardening, durable state, email invites)