Spaces:
Sleeping
Sleeping
|
Download docs/COLLABORATIVE_WORKFLOW_GUIDE.md from jrrribeiro/BirdNET-Validator-App: direct link, hf CLI and curl.
- Browser
- Download file 12.2 kB
-
https://huggingface.co/spaces/jrrribeiro/BirdNET-Validator-App/resolve/main/docs/COLLABORATIVE_WORKFLOW_GUIDE.md
- Command line
-
hf download hf://spaces/jrrribeiro/BirdNET-Validator-App/docs/COLLABORATIVE_WORKFLOW_GUIDE.md
-
curl -L -o COLLABORATIVE_WORKFLOW_GUIDE.md https://huggingface.co/spaces/jrrribeiro/BirdNET-Validator-App/resolve/main/docs/COLLABORATIVE_WORKFLOW_GUIDE.md
12.2 kB
| # End-to-End Collaborative Workflow Guide | |
| ## Overview | |
| The BirdNET Validator App supports two project visibility modes: | |
| - **Collaborative**: Multiple users can contribute validations. Any admin can invite collaborators via email. | |
| - **Private**: Single owner only. No invites allowed. Owner retains exclusive access. | |
| ## Feature Summary | |
| ### β What's Implemented | |
| | Feature | Status | Details | | |
| |---------|--------|---------| | |
| | **Collaborative Projects** | β | Multiple users, email invites, per-user tokens | | |
| | **Private Projects** | β | Owner-only, no invites, strict ACL enforcement | | |
| | **Email Notifications** | β | SMTP + NoOp fallback, transactional delivery | | |
| | **Per-User Tokens** | β | Each user's HF token stored separately (secure) | | |
| | **Session Revocation** | β | Immediate when user removed from project | | |
| | **Durable State** | β | `/data` persistence, atomic writes, crash-safe | | |
| | **Invite TTL** | β | Expiring invites (72h default, configurable) | | |
| | **ACL Enforcement** | β | Private vs Collaborative, admin-only operations | | |
| --- | |
| ## Workflow Example: Creating & Sharing a Collaborative Project | |
| ### Phase 1: Admin Creates Project | |
| ```bash | |
| # Admin logs in (first user = automatic admin) | |
| # β Session created with admin role | |
| # Admin creates project via CLI: | |
| python -m src.cli.project_cli create-project \ | |
| --projects-file /data/bootstrap/projects.json \ | |
| --slug "parrots-2026" \ | |
| --name "Parrots Dataset 2026" \ | |
| --dataset-repo-id "birdnet/parrots-2026" \ | |
| --visibility "collaborative" | |
| # OK: project 'parrots-2026' created | |
| ``` | |
| **What happened:** | |
| - Project created in `/data/bootstrap/projects.json` | |
| - No `owner_username` (collaborative mode) | |
| - Admin automatically has validator role | |
| ### Phase 2: Admin Invites Collaborator | |
| Using the Admin Panel UI, admin clicks "Invite User": | |
| ``` | |
| Actor: alice_admin (project admin) | |
| Invitee Username: bob_validator | |
| Invitee Email: bob@example.com | |
| Role: validator | |
| ``` | |
| **What happens:** | |
| 1. Authorization check: Alice is admin for this project β | |
| 2. Project visibility check: Collaborative (invites allowed) β | |
| 3. Invite created with 72h TTL | |
| 4. Email sent via SMTP: | |
| ``` | |
| Subject: You've been invited to validate "Parrots Dataset 2026" | |
| Hello Bob, | |
| Alice has invited you to join the "Parrots Dataset 2026" project as a validator. | |
| Project: parrots-2026 | |
| Role: validator | |
| Expires: [72 hours from now] | |
| To accept this invitation, log in with your Hugging Face token at: | |
| https://birdnet-validator.example.com/login | |
| After logging in, your invite will appear. Click "Accept" to join. | |
| ``` | |
| 5. Invite stored in `/data/bootstrap/invites.json`: | |
| ```json | |
| { | |
| "bob_validator": { | |
| "parrots-2026": { | |
| "role": "validator", | |
| "invited_by": "alice_admin", | |
| "created_at": "2026-04-01T12:00:00Z", | |
| "expires_at": "2026-04-04T12:00:00Z" | |
| } | |
| } | |
| } | |
| ``` | |
| ### Phase 3: Collaborator Receives & Accepts Invite | |
| **Bob receives email, opens Space app, logs in:** | |
| 1. Bob enters his HF token: `hf_user_***` | |
| 2. AuthService resolves username from HF API (`whoami()`) | |
| 3. Bob's session created | |
| 4. Bob's token stored separately: `_hf_tokens_by_username["bob_validator"] = "hf_user_***"` | |
| 5. **Pending invites auto-appear** in project selector | |
| 6. Bob clicks "Accept Invite" | |
| **What happens:** | |
| - Invite moved from pending β accepted | |
| - Bob added to `user_access.json`: | |
| ```json | |
| { | |
| "bob_validator": { | |
| "parrots-2026": "validator" | |
| } | |
| } | |
| ``` | |
| - Invite removed from `invites.json` | |
| ### Phase 4: Both Users Have Access & Own Tokens | |
| **Alice (admin):** | |
| - Token: `hf_alice_***` (stored in `_hf_tokens_by_username["alice_admin"]`) | |
| - Role: admin for "parrots-2026" | |
| - Can: create, assign, remove, validate, manage project | |
| **Bob (validator):** | |
| - Token: `hf_bob_***` (stored in `_hf_tokens_by_username["bob_validator"]`) | |
| - Role: validator for "parrots-2026" | |
| - Can: validate detections, see project data | |
| **Key Security Feature:** | |
| - Tokens are **NOT** stored in Session objects | |
| - Token lookup happens only when needed (validation flow) | |
| - Each user brings own token β no token sharing | |
| - If Alice's token revoked on HF, only Alice affected | |
| --- | |
| ## Private Project Workflow | |
| ### Create Private Project | |
| ```bash | |
| python -m src.cli.project_cli create-project \ | |
| --projects-file /data/bootstrap/projects.json \ | |
| --user-access-file /data/bootstrap/user_access.json \ | |
| --slug "proprietary-2026" \ | |
| --name "Proprietary Dataset" \ | |
| --dataset-repo-id "owner/proprietary-2026" \ | |
| --visibility "private" \ | |
| --owner "carol_owner" | |
| ``` | |
| **Result:** | |
| - `owner_username = "carol_owner"` is set (required) | |
| - Carol automatically gets admin role | |
| - Invites are **rejected** if attempted | |
| ### Try to Invite (Fails) | |
| If Carol or another admin tries to invite a collaborator: | |
| ``` | |
| Actor: carol_owner | |
| Invitee: dave_validator | |
| β ERROR: "Private projects do not accept collaborators" | |
| ``` | |
| **No invite created.** Private projects are owner-only. | |
| --- | |
| ## Configuration for HF Spaces Deployment | |
| ### Set Environment Variables | |
| On your [HF Space Settings](https://huggingface.co/spaces/YOUR_ORG/BirdNET-Validator-App/settings): | |
| ```bash | |
| # Basic config | |
| BIRDNET_ENABLE_DEMO_BOOTSTRAP=false | |
| BIRDNET_BOOTSTRAP_DIR=/data/bootstrap | |
| BIRDNET_VALIDATIONS_DIR=/data/validations | |
| # Invite settings | |
| BIRDNET_INVITE_EMAIL_ENABLED=true | |
| BIRDNET_INVITE_EMAIL_SENDER=noreply@birdnet-validator.example.com | |
| BIRDNET_INVITE_EMAIL_LOGIN_URL=https://huggingface.co/spaces/YOUR_ORG/BirdNET-Validator-App | |
| BIRDNET_INVITE_TTL_HOURS=72 | |
| # SMTP settings (transactional email service) | |
| BIRDNET_SMTP_HOST=smtp.gmail.com # or your email provider | |
| BIRDNET_SMTP_PORT=587 | |
| BIRDNET_SMTP_USERNAME=your-email@gmail.com | |
| BIRDNET_SMTP_PASSWORD=your-app-password # Use app-specific password for 2FA | |
| BIRDNET_SMTP_USE_TLS=true | |
| ``` | |
| ### Test SMTP Configuration Locally | |
| ```python | |
| from src.config.runtime_config import RuntimeConfig | |
| from src.services.invite_email_notifier import SmtpInviteEmailNotifier, InviteEmailPayload | |
| from datetime import datetime, timedelta, UTC | |
| config = RuntimeConfig.from_env() | |
| if config.invite_email_enabled and config.smtp_host: | |
| notifier = SmtpInviteEmailNotifier( | |
| host=config.smtp_host, | |
| port=config.smtp_port, | |
| username=config.smtp_username, | |
| password=config.smtp_password, | |
| use_tls=config.smtp_use_tls, | |
| ) | |
| test_payload = InviteEmailPayload( | |
| invitee_username="test_user", | |
| invitee_email="test@example.com", | |
| project_slug="test-project", | |
| role="validator", | |
| invited_by="admin_user", | |
| expires_at=datetime.now(UTC) + timedelta(hours=72), | |
| login_url=config.invite_email_login_url, | |
| ) | |
| ok, message = notifier.send(test_payload) | |
| print(f"Email send: {ok=}, {message=}") | |
| else: | |
| print("Email invites disabled or SMTP not configured") | |
| ``` | |
| --- | |
| ## Data Durability & Persistence | |
| ### Bootstrap Files Location | |
| On HF Spaces, all bootstrap and validation data persists in `/data/`: | |
| ``` | |
| /data/ | |
| βββ bootstrap/ | |
| β βββ projects.json # All projects (collaborative + private) | |
| β βββ user_access.json # User β Project β Role mapping | |
| β βββ invites.json # Pending invites with TTL | |
| βββ validations/ | |
| βββ project-1/ | |
| β βββ 2026-04-01.jsonl # Append-only validation events | |
| β βββ 2026-04-02.jsonl | |
| β βββ ... | |
| βββ project-2/ | |
| βββ ... | |
| ``` | |
| ### Atomic Writes (Crash-Safe) | |
| All JSON writes use atomic pattern: | |
| ```python | |
| temp_file = target.with_suffix(target.suffix + ".tmp") | |
| temp_file.write_text(json.dumps(data)) | |
| temp_file.replace(target) # Atomic on modern filesystems | |
| ``` | |
| **Benefit:** If Space crashes during write, file is not corrupted. | |
| ### Data Loss Prevention | |
| Since `/data` is persistent on HF Spaces: | |
| - Projects, ACL, invites **survive** Space restarts β | |
| - Unfinished validations are **not** lost β | |
| - User data is **durable** across redeploys β | |
| --- | |
| ## Testing the Workflow Locally | |
| ### 1. Create Users & Projects | |
| ```bash | |
| # Set environment for local testing | |
| export BIRDNET_ENABLE_DEMO_BOOTSTRAP=false | |
| export BIRDNET_PROJECTS_FILE=/tmp/projects.json | |
| export BIRDNET_USER_ACCESS_FILE=/tmp/user_access.json | |
| export BIRDNET_INVITES_FILE=/tmp/invites.json | |
| export BIRDNET_INVITE_EMAIL_ENABLED=true | |
| export BIRDNET_INVITE_EMAIL_SENDER=test@example.com | |
| export BIRDNET_INVITE_EMAIL_LOGIN_URL=http://localhost:7860 | |
| # Create initial files | |
| echo '[]' > /tmp/projects.json | |
| echo '{}' > /tmp/user_access.json | |
| echo '{}' > /tmp/invites.json | |
| # Create project | |
| python -m src.cli.project_cli create-project \ | |
| --projects-file /tmp/projects.json \ | |
| --user-access-file /tmp/user_access.json \ | |
| --slug test-project \ | |
| --name "Test Project" \ | |
| --dataset-repo-id "test/project" \ | |
| --visibility collaborative \ | |
| --owner alice | |
| ``` | |
| ### 2. Start App & Test UI | |
| ```bash | |
| # Terminal 1: Start server | |
| python app.py # Runs on http://localhost:7860 | |
| # Terminal 2: Test workflow in UI | |
| # - Log in with first user (becomes admin) | |
| # - Create collaborative project | |
| # - Invite second user | |
| # - Log out, log in as second user | |
| # - Accept invite from project selector | |
| # - Both users see project in dropdown | |
| ``` | |
| ### 3. Verify Data Files | |
| ```bash | |
| # Check projects file | |
| cat /tmp/projects.json | python -m json.tool | |
| # Check user access | |
| cat /tmp/user_access.json | python -m json.tool | |
| # Check invites | |
| cat /tmp/invites.json | python -m json.tool | |
| ``` | |
| --- | |
| ## Troubleshooting | |
| ### "Email not sent" but project created | |
| **Cause:** SMTP not configured or Email invites disabled | |
| **Fix:** | |
| 1. Check `BIRDNET_INVITE_EMAIL_ENABLED=true` | |
| 2. Check SMTP credentials are correct | |
| 3. For Gmail: use [App Password](https://support.google.com/accounts/answer/185833), not account password | |
| 4. Check firewall allows outbound SMTP (port 587) | |
| **Fallback:** System will create invite without email. User must be told invite URL manually. | |
| ### User can't accept invite | |
| **Cause:** Session TTL expired, or invite expired (>72h) | |
| **Fix:** | |
| 1. Re-login (creates fresh session) | |
| 2. Resend invite | |
| 3. Adjust `BIRDNET_INVITE_TTL_HOURS` if needed | |
| ### Private project invite fails | |
| **Expected behavior:** Private projects reject invites | |
| **Workaround:** Assign users directly in `user_access.json` if bootstrap: | |
| ```json | |
| { | |
| "username": { | |
| "private-project": "validator" | |
| } | |
| } | |
| ``` | |
| ### Tokens not being used in validation | |
| **Cause:** Check project token precedence in `app_factory.py`: | |
| ```python | |
| # Preference order: | |
| # 1. User's HF token (from session login) | |
| # 2. Project token (for private project owners only) | |
| # 3. None (validation fails) | |
| ``` | |
| Set users' HF token via login, or set project token via CLI/UI. | |
| --- | |
| ## Next Steps | |
| ### For Production Deployment | |
| 1. β **Configure SMTP** on HF Space with real email service | |
| 2. β **Test end-to-end**: Create project β Invite user β Accept β Validate | |
| 3. β **Verify durability**: Create data, restart Space, confirm persistence | |
| 4. π **Monitor invites**: Set up alerts if emails bounce | |
| 5. π **Audit trail**: Review `/data/bootstrap/invites.json` periodically | |
| ### For Future Enhancements | |
| - [ ] Invite expiration auto-cleanup background task | |
| - [ ] Invite revocation by admin | |
| - [ ] User deactivation cascade (revoke all invites, remove all roles) | |
| - [ ] Email template customization | |
| - [ ] Multi-language invite emails | |
| - [ ] Webhook notifications (Discord, Slack) | |
| --- | |
| ## Test Results Summary | |
| | Test Suite | Pass Rate | Key Tests | | |
| |-----------|-----------|-----------| | |
| | **Unit: AuthService** | 37/37 β | Login, session TTL, ACL, token storage, invite lifecycle | | |
| | **Unit: AdminPanel** | 5/5 β | Private isolation, email dispatch, authorization | | |
| | **Unit: AppFactory** | 45/45 β | Bootstrap loading, config handling, audio helpers | | |
| | **Unit: CLI** | 10/10 β | Create-project, init-dataset, build-index, verify | | |
| | **Integration** | 2/2 β | Validation flow, detection repository | | |
| | **Total** | **139/139 β ** | All workflows tested and passing | | |
| --- | |
| **Last Updated:** 2026-04-01 | |
| **Deployment Status:** Ready for HF Spaces production | |
| **Latest Commit:** 5e463e3 (feat: security hardening, durable state, email invites) | |