BirdNET-Validator-App / docs /COLLABORATIVE_WORKFLOW_GUIDE.md
jrrribeiro's picture
docs: add collaborative workflow operation guide
05ee5e6
|
Raw History Blame Contribute Delete
12.2 kB
# End-to-End Collaborative Workflow Guide
## Overview
The BirdNET Validator App supports two project visibility modes:
- **Collaborative**: Multiple users can contribute validations. Any admin can invite collaborators via email.
- **Private**: Single owner only. No invites allowed. Owner retains exclusive access.
## Feature Summary
### βœ… What's Implemented
| Feature | Status | Details |
|---------|--------|---------|
| **Collaborative Projects** | βœ… | Multiple users, email invites, per-user tokens |
| **Private Projects** | βœ… | Owner-only, no invites, strict ACL enforcement |
| **Email Notifications** | βœ… | SMTP + NoOp fallback, transactional delivery |
| **Per-User Tokens** | βœ… | Each user's HF token stored separately (secure) |
| **Session Revocation** | βœ… | Immediate when user removed from project |
| **Durable State** | βœ… | `/data` persistence, atomic writes, crash-safe |
| **Invite TTL** | βœ… | Expiring invites (72h default, configurable) |
| **ACL Enforcement** | βœ… | Private vs Collaborative, admin-only operations |
---
## Workflow Example: Creating & Sharing a Collaborative Project
### Phase 1: Admin Creates Project
```bash
# Admin logs in (first user = automatic admin)
# β†’ Session created with admin role
# Admin creates project via CLI:
python -m src.cli.project_cli create-project \
--projects-file /data/bootstrap/projects.json \
--slug "parrots-2026" \
--name "Parrots Dataset 2026" \
--dataset-repo-id "birdnet/parrots-2026" \
--visibility "collaborative"
# OK: project 'parrots-2026' created
```
**What happened:**
- Project created in `/data/bootstrap/projects.json`
- No `owner_username` (collaborative mode)
- Admin automatically has validator role
### Phase 2: Admin Invites Collaborator
Using the Admin Panel UI, admin clicks "Invite User":
```
Actor: alice_admin (project admin)
Invitee Username: bob_validator
Invitee Email: bob@example.com
Role: validator
```
**What happens:**
1. Authorization check: Alice is admin for this project βœ“
2. Project visibility check: Collaborative (invites allowed) βœ“
3. Invite created with 72h TTL
4. Email sent via SMTP:
```
Subject: You've been invited to validate "Parrots Dataset 2026"
Hello Bob,
Alice has invited you to join the "Parrots Dataset 2026" project as a validator.
Project: parrots-2026
Role: validator
Expires: [72 hours from now]
To accept this invitation, log in with your Hugging Face token at:
https://birdnet-validator.example.com/login
After logging in, your invite will appear. Click "Accept" to join.
```
5. Invite stored in `/data/bootstrap/invites.json`:
```json
{
"bob_validator": {
"parrots-2026": {
"role": "validator",
"invited_by": "alice_admin",
"created_at": "2026-04-01T12:00:00Z",
"expires_at": "2026-04-04T12:00:00Z"
}
}
}
```
### Phase 3: Collaborator Receives & Accepts Invite
**Bob receives email, opens Space app, logs in:**
1. Bob enters his HF token: `hf_user_***`
2. AuthService resolves username from HF API (`whoami()`)
3. Bob's session created
4. Bob's token stored separately: `_hf_tokens_by_username["bob_validator"] = "hf_user_***"`
5. **Pending invites auto-appear** in project selector
6. Bob clicks "Accept Invite"
**What happens:**
- Invite moved from pending β†’ accepted
- Bob added to `user_access.json`:
```json
{
"bob_validator": {
"parrots-2026": "validator"
}
}
```
- Invite removed from `invites.json`
### Phase 4: Both Users Have Access & Own Tokens
**Alice (admin):**
- Token: `hf_alice_***` (stored in `_hf_tokens_by_username["alice_admin"]`)
- Role: admin for "parrots-2026"
- Can: create, assign, remove, validate, manage project
**Bob (validator):**
- Token: `hf_bob_***` (stored in `_hf_tokens_by_username["bob_validator"]`)
- Role: validator for "parrots-2026"
- Can: validate detections, see project data
**Key Security Feature:**
- Tokens are **NOT** stored in Session objects
- Token lookup happens only when needed (validation flow)
- Each user brings own token β†’ no token sharing
- If Alice's token revoked on HF, only Alice affected
---
## Private Project Workflow
### Create Private Project
```bash
python -m src.cli.project_cli create-project \
--projects-file /data/bootstrap/projects.json \
--user-access-file /data/bootstrap/user_access.json \
--slug "proprietary-2026" \
--name "Proprietary Dataset" \
--dataset-repo-id "owner/proprietary-2026" \
--visibility "private" \
--owner "carol_owner"
```
**Result:**
- `owner_username = "carol_owner"` is set (required)
- Carol automatically gets admin role
- Invites are **rejected** if attempted
### Try to Invite (Fails)
If Carol or another admin tries to invite a collaborator:
```
Actor: carol_owner
Invitee: dave_validator
β†’ ERROR: "Private projects do not accept collaborators"
```
**No invite created.** Private projects are owner-only.
---
## Configuration for HF Spaces Deployment
### Set Environment Variables
On your [HF Space Settings](https://huggingface.co/spaces/YOUR_ORG/BirdNET-Validator-App/settings):
```bash
# Basic config
BIRDNET_ENABLE_DEMO_BOOTSTRAP=false
BIRDNET_BOOTSTRAP_DIR=/data/bootstrap
BIRDNET_VALIDATIONS_DIR=/data/validations
# Invite settings
BIRDNET_INVITE_EMAIL_ENABLED=true
BIRDNET_INVITE_EMAIL_SENDER=noreply@birdnet-validator.example.com
BIRDNET_INVITE_EMAIL_LOGIN_URL=https://huggingface.co/spaces/YOUR_ORG/BirdNET-Validator-App
BIRDNET_INVITE_TTL_HOURS=72
# SMTP settings (transactional email service)
BIRDNET_SMTP_HOST=smtp.gmail.com # or your email provider
BIRDNET_SMTP_PORT=587
BIRDNET_SMTP_USERNAME=your-email@gmail.com
BIRDNET_SMTP_PASSWORD=your-app-password # Use app-specific password for 2FA
BIRDNET_SMTP_USE_TLS=true
```
### Test SMTP Configuration Locally
```python
from src.config.runtime_config import RuntimeConfig
from src.services.invite_email_notifier import SmtpInviteEmailNotifier, InviteEmailPayload
from datetime import datetime, timedelta, UTC
config = RuntimeConfig.from_env()
if config.invite_email_enabled and config.smtp_host:
notifier = SmtpInviteEmailNotifier(
host=config.smtp_host,
port=config.smtp_port,
username=config.smtp_username,
password=config.smtp_password,
use_tls=config.smtp_use_tls,
)
test_payload = InviteEmailPayload(
invitee_username="test_user",
invitee_email="test@example.com",
project_slug="test-project",
role="validator",
invited_by="admin_user",
expires_at=datetime.now(UTC) + timedelta(hours=72),
login_url=config.invite_email_login_url,
)
ok, message = notifier.send(test_payload)
print(f"Email send: {ok=}, {message=}")
else:
print("Email invites disabled or SMTP not configured")
```
---
## Data Durability & Persistence
### Bootstrap Files Location
On HF Spaces, all bootstrap and validation data persists in `/data/`:
```
/data/
β”œβ”€β”€ bootstrap/
β”‚ β”œβ”€β”€ projects.json # All projects (collaborative + private)
β”‚ β”œβ”€β”€ user_access.json # User β†’ Project β†’ Role mapping
β”‚ └── invites.json # Pending invites with TTL
└── validations/
β”œβ”€β”€ project-1/
β”‚ β”œβ”€β”€ 2026-04-01.jsonl # Append-only validation events
β”‚ β”œβ”€β”€ 2026-04-02.jsonl
β”‚ └── ...
└── project-2/
└── ...
```
### Atomic Writes (Crash-Safe)
All JSON writes use atomic pattern:
```python
temp_file = target.with_suffix(target.suffix + ".tmp")
temp_file.write_text(json.dumps(data))
temp_file.replace(target) # Atomic on modern filesystems
```
**Benefit:** If Space crashes during write, file is not corrupted.
### Data Loss Prevention
Since `/data` is persistent on HF Spaces:
- Projects, ACL, invites **survive** Space restarts βœ“
- Unfinished validations are **not** lost βœ“
- User data is **durable** across redeploys βœ“
---
## Testing the Workflow Locally
### 1. Create Users & Projects
```bash
# Set environment for local testing
export BIRDNET_ENABLE_DEMO_BOOTSTRAP=false
export BIRDNET_PROJECTS_FILE=/tmp/projects.json
export BIRDNET_USER_ACCESS_FILE=/tmp/user_access.json
export BIRDNET_INVITES_FILE=/tmp/invites.json
export BIRDNET_INVITE_EMAIL_ENABLED=true
export BIRDNET_INVITE_EMAIL_SENDER=test@example.com
export BIRDNET_INVITE_EMAIL_LOGIN_URL=http://localhost:7860
# Create initial files
echo '[]' > /tmp/projects.json
echo '{}' > /tmp/user_access.json
echo '{}' > /tmp/invites.json
# Create project
python -m src.cli.project_cli create-project \
--projects-file /tmp/projects.json \
--user-access-file /tmp/user_access.json \
--slug test-project \
--name "Test Project" \
--dataset-repo-id "test/project" \
--visibility collaborative \
--owner alice
```
### 2. Start App & Test UI
```bash
# Terminal 1: Start server
python app.py # Runs on http://localhost:7860
# Terminal 2: Test workflow in UI
# - Log in with first user (becomes admin)
# - Create collaborative project
# - Invite second user
# - Log out, log in as second user
# - Accept invite from project selector
# - Both users see project in dropdown
```
### 3. Verify Data Files
```bash
# Check projects file
cat /tmp/projects.json | python -m json.tool
# Check user access
cat /tmp/user_access.json | python -m json.tool
# Check invites
cat /tmp/invites.json | python -m json.tool
```
---
## Troubleshooting
### "Email not sent" but project created
**Cause:** SMTP not configured or Email invites disabled
**Fix:**
1. Check `BIRDNET_INVITE_EMAIL_ENABLED=true`
2. Check SMTP credentials are correct
3. For Gmail: use [App Password](https://support.google.com/accounts/answer/185833), not account password
4. Check firewall allows outbound SMTP (port 587)
**Fallback:** System will create invite without email. User must be told invite URL manually.
### User can't accept invite
**Cause:** Session TTL expired, or invite expired (>72h)
**Fix:**
1. Re-login (creates fresh session)
2. Resend invite
3. Adjust `BIRDNET_INVITE_TTL_HOURS` if needed
### Private project invite fails
**Expected behavior:** Private projects reject invites
**Workaround:** Assign users directly in `user_access.json` if bootstrap:
```json
{
"username": {
"private-project": "validator"
}
}
```
### Tokens not being used in validation
**Cause:** Check project token precedence in `app_factory.py`:
```python
# Preference order:
# 1. User's HF token (from session login)
# 2. Project token (for private project owners only)
# 3. None (validation fails)
```
Set users' HF token via login, or set project token via CLI/UI.
---
## Next Steps
### For Production Deployment
1. βœ… **Configure SMTP** on HF Space with real email service
2. βœ… **Test end-to-end**: Create project β†’ Invite user β†’ Accept β†’ Validate
3. βœ… **Verify durability**: Create data, restart Space, confirm persistence
4. πŸ“‹ **Monitor invites**: Set up alerts if emails bounce
5. πŸ“‹ **Audit trail**: Review `/data/bootstrap/invites.json` periodically
### For Future Enhancements
- [ ] Invite expiration auto-cleanup background task
- [ ] Invite revocation by admin
- [ ] User deactivation cascade (revoke all invites, remove all roles)
- [ ] Email template customization
- [ ] Multi-language invite emails
- [ ] Webhook notifications (Discord, Slack)
---
## Test Results Summary
| Test Suite | Pass Rate | Key Tests |
|-----------|-----------|-----------|
| **Unit: AuthService** | 37/37 βœ… | Login, session TTL, ACL, token storage, invite lifecycle |
| **Unit: AdminPanel** | 5/5 βœ… | Private isolation, email dispatch, authorization |
| **Unit: AppFactory** | 45/45 βœ… | Bootstrap loading, config handling, audio helpers |
| **Unit: CLI** | 10/10 βœ… | Create-project, init-dataset, build-index, verify |
| **Integration** | 2/2 βœ… | Validation flow, detection repository |
| **Total** | **139/139 βœ…** | All workflows tested and passing |
---
**Last Updated:** 2026-04-01
**Deployment Status:** Ready for HF Spaces production
**Latest Commit:** 5e463e3 (feat: security hardening, durable state, email invites)