Spaces:
Sleeping
Sleeping
Sync app from GitHub Actions (feature/project-state-security-privacy-review)
Browse files- README.md +23 -7
- docs/TEMP_BYO_SUPABASE_EDGE_FUNCTION_HF_IDENTITY_PLAN.md +381 -0
- docs/TEMP_HF_TRUSTED_TEAM_PRIVATE_STORAGE_DELIVERY_PLAN.md +362 -0
- src/config/runtime_config.py +4 -0
- src/repositories/hf_bucket_validation_repository.py +90 -0
- src/ui/app_factory.py +264 -22
- src/ui/login_page.py +49 -16
- src/ui/theme.py +9 -2
- tests/unit/test_app_factory_audio_helpers.py +116 -0
- tests/unit/test_hf_bucket_validation_repository.py +17 -0
- tests/unit/test_runtime_config.py +16 -0
README.md
CHANGED
|
@@ -10,7 +10,6 @@ app_file: app.py
|
|
| 10 |
hf_oauth: true
|
| 11 |
hf_oauth_scopes:
|
| 12 |
- read-repos
|
| 13 |
-
- contribute-repos
|
| 14 |
pinned: false
|
| 15 |
---
|
| 16 |
|
|
@@ -178,7 +177,7 @@ User access bootstrap file example (`BIRDNET_USER_ACCESS_FILE`):
|
|
| 178 |
- SDK: `Gradio`
|
| 179 |
- Python: `3.11`
|
| 180 |
- OAuth: enabled (`hf_oauth: true` in this README metadata)
|
| 181 |
-
- OAuth scopes: `read-repos`
|
| 182 |
|
| 183 |
2. Push this repository to the Space.
|
| 184 |
|
|
@@ -199,11 +198,24 @@ Optional runtime settings:
|
|
| 199 |
- `BIRDNET_PAGE_SIZE` (default `25`)
|
| 200 |
- `BIRDNET_ENABLE_DEMO_BOOTSTRAP` (`false` in production)
|
| 201 |
- `BIRDNET_HF_PROJECT_STATE_WRITES_ENABLED=true` (experimental `_state` diagnostics and recovery only while a new collaborative backend is selected)
|
| 202 |
-
- `BIRDNET_HF_BUCKET_VALIDATIONS_ENABLED=true` (legacy experimental access
|
|
|
|
|
|
|
| 203 |
|
| 204 |
-
|
| 205 |
|
| 206 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 207 |
|
| 208 |
Legacy Supabase state backend, available while migrating existing projects:
|
| 209 |
|
|
@@ -211,9 +223,9 @@ Legacy Supabase state backend, available while migrating existing projects:
|
|
| 211 |
- `SUPABASE_URL` (Secret, example: `https://xxxx.supabase.co`)
|
| 212 |
- `SUPABASE_SERVICE_ROLE_KEY` (Secret)
|
| 213 |
|
| 214 |
-
When Supabase is enabled, projects, ACL, invites, validation events, and current validation snapshots are stored in Supabase instead of `/data`.
|
| 215 |
|
| 216 |
-
The
|
| 217 |
|
| 218 |
Optional invite email settings:
|
| 219 |
|
|
@@ -532,6 +544,10 @@ python -m src.uploader_cli.main start --repo-id alice/birdnet-2026 --segments C:
|
|
| 532 |
- `BIRDNET_STATE_BACKEND`
|
| 533 |
- `SUPABASE_URL`
|
| 534 |
- `SUPABASE_SERVICE_ROLE_KEY`
|
|
|
|
|
|
|
|
|
|
|
|
|
| 535 |
|
| 536 |
### Uploader CLI Environment Variables
|
| 537 |
|
|
|
|
| 10 |
hf_oauth: true
|
| 11 |
hf_oauth_scopes:
|
| 12 |
- read-repos
|
|
|
|
| 13 |
pinned: false
|
| 14 |
---
|
| 15 |
|
|
|
|
| 177 |
- SDK: `Gradio`
|
| 178 |
- Python: `3.11`
|
| 179 |
- OAuth: enabled (`hf_oauth: true` in this README metadata)
|
| 180 |
+
- OAuth scopes: `read-repos` only; trusted-team validation writes use each collaborator's explicitly entered personal token
|
| 181 |
|
| 182 |
2. Push this repository to the Space.
|
| 183 |
|
|
|
|
| 198 |
- `BIRDNET_PAGE_SIZE` (default `25`)
|
| 199 |
- `BIRDNET_ENABLE_DEMO_BOOTSTRAP` (`false` in production)
|
| 200 |
- `BIRDNET_HF_PROJECT_STATE_WRITES_ENABLED=true` (experimental `_state` diagnostics and recovery only while a new collaborative backend is selected)
|
| 201 |
+
- `BIRDNET_HF_BUCKET_VALIDATIONS_ENABLED=true` (legacy experimental access for existing Bucket projects)
|
| 202 |
+
- `BIRDNET_HF_TRUSTED_TEAM_MODE_ENABLED=true` (private HF-only delivery mode for small trusted teams)
|
| 203 |
+
- `BIRDNET_HF_PROJECT_STATE_REPOS=project-org/audio-dataset_state` (comma-separated private state repos restored after Space rebuild)
|
| 204 |
|
| 205 |
+
The HF trusted-team mode is intended only for confidential projects whose validators are known collaborators. It creates a private `_state` repository for infrequent configuration/ACL updates and a private Storage Bucket for validation events and snapshots. The source dataset must already be private and owned by a Hugging Face organization; personal-namespace datasets are rejected in this mode because they did not support the required second-user write workflow in testing. The demonstrated free-tier permission path uses a dedicated project organization whose trusted validators have Hugging Face `write` membership; users outside the organization cannot access its private resources. Select `collaborative` when creating the project inside the app: this authorizes approved app members and does not make the Hugging Face resources public. Each member works with their own HF authorization, and the app does not persist a shared project token for projects created in this mode.
|
| 206 |
|
| 207 |
+
Recommended trusted-team Space settings:
|
| 208 |
+
|
| 209 |
+
```text
|
| 210 |
+
BIRDNET_HF_TRUSTED_TEAM_MODE_ENABLED=true
|
| 211 |
+
BIRDNET_AUTH_MODE=hf_token
|
| 212 |
+
BIRDNET_HF_PROJECT_STATE_REPOS=project-org/audio-dataset_state
|
| 213 |
+
HF_TOKEN=<administrator token stored as a Space secret for startup restore>
|
| 214 |
+
```
|
| 215 |
+
|
| 216 |
+
After creating a project, add its generated `_state` repository to `BIRDNET_HF_PROJECT_STATE_REPOS` before relying on rebuild recovery. Each administrator and validator should open **Projects > Private project storage access** and pass the access check with their own token before validating real records; the check writes and removes only a small diagnostic marker. A private personal Bucket did not permit the invited second-user workflow in testing; the confirmed collaborative route is an organization-owned private dataset and Bucket with trusted `write` members. The private `_state` OAuth experiment also failed as a high-frequency direct validation store because `contribute-repos` permitted contribution through Pull Requests rather than direct durable writes to `main`.
|
| 217 |
+
|
| 218 |
+
Projects previously created with a private `_state` repository can be recovered from the **Admin** tab using **Connect Existing State**. Sign in with the Hugging Face account that is recorded as an administrator in that repository's `acl.json`, then provide the `_state` repo id (for example, `project-org/audio_dataset_state`). In trusted-team mode, recovery accepts only a manifest that declares a private validation Bucket and still points to a private, organization-owned collaborative project. The app loads the saved manifest, ACL, and pending invites; it does not accept a new local definition over an existing state repository.
|
| 219 |
|
| 220 |
Legacy Supabase state backend, available while migrating existing projects:
|
| 221 |
|
|
|
|
| 223 |
- `SUPABASE_URL` (Secret, example: `https://xxxx.supabase.co`)
|
| 224 |
- `SUPABASE_SERVICE_ROLE_KEY` (Secret)
|
| 225 |
|
| 226 |
+
When Supabase is enabled, projects, ACL, invites, validation events, and current validation snapshots are stored in Supabase instead of `/data`. For eventual public distribution, the planned backend is administrator-owned Supabase provisioned through OAuth with an Edge Function validating Hugging Face identity; see `docs/TEMP_BYO_SUPABASE_EDGE_FUNCTION_HF_IDENTITY_PLAN.md`.
|
| 227 |
|
| 228 |
+
The **Private state OAuth diagnostic** action in **Projects** applies to experimental direct `_state` writes. In trusted-team mode, frequent validation state uses the Bucket rather than direct repository commits.
|
| 229 |
|
| 230 |
Optional invite email settings:
|
| 231 |
|
|
|
|
| 544 |
- `BIRDNET_STATE_BACKEND`
|
| 545 |
- `SUPABASE_URL`
|
| 546 |
- `SUPABASE_SERVICE_ROLE_KEY`
|
| 547 |
+
- `BIRDNET_HF_TRUSTED_TEAM_MODE_ENABLED`
|
| 548 |
+
- `BIRDNET_HF_PROJECT_STATE_REPOS`
|
| 549 |
+
- `BIRDNET_HF_PROJECT_STATE_WRITES_ENABLED`
|
| 550 |
+
- `BIRDNET_HF_BUCKET_VALIDATIONS_ENABLED`
|
| 551 |
|
| 552 |
### Uploader CLI Environment Variables
|
| 553 |
|
docs/TEMP_BYO_SUPABASE_EDGE_FUNCTION_HF_IDENTITY_PLAN.md
ADDED
|
@@ -0,0 +1,381 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# BYO Supabase With Edge Function And Hugging Face Identity Plan
|
| 2 |
+
|
| 3 |
+
Status: deferred target architecture for public distribution
|
| 4 |
+
Created: 2026-05-26
|
| 5 |
+
Implementation target: a future branch after the trusted-team HF delivery is stable
|
| 6 |
+
|
| 7 |
+
## Decision Summary
|
| 8 |
+
|
| 9 |
+
For a distributed public application, each project administrator should own a
|
| 10 |
+
Supabase backend in their own account. The BirdNET Validator app should:
|
| 11 |
+
|
| 12 |
+
1. authenticate scientists through Hugging Face;
|
| 13 |
+
2. let administrators connect a Supabase account through Supabase OAuth;
|
| 14 |
+
3. provision or connect an administrator-owned Supabase project;
|
| 15 |
+
4. install the validation schema and a project write API in an Edge Function;
|
| 16 |
+
5. issue short-lived validation authorization tokens after checking the user's
|
| 17 |
+
Hugging Face identity and project role; and
|
| 18 |
+
6. never distribute an administrator database secret to validators.
|
| 19 |
+
|
| 20 |
+
This model keeps relational-database performance while shifting storage quota,
|
| 21 |
+
state ownership, exports, and eventual billing responsibility to the project
|
| 22 |
+
administrator.
|
| 23 |
+
|
| 24 |
+
## Why This Is The Distribution Target
|
| 25 |
+
|
| 26 |
+
The application must support:
|
| 27 |
+
|
| 28 |
+
1. thousands or millions of source detections;
|
| 29 |
+
2. multiple simultaneous validators;
|
| 30 |
+
3. validators capable of submitting one decision every few seconds;
|
| 31 |
+
4. complete audit history and reconstructable current state;
|
| 32 |
+
5. fast progress dashboards and scientific exports;
|
| 33 |
+
6. private pre-publication research data; and
|
| 34 |
+
7. projects owned by independent administrators, not by the Space maintainer.
|
| 35 |
+
|
| 36 |
+
A Supabase database is structurally appropriate for concurrent events,
|
| 37 |
+
transactions, queries, and reports. Google Drive is useful for backups but
|
| 38 |
+
would require a custom append-only synchronization engine as the primary
|
| 39 |
+
backend. Hugging Face organization storage is appropriate for a small trusted
|
| 40 |
+
team but requires broad organization write access for validators on the
|
| 41 |
+
currently proven free-tier path.
|
| 42 |
+
|
| 43 |
+
## Verified Constraints And Required Proofs
|
| 44 |
+
|
| 45 |
+
### Confirmed Through Official Documentation
|
| 46 |
+
|
| 47 |
+
1. Supabase exposes a Management API intended for platform integrations.
|
| 48 |
+
2. Supabase integrations can use OAuth rather than requiring administrators to
|
| 49 |
+
paste personal access tokens.
|
| 50 |
+
3. OAuth scopes include project, database, secret, and Edge Function
|
| 51 |
+
management permissions.
|
| 52 |
+
4. Supabase Edge Functions can run trusted server code close to the
|
| 53 |
+
administrator-owned database.
|
| 54 |
+
5. Hugging Face identity is not listed as a built-in Supabase third-party auth
|
| 55 |
+
provider.
|
| 56 |
+
|
| 57 |
+
### Must Be Proven Before Product Implementation
|
| 58 |
+
|
| 59 |
+
1. A regular free Supabase administrator can authorize our OAuth integration.
|
| 60 |
+
2. The granted scopes allow creating or selecting a project without manual
|
| 61 |
+
administrator secrets.
|
| 62 |
+
3. The integration can install schema migrations in that project, or there is
|
| 63 |
+
an acceptable one-time SQL fallback.
|
| 64 |
+
4. The integration can deploy/update an Edge Function in a free project.
|
| 65 |
+
5. A validator authenticated only with Hugging Face can submit through that
|
| 66 |
+
function using a short-lived app authorization token.
|
| 67 |
+
6. Revocation, replay protection, simultaneous writes, exports, and restore
|
| 68 |
+
paths work under real use.
|
| 69 |
+
|
| 70 |
+
No production redesign begins until this spike has passed with two distinct
|
| 71 |
+
Hugging Face users and a Supabase account owned by the test administrator.
|
| 72 |
+
|
| 73 |
+
## Ownership Model
|
| 74 |
+
|
| 75 |
+
### Central Space Owns
|
| 76 |
+
|
| 77 |
+
The shared Space owns only application runtime concerns:
|
| 78 |
+
|
| 79 |
+
1. UI and validation workflows;
|
| 80 |
+
2. Hugging Face login/session handling;
|
| 81 |
+
3. Supabase OAuth integration identity;
|
| 82 |
+
4. the private signing key used to issue short-lived validation assertions;
|
| 83 |
+
5. minimal encrypted connection registry, only if required to reconnect an
|
| 84 |
+
administrator's backend; and
|
| 85 |
+
6. diagnostics and compatibility migrations.
|
| 86 |
+
|
| 87 |
+
### Administrator-Owned Supabase Project Owns
|
| 88 |
+
|
| 89 |
+
One Supabase project may contain several BirdNET projects belonging to the
|
| 90 |
+
same administrator:
|
| 91 |
+
|
| 92 |
+
```text
|
| 93 |
+
admin Supabase account
|
| 94 |
+
birdnet-validator-state project
|
| 95 |
+
birdnet_projects
|
| 96 |
+
project_members
|
| 97 |
+
project_invites
|
| 98 |
+
detection_catalog
|
| 99 |
+
validation_events
|
| 100 |
+
current_validations
|
| 101 |
+
export_jobs
|
| 102 |
+
audit_events
|
| 103 |
+
edge function: submit-validation
|
| 104 |
+
```
|
| 105 |
+
|
| 106 |
+
It stores project metadata, authorization, validation history, derived current
|
| 107 |
+
state, and exports. It does not need to store audio files.
|
| 108 |
+
|
| 109 |
+
### Hugging Face Dataset Owns
|
| 110 |
+
|
| 111 |
+
The administrator's Hugging Face dataset continues to own:
|
| 112 |
+
|
| 113 |
+
1. audio segments;
|
| 114 |
+
2. uploader-generated indexes;
|
| 115 |
+
3. original BirdNET detection metadata; and
|
| 116 |
+
4. stable identifiers required to join exports.
|
| 117 |
+
|
| 118 |
+
For private audio, validators still need either individual read permission in
|
| 119 |
+
Hugging Face or a separately designed read proxy. BYO Supabase solves state
|
| 120 |
+
ownership, not private source-audio authorization.
|
| 121 |
+
|
| 122 |
+
## Authentication And Authorization Model
|
| 123 |
+
|
| 124 |
+
### Administrator Setup
|
| 125 |
+
|
| 126 |
+
1. Administrator logs into BirdNET Validator with Hugging Face.
|
| 127 |
+
2. Administrator chooses **Connect Supabase Storage**.
|
| 128 |
+
3. App redirects to Supabase OAuth consent.
|
| 129 |
+
4. Administrator grants only required scopes.
|
| 130 |
+
5. App creates or selects a Supabase project owned by that administrator.
|
| 131 |
+
6. App provisions schema and Edge Function.
|
| 132 |
+
7. App records a backend reference for the BirdNET project.
|
| 133 |
+
|
| 134 |
+
### Validator Session
|
| 135 |
+
|
| 136 |
+
1. Validator logs in with Hugging Face only.
|
| 137 |
+
2. App verifies identity from the HF session/token.
|
| 138 |
+
3. App reads project membership from the administrator-owned backend through a
|
| 139 |
+
read path or cached signed project mapping.
|
| 140 |
+
4. When the validator submits a decision, the app issues a signed, short-lived
|
| 141 |
+
authorization assertion containing:
|
| 142 |
+
|
| 143 |
+
```json
|
| 144 |
+
{
|
| 145 |
+
"iss": "birdnet-validator-space",
|
| 146 |
+
"sub": "hf:validator_username",
|
| 147 |
+
"project_id": "uuid",
|
| 148 |
+
"role": "validator",
|
| 149 |
+
"action": "validation:submit",
|
| 150 |
+
"jti": "unique-id",
|
| 151 |
+
"iat": 0,
|
| 152 |
+
"exp": 0
|
| 153 |
+
}
|
| 154 |
+
```
|
| 155 |
+
|
| 156 |
+
5. The Supabase Edge Function validates signature, expiry, action, project
|
| 157 |
+
membership, and replay rules before performing the database transaction.
|
| 158 |
+
|
| 159 |
+
### Trust Boundary
|
| 160 |
+
|
| 161 |
+
The Space remains the identity broker: it asserts that a Hugging Face user is
|
| 162 |
+
currently signed in and permitted to take an action. It does not own the
|
| 163 |
+
scientific validation database or distribute database administrator keys.
|
| 164 |
+
|
| 165 |
+
## Supabase OAuth And Secret Policy
|
| 166 |
+
|
| 167 |
+
Required principles:
|
| 168 |
+
|
| 169 |
+
1. Prefer Supabase OAuth over personal access tokens.
|
| 170 |
+
2. Request the minimum scopes needed for the provisioning phase.
|
| 171 |
+
3. Keep OAuth refresh tokens encrypted at rest if persistent reconnection is
|
| 172 |
+
required.
|
| 173 |
+
4. Separate provisioning authority from runtime validation authority.
|
| 174 |
+
5. Do not save `service_role` or modern `secret` API keys in project rows.
|
| 175 |
+
6. Never send privileged Supabase credentials to a browser or validator.
|
| 176 |
+
7. Provide a **Disconnect Supabase** action that revokes local references and
|
| 177 |
+
explains how to revoke authorization in Supabase.
|
| 178 |
+
|
| 179 |
+
Candidate scopes must be finalized after the spike. Expected needs include:
|
| 180 |
+
|
| 181 |
+
| Capability | Expected purpose |
|
| 182 |
+
|---|---|
|
| 183 |
+
| Project read/write | Select or create admin-owned Supabase project |
|
| 184 |
+
| Database write | Install/update schema if available to the integration |
|
| 185 |
+
| Edge Function write | Deploy validation submission handler |
|
| 186 |
+
| Secret write | Install function verification material without exposing it |
|
| 187 |
+
|
| 188 |
+
## Database Schema
|
| 189 |
+
|
| 190 |
+
### `birdnet_projects`
|
| 191 |
+
|
| 192 |
+
| Column | Purpose |
|
| 193 |
+
|---|---|
|
| 194 |
+
| `project_id` | Stable UUID |
|
| 195 |
+
| `project_slug` | User-facing key |
|
| 196 |
+
| `owner_hf_username` | Project administrator identity |
|
| 197 |
+
| `dataset_repo_id` | HF source dataset |
|
| 198 |
+
| `created_at`, `updated_at` | Audit timestamps |
|
| 199 |
+
| `active` | Archive control |
|
| 200 |
+
|
| 201 |
+
### `project_members`
|
| 202 |
+
|
| 203 |
+
| Column | Purpose |
|
| 204 |
+
|---|---|
|
| 205 |
+
| `project_id` | Project scope |
|
| 206 |
+
| `hf_username` | Verified collaborator identity |
|
| 207 |
+
| `role` | `admin`, `validator`, or `viewer` |
|
| 208 |
+
| `active` | Revocation control |
|
| 209 |
+
| `granted_by`, `granted_at` | Audit |
|
| 210 |
+
|
| 211 |
+
### `validation_events`
|
| 212 |
+
|
| 213 |
+
Append-only source of truth:
|
| 214 |
+
|
| 215 |
+
| Column | Purpose |
|
| 216 |
+
|---|---|
|
| 217 |
+
| `event_id` | UUID/idempotency key |
|
| 218 |
+
| `project_id` | Scope |
|
| 219 |
+
| `detection_key` | Stable segment identifier |
|
| 220 |
+
| `actor_hf_username` | Validator |
|
| 221 |
+
| `status`, `corrected_species`, `notes` | Scientific review |
|
| 222 |
+
| `expected_version`, `new_version` | Concurrency control |
|
| 223 |
+
| `created_at` | Audit ordering |
|
| 224 |
+
| `auth_jti` | Replay/audit link |
|
| 225 |
+
|
| 226 |
+
### `current_validations`
|
| 227 |
+
|
| 228 |
+
Fast derived state keyed by `(project_id, detection_key)`. It must be updated
|
| 229 |
+
within the same database transaction as event insertion.
|
| 230 |
+
|
| 231 |
+
### Detection Catalog Policy
|
| 232 |
+
|
| 233 |
+
The full source detection table may be loaded from HF on demand or cached in
|
| 234 |
+
Supabase when necessary for reports. The original dataset index remains the
|
| 235 |
+
authoritative source for audio, species, confidence, start time, and end time.
|
| 236 |
+
|
| 237 |
+
## Edge Function Write Contract
|
| 238 |
+
|
| 239 |
+
Suggested endpoint:
|
| 240 |
+
|
| 241 |
+
```http
|
| 242 |
+
POST /functions/v1/submit-validation
|
| 243 |
+
Authorization: Bearer <short-lived app assertion>
|
| 244 |
+
Content-Type: application/json
|
| 245 |
+
Idempotency-Key: <event uuid>
|
| 246 |
+
```
|
| 247 |
+
|
| 248 |
+
Request:
|
| 249 |
+
|
| 250 |
+
```json
|
| 251 |
+
{
|
| 252 |
+
"project_id": "uuid",
|
| 253 |
+
"detection_key": "stable-key",
|
| 254 |
+
"status": "positive",
|
| 255 |
+
"corrected_species": null,
|
| 256 |
+
"notes": "",
|
| 257 |
+
"expected_version": 0
|
| 258 |
+
}
|
| 259 |
+
```
|
| 260 |
+
|
| 261 |
+
Function requirements:
|
| 262 |
+
|
| 263 |
+
1. verify signature against installed public key or shared verifier secret;
|
| 264 |
+
2. reject expired or replayed assertions;
|
| 265 |
+
3. verify active member role;
|
| 266 |
+
4. verify request project equals asserted project;
|
| 267 |
+
5. execute atomic event insert and snapshot update;
|
| 268 |
+
6. return the new version or a structured conflict;
|
| 269 |
+
7. append an audit record for authorization failures and accepted writes.
|
| 270 |
+
|
| 271 |
+
## Conflict And Throughput Strategy
|
| 272 |
+
|
| 273 |
+
1. Every decision is an immutable event.
|
| 274 |
+
2. Current state updates use an atomic database transaction or SQL RPC.
|
| 275 |
+
3. Two decisions for the same expected version return a conflict for review;
|
| 276 |
+
neither is silently discarded.
|
| 277 |
+
4. The UI may batch network calls only if every event keeps an idempotency key.
|
| 278 |
+
5. Dashboard queries read `current_validations`, not all historical events.
|
| 279 |
+
|
| 280 |
+
The expected workload of rapid human validation is natural for PostgreSQL and
|
| 281 |
+
must be tested with simulated multi-user bursts.
|
| 282 |
+
|
| 283 |
+
## Administrator Experience
|
| 284 |
+
|
| 285 |
+
New UI surfaces:
|
| 286 |
+
|
| 287 |
+
1. **Storage Setup**: connect Supabase, choose existing/create backend, verify
|
| 288 |
+
schema and function health.
|
| 289 |
+
2. **Project Management**: create BirdNET project in connected backend.
|
| 290 |
+
3. **Member Management**: app-level invitation and role controls persisted in
|
| 291 |
+
the administrator's Supabase.
|
| 292 |
+
4. **Health**: backend owner, connection state, migration version, last
|
| 293 |
+
successful write, last export, and reconnect action.
|
| 294 |
+
5. **Export And Backup**: prepare CSV/XLSX and optionally back them up to
|
| 295 |
+
administrator-owned Google Drive or HF resources.
|
| 296 |
+
|
| 297 |
+
## Implementation Phases
|
| 298 |
+
|
| 299 |
+
### Phase 0 - Feasibility Spike
|
| 300 |
+
|
| 301 |
+
1. Register a private Supabase OAuth integration.
|
| 302 |
+
2. Complete OAuth from a local test app and from an HF Space.
|
| 303 |
+
3. List/create a free Supabase project through Management API.
|
| 304 |
+
4. Attempt automated schema provisioning.
|
| 305 |
+
5. Attempt Edge Function deployment and secret installation.
|
| 306 |
+
6. Submit one validation using a separately authenticated HF validator.
|
| 307 |
+
7. Revoke OAuth and confirm configuration writes stop.
|
| 308 |
+
|
| 309 |
+
Exit gate: all required operations succeed without a manually supplied
|
| 310 |
+
administrator database key, or the remaining one-time manual step is explicitly
|
| 311 |
+
accepted.
|
| 312 |
+
|
| 313 |
+
### Phase 1 - Backend Contracts
|
| 314 |
+
|
| 315 |
+
1. Add `supabase_owned` project backend type.
|
| 316 |
+
2. Add encrypted connection-registry interface.
|
| 317 |
+
3. Implement provisioning service behind a mockable client.
|
| 318 |
+
4. Implement Edge Function source and schema migrations.
|
| 319 |
+
5. Add connection health and migration version checks.
|
| 320 |
+
|
| 321 |
+
### Phase 2 - Identity Broker
|
| 322 |
+
|
| 323 |
+
1. Require verified HF login in production.
|
| 324 |
+
2. Implement short-lived assertion signer and key rotation.
|
| 325 |
+
3. Deploy verifier configuration to administrator Edge Functions.
|
| 326 |
+
4. Add membership checks, expiration, idempotency, and replay defense.
|
| 327 |
+
|
| 328 |
+
### Phase 3 - Product Workflows
|
| 329 |
+
|
| 330 |
+
1. Implement **Connect Supabase Storage** UI.
|
| 331 |
+
2. Create projects and membership records in admin-owned storage.
|
| 332 |
+
3. Route validation, progress, and export operations through that backend.
|
| 333 |
+
4. Remove shared project token persistence from the default flow.
|
| 334 |
+
|
| 335 |
+
### Phase 4 - Reliability And Migration
|
| 336 |
+
|
| 337 |
+
1. Import projects/events from central Supabase or trusted-team HF Buckets.
|
| 338 |
+
2. Add automated export backups.
|
| 339 |
+
3. Load-test simultaneous validation.
|
| 340 |
+
4. Security review before enabling for third-party projects.
|
| 341 |
+
|
| 342 |
+
## Test Matrix
|
| 343 |
+
|
| 344 |
+
| Area | Required tests |
|
| 345 |
+
|---|---|
|
| 346 |
+
| OAuth setup | consent, revoked token, expired token, denied scopes |
|
| 347 |
+
| Provisioning | new project, existing project, migration retry, failed setup recovery |
|
| 348 |
+
| Identity | valid HF actor, removed member, forged assertion, expired assertion |
|
| 349 |
+
| Writes | idempotent retry, same-audio conflict, parallel different audio submissions |
|
| 350 |
+
| Reports | ongoing project export, completed export, species aggregation |
|
| 351 |
+
| Privacy | no secret in UI/log/export, unauthorized user cannot query or submit |
|
| 352 |
+
| Resilience | Space rebuild, Edge Function redeploy, Supabase outage and recovery |
|
| 353 |
+
|
| 354 |
+
## Acceptance Criteria
|
| 355 |
+
|
| 356 |
+
The distribution architecture is acceptable only when:
|
| 357 |
+
|
| 358 |
+
1. project state is durably stored in the administrator's Supabase account;
|
| 359 |
+
2. validators require only their Hugging Face identity during routine work;
|
| 360 |
+
3. no validator receives database administrator secrets;
|
| 361 |
+
4. concurrent validation writes do not silently overwrite each other;
|
| 362 |
+
5. the administrator can revoke a validator and immediately stop new writes;
|
| 363 |
+
6. a Space rebuild does not erase projects or validation history;
|
| 364 |
+
7. scientific CSV/XLSX exports remain complete and reproducible; and
|
| 365 |
+
8. the app exposes the ownership, quota, and recovery status clearly.
|
| 366 |
+
|
| 367 |
+
## Official References
|
| 368 |
+
|
| 369 |
+
1. Supabase Management API: <https://supabase.com/docs/reference/api/introduction>
|
| 370 |
+
2. Supabase OAuth integrations:
|
| 371 |
+
<https://supabase.com/docs/guides/integrations/build-a-supabase-oauth-integration>
|
| 372 |
+
3. Supabase for Platforms:
|
| 373 |
+
<https://supabase.com/docs/guides/integrations/supabase-for-platforms>
|
| 374 |
+
4. Supabase Edge Functions:
|
| 375 |
+
<https://supabase.com/docs/guides/functions>
|
| 376 |
+
5. Supabase Row Level Security:
|
| 377 |
+
<https://supabase.com/docs/guides/database/postgres/row-level-security>
|
| 378 |
+
6. Supabase third-party auth:
|
| 379 |
+
<https://supabase.com/docs/guides/auth/third-party/overview>
|
| 380 |
+
7. Hugging Face OAuth:
|
| 381 |
+
<https://huggingface.co/docs/hub/oauth>
|
docs/TEMP_HF_TRUSTED_TEAM_PRIVATE_STORAGE_DELIVERY_PLAN.md
ADDED
|
@@ -0,0 +1,362 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# HF Trusted-Team Private Storage Delivery Plan
|
| 2 |
+
|
| 3 |
+
Status: immediate delivery architecture, implementation in progress on
|
| 4 |
+
`feature/project-state-security-privacy-review`
|
| 5 |
+
Created: 2026-05-26
|
| 6 |
+
Purpose: enable real validation by a small trusted research group while
|
| 7 |
+
protecting unpublished material from external access
|
| 8 |
+
|
| 9 |
+
## Decision Summary
|
| 10 |
+
|
| 11 |
+
For the immediate research workflow, all durable project data will live in
|
| 12 |
+
private Hugging Face resources under a dedicated project namespace:
|
| 13 |
+
|
| 14 |
+
```text
|
| 15 |
+
project organization namespace/
|
| 16 |
+
audio-dataset private Dataset repository
|
| 17 |
+
audio-dataset_state private Dataset repository for manifest/ACL/invites
|
| 18 |
+
audio-dataset_validation_state private Storage Bucket for validation events/snapshots
|
| 19 |
+
```
|
| 20 |
+
|
| 21 |
+
The working permission model is intentionally broad:
|
| 22 |
+
|
| 23 |
+
1. the project administrator owns or administers the namespace;
|
| 24 |
+
2. each trusted validator is granted Hugging Face `write` membership in the
|
| 25 |
+
project organization;
|
| 26 |
+
3. each user signs in using their own Hugging Face authorization;
|
| 27 |
+
4. the app writes validation decisions to the private Bucket using the current
|
| 28 |
+
validator's token, never a shared administrator token; and
|
| 29 |
+
5. users outside the organization cannot read the private dataset or state.
|
| 30 |
+
|
| 31 |
+
This is not the public distribution architecture. Trusted validators can
|
| 32 |
+
technically modify resources within the project organization outside the app.
|
| 33 |
+
That trade-off is explicitly accepted for a small trained research team.
|
| 34 |
+
|
| 35 |
+
## Why This Route Is Appropriate Now
|
| 36 |
+
|
| 37 |
+
The project needs to begin real validation before the BYO Supabase integration
|
| 38 |
+
can be researched and implemented. This route has already produced the key
|
| 39 |
+
permission proof:
|
| 40 |
+
|
| 41 |
+
| Actor | Organization role | Read private Bucket | Write private Bucket |
|
| 42 |
+
|---|---|---:|---:|
|
| 43 |
+
| Administrator | `admin` | passed | passed |
|
| 44 |
+
| Trusted validator | `write` | passed | passed |
|
| 45 |
+
| Contributor | `contributor` | passed | failed (`403`) |
|
| 46 |
+
|
| 47 |
+
Therefore, the free and currently demonstrated HF-only workflow requires
|
| 48 |
+
trusted validators to be `write` members of an isolated project organization.
|
| 49 |
+
|
| 50 |
+
## Current Branch Implementation Status
|
| 51 |
+
|
| 52 |
+
Implemented in `feature/project-state-security-privacy-review`:
|
| 53 |
+
|
| 54 |
+
1. explicit `BIRDNET_HF_TRUSTED_TEAM_MODE_ENABLED` runtime switch, disabled by
|
| 55 |
+
default for existing deployments;
|
| 56 |
+
2. automatic creation of a private validation Bucket during new project setup
|
| 57 |
+
in trusted-team mode;
|
| 58 |
+
3. automatic private `_state` repository creation with the Bucket reference in
|
| 59 |
+
the project manifest;
|
| 60 |
+
4. rejection of public source datasets, personal namespaces, and app-private
|
| 61 |
+
visibility in trusted-team mode;
|
| 62 |
+
5. no persisted shared project token for newly created Bucket-backed projects;
|
| 63 |
+
6. validation routing through the signed-in actor's HF token with no
|
| 64 |
+
administrator-token fallback;
|
| 65 |
+
7. HF-mode health status and an Admin warning explaining organization `write`
|
| 66 |
+
membership; and
|
| 67 |
+
8. a private Bucket access probe that reads, writes, verifies, and removes a
|
| 68 |
+
diagnostic marker using the signed-in collaborator's own token; and
|
| 69 |
+
9. unit tests for configuration, private-dataset enforcement, organization
|
| 70 |
+
enforcement, token removal, Bucket access/routing, conflict/reconciliation,
|
| 71 |
+
state recovery, and UI creation.
|
| 72 |
+
|
| 73 |
+
Recovery through **Connect Existing State** also refuses manifests that do not
|
| 74 |
+
declare Bucket storage or no longer identify a compatible private
|
| 75 |
+
organization-owned collaborative project.
|
| 76 |
+
|
| 77 |
+
Remaining before merge with `main`:
|
| 78 |
+
|
| 79 |
+
1. push/deploy this branch to the Space;
|
| 80 |
+
2. configure the runtime variables/secrets;
|
| 81 |
+
3. run the two-account private organization test described in the merge gate;
|
| 82 |
+
4. confirm rebuild recovery through `BIRDNET_HF_PROJECT_STATE_REPOS`; and
|
| 83 |
+
5. confirm exports after real Bucket-backed validations.
|
| 84 |
+
|
| 85 |
+
Automated verification completed on 2026-05-26:
|
| 86 |
+
|
| 87 |
+
1. `python -m pytest -q`: `246 passed`;
|
| 88 |
+
2. trusted-team UI construction with
|
| 89 |
+
`BIRDNET_HF_TRUSTED_TEAM_MODE_ENABLED=true`: passed; and
|
| 90 |
+
3. default UI construction without the new mode: passed; and
|
| 91 |
+
4. `git diff --check`: no patch whitespace errors.
|
| 92 |
+
|
| 93 |
+
## Security Boundary
|
| 94 |
+
|
| 95 |
+
### Protected Against
|
| 96 |
+
|
| 97 |
+
1. external users not granted organization membership;
|
| 98 |
+
2. accidental loss caused by Space rebuilds or lack of persistent disk;
|
| 99 |
+
3. disclosure of a shared administrator dataset token through the app;
|
| 100 |
+
4. high-frequency Git commit limits for individual validation actions, because
|
| 101 |
+
validation events are written to a Bucket rather than a dataset repository.
|
| 102 |
+
|
| 103 |
+
### Not Protected Against
|
| 104 |
+
|
| 105 |
+
1. intentional or accidental modification by a trusted organization member
|
| 106 |
+
with `write` permission;
|
| 107 |
+
2. a compromised member Hugging Face token;
|
| 108 |
+
3. project-to-project privilege spillover if multiple confidential projects
|
| 109 |
+
are placed in the same organization.
|
| 110 |
+
|
| 111 |
+
Mitigation: create a separate HF organization for each confidential project or
|
| 112 |
+
for each group of projects whose validators may safely share broad access.
|
| 113 |
+
|
| 114 |
+
## Resource Responsibilities
|
| 115 |
+
|
| 116 |
+
| Resource | Purpose | Write pattern | Authority |
|
| 117 |
+
|---|---|---|---|
|
| 118 |
+
| Private audio dataset | Audio and original detection index | Uploader only before/during ingest | Project admin/uploader |
|
| 119 |
+
| Private `_state` dataset | `project.json`, `acl.json`, `invites.json` | Infrequent admin operations | App admin flow |
|
| 120 |
+
| Private validation Bucket | events, snapshots, archives | Frequent validator writes | Current signed-in HF user |
|
| 121 |
+
| Space runtime/local files | Cache only | Ephemeral | Never authoritative |
|
| 122 |
+
|
| 123 |
+
The audio dataset must be private before a real project is created in this
|
| 124 |
+
mode. The source dataset should be treated as immutable once active validation
|
| 125 |
+
begins, other than controlled uploader completion.
|
| 126 |
+
|
| 127 |
+
## State Layout
|
| 128 |
+
|
| 129 |
+
### Private `_state` Repository
|
| 130 |
+
|
| 131 |
+
```text
|
| 132 |
+
project.json
|
| 133 |
+
acl.json
|
| 134 |
+
invites.json
|
| 135 |
+
README.md
|
| 136 |
+
```
|
| 137 |
+
|
| 138 |
+
The repository contains project configuration and app-level access metadata.
|
| 139 |
+
It must not contain HF tokens.
|
| 140 |
+
|
| 141 |
+
### Private Validation Bucket
|
| 142 |
+
|
| 143 |
+
```text
|
| 144 |
+
metadata/project.json
|
| 145 |
+
snapshots/current.json
|
| 146 |
+
events/YYYYMMDD/<event-id>.json
|
| 147 |
+
archives/events/<archive-id>.jsonl
|
| 148 |
+
```
|
| 149 |
+
|
| 150 |
+
Rules:
|
| 151 |
+
|
| 152 |
+
1. individual decisions are append-oriented events;
|
| 153 |
+
2. snapshots accelerate queues and reports but are reconstructable;
|
| 154 |
+
3. compaction archives active events instead of discarding audit history;
|
| 155 |
+
4. parallel same-version decisions are surfaced as conflicts; and
|
| 156 |
+
5. exports join validation state with the source detection metadata.
|
| 157 |
+
|
| 158 |
+
## Application Configuration
|
| 159 |
+
|
| 160 |
+
The immediate mode is activated explicitly:
|
| 161 |
+
|
| 162 |
+
```text
|
| 163 |
+
BIRDNET_HF_TRUSTED_TEAM_MODE_ENABLED=true
|
| 164 |
+
BIRDNET_AUTH_MODE=hf_token
|
| 165 |
+
BIRDNET_HF_PROJECT_STATE_REPOS=project-org/audio-dataset_state
|
| 166 |
+
HF_TOKEN=<admin read/write token for startup recovery>
|
| 167 |
+
```
|
| 168 |
+
|
| 169 |
+
Notes:
|
| 170 |
+
|
| 171 |
+
1. `BIRDNET_HF_TRUSTED_TEAM_MODE_ENABLED=true` implies HF `_state` syncing and
|
| 172 |
+
HF Bucket validation routing for projects created in the mode.
|
| 173 |
+
2. `BIRDNET_AUTH_MODE=hf_token` permits each team member to use their own
|
| 174 |
+
verified token. OAuth-only Bucket capability has not been proven.
|
| 175 |
+
3. `HF_TOKEN` allows the Space to restore project configuration after a
|
| 176 |
+
restart before any user has logged in. It belongs to the administrator of
|
| 177 |
+
this specific deployment and must be stored only as a Space secret.
|
| 178 |
+
4. `BIRDNET_HF_PROJECT_STATE_REPOS` is updated after each project is created,
|
| 179 |
+
so the Space can reload it after rebuilds.
|
| 180 |
+
5. Project tokens stored in application state are disabled for new
|
| 181 |
+
trusted-team Bucket projects. Users supply their own authorization.
|
| 182 |
+
6. The hosted OAuth button requests read access only in this interim mode;
|
| 183 |
+
it does not replace the personal token required for the proven Bucket
|
| 184 |
+
write workflow.
|
| 185 |
+
|
| 186 |
+
## Administrator Workflow
|
| 187 |
+
|
| 188 |
+
### Prerequisites
|
| 189 |
+
|
| 190 |
+
1. Create a dedicated Hugging Face organization for the confidential project.
|
| 191 |
+
2. Ensure the audio dataset exists in that namespace and is private.
|
| 192 |
+
3. Ensure the administrator token can read/write resources in the
|
| 193 |
+
organization.
|
| 194 |
+
4. Configure the Space secrets/variables above.
|
| 195 |
+
|
| 196 |
+
### Create Project In The App
|
| 197 |
+
|
| 198 |
+
1. Log into the app with the administrator's own HF token.
|
| 199 |
+
2. Open **Admin > Project management**.
|
| 200 |
+
3. Enter:
|
| 201 |
+
- project slug;
|
| 202 |
+
- display name;
|
| 203 |
+
- private dataset repo id, for example `project-org/audio-dataset`;
|
| 204 |
+
- app visibility `collaborative` for a team project.
|
| 205 |
+
4. Select **Create Project**.
|
| 206 |
+
5. The app verifies that the source dataset is private, organization-owned,
|
| 207 |
+
and uses app visibility `collaborative`.
|
| 208 |
+
6. The app creates:
|
| 209 |
+
- `project-org/audio-dataset_validation_state` as a private Bucket;
|
| 210 |
+
- `project-org/audio-dataset_state` as a private state repository.
|
| 211 |
+
7. Copy the new `_state` repo id into the Space variable
|
| 212 |
+
`BIRDNET_HF_PROJECT_STATE_REPOS`, comma-separated with existing projects.
|
| 213 |
+
8. Restart/rebuild the Space and confirm the project is restored.
|
| 214 |
+
|
| 215 |
+
### Add Trusted Validators
|
| 216 |
+
|
| 217 |
+
1. Add the validator to the dedicated project organization in Hugging Face
|
| 218 |
+
with `write` role.
|
| 219 |
+
2. Assign or invite the same HF username in the app to retain workflow roles
|
| 220 |
+
and UI filtering.
|
| 221 |
+
3. Validator logs in using their own HF token.
|
| 222 |
+
4. Validator opens **Projects > Private project storage access** and runs the
|
| 223 |
+
check, which writes and removes a small diagnostic marker without changing
|
| 224 |
+
validation records.
|
| 225 |
+
5. Validator verifies that the private audio opens and submits a test
|
| 226 |
+
validation.
|
| 227 |
+
|
| 228 |
+
Both membership layers are intentionally required in this interim model:
|
| 229 |
+
Hugging Face controls actual private resource access, while app roles control
|
| 230 |
+
the validation interface.
|
| 231 |
+
|
| 232 |
+
## Implementation Work Packages
|
| 233 |
+
|
| 234 |
+
### Package 1 - Explicit Runtime Mode
|
| 235 |
+
|
| 236 |
+
Implementation:
|
| 237 |
+
|
| 238 |
+
1. Add `BIRDNET_HF_TRUSTED_TEAM_MODE_ENABLED`.
|
| 239 |
+
2. Make the mode enable project-state synchronization and Bucket validation
|
| 240 |
+
routing without changing existing deployments by default.
|
| 241 |
+
3. Display active private HF storage status in Settings and app header.
|
| 242 |
+
|
| 243 |
+
Tests:
|
| 244 |
+
|
| 245 |
+
1. environment flag parsing;
|
| 246 |
+
2. default disabled behavior;
|
| 247 |
+
3. UI construction smoke test.
|
| 248 |
+
|
| 249 |
+
### Package 2 - Secure Project Provisioning
|
| 250 |
+
|
| 251 |
+
Implementation:
|
| 252 |
+
|
| 253 |
+
1. During project creation, require an organization-owned private source
|
| 254 |
+
dataset and app visibility `collaborative`.
|
| 255 |
+
2. Initialize the private Bucket in the dataset namespace.
|
| 256 |
+
3. Store Bucket reference in the `_state` manifest.
|
| 257 |
+
4. Initialize the private `_state` repository.
|
| 258 |
+
5. Do not persist setup/project tokens for Bucket projects.
|
| 259 |
+
6. Refuse automatic overwrite if either store contains unrelated content.
|
| 260 |
+
|
| 261 |
+
Tests:
|
| 262 |
+
|
| 263 |
+
1. private dataset accepted and public dataset rejected;
|
| 264 |
+
2. Bucket initialization and manifest linkage;
|
| 265 |
+
3. existing safe state reuse and unsafe content refusal;
|
| 266 |
+
4. setup token does not appear in serialized project state.
|
| 267 |
+
|
| 268 |
+
### Package 3 - Trusted Validator Writes
|
| 269 |
+
|
| 270 |
+
Implementation:
|
| 271 |
+
|
| 272 |
+
1. Route validation reads/writes to the Bucket for projects declaring it.
|
| 273 |
+
2. Use the current signed-in user's HF token for writes.
|
| 274 |
+
3. Never fall back to the administrator token for a validator write.
|
| 275 |
+
4. Give clear access errors when organization permission is missing.
|
| 276 |
+
|
| 277 |
+
Tests:
|
| 278 |
+
|
| 279 |
+
1. actor token routing;
|
| 280 |
+
2. no fallback to admin token;
|
| 281 |
+
3. disabled-backend hard failure;
|
| 282 |
+
4. two-user Space proof with private audio and Bucket.
|
| 283 |
+
|
| 284 |
+
### Package 4 - Recovery And Audit
|
| 285 |
+
|
| 286 |
+
Implementation:
|
| 287 |
+
|
| 288 |
+
1. Reload project metadata from configured `_state` repositories on startup.
|
| 289 |
+
2. Keep events append-oriented and snapshots reconstructable.
|
| 290 |
+
3. Preserve conflict reporting and archive compaction.
|
| 291 |
+
4. Expose CSV/XLSX exports from reconciled current state.
|
| 292 |
+
|
| 293 |
+
Tests:
|
| 294 |
+
|
| 295 |
+
1. rebuild Space and restore project/ACL;
|
| 296 |
+
2. restore snapshot from events;
|
| 297 |
+
3. simultaneous different-audio validation;
|
| 298 |
+
4. same-audio conflict;
|
| 299 |
+
5. export before and after restart.
|
| 300 |
+
|
| 301 |
+
### Package 5 - Operational Guardrails
|
| 302 |
+
|
| 303 |
+
Implementation:
|
| 304 |
+
|
| 305 |
+
1. Add visible wording that trusted-team mode requires HF organization
|
| 306 |
+
membership separately from app invitations.
|
| 307 |
+
2. Provide a collaborator-run Bucket read/write diagnostic that removes its
|
| 308 |
+
own marker and does not create a validation decision.
|
| 309 |
+
3. Warn when no `_state` repository is configured for restart recovery.
|
| 310 |
+
4. Document backup/export cadence.
|
| 311 |
+
5. Document token revocation after team changes.
|
| 312 |
+
6. Record that organization isolation is the privacy boundary.
|
| 313 |
+
|
| 314 |
+
## Merge Gate For `main`
|
| 315 |
+
|
| 316 |
+
The branch may be merged only after:
|
| 317 |
+
|
| 318 |
+
1. the existing automated test suite passes;
|
| 319 |
+
2. a private test dataset in a dedicated organization is loaded in the Space;
|
| 320 |
+
3. the administrator creates a project and sees both private state resources;
|
| 321 |
+
4. the Space is rebuilt and restores the project from `_state`;
|
| 322 |
+
5. administrator and second trusted account pass **Private project storage
|
| 323 |
+
access** using their own tokens;
|
| 324 |
+
6. a second trusted account with organization `write` role loads private
|
| 325 |
+
audio and writes a validation using its own token;
|
| 326 |
+
7. an account outside the organization cannot load audio or validation state;
|
| 327 |
+
8. twenty rapid validations persist after refresh/rebuild;
|
| 328 |
+
9. a CSV and XLSX export contain the expected validation decisions; and
|
| 329 |
+
10. the administrator keeps a source backup of audio/index data.
|
| 330 |
+
|
| 331 |
+
## Known Limitations
|
| 332 |
+
|
| 333 |
+
1. Creating organizations and managing membership remain Hugging Face web UI
|
| 334 |
+
steps in the free supported workflow investigated to date.
|
| 335 |
+
2. A trusted validator receives broad write permission in that project's
|
| 336 |
+
organization.
|
| 337 |
+
3. The startup recovery secret is acceptable for the administrator's immediate
|
| 338 |
+
private deployment, but is not the distributed public-product model.
|
| 339 |
+
4. OAuth-only access to private Buckets remains unproven; routine immediate
|
| 340 |
+
testing uses each member's own verified HF token.
|
| 341 |
+
|
| 342 |
+
## Transition To Distribution Architecture
|
| 343 |
+
|
| 344 |
+
When BYO Supabase with Edge Function support is proven:
|
| 345 |
+
|
| 346 |
+
1. retain the private HF audio dataset as source storage;
|
| 347 |
+
2. export/replay Bucket validation events into the administrator-owned
|
| 348 |
+
Supabase database;
|
| 349 |
+
3. replace organization-wide write membership with app-issued validation
|
| 350 |
+
authorization; and
|
| 351 |
+
4. archive the HF Bucket as a historical backup or retain it read-only.
|
| 352 |
+
|
| 353 |
+
## Official References
|
| 354 |
+
|
| 355 |
+
1. Hugging Face Storage Buckets:
|
| 356 |
+
<https://huggingface.co/docs/hub/storage-buckets>
|
| 357 |
+
2. Hugging Face dataset repositories:
|
| 358 |
+
<https://huggingface.co/docs/hub/datasets-overview>
|
| 359 |
+
3. Hugging Face organizations:
|
| 360 |
+
<https://huggingface.co/docs/hub/organizations>
|
| 361 |
+
4. Hugging Face OAuth:
|
| 362 |
+
<https://huggingface.co/docs/hub/oauth>
|
src/config/runtime_config.py
CHANGED
|
@@ -33,6 +33,7 @@ class RuntimeConfig:
|
|
| 33 |
auth_mode: str = "auto"
|
| 34 |
hf_project_state_writes_enabled: bool = False
|
| 35 |
hf_bucket_validations_enabled: bool = False
|
|
|
|
| 36 |
hf_project_state_repos: tuple[str, ...] = ()
|
| 37 |
|
| 38 |
@classmethod
|
|
@@ -111,6 +112,8 @@ class RuntimeConfig:
|
|
| 111 |
hf_project_state_writes_enabled = raw_hf_project_state_writes in {"1", "true", "yes", "on"}
|
| 112 |
raw_hf_bucket_validations = (os.getenv("BIRDNET_HF_BUCKET_VALIDATIONS_ENABLED") or "").strip().lower()
|
| 113 |
hf_bucket_validations_enabled = raw_hf_bucket_validations in {"1", "true", "yes", "on"}
|
|
|
|
|
|
|
| 114 |
raw_hf_project_state_repos = (os.getenv("BIRDNET_HF_PROJECT_STATE_REPOS") or "").strip()
|
| 115 |
hf_project_state_repos = tuple(
|
| 116 |
repo.strip()
|
|
@@ -146,6 +149,7 @@ class RuntimeConfig:
|
|
| 146 |
auth_mode=auth_mode,
|
| 147 |
hf_project_state_writes_enabled=hf_project_state_writes_enabled,
|
| 148 |
hf_bucket_validations_enabled=hf_bucket_validations_enabled,
|
|
|
|
| 149 |
hf_project_state_repos=hf_project_state_repos,
|
| 150 |
)
|
| 151 |
|
|
|
|
| 33 |
auth_mode: str = "auto"
|
| 34 |
hf_project_state_writes_enabled: bool = False
|
| 35 |
hf_bucket_validations_enabled: bool = False
|
| 36 |
+
hf_trusted_team_mode_enabled: bool = False
|
| 37 |
hf_project_state_repos: tuple[str, ...] = ()
|
| 38 |
|
| 39 |
@classmethod
|
|
|
|
| 112 |
hf_project_state_writes_enabled = raw_hf_project_state_writes in {"1", "true", "yes", "on"}
|
| 113 |
raw_hf_bucket_validations = (os.getenv("BIRDNET_HF_BUCKET_VALIDATIONS_ENABLED") or "").strip().lower()
|
| 114 |
hf_bucket_validations_enabled = raw_hf_bucket_validations in {"1", "true", "yes", "on"}
|
| 115 |
+
raw_hf_trusted_team_mode = (os.getenv("BIRDNET_HF_TRUSTED_TEAM_MODE_ENABLED") or "").strip().lower()
|
| 116 |
+
hf_trusted_team_mode_enabled = raw_hf_trusted_team_mode in {"1", "true", "yes", "on"}
|
| 117 |
raw_hf_project_state_repos = (os.getenv("BIRDNET_HF_PROJECT_STATE_REPOS") or "").strip()
|
| 118 |
hf_project_state_repos = tuple(
|
| 119 |
repo.strip()
|
|
|
|
| 149 |
auth_mode=auth_mode,
|
| 150 |
hf_project_state_writes_enabled=hf_project_state_writes_enabled,
|
| 151 |
hf_bucket_validations_enabled=hf_bucket_validations_enabled,
|
| 152 |
+
hf_trusted_team_mode_enabled=hf_trusted_team_mode_enabled,
|
| 153 |
hf_project_state_repos=hf_project_state_repos,
|
| 154 |
)
|
| 155 |
|
src/repositories/hf_bucket_validation_repository.py
CHANGED
|
@@ -123,6 +123,14 @@ class HfBucketCompactionResult:
|
|
| 123 |
compacted_event_count: int
|
| 124 |
|
| 125 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 126 |
class HfBucketValidationInitializer:
|
| 127 |
"""Create a private mutable validation store owned by the project admin."""
|
| 128 |
|
|
@@ -173,6 +181,88 @@ class HfBucketValidationInitializer:
|
|
| 173 |
return HfBucketValidationInitResult(bucket_id=bucket_id, initialized=True, reused_existing=False)
|
| 174 |
|
| 175 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 176 |
def _json_bytes(payload: object) -> bytes:
|
| 177 |
return json.dumps(payload, ensure_ascii=True, indent=2, sort_keys=True).encode("utf-8")
|
| 178 |
|
|
|
|
| 123 |
compacted_event_count: int
|
| 124 |
|
| 125 |
|
| 126 |
+
@dataclass(frozen=True)
|
| 127 |
+
class HfBucketPermissionProbeResult:
|
| 128 |
+
bucket_id: str
|
| 129 |
+
actor_username: str
|
| 130 |
+
diagnostic_path: str
|
| 131 |
+
verified_at: str
|
| 132 |
+
|
| 133 |
+
|
| 134 |
class HfBucketValidationInitializer:
|
| 135 |
"""Create a private mutable validation store owned by the project admin."""
|
| 136 |
|
|
|
|
| 181 |
return HfBucketValidationInitResult(bucket_id=bucket_id, initialized=True, reused_existing=False)
|
| 182 |
|
| 183 |
|
| 184 |
+
class HfBucketPermissionProbe:
|
| 185 |
+
"""Verify a user's private Bucket access without changing validation state."""
|
| 186 |
+
|
| 187 |
+
def __init__(self, api: HfBucketFilesApi | None = None) -> None:
|
| 188 |
+
self._api = api or HuggingFaceBucketFilesApi()
|
| 189 |
+
|
| 190 |
+
def probe(
|
| 191 |
+
self,
|
| 192 |
+
*,
|
| 193 |
+
bucket_id: str,
|
| 194 |
+
actor_username: str,
|
| 195 |
+
token: str | None,
|
| 196 |
+
) -> HfBucketPermissionProbeResult:
|
| 197 |
+
bucket = (bucket_id or "").strip()
|
| 198 |
+
actor = (actor_username or "").strip()
|
| 199 |
+
token_value = (token or "").strip()
|
| 200 |
+
if not bucket or not actor or not token_value:
|
| 201 |
+
raise HfBucketValidationError(
|
| 202 |
+
"Select a project and sign in with your own Hugging Face authorization before testing private storage access."
|
| 203 |
+
)
|
| 204 |
+
|
| 205 |
+
verified_at = datetime.now(UTC).isoformat()
|
| 206 |
+
diagnostic_path = f"diagnostics/access-proof/{uuid4()}.json"
|
| 207 |
+
payload = {
|
| 208 |
+
"schema_version": HF_BUCKET_VALIDATION_SCHEMA_VERSION,
|
| 209 |
+
"diagnostic_type": "private_bucket_access_proof",
|
| 210 |
+
"actor_username": actor,
|
| 211 |
+
"bucket_id": bucket,
|
| 212 |
+
"verified_at": verified_at,
|
| 213 |
+
"contains_project_data": False,
|
| 214 |
+
}
|
| 215 |
+
marker_written = False
|
| 216 |
+
try:
|
| 217 |
+
self._api.read_text(
|
| 218 |
+
bucket_id=bucket,
|
| 219 |
+
path_in_bucket="metadata/project.json",
|
| 220 |
+
token=token_value,
|
| 221 |
+
)
|
| 222 |
+
self._api.write_files(
|
| 223 |
+
bucket_id=bucket,
|
| 224 |
+
token=token_value,
|
| 225 |
+
files={diagnostic_path: _json_bytes(payload)},
|
| 226 |
+
)
|
| 227 |
+
marker_written = True
|
| 228 |
+
written_payload = json.loads(
|
| 229 |
+
self._api.read_text(
|
| 230 |
+
bucket_id=bucket,
|
| 231 |
+
path_in_bucket=diagnostic_path,
|
| 232 |
+
token=token_value,
|
| 233 |
+
)
|
| 234 |
+
)
|
| 235 |
+
if written_payload.get("actor_username") != actor:
|
| 236 |
+
raise HfBucketValidationError("The private storage diagnostic marker could not be verified.")
|
| 237 |
+
except HfBucketValidationError:
|
| 238 |
+
raise
|
| 239 |
+
except Exception as exc:
|
| 240 |
+
raise HfBucketValidationError(
|
| 241 |
+
f"Private Bucket read/write test failed for '{bucket}' using the signed-in Hugging Face account: {exc}"
|
| 242 |
+
) from exc
|
| 243 |
+
finally:
|
| 244 |
+
if marker_written:
|
| 245 |
+
try:
|
| 246 |
+
self._api.write_files(
|
| 247 |
+
bucket_id=bucket,
|
| 248 |
+
token=token_value,
|
| 249 |
+
files={},
|
| 250 |
+
delete_paths=[diagnostic_path],
|
| 251 |
+
)
|
| 252 |
+
except Exception as exc:
|
| 253 |
+
raise HfBucketValidationError(
|
| 254 |
+
"Private Bucket access was verified, but the diagnostic marker could not be removed: "
|
| 255 |
+
f"{exc}"
|
| 256 |
+
) from exc
|
| 257 |
+
|
| 258 |
+
return HfBucketPermissionProbeResult(
|
| 259 |
+
bucket_id=bucket,
|
| 260 |
+
actor_username=actor,
|
| 261 |
+
diagnostic_path=diagnostic_path,
|
| 262 |
+
verified_at=verified_at,
|
| 263 |
+
)
|
| 264 |
+
|
| 265 |
+
|
| 266 |
def _json_bytes(payload: object) -> bytes:
|
| 267 |
return json.dumps(payload, ensure_ascii=True, indent=2, sort_keys=True).encode("utf-8")
|
| 268 |
|
src/ui/app_factory.py
CHANGED
|
@@ -26,7 +26,9 @@ from src.repositories.append_only_validation_repository import AppendOnlyValidat
|
|
| 26 |
from src.repositories.in_memory_detection_repository import InMemoryDetectionRepository
|
| 27 |
from src.repositories.hf_bucket_validation_repository import (
|
| 28 |
HF_BUCKET_VALIDATION_BACKEND,
|
|
|
|
| 29 |
HfBucketValidationError,
|
|
|
|
| 30 |
)
|
| 31 |
from src.repositories.project_aware_validation_repository import ProjectAwareValidationRepository
|
| 32 |
from src.repositories.state_safety import (
|
|
@@ -1559,6 +1561,67 @@ def _is_running_in_hf_space() -> bool:
|
|
| 1559 |
return bool((os.getenv("SPACE_ID") or os.getenv("SPACE_HOST") or "").strip())
|
| 1560 |
|
| 1561 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1562 |
def _resolve_username_login_policy(runtime_config: RuntimeConfig) -> tuple[bool, str, str]:
|
| 1563 |
mode = (runtime_config.auth_mode or "auto").strip().lower()
|
| 1564 |
if mode == "username":
|
|
@@ -2904,6 +2967,8 @@ def create_app() -> gr.Blocks:
|
|
| 2904 |
|
| 2905 |
projects_file_path, user_access_file_path, invites_file_path = _resolve_bootstrap_file_paths(runtime_config)
|
| 2906 |
state_store, supabase_validation_repository, state_backend_message = _build_supabase_state(runtime_config)
|
|
|
|
|
|
|
| 2907 |
allow_username_login, auth_mode_label, auth_mode_description = _resolve_username_login_policy(runtime_config)
|
| 2908 |
bootstrap_warning = _bootstrap_auth_and_projects(
|
| 2909 |
auth_service,
|
|
@@ -2938,6 +3003,8 @@ def create_app() -> gr.Blocks:
|
|
| 2938 |
max_loaded_projects = 3
|
| 2939 |
audio_service = AudioFetchService(EphemeralCacheManager(ttl_seconds=300, max_files=128))
|
| 2940 |
base_validation_repository = supabase_validation_repository or AppendOnlyValidationRepository(base_dir=runtime_config.validation_base_dir)
|
|
|
|
|
|
|
| 2941 |
validation_repository = ProjectAwareValidationRepository(
|
| 2942 |
fallback_repository=base_validation_repository,
|
| 2943 |
project_lookup=admin_manager.get_project,
|
|
@@ -2948,11 +3015,13 @@ def create_app() -> gr.Blocks:
|
|
| 2948 |
or None
|
| 2949 |
),
|
| 2950 |
actor_token_provider=lambda _project, username: auth_service.get_hf_token_for_user(username),
|
| 2951 |
-
enable_hf_project_state=
|
| 2952 |
-
enable_hf_bucket_validations=
|
| 2953 |
)
|
| 2954 |
validation_service = ValidationService(validation_repository)
|
| 2955 |
hf_state_initializer = HfProjectStateStoreInitializer()
|
|
|
|
|
|
|
| 2956 |
hf_state_connector = HfProjectStateStoreConnector()
|
| 2957 |
hf_state_permission_probe = HfProjectStatePermissionProbe()
|
| 2958 |
hf_state_sync = HfProjectStateStoreSync()
|
|
@@ -3124,7 +3193,7 @@ def create_app() -> gr.Blocks:
|
|
| 3124 |
sync_project_slugs: set[str] | None = None,
|
| 3125 |
archived_projects: list[Project] | None = None,
|
| 3126 |
) -> list[str]:
|
| 3127 |
-
if not
|
| 3128 |
return []
|
| 3129 |
|
| 3130 |
user_access = auth_service.export_user_access_map(include_inactive=True)
|
|
@@ -3214,14 +3283,24 @@ def create_app() -> gr.Blocks:
|
|
| 3214 |
section_header_html(
|
| 3215 |
"Secure access",
|
| 3216 |
"Sign in with your Hugging Face identity",
|
| 3217 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3218 |
)
|
| 3219 |
)
|
| 3220 |
username_input, session_output, login_button, error_message = create_login_page(
|
| 3221 |
auth_service,
|
| 3222 |
allow_username_login=allow_username_login,
|
| 3223 |
enable_oauth_login=_is_running_in_hf_space(),
|
| 3224 |
-
auth_mode_label=
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3225 |
)
|
| 3226 |
|
| 3227 |
# Store session ID when login succeeds
|
|
@@ -3300,6 +3379,16 @@ def create_app() -> gr.Blocks:
|
|
| 3300 |
class_name="bn-panel-soft",
|
| 3301 |
)
|
| 3302 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3303 |
with gr.Row():
|
| 3304 |
create_project_slug = gr.Textbox(
|
| 3305 |
label="New Project Slug",
|
|
@@ -3319,7 +3408,11 @@ def create_app() -> gr.Blocks:
|
|
| 3319 |
value="collaborative",
|
| 3320 |
)
|
| 3321 |
create_project_token = gr.Textbox(
|
| 3322 |
-
label=
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3323 |
placeholder="hf_xxx...",
|
| 3324 |
type="password",
|
| 3325 |
)
|
|
@@ -3373,7 +3466,8 @@ def create_app() -> gr.Blocks:
|
|
| 3373 |
return "Visibility must be private or collaborative.", gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), session
|
| 3374 |
if (
|
| 3375 |
_is_running_in_hf_space()
|
| 3376 |
-
and
|
|
|
|
| 3377 |
and getattr(session, "authentication_method", "") != "oauth"
|
| 3378 |
):
|
| 3379 |
return (
|
|
@@ -3417,9 +3511,33 @@ def create_app() -> gr.Blocks:
|
|
| 3417 |
dataset_repo_id=repo_id,
|
| 3418 |
visibility=visibility_value,
|
| 3419 |
owner_username=session.username,
|
| 3420 |
-
dataset_token=explicit_project_token,
|
| 3421 |
active=True,
|
| 3422 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3423 |
try:
|
| 3424 |
state_result = hf_state_initializer.initialize(
|
| 3425 |
project=project,
|
|
@@ -3466,6 +3584,17 @@ def create_app() -> gr.Blocks:
|
|
| 3466 |
project.state_schema_version = HF_PROJECT_STATE_SCHEMA_VERSION
|
| 3467 |
project.state_status = "ready"
|
| 3468 |
state_repo_action = "initialized" if state_result.initialized else "connected"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3469 |
|
| 3470 |
created = admin_manager.register_project(project)
|
| 3471 |
if not created:
|
|
@@ -3499,9 +3628,9 @@ def create_app() -> gr.Blocks:
|
|
| 3499 |
|
| 3500 |
return (
|
| 3501 |
(
|
| 3502 |
-
f"Project '{slug}' created successfully. Private state repo {state_repo_action}: {state_result.state_repo_id}."
|
| 3503 |
if persisted
|
| 3504 |
-
else f"Project '{slug}' created with private state repo {state_repo_action} ({state_result.state_repo_id}). Could not persist bootstrap files: {persist_error}"
|
| 3505 |
),
|
| 3506 |
_project_rows(),
|
| 3507 |
gr.update(choices=admin_projects, value=slug),
|
|
@@ -3560,6 +3689,39 @@ def create_app() -> gr.Blocks:
|
|
| 3560 |
session,
|
| 3561 |
)
|
| 3562 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3563 |
slug = loaded.project.project_slug
|
| 3564 |
if admin_manager.get_project(slug) is not None:
|
| 3565 |
return (
|
|
@@ -3703,6 +3865,18 @@ def create_app() -> gr.Blocks:
|
|
| 3703 |
project = admin_manager.get_project(project_slug)
|
| 3704 |
if project is None:
|
| 3705 |
return "Select a valid project.", gr.update(), gr.update(), gr.update(), gr.update()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3706 |
|
| 3707 |
if bool(clear_token):
|
| 3708 |
project.dataset_token = None
|
|
@@ -4118,7 +4292,7 @@ def create_app() -> gr.Blocks:
|
|
| 4118 |
|
| 4119 |
session_state.change(
|
| 4120 |
fn=lambda s: (
|
| 4121 |
-
gr.update(visible=bool(s is not None)),
|
| 4122 |
gr.update(visible=bool(s is not None)),
|
| 4123 |
gr.update(visible=bool(s is not None)),
|
| 4124 |
gr.update(visible=bool(s is not None)),
|
|
@@ -4172,7 +4346,10 @@ def create_app() -> gr.Blocks:
|
|
| 4172 |
interactive=False,
|
| 4173 |
allow_custom_value=True,
|
| 4174 |
)
|
| 4175 |
-
with gr.Group(
|
|
|
|
|
|
|
|
|
|
| 4176 |
gr.Markdown("### Private state OAuth diagnostic")
|
| 4177 |
state_authorization_status = gr.Markdown(
|
| 4178 |
"The current OAuth scope may allow Pull Requests rather than direct state writes. "
|
|
@@ -4182,6 +4359,19 @@ def create_app() -> gr.Blocks:
|
|
| 4182 |
"Check OAuth state write capability",
|
| 4183 |
elem_classes=["bn-soft-action"],
|
| 4184 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4185 |
invitations_info = gr.Markdown(value="")
|
| 4186 |
invitations_overview = gr.HTML(value=invite_panel_html(0))
|
| 4187 |
invite_selector = gr.Dropdown(choices=[], label="Pending Invites", interactive=False)
|
|
@@ -4435,6 +4625,28 @@ def create_app() -> gr.Blocks:
|
|
| 4435 |
f"A private diagnostic write was confirmed in `{result.state_repo_id}`."
|
| 4436 |
)
|
| 4437 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4438 |
project_selector.change(
|
| 4439 |
fn=update_selected_project,
|
| 4440 |
inputs=[project_selector, session_state],
|
|
@@ -4446,6 +4658,11 @@ def create_app() -> gr.Blocks:
|
|
| 4446 |
inputs=[project_selector, session_state],
|
| 4447 |
outputs=[state_authorization_status],
|
| 4448 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4449 |
|
| 4450 |
create_project_event.then(
|
| 4451 |
fn=update_project_selector,
|
|
@@ -5793,11 +6010,18 @@ def create_app() -> gr.Blocks:
|
|
| 5793 |
def _render_settings_health():
|
| 5794 |
backend = (runtime_config.state_backend or "filesystem").strip().lower()
|
| 5795 |
supabase_ready = bool(runtime_config.supabase_url and runtime_config.supabase_service_role_key)
|
| 5796 |
-
|
| 5797 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5798 |
normalized_bootstrap_dir = str(runtime_config.bootstrap_base_dir).replace("\\", "/").rstrip("/")
|
| 5799 |
filesystem_durable = normalized_bootstrap_dir == "/data" or normalized_bootstrap_dir.startswith("/data/")
|
| 5800 |
-
if state_label == "
|
|
|
|
|
|
|
|
|
|
| 5801 |
durability_label = "external durable"
|
| 5802 |
durability_tone = "ok"
|
| 5803 |
elif filesystem_durable:
|
|
@@ -5809,16 +6033,24 @@ def create_app() -> gr.Blocks:
|
|
| 5809 |
demo_label = "enabled" if runtime_config.enable_demo_bootstrap else "disabled"
|
| 5810 |
demo_tone = "warn" if runtime_config.enable_demo_bootstrap else "ok"
|
| 5811 |
invite_email_label = "enabled" if runtime_config.invite_email_enabled and runtime_config.emailjs_enabled else "disabled"
|
| 5812 |
-
hf_project_state_label = "enabled" if
|
| 5813 |
-
hf_bucket_validation_label =
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5814 |
hf_project_state_repos_label = str(len(runtime_config.hf_project_state_repos))
|
| 5815 |
hf_space_label = os.getenv("SPACE_ID") or "local runtime"
|
| 5816 |
health_html = settings_health_html(
|
| 5817 |
[
|
| 5818 |
("State backend", state_label, state_tone),
|
| 5819 |
-
("HF project-state writes", hf_project_state_label, "ok" if
|
| 5820 |
-
("HF Bucket validations", hf_bucket_validation_label, "warn" if runtime_config.hf_bucket_validations_enabled else "info"),
|
| 5821 |
-
(
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5822 |
("Auth mode", auth_mode_label, "ok" if not allow_username_login else "warn"),
|
| 5823 |
("Destructive write guard", "enabled", "ok"),
|
| 5824 |
("Local JSON backups", "enabled" if state_label == "Filesystem" else "not used", "ok" if state_label == "Filesystem" else "info"),
|
|
@@ -5833,8 +6065,18 @@ def create_app() -> gr.Blocks:
|
|
| 5833 |
]
|
| 5834 |
)
|
| 5835 |
status_text = (
|
| 5836 |
-
|
| 5837 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5838 |
else (
|
| 5839 |
"Filesystem persistence is active with local JSON backups and destructive-write guards. "
|
| 5840 |
"On free Spaces without persistent storage, local files are still not durable across rebuilds."
|
|
|
|
| 26 |
from src.repositories.in_memory_detection_repository import InMemoryDetectionRepository
|
| 27 |
from src.repositories.hf_bucket_validation_repository import (
|
| 28 |
HF_BUCKET_VALIDATION_BACKEND,
|
| 29 |
+
HfBucketPermissionProbe,
|
| 30 |
HfBucketValidationError,
|
| 31 |
+
HfBucketValidationInitializer,
|
| 32 |
)
|
| 33 |
from src.repositories.project_aware_validation_repository import ProjectAwareValidationRepository
|
| 34 |
from src.repositories.state_safety import (
|
|
|
|
| 1561 |
return bool((os.getenv("SPACE_ID") or os.getenv("SPACE_HOST") or "").strip())
|
| 1562 |
|
| 1563 |
|
| 1564 |
+
def _validate_hf_trusted_team_source_dataset(
|
| 1565 |
+
*,
|
| 1566 |
+
project: Project,
|
| 1567 |
+
token: str | None,
|
| 1568 |
+
api: HfApi | None = None,
|
| 1569 |
+
) -> None:
|
| 1570 |
+
if (project.visibility or "").strip().lower() != "collaborative":
|
| 1571 |
+
raise HfBucketValidationError(
|
| 1572 |
+
"Trusted-team projects must use app visibility 'collaborative'. "
|
| 1573 |
+
"The Hugging Face dataset and storage remain private; collaborative visibility enables approved "
|
| 1574 |
+
"team members inside this app."
|
| 1575 |
+
)
|
| 1576 |
+
token_value = (token or "").strip()
|
| 1577 |
+
if not token_value:
|
| 1578 |
+
raise HfBucketValidationError("Trusted-team HF storage requires the administrator's Hugging Face authorization during project setup.")
|
| 1579 |
+
|
| 1580 |
+
client = api or HfApi()
|
| 1581 |
+
namespace = project.dataset_repo_id.split("/", 1)[0].strip()
|
| 1582 |
+
try:
|
| 1583 |
+
client.get_organization_overview(namespace, token=token_value)
|
| 1584 |
+
except Exception as exc:
|
| 1585 |
+
raise HfBucketValidationError(
|
| 1586 |
+
"Trusted-team HF storage requires the private dataset to be owned by a dedicated Hugging Face "
|
| 1587 |
+
f"organization. Could not verify organization '{namespace}': {exc}"
|
| 1588 |
+
) from exc
|
| 1589 |
+
try:
|
| 1590 |
+
dataset_info = client.repo_info(
|
| 1591 |
+
repo_id=project.dataset_repo_id,
|
| 1592 |
+
repo_type="dataset",
|
| 1593 |
+
token=token_value,
|
| 1594 |
+
)
|
| 1595 |
+
except Exception as exc:
|
| 1596 |
+
raise HfBucketValidationError(f"Could not verify the Hugging Face dataset before creating private state: {exc}") from exc
|
| 1597 |
+
if not bool(getattr(dataset_info, "private", False)):
|
| 1598 |
+
raise HfBucketValidationError(
|
| 1599 |
+
"Trusted-team HF storage only accepts a private dataset. Make the dataset private in Hugging Face "
|
| 1600 |
+
"before creating the validation project."
|
| 1601 |
+
)
|
| 1602 |
+
|
| 1603 |
+
|
| 1604 |
+
def _initialize_hf_trusted_team_storage(
|
| 1605 |
+
*,
|
| 1606 |
+
project: Project,
|
| 1607 |
+
token: str | None,
|
| 1608 |
+
api: HfApi | None = None,
|
| 1609 |
+
bucket_initializer: HfBucketValidationInitializer | None = None,
|
| 1610 |
+
):
|
| 1611 |
+
_validate_hf_trusted_team_source_dataset(project=project, token=token, api=api)
|
| 1612 |
+
token_value = (token or "").strip()
|
| 1613 |
+
initializer = bucket_initializer or HfBucketValidationInitializer()
|
| 1614 |
+
result = initializer.initialize(
|
| 1615 |
+
project_slug=project.project_slug,
|
| 1616 |
+
dataset_repo_id=project.dataset_repo_id,
|
| 1617 |
+
token=token_value,
|
| 1618 |
+
)
|
| 1619 |
+
project.dataset_token = None
|
| 1620 |
+
project.validation_backend = HF_BUCKET_VALIDATION_BACKEND
|
| 1621 |
+
project.validation_bucket_id = result.bucket_id
|
| 1622 |
+
return result
|
| 1623 |
+
|
| 1624 |
+
|
| 1625 |
def _resolve_username_login_policy(runtime_config: RuntimeConfig) -> tuple[bool, str, str]:
|
| 1626 |
mode = (runtime_config.auth_mode or "auto").strip().lower()
|
| 1627 |
if mode == "username":
|
|
|
|
| 2967 |
|
| 2968 |
projects_file_path, user_access_file_path, invites_file_path = _resolve_bootstrap_file_paths(runtime_config)
|
| 2969 |
state_store, supabase_validation_repository, state_backend_message = _build_supabase_state(runtime_config)
|
| 2970 |
+
if runtime_config.hf_trusted_team_mode_enabled:
|
| 2971 |
+
state_backend_message = "HF trusted-team project storage enabled."
|
| 2972 |
allow_username_login, auth_mode_label, auth_mode_description = _resolve_username_login_policy(runtime_config)
|
| 2973 |
bootstrap_warning = _bootstrap_auth_and_projects(
|
| 2974 |
auth_service,
|
|
|
|
| 3003 |
max_loaded_projects = 3
|
| 3004 |
audio_service = AudioFetchService(EphemeralCacheManager(ttl_seconds=300, max_files=128))
|
| 3005 |
base_validation_repository = supabase_validation_repository or AppendOnlyValidationRepository(base_dir=runtime_config.validation_base_dir)
|
| 3006 |
+
hf_state_writes_enabled = runtime_config.hf_project_state_writes_enabled or runtime_config.hf_trusted_team_mode_enabled
|
| 3007 |
+
hf_bucket_validations_enabled = runtime_config.hf_bucket_validations_enabled or runtime_config.hf_trusted_team_mode_enabled
|
| 3008 |
validation_repository = ProjectAwareValidationRepository(
|
| 3009 |
fallback_repository=base_validation_repository,
|
| 3010 |
project_lookup=admin_manager.get_project,
|
|
|
|
| 3015 |
or None
|
| 3016 |
),
|
| 3017 |
actor_token_provider=lambda _project, username: auth_service.get_hf_token_for_user(username),
|
| 3018 |
+
enable_hf_project_state=hf_state_writes_enabled,
|
| 3019 |
+
enable_hf_bucket_validations=hf_bucket_validations_enabled,
|
| 3020 |
)
|
| 3021 |
validation_service = ValidationService(validation_repository)
|
| 3022 |
hf_state_initializer = HfProjectStateStoreInitializer()
|
| 3023 |
+
hf_bucket_initializer = HfBucketValidationInitializer()
|
| 3024 |
+
hf_bucket_permission_probe = HfBucketPermissionProbe()
|
| 3025 |
hf_state_connector = HfProjectStateStoreConnector()
|
| 3026 |
hf_state_permission_probe = HfProjectStatePermissionProbe()
|
| 3027 |
hf_state_sync = HfProjectStateStoreSync()
|
|
|
|
| 3193 |
sync_project_slugs: set[str] | None = None,
|
| 3194 |
archived_projects: list[Project] | None = None,
|
| 3195 |
) -> list[str]:
|
| 3196 |
+
if not hf_state_writes_enabled:
|
| 3197 |
return []
|
| 3198 |
|
| 3199 |
user_access = auth_service.export_user_access_map(include_inactive=True)
|
|
|
|
| 3283 |
section_header_html(
|
| 3284 |
"Secure access",
|
| 3285 |
"Sign in with your Hugging Face identity",
|
| 3286 |
+
(
|
| 3287 |
+
"Each trusted collaborator accesses private data and validation storage with their own Hugging Face token."
|
| 3288 |
+
if runtime_config.hf_trusted_team_mode_enabled
|
| 3289 |
+
else "Your account controls project access, validator attribution, and private dataset access."
|
| 3290 |
+
),
|
| 3291 |
)
|
| 3292 |
)
|
| 3293 |
username_input, session_output, login_button, error_message = create_login_page(
|
| 3294 |
auth_service,
|
| 3295 |
allow_username_login=allow_username_login,
|
| 3296 |
enable_oauth_login=_is_running_in_hf_space(),
|
| 3297 |
+
auth_mode_label=(
|
| 3298 |
+
"Private HF trusted-team mode is active. Personal token login is the validated workflow; "
|
| 3299 |
+
"project data remain private to authorized Hugging Face organization members."
|
| 3300 |
+
if runtime_config.hf_trusted_team_mode_enabled
|
| 3301 |
+
else auth_mode_description
|
| 3302 |
+
),
|
| 3303 |
+
trusted_team_mode=runtime_config.hf_trusted_team_mode_enabled,
|
| 3304 |
)
|
| 3305 |
|
| 3306 |
# Store session ID when login succeeds
|
|
|
|
| 3379 |
class_name="bn-panel-soft",
|
| 3380 |
)
|
| 3381 |
)
|
| 3382 |
+
if runtime_config.hf_trusted_team_mode_enabled:
|
| 3383 |
+
gr.HTML(
|
| 3384 |
+
inline_hint_html(
|
| 3385 |
+
"Private HF trusted-team mode is active. Source datasets must be private. "
|
| 3386 |
+
"Choose collaborative visibility; the data remain private on Hugging Face. Before "
|
| 3387 |
+
"assigning validators here, add each trusted collaborator with write access to the "
|
| 3388 |
+
"project's dedicated Hugging Face organization.",
|
| 3389 |
+
"info",
|
| 3390 |
+
)
|
| 3391 |
+
)
|
| 3392 |
with gr.Row():
|
| 3393 |
create_project_slug = gr.Textbox(
|
| 3394 |
label="New Project Slug",
|
|
|
|
| 3408 |
value="collaborative",
|
| 3409 |
)
|
| 3410 |
create_project_token = gr.Textbox(
|
| 3411 |
+
label=(
|
| 3412 |
+
"One-time setup HF token (not stored)"
|
| 3413 |
+
if runtime_config.hf_trusted_team_mode_enabled
|
| 3414 |
+
else "Project HF Token (optional)"
|
| 3415 |
+
),
|
| 3416 |
placeholder="hf_xxx...",
|
| 3417 |
type="password",
|
| 3418 |
)
|
|
|
|
| 3466 |
return "Visibility must be private or collaborative.", gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), session
|
| 3467 |
if (
|
| 3468 |
_is_running_in_hf_space()
|
| 3469 |
+
and hf_state_writes_enabled
|
| 3470 |
+
and not runtime_config.hf_trusted_team_mode_enabled
|
| 3471 |
and getattr(session, "authentication_method", "") != "oauth"
|
| 3472 |
):
|
| 3473 |
return (
|
|
|
|
| 3511 |
dataset_repo_id=repo_id,
|
| 3512 |
visibility=visibility_value,
|
| 3513 |
owner_username=session.username,
|
| 3514 |
+
dataset_token=None if runtime_config.hf_trusted_team_mode_enabled else explicit_project_token,
|
| 3515 |
active=True,
|
| 3516 |
)
|
| 3517 |
+
bucket_result = None
|
| 3518 |
+
if runtime_config.hf_trusted_team_mode_enabled:
|
| 3519 |
+
try:
|
| 3520 |
+
bucket_result = _initialize_hf_trusted_team_storage(
|
| 3521 |
+
project=project,
|
| 3522 |
+
token=operation_token,
|
| 3523 |
+
bucket_initializer=hf_bucket_initializer,
|
| 3524 |
+
)
|
| 3525 |
+
except HfBucketValidationError as exc:
|
| 3526 |
+
return (
|
| 3527 |
+
f"Project was not created because its private validation Bucket could not be initialized: {exc}",
|
| 3528 |
+
_project_rows(),
|
| 3529 |
+
gr.update(),
|
| 3530 |
+
gr.update(),
|
| 3531 |
+
gr.update(),
|
| 3532 |
+
gr.update(),
|
| 3533 |
+
gr.update(),
|
| 3534 |
+
gr.update(),
|
| 3535 |
+
gr.update(),
|
| 3536 |
+
gr.update(),
|
| 3537 |
+
gr.update(),
|
| 3538 |
+
gr.update(),
|
| 3539 |
+
session,
|
| 3540 |
+
)
|
| 3541 |
try:
|
| 3542 |
state_result = hf_state_initializer.initialize(
|
| 3543 |
project=project,
|
|
|
|
| 3584 |
project.state_schema_version = HF_PROJECT_STATE_SCHEMA_VERSION
|
| 3585 |
project.state_status = "ready"
|
| 3586 |
state_repo_action = "initialized" if state_result.initialized else "connected"
|
| 3587 |
+
bucket_message = (
|
| 3588 |
+
f" Private validation Bucket ready: {bucket_result.bucket_id}."
|
| 3589 |
+
if bucket_result is not None
|
| 3590 |
+
else ""
|
| 3591 |
+
)
|
| 3592 |
+
recovery_message = (
|
| 3593 |
+
f" Add `{state_result.state_repo_id}` to `BIRDNET_HF_PROJECT_STATE_REPOS` before rebuilding "
|
| 3594 |
+
"the Space so this project is restored after restart."
|
| 3595 |
+
if runtime_config.hf_trusted_team_mode_enabled
|
| 3596 |
+
else ""
|
| 3597 |
+
)
|
| 3598 |
|
| 3599 |
created = admin_manager.register_project(project)
|
| 3600 |
if not created:
|
|
|
|
| 3628 |
|
| 3629 |
return (
|
| 3630 |
(
|
| 3631 |
+
f"Project '{slug}' created successfully. Private state repo {state_repo_action}: {state_result.state_repo_id}.{bucket_message}{recovery_message}"
|
| 3632 |
if persisted
|
| 3633 |
+
else f"Project '{slug}' created with private state repo {state_repo_action} ({state_result.state_repo_id}).{bucket_message}{recovery_message} Could not persist bootstrap files: {persist_error}"
|
| 3634 |
),
|
| 3635 |
_project_rows(),
|
| 3636 |
gr.update(choices=admin_projects, value=slug),
|
|
|
|
| 3689 |
session,
|
| 3690 |
)
|
| 3691 |
|
| 3692 |
+
if runtime_config.hf_trusted_team_mode_enabled:
|
| 3693 |
+
if (
|
| 3694 |
+
(loaded.project.validation_backend or "").strip() != HF_BUCKET_VALIDATION_BACKEND
|
| 3695 |
+
or not (loaded.project.validation_bucket_id or "").strip()
|
| 3696 |
+
):
|
| 3697 |
+
return (
|
| 3698 |
+
"Project state could not be connected in trusted-team mode because its manifest "
|
| 3699 |
+
"does not declare private Hugging Face Bucket validation storage.",
|
| 3700 |
+
_project_rows(),
|
| 3701 |
+
gr.update(),
|
| 3702 |
+
gr.update(),
|
| 3703 |
+
gr.update(),
|
| 3704 |
+
gr.update(),
|
| 3705 |
+
gr.update(),
|
| 3706 |
+
session,
|
| 3707 |
+
)
|
| 3708 |
+
try:
|
| 3709 |
+
_validate_hf_trusted_team_source_dataset(
|
| 3710 |
+
project=loaded.project,
|
| 3711 |
+
token=_session_hf_token(session),
|
| 3712 |
+
)
|
| 3713 |
+
except HfBucketValidationError as exc:
|
| 3714 |
+
return (
|
| 3715 |
+
f"Project state could not be connected in trusted-team mode: {exc}",
|
| 3716 |
+
_project_rows(),
|
| 3717 |
+
gr.update(),
|
| 3718 |
+
gr.update(),
|
| 3719 |
+
gr.update(),
|
| 3720 |
+
gr.update(),
|
| 3721 |
+
gr.update(),
|
| 3722 |
+
session,
|
| 3723 |
+
)
|
| 3724 |
+
|
| 3725 |
slug = loaded.project.project_slug
|
| 3726 |
if admin_manager.get_project(slug) is not None:
|
| 3727 |
return (
|
|
|
|
| 3865 |
project = admin_manager.get_project(project_slug)
|
| 3866 |
if project is None:
|
| 3867 |
return "Select a valid project.", gr.update(), gr.update(), gr.update(), gr.update()
|
| 3868 |
+
if (
|
| 3869 |
+
runtime_config.hf_trusted_team_mode_enabled
|
| 3870 |
+
and (project.validation_backend or "").strip() == HF_BUCKET_VALIDATION_BACKEND
|
| 3871 |
+
):
|
| 3872 |
+
return (
|
| 3873 |
+
"Trusted-team HF projects never store a shared project token. Each collaborator must sign "
|
| 3874 |
+
"in using their own Hugging Face authorization.",
|
| 3875 |
+
gr.update(value=""),
|
| 3876 |
+
gr.update(value=False),
|
| 3877 |
+
_project_rows(),
|
| 3878 |
+
"",
|
| 3879 |
+
)
|
| 3880 |
|
| 3881 |
if bool(clear_token):
|
| 3882 |
project.dataset_token = None
|
|
|
|
| 4292 |
|
| 4293 |
session_state.change(
|
| 4294 |
fn=lambda s: (
|
| 4295 |
+
gr.update(visible=bool(s is not None and not runtime_config.hf_trusted_team_mode_enabled)),
|
| 4296 |
gr.update(visible=bool(s is not None)),
|
| 4297 |
gr.update(visible=bool(s is not None)),
|
| 4298 |
gr.update(visible=bool(s is not None)),
|
|
|
|
| 4346 |
interactive=False,
|
| 4347 |
allow_custom_value=True,
|
| 4348 |
)
|
| 4349 |
+
with gr.Group(
|
| 4350 |
+
visible=not runtime_config.hf_trusted_team_mode_enabled,
|
| 4351 |
+
elem_classes=["bn-panel-soft"],
|
| 4352 |
+
):
|
| 4353 |
gr.Markdown("### Private state OAuth diagnostic")
|
| 4354 |
state_authorization_status = gr.Markdown(
|
| 4355 |
"The current OAuth scope may allow Pull Requests rather than direct state writes. "
|
|
|
|
| 4359 |
"Check OAuth state write capability",
|
| 4360 |
elem_classes=["bn-soft-action"],
|
| 4361 |
)
|
| 4362 |
+
with gr.Group(
|
| 4363 |
+
visible=runtime_config.hf_trusted_team_mode_enabled,
|
| 4364 |
+
elem_classes=["bn-panel-soft"],
|
| 4365 |
+
):
|
| 4366 |
+
gr.Markdown("### Private project storage access")
|
| 4367 |
+
trusted_storage_status = gr.Markdown(
|
| 4368 |
+
"Run this check once for each trusted collaborator before validation. "
|
| 4369 |
+
"It verifies private validation storage write access with that user's own Hugging Face token."
|
| 4370 |
+
)
|
| 4371 |
+
test_trusted_storage_btn = gr.Button(
|
| 4372 |
+
"Check private validation storage access",
|
| 4373 |
+
elem_classes=["bn-soft-action"],
|
| 4374 |
+
)
|
| 4375 |
invitations_info = gr.Markdown(value="")
|
| 4376 |
invitations_overview = gr.HTML(value=invite_panel_html(0))
|
| 4377 |
invite_selector = gr.Dropdown(choices=[], label="Pending Invites", interactive=False)
|
|
|
|
| 4625 |
f"A private diagnostic write was confirmed in `{result.state_repo_id}`."
|
| 4626 |
)
|
| 4627 |
|
| 4628 |
+
def test_trusted_storage_authorization(selected: str, session):
|
| 4629 |
+
if session is None:
|
| 4630 |
+
return "Login with your Hugging Face account before testing private validation storage."
|
| 4631 |
+
project_slug = (selected or "").strip()
|
| 4632 |
+
project = admin_manager.get_project(project_slug) if project_slug else None
|
| 4633 |
+
if project is None or project_slug not in session.authorized_projects:
|
| 4634 |
+
return "Select a project that is authorized for the signed-in account."
|
| 4635 |
+
if (project.validation_backend or "").strip() != HF_BUCKET_VALIDATION_BACKEND:
|
| 4636 |
+
return "This project does not use private Hugging Face Bucket validation storage."
|
| 4637 |
+
try:
|
| 4638 |
+
result = hf_bucket_permission_probe.probe(
|
| 4639 |
+
bucket_id=project.validation_bucket_id or "",
|
| 4640 |
+
actor_username=session.username,
|
| 4641 |
+
token=_session_hf_token(session),
|
| 4642 |
+
)
|
| 4643 |
+
except HfBucketValidationError as exc:
|
| 4644 |
+
return f"Private storage access test failed: {exc}"
|
| 4645 |
+
return (
|
| 4646 |
+
f"Private validation storage verified for **{result.actor_username}**. "
|
| 4647 |
+
f"Read/write access was confirmed in `{result.bucket_id}` and the diagnostic marker was removed."
|
| 4648 |
+
)
|
| 4649 |
+
|
| 4650 |
project_selector.change(
|
| 4651 |
fn=update_selected_project,
|
| 4652 |
inputs=[project_selector, session_state],
|
|
|
|
| 4658 |
inputs=[project_selector, session_state],
|
| 4659 |
outputs=[state_authorization_status],
|
| 4660 |
)
|
| 4661 |
+
test_trusted_storage_btn.click(
|
| 4662 |
+
fn=test_trusted_storage_authorization,
|
| 4663 |
+
inputs=[project_selector, session_state],
|
| 4664 |
+
outputs=[trusted_storage_status],
|
| 4665 |
+
)
|
| 4666 |
|
| 4667 |
create_project_event.then(
|
| 4668 |
fn=update_project_selector,
|
|
|
|
| 6010 |
def _render_settings_health():
|
| 6011 |
backend = (runtime_config.state_backend or "filesystem").strip().lower()
|
| 6012 |
supabase_ready = bool(runtime_config.supabase_url and runtime_config.supabase_service_role_key)
|
| 6013 |
+
if runtime_config.hf_trusted_team_mode_enabled:
|
| 6014 |
+
state_label = "HF trusted-team"
|
| 6015 |
+
state_tone = "ok"
|
| 6016 |
+
else:
|
| 6017 |
+
state_label = "Supabase" if backend in {"supabase", "postgres", "postgresql"} and supabase_ready else "Filesystem"
|
| 6018 |
+
state_tone = "ok" if state_label == "Supabase" else "warn"
|
| 6019 |
normalized_bootstrap_dir = str(runtime_config.bootstrap_base_dir).replace("\\", "/").rstrip("/")
|
| 6020 |
filesystem_durable = normalized_bootstrap_dir == "/data" or normalized_bootstrap_dir.startswith("/data/")
|
| 6021 |
+
if state_label == "HF trusted-team":
|
| 6022 |
+
durability_label = "private HF state/Bucket"
|
| 6023 |
+
durability_tone = "ok"
|
| 6024 |
+
elif state_label == "Supabase":
|
| 6025 |
durability_label = "external durable"
|
| 6026 |
durability_tone = "ok"
|
| 6027 |
elif filesystem_durable:
|
|
|
|
| 6033 |
demo_label = "enabled" if runtime_config.enable_demo_bootstrap else "disabled"
|
| 6034 |
demo_tone = "warn" if runtime_config.enable_demo_bootstrap else "ok"
|
| 6035 |
invite_email_label = "enabled" if runtime_config.invite_email_enabled and runtime_config.emailjs_enabled else "disabled"
|
| 6036 |
+
hf_project_state_label = "enabled" if hf_state_writes_enabled else "disabled"
|
| 6037 |
+
hf_bucket_validation_label = (
|
| 6038 |
+
"trusted-team active"
|
| 6039 |
+
if runtime_config.hf_trusted_team_mode_enabled
|
| 6040 |
+
else ("legacy experimental" if runtime_config.hf_bucket_validations_enabled else "disabled")
|
| 6041 |
+
)
|
| 6042 |
hf_project_state_repos_label = str(len(runtime_config.hf_project_state_repos))
|
| 6043 |
hf_space_label = os.getenv("SPACE_ID") or "local runtime"
|
| 6044 |
health_html = settings_health_html(
|
| 6045 |
[
|
| 6046 |
("State backend", state_label, state_tone),
|
| 6047 |
+
("HF project-state writes", hf_project_state_label, "ok" if hf_state_writes_enabled else "info"),
|
| 6048 |
+
("HF Bucket validations", hf_bucket_validation_label, "ok" if runtime_config.hf_trusted_team_mode_enabled else ("warn" if runtime_config.hf_bucket_validations_enabled else "info")),
|
| 6049 |
+
(
|
| 6050 |
+
"HF project-state repos",
|
| 6051 |
+
hf_project_state_repos_label,
|
| 6052 |
+
"ok" if runtime_config.hf_project_state_repos else ("warn" if runtime_config.hf_trusted_team_mode_enabled else "info"),
|
| 6053 |
+
),
|
| 6054 |
("Auth mode", auth_mode_label, "ok" if not allow_username_login else "warn"),
|
| 6055 |
("Destructive write guard", "enabled", "ok"),
|
| 6056 |
("Local JSON backups", "enabled" if state_label == "Filesystem" else "not used", "ok" if state_label == "Filesystem" else "info"),
|
|
|
|
| 6065 |
]
|
| 6066 |
)
|
| 6067 |
status_text = (
|
| 6068 |
+
(
|
| 6069 |
+
"HF trusted-team storage is active. Private dataset access and Bucket writes require each "
|
| 6070 |
+
"trusted collaborator's own Hugging Face permissions; "
|
| 6071 |
+
+ (
|
| 6072 |
+
"no project-state repository is configured, so projects will not be restored after a rebuild."
|
| 6073 |
+
if not runtime_config.hf_project_state_repos
|
| 6074 |
+
else "configured state repositories will be used for restart recovery."
|
| 6075 |
+
)
|
| 6076 |
+
if state_label == "HF trusted-team"
|
| 6077 |
+
else "Supabase persistence is active. Project and ACL removals are guarded against accidental broad overwrites."
|
| 6078 |
+
)
|
| 6079 |
+
if state_label in {"HF trusted-team", "Supabase"}
|
| 6080 |
else (
|
| 6081 |
"Filesystem persistence is active with local JSON backups and destructive-write guards. "
|
| 6082 |
"On free Spaces without persistent storage, local files are still not durable across rebuilds."
|
src/ui/login_page.py
CHANGED
|
@@ -84,6 +84,7 @@ def create_login_page(
|
|
| 84 |
allow_username_login: bool = True,
|
| 85 |
enable_oauth_login: bool = False,
|
| 86 |
auth_mode_label: str = "",
|
|
|
|
| 87 |
) -> Tuple[gr.Textbox, gr.Textbox, gr.Button, gr.Markdown]:
|
| 88 |
"""Create a Gradio login page with username input and session tracking.
|
| 89 |
|
|
@@ -98,34 +99,41 @@ def create_login_page(
|
|
| 98 |
gr.Markdown("")
|
| 99 |
with gr.Column(scale=6):
|
| 100 |
gr.Markdown("# BirdNET Validation Platform")
|
| 101 |
-
gr.Markdown(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 102 |
if auth_mode_label:
|
| 103 |
gr.Markdown(auth_mode_label)
|
| 104 |
|
| 105 |
-
|
| 106 |
-
if enable_oauth_login:
|
| 107 |
gr.Markdown(
|
| 108 |
-
"
|
| 109 |
-
"
|
|
|
|
|
|
|
| 110 |
)
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
"
|
| 114 |
-
|
| 115 |
)
|
| 116 |
-
|
| 117 |
-
gr.Markdown(
|
| 118 |
-
"Manual token access is available for development or legacy access, "
|
| 119 |
-
"but cannot be used for the private collaborative state authorization test."
|
| 120 |
-
)
|
| 121 |
username_input = gr.Textbox(
|
| 122 |
label="Username",
|
| 123 |
placeholder="Enter your username" if allow_username_login else "Username login disabled in this deployment",
|
| 124 |
lines=1,
|
| 125 |
interactive=allow_username_login,
|
|
|
|
| 126 |
)
|
| 127 |
hf_token_input = gr.Textbox(
|
| 128 |
-
label=
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
placeholder="hf_xxx...",
|
| 130 |
type="password",
|
| 131 |
lines=1,
|
|
@@ -133,7 +141,32 @@ def create_login_page(
|
|
| 133 |
|
| 134 |
error_message = gr.Markdown()
|
| 135 |
|
| 136 |
-
login_button = gr.Button(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 137 |
|
| 138 |
session_output = gr.Textbox(
|
| 139 |
label="Session ID",
|
|
|
|
| 84 |
allow_username_login: bool = True,
|
| 85 |
enable_oauth_login: bool = False,
|
| 86 |
auth_mode_label: str = "",
|
| 87 |
+
trusted_team_mode: bool = False,
|
| 88 |
) -> Tuple[gr.Textbox, gr.Textbox, gr.Button, gr.Markdown]:
|
| 89 |
"""Create a Gradio login page with username input and session tracking.
|
| 90 |
|
|
|
|
| 99 |
gr.Markdown("")
|
| 100 |
with gr.Column(scale=6):
|
| 101 |
gr.Markdown("# BirdNET Validation Platform")
|
| 102 |
+
gr.Markdown(
|
| 103 |
+
(
|
| 104 |
+
"Use your own Hugging Face authorization to access private project validation workflows."
|
| 105 |
+
if trusted_team_mode
|
| 106 |
+
else "Sign in with your Hugging Face account to access project validation workflows."
|
| 107 |
+
)
|
| 108 |
+
)
|
| 109 |
if auth_mode_label:
|
| 110 |
gr.Markdown(auth_mode_label)
|
| 111 |
|
| 112 |
+
if trusted_team_mode:
|
|
|
|
| 113 |
gr.Markdown(
|
| 114 |
+
"### Personal token access\n"
|
| 115 |
+
"This is the validated access method for private trusted-team projects. "
|
| 116 |
+
"Use a token from your own account with permission to read the project's private dataset "
|
| 117 |
+
"and write to its private validation Bucket."
|
| 118 |
)
|
| 119 |
+
else:
|
| 120 |
+
gr.Markdown(
|
| 121 |
+
"Manual token access is available for development or legacy access, "
|
| 122 |
+
"but cannot be used for the private collaborative state authorization test."
|
| 123 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
username_input = gr.Textbox(
|
| 125 |
label="Username",
|
| 126 |
placeholder="Enter your username" if allow_username_login else "Username login disabled in this deployment",
|
| 127 |
lines=1,
|
| 128 |
interactive=allow_username_login,
|
| 129 |
+
visible=allow_username_login,
|
| 130 |
)
|
| 131 |
hf_token_input = gr.Textbox(
|
| 132 |
+
label=(
|
| 133 |
+
"Personal Hugging Face Token"
|
| 134 |
+
if trusted_team_mode
|
| 135 |
+
else "Hugging Face Token (legacy/development only)"
|
| 136 |
+
),
|
| 137 |
placeholder="hf_xxx...",
|
| 138 |
type="password",
|
| 139 |
lines=1,
|
|
|
|
| 141 |
|
| 142 |
error_message = gr.Markdown()
|
| 143 |
|
| 144 |
+
login_button = gr.Button(
|
| 145 |
+
"Enter with personal token" if trusted_team_mode else "Login",
|
| 146 |
+
variant="primary",
|
| 147 |
+
scale=1,
|
| 148 |
+
)
|
| 149 |
+
|
| 150 |
+
oauth_continue_button = None
|
| 151 |
+
if enable_oauth_login:
|
| 152 |
+
gr.Markdown(
|
| 153 |
+
(
|
| 154 |
+
"### Optional OAuth test\n"
|
| 155 |
+
"The hosted Space can sign you in with Hugging Face OAuth. "
|
| 156 |
+
"For this interim private-storage mode, complete the personal-token test above before "
|
| 157 |
+
"using real validation data because OAuth Bucket writes have not yet been proven."
|
| 158 |
+
if trusted_team_mode
|
| 159 |
+
else (
|
| 160 |
+
"**OAuth access required for private collaborative state.** "
|
| 161 |
+
"Complete both steps below; signing into Hugging Face alone does not open an app session."
|
| 162 |
+
)
|
| 163 |
+
)
|
| 164 |
+
)
|
| 165 |
+
gr.LoginButton("1. Sign in with Hugging Face", logout_value="Sign out ({})")
|
| 166 |
+
oauth_continue_button = gr.Button(
|
| 167 |
+
"2. Enter workspace with OAuth authorization",
|
| 168 |
+
variant="secondary" if trusted_team_mode else "primary",
|
| 169 |
+
)
|
| 170 |
|
| 171 |
session_output = gr.Textbox(
|
| 172 |
label="Session ID",
|
src/ui/theme.py
CHANGED
|
@@ -1449,8 +1449,15 @@ select {
|
|
| 1449 |
|
| 1450 |
|
| 1451 |
def app_header_html(state_backend_message: str = "") -> str:
|
| 1452 |
-
|
| 1453 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1454 |
return f"""
|
| 1455 |
<div class="bn-app-header">
|
| 1456 |
<div>
|
|
|
|
| 1449 |
|
| 1450 |
|
| 1451 |
def app_header_html(state_backend_message: str = "") -> str:
|
| 1452 |
+
if "HF trusted-team project storage enabled" in state_backend_message:
|
| 1453 |
+
backend_text = "Private HF state ready"
|
| 1454 |
+
backend_class = "bn-pill-ok"
|
| 1455 |
+
elif "Supabase state backend enabled" in state_backend_message:
|
| 1456 |
+
backend_text = "Supabase ready"
|
| 1457 |
+
backend_class = "bn-pill-ok"
|
| 1458 |
+
else:
|
| 1459 |
+
backend_text = "Filesystem state"
|
| 1460 |
+
backend_class = "bn-pill-warn"
|
| 1461 |
return f"""
|
| 1462 |
<div class="bn-app-header">
|
| 1463 |
<div>
|
tests/unit/test_app_factory_audio_helpers.py
CHANGED
|
@@ -3,6 +3,7 @@ from datetime import date
|
|
| 3 |
from pathlib import Path
|
| 4 |
import hashlib
|
| 5 |
import json
|
|
|
|
| 6 |
|
| 7 |
import pandas as pd
|
| 8 |
import pytest
|
|
@@ -35,11 +36,13 @@ from src.ui.app_factory import (
|
|
| 35 |
_persist_bootstrap_state,
|
| 36 |
_resolve_username_login_policy,
|
| 37 |
_resolve_project_fetch_token,
|
|
|
|
| 38 |
_write_validation_export,
|
| 39 |
)
|
| 40 |
from src.auth.auth_service import AuthService
|
| 41 |
from src.config.runtime_config import RuntimeConfig
|
| 42 |
from src.domain.models import Detection, Project, Role
|
|
|
|
| 43 |
from src.repositories.state_safety import StateSafetyError
|
| 44 |
from src.services.hf_project_state_store import HfProjectStateStoreLoadedProject
|
| 45 |
from src.ui.admin_panel import AdminPanelManager
|
|
@@ -64,6 +67,38 @@ class FakeAudioService:
|
|
| 64 |
self.cleaned.append(cache_key)
|
| 65 |
|
| 66 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
class InaccessibleAudioService:
|
| 68 |
def fetch(self, dataset_repo: str, audio_id: str, **kwargs: object) -> FakeFetchResult:
|
| 69 |
_ = (dataset_repo, audio_id, kwargs)
|
|
@@ -197,6 +232,87 @@ class FakeQueueService:
|
|
| 197 |
return FakeQueueService._Page()
|
| 198 |
|
| 199 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 200 |
def test_extract_audio_id_from_list_rows() -> None:
|
| 201 |
rows = [["k1", "audio_01", "sp", 0.9, 0.0, 1.0]]
|
| 202 |
assert _extract_audio_id(rows, 0) == "audio_01"
|
|
|
|
| 3 |
from pathlib import Path
|
| 4 |
import hashlib
|
| 5 |
import json
|
| 6 |
+
from types import SimpleNamespace
|
| 7 |
|
| 8 |
import pandas as pd
|
| 9 |
import pytest
|
|
|
|
| 36 |
_persist_bootstrap_state,
|
| 37 |
_resolve_username_login_policy,
|
| 38 |
_resolve_project_fetch_token,
|
| 39 |
+
_initialize_hf_trusted_team_storage,
|
| 40 |
_write_validation_export,
|
| 41 |
)
|
| 42 |
from src.auth.auth_service import AuthService
|
| 43 |
from src.config.runtime_config import RuntimeConfig
|
| 44 |
from src.domain.models import Detection, Project, Role
|
| 45 |
+
from src.repositories.hf_bucket_validation_repository import HfBucketValidationError, HfBucketValidationInitResult
|
| 46 |
from src.repositories.state_safety import StateSafetyError
|
| 47 |
from src.services.hf_project_state_store import HfProjectStateStoreLoadedProject
|
| 48 |
from src.ui.admin_panel import AdminPanelManager
|
|
|
|
| 67 |
self.cleaned.append(cache_key)
|
| 68 |
|
| 69 |
|
| 70 |
+
class FakeDatasetInfoApi:
|
| 71 |
+
def __init__(self, private: bool, organization_exists: bool = True) -> None:
|
| 72 |
+
self.private = private
|
| 73 |
+
self.organization_exists = organization_exists
|
| 74 |
+
|
| 75 |
+
def get_organization_overview(self, organization: str, token: str): # noqa: ANN001
|
| 76 |
+
assert organization == "project-org"
|
| 77 |
+
assert token == "hf_admin"
|
| 78 |
+
if not self.organization_exists:
|
| 79 |
+
raise RuntimeError("404 organization not found")
|
| 80 |
+
return SimpleNamespace(name=organization)
|
| 81 |
+
|
| 82 |
+
def repo_info(self, *, repo_id: str, repo_type: str, token: str): # noqa: ANN001
|
| 83 |
+
assert repo_id == "project-org/private-audio"
|
| 84 |
+
assert repo_type == "dataset"
|
| 85 |
+
assert token == "hf_admin"
|
| 86 |
+
return SimpleNamespace(private=self.private)
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
class FakeBucketInitializer:
|
| 90 |
+
def __init__(self) -> None:
|
| 91 |
+
self.calls: list[tuple[str, str, str | None]] = []
|
| 92 |
+
|
| 93 |
+
def initialize(self, *, project_slug: str, dataset_repo_id: str, token: str | None):
|
| 94 |
+
self.calls.append((project_slug, dataset_repo_id, token))
|
| 95 |
+
return HfBucketValidationInitResult(
|
| 96 |
+
bucket_id="project-org/private-audio_validation_state",
|
| 97 |
+
initialized=True,
|
| 98 |
+
reused_existing=False,
|
| 99 |
+
)
|
| 100 |
+
|
| 101 |
+
|
| 102 |
class InaccessibleAudioService:
|
| 103 |
def fetch(self, dataset_repo: str, audio_id: str, **kwargs: object) -> FakeFetchResult:
|
| 104 |
_ = (dataset_repo, audio_id, kwargs)
|
|
|
|
| 232 |
return FakeQueueService._Page()
|
| 233 |
|
| 234 |
|
| 235 |
+
def test_initialize_hf_trusted_team_storage_sets_bucket_without_persisting_shared_token() -> None:
|
| 236 |
+
project = Project(
|
| 237 |
+
project_slug="private-audio",
|
| 238 |
+
name="Private Audio",
|
| 239 |
+
dataset_repo_id="project-org/private-audio",
|
| 240 |
+
dataset_token="hf_shared_should_be_removed",
|
| 241 |
+
)
|
| 242 |
+
bucket_initializer = FakeBucketInitializer()
|
| 243 |
+
|
| 244 |
+
result = _initialize_hf_trusted_team_storage(
|
| 245 |
+
project=project,
|
| 246 |
+
token="hf_admin",
|
| 247 |
+
api=FakeDatasetInfoApi(private=True), # type: ignore[arg-type]
|
| 248 |
+
bucket_initializer=bucket_initializer, # type: ignore[arg-type]
|
| 249 |
+
)
|
| 250 |
+
|
| 251 |
+
assert result.bucket_id == "project-org/private-audio_validation_state"
|
| 252 |
+
assert project.validation_bucket_id == result.bucket_id
|
| 253 |
+
assert project.validation_backend == "hf_bucket"
|
| 254 |
+
assert project.dataset_token is None
|
| 255 |
+
assert bucket_initializer.calls == [("private-audio", "project-org/private-audio", "hf_admin")]
|
| 256 |
+
|
| 257 |
+
|
| 258 |
+
def test_initialize_hf_trusted_team_storage_rejects_public_source_dataset() -> None:
|
| 259 |
+
project = Project(
|
| 260 |
+
project_slug="private-audio",
|
| 261 |
+
name="Private Audio",
|
| 262 |
+
dataset_repo_id="project-org/private-audio",
|
| 263 |
+
)
|
| 264 |
+
bucket_initializer = FakeBucketInitializer()
|
| 265 |
+
|
| 266 |
+
with pytest.raises(HfBucketValidationError, match="only accepts a private dataset"):
|
| 267 |
+
_initialize_hf_trusted_team_storage(
|
| 268 |
+
project=project,
|
| 269 |
+
token="hf_admin",
|
| 270 |
+
api=FakeDatasetInfoApi(private=False), # type: ignore[arg-type]
|
| 271 |
+
bucket_initializer=bucket_initializer, # type: ignore[arg-type]
|
| 272 |
+
)
|
| 273 |
+
|
| 274 |
+
assert bucket_initializer.calls == []
|
| 275 |
+
|
| 276 |
+
|
| 277 |
+
def test_initialize_hf_trusted_team_storage_requires_organization_namespace() -> None:
|
| 278 |
+
project = Project(
|
| 279 |
+
project_slug="private-audio",
|
| 280 |
+
name="Private Audio",
|
| 281 |
+
dataset_repo_id="project-org/private-audio",
|
| 282 |
+
)
|
| 283 |
+
bucket_initializer = FakeBucketInitializer()
|
| 284 |
+
|
| 285 |
+
with pytest.raises(HfBucketValidationError, match="dedicated Hugging Face organization"):
|
| 286 |
+
_initialize_hf_trusted_team_storage(
|
| 287 |
+
project=project,
|
| 288 |
+
token="hf_admin",
|
| 289 |
+
api=FakeDatasetInfoApi(private=True, organization_exists=False), # type: ignore[arg-type]
|
| 290 |
+
bucket_initializer=bucket_initializer, # type: ignore[arg-type]
|
| 291 |
+
)
|
| 292 |
+
|
| 293 |
+
assert bucket_initializer.calls == []
|
| 294 |
+
|
| 295 |
+
|
| 296 |
+
def test_initialize_hf_trusted_team_storage_requires_collaborative_app_visibility() -> None:
|
| 297 |
+
project = Project(
|
| 298 |
+
project_slug="private-audio",
|
| 299 |
+
name="Private Audio",
|
| 300 |
+
dataset_repo_id="project-org/private-audio",
|
| 301 |
+
visibility="private",
|
| 302 |
+
)
|
| 303 |
+
bucket_initializer = FakeBucketInitializer()
|
| 304 |
+
|
| 305 |
+
with pytest.raises(HfBucketValidationError, match="visibility 'collaborative'"):
|
| 306 |
+
_initialize_hf_trusted_team_storage(
|
| 307 |
+
project=project,
|
| 308 |
+
token="hf_admin",
|
| 309 |
+
api=FakeDatasetInfoApi(private=True), # type: ignore[arg-type]
|
| 310 |
+
bucket_initializer=bucket_initializer, # type: ignore[arg-type]
|
| 311 |
+
)
|
| 312 |
+
|
| 313 |
+
assert bucket_initializer.calls == []
|
| 314 |
+
|
| 315 |
+
|
| 316 |
def test_extract_audio_id_from_list_rows() -> None:
|
| 317 |
rows = [["k1", "audio_01", "sp", 0.9, 0.0, 1.0]]
|
| 318 |
assert _extract_audio_id(rows, 0) == "audio_01"
|
tests/unit/test_hf_bucket_validation_repository.py
CHANGED
|
@@ -6,6 +6,7 @@ from src.domain.models import Validation
|
|
| 6 |
from src.repositories.append_only_validation_repository import OptimisticLockError
|
| 7 |
from src.repositories.hf_bucket_validation_repository import (
|
| 8 |
HF_BUCKET_VALIDATION_BACKEND,
|
|
|
|
| 9 |
HfBucketValidationError,
|
| 10 |
HfBucketValidationInitializer,
|
| 11 |
HfBucketValidationRepository,
|
|
@@ -99,6 +100,22 @@ def test_initializer_reuses_manifest_but_refuses_unknown_existing_data() -> None
|
|
| 99 |
unsafe.initialize(project_slug="project-a", dataset_repo_id="owner/audio", token="hf_admin")
|
| 100 |
|
| 101 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 102 |
def test_save_validation_writes_event_and_snapshot_without_repo_commit() -> None:
|
| 103 |
api = FakeBucketApi({"snapshots/current.json": json.dumps({"project_slug": "project-a", "items": {}})})
|
| 104 |
repository = HfBucketValidationRepository(bucket_id="owner/audio_validation_state", token="hf_user", api=api)
|
|
|
|
| 6 |
from src.repositories.append_only_validation_repository import OptimisticLockError
|
| 7 |
from src.repositories.hf_bucket_validation_repository import (
|
| 8 |
HF_BUCKET_VALIDATION_BACKEND,
|
| 9 |
+
HfBucketPermissionProbe,
|
| 10 |
HfBucketValidationError,
|
| 11 |
HfBucketValidationInitializer,
|
| 12 |
HfBucketValidationRepository,
|
|
|
|
| 100 |
unsafe.initialize(project_slug="project-a", dataset_repo_id="owner/audio", token="hf_admin")
|
| 101 |
|
| 102 |
|
| 103 |
+
def test_bucket_permission_probe_reads_writes_verifies_and_removes_marker() -> None:
|
| 104 |
+
api = FakeBucketApi({"metadata/project.json": "{}"})
|
| 105 |
+
probe = HfBucketPermissionProbe(api=api)
|
| 106 |
+
|
| 107 |
+
result = probe.probe(
|
| 108 |
+
bucket_id="owner/audio_validation_state",
|
| 109 |
+
actor_username="validator-a",
|
| 110 |
+
token="hf_validator",
|
| 111 |
+
)
|
| 112 |
+
|
| 113 |
+
assert result.bucket_id == "owner/audio_validation_state"
|
| 114 |
+
assert result.actor_username == "validator-a"
|
| 115 |
+
assert result.diagnostic_path not in api.files
|
| 116 |
+
assert api.deletes[-1] == [result.diagnostic_path]
|
| 117 |
+
|
| 118 |
+
|
| 119 |
def test_save_validation_writes_event_and_snapshot_without_repo_commit() -> None:
|
| 120 |
api = FakeBucketApi({"snapshots/current.json": json.dumps({"project_slug": "project-a", "items": {}})})
|
| 121 |
repository = HfBucketValidationRepository(bucket_id="owner/audio_validation_state", token="hf_user", api=api)
|
tests/unit/test_runtime_config.py
CHANGED
|
@@ -31,3 +31,19 @@ def test_runtime_config_reads_bucket_validation_flag(monkeypatch) -> None: # no
|
|
| 31 |
config = RuntimeConfig.from_env()
|
| 32 |
|
| 33 |
assert config.hf_bucket_validations_enabled is True
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
config = RuntimeConfig.from_env()
|
| 32 |
|
| 33 |
assert config.hf_bucket_validations_enabled is True
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def test_runtime_config_reads_hf_trusted_team_mode_flag(monkeypatch) -> None: # noqa: ANN001
|
| 37 |
+
monkeypatch.setenv("BIRDNET_HF_TRUSTED_TEAM_MODE_ENABLED", "true")
|
| 38 |
+
|
| 39 |
+
config = RuntimeConfig.from_env()
|
| 40 |
+
|
| 41 |
+
assert config.hf_trusted_team_mode_enabled is True
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def test_runtime_config_disables_hf_trusted_team_mode_by_default(monkeypatch) -> None: # noqa: ANN001
|
| 45 |
+
monkeypatch.delenv("BIRDNET_HF_TRUSTED_TEAM_MODE_ENABLED", raising=False)
|
| 46 |
+
|
| 47 |
+
config = RuntimeConfig.from_env()
|
| 48 |
+
|
| 49 |
+
assert config.hf_trusted_team_mode_enabled is False
|