Spaces:
Sleeping
Sleeping
Sync app from GitHub Actions (feature/project-state-security-privacy-review)
Browse files- README.md +16 -13
- docs/TEMP_HF_ADMIN_OWNED_PRIVATE_STORAGE_DELIVERY_PLAN.md +88 -0
- src/config/runtime_config.py +4 -4
- src/repositories/hf_bucket_validation_repository.py +4 -3
- src/repositories/project_aware_validation_repository.py +12 -2
- src/ui/app_factory.py +152 -75
- src/ui/login_page.py +15 -17
- src/ui/theme.py +2 -2
- tests/unit/test_app_factory_audio_helpers.py +23 -26
- tests/unit/test_hf_bucket_validation_repository.py +2 -2
- tests/unit/test_project_aware_validation_repository.py +45 -0
- tests/unit/test_runtime_config.py +6 -6
README.md
CHANGED
|
@@ -177,7 +177,7 @@ User access bootstrap file example (`BIRDNET_USER_ACCESS_FILE`):
|
|
| 177 |
- SDK: `Gradio`
|
| 178 |
- Python: `3.11`
|
| 179 |
- OAuth: enabled (`hf_oauth: true` in this README metadata)
|
| 180 |
-
- OAuth scopes: `read-repos` only;
|
| 181 |
|
| 182 |
2. Push this repository to the Space.
|
| 183 |
|
|
@@ -199,23 +199,25 @@ Optional runtime settings:
|
|
| 199 |
- `BIRDNET_ENABLE_DEMO_BOOTSTRAP` (`false` in production)
|
| 200 |
- `BIRDNET_HF_PROJECT_STATE_WRITES_ENABLED=true` (experimental `_state` diagnostics and recovery only while a new collaborative backend is selected)
|
| 201 |
- `BIRDNET_HF_BUCKET_VALIDATIONS_ENABLED=true` (legacy experimental access for existing Bucket projects)
|
| 202 |
-
- `
|
| 203 |
-
- `BIRDNET_HF_PROJECT_STATE_REPOS=
|
| 204 |
|
| 205 |
-
The HF
|
| 206 |
|
| 207 |
-
|
|
|
|
|
|
|
| 208 |
|
| 209 |
```text
|
| 210 |
-
|
| 211 |
BIRDNET_AUTH_MODE=hf_token
|
| 212 |
-
BIRDNET_HF_PROJECT_STATE_REPOS=
|
| 213 |
-
|
| 214 |
```
|
| 215 |
|
| 216 |
-
|
| 217 |
|
| 218 |
-
Projects previously created with a private `_state` repository can be recovered from the **Admin** tab using **Connect Existing State**. Sign in
|
| 219 |
|
| 220 |
Legacy Supabase state backend, available while migrating existing projects:
|
| 221 |
|
|
@@ -225,7 +227,7 @@ Legacy Supabase state backend, available while migrating existing projects:
|
|
| 225 |
|
| 226 |
When Supabase is enabled, projects, ACL, invites, validation events, and current validation snapshots are stored in Supabase instead of `/data`. For eventual public distribution, the planned backend is administrator-owned Supabase provisioned through OAuth with an Edge Function validating Hugging Face identity; see `docs/TEMP_BYO_SUPABASE_EDGE_FUNCTION_HF_IDENTITY_PLAN.md`.
|
| 227 |
|
| 228 |
-
The **Private state OAuth diagnostic** action in **Projects** applies to experimental direct `_state` writes. In
|
| 229 |
|
| 230 |
Optional invite email settings:
|
| 231 |
|
|
@@ -259,7 +261,7 @@ Notes:
|
|
| 259 |
- The app entrypoint reads `PORT` automatically in Spaces.
|
| 260 |
- Keep user/project bootstrap files private if they contain sensitive assignments.
|
| 261 |
- Use Supabase or persistent Space storage to keep projects, invites, ACL, and validations across redeploys. The free Space filesystem is ephemeral.
|
| 262 |
-
-
|
| 263 |
- Datasets uploaded by `HF_Dataset_Uploader` are read from `index/files.parquet` first; older `index/shards/*.parquet`, CSV, JSONL, and audio-path fallback layouts remain supported.
|
| 264 |
|
| 265 |
## Local Uploader CLI
|
|
@@ -544,7 +546,8 @@ python -m src.uploader_cli.main start --repo-id alice/birdnet-2026 --segments C:
|
|
| 544 |
- `BIRDNET_STATE_BACKEND`
|
| 545 |
- `SUPABASE_URL`
|
| 546 |
- `SUPABASE_SERVICE_ROLE_KEY`
|
| 547 |
-
- `
|
|
|
|
| 548 |
- `BIRDNET_HF_PROJECT_STATE_REPOS`
|
| 549 |
- `BIRDNET_HF_PROJECT_STATE_WRITES_ENABLED`
|
| 550 |
- `BIRDNET_HF_BUCKET_VALIDATIONS_ENABLED`
|
|
|
|
| 177 |
- SDK: `Gradio`
|
| 178 |
- Python: `3.11`
|
| 179 |
- OAuth: enabled (`hf_oauth: true` in this README metadata)
|
| 180 |
+
- OAuth scopes: `read-repos` only; in administrator-owned storage mode user OAuth is used only for identity
|
| 181 |
|
| 182 |
2. Push this repository to the Space.
|
| 183 |
|
|
|
|
| 199 |
- `BIRDNET_ENABLE_DEMO_BOOTSTRAP` (`false` in production)
|
| 200 |
- `BIRDNET_HF_PROJECT_STATE_WRITES_ENABLED=true` (experimental `_state` diagnostics and recovery only while a new collaborative backend is selected)
|
| 201 |
- `BIRDNET_HF_BUCKET_VALIDATIONS_ENABLED=true` (legacy experimental access for existing Bucket projects)
|
| 202 |
+
- `BIRDNET_HF_ADMIN_STORAGE_MODE_ENABLED=true` (private HF-only delivery mode for a small trusted team)
|
| 203 |
+
- `BIRDNET_HF_PROJECT_STATE_REPOS=jrrribeiro/audio-dataset_state` (comma-separated private state repos restored after Space rebuild)
|
| 204 |
|
| 205 |
+
The HF administrator-owned storage mode is intended for confidential projects run by a small trusted research team. The source audio dataset must already be private in the administrator's personal Hugging Face namespace. When a project is created, the app creates a private `_state` repository for project/ACL/invite recovery and a private Storage Bucket for validation events and snapshots in that same namespace. Select `collaborative` inside the app: this enables app-managed validators and does not make any Hugging Face resource public.
|
| 206 |
|
| 207 |
+
The Space stores the administrator storage credential as a protected secret. Validators log in with their own Hugging Face identity and are invited or assigned inside the app; they do not need repository or Bucket permissions, organization membership, or the administrator token. Resource operations are performed server-side only after the app confirms the user's project role.
|
| 208 |
+
|
| 209 |
+
Recommended administrator-owned Space settings:
|
| 210 |
|
| 211 |
```text
|
| 212 |
+
BIRDNET_HF_ADMIN_STORAGE_MODE_ENABLED=true
|
| 213 |
BIRDNET_AUTH_MODE=hf_token
|
| 214 |
+
BIRDNET_HF_PROJECT_STATE_REPOS=jrrribeiro/audio-dataset_state
|
| 215 |
+
BIRDNET_HF_STORAGE_TOKEN=<administrator token stored only as a Space secret>
|
| 216 |
```
|
| 217 |
|
| 218 |
+
`HF_TOKEN` remains a compatibility fallback for the storage credential, but `BIRDNET_HF_STORAGE_TOKEN` is preferred because its purpose is explicit. Never enter this administrator token in an ordinary user login form. After creating a project, add its generated `_state` repository to `BIRDNET_HF_PROJECT_STATE_REPOS` before relying on rebuild recovery. An administrator may run **Projects > Private storage backend health** to verify the protected backend credential with a temporary marker that is immediately removed.
|
| 219 |
|
| 220 |
+
Projects previously created with a private `_state` repository in this mode can be recovered from the **Admin** tab using **Connect Existing State**. Sign in as an administrator recorded in its `acl.json`, then provide the `_state` repo id (for example, `jrrribeiro/audio_dataset_state`). Recovery accepts only a manifest with private Bucket validation storage and a private personal-namespace source dataset accessible to the configured storage account. The app loads the saved manifest, ACL, and pending invites.
|
| 221 |
|
| 222 |
Legacy Supabase state backend, available while migrating existing projects:
|
| 223 |
|
|
|
|
| 227 |
|
| 228 |
When Supabase is enabled, projects, ACL, invites, validation events, and current validation snapshots are stored in Supabase instead of `/data`. For eventual public distribution, the planned backend is administrator-owned Supabase provisioned through OAuth with an Edge Function validating Hugging Face identity; see `docs/TEMP_BYO_SUPABASE_EDGE_FUNCTION_HF_IDENTITY_PLAN.md`.
|
| 229 |
|
| 230 |
+
The **Private state OAuth diagnostic** action in **Projects** applies only to experimental direct `_state` writes. In administrator-owned storage mode, frequent validation state uses the Bucket rather than dataset repository commits.
|
| 231 |
|
| 232 |
Optional invite email settings:
|
| 233 |
|
|
|
|
| 261 |
- The app entrypoint reads `PORT` automatically in Spaces.
|
| 262 |
- Keep user/project bootstrap files private if they contain sensitive assignments.
|
| 263 |
- Use Supabase or persistent Space storage to keep projects, invites, ACL, and validations across redeploys. The free Space filesystem is ephemeral.
|
| 264 |
+
- In administrator-owned storage mode, each collaborator signs in with their own Hugging Face identity, while only the Space backend credential accesses private project storage.
|
| 265 |
- Datasets uploaded by `HF_Dataset_Uploader` are read from `index/files.parquet` first; older `index/shards/*.parquet`, CSV, JSONL, and audio-path fallback layouts remain supported.
|
| 266 |
|
| 267 |
## Local Uploader CLI
|
|
|
|
| 546 |
- `BIRDNET_STATE_BACKEND`
|
| 547 |
- `SUPABASE_URL`
|
| 548 |
- `SUPABASE_SERVICE_ROLE_KEY`
|
| 549 |
+
- `BIRDNET_HF_ADMIN_STORAGE_MODE_ENABLED`
|
| 550 |
+
- `BIRDNET_HF_STORAGE_TOKEN` (Secret)
|
| 551 |
- `BIRDNET_HF_PROJECT_STATE_REPOS`
|
| 552 |
- `BIRDNET_HF_PROJECT_STATE_WRITES_ENABLED`
|
| 553 |
- `BIRDNET_HF_BUCKET_VALIDATIONS_ENABLED`
|
docs/TEMP_HF_ADMIN_OWNED_PRIVATE_STORAGE_DELIVERY_PLAN.md
ADDED
|
@@ -0,0 +1,88 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# HF Administrator-Owned Private Storage Delivery Plan
|
| 2 |
+
|
| 3 |
+
Status: immediate delivery architecture implemented on `feature/project-state-security-privacy-review`
|
| 4 |
+
Created: 2026-05-26
|
| 5 |
+
Purpose: begin real validation with a small trusted team while keeping unpublished data inaccessible outside the app.
|
| 6 |
+
|
| 7 |
+
## Decision
|
| 8 |
+
|
| 9 |
+
For the immediate workflow, all durable project resources belong to the project administrator's personal Hugging Face account:
|
| 10 |
+
|
| 11 |
+
```text
|
| 12 |
+
administrator namespace/
|
| 13 |
+
audio-dataset private Dataset repository
|
| 14 |
+
audio-dataset_state private Dataset repository for project, ACL, and invites
|
| 15 |
+
audio-dataset_validation_state private Storage Bucket for validation events and snapshots
|
| 16 |
+
```
|
| 17 |
+
|
| 18 |
+
The deployed Space receives one protected administrator storage secret. Team members authenticate with their own Hugging Face identities, but the app uses their identity only for its persisted ACL and validation attribution. Authorized reads and writes to private project resources are performed by the backend with the storage secret.
|
| 19 |
+
|
| 20 |
+
This avoids organization membership and lets administrators invite or assign validators entirely inside the app.
|
| 21 |
+
|
| 22 |
+
## Security Boundary
|
| 23 |
+
|
| 24 |
+
This mode protects against external Hub users reading a private dataset, state repository, or Bucket. An invited validator does not receive direct Hub access and cannot browse those resources outside this app.
|
| 25 |
+
|
| 26 |
+
The trusted boundary is the deployed Space:
|
| 27 |
+
|
| 28 |
+
1. its owner controls the administrator storage secret;
|
| 29 |
+
2. its callbacks must validate app authorization before serving private audio or state;
|
| 30 |
+
3. trusted project validators can receive project audio through the app and submit decisions; and
|
| 31 |
+
4. compromise of the Space or its storage secret exposes projects accessible to that credential.
|
| 32 |
+
|
| 33 |
+
For public distribution to independent administrators, this model is not sufficient because the operator would hold their storage credentials. The planned distribution architecture remains administrator-owned Supabase with an Edge Function validating Hugging Face identity, documented in `TEMP_BYO_SUPABASE_EDGE_FUNCTION_HF_IDENTITY_PLAN.md`.
|
| 34 |
+
|
| 35 |
+
## Resource Roles
|
| 36 |
+
|
| 37 |
+
| Resource | Purpose | Writer |
|
| 38 |
+
|---|---|---|
|
| 39 |
+
| Private audio dataset | audio and source detection index | administrator/uploader |
|
| 40 |
+
| Private `_state` dataset | manifest, ACL, invitations, recovery metadata | app backend using administrator secret |
|
| 41 |
+
| Private validation Bucket | events, current snapshot, archives | app backend using administrator secret |
|
| 42 |
+
| Local Space disk | temporary cache and generated downloads | non-authoritative |
|
| 43 |
+
|
| 44 |
+
## Implemented Behavior
|
| 45 |
+
|
| 46 |
+
1. `BIRDNET_HF_ADMIN_STORAGE_MODE_ENABLED=true` activates this mode.
|
| 47 |
+
2. `BIRDNET_HF_STORAGE_TOKEN` is read server-side and never saved in project records or shown in the UI. `HF_TOKEN` remains a compatibility fallback.
|
| 48 |
+
3. New projects require a private dataset in the personal namespace belonging to the configured storage credential.
|
| 49 |
+
4. New projects automatically initialize a private validation Bucket and private companion `_state` repository.
|
| 50 |
+
5. Project manifests persist the Bucket reference; ACL and invite changes sync to `_state`.
|
| 51 |
+
6. Validation Bucket reads and writes route through the backend storage credential while retaining the actual validator username in each event.
|
| 52 |
+
7. Dataset metadata and audio fetches route through the backend credential only after server-side authorization of the session and project role.
|
| 53 |
+
8. The administrator can run a backend storage health check; validators do not test or receive direct Bucket permissions.
|
| 54 |
+
|
| 55 |
+
## Space Configuration
|
| 56 |
+
|
| 57 |
+
Configure:
|
| 58 |
+
|
| 59 |
+
```text
|
| 60 |
+
BIRDNET_HF_ADMIN_STORAGE_MODE_ENABLED=true
|
| 61 |
+
BIRDNET_AUTH_MODE=hf_token
|
| 62 |
+
BIRDNET_HF_PROJECT_STATE_REPOS=jrrribeiro/audio-dataset_state
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
Configure as a Secret:
|
| 66 |
+
|
| 67 |
+
```text
|
| 68 |
+
BIRDNET_HF_STORAGE_TOKEN=<administrator credential with required private dataset/repo/Bucket read-write access>
|
| 69 |
+
```
|
| 70 |
+
|
| 71 |
+
Use a scoped credential limited to the administrator's required resources when Hugging Face token controls permit it. Do not share it with validators or enter it in the app login input.
|
| 72 |
+
|
| 73 |
+
## Acceptance Gate Before Real Validation
|
| 74 |
+
|
| 75 |
+
1. Administrator creates a private test dataset in their personal namespace.
|
| 76 |
+
2. Administrator creates the app project; the app reports the generated private `_state` repository and validation Bucket.
|
| 77 |
+
3. Administrator adds that `_state` repository to `BIRDNET_HF_PROJECT_STATE_REPOS` and restarts the Space.
|
| 78 |
+
4. Project is restored with its ACL and Bucket reference intact.
|
| 79 |
+
5. Administrator assigns or invites a second Hugging Face account only through the app.
|
| 80 |
+
6. The second account logs in with its own identity, opens the project, loads private audio, and submits validations without Hub resource permissions.
|
| 81 |
+
7. The second account cannot access the private audio dataset, `_state`, or Bucket directly on Hugging Face.
|
| 82 |
+
8. Administrator exports CSV/XLSX and confirms the second user's validation attribution.
|
| 83 |
+
9. A rebuild restores the project and previous validations.
|
| 84 |
+
10. A short rapid-validation test confirms correctness under normal research throughput before full production use.
|
| 85 |
+
|
| 86 |
+
## Performance Note
|
| 87 |
+
|
| 88 |
+
Bucket events avoid one Git commit per validation. Before millions of validations or very high concurrency, test event compaction and snapshot behavior under sustained writes; if necessary, introduce additional snapshot sharding or buffered batches without changing this authorization model.
|
src/config/runtime_config.py
CHANGED
|
@@ -33,7 +33,7 @@ class RuntimeConfig:
|
|
| 33 |
auth_mode: str = "auto"
|
| 34 |
hf_project_state_writes_enabled: bool = False
|
| 35 |
hf_bucket_validations_enabled: bool = False
|
| 36 |
-
|
| 37 |
hf_project_state_repos: tuple[str, ...] = ()
|
| 38 |
|
| 39 |
@classmethod
|
|
@@ -112,8 +112,8 @@ class RuntimeConfig:
|
|
| 112 |
hf_project_state_writes_enabled = raw_hf_project_state_writes in {"1", "true", "yes", "on"}
|
| 113 |
raw_hf_bucket_validations = (os.getenv("BIRDNET_HF_BUCKET_VALIDATIONS_ENABLED") or "").strip().lower()
|
| 114 |
hf_bucket_validations_enabled = raw_hf_bucket_validations in {"1", "true", "yes", "on"}
|
| 115 |
-
|
| 116 |
-
|
| 117 |
raw_hf_project_state_repos = (os.getenv("BIRDNET_HF_PROJECT_STATE_REPOS") or "").strip()
|
| 118 |
hf_project_state_repos = tuple(
|
| 119 |
repo.strip()
|
|
@@ -149,7 +149,7 @@ class RuntimeConfig:
|
|
| 149 |
auth_mode=auth_mode,
|
| 150 |
hf_project_state_writes_enabled=hf_project_state_writes_enabled,
|
| 151 |
hf_bucket_validations_enabled=hf_bucket_validations_enabled,
|
| 152 |
-
|
| 153 |
hf_project_state_repos=hf_project_state_repos,
|
| 154 |
)
|
| 155 |
|
|
|
|
| 33 |
auth_mode: str = "auto"
|
| 34 |
hf_project_state_writes_enabled: bool = False
|
| 35 |
hf_bucket_validations_enabled: bool = False
|
| 36 |
+
hf_admin_storage_mode_enabled: bool = False
|
| 37 |
hf_project_state_repos: tuple[str, ...] = ()
|
| 38 |
|
| 39 |
@classmethod
|
|
|
|
| 112 |
hf_project_state_writes_enabled = raw_hf_project_state_writes in {"1", "true", "yes", "on"}
|
| 113 |
raw_hf_bucket_validations = (os.getenv("BIRDNET_HF_BUCKET_VALIDATIONS_ENABLED") or "").strip().lower()
|
| 114 |
hf_bucket_validations_enabled = raw_hf_bucket_validations in {"1", "true", "yes", "on"}
|
| 115 |
+
raw_hf_admin_storage_mode = (os.getenv("BIRDNET_HF_ADMIN_STORAGE_MODE_ENABLED") or "").strip().lower()
|
| 116 |
+
hf_admin_storage_mode_enabled = raw_hf_admin_storage_mode in {"1", "true", "yes", "on"}
|
| 117 |
raw_hf_project_state_repos = (os.getenv("BIRDNET_HF_PROJECT_STATE_REPOS") or "").strip()
|
| 118 |
hf_project_state_repos = tuple(
|
| 119 |
repo.strip()
|
|
|
|
| 149 |
auth_mode=auth_mode,
|
| 150 |
hf_project_state_writes_enabled=hf_project_state_writes_enabled,
|
| 151 |
hf_bucket_validations_enabled=hf_bucket_validations_enabled,
|
| 152 |
+
hf_admin_storage_mode_enabled=hf_admin_storage_mode_enabled,
|
| 153 |
hf_project_state_repos=hf_project_state_repos,
|
| 154 |
)
|
| 155 |
|
src/repositories/hf_bucket_validation_repository.py
CHANGED
|
@@ -238,7 +238,7 @@ class HfBucketPermissionProbe:
|
|
| 238 |
raise
|
| 239 |
except Exception as exc:
|
| 240 |
raise HfBucketValidationError(
|
| 241 |
-
f"Private Bucket read/write test failed for '{bucket}' using the
|
| 242 |
) from exc
|
| 243 |
finally:
|
| 244 |
if marker_written:
|
|
@@ -309,8 +309,9 @@ class HfBucketValidationRepository:
|
|
| 309 |
access_markers = ("401", "403", "404", "bucketnotfound", "bucket not found", "unauthorized", "forbidden")
|
| 310 |
if any(marker in message for marker in access_markers):
|
| 311 |
return HfBucketValidationError(
|
| 312 |
-
f"The
|
| 313 |
-
"
|
|
|
|
| 314 |
)
|
| 315 |
return exc
|
| 316 |
|
|
|
|
| 238 |
raise
|
| 239 |
except Exception as exc:
|
| 240 |
raise HfBucketValidationError(
|
| 241 |
+
f"Private Bucket read/write test failed for '{bucket}' using the supplied storage credential: {exc}"
|
| 242 |
) from exc
|
| 243 |
finally:
|
| 244 |
if marker_written:
|
|
|
|
| 309 |
access_markers = ("401", "403", "404", "bucketnotfound", "bucket not found", "unauthorized", "forbidden")
|
| 310 |
if any(marker in message for marker in access_markers):
|
| 311 |
return HfBucketValidationError(
|
| 312 |
+
f"The configured credential cannot access validation bucket '{self._bucket_id}'. "
|
| 313 |
+
"In administrator-owned storage mode, check the protected Space storage secret. "
|
| 314 |
+
"An invitation inside this app authorizes app access but does not grant direct Hub Bucket permissions."
|
| 315 |
)
|
| 316 |
return exc
|
| 317 |
|
src/repositories/project_aware_validation_repository.py
CHANGED
|
@@ -28,6 +28,7 @@ class ProjectAwareValidationRepository:
|
|
| 28 |
actor_token_provider: Callable[[Project, str], str | None] | None = None,
|
| 29 |
enable_hf_project_state: bool = False,
|
| 30 |
enable_hf_bucket_validations: bool = False,
|
|
|
|
| 31 |
hf_repository_factory: Callable[[str, str], _ValidationStateRepository] | None = None,
|
| 32 |
bucket_repository_factory: Callable[[str, str], _ValidationStateRepository] | None = None,
|
| 33 |
) -> None:
|
|
@@ -37,6 +38,7 @@ class ProjectAwareValidationRepository:
|
|
| 37 |
self._actor_token_provider = actor_token_provider
|
| 38 |
self._enable_hf_project_state = enable_hf_project_state
|
| 39 |
self._enable_hf_bucket_validations = enable_hf_bucket_validations
|
|
|
|
| 40 |
self._hf_repository_factory = hf_repository_factory or (
|
| 41 |
lambda state_repo_id, token: HfProjectStateValidationRepository(
|
| 42 |
state_repo_id=state_repo_id,
|
|
@@ -74,10 +76,18 @@ class ProjectAwareValidationRepository:
|
|
| 74 |
raise HfBucketValidationError(
|
| 75 |
"This project declares HF Bucket validation storage but has no validation bucket id."
|
| 76 |
)
|
| 77 |
-
token =
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
if not token:
|
| 79 |
raise HfBucketValidationError(
|
| 80 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 81 |
)
|
| 82 |
bucket_id = (project.validation_bucket_id or "").strip()
|
| 83 |
cache_key = (f"bucket:{bucket_id}", token)
|
|
|
|
| 28 |
actor_token_provider: Callable[[Project, str], str | None] | None = None,
|
| 29 |
enable_hf_project_state: bool = False,
|
| 30 |
enable_hf_bucket_validations: bool = False,
|
| 31 |
+
use_backend_token_for_bucket: bool = False,
|
| 32 |
hf_repository_factory: Callable[[str, str], _ValidationStateRepository] | None = None,
|
| 33 |
bucket_repository_factory: Callable[[str, str], _ValidationStateRepository] | None = None,
|
| 34 |
) -> None:
|
|
|
|
| 38 |
self._actor_token_provider = actor_token_provider
|
| 39 |
self._enable_hf_project_state = enable_hf_project_state
|
| 40 |
self._enable_hf_bucket_validations = enable_hf_bucket_validations
|
| 41 |
+
self._use_backend_token_for_bucket = use_backend_token_for_bucket
|
| 42 |
self._hf_repository_factory = hf_repository_factory or (
|
| 43 |
lambda state_repo_id, token: HfProjectStateValidationRepository(
|
| 44 |
state_repo_id=state_repo_id,
|
|
|
|
| 76 |
raise HfBucketValidationError(
|
| 77 |
"This project declares HF Bucket validation storage but has no validation bucket id."
|
| 78 |
)
|
| 79 |
+
token = (
|
| 80 |
+
(self._token_provider(project) or "").strip()
|
| 81 |
+
if self._use_backend_token_for_bucket
|
| 82 |
+
else (actor_token if actor_username else (self._token_provider(project) or "").strip())
|
| 83 |
+
)
|
| 84 |
if not token:
|
| 85 |
raise HfBucketValidationError(
|
| 86 |
+
(
|
| 87 |
+
"The backend storage credential required for private Bucket validation is not configured."
|
| 88 |
+
if self._use_backend_token_for_bucket
|
| 89 |
+
else "Bucket-backed validation requires the signed-in validator's Hugging Face authorization."
|
| 90 |
+
)
|
| 91 |
)
|
| 92 |
bucket_id = (project.validation_bucket_id or "").strip()
|
| 93 |
cache_key = (f"bucket:{bucket_id}", token)
|
src/ui/app_factory.py
CHANGED
|
@@ -452,6 +452,12 @@ def _env_hf_token() -> str | None:
|
|
| 452 |
return token or None
|
| 453 |
|
| 454 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 455 |
def _project_dataset_token(project: Project, fallback_token: str | None = None) -> str | None:
|
| 456 |
return (project.dataset_token or "").strip() or (fallback_token or "").strip() or _env_hf_token()
|
| 457 |
|
|
@@ -1561,7 +1567,7 @@ def _is_running_in_hf_space() -> bool:
|
|
| 1561 |
return bool((os.getenv("SPACE_ID") or os.getenv("SPACE_HOST") or "").strip())
|
| 1562 |
|
| 1563 |
|
| 1564 |
-
def
|
| 1565 |
*,
|
| 1566 |
project: Project,
|
| 1567 |
token: str | None,
|
|
@@ -1569,23 +1575,30 @@ def _validate_hf_trusted_team_source_dataset(
|
|
| 1569 |
) -> None:
|
| 1570 |
if (project.visibility or "").strip().lower() != "collaborative":
|
| 1571 |
raise HfBucketValidationError(
|
| 1572 |
-
"
|
| 1573 |
-
"The Hugging Face dataset and storage remain private;
|
| 1574 |
-
"team members inside this app."
|
| 1575 |
)
|
| 1576 |
token_value = (token or "").strip()
|
| 1577 |
if not token_value:
|
| 1578 |
-
raise HfBucketValidationError(
|
|
|
|
|
|
|
| 1579 |
|
| 1580 |
client = api or HfApi()
|
| 1581 |
namespace = project.dataset_repo_id.split("/", 1)[0].strip()
|
| 1582 |
try:
|
| 1583 |
-
client.
|
| 1584 |
except Exception as exc:
|
| 1585 |
raise HfBucketValidationError(
|
| 1586 |
-
"
|
| 1587 |
-
f"organization. Could not verify organization '{namespace}': {exc}"
|
| 1588 |
) from exc
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1589 |
try:
|
| 1590 |
dataset_info = client.repo_info(
|
| 1591 |
repo_id=project.dataset_repo_id,
|
|
@@ -1596,19 +1609,19 @@ def _validate_hf_trusted_team_source_dataset(
|
|
| 1596 |
raise HfBucketValidationError(f"Could not verify the Hugging Face dataset before creating private state: {exc}") from exc
|
| 1597 |
if not bool(getattr(dataset_info, "private", False)):
|
| 1598 |
raise HfBucketValidationError(
|
| 1599 |
-
"
|
| 1600 |
"before creating the validation project."
|
| 1601 |
)
|
| 1602 |
|
| 1603 |
|
| 1604 |
-
def
|
| 1605 |
*,
|
| 1606 |
project: Project,
|
| 1607 |
token: str | None,
|
| 1608 |
api: HfApi | None = None,
|
| 1609 |
bucket_initializer: HfBucketValidationInitializer | None = None,
|
| 1610 |
):
|
| 1611 |
-
|
| 1612 |
token_value = (token or "").strip()
|
| 1613 |
initializer = bucket_initializer or HfBucketValidationInitializer()
|
| 1614 |
result = initializer.initialize(
|
|
@@ -2967,8 +2980,8 @@ def create_app() -> gr.Blocks:
|
|
| 2967 |
|
| 2968 |
projects_file_path, user_access_file_path, invites_file_path = _resolve_bootstrap_file_paths(runtime_config)
|
| 2969 |
state_store, supabase_validation_repository, state_backend_message = _build_supabase_state(runtime_config)
|
| 2970 |
-
if runtime_config.
|
| 2971 |
-
state_backend_message = "HF
|
| 2972 |
allow_username_login, auth_mode_label, auth_mode_description = _resolve_username_login_policy(runtime_config)
|
| 2973 |
bootstrap_warning = _bootstrap_auth_and_projects(
|
| 2974 |
auth_service,
|
|
@@ -2978,7 +2991,7 @@ def create_app() -> gr.Blocks:
|
|
| 2978 |
user_access_file_path=str(user_access_file_path),
|
| 2979 |
invites_file_path=str(invites_file_path),
|
| 2980 |
state_store=state_store,
|
| 2981 |
-
hf_project_state_token=_env_hf_token(),
|
| 2982 |
)
|
| 2983 |
|
| 2984 |
def _current_project_map() -> dict[str, Project]:
|
|
@@ -3003,20 +3016,25 @@ def create_app() -> gr.Blocks:
|
|
| 3003 |
max_loaded_projects = 3
|
| 3004 |
audio_service = AudioFetchService(EphemeralCacheManager(ttl_seconds=300, max_files=128))
|
| 3005 |
base_validation_repository = supabase_validation_repository or AppendOnlyValidationRepository(base_dir=runtime_config.validation_base_dir)
|
| 3006 |
-
hf_state_writes_enabled = runtime_config.hf_project_state_writes_enabled or runtime_config.
|
| 3007 |
-
hf_bucket_validations_enabled = runtime_config.hf_bucket_validations_enabled or runtime_config.
|
| 3008 |
validation_repository = ProjectAwareValidationRepository(
|
| 3009 |
fallback_repository=base_validation_repository,
|
| 3010 |
project_lookup=admin_manager.get_project,
|
| 3011 |
token_provider=lambda project: (
|
| 3012 |
-
(
|
| 3013 |
-
|
| 3014 |
-
|
| 3015 |
-
|
| 3016 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3017 |
actor_token_provider=lambda _project, username: auth_service.get_hf_token_for_user(username),
|
| 3018 |
enable_hf_project_state=hf_state_writes_enabled,
|
| 3019 |
enable_hf_bucket_validations=hf_bucket_validations_enabled,
|
|
|
|
| 3020 |
)
|
| 3021 |
validation_service = ValidationService(validation_repository)
|
| 3022 |
hf_state_initializer = HfProjectStateStoreInitializer()
|
|
@@ -3086,6 +3104,14 @@ def create_app() -> gr.Blocks:
|
|
| 3086 |
role = auth_service.get_user_role_for_project(session.username, slug)
|
| 3087 |
return role == Role.admin
|
| 3088 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3089 |
def _refresh_session_copy(session):
|
| 3090 |
if session is None:
|
| 3091 |
return None
|
|
@@ -3154,6 +3180,8 @@ def create_app() -> gr.Blocks:
|
|
| 3154 |
slug = (project_slug or "").strip()
|
| 3155 |
if not slug:
|
| 3156 |
return ""
|
|
|
|
|
|
|
| 3157 |
|
| 3158 |
token = _project_fetch_token(slug, session)
|
| 3159 |
signature = _project_queue_signature(slug, token)
|
|
@@ -3175,6 +3203,8 @@ def create_app() -> gr.Blocks:
|
|
| 3175 |
return warning
|
| 3176 |
|
| 3177 |
def _project_state_write_token(project: Project, session=None) -> str | None:
|
|
|
|
|
|
|
| 3178 |
session_username = str(getattr(session, "username", "") or "").strip()
|
| 3179 |
session_token = auth_service.get_hf_token_for_user(session_username) if session_username else None
|
| 3180 |
owner_username = (project.owner_username or "").strip()
|
|
@@ -3284,8 +3314,8 @@ def create_app() -> gr.Blocks:
|
|
| 3284 |
"Secure access",
|
| 3285 |
"Sign in with your Hugging Face identity",
|
| 3286 |
(
|
| 3287 |
-
"
|
| 3288 |
-
if runtime_config.
|
| 3289 |
else "Your account controls project access, validator attribution, and private dataset access."
|
| 3290 |
),
|
| 3291 |
)
|
|
@@ -3295,12 +3325,12 @@ def create_app() -> gr.Blocks:
|
|
| 3295 |
allow_username_login=allow_username_login,
|
| 3296 |
enable_oauth_login=_is_running_in_hf_space(),
|
| 3297 |
auth_mode_label=(
|
| 3298 |
-
"Private HF
|
| 3299 |
-
"project
|
| 3300 |
-
if runtime_config.
|
| 3301 |
else auth_mode_description
|
| 3302 |
),
|
| 3303 |
-
|
| 3304 |
)
|
| 3305 |
|
| 3306 |
# Store session ID when login succeeds
|
|
@@ -3379,13 +3409,12 @@ def create_app() -> gr.Blocks:
|
|
| 3379 |
class_name="bn-panel-soft",
|
| 3380 |
)
|
| 3381 |
)
|
| 3382 |
-
if runtime_config.
|
| 3383 |
gr.HTML(
|
| 3384 |
inline_hint_html(
|
| 3385 |
-
"Private
|
| 3386 |
-
"
|
| 3387 |
-
"
|
| 3388 |
-
"project's dedicated Hugging Face organization.",
|
| 3389 |
"info",
|
| 3390 |
)
|
| 3391 |
)
|
|
@@ -3409,12 +3438,13 @@ def create_app() -> gr.Blocks:
|
|
| 3409 |
)
|
| 3410 |
create_project_token = gr.Textbox(
|
| 3411 |
label=(
|
| 3412 |
-
"
|
| 3413 |
-
if runtime_config.
|
| 3414 |
else "Project HF Token (optional)"
|
| 3415 |
),
|
| 3416 |
placeholder="hf_xxx...",
|
| 3417 |
type="password",
|
|
|
|
| 3418 |
)
|
| 3419 |
|
| 3420 |
create_project_message = gr.Markdown()
|
|
@@ -3459,7 +3489,11 @@ def create_app() -> gr.Blocks:
|
|
| 3459 |
repo_id = (repo_id or "").strip()
|
| 3460 |
visibility_value = (visibility or "collaborative").strip().lower()
|
| 3461 |
explicit_project_token = (project_token or "").strip() or None
|
| 3462 |
-
operation_token =
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3463 |
if not slug or not name or not repo_id:
|
| 3464 |
return "Fill slug, name, and repo id.", gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), session
|
| 3465 |
if visibility_value not in {"private", "collaborative"}:
|
|
@@ -3467,7 +3501,7 @@ def create_app() -> gr.Blocks:
|
|
| 3467 |
if (
|
| 3468 |
_is_running_in_hf_space()
|
| 3469 |
and hf_state_writes_enabled
|
| 3470 |
-
and not runtime_config.
|
| 3471 |
and getattr(session, "authentication_method", "") != "oauth"
|
| 3472 |
):
|
| 3473 |
return (
|
|
@@ -3511,13 +3545,13 @@ def create_app() -> gr.Blocks:
|
|
| 3511 |
dataset_repo_id=repo_id,
|
| 3512 |
visibility=visibility_value,
|
| 3513 |
owner_username=session.username,
|
| 3514 |
-
dataset_token=None if runtime_config.
|
| 3515 |
active=True,
|
| 3516 |
)
|
| 3517 |
bucket_result = None
|
| 3518 |
-
if runtime_config.
|
| 3519 |
try:
|
| 3520 |
-
bucket_result =
|
| 3521 |
project=project,
|
| 3522 |
token=operation_token,
|
| 3523 |
bucket_initializer=hf_bucket_initializer,
|
|
@@ -3592,7 +3626,7 @@ def create_app() -> gr.Blocks:
|
|
| 3592 |
recovery_message = (
|
| 3593 |
f" Add `{state_result.state_repo_id}` to `BIRDNET_HF_PROJECT_STATE_REPOS` before rebuilding "
|
| 3594 |
"the Space so this project is restored after restart."
|
| 3595 |
-
if runtime_config.
|
| 3596 |
else ""
|
| 3597 |
)
|
| 3598 |
|
|
@@ -3674,7 +3708,11 @@ def create_app() -> gr.Blocks:
|
|
| 3674 |
try:
|
| 3675 |
loaded = hf_state_connector.connect_admin_project(
|
| 3676 |
state_repo_id=repo_id,
|
| 3677 |
-
token=
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3678 |
actor_username=session.username,
|
| 3679 |
)
|
| 3680 |
except HfProjectStateStoreError as exc:
|
|
@@ -3689,14 +3727,14 @@ def create_app() -> gr.Blocks:
|
|
| 3689 |
session,
|
| 3690 |
)
|
| 3691 |
|
| 3692 |
-
if runtime_config.
|
| 3693 |
if (
|
| 3694 |
(loaded.project.validation_backend or "").strip() != HF_BUCKET_VALIDATION_BACKEND
|
| 3695 |
or not (loaded.project.validation_bucket_id or "").strip()
|
| 3696 |
):
|
| 3697 |
return (
|
| 3698 |
-
"Project state could not be connected in
|
| 3699 |
-
"does not declare private Hugging Face Bucket validation storage.",
|
| 3700 |
_project_rows(),
|
| 3701 |
gr.update(),
|
| 3702 |
gr.update(),
|
|
@@ -3706,13 +3744,13 @@ def create_app() -> gr.Blocks:
|
|
| 3706 |
session,
|
| 3707 |
)
|
| 3708 |
try:
|
| 3709 |
-
|
| 3710 |
project=loaded.project,
|
| 3711 |
-
token=
|
| 3712 |
)
|
| 3713 |
except HfBucketValidationError as exc:
|
| 3714 |
return (
|
| 3715 |
-
f"Project state could not be connected in
|
| 3716 |
_project_rows(),
|
| 3717 |
gr.update(),
|
| 3718 |
gr.update(),
|
|
@@ -3866,12 +3904,12 @@ def create_app() -> gr.Blocks:
|
|
| 3866 |
if project is None:
|
| 3867 |
return "Select a valid project.", gr.update(), gr.update(), gr.update(), gr.update()
|
| 3868 |
if (
|
| 3869 |
-
runtime_config.
|
| 3870 |
and (project.validation_backend or "").strip() == HF_BUCKET_VALIDATION_BACKEND
|
| 3871 |
):
|
| 3872 |
return (
|
| 3873 |
-
"
|
| 3874 |
-
"
|
| 3875 |
gr.update(value=""),
|
| 3876 |
gr.update(value=False),
|
| 3877 |
_project_rows(),
|
|
@@ -4292,7 +4330,7 @@ def create_app() -> gr.Blocks:
|
|
| 4292 |
|
| 4293 |
session_state.change(
|
| 4294 |
fn=lambda s: (
|
| 4295 |
-
gr.update(visible=bool(s is not None and not runtime_config.
|
| 4296 |
gr.update(visible=bool(s is not None)),
|
| 4297 |
gr.update(visible=bool(s is not None)),
|
| 4298 |
gr.update(visible=bool(s is not None)),
|
|
@@ -4347,7 +4385,7 @@ def create_app() -> gr.Blocks:
|
|
| 4347 |
allow_custom_value=True,
|
| 4348 |
)
|
| 4349 |
with gr.Group(
|
| 4350 |
-
visible=not runtime_config.
|
| 4351 |
elem_classes=["bn-panel-soft"],
|
| 4352 |
):
|
| 4353 |
gr.Markdown("### Private state OAuth diagnostic")
|
|
@@ -4360,16 +4398,16 @@ def create_app() -> gr.Blocks:
|
|
| 4360 |
elem_classes=["bn-soft-action"],
|
| 4361 |
)
|
| 4362 |
with gr.Group(
|
| 4363 |
-
visible=runtime_config.
|
| 4364 |
elem_classes=["bn-panel-soft"],
|
| 4365 |
):
|
| 4366 |
-
gr.Markdown("### Private
|
| 4367 |
trusted_storage_status = gr.Markdown(
|
| 4368 |
-
"
|
| 4369 |
-
"
|
| 4370 |
)
|
| 4371 |
test_trusted_storage_btn = gr.Button(
|
| 4372 |
-
"Check
|
| 4373 |
elem_classes=["bn-soft-action"],
|
| 4374 |
)
|
| 4375 |
invitations_info = gr.Markdown(value="")
|
|
@@ -4588,7 +4626,7 @@ def create_app() -> gr.Blocks:
|
|
| 4588 |
|
| 4589 |
def update_selected_project(selected: str, session):
|
| 4590 |
"""Update state when project is selected."""
|
| 4591 |
-
if session and selected:
|
| 4592 |
selected_project = admin_manager.get_project(selected)
|
| 4593 |
dataset_repo_id = selected_project.dataset_repo_id if selected_project else ""
|
| 4594 |
role = auth_service.get_user_role_for_project(session.username, selected)
|
|
@@ -4627,24 +4665,30 @@ def create_app() -> gr.Blocks:
|
|
| 4627 |
|
| 4628 |
def test_trusted_storage_authorization(selected: str, session):
|
| 4629 |
if session is None:
|
| 4630 |
-
return "Login
|
| 4631 |
project_slug = (selected or "").strip()
|
| 4632 |
project = admin_manager.get_project(project_slug) if project_slug else None
|
| 4633 |
if project is None or project_slug not in session.authorized_projects:
|
| 4634 |
return "Select a project that is authorized for the signed-in account."
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4635 |
if (project.validation_backend or "").strip() != HF_BUCKET_VALIDATION_BACKEND:
|
| 4636 |
return "This project does not use private Hugging Face Bucket validation storage."
|
| 4637 |
try:
|
| 4638 |
result = hf_bucket_permission_probe.probe(
|
| 4639 |
bucket_id=project.validation_bucket_id or "",
|
| 4640 |
actor_username=session.username,
|
| 4641 |
-
token=
|
| 4642 |
)
|
| 4643 |
except HfBucketValidationError as exc:
|
| 4644 |
return f"Private storage access test failed: {exc}"
|
| 4645 |
return (
|
| 4646 |
-
f"
|
| 4647 |
-
f"
|
|
|
|
| 4648 |
)
|
| 4649 |
|
| 4650 |
project_selector.change(
|
|
@@ -4887,6 +4931,10 @@ def create_app() -> gr.Blocks:
|
|
| 4887 |
return auth_service.get_hf_token_for_user(session.username)
|
| 4888 |
|
| 4889 |
def _project_fetch_token(project_slug: str, session) -> str | None:
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4890 |
session_token = _session_hf_token(session)
|
| 4891 |
project = admin_manager.get_project(project_slug) if project_slug else None
|
| 4892 |
return _resolve_project_fetch_token(project, session_token)
|
|
@@ -4904,6 +4952,8 @@ def create_app() -> gr.Blocks:
|
|
| 4904 |
):
|
| 4905 |
if not project_slug:
|
| 4906 |
return [], "", 1
|
|
|
|
|
|
|
| 4907 |
species_name = (species or "").strip()
|
| 4908 |
if not species_name:
|
| 4909 |
return [], "Select a species to start validation", 1
|
|
@@ -4978,6 +5028,9 @@ def create_app() -> gr.Blocks:
|
|
| 4978 |
if not project_slug:
|
| 4979 |
return gr.update(choices=[], value=None, interactive=False), [], "", 1, None, None, _spectrogram_title(None, None), _build_validation_summary_cards([]), gr.update(choices=["Noise", "Undetermined"], value=None), []
|
| 4980 |
|
|
|
|
|
|
|
|
|
|
| 4981 |
warning = _ensure_project_queue_loaded(project_slug, session)
|
| 4982 |
|
| 4983 |
species_options = _extract_species_options_from_queue(
|
|
@@ -5021,6 +5074,8 @@ def create_app() -> gr.Blocks:
|
|
| 5021 |
validator_name_value = _validator_name_from_session(session)
|
| 5022 |
if not validator_name_value:
|
| 5023 |
return "Login before validating", cache_key, None, rows, page, idx, "", ""
|
|
|
|
|
|
|
| 5024 |
result = _save_selected_validation_with_refresh(
|
| 5025 |
validation_service=validation_service,
|
| 5026 |
audio_service=audio_service,
|
|
@@ -5061,9 +5116,12 @@ def create_app() -> gr.Blocks:
|
|
| 5061 |
status_filter_value: str,
|
| 5062 |
updated_after_value: object,
|
| 5063 |
only_conflicts: bool,
|
|
|
|
| 5064 |
):
|
| 5065 |
if not project_slug:
|
| 5066 |
return "Select a project before reapplying", cache_key, None, rows, page, idx, pending_status, conflict_key
|
|
|
|
|
|
|
| 5067 |
result = _reapply_last_conflict_validation_with_refresh(
|
| 5068 |
validation_service=validation_service,
|
| 5069 |
audio_service=audio_service,
|
|
@@ -5101,9 +5159,12 @@ def create_app() -> gr.Blocks:
|
|
| 5101 |
validator_filter_value: str,
|
| 5102 |
status_filter_value: str,
|
| 5103 |
updated_after_value: object,
|
|
|
|
| 5104 |
):
|
| 5105 |
if not project_slug:
|
| 5106 |
return "Select a project before validating", cache_key, None, rows, page
|
|
|
|
|
|
|
| 5107 |
result = _batch_validate_conflicts(
|
| 5108 |
validation_service=validation_service,
|
| 5109 |
audio_service=audio_service,
|
|
@@ -5128,6 +5189,8 @@ def create_app() -> gr.Blocks:
|
|
| 5128 |
def build_report_for_project(project_slug: str, session=None) -> str:
|
| 5129 |
if not project_slug:
|
| 5130 |
return "Select a project to generate report"
|
|
|
|
|
|
|
| 5131 |
return _build_validation_report(validation_repository, project_slug, _validator_name_from_session(session))
|
| 5132 |
|
| 5133 |
def save_corrected_species_option(
|
|
@@ -5194,6 +5257,8 @@ def create_app() -> gr.Blocks:
|
|
| 5194 |
return gr.update(value="Favorite", variant="secondary")
|
| 5195 |
|
| 5196 |
def on_table_select(project_slug: str, repo: str, rows: object, cache_key: str, session, evt: gr.SelectData):
|
|
|
|
|
|
|
| 5197 |
return _select_and_fetch_audio_with_title(
|
| 5198 |
audio_service=audio_service,
|
| 5199 |
dataset_repo=repo,
|
|
@@ -5262,7 +5327,7 @@ def create_app() -> gr.Blocks:
|
|
| 5262 |
)
|
| 5263 |
|
| 5264 |
species_filter.change(
|
| 5265 |
-
fn=lambda project_slug, species, confidence, validator_filter_value, status_filter_value, updated_after_value, only_conflicts: refresh(
|
| 5266 |
project_slug,
|
| 5267 |
1,
|
| 5268 |
species,
|
|
@@ -5271,6 +5336,7 @@ def create_app() -> gr.Blocks:
|
|
| 5271 |
status_filter_value,
|
| 5272 |
updated_after_value,
|
| 5273 |
only_conflicts,
|
|
|
|
| 5274 |
),
|
| 5275 |
inputs=[
|
| 5276 |
selected_project_state,
|
|
@@ -5280,6 +5346,7 @@ def create_app() -> gr.Blocks:
|
|
| 5280 |
validation_status_filter,
|
| 5281 |
updated_after_filter,
|
| 5282 |
show_conflicts_only,
|
|
|
|
| 5283 |
],
|
| 5284 |
outputs=[table, status, page_state],
|
| 5285 |
).then(
|
|
@@ -5739,7 +5806,7 @@ def create_app() -> gr.Blocks:
|
|
| 5739 |
slug = (project_slug or "").strip()
|
| 5740 |
if session is None:
|
| 5741 |
return None, "Login before exporting project data."
|
| 5742 |
-
if not
|
| 5743 |
return None, "Choose an authorized project before exporting."
|
| 5744 |
|
| 5745 |
project = _project_map().get(slug)
|
|
@@ -5791,6 +5858,16 @@ def create_app() -> gr.Blocks:
|
|
| 5791 |
1,
|
| 5792 |
"Select a project to view the dashboard",
|
| 5793 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5794 |
|
| 5795 |
warning = _ensure_project_queue_loaded(slug, session)
|
| 5796 |
signature = loaded_project_signatures.get(slug, "")
|
|
@@ -6010,15 +6087,15 @@ def create_app() -> gr.Blocks:
|
|
| 6010 |
def _render_settings_health():
|
| 6011 |
backend = (runtime_config.state_backend or "filesystem").strip().lower()
|
| 6012 |
supabase_ready = bool(runtime_config.supabase_url and runtime_config.supabase_service_role_key)
|
| 6013 |
-
if runtime_config.
|
| 6014 |
-
state_label = "HF
|
| 6015 |
state_tone = "ok"
|
| 6016 |
else:
|
| 6017 |
state_label = "Supabase" if backend in {"supabase", "postgres", "postgresql"} and supabase_ready else "Filesystem"
|
| 6018 |
state_tone = "ok" if state_label == "Supabase" else "warn"
|
| 6019 |
normalized_bootstrap_dir = str(runtime_config.bootstrap_base_dir).replace("\\", "/").rstrip("/")
|
| 6020 |
filesystem_durable = normalized_bootstrap_dir == "/data" or normalized_bootstrap_dir.startswith("/data/")
|
| 6021 |
-
if state_label == "HF
|
| 6022 |
durability_label = "private HF state/Bucket"
|
| 6023 |
durability_tone = "ok"
|
| 6024 |
elif state_label == "Supabase":
|
|
@@ -6035,8 +6112,8 @@ def create_app() -> gr.Blocks:
|
|
| 6035 |
invite_email_label = "enabled" if runtime_config.invite_email_enabled and runtime_config.emailjs_enabled else "disabled"
|
| 6036 |
hf_project_state_label = "enabled" if hf_state_writes_enabled else "disabled"
|
| 6037 |
hf_bucket_validation_label = (
|
| 6038 |
-
"
|
| 6039 |
-
if runtime_config.
|
| 6040 |
else ("legacy experimental" if runtime_config.hf_bucket_validations_enabled else "disabled")
|
| 6041 |
)
|
| 6042 |
hf_project_state_repos_label = str(len(runtime_config.hf_project_state_repos))
|
|
@@ -6045,11 +6122,11 @@ def create_app() -> gr.Blocks:
|
|
| 6045 |
[
|
| 6046 |
("State backend", state_label, state_tone),
|
| 6047 |
("HF project-state writes", hf_project_state_label, "ok" if hf_state_writes_enabled else "info"),
|
| 6048 |
-
("HF Bucket validations", hf_bucket_validation_label, "ok" if runtime_config.
|
| 6049 |
(
|
| 6050 |
"HF project-state repos",
|
| 6051 |
hf_project_state_repos_label,
|
| 6052 |
-
"ok" if runtime_config.hf_project_state_repos else ("warn" if runtime_config.
|
| 6053 |
),
|
| 6054 |
("Auth mode", auth_mode_label, "ok" if not allow_username_login else "warn"),
|
| 6055 |
("Destructive write guard", "enabled", "ok"),
|
|
@@ -6066,17 +6143,17 @@ def create_app() -> gr.Blocks:
|
|
| 6066 |
)
|
| 6067 |
status_text = (
|
| 6068 |
(
|
| 6069 |
-
"HF
|
| 6070 |
-
"
|
| 6071 |
+ (
|
| 6072 |
"no project-state repository is configured, so projects will not be restored after a rebuild."
|
| 6073 |
if not runtime_config.hf_project_state_repos
|
| 6074 |
else "configured state repositories will be used for restart recovery."
|
| 6075 |
)
|
| 6076 |
-
if state_label == "HF
|
| 6077 |
else "Supabase persistence is active. Project and ACL removals are guarded against accidental broad overwrites."
|
| 6078 |
)
|
| 6079 |
-
if state_label in {"HF
|
| 6080 |
else (
|
| 6081 |
"Filesystem persistence is active with local JSON backups and destructive-write guards. "
|
| 6082 |
"On free Spaces without persistent storage, local files are still not durable across rebuilds."
|
|
|
|
| 452 |
return token or None
|
| 453 |
|
| 454 |
|
| 455 |
+
def _storage_hf_token() -> str | None:
|
| 456 |
+
"""Return the server-side credential used only for admin-owned private project storage."""
|
| 457 |
+
token = (os.getenv("BIRDNET_HF_STORAGE_TOKEN") or "").strip()
|
| 458 |
+
return token or _env_hf_token()
|
| 459 |
+
|
| 460 |
+
|
| 461 |
def _project_dataset_token(project: Project, fallback_token: str | None = None) -> str | None:
|
| 462 |
return (project.dataset_token or "").strip() or (fallback_token or "").strip() or _env_hf_token()
|
| 463 |
|
|
|
|
| 1567 |
return bool((os.getenv("SPACE_ID") or os.getenv("SPACE_HOST") or "").strip())
|
| 1568 |
|
| 1569 |
|
| 1570 |
+
def _validate_hf_admin_storage_source_dataset(
|
| 1571 |
*,
|
| 1572 |
project: Project,
|
| 1573 |
token: str | None,
|
|
|
|
| 1575 |
) -> None:
|
| 1576 |
if (project.visibility or "").strip().lower() != "collaborative":
|
| 1577 |
raise HfBucketValidationError(
|
| 1578 |
+
"Admin-owned private storage projects must use app visibility 'collaborative'. "
|
| 1579 |
+
"The Hugging Face dataset and storage remain private; this app visibility enables assigned validators."
|
|
|
|
| 1580 |
)
|
| 1581 |
token_value = (token or "").strip()
|
| 1582 |
if not token_value:
|
| 1583 |
+
raise HfBucketValidationError(
|
| 1584 |
+
"Admin-owned HF storage requires BIRDNET_HF_STORAGE_TOKEN to be configured as a Space secret."
|
| 1585 |
+
)
|
| 1586 |
|
| 1587 |
client = api or HfApi()
|
| 1588 |
namespace = project.dataset_repo_id.split("/", 1)[0].strip()
|
| 1589 |
try:
|
| 1590 |
+
identity = client.whoami(token=token_value)
|
| 1591 |
except Exception as exc:
|
| 1592 |
raise HfBucketValidationError(
|
| 1593 |
+
f"Could not verify the configured administrator storage credential: {exc}"
|
|
|
|
| 1594 |
) from exc
|
| 1595 |
+
owner_username = str(identity.get("name") if isinstance(identity, dict) else "").strip()
|
| 1596 |
+
if not owner_username or owner_username.casefold() != namespace.casefold():
|
| 1597 |
+
raise HfBucketValidationError(
|
| 1598 |
+
"Admin-owned HF storage requires the dataset to be in the personal namespace of the configured "
|
| 1599 |
+
f"storage account. Expected '{owner_username or 'authenticated user'}/...', received "
|
| 1600 |
+
f"'{project.dataset_repo_id}'."
|
| 1601 |
+
)
|
| 1602 |
try:
|
| 1603 |
dataset_info = client.repo_info(
|
| 1604 |
repo_id=project.dataset_repo_id,
|
|
|
|
| 1609 |
raise HfBucketValidationError(f"Could not verify the Hugging Face dataset before creating private state: {exc}") from exc
|
| 1610 |
if not bool(getattr(dataset_info, "private", False)):
|
| 1611 |
raise HfBucketValidationError(
|
| 1612 |
+
"Admin-owned HF storage only accepts a private dataset. Make the dataset private in Hugging Face "
|
| 1613 |
"before creating the validation project."
|
| 1614 |
)
|
| 1615 |
|
| 1616 |
|
| 1617 |
+
def _initialize_hf_admin_storage(
|
| 1618 |
*,
|
| 1619 |
project: Project,
|
| 1620 |
token: str | None,
|
| 1621 |
api: HfApi | None = None,
|
| 1622 |
bucket_initializer: HfBucketValidationInitializer | None = None,
|
| 1623 |
):
|
| 1624 |
+
_validate_hf_admin_storage_source_dataset(project=project, token=token, api=api)
|
| 1625 |
token_value = (token or "").strip()
|
| 1626 |
initializer = bucket_initializer or HfBucketValidationInitializer()
|
| 1627 |
result = initializer.initialize(
|
|
|
|
| 2980 |
|
| 2981 |
projects_file_path, user_access_file_path, invites_file_path = _resolve_bootstrap_file_paths(runtime_config)
|
| 2982 |
state_store, supabase_validation_repository, state_backend_message = _build_supabase_state(runtime_config)
|
| 2983 |
+
if runtime_config.hf_admin_storage_mode_enabled:
|
| 2984 |
+
state_backend_message = "HF admin-owned private storage enabled."
|
| 2985 |
allow_username_login, auth_mode_label, auth_mode_description = _resolve_username_login_policy(runtime_config)
|
| 2986 |
bootstrap_warning = _bootstrap_auth_and_projects(
|
| 2987 |
auth_service,
|
|
|
|
| 2991 |
user_access_file_path=str(user_access_file_path),
|
| 2992 |
invites_file_path=str(invites_file_path),
|
| 2993 |
state_store=state_store,
|
| 2994 |
+
hf_project_state_token=_storage_hf_token() if runtime_config.hf_admin_storage_mode_enabled else _env_hf_token(),
|
| 2995 |
)
|
| 2996 |
|
| 2997 |
def _current_project_map() -> dict[str, Project]:
|
|
|
|
| 3016 |
max_loaded_projects = 3
|
| 3017 |
audio_service = AudioFetchService(EphemeralCacheManager(ttl_seconds=300, max_files=128))
|
| 3018 |
base_validation_repository = supabase_validation_repository or AppendOnlyValidationRepository(base_dir=runtime_config.validation_base_dir)
|
| 3019 |
+
hf_state_writes_enabled = runtime_config.hf_project_state_writes_enabled or runtime_config.hf_admin_storage_mode_enabled
|
| 3020 |
+
hf_bucket_validations_enabled = runtime_config.hf_bucket_validations_enabled or runtime_config.hf_admin_storage_mode_enabled
|
| 3021 |
validation_repository = ProjectAwareValidationRepository(
|
| 3022 |
fallback_repository=base_validation_repository,
|
| 3023 |
project_lookup=admin_manager.get_project,
|
| 3024 |
token_provider=lambda project: (
|
| 3025 |
+
(_storage_hf_token() or "").strip()
|
| 3026 |
+
if runtime_config.hf_admin_storage_mode_enabled
|
| 3027 |
+
and (project.validation_backend or "").strip() == HF_BUCKET_VALIDATION_BACKEND
|
| 3028 |
+
else (
|
| 3029 |
+
(project.dataset_token or "").strip()
|
| 3030 |
+
or (auth_service.get_hf_token_for_user((project.owner_username or "").strip()) or "").strip()
|
| 3031 |
+
or (_env_hf_token() or "").strip()
|
| 3032 |
+
)
|
| 3033 |
+
) or None,
|
| 3034 |
actor_token_provider=lambda _project, username: auth_service.get_hf_token_for_user(username),
|
| 3035 |
enable_hf_project_state=hf_state_writes_enabled,
|
| 3036 |
enable_hf_bucket_validations=hf_bucket_validations_enabled,
|
| 3037 |
+
use_backend_token_for_bucket=runtime_config.hf_admin_storage_mode_enabled,
|
| 3038 |
)
|
| 3039 |
validation_service = ValidationService(validation_repository)
|
| 3040 |
hf_state_initializer = HfProjectStateStoreInitializer()
|
|
|
|
| 3104 |
role = auth_service.get_user_role_for_project(session.username, slug)
|
| 3105 |
return role == Role.admin
|
| 3106 |
|
| 3107 |
+
def _can_access_project(session, project_slug: str) -> bool:
|
| 3108 |
+
if session is None:
|
| 3109 |
+
return False
|
| 3110 |
+
slug = (project_slug or "").strip()
|
| 3111 |
+
if not slug or slug not in session.authorized_projects:
|
| 3112 |
+
return False
|
| 3113 |
+
return auth_service.get_user_role_for_project(session.username, slug) in {Role.admin, Role.validator}
|
| 3114 |
+
|
| 3115 |
def _refresh_session_copy(session):
|
| 3116 |
if session is None:
|
| 3117 |
return None
|
|
|
|
| 3180 |
slug = (project_slug or "").strip()
|
| 3181 |
if not slug:
|
| 3182 |
return ""
|
| 3183 |
+
if not _can_access_project(session, slug):
|
| 3184 |
+
return "Access denied. Select a project assigned to your signed-in account."
|
| 3185 |
|
| 3186 |
token = _project_fetch_token(slug, session)
|
| 3187 |
signature = _project_queue_signature(slug, token)
|
|
|
|
| 3203 |
return warning
|
| 3204 |
|
| 3205 |
def _project_state_write_token(project: Project, session=None) -> str | None:
|
| 3206 |
+
if runtime_config.hf_admin_storage_mode_enabled:
|
| 3207 |
+
return _storage_hf_token()
|
| 3208 |
session_username = str(getattr(session, "username", "") or "").strip()
|
| 3209 |
session_token = auth_service.get_hf_token_for_user(session_username) if session_username else None
|
| 3210 |
owner_username = (project.owner_username or "").strip()
|
|
|
|
| 3314 |
"Secure access",
|
| 3315 |
"Sign in with your Hugging Face identity",
|
| 3316 |
(
|
| 3317 |
+
"Validators sign in with their own identity; private datasets and validation storage remain accessible only through this app."
|
| 3318 |
+
if runtime_config.hf_admin_storage_mode_enabled
|
| 3319 |
else "Your account controls project access, validator attribution, and private dataset access."
|
| 3320 |
),
|
| 3321 |
)
|
|
|
|
| 3325 |
allow_username_login=allow_username_login,
|
| 3326 |
enable_oauth_login=_is_running_in_hf_space(),
|
| 3327 |
auth_mode_label=(
|
| 3328 |
+
"Private HF administrator storage is active. Your login verifies identity only; "
|
| 3329 |
+
"project access is granted by assignments or invitations inside this app."
|
| 3330 |
+
if runtime_config.hf_admin_storage_mode_enabled
|
| 3331 |
else auth_mode_description
|
| 3332 |
),
|
| 3333 |
+
admin_storage_mode=runtime_config.hf_admin_storage_mode_enabled,
|
| 3334 |
)
|
| 3335 |
|
| 3336 |
# Store session ID when login succeeds
|
|
|
|
| 3409 |
class_name="bn-panel-soft",
|
| 3410 |
)
|
| 3411 |
)
|
| 3412 |
+
if runtime_config.hf_admin_storage_mode_enabled:
|
| 3413 |
gr.HTML(
|
| 3414 |
inline_hint_html(
|
| 3415 |
+
"Private administrator-owned HF storage is active. Use a private dataset in your "
|
| 3416 |
+
"personal Hugging Face namespace and collaborative visibility in this app. "
|
| 3417 |
+
"Validators are assigned here and do not need direct Hub repository access.",
|
|
|
|
| 3418 |
"info",
|
| 3419 |
)
|
| 3420 |
)
|
|
|
|
| 3438 |
)
|
| 3439 |
create_project_token = gr.Textbox(
|
| 3440 |
label=(
|
| 3441 |
+
"Storage credential configured as Space secret"
|
| 3442 |
+
if runtime_config.hf_admin_storage_mode_enabled
|
| 3443 |
else "Project HF Token (optional)"
|
| 3444 |
),
|
| 3445 |
placeholder="hf_xxx...",
|
| 3446 |
type="password",
|
| 3447 |
+
visible=not runtime_config.hf_admin_storage_mode_enabled,
|
| 3448 |
)
|
| 3449 |
|
| 3450 |
create_project_message = gr.Markdown()
|
|
|
|
| 3489 |
repo_id = (repo_id or "").strip()
|
| 3490 |
visibility_value = (visibility or "collaborative").strip().lower()
|
| 3491 |
explicit_project_token = (project_token or "").strip() or None
|
| 3492 |
+
operation_token = (
|
| 3493 |
+
_storage_hf_token()
|
| 3494 |
+
if runtime_config.hf_admin_storage_mode_enabled
|
| 3495 |
+
else explicit_project_token or _session_hf_token(session) or _env_hf_token()
|
| 3496 |
+
)
|
| 3497 |
if not slug or not name or not repo_id:
|
| 3498 |
return "Fill slug, name, and repo id.", gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), session
|
| 3499 |
if visibility_value not in {"private", "collaborative"}:
|
|
|
|
| 3501 |
if (
|
| 3502 |
_is_running_in_hf_space()
|
| 3503 |
and hf_state_writes_enabled
|
| 3504 |
+
and not runtime_config.hf_admin_storage_mode_enabled
|
| 3505 |
and getattr(session, "authentication_method", "") != "oauth"
|
| 3506 |
):
|
| 3507 |
return (
|
|
|
|
| 3545 |
dataset_repo_id=repo_id,
|
| 3546 |
visibility=visibility_value,
|
| 3547 |
owner_username=session.username,
|
| 3548 |
+
dataset_token=None if runtime_config.hf_admin_storage_mode_enabled else explicit_project_token,
|
| 3549 |
active=True,
|
| 3550 |
)
|
| 3551 |
bucket_result = None
|
| 3552 |
+
if runtime_config.hf_admin_storage_mode_enabled:
|
| 3553 |
try:
|
| 3554 |
+
bucket_result = _initialize_hf_admin_storage(
|
| 3555 |
project=project,
|
| 3556 |
token=operation_token,
|
| 3557 |
bucket_initializer=hf_bucket_initializer,
|
|
|
|
| 3626 |
recovery_message = (
|
| 3627 |
f" Add `{state_result.state_repo_id}` to `BIRDNET_HF_PROJECT_STATE_REPOS` before rebuilding "
|
| 3628 |
"the Space so this project is restored after restart."
|
| 3629 |
+
if runtime_config.hf_admin_storage_mode_enabled
|
| 3630 |
else ""
|
| 3631 |
)
|
| 3632 |
|
|
|
|
| 3708 |
try:
|
| 3709 |
loaded = hf_state_connector.connect_admin_project(
|
| 3710 |
state_repo_id=repo_id,
|
| 3711 |
+
token=(
|
| 3712 |
+
_storage_hf_token()
|
| 3713 |
+
if runtime_config.hf_admin_storage_mode_enabled
|
| 3714 |
+
else _session_hf_token(session)
|
| 3715 |
+
),
|
| 3716 |
actor_username=session.username,
|
| 3717 |
)
|
| 3718 |
except HfProjectStateStoreError as exc:
|
|
|
|
| 3727 |
session,
|
| 3728 |
)
|
| 3729 |
|
| 3730 |
+
if runtime_config.hf_admin_storage_mode_enabled:
|
| 3731 |
if (
|
| 3732 |
(loaded.project.validation_backend or "").strip() != HF_BUCKET_VALIDATION_BACKEND
|
| 3733 |
or not (loaded.project.validation_bucket_id or "").strip()
|
| 3734 |
):
|
| 3735 |
return (
|
| 3736 |
+
"Project state could not be connected in administrator-storage mode because its "
|
| 3737 |
+
"manifest does not declare private Hugging Face Bucket validation storage.",
|
| 3738 |
_project_rows(),
|
| 3739 |
gr.update(),
|
| 3740 |
gr.update(),
|
|
|
|
| 3744 |
session,
|
| 3745 |
)
|
| 3746 |
try:
|
| 3747 |
+
_validate_hf_admin_storage_source_dataset(
|
| 3748 |
project=loaded.project,
|
| 3749 |
+
token=_storage_hf_token(),
|
| 3750 |
)
|
| 3751 |
except HfBucketValidationError as exc:
|
| 3752 |
return (
|
| 3753 |
+
f"Project state could not be connected in administrator-storage mode: {exc}",
|
| 3754 |
_project_rows(),
|
| 3755 |
gr.update(),
|
| 3756 |
gr.update(),
|
|
|
|
| 3904 |
if project is None:
|
| 3905 |
return "Select a valid project.", gr.update(), gr.update(), gr.update(), gr.update()
|
| 3906 |
if (
|
| 3907 |
+
runtime_config.hf_admin_storage_mode_enabled
|
| 3908 |
and (project.validation_backend or "").strip() == HF_BUCKET_VALIDATION_BACKEND
|
| 3909 |
):
|
| 3910 |
return (
|
| 3911 |
+
"Administrator-owned HF projects never store a project token. Private storage access "
|
| 3912 |
+
"uses only the configured Space secret; validator logins verify identity.",
|
| 3913 |
gr.update(value=""),
|
| 3914 |
gr.update(value=False),
|
| 3915 |
_project_rows(),
|
|
|
|
| 4330 |
|
| 4331 |
session_state.change(
|
| 4332 |
fn=lambda s: (
|
| 4333 |
+
gr.update(visible=bool(s is not None and not runtime_config.hf_admin_storage_mode_enabled)),
|
| 4334 |
gr.update(visible=bool(s is not None)),
|
| 4335 |
gr.update(visible=bool(s is not None)),
|
| 4336 |
gr.update(visible=bool(s is not None)),
|
|
|
|
| 4385 |
allow_custom_value=True,
|
| 4386 |
)
|
| 4387 |
with gr.Group(
|
| 4388 |
+
visible=not runtime_config.hf_admin_storage_mode_enabled,
|
| 4389 |
elem_classes=["bn-panel-soft"],
|
| 4390 |
):
|
| 4391 |
gr.Markdown("### Private state OAuth diagnostic")
|
|
|
|
| 4398 |
elem_classes=["bn-soft-action"],
|
| 4399 |
)
|
| 4400 |
with gr.Group(
|
| 4401 |
+
visible=runtime_config.hf_admin_storage_mode_enabled,
|
| 4402 |
elem_classes=["bn-panel-soft"],
|
| 4403 |
):
|
| 4404 |
+
gr.Markdown("### Private storage backend health")
|
| 4405 |
trusted_storage_status = gr.Markdown(
|
| 4406 |
+
"Administrators can verify that this Space can read and write the project's private "
|
| 4407 |
+
"validation storage. Validators do not require direct Hub storage access."
|
| 4408 |
)
|
| 4409 |
test_trusted_storage_btn = gr.Button(
|
| 4410 |
+
"Check backend storage access",
|
| 4411 |
elem_classes=["bn-soft-action"],
|
| 4412 |
)
|
| 4413 |
invitations_info = gr.Markdown(value="")
|
|
|
|
| 4626 |
|
| 4627 |
def update_selected_project(selected: str, session):
|
| 4628 |
"""Update state when project is selected."""
|
| 4629 |
+
if session and selected and selected in session.authorized_projects:
|
| 4630 |
selected_project = admin_manager.get_project(selected)
|
| 4631 |
dataset_repo_id = selected_project.dataset_repo_id if selected_project else ""
|
| 4632 |
role = auth_service.get_user_role_for_project(session.username, selected)
|
|
|
|
| 4665 |
|
| 4666 |
def test_trusted_storage_authorization(selected: str, session):
|
| 4667 |
if session is None:
|
| 4668 |
+
return "Login before testing private validation storage."
|
| 4669 |
project_slug = (selected or "").strip()
|
| 4670 |
project = admin_manager.get_project(project_slug) if project_slug else None
|
| 4671 |
if project is None or project_slug not in session.authorized_projects:
|
| 4672 |
return "Select a project that is authorized for the signed-in account."
|
| 4673 |
+
if not _is_admin_for_project(session, project_slug):
|
| 4674 |
+
return (
|
| 4675 |
+
"Only a project administrator can run this storage health check. "
|
| 4676 |
+
"Validators receive data through the app after assignment."
|
| 4677 |
+
)
|
| 4678 |
if (project.validation_backend or "").strip() != HF_BUCKET_VALIDATION_BACKEND:
|
| 4679 |
return "This project does not use private Hugging Face Bucket validation storage."
|
| 4680 |
try:
|
| 4681 |
result = hf_bucket_permission_probe.probe(
|
| 4682 |
bucket_id=project.validation_bucket_id or "",
|
| 4683 |
actor_username=session.username,
|
| 4684 |
+
token=_storage_hf_token(),
|
| 4685 |
)
|
| 4686 |
except HfBucketValidationError as exc:
|
| 4687 |
return f"Private storage access test failed: {exc}"
|
| 4688 |
return (
|
| 4689 |
+
f"Backend storage verified for **{result.actor_username}**. "
|
| 4690 |
+
f"The Space read and wrote `{result.bucket_id}` using its protected storage credential; "
|
| 4691 |
+
"the diagnostic marker was removed."
|
| 4692 |
)
|
| 4693 |
|
| 4694 |
project_selector.change(
|
|
|
|
| 4931 |
return auth_service.get_hf_token_for_user(session.username)
|
| 4932 |
|
| 4933 |
def _project_fetch_token(project_slug: str, session) -> str | None:
|
| 4934 |
+
if not _can_access_project(session, project_slug):
|
| 4935 |
+
return None
|
| 4936 |
+
if runtime_config.hf_admin_storage_mode_enabled:
|
| 4937 |
+
return _storage_hf_token()
|
| 4938 |
session_token = _session_hf_token(session)
|
| 4939 |
project = admin_manager.get_project(project_slug) if project_slug else None
|
| 4940 |
return _resolve_project_fetch_token(project, session_token)
|
|
|
|
| 4952 |
):
|
| 4953 |
if not project_slug:
|
| 4954 |
return [], "", 1
|
| 4955 |
+
if not _can_access_project(session, project_slug):
|
| 4956 |
+
return [], "Access denied. Select a project assigned to your signed-in account.", 1
|
| 4957 |
species_name = (species or "").strip()
|
| 4958 |
if not species_name:
|
| 4959 |
return [], "Select a species to start validation", 1
|
|
|
|
| 5028 |
if not project_slug:
|
| 5029 |
return gr.update(choices=[], value=None, interactive=False), [], "", 1, None, None, _spectrogram_title(None, None), _build_validation_summary_cards([]), gr.update(choices=["Noise", "Undetermined"], value=None), []
|
| 5030 |
|
| 5031 |
+
if not _can_access_project(session, project_slug):
|
| 5032 |
+
return gr.update(choices=[], value=None, interactive=False), [], "Access denied. Select an assigned project.", 1, None, None, _spectrogram_title(None, None), _build_validation_summary_cards([]), gr.update(choices=["Noise", "Undetermined"], value=None), []
|
| 5033 |
+
|
| 5034 |
warning = _ensure_project_queue_loaded(project_slug, session)
|
| 5035 |
|
| 5036 |
species_options = _extract_species_options_from_queue(
|
|
|
|
| 5074 |
validator_name_value = _validator_name_from_session(session)
|
| 5075 |
if not validator_name_value:
|
| 5076 |
return "Login before validating", cache_key, None, rows, page, idx, "", ""
|
| 5077 |
+
if not _can_access_project(session, project_slug):
|
| 5078 |
+
return "Access denied for this project", cache_key, None, rows, page, idx, "", ""
|
| 5079 |
result = _save_selected_validation_with_refresh(
|
| 5080 |
validation_service=validation_service,
|
| 5081 |
audio_service=audio_service,
|
|
|
|
| 5116 |
status_filter_value: str,
|
| 5117 |
updated_after_value: object,
|
| 5118 |
only_conflicts: bool,
|
| 5119 |
+
session=None,
|
| 5120 |
):
|
| 5121 |
if not project_slug:
|
| 5122 |
return "Select a project before reapplying", cache_key, None, rows, page, idx, pending_status, conflict_key
|
| 5123 |
+
if not _can_access_project(session, project_slug):
|
| 5124 |
+
return "Access denied for this project", cache_key, None, rows, page, idx, pending_status, conflict_key
|
| 5125 |
result = _reapply_last_conflict_validation_with_refresh(
|
| 5126 |
validation_service=validation_service,
|
| 5127 |
audio_service=audio_service,
|
|
|
|
| 5159 |
validator_filter_value: str,
|
| 5160 |
status_filter_value: str,
|
| 5161 |
updated_after_value: object,
|
| 5162 |
+
session=None,
|
| 5163 |
):
|
| 5164 |
if not project_slug:
|
| 5165 |
return "Select a project before validating", cache_key, None, rows, page
|
| 5166 |
+
if not _can_access_project(session, project_slug):
|
| 5167 |
+
return "Access denied for this project", cache_key, None, rows, page
|
| 5168 |
result = _batch_validate_conflicts(
|
| 5169 |
validation_service=validation_service,
|
| 5170 |
audio_service=audio_service,
|
|
|
|
| 5189 |
def build_report_for_project(project_slug: str, session=None) -> str:
|
| 5190 |
if not project_slug:
|
| 5191 |
return "Select a project to generate report"
|
| 5192 |
+
if not _can_access_project(session, project_slug):
|
| 5193 |
+
return "Access denied for this project"
|
| 5194 |
return _build_validation_report(validation_repository, project_slug, _validator_name_from_session(session))
|
| 5195 |
|
| 5196 |
def save_corrected_species_option(
|
|
|
|
| 5257 |
return gr.update(value="Favorite", variant="secondary")
|
| 5258 |
|
| 5259 |
def on_table_select(project_slug: str, repo: str, rows: object, cache_key: str, session, evt: gr.SelectData):
|
| 5260 |
+
if not _can_access_project(session, project_slug):
|
| 5261 |
+
return None, None, cache_key, "Access denied for this project", None, _spectrogram_title(None, None)
|
| 5262 |
return _select_and_fetch_audio_with_title(
|
| 5263 |
audio_service=audio_service,
|
| 5264 |
dataset_repo=repo,
|
|
|
|
| 5327 |
)
|
| 5328 |
|
| 5329 |
species_filter.change(
|
| 5330 |
+
fn=lambda project_slug, species, confidence, validator_filter_value, status_filter_value, updated_after_value, only_conflicts, session: refresh(
|
| 5331 |
project_slug,
|
| 5332 |
1,
|
| 5333 |
species,
|
|
|
|
| 5336 |
status_filter_value,
|
| 5337 |
updated_after_value,
|
| 5338 |
only_conflicts,
|
| 5339 |
+
session,
|
| 5340 |
),
|
| 5341 |
inputs=[
|
| 5342 |
selected_project_state,
|
|
|
|
| 5346 |
validation_status_filter,
|
| 5347 |
updated_after_filter,
|
| 5348 |
show_conflicts_only,
|
| 5349 |
+
session_state,
|
| 5350 |
],
|
| 5351 |
outputs=[table, status, page_state],
|
| 5352 |
).then(
|
|
|
|
| 5806 |
slug = (project_slug or "").strip()
|
| 5807 |
if session is None:
|
| 5808 |
return None, "Login before exporting project data."
|
| 5809 |
+
if not _can_access_project(session, slug):
|
| 5810 |
return None, "Choose an authorized project before exporting."
|
| 5811 |
|
| 5812 |
project = _project_map().get(slug)
|
|
|
|
| 5858 |
1,
|
| 5859 |
"Select a project to view the dashboard",
|
| 5860 |
)
|
| 5861 |
+
if not _can_access_project(session, slug):
|
| 5862 |
+
return (
|
| 5863 |
+
"",
|
| 5864 |
+
coverage_bars_html([]),
|
| 5865 |
+
paged_activity_html("Validator activity", ["Validator", "Validations"], []),
|
| 5866 |
+
paged_activity_html("Recent activity", ["Timestamp", "Validator", "Status", "Detection"], []),
|
| 5867 |
+
1,
|
| 5868 |
+
1,
|
| 5869 |
+
"Access denied. Select a project assigned to your signed-in account.",
|
| 5870 |
+
)
|
| 5871 |
|
| 5872 |
warning = _ensure_project_queue_loaded(slug, session)
|
| 5873 |
signature = loaded_project_signatures.get(slug, "")
|
|
|
|
| 6087 |
def _render_settings_health():
|
| 6088 |
backend = (runtime_config.state_backend or "filesystem").strip().lower()
|
| 6089 |
supabase_ready = bool(runtime_config.supabase_url and runtime_config.supabase_service_role_key)
|
| 6090 |
+
if runtime_config.hf_admin_storage_mode_enabled:
|
| 6091 |
+
state_label = "HF admin-owned private storage"
|
| 6092 |
state_tone = "ok"
|
| 6093 |
else:
|
| 6094 |
state_label = "Supabase" if backend in {"supabase", "postgres", "postgresql"} and supabase_ready else "Filesystem"
|
| 6095 |
state_tone = "ok" if state_label == "Supabase" else "warn"
|
| 6096 |
normalized_bootstrap_dir = str(runtime_config.bootstrap_base_dir).replace("\\", "/").rstrip("/")
|
| 6097 |
filesystem_durable = normalized_bootstrap_dir == "/data" or normalized_bootstrap_dir.startswith("/data/")
|
| 6098 |
+
if state_label == "HF admin-owned private storage":
|
| 6099 |
durability_label = "private HF state/Bucket"
|
| 6100 |
durability_tone = "ok"
|
| 6101 |
elif state_label == "Supabase":
|
|
|
|
| 6112 |
invite_email_label = "enabled" if runtime_config.invite_email_enabled and runtime_config.emailjs_enabled else "disabled"
|
| 6113 |
hf_project_state_label = "enabled" if hf_state_writes_enabled else "disabled"
|
| 6114 |
hf_bucket_validation_label = (
|
| 6115 |
+
"admin-owned storage active"
|
| 6116 |
+
if runtime_config.hf_admin_storage_mode_enabled
|
| 6117 |
else ("legacy experimental" if runtime_config.hf_bucket_validations_enabled else "disabled")
|
| 6118 |
)
|
| 6119 |
hf_project_state_repos_label = str(len(runtime_config.hf_project_state_repos))
|
|
|
|
| 6122 |
[
|
| 6123 |
("State backend", state_label, state_tone),
|
| 6124 |
("HF project-state writes", hf_project_state_label, "ok" if hf_state_writes_enabled else "info"),
|
| 6125 |
+
("HF Bucket validations", hf_bucket_validation_label, "ok" if runtime_config.hf_admin_storage_mode_enabled else ("warn" if runtime_config.hf_bucket_validations_enabled else "info")),
|
| 6126 |
(
|
| 6127 |
"HF project-state repos",
|
| 6128 |
hf_project_state_repos_label,
|
| 6129 |
+
"ok" if runtime_config.hf_project_state_repos else ("warn" if runtime_config.hf_admin_storage_mode_enabled else "info"),
|
| 6130 |
),
|
| 6131 |
("Auth mode", auth_mode_label, "ok" if not allow_username_login else "warn"),
|
| 6132 |
("Destructive write guard", "enabled", "ok"),
|
|
|
|
| 6143 |
)
|
| 6144 |
status_text = (
|
| 6145 |
(
|
| 6146 |
+
"HF administrator-owned storage is active. Private dataset and Bucket access use only the "
|
| 6147 |
+
"protected Space storage credential; validator identities are authorized inside this app; "
|
| 6148 |
+ (
|
| 6149 |
"no project-state repository is configured, so projects will not be restored after a rebuild."
|
| 6150 |
if not runtime_config.hf_project_state_repos
|
| 6151 |
else "configured state repositories will be used for restart recovery."
|
| 6152 |
)
|
| 6153 |
+
if state_label == "HF admin-owned private storage"
|
| 6154 |
else "Supabase persistence is active. Project and ACL removals are guarded against accidental broad overwrites."
|
| 6155 |
)
|
| 6156 |
+
if state_label in {"HF admin-owned private storage", "Supabase"}
|
| 6157 |
else (
|
| 6158 |
"Filesystem persistence is active with local JSON backups and destructive-write guards. "
|
| 6159 |
"On free Spaces without persistent storage, local files are still not durable across rebuilds."
|
src/ui/login_page.py
CHANGED
|
@@ -84,7 +84,7 @@ def create_login_page(
|
|
| 84 |
allow_username_login: bool = True,
|
| 85 |
enable_oauth_login: bool = False,
|
| 86 |
auth_mode_label: str = "",
|
| 87 |
-
|
| 88 |
) -> Tuple[gr.Textbox, gr.Textbox, gr.Button, gr.Markdown]:
|
| 89 |
"""Create a Gradio login page with username input and session tracking.
|
| 90 |
|
|
@@ -101,20 +101,19 @@ def create_login_page(
|
|
| 101 |
gr.Markdown("# BirdNET Validation Platform")
|
| 102 |
gr.Markdown(
|
| 103 |
(
|
| 104 |
-
"
|
| 105 |
-
if
|
| 106 |
else "Sign in with your Hugging Face account to access project validation workflows."
|
| 107 |
)
|
| 108 |
)
|
| 109 |
if auth_mode_label:
|
| 110 |
gr.Markdown(auth_mode_label)
|
| 111 |
|
| 112 |
-
if
|
| 113 |
gr.Markdown(
|
| 114 |
-
"###
|
| 115 |
-
"
|
| 116 |
-
"
|
| 117 |
-
"and write to its private validation Bucket."
|
| 118 |
)
|
| 119 |
else:
|
| 120 |
gr.Markdown(
|
|
@@ -130,8 +129,8 @@ def create_login_page(
|
|
| 130 |
)
|
| 131 |
hf_token_input = gr.Textbox(
|
| 132 |
label=(
|
| 133 |
-
"Personal Hugging Face Token"
|
| 134 |
-
if
|
| 135 |
else "Hugging Face Token (legacy/development only)"
|
| 136 |
),
|
| 137 |
placeholder="hf_xxx...",
|
|
@@ -142,7 +141,7 @@ def create_login_page(
|
|
| 142 |
error_message = gr.Markdown()
|
| 143 |
|
| 144 |
login_button = gr.Button(
|
| 145 |
-
"Enter with
|
| 146 |
variant="primary",
|
| 147 |
scale=1,
|
| 148 |
)
|
|
@@ -151,11 +150,10 @@ def create_login_page(
|
|
| 151 |
if enable_oauth_login:
|
| 152 |
gr.Markdown(
|
| 153 |
(
|
| 154 |
-
"###
|
| 155 |
-
"The hosted Space can
|
| 156 |
-
"
|
| 157 |
-
|
| 158 |
-
if trusted_team_mode
|
| 159 |
else (
|
| 160 |
"**OAuth access required for private collaborative state.** "
|
| 161 |
"Complete both steps below; signing into Hugging Face alone does not open an app session."
|
|
@@ -165,7 +163,7 @@ def create_login_page(
|
|
| 165 |
gr.LoginButton("1. Sign in with Hugging Face", logout_value="Sign out ({})")
|
| 166 |
oauth_continue_button = gr.Button(
|
| 167 |
"2. Enter workspace with OAuth authorization",
|
| 168 |
-
variant="
|
| 169 |
)
|
| 170 |
|
| 171 |
session_output = gr.Textbox(
|
|
|
|
| 84 |
allow_username_login: bool = True,
|
| 85 |
enable_oauth_login: bool = False,
|
| 86 |
auth_mode_label: str = "",
|
| 87 |
+
admin_storage_mode: bool = False,
|
| 88 |
) -> Tuple[gr.Textbox, gr.Textbox, gr.Button, gr.Markdown]:
|
| 89 |
"""Create a Gradio login page with username input and session tracking.
|
| 90 |
|
|
|
|
| 101 |
gr.Markdown("# BirdNET Validation Platform")
|
| 102 |
gr.Markdown(
|
| 103 |
(
|
| 104 |
+
"Sign in with your Hugging Face identity. Private project storage is accessed only by the app backend."
|
| 105 |
+
if admin_storage_mode
|
| 106 |
else "Sign in with your Hugging Face account to access project validation workflows."
|
| 107 |
)
|
| 108 |
)
|
| 109 |
if auth_mode_label:
|
| 110 |
gr.Markdown(auth_mode_label)
|
| 111 |
|
| 112 |
+
if admin_storage_mode:
|
| 113 |
gr.Markdown(
|
| 114 |
+
"### Identity verification\n"
|
| 115 |
+
"Use a token from your own account only to prove your identity. "
|
| 116 |
+
"You do not need direct Hugging Face access to the project's private dataset or validation storage."
|
|
|
|
| 117 |
)
|
| 118 |
else:
|
| 119 |
gr.Markdown(
|
|
|
|
| 129 |
)
|
| 130 |
hf_token_input = gr.Textbox(
|
| 131 |
label=(
|
| 132 |
+
"Personal Hugging Face Token (identity verification)"
|
| 133 |
+
if admin_storage_mode
|
| 134 |
else "Hugging Face Token (legacy/development only)"
|
| 135 |
),
|
| 136 |
placeholder="hf_xxx...",
|
|
|
|
| 141 |
error_message = gr.Markdown()
|
| 142 |
|
| 143 |
login_button = gr.Button(
|
| 144 |
+
"Enter with Hugging Face token" if admin_storage_mode else "Login",
|
| 145 |
variant="primary",
|
| 146 |
scale=1,
|
| 147 |
)
|
|
|
|
| 150 |
if enable_oauth_login:
|
| 151 |
gr.Markdown(
|
| 152 |
(
|
| 153 |
+
"### Sign in with OAuth\n"
|
| 154 |
+
"The hosted Space can verify your Hugging Face identity without requiring a personal token. "
|
| 155 |
+
"Project access remains controlled by assignments and invitations inside this app."
|
| 156 |
+
if admin_storage_mode
|
|
|
|
| 157 |
else (
|
| 158 |
"**OAuth access required for private collaborative state.** "
|
| 159 |
"Complete both steps below; signing into Hugging Face alone does not open an app session."
|
|
|
|
| 163 |
gr.LoginButton("1. Sign in with Hugging Face", logout_value="Sign out ({})")
|
| 164 |
oauth_continue_button = gr.Button(
|
| 165 |
"2. Enter workspace with OAuth authorization",
|
| 166 |
+
variant="primary",
|
| 167 |
)
|
| 168 |
|
| 169 |
session_output = gr.Textbox(
|
src/ui/theme.py
CHANGED
|
@@ -1449,8 +1449,8 @@ select {
|
|
| 1449 |
|
| 1450 |
|
| 1451 |
def app_header_html(state_backend_message: str = "") -> str:
|
| 1452 |
-
if "HF
|
| 1453 |
-
backend_text = "Private HF
|
| 1454 |
backend_class = "bn-pill-ok"
|
| 1455 |
elif "Supabase state backend enabled" in state_backend_message:
|
| 1456 |
backend_text = "Supabase ready"
|
|
|
|
| 1449 |
|
| 1450 |
|
| 1451 |
def app_header_html(state_backend_message: str = "") -> str:
|
| 1452 |
+
if "HF admin-owned private storage enabled" in state_backend_message:
|
| 1453 |
+
backend_text = "Private HF storage ready"
|
| 1454 |
backend_class = "bn-pill-ok"
|
| 1455 |
elif "Supabase state backend enabled" in state_backend_message:
|
| 1456 |
backend_text = "Supabase ready"
|
tests/unit/test_app_factory_audio_helpers.py
CHANGED
|
@@ -36,7 +36,7 @@ from src.ui.app_factory import (
|
|
| 36 |
_persist_bootstrap_state,
|
| 37 |
_resolve_username_login_policy,
|
| 38 |
_resolve_project_fetch_token,
|
| 39 |
-
|
| 40 |
_write_validation_export,
|
| 41 |
)
|
| 42 |
from src.auth.auth_service import AuthService
|
|
@@ -68,19 +68,16 @@ class FakeAudioService:
|
|
| 68 |
|
| 69 |
|
| 70 |
class FakeDatasetInfoApi:
|
| 71 |
-
def __init__(self, private: bool,
|
| 72 |
self.private = private
|
| 73 |
-
self.
|
| 74 |
|
| 75 |
-
def
|
| 76 |
-
assert organization == "project-org"
|
| 77 |
assert token == "hf_admin"
|
| 78 |
-
|
| 79 |
-
raise RuntimeError("404 organization not found")
|
| 80 |
-
return SimpleNamespace(name=organization)
|
| 81 |
|
| 82 |
def repo_info(self, *, repo_id: str, repo_type: str, token: str): # noqa: ANN001
|
| 83 |
-
assert repo_id == "
|
| 84 |
assert repo_type == "dataset"
|
| 85 |
assert token == "hf_admin"
|
| 86 |
return SimpleNamespace(private=self.private)
|
|
@@ -93,7 +90,7 @@ class FakeBucketInitializer:
|
|
| 93 |
def initialize(self, *, project_slug: str, dataset_repo_id: str, token: str | None):
|
| 94 |
self.calls.append((project_slug, dataset_repo_id, token))
|
| 95 |
return HfBucketValidationInitResult(
|
| 96 |
-
bucket_id="
|
| 97 |
initialized=True,
|
| 98 |
reused_existing=False,
|
| 99 |
)
|
|
@@ -232,39 +229,39 @@ class FakeQueueService:
|
|
| 232 |
return FakeQueueService._Page()
|
| 233 |
|
| 234 |
|
| 235 |
-
def
|
| 236 |
project = Project(
|
| 237 |
project_slug="private-audio",
|
| 238 |
name="Private Audio",
|
| 239 |
-
dataset_repo_id="
|
| 240 |
dataset_token="hf_shared_should_be_removed",
|
| 241 |
)
|
| 242 |
bucket_initializer = FakeBucketInitializer()
|
| 243 |
|
| 244 |
-
result =
|
| 245 |
project=project,
|
| 246 |
token="hf_admin",
|
| 247 |
api=FakeDatasetInfoApi(private=True), # type: ignore[arg-type]
|
| 248 |
bucket_initializer=bucket_initializer, # type: ignore[arg-type]
|
| 249 |
)
|
| 250 |
|
| 251 |
-
assert result.bucket_id == "
|
| 252 |
assert project.validation_bucket_id == result.bucket_id
|
| 253 |
assert project.validation_backend == "hf_bucket"
|
| 254 |
assert project.dataset_token is None
|
| 255 |
-
assert bucket_initializer.calls == [("private-audio", "
|
| 256 |
|
| 257 |
|
| 258 |
-
def
|
| 259 |
project = Project(
|
| 260 |
project_slug="private-audio",
|
| 261 |
name="Private Audio",
|
| 262 |
-
dataset_repo_id="
|
| 263 |
)
|
| 264 |
bucket_initializer = FakeBucketInitializer()
|
| 265 |
|
| 266 |
with pytest.raises(HfBucketValidationError, match="only accepts a private dataset"):
|
| 267 |
-
|
| 268 |
project=project,
|
| 269 |
token="hf_admin",
|
| 270 |
api=FakeDatasetInfoApi(private=False), # type: ignore[arg-type]
|
|
@@ -274,36 +271,36 @@ def test_initialize_hf_trusted_team_storage_rejects_public_source_dataset() -> N
|
|
| 274 |
assert bucket_initializer.calls == []
|
| 275 |
|
| 276 |
|
| 277 |
-
def
|
| 278 |
project = Project(
|
| 279 |
project_slug="private-audio",
|
| 280 |
name="Private Audio",
|
| 281 |
-
dataset_repo_id="
|
| 282 |
)
|
| 283 |
bucket_initializer = FakeBucketInitializer()
|
| 284 |
|
| 285 |
-
with pytest.raises(HfBucketValidationError, match="
|
| 286 |
-
|
| 287 |
project=project,
|
| 288 |
token="hf_admin",
|
| 289 |
-
api=FakeDatasetInfoApi(private=True,
|
| 290 |
bucket_initializer=bucket_initializer, # type: ignore[arg-type]
|
| 291 |
)
|
| 292 |
|
| 293 |
assert bucket_initializer.calls == []
|
| 294 |
|
| 295 |
|
| 296 |
-
def
|
| 297 |
project = Project(
|
| 298 |
project_slug="private-audio",
|
| 299 |
name="Private Audio",
|
| 300 |
-
dataset_repo_id="
|
| 301 |
visibility="private",
|
| 302 |
)
|
| 303 |
bucket_initializer = FakeBucketInitializer()
|
| 304 |
|
| 305 |
with pytest.raises(HfBucketValidationError, match="visibility 'collaborative'"):
|
| 306 |
-
|
| 307 |
project=project,
|
| 308 |
token="hf_admin",
|
| 309 |
api=FakeDatasetInfoApi(private=True), # type: ignore[arg-type]
|
|
|
|
| 36 |
_persist_bootstrap_state,
|
| 37 |
_resolve_username_login_policy,
|
| 38 |
_resolve_project_fetch_token,
|
| 39 |
+
_initialize_hf_admin_storage,
|
| 40 |
_write_validation_export,
|
| 41 |
)
|
| 42 |
from src.auth.auth_service import AuthService
|
|
|
|
| 68 |
|
| 69 |
|
| 70 |
class FakeDatasetInfoApi:
|
| 71 |
+
def __init__(self, private: bool, owner_username: str = "jrrribeiro") -> None:
|
| 72 |
self.private = private
|
| 73 |
+
self.owner_username = owner_username
|
| 74 |
|
| 75 |
+
def whoami(self, token: str): # noqa: ANN001
|
|
|
|
| 76 |
assert token == "hf_admin"
|
| 77 |
+
return {"name": self.owner_username}
|
|
|
|
|
|
|
| 78 |
|
| 79 |
def repo_info(self, *, repo_id: str, repo_type: str, token: str): # noqa: ANN001
|
| 80 |
+
assert repo_id == "jrrribeiro/private-audio"
|
| 81 |
assert repo_type == "dataset"
|
| 82 |
assert token == "hf_admin"
|
| 83 |
return SimpleNamespace(private=self.private)
|
|
|
|
| 90 |
def initialize(self, *, project_slug: str, dataset_repo_id: str, token: str | None):
|
| 91 |
self.calls.append((project_slug, dataset_repo_id, token))
|
| 92 |
return HfBucketValidationInitResult(
|
| 93 |
+
bucket_id="jrrribeiro/private-audio_validation_state",
|
| 94 |
initialized=True,
|
| 95 |
reused_existing=False,
|
| 96 |
)
|
|
|
|
| 229 |
return FakeQueueService._Page()
|
| 230 |
|
| 231 |
|
| 232 |
+
def test_initialize_hf_admin_storage_sets_bucket_without_persisting_shared_token() -> None:
|
| 233 |
project = Project(
|
| 234 |
project_slug="private-audio",
|
| 235 |
name="Private Audio",
|
| 236 |
+
dataset_repo_id="jrrribeiro/private-audio",
|
| 237 |
dataset_token="hf_shared_should_be_removed",
|
| 238 |
)
|
| 239 |
bucket_initializer = FakeBucketInitializer()
|
| 240 |
|
| 241 |
+
result = _initialize_hf_admin_storage(
|
| 242 |
project=project,
|
| 243 |
token="hf_admin",
|
| 244 |
api=FakeDatasetInfoApi(private=True), # type: ignore[arg-type]
|
| 245 |
bucket_initializer=bucket_initializer, # type: ignore[arg-type]
|
| 246 |
)
|
| 247 |
|
| 248 |
+
assert result.bucket_id == "jrrribeiro/private-audio_validation_state"
|
| 249 |
assert project.validation_bucket_id == result.bucket_id
|
| 250 |
assert project.validation_backend == "hf_bucket"
|
| 251 |
assert project.dataset_token is None
|
| 252 |
+
assert bucket_initializer.calls == [("private-audio", "jrrribeiro/private-audio", "hf_admin")]
|
| 253 |
|
| 254 |
|
| 255 |
+
def test_initialize_hf_admin_storage_rejects_public_source_dataset() -> None:
|
| 256 |
project = Project(
|
| 257 |
project_slug="private-audio",
|
| 258 |
name="Private Audio",
|
| 259 |
+
dataset_repo_id="jrrribeiro/private-audio",
|
| 260 |
)
|
| 261 |
bucket_initializer = FakeBucketInitializer()
|
| 262 |
|
| 263 |
with pytest.raises(HfBucketValidationError, match="only accepts a private dataset"):
|
| 264 |
+
_initialize_hf_admin_storage(
|
| 265 |
project=project,
|
| 266 |
token="hf_admin",
|
| 267 |
api=FakeDatasetInfoApi(private=False), # type: ignore[arg-type]
|
|
|
|
| 271 |
assert bucket_initializer.calls == []
|
| 272 |
|
| 273 |
|
| 274 |
+
def test_initialize_hf_admin_storage_requires_storage_owner_personal_namespace() -> None:
|
| 275 |
project = Project(
|
| 276 |
project_slug="private-audio",
|
| 277 |
name="Private Audio",
|
| 278 |
+
dataset_repo_id="jrrribeiro/private-audio",
|
| 279 |
)
|
| 280 |
bucket_initializer = FakeBucketInitializer()
|
| 281 |
|
| 282 |
+
with pytest.raises(HfBucketValidationError, match="personal namespace"):
|
| 283 |
+
_initialize_hf_admin_storage(
|
| 284 |
project=project,
|
| 285 |
token="hf_admin",
|
| 286 |
+
api=FakeDatasetInfoApi(private=True, owner_username="another-admin"), # type: ignore[arg-type]
|
| 287 |
bucket_initializer=bucket_initializer, # type: ignore[arg-type]
|
| 288 |
)
|
| 289 |
|
| 290 |
assert bucket_initializer.calls == []
|
| 291 |
|
| 292 |
|
| 293 |
+
def test_initialize_hf_admin_storage_requires_collaborative_app_visibility() -> None:
|
| 294 |
project = Project(
|
| 295 |
project_slug="private-audio",
|
| 296 |
name="Private Audio",
|
| 297 |
+
dataset_repo_id="jrrribeiro/private-audio",
|
| 298 |
visibility="private",
|
| 299 |
)
|
| 300 |
bucket_initializer = FakeBucketInitializer()
|
| 301 |
|
| 302 |
with pytest.raises(HfBucketValidationError, match="visibility 'collaborative'"):
|
| 303 |
+
_initialize_hf_admin_storage(
|
| 304 |
project=project,
|
| 305 |
token="hf_admin",
|
| 306 |
api=FakeDatasetInfoApi(private=True), # type: ignore[arg-type]
|
tests/unit/test_hf_bucket_validation_repository.py
CHANGED
|
@@ -129,14 +129,14 @@ def test_save_validation_writes_event_and_snapshot_without_repo_commit() -> None
|
|
| 129 |
assert repository.load_current_snapshot("project-a")["audio-a-0000000001"]["version"] == 1
|
| 130 |
|
| 131 |
|
| 132 |
-
def
|
| 133 |
repository = HfBucketValidationRepository(
|
| 134 |
bucket_id="owner/audio_validation_state",
|
| 135 |
token="hf_user",
|
| 136 |
api=InaccessibleBucketApi(),
|
| 137 |
)
|
| 138 |
|
| 139 |
-
with pytest.raises(HfBucketValidationError, match="
|
| 140 |
repository.load_current_snapshot("project-a")
|
| 141 |
|
| 142 |
|
|
|
|
| 129 |
assert repository.load_current_snapshot("project-a")["audio-a-0000000001"]["version"] == 1
|
| 130 |
|
| 131 |
|
| 132 |
+
def test_bucket_access_error_explains_that_app_invite_is_not_direct_hub_permission() -> None:
|
| 133 |
repository = HfBucketValidationRepository(
|
| 134 |
bucket_id="owner/audio_validation_state",
|
| 135 |
token="hf_user",
|
| 136 |
api=InaccessibleBucketApi(),
|
| 137 |
)
|
| 138 |
|
| 139 |
+
with pytest.raises(HfBucketValidationError, match="does not grant direct Hub Bucket"):
|
| 140 |
repository.load_current_snapshot("project-a")
|
| 141 |
|
| 142 |
|
tests/unit/test_project_aware_validation_repository.py
CHANGED
|
@@ -163,6 +163,51 @@ def test_project_aware_repository_bucket_write_does_not_fall_back_to_admin_token
|
|
| 163 |
router.save_validation("project-a", _validation(), expected_version=0)
|
| 164 |
|
| 165 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 166 |
def test_project_aware_repository_never_falls_back_when_declared_bucket_is_disabled() -> None:
|
| 167 |
fallback = FakeValidationStateRepository("fallback")
|
| 168 |
project = Project(
|
|
|
|
| 163 |
router.save_validation("project-a", _validation(), expected_version=0)
|
| 164 |
|
| 165 |
|
| 166 |
+
def test_project_aware_repository_admin_storage_routes_validator_write_through_backend_token() -> None:
|
| 167 |
+
bucket_repo = FakeValidationStateRepository("bucket")
|
| 168 |
+
received: list[tuple[str, str]] = []
|
| 169 |
+
project = Project(
|
| 170 |
+
project_slug="project-a",
|
| 171 |
+
name="Project A",
|
| 172 |
+
dataset_repo_id="owner/project-a",
|
| 173 |
+
validation_backend=HF_BUCKET_VALIDATION_BACKEND,
|
| 174 |
+
validation_bucket_id="owner/project-a_validation_state",
|
| 175 |
+
)
|
| 176 |
+
router = ProjectAwareValidationRepository(
|
| 177 |
+
fallback_repository=FakeValidationStateRepository("fallback"),
|
| 178 |
+
project_lookup=lambda _: project,
|
| 179 |
+
token_provider=lambda _: "hf_storage_secret",
|
| 180 |
+
actor_token_provider=lambda _project, _username: "hf_validator_identity_only",
|
| 181 |
+
enable_hf_bucket_validations=True,
|
| 182 |
+
use_backend_token_for_bucket=True,
|
| 183 |
+
bucket_repository_factory=lambda bucket, token: (received.append((bucket, token)) or bucket_repo),
|
| 184 |
+
)
|
| 185 |
+
|
| 186 |
+
assert router.save_validation("project-a", _validation(), expected_version=0) == 1
|
| 187 |
+
assert received == [("owner/project-a_validation_state", "hf_storage_secret")]
|
| 188 |
+
|
| 189 |
+
|
| 190 |
+
def test_project_aware_repository_admin_storage_fails_without_backend_credential() -> None:
|
| 191 |
+
project = Project(
|
| 192 |
+
project_slug="project-a",
|
| 193 |
+
name="Project A",
|
| 194 |
+
dataset_repo_id="owner/project-a",
|
| 195 |
+
validation_backend=HF_BUCKET_VALIDATION_BACKEND,
|
| 196 |
+
validation_bucket_id="owner/project-a_validation_state",
|
| 197 |
+
)
|
| 198 |
+
router = ProjectAwareValidationRepository(
|
| 199 |
+
fallback_repository=FakeValidationStateRepository("fallback"),
|
| 200 |
+
project_lookup=lambda _: project,
|
| 201 |
+
token_provider=lambda _: None,
|
| 202 |
+
actor_token_provider=lambda _project, _username: "hf_validator_identity_only",
|
| 203 |
+
enable_hf_bucket_validations=True,
|
| 204 |
+
use_backend_token_for_bucket=True,
|
| 205 |
+
)
|
| 206 |
+
|
| 207 |
+
with pytest.raises(HfBucketValidationError, match="backend storage credential"):
|
| 208 |
+
router.save_validation("project-a", _validation(), expected_version=0)
|
| 209 |
+
|
| 210 |
+
|
| 211 |
def test_project_aware_repository_never_falls_back_when_declared_bucket_is_disabled() -> None:
|
| 212 |
fallback = FakeValidationStateRepository("fallback")
|
| 213 |
project = Project(
|
tests/unit/test_runtime_config.py
CHANGED
|
@@ -33,17 +33,17 @@ def test_runtime_config_reads_bucket_validation_flag(monkeypatch) -> None: # no
|
|
| 33 |
assert config.hf_bucket_validations_enabled is True
|
| 34 |
|
| 35 |
|
| 36 |
-
def
|
| 37 |
-
monkeypatch.setenv("
|
| 38 |
|
| 39 |
config = RuntimeConfig.from_env()
|
| 40 |
|
| 41 |
-
assert config.
|
| 42 |
|
| 43 |
|
| 44 |
-
def
|
| 45 |
-
monkeypatch.delenv("
|
| 46 |
|
| 47 |
config = RuntimeConfig.from_env()
|
| 48 |
|
| 49 |
-
assert config.
|
|
|
|
| 33 |
assert config.hf_bucket_validations_enabled is True
|
| 34 |
|
| 35 |
|
| 36 |
+
def test_runtime_config_reads_hf_admin_storage_mode_flag(monkeypatch) -> None: # noqa: ANN001
|
| 37 |
+
monkeypatch.setenv("BIRDNET_HF_ADMIN_STORAGE_MODE_ENABLED", "true")
|
| 38 |
|
| 39 |
config = RuntimeConfig.from_env()
|
| 40 |
|
| 41 |
+
assert config.hf_admin_storage_mode_enabled is True
|
| 42 |
|
| 43 |
|
| 44 |
+
def test_runtime_config_disables_hf_admin_storage_mode_by_default(monkeypatch) -> None: # noqa: ANN001
|
| 45 |
+
monkeypatch.delenv("BIRDNET_HF_ADMIN_STORAGE_MODE_ENABLED", raising=False)
|
| 46 |
|
| 47 |
config = RuntimeConfig.from_env()
|
| 48 |
|
| 49 |
+
assert config.hf_admin_storage_mode_enabled is False
|