jrrribeiro commited on
Commit
7df33f3
·
verified ·
1 Parent(s): 5d646a8

Sync app from GitHub Actions (feature/project-state-security-privacy-review)

Browse files
README.md CHANGED
@@ -200,22 +200,23 @@ Optional runtime settings:
200
  - `BIRDNET_HF_PROJECT_STATE_WRITES_ENABLED=true` (experimental `_state` diagnostics and recovery only while a new collaborative backend is selected)
201
  - `BIRDNET_HF_BUCKET_VALIDATIONS_ENABLED=true` (legacy experimental access for existing Bucket projects)
202
  - `BIRDNET_HF_ADMIN_STORAGE_MODE_ENABLED=true` (private HF-only delivery mode for a small trusted team)
203
- - `BIRDNET_HF_PROJECT_STATE_REPOS=jrrribeiro/audio-dataset_state` (comma-separated private state repos restored after Space rebuild)
204
 
205
  The HF administrator-owned storage mode is intended for confidential projects run by a small trusted research team. The source audio dataset must already be private in the administrator's personal Hugging Face namespace. When a project is created, the app creates a private `_state` repository for project/ACL/invite recovery and a private Storage Bucket for validation events and snapshots in that same namespace. Select `collaborative` inside the app: this enables app-managed validators and does not make any Hugging Face resource public.
206
 
207
  The Space stores the administrator storage credential as a protected secret. Validators log in with their own Hugging Face identity and are invited or assigned inside the app; they do not need repository or Bucket permissions, organization membership, or the administrator token. Resource operations are performed server-side only after the app confirms the user's project role.
208
 
 
 
209
  Recommended administrator-owned Space settings:
210
 
211
  ```text
212
  BIRDNET_HF_ADMIN_STORAGE_MODE_ENABLED=true
213
  BIRDNET_AUTH_MODE=hf_token
214
- BIRDNET_HF_PROJECT_STATE_REPOS=jrrribeiro/audio-dataset_state
215
  BIRDNET_HF_STORAGE_TOKEN=<administrator token stored only as a Space secret>
216
  ```
217
 
218
- `HF_TOKEN` remains a compatibility fallback for the storage credential, but `BIRDNET_HF_STORAGE_TOKEN` is preferred because its purpose is explicit. Never enter this administrator token in an ordinary user login form. After creating a project, add its generated `_state` repository to `BIRDNET_HF_PROJECT_STATE_REPOS` before relying on rebuild recovery. An administrator may run **Projects > Private storage backend health** to verify the protected backend credential with a temporary marker that is immediately removed.
219
 
220
  Projects previously created with a private `_state` repository in this mode can be recovered from the **Admin** tab using **Connect Existing State**. Sign in as an administrator recorded in its `acl.json`, then provide the `_state` repo id (for example, `jrrribeiro/audio_dataset_state`). Recovery accepts only a manifest with private Bucket validation storage and a private personal-namespace source dataset accessible to the configured storage account. The app loads the saved manifest, ACL, and pending invites.
221
 
 
200
  - `BIRDNET_HF_PROJECT_STATE_WRITES_ENABLED=true` (experimental `_state` diagnostics and recovery only while a new collaborative backend is selected)
201
  - `BIRDNET_HF_BUCKET_VALIDATIONS_ENABLED=true` (legacy experimental access for existing Bucket projects)
202
  - `BIRDNET_HF_ADMIN_STORAGE_MODE_ENABLED=true` (private HF-only delivery mode for a small trusted team)
203
+ - `BIRDNET_HF_PROJECT_STATE_REPOS=jrrribeiro/audio-dataset_state` (optional manual recovery list; administrator-owned mode discovers companion state repos automatically)
204
 
205
  The HF administrator-owned storage mode is intended for confidential projects run by a small trusted research team. The source audio dataset must already be private in the administrator's personal Hugging Face namespace. When a project is created, the app creates a private `_state` repository for project/ACL/invite recovery and a private Storage Bucket for validation events and snapshots in that same namespace. Select `collaborative` inside the app: this enables app-managed validators and does not make any Hugging Face resource public.
206
 
207
  The Space stores the administrator storage credential as a protected secret. Validators log in with their own Hugging Face identity and are invited or assigned inside the app; they do not need repository or Bucket permissions, organization membership, or the administrator token. Resource operations are performed server-side only after the app confirms the user's project role.
208
 
209
+ When this mode is enabled it takes precedence over any existing Supabase settings: project ACL/invites are restored from private `*_state` repositories and validations are read and written in the project's private Bucket.
210
+
211
  Recommended administrator-owned Space settings:
212
 
213
  ```text
214
  BIRDNET_HF_ADMIN_STORAGE_MODE_ENABLED=true
215
  BIRDNET_AUTH_MODE=hf_token
 
216
  BIRDNET_HF_STORAGE_TOKEN=<administrator token stored only as a Space secret>
217
  ```
218
 
219
+ `HF_TOKEN` remains a compatibility fallback for the storage credential, but `BIRDNET_HF_STORAGE_TOKEN` is preferred because its purpose is explicit. Never enter this administrator token in an ordinary user login form. On restart the app uses the protected credential to find the administrator's companion `*_state` repositories automatically; `BIRDNET_HF_PROJECT_STATE_REPOS` remains available as an explicit fallback or migration list. An administrator may run **Projects > Private storage backend health** to verify the protected backend credential with a temporary marker that is immediately removed.
220
 
221
  Projects previously created with a private `_state` repository in this mode can be recovered from the **Admin** tab using **Connect Existing State**. Sign in as an administrator recorded in its `acl.json`, then provide the `_state` repo id (for example, `jrrribeiro/audio_dataset_state`). Recovery accepts only a manifest with private Bucket validation storage and a private personal-namespace source dataset accessible to the configured storage account. The app loads the saved manifest, ACL, and pending invites.
222
 
docs/TEMP_HF_ADMIN_OWNED_PRIVATE_STORAGE_DELIVERY_PLAN.md CHANGED
@@ -50,7 +50,9 @@ For public distribution to independent administrators, this model is not suffici
50
  5. Project manifests persist the Bucket reference; ACL and invite changes sync to `_state`.
51
  6. Validation Bucket reads and writes route through the backend storage credential while retaining the actual validator username in each event.
52
  7. Dataset metadata and audio fetches route through the backend credential only after server-side authorization of the session and project role.
53
- 8. The administrator can run a backend storage health check; validators do not test or receive direct Bucket permissions.
 
 
54
 
55
  ## Space Configuration
56
 
@@ -59,7 +61,6 @@ Configure:
59
  ```text
60
  BIRDNET_HF_ADMIN_STORAGE_MODE_ENABLED=true
61
  BIRDNET_AUTH_MODE=hf_token
62
- BIRDNET_HF_PROJECT_STATE_REPOS=jrrribeiro/audio-dataset_state
63
  ```
64
 
65
  Configure as a Secret:
@@ -74,8 +75,8 @@ Use a scoped credential limited to the administrator's required resources when H
74
 
75
  1. Administrator creates a private test dataset in their personal namespace.
76
  2. Administrator creates the app project; the app reports the generated private `_state` repository and validation Bucket.
77
- 3. Administrator adds that `_state` repository to `BIRDNET_HF_PROJECT_STATE_REPOS` and restarts the Space.
78
- 4. Project is restored with its ACL and Bucket reference intact.
79
  5. Administrator assigns or invites a second Hugging Face account only through the app.
80
  6. The second account logs in with its own identity, opens the project, loads private audio, and submits validations without Hub resource permissions.
81
  7. The second account cannot access the private audio dataset, `_state`, or Bucket directly on Hugging Face.
 
50
  5. Project manifests persist the Bucket reference; ACL and invite changes sync to `_state`.
51
  6. Validation Bucket reads and writes route through the backend storage credential while retaining the actual validator username in each event.
52
  7. Dataset metadata and audio fetches route through the backend credential only after server-side authorization of the session and project role.
53
+ 8. At startup the app automatically discovers the administrator's private companion `*_state` datasets for rebuild recovery; a manual repo list remains an optional fallback.
54
+ 9. This mode takes precedence over configured Supabase persistence so durable project state does not remain dependent on the app operator's Supabase database.
55
+ 10. The administrator can run a backend storage health check; validators do not test or receive direct Bucket permissions.
56
 
57
  ## Space Configuration
58
 
 
61
  ```text
62
  BIRDNET_HF_ADMIN_STORAGE_MODE_ENABLED=true
63
  BIRDNET_AUTH_MODE=hf_token
 
64
  ```
65
 
66
  Configure as a Secret:
 
75
 
76
  1. Administrator creates a private test dataset in their personal namespace.
77
  2. Administrator creates the app project; the app reports the generated private `_state` repository and validation Bucket.
78
+ 3. Administrator restarts the Space without manually registering the new `_state` repository.
79
+ 4. Project is discovered and restored with its ACL and Bucket reference intact.
80
  5. Administrator assigns or invites a second Hugging Face account only through the app.
81
  6. The second account logs in with its own identity, opens the project, loads private audio, and submits validations without Hub resource permissions.
82
  7. The second account cannot access the private audio dataset, `_state`, or Bucket directly on Hugging Face.
src/ui/app_factory.py CHANGED
@@ -1388,6 +1388,32 @@ def _load_hf_project_state_repos(
1388
  return loaded, errors
1389
 
1390
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1391
  def _resolve_bootstrap_file_paths(runtime_config: RuntimeConfig) -> tuple[Path, Path, Path]:
1392
  bootstrap_dir = Path(runtime_config.bootstrap_base_dir)
1393
  projects_path = Path(runtime_config.projects_file_path) if runtime_config.projects_file_path else (bootstrap_dir / "projects.json")
@@ -1462,6 +1488,7 @@ def _bootstrap_auth_and_projects(
1462
  state_store: SupabaseBootstrapStore | None = None,
1463
  hf_project_state_token: str | None = None,
1464
  hf_project_state_loader: HfProjectStateStoreLoader | None = None,
 
1465
  ) -> str:
1466
  if state_store is not None:
1467
  projects = state_store.load_projects()
@@ -1472,8 +1499,17 @@ def _bootstrap_auth_and_projects(
1472
  user_access = _load_user_access_from_file(user_access_file_path or runtime_config.user_access_file_path)
1473
  pending_invites = _load_pending_invites_from_file(invites_file_path or runtime_config.invites_file_path)
1474
 
 
 
 
 
 
 
 
 
 
1475
  hf_loaded_state, hf_load_errors = _load_hf_project_state_repos(
1476
- state_repo_ids=runtime_config.hf_project_state_repos,
1477
  token=hf_project_state_token,
1478
  loader=hf_project_state_loader,
1479
  )
@@ -1532,6 +1568,8 @@ def _bootstrap_auth_and_projects(
1532
  )
1533
 
1534
  warnings = [emergency_admin_message] if emergency_admin_message else []
 
 
1535
  if hf_load_errors:
1536
  warnings.append("⚠️ Some HF project-state repos could not be loaded: " + " | ".join(hf_load_errors))
1537
  if hf_loaded_state:
@@ -2981,6 +3019,8 @@ def create_app() -> gr.Blocks:
2981
  projects_file_path, user_access_file_path, invites_file_path = _resolve_bootstrap_file_paths(runtime_config)
2982
  state_store, supabase_validation_repository, state_backend_message = _build_supabase_state(runtime_config)
2983
  if runtime_config.hf_admin_storage_mode_enabled:
 
 
2984
  state_backend_message = "HF admin-owned private storage enabled."
2985
  allow_username_login, auth_mode_label, auth_mode_description = _resolve_username_login_policy(runtime_config)
2986
  bootstrap_warning = _bootstrap_auth_and_projects(
@@ -3624,8 +3664,7 @@ def create_app() -> gr.Blocks:
3624
  else ""
3625
  )
3626
  recovery_message = (
3627
- f" Add `{state_result.state_repo_id}` to `BIRDNET_HF_PROJECT_STATE_REPOS` before rebuilding "
3628
- "the Space so this project is restored after restart."
3629
  if runtime_config.hf_admin_storage_mode_enabled
3630
  else ""
3631
  )
@@ -6116,7 +6155,11 @@ def create_app() -> gr.Blocks:
6116
  if runtime_config.hf_admin_storage_mode_enabled
6117
  else ("legacy experimental" if runtime_config.hf_bucket_validations_enabled else "disabled")
6118
  )
6119
- hf_project_state_repos_label = str(len(runtime_config.hf_project_state_repos))
 
 
 
 
6120
  hf_space_label = os.getenv("SPACE_ID") or "local runtime"
6121
  health_html = settings_health_html(
6122
  [
@@ -6126,7 +6169,7 @@ def create_app() -> gr.Blocks:
6126
  (
6127
  "HF project-state repos",
6128
  hf_project_state_repos_label,
6129
- "ok" if runtime_config.hf_project_state_repos else ("warn" if runtime_config.hf_admin_storage_mode_enabled else "info"),
6130
  ),
6131
  ("Auth mode", auth_mode_label, "ok" if not allow_username_login else "warn"),
6132
  ("Destructive write guard", "enabled", "ok"),
@@ -6145,11 +6188,7 @@ def create_app() -> gr.Blocks:
6145
  (
6146
  "HF administrator-owned storage is active. Private dataset and Bucket access use only the "
6147
  "protected Space storage credential; validator identities are authorized inside this app; "
6148
- + (
6149
- "no project-state repository is configured, so projects will not be restored after a rebuild."
6150
- if not runtime_config.hf_project_state_repos
6151
- else "configured state repositories will be used for restart recovery."
6152
- )
6153
  if state_label == "HF admin-owned private storage"
6154
  else "Supabase persistence is active. Project and ACL removals are guarded against accidental broad overwrites."
6155
  )
 
1388
  return loaded, errors
1389
 
1390
 
1391
+ def _discover_hf_admin_state_repos(
1392
+ *,
1393
+ token: str | None,
1394
+ api: HfApi | None = None,
1395
+ ) -> tuple[tuple[str, ...], str]:
1396
+ """Find companion state datasets owned by the configured administrator account."""
1397
+ token_value = (token or "").strip()
1398
+ if not token_value:
1399
+ return (), "Administrator storage mode is enabled, but no HF storage secret is configured for state recovery."
1400
+ client = api or HfApi()
1401
+ try:
1402
+ identity = client.whoami(token=token_value)
1403
+ owner = str(identity.get("name") if isinstance(identity, dict) else "").strip()
1404
+ if not owner:
1405
+ return (), "Could not determine the administrator storage account for automatic state recovery."
1406
+ datasets = client.list_datasets(author=owner, token=token_value)
1407
+ repo_ids = []
1408
+ for dataset in datasets:
1409
+ repo_id = str(getattr(dataset, "id", None) or getattr(dataset, "repo_id", None) or "").strip()
1410
+ if repo_id.casefold().startswith(f"{owner}/".casefold()) and repo_id.casefold().endswith("_state"):
1411
+ repo_ids.append(repo_id)
1412
+ return tuple(sorted(set(repo_ids))), ""
1413
+ except Exception as exc:
1414
+ return (), f"Could not discover private companion state repos automatically: {exc}"
1415
+
1416
+
1417
  def _resolve_bootstrap_file_paths(runtime_config: RuntimeConfig) -> tuple[Path, Path, Path]:
1418
  bootstrap_dir = Path(runtime_config.bootstrap_base_dir)
1419
  projects_path = Path(runtime_config.projects_file_path) if runtime_config.projects_file_path else (bootstrap_dir / "projects.json")
 
1488
  state_store: SupabaseBootstrapStore | None = None,
1489
  hf_project_state_token: str | None = None,
1490
  hf_project_state_loader: HfProjectStateStoreLoader | None = None,
1491
+ hf_project_state_discovery_api: HfApi | None = None,
1492
  ) -> str:
1493
  if state_store is not None:
1494
  projects = state_store.load_projects()
 
1499
  user_access = _load_user_access_from_file(user_access_file_path or runtime_config.user_access_file_path)
1500
  pending_invites = _load_pending_invites_from_file(invites_file_path or runtime_config.invites_file_path)
1501
 
1502
+ configured_state_repos = tuple(runtime_config.hf_project_state_repos)
1503
+ discovery_message = ""
1504
+ discovered_state_repos: tuple[str, ...] = ()
1505
+ if runtime_config.hf_admin_storage_mode_enabled:
1506
+ discovered_state_repos, discovery_message = _discover_hf_admin_state_repos(
1507
+ token=hf_project_state_token,
1508
+ api=hf_project_state_discovery_api,
1509
+ )
1510
+ state_repo_ids = tuple(dict.fromkeys([*configured_state_repos, *discovered_state_repos]))
1511
  hf_loaded_state, hf_load_errors = _load_hf_project_state_repos(
1512
+ state_repo_ids=state_repo_ids,
1513
  token=hf_project_state_token,
1514
  loader=hf_project_state_loader,
1515
  )
 
1568
  )
1569
 
1570
  warnings = [emergency_admin_message] if emergency_admin_message else []
1571
+ if discovery_message:
1572
+ warnings.append(f"⚠️ {discovery_message}")
1573
  if hf_load_errors:
1574
  warnings.append("⚠️ Some HF project-state repos could not be loaded: " + " | ".join(hf_load_errors))
1575
  if hf_loaded_state:
 
3019
  projects_file_path, user_access_file_path, invites_file_path = _resolve_bootstrap_file_paths(runtime_config)
3020
  state_store, supabase_validation_repository, state_backend_message = _build_supabase_state(runtime_config)
3021
  if runtime_config.hf_admin_storage_mode_enabled:
3022
+ state_store = None
3023
+ supabase_validation_repository = None
3024
  state_backend_message = "HF admin-owned private storage enabled."
3025
  allow_username_login, auth_mode_label, auth_mode_description = _resolve_username_login_policy(runtime_config)
3026
  bootstrap_warning = _bootstrap_auth_and_projects(
 
3664
  else ""
3665
  )
3666
  recovery_message = (
3667
+ " This private state repository will be discovered automatically after a Space restart."
 
3668
  if runtime_config.hf_admin_storage_mode_enabled
3669
  else ""
3670
  )
 
6155
  if runtime_config.hf_admin_storage_mode_enabled
6156
  else ("legacy experimental" if runtime_config.hf_bucket_validations_enabled else "disabled")
6157
  )
6158
+ hf_project_state_repos_label = (
6159
+ "automatic discovery"
6160
+ if runtime_config.hf_admin_storage_mode_enabled
6161
+ else str(len(runtime_config.hf_project_state_repos))
6162
+ )
6163
  hf_space_label = os.getenv("SPACE_ID") or "local runtime"
6164
  health_html = settings_health_html(
6165
  [
 
6169
  (
6170
  "HF project-state repos",
6171
  hf_project_state_repos_label,
6172
+ "ok" if runtime_config.hf_admin_storage_mode_enabled or runtime_config.hf_project_state_repos else "info",
6173
  ),
6174
  ("Auth mode", auth_mode_label, "ok" if not allow_username_login else "warn"),
6175
  ("Destructive write guard", "enabled", "ok"),
 
6188
  (
6189
  "HF administrator-owned storage is active. Private dataset and Bucket access use only the "
6190
  "protected Space storage credential; validator identities are authorized inside this app; "
6191
+ "private companion state repositories are discovered automatically for restart recovery."
 
 
 
 
6192
  if state_label == "HF admin-owned private storage"
6193
  else "Supabase persistence is active. Project and ACL removals are guarded against accidental broad overwrites."
6194
  )
tests/unit/test_app_factory_audio_helpers.py CHANGED
@@ -33,6 +33,7 @@ from src.ui.app_factory import (
33
  _load_projects_from_file,
34
  _load_user_access_from_file,
35
  _bootstrap_auth_and_projects,
 
36
  _persist_bootstrap_state,
37
  _resolve_username_login_policy,
38
  _resolve_project_fetch_token,
@@ -96,6 +97,21 @@ class FakeBucketInitializer:
96
  )
97
 
98
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
99
  class InaccessibleAudioService:
100
  def fetch(self, dataset_repo: str, audio_id: str, **kwargs: object) -> FakeFetchResult:
101
  _ = (dataset_repo, audio_id, kwargs)
@@ -310,6 +326,16 @@ def test_initialize_hf_admin_storage_requires_collaborative_app_visibility() ->
310
  assert bucket_initializer.calls == []
311
 
312
 
 
 
 
 
 
 
 
 
 
 
313
  def test_extract_audio_id_from_list_rows() -> None:
314
  rows = [["k1", "audio_01", "sp", 0.9, 0.0, 1.0]]
315
  assert _extract_audio_id(rows, 0) == "audio_01"
 
33
  _load_projects_from_file,
34
  _load_user_access_from_file,
35
  _bootstrap_auth_and_projects,
36
+ _discover_hf_admin_state_repos,
37
  _persist_bootstrap_state,
38
  _resolve_username_login_policy,
39
  _resolve_project_fetch_token,
 
97
  )
98
 
99
 
100
+ class FakeStateDiscoveryApi:
101
+ def whoami(self, token: str): # noqa: ANN001
102
+ assert token == "hf_storage"
103
+ return {"name": "jrrribeiro"}
104
+
105
+ def list_datasets(self, *, author: str, token: str): # noqa: ANN001
106
+ assert author == "jrrribeiro"
107
+ assert token == "hf_storage"
108
+ return [
109
+ SimpleNamespace(id="jrrribeiro/project-a_state"),
110
+ SimpleNamespace(id="jrrribeiro/audio-source"),
111
+ SimpleNamespace(id="someone-else/project-b_state"),
112
+ ]
113
+
114
+
115
  class InaccessibleAudioService:
116
  def fetch(self, dataset_repo: str, audio_id: str, **kwargs: object) -> FakeFetchResult:
117
  _ = (dataset_repo, audio_id, kwargs)
 
326
  assert bucket_initializer.calls == []
327
 
328
 
329
+ def test_discover_hf_admin_state_repos_filters_personal_companion_datasets() -> None:
330
+ repo_ids, warning = _discover_hf_admin_state_repos(
331
+ token="hf_storage",
332
+ api=FakeStateDiscoveryApi(), # type: ignore[arg-type]
333
+ )
334
+
335
+ assert warning == ""
336
+ assert repo_ids == ("jrrribeiro/project-a_state",)
337
+
338
+
339
  def test_extract_audio_id_from_list_rows() -> None:
340
  rows = [["k1", "audio_01", "sp", 0.9, 0.0, 1.0]]
341
  assert _extract_audio_id(rows, 0) == "audio_01"