File size: 20,121 Bytes
c5ce085
4e31d1f
b171a3e
 
 
 
0aec723
b171a3e
 
 
 
 
055fe47
cc88305
2b0510e
cc88305
055fe47
2b0510e
cc88305
2f1a5fb
cc88305
2f1a5fb
055fe47
2f1a5fb
055fe47
 
2f1a5fb
055fe47
cc88305
2f1a5fb
cc88305
055fe47
cc88305
2b0510e
055fe47
2b0510e
055fe47
2b0510e
f0bff77
055fe47
cc88305
055fe47
cc88305
055fe47
 
 
 
 
20568c9
055fe47
20568c9
055fe47
20568c9
4e68920
64cfee6
055fe47
 
64cfee6
54ead58
20568c9
055fe47
2f1a5fb
055fe47
 
 
2f1a5fb
055fe47
 
2f1a5fb
055fe47
 
 
 
 
 
2f1a5fb
055fe47
 
2f1a5fb
 
055fe47
2f1a5fb
055fe47
 
 
 
 
2f1a5fb
055fe47
 
2b0510e
 
 
2f1a5fb
055fe47
2b0510e
2f1a5fb
055fe47
 
2b0510e
 
2f1a5fb
 
055fe47
2f1a5fb
055fe47
2f1a5fb
055fe47
 
 
2f1a5fb
055fe47
 
2f1a5fb
2b0510e
 
 
055fe47
 
2f1a5fb
055fe47
 
2f1a5fb
055fe47
 
 
2f1a5fb
055fe47
2f1a5fb
055fe47
 
 
 
 
2f1a5fb
055fe47
 
 
 
 
2f1a5fb
055fe47
 
 
 
 
2f1a5fb
055fe47
2f1a5fb
055fe47
2f1a5fb
055fe47
 
 
 
2f1a5fb
055fe47
2f1a5fb
055fe47
 
 
 
2b0510e
 
 
055fe47
2b0510e
055fe47
 
 
2f1a5fb
055fe47
2f1a5fb
055fe47
 
 
 
2f1a5fb
055fe47
2f1a5fb
055fe47
 
 
 
2f1a5fb
055fe47
2f1a5fb
055fe47
 
 
 
 
 
2f1a5fb
055fe47
2f1a5fb
055fe47
2f1a5fb
055fe47
 
 
 
 
 
 
 
 
2f1a5fb
 
055fe47
2f1a5fb
055fe47
2f1a5fb
055fe47
 
 
 
2f1a5fb
055fe47
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2f1a5fb
055fe47
2f1a5fb
 
055fe47
2f1a5fb
 
055fe47
 
2f1a5fb
055fe47
 
2f1a5fb
055fe47
2f1a5fb
 
055fe47
 
 
 
 
 
 
 
2f1a5fb
 
055fe47
2f1a5fb
055fe47
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2f1a5fb
055fe47
 
 
 
2f1a5fb
055fe47
2f1a5fb
055fe47
2f1a5fb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
20568c9
 
2f1a5fb
20568c9
 
2f1a5fb
 
 
 
 
 
 
 
 
 
20568c9
2f1a5fb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
20568c9
 
2f1a5fb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
cc88305
2f1a5fb
 
 
 
 
 
 
 
 
 
cc88305
2f1a5fb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
cc88305
 
 
 
 
 
 
 
2f1a5fb
cc88305
 
 
 
 
2f1a5fb
cc88305
2f1a5fb
 
 
cc88305
2f1a5fb
cc88305
2f1a5fb
cc88305
2f1a5fb
 
 
 
 
cc88305
 
2f1a5fb
cc88305
 
2f1a5fb
 
cc88305
2f1a5fb
cc88305
 
2f1a5fb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
cc88305
 
2f1a5fb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
cc88305
2f1a5fb
cc88305
2f1a5fb
cc88305
2f1a5fb
 
 
 
 
 
cc88305
2f1a5fb
cb278cb
2f1a5fb
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
---
title: BirdNET-Uploader-App
emoji: "🐦"
colorFrom: green
colorTo: blue
sdk: gradio
sdk_version: "5.23.1"
python_version: "3.11"
app_file: app.py
pinned: false
---

# BirdNET Uploader

A robust uploader for audio segments and detection metadata to Hugging Face datasets. Choose your upload path based on data size:

- **Web Path** (~1 GB): Browser-based archive upload with live progress
- **CLI Path** (unlimited): Direct folder upload for large datasets and Windows portable use

## Contents

- [Quick Start](#quick-start)
- [Features](#features)
- [Configuration](#configuration)
- [Typical Workflows](#typical-workflows)
- [CLI Reference](#cli-reference)
- [Troubleshooting](#troubleshooting)
- [Development](#development)

## Overview

This repository is a focused tool for reliably uploading large audio collections and optional detection CSVs to Hugging Face datasets. It emphasizes:

- **Two upload paths**: Small archives via web UI (≀1 GB) or direct folder uploads via CLI/Windows portable
- **Resumable sessions**: Checkpoints stored locally allow recovery from interruptions
- **Fast direct uploads**: The CLI uploads the chosen folder directly to the dataset without packing it into a single archive
- **Live progress**: Real-time feedback during uploads
- **Optional metadata**: CSV uploads remain supported, but manifest/shard generation is only for legacy workflows

## Quick Start

### Web UI (Browser-Based, ≀1 GB)

1. Open [BirdNET Uploader on Hugging Face Spaces](https://huggingface.co/spaces/YOUR_ORG/birdnet-uploader)
2. Enter your **Hugging Face token** (from [hf.co/settings/tokens](https://huggingface.co/settings/tokens))
3. Specify your dataset **repo ID** (`username/dataset-name`)
4. Upload a `.tar.gz`, `.zip`, or `.tar` archive containing audio files (optionally include a CSV)
5. Watch live progress and get a summary when complete

**Note**: Archive size is limited to ~1 GB due to browser memory constraints. For larger datasets, use the Windows portable or CLI.

### Windows Portable (Unlimited Size)

1. [Download birdnet-uploader-1.0.4-windows.zip](https://huggingface.co/datasets/jrrribeiro/birdnet-uploader-releases/resolve/main/releases/v1.0.4/birdnet-uploader-1.0.4-windows.zip) (~109 MB)
2. Extract the archive
3. Double-click `birdnet-uploader.exe` to launch the web UI
   - Or use CLI commands: `birdnet-uploader login`, `birdnet-uploader upload --help`
   - See [WINDOWS_PORTABLE_SETUP.md](./WINDOWS_PORTABLE_SETUP.md) for detailed instructions
   - Issues? Check [TROUBLESHOOTING.md](./TROUBLESHOOTING.md)

### Local CLI (Python 3.11+)

```bash
# Install
pip install -r requirements.txt

# Login (stores token securely)
python app.py login

# Upload local folder
python app.py upload \
  --repo-id username/dataset-name \
  --segments /path/to/audio/folder \
  --csv /path/to/detections.csv (optional) \
  --workers 4

# Check upload status
python app.py resume <session-id>
```

## Features

### Archive Upload (Web)
- **Supported formats**: `.tar`, `.tar.gz`, `.zip`
- **Size limit**: ~1 GB (due to browser constraints)
- **Automatic extraction**: Archives are extracted server-side
- **Live progress**: Real-time upload percentage with file counts

### Large Dataset Upload (CLI)
- **Unlimited size**: No practical file size limit
- **Direct folder streaming**: Uploads the local folder directly to the dataset path you choose
- **Parallel workers**: Optimized Hub-side transfer and retries for large uploads
- **Progress tracking**: Folder-level status and local checkpoints for recovery metadata

### Dataset Structure
The direct CLI path uploads your local folder structure as-is into the dataset, typically under `audio/`:
```
dataset-repo/
β”œβ”€β”€ audio/              # Uploaded audio files (organized by species if CSV)
└── index/
  └── detections.csv  # Optional metadata CSV uploaded directly, if provided
```

## Configuration

### Environment Variables

```bash
# Web UI port (default: 7860)
BIRDNET_UPLOADER_PORT=8000

# CLI mode (use app.py as CLI instead of web UI)
BIRDNET_UPLOADER_CLI=true

# High-throughput Xet mode for large uploads
HF_XET_HIGH_PERFORMANCE=1

# Session storage (default: ~/.birdnet-uploader/sessions)
BIRDNET_UPLOADER_SESSION_DIR=/custom/path/sessions

# Cache directory (default: ~/.birdnet-uploader/cache)
BIRDNET_UPLOADER_CACHE_DIR=/custom/path/cache

# Hugging Face token (alternative to secure storage)
HF_TOKEN=hf_xxxxxxxxxxxx
```

## Typical Workflows

### Scenario 1: Small Archive Upload (≀1 GB)
1. Create a `.tar.gz` archive of audio files with optional CSV
2. Use web UI at [Hugging Face Spaces](https://huggingface.co/spaces/YOUR_ORG/birdnet-uploader)
3. Paste token, enter dataset ID, upload archive
4. **Done** - files appear in dataset within minutes

### Scenario 2: Large Archive on Windows
1. Create a `.tar.gz` or `.zip` archive (any size)
2. Download and extract Windows portable
3. Double-click `birdnet-uploader.exe` β†’ launches web UI on localhost:7860
4. Same workflow as web UI, but runs locally (no size limits)

### Scenario 3: Local Folder Upload (CLI)
1. Navigate to audio folder root in terminal
2. Run: `python app.py upload --repo-id user/dataset --segments .`
3. Watch progress in real-time
4. Resume if interrupted: `python app.py resume <session-id>`

## CLI Reference

### Login

```bash
python app.py login
```
Stores Hugging Face token securely in system keyring.

### Upload

```bash
python app.py upload \
  --repo-id user/dataset \
  --segments /path/to/audio \
  --csv /path/to/detections.csv \      # Optional, uploaded directly as index/detections.csv
  --workers 4 \                         # Default: 4 (legacy mode only)
  --session-id my-session \             # Optional metadata checkpoint
  --remote-base my_audio_folder \       # Default: "audio"
  --upload-mode direct \                # Default: direct folder upload
  --dry-run                             # Show what would be uploaded
  --verbose                             # Detailed logging
```

### Scan

```bash
python app.py scan --segments /path/to/audio
```
Shows file count, total size, and species breakdown.

### Resume

```bash
python app.py resume <session-id>
```
Checks session status and shows last checkpoint.

### Init Repo

```bash
python app.py init-repo \
  --repo-id user/dataset \
  --private                             # Default: private
```
Creates dataset structure on Hugging Face.

## Resumable Uploads

Session checkpoints are saved locally as JSON. If an upload is interrupted:

```bash
# Check session status
python app.py resume abc-session-id-xyz

# Resume from last checkpoint (automatically skips already-uploaded files)
python app.py upload \
  --repo-id user/dataset \
  --segments ./audio \
  --session-id abc-session-id-xyz
```

Checkpoint location: `~/.birdnet-uploader/sessions/upload-YYYYMMDDTHHMMSSZ/checkpoint.json`

## Authentication

### Secure Token Storage
- **First time**: `python app.py login` stores token in system keyring (Windows, macOS, Linux)
- **Environment override**: `HF_TOKEN=hf_xxxx python app.py upload ...`
- **Explicit flag**: `python app.py upload --token hf_xxxx ...`

### Token Permissions
Ensure your token has `write` permissions on the target dataset:
1. Go to [hf.co/settings/tokens](https://huggingface.co/settings/tokens)
2. Create or edit a token
3. Set permissions: `repo > write` (not `read`)
4. Restrict to specific repo if desired

## Troubleshooting

### "Upload exceeds 1 GB limit"
- Switch to Windows portable or CLI for larger datasets
- Compress archive more aggressively

### "CSV upload failed"
- Ensure CSV has proper headers and encoding (UTF-8)
- File size should match audio files metadata (e.g., `audio_file` column)

### "Session has no checkpoint"
- Session ID may be invalid or expired
- Check `~/.birdnet-uploader/sessions/` for available sessions
- Restart upload without `--session-id` to create a new session

### Network timeouts on large files
- Use CLI with more workers: `--workers 8`
- Check internet stability
- Resume from checkpoint if interrupted

## Development

### Build Windows Portable

```bash
# Install build dependencies
pip install pyinstaller

# Build release
python build/release_uploader.py --version 1.0.0

# Output: build/release/birdnet-uploader-1.0.0-windows.zip
```

### Local Testing

```bash
# Web UI
python app.py

# CLI
python app.py --help
python app.py login
python app.py scan --segments ./test_audio
python app.py upload --repo-id my-test/uploader --segments ./test_audio --dry-run
```

## Architecture

### Core Components

- **`src/uploader/web_ui.py`**: Gradio interface for browser uploads
- **`src/uploader/main.py`**: Click CLI for local uploads
- **`src/uploader/batch_uploader.py`**: File upload orchestration with retries
- **`src/uploader/session_manager.py`**: Checkpoint persistence and resumability
- **`src/uploader/deduplicator.py`**: Remote hash checking to avoid re-uploads
- **`src/uploader/scanner.py`**: Local folder scanning and metadata extraction
- **`src/uploader/manifest.py`**: Dataset metadata and Parquet shard generation

### Data Flow

1. **Upload**: Archive extraction or folder scan
2. **Deduplication**: Check remote hashes to skip existing files
3. **Upload**: Push files to HF dataset in parallel
4. **Metadata**: Generate and upload manifest + CSV + Parquet shards
5. **Checkpoint**: Save session state for resumable recovery

## Limitations

- Web UI: ~1 GB per upload (browser constraint)
- CSV: Must have `audio_file` or `file` column for audio file names
- Audio formats: `.wav`, `.mp3`, `.flac`, `.ogg`, `.m4a`
- Archive formats: `.tar`, `.tar.gz`, `.zip`

## License

[MIT License](LICENSE)

1. Download the zip from Hugging Face.
2. Unzip into a clean folder.
3. Run the executable without Python installed.
4. Authenticate with a Hugging Face token.
5. Create a session and confirm resume/recovery works.

### What It Does

- authenticates with a Hugging Face token stored in the OS keyring
- creates or validates dataset repositories
- scans a local audio folder recursively
- computes per-file SHA-256 hashes
- deduplicates against a cached remote index
- uploads files with retries and session checkpoints
- resumes from the last checkpoint after close or crash

### Main Commands


The CLI entrypoint is:

```bash
python -m src.uploader_cli.main --help
```

Commands:

- `login`: validate and store a Hugging Face token
- `init-repo`: create and initialize a dataset repository
- `scan`: scan a local folder and summarize audio files
- `start`: create a local upload session checkpoint
- `resume`: print the saved checkpoint for a session

### Example Workflow

```bash
# Authenticate once
python -m src.uploader_cli.main login

# Create or initialize the dataset repository
python -m src.uploader_cli.main init-repo --repo-id alice/birdnet-2026 --private

# Inspect the local folder before upload
python -m src.uploader_cli.main scan --segments C:\audio\segments

# Create a resumable local session checkpoint
python -m src.uploader_cli.main start --repo-id alice/birdnet-2026 --segments C:\audio\segments


# Resume and inspect the checkpoint later
python -m src.uploader_cli.main resume upload-20260429T120000Z
```

### Upload Engine Building Blocks

The uploader CLI is built from the following modules:

- `src/uploader_cli/auth_service.py`: token validation and keyring storage
- `src/uploader_cli/repo_service.py`: dataset creation and structure initialization
- `src/uploader_cli/scanner.py`: recursive audio discovery and metadata collection
- `src/uploader_cli/hash_utils.py`: streaming SHA-256 helpers
- `src/uploader_cli/deduplicator.py`: remote index cache and skip/upload decisions
- `src/uploader_cli/session_manager.py`: checkpoint and session metadata persistence
- `src/uploader_cli/batch_uploader.py`: retrying upload orchestration
- `src/uploader_cli/error_handler.py`: user-friendly error formatting

### Local Data Layout

The uploader stores local state under the user's profile directory by default:

```text
~/.birdnet-uploader/
  sessions/
    upload-YYYYMMDDTHHMMSSZ/
      metadata.json
      checkpoint.json
  cache/
    dedup/
      <repo-hash>.json
  logs/
```

Environment overrides are supported:

- `BIRDNET_UPLOADER_SESSION_DIR`
- `BIRDNET_UPLOADER_CACHE_DIR`
- `BIRDNET_UPLOADER_LOG_DIR`

### Dataset Layout Expected by the Validator

The uploader is designed to preserve the validator's expected structure:

```text
audio/{project_slug}/...        # uploaded audio files
index/shards/...                # parquet shard files
index/manifest.json             # index manifest
validations/...                 # validation outputs
audit/ingestion-runs/...        # ingestion run history
```

### Uploader Configuration

Runtime constants are centralized in `src/uploader_cli/config.py`:

- `APP_NAME`
- `SCHEMA_VERSION`
- `AUDIO_EXTENSIONS`
- `INDEX_SHARD_SIZE`
- `MAX_BATCH_SIZE`
- `RETRY_MAX_ATTEMPTS`
- `RETRY_INITIAL_BACKOFF_SECONDS`
- `RETRY_MAX_BACKOFF_SECONDS`
- `KEYRING_SERVICE`
- `KEYRING_ACCOUNT`

### Resumability Guarantees

The uploader uses local checkpoints so it can recover from:

- process crashes
- app close
- transient network failures
- partial progress during large uploads

The checkpoint stores session metadata and cumulative counters such as:

- uploaded count
- skipped count
- failed count
- last completed file
- last failed file
- total uploaded bytes

## Quick Start

### Local Development

1. Create and activate a Python 3.11+ virtual environment.
2. Install dependencies:

```bash
pip install -r requirements.txt
```

3. Run the validator app:

```bash
python app.py
```

4. Run the uploader CLI help:

```bash
python -m src.uploader_cli.main --help
```

### First-Time Uploader Setup

1. Log in with a Hugging Face token:

```bash
python -m src.uploader_cli.main login
```

2. Validate your target repository exists or create it:

```bash
python -m src.uploader_cli.main init-repo --repo-id alice/birdnet-2026 --private
```

3. Scan your local dataset folder:

```bash
python -m src.uploader_cli.main scan --segments C:\audio\segments
```

4. Start a resumable session:

```bash
python -m src.uploader_cli.main start --repo-id alice/birdnet-2026 --segments C:\audio\segments
```

## Configuration

### Validator App Environment Variables

- `BIRDNET_DETECTIONS_FILE`
- `BIRDNET_VALIDATIONS_DIR`
- `BIRDNET_PAGE_SIZE`
- `BIRDNET_PROJECTS_FILE`
- `BIRDNET_USER_ACCESS_FILE`
- `BIRDNET_INVITES_FILE`
- `BIRDNET_INVITE_TTL_HOURS`
- `BIRDNET_BOOTSTRAP_DIR`
- `BIRDNET_ENABLE_DEMO_BOOTSTRAP`
- `BIRDNET_EMAILJS_ENABLED`
- `BIRDNET_INVITE_EMAIL_ENABLED`
- `BIRDNET_INVITE_EMAIL_SENDER`
- `BIRDNET_INVITE_EMAIL_LOGIN_URL`
- `BIRDNET_EMAILJS_SERVICE_ID`
- `BIRDNET_EMAILJS_TEMPLATE_ID`
- `BIRDNET_EMAILJS_TEMPLATE_ID_USERNAME_ONLY`
- `BIRDNET_EMAILJS_TEMPLATE_ID_EMAIL_ONLY`
- `BIRDNET_EMAILJS_TEMPLATE_ID_DUAL`
- `BIRDNET_EMAILJS_PUBLIC_KEY`
- `BIRDNET_EMAILJS_ENDPOINT`
- `BIRDNET_EMAILJS_TIMEOUT_SECONDS`

### Uploader CLI Environment Variables

- `BIRDNET_UPLOADER_SESSION_DIR`
- `BIRDNET_UPLOADER_CACHE_DIR`
- `BIRDNET_UPLOADER_LOG_DIR`

## Typical Flows

### Validation Flow

1. Login with a valid user.
2. Select an authorized project.
3. Load queue items and apply filters when needed.
4. Listen to selected audio and submit validation status.
5. Resolve conflicts when concurrent updates occur.
6. Export or inspect project validation report.

### Upload Flow

1. Authenticate once with a Hugging Face token.
2. Create or validate a dataset repository.
3. Scan the local folder recursively.
4. Deduplicate against remote files.
5. Upload with retries and checkpoints.
6. Resume later from the saved session metadata if needed.

## Repository Structure

- `app.py`: validator app entrypoint
- `src/auth`: authentication/session and ACL logic
- `src/domain`: domain models
- `src/repositories`: in-memory and append-only persistence
- `src/services`: queue, validation, and audio fetch services
- `src/ui`: Gradio interface composition and callbacks
- `src/uploader_cli`: local uploader CLI, session persistence, deduplication, and retrying uploads
- `tests`: unit and integration test suites
- `docs`: collaborative workflow and QA checklists

## Project Status

Validator workflow is active and under continuous development, with emphasis on reliability, auditability, and operator productivity.

The uploader CLI is being built as a companion local workflow for large ingest tasks, with a focus on resumability and validator compatibility.

## FAQ

### Is this app intended for production use?

Yes, for validator workflows. It is structured for multi-project access control, audit-friendly writes, and collaborative validation.

### Is the uploader CLI meant to replace the validator app?

No. It is a companion tool for large dataset uploads and repository initialization.

### Does the app preload all audio files?

No. Audio is fetched on demand for the selected detection, helping keep memory and network usage under control.

### Can multiple validators work at the same time?

Yes. The validation flow uses optimistic concurrency checks to detect and handle conflicting updates.

### Can this run on Hugging Face Spaces?

Yes. The validator app is designed for Gradio deployment on Hugging Face Spaces.

### Can the uploader resume after closing the app?

Yes. It persists checkpoints and session metadata locally and can be resumed later.

### Does the uploader support deduplication?

Yes. It checks the remote dataset index, caches the remote file list locally, and skips files that are already present.

## Troubleshooting

### The app does not start locally

1. Confirm Python 3.11+ is active.
2. Reinstall dependencies with `pip install -r requirements.txt`.
3. Run `python app.py` from the repository root.

### Port 7860 is already in use

Stop the existing process using that port, or launch after setting a different `GRADIO_SERVER_PORT` environment variable.

### Login works but no project appears

Your user likely has no project assignment. Verify `BIRDNET_PROJECTS_FILE` and `BIRDNET_USER_ACCESS_FILE`, or add project access through the admin flow.

### Seed warning appears in Validation tab

If you see a seed warning banner:

1. Verify `BIRDNET_DETECTIONS_FILE` points to an existing file.
2. Confirm the file is valid UTF-8 JSON.
3. Confirm each project maps to a list of detections, or each list item includes `project_slug`.
4. If needed, unset `BIRDNET_DETECTIONS_FILE` to fall back to default demo detections.

### Audio does not load for a detection

Check that:

1. `audio_id` exists and is valid for the selected project.
2. The dataset repository is reachable.
3. You have permission to read the project dataset.

### I get conflict messages while validating

This means another validator updated the same detection first. Refresh and reapply your decision on the newest version.

### The uploader says the token cannot be saved

Make sure your OS keyring backend is available. On Windows, the default credential store should work automatically.

### The uploader skips files unexpectedly

It may have found them in the remote dataset index cache. Clear the uploader cache directory if you want to force a fresh remote listing.

### Resume shows an empty checkpoint

Check that you ran the uploader `start` command at least once and that the session directory still exists.

## Recent Updates

- Integrated Validation tab flow in the multi-project app (no placeholder stage).
- Added project-scoped queue badge (`Queue: N`) and improved queue context feedback.
- Added seed-file warning banner with actionable remediation guidance in Validation.
- Standardized all user-facing UI text and feedback messages in English.
- Updated authentication/session timestamps to timezone-aware UTC handling.
- Added a local uploader CLI with login, repository initialization, scan, deduplication, checkpointing, and resumable upload foundations.

## Related Documents

- `BIRDNET_UPLOADER_REDESIGN.md`: uploader UX and architecture notes
- `SPRINT_ROADMAP.md`: implementation roadmap
- `SPRINT1_KICKOFF.md`: sprint kickoff details