Spaces:
Sleeping
Download README.md from jrrribeiro/BirdNET-Uploader-App: direct link, hf CLI and curl.
- Browser
- Download file 20.1 kB
-
https://huggingface.co/spaces/jrrribeiro/BirdNET-Uploader-App/resolve/main/README.md
- Command line
-
hf download hf://spaces/jrrribeiro/BirdNET-Uploader-App/README.md
-
curl -L -o README.md https://huggingface.co/spaces/jrrribeiro/BirdNET-Uploader-App/resolve/main/README.md
A newer version of the Gradio SDK is available: 6.32.0
title: BirdNET-Uploader-App
emoji: 🐦
colorFrom: green
colorTo: blue
sdk: gradio
sdk_version: 5.23.1
python_version: '3.11'
app_file: app.py
pinned: false
BirdNET Uploader
A robust uploader for audio segments and detection metadata to Hugging Face datasets. Choose your upload path based on data size:
- Web Path (~1 GB): Browser-based archive upload with live progress
- CLI Path (unlimited): Direct folder upload for large datasets and Windows portable use
Contents
Overview
This repository is a focused tool for reliably uploading large audio collections and optional detection CSVs to Hugging Face datasets. It emphasizes:
- Two upload paths: Small archives via web UI (≤1 GB) or direct folder uploads via CLI/Windows portable
- Resumable sessions: Checkpoints stored locally allow recovery from interruptions
- Fast direct uploads: The CLI uploads the chosen folder directly to the dataset without packing it into a single archive
- Live progress: Real-time feedback during uploads
- Optional metadata: CSV uploads remain supported, but manifest/shard generation is only for legacy workflows
Quick Start
Web UI (Browser-Based, ≤1 GB)
- Open BirdNET Uploader on Hugging Face Spaces
- Enter your Hugging Face token (from hf.co/settings/tokens)
- Specify your dataset repo ID (
username/dataset-name) - Upload a
.tar.gz,.zip, or.tararchive containing audio files (optionally include a CSV) - Watch live progress and get a summary when complete
Note: Archive size is limited to ~1 GB due to browser memory constraints. For larger datasets, use the Windows portable or CLI.
Windows Portable (Unlimited Size)
- Download birdnet-uploader-1.0.4-windows.zip (~109 MB)
- Extract the archive
- Double-click
birdnet-uploader.exeto launch the web UI- Or use CLI commands:
birdnet-uploader login,birdnet-uploader upload --help - See WINDOWS_PORTABLE_SETUP.md for detailed instructions
- Issues? Check TROUBLESHOOTING.md
- Or use CLI commands:
Local CLI (Python 3.11+)
# Install
pip install -r requirements.txt
# Login (stores token securely)
python app.py login
# Upload local folder
python app.py upload \
--repo-id username/dataset-name \
--segments /path/to/audio/folder \
--csv /path/to/detections.csv (optional) \
--workers 4
# Check upload status
python app.py resume <session-id>
Features
Archive Upload (Web)
- Supported formats:
.tar,.tar.gz,.zip - Size limit: ~1 GB (due to browser constraints)
- Automatic extraction: Archives are extracted server-side
- Live progress: Real-time upload percentage with file counts
Large Dataset Upload (CLI)
- Unlimited size: No practical file size limit
- Direct folder streaming: Uploads the local folder directly to the dataset path you choose
- Parallel workers: Optimized Hub-side transfer and retries for large uploads
- Progress tracking: Folder-level status and local checkpoints for recovery metadata
Dataset Structure
The direct CLI path uploads your local folder structure as-is into the dataset, typically under audio/:
dataset-repo/
├── audio/ # Uploaded audio files (organized by species if CSV)
└── index/
└── detections.csv # Optional metadata CSV uploaded directly, if provided
Configuration
Environment Variables
# Web UI port (default: 7860)
BIRDNET_UPLOADER_PORT=8000
# CLI mode (use app.py as CLI instead of web UI)
BIRDNET_UPLOADER_CLI=true
# High-throughput Xet mode for large uploads
HF_XET_HIGH_PERFORMANCE=1
# Session storage (default: ~/.birdnet-uploader/sessions)
BIRDNET_UPLOADER_SESSION_DIR=/custom/path/sessions
# Cache directory (default: ~/.birdnet-uploader/cache)
BIRDNET_UPLOADER_CACHE_DIR=/custom/path/cache
# Hugging Face token (alternative to secure storage)
HF_TOKEN=hf_xxxxxxxxxxxx
Typical Workflows
Scenario 1: Small Archive Upload (≤1 GB)
- Create a
.tar.gzarchive of audio files with optional CSV - Use web UI at Hugging Face Spaces
- Paste token, enter dataset ID, upload archive
- Done - files appear in dataset within minutes
Scenario 2: Large Archive on Windows
- Create a
.tar.gzor.ziparchive (any size) - Download and extract Windows portable
- Double-click
birdnet-uploader.exe→ launches web UI on localhost:7860 - Same workflow as web UI, but runs locally (no size limits)
Scenario 3: Local Folder Upload (CLI)
- Navigate to audio folder root in terminal
- Run:
python app.py upload --repo-id user/dataset --segments . - Watch progress in real-time
- Resume if interrupted:
python app.py resume <session-id>
CLI Reference
Login
python app.py login
Stores Hugging Face token securely in system keyring.
Upload
python app.py upload \
--repo-id user/dataset \
--segments /path/to/audio \
--csv /path/to/detections.csv \ # Optional, uploaded directly as index/detections.csv
--workers 4 \ # Default: 4 (legacy mode only)
--session-id my-session \ # Optional metadata checkpoint
--remote-base my_audio_folder \ # Default: "audio"
--upload-mode direct \ # Default: direct folder upload
--dry-run # Show what would be uploaded
--verbose # Detailed logging
Scan
python app.py scan --segments /path/to/audio
Shows file count, total size, and species breakdown.
Resume
python app.py resume <session-id>
Checks session status and shows last checkpoint.
Init Repo
python app.py init-repo \
--repo-id user/dataset \
--private # Default: private
Creates dataset structure on Hugging Face.
Resumable Uploads
Session checkpoints are saved locally as JSON. If an upload is interrupted:
# Check session status
python app.py resume abc-session-id-xyz
# Resume from last checkpoint (automatically skips already-uploaded files)
python app.py upload \
--repo-id user/dataset \
--segments ./audio \
--session-id abc-session-id-xyz
Checkpoint location: ~/.birdnet-uploader/sessions/upload-YYYYMMDDTHHMMSSZ/checkpoint.json
Authentication
Secure Token Storage
- First time:
python app.py loginstores token in system keyring (Windows, macOS, Linux) - Environment override:
HF_TOKEN=hf_xxxx python app.py upload ... - Explicit flag:
python app.py upload --token hf_xxxx ...
Token Permissions
Ensure your token has write permissions on the target dataset:
- Go to hf.co/settings/tokens
- Create or edit a token
- Set permissions:
repo > write(notread) - Restrict to specific repo if desired
Troubleshooting
"Upload exceeds 1 GB limit"
- Switch to Windows portable or CLI for larger datasets
- Compress archive more aggressively
"CSV upload failed"
- Ensure CSV has proper headers and encoding (UTF-8)
- File size should match audio files metadata (e.g.,
audio_filecolumn)
"Session has no checkpoint"
- Session ID may be invalid or expired
- Check
~/.birdnet-uploader/sessions/for available sessions - Restart upload without
--session-idto create a new session
Network timeouts on large files
- Use CLI with more workers:
--workers 8 - Check internet stability
- Resume from checkpoint if interrupted
Development
Build Windows Portable
# Install build dependencies
pip install pyinstaller
# Build release
python build/release_uploader.py --version 1.0.0
# Output: build/release/birdnet-uploader-1.0.0-windows.zip
Local Testing
# Web UI
python app.py
# CLI
python app.py --help
python app.py login
python app.py scan --segments ./test_audio
python app.py upload --repo-id my-test/uploader --segments ./test_audio --dry-run
Architecture
Core Components
src/uploader/web_ui.py: Gradio interface for browser uploadssrc/uploader/main.py: Click CLI for local uploadssrc/uploader/batch_uploader.py: File upload orchestration with retriessrc/uploader/session_manager.py: Checkpoint persistence and resumabilitysrc/uploader/deduplicator.py: Remote hash checking to avoid re-uploadssrc/uploader/scanner.py: Local folder scanning and metadata extractionsrc/uploader/manifest.py: Dataset metadata and Parquet shard generation
Data Flow
- Upload: Archive extraction or folder scan
- Deduplication: Check remote hashes to skip existing files
- Upload: Push files to HF dataset in parallel
- Metadata: Generate and upload manifest + CSV + Parquet shards
- Checkpoint: Save session state for resumable recovery
Limitations
- Web UI: ~1 GB per upload (browser constraint)
- CSV: Must have
audio_fileorfilecolumn for audio file names - Audio formats:
.wav,.mp3,.flac,.ogg,.m4a - Archive formats:
.tar,.tar.gz,.zip
License
- Download the zip from Hugging Face.
- Unzip into a clean folder.
- Run the executable without Python installed.
- Authenticate with a Hugging Face token.
- Create a session and confirm resume/recovery works.
What It Does
- authenticates with a Hugging Face token stored in the OS keyring
- creates or validates dataset repositories
- scans a local audio folder recursively
- computes per-file SHA-256 hashes
- deduplicates against a cached remote index
- uploads files with retries and session checkpoints
- resumes from the last checkpoint after close or crash
Main Commands
The CLI entrypoint is:
python -m src.uploader_cli.main --help
Commands:
login: validate and store a Hugging Face tokeninit-repo: create and initialize a dataset repositoryscan: scan a local folder and summarize audio filesstart: create a local upload session checkpointresume: print the saved checkpoint for a session
Example Workflow
# Authenticate once
python -m src.uploader_cli.main login
# Create or initialize the dataset repository
python -m src.uploader_cli.main init-repo --repo-id alice/birdnet-2026 --private
# Inspect the local folder before upload
python -m src.uploader_cli.main scan --segments C:\audio\segments
# Create a resumable local session checkpoint
python -m src.uploader_cli.main start --repo-id alice/birdnet-2026 --segments C:\audio\segments
# Resume and inspect the checkpoint later
python -m src.uploader_cli.main resume upload-20260429T120000Z
Upload Engine Building Blocks
The uploader CLI is built from the following modules:
src/uploader_cli/auth_service.py: token validation and keyring storagesrc/uploader_cli/repo_service.py: dataset creation and structure initializationsrc/uploader_cli/scanner.py: recursive audio discovery and metadata collectionsrc/uploader_cli/hash_utils.py: streaming SHA-256 helperssrc/uploader_cli/deduplicator.py: remote index cache and skip/upload decisionssrc/uploader_cli/session_manager.py: checkpoint and session metadata persistencesrc/uploader_cli/batch_uploader.py: retrying upload orchestrationsrc/uploader_cli/error_handler.py: user-friendly error formatting
Local Data Layout
The uploader stores local state under the user's profile directory by default:
~/.birdnet-uploader/
sessions/
upload-YYYYMMDDTHHMMSSZ/
metadata.json
checkpoint.json
cache/
dedup/
<repo-hash>.json
logs/
Environment overrides are supported:
BIRDNET_UPLOADER_SESSION_DIRBIRDNET_UPLOADER_CACHE_DIRBIRDNET_UPLOADER_LOG_DIR
Dataset Layout Expected by the Validator
The uploader is designed to preserve the validator's expected structure:
audio/{project_slug}/... # uploaded audio files
index/shards/... # parquet shard files
index/manifest.json # index manifest
validations/... # validation outputs
audit/ingestion-runs/... # ingestion run history
Uploader Configuration
Runtime constants are centralized in src/uploader_cli/config.py:
APP_NAMESCHEMA_VERSIONAUDIO_EXTENSIONSINDEX_SHARD_SIZEMAX_BATCH_SIZERETRY_MAX_ATTEMPTSRETRY_INITIAL_BACKOFF_SECONDSRETRY_MAX_BACKOFF_SECONDSKEYRING_SERVICEKEYRING_ACCOUNT
Resumability Guarantees
The uploader uses local checkpoints so it can recover from:
- process crashes
- app close
- transient network failures
- partial progress during large uploads
The checkpoint stores session metadata and cumulative counters such as:
- uploaded count
- skipped count
- failed count
- last completed file
- last failed file
- total uploaded bytes
Quick Start
Local Development
- Create and activate a Python 3.11+ virtual environment.
- Install dependencies:
pip install -r requirements.txt
- Run the validator app:
python app.py
- Run the uploader CLI help:
python -m src.uploader_cli.main --help
First-Time Uploader Setup
- Log in with a Hugging Face token:
python -m src.uploader_cli.main login
- Validate your target repository exists or create it:
python -m src.uploader_cli.main init-repo --repo-id alice/birdnet-2026 --private
- Scan your local dataset folder:
python -m src.uploader_cli.main scan --segments C:\audio\segments
- Start a resumable session:
python -m src.uploader_cli.main start --repo-id alice/birdnet-2026 --segments C:\audio\segments
Configuration
Validator App Environment Variables
BIRDNET_DETECTIONS_FILEBIRDNET_VALIDATIONS_DIRBIRDNET_PAGE_SIZEBIRDNET_PROJECTS_FILEBIRDNET_USER_ACCESS_FILEBIRDNET_INVITES_FILEBIRDNET_INVITE_TTL_HOURSBIRDNET_BOOTSTRAP_DIRBIRDNET_ENABLE_DEMO_BOOTSTRAPBIRDNET_EMAILJS_ENABLEDBIRDNET_INVITE_EMAIL_ENABLEDBIRDNET_INVITE_EMAIL_SENDERBIRDNET_INVITE_EMAIL_LOGIN_URLBIRDNET_EMAILJS_SERVICE_IDBIRDNET_EMAILJS_TEMPLATE_IDBIRDNET_EMAILJS_TEMPLATE_ID_USERNAME_ONLYBIRDNET_EMAILJS_TEMPLATE_ID_EMAIL_ONLYBIRDNET_EMAILJS_TEMPLATE_ID_DUALBIRDNET_EMAILJS_PUBLIC_KEYBIRDNET_EMAILJS_ENDPOINTBIRDNET_EMAILJS_TIMEOUT_SECONDS
Uploader CLI Environment Variables
BIRDNET_UPLOADER_SESSION_DIRBIRDNET_UPLOADER_CACHE_DIRBIRDNET_UPLOADER_LOG_DIR
Typical Flows
Validation Flow
- Login with a valid user.
- Select an authorized project.
- Load queue items and apply filters when needed.
- Listen to selected audio and submit validation status.
- Resolve conflicts when concurrent updates occur.
- Export or inspect project validation report.
Upload Flow
- Authenticate once with a Hugging Face token.
- Create or validate a dataset repository.
- Scan the local folder recursively.
- Deduplicate against remote files.
- Upload with retries and checkpoints.
- Resume later from the saved session metadata if needed.
Repository Structure
app.py: validator app entrypointsrc/auth: authentication/session and ACL logicsrc/domain: domain modelssrc/repositories: in-memory and append-only persistencesrc/services: queue, validation, and audio fetch servicessrc/ui: Gradio interface composition and callbackssrc/uploader_cli: local uploader CLI, session persistence, deduplication, and retrying uploadstests: unit and integration test suitesdocs: collaborative workflow and QA checklists
Project Status
Validator workflow is active and under continuous development, with emphasis on reliability, auditability, and operator productivity.
The uploader CLI is being built as a companion local workflow for large ingest tasks, with a focus on resumability and validator compatibility.
FAQ
Is this app intended for production use?
Yes, for validator workflows. It is structured for multi-project access control, audit-friendly writes, and collaborative validation.
Is the uploader CLI meant to replace the validator app?
No. It is a companion tool for large dataset uploads and repository initialization.
Does the app preload all audio files?
No. Audio is fetched on demand for the selected detection, helping keep memory and network usage under control.
Can multiple validators work at the same time?
Yes. The validation flow uses optimistic concurrency checks to detect and handle conflicting updates.
Can this run on Hugging Face Spaces?
Yes. The validator app is designed for Gradio deployment on Hugging Face Spaces.
Can the uploader resume after closing the app?
Yes. It persists checkpoints and session metadata locally and can be resumed later.
Does the uploader support deduplication?
Yes. It checks the remote dataset index, caches the remote file list locally, and skips files that are already present.
Troubleshooting
The app does not start locally
- Confirm Python 3.11+ is active.
- Reinstall dependencies with
pip install -r requirements.txt. - Run
python app.pyfrom the repository root.
Port 7860 is already in use
Stop the existing process using that port, or launch after setting a different GRADIO_SERVER_PORT environment variable.
Login works but no project appears
Your user likely has no project assignment. Verify BIRDNET_PROJECTS_FILE and BIRDNET_USER_ACCESS_FILE, or add project access through the admin flow.
Seed warning appears in Validation tab
If you see a seed warning banner:
- Verify
BIRDNET_DETECTIONS_FILEpoints to an existing file. - Confirm the file is valid UTF-8 JSON.
- Confirm each project maps to a list of detections, or each list item includes
project_slug. - If needed, unset
BIRDNET_DETECTIONS_FILEto fall back to default demo detections.
Audio does not load for a detection
Check that:
audio_idexists and is valid for the selected project.- The dataset repository is reachable.
- You have permission to read the project dataset.
I get conflict messages while validating
This means another validator updated the same detection first. Refresh and reapply your decision on the newest version.
The uploader says the token cannot be saved
Make sure your OS keyring backend is available. On Windows, the default credential store should work automatically.
The uploader skips files unexpectedly
It may have found them in the remote dataset index cache. Clear the uploader cache directory if you want to force a fresh remote listing.
Resume shows an empty checkpoint
Check that you ran the uploader start command at least once and that the session directory still exists.
Recent Updates
- Integrated Validation tab flow in the multi-project app (no placeholder stage).
- Added project-scoped queue badge (
Queue: N) and improved queue context feedback. - Added seed-file warning banner with actionable remediation guidance in Validation.
- Standardized all user-facing UI text and feedback messages in English.
- Updated authentication/session timestamps to timezone-aware UTC handling.
- Added a local uploader CLI with login, repository initialization, scan, deduplication, checkpointing, and resumable upload foundations.
Related Documents
BIRDNET_UPLOADER_REDESIGN.md: uploader UX and architecture notesSPRINT_ROADMAP.md: implementation roadmapSPRINT1_KICKOFF.md: sprint kickoff details