jrrribeiro's picture
Add tests for batch uploader and deduplicator logic
2b0510e
|
Raw History Blame Contribute Delete
20.1 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade
metadata
title: BirdNET-Uploader-App
emoji: 🐦
colorFrom: green
colorTo: blue
sdk: gradio
sdk_version: 5.23.1
python_version: '3.11'
app_file: app.py
pinned: false

BirdNET Uploader

A robust uploader for audio segments and detection metadata to Hugging Face datasets. Choose your upload path based on data size:

  • Web Path (~1 GB): Browser-based archive upload with live progress
  • CLI Path (unlimited): Direct folder upload for large datasets and Windows portable use

Contents

Overview

This repository is a focused tool for reliably uploading large audio collections and optional detection CSVs to Hugging Face datasets. It emphasizes:

  • Two upload paths: Small archives via web UI (≤1 GB) or direct folder uploads via CLI/Windows portable
  • Resumable sessions: Checkpoints stored locally allow recovery from interruptions
  • Fast direct uploads: The CLI uploads the chosen folder directly to the dataset without packing it into a single archive
  • Live progress: Real-time feedback during uploads
  • Optional metadata: CSV uploads remain supported, but manifest/shard generation is only for legacy workflows

Quick Start

Web UI (Browser-Based, ≤1 GB)

  1. Open BirdNET Uploader on Hugging Face Spaces
  2. Enter your Hugging Face token (from hf.co/settings/tokens)
  3. Specify your dataset repo ID (username/dataset-name)
  4. Upload a .tar.gz, .zip, or .tar archive containing audio files (optionally include a CSV)
  5. Watch live progress and get a summary when complete

Note: Archive size is limited to ~1 GB due to browser memory constraints. For larger datasets, use the Windows portable or CLI.

Windows Portable (Unlimited Size)

  1. Download birdnet-uploader-1.0.4-windows.zip (~109 MB)
  2. Extract the archive
  3. Double-click birdnet-uploader.exe to launch the web UI

Local CLI (Python 3.11+)

# Install
pip install -r requirements.txt

# Login (stores token securely)
python app.py login

# Upload local folder
python app.py upload \
  --repo-id username/dataset-name \
  --segments /path/to/audio/folder \
  --csv /path/to/detections.csv (optional) \
  --workers 4

# Check upload status
python app.py resume <session-id>

Features

Archive Upload (Web)

  • Supported formats: .tar, .tar.gz, .zip
  • Size limit: ~1 GB (due to browser constraints)
  • Automatic extraction: Archives are extracted server-side
  • Live progress: Real-time upload percentage with file counts

Large Dataset Upload (CLI)

  • Unlimited size: No practical file size limit
  • Direct folder streaming: Uploads the local folder directly to the dataset path you choose
  • Parallel workers: Optimized Hub-side transfer and retries for large uploads
  • Progress tracking: Folder-level status and local checkpoints for recovery metadata

Dataset Structure

The direct CLI path uploads your local folder structure as-is into the dataset, typically under audio/:

dataset-repo/
├── audio/              # Uploaded audio files (organized by species if CSV)
└── index/
  └── detections.csv  # Optional metadata CSV uploaded directly, if provided

Configuration

Environment Variables

# Web UI port (default: 7860)
BIRDNET_UPLOADER_PORT=8000

# CLI mode (use app.py as CLI instead of web UI)
BIRDNET_UPLOADER_CLI=true

# High-throughput Xet mode for large uploads
HF_XET_HIGH_PERFORMANCE=1

# Session storage (default: ~/.birdnet-uploader/sessions)
BIRDNET_UPLOADER_SESSION_DIR=/custom/path/sessions

# Cache directory (default: ~/.birdnet-uploader/cache)
BIRDNET_UPLOADER_CACHE_DIR=/custom/path/cache

# Hugging Face token (alternative to secure storage)
HF_TOKEN=hf_xxxxxxxxxxxx

Typical Workflows

Scenario 1: Small Archive Upload (≤1 GB)

  1. Create a .tar.gz archive of audio files with optional CSV
  2. Use web UI at Hugging Face Spaces
  3. Paste token, enter dataset ID, upload archive
  4. Done - files appear in dataset within minutes

Scenario 2: Large Archive on Windows

  1. Create a .tar.gz or .zip archive (any size)
  2. Download and extract Windows portable
  3. Double-click birdnet-uploader.exe → launches web UI on localhost:7860
  4. Same workflow as web UI, but runs locally (no size limits)

Scenario 3: Local Folder Upload (CLI)

  1. Navigate to audio folder root in terminal
  2. Run: python app.py upload --repo-id user/dataset --segments .
  3. Watch progress in real-time
  4. Resume if interrupted: python app.py resume <session-id>

CLI Reference

Login

python app.py login

Stores Hugging Face token securely in system keyring.

Upload

python app.py upload \
  --repo-id user/dataset \
  --segments /path/to/audio \
  --csv /path/to/detections.csv \      # Optional, uploaded directly as index/detections.csv
  --workers 4 \                         # Default: 4 (legacy mode only)
  --session-id my-session \             # Optional metadata checkpoint
  --remote-base my_audio_folder \       # Default: "audio"
  --upload-mode direct \                # Default: direct folder upload
  --dry-run                             # Show what would be uploaded
  --verbose                             # Detailed logging

Scan

python app.py scan --segments /path/to/audio

Shows file count, total size, and species breakdown.

Resume

python app.py resume <session-id>

Checks session status and shows last checkpoint.

Init Repo

python app.py init-repo \
  --repo-id user/dataset \
  --private                             # Default: private

Creates dataset structure on Hugging Face.

Resumable Uploads

Session checkpoints are saved locally as JSON. If an upload is interrupted:

# Check session status
python app.py resume abc-session-id-xyz

# Resume from last checkpoint (automatically skips already-uploaded files)
python app.py upload \
  --repo-id user/dataset \
  --segments ./audio \
  --session-id abc-session-id-xyz

Checkpoint location: ~/.birdnet-uploader/sessions/upload-YYYYMMDDTHHMMSSZ/checkpoint.json

Authentication

Secure Token Storage

  • First time: python app.py login stores token in system keyring (Windows, macOS, Linux)
  • Environment override: HF_TOKEN=hf_xxxx python app.py upload ...
  • Explicit flag: python app.py upload --token hf_xxxx ...

Token Permissions

Ensure your token has write permissions on the target dataset:

  1. Go to hf.co/settings/tokens
  2. Create or edit a token
  3. Set permissions: repo > write (not read)
  4. Restrict to specific repo if desired

Troubleshooting

"Upload exceeds 1 GB limit"

  • Switch to Windows portable or CLI for larger datasets
  • Compress archive more aggressively

"CSV upload failed"

  • Ensure CSV has proper headers and encoding (UTF-8)
  • File size should match audio files metadata (e.g., audio_file column)

"Session has no checkpoint"

  • Session ID may be invalid or expired
  • Check ~/.birdnet-uploader/sessions/ for available sessions
  • Restart upload without --session-id to create a new session

Network timeouts on large files

  • Use CLI with more workers: --workers 8
  • Check internet stability
  • Resume from checkpoint if interrupted

Development

Build Windows Portable

# Install build dependencies
pip install pyinstaller

# Build release
python build/release_uploader.py --version 1.0.0

# Output: build/release/birdnet-uploader-1.0.0-windows.zip

Local Testing

# Web UI
python app.py

# CLI
python app.py --help
python app.py login
python app.py scan --segments ./test_audio
python app.py upload --repo-id my-test/uploader --segments ./test_audio --dry-run

Architecture

Core Components

  • src/uploader/web_ui.py: Gradio interface for browser uploads
  • src/uploader/main.py: Click CLI for local uploads
  • src/uploader/batch_uploader.py: File upload orchestration with retries
  • src/uploader/session_manager.py: Checkpoint persistence and resumability
  • src/uploader/deduplicator.py: Remote hash checking to avoid re-uploads
  • src/uploader/scanner.py: Local folder scanning and metadata extraction
  • src/uploader/manifest.py: Dataset metadata and Parquet shard generation

Data Flow

  1. Upload: Archive extraction or folder scan
  2. Deduplication: Check remote hashes to skip existing files
  3. Upload: Push files to HF dataset in parallel
  4. Metadata: Generate and upload manifest + CSV + Parquet shards
  5. Checkpoint: Save session state for resumable recovery

Limitations

  • Web UI: ~1 GB per upload (browser constraint)
  • CSV: Must have audio_file or file column for audio file names
  • Audio formats: .wav, .mp3, .flac, .ogg, .m4a
  • Archive formats: .tar, .tar.gz, .zip

License

MIT License

  1. Download the zip from Hugging Face.
  2. Unzip into a clean folder.
  3. Run the executable without Python installed.
  4. Authenticate with a Hugging Face token.
  5. Create a session and confirm resume/recovery works.

What It Does

  • authenticates with a Hugging Face token stored in the OS keyring
  • creates or validates dataset repositories
  • scans a local audio folder recursively
  • computes per-file SHA-256 hashes
  • deduplicates against a cached remote index
  • uploads files with retries and session checkpoints
  • resumes from the last checkpoint after close or crash

Main Commands

The CLI entrypoint is:

python -m src.uploader_cli.main --help

Commands:

  • login: validate and store a Hugging Face token
  • init-repo: create and initialize a dataset repository
  • scan: scan a local folder and summarize audio files
  • start: create a local upload session checkpoint
  • resume: print the saved checkpoint for a session

Example Workflow

# Authenticate once
python -m src.uploader_cli.main login

# Create or initialize the dataset repository
python -m src.uploader_cli.main init-repo --repo-id alice/birdnet-2026 --private

# Inspect the local folder before upload
python -m src.uploader_cli.main scan --segments C:\audio\segments

# Create a resumable local session checkpoint
python -m src.uploader_cli.main start --repo-id alice/birdnet-2026 --segments C:\audio\segments


# Resume and inspect the checkpoint later
python -m src.uploader_cli.main resume upload-20260429T120000Z

Upload Engine Building Blocks

The uploader CLI is built from the following modules:

  • src/uploader_cli/auth_service.py: token validation and keyring storage
  • src/uploader_cli/repo_service.py: dataset creation and structure initialization
  • src/uploader_cli/scanner.py: recursive audio discovery and metadata collection
  • src/uploader_cli/hash_utils.py: streaming SHA-256 helpers
  • src/uploader_cli/deduplicator.py: remote index cache and skip/upload decisions
  • src/uploader_cli/session_manager.py: checkpoint and session metadata persistence
  • src/uploader_cli/batch_uploader.py: retrying upload orchestration
  • src/uploader_cli/error_handler.py: user-friendly error formatting

Local Data Layout

The uploader stores local state under the user's profile directory by default:

~/.birdnet-uploader/
  sessions/
    upload-YYYYMMDDTHHMMSSZ/
      metadata.json
      checkpoint.json
  cache/
    dedup/
      <repo-hash>.json
  logs/

Environment overrides are supported:

  • BIRDNET_UPLOADER_SESSION_DIR
  • BIRDNET_UPLOADER_CACHE_DIR
  • BIRDNET_UPLOADER_LOG_DIR

Dataset Layout Expected by the Validator

The uploader is designed to preserve the validator's expected structure:

audio/{project_slug}/...        # uploaded audio files
index/shards/...                # parquet shard files
index/manifest.json             # index manifest
validations/...                 # validation outputs
audit/ingestion-runs/...        # ingestion run history

Uploader Configuration

Runtime constants are centralized in src/uploader_cli/config.py:

  • APP_NAME
  • SCHEMA_VERSION
  • AUDIO_EXTENSIONS
  • INDEX_SHARD_SIZE
  • MAX_BATCH_SIZE
  • RETRY_MAX_ATTEMPTS
  • RETRY_INITIAL_BACKOFF_SECONDS
  • RETRY_MAX_BACKOFF_SECONDS
  • KEYRING_SERVICE
  • KEYRING_ACCOUNT

Resumability Guarantees

The uploader uses local checkpoints so it can recover from:

  • process crashes
  • app close
  • transient network failures
  • partial progress during large uploads

The checkpoint stores session metadata and cumulative counters such as:

  • uploaded count
  • skipped count
  • failed count
  • last completed file
  • last failed file
  • total uploaded bytes

Quick Start

Local Development

  1. Create and activate a Python 3.11+ virtual environment.
  2. Install dependencies:
pip install -r requirements.txt
  1. Run the validator app:
python app.py
  1. Run the uploader CLI help:
python -m src.uploader_cli.main --help

First-Time Uploader Setup

  1. Log in with a Hugging Face token:
python -m src.uploader_cli.main login
  1. Validate your target repository exists or create it:
python -m src.uploader_cli.main init-repo --repo-id alice/birdnet-2026 --private
  1. Scan your local dataset folder:
python -m src.uploader_cli.main scan --segments C:\audio\segments
  1. Start a resumable session:
python -m src.uploader_cli.main start --repo-id alice/birdnet-2026 --segments C:\audio\segments

Configuration

Validator App Environment Variables

  • BIRDNET_DETECTIONS_FILE
  • BIRDNET_VALIDATIONS_DIR
  • BIRDNET_PAGE_SIZE
  • BIRDNET_PROJECTS_FILE
  • BIRDNET_USER_ACCESS_FILE
  • BIRDNET_INVITES_FILE
  • BIRDNET_INVITE_TTL_HOURS
  • BIRDNET_BOOTSTRAP_DIR
  • BIRDNET_ENABLE_DEMO_BOOTSTRAP
  • BIRDNET_EMAILJS_ENABLED
  • BIRDNET_INVITE_EMAIL_ENABLED
  • BIRDNET_INVITE_EMAIL_SENDER
  • BIRDNET_INVITE_EMAIL_LOGIN_URL
  • BIRDNET_EMAILJS_SERVICE_ID
  • BIRDNET_EMAILJS_TEMPLATE_ID
  • BIRDNET_EMAILJS_TEMPLATE_ID_USERNAME_ONLY
  • BIRDNET_EMAILJS_TEMPLATE_ID_EMAIL_ONLY
  • BIRDNET_EMAILJS_TEMPLATE_ID_DUAL
  • BIRDNET_EMAILJS_PUBLIC_KEY
  • BIRDNET_EMAILJS_ENDPOINT
  • BIRDNET_EMAILJS_TIMEOUT_SECONDS

Uploader CLI Environment Variables

  • BIRDNET_UPLOADER_SESSION_DIR
  • BIRDNET_UPLOADER_CACHE_DIR
  • BIRDNET_UPLOADER_LOG_DIR

Typical Flows

Validation Flow

  1. Login with a valid user.
  2. Select an authorized project.
  3. Load queue items and apply filters when needed.
  4. Listen to selected audio and submit validation status.
  5. Resolve conflicts when concurrent updates occur.
  6. Export or inspect project validation report.

Upload Flow

  1. Authenticate once with a Hugging Face token.
  2. Create or validate a dataset repository.
  3. Scan the local folder recursively.
  4. Deduplicate against remote files.
  5. Upload with retries and checkpoints.
  6. Resume later from the saved session metadata if needed.

Repository Structure

  • app.py: validator app entrypoint
  • src/auth: authentication/session and ACL logic
  • src/domain: domain models
  • src/repositories: in-memory and append-only persistence
  • src/services: queue, validation, and audio fetch services
  • src/ui: Gradio interface composition and callbacks
  • src/uploader_cli: local uploader CLI, session persistence, deduplication, and retrying uploads
  • tests: unit and integration test suites
  • docs: collaborative workflow and QA checklists

Project Status

Validator workflow is active and under continuous development, with emphasis on reliability, auditability, and operator productivity.

The uploader CLI is being built as a companion local workflow for large ingest tasks, with a focus on resumability and validator compatibility.

FAQ

Is this app intended for production use?

Yes, for validator workflows. It is structured for multi-project access control, audit-friendly writes, and collaborative validation.

Is the uploader CLI meant to replace the validator app?

No. It is a companion tool for large dataset uploads and repository initialization.

Does the app preload all audio files?

No. Audio is fetched on demand for the selected detection, helping keep memory and network usage under control.

Can multiple validators work at the same time?

Yes. The validation flow uses optimistic concurrency checks to detect and handle conflicting updates.

Can this run on Hugging Face Spaces?

Yes. The validator app is designed for Gradio deployment on Hugging Face Spaces.

Can the uploader resume after closing the app?

Yes. It persists checkpoints and session metadata locally and can be resumed later.

Does the uploader support deduplication?

Yes. It checks the remote dataset index, caches the remote file list locally, and skips files that are already present.

Troubleshooting

The app does not start locally

  1. Confirm Python 3.11+ is active.
  2. Reinstall dependencies with pip install -r requirements.txt.
  3. Run python app.py from the repository root.

Port 7860 is already in use

Stop the existing process using that port, or launch after setting a different GRADIO_SERVER_PORT environment variable.

Login works but no project appears

Your user likely has no project assignment. Verify BIRDNET_PROJECTS_FILE and BIRDNET_USER_ACCESS_FILE, or add project access through the admin flow.

Seed warning appears in Validation tab

If you see a seed warning banner:

  1. Verify BIRDNET_DETECTIONS_FILE points to an existing file.
  2. Confirm the file is valid UTF-8 JSON.
  3. Confirm each project maps to a list of detections, or each list item includes project_slug.
  4. If needed, unset BIRDNET_DETECTIONS_FILE to fall back to default demo detections.

Audio does not load for a detection

Check that:

  1. audio_id exists and is valid for the selected project.
  2. The dataset repository is reachable.
  3. You have permission to read the project dataset.

I get conflict messages while validating

This means another validator updated the same detection first. Refresh and reapply your decision on the newest version.

The uploader says the token cannot be saved

Make sure your OS keyring backend is available. On Windows, the default credential store should work automatically.

The uploader skips files unexpectedly

It may have found them in the remote dataset index cache. Clear the uploader cache directory if you want to force a fresh remote listing.

Resume shows an empty checkpoint

Check that you ran the uploader start command at least once and that the session directory still exists.

Recent Updates

  • Integrated Validation tab flow in the multi-project app (no placeholder stage).
  • Added project-scoped queue badge (Queue: N) and improved queue context feedback.
  • Added seed-file warning banner with actionable remediation guidance in Validation.
  • Standardized all user-facing UI text and feedback messages in English.
  • Updated authentication/session timestamps to timezone-aware UTC handling.
  • Added a local uploader CLI with login, repository initialization, scan, deduplication, checkpointing, and resumable upload foundations.

Related Documents

  • BIRDNET_UPLOADER_REDESIGN.md: uploader UX and architecture notes
  • SPRINT_ROADMAP.md: implementation roadmap
  • SPRINT1_KICKOFF.md: sprint kickoff details