jrrribeiro's picture
Add comprehensive usage guide for BirdNET Uploader
f961d5d
|
Raw History Blame Contribute Delete
9.13 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

BirdNET Uploader Usage Guide

This guide covers how to download, install, and use the BirdNET Uploader CLI for uploading audio datasets to Hugging Face.

Installation

Option 1: Windows Standalone (Recommended)

  1. Download the latest release from Releases
  2. Extract the ZIP file to a folder
  3. No Python installation required!
  4. Run: birdnet-uploader.exe --help

Option 2: Python + Pip

pip install birdnet-uploader
birdnet-uploader --help

Option 3: Docker

docker pull birdnet-uploader:latest
docker run --rm \
  -e HF_TOKEN="hf_your_token" \
  -v C:\segments:C:\data\segments \
  birdnet-uploader:latest \
  upload --repo-id user/dataset --segments C:\data\segments

Option 4: Build from Source

git clone https://github.com/jrrribeiro/BirdNET-Uploader-App
cd BirdNET-Uploader-App
pip install -r requirements.txt
python -m src.uploader_cli.main --help

Quick Start (5 Minutes)

Step 1: Authenticate

Store your Hugging Face token securely:

birdnet-uploader login
# Follow the prompt to enter your token
# Token is stored securely in your OS credential manager

Step 2: Create/Validate Dataset

birdnet-uploader init-repo --repo-id username/my-dataset --private
# Creates or validates the dataset exists

Step 3: Prepare Audio Files

Organize your audio files in a folder:

C:\my-segments\
β”œβ”€β”€ species-A\
β”‚   β”œβ”€β”€ recording-001.wav
β”‚   β”œβ”€β”€ recording-002.wav
β”‚   └── ...
β”œβ”€β”€ species-B\
β”‚   β”œβ”€β”€ recording-003.wav
β”‚   └── ...
└── detections.csv  (optional)

Step 4: Dry Run (Recommended)

Preview what will be uploaded without actually uploading:

birdnet-uploader upload \
  --repo-id username/my-dataset \
  --segments C:\my-segments \
  --dry-run

Step 5: Upload

birdnet-uploader upload \
  --repo-id username/my-dataset \
  --segments C:\my-segments \
  --csv detections.csv

Done! Audio files are now in audio/ and CSV is in index/detections.csv.

Common Workflows

Upload with Progress Tracking

birdnet-uploader upload \
  --repo-id username/my-dataset \
  --segments C:\my-segments \
  --workers 4

The --workers option enables parallel uploads (faster on multi-core systems).

Resume Interrupted Upload

# Check existing sessions
ls ~/.birdnet-uploader/sessions/

# Resume a specific session
birdnet-uploader resume upload-20260429T120000Z

# Or start fresh
birdnet-uploader upload --repo-id username/my-dataset --segments C:\my-segments

Upload with Custom Token (CI/CD)

birdnet-uploader upload \
  --repo-id username/my-dataset \
  --segments C:\my-segments \
  --token hf_your_token

Batch Upload Multiple Folders

$folders = @("C:\data\2024", "C:\data\2025")
foreach ($folder in $folders) {
  birdnet-uploader upload `
    --repo-id username/my-dataset `
    --segments $folder
}

Upload Only CSV (Index Only)

birdnet-uploader upload \
  --repo-id username/my-dataset \
  --segments C:\dummy-empty \
  --csv detections.csv \
  --dry-run

Token Management

Method 1: Stored Token (Recommended for Local Use)

# Store token securely (OS Credential Manager)
birdnet-uploader login

# Use stored token automatically
birdnet-uploader upload --repo-id username/my-dataset --segments C:\segments

Method 2: Environment Variable (Docker/CI)

# Linux/macOS
export HF_TOKEN="hf_..."
birdnet-uploader upload ...

# Windows PowerShell
$env:HF_TOKEN = "hf_..."
birdnet-uploader.exe upload ...

# Windows CMD
set HF_TOKEN=hf_...
birdnet-uploader.exe upload ...

Method 3: CLI Option (One-Off)

birdnet-uploader upload \
  --repo-id username/my-dataset \
  --segments C:\segments \
  --token hf_...

Advanced Usage

Parallel Uploads

Speed up uploads on multi-core systems:

birdnet-uploader upload \
  --repo-id username/my-dataset \
  --segments C:\segments \
  --workers 8

Recommended values:

  • 4-8 workers for residential/office internet
  • 16-32 workers for datacenter/gigabit internet

Custom Dataset Path

Upload to a subdirectory in the dataset:

birdnet-uploader upload \
  --repo-id username/my-dataset \
  --segments C:\segments \
  --remote-base audio/my-collection

Files uploaded to: audio/my-collection/... instead of audio/...

Session Management

# List all sessions
ls ~/.birdnet-uploader/sessions/

# View session checkpoint
birdnet-uploader resume upload-20260429T120000Z

# Remove old session (careful!)
rm -r ~/.birdnet-uploader/sessions/upload-20260101T000000Z

Clear Deduplication Cache

If you need to re-upload files that were marked as already uploaded:

# Remove cache (forces re-upload check)
rm -r ~/.birdnet-uploader/cache/

# Then retry upload
birdnet-uploader upload --repo-id username/my-dataset --segments C:\segments

Troubleshooting

"No stored token found"

Problem: You haven't logged in yet.

Solution:

birdnet-uploader login
# Enter your Hugging Face token when prompted

"Token validation failed"

Problem: Token is invalid or expired.

Solution:

  1. Go to https://huggingface.co/settings/tokens
  2. Create a new token
  3. Run birdnet-uploader login and enter the new token
  4. Or use --token hf_new_token directly

"Dataset not found"

Problem: Dataset doesn't exist or you don't have access.

Solution:

# Create the dataset
birdnet-uploader init-repo --repo-id username/my-dataset

# Or verify access
curl -H "Authorization: Bearer hf_..." \
  https://huggingface.co/api/datasets/username/my-dataset

"Upload stuck or slow"

Problem: Network issues or single-threaded upload.

Solution:

# Use parallel workers
birdnet-uploader upload \
  --repo-id username/my-dataset \
  --segments C:\segments \
  --workers 4

"DLL not found" (Windows executable)

Problem: Missing runtime dependencies in bundled version.

Solution:

  1. Download latest release (should include all DLLs)
  2. Or run from Python: python -m src.uploader_cli.main upload ...
  3. Or report issue on GitHub

Out of Disk Space

Problem: Destination dataset is full.

Solution:

  1. Check dataset size: huggingface_hub
  2. Delete old uploads if not needed
  3. Create a new dataset: birdnet-uploader init-repo --repo-id username/my-dataset-2

Dataset Structure

After upload, your Hugging Face dataset will look like:

my-dataset/
β”œβ”€β”€ audio/
β”‚   β”œβ”€β”€ species-A/
β”‚   β”‚   β”œβ”€β”€ recording-001.wav
β”‚   β”‚   β”œβ”€β”€ recording-002.wav
β”‚   β”‚   └── ...
β”‚   └── species-B/
β”‚       β”œβ”€β”€ recording-003.wav
β”‚       └── ...
β”œβ”€β”€ index/
β”‚   β”œβ”€β”€ manifest.json          # Index metadata
β”‚   β”œβ”€β”€ detections.csv         # Your CSV (if provided)
β”‚   └── shards/
β”‚       β”œβ”€β”€ detections_0.parquet
β”‚       └── detections_1.parquet
└── README.md

Performance Tips

  1. Use SSD for local segments: Faster scanning and hashing
  2. Enable parallel uploads: --workers 4 or higher
  3. Stable network: Wired internet > WiFi
  4. Large batches: Consider --batch-size 100 for many files
  5. Monitor system resources: CPU and memory usage during upload

Security

  • Never hardcode tokens: Use birdnet-uploader login or HF_TOKEN env var
  • Rotate tokens: Generate new token every 3-6 months at https://huggingface.co/settings/tokens
  • Limit token scope: Use fine-grained tokens with minimal permissions
  • Don't share tokens: Each user should have their own token
  • Use dataset privacy: Set --private when creating repos with sensitive data

Getting Help

FAQ

Q: Can I cancel an upload? A: Yes, press Ctrl+C. The session checkpoint saves progress, so you can resume later with the same command.

Q: How large can the audio collection be? A: Hugging Face datasets support large collections (100GB+). Upload speed depends on your internet.

Q: Can multiple users upload to the same dataset? A: Yes, but they need write access. Use separate tokens for each user to track uploads separately.

Q: Does it support audio formats other than WAV? A: Yes: WAV, MP3, FLAC, OGG, M4A. Extend AUDIO_EXTENSIONS in src/uploader_cli/config.py if needed.

Q: Can I upload without audio files (CSV only)? A: Yes, but you need to provide a dummy --segments directory (can be empty).

Q: How do I verify upload integrity? A: The uploader computes SHA-256 hashes and deduplicates against remote index. Files are verified on upload.