Spaces:
Sleeping
Sleeping
|
Download docs/USAGE.md from jrrribeiro/BirdNET-Uploader-App: direct link, hf CLI and curl.
- Browser
- Download file 9.13 kB
-
https://huggingface.co/spaces/jrrribeiro/BirdNET-Uploader-App/resolve/main/docs/USAGE.md
- Command line
-
hf download hf://spaces/jrrribeiro/BirdNET-Uploader-App/docs/USAGE.md
-
curl -L -o USAGE.md https://huggingface.co/spaces/jrrribeiro/BirdNET-Uploader-App/resolve/main/docs/USAGE.md
9.13 kB
| # BirdNET Uploader Usage Guide | |
| This guide covers how to download, install, and use the BirdNET Uploader CLI for uploading audio datasets to Hugging Face. | |
| ## Installation | |
| ### Option 1: Windows Standalone (Recommended) | |
| 1. Download the latest release from [Releases](https://huggingface.co/spaces/jrrribeiro/BirdNET-Uploader-App/discussions) | |
| 2. Extract the ZIP file to a folder | |
| 3. No Python installation required! | |
| 4. Run: `birdnet-uploader.exe --help` | |
| ### Option 2: Python + Pip | |
| ```bash | |
| pip install birdnet-uploader | |
| birdnet-uploader --help | |
| ``` | |
| ### Option 3: Docker | |
| ```bash | |
| docker pull birdnet-uploader:latest | |
| docker run --rm \ | |
| -e HF_TOKEN="hf_your_token" \ | |
| -v C:\segments:C:\data\segments \ | |
| birdnet-uploader:latest \ | |
| upload --repo-id user/dataset --segments C:\data\segments | |
| ``` | |
| ### Option 4: Build from Source | |
| ```bash | |
| git clone https://github.com/jrrribeiro/BirdNET-Uploader-App | |
| cd BirdNET-Uploader-App | |
| pip install -r requirements.txt | |
| python -m src.uploader_cli.main --help | |
| ``` | |
| ## Quick Start (5 Minutes) | |
| ### Step 1: Authenticate | |
| Store your Hugging Face token securely: | |
| ```bash | |
| birdnet-uploader login | |
| # Follow the prompt to enter your token | |
| # Token is stored securely in your OS credential manager | |
| ``` | |
| ### Step 2: Create/Validate Dataset | |
| ```bash | |
| birdnet-uploader init-repo --repo-id username/my-dataset --private | |
| # Creates or validates the dataset exists | |
| ``` | |
| ### Step 3: Prepare Audio Files | |
| Organize your audio files in a folder: | |
| ``` | |
| C:\my-segments\ | |
| βββ species-A\ | |
| β βββ recording-001.wav | |
| β βββ recording-002.wav | |
| β βββ ... | |
| βββ species-B\ | |
| β βββ recording-003.wav | |
| β βββ ... | |
| βββ detections.csv (optional) | |
| ``` | |
| ### Step 4: Dry Run (Recommended) | |
| Preview what will be uploaded without actually uploading: | |
| ```bash | |
| birdnet-uploader upload \ | |
| --repo-id username/my-dataset \ | |
| --segments C:\my-segments \ | |
| --dry-run | |
| ``` | |
| ### Step 5: Upload | |
| ```bash | |
| birdnet-uploader upload \ | |
| --repo-id username/my-dataset \ | |
| --segments C:\my-segments \ | |
| --csv detections.csv | |
| ``` | |
| Done! Audio files are now in `audio/` and CSV is in `index/detections.csv`. | |
| ## Common Workflows | |
| ### Upload with Progress Tracking | |
| ```bash | |
| birdnet-uploader upload \ | |
| --repo-id username/my-dataset \ | |
| --segments C:\my-segments \ | |
| --workers 4 | |
| ``` | |
| The `--workers` option enables parallel uploads (faster on multi-core systems). | |
| ### Resume Interrupted Upload | |
| ```bash | |
| # Check existing sessions | |
| ls ~/.birdnet-uploader/sessions/ | |
| # Resume a specific session | |
| birdnet-uploader resume upload-20260429T120000Z | |
| # Or start fresh | |
| birdnet-uploader upload --repo-id username/my-dataset --segments C:\my-segments | |
| ``` | |
| ### Upload with Custom Token (CI/CD) | |
| ```bash | |
| birdnet-uploader upload \ | |
| --repo-id username/my-dataset \ | |
| --segments C:\my-segments \ | |
| --token hf_your_token | |
| ``` | |
| ### Batch Upload Multiple Folders | |
| ```powershell | |
| $folders = @("C:\data\2024", "C:\data\2025") | |
| foreach ($folder in $folders) { | |
| birdnet-uploader upload ` | |
| --repo-id username/my-dataset ` | |
| --segments $folder | |
| } | |
| ``` | |
| ### Upload Only CSV (Index Only) | |
| ```bash | |
| birdnet-uploader upload \ | |
| --repo-id username/my-dataset \ | |
| --segments C:\dummy-empty \ | |
| --csv detections.csv \ | |
| --dry-run | |
| ``` | |
| ## Token Management | |
| ### Method 1: Stored Token (Recommended for Local Use) | |
| ```bash | |
| # Store token securely (OS Credential Manager) | |
| birdnet-uploader login | |
| # Use stored token automatically | |
| birdnet-uploader upload --repo-id username/my-dataset --segments C:\segments | |
| ``` | |
| ### Method 2: Environment Variable (Docker/CI) | |
| ```bash | |
| # Linux/macOS | |
| export HF_TOKEN="hf_..." | |
| birdnet-uploader upload ... | |
| # Windows PowerShell | |
| $env:HF_TOKEN = "hf_..." | |
| birdnet-uploader.exe upload ... | |
| # Windows CMD | |
| set HF_TOKEN=hf_... | |
| birdnet-uploader.exe upload ... | |
| ``` | |
| ### Method 3: CLI Option (One-Off) | |
| ```bash | |
| birdnet-uploader upload \ | |
| --repo-id username/my-dataset \ | |
| --segments C:\segments \ | |
| --token hf_... | |
| ``` | |
| ## Advanced Usage | |
| ### Parallel Uploads | |
| Speed up uploads on multi-core systems: | |
| ```bash | |
| birdnet-uploader upload \ | |
| --repo-id username/my-dataset \ | |
| --segments C:\segments \ | |
| --workers 8 | |
| ``` | |
| Recommended values: | |
| - 4-8 workers for residential/office internet | |
| - 16-32 workers for datacenter/gigabit internet | |
| ### Custom Dataset Path | |
| Upload to a subdirectory in the dataset: | |
| ```bash | |
| birdnet-uploader upload \ | |
| --repo-id username/my-dataset \ | |
| --segments C:\segments \ | |
| --remote-base audio/my-collection | |
| ``` | |
| Files uploaded to: `audio/my-collection/...` instead of `audio/...` | |
| ### Session Management | |
| ```bash | |
| # List all sessions | |
| ls ~/.birdnet-uploader/sessions/ | |
| # View session checkpoint | |
| birdnet-uploader resume upload-20260429T120000Z | |
| # Remove old session (careful!) | |
| rm -r ~/.birdnet-uploader/sessions/upload-20260101T000000Z | |
| ``` | |
| ### Clear Deduplication Cache | |
| If you need to re-upload files that were marked as already uploaded: | |
| ```bash | |
| # Remove cache (forces re-upload check) | |
| rm -r ~/.birdnet-uploader/cache/ | |
| # Then retry upload | |
| birdnet-uploader upload --repo-id username/my-dataset --segments C:\segments | |
| ``` | |
| ## Troubleshooting | |
| ### "No stored token found" | |
| **Problem**: You haven't logged in yet. | |
| **Solution**: | |
| ```bash | |
| birdnet-uploader login | |
| # Enter your Hugging Face token when prompted | |
| ``` | |
| ### "Token validation failed" | |
| **Problem**: Token is invalid or expired. | |
| **Solution**: | |
| 1. Go to https://huggingface.co/settings/tokens | |
| 2. Create a new token | |
| 3. Run `birdnet-uploader login` and enter the new token | |
| 4. Or use `--token hf_new_token` directly | |
| ### "Dataset not found" | |
| **Problem**: Dataset doesn't exist or you don't have access. | |
| **Solution**: | |
| ```bash | |
| # Create the dataset | |
| birdnet-uploader init-repo --repo-id username/my-dataset | |
| # Or verify access | |
| curl -H "Authorization: Bearer hf_..." \ | |
| https://huggingface.co/api/datasets/username/my-dataset | |
| ``` | |
| ### "Upload stuck or slow" | |
| **Problem**: Network issues or single-threaded upload. | |
| **Solution**: | |
| ```bash | |
| # Use parallel workers | |
| birdnet-uploader upload \ | |
| --repo-id username/my-dataset \ | |
| --segments C:\segments \ | |
| --workers 4 | |
| ``` | |
| ### "DLL not found" (Windows executable) | |
| **Problem**: Missing runtime dependencies in bundled version. | |
| **Solution**: | |
| 1. Download latest release (should include all DLLs) | |
| 2. Or run from Python: `python -m src.uploader_cli.main upload ...` | |
| 3. Or report issue on GitHub | |
| ### Out of Disk Space | |
| **Problem**: Destination dataset is full. | |
| **Solution**: | |
| 1. Check dataset size: `huggingface_hub` | |
| 2. Delete old uploads if not needed | |
| 3. Create a new dataset: `birdnet-uploader init-repo --repo-id username/my-dataset-2` | |
| ## Dataset Structure | |
| After upload, your Hugging Face dataset will look like: | |
| ```text | |
| my-dataset/ | |
| βββ audio/ | |
| β βββ species-A/ | |
| β β βββ recording-001.wav | |
| β β βββ recording-002.wav | |
| β β βββ ... | |
| β βββ species-B/ | |
| β βββ recording-003.wav | |
| β βββ ... | |
| βββ index/ | |
| β βββ manifest.json # Index metadata | |
| β βββ detections.csv # Your CSV (if provided) | |
| β βββ shards/ | |
| β βββ detections_0.parquet | |
| β βββ detections_1.parquet | |
| βββ README.md | |
| ``` | |
| ## Performance Tips | |
| 1. **Use SSD for local segments**: Faster scanning and hashing | |
| 2. **Enable parallel uploads**: `--workers 4` or higher | |
| 3. **Stable network**: Wired internet > WiFi | |
| 4. **Large batches**: Consider `--batch-size 100` for many files | |
| 5. **Monitor system resources**: CPU and memory usage during upload | |
| ## Security | |
| - **Never hardcode tokens**: Use `birdnet-uploader login` or `HF_TOKEN` env var | |
| - **Rotate tokens**: Generate new token every 3-6 months at https://huggingface.co/settings/tokens | |
| - **Limit token scope**: Use fine-grained tokens with minimal permissions | |
| - **Don't share tokens**: Each user should have their own token | |
| - **Use dataset privacy**: Set `--private` when creating repos with sensitive data | |
| ## Getting Help | |
| - Check [TOKEN_SECURITY.md](docs/TOKEN_SECURITY.md) for token security best practices | |
| - Check [TOKEN_USAGE_EXAMPLES.md](docs/TOKEN_USAGE_EXAMPLES.md) for Docker/CI/CD examples | |
| - Check [CI_CD_PIPELINE.md](docs/CI_CD_PIPELINE.md) for build and release info | |
| - Report issues: Create a GitHub issue with error logs and reproduction steps | |
| ## FAQ | |
| **Q: Can I cancel an upload?** | |
| A: Yes, press Ctrl+C. The session checkpoint saves progress, so you can resume later with the same command. | |
| **Q: How large can the audio collection be?** | |
| A: Hugging Face datasets support large collections (100GB+). Upload speed depends on your internet. | |
| **Q: Can multiple users upload to the same dataset?** | |
| A: Yes, but they need write access. Use separate tokens for each user to track uploads separately. | |
| **Q: Does it support audio formats other than WAV?** | |
| A: Yes: WAV, MP3, FLAC, OGG, M4A. Extend `AUDIO_EXTENSIONS` in `src/uploader_cli/config.py` if needed. | |
| **Q: Can I upload without audio files (CSV only)?** | |
| A: Yes, but you need to provide a dummy `--segments` directory (can be empty). | |
| **Q: How do I verify upload integrity?** | |
| A: The uploader computes SHA-256 hashes and deduplicates against remote index. Files are verified on upload. | |