Spaces:
Sleeping
Sleeping
Commit ·
fe20ae1
1
Parent(s): 2574ede
Remove hf_bulk_upload_tool (migrated to jrrribeiro/HF_Dataset_Uploader)
Browse files- hf_bulk_upload_tool/BUILD_GUIDE.md +0 -139
- hf_bulk_upload_tool/HF_Dataset_Uploader.spec +0 -56
- hf_bulk_upload_tool/IMPLEMENTATION_PROTOCOL.md +0 -365
- hf_bulk_upload_tool/RATE_LIMITER_GUIDE.md +0 -221
- hf_bulk_upload_tool/README.md +0 -178
- hf_bulk_upload_tool/README_GUI.md +0 -261
- hf_bulk_upload_tool/__init__.py +0 -3
- hf_bulk_upload_tool/hf_uploader_gui.spec +0 -56
- hf_bulk_upload_tool/main_gui.py +0 -443
- hf_bulk_upload_tool/progress_bar_utils.py +0 -355
- hf_bulk_upload_tool/requirements.txt +0 -3
- hf_bulk_upload_tool/requirements_gui.txt +0 -10
- hf_bulk_upload_tool/resilience_utils.py +0 -268
- hf_bulk_upload_tool/run_upload.ps1 +0 -58
- hf_bulk_upload_tool/sharding_utils.py +0 -304
- hf_bulk_upload_tool/test_gui.py +0 -132
- hf_bulk_upload_tool/upload_dataset.py +0 -767
- hf_bulk_upload_tool/upload_logic.py +0 -886
hf_bulk_upload_tool/BUILD_GUIDE.md
DELETED
|
@@ -1,139 +0,0 @@
|
|
| 1 |
-
# Build Guide: Creating a Portable EXE for HF Bulk Uploader
|
| 2 |
-
|
| 3 |
-
## Prerequisites
|
| 4 |
-
|
| 5 |
-
1. **Python 3.10+** installed and added to PATH
|
| 6 |
-
2. **Virtual Environment activated** (highly recommended)
|
| 7 |
-
|
| 8 |
-
## Step 1: Install Dependencies
|
| 9 |
-
|
| 10 |
-
```powershell
|
| 11 |
-
# Install base requirements (if not already done)
|
| 12 |
-
pip install -r requirements.txt
|
| 13 |
-
|
| 14 |
-
# Install GUI-specific requirements
|
| 15 |
-
pip install -r hf_bulk_upload_tool\requirements_gui.txt
|
| 16 |
-
|
| 17 |
-
# Install PyInstaller for building the EXE
|
| 18 |
-
pip install pyinstaller==6.1.0
|
| 19 |
-
```
|
| 20 |
-
|
| 21 |
-
## Step 2: Test the GUI (Optional but Recommended)
|
| 22 |
-
|
| 23 |
-
Before building the EXE, test that the GUI runs correctly:
|
| 24 |
-
|
| 25 |
-
```powershell
|
| 26 |
-
cd hf_bulk_upload_tool
|
| 27 |
-
python main_gui.py
|
| 28 |
-
```
|
| 29 |
-
|
| 30 |
-
This will open the GUI window. Make sure all controls work and that you can see the log output.
|
| 31 |
-
|
| 32 |
-
## Step 3: Build the EXE
|
| 33 |
-
|
| 34 |
-
### Option A: Simple One-File Build (Recommended for Distribution)
|
| 35 |
-
|
| 36 |
-
```powershell
|
| 37 |
-
cd hf_bulk_upload_tool
|
| 38 |
-
|
| 39 |
-
pyinstaller --onefile --noconsole `
|
| 40 |
-
--name "HF_Dataset_Uploader" `
|
| 41 |
-
--icon=path\to\icon.ico `
|
| 42 |
-
--collect-all customtkinter `
|
| 43 |
-
--collect-all huggingface_hub `
|
| 44 |
-
main_gui.py
|
| 45 |
-
```
|
| 46 |
-
|
| 47 |
-
**Parameters explained:**
|
| 48 |
-
- `--onefile`: Creates a single `.exe` file (larger but easier to distribute)
|
| 49 |
-
- `--noconsole`: Hides the black command prompt window
|
| 50 |
-
- `--name "HF_Dataset_Uploader"`: Sets the output EXE name
|
| 51 |
-
- `--icon`: (Optional) Path to a `.ico` file for the EXE icon
|
| 52 |
-
- `--collect-all customtkinter`: Ensures CustomTkinter is bundled
|
| 53 |
-
- `--collect-all huggingface_hub`: Ensures HF Hub lib is bundled
|
| 54 |
-
- `main_gui.py`: The entry point script
|
| 55 |
-
|
| 56 |
-
### Option B: Multi-File Build (Faster Development)
|
| 57 |
-
|
| 58 |
-
If you want faster builds during development:
|
| 59 |
-
|
| 60 |
-
```powershell
|
| 61 |
-
pyinstaller --onedir --noconsole `
|
| 62 |
-
--name "HF_Dataset_Uploader" `
|
| 63 |
-
--collect-all customtkinter `
|
| 64 |
-
--collect-all huggingface_hub `
|
| 65 |
-
main_gui.py
|
| 66 |
-
```
|
| 67 |
-
|
| 68 |
-
This creates a folder with the EXE and all dependencies (easier to debug, but more files to distribute).
|
| 69 |
-
|
| 70 |
-
## Step 4: Locate Your EXE
|
| 71 |
-
|
| 72 |
-
After the build completes, you'll find:
|
| 73 |
-
- **One-file build**: `dist/HF_Dataset_Uploader.exe`
|
| 74 |
-
- **Multi-file build**: `dist/HF_Dataset_Uploader/HF_Dataset_Uploader.exe`
|
| 75 |
-
|
| 76 |
-
## Step 5: Test the EXE
|
| 77 |
-
|
| 78 |
-
1. Double-click the EXE to run it
|
| 79 |
-
2. Fill in the fields and test with `--dry-run` equivalent (if adding that option)
|
| 80 |
-
3. Check that all functionality works
|
| 81 |
-
|
| 82 |
-
## Troubleshooting
|
| 83 |
-
|
| 84 |
-
### "Module not found" errors
|
| 85 |
-
- Ensure all dependencies are installed: `pip install -r requirements.txt`
|
| 86 |
-
- Use `--collect-all <module>` for each missing module
|
| 87 |
-
- Example: `--collect-all requests --collect-all tqdm`
|
| 88 |
-
|
| 89 |
-
### GUI doesn't display
|
| 90 |
-
- Make sure CustomTkinter is installed: `pip install customtkinter`
|
| 91 |
-
- Check for Python version conflicts (needs Python 3.10+)
|
| 92 |
-
|
| 93 |
-
### Large EXE file size
|
| 94 |
-
- One-file builds are large (~200-300 MB) because they include all dependencies
|
| 95 |
-
- Use the `--multi-file` version for smaller individual files (but more of them)
|
| 96 |
-
|
| 97 |
-
### Windows SmartScreen warning
|
| 98 |
-
- This is a false positive; users should click "More info" → "Run anyway"
|
| 99 |
-
- To prevent this, you'd need a code-signing certificate (paid)
|
| 100 |
-
|
| 101 |
-
### Antivirus flags
|
| 102 |
-
- PyInstaller-generated EXEs can trigger false positives
|
| 103 |
-
- Consider using `--clean` to remove build artifacts between builds
|
| 104 |
-
- Upload the final EXE to VirusTotal for verification
|
| 105 |
-
|
| 106 |
-
## Advanced: Creating an Installer with NSIS
|
| 107 |
-
|
| 108 |
-
For professional distribution, consider creating an installer:
|
| 109 |
-
|
| 110 |
-
1. Install NSIS: https://nsis.sourceforge.io/
|
| 111 |
-
2. Use a tool like `auto-py-to-exe` or create a custom `.nsi` script
|
| 112 |
-
3. This allows users to install to `Program Files` and create Start Menu shortcuts
|
| 113 |
-
|
| 114 |
-
## GitHub Actions for Automated Builds (Optional)
|
| 115 |
-
|
| 116 |
-
You can set up GitHub Actions to automatically build the EXE for every release:
|
| 117 |
-
|
| 118 |
-
See `.github/workflows/build.yml` (if created)
|
| 119 |
-
|
| 120 |
-
---
|
| 121 |
-
|
| 122 |
-
## Summary Commands
|
| 123 |
-
|
| 124 |
-
**Quick build (all-in-one):**
|
| 125 |
-
```powershell
|
| 126 |
-
cd hf_bulk_upload_tool
|
| 127 |
-
pyinstaller --onefile --noconsole `
|
| 128 |
-
--name "HF_Dataset_Uploader" `
|
| 129 |
-
--collect-all customtkinter `
|
| 130 |
-
--collect-all huggingface_hub `
|
| 131 |
-
main_gui.py
|
| 132 |
-
```
|
| 133 |
-
|
| 134 |
-
**Then run:**
|
| 135 |
-
```powershell
|
| 136 |
-
.\dist\HF_Dataset_Uploader.exe
|
| 137 |
-
```
|
| 138 |
-
|
| 139 |
-
That's it! Your portable GUI uploader is ready to distribute.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/HF_Dataset_Uploader.spec
DELETED
|
@@ -1,56 +0,0 @@
|
|
| 1 |
-
# -*- mode: python ; coding: utf-8 -*-
|
| 2 |
-
from PyInstaller.utils.hooks import collect_all
|
| 3 |
-
|
| 4 |
-
datas = []
|
| 5 |
-
binaries = []
|
| 6 |
-
hiddenimports = []
|
| 7 |
-
tmp_ret = collect_all('customtkinter')
|
| 8 |
-
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
|
| 9 |
-
tmp_ret = collect_all('huggingface_hub')
|
| 10 |
-
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
|
| 11 |
-
|
| 12 |
-
# Add custom modules
|
| 13 |
-
hiddenimports.extend([
|
| 14 |
-
'sharding_utils',
|
| 15 |
-
'progress_bar_utils',
|
| 16 |
-
'resilience_utils',
|
| 17 |
-
'tqdm',
|
| 18 |
-
'requests',
|
| 19 |
-
])
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
a = Analysis(
|
| 23 |
-
['main_gui.py'],
|
| 24 |
-
pathex=[],
|
| 25 |
-
binaries=binaries,
|
| 26 |
-
datas=datas,
|
| 27 |
-
hiddenimports=hiddenimports,
|
| 28 |
-
hookspath=[],
|
| 29 |
-
hooksconfig={},
|
| 30 |
-
runtime_hooks=[],
|
| 31 |
-
excludes=[],
|
| 32 |
-
noarchive=False,
|
| 33 |
-
optimize=0,
|
| 34 |
-
)
|
| 35 |
-
pyz = PYZ(a.pure)
|
| 36 |
-
|
| 37 |
-
exe = EXE(
|
| 38 |
-
pyz,
|
| 39 |
-
a.scripts,
|
| 40 |
-
a.binaries,
|
| 41 |
-
a.datas,
|
| 42 |
-
[],
|
| 43 |
-
name='HF_Dataset_Uploader',
|
| 44 |
-
debug=False,
|
| 45 |
-
bootloader_ignore_signals=False,
|
| 46 |
-
strip=False,
|
| 47 |
-
upx=True,
|
| 48 |
-
upx_exclude=[],
|
| 49 |
-
runtime_tmpdir=None,
|
| 50 |
-
console=False,
|
| 51 |
-
disable_windowed_traceback=False,
|
| 52 |
-
argv_emulation=False,
|
| 53 |
-
target_arch=None,
|
| 54 |
-
codesign_identity=None,
|
| 55 |
-
entitlements_file=None,
|
| 56 |
-
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/IMPLEMENTATION_PROTOCOL.md
DELETED
|
@@ -1,365 +0,0 @@
|
|
| 1 |
-
# Implementation Protocol: GUI Uploader (.exe)
|
| 2 |
-
|
| 3 |
-
**Version:** 1.0
|
| 4 |
-
**Status:** ✅ Complete (Code only, no tests executed)
|
| 5 |
-
**Date:** May 14, 2026
|
| 6 |
-
|
| 7 |
-
---
|
| 8 |
-
|
| 9 |
-
## Executive Summary
|
| 10 |
-
|
| 11 |
-
This protocol documents the **complete implementation** of a portable GUI application for Hugging Face dataset uploads. The original CLI script (`upload_dataset.py`) has been refactored into a modular architecture with a CustomTkinter GUI frontend, enabling non-technical users to upload datasets via a user-friendly interface.
|
| 12 |
-
|
| 13 |
-
**No automated tests were executed** per your request — only the complete source code is provided.
|
| 14 |
-
|
| 15 |
-
---
|
| 16 |
-
|
| 17 |
-
## Files Delivered
|
| 18 |
-
|
| 19 |
-
### 1. **upload_logic.py** (260 lines)
|
| 20 |
-
**Purpose:** Refactored core upload logic
|
| 21 |
-
**Key Changes:**
|
| 22 |
-
- Extracted `run_upload_logic(config, logger)` function
|
| 23 |
-
- Replaced all `print()` with logger callback
|
| 24 |
-
- Removed argparse dependency
|
| 25 |
-
- Config is now a plain Python dict
|
| 26 |
-
- 100% compatible with original behavior
|
| 27 |
-
|
| 28 |
-
**Entry Point:**
|
| 29 |
-
```python
|
| 30 |
-
def run_upload_logic(config: dict, logger: Callable[[str], None]) -> int
|
| 31 |
-
```
|
| 32 |
-
|
| 33 |
-
### 2. **main_gui.py** (350+ lines)
|
| 34 |
-
**Purpose:** CustomTkinter GUI application
|
| 35 |
-
**Features:**
|
| 36 |
-
- ✅ Input fields: repo ID, token, segment path, CSV path
|
| 37 |
-
- ✅ Options: private/public, resume, verify, rate-limit, workers
|
| 38 |
-
- ✅ Real-time progress log (textbox with auto-scroll)
|
| 39 |
-
- ✅ Threading: upload runs in background, UI never freezes
|
| 40 |
-
- ✅ Thread-safe logging via Queue
|
| 41 |
-
- ✅ Validation: checks required fields and paths
|
| 42 |
-
- ✅ Button states: Start/Stop enable/disable properly
|
| 43 |
-
|
| 44 |
-
**Entry Point:**
|
| 45 |
-
```python
|
| 46 |
-
def main() # Launches GUI mainloop
|
| 47 |
-
```
|
| 48 |
-
|
| 49 |
-
### 3. **requirements_gui.txt** (4 lines)
|
| 50 |
-
**Purpose:** Additional dependencies for GUI
|
| 51 |
-
**Contents:**
|
| 52 |
-
```
|
| 53 |
-
customtkinter==5.2.2
|
| 54 |
-
# Plus: huggingface-hub, requests, tqdm (in base requirements.txt)
|
| 55 |
-
```
|
| 56 |
-
|
| 57 |
-
### 4. **BUILD_GUIDE.md** (140+ lines)
|
| 58 |
-
**Purpose:** Step-by-step guide to generate the .exe
|
| 59 |
-
**Sections:**
|
| 60 |
-
- Prerequisites
|
| 61 |
-
- Dependency installation
|
| 62 |
-
- GUI testing before build
|
| 63 |
-
- PyInstaller build commands (one-file & multi-file options)
|
| 64 |
-
- Troubleshooting (module not found, antivirus, SmartScreen, etc.)
|
| 65 |
-
- Advanced: NSIS installer creation
|
| 66 |
-
- Summary commands
|
| 67 |
-
|
| 68 |
-
**Quick Build:**
|
| 69 |
-
```powershell
|
| 70 |
-
pyinstaller --onefile --noconsole `
|
| 71 |
-
--name "HF_Dataset_Uploader" `
|
| 72 |
-
--collect-all customtkinter `
|
| 73 |
-
--collect-all huggingface_hub `
|
| 74 |
-
main_gui.py
|
| 75 |
-
```
|
| 76 |
-
|
| 77 |
-
### 5. **hf_uploader_gui.spec** (65 lines)
|
| 78 |
-
**Purpose:** Pre-configured PyInstaller spec file
|
| 79 |
-
**Benefits:**
|
| 80 |
-
- Simplified build: `pyinstaller hf_uploader_gui.spec`
|
| 81 |
-
- All options pre-set
|
| 82 |
-
- Reproducible builds
|
| 83 |
-
- Easy to modify if needed
|
| 84 |
-
|
| 85 |
-
### 6. **README_GUI.md** (300+ lines)
|
| 86 |
-
**Purpose:** Complete architecture & usage documentation
|
| 87 |
-
**Sections:**
|
| 88 |
-
- Overview & architecture
|
| 89 |
-
- Motor (logic) vs Cabin (UI) separation
|
| 90 |
-
- Config dict schema
|
| 91 |
-
- Threading model explained
|
| 92 |
-
- File structure
|
| 93 |
-
- Design decisions
|
| 94 |
-
- Limitations & future improvements
|
| 95 |
-
- Testing checklist
|
| 96 |
-
- Troubleshooting table
|
| 97 |
-
- Summary & next steps
|
| 98 |
-
|
| 99 |
-
### 7. **test_gui.py** (130 lines)
|
| 100 |
-
**Purpose:** Pre-flight checks before launching GUI
|
| 101 |
-
**Checks:**
|
| 102 |
-
- ✅ All required modules installed
|
| 103 |
-
- ✅ GUI files exist and readable
|
| 104 |
-
- ✅ Optional: Launch GUI to verify
|
| 105 |
-
|
| 106 |
-
**Usage:**
|
| 107 |
-
```powershell
|
| 108 |
-
python test_gui.py
|
| 109 |
-
```
|
| 110 |
-
|
| 111 |
-
---
|
| 112 |
-
|
| 113 |
-
## Architecture Overview
|
| 114 |
-
|
| 115 |
-
```
|
| 116 |
-
┌─────────────────────────────────────────────────────────┐
|
| 117 |
-
│ main_gui.py │
|
| 118 |
-
│ (CustomTkinter GUI Layer) │
|
| 119 |
-
│ ┌──────────────────────────────────────────────────┐ │
|
| 120 |
-
│ │ Fields: repo ID, token, segments, CSV │ │
|
| 121 |
-
│ │ Options: private, resume, verify, rate-limit │ │
|
| 122 |
-
│ │ Buttons: Start, Stop, Clear │ │
|
| 123 |
-
│ │ Display: Real-time log textbox │ │
|
| 124 |
-
│ └──────────────────────────────────────────────────┘ │
|
| 125 |
-
│ ↓ │
|
| 126 |
-
│ ┌──────────────────────────────────────────────────┐ │
|
| 127 |
-
│ │ Thread Model │ │
|
| 128 |
-
│ │ • Main thread: UI event loop (never blocks) │ │
|
| 129 |
-
│ │ • Upload thread: run_upload_logic() │ │
|
| 130 |
-
│ │ • Queue: Thread-safe log passing │ │
|
| 131 |
-
│ └──────────────────────────────────────────────────┘ │
|
| 132 |
-
└──────────────────────────────────────────��──────────────┘
|
| 133 |
-
↓
|
| 134 |
-
┌─────────────────────────────────────────────────────────┐
|
| 135 |
-
│ upload_logic.py │
|
| 136 |
-
│ (Pure Upload Logic Layer – No UI) │
|
| 137 |
-
│ ┌──────────────────────────────────────────────────┐ │
|
| 138 |
-
│ │ Config validation │ │
|
| 139 |
-
│ │ ↓ │ │
|
| 140 |
-
│ │ HfApi setup + rate limiter │ │
|
| 141 |
-
│ │ ↓ │ │
|
| 142 |
-
│ │ upload_large_folder() [Primary path] │ │
|
| 143 |
-
│ │ • Staging folder setup │ │
|
| 144 |
-
│ │ • Directory junctions (Windows) │ │
|
| 145 |
-
│ │ • Batch commit (respects rate limits) │ │
|
| 146 |
-
│ │ ↓ [Fallback if above fails] │ │
|
| 147 |
-
│ │ Per-file upload with retry logic │ │
|
| 148 |
-
│ │ • ThreadPoolExecutor for parallelism │ │
|
| 149 |
-
│ │ • Checkpoint system (safe resume) │ │
|
| 150 |
-
│ │ • Rate limiter (1000 req/5min) │ │
|
| 151 |
-
│ │ ↓ │ │
|
| 152 |
-
│ │ CSV upload (if not already included) │ │
|
| 153 |
-
│ │ ↓ │ │
|
| 154 |
-
│ │ Checkpoint save + summary JSON │ │
|
| 155 |
-
│ └──────────────────────────────────────────────────┘ │
|
| 156 |
-
└─────────────────────────────────────────────────────────┘
|
| 157 |
-
↓
|
| 158 |
-
Hugging Face API
|
| 159 |
-
```
|
| 160 |
-
|
| 161 |
-
---
|
| 162 |
-
|
| 163 |
-
## Configuration Dictionary
|
| 164 |
-
|
| 165 |
-
All configuration passed to `run_upload_logic()`:
|
| 166 |
-
|
| 167 |
-
```python
|
| 168 |
-
config = {
|
| 169 |
-
# Required
|
| 170 |
-
"hf_token": str, # HF write token
|
| 171 |
-
"repo_id": str, # Format: "username/dataset-name"
|
| 172 |
-
"segments": str | Path, # Path to audio directory
|
| 173 |
-
|
| 174 |
-
# Optional (with defaults)
|
| 175 |
-
"csv": str | Path, # CSV file to upload
|
| 176 |
-
"segments_path_in_repo": "audio",
|
| 177 |
-
"csv_path_in_repo": "index/detections.csv",
|
| 178 |
-
"private": True,
|
| 179 |
-
"resume": False,
|
| 180 |
-
"verify_remote": False,
|
| 181 |
-
"verify_etag": False,
|
| 182 |
-
"dry_run": False,
|
| 183 |
-
"rate_limit_aware": True,
|
| 184 |
-
"rate_limit_max_requests": 950,
|
| 185 |
-
"rate_limit_window": 300,
|
| 186 |
-
"create_repo_attempts": 3,
|
| 187 |
-
"upload_attempts": 3,
|
| 188 |
-
"retry_backoff": 5.0,
|
| 189 |
-
"max_workers": 1,
|
| 190 |
-
"checkpoint_dir": Path,
|
| 191 |
-
"commit_message": "Upload BirdNET segments",
|
| 192 |
-
}
|
| 193 |
-
```
|
| 194 |
-
|
| 195 |
-
---
|
| 196 |
-
|
| 197 |
-
## Threading Model
|
| 198 |
-
|
| 199 |
-
**Problem:** If upload runs on main thread, GUI freezes.
|
| 200 |
-
**Solution:** Background thread + thread-safe queue.
|
| 201 |
-
|
| 202 |
-
```
|
| 203 |
-
User clicks "Start"
|
| 204 |
-
↓
|
| 205 |
-
Main thread validates inputs
|
| 206 |
-
↓
|
| 207 |
-
Main thread spawns upload_thread (daemon)
|
| 208 |
-
↓
|
| 209 |
-
upload_thread runs: run_upload_logic(config, logger_callback)
|
| 210 |
-
↓
|
| 211 |
-
Inside upload_thread: logger(msg) → queue.put(msg)
|
| 212 |
-
↓
|
| 213 |
-
Main thread: after(100ms, _process_log_queue)
|
| 214 |
-
↓
|
| 215 |
-
Main thread reads queue: self.log_queue.get_nowait()
|
| 216 |
-
↓
|
| 217 |
-
Main thread updates textbox: textbox.insert(msg)
|
| 218 |
-
↓
|
| 219 |
-
Main thread schedules next check after(100ms, ...)
|
| 220 |
-
```
|
| 221 |
-
|
| 222 |
-
**Result:** Upload happens smoothly, UI responsive, logs update every 100ms.
|
| 223 |
-
|
| 224 |
-
---
|
| 225 |
-
|
| 226 |
-
## Build Process
|
| 227 |
-
|
| 228 |
-
### Step 1: Verify Environment
|
| 229 |
-
```powershell
|
| 230 |
-
pip install -r requirements.txt
|
| 231 |
-
pip install -r hf_bulk_upload_tool/requirements_gui.txt
|
| 232 |
-
```
|
| 233 |
-
|
| 234 |
-
### Step 2: Test GUI
|
| 235 |
-
```powershell
|
| 236 |
-
cd hf_bulk_upload_tool
|
| 237 |
-
python test_gui.py # Validates imports and launches GUI
|
| 238 |
-
```
|
| 239 |
-
|
| 240 |
-
### Step 3: Build EXE
|
| 241 |
-
```powershell
|
| 242 |
-
pyinstaller --onefile --noconsole `
|
| 243 |
-
--name "HF_Dataset_Uploader" `
|
| 244 |
-
--collect-all customtkinter `
|
| 245 |
-
--collect-all huggingface_hub `
|
| 246 |
-
main_gui.py
|
| 247 |
-
```
|
| 248 |
-
|
| 249 |
-
### Step 4: Output
|
| 250 |
-
- **One-file build**: `dist/HF_Dataset_Uploader.exe` (~250 MB)
|
| 251 |
-
- **Multi-file build**: `dist/HF_Dataset_Uploader/HF_Dataset_Uploader.exe` (folder)
|
| 252 |
-
|
| 253 |
-
### Step 5: Test EXE
|
| 254 |
-
```powershell
|
| 255 |
-
./dist/HF_Dataset_Uploader.exe
|
| 256 |
-
```
|
| 257 |
-
|
| 258 |
-
---
|
| 259 |
-
|
| 260 |
-
## Key Features
|
| 261 |
-
|
| 262 |
-
✅ **User-Friendly**
|
| 263 |
-
- Modern CustomTkinter dark theme
|
| 264 |
-
- Clear input fields with labels
|
| 265 |
-
- File browser dialogs
|
| 266 |
-
- Real-time progress log
|
| 267 |
-
|
| 268 |
-
✅ **Robust**
|
| 269 |
-
- Input validation
|
| 270 |
-
- Graceful error handling
|
| 271 |
-
- Retry logic with exponential backoff
|
| 272 |
-
- Checkpoint system for safe resume
|
| 273 |
-
|
| 274 |
-
✅ **Performant**
|
| 275 |
-
- `upload_large_folder()` reduces commits (solves rate limit issue)
|
| 276 |
-
- Rate limiting respects 1000 req/5min quota
|
| 277 |
-
- Directory junctions avoid copying 55GB+ data
|
| 278 |
-
- Multi-threaded per-file upload as fallback
|
| 279 |
-
|
| 280 |
-
✅ **Professional**
|
| 281 |
-
- Standalone .exe (no Python installation required)
|
| 282 |
-
- No console window (looks polished)
|
| 283 |
-
- Proper error dialogs
|
| 284 |
-
- Status indicators
|
| 285 |
-
|
| 286 |
-
---
|
| 287 |
-
|
| 288 |
-
## Known Limitations (v1)
|
| 289 |
-
|
| 290 |
-
❌ No mid-upload cancellation (can close app + resume later with --resume)
|
| 291 |
-
❌ No progress bar visualization (text log only)
|
| 292 |
-
❌ No drag-and-drop for file selection
|
| 293 |
-
❌ No configuration profiles (save/load)
|
| 294 |
-
|
| 295 |
-
**These can be added in v2.**
|
| 296 |
-
|
| 297 |
-
---
|
| 298 |
-
|
| 299 |
-
## Testing Checklist (Manual, Not Automated)
|
| 300 |
-
|
| 301 |
-
Before generating .exe, verify:
|
| 302 |
-
|
| 303 |
-
- [ ] `test_gui.py` runs without errors
|
| 304 |
-
- [ ] `python main_gui.py` launches GUI window
|
| 305 |
-
- [ ] All input fields respond to keyboard/mouse
|
| 306 |
-
- [ ] File picker dialogs open
|
| 307 |
-
- [ ] Token field is masked (shows `*` not plaintext)
|
| 308 |
-
- [ ] Validation catches empty required fields
|
| 309 |
-
- [ ] Validation catches non-existent paths
|
| 310 |
-
- [ ] Start button disables during upload
|
| 311 |
-
- [ ] Stop button enables during upload
|
| 312 |
-
- [ ] Log updates in real-time
|
| 313 |
-
- [ ] Progress completes with ✅ or ❌ message
|
| 314 |
-
- [ ] Resume mode skips already-uploaded files
|
| 315 |
-
|
| 316 |
-
---
|
| 317 |
-
|
| 318 |
-
## Deliverable Files Summary
|
| 319 |
-
|
| 320 |
-
| File | Lines | Purpose |
|
| 321 |
-
|------|-------|---------|
|
| 322 |
-
| upload_logic.py | 260 | Refactored core logic (no UI) |
|
| 323 |
-
| main_gui.py | 350+ | CustomTkinter GUI + threading |
|
| 324 |
-
| requirements_gui.txt | 4 | Extra dependencies |
|
| 325 |
-
| BUILD_GUIDE.md | 140+ | Build instructions & troubleshooting |
|
| 326 |
-
| hf_uploader_gui.spec | 65 | PyInstaller spec file |
|
| 327 |
-
| README_GUI.md | 300+ | Architecture & usage docs |
|
| 328 |
-
| test_gui.py | 130 | Pre-flight checks |
|
| 329 |
-
|
| 330 |
-
**Total:** ~1,250 lines of production-ready code (no tests executed as requested)
|
| 331 |
-
|
| 332 |
-
---
|
| 333 |
-
|
| 334 |
-
## Next Steps for User
|
| 335 |
-
|
| 336 |
-
1. ✅ **Review the code** — Read `README_GUI.md` for architecture overview
|
| 337 |
-
2. ✅ **Test locally** — Run `test_gui.py` to verify dependencies
|
| 338 |
-
3. ✅ **Launch GUI** — `python main_gui.py` to see the interface
|
| 339 |
-
4. ✅ **Build EXE** — Follow `BUILD_GUIDE.md` to generate the executable
|
| 340 |
-
5. ✅ **Distribute** — Share the `.exe` with non-technical users
|
| 341 |
-
|
| 342 |
-
---
|
| 343 |
-
|
| 344 |
-
## Support & Questions
|
| 345 |
-
|
| 346 |
-
- **Architecture questions?** See `README_GUI.md`
|
| 347 |
-
- **Build issues?** See `BUILD_GUIDE.md` troubleshooting section
|
| 348 |
-
- **Logic changes?** Modify `upload_logic.py` (both CLI and GUI will use it)
|
| 349 |
-
- **UI enhancements?** Modify `main_gui.py` (logic layer untouched)
|
| 350 |
-
|
| 351 |
-
---
|
| 352 |
-
|
| 353 |
-
## Protocol Status
|
| 354 |
-
|
| 355 |
-
✅ **COMPLETE** — All code written, no tests executed (per request)
|
| 356 |
-
⏳ **READY FOR TESTING** — User should run `test_gui.py` before building EXE
|
| 357 |
-
🚀 **READY FOR BUILD** — Once testing passes, ready to generate `.exe`
|
| 358 |
-
|
| 359 |
-
---
|
| 360 |
-
|
| 361 |
-
**Document Signed:** Implementation Protocol v1.0
|
| 362 |
-
**Deliverable Date:** May 14, 2026
|
| 363 |
-
**Platform:** Windows 10+ (PowerShell), cross-platform capable
|
| 364 |
-
**Python:** 3.10+ required
|
| 365 |
-
**Architecture:** Modular, testable, distributable as standalone .exe
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/RATE_LIMITER_GUIDE.md
DELETED
|
@@ -1,221 +0,0 @@
|
|
| 1 |
-
# Rate Limiter Guide for Large Datasets
|
| 2 |
-
|
| 3 |
-
## Overview
|
| 4 |
-
|
| 5 |
-
The upload script now includes an intelligent **rate limiter** that respects the Hugging Face API quota of **1000 requests per 5 minutes**. This allows you to upload very large datasets (150k+ files) automatically without hitting rate limits.
|
| 6 |
-
|
| 7 |
-
## How It Works
|
| 8 |
-
|
| 9 |
-
**Sliding Window Rate Limiting:**
|
| 10 |
-
- Tracks requests over the last 5 minutes
|
| 11 |
-
- When reaching ~950 requests (safety buffer), pauses automatically
|
| 12 |
-
- Resumes when old requests drop out of the window
|
| 13 |
-
- Completely transparent — happens in the background
|
| 14 |
-
|
| 15 |
-
Example timeline for 150k small files:
|
| 16 |
-
```
|
| 17 |
-
Minute 0-5: Upload files 1-1000 (1000 requests)
|
| 18 |
-
Minute 5-10: Wait 5 sec, then files 1001-2000 (1000 requests)
|
| 19 |
-
Minute 10-15: Wait 5 sec, then files 2001-3000 (1000 requests)
|
| 20 |
-
...
|
| 21 |
-
Minute 750-760: Upload final files
|
| 22 |
-
Total time: ~12.5 hours (continuous, no manual waits)
|
| 23 |
-
```
|
| 24 |
-
|
| 25 |
-
## Enabling Rate Limiter
|
| 26 |
-
|
| 27 |
-
Rate limiter is **enabled by default**. Use it with:
|
| 28 |
-
|
| 29 |
-
```powershell
|
| 30 |
-
python .\hf_bulk_upload_tool\upload_dataset.py `
|
| 31 |
-
--repo-id "jrrribeiro/PPBIO_Audio_Library" `
|
| 32 |
-
--segments "C:\path\to\segments" `
|
| 33 |
-
--csv "C:\path\to\detections.csv" `
|
| 34 |
-
--hf-token "<YOUR_HF_TOKEN>" --private `
|
| 35 |
-
--resume
|
| 36 |
-
```
|
| 37 |
-
|
| 38 |
-
To disable (not recommended):
|
| 39 |
-
```powershell
|
| 40 |
-
python ... --rate-limit-aware False
|
| 41 |
-
```
|
| 42 |
-
|
| 43 |
-
## Disabling Verify-Remote for Speed
|
| 44 |
-
|
| 45 |
-
**Important:** If resuming with `--resume`, do NOT use `--verify-remote` by default — it doubles the requests!
|
| 46 |
-
|
| 47 |
-
✅ **Good (safe for 150k files):**
|
| 48 |
-
```powershell
|
| 49 |
-
python .\hf_bulk_upload_tool\upload_dataset.py `
|
| 50 |
-
--repo-id "jrrribeiro/PPBIO_Audio_Library" `
|
| 51 |
-
--segments "C:\path\to\segments" `
|
| 52 |
-
--csv "C:\path\to\detections.csv" `
|
| 53 |
-
--hf-token "<YOUR_HF_TOKEN>" --private `
|
| 54 |
-
--resume
|
| 55 |
-
```
|
| 56 |
-
Estimated time: ~12 hours (150k requests at 1000/5min)
|
| 57 |
-
|
| 58 |
-
❌ **Slow (not recommended for 150k files):**
|
| 59 |
-
```powershell
|
| 60 |
-
python ... --resume --verify-remote
|
| 61 |
-
```
|
| 62 |
-
Estimated time: ~24 hours (300k requests)
|
| 63 |
-
|
| 64 |
-
## Handling Interruptions
|
| 65 |
-
|
| 66 |
-
If the script crashes or network fails mid-upload:
|
| 67 |
-
|
| 68 |
-
1. **Wait a moment** (to ensure checkpoint is flushed)
|
| 69 |
-
2. **Re-run the exact same command:**
|
| 70 |
-
```powershell
|
| 71 |
-
python .\hf_bulk_upload_tool\upload_dataset.py `
|
| 72 |
-
--repo-id "jrrribeiro/PPBIO_Audio_Library" `
|
| 73 |
-
--segments "C:\path\to\segments" `
|
| 74 |
-
--csv "C:\path\to\detections.csv" `
|
| 75 |
-
--hf-token "<YOUR_HF_TOKEN>" --private `
|
| 76 |
-
--resume
|
| 77 |
-
```
|
| 78 |
-
|
| 79 |
-
The script will:
|
| 80 |
-
- Load the checkpoint (files already uploaded)
|
| 81 |
-
- Skip all completed files
|
| 82 |
-
- Continue from where it left off
|
| 83 |
-
- **No re-uploading, no wasted time**
|
| 84 |
-
|
| 85 |
-
## Checkpoint Storage
|
| 86 |
-
|
| 87 |
-
Checkpoints are stored in `.checkpoints/`:
|
| 88 |
-
```
|
| 89 |
-
.checkpoints/
|
| 90 |
-
jrrribeiro__PPBIO_Audio_Library.json # List of uploaded files + SHA-256 hashes
|
| 91 |
-
progress.csv # Per-file logs
|
| 92 |
-
jrrribeiro__PPBIO_Audio_Library.summary.json # Final stats
|
| 93 |
-
```
|
| 94 |
-
|
| 95 |
-
Customize location:
|
| 96 |
-
```powershell
|
| 97 |
-
python ... --checkpoint-dir "C:\my\custom\path"
|
| 98 |
-
```
|
| 99 |
-
|
| 100 |
-
## Rate Limiter Configuration
|
| 101 |
-
|
| 102 |
-
Override defaults if needed:
|
| 103 |
-
|
| 104 |
-
| Option | Default | Description |
|
| 105 |
-
|--------|---------|-------------|
|
| 106 |
-
| `--rate-limit-aware` | True | Enable/disable rate limiter |
|
| 107 |
-
| `--rate-limit-max-requests` | 950 | Max requests before pause (safety buffer below 1000) |
|
| 108 |
-
| `--rate-limit-window` | 300 | Window duration in seconds (300 = 5min) |
|
| 109 |
-
|
| 110 |
-
Example (more conservative):
|
| 111 |
-
```powershell
|
| 112 |
-
python ... `
|
| 113 |
-
--rate-limit-max-requests 800 `
|
| 114 |
-
--rate-limit-window 300
|
| 115 |
-
```
|
| 116 |
-
|
| 117 |
-
## Request Counting
|
| 118 |
-
|
| 119 |
-
The script counts these as API requests:
|
| 120 |
-
- **File upload** (each file) = 1 request
|
| 121 |
-
- **List repo files** (resume mode, once) = 1 request
|
| 122 |
-
- **Verify remote** (if --verify-remote per file) = 1 request
|
| 123 |
-
- **CSV upload** = 1 request
|
| 124 |
-
|
| 125 |
-
For 150k small files with `--resume` (no verify-remote):
|
| 126 |
-
- ~150,000 file uploads
|
| 127 |
-
- ~1 list_repo_files
|
| 128 |
-
- ~1 CSV upload
|
| 129 |
-
- **Total: ~150,002 requests ≈ 150 windows ≈ 750 minutes ≈ 12.5 hours**
|
| 130 |
-
|
| 131 |
-
## Monitoring Progress
|
| 132 |
-
|
| 133 |
-
Progress is shown in real-time:
|
| 134 |
-
```
|
| 135 |
-
Uploading: |████████░░░░░░░░░░| 25% [2.5 GB / 10 GB] ETA 9h 15m
|
| 136 |
-
[RateLimit] 948 / 950 requests in current window (3 min left)
|
| 137 |
-
```
|
| 138 |
-
|
| 139 |
-
Key metrics in logs:
|
| 140 |
-
- Bytes uploaded
|
| 141 |
-
- Current rate limiter window status
|
| 142 |
-
- Estimated time remaining
|
| 143 |
-
|
| 144 |
-
## Best Practices for 150k Files
|
| 145 |
-
|
| 146 |
-
1. **Start fresh or resume:**
|
| 147 |
-
```powershell
|
| 148 |
-
# First run
|
| 149 |
-
python ... --resume
|
| 150 |
-
```
|
| 151 |
-
|
| 152 |
-
2. **Keep machine awake** (disable sleep/screensaver)
|
| 153 |
-
|
| 154 |
-
3. **Stable network** (hardwired Ethernet if possible)
|
| 155 |
-
|
| 156 |
-
4. **One worker** (default):
|
| 157 |
-
```powershell
|
| 158 |
-
python ... --max-workers 1
|
| 159 |
-
```
|
| 160 |
-
|
| 161 |
-
5. **Monitor first hour** to ensure everything is working, then let it run
|
| 162 |
-
|
| 163 |
-
6. **Use `--checkpoint-dir` on a reliable disk** (not network drive)
|
| 164 |
-
|
| 165 |
-
## Troubleshooting
|
| 166 |
-
|
| 167 |
-
**Q: Script paused and shows "Waiting Xs before next request"**
|
| 168 |
-
A: Normal! This is the rate limiter preventing 429 errors. Let it continue.
|
| 169 |
-
|
| 170 |
-
**Q: Got 429 anyway?**
|
| 171 |
-
A: Rare, but possible if:
|
| 172 |
-
- Another client is using the same token
|
| 173 |
-
- HF already counted background requests in your quota
|
| 174 |
-
- Solution: Wait 10 min, then resume with `--rate-limit-max-requests 800`
|
| 175 |
-
|
| 176 |
-
**Q: Can I speed this up?**
|
| 177 |
-
A: Only if HF allows higher limits (PRO account). Otherwise, the 1000 req/5min is the hard limit.
|
| 178 |
-
|
| 179 |
-
**Q: How do I monitor progress without staying in the terminal?**
|
| 180 |
-
A: Script logs to `.checkpoints/progress.csv` — check that anytime.
|
| 181 |
-
|
| 182 |
-
## Example Commands
|
| 183 |
-
|
| 184 |
-
### First run (full upload):
|
| 185 |
-
```powershell
|
| 186 |
-
python .\hf_bulk_upload_tool\upload_dataset.py `
|
| 187 |
-
--repo-id "jrrribeiro/PPBIO_Audio_Library" `
|
| 188 |
-
--segments "C:\Users\jonat\OneDrive - MSFT\Doutorado PPGEtno\BirdNET Segments" `
|
| 189 |
-
--csv "C:\Users\jonat\OneDrive - MSFT\Doutorado PPGEtno\BirdNET Analyzer\00_ALL_DETECTIONS.csv" `
|
| 190 |
-
--hf-token "<YOUR_HF_TOKEN>" --private `
|
| 191 |
-
--resume
|
| 192 |
-
```
|
| 193 |
-
ETA: ~12 hours (150k files)
|
| 194 |
-
|
| 195 |
-
### Resume after interruption:
|
| 196 |
-
```powershell
|
| 197 |
-
# Same command as above
|
| 198 |
-
python .\hf_bulk_upload_tool\upload_dataset.py `
|
| 199 |
-
--repo-id "jrrribeiro/PPBIO_Audio_Library" `
|
| 200 |
-
--segments "C:\Users\jonat\OneDrive - MSFT\Doutorado PPGEtno\BirdNET Segments" `
|
| 201 |
-
--csv "C:\Users\jonat\OneDrive - MSFT\Doutorado PPGEtno\BirdNET Analyzer\00_ALL_DETECTIONS.csv" `
|
| 202 |
-
--hf-token "<YOUR_HF_TOKEN>" --private `
|
| 203 |
-
--resume
|
| 204 |
-
```
|
| 205 |
-
Continues from where it left off — only uploads new/missing files.
|
| 206 |
-
|
| 207 |
-
### Conservative (slower, more stable):
|
| 208 |
-
```powershell
|
| 209 |
-
python .\hf_bulk_upload_tool\upload_dataset.py `
|
| 210 |
-
--repo-id "jrrribeiro/PPBIO_Audio_Library" `
|
| 211 |
-
--segments "C:\Users\jonat\OneDrive - MSFT\Doutorado PPGEtno\BirdNET Segments" `
|
| 212 |
-
--csv "C:\Users\jonat\OneDrive - MSFT\Doutorado PPGEtno\BirdNET Analyzer\00_ALL_DETECTIONS.csv" `
|
| 213 |
-
--hf-token "<YOUR_HF_TOKEN>" --private `
|
| 214 |
-
--resume `
|
| 215 |
-
--rate-limit-max-requests 800
|
| 216 |
-
```
|
| 217 |
-
ETA: ~15 hours (more safety margin)
|
| 218 |
-
|
| 219 |
-
---
|
| 220 |
-
|
| 221 |
-
**Ready to start?** Run the first command above and let it work for 12+ hours. The rate limiter will handle everything automatically!
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/README.md
DELETED
|
@@ -1,178 +0,0 @@
|
|
| 1 |
-
# BirdNET HF Bulk Upload Tool
|
| 2 |
-
|
| 3 |
-
Standalone, production-ready script for uploading large BirdNET datasets to Hugging Face with built-in rate-limit handling, resume support, and progress tracking.
|
| 4 |
-
|
| 5 |
-
## Features
|
| 6 |
-
|
| 7 |
-
✅ **Fast uploads** – Uses `HfApi.upload_large_folder` when available; falls back to parallel per-file upload
|
| 8 |
-
✅ **Rate-limit safe** – 429-aware backoff; per-file throttling (default 0.5s delay) to avoid HF API quota exhaustion
|
| 9 |
-
✅ **Resume-friendly** – Checkpoints track uploaded files + SHA-256 hashes; skip already-uploaded files on retry
|
| 10 |
-
✅ **Progress visibility** – Aggregated progress bar with ETA (not verbose per-file logs)
|
| 11 |
-
✅ **Audit logs** – CSV progress log + JSON summary with upload stats
|
| 12 |
-
✅ **Flexible** – Verify remote files by size/ETag; customize worker count, delays, and retry policy
|
| 13 |
-
|
| 14 |
-
## Install
|
| 15 |
-
|
| 16 |
-
Activate the venv and install dependencies:
|
| 17 |
-
|
| 18 |
-
```powershell
|
| 19 |
-
& .\.venv\Scripts\Activate.ps1
|
| 20 |
-
pip install -r .\hf_bulk_upload_tool\requirements.txt
|
| 21 |
-
```
|
| 22 |
-
|
| 23 |
-
## Quick Start
|
| 24 |
-
|
| 25 |
-
### Dry-run (validate before uploading)
|
| 26 |
-
|
| 27 |
-
```powershell
|
| 28 |
-
python .\hf_bulk_upload_tool\upload_dataset.py `
|
| 29 |
-
--repo-id "jrrribeiro/PPBIO_Audio_Library" `
|
| 30 |
-
--segments "C:\path\to\segments" `
|
| 31 |
-
--csv "C:\path\to\detections.csv" `
|
| 32 |
-
--private --dry-run
|
| 33 |
-
```
|
| 34 |
-
|
| 35 |
-
### Upload (rate-limit safe defaults)
|
| 36 |
-
|
| 37 |
-
```powershell
|
| 38 |
-
python .\hf_bulk_upload_tool\upload_dataset.py `
|
| 39 |
-
--repo-id "jrrribeiro/PPBIO_Audio_Library" `
|
| 40 |
-
--segments "C:\path\to\segments" `
|
| 41 |
-
--csv "C:\path\to\detections.csv" `
|
| 42 |
-
--hf-token "<YOUR_HF_TOKEN>" --private
|
| 43 |
-
```
|
| 44 |
-
|
| 45 |
-
### Resume (skip already-uploaded files)
|
| 46 |
-
|
| 47 |
-
```powershell
|
| 48 |
-
python .\hf_bulk_upload_tool\upload_dataset.py `
|
| 49 |
-
--repo-id "jrrribeiro/PPBIO_Audio_Library" `
|
| 50 |
-
--segments "C:\path\to\segments" `
|
| 51 |
-
--csv "C:\path\to\detections.csv" `
|
| 52 |
-
--hf-token "<YOUR_HF_TOKEN>" --private `
|
| 53 |
-
--resume --verify-remote
|
| 54 |
-
```
|
| 55 |
-
|
| 56 |
-
### Inspect only (no upload)
|
| 57 |
-
|
| 58 |
-
```powershell
|
| 59 |
-
python .\hf_bulk_upload_tool\upload_dataset.py `
|
| 60 |
-
--repo-id "jrrribeiro/PPBIO_Audio_Library" `
|
| 61 |
-
--segments "C:\path\to\segments" `
|
| 62 |
-
--private --resume --resume-only
|
| 63 |
-
```
|
| 64 |
-
|
| 65 |
-
## Dataset Layout
|
| 66 |
-
|
| 67 |
-
By default, the script uploads:
|
| 68 |
-
- `segments/` folder → `audio/` in the dataset (preserves subfolder structure)
|
| 69 |
-
- `CSV` file → `index/detections.csv` in the dataset
|
| 70 |
-
|
| 71 |
-
Example:
|
| 72 |
-
```
|
| 73 |
-
Local:
|
| 74 |
-
segments/
|
| 75 |
-
Species A/
|
| 76 |
-
file1.wav
|
| 77 |
-
file2.wav
|
| 78 |
-
Species B/
|
| 79 |
-
file3.wav
|
| 80 |
-
|
| 81 |
-
Dataset:
|
| 82 |
-
audio/Species A/file1.wav
|
| 83 |
-
audio/Species A/file2.wav
|
| 84 |
-
audio/Species B/file3.wav
|
| 85 |
-
index/detections.csv
|
| 86 |
-
```
|
| 87 |
-
|
| 88 |
-
## Large Upload Tuning
|
| 89 |
-
|
| 90 |
-
### If hitting rate limits (429)
|
| 91 |
-
|
| 92 |
-
1. **Wait and retry** (~5 min cooldown): the script uses checkpoints, so files already uploaded are skipped
|
| 93 |
-
2. **Increase per-file delay**: `--per-file-delay 1.0` (default is 0.5s)
|
| 94 |
-
3. **Reduce workers**: `--max-workers 1` (default is 1, but adjust if needed)
|
| 95 |
-
|
| 96 |
-
Example conservative command:
|
| 97 |
-
```powershell
|
| 98 |
-
python .\hf_bulk_upload_tool\upload_dataset.py `
|
| 99 |
-
--repo-id "jrrribeiro/PPBIO_Audio_Library" `
|
| 100 |
-
--segments "C:\path\to\segments" `
|
| 101 |
-
--hf-token "<YOUR_HF_TOKEN>" --private `
|
| 102 |
-
--max-workers 1 --per-file-delay 1.0
|
| 103 |
-
```
|
| 104 |
-
|
| 105 |
-
### For very large datasets (50+ GB)
|
| 106 |
-
|
| 107 |
-
Increase HF timeouts:
|
| 108 |
-
|
| 109 |
-
```powershell
|
| 110 |
-
$env:HF_HUB_ETAG_TIMEOUT = '60'
|
| 111 |
-
$env:HF_HUB_DOWNLOAD_TIMEOUT = '300'
|
| 112 |
-
$env:HF_XET_HIGH_PERFORMANCE = '1'
|
| 113 |
-
```
|
| 114 |
-
|
| 115 |
-
Then run the upload.
|
| 116 |
-
|
| 117 |
-
## Checkpoints & Logging
|
| 118 |
-
|
| 119 |
-
The script saves:
|
| 120 |
-
- `.checkpoints/<repo_id>.json` – list of uploaded files + SHA-256 hashes
|
| 121 |
-
- `.checkpoints/progress.csv` – per-file status, elapsed time, errors
|
| 122 |
-
- `.checkpoints/<repo_id>.summary.json` – total stats (files, bytes, elapsed time)
|
| 123 |
-
|
| 124 |
-
Use `--checkpoint-dir <path>` to customize the checkpoint location.
|
| 125 |
-
|
| 126 |
-
## All Options
|
| 127 |
-
|
| 128 |
-
```
|
| 129 |
-
python .\hf_bulk_upload_tool\upload_dataset.py --help
|
| 130 |
-
```
|
| 131 |
-
|
| 132 |
-
Key arguments:
|
| 133 |
-
| Option | Default | Description |
|
| 134 |
-
|--------|---------|-------------|
|
| 135 |
-
| `--repo-id` | (required) | `username/dataset-name` |
|
| 136 |
-
| `--segments` | (required) | Local folder path |
|
| 137 |
-
| `--csv` | (none) | Optional CSV file to upload |
|
| 138 |
-
| `--hf-token` | (prompt) | HF write token (or use `HF_TOKEN` env var) |
|
| 139 |
-
| `--private` | true | Create private dataset (or `--public`) |
|
| 140 |
-
| `--max-workers` | 1 | Parallel upload threads (per-file fallback) |
|
| 141 |
-
| `--per-file-delay` | 0.5 | Seconds between per-file uploads |
|
| 142 |
-
| `--resume` | false | Skip already-uploaded files |
|
| 143 |
-
| `--verify-remote` | false | Verify remote file size/ETag on resume |
|
| 144 |
-
| `--verify-etag` | false | Prefer ETag over size checks |
|
| 145 |
-
| `--resume-only` | false | Inspect repo without uploading |
|
| 146 |
-
| `--dry-run` | false | Validate args; no network calls |
|
| 147 |
-
| `--checkpoint-dir` | `.checkpoints` | Where to store checkpoints |
|
| 148 |
-
|
| 149 |
-
## Troubleshooting
|
| 150 |
-
|
| 151 |
-
**"Failed to get upload mode: 429"**
|
| 152 |
-
HF rate limit hit. Wait 5 min, then re-run (script will skip completed files).
|
| 153 |
-
|
| 154 |
-
**"No such file or directory" (temp junction)**
|
| 155 |
-
Rare on Windows. Try increasing `--per-file-delay` or reduce `--max-workers`.
|
| 156 |
-
|
| 157 |
-
**"Failed to create audio junction" (Windows)**
|
| 158 |
-
Ensure temp folder has write permissions. Or use `--max-workers 1 --per-file-delay 0.5`.
|
| 159 |
-
|
| 160 |
-
**"Authentication failed"**
|
| 161 |
-
Check your HF token is valid and has write permission to the target repo.
|
| 162 |
-
|
| 163 |
-
## Notes
|
| 164 |
-
|
| 165 |
-
- **Keep machine awake** during uploads (no sleep mode)
|
| 166 |
-
- **Stable network** strongly recommended for 50+ GB uploads
|
| 167 |
-
- **Re-run is safe** – checkpoints prevent re-uploading files
|
| 168 |
-
- **Start small** – test with a tiny folder first if unsure about permissions
|
| 169 |
-
|
| 170 |
-
## Environment Variables
|
| 171 |
-
|
| 172 |
-
Override defaults:
|
| 173 |
-
```powershell
|
| 174 |
-
$env:HF_HUB_ETAG_TIMEOUT = '60'
|
| 175 |
-
$env:HF_HUB_DOWNLOAD_TIMEOUT = '300'
|
| 176 |
-
$env:HF_XET_HIGH_PERFORMANCE = '1'
|
| 177 |
-
$env:HF_TOKEN = '<your_token>' # avoid typing --hf-token each time
|
| 178 |
-
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/README_GUI.md
DELETED
|
@@ -1,261 +0,0 @@
|
|
| 1 |
-
# HF Dataset Uploader GUI – Protocol & Implementation Guide
|
| 2 |
-
|
| 3 |
-
## Overview
|
| 4 |
-
|
| 5 |
-
This directory contains a refactored version of `upload_dataset.py` designed for packaging as a portable, user-friendly GUI application using **CustomTkinter** and **PyInstaller**.
|
| 6 |
-
|
| 7 |
-
The project follows a **"Wrapper" architecture**:
|
| 8 |
-
- **Motor (Logic Layer)**: `upload_logic.py` — Pure upload logic, no UI dependencies
|
| 9 |
-
- **Cabin (UI Layer)**: `main_gui.py` — CustomTkinter GUI that calls the motor via callbacks
|
| 10 |
-
- **Build System**: PyInstaller configuration and build guide
|
| 11 |
-
|
| 12 |
-
---
|
| 13 |
-
|
| 14 |
-
## Architecture
|
| 15 |
-
|
| 16 |
-
### 1. **upload_logic.py** (Refactored Core)
|
| 17 |
-
|
| 18 |
-
**Key Changes from `upload_dataset.py`:**
|
| 19 |
-
- ✅ Removed `argparse` dependency (no CLI parsing in this version)
|
| 20 |
-
- ✅ Extracted `main()` into `run_upload_logic(config, logger_callback)`
|
| 21 |
-
- ✅ Replaced all `print()` calls with `logger(message)` callback
|
| 22 |
-
- ✅ Config is a Python dict instead of argparse Namespace
|
| 23 |
-
- ✅ 100% backward compatible with the original logic
|
| 24 |
-
|
| 25 |
-
**Function Signature:**
|
| 26 |
-
```python
|
| 27 |
-
def run_upload_logic(config: dict, logger: Callable[[str], None]) -> int:
|
| 28 |
-
"""
|
| 29 |
-
Execute the upload process.
|
| 30 |
-
|
| 31 |
-
Args:
|
| 32 |
-
config: Dictionary with keys like 'repo_id', 'hf_token', 'segments', etc.
|
| 33 |
-
logger: Callable that accepts strings for logging (e.g., textbox.insert)
|
| 34 |
-
|
| 35 |
-
Returns:
|
| 36 |
-
0 on success, raises Exception on failure
|
| 37 |
-
"""
|
| 38 |
-
```
|
| 39 |
-
|
| 40 |
-
**Config Keys (All Optional Except Marked):**
|
| 41 |
-
```python
|
| 42 |
-
config = {
|
| 43 |
-
"hf_token": "...", # ✅ Required
|
| 44 |
-
"repo_id": "username/repo", # ✅ Required (format: "owner/name")
|
| 45 |
-
"segments": "/path/to/audio", # ✅ Required (directory with files)
|
| 46 |
-
"csv": "/path/to/file.csv", # Optional
|
| 47 |
-
"segments_path_in_repo": "audio", # Default: "audio"
|
| 48 |
-
"csv_path_in_repo": "index/detections.csv", # Default
|
| 49 |
-
"private": True, # Default: True
|
| 50 |
-
"resume": True, # Default: False
|
| 51 |
-
"verify_remote": False, # Default: False
|
| 52 |
-
"verify_etag": False, # Default: False
|
| 53 |
-
"dry_run": False, # Default: False
|
| 54 |
-
"rate_limit_aware": True, # Default: True
|
| 55 |
-
"rate_limit_max_requests": 950, # Default: 950
|
| 56 |
-
"rate_limit_window": 300, # Default: 300 (5 min)
|
| 57 |
-
"create_repo_attempts": 3, # Default: 3
|
| 58 |
-
"upload_attempts": 5, # Default: 3
|
| 59 |
-
"retry_backoff": 10.0, # Default: 5.0
|
| 60 |
-
"max_workers": 2, # Default: 1
|
| 61 |
-
"checkpoint_dir": "/path", # Default: .checkpoints
|
| 62 |
-
"commit_message": "Upload BirdNET segments", # Default
|
| 63 |
-
}
|
| 64 |
-
```
|
| 65 |
-
|
| 66 |
-
### 2. **main_gui.py** (CustomTkinter GUI)
|
| 67 |
-
|
| 68 |
-
**UI Sections:**
|
| 69 |
-
|
| 70 |
-
1. **📋 Upload Configuration**
|
| 71 |
-
- Repository ID input (username/dataset-name)
|
| 72 |
-
- HF Token input (password field, hidden)
|
| 73 |
-
- Segments folder picker (browse button)
|
| 74 |
-
- CSV file picker (optional, browse button)
|
| 75 |
-
|
| 76 |
-
2. **⚙️ Options**
|
| 77 |
-
- Private Repository (checkbox)
|
| 78 |
-
- Resume mode (checkbox)
|
| 79 |
-
- Verify remote files (checkbox)
|
| 80 |
-
- Rate limiting (checkbox)
|
| 81 |
-
- Worker threads (spinbox, 1-8)
|
| 82 |
-
|
| 83 |
-
3. **📝 Upload Progress**
|
| 84 |
-
- Large textbox for real-time log output
|
| 85 |
-
- Auto-scrolls to latest message
|
| 86 |
-
- Read-only to prevent user edits
|
| 87 |
-
|
| 88 |
-
4. **🚀 Buttons**
|
| 89 |
-
- Start Upload (green, initiates upload in background thread)
|
| 90 |
-
- Stop (disabled until upload starts)
|
| 91 |
-
- Clear Log (clears textbox)
|
| 92 |
-
- Status label (Ready/Uploading/Complete/Failed)
|
| 93 |
-
|
| 94 |
-
**Threading Model:**
|
| 95 |
-
- Main thread: UI event loop (never blocks)
|
| 96 |
-
- Upload thread: Runs `run_upload_logic()` in background
|
| 97 |
-
- Log queue: Thread-safe message passing (Queue.Queue)
|
| 98 |
-
|
| 99 |
-
```python
|
| 100 |
-
# Upload runs in background; logs appear in real-time
|
| 101 |
-
Thread 1 (Main UI):
|
| 102 |
-
→ user clicks "Start"
|
| 103 |
-
→ spawns Thread 2
|
| 104 |
-
→ continues processing UI events
|
| 105 |
-
↓
|
| 106 |
-
Thread 2 (Upload):
|
| 107 |
-
→ calls run_upload_logic()
|
| 108 |
-
→ logs messages → queue → picked up by Thread 1 every 100ms
|
| 109 |
-
```
|
| 110 |
-
|
| 111 |
-
---
|
| 112 |
-
|
| 113 |
-
## How to Use (Development)
|
| 114 |
-
|
| 115 |
-
### 1. **Install Dependencies**
|
| 116 |
-
|
| 117 |
-
```powershell
|
| 118 |
-
# Base requirements
|
| 119 |
-
pip install -r ../requirements.txt
|
| 120 |
-
|
| 121 |
-
# GUI additions
|
| 122 |
-
pip install -r requirements_gui.txt
|
| 123 |
-
```
|
| 124 |
-
|
| 125 |
-
### 2. **Test the GUI**
|
| 126 |
-
|
| 127 |
-
```powershell
|
| 128 |
-
python main_gui.py
|
| 129 |
-
```
|
| 130 |
-
|
| 131 |
-
This opens the GUI window. Fill in test values and click "Start Upload" to test the flow.
|
| 132 |
-
|
| 133 |
-
### 3. **Build the EXE**
|
| 134 |
-
|
| 135 |
-
Follow instructions in **BUILD_GUIDE.md**, or quick version:
|
| 136 |
-
|
| 137 |
-
```powershell
|
| 138 |
-
pyinstaller --onefile --noconsole `
|
| 139 |
-
--name "HF_Dataset_Uploader" `
|
| 140 |
-
--collect-all customtkinter `
|
| 141 |
-
--collect-all huggingface_hub `
|
| 142 |
-
main_gui.py
|
| 143 |
-
```
|
| 144 |
-
|
| 145 |
-
Output: `dist/HF_Dataset_Uploader.exe` (standalone, ~250 MB)
|
| 146 |
-
|
| 147 |
-
---
|
| 148 |
-
|
| 149 |
-
## File Structure
|
| 150 |
-
|
| 151 |
-
```
|
| 152 |
-
hf_bulk_upload_tool/
|
| 153 |
-
├── upload_dataset.py ← Original CLI version (unchanged)
|
| 154 |
-
├── upload_logic.py ← NEW: Refactored logic (no CLI)
|
| 155 |
-
├── main_gui.py ← NEW: CustomTkinter GUI
|
| 156 |
-
├── requirements_gui.txt ← NEW: GUI dependencies
|
| 157 |
-
├── BUILD_GUIDE.md ← NEW: Step-by-step build instructions
|
| 158 |
-
├��─ hf_uploader_gui.spec ← NEW: PyInstaller spec file
|
| 159 |
-
└── .checkpoints/ ← Auto-created: stores resume checkpoints
|
| 160 |
-
```
|
| 161 |
-
|
| 162 |
-
---
|
| 163 |
-
|
| 164 |
-
## Key Design Decisions
|
| 165 |
-
|
| 166 |
-
### 1. **Why Refactor Logic?**
|
| 167 |
-
- Separating UI from logic makes the code reusable and testable
|
| 168 |
-
- The original `upload_dataset.py` is untouched; we added a new layer on top
|
| 169 |
-
- Future: can plug in different UIs (CLI, web, REST API) without changing logic
|
| 170 |
-
|
| 171 |
-
### 2. **Why Thread-Safe Logging?**
|
| 172 |
-
- If we called `textbox.insert()` directly from the upload thread, the GUI could crash
|
| 173 |
-
- Using `queue.Queue` ensures thread safety and smooth updates
|
| 174 |
-
|
| 175 |
-
### 3. **Why CustomTkinter?**
|
| 176 |
-
- **Modern look** compared to vanilla tkinter (dark mode, rounded buttons, etc.)
|
| 177 |
-
- **Pure Python** (no compiled dependencies)
|
| 178 |
-
- **Lightweight** (~5 MB)
|
| 179 |
-
- **Cross-platform** (Windows, macOS, Linux)
|
| 180 |
-
|
| 181 |
-
### 4. **Resume as Default?**
|
| 182 |
-
- `resume=True` by default in the GUI
|
| 183 |
-
- Prevents re-uploading files and respects HF rate limits
|
| 184 |
-
- Safe: checkpoint system tracks uploaded files locally
|
| 185 |
-
|
| 186 |
-
---
|
| 187 |
-
|
| 188 |
-
## Limitations & Future Improvements
|
| 189 |
-
|
| 190 |
-
### Current Limitations
|
| 191 |
-
1. ❌ **No mid-upload cancellation** — user can close app, but upload thread keeps running
|
| 192 |
-
2. ❌ **No progress bar** — only text log (could add % completion)
|
| 193 |
-
3. ❌ **No drag-and-drop** for file selection
|
| 194 |
-
4. ❌ **No dark mode toggle** (always dark)
|
| 195 |
-
|
| 196 |
-
### Potential Improvements (v2)
|
| 197 |
-
- ✅ Real-time progress bar showing % completion
|
| 198 |
-
- ✅ Ability to pause/resume upload mid-flight
|
| 199 |
-
- ✅ Save favorite configs (profiles)
|
| 200 |
-
- ✅ Drag-and-drop for folder/CSV inputs
|
| 201 |
-
- ✅ System tray icon with notifications
|
| 202 |
-
- ✅ Embedded browser to validate HF token in-app
|
| 203 |
-
- ✅ Light mode toggle
|
| 204 |
-
- ✅ Auto-update mechanism
|
| 205 |
-
|
| 206 |
-
---
|
| 207 |
-
|
| 208 |
-
## Testing Checklist
|
| 209 |
-
|
| 210 |
-
Before building the EXE, verify:
|
| 211 |
-
|
| 212 |
-
- [ ] `python main_gui.py` launches without errors
|
| 213 |
-
- [ ] All buttons and inputs are clickable
|
| 214 |
-
- [ ] File picker dialogs open correctly
|
| 215 |
-
- [ ] Token field is masked (shows `*` not plaintext)
|
| 216 |
-
- [ ] Log textbox updates in real-time
|
| 217 |
-
- [ ] Empty fields show validation errors
|
| 218 |
-
- [ ] Non-existent paths are caught and reported
|
| 219 |
-
- [ ] Start/Stop button states toggle correctly
|
| 220 |
-
|
| 221 |
-
---
|
| 222 |
-
|
| 223 |
-
## Troubleshooting
|
| 224 |
-
|
| 225 |
-
| Issue | Solution |
|
| 226 |
-
|-------|----------|
|
| 227 |
-
| `ModuleNotFoundError: customtkinter` | `pip install customtkinter` |
|
| 228 |
-
| `ModuleNotFoundError: huggingface_hub` | `pip install huggingface-hub` |
|
| 229 |
-
| GUI shows blank window | Run with `python -u main_gui.py` to force unbuffered output |
|
| 230 |
-
| EXE won't start | Double-click and check Windows event log for errors |
|
| 231 |
-
| "Only administrator can run this" | Right-click EXE → Properties → Compatibility → Run as admin |
|
| 232 |
-
| Antivirus blocks EXE | Add exception for the file or upload to VirusTotal.com to verify |
|
| 233 |
-
|
| 234 |
-
---
|
| 235 |
-
|
| 236 |
-
## Summary
|
| 237 |
-
|
| 238 |
-
This protocol transforms the CLI upload script into a professional, distributable GUI application:
|
| 239 |
-
|
| 240 |
-
1. **Logic Layer** (`upload_logic.py`): Pure, testable upload code
|
| 241 |
-
2. **UI Layer** (`main_gui.py`): User-friendly CustomTkinter interface
|
| 242 |
-
3. **Build System** (PyInstaller): Creates a standalone `.exe` for non-technical users
|
| 243 |
-
4. **Threading**: Ensures UI never freezes during long uploads
|
| 244 |
-
5. **Logging**: Real-time progress visible in the GUI
|
| 245 |
-
|
| 246 |
-
Users can now:
|
| 247 |
-
- ✅ Double-click an `.exe` to upload datasets
|
| 248 |
-
- ✅ No need to understand Python, CLI, or argparse
|
| 249 |
-
- ✅ Safe resume if interrupted
|
| 250 |
-
- ✅ Real-time progress feedback
|
| 251 |
-
|
| 252 |
-
---
|
| 253 |
-
|
| 254 |
-
## Next Steps
|
| 255 |
-
|
| 256 |
-
1. **Test locally**: `python main_gui.py`
|
| 257 |
-
2. **Build EXE**: Follow BUILD_GUIDE.md
|
| 258 |
-
3. **Distribute**: Share the `.exe` with non-technical users
|
| 259 |
-
4. **Iterate**: Gather feedback, add improvements from "Future Improvements" section
|
| 260 |
-
|
| 261 |
-
Happy uploading! 🚀
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/__init__.py
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
"""HF Bulk Upload Tool - Hugging Face Dataset Upload Utilities"""
|
| 2 |
-
|
| 3 |
-
__version__ = "1.0.0"
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/hf_uploader_gui.spec
DELETED
|
@@ -1,56 +0,0 @@
|
|
| 1 |
-
# -*- mode: python ; coding: utf-8 -*-
|
| 2 |
-
import sys
|
| 3 |
-
import os
|
| 4 |
-
from pathlib import Path
|
| 5 |
-
|
| 6 |
-
block_cipher = None
|
| 7 |
-
script_dir = Path(os.path.dirname(os.path.abspath(SPEC)))
|
| 8 |
-
main_gui_path = script_dir / 'main_gui.py'
|
| 9 |
-
|
| 10 |
-
a = Analysis(
|
| 11 |
-
[str(main_gui_path)],
|
| 12 |
-
pathex=[str(script_dir)],
|
| 13 |
-
binaries=[],
|
| 14 |
-
datas=[],
|
| 15 |
-
hiddenimports=[
|
| 16 |
-
'upload_logic',
|
| 17 |
-
'customtkinter',
|
| 18 |
-
'huggingface_hub',
|
| 19 |
-
'requests',
|
| 20 |
-
'tqdm',
|
| 21 |
-
'sharding_utils',
|
| 22 |
-
'progress_bar_utils',
|
| 23 |
-
'resilience_utils',
|
| 24 |
-
],
|
| 25 |
-
hookspath=[],
|
| 26 |
-
hooksconfig={},
|
| 27 |
-
runtime_hooks=[],
|
| 28 |
-
excludedimports=[],
|
| 29 |
-
win_no_prefer_redirects=False,
|
| 30 |
-
win_private_assemblies=False,
|
| 31 |
-
cipher=block_cipher,
|
| 32 |
-
noarchive=False,
|
| 33 |
-
)
|
| 34 |
-
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)
|
| 35 |
-
exe = EXE(
|
| 36 |
-
pyz,
|
| 37 |
-
a.scripts,
|
| 38 |
-
a.binaries,
|
| 39 |
-
a.zipfiles,
|
| 40 |
-
a.datas,
|
| 41 |
-
[],
|
| 42 |
-
name='HF_Dataset_Uploader',
|
| 43 |
-
debug=False,
|
| 44 |
-
bootloader_ignore_signals=False,
|
| 45 |
-
strip=False,
|
| 46 |
-
upx=True,
|
| 47 |
-
upx_exclude=[],
|
| 48 |
-
runtime_tmpdir=None,
|
| 49 |
-
console=False,
|
| 50 |
-
disable_windowed_traceback=False,
|
| 51 |
-
argv_emulation=False,
|
| 52 |
-
target_arch=None,
|
| 53 |
-
codesign_identity=None,
|
| 54 |
-
entitlements_file=None,
|
| 55 |
-
icon=None,
|
| 56 |
-
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/main_gui.py
DELETED
|
@@ -1,443 +0,0 @@
|
|
| 1 |
-
#!/usr/bin/env python3
|
| 2 |
-
"""
|
| 3 |
-
GUI for Hugging Face Bulk Uploader using CustomTkinter.
|
| 4 |
-
|
| 5 |
-
This is a portable GUI wrapper around the upload_logic module, designed to be
|
| 6 |
-
packaged as a standalone .exe using PyInstaller.
|
| 7 |
-
"""
|
| 8 |
-
|
| 9 |
-
import customtkinter as ctk
|
| 10 |
-
from tkinter import filedialog, messagebox
|
| 11 |
-
import threading
|
| 12 |
-
import queue
|
| 13 |
-
from pathlib import Path
|
| 14 |
-
import sys
|
| 15 |
-
|
| 16 |
-
# Import the refactored logic using a robust lookup that works both
|
| 17 |
-
# in development and when packaged by PyInstaller.
|
| 18 |
-
import importlib
|
| 19 |
-
|
| 20 |
-
def _find_run_upload_logic():
|
| 21 |
-
# Try explicit module names first
|
| 22 |
-
for name in ("hf_bulk_upload_tool.upload_logic", "upload_logic"):
|
| 23 |
-
try:
|
| 24 |
-
m = importlib.import_module(name)
|
| 25 |
-
if hasattr(m, "run_upload_logic"):
|
| 26 |
-
return getattr(m, "run_upload_logic")
|
| 27 |
-
except Exception:
|
| 28 |
-
pass
|
| 29 |
-
|
| 30 |
-
# Search already-imported modules for a candidate (useful in PyInstaller)
|
| 31 |
-
for m in list(sys.modules.values()):
|
| 32 |
-
try:
|
| 33 |
-
if hasattr(m, "run_upload_logic"):
|
| 34 |
-
return getattr(m, "run_upload_logic")
|
| 35 |
-
except Exception:
|
| 36 |
-
continue
|
| 37 |
-
|
| 38 |
-
# As a last resort, if this file is in a package, try relative import
|
| 39 |
-
pkg = __package__
|
| 40 |
-
if pkg:
|
| 41 |
-
try:
|
| 42 |
-
m = importlib.import_module(pkg + ".upload_logic")
|
| 43 |
-
if hasattr(m, "run_upload_logic"):
|
| 44 |
-
return getattr(m, "run_upload_logic")
|
| 45 |
-
except Exception:
|
| 46 |
-
pass
|
| 47 |
-
|
| 48 |
-
return None
|
| 49 |
-
|
| 50 |
-
run_upload_logic = _find_run_upload_logic()
|
| 51 |
-
if run_upload_logic is None:
|
| 52 |
-
raise ImportError("Could not import `run_upload_logic` from hf_bulk_upload_tool or upload_logic; searched known modules and sys.modules")
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
class UploadConfig:
|
| 56 |
-
"""Simple config holder for upload parameters."""
|
| 57 |
-
def __init__(self):
|
| 58 |
-
self.repo_id = ""
|
| 59 |
-
self.hf_token = ""
|
| 60 |
-
self.segments_path = ""
|
| 61 |
-
self.csv_path = ""
|
| 62 |
-
self.private = True
|
| 63 |
-
self.resume = True
|
| 64 |
-
self.verify_remote = False
|
| 65 |
-
self.verify_etag = False
|
| 66 |
-
self.rate_limit_aware = True
|
| 67 |
-
self.max_workers = 1
|
| 68 |
-
self.upload_attempts = 3
|
| 69 |
-
self.retry_backoff = 5.0
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
class HFUploaderApp(ctk.CTk):
|
| 73 |
-
"""Main GUI application for Hugging Face dataset uploads."""
|
| 74 |
-
|
| 75 |
-
def __init__(self):
|
| 76 |
-
super().__init__()
|
| 77 |
-
|
| 78 |
-
self.title("Hugging Face Dataset Uploader")
|
| 79 |
-
self.geometry("900x750")
|
| 80 |
-
self.resizable(True, True)
|
| 81 |
-
|
| 82 |
-
# Configure appearance
|
| 83 |
-
ctk.set_appearance_mode("dark")
|
| 84 |
-
ctk.set_default_color_theme("blue")
|
| 85 |
-
|
| 86 |
-
# Upload state
|
| 87 |
-
self.uploading = False
|
| 88 |
-
self.upload_thread = None
|
| 89 |
-
self.log_queue: queue.Queue = queue.Queue()
|
| 90 |
-
self.stop_event = threading.Event()
|
| 91 |
-
self._dynamic_line_active = False
|
| 92 |
-
self._dynamic_line_start = None
|
| 93 |
-
|
| 94 |
-
# Build UI
|
| 95 |
-
self._build_ui()
|
| 96 |
-
|
| 97 |
-
# Start log queue processor
|
| 98 |
-
self.after(100, self._process_log_queue)
|
| 99 |
-
|
| 100 |
-
def _build_ui(self):
|
| 101 |
-
"""Build the user interface."""
|
| 102 |
-
# Main container with padding
|
| 103 |
-
main_frame = ctk.CTkFrame(self, fg_color="transparent")
|
| 104 |
-
main_frame.pack(fill="both", expand=True, padx=10, pady=10)
|
| 105 |
-
|
| 106 |
-
# ===== CONFIGURATION SECTION =====
|
| 107 |
-
config_label = ctk.CTkLabel(
|
| 108 |
-
main_frame, text="📋 Upload Configuration",
|
| 109 |
-
font=("Arial", 14, "bold")
|
| 110 |
-
)
|
| 111 |
-
config_label.pack(anchor="w", pady=(0, 10))
|
| 112 |
-
|
| 113 |
-
config_frame = ctk.CTkFrame(main_frame)
|
| 114 |
-
config_frame.pack(fill="x", pady=(0, 15))
|
| 115 |
-
|
| 116 |
-
# Row 1: Repo ID
|
| 117 |
-
ctk.CTkLabel(config_frame, text="Repository ID:", font=("Arial", 11)).grid(
|
| 118 |
-
row=0, column=0, sticky="w", padx=5, pady=5
|
| 119 |
-
)
|
| 120 |
-
self.repo_id_entry = ctk.CTkEntry(
|
| 121 |
-
config_frame, placeholder_text="username/dataset-name", width=300
|
| 122 |
-
)
|
| 123 |
-
self.repo_id_entry.grid(row=0, column=1, sticky="ew", padx=5, pady=5)
|
| 124 |
-
|
| 125 |
-
# Row 2: Token
|
| 126 |
-
ctk.CTkLabel(config_frame, text="HF Token:", font=("Arial", 11)).grid(
|
| 127 |
-
row=1, column=0, sticky="w", padx=5, pady=5
|
| 128 |
-
)
|
| 129 |
-
self.token_entry = ctk.CTkEntry(
|
| 130 |
-
config_frame, placeholder_text="hf_...", show="*", width=300
|
| 131 |
-
)
|
| 132 |
-
self.token_entry.grid(row=1, column=1, sticky="ew", padx=5, pady=5)
|
| 133 |
-
|
| 134 |
-
# Row 3: Segments path
|
| 135 |
-
ctk.CTkLabel(config_frame, text="Segments Folder:", font=("Arial", 11)).grid(
|
| 136 |
-
row=2, column=0, sticky="w", padx=5, pady=5
|
| 137 |
-
)
|
| 138 |
-
segments_row = ctk.CTkFrame(config_frame, fg_color="transparent")
|
| 139 |
-
segments_row.grid(row=2, column=1, sticky="ew", padx=5, pady=5)
|
| 140 |
-
|
| 141 |
-
self.segments_entry = ctk.CTkEntry(segments_row, width=200)
|
| 142 |
-
self.segments_entry.pack(side="left", fill="x", expand=True, padx=(0, 5))
|
| 143 |
-
|
| 144 |
-
ctk.CTkButton(
|
| 145 |
-
segments_row, text="Browse", width=80,
|
| 146 |
-
command=self._browse_segments
|
| 147 |
-
).pack(side="left")
|
| 148 |
-
|
| 149 |
-
# Row 4: CSV path
|
| 150 |
-
ctk.CTkLabel(config_frame, text="CSV File (Optional):", font=("Arial", 11)).grid(
|
| 151 |
-
row=3, column=0, sticky="w", padx=5, pady=5
|
| 152 |
-
)
|
| 153 |
-
csv_row = ctk.CTkFrame(config_frame, fg_color="transparent")
|
| 154 |
-
csv_row.grid(row=3, column=1, sticky="ew", padx=5, pady=5)
|
| 155 |
-
|
| 156 |
-
self.csv_entry = ctk.CTkEntry(csv_row, width=200)
|
| 157 |
-
self.csv_entry.pack(side="left", fill="x", expand=True, padx=(0, 5))
|
| 158 |
-
|
| 159 |
-
ctk.CTkButton(
|
| 160 |
-
csv_row, text="Browse", width=80,
|
| 161 |
-
command=self._browse_csv
|
| 162 |
-
).pack(side="left")
|
| 163 |
-
|
| 164 |
-
# Configure grid weights for proper stretching
|
| 165 |
-
config_frame.columnconfigure(1, weight=1)
|
| 166 |
-
|
| 167 |
-
# ===== OPTIONS SECTION =====
|
| 168 |
-
options_label = ctk.CTkLabel(
|
| 169 |
-
main_frame, text="⚙️ Options",
|
| 170 |
-
font=("Arial", 14, "bold")
|
| 171 |
-
)
|
| 172 |
-
options_label.pack(anchor="w", pady=(10, 10))
|
| 173 |
-
|
| 174 |
-
options_frame = ctk.CTkFrame(main_frame)
|
| 175 |
-
options_frame.pack(fill="x", pady=(0, 15))
|
| 176 |
-
|
| 177 |
-
# Checkboxes
|
| 178 |
-
self.private_var = ctk.BooleanVar(value=True)
|
| 179 |
-
ctk.CTkCheckBox(
|
| 180 |
-
options_frame, text="Private Repository",
|
| 181 |
-
variable=self.private_var, font=("Arial", 10)
|
| 182 |
-
).pack(anchor="w", padx=5, pady=2)
|
| 183 |
-
|
| 184 |
-
self.resume_var = ctk.BooleanVar(value=True)
|
| 185 |
-
ctk.CTkCheckBox(
|
| 186 |
-
options_frame, text="Resume (skip already uploaded files)",
|
| 187 |
-
variable=self.resume_var, font=("Arial", 10)
|
| 188 |
-
).pack(anchor="w", padx=5, pady=2)
|
| 189 |
-
|
| 190 |
-
self.verify_var = ctk.BooleanVar(value=False)
|
| 191 |
-
ctk.CTkCheckBox(
|
| 192 |
-
options_frame, text="Verify remote files before skipping",
|
| 193 |
-
variable=self.verify_var, font=("Arial", 10)
|
| 194 |
-
).pack(anchor="w", padx=5, pady=2)
|
| 195 |
-
|
| 196 |
-
self.rate_limit_var = ctk.BooleanVar(value=True)
|
| 197 |
-
ctk.CTkCheckBox(
|
| 198 |
-
options_frame, text="Enable rate limiting (1000 req/5min)",
|
| 199 |
-
variable=self.rate_limit_var, font=("Arial", 10)
|
| 200 |
-
).pack(anchor="w", padx=5, pady=2)
|
| 201 |
-
|
| 202 |
-
# Worker threads
|
| 203 |
-
worker_frame = ctk.CTkFrame(options_frame, fg_color="transparent")
|
| 204 |
-
worker_frame.pack(anchor="w", padx=5, pady=5)
|
| 205 |
-
|
| 206 |
-
ctk.CTkLabel(worker_frame, text="Worker threads:", font=("Arial", 10)).pack(side="left", padx=(0, 10))
|
| 207 |
-
self.workers_combobox = ctk.CTkComboBox(
|
| 208 |
-
worker_frame,
|
| 209 |
-
values=["1", "2", "3", "4", "5", "6", "7", "8"],
|
| 210 |
-
width=60,
|
| 211 |
-
font=("Arial", 10),
|
| 212 |
-
state="readonly"
|
| 213 |
-
)
|
| 214 |
-
self.workers_combobox.set("2")
|
| 215 |
-
self.workers_combobox.pack(side="left")
|
| 216 |
-
|
| 217 |
-
# ===== CONSOLE / LOG SECTION =====
|
| 218 |
-
console_label = ctk.CTkLabel(
|
| 219 |
-
main_frame, text="📝 Upload Progress",
|
| 220 |
-
font=("Arial", 14, "bold")
|
| 221 |
-
)
|
| 222 |
-
console_label.pack(anchor="w", pady=(10, 5))
|
| 223 |
-
|
| 224 |
-
self.log_textbox = ctk.CTkTextbox(
|
| 225 |
-
main_frame, width=800, height=250,
|
| 226 |
-
font=("Courier", 10), wrap="none"
|
| 227 |
-
)
|
| 228 |
-
self.log_textbox.pack(fill="both", expand=True, pady=(0, 15))
|
| 229 |
-
|
| 230 |
-
# Disable editing of log textbox
|
| 231 |
-
self.log_textbox.configure(state="disabled")
|
| 232 |
-
|
| 233 |
-
# ===== BUTTON SECTION =====
|
| 234 |
-
button_frame = ctk.CTkFrame(main_frame, fg_color="transparent")
|
| 235 |
-
button_frame.pack(fill="x", pady=(0, 10))
|
| 236 |
-
|
| 237 |
-
self.start_btn = ctk.CTkButton(
|
| 238 |
-
button_frame, text="🚀 Start Upload",
|
| 239 |
-
command=self._start_upload, width=150, height=40,
|
| 240 |
-
font=("Arial", 12, "bold"), fg_color="#2ecc71"
|
| 241 |
-
)
|
| 242 |
-
self.start_btn.pack(side="left", padx=5)
|
| 243 |
-
|
| 244 |
-
self.stop_btn = ctk.CTkButton(
|
| 245 |
-
button_frame, text="⏹ Stop",
|
| 246 |
-
command=self._stop_upload, width=150, height=40,
|
| 247 |
-
state="disabled", font=("Arial", 12)
|
| 248 |
-
)
|
| 249 |
-
self.stop_btn.pack(side="left", padx=5)
|
| 250 |
-
|
| 251 |
-
self.clear_btn = ctk.CTkButton(
|
| 252 |
-
button_frame, text="🗑 Clear Log",
|
| 253 |
-
command=self._clear_log, width=150, height=40,
|
| 254 |
-
font=("Arial", 12)
|
| 255 |
-
)
|
| 256 |
-
self.clear_btn.pack(side="left", padx=5)
|
| 257 |
-
|
| 258 |
-
# Status label
|
| 259 |
-
self.status_label = ctk.CTkLabel(
|
| 260 |
-
button_frame, text="Ready", font=("Arial", 10), text_color="gray"
|
| 261 |
-
)
|
| 262 |
-
self.status_label.pack(side="right", padx=5)
|
| 263 |
-
|
| 264 |
-
def _browse_segments(self):
|
| 265 |
-
"""Open folder browser for segments."""
|
| 266 |
-
folder = filedialog.askdirectory(title="Select Segments Folder")
|
| 267 |
-
if folder:
|
| 268 |
-
self.segments_entry.delete(0, "end")
|
| 269 |
-
self.segments_entry.insert(0, folder)
|
| 270 |
-
|
| 271 |
-
def _browse_csv(self):
|
| 272 |
-
"""Open file browser for CSV."""
|
| 273 |
-
file = filedialog.askopenfilename(
|
| 274 |
-
title="Select CSV File",
|
| 275 |
-
filetypes=[("CSV files", "*.csv"), ("All files", "*.*")]
|
| 276 |
-
)
|
| 277 |
-
if file:
|
| 278 |
-
self.csv_entry.delete(0, "end")
|
| 279 |
-
self.csv_entry.insert(0, file)
|
| 280 |
-
|
| 281 |
-
def _log(self, message: str):
|
| 282 |
-
"""Add message to log queue (thread-safe)."""
|
| 283 |
-
self.log_queue.put(message)
|
| 284 |
-
|
| 285 |
-
def _process_log_queue(self):
|
| 286 |
-
"""Process queued log messages and update textbox."""
|
| 287 |
-
drained_messages = []
|
| 288 |
-
try:
|
| 289 |
-
while True:
|
| 290 |
-
drained_messages.append(self.log_queue.get_nowait())
|
| 291 |
-
except queue.Empty:
|
| 292 |
-
pass
|
| 293 |
-
|
| 294 |
-
if drained_messages:
|
| 295 |
-
self.log_textbox.configure(state="normal")
|
| 296 |
-
for message in drained_messages:
|
| 297 |
-
dynamic = message.startswith("[Progress]") or message.startswith("[Status]")
|
| 298 |
-
|
| 299 |
-
if dynamic:
|
| 300 |
-
if self._dynamic_line_active and self._dynamic_line_start is not None:
|
| 301 |
-
try:
|
| 302 |
-
self.log_textbox.delete(self._dynamic_line_start, "end")
|
| 303 |
-
except Exception:
|
| 304 |
-
pass
|
| 305 |
-
self.log_textbox.insert("end", message)
|
| 306 |
-
self._dynamic_line_active = True
|
| 307 |
-
self._dynamic_line_start = self.log_textbox.index("end-1c linestart")
|
| 308 |
-
else:
|
| 309 |
-
if self._dynamic_line_active:
|
| 310 |
-
self.log_textbox.insert("end", "\n")
|
| 311 |
-
self._dynamic_line_active = False
|
| 312 |
-
self._dynamic_line_start = None
|
| 313 |
-
self.log_textbox.insert("end", message + "\n")
|
| 314 |
-
|
| 315 |
-
self.log_textbox.see("end")
|
| 316 |
-
self.log_textbox.configure(state="disabled")
|
| 317 |
-
|
| 318 |
-
# Schedule next check
|
| 319 |
-
self.after(100, self._process_log_queue)
|
| 320 |
-
|
| 321 |
-
def _clear_log(self):
|
| 322 |
-
"""Clear the log textbox."""
|
| 323 |
-
self.log_textbox.configure(state="normal")
|
| 324 |
-
self.log_textbox.delete("1.0", "end")
|
| 325 |
-
self.log_textbox.configure(state="disabled")
|
| 326 |
-
self._dynamic_line_active = False
|
| 327 |
-
self._dynamic_line_start = None
|
| 328 |
-
|
| 329 |
-
def _validate_inputs(self) -> tuple[bool, str]:
|
| 330 |
-
"""Validate all input fields."""
|
| 331 |
-
repo_id = self.repo_id_entry.get().strip()
|
| 332 |
-
token = self.token_entry.get().strip()
|
| 333 |
-
segments = self.segments_entry.get().strip()
|
| 334 |
-
|
| 335 |
-
if not repo_id:
|
| 336 |
-
return False, "Repository ID is required (format: username/repo-name)"
|
| 337 |
-
if not token:
|
| 338 |
-
return False, "Hugging Face token is required"
|
| 339 |
-
if not segments:
|
| 340 |
-
return False, "Segments folder is required"
|
| 341 |
-
if not Path(segments).is_dir():
|
| 342 |
-
return False, f"Segments folder does not exist: {segments}"
|
| 343 |
-
|
| 344 |
-
csv_path = self.csv_entry.get().strip()
|
| 345 |
-
if csv_path and not Path(csv_path).is_file():
|
| 346 |
-
return False, f"CSV file does not exist: {csv_path}"
|
| 347 |
-
|
| 348 |
-
return True, "OK"
|
| 349 |
-
|
| 350 |
-
def _start_upload(self):
|
| 351 |
-
"""Start the upload process in a separate thread."""
|
| 352 |
-
if self.uploading:
|
| 353 |
-
messagebox.showwarning("Warning", "Upload already in progress")
|
| 354 |
-
return
|
| 355 |
-
|
| 356 |
-
valid, msg = self._validate_inputs()
|
| 357 |
-
if not valid:
|
| 358 |
-
messagebox.showerror("Validation Error", msg)
|
| 359 |
-
return
|
| 360 |
-
|
| 361 |
-
# Build config
|
| 362 |
-
config = {
|
| 363 |
-
"repo_id": self.repo_id_entry.get().strip(),
|
| 364 |
-
"hf_token": self.token_entry.get().strip(),
|
| 365 |
-
"segments": self.segments_entry.get().strip(),
|
| 366 |
-
"csv": self.csv_entry.get().strip() or None,
|
| 367 |
-
"private": self.private_var.get(),
|
| 368 |
-
"resume": self.resume_var.get(),
|
| 369 |
-
"verify_remote": self.verify_var.get(),
|
| 370 |
-
"verify_etag": self.verify_var.get(),
|
| 371 |
-
"rate_limit_aware": self.rate_limit_var.get(),
|
| 372 |
-
"max_workers": int(self.workers_combobox.get()),
|
| 373 |
-
"upload_attempts": 5,
|
| 374 |
-
"retry_backoff": 10.0,
|
| 375 |
-
"stop_event": self.stop_event,
|
| 376 |
-
"suppress_console_progress": True,
|
| 377 |
-
}
|
| 378 |
-
|
| 379 |
-
# Disable buttons
|
| 380 |
-
self.uploading = True
|
| 381 |
-
self.start_btn.configure(state="disabled")
|
| 382 |
-
self.stop_btn.configure(state="normal")
|
| 383 |
-
self.status_label.configure(text="Uploading...", text_color="orange")
|
| 384 |
-
|
| 385 |
-
# Clear log
|
| 386 |
-
self._clear_log()
|
| 387 |
-
self._log(f"Starting upload to {config['repo_id']}...")
|
| 388 |
-
self._log("")
|
| 389 |
-
|
| 390 |
-
# Start upload thread
|
| 391 |
-
# Clear any previous stop event and start the worker
|
| 392 |
-
self.stop_event.clear()
|
| 393 |
-
self.upload_thread = threading.Thread(target=self._run_upload, args=(config,), daemon=True)
|
| 394 |
-
self.upload_thread.start()
|
| 395 |
-
|
| 396 |
-
def _run_upload(self, config: dict):
|
| 397 |
-
"""Run the upload logic in a separate thread."""
|
| 398 |
-
try:
|
| 399 |
-
run_upload_logic(config, self._log)
|
| 400 |
-
self.log_queue.put("✅ Upload completed successfully!")
|
| 401 |
-
self.after(0, self._upload_finished, True)
|
| 402 |
-
except Exception as e:
|
| 403 |
-
msg = str(e)
|
| 404 |
-
if "cancel" in msg.lower():
|
| 405 |
-
self.log_queue.put("⚠️ Upload cancelled by user")
|
| 406 |
-
self.after(0, self._upload_finished, False)
|
| 407 |
-
else:
|
| 408 |
-
self.log_queue.put(f"❌ Upload failed: {msg}")
|
| 409 |
-
self.after(0, self._upload_finished, False)
|
| 410 |
-
|
| 411 |
-
def _upload_finished(self, success: bool):
|
| 412 |
-
"""Called when upload finishes (from main thread)."""
|
| 413 |
-
self.uploading = False
|
| 414 |
-
self.start_btn.configure(state="normal")
|
| 415 |
-
self.stop_btn.configure(state="disabled")
|
| 416 |
-
|
| 417 |
-
if success:
|
| 418 |
-
self.status_label.configure(text="✅ Complete", text_color="green")
|
| 419 |
-
messagebox.showinfo("Success", "Upload completed successfully!")
|
| 420 |
-
else:
|
| 421 |
-
self.status_label.configure(text="❌ Failed", text_color="red")
|
| 422 |
-
messagebox.showerror("Error", "Upload failed. Check the log for details.")
|
| 423 |
-
|
| 424 |
-
def _stop_upload(self):
|
| 425 |
-
"""Stop the upload (note: this is a basic implementation)."""
|
| 426 |
-
if not self.uploading:
|
| 427 |
-
return
|
| 428 |
-
# Signal cancellation to worker
|
| 429 |
-
self.stop_event.set()
|
| 430 |
-
self._log("[Control] Stop requested; attempting to cancel upload...")
|
| 431 |
-
self.status_label.configure(text="Cancelling...", text_color="orange")
|
| 432 |
-
# Disable stop button to avoid repeated clicks
|
| 433 |
-
self.stop_btn.configure(state="disabled")
|
| 434 |
-
|
| 435 |
-
|
| 436 |
-
def main():
|
| 437 |
-
"""Entry point for the GUI application."""
|
| 438 |
-
app = HFUploaderApp()
|
| 439 |
-
app.mainloop()
|
| 440 |
-
|
| 441 |
-
|
| 442 |
-
if __name__ == "__main__":
|
| 443 |
-
main()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/progress_bar_utils.py
DELETED
|
@@ -1,355 +0,0 @@
|
|
| 1 |
-
#!/usr/bin/env python3
|
| 2 |
-
"""
|
| 3 |
-
Progress bar management for HF uploads.
|
| 4 |
-
|
| 5 |
-
Replaces verbose per-file logs with a clean progress bar and filters
|
| 6 |
-
trivial messages while preserving errors and warnings.
|
| 7 |
-
"""
|
| 8 |
-
|
| 9 |
-
import re
|
| 10 |
-
import io
|
| 11 |
-
from typing import Callable, Optional
|
| 12 |
-
from pathlib import Path
|
| 13 |
-
|
| 14 |
-
try:
|
| 15 |
-
from tqdm import tqdm
|
| 16 |
-
except ImportError:
|
| 17 |
-
tqdm = None
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
class ProgressFilter(io.TextIOBase):
|
| 21 |
-
"""
|
| 22 |
-
Filters upload_large_folder() output to show only a progress bar.
|
| 23 |
-
|
| 24 |
-
- Captures "Uploaded X of Y files" patterns
|
| 25 |
-
- Updates tqdm progress bar
|
| 26 |
-
- Filters trivial per-file logs
|
| 27 |
-
- Preserves error/warning messages
|
| 28 |
-
|
| 29 |
-
Usage:
|
| 30 |
-
pbar = tqdm(total=total_files, unit="file", desc="Uploading", leave=True)
|
| 31 |
-
filter = ProgressFilter(pbar, logger_callback)
|
| 32 |
-
|
| 33 |
-
with contextlib.redirect_stdout(filter):
|
| 34 |
-
api.upload_large_folder(...)
|
| 35 |
-
"""
|
| 36 |
-
|
| 37 |
-
def __init__(self, pbar: 'tqdm', logger: Callable[[str], None]):
|
| 38 |
-
"""
|
| 39 |
-
Args:
|
| 40 |
-
pbar: tqdm progress bar instance
|
| 41 |
-
logger: Callback function for important messages (errors, warnings)
|
| 42 |
-
"""
|
| 43 |
-
super().__init__()
|
| 44 |
-
self.pbar = pbar
|
| 45 |
-
self.logger = logger
|
| 46 |
-
self._buffer = ""
|
| 47 |
-
self.last_reported_files = 0
|
| 48 |
-
self.last_reported_bytes = 0
|
| 49 |
-
self.last_metadata_files = 0
|
| 50 |
-
self.last_gui_progress_pct = -1.0
|
| 51 |
-
self.last_gui_metadata_pct = -1.0
|
| 52 |
-
|
| 53 |
-
def write(self, s: str) -> int:
|
| 54 |
-
"""Process output line by line."""
|
| 55 |
-
if not s:
|
| 56 |
-
return 0
|
| 57 |
-
|
| 58 |
-
self._buffer += s
|
| 59 |
-
|
| 60 |
-
# Process complete lines
|
| 61 |
-
while "\n" in self._buffer:
|
| 62 |
-
line, self._buffer = self._buffer.split("\n", 1)
|
| 63 |
-
self._process_line(line)
|
| 64 |
-
|
| 65 |
-
return len(s)
|
| 66 |
-
|
| 67 |
-
def flush(self) -> None:
|
| 68 |
-
"""Flush any remaining buffered content."""
|
| 69 |
-
if self._buffer.strip():
|
| 70 |
-
self._process_line(self._buffer)
|
| 71 |
-
self._buffer = ""
|
| 72 |
-
|
| 73 |
-
def _process_line(self, line: str) -> None:
|
| 74 |
-
"""Process a single line of output."""
|
| 75 |
-
if not line.strip():
|
| 76 |
-
return
|
| 77 |
-
|
| 78 |
-
# Strip common ANSI cursor/control sequences emitted by progress renderers
|
| 79 |
-
line = re.sub(r'\x1b\[[0-9;?]*[A-Za-z]', '', line)
|
| 80 |
-
if not line.strip():
|
| 81 |
-
return
|
| 82 |
-
|
| 83 |
-
# Try to extract progress: "Uploaded X of Y files"
|
| 84 |
-
progress_match = re.search(
|
| 85 |
-
r'Uploaded\s+(\d+)\s+of\s+(\d+)\s+files',
|
| 86 |
-
line,
|
| 87 |
-
re.IGNORECASE
|
| 88 |
-
)
|
| 89 |
-
|
| 90 |
-
if progress_match:
|
| 91 |
-
uploaded = int(progress_match.group(1))
|
| 92 |
-
total = int(progress_match.group(2))
|
| 93 |
-
|
| 94 |
-
# Update progress bar if new files since last report
|
| 95 |
-
if uploaded > self.last_reported_files:
|
| 96 |
-
increment = uploaded - self.last_reported_files
|
| 97 |
-
if self.pbar:
|
| 98 |
-
try:
|
| 99 |
-
self.pbar.update(increment)
|
| 100 |
-
except Exception:
|
| 101 |
-
pass
|
| 102 |
-
self.last_reported_files = uploaded
|
| 103 |
-
|
| 104 |
-
# Update description with percentage
|
| 105 |
-
if total > 0:
|
| 106 |
-
pct = (uploaded / total) * 100
|
| 107 |
-
# If we have a console tqdm bar, update it; otherwise emit concise progress to logger
|
| 108 |
-
if self.pbar:
|
| 109 |
-
try:
|
| 110 |
-
self.pbar.set_description(f"Uploading ({pct:.1f}%)")
|
| 111 |
-
except Exception:
|
| 112 |
-
pass
|
| 113 |
-
else:
|
| 114 |
-
if pct >= 100.0 or (pct - self.last_gui_progress_pct) >= 0.5:
|
| 115 |
-
self.last_gui_progress_pct = pct
|
| 116 |
-
try:
|
| 117 |
-
self.logger(f"[Progress] Uploading ({pct:.1f}%) {uploaded}/{total} files")
|
| 118 |
-
except Exception:
|
| 119 |
-
pass
|
| 120 |
-
|
| 121 |
-
return # Don't log this line
|
| 122 |
-
|
| 123 |
-
# HF metadata recovery progress: keep only a single dynamic line in GUI
|
| 124 |
-
metadata_match = re.search(
|
| 125 |
-
r'Recovering\s+from\s+metadata\s+files:.*?(\d+)\/(\d+)',
|
| 126 |
-
line,
|
| 127 |
-
re.IGNORECASE,
|
| 128 |
-
)
|
| 129 |
-
if metadata_match:
|
| 130 |
-
current = int(metadata_match.group(1))
|
| 131 |
-
total = int(metadata_match.group(2))
|
| 132 |
-
if current >= self.last_metadata_files:
|
| 133 |
-
self.last_metadata_files = current
|
| 134 |
-
pct = (current / total) * 100 if total > 0 else 0.0
|
| 135 |
-
if pct >= 100.0 or (pct - self.last_gui_metadata_pct) >= 0.5:
|
| 136 |
-
self.last_gui_metadata_pct = pct
|
| 137 |
-
self.logger(f"[Progress] Metadata recovery ({pct:.1f}%) {current}/{total}")
|
| 138 |
-
return
|
| 139 |
-
|
| 140 |
-
# HF periodic report blocks are useful but very noisy in GUI: compress to one status line.
|
| 141 |
-
files_report = re.search(
|
| 142 |
-
r'Files:\s+hashed\s+(\d+)\/(\d+).*?committed:\s+(\d+)\/(\d+)',
|
| 143 |
-
line,
|
| 144 |
-
re.IGNORECASE,
|
| 145 |
-
)
|
| 146 |
-
if files_report:
|
| 147 |
-
hashed = int(files_report.group(1))
|
| 148 |
-
total = int(files_report.group(2))
|
| 149 |
-
committed = int(files_report.group(3))
|
| 150 |
-
self.logger(f"[Status] Hashed {hashed}/{total} | Committed {committed}/{total}")
|
| 151 |
-
return
|
| 152 |
-
|
| 153 |
-
# Ignore report separators and worker summary lines
|
| 154 |
-
if re.search(r'^-+\s+\d{4}-\d{2}-\d{2}|^Workers:|^-{10,}', line, re.IGNORECASE):
|
| 155 |
-
return
|
| 156 |
-
|
| 157 |
-
# Check for bytes uploaded pattern (optional)
|
| 158 |
-
bytes_match = re.search(
|
| 159 |
-
r'Uploaded\s+([\d.]+\s*[KMGT]?B)',
|
| 160 |
-
line,
|
| 161 |
-
re.IGNORECASE
|
| 162 |
-
)
|
| 163 |
-
|
| 164 |
-
# Filter trivial logs (per-file progress)
|
| 165 |
-
trivial_patterns = [
|
| 166 |
-
r'^\s*\[.*\]\s+Uploading.*\.wav', # "[...] Uploading file.wav"
|
| 167 |
-
r'^\s*Uploading\s+file.*to', # "Uploading file to..."
|
| 168 |
-
r'^\s*\d+%', # "50% uploaded"
|
| 169 |
-
r'^\s*[├└┌┐]', # Tree characters
|
| 170 |
-
r'Connection pooling', # Connection messages
|
| 171 |
-
r'Retry-After', # Rate limit header
|
| 172 |
-
r'Commit message', # Commit messages
|
| 173 |
-
r'^\s*100%', # Final percentage
|
| 174 |
-
r'^\s*\|', # progress bar pipes
|
| 175 |
-
r'\[A|\[K', # escaped cursor controls
|
| 176 |
-
]
|
| 177 |
-
|
| 178 |
-
is_trivial = any(re.search(pattern, line, re.IGNORECASE) for pattern in trivial_patterns)
|
| 179 |
-
|
| 180 |
-
if is_trivial:
|
| 181 |
-
return # Suppress trivial logs
|
| 182 |
-
|
| 183 |
-
# Log important messages (errors, warnings, etc.)
|
| 184 |
-
error_patterns = [
|
| 185 |
-
r'error|exception|failed|traceback',
|
| 186 |
-
r'warning|warn',
|
| 187 |
-
r'timeout|504|500|503|429|too many',
|
| 188 |
-
r'refused|connection reset|broken pipe',
|
| 189 |
-
]
|
| 190 |
-
|
| 191 |
-
is_important = any(
|
| 192 |
-
re.search(pattern, line, re.IGNORECASE)
|
| 193 |
-
for pattern in error_patterns
|
| 194 |
-
)
|
| 195 |
-
|
| 196 |
-
if is_important or not any(
|
| 197 |
-
re.search(p, line, re.IGNORECASE)
|
| 198 |
-
for p in trivial_patterns + [
|
| 199 |
-
r'progress',
|
| 200 |
-
r'mb/s|kb/s',
|
| 201 |
-
r'elapsed',
|
| 202 |
-
]
|
| 203 |
-
):
|
| 204 |
-
# Log important messages or unknowns that aren't clearly trivial
|
| 205 |
-
if len(line) > 3: # Skip very short lines
|
| 206 |
-
self.logger(f"[HF] {line.strip()}")
|
| 207 |
-
|
| 208 |
-
|
| 209 |
-
class FilesProgressBar:
|
| 210 |
-
"""
|
| 211 |
-
Manages a clean progress bar for file uploads.
|
| 212 |
-
|
| 213 |
-
- Shows: "Uploading [████████..] 1,234 / 5,000 files | ETA: 2m 30s"
|
| 214 |
-
- Updates when files complete
|
| 215 |
-
- Thread-safe
|
| 216 |
-
|
| 217 |
-
Usage (for per-file upload fallback):
|
| 218 |
-
bar = FilesProgressBar(total_files=5000, logger=print)
|
| 219 |
-
for file in files:
|
| 220 |
-
try:
|
| 221 |
-
upload_file(file)
|
| 222 |
-
bar.update_file(1)
|
| 223 |
-
except Exception as e:
|
| 224 |
-
bar.update_file_error(1, str(e))
|
| 225 |
-
"""
|
| 226 |
-
|
| 227 |
-
def __init__(self, total_files: int, logger: Callable[[str], None], suppress_console: bool = False):
|
| 228 |
-
"""
|
| 229 |
-
Args:
|
| 230 |
-
total_files: Total number of files to upload
|
| 231 |
-
logger: Callback function for logging
|
| 232 |
-
"""
|
| 233 |
-
self.total_files = total_files
|
| 234 |
-
self.logger = logger
|
| 235 |
-
self.uploaded = 0
|
| 236 |
-
self.errors = []
|
| 237 |
-
self.pbar = None
|
| 238 |
-
self.suppress_console = bool(suppress_console)
|
| 239 |
-
|
| 240 |
-
if not self.suppress_console and tqdm is not None:
|
| 241 |
-
try:
|
| 242 |
-
self.pbar = tqdm(
|
| 243 |
-
total=total_files,
|
| 244 |
-
unit="file",
|
| 245 |
-
unit_scale=False,
|
| 246 |
-
desc="Uploading",
|
| 247 |
-
leave=True,
|
| 248 |
-
ncols=80,
|
| 249 |
-
miniters=1,
|
| 250 |
-
)
|
| 251 |
-
except Exception:
|
| 252 |
-
self.pbar = None
|
| 253 |
-
|
| 254 |
-
def update_file(self, count: int = 1) -> None:
|
| 255 |
-
"""Mark files as successfully uploaded."""
|
| 256 |
-
self.uploaded += count
|
| 257 |
-
if self.pbar:
|
| 258 |
-
try:
|
| 259 |
-
self.pbar.update(count)
|
| 260 |
-
pct = (self.uploaded / self.total_files) * 100 if self.total_files > 0 else 0
|
| 261 |
-
self.pbar.set_description(f"Uploading ({pct:.1f}%)")
|
| 262 |
-
except Exception:
|
| 263 |
-
pass
|
| 264 |
-
else:
|
| 265 |
-
# Emit concise progress to logger for GUI mode
|
| 266 |
-
try:
|
| 267 |
-
pct = (self.uploaded / self.total_files) * 100 if self.total_files > 0 else 0
|
| 268 |
-
if pct >= 100.0 or pct == 0.0 or (pct - getattr(self, "_last_gui_file_pct", -1.0)) >= 0.5:
|
| 269 |
-
self._last_gui_file_pct = pct
|
| 270 |
-
self.logger(f"[Progress] Uploading ({pct:.1f}%) {self.uploaded}/{self.total_files} files")
|
| 271 |
-
except Exception:
|
| 272 |
-
pass
|
| 273 |
-
|
| 274 |
-
def update_file_error(self, count: int, error_msg: str) -> None:
|
| 275 |
-
"""Mark files as failed with error message."""
|
| 276 |
-
self.errors.append(error_msg)
|
| 277 |
-
# Still count as "processed" (though failed) for progress
|
| 278 |
-
self.update_file(count)
|
| 279 |
-
|
| 280 |
-
def close(self) -> None:
|
| 281 |
-
"""Close the progress bar."""
|
| 282 |
-
if self.pbar:
|
| 283 |
-
try:
|
| 284 |
-
self.pbar.close()
|
| 285 |
-
except Exception:
|
| 286 |
-
pass
|
| 287 |
-
|
| 288 |
-
def log_errors(self) -> None:
|
| 289 |
-
"""Print all collected errors/warnings below progress bar."""
|
| 290 |
-
if not self.errors:
|
| 291 |
-
return
|
| 292 |
-
|
| 293 |
-
self.logger("")
|
| 294 |
-
self.logger("=" * 70)
|
| 295 |
-
self.logger(f"Upload completed with {len(self.errors)} error(s):")
|
| 296 |
-
self.logger("=" * 70)
|
| 297 |
-
|
| 298 |
-
for error in self.errors:
|
| 299 |
-
self.logger(f" ⚠️ {error}")
|
| 300 |
-
|
| 301 |
-
self.logger("=" * 70)
|
| 302 |
-
|
| 303 |
-
|
| 304 |
-
class BytesProgressBar:
|
| 305 |
-
"""
|
| 306 |
-
Progress bar for byte-based uploads (alternative metric).
|
| 307 |
-
|
| 308 |
-
Usage:
|
| 309 |
-
bar = BytesProgressBar(total_bytes=5_000_000, logger=print)
|
| 310 |
-
bar.update_bytes(1_000_000) # Update with bytes uploaded
|
| 311 |
-
"""
|
| 312 |
-
|
| 313 |
-
def __init__(self, total_bytes: int, logger: Callable[[str], None]):
|
| 314 |
-
"""
|
| 315 |
-
Args:
|
| 316 |
-
total_bytes: Total bytes to upload
|
| 317 |
-
logger: Callback function for logging
|
| 318 |
-
"""
|
| 319 |
-
self.total_bytes = total_bytes
|
| 320 |
-
self.logger = logger
|
| 321 |
-
self.uploaded_bytes = 0
|
| 322 |
-
self.pbar = None
|
| 323 |
-
|
| 324 |
-
if tqdm is not None:
|
| 325 |
-
try:
|
| 326 |
-
self.pbar = tqdm(
|
| 327 |
-
total=total_bytes,
|
| 328 |
-
unit="B",
|
| 329 |
-
unit_scale=True,
|
| 330 |
-
unit_divisor=1024,
|
| 331 |
-
desc="Uploading",
|
| 332 |
-
leave=True,
|
| 333 |
-
ncols=80,
|
| 334 |
-
)
|
| 335 |
-
except Exception:
|
| 336 |
-
pass
|
| 337 |
-
|
| 338 |
-
def update_bytes(self, byte_count: int) -> None:
|
| 339 |
-
"""Update progress with bytes uploaded."""
|
| 340 |
-
self.uploaded_bytes += byte_count
|
| 341 |
-
if self.pbar:
|
| 342 |
-
try:
|
| 343 |
-
self.pbar.update(byte_count)
|
| 344 |
-
pct = (self.uploaded_bytes / self.total_bytes) * 100 if self.total_bytes > 0 else 0
|
| 345 |
-
self.pbar.set_description(f"Uploading ({pct:.1f}%)")
|
| 346 |
-
except Exception:
|
| 347 |
-
pass
|
| 348 |
-
|
| 349 |
-
def close(self) -> None:
|
| 350 |
-
"""Close the progress bar."""
|
| 351 |
-
if self.pbar:
|
| 352 |
-
try:
|
| 353 |
-
self.pbar.close()
|
| 354 |
-
except Exception:
|
| 355 |
-
pass
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/requirements.txt
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
huggingface_hub==0.28.1
|
| 2 |
-
requests>=2.28
|
| 3 |
-
tqdm>=4.65
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/requirements_gui.txt
DELETED
|
@@ -1,10 +0,0 @@
|
|
| 1 |
-
# Requirements for the GUI version of HF Bulk Uploader
|
| 2 |
-
# These are IN ADDITION to the base requirements.txt
|
| 3 |
-
|
| 4 |
-
# GUI Framework
|
| 5 |
-
customtkinter==5.2.2
|
| 6 |
-
|
| 7 |
-
# Base requirements (should already be installed)
|
| 8 |
-
# huggingface-hub>=0.19.0
|
| 9 |
-
# requests>=2.31.0
|
| 10 |
-
# tqdm>=4.65.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/resilience_utils.py
DELETED
|
@@ -1,268 +0,0 @@
|
|
| 1 |
-
#!/usr/bin/env python3
|
| 2 |
-
"""
|
| 3 |
-
Enhanced resilience utilities for HF uploads.
|
| 4 |
-
|
| 5 |
-
Adds robustness against:
|
| 6 |
-
- HTTP timeouts and network interruptions
|
| 7 |
-
- Token expiration detection
|
| 8 |
-
- Improved retry strategies for different error types
|
| 9 |
-
- Monitoring/alerting hooks
|
| 10 |
-
"""
|
| 11 |
-
|
| 12 |
-
import time
|
| 13 |
-
from typing import Callable, Optional, Dict, Any
|
| 14 |
-
from pathlib import Path
|
| 15 |
-
from datetime import datetime, timedelta
|
| 16 |
-
import json
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
class TokenManager:
|
| 20 |
-
"""Manages HF token lifecycle and expiration."""
|
| 21 |
-
|
| 22 |
-
def __init__(self, token: str, warning_days_before_expiry: int = 7):
|
| 23 |
-
"""
|
| 24 |
-
Args:
|
| 25 |
-
token: HF API token
|
| 26 |
-
warning_days_before_expiry: Days before expiry to warn user
|
| 27 |
-
"""
|
| 28 |
-
self.token = token
|
| 29 |
-
self.warning_days_before_expiry = warning_days_before_expiry
|
| 30 |
-
|
| 31 |
-
def check_token_expiry(self, logger: Callable[[str], None]) -> bool:
|
| 32 |
-
"""
|
| 33 |
-
Check if token is about to expire and log warning.
|
| 34 |
-
|
| 35 |
-
Note: HF tokens don't include expiry in the token itself.
|
| 36 |
-
This is a placeholder for user to manually set expiry date in config.
|
| 37 |
-
|
| 38 |
-
Returns:
|
| 39 |
-
True if token is safe to use, False if expiring soon
|
| 40 |
-
"""
|
| 41 |
-
# In practice, this would need to be set by the user in their config
|
| 42 |
-
# Example: --token-expiry-date "2026-06-15"
|
| 43 |
-
# For now, just warn to check manually
|
| 44 |
-
logger("[TokenManager] HF tokens do not self-report expiry.")
|
| 45 |
-
logger("[TokenManager] If you created your token >30 days ago, consider creating a new one.")
|
| 46 |
-
return True
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
class SmartRetryStrategy:
|
| 50 |
-
"""
|
| 51 |
-
Intelligent retry strategy that adapts to different error types.
|
| 52 |
-
|
| 53 |
-
Handles:
|
| 54 |
-
- 429 (Too Many Requests) - respect Retry-After header
|
| 55 |
-
- 504 (Gateway Timeout) - exponential backoff up to 5 min
|
| 56 |
-
- 503 (Service Unavailable) - slow backoff (GC detected)
|
| 57 |
-
- 500-series - exponential backoff
|
| 58 |
-
- Network errors - exponential backoff
|
| 59 |
-
"""
|
| 60 |
-
|
| 61 |
-
def __init__(self, max_retries: int = 5, initial_backoff: float = 2.0):
|
| 62 |
-
self.max_retries = max_retries
|
| 63 |
-
self.initial_backoff = initial_backoff
|
| 64 |
-
self.error_history: list[Dict[str, Any]] = []
|
| 65 |
-
|
| 66 |
-
def should_retry(self, error_response: Dict[str, Any]) -> bool:
|
| 67 |
-
"""Determine if error is retryable."""
|
| 68 |
-
status = error_response.get('status_code')
|
| 69 |
-
|
| 70 |
-
# Retryable status codes
|
| 71 |
-
retryable_statuses = [408, 429, 500, 502, 503, 504]
|
| 72 |
-
return status in retryable_statuses
|
| 73 |
-
|
| 74 |
-
def get_wait_time(self, error_response: Dict[str, Any], attempt: int) -> float:
|
| 75 |
-
"""
|
| 76 |
-
Calculate wait time based on error type and attempt number.
|
| 77 |
-
|
| 78 |
-
Returns:
|
| 79 |
-
Seconds to wait before retry
|
| 80 |
-
"""
|
| 81 |
-
status = error_response.get('status_code')
|
| 82 |
-
|
| 83 |
-
# Check for Retry-After header (HF may suggest wait time)
|
| 84 |
-
retry_after = error_response.get('retry_after_header')
|
| 85 |
-
if retry_after:
|
| 86 |
-
try:
|
| 87 |
-
return max(float(retry_after), self.initial_backoff * (2 ** (attempt - 1)))
|
| 88 |
-
except (ValueError, TypeError):
|
| 89 |
-
pass
|
| 90 |
-
|
| 91 |
-
# Status-specific backoff strategies
|
| 92 |
-
if status == 429:
|
| 93 |
-
# Rate limit: aggressive backoff
|
| 94 |
-
return min(300, self.initial_backoff * (4 ** attempt)) # Cap at 5 min
|
| 95 |
-
|
| 96 |
-
elif status == 503:
|
| 97 |
-
# Service unavailable (likely GC): slow backoff
|
| 98 |
-
return min(600, self.initial_backoff * (2 ** attempt)) # Cap at 10 min
|
| 99 |
-
|
| 100 |
-
elif status == 504:
|
| 101 |
-
# Gateway timeout: exponential backoff
|
| 102 |
-
return min(300, self.initial_backoff * (3 ** attempt)) # Cap at 5 min
|
| 103 |
-
|
| 104 |
-
elif status in [500, 502]:
|
| 105 |
-
# Server error: standard exponential backoff
|
| 106 |
-
return self.initial_backoff * (2 ** attempt)
|
| 107 |
-
|
| 108 |
-
else:
|
| 109 |
-
# Generic network error
|
| 110 |
-
return self.initial_backoff * (2 ** attempt)
|
| 111 |
-
|
| 112 |
-
def log_retry_decision(
|
| 113 |
-
self,
|
| 114 |
-
error_response: Dict[str, Any],
|
| 115 |
-
attempt: int,
|
| 116 |
-
wait_time: float,
|
| 117 |
-
logger: Callable[[str], None]
|
| 118 |
-
) -> None:
|
| 119 |
-
"""Log retry decision for monitoring."""
|
| 120 |
-
status = error_response.get('status_code')
|
| 121 |
-
error_msg = error_response.get('message', 'Unknown error')
|
| 122 |
-
|
| 123 |
-
logger(f"[SmartRetry] Attempt {attempt}/{self.max_retries} failed with status {status}")
|
| 124 |
-
logger(f"[SmartRetry] Error: {error_msg}")
|
| 125 |
-
logger(f"[SmartRetry] Waiting {wait_time:.0f}s before retry...")
|
| 126 |
-
|
| 127 |
-
# Store for historical analysis
|
| 128 |
-
self.error_history.append({
|
| 129 |
-
'timestamp': datetime.now().isoformat(),
|
| 130 |
-
'status': status,
|
| 131 |
-
'attempt': attempt,
|
| 132 |
-
'wait_time': wait_time
|
| 133 |
-
})
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
class NetworkResilience:
|
| 137 |
-
"""Handles network-specific issues."""
|
| 138 |
-
|
| 139 |
-
@staticmethod
|
| 140 |
-
def detect_slow_network(
|
| 141 |
-
bytes_uploaded: int,
|
| 142 |
-
elapsed_seconds: float,
|
| 143 |
-
threshold_kbps: float = 100.0
|
| 144 |
-
) -> bool:
|
| 145 |
-
"""
|
| 146 |
-
Detect if network is unusually slow.
|
| 147 |
-
|
| 148 |
-
Args:
|
| 149 |
-
bytes_uploaded: Total bytes successfully uploaded
|
| 150 |
-
elapsed_seconds: Time elapsed
|
| 151 |
-
threshold_kbps: Minimum expected speed (default 100 KB/s)
|
| 152 |
-
|
| 153 |
-
Returns:
|
| 154 |
-
True if network speed is below threshold
|
| 155 |
-
"""
|
| 156 |
-
if elapsed_seconds < 1:
|
| 157 |
-
return False
|
| 158 |
-
|
| 159 |
-
kbps = (bytes_uploaded / 1024) / elapsed_seconds
|
| 160 |
-
return kbps < threshold_kbps
|
| 161 |
-
|
| 162 |
-
@staticmethod
|
| 163 |
-
def estimate_remaining_time(
|
| 164 |
-
bytes_remaining: int,
|
| 165 |
-
current_speed_kbps: float
|
| 166 |
-
) -> float:
|
| 167 |
-
"""Estimate remaining upload time based on current speed."""
|
| 168 |
-
if current_speed_kbps <= 0:
|
| 169 |
-
return float('inf')
|
| 170 |
-
return (bytes_remaining / 1024) / current_speed_kbps
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
class UploadMonitor:
|
| 174 |
-
"""Monitors upload progress and detects anomalies."""
|
| 175 |
-
|
| 176 |
-
def __init__(self, checkpoint_file: Path, logger: Callable[[str], None]):
|
| 177 |
-
self.checkpoint_file = checkpoint_file
|
| 178 |
-
self.logger = logger
|
| 179 |
-
self.start_time = time.time()
|
| 180 |
-
self.last_checkpoint_time = self.start_time
|
| 181 |
-
self.checkpoint_interval_seconds = 300 # Every 5 min
|
| 182 |
-
|
| 183 |
-
def should_save_checkpoint(self) -> bool:
|
| 184 |
-
"""Determine if checkpoint should be saved based on time interval."""
|
| 185 |
-
elapsed = time.time() - self.last_checkpoint_time
|
| 186 |
-
return elapsed > self.checkpoint_interval_seconds
|
| 187 |
-
|
| 188 |
-
def record_checkpoint(self, uploaded_count: int, total_count: int) -> None:
|
| 189 |
-
"""Save checkpoint with upload progress."""
|
| 190 |
-
self.last_checkpoint_time = time.time()
|
| 191 |
-
elapsed = time.time() - self.start_time
|
| 192 |
-
|
| 193 |
-
progress = {
|
| 194 |
-
'timestamp': datetime.now().isoformat(),
|
| 195 |
-
'uploaded': uploaded_count,
|
| 196 |
-
'total': total_count,
|
| 197 |
-
'elapsed_seconds': elapsed,
|
| 198 |
-
'rate_files_per_hour': (uploaded_count / elapsed * 3600) if elapsed > 0 else 0,
|
| 199 |
-
'percent_complete': (uploaded_count / total_count * 100) if total_count > 0 else 0,
|
| 200 |
-
}
|
| 201 |
-
|
| 202 |
-
try:
|
| 203 |
-
with open(self.checkpoint_file, 'w') as f:
|
| 204 |
-
json.dump(progress, f, indent=2)
|
| 205 |
-
except Exception as e:
|
| 206 |
-
self.logger(f"[Monitor] Warning: Could not save checkpoint: {e}")
|
| 207 |
-
|
| 208 |
-
def estimate_time_remaining(self, uploaded_count: int, total_count: int) -> str:
|
| 209 |
-
"""Estimate remaining upload time."""
|
| 210 |
-
if uploaded_count == 0 or total_count == 0:
|
| 211 |
-
return "unknown"
|
| 212 |
-
|
| 213 |
-
elapsed = time.time() - self.start_time
|
| 214 |
-
rate_per_sec = uploaded_count / elapsed if elapsed > 0 else 0
|
| 215 |
-
remaining = total_count - uploaded_count
|
| 216 |
-
remaining_seconds = remaining / rate_per_sec if rate_per_sec > 0 else 0
|
| 217 |
-
|
| 218 |
-
hours = remaining_seconds / 3600
|
| 219 |
-
minutes = (remaining_seconds % 3600) / 60
|
| 220 |
-
|
| 221 |
-
return f"{hours:.1f}h {minutes:.0f}m"
|
| 222 |
-
|
| 223 |
-
def log_progress(self, uploaded_count: int, total_count: int) -> None:
|
| 224 |
-
"""Log current progress."""
|
| 225 |
-
elapsed = time.time() - self.start_time
|
| 226 |
-
rate = (uploaded_count / elapsed * 3600) if elapsed > 0 else 0
|
| 227 |
-
remaining = self.estimate_time_remaining(uploaded_count, total_count)
|
| 228 |
-
|
| 229 |
-
self.logger(f"[Progress] {uploaded_count}/{total_count} files ({uploaded_count/total_count*100:.1f}%)")
|
| 230 |
-
self.logger(f"[Progress] Rate: {rate:.0f} files/hour | ETA: {remaining}")
|
| 231 |
-
|
| 232 |
-
|
| 233 |
-
class GarbageCollectionDetector:
|
| 234 |
-
"""Detects when HF is doing garbage collection (repo locked state)."""
|
| 235 |
-
|
| 236 |
-
def __init__(self):
|
| 237 |
-
self.gc_indicators = []
|
| 238 |
-
|
| 239 |
-
def record_error(self, status_code: int, retry_after: Optional[str] = None) -> None:
|
| 240 |
-
"""Record error that might indicate GC."""
|
| 241 |
-
if status_code == 503 or (status_code == 429 and retry_after and int(retry_after) > 300):
|
| 242 |
-
self.gc_indicators.append({
|
| 243 |
-
'timestamp': datetime.now().isoformat(),
|
| 244 |
-
'status': status_code,
|
| 245 |
-
'retry_after': retry_after
|
| 246 |
-
})
|
| 247 |
-
|
| 248 |
-
def might_be_in_gc(self) -> bool:
|
| 249 |
-
"""Check if GC might be happening based on recent errors."""
|
| 250 |
-
# If we have 503 errors in the last few minutes, likely GC
|
| 251 |
-
if not self.gc_indicators:
|
| 252 |
-
return False
|
| 253 |
-
|
| 254 |
-
recent_errors = [
|
| 255 |
-
e for e in self.gc_indicators
|
| 256 |
-
if (datetime.now() - datetime.fromisoformat(e['timestamp'])).total_seconds() < 600
|
| 257 |
-
]
|
| 258 |
-
|
| 259 |
-
return len(recent_errors) >= 2
|
| 260 |
-
|
| 261 |
-
def suggest_action(self, logger: Callable[[str], None]) -> None:
|
| 262 |
-
"""Suggest action if GC detected."""
|
| 263 |
-
if self.might_be_in_gc():
|
| 264 |
-
logger("")
|
| 265 |
-
logger("[GCDetector] ⚠️ Possible HF garbage collection detected!")
|
| 266 |
-
logger("[GCDetector] Repository may be temporarily locked for optimization.")
|
| 267 |
-
logger("[GCDetector] Switching to slower retry strategy (waiting longer between attempts)...")
|
| 268 |
-
logger("")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/run_upload.ps1
DELETED
|
@@ -1,58 +0,0 @@
|
|
| 1 |
-
param(
|
| 2 |
-
[string]$HfUsername,
|
| 3 |
-
[string]$HfToken,
|
| 4 |
-
[string]$RepoName,
|
| 5 |
-
[string]$RepoId,
|
| 6 |
-
[Parameter(Mandatory = $true)]
|
| 7 |
-
[string]$Segments,
|
| 8 |
-
[string]$Csv,
|
| 9 |
-
[switch]$Private,
|
| 10 |
-
[switch]$Public,
|
| 11 |
-
[switch]$DryRun,
|
| 12 |
-
[switch]$Resume,
|
| 13 |
-
[switch]$VerifyRemote,
|
| 14 |
-
[switch]$VerifyEtag,
|
| 15 |
-
[string]$ProgressLog,
|
| 16 |
-
[switch]$ResumeOnly,
|
| 17 |
-
[string]$CheckpointDir,
|
| 18 |
-
[int]$CreateRepoAttempts = 3,
|
| 19 |
-
[int]$UploadAttempts = 3,
|
| 20 |
-
[double]$RetryBackoff = 5
|
| 21 |
-
)
|
| 22 |
-
|
| 23 |
-
$ErrorActionPreference = 'Stop'
|
| 24 |
-
|
| 25 |
-
$env:HF_HUB_ETAG_TIMEOUT = $env:HF_HUB_ETAG_TIMEOUT ?? '20'
|
| 26 |
-
$env:HF_HUB_DOWNLOAD_TIMEOUT = $env:HF_HUB_DOWNLOAD_TIMEOUT ?? '120'
|
| 27 |
-
$env:HF_XET_HIGH_PERFORMANCE = $env:HF_XET_HIGH_PERFORMANCE ?? '1'
|
| 28 |
-
|
| 29 |
-
$scriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
|
| 30 |
-
$pythonExe = Join-Path (Split-Path -Parent $scriptDir) '.venv\Scripts\python.exe'
|
| 31 |
-
$scriptPath = Join-Path $scriptDir 'upload_dataset.py'
|
| 32 |
-
|
| 33 |
-
if (-not (Test-Path $pythonExe)) {
|
| 34 |
-
throw "Python executable not found at: $pythonExe"
|
| 35 |
-
}
|
| 36 |
-
|
| 37 |
-
$argsList = @($scriptPath, '--segments', $Segments)
|
| 38 |
-
|
| 39 |
-
if ($HfUsername) { $argsList += @('--hf-username', $HfUsername) }
|
| 40 |
-
if ($HfToken) { $argsList += @('--hf-token', $HfToken) }
|
| 41 |
-
if ($RepoName) { $argsList += @('--repo-name', $RepoName) }
|
| 42 |
-
if ($RepoId) { $argsList += @('--repo-id', $RepoId) }
|
| 43 |
-
if ($Csv) { $argsList += @('--csv', $Csv) }
|
| 44 |
-
if ($Private) { $argsList += '--private' }
|
| 45 |
-
if ($Public) { $argsList += '--public' }
|
| 46 |
-
if ($DryRun) { $argsList += '--dry-run' }
|
| 47 |
-
if ($Resume) { $argsList += '--resume' }
|
| 48 |
-
if ($VerifyRemote) { $argsList += '--verify-remote' }
|
| 49 |
-
if ($CheckpointDir) { $argsList += @('--checkpoint-dir', $CheckpointDir) }
|
| 50 |
-
if ($VerifyEtag) { $argsList += '--verify-etag' }
|
| 51 |
-
if ($ProgressLog) { $argsList += @('--progress-log', $ProgressLog) }
|
| 52 |
-
if ($ResumeOnly) { $argsList += '--resume-only' }
|
| 53 |
-
|
| 54 |
-
$argsList += @('--create-repo-attempts', $CreateRepoAttempts.ToString())
|
| 55 |
-
$argsList += @('--upload-attempts', $UploadAttempts.ToString())
|
| 56 |
-
$argsList += @('--retry-backoff', $RetryBackoff.ToString([System.Globalization.CultureInfo]::InvariantCulture))
|
| 57 |
-
|
| 58 |
-
& $pythonExe $argsList
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/sharding_utils.py
DELETED
|
@@ -1,304 +0,0 @@
|
|
| 1 |
-
#!/usr/bin/env python3
|
| 2 |
-
"""
|
| 3 |
-
Transparent sharding utilities for BirdNET dataset uploads.
|
| 4 |
-
|
| 5 |
-
Handles the splitting of large species directories (>10k files) for HF compliance,
|
| 6 |
-
while keeping them transparent to validation code.
|
| 7 |
-
|
| 8 |
-
Sharding format:
|
| 9 |
-
Elaenia albiceps/ (original, <10k files)
|
| 10 |
-
Elaenia albiceps__shard_00/ (part 1, if original had >10k)
|
| 11 |
-
Elaenia albiceps__shard_01/ (part 2)
|
| 12 |
-
...
|
| 13 |
-
|
| 14 |
-
When reading in validation: transparently combines all shards into a single virtual directory.
|
| 15 |
-
"""
|
| 16 |
-
|
| 17 |
-
import os
|
| 18 |
-
from pathlib import Path
|
| 19 |
-
from typing import Dict, List, Tuple
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
SHARD_SEPARATOR = "__shard_"
|
| 23 |
-
MAX_FILES_PER_DIR = 9500 # Stay below 10k limit with safety margin
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
def is_shard_folder(folder_name: str) -> Tuple[bool, str]:
|
| 27 |
-
"""
|
| 28 |
-
Check if a folder name is a shard and extract the base species name.
|
| 29 |
-
|
| 30 |
-
Returns:
|
| 31 |
-
(is_shard: bool, base_species_name: str)
|
| 32 |
-
"""
|
| 33 |
-
if SHARD_SEPARATOR not in folder_name:
|
| 34 |
-
return False, folder_name
|
| 35 |
-
|
| 36 |
-
base = folder_name.split(SHARD_SEPARATOR)[0]
|
| 37 |
-
return True, base
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
def get_all_shards(base_species_path: Path) -> List[Path]:
|
| 41 |
-
"""
|
| 42 |
-
Get all shards for a species, including the base folder if it exists.
|
| 43 |
-
|
| 44 |
-
Args:
|
| 45 |
-
base_species_path: Path to the base species folder (e.g., Species Name/)
|
| 46 |
-
|
| 47 |
-
Returns:
|
| 48 |
-
List of all shard paths for this species, sorted by shard index
|
| 49 |
-
"""
|
| 50 |
-
parent = base_species_path.parent
|
| 51 |
-
base_name = base_species_path.name
|
| 52 |
-
|
| 53 |
-
shards = []
|
| 54 |
-
|
| 55 |
-
# Add the base folder if it exists and is not empty
|
| 56 |
-
if base_species_path.is_dir():
|
| 57 |
-
files = list(base_species_path.glob("*"))
|
| 58 |
-
if files:
|
| 59 |
-
shards.append(base_species_path)
|
| 60 |
-
|
| 61 |
-
# Find all shards (Elaenia albiceps__shard_00, __shard_01, etc.)
|
| 62 |
-
shard_pattern = f"{base_name}{SHARD_SEPARATOR}*"
|
| 63 |
-
shard_folders = sorted(parent.glob(shard_pattern))
|
| 64 |
-
|
| 65 |
-
shards.extend(shard_folders)
|
| 66 |
-
return shards
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
def iter_files_all_shards(base_species_path: Path):
|
| 70 |
-
"""
|
| 71 |
-
Generator that yields all files from a species and all its shards.
|
| 72 |
-
|
| 73 |
-
Args:
|
| 74 |
-
base_species_path: Path to the base species folder
|
| 75 |
-
|
| 76 |
-
Yields:
|
| 77 |
-
Tuple of (file_path, relative_path_from_base_species)
|
| 78 |
-
"""
|
| 79 |
-
shards = get_all_shards(base_species_path)
|
| 80 |
-
|
| 81 |
-
for shard_path in shards:
|
| 82 |
-
if not shard_path.is_dir():
|
| 83 |
-
continue
|
| 84 |
-
|
| 85 |
-
for file_path in shard_path.rglob("*"):
|
| 86 |
-
if file_path.is_file():
|
| 87 |
-
# Keep the relative structure from within the shard
|
| 88 |
-
rel_from_shard = file_path.relative_to(shard_path)
|
| 89 |
-
yield file_path, rel_from_shard
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
def should_shard_directory(directory_path: Path) -> bool:
|
| 93 |
-
"""
|
| 94 |
-
Check if a directory should be sharded (has >MAX_FILES_PER_DIR files).
|
| 95 |
-
|
| 96 |
-
Args:
|
| 97 |
-
directory_path: Path to the directory
|
| 98 |
-
|
| 99 |
-
Returns:
|
| 100 |
-
True if should be sharded
|
| 101 |
-
"""
|
| 102 |
-
if not directory_path.is_dir():
|
| 103 |
-
return False
|
| 104 |
-
|
| 105 |
-
count = sum(1 for _ in directory_path.rglob("*") if _.is_file())
|
| 106 |
-
return count > MAX_FILES_PER_DIR
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
def shard_directory(source_dir: Path, dest_parent: Path) -> List[Path]:
|
| 110 |
-
"""
|
| 111 |
-
Shard a large directory into multiple folders respecting the 10k file limit.
|
| 112 |
-
|
| 113 |
-
Args:
|
| 114 |
-
source_dir: Source directory to shard (e.g., "Elaenia albiceps/")
|
| 115 |
-
dest_parent: Destination parent folder (staging root)
|
| 116 |
-
|
| 117 |
-
Returns:
|
| 118 |
-
List of created shard paths (destination folders)
|
| 119 |
-
|
| 120 |
-
Example:
|
| 121 |
-
shards = shard_directory(
|
| 122 |
-
Path("Elaenia albiceps"),
|
| 123 |
-
Path("staging/audio")
|
| 124 |
-
)
|
| 125 |
-
# Creates:
|
| 126 |
-
# staging/audio/Elaenia albiceps__shard_00/
|
| 127 |
-
# staging/audio/Elaenia albiceps__shard_01/
|
| 128 |
-
# ...
|
| 129 |
-
"""
|
| 130 |
-
if not source_dir.is_dir():
|
| 131 |
-
raise ValueError(f"Source directory does not exist: {source_dir}")
|
| 132 |
-
|
| 133 |
-
dest_parent.mkdir(parents=True, exist_ok=True)
|
| 134 |
-
base_name = source_dir.name
|
| 135 |
-
|
| 136 |
-
# Collect all files
|
| 137 |
-
all_files = sorted([f for f in source_dir.rglob("*") if f.is_file()])
|
| 138 |
-
|
| 139 |
-
if not all_files:
|
| 140 |
-
return []
|
| 141 |
-
|
| 142 |
-
shards = []
|
| 143 |
-
current_shard_idx = 0
|
| 144 |
-
current_shard_count = 0
|
| 145 |
-
current_shard_dir = None
|
| 146 |
-
|
| 147 |
-
for file_path in all_files:
|
| 148 |
-
# Create new shard if needed
|
| 149 |
-
if current_shard_count >= MAX_FILES_PER_DIR or current_shard_dir is None:
|
| 150 |
-
shard_name = f"{base_name}{SHARD_SEPARATOR}{current_shard_idx:02d}"
|
| 151 |
-
current_shard_dir = dest_parent / shard_name
|
| 152 |
-
current_shard_dir.mkdir(parents=True, exist_ok=True)
|
| 153 |
-
shards.append(current_shard_dir)
|
| 154 |
-
current_shard_count = 0
|
| 155 |
-
current_shard_idx += 1
|
| 156 |
-
|
| 157 |
-
# Copy file preserving directory structure
|
| 158 |
-
rel_path = file_path.relative_to(source_dir)
|
| 159 |
-
dest_file = current_shard_dir / rel_path
|
| 160 |
-
dest_file.parent.mkdir(parents=True, exist_ok=True)
|
| 161 |
-
|
| 162 |
-
# Create symlink or hardlink instead of copying to save space
|
| 163 |
-
if os.name == "nt":
|
| 164 |
-
# Windows: use hardlink
|
| 165 |
-
os.link(str(file_path), str(dest_file))
|
| 166 |
-
else:
|
| 167 |
-
# Unix: use symlink
|
| 168 |
-
os.symlink(file_path, dest_file)
|
| 169 |
-
|
| 170 |
-
current_shard_count += 1
|
| 171 |
-
|
| 172 |
-
return shards
|
| 173 |
-
|
| 174 |
-
|
| 175 |
-
def get_species_map(staging_audio_dir: Path) -> Dict[str, List[Path]]:
|
| 176 |
-
"""
|
| 177 |
-
Build a map of all species and their shards in the staging directory.
|
| 178 |
-
|
| 179 |
-
Args:
|
| 180 |
-
staging_audio_dir: Path to the staging audio directory (audio/)
|
| 181 |
-
|
| 182 |
-
Returns:
|
| 183 |
-
Dict mapping base species name -> list of shard paths
|
| 184 |
-
|
| 185 |
-
Example:
|
| 186 |
-
{
|
| 187 |
-
"Elaenia albiceps": [
|
| 188 |
-
Path("staging/audio/Elaenia albiceps__shard_00"),
|
| 189 |
-
Path("staging/audio/Elaenia albiceps__shard_01"),
|
| 190 |
-
],
|
| 191 |
-
"Other species": [
|
| 192 |
-
Path("staging/audio/Other species"),
|
| 193 |
-
]
|
| 194 |
-
}
|
| 195 |
-
"""
|
| 196 |
-
if not staging_audio_dir.is_dir():
|
| 197 |
-
return {}
|
| 198 |
-
|
| 199 |
-
species_map = {}
|
| 200 |
-
processed = set()
|
| 201 |
-
|
| 202 |
-
for item in sorted(staging_audio_dir.iterdir()):
|
| 203 |
-
if item.name in processed:
|
| 204 |
-
continue
|
| 205 |
-
|
| 206 |
-
is_shard, base_name = is_shard_folder(item.name)
|
| 207 |
-
|
| 208 |
-
if is_shard:
|
| 209 |
-
# This is a shard, process its base species
|
| 210 |
-
if base_name not in species_map:
|
| 211 |
-
species_map[base_name] = get_all_shards(staging_audio_dir / base_name)
|
| 212 |
-
processed.add(item.name)
|
| 213 |
-
elif item.is_dir():
|
| 214 |
-
# This is a regular species folder (non-shard)
|
| 215 |
-
if base_name not in species_map:
|
| 216 |
-
species_map[base_name] = get_all_shards(item)
|
| 217 |
-
processed.add(base_name)
|
| 218 |
-
|
| 219 |
-
return species_map
|
| 220 |
-
|
| 221 |
-
|
| 222 |
-
# ===== Validation Helper =====
|
| 223 |
-
|
| 224 |
-
class TransparentSpeciesReader:
|
| 225 |
-
"""
|
| 226 |
-
Helper for reading sharded species in validation code.
|
| 227 |
-
|
| 228 |
-
Transparently combines all shards of a species into a single iterable view.
|
| 229 |
-
|
| 230 |
-
Usage:
|
| 231 |
-
reader = TransparentSpeciesReader(Path("dataset/audio"))
|
| 232 |
-
for species_name, files in reader.iter_species():
|
| 233 |
-
for file_path in files:
|
| 234 |
-
# Process file
|
| 235 |
-
pass
|
| 236 |
-
"""
|
| 237 |
-
|
| 238 |
-
def __init__(self, dataset_audio_dir: Path):
|
| 239 |
-
"""
|
| 240 |
-
Args:
|
| 241 |
-
dataset_audio_dir: Path to the dataset's audio directory
|
| 242 |
-
"""
|
| 243 |
-
self.audio_dir = Path(dataset_audio_dir)
|
| 244 |
-
self._species_cache = None
|
| 245 |
-
|
| 246 |
-
def get_species_list(self) -> List[str]:
|
| 247 |
-
"""Get list of all species (base names, deduped)."""
|
| 248 |
-
species_map = get_species_map(self.audio_dir)
|
| 249 |
-
return sorted(species_map.keys())
|
| 250 |
-
|
| 251 |
-
def get_files_for_species(self, species_name: str) -> List[Path]:
|
| 252 |
-
"""
|
| 253 |
-
Get all files for a species, combining all shards.
|
| 254 |
-
|
| 255 |
-
Args:
|
| 256 |
-
species_name: Base species name (e.g., "Elaenia albiceps")
|
| 257 |
-
|
| 258 |
-
Returns:
|
| 259 |
-
List of all file paths from this species (all shards combined)
|
| 260 |
-
"""
|
| 261 |
-
species_map = get_species_map(self.audio_dir)
|
| 262 |
-
if species_name not in species_map:
|
| 263 |
-
return []
|
| 264 |
-
|
| 265 |
-
files = []
|
| 266 |
-
for shard_dir in species_map[species_name]:
|
| 267 |
-
shard_files = sorted([f for f in shard_dir.rglob("*") if f.is_file()])
|
| 268 |
-
files.extend(shard_files)
|
| 269 |
-
|
| 270 |
-
return sorted(files)
|
| 271 |
-
|
| 272 |
-
def iter_species(self):
|
| 273 |
-
"""
|
| 274 |
-
Iterate over all species and their files.
|
| 275 |
-
|
| 276 |
-
Yields:
|
| 277 |
-
Tuple of (species_name: str, files: List[Path])
|
| 278 |
-
"""
|
| 279 |
-
species_map = get_species_map(self.audio_dir)
|
| 280 |
-
|
| 281 |
-
for species_name in sorted(species_map.keys()):
|
| 282 |
-
files = self.get_files_for_species(species_name)
|
| 283 |
-
yield species_name, files
|
| 284 |
-
|
| 285 |
-
def iter_species_with_metadata(self):
|
| 286 |
-
"""
|
| 287 |
-
Iterate over species with metadata (total files, shard count).
|
| 288 |
-
|
| 289 |
-
Yields:
|
| 290 |
-
Tuple of (species_name: str, files: List[Path], metadata: dict)
|
| 291 |
-
"""
|
| 292 |
-
species_map = get_species_map(self.audio_dir)
|
| 293 |
-
|
| 294 |
-
for species_name in sorted(species_map.keys()):
|
| 295 |
-
shards = species_map[species_name]
|
| 296 |
-
files = self.get_files_for_species(species_name)
|
| 297 |
-
|
| 298 |
-
metadata = {
|
| 299 |
-
"num_files": len(files),
|
| 300 |
-
"num_shards": len(shards),
|
| 301 |
-
"shard_dirs": shards,
|
| 302 |
-
}
|
| 303 |
-
|
| 304 |
-
yield species_name, files, metadata
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/test_gui.py
DELETED
|
@@ -1,132 +0,0 @@
|
|
| 1 |
-
#!/usr/bin/env python3
|
| 2 |
-
"""
|
| 3 |
-
Quick test script for the GUI application.
|
| 4 |
-
|
| 5 |
-
This script tests that all dependencies are installed and the GUI can launch.
|
| 6 |
-
No actual uploads are performed in this test.
|
| 7 |
-
|
| 8 |
-
Usage:
|
| 9 |
-
python test_gui.py
|
| 10 |
-
"""
|
| 11 |
-
|
| 12 |
-
import sys
|
| 13 |
-
from pathlib import Path
|
| 14 |
-
|
| 15 |
-
def check_imports():
|
| 16 |
-
"""Check if all required modules are available."""
|
| 17 |
-
modules = {
|
| 18 |
-
"customtkinter": "CustomTkinter (GUI framework)",
|
| 19 |
-
"huggingface_hub": "Hugging Face Hub API",
|
| 20 |
-
"requests": "HTTP library",
|
| 21 |
-
"tqdm": "Progress bars",
|
| 22 |
-
}
|
| 23 |
-
|
| 24 |
-
print("🔍 Checking required modules...\n")
|
| 25 |
-
|
| 26 |
-
missing = []
|
| 27 |
-
for module_name, description in modules.items():
|
| 28 |
-
try:
|
| 29 |
-
__import__(module_name)
|
| 30 |
-
print(f"✅ {module_name:20} — {description}")
|
| 31 |
-
except ImportError:
|
| 32 |
-
print(f"❌ {module_name:20} — {description} [MISSING]")
|
| 33 |
-
missing.append(module_name)
|
| 34 |
-
|
| 35 |
-
if missing:
|
| 36 |
-
print(f"\n⚠️ Missing modules: {', '.join(missing)}")
|
| 37 |
-
print(f"\nInstall with:")
|
| 38 |
-
print(f" pip install {' '.join(missing)}")
|
| 39 |
-
return False
|
| 40 |
-
|
| 41 |
-
print(f"\n✅ All modules installed!")
|
| 42 |
-
return True
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
def check_local_files():
|
| 46 |
-
"""Check if GUI files exist."""
|
| 47 |
-
print("\n🔍 Checking local files...\n")
|
| 48 |
-
|
| 49 |
-
files_to_check = [
|
| 50 |
-
"upload_logic.py",
|
| 51 |
-
"main_gui.py",
|
| 52 |
-
"requirements_gui.txt",
|
| 53 |
-
]
|
| 54 |
-
|
| 55 |
-
all_exist = True
|
| 56 |
-
for filename in files_to_check:
|
| 57 |
-
filepath = Path(__file__).parent / filename
|
| 58 |
-
if filepath.exists():
|
| 59 |
-
size_kb = filepath.stat().st_size / 1024
|
| 60 |
-
print(f"✅ {filename:25} ({size_kb:.1f} KB)")
|
| 61 |
-
else:
|
| 62 |
-
print(f"❌ {filename:25} [NOT FOUND]")
|
| 63 |
-
all_exist = False
|
| 64 |
-
|
| 65 |
-
return all_exist
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
def test_gui_launch():
|
| 69 |
-
"""Attempt to launch the GUI."""
|
| 70 |
-
print("\n🚀 Launching GUI...\n")
|
| 71 |
-
|
| 72 |
-
try:
|
| 73 |
-
from main_gui import HFUploaderApp
|
| 74 |
-
|
| 75 |
-
print("✅ GUI imported successfully!")
|
| 76 |
-
print("📋 Creating application window...")
|
| 77 |
-
|
| 78 |
-
app = HFUploaderApp()
|
| 79 |
-
|
| 80 |
-
print("✅ GUI window created successfully!")
|
| 81 |
-
print("\n💡 Tips:")
|
| 82 |
-
print(" • Fill in test values (repo ID, token, folder paths)")
|
| 83 |
-
print(" • Click 'Start Upload' to trigger the upload logic")
|
| 84 |
-
print(" • Watch for logs in the progress textbox")
|
| 85 |
-
print(" • Close the window to exit")
|
| 86 |
-
print("\nPress Ctrl+C to close the window from this terminal.\n")
|
| 87 |
-
|
| 88 |
-
# Run the GUI
|
| 89 |
-
app.mainloop()
|
| 90 |
-
|
| 91 |
-
return True
|
| 92 |
-
|
| 93 |
-
except Exception as e:
|
| 94 |
-
print(f"❌ Failed to launch GUI: {e}")
|
| 95 |
-
import traceback
|
| 96 |
-
traceback.print_exc()
|
| 97 |
-
return False
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
def main():
|
| 101 |
-
"""Run all checks and optionally launch GUI."""
|
| 102 |
-
print("=" * 60)
|
| 103 |
-
print("HF Dataset Uploader — GUI Test Suite")
|
| 104 |
-
print("=" * 60)
|
| 105 |
-
|
| 106 |
-
# Step 1: Check imports
|
| 107 |
-
if not check_imports():
|
| 108 |
-
print("\n⚠️ Cannot proceed without missing modules.")
|
| 109 |
-
print("Please install them and try again.")
|
| 110 |
-
return 1
|
| 111 |
-
|
| 112 |
-
# Step 2: Check files
|
| 113 |
-
if not check_local_files():
|
| 114 |
-
print("\n⚠️ Some GUI files are missing.")
|
| 115 |
-
print("Ensure you're running this from the hf_bulk_upload_tool/ directory.")
|
| 116 |
-
return 1
|
| 117 |
-
|
| 118 |
-
# Step 3: Launch GUI
|
| 119 |
-
print("\n" + "=" * 60)
|
| 120 |
-
answer = input("Launch GUI? (yes/no): ").strip().lower()
|
| 121 |
-
|
| 122 |
-
if answer in ['y', 'yes']:
|
| 123 |
-
if not test_gui_launch():
|
| 124 |
-
return 1
|
| 125 |
-
else:
|
| 126 |
-
print("\n✅ All checks passed! You can now run: python main_gui.py")
|
| 127 |
-
|
| 128 |
-
return 0
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
if __name__ == "__main__":
|
| 132 |
-
sys.exit(main())
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/upload_dataset.py
DELETED
|
@@ -1,767 +0,0 @@
|
|
| 1 |
-
#!/usr/bin/env python3
|
| 2 |
-
from __future__ import annotations
|
| 3 |
-
|
| 4 |
-
import argparse
|
| 5 |
-
import getpass
|
| 6 |
-
import hashlib
|
| 7 |
-
import os
|
| 8 |
-
import shutil
|
| 9 |
-
import sys
|
| 10 |
-
import subprocess
|
| 11 |
-
import time
|
| 12 |
-
import contextlib
|
| 13 |
-
from pathlib import Path
|
| 14 |
-
import concurrent.futures
|
| 15 |
-
from typing import List, Tuple
|
| 16 |
-
import json
|
| 17 |
-
import csv
|
| 18 |
-
from urllib.parse import quote
|
| 19 |
-
|
| 20 |
-
try:
|
| 21 |
-
import requests
|
| 22 |
-
except Exception:
|
| 23 |
-
requests = None
|
| 24 |
-
try:
|
| 25 |
-
from tqdm import tqdm
|
| 26 |
-
except Exception:
|
| 27 |
-
tqdm = None
|
| 28 |
-
import threading
|
| 29 |
-
|
| 30 |
-
try:
|
| 31 |
-
from sharding_utils import should_shard_directory, shard_directory
|
| 32 |
-
from progress_bar_utils import ProgressFilter, FilesProgressBar
|
| 33 |
-
except ImportError:
|
| 34 |
-
# Fallback for relative imports during development
|
| 35 |
-
from .sharding_utils import should_shard_directory, shard_directory
|
| 36 |
-
from .progress_bar_utils import ProgressFilter, FilesProgressBar
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
def configure_hf_env() -> None:
|
| 40 |
-
os.environ.setdefault("HF_HUB_ETAG_TIMEOUT", "20")
|
| 41 |
-
os.environ.setdefault("HF_HUB_DOWNLOAD_TIMEOUT", "120")
|
| 42 |
-
os.environ.setdefault("HF_XET_HIGH_PERFORMANCE", "1")
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
configure_hf_env()
|
| 46 |
-
|
| 47 |
-
from huggingface_hub import HfApi
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
class RateLimiter:
|
| 51 |
-
"""Sliding window rate limiter for HF API (1000 requests per 5 minutes)."""
|
| 52 |
-
def __init__(self, max_requests: int = 1000, window_seconds: int = 300):
|
| 53 |
-
self.max_requests = max_requests
|
| 54 |
-
self.window_seconds = window_seconds
|
| 55 |
-
self.request_times: list[tuple[float, str]] = []
|
| 56 |
-
|
| 57 |
-
def acquire(self, request_type: str = "upload") -> None:
|
| 58 |
-
"""Wait if necessary to stay within rate limits, then register the request."""
|
| 59 |
-
now = time.time()
|
| 60 |
-
|
| 61 |
-
# Remove requests outside the current window
|
| 62 |
-
self.request_times = [
|
| 63 |
-
(ts, req_type) for ts, req_type in self.request_times
|
| 64 |
-
if now - ts < self.window_seconds
|
| 65 |
-
]
|
| 66 |
-
|
| 67 |
-
# If at limit, wait until oldest request drops out of window
|
| 68 |
-
if len(self.request_times) >= self.max_requests:
|
| 69 |
-
oldest_ts = self.request_times[0][0]
|
| 70 |
-
wait_time = (oldest_ts + self.window_seconds) - now
|
| 71 |
-
if wait_time > 0:
|
| 72 |
-
print(f"[RateLimit] Reached {self.max_requests} requests in {self.window_seconds}s window.")
|
| 73 |
-
print(f"[RateLimit] Waiting {wait_time:.0f}s before next request...")
|
| 74 |
-
time.sleep(wait_time)
|
| 75 |
-
self.request_times = [] # Reset window
|
| 76 |
-
|
| 77 |
-
# Register this request
|
| 78 |
-
self.request_times.append((now, request_type))
|
| 79 |
-
|
| 80 |
-
def get_window_usage(self) -> tuple[int, int]:
|
| 81 |
-
"""Return (requests_in_window, max_requests)."""
|
| 82 |
-
now = time.time()
|
| 83 |
-
self.request_times = [
|
| 84 |
-
(ts, req_type) for ts, req_type in self.request_times
|
| 85 |
-
if now - ts < self.window_seconds
|
| 86 |
-
]
|
| 87 |
-
return len(self.request_times), self.max_requests
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
def ensure_directory_link(link_path: Path, target_path: Path) -> None:
|
| 91 |
-
"""Create a directory link on the local filesystem.
|
| 92 |
-
|
| 93 |
-
On Windows, prefer a junction so the uploader can stage the repo structure
|
| 94 |
-
without copying the full audio tree.
|
| 95 |
-
"""
|
| 96 |
-
if link_path.exists() or link_path.is_symlink():
|
| 97 |
-
try:
|
| 98 |
-
if link_path.resolve() == target_path.resolve():
|
| 99 |
-
return
|
| 100 |
-
except Exception:
|
| 101 |
-
pass
|
| 102 |
-
if link_path.is_symlink() or link_path.is_file():
|
| 103 |
-
link_path.unlink()
|
| 104 |
-
else:
|
| 105 |
-
shutil.rmtree(link_path)
|
| 106 |
-
|
| 107 |
-
link_path.parent.mkdir(parents=True, exist_ok=True)
|
| 108 |
-
|
| 109 |
-
if os.name == "nt":
|
| 110 |
-
try:
|
| 111 |
-
subprocess.run(
|
| 112 |
-
["cmd", "/c", "mklink", "/J", str(link_path), str(target_path)],
|
| 113 |
-
check=True,
|
| 114 |
-
capture_output=True,
|
| 115 |
-
text=True,
|
| 116 |
-
creationflags=subprocess.CREATE_NO_WINDOW,
|
| 117 |
-
)
|
| 118 |
-
return
|
| 119 |
-
except Exception:
|
| 120 |
-
pass
|
| 121 |
-
|
| 122 |
-
os.symlink(target_path, link_path, target_is_directory=True)
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
def prepare_staging_folder(staging_root: Path, segments_path: Path, csv_path: Path | None) -> Path:
|
| 126 |
-
"""Build a persistent staging tree with the repo layout expected by HF.
|
| 127 |
-
|
| 128 |
-
The stage keeps the audio directory as a link to the real data and copies
|
| 129 |
-
only small metadata files locally.
|
| 130 |
-
"""
|
| 131 |
-
staging_root.mkdir(parents=True, exist_ok=True)
|
| 132 |
-
|
| 133 |
-
audio_link = staging_root / "audio"
|
| 134 |
-
ensure_directory_link(audio_link, segments_path)
|
| 135 |
-
|
| 136 |
-
if csv_path is not None:
|
| 137 |
-
csv_dest = staging_root / "index" / "detections.csv"
|
| 138 |
-
csv_dest.parent.mkdir(parents=True, exist_ok=True)
|
| 139 |
-
shutil.copy2(csv_path, csv_dest)
|
| 140 |
-
|
| 141 |
-
return staging_root
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
def _prepare_sharded_staging(staging_root: Path, segments_path: Path, csv_path: Path | None) -> Path:
|
| 145 |
-
"""Prepare staging folder with automatic sharding for large species directories.
|
| 146 |
-
|
| 147 |
-
Detects species folders with >9500 files and automatically shards them into
|
| 148 |
-
numbered subdirectories to comply with HF's 10k files per directory limit.
|
| 149 |
-
Validation tools can transparently read shards via sharding_utils.
|
| 150 |
-
|
| 151 |
-
Returns:
|
| 152 |
-
Path to the prepared staging root directory
|
| 153 |
-
"""
|
| 154 |
-
staging_root.mkdir(parents=True, exist_ok=True)
|
| 155 |
-
audio_staging = staging_root / "audio"
|
| 156 |
-
audio_staging.mkdir(parents=True, exist_ok=True)
|
| 157 |
-
|
| 158 |
-
sharded_species = []
|
| 159 |
-
|
| 160 |
-
# Check each species folder for sharding
|
| 161 |
-
for species_dir in sorted(segments_path.iterdir()):
|
| 162 |
-
if not species_dir.is_dir():
|
| 163 |
-
continue
|
| 164 |
-
|
| 165 |
-
if should_shard_directory(species_dir):
|
| 166 |
-
# Need to shard this species
|
| 167 |
-
print(f"[Sharding] {species_dir.name} exceeds 9500 files, dividing into shards...")
|
| 168 |
-
try:
|
| 169 |
-
shards = shard_directory(species_dir, audio_staging)
|
| 170 |
-
sharded_species.append((species_dir.name, len(shards)))
|
| 171 |
-
print(f"[Sharding] Created {len(shards)} shards for {species_dir.name}")
|
| 172 |
-
except Exception as e:
|
| 173 |
-
print(f"[Sharding] ERROR sharding {species_dir.name}: {e}")
|
| 174 |
-
print(f"[Sharding] Falling back to single directory link")
|
| 175 |
-
ensure_directory_link(audio_staging / species_dir.name, species_dir)
|
| 176 |
-
else:
|
| 177 |
-
# Small species, just link it
|
| 178 |
-
ensure_directory_link(audio_staging / species_dir.name, species_dir)
|
| 179 |
-
|
| 180 |
-
# Log sharding summary
|
| 181 |
-
if sharded_species:
|
| 182 |
-
print(f"\n[Sharding] Summary: {len(sharded_species)} species were divided")
|
| 183 |
-
for species_name, num_shards in sharded_species:
|
| 184 |
-
print(f" - {species_name}: {num_shards} shards")
|
| 185 |
-
print("[Sharding] Note: Validation tools will transparently read all shards as single species\n")
|
| 186 |
-
|
| 187 |
-
# Copy CSV if provided
|
| 188 |
-
if csv_path is not None:
|
| 189 |
-
csv_dest = staging_root / "index" / "detections.csv"
|
| 190 |
-
csv_dest.parent.mkdir(parents=True, exist_ok=True)
|
| 191 |
-
shutil.copy2(csv_path, csv_dest)
|
| 192 |
-
|
| 193 |
-
return staging_root
|
| 194 |
-
|
| 195 |
-
|
| 196 |
-
def parse_args() -> argparse.Namespace:
|
| 197 |
-
parser = argparse.ArgumentParser(
|
| 198 |
-
description=(
|
| 199 |
-
"Upload a large BirdNET segment folder to a Hugging Face dataset "
|
| 200 |
-
"while preserving the local species subfolder structure."
|
| 201 |
-
)
|
| 202 |
-
)
|
| 203 |
-
parser.add_argument("--hf-username", help="Hugging Face username or organization name.")
|
| 204 |
-
parser.add_argument("--hf-token", help="Hugging Face write token. If omitted, the script prompts for it.")
|
| 205 |
-
parser.add_argument("--repo-name", help="Dataset name to create, for example birdnet-segments-2026.")
|
| 206 |
-
parser.add_argument("--repo-id", help="Full dataset repo id, for example username/dataset-name.")
|
| 207 |
-
parser.add_argument("--segments", required=True, help="Path to the local folder containing the species subfolders.")
|
| 208 |
-
parser.add_argument("--csv", help="Optional CSV file to upload alongside the segments.")
|
| 209 |
-
parser.add_argument("--segments-path-in-repo", default="audio", help="Destination folder inside the dataset repo.")
|
| 210 |
-
parser.add_argument("--csv-path-in-repo", default="index/detections.csv", help="Destination path for the CSV inside the repo.")
|
| 211 |
-
parser.add_argument("--private", action="store_true", help="Create the dataset as private. This is the default.")
|
| 212 |
-
parser.add_argument("--public", action="store_true", help="Create the dataset as public.")
|
| 213 |
-
parser.add_argument("--commit-message", default="Upload BirdNET segments", help="Commit message used for the upload.")
|
| 214 |
-
parser.add_argument("--create-repo-attempts", type=int, default=3, help="Number of retries for repo creation.")
|
| 215 |
-
parser.add_argument("--upload-attempts", type=int, default=3, help="Number of retries for folder and CSV uploads.")
|
| 216 |
-
parser.add_argument("--retry-backoff", type=float, default=5.0, help="Base delay in seconds between retries.")
|
| 217 |
-
parser.add_argument("--max-workers", type=int, default=1, help="Number of parallel worker threads for per-file uploads (fallback path).")
|
| 218 |
-
parser.add_argument("--per-file-delay", type=float, default=0.5, help="Seconds to wait between per-file uploads to avoid rate limits.")
|
| 219 |
-
parser.add_argument("--resume", action="store_true", help="Check repository and skip files that are already uploaded (resume mode).")
|
| 220 |
-
parser.add_argument("--checkpoint-dir", default=None, help="Directory to store upload checkpoints. Defaults to ./hf_bulk_upload_tool/.checkpoints.")
|
| 221 |
-
parser.add_argument("--verify-remote", action="store_true", help="When resuming, verify remote file size via HTTP HEAD to avoid skipping mismatched files.")
|
| 222 |
-
parser.add_argument("--verify-etag", action="store_true", help="When resuming, prefer ETag comparison if available from the server.")
|
| 223 |
-
parser.add_argument("--progress-log", default=None, help="Path to CSV progress log. Defaults to <checkpoint-dir>/progress.csv")
|
| 224 |
-
parser.add_argument("--resume-only", action="store_true", help="Only compare local files with the repo and report what would be uploaded; do not upload anything.")
|
| 225 |
-
parser.add_argument("--dry-run", action="store_true", help="Print the planned actions without uploading anything.")
|
| 226 |
-
parser.add_argument("--rate-limit-aware", action="store_true", default=True, help="Enable rate limiter to respect HF 1000 req/5min quota (default: True).")
|
| 227 |
-
parser.add_argument("--rate-limit-max-requests", type=int, default=950, help="Max requests per window before waiting (default: 950 for safety buffer).")
|
| 228 |
-
parser.add_argument("--rate-limit-window", type=int, default=300, help="Rate limit window in seconds (default: 300 = 5min).")
|
| 229 |
-
return parser.parse_args()
|
| 230 |
-
|
| 231 |
-
|
| 232 |
-
def resolve_repo_id(args: argparse.Namespace) -> str:
|
| 233 |
-
if args.repo_id:
|
| 234 |
-
return args.repo_id.strip()
|
| 235 |
-
|
| 236 |
-
if not args.hf_username:
|
| 237 |
-
args.hf_username = input("Hugging Face username/organization: ").strip()
|
| 238 |
-
if not args.repo_name:
|
| 239 |
-
args.repo_name = input("Dataset name: ").strip()
|
| 240 |
-
|
| 241 |
-
if not args.hf_username or not args.repo_name:
|
| 242 |
-
raise SystemExit("Both --hf-username and --repo-name are required when --repo-id is not provided.")
|
| 243 |
-
|
| 244 |
-
return f"{args.hf_username.strip()}/{args.repo_name.strip()}"
|
| 245 |
-
|
| 246 |
-
|
| 247 |
-
def get_token(args: argparse.Namespace) -> str:
|
| 248 |
-
token = args.hf_token or os.environ.get("HF_TOKEN")
|
| 249 |
-
if not token:
|
| 250 |
-
token = getpass.getpass("Hugging Face token: ").strip()
|
| 251 |
-
if not token:
|
| 252 |
-
raise SystemExit("A Hugging Face token is required.")
|
| 253 |
-
return token.strip()
|
| 254 |
-
|
| 255 |
-
|
| 256 |
-
def retry_call(label: str, attempts: int, backoff: float, func):
|
| 257 |
-
last_error: Exception | None = None
|
| 258 |
-
for attempt in range(1, attempts + 1):
|
| 259 |
-
try:
|
| 260 |
-
print(f"[{label}] attempt {attempt}/{attempts}")
|
| 261 |
-
return func()
|
| 262 |
-
except Exception as exc: # noqa: BLE001 - network operations can fail in many ways
|
| 263 |
-
last_error = exc
|
| 264 |
-
if attempt >= attempts:
|
| 265 |
-
break
|
| 266 |
-
# Detect possible 429 / rate-limit signals and increase backoff
|
| 267 |
-
wait_seconds = backoff * attempt
|
| 268 |
-
try:
|
| 269 |
-
resp = getattr(exc, 'response', None)
|
| 270 |
-
if resp is not None:
|
| 271 |
-
status = getattr(resp, 'status_code', None)
|
| 272 |
-
if status == 429:
|
| 273 |
-
# obey Retry-After header when present
|
| 274 |
-
ra = resp.headers.get('Retry-After') if hasattr(resp, 'headers') else None
|
| 275 |
-
if ra:
|
| 276 |
-
try:
|
| 277 |
-
wait_seconds = max(wait_seconds, int(ra))
|
| 278 |
-
except Exception:
|
| 279 |
-
wait_seconds = max(wait_seconds, backoff * attempt * 4)
|
| 280 |
-
else:
|
| 281 |
-
wait_seconds = max(wait_seconds, backoff * attempt * 4)
|
| 282 |
-
elif '429' in str(exc) or 'Too Many Requests' in str(exc):
|
| 283 |
-
wait_seconds = max(wait_seconds, backoff * attempt * 4)
|
| 284 |
-
except Exception:
|
| 285 |
-
pass
|
| 286 |
-
print(f"[{label}] attempt {attempt} failed: {exc}")
|
| 287 |
-
print(f"[{label}] retrying in {wait_seconds:.0f}s")
|
| 288 |
-
time.sleep(wait_seconds)
|
| 289 |
-
|
| 290 |
-
raise SystemExit(f"{label} failed after {attempts} attempts: {last_error}")
|
| 291 |
-
|
| 292 |
-
|
| 293 |
-
def main() -> int:
|
| 294 |
-
args = parse_args()
|
| 295 |
-
|
| 296 |
-
if args.private and args.public:
|
| 297 |
-
raise SystemExit("Choose either --private or --public, not both.")
|
| 298 |
-
|
| 299 |
-
segments_path = Path(args.segments).expanduser().resolve()
|
| 300 |
-
if not segments_path.is_dir():
|
| 301 |
-
raise SystemExit(f"Segments path does not exist or is not a directory: {segments_path}")
|
| 302 |
-
|
| 303 |
-
csv_path = None
|
| 304 |
-
if args.csv:
|
| 305 |
-
csv_path = Path(args.csv).expanduser().resolve()
|
| 306 |
-
if not csv_path.is_file():
|
| 307 |
-
raise SystemExit(f"CSV path does not exist or is not a file: {csv_path}")
|
| 308 |
-
|
| 309 |
-
repo_id = resolve_repo_id(args)
|
| 310 |
-
token = get_token(args)
|
| 311 |
-
is_private = True if args.private or not args.public else False
|
| 312 |
-
|
| 313 |
-
print(f"Repository: {repo_id}")
|
| 314 |
-
print(f"Segments: {segments_path}")
|
| 315 |
-
print(f"Segments destination: {args.segments_path_in_repo}")
|
| 316 |
-
# Track if CSV was uploaded as part of the staged large-folder upload
|
| 317 |
-
csv_uploaded_with_segments = False
|
| 318 |
-
|
| 319 |
-
if csv_path and not csv_uploaded_with_segments:
|
| 320 |
-
print(f"CSV: {csv_path}")
|
| 321 |
-
print(f"CSV destination: {args.csv_path_in_repo}")
|
| 322 |
-
print(f"Visibility: {'private' if is_private else 'public'}")
|
| 323 |
-
|
| 324 |
-
if args.dry_run:
|
| 325 |
-
print("Dry run selected. No network calls will be made.")
|
| 326 |
-
return 0
|
| 327 |
-
|
| 328 |
-
api = HfApi(token=token)
|
| 329 |
-
|
| 330 |
-
# Initialize rate limiter for large uploads (150k files = ~12.5 hours at 1000 req/5min)
|
| 331 |
-
if args.rate_limit_aware:
|
| 332 |
-
rate_limiter = RateLimiter(max_requests=args.rate_limit_max_requests, window_seconds=args.rate_limit_window)
|
| 333 |
-
print(f"[RateLimit] Enabled: max {args.rate_limit_max_requests} requests per {args.rate_limit_window}s")
|
| 334 |
-
else:
|
| 335 |
-
rate_limiter = None
|
| 336 |
-
|
| 337 |
-
def list_existing_files() -> set:
|
| 338 |
-
try:
|
| 339 |
-
print("Listing files in target repo to determine already-uploaded files...")
|
| 340 |
-
if rate_limiter:
|
| 341 |
-
rate_limiter.acquire("list_repo_files")
|
| 342 |
-
files = api.list_repo_files(repo_id=repo_id, repo_type="dataset")
|
| 343 |
-
return set(files or [])
|
| 344 |
-
except Exception as e:
|
| 345 |
-
print(f"Warning: could not list repo files: {e}")
|
| 346 |
-
return set()
|
| 347 |
-
|
| 348 |
-
|
| 349 |
-
# Checkpoint helpers
|
| 350 |
-
default_checkpoint_dir = Path(__file__).parent / ".checkpoints"
|
| 351 |
-
checkpoint_dir = Path(args.checkpoint_dir) if args.checkpoint_dir else default_checkpoint_dir
|
| 352 |
-
checkpoint_dir.mkdir(parents=True, exist_ok=True)
|
| 353 |
-
checkpoint_file = checkpoint_dir / f"{repo_id.replace('/', '__')}.json"
|
| 354 |
-
|
| 355 |
-
def load_checkpoint() -> tuple[set, dict[str, str]]:
|
| 356 |
-
if not checkpoint_file.exists():
|
| 357 |
-
return set(), {}
|
| 358 |
-
try:
|
| 359 |
-
with open(checkpoint_file, 'r', encoding='utf-8') as fh:
|
| 360 |
-
data = json.load(fh)
|
| 361 |
-
if isinstance(data, list):
|
| 362 |
-
return set(data), {}
|
| 363 |
-
uploaded = set(data.get('uploaded', []))
|
| 364 |
-
file_hashes = data.get('file_hashes', {})
|
| 365 |
-
if not isinstance(file_hashes, dict):
|
| 366 |
-
file_hashes = {}
|
| 367 |
-
return uploaded, {str(k): str(v) for k, v in file_hashes.items()}
|
| 368 |
-
except Exception as e:
|
| 369 |
-
print(f"Warning: failed to load checkpoint {checkpoint_file}: {e}")
|
| 370 |
-
return set(), {}
|
| 371 |
-
|
| 372 |
-
def save_checkpoint(uploaded_set: set, file_hashes: dict[str, str]) -> None:
|
| 373 |
-
try:
|
| 374 |
-
with open(checkpoint_file, 'w', encoding='utf-8') as fh:
|
| 375 |
-
json.dump({'uploaded': sorted(list(uploaded_set)), 'file_hashes': file_hashes}, fh, indent=2)
|
| 376 |
-
except Exception as e:
|
| 377 |
-
print(f"Warning: failed to save checkpoint {checkpoint_file}: {e}")
|
| 378 |
-
|
| 379 |
-
def compute_sha256(file_path: Path) -> str:
|
| 380 |
-
digest = hashlib.sha256()
|
| 381 |
-
with open(file_path, 'rb') as fh:
|
| 382 |
-
for chunk in iter(lambda: fh.read(1024 * 1024), b''):
|
| 383 |
-
digest.update(chunk)
|
| 384 |
-
return digest.hexdigest()
|
| 385 |
-
|
| 386 |
-
def remote_file_info(repo_path: str) -> Tuple[int | None, str | None]:
|
| 387 |
-
"""Return (size, etag) for a repo file via HTTP HEAD, or (None, None)."""
|
| 388 |
-
if requests is None:
|
| 389 |
-
return None, None
|
| 390 |
-
quoted = '/'.join(quote(p) for p in repo_path.split('/'))
|
| 391 |
-
url = f"https://huggingface.co/datasets/{repo_id}/resolve/main/{quoted}"
|
| 392 |
-
try:
|
| 393 |
-
r = requests.head(url, allow_redirects=True, timeout=10)
|
| 394 |
-
if r.status_code >= 400:
|
| 395 |
-
return None, None
|
| 396 |
-
cl = r.headers.get('content-length')
|
| 397 |
-
etag = r.headers.get('etag')
|
| 398 |
-
size = int(cl) if cl is not None else None
|
| 399 |
-
return size, etag
|
| 400 |
-
except Exception:
|
| 401 |
-
return None, None
|
| 402 |
-
|
| 403 |
-
# Progress logging
|
| 404 |
-
progress_log_path = Path(args.progress_log) if args.progress_log else (checkpoint_dir / 'progress.csv')
|
| 405 |
-
def append_progress(local_path: Path | None, repo_path: str, status: str, msg: str | None = None, elapsed: float | None = None) -> None:
|
| 406 |
-
header = ['timestamp','local_path','repo_path','status','message','elapsed_seconds']
|
| 407 |
-
row = [time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()), str(local_path) if local_path else '', repo_path, status, (msg or '').replace('\n',' '), f"{elapsed:.2f}" if elapsed is not None else '']
|
| 408 |
-
write_header = not progress_log_path.exists()
|
| 409 |
-
try:
|
| 410 |
-
with open(progress_log_path, 'a', newline='', encoding='utf-8') as fh:
|
| 411 |
-
writer = csv.writer(fh)
|
| 412 |
-
if write_header:
|
| 413 |
-
writer.writerow(header)
|
| 414 |
-
writer.writerow(row)
|
| 415 |
-
except Exception as e:
|
| 416 |
-
print(f"Warning: failed to write progress log {progress_log_path}: {e}")
|
| 417 |
-
|
| 418 |
-
run_stats = {
|
| 419 |
-
'planned': 0,
|
| 420 |
-
'skipped': 0,
|
| 421 |
-
'uploaded': 0,
|
| 422 |
-
'failed': 0,
|
| 423 |
-
'bytes_planned': 0,
|
| 424 |
-
'bytes_uploaded': 0,
|
| 425 |
-
'started_at': time.time(),
|
| 426 |
-
}
|
| 427 |
-
|
| 428 |
-
def record_summary() -> None:
|
| 429 |
-
elapsed = time.time() - run_stats['started_at']
|
| 430 |
-
summary_message = (
|
| 431 |
-
f"planned={run_stats['planned']} skipped={run_stats['skipped']} "
|
| 432 |
-
f"uploaded={run_stats['uploaded']} failed={run_stats['failed']} "
|
| 433 |
-
f"bytes_planned={run_stats['bytes_planned']} bytes_uploaded={run_stats['bytes_uploaded']}"
|
| 434 |
-
)
|
| 435 |
-
append_progress(None, '', 'summary', summary_message, elapsed)
|
| 436 |
-
try:
|
| 437 |
-
summary_json = checkpoint_dir / f"{repo_id.replace('/', '__')}.summary.json"
|
| 438 |
-
with open(summary_json, 'w', encoding='utf-8') as fh:
|
| 439 |
-
json.dump({
|
| 440 |
-
'repo_id': repo_id,
|
| 441 |
-
'planned': run_stats['planned'],
|
| 442 |
-
'skipped': run_stats['skipped'],
|
| 443 |
-
'uploaded': run_stats['uploaded'],
|
| 444 |
-
'failed': run_stats['failed'],
|
| 445 |
-
'bytes_planned': run_stats['bytes_planned'],
|
| 446 |
-
'bytes_uploaded': run_stats['bytes_uploaded'],
|
| 447 |
-
'elapsed_seconds': round(elapsed, 2),
|
| 448 |
-
}, fh, indent=2)
|
| 449 |
-
except Exception as e:
|
| 450 |
-
print(f"Warning: failed to write summary json: {e}")
|
| 451 |
-
|
| 452 |
-
def remote_file_size(repo_path: str) -> int | None:
|
| 453 |
-
if requests is None:
|
| 454 |
-
return None
|
| 455 |
-
# Construct raw file URL: https://huggingface.co/datasets/{repo_id}/resolve/main/{path}
|
| 456 |
-
# Path components must be URL-quoted
|
| 457 |
-
quoted = '/'.join(quote(p) for p in repo_path.split('/'))
|
| 458 |
-
url = f"https://huggingface.co/datasets/{repo_id}/resolve/main/{quoted}"
|
| 459 |
-
try:
|
| 460 |
-
r = requests.head(url, allow_redirects=True, timeout=10)
|
| 461 |
-
if r.status_code >= 400:
|
| 462 |
-
return None
|
| 463 |
-
cl = r.headers.get('content-length')
|
| 464 |
-
if cl is None:
|
| 465 |
-
return None
|
| 466 |
-
return int(cl)
|
| 467 |
-
except Exception:
|
| 468 |
-
return None
|
| 469 |
-
|
| 470 |
-
def create_repo() -> None:
|
| 471 |
-
api.create_repo(repo_id=repo_id, repo_type="dataset", private=is_private, exist_ok=True)
|
| 472 |
-
|
| 473 |
-
retry_call("create_repo", args.create_repo_attempts, args.retry_backoff, create_repo)
|
| 474 |
-
# Prepare set of existing files if resume requested
|
| 475 |
-
existing_files: set = set()
|
| 476 |
-
checkpoint_uploaded: set = set()
|
| 477 |
-
checkpoint_hashes: dict[str, str] = {}
|
| 478 |
-
if args.resume:
|
| 479 |
-
existing_files = list_existing_files()
|
| 480 |
-
checkpoint_uploaded, checkpoint_hashes = load_checkpoint()
|
| 481 |
-
|
| 482 |
-
# union of already-known uploaded files
|
| 483 |
-
already_uploaded = existing_files.union(checkpoint_uploaded)
|
| 484 |
-
|
| 485 |
-
def upload_segments() -> None:
|
| 486 |
-
nonlocal csv_uploaded_with_segments
|
| 487 |
-
# existing_files is computed above when resume is requested
|
| 488 |
-
|
| 489 |
-
if args.resume_only:
|
| 490 |
-
print("Resume-only mode: no upload will be performed.")
|
| 491 |
-
files: List[Tuple[Path, str]] = []
|
| 492 |
-
for p in segments_path.rglob("*"):
|
| 493 |
-
if p.is_file():
|
| 494 |
-
rel = p.relative_to(segments_path).as_posix()
|
| 495 |
-
repo_path = f"{args.segments_path_in_repo}/{rel}"
|
| 496 |
-
run_stats['bytes_planned'] += p.stat().st_size
|
| 497 |
-
if repo_path in already_uploaded:
|
| 498 |
-
checkpoint_hash = checkpoint_hashes.get(repo_path)
|
| 499 |
-
if checkpoint_hash:
|
| 500 |
-
local_hash = compute_sha256(p)
|
| 501 |
-
if local_hash == checkpoint_hash:
|
| 502 |
-
run_stats['skipped'] += 1
|
| 503 |
-
append_progress(p, repo_path, 'skipped', 'already uploaded')
|
| 504 |
-
continue
|
| 505 |
-
files.append((p, repo_path))
|
| 506 |
-
|
| 507 |
-
run_stats['planned'] = len(files)
|
| 508 |
-
for local_path, repo_path in files:
|
| 509 |
-
append_progress(local_path, repo_path, 'planned', 'resume-only')
|
| 510 |
-
return
|
| 511 |
-
|
| 512 |
-
if hasattr(api, "upload_large_folder"):
|
| 513 |
-
staging_root = checkpoint_dir / "staging" / repo_id.replace("/", "__")
|
| 514 |
-
print(f"Preparing persistent staging folder at {staging_root} ...")
|
| 515 |
-
print(f"Checking for large species directories that need sharding...\n")
|
| 516 |
-
staged_root = _prepare_sharded_staging(staging_root, segments_path, csv_path)
|
| 517 |
-
|
| 518 |
-
total_bytes = 0
|
| 519 |
-
total_files = 0
|
| 520 |
-
for root, _, files in os.walk(staged_root):
|
| 521 |
-
for name in files:
|
| 522 |
-
file_path = Path(root) / name
|
| 523 |
-
if ".cache" in file_path.parts:
|
| 524 |
-
continue
|
| 525 |
-
try:
|
| 526 |
-
total_bytes += file_path.stat().st_size
|
| 527 |
-
total_files += 1
|
| 528 |
-
except Exception:
|
| 529 |
-
continue
|
| 530 |
-
|
| 531 |
-
print(f"Found {total_files} staged files to upload with upload_large_folder()")
|
| 532 |
-
|
| 533 |
-
kwargs = {
|
| 534 |
-
"folder_path": str(staged_root),
|
| 535 |
-
"repo_id": repo_id,
|
| 536 |
-
"repo_type": "dataset",
|
| 537 |
-
"private": is_private,
|
| 538 |
-
"num_workers": max(1, args.max_workers),
|
| 539 |
-
"print_report": True,
|
| 540 |
-
"print_report_every": 60,
|
| 541 |
-
}
|
| 542 |
-
|
| 543 |
-
def _run_upload():
|
| 544 |
-
# Create progress bar and filter
|
| 545 |
-
if tqdm is not None:
|
| 546 |
-
pbar = tqdm(total=total_files, unit="file", unit_scale=False, desc="Uploading", leave=True)
|
| 547 |
-
filter_writer = ProgressFilter(pbar, print)
|
| 548 |
-
else:
|
| 549 |
-
pbar = None
|
| 550 |
-
filter_writer = None
|
| 551 |
-
|
| 552 |
-
try:
|
| 553 |
-
if filter_writer:
|
| 554 |
-
with contextlib.redirect_stdout(filter_writer), contextlib.redirect_stderr(filter_writer):
|
| 555 |
-
retry_call("upload_large_folder", args.upload_attempts, args.retry_backoff, lambda: api.upload_large_folder(**kwargs))
|
| 556 |
-
filter_writer.flush()
|
| 557 |
-
else:
|
| 558 |
-
retry_call("upload_large_folder", args.upload_attempts, args.retry_backoff, lambda: api.upload_large_folder(**kwargs))
|
| 559 |
-
finally:
|
| 560 |
-
if pbar:
|
| 561 |
-
try:
|
| 562 |
-
pbar.close()
|
| 563 |
-
except Exception:
|
| 564 |
-
pass
|
| 565 |
-
|
| 566 |
-
try:
|
| 567 |
-
_run_upload()
|
| 568 |
-
except Exception as upload_exc:
|
| 569 |
-
print(f"upload_large_folder failed: {upload_exc}; falling back to per-file upload with throttling.")
|
| 570 |
-
else:
|
| 571 |
-
print("upload_large_folder completed successfully.")
|
| 572 |
-
run_stats['planned'] = total_files
|
| 573 |
-
run_stats['bytes_planned'] = total_bytes
|
| 574 |
-
run_stats['uploaded'] = total_files
|
| 575 |
-
run_stats['bytes_uploaded'] = total_bytes
|
| 576 |
-
csv_uploaded_with_segments = True
|
| 577 |
-
return
|
| 578 |
-
|
| 579 |
-
# Otherwise, perform per-file upload and skip existing files
|
| 580 |
-
print("Performing per-file upload (skipping already uploaded files if any)...")
|
| 581 |
-
|
| 582 |
-
files: List[Tuple[Path, str]] = []
|
| 583 |
-
for p in segments_path.rglob("*"):
|
| 584 |
-
if p.is_file():
|
| 585 |
-
rel = p.relative_to(segments_path).as_posix()
|
| 586 |
-
repo_path = f"{args.segments_path_in_repo}/{rel}"
|
| 587 |
-
run_stats['bytes_planned'] += p.stat().st_size
|
| 588 |
-
skip = False
|
| 589 |
-
if repo_path in already_uploaded:
|
| 590 |
-
local_hash = None
|
| 591 |
-
checkpoint_hash = checkpoint_hashes.get(repo_path)
|
| 592 |
-
if checkpoint_hash:
|
| 593 |
-
local_hash = compute_sha256(p)
|
| 594 |
-
if local_hash != checkpoint_hash:
|
| 595 |
-
skip = False
|
| 596 |
-
print(f"Hash mismatch for {repo_path}; will re-upload.")
|
| 597 |
-
else:
|
| 598 |
-
skip = True
|
| 599 |
-
elif args.verify_remote and requests is not None:
|
| 600 |
-
local_size = p.stat().st_size
|
| 601 |
-
if rate_limiter:
|
| 602 |
-
rate_limiter.acquire("verify_remote")
|
| 603 |
-
remote_size, remote_etag = remote_file_info(repo_path)
|
| 604 |
-
if args.verify_etag and remote_etag:
|
| 605 |
-
# We do not compute a remote-matching hash locally here; rely on size and the checkpoint hash path.
|
| 606 |
-
skip = (remote_size is not None and local_size == remote_size)
|
| 607 |
-
else:
|
| 608 |
-
skip = (remote_size is not None and local_size == remote_size)
|
| 609 |
-
else:
|
| 610 |
-
skip = True
|
| 611 |
-
|
| 612 |
-
if skip:
|
| 613 |
-
run_stats['skipped'] += 1
|
| 614 |
-
append_progress(p, repo_path, 'skipped', 'already uploaded')
|
| 615 |
-
continue
|
| 616 |
-
files.append((p, repo_path))
|
| 617 |
-
|
| 618 |
-
run_stats['planned'] = len(files)
|
| 619 |
-
|
| 620 |
-
if args.resume_only:
|
| 621 |
-
print("Resume-only mode: no upload will be performed.")
|
| 622 |
-
for local_path, repo_path in files:
|
| 623 |
-
append_progress(local_path, repo_path, 'planned', 'resume-only')
|
| 624 |
-
return
|
| 625 |
-
|
| 626 |
-
print(f"Found {len(files)} files to upload; launching {args.max_workers} workers")
|
| 627 |
-
|
| 628 |
-
# Progress bar for per-file uploads (counts files, not bytes)
|
| 629 |
-
file_pbar = FilesProgressBar(total_files=len(files), logger=print)
|
| 630 |
-
|
| 631 |
-
def upload_one(pair: Tuple[Path, str]):
|
| 632 |
-
local_path, repo_path = pair
|
| 633 |
-
start = time.time()
|
| 634 |
-
try:
|
| 635 |
-
def do_upload():
|
| 636 |
-
if rate_limiter:
|
| 637 |
-
rate_limiter.acquire("file_upload")
|
| 638 |
-
api.upload_file(
|
| 639 |
-
path_or_fileobj=str(local_path),
|
| 640 |
-
path_in_repo=repo_path,
|
| 641 |
-
repo_id=repo_id,
|
| 642 |
-
repo_type="dataset",
|
| 643 |
-
commit_message=args.commit_message,
|
| 644 |
-
)
|
| 645 |
-
|
| 646 |
-
retry_call(f"upload_file:{repo_path}", args.upload_attempts, args.retry_backoff, do_upload)
|
| 647 |
-
except Exception as e:
|
| 648 |
-
elapsed = time.time() - start
|
| 649 |
-
# Log error to CSV for tracking
|
| 650 |
-
append_progress(local_path, repo_path, 'failed', str(e), elapsed)
|
| 651 |
-
# Update progress bar with error
|
| 652 |
-
file_pbar.update_file_error(1, f"{repo_path}: {str(e)[:60]}")
|
| 653 |
-
raise
|
| 654 |
-
else:
|
| 655 |
-
elapsed = time.time() - start
|
| 656 |
-
# Only log successful uploads to CSV (optional, can be verbose)
|
| 657 |
-
# append_progress(local_path, repo_path, 'uploaded', None, elapsed)
|
| 658 |
-
# Update progress bar
|
| 659 |
-
file_pbar.update_file(1)
|
| 660 |
-
return repo_path, local_path
|
| 661 |
-
|
| 662 |
-
try:
|
| 663 |
-
with concurrent.futures.ThreadPoolExecutor(max_workers=args.max_workers) as ex:
|
| 664 |
-
futures = [ex.submit(upload_one, f) for f in files]
|
| 665 |
-
for fut in concurrent.futures.as_completed(futures):
|
| 666 |
-
try:
|
| 667 |
-
repo_path, local_path = fut.result()
|
| 668 |
-
except Exception as e:
|
| 669 |
-
# Error already logged above, just continue
|
| 670 |
-
continue
|
| 671 |
-
finally:
|
| 672 |
-
file_pbar.close()
|
| 673 |
-
file_pbar.log_errors()
|
| 674 |
-
# mark as uploaded in checkpoint
|
| 675 |
-
if repo_path:
|
| 676 |
-
already_uploaded.add(repo_path)
|
| 677 |
-
try:
|
| 678 |
-
checkpoint_hashes[repo_path] = compute_sha256(local_path)
|
| 679 |
-
except Exception as e:
|
| 680 |
-
print(f"Warning: failed to hash {repo_path} after upload: {e}")
|
| 681 |
-
run_stats['uploaded'] += 1
|
| 682 |
-
try:
|
| 683 |
-
run_stats['bytes_uploaded'] += Path(local_path).stat().st_size
|
| 684 |
-
except Exception:
|
| 685 |
-
pass
|
| 686 |
-
save_checkpoint(already_uploaded, checkpoint_hashes)
|
| 687 |
-
if pbar is not None:
|
| 688 |
-
try:
|
| 689 |
-
pbar.close()
|
| 690 |
-
except Exception:
|
| 691 |
-
pass
|
| 692 |
-
|
| 693 |
-
retry_call("upload_segments", args.upload_attempts, args.retry_backoff, upload_segments)
|
| 694 |
-
|
| 695 |
-
if args.resume_only:
|
| 696 |
-
print("Resume-only mode completed; no uploads were performed.")
|
| 697 |
-
record_summary()
|
| 698 |
-
print()
|
| 699 |
-
print("Upload complete.")
|
| 700 |
-
print(f"Dataset URL: https://huggingface.co/datasets/{repo_id}")
|
| 701 |
-
return 0
|
| 702 |
-
|
| 703 |
-
if csv_path:
|
| 704 |
-
|
| 705 |
-
def upload_csv() -> None:
|
| 706 |
-
skip_csv = False
|
| 707 |
-
if args.resume and args.csv_path_in_repo in already_uploaded:
|
| 708 |
-
if args.verify_remote and requests is not None:
|
| 709 |
-
local_size = csv_path.stat().st_size
|
| 710 |
-
if rate_limiter:
|
| 711 |
-
rate_limiter.acquire("verify_remote")
|
| 712 |
-
remote_size, remote_etag = remote_file_info(args.csv_path_in_repo)
|
| 713 |
-
if args.verify_etag and remote_etag:
|
| 714 |
-
# no local etag available -> fall back to size check
|
| 715 |
-
if remote_size is not None and local_size == remote_size:
|
| 716 |
-
skip_csv = True
|
| 717 |
-
else:
|
| 718 |
-
if remote_size is not None and local_size == remote_size:
|
| 719 |
-
skip_csv = True
|
| 720 |
-
|
| 721 |
-
if skip_csv:
|
| 722 |
-
print(f"CSV already exists at {args.csv_path_in_repo}; skipping upload.")
|
| 723 |
-
run_stats['skipped'] += 1
|
| 724 |
-
append_progress(csv_path, args.csv_path_in_repo, 'skipped', 'already uploaded')
|
| 725 |
-
return
|
| 726 |
-
|
| 727 |
-
start = time.time()
|
| 728 |
-
try:
|
| 729 |
-
if rate_limiter:
|
| 730 |
-
rate_limiter.acquire("csv_upload")
|
| 731 |
-
api.upload_file(
|
| 732 |
-
path_or_fileobj=str(csv_path),
|
| 733 |
-
path_in_repo=args.csv_path_in_repo,
|
| 734 |
-
repo_id=repo_id,
|
| 735 |
-
repo_type="dataset",
|
| 736 |
-
commit_message=f"Upload CSV for {repo_id}",
|
| 737 |
-
)
|
| 738 |
-
except Exception as e:
|
| 739 |
-
elapsed = time.time() - start
|
| 740 |
-
run_stats['failed'] += 1
|
| 741 |
-
append_progress(csv_path, args.csv_path_in_repo, 'failed', str(e), elapsed)
|
| 742 |
-
raise
|
| 743 |
-
else:
|
| 744 |
-
elapsed = time.time() - start
|
| 745 |
-
run_stats['uploaded'] += 1
|
| 746 |
-
try:
|
| 747 |
-
run_stats['bytes_uploaded'] += csv_path.stat().st_size
|
| 748 |
-
except Exception:
|
| 749 |
-
pass
|
| 750 |
-
try:
|
| 751 |
-
checkpoint_hashes[args.csv_path_in_repo] = compute_sha256(csv_path)
|
| 752 |
-
save_checkpoint(already_uploaded.union({args.csv_path_in_repo}), checkpoint_hashes)
|
| 753 |
-
except Exception as e:
|
| 754 |
-
print(f"Warning: failed to update CSV checkpoint hash: {e}")
|
| 755 |
-
append_progress(csv_path, args.csv_path_in_repo, 'uploaded', None, elapsed)
|
| 756 |
-
|
| 757 |
-
retry_call("upload_csv", args.upload_attempts, args.retry_backoff, upload_csv)
|
| 758 |
-
|
| 759 |
-
print()
|
| 760 |
-
print("Upload complete.")
|
| 761 |
-
print(f"Dataset URL: https://huggingface.co/datasets/{repo_id}")
|
| 762 |
-
record_summary()
|
| 763 |
-
return 0
|
| 764 |
-
|
| 765 |
-
|
| 766 |
-
if __name__ == "__main__":
|
| 767 |
-
raise SystemExit(main())
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hf_bulk_upload_tool/upload_logic.py
DELETED
|
@@ -1,886 +0,0 @@
|
|
| 1 |
-
#!/usr/bin/env python3
|
| 2 |
-
"""
|
| 3 |
-
Upload logic refactored for GUI integration.
|
| 4 |
-
This module contains the core upload functionality extracted from upload_dataset.py
|
| 5 |
-
and refactored to accept a config dict and a logger callback instead of argparse.
|
| 6 |
-
"""
|
| 7 |
-
|
| 8 |
-
from __future__ import annotations
|
| 9 |
-
|
| 10 |
-
import hashlib
|
| 11 |
-
import io
|
| 12 |
-
import os
|
| 13 |
-
import sys
|
| 14 |
-
import shutil
|
| 15 |
-
import subprocess
|
| 16 |
-
import time
|
| 17 |
-
import contextlib
|
| 18 |
-
from pathlib import Path
|
| 19 |
-
import concurrent.futures
|
| 20 |
-
from typing import List, Tuple, Callable, Optional
|
| 21 |
-
import json
|
| 22 |
-
import csv
|
| 23 |
-
from datetime import datetime
|
| 24 |
-
from urllib.parse import quote
|
| 25 |
-
|
| 26 |
-
try:
|
| 27 |
-
import requests
|
| 28 |
-
except Exception:
|
| 29 |
-
requests = None
|
| 30 |
-
try:
|
| 31 |
-
from tqdm import tqdm
|
| 32 |
-
except Exception:
|
| 33 |
-
tqdm = None
|
| 34 |
-
|
| 35 |
-
try:
|
| 36 |
-
from sharding_utils import shard_directory, MAX_FILES_PER_DIR
|
| 37 |
-
from progress_bar_utils import ProgressFilter, FilesProgressBar
|
| 38 |
-
except ImportError:
|
| 39 |
-
# Fallback for relative imports during development
|
| 40 |
-
from .sharding_utils import shard_directory, MAX_FILES_PER_DIR
|
| 41 |
-
from .progress_bar_utils import ProgressFilter, FilesProgressBar
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
def configure_hf_env() -> None:
|
| 45 |
-
"""Configure Hugging Face environment variables."""
|
| 46 |
-
os.environ.setdefault("HF_HUB_ETAG_TIMEOUT", "20")
|
| 47 |
-
os.environ.setdefault("HF_HUB_DOWNLOAD_TIMEOUT", "120")
|
| 48 |
-
os.environ.setdefault("HF_XET_HIGH_PERFORMANCE", "1")
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
configure_hf_env()
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
def _ensure_std_streams() -> None:
|
| 55 |
-
"""Ensure std streams exist in windowed/frozen runs.
|
| 56 |
-
|
| 57 |
-
In PyInstaller windowed mode, sys.stderr/sys.stdout may be None.
|
| 58 |
-
Some libraries (including logging handlers inside huggingface_hub flow)
|
| 59 |
-
expect writable streams and crash otherwise.
|
| 60 |
-
"""
|
| 61 |
-
if sys.stdout is None:
|
| 62 |
-
sys.stdout = open(os.devnull, "w", encoding="utf-8")
|
| 63 |
-
if sys.stderr is None:
|
| 64 |
-
sys.stderr = open(os.devnull, "w", encoding="utf-8")
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
_ensure_std_streams()
|
| 68 |
-
|
| 69 |
-
from huggingface_hub import HfApi
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
class _CallbackWriter(io.TextIOBase):
|
| 73 |
-
"""File-like writer that forwards complete lines to a callback logger."""
|
| 74 |
-
|
| 75 |
-
def __init__(self, callback: Callable[[str], None]):
|
| 76 |
-
super().__init__()
|
| 77 |
-
self._callback = callback
|
| 78 |
-
self._buffer = ""
|
| 79 |
-
|
| 80 |
-
def write(self, s: str) -> int:
|
| 81 |
-
if not s:
|
| 82 |
-
return 0
|
| 83 |
-
self._buffer += s
|
| 84 |
-
while "\n" in self._buffer:
|
| 85 |
-
line, self._buffer = self._buffer.split("\n", 1)
|
| 86 |
-
if line.strip():
|
| 87 |
-
self._callback(line)
|
| 88 |
-
return len(s)
|
| 89 |
-
|
| 90 |
-
def flush(self) -> None:
|
| 91 |
-
if self._buffer.strip():
|
| 92 |
-
self._callback(self._buffer.strip())
|
| 93 |
-
self._buffer = ""
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
class RateLimiter:
|
| 97 |
-
"""Sliding window rate limiter for HF API (1000 requests per 5 minutes)."""
|
| 98 |
-
def __init__(self, max_requests: int = 1000, window_seconds: int = 300):
|
| 99 |
-
self.max_requests = max_requests
|
| 100 |
-
self.window_seconds = window_seconds
|
| 101 |
-
self.request_times: list[tuple[float, str]] = []
|
| 102 |
-
|
| 103 |
-
def acquire(self, request_type: str = "upload") -> None:
|
| 104 |
-
"""Wait if necessary to stay within rate limits, then register the request."""
|
| 105 |
-
now = time.time()
|
| 106 |
-
|
| 107 |
-
self.request_times = [
|
| 108 |
-
(ts, req_type) for ts, req_type in self.request_times
|
| 109 |
-
if now - ts < self.window_seconds
|
| 110 |
-
]
|
| 111 |
-
|
| 112 |
-
if len(self.request_times) >= self.max_requests:
|
| 113 |
-
oldest_ts = self.request_times[0][0]
|
| 114 |
-
wait_time = (oldest_ts + self.window_seconds) - now
|
| 115 |
-
if wait_time > 0:
|
| 116 |
-
time.sleep(wait_time)
|
| 117 |
-
self.request_times = []
|
| 118 |
-
|
| 119 |
-
self.request_times.append((now, request_type))
|
| 120 |
-
|
| 121 |
-
def get_window_usage(self) -> tuple[int, int]:
|
| 122 |
-
"""Return (requests_in_window, max_requests)."""
|
| 123 |
-
now = time.time()
|
| 124 |
-
self.request_times = [
|
| 125 |
-
(ts, req_type) for ts, req_type in self.request_times
|
| 126 |
-
if now - ts < self.window_seconds
|
| 127 |
-
]
|
| 128 |
-
return len(self.request_times), self.max_requests
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
def ensure_directory_link(link_path: Path, target_path: Path) -> None:
|
| 132 |
-
"""Create a directory link (Windows junction or Unix symlink)."""
|
| 133 |
-
if link_path.exists() or link_path.is_symlink():
|
| 134 |
-
try:
|
| 135 |
-
if link_path.resolve() == target_path.resolve():
|
| 136 |
-
return
|
| 137 |
-
except Exception:
|
| 138 |
-
pass
|
| 139 |
-
if link_path.is_symlink() or link_path.is_file():
|
| 140 |
-
link_path.unlink()
|
| 141 |
-
else:
|
| 142 |
-
shutil.rmtree(link_path)
|
| 143 |
-
|
| 144 |
-
link_path.parent.mkdir(parents=True, exist_ok=True)
|
| 145 |
-
|
| 146 |
-
if os.name == "nt":
|
| 147 |
-
try:
|
| 148 |
-
subprocess.run(
|
| 149 |
-
["cmd", "/c", "mklink", "/J", str(link_path), str(target_path)],
|
| 150 |
-
check=True,
|
| 151 |
-
capture_output=True,
|
| 152 |
-
text=True,
|
| 153 |
-
creationflags=subprocess.CREATE_NO_WINDOW,
|
| 154 |
-
)
|
| 155 |
-
return
|
| 156 |
-
except Exception:
|
| 157 |
-
pass
|
| 158 |
-
|
| 159 |
-
os.symlink(target_path, link_path, target_is_directory=True)
|
| 160 |
-
|
| 161 |
-
|
| 162 |
-
def prepare_staging_folder(staging_root: Path, segments_path: Path, csv_path: Path | None) -> Path:
|
| 163 |
-
"""Build a persistent staging tree with the repo layout expected by HF."""
|
| 164 |
-
staging_root.mkdir(parents=True, exist_ok=True)
|
| 165 |
-
|
| 166 |
-
audio_link = staging_root / "audio"
|
| 167 |
-
ensure_directory_link(audio_link, segments_path)
|
| 168 |
-
|
| 169 |
-
if csv_path is not None:
|
| 170 |
-
csv_dest = staging_root / "index" / "detections.csv"
|
| 171 |
-
csv_dest.parent.mkdir(parents=True, exist_ok=True)
|
| 172 |
-
shutil.copy2(csv_path, csv_dest)
|
| 173 |
-
|
| 174 |
-
return staging_root
|
| 175 |
-
|
| 176 |
-
|
| 177 |
-
def _inventory_cache_path(checkpoint_dir: Path, segments_path: Path) -> Path:
|
| 178 |
-
"""Return cache file path for local inventory of a segments directory."""
|
| 179 |
-
key = hashlib.sha1(str(segments_path).encode("utf-8")).hexdigest()[:12]
|
| 180 |
-
return checkpoint_dir / f"inventory_{key}.json"
|
| 181 |
-
|
| 182 |
-
|
| 183 |
-
def _scan_species_dir(species_dir: Path) -> tuple[int, int]:
|
| 184 |
-
"""Scan one species directory and return (file_count, total_bytes)."""
|
| 185 |
-
count = 0
|
| 186 |
-
total_bytes = 0
|
| 187 |
-
for p in species_dir.rglob("*"):
|
| 188 |
-
if p.is_file():
|
| 189 |
-
count += 1
|
| 190 |
-
try:
|
| 191 |
-
total_bytes += p.stat().st_size
|
| 192 |
-
except Exception:
|
| 193 |
-
pass
|
| 194 |
-
return count, total_bytes
|
| 195 |
-
|
| 196 |
-
|
| 197 |
-
def _build_species_inventory(segments_path: Path, checkpoint_dir: Path, logger: Callable[[str], None]) -> list[dict]:
|
| 198 |
-
"""Build per-species inventory with lightweight persistent cache.
|
| 199 |
-
|
| 200 |
-
Cache key is based on top-level species folder mtime. If folder mtime changed,
|
| 201 |
-
species is rescanned. This preserves correctness while avoiding repeated full scans
|
| 202 |
-
across reruns.
|
| 203 |
-
"""
|
| 204 |
-
cache_path = _inventory_cache_path(checkpoint_dir, segments_path)
|
| 205 |
-
cached_species: dict = {}
|
| 206 |
-
|
| 207 |
-
if cache_path.exists():
|
| 208 |
-
try:
|
| 209 |
-
with open(cache_path, "r", encoding="utf-8") as fh:
|
| 210 |
-
payload = json.load(fh)
|
| 211 |
-
cached_species = payload.get("species", {}) if isinstance(payload, dict) else {}
|
| 212 |
-
except Exception:
|
| 213 |
-
cached_species = {}
|
| 214 |
-
|
| 215 |
-
inventory: list[dict] = []
|
| 216 |
-
cache_hits = 0
|
| 217 |
-
total_species = 0
|
| 218 |
-
new_cache_species: dict = {}
|
| 219 |
-
|
| 220 |
-
for species_dir in sorted(segments_path.iterdir()):
|
| 221 |
-
if not species_dir.is_dir():
|
| 222 |
-
continue
|
| 223 |
-
total_species += 1
|
| 224 |
-
species_name = species_dir.name
|
| 225 |
-
try:
|
| 226 |
-
mtime_ns = species_dir.stat().st_mtime_ns
|
| 227 |
-
except Exception:
|
| 228 |
-
mtime_ns = 0
|
| 229 |
-
|
| 230 |
-
cached = cached_species.get(species_name)
|
| 231 |
-
if isinstance(cached, dict) and int(cached.get("mtime_ns", -1)) == int(mtime_ns):
|
| 232 |
-
file_count = int(cached.get("file_count", 0))
|
| 233 |
-
total_bytes = int(cached.get("total_bytes", 0))
|
| 234 |
-
cache_hits += 1
|
| 235 |
-
else:
|
| 236 |
-
file_count, total_bytes = _scan_species_dir(species_dir)
|
| 237 |
-
|
| 238 |
-
entry = {
|
| 239 |
-
"species_name": species_name,
|
| 240 |
-
"path": species_dir,
|
| 241 |
-
"file_count": file_count,
|
| 242 |
-
"total_bytes": total_bytes,
|
| 243 |
-
"mtime_ns": int(mtime_ns),
|
| 244 |
-
}
|
| 245 |
-
inventory.append(entry)
|
| 246 |
-
|
| 247 |
-
new_cache_species[species_name] = {
|
| 248 |
-
"mtime_ns": int(mtime_ns),
|
| 249 |
-
"file_count": int(file_count),
|
| 250 |
-
"total_bytes": int(total_bytes),
|
| 251 |
-
}
|
| 252 |
-
|
| 253 |
-
try:
|
| 254 |
-
cache_path.parent.mkdir(parents=True, exist_ok=True)
|
| 255 |
-
with open(cache_path, "w", encoding="utf-8") as fh:
|
| 256 |
-
json.dump(
|
| 257 |
-
{
|
| 258 |
-
"segments_root": str(segments_path),
|
| 259 |
-
"updated_at": datetime.utcnow().isoformat() + "Z",
|
| 260 |
-
"species": new_cache_species,
|
| 261 |
-
},
|
| 262 |
-
fh,
|
| 263 |
-
indent=2,
|
| 264 |
-
)
|
| 265 |
-
except Exception as e:
|
| 266 |
-
logger(f"Warning: failed to save inventory cache {cache_path}: {e}")
|
| 267 |
-
|
| 268 |
-
logger(f"[Scan] Species inventory cache hit: {cache_hits}/{total_species}")
|
| 269 |
-
return inventory
|
| 270 |
-
|
| 271 |
-
|
| 272 |
-
def _prepare_sharded_staging(
|
| 273 |
-
staging_root: Path,
|
| 274 |
-
segments_path: Path,
|
| 275 |
-
csv_path: Path | None,
|
| 276 |
-
logger: Callable[[str], None],
|
| 277 |
-
inventory: list[dict] | None = None,
|
| 278 |
-
) -> Path:
|
| 279 |
-
"""Prepare staging folder with automatic sharding for large species directories.
|
| 280 |
-
|
| 281 |
-
Detects species folders with >9500 files and automatically shards them into
|
| 282 |
-
numbered subdirectories to comply with HF's 10k files per directory limit.
|
| 283 |
-
Validation tools can transparently read shards via sharding_utils.
|
| 284 |
-
|
| 285 |
-
Args:
|
| 286 |
-
staging_root: Root directory for staging
|
| 287 |
-
segments_path: Path to source segments (species directories)
|
| 288 |
-
csv_path: Optional path to CSV to copy
|
| 289 |
-
logger: Callback function for logging messages
|
| 290 |
-
|
| 291 |
-
Returns:
|
| 292 |
-
Path to the prepared staging root directory
|
| 293 |
-
"""
|
| 294 |
-
staging_root.mkdir(parents=True, exist_ok=True)
|
| 295 |
-
audio_staging = staging_root / "audio"
|
| 296 |
-
audio_staging.mkdir(parents=True, exist_ok=True)
|
| 297 |
-
|
| 298 |
-
sharded_species = []
|
| 299 |
-
|
| 300 |
-
# Check each species folder for sharding using precomputed inventory when available
|
| 301 |
-
source_items = inventory
|
| 302 |
-
if source_items is None:
|
| 303 |
-
source_items = []
|
| 304 |
-
for species_dir in sorted(segments_path.iterdir()):
|
| 305 |
-
if species_dir.is_dir():
|
| 306 |
-
source_items.append(
|
| 307 |
-
{
|
| 308 |
-
"species_name": species_dir.name,
|
| 309 |
-
"path": species_dir,
|
| 310 |
-
"file_count": 0,
|
| 311 |
-
}
|
| 312 |
-
)
|
| 313 |
-
|
| 314 |
-
for item in source_items:
|
| 315 |
-
species_dir = Path(item["path"])
|
| 316 |
-
file_count = int(item.get("file_count", 0))
|
| 317 |
-
|
| 318 |
-
if file_count > MAX_FILES_PER_DIR:
|
| 319 |
-
# Need to shard this species
|
| 320 |
-
logger(f"[Sharding] {species_dir.name} exceeds {MAX_FILES_PER_DIR} files, dividing into shards...")
|
| 321 |
-
try:
|
| 322 |
-
shards = shard_directory(species_dir, audio_staging)
|
| 323 |
-
sharded_species.append((species_dir.name, len(shards)))
|
| 324 |
-
logger(f"[Sharding] Created {len(shards)} shards for {species_dir.name}")
|
| 325 |
-
except Exception as e:
|
| 326 |
-
logger(f"[Sharding] ERROR sharding {species_dir.name}: {e}")
|
| 327 |
-
logger(f"[Sharding] Falling back to single directory link")
|
| 328 |
-
ensure_directory_link(audio_staging / species_dir.name, species_dir)
|
| 329 |
-
else:
|
| 330 |
-
# Small species, just link it
|
| 331 |
-
ensure_directory_link(audio_staging / species_dir.name, species_dir)
|
| 332 |
-
|
| 333 |
-
# Log sharding summary
|
| 334 |
-
if sharded_species:
|
| 335 |
-
logger(f"")
|
| 336 |
-
logger(f"[Sharding] Summary: {len(sharded_species)} species were divided")
|
| 337 |
-
for species_name, num_shards in sharded_species:
|
| 338 |
-
logger(f" - {species_name}: {num_shards} shards")
|
| 339 |
-
logger(f"[Sharding] Validation tools will transparently read all shards as single species")
|
| 340 |
-
logger(f"")
|
| 341 |
-
|
| 342 |
-
# Copy CSV if provided
|
| 343 |
-
if csv_path is not None:
|
| 344 |
-
csv_dest = staging_root / "index" / "detections.csv"
|
| 345 |
-
csv_dest.parent.mkdir(parents=True, exist_ok=True)
|
| 346 |
-
shutil.copy2(csv_path, csv_dest)
|
| 347 |
-
|
| 348 |
-
return staging_root
|
| 349 |
-
|
| 350 |
-
|
| 351 |
-
def retry_call(label: str, attempts: int, backoff: float, func, logger: Callable[[str], None], stop_event=None):
|
| 352 |
-
"""Execute a function with retry logic, logging via callback."""
|
| 353 |
-
last_error: Exception | None = None
|
| 354 |
-
for attempt in range(1, attempts + 1):
|
| 355 |
-
# If cancellation requested, abort immediately
|
| 356 |
-
try:
|
| 357 |
-
if stop_event is not None and getattr(stop_event, 'is_set', lambda: False)():
|
| 358 |
-
raise Exception(f"{label} cancelled by user")
|
| 359 |
-
except Exception:
|
| 360 |
-
raise
|
| 361 |
-
try:
|
| 362 |
-
logger(f"[{label}] attempt {attempt}/{attempts}")
|
| 363 |
-
return func()
|
| 364 |
-
except Exception as exc:
|
| 365 |
-
last_error = exc
|
| 366 |
-
if attempt >= attempts:
|
| 367 |
-
break
|
| 368 |
-
|
| 369 |
-
wait_seconds = backoff * attempt
|
| 370 |
-
try:
|
| 371 |
-
resp = getattr(exc, 'response', None)
|
| 372 |
-
if resp is not None:
|
| 373 |
-
status = getattr(resp, 'status_code', None)
|
| 374 |
-
if status == 429:
|
| 375 |
-
ra = resp.headers.get('Retry-After') if hasattr(resp, 'headers') else None
|
| 376 |
-
if ra:
|
| 377 |
-
try:
|
| 378 |
-
wait_seconds = max(wait_seconds, int(ra))
|
| 379 |
-
except Exception:
|
| 380 |
-
wait_seconds = max(wait_seconds, backoff * attempt * 4)
|
| 381 |
-
else:
|
| 382 |
-
wait_seconds = max(wait_seconds, backoff * attempt * 4)
|
| 383 |
-
elif '429' in str(exc) or 'Too Many Requests' in str(exc):
|
| 384 |
-
wait_seconds = max(wait_seconds, backoff * attempt * 4)
|
| 385 |
-
except Exception:
|
| 386 |
-
pass
|
| 387 |
-
|
| 388 |
-
logger(f"[{label}] attempt {attempt} failed: {exc}")
|
| 389 |
-
logger(f"[{label}] retrying in {wait_seconds:.0f}s")
|
| 390 |
-
time.sleep(wait_seconds)
|
| 391 |
-
|
| 392 |
-
raise Exception(f"{label} failed after {attempts} attempts: {last_error}")
|
| 393 |
-
|
| 394 |
-
|
| 395 |
-
def run_upload_logic(config: dict, logger: Callable[[str], None]) -> int:
|
| 396 |
-
"""
|
| 397 |
-
Core upload logic refactored for GUI integration.
|
| 398 |
-
|
| 399 |
-
Args:
|
| 400 |
-
config: Dictionary with keys:
|
| 401 |
-
- hf_token: str (required)
|
| 402 |
-
- repo_id: str (required, format: "username/repo-name")
|
| 403 |
-
- segments: str or Path (required, directory with audio files)
|
| 404 |
-
- csv: str or Path (optional, path to CSV file)
|
| 405 |
-
- segments_path_in_repo: str (default: "audio")
|
| 406 |
-
- csv_path_in_repo: str (default: "index/detections.csv")
|
| 407 |
-
- private: bool (default: True)
|
| 408 |
-
- resume: bool (default: False)
|
| 409 |
-
- verify_remote: bool (default: False)
|
| 410 |
-
- verify_etag: bool (default: False)
|
| 411 |
-
- dry_run: bool (default: False)
|
| 412 |
-
- rate_limit_aware: bool (default: True)
|
| 413 |
-
- rate_limit_max_requests: int (default: 950)
|
| 414 |
-
- rate_limit_window: int (default: 300)
|
| 415 |
-
- create_repo_attempts: int (default: 3)
|
| 416 |
-
- upload_attempts: int (default: 3)
|
| 417 |
-
- retry_backoff: float (default: 5.0)
|
| 418 |
-
- max_workers: int (default: 1)
|
| 419 |
-
- checkpoint_dir: str or Path (default: .checkpoints)
|
| 420 |
-
- commit_message: str (default: "Upload BirdNET segments")
|
| 421 |
-
|
| 422 |
-
logger: Callable that accepts a string message (for GUI textbox updates)
|
| 423 |
-
|
| 424 |
-
Returns:
|
| 425 |
-
0 on success, raises Exception on failure
|
| 426 |
-
"""
|
| 427 |
-
|
| 428 |
-
# Extract and validate config
|
| 429 |
-
token = config.get("hf_token")
|
| 430 |
-
if not token:
|
| 431 |
-
raise ValueError("hf_token is required")
|
| 432 |
-
|
| 433 |
-
repo_id = config.get("repo_id")
|
| 434 |
-
if not repo_id:
|
| 435 |
-
raise ValueError("repo_id is required")
|
| 436 |
-
|
| 437 |
-
segments = config.get("segments")
|
| 438 |
-
if not segments:
|
| 439 |
-
raise ValueError("segments is required")
|
| 440 |
-
|
| 441 |
-
segments_path = Path(segments).expanduser().resolve()
|
| 442 |
-
if not segments_path.is_dir():
|
| 443 |
-
raise ValueError(f"Segments path does not exist or is not a directory: {segments_path}")
|
| 444 |
-
|
| 445 |
-
csv_path = None
|
| 446 |
-
if config.get("csv"):
|
| 447 |
-
csv_path = Path(config.get("csv")).expanduser().resolve()
|
| 448 |
-
if not csv_path.is_file():
|
| 449 |
-
raise ValueError(f"CSV path does not exist or is not a file: {csv_path}")
|
| 450 |
-
|
| 451 |
-
# Config parameters
|
| 452 |
-
segments_path_in_repo = config.get("segments_path_in_repo", "audio")
|
| 453 |
-
csv_path_in_repo = config.get("csv_path_in_repo", "index/detections.csv")
|
| 454 |
-
is_private = config.get("private", True)
|
| 455 |
-
resume = config.get("resume", False)
|
| 456 |
-
verify_remote = config.get("verify_remote", False)
|
| 457 |
-
verify_etag = config.get("verify_etag", False)
|
| 458 |
-
dry_run = config.get("dry_run", False)
|
| 459 |
-
rate_limit_aware = config.get("rate_limit_aware", True)
|
| 460 |
-
rate_limit_max_requests = config.get("rate_limit_max_requests", 950)
|
| 461 |
-
rate_limit_window = config.get("rate_limit_window", 300)
|
| 462 |
-
create_repo_attempts = config.get("create_repo_attempts", 3)
|
| 463 |
-
upload_attempts = config.get("upload_attempts", 3)
|
| 464 |
-
retry_backoff = config.get("retry_backoff", 5.0)
|
| 465 |
-
max_workers = config.get("max_workers", 1)
|
| 466 |
-
checkpoint_dir = Path(config.get("checkpoint_dir", Path(__file__).parent / ".checkpoints"))
|
| 467 |
-
commit_message = config.get("commit_message", "Upload BirdNET segments")
|
| 468 |
-
|
| 469 |
-
# Log configuration
|
| 470 |
-
logger(f"Repository: {repo_id}")
|
| 471 |
-
logger(f"Segments: {segments_path}")
|
| 472 |
-
logger(f"Segments destination: {segments_path_in_repo}")
|
| 473 |
-
|
| 474 |
-
csv_uploaded_with_segments = False
|
| 475 |
-
if csv_path and not csv_uploaded_with_segments:
|
| 476 |
-
logger(f"CSV: {csv_path}")
|
| 477 |
-
logger(f"CSV destination: {csv_path_in_repo}")
|
| 478 |
-
|
| 479 |
-
logger(f"Visibility: {'private' if is_private else 'public'}")
|
| 480 |
-
|
| 481 |
-
if dry_run:
|
| 482 |
-
logger("Dry run selected. No network calls will be made.")
|
| 483 |
-
return 0
|
| 484 |
-
|
| 485 |
-
# Cancellation event (optional). GUI should pass a threading.Event via config['stop_event']
|
| 486 |
-
stop_event = config.get('stop_event')
|
| 487 |
-
# Suppress console progress (GUI mode): when True, do not create tqdm bars
|
| 488 |
-
suppress_console = config.get('suppress_console_progress', False)
|
| 489 |
-
|
| 490 |
-
api = HfApi(token=token)
|
| 491 |
-
|
| 492 |
-
# Initialize rate limiter
|
| 493 |
-
rate_limiter = None
|
| 494 |
-
if rate_limit_aware:
|
| 495 |
-
rate_limiter = RateLimiter(max_requests=rate_limit_max_requests, window_seconds=rate_limit_window)
|
| 496 |
-
logger(f"[RateLimit] Enabled: max {rate_limit_max_requests} requests per {rate_limit_window}s")
|
| 497 |
-
|
| 498 |
-
def list_existing_files() -> set:
|
| 499 |
-
try:
|
| 500 |
-
logger("Listing files in target repo to determine already-uploaded files...")
|
| 501 |
-
if rate_limiter:
|
| 502 |
-
rate_limiter.acquire("list_repo_files")
|
| 503 |
-
files = api.list_repo_files(repo_id=repo_id, repo_type="dataset")
|
| 504 |
-
return set(files or [])
|
| 505 |
-
except Exception as e:
|
| 506 |
-
logger(f"Warning: could not list repo files: {e}")
|
| 507 |
-
return set()
|
| 508 |
-
|
| 509 |
-
# Checkpoint management
|
| 510 |
-
checkpoint_dir.mkdir(parents=True, exist_ok=True)
|
| 511 |
-
checkpoint_file = checkpoint_dir / f"{repo_id.replace('/', '__')}.json"
|
| 512 |
-
progress_log_path = checkpoint_dir / 'progress.csv'
|
| 513 |
-
|
| 514 |
-
def load_checkpoint() -> tuple[set, dict[str, str]]:
|
| 515 |
-
if not checkpoint_file.exists():
|
| 516 |
-
return set(), {}
|
| 517 |
-
try:
|
| 518 |
-
with open(checkpoint_file, 'r', encoding='utf-8') as fh:
|
| 519 |
-
data = json.load(fh)
|
| 520 |
-
if isinstance(data, list):
|
| 521 |
-
return set(data), {}
|
| 522 |
-
uploaded = set(data.get('uploaded', []))
|
| 523 |
-
file_hashes = data.get('file_hashes', {})
|
| 524 |
-
if not isinstance(file_hashes, dict):
|
| 525 |
-
file_hashes = {}
|
| 526 |
-
return uploaded, {str(k): str(v) for k, v in file_hashes.items()}
|
| 527 |
-
except Exception as e:
|
| 528 |
-
logger(f"Warning: failed to load checkpoint {checkpoint_file}: {e}")
|
| 529 |
-
return set(), {}
|
| 530 |
-
|
| 531 |
-
def save_checkpoint(uploaded_set: set, file_hashes: dict[str, str]) -> None:
|
| 532 |
-
try:
|
| 533 |
-
with open(checkpoint_file, 'w', encoding='utf-8') as fh:
|
| 534 |
-
json.dump({'uploaded': sorted(list(uploaded_set)), 'file_hashes': file_hashes}, fh, indent=2)
|
| 535 |
-
except Exception as e:
|
| 536 |
-
logger(f"Warning: failed to save checkpoint {checkpoint_file}: {e}")
|
| 537 |
-
|
| 538 |
-
def compute_sha256(file_path: Path) -> str:
|
| 539 |
-
digest = hashlib.sha256()
|
| 540 |
-
with open(file_path, 'rb') as fh:
|
| 541 |
-
for chunk in iter(lambda: fh.read(1024 * 1024), b''):
|
| 542 |
-
digest.update(chunk)
|
| 543 |
-
return digest.hexdigest()
|
| 544 |
-
|
| 545 |
-
def remote_file_info(repo_path: str) -> Tuple[int | None, str | None]:
|
| 546 |
-
"""Return (size, etag) for a repo file via HTTP HEAD, or (None, None)."""
|
| 547 |
-
if requests is None:
|
| 548 |
-
return None, None
|
| 549 |
-
quoted = '/'.join(quote(p) for p in repo_path.split('/'))
|
| 550 |
-
url = f"https://huggingface.co/datasets/{repo_id}/resolve/main/{quoted}"
|
| 551 |
-
try:
|
| 552 |
-
r = requests.head(url, allow_redirects=True, timeout=10)
|
| 553 |
-
if r.status_code >= 400:
|
| 554 |
-
return None, None
|
| 555 |
-
cl = r.headers.get('content-length')
|
| 556 |
-
etag = r.headers.get('etag')
|
| 557 |
-
size = int(cl) if cl is not None else None
|
| 558 |
-
return size, etag
|
| 559 |
-
except Exception:
|
| 560 |
-
return None, None
|
| 561 |
-
|
| 562 |
-
def append_progress(local_path: Path | None, repo_path: str, status: str, msg: str | None = None, elapsed: float | None = None) -> None:
|
| 563 |
-
header = ['timestamp','local_path','repo_path','status','message','elapsed_seconds']
|
| 564 |
-
row = [time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()), str(local_path) if local_path else '', repo_path, status, (msg or '').replace('\n',' '), f"{elapsed:.2f}" if elapsed is not None else '']
|
| 565 |
-
write_header = not progress_log_path.exists()
|
| 566 |
-
try:
|
| 567 |
-
with open(progress_log_path, 'a', newline='', encoding='utf-8') as fh:
|
| 568 |
-
writer = csv.writer(fh)
|
| 569 |
-
if write_header:
|
| 570 |
-
writer.writerow(header)
|
| 571 |
-
writer.writerow(row)
|
| 572 |
-
except Exception as e:
|
| 573 |
-
logger(f"Warning: failed to write progress log {progress_log_path}: {e}")
|
| 574 |
-
|
| 575 |
-
run_stats = {
|
| 576 |
-
'planned': 0,
|
| 577 |
-
'skipped': 0,
|
| 578 |
-
'uploaded': 0,
|
| 579 |
-
'failed': 0,
|
| 580 |
-
'bytes_planned': 0,
|
| 581 |
-
'bytes_uploaded': 0,
|
| 582 |
-
'started_at': time.time(),
|
| 583 |
-
}
|
| 584 |
-
|
| 585 |
-
def record_summary() -> None:
|
| 586 |
-
elapsed = time.time() - run_stats['started_at']
|
| 587 |
-
summary_message = (
|
| 588 |
-
f"planned={run_stats['planned']} skipped={run_stats['skipped']} "
|
| 589 |
-
f"uploaded={run_stats['uploaded']} failed={run_stats['failed']} "
|
| 590 |
-
f"bytes_planned={run_stats['bytes_planned']} bytes_uploaded={run_stats['bytes_uploaded']}"
|
| 591 |
-
)
|
| 592 |
-
append_progress(None, '', 'summary', summary_message, elapsed)
|
| 593 |
-
try:
|
| 594 |
-
summary_json = checkpoint_dir / f"{repo_id.replace('/', '__')}.summary.json"
|
| 595 |
-
with open(summary_json, 'w', encoding='utf-8') as fh:
|
| 596 |
-
json.dump({
|
| 597 |
-
'repo_id': repo_id,
|
| 598 |
-
'planned': run_stats['planned'],
|
| 599 |
-
'skipped': run_stats['skipped'],
|
| 600 |
-
'uploaded': run_stats['uploaded'],
|
| 601 |
-
'failed': run_stats['failed'],
|
| 602 |
-
'bytes_planned': run_stats['bytes_planned'],
|
| 603 |
-
'bytes_uploaded': run_stats['bytes_uploaded'],
|
| 604 |
-
'elapsed_seconds': round(elapsed, 2),
|
| 605 |
-
}, fh, indent=2)
|
| 606 |
-
except Exception as e:
|
| 607 |
-
logger(f"Warning: failed to write summary json: {e}")
|
| 608 |
-
|
| 609 |
-
# Create repo
|
| 610 |
-
def create_repo() -> None:
|
| 611 |
-
api.create_repo(repo_id=repo_id, repo_type="dataset", private=is_private, exist_ok=True)
|
| 612 |
-
|
| 613 |
-
# Create repo (with retry and cancellation support)
|
| 614 |
-
retry_call("create_repo", create_repo_attempts, retry_backoff, create_repo, logger, stop_event)
|
| 615 |
-
|
| 616 |
-
# Load checkpoints
|
| 617 |
-
existing_files: set = set()
|
| 618 |
-
checkpoint_uploaded: set = set()
|
| 619 |
-
checkpoint_hashes: dict[str, str] = {}
|
| 620 |
-
existing_files_loaded = False
|
| 621 |
-
if resume:
|
| 622 |
-
checkpoint_uploaded, checkpoint_hashes = load_checkpoint()
|
| 623 |
-
|
| 624 |
-
already_uploaded = existing_files.union(checkpoint_uploaded)
|
| 625 |
-
|
| 626 |
-
def ensure_existing_files_loaded() -> None:
|
| 627 |
-
nonlocal existing_files_loaded, existing_files, already_uploaded
|
| 628 |
-
if not resume or existing_files_loaded:
|
| 629 |
-
return
|
| 630 |
-
existing_files = list_existing_files()
|
| 631 |
-
existing_files_loaded = True
|
| 632 |
-
already_uploaded = existing_files.union(checkpoint_uploaded)
|
| 633 |
-
|
| 634 |
-
def upload_segments() -> None:
|
| 635 |
-
nonlocal csv_uploaded_with_segments
|
| 636 |
-
|
| 637 |
-
if hasattr(api, "upload_large_folder"):
|
| 638 |
-
staging_root = checkpoint_dir / "staging" / repo_id.replace("/", "__")
|
| 639 |
-
logger(f"Preparing persistent staging folder at {staging_root} ...")
|
| 640 |
-
logger(f"Checking for large species directories that need sharding...\n")
|
| 641 |
-
inventory = _build_species_inventory(segments_path, checkpoint_dir, logger)
|
| 642 |
-
staged_root = _prepare_sharded_staging(staging_root, segments_path, csv_path, logger, inventory=inventory)
|
| 643 |
-
|
| 644 |
-
total_files = sum(int(item.get("file_count", 0)) for item in inventory)
|
| 645 |
-
total_bytes = sum(int(item.get("total_bytes", 0)) for item in inventory)
|
| 646 |
-
if csv_path is not None:
|
| 647 |
-
total_files += 1
|
| 648 |
-
try:
|
| 649 |
-
total_bytes += csv_path.stat().st_size
|
| 650 |
-
except Exception:
|
| 651 |
-
pass
|
| 652 |
-
|
| 653 |
-
logger(f"Found {total_files} staged files to upload with upload_large_folder()")
|
| 654 |
-
|
| 655 |
-
kwargs = {
|
| 656 |
-
"folder_path": str(staged_root),
|
| 657 |
-
"repo_id": repo_id,
|
| 658 |
-
"repo_type": "dataset",
|
| 659 |
-
"private": is_private,
|
| 660 |
-
"num_workers": max(1, max_workers),
|
| 661 |
-
# Keep HF periodic reports disabled; GUI receives compact status via ProgressFilter.
|
| 662 |
-
"print_report": False,
|
| 663 |
-
}
|
| 664 |
-
|
| 665 |
-
def _run_upload():
|
| 666 |
-
# Create progress bar and filter
|
| 667 |
-
if tqdm is not None and not suppress_console:
|
| 668 |
-
pbar = tqdm(total=total_files, unit="file", unit_scale=False, desc="Uploading", leave=True)
|
| 669 |
-
filter_writer = ProgressFilter(pbar, logger)
|
| 670 |
-
else:
|
| 671 |
-
pbar = None
|
| 672 |
-
filter_writer = ProgressFilter(None, logger)
|
| 673 |
-
|
| 674 |
-
try:
|
| 675 |
-
if filter_writer:
|
| 676 |
-
with contextlib.redirect_stdout(filter_writer), contextlib.redirect_stderr(filter_writer):
|
| 677 |
-
retry_call("upload_large_folder", upload_attempts, retry_backoff, lambda: api.upload_large_folder(**kwargs), logger, stop_event)
|
| 678 |
-
filter_writer.flush()
|
| 679 |
-
else:
|
| 680 |
-
retry_call("upload_large_folder", upload_attempts, retry_backoff, lambda: api.upload_large_folder(**kwargs), logger, stop_event)
|
| 681 |
-
finally:
|
| 682 |
-
if pbar:
|
| 683 |
-
try:
|
| 684 |
-
pbar.close()
|
| 685 |
-
except Exception:
|
| 686 |
-
pass
|
| 687 |
-
|
| 688 |
-
try:
|
| 689 |
-
# Check cancellation before starting heavy upload
|
| 690 |
-
if stop_event is not None and getattr(stop_event, 'is_set', lambda: False)():
|
| 691 |
-
raise Exception("Upload cancelled by user before starting batch upload")
|
| 692 |
-
_run_upload()
|
| 693 |
-
except Exception as upload_exc:
|
| 694 |
-
logger(f"upload_large_folder failed: {upload_exc}; falling back to per-file upload with throttling.")
|
| 695 |
-
else:
|
| 696 |
-
logger("upload_large_folder completed successfully.")
|
| 697 |
-
run_stats['planned'] = total_files
|
| 698 |
-
run_stats['bytes_planned'] = total_bytes
|
| 699 |
-
run_stats['uploaded'] = total_files
|
| 700 |
-
run_stats['bytes_uploaded'] = total_bytes
|
| 701 |
-
csv_uploaded_with_segments = True
|
| 702 |
-
return
|
| 703 |
-
|
| 704 |
-
# Per-file fallback
|
| 705 |
-
logger("Performing per-file upload (skipping already uploaded files if any)...")
|
| 706 |
-
|
| 707 |
-
# Remote repo listing is only needed for per-file resume logic.
|
| 708 |
-
ensure_existing_files_loaded()
|
| 709 |
-
|
| 710 |
-
files: List[Tuple[Path, str]] = []
|
| 711 |
-
for p in segments_path.rglob("*"):
|
| 712 |
-
if p.is_file():
|
| 713 |
-
rel = p.relative_to(segments_path).as_posix()
|
| 714 |
-
repo_path = f"{segments_path_in_repo}/{rel}"
|
| 715 |
-
run_stats['bytes_planned'] += p.stat().st_size
|
| 716 |
-
skip = False
|
| 717 |
-
if repo_path in already_uploaded:
|
| 718 |
-
checkpoint_hash = checkpoint_hashes.get(repo_path)
|
| 719 |
-
if checkpoint_hash:
|
| 720 |
-
local_hash = compute_sha256(p)
|
| 721 |
-
if local_hash != checkpoint_hash:
|
| 722 |
-
skip = False
|
| 723 |
-
logger(f"Hash mismatch for {repo_path}; will re-upload.")
|
| 724 |
-
else:
|
| 725 |
-
skip = True
|
| 726 |
-
elif verify_remote and requests is not None:
|
| 727 |
-
local_size = p.stat().st_size
|
| 728 |
-
if rate_limiter:
|
| 729 |
-
rate_limiter.acquire("verify_remote")
|
| 730 |
-
remote_size, remote_etag = remote_file_info(repo_path)
|
| 731 |
-
if verify_etag and remote_etag:
|
| 732 |
-
skip = (remote_size is not None and local_size == remote_size)
|
| 733 |
-
else:
|
| 734 |
-
skip = (remote_size is not None and local_size == remote_size)
|
| 735 |
-
else:
|
| 736 |
-
skip = True
|
| 737 |
-
|
| 738 |
-
if skip:
|
| 739 |
-
run_stats['skipped'] += 1
|
| 740 |
-
append_progress(p, repo_path, 'skipped', 'already uploaded')
|
| 741 |
-
continue
|
| 742 |
-
files.append((p, repo_path))
|
| 743 |
-
|
| 744 |
-
run_stats['planned'] = len(files)
|
| 745 |
-
logger(f"Found {len(files)} files to upload; launching {max_workers} workers")
|
| 746 |
-
|
| 747 |
-
# Progress bar for per-file uploads (counts files, not bytes)
|
| 748 |
-
file_pbar = FilesProgressBar(total_files=len(files), logger=logger, suppress_console=suppress_console)
|
| 749 |
-
|
| 750 |
-
def upload_one(pair: Tuple[Path, str]):
|
| 751 |
-
local_path, repo_path = pair
|
| 752 |
-
start = time.time()
|
| 753 |
-
try:
|
| 754 |
-
def do_upload():
|
| 755 |
-
if rate_limiter:
|
| 756 |
-
rate_limiter.acquire("file_upload")
|
| 757 |
-
api.upload_file(
|
| 758 |
-
path_or_fileobj=str(local_path),
|
| 759 |
-
path_in_repo=repo_path,
|
| 760 |
-
repo_id=repo_id,
|
| 761 |
-
repo_type="dataset",
|
| 762 |
-
commit_message=commit_message,
|
| 763 |
-
)
|
| 764 |
-
|
| 765 |
-
retry_call(f"upload_file:{repo_path}", upload_attempts, retry_backoff, do_upload, logger, stop_event)
|
| 766 |
-
except Exception as e:
|
| 767 |
-
elapsed = time.time() - start
|
| 768 |
-
# Log error to CSV for tracking
|
| 769 |
-
append_progress(local_path, repo_path, 'failed', str(e), elapsed)
|
| 770 |
-
# Update progress bar with error
|
| 771 |
-
file_pbar.update_file_error(1, f"{repo_path}: {str(e)[:60]}")
|
| 772 |
-
raise
|
| 773 |
-
else:
|
| 774 |
-
elapsed = time.time() - start
|
| 775 |
-
# Only log successful uploads to CSV (optional, can be verbose)
|
| 776 |
-
# append_progress(local_path, repo_path, 'uploaded', None, elapsed)
|
| 777 |
-
# Update progress bar
|
| 778 |
-
file_pbar.update_file(1)
|
| 779 |
-
return repo_path, local_path
|
| 780 |
-
|
| 781 |
-
try:
|
| 782 |
-
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as ex:
|
| 783 |
-
futures = [ex.submit(upload_one, f) for f in files]
|
| 784 |
-
for fut in concurrent.futures.as_completed(futures):
|
| 785 |
-
# Respect cancellation
|
| 786 |
-
if stop_event is not None and getattr(stop_event, 'is_set', lambda: False)():
|
| 787 |
-
logger("[Control] Cancellation requested; cancelling pending uploads...")
|
| 788 |
-
# Attempt to cancel pending futures
|
| 789 |
-
for f in futures:
|
| 790 |
-
try:
|
| 791 |
-
f.cancel()
|
| 792 |
-
except Exception:
|
| 793 |
-
pass
|
| 794 |
-
try:
|
| 795 |
-
ex.shutdown(wait=False, cancel_futures=True)
|
| 796 |
-
except Exception:
|
| 797 |
-
pass
|
| 798 |
-
raise Exception("Upload cancelled by user")
|
| 799 |
-
|
| 800 |
-
try:
|
| 801 |
-
repo_path, local_path = fut.result()
|
| 802 |
-
except Exception as e:
|
| 803 |
-
# Error already logged above, just continue
|
| 804 |
-
continue
|
| 805 |
-
|
| 806 |
-
if repo_path:
|
| 807 |
-
already_uploaded.add(repo_path)
|
| 808 |
-
try:
|
| 809 |
-
checkpoint_hashes[repo_path] = compute_sha256(local_path)
|
| 810 |
-
except Exception as e:
|
| 811 |
-
logger(f"Warning: failed to hash {repo_path} after upload: {e}")
|
| 812 |
-
run_stats['uploaded'] += 1
|
| 813 |
-
try:
|
| 814 |
-
run_stats['bytes_uploaded'] += Path(local_path).stat().st_size
|
| 815 |
-
except Exception:
|
| 816 |
-
pass
|
| 817 |
-
save_checkpoint(already_uploaded, checkpoint_hashes)
|
| 818 |
-
finally:
|
| 819 |
-
file_pbar.close()
|
| 820 |
-
file_pbar.log_errors()
|
| 821 |
-
|
| 822 |
-
retry_call("upload_segments", upload_attempts, retry_backoff, upload_segments, logger, stop_event)
|
| 823 |
-
|
| 824 |
-
# Upload CSV if not already uploaded with segments
|
| 825 |
-
if csv_path and not csv_uploaded_with_segments:
|
| 826 |
-
def upload_csv() -> None:
|
| 827 |
-
skip_csv = False
|
| 828 |
-
if resume:
|
| 829 |
-
ensure_existing_files_loaded()
|
| 830 |
-
if resume and csv_path_in_repo in already_uploaded:
|
| 831 |
-
if verify_remote and requests is not None:
|
| 832 |
-
local_size = csv_path.stat().st_size
|
| 833 |
-
if rate_limiter:
|
| 834 |
-
rate_limiter.acquire("verify_remote")
|
| 835 |
-
remote_size, remote_etag = remote_file_info(csv_path_in_repo)
|
| 836 |
-
if verify_etag and remote_etag:
|
| 837 |
-
if remote_size is not None and local_size == remote_size:
|
| 838 |
-
skip_csv = True
|
| 839 |
-
else:
|
| 840 |
-
if remote_size is not None and local_size == remote_size:
|
| 841 |
-
skip_csv = True
|
| 842 |
-
|
| 843 |
-
if skip_csv:
|
| 844 |
-
logger(f"CSV already exists at {csv_path_in_repo}; skipping upload.")
|
| 845 |
-
run_stats['skipped'] += 1
|
| 846 |
-
append_progress(csv_path, csv_path_in_repo, 'skipped', 'already uploaded')
|
| 847 |
-
return
|
| 848 |
-
|
| 849 |
-
start = time.time()
|
| 850 |
-
try:
|
| 851 |
-
if rate_limiter:
|
| 852 |
-
rate_limiter.acquire("csv_upload")
|
| 853 |
-
api.upload_file(
|
| 854 |
-
path_or_fileobj=str(csv_path),
|
| 855 |
-
path_in_repo=csv_path_in_repo,
|
| 856 |
-
repo_id=repo_id,
|
| 857 |
-
repo_type="dataset",
|
| 858 |
-
commit_message=f"Upload CSV for {repo_id}",
|
| 859 |
-
)
|
| 860 |
-
except Exception as e:
|
| 861 |
-
elapsed = time.time() - start
|
| 862 |
-
run_stats['failed'] += 1
|
| 863 |
-
append_progress(csv_path, csv_path_in_repo, 'failed', str(e), elapsed)
|
| 864 |
-
raise
|
| 865 |
-
else:
|
| 866 |
-
elapsed = time.time() - start
|
| 867 |
-
run_stats['uploaded'] += 1
|
| 868 |
-
try:
|
| 869 |
-
run_stats['bytes_uploaded'] += csv_path.stat().st_size
|
| 870 |
-
except Exception:
|
| 871 |
-
pass
|
| 872 |
-
try:
|
| 873 |
-
checkpoint_hashes[csv_path_in_repo] = compute_sha256(csv_path)
|
| 874 |
-
save_checkpoint(already_uploaded.union({csv_path_in_repo}), checkpoint_hashes)
|
| 875 |
-
except Exception as e:
|
| 876 |
-
logger(f"Warning: failed to update CSV checkpoint hash: {e}")
|
| 877 |
-
append_progress(csv_path, csv_path_in_repo, 'uploaded', None, elapsed)
|
| 878 |
-
|
| 879 |
-
retry_call("upload_csv", upload_attempts, retry_backoff, upload_csv, logger, stop_event)
|
| 880 |
-
|
| 881 |
-
logger("")
|
| 882 |
-
logger("Upload complete.")
|
| 883 |
-
logger(f"Dataset URL: https://huggingface.co/datasets/{repo_id}")
|
| 884 |
-
record_summary()
|
| 885 |
-
|
| 886 |
-
return 0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|