jrrribeiro commited on
Commit
fe20ae1
·
1 Parent(s): 2574ede

Remove hf_bulk_upload_tool (migrated to jrrribeiro/HF_Dataset_Uploader)

Browse files
hf_bulk_upload_tool/BUILD_GUIDE.md DELETED
@@ -1,139 +0,0 @@
1
- # Build Guide: Creating a Portable EXE for HF Bulk Uploader
2
-
3
- ## Prerequisites
4
-
5
- 1. **Python 3.10+** installed and added to PATH
6
- 2. **Virtual Environment activated** (highly recommended)
7
-
8
- ## Step 1: Install Dependencies
9
-
10
- ```powershell
11
- # Install base requirements (if not already done)
12
- pip install -r requirements.txt
13
-
14
- # Install GUI-specific requirements
15
- pip install -r hf_bulk_upload_tool\requirements_gui.txt
16
-
17
- # Install PyInstaller for building the EXE
18
- pip install pyinstaller==6.1.0
19
- ```
20
-
21
- ## Step 2: Test the GUI (Optional but Recommended)
22
-
23
- Before building the EXE, test that the GUI runs correctly:
24
-
25
- ```powershell
26
- cd hf_bulk_upload_tool
27
- python main_gui.py
28
- ```
29
-
30
- This will open the GUI window. Make sure all controls work and that you can see the log output.
31
-
32
- ## Step 3: Build the EXE
33
-
34
- ### Option A: Simple One-File Build (Recommended for Distribution)
35
-
36
- ```powershell
37
- cd hf_bulk_upload_tool
38
-
39
- pyinstaller --onefile --noconsole `
40
- --name "HF_Dataset_Uploader" `
41
- --icon=path\to\icon.ico `
42
- --collect-all customtkinter `
43
- --collect-all huggingface_hub `
44
- main_gui.py
45
- ```
46
-
47
- **Parameters explained:**
48
- - `--onefile`: Creates a single `.exe` file (larger but easier to distribute)
49
- - `--noconsole`: Hides the black command prompt window
50
- - `--name "HF_Dataset_Uploader"`: Sets the output EXE name
51
- - `--icon`: (Optional) Path to a `.ico` file for the EXE icon
52
- - `--collect-all customtkinter`: Ensures CustomTkinter is bundled
53
- - `--collect-all huggingface_hub`: Ensures HF Hub lib is bundled
54
- - `main_gui.py`: The entry point script
55
-
56
- ### Option B: Multi-File Build (Faster Development)
57
-
58
- If you want faster builds during development:
59
-
60
- ```powershell
61
- pyinstaller --onedir --noconsole `
62
- --name "HF_Dataset_Uploader" `
63
- --collect-all customtkinter `
64
- --collect-all huggingface_hub `
65
- main_gui.py
66
- ```
67
-
68
- This creates a folder with the EXE and all dependencies (easier to debug, but more files to distribute).
69
-
70
- ## Step 4: Locate Your EXE
71
-
72
- After the build completes, you'll find:
73
- - **One-file build**: `dist/HF_Dataset_Uploader.exe`
74
- - **Multi-file build**: `dist/HF_Dataset_Uploader/HF_Dataset_Uploader.exe`
75
-
76
- ## Step 5: Test the EXE
77
-
78
- 1. Double-click the EXE to run it
79
- 2. Fill in the fields and test with `--dry-run` equivalent (if adding that option)
80
- 3. Check that all functionality works
81
-
82
- ## Troubleshooting
83
-
84
- ### "Module not found" errors
85
- - Ensure all dependencies are installed: `pip install -r requirements.txt`
86
- - Use `--collect-all <module>` for each missing module
87
- - Example: `--collect-all requests --collect-all tqdm`
88
-
89
- ### GUI doesn't display
90
- - Make sure CustomTkinter is installed: `pip install customtkinter`
91
- - Check for Python version conflicts (needs Python 3.10+)
92
-
93
- ### Large EXE file size
94
- - One-file builds are large (~200-300 MB) because they include all dependencies
95
- - Use the `--multi-file` version for smaller individual files (but more of them)
96
-
97
- ### Windows SmartScreen warning
98
- - This is a false positive; users should click "More info" → "Run anyway"
99
- - To prevent this, you'd need a code-signing certificate (paid)
100
-
101
- ### Antivirus flags
102
- - PyInstaller-generated EXEs can trigger false positives
103
- - Consider using `--clean` to remove build artifacts between builds
104
- - Upload the final EXE to VirusTotal for verification
105
-
106
- ## Advanced: Creating an Installer with NSIS
107
-
108
- For professional distribution, consider creating an installer:
109
-
110
- 1. Install NSIS: https://nsis.sourceforge.io/
111
- 2. Use a tool like `auto-py-to-exe` or create a custom `.nsi` script
112
- 3. This allows users to install to `Program Files` and create Start Menu shortcuts
113
-
114
- ## GitHub Actions for Automated Builds (Optional)
115
-
116
- You can set up GitHub Actions to automatically build the EXE for every release:
117
-
118
- See `.github/workflows/build.yml` (if created)
119
-
120
- ---
121
-
122
- ## Summary Commands
123
-
124
- **Quick build (all-in-one):**
125
- ```powershell
126
- cd hf_bulk_upload_tool
127
- pyinstaller --onefile --noconsole `
128
- --name "HF_Dataset_Uploader" `
129
- --collect-all customtkinter `
130
- --collect-all huggingface_hub `
131
- main_gui.py
132
- ```
133
-
134
- **Then run:**
135
- ```powershell
136
- .\dist\HF_Dataset_Uploader.exe
137
- ```
138
-
139
- That's it! Your portable GUI uploader is ready to distribute.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
hf_bulk_upload_tool/HF_Dataset_Uploader.spec DELETED
@@ -1,56 +0,0 @@
1
- # -*- mode: python ; coding: utf-8 -*-
2
- from PyInstaller.utils.hooks import collect_all
3
-
4
- datas = []
5
- binaries = []
6
- hiddenimports = []
7
- tmp_ret = collect_all('customtkinter')
8
- datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
9
- tmp_ret = collect_all('huggingface_hub')
10
- datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
11
-
12
- # Add custom modules
13
- hiddenimports.extend([
14
- 'sharding_utils',
15
- 'progress_bar_utils',
16
- 'resilience_utils',
17
- 'tqdm',
18
- 'requests',
19
- ])
20
-
21
-
22
- a = Analysis(
23
- ['main_gui.py'],
24
- pathex=[],
25
- binaries=binaries,
26
- datas=datas,
27
- hiddenimports=hiddenimports,
28
- hookspath=[],
29
- hooksconfig={},
30
- runtime_hooks=[],
31
- excludes=[],
32
- noarchive=False,
33
- optimize=0,
34
- )
35
- pyz = PYZ(a.pure)
36
-
37
- exe = EXE(
38
- pyz,
39
- a.scripts,
40
- a.binaries,
41
- a.datas,
42
- [],
43
- name='HF_Dataset_Uploader',
44
- debug=False,
45
- bootloader_ignore_signals=False,
46
- strip=False,
47
- upx=True,
48
- upx_exclude=[],
49
- runtime_tmpdir=None,
50
- console=False,
51
- disable_windowed_traceback=False,
52
- argv_emulation=False,
53
- target_arch=None,
54
- codesign_identity=None,
55
- entitlements_file=None,
56
- )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
hf_bulk_upload_tool/IMPLEMENTATION_PROTOCOL.md DELETED
@@ -1,365 +0,0 @@
1
- # Implementation Protocol: GUI Uploader (.exe)
2
-
3
- **Version:** 1.0
4
- **Status:** ✅ Complete (Code only, no tests executed)
5
- **Date:** May 14, 2026
6
-
7
- ---
8
-
9
- ## Executive Summary
10
-
11
- This protocol documents the **complete implementation** of a portable GUI application for Hugging Face dataset uploads. The original CLI script (`upload_dataset.py`) has been refactored into a modular architecture with a CustomTkinter GUI frontend, enabling non-technical users to upload datasets via a user-friendly interface.
12
-
13
- **No automated tests were executed** per your request — only the complete source code is provided.
14
-
15
- ---
16
-
17
- ## Files Delivered
18
-
19
- ### 1. **upload_logic.py** (260 lines)
20
- **Purpose:** Refactored core upload logic
21
- **Key Changes:**
22
- - Extracted `run_upload_logic(config, logger)` function
23
- - Replaced all `print()` with logger callback
24
- - Removed argparse dependency
25
- - Config is now a plain Python dict
26
- - 100% compatible with original behavior
27
-
28
- **Entry Point:**
29
- ```python
30
- def run_upload_logic(config: dict, logger: Callable[[str], None]) -> int
31
- ```
32
-
33
- ### 2. **main_gui.py** (350+ lines)
34
- **Purpose:** CustomTkinter GUI application
35
- **Features:**
36
- - ✅ Input fields: repo ID, token, segment path, CSV path
37
- - ✅ Options: private/public, resume, verify, rate-limit, workers
38
- - ✅ Real-time progress log (textbox with auto-scroll)
39
- - ✅ Threading: upload runs in background, UI never freezes
40
- - ✅ Thread-safe logging via Queue
41
- - ✅ Validation: checks required fields and paths
42
- - ✅ Button states: Start/Stop enable/disable properly
43
-
44
- **Entry Point:**
45
- ```python
46
- def main() # Launches GUI mainloop
47
- ```
48
-
49
- ### 3. **requirements_gui.txt** (4 lines)
50
- **Purpose:** Additional dependencies for GUI
51
- **Contents:**
52
- ```
53
- customtkinter==5.2.2
54
- # Plus: huggingface-hub, requests, tqdm (in base requirements.txt)
55
- ```
56
-
57
- ### 4. **BUILD_GUIDE.md** (140+ lines)
58
- **Purpose:** Step-by-step guide to generate the .exe
59
- **Sections:**
60
- - Prerequisites
61
- - Dependency installation
62
- - GUI testing before build
63
- - PyInstaller build commands (one-file & multi-file options)
64
- - Troubleshooting (module not found, antivirus, SmartScreen, etc.)
65
- - Advanced: NSIS installer creation
66
- - Summary commands
67
-
68
- **Quick Build:**
69
- ```powershell
70
- pyinstaller --onefile --noconsole `
71
- --name "HF_Dataset_Uploader" `
72
- --collect-all customtkinter `
73
- --collect-all huggingface_hub `
74
- main_gui.py
75
- ```
76
-
77
- ### 5. **hf_uploader_gui.spec** (65 lines)
78
- **Purpose:** Pre-configured PyInstaller spec file
79
- **Benefits:**
80
- - Simplified build: `pyinstaller hf_uploader_gui.spec`
81
- - All options pre-set
82
- - Reproducible builds
83
- - Easy to modify if needed
84
-
85
- ### 6. **README_GUI.md** (300+ lines)
86
- **Purpose:** Complete architecture & usage documentation
87
- **Sections:**
88
- - Overview & architecture
89
- - Motor (logic) vs Cabin (UI) separation
90
- - Config dict schema
91
- - Threading model explained
92
- - File structure
93
- - Design decisions
94
- - Limitations & future improvements
95
- - Testing checklist
96
- - Troubleshooting table
97
- - Summary & next steps
98
-
99
- ### 7. **test_gui.py** (130 lines)
100
- **Purpose:** Pre-flight checks before launching GUI
101
- **Checks:**
102
- - ✅ All required modules installed
103
- - ✅ GUI files exist and readable
104
- - ✅ Optional: Launch GUI to verify
105
-
106
- **Usage:**
107
- ```powershell
108
- python test_gui.py
109
- ```
110
-
111
- ---
112
-
113
- ## Architecture Overview
114
-
115
- ```
116
- ┌─────────────────────────────────────────────────────────┐
117
- │ main_gui.py │
118
- │ (CustomTkinter GUI Layer) │
119
- │ ┌──────────────────────────────────────────────────┐ │
120
- │ │ Fields: repo ID, token, segments, CSV │ │
121
- │ │ Options: private, resume, verify, rate-limit │ │
122
- │ │ Buttons: Start, Stop, Clear │ │
123
- │ │ Display: Real-time log textbox │ │
124
- │ └──────────────────────────────────────────────────┘ │
125
- │ ↓ │
126
- │ ┌──────────────────────────────────────────────────┐ │
127
- │ │ Thread Model │ │
128
- │ │ • Main thread: UI event loop (never blocks) │ │
129
- │ │ • Upload thread: run_upload_logic() │ │
130
- │ │ • Queue: Thread-safe log passing │ │
131
- │ └──────────────────────────────────────────────────┘ │
132
- └──────────────────────────────────────────��──────────────┘
133
- ↓
134
- ┌─────────────────────────────────────────────────────────┐
135
- │ upload_logic.py │
136
- │ (Pure Upload Logic Layer – No UI) │
137
- │ ┌──────────────────────────────────────────────────┐ │
138
- │ │ Config validation │ │
139
- │ │ ↓ │ │
140
- │ │ HfApi setup + rate limiter │ │
141
- │ │ ↓ │ │
142
- │ │ upload_large_folder() [Primary path] │ │
143
- │ │ • Staging folder setup │ │
144
- │ │ • Directory junctions (Windows) │ │
145
- │ │ • Batch commit (respects rate limits) │ │
146
- │ │ ↓ [Fallback if above fails] │ │
147
- │ │ Per-file upload with retry logic │ │
148
- │ │ • ThreadPoolExecutor for parallelism │ │
149
- │ │ • Checkpoint system (safe resume) │ │
150
- │ │ • Rate limiter (1000 req/5min) │ │
151
- │ │ ↓ │ │
152
- │ │ CSV upload (if not already included) │ │
153
- │ │ ↓ │ │
154
- │ │ Checkpoint save + summary JSON │ │
155
- │ └──────────────────────────────────────────────────┘ │
156
- └─────────────────────────────────────────────────────────┘
157
- ↓
158
- Hugging Face API
159
- ```
160
-
161
- ---
162
-
163
- ## Configuration Dictionary
164
-
165
- All configuration passed to `run_upload_logic()`:
166
-
167
- ```python
168
- config = {
169
- # Required
170
- "hf_token": str, # HF write token
171
- "repo_id": str, # Format: "username/dataset-name"
172
- "segments": str | Path, # Path to audio directory
173
-
174
- # Optional (with defaults)
175
- "csv": str | Path, # CSV file to upload
176
- "segments_path_in_repo": "audio",
177
- "csv_path_in_repo": "index/detections.csv",
178
- "private": True,
179
- "resume": False,
180
- "verify_remote": False,
181
- "verify_etag": False,
182
- "dry_run": False,
183
- "rate_limit_aware": True,
184
- "rate_limit_max_requests": 950,
185
- "rate_limit_window": 300,
186
- "create_repo_attempts": 3,
187
- "upload_attempts": 3,
188
- "retry_backoff": 5.0,
189
- "max_workers": 1,
190
- "checkpoint_dir": Path,
191
- "commit_message": "Upload BirdNET segments",
192
- }
193
- ```
194
-
195
- ---
196
-
197
- ## Threading Model
198
-
199
- **Problem:** If upload runs on main thread, GUI freezes.
200
- **Solution:** Background thread + thread-safe queue.
201
-
202
- ```
203
- User clicks "Start"
204
- ↓
205
- Main thread validates inputs
206
- ↓
207
- Main thread spawns upload_thread (daemon)
208
- ↓
209
- upload_thread runs: run_upload_logic(config, logger_callback)
210
- ↓
211
- Inside upload_thread: logger(msg) → queue.put(msg)
212
- ↓
213
- Main thread: after(100ms, _process_log_queue)
214
- ↓
215
- Main thread reads queue: self.log_queue.get_nowait()
216
- ↓
217
- Main thread updates textbox: textbox.insert(msg)
218
- ↓
219
- Main thread schedules next check after(100ms, ...)
220
- ```
221
-
222
- **Result:** Upload happens smoothly, UI responsive, logs update every 100ms.
223
-
224
- ---
225
-
226
- ## Build Process
227
-
228
- ### Step 1: Verify Environment
229
- ```powershell
230
- pip install -r requirements.txt
231
- pip install -r hf_bulk_upload_tool/requirements_gui.txt
232
- ```
233
-
234
- ### Step 2: Test GUI
235
- ```powershell
236
- cd hf_bulk_upload_tool
237
- python test_gui.py # Validates imports and launches GUI
238
- ```
239
-
240
- ### Step 3: Build EXE
241
- ```powershell
242
- pyinstaller --onefile --noconsole `
243
- --name "HF_Dataset_Uploader" `
244
- --collect-all customtkinter `
245
- --collect-all huggingface_hub `
246
- main_gui.py
247
- ```
248
-
249
- ### Step 4: Output
250
- - **One-file build**: `dist/HF_Dataset_Uploader.exe` (~250 MB)
251
- - **Multi-file build**: `dist/HF_Dataset_Uploader/HF_Dataset_Uploader.exe` (folder)
252
-
253
- ### Step 5: Test EXE
254
- ```powershell
255
- ./dist/HF_Dataset_Uploader.exe
256
- ```
257
-
258
- ---
259
-
260
- ## Key Features
261
-
262
- ✅ **User-Friendly**
263
- - Modern CustomTkinter dark theme
264
- - Clear input fields with labels
265
- - File browser dialogs
266
- - Real-time progress log
267
-
268
- ✅ **Robust**
269
- - Input validation
270
- - Graceful error handling
271
- - Retry logic with exponential backoff
272
- - Checkpoint system for safe resume
273
-
274
- ✅ **Performant**
275
- - `upload_large_folder()` reduces commits (solves rate limit issue)
276
- - Rate limiting respects 1000 req/5min quota
277
- - Directory junctions avoid copying 55GB+ data
278
- - Multi-threaded per-file upload as fallback
279
-
280
- ✅ **Professional**
281
- - Standalone .exe (no Python installation required)
282
- - No console window (looks polished)
283
- - Proper error dialogs
284
- - Status indicators
285
-
286
- ---
287
-
288
- ## Known Limitations (v1)
289
-
290
- ❌ No mid-upload cancellation (can close app + resume later with --resume)
291
- ❌ No progress bar visualization (text log only)
292
- ❌ No drag-and-drop for file selection
293
- ❌ No configuration profiles (save/load)
294
-
295
- **These can be added in v2.**
296
-
297
- ---
298
-
299
- ## Testing Checklist (Manual, Not Automated)
300
-
301
- Before generating .exe, verify:
302
-
303
- - [ ] `test_gui.py` runs without errors
304
- - [ ] `python main_gui.py` launches GUI window
305
- - [ ] All input fields respond to keyboard/mouse
306
- - [ ] File picker dialogs open
307
- - [ ] Token field is masked (shows `*` not plaintext)
308
- - [ ] Validation catches empty required fields
309
- - [ ] Validation catches non-existent paths
310
- - [ ] Start button disables during upload
311
- - [ ] Stop button enables during upload
312
- - [ ] Log updates in real-time
313
- - [ ] Progress completes with ✅ or ❌ message
314
- - [ ] Resume mode skips already-uploaded files
315
-
316
- ---
317
-
318
- ## Deliverable Files Summary
319
-
320
- | File | Lines | Purpose |
321
- |------|-------|---------|
322
- | upload_logic.py | 260 | Refactored core logic (no UI) |
323
- | main_gui.py | 350+ | CustomTkinter GUI + threading |
324
- | requirements_gui.txt | 4 | Extra dependencies |
325
- | BUILD_GUIDE.md | 140+ | Build instructions & troubleshooting |
326
- | hf_uploader_gui.spec | 65 | PyInstaller spec file |
327
- | README_GUI.md | 300+ | Architecture & usage docs |
328
- | test_gui.py | 130 | Pre-flight checks |
329
-
330
- **Total:** ~1,250 lines of production-ready code (no tests executed as requested)
331
-
332
- ---
333
-
334
- ## Next Steps for User
335
-
336
- 1. ✅ **Review the code** — Read `README_GUI.md` for architecture overview
337
- 2. ✅ **Test locally** — Run `test_gui.py` to verify dependencies
338
- 3. ✅ **Launch GUI** — `python main_gui.py` to see the interface
339
- 4. ✅ **Build EXE** — Follow `BUILD_GUIDE.md` to generate the executable
340
- 5. ✅ **Distribute** — Share the `.exe` with non-technical users
341
-
342
- ---
343
-
344
- ## Support & Questions
345
-
346
- - **Architecture questions?** See `README_GUI.md`
347
- - **Build issues?** See `BUILD_GUIDE.md` troubleshooting section
348
- - **Logic changes?** Modify `upload_logic.py` (both CLI and GUI will use it)
349
- - **UI enhancements?** Modify `main_gui.py` (logic layer untouched)
350
-
351
- ---
352
-
353
- ## Protocol Status
354
-
355
- ✅ **COMPLETE** — All code written, no tests executed (per request)
356
- ⏳ **READY FOR TESTING** — User should run `test_gui.py` before building EXE
357
- 🚀 **READY FOR BUILD** — Once testing passes, ready to generate `.exe`
358
-
359
- ---
360
-
361
- **Document Signed:** Implementation Protocol v1.0
362
- **Deliverable Date:** May 14, 2026
363
- **Platform:** Windows 10+ (PowerShell), cross-platform capable
364
- **Python:** 3.10+ required
365
- **Architecture:** Modular, testable, distributable as standalone .exe
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
hf_bulk_upload_tool/RATE_LIMITER_GUIDE.md DELETED
@@ -1,221 +0,0 @@
1
- # Rate Limiter Guide for Large Datasets
2
-
3
- ## Overview
4
-
5
- The upload script now includes an intelligent **rate limiter** that respects the Hugging Face API quota of **1000 requests per 5 minutes**. This allows you to upload very large datasets (150k+ files) automatically without hitting rate limits.
6
-
7
- ## How It Works
8
-
9
- **Sliding Window Rate Limiting:**
10
- - Tracks requests over the last 5 minutes
11
- - When reaching ~950 requests (safety buffer), pauses automatically
12
- - Resumes when old requests drop out of the window
13
- - Completely transparent — happens in the background
14
-
15
- Example timeline for 150k small files:
16
- ```
17
- Minute 0-5: Upload files 1-1000 (1000 requests)
18
- Minute 5-10: Wait 5 sec, then files 1001-2000 (1000 requests)
19
- Minute 10-15: Wait 5 sec, then files 2001-3000 (1000 requests)
20
- ...
21
- Minute 750-760: Upload final files
22
- Total time: ~12.5 hours (continuous, no manual waits)
23
- ```
24
-
25
- ## Enabling Rate Limiter
26
-
27
- Rate limiter is **enabled by default**. Use it with:
28
-
29
- ```powershell
30
- python .\hf_bulk_upload_tool\upload_dataset.py `
31
- --repo-id "jrrribeiro/PPBIO_Audio_Library" `
32
- --segments "C:\path\to\segments" `
33
- --csv "C:\path\to\detections.csv" `
34
- --hf-token "<YOUR_HF_TOKEN>" --private `
35
- --resume
36
- ```
37
-
38
- To disable (not recommended):
39
- ```powershell
40
- python ... --rate-limit-aware False
41
- ```
42
-
43
- ## Disabling Verify-Remote for Speed
44
-
45
- **Important:** If resuming with `--resume`, do NOT use `--verify-remote` by default — it doubles the requests!
46
-
47
- ✅ **Good (safe for 150k files):**
48
- ```powershell
49
- python .\hf_bulk_upload_tool\upload_dataset.py `
50
- --repo-id "jrrribeiro/PPBIO_Audio_Library" `
51
- --segments "C:\path\to\segments" `
52
- --csv "C:\path\to\detections.csv" `
53
- --hf-token "<YOUR_HF_TOKEN>" --private `
54
- --resume
55
- ```
56
- Estimated time: ~12 hours (150k requests at 1000/5min)
57
-
58
- ❌ **Slow (not recommended for 150k files):**
59
- ```powershell
60
- python ... --resume --verify-remote
61
- ```
62
- Estimated time: ~24 hours (300k requests)
63
-
64
- ## Handling Interruptions
65
-
66
- If the script crashes or network fails mid-upload:
67
-
68
- 1. **Wait a moment** (to ensure checkpoint is flushed)
69
- 2. **Re-run the exact same command:**
70
- ```powershell
71
- python .\hf_bulk_upload_tool\upload_dataset.py `
72
- --repo-id "jrrribeiro/PPBIO_Audio_Library" `
73
- --segments "C:\path\to\segments" `
74
- --csv "C:\path\to\detections.csv" `
75
- --hf-token "<YOUR_HF_TOKEN>" --private `
76
- --resume
77
- ```
78
-
79
- The script will:
80
- - Load the checkpoint (files already uploaded)
81
- - Skip all completed files
82
- - Continue from where it left off
83
- - **No re-uploading, no wasted time**
84
-
85
- ## Checkpoint Storage
86
-
87
- Checkpoints are stored in `.checkpoints/`:
88
- ```
89
- .checkpoints/
90
- jrrribeiro__PPBIO_Audio_Library.json # List of uploaded files + SHA-256 hashes
91
- progress.csv # Per-file logs
92
- jrrribeiro__PPBIO_Audio_Library.summary.json # Final stats
93
- ```
94
-
95
- Customize location:
96
- ```powershell
97
- python ... --checkpoint-dir "C:\my\custom\path"
98
- ```
99
-
100
- ## Rate Limiter Configuration
101
-
102
- Override defaults if needed:
103
-
104
- | Option | Default | Description |
105
- |--------|---------|-------------|
106
- | `--rate-limit-aware` | True | Enable/disable rate limiter |
107
- | `--rate-limit-max-requests` | 950 | Max requests before pause (safety buffer below 1000) |
108
- | `--rate-limit-window` | 300 | Window duration in seconds (300 = 5min) |
109
-
110
- Example (more conservative):
111
- ```powershell
112
- python ... `
113
- --rate-limit-max-requests 800 `
114
- --rate-limit-window 300
115
- ```
116
-
117
- ## Request Counting
118
-
119
- The script counts these as API requests:
120
- - **File upload** (each file) = 1 request
121
- - **List repo files** (resume mode, once) = 1 request
122
- - **Verify remote** (if --verify-remote per file) = 1 request
123
- - **CSV upload** = 1 request
124
-
125
- For 150k small files with `--resume` (no verify-remote):
126
- - ~150,000 file uploads
127
- - ~1 list_repo_files
128
- - ~1 CSV upload
129
- - **Total: ~150,002 requests ≈ 150 windows ≈ 750 minutes ≈ 12.5 hours**
130
-
131
- ## Monitoring Progress
132
-
133
- Progress is shown in real-time:
134
- ```
135
- Uploading: |████████░░░░░░░░░░| 25% [2.5 GB / 10 GB] ETA 9h 15m
136
- [RateLimit] 948 / 950 requests in current window (3 min left)
137
- ```
138
-
139
- Key metrics in logs:
140
- - Bytes uploaded
141
- - Current rate limiter window status
142
- - Estimated time remaining
143
-
144
- ## Best Practices for 150k Files
145
-
146
- 1. **Start fresh or resume:**
147
- ```powershell
148
- # First run
149
- python ... --resume
150
- ```
151
-
152
- 2. **Keep machine awake** (disable sleep/screensaver)
153
-
154
- 3. **Stable network** (hardwired Ethernet if possible)
155
-
156
- 4. **One worker** (default):
157
- ```powershell
158
- python ... --max-workers 1
159
- ```
160
-
161
- 5. **Monitor first hour** to ensure everything is working, then let it run
162
-
163
- 6. **Use `--checkpoint-dir` on a reliable disk** (not network drive)
164
-
165
- ## Troubleshooting
166
-
167
- **Q: Script paused and shows "Waiting Xs before next request"**
168
- A: Normal! This is the rate limiter preventing 429 errors. Let it continue.
169
-
170
- **Q: Got 429 anyway?**
171
- A: Rare, but possible if:
172
- - Another client is using the same token
173
- - HF already counted background requests in your quota
174
- - Solution: Wait 10 min, then resume with `--rate-limit-max-requests 800`
175
-
176
- **Q: Can I speed this up?**
177
- A: Only if HF allows higher limits (PRO account). Otherwise, the 1000 req/5min is the hard limit.
178
-
179
- **Q: How do I monitor progress without staying in the terminal?**
180
- A: Script logs to `.checkpoints/progress.csv` — check that anytime.
181
-
182
- ## Example Commands
183
-
184
- ### First run (full upload):
185
- ```powershell
186
- python .\hf_bulk_upload_tool\upload_dataset.py `
187
- --repo-id "jrrribeiro/PPBIO_Audio_Library" `
188
- --segments "C:\Users\jonat\OneDrive - MSFT\Doutorado PPGEtno\BirdNET Segments" `
189
- --csv "C:\Users\jonat\OneDrive - MSFT\Doutorado PPGEtno\BirdNET Analyzer\00_ALL_DETECTIONS.csv" `
190
- --hf-token "<YOUR_HF_TOKEN>" --private `
191
- --resume
192
- ```
193
- ETA: ~12 hours (150k files)
194
-
195
- ### Resume after interruption:
196
- ```powershell
197
- # Same command as above
198
- python .\hf_bulk_upload_tool\upload_dataset.py `
199
- --repo-id "jrrribeiro/PPBIO_Audio_Library" `
200
- --segments "C:\Users\jonat\OneDrive - MSFT\Doutorado PPGEtno\BirdNET Segments" `
201
- --csv "C:\Users\jonat\OneDrive - MSFT\Doutorado PPGEtno\BirdNET Analyzer\00_ALL_DETECTIONS.csv" `
202
- --hf-token "<YOUR_HF_TOKEN>" --private `
203
- --resume
204
- ```
205
- Continues from where it left off — only uploads new/missing files.
206
-
207
- ### Conservative (slower, more stable):
208
- ```powershell
209
- python .\hf_bulk_upload_tool\upload_dataset.py `
210
- --repo-id "jrrribeiro/PPBIO_Audio_Library" `
211
- --segments "C:\Users\jonat\OneDrive - MSFT\Doutorado PPGEtno\BirdNET Segments" `
212
- --csv "C:\Users\jonat\OneDrive - MSFT\Doutorado PPGEtno\BirdNET Analyzer\00_ALL_DETECTIONS.csv" `
213
- --hf-token "<YOUR_HF_TOKEN>" --private `
214
- --resume `
215
- --rate-limit-max-requests 800
216
- ```
217
- ETA: ~15 hours (more safety margin)
218
-
219
- ---
220
-
221
- **Ready to start?** Run the first command above and let it work for 12+ hours. The rate limiter will handle everything automatically!
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
hf_bulk_upload_tool/README.md DELETED
@@ -1,178 +0,0 @@
1
- # BirdNET HF Bulk Upload Tool
2
-
3
- Standalone, production-ready script for uploading large BirdNET datasets to Hugging Face with built-in rate-limit handling, resume support, and progress tracking.
4
-
5
- ## Features
6
-
7
- ✅ **Fast uploads** – Uses `HfApi.upload_large_folder` when available; falls back to parallel per-file upload
8
- ✅ **Rate-limit safe** – 429-aware backoff; per-file throttling (default 0.5s delay) to avoid HF API quota exhaustion
9
- ✅ **Resume-friendly** – Checkpoints track uploaded files + SHA-256 hashes; skip already-uploaded files on retry
10
- ✅ **Progress visibility** – Aggregated progress bar with ETA (not verbose per-file logs)
11
- ✅ **Audit logs** – CSV progress log + JSON summary with upload stats
12
- ✅ **Flexible** – Verify remote files by size/ETag; customize worker count, delays, and retry policy
13
-
14
- ## Install
15
-
16
- Activate the venv and install dependencies:
17
-
18
- ```powershell
19
- & .\.venv\Scripts\Activate.ps1
20
- pip install -r .\hf_bulk_upload_tool\requirements.txt
21
- ```
22
-
23
- ## Quick Start
24
-
25
- ### Dry-run (validate before uploading)
26
-
27
- ```powershell
28
- python .\hf_bulk_upload_tool\upload_dataset.py `
29
- --repo-id "jrrribeiro/PPBIO_Audio_Library" `
30
- --segments "C:\path\to\segments" `
31
- --csv "C:\path\to\detections.csv" `
32
- --private --dry-run
33
- ```
34
-
35
- ### Upload (rate-limit safe defaults)
36
-
37
- ```powershell
38
- python .\hf_bulk_upload_tool\upload_dataset.py `
39
- --repo-id "jrrribeiro/PPBIO_Audio_Library" `
40
- --segments "C:\path\to\segments" `
41
- --csv "C:\path\to\detections.csv" `
42
- --hf-token "<YOUR_HF_TOKEN>" --private
43
- ```
44
-
45
- ### Resume (skip already-uploaded files)
46
-
47
- ```powershell
48
- python .\hf_bulk_upload_tool\upload_dataset.py `
49
- --repo-id "jrrribeiro/PPBIO_Audio_Library" `
50
- --segments "C:\path\to\segments" `
51
- --csv "C:\path\to\detections.csv" `
52
- --hf-token "<YOUR_HF_TOKEN>" --private `
53
- --resume --verify-remote
54
- ```
55
-
56
- ### Inspect only (no upload)
57
-
58
- ```powershell
59
- python .\hf_bulk_upload_tool\upload_dataset.py `
60
- --repo-id "jrrribeiro/PPBIO_Audio_Library" `
61
- --segments "C:\path\to\segments" `
62
- --private --resume --resume-only
63
- ```
64
-
65
- ## Dataset Layout
66
-
67
- By default, the script uploads:
68
- - `segments/` folder → `audio/` in the dataset (preserves subfolder structure)
69
- - `CSV` file → `index/detections.csv` in the dataset
70
-
71
- Example:
72
- ```
73
- Local:
74
- segments/
75
- Species A/
76
- file1.wav
77
- file2.wav
78
- Species B/
79
- file3.wav
80
-
81
- Dataset:
82
- audio/Species A/file1.wav
83
- audio/Species A/file2.wav
84
- audio/Species B/file3.wav
85
- index/detections.csv
86
- ```
87
-
88
- ## Large Upload Tuning
89
-
90
- ### If hitting rate limits (429)
91
-
92
- 1. **Wait and retry** (~5 min cooldown): the script uses checkpoints, so files already uploaded are skipped
93
- 2. **Increase per-file delay**: `--per-file-delay 1.0` (default is 0.5s)
94
- 3. **Reduce workers**: `--max-workers 1` (default is 1, but adjust if needed)
95
-
96
- Example conservative command:
97
- ```powershell
98
- python .\hf_bulk_upload_tool\upload_dataset.py `
99
- --repo-id "jrrribeiro/PPBIO_Audio_Library" `
100
- --segments "C:\path\to\segments" `
101
- --hf-token "<YOUR_HF_TOKEN>" --private `
102
- --max-workers 1 --per-file-delay 1.0
103
- ```
104
-
105
- ### For very large datasets (50+ GB)
106
-
107
- Increase HF timeouts:
108
-
109
- ```powershell
110
- $env:HF_HUB_ETAG_TIMEOUT = '60'
111
- $env:HF_HUB_DOWNLOAD_TIMEOUT = '300'
112
- $env:HF_XET_HIGH_PERFORMANCE = '1'
113
- ```
114
-
115
- Then run the upload.
116
-
117
- ## Checkpoints & Logging
118
-
119
- The script saves:
120
- - `.checkpoints/<repo_id>.json` – list of uploaded files + SHA-256 hashes
121
- - `.checkpoints/progress.csv` – per-file status, elapsed time, errors
122
- - `.checkpoints/<repo_id>.summary.json` – total stats (files, bytes, elapsed time)
123
-
124
- Use `--checkpoint-dir <path>` to customize the checkpoint location.
125
-
126
- ## All Options
127
-
128
- ```
129
- python .\hf_bulk_upload_tool\upload_dataset.py --help
130
- ```
131
-
132
- Key arguments:
133
- | Option | Default | Description |
134
- |--------|---------|-------------|
135
- | `--repo-id` | (required) | `username/dataset-name` |
136
- | `--segments` | (required) | Local folder path |
137
- | `--csv` | (none) | Optional CSV file to upload |
138
- | `--hf-token` | (prompt) | HF write token (or use `HF_TOKEN` env var) |
139
- | `--private` | true | Create private dataset (or `--public`) |
140
- | `--max-workers` | 1 | Parallel upload threads (per-file fallback) |
141
- | `--per-file-delay` | 0.5 | Seconds between per-file uploads |
142
- | `--resume` | false | Skip already-uploaded files |
143
- | `--verify-remote` | false | Verify remote file size/ETag on resume |
144
- | `--verify-etag` | false | Prefer ETag over size checks |
145
- | `--resume-only` | false | Inspect repo without uploading |
146
- | `--dry-run` | false | Validate args; no network calls |
147
- | `--checkpoint-dir` | `.checkpoints` | Where to store checkpoints |
148
-
149
- ## Troubleshooting
150
-
151
- **"Failed to get upload mode: 429"**
152
- HF rate limit hit. Wait 5 min, then re-run (script will skip completed files).
153
-
154
- **"No such file or directory" (temp junction)**
155
- Rare on Windows. Try increasing `--per-file-delay` or reduce `--max-workers`.
156
-
157
- **"Failed to create audio junction" (Windows)**
158
- Ensure temp folder has write permissions. Or use `--max-workers 1 --per-file-delay 0.5`.
159
-
160
- **"Authentication failed"**
161
- Check your HF token is valid and has write permission to the target repo.
162
-
163
- ## Notes
164
-
165
- - **Keep machine awake** during uploads (no sleep mode)
166
- - **Stable network** strongly recommended for 50+ GB uploads
167
- - **Re-run is safe** – checkpoints prevent re-uploading files
168
- - **Start small** – test with a tiny folder first if unsure about permissions
169
-
170
- ## Environment Variables
171
-
172
- Override defaults:
173
- ```powershell
174
- $env:HF_HUB_ETAG_TIMEOUT = '60'
175
- $env:HF_HUB_DOWNLOAD_TIMEOUT = '300'
176
- $env:HF_XET_HIGH_PERFORMANCE = '1'
177
- $env:HF_TOKEN = '<your_token>' # avoid typing --hf-token each time
178
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
hf_bulk_upload_tool/README_GUI.md DELETED
@@ -1,261 +0,0 @@
1
- # HF Dataset Uploader GUI – Protocol & Implementation Guide
2
-
3
- ## Overview
4
-
5
- This directory contains a refactored version of `upload_dataset.py` designed for packaging as a portable, user-friendly GUI application using **CustomTkinter** and **PyInstaller**.
6
-
7
- The project follows a **"Wrapper" architecture**:
8
- - **Motor (Logic Layer)**: `upload_logic.py` — Pure upload logic, no UI dependencies
9
- - **Cabin (UI Layer)**: `main_gui.py` — CustomTkinter GUI that calls the motor via callbacks
10
- - **Build System**: PyInstaller configuration and build guide
11
-
12
- ---
13
-
14
- ## Architecture
15
-
16
- ### 1. **upload_logic.py** (Refactored Core)
17
-
18
- **Key Changes from `upload_dataset.py`:**
19
- - ✅ Removed `argparse` dependency (no CLI parsing in this version)
20
- - ✅ Extracted `main()` into `run_upload_logic(config, logger_callback)`
21
- - ✅ Replaced all `print()` calls with `logger(message)` callback
22
- - ✅ Config is a Python dict instead of argparse Namespace
23
- - ✅ 100% backward compatible with the original logic
24
-
25
- **Function Signature:**
26
- ```python
27
- def run_upload_logic(config: dict, logger: Callable[[str], None]) -> int:
28
- """
29
- Execute the upload process.
30
-
31
- Args:
32
- config: Dictionary with keys like 'repo_id', 'hf_token', 'segments', etc.
33
- logger: Callable that accepts strings for logging (e.g., textbox.insert)
34
-
35
- Returns:
36
- 0 on success, raises Exception on failure
37
- """
38
- ```
39
-
40
- **Config Keys (All Optional Except Marked):**
41
- ```python
42
- config = {
43
- "hf_token": "...", # ✅ Required
44
- "repo_id": "username/repo", # ✅ Required (format: "owner/name")
45
- "segments": "/path/to/audio", # ✅ Required (directory with files)
46
- "csv": "/path/to/file.csv", # Optional
47
- "segments_path_in_repo": "audio", # Default: "audio"
48
- "csv_path_in_repo": "index/detections.csv", # Default
49
- "private": True, # Default: True
50
- "resume": True, # Default: False
51
- "verify_remote": False, # Default: False
52
- "verify_etag": False, # Default: False
53
- "dry_run": False, # Default: False
54
- "rate_limit_aware": True, # Default: True
55
- "rate_limit_max_requests": 950, # Default: 950
56
- "rate_limit_window": 300, # Default: 300 (5 min)
57
- "create_repo_attempts": 3, # Default: 3
58
- "upload_attempts": 5, # Default: 3
59
- "retry_backoff": 10.0, # Default: 5.0
60
- "max_workers": 2, # Default: 1
61
- "checkpoint_dir": "/path", # Default: .checkpoints
62
- "commit_message": "Upload BirdNET segments", # Default
63
- }
64
- ```
65
-
66
- ### 2. **main_gui.py** (CustomTkinter GUI)
67
-
68
- **UI Sections:**
69
-
70
- 1. **📋 Upload Configuration**
71
- - Repository ID input (username/dataset-name)
72
- - HF Token input (password field, hidden)
73
- - Segments folder picker (browse button)
74
- - CSV file picker (optional, browse button)
75
-
76
- 2. **⚙️ Options**
77
- - Private Repository (checkbox)
78
- - Resume mode (checkbox)
79
- - Verify remote files (checkbox)
80
- - Rate limiting (checkbox)
81
- - Worker threads (spinbox, 1-8)
82
-
83
- 3. **📝 Upload Progress**
84
- - Large textbox for real-time log output
85
- - Auto-scrolls to latest message
86
- - Read-only to prevent user edits
87
-
88
- 4. **🚀 Buttons**
89
- - Start Upload (green, initiates upload in background thread)
90
- - Stop (disabled until upload starts)
91
- - Clear Log (clears textbox)
92
- - Status label (Ready/Uploading/Complete/Failed)
93
-
94
- **Threading Model:**
95
- - Main thread: UI event loop (never blocks)
96
- - Upload thread: Runs `run_upload_logic()` in background
97
- - Log queue: Thread-safe message passing (Queue.Queue)
98
-
99
- ```python
100
- # Upload runs in background; logs appear in real-time
101
- Thread 1 (Main UI):
102
- → user clicks "Start"
103
- → spawns Thread 2
104
- → continues processing UI events
105
- ↓
106
- Thread 2 (Upload):
107
- → calls run_upload_logic()
108
- → logs messages → queue → picked up by Thread 1 every 100ms
109
- ```
110
-
111
- ---
112
-
113
- ## How to Use (Development)
114
-
115
- ### 1. **Install Dependencies**
116
-
117
- ```powershell
118
- # Base requirements
119
- pip install -r ../requirements.txt
120
-
121
- # GUI additions
122
- pip install -r requirements_gui.txt
123
- ```
124
-
125
- ### 2. **Test the GUI**
126
-
127
- ```powershell
128
- python main_gui.py
129
- ```
130
-
131
- This opens the GUI window. Fill in test values and click "Start Upload" to test the flow.
132
-
133
- ### 3. **Build the EXE**
134
-
135
- Follow instructions in **BUILD_GUIDE.md**, or quick version:
136
-
137
- ```powershell
138
- pyinstaller --onefile --noconsole `
139
- --name "HF_Dataset_Uploader" `
140
- --collect-all customtkinter `
141
- --collect-all huggingface_hub `
142
- main_gui.py
143
- ```
144
-
145
- Output: `dist/HF_Dataset_Uploader.exe` (standalone, ~250 MB)
146
-
147
- ---
148
-
149
- ## File Structure
150
-
151
- ```
152
- hf_bulk_upload_tool/
153
- ├── upload_dataset.py ← Original CLI version (unchanged)
154
- ├── upload_logic.py ← NEW: Refactored logic (no CLI)
155
- ├── main_gui.py ← NEW: CustomTkinter GUI
156
- ├── requirements_gui.txt ← NEW: GUI dependencies
157
- ├── BUILD_GUIDE.md ← NEW: Step-by-step build instructions
158
- ├��─ hf_uploader_gui.spec ← NEW: PyInstaller spec file
159
- └── .checkpoints/ ← Auto-created: stores resume checkpoints
160
- ```
161
-
162
- ---
163
-
164
- ## Key Design Decisions
165
-
166
- ### 1. **Why Refactor Logic?**
167
- - Separating UI from logic makes the code reusable and testable
168
- - The original `upload_dataset.py` is untouched; we added a new layer on top
169
- - Future: can plug in different UIs (CLI, web, REST API) without changing logic
170
-
171
- ### 2. **Why Thread-Safe Logging?**
172
- - If we called `textbox.insert()` directly from the upload thread, the GUI could crash
173
- - Using `queue.Queue` ensures thread safety and smooth updates
174
-
175
- ### 3. **Why CustomTkinter?**
176
- - **Modern look** compared to vanilla tkinter (dark mode, rounded buttons, etc.)
177
- - **Pure Python** (no compiled dependencies)
178
- - **Lightweight** (~5 MB)
179
- - **Cross-platform** (Windows, macOS, Linux)
180
-
181
- ### 4. **Resume as Default?**
182
- - `resume=True` by default in the GUI
183
- - Prevents re-uploading files and respects HF rate limits
184
- - Safe: checkpoint system tracks uploaded files locally
185
-
186
- ---
187
-
188
- ## Limitations & Future Improvements
189
-
190
- ### Current Limitations
191
- 1. ❌ **No mid-upload cancellation** — user can close app, but upload thread keeps running
192
- 2. ❌ **No progress bar** — only text log (could add % completion)
193
- 3. ❌ **No drag-and-drop** for file selection
194
- 4. ❌ **No dark mode toggle** (always dark)
195
-
196
- ### Potential Improvements (v2)
197
- - ✅ Real-time progress bar showing % completion
198
- - ✅ Ability to pause/resume upload mid-flight
199
- - ✅ Save favorite configs (profiles)
200
- - ✅ Drag-and-drop for folder/CSV inputs
201
- - ✅ System tray icon with notifications
202
- - ✅ Embedded browser to validate HF token in-app
203
- - ✅ Light mode toggle
204
- - ✅ Auto-update mechanism
205
-
206
- ---
207
-
208
- ## Testing Checklist
209
-
210
- Before building the EXE, verify:
211
-
212
- - [ ] `python main_gui.py` launches without errors
213
- - [ ] All buttons and inputs are clickable
214
- - [ ] File picker dialogs open correctly
215
- - [ ] Token field is masked (shows `*` not plaintext)
216
- - [ ] Log textbox updates in real-time
217
- - [ ] Empty fields show validation errors
218
- - [ ] Non-existent paths are caught and reported
219
- - [ ] Start/Stop button states toggle correctly
220
-
221
- ---
222
-
223
- ## Troubleshooting
224
-
225
- | Issue | Solution |
226
- |-------|----------|
227
- | `ModuleNotFoundError: customtkinter` | `pip install customtkinter` |
228
- | `ModuleNotFoundError: huggingface_hub` | `pip install huggingface-hub` |
229
- | GUI shows blank window | Run with `python -u main_gui.py` to force unbuffered output |
230
- | EXE won't start | Double-click and check Windows event log for errors |
231
- | "Only administrator can run this" | Right-click EXE → Properties → Compatibility → Run as admin |
232
- | Antivirus blocks EXE | Add exception for the file or upload to VirusTotal.com to verify |
233
-
234
- ---
235
-
236
- ## Summary
237
-
238
- This protocol transforms the CLI upload script into a professional, distributable GUI application:
239
-
240
- 1. **Logic Layer** (`upload_logic.py`): Pure, testable upload code
241
- 2. **UI Layer** (`main_gui.py`): User-friendly CustomTkinter interface
242
- 3. **Build System** (PyInstaller): Creates a standalone `.exe` for non-technical users
243
- 4. **Threading**: Ensures UI never freezes during long uploads
244
- 5. **Logging**: Real-time progress visible in the GUI
245
-
246
- Users can now:
247
- - ✅ Double-click an `.exe` to upload datasets
248
- - ✅ No need to understand Python, CLI, or argparse
249
- - ✅ Safe resume if interrupted
250
- - ✅ Real-time progress feedback
251
-
252
- ---
253
-
254
- ## Next Steps
255
-
256
- 1. **Test locally**: `python main_gui.py`
257
- 2. **Build EXE**: Follow BUILD_GUIDE.md
258
- 3. **Distribute**: Share the `.exe` with non-technical users
259
- 4. **Iterate**: Gather feedback, add improvements from "Future Improvements" section
260
-
261
- Happy uploading! 🚀
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
hf_bulk_upload_tool/__init__.py DELETED
@@ -1,3 +0,0 @@
1
- """HF Bulk Upload Tool - Hugging Face Dataset Upload Utilities"""
2
-
3
- __version__ = "1.0.0"
 
 
 
 
hf_bulk_upload_tool/hf_uploader_gui.spec DELETED
@@ -1,56 +0,0 @@
1
- # -*- mode: python ; coding: utf-8 -*-
2
- import sys
3
- import os
4
- from pathlib import Path
5
-
6
- block_cipher = None
7
- script_dir = Path(os.path.dirname(os.path.abspath(SPEC)))
8
- main_gui_path = script_dir / 'main_gui.py'
9
-
10
- a = Analysis(
11
- [str(main_gui_path)],
12
- pathex=[str(script_dir)],
13
- binaries=[],
14
- datas=[],
15
- hiddenimports=[
16
- 'upload_logic',
17
- 'customtkinter',
18
- 'huggingface_hub',
19
- 'requests',
20
- 'tqdm',
21
- 'sharding_utils',
22
- 'progress_bar_utils',
23
- 'resilience_utils',
24
- ],
25
- hookspath=[],
26
- hooksconfig={},
27
- runtime_hooks=[],
28
- excludedimports=[],
29
- win_no_prefer_redirects=False,
30
- win_private_assemblies=False,
31
- cipher=block_cipher,
32
- noarchive=False,
33
- )
34
- pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)
35
- exe = EXE(
36
- pyz,
37
- a.scripts,
38
- a.binaries,
39
- a.zipfiles,
40
- a.datas,
41
- [],
42
- name='HF_Dataset_Uploader',
43
- debug=False,
44
- bootloader_ignore_signals=False,
45
- strip=False,
46
- upx=True,
47
- upx_exclude=[],
48
- runtime_tmpdir=None,
49
- console=False,
50
- disable_windowed_traceback=False,
51
- argv_emulation=False,
52
- target_arch=None,
53
- codesign_identity=None,
54
- entitlements_file=None,
55
- icon=None,
56
- )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
hf_bulk_upload_tool/main_gui.py DELETED
@@ -1,443 +0,0 @@
1
- #!/usr/bin/env python3
2
- """
3
- GUI for Hugging Face Bulk Uploader using CustomTkinter.
4
-
5
- This is a portable GUI wrapper around the upload_logic module, designed to be
6
- packaged as a standalone .exe using PyInstaller.
7
- """
8
-
9
- import customtkinter as ctk
10
- from tkinter import filedialog, messagebox
11
- import threading
12
- import queue
13
- from pathlib import Path
14
- import sys
15
-
16
- # Import the refactored logic using a robust lookup that works both
17
- # in development and when packaged by PyInstaller.
18
- import importlib
19
-
20
- def _find_run_upload_logic():
21
- # Try explicit module names first
22
- for name in ("hf_bulk_upload_tool.upload_logic", "upload_logic"):
23
- try:
24
- m = importlib.import_module(name)
25
- if hasattr(m, "run_upload_logic"):
26
- return getattr(m, "run_upload_logic")
27
- except Exception:
28
- pass
29
-
30
- # Search already-imported modules for a candidate (useful in PyInstaller)
31
- for m in list(sys.modules.values()):
32
- try:
33
- if hasattr(m, "run_upload_logic"):
34
- return getattr(m, "run_upload_logic")
35
- except Exception:
36
- continue
37
-
38
- # As a last resort, if this file is in a package, try relative import
39
- pkg = __package__
40
- if pkg:
41
- try:
42
- m = importlib.import_module(pkg + ".upload_logic")
43
- if hasattr(m, "run_upload_logic"):
44
- return getattr(m, "run_upload_logic")
45
- except Exception:
46
- pass
47
-
48
- return None
49
-
50
- run_upload_logic = _find_run_upload_logic()
51
- if run_upload_logic is None:
52
- raise ImportError("Could not import `run_upload_logic` from hf_bulk_upload_tool or upload_logic; searched known modules and sys.modules")
53
-
54
-
55
- class UploadConfig:
56
- """Simple config holder for upload parameters."""
57
- def __init__(self):
58
- self.repo_id = ""
59
- self.hf_token = ""
60
- self.segments_path = ""
61
- self.csv_path = ""
62
- self.private = True
63
- self.resume = True
64
- self.verify_remote = False
65
- self.verify_etag = False
66
- self.rate_limit_aware = True
67
- self.max_workers = 1
68
- self.upload_attempts = 3
69
- self.retry_backoff = 5.0
70
-
71
-
72
- class HFUploaderApp(ctk.CTk):
73
- """Main GUI application for Hugging Face dataset uploads."""
74
-
75
- def __init__(self):
76
- super().__init__()
77
-
78
- self.title("Hugging Face Dataset Uploader")
79
- self.geometry("900x750")
80
- self.resizable(True, True)
81
-
82
- # Configure appearance
83
- ctk.set_appearance_mode("dark")
84
- ctk.set_default_color_theme("blue")
85
-
86
- # Upload state
87
- self.uploading = False
88
- self.upload_thread = None
89
- self.log_queue: queue.Queue = queue.Queue()
90
- self.stop_event = threading.Event()
91
- self._dynamic_line_active = False
92
- self._dynamic_line_start = None
93
-
94
- # Build UI
95
- self._build_ui()
96
-
97
- # Start log queue processor
98
- self.after(100, self._process_log_queue)
99
-
100
- def _build_ui(self):
101
- """Build the user interface."""
102
- # Main container with padding
103
- main_frame = ctk.CTkFrame(self, fg_color="transparent")
104
- main_frame.pack(fill="both", expand=True, padx=10, pady=10)
105
-
106
- # ===== CONFIGURATION SECTION =====
107
- config_label = ctk.CTkLabel(
108
- main_frame, text="📋 Upload Configuration",
109
- font=("Arial", 14, "bold")
110
- )
111
- config_label.pack(anchor="w", pady=(0, 10))
112
-
113
- config_frame = ctk.CTkFrame(main_frame)
114
- config_frame.pack(fill="x", pady=(0, 15))
115
-
116
- # Row 1: Repo ID
117
- ctk.CTkLabel(config_frame, text="Repository ID:", font=("Arial", 11)).grid(
118
- row=0, column=0, sticky="w", padx=5, pady=5
119
- )
120
- self.repo_id_entry = ctk.CTkEntry(
121
- config_frame, placeholder_text="username/dataset-name", width=300
122
- )
123
- self.repo_id_entry.grid(row=0, column=1, sticky="ew", padx=5, pady=5)
124
-
125
- # Row 2: Token
126
- ctk.CTkLabel(config_frame, text="HF Token:", font=("Arial", 11)).grid(
127
- row=1, column=0, sticky="w", padx=5, pady=5
128
- )
129
- self.token_entry = ctk.CTkEntry(
130
- config_frame, placeholder_text="hf_...", show="*", width=300
131
- )
132
- self.token_entry.grid(row=1, column=1, sticky="ew", padx=5, pady=5)
133
-
134
- # Row 3: Segments path
135
- ctk.CTkLabel(config_frame, text="Segments Folder:", font=("Arial", 11)).grid(
136
- row=2, column=0, sticky="w", padx=5, pady=5
137
- )
138
- segments_row = ctk.CTkFrame(config_frame, fg_color="transparent")
139
- segments_row.grid(row=2, column=1, sticky="ew", padx=5, pady=5)
140
-
141
- self.segments_entry = ctk.CTkEntry(segments_row, width=200)
142
- self.segments_entry.pack(side="left", fill="x", expand=True, padx=(0, 5))
143
-
144
- ctk.CTkButton(
145
- segments_row, text="Browse", width=80,
146
- command=self._browse_segments
147
- ).pack(side="left")
148
-
149
- # Row 4: CSV path
150
- ctk.CTkLabel(config_frame, text="CSV File (Optional):", font=("Arial", 11)).grid(
151
- row=3, column=0, sticky="w", padx=5, pady=5
152
- )
153
- csv_row = ctk.CTkFrame(config_frame, fg_color="transparent")
154
- csv_row.grid(row=3, column=1, sticky="ew", padx=5, pady=5)
155
-
156
- self.csv_entry = ctk.CTkEntry(csv_row, width=200)
157
- self.csv_entry.pack(side="left", fill="x", expand=True, padx=(0, 5))
158
-
159
- ctk.CTkButton(
160
- csv_row, text="Browse", width=80,
161
- command=self._browse_csv
162
- ).pack(side="left")
163
-
164
- # Configure grid weights for proper stretching
165
- config_frame.columnconfigure(1, weight=1)
166
-
167
- # ===== OPTIONS SECTION =====
168
- options_label = ctk.CTkLabel(
169
- main_frame, text="⚙️ Options",
170
- font=("Arial", 14, "bold")
171
- )
172
- options_label.pack(anchor="w", pady=(10, 10))
173
-
174
- options_frame = ctk.CTkFrame(main_frame)
175
- options_frame.pack(fill="x", pady=(0, 15))
176
-
177
- # Checkboxes
178
- self.private_var = ctk.BooleanVar(value=True)
179
- ctk.CTkCheckBox(
180
- options_frame, text="Private Repository",
181
- variable=self.private_var, font=("Arial", 10)
182
- ).pack(anchor="w", padx=5, pady=2)
183
-
184
- self.resume_var = ctk.BooleanVar(value=True)
185
- ctk.CTkCheckBox(
186
- options_frame, text="Resume (skip already uploaded files)",
187
- variable=self.resume_var, font=("Arial", 10)
188
- ).pack(anchor="w", padx=5, pady=2)
189
-
190
- self.verify_var = ctk.BooleanVar(value=False)
191
- ctk.CTkCheckBox(
192
- options_frame, text="Verify remote files before skipping",
193
- variable=self.verify_var, font=("Arial", 10)
194
- ).pack(anchor="w", padx=5, pady=2)
195
-
196
- self.rate_limit_var = ctk.BooleanVar(value=True)
197
- ctk.CTkCheckBox(
198
- options_frame, text="Enable rate limiting (1000 req/5min)",
199
- variable=self.rate_limit_var, font=("Arial", 10)
200
- ).pack(anchor="w", padx=5, pady=2)
201
-
202
- # Worker threads
203
- worker_frame = ctk.CTkFrame(options_frame, fg_color="transparent")
204
- worker_frame.pack(anchor="w", padx=5, pady=5)
205
-
206
- ctk.CTkLabel(worker_frame, text="Worker threads:", font=("Arial", 10)).pack(side="left", padx=(0, 10))
207
- self.workers_combobox = ctk.CTkComboBox(
208
- worker_frame,
209
- values=["1", "2", "3", "4", "5", "6", "7", "8"],
210
- width=60,
211
- font=("Arial", 10),
212
- state="readonly"
213
- )
214
- self.workers_combobox.set("2")
215
- self.workers_combobox.pack(side="left")
216
-
217
- # ===== CONSOLE / LOG SECTION =====
218
- console_label = ctk.CTkLabel(
219
- main_frame, text="📝 Upload Progress",
220
- font=("Arial", 14, "bold")
221
- )
222
- console_label.pack(anchor="w", pady=(10, 5))
223
-
224
- self.log_textbox = ctk.CTkTextbox(
225
- main_frame, width=800, height=250,
226
- font=("Courier", 10), wrap="none"
227
- )
228
- self.log_textbox.pack(fill="both", expand=True, pady=(0, 15))
229
-
230
- # Disable editing of log textbox
231
- self.log_textbox.configure(state="disabled")
232
-
233
- # ===== BUTTON SECTION =====
234
- button_frame = ctk.CTkFrame(main_frame, fg_color="transparent")
235
- button_frame.pack(fill="x", pady=(0, 10))
236
-
237
- self.start_btn = ctk.CTkButton(
238
- button_frame, text="🚀 Start Upload",
239
- command=self._start_upload, width=150, height=40,
240
- font=("Arial", 12, "bold"), fg_color="#2ecc71"
241
- )
242
- self.start_btn.pack(side="left", padx=5)
243
-
244
- self.stop_btn = ctk.CTkButton(
245
- button_frame, text="⏹ Stop",
246
- command=self._stop_upload, width=150, height=40,
247
- state="disabled", font=("Arial", 12)
248
- )
249
- self.stop_btn.pack(side="left", padx=5)
250
-
251
- self.clear_btn = ctk.CTkButton(
252
- button_frame, text="🗑 Clear Log",
253
- command=self._clear_log, width=150, height=40,
254
- font=("Arial", 12)
255
- )
256
- self.clear_btn.pack(side="left", padx=5)
257
-
258
- # Status label
259
- self.status_label = ctk.CTkLabel(
260
- button_frame, text="Ready", font=("Arial", 10), text_color="gray"
261
- )
262
- self.status_label.pack(side="right", padx=5)
263
-
264
- def _browse_segments(self):
265
- """Open folder browser for segments."""
266
- folder = filedialog.askdirectory(title="Select Segments Folder")
267
- if folder:
268
- self.segments_entry.delete(0, "end")
269
- self.segments_entry.insert(0, folder)
270
-
271
- def _browse_csv(self):
272
- """Open file browser for CSV."""
273
- file = filedialog.askopenfilename(
274
- title="Select CSV File",
275
- filetypes=[("CSV files", "*.csv"), ("All files", "*.*")]
276
- )
277
- if file:
278
- self.csv_entry.delete(0, "end")
279
- self.csv_entry.insert(0, file)
280
-
281
- def _log(self, message: str):
282
- """Add message to log queue (thread-safe)."""
283
- self.log_queue.put(message)
284
-
285
- def _process_log_queue(self):
286
- """Process queued log messages and update textbox."""
287
- drained_messages = []
288
- try:
289
- while True:
290
- drained_messages.append(self.log_queue.get_nowait())
291
- except queue.Empty:
292
- pass
293
-
294
- if drained_messages:
295
- self.log_textbox.configure(state="normal")
296
- for message in drained_messages:
297
- dynamic = message.startswith("[Progress]") or message.startswith("[Status]")
298
-
299
- if dynamic:
300
- if self._dynamic_line_active and self._dynamic_line_start is not None:
301
- try:
302
- self.log_textbox.delete(self._dynamic_line_start, "end")
303
- except Exception:
304
- pass
305
- self.log_textbox.insert("end", message)
306
- self._dynamic_line_active = True
307
- self._dynamic_line_start = self.log_textbox.index("end-1c linestart")
308
- else:
309
- if self._dynamic_line_active:
310
- self.log_textbox.insert("end", "\n")
311
- self._dynamic_line_active = False
312
- self._dynamic_line_start = None
313
- self.log_textbox.insert("end", message + "\n")
314
-
315
- self.log_textbox.see("end")
316
- self.log_textbox.configure(state="disabled")
317
-
318
- # Schedule next check
319
- self.after(100, self._process_log_queue)
320
-
321
- def _clear_log(self):
322
- """Clear the log textbox."""
323
- self.log_textbox.configure(state="normal")
324
- self.log_textbox.delete("1.0", "end")
325
- self.log_textbox.configure(state="disabled")
326
- self._dynamic_line_active = False
327
- self._dynamic_line_start = None
328
-
329
- def _validate_inputs(self) -> tuple[bool, str]:
330
- """Validate all input fields."""
331
- repo_id = self.repo_id_entry.get().strip()
332
- token = self.token_entry.get().strip()
333
- segments = self.segments_entry.get().strip()
334
-
335
- if not repo_id:
336
- return False, "Repository ID is required (format: username/repo-name)"
337
- if not token:
338
- return False, "Hugging Face token is required"
339
- if not segments:
340
- return False, "Segments folder is required"
341
- if not Path(segments).is_dir():
342
- return False, f"Segments folder does not exist: {segments}"
343
-
344
- csv_path = self.csv_entry.get().strip()
345
- if csv_path and not Path(csv_path).is_file():
346
- return False, f"CSV file does not exist: {csv_path}"
347
-
348
- return True, "OK"
349
-
350
- def _start_upload(self):
351
- """Start the upload process in a separate thread."""
352
- if self.uploading:
353
- messagebox.showwarning("Warning", "Upload already in progress")
354
- return
355
-
356
- valid, msg = self._validate_inputs()
357
- if not valid:
358
- messagebox.showerror("Validation Error", msg)
359
- return
360
-
361
- # Build config
362
- config = {
363
- "repo_id": self.repo_id_entry.get().strip(),
364
- "hf_token": self.token_entry.get().strip(),
365
- "segments": self.segments_entry.get().strip(),
366
- "csv": self.csv_entry.get().strip() or None,
367
- "private": self.private_var.get(),
368
- "resume": self.resume_var.get(),
369
- "verify_remote": self.verify_var.get(),
370
- "verify_etag": self.verify_var.get(),
371
- "rate_limit_aware": self.rate_limit_var.get(),
372
- "max_workers": int(self.workers_combobox.get()),
373
- "upload_attempts": 5,
374
- "retry_backoff": 10.0,
375
- "stop_event": self.stop_event,
376
- "suppress_console_progress": True,
377
- }
378
-
379
- # Disable buttons
380
- self.uploading = True
381
- self.start_btn.configure(state="disabled")
382
- self.stop_btn.configure(state="normal")
383
- self.status_label.configure(text="Uploading...", text_color="orange")
384
-
385
- # Clear log
386
- self._clear_log()
387
- self._log(f"Starting upload to {config['repo_id']}...")
388
- self._log("")
389
-
390
- # Start upload thread
391
- # Clear any previous stop event and start the worker
392
- self.stop_event.clear()
393
- self.upload_thread = threading.Thread(target=self._run_upload, args=(config,), daemon=True)
394
- self.upload_thread.start()
395
-
396
- def _run_upload(self, config: dict):
397
- """Run the upload logic in a separate thread."""
398
- try:
399
- run_upload_logic(config, self._log)
400
- self.log_queue.put("✅ Upload completed successfully!")
401
- self.after(0, self._upload_finished, True)
402
- except Exception as e:
403
- msg = str(e)
404
- if "cancel" in msg.lower():
405
- self.log_queue.put("⚠️ Upload cancelled by user")
406
- self.after(0, self._upload_finished, False)
407
- else:
408
- self.log_queue.put(f"❌ Upload failed: {msg}")
409
- self.after(0, self._upload_finished, False)
410
-
411
- def _upload_finished(self, success: bool):
412
- """Called when upload finishes (from main thread)."""
413
- self.uploading = False
414
- self.start_btn.configure(state="normal")
415
- self.stop_btn.configure(state="disabled")
416
-
417
- if success:
418
- self.status_label.configure(text="✅ Complete", text_color="green")
419
- messagebox.showinfo("Success", "Upload completed successfully!")
420
- else:
421
- self.status_label.configure(text="❌ Failed", text_color="red")
422
- messagebox.showerror("Error", "Upload failed. Check the log for details.")
423
-
424
- def _stop_upload(self):
425
- """Stop the upload (note: this is a basic implementation)."""
426
- if not self.uploading:
427
- return
428
- # Signal cancellation to worker
429
- self.stop_event.set()
430
- self._log("[Control] Stop requested; attempting to cancel upload...")
431
- self.status_label.configure(text="Cancelling...", text_color="orange")
432
- # Disable stop button to avoid repeated clicks
433
- self.stop_btn.configure(state="disabled")
434
-
435
-
436
- def main():
437
- """Entry point for the GUI application."""
438
- app = HFUploaderApp()
439
- app.mainloop()
440
-
441
-
442
- if __name__ == "__main__":
443
- main()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
hf_bulk_upload_tool/progress_bar_utils.py DELETED
@@ -1,355 +0,0 @@
1
- #!/usr/bin/env python3
2
- """
3
- Progress bar management for HF uploads.
4
-
5
- Replaces verbose per-file logs with a clean progress bar and filters
6
- trivial messages while preserving errors and warnings.
7
- """
8
-
9
- import re
10
- import io
11
- from typing import Callable, Optional
12
- from pathlib import Path
13
-
14
- try:
15
- from tqdm import tqdm
16
- except ImportError:
17
- tqdm = None
18
-
19
-
20
- class ProgressFilter(io.TextIOBase):
21
- """
22
- Filters upload_large_folder() output to show only a progress bar.
23
-
24
- - Captures "Uploaded X of Y files" patterns
25
- - Updates tqdm progress bar
26
- - Filters trivial per-file logs
27
- - Preserves error/warning messages
28
-
29
- Usage:
30
- pbar = tqdm(total=total_files, unit="file", desc="Uploading", leave=True)
31
- filter = ProgressFilter(pbar, logger_callback)
32
-
33
- with contextlib.redirect_stdout(filter):
34
- api.upload_large_folder(...)
35
- """
36
-
37
- def __init__(self, pbar: 'tqdm', logger: Callable[[str], None]):
38
- """
39
- Args:
40
- pbar: tqdm progress bar instance
41
- logger: Callback function for important messages (errors, warnings)
42
- """
43
- super().__init__()
44
- self.pbar = pbar
45
- self.logger = logger
46
- self._buffer = ""
47
- self.last_reported_files = 0
48
- self.last_reported_bytes = 0
49
- self.last_metadata_files = 0
50
- self.last_gui_progress_pct = -1.0
51
- self.last_gui_metadata_pct = -1.0
52
-
53
- def write(self, s: str) -> int:
54
- """Process output line by line."""
55
- if not s:
56
- return 0
57
-
58
- self._buffer += s
59
-
60
- # Process complete lines
61
- while "\n" in self._buffer:
62
- line, self._buffer = self._buffer.split("\n", 1)
63
- self._process_line(line)
64
-
65
- return len(s)
66
-
67
- def flush(self) -> None:
68
- """Flush any remaining buffered content."""
69
- if self._buffer.strip():
70
- self._process_line(self._buffer)
71
- self._buffer = ""
72
-
73
- def _process_line(self, line: str) -> None:
74
- """Process a single line of output."""
75
- if not line.strip():
76
- return
77
-
78
- # Strip common ANSI cursor/control sequences emitted by progress renderers
79
- line = re.sub(r'\x1b\[[0-9;?]*[A-Za-z]', '', line)
80
- if not line.strip():
81
- return
82
-
83
- # Try to extract progress: "Uploaded X of Y files"
84
- progress_match = re.search(
85
- r'Uploaded\s+(\d+)\s+of\s+(\d+)\s+files',
86
- line,
87
- re.IGNORECASE
88
- )
89
-
90
- if progress_match:
91
- uploaded = int(progress_match.group(1))
92
- total = int(progress_match.group(2))
93
-
94
- # Update progress bar if new files since last report
95
- if uploaded > self.last_reported_files:
96
- increment = uploaded - self.last_reported_files
97
- if self.pbar:
98
- try:
99
- self.pbar.update(increment)
100
- except Exception:
101
- pass
102
- self.last_reported_files = uploaded
103
-
104
- # Update description with percentage
105
- if total > 0:
106
- pct = (uploaded / total) * 100
107
- # If we have a console tqdm bar, update it; otherwise emit concise progress to logger
108
- if self.pbar:
109
- try:
110
- self.pbar.set_description(f"Uploading ({pct:.1f}%)")
111
- except Exception:
112
- pass
113
- else:
114
- if pct >= 100.0 or (pct - self.last_gui_progress_pct) >= 0.5:
115
- self.last_gui_progress_pct = pct
116
- try:
117
- self.logger(f"[Progress] Uploading ({pct:.1f}%) {uploaded}/{total} files")
118
- except Exception:
119
- pass
120
-
121
- return # Don't log this line
122
-
123
- # HF metadata recovery progress: keep only a single dynamic line in GUI
124
- metadata_match = re.search(
125
- r'Recovering\s+from\s+metadata\s+files:.*?(\d+)\/(\d+)',
126
- line,
127
- re.IGNORECASE,
128
- )
129
- if metadata_match:
130
- current = int(metadata_match.group(1))
131
- total = int(metadata_match.group(2))
132
- if current >= self.last_metadata_files:
133
- self.last_metadata_files = current
134
- pct = (current / total) * 100 if total > 0 else 0.0
135
- if pct >= 100.0 or (pct - self.last_gui_metadata_pct) >= 0.5:
136
- self.last_gui_metadata_pct = pct
137
- self.logger(f"[Progress] Metadata recovery ({pct:.1f}%) {current}/{total}")
138
- return
139
-
140
- # HF periodic report blocks are useful but very noisy in GUI: compress to one status line.
141
- files_report = re.search(
142
- r'Files:\s+hashed\s+(\d+)\/(\d+).*?committed:\s+(\d+)\/(\d+)',
143
- line,
144
- re.IGNORECASE,
145
- )
146
- if files_report:
147
- hashed = int(files_report.group(1))
148
- total = int(files_report.group(2))
149
- committed = int(files_report.group(3))
150
- self.logger(f"[Status] Hashed {hashed}/{total} | Committed {committed}/{total}")
151
- return
152
-
153
- # Ignore report separators and worker summary lines
154
- if re.search(r'^-+\s+\d{4}-\d{2}-\d{2}|^Workers:|^-{10,}', line, re.IGNORECASE):
155
- return
156
-
157
- # Check for bytes uploaded pattern (optional)
158
- bytes_match = re.search(
159
- r'Uploaded\s+([\d.]+\s*[KMGT]?B)',
160
- line,
161
- re.IGNORECASE
162
- )
163
-
164
- # Filter trivial logs (per-file progress)
165
- trivial_patterns = [
166
- r'^\s*\[.*\]\s+Uploading.*\.wav', # "[...] Uploading file.wav"
167
- r'^\s*Uploading\s+file.*to', # "Uploading file to..."
168
- r'^\s*\d+%', # "50% uploaded"
169
- r'^\s*[├└┌┐]', # Tree characters
170
- r'Connection pooling', # Connection messages
171
- r'Retry-After', # Rate limit header
172
- r'Commit message', # Commit messages
173
- r'^\s*100%', # Final percentage
174
- r'^\s*\|', # progress bar pipes
175
- r'\[A|\[K', # escaped cursor controls
176
- ]
177
-
178
- is_trivial = any(re.search(pattern, line, re.IGNORECASE) for pattern in trivial_patterns)
179
-
180
- if is_trivial:
181
- return # Suppress trivial logs
182
-
183
- # Log important messages (errors, warnings, etc.)
184
- error_patterns = [
185
- r'error|exception|failed|traceback',
186
- r'warning|warn',
187
- r'timeout|504|500|503|429|too many',
188
- r'refused|connection reset|broken pipe',
189
- ]
190
-
191
- is_important = any(
192
- re.search(pattern, line, re.IGNORECASE)
193
- for pattern in error_patterns
194
- )
195
-
196
- if is_important or not any(
197
- re.search(p, line, re.IGNORECASE)
198
- for p in trivial_patterns + [
199
- r'progress',
200
- r'mb/s|kb/s',
201
- r'elapsed',
202
- ]
203
- ):
204
- # Log important messages or unknowns that aren't clearly trivial
205
- if len(line) > 3: # Skip very short lines
206
- self.logger(f"[HF] {line.strip()}")
207
-
208
-
209
- class FilesProgressBar:
210
- """
211
- Manages a clean progress bar for file uploads.
212
-
213
- - Shows: "Uploading [████████..] 1,234 / 5,000 files | ETA: 2m 30s"
214
- - Updates when files complete
215
- - Thread-safe
216
-
217
- Usage (for per-file upload fallback):
218
- bar = FilesProgressBar(total_files=5000, logger=print)
219
- for file in files:
220
- try:
221
- upload_file(file)
222
- bar.update_file(1)
223
- except Exception as e:
224
- bar.update_file_error(1, str(e))
225
- """
226
-
227
- def __init__(self, total_files: int, logger: Callable[[str], None], suppress_console: bool = False):
228
- """
229
- Args:
230
- total_files: Total number of files to upload
231
- logger: Callback function for logging
232
- """
233
- self.total_files = total_files
234
- self.logger = logger
235
- self.uploaded = 0
236
- self.errors = []
237
- self.pbar = None
238
- self.suppress_console = bool(suppress_console)
239
-
240
- if not self.suppress_console and tqdm is not None:
241
- try:
242
- self.pbar = tqdm(
243
- total=total_files,
244
- unit="file",
245
- unit_scale=False,
246
- desc="Uploading",
247
- leave=True,
248
- ncols=80,
249
- miniters=1,
250
- )
251
- except Exception:
252
- self.pbar = None
253
-
254
- def update_file(self, count: int = 1) -> None:
255
- """Mark files as successfully uploaded."""
256
- self.uploaded += count
257
- if self.pbar:
258
- try:
259
- self.pbar.update(count)
260
- pct = (self.uploaded / self.total_files) * 100 if self.total_files > 0 else 0
261
- self.pbar.set_description(f"Uploading ({pct:.1f}%)")
262
- except Exception:
263
- pass
264
- else:
265
- # Emit concise progress to logger for GUI mode
266
- try:
267
- pct = (self.uploaded / self.total_files) * 100 if self.total_files > 0 else 0
268
- if pct >= 100.0 or pct == 0.0 or (pct - getattr(self, "_last_gui_file_pct", -1.0)) >= 0.5:
269
- self._last_gui_file_pct = pct
270
- self.logger(f"[Progress] Uploading ({pct:.1f}%) {self.uploaded}/{self.total_files} files")
271
- except Exception:
272
- pass
273
-
274
- def update_file_error(self, count: int, error_msg: str) -> None:
275
- """Mark files as failed with error message."""
276
- self.errors.append(error_msg)
277
- # Still count as "processed" (though failed) for progress
278
- self.update_file(count)
279
-
280
- def close(self) -> None:
281
- """Close the progress bar."""
282
- if self.pbar:
283
- try:
284
- self.pbar.close()
285
- except Exception:
286
- pass
287
-
288
- def log_errors(self) -> None:
289
- """Print all collected errors/warnings below progress bar."""
290
- if not self.errors:
291
- return
292
-
293
- self.logger("")
294
- self.logger("=" * 70)
295
- self.logger(f"Upload completed with {len(self.errors)} error(s):")
296
- self.logger("=" * 70)
297
-
298
- for error in self.errors:
299
- self.logger(f" ⚠️ {error}")
300
-
301
- self.logger("=" * 70)
302
-
303
-
304
- class BytesProgressBar:
305
- """
306
- Progress bar for byte-based uploads (alternative metric).
307
-
308
- Usage:
309
- bar = BytesProgressBar(total_bytes=5_000_000, logger=print)
310
- bar.update_bytes(1_000_000) # Update with bytes uploaded
311
- """
312
-
313
- def __init__(self, total_bytes: int, logger: Callable[[str], None]):
314
- """
315
- Args:
316
- total_bytes: Total bytes to upload
317
- logger: Callback function for logging
318
- """
319
- self.total_bytes = total_bytes
320
- self.logger = logger
321
- self.uploaded_bytes = 0
322
- self.pbar = None
323
-
324
- if tqdm is not None:
325
- try:
326
- self.pbar = tqdm(
327
- total=total_bytes,
328
- unit="B",
329
- unit_scale=True,
330
- unit_divisor=1024,
331
- desc="Uploading",
332
- leave=True,
333
- ncols=80,
334
- )
335
- except Exception:
336
- pass
337
-
338
- def update_bytes(self, byte_count: int) -> None:
339
- """Update progress with bytes uploaded."""
340
- self.uploaded_bytes += byte_count
341
- if self.pbar:
342
- try:
343
- self.pbar.update(byte_count)
344
- pct = (self.uploaded_bytes / self.total_bytes) * 100 if self.total_bytes > 0 else 0
345
- self.pbar.set_description(f"Uploading ({pct:.1f}%)")
346
- except Exception:
347
- pass
348
-
349
- def close(self) -> None:
350
- """Close the progress bar."""
351
- if self.pbar:
352
- try:
353
- self.pbar.close()
354
- except Exception:
355
- pass
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
hf_bulk_upload_tool/requirements.txt DELETED
@@ -1,3 +0,0 @@
1
- huggingface_hub==0.28.1
2
- requests>=2.28
3
- tqdm>=4.65
 
 
 
 
hf_bulk_upload_tool/requirements_gui.txt DELETED
@@ -1,10 +0,0 @@
1
- # Requirements for the GUI version of HF Bulk Uploader
2
- # These are IN ADDITION to the base requirements.txt
3
-
4
- # GUI Framework
5
- customtkinter==5.2.2
6
-
7
- # Base requirements (should already be installed)
8
- # huggingface-hub>=0.19.0
9
- # requests>=2.31.0
10
- # tqdm>=4.65.0
 
 
 
 
 
 
 
 
 
 
 
hf_bulk_upload_tool/resilience_utils.py DELETED
@@ -1,268 +0,0 @@
1
- #!/usr/bin/env python3
2
- """
3
- Enhanced resilience utilities for HF uploads.
4
-
5
- Adds robustness against:
6
- - HTTP timeouts and network interruptions
7
- - Token expiration detection
8
- - Improved retry strategies for different error types
9
- - Monitoring/alerting hooks
10
- """
11
-
12
- import time
13
- from typing import Callable, Optional, Dict, Any
14
- from pathlib import Path
15
- from datetime import datetime, timedelta
16
- import json
17
-
18
-
19
- class TokenManager:
20
- """Manages HF token lifecycle and expiration."""
21
-
22
- def __init__(self, token: str, warning_days_before_expiry: int = 7):
23
- """
24
- Args:
25
- token: HF API token
26
- warning_days_before_expiry: Days before expiry to warn user
27
- """
28
- self.token = token
29
- self.warning_days_before_expiry = warning_days_before_expiry
30
-
31
- def check_token_expiry(self, logger: Callable[[str], None]) -> bool:
32
- """
33
- Check if token is about to expire and log warning.
34
-
35
- Note: HF tokens don't include expiry in the token itself.
36
- This is a placeholder for user to manually set expiry date in config.
37
-
38
- Returns:
39
- True if token is safe to use, False if expiring soon
40
- """
41
- # In practice, this would need to be set by the user in their config
42
- # Example: --token-expiry-date "2026-06-15"
43
- # For now, just warn to check manually
44
- logger("[TokenManager] HF tokens do not self-report expiry.")
45
- logger("[TokenManager] If you created your token >30 days ago, consider creating a new one.")
46
- return True
47
-
48
-
49
- class SmartRetryStrategy:
50
- """
51
- Intelligent retry strategy that adapts to different error types.
52
-
53
- Handles:
54
- - 429 (Too Many Requests) - respect Retry-After header
55
- - 504 (Gateway Timeout) - exponential backoff up to 5 min
56
- - 503 (Service Unavailable) - slow backoff (GC detected)
57
- - 500-series - exponential backoff
58
- - Network errors - exponential backoff
59
- """
60
-
61
- def __init__(self, max_retries: int = 5, initial_backoff: float = 2.0):
62
- self.max_retries = max_retries
63
- self.initial_backoff = initial_backoff
64
- self.error_history: list[Dict[str, Any]] = []
65
-
66
- def should_retry(self, error_response: Dict[str, Any]) -> bool:
67
- """Determine if error is retryable."""
68
- status = error_response.get('status_code')
69
-
70
- # Retryable status codes
71
- retryable_statuses = [408, 429, 500, 502, 503, 504]
72
- return status in retryable_statuses
73
-
74
- def get_wait_time(self, error_response: Dict[str, Any], attempt: int) -> float:
75
- """
76
- Calculate wait time based on error type and attempt number.
77
-
78
- Returns:
79
- Seconds to wait before retry
80
- """
81
- status = error_response.get('status_code')
82
-
83
- # Check for Retry-After header (HF may suggest wait time)
84
- retry_after = error_response.get('retry_after_header')
85
- if retry_after:
86
- try:
87
- return max(float(retry_after), self.initial_backoff * (2 ** (attempt - 1)))
88
- except (ValueError, TypeError):
89
- pass
90
-
91
- # Status-specific backoff strategies
92
- if status == 429:
93
- # Rate limit: aggressive backoff
94
- return min(300, self.initial_backoff * (4 ** attempt)) # Cap at 5 min
95
-
96
- elif status == 503:
97
- # Service unavailable (likely GC): slow backoff
98
- return min(600, self.initial_backoff * (2 ** attempt)) # Cap at 10 min
99
-
100
- elif status == 504:
101
- # Gateway timeout: exponential backoff
102
- return min(300, self.initial_backoff * (3 ** attempt)) # Cap at 5 min
103
-
104
- elif status in [500, 502]:
105
- # Server error: standard exponential backoff
106
- return self.initial_backoff * (2 ** attempt)
107
-
108
- else:
109
- # Generic network error
110
- return self.initial_backoff * (2 ** attempt)
111
-
112
- def log_retry_decision(
113
- self,
114
- error_response: Dict[str, Any],
115
- attempt: int,
116
- wait_time: float,
117
- logger: Callable[[str], None]
118
- ) -> None:
119
- """Log retry decision for monitoring."""
120
- status = error_response.get('status_code')
121
- error_msg = error_response.get('message', 'Unknown error')
122
-
123
- logger(f"[SmartRetry] Attempt {attempt}/{self.max_retries} failed with status {status}")
124
- logger(f"[SmartRetry] Error: {error_msg}")
125
- logger(f"[SmartRetry] Waiting {wait_time:.0f}s before retry...")
126
-
127
- # Store for historical analysis
128
- self.error_history.append({
129
- 'timestamp': datetime.now().isoformat(),
130
- 'status': status,
131
- 'attempt': attempt,
132
- 'wait_time': wait_time
133
- })
134
-
135
-
136
- class NetworkResilience:
137
- """Handles network-specific issues."""
138
-
139
- @staticmethod
140
- def detect_slow_network(
141
- bytes_uploaded: int,
142
- elapsed_seconds: float,
143
- threshold_kbps: float = 100.0
144
- ) -> bool:
145
- """
146
- Detect if network is unusually slow.
147
-
148
- Args:
149
- bytes_uploaded: Total bytes successfully uploaded
150
- elapsed_seconds: Time elapsed
151
- threshold_kbps: Minimum expected speed (default 100 KB/s)
152
-
153
- Returns:
154
- True if network speed is below threshold
155
- """
156
- if elapsed_seconds < 1:
157
- return False
158
-
159
- kbps = (bytes_uploaded / 1024) / elapsed_seconds
160
- return kbps < threshold_kbps
161
-
162
- @staticmethod
163
- def estimate_remaining_time(
164
- bytes_remaining: int,
165
- current_speed_kbps: float
166
- ) -> float:
167
- """Estimate remaining upload time based on current speed."""
168
- if current_speed_kbps <= 0:
169
- return float('inf')
170
- return (bytes_remaining / 1024) / current_speed_kbps
171
-
172
-
173
- class UploadMonitor:
174
- """Monitors upload progress and detects anomalies."""
175
-
176
- def __init__(self, checkpoint_file: Path, logger: Callable[[str], None]):
177
- self.checkpoint_file = checkpoint_file
178
- self.logger = logger
179
- self.start_time = time.time()
180
- self.last_checkpoint_time = self.start_time
181
- self.checkpoint_interval_seconds = 300 # Every 5 min
182
-
183
- def should_save_checkpoint(self) -> bool:
184
- """Determine if checkpoint should be saved based on time interval."""
185
- elapsed = time.time() - self.last_checkpoint_time
186
- return elapsed > self.checkpoint_interval_seconds
187
-
188
- def record_checkpoint(self, uploaded_count: int, total_count: int) -> None:
189
- """Save checkpoint with upload progress."""
190
- self.last_checkpoint_time = time.time()
191
- elapsed = time.time() - self.start_time
192
-
193
- progress = {
194
- 'timestamp': datetime.now().isoformat(),
195
- 'uploaded': uploaded_count,
196
- 'total': total_count,
197
- 'elapsed_seconds': elapsed,
198
- 'rate_files_per_hour': (uploaded_count / elapsed * 3600) if elapsed > 0 else 0,
199
- 'percent_complete': (uploaded_count / total_count * 100) if total_count > 0 else 0,
200
- }
201
-
202
- try:
203
- with open(self.checkpoint_file, 'w') as f:
204
- json.dump(progress, f, indent=2)
205
- except Exception as e:
206
- self.logger(f"[Monitor] Warning: Could not save checkpoint: {e}")
207
-
208
- def estimate_time_remaining(self, uploaded_count: int, total_count: int) -> str:
209
- """Estimate remaining upload time."""
210
- if uploaded_count == 0 or total_count == 0:
211
- return "unknown"
212
-
213
- elapsed = time.time() - self.start_time
214
- rate_per_sec = uploaded_count / elapsed if elapsed > 0 else 0
215
- remaining = total_count - uploaded_count
216
- remaining_seconds = remaining / rate_per_sec if rate_per_sec > 0 else 0
217
-
218
- hours = remaining_seconds / 3600
219
- minutes = (remaining_seconds % 3600) / 60
220
-
221
- return f"{hours:.1f}h {minutes:.0f}m"
222
-
223
- def log_progress(self, uploaded_count: int, total_count: int) -> None:
224
- """Log current progress."""
225
- elapsed = time.time() - self.start_time
226
- rate = (uploaded_count / elapsed * 3600) if elapsed > 0 else 0
227
- remaining = self.estimate_time_remaining(uploaded_count, total_count)
228
-
229
- self.logger(f"[Progress] {uploaded_count}/{total_count} files ({uploaded_count/total_count*100:.1f}%)")
230
- self.logger(f"[Progress] Rate: {rate:.0f} files/hour | ETA: {remaining}")
231
-
232
-
233
- class GarbageCollectionDetector:
234
- """Detects when HF is doing garbage collection (repo locked state)."""
235
-
236
- def __init__(self):
237
- self.gc_indicators = []
238
-
239
- def record_error(self, status_code: int, retry_after: Optional[str] = None) -> None:
240
- """Record error that might indicate GC."""
241
- if status_code == 503 or (status_code == 429 and retry_after and int(retry_after) > 300):
242
- self.gc_indicators.append({
243
- 'timestamp': datetime.now().isoformat(),
244
- 'status': status_code,
245
- 'retry_after': retry_after
246
- })
247
-
248
- def might_be_in_gc(self) -> bool:
249
- """Check if GC might be happening based on recent errors."""
250
- # If we have 503 errors in the last few minutes, likely GC
251
- if not self.gc_indicators:
252
- return False
253
-
254
- recent_errors = [
255
- e for e in self.gc_indicators
256
- if (datetime.now() - datetime.fromisoformat(e['timestamp'])).total_seconds() < 600
257
- ]
258
-
259
- return len(recent_errors) >= 2
260
-
261
- def suggest_action(self, logger: Callable[[str], None]) -> None:
262
- """Suggest action if GC detected."""
263
- if self.might_be_in_gc():
264
- logger("")
265
- logger("[GCDetector] ⚠️ Possible HF garbage collection detected!")
266
- logger("[GCDetector] Repository may be temporarily locked for optimization.")
267
- logger("[GCDetector] Switching to slower retry strategy (waiting longer between attempts)...")
268
- logger("")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
hf_bulk_upload_tool/run_upload.ps1 DELETED
@@ -1,58 +0,0 @@
1
- param(
2
- [string]$HfUsername,
3
- [string]$HfToken,
4
- [string]$RepoName,
5
- [string]$RepoId,
6
- [Parameter(Mandatory = $true)]
7
- [string]$Segments,
8
- [string]$Csv,
9
- [switch]$Private,
10
- [switch]$Public,
11
- [switch]$DryRun,
12
- [switch]$Resume,
13
- [switch]$VerifyRemote,
14
- [switch]$VerifyEtag,
15
- [string]$ProgressLog,
16
- [switch]$ResumeOnly,
17
- [string]$CheckpointDir,
18
- [int]$CreateRepoAttempts = 3,
19
- [int]$UploadAttempts = 3,
20
- [double]$RetryBackoff = 5
21
- )
22
-
23
- $ErrorActionPreference = 'Stop'
24
-
25
- $env:HF_HUB_ETAG_TIMEOUT = $env:HF_HUB_ETAG_TIMEOUT ?? '20'
26
- $env:HF_HUB_DOWNLOAD_TIMEOUT = $env:HF_HUB_DOWNLOAD_TIMEOUT ?? '120'
27
- $env:HF_XET_HIGH_PERFORMANCE = $env:HF_XET_HIGH_PERFORMANCE ?? '1'
28
-
29
- $scriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
30
- $pythonExe = Join-Path (Split-Path -Parent $scriptDir) '.venv\Scripts\python.exe'
31
- $scriptPath = Join-Path $scriptDir 'upload_dataset.py'
32
-
33
- if (-not (Test-Path $pythonExe)) {
34
- throw "Python executable not found at: $pythonExe"
35
- }
36
-
37
- $argsList = @($scriptPath, '--segments', $Segments)
38
-
39
- if ($HfUsername) { $argsList += @('--hf-username', $HfUsername) }
40
- if ($HfToken) { $argsList += @('--hf-token', $HfToken) }
41
- if ($RepoName) { $argsList += @('--repo-name', $RepoName) }
42
- if ($RepoId) { $argsList += @('--repo-id', $RepoId) }
43
- if ($Csv) { $argsList += @('--csv', $Csv) }
44
- if ($Private) { $argsList += '--private' }
45
- if ($Public) { $argsList += '--public' }
46
- if ($DryRun) { $argsList += '--dry-run' }
47
- if ($Resume) { $argsList += '--resume' }
48
- if ($VerifyRemote) { $argsList += '--verify-remote' }
49
- if ($CheckpointDir) { $argsList += @('--checkpoint-dir', $CheckpointDir) }
50
- if ($VerifyEtag) { $argsList += '--verify-etag' }
51
- if ($ProgressLog) { $argsList += @('--progress-log', $ProgressLog) }
52
- if ($ResumeOnly) { $argsList += '--resume-only' }
53
-
54
- $argsList += @('--create-repo-attempts', $CreateRepoAttempts.ToString())
55
- $argsList += @('--upload-attempts', $UploadAttempts.ToString())
56
- $argsList += @('--retry-backoff', $RetryBackoff.ToString([System.Globalization.CultureInfo]::InvariantCulture))
57
-
58
- & $pythonExe $argsList
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
hf_bulk_upload_tool/sharding_utils.py DELETED
@@ -1,304 +0,0 @@
1
- #!/usr/bin/env python3
2
- """
3
- Transparent sharding utilities for BirdNET dataset uploads.
4
-
5
- Handles the splitting of large species directories (>10k files) for HF compliance,
6
- while keeping them transparent to validation code.
7
-
8
- Sharding format:
9
- Elaenia albiceps/ (original, <10k files)
10
- Elaenia albiceps__shard_00/ (part 1, if original had >10k)
11
- Elaenia albiceps__shard_01/ (part 2)
12
- ...
13
-
14
- When reading in validation: transparently combines all shards into a single virtual directory.
15
- """
16
-
17
- import os
18
- from pathlib import Path
19
- from typing import Dict, List, Tuple
20
-
21
-
22
- SHARD_SEPARATOR = "__shard_"
23
- MAX_FILES_PER_DIR = 9500 # Stay below 10k limit with safety margin
24
-
25
-
26
- def is_shard_folder(folder_name: str) -> Tuple[bool, str]:
27
- """
28
- Check if a folder name is a shard and extract the base species name.
29
-
30
- Returns:
31
- (is_shard: bool, base_species_name: str)
32
- """
33
- if SHARD_SEPARATOR not in folder_name:
34
- return False, folder_name
35
-
36
- base = folder_name.split(SHARD_SEPARATOR)[0]
37
- return True, base
38
-
39
-
40
- def get_all_shards(base_species_path: Path) -> List[Path]:
41
- """
42
- Get all shards for a species, including the base folder if it exists.
43
-
44
- Args:
45
- base_species_path: Path to the base species folder (e.g., Species Name/)
46
-
47
- Returns:
48
- List of all shard paths for this species, sorted by shard index
49
- """
50
- parent = base_species_path.parent
51
- base_name = base_species_path.name
52
-
53
- shards = []
54
-
55
- # Add the base folder if it exists and is not empty
56
- if base_species_path.is_dir():
57
- files = list(base_species_path.glob("*"))
58
- if files:
59
- shards.append(base_species_path)
60
-
61
- # Find all shards (Elaenia albiceps__shard_00, __shard_01, etc.)
62
- shard_pattern = f"{base_name}{SHARD_SEPARATOR}*"
63
- shard_folders = sorted(parent.glob(shard_pattern))
64
-
65
- shards.extend(shard_folders)
66
- return shards
67
-
68
-
69
- def iter_files_all_shards(base_species_path: Path):
70
- """
71
- Generator that yields all files from a species and all its shards.
72
-
73
- Args:
74
- base_species_path: Path to the base species folder
75
-
76
- Yields:
77
- Tuple of (file_path, relative_path_from_base_species)
78
- """
79
- shards = get_all_shards(base_species_path)
80
-
81
- for shard_path in shards:
82
- if not shard_path.is_dir():
83
- continue
84
-
85
- for file_path in shard_path.rglob("*"):
86
- if file_path.is_file():
87
- # Keep the relative structure from within the shard
88
- rel_from_shard = file_path.relative_to(shard_path)
89
- yield file_path, rel_from_shard
90
-
91
-
92
- def should_shard_directory(directory_path: Path) -> bool:
93
- """
94
- Check if a directory should be sharded (has >MAX_FILES_PER_DIR files).
95
-
96
- Args:
97
- directory_path: Path to the directory
98
-
99
- Returns:
100
- True if should be sharded
101
- """
102
- if not directory_path.is_dir():
103
- return False
104
-
105
- count = sum(1 for _ in directory_path.rglob("*") if _.is_file())
106
- return count > MAX_FILES_PER_DIR
107
-
108
-
109
- def shard_directory(source_dir: Path, dest_parent: Path) -> List[Path]:
110
- """
111
- Shard a large directory into multiple folders respecting the 10k file limit.
112
-
113
- Args:
114
- source_dir: Source directory to shard (e.g., "Elaenia albiceps/")
115
- dest_parent: Destination parent folder (staging root)
116
-
117
- Returns:
118
- List of created shard paths (destination folders)
119
-
120
- Example:
121
- shards = shard_directory(
122
- Path("Elaenia albiceps"),
123
- Path("staging/audio")
124
- )
125
- # Creates:
126
- # staging/audio/Elaenia albiceps__shard_00/
127
- # staging/audio/Elaenia albiceps__shard_01/
128
- # ...
129
- """
130
- if not source_dir.is_dir():
131
- raise ValueError(f"Source directory does not exist: {source_dir}")
132
-
133
- dest_parent.mkdir(parents=True, exist_ok=True)
134
- base_name = source_dir.name
135
-
136
- # Collect all files
137
- all_files = sorted([f for f in source_dir.rglob("*") if f.is_file()])
138
-
139
- if not all_files:
140
- return []
141
-
142
- shards = []
143
- current_shard_idx = 0
144
- current_shard_count = 0
145
- current_shard_dir = None
146
-
147
- for file_path in all_files:
148
- # Create new shard if needed
149
- if current_shard_count >= MAX_FILES_PER_DIR or current_shard_dir is None:
150
- shard_name = f"{base_name}{SHARD_SEPARATOR}{current_shard_idx:02d}"
151
- current_shard_dir = dest_parent / shard_name
152
- current_shard_dir.mkdir(parents=True, exist_ok=True)
153
- shards.append(current_shard_dir)
154
- current_shard_count = 0
155
- current_shard_idx += 1
156
-
157
- # Copy file preserving directory structure
158
- rel_path = file_path.relative_to(source_dir)
159
- dest_file = current_shard_dir / rel_path
160
- dest_file.parent.mkdir(parents=True, exist_ok=True)
161
-
162
- # Create symlink or hardlink instead of copying to save space
163
- if os.name == "nt":
164
- # Windows: use hardlink
165
- os.link(str(file_path), str(dest_file))
166
- else:
167
- # Unix: use symlink
168
- os.symlink(file_path, dest_file)
169
-
170
- current_shard_count += 1
171
-
172
- return shards
173
-
174
-
175
- def get_species_map(staging_audio_dir: Path) -> Dict[str, List[Path]]:
176
- """
177
- Build a map of all species and their shards in the staging directory.
178
-
179
- Args:
180
- staging_audio_dir: Path to the staging audio directory (audio/)
181
-
182
- Returns:
183
- Dict mapping base species name -> list of shard paths
184
-
185
- Example:
186
- {
187
- "Elaenia albiceps": [
188
- Path("staging/audio/Elaenia albiceps__shard_00"),
189
- Path("staging/audio/Elaenia albiceps__shard_01"),
190
- ],
191
- "Other species": [
192
- Path("staging/audio/Other species"),
193
- ]
194
- }
195
- """
196
- if not staging_audio_dir.is_dir():
197
- return {}
198
-
199
- species_map = {}
200
- processed = set()
201
-
202
- for item in sorted(staging_audio_dir.iterdir()):
203
- if item.name in processed:
204
- continue
205
-
206
- is_shard, base_name = is_shard_folder(item.name)
207
-
208
- if is_shard:
209
- # This is a shard, process its base species
210
- if base_name not in species_map:
211
- species_map[base_name] = get_all_shards(staging_audio_dir / base_name)
212
- processed.add(item.name)
213
- elif item.is_dir():
214
- # This is a regular species folder (non-shard)
215
- if base_name not in species_map:
216
- species_map[base_name] = get_all_shards(item)
217
- processed.add(base_name)
218
-
219
- return species_map
220
-
221
-
222
- # ===== Validation Helper =====
223
-
224
- class TransparentSpeciesReader:
225
- """
226
- Helper for reading sharded species in validation code.
227
-
228
- Transparently combines all shards of a species into a single iterable view.
229
-
230
- Usage:
231
- reader = TransparentSpeciesReader(Path("dataset/audio"))
232
- for species_name, files in reader.iter_species():
233
- for file_path in files:
234
- # Process file
235
- pass
236
- """
237
-
238
- def __init__(self, dataset_audio_dir: Path):
239
- """
240
- Args:
241
- dataset_audio_dir: Path to the dataset's audio directory
242
- """
243
- self.audio_dir = Path(dataset_audio_dir)
244
- self._species_cache = None
245
-
246
- def get_species_list(self) -> List[str]:
247
- """Get list of all species (base names, deduped)."""
248
- species_map = get_species_map(self.audio_dir)
249
- return sorted(species_map.keys())
250
-
251
- def get_files_for_species(self, species_name: str) -> List[Path]:
252
- """
253
- Get all files for a species, combining all shards.
254
-
255
- Args:
256
- species_name: Base species name (e.g., "Elaenia albiceps")
257
-
258
- Returns:
259
- List of all file paths from this species (all shards combined)
260
- """
261
- species_map = get_species_map(self.audio_dir)
262
- if species_name not in species_map:
263
- return []
264
-
265
- files = []
266
- for shard_dir in species_map[species_name]:
267
- shard_files = sorted([f for f in shard_dir.rglob("*") if f.is_file()])
268
- files.extend(shard_files)
269
-
270
- return sorted(files)
271
-
272
- def iter_species(self):
273
- """
274
- Iterate over all species and their files.
275
-
276
- Yields:
277
- Tuple of (species_name: str, files: List[Path])
278
- """
279
- species_map = get_species_map(self.audio_dir)
280
-
281
- for species_name in sorted(species_map.keys()):
282
- files = self.get_files_for_species(species_name)
283
- yield species_name, files
284
-
285
- def iter_species_with_metadata(self):
286
- """
287
- Iterate over species with metadata (total files, shard count).
288
-
289
- Yields:
290
- Tuple of (species_name: str, files: List[Path], metadata: dict)
291
- """
292
- species_map = get_species_map(self.audio_dir)
293
-
294
- for species_name in sorted(species_map.keys()):
295
- shards = species_map[species_name]
296
- files = self.get_files_for_species(species_name)
297
-
298
- metadata = {
299
- "num_files": len(files),
300
- "num_shards": len(shards),
301
- "shard_dirs": shards,
302
- }
303
-
304
- yield species_name, files, metadata
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
hf_bulk_upload_tool/test_gui.py DELETED
@@ -1,132 +0,0 @@
1
- #!/usr/bin/env python3
2
- """
3
- Quick test script for the GUI application.
4
-
5
- This script tests that all dependencies are installed and the GUI can launch.
6
- No actual uploads are performed in this test.
7
-
8
- Usage:
9
- python test_gui.py
10
- """
11
-
12
- import sys
13
- from pathlib import Path
14
-
15
- def check_imports():
16
- """Check if all required modules are available."""
17
- modules = {
18
- "customtkinter": "CustomTkinter (GUI framework)",
19
- "huggingface_hub": "Hugging Face Hub API",
20
- "requests": "HTTP library",
21
- "tqdm": "Progress bars",
22
- }
23
-
24
- print("🔍 Checking required modules...\n")
25
-
26
- missing = []
27
- for module_name, description in modules.items():
28
- try:
29
- __import__(module_name)
30
- print(f"✅ {module_name:20} — {description}")
31
- except ImportError:
32
- print(f"❌ {module_name:20} — {description} [MISSING]")
33
- missing.append(module_name)
34
-
35
- if missing:
36
- print(f"\n⚠️ Missing modules: {', '.join(missing)}")
37
- print(f"\nInstall with:")
38
- print(f" pip install {' '.join(missing)}")
39
- return False
40
-
41
- print(f"\n✅ All modules installed!")
42
- return True
43
-
44
-
45
- def check_local_files():
46
- """Check if GUI files exist."""
47
- print("\n🔍 Checking local files...\n")
48
-
49
- files_to_check = [
50
- "upload_logic.py",
51
- "main_gui.py",
52
- "requirements_gui.txt",
53
- ]
54
-
55
- all_exist = True
56
- for filename in files_to_check:
57
- filepath = Path(__file__).parent / filename
58
- if filepath.exists():
59
- size_kb = filepath.stat().st_size / 1024
60
- print(f"✅ {filename:25} ({size_kb:.1f} KB)")
61
- else:
62
- print(f"❌ {filename:25} [NOT FOUND]")
63
- all_exist = False
64
-
65
- return all_exist
66
-
67
-
68
- def test_gui_launch():
69
- """Attempt to launch the GUI."""
70
- print("\n🚀 Launching GUI...\n")
71
-
72
- try:
73
- from main_gui import HFUploaderApp
74
-
75
- print("✅ GUI imported successfully!")
76
- print("📋 Creating application window...")
77
-
78
- app = HFUploaderApp()
79
-
80
- print("✅ GUI window created successfully!")
81
- print("\n💡 Tips:")
82
- print(" • Fill in test values (repo ID, token, folder paths)")
83
- print(" • Click 'Start Upload' to trigger the upload logic")
84
- print(" • Watch for logs in the progress textbox")
85
- print(" • Close the window to exit")
86
- print("\nPress Ctrl+C to close the window from this terminal.\n")
87
-
88
- # Run the GUI
89
- app.mainloop()
90
-
91
- return True
92
-
93
- except Exception as e:
94
- print(f"❌ Failed to launch GUI: {e}")
95
- import traceback
96
- traceback.print_exc()
97
- return False
98
-
99
-
100
- def main():
101
- """Run all checks and optionally launch GUI."""
102
- print("=" * 60)
103
- print("HF Dataset Uploader — GUI Test Suite")
104
- print("=" * 60)
105
-
106
- # Step 1: Check imports
107
- if not check_imports():
108
- print("\n⚠️ Cannot proceed without missing modules.")
109
- print("Please install them and try again.")
110
- return 1
111
-
112
- # Step 2: Check files
113
- if not check_local_files():
114
- print("\n⚠️ Some GUI files are missing.")
115
- print("Ensure you're running this from the hf_bulk_upload_tool/ directory.")
116
- return 1
117
-
118
- # Step 3: Launch GUI
119
- print("\n" + "=" * 60)
120
- answer = input("Launch GUI? (yes/no): ").strip().lower()
121
-
122
- if answer in ['y', 'yes']:
123
- if not test_gui_launch():
124
- return 1
125
- else:
126
- print("\n✅ All checks passed! You can now run: python main_gui.py")
127
-
128
- return 0
129
-
130
-
131
- if __name__ == "__main__":
132
- sys.exit(main())
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
hf_bulk_upload_tool/upload_dataset.py DELETED
@@ -1,767 +0,0 @@
1
- #!/usr/bin/env python3
2
- from __future__ import annotations
3
-
4
- import argparse
5
- import getpass
6
- import hashlib
7
- import os
8
- import shutil
9
- import sys
10
- import subprocess
11
- import time
12
- import contextlib
13
- from pathlib import Path
14
- import concurrent.futures
15
- from typing import List, Tuple
16
- import json
17
- import csv
18
- from urllib.parse import quote
19
-
20
- try:
21
- import requests
22
- except Exception:
23
- requests = None
24
- try:
25
- from tqdm import tqdm
26
- except Exception:
27
- tqdm = None
28
- import threading
29
-
30
- try:
31
- from sharding_utils import should_shard_directory, shard_directory
32
- from progress_bar_utils import ProgressFilter, FilesProgressBar
33
- except ImportError:
34
- # Fallback for relative imports during development
35
- from .sharding_utils import should_shard_directory, shard_directory
36
- from .progress_bar_utils import ProgressFilter, FilesProgressBar
37
-
38
-
39
- def configure_hf_env() -> None:
40
- os.environ.setdefault("HF_HUB_ETAG_TIMEOUT", "20")
41
- os.environ.setdefault("HF_HUB_DOWNLOAD_TIMEOUT", "120")
42
- os.environ.setdefault("HF_XET_HIGH_PERFORMANCE", "1")
43
-
44
-
45
- configure_hf_env()
46
-
47
- from huggingface_hub import HfApi
48
-
49
-
50
- class RateLimiter:
51
- """Sliding window rate limiter for HF API (1000 requests per 5 minutes)."""
52
- def __init__(self, max_requests: int = 1000, window_seconds: int = 300):
53
- self.max_requests = max_requests
54
- self.window_seconds = window_seconds
55
- self.request_times: list[tuple[float, str]] = []
56
-
57
- def acquire(self, request_type: str = "upload") -> None:
58
- """Wait if necessary to stay within rate limits, then register the request."""
59
- now = time.time()
60
-
61
- # Remove requests outside the current window
62
- self.request_times = [
63
- (ts, req_type) for ts, req_type in self.request_times
64
- if now - ts < self.window_seconds
65
- ]
66
-
67
- # If at limit, wait until oldest request drops out of window
68
- if len(self.request_times) >= self.max_requests:
69
- oldest_ts = self.request_times[0][0]
70
- wait_time = (oldest_ts + self.window_seconds) - now
71
- if wait_time > 0:
72
- print(f"[RateLimit] Reached {self.max_requests} requests in {self.window_seconds}s window.")
73
- print(f"[RateLimit] Waiting {wait_time:.0f}s before next request...")
74
- time.sleep(wait_time)
75
- self.request_times = [] # Reset window
76
-
77
- # Register this request
78
- self.request_times.append((now, request_type))
79
-
80
- def get_window_usage(self) -> tuple[int, int]:
81
- """Return (requests_in_window, max_requests)."""
82
- now = time.time()
83
- self.request_times = [
84
- (ts, req_type) for ts, req_type in self.request_times
85
- if now - ts < self.window_seconds
86
- ]
87
- return len(self.request_times), self.max_requests
88
-
89
-
90
- def ensure_directory_link(link_path: Path, target_path: Path) -> None:
91
- """Create a directory link on the local filesystem.
92
-
93
- On Windows, prefer a junction so the uploader can stage the repo structure
94
- without copying the full audio tree.
95
- """
96
- if link_path.exists() or link_path.is_symlink():
97
- try:
98
- if link_path.resolve() == target_path.resolve():
99
- return
100
- except Exception:
101
- pass
102
- if link_path.is_symlink() or link_path.is_file():
103
- link_path.unlink()
104
- else:
105
- shutil.rmtree(link_path)
106
-
107
- link_path.parent.mkdir(parents=True, exist_ok=True)
108
-
109
- if os.name == "nt":
110
- try:
111
- subprocess.run(
112
- ["cmd", "/c", "mklink", "/J", str(link_path), str(target_path)],
113
- check=True,
114
- capture_output=True,
115
- text=True,
116
- creationflags=subprocess.CREATE_NO_WINDOW,
117
- )
118
- return
119
- except Exception:
120
- pass
121
-
122
- os.symlink(target_path, link_path, target_is_directory=True)
123
-
124
-
125
- def prepare_staging_folder(staging_root: Path, segments_path: Path, csv_path: Path | None) -> Path:
126
- """Build a persistent staging tree with the repo layout expected by HF.
127
-
128
- The stage keeps the audio directory as a link to the real data and copies
129
- only small metadata files locally.
130
- """
131
- staging_root.mkdir(parents=True, exist_ok=True)
132
-
133
- audio_link = staging_root / "audio"
134
- ensure_directory_link(audio_link, segments_path)
135
-
136
- if csv_path is not None:
137
- csv_dest = staging_root / "index" / "detections.csv"
138
- csv_dest.parent.mkdir(parents=True, exist_ok=True)
139
- shutil.copy2(csv_path, csv_dest)
140
-
141
- return staging_root
142
-
143
-
144
- def _prepare_sharded_staging(staging_root: Path, segments_path: Path, csv_path: Path | None) -> Path:
145
- """Prepare staging folder with automatic sharding for large species directories.
146
-
147
- Detects species folders with >9500 files and automatically shards them into
148
- numbered subdirectories to comply with HF's 10k files per directory limit.
149
- Validation tools can transparently read shards via sharding_utils.
150
-
151
- Returns:
152
- Path to the prepared staging root directory
153
- """
154
- staging_root.mkdir(parents=True, exist_ok=True)
155
- audio_staging = staging_root / "audio"
156
- audio_staging.mkdir(parents=True, exist_ok=True)
157
-
158
- sharded_species = []
159
-
160
- # Check each species folder for sharding
161
- for species_dir in sorted(segments_path.iterdir()):
162
- if not species_dir.is_dir():
163
- continue
164
-
165
- if should_shard_directory(species_dir):
166
- # Need to shard this species
167
- print(f"[Sharding] {species_dir.name} exceeds 9500 files, dividing into shards...")
168
- try:
169
- shards = shard_directory(species_dir, audio_staging)
170
- sharded_species.append((species_dir.name, len(shards)))
171
- print(f"[Sharding] Created {len(shards)} shards for {species_dir.name}")
172
- except Exception as e:
173
- print(f"[Sharding] ERROR sharding {species_dir.name}: {e}")
174
- print(f"[Sharding] Falling back to single directory link")
175
- ensure_directory_link(audio_staging / species_dir.name, species_dir)
176
- else:
177
- # Small species, just link it
178
- ensure_directory_link(audio_staging / species_dir.name, species_dir)
179
-
180
- # Log sharding summary
181
- if sharded_species:
182
- print(f"\n[Sharding] Summary: {len(sharded_species)} species were divided")
183
- for species_name, num_shards in sharded_species:
184
- print(f" - {species_name}: {num_shards} shards")
185
- print("[Sharding] Note: Validation tools will transparently read all shards as single species\n")
186
-
187
- # Copy CSV if provided
188
- if csv_path is not None:
189
- csv_dest = staging_root / "index" / "detections.csv"
190
- csv_dest.parent.mkdir(parents=True, exist_ok=True)
191
- shutil.copy2(csv_path, csv_dest)
192
-
193
- return staging_root
194
-
195
-
196
- def parse_args() -> argparse.Namespace:
197
- parser = argparse.ArgumentParser(
198
- description=(
199
- "Upload a large BirdNET segment folder to a Hugging Face dataset "
200
- "while preserving the local species subfolder structure."
201
- )
202
- )
203
- parser.add_argument("--hf-username", help="Hugging Face username or organization name.")
204
- parser.add_argument("--hf-token", help="Hugging Face write token. If omitted, the script prompts for it.")
205
- parser.add_argument("--repo-name", help="Dataset name to create, for example birdnet-segments-2026.")
206
- parser.add_argument("--repo-id", help="Full dataset repo id, for example username/dataset-name.")
207
- parser.add_argument("--segments", required=True, help="Path to the local folder containing the species subfolders.")
208
- parser.add_argument("--csv", help="Optional CSV file to upload alongside the segments.")
209
- parser.add_argument("--segments-path-in-repo", default="audio", help="Destination folder inside the dataset repo.")
210
- parser.add_argument("--csv-path-in-repo", default="index/detections.csv", help="Destination path for the CSV inside the repo.")
211
- parser.add_argument("--private", action="store_true", help="Create the dataset as private. This is the default.")
212
- parser.add_argument("--public", action="store_true", help="Create the dataset as public.")
213
- parser.add_argument("--commit-message", default="Upload BirdNET segments", help="Commit message used for the upload.")
214
- parser.add_argument("--create-repo-attempts", type=int, default=3, help="Number of retries for repo creation.")
215
- parser.add_argument("--upload-attempts", type=int, default=3, help="Number of retries for folder and CSV uploads.")
216
- parser.add_argument("--retry-backoff", type=float, default=5.0, help="Base delay in seconds between retries.")
217
- parser.add_argument("--max-workers", type=int, default=1, help="Number of parallel worker threads for per-file uploads (fallback path).")
218
- parser.add_argument("--per-file-delay", type=float, default=0.5, help="Seconds to wait between per-file uploads to avoid rate limits.")
219
- parser.add_argument("--resume", action="store_true", help="Check repository and skip files that are already uploaded (resume mode).")
220
- parser.add_argument("--checkpoint-dir", default=None, help="Directory to store upload checkpoints. Defaults to ./hf_bulk_upload_tool/.checkpoints.")
221
- parser.add_argument("--verify-remote", action="store_true", help="When resuming, verify remote file size via HTTP HEAD to avoid skipping mismatched files.")
222
- parser.add_argument("--verify-etag", action="store_true", help="When resuming, prefer ETag comparison if available from the server.")
223
- parser.add_argument("--progress-log", default=None, help="Path to CSV progress log. Defaults to <checkpoint-dir>/progress.csv")
224
- parser.add_argument("--resume-only", action="store_true", help="Only compare local files with the repo and report what would be uploaded; do not upload anything.")
225
- parser.add_argument("--dry-run", action="store_true", help="Print the planned actions without uploading anything.")
226
- parser.add_argument("--rate-limit-aware", action="store_true", default=True, help="Enable rate limiter to respect HF 1000 req/5min quota (default: True).")
227
- parser.add_argument("--rate-limit-max-requests", type=int, default=950, help="Max requests per window before waiting (default: 950 for safety buffer).")
228
- parser.add_argument("--rate-limit-window", type=int, default=300, help="Rate limit window in seconds (default: 300 = 5min).")
229
- return parser.parse_args()
230
-
231
-
232
- def resolve_repo_id(args: argparse.Namespace) -> str:
233
- if args.repo_id:
234
- return args.repo_id.strip()
235
-
236
- if not args.hf_username:
237
- args.hf_username = input("Hugging Face username/organization: ").strip()
238
- if not args.repo_name:
239
- args.repo_name = input("Dataset name: ").strip()
240
-
241
- if not args.hf_username or not args.repo_name:
242
- raise SystemExit("Both --hf-username and --repo-name are required when --repo-id is not provided.")
243
-
244
- return f"{args.hf_username.strip()}/{args.repo_name.strip()}"
245
-
246
-
247
- def get_token(args: argparse.Namespace) -> str:
248
- token = args.hf_token or os.environ.get("HF_TOKEN")
249
- if not token:
250
- token = getpass.getpass("Hugging Face token: ").strip()
251
- if not token:
252
- raise SystemExit("A Hugging Face token is required.")
253
- return token.strip()
254
-
255
-
256
- def retry_call(label: str, attempts: int, backoff: float, func):
257
- last_error: Exception | None = None
258
- for attempt in range(1, attempts + 1):
259
- try:
260
- print(f"[{label}] attempt {attempt}/{attempts}")
261
- return func()
262
- except Exception as exc: # noqa: BLE001 - network operations can fail in many ways
263
- last_error = exc
264
- if attempt >= attempts:
265
- break
266
- # Detect possible 429 / rate-limit signals and increase backoff
267
- wait_seconds = backoff * attempt
268
- try:
269
- resp = getattr(exc, 'response', None)
270
- if resp is not None:
271
- status = getattr(resp, 'status_code', None)
272
- if status == 429:
273
- # obey Retry-After header when present
274
- ra = resp.headers.get('Retry-After') if hasattr(resp, 'headers') else None
275
- if ra:
276
- try:
277
- wait_seconds = max(wait_seconds, int(ra))
278
- except Exception:
279
- wait_seconds = max(wait_seconds, backoff * attempt * 4)
280
- else:
281
- wait_seconds = max(wait_seconds, backoff * attempt * 4)
282
- elif '429' in str(exc) or 'Too Many Requests' in str(exc):
283
- wait_seconds = max(wait_seconds, backoff * attempt * 4)
284
- except Exception:
285
- pass
286
- print(f"[{label}] attempt {attempt} failed: {exc}")
287
- print(f"[{label}] retrying in {wait_seconds:.0f}s")
288
- time.sleep(wait_seconds)
289
-
290
- raise SystemExit(f"{label} failed after {attempts} attempts: {last_error}")
291
-
292
-
293
- def main() -> int:
294
- args = parse_args()
295
-
296
- if args.private and args.public:
297
- raise SystemExit("Choose either --private or --public, not both.")
298
-
299
- segments_path = Path(args.segments).expanduser().resolve()
300
- if not segments_path.is_dir():
301
- raise SystemExit(f"Segments path does not exist or is not a directory: {segments_path}")
302
-
303
- csv_path = None
304
- if args.csv:
305
- csv_path = Path(args.csv).expanduser().resolve()
306
- if not csv_path.is_file():
307
- raise SystemExit(f"CSV path does not exist or is not a file: {csv_path}")
308
-
309
- repo_id = resolve_repo_id(args)
310
- token = get_token(args)
311
- is_private = True if args.private or not args.public else False
312
-
313
- print(f"Repository: {repo_id}")
314
- print(f"Segments: {segments_path}")
315
- print(f"Segments destination: {args.segments_path_in_repo}")
316
- # Track if CSV was uploaded as part of the staged large-folder upload
317
- csv_uploaded_with_segments = False
318
-
319
- if csv_path and not csv_uploaded_with_segments:
320
- print(f"CSV: {csv_path}")
321
- print(f"CSV destination: {args.csv_path_in_repo}")
322
- print(f"Visibility: {'private' if is_private else 'public'}")
323
-
324
- if args.dry_run:
325
- print("Dry run selected. No network calls will be made.")
326
- return 0
327
-
328
- api = HfApi(token=token)
329
-
330
- # Initialize rate limiter for large uploads (150k files = ~12.5 hours at 1000 req/5min)
331
- if args.rate_limit_aware:
332
- rate_limiter = RateLimiter(max_requests=args.rate_limit_max_requests, window_seconds=args.rate_limit_window)
333
- print(f"[RateLimit] Enabled: max {args.rate_limit_max_requests} requests per {args.rate_limit_window}s")
334
- else:
335
- rate_limiter = None
336
-
337
- def list_existing_files() -> set:
338
- try:
339
- print("Listing files in target repo to determine already-uploaded files...")
340
- if rate_limiter:
341
- rate_limiter.acquire("list_repo_files")
342
- files = api.list_repo_files(repo_id=repo_id, repo_type="dataset")
343
- return set(files or [])
344
- except Exception as e:
345
- print(f"Warning: could not list repo files: {e}")
346
- return set()
347
-
348
-
349
- # Checkpoint helpers
350
- default_checkpoint_dir = Path(__file__).parent / ".checkpoints"
351
- checkpoint_dir = Path(args.checkpoint_dir) if args.checkpoint_dir else default_checkpoint_dir
352
- checkpoint_dir.mkdir(parents=True, exist_ok=True)
353
- checkpoint_file = checkpoint_dir / f"{repo_id.replace('/', '__')}.json"
354
-
355
- def load_checkpoint() -> tuple[set, dict[str, str]]:
356
- if not checkpoint_file.exists():
357
- return set(), {}
358
- try:
359
- with open(checkpoint_file, 'r', encoding='utf-8') as fh:
360
- data = json.load(fh)
361
- if isinstance(data, list):
362
- return set(data), {}
363
- uploaded = set(data.get('uploaded', []))
364
- file_hashes = data.get('file_hashes', {})
365
- if not isinstance(file_hashes, dict):
366
- file_hashes = {}
367
- return uploaded, {str(k): str(v) for k, v in file_hashes.items()}
368
- except Exception as e:
369
- print(f"Warning: failed to load checkpoint {checkpoint_file}: {e}")
370
- return set(), {}
371
-
372
- def save_checkpoint(uploaded_set: set, file_hashes: dict[str, str]) -> None:
373
- try:
374
- with open(checkpoint_file, 'w', encoding='utf-8') as fh:
375
- json.dump({'uploaded': sorted(list(uploaded_set)), 'file_hashes': file_hashes}, fh, indent=2)
376
- except Exception as e:
377
- print(f"Warning: failed to save checkpoint {checkpoint_file}: {e}")
378
-
379
- def compute_sha256(file_path: Path) -> str:
380
- digest = hashlib.sha256()
381
- with open(file_path, 'rb') as fh:
382
- for chunk in iter(lambda: fh.read(1024 * 1024), b''):
383
- digest.update(chunk)
384
- return digest.hexdigest()
385
-
386
- def remote_file_info(repo_path: str) -> Tuple[int | None, str | None]:
387
- """Return (size, etag) for a repo file via HTTP HEAD, or (None, None)."""
388
- if requests is None:
389
- return None, None
390
- quoted = '/'.join(quote(p) for p in repo_path.split('/'))
391
- url = f"https://huggingface.co/datasets/{repo_id}/resolve/main/{quoted}"
392
- try:
393
- r = requests.head(url, allow_redirects=True, timeout=10)
394
- if r.status_code >= 400:
395
- return None, None
396
- cl = r.headers.get('content-length')
397
- etag = r.headers.get('etag')
398
- size = int(cl) if cl is not None else None
399
- return size, etag
400
- except Exception:
401
- return None, None
402
-
403
- # Progress logging
404
- progress_log_path = Path(args.progress_log) if args.progress_log else (checkpoint_dir / 'progress.csv')
405
- def append_progress(local_path: Path | None, repo_path: str, status: str, msg: str | None = None, elapsed: float | None = None) -> None:
406
- header = ['timestamp','local_path','repo_path','status','message','elapsed_seconds']
407
- row = [time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()), str(local_path) if local_path else '', repo_path, status, (msg or '').replace('\n',' '), f"{elapsed:.2f}" if elapsed is not None else '']
408
- write_header = not progress_log_path.exists()
409
- try:
410
- with open(progress_log_path, 'a', newline='', encoding='utf-8') as fh:
411
- writer = csv.writer(fh)
412
- if write_header:
413
- writer.writerow(header)
414
- writer.writerow(row)
415
- except Exception as e:
416
- print(f"Warning: failed to write progress log {progress_log_path}: {e}")
417
-
418
- run_stats = {
419
- 'planned': 0,
420
- 'skipped': 0,
421
- 'uploaded': 0,
422
- 'failed': 0,
423
- 'bytes_planned': 0,
424
- 'bytes_uploaded': 0,
425
- 'started_at': time.time(),
426
- }
427
-
428
- def record_summary() -> None:
429
- elapsed = time.time() - run_stats['started_at']
430
- summary_message = (
431
- f"planned={run_stats['planned']} skipped={run_stats['skipped']} "
432
- f"uploaded={run_stats['uploaded']} failed={run_stats['failed']} "
433
- f"bytes_planned={run_stats['bytes_planned']} bytes_uploaded={run_stats['bytes_uploaded']}"
434
- )
435
- append_progress(None, '', 'summary', summary_message, elapsed)
436
- try:
437
- summary_json = checkpoint_dir / f"{repo_id.replace('/', '__')}.summary.json"
438
- with open(summary_json, 'w', encoding='utf-8') as fh:
439
- json.dump({
440
- 'repo_id': repo_id,
441
- 'planned': run_stats['planned'],
442
- 'skipped': run_stats['skipped'],
443
- 'uploaded': run_stats['uploaded'],
444
- 'failed': run_stats['failed'],
445
- 'bytes_planned': run_stats['bytes_planned'],
446
- 'bytes_uploaded': run_stats['bytes_uploaded'],
447
- 'elapsed_seconds': round(elapsed, 2),
448
- }, fh, indent=2)
449
- except Exception as e:
450
- print(f"Warning: failed to write summary json: {e}")
451
-
452
- def remote_file_size(repo_path: str) -> int | None:
453
- if requests is None:
454
- return None
455
- # Construct raw file URL: https://huggingface.co/datasets/{repo_id}/resolve/main/{path}
456
- # Path components must be URL-quoted
457
- quoted = '/'.join(quote(p) for p in repo_path.split('/'))
458
- url = f"https://huggingface.co/datasets/{repo_id}/resolve/main/{quoted}"
459
- try:
460
- r = requests.head(url, allow_redirects=True, timeout=10)
461
- if r.status_code >= 400:
462
- return None
463
- cl = r.headers.get('content-length')
464
- if cl is None:
465
- return None
466
- return int(cl)
467
- except Exception:
468
- return None
469
-
470
- def create_repo() -> None:
471
- api.create_repo(repo_id=repo_id, repo_type="dataset", private=is_private, exist_ok=True)
472
-
473
- retry_call("create_repo", args.create_repo_attempts, args.retry_backoff, create_repo)
474
- # Prepare set of existing files if resume requested
475
- existing_files: set = set()
476
- checkpoint_uploaded: set = set()
477
- checkpoint_hashes: dict[str, str] = {}
478
- if args.resume:
479
- existing_files = list_existing_files()
480
- checkpoint_uploaded, checkpoint_hashes = load_checkpoint()
481
-
482
- # union of already-known uploaded files
483
- already_uploaded = existing_files.union(checkpoint_uploaded)
484
-
485
- def upload_segments() -> None:
486
- nonlocal csv_uploaded_with_segments
487
- # existing_files is computed above when resume is requested
488
-
489
- if args.resume_only:
490
- print("Resume-only mode: no upload will be performed.")
491
- files: List[Tuple[Path, str]] = []
492
- for p in segments_path.rglob("*"):
493
- if p.is_file():
494
- rel = p.relative_to(segments_path).as_posix()
495
- repo_path = f"{args.segments_path_in_repo}/{rel}"
496
- run_stats['bytes_planned'] += p.stat().st_size
497
- if repo_path in already_uploaded:
498
- checkpoint_hash = checkpoint_hashes.get(repo_path)
499
- if checkpoint_hash:
500
- local_hash = compute_sha256(p)
501
- if local_hash == checkpoint_hash:
502
- run_stats['skipped'] += 1
503
- append_progress(p, repo_path, 'skipped', 'already uploaded')
504
- continue
505
- files.append((p, repo_path))
506
-
507
- run_stats['planned'] = len(files)
508
- for local_path, repo_path in files:
509
- append_progress(local_path, repo_path, 'planned', 'resume-only')
510
- return
511
-
512
- if hasattr(api, "upload_large_folder"):
513
- staging_root = checkpoint_dir / "staging" / repo_id.replace("/", "__")
514
- print(f"Preparing persistent staging folder at {staging_root} ...")
515
- print(f"Checking for large species directories that need sharding...\n")
516
- staged_root = _prepare_sharded_staging(staging_root, segments_path, csv_path)
517
-
518
- total_bytes = 0
519
- total_files = 0
520
- for root, _, files in os.walk(staged_root):
521
- for name in files:
522
- file_path = Path(root) / name
523
- if ".cache" in file_path.parts:
524
- continue
525
- try:
526
- total_bytes += file_path.stat().st_size
527
- total_files += 1
528
- except Exception:
529
- continue
530
-
531
- print(f"Found {total_files} staged files to upload with upload_large_folder()")
532
-
533
- kwargs = {
534
- "folder_path": str(staged_root),
535
- "repo_id": repo_id,
536
- "repo_type": "dataset",
537
- "private": is_private,
538
- "num_workers": max(1, args.max_workers),
539
- "print_report": True,
540
- "print_report_every": 60,
541
- }
542
-
543
- def _run_upload():
544
- # Create progress bar and filter
545
- if tqdm is not None:
546
- pbar = tqdm(total=total_files, unit="file", unit_scale=False, desc="Uploading", leave=True)
547
- filter_writer = ProgressFilter(pbar, print)
548
- else:
549
- pbar = None
550
- filter_writer = None
551
-
552
- try:
553
- if filter_writer:
554
- with contextlib.redirect_stdout(filter_writer), contextlib.redirect_stderr(filter_writer):
555
- retry_call("upload_large_folder", args.upload_attempts, args.retry_backoff, lambda: api.upload_large_folder(**kwargs))
556
- filter_writer.flush()
557
- else:
558
- retry_call("upload_large_folder", args.upload_attempts, args.retry_backoff, lambda: api.upload_large_folder(**kwargs))
559
- finally:
560
- if pbar:
561
- try:
562
- pbar.close()
563
- except Exception:
564
- pass
565
-
566
- try:
567
- _run_upload()
568
- except Exception as upload_exc:
569
- print(f"upload_large_folder failed: {upload_exc}; falling back to per-file upload with throttling.")
570
- else:
571
- print("upload_large_folder completed successfully.")
572
- run_stats['planned'] = total_files
573
- run_stats['bytes_planned'] = total_bytes
574
- run_stats['uploaded'] = total_files
575
- run_stats['bytes_uploaded'] = total_bytes
576
- csv_uploaded_with_segments = True
577
- return
578
-
579
- # Otherwise, perform per-file upload and skip existing files
580
- print("Performing per-file upload (skipping already uploaded files if any)...")
581
-
582
- files: List[Tuple[Path, str]] = []
583
- for p in segments_path.rglob("*"):
584
- if p.is_file():
585
- rel = p.relative_to(segments_path).as_posix()
586
- repo_path = f"{args.segments_path_in_repo}/{rel}"
587
- run_stats['bytes_planned'] += p.stat().st_size
588
- skip = False
589
- if repo_path in already_uploaded:
590
- local_hash = None
591
- checkpoint_hash = checkpoint_hashes.get(repo_path)
592
- if checkpoint_hash:
593
- local_hash = compute_sha256(p)
594
- if local_hash != checkpoint_hash:
595
- skip = False
596
- print(f"Hash mismatch for {repo_path}; will re-upload.")
597
- else:
598
- skip = True
599
- elif args.verify_remote and requests is not None:
600
- local_size = p.stat().st_size
601
- if rate_limiter:
602
- rate_limiter.acquire("verify_remote")
603
- remote_size, remote_etag = remote_file_info(repo_path)
604
- if args.verify_etag and remote_etag:
605
- # We do not compute a remote-matching hash locally here; rely on size and the checkpoint hash path.
606
- skip = (remote_size is not None and local_size == remote_size)
607
- else:
608
- skip = (remote_size is not None and local_size == remote_size)
609
- else:
610
- skip = True
611
-
612
- if skip:
613
- run_stats['skipped'] += 1
614
- append_progress(p, repo_path, 'skipped', 'already uploaded')
615
- continue
616
- files.append((p, repo_path))
617
-
618
- run_stats['planned'] = len(files)
619
-
620
- if args.resume_only:
621
- print("Resume-only mode: no upload will be performed.")
622
- for local_path, repo_path in files:
623
- append_progress(local_path, repo_path, 'planned', 'resume-only')
624
- return
625
-
626
- print(f"Found {len(files)} files to upload; launching {args.max_workers} workers")
627
-
628
- # Progress bar for per-file uploads (counts files, not bytes)
629
- file_pbar = FilesProgressBar(total_files=len(files), logger=print)
630
-
631
- def upload_one(pair: Tuple[Path, str]):
632
- local_path, repo_path = pair
633
- start = time.time()
634
- try:
635
- def do_upload():
636
- if rate_limiter:
637
- rate_limiter.acquire("file_upload")
638
- api.upload_file(
639
- path_or_fileobj=str(local_path),
640
- path_in_repo=repo_path,
641
- repo_id=repo_id,
642
- repo_type="dataset",
643
- commit_message=args.commit_message,
644
- )
645
-
646
- retry_call(f"upload_file:{repo_path}", args.upload_attempts, args.retry_backoff, do_upload)
647
- except Exception as e:
648
- elapsed = time.time() - start
649
- # Log error to CSV for tracking
650
- append_progress(local_path, repo_path, 'failed', str(e), elapsed)
651
- # Update progress bar with error
652
- file_pbar.update_file_error(1, f"{repo_path}: {str(e)[:60]}")
653
- raise
654
- else:
655
- elapsed = time.time() - start
656
- # Only log successful uploads to CSV (optional, can be verbose)
657
- # append_progress(local_path, repo_path, 'uploaded', None, elapsed)
658
- # Update progress bar
659
- file_pbar.update_file(1)
660
- return repo_path, local_path
661
-
662
- try:
663
- with concurrent.futures.ThreadPoolExecutor(max_workers=args.max_workers) as ex:
664
- futures = [ex.submit(upload_one, f) for f in files]
665
- for fut in concurrent.futures.as_completed(futures):
666
- try:
667
- repo_path, local_path = fut.result()
668
- except Exception as e:
669
- # Error already logged above, just continue
670
- continue
671
- finally:
672
- file_pbar.close()
673
- file_pbar.log_errors()
674
- # mark as uploaded in checkpoint
675
- if repo_path:
676
- already_uploaded.add(repo_path)
677
- try:
678
- checkpoint_hashes[repo_path] = compute_sha256(local_path)
679
- except Exception as e:
680
- print(f"Warning: failed to hash {repo_path} after upload: {e}")
681
- run_stats['uploaded'] += 1
682
- try:
683
- run_stats['bytes_uploaded'] += Path(local_path).stat().st_size
684
- except Exception:
685
- pass
686
- save_checkpoint(already_uploaded, checkpoint_hashes)
687
- if pbar is not None:
688
- try:
689
- pbar.close()
690
- except Exception:
691
- pass
692
-
693
- retry_call("upload_segments", args.upload_attempts, args.retry_backoff, upload_segments)
694
-
695
- if args.resume_only:
696
- print("Resume-only mode completed; no uploads were performed.")
697
- record_summary()
698
- print()
699
- print("Upload complete.")
700
- print(f"Dataset URL: https://huggingface.co/datasets/{repo_id}")
701
- return 0
702
-
703
- if csv_path:
704
-
705
- def upload_csv() -> None:
706
- skip_csv = False
707
- if args.resume and args.csv_path_in_repo in already_uploaded:
708
- if args.verify_remote and requests is not None:
709
- local_size = csv_path.stat().st_size
710
- if rate_limiter:
711
- rate_limiter.acquire("verify_remote")
712
- remote_size, remote_etag = remote_file_info(args.csv_path_in_repo)
713
- if args.verify_etag and remote_etag:
714
- # no local etag available -> fall back to size check
715
- if remote_size is not None and local_size == remote_size:
716
- skip_csv = True
717
- else:
718
- if remote_size is not None and local_size == remote_size:
719
- skip_csv = True
720
-
721
- if skip_csv:
722
- print(f"CSV already exists at {args.csv_path_in_repo}; skipping upload.")
723
- run_stats['skipped'] += 1
724
- append_progress(csv_path, args.csv_path_in_repo, 'skipped', 'already uploaded')
725
- return
726
-
727
- start = time.time()
728
- try:
729
- if rate_limiter:
730
- rate_limiter.acquire("csv_upload")
731
- api.upload_file(
732
- path_or_fileobj=str(csv_path),
733
- path_in_repo=args.csv_path_in_repo,
734
- repo_id=repo_id,
735
- repo_type="dataset",
736
- commit_message=f"Upload CSV for {repo_id}",
737
- )
738
- except Exception as e:
739
- elapsed = time.time() - start
740
- run_stats['failed'] += 1
741
- append_progress(csv_path, args.csv_path_in_repo, 'failed', str(e), elapsed)
742
- raise
743
- else:
744
- elapsed = time.time() - start
745
- run_stats['uploaded'] += 1
746
- try:
747
- run_stats['bytes_uploaded'] += csv_path.stat().st_size
748
- except Exception:
749
- pass
750
- try:
751
- checkpoint_hashes[args.csv_path_in_repo] = compute_sha256(csv_path)
752
- save_checkpoint(already_uploaded.union({args.csv_path_in_repo}), checkpoint_hashes)
753
- except Exception as e:
754
- print(f"Warning: failed to update CSV checkpoint hash: {e}")
755
- append_progress(csv_path, args.csv_path_in_repo, 'uploaded', None, elapsed)
756
-
757
- retry_call("upload_csv", args.upload_attempts, args.retry_backoff, upload_csv)
758
-
759
- print()
760
- print("Upload complete.")
761
- print(f"Dataset URL: https://huggingface.co/datasets/{repo_id}")
762
- record_summary()
763
- return 0
764
-
765
-
766
- if __name__ == "__main__":
767
- raise SystemExit(main())
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
hf_bulk_upload_tool/upload_logic.py DELETED
@@ -1,886 +0,0 @@
1
- #!/usr/bin/env python3
2
- """
3
- Upload logic refactored for GUI integration.
4
- This module contains the core upload functionality extracted from upload_dataset.py
5
- and refactored to accept a config dict and a logger callback instead of argparse.
6
- """
7
-
8
- from __future__ import annotations
9
-
10
- import hashlib
11
- import io
12
- import os
13
- import sys
14
- import shutil
15
- import subprocess
16
- import time
17
- import contextlib
18
- from pathlib import Path
19
- import concurrent.futures
20
- from typing import List, Tuple, Callable, Optional
21
- import json
22
- import csv
23
- from datetime import datetime
24
- from urllib.parse import quote
25
-
26
- try:
27
- import requests
28
- except Exception:
29
- requests = None
30
- try:
31
- from tqdm import tqdm
32
- except Exception:
33
- tqdm = None
34
-
35
- try:
36
- from sharding_utils import shard_directory, MAX_FILES_PER_DIR
37
- from progress_bar_utils import ProgressFilter, FilesProgressBar
38
- except ImportError:
39
- # Fallback for relative imports during development
40
- from .sharding_utils import shard_directory, MAX_FILES_PER_DIR
41
- from .progress_bar_utils import ProgressFilter, FilesProgressBar
42
-
43
-
44
- def configure_hf_env() -> None:
45
- """Configure Hugging Face environment variables."""
46
- os.environ.setdefault("HF_HUB_ETAG_TIMEOUT", "20")
47
- os.environ.setdefault("HF_HUB_DOWNLOAD_TIMEOUT", "120")
48
- os.environ.setdefault("HF_XET_HIGH_PERFORMANCE", "1")
49
-
50
-
51
- configure_hf_env()
52
-
53
-
54
- def _ensure_std_streams() -> None:
55
- """Ensure std streams exist in windowed/frozen runs.
56
-
57
- In PyInstaller windowed mode, sys.stderr/sys.stdout may be None.
58
- Some libraries (including logging handlers inside huggingface_hub flow)
59
- expect writable streams and crash otherwise.
60
- """
61
- if sys.stdout is None:
62
- sys.stdout = open(os.devnull, "w", encoding="utf-8")
63
- if sys.stderr is None:
64
- sys.stderr = open(os.devnull, "w", encoding="utf-8")
65
-
66
-
67
- _ensure_std_streams()
68
-
69
- from huggingface_hub import HfApi
70
-
71
-
72
- class _CallbackWriter(io.TextIOBase):
73
- """File-like writer that forwards complete lines to a callback logger."""
74
-
75
- def __init__(self, callback: Callable[[str], None]):
76
- super().__init__()
77
- self._callback = callback
78
- self._buffer = ""
79
-
80
- def write(self, s: str) -> int:
81
- if not s:
82
- return 0
83
- self._buffer += s
84
- while "\n" in self._buffer:
85
- line, self._buffer = self._buffer.split("\n", 1)
86
- if line.strip():
87
- self._callback(line)
88
- return len(s)
89
-
90
- def flush(self) -> None:
91
- if self._buffer.strip():
92
- self._callback(self._buffer.strip())
93
- self._buffer = ""
94
-
95
-
96
- class RateLimiter:
97
- """Sliding window rate limiter for HF API (1000 requests per 5 minutes)."""
98
- def __init__(self, max_requests: int = 1000, window_seconds: int = 300):
99
- self.max_requests = max_requests
100
- self.window_seconds = window_seconds
101
- self.request_times: list[tuple[float, str]] = []
102
-
103
- def acquire(self, request_type: str = "upload") -> None:
104
- """Wait if necessary to stay within rate limits, then register the request."""
105
- now = time.time()
106
-
107
- self.request_times = [
108
- (ts, req_type) for ts, req_type in self.request_times
109
- if now - ts < self.window_seconds
110
- ]
111
-
112
- if len(self.request_times) >= self.max_requests:
113
- oldest_ts = self.request_times[0][0]
114
- wait_time = (oldest_ts + self.window_seconds) - now
115
- if wait_time > 0:
116
- time.sleep(wait_time)
117
- self.request_times = []
118
-
119
- self.request_times.append((now, request_type))
120
-
121
- def get_window_usage(self) -> tuple[int, int]:
122
- """Return (requests_in_window, max_requests)."""
123
- now = time.time()
124
- self.request_times = [
125
- (ts, req_type) for ts, req_type in self.request_times
126
- if now - ts < self.window_seconds
127
- ]
128
- return len(self.request_times), self.max_requests
129
-
130
-
131
- def ensure_directory_link(link_path: Path, target_path: Path) -> None:
132
- """Create a directory link (Windows junction or Unix symlink)."""
133
- if link_path.exists() or link_path.is_symlink():
134
- try:
135
- if link_path.resolve() == target_path.resolve():
136
- return
137
- except Exception:
138
- pass
139
- if link_path.is_symlink() or link_path.is_file():
140
- link_path.unlink()
141
- else:
142
- shutil.rmtree(link_path)
143
-
144
- link_path.parent.mkdir(parents=True, exist_ok=True)
145
-
146
- if os.name == "nt":
147
- try:
148
- subprocess.run(
149
- ["cmd", "/c", "mklink", "/J", str(link_path), str(target_path)],
150
- check=True,
151
- capture_output=True,
152
- text=True,
153
- creationflags=subprocess.CREATE_NO_WINDOW,
154
- )
155
- return
156
- except Exception:
157
- pass
158
-
159
- os.symlink(target_path, link_path, target_is_directory=True)
160
-
161
-
162
- def prepare_staging_folder(staging_root: Path, segments_path: Path, csv_path: Path | None) -> Path:
163
- """Build a persistent staging tree with the repo layout expected by HF."""
164
- staging_root.mkdir(parents=True, exist_ok=True)
165
-
166
- audio_link = staging_root / "audio"
167
- ensure_directory_link(audio_link, segments_path)
168
-
169
- if csv_path is not None:
170
- csv_dest = staging_root / "index" / "detections.csv"
171
- csv_dest.parent.mkdir(parents=True, exist_ok=True)
172
- shutil.copy2(csv_path, csv_dest)
173
-
174
- return staging_root
175
-
176
-
177
- def _inventory_cache_path(checkpoint_dir: Path, segments_path: Path) -> Path:
178
- """Return cache file path for local inventory of a segments directory."""
179
- key = hashlib.sha1(str(segments_path).encode("utf-8")).hexdigest()[:12]
180
- return checkpoint_dir / f"inventory_{key}.json"
181
-
182
-
183
- def _scan_species_dir(species_dir: Path) -> tuple[int, int]:
184
- """Scan one species directory and return (file_count, total_bytes)."""
185
- count = 0
186
- total_bytes = 0
187
- for p in species_dir.rglob("*"):
188
- if p.is_file():
189
- count += 1
190
- try:
191
- total_bytes += p.stat().st_size
192
- except Exception:
193
- pass
194
- return count, total_bytes
195
-
196
-
197
- def _build_species_inventory(segments_path: Path, checkpoint_dir: Path, logger: Callable[[str], None]) -> list[dict]:
198
- """Build per-species inventory with lightweight persistent cache.
199
-
200
- Cache key is based on top-level species folder mtime. If folder mtime changed,
201
- species is rescanned. This preserves correctness while avoiding repeated full scans
202
- across reruns.
203
- """
204
- cache_path = _inventory_cache_path(checkpoint_dir, segments_path)
205
- cached_species: dict = {}
206
-
207
- if cache_path.exists():
208
- try:
209
- with open(cache_path, "r", encoding="utf-8") as fh:
210
- payload = json.load(fh)
211
- cached_species = payload.get("species", {}) if isinstance(payload, dict) else {}
212
- except Exception:
213
- cached_species = {}
214
-
215
- inventory: list[dict] = []
216
- cache_hits = 0
217
- total_species = 0
218
- new_cache_species: dict = {}
219
-
220
- for species_dir in sorted(segments_path.iterdir()):
221
- if not species_dir.is_dir():
222
- continue
223
- total_species += 1
224
- species_name = species_dir.name
225
- try:
226
- mtime_ns = species_dir.stat().st_mtime_ns
227
- except Exception:
228
- mtime_ns = 0
229
-
230
- cached = cached_species.get(species_name)
231
- if isinstance(cached, dict) and int(cached.get("mtime_ns", -1)) == int(mtime_ns):
232
- file_count = int(cached.get("file_count", 0))
233
- total_bytes = int(cached.get("total_bytes", 0))
234
- cache_hits += 1
235
- else:
236
- file_count, total_bytes = _scan_species_dir(species_dir)
237
-
238
- entry = {
239
- "species_name": species_name,
240
- "path": species_dir,
241
- "file_count": file_count,
242
- "total_bytes": total_bytes,
243
- "mtime_ns": int(mtime_ns),
244
- }
245
- inventory.append(entry)
246
-
247
- new_cache_species[species_name] = {
248
- "mtime_ns": int(mtime_ns),
249
- "file_count": int(file_count),
250
- "total_bytes": int(total_bytes),
251
- }
252
-
253
- try:
254
- cache_path.parent.mkdir(parents=True, exist_ok=True)
255
- with open(cache_path, "w", encoding="utf-8") as fh:
256
- json.dump(
257
- {
258
- "segments_root": str(segments_path),
259
- "updated_at": datetime.utcnow().isoformat() + "Z",
260
- "species": new_cache_species,
261
- },
262
- fh,
263
- indent=2,
264
- )
265
- except Exception as e:
266
- logger(f"Warning: failed to save inventory cache {cache_path}: {e}")
267
-
268
- logger(f"[Scan] Species inventory cache hit: {cache_hits}/{total_species}")
269
- return inventory
270
-
271
-
272
- def _prepare_sharded_staging(
273
- staging_root: Path,
274
- segments_path: Path,
275
- csv_path: Path | None,
276
- logger: Callable[[str], None],
277
- inventory: list[dict] | None = None,
278
- ) -> Path:
279
- """Prepare staging folder with automatic sharding for large species directories.
280
-
281
- Detects species folders with >9500 files and automatically shards them into
282
- numbered subdirectories to comply with HF's 10k files per directory limit.
283
- Validation tools can transparently read shards via sharding_utils.
284
-
285
- Args:
286
- staging_root: Root directory for staging
287
- segments_path: Path to source segments (species directories)
288
- csv_path: Optional path to CSV to copy
289
- logger: Callback function for logging messages
290
-
291
- Returns:
292
- Path to the prepared staging root directory
293
- """
294
- staging_root.mkdir(parents=True, exist_ok=True)
295
- audio_staging = staging_root / "audio"
296
- audio_staging.mkdir(parents=True, exist_ok=True)
297
-
298
- sharded_species = []
299
-
300
- # Check each species folder for sharding using precomputed inventory when available
301
- source_items = inventory
302
- if source_items is None:
303
- source_items = []
304
- for species_dir in sorted(segments_path.iterdir()):
305
- if species_dir.is_dir():
306
- source_items.append(
307
- {
308
- "species_name": species_dir.name,
309
- "path": species_dir,
310
- "file_count": 0,
311
- }
312
- )
313
-
314
- for item in source_items:
315
- species_dir = Path(item["path"])
316
- file_count = int(item.get("file_count", 0))
317
-
318
- if file_count > MAX_FILES_PER_DIR:
319
- # Need to shard this species
320
- logger(f"[Sharding] {species_dir.name} exceeds {MAX_FILES_PER_DIR} files, dividing into shards...")
321
- try:
322
- shards = shard_directory(species_dir, audio_staging)
323
- sharded_species.append((species_dir.name, len(shards)))
324
- logger(f"[Sharding] Created {len(shards)} shards for {species_dir.name}")
325
- except Exception as e:
326
- logger(f"[Sharding] ERROR sharding {species_dir.name}: {e}")
327
- logger(f"[Sharding] Falling back to single directory link")
328
- ensure_directory_link(audio_staging / species_dir.name, species_dir)
329
- else:
330
- # Small species, just link it
331
- ensure_directory_link(audio_staging / species_dir.name, species_dir)
332
-
333
- # Log sharding summary
334
- if sharded_species:
335
- logger(f"")
336
- logger(f"[Sharding] Summary: {len(sharded_species)} species were divided")
337
- for species_name, num_shards in sharded_species:
338
- logger(f" - {species_name}: {num_shards} shards")
339
- logger(f"[Sharding] Validation tools will transparently read all shards as single species")
340
- logger(f"")
341
-
342
- # Copy CSV if provided
343
- if csv_path is not None:
344
- csv_dest = staging_root / "index" / "detections.csv"
345
- csv_dest.parent.mkdir(parents=True, exist_ok=True)
346
- shutil.copy2(csv_path, csv_dest)
347
-
348
- return staging_root
349
-
350
-
351
- def retry_call(label: str, attempts: int, backoff: float, func, logger: Callable[[str], None], stop_event=None):
352
- """Execute a function with retry logic, logging via callback."""
353
- last_error: Exception | None = None
354
- for attempt in range(1, attempts + 1):
355
- # If cancellation requested, abort immediately
356
- try:
357
- if stop_event is not None and getattr(stop_event, 'is_set', lambda: False)():
358
- raise Exception(f"{label} cancelled by user")
359
- except Exception:
360
- raise
361
- try:
362
- logger(f"[{label}] attempt {attempt}/{attempts}")
363
- return func()
364
- except Exception as exc:
365
- last_error = exc
366
- if attempt >= attempts:
367
- break
368
-
369
- wait_seconds = backoff * attempt
370
- try:
371
- resp = getattr(exc, 'response', None)
372
- if resp is not None:
373
- status = getattr(resp, 'status_code', None)
374
- if status == 429:
375
- ra = resp.headers.get('Retry-After') if hasattr(resp, 'headers') else None
376
- if ra:
377
- try:
378
- wait_seconds = max(wait_seconds, int(ra))
379
- except Exception:
380
- wait_seconds = max(wait_seconds, backoff * attempt * 4)
381
- else:
382
- wait_seconds = max(wait_seconds, backoff * attempt * 4)
383
- elif '429' in str(exc) or 'Too Many Requests' in str(exc):
384
- wait_seconds = max(wait_seconds, backoff * attempt * 4)
385
- except Exception:
386
- pass
387
-
388
- logger(f"[{label}] attempt {attempt} failed: {exc}")
389
- logger(f"[{label}] retrying in {wait_seconds:.0f}s")
390
- time.sleep(wait_seconds)
391
-
392
- raise Exception(f"{label} failed after {attempts} attempts: {last_error}")
393
-
394
-
395
- def run_upload_logic(config: dict, logger: Callable[[str], None]) -> int:
396
- """
397
- Core upload logic refactored for GUI integration.
398
-
399
- Args:
400
- config: Dictionary with keys:
401
- - hf_token: str (required)
402
- - repo_id: str (required, format: "username/repo-name")
403
- - segments: str or Path (required, directory with audio files)
404
- - csv: str or Path (optional, path to CSV file)
405
- - segments_path_in_repo: str (default: "audio")
406
- - csv_path_in_repo: str (default: "index/detections.csv")
407
- - private: bool (default: True)
408
- - resume: bool (default: False)
409
- - verify_remote: bool (default: False)
410
- - verify_etag: bool (default: False)
411
- - dry_run: bool (default: False)
412
- - rate_limit_aware: bool (default: True)
413
- - rate_limit_max_requests: int (default: 950)
414
- - rate_limit_window: int (default: 300)
415
- - create_repo_attempts: int (default: 3)
416
- - upload_attempts: int (default: 3)
417
- - retry_backoff: float (default: 5.0)
418
- - max_workers: int (default: 1)
419
- - checkpoint_dir: str or Path (default: .checkpoints)
420
- - commit_message: str (default: "Upload BirdNET segments")
421
-
422
- logger: Callable that accepts a string message (for GUI textbox updates)
423
-
424
- Returns:
425
- 0 on success, raises Exception on failure
426
- """
427
-
428
- # Extract and validate config
429
- token = config.get("hf_token")
430
- if not token:
431
- raise ValueError("hf_token is required")
432
-
433
- repo_id = config.get("repo_id")
434
- if not repo_id:
435
- raise ValueError("repo_id is required")
436
-
437
- segments = config.get("segments")
438
- if not segments:
439
- raise ValueError("segments is required")
440
-
441
- segments_path = Path(segments).expanduser().resolve()
442
- if not segments_path.is_dir():
443
- raise ValueError(f"Segments path does not exist or is not a directory: {segments_path}")
444
-
445
- csv_path = None
446
- if config.get("csv"):
447
- csv_path = Path(config.get("csv")).expanduser().resolve()
448
- if not csv_path.is_file():
449
- raise ValueError(f"CSV path does not exist or is not a file: {csv_path}")
450
-
451
- # Config parameters
452
- segments_path_in_repo = config.get("segments_path_in_repo", "audio")
453
- csv_path_in_repo = config.get("csv_path_in_repo", "index/detections.csv")
454
- is_private = config.get("private", True)
455
- resume = config.get("resume", False)
456
- verify_remote = config.get("verify_remote", False)
457
- verify_etag = config.get("verify_etag", False)
458
- dry_run = config.get("dry_run", False)
459
- rate_limit_aware = config.get("rate_limit_aware", True)
460
- rate_limit_max_requests = config.get("rate_limit_max_requests", 950)
461
- rate_limit_window = config.get("rate_limit_window", 300)
462
- create_repo_attempts = config.get("create_repo_attempts", 3)
463
- upload_attempts = config.get("upload_attempts", 3)
464
- retry_backoff = config.get("retry_backoff", 5.0)
465
- max_workers = config.get("max_workers", 1)
466
- checkpoint_dir = Path(config.get("checkpoint_dir", Path(__file__).parent / ".checkpoints"))
467
- commit_message = config.get("commit_message", "Upload BirdNET segments")
468
-
469
- # Log configuration
470
- logger(f"Repository: {repo_id}")
471
- logger(f"Segments: {segments_path}")
472
- logger(f"Segments destination: {segments_path_in_repo}")
473
-
474
- csv_uploaded_with_segments = False
475
- if csv_path and not csv_uploaded_with_segments:
476
- logger(f"CSV: {csv_path}")
477
- logger(f"CSV destination: {csv_path_in_repo}")
478
-
479
- logger(f"Visibility: {'private' if is_private else 'public'}")
480
-
481
- if dry_run:
482
- logger("Dry run selected. No network calls will be made.")
483
- return 0
484
-
485
- # Cancellation event (optional). GUI should pass a threading.Event via config['stop_event']
486
- stop_event = config.get('stop_event')
487
- # Suppress console progress (GUI mode): when True, do not create tqdm bars
488
- suppress_console = config.get('suppress_console_progress', False)
489
-
490
- api = HfApi(token=token)
491
-
492
- # Initialize rate limiter
493
- rate_limiter = None
494
- if rate_limit_aware:
495
- rate_limiter = RateLimiter(max_requests=rate_limit_max_requests, window_seconds=rate_limit_window)
496
- logger(f"[RateLimit] Enabled: max {rate_limit_max_requests} requests per {rate_limit_window}s")
497
-
498
- def list_existing_files() -> set:
499
- try:
500
- logger("Listing files in target repo to determine already-uploaded files...")
501
- if rate_limiter:
502
- rate_limiter.acquire("list_repo_files")
503
- files = api.list_repo_files(repo_id=repo_id, repo_type="dataset")
504
- return set(files or [])
505
- except Exception as e:
506
- logger(f"Warning: could not list repo files: {e}")
507
- return set()
508
-
509
- # Checkpoint management
510
- checkpoint_dir.mkdir(parents=True, exist_ok=True)
511
- checkpoint_file = checkpoint_dir / f"{repo_id.replace('/', '__')}.json"
512
- progress_log_path = checkpoint_dir / 'progress.csv'
513
-
514
- def load_checkpoint() -> tuple[set, dict[str, str]]:
515
- if not checkpoint_file.exists():
516
- return set(), {}
517
- try:
518
- with open(checkpoint_file, 'r', encoding='utf-8') as fh:
519
- data = json.load(fh)
520
- if isinstance(data, list):
521
- return set(data), {}
522
- uploaded = set(data.get('uploaded', []))
523
- file_hashes = data.get('file_hashes', {})
524
- if not isinstance(file_hashes, dict):
525
- file_hashes = {}
526
- return uploaded, {str(k): str(v) for k, v in file_hashes.items()}
527
- except Exception as e:
528
- logger(f"Warning: failed to load checkpoint {checkpoint_file}: {e}")
529
- return set(), {}
530
-
531
- def save_checkpoint(uploaded_set: set, file_hashes: dict[str, str]) -> None:
532
- try:
533
- with open(checkpoint_file, 'w', encoding='utf-8') as fh:
534
- json.dump({'uploaded': sorted(list(uploaded_set)), 'file_hashes': file_hashes}, fh, indent=2)
535
- except Exception as e:
536
- logger(f"Warning: failed to save checkpoint {checkpoint_file}: {e}")
537
-
538
- def compute_sha256(file_path: Path) -> str:
539
- digest = hashlib.sha256()
540
- with open(file_path, 'rb') as fh:
541
- for chunk in iter(lambda: fh.read(1024 * 1024), b''):
542
- digest.update(chunk)
543
- return digest.hexdigest()
544
-
545
- def remote_file_info(repo_path: str) -> Tuple[int | None, str | None]:
546
- """Return (size, etag) for a repo file via HTTP HEAD, or (None, None)."""
547
- if requests is None:
548
- return None, None
549
- quoted = '/'.join(quote(p) for p in repo_path.split('/'))
550
- url = f"https://huggingface.co/datasets/{repo_id}/resolve/main/{quoted}"
551
- try:
552
- r = requests.head(url, allow_redirects=True, timeout=10)
553
- if r.status_code >= 400:
554
- return None, None
555
- cl = r.headers.get('content-length')
556
- etag = r.headers.get('etag')
557
- size = int(cl) if cl is not None else None
558
- return size, etag
559
- except Exception:
560
- return None, None
561
-
562
- def append_progress(local_path: Path | None, repo_path: str, status: str, msg: str | None = None, elapsed: float | None = None) -> None:
563
- header = ['timestamp','local_path','repo_path','status','message','elapsed_seconds']
564
- row = [time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()), str(local_path) if local_path else '', repo_path, status, (msg or '').replace('\n',' '), f"{elapsed:.2f}" if elapsed is not None else '']
565
- write_header = not progress_log_path.exists()
566
- try:
567
- with open(progress_log_path, 'a', newline='', encoding='utf-8') as fh:
568
- writer = csv.writer(fh)
569
- if write_header:
570
- writer.writerow(header)
571
- writer.writerow(row)
572
- except Exception as e:
573
- logger(f"Warning: failed to write progress log {progress_log_path}: {e}")
574
-
575
- run_stats = {
576
- 'planned': 0,
577
- 'skipped': 0,
578
- 'uploaded': 0,
579
- 'failed': 0,
580
- 'bytes_planned': 0,
581
- 'bytes_uploaded': 0,
582
- 'started_at': time.time(),
583
- }
584
-
585
- def record_summary() -> None:
586
- elapsed = time.time() - run_stats['started_at']
587
- summary_message = (
588
- f"planned={run_stats['planned']} skipped={run_stats['skipped']} "
589
- f"uploaded={run_stats['uploaded']} failed={run_stats['failed']} "
590
- f"bytes_planned={run_stats['bytes_planned']} bytes_uploaded={run_stats['bytes_uploaded']}"
591
- )
592
- append_progress(None, '', 'summary', summary_message, elapsed)
593
- try:
594
- summary_json = checkpoint_dir / f"{repo_id.replace('/', '__')}.summary.json"
595
- with open(summary_json, 'w', encoding='utf-8') as fh:
596
- json.dump({
597
- 'repo_id': repo_id,
598
- 'planned': run_stats['planned'],
599
- 'skipped': run_stats['skipped'],
600
- 'uploaded': run_stats['uploaded'],
601
- 'failed': run_stats['failed'],
602
- 'bytes_planned': run_stats['bytes_planned'],
603
- 'bytes_uploaded': run_stats['bytes_uploaded'],
604
- 'elapsed_seconds': round(elapsed, 2),
605
- }, fh, indent=2)
606
- except Exception as e:
607
- logger(f"Warning: failed to write summary json: {e}")
608
-
609
- # Create repo
610
- def create_repo() -> None:
611
- api.create_repo(repo_id=repo_id, repo_type="dataset", private=is_private, exist_ok=True)
612
-
613
- # Create repo (with retry and cancellation support)
614
- retry_call("create_repo", create_repo_attempts, retry_backoff, create_repo, logger, stop_event)
615
-
616
- # Load checkpoints
617
- existing_files: set = set()
618
- checkpoint_uploaded: set = set()
619
- checkpoint_hashes: dict[str, str] = {}
620
- existing_files_loaded = False
621
- if resume:
622
- checkpoint_uploaded, checkpoint_hashes = load_checkpoint()
623
-
624
- already_uploaded = existing_files.union(checkpoint_uploaded)
625
-
626
- def ensure_existing_files_loaded() -> None:
627
- nonlocal existing_files_loaded, existing_files, already_uploaded
628
- if not resume or existing_files_loaded:
629
- return
630
- existing_files = list_existing_files()
631
- existing_files_loaded = True
632
- already_uploaded = existing_files.union(checkpoint_uploaded)
633
-
634
- def upload_segments() -> None:
635
- nonlocal csv_uploaded_with_segments
636
-
637
- if hasattr(api, "upload_large_folder"):
638
- staging_root = checkpoint_dir / "staging" / repo_id.replace("/", "__")
639
- logger(f"Preparing persistent staging folder at {staging_root} ...")
640
- logger(f"Checking for large species directories that need sharding...\n")
641
- inventory = _build_species_inventory(segments_path, checkpoint_dir, logger)
642
- staged_root = _prepare_sharded_staging(staging_root, segments_path, csv_path, logger, inventory=inventory)
643
-
644
- total_files = sum(int(item.get("file_count", 0)) for item in inventory)
645
- total_bytes = sum(int(item.get("total_bytes", 0)) for item in inventory)
646
- if csv_path is not None:
647
- total_files += 1
648
- try:
649
- total_bytes += csv_path.stat().st_size
650
- except Exception:
651
- pass
652
-
653
- logger(f"Found {total_files} staged files to upload with upload_large_folder()")
654
-
655
- kwargs = {
656
- "folder_path": str(staged_root),
657
- "repo_id": repo_id,
658
- "repo_type": "dataset",
659
- "private": is_private,
660
- "num_workers": max(1, max_workers),
661
- # Keep HF periodic reports disabled; GUI receives compact status via ProgressFilter.
662
- "print_report": False,
663
- }
664
-
665
- def _run_upload():
666
- # Create progress bar and filter
667
- if tqdm is not None and not suppress_console:
668
- pbar = tqdm(total=total_files, unit="file", unit_scale=False, desc="Uploading", leave=True)
669
- filter_writer = ProgressFilter(pbar, logger)
670
- else:
671
- pbar = None
672
- filter_writer = ProgressFilter(None, logger)
673
-
674
- try:
675
- if filter_writer:
676
- with contextlib.redirect_stdout(filter_writer), contextlib.redirect_stderr(filter_writer):
677
- retry_call("upload_large_folder", upload_attempts, retry_backoff, lambda: api.upload_large_folder(**kwargs), logger, stop_event)
678
- filter_writer.flush()
679
- else:
680
- retry_call("upload_large_folder", upload_attempts, retry_backoff, lambda: api.upload_large_folder(**kwargs), logger, stop_event)
681
- finally:
682
- if pbar:
683
- try:
684
- pbar.close()
685
- except Exception:
686
- pass
687
-
688
- try:
689
- # Check cancellation before starting heavy upload
690
- if stop_event is not None and getattr(stop_event, 'is_set', lambda: False)():
691
- raise Exception("Upload cancelled by user before starting batch upload")
692
- _run_upload()
693
- except Exception as upload_exc:
694
- logger(f"upload_large_folder failed: {upload_exc}; falling back to per-file upload with throttling.")
695
- else:
696
- logger("upload_large_folder completed successfully.")
697
- run_stats['planned'] = total_files
698
- run_stats['bytes_planned'] = total_bytes
699
- run_stats['uploaded'] = total_files
700
- run_stats['bytes_uploaded'] = total_bytes
701
- csv_uploaded_with_segments = True
702
- return
703
-
704
- # Per-file fallback
705
- logger("Performing per-file upload (skipping already uploaded files if any)...")
706
-
707
- # Remote repo listing is only needed for per-file resume logic.
708
- ensure_existing_files_loaded()
709
-
710
- files: List[Tuple[Path, str]] = []
711
- for p in segments_path.rglob("*"):
712
- if p.is_file():
713
- rel = p.relative_to(segments_path).as_posix()
714
- repo_path = f"{segments_path_in_repo}/{rel}"
715
- run_stats['bytes_planned'] += p.stat().st_size
716
- skip = False
717
- if repo_path in already_uploaded:
718
- checkpoint_hash = checkpoint_hashes.get(repo_path)
719
- if checkpoint_hash:
720
- local_hash = compute_sha256(p)
721
- if local_hash != checkpoint_hash:
722
- skip = False
723
- logger(f"Hash mismatch for {repo_path}; will re-upload.")
724
- else:
725
- skip = True
726
- elif verify_remote and requests is not None:
727
- local_size = p.stat().st_size
728
- if rate_limiter:
729
- rate_limiter.acquire("verify_remote")
730
- remote_size, remote_etag = remote_file_info(repo_path)
731
- if verify_etag and remote_etag:
732
- skip = (remote_size is not None and local_size == remote_size)
733
- else:
734
- skip = (remote_size is not None and local_size == remote_size)
735
- else:
736
- skip = True
737
-
738
- if skip:
739
- run_stats['skipped'] += 1
740
- append_progress(p, repo_path, 'skipped', 'already uploaded')
741
- continue
742
- files.append((p, repo_path))
743
-
744
- run_stats['planned'] = len(files)
745
- logger(f"Found {len(files)} files to upload; launching {max_workers} workers")
746
-
747
- # Progress bar for per-file uploads (counts files, not bytes)
748
- file_pbar = FilesProgressBar(total_files=len(files), logger=logger, suppress_console=suppress_console)
749
-
750
- def upload_one(pair: Tuple[Path, str]):
751
- local_path, repo_path = pair
752
- start = time.time()
753
- try:
754
- def do_upload():
755
- if rate_limiter:
756
- rate_limiter.acquire("file_upload")
757
- api.upload_file(
758
- path_or_fileobj=str(local_path),
759
- path_in_repo=repo_path,
760
- repo_id=repo_id,
761
- repo_type="dataset",
762
- commit_message=commit_message,
763
- )
764
-
765
- retry_call(f"upload_file:{repo_path}", upload_attempts, retry_backoff, do_upload, logger, stop_event)
766
- except Exception as e:
767
- elapsed = time.time() - start
768
- # Log error to CSV for tracking
769
- append_progress(local_path, repo_path, 'failed', str(e), elapsed)
770
- # Update progress bar with error
771
- file_pbar.update_file_error(1, f"{repo_path}: {str(e)[:60]}")
772
- raise
773
- else:
774
- elapsed = time.time() - start
775
- # Only log successful uploads to CSV (optional, can be verbose)
776
- # append_progress(local_path, repo_path, 'uploaded', None, elapsed)
777
- # Update progress bar
778
- file_pbar.update_file(1)
779
- return repo_path, local_path
780
-
781
- try:
782
- with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as ex:
783
- futures = [ex.submit(upload_one, f) for f in files]
784
- for fut in concurrent.futures.as_completed(futures):
785
- # Respect cancellation
786
- if stop_event is not None and getattr(stop_event, 'is_set', lambda: False)():
787
- logger("[Control] Cancellation requested; cancelling pending uploads...")
788
- # Attempt to cancel pending futures
789
- for f in futures:
790
- try:
791
- f.cancel()
792
- except Exception:
793
- pass
794
- try:
795
- ex.shutdown(wait=False, cancel_futures=True)
796
- except Exception:
797
- pass
798
- raise Exception("Upload cancelled by user")
799
-
800
- try:
801
- repo_path, local_path = fut.result()
802
- except Exception as e:
803
- # Error already logged above, just continue
804
- continue
805
-
806
- if repo_path:
807
- already_uploaded.add(repo_path)
808
- try:
809
- checkpoint_hashes[repo_path] = compute_sha256(local_path)
810
- except Exception as e:
811
- logger(f"Warning: failed to hash {repo_path} after upload: {e}")
812
- run_stats['uploaded'] += 1
813
- try:
814
- run_stats['bytes_uploaded'] += Path(local_path).stat().st_size
815
- except Exception:
816
- pass
817
- save_checkpoint(already_uploaded, checkpoint_hashes)
818
- finally:
819
- file_pbar.close()
820
- file_pbar.log_errors()
821
-
822
- retry_call("upload_segments", upload_attempts, retry_backoff, upload_segments, logger, stop_event)
823
-
824
- # Upload CSV if not already uploaded with segments
825
- if csv_path and not csv_uploaded_with_segments:
826
- def upload_csv() -> None:
827
- skip_csv = False
828
- if resume:
829
- ensure_existing_files_loaded()
830
- if resume and csv_path_in_repo in already_uploaded:
831
- if verify_remote and requests is not None:
832
- local_size = csv_path.stat().st_size
833
- if rate_limiter:
834
- rate_limiter.acquire("verify_remote")
835
- remote_size, remote_etag = remote_file_info(csv_path_in_repo)
836
- if verify_etag and remote_etag:
837
- if remote_size is not None and local_size == remote_size:
838
- skip_csv = True
839
- else:
840
- if remote_size is not None and local_size == remote_size:
841
- skip_csv = True
842
-
843
- if skip_csv:
844
- logger(f"CSV already exists at {csv_path_in_repo}; skipping upload.")
845
- run_stats['skipped'] += 1
846
- append_progress(csv_path, csv_path_in_repo, 'skipped', 'already uploaded')
847
- return
848
-
849
- start = time.time()
850
- try:
851
- if rate_limiter:
852
- rate_limiter.acquire("csv_upload")
853
- api.upload_file(
854
- path_or_fileobj=str(csv_path),
855
- path_in_repo=csv_path_in_repo,
856
- repo_id=repo_id,
857
- repo_type="dataset",
858
- commit_message=f"Upload CSV for {repo_id}",
859
- )
860
- except Exception as e:
861
- elapsed = time.time() - start
862
- run_stats['failed'] += 1
863
- append_progress(csv_path, csv_path_in_repo, 'failed', str(e), elapsed)
864
- raise
865
- else:
866
- elapsed = time.time() - start
867
- run_stats['uploaded'] += 1
868
- try:
869
- run_stats['bytes_uploaded'] += csv_path.stat().st_size
870
- except Exception:
871
- pass
872
- try:
873
- checkpoint_hashes[csv_path_in_repo] = compute_sha256(csv_path)
874
- save_checkpoint(already_uploaded.union({csv_path_in_repo}), checkpoint_hashes)
875
- except Exception as e:
876
- logger(f"Warning: failed to update CSV checkpoint hash: {e}")
877
- append_progress(csv_path, csv_path_in_repo, 'uploaded', None, elapsed)
878
-
879
- retry_call("upload_csv", upload_attempts, retry_backoff, upload_csv, logger, stop_event)
880
-
881
- logger("")
882
- logger("Upload complete.")
883
- logger(f"Dataset URL: https://huggingface.co/datasets/{repo_id}")
884
- record_summary()
885
-
886
- return 0