--- title: Dataset Utils emoji: 🧰 colorFrom: indigo colorTo: purple sdk: gradio sdk_version: 6.28.0 app_file: app.py pinned: false license: mit hf_oauth: true hf_oauth_expiration_minutes: 480 hf_oauth_scopes: - email - read-repos - write-repos - manage-repos --- # 🧰 Dataset Utils A Gradio Space with a set of utilities for datasets on the Hugging Face Hub — **one tab per feature**. | Tab | What it does | | --- | --- | | 🧲 **Text Column Extract** | Extract one text column from a dataset into a split of another dataset. | | 🗜️ **Audio Compress** | Re-encode the audio column(s) of a subset / split to FLAC / Opus / MP3, shard by shard, into another repo. | | 📋 **Jobs** | Status, progress, logs and cancellation of the tasks you submitted. | All tabs share the sign-in section: **Sign in with Hugging Face** (Spaces OAuth) or paste an Access Token with *Write* permission. ## Architecture: a server-side job queue The **job queue is the backbone** of the app. Feature tabs never do long work inside a Gradio event: clicking *Submit job* validates the sign-in, puts the task on an in-process queue and returns immediately. Worker threads run the jobs **independently of the browser** — closing the page, losing the network or a laptop going to sleep does not stop them. The 📋 **Jobs** tab refreshes every 2 s to show each job's status, progress, log (and preview), and can cancel a job (it stops after the current step, e.g. the current shard). - Jobs run one at a time by default (`DATASET_UTILS_WORKERS` env var to change it). - Each user only sees and cancels their own jobs. The token is held in memory for the job's lifetime only, and dropped when it ends. - Jobs live in memory: a Space restart clears the list (data already pushed to the Hub is kept; Audio Compress resumes where it stopped when submitted again). - An OAuth token expires after `hf_oauth_expiration_minutes` (8 h here): for jobs that may run longer, sign in with a pasted Access Token instead. ## Project layout ``` app.py # routing: shared sign-in + one gr.Tab per feature + the Jobs tab core/ # backbone, shared by every feature common.py # token / sign-in helpers, shared UI constants, AppContext jobs.py # job queue: Job, JobManager, submit_job jobs_tab.py # 📋 Jobs tab features/ # extensions — one module per tab __init__.py # FEATURES registry + the feature contract text_extract.py # 🧲 Text Column Extract audio_compress.py # 🗜️ Audio Compress ``` To add a feature, create `features/.py` exposing `TITLE` and `build(ctx)`, write its long-running work as a job generator `fn(token, **params, progress)` yielding `(status_md, log_text[, preview])`, submit it with `core.jobs.submit_job(...)`, and append the module to `FEATURES` (see the docstring of `features/__init__.py`). --- ## 🧲 Text Column Extract Extract **one text column** from a dataset on the Hub and write it as **one split** of another dataset, using [`datasets`](https://huggingface.co/docs/datasets). - Source: `repo`, `subset` (config), `split`, `column_name`. - Destination: `repo` (free-form — any namespace you can write to), `config`, `split`. - **✨ Generate split name** derives a default split name from the source `repo + subset` (optionally plus the source split), sanitized for the `datasets` split-name rules. - **🔎 Inspect source dataset** lists subsets, splits, columns and row counts without downloading any data. - Writes **Parquet** to the Hub, with 3 modes when the split already exists: **Overwrite**, **Append** (checks the columns match), or **Fail**. - Advanced: `revision`, rename the output column, row limit, `num_proc`, `streaming` for very large datasets, `max_shard_size`, commit message, private repo. | Output config | File path | | --- | --- | | `default` | `data/-00000-of-00001.parquet` | | anything else | `/-00000-of-00001.parquet` | > ⚠️ Every split of the **same config** must have the **same columns** (a `datasets` > constraint). If you rename the column, write it to a **different Output config**. --- ## 🗜️ Audio Compress Re-encode audio with [`soundfile`](https://github.com/bastibe/python-soundfile) (libsndfile) and resample with [`soxr`](https://github.com/dofuuz/python-soxr). 1. Enter the source repo and click **📂 Load** → pick the **subset**, **split** and **audio column(s)** from the dropdowns (shard count, size and rows are shown per split). 2. Choose the **format** (FLAC / Opus / MP3), **compression_level**, **output sample rate** (or keep the original), optional **mono**, MP3 bitrate mode / FLAC bit depth. A live estimate (kbps, MB per hour) and a measured bitrate reference table are shown. 3. Enter the destination repo (**✨ Suggest name** → `/-`). 4. **🚀 Submit job** — the subset and split keep **the same names** as the source; follow it in 📋 Jobs. How it works: - **One parquet shard at a time**: download the source shard → decode / (mono) / resample / re-encode every audio value (multi-threaded) → write a new parquet with the same row groups → upload → delete temp files. Memory and disk only need to hold one shard. - One destination shard per source shard: `/-0000i-of-0000N.parquet` (`data_dir` = `data` for the `default` subset, otherwise the subset name). - Non-audio columns are copied as is; the `Audio` feature metadata gets the new `sampling_rate`. - **Resume**: shards already at the destination are skipped, so re-running after an interruption continues where it stopped. *Delete the destination split* starts over. - **Dataset card**: if the destination has no README, one is created that keeps the original's `license` (plus language / task categories), states clearly that it is a **compressed version** linking back to the original repo, and keeps a per-split table of the settings used (format, level, sample rate, rows, hours, size before → after). The `configs` YAML is updated so `load_dataset(repo, subset, split=...)` works. - Non-parquet sources fall back to the Hub's auto-converted parquet (`refs/convert/parquet`). ### Expected bitrates Measured on mono English speech (LibriSpeech) with soundfile 0.14 / libsndfile 1.2.2: | Format | compression_level → kbps | Notes | | --- | --- | --- | | Opus | 0 → 256, 0.5 → 133, 0.75 → 70, 0.9 → 31, 1 → 7 (per channel) | Sample rate must be 8k / 12k / 16k / 24k / 48k | | MP3 VBR @ 16k | 0 → 64, 0.5 → 39, 0.9 → 29, 0.95 → 25 | level is capped at 0.95 | | MP3 CBR @ 48k | 0 → 320, 0.5 → 160, 0.9 → 56, 0.95 → 48 | ≤ 24 kHz caps at 160 kbps | | FLAC | ~60% of 16-bit PCM for speech | lossless; level mostly trades speed | Decoding the compressed audio with `datasets` ≥ 4 requires `torchcodec` on the reader's side (as for any audio dataset). --- ## Technical notes - The token is kept in each browser session's `gr.State` (and in memory while its job runs), never written to the Space's files. - OAuth requests the `read-repos`, `write-repos`, `manage-repos` scopes to create repos and push data on behalf of the user. - If the Space has an `HF_TOKEN` secret, the app also works without signing in. - Run locally: `pip install -r requirements.txt gradio && python app.py` (needs `HF_TOKEN` in the environment so Gradio can mock OAuth outside Spaces). ## Deployment This Space uses the **`gradio` SDK** on **ZeroGPU** hardware — the free tier for personal accounts. The app is CPU-only, but ZeroGPU **requires** at least one `@spaces.GPU` function at startup (otherwise *"No @spaces.GPU function detected during startup"*), so `app.py` defines a `_zerogpu_heartbeat` function that is never called from the UI and is a no-op outside ZeroGPU. Deploy by uploading the code straight to the Space. Use `upload_folder` rather than `hf upload`: the CLI first calls `create_repo`, which the Hub rejects with *402 Payment Required* for Gradio Spaces without PRO, even when the Space already exists. ```python from huggingface_hub import HfApi HfApi().upload_folder( repo_id="giangndm/dataset-utils", repo_type="space", folder_path=".", allow_patterns=["app.py", "README.md", "requirements.txt", "core/*.py", "features/*.py"], delete_patterns=["core/*.py", "features/*.py"], # drop modules removed locally ) ```