dataset-utils / README.md
giangndm's picture
Server-side job queue as the backbone + Jobs tab
ebc294a verified
|
Raw History Blame Contribute Delete
8.52 kB
---
title: Dataset Utils
emoji: 🧰
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 6.28.0
app_file: app.py
pinned: false
license: mit
hf_oauth: true
hf_oauth_expiration_minutes: 480
hf_oauth_scopes:
- email
- read-repos
- write-repos
- manage-repos
---
# 🧰 Dataset Utils
A Gradio Space with a set of utilities for datasets on the Hugging Face Hub β€” **one tab per feature**.
| Tab | What it does |
| --- | --- |
| 🧲 **Text Column Extract** | Extract one text column from a dataset into a split of another dataset. |
| πŸ—œοΈ **Audio Compress** | Re-encode the audio column(s) of a subset / split to FLAC / Opus / MP3, shard by shard, into another repo. |
| πŸ“‹ **Jobs** | Status, progress, logs and cancellation of the tasks you submitted. |
All tabs share the sign-in section: **Sign in with Hugging Face** (Spaces OAuth) or paste an
Access Token with *Write* permission.
## Architecture: a server-side job queue
The **job queue is the backbone** of the app. Feature tabs never do long work inside a
Gradio event: clicking *Submit job* validates the sign-in, puts the task on an in-process
queue and returns immediately. Worker threads run the jobs **independently of the browser**
β€” closing the page, losing the network or a laptop going to sleep does not stop them. The
πŸ“‹ **Jobs** tab refreshes every 2 s to show each job's status, progress, log (and preview),
and can cancel a job (it stops after the current step, e.g. the current shard).
- Jobs run one at a time by default (`DATASET_UTILS_WORKERS` env var to change it).
- Each user only sees and cancels their own jobs. The token is held in memory for the job's
lifetime only, and dropped when it ends.
- Jobs live in memory: a Space restart clears the list (data already pushed to the Hub is
kept; Audio Compress resumes where it stopped when submitted again).
- An OAuth token expires after `hf_oauth_expiration_minutes` (8 h here): for jobs that may
run longer, sign in with a pasted Access Token instead.
## Project layout
```
app.py # routing: shared sign-in + one gr.Tab per feature + the Jobs tab
core/ # backbone, shared by every feature
common.py # token / sign-in helpers, shared UI constants, AppContext
jobs.py # job queue: Job, JobManager, submit_job
jobs_tab.py # πŸ“‹ Jobs tab
features/ # extensions β€” one module per tab
__init__.py # FEATURES registry + the feature contract
text_extract.py # 🧲 Text Column Extract
audio_compress.py # πŸ—œοΈ Audio Compress
```
To add a feature, create `features/<name>.py` exposing `TITLE` and `build(ctx)`, write its
long-running work as a job generator `fn(token, **params, progress)` yielding
`(status_md, log_text[, preview])`, submit it with `core.jobs.submit_job(...)`, and append the
module to `FEATURES` (see the docstring of `features/__init__.py`).
---
## 🧲 Text Column Extract
Extract **one text column** from a dataset on the Hub and write it as **one split** of
another dataset, using [`datasets`](https://huggingface.co/docs/datasets).
- Source: `repo`, `subset` (config), `split`, `column_name`.
- Destination: `repo` (free-form β€” any namespace you can write to), `config`, `split`.
- **✨ Generate split name** derives a default split name from the source `repo + subset`
(optionally plus the source split), sanitized for the `datasets` split-name rules.
- **πŸ”Ž Inspect source dataset** lists subsets, splits, columns and row counts without
downloading any data.
- Writes **Parquet** to the Hub, with 3 modes when the split already exists:
**Overwrite**, **Append** (checks the columns match), or **Fail**.
- Advanced: `revision`, rename the output column, row limit, `num_proc`, `streaming` for very
large datasets, `max_shard_size`, commit message, private repo.
| Output config | File path |
| --- | --- |
| `default` | `data/<split>-00000-of-00001.parquet` |
| anything else | `<config>/<split>-00000-of-00001.parquet` |
> ⚠️ Every split of the **same config** must have the **same columns** (a `datasets`
> constraint). If you rename the column, write it to a **different Output config**.
---
## πŸ—œοΈ Audio Compress
Re-encode audio with [`soundfile`](https://github.com/bastibe/python-soundfile) (libsndfile)
and resample with [`soxr`](https://github.com/dofuuz/python-soxr).
1. Enter the source repo and click **πŸ“‚ Load** β†’ pick the **subset**, **split** and **audio
column(s)** from the dropdowns (shard count, size and rows are shown per split).
2. Choose the **format** (FLAC / Opus / MP3), **compression_level**, **output sample rate**
(or keep the original), optional **mono**, MP3 bitrate mode / FLAC bit depth. A live
estimate (kbps, MB per hour) and a measured bitrate reference table are shown.
3. Enter the destination repo (**✨ Suggest name** β†’ `<you>/<source-name>-<format>`).
4. **πŸš€ Submit job** β€” the subset and split keep **the same names** as the source; follow it in πŸ“‹ Jobs.
How it works:
- **One parquet shard at a time**: download the source shard β†’ decode / (mono) / resample /
re-encode every audio value (multi-threaded) β†’ write a new parquet with the same row groups β†’
upload β†’ delete temp files. Memory and disk only need to hold one shard.
- One destination shard per source shard: `<data_dir>/<split>-0000i-of-0000N.parquet`
(`data_dir` = `data` for the `default` subset, otherwise the subset name).
- Non-audio columns are copied as is; the `Audio` feature metadata gets the new `sampling_rate`.
- **Resume**: shards already at the destination are skipped, so re-running after an
interruption continues where it stopped. *Delete the destination split* starts over.
- **Dataset card**: if the destination has no README, one is created that keeps the
original's `license` (plus language / task categories), states clearly that it is a
**compressed version** linking back to the original repo, and keeps a per-split table of the
settings used (format, level, sample rate, rows, hours, size before β†’ after). The `configs`
YAML is updated so `load_dataset(repo, subset, split=...)` works.
- Non-parquet sources fall back to the Hub's auto-converted parquet (`refs/convert/parquet`).
### Expected bitrates
Measured on mono English speech (LibriSpeech) with soundfile 0.14 / libsndfile 1.2.2:
| Format | compression_level β†’ kbps | Notes |
| --- | --- | --- |
| Opus | 0 β†’ 256, 0.5 β†’ 133, 0.75 β†’ 70, 0.9 β†’ 31, 1 β†’ 7 (per channel) | Sample rate must be 8k / 12k / 16k / 24k / 48k |
| MP3 VBR @ 16k | 0 β†’ 64, 0.5 β†’ 39, 0.9 β†’ 29, 0.95 β†’ 25 | level is capped at 0.95 |
| MP3 CBR @ 48k | 0 β†’ 320, 0.5 β†’ 160, 0.9 β†’ 56, 0.95 β†’ 48 | ≀ 24 kHz caps at 160 kbps |
| FLAC | ~60% of 16-bit PCM for speech | lossless; level mostly trades speed |
Decoding the compressed audio with `datasets` β‰₯ 4 requires `torchcodec` on the reader's side
(as for any audio dataset).
---
## Technical notes
- The token is kept in each browser session's `gr.State` (and in memory while its job runs),
never written to the Space's files.
- OAuth requests the `read-repos`, `write-repos`, `manage-repos` scopes to create repos and push
data on behalf of the user.
- If the Space has an `HF_TOKEN` secret, the app also works without signing in.
- Run locally: `pip install -r requirements.txt gradio && python app.py`
(needs `HF_TOKEN` in the environment so Gradio can mock OAuth outside Spaces).
## Deployment
This Space uses the **`gradio` SDK** on **ZeroGPU** hardware β€” the free tier for personal
accounts. The app is CPU-only, but ZeroGPU **requires** at least one `@spaces.GPU` function at
startup (otherwise *"No @spaces.GPU function detected during startup"*), so `app.py` defines a
`_zerogpu_heartbeat` function that is never called from the UI and is a no-op outside ZeroGPU.
Deploy by uploading the code straight to the Space. Use `upload_folder` rather than
`hf upload`: the CLI first calls `create_repo`, which the Hub rejects with *402 Payment
Required* for Gradio Spaces without PRO, even when the Space already exists.
```python
from huggingface_hub import HfApi
HfApi().upload_folder(
repo_id="giangndm/dataset-utils",
repo_type="space",
folder_path=".",
allow_patterns=["app.py", "README.md", "requirements.txt", "core/*.py", "features/*.py"],
delete_patterns=["core/*.py", "features/*.py"], # drop modules removed locally
)
```