Spaces:
Running on Zero
Download README.md from giangndm/dataset-utils: direct link, hf CLI and curl.
- Browser
- Download file 8.52 kB
-
https://huggingface.co/spaces/giangndm/dataset-utils/resolve/main/README.md
- Command line
-
hf download hf://spaces/giangndm/dataset-utils/README.md
-
curl -L -o README.md https://huggingface.co/spaces/giangndm/dataset-utils/resolve/main/README.md
A newer version of the Gradio SDK is available: 6.32.0
title: Dataset Utils
emoji: π§°
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 6.28.0
app_file: app.py
pinned: false
license: mit
hf_oauth: true
hf_oauth_expiration_minutes: 480
hf_oauth_scopes:
- email
- read-repos
- write-repos
- manage-repos
π§° Dataset Utils
A Gradio Space with a set of utilities for datasets on the Hugging Face Hub β one tab per feature.
| Tab | What it does |
|---|---|
| π§² Text Column Extract | Extract one text column from a dataset into a split of another dataset. |
| ποΈ Audio Compress | Re-encode the audio column(s) of a subset / split to FLAC / Opus / MP3, shard by shard, into another repo. |
| π Jobs | Status, progress, logs and cancellation of the tasks you submitted. |
All tabs share the sign-in section: Sign in with Hugging Face (Spaces OAuth) or paste an Access Token with Write permission.
Architecture: a server-side job queue
The job queue is the backbone of the app. Feature tabs never do long work inside a Gradio event: clicking Submit job validates the sign-in, puts the task on an in-process queue and returns immediately. Worker threads run the jobs independently of the browser β closing the page, losing the network or a laptop going to sleep does not stop them. The π Jobs tab refreshes every 2 s to show each job's status, progress, log (and preview), and can cancel a job (it stops after the current step, e.g. the current shard).
- Jobs run one at a time by default (
DATASET_UTILS_WORKERSenv var to change it). - Each user only sees and cancels their own jobs. The token is held in memory for the job's lifetime only, and dropped when it ends.
- Jobs live in memory: a Space restart clears the list (data already pushed to the Hub is kept; Audio Compress resumes where it stopped when submitted again).
- An OAuth token expires after
hf_oauth_expiration_minutes(8 h here): for jobs that may run longer, sign in with a pasted Access Token instead.
Project layout
app.py # routing: shared sign-in + one gr.Tab per feature + the Jobs tab
core/ # backbone, shared by every feature
common.py # token / sign-in helpers, shared UI constants, AppContext
jobs.py # job queue: Job, JobManager, submit_job
jobs_tab.py # π Jobs tab
features/ # extensions β one module per tab
__init__.py # FEATURES registry + the feature contract
text_extract.py # π§² Text Column Extract
audio_compress.py # ποΈ Audio Compress
To add a feature, create features/<name>.py exposing TITLE and build(ctx), write its
long-running work as a job generator fn(token, **params, progress) yielding
(status_md, log_text[, preview]), submit it with core.jobs.submit_job(...), and append the
module to FEATURES (see the docstring of features/__init__.py).
π§² Text Column Extract
Extract one text column from a dataset on the Hub and write it as one split of
another dataset, using datasets.
- Source:
repo,subset(config),split,column_name. - Destination:
repo(free-form β any namespace you can write to),config,split. - β¨ Generate split name derives a default split name from the source
repo + subset(optionally plus the source split), sanitized for thedatasetssplit-name rules. - π Inspect source dataset lists subsets, splits, columns and row counts without downloading any data.
- Writes Parquet to the Hub, with 3 modes when the split already exists: Overwrite, Append (checks the columns match), or Fail.
- Advanced:
revision, rename the output column, row limit,num_proc,streamingfor very large datasets,max_shard_size, commit message, private repo.
| Output config | File path |
|---|---|
default |
data/<split>-00000-of-00001.parquet |
| anything else | <config>/<split>-00000-of-00001.parquet |
β οΈ Every split of the same config must have the same columns (a
datasetsconstraint). If you rename the column, write it to a different Output config.
ποΈ Audio Compress
Re-encode audio with soundfile (libsndfile)
and resample with soxr.
- Enter the source repo and click π Load β pick the subset, split and audio column(s) from the dropdowns (shard count, size and rows are shown per split).
- Choose the format (FLAC / Opus / MP3), compression_level, output sample rate (or keep the original), optional mono, MP3 bitrate mode / FLAC bit depth. A live estimate (kbps, MB per hour) and a measured bitrate reference table are shown.
- Enter the destination repo (β¨ Suggest name β
<you>/<source-name>-<format>). - π Submit job β the subset and split keep the same names as the source; follow it in π Jobs.
How it works:
- One parquet shard at a time: download the source shard β decode / (mono) / resample / re-encode every audio value (multi-threaded) β write a new parquet with the same row groups β upload β delete temp files. Memory and disk only need to hold one shard.
- One destination shard per source shard:
<data_dir>/<split>-0000i-of-0000N.parquet(data_dir=datafor thedefaultsubset, otherwise the subset name). - Non-audio columns are copied as is; the
Audiofeature metadata gets the newsampling_rate. - Resume: shards already at the destination are skipped, so re-running after an interruption continues where it stopped. Delete the destination split starts over.
- Dataset card: if the destination has no README, one is created that keeps the
original's
license(plus language / task categories), states clearly that it is a compressed version linking back to the original repo, and keeps a per-split table of the settings used (format, level, sample rate, rows, hours, size before β after). TheconfigsYAML is updated soload_dataset(repo, subset, split=...)works. - Non-parquet sources fall back to the Hub's auto-converted parquet (
refs/convert/parquet).
Expected bitrates
Measured on mono English speech (LibriSpeech) with soundfile 0.14 / libsndfile 1.2.2:
| Format | compression_level β kbps | Notes |
|---|---|---|
| Opus | 0 β 256, 0.5 β 133, 0.75 β 70, 0.9 β 31, 1 β 7 (per channel) | Sample rate must be 8k / 12k / 16k / 24k / 48k |
| MP3 VBR @ 16k | 0 β 64, 0.5 β 39, 0.9 β 29, 0.95 β 25 | level is capped at 0.95 |
| MP3 CBR @ 48k | 0 β 320, 0.5 β 160, 0.9 β 56, 0.95 β 48 | β€ 24 kHz caps at 160 kbps |
| FLAC | ~60% of 16-bit PCM for speech | lossless; level mostly trades speed |
Decoding the compressed audio with datasets β₯ 4 requires torchcodec on the reader's side
(as for any audio dataset).
Technical notes
- The token is kept in each browser session's
gr.State(and in memory while its job runs), never written to the Space's files. - OAuth requests the
read-repos,write-repos,manage-reposscopes to create repos and push data on behalf of the user. - If the Space has an
HF_TOKENsecret, the app also works without signing in. - Run locally:
pip install -r requirements.txt gradio && python app.py(needsHF_TOKENin the environment so Gradio can mock OAuth outside Spaces).
Deployment
This Space uses the gradio SDK on ZeroGPU hardware β the free tier for personal
accounts. The app is CPU-only, but ZeroGPU requires at least one @spaces.GPU function at
startup (otherwise "No @spaces.GPU function detected during startup"), so app.py defines a
_zerogpu_heartbeat function that is never called from the UI and is a no-op outside ZeroGPU.
Deploy by uploading the code straight to the Space. Use upload_folder rather than
hf upload: the CLI first calls create_repo, which the Hub rejects with 402 Payment
Required for Gradio Spaces without PRO, even when the Space already exists.
from huggingface_hub import HfApi
HfApi().upload_folder(
repo_id="giangndm/dataset-utils",
repo_type="space",
folder_path=".",
allow_patterns=["app.py", "README.md", "requirements.txt", "core/*.py", "features/*.py"],
delete_patterns=["core/*.py", "features/*.py"], # drop modules removed locally
)