dataset-utils / README.md
giangndm's picture
Server-side job queue as the backbone + Jobs tab
ebc294a verified
|
Raw History Blame Contribute Delete
8.52 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade
metadata
title: Dataset Utils
emoji: 🧰
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 6.28.0
app_file: app.py
pinned: false
license: mit
hf_oauth: true
hf_oauth_expiration_minutes: 480
hf_oauth_scopes:
  - email
  - read-repos
  - write-repos
  - manage-repos

🧰 Dataset Utils

A Gradio Space with a set of utilities for datasets on the Hugging Face Hub β€” one tab per feature.

Tab What it does
🧲 Text Column Extract Extract one text column from a dataset into a split of another dataset.
πŸ—œοΈ Audio Compress Re-encode the audio column(s) of a subset / split to FLAC / Opus / MP3, shard by shard, into another repo.
πŸ“‹ Jobs Status, progress, logs and cancellation of the tasks you submitted.

All tabs share the sign-in section: Sign in with Hugging Face (Spaces OAuth) or paste an Access Token with Write permission.

Architecture: a server-side job queue

The job queue is the backbone of the app. Feature tabs never do long work inside a Gradio event: clicking Submit job validates the sign-in, puts the task on an in-process queue and returns immediately. Worker threads run the jobs independently of the browser β€” closing the page, losing the network or a laptop going to sleep does not stop them. The πŸ“‹ Jobs tab refreshes every 2 s to show each job's status, progress, log (and preview), and can cancel a job (it stops after the current step, e.g. the current shard).

  • Jobs run one at a time by default (DATASET_UTILS_WORKERS env var to change it).
  • Each user only sees and cancels their own jobs. The token is held in memory for the job's lifetime only, and dropped when it ends.
  • Jobs live in memory: a Space restart clears the list (data already pushed to the Hub is kept; Audio Compress resumes where it stopped when submitted again).
  • An OAuth token expires after hf_oauth_expiration_minutes (8 h here): for jobs that may run longer, sign in with a pasted Access Token instead.

Project layout

app.py                     # routing: shared sign-in + one gr.Tab per feature + the Jobs tab
core/                      # backbone, shared by every feature
  common.py                # token / sign-in helpers, shared UI constants, AppContext
  jobs.py                  # job queue: Job, JobManager, submit_job
  jobs_tab.py              # πŸ“‹ Jobs tab
features/                  # extensions β€” one module per tab
  __init__.py              # FEATURES registry + the feature contract
  text_extract.py          # 🧲 Text Column Extract
  audio_compress.py        # πŸ—œοΈ Audio Compress

To add a feature, create features/<name>.py exposing TITLE and build(ctx), write its long-running work as a job generator fn(token, **params, progress) yielding (status_md, log_text[, preview]), submit it with core.jobs.submit_job(...), and append the module to FEATURES (see the docstring of features/__init__.py).


🧲 Text Column Extract

Extract one text column from a dataset on the Hub and write it as one split of another dataset, using datasets.

  • Source: repo, subset (config), split, column_name.
  • Destination: repo (free-form β€” any namespace you can write to), config, split.
  • ✨ Generate split name derives a default split name from the source repo + subset (optionally plus the source split), sanitized for the datasets split-name rules.
  • πŸ”Ž Inspect source dataset lists subsets, splits, columns and row counts without downloading any data.
  • Writes Parquet to the Hub, with 3 modes when the split already exists: Overwrite, Append (checks the columns match), or Fail.
  • Advanced: revision, rename the output column, row limit, num_proc, streaming for very large datasets, max_shard_size, commit message, private repo.
Output config File path
default data/<split>-00000-of-00001.parquet
anything else <config>/<split>-00000-of-00001.parquet

⚠️ Every split of the same config must have the same columns (a datasets constraint). If you rename the column, write it to a different Output config.


πŸ—œοΈ Audio Compress

Re-encode audio with soundfile (libsndfile) and resample with soxr.

  1. Enter the source repo and click πŸ“‚ Load β†’ pick the subset, split and audio column(s) from the dropdowns (shard count, size and rows are shown per split).
  2. Choose the format (FLAC / Opus / MP3), compression_level, output sample rate (or keep the original), optional mono, MP3 bitrate mode / FLAC bit depth. A live estimate (kbps, MB per hour) and a measured bitrate reference table are shown.
  3. Enter the destination repo (✨ Suggest name β†’ <you>/<source-name>-<format>).
  4. πŸš€ Submit job β€” the subset and split keep the same names as the source; follow it in πŸ“‹ Jobs.

How it works:

  • One parquet shard at a time: download the source shard β†’ decode / (mono) / resample / re-encode every audio value (multi-threaded) β†’ write a new parquet with the same row groups β†’ upload β†’ delete temp files. Memory and disk only need to hold one shard.
  • One destination shard per source shard: <data_dir>/<split>-0000i-of-0000N.parquet (data_dir = data for the default subset, otherwise the subset name).
  • Non-audio columns are copied as is; the Audio feature metadata gets the new sampling_rate.
  • Resume: shards already at the destination are skipped, so re-running after an interruption continues where it stopped. Delete the destination split starts over.
  • Dataset card: if the destination has no README, one is created that keeps the original's license (plus language / task categories), states clearly that it is a compressed version linking back to the original repo, and keeps a per-split table of the settings used (format, level, sample rate, rows, hours, size before β†’ after). The configs YAML is updated so load_dataset(repo, subset, split=...) works.
  • Non-parquet sources fall back to the Hub's auto-converted parquet (refs/convert/parquet).

Expected bitrates

Measured on mono English speech (LibriSpeech) with soundfile 0.14 / libsndfile 1.2.2:

Format compression_level β†’ kbps Notes
Opus 0 β†’ 256, 0.5 β†’ 133, 0.75 β†’ 70, 0.9 β†’ 31, 1 β†’ 7 (per channel) Sample rate must be 8k / 12k / 16k / 24k / 48k
MP3 VBR @ 16k 0 β†’ 64, 0.5 β†’ 39, 0.9 β†’ 29, 0.95 β†’ 25 level is capped at 0.95
MP3 CBR @ 48k 0 β†’ 320, 0.5 β†’ 160, 0.9 β†’ 56, 0.95 β†’ 48 ≀ 24 kHz caps at 160 kbps
FLAC ~60% of 16-bit PCM for speech lossless; level mostly trades speed

Decoding the compressed audio with datasets β‰₯ 4 requires torchcodec on the reader's side (as for any audio dataset).


Technical notes

  • The token is kept in each browser session's gr.State (and in memory while its job runs), never written to the Space's files.
  • OAuth requests the read-repos, write-repos, manage-repos scopes to create repos and push data on behalf of the user.
  • If the Space has an HF_TOKEN secret, the app also works without signing in.
  • Run locally: pip install -r requirements.txt gradio && python app.py (needs HF_TOKEN in the environment so Gradio can mock OAuth outside Spaces).

Deployment

This Space uses the gradio SDK on ZeroGPU hardware β€” the free tier for personal accounts. The app is CPU-only, but ZeroGPU requires at least one @spaces.GPU function at startup (otherwise "No @spaces.GPU function detected during startup"), so app.py defines a _zerogpu_heartbeat function that is never called from the UI and is a no-op outside ZeroGPU.

Deploy by uploading the code straight to the Space. Use upload_folder rather than hf upload: the CLI first calls create_repo, which the Hub rejects with 402 Payment Required for Gradio Spaces without PRO, even when the Space already exists.

from huggingface_hub import HfApi

HfApi().upload_folder(
    repo_id="giangndm/dataset-utils",
    repo_type="space",
    folder_path=".",
    allow_patterns=["app.py", "README.md", "requirements.txt", "core/*.py", "features/*.py"],
    delete_patterns=["core/*.py", "features/*.py"],  # drop modules removed locally
)