Spaces:
Running on Zero
Running on Zero
|
Download README.md from giangndm/dataset-utils: direct link, hf CLI and curl.
- Browser
- Download file 8.52 kB
-
https://huggingface.co/spaces/giangndm/dataset-utils/resolve/main/README.md
- Command line
-
hf download hf://spaces/giangndm/dataset-utils/README.md
-
curl -L -o README.md https://huggingface.co/spaces/giangndm/dataset-utils/resolve/main/README.md
8.52 kB
| title: Dataset Utils | |
| emoji: π§° | |
| colorFrom: indigo | |
| colorTo: purple | |
| sdk: gradio | |
| sdk_version: 6.28.0 | |
| app_file: app.py | |
| pinned: false | |
| license: mit | |
| hf_oauth: true | |
| hf_oauth_expiration_minutes: 480 | |
| hf_oauth_scopes: | |
| - read-repos | |
| - write-repos | |
| - manage-repos | |
| # π§° Dataset Utils | |
| A Gradio Space with a set of utilities for datasets on the Hugging Face Hub β **one tab per feature**. | |
| | Tab | What it does | | |
| | --- | --- | | |
| | π§² **Text Column Extract** | Extract one text column from a dataset into a split of another dataset. | | |
| | ποΈ **Audio Compress** | Re-encode the audio column(s) of a subset / split to FLAC / Opus / MP3, shard by shard, into another repo. | | |
| | π **Jobs** | Status, progress, logs and cancellation of the tasks you submitted. | | |
| All tabs share the sign-in section: **Sign in with Hugging Face** (Spaces OAuth) or paste an | |
| Access Token with *Write* permission. | |
| ## Architecture: a server-side job queue | |
| The **job queue is the backbone** of the app. Feature tabs never do long work inside a | |
| Gradio event: clicking *Submit job* validates the sign-in, puts the task on an in-process | |
| queue and returns immediately. Worker threads run the jobs **independently of the browser** | |
| β closing the page, losing the network or a laptop going to sleep does not stop them. The | |
| π **Jobs** tab refreshes every 2 s to show each job's status, progress, log (and preview), | |
| and can cancel a job (it stops after the current step, e.g. the current shard). | |
| - Jobs run one at a time by default (`DATASET_UTILS_WORKERS` env var to change it). | |
| - Each user only sees and cancels their own jobs. The token is held in memory for the job's | |
| lifetime only, and dropped when it ends. | |
| - Jobs live in memory: a Space restart clears the list (data already pushed to the Hub is | |
| kept; Audio Compress resumes where it stopped when submitted again). | |
| - An OAuth token expires after `hf_oauth_expiration_minutes` (8 h here): for jobs that may | |
| run longer, sign in with a pasted Access Token instead. | |
| ## Project layout | |
| ``` | |
| app.py # routing: shared sign-in + one gr.Tab per feature + the Jobs tab | |
| core/ # backbone, shared by every feature | |
| common.py # token / sign-in helpers, shared UI constants, AppContext | |
| jobs.py # job queue: Job, JobManager, submit_job | |
| jobs_tab.py # π Jobs tab | |
| features/ # extensions β one module per tab | |
| __init__.py # FEATURES registry + the feature contract | |
| text_extract.py # π§² Text Column Extract | |
| audio_compress.py # ποΈ Audio Compress | |
| ``` | |
| To add a feature, create `features/<name>.py` exposing `TITLE` and `build(ctx)`, write its | |
| long-running work as a job generator `fn(token, **params, progress)` yielding | |
| `(status_md, log_text[, preview])`, submit it with `core.jobs.submit_job(...)`, and append the | |
| module to `FEATURES` (see the docstring of `features/__init__.py`). | |
| --- | |
| ## π§² Text Column Extract | |
| Extract **one text column** from a dataset on the Hub and write it as **one split** of | |
| another dataset, using [`datasets`](https://huggingface.co/docs/datasets). | |
| - Source: `repo`, `subset` (config), `split`, `column_name`. | |
| - Destination: `repo` (free-form β any namespace you can write to), `config`, `split`. | |
| - **β¨ Generate split name** derives a default split name from the source `repo + subset` | |
| (optionally plus the source split), sanitized for the `datasets` split-name rules. | |
| - **π Inspect source dataset** lists subsets, splits, columns and row counts without | |
| downloading any data. | |
| - Writes **Parquet** to the Hub, with 3 modes when the split already exists: | |
| **Overwrite**, **Append** (checks the columns match), or **Fail**. | |
| - Advanced: `revision`, rename the output column, row limit, `num_proc`, `streaming` for very | |
| large datasets, `max_shard_size`, commit message, private repo. | |
| | Output config | File path | | |
| | --- | --- | | |
| | `default` | `data/<split>-00000-of-00001.parquet` | | |
| | anything else | `<config>/<split>-00000-of-00001.parquet` | | |
| > β οΈ Every split of the **same config** must have the **same columns** (a `datasets` | |
| > constraint). If you rename the column, write it to a **different Output config**. | |
| --- | |
| ## ποΈ Audio Compress | |
| Re-encode audio with [`soundfile`](https://github.com/bastibe/python-soundfile) (libsndfile) | |
| and resample with [`soxr`](https://github.com/dofuuz/python-soxr). | |
| 1. Enter the source repo and click **π Load** β pick the **subset**, **split** and **audio | |
| column(s)** from the dropdowns (shard count, size and rows are shown per split). | |
| 2. Choose the **format** (FLAC / Opus / MP3), **compression_level**, **output sample rate** | |
| (or keep the original), optional **mono**, MP3 bitrate mode / FLAC bit depth. A live | |
| estimate (kbps, MB per hour) and a measured bitrate reference table are shown. | |
| 3. Enter the destination repo (**β¨ Suggest name** β `<you>/<source-name>-<format>`). | |
| 4. **π Submit job** β the subset and split keep **the same names** as the source; follow it in π Jobs. | |
| How it works: | |
| - **One parquet shard at a time**: download the source shard β decode / (mono) / resample / | |
| re-encode every audio value (multi-threaded) β write a new parquet with the same row groups β | |
| upload β delete temp files. Memory and disk only need to hold one shard. | |
| - One destination shard per source shard: `<data_dir>/<split>-0000i-of-0000N.parquet` | |
| (`data_dir` = `data` for the `default` subset, otherwise the subset name). | |
| - Non-audio columns are copied as is; the `Audio` feature metadata gets the new `sampling_rate`. | |
| - **Resume**: shards already at the destination are skipped, so re-running after an | |
| interruption continues where it stopped. *Delete the destination split* starts over. | |
| - **Dataset card**: if the destination has no README, one is created that keeps the | |
| original's `license` (plus language / task categories), states clearly that it is a | |
| **compressed version** linking back to the original repo, and keeps a per-split table of the | |
| settings used (format, level, sample rate, rows, hours, size before β after). The `configs` | |
| YAML is updated so `load_dataset(repo, subset, split=...)` works. | |
| - Non-parquet sources fall back to the Hub's auto-converted parquet (`refs/convert/parquet`). | |
| ### Expected bitrates | |
| Measured on mono English speech (LibriSpeech) with soundfile 0.14 / libsndfile 1.2.2: | |
| | Format | compression_level β kbps | Notes | | |
| | --- | --- | --- | | |
| | Opus | 0 β 256, 0.5 β 133, 0.75 β 70, 0.9 β 31, 1 β 7 (per channel) | Sample rate must be 8k / 12k / 16k / 24k / 48k | | |
| | MP3 VBR @ 16k | 0 β 64, 0.5 β 39, 0.9 β 29, 0.95 β 25 | level is capped at 0.95 | | |
| | MP3 CBR @ 48k | 0 β 320, 0.5 β 160, 0.9 β 56, 0.95 β 48 | β€ 24 kHz caps at 160 kbps | | |
| | FLAC | ~60% of 16-bit PCM for speech | lossless; level mostly trades speed | | |
| Decoding the compressed audio with `datasets` β₯ 4 requires `torchcodec` on the reader's side | |
| (as for any audio dataset). | |
| --- | |
| ## Technical notes | |
| - The token is kept in each browser session's `gr.State` (and in memory while its job runs), | |
| never written to the Space's files. | |
| - OAuth requests the `read-repos`, `write-repos`, `manage-repos` scopes to create repos and push | |
| data on behalf of the user. | |
| - If the Space has an `HF_TOKEN` secret, the app also works without signing in. | |
| - Run locally: `pip install -r requirements.txt gradio && python app.py` | |
| (needs `HF_TOKEN` in the environment so Gradio can mock OAuth outside Spaces). | |
| ## Deployment | |
| This Space uses the **`gradio` SDK** on **ZeroGPU** hardware β the free tier for personal | |
| accounts. The app is CPU-only, but ZeroGPU **requires** at least one `@spaces.GPU` function at | |
| startup (otherwise *"No @spaces.GPU function detected during startup"*), so `app.py` defines a | |
| `_zerogpu_heartbeat` function that is never called from the UI and is a no-op outside ZeroGPU. | |
| Deploy by uploading the code straight to the Space. Use `upload_folder` rather than | |
| `hf upload`: the CLI first calls `create_repo`, which the Hub rejects with *402 Payment | |
| Required* for Gradio Spaces without PRO, even when the Space already exists. | |
| ```python | |
| from huggingface_hub import HfApi | |
| HfApi().upload_folder( | |
| repo_id="giangndm/dataset-utils", | |
| repo_type="space", | |
| folder_path=".", | |
| allow_patterns=["app.py", "README.md", "requirements.txt", "core/*.py", "features/*.py"], | |
| delete_patterns=["core/*.py", "features/*.py"], # drop modules removed locally | |
| ) | |
| ``` | |