| # VPdata_processing | |
| Extracted [VideoPainter VPData](https://huggingface.co/datasets/TencentARC/VPData) assets, | |
| plus documentation for the processing scripts that turn them into **inpainting / outpainting | |
| control-video training pairs**. | |
| A "control video" here is the original footage with part of it painted solid black — the | |
| region a generative model is asked to reconstruct. Inpainting blacks out the *inside* of an | |
| object's box; outpainting blacks out everything *outside* a visible box. | |
| --- | |
| ## Paths | |
| The data lives here; the scripts live in the sibling `VPData/processing/`. Every command | |
| below is anchored on these two roots, so it runs from any working directory: | |
| ```bash | |
| export VPDATA=/mnt/dataset/xinyuy/datasets/VPData | |
| export PROC=/mnt/dataset/xinyuy/datasets/VPdata_processing | |
| ``` | |
| --- | |
| ## What's in this folder | |
| ### Extracted from the VPData zips | |
| | Path | Contents | Source | | |
| |---|---|---| | |
| | `video_inpainting/pexels/<idx>_<videoId>/all_masks.npz` | 124,672 mask sequences | `VPData/pexels_masks/*.zip` (1,247) | | |
| | `video_inpainting/videovo/<clipId>/all_masks.npz` | 42,157 mask sequences | `VPData/videovo_masks/*.zip` (422) | | |
| Layout follows the VPData README's canonical structure, matching what the VideoPainter | |
| training code expects and what `../VPBench/` already uses. | |
| **The videovo raw videos are deliberately not extracted here.** They remain as 74 zips in | |
| `$VPDATA/videovo_raw_videos/` (475 GB, 73,297 clips). Nothing in this pipeline uses them — | |
| it operates on the Pexels subset only. To unpack them into the canonical | |
| `videovo/raw_video/<group>/<clip>.mp4` layout: | |
| ```bash | |
| mkdir -p $PROC/videovo/raw_video | |
| ls $VPDATA/videovo_raw_videos/*.zip | parallel -j 12 unzip -qq -o {} -d $PROC/videovo/raw_video | |
| ``` | |
| Note that four clips are **0 bytes upstream** — declared 0-length inside the source zips, so | |
| no re-extraction will fix them: `000005036/000005036288.0.mp4`, `000005050/000005050262.0.mp4`, | |
| `000005055/000005055471.0.mp4`, `000005065/000005065795.0.mp4`. | |
| ### Pipeline outputs (hardlinked from `VPData/`, so they cost no extra disk) | |
| | Path | Clips | Produced by | | |
| |---|---|---| | |
| | `selected_videos/` | 3,000 | `download_pexels.py` | | |
| | `inpainting/` | 3,404 | `overlay_masks.py --inpaint-mask-dir` | | |
| | `overlay/` | 3,402 | `overlay_masks.py --overlay-dir` | | |
| | `outpainting/` | 3,000 | `outpaint_masks.py` | | |
| | `test_videos/` | 202 (+ `inpainting/`, `overlay/` subdirs) | same scripts, held-out split | | |
| `inpainting` ⊃ `overlay` (3,402 of 3,404) ⊃ `outpainting` (all 3,000). Clip counts exceed | |
| `selected_videos` because the folders accumulated across runs with different inputs; the | |
| per-`videoId` dedupe applies within a single run, not across runs. | |
| --- | |
| ## Pipeline at a glance | |
| ``` | |
| pexels.csv ─────────────────► download_pexels.py ──► selected_videos/<videoId>.mp4 | |
| │ | |
| video_inpainting/pexels/*/all_masks.npz ─┐ │ | |
| pexels_videovo_train_dataset.csv ────────┤ │ | |
| ▼ ▼ | |
| ┌───────────────────────────────────────────────┐ | |
| │ overlay_masks.py (one decode pass, 2 outputs)│ | |
| │ --overlay-dir ──► overlay/ (QA) │ | |
| │ --inpaint-mask-dir ──► inpainting/ (train) │ | |
| ├───────────────────────────────────────────────┤ | |
| │ outpaint_masks.py │ | |
| │ --outpaint-dir ──► outpainting/ (train) │ | |
| └───────────────────────────────────────────────┘ | |
| │ | |
| ▼ | |
| build_inpainting_outpainting_dataset.py | |
| │ | |
| ▼ | |
| <output>/{stem}.mp4 + {stem}_painting.mp4 + dataset.json | |
| ``` | |
| Stages 2a and 2b are independent — run either or both. Stage 3 needs both folders. | |
| --- | |
| ## Requirements | |
| ```bash | |
| pip install requests av opencv-python numpy tqdm | |
| ``` | |
| `av` (PyAV) is used for encoding rather than `cv2.VideoWriter`: the bundled FFmpeg's | |
| VideoWriter backend can only open the `mp4v` fourcc (MPEG-4 Part 2), which most players | |
| and browsers refuse to play. All scripts write H.264 / yuv420p instead. | |
| --- | |
| ## Input formats | |
| ### `$VPDATA/pexels.csv` | |
| | column | meaning | | |
| |---|---| | |
| | `link` | direct Pexels CDN URL, e.g. `.../852038-hd_1920_1080_30fps.mp4` | | |
| | `videoId` | Pexels numeric id, `852038` | | |
| The resolution filter in `download_pexels.py` parses `_<W>_<H>_<fps>fps` out of `link`. | |
| ### `$VPDATA/pexels_videovo_{train,val,test}_dataset.csv` | |
| 391,416 / 1,000 / 568 rows. | |
| | column | meaning | | |
| |---|---| | |
| | `path` | `{12-digit index}_{videoId}.mp4` — also the **mask folder name** | | |
| | `start_frame`, `end_frame` | clip range, **inclusive** on both ends | | |
| | `fps` | source frame rate | | |
| | `mask_id` | which instance id in the mask array this row refers to | | |
| | `caption` | dense text description of the clip | | |
| One raw video can appear in many rows (different objects / ranges). Both mask scripts | |
| **dedupe to the first row per `videoId`**, so each raw video yields at most one clip. | |
| ### `all_masks.npz` | |
| Single key `arr_0`: a `uint8` array of shape `(num_frames, H, W)`. `0` is background, | |
| `1..N` are per-instance ids. A row's mask for frame `i` is `arr_0[i] == mask_id`. | |
| Mask resolution may differ from the video's; `overlay_masks.py` nearest-neighbour resizes | |
| to match. | |
| Both mask scripts accept `--masks-dir` pointing at a folder containing **extracted | |
| `<index>_<videoId>/` folders, `pexels-*.zip` archives, or a mix** — `build_mask_index()` | |
| indexes whichever it finds and reads `.npz` bytes straight out of the zip when needed. | |
| Point it at `$PROC/video_inpainting/pexels` (extracted) rather than `$VPDATA/pexels_masks` | |
| (zips) to skip reading 1,247 zip central directories at every startup. | |
| --- | |
| ## Scripts | |
| ### 1. `download_pexels.py` — fetch raw Pexels videos | |
| Samples rows from `pexels.csv` and downloads them concurrently to `<out>/<videoId>.mp4`. | |
| ```bash | |
| python3 $VPDATA/processing/download_pexels.py \ | |
| --csv $VPDATA/pexels.csv \ | |
| --mask-info-csv $VPDATA/pexels_videovo_train_dataset.csv \ | |
| --out $PROC/selected_videos \ | |
| -n 3000 --resolution 1920x1080 --workers 8 --seed 42 | |
| ``` | |
| | flag | default | notes | | |
| |---|---|---| | |
| | `--csv` | `pexels.csv` | resolved against **cwd**, not the script dir — always pass it | | |
| | `--mask-info-csv` | none | keep only videoIds that also have a mask/caption row — use this, or you will download videos you have no mask for | | |
| | `-n/--num-samples` | 2000 | target count of *successful* downloads | | |
| | `--resolution` | `1920x1080` | `any` disables filtering | | |
| | `--seed` | 42 | seeds the shuffle, so the sample is reproducible | | |
| | `--workers` | 8 | concurrent HTTP downloads | | |
| Behaviour worth knowing: downloads go to a `.part` file and are renamed only on success, so | |
| a killed run leaves no truncated `.mp4`. Existing non-empty files are skipped, making re-runs | |
| resumable. Failed URLs (Pexels links do expire) are **replaced by drawing further samples** | |
| until `-n` successes are reached — so the output count is exact, but the specific videos | |
| depend on how many links were dead. | |
| --- | |
| ### 2a. `overlay_masks.py` — QA overlays and inpainting control clips | |
| Produces up to two outputs from a single decode pass over each clip's frame range: | |
| - `--overlay-dir` — the frame alpha-blended with a deterministic colour per instance id. | |
| **Visualization only**, for eyeballing what the mask actually selected. | |
| - `--inpaint-mask-dir` — the frame with the `mask_id` region's axis-aligned **bounding box | |
| painted solid black**. | |
| ```bash | |
| python3 $VPDATA/processing/overlay_masks.py \ | |
| --csv $VPDATA/pexels_videovo_train_dataset.csv \ | |
| --videos-dir $PROC/selected_videos \ | |
| --masks-dir $PROC/video_inpainting/pexels \ | |
| --overlay-dir $PROC/overlay \ | |
| --inpaint-mask-dir $PROC/inpainting \ | |
| --alpha 0.5 --workers 32 | |
| ``` | |
| Two deliberate design choices in the inpainting output: | |
| - **A bounding box, not the precise silhouette.** Using the exact mask contour would leak the | |
| object's shape into the "inpaint this region" signal, letting a model cheat. | |
| - **A hard black replace, not an alpha blend**, encoded **losslessly (`crf=0`)**. The control | |
| clip is read back downstream as a strict binary mask; lossy encoding would smear the box | |
| edges into intermediate values. | |
| The overlay output is *not* lossless — it is only ever looked at. | |
| Output naming: `{index}_{videoId}_{start}-{end}.mp4`. | |
| --- | |
| ### 2b. `outpaint_masks.py` — outpainting control clips | |
| The inverse of the inpainting output: everything **outside** a visible box is painted black, | |
| so the model sees a crop and must hallucinate the surrounding context. | |
| ```bash | |
| python3 $VPDATA/processing/outpaint_masks.py \ | |
| --csv $VPDATA/pexels_videovo_train_dataset.csv \ | |
| --videos-dir $PROC/selected_videos \ | |
| --masks-dir $PROC/video_inpainting/pexels \ | |
| --outpaint-dir $PROC/outpainting \ | |
| --bbox-frac-range 0.4-0.8 --seed vpdata --workers 32 | |
| ``` | |
| **How the visible box is sized** (`--bbox-frac-range`, default `0.4-0.8`): | |
| 1. One target fraction is drawn **per clip**, not per frame — otherwise the visible region | |
| would jitter every time the object drifted in and out of range. | |
| 2. For each frame, if the object's own bbox already covers 40–80% of the frame area, it is | |
| used **as-is**. | |
| 3. Otherwise the box is rescaled about its centre to hit the target fraction — grown outward | |
| if the object was too small, shrunk inward (cutting into the object) if too large — then | |
| shifted to stay inside the frame. Very elongated boxes that cannot fit are clamped to the | |
| frame extent, so the achieved area can fall short of the target. | |
| 4. Frames where `mask_id` is absent become fully black. | |
| `--seed` takes a **string** and is combined per-clip as `random.Random(f"{seed}:{key}")`. | |
| `random.Random(str)` is stable across processes and `PYTHONHASHSEED` values (unlike `hash()`), | |
| so a seeded run is exactly reproducible across the worker pool. Omitting `--seed` is | |
| nondeterministic. | |
| Also encoded losslessly, for the same binary-mask reason as above. Clips whose `mask_id` | |
| never appears in the frame range are skipped **before** the video is decoded, which is the | |
| expensive part. | |
| --- | |
| ### 3. `build_inpainting_outpainting_dataset.py` — assemble the training set | |
| Selects distinct videos across `inpainting/` and `outpainting/` and materializes a flat | |
| dataset directory. | |
| ```bash | |
| python3 $VPDATA/processing/build_inpainting_outpainting_dataset.py \ | |
| --csv $VPDATA/pexels_videovo_train_dataset.csv \ | |
| --videos-dir $PROC/selected_videos \ | |
| --inpainting-dir $PROC/inpainting \ | |
| --outpainting-dir $PROC/outpainting \ | |
| --output $PROC/train_dataset \ | |
| --num-videos 2000 --inpaint-ratio 0.5 --seed 0 --link-mode hardlink | |
| ``` | |
| Output layout: | |
| ``` | |
| train_dataset/ | |
| ├── {stem}.mp4 # raw video, freshly trimmed to [start_frame, end_frame] | |
| ├── {stem}_painting.mp4 # the control clip (hardlinked from inpainting/ or outpainting/) | |
| ├── ... | |
| └── dataset.json # [{caption, media_path, reference_path}, ...] | |
| ``` | |
| Two things this script gets deliberately right: | |
| - **`{stem}.mp4` is re-encoded, not hardlinked.** The control clips cover only a | |
| `[start_frame, end_frame]` sub-range, while `selected_videos/<videoId>.mp4` is the full | |
| video. Linking the full video would leave the pair frame-misaligned, so the raw footage is | |
| re-trimmed to the identical range. | |
| - **Distinct-videoId allocation.** Every outpainting clip's videoId also has an inpainting | |
| clip (outpainting is a strict subset), so naive independent sampling would put the same | |
| source video in both halves. `allocate()` fills the **smaller pool first** and excludes its | |
| videoIds from the other pool's candidates. If a pool runs short it warns and returns fewer | |
| rather than failing. | |
| `--link-mode hardlink` (default) costs no extra disk and falls back to a real copy across | |
| filesystems. Use `copy` if the output must survive deletion of `inpainting/`. | |
| Only clips with **both** a present raw video and a caption in `--csv` are eligible. | |
| --- | |
| ## Gotchas | |
| **Three built-in defaults are stale — pass these flags explicitly** (the commands above | |
| already do). | |
| | Script | Bad default | Pass instead | | |
| |---|---|---| | |
| | `overlay_masks.py` | `--videos-dir` → `VPData/videos` (does not exist) | `--videos-dir $PROC/selected_videos` | | |
| | `build_inpainting_outpainting_dataset.py` | `--csv` → `datasets/pexels_videovo_train_dataset.csv` (resolves one level too high) | `--csv $VPDATA/pexels_videovo_train_dataset.csv` | | |
| | both mask scripts | `--masks-dir` → `VPData/pexels_masks`, which now holds **only the 1,247 zips** | `--masks-dir $PROC/video_inpainting/pexels` | | |
| The last one still works — the scripts read masks out of zips fine — but it re-indexes 1,247 | |
| zip central directories on every startup. | |
| **Other notes** | |
| - `--workers` defaults to every logical CPU. Each clip's decode+encode is independent and | |
| CPU-bound, so it scales near-linearly — lower it on a shared machine. | |
| - `end_frame` is **inclusive** in the CSVs but the loops use `min(end_frame + 1, num_masks)`, | |
| so clips are silently truncated when the mask array is shorter than the CSV claims. | |
| - Failures are reported per clip and tallied, but never abort the run. Check the final | |
| `ok/fail` line. | |
| - `/mnt/dataset/xinyuy/datasets/processing/` (`build_dataset/`, `select_and_trim/`) contains | |
| **0-byte placeholder files** — unrelated to this pipeline and not runnable. | |
| - This pipeline uses the **Pexels subset only**. The videovo masks are extracted here for | |
| other consumers; the videovo raw videos stay zipped (see "What's in this folder"). | |
Xet Storage Details
- Size:
- 14.2 kB
- Xet hash:
- c9f1b066fe75ec11f9d368db6a39c34ea8793988e89466dd473613bfc67b76d5
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.