|
Download video-depth-pointdit/README.md from yslan/dev-pointdit-video-depth: direct link, hf CLI and curl.
- Browser
- Download file 17.8 kB
-
https://huggingface.co/yslan/dev-pointdit-video-depth/resolve/main/video-depth-pointdit/README.md
- Command line
-
hf download hf://yslan/dev-pointdit-video-depth/video-depth-pointdit/README.md
-
curl -L -o README.md https://huggingface.co/yslan/dev-pointdit-video-depth/resolve/main/video-depth-pointdit/README.md
17.8 kB
| # Video-depth PointDiT-B (update 32,500, EMA2) | |
| A temporal PointDiT-B that predicts, for every frame of a short video, that frame's depth in its own camera. It is | |
| the video-depth expert of the vgg-forecast project (`github.com/NIRVANALAN/vgg_forecast_4d`), initialized from the | |
| released PointDiT-B 512 checkpoint and trained for 32,500 updates on a mixture of 22 synthetic and reconstructed | |
| depth sources. This folder holds the EMA weights of update 32,500, the exact training config, the recorded | |
| evaluation rows and everything needed to rebuild the model and score it with the repository's evaluator. | |
| Status of the recipe: it has plateaued. Between update 30,000 and update 32,500 no benchmark moved by more than | |
| 0.002 AbsRel. The model is clearly behind VGGT-Omega on Sintel and KITTI (table below). More updates of this recipe | |
| are not expected to close that gap. | |
| ## Contents | |
| | Path | What it is | | |
| |---|---| | |
| | `train/ckpt_32500/ema2/model.safetensors` | the weights: EMA with decay 0.9996; 216 tensors (213 parameters, 140,204,928 values, plus 3 fixed position buffers), fp32 | | |
| | `train/ckpt_32500/ema2/model.json` | the artifact record: architecture, tensor manifest, model profile, parent checkpoint hashes, DINO identity | | |
| | `train/ckpt_32500/point_training.json` | the training progress and config hashes at update 32,500 | | |
| | `train/ckpt_32500/vggforecast_checkpoint.json` | the checkpoint format marker | | |
| | `train/resolved_resume_29500.yaml` | the resolved training config of the last training start (resumed from update 29,500, ran to 32,500) | | |
| | `train/launch_resume_29500.json` | that start's launch record: source commit `321afca216fc13eaf6e8264d1f966f065d61ce4b`, resume ancestry, data census | | |
| | `rows/*.jsonl` | the recorded evaluation rows (raw and EMA2 weights; video and single-frame calls) | | |
| | `manifest.json`, `manifest.sha256` | file names, sizes, sha256 (`manifest.sha256` is `sha256sum -c` input), code commit, update count | | |
| | `infer_video_depth.py` | depth for your own video or frames: npy, colour-mapped PNGs, MP4 (see "Run inference on your own video") | | |
| | `reproduce_evaluation.sh` | the evaluation commands, taking every external path by flag (see Quick start) | | |
| | `load_point.py` | builds the model through the repository's builder and loads the EMA weights (reads only this folder) | | |
| There is no optimizer state and no raw (non-EMA) weight file here. The evaluator needs only the EMA2 artifact. | |
| The frozen DINOv3 encoder is not included (see Licence). | |
| ## What the model predicts | |
| - **Architecture:** PointDiT-B/16 (12 blocks, width 768, 12 heads, bottleneck patch embedding), made temporal: | |
| even blocks attend within a frame, odd blocks attend over all frames; the private 3D rotary embedding is | |
| sectioned (time | row | column); one residual adaptive-LayerNorm branch carries the physical frame time | |
| (`t_j - t_0` in frames). One generated channel: the released three-channel (XYZ) head is sliced to its Z column. | |
| - **Conditioning:** frozen DINOv3 ViT-B/16 features of every frame (layers 2, 5, 8, 11, concatenated: 3,072 | |
| channels per 16 x 16 patch). No raw RGB enters the Point network. | |
| - **Input:** a window of T RGB frames (trained on T = 1 to 25, evaluated with T = 21), height and width multiples | |
| of 16 at about 512 x 512 pixels of area (ten aspect buckets, never squashed; for example 336 x 784 for Sintel, | |
| 288 x 928 for KITTI), and strictly increasing integer frame indices starting at 0 (the frames' true indices; they | |
| set the rotary time and the physical-time branch). | |
| - **Output:** `[1, T, 1, H, W]`: each frame's own-camera depth Z. The depth is in the units of the window's | |
| gauge: a single scale shared by every frame of the window, the mean valid depth of the window's frames. The model | |
| predicts a contracted state (symmetric mip-NeRF 360 contraction); the repository decodes it, capping the state at | |
| 1.984375 (a depth of at most 64 gauge units). | |
| - **Metric-free depth.** The output is depth up to one unknown scale per window (and, in the benchmark, one fitted | |
| shift): it is not metric. To get depth in metres, multiply by a known scale. The evaluation recovers it the way | |
| Video-Depth-Anything does: one least-squares scale and shift per sequence, fitted in depth against the ground | |
| truth. Nothing inside the model predicts the scale. | |
| - **Sampling for evaluation:** one Euler step from Gaussian noise (the x0 head makes one step a one-shot | |
| prediction), `--base-steps 1`, on the logit-normal quantile grid the model was trained with (its native sampler is | |
| four steps, noise scale 2.0). | |
| ## Quick start without Civo | |
| Everything below uses only this folder, the project repository on GitHub, two public checkouts and files you request or | |
| download yourself. No path of the Civo cluster is on any command line. | |
| **1. Code.** `git clone git@github.com:NIRVANALAN/vgg_forecast_4d.git` (you need access to the repository) and check out | |
| `main`. Loading and inference need commit `2310eec5` or later. The evaluation flag `--data-root` (used below) is on the | |
| branch `docs/video-depth-pointdit-off-civo` until its pull request is merged; check that branch out for the evaluation. | |
| Run every command from the clone root with `export PYTHONPATH=$PWD/src`. | |
| **2. Environment.** The scripts were run in Python 3.12.13 with torch 2.9.0+cu130 (CUDA 13.0), numpy 2.4.6, omegaconf | |
| 2.3.1, hydra-core 1.3.4, einops 0.8.2, safetensors 0.8.0, opencv-python 4.11.0.86, imageio 2.37.4 with imageio-ffmpeg | |
| 0.6.0, scipy 1.18.0, pillow 12.3.0, matplotlib 3.11.1, accelerate 1.14.0, timm 1.0.28, transformers 5.14.1. The recorded | |
| rows used FlashAttention 2.8.3 (`fa2`); the scripts default to PyTorch's `sdpa` attention, which needs no extra | |
| package (pass `--attention-backend fa2` where it is offered if you installed flash-attn). A GPU is needed for evaluation | |
| and recommended for inference; loading runs on a CPU. | |
| **3. This folder.** `hf download yslan/dev-pointdit-video-depth --include "video-depth-pointdit/*" --local-dir <dir>`; the | |
| folder is `<dir>/video-depth-pointdit`, called `$FOLDER` below. `cd $FOLDER && sha256sum -c manifest.sha256` checks it. | |
| **4. DINOv3 ViT-B/16 (not included).** Request it at `https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/` | |
| (Meta's licence). The file is `dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth`, sha256 | |
| `73cec8be7427c8655ceced13ce62f6e20a1fa90d1b4d4a550df17a1144081a7c` (the scripts refuse another file). The code is | |
| `git clone https://github.com/facebookresearch/dinov3` (tested at `6876159a11b4df116f30f667f8c9888617df0751`). Both go | |
| to the scripts by flag: `--dino-weights <file> --dinov3-repo <checkout>`. | |
| **5. Load (CPU, about 15 s).** | |
| ```bash | |
| python $FOLDER/load_point.py $FOLDER | |
| # temporal_pointdit window_valid_mean_depth 32500 0.9996 140204928 | |
| ``` | |
| **6. Inference on your video (section below).** | |
| ```bash | |
| python $FOLDER/infer_video_depth.py $FOLDER --input clip.mp4 --out out/ \ | |
| --dino-weights <dinov3 file> --dinov3-repo <dinov3 checkout> | |
| ``` | |
| **7. Evaluation (one GPU).** Needs the benchmark data in the layout of the repository's evaluator | |
| (`monocular_depth.cohort_pairs`; the layout and the 21 frames per sequence are written out in the repository's | |
| `doc/handoff/video_depth_and_camera_pose_evaluation_for_external_models.md`, section 3.1) and a checkout of | |
| `https://github.com/NIRVANALAN/STream3R` at `7bdca8cf7ec206ff8b08c0941ad8286d72087d4f`, whose depth readers the evaluator | |
| uses (`STREAM3R_ROOT`): | |
| ```bash | |
| export STREAM3R_ROOT=<STream3R checkout> | |
| bash $FOLDER/reproduce_evaluation.sh $FOLDER out/ --dino-weights <dinov3 file> --dinov3-repo <dinov3 checkout> \ | |
| --data-root <dir holding sintel/ bonn/ kitti/> [--datasets "sintel bonn kitti"] | |
| ``` | |
| It checks `manifest.sha256`, writes a temporary run directory (this folder's config with the three external paths | |
| replaced; the shipped config is not edited) and runs `eval/temporal_point/checkpoint_evaluation.py --task video_depth` | |
| per dataset with the aspect-preserving depth-fit protocol, EMA2, seed 0, `--base-steps 1`, `--depth-state-cap 1.984375`. | |
| **What reads what.** The released PointDiT-B 512 checkpoint is never read: the scripts set `model.point.weights: random`, | |
| which builds the same architecture, and all 216 tensors then come from the EMA2 file. The model after loading EMA2 is | |
| bit-identical to one built over the released checkpoint (checked: 216 of 216 tensors equal). The recorded training | |
| config keeps its Civo paths as history. Which keys still name one: | |
| | Config key (`train/resolved_resume_29500.yaml`) | Read by load / inference / evaluation? | How to override | | |
| |---|---|---| | |
| | `model.point.checkpoint` (the released PointDiT-B 512 file) | no: skipped (see above) | nothing to do | | |
| | `model.dino.weights` (DINOv3 file), `model.dino.repo` (DINOv3 checkout) | yes, by inference and evaluation | `--dino-weights`, `--dinov3-repo` | | |
| | the evaluation data root (not in the config: `monocular_depth.DATA_ROOT`) | yes, by evaluation | `--data-root` | | |
| | STream3R checkout (not in the config: default server path) | yes, by evaluation | environment variable `STREAM3R_ROOT` | | |
| | `data.track4d.sources[*].kwargs.ROOT`, `metric_scales_csv`, `index_cache_dir`, `camera_exclusion_manifest` roots | no: training data, not read by these scripts | none needed | | |
| | `train.output_dir`, `train.resume_from` | no | none needed | | |
| The last two rows are from reading the code (the evaluator reads only `data.track4d.frames`, the model, DINO and schedule | |
| sections); the off-Civo check below ran on a machine where those directories still exist, so it does not prove it. | |
| ## Run inference on your own video | |
| `infer_video_depth.py` takes a video file or a folder of frames and writes per-frame depth. It builds the model through | |
| the repository's builder (`build_point_model`), loads the EMA2 weights, computes the frozen DINOv3 ViT-B/16 features of the | |
| frames and samples with the repository's sampler exactly as the evaluation does: one Euler step from Gaussian noise, | |
| the contracted depth state capped at 1.984375, bf16 autocast. Run it from a clone of the repository (steps 1 to 4 | |
| of the Quick start), on one GPU (a CPU works but takes minutes per frame set): | |
| ```bash | |
| D="--dino-weights <dinov3 file> --dinov3-repo <dinov3 checkout>" | |
| python $FOLDER/infer_video_depth.py $FOLDER --input clip.mp4 --out out/ $D # first 21 frames | |
| python $FOLDER/infer_video_depth.py $FOLDER --input frames_dir/ --out out/ --max-frames 25 --stride 2 $D | |
| ``` | |
| - **Frames:** the first `--max-frames` (default 21, at most 25) frames at `--stride`, in one call. Each is resized, whole | |
| and without a crop, to the raster that keeps its aspect ratio at about 512 x 512 pixels in 16-pixel patches | |
| (for example 480 x 640 becomes 448 x 592; 436 x 1024 becomes 336 x 784), with `cv2.INTER_AREA`, as the benchmark does. | |
| The frame indices given to the model are `0, stride, 2 stride, ...`; they set its rotary time and physical-time input, | |
| so give the stride the clip really has between frames (training used strides 1 to 8). For a longer video, call it on | |
| successive chunks; the chunks are then separate gauges (one unknown scale each). | |
| - **Outputs** in `--out`: `depth.npy` ([T,H,W] float32 on the model raster, in units of the window's mean depth: one | |
| unknown scale for the whole window, not metric); `rgb.npy` (the resized input); `depth_NNN.png` (colour-mapped | |
| disparity, 1 / depth, inferno, normalized over the whole clip, as Video-Depth-Anything draws it); `depth.mp4` (input and | |
| colour map side by side); `info.json` (raster, frame indices, sampler settings, seed). | |
| - **Scale.** `depth.npy` is the model's output unchanged. With `--hfov-degrees <fov>` the script also writes | |
| `depth_frame0_gauge.npy`: the same depth times the evaluator's frame-0 gauge factor (frame 0's mean point norm | |
| becomes 1; it needs a field of view). Either way the depth is known only up to one scale: to get metres multiply by one | |
| known scale, or fit one scale and shift per clip against a reference, as the benchmark does. | |
| - **Seed.** The sampler starts from Gaussian noise: `--seed` fixes it. `--seed-key 0:<dataset>:<sequence>` uses the | |
| benchmark's noise key, which reproduces a benchmark row's call exactly. | |
| - **Weights it needs:** only this folder and the DINOv3 ViT-B/16 file and checkout (`--dino-weights`, `--dinov3-repo`, | |
| both required). `--attention-backend` is `sdpa` by default; `fa2` is the recorded run's. | |
| **Checked.** (1) Before the off-Civo flags were added, on one H100 (2026-10-07, repository commit `2310eec5`), the script | |
| run on the first 21 frames of Bonn `balloon2` and Sintel `alley_2` with the benchmark noise key | |
| (`--seed-key 0:<dataset>:<sequence>`) scores, after one scale and shift per sequence in depth against the ground truth | |
| (the registered protocol), AbsRel 0.0970 on `balloon2` (recorded row: 0.0970) and 0.2377 on `alley_2` (recorded: 0.2368); | |
| delta<1.25 0.9480 and 0.6414 (recorded 0.9480 and 0.6420). The recorded rows came from an A100; the small differences are | |
| GPU and kernel numerics. (2) With the flags of this README, on one H100 (repository branch `docs/video-depth-pointdit-off-civo`), | |
| every external input copied to node-local `/tmp` and passed by flag (this folder, the DINOv3 file and checkout, a STream3R | |
| checkout, the Bonn frames), with `TRACK4D_ROOT` and `VGGT_OMEGA_ROOT` pointing at nonexistent paths: `load_point.py` printed | |
| `temporal_pointdit window_valid_mean_depth 32500 0.9996 140204928`; `infer_video_depth.py` ran on a folder of 21 Bonn frames | |
| and on an mp4 made from them (448 x 592 raster, `sdpa` attention); and `reproduce_evaluation.sh --datasets bonn` re-scored | |
| Bonn at AbsRel 0.060529 against the recorded 0.060454 (difference +7.4e-5; `sdpa` here, `fa2` in the recorded row). | |
| (3) A model built without the released checkpoint holds exactly the same 216 tensors as one built over it, after loading | |
| EMA2 (all equal, bit for bit). A full three-set re-score from a downloaded copy of this folder with the earlier scripts | |
| (`fa2`, Civo paths) gave Sintel 0.381766, Bonn 0.060523, KITTI 0.090556 against the recorded 0.381446, 0.060454, 0.090497. | |
| ## Recorded results | |
| Mean AbsRel over sequences (median over sequences in brackets); the first 21 frames of each sequence; the whole | |
| frame at the dataset's native aspect, scored on the ground-truth grid; one least-squares scale and shift per | |
| sequence, fitted in depth (`video_depth_anything_aspect_preserving_depth_fit`). EMA2, one step from noise. | |
| | | Sintel (13 sequences) | Bonn (5) | KITTI (13) | | |
| |---|---|---|---| | |
| | This model, 21-frame call | 0.381 (0.262) | 0.060 (0.054) | 0.090 (0.091) | | |
| | This model, raw weights (not shipped) | 0.383 | 0.060 | 0.100 | | |
| | This model, each frame its own one-frame call (per-frame fit) | 0.379 (0.254) | 0.061 (0.054) | 0.092 (0.090) | | |
| | VGGT-Omega (1B), 21-frame call | 0.145 (0.114) | 0.053 (0.042) | 0.038 (0.031) | | |
| | Per-frame PointDiT-B 512 (released), 21 frames | 0.394 (0.415) | 0.071 (0.053) | 0.118 (0.107) | | |
| Learning curve of the same recipe (EMA2, same protocol): update 30,000: 0.379 / 0.062 / 0.089; update 32,500: | |
| 0.381 / 0.060 / 0.090. Sintel's mean is carried by two sequences (`ambush_5` 1.85, `market_6` 0.57) whose single | |
| depth-space scale and shift is pulled by far pixels; the rank correlation of the raw prediction with the ground truth | |
| is high (0.84 and 0.97 on those two). KITTI's near third (closer than about 10 m) is under-predicted. | |
| ## Training summary | |
| - **Parent:** released PointDiT-B 512 mixdata EMA branch (`pointditb-512-mixdata-nodinov3-1d42aacf.pth`, sha256 | |
| `1d42aacfd6aaf97c8bd96b0ce5e23bd880cd01b2efbb2ac80ccf8876fde3034b`), Z channel sliced. | |
| - **Updates:** 32,500 in four segments; global batch 64 (4 A100-80GB x 16), AdamW (0.9, 0.95), constant learning | |
| rate 5e-5, gradient clip 1.0, bf16 autocast, EMA decays 0.9999 and 0.9996. | |
| - **Objective:** PointDiT's velocity loss with noise scale 2.0 plus 0.1 x a bounded relative depth term, per-sample | |
| mean over valid pixels. | |
| - **Data (last segments):** BEDLAM, Spring, vKITTI 2, OmniWorld-Game (depth within 20 m), Syn4D v1 / v2, TartanAir, | |
| MVS-Synth, DL3DV (reconstructed depth), Dynamic Replica, PointOdyssey, Kubric49 / 81, Waymo (LiDAR), Hypersim, | |
| ScanNet++, ScanNet, Mid-Air, WildRGB-D, SynthVerse and BlendedMVS; 65 % video windows, 15 % single frames, 20 % | |
| unordered multi-view sets; 1 to 25 frames per sample. | |
| - **Evaluation data is held out:** Sintel, Bonn and KITTI (video-depth benchmark sets) are not training sources. | |
| ## Limits | |
| - The error that remains is not fixable by more steps of this recipe: it plateaus at update ~30,000. | |
| - Output is scale-free; the gauge is the window's own mean depth, not frame 0's depth (the camera expert of the | |
| MoT normalizes translation by frame 0's depth: the two gauges must be reconciled before they are combined). | |
| - One step from noise is the evaluated sampler. The segment-4 offset diagnosis (update 27,000) found four and | |
| sixteen steps give the same depth offset as one step. | |
| ## Licence and third-party notes | |
| - The repository code is the project's; the PointDiT trunk code and the released PointDiT checkpoint come from | |
| `github.com/google-research/pointdit` (Apache-2.0 code; read the checkpoint's own terms before redistribution). | |
| - The weights were trained on datasets with their own licences, several research-only (for example Hypersim, ScanNet, | |
| ScanNet++, Waymo, DL3DV). Treat the weights as research-only until the owner chooses a licence. | |
| - DINOv3 (Meta) is under the DINOv3 licence and is NOT redistributed here. Request it at | |
| `https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/`. The model needs the ViT-B/16 LVD-1689M | |
| weights (`dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth`, sha256 | |
| `73cec8be7427c8655ceced13ce62f6e20a1fa90d1b4d4a550df17a1144081a7c`). | |