yslan's picture
Video-depth PointDiT-B, update 32500, EMA2
9479bc0 verified
|
Raw History Blame Contribute Delete
17.8 kB

Video-depth PointDiT-B (update 32,500, EMA2)

A temporal PointDiT-B that predicts, for every frame of a short video, that frame's depth in its own camera. It is the video-depth expert of the vgg-forecast project (github.com/NIRVANALAN/vgg_forecast_4d), initialized from the released PointDiT-B 512 checkpoint and trained for 32,500 updates on a mixture of 22 synthetic and reconstructed depth sources. This folder holds the EMA weights of update 32,500, the exact training config, the recorded evaluation rows and everything needed to rebuild the model and score it with the repository's evaluator.

Status of the recipe: it has plateaued. Between update 30,000 and update 32,500 no benchmark moved by more than 0.002 AbsRel. The model is clearly behind VGGT-Omega on Sintel and KITTI (table below). More updates of this recipe are not expected to close that gap.

Contents

Path What it is
train/ckpt_32500/ema2/model.safetensors the weights: EMA with decay 0.9996; 216 tensors (213 parameters, 140,204,928 values, plus 3 fixed position buffers), fp32
train/ckpt_32500/ema2/model.json the artifact record: architecture, tensor manifest, model profile, parent checkpoint hashes, DINO identity
train/ckpt_32500/point_training.json the training progress and config hashes at update 32,500
train/ckpt_32500/vggforecast_checkpoint.json the checkpoint format marker
train/resolved_resume_29500.yaml the resolved training config of the last training start (resumed from update 29,500, ran to 32,500)
train/launch_resume_29500.json that start's launch record: source commit 321afca216fc13eaf6e8264d1f966f065d61ce4b, resume ancestry, data census
rows/*.jsonl the recorded evaluation rows (raw and EMA2 weights; video and single-frame calls)
manifest.json, manifest.sha256 file names, sizes, sha256 (manifest.sha256 is sha256sum -c input), code commit, update count
infer_video_depth.py depth for your own video or frames: npy, colour-mapped PNGs, MP4 (see "Run inference on your own video")
reproduce_evaluation.sh the evaluation commands, taking every external path by flag (see Quick start)
load_point.py builds the model through the repository's builder and loads the EMA weights (reads only this folder)

There is no optimizer state and no raw (non-EMA) weight file here. The evaluator needs only the EMA2 artifact. The frozen DINOv3 encoder is not included (see Licence).

What the model predicts

  • Architecture: PointDiT-B/16 (12 blocks, width 768, 12 heads, bottleneck patch embedding), made temporal: even blocks attend within a frame, odd blocks attend over all frames; the private 3D rotary embedding is sectioned (time | row | column); one residual adaptive-LayerNorm branch carries the physical frame time (t_j - t_0 in frames). One generated channel: the released three-channel (XYZ) head is sliced to its Z column.
  • Conditioning: frozen DINOv3 ViT-B/16 features of every frame (layers 2, 5, 8, 11, concatenated: 3,072 channels per 16 x 16 patch). No raw RGB enters the Point network.
  • Input: a window of T RGB frames (trained on T = 1 to 25, evaluated with T = 21), height and width multiples of 16 at about 512 x 512 pixels of area (ten aspect buckets, never squashed; for example 336 x 784 for Sintel, 288 x 928 for KITTI), and strictly increasing integer frame indices starting at 0 (the frames' true indices; they set the rotary time and the physical-time branch).
  • Output: [1, T, 1, H, W]: each frame's own-camera depth Z. The depth is in the units of the window's gauge: a single scale shared by every frame of the window, the mean valid depth of the window's frames. The model predicts a contracted state (symmetric mip-NeRF 360 contraction); the repository decodes it, capping the state at 1.984375 (a depth of at most 64 gauge units).
  • Metric-free depth. The output is depth up to one unknown scale per window (and, in the benchmark, one fitted shift): it is not metric. To get depth in metres, multiply by a known scale. The evaluation recovers it the way Video-Depth-Anything does: one least-squares scale and shift per sequence, fitted in depth against the ground truth. Nothing inside the model predicts the scale.
  • Sampling for evaluation: one Euler step from Gaussian noise (the x0 head makes one step a one-shot prediction), --base-steps 1, on the logit-normal quantile grid the model was trained with (its native sampler is four steps, noise scale 2.0).

Quick start without Civo

Everything below uses only this folder, the project repository on GitHub, two public checkouts and files you request or download yourself. No path of the Civo cluster is on any command line.

1. Code. git clone git@github.com:NIRVANALAN/vgg_forecast_4d.git (you need access to the repository) and check out main. Loading and inference need commit 2310eec5 or later. The evaluation flag --data-root (used below) is on the branch docs/video-depth-pointdit-off-civo until its pull request is merged; check that branch out for the evaluation. Run every command from the clone root with export PYTHONPATH=$PWD/src.

2. Environment. The scripts were run in Python 3.12.13 with torch 2.9.0+cu130 (CUDA 13.0), numpy 2.4.6, omegaconf 2.3.1, hydra-core 1.3.4, einops 0.8.2, safetensors 0.8.0, opencv-python 4.11.0.86, imageio 2.37.4 with imageio-ffmpeg 0.6.0, scipy 1.18.0, pillow 12.3.0, matplotlib 3.11.1, accelerate 1.14.0, timm 1.0.28, transformers 5.14.1. The recorded rows used FlashAttention 2.8.3 (fa2); the scripts default to PyTorch's sdpa attention, which needs no extra package (pass --attention-backend fa2 where it is offered if you installed flash-attn). A GPU is needed for evaluation and recommended for inference; loading runs on a CPU.

3. This folder. hf download yslan/dev-pointdit-video-depth --include "video-depth-pointdit/*" --local-dir <dir>; the folder is <dir>/video-depth-pointdit, called $FOLDER below. cd $FOLDER && sha256sum -c manifest.sha256 checks it.

4. DINOv3 ViT-B/16 (not included). Request it at https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/ (Meta's licence). The file is dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth, sha256 73cec8be7427c8655ceced13ce62f6e20a1fa90d1b4d4a550df17a1144081a7c (the scripts refuse another file). The code is git clone https://github.com/facebookresearch/dinov3 (tested at 6876159a11b4df116f30f667f8c9888617df0751). Both go to the scripts by flag: --dino-weights <file> --dinov3-repo <checkout>.

5. Load (CPU, about 15 s).

python $FOLDER/load_point.py $FOLDER
# temporal_pointdit window_valid_mean_depth 32500 0.9996 140204928

6. Inference on your video (section below).

python $FOLDER/infer_video_depth.py $FOLDER --input clip.mp4 --out out/ \
  --dino-weights <dinov3 file> --dinov3-repo <dinov3 checkout>

7. Evaluation (one GPU). Needs the benchmark data in the layout of the repository's evaluator (monocular_depth.cohort_pairs; the layout and the 21 frames per sequence are written out in the repository's doc/handoff/video_depth_and_camera_pose_evaluation_for_external_models.md, section 3.1) and a checkout of https://github.com/NIRVANALAN/STream3R at 7bdca8cf7ec206ff8b08c0941ad8286d72087d4f, whose depth readers the evaluator uses (STREAM3R_ROOT):

export STREAM3R_ROOT=<STream3R checkout>
bash $FOLDER/reproduce_evaluation.sh $FOLDER out/ --dino-weights <dinov3 file> --dinov3-repo <dinov3 checkout> \
  --data-root <dir holding sintel/ bonn/ kitti/> [--datasets "sintel bonn kitti"]

It checks manifest.sha256, writes a temporary run directory (this folder's config with the three external paths replaced; the shipped config is not edited) and runs eval/temporal_point/checkpoint_evaluation.py --task video_depth per dataset with the aspect-preserving depth-fit protocol, EMA2, seed 0, --base-steps 1, --depth-state-cap 1.984375.

What reads what. The released PointDiT-B 512 checkpoint is never read: the scripts set model.point.weights: random, which builds the same architecture, and all 216 tensors then come from the EMA2 file. The model after loading EMA2 is bit-identical to one built over the released checkpoint (checked: 216 of 216 tensors equal). The recorded training config keeps its Civo paths as history. Which keys still name one:

Config key (train/resolved_resume_29500.yaml) Read by load / inference / evaluation? How to override
model.point.checkpoint (the released PointDiT-B 512 file) no: skipped (see above) nothing to do
model.dino.weights (DINOv3 file), model.dino.repo (DINOv3 checkout) yes, by inference and evaluation --dino-weights, --dinov3-repo
the evaluation data root (not in the config: monocular_depth.DATA_ROOT) yes, by evaluation --data-root
STream3R checkout (not in the config: default server path) yes, by evaluation environment variable STREAM3R_ROOT
data.track4d.sources[*].kwargs.ROOT, metric_scales_csv, index_cache_dir, camera_exclusion_manifest roots no: training data, not read by these scripts none needed
train.output_dir, train.resume_from no none needed

The last two rows are from reading the code (the evaluator reads only data.track4d.frames, the model, DINO and schedule sections); the off-Civo check below ran on a machine where those directories still exist, so it does not prove it.

Run inference on your own video

infer_video_depth.py takes a video file or a folder of frames and writes per-frame depth. It builds the model through the repository's builder (build_point_model), loads the EMA2 weights, computes the frozen DINOv3 ViT-B/16 features of the frames and samples with the repository's sampler exactly as the evaluation does: one Euler step from Gaussian noise, the contracted depth state capped at 1.984375, bf16 autocast. Run it from a clone of the repository (steps 1 to 4 of the Quick start), on one GPU (a CPU works but takes minutes per frame set):

D="--dino-weights <dinov3 file> --dinov3-repo <dinov3 checkout>"
python $FOLDER/infer_video_depth.py $FOLDER --input clip.mp4 --out out/ $D            # first 21 frames
python $FOLDER/infer_video_depth.py $FOLDER --input frames_dir/ --out out/ --max-frames 25 --stride 2 $D
  • Frames: the first --max-frames (default 21, at most 25) frames at --stride, in one call. Each is resized, whole and without a crop, to the raster that keeps its aspect ratio at about 512 x 512 pixels in 16-pixel patches (for example 480 x 640 becomes 448 x 592; 436 x 1024 becomes 336 x 784), with cv2.INTER_AREA, as the benchmark does. The frame indices given to the model are 0, stride, 2 stride, ...; they set its rotary time and physical-time input, so give the stride the clip really has between frames (training used strides 1 to 8). For a longer video, call it on successive chunks; the chunks are then separate gauges (one unknown scale each).
  • Outputs in --out: depth.npy ([T,H,W] float32 on the model raster, in units of the window's mean depth: one unknown scale for the whole window, not metric); rgb.npy (the resized input); depth_NNN.png (colour-mapped disparity, 1 / depth, inferno, normalized over the whole clip, as Video-Depth-Anything draws it); depth.mp4 (input and colour map side by side); info.json (raster, frame indices, sampler settings, seed).
  • Scale. depth.npy is the model's output unchanged. With --hfov-degrees <fov> the script also writes depth_frame0_gauge.npy: the same depth times the evaluator's frame-0 gauge factor (frame 0's mean point norm becomes 1; it needs a field of view). Either way the depth is known only up to one scale: to get metres multiply by one known scale, or fit one scale and shift per clip against a reference, as the benchmark does.
  • Seed. The sampler starts from Gaussian noise: --seed fixes it. --seed-key 0:<dataset>:<sequence> uses the benchmark's noise key, which reproduces a benchmark row's call exactly.
  • Weights it needs: only this folder and the DINOv3 ViT-B/16 file and checkout (--dino-weights, --dinov3-repo, both required). --attention-backend is sdpa by default; fa2 is the recorded run's.

Checked. (1) Before the off-Civo flags were added, on one H100 (2026-10-07, repository commit 2310eec5), the script run on the first 21 frames of Bonn balloon2 and Sintel alley_2 with the benchmark noise key (--seed-key 0:<dataset>:<sequence>) scores, after one scale and shift per sequence in depth against the ground truth (the registered protocol), AbsRel 0.0970 on balloon2 (recorded row: 0.0970) and 0.2377 on alley_2 (recorded: 0.2368); delta<1.25 0.9480 and 0.6414 (recorded 0.9480 and 0.6420). The recorded rows came from an A100; the small differences are GPU and kernel numerics. (2) With the flags of this README, on one H100 (repository branch docs/video-depth-pointdit-off-civo), every external input copied to node-local /tmp and passed by flag (this folder, the DINOv3 file and checkout, a STream3R checkout, the Bonn frames), with TRACK4D_ROOT and VGGT_OMEGA_ROOT pointing at nonexistent paths: load_point.py printed temporal_pointdit window_valid_mean_depth 32500 0.9996 140204928; infer_video_depth.py ran on a folder of 21 Bonn frames and on an mp4 made from them (448 x 592 raster, sdpa attention); and reproduce_evaluation.sh --datasets bonn re-scored Bonn at AbsRel 0.060529 against the recorded 0.060454 (difference +7.4e-5; sdpa here, fa2 in the recorded row). (3) A model built without the released checkpoint holds exactly the same 216 tensors as one built over it, after loading EMA2 (all equal, bit for bit). A full three-set re-score from a downloaded copy of this folder with the earlier scripts (fa2, Civo paths) gave Sintel 0.381766, Bonn 0.060523, KITTI 0.090556 against the recorded 0.381446, 0.060454, 0.090497.

Recorded results

Mean AbsRel over sequences (median over sequences in brackets); the first 21 frames of each sequence; the whole frame at the dataset's native aspect, scored on the ground-truth grid; one least-squares scale and shift per sequence, fitted in depth (video_depth_anything_aspect_preserving_depth_fit). EMA2, one step from noise.

Sintel (13 sequences) Bonn (5) KITTI (13)
This model, 21-frame call 0.381 (0.262) 0.060 (0.054) 0.090 (0.091)
This model, raw weights (not shipped) 0.383 0.060 0.100
This model, each frame its own one-frame call (per-frame fit) 0.379 (0.254) 0.061 (0.054) 0.092 (0.090)
VGGT-Omega (1B), 21-frame call 0.145 (0.114) 0.053 (0.042) 0.038 (0.031)
Per-frame PointDiT-B 512 (released), 21 frames 0.394 (0.415) 0.071 (0.053) 0.118 (0.107)

Learning curve of the same recipe (EMA2, same protocol): update 30,000: 0.379 / 0.062 / 0.089; update 32,500: 0.381 / 0.060 / 0.090. Sintel's mean is carried by two sequences (ambush_5 1.85, market_6 0.57) whose single depth-space scale and shift is pulled by far pixels; the rank correlation of the raw prediction with the ground truth is high (0.84 and 0.97 on those two). KITTI's near third (closer than about 10 m) is under-predicted.

Training summary

  • Parent: released PointDiT-B 512 mixdata EMA branch (pointditb-512-mixdata-nodinov3-1d42aacf.pth, sha256 1d42aacfd6aaf97c8bd96b0ce5e23bd880cd01b2efbb2ac80ccf8876fde3034b), Z channel sliced.
  • Updates: 32,500 in four segments; global batch 64 (4 A100-80GB x 16), AdamW (0.9, 0.95), constant learning rate 5e-5, gradient clip 1.0, bf16 autocast, EMA decays 0.9999 and 0.9996.
  • Objective: PointDiT's velocity loss with noise scale 2.0 plus 0.1 x a bounded relative depth term, per-sample mean over valid pixels.
  • Data (last segments): BEDLAM, Spring, vKITTI 2, OmniWorld-Game (depth within 20 m), Syn4D v1 / v2, TartanAir, MVS-Synth, DL3DV (reconstructed depth), Dynamic Replica, PointOdyssey, Kubric49 / 81, Waymo (LiDAR), Hypersim, ScanNet++, ScanNet, Mid-Air, WildRGB-D, SynthVerse and BlendedMVS; 65 % video windows, 15 % single frames, 20 % unordered multi-view sets; 1 to 25 frames per sample.
  • Evaluation data is held out: Sintel, Bonn and KITTI (video-depth benchmark sets) are not training sources.

Limits

  • The error that remains is not fixable by more steps of this recipe: it plateaus at update ~30,000.
  • Output is scale-free; the gauge is the window's own mean depth, not frame 0's depth (the camera expert of the MoT normalizes translation by frame 0's depth: the two gauges must be reconciled before they are combined).
  • One step from noise is the evaluated sampler. The segment-4 offset diagnosis (update 27,000) found four and sixteen steps give the same depth offset as one step.

Licence and third-party notes

  • The repository code is the project's; the PointDiT trunk code and the released PointDiT checkpoint come from github.com/google-research/pointdit (Apache-2.0 code; read the checkpoint's own terms before redistribution).
  • The weights were trained on datasets with their own licences, several research-only (for example Hypersim, ScanNet, ScanNet++, Waymo, DL3DV). Treat the weights as research-only until the owner chooses a licence.
  • DINOv3 (Meta) is under the DINOv3 licence and is NOT redistributed here. Request it at https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/. The model needs the ViT-B/16 LVD-1689M weights (dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth, sha256 73cec8be7427c8655ceced13ce62f6e20a1fa90d1b4d4a550df17a1144081a7c).