# Video-depth PointDiT-B (update 32,500, EMA2) A temporal PointDiT-B that predicts, for every frame of a short video, that frame's depth in its own camera. It is the video-depth expert of the vgg-forecast project (`github.com/NIRVANALAN/vgg_forecast_4d`), initialized from the released PointDiT-B 512 checkpoint and trained for 32,500 updates on a mixture of 22 synthetic and reconstructed depth sources. This folder holds the EMA weights of update 32,500, the exact training config, the recorded evaluation rows and everything needed to rebuild the model and score it with the repository's evaluator. Status of the recipe: it has plateaued. Between update 30,000 and update 32,500 no benchmark moved by more than 0.002 AbsRel. The model is clearly behind VGGT-Omega on Sintel and KITTI (table below). More updates of this recipe are not expected to close that gap. ## Contents | Path | What it is | |---|---| | `train/ckpt_32500/ema2/model.safetensors` | the weights: EMA with decay 0.9996; 216 tensors (213 parameters, 140,204,928 values, plus 3 fixed position buffers), fp32 | | `train/ckpt_32500/ema2/model.json` | the artifact record: architecture, tensor manifest, model profile, parent checkpoint hashes, DINO identity | | `train/ckpt_32500/point_training.json` | the training progress and config hashes at update 32,500 | | `train/ckpt_32500/vggforecast_checkpoint.json` | the checkpoint format marker | | `train/resolved_resume_29500.yaml` | the resolved training config of the last training start (resumed from update 29,500, ran to 32,500) | | `train/launch_resume_29500.json` | that start's launch record: source commit `321afca216fc13eaf6e8264d1f966f065d61ce4b`, resume ancestry, data census | | `rows/*.jsonl` | the recorded evaluation rows (raw and EMA2 weights; video and single-frame calls) | | `manifest.json`, `manifest.sha256` | file names, sizes, sha256 (`manifest.sha256` is `sha256sum -c` input), code commit, update count | | `infer_video_depth.py` | depth for your own video or frames: npy, colour-mapped PNGs, MP4 (see "Run inference on your own video") | | `reproduce_evaluation.sh` | the evaluation commands, taking every external path by flag (see Quick start) | | `load_point.py` | builds the model through the repository's builder and loads the EMA weights (reads only this folder) | There is no optimizer state and no raw (non-EMA) weight file here. The evaluator needs only the EMA2 artifact. The frozen DINOv3 encoder is not included (see Licence). ## What the model predicts - **Architecture:** PointDiT-B/16 (12 blocks, width 768, 12 heads, bottleneck patch embedding), made temporal: even blocks attend within a frame, odd blocks attend over all frames; the private 3D rotary embedding is sectioned (time | row | column); one residual adaptive-LayerNorm branch carries the physical frame time (`t_j - t_0` in frames). One generated channel: the released three-channel (XYZ) head is sliced to its Z column. - **Conditioning:** frozen DINOv3 ViT-B/16 features of every frame (layers 2, 5, 8, 11, concatenated: 3,072 channels per 16 x 16 patch). No raw RGB enters the Point network. - **Input:** a window of T RGB frames (trained on T = 1 to 25, evaluated with T = 21), height and width multiples of 16 at about 512 x 512 pixels of area (ten aspect buckets, never squashed; for example 336 x 784 for Sintel, 288 x 928 for KITTI), and strictly increasing integer frame indices starting at 0 (the frames' true indices; they set the rotary time and the physical-time branch). - **Output:** `[1, T, 1, H, W]`: each frame's own-camera depth Z. The depth is in the units of the window's gauge: a single scale shared by every frame of the window, the mean valid depth of the window's frames. The model predicts a contracted state (symmetric mip-NeRF 360 contraction); the repository decodes it, capping the state at 1.984375 (a depth of at most 64 gauge units). - **Metric-free depth.** The output is depth up to one unknown scale per window (and, in the benchmark, one fitted shift): it is not metric. To get depth in metres, multiply by a known scale. The evaluation recovers it the way Video-Depth-Anything does: one least-squares scale and shift per sequence, fitted in depth against the ground truth. Nothing inside the model predicts the scale. - **Sampling for evaluation:** one Euler step from Gaussian noise (the x0 head makes one step a one-shot prediction), `--base-steps 1`, on the logit-normal quantile grid the model was trained with (its native sampler is four steps, noise scale 2.0). ## Quick start without Civo Everything below uses only this folder, the project repository on GitHub, two public checkouts and files you request or download yourself. No path of the Civo cluster is on any command line. **1. Code.** `git clone git@github.com:NIRVANALAN/vgg_forecast_4d.git` (you need access to the repository) and check out `main`. Loading and inference need commit `2310eec5` or later. The evaluation flag `--data-root` (used below) is on the branch `docs/video-depth-pointdit-off-civo` until its pull request is merged; check that branch out for the evaluation. Run every command from the clone root with `export PYTHONPATH=$PWD/src`. **2. Environment.** The scripts were run in Python 3.12.13 with torch 2.9.0+cu130 (CUDA 13.0), numpy 2.4.6, omegaconf 2.3.1, hydra-core 1.3.4, einops 0.8.2, safetensors 0.8.0, opencv-python 4.11.0.86, imageio 2.37.4 with imageio-ffmpeg 0.6.0, scipy 1.18.0, pillow 12.3.0, matplotlib 3.11.1, accelerate 1.14.0, timm 1.0.28, transformers 5.14.1. The recorded rows used FlashAttention 2.8.3 (`fa2`); the scripts default to PyTorch's `sdpa` attention, which needs no extra package (pass `--attention-backend fa2` where it is offered if you installed flash-attn). A GPU is needed for evaluation and recommended for inference; loading runs on a CPU. **3. This folder.** `hf download yslan/dev-pointdit-video-depth --include "video-depth-pointdit/*" --local-dir `; the folder is `/video-depth-pointdit`, called `$FOLDER` below. `cd $FOLDER && sha256sum -c manifest.sha256` checks it. **4. DINOv3 ViT-B/16 (not included).** Request it at `https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/` (Meta's licence). The file is `dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth`, sha256 `73cec8be7427c8655ceced13ce62f6e20a1fa90d1b4d4a550df17a1144081a7c` (the scripts refuse another file). The code is `git clone https://github.com/facebookresearch/dinov3` (tested at `6876159a11b4df116f30f667f8c9888617df0751`). Both go to the scripts by flag: `--dino-weights --dinov3-repo `. **5. Load (CPU, about 15 s).** ```bash python $FOLDER/load_point.py $FOLDER # temporal_pointdit window_valid_mean_depth 32500 0.9996 140204928 ``` **6. Inference on your video (section below).** ```bash python $FOLDER/infer_video_depth.py $FOLDER --input clip.mp4 --out out/ \ --dino-weights --dinov3-repo ``` **7. Evaluation (one GPU).** Needs the benchmark data in the layout of the repository's evaluator (`monocular_depth.cohort_pairs`; the layout and the 21 frames per sequence are written out in the repository's `doc/handoff/video_depth_and_camera_pose_evaluation_for_external_models.md`, section 3.1) and a checkout of `https://github.com/NIRVANALAN/STream3R` at `7bdca8cf7ec206ff8b08c0941ad8286d72087d4f`, whose depth readers the evaluator uses (`STREAM3R_ROOT`): ```bash export STREAM3R_ROOT= bash $FOLDER/reproduce_evaluation.sh $FOLDER out/ --dino-weights --dinov3-repo \ --data-root [--datasets "sintel bonn kitti"] ``` It checks `manifest.sha256`, writes a temporary run directory (this folder's config with the three external paths replaced; the shipped config is not edited) and runs `eval/temporal_point/checkpoint_evaluation.py --task video_depth` per dataset with the aspect-preserving depth-fit protocol, EMA2, seed 0, `--base-steps 1`, `--depth-state-cap 1.984375`. **What reads what.** The released PointDiT-B 512 checkpoint is never read: the scripts set `model.point.weights: random`, which builds the same architecture, and all 216 tensors then come from the EMA2 file. The model after loading EMA2 is bit-identical to one built over the released checkpoint (checked: 216 of 216 tensors equal). The recorded training config keeps its Civo paths as history. Which keys still name one: | Config key (`train/resolved_resume_29500.yaml`) | Read by load / inference / evaluation? | How to override | |---|---|---| | `model.point.checkpoint` (the released PointDiT-B 512 file) | no: skipped (see above) | nothing to do | | `model.dino.weights` (DINOv3 file), `model.dino.repo` (DINOv3 checkout) | yes, by inference and evaluation | `--dino-weights`, `--dinov3-repo` | | the evaluation data root (not in the config: `monocular_depth.DATA_ROOT`) | yes, by evaluation | `--data-root` | | STream3R checkout (not in the config: default server path) | yes, by evaluation | environment variable `STREAM3R_ROOT` | | `data.track4d.sources[*].kwargs.ROOT`, `metric_scales_csv`, `index_cache_dir`, `camera_exclusion_manifest` roots | no: training data, not read by these scripts | none needed | | `train.output_dir`, `train.resume_from` | no | none needed | The last two rows are from reading the code (the evaluator reads only `data.track4d.frames`, the model, DINO and schedule sections); the off-Civo check below ran on a machine where those directories still exist, so it does not prove it. ## Run inference on your own video `infer_video_depth.py` takes a video file or a folder of frames and writes per-frame depth. It builds the model through the repository's builder (`build_point_model`), loads the EMA2 weights, computes the frozen DINOv3 ViT-B/16 features of the frames and samples with the repository's sampler exactly as the evaluation does: one Euler step from Gaussian noise, the contracted depth state capped at 1.984375, bf16 autocast. Run it from a clone of the repository (steps 1 to 4 of the Quick start), on one GPU (a CPU works but takes minutes per frame set): ```bash D="--dino-weights --dinov3-repo " python $FOLDER/infer_video_depth.py $FOLDER --input clip.mp4 --out out/ $D # first 21 frames python $FOLDER/infer_video_depth.py $FOLDER --input frames_dir/ --out out/ --max-frames 25 --stride 2 $D ``` - **Frames:** the first `--max-frames` (default 21, at most 25) frames at `--stride`, in one call. Each is resized, whole and without a crop, to the raster that keeps its aspect ratio at about 512 x 512 pixels in 16-pixel patches (for example 480 x 640 becomes 448 x 592; 436 x 1024 becomes 336 x 784), with `cv2.INTER_AREA`, as the benchmark does. The frame indices given to the model are `0, stride, 2 stride, ...`; they set its rotary time and physical-time input, so give the stride the clip really has between frames (training used strides 1 to 8). For a longer video, call it on successive chunks; the chunks are then separate gauges (one unknown scale each). - **Outputs** in `--out`: `depth.npy` ([T,H,W] float32 on the model raster, in units of the window's mean depth: one unknown scale for the whole window, not metric); `rgb.npy` (the resized input); `depth_NNN.png` (colour-mapped disparity, 1 / depth, inferno, normalized over the whole clip, as Video-Depth-Anything draws it); `depth.mp4` (input and colour map side by side); `info.json` (raster, frame indices, sampler settings, seed). - **Scale.** `depth.npy` is the model's output unchanged. With `--hfov-degrees ` the script also writes `depth_frame0_gauge.npy`: the same depth times the evaluator's frame-0 gauge factor (frame 0's mean point norm becomes 1; it needs a field of view). Either way the depth is known only up to one scale: to get metres multiply by one known scale, or fit one scale and shift per clip against a reference, as the benchmark does. - **Seed.** The sampler starts from Gaussian noise: `--seed` fixes it. `--seed-key 0::` uses the benchmark's noise key, which reproduces a benchmark row's call exactly. - **Weights it needs:** only this folder and the DINOv3 ViT-B/16 file and checkout (`--dino-weights`, `--dinov3-repo`, both required). `--attention-backend` is `sdpa` by default; `fa2` is the recorded run's. **Checked.** (1) Before the off-Civo flags were added, on one H100 (2026-10-07, repository commit `2310eec5`), the script run on the first 21 frames of Bonn `balloon2` and Sintel `alley_2` with the benchmark noise key (`--seed-key 0::`) scores, after one scale and shift per sequence in depth against the ground truth (the registered protocol), AbsRel 0.0970 on `balloon2` (recorded row: 0.0970) and 0.2377 on `alley_2` (recorded: 0.2368); delta<1.25 0.9480 and 0.6414 (recorded 0.9480 and 0.6420). The recorded rows came from an A100; the small differences are GPU and kernel numerics. (2) With the flags of this README, on one H100 (repository branch `docs/video-depth-pointdit-off-civo`), every external input copied to node-local `/tmp` and passed by flag (this folder, the DINOv3 file and checkout, a STream3R checkout, the Bonn frames), with `TRACK4D_ROOT` and `VGGT_OMEGA_ROOT` pointing at nonexistent paths: `load_point.py` printed `temporal_pointdit window_valid_mean_depth 32500 0.9996 140204928`; `infer_video_depth.py` ran on a folder of 21 Bonn frames and on an mp4 made from them (448 x 592 raster, `sdpa` attention); and `reproduce_evaluation.sh --datasets bonn` re-scored Bonn at AbsRel 0.060529 against the recorded 0.060454 (difference +7.4e-5; `sdpa` here, `fa2` in the recorded row). (3) A model built without the released checkpoint holds exactly the same 216 tensors as one built over it, after loading EMA2 (all equal, bit for bit). A full three-set re-score from a downloaded copy of this folder with the earlier scripts (`fa2`, Civo paths) gave Sintel 0.381766, Bonn 0.060523, KITTI 0.090556 against the recorded 0.381446, 0.060454, 0.090497. ## Recorded results Mean AbsRel over sequences (median over sequences in brackets); the first 21 frames of each sequence; the whole frame at the dataset's native aspect, scored on the ground-truth grid; one least-squares scale and shift per sequence, fitted in depth (`video_depth_anything_aspect_preserving_depth_fit`). EMA2, one step from noise. | | Sintel (13 sequences) | Bonn (5) | KITTI (13) | |---|---|---|---| | This model, 21-frame call | 0.381 (0.262) | 0.060 (0.054) | 0.090 (0.091) | | This model, raw weights (not shipped) | 0.383 | 0.060 | 0.100 | | This model, each frame its own one-frame call (per-frame fit) | 0.379 (0.254) | 0.061 (0.054) | 0.092 (0.090) | | VGGT-Omega (1B), 21-frame call | 0.145 (0.114) | 0.053 (0.042) | 0.038 (0.031) | | Per-frame PointDiT-B 512 (released), 21 frames | 0.394 (0.415) | 0.071 (0.053) | 0.118 (0.107) | Learning curve of the same recipe (EMA2, same protocol): update 30,000: 0.379 / 0.062 / 0.089; update 32,500: 0.381 / 0.060 / 0.090. Sintel's mean is carried by two sequences (`ambush_5` 1.85, `market_6` 0.57) whose single depth-space scale and shift is pulled by far pixels; the rank correlation of the raw prediction with the ground truth is high (0.84 and 0.97 on those two). KITTI's near third (closer than about 10 m) is under-predicted. ## Training summary - **Parent:** released PointDiT-B 512 mixdata EMA branch (`pointditb-512-mixdata-nodinov3-1d42aacf.pth`, sha256 `1d42aacfd6aaf97c8bd96b0ce5e23bd880cd01b2efbb2ac80ccf8876fde3034b`), Z channel sliced. - **Updates:** 32,500 in four segments; global batch 64 (4 A100-80GB x 16), AdamW (0.9, 0.95), constant learning rate 5e-5, gradient clip 1.0, bf16 autocast, EMA decays 0.9999 and 0.9996. - **Objective:** PointDiT's velocity loss with noise scale 2.0 plus 0.1 x a bounded relative depth term, per-sample mean over valid pixels. - **Data (last segments):** BEDLAM, Spring, vKITTI 2, OmniWorld-Game (depth within 20 m), Syn4D v1 / v2, TartanAir, MVS-Synth, DL3DV (reconstructed depth), Dynamic Replica, PointOdyssey, Kubric49 / 81, Waymo (LiDAR), Hypersim, ScanNet++, ScanNet, Mid-Air, WildRGB-D, SynthVerse and BlendedMVS; 65 % video windows, 15 % single frames, 20 % unordered multi-view sets; 1 to 25 frames per sample. - **Evaluation data is held out:** Sintel, Bonn and KITTI (video-depth benchmark sets) are not training sources. ## Limits - The error that remains is not fixable by more steps of this recipe: it plateaus at update ~30,000. - Output is scale-free; the gauge is the window's own mean depth, not frame 0's depth (the camera expert of the MoT normalizes translation by frame 0's depth: the two gauges must be reconciled before they are combined). - One step from noise is the evaluated sampler. The segment-4 offset diagnosis (update 27,000) found four and sixteen steps give the same depth offset as one step. ## Licence and third-party notes - The repository code is the project's; the PointDiT trunk code and the released PointDiT checkpoint come from `github.com/google-research/pointdit` (Apache-2.0 code; read the checkpoint's own terms before redistribution). - The weights were trained on datasets with their own licences, several research-only (for example Hypersim, ScanNet, ScanNet++, Waymo, DL3DV). Treat the weights as research-only until the owner chooses a licence. - DINOv3 (Meta) is under the DINOv3 licence and is NOT redistributed here. Request it at `https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/`. The model needs the ViT-B/16 LVD-1689M weights (`dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth`, sha256 `73cec8be7427c8655ceced13ce62f6e20a1fa90d1b4d4a550df17a1144081a7c`).