ZIL: Zero-shot Image-to-LiDAR Registration

This repository hosts the public inference checkpoint for ZIL, a foundation model for zero-shot, non-synchronized image-to-LiDAR registration. The model estimates the camera pose in an unaligned LiDAR point cloud without dataset-specific fine-tuning.

Paper: ZIL: Zero-shot Image-to-LiDAR Registration

Implementation, training, preprocessing, and evaluation code live in the ZIL GitHub repository. This model repository contains artifacts only.

Try the interactive Gradio Space or visit the project page.

Usage

From a CUDA 12.4 environment, install PyTorch before installing ZIL so FlashAttention can use the installed torch during its build:

python -m pip install \
  torch==2.6.0+cu124 torchvision==0.21.0+cu124 \
  --index-url https://download.pytorch.org/whl/cu124
python -m pip install packaging==26.0 ninja==1.13.0 wheel==0.45.1 setuptools==69.5.1
python -m pip install --no-build-isolation "git+https://github.com/ZijunLi7/ZIL.git"

From a checkout of the ZIL repository root, use its bundled demonstration sample in examples/:

from zil import ZIL

model = ZIL.from_pretrained(device="cuda")
result = model.register(
    "examples/image.png",
    "examples/pointcloud.npy",
    "examples/intrinsics.json",
)
print(result.transform)

The first call downloads this repository's checkpoint to the standard Hugging Face cache; later calls reuse the cached file.

The returned matrix is the camera-to-LiDAR pose for column vectors: it maps camera-frame points into the LiDAR frame, with translation in metres. Input LiDAR points must use FLU axes: x forward, y left, z up. A .bin file is interpreted as float32 N x 4; only XYZ is used.

Files

• zil-inference.safetensors: model weights with the zil.pose.v1 metadata schema.
• config.json: input conventions and the public inference recipe.

Limitations

• The released implementation requires an NVIDIA CUDA GPU because its sparse-convolution backbone has no supported CPU path. FlashAttention is optional; PyTorch SDPA is used when that extension is unavailable.
• The model is intended for outdoor camera/LiDAR systems. Performance outside the training distribution should be independently validated.
• Pose quality depends on correct image intrinsics, metric point coordinates, and the documented axis convention.

License

The checkpoint is released under CC BY-NC 4.0, including non-commercial restrictions inherited from its VGGT-1B and Utonia initialization. Source-code licensing and all third-party notices are documented in the GitHub repository.