ZIL: Zero-shot Image-to-LiDAR Registration

Abstract

Image-to-LiDAR registration estimates the six-degree-of-freedom camera pose of an image with respect to a LiDAR point cloud. Existing methods commonly assume synchronized inputs and train a separate model for each dataset, which limits their use with distant frames and unseen sensor systems. We introduce ZIL, a foundation model for zero-shot, non-synchronized image-to-LiDAR registration. ZIL combines Vision and Point Transformers with global cross-modal attention, harmonizes heterogeneous sensors through camera-intrinsic and LiDAR vertical-axis normalization, and learns an auxiliary per-pixel 3D prediction task without additional annotation. Trained on seven public datasets, one ZIL checkpoint achieves sub-metre mean translation error on all five in-domain and zero-shot benchmarks.

Method

ZIL estimates the camera-to-LiDAR transform T_L←C ∈ SE(3) from a partially overlapping RGB image and point cloud captured at different time steps.

ZIL architecture with camera and LiDAR normalization, dual encoders, global attention, pose head, and point head — ZIL uses dual encoders, modality and register tokens, global cross-modal attention, and complementary pose and point heads.

01

Geometric normalization

Images are mapped to a canonical virtual camera, while the LiDAR vertical origin is shifted to the lowest observed point. These operations reduce ambiguity from camera intrinsics and sensor mounting height.

02

Global cross-modal attention

Vision and Point Transformers produce image and LiDAR tokens. A shared decoder lets every image token attend to all LiDAR tokens—and vice versa—while preserving 2D and 3D positional structure.

03

Dual-head learning

The pose head regresses translation and rotation. A complementary point head predicts image-pixel coordinates in the LiDAR frame, adding geometric supervision without new manual labels.

Results

All values use one ZIL model in the paper’s non-synchronized protocol. RTE is mean relative translation error, RRE is mean relative rotation error, and RR is registration recall for RTE < 2 m and RRE < 5°.

Benchmark	Setting	RTE (m) ↓	RRE (°) ↓	RR (%) ↑
KITTI	In-domain	0.39 ± 0.46	1.08 ± 2.95	97.5
nuScenes	In-domain	0.57 ± 1.03	0.88 ± 2.17	95.8
KITTI-360	Zero-shot	0.53 ± 0.81	1.93 ± 8.76	96.9
Lyft	Zero-shot	0.63 ± 0.60	0.77 ± 0.38	98.3
PandaSet	Zero-shot	0.67 ± 0.40	1.60 ± 0.36	98.6

Interactive qualitative visualization

Drag to orbit, scroll to zoom, and compare the ZIL camera pose with ground truth in the reference LiDAR cloud.

Reference LiDAR Ground truth ZIL prediction

KITTI

nuScenes

KITTI-360

PandaSet

*The visualization samples are provided solely for demonstration purposes and non-commercial research.

Use ZIL

In a CUDA 12.4 environment, install PyTorch first, then ZIL:

python -m pip install torch==2.6.0+cu124 torchvision==0.21.0+cu124 \
  --index-url https://download.pytorch.org/whl/cu124
python -m pip install packaging ninja wheel setuptools
python -m pip install --no-build-isolation "git+https://github.com/ZijunLi7/ZIL.git"

Model weights download automatically on first use. Full setup guide

Source code Open demo

Python API

from zil import ZIL

model = ZIL.from_pretrained(device="cuda")
result = model.register(
    image="image.png",
    pointcloud="pointcloud.npy",
    intrinsic="intrinsics.json",
)
print(result.transform)

Citation

arXiv:2609.22716

@article{li2026zil,
  title={ZIL: Zero-shot Image-to-LiDAR Registration},
  author={Li, Zijun and Sun, Xiaotian and Shen, Xuelun and Dai, Yao and Ao, Sheng and Shi, Yangyang and Engel, Jakob and Cai, Zhipeng and Wang, Cheng},
  journal={arXiv preprint arXiv:2609.22716},
  year={2026}
}

License

ZIL-original source is Apache-2.0. Incorporated source retains its upstream license. The released inference checkpoint is CC BY-NC 4.0 and subject to the model-weight notices in the repository.