Remote Sensing, vol. 15, no. 18, 2023 · MDPI

Deep LiDAR-Radar-Visual Fusion for Object Detection in Urban Environments

Yuhan Xiao¹, Yufei Liu¹, Kai Luan¹, Yuwei Cheng^2,3, Xieyuanli Chen¹, Huimin Lu¹

1 National University of Defense Technology
2 Tsinghua University 3 ORCA-TECH

Bird's-eye view of a fused scan: 0.1 m cells over 40 × 40 m, points coloured by height, orange for detections. Ego vehicle at the bottom.

Abstract

Robust sensing and accurate object detection are crucial for autonomous driving in cities. We propose LRVFNet, a multi-modal sensor fusion network that combines LiDAR, mmWave radar and visual data through a deep multi-scale attention architecture, and show that fusing complementary modalities improves both accuracy and robustness of 2D detection in urban scenes.

The network has three parts: a backbone that generates distinct features from each sensor modality, a fusion module that combines those features through attention, and a pyramid module that reasons about objects at different scales. Camera data alone degrades under poor lighting and weather; LiDAR provides precise geometry but suffers in rain, fog and dust; mmWave radar penetrates those conditions but is sparse and noisy. Fusing all three addresses the occlusion and viewpoint limits of a single sensor. Experiments on the public VoD and Flow datasets show consistent gains over state-of-the-art baselines, and the method holds up when one modality degrades.

Architecture fig. 2

A nine-channel tensor enters the network: three channels of rectified RGB, three of LiDAR bird's-eye view and three of radar bird's-eye view. Two branches extract features in parallel, a shared attention gate merges them at every scale, and a pyramid head predicts small, medium and large objects.

LRVFNet pipeline: image and BEV inputs, two feature extractors, feature fusion, FPN prediction — Pipeline overview — two-branch feature extraction, attention fusion, pyramid prediction.

Two-branch backbone

ELAN blocks for the image, CSPDarknet53 residual stages for the two BEV inputs, each producing features at three scales.

Feature fusion

Per-modality self-attention followed by a shared global-attention gate, applied at every scale so noisy radar points are attenuated rather than trusted.

Pyramid network

A PANet-style neck with SPPCSPC widening the receptive field, then three heads for small, medium and large objects.

The two backbones: ELAN for the image, CSPDarknet53 for the BEV inputs — Backbones — ELAN on the image branch, CSPDarknet53 on the BEV branch.

Input representation fig. 1

Point clouds are projected onto a 0.1 m grid over X ∈ [0, 40] m, Y ∈ [−20, 20] m and Z ∈ [−2, 0.5] m. Each cell takes the maximum height of the points that fall into it, normalised to 0–255, which keeps the representation compact enough for mature 2D detection machinery while preserving the geometry that matters.

RGB image, LiDAR BEV and radar BEV over three consecutive frames — Image, LiDAR BEV and radar BEV across consecutive frames; the radar view is far sparser, which is why fusion matters.

Results

AP is reported at IoU thresholds of 0.35 and 0.50. Adding LiDAR on top of radar is the largest single step; the attention fusion adds the rest.

View-of-Delft, on request
Modality	Method	AP³⁵	AP⁵⁰
Image	YOLOv4	31.06	29.78
Image	YOLOv7	83.74	80.65
Image + radar	RISFNet	66.78	45.55
Image + radar	LRVFNet	88.30	85.03
Image + LiDAR + radar	RISFNet	76.72	54.23
Image + LiDAR + radar	LRVFNet	89.37	86.50

FloW, on request
Modality	Method	AP³⁵	AP⁵⁰
Image	Faster R-CNN	77.35	57.58
Image	YOLOv4	78.46	57.04
Image	EfficientDet	78.62	58.52
Image	FCOS	68.71	58.56
Image + radar	CRF-Net	79.63	57.74
Image + radar	Li et al.	85.28	64.64
Image + radar	RISFNet	90.05	75.09
Image + radar	LRVFNet	91.12	78.46

Qualitative comparison on VoD: YOLOv4, RISFNet, LRVFNet and ground truth — Qualitative comparison on VoD — image-only and radar-only baselines miss the occluded objects that the fused model recovers.

Running it

The environment is pinned with uv: Python 3.10 and PyTorch 1.13.1 built for CUDA 11.7. Pretrained CSPDarknet53 backbone weights come from the Hugging Face Hub and are fetched automatically when pretrained=True.

# clone and build the environment
git clone https://github.com/nubot-nudt/LRVFNet.git
cd LRVFNet && uv sync

# train, or run inference on one frame
uv run python train.py
uv run python infer.py

A synthetic dataset generator is included, so the whole pipeline can be checked end to end before a real dataset is in place.

Citation

@article{xiao2023lrvfnet,
  title   = {Deep LiDAR-Radar-Visual Fusion for Object Detection
             in Urban Environments},
  author  = {Xiao, Yuhan and Liu, Yufei and Luan, Kai and
             Cheng, Yuwei and Chen, Xieyuanli and Lu, Huimin},
  journal = {Remote Sensing},
  volume  = {15}, number = {18}, pages = {4433}, year = {2023},
  doi     = {10.3390/rs15184433}
}