Dual-Proposal Camera-LiDAR Fusion and Two-Pass 3D Multi-Object Tracking with SE(3) Ego-Motion Compensation

We present MultiObject, a modular, hardware-agnostic 3D multimodal perception and multi-object tracking (MOT) stackdesigned to bridge offline autonomous driving benchmarks (nuScenes) and live embedded robotic platforms (ZED stereo camera + Livox3D LiDAR on ROS 2). Multimodal perception pipelines for small autonomous vehicles face three competing operational constraints: (1)preserving high 3D recall under single-sensor occlusion or sparse far-range LiDAR returns, (2) suppressing false-positive “ghost braking”triggers without inducing multi-frame track birth lag, and (3) maintaining zero coupling between core geometric/tracking algorithmsand dataset-specific devkits. Our central architectural contribution is a decoupled Dual-Proposal Camera–LiDAR Fusion andTwo-Pass 3D Tracking Spine. In the detection plane, 10-sweep aggregated CenterPoint (Pillar02) 3D LiDAR proposals operatein parallel with YOLOv11 2D camera detections that are lifted into 3D LiDAR coordinates via pinhole frustum back-projection,15th-percentile ground-plane stripping, and DBSCAN foreground clustering. Proposals are unified through a priority Hungariansolver governed by class-conditional Bird’s-Eye-View (BEV) distance gating rather than brittle 3D IoU gating, overcoming monoculardepth and orientation variance. In the temporal plane, MultiObjectTracker implements a ByteTrack-inspired two-pass 3D Kalmanfilter with explicit SE(3) ego-motion compensation (Rt−1→t , tt−1→t ) and per-class calibrated confidence gates. Evaluated across all404 keyframes (10 scenes) of nuScenes v1.0-mini, our root-cause diagnostic and architectural refactor elevates CAM_FRONT 3D IoUrecall from 32.5% to 88.2% (93.8% under the standard 2.0 m BEV gate for visible objects), achieves 100.0% (20/20) physicalcorrespondence accuracy on fused proposals, reaches a 91.3% 3D F1-score on cars (84.2% overall across all 8 classes), and delivers79.2% Car MOTA / 90.5% Car Tracking F1 / 80.3% Overall IDF1 at a mean frame latency of 19.74 ms (∼ 50.7 FPS).