Visual 3D Spatiotemporal Fields‐Driven Obstacle Avoidance Using Model Predictive Control for Nursing Robots in Unstructured Environments

ABSTRACT Monocular detectors typically provide 2D features, which limit a nursing robot's ability to perceive and adapt to changes in complex 3D environments. To address this challenge, this paper proposes a monocular 3D object detection framework capable of estimating objects' 3D locations, sizes, velocities, and predicted trajectories. The framework employs a Nursing Robot Convolutional Neural Network (NRCNN) to generate 2D bounding boxes for objects, followed by a 2D‐pixel‐to‐3D‐world coordinate transformation algorithm to recover their 3D spatial positions. Subsequently, an ID‐assignment‐based tracking module associates detections across sequential D435i video frames, enabling prediction of objects' spatiotemporal dynamics. To enhance 2D detection accuracy, a multibranch efficient channel attention (MECA) module is embedded within a cross‐stage partial network in the backbone of the NRCNN detector. This design enhances interchannel and cross‐layer information interactions to adaptively capture the cross‐channel and cross‐layer dimensional features. It departs from the traditional approach of improving feature extraction solely through increasingly deep dilated convolutional network structures, thereby enabling more effective feature extraction. Additionally, a bi‐directional channel spatial feature fusion pyramid network (Bi‐CSFFPN) is integrated into the neck of the NRCNN detector to fuse multi‐level, cross‐channel, and spatial features. This approach overcomes the limitation of conventional feature fusion networks that only fuse features along a single dimension and results in substantial performance improvements. Using outputs from the monocular 3D object detector, a dynamic lattice map is constructed that integrates real‐time object volume, position, and velocity information. This map allows the nursing robot to plan the shortest feasible path from its initial position to the target location while avoiding moving obstacles via a model predictive control (MPC)‐based trajectory tracking controller. Extensive experimental results demonstrate that the proposed MECA‐CSP and Bi‐CSFFPN modules significantly improve NR‐CNN detection performance and enhance downstream tasks, including 3D object localization and dynamic lattice‐map‐based MPC obstacle avoidance control.