Few-Shot Learning With Manifold-Enhanced LLM for Handling Anomalous Perception Inputs in Autonomous Driving

With the widespread adoption of advanced driving systems (ADS), these systems must cope with increasingly complex driving scenarios. However, data-driven deep learning models heavily rely on large amounts of training data and may perform inadequately when faced with novel situations. Consequently, few-shot learning has become a key topic in the field of autonomous driving. Yet, few-shot learning demands models with strong generalization and extrapolation abilities and tends to be vulnerable to anomalous inputs. The perception modules in current ADS systems are not infallible, and there remains a small chance of inaccurate information being generated, which could potentially affect the decision-making process. To address this issue, we propose an innovative few-shot learning framework based on large language models (LLMs) that can comprehend context and make correct decisions despite the presence of anomalous inputs. This framework decouples the high-dimensional textual space of LLMs into a low-dimensional space tailored for autonomous driving, enabling commonsense reasoning within this space. By integrating this space with the textual space, we create a decision manifold that enables effective reasoning and decision-making processes. The framework also maintains an external self-correction database that continually updates experiences to guide manifold construction, facilitating continual Learning. Experimental results demonstrate our framework reduces collision rates by 14% compared to GPT-Driver when tested with anomalous data. This confirms enhanced safety in complex driving environments with perceptual irregularities.