Lingo2Action: Fusing Semantic Risk and Perceptual Uncertainty for Adaptive 3D Value Maps

While LLMs such as GPT, LLaMA, and Gemini have achieved significant progress in language understanding and reasoning, their capability to act reliably within the physical world remains constrained. Existing approaches predominantly rely on static spatial representations, which fail to capture the semantic risks embedded within language instructions and the perceptual uncertainties inherent in real-world sensing. To address these limitations, this work introduces Lingo2Action, a unified semantic-driven robotic framework based on the LAM paradigm. This framework incorporates a Semantic-Perceptual Adaptive Field (SPAF) that dynamically integrates Semantic-Aware Parameterization (SAP) derived from the LLM with Risk-Aware Perception (RAP) generated by the perception module. Through this integration, the system jointly encodes task semantics, environmental risks, and perceptual confidence into an Adaptive Semantic Value Map (ASVM). The ASVM directs an enhanced context-aware path planner to generate safe and efficient trajectories. Empirical evaluations demonstrate that, within tabletop manipulation scenarios characterized by varying semantic risks and perceptual uncertainties, the Lingo2Action framework equipped with the SPAF engine consistently improves both operational safety and task success rates.