A closed loop multimodal auditing framework for enhancing artificial intelligence reliability in medical emergency response

Ensuring the reliability of AI-powered information retrieval systems is critical in high-stakes domains such as first aid. Traditional audit methodologies are often static, manual, and unimodal, failing to address the dynamic, time-sensitive, and multimodal requirements of these applications. To overcome these limitations, we introduce RADAR-FirstAid (Reliability-Aware Dynamic Auditing Framework), a closed-loop auditing pipeline — not a competing information system — for real-time, multimodal, and edge-aware auditing. Its key components include a Dynamic Query Simulator for generating criticality-weighted scenarios, a Multimodal Audit & Fusion Agent for verifying cross-modal consistency, a Differentiable Reliability Evaluator that employs LLM-based judges for automated assessment, and an Edge-Optimization Scheduler for efficient deployment. Applying the framework to a curated dataset of 1, 508 first-aid queries, we conduct a large-scale audit of Google’s AI Overviews (AIO) and Featured Snippets (FS), operated purely as black-box Systems under Audit. The audit reveals substantial reliability gaps: automated consistency scores of Φ cons=0.68 (FS) and 0.72 (AIO), safety/relevance scores of Φ safe=0.71 (FS) and 0.76 (AIO), and pronounced weaknesses on safety-critical “Binary” and “How-to” queries and on negative-sentiment queries where cautious tone matters most. Manual annotation on a 200-query subset validates the automated evaluation (Spearman ρ = 0.74 –0.87). Ablation studies confirm the essential role of each pipeline module, and geographical variance is minimal (SD $$<0.03$$ ). RADAR-FirstAid offers a scalable, automated, and reproducible audit instrument for AI-powered search systems in safety-critical domains.