SupplyQual: an approach for using customer reviews for evaluating last-mile delivery performance

As e-commerce platforms scale, the volume of unstructured customer feedback has outpaced the ability of logistics providers to extract actionable insights. Traditional evaluation methods which rely on explicitnumerical ratings fail to capture the nuances of last-mile delivery failures. We propose SupplyQual, a novel framework that transforms customer feedback into a structured knowledge graph, enabling sophisticated multi-hop queries via an agentic framework. By integrating Model Context Protocol (MCP)based access, our framework allows LLM-based agents to perform grounded natural language querying over this data. We evaluate our approach using a curated set of queries, demonstrating that the graph-based agentic system achieves a 6.3%-point improvement in faithfulness and a 12.7%-point increase in context precision over a standard RAG‑based baseline, with both differences statistically significant (p<0.05). These gains come with a measurable trade-off: because the framework prioritizes grounded, schema-constrained retrieval, it answers fewer queries fully and records lower answer correctness than free-text RAG. We therefore position SupplyQual as a high-faithfulness, auditable decision-support tool rather than a general-purpose question-answering system, and outline a hybrid graph–vector strategy to close the correctness gap. Our results suggest that SupplyQual provides a scalable and traceable method for automating qualitative assessments in high-growth, fragmented logistics environments like India.