Real-time conversational voice agents, which combine speech-to-text (STT), large language model (LLM) based dialogue management, and text-to-speech (TTS) in a single streaming pipeline, are increasingly deployed in settings that demand human-comparable response latency. This paper reports architecture patterns and operational lessons from a production voice platform that serves inbound and outbound business calls over SIP telephony and web channels, built on a self-hosted open-source media server with stateless agent workers and per-agent selection of STT, LLM, and TTS providers. We organize the design space around four patterns: a streaming-first pipeline with a short call-setup critical path, a dedicated turn-taking subsystem, per-session isolation with independently scalable resource pools, and latency-aware LLM orchestration. For each pattern we state whether the platform adopted it, relied on a framework default, or did not adopt it. The platform uses voice-activity-detection-based turn handling without a semantic turn detector, does not use preemptive generation, and provides provider fallback only when a session is constructed. We document ten production incidents, in which the dominant failure mode was silent degradation (silence, dead air, or ignored configuration) rather than slow inference alone, and we report the measurements available from the deployment: a 33-turn single-call latency decomposition in which the end-of-turn delay, a dynamic quantity set largely by a configurable silence threshold (deliberately long in this interpreter configuration), exceeded LLM and TTS latency combined, measured effects of reasoning-effort settings on time to first token, and anecdotal outbound burst failures. No load test exists, and we state that plainly. The paper is intended as a structured, candid applied account for practitioners.
No takes yet. Share an insight, caveat, or question.
Altaf Rehman (2026) studied this question.