Multi-Tier DLQ Architecture for Fault-Resilient Data Consistency in Distributed Near/Real-Time Systems

Distributed real-time data ingestion pipelines operating at enterprise scale must handle a continuous spectrum of failure modes: from upstream schema violations and network-level transient faults, to sink-specific write failures and orchestration-layer resolution anomalies without sacrificing throughput, data consistency, or operational observability. Existing literature and industry practice treat the Dead Letter Queue (DLQ) as a passive overflow mechanism: a place to discard non-processable records. This paper challenges the framing and presents a production-validated architecture in which the DLQ is implemented as an active, checkpointed, schema-enforced, and metric-observable sub-pipeline operating in coordinated execution with the primary data path. The architecture supports three ingestion modes: real-time (RT), Near-Real-Time (NRT) micro-batch orchestration, each with mode-appropriate DLQ strategies that preserve full Kafka lineage metadata (topic, partition, offset, timestamp) sufficient for deterministic record replay with offset-level precision. We formalize a five-class error taxonomy for streaming pipelines, introduce a dual-path DLQ architecture that eliminates recursive routing, characterize the delivery guarantee semantics of each DLQ path, and propose a DLQ-safe offset commit protocol for N/RT ingestion that prevents silent record loss under arbitrary component failures. An observability model grounded in Stream Processing Engine (SPE) derives DLQ rate as a first-class streaming metric, demonstrating that sustained DLQ rate elevation is a reliable leading indicator of upstream schema drift. Empirical observations from a production deployment serving enterprise-scale financial data ingestion workloads validate the architectural claims and quantify the operational benefits.