The Forgotten Middle Layer: Metadata-Driven Governance For Semi-Structured Big Data

Big data technologies enable advanced analytical capabilities, yet they produce numerous project failures because of a fundamental architectural deficiency. The paper reveals a "forgotten middle layer" in contemporary data systems, which arises from the implementation of schema-on-read data lakes in conjunction with semi-structured formats, such as JSON and XML. The complexity of modern data systems exceeds the capabilities of traditional governance models, which results in data becoming both untrustworthy and unusable. The proposed metadata-driven governance framework utilizes active metadata management to deliver data discovery, lineage, and quality, as well as policy enforcement capabilities. The paper demonstrates architectural patterns for this method while showing its implementation in Apache Atlas platforms and developing a step-by-step adoption framework. Metadata governance establishes structure and trust in semi-structured data environments, connecting raw infrastructure with effective analytics.