Context: The digital transformation of scientific literature relies heavily on the creation of structured Knowledge Graphs (KGs). Transitioning dense research papers into machine-readable data requires mapping complex text to strict curation templates for comparability. However, manual curation is a highly demanding and time-consuming bottleneck that requires specialized domain expertise and cannot scale with the growing volume of publications. While Large Language Models (LLMs) offer rapid extraction capabilities, applying them in a fully automated, “black-box” manner often results in hallucinated evidence, loss of structural context, and untraceable answers, making them unreliable for rigorous scientific documentation. Objective: To overcome these limitations, this thesis proposes a steerable Human-in-the-Loop (HITL) extraction framework built upon a flexible, modelagnostic architecture to prevent vendor lock-in. The primary objective is to develop an intelligent helper tool that assists human curators in finding specific values within complex documents. Rather than attempting to fully automate data entry, this approach ensures that the human expert remains firmly in control. It utilizes the AI to accelerate the search process while empowering the curator to seamlessly accept, refine, or reject generated suggestions and actively steer the model’s reasoning through natural language feedback. Method: To structure the curation task, the proposed method utilizes predefined, hierarchical questionnaires derived from strict scientific schemas. Rather than allowing the AI to generate unstructured text, these questionnaires act as a rigid framework, constraining the model to locate and extract specific, targeted data points. To generate highly accurate suggestions for these fields, the framework employs targeted context engineering, dynamically supplying the LLM with relevant source text, structural awareness of the schema (such as parent and sibling question relationships), and iterative user feedback. To resolve the lack of traceability in standard LLMs, the method introduces a multi-stage spatial mapping pipeline paired with a two-pass hybrid search algorithm. This mathematically anchors the AI’s abstract semantic output back to exact geometric nodes within the physical PDF document. Finally, to rigorously validate the underlying logic, an automated, headless evaluation pipeline was developed to benchmark the AI’s zero-shot extraction performance against the KG-EmpiRE dataset—a gold-standard collection of manually annotated literature—by applying standard evaluation metrics. Results: The empirical evaluation revealed that dynamically injecting structural context significantly improves the LLM’s logical reasoning, particularly for extracting dependent variables. However, fully autonomous extraction faced severe bottlenecks due to inherent model biases—such as affirmative biases in boolean questions, semantic confusion in single-select variables and “lazy generation” in multi-select arrays—as well as strict automated evaluation penalties for text length. Crucially, when evaluating the model’s ability to generate multiple viable extraction pathways, the context-aware pipeline successfully placed the correct answer within its top three suggestions in 77.9% of all interactions. Conclusion: The findings demonstrate that fully automated LLM pipelines remain fundamentally unsafe for rigorous structured literature curation due to semantic ambiguities and formatting unpredictability. However, the high Top-3 accuracy strongly validates the proposed architecture. By framing the AI as a targeted “evidence hunter” rather than an autonomous decision-maker, the curator-focused HITL assistant effectively neutralizes foundational model flaws. This collaborative workflow ensures strict scientific rigor while drastically accelerating the manual data curation process.
Sushant Aggarwal (Thu,) studied this question.