Automated Evaluation of Expert-Like Discussion Patterns in Large Language Models for Intertextual Analysis

Abstract Automated systems capable of generating human-like text have become increasingly relevant in fields requiring complex intellectual synthesis. Intertextual analysis, which involves identifying and integrating references from prior works into new content, presents a unique challenge for automated systems due to the complexity and depth of the reasoning required. The concept of employing models to simulate expert-level intertextuality represents a novel approach, offering significant advantages in terms of efficiency and accuracy, especially in academic and literary contexts. This study evaluated the performance of Mistral Large, an advanced language model, in generating expert-like intertextual responses across diverse prompt types. The model demonstrated strong capabilities in maintaining thematic consistency, structural coherence, and citation accuracy, particularly when tasked with complex intellectual prompts. Results revealed that while the model successfully replicated expert discussion patterns and intertextual references, it was limited in generating novel interpretations or critical insights beyond the scope of its pre-trained corpus. Overall, the findings highlight both the potential and limitations of automated intertextual analysis, providing valuable insights into the future integration of such systems in academic writing and intellectual content generation.