What question did this study set out to answer?

This study aims to establish a reliable methodological framework for applying large language models in engineering management research through Context Modeling.

June 20, 2026

LLMs as Research Instruments for Problems in Engineering Management: A Context Modeling Framework

Puntos clave

This study aims to establish a reliable methodological framework for applying large language models in engineering management research through Context Modeling.
Introduced Context Modeling methodology to structure the use of LLMs as research instruments.
Constructed a synthetic population of 3,154 agents based on public survey data.
Evaluated LLM outputs for fidelity, robustness, consistency, and reproducibility across policy alternatives.
Overall Fidelity exceeds 98%; Robustness ranges from 87% to 91%; Consistency from 87% to 93%.
High subgroup Fidelity (98%–99%), with most Robustness and Consistency values exceeding 90%.
Localized reductions in robustness reflect context-dependent sensitivity, not instability.

Resumen

In engineering management research, large language models (LLMs) are increasingly used as analytical tools, yet their application often lacks a systematic and scientifically verifiable methodological foundation. This study introduces Context Modeling, a structured methodology that transforms LLMs from heuristic tools into scientifically verifiable research methods. The framework provides a unified process for prompt structuring, semantic constraint design, and rigorous output evaluation within a traceable dual-layer architecture. The structural layer assesses reliability through fidelity, robustness, consistency, and reproducibility, while the mechanistic layer applies feature attribution to reveal decision logic and feature dependencies. The framework is demonstrated using the Rethinking I-94 infrastructure redevelopment case led by the Minnesota Department of Transportation. Based on public survey data, a synthetic population of 3,154 agents is constructed to simulate attitudes toward two contrasting policy alternatives. Results show that the LLM reproduces stable structural distributions across scenarios: overall Fidelity exceeds 98%, Robustness ranges from 87% to 91%, Consistency from 87% to 93%, and Reproducibility exceeds 60% across both policy alternatives. At the subgroup level, Fidelity remains high (approximately 98%–99%), and most Robustness and Consistency values exceed 90%, with localized reductions under one scenario reflecting context-dependent semantic sensitivity rather than structural instability. These findings indicate that Context Modeling enables structurally stable and mechanistically interpretable LLM-based decision patterns grounded in real data. Through quantitative validation and feature-level interpretation, the results indicate that the proposed Context Modeling framework can provide a reliable research method in the field of engineering management based on LLMs and problem context simulation.

Preguntar a la IA

Me gusta

Guardar