Enhancing Analytic Hierarchy Process Modelling Under Uncertainty With Fine‐Tuning LLM

ABSTRACT Given that decision‐making typically encompasses stages such as problem recognition, the generation of alternatives, and the selection of the optimal choice, Large Language Models (LLMs) are progressively being integrated into tasks requiring the enumeration and comparative evaluation of alternatives, thereby promoting more rational decision‐making frameworks. Analysing the extent to which LLMs exhibit meaningful performance at each stage of the decision‐making process has thus become a critical area of inquiry. In particular, LLMs hold the potential to identify latent relationships within contextual information and data related to the problem domain. This capability enables them to propose novel evaluation criteria or alternatives that may otherwise be overlooked by human designers. This study seeks to advance the modelling and evaluation of the analytical hierarchy process (AHP), a widely utilised multiple criteria decision making (MCDM) method, by leveraging LLMs. To achieve this, a methodology was developed for constructing AHP models using LLMs fine‐tuned with domain‐specific documents. The performance of the proposed methodology was assessed by evaluating the extent to which its outputs aligned with reference hierarchies and criteria created by human experts under predefined AHP frameworks. Additionally, the study examined the model's efficacy in generating complete AHP hierarchies and criteria in scenarios where these were not predefined. For empirical validation, the proposed methodology was applied to assess and improve the management performance of six‐sector agricultural enterprises. Comparative analysis of the LLM‐based AHP results with human expert evaluations was conducted to determine the validity and robustness of the approach. The findings provide insights into the potential of LLMs to contribute to structured decision‐making and enhance the application of MCDM methods.