Evaluating the Prototype of a Clinical Decision Support System in Primary Care: A Qualitative Study (Preprint)

BACKGROUND In the project SATURN (Smart physician portal for patients with unclear disease) the prototype of a clinical decision support system based on artificial intelligence is being developed specifically for primary care in Germany. It aims to reduce diagnostic uncertainty in cases of unclear and rare diseases and focuses on three medical fields. A user-centered design approach is applied for prototype development and evaluation. OBJECTIVE This study explores the usability of a high-fidelity prototype. Aspects of user experience like the subjective impression, satisfaction, and improvement requests are also investigated. METHODS Five general practitioners participated in the evaluation of the prototype which consisted of (1) a remote think-aloud test, (2) a post-session interview, and (3) a survey with the System Usability Scale. All three parts were consecutively carried out in individual remote sessions. During the think-aloud tests, which were video- and audiotaped, the participants verbalized their thoughts and actions and had to solve several tasks which were based on a primary care case vignette. Remarkable observations were logged, transcribed with quotes, and analyzed for usability problems and positive findings. All observations and interview responses were deductively assigned to the following categories: (1) Content, (2) Comprehensibility, (3) User-friendliness, (4) Layout, (5) Feedback, (6) Navigation. Usability problems were described in detail and solutions for improvement proposed. Median and total scores were calculated for all questionnaire items. RESULTS The evaluation detected both strengths and areas for improvement. Key issues identified were content-related limitations, such as the inability to enter unlisted symptoms, medications, and examination findings in the dropdown menus. Participants also found the terminology for laboratory values did not match their day-to-day vocabulary, as common abbreviations were not recognized. Suggestions for improving the content of the system were also made and included adding symptom duration, weighting symptoms, and incorporating hereditary factors. Another key issue was a lack of user-friendliness concerning the time required to input medication plans and lab values. This aspect was criticized for being cumbersome, with participants expressing a need for faster data entry, potentially through direct imports from practice management systems or laboratory files. Despite these challenges, participants praised other aspects of user-friendliness (use of stored diagnoses and symptoms) and navigation (top navigation bar), and particularly liked the clear and well-structured layout. Overall, the SATURN prototype was deemed useful and promising for future clinical use, despite the need for further refinements, particularly in the areas of data entry. CONCLUSIONS The usability evaluation methods combined proved to be location independent and easy to use, and were apt to detect usability problems in detail. Technically demanding user requirements, such as direct data transfer from the practice management system and entry options that require complex data models were beyond the scope of this project. However, they should be considered in future development projects. CLINICALTRIAL not applicable