Assessing the quality of large generative models for basic healthcare applications

Large generative models (LGMs) have significant potential for healthcare and medical science. While publications are growing exponentially, LGM studies lack quality and breakthrough findings. Research articles call for standardized approaches to ensure safe and effective integration of LGMs into clinical practice. Currently, the Moscow healthcare system is testing LGMs as tools for supporting medical decision-making, which has required development of specialized methods and techniques for assessing LGM quality. This paper presents two methods for assessing the quality of large generative models. Both methods are based on analysis of literature data (over 200 sources), results from comprehensive testing of 204 LGMs, and hands-on experience in assessing model quality using a sample of more than 12,000 cases. Designed for two main LGM application scenarios, the methods incorporate a dedicated approach to building test samples, tailored and validated questionnaires, testing methodologies, and unified requirements for the composition and structure of quality assessment outputs.