Prompting alone is insufficient for large language model-based clinical variant classification

Accurate clinical genetic variant classification is becoming increasingly challenging as data volume and knowledge complexity increase. Large Language Models (LLMs) could potentially automate evidence retrieval and assist variant interpretation. To systematically evaluate the capacity of general-purpose LLMs (Claude, ChatGPT, Gemini and Perplexity) to support clinical genetic variant classification. We analyzed 475 variants across 185 genes, with reference classifications established by expert variant scientists. Classification was performed by the four LLMs using a detailed structured prompting strategy and compared with a commercial non-LLM specialized platform. Performance was evaluated at the classification and criteria levels and stratified by various variant characteristics. Detailed prompting generally improved performance. Mean LLM classification concordance was 69.0% (70.2% precision and 68.2% recall), which was substantially lower than the 81.1% concordance (80.2% precision and 84.5% recall) achieved by the specialized commercial non-LLM platform tested. Concordance varied across variant types, with unadjusted mean concordance ranging from 87.2% for intronic variants to 56.0% for splice-region variants; adjusted associations were model-specific. At the criteria level, performance was highly criterion-specific: sensitivity exceeded 80% for BP7, PP3, PVS1, and PM2, but criterion-call precision and exact-strength agreement were substantially lower for several criteria. General-purpose LLMs used through direct or structured prompting demonstrated substantial limitations in clinical genetic variant classification compared with a specialized interpretation platform. These findings indicate that the common conversational approach of asking an LLM to classify a variant, even with detailed instructions, is insufficient for autonomous clinical use.