A critical perspective on finite sample conformal prediction theory in medical applications

Key Points

• The study critically evaluates the validity of finite sample conformal prediction theory in medical contexts, particularly focusing on calibration issues.
• Analyzed the performance of marginal conformal coverage under various calibration workflows.
• Examined histology classification to assess under-coverage linked to small calibration sets.
• Identified the need for larger calibration sets to improve conditional guarantees.
• Marginal conformal coverage often misleads when using one-time calibration workflows.
• Limited calibration data resulted in significant variability in conditional coverage rates.
• Frequent under-coverage was observed in histology classification with small calibration sets.