The Inaugural Flatiron Institute Cryo-EM Conformational Heterogeneity Challenge

Despite the rise of single particle cryo-electron microscopy (cryo-EM) as a premier method for resolving macromolecular structures at atomic resolution, methods to address molecular heterogeneity in vitrified samples have yet to reach maturity. With an increasing number of new methods to analyze the multitude of heterogeneous states captured in single particle images, a systematic approach to validation in this field is needed. With this motivation, we issued a challenge to the community to analyze two cryo-EM image particle sets of the thyroglobulin molecule with continuous conformational heterogeneity. The first dataset was a experimental, and the second was generated with a simulator, allowing control over the distribution of molecular structures in the particle images. This simulated dataset also enabled direct comparison between participants' submissions and the ground truth molecular structures and distributions. Participants were asked to submit 80 volumes representing the heterogeneous ensemble and estimate their respective populations in the image sets provided. Participation of the research community in the challenge was strong, with submissions from nearly all developers of heterogeneity methods, resulting in 41 submissions across both datasets. Submissions qualitatively exceeded expectations, with the molecular motions identified by methods resembling both each other and the ground truth motion. However, quantitatively assessing these similarities was a challenge in and of itself. In the process of assessing the submissions to this challenge, we developed several validation metrics, most of which require reference to the underlying ground truth volumes. However, we have also explored the use of metrics which do not necessarily reference ground truth. This is particularly apt for experimental datasets where ground truth is inaccessible. These approaches allowed us to assess the similarity and accuracy in volume quality, molecular motions, and conformational distribution of different submissions. These metrics and the efforts of all participants will help chart a path forward for the improvements of heterogeneity methods for cryo-EM and future challenges to validate these new methods as they continue to be developed by the community.