Are experts in agreement? A comprehensive study on consensus in echocardiographic view identification

Introduction Accurate identification of echocardiographic views is crucial for quantitative assessment and clinical decision-making. While this task is often considered routine, variability in image quality and expertise introduce discrepancies. Prior studies have noted these factors but have not systematically quantified their impact across a large cohort of experts [1]. Purpose This study aims to rigorously evaluate the performance of clinical experts in classifying echocardiographic images, considering image quality and experience level. Methods A total of 1663 sequences from 100 patient studies (mean age 60 ± 13.8 years; 60% male) were classified by 22 experts (experience: 1–14 years) into 13 views grouped by projection: apical (2-, 3-, 4-, 5-chamber, and other); parasternal short axis (apex, papillary muscles/mitral valve, great vessels, and other); parasternal long axis; subcostal; suprasternal; and other. The perceived image quality of each sequence was rated on 5-point Likert scale (1 = lowest, 5 = highest) across five categories: sharpness, absence of artifacts, acquisition plane, contour visibility, and validity for clinical measurements. Performance was assessed as inter-rater agreement and measured using Fleiss' Kappa (FK) and percentage agreement (PA) [2,3]. Global FK was calculated at both projection and view level. The impact of image quality on relative observed agreement and FK was assessed for each quality category by grouping sequences based on their mean quality score for that category. PA for each expert was computed at projection level to analyse the effect of experience level. Results Overall FK was 0.94 and 0.87 for projection and view classification, respectively, indicating a high level of agreement among experts, though not ideal. Regardless of the quality category, low-quality scores were associated with dispersed and compromised performance (FK: 0.50 to 0.62), while higher quality scores led to near-perfect agreement (FK ~1) in view classification (Figure 1). No relevant differences in classification performance were observed across quality categories. Experience level did not affect agreement in projection classification, except for the 'other' projection class, suggesting that the ability to identify atypical sequences was proportional to experience (Figure 2). The lack of differences in global performance by experience may explain why this task is often assumed to be routine. Conclusion This work provides a large-scale systematic quantification of view identification performance and demonstrates that: (1) inter-rater agreement is high, albeit not absolute, with experience playing a key role only in identifying atypical views; and (2) low-quality images, regardless of the specific deficiency, significantly reduces classification reliability. This challenges the assumption that view identification is straightforward and underscore the need for improved standardization in clinical practice.