Subjectivity in paediatric urological imaging: a European Association of Urology‐Young Academic Urologists (EAU ‐ YAU) reliability study with consensus datasets

The diagnoses for congenital anomalies of the kidney and urinary tract can be subjective on radiographic imaging. For example, the severity of VUR is determined on voiding cystourethrography (VCUG) and has shown historically poor agreement [1]. Similarly, the diagnostic suspicion for posterior urethral valves (PUV) is dependent on the appearance of the urethra and bladder on VCUG, and the severity of paediatric hydronephrosis is determined by the appearance of the renal calyces on ultrasonography [2, 3]. Both diagnosis and severity assessment can influence the clinical management for children in each of these conditions [3]. In addition to clinical care, these subjective outcomes are routinely used for prediction in modelling and artificial intelligence studies, which raises validity concerns inherent to these models [4]. While objective diagnostic metrics would be ideal, this is not in keeping with current clinical practice standards, and individual clinician decision remains the ‘gold standard’. In this work, we performed a reliability study across three paediatric urological conditions that require radiographic assessment and provide a consensus dataset to inform future model development and education. We first performed a reliability study on publicly available radiographic images for (i) VUR grading (Grades I–V), (ii) diagnosis of PUV (yes vs possibly vs no), and (iii) hydronephrosis severity assessment (Society for Fetal Urology [SFU] Grade I–IV). The classification framework was determined based on clinical usability and current standards, and for PUV where a standardised staging is unavailable, categorisation labels were reached through group discussion. The aim of this study was to assess the inter- and intra-rater reliability of radiographic image assessments between paediatric urologists with regards to each condition. Real-world cases of each of these conditions were extracted from publicly available sources and Kabir et al. [5] to a convenience sample. Within each of the three datasets, randomly selected images were repeated for intra-rater statistics. The image order was randomised for each rater. A total of 10 fellowship-trained paediatric urologists independently assessed each dataset and the ratings were tabulated. Inter-rater agreement was determined using percentage agreement with consensus decision (determined from the mode of choices), Kendall's W (to account for near errors), and Fleiss’ Kappa statistic, which was deemed the most clinically relevant. Intra-rater agreement was determined using percentage agreement with self and intra-class coefficient (ICC). Thresholds for agreement in each statistic were provided. The CIs were calculated from bootstrapping of 1000 iterations, and all analysis was performed on R, version 4.3.2 (R Foundation for Statistical Computing, Vienna, Austria). After completion, all results were pooled into consensus datasets, which included the image and labelled consensus decision with measure of certainty and provided within a public data repository. The inter- and intra-rater statistics for each condition are provided in Table 1. For VUR, 61 unique and eight repeat images were presented. The agreement with consensus was 65% and Fleiss’ Kappa was 0.34 (95% CI 0.26–0.40), which represents fair agreement. The median (interquartile range [IQR]) percentage of raters agreeing on the grade was 60% (50–70%) and four images (7%) had complete agreement. The intra-rater ICC was 0.89 (95% CI 0.84–0.93), which is near-perfect and 72% agreement with self. For PUV, 62 unique and eight repeat images were presented. The agreement with consensus was 68% and Fleiss’ Kappa was 0.25 (95% CI 0.15–0.32), which represents fair agreement. The median (IQR) percentage of raters agreeing on the likelihood of PUV was 70% (50–80%) and nine images (15%) had complete agreement. The intra-rater ICC was 0.90 (95% CI 0.86–0.95), which is near-perfect and 92% agreement with self. For hydronephrosis, 41 unique and nine repeat images were presented. The agreement with consensus was 80% and Fleiss’ Kappa was 0.51 (95% CI 0.41–0.58), which represents strong agreement. The median (IQR) percentage of raters agreeing on the grade was 80% (65–90%) and six images (15%) had complete agreement. The intra-rater ICC was 0.77 (95% CI 0.62–0.86), which is strong and 68% agreement with self. The consensus dataset is provided in https://sites.google.com/view/yau-radar/consensus-datasets. The diagnostic standard for radiographic images in paediatric urology has been a long-standing concern. In clinical practice, the grade or imaging is used as an adjunct to other clinically relevant variables to guide decisions. However, for education and model development purposes, a ‘gold standard’ is necessary to ensure model predictions are accurate and generalisable as models trained on subjective outcomes are more likely to reflect subjectivity and practice variation [6]. This study demonstrates concerns in the agreement among paediatric urologists for three paediatric conditions. The cause for this is likely practice heterogeneity, but it further highlights the need for more objective and standardised approaches to diagnosis and assessment. Our results for VUR inter-rater agreeability are in line with previous literature with concerning rates of consensus disagreement [7]. In PUV, there is no true reference standard on VCUG and in a study by de Jong et al. [2], there was significant heterogeneity in the decision-making of 25 paediatric urologists when decisions were made on VCUG without supplementary cystoscopy – despite its critical role in guiding management. Last, our results in agreement with SFU grading for hydronephrosis with fair inter-rater agreement and strong intra-rater agreement [8]. Interestingly, this study demonstrates that inter-rater reliability is consistently poorer than intra-rater reliability, which suggests variability arises mainly between clinicians rather than within the individual rater. This study stresses the concerns that are longstanding within paediatric urology – there is a need for improved standardisation of radiographic diagnoses. For example, among all the published models in paediatric urology [4], <10% of models include external validation cohorts, and this provides a validated consensus dataset to improve model prediction when incorporating imaging. Next, the dataset from this work provides a standardisation tool for learning grading and diagnosing by radiographic images for trainees. This study is limited by sample size, but this was necessary for ethical dissemination of consensus datasets. In addition, publicly available images may represent selection bias and not reflect all routine cases of each disease. Next, this study reflects practice patterns of paediatric urologists trained in Europe based on the rater selection within the working group and may be different from practices of other locations. While a consensus-based reference standard may introduce consensus bias, this approach reflects real-world clinical practice in settings where no objective ‘gold standard’ exists for radiographic diagnosis. Lastly, this study may not replicate real-world decision-making that integrates additional clinical data or subsequent investigations. This study demonstrates meaningful concern for the reliability of radiographic diagnoses in paediatric urology for VUR, PUV, and hydronephrosis. Unlike prior reliability studies that primarily report agreement statistics, this work translates observed variability into a practical solution by providing openly accessible consensus datasets for education, model development and external validation via the ‘Radiographic Assessment and Diagnostic AgReement in paediatric urology’ (RADAR) website, and as benchmark for others to utilise that will continue to expand with an active invitation to practicing urologists to contribute. The authors declare no conflicts of interest.