The intra- and inter-rater reliability of plain radiographs for Hill-Sachs and bony glenoid lesions: evaluation of the radiographic portion of the instability severity index score

Background The Instability Severity Index Score (ISIS) was developed to help determine the prognosis for recurrent shoulder instability and to assist surgical decision-making. The radiographic portion of the ISIS represents a substantial portion of the total score. The present study examined the intra- and inter-rater agreement, reliability and accuracy of the radiographic components of the ISIS. Methods Four assessors evaluated 49 blinded shoulder radiographs. Assessors documented their observation of the presence of a Hill–Sachs lesion and/or a loss of glenoid contour. Radiographs were reviewed twice in random order over two sessions. Intra- and inter-rater reliability and accuracy were calculated. Results Intra-rater agreement ranged from 71% to 94% for the presence of a Hill–Sachs lesion and 85% to 94% for a loss of glenoid contour. Intra-rater kappa values ranged from 0.41 to 0.86 and 0.56 to 0.74, respectively. Inter-rater agreement was found to be 63% to 78% and 78% to 90% for Hill–Sachs and glenoid lesions, respectively. Corresponding kappa values were 0.31 and 0.48. Accuracy ranged from 29% to 57% for Hill–Sachs and 65% to 73% for glenoid lesions. Conclusions In the present study, intra- and inter-rater reliability of the radiographic portion of the ISIS demonstrated limited kappa and accuracy. We suggest that the ISIS should be used with caution as a guide for surgical management.