Inter‐rater and intrarater agreement and reliability in clinical staging of hidradenitis suppurativa/acne inversa

Dear Editor, Core outcome domains represent a major need in hidradenitis suppurativa/acne inversa (HS) as reliable disease staging systems and outcome measurement instruments (OMIs). Thorlacius et al.1 reported recently that none of the HS OMI tested by them could be recommended when used with HS experts as raters without a preceding bedside training session. Diverse staging systems and 30 OMIs were applied in 12 randomized controlled trials, with only three studies investigating inter‐rater reliability.2 We therefore investigated the inter‐rater and intrarater reliability and agreement of four widely used HS staging systems [Hurley score (Hurley),3 refined Hurley staging (RefHurley),4 International HS Severity Scoring System (IHS4)5 and HS Physician's Global Assessment (HS‐PGA)6] in a prospective study approved by the ethics committee of the Wroclaw Medical University. Five patients (mean ± SD age 42·7 ± 11·9 years) with a clinically confirmed diagnosis of HS7 and widely differing disease severity were included. Raters were 32 physicians (24 dermatology residents and eight specialists) from different countries, who participated in the European Academy of Dermatology and Venereology (EADV) HS School (held 4–5 February 2019 in Wroclaw, Poland). The study protocol and the HS staging systems to be tested were provided to the raters before rating. All raters assessed the same patients on two consecutive days. After the first staging an educational HS staging session was held. Intrarater agreement (IAA) was assessed by the Wilcoxon test for paired samples, inter‐rater agreement (IEA) by Spearman's coefficient of rank correlation (rho), intrarater reliability (IAR) by the intraclass correlation coefficient (ICC) and interrater reliability (IER) by Cohen's kappa coefficient. Results were interpreted using Altman's kappa Benchmark Scale (0–0·20, poor agreement/reliability; 0·21–0·40, fair; 0·41–0·60, moderate; 0·61–0·80, good and 0·81–1·00 very good). Owing to the limited patient sample size, the lower limits of the 95% confidence intervals (CIs) were used for critical reliability interpretation. Calculations were carried out using the MedCalc Statistical Software version 18·11·3 (MedCalc, Ostend, Belgium). After the first examination, the lowest CI of rho showed a good IEA for IHS4 and HS‐PGA, and a moderate IEA for Hurley and RefHurley (Table 1). A marked IEA improvement was identified after the second staging, with a good agreement level for all staging systems but without statistical significance, probably due to the low patient numbers. Inter‐ratera agreement using the HS staging systems tested HS‐PGA, Hidradenitis Suppurativa Physicians’ Global Assessment; IHS4, International Hidradenitis Suppurativa Severity Scoring System; lower confidence interval (CI), the lower 95% CI of Spearman's rho; P, patient. aNumber of patients examined = 5; number of raters = 32. bNo lesions; cthe stages IA, IB, IC, IIA, IIB, IIC, III of the Refined Hurley score are presented here as stages 1–7, respectively. Inter‐ratera agreement using the HS staging systems tested HS‐PGA, Hidradenitis Suppurativa Physicians’ Global Assessment; IHS4, International Hidradenitis Suppurativa Severity Scoring System; lower confidence interval (CI), the lower 95% CI of Spearman's rho; P, patient. aNumber of patients examined = 5; number of raters = 32. bNo lesions; cthe stages IA, IB, IC, IIA, IIB, IIC, III of the Refined Hurley score are presented here as stages 1–7, respectively. The IER was good with IHS4 (0·63); the raters could distinguish the disease stage in 35 of 40 comparisons (88%; 17 of 20 after the first and 18 of 20 after the second staging). IER was fair with Hurley (0·58; 75%, 30 of 40), RefHurley (0·47; 75%, 30 of 40) and HS‐PGA (0·46; 82·5%, 33 of 40). A good IAA was assessed by the lowest CI of rho of all staging systems. IHS4 (0·75) and HS‐PGA (0·72) provided a higher number of identical assessments in the first and second stagings than did Hurley (0·68) and RefHurley (0·67). IAR assessed by the lowest CI of ICC was very good for HS‐PGA (0·84) and HIS4 (0·84), and good for Hurley (0·76) and RefHurley (0·74). In all cases, the patients’ stage distinguishability increased at the second staging as shown by the rates of identical assessments and the reduction of the minimum required difference of mean rank using Hurley (0·32 vs. 0·45), IHS4 (0·34 vs. 0·42), RefHurley (0·35 vs. 0·44) and HS‐PGA (0·36 vs. 0·38). To avoid intrinsic system problems, the possibility of using two systems for better assessment of IEA was evaluated. Hurley correlated better with RefHurley (0·89) and IHS4 with HS‐PGA (0·88). The combination of HS inexperienced with a few experienced raters in this study corresponds to the ‘real‐life’ IER and IEA among dermatologists, and is complementary to the work by Thorlacius et al.,1 who used only HS experts as raters. The good IEA achieved by all systems tested at the second staging indicates the importance of training raters in the application of HS staging systems. A good IAA was achieved by all four staging systems tested. IHS4 and Hurley provided the highest IAA when the first and second stagings were compared, probably due to their simpler three‐step scale, whereas HS‐PGA is a five‐step and RefHurley a seven‐step scale. By contrast, the IAR (test–retest reliability) and patient's stage distinguishability were very good for IHS4 and HS‐PGA. The IAR using IHS4 and HS‐PGA was very good, regarding both the capacity to distinguish the patient's stage and the reproducibility of the individual patient assessment. RefHurley and Hurley performed at lower levels, possibly indicating intrinsic problems regarding IAR. In general, observed agreement limits were wide compared with the scale ranges, especially for RefHurley and in patients with moderate HS. Thus, substantial changes would be needed in order to rule out measurement error over real change in cases of moderate severity. The apparent discrepancy with the work of Thorlacius et al.,1 who compared IER and IEA between patients with mild/moderate vs. severe disease and found that the measurement error increases with increasing HS severity, is probably due to the fact that the present work studied HS staging, whereas Thorlacius et al. studied the dynamic evaluation of HS severity. We confirmed the findings of Thorlacius et al.1 that the proper use of HS staging systems, especially on IAA and IAR,8 require a preceding bedside training session. Intrinsic problems of the staging systems could be avoided by the use of two systems for better assessment of IEA; Hurley correlating better with RefHurley and IHS4 with HS‐PGA. Nevertheless, our results illustrate the difficulty of HS staging by a simple rater evaluation of physical signs. Baseline and end point assessments by the same rater are recommended if lesion count‐based systems are used as HS trial end points. Based on the results of this study, IHS4 in particular, but also HS‐PGA, can be recommended for staging of patients with HS. We thank the included patients for their willingness to participate and the staff of the Department of Dermatology, Venereology and Allergology, Wroclaw Medical University, Wroclaw, as well as Mr Marc Somja, EADV employee, for logistical support. A full list of affiliations is available in Appendix S1 (see Supporting Information). Funding sources: this work was indirectly supported by an educational grant of the European Academy of Dermatology and Venereology (EADV) provided for a Hidradenitis Suppurativa School. Conflicts of interest can be found in Appendix S1 (see Supporting Information). Appendix S1 Full list of affiliations and conflicts of interest.