Diagnosing sepsis is subjective and highly variable: a survey of intensivists using case vignettes

BACKGROUND: Sepsis is the focus of national quality improvement programs and a recent public reporting measure from the Centers for Medicare and Medicaid Services. However, diagnosing sepsis requires interpreting nonspecific signs and can therefore be subjective. We sought to quantify interobserver variability in diagnosing sepsis. METHODS: We distributed five case vignettes of patients with suspected or confirmed infection and organ dysfunction to a sample of practicing intensivists. Respondents classified cases as systemic inflammatory response syndrome, sepsis, severe sepsis, septic shock, or none of the above. Interobserver variability was calculated using Fleiss' κ for the five-level classification, and for answers dichotomized as severe sepsis/septic shock versus not-severe sepsis/septic shock and any sepsis category (sepsis, severe sepsis, or septic shock) versus not-sepsis. RESULTS: Ninety-four physicians completed the survey. Most respondents (88%) identified as critical care specialists; other specialties included pulmonology (39%), anesthesia (19%), surgery (9%), and emergency medicine (9%). Respondents had been in practice for a median of 8 years, and 90% practiced at academic hospitals. Almost all respondents (83%) felt strongly or somewhat confident in their ability to apply the traditional consensus sepsis definitions. However, overall interrater agreement in sepsis diagnoses was poor (Fleiss' κ 0.29). When responses were dichotomized into severe sepsis/septic shock versus not-severe sepsis/septic shock or any sepsis category versus not-sepsis, agreement was still poor (Fleiss' κ 0.23 and 0.18, respectively). Seventeen percent of respondents classified one of the five cases as severe sepsis/septic shock, 27.7% rated two cases, 33.0% respondents rated three cases, 19.2% rated four cases, and 3.2% rated all five cases as severe sepsis/septic shock. Among respondents who felt strongly confident in their ability to use sepsis definitions (n = 45), agreement was no better (Fleiss' κ 0.28 for the five-category classification, and Fleiss' κ 0.21 for the dichotomized severe sepsis/septic shock classification). Cases were felt to be extremely or very realistic in 74% of responses; only 3% were deemed unrealistic. CONCLUSIONS: Diagnosing sepsis is extremely subjective and variable. Objective criteria and standardized methodology are needed to enhance consistency and comparability in sepsis research, surveillance, benchmarking, and reporting.