Universal psychosocial screening for post-secondary students with HEARTSMAP-U : evidence for inter-rater reliability

Post-secondary students are typically assuming greater life responsibilities and independence, which can become emotionally overwhelming, resulting in a high prevalence of psychosocial issues. At a time when many young adults are moving away from their support networks and transitioning from pediatric to adult health care systems, there is a perceived lack of resources and numerous barriers to mental health treatment. A digital psychosocial self-assessment and guidance tool for post-secondary students (HEARTSMAP-U) was adapted to address this growing concern. HEARTSMAP-U has already undergone comprehensive adaptation and evaluation work among both a broad population of clinicians who support young adults in post-secondary education, as well as among post-secondary students themselves. Through a multiphasic study, this research aimed to 1) evaluate the inter-rater reliability of HEARTSMAP-U among young adults pursuing post-secondary education, as applied to a set of fictional cases, and 2) evaluate, on the same set of fictional cases, the scoring agreement between student and clinician assessors. In Phase 1 (n = 15), an iterative process was used to evaluate the fictional vignettes for comprehensiveness and clarity. Feedback received was reviewed and incorporated into the next version of the vignette and this process was conducted four times until all comments were positive and saturated. In Phase 2 (n = 34), HEARTSMAP-U’s inter-rater reliability was evaluated among post-secondary students. Students displayed substantial to near perfect inter-rater scoring agreement in applying HEARTSMAP-U to the finalized fictional clinical vignettes, with weighted kappas on tool domains ranging from 0.72 (Student Health; 95% CI: 0.71, 0.73) to 0.81 (Psychiatry; 95% CI: 0.80, 0.82). In Phase 3, a clinician applied HEARTSMAP-U to the same vignettes and a large proportion of scoring agreement was found between student and clinician responses (median range 0.82-0.85) on concern severity and on whether the individual described in the vignette had resources in place (97%). Together, these results indicate that HEARTSMAP-U can be consistently interpreted by young adults pursuing post-secondary education. These study results will add to HEARTSMAP-U’s ongoing evaluation in which HEARTSMAP-U’s predictive validity will be assessed.