Rater (dis)agreement on risk assessment measures in sexually violent predator proceedings: Evidence of adversarial allegiance in forensic evaluation?

Actuarial risk assessment measures are often admitted in court, partly because strong psychometric properties such as interrater agreement suggest that they increase reliability and reduce subjectivity in forensic evaluation. But how strong is rater agreement when raters are retained by opposing sides in adversarial legal proceedings? The authors review sexual offender civil commitment cases in which opposing evaluators reported scores on the STATIC-99, the Minnesota Sex Offender Sex Offender Screening Tool—Revised (MnSOST–R), or the Psychopathy Checklist—Revised (PCL–R) for the same individual. Differences between scores from opposing evaluators were often greater than expected based on rater agreement values reported in the instrument manuals and research literature. Score differences were often in a direction that supported the party who retained each evaluator. Rater agreement was stronger for the STATIC-99, intraclass correlation coefficient (ICCA,1) = .64; than for the MnSOST–R, ICC(A,1) = .48; and the PCL–R, ICC(A,1) = .42. STATIC-99 scores appeared less influenced by adversarial allegiance. Overall, however, results raise concern that an evaluator's adversarial allegiance could influence some assessment instrument scores in forensic evaluation.