Validity and reliability of on‑site and off‑site direct observation of procedural skills (DOPS) for knee mobilization in Physical therapy education

Objectives The direct observation of procedural skills for knee joint mobilization (DOPSknee) tool requires validation to ensure its reliability and validity for providing immediate feedback and assessing its applicability in off-site settings. We aimed to describe the implementation and clinimetric properties of DOPSkneein both on-site and off-site environments and to evaluate the preclinical competencies of physical therapy (PT) students.Methods Following the development of DOPSknee, 42 preclinical PT students were filmed performing knee joint mobilization. Their performance was assessed by two clinical instructors using DOPSknee, in on-site and off-site conditions, with an interval of 1.5–4.5 months between assessments.Results DOPSknee exhibited good on-site inter-rater reliability (ICC = 0.78). However, it had poor off-site inter-rater reliability (ICC = 0.37) and intra-rater reliability differed substantially between on- and off-site evaluations (evaluator 1: ICC = 0.42; evaluator 2: ICC = 0.88). Nevertheless, DOPSknee exhibited good reliability for average measures from on- and off-site evaluations (ICC = 0.84). Average total DOPSknee scores were significantly correlated with average global rating scores in both on-site (σ = 0.47) and off-site conditions (σ = 0.75), as well as in cross-over conditions (σ = 0.54 and 0.71). The cutoff score for DOPSknee was determined to be 14 points. Construct validity analysis of both on- and off-site evaluations revealed that students who passed the DOPSknee evaluation had significantly higher average global rating scores (p < 0.05).Conclusions DOPSknee exhibited good validity and reliability for assessing student performance in on‑site conditions, while average measures may improve reliability in off‑site assessments. These findings highlight its potential as a structured procedural assessment tool in physical therapy education.