Which data source in clinical performance assessment? A pilot study comparing self-recording with patient records and observation

OBJECTIVE: A pilot study aimed to determine the extent to which each of three data sources could provide complete and reliable data for valid assessment of clinical performance. DESIGN: Clinical decisions taken in 168 consultations by seven family physicians were reviewed against guidelines for 15 clinical conditions. In total, 206 criteria were reviewed using three sources: medical records, observation in surgery, and structured self-recording by the physicians. SETTING AND PARTICIPANTS: Seven family practices in the Netherlands. MAIN MEASURE: Scores (%) of data recorded/total were obtained for each method. Kappa scores for the agreement between the three data sources were also obtained. RESULTS: Medical record examination provided 40%, observation 72%, and physician self-recording 95% of the data required for the review against guidelines. Nine per cent of the clinical decisions could be reviewed when using medical records, 46% when using observation data, and 69% when using data from prospective self-recording. In particular, decisions in the area of patient education and diagnostic examinations could not be reviewed validly using medical records only. Kappa agreements between the data available from the three sources as well as between the review results appeared to be 0.79. CONCLUSIONS: Medical records alone only supply sufficient information for the review of a very limited set of clinical decisions. Physician self-recording has significantly more potential for valid review of a broad range of clinical decisions. Furthermore, self-recording seems a reliable data collection method that deserves further research.