Estimating the Consistency of Third-Party Evaluator Scoring of Organizational Self-Assessments

Organizational self-assessments are an important tool for organizations attempting to improve performance. Little is known about the measurement properties of assessment processes, particularly the consistency of the evaluator scoring that provides the validation for self-assessments. Similarly, little is known about the magnitude of inconsistencies that must exist. A critical step toward improving the self-assessment process is understanding its properties and designing strategies for improvement based on this understanding.This article reports on research conducted to determine the level of consistency of third-party evaluator scoring of written organizational self-assessments using the Malcolm Baldrige National Quality Award as the assessment standard. Using practicing professionals, an experiment was conducted that measured consistency at several levels: among untrained evaluators; among trained evaluators; between trained and untrained evaluators; and between the various items and categories constituting the Baldrige Award criteria.The results indicate that consistency for the more quantitative categories was better than for qualitative categories. Frame of reference (FOR) training had an effect on evaluator consistency (for example, reduced leniency) and seemed to have greater effect on the categories with more quantitative content. No evidence was seen that FOR training reduced score variance among evaluators.