Rating the Rater

Introduction IN psychiatric clinical research, the description and evaluation of behavior is very often based on judgements made by human observers. Of the various psychometric methods employing human judgement, the rating scale is perhaps the most widely used because it is easy to administer and usually yields quantifiable data. The apparent simplicity of the method, however, is deceiving, for the rating scale is subject to error and considerable care must be taken to ensure the validity and consistency of the observers' judgements. To establish such consistency of judgements, the usual method is to systematically train all those who are expected to use a certain rating scale. This is done within the setting where the study will be undertaken and prior to its beginning. It results mostly in a high interrater reliability. However, there are certain reasons why it might be desirable to know how closely different