Lems of Analyst and Observer Agreement in Naturalistic Narrative Data

The use of naturalistic narrative data is increasing but without increasing methodological rigor. Assumptions underlying such methods prohibit an investigator from interrupting the natural habitat by use of such standard laboratory procedures as equal observation intervals for all subjects, equal behavior units, or other artificial equalization of the behavioral data stream. Data analytic techniques commonly used in standard laboratory research present problems in the analysis of naturalistic data of this type inasmuch as most assume equal observation intervals, equal unit intervals, or some other fundamental character of equality. These problems are particularly acute with respect to estimates of analyst and observer agreement. This paper addresses these problems and discusses the advantages and disadvantages of several possible solutions.