Beyond classical metrics: Generalizability theory across psychophysiological modalities

Psychophysiological research relies on biological measures to understand cognitive, affective, and behavioral processes, but the utility of these measures for studying individual differences depends on their psychometric reliability. Traditional reliability methods, such as classical test theory, often fail to account for the multiple sources of variance inherent in psychophysiological data. Generalizability theory (GT) provides a robust, multifaceted approach to reliability estimation by decomposing variance across multiple facets, such as trials, tasks, and sessions. This article introduces GT to psychophysiological researchers, detailing its advantages over classical approaches and demonstrating its application to a variety of psychophysiological modalities: event-related potentials (ERPs), electroencephalography (EEG), electrodermal activity (EDA), electromyography (EMG), and electrocardiography (ECG). We outline the two-phase process of GT: generalizability (G) studies, which quantify variance components, and decision (D) studies, which optimize reliability within study designs intended for specific purposes. Psychometric formulas are provided for estimating indices of generalizability, dependability, and measurement error for numerous designs, including designs based on difference scores. Additionally, we discuss best practices for variance component estimation, highlighting the advantages of multilevel modeling in handling unbalanced data and non-normal distributions, typical of psychophysiological data. By applying GT, researchers can enhance the replicability and interpretability of psychophysiological measures, ultimately strengthening their ability to link biological signals to psychological constructs. This framework represents a necessary evolution in psychophysiological science, ensuring that biological measurements are grounded in fundamental psychometric principles.