The Stability of Post Hoc Model Modifications in Confirmatory Factor Analysis Models

In applications of covariance structure modeling, practitioners frequently engage in post hoc model modification when confronted with models exhibiting unacceptable fit. Little is currently known about the extent to which such procedures capitalize on sampling error. In the present study, the problem of chance model modifications under varying levels of sample size, model size, and severity of misspecification in CFA models was examined. Under low levels of misspecification, specification searches yielded consistent modifications when N ≥ 800, but under more severe misspecifications, modifications tended to be unstable unless N ≥ 1,200. Modifications were somewhat more stable for the larger model. The findings suggest that practitioners should exercise caution when interpreting modified models unless sample size is quite large.