A Review of Psychometric Assessment and Reporting Practices: An Examination of Measurement-Oriented Versus Non-Measurement-Oriented Domains

The primary aim of the present study is to determine whether the psychometric evaluation practices and test-analytic rationales of researchers publishing in journals with a measurement focus differ from those of researchers publishing in journals with varying substantive research foci. Several components of two different samples of articles were examined and compared; one contained articles from a set of measurement-oriented journals ( n = 402) and the other contained articles published in journals representing a cross-section of research domains ( n = 289). Findings indicate that, contrary to expectations, articles published in measurement-oriented journals, as compared with general journals, generally may not reflect better psychometric analysis and reporting practices or sounder test-analytic rationales on the part of the researchers. It was also found that although researchers are generally evaluating either score precision/reliability or validity, they seldom evaluate both, indicating that there may be a general lack of appreciation for the importance of conducting a full and coherent data-based test analysis whenever a measure is employed. A number of limitations of the study and recommendations for future research are also addressed.