The Influence of Translator Backgrounds and Machine Translation on Statistical Properties of Surveys: Evidence from a Survey Experiment

Comparable questionnaire translation is essential for drawing valid conclusions in cross-cultural survey research. Sound translation methodology, including the use of adequate personnel, is seen as crucial for reaching this goal. The recommended methodology should be empirically backed and stay tuned to latest developments, say machine translation. Against this backdrop, to investigate the potential effect of varied translators’ backgrounds and machine translation on the statistical properties of surveys, we conducted an experiment in which an English questionnaire was translated into German by 16 professional translators and 16 social scientists. We introduced two translation conditions: translation from scratch and post-editing (machine translation corrected by a human translator). Translations were subsequently fielded in web surveys. To investigate the quality of the survey data from these 32 translation versions (approx. 250 responses each), we use standardized mean distance and Cohen’s d with the official translation as a benchmark. We have four key findings: First, the resulting statistical means of the survey items vary, sometimes substantially, across translations. Second, post-editing, with its potential to prevent unwanted creativity and guard against gross mistranslations, is associated with a reduced gap between the survey data from the experimental questionnaires and the official translation; it also lowers the variability among different translations. Third, post-editing can lead to systematic bias if translation errors made by the machine are not identified or miscorrected. This applies to both expert groups. Fourth, social scientists are slightly more likely to deviate from a benchmark and, when translating from scratch, to produce translations leading to statistical outliers of survey data. This study highlights to what extent decisions concerning the choice of translators and the integration of machine translation can impact the statistical properties of survey data. We offer evidence to implement multi-step, collaborative translation procedures to enhance data comparability in cross-cultural studies.