Effect of Machine Learning Cross-validation Algorithms Considering Human Participants and Time-series: Application on Biometric Data Obtained from a Virtual Reality Experiment

Data containing human participants and time-series features, as is commonplace in Human Factors research, require special considerations when used in machine learning applications. Ignoring such features during cross-validation procedures might lead to artificially increased model performances due to temporal (i.e. using future observations to predict the present) and participant (i.e using sub-data sets coming from the same participant for training and testing) data leakage. We propose a comparison approach to assess the model performance when machine learning algorithms are trained with two distinctly different cross-validation algorithms: k-fold, which assumes data independence, and population-informed forward chain (PIFC), which accounts for human participants and time-series features. A case study was conducted by using biometric measurements collected from a virtual reality chess experiment. The results show that substantial overestimation might occur when applying the k-fold algorithm instead of the PIFC algorithm.