Important Considerations of Data Collection and Curation for Reliable Benchmarking of End-User Eye-Tracking Systems

In this article we discuss how to build a reliable system to estimate the quality of a VR eye-tracker from an accuracy and robustness point of view. We list up and discuss problems that occur at the data collection, data curation and data processing stages. We address this article to academic eye-tracking researchers and commercial eye-tracker developers with the purpose of raising the problem of standardization of eye-tracking benchmarks, and to make a step towards repeatability of benchmarking results. The main scope of this article is consumer-focused eye-tracking VR headsets, however some parts also apply to AR and remote eye-trackers, and to research environments. As an example, we demonstrate how to use the proposed methodology to build, benchmark and estimate the accuracy of the FOVE0 eye-tracking headset.