The NTCIR-16 RCIR pilot task aimed to motivate the development of a first generation of personalised retrieval techniques that integrate reading comprehension measures and eye tracker signals as a source of information when ranking text content. The dataset used in the challenge was newly generated by capturing eye movement measures while experimental participants read text passages on a computer screen. The RCIR challenge included two sub-tasks: a) the comprehension-evaluation task (CET) that involved predicting a measure of a reader’s comprehension for text passages and, b) the comprehension-based retrieval task (CRT) that involved retrieving relevant passage texts ranked by comprehension score. The participating teams were ranked using Spearman’s correlation coefficient (rho) for the CET sub-task and normalised Discounted Cumulative Gain (nDCG score) for the CRT sub-task.
Healy et al. (Tue,) studied this question.