Automatic TimeML Corpus Validation: Uncovering Errors and Inconsistencies in Gold-Standard Corpora

TimeML serves as a foundational annotation scheme for capturing temporal information within textual data. Despite the availability of several gold-standard TimeML corpora, the adherence to TimeML annotation rules and principles of temporal reasoning remains uncertain. This paper proposes a fully automatic sanity check algorithm for TimeML corpus validation and presents a comprehensive evaluation of four mainstream TimeML corpora. Through this investigation, numerous previously unrecognized issues were identified, potentially impacting downstream applications such as graphs and timelines. The analysis revealed a total of 682 errors across the evaluated corpora, encompassing TimeML compatibility errors, violations of TimeML annotation guidelines, temporal reasoning inaccuracies, annotation inconsistencies, and redundancy. These findings underscore the suboptimal nature of current gold-standard TimeML corpora, emphasizing the necessity for corpus validation procedures prior to their release.