Improving Document Quality via Updatable Extracted Views

Data cleaning is a critical component of modern data intelligence systems. However, most existing approaches have been proposed for tabular data, despite the fact that much real-world data is unstructured. We propose to leverage information extraction programs to disclose where to apply data cleaning processes for documents. After using information extraction to produce a table of values extracted from the document (an “extracted view” of the documents, in the sense of a database view ), the resulting table is subject to any desired tabular-based data cleaning protocol. However, in order to reflect such cleaning back in the source documents, we require that updates to the tabular views can be translated into document updates without introducing any unintended changes to the contents in those views. With this approach, we consider a document to be clean if all its extracted views are clean. In this paper, we characterize and verify a set of sufficient conditions for rule-based extraction programs that ensure document updates do not introduce unintended changes to the extracted views, thus qualifying them for inclusion in a document cleaning pipeline. Through experiments conducted on medical records and on math-intensive documents, we demonstrate that our approach provides an effective, practical pipeline for correcting data quality problems in documents. The second of these experiments demonstrates that a math retrieval system can perform at least twice as well on the cleaned documents as it does when applied to the documents without cleaning, matching the improvements achieved by specially tailored cleaning methods applied to those documents during the search pipeline but preserving the cleaned documents for use in other applications as well.