Discovering Data Sets in Unstructured Corpora: Discovering Use and Identifying New Opportunities

Federal statistical agencies are keenly aware that scientific research is not the only way in which their data assets are used for evidence building. Entities around the world have been building services on top of curated corpuses of scientific research, to help provide insight as to the importance of the works in the collection. This also provides a much-needed framework for statistical agencies and other dataset creators to search and find the usages and impacts of those datasets. While the other papers in this special issue largely apply machine learning models to find datasets in an extensively curated corpus, this paper starts with a much less structured framework and examines the potential to discover how datasets are used in writing that’s targeted at a broader base of users than scientific researchers, and describes the challenges and lessons learned from the exercise.