Text-Mining and the History of Medicine: Big Data, Big Questions?

Text-Mining and the History of Medicine: Big Data, Big Questions?Most of us have heard about 'Big Data', often as part of discussions about the information collected about us as consumers and citizens, and the increasingly sophisticated tools that analyse such information.But can we as historians of medicine benefit from thinking about our historical sources as 'Big Data', and 'mine' this data by adapting the tools used by commerce, computing science and intelligence?What possibilities for historical scholarship would such tools open up; what challenges do they present?These questions motivated our involvement in a collaborative project using text mining tools with medical history sources.In early 2014 we joined University of Manchester colleagues from the National Centre for Text Mining (NaCTeM) to work on a project funded by the UK Arts and Humanities Research Council, under their Digital Transformations theme. 1 As their name suggests, NaCTeM 2 develops text mining tools, mostly for academic use. 3 Our team set out to create a semantic search engine, one that would go beyond finding a specific keyword or string of text in a document.Semantic searches consider the context of use in order to locate terms and their variants representing particular concepts.We wanted to explore how such a search could provide new ways of working with series of medical texts covering a long period of major change in medical knowledge, practice and language.We chose two sources to form our corpus, as large-scale collections of structured text are known in digital humanities: the digitised run of the British Medical Journal from 1840 onwards, and the more recently digitised London-area Medical Officer of Health Reports that form the Wellcome Library's London's Pulse collection.Text mining (TM) uses digital tools to detect the structure of textual information, then find and recognise patterns and relationships in the structured data.4 For instance, one TM task is to find and compare the number of instances of particular terms over time in a defined corpus, as Google's N-Gram viewer does using the 'millions and millions of books' that Google has digitised or has access to in digital form.5 Another common TM application is finding the relative frequency of the words in a text and then visualising these in a way that makes the different frequencies apparent, for example, by size and position in word clouds.Other TM tools track and compare the relative locations of terms and their variants in texts, or, in the case of topic modelling, identify groups of terms that tend to be representative of a given topic.As Tom Ewing's use of these approaches demonstrates, they can provide insights that are not readily apparent in traditional reading, however 1 Our great thanks go to our NaCTeM colleagues and collaborators Sophia Ananiadou, John McNaught and Paul Thompson, and also to Nick Duvall, now at Warwick, who made important contributions to the historical side of the project.We also thank NaCTeM's Riza Batista-Navarro, Jacob Carter, Georgios Kontonatsios and Claudiu Mihȃilȃ.