Helping Researchers Find The Way: Optimising the Preparation of Analytical-Ready Geospatial Datasets

Public Sector organisations have large repositories of geospatial data that researchers have long been unable to take advantage of due to lack of publication in relevant catalogues, being in inappropriate formats, or lacking the computational skills to prepare the data into a research-ready dataset that can be easily analysed. This talk will describe how we have worked with the Scottish Government to create publishable geospatial data products, with the first series of products being the application of an efficient route-finding algorithm to generate routes from each residential address in Scotland to its nearest GP Surgery. This has resulted in a generic framework that can accept any other “Point of Interest” (POI) that contains coordinates, depending on the researchers’ needs, and we will highlight the diversity of POI data held by the Scottish Government. The resulting data products not only show summary statistics for the whole processed dataset, but the associated metadata allows researchers to determine trends based on one of a number of criteria. We will discuss the computational challenges associated with processing the data, including improving the runtime from 30 hours to 50 minutes on a dedicated high-memory Virtual Machine, allowing us to support researchers with much more agility. Finally, we will describe how these real-world raw datasets have had data cleanliness challenges, and the steps we have taken to mitigate these such that the resulting data products are still of value to researchers.