Data Resource Profile: The New Zealand Integrated Data Infrastructure (IDI)

The Integrated Data Infrastructure (IDI) is a collection of New Zealand whole-population administrative data sources from government agencies, the 2013 Census and several questionnaire-based social and socioeconomic surveys from samples of the population. The IDI allows whole-population analysis across different sectors of government (e.g. health, social services, education). Data are available and linkable at the individual level for an ‘ever resident’ New Zealand population, including ‘people born in New Zealand, permanent residents, people with visas that allow them to reside, work, or study in New Zealand (including international students and temporary workers), and those who live and work … [in New Zealand] without requiring a formal visa’.1 Data provide a longitudinal record of events (e.g. hospitalizations, pharmaceuticals dispensed) over time, with different datasets covering different periods (e.g. hospitalizations have been captured since 1988 but maternity data only since 2002) (see Table 1). As of September 2018, the IDI holds 166 billion pieces of information.2 Data available (start date) in the New Zealand Integrated Data Infrastructure (IDI) Data available (start date) in the New Zealand Integrated Data Infrastructure (IDI) The IDI was first established in 2011, but has been continually updated with the addition of new records, new data tables and new fields within tables. The information presented in this profile is current as at September 2018. As New Zealand’s national statistics office, Statistics New Zealand (Stats NZ) has a data and information leadership role throughout the public sector, and is funded through ‘Vote Statistics’.3 This role includes overseeing the IDI on behalf of the system. Data are sourced from a variety of public sector organizations (e.g. Ministry of Health, Ministry of Justice) and non-government organizations (e.g. Auckland City Mission), who are responsible for maintaining and providing up-to-date data tables to Stats NZ for inclusion in the IDI. IDI data are stored in separate data tables in an SQL database (see Table 1 for an overview of data available in the IDI). Depending on the table, data may be at the individual, household, provider or small-area geographical level. Personal identifiers (i.e. names, addresses, government agency ID) have been removed to protect privacy and confidentiality; instead, unique but anonymous identifiers identify individuals and households. The main unique identifier for individuals (snz_uid) enables the same individual to be linked across different datasets; household- or area-level tables typically link to individuals using concordance tables linking household- or area-level identifiers to the snz_uid. The IDI is updated (‘refreshed’) up to four times per year, which includes addition of new datasets and updates of existing data. For each new refresh, datasets are linked to the IDI ‘spine’ using probabilistic linkage (the Fellegi-Sunter method).4 The IDI spine aims to include all people who have ever been a resident in New Zealand, and it is constructed for each new refresh by linking together tax records since 1999, New Zealand birth records from 1920, and long-term visas from 1997 (currently about 10 million individuals).1,2 Linkage rates and linkage quality (e.g. false-positive error rates) differ for each dataset linkage to the spine. For health records linked to the spine, the linkage rate was estimated in September 2018 to be 85% (88% among New Zealand residents) and the false-positive rate is estimated to be 0.8%.5,6 There are differences in linkage rates by sex (males are about 2% higher), ethnicity (lower for Pacific and Asian New Zealanders compared with Māori and European New Zealanders), and age-by-ethnicity (declines for European New Zealanders older than 85 years, Māori New Zealanders older than 65 years and Pacific and Asian New Zealanders older than 35 years).6 Under the ‘Five Safes’ framework followed by Stats NZ, IDI data can only be used for public-good research purposes.7 IDI data cannot be used for individual case management or for regulatory purposes. A searchable database of all IDI research projects is available via the Stats NZ website.8 It is a condition of access that all projects and lead researcher details are published on this website, to ensure transparency about how the IDI is being used. Broadly speaking, the IDI data can be used for four types of research: descriptive, analytical, methodological and evaluation of policies and interventions. Examples of each of these are provided below. Shackleton et al.9 used repeated cross-sections of B4 School Check data in the IDI to identify decreasing trends in pre-school obesity in New Zealand from 2010 to 2016. McLeod et al.10 used IDI data to identify and characterize adolescents who experience poor health and other outcomes. IDI data can be used to define population cohorts which can be assessed over time.11 For example, Dixon12 defined cohorts of sufferers of chronic disease and assessed short- to mid-term effects on work and income; Berry et al.13 assessed long-term outcomes for a cohort of very pre-term babies; Donovan et al.14 used a cohort approach to show that living near green and more biodiverse vegetation lowered the risk of asthma; Davie and Lilley15 assessed the financial impacts for an older cohort who have experienced injury; and Teng et al.16 examined a cohort of earthquake survivors and showed that cardiovascular disease admission rates after the Canterbury earthquakes were associated with area-level residential damage. IDI data have also been used to assess socioeconomic and ethnic inequalities (e.g. in pre-school oral health17 and in immunization18). Methodological research using IDI data is an under-explored area, especially in regard to data quality and linkage bias and what can be done to overcome these issues. However, Zhao et al.11 and Stats NZ, through their work on ‘Census transformation’,19,20 have conducted methodological investigations regarding the ability of administrative data from the IDI to produce population estimates as accurately as the New Zealand Census. The IDI lends itself to evaluation work because researchers are able to assess the long-term impact of life events, policies or interventions for individuals. For example, Vaithianathan et al.21 compared health and other outcomes for children and mothers who received the ‘Family Start’ intervention, a home visiting programme for at-risk and low-income mothers, against a propensity-scored-matched control group. The IDI has several analytical advantages. First, the IDI links health data to data from various government sectors. This adds enormous value to already existing health data, in determining both the drivers of health and the consequences of ill health. Second, the IDI allows for whole-population data analysis; few other countries have this ability. Many datasets in the IDI have national coverage and contain service use data for the whole population of New Zealand. The large sample sizes allow for analysis of small groups and rare events in ways that are not possible in projects that are dependent on primary collection of new data. Further, methods have been developed to accurately estimate the population under investigation at specific points in time (e.g. for use as a denominator), taking account of border movements, births and deaths.11 Third, the IDI’s use of administrative data eliminates the risk of recall bias, which is a problem if data collection relies on self-reports of service use (e.g. hospitalization or pharmaceutical dispensing). Fourth, the IDI can be used to analyse other information about study participants for whom detailed information was obtained in field studies at baseline (e.g. birth cohort studies), thus allowing longitudinal analyses of health outcomes identified through the IDI. In both cases, this requires loading data from field studies to the IDI. This is done by Stats NZ upon request (with anonymity being preserved throughout the process), so long as appropriate permissions (e.g. consent) are in place. However, research fully capitalizing on the longitudinal nature of the data (e.g. estimates of cumulative exposure or exposure patterns using marginal structural models or other G-methods) is rare to this point—an opportunity that should be utilized.22 Fifth, there are analytical advantages that have been touched on above: the ability to define and analyse long-term, system-wide trends for cohorts, and the ability to evaluate interventions. One key weakness is data quality. Data have been collected by a number of agencies, and by multiple people within multiple divisions within those agencies. Also, for the most part, data have been collected for operational, monitoring or accounting purposes, rather than for research purposes. As such, quality of data is variable across different agencies and across different data tables. However, quality of data is improving over time. For example, the pre-school health screen (B4 School Check) had incomplete coverage in early years, but more than 90% coverage from mid 2012. Also, pharmaceutical dispensing was not always recorded against a specific patient in the early years of the collection, but was recorded for more than 97% of all pharmaceuticals dispensed from mid 2007. A second weakness is incomplete data documentation and metadata. Because the data have only recently been made available for wider research use, comprehensive documentation has lagged behind data availability. Furthermore, documentation about measures and constructs is stored at the agency level, with no central database of the measures contained in the IDI. A third weakness is incorrect linkage. Linkage is probabilistic and, although the estimated false-positive rates are low (typically <2%),5 incorrect linkage will cause errors in analyses of exposure–outcome associations—via selection bias (i.e. systematically missing observations), measurement error (e.g. incorrect variables from the wrong linked person assumed to apply to the index person) or confounding (e.g. missing confounders or mis-measured confounders). Rates of missing links (i.e. false-negatives) are unknown, and these may introduce similar biases. Additionally, as datasets are linked via a central spine, links between datasets external to the spine necessarily involve multiple linkage steps, each with potential error. For example, links between health and benefit data involve links between health data and the spine, as well as links between benefit data and the spine (i.e. two links and two possibilities for linkage error). Furthermore, and as noted above, health data for Pacific and Asian New Zealanders and older Māori New Zealanders are linked to the IDI spine at a lower rate, suggesting inconsistent coverage across population subgroups.6 A fourth weakness is that there will be no administrative data for individuals who have not accessed government agencies and services. This means, for example, that the IDI will not include individuals with health problems who do not access services for those health problems; and that only individuals charged or convicted for crimes will be included, which will be a (biased) subset of individuals committing crimes. Further, when a research question is reliant on data from multiple agencies, the proportion of individuals with data across all datasets may be low (i.e. if the proportion of individuals accessing services from all of the agencies is low). There is also the problem that a non-event (e.g. no hospital admission) is indistinguishable from a non-linked event (e.g. a hospital admission that was unable to be linked). A fifth weakness is current limitations with Stats NZ IT infrastructure which limit the ability to use machine learning and other computationally intense methods. Finally, researchers need to be aware that different datasets cover different periods (summarized in Table 1). As roughly two-thirds of data collections began in the year 2000 or later, this limits some longitudinal investigations. For example, long-term follow-up of children from the B4 School Check health screen is not yet feasible, as that collection only began in 2008. Access to the IDI is by application to Stats NZ [https://www.stats.govt.nz/integrated-data/access-microdata-in-the-data-lab/]. Stats NZ applies the ‘Five Safes’ framework to statistical disclosure control—Safe People, Safe Projects, Safe Settings, Safe Data and Safe Output7—and researchers wishing to access the IDI are required to work within this framework. Researchers applying to use IDI data are vetted by Stats NZ (Safe People). Researchers must supply a curriculum vitae and the names of two referees, and Stats NZ uses this information to assess whether the researcher is a bona fide researcher, belongs to a bona fide research institution, has a history of trustworthy data use and has an ability to analyse large datasets. Projects are also vetted (Safe Projects) to ensure they are public-good research, which analyses and reports on groups of people rather than individuals. Under the Statistics Act 1975, Stats NZ are legally required to protect the privacy and confidentiality of the people and businesses they hold information about. Once a project is approved, first-time researchers participate in a one-hour training session on the confidentiality requirements of using IDI data. Researchers must sign a Confidentiality and Secrecy Agreement before accessing the data. Data access for the project is then provided through a secure ‘Data Lab’ environment (Safe Settings). Researchers access data through a protected virtual environment and only in secure research facilities on computers that can access the IDI server, but nothing else (i.e. computer hard and soft drives cannot be accessed, and there is no access to the worldwide web). As such, IDI data are never sent to researchers, but instead access is granted to analyse data within the Data Lab environment. Data Lab facilities exist in Stats NZ offices and some government departments, universities and research agencies throughout New Zealand. Researchers from outside New Zealand can and do work on IDI projects but need to either travel to New Zealand to do so or collaborate with researchers able to undertake analyses in a New Zealand Data Lab. Researchers access their project(s) through a login procedure that ‘unlocks’ only the data for which access has been granted; all identifiers have been removed from these data (Safe Data). Further value can be added by a researcher applying to link new external datasets into the IDI. An agreement between the data owners and Stats NZ is made to facilitate the new data linkage. The statistical and programming packages SAS, STATA, R and Python are available to use for data analysis, as well as SQL for database management. Researchers can request results (but not individual-level data) to be released from the Data Lab, after they have applied confidentialization procedures to these results to ensure neither individuals nor attributes of individuals can be identified (Safe Output). Released output must also be accompanied by a standard disclaimer, indicating (among other things) that access to the data is provided under the Statistics Act (1975) and that ‘careful consideration has been given to the privacy, security and confidentiality issues associated with using administrative and survey data in the IDI’.23 Support for IDI users is available through a Virtual Health Information Network [www.vhin.co.nz], which offers online guides, a discussion forum for users, a shared code repository and courses for users getting started in the IDI. The New Zealand Ministry of Health requires researchers to apply to the New Zealand Health and Disability Ethics Committee if they are requesting access to health data in the IDI [https://ethics.health.govt.nz/]. Stats NZ require that a researcher’s organization supports the research proposal, but do not require institutional ethical review of IDI research projects. However, institutional ethical review may be a requirement for universities, government agencies and other institutions, as well as for journals and funders. The New Zealand Integrated Data Infrastructure (IDI) was set up to allow whole-population analysis across different sectors of government (e.g. health, social services, education). Established in 2011, the IDI has been continually updated with new records and data tables. Data are available and linkable for an ‘ever resident’ New Zealand population (as at September 2018, about 10 million individuals). Data are secondary administrative data. Data fields may be at the individual, household, provider or small-area geographical level. Data capture individuals’ interactions with government agencies in the areas of health, social services, education, justice, geography, housing and economics (tax and income). Data from the 2013 Census and several questionnaire-based social and socioeconomic surveys from samples of the population are also included. Access to the IDI is by application to Stats NZ [https://www.stats.govt.nz/integrated-data/access-microdata-in-the-data-lab/]. Data access is through a secure ‘Data Lab’ environment, which exists in Stats NZ offices and some government departments, universities and research agencies throughout New Zealand. Researchers from outside New Zealand can work on IDI projects but need to travel to New Zealand to do so. The authors would like to acknowledge Lucy Ashby from Stats NZ for useful comments on drafts of the manuscript. Conflict of interest: None declared.