UK Biobank, big data, and the consequences of non-representativeness

UK Biobank is an unparalleled resource of extensive health information from 500 000 individuals and with more than 400 peer­reviewed publications to date. The sampling population is volunteer­based and is not repre­ sentative of the UK population. Investigators state that although the estimates of prevalence and incidence should be interpreted with caution, valid measures of associa tion and estimates of causal effect can be more readily interpreted as they do “not require participants to be repre sentative of the population at large”. This statement is a puzzling claim: sample selection can indeed influence measures of association. Specifically, whether or not an association ob­ served in a study is similar in some other target population (ie, has external validity) depends on a number of factors, including the distribution of effect measure modifiers of the exposure–outcome relationship in the study sample and target population. Critically, a study can have restricted external validity even when it has internal validity, which might occur in a randomised trial. Thus, researchers should not be quick to set aside issues of representativeness in interpreting UK Biobank results. The situation can be illustrated with a numerical example. The appendix shows the unobservable truth with regard to an exposure–disease rela­ tionship in a target population of 66 million individuals (approximately the current population of the UK). Those exposed to X (a gene, or an environmental factor) have twice the risk of the outcome Y compared with those that are unexposed. Suppose further that X is only causal among people who are also exposed to a third factor A (eg, lack of exercise, present in 50% of the population). Among those with A, the exposed population have 3 times the risk compared with those who are unexposed; among those without A, there is no association between X and Y. We hypothetically recruit 500 000 participants (approx­ imately the size of the UK Biobank sample). But, during recruitment, those unexposed to A (healthier volunteers) are more likely to join, than those exposed to A are (by about 2:1, appendix). In the study sample, the risk ratio is 1·67 (appendix). The association between those exposed to X and the disease differs between the study sample and the target population because the prevalence of A differs; indeed, the association would differ in any population with a different prevalence of A. The magnitude of an exposure’s association with an outcome depends on the prevalence of other factors that interact with the exposure; in our case, A interacts positively with X, and because the prevalence of A decreases, the magnitude of the association between X and Y also decreases. When the target population is well defined and the relevant modifiers are measured, weighting methods can be used to map the results in the study sample to match the target population. But increases in sample size alone cannot overcome selection issues. Indeed, larger sample size in a skewed sample only leads to confidence in answers that might not apply to the target population. Thus, it is paramount that external validity be taken more seriously in the UK Biobank and other large data resources. We suggest that investigators in the UK Biobank and elsewhere consider these assumptions when making inference, and reconsider the idea that associations are generalisable to all possible target populations, or relevant to public health and clinical medicine, simply because the sample size is large.