Small Data Challenges of Studying Rare Diseases

The age of big data is in full swing, with researchers in both clinical medicine and public health seeking to take advantage of the increasing availability of massive amounts of electronic and administrative health data. In turn, this has led to substantial resources and efforts being poured into the development and teaching of methods for data collection and storage as well as machine learning analytic methods.1 However, big data are not always available, especially in the study of rare diseases. Indeed, in the study of rare diseases, small sample sizes are inevitable, especially when the primary end point is also uncommon. As an example, Avadhanula et al2 used data from a cohort of 125 patients with alkaptonuria, a rare autosomal recessive disorder. Patients were recruited between 2000 and 2018 as part of a prospective longitudinal study conducted at the National Human Genome Research Institute to investigate the incidence of thyroid dysfunction among patients with alkaptonuria. While this is by no means a generous sample size, the cohort is the largest of its kind for patients with alkaptonuria, according to the authors. In the US, a rare disease is defined as a health condition that affects fewer than 200 000 individuals.3 This definition was created by Congress as part of the Orphan Drug Act of 1983, which aimed to use financial incentives to motivate pharmaceutical and medical device companies to develop new treatments for patients with rare diseases. Close to 7000 conditions meet this definition. Although a relatively small number of individuals are affected by each rare disease, the estimated total number of individuals living with any rare disease is between 25 million and 30 million.3 Support for rare disease research continues today. In 2016, the US Food and Drug Administration awarded $23 million over 4 years to support research in 21 different rare diseases.4 The Patient-Centered Outcomes Research Institute also has a special advisory board for rare disease research and thus far has funded more than 28 patient-centered comparative effectiveness studies that focus on the treatment and management of rare diseases.5 That the study of rare diseases poses unique challenges has been recognized. From the perspective of study design, researchers investigating rare diseases have many options, including crossover and adaptive trials.6 For observational studies, Whicher et al7 list self-controlled study designs, case-control designs, and prospective inception cohorts as potential designs suitable for rare disease research. Beyond the choice of study design, researchers must also be wary of the analytic challenges that arise from studying rare diseases, including the extent to which the available data can be viewed as representative of the entire population of patients with the condition and whether there is sufficient (statistical) power to draw definitive conclusions (ie, those that could inform decision-making). It is perhaps less well recognized that, when the sample size is small, P values are especially vulnerable to small deviations in the observed number of outcomes. For example, in the study by Avadhanula et al,2 1 patient was diagnosed with hyperthyroidism in the cohort of 125 individuals. Based on the exact test for 1-sample proportion, Avadhanula et al2 found insufficient evidence that the estimated prevalence in the study population (ie, 1 of 125 [0.8%]) was different than that in the general population (ie, 0.5%), with a P value of .88. As a thought experiment, suppose 2 patients instead of 1 had been diagnosed with hyperthyroidism. The same test would yield a P value of .23. Furthermore, if 3 patients were diagnosed, the resulting P value would then be .04. Thus, by hypothetically observing just 2 more cases, there is a dramatic change in the P value, a change that would likely alter decision-making. + Related article