Synthetic data in Multiple Sclerosis: replicating the results of a clinical trial with artificially generated data.

Abstract Background Collecting high quality, patient level data demands significant efforts and resources. A potential solution, well-suited for exploratory hypothesis testing, gathering evidence, and assessing the heterogeneity and generalizability of estimates, is the usage of synthetic data. Synthetic data are newly generated data from real, original data. They share a fundamental set of statistical properties, which is sufficient to replicate statistical analysis findings. Aim of this work is to explore the feasibility of obtaining clinically relevant insights from analyses on synthetic data. Methods We used baseline characteristics and follow-up data from the 641 subjects with progressive multiple sclerosis enrolled in the SPI2 randomized controlled clinical trial evaluating the effect of high dose biotin on disability improvement. We used four strategies (noise addition, chains of conditional distributions, multivariate modeling, and generative adversarial networks) to generate artificial data from the original SPI2 data. We simulated six research scenarios with different methodology, complexity, and data usage level. For each strategy we generated multiple samples and averaged the results of each scenario. Then, we compared synthetic-data-analysis results with original-data-analysis results by standardized mean difference (SMD) and percentage of confidence interval overlap (CIO). All the Python and R code used to produce these findings is available and public. Results Median SMD was 0.73 with inter-quartile range from 0.28 to 2.25. Median CIO was 77.3% with inter-quartile range from 46.4–89.2%. Bias direction was an underestimate 66.7% of the times (16/24) but largely differed accordingly to the research scenario. In 75% of the cases (18/24) the formal test of the null hypothesis of the corresponding scenario on synthetic data offered the same answer of the original data. Conclusion Although the selected synthesis approaches may not be exhaustive or state-of-the-art, and methods were minimally optimized, it is feasible to obtain clinically relevant insights from synthetic data, particularly for exploratory purposes, albeit with certain caveats. Assessing the quality of synthetic data before initiating any analysis is crucial. While a consistent underestimation of an association may occur, the bias inconsistency by synthesis strategy implies that a combination of these methods could enhance future synthetic generation strategies.