Challenges of clustering disease trajectories in people with multiple long-term conditions using data from 7.2 million patients

Background Generating clusters of people with similar patterns of Multiple Long-Term Conditions (MLTC) could help target healthcare services for specific patient groups. We aimed to generate data-driven clusters of patients based on their disease trajectories and assess clinical interpretability and associations with future health outcomes. Methods We used structured general practice data from the Clinical Practice Research Datalink Aurum, linked to Hospital Episode Statistics data on emergency department (ED) attendances and hospital admissions, including all adults registered on 1 st January 2015. For each patient, we generated a vector embedding representing their trajectory of diseases developed over time, using a transformer model, followed by clustering using k-means. Patients were subsequently followed up for 1 year to estimate associations of cluster membership with ED attendance, hospitalisation and mortality, compared to patients with no long-term conditions. We also evaluated the use of the clusters for predicting these outcomes compared with using number of long-term conditions, individual diseases or embeddings alone. Results Analysis included 5,981,091 (82.1%) patients with MLTC and 1,304,119 (17.9%) with no chronic conditions. We identified eight clusters of patients with MLTC as optimal. The largest, representing 21.3% of the population included strong contributions from cardio-kidney-metabolic conditions, but there was substantial overlap in prevalent conditions across clusters. Although large differences were found between clusters in 1-year odds of ED attendance, hospitalisation and mortality, with the highest odds in cardio-kidney-metabolic clusters, clusters explained only between 1.2-3.1% of total variance, and associations were substantially attenuated after adjustment for age, sex, ethnicity and deprivation. Clusters performed substantially worse at predicting outcomes than using the individual diseases or embeddings, and performed worse than a person’s count of long-term conditions at predicting ED attendance and hospitalisation. Conclusions We generated eight data-driven clusters of adults with similar trajectories of MLTCs. Interpretation of the clusters was challenging due to overlap of the dominant conditions across clusters, while clusters explained little of the variation in 1-year health outcomes. Our findings highlight challenges in translating data-driven clusters of disease trajectories into interpretable groups that can inform future care needs.