Optimizing workflows: a case study on identifying medication-cancer associations in the Lleida population cancer registry

Exploratory analysis of medication–cancer associations in large patient populations presents substantial computational challenges that constrain research feasibility. Although individual survival models are computationally efficient, exhaustively applying them across thousands of combinations exposes architectural bottlenecks in data movement, serialization, and execution orchestration. This study evaluates a multi-level optimization framework for analyzing longitudinal data from the Lleida Population Cancer Registry in Catalonia, Spain. The workload comprises thousands of medication–cancer associations using Cox proportional hazards models, with an initial runtime exceeding two months under a strictly sequential reference configuration maintained to profile worst-case performance. Profiling revealed that runtime was dominated not by Cox regression computations, but by repeated database access, BSON deserialization, intermediate disk I/O, and inter-process communication, characterizing the workload as I/O-dominated rather than compute-bound. The framework incorporated schema redesign for efficient data access, computational enhancements through vectorized operations and precomputed, and containerized deployment using Kubernetes. A fundamental schema redesign, which transformed nested year-based medication records into a flattened, denormalized structure, improved query latency by 72 percent and enabled queries to execute 3.2 times faster than wildcard-indexed nested designs. The layered optimization strategy reduced per-task execution time from 66 seconds to less than one second, converting a two-month sequential workload into a scalable, distributed analytical pipeline. These findings demonstrate that multi-level architectural optimization can transform computationally infeasible analyses into practical tools, thereby enabling large-scale medication–cancer association studies to support evidence-based healthcare.