Bridging biology and statistics with hybrid Bayesian experimental design for drug dose-response assays

Abstract Dose-response cytotoxicity assays are central to evaluating drug potency, yet selecting concentrations that capture both the full response range and key parameters such as the half-maximal inhibitory concentration (IC 50 ) and Hill slope remains challenging. Practical constraints, including limited replicates, variability across culture conditions, and the trial-and-error nature of current practice, make assays time- and resource-intensive. To address this, we introduce a Bayesian inference framework that quantifies uncertainty, incorporates prior knowledge, and extend it with a Bayesian optimal experimental design (OED) strategy to systematically refine concentration selection. We further propose a hybrid OED approach that integrates information-theoretic design with space-filling principles, aligning statistical rigor with biological intuition. Applied to ER+ breast cancer cells in monoculture and co-culture with stromal cells, this framework revealed differential drug responses while improving information efficiency. More broadly, our results highlight how Bayesian OED can bridge computational and biological perspectives, offering a path toward more efficient, reproducible, and interpretable experimental design in cancer research and beyond. Author summary We present a statistical framework that systematically captures uncertainty in drug cytotoxicity assays, which are standard laboratory tests that measure how sensitive cells are to drug treatments. Applying this approach to breast cancer cells grown with and without supportive bone marrow cells, we revealed how the surrounding environment can protect cancer cells from therapy. By guiding the selection of drug concentrations, our framework reduces trial-and-error, thereby saving time and resources. Beyond cancer research, this strategy offers a general way to design more efficient biological experiments, ultimately supporting the development of new treatments.