Deep learning has advanced cancer imaging and cancer-related risk prediction, but many high-performing models remain difficult to interrogate in clinically meaningful terms. This creates an interpretability paradox: gains in predictive performance often coincide with reduced transparency, while widely used post-hoc explanations can be persuasive without providing reliable evidence of model reasoning. Here, we present a critical narrative review and position argument, supported by a semi-systematic mapping of the 2017–2025 literature across major biomedical and technical databases, to reassess explainability practices in oncology and identify design alternatives that are better aligned with clinical oversight. Across cancer applications, explainability is most operationalised through post-hoc attribution methods, including saliency-style heatmaps for imaging and local feature attribution for structured risk models, with evaluation frequently limited to plausibility narratives or internal technical proxies. We argue that this practice leaves an evaluation gap: limited evidence on explanation fidelity, stability, calibration-aware interpretation of risk outputs, and impact on clinician decision-making under realistic workflow constraints and distribution shift. To address this gap, we delineate three underused but clinically aligned design families: concept-based modelling, counterfactual and contrastive reasoning, and prototype- or case-based approaches, alongside emerging hybrid and interpretable-by-design architectures. We synthesise these insights into a pragmatic lifecycle framework in which post-hoc Explainable AI (XAI) is treated primarily as a diagnostic tool for critique, auditing, and monitoring, while transparency-by-design is evaluated as a primary modelling target for high-stakes use. We conclude with a research and governance agenda centred on domain-grounded concept resources, clinically feasible counterfactuals with validity checks, curated exemplar libraries, and prospective human-in-the-loop evaluation of transparency as an intervention.
Alkhnbashi et al. (Tue,) studied this question.