Answers are grounded in this paper and cite their sources.
Medication errors remain a major source of preventable harm, and identifying a pill from a photograph taken in an uncontrolled setting is still difficult because the informative cues-color, shape, surface texture, and imprinted text-are unequally reliable from one image to the next. We present a recognition framework for natural environments in which a fusion module, context-aware cross-modal attention (CACMA), weighs these four modalities separately for every image using a query formed from their joint global context, an explicit confidence branch that estimates per-modality reliability, and a three-way integration of attended, confidence-weighted, and residual features, on top of \(U^2\) -Net segmentation (98.06% validation IoU) and connected-component localization. All results follow a unified end-to-end protocol, with segmentation, localization, OCR, and fusion executed at inference time, over three random seeds on two real-world datasets. On CURE (196 classes) CACMA attains 91.13±0.27% Top-1 accuracy and \(95.50± 0.16%\) mAP, exceeding the published results of Ling et al. and Rádli et al. by 22.83 and 5.41 pp and summation fusion by 3.15 pp ( \(p<0.001\) ); the 0.76 pp margin over weighted summation is not statistically significant, which we report as parity rather than superiority, while CACMA attains the smallest across-seed variability of any method evaluated ( \(± 0.27\) pp). On OGYEI-v2 (112 classes) it reaches \(83.23± 0.73%\) Top-1 accuracy. What fixed-weight fusion cannot reproduce is directly measurable: the confidence assigned to the text modality rises from 0.362 to 0.602 when an imprint is legible ( \(p = 1 × 10⁻⁵⁵\) ), with the released weight redistributed to the visual modalities. A fully fine-tuned ViT-B/16 is not significantly more accurate ( \(92.07± 0.64%\) ; McNemar \(p=0.32\) ) while using 6.4 \(× \) the parameters and offering no modality-level explanation, placing CACMA at a favorable accuracy-efficiency-interpretability operating point. Paired significance testing applied to the ablations shows that no individual component accounts for the result on its own under this segmentation quality; together with calibration, saliency and failure analyses, these tests delimit where multimodal attention fusion helps and where it does not.
No takes yet. Share an insight, caveat, or question.
My et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: