Low-latency deep learning framework for real-time conventional Pap smear cytology using YOLO-based instance segmentation

Purpose Compact YOLOv11 nano/small variants were trained on public cervical cytology datasets, harmonized via Bethesda labeling for cells and nuclei at high resolution. Models performed joint detection and segmentation, assessed by mAP, recall, and F1-score per category. Real-time use was tested on video from CMOS cameras on a microscope, tracking inference time and frame rate. Methods Compact YOLOv11 nano and small variants were trained on public cervical cytology images. These datasets were aligned using a unified Bethesda labeling scheme for whole cells and nuclei at high resolution. The models were trained for joint box detection and segmentation, and their results were assessed via mAP, recall, and F1-score for each Bethesda category. The real-time use of the models was tested by deploying them on video streams from low-cost CMOS cameras on a microscope, recording the per-frame inference time and frame rate. Results The YOLOv11 small model achieved high mAP and recall for both bounding-box and mask predictions across Bethesda classes while maintaining low latency for real-time applications. Inference on microscope video sequences showed frame rates above the 25 fps threshold for real-time performance, even under tough conditions with overlapping cells and complex inflammatory or hemorrhagic backgrounds. Conclusion These findings suggest that lightweight YOLOv11-based models can offer fast and reliable analysis of conventional Pap smears directly at the microscope, which can support the development of affordable, point-of-care computer-assisted screening tools for cervical cancer, notably in low- and middle-income countries where access to liquid-based cytology and whole-slide scanners is limited.