Automated Coral Detection and Instance Segmentation in Underwater Imagery: Evaluating YOLO Model Performance and Transferability to Mesophotic Coral Ecosystems

Automated coral image analysis remains challenging because methods must resolve individual colonies while preserving taxonomic information. We present a YOLO-based in-stance-segmentation framework trained on more than 9,500 hand-annotated images of Caribbean scleractinian corals representing 57 species and 24 genera. Four architectures, YOLOv8x, YOLOv9e, YOLOv11x, and YOLOv12x, were evaluated under consistent training conditions at both species and genus levels. On the held-out test set, species-level mask mAP₅₀ reached 0.789 and genus-level mask mAP₅₀ reached 0.870 (both YOLOv9e), with inference times of 4.1–5.5 ms per image. YOLOv8x, YOLOv9e, and YOLOv11x performed similarly (mask mAP₅₀ within 0.01 of one another), with YOLOv8x and YOLOv11x the fastest, while YOLOv12x showed no improve-ment despite added architectural complexity. Segmentation performance varied among taxa and growth forms, with compact or visually distinct colonies generally producing stronger masks than branching, encrusting, low-contrast, or substrate-integrated forms. External model transfer was evaluated on independent mesophotic coral ecosystem imagery using published results and expert review. Expert-confirmed precision was higher at genus level than species level and in-creased with confidence filtering, reaching 82.8% for genus-level and 71.1% for species-level pre-dictions at confidence >0.90. These results indicate that YOLO-based instance segmentation can produce rapid, taxonomically resolved coral observations, but external applications require con-fidence-based filtering and expert review, particularly for species-level inference in visually com-plex reef imagery.