Designing A Sustainable Marine Debris Clean-up Framework without Human Labels

Marine debris poses a significant ecological threat to birds, fish, and other animal life.Traditional methods for assessing debris accumulation involve labor-intensive and costly manual surveys.This study introduces a framework that utilizes aerial imagery captured by drones to conduct remote trash surveys.Leveraging computer vision techniques, our approach detects, classifies, and maps marine debris distributions.The framework uses Grounding DINO, a transformer-based zero-shot object detector, and CLIP, a visionlanguage model for zero-shot object classification, enabling the detection and classification of debris objects based on material type without the need for training labels.To mitigate over-counting due to different views of the same object, Scale-Invariant Feature Transform (SIFT) is employed for duplicate matching using local object features.Additionally, we have developed a user-friendly web application that facilitates end-to-end analysis of drone images, including object detection, classification, and visualization on a map to support cleanup efforts.Our method achieves competitive performance in detection (0.69 mean IoU) and classification (0.74 F1 score) across seven debris object classes without labeled data, comparable to state-of-the-art supervised methods.This framework has the potential to streamline automated trash sampling surveys, fostering efficient and sustainable community-led cleanup initiatives.