Multi-modal sorting in plastic and wood waste streams

Addressing the escalating waste crisis necessitates innovative waste management strategies, particularly valorisation techniques, the efficiency of which is dictated by the purity of the feedstock. In order to mitigate the segregation challenges encountered in complex and non-homogeneous waste streams, this work proposes a vision-based architecture ample for effective sorting of parts based on shape and material-related properties. The proposed work encapsulates a novel deep learning multi-modal approach, in which multiple parallel auto-encoders are used to extract spatio-spectral information from the RGB and multi-spectral sensors and project them in a common latent space. By decoding the latent space representations, the class of each object is picked out, thus guiding the robotic sub-system accordingly. To support the proposed deep architecture, a dataset, called Multispectral Mixed Waste Dataset (MMWD) was produced, containing multi-spectral data from the visible (16 bands), near-infrared (25 bands) regions of the electromagnetic spectrum and RGB (3 bands) data. The dataset includes the following seven plastic and wood wastes: Polypropylene (PP), PolyEthylene Terephthalate (PET), Low-Density PolyEthylene (LDPE), High-Density PolyEthylene (HDPE), Medium Density Fibreboard (MDF), Melamine Faced Chipboards (MFC), and Oak Veneer samples. For the localisation of waste along the conveyor belt, YOLO v8 is used to achieve 99.5% mean average precision (mAP50). In the classification task, where the multi-modal approach was followed, the overall accuracy achieved is 96% with the prediction recall being greater than 95% for the majority of classes under examination.