Automated tissue analysis and differentiation using intraoperative multispectral analysis with a digital surgical microscope

The success of complex surgical procedures, particularly in the head and neck, depends on the expertise and experience of the surgeon. Despite the utmost care, iatrogenic injuries to structures located or running in the surgical area, such as cranial nerves and vessels, can occur. To minimize these risks, an in-vivo multispectral analysis tool is integrated into a fully digital surgical microscope. The multispectral analysis enables non-invasive and automated simultaneous analysis and differentiation of tissues during the surgical procedure. This helps to reduce intraoperative complications and improve the surgeon's learning curve. The method is based on the directional, individual control of four different LEDs in the microscope and the three RGB color channels on the sensor and uses the visual differentiation of different optical tissue behaviors. In 47 planned neck dissections and parotidectomies in the period from December 2021 to October 2023, the occurring tissue structures glandular tissue, nerves, vessels and connective tissue as well as muscles were annotated and the optical tissue behavior was assigned. A 3D Convolutional Neural Network was trained with the annotated data, whereby individual patients were previously excluded from the training data set for evaluation ('leave-one-subject-out'). All remaining data was randomly divided into a training (70%), validation (15%) and test set (15%) for each tissue type. In the evaluation on the random test set, an overall accuracy of 98.8% was achieved with sensitivities for the five tissue types of 98.0% to 99.5%. On the set of previously excluded patients, an accuracy of 77.3% was achieved with sensitivities up to 93.8%.