Augmented reality with Microsoft HoloLens holograms for near infrared fluorescence based image guided surgery

Near infrared fluorescence (NIRF) based image guided surgery aims to provide vital information to the surgeon in the operating room, such as locations of cancerous tissue that should be resected and healthy tissue that should to be preserved. Targeted molecular markers, such as tumor or nerve specific probes, are used in conjunctions with NIRF imaging and display systems to provide key information to the operator in real-time. One of the major hurdles for the wide adaptation of these imaging systems is the high cost to operate the instruments, large footprint and complexity of operating the systems. The emergence of wearable NIRF systems has addressed these shortcomings by minimizing the imaging and display systems’ footprint and reducing the operational cost. However, one of the major shortcomings for this technology is the replacement of the surgeon’s natural vision with an augmented reality view of the operating room. In this paper, we have addressed this major shortcoming by exploiting hologram technology from Microsoft HoloLens to present NIR information on a color image captured by the surgeon’s natural vision. NIR information is captured with a CMOS sensor with high quantum efficiency in the 800 nm wavelength together with a laser light illumination light source. The NIR image is converted to a hologram that is displayed on Microsoft HoloLens and is correctly co-registered with the operator’s natural eyesight.