First-in-man experience with real-time holographic mixed reality display of three-dimensional echocardiography during structural intervention: balloon mitral commissurotomy

Three-dimensional transoesophageal echocardiography (3DTOE) is widely used as an aid during structural percutaneous cardiac interventions. However, displaying 3D information on standard flat monitors is limited in this setting. Therefore, we developed the method of real-time streaming of 3DTOE data into head-mounted mixed-reality holographic display allowing for touchless control and data sharing within cath-lab. The method was tested for the first time in human during percutaneous mitral balloon commissurotomy (PMC). PMC was performed using Inoue balloon in a 74-year-old woman with New York Heart Association (NYHA) class III rheumatic mitral stenosis, 43 years after surgical valvuloplasty (area 1.3 cm2, mean gradient 8 mmHg, conscious sedation with midazolam/fentanyl) and monitored with real-time 3DTOE (Vivid e95, GE Healthcare, Horten, Norway). Raw 3D data were streamed from standard echocardiograph using custom connection to 3D DICOM viewer workstation (CarnaLife Holo, MedApp, Krakow, Poland) for real-time, dynamic 3D rendering and wirelessly transferred into HoloLens mixed reality display (Microsoft, Redmond, USA) to overlay non-obstructive 3D data hologram upon reality view. Data were visible as a semitransparent holographic cube positioned in a convenient sector of visual field of echocardiographist and shared by interventional cardiologist (Figure). Hologram could be controlled (e.g. zoom, position, cropping) in a touchless manner by voice commands and gestures recognized by HoloLens without perceptible delay. Specific views were used to assess preprocedural anatomy, support transseptal puncture, positioning of wires, orifice crossing, and balloon positioning for inflation, and later, to verify outcomes. The procedure was successfully completed within 60 min with 81 mGy irradiation, with 1.8 cm2 final mitral valve area. In conclusion, we demonstrate for the first time the feasibility of real-time intraprocedural use of holographic mixed reality, displaying 3DTOE data stream with touchless control over data shared by echocardiographist and invasive cardiologist, to support percutaneous mitral valve structural intervention. Implementation of holographic mixed reality 3DTOE data display in cath-lab. Real-time 3DTOE imaging is transferred live into CarnaLife®Holo workstation wirelessly feeding head-mounted HoloLens display. Bottom row shows, along with conventional 3DTOE echo display and fluoroscopic data, a synthetic concept of a 3DTOE data streamed into hologram visible (only to personnel carrying HoloLens, as exemplified by the presented echocardiographist applying a touchless control gesture) simultaneously with real world perception. We gratefully acknowledge the technical cooperation of MedApp and GE Healthcare teams (Poland, Norway)—special credits to Bartosz Machura, Krzysztof Janc, Gunnar Hansen, Eigil Samset, Katarzyna Olszowska-Pawluczuk, and Tomasz Ogonowski.