ARssist: augmented reality on a head‐mounted display for the first assistant in robotic surgery

In robot‐assisted laparoscopic surgery, the first assistant (FA) is responsible for tasks such as robot docking, passing necessary materials, manipulating hand‐held instruments, and helping with trocar planning and placement. The performance of the FA is critical for the outcome of the surgery. The authors introduce ARssist , an augmented reality application based on an optical see‐through head‐mounted display, to help the FA perform these tasks. ARssist offers (i) real‐time three‐dimensional rendering of the robotic instruments, hand‐held instruments, and endoscope based on a hybrid tracking scheme and (ii) real‐time stereo endoscopy that is configurable to suit the FA's hand–eye coordination when operating based on endoscopy feedback. ARssist has the potential to help the FA perform his/her task more efficiently, and hence improve the outcome of robot‐assisted laparoscopic surgeries.