Automated Extraction of Physical Examination Interactions in General Practice Consultations: A Multi-Modal Using Images and Text Base Approaches

Teleconsultation, which proliferated during the COVID-19 pandemic, is projected to keep evolving in primary care; however, virtual physical examinations continue to hinder telehealth development. This study aimed to automatically recognise physical examinations performed during in-person GP consultations by text and image-based recognition. Text-based methods used regular expressions on transcript keywords to identify physical examinations, while image-based methods utilised transfer learning (fine-tuning the Resnet-18 model) to detect physical examination interactions.