Recognition of Human Computer Operations Based on Keystroke Sensing by Smartphone Microphone

Human computer operations such as writing documents and playing games have become popular in our daily lives. These activities (especially if identified in a non-intrusive manner) can be used to facilitate context-aware services. In this paper, we propose to recognize human computer operations through keystroke sensing with a smartphone. Specifically, we first utilize the microphone embedded in a smartphone to sense the input audio from a computer keyboard. We then identify keystrokes using fingerprint identification techniques. The determined keystrokes are then corrected with a word recognition procedure, which utilizes the relations of adjacent letters in a word. Finally, by fusing both semantic and acoustic features, a classification model is constructed to recognize four typical human computer operations: 1) chatting; 2) coding; 3) writing documents; and 4) playing games. We recruited 15 volunteers to complete these operations, and evaluated the proposed approach from multiple aspects in realistic environments. Experimental results validated the effectiveness of our approach.