Deep Learning Based Inference of Private Information Using Embedded Sensors in Smart Devices

Smart mobile devices and mobile apps have been rolling out at swift speeds over the last decade, turning these devices into convenient and general-purpose computing platforms. Sensory data from smart devices are important resources to nourish mobile services, and they are regarded as innocuous information that can be obtained without user permissions. In this article, we show that this seemingly innocuous information could cause serious privacy issues. First, we demonstrate that users' tap positions on the screens of smart devices can be identified based on sensory data by employing some deep learning techniques. Second, it is shown that tap stream profiles for each type of apps can be collected, so that a user's app usage habit can be accurately inferred. In our experiments, the sensory data and mobile app usage information of 102 volunteers are collected. The experiment results demonstrate that the prediction accuracy of tap position inference can be at least 90 percent by utilizing convolutional neural networks. Furthermore, based on the inferred tap position information, users' app usage habits and passwords may be inferred with high accuracy.