Estimation of Behavior in a Bed using Deep Learning with Multimodal Input of Acoustic Spectrum and Sound Source Image

We are developing a smart bed system with the aim of improving the quality of nursing care. Here we report on a system that estimates behavior of target persons from the sounds they make in bed, which is one of the main elements of this system. When estimating behavior from sounds, not only the spectral information but also the location and movement of the sound source are important information. For this reason, we devised a method to generate a two-dimensional distribution of sound sources, and we performed long short term memory deep learning using a sound source image sequence and a mel spectrogram as multimodal input and evaluated the behavior estimation performance. As a result, we obtained generally good estimation results, but it became clear that there were issues with the accuracy of identifying the location of scratching.