Mechanical model of lips with a loudspeaker mimicking human speech production

We focused on lip movement and physically implemented a mechanical lip model by combining it with an actuator and a loudspeaker. To acquire data on lip movements, we first extracted facial landmarks using FaceMesh, a machine-learning library from MediaPipe provided by Google. We used these coordinates to create an animation of the lips and then moved the actual lip model accordingly. Our lip model consisted of upper and lower lips, with a linear actuator attached to the lower lip, enabling the lips to open and close by controlling the actuator. A loudspeaker was placed behind the lips to produce speech sounds through the mouth opening. The actuator's movement was controlled using a MATLAB program, which also played the corresponding speech sounds synchronized with the lip movement. We conducted an evaluation experiment using phrases such as “Bai-bai (bye-bye).” The results showed that, although there were individual differences, half of the respondents preferred placing the loudspeaker behind the mouth opening rather than outside the mouth at a distance of approximately 120 mm. [Work supported by JSPS KAKENHI Grant No. 24K06423.]