Measuring lip movement outside the laboratory using face-tracked digital video

Direct measurement of speech articulation can furnish information unavailable from the acoustic signal. Historically, articulatory measurement required running participants in a laboratory using specialized (and often expensive) equipment. Recent advances in markerless pose estimation allow measuring lip articulation from simple digital video. While video does not afford the millimeter precision or multi-articulator imaging offered by other approaches, it is accessible, portable, and requires no special preparation (e.g., sensor application). This talk will review techniques co-developed by the author to be used with publicly available, cost-free face-trackers (OpenFace 2.0 and Google MediaPipe’s FaceMesh module). The talk will use two recent studies to showcase how these techniques can be deployed under quite unusual conditions. In the first study Broś & Krause, 2024, a laptop with webcam was used to obtain field recordings of native Gran Canarian Spanish speech, inside participants’ homes. Analysis revealed important articulatory correlates of obstruent lenition. In the second study Krause & Tilsen, in prep, American English speakers participated remotely over the Internet, while their lip movements were tracked through their own webcams. Preliminary results from this study offer novel insights into the labial motor control strategies used for both labial and non-labial consonants.