Do People Mirror Emotion Differently with a Human or TTS Voice? Comparing Listener Ratings and Word Embeddings

Speakers increasingly communicate with technology using spoken language and display behaviors from human-human interaction, such as mirroring the pronunciation patterns of text-to-speech (TTS) voices. Yet, the magnitude and directionality of mirroring varies across prior studies comparing human and TTS addressees. The current study uses two approaches to assess mirroring of emotionally expressive speech produced by a human and a TTS voice. We compare AXB perceptual similarity (holistic human listener judgment, n=109 raters), and distance in latent space (wav2vec 2.0) on a dataset of single word shadowing in a cohort of 36 United States English speakers. Results show that both AXB and wav2vec 2.0 capture mirroring toward emotional prosody for both human and TTS voices. We discuss these findings in terms of theories of computer personification and emotional mirroring, as well as their contributions toward methodological advancements in phonetic entrainment.