Semantic content outperforms speech prosody in predicting affective experience in naturalistic settings

Many commercial products use algorithms to recognize affect based on speech prosody (i.e.,voice acoustics). These algorithms are typically trained on enacted or labeled speech samplescollected in lab settings. However, they are used to infer affective experiences occurring ineveryday life. Here, we investigate whether the experience of affective states can be predictedfrom speech samples collected using smartphones in naturalistic settings. In two field studies(experimental Study 1: N = 409; observational Study 2: N = 687), we collected 25,403speech samples from participants along with their self-reported affective experiences.Machine learning analyses show that prosody reveals limited affective information (r_md =.17) and is outperformed by semantic content (r_md = .33). Our findings demonstrate theimportance of semantic content and challenge whether previously reported predictionperformances for affective expression from prosody in controlled settings generalize to therecognition of subjective affective experience in naturalistic settings.