An Evaluation of Automated Sentiment Analysis Methods: YouTube comments on the opioid crisis (Preprint)

BACKGROUND Sentiment analysis is the most popular artificial intelligence method to mine and examine text. Popular sentiment analysis tools were used to analyze YouTube comments from videos discussing the opioid epidemic, inform health policy, and improve population health. OBJECTIVE This study compared manually coded sentiment scores to scores from three automated sentiment analysis tools: VADER, TEXT2DATA, and the tone summary measure from LIWC-22. METHODS Evaluation methods included descriptive statistics, ROC curves, confusion matrices, Cohen’s kappa, accuracy, specificity, precision, sensitivity (recall), and F1 harmonic means. RESULTS Overall, LIWC’s tone sentiment analysis measure most closely matched manual coding (88% F1 score), followed by VADER (83% F1 score) and TEXT2DATA (82% F1 score). CONCLUSIONS We recommend applying these tools to social media analysis, considering imbalances in positive and negative affect, comment length, and other practical considerations, such as costs and programming needs.