Analysis of a dataset for evaluating semantic relevance between text and audio

In text-to-audio generation (TTA), the relevance between input text and generated audio is an important metric. Prior work created the RELATE dataset, an open-source dataset for the subjective evaluation of relevance, by collecting 11-point scores across three perspectives: REL (overall relevance), IS (inclusion of sound event), and OS (order of sound event). In conventional work, only the REL score was analyzed based on text attributes such as complexity and audio attributes like audio categories (e.g., Animal, Music). This study expands upon that research by comprehensively analyzing all three scores (REL, IS, and OS). Text and audio attribute analyses, consistent with those previously applied to the REL score, were conducted for the IS and OS scores. Additionally, pairwise correlation coefficients among all three scores were computed. Analysis of text and audio attributes indicated that nearly identical trends for REL and IS scores. A moderate correlation of 0.56 was observed between IS and REL scores. On the other hand, IS and OS scores demonstrated a high correlation of 0.78. Furthermore, REL and OS scores showed a moderate correlation of 0.52. These results suggest that their independent assessment is necessary to ensure an appropriate evaluation of relevance.