The proper use of t -tests in computational stylistics

The t-test is an important and commonly used statistical tool that produces a probability known as a p-value. In computational stylistics, a t-test is often used to make sense of the data arising from counting the frequencies of certain words in different texts. These frequencies may vary across different texts from different eras, in different genres, and by different authors. In a recent essay in Digital Scholarship in the Humanities, Pervez Rizvi argued that Hugh Craig had, in an article published elsewhere, misused the t-test in an attempt to show that in early modern plays word-choice frequencies vary more significantly across different authors than they vary across writings in different genres by one author. The present essay shows that Craig in fact used t-test’s responsibly, that Rizvi’s essay misrepresents this branch of statistics, and that Rizvi recurrently committed a well-known statistical ‘howler’—his claim that a p-value tells us how likely it is that our null hypothesis is true—that vitiates his entire critique.