Style matters: Writing styles of specific authors are a key factor in determining the perceived credibility and quality of AI-generated and human-written poetry

: Previous research (Köbis & Mossink, 2021) found a 50% misclassification rate between AI-generated and human-written poems, indicating chance level, but a growing body of literature is still needed. Thus, this study aims at generalizing these findings using both a fine-tuned GPT-2 model (Study 1) and a base GPT-2 model allowing for human re-editing (more common use case; Study 2). Poems from classic authors with distinct styles, such as Kafka, Gernhardt, Jandl, Celan, and Hölderlin, were used. Participants were presented the first lines of original poems followed by either AI-generated or original continuations. Their task was identifying the source (AI-generated vs. human-written originals) and evaluating the texts on writing quality, inspiration, fascination, interest, and aesthetics. In Study 1, the misclassification rate was 30% for human-written and 34% for AI-generated poems. In Study 2, allowing for human re-editing, the misclassification rate was 34% for human-written and 40% for AI-generated poems. These rates were lower than those reported by Köbis and Mossink (2021). Results varied significantly by author: In Study 2, misclassification rates were 57% for AI-generated Kafka continuations and 28% for Kafka originals, whereas for vocal style poems by Gernhardt and Jandl they were 30% for AI-generated continuations and 55% for originals. Participants generally preferred human-written texts over AI-generated ones in all evaluated aspects and were overconfident in their performance. This study extends Köbis and Mossink’s (2021) work, showing lower misclassification rates, but particularly highlighting the influence of specific authors and writing styles on the results.