Randomized trial compares accuracy of AI detection tools in higher education, indicating potential for enhanced assessments.
Key Points
This study aims to evaluate the reliability of AI detection tools used in academic settings by comparing their accuracy with different types of academic papers.
Compared four AI detection tools: GPTZero, Pangram, Copyleaks, and Turnitin.
Used a synthetic dataset of 160 documents with known ground truth values to assess detection accuracy.
Applied Pangram to 1,163 master’s theses submitted in academic year 2024–2025 for real-world analysis.
Pangram consistently outperformed other tools in detecting fully AI-generated and hybrid texts.
All tools correctly identified fully human-written texts, with Pangram flagging 45.5% of the master’s theses analyzed.
False positives were rare across all tools, indicating improvement compared to previous studies.