Not all tokens are created equal: Perplexity Attention Weighted Networks for AI-generated text detection

The rapid advancement in large language models (LLMs) has enhanced their ability to generate coherent text, raising concerns about misuse and making detection critical. However, this task remains challenging, particularly in unseen domains or with unfamiliar LLMs. Leveraging next-token distribution outputs offers a theoretically appealing detection approach, as they reflect the models’ extensive pre-training. Despite this promise, zero-shot methods using these outputs have had limited success. We hypothesize this is partly due to using the mean to aggregate next-token metrics across tokens, ignoring that some tokens are inherently easier or harder to predict. We propose the Perplexity Attention Weighted Network (PAWN), which uses last hidden states and positions to weight features derived from next-token distribution metrics across the sequence. Though not zero-shot, our method allows caching hidden states and metrics to reduce training resource needs. PAWN performs competitively or better in-distribution than the strongest baselines (fine-tuned LMs) using a fraction of their trainable parameters. It also generalizes better to unseen domains and source models, with less variability in the decision boundary under distribution shifts. PAWN is more robust to adversarial attacks and, with a multilingual backbone, generalizes well to languages unseen in supervised training, with LLaMA3-1B reaching a mean macro F1 score of 81.46% in cross-validation across nine languages. • We propose a novel AI-generated text detection framework PAWN. • PAWN uses a frozen LLM backbone, needing minimal resources. • Competitive performance in domains and generative models seen during training. • Improved generalization to new domains and generative models. • More robust to adversarial attacks and can generalize to new languages.