User Inference Attacks on Large Language Models

Text written by humans makes up the vast majority of the data used to pre-train and finetune large language models (LLMs).Many sources of this data-like code, forum posts, personal websites, and books-are easily attributed to one or a few "users".In this paper, we ask if it is possible to infer if any of a user's data was used to train an LLM.Not only would this constitute a breach of privacy, but it would also enable users to detect when their data was used for training.We develop the first effective attacks for user inferenceat times, with near-perfect success-against LLMs.Our attacks are easy to employ, requiring only black-box access to an LLM and a few samples from the user, which need not be the ones that were trained on.We find, both theoretically and empirically, that certain properties make users more susceptible to user inference: being an outlier, having highly correlated examples, and contributing a larger fraction of data.Based on these findings, we identify several methods for mitigating user inference including training with example-level differential privacy, removing within-user duplicate examples, and reducing a user's contribution to the training data.Though these provide partial mitigation, our work highlights the need to develop methods to fully protect LLMs from user inference.Pre-trained LLM Finetuned LLM !User-level finetuned data Training samples Samples known by attacker Query access Adversary Target User 2. For each (#) compute !( (#) ) 3. Test statistic 4 (%) , , (&) = % & #'% & log ( ! () (#) ) ( %&' () (#) ) 4. was in training if 4 (%) , , (&) > 1. Sample (%) , , & from +