Assessing ChatGPT as a Power Analysis Tool: An Empirical Investigation

Power analysis plays a crucial role in quantitative research for determining adequate sample sizes required to reject null hypotheses. However, conducting power analysis properly can be challenging since the procedures vary for different data analyses and often demand specialized statistical expertise. Although large language models (LLMs) have the potential to address these issues by providing intuitive explanations and guiding researchers through the process to reduce the barriers, the performance of LLMs in conducting power analysis has not been empirically evaluated. Thus, this study aimed to empirically evaluate ChatGPT’s capacity to assist in calculating minimum sample sizes in power analysis, leveraging its capabilities in text and code generation. Using OpenAI API, we conducted a series of evaluation experiments focusing on its capacity to help sample size calculation in power analysis for commonly used statistical tests. Specifically, Experiment 1 assessed ChatGPT's capability in computing the required sample size by proposing multiple strategies to enable GPT to conduct power analysis. Experiment 2 assessed its performance in identifying missing information for sample size calculation when the input information was incomplete. The findings suggest ChatGPT's adequate performance under the guidance of certain strategies for sample size calculation for considered statistical tests. Overall, ChatGPT proved viable in supporting researchers as an alternative to existing power analysis software packages, offering interpretation of results when performing power analysis. We conclude by discussing the potential of ChatGPT and Generative AI in research practice.