An experiment in forecasting out-of-sample coup events with artificial intelligence

Out-of-sample forecasting of political violence is now common among academics and security practitioners, with varying degrees of success. While forecasting methods range from informal expert estimates to “big data” algorithmic models, large-language models (LLMs) are only now emerging as a potential tool to supplement these established methods. LLMs, however, are known to be sensitive to variations in prompt narrative style, prompt syntax, randomization parameters (i.e. “temperature”), response length limits, whether the platform accesses its previous responses, and other manipulable variables. This experiment compares out-of-sample forecasts in ChatGPT-4 across 10 different prompt narratives designed to extract responses about Sub-Saharan African states most likely to experience coups beginning in May 2023, the first month past the LLM’s training data, through the end of the year. Two subsequent experiments take the most accurate forecasting prompts and re-test these with variations to the LLM’s temperature and response length; a final series of experiments examines language bias by comparing English to French inputs. Cumulatively, these experiments find significant but not conclusive support for AI’s potential as a political forecasting tool.