When ChatGPT Meets Vulnerability Management: The Good, the Bad, and the Ugly

Vulnerability management is a very challenging and time-consuming task. For many organizations, security operators need to learn about the properties of vulnerabilities to prioritize and mitigate them. Due to the lack of automated tools for vulnerability assessment, operators usually manually search for and read related information from sources online. Recent advances in large language models, like ChatGPT, open up an opportunity for time savings and may prompt operators to use these models as vulnerability information sources. In this work, we evaluate the ability of ChatGPT and several of its siblings to accurately answer user questions about vulnerability properties as well as to provide information for how to mitigate a vulnerability. We also explore their summarization capabilities when multiple vulnerability advisory documents are provided. We find that the models perform poorly on information retrieval tasks, but they perform quite well on summarization.