Analyzing Cyber Security Research Practices through a Meta-Research Framework

Sound research practices are the foundation of valid, reliable, and trustworthy research results. The discipline of meta-research critically evaluates research practices and proposes new methods to improve and refine the way in which research is conducted. In this paper, we apply the framework by Ioannidis et al. 59 for categorizing meta-research work that analyzes cyber security research practices, with the goal of gaining a better understanding on the research community’s efforts to examine its own research practices. We use this framework to characterize which areas of meta-research are most commonly studied, and which areas receive the most attention in terms of developing and enforcing improved research practices. We also compare these meta-research findings with experiences from another academic community, in this case the Internet measurement community, to observe areas where academic communities can learn from each other. Our work is meant as an encouragement for the research community to continue its self-reflective practices, and we hope that it can contribute to these ongoing efforts to improve cyber security research.