Artifact evaluation (AE) is gaining traction across the computer science community as a means of advancing reproducible research and strengthening readers’ confidence in published results. Applying reproducibility to computer systems and architecture research has proven particularly challenging, and the FPGA community faces its own distinct hurdles — namely, the use of non-standard hardware platforms and dependence on specific software tools and versions. In this editorial, we review the history of AE in the FPGA community, compare it to practices in related fields, and discuss challenges and future directions. To date, AE has meaningfully improved the availability and accessibility of artifacts and their documentation, while also increasing readers’ confidence in published findings. Looking ahead, AE is poised to continue growing across the reconfigurable hardware community. Notably, ACM Transactions on Reconfigurable Technology and Systems will now offer artifact evaluation with the opportunity to earn artifact badges for all accepted papers.
Miriam Leeser (Sat,) studied this question.