SBOM Generation Tools Under Microscope: A Focus on The npm Ecosystem

Generating accurate Software Bill of Materials (SBOM) is challenging due to the complex dependencies in the diverse components used in software and also the way software is built into executables. A handful of tools claim the capability of automatic SBOM generation from software distributions while little is known about their applicability, strengths, and limitations. Our study makes quantitative and qualitative comparisons of the four such tools (i.e., ORT, cnn, syft, cdxgen) that claim to be capable of generating SBOM from JavaScript projects. For the comparison, we operate these four tools on 50 open-source JavaScript npm projects. We find significant performance variations when evaluating their ability to extract component details, especially in detecting dependencies. The findings of this study are useful in the design and development of SBOM generator tools, in end-users' selections of such tools, and thus in the overall improvement of the security and transparency in software supply chain.