Using a testbed to assess XML Database Performance

We have reviewed the history of XML Benchmarking, and we have argued that the benchmarking of XML Databases has stagnated Paxton 2024 . Consequently we have proposed that by adopting the testbed approach Difallah 2013 Shook 2020 , and adapting it to XML databases, we can bring recent progress in OLTP and NoSQL benchmarking to XML Paxton 2024 . Our goal is to make it easier to run flexible, repeatable, and potentially large benchmarks against any standard-compliant XQuery or XML database implementation in order to perform: Application level benchmarking, to provide a broad characterisation of the performance of a database against a particular workload. Microbenchmarking, in which more tailored queries and workloads are run to characterise in detail the performance of particular code or data paths within a database implementation. Microbenchmarks are often run in the presence of profiling tools which can analyse detailed CPU and memory usage patterns of a system while it is executing a workload. We have implemented the 2 key components necessary for adopting the NoSQLBench testbed in XML Database benchmarking: An adapter for NoSQLBench, to drive XML databases. NoSQLBench Shook 2020 is a testbed for benchmarking NoSQL databases. It has an extensible adapter-based architecture for interacting with different NoSQL database APIs. Our adapter implements the XML:DB standard XML:DB . A tool xmlgen2 for the efficient generation of synthetic XML data which can be ingested into a database as a prelude to benchmarking. Of course where data ingestion is an important part of a workload, the ingestion itself can become part of the benchmark workload. This tool is modelled on the xmlgen tool of Schmidt 2002 but is designed to be flexible enough to generate entirely diverse workloads in a straightforward way. To do this we exploit the powerful Virtual Data Set features which are part of NoSQLBench NoSQLBench . This is an advance over the original xmlgen tool which was only designed to generate one specific XML corpus.