mz5: Space- and Time-efficient Storage of Mass Spectrometry Data Sets

Across a host of MS-driven-omics fields, researchers witness the acquisition of ever increasing amounts of high throughput MS data and face the need for their compact yet efficiently accessible storage. Addressing the need for an open data exchange format, the Proteomics Standards Initiative and the Seattle Proteome Center at the Institute for Systems Biology independently developed the mzData and mzXML formats, respectively. In a subsequent joint effort, they defined an ontology and associated controlled vocabulary that specifies the contents of MS data files, implemented as the newer mzML format. All three formats are based on XML and are thus not particularly efficient in either storage space requirements or read/write speed. This contribution introduces mz5, a complete reimplementation of the mzML ontology that is based on the efficient, industrial strength storage backend HDF5. Compared with the current mzML standard, this strategy yields an average file size reduction to ∼54% and increases linear read and write speeds ∼3–4-fold. The format is implemented as part of the ProteoWizard project and is available under a permissive Apache license. Additional information and download links are available from http://software.steenlab.org/mz5. Across a host of MS-driven-omics fields, researchers witness the acquisition of ever increasing amounts of high throughput MS data and face the need for their compact yet efficiently accessible storage. Addressing the need for an open data exchange format, the Proteomics Standards Initiative and the Seattle Proteome Center at the Institute for Systems Biology independently developed the mzData and mzXML formats, respectively. In a subsequent joint effort, they defined an ontology and associated controlled vocabulary that specifies the contents of MS data files, implemented as the newer mzML format. All three formats are based on XML and are thus not particularly efficient in either storage space requirements or read/write speed. This contribution introduces mz5, a complete reimplementation of the mzML ontology that is based on the efficient, industrial strength storage backend HDF5. Compared with the current mzML standard, this strategy yields an average file size reduction to ∼54% and increases linear read and write speeds ∼3–4-fold. The format is implemented as part of the ProteoWizard project and is available under a permissive Apache license. Additional information and download links are available from http://software.steenlab.org/mz5. MS data are acquired on a wide variety of mass analyzer technologies and brands that deliver data sets in various proprietary data formats and make use of a multitude of architecture-dependent libraries. In the past, this situation has severely complicated the development and application of alternative, vendor-independent data analysis pipelines (1Kessner D. Chambers M. Burke R. Agus D. Mallick P. ProteoWizard: Open source software for rapid proteomics tools development.Bioinformatics. 2008; 24: 2534-2536Crossref PubMed Scopus (1217) Google Scholar, 2Deutsch E.W. Mendoza L. Shteynberg D. Farrah T. Lam H. Tasman N. Sun Z. Nilsson E. Pratt B. Prazen B. Eng J.K. Martin D.B. Nesvizhskii A.I. Aebersold R. A guided tour of the trans-proteomic pipeline.Proteomics. 2010; 10: 1150-1159Crossref PubMed Scopus (601) Google Scholar, 3Bertsch A. Gröpl C. Reinert K. Kohlbacher O. OpenMS and TOPP: Open source software for LC-MS data analysis.Methods Mol. Biol. 2011; 696: 353-367Crossref PubMed Scopus (66) Google Scholar), highlighting the need for and fueling the development of a common and open storage format for MS data sets (4Pedrioli P.G. Eng J.K. Hubley R. Vogelzang M. Deutsch E.W. Raught B. Pratt B. Nilsson E. Angeletti R.H. Apweiler R. Cheung K. Costello C.E. Hermjakob H. Huang S. Julian R.K. Kapp E. McComb M.E. Oliver S.G. Omenn G. Paton N.W. Simpson R. Smith R. Taylor C.F. Zhu W. Aebersold R. A common open representation of mass spectrometry data and its application to proteomics research.Nat. Biotechnol. 2004; 22: 1459-1466Crossref PubMed Scopus (652) Google Scholar, 5Orchard S. Montechi-Palazzi L. Deutsch E.W. Binz P.A. Jones A.R. Paton N. Pizarro A. Creasy D.M. Wojcik J. Hermjakob H. Five years of progress in the standardization of proteomics data 4th annual spring workshop of the HUPO-proteomics standards initiative April 23–25, 2007 Ecole Nationale Suprieure (ens), Lyon, France.Proteomics. 2007; 7: 3436-3440Crossref PubMed Scopus (46) Google Scholar). The Proteomics Standards Initiative and the Seattle Proteome Center at the Institute for Systems Biology independently developed the mzData and mzXML formats, respectively (4Pedrioli P.G. Eng J.K. Hubley R. Vogelzang M. Deutsch E.W. Raught B. Pratt B. Nilsson E. Angeletti R.H. Apweiler R. Cheung K. Costello C.E. Hermjakob H. Huang S. Julian R.K. Kapp E. McComb M.E. Oliver S.G. Omenn G. Paton N.W. Simpson R. Smith R. Taylor C.F. Zhu W. Aebersold R. A common open representation of mass spectrometry data and its application to proteomics research.Nat. Biotechnol. 2004; 22: 1459-1466Crossref PubMed Scopus (652) Google Scholar, 6PSI-MS: Mass Spectrometer Standards Working Group (2010) http://www.psidev.info/index.php?q=node/80Google Scholar). They subsequently merged their efforts, leading to the development of mzML (7Martens L. Chambers M. Sturm M. Kessner D. Levander F. Shofstahl J. Tang W.H. Römpp A. Neumann S. Pizarro A.D. Montecchi-Palazzi L. Tasman N. Coleman M. Reisinger F. Souda P. Hermjakob H. Binz P.A. Deutsch E.W. mzML: A community standard for mass spectrometry data.Mol. Cell. Proteomics. 2011; 10 (R110.000133)Abstract Full Text Full Text PDF PubMed Scopus (452) Google Scholar), which features a generic ontology for the representation of MS data. The mzML format is universally applicable, and combined with the readily available open source reference implementation ProteoWizard (1Kessner D. Chambers M. Burke R. Agus D. Mallick P. ProteoWizard: Open source software for rapid proteomics tools development.Bioinformatics. 2008; 24: 2534-2536Crossref PubMed Scopus (1217) Google Scholar), it significantly simplifies data import and export as well as general data handling. Most notably, mzML introduces a controlled vocabulary that enables the addition of novel as well as user-defined data types without requiring changes to the underlying XML schema. As a consequence, laboratories are able to store, analyze, and share MS data using an open exchange format, even for highly specialized workflows (8Orchard S. Data deposition as an integral part of the publication process.J. Proteomics Bioinf. 2009; 2: 334-335Crossref Scopus (1) Google Scholar, 9Jones P. Côté R.G. Cho S.Y. Klie S. Martens L. Quinn A.F. Thorneycroft D. Hermjakob H. PRIDE: New developments and new datasets.Nucleic Acids Res. 2008; 36: D878-D883Crossref PubMed Scopus (123) Google Scholar, 10Desiere F. Deutsch E.W. Nesvizhskii A.I. Mallick P. King N.L. Eng J.K. Aderem A. Boyle R. Brunner E. Donohoe S. Fausto N. Hafen E. Hood L. Katze M.G. Kennedy K.A. Kregenow F. Lee H. Lin B. Martin D. Ranish J.A. Rawlings D.J. Samelson L.E. Shiio Y. Watts J.D. Wollscheid B. Wright M.E. Yan W. Yang L. Yi E.C. Zhang H. Aebersold R. Integration with the human genome of peptide sequences obtained by high-throughput mass spectrometry.Genome Biol. 2005; 6: R9Crossref PubMed Google Scholar, 11Deutsch E.W. Lam H. Aebersold R. PeptideAtlas: A resource for target selection for emerging targeted proteomics workflows.EMBO Rep. 2008; 9: 429-434Crossref PubMed Scopus (444) Google Scholar). Continuous improvements in mass resolution and acquisition speed pose a serious challenge for existing data formats: current MS setups commonly acquire several hundreds of megabytes of data for each run, and space requirements for complete proteomics experiments easily exceed tens of gigabytes. As a consequence, the space and time efficiency of data format implementations have become increasingly critical. Although based on excellent ontologies, relying on the extended markup language (XML) 1The abbreviations used are:XMLextended markup languageHDF5Hierarchical Data Format, version 5I/OInput/Output. 1The abbreviations used are:XMLextended markup languageHDF5Hierarchical Data Format, version 5I/OInput/Output. for the straightforward implementation of mzData, mzXML, and mzML makes for a major efficiency bottleneck. XML was designed to be a human readable, textual data format with considerable inherent verbosity and redundancy. XML was not designed for efficient bulk data storage, and the general modus operandi requires reading complete files to construct the XML parse tree. The mzXML and mzML formats partly circumvent these limitations by using base-64 encoding and (optional) compression of the raw MS scan data in combination with an application-specific indexing system. Despite the improvements gained from these efforts, vendor formats in general outperform mzXML and mzML in terms of space requirements, as well as in read and write efficiency. extended markup language Hierarchical Data Format, version 5 Input/Output. extended markup language Hierarchical Data Format, version 5 Input/Output. This contribution introduces mz5, a novel data representation that combines the merits of the mzML ontology with the efficiency of the Hierarchical Data Format (HDF5) (12HDF Group (2000–2010), Hierarchical data format version 5. http://www.hdfgroup.org/HDF5Google Scholar). HDF5 is an established industrial standard for efficient storage and retrieval of large amounts of complex data and is based on a portable binary representation. HDF5 was specifically designed for large data sets; the key features used in our mz5 implementation are its native support for compression and cached partial read and write access. HDF5 library implementations are available for a range of languages and computational platforms, and the format is distributed under a permissive licensing scheme. Introduced more than 20 years ago, the continued use of HDF5 as the standard format for the NASA earth observing system, as well as its adaption as a prime format for data intensive applications in fields as diverse as astronomy, geology, remote sensing, and avionics (13Millard B.L. Niepel M. Menden M.P. Muhlich J.L. Sorger P.K. Adaptive informatics for multifactorial and high-content biological data.Nat. Methods. 2011; 8: 487-493Crossref PubMed Scopus (52) Google Scholar, 14Anderson K. Alexov A. Baehren L. Griessmeier J.M. Wise M. Renting A. LOFAR and HDF5: Toward a new radio data standard.Int. SKA Forum. 2010; (2010)Google Scholar, 15Bauer B. Carr L.D. Evertz H.G. Feiguin A. Freire J. Fuchs S. Gamper L. Gukelberger J. Gull E. Guertler S. Hehn A. Igarashi R. Isakov S.V. Koop D. Ma P.N. Mates P. Matsuo H. Parcollet O. Pawlowski G. Picon J.D. Pollet L. Santos E. Scarola V.W. Schollwöck U. Silva C. Surer B. Todo S. Trebst S. Troyer M. Wall M.L. Werner P. Wessel S. The ALPS project release 2.0: Open source software for strongly correlated systems.J. Stat. Mech. Theory Exp. 2011; 2011: P05001Crossref Scopus (550) Google Scholar, 16Dougherty M.T. Folk M.J. Zadok E. Bernstein H.J. Bernstein F.C. Eliceiri K.W. Benger W. Best C. Unifying biological image formats with HDF5.Commun. ACM. 2009; 52: 42-47Crossref PubMed Scopus (34) Google Scholar), ensures broad, ongoing use and technical support for many years to come. Our analyses show that mz5 dramatically outperforms the XML-based representations of the mzXML and mzML formats in terms of both space requirements and Input/Output (I/O) speed and is competitive with proprietary vendor formats in terms of space requirements (Fig. 1). Because mz5 is implemented inside the ProteoWizard (1Kessner D. Chambers M. Burke R. Agus D. Mallick P. ProteoWizard: Open source software for rapid proteomics tools development.Bioinformatics. 2008; 24: 2534-2536Crossref PubMed Scopus (1217) Google Scholar) framework, it is already available for use by the mass spectrometry community, and mz5 files can immediately be created from all open or proprietary formats supported by ProteoWizard (Fig. 2).Fig. 2The mz5 format and its relation to common input and output formats. Format conversions that are supported by the ProteoWizard (1Kessner D. Chambers M. Burke R. Agus D. Mallick P. ProteoWizard: Open source software for rapid proteomics tools development.Bioinformatics. 2008; 24: 2534-2536Crossref PubMed Scopus (1217) Google Scholar) framework are represented by solid arrows; format conversions that currently require additional software are shown with dashed arrows. Because the mz5 format has been implemented within the ProteoWizard framework, it can readily be generated from a wide range of proprietary vendor raw formats.View Large Image Figure ViewerDownload Hi-res image Download (PPT) Twenty LC-MS/MS runs of fractionated HeLa S3 cell lysate were acquired in data-dependent acquisition mode on an LTQ/Orbitrap classic (Thermo Scientific) system hyphenated to a nanoflow HPLC system (Eksigent). The survey scan range was limited to m/z 400–2000 at a nominal resolution of 60,000, and MS/MS scans were limited to the eight most abundant precursor ions. The resulting raw data files were converted to mzML using the ProteoWizard (1Kessner D. Chambers M. Burke R. Agus D. Mallick P. ProteoWizard: Open source software for rapid proteomics tools development.Bioinformatics. 2008; 24: 2534-2536Crossref PubMed Scopus (1217) Google Scholar) tool set, mzML files were converted to mz5, and the identity of the mzML and mz5 file contents was confirmed. For all formats, the data were stored in compressed double precision. For the mzXML and mzML formats, indexing was turned on. All of the experiments were conducted using the same hardware, with all resources dedicated to the test runs (no parallel jobs). Overall, we consider the selected data set to be a reasonably common example of LC-MS/MS data acquisition and hence expect that the results generalize to a large majority of practical use cases. To assess mz5 performance, we conducted three types of comparative experiments, testing (i) write performance; (ii) read performance; and (iii) space requirements. For each of the 20 input files, we generated a set of 10 subsets of increasing sizes (sets of 800, 1600, up to 8000 spectra, and the complete set of all spectra present in the respective input file). This strategy yielded 220 test cases. All of the time measurements were averaged over 10 repeats for every test case, after discarding the minimum and maximum timings. For performance comparisons between different formats, we fitted linear models with intercepts and determined relative slopes over time and space (see supplemental materials). mz5 files were used to generate each test case for mzXML, mzML, and mz5, to guarantee that all three file formats contain the same data. To test linear read performance, we measure the time it takes to read each of the test cases from the file it is stored in. Fig. 1a shows the linear read times of all test cases versus the number of (m/z, abundance) pairs. The mz5 format outperforms the mzXML and mzML formats by factors of ∼2.3 and ∼3.7, respectively. We note in passing that the mz5 gains over mzML are not strictly linear; this indicates optimization opportunities for the parsing of smaller mzML files. Interestingly, the mz5 linear read times also exhibit smaller variation compared with mzXML and, in particular, mzML read times. For strictly random reading, mz5 is significantly faster than mzML but performs slightly slower than mzXML. This is due to the underlying HDF5 caching/compression tradeoff. We focused on the vast majority of conceivable use cases and geared the tradeoff toward efficient linear reading. The random read and initialization performance tests, as well as benchmarks for different mass analyzer types, can be found in the supplemental materials. To test write performance, we extracted every test case from the respective mz5 input file and measured the time it takes to write the data to disk, using mzXML, mzML, and mz5. Consequently, potential read time delays only cause a constant offset that is identical between the different formats and does not affect the slopes. supplemental Fig. 1a illustrates the results for all 220 test cases. Time measurements for mzXML, mzML, and mz5 are shown as diamonds, triangles, and circles, respectively. It is apparent that mz5 is consistently faster than mzXML and mzML. The read time-corrected relative slopes show improvements of a factor of ∼4.7 for mz5 over mzXML and ∼3.9 over mzML (Table I).Table IAverage read/write times and space requirements for 106 (m/z, abundance) pairs for the mz5, mzML, and mzXML file formatsmz5mzMLmzXMLmzML/mz5mzXML/mz5tlr0.130.490.33.652.28tw0.632.082.463.283.9tw − tlrmz50.51.952.333.894.66Space7.5713.7214.131.811.87 Open table in a new tab Fig. 1b illustrates storage consumption versus the number of (m/z, abundance) pairs. It is evident that the mz5 format requires little more than half the disk space (54%) of the mzXML and mzML formats. The mzXML and mzML storage requirements are practically identical. Fig. 1c illustrates that mz5 even supersedes XML-based formats that are compressed at the file and spectrum levels for off-line long term storage. Hence, mz5 provides better compression levels while still offering immediate data access and avoiding the need for time-consuming file level compression and decompression. Fig. 1c also provides a rough comparison of space requirements for three popular proprietary vendor formats (Thermo raw, Bruker yep, and ABSciex wiff) against the open mzXML, mzML, and mz5 formats. The mz5 format compares very favorably against the Thermo and Bruker formats and is the best open alternative for the ABSciex format. The Bruker yep file data was generated from Escherichia coli whole cell lysate and was acquired on a Bruker amaZon instrument. The ABSciex wiff file data stems from subcellular fractions of a HeLa cell lysate and was analyzed by on an The mz5 format the mzML as implemented in the ProteoWizard a of HDF5 The HDF5 library provides HDF5 The of mz5 is to to an open exchange format with both speed and storage space requirements that existing vendor formats. The practical challenge was to data representations that the use of the HDF5 to the In mz5, we mzML as HDF5 data types, which to data of are stored in of a are compressed and stored to It be that the straightforward representation of mzML in terms of HDF5 is not a an data compression for technical The HDF5 library is already highly for file M. D. J. J. HDF5 for file on and for Data 2010; Google Scholar). The mz5 implementation sizes to the number of raw data bulk are HDF5 compression and to compression our implementation makes use of HDF5 data types that are to in avoiding data types In mz5 scans and m/z measurements in a mass between m/z The a storage reduction of that were by storage. The implementation at features considerable it is to the data of raw data compression level and and HDF5 The provides a for most common use cases. mz5 is currently limited to of (m/z, abundance) and abundance) for and respectively. The to of measurements is and For practical the of the mz5 format is to be the in storage space requirements. proteomics have data storage and and the efficient use of available space can to these Although standard as they are currently are to from read and write we expect that mz5 be of for library which are to become more in the N. D. A. P.A. by 2007; PubMed Scopus Google Scholar, H. Deutsch E.W. Eng J.K. King N. Aebersold R. and of a library for peptide from 2007; 7: PubMed Scopus Google Scholar). mz5 be of for data setups the read and write speeds and storage space requirements by mz5 the efficiency of data exchange between different of a tool and the by data from proprietary vendor formats to an open To the straightforward use of the mz5 format in existing and software we have implemented mz5 within the ProteoWizard application that already makes use of data file access support mz5. This the need for tools or software the ProteoWizard already enables the use of many proprietary and open formats and requires only a implementation on the We have used the combination under and This contribution introduces an efficient open data format for bulk mass spectrometry storage mz5. It combines the merits of an established standard, and the mzML ontology developed by the Proteomics Standards The mz5 format dramatically faster than the mzXML and mzML formats and requires only half (54%) the storage The current mz5 implementation is the ProteoWizard library and for all proprietary data formats supported by ProteoWizard All software is available http://software.steenlab.org/mz5. The mz5 format is a toward extended for MS data storage. on mzML ontology development and this and of large data distributed read and write access for high throughput and novel for large data M. J.A. access for mass spectrometry data Cell. Proteomics. 2011; 10 Full Text Full Text PDF Scopus Google Scholar, M. J.A. for and of proteomics Cell. Proteomics. 2011; 10 Full Text Full Text PDF Scopus Google Scholar). We Chambers from the by of for ProteoWizard support and and for Bruker yep data Download with files