Improve data management in register-based research: Transition from CSV to Parquet

Aims: To identify an efficient file format for data delivery from large administrative registers for research, testing the full workflow from data extraction to research data management. Methods: Through collaboration between a data delivery and a research department within the same institute, we evaluated each step from data extraction and delivery to management and usage, comparing CSV and Parquet formats. Results: Switching from the unstructured, text-based CSV format to the highly structured Parquet format significantly optimized all processes by reducing file sizes and saving processing time. The Parquet format also provided access to advanced data management techniques, simplifying further work. Despite these advantages, the basic programming required for Parquet format is not very different from that for CSV. We provide a tutorial and examples as online supplement. Conclusions: We strongly recommend replacing CSV files with contemporary data formats. The Parquet file format proved to be an excellent option throughout the entire process from data extraction to research implementation.