Slicing and Dictionaries: A New Approach to Medical Big Data

We present SliceDB, a dual-dimensional data partitioning framework that optimizes healthcare data access through strategic row-wise and column-wise slicing coupled with semantic dictionaries, enabling efficient querying of massive clinical datasets on standard computing resources without specialized database expertise. Medical databases are continuously updated and iterated, with the number of observed values and the variety of variables constantly increasing, leading to increasingly rich data content. This results in a significant increase in the size of individual data files, including both an increase in the number of rows (length) and an expansion in the number of columns (width) [1]. For example, in the MIMIC 3.0 database, the number of records in the chartevents file has reached the hundred-million level [2]; the numeric file in the Amsterdam Critical Care Database version 1.0.2 is similarly large [3]. Despite their massive number of records, these files have relatively simple variable structures. For example, the chartevents file primarily contains fields such as patient ID, item, and value. In contrast, some other databases, though having fewer observations, feature a large number of variables. For example, the core file in the first wave (Wave 1) of the ELSA data set contains only 12,099 records but includes as many as 4484 variables [4]. Such a large-scale database faces numerous challenges during querying, cleaning, and processing. Currently, common data access methods include SQL queries, distributed storage system conversion, and specialized data platforms. SQL queries, as the standard method, are versatile but require users to master the SQL language, have a steep learning curve, and suffer from low efficiency for complex queries [5]. Distributed storage systems (e.g., Hadoop) offer strong scalability but come with high deployment and maintenance costs, requiring specialized technical teams, making them unsuitable for most clinical researchers [6, 7]. As a result, ordinary researchers often struggle to independently apply such systems for analysis. To address these challenges, this paper proposes an innovative data processing strategy based on "slicing + dictionary" to enable flexible querying and efficient extraction of medical big data, thereby enhancing data accessibility and practicality. The "slicing + dictionary" method is based on the theory of data decomposition and restructuring, and has been extended and applied to specific application scenarios in medical big data. This method has the following theoretical advantages: (1) It effectively reduces the amount of data processed in a single operation, thereby significantly reducing the consumption of computing resources; (2) It constructs an efficient indexing system through preprocessing, enabling rapid data location and extraction, and significantly improving query efficiency; (3) It maintains clinical relevance during data decomposition and reorganization, ensuring the scientific validity and effectiveness of the analysis results. This method consists of two core components: data slicing and dictionary construction. This study designed a multi-dimensional data slicing strategy to adapt to the needs of different clinical studies. Specifically, it includes: Clinical Dimension Slicing: Data is divided according to parameter types (such as vital signs, laboratory tests, drug use, etc.); Event Dimension Slicing: Data views are constructed around key clinical events (such as intubation, medication, surgery, etc.); Hybrid Dimension Slicing: Composite slices are created by combining multiple features to meet complex analysis needs. The slicing granularity can be flexibly adjusted according to data characteristics and application scenarios to achieve a balance between query performance and storage efficiency. Inspired by distributed database sharding technology, this design introduces clinical semantic considerations to make data slicing more in line with medical research thinking patterns [8, 9]. For example, the chartevents table in the MIMIC database can be sliced according to different clinical measurement items (such as blood pressure, blood oxygen saturation, body temperature, etc.), or drug-related data can be sliced according to drug names. The slicing process supports multi-level granularity, allowing researchers to flexibly adjust the level of detail of the slicing according to actual needs. The dictionary system serves as a bridge connecting user query intentions with data slices, featuring the following key elements: Encoding-Description-Location-Attribute Structure: Each dictionary entry includes standard encoding, clinical description, data location, and attribute information; Multi-level Classification System: Construct a hierarchical dictionary system based on clinical domains, parameter types, measurement methods, etc.; Synonym Mapping: enabling unified retrieval of terms from different terminology systems and expressions; Cross-database compatibility design: enabling mapping and connectivity between different databases for the same clinical concepts. This dictionary system draws on the experience of the Unified Medical Language System in integrating biomedical terminology, adopting a multi-level structure of encoding-description-location-attributes to achieve semantic mapping between clinical concepts and data locations [10]. By integrating multidimensional information such as clinical terms, parameter types, and measurement methods, and establishing cross-database term mapping relationships, researchers can retrieve and analyze data using standardized clinical terms [11, 12]. The aim is to establish a semantic bridge between clinical concepts and data locations, enabling researchers to directly retrieve and analyze data using familiar clinical terms (rather than database field names). Through dictionary indexing, researchers can precisely locate and extract the required specific data subsets, avoiding unnecessary processing of the entire data set, thereby significantly improving the efficiency of retrieval and analysis. Based on the direction of data segmentation, slices can be categorized into vertical slices and horizontal slices, also known as row slices and column slices, or observation slices and variable slices. In scenarios where the volume of data is dominated by the number of rows, that is, typical long-format data, the number of records continues to increase with database version updates, while the number of variables remains largely unchanged. In such cases, using vertical slicing (row slicing/observation slicing) is more appropriate. For example, in the MIMIC database's chartevents file, the number of records expands over time, but the field structure remains largely fixed. Conversely, when the data volume is primarily influenced by the expansion of the number of columns, i.e., when the number of variables is large and the number of records changes little, horizontal slicing (column slicing/variable slicing) is more suitable. For example, in certain social medicine surveys (such as ELSA), although the sample size is limited, the data set becomes extremely large due to the extremely high number of variables. In extremely large datasets where both the number of rows and columns influence the data set size, both longitudinal slicing and horizontal slicing strategies can be employed simultaneously to achieve more flexible and efficient storage and querying. The core advantages of the slicing and dictionary method lie in: reduced resource requirements (by loading slices on demand, enabling ordinary computing devices to process large-scale data), improved query efficiency (accurately locating the required data, avoiding full table scans, and significantly improving query speed), enhanced flexibility (supporting multi-dimensional slicing and customizable dictionaries to adapt to diverse research needs), and cross-database universality (the method design does not depend on specific database structures). These advantages directly address the limitations of traditional analysis methods in medical big data analysis proposed by Binder and Blettner, and as validated by Chrimes and Zamani, appropriate data structure design can effectively solve the core issues of medical data integration and querying [7, 13]. This method also faces several limitations: slice design trade-offs (slice granularity must balance access efficiency and storage redundancy), update and maintenance costs (database updates require synchronized updates to slices and dictionaries), support for non-standard queries (extremely specialized queries may require custom slice strategies), and initial setup effort (initial implementation requires significant effort for slice and dictionary construction). Dinov points out that the processing and analysis of large-scale medical data are characterized by high costs and high complexity. Khan et al. emphasize that the technical challenges of big data processing primarily lie in rapid data growth, transmission speed, diversity, and security. These challenges are also present in the slicing dictionary method and must be fully considered during implementation. This initial investment may be offset by long-term efficiency improvements and enhanced accessibility, particularly for institutions conducting multiple studies [14, 15]. Future research will focus on: automated slicing optimization (developing intelligent algorithms to automatically determine the optimal slicing strategy), dictionary self-learning (continuously optimizing dictionary structure based on user query patterns), cloud-based deployment models (researching cloud-based slicing dictionary services to further lower the usage threshold), and integration with AI/ML workflows (optimizing the integration of slicing dictionary methods with machine learning workflows). With the advancement of natural language processing and machine learning technologies, slicing dictionary methods are expected to further enhance their intelligence and usability. Slicing and dictionary methods offer a new paradigm for addressing the challenges of accessing large-scale medical databases. By partitioning large data files according to clinical dimensions and establishing standardized dictionary mappings, this method has the potential to significantly reduce technical barriers and resource requirements while improving query efficiency and flexibility. The theoretical foundation of this method aligns with existing research findings, indicating that it could enable ordinary clinical researchers to efficiently access and analyze large-scale medical data that was previously difficult to handle, thereby providing an important tool for advancing medical research and evidence-based medical practice. In the future, this method is expected to be extended to broader medical data domains, promoting the democratization and value extraction of medical big data, and ultimately accelerating the transformation from data to knowledge, improving patient outcomes, and optimizing the allocation of medical resources. Future work will focus on the practical implementation and performance validation of the method, as well as its optimization and adaptation for different clinical research scenarios. Jing Zhang: visualization, writing – original draft, writing – review and editing, project administration. Kailai Li: writing – original draft, writing – review and editing, project administration. Yasi Zhang: visualization. Peng Luo: project administration, supervision. The authors have nothing to report. The authors have nothing to report. The authors have nothing to report. Peng Luo hold the position of Executive Editors-in-Chief for iMetaMed. He was excluded from editorial decision-making related to the acceptance of this article for publication in the journal. The authors have nothing to report.