An efficient and scalable real-time health data processing model

Big Data and Cloud Computing has became important technologies in the information technology (IT) industry. Huge volumes of data are generated daily from various sources. The datasets are so large and complex, Traditional methods are difficult to deal with them. These data focuses on storing, processing, and analysing these large and complex datasets, while Cloud Computing provides infrastructure which is cost effective and scalable. Many areas, including business, healthcare, and education, use Big Data to make decisions and work more efficiently. In healthcare, it helps to cut down the cost of treatments and predict diseases when they are going to spread and take care of people before they get into sick. The proposed system integrates a serverless data warehouse to improve system scalability and Platform as a Service provides querying to retrieve data and processing through data pipelines. Different kinds of experiments were conducted on real world datasets to measure average, read, compute and write times across different datasets and its sizes it enhances the system stability and visibility, Centralized monitoring and logging implemented to track the resource utilization and system health check-up in real-time. The Experiment results shows that Extract-Load-Transform(ELT) is achieving lower latency and high throughput that Extract-Transform-Load(ETL). The Proposed ELT methodology achieves 12% higher processing efficiency and latency reduce by 35% compared to ETL.