Enabling Efficient Cloud-Based Data Collection System for Smart Home IoT Data Using Data Binning and Lossless Compression

The ever-growing volume of data generated by Internet of Things (IoT) devices necessitates efficient storage and retrieval mechanisms. This paper proposes a data collection system tailored to accommodate the unique characteristics of IoT data. The system aimed at optimizing the storage space within the Firestore, a NoSQL schemeless cloud-based database, by implementing the data binning technique, grouping similar values and replacing the original data with its group name, and the GZIP lossless data compression technique to enhance data retrieval performance to dynamic web pages. The proposed data storage scheme comprises two collections: the Temporary Collection, where formatted data of IoT devices is temporarily stored every one-second interval, and the Archived Collection, where data older than one hour is compressed and archived. We also developed a web-based dashboard to visualize the data of the past hour by extracting a dataset from the Temporary Collection, while historical data is retrieved by decompressing data from the Archived Collection only. Experimental results reveal a compression ratio of 4.538:1 and a space-saving ratio of 0.789, promising substantial cost savings on cloud-based platforms by reducing data size for storage and transmission. With a reasonable decompression time of 11.26 milliseconds and a significantly lower response time of 130.2 milliseconds compared to 1916.36 milliseconds of test cases, the proposed system remarkably improves real-time data processing, contributing to enriched user experiences and informed decision-making within the IoT web-based dashboard.