What question did this study set out to answer?

Develop and evaluate an end-to-end industrial big data processing framework based on Apache Spark for real-time analysis of heterogeneous Industrial Internet of Things data streams.

August 16, 2026Advanced ElectromagneticsOpen Access

Application Research of Big Data Real-Time Processing Framework Based on Spark in Industrial Data Stream Analysis

View Full Paper

Ask AI

Bookmark

Authors

YLY. LiuZhejiang International Studies University

Discussion

Member takes

Implication

Framework evaluation demonstrates high-throughput real-time processing with low latency in industrial IoT environments, indicating viable low-resource edge-to-cloud analytics.

Key Points

Develop and evaluate an end-to-end industrial big data processing framework based on Apache Spark for real-time analysis of heterogeneous Industrial Internet of Things data streams.
Integrated Spark Structured Streaming, Spark MLlib, and time-series storage to build an ingestion, preprocessing, computation, and visualization pipeline.
Implemented gateway-level edge preprocessing (protocol parsing, outlier filtering, compression) alongside dynamic partition pruning and shuffle optimization across three industrial deployment scenarios.
Conducted multidimensional benchmarking against traditional Spark configurations and Apache Flink using environments with millions of collection points.
Achieved a processing throughput of 89,000 records per second with an end-to-end latency capped within 280 ms.
Attained a 94.2% equipment fault prediction accuracy and reduced data storage costs by 70%.
Delivered over 40% performance improvement over standard Spark setups and reduced CPU, memory, and network I/O resource occupancy by 25% compared to Flink.

Cite This Study

Y. Liu (2026) studied this question.

synapsesocial.com/papers/6a81796af2fb91fc834ac8c3 https://doi.org/10.7716/aem.v15i3.3233

View Full Paper

Ask AI

Bookmark

Implication

Framework evaluation demonstrates high-throughput real-time processing with low latency in industrial IoT environments, indicating viable low-resource edge-to-cloud analytics.

Application Research of Big Data Real-Time Processing Framework Based on Spark in Industrial Data Stream Analysis

Discussion

Member takes

Implication

Key Points

Cite This Study

Application Research of Big Data Real-Time Processing Framework Based on Spark in Industrial Data Stream Analysis

Discussion

Member takes

Implication

Key Points

Cite This Study

Also Consider

Also Consider