A Scalable Hybrid AI Security Framework for Adaptive Big Data

ABSTRACT Big Data has emerged as a demanding and fast‐growing technology in various fields, such as healthcare, finance, banking, education, and the Internet of Things (IoT). Nevertheless, such exponential growth also implies serious challenges and new threats associated with information security, real‐time data processing, and system flexibility. These are compounded by the fact that heterogeneous (non‐independent and Identically Distributed IID) data exist both in IoT and edge clients, and that there are stringent latency and energy constraints on resource‐limited devices. Despite various suggested security models, there are still too few to implement full security protection in Big Data settings. This study aims to address these drawbacks by introducing a new Adaptive Hybrid Security Framework that implements federated learning, near‐lossless AI‐based data compression, reinforcement learning (RL), and a lightweight blockchain with Edge AI for anomaly detection in a single architecture. In contrast to current methods, the framework proposed works by categorizing data dynamically (in accordance with its sensitivity) and dynamically encrypting data according to system load and other security needs, and provides decentralized key management to avoid the presence of single points of failure, which reduces computational requirements. The use of the CICIDS2017 dataset with experimental evaluation showed 94.49% federated classification accuracy, a compression ratio of 3.5–1, a reconstruction loss of 0.004, and an F1‐score of 93% for anomaly detection. The system had a latency of 180 ms during encryption and 15 ms during anomaly detection. Overall, the proposed framework provides scalable, efficient, and adaptive security for Big Data, and its lightweight architecture is compatible with real‐time usage in IoT and edge setups, which are resource‐constrained.