Anomaly Detection in an Open Set Environment Using Reinforcement Learning

In this research, we attempt to develop unknown anomaly detection models using a large‐scale unlabeled dataset and a limited number of partially labeled anomalies, addressing an important but mostly unsolved anomaly detection problem. This is a common situation in many important applications. Currently used related methods work with unlabeled data in an unsupervised manner or fit only a small number of anomalies, often without all the anomalies. Here, we present a new decision‐making method using the deep Q network (DQN) and the neutrosophic soft expert set (NSES) to actively search for new sets of anomalies that exist outside the boundaries of the labeled training data. The Jaccard similarity algorithm is used to calculate the distance between two states. This approach learns to find a balance between finding new anomaly classes and using its current data model. Consequently, it can increase detection accuracy by using the labeled anomaly data without limiting the types of anomalies it requires to a given anomalous case. Then, the gradient‐based marine predator (GMP) algorithm—a hybrid of the marine predator algorithm (MPA) and the gradient‐based optimizer (GBO)—is applied, and the parameters of the DQN model are adjusted at each iteration. This has major practical implications because abnormalities are inherently unobservable in nature and can be costly to ignore. Comprehensive tests on three real‐world datasets demonstrate that our strategy performs noticeably better than five state‐of‐the‐art competing approaches.