Abstract The article presents an approach to the development of an intelligent multimodal monitoring service for the surveillance area using large neural network models. The proposed solution is capable of analyzing heterogeneous data from video streams, environmental sensor signals (temperature, humidity, etc.), and event logs to obtain a complete picture of what is happening. The main tools used are large language and visual models (for example, LLaMA, MiniCPM‑V, etc.) deployed locally using the Ollama platform, which provides autonomous and secure information processing without the need to transfer data to the cloud. A prototype system has been developed that works offline and is capable of detecting critical situations, abnormal deviations from the norm and contextually significant events in the observed area. The method of forming test scenarios and conducting a qualitative assessment of the model’s performance using the metrics F1-score, Precision, and Recall on a set of various situations is described. The experimental results confirm the applicability of multimodal models for monitoring tasks: the prototype successfully recognizes complex patterns of behavior, demonstrating the potential of large models in building adaptive and scalable surveillance systems.
R. R. Minneakmetov (Fri,) studied this question.