A technique and dataset of HTTP traffic for web application classification in business logic vulnerability detection

This article explores creating a specialized HyperText Transfer Protocol (HTTP) traffic dataset for multi-class web application classification by topic (domain) to identify business logic vulnerabilities. Each record contains the full text of the HTTP request and response, including headers, body, path, and parameters, which distinguishes the dataset from existing analogs limited to either a single category or a unique resource address on the Internet (Uniform Resource Locator) without any content, and is the first publicly available HTTP traffic dataset with multi-domain markup by categories relevant to the tasks of detecting business logic vulnerabilities. A technique for automated web application scanning is proposed, comprising five sequential stages: defining the composition of subject categories, selecting auxiliary sources and a collection method, preparing an initial list of web resources, multi-level web crawling with preservation of the full content of requests and responses and simulating user actions, as well as post-processing and filtering of the collected data. A comparative study of five classification models was conducted on the collected data: three models based on Term Frequency — Inverse Document Frequency (logistic regression, linear support vector machine, and naive Bayes classifier) and two neural network architectures (the lightweight and fast FastText model and a convolutional neural network adapted for the texts based on Kim’s TextCNN architecture). Web application crawling resulted in a dataset of 53,074 records in Parquet format, covering four categories: e-commerce, games, finance, and social networking and messaging. The best result of model comparison was achieved by the TextCNN model. The dataset and technique are published openly. The limitations of the proposed approach to creating a dataset of HTTP traffic for the task of multi-class classification of web applications by topic are the limitation of the coverage of the dataset to four topic domains, the minimal thematically significant content of publicly accessible pages of applications in the category of social networks and instant messengers, and the introduction of translation artifacts into the feature space of about 30 % of the records.