A Stateful Protocol‐Based Detection Engine Combining Behavior Use Cases and System Specifications

ABSTRACT

Faced with the increasing need for network monitoring, many detection methods have been proposed. In the last few years, AI‐based methods, especially Machine Learning, have been the most popular. However, these methods are not yet fully operational and detection methods based on signatures or on specifications still keep all their legitimacy. In this letter, we propose a technique that combines a detection method based on protocol specification with a learning method train on a dataset specific to a use case. This combination leads to the definition of the notion of protocol profile. Our solution is a continuation of a previous work which proposes an anomaly detection over‐layer that are complementary to the pre‐existing ones within a NIDS. The latter keeps its usual detection technique to which is added a stateful monitoring layer based on protocol specifications represented using Harel statecharts as well as our protocol profile layer. An algorithm has been proposed to automatically generate a protocol profile. It is based on event occurrence probabilities and an intermediate data format that we introduce: the Flow Graph Execution Log (FGEL). Other algorithms are also mentioned. A prototype has been realized and an experimentation with the POP3 protocol and simulated data sets has allowed to validate the concept.