A Multi-Module Anomaly Detection Scheme based on System Call Prediction

Authors: Zhenghua Xu, Xinghuo Yu, Yong Feng, Jiankun Hu, Zahir Tari and Fengling Han.
Abstract: Due to the rapid and continuous increase of network intrusion, the need of protecting our systems becomes more and more compelling. In many situations, there exists a weak anomaly signal detection problem: due to the little number of anomalous system calls, the anomalous patterns of some intrusions may not be enough to distinguish themselves from normal activities so the existing anomaly detection systems can not detect this kind of sequences accurately. Motivated by this, we propose a multi-module anomaly detection scheme to solve this problem through utilizing system call prediction to enlarge the patterns of weak anomaly signal sequences and make them more distinguishable. Besides this, a variation of the Viterbi algorithm (called VV algorithm) is developed to predict the most probable future system calls more efficiently and a Markov-based intrusion detection method is adopted for the pattern value calculation and anomaly detection. The results of our experimental study conclude the followings: (i) the proposed scheme can greatly improve the intrusion detection accuracy of this Markov-based intrusion detection method in terms of hit rates under small false alarm rate bounds; (ii) the performance of the proposed scheme depends on the prediction accuracy of the adopted prediction technique; (iii) the developed VV algorithm is exponentially more efficient than a baseline method.

[Download paper here]