What are the best methods for detecting network anomalies in large data sets?

[cvandolson]

I wanted to get some insight about which direction I should look in about Network Detection anomalies. I have been doing some research and I have found CLAD (Clustering and Anomaly Detection) which is an algorithm, however, I'm trying to find something simpler to understand to implement. Basically, I have a large set of data and I need to find outliers from the normal data. With the set of data I'm working with I'm assuming using just a basic outlier statement wouldn't work very well and was wondering if there are alternatives to something that will produce less errors in the final outcome. I just need an idea to do research and figure out if that is where I need to start.

 

[Valkarie]

One way you can approach this problem is by using machine learning algorithms for anomaly detection. This involves training a model on the normal data set, and then letting it detect outliers from the new data. For example, you could use a supervised machine learning algorithm such as Support Vector Machines (SVM) or Random Forests to classify anomalies from normal data points. Unsupervised approaches such as clustering algorithms like K-Means or DBSCAN can also be used to detect outliers. Additionally, you could use Deep Learning models such as Autoencoders or Variational Autoencoders to detect anomalies. These models are trained on normal data, and upon detecting new data, they can identify outliers. Another approach is to use statistical methods such as hypothesis testing or outlier detection. This involves looking for data points that are significantly different from the rest of the data. For example, you could use Grubbs’ test to detect outliers, or you could use an algorithm such as Isolation Forest to detect anomalies in the data.Ultimately, the approach you take depends on your data and the type of anomalies you are trying to detect. We recommend exploring both machine learning and statistical approaches to see which one works best for your application.