Lecture 17: 12th November 2019

Question 1

What are Intelligent Intrusion Detection Systems?

Answer 1

Conceptual IDSes that are accurate enough in their detection and classification of instructions to be called intelligent.

Question 2

What are some types of IDS?

Answer 2

• Host Based (HIDS)
• Network Based (NIDS)
• Hypervisor Based
• Application Based
• Protocol Based
• Server Based

Question 3

What are HIDS?

Answer 3

Host-Based IDS (HIDS): These look at logfiles to verify Message digests/checksums of key system files. They are effectively an internal system monitor. An intrusion detection system that monitors the computer infrastructure on which it is installed, analyzing traffic and logging malicious behavior.

A Network-based IDS (NIDS) looks for attack signatures in network traffic whereas s Host-based IDS (HIDS) looks for attack signatures in log files of hosts.

Question 4

What are NIDS?

Answer 4

Network-Based IDS (NIDS): these can be trained to recognise attack signatures via pattern matching (e.g. expressions, bytecode), frequency or threshold crossing (uncommon port usage). IDSes that are intelligently distributed within networks that passively inspect traffic traversing the devices on which they sit and report back to an administrator.

A Network-based IDS (NIDS) looks for attack signatures in network traffic whereas s Host-based IDS (HIDS) looks for attack signatures in log files of hosts.

Question 5

What are hypervisor-based IDSes?

Answer 5

A proposed cloud-based IDS to overcome the limitations of network load in a cloud.

Question 6

What are application-based IDSes?

Answer 6

IDSes which control data exchange per application, such as a web browser or email client.

Question 7

What are protocol-based IDSes?

Answer 7

IDSes which control one protocol, e.g. all HTTP traffic

Question 8

What are server-based IDSes?

Answer 8

IDSes shared by a subnetwork or server group

Question 9

What are some methods of intrusion detection used in IDSes? Which of them require a training phase?

Answer 9

• Misuse Based
• Anomaly Based
• Classification Based
• Combination Based or Hybrid Approaches

last 3 require a training phase

Question 10

What is Misuse Based intrusion detection?

Answer 10

The misuse-based approach uses a set of signatures representing the patterns of already known attacks to filter malicious activities. They are matched against previously defined patterns - can’t be applied to new intrusion types dynamically. Uses general-purpose GPUs.

Question 11

What is Anomaly Based intrusion detection?

Answer 11

An anomaly-based intrusion detection system is an intrusion detection system for detecting both network and computer intrusions and misuse by monitoring system activity and classifying it as either normal or anomalous. The classification is based on statistics, ML algorithms, or heuristics, rather than discrete patterns or signatures, and attempts to detect any type of misuse that falls out of normal system operation. This is as opposed to signature-based systems, which can only detect attacks for which a signature has previously been created.

Question 12

What is Classification Based intrusion detection?

Answer 12

IDSes using classification algorithms or methods such as Binary or Multi-Classification, Decision Trees, SVM, NN, Bayes, KNN (K Nearest Neighbour). These require
pre-phase of labelling but can be applied to new intrustion types.

Question 13

What is Combination Based/Hybrid Approaches intrusion detection?

Answer 13

use the best of each other type of technique but suffer from high computational costs

Question 14

Which new technologies might give rise to new types of intrusion detection?

Answer 14

cloud computing and IoT

Question 15

What is batch processing? What is stream processing? Which types of IDS use each?

Answer 15

Batch processing = load in large chunks of data and then process it.

Stream processing = process data as it comes in.

Traditional ones (all discussed) are almost always batch processing but with cloud and IoT (and large-scale applications) stream processing makes more sense - give you real-time results.

Question 16

What are ensemble methods? When are they used?

Answer 16

Ensemble methods are meta-algorithms that combine several machine learning techniques into one predictive model in order to decrease variance (bagging), bias (boosting), or improve predictions (stacking). Ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone.

They’re used to build single models from big datasets.

Question 17

What is drift?

Answer 17

Change in a model property over time

Question 18

What are the two types of drift we talk about?

Answer 18

concept and feature

Question 19

What is concept drift?

Answer 19

Concept Drift is when a data distribution varies over time (abrupt, incremental, gradual or recurring drifts) and describes the nature of network traffic.

In predictive analytics and machine learning, the concept drift means that the statistical properties of the target variable, which the model is trying to predict, change over time in unforeseen ways. This causes problems because the predictions become less accurate as time passes.

Question 20

What is feature drift?

Answer 20

Feature Drift is when features change over time as changes in data patterns dictate different levels of features (packet types). Feature drift occurs whenever a subset of features becomes, or ceases to be, relevant to the learning task; thus, learners must detect and adapt to these changes accordingly.

Question 21

What is MOA?

Answer 21

Massive Online Analysis = an open-source framework software that allows to build and run experiments of machine learning or data mining on evolving data streams.

Question 22

What is the quality of datasets like in IDS?

Answer 22

Poor. Most are outdated and sparse because companies don’t want to share their data, which has an adverse effect on the quality of IDSes, for commercial and privacy-related reasons. Datasets from before 2000 are still commonly used today in academia - companies probably have their own in-house datasets.

Question 23

What are some common problems with IDS?

Answer 23

Some pre-processing to address data cleaning and to format and remove redundant, duplicate, or incomplete values. Some fragmeneted packets may be removed. Some data may also be transformed into normalised forms.

With any predictive model building, a training phase must occur. This often uses features selection such as source and destination addresses, protocol types, length of the packet, timings, flags, etc. Another technique is to use dimension reduction which combines features into subgroups to create a lower number of manipulable dimensions.

Also class balancing: having accurate IDSes even with few anomalous packets vs vast majority normal - overweights normal ones

Lack of realistic modern datasets to develop new IDS techniques. Old and synthetic data are used.

Compensating for and mitigating drift.

Maintaining high network performance (speed) whilst still detecting intrusions and maintaining security.

Lots of time needed for training with some types of predictive models. Tradeoffs of different algorithms and techniques - none at all near perfect.

Question 24

What are classes in IDS?

Answer 24

Labelled groups applied to packets, such as normal vs anomalous

Question 25

What is the class balancing problem?

Answer 25

The idea that the vast majority of traffic being normal leads to inaccurate prediction algorithms in IDSes. The difference can make ML/DM algorithms favour normality too much to be correct more often, resulting in a high FAR/FPR.

Question 26

How can you gain or split data samples for training and sampling with predictive models?

Answer 26

• Leave One Out (LOO) method uses K-fold partitioning which is a technique to build a model on K-1 folds of the data (splits of the dataset) and evaluate against the final fold and
• Hold out (K=2, implying one part for training, one for testing)
• Prospective sampling uses a new sampled dataset separate from the training set.
• Randomization methods use sample instances without replacement.

Question 27

What is a confusion matrix?

Answer 27

A confusion matrix is a table that is often used to describe the performance of a classification model (or “classifier”) on a set of test data for which the true values are known. It is a specific table layout that allows visualization of the performance of an algorithm, typically a supervised learning one.

Question 28

How can confusion matrices evaluate the accuracy of a predictive data model for IDSes?

Answer 28

The column headers are the predicted values (e.g. true and false) and the row headers are the actual values (e.g. true false). Cells hold the frequency at which each combo (e.g. predicted false, was actually true) is seen.

This lets you find the true pos, true neg, false pos, and false neg counts and rates.

Question 29

What are some accuracy measures and their formulae to calculate them?

Answer 29

Sensitivity is the True Positive Rate, i.e. the detection rate or recall (TPR) = TP / (TP + FN)

Specificity is the True Negative Rate (TNR) = TN / (TN + FP)

Precision = TP / (TP + FP)

The F-measure is the Harmonic Mean of Precision and Sensitivity = 2TP / (2TP + FN + FN)

The Geometric Mean (G-mean) measure computes accuracy over every class separately and then computes their geometric mean.

G-mean = square root (TPR x TNR)
= square root [(TP / [TP + FN]) x (TN / [TN + FP])]

Question 30

In what forms can a ML algorithm return a prediction?

Answer 30

Either into a discrete class/group or as a probability/et of probabilities. For example, could classify a day as “going to be sunny” or “90% chance sunny, 2% cloudy, 8% rain”.

Question 31

What is the discriminating threshold?

Answer 31

The cutoff point for a probability value to fall within a certain group or classification. For example, if there is a 95% or more chance a day will be sunny, class it as predicted to be sunny.

Question 32

What is an ROC curve?

Answer 32

Receiver Operator Characteristic = a plot of the FPR (x) against TPR (y) for every discriminating threshold value used to classify instances. Since only examining positive rates, y = x is poor: want high y and low x.

Question 33

What is the AUC measure?

Answer 33

AUC = area under the curve for ROC graphs. It’s a binary measure of the performance of the model (1 = good, 0 = bad). Since only examining positive rates, y = x is poor: want high y and low x.

Question 34

What are some research issues with IDS?

Answer 34

IDS performance depends on the patterns of users and the characteristics of various services and protocols. The use of adaptive thresholds is a way to approach changing traffic profiles.

Anomaly-based systems tend to have high false positive rates (alarms) because of their current model’s inability to adapt to changes in data patterns. Batch learning systems of training followed by testing are slow to react to network traffic changes.

Proposed real-time learning systems would aggregate data from distributed sensors on hosts into a single data warehouse. However, using fixed thresholds to flag anomalies may not be as accurate as adaptable thresholds. Scaleability issues also occur here and various Data Mining, Markov chain models etc. have been tried but all perform poorly when using high volume and realistic data.

Question 35

What did Ishbel and Al Tobi’s IDS research involve?

Answer 35

Investigating how well thresholds adapt and their accuracy with different algorithms.

Question 36

What results did Ishbel and Al Tobi’s IDS research give?

Answer 36

Changing (tuning) thresholds improve detection rates. Some algorithms are more adaptable (tunable) than others - improve the most by changing (tuning) thresholds.

Threshold tuning reduces the severe effects of traffic variability on detection models.

10% (or less) of a full dataset is required to tune thresholds to get a G-mean accuracy of 90%. Random forest only needed .05%: little data needed for training (or retraining) and a lot can be left for use in testing.

Question 37

What can combat drift?

Answer 37

Use MOA to help build changing predictive models by either the strategic training of multiple weak models or to adapt a model once a drift is detected.

Question 38

How have researches sought to combat the class balancing problem?

Answer 38

• undersampling of the majority class or oversampling of the minority class
• cost function, based approaches to assign costs/ weights to minority instances.
• one-class anomaly learning methods to build a model using one class