Chapter Three Thesis

Download as docx, pdf, or txt
Download as docx, pdf, or txt
You are on page 1of 8

CHAPTER THREE

3. RESEARCH METHODOLOGY

3.1. DATA COLLECTION AND PREPROCESSING

Considering emerging cyber-attacks, an intrusion detection system


was built by training a model with the most up-to-date realistic
dataset that plays a major role for an efficient solution. Therefore,
within the scope of this research, Satellite Terrestrial Integrated
Network (STIN) data set was used.

STIN DATA SET

The STIN dataset represent a satellite dataset that includes various


types of attacks in modern satellite and terrestrial network
environments. This dataset contains two types of traffic: TER20 and
SAT20, in csv format. These two files contain 32 features with labels
and nine different types of attacks. The distribution of the samples
for types of attacks in the training set for each category is mainly
one for terrestrial network attacks, which include 7 various types of
attacks like Botnet with 14,622 records, Web Attack with 13,017
records, Backdoor with 12,762 records, LDAP_DDos with 15,694
records, MSSQL_DDos with 15,688 records, NetBIOS_DDos with
11,530 records, and the last type is Portmap_DDos with 14,380
records. Another one is for satellite network attacks, which include
two types of attacks: Syn_DDos with 54,789 records and UDP_DDos
with 57,082 records [1].

EXPERIMENT SETUP

For this experiment, the following were included in the experiment


environment; Latest stable version of Jupyter Notebook Edition with
Python distribution platform. Tensorflow Platform for Machine
Learning and Keras as a deep learning API of Tensorflow were
employed. Scikit-learn was used for preprocessing the dataset and
evaluating the model
Jupyter Notebook Edition with Python distribution platform was
installed and used as the Integrated Development Environment
(IDE) which is pre-installed with Anaconda.
HARDWARE SYSTEM SPECIFICATIONS OF THE EXPERIMENT
ENVIRONMENT;

• Processors: Intel® CoreTM i5-6300U CPU @2.40 GHz


• RAM: 16 GB
• GPU: Intel ® HD Graphics 520 8 GB
• Operating System: Linux Mint

3.1 DATA PREPROCESSING:


The aim of this research was to build a integrated model that
predicts benign traffic by outperforming traditional intrusion
detection systems. The methodology followed during the
experiments was as shown. The Preprocessing was divided into
three main Phases such as “Data Initialization”, “Data Preparation”
and “Split and Normalize”.

Data was cleaned by addressing missing values, outliers, and


inconsistencies. Feature scaling was employed to ensure all features
are on a comparable scale for effective model training. Random
Forest was utilized for feature selection. Random Forest's strength
lies in identifying important features from the data. This reduces
complexity and improves training efficiency.

FEATURE SELECTION WITH RANDOM FOREST

Random Forest was utilized for feature selection. Random Forest's


strength lies in identifying important features from the data. We will
use the feature importance scores from the Random Forest to
identify the most relevant features for intrusion detection. This will
reduce model complexity and improve efficiency.

3.2. MODEL ARCHITECTURE AND TRAINING

RANDOM FOREST MODEL

The Random Forest model was trained using a portion of the


preprocessed data and tuned the hyperparameters using random
search to optimize performance.

Tell us

sudo code(schematic diagram)


TRANSFORMER MODEL

A transformer-based architecture suitable for the satellite network


traffic data was designed. This involved adapting existing
transformer architectures such as Transformer-XL for sequential
data. The transformer model was trained on a separate portion of
the preprocessed data, focusing on learning long-range
dependencies and contextual relationships within the traffic
sequences. The Transformer's self-attention mechanism can capture
long-range dependencies between features, useful for detecting
anomalies.
ARCHITECTURAL DESIGN

3.MODEL INTEGRATION AND EVALUATION

In this experiment, ensemble learning strategy was employed and


the models were trained separately and their predictions combined
using weighted averaging technique. Random Forest for robust
feature selection and initial classification and transformer for
capturing complex patterns and identifying anomalies in sequential
data

5. EVALUATION AND REFINEMENT:

 Evaluation Metrics: Standard NIDS evaluation metrics like


accuracy, precision, recall, F1-score, and detection rate were
employed. Additionally, metrics specific to satellite network
traffic, such as false positives that could disrupt critical
operations were considered.
 Evaluation Methodology: Rigorous evaluation using a hold-
out test set or k-fold cross-validation was conducted to ensure
generalizability and avoid overfitting. Model performance on
unseen data was monitored to assess its ability to detect
novel attacks.
 The performance of the integrated model was evaluated on a
separate test dataset and the hyperparameters of both
models were fine-tuned for optimal performance.

EVALUATION
Performance Evaluation Metrics

The performance evaluation metrics for the model was directly


dependent on the dataset characteristics and performance criteria.
First of all, the dataset distribution is as shown in terms of
percentages;

Imbalance Ratio = Majority Class Instances/ Minority Class Instances

Accuracy

The accuracy function is given as in formula. It is beneficial to


evaluate the correct prediction rate of the classifier.
Having the accuracy score closer to 1.0 is accepted as ideal for a
good classifier.

Accuracy = (TP + TN)/ (TP + TN + FP + FN)

Precision

Precision summarizes the fraction of examples assigned the positive


class that belong to the positive class.
The formula is given as;
Precision =TP/ (TP + FP)

Recall

Recall summarizes how well the positive class is predicted and is the
same calculation as sensitivity. The result varies between (0.0, 1.0).
Recall =TP/ (TP + FN)

Geometric Mean Score (G-mean)

The geometric mean is calculated by taking the square root of the


product of true positive rate (TPR) and true negative rate (TNR) also
known as sensitivity and specificity, respectively. The root degree is
equal to the number of classes. The result varies between (0.0, 1.0).
It is more acceptable to have score close to 1.0.

The formula for the G-mean score is given by [4]

𝐺−𝑚𝑒𝑎𝑛=(𝑇𝑃𝑅∗𝑇𝑁𝑅)
𝐺𝑚𝑒𝑎𝑛=𝑇𝑃𝑅∗𝑇𝑁𝑅
where TPR (True Positive Rate) is defined as

$𝑇𝑃𝑅=(𝑇𝑃/(𝑇𝑃+𝐹𝑁))
and TNR (True Negative Rate) is defined as

$𝑇𝑁𝑅=(𝑇𝑁/𝑇𝑁+𝐹𝑃))

with TP (True Positives) as the number of instances that are


correctly classified as positive, TN (True Negatives) as the number
of instances that are correctly classified as negative, FP (False
Positives) as the number of instances that are wrongly classified as
positive, and FN (False Negatives) as the number of instances that
are wrongly classified as negative.

In a multi-class classification problem, the G-mean score can be


calculated for each class and then averaged over all classes to
provide a single value that represents the overall accuracy of the
classifier. The average can be weighted or unweighted, depending
on the desired interpretation of the results.
The G-mean score provides a way to balance the accuracy of a
classifier between positive and negative classes, and is particularly
useful in cases where the class distribution is imbalanced, or when
one class is more important than the other.

Evaluation Methodology: Conduct a rigorous evaluation using a


k-fold cross-validation to ensure generalizability and avoid
overfitting. Monitor model performance on unseen data to assess its
ability to detect novel attacks.5

F-Score (F-Measure)

Precision and recall can be combined into a single score that seeks
to balance both concerns, called the F-score or the F-measure.

F − Score =2 ∗ (Precision ∗ Recall) / (Precision + Recall)

We will Evaluate the performance of the integrated model on a


separate test dataset and fine-tune the hyperparameters of both
models and the integration approach for optimal performance.

CHAPTER FOUR.

RESULTS AND DISCUSSION

We will analyze the performance of individual models (RF and


Transformer) and compare them to the integrated model, discuss
the effectiveness of different integration strategies and identify the
most advantageous approach for satellite network NIDS.

The impact of feature selection by the RF on the integrated model's


performance will be explored and will investigate the
generalizability of the integrated model to unseen attack scenarios.

We aim at;

Improving Accuracy: Random Forest helps select relevant


features, while the Transformer tackles complex network traffic
patterns, potentially leading to higher intrusion detection accuracy.

Reducing False Positives: Random Forest can help reduce false


positives by filtering out normal traffic in Stage 1.
And enhancing Anomaly Classification: The Transformer allows
for finer-grained classification of attacks.

5. CONCLUSION

This research proposes a novel integrated model for NIDS in satellite


networks that leverages the combined strengths of random forests
and transformers. The focus on data collection, preprocessing,
model architecture design, integration strategies, and rigorous
evaluation will contribute to the development of more robust and
effective NIDS solutions for safeguarding critical satellite network
infrastructure.

REFERENCES

[1] Ahmad Taher Azar, Esraa Shehab, Ahmed M. Mattar, Ibrahim A.


Hameed & Shaimaa Ahmed Elsaid: Deep Learning Based Hybrid
Intrusion Detection Systems to Protect Satellite Networks.

[2] Mohammed Hassanin, Marwa Keshk, Sara Salim, Majid lsubaie,


Dharmendra Sharma. PLLM-CS: Pre-trained Large Language Model
(LLM) for Cyber Threat Detection in Satellite Networks
DOI:10.48550/arXiv.2405.05469.

[3] Nabila Farnaaz, and M. A. Jabbar. Random Forest Modeling for


Network Intrusion Detection System.
https://doi.org/10.1016/j.procs.2016.06.047

[4] Permetrics: G-Mean Score (GMS)


https://permetrics.readthedocs.io/en/stable/pages/classification/
GMS.html.

You might also like