From 5G To 6G Networks A Survey On AI-Based Jammin

This article has been accepted for publication in IEEE Open Journal of the Communications Society.
This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/OJCOMS.2024.3416808
Received XX Month, XXXX; revised XX Month, XXXX; accepted XX Month, XXXX; Date of publication XX Month, XXXX; date of
current version 17 April, 2024.
Digital Object Identifier 10.1109/OJCOMS.2024.011100
From 5G to 6G Networks, a Survey on

AI-Based Jamming and Interference
Detection and Mitigation
Poonam Lohan 1 (Member, IEEE) , Burak Kantarci1 (Senior Member, IEEE) ,
Mohamed Amine Ferrag2 (Senior Member, IEEE) , Norbert Tihanyi2 (Member, IEEE) ,
Yi Shi3 (Senior Member, IEEE)
1
University of Ottawa, Ottawa, ON, Canada
2
Technology Innovation Institute, Abu Dhabi, UAE
3
Virginia Tech, VA, USA
CORRESPONDING AUTHOR: Burak Kantarci (e-mail: [email protected]).
This work was supported in part by funding from the Natural Sciences and Engineering Research Council (NSERC) Discovery program, in
part by the NSERC CREATE TRAVERSAL Program, and in part by in part by funding from the Innovation for Defence Excellence and
Security (IDEaS) program from the Department of National Defence (DND).
ABSTRACT Fifth-generation and Beyond (5GB) networks are transformational technologies to revolu-
tionize future wireless communications in terms of massive connectivity, higher capacity, lower latency,
and ultra-high reliability. To this end, 5GB networks are designed as a coalescence of various schemes
and enabling technologies such as unmanned aerial vehicles (UAV)-assisted networks, vehicular networks,
heterogeneous cellular networks (HCNs), Internet of things (IoT), device-to-device (D2D) communication,
millimeter-wave (mm-wave), massive multiple-input multiple-output (mMIMO), non-orthogonal multiple
access (NOMA), re-configurable intelligent surface (RIS) and Terahertz (THz) communications. Due
to the scarcity of licensed bands and the co-existence of multiple technologies in unlicensed bands,
interference management is a pivotal factor in enhancing the user experience and quality of service (QoS)
in future-generation networks. However, due to the highly complex scenarios, conventional interference
mitigation techniques may not be suitable in 5GB networks. To cope with this, researchers have investigated
artificial intelligence (AI)-based interference management techniques to tackle complex environments.
Existing surveys either focus on conventional interference management methods or AI-based interference
management only for a specific scheme or technology. This survey article complements the existing
survey literature by providing a detailed review of AI-based intentional-interference management such as
jamming detection and mitigation, and AI-enabled unintentional-interference mitigation techniques from
the standpoints of UAV-assisted networks, vehicular networks, HCNs, D2D, IoT, mmWave-MIMO, NOMA,
and THz communications. While identifying and presenting the AI-based techniques for interference
management in 5G and beyond networks, this article also points out the challenges, open issues, and
future research directions to adopt AI-enabled techniques to curtail the effects of interference in 5GB and
towards 6G networks.
INDEX TERMS 5G, 6G, Artificial Intelligence, Machine Learning, Jamming Detection, Jamming Mitiga-
tion, Interference Mitigation, Physical Layer.
I. INTRODUCTION communication (mMTC), and ultra-reliable low latency com-
T HE third-generation partnership project (3GPP) intro-

duced Release 15 of fifth-generation (5G) mobile com-
munication in June 2018 with three main use cases, namely,
munication (URLLC) which laid the basis for worldwide
deployment of commercial 5G networks [1].1 In Release 15,
1 Table 1 provides the list of abbreviations used in this survey, accompa-
enhanced mobile broadband (eMBB), massive machine type
nied by their corresponding explanations.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
VOLUME , 1
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
This article has been accepted for publication in IEEE Open Journal of the Communications Society. This is the author's version which has not been fully edited and
Lohan et al.: From 5G to 6G Networks, a Survey on AI-Based Jamming and Interference Detection and Mitigation
TABLE 1. List of Abbreviations
Abbreviation Explanation Abbreviation Explanation Abbreviation Explanation

Third Generation Generative Adversarial
3GPP GAN NSA Non-Standalone
Partnership Project Network
Orthogonal Approximate
5G Fifth-Generation GMM Gaussian Mixture Model OAMP
Message Passing
Orthogonal Frequency
6G Sixth-Generation GNN Graph Neural Network OFDM
Division Multiplexing
Global Navigation Orthogonal Multiple
ABS Almost Blank Subframe GNSS OMA
Satellite System Access
Alternating Direction
ADMM GRU Gated Recurrent Unit PA Power Allocation
Method for Multiplier
AI Artificial Intelligence GUE Ground User Equipment PD-NOMA Power-Domain NOMA
AN Artificial Noise HBF Hybrid Beamforming PDR Packet Delivery Ratio
Heterogeneous Cellular Proximal Policy
AoA Angle of Arrival HCN PPO
Network Optimization
Particle Swarm
AoD Angle of Departure HetNet Heterogeneous Network PSO
Optimization
AoI Age of Information HMM Hidden Markov Model PU Primary User
AP Access Point I/Q In-phase/Quadrature QoS Quality of Service
Affinity Propagation
APC IA Interference Alignment RAN Radio Access Network
Clustering
Affinity Propagation Inter-Cell Interference
APIM ICIC RB Resource Block
Interference Management Coordination
AR Augmented Reality IF Isolation Forest RedCAP Reduced Capability
ASE Aerial Spectrum Efficiency IoD Internet of Drone RF Random Forest
Reconfigurable
BER Bit Error Rate IoT Internet of Things RIS
Intelligent Surface
BLE Bluetooth Low Energy IoTG IoT Gateway RL Reinforcement Learning
BPR Bad Packet Ratio IoV Internet of Vehicles RMA Random Multiple Access
BS Base Station ISI Inter Symbol Interference RNN Recurrent Neural Network
BW Bandwidth IVD Inter-Vehicle Distance RSMA Rate Split Multiple Access
K-means Position Received Signal
CA Carrier Aggregation KMPA RSSI
Adjustment Strength Indicator
CCI Co-Channel Interference LLM Large Language Model RSU Road Side Unit
CD-NOMA Code-Domain NOMA LoS Line-of-Sight SA Spectrum Allocation
Convolutional Neural
CNN LSTM Long Short-Term Memory SAC Soft Actor-Critic
Network
State-Action-Reward
CRE Cell range Expansion LTE Long-Term Evolution SARSA
-State-Action
CSI Channel State Information MAB Multi-Arm Bandit SCMA Sparse Code Multiple Access
CTI Cross Technology Interference MAC Medium Access Control SDR Software Defined Radio
Multi-agent Layered
D2D Device-to-Device MALQL SE Spectral Efficiency
Q-Learning
Dueling Double Deep Multi-Agent
D3QN MARL SFW Spatial Frequency Wideband
Q-Network Reinforcement Learning
Multiple Actor Shared Successive Interference
DBN Dynamic Bayesian Network MASC SIC
Critic Cancellation
Density-based spatial clustering Signal to Interference
DBSCAN MD Modulation SINR
of application with noise plus Noise Ratio
Deep Deterministic Policy
DDPG MDP Markov Decision Process SL Supervised Learning
Gradient
DDQN Double Deep Q-Network MEC Mobile Edge Computing SPS Semi-persistent Scheduling
DL Deep Learning MFG Mean Field Game SS Spectrum Sensing
DNN Deep Neural Network MFRL Mean Field RL SU Secondary User
DSA Dynamic Spectrum Access MISO Multiple Input Single Output SVM Support Vector Machine
DSC Drone Small Cell ML Machine Learning TD3 Twin Delayed DDPG
Direct Short Range Massive Multiple Input
DSRC mMIMO THz Tera Hertz
Communication Multiple Output
DT Decision Tree MLD Maximum likelihood Detector UAV Unmanned Aerial Vehicle
EE Energy Efficiency MMSE Minimum Mean Square Error UE User Equipment
Massive Machine Ultra-Reliable Low-
eMBB Enhanced Mobile Broadband mMTC URLLC
Type Communication Latency Communication
EOD Ergodic Outage Duration mmWave Millimeter Wave V2I Vehicle to Infrastructure
EPM Embedded Positioning Method MRT Maximum Ratio TX V2N Vehicle to Networks
ESN Echo State Network MUD Multi-user Detection V2P Vehicle to Pedestrians
FDD Frequency Division Duplex NB-IoT Narrow Band IoT V2V Vehicle to Vehicle
FL Federated Learning NE Nash Equilibrium V2X Vehicle-to-Everything
FN False Negative NLoS Non Line-of-Sight VNF Virtual Network Function
FP False Positive NN Neural Network VR Virtual Reality
Non-Orthogonal
FrP Fractional Programming NOMA XR Extended Reality
Multiple Access
GA Genetic Algorithm NR New Radio ZF Zero Forcing
2 VOLUME ,
Global
Evolution multiple output (MIMO), and multi-carrier operations. In 5G
Broadband
Advanced, eMBB performance would be further enhanced.
Revolution LTE and NR dynamic spectrum sharing enhancements in
eMBB+ Release 18 facilitate an efficient switch from LTE to NR.
Compute-AI
Spatio-Temporal
services 5G-Advanced Release 19 forms the basis for the sixth-
services
generation (6G) communication system. After initial inves-
eMBB tigation and studies, the standardization process of the new
6G system is scheduled to align with Release 21. As per
Ericssonâ C™s perspective and the 3GPPâC™s initiative, the
5G
release of 5G-advanced is scheduled in 2024, and the fun-
mMTC URLLC damental components of 6G specification in Release 21 are
5G-Adv
scheduled for 2028, followed by ongoing 6G evolution [4].
mMTC+ URLLC+
6G will bring together evolutionary and revolutionary wire-
Omnipresent Critical less advances. 6G will provide an evolving path for fixed and
IoT Services
6G mobile broadband characterized by widespread coverage,
reduced latency, increased capacity, and higher throughput
while supporting mission-critical services through improved
Immersive
Communication
reliability and availability. Furthermore, 6G will empower
enhanced services and new use cases such as compute-AI
FIGURE 1. Communication capabilities of 5G, 5G-advance, and 6G vision
services, spatiotemporal services, and immersive communi-
cation in the digital and virtual worlds, exploiting the full
potential of the metaverse, including boundless XR, virtual
a new 5G air interface, known as new radio (NR) is identified reality (VR), and augmented reality (AR). 6G will derive
which supports both non-standalone (NSA) and standalone the innovation and development for devices, experiences, and
operations. NSA operation in 5G communication relies on use cases that emerge around 2030 and beyond. The commu-
long-term evolution (LTE) for mobility handling and initial nication capabilities and use cases of 5G, 5G-advanced, and
access, while 5G NR is self-sufficient for standalone op- vision for 6G with evolutionary and revolutionary advances
eration. NR key features include high-frequency operation are shown in Fig. 1. The timeline, including the 3GPP 5G
(FR1: 410 MHz- 7.125 GHz; FR2: 24.25 GHz- 52.6 GHz), Releases and anticipated time plan for 6G advancements, is
spectrum flexibility, ultra-lean design to reduce interfer- represented in Fig. 2.
ence and enhance energy performance, low latency support, Various advanced techniques and technologies are being
forward compatibility, and advanced antenna technologies investigated to fulfill the ever-evolving and revolutionary
with beamforming design, among others [2]. Following this, demands of modern communication facilitated by 5GB net-
3GPP has introduced Releases 16 and 17 on evolving 5G works. Network densification through deploying heteroge-
technology to address new vertical use cases and enhance neous cellular networks (HCNs) (incorporating a mix of
network performance. Release 16 addresses enhanced sup- macro, small, micro, and pico cells), spectrum sharing,
port of the critical Internet of Things (IoT) and URLLC, beamforming, and massive MIMO are technical solutions
vehicle-to-everything (V2X) communications, relay-assisted that enhance coverage capacity, spectral efficiency, and
communications, device-to-device (D2D) communications, communication reliability. However, effective interference
positioning services, and communication in unlicensed spec- management is essential to ensure their successful implemen-
trum. Similarly, Release 17 focuses on addressing support tation and operation. Interference can result in dropped calls,
for reduced capability (RedCap) user equipment (UE), non- reduced data rates, and deteriorated overall user experience.
terrestrial networks such as Unmanned aerial vehicle (UAV)- To meet the high standards of users’ expectations in 5GB
assisted communications, multicast and broadcast services, networks, interference management is crucial for ensuring
and operations in beyond 52 GHz spectrum band [3]. secure and reliable connectivity. In the following subsection,
Besides, 3GPP recently endorsed Release 18, introducing we explore the rationale for using artificial intelligence (AI)
the features of 5G-advanced evolution [5], which focuses on and machine learning (ML)-based approaches for interfer-
enhanced network performance regarding coverage, capacity, ence management in 5GB and towards 6G networks.
and energy efficiency (EE) with new use cases such as
eXtended Reality (XR). In Release 19, scheduled for 2024,
this evolution continues and focuses on enhanced mobil- A. Research Motivation: Role of AI/ML in 5G-advanced
ity support along with all objectives presented in Release and 5G Evolution Towards 6G
18. In Release 19, it is anticipated that massive antennas The complexity of continuously evolving wireless communi-
will play a crucial role in further improvement of network cation technologies beyond 5G is increasing as they support
performance. To support eMBB use cases, 5G NR utilizes diverse deployment scenarios and applications. Integrating
different frequency ranges, duplex schemes, multiple input AI and ML techniques into the evolution of 5G is critical
VOLUME , 3
3GPP Release Timeline
2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028
REL-15 REL-16 REL-17 REL-18 REL-19 REL-20* REL-21*
* Indicative timline
5G 5G Evo 5G Evo 5G-A 5G-A
eMBB Relaying Non-terrestrial AI/ML for RAN AI/ML
Networks
mMTC Unlicenced automation Energy
RedCap efficiency 6G Specifications
URLLC D2D N/w energy
Broadcast XR
efficiency
Positioning
52.6-71GHz Mobility
XR 6G Studies
MIMO
6G Requirements
FIGURE 2. 3GPP timeline for 5G advance and 6G based on [4]
for effectively handling this complexity and propelling 5G- UAV-assisted communication V2X communication, and in-
advanced evolution towards 6G. 3GPP Release 18 provides terference from non-cooperative sources, present challenges
details of AI/ML-enabled 5G NR air interface design to that conventional interference mitigation techniques may not
reduce complexity and enhance network performance for effectively address. Hence, it is essential to design cus-
specific use cases such as enhanced channel state information tomized interference mitigation strategies that deal with the
(CSI) feedback, beam management, and positioning accu- specific requirements of 5GB networks to optimize perfor-
racy. It also specifies enhanced data collection and signaling mance and ensure URLLC communication. These problems
support for AI/ML-based load balancing, mobility optimiza- encompass continuously changing interfering conditions,
tion, and network energy-saving [6]. AI/ML-powered end-to- spectrum management, coexistence challenges, and dynamic
end (E2E) communication is a key enabler for removing the resource allocation. AI/ ML techniques, with their situa-
barriers toward 6G where E2E information and application tional awareness, complexity handling, data-driven, predic-
of dynamic distributed computing infrastructure (e.g., cloud, tive, adaptive, and automation proficiency, have been investi-
edge, devices) for new capabilities will pave the way for gated to tackle these complex issues. AI/ML algorithms can
enhanced performance of networks [7]. In line with this, dynamically allocate resources, adjust network parameters,
extensive research is actively underway, led by dedicated and predict interference patterns in real-time for efficient
researchers, to explore the application of AI/ML techniques and smarter interference management/mitigation, ensuring
to next-generation radio access networks (physical layer that wireless networks operate seamlessly in the presence
techniques, radio resource management/control, medium ac- of intentional or unintentional interference sources. There-
cess control), system architecture aspects (management, or- fore, this survey covers various “Beyond 5G” technologies
chestration, charging), and core network (network slicing, such as UAV-based networks, vehicular networks, heteroge-
service function chain (SFC) management, virtual network neous networks, D2D networks, IoT networks, mmWave-
function (VNF) placements, etc.) [8]. MIMO networks, THz Communication networks, and Non-
To review interference detection and mitigation in 5G and Orthogonal Multiple Access (NOMA)-based communication
6G networks, this survey focuses on AI/ML-enabled radio networks providing an exhaustive and detailed review of
access networks (RAN). As 5GB networks are delineated research works on AI/ML-enabled intentional/unintentional
to support various services and deployment scenarios, from interference management techniques in these domains.
autonomous vehicles and augmented reality to massive IoT Before delving into the survey content, we provide a
deployments, interference issues become increasingly com- concise introduction to various ML techniques. ML, a major
plex. Conventional interference mitigation techniques such field of AI, enables network nodes and devices to perform
as frequency reuse, spatial reuse, power control, channel- tasks without explicit programming. ML algorithms can
ization, equalization, interference cancellation, and adap- generally be categorized into three types [9]: Supervised
tive beamforming that are commonly used in traditional Learning (SL), Unsupervised Learning, and Reinforcement
wireless communication systems may not be suitable for Learning (RL) [10]. SL algorithms use labeled training
handling unique challenges posed by emerging enabling data to build models for predicting outputs from input fea-
technologies of 5GB networks. The distinctive attributes of tures. Examples include classification (e.g., KNN, Decision
5GB networks, including massive MIMO, mmWave, THz Tree, Random Forest, SVM) and regression (e.g., Linear,
frequencies, Hybrid beamforming, dynamic spectrum access, Polynomial, Exponential, Logistic). Unsupervised Learning
advanced modulation schemes, heterogeneous deployments, algorithms detect patterns in unlabeled data, including clus-
4 VOLUME ,
tering (e.g., K-means, DBSCAN, Gaussian Mixture) and interference management techniques focus on power control,
dimensionality reduction. RL enables nodes to learn control coordinated multi-point communication, and MIMO-based
strategies by interacting with the environment to maximize interference mitigation techniques in the D2D and HetNets
rewards. Examples include policy optimization (e.g., DDPG, domains. In [19], the survey work is focused on a radio ac-
TD3, PPO, SAC, A2C) and Q-learning (e.g., DQN, SARSA). cess technique, i.e., power-domain NOMA. NOMA seems to
Also, Deep Learning (DL) is a subset of ML that leverages be a very promising access technique for future technologies.
multi-layer Artificial Neural Networks. These methods can This survey paper provides detailed information on power
be employed in SL, Unsupervised Learning, and RL scenar- allocation techniques, capacity analysis, and user-pairing
ios. DL encompasses various architectural models, including methods in the power-domain NOMA, where superposition
Deep Neural Networks (DNN), Deep RL (DRL), Recurrent coding is used at the transmitter, and SIC is utilized at the
Neural Networks (RNN), Convolutional Neural Networks receiver. In [20], the multi-user MIMO (MU-MIMO) system
(CNN), Autoencoders, and Transformers [11]. is considered, and a detailed survey is provided for resource
allocation techniques in the downlink transmission (TX) of
the MU-MIMO system. Different precoding techniques and
B. Previous Interference Management Survey Works
TX technologies exploiting spatial dimensions are surveyed
As reported in Table 2, some surveys and tutorials have precisely. Also, different methodologies are reviewed to
been published on conventional interference management optimize user selection jointly, the number of users and
techniques such as [12]–[22]. In [12], cellular, ad-hoc, and antennas, spatial signaling, and power allocation to maximize
infrastructure-based platforms are considered with single the performance and mitigate the interference in downlink
and multiple antenna OFDM systems. The study reviews MU-MIMO communication. In [21], a detailed overview
successive interference cancellation (SIC) techniques and of interference management and resource management in
compares the effectiveness of various solutions in terms in-band and out-band D2D communication is provided. In
of bit error rate (BER) performances and computational [22], the authors thoroughly review interference management
complexity. In [13], a multi-cell OFDMA-based system is schemes in MIMO-based WLAN systems through the MAC
considered and the survey focuses on interference avoidance and physical layers’ mechanism design.
via inter-cell interference coordination (ICIC) such as fixed, It is to be noted that all the above studies focus on
adaptive, and real-time techniques with selective frequency conventional interference management techniques. However,
reuse, selective power reuse, and selective invert power in the current era of AI and ML, it is highly necessary to
frequency reuse. Similarly, in [14], the authors survey the develop intelligent techniques for interference and network
conventional frequency reuse, fractional frequency reuse, and management to deal with the complex environment of next-
cell coordination-based ICIC techniques in OFDMA-based generation networks. In [30] and [31], the authors provide a
cellular networks. Further, cell coordination is classified as comprehensive review of methodologies applying deep learn-
centralized, semi-centralized, coordinated, distributed, and ing to enhance the performance of wireless networks. In [31],
autonomously distributed. In [15], cognitive femtocells in the authors focus more on using artificial neural networks
HCNs are considered, where different cognitive interference (mainly spiking, deep, and recurrent neural networks) in the
mitigation schemes are surveyed, such as power control, domains such as UAV-based communication, virtual reality,
spectrum access, and antenna techniques with varying sce- IoT, and cache computing. However, these works are not
narios of deployment for femtocells such as dedicated chan- focused on interference mitigation techniques. In [24], the
nels, co-channel, open access, closed access, fixed downlink authors provide a detailed review of intentional interference
power, and adaptive downlink power. Similarly, in [16], to mitigation works utilizing ML techniques in the Global Nav-
control the inter-tier interference among low power nodes igation Satellite System (GNSS) for unmanned and manned
(small cells) and high power nodes (macro cells) and intra- aerial vehicles. There are some survey works [25], [27], [29],
tier interference in low power nodes (small cells) in HCNs, [32] that focus on ML techniques partially along with the
different interference control schemes are surveyed such conventional interference mitigation techniques with their
as spatial interference coordination at the transmitter and scope to some fixed wireless applications such as IoT [25],
interference cancellation at the receiver. D2D [25], [29], UAV [23], [29], HetNets [25], [27], [29],
In [17], to control the interference, interference alignment MIMO [29], and radar communication [32]. Further, we can
(IA) techniques are surveyed in different dimensions such as find a detailed survey on AI-based technologies for resource
time, frequency, and space, and its application is discussed allocation and interference management in the Internet of
in different domains such as HetNets, D2D, cognitive radio, Vehicles (IoV) [28] and D2D communication with mmWaves
and cellular networks. However, the main challenges in the [26]. In [33], the authors use DL techniques to focus on in-
practical adoption of IA schemes are also discussed, such terference mitigation in radar communication. In comparison
as the high overhead of CSI feedback and poor performance to existing surveys and tutorials, which mainly focus either
in low SNR scenarios. In [18], a survey of different radio on conventional interference mitigation techniques [12]–[22]
interference and resource management techniques (spectrum- or on a few wireless applications with partial or full attention
and energy-efficient) techniques is conducted. Specifically,
VOLUME , 5
TABLE 2. Comparison of our tutorial on AI/ML-enabled intentional and unintentional interference management with related existing survey and tutorial
works (Not Considered ; Partial discussion ; Considered )
AI/ML- Intentional THz

Ref. Year UAV IoV/V2X HetNets D2D IoT M-MIMO NOMA OFDM
enabled Jamming comm.
[12] 2013
[13] 2013
[14] 2013
[15] 2015
[16] 2015
[17] 2016
[18] 2016
[19] 2017
[20] 2017
[21] 2018
[22] 2019
[23] 2019
[24] 2020
[25] 2021
[26] 2021
[27] 2021
[28] 2021
[29] 2022
This
2024
survey
on AI/ML-enabled resource and interference management To the best of our knowledge, this is the first survey that
techniques [23]–[29], [32], [33], in this tutorial, to cover assembles the recent state-of-the-art research advances in
the literature gap we provide an in-depth literature review the use of AI/ML-based techniques for both intentional and
of AI/ML-enabled intentional and unintentional interference unintentional interference management in different domains
mitigation methodologies in vast domains of 5G and be- of Beyond-5G networks towards 6G (see Table 2). The main
yond networks such as UAV-based communication, Vehicular contributions of this survey are summarized as follows:
networks (V2V and V2X), Hetnets, D2D, IoT, mmWave (i) A two-fold study of interference management based on
communication, MIMO, NOMA, and THz communication. the physical-layer perspective: intentional and uninten-
As summarized in Table 2, the only survey that re- tional interference. For intentional interference man-
views AI/ML-enabled solutions for intentional jamming is agement, first, we provide the key characteristics of
presented in [24]; however, the survey is limited to UAV different types of jamming attacks and then provide a
communications. Thus, this work bridges the gap in the detailed review of a wide range of works that use AI-
literature by not only covering AI/ML-enabled interference based techniques for jamming detection and mitigation
management solutions against intentional jamming but also in various wireless communication scenarios.
reviewing the solutions by considering the Beyond-5G en- (ii) An in-depth investigation of the main features and
abling technologies listed in the table, including UAVs, types of unintentional interference affecting the perfor-
IoVs/V2X, HetNets, D2D, IoT, mm-MIMO, THz commu- mance of various schemes and enabling technologies
nications, NOMA and OFDM. of 5G and beyond network, followed by an extensive
review of AI-based interference management works
C. Contributions in different domains such as UAV-assisted networks,
The main objective of this survey work is to provide a Vehicular Networks, Heterogeneous Networks, Device-
comprehensive review of the AI/ML-enabled techniques for to-Device (D2D) communication, Internet of Things
interference mitigation in Beyond-5G networks. Here, we (IoT), millimeter-wave (mm-wave) Massive Multiple-
provide an in-depth tutorial on the recently emerging re- Input Multiple-Output (mMIMO), THz communica-
search directions and contributions in AI/ML-enabled inter- tions, and Non-Orthogonal Multiple Access (NOMA)
ference management of next-generation wireless networks. networks. We provide a summary and comprehensive
It also addresses the major advantages and challenges of taxonomy of the network systems considered, objec-
developing and deploying AI/ML-based interference mit- tives, and the ML-based approaches used to implement
igation techniques in different domains and technologies. different wireless techniques to mitigate interference
6 VOLUME ,
Survey Structure
Section I: Introduction
A. Research Motivation
Section XII: Conclusion
B. Previous Works
C. Contribution
Section II: Intentional Interference
A. Types of Jamming Attacks
B. Mathematical Modeling of
Section XI: Lesson Learned, Interference/Jamming Signals
Challenges, and Open Issues C. Jamming Detection
D. Jamming Mitigation
Intentional-
Interference Section III: UAV-based Networks
Section X: NOMA-based Networks mmWave- Management A. Features and types of interference
A. Features and types of interference MIMO, THz in UAV-based networks
in NOMA Comm. and B. Related works in UAVs and their
B. Related works in NOMA and their NOMA-based qualitative comparison
qualitative comparison UAV-based
Networks and V2X
AI-ML Networks Section IV: Vehicular Networks (V2X)
Section IX: THz Comm. Networks Enabled A. Features and types of interference
A. Features and types of interference in V2X
in THZ comm. HetNets B. Related works in V2X and their
B. Related works in THz comm. and qualitative comparison
their qualitative comparison
IoT
D2D
Networks
Networks Section V: Heterogeneous Networks
Section VIII: mmWave-MIMO Networks
(HetNets)
A. Features and types of interference
A. Features and types of interference
in MIMO
in HetNets
B. Related works in MIMO and their
B. Related works in HetNets and their
qualitative comparison
qualitative comparison
Section VII: IoT Networks Section VI: D2D Networks

A. Features and types of interference A. Features and types of interference
in IoT in D2D
B. Related works in IoT and their B. Related works in D2D and their
qualitative comparison qualitative comparison
FIGURE 3. Structure of the survey
and achieve the set objectives. Further, we provide for THz communication networks. Section X demonstrates
a qualitative comparison of the different ML-based the reviews for NOMA-based networks. Then, Section XI
approaches regarding performance and complexity. describes the lessons learned, challenges, and future direc-
(iii) Unveiling the challenges associated with developing and tion in utilizing AI/ML-enabled techniques for performance
deploying AI/ML-based interference mitigation tech- enhancement in next-generation wireless networks. Finally,
niques in different enabling technologies of 5G and Section XII provides the concluding remarks of the study.
beyond networks at the dawn of 6G. We also eluci-
date the potential future directions for using AI/ML-
based techniques to enhance the performance of future- II. AI-ENABLED INTENTIONAL INTERFERENCE
generation wireless networks. MANAGEMENT IN 5G and BEYOND NETWORKS
The rest of this survey is organized as per the high-level This section comprises three subsections. The first subsection
structure presented in Fig. 3. Section II focuses on inten- characterizes the various categories of potential jamming
tional interference, i.e., different types of jamming attacks attacks at the physical and medium access control (MAC)
from a physical layer perspective, and provides a detailed layer. The second subsection focuses on works centered on
review of AI/ML-enabled jamming detection and jamming AI-enabled intentional interference detection, and the last
mitigation techniques. Section III onwards, this tutorial fo- subsection showcases the research works employing AI-
cuses on unintentional interference, providing the features, enabled techniques for intentional interference mitigation
types of interference, and AI/ML-enabled techniques to mit- across diverse domains of 5G and beyond networks.
igate that interference in different 5G and beyond enabling
technologies. Section III begins with UAV-assisted wireless A. Different Types of Jamming Attacks at Physical and
networks and reviews recently developed AI/ML-enabled MAC layers
interference mitigation techniques in this domain. Section A jamming attack specific to physical or MAC layers
IV covers the Vehicular networks, and Section V focuses emits interference in radio signals as noise to disturb the
on Heterogeneous Networks. Then, Section VI and Section communication between transmitter and receiver. Jamming
VII discuss the intelligent interference mitigation in D2D deteriorates the SNR and makes it difficult to retrieve the
and IoT Networks, respectively. Further, Section VIII sheds information from the received signal [34]. Jamming is also
light on AI/ML-enabled interference management in the called intentional interference created against a specific tar-
mmWave-MIMO networks, and Section IX does the same get by the jammer. In contrast, unintentional interference is
VOLUME , 7
Non-Deceptive Constant
Jamming Jamming
Random Jamming
Deceptive
Jamming
Network Pulse Jamming

Active Time-based
Awareness Based
Proactive
Jamming
Spot Jamming
Jamming Attacks at Frequency

Sensing Based
Physiical/Mac Layers Range Based
Barrage
Jamming
Reactive Jamming
Frequency
Smartness based
Selction Based
Random select
Jamming
Smart Jamming
Non Smart Static Sweep

Jamming Jamming Jamming
FIGURE 4. Types of jamming attacks at physical and MAC layers
generated due to the use of the same frequency bands by transmits continuous and steady interfering signals on
nearby transmitters. Jamming attacks can be classified into the radio frequency the target system uses for com-
different categories, which are presented in Fig. 4 and are munication. This type of jammer aims to maintain
explained as follows: constant interference on a frequency and disrupt de-
(i) Sensing-based jamming attacks: In this category, jam- vices’ communication operating on that frequency [38].
ming attacks are divided into two types: reactive jam- It is worth noting that constant jamming is a type
ming and proactive jamming [35]. The reactive jam- of proactive jamming. Similarly, random jamming can
mer first senses the activity on the channel and then also be proactive because it does not listen to ongoing
activates in response to the detection of certain events communication over the channel before transmitting the
or signals, such as initiating communication over the jamming signals for random intervals. A random jam-
channel. Reactive jamming is normally used to avoid mer transmits interfering signals that randomly change
continuous TX and minimize the detection risk and the frequency and timing of TX. Random jamming
energy consumption [36], [37]. In proactive jamming, makes it difficult for the target system to predict or
the jammer transmits interfering signals irrespective of detect the jamming signal and counteract the random in-
ongoing communication. Proactive jamming disrupts terference [38]. In pulse jamming, the jammer transmits
communication more generally and is effective against intermittent interference pulses instead of continuous
communication systems with predictable patterns and TX. The jamming signal occurs in periodic bursts,
constant TX characteristics. creating confusion for the target to establish reliable
(ii) Network-awareness jamming attacks: Deceptive and communication. Pulse jamming avoids easy detection
non-deceptive are two classifications of jamming at- and is an energy-efficient jamming technique that trans-
tacks in this category. In deceptive jamming, based mits periodically, not constantly.
on network awareness, the jammer transmits signals (iv) Frequency Range-based jamming attacks: Spot and
that copy the protocol, format, and characteristics of barrage jamming come under this category. In spot
the legitimate communication signals [36]. A deceptive jamming, the jammer transmits all its power on a
jammer aims to make the target system accept the single frequency at a time to disturb the target com-
jamming signal as authentic. On the other hand, a munication. It is a very powerful jamming technique
non-deceptive jammer interferes with communication but can be easily avoided by continuously changing
signals to disrupt the communication without aiming communication frequency. In barrage jamming, a range
to deceive the target system. of frequencies are jammed at once instead of a single
(iii) Active-time-based jamming attacks: In this category, frequency to decrease the SNR of the target system [34].
jamming attacks are classified into three types: constant, However, as the jammer divides its power in the range
random, and pulse jamming [35]. A constant jammer
8 VOLUME ,
of the frequencies, the output power of the jamming is

reduced, and jamming becomes less effective. Transmission Link
(v) Frequency selection-based jamming attacks: Based on Interference Link hIR
Interfering Tx
the frequency selection, jamming can be classified into Jamming Link
three types: static, sweep, and random select jamming.
Static jammer transmits interfering signals continuously
hSR
on a single frequency or over a range of frequencies Receiver
and effectively blocks communication. However, static hJR

Source
jamming can be easily detected compared to dynamic
jamming techniques. Sweep jamming rapidly shifts the
frequency of the jamming signal over a range of fre-
quencies and can jam multiple frequencies in succession Jammer
[39]. Sweep jamming can overcome countermeasures
FIGURE 5. Simple Scenario to Model Jamming and Interfering Signals
that might be effective against static jamming. In ran-
dom select jamming, a jammer transmits interfering
signals randomly on different frequencies. It is chal- The transmit power of the source, jammer, and inter-
lenging to counteract random select jamming because fering transmitter can be defined as PS “ Et|XS |2 u,
of its unpredictable nature. PJ “ Et|XJ |2 u, and PI “ Et|XI |2 u, respectively. Follow-
(vi) Smartness-based jamming attacks: Smart and non-smart ing equation (1), the signal-to-interference plus noise ratio
jamming, in this category, refer to a level of intelligence (SINR) at the receiver can be expressed as:
and adaptability a jammer applies in creating interfer- PS |hSR |2
ence signals [40]. Smart jamming uses intelligence and SIN R “ (2)
λJ PJ |hJR |2 ` λI PI |hIR |2 ` σo2
advanced techniques to adapt and optimize the jamming
Based on the operations of the jammer J and interfering
process and minimizes the chances of detection and
transmitter T in time slot t, the SINR value at the receiver
counteraction [35]. Non-smart jamming uses simple and
can be defined as follows:
static methods of interference without any intelligence
1) Channel c is ideal because another transmitter T is not
and adaptability used for jamming.
using this channel i.e. λI “ 0, and jammer is not active
B. Mathematical Modelling of Intentional/Jamming and
in channel c i.e. λJ “ 0. In this case, both intentional and
Unintentional Interfering Signals
unintentional interference do not exist and the SNR value of
the received signal at receiver R is presented as:
To present the mathematical modeling of jamming and
interfering signals, a simple scenario is considered with a PS |hSR |2
SN R “ (3)
source and receiver pair, one jammer, and one interfering σo2
transmitter, as illustrated in Figure 5. It is worth noting that 2) Channel c is in use by another transmitter T i.e. λI “ 1,
the following explanation is easily extensible to the scenario and jammer is not active in channel c i.e. λJ “ 0. In this
with multiple jammers and multiple interfering transmitters. case, Jamming or intentional interference does not exist and
The signal transmitted by source S in timeslot t can be the SINR value at receiver R is presented as:
denoted by XS , while XJ and XI represent the jamming PS |hSR |2
signal emitted by jammer J and interfering signal emitted SIN R “ (4)
PI |hIR |2 ` σo2
by transmitter T , respectively, in timeslot t. Let hSR denote
3) Channel c is ideal i.e. λI “ 0, and jammer is active in
the channel coefficient of the channel between source S and
channel c i.e. λJ “ 1. In this case, unintentional interference
receiver R and hJR represent the channel coefficient between
does not exist but the jamming i.e intentional interference is
jammer J and receiver R, while hIR characterize the same
present and the SINR value at receiver R is presented as:
for the channel between interfering T and receiver R. It
is worth noting that the channel coefficient considers the PS |hSR |2
SIN R “ (5)
channel attenuation, antenna characteristics, and distance- PJ |hJR |2 ` σo2
dependent path loss. The received signal YR at receiver R in 4) Lastly, channel c is in use by another transmitter T i.e.
timeslot t with respect to channel c can be mathematically λI “ 1, and jammer is also active in channel c i.e. λJ “ 1.
expressed as: In this case, both unintentional and jamming i.e intentional
YR “ hSR XS ` λJ hJR XJ ` λI hIR XI ` no (1) interference exist. Hence, the SINR value of the received
signal at receiver R is presented as:
where λJ P t0, 1u and λI P t0, 1u, respectively, represent
whether channel c is jammed by jammer J and whether PS |hSR |2
SIN R “ (6)
transmitter T is emitting interfering signal using same chan- PJ |hJR |2 ` PI |hIR |2 ` σo2
nel c for its own TX, in timeslot t. n0 represents a zero-mean A TX from source to receiver is successful if the re-
complex Gaussian random noise with variance σo2 . ceiver decodes the information without error, which requires
VOLUME , 9
the SINR value to exceed a threshold. Jamming signals Decision

intentionally degrade the received signal quality. While Random Forest Tree [42-43] Federated
[42-45], learning [47]
unintentional interference can be managed through proper
SVM [41],[44-45] Le Dee
nin d
NN [44],[47]
ar se
resource allocation and power control of known transmitters,
g
ar p
Le ervi
nin
KNN [43],[45] g
p
Su
jammers are unknown and beyond the service provider’s ML CNN [46],[49]
GAN [51-52] Algorithm
t
control. Therefore, developing effective jamming detection
ng en
ns Lea
ar cem
SAC [52]
up rn
er ing
Le for
ni
and mitigation techniques, alongside managing unintentional Auto-
vi
n
se
ei
R
Encoders [53]
d
interference, is crucial for 5G and beyond networks. PDR BPR
Synthetic
SNR
Dataset
Real Metric
Dataset Used RSSI
C. AI/ML-based Jamming Detection in 5G and Beyond Jamming
Data Set Spectrogram
Networks Detection [46]
Powder RF Finger
This subsection provides state-of-the-art information related Dataset [49-50]
CRAWDAD
Printing [49]
to AI/ML-based research, specifying network systems, pa- dataset [47-48] Jamming

Classification
rameters, techniques, and ML algorithms employed for jam- IoTNetwork
[45],[49]
ming detection and classification purposes. A summary of Abnormality
Cognitive Detection
all types of terrestrial and non-terrestrial network systems, Adhoc UAV- Objectives
Network [51]
Network [46-47] Detection
objectives, metrics, data sets, and ML techniques used in Vehicular
Network
Systems
Accuracy
Sensor
jamming detection-related works is depicted in Fig. 6. Network [41] Network [52] Jamming Legitimacy
Detection Detection
The work in [41] proposes a support vector machine Wireless mmWave
Network [42-44] Network [53]
(SVM)-based mechanism to detect the jamming attack in
a vehicular network. In this work, joint sufficient statistics
FIGURE 6. Summary of jamming detection related works
of packet drop probabilities are used to generate the training
data. The simulation results present the effectiveness of the
proposed technique in terms of jamming attack detection the jamming attack. Then, different ML-based algorithms
probability. Then, a novel ML-based hybrid jamming detec- such as random forest, SVM, and KNN are used for jamming
tion algorithm is proposed in [42], where the decision tree detection. Numerical results show that Random Forest per-
(DT) algorithm is used to classify known attacks, and an iso- forms the best among all other algorithms, and the detection
lation forest (IF) is used to detect unknown attacks. Known accuracy of the Random Forest algorithm with the Real
attacks are those that are used for training purposes. To AdaBoost with five anchor nodes for real network data and
evaluate the proposed algorithm, a real dataset is generated simulated data are 89.7% and 98.6%, respectively. From the
from wireless modems transmitting in the C-band with four above works [41]–[45], it can be concluded that although
types of different jamming attacks (spot, barrage, pulse, and there are multiple available ML techniques for jamming
sweep jamming). Numerical results show that the proposed detection and classification such as SVM, Decision tree,
algorithm outperforms the baseline algorithm (hierarchical KNN, NN, and Random Forest, however, Random Forest is
combination of DT and one class-SVM) in terms of jamming the best choice for resulting the highest detection accuracy
classification and detection accuracy. in different datasets of different application scenarios.
Furthermore, three ML-based algorithms are used to detect In [46], OFDM-based UAVs’ network is considered where
and classify constant, random, and reactive jamming attacks to detect and classify different jamming attacks two different
in [43] that are random forest, k-nearest neighbors, and techniques are proposed. First, an ML-based feature classi-
decision tree. In this work, simulation data is generated using fication technique is proposed using numeric features such
the NS-3 simulator to train the algorithm, and the metrics as SINR, energy threshold, and OFDM parameters. Second,
used for jamming detection are received signal strength using the spectrogram images collected through the experi-
indicator (RSSI) and packet delivery ratio (PDR). Numerical mental testing procedure, a spectrogram-based classification
results show that the random forest algorithm is the best for model is designed using the CNN-deep learning algorithm.
detection accuracy. In the same line, the efficiencies of sev- Also, the performance of both techniques is evaluated in
eral ML-based algorithms such as SVM, random forest, and terms of accuracy and false alarm (false positive- FP) rates.
neural network (NN) are compared in detecting the jamming This work claims that the spectrogram-based CNN model
signals in [44]. In this work, a dataset is generated using outperforms the feature-based ML techniques such as KNN,
features such as PDR, bad packet ratio (BPR), and RSSI Decision Tree, Logistic Regression, Naive Bayes, and Ran-
that identify jamming signals and is used to train, test, and dom Forest in terms of jamming detection and classification
evaluate the ML-based models. The simulation results show accuracy and FP rates. Furthermore, for a flying Ad-
that Random Forest has higher jamming detection probability hoc network, a federated learning-based on-device jamming
and accuracy than SVM and NN-based techniques. attack detection scheme is proposed in [47]. Also, leveraging
The authors in [45] consider the wireless IoT network the Dempsterâ C“Shafer theory, the client group prioritization
where RSSI is collected via dedicated anchor nodes to detect technique is used to enhance the performance of the proposed
10 VOLUME ,
three-layer NN-based algorithm. The proposed algorithm is from above stated works that all different ranges of ML
evaluated using CRAWDAD [48] and the ns-3 simulated techniques such as discriminative, autoencoder, and gener-
datasets. Numerical results show that the proposed federated ative models have been investigated and proven effective for
learning scheme outperforms the traditional distributed learn- jamming detection and classification tasks.
ing solution regarding jamming attack detection accuracy.
In the work [49], an efficient fine-grained augmentation-
D. AI/ML-based Jamming Mitigation Techniques in 5G
driven DL approach is proposed to identify legitimate trans-
and Beyond Networks
mitters via RF fingerprinting to ensure the security of in-
dustrial IoT-enabled cyber-physical systems. A fine-grained Building on the previous section, this subsection delves into
augmentation approach improves the learning performance AI/ML-based techniques for jamming mitigation. Figure 7
of a DL model while tackling the varying propagation summarizes the related works, covering network systems,
channels. Further, an enhanced classifier structure is also objectives, wireless techniques, and ML algorithms used for
presented to enhance the accuracy of the proposed approach. interference mitigation.
The POWDER dataset [50] is used to experiment. It is A novel framework using a deception mechanism is
claimed and demonstrated that the RF fingerprinting identi- proposed to defeat a super-reactive jammer that makes
fication accuracy performance improves when training data the jammer continuously attack the channel [54], and the
are augmented in a waveform-specific fine-grained manner jamming signals are used to transmit data using ambient
compared to the tapped delay line and clustered delay line- backscatter communication technology. Further, a DL-based
based augmentation approach. Also, a sensitivity analysis is detector with an LSTM network is proposed to detect the
presented for the fine-grained approach concerning different backscattered signals that adapt to noise and channel dis-
SNR, SIR, and SINR levels regarding accuracy performance. tributions and learn the received signalsâC™ dependencies.
The authors in [51] consider a cognitive radio communi- Simulation results demonstrate an interesting outcome that
cation environment where to detect the abnormalities inside the transmitter can achieve better BER performance using
the radio spectrum and to make the physical layer secure, the proposed method if the jammer attacks the channel with
two AI-based techniques, (Generative Adversarial Networks a higher power. In the case of a multi-user communication
(GANs) and the dynamic Bayesian networks (DBNs)) are scenario, a Markov game framework is used to model the
proposed. Conditional-GAN is applied for high dimensional anti-jamming problem, and to solve this problem, a collab-
data, while DBN is applied on low dimensional data. orative MARL-based anti-jamming algorithm is proposed
Simulation results demonstrate the effectiveness of C-GAN [55]. The proposed solution mitigates mutual interference
and DBN models in abnormality detection accuracy over while tackling external sweep-jamming attacks. The impact
the Cyclostationary Feature Detection (CFD) and Energy of FP and false negative (FN) is considered in sensing error.
detector, respectively. Also, the work in [52] proposes a Simulation results claim that the proposed collaborative
soft actor-critic (SAC) deep-RL framework for anomaly MARL-based algorithm outperforms independent Q-learning
detection in a target process where remote sensors moni- and sensing-based methods regarding jamming mitigation
tor the process. Further, to detect the adversarial jamming and normalized rate.
attacks on the communication link between sensors and the To mitigate the jamming attacks in a flying ad-hoc net-
decision-making unit, a GAN-based framework is proposed. work, an RL-based model-free Q-learning algorithm with
Simulation results show the effectiveness of the proposed an epsilon-greedy policy is proposed [56]. Also, an on-
techniques in terms of detection accuracy and time com- device federated learning-based jamming detection technique
plexity over the conventional Actor-Critic algorithm. Thus, directs the jamming mitigation algorithm. The simulation
[51] and [52] state that using GAN-based approaches is results show that the proposed algorithm outperforms the
quite useful in detecting the abnormalities in spectrum or baseline methods, such as random policy and Dijkstra’s
adversarial jamming attacks. algorithm, to reduce the number of en-route jammer lo-
An autoencoder-based jamming detection and mitigation cation hop counts from source to destination. A novel
approach is proposed against beam training jamming attacks active inference framework is proposed for an anti-jamming
in a manufacturing environment where an automated guided resource allocation strategy in a cognitive-UAV to ground
vehicle communicates with a base station (BS) via mmWave base station communication environment [57]. Specifically,
carrier [53]. The proposed auto-encoder-based jamming de- an active generalized dynamic Bayesian network is proposed
tection algorithm learns through RSSI and can capture more to represent the external environment and jointly encode
than 80% of attack events, while the proposed auto-encoder- the spectrum’s dynamic physical signal and UAV-jammer
based jamming mitigation algorithm improves the average interaction. Further, the action and planning are considered
SINR up to 15 dB over the case without using any jamming a Bayesian inference problem that can be solved by avoid-
mitigation strategy. However, the accuracy performance of ing states under attack (abnormal state). Simulation results
the autoencoder-based jamming detection algorithm is not verify that the proposed technique outperforms the frequency
compared with any benchmark schemes. It can be observed hopping and Q-learning-based techniques regarding jamming
mitigation and convergence speed.
VOLUME , 11
Jamming Mitigation
Jamming Mitigation
FIGURE 7. Summary of jamming mitigation related works
A hot-booting Policy Hill Climbing (PHC)-based UAV power on multiple radio channels to mitigate smart jamming,
relay strategy is proposed to enhance the performance of eavesdropping, and spoofing attacks. Numerical results show
UAV-assisted vehicular networks against jamming attacks on that the proposed algorithm enhances the utility and secrecy
roadside units (RSUs) [58]. Simulation results demonstrate capacity of the UAV in comparison to Q-learning and Win
that the proposed relay strategy outperforms the Q-learning- or Fast Learn (WoFL)-PHC-based schemes.
based scheme regarding BER and UAV utility performance. To mitigate the jamming attacks and improve the BER
In work [59], a UAV-aided cellular framework is considered and energy consumption performance in a UAV swarm
to resist smart jamming where a UAV selects a DRL-based communication network, a MARL-based scheme is proposed
relay policy to serve those mobile users whose associated to control the Tx power and relay selection [61]. In this
BSs are under jamming attack. Specifically, the problem is scheme, policy distribution is formulated to enhance policy
formulated as a dynamic relay game and its NE solution is exploration, and transfer learning is used to exploit the
derived in terms of BER and UAV energy consumption. The shared swarm experiences to fasten the initial learning.
NE solution is obtained by applying deep RL and transfer Further, a deep RL-based scheme is proposed to remove state
learning utilizing the observed current communication status quantization error for dynamic channel states due to high
and the previous anti-jamming relay experiences. Simulation swarm mobility. Simulation results show that the proposed
results show that the proposed DRL-based scheme outper- scheme outperforms the MAPPO and MADDPG-based ap-
forms the Q-learning and PHC-based approaches regarding proaches regarding BER, utility, and energy consumption.
BER and UAV energy consumption performance. The au- However, the computational complexity of the proposed
thors in [60] utilize prospect theory to formulate a smart- scheme increases with the number of UAVs, batch size,
attack game on UAV TXs. Further, a deep Q-learning- quantized power levels, and the number of learning samples
based algorithm is proposed to optimally allocate UAV TX [61]. Furthermore, a distributed DRL-based energy-efficient
12 VOLUME ,
and low complexity framework is proposed to enhance the Femto-cell users in 5G HetNets in the presence of one
BER performance under jamming attacks in UAVs commu- jammer in each femtocell [68]. Specifically, the Markov
nication network [62]. Specifically, using this framework, decision process (MDP) is used to transform the non-convex
each relay UAV selects its Tx power independently based optimization problem into a MARL problem, and a dueling
on historical state-related information without knowing the DDQN (double deep Q-network) with federated learning is
location, trajectory, and battery level of other UAVs and used to solve the problem. Simulation results show that the
jammers. Then, a power control game is formulated and proposed scheme outperforms DDQN and DQN schemes
solved using a deep RL-based anti-jamming relay approach regarding the achievable rate. In work [69], an autonomous
to achieve faster and better performance. Simulation results vehicular network is considered, where to mitigate jamming
show that the proposed schemes outperform the optimal attacks and to enhance the anti-eavesdropping secrecy capac-
power allocation scheme against jamming regarding BER ity, a distributed Kalman Filtering algorithm is proposed to
and energy consumption performance. track jammers, and a hierarchical DQN-based technique is
For IoT devices to improve SINR and save their energy proposed to design anti-eavesdropping channel selection and
under jamming attacks, a DRL-based power control scheme power control policy. Simulation results show that the pro-
is proposed [63]. Knowing the current IoT Tx status and posed technique outperforms available techniques regarding
the jamming strength, DQN determines the Tx power of secrecy capacity and data rate under jamming attacks.
the IoT devices. The authors implemented the proposed Along with jamming attacks, eavesdropping is also a
scheme on USRP to evaluate the performance of the anti- serious problem from a secrecy point of view, where an illicit
jamming technique. Experimental results show that the user tries to steal the information of ongoing communica-
proposed scheme outperforms the benchmark Q-learning- tion. The following are related works utilizing AI/ML-based
based power control scheme against jamming. Especially techniques to tackle the eavesdropping problem and enhance
for the uplink TX of IoT devices, a novel spectrum anti- secrecy rate. A multi-agent cooperative DRL-based secrecy-
jamming technique is proposed in [64]. Specifically, a value aware joint channel, power, and beam-forming control al-
function approximation-based MARL technique is used to gorithm is proposed to maximize the secrecy EE of pico-
optimize the configuration policies for the IoT devices se- cell users in a 5G heterogeneous network [70]. Specifically,
quentially. Simulation results show that the proposed scheme the problem is transformed into a MARL problem utilizing
outperforms various benchmark schemes regarding average MDP, and dueling DDQN is used to find the optimal solution
normalized throughput. In [65], a model-free MARL-based of MDP. Simulation results demonstrate that the proposed
algorithm is proposed to mitigate the jamming attack in ultra- scheme outperforms MA-DQN and MARL schemes re-
dense IoT networks. In this paper, a quality of service (QoS) garding the secrecy EE of pico-cell users. Furthermore,
constrained anti-jamming problem is modeled as a Markov a deep RL-based power control algorithm is proposed to
game, and to get its NE solution, a mean-field deep Mellow- improve the secrecy rate of femtocell users in 6G HetNets
based anti-jamming technique is developed. The simulation [71]. Particularly, the non-convex optimization problem is
results show that the proposed technique simultaneously transformed into a multi-agent deep RL problem using MDP,
enables spectrum sharing and jamming attack mitigation. and a DDPG-based power control scheme is used to solve the
In the downlink mmWave-MIMO communication sce- problem. Simulation results show that the proposed scheme
nario, an RL-based power control technique is proposed to outperforms existing schemes regarding the average secrecy
control the jamming attack [66]. Specifically, a fast PHC- rate per femtocell user.
based power control algorithm is used to choose the BS Tx To secure V2X links in the cellular-based vehicle to every-
power over multiple antennas. Simulation results show that thing (V2X) network [72], a deep Q-learning-based power
the proposed scheme outperforms the benchmark technique allocation scheme is proposed to maximize the secrecy rate
regarding sum data rate and average SINR. Furthermore, of vehicles while keeping the constraints on power and
to mitigate the smart jamming attack in downlink NOMA interference. Further, a reward function is developed for the
TXs [67], an RL-based power control algorithm is proposed deep Q-learning network to allocate power to vehicles effi-
for BS power allocation equipped with multiple antennas ciently. The simulation results demonstrate the effectiveness
without knowing the radio channel and jamming parameters. of the proposed algorithm. A THz communication network
The power allocation problem is formulated as a zero-sum with randomly distributed eavesdroppers is considered in
game and its Stackelberg equilibrium is derived. Further, work [73]. In this paper, to increase the secrecy rate and
the use of Dyna architecture and hot-booting techniques to decrease the secrecy outage probability, an artificial noise
accelerate the learning speed of the proposed power allo- (AN) aided DNN-based algorithm is proposed to control the
cation algorithm. Simulation results show that the proposed Tx power, carrier frequency, and AN signal power. Simu-
scheme outperforms the standard Q-learning-based anti- lation results show that the proposed scheme outperforms
jamming techniques in terms of sum data rates of users. the conventional transmitter AN scheme regarding secrecy
A federated DRL-based beamforming and power control outage probability.
algorithm is proposed to enhance the achievable rate of
VOLUME , 13
AI/ML-based techniques are indeed helpful in enhancing III. INTERFERENCE MITIGATION IN UAV-BASED N/w
the performance of communication networks. However, in A. Features and types of interference in UAV-based
some situations, learning techniques can also be a reason communication systems
for deteriorating network performance where a learning tech- Nowadays, UAVs are widely utilized for monitoring, surveil-
nique itself can be used to create jamming attacks [74]. The lance, rescue operations, military, and public safety opera-
authors in [74], propose a DeepJam strategy for creating a tions [77]. In the above-stated scenarios, UAVs act as aerial
jamming attack on the Zigbee traffic by exploiting the cross- users which communicate with the existing cellular ground
technology interference (CTI) in the coexisting scenario BS for information sharing (uplink) and getting instructions
of Wifi-Zigbee networks. DeepJam captures the temporal (downlink) for further actions. However, due to high altitude
pattern of past traffic and predicts future traffic using a and favorable line-of-sight (LoS) channel conditions, aerial
deep-learning technique. DeepJam attacks the Zigbee Tx UAV users can cause high interference to terrestrial ground
that is not disrupted by CTI and reduces the victimâC™s users during uplink communication. On the other side, UAV
throughput significantly compared to conventional jamming aerial users can receive more interference from neighboring
strategies. Furthermore, to take advantage of learning tech- base stations during downlink communication [78]. Also,
niques, securing the training and testing data is highly UAVs act as aerial BSs to provide effective solutions to serve
important. Otherwise, the performance can be negatively temporary high-traffic demands during high-crowd events
affected if the data is distorted. Some works represent the such as concerts, festivals, and stadium games [79]. In
effect of distorting training and testing data on the system such scenarios, UAVs can face and create intercell inter-
performance. For example, in [75], to reduce the spectrum ference with other UAVs aerial BSs, and cellular ground
sensing accuracy of ML-based spectrum sensing methods BSs utilizing the same spectrum bands. In both scenarios,
in a cognitive radio-based IoT environment, an embedded the illustration of UAV-based communication systems having
poisoning method (EPM) is proposed. The proposed EPM is both desired and interfering signals is presented in Fig. 8.
a combination of embedded communication method wave- Since UAVs are power-constrained devices, in both scenar-
forms that produce interference and poison data label hidden ios, interference control via proper resource management is
attacks on the training data of ML-based spectrum sensing mandatory for better EE and throughput results. This section
techniques. Simulation results show the efficiency of the provides a detailed survey of AI/ML-related works utiliz-
proposed method in reducing the sensing accuracy compared ing intelligent techniques to avoid inter-cell or co-channel
with traditional white-box attack methods. In work [76], interference (CCI) in UAV-based communication systems.
the poisoning and evasion attacks using particle swarm This section is mainly divided into two parts: One for
optimization are exploited in training and testing datasets to UAVs as aerial BSs and the other for UAVs as aerial users.
reduce the classification accuracy of DL-based algorithms The summary of all discussed related works comprising the
used in traffic sign recognition systems for autonomous network systems, objectives, wireless techniques, and ML
vehicles. Simulation results demonstrate the effectiveness of algorithms used for interference mitigation in UAV-based
the proposed attacks in terms of accuracy reduction. Hence, networks is presented in Fig. 9.
to take full advantage of AI/ML techniques, the data used for
training and testing of the model should be highly secure.
B. Related works in UAV-based communication systems
From the next section onwards, this survey paper focuses
and their qualitative comparison
on providing a detailed review of AI/ML-related works for
1) UAVs as Aerial Base Stations (BSs):
unintentional interference mitigation in different domains
of communication networks, such as UAV-based, vehicular The research in [80] considers a drone small cells (DSCs)-
networks, HCNs, D2D, IoT, mmWave-MIMO, THz, and aided wireless communication scenario, in which to miti-
NOMA based networks. In this context, every section is gate the inter-DSC (co-channel) interference and to enhance
dedicated to a particular domain and consists of two sub- the EE and throughput, two key unsupervised learning
sections. The first subsection, discusses the key features, techniques are proposed. The first is the APIM (affinity
characteristics, different challenges, and types of interference propagation interference management) scheme, designed to
related to a specific domain. Then, in the second subsection, mitigate interference by putting the interfering DSC in sleep
a detailed review and qualitative comparison of AI/ML- mode. The second scheme is KMPA (K-means position
based research works are presented that focus on tackling adjustment), which associates the uncovered GUE (ground
these challenges and enhancing the performance of wireless users) with enhanced signal quality to the serving active
communication networks in that particular domain. DSC. Simulation results show that the proposed learning-
based multiple drone management framework enhances the
EE of DSCs by 25% and the SINR of GUEs by 56%,
respectively, when compared to the baseline scheme where
all drones transmit with maximum power. In the same
line, a multiple DSCs scenario is considered to provide
coverage to GUEs in [81]. In this paper, a prediction-based
14 VOLUME ,
Downlink Desired
Interference Signal
Intercell
Interference
Interference
Uplink
Interference
Scenario 1: UAVs as aerial users Scenario 2: UAVs as aerial base-stations
FIGURE 8. Illustration of UAV-based communication system with desired and interfering signals
proactive drone management (P2DM) framework is designed Q-learning algorithms is proposed to mitigate the CCI and
to mitigate network inter-cell interference and improve the maximize the sum capacity of the network by keeping
EE of DSCs. The proposed technique works in two phases. minimum data rate requirement constraints. Specifically, the
Firstly, a highly accurate mobility prediction model is built K-means algorithm is used to optimize the 2D horizon-
in the offline phase using Random forest as per the historical tal placement of UAVs, and the Q-learning algorithm is
data. Then, the prediction model is launched in the online used to optimize the power allocation to UAVs for vary-
phase to predict the user position, which helps proactively ing heights. The proposed approach outperforms the equal
determine whether a DSC should be awake or asleep at power allocation and particle swarm optimization (PSO)-
the next timeslot. Further, in the online phase, the K-means based power allocation schemes regarding sum capacity. To
technique is utilized to control the position and TX power mitigate the CCI in UAVs network, an effective multi-agent
of DSCs to improve their EE. Simulation results show that collaborative environment learning algorithm is proposed
the proposed P2DM technique demonstrates better results in by realizing the dynamic UAV positioning and resource
terms of execution time and EE compared to the existing allocation (in terms of power levels and sub-channels) to
method based on genetic or heuristic algorithms. The re- the GUEs [85]. In this work, the combination of DQN and
searchers in [82] consider the UAV-assisted mission-critical federated learning (FL) is considered where each UAV learns
services where UAVs provide extended network coverage and gets trained independently based on the DQN algorithm
to the affected area using spectrum-sharing cognitive radio and regularly updates network training parameters through
networks. This work proposes an optimization scheme to the sharing of non-private data to the FL cloud server to
maximize the throughput of both primary and secondary realize the collaboration of multiple UAVs. It is claimed
networks by optimizing the power allocation and number through simulation results that the proposed method is better
of UAVs deployed as relay nodes. The main constraint adaptable to a dynamic environment and outperforms the
is that the interference on the primary users should not multi-agent DQN without collaboration in terms of network
be beyond a tolerable limit. A DNN model is proposed utility (throughput maximization and power consumption
to significantly reduce the execution time under real-time minimization) optimization.
solutions of mixed-integer UAV deployment problems. It Next, the research in [86] considers a system with multiple
is claimed that low computation complexity and execution UAVs as aerial BSs to serve multiple GUEs using the
time are the main benefits of the proposed approach. The same spectrum without knowing users’ positions and channel
authors in [83] propose a hybrid unsupervised and RL-based parameters. In this paper, the authors propose a soft actor-
technique to mitigate the inter-cell interference in multiple critic (SAC), i.e., a DRL-based solution to control the CCI
UAVs-mounted small-cell scenarios. In this, the unsuper- and maximize the total throughput and fairness among users
vised affinity propagation clustering (APC) scheme explores by optimizing the trajectory and TX power of UAVs in a
the interference structure of multiple UAVs and puts the most centralized way. To achieve the balance between exploitation
serious interfering UAVs into a sleep state. Further, RL is and exploration, the reward function that combines sparse
applied to adjust the TX power of active UAVs to optimize with non-sparse reward is carefully designed. Simulation
the total throughput. It is shown that the proposed scheme results show that the proposed SAC outperforms the SAC
outperforms the pure Q-learning and genetic algorithm (GA) without power control regarding throughput and fairness.
regarding execution time and throughput. The study in [87] considers an ultra-dense UAV network
The multiple UAVs’ deployments as aerial BSs are con- to serve the GUEs. Here, a discrete mean field game (MFG)
sidered to provide emergency communication services to framework is used to formulate the power control problem
GUEs [84]. In this work, the combination of K-means and to improve EE and mitigate inter-cell interference. The MFG
VOLUME , 15
UAV-based Communication
UAV positioning control/Optimal Deployment

Random Forest [81]
UAVs as Aerial BSs [80-90] Power allocation Regression [96]

Discriminative
Network approach K-means [80-81], [84]
UAVs as Aerial Users [91-99] Channel allocation
Systems
UAVs as
Aerial BSs Tx power control APC [80], [83]
Sleep/Awake mode election
Co-Channel/ ICI mitigation
Trajectory design/ otpimization
Network Throughput
Q-learning [84], [91]; SAC-DRL [86]
Rate-splitting multiple access
Network Utility Wireless
UAVs as DQN [87], [92-95]; Dueling DQN [96]
Techniques
Aerial BSs Decoupled UL/DL BS association
Spectral Efficiency
TD3 [97], D3QN [97]; Federated DRL [85]
Trajectory Optimization Reinforcement
Energy Efficiency Learning
UAVs as MARL-PPO [88]; MARL-POMDP [89]
Resource block allocation
Aerial Users
Objectives
Power allocation MADDQN [90]; MARL [98-99]
UAV to GUE co-channel interference control Drone Clustering
Rate per UAV/GUE

ML Algorithms
Transmission latency per UAV UAVs as
Aerial Users DNN [82], [94]
Neural
Energy Efficiency Networks
Outage Minimization
FIGURE 9. Summary of related works for interference mitigation in UAV-based networks
helps imitate the interactions among UAVs in ultra-dense net- approach that combines the coalition formation game (CFG)
works. Then, MFG framework is transformed into an MDP for user association and MADRL based technique for tra-
and a DRL-based MFG (DRL-MFG) algorithm is proposed jectory optimization and energy consumption minimization.
to obtain the equilibrium solution. Simulation results show The benefit of the proposed technique is that it is trained
that the proposed methodology outperforms the random offline in a centralized manner and gets executed online
power allocation and Q-learning-based techniques for power in a decentralized manner. Simulation results demonstrate
allocation regarding EE and interference mitigation. that the proposed algorithm outperforms greedy association,
In the study [88], multiple UAVs-assisted full-duplex the best response-based association, and iterative algorithm-
cellular networks are considered. In this paper, to miti- based techniques in terms of fair throughput and energy
gate the interference and enhance the spectral efficiency, consumption in a distributed manner. The authors in [90]
the downlink-uplink decoupled association technique is consider the multiple UAVs as aerial BSs that provide
proposed utilizing the rate split multiple access (RSMA) wireless coverage to static and mobile GUEs. In this paper, to
method. Furthermore, for optimizing the sum rate of users maximize the EE of the network, a cooperative multi-agent
in the uplink and for the multicast group in the downlink decentralized DDQN (MAD-DDQN) approach is proposed
having constraints on power budget and backhaul capacity to mitigate the inter-cell interference from nearby UAV cells
of UAV, a multi-agent DRL (MADRL) technique is utilized and to jointly optimize each UAVâC™s 3D trajectory and
in a distributed manner to optimize beam-forming and user the number of connected users. The proposed technique
association in the network jointly. Compared with standard outperforms the random policy and MADDPG (multi-agent
coupled association techniques and multiple access tech- deep deterministic policy gradient) approach regarding EE
niques such as SDMA, TDMA, and NOMA, the proposed without degrading the outage performance.
method performs better regarding the sum rate of users. The
authors in [89] consider an uplink communication scenario
in which multiple UAVs cooperatively fly over the GUEs as 2) UAVs as Aerial Users:
aerial BSs to collect data from them without any centralized The authors in [91], propose a multimodal Q-learning
controller. While considering the interference management framework to locate an unknown interference source in a
through the SINR metric and fairness among users, an dynamic environment. The proposed framework combines
optimization problem is formulated to maximize the total pattern recognition with Q-learning and can dynamically
throughput and EE. In this paper, the optimization problem is adjust the algorithm’s parameters based on the changing
formulated as a decentralized partially observed MDP (DEC- environments. The simulation results confirm that the pro-
POMDP), and to solve this problem, the trajectories of UAVs posed approach can outperform various baselines in terms
and the GUEs associations are jointly optimized using an of both accuracy and distance detection. To mitigate the
16 VOLUME ,
interference among cellular-enabled UAVs and the GUEs, aware of BS locations, received signal power, surrounding
the authors propose the influential decoupled base station building topology, and interference, the proposed solution
association and the different spectrum resource allocations selects the best BS to be associated with to maximize the data
for uplink and downlink communications [92]. To avoid throughput received by the UAV during its trip. Simulation
CCI among UAVs and GUEs uplink communications, UAVs results demonstrate that the proposed technique outperforms
are served using mmWaves, and UHFs are considered for the heuristic association methods regarding data throughput.
GUEs. Also, different spectrums are allocated to UAVs’ data In study [97], a cellular connected-UAVs scenario is
links and control links. Since UAV communication is power- considered where for harmonious coexistence of UAVs and
constrained, two power allocation algorithms based on Q- GUEs, CCI is mitigated by joint optimization of time-
learning and DQN are proposed to optimize the EE of the frequency resource block (RB) allocation and beam-forming
network. The proposed scheme outperforms EE and sum design to minimize the ergodic outage duration (EOD) of
rate per UAV/GUE compared to the conventional coupled UAVs. This work considers a practical building distribution-
benchmark schemes. based path loss model and small-scale fading. To solve the
A DRL-based algorithm is proposed to maximize the radio resource management problem, a DRL-aided technique
downlink SINR of 5G cellular network-connected UAVs by is proposed, where deep double dueling Q network (D3QN)
mitigating the severe LoS link interference from the non- deals with RB coordination in the discrete action domain,
associated ground BSs (GBSs) [93]. The proposed algorithm and twin delayed deep deterministic policy gradient (TD3)
controls the TX power of GBSs in a centralized way. Instead handles beam-forming design in continuous action regime.
of CSI, the proposed algorithm requires only the UAVs’ To optimize the EOD performance, the hybrid D3QN-TD3
coordinates and their received SINR every millisecond. solution is trained via interacting with the MDP-mapped
The proposed technique provides better results regarding outer and inner environments in an online centralized man-
UAVs’ downlink SINR than Q-learning and the industry- ner. Then, the trained D3QN and TD3 agents are exploited
standard fixed power allocation algorithm. The authors in offline in a decentralized manner to achieve independent
[94] consider the joint trajectory design and GBS association EOD performance gain. Simulation results present the ef-
optimization while managing the CCI antennas in cellular- fectiveness of the proposed technique in terms of EOD per-
connected UAVs networks. This paper aims to minimize formance compared to several baseline techniques, such as
the trajectory length with connectivity outage constraints. Maximum ratio beam-forming and bufferless DRL methods.
Here, the DNN is used to provide a better GBS association, In research [98], a wireless network of multiple UAVs is
and to train a DNN, the signal strength radio map of a considered, controlled by ground control stations (GCS).
given area is constructed. Also, a novel DRL approach is This paper proposes a hierarchical MARL-based algorithm
proposed to learn optimal trajectory, in which the classical to mitigate inter-cell interference and efficiently utilize the
DRL framework is supported with a self-imitation learning battery power of UAVs. The proposed algorithm works in
model to accelerate training by utilizing successful transi- two steps to maximize UAVs’ EE. Firstly, a distributed
tion experiences. Simulation results show that the proposed inner-loop RL algorithm is used to control the TX power
approach outperforms the nearest BS association and the of UAVs. Then, a centralized outer-loop RL algorithm is
highest SINR association techniques regarding trajectory used to find the optimal frequency reuse factor to control the
length and connectivity outage performance. inter-cell interference. The proposed algorithm outperforms
The uplink communication scenario for the cellular- the random action algorithm with a fixed frequency reuse
connected UAVs is considered in [95]. In this paper, an echo factor and a hierarchical multi-agent Q-learning algorithm
state network (ESN) cells-based DRL framework is utilized with binary transmit power controls regarding EE.
online by UAVs in a distributed manner for optimizing their The Internet-of-drones (IoD) system served by cellular
path planning, power allocation, and association vectors with BSs is considered in [99]. To jointly mitigate the interference
ground BSs along the path toward their destination. The and enhance the physical layer security in the presence
objective of utilizing this framework is to minimize the of an eavesdropping drone, a resource allocation scheme
interference that UAVs cause on the GUEs and the UAVs’ consisting of clustering the legitimate drones and TX power
TX latency. Simulation results show that in comparison to the control of cellular BSs is proposed. Different clusters are
heuristic approach of following the shortest path toward the allocated with orthogonal RB to mitigate the CCI, and in
corresponding destinations, the proposed scheme performs each cluster, the NOMA scheme is used for uplink commu-
better regarding the rate per GUE and TX latency per UAV. nication to enhance spectral efficiency. The cellular BSs are
The study in [96] considers a single UAV connected to the assumed to have full-duplex communication capability and
cellular network, which flies over the urban area straight simultaneously transmit artificial noise signals to weaken the
from a starting point to the destination point. To maintain the eavesdropper links. To solve the optimization problem with
QoS and to reduce the handover frequency, a regression and the objective of sum-secrecy-rate maximization, the problem
dueling deep Q-learning-based solution is proposed for intel- is decomposed into drone clustering and power allocation
ligent BS association of the UAV. Assuming that the UAV is sub-problems. To address the drone-clustering sub-problem,
VOLUME , 17
a MARL framework is devised, and the transmit and jam- spectrum sensing (SS) and resource allocation algorithm
ming power allocation sub-problem is solved by employing is proposed to avoid CCI between the primary cellular
fractional programming, successive convex approximation, users and the secondary vehicle users and enhance spec-
and alternating optimization techniques. The better per- trum efficiency. Specifically, the hybrid learning spectrum
formance of the proposed technique SIREN (security and agents (HLSAs) perform SS based on DL to detect the
interference management with RL and NOMA) over the vacant spectrum band and then allocate those spectrum
benchmark clustering and power control schemes in terms bands to the requesting vehicle using a trained SVM clas-
of secrecy capacity is verified via simulations. sifier. Simulation results demonstrate the efficacy of the
proposed scheme regarding detection probability, sensing,
IV. INTERFERENCE MITIGATION IN VEHICULAR COMM. and allocation accuracy compared to the existing state-of-
A. Features and types of Interference in V2X the-art schemes. Similar to [102], in [103], a cognitive IoV
communication systems network is considered that opportunistically accesses primary
With the rapid evolution of autonomous driving and inter- users’ licensed spectrum without interrupting their normal
connected vehicles, the V2X communication technology is communications. In this paper, to enhance spectral efficiency
pivotal in facilitating data exchange among vehicles, RSUs, and avoid harmful interference to primary users, a joint col-
edge servers, and pedestrians. Two primary radio standards laborative big spectrum data sensing and Q-learning-based
enabling V2X communication are Dedicated Short-Range dynamic spectrum access scheme is proposed. Simulation
Communication (DSRC) and Cellular V2X (C-V2X). DSRC results indicate that the collaborative spectrum access mode
is an IEEE 802.11p technology developed from standard achieves a higher average throughput, lower interference
Wi-Fi, specifically for use within vehicular communication power, and lower communication outage than the underlay
[100]. Meanwhile, C-V2X is defined by 3GPP standards, and overlay spectrum access modes.
where LTE-V2X is presented in Releases 14 and 15, and In the research in [104], a platoon-based vehicular cyber-
5G-V2X is presented in Releases 16 and 17. Both DSRC physical system is considered, where the platoon vehicles
and C-V2X standards use the 5.9GHz band to transmit are served by a RSU. This paper proposes a soft actor-critic-
the necessary information to deliver basic safety messages based two-timescale DRL algorithm to balance the tradeoff
between vehicle-to-vehicle (V2V), vehicle-to-infrastructure between traffic capacity and communication efficiency. The
(V2I), and Vehicle-to-pedestrians (V2P). In addition to us- objective of this work is to minimize the inter-vehicle
ing the 5.9GHz band, C-V2X also uses existing cellular distance (IVD) and to maintain the QoS requirement of the
networks, forming vehicle-to-network (V2N) communication V2I link by jointly optimizing the transmit power and the
alongside V2V, V2I, and V2P [101]. As C-V2X uses direct beamwidth of the RSU and IVD of the platoon, which in
(PC5 interface/sidelink) and indirect (Uu interface/network) turn controls the inter-beam interference among different V2I
communication methods, fatal and time-critical safety mes- links. Further, a decentralized technique using the Gaussian
sages can be delivered through direct communication. In process regression (GPR) approach is proposed to allocate
contrast, less time-sensitive messages can be efficiently the power and resource blocks to vehicles so that their age
delivered through indirect communication. Therefore, this of information (AoI) does not exceed a predefined threshold
adaptability ensures that the most appropriate and efficient [105]. The proposed technique learns the network dynamics
communication method is allocated based on the specific (such as wireless channels and interference), estimates the
use cases. Thus, to provide high-quality communication future AoI of vehicles, and allocates resources in a dis-
for on-road vehicle safety, it becomes highly important to tributed manner. Simulation results show that the proposed
maintain QoS constraints such as ultra-high reliability and technique outperforms the baseline techniques with random
low latency of V2V link and V2I link communication. In power allocation and GPR-based power allocation without
cellular underlying V2V, the co-existence of V2V and V2I exploration regarding AoI violation probability.
link, and cognitive IoV networks, CCI degrades the quality A 5G V2V network is considered in [106], where to
metrics. An illustration of the V2X communication system minimize the CCI and maximize link capacity and packet
with interference effects is presented in Fig. 10. Hence, this reception ratio, a novel MADRL-based resource allocation
section provides a detailed survey of intelligent AI/ML-based algorithm is proposed. Here, each vehicle acts as an indi-
techniques that mitigate CCI via proper resource allocation vidual agent and use the actor-critic RL technique. Specifi-
and spectrum sensing. The summary of the network systems, cally, all agents share their observations over critic networks
objectives, wireless techniques, and ML algorithm utilized for centralized training using DDQN to tackle the non-
by state-of-art works in this section can be seen in Fig. 11. stationarity in a multi-agent setting. Further, each agent
uses DPG in the actor network to select the appropriate
combination of RB and Tx power in a decentralized way.
B. Related works in V2X communication systems and
This paper also addresses the hidden node problem, half
their qualitative comparison
duplex problem, and merging collision issues. Simulation re-
A cognitive radio-assisted IoV network is considered in sults demonstrate that the proposed method outperforms the
[102]. Here, a novel congestion-aware hybrid learning-based
18 VOLUME ,
V2I Link
Co-channel Cellular
Interference User link
V2P Link
V2V Link
V2I
Cellular V2I V2P

V2P user
V2V V2V
FIGURE 10. Illustration of V2X communication system with desired and interfering signals.
DDQN-based schemes. In [107], a semi-persistent schedul- vehicles in high-mobility settings. To reduce complexity,
ing (SPS) scheme is considered for resource allocation and a sequential Thompson sampling (TS) algorithm performs
avoid CCI in cellular-C2X Mode 4 V2V communication to individual beam selection. Simulations show both strategies
maximize the distance-weighted TX utility while enhancing approach the optimal rate of the genie-aided solution using
the reliability and latency-based QoS performance. Specifi- the best beam each time slot.
cally, the maximization problem is formulated as a Markov In [111], a UAV-assisted cellular V2X communication
game. A MARL approach is designed to solve it, which helps system is explored, where UAVs and cellular BS provide
vehicles select radio resources independently based on their V2I communication with multiple orthogonal spectrum re-
local observations. Further, a multi-head attention method sources, shared with V2V links. To avoid co-channel inter-
is proposed to improve the training efficiency. Simulation ference (CCI) and maximize bandwidth efficiency, a resource
results show that the proposed RL-SPS outperforms the allocation scheme for air-slicing is proposed, meeting data
baseline approach with random allocation. rate and latency constraints. An LSTM-based algorithm
V2V communication systems underlaid uplink cellular optimizes resource allocation by forecasting vehicle and
networks are considered in [108]. This paper proposes a UAV mobility. Simulations show this scheme outperforms
DNN-based transmit power control algorithm to minimize DQN and DDPG-based methods. Further, an uplink random
the interference among cellular users and vehicles and multiple access (RMA) oriented sparse code multiple access
maximize the sum rate of all users. Simulation results (SCMA)-V2X network is considered in [112], where V2I
verify that the proposed algorithm outperforms fixed transmit users and V2V users randomly select codebooks from a
power, weighted minimum mean square error (MMSE), and codebook pool. The total available bandwidth is divided
DinkelbachâC™s methods in terms of achievable sum rate. into two parts for V2I and V2V user’s TX to avoid mutual
Further, a cellular-underlaying V2V communications system interference. In this paper, to maximize the sum rate of V2I
is considered in [109]. In this paper, to mitigate the CCI links while maintaining the reliability (outage probability)
between V2V and cellular communication, a novel federated performance of V2V links, a decoupled Q-learning-based
MADRL-D3QN approach is proposed for the decentralized algorithm is proposed for codebook selection, and a convex
joint optimization of channel selection and power control optimization method is used to find the optimal bandwidth
for V2V communication. The objective of the proposed allocation (OBA) to V2I users and V2V users. Simulation
approach is to maximize the sum rate of cellular users results show that the proposed RMA-SCMA with OBA
and the packet delivery rate of V2V communication. Also, outperforms the conventional schemes, such as fixed band-
to accelerate the multi-agent learning training process and width allocation, regarding sum rate. A high-mobility V2X
address limited partial observation, federated learning is network is considered in [113]. This work aims to satisfy
adopted to synchronize the DRL models of all agents and the diverse QoS requirements of different types of links,
maintain a global model. such as low delay for V2I connections and ultra-reliability
In [110], a dynamic multi-vehicle interference environ- in terms of outage performance for V2V connections while
ment using mmWaves for BS communication is studied. A managing the CCI between V2I and V2V links. To achieve
combinatorial multi-armed bandit (CMAB) RL framework the objective, a parallel DNN framework is designed for
with adaptive combinatorial Thompson sampling (CTS) TX power allocation and vehicle local computing ratio
maximizes network throughput by selecting beams for all allocation. Further, the Hungarian method is employed to
VOLUME , 19
In research [116], a cellular-V2X is considered where

Vehicular Communication to mitigate the CCI between V2I and V2V links, and a
hybrid centralized-distributed approach is proposed for radio
resource management. Specifically, to maximize the system
Network Wireless
Systems
Objectives
Techniques
ML Algorithms capacity while guaranteeing the reliability of V2V links, a
Co-channel Spectrum/
graph-matching method is utilized for centralized channel
Cognitive IoV Supervised learning:
N/w [102-103]
interference
mitigation
Channel/RB
Allocation Regression[105]; allocation, and a distributed DDPG-based algorithm is used
SVM [102]
Platoon based Power for power allocation through actor-critic learning. Numerical
Reliability and
Vehicular allocation/
system [104]
latency of V2V
control
Neural Network: results show that the proposed method outperforms other
links DNN [108],[113],[117];
V2V
Sum rate of V2I Spectrum
LSTM [106],[111] DQN-based techniques regarding system capacity. Further, to
communication
N/w [105-106] links, Sensing
Reinforcement learning:
avoid the CCI among V2I and V2V links in V2X network, a
Spectral
Q-learning [103],[112];
Cellular efficiency
Actor-Critic RL [104], hybrid architecture consisting of centralized decision-making
underlying V2V
[106],[115-116];
links [107-109] Packet Delivery
DQN [114],[117];
and distributed resource sharing is proposed to maximize
rate,
DDPG [116];
RSU-assisted
Vehicular N/w
Packet Delivery FL-DQN [118]; the long-term sum rate of all vehicles In [117]. Further, a
Ratio FL-D3QN [109];
[110]
MARL [107],[120]; DNN-based algorithm and quantization technique is used
Lower outage MADDQN [119];
UAV-assisted
V2X comm.
Probability MAB-RL [110] by each vehicle to compress the CSI information before
N/w [111]
Age of
sending the feedback to centralized decision-making BS so
Information
Coexistence of
V2I & V2V links Minimization
that the network signaling overhead can be reduced. To
[113-120] decide on resource allocation to all vehicles, DQN is utilized
by centralized BS. Also, a distributed decision-making and
FIGURE 11. Summary of related works for interference mitigation in spectrum-sharing architecture is proposed for each V2V
vehicular communication networks link to reduce the execution time for resource allocation.
Simulation results demonstrate that the proposed schemes
are robust to feedback interval variations, input, and feedback
noise and achieve near-optimal performance.
obtain the optimal spectrum allocation to V2V and V2I links. The authors in [118] present a DRL-based decentralized
Simulation results show that the proposed scheme performs mode selection and resource allocation (channel and power)
better than other general algorithms in terms of diverse QoS approach for cellular V2X communications to maximize
achievements. the sum capacity of V2I users while ensuring the latency
In the V2X communication system, shared spectrum re- and reliability requirements of V2V pairs. A two-timescale
sources are allocated to V2I and V2V links along with their federated DRL-based algorithm is further developed to over-
Tx power control to manage the CCI and to maximize the come limited local training data at vehicles and obtain robust
data rate of V2V links and successful TX probability of V2I models. On a large time scale, clusters of nearby vehicles
links In [114]. Specifically, the DQN algorithm is used to are formed using a graph-based clustering technique. Then
solve the sub-band assignment and the DDPG algorithm is on a small timescale, vehicles in the same cluster cooperate
used for continuous power allocation. Also, a meta-based to train the robust global DRL model through federated
DRL algorithm using the off-policy model-agnostic meta- learning. Further, the global DRL model can be directly
learning (MAML) is used to adapt trained policy quickly downloaded to newly activated V2V pairs to avoid a time-
in the new dynamic environment. Numerical results verify consuming training process.
that the proposed DRL-based approach outperforms the To mitigate the interference caused by unstable V2V links
DQN-based algorithm that quantizes continuous power and to the V2I links [119], the authors propose a multi-agent
easily adapts to the new dynamic environment. An IoV DDQN-based scheme to efficiently allocate the RBs to V2V
communication network is considered in [115]. This paper links such that the sum capacity of V2I network can be
proposes a distributed model-free actor-critic RL framework maximized while satisfying the delay and reliability con-
for joint mode selection (reuse mode, dedicated mode, and straints for V2V links. Further, the proposed scheme adopts
cellular mode), RB assignment, and power control in D2D- centralized training processes and distributed execution to
V2V-based IoV networks. This work aims to maximize the ease the implementation. Also, a binarized weight algorithm
sum capacity of V2I and V2V communication and minimize is introduced to accelerate the DNN learning process and
the outage probability to ensure the URLLC requirements improve the computational complexity. A novel decentral-
of unicast V2V links. Moreover, the efficient transfer actor- ized MARL-based spectrum access algorithm is proposed
critic (ETAC) learning approach is proposed to enhance to maximize the throughput of V2I links while maintaining
the learning efficiency and convergence speed. Simulation the successful PDR above a threshold for better reliability
results prove that the proposed ETAC RL approach outper- in V2I links in a cellular V2X network [120]. Each vehicle
forms the classical actor-critic learning, the Q-learning, and agent consists of an action selector module and a message
the random search approaches regarding sum capacity and generator module, and the agents learn emergent communi-
outage probability.
20 VOLUME ,
cation to enable collaboration among themselves. For inter-

agent communication, an additional dedicated channel is
used. Simulation results show that the proposed algorithm
outperforms the baseline techniques without inter-agent com-
munication regarding throughput and PDR.
Small
V. INTERFERENCE MITIGATION IN HETEROGENEOUS Small Cell
Cell
CELLULAR NETWORKS (HCNs)
A. Features and types of Interference in HCNs
Macro
communication systems Cell
Small
HCNs represent a paradigm shift in wireless communication, Cell
offering a diverse and dynamic environment where macro-

Desired Cross-tier Co-tier
cells coexist with small cells. This architectural innovation Signal Interference Interference
is significant in the context of next-generation mobile com-
munication technology. This flexible and efficient approach FIGURE 12. Illustration of HCNs communication system with co-tier and
cross-tier interference
enhances the overall network performance by increasing net-
work capacity, improving spectral efficiency, meeting higher
data rate demands, and improving user experience [121]. portional fair scheduling and cognitive approach regarding
However, with all these advantages, the coexistence of macro delay, throughput, and packet loss ratio. The study in [126],
and small cells introduces the challenge of co-tier (within the proposes a contextual multi-arm bandit (MAB) approach
same tier) and cross-tier (among different tiers) interference based semi-distributed scheme having one global learning
management [122]. An illustration of a two-tier HCN with agent and multiple local controllers for joint interference
one macro cell and multiple small cells causing co-tier and coordination and energy saving in Hetnets. In research [127],
cross-tier interference with each other is presented in Fig. the authors proposed a big data-driven framework and MAB
12. Consequently, effective interference mitigation strategies algorithm-based online learning method to mitigate the inter-
such as spectrum allocation, power control, cell range ex- cell interference in HetNets consisting of macro and small
pansion (CRE) with biasing, and on-off switching of BSs cells sharing the same spectrum. Specifically, the sampled
are used to mitigate these challenges and ensure seamless data from the network is used by three modules i.e. action
connectivity. Spectral allocation, such as co-channel and availability selector, MAB with limited action availability,
orthogonal channel deployment to different tiers, is critical and network change detector to effectively configure the
in controlling cross-tier interference. In the co-channel de- eICIC parameters such as ABS ratio and CRE bias with-
ployment scenario, almost blank subframe (ABS) [123] and out any prior information of networks about traffic load,
fractional frequency reuse [124] are widely utilized tech- topology, and scheduling algorithm. Numerical results show
niques to manage cross-tier interference. With the orthogonal that the proposed scheme provides better results in terms
spectrum deployment, mainly co-tier interference needs to of throughput when compared to benchmark online learning
be tackled. This section provides an in-depth study of state- techniques such as stochastic gradient ascent (SGA) and
of-the-art works utilizing intelligent ML-based techniques MAB algorithms.
to provide effective strategies to mitigate co-tier and cross- In study [128], a joint problem of interference coordina-
tier interference in the HCNs domain. Fig. 13 summarizes tion and energy saving is investigated in HetNets consisting
network systems, objectives, wireless techniques, and ML of macro and pico cells. In this paper, the joint problem
algorithms utilized by the works discussed in this section. is formulated as an MDP assuming periodic patterns in
users’ traffic demand so that efficient network configuration
B. Related works in HCNs communication systems and can be selected by predicting network load. Further, to
their qualitative comparison solve MDP, an approximate dynamic programming (ADP)
The authors in [125] consider a HetNet consisting of a algorithm is proposed using the certainty equivalent control
macrocell overlaid with femtocells. In this work, an ML- (CEC) principle and ML techniques (neural network and
based interference classification and offloading scheme is polynomial regressor) while satisfying QoS constraints. It
proposed to mitigate the co-tier interference of femtocells is claimed that the proposed scheme for joint control of
and maintain the QoS for video delivery in terms of interference coordination and energy saving performs better
throughput, delay, and packet loss rate. Specifically, using than the disjoint control regarding both QoS and energy
the random forest and CNN, two different algorithms are consumption reduction. Furthermore, for the joint control of
designed to classify the users with high and low interference interference coordination and energy saving in HetNets, an
classes. Then, the high interference traffic is offloaded to the online learning framework based on a new variant of the con-
nearby small cell to maintain the QoS. Simulation results textual bandit approach is proposed while fulfilling the rate
show that the proposed scheme performs better than the pro- threshold-based QoS constraints of UEs [129]. Specifically,
VOLUME , 21
Heterogeneous Cellular Network

Communication
Network Objectives Wireless

ML Algorithms
Systems Techniques
Coexistence of Interference Supervised

Performance Resource block
macro and small mitigation-based learning: Reinforcement Learning:
based scheduling
cells sharing Polynomial Q-learning [131-132];
same spectrum regression DQN [133-135], [143];
[127-130], [138- Co-tier/Cross Throughput/ Power [128], DDQN [136], [146];
140] tier interference sum rate allocation/ SVM[ 125], DDPG [137];
mitigation maximization control Random Forest MARL [130], [139-140];
Multiple small [125] MADRL [138];
cells with same Packet loss On-off switching Actor-critic MARL
spectrum Co-channel [141],[145];
interference minimization of BSs Neural Network:
[125], [134], CNN [125], Multi-actor shared critic
[137], [144-146] mitigation [144];
Power/energy [143]
Optimal ABS Wasserstein MAB-RL [127];
consumption Contextual bandit arm
One macro cell ratio, GAN [146]
Inter-cell reduction approach [129]
and multiple CRE bias
interference
UAV-based mitigation Outage
small cells ratio/probability Cell association/
[135-136] minimization User
association
Ultra dense Spectral
network
efficiency
[131-133],
[141-143]
FIGURE 13. Summary of related works for interference mitigation in heterogeneous cellular networks communication
a two-level framework is proposed with a central controller the co-tier and cross-tier interference in ultra-dense HetNet
and multiple distributed local controllers (one per macro consisting of macro cells and small cells [133], where the
cell). The algorithm is named Bayesian Response Estimation. ϵ-greedy policy is followed for optimal convergence. It is
Threshold Search is implemented on the central controller demonstrated that cooperative learning significantly impacts
to learn each control action and its suitability for QoS and users’ capacity and the sum capacity of cooperating small
energy consumption as a function of traffic intensity. Further, cells. Also, an optimal clustering algorithm is proposed to
The local controllers make local decisions based on control reduce the computational complexity, which decreases the
actions learned by the central controller. Numerical results cooperative Q-learning computation time by four times.
demonstrate that the proposed algorithm outperforms the A 5G Hetnet consisting of one macro cell and multiple
benchmark schemes regarding energy saving, rate threshold, small cells with mobile users is considered in [134]. This
and convergence speed. paper proposes a DRL-based dynamic TDD configuration
In research [130], a power optimization problem is mod- algorithm to allocate the uplink and downlink resources to
eled using a multi-agent MDP-based distributed framework tackle inter-cell interference (ICI) and D2D link interfer-
to mitigate interference in a dense HetNet consisting of a ence. The proposed Q-value iteration-based DRL algorithm
macro cell overlaid with small cells. To solve the problem, reflects the complex states of HetNet and adaptively changes
a reward function is designed based on an optimization the TDD uplink and downlink ratio following evaluated
problem, and a Q-learning-based distributed small cells’ rewards. Numerical results demonstrate that the proposed
power allocation algorithm is proposed to maintain the QoS algorithm outperforms the conventional TDD resource al-
of macro UEs. it is claimed that the macro UE QoS can location algorithm regarding packet loss rate and network
be maintained even in a high density of small cells by throughput. Further, a HetNet with a macro cell and multiple
adequately selecting the distributed or cooperative learning UAV-based small cells is considered to offload the traffic of
approach. Then, In studies [131] and [132], the authors congested macro cells or to provide service at emergency
consider the small cells- based ultra-dense HetNet where Q- sites [135]. In this scenario, sequential and DQN-based
learning [131] and cooperative Q-learning [132] techniques algorithms are proposed to optimize UAVs’ positions and
are used to adaptively allocate the TX power to small cell the ICI coordination parameters to control ICI and maxi-
base stations such that co-tier and cross-tier interference mize spectral efficiency. Simulation results compare the two
can be controlled while maintaining the minimum capacity computationally less complex proposed algorithms with the
requirements of macro cell UEs and small cell UEs. The brute force algorithm and show that the proposed techniques
numerical results show the improvement in the minimum provide results very close to that achieved by the brute force
capacity thresholds of macro and small cell UEs. Further- method. Then, in research [136], a DDQN-based algorithm
more, both independent and cooperative Q-learning-based is proposed to solve a similar problem stated in [135] with
TX power management schemes are considered to control better spectral efficiency performance.
22 VOLUME ,
An interference alignment scheme is adopted to mitigate function. Numerical results show that the proposed scheme
the intra-cell interference and suppress the inter-cell inter- outperforms the benchmark schemes, such as random power
ference in HetNets consisting of dense small cells [137]. control, soft frequency reuse, and gradient-based distributed
A centralized power control scheme is adopted in macro power control, regarding TX rate and power consumption
BSs. Specifically, the power control problem to maximize reduction. Furthermore, a joint channel allocation and BS
the sum spectral efficiency is formulated as an MDP and is ON-OFF switching scheme is proposed to manage the CCI
solved by the DDPG approach. Numerical results show that in densely deployed HetNets [142]. Specifically, for channel
the proposed scheme outperforms the existing benchmark allocation, two algorithms are proposed. The first algorithm
algorithms (random, Q-learning, DQN, etc.) regarding con- is implemented at the BSs level, where allocating the channel
vergence, complexity, and spectral efficiency performance. of a non-cooperative game dynamically is solved using a no-
The study in [138] considers a beam-forming framework, regret learning approach. The second algorithm is executed
for HetNet with multiple APs such as Macro, micro, and at the UE level, where each UE provides feedback about
femtocells, that utilizes the multi-agent DRL to mitigate the the minimum CCI channel to its associated BSs. Then, BSs
inter-AP interference and to maximize the system downlink use the UE feedback to select their channels. Further, both
sum rate. Specifically, each AP is treated as an agent, which channel allocation algorithms use an ON-OFF switching
exploits an online policy DNN to generate an APâC™s scheme to minimize energy consumption. Simulation results
beamforming vector and an online Q-function DNN to demonstrate that the second channel allocation algorithm
evaluate the appropriateness of this beamforming vector. combined with the ON-OFF switching scheme performs
Further, a centralized reward scheme is proposed to train Q- better in load balancing, users’ rate, and energy consumption
function DNNs and the policy DNNs of all the APs, which compared to an interference-aware dynamic channel alloca-
gets updated in a distributed manner. Numerical results tion benchmark scheme.
show that the proposed method outperforms the benchmark In [143], the authors propose an RL-based TX power opti-
beamforming schemes in terms of convergence speed and mization scheme for ultra-dense small-cell systems to control
the downlink sum-rate performance of the system. downlink inter-cell interference without knowing neighbor-
MARL-based decentralized cell association and interfering cells’ channel states and interference distribution. Ad-
ence management strategies are proposed by optimizing the ditionally, a DRL-based framework reduces computational
downlink TX power and CRE bias parameter in a Hetnet complexity and enhances communication efficiency. Numer-
consisting of macro and pico cells [139]. Simulation results ical results show these schemes outperform the benchmark
show that the proposed scheme outperforms the static eICIC big-data self-organizing network (Bi-SON) in throughput,
solutions, such as fixed CRE bias, fixed ABS, and resource energy consumption, and interference mitigation.
partitioning, regarding per-user average throughput. Further, In [144], a DRL-based multi-agent power control algo-
a joint power control and cell association-based ICIC ap- rithm with centralized training and distributed execution is
proach is proposed in HetNet consisting of macro cells and designed to mitigate interference in Hetnets with multiple
pico cells [140]. Here, in the time domain, pico cells learn APs sharing the same spectrum. The multiple-actor-shared-
power allocation and CRE bias to fulfill the QoS require- critic (MASC) method trains local DNNs of each AP agent
ments of its UEs and macro cell self-organizes its TX power separately, controlling TX power based on local observa-
to serve macro UEs while satisfying pico cells’ interference tions. The proposed scheme outperforms conventional power
constraints. In the frequency domain, single-flow CA (carrier control algorithms, such as fractional programming and
aggregation) and multi-flow CA approaches are investigated weighted MMSE, in computational complexity and sum rate.
in which users can be served simultaneously by one BS and In research [145], the power control problem is solved
two (or more) BSs. The proposed MARL-based solution is by a single agent and MARL utilizing a soft actor-critic
compared with several benchmark schemes, such as static (SAC) framework to mitigate the multi-cell interference in
CRE and resource partitioning. Numerical results state that the uplink communication scenario and enhance the EE.
the proposed scheme outperforms the benchmark schemes Numerical results show that the proposed scheme approaches
regarding average UE throughput in the time domain and the global optimum provided by the branch and bound
cell-edge UE throughput in the frequency domain. algorithm with less computation complexity. The authors
In research [141], the authors investigate the inter-cell in [146] consider a HetNet with multiple small cells where
interference mitigation problem for autonomous ultra-dense an intelligent resource (sub-channels) scheduling scheme is
networks by proposing a fully distributed self-learning in- proposed for ICIC. This paper’s ICIC problem is formulated
terference mitigation (SLIM) scheme using a model-free as a distributed inverse RL problem. Further, to solve this
MARL framework. Precisely, the actor-critic algorithm con- problem from both Q-value and policy perspectives, a self-
trols power to mitigate inter-cell interference while guaran- imitating learning (SIL) algorithm incorporating Wasserstein
teeing QoS to users. Further, the Mean Field Theory avoids GAN and DDQN algorithms is proposed. Numerical re-
agents’ interaction and addresses the dimensional complexity sults demonstrate that the proposed scheme outperforms the
of joint action in MARL by approximating the action value benchmark schemes regarding overall network throughput.
VOLUME , 23
Uplink
Downlink Uplink
Downlink
Cell UE
Cell UE
Cell UE
Cell UE
Cellular
Cellular
Cell
Tx Cell
Tx D2D Pair Rx
Rx Rx
D2D Pair Rx D2D Pair
Tx D2D Pair
Overlay Tx
Underlay Deployment
Deployment
Desired Cellular-D2D Intra D2D

Signal interference Interference
FIGURE 14. Illustration of interfering signals in D2D communication networks for both underlay and overlay deployment
VI. INTERFERENCE MITIGATION IN in D2D wireless techniques, and ML algorithms utilized by works
COMMUNICATION NETWORKS discussed in this section can be visualized in Fig. 15.
A. Features and types of Interference in D2D
communication networks B. Related works in D2D communication networks and
Due to the significant increase in users and the growing their qualitative comparison
demand for seamless connectivity everywhere, integrating 1) Overlay D2D Communication Networks:
innovative technologies and concepts has become imperative. In research [150], the RF or VLC band selection problem
Consequently, incorporating D2D communication technol- is considered for D2D communication to maximize EE
ogy is a pivotal consideration for 5G and beyond networks and minimize the outage probability. In this work, utilizing
[147]. D2D communication has emerged as a transformative the limited channel knowledge in terms of received power
paradigm in wireless communication, allowing devices close and sum interference at D2D receivers, a DNN-based band
to each other to communicate directly without central BS. selection solution is proposed. To further improve the band
D2D communication enhances the network capacity and selection decision, a heuristic algorithm, which utilizes the
performance by achieving high data rates, spectral effi- output of the DNN algorithm, is proposed. Numerical results
ciency, EE, and expanding network coverage with minimal show that the proposed scheme outperforms the heuristic
latency [148]. However, integrating D2D technology into algorithm without DNN regarding EE, outage ratio, and com-
conventional networks presents several challenges, includ- putation complexity. In [151], the authors investigate a link-
ing power control, spectrum allocation, and interference scheduling scheme to jointly optimize the throughput and
management. Two deployment scenarios exist for the In- AoI for D2D communication networks having a last come,
band D2D communication: underlay and overlay deploy- first serve-packet replacement (LCFS-PR) policy for packet
ments with the existing cellular networks [149]. In underlay TX. Specifically, a DNN structure maps the geographic lo-
deployment, D2D pair communication relies on the cellular cation information to optimal scheduling parameters without
BS for coordination, control, and resource allocation. In this CSI feedback. Numerical results show the trade-off between
scenario, D2D pairs communication coexists with cellular AoI and throughput, i.e., one metric can be improved at the
communication, sharing the spectrum with existing cellular cost of another metric. Also, as the number of scheduling
users. Therefore, devices engaged in D2D communication links increases, DNN-based algorithms can approach near-
can create CCI with cellular users and other D2D users. optimal performance while offering lower computational
In overlay deployment, D2D communication is independent complexity than iterative methods.
of existing cellular infrastructure, and a separate/dedicated The authors in [152] consider frequency division duplex
set of frequency bands is explicitly allocated for D2D (FDD)-based overlay D2D communication network, mainly
communication. Hence, CCI management is required among focusing on feedback overhead reduction and resource man-
different D2D pairs in overlay deployment. An illustration agement for CCI. Specifically, the partial or binary feedback
of D2D communication networks consisting of desired and system is used to reduce the feedback overhead, and a DRL-
interfering signals is presented in Fig. 14 for underlay and based TX policy is used to maintain the average sum rate
overlay scenarios. This section provides a detailed survey of per the level with full feedback. Numerical results show that
state-of-the-art works for interference management in both compared to the full feedback method, the proposed partial
D2D deployment scenarios through proper TX power control and binary feedback methods reduce the feedback overhead
and spectrum allocation schemes utilizing different ML 49% and 92%, respectively. In work [153], two algorithms
techniques. The summary of network systems, objectives, are proposed for solving the joint channel allocation and
power control problem to mitigate the CCI among D2D
24 VOLUME ,
Device-to-Device Communication
Link Scheduling/Channel Selection DNN [150-151], [158-159]

Overlay D2D deployment [150-156]
Resource management Grpah NN [157]
Network Overlay D2D
Underlay D2D deployment [157-169] Systems Power Control Neural Network CNN [157], RNN [155],
deployment
CSI based Tx Scheme
Spiking NN [169]
Co-Channel interference mitigation
Wireless
Sun rate/ Throughput Maximization Techniques
DQN [152-153], [160], [166]
Energy Efficiency Overlay D2D Spectrum Sensing
deployment Duelling DQN [154]
Outage Minimization
Underlay Channel/ Spectrum Allocation
D2D DDQN [161], [163]; DDPG [166]
Age of Information Minimization Reinforcement
deployment Link Scheduling
Learning
Objectives MADRL [165-168]
Power Control
FL-aided DRL [164]
D2D and cellular co-channel interference control
Throughput/sum rate of D2D and Cellular links

ML Algorithms
QoS of Cellular links Underlay
D2D
Energy Efficiency deployment
Non-linear Regression [156]
Supervised
Outage Minimization learning
D2D users fairness
FIGURE 15. Summary of related works for interference mitigation in D2D communication domain
communications and to maximize the weighted sum rate. 2) Underlay D2D Communication Networks:
The first algorithm is based on fractional programming (FrP), The study in [156] considers the coexistence of D2D and
a centralized approach that requires global CSI, which is UAV communication networks is considered where UAV
unsuitable for scalability and creates a high signal overhead. senses the spectrum used by D2D communication before
The second algorithm is based on DRL, which uses the D2D utilizing it to avoid CCI. This work develops an ML-assisted
pairs to optimally select the channel and control the TX stochastic geometry framework to characterize the aerial
power in a distributed manner by utilizing local information. spectrum efficiency (ASE) and coverage probability of UAV
Numerical results indicate that the DRL-based algorithm and D2D communication by analyzing the spatial false alarm
performs near to the FrP-based algorithm without global CSI probability and the spatial missed detection probability of
and with significantly less signaling overhead. a typical UAV spatial spectrum sensing. Specifically, the
Further, for CCI mitigation and average sum rate max- Gaussian non-linear regression method determines the PDF
imization in a D2D communication network, a distributed (probability density function) parameters for the received
TX scheme is proposed utilizing a dueling DQN algorithm signal strength of UAV networks. This helps in further
that learns each state’s value without learning the effect analysis of ASE and coverage probability. The proposed
of each action [154]. Numerical results indicate that the method helps find the optimal radius for spatial spectrum
proposed scheme outperforms no-control methods in average sensing and UAV TX power to maximize the ASE of UAV
sum rate and matches the performance of centralized oppor- networks while maintaining the minimum ASE requirement
tunistic and sub-optimal schemes at high SNR. However, for D2D communication. The authors in [157] consider D2D
the feedback and signaling overhead is significantly low in communication-based IoT systems where to solve the re-
the proposed distributed scheme compared to the centralized source allocation problem a Graph Neural Network (GNN)-
schemes. The research in [155] proposes an interference based supervised framework is proposed. Specifically, a lay-
feature extraction-aided RNN scheme to control the TX ered structure is investigated where CNN is used for feature
power of transmitters in a distributed manner with full aggregation, and DNN is used for feature combination to
frequency reuse for D2D pairs. Linear filters are designed update the feature vector at each node iteratively. Further,
to extract different local interference patterns from outdated two algorithms are proposed for resource management. First
channel information, and they enable the network to learn for link scheduling in D2D communication and second
and predict the real-time interference pattern to make power for joint power and channel allocation for D2D under-laid
decisions. Further, an input reduction process is designed to cellular network scenario. Numerical results demonstrate that
reduce the computational time and system overhead. Sim- the proposed methods outperform the unsupervised GNN and
ulation results show that the proposed scheme outperforms conventional DNN approaches regarding average sum rate,
the benchmark schemes such as maximum power Tx and training sample efficiency, and computational complexity.
RNN-RL regarding the average sum rate. In research [158], an underlay D2D communication net-
work is considered. In this work, the authors utilize a
VOLUME , 25
heuristic equally reduced power scheme together with DNN resource allocation scheme is proposed in [164] to maximize
to control the TX power of D2D users such that the sum the throughput and minimize the power consumption while
rate of D2D users can be maximized while fulfilling the guaranteeing the QoS requirement of both D2D and cellular
rate constraints of D2D users and interference constraints users. The proposed scheme is evaluated through simulation
to cellular users. Simulation results show that the proposed for 5G mm-wave and 6G THz scenarios. It is shown that
scheme achieves a near-optimal sum rate with low com- the proposed scheme outperforms pure DRL and random
putation complexity. However, the channel allocation was methods in terms of throughput and power consumption.
not discussed in this work. Further, a DNN-aided resource The authors in [165] propose an actor critic-based MARL
allocation (power allocation and channel allocation) scheme distributed algorithm for spectrum allocation to mitigate the
is proposed in [159]. The proposed scheme can work in a interference to cellular and other D2D links and to enhance
centralized manner with full CSI and in a distributed manner the sum rate of D2D links. The training is performed in a
with partial CSI such that the spectral efficiency of the D2D centralized manner by sharing global policies, actions, and
users is maximized while maintaining a minimum data rate historical states. Further, neighbor users’ historical informa-
for cellular users. Further, a novel hybrid training strategy tion is used to reduce training complexity. Simulation results
combining supervised and unsupervised training is proposed. demonstrate that the proposed algorithm outperforms the Q-
It is reported that the proposed scheme can achieve near- learning and DQN-based algorithms regarding sum rate, the
optimal performance with low computation complexity. outage probability of cellular links, and convergence time.
A centralized DRL algorithm is proposed for power al- A DRL-based scheme for joint RB allocation and power
location to D2D transmitters in an under-laid D2D com- control of D2D pairs is introduced to maximize the sum
munication network having a time-varying environment to rate while ensuring fairness among all users, as described
maximize the system capacity and user quality of experience in [166]. Specifically, the MDP converts the optimization
[160]. A new reward function is adopted to improve the user problem into the MADRL problem. To solve this problem,
experience quality. Numerical results show that the proposed multi-DQN is used for RB allocation and distributed DDPG
method outperforms the Q-learning and random assignment is used to convert stochastic policy to deterministic pol-
schemes regarding system capacity and user experience icy. Furthermore, a conventional optimization scheme with
quality. In [161], an under-laid D2D communication network DDPG is used for power control of D2D pairs and cellular
is considered where to avoid CCI and to maximize the sum UEs. Simulation results show that the proposed scheme
throughput while maintaining the communication quality of outperforms distributed DRL schemes regarding sum rate
cell edge users; a centralized DDQN-based spectrum access and fairness among all users. Further, in [167], a Stackel-
technique is proposed for D2D pairs to access the spectrum berg game (SG)-guided MADRL scheme is proposed for
shared with cellular UEs. Further, to maintain resource power control and channel allocation to D2D users in a
allocation fairness among D2D pairs, a fairness factor is distributed manner. Specifically, based on SG equilibrium, a
included in the objective function. Numerical results indicate Stackelberg Q-value is defined to guide the learning direction
that the proposed algorithm achieves optimal performance and reduce the convergence time. Simulation results validate
and outperforms the scheme, which requires BS cooperation. that the proposed SG-guided MADRL outperforms normal
However, the proposed schemes are centralized in nature MADRL regarding channel capacity, average utility, and
and the investigation for joint power control and spectrum training time. In research [168], a MADRL-based power
allocation is missing in [160] and [161]. control and spectrum allocation scheme is investigated to
The research in [162] proposes a distributed DL-based mitigate the CCI and optimize the throughput of cellular
power allocation algorithm to maximize the spectral effi- and D2D links while satisfying the delay constraints of D2D
ciency of the under-laid D2D communication network while links. Specifically, priority sampling (PS) is used to learn
controlling the CCI to cellular communication. Specifically, the dominant features to overcome the impact of large state
the mapping between transmit power and location informa- space. The PS-based D3QN (PS-D3QN) algorithm is used in
tion is memorized by D2D devices to achieve the near- a distributed manner for resource management. Simulation
optimal result. Numerical results show that the proposed results show the proposed algorithm outperforms the DRL
scheme achieves similar spectral efficiency to the Iterative baselines regarding sum throughput. Then, in [169], the
approximated distributed rate maximization approach with- authors consider the spiking neural network cooperated with
out involving other nodes. In this line, the authors in [163] DRL for power control and channel selection to optimize the
propose a distributed resource matching (wireless channel resource allocation for many D2D users in an underlaid D2D
matching and power allocation) scheme based on DDQN to communication network. Further, mean-field RL (MFRL) is
maximize the sum rate and EE of D2D links while maintain- utilized to approximate the D2D users’ interactions. Specif-
ing the SINR threshold of cellular users. Simulation results ically, two algorithms are developed by integrating MFRL
demonstrate that the proposed scheme outperforms the DQN- with spiking actor-critic (AC) and spiking proximal policy
based resource allocation scheme regarding the sum rate. optimization (PPO). It is asserted that the proposed algorithm
Further, a DRL-based federated learning-aided decentralized is effective for many D2D users and outperforms the simple
26 VOLUME ,
Network Layer Communication Architecture Layer

Layer
Cloud storage,
Data Processing/
analysis
LTE/5G
link
IoT Gateway
WiFi.
Zigbee,
Bluetooth,
LoRa,
Sensing Layer NB-IoT,
5G-NR
Smart Smart Smart Smart Smart Smart
Cities Transport Health Energy Living Industry
Application Layer
FIGURE 16. IoT network architecture
PPO and AC-based algorithms regarding overall throughput and machine learning-based algorithms examined across the
and convergence rate. studies discussed in this section.
B. Related works in IoT communication networks and
VII. INTERFERENCE MITIGATION IN IOT NETWORKS their qualitative comparison
A. Features and types of Interference in IoT An IoT network with Wi-Fi and Bluetooth low Energy (BLE)
communication networks communications sharing the 2.4 GHz band is considered
IoT is a communication environment where many devices in [175]. This work presents an LSTM-based lightweight
communicate with each other, and 5G is considered an es- interference prediction model to predict the Wi-Fi interfer-
sential and emerging technique for supporting IoT communi- ence on BLE communication for resource-constraint IoT
cation. It is anticipated that, by the year 2030, approximately devices. The proposed algorithm is compared with the Auto-
80 billion devices will be interconnected within a network Regressive Integrated Moving Average (ARIMA) model
[170]. 5G networks address the major existing challenges where the RMSE results on prediction error indicate the
in cellular networks, such as low latency, ultra-reliability, better performance of the LSTM-based proposed approach
and efficient massive connectivity while supporting mobil- than ARIMA model. Further, a massive IoT network is
ity [171]. 5G-enabled IoT applications include services in considered in [176], where to enhance the communication
smart cities, smart homes, self-driving cars, drones, virtual success rate, a Tug-of-War dynamics-based MAB algorithm
reality, healthcare, and industrial automation, which improve is proposed for channel selection by each IoT device in a
peopleâ C™s daily lives [172]. All the devices used in these distributed manner. The proposed algorithm is implemented
applications communicate with each other or an IoT gateway on an IoT device (Lazurite Pi Gateway) for experiment
(IoTG) or an access point through the low-power wide- and on the ns-3 platform for simulation. Numerical results
area networks technologies, such as LoRa, WiFi, SigFox, demonstrate that the proposed approach for channel selection
ZigBee, and NB-IoT (Narrow-band IoT) [173]. Also, IoT outperforms the random frequency hopping and ϵ-greedy
devices and cellular users may transmit and receive data algorithm in terms of communication success rate in high-
while sharing the same spectrum in 5G-enabled IoT. Usually, density IoT network environments.
IoT devices communicate with an IoTG, and cellular users The research in [177] considers a cognitive IoT network
communicate with BS, using the same spectrum, causing where a new receiver is designed for the secondary user (SU)
interference at BS, cellular user receiver, and IoTG, which in a concurrent spectrum access (CSA) model to efficiently
affects the IoT system’s efficiency and reliability [174]. handle the primary user (PU) interference with minimal
The layered architecture of an IoT network with sensing, knowledge of PU system such as modulation type of PU
network, communication, architecture, and application layer interference. Specifically, an iterative framework consisting
is presented in Fig. 16. In the literature, diverse solutions of a linear estimator, a demodulation and decoding module,
have been proposed to overcome the interference problem and a clustering module is designed. The main focus of
and enhance IoT networks’ overall system throughput and this work is on the realization of clustering module to
EE via proper resource allocation, spectrum management, cluster interference using Gaussian mixture model (GMM)-
and TX power control. This section provides a detailed based expectation maximization (EM) algorithm. Further,
state-of-the-art survey of ML-based intelligent interference extrinsic information technique and modulation constraints
mitigation techniques in IoT networks. Fig. 17 illustrates (MC) are used to enhance the performance of the conven-
the various network systems, objectives, wireless techniques, tional GMM-based EM algorithm, resulting in the Ext-MC-
VOLUME , 27
bands, which determines the interference, a software-defined

Internet-of-Things
radio (SDR) is used with an energy detection module.
Communication Also, to capture the frequency and temporal behavior of
the spectrum, two models are used, namely the Poisson
Network Wireless ML
distribution arrival rate and semi-Markov chains. To improve
Objectives
Systems Techniques Algorithms energy detection performance, the use of an unsupervised
Massive IoT Co-channel
Dynamic Unsupervised
ML algorithm, DBSCAN, is proposed that reduces the FP
Network [176], Interference learning:
Mitigation
Clustering
K-means [192];
and FN rate in capturing the spectrum occupancy, The
Industrial IoT DBSCAN [180];
network [183] Throughput/ Spectrum Gaussian mixture
authors in [181] utilize the sparse cross-entropy minimization
Sum rate Sensing model [177];
LoRa [182], WiFi
maximization Affinity
learning algorithm to recover the narrow-band (NB) interfer-
Propagation
and Bluetooth
[175], Spectral Channel/
[178]
ence of the NB-IoT network to the broadband LTE systems.
efficiency Spectrum
Narrow Band
[181]- enabled IoT
Allocation Further, regularization is introduced to the loss function to
Network Low Latency Neural Networks:
Link DNN [182-183]; enhance the recovery accuracy and convergence rate. Also,
Scheduling Graph NN [184];
B5G-enabled
relay-assisted IoT
Energy CNN [185]; the proposed framework is extended to the MIMO system
Efficiency RNN-LSTM [175]
network [185] Power by exploiting the spatial correlation of the NB interference
SINR Control
UAV-assisted IoT
network [190], maximization
Reinforcement at multi-antennas. The proposed approach is claimed to be
Random Learning:
[192]
BER Channel Access Q-learning more efficient than conventional methods in reducing NB
Protocol [186-188];
Backscatter Comm
based IoT N/w
Minimization SARSA [189]; interference in IoT networks within the LTE system and
DQN [189-190];
[188] Resource DDQN [191]; improving spectral efficiency.
blocks and MA-DRL
Uplink NOMA-
based IoT N/w
network traffic [191-192]; A long-range(LoRA)-enabled static and dynamic IoT net-
balancing MAB-RL [176]
[189] work is considered in [182], where to tackle the resource
Cognitive IoT N/w allocation issue an AI-empowered resource allocation (AI-
[177-179]
ERA) approach is proposed. In the proposed approach, the
DNN model is first trained offline using a synthetic dataset
FIGURE 17. Summary of related works for interference mitigation in the generated by ns-3. Then the pre-trained DNN model is
IoT communications domain utilized online to assign a suitable spreading factor to the
end device before each uplink TX. Numerical results demon-
strate that the proposed scheme outperforms the conventional
EM algorithm. It is shown that the proposed Ext-MC-EM adaptive data rate approaches and SVM in terms of packet
algorithm outperforms the K-means algorithm and MC-EM success ratio and convergence period. Then, a DNN-based
and normal EM algorithm regarding interference mitigation directional MAC (DMAC) protocol is proposed to tackle
and BER minimization. Further, in work [178], a similar the deafness problem for industrial IoT networks [183] that
system setup is considered as stated in [177] with a different utilize mmWave band for communication among IoT devices
objective: multiple PU-interference cancellation at the SU through direction antennas. The proposed DNN model first
receiver. In this work, an affinity propagation (AP) clustering identifies the reason for communication failure, which can
algorithm is used, which works well without cluster centroid be due to interference or deafness, and then avoids deafness
initialization, which is not the case with the GMM-based EM by rescheduling the TX through another beam direction. It
clustering algorithm. Numerical results indicate that the pro- is demonstrated that the proposed ML-based DMAC outper-
posed AP clustering algorithm outperforms the k-means and forms the conventional DMAC protocols regarding deafness
GMM-based EM algorithms to handle multi-interference. In duration minimization and throughput maximization.
the same line, the authors in [179] consider an asynchronous In research [184], a wireless IoT network is modeled
cognitive IoT (C-IoT) network with a system model similar with a directed graph, in which nodes represent the desired
to that considered in [177]. Since the PU and C-IoT devices links and edges represent the interfering links. Further, a
are not synchronized at the symbol level, the interference GNN-based supervised learning framework is used to solve
at the C-IoT device receiver has the first-order Markov resource allocation problems involving integer parameters
property over time. Considering this property, to improve with less execution time. Specifically, a constrained cross-
the performance of the iterative receiver of SU, a hidden entropy algorithm generates the samples to reduce the com-
Markov Model-based AP (HMM-AP) clustering algorithm, putational complexity. Considering D2D communication as
which exploits the temporal correlation of the interference a promising technology for IoT networks, the proposed
from PU, is proposed. It is claimed that the proposed HMM- scheme is verified for D2D link scheduling and joint channel
AP algorithm outperforms the K-means, EM, and AP-based and power allocation for D2D underlaid cellular networks.
clustering algorithms in terms of BER minimization and Numerical results demonstrate that the proposed scheme
interference mitigation. outperforms the unsupervised GNN, the graph embedding
In study [180], an ML framework is proposed to sense method, and conventional DNN regarding average sum rate,
and model the interference in the IoT frequency band. To stability, and computation complexity. The authors in [185]
capture the short-time spectrum occupancy in the shared
28 VOLUME ,
consider the multiband relay nodes assisted B5G-enabled A wireless IoT network is considered with multiple solar-
massive IoT network where relay nodes are used for traffic powered UAVs as relay nodes that leverage NOMA to serve
offloading of cellular base stations. This work proposes a the IoT devices [190]. Specifically, a p-persistent slotted
lightweight DL technique based on the CNN model to select ALOHA protocol is used by IoT devices to access the shared
the best channel for the relay node to transmit and receive channel, and UAVs utilize SIC to decode multiple pieces of
data based on SINR metric quality. Also, the scheduling information received from IoT devices. This work proposes
algorithm at the relay node utilizes channel quality measures a joint problem of dynamic multi-UAV altitude control and
to determine the coding and modulation rates to transmit data random channel access control as constrained MDP. Further,
to the next node. it is claimed that the Shallow CNN model to solve this problem, an online model-free constrained DRL
performs better than Deep CNN concerning throughput, algorithm is proposed to learn an optimal control policy. It
processing delay, and memory consumption. However, the is asserted that the proposed approach outperforms baseline
CSI feedback requirement for the proposed approach creates solutions such as Vanilla policy gradient for total network
an extra overhead on the network. capacity and energy sustainability of solar-powered UAVs.
A massive-scale IoT network of machine-type commu- To ensure low latency and ultra-reliability in D2D com-
nication devices (MTCDs) is considered in [186], where munication for the Internet of Controllable Things, a multi-
to reduce congestion and overload problems, MTCDs are agent DRL algorithm addresses joint subcarrier and power
clustered by the K-means algorithm based on their spa- allocation for D2D pairs [191]. Each D2D pair uses multiple
tial distribution. Then, a centralized approach is presented subcarriers, and subcarriers can be shared among pairs. The
for slot allocation utilizing a graph coloring-based algo- problem is framed as an MDP, and a DDQN-based algorithm
rithm. Furthermore, to mitigate inter-tier interference, a learns the optimal policy without CSI knowledge. Each D2D
distributed block-based Q-learning algorithm with various pair, as a learning agent, iteratively adjusts its resource
reward schemes is proposed for block of slots scheduling allocation strategy through trial and error, outperforming ran-
in MTCDs constituting clustered IoT networks. It is shown dom subcarrier and power assignment schemes in spectrum
that the proposed distributed slot allocation scheme outper- efficiency and interference control. In a related study, mul-
forms the centralized approach in terms of average SIR and tiple UAVs collect data from IoT devices [192]. The multi-
convergence capabilities. Then, a spectrum sensors-aided IoT UAV trajectory design problem, also formulated as an MDP,
network is considered In [187], where to enhance the spatial seeks to minimize mission completion time while avoiding
spectrum re-usability, an RL-based dynamic spectrum access collisions and interference. A multi-agent DRL algorithm
(DSA) algorithm is proposed which learns the spectrum optimizes UAV trajectories both distributively and centrally.
access probability for the slotted ALOHA protocol utilized Numerical results show the proposed approach outperforms
by IoT devices. The proposed RL-based DSA approach is IAQL and fully centralized methods in minimizing mission
claimed to outperform the distributed Carrier Sensing Multi- time.
ple Access with Collision Avoidance (CSMA/CA) protocol
regarding area spectral efficiency.
VIII. INTERFERENCE MITIGATION IN mmWAVE-MIMO
The study in [188] proposes an RL-based power allocation
COMMUNICATION NETWORKS
scheme in a backscatter communications-based clustered IoT
A. Features and types of Interference in mmWave-MIMO
network. First, based on the considered radio environment,
communication networks
states, actions, and rewards of the agent are presented, and
then, the Q-learning-based power allocation scheme is pro- Millimeter wave (mmWave)-massive multiple input multiple
posed to minimize interference and to improve the received output (MIMO) technology is one of the dominant enabling
SINR of the ambient backscatter IoT network. Numerical technologies that are being explored towards achieving the
results demonstrate the proposed method performs better set goals of network performance for the next-generation
than the equal power allocation technique in terms of inter- wireless networks. The use of the mmWave-MIMO system
ference control and received signal strength improvement. brings together the advantages of huge bandwidth availability
The authors in [189] consider the uplink TX in a power in the mmWave spectrum band, and the spatial diversity and
domain NOMA-based IoT Network. This work proposes two multiplexing gains of a massive MIMO system [193]. In
RL-based optimal clustering and resource allocation strate- the mmWave-MIMO system, precoding (via Zero forcing
gies for IoT users in multi-cell networks to control inter- (ZF), Maximum ratio TX (MRT), MMSE, and Dirty paper
cell interference and enhance the long-term average sum coding), beamforming (via analog, digital, and hybrid beam-
rate. The first strategy is based on SARSA (State-Action- forming), and channel estimation with proper feedback are
Reward-state-Action)-learning is suitable for light traffic, and of vital importance to take advantage of spatial diversity and
the other strategy utilizing DRL can handle heavy traffic. multiplexing gain [194]. These techniques also help miti-
Numerical results demonstrate that the proposed schemes gate intra-cell and inter-cell interference to enhance spectral
outperform the conventional OMA (Orthogonal multiple efficiency, EE, and achievable rate performance in ultra-
access) based IoT network in terms of mean sum rate. dense networks [195]. An example of a dense mmWave-
MIMO communication network with inter-cell and intra-cell
VOLUME , 29
Desired Signal Inter-cell Interference Intra-cell interference

mmWave-MIMO Communication
N Antennas-BS
Network Wireless
N Antennas-BS Objectives ML Algorithms
N Antennas-BS Systems Techniques
mmWave-MIMO Inter-beam/ Precoding/

Discriminative
Network Inter-cell Beamforming
Model:
[196-197], [201], interference
mitigation CSI-based Gaussian
[203-204], [208] Clustering Process [198];
SINR Bayesian
N Antennas-BS Cell-free N/w Maximization User learning [199];
[199], [209] Scheduling K-means [197]
Energy
UAV-aided N/w Efficiency Channel Neural Network:
[198], [205-206], Estimation
Spectral GNN [200];
efficiency CNN [203-204],
Ultra-dense N/w Tx Power [212-213];
[200], [202] Sum-rate/ Allocation DNN [201],[211]
Capacity
High-mobility Maximization Reinforcement
Network [210] Beam Steering Learning:
BER Q-learning
NOMA-based minimization [205-207];
Network [207] DQN [208];
FIGURE 18. Illustration of a dense mmWave-MIMO Network Edge user QoS SAC [209];
MIMO-OFDM Constraint Federated- DRL
Network [211] [210]
interference can be visualized in Fig. 18. Nowadays, with the

advancements in ML algorithms, it is being widely explored FIGURE 19. Summary of related works for interference mitigation in
mmWave-MIMO communication networks
by researchers to assist channel prediction, beamforming
optimization, dynamic resource allocation, and interference
management with intelligent ML algorithms. This section
complexity. A UAV-aided mmWave communication scheme
provides an in-depth study of ML-based state-of-the-art
is proposed to improve the QoS of edge users in [198],
works for interference management and resource allocation
where LoS links from primary and coordinated BSs to edge
in mmWave-MIMO systems. Fig. 19 presents the summary
users are blocked, and therefore, UAV acts as a relay to
of different mmWave-MIMO network systems, different ob-
communicate to edge users. In this work, it is assumed that
jectives, wireless techniques, and ML algorithms considered
due to resource constraints, UAV feeds back CSI only to the
in the works covered in this section.
primary BS, and then the primary BS sends this information
to the coordinated BS through the backbone network. To
B. Related works in mmWave-MIMO communication tackle the CSI feedback delay, a Gaussian process-based
networks and their qualitative comparison ML technique is proposed at coordinated BS to estimate
The study in [196] considers the initial access problem for the channel used to perform hybrid beamforming towards
mmWave communication to overcome the signal attenua- the UAV. Furthermore, a Max-SINR-based beamforming
tion and interference problem with optimal beamforming compensation scheme is proposed for the primary BS and
techniques. In this work, a single-path channel model is UAV to eliminate the inter-BS interference. Simulation re-
considered where to get the optimal beamforming vector, sults demonstrate that the proposed method outperforms the
an adaptive and sequential beam search algorithm named schemes without compensation in terms of BER and sum
hierarchical posterior matching algorithm is proposed. The rate, and performs close to perfect CSI feedback scenarios
performance of the proposed algorithm is characterized by in terms of sum rate in the low and medium range of SNR.
the error probability and the resolution of actively learning The authors in [199] design optimal hybrid beamformers
the Angle of Arrival (AoA), which indicates the link quality for unicast, multicast, and broadcast cell-free multi-user
between users and BS. Numerical results demonstrate that mmWave-MIMO systems for capacity maximization and
the proposed algorithm can reach an acceptable resolution inter-user and inter-group interference mitigation with low
and lower probability of error of estimated AoA even in a complexity. In this work, the optimal beamforming solutions
low SNR regime and provide better spectral efficiency than are first derived for the broadcast scenario while considering
bisection and random-search algorithms. In research [197], constraints of total power and per-AP power. Then, optimal
an unsupervised learning-based user scheduling scheme is successive minimum variance hybrid beamforming schemes
proposed for a large-scale multi-antenna system to improve are designed for unicast and multicast scenarios to mitigate
spatial diversity in uplink MAC and downlink broadcast sce- inter-user and inter-group interference. Next, a Bayesian
narios. In this work, the Grassmannian K-Means algorithm learning (BL)-based joint hybrid transceiver (with RF and
is first used for clustering the users with high CSI similarity, baseband transmit precoding and receive combiners (RC)
and then, scheduling is done into MIMO access groups with is designed that does not require AoAs, AoDs, or channel
low CCI. Numerical results show that the proposed scheme gain information. Lastly, a successive cooperative uplink
outperforms direct greedy scheduling in terms of CCI sup- hybrid beamforming scheme is proposed to maximize the
pression and spectrum efficiency with lower computation SINR and data rate for the uplink scenario. Numerical results
30 VOLUME ,
demonstrate that the proposed schemes outperform the no- method of multipliers-based algorithm is proposed to elimi-
coordination scenario in terms of interference cancellation nate residual self-interference and for fully digital beamform-
and spectral efficiency and perform close to fully digital ers optimization. Then, a majorization-minimization-based
design. algorithm is proposed to optimize the Tx and Rx hybrid
The research in [200] considers an ultra-dense D2D beamforming jointly. Furthermore, two learning frameworks
mmWave network and formulates the joint beam selection (ELM-Extreme learning machine-based and CNN-based) are
and link activation problem for interference control between proposed for HBF design, which estimates the precoders
communication pairs and sum-rate maximization. To solve and combiners of full duplex mmWave systems. Numerical
this non-convex and NP-hard integer programming problem, results demonstrate that the proposed schemes perform better
a GNN-based DL framework is proposed by considering in terms of spectral efficiency and computation complexity
the network’s topology. Furthermore, a Lagrangian dual compared to the orthogonal matching pursuit algorithm.
learning framework is proposed to train the parameters The study in [205] considers mmWave multi-UAV com-
of the GNN model. Simulation results demonstrate that munication network consisting of one leader UAV and mul-
the proposed scheme outperforms the greedy approach in tiple follower UAVs. In this work, a Q-learning-based algo-
terms of weighted sum rate and reaches a near-optimal rithm is proposed to deal with the high dynamics of UAVs
solution by comparing it with an exhaustive search so- for analog beam tracking, which uses coupling coefficients
lution in small-scale D2D mmWave networks. Further, a and past and current observations to solve the prediction
DNN-based framework is presented for fast and reliable problem. Further, the desired signal and interference powers
initial access for mmWave networks in [201]. The proposed are estimated using the received pilot signal for the selected
scheme utilizes only a subset of the available beams by analog beams. This helps decide the digital weights for op-
mapping RSSs obtained from a subset of beams to the timal digital beamforming with the objective of SINR max-
beam that is best oriented to the receiver and reduces the imization. It is noted that the received power-based selected
initial access time compared to the conventional continuous analog beams may not result in maximum SINR, hence,
beam sweeping technique. Simulation results show that the additional analog beams are reserved as candidates during
proposed scheme enhances the beam prediction accuracy the beam tracking. Thus, combining analog beams with their
in LoS and NLoS conditions compared to the conventional digital weights, and achieving the maximum SINR provides
initial access scheme. The authors in [202], consider a dense the optimal hybrid beamforming solution. Numerical results
mmWave network where a DL-based centralized solution demonstrate that the proposed scheme significantly increases
is proposed for beam management and interference coor- beam switching and data TX efficiency. Further, in [206],
dination to reduce the computational complexity of radio the authors investigate the beamforming and beam-steering
resource management. Specifically, an optimization problem problem in a multi-UAV and multi-antenna mmWave net-
is formulated for sum rate maximization in a dense mmWave work. In this work, cross-entropy estimation using a robust
network, and to solve this problem, a beamforming training- algorithm is proposed for optimal hybrid beamforming, and a
aided DNN-based algorithm is used to find the optimal beam mean field game (MFG) framework is proposed for realizing
directions, beam widths and transmit power allocation to beam-steering to find the optimal mmWave channel. An
each beam. Numerical results demonstrate that the proposed RL-based learning approach is used to obtain the mean
approach achieves a comparable sum rate to the conventional field equilibrium. Finally, a joint beamforming and beam-
algorithm with much less computation complexity. steering algorithm is presented to maximize the sum rate of
An interference-free achievable rate is introduced for the considered network. Numerical results demonstrate that
wideband multiuser mmWave massive MIMO systems [203]. the proposed scheme outperforms the classical algorithms in
To avoid beam conflict and to maximize the sum rate, the terms of sum rate, spectral efficiency, and EE. The authors in
beam allocation problem is investigated by decoupling the [207] consider a NOMA-based mmWave network where to
analog and digital precoders. Then, the beam allocation mitigate the inter-beam inter-cell interference and to enhance
problem is treated as a multi-label classification problem, and the sum-rate capacity of the network, an RL-based algorithm
to solve this problem, a CNN-based beam allocation scheme is proposed to optimize the user-cell association and inter-
is proposed. A simulated environment is used to generate beam power allocation jointly. Optimal user-cell association
synthetic data to train CNN offline, and then the trained balances usersâ C™ attachments across cells, improving the
CNN is used online to predict the beam allocation for all SIC performance. Numerical results demonstrate that the
the users. Numerical results show that the proposed scheme proposed scheme outperforms the equal power allocation
reduces the computational complexity to a large extent with scheme in terms of network sum rate.
marginal sacrifice in sum-rate performance compared to In research [208], the authors consider the downlink com-
fully digital and Hungarian schemes. The authors in [204] munication scenario in a multi-access OFDM cellular net-
investigate self-interference cancellation and hybrid beam- work with multi-antenna BSs and single-antenna receivers.
forming (HBF) to enhance the spectral efficiency for full This work proposes a centralized DRL-based algorithm for
duplex mmWave relay systems. First, an alternating direction joint beamforming, power control, and interference coordina-
VOLUME , 31
tion to maximize the downlink SINR and sum-rate capacity. prefix-free MIMO-OFDM receiver is proposed to improve
This algorithm does not require CSI and channel estimation, the spectral efficiency. It is demonstrated that the proposed
while it needs users’ locations and SINR information for scheme reduces the detector’s computational complexity and
every millisecond as feedback. Numerical results show that significantly improves detection performance in terms of
the proposed scheme outperforms tabular Q-learning and BER. Further, through experiment, the robustness of the
fixed power allocation algorithms in terms of SINR and proposed scheme is verified in realistic over-the-air (OTA)
sum-rate capacity for voice bearers in the sub-6 GHz band, scenarios. The research in [212] investigates the detection
and achieves maximum sum rate with less time complexity problem at the receiver for MIMO networks with multiple
for data bearers in the mmWave band. Further, the authors antennas while considering the interfering signals correlating
in [209] consider a mmWave MIMO cell-free network in the time and frequency domain. In this work, the joint
which solving the beamforming problem in a centralized use of an MLD (Maximum likelihood Detector) and a
manner is highly complex. Therefore, to solve this problem deep CNN is proposed to tackle the highly complex joint
in a distributed manner, dynamic clustering of the network distribution estimation of the interfering signals. Specifically,
into a group of cell-free subnetworks consisting of an in- MLD produces an initial detection result, and then deep CNN
dependent set of APs and UEs is proposed based on the suppresses the interference and improves the detection by
DRL algorithm. To mitigate inter-subnetwork and intra-UE exploiting the local correlation among interference signals.
interference while maximizing TX rate per UE, a hybrid Furthermore, an improved version of deep CCN is presented
beamforming scheme is proposed. Specifically, a DRL model utilizing the cross-entropy of detection for defining the loss
continuous-action subspace is proposed for analog beam- function of deep CNN which makes the residual interference
steering, and within the environment of the DRL agent for follow the Gaussian distribution. Numerical results verify
analog beam-steering, digital beamforming is solved as a that the proposed detector with Deep CNN outperforms the
convex optimization problem. Numerical results demonstrate conventional MLD regarding BER and MSE minimization.
that the proposed scheme outperforms the conventional all- In [213], the authors investigate the DL-based linear detec-
digital beamforming scheme regarding TX rate. The study tor for MIMO systems over correlated noise environment.
in [210] proposes a DRL-based algorithm for fast and effi- Specifically, the iterative framework of linear detectors and
cient mmWaves beamforming in MIMO system to mitigate deep CNN is employed where the linear detector provides an
the signal attenuation and interference problem for serving initial estimate of the transmitted signals and then deep CNN
highly mobile users (for eg. V2I and high-speed train sce- is used to estimate the transmitted signals more accurately
nario). In this work, to solve the EE maximization problem by capturing the local correlation among noise. The linear
while fulfilling the QoS constraint of each user, access detectors used in this work are ZF (zero-forcing), MMSE ,
points (APs) association per user is considered, and hybrid ZF-SIC (ZF with SIC), and MMSE-SIC. Numerical results
beamforming matrices are obtained using the proposed DRL- demonstrate that the proposed scheme outperforms the con-
based algorithm. Furthermore, to reduce the communication ventional linear detectors in terms of BER.
overhead a federated DRL-based distributed scheme is also
proposed.Numerical results demonstrate that the proposed
IX. INTERFERENCE MITIGATION IN THz
approach outperforms the existing pilot-based beamforming
COMMUNICATION NETWORKS
schemes in terms of achievable rate and EE.
A. Features and types of challenges/interference in THz
Along with working on the transmitter side, the ML
communication Networks
techniques have been proven very effective in improving the
desired signal detection performance at the receiver side of THz-band communication is envisioned as a fundamental
the MIMO systems. The following are some works utilizing enabler of the 6G wireless networks. The THz frequency
neural networks to improve computation complexity, BER, band, typically ranging from 0.1 to 10 terahertz (THz),
and interference mitigation performance at the receiver side holds promise to address the growing demands for ultra-
of the MIMO systems. The authors in [211] present a DL- high data rates (peak data rate: 1 Tbit/s and experience
based signal detector model for the MIMO-OFDM commu- data rate: 10-100 Gbps), low latency (ă 0.1 msec), high
nication system. The proposed scheme considers the iterative EE (100X) compared to 5G, ultra-reliability (99.9999%),
orthogonal approximate message passing (OAMP) algorithm high sensing and positioning accuracy, and massive and
which suppresses interference to a large extent and replaces seamless connectivity that are expected in the 6G era [214].
its matrix inversion process per iteration with a conjugate The THz band has capability of multiplexing thousands of
gradient method to reduce its computational complexity. users simultaneously and can establish ultra-high-BW links.
Then, inspired by an iterative soft-thresholding algorithm Beyond just addressing spectrum congestion, the THz band
(ISTA), the revised OAMP detector is unfolded into a neural is poised to revolutionize society by empowering transfor-
network (NN) through which the detection performance of mative applications [215] such as real-time virtual reality,
the detector could be enhanced by tuning some key trainable augmented reality, holographic telepresence, e-healthcare,
parameters using DL techniques. Furthermore, a cyclic- autonomous vehicles, imaging, localization, positioning, and
tracking of drones etc. While THz channels offer distinct
32 VOLUME ,
opportunities, they also pose unique challenges that are

not observed in traditional wireless environments which are THz Communication
Terahertz band (0.1–10THz)
discussed as follows:
(i) Atmospheric and molecular absorption: One of the Network System Objectives Objectives ML Algorithms
significant challenges is the substantial impact of molec-
Interference Dynamic hybrid Discriminative
ular absorption on path loss within THz band. This THz MIMO-NOMA
mitigation beamforming and Algorithms:
system [223] K-Means [223],
beamselection
absorption loss is contingent on various factors tied to Energy efficiency
Random Forest [224]
Dynamic PHY
time and distance, such as the concentration and specific THz Comm. N/w
with single & layer/Adaptive Neural Networks:
multiple BSs Throughput/sum Modulation and
molecular composition encountered during propaga- [224],[227],[234], rate maximization Coding
RNN [225],
CNN [227],
[237] DNN [229-231]
tion, notably water vapor. Consequently, THz channels Learning-based
BER minimization demodulation
exhibit extreme frequency selectivity, with bandwidth UAV-assisted THz Reinforcement
Learning:
comm. N/w [225], UAV trajectory
significantly contracting or expanding over time. These [232],[236],[238] Computational
complexity
control
RL-Echo State
Network [232],
RL-MAB [233-234],
variations are dependent on factors like the distance RIS-assisted THz
minimization
Dynamic PHY DRL [235-236],
comm. N/w layer/Adaptive BW Federated DRL
between the Tx and Rx (also called distance selectivity) [225],[231],[235]
Coverage
enhancement
& power allocation [237],
MADRL-PPO [238]
or the ambient humidity [216]. As a result, addressing Multi-hop THz
Molecular User Clustering
comm. network and Association
these bandwidth and absorption loss variations necessi- [233],[235]
absorption,
distortions and
tates substantial adaptability, especially concerning the path loss
mitigation
physical layer (PHY) waveforms utilized for TX.
(ii) Propagation loss: The increase in carrier frequencies
FIGURE 20. THz communications network system, objectives, and
and extremely short wavelength of THz waves lead to solution techniques.
higher spreading and intensified NLoS link propagation
losses. These losses are primarily a result of diffuse
scattering, reflection, diffraction, and shadowing [217]. solutions that can effectively address these issues and
(iii) Device and infrastructure development: THz commu- enable efficient deployment of THz networks [222].
nication hardware is still in the development / experi- AI/ML can effectively address THz band challenges (e.g.,
mental stage, and challenges exist in building compact, atmospheric loss, co-channel, and inter-symbol interference)
power-efficient devices and infrastructure [218]. in THz communication systems. Techniques such as intelli-
(iv) Phase Noise: Phase noise is a critical problem in THz gent dynamic hybrid beamforming, beam steering, channel
band that causes inter-carrier interference and severely estimation and prediction, adaptive modulation and coding,
impacts demodulation performance at Rx [219]. OTFS modulation, orbital angular momentum, user cluster-
(v) Beam squint: In THz massive-MIMO systems, wide- ing and association, and optimal resource allocation can
band TXs with large array apertures can introduce mitigate these issues. We will discuss related works that
spatial-frequency wideband (SFW) effects. In this propose AI/ML-based solutions for THz communication
scenario, the channel response becomes frequency- challenges. A summary of these works is shown in Fig. 20.
selective not just due to the delay spread of the multi-
path channel but also because of extensive propagation B. Related works in THz communication networks and
delay across the array aperture [220]. Consequently, the their qualitative comparison
response of the BS array can be frequency-dependent, The authors in [223] investigate EE maximization problem
even in a LoS scenario. When using OFDM to combat in the THz-MIMO-NOMA cache-enabled network. Specifi-
intersymbol interference (ISI), the SFW effect causes cally, the EE maximization problem is divided into three sub-
the Direction of Arrival (DoA) and Direction of Depar- problems: channel correlation-based user clustering, hybrid
ture (DoD) of signals to vary across subcarriers. This precoding, and optimal power allocation to users in clusters
phenomenon, termed beam squint, requires frequency- to mitigate intra-cluster and multi-cluster interference. For
dependent beamforming/combining. However, typical user clustering, an enhanced K-means algorithm is proposed
hybrid array architectures of THz massive MIMO lack for fast convergence. Then, a hybrid precoding scheme is
this capability. Hence, compensating for beam squint is adopted based on the sub-connection structure (where each
crucial for THz massive MIMO-OFDM systems [221]. RF chain is connected to only part of the antennas through
(vi) Limited coverage range: To tackle the restricted cov- phase shifters, not to all antennas in the array) to reduce
erage range of THz networks due to high penetration power consumption and implementation complexity. Lastly,
and propagation loss, strategies involving ultra-dense a distributed ADMM (Alternating Direction Method of Mul-
deployments and highly directional antennas are envis- tipliers) algorithm is designed for power allocation to the
aged in THz networks. However, such deployments sub- users in clusters with imperfect SIC. It is demonstrated that
stantially elevate CCI levels, limiting network density the proposed scheme achieves better clustering convergence
and presenting challenges regarding beam alignment rate and EE performance than benchmark schemes such
and tracking. Hence, there is a need for innovative as conventional k-means-based clustering and equal/random
VOLUME , 33
power allocation. The study in [224] investigates the beam- demodulation technique is proposed to enhance the BER
selection problem in THz communication system as a multi- performance of the Rx system. Specifically, a multi-output
class classification problem. To solve this problem, the deep feed-forward neural network is proposed for nonlinear
authors propose the use of a random forest ML algorithm mapping between the Rx symbols and the demodulation
that is trained by using the simulated data. The proposed likelihood information vector. Furthermore, a training data
scheme provides a better tradeoff between the computational generation method is presented in which prior knowledge
complexity of hybrid beamforming and sum-rate maximiza- of likelihood information is not required. Simulation results
tion in comparison to classical beam selection schemes such show that the proposed demodulation methodology outper-
as exhaustive search method and SVM-based beam selection. forms the MMSE equalization and SVM-based schemes in
terms of BER performance for THz communication systems
A reconfigurable intelligent surface (RIS)-assisted drone under severe hybrid distortions.
THz-communication network is considered in [225], where In research [230], a multiuser THz communication system
a mobile drone establishes and maintains a reliable com- is considered with rapid variation in molecular absorption
munication link with the BS through either a direct or a coefficient (MBC) within the spectrum of interest. In this
RIS-assisted link. To ensure a seamless communication link work, to minimize the variation of molecular absorption loss
between the BS and the drone while considering the drone among users and to maximize the aggregated multiuser data
mobility, the authors propose a gated recurrent unit (GRU)- rate, a DNN-based adaptive sub-band bandwidth allocation
based RNN DL algorithm to predict the most optimal com- and power control algorithm is proposed. The proposed DNN
munication link (be it direct or RIS-assisted) and to forecast algorithm is trained offline by using a Lagrangian-based Loss
the most suitable beamforming vector for that specific link. function. Then, the trained DNN is used to find the near-
Furthermore, the deep MIMO data-generation framework optimal solutions to the optimization problem. Simulation
[226] is used to build the scenario and to develop a dataset to outcomes demonstrate that the aggregated multiuser data rate
evaluate the proposed scheme. Numerical results show that achieved using the proposed DNN-based algorithm surpasses
the proposed approach achieves high accuracy in determining the results obtained by the existing convex optimization
the communication link (handoff) and serving beam selec- approach, particularly in scenarios where the MBC exhibits
tion. In study [227], to enhance the security against smart highly non-linear variations within the spectrum of interest.
interference and to tackle the communication challenges in The study in [231] tackles the coexistence challenge of
THz-band such as molecular absorption loss, frequency- eMBB and URLLC services in RIS-assisted THz communi-
selectivity, and distance-dependent bandwidth change, a DL- cations and proposes a resource management framework op-
based bandwidth (BW) and modulation (MD) classification timizing parameters such as BS transmit power, RIS reflec-
technique is presented to make the physical layer very tion coefficients, and resource block allocation. Specifically,
flexible where Tx and Rx can change their PHY parameters an alternative decomposition method-based optimization pol-
such as BW and MD without coordination. Here, the Tx icy is proposed that decomposes the resource management
selects its BW and MD parameters using CSI reports Rx problem in two parts: one for resource block allocation and
sends periodically. At the Rx side, the processed and down- the second for reflection coefficient and power allocation
converted baseband signal is sent to CNN to infer BW and addresses the joint problem iteratively. Next, an LSTM
and MD of the incoming THz signal. Then CNN inference network is trained using optimized parameters for real-time
is used by a reconfigurable DSP logic to demodulate and prediction and its learning rate hyper-parameter is tuned
recover the application data. This work extensively trains and via the Bayesian optimizer. Finally, multiple LSTMs are
tests CNN classifiers using the experimental data collected integrated into an ensemble model to enhance the prediction
through TeraNova testbed [228]. The experimental dataset accuracy. The numerical results show substantial eMBB
consists of In-phase/Quadrature (I/Q) samples collected at spectral efficiency gains with RIS of size 11x11, reaching
the 120-GHz RF frequency with five MD schemes (BPSK, up to 49% with the proposed optimization approach, 34%
QPSK, 8PSK, 16QAM, and 64QAM), three BWs (5, 10, and with the LSTM DL method, and 48% with the ensemble
20 GHz), and two SNR levels (low, high). Numerical results learning method, compared to a DRL-based scheme.
prove the effectiveness of the proposed DL-based modulation The authors in [232] consider a multi-drone-based net-
and BW classification scheme at THz frequencies with 78% work in which different spectrum bands such as microwave,
and 90% accuracy in low and high SNR conditions. mmWave, and THz bands provide data streaming services to
The authors in [229] focus on tackling the hybrid dis- users on the ground in blockage-rich mobile environments.
tortions in THz communication systems, such as phase This work investigates the sum rate maximization problem
noise, phase shift, I/Q imbalance, and non-linearity of power by jointly controlling the drones’ flight, user association,
amplifiers. These distortions cause a mathematical-model- and spectrum band selection for serving the users. It is
deficit scenario that limits the performance and application assumed that a user can be served one spectrum band at a
of the conventional demodulation techniques in THz commu- time, while a drone can serve multiple users using different
nication systems. Consequently, in this work, a DL-assisted spectrum bands based on channel conditions. To solve the
34 VOLUME ,
formulated MINLP (Mixed Integer nonlinear programming) time delay, considering the resource and QoS constraints and
throughput maximization problem, an echo state learning- swiftly changing environment. To optimize UAV placement,
based utility prediction and flight optimization framework is computation offloading, and resource allocation within this
proposed while combining it with an RL-based flight control architecture, the study introduces the DDQN and DDPG
distributed algorithm. Simulation results through a newly algorithms. The simulation outcomes demonstrate the al-
developed event-driven simulator, SimBAG (Broadband Air- gorithm’s efficient convergence and lower latency than full
to-Ground network simulator), demonstrate that to fulfill the local computing and complete offloading methods.
demands of comprehensive coverage and high data rate, In [237], a THz-communication network with multiple
using the combination of different spectrum bands is a BSs coordinated by an edge server is studied. To minimize
desirable solution, and THz-band has excellent potential to inter-cell interference (ICI) and maximize network through-
enhance the throughput in a blockage-rich environment by put, a federated DRL algorithm for THz-beam searching
taking advantage of the mobility of flying drone BSs. is proposed, using DDPG-based learning with limited CSI.
The study in [233] focuses on a multi-hop THz communi- BSs update their DDPG models with partially estimated
cation system and introduces a novel RL-based beam training CSI through Federated Edge Learning (FEL) to mitigate
approach. This approach aims to minimize the time spent ICI. Numerical results show the federated DRL-based beam-
on beam searching and enhance network spectral efficiency. forming scheme outperforms conventional ZF and MMSE
Specifically, an MAB-based hierarchical beam training algo- beamforming in computation complexity and throughput.
rithm is proposed to optimize the number of beam training In [238], a mobile edge computing (MEC)-assisted UAV
levels at each hop, facilitating rapid convergence and re- swarm network is explored for task offloading of ground
ducing beam searching time overhead. Ultimately, this leads users using THz-band communication. The goal is to max-
to improved spectral efficiency in the multi-hop THz com- imize network utility by optimizing UAVs’ resource allo-
munication system. Numerical results substantiate that the cation and trajectories. The NP-hard utility maximization
proposed scheme surpasses existing beam training schemes problem is divided into two subproblems: local and of-
having a fixed and random number of training levels in terms floaded computation resource allocation, solved by convex
of spectral efficiency. The research in [234] focuses on a optimization, and optimizing MEC-UAV trajectories, data
dense THz communication system and introduces a time- offloading, and communication resources, solved by a multi-
domain interference mitigation technique using an RL-based agent PPO DRL algorithm with an Embedded Multi-Head
algorithm at the Rx to improve the BER performance of Attention (MHA) mechanism. Numerical results show this
the communication system. The proposed technique utilizes scheme outperforms fairness-based resource block and power
an MAB-based strategy to determine the best threshold value allocation and general MAPPO in network utility.
from multiple threshold clipping levels for a threshold-based
nonlinear preprocessor at the Rx. Simulation results indicate
C. Features and types of interference in NOMA-based
that this approach converges rapidly and surpasses traditional
communication networks
interference mitigation methods like blanking and single-
threshold clipping schemes in terms of BER performance. NOMA is a promising multiple-access technique to en-
The authors in [235] consider the multi-hop RIS- hance spectral efficiency, user fairness, massive connectivity,
empowered THz communication system to tackle the prop- and cell edge throughput in future wireless technologies
agation attenuation and atmospheric absorption challenges [239]. Different multiple access techniques were utilized
and enhance the coverage range in THz-band. Specifically, in previous wireless technologies such as FDMA in 1G,
to increase the TX range between BS and users, multiple TDMA in 2G, CDMA in 3G, and OFDMA in 4G and
controllable RISs are deployed, and the joint design of 5G, where frequency, time, and code resources are divided
analog beamforming at RISs and digital beamforming at among users to avoid interference for simultaneous infor-
BS is investigated using the DRL-based algorithm. Fur- mation TX. In contrast with previous orthogonal multiple
thermore, two alternating optimization-based methods are access (OMA) schemes, NOMA allows simultaneous TX
proposed for initializing the beamforming metrics, which of multiple users using the same frequency-time resources
enhance the convergence rate of the proposed DRL-based using power domain (PD-NOMA) or code-domain (CD-
algorithm. Numerical results show that the proposed DRL- NOMA) multiplexing [240]. In PD-NOMA, multiple users’
based scheme outperforms the benchmarks regarding the signals are superimposed using different power levels, while
THz communications system’s total throughput and coverage in CD-NOMA it is performed by using different signature
range. The research in [236] explores an architecture uti- codes [241]. Also, NOMA can be integrated with advanced
lizing UAVs for offloading computations of wireless users’ technologies such as mmWave, massive MIMO, beamform-
devices in the THz band. The goal is to increase system ing [239], and machine-type communication [242] to ful-
capacity while meeting the real-time latency needs of wire- fill the ever-increasing demands of next-generation wireless
less users having limited computational capacity. This work networks. NOMA indeed enables the efficient utilization of
proposes an optimization problem to minimize the overall resources at the Tx, however, it leads to a non-orthogonal
mixture of signals interfering with each other at the Rx side
VOLUME , 35
Received Subtract Subtract Subtract

Signal UE1 Signal UE2 Signal UEn-1 Signal SIC
SIC SIC SIC Downlink
Scenario Decode and subtract
UE2 to UEm signals
UE1 Signal UE2 Signal UE3 Signal UEn Signal sequentially
Decoding Decoding Decoding Decoding
Decode
UE1 Signal
UE1
SIC
Uplink
Scenario UE1 Decode and subtract
UE3 to UEm signals
UE2 sequentially
UE2 Decode
P1 UEm Pm UE2 Signal
UE1
UEn Power Power
Decode
UEm Signal
P2 UE2 P2
UE2
Pn UE1 P1 UEm
UEn
Frequency Frequency
FIGURE 21. Illustration of uplink and downlink NOMA-based communication with Successive Interference Cancellation (SIC)
[243]. To tackle this interference, more efficient protocols extraction process is based on the AndersonâC“Darling
and algorithms are required for channel estimation, resource test, followed by a logistic regression-based classification
allocation, dynamic clustering, and user scheduling at Tx to determine the modulation order of the interfering user.
side, and more advanced signal processing techniques are Numerical results demonstrate the superior performance of
required for SIC, and proper decoding of information at Rx the proposed approach over the max-log algorithm in terms
[244]. An illustration of Uplink and downlink NOMA-based of detection rate and throughput. In research [246], a cell-free
communication with SIC is presented in Fig. 21. Recently, massive MIMO-NOMA network is considered to enhance
ML techniques have been widely used to assist the NOMA the sum spectral efficiency (SSE). In this work, an advanced
technology by optimal user clustering, power allocation, K-means algorithm is used for user clustering which defines
channel allocation, optimal order of SIC decoding, and the initialization of centroids of the clusters, and ZF precod-
interference management based on learned patterns from ing is used to mitigate inter-cluster interference. Then, to
historical data to enhance the network performance. This solve SSE optimization problem, optimal power is allocated
section provides a detailed study of ML-based works in the to users for proper SIC decoding while considering the power
NOMA domain with different network systems architectures constraints at APs and QoS constraints of user equipment.
focusing on downlink and uplink communications, with An efficient iterative algorithm is proposed based on the
different objectives, wireless techniques, and ML algorithms inner approximation method to obtain the optimal power
used to achieve set objectives. A summary of these details allocation. Simulation results show that the proposed user
can be visualized in Fig. 22. clustering scheme outperforms the random and conventional
K-means-based clustering schemes regarding SSE.
Then, the authors in [247] propose a novel cluster-free
X. INTERFERENCE MITIGATION IN NOMA-BASED NOMA framework in a multi-cell network system. In this
COMMUNICATION NETWORKS work, an auto-learning GNN is used for distributed multi-cell
A. Related works in NOMA-based communication coordinated beamforming and cluster-free SIC decoding. The
Networks and their qualitative comparison objectives of this work are sum rate maximization and inter-
1) NOMA-based downlink communication scenarios: cell and intra-cell interference mitigation while fulfilling the
Initially, we delve into the works centered on NOMA- usersâC™ minimum rate constraints. Moreover, a bi-level
based downlink communication scenarios leveraging ML learning algorithm is proposed to train the GNN architecture
techniques to augment the performance of various network parameters and model weights to reduce the complexity
metrics. The NOMA-based communication system is con- and information overhead for distributed coordinated beam-
sidered in [245], where assistance information such as mod- forming and cluster-free SIC optimization. Simulation results
ulation order of interfering users is necessary for canceling show that the proposed cluster-free NOMA framework out-
the inter-superposition-layer interference at the Rx using the performs the conventional cluster-based NOMA in terms of
SIC technique. In this work, a feature-based blind detec- sum rate under multi-cell scenarios and reduces the informa-
tion method is employed to derive assistance information tion and computation overhead in comparison to fixed GNN
from received constellation points. Specifically, the feature
36 VOLUME ,
NOMA-based Communication
FIGURE 22. Summary of related works for interference mitigation in NOMA-based communication networks
and distributed ADMM (Alternating Direction Method for the cost of high power consumption. A NOMA framework
Multipliers) techniques. A NOMA-based heterogeneous IoT is considered for cognitive radio networks [249], wherein
system is considered in In [248], wherein based on QoS a DL-based technique is utilized for joint signal detection
targets, mobile users and IoT users are classified as PUs and fair power allocation for PUs and SUs accessing the
(Primary users) and SUs (Secondary users). In this work, to same frequency bands. Specifically, an LSTM-based trained
enhance the EE of the NOMA system with imperfect SIC, algorithm is used in online mode using which multi-user
a peeling-based subchannel assignment scheme is proposed detection (MUD) is performed in one shot. The DNN algo-
for PUs, and reweighted message-passing algorithm-based rithm’s training is done offline using the pilot sequences. The
resource allocation is done for SUs. Also, a deep RNN-based proposed framework is implemented through an experimen-
algorithm is proposed to solve the problem of resource allo- tal setup using SDR with one PU and two SUs. Analytical
cation for SUs. Furthermore, a rate-demands and priorities- and experimental results demonstrate the superiority of the
based user scheduling method is put in use to synchronize the proposed technique over the conventional SIC techniques in
access of heterogeneous users with limited radio resources. terms of Symbol error rate and system throughput.
Numerical results demonstrate that the proposed scheme A NOMA-based mmWave heterogeneous network is con-
outperforms the OFDMA system in terms of sum rate but at sidered in [250] where it is assumed that up to two users
VOLUME , 37
can utilize a subchannel through SIC. This work aims to results show it outperforms conventional SIC in terms of
maximize the network’s EE while fulfilling the users’ QoS BER and network capacity, especially with more IoT de-
requirements and limiting inter-tier interference and power vices. The research in [255] proposes a DNN-based SIC
consumption. To achieve the set target, the Lagrange dual technique for intelligent detection sequencing in uplink
decomposition method is used for user association, a semi- NOMA scenarios. The algorithm uses received signal infor-
supervised learning-based regression algorithm is utilized mation, channel characteristics, and Tx power to optimize
for channel allocation and DNN is employed for power the detection sequence. Numerical results demonstrate better
allocation. An iterative algorithm is used for generating the BER and detection performance than conventional CSI-based
sample data. Simulation results verify the effectiveness of the SIC schemes.
proposed schemes over the conventional methods for power Research focused on improving uplink communication
and subchannel allocation. The authors in [251] consider in NOMA systems with machine learning techniques has
the MIMO-NOMA system wherein to tackle the issue of shown promising results. Considered in one study, an up-
imperfect SIC, a FNNs (Feed-forward Neural networks)- link MIMO-NOMA system employs a DL-based successive
based DL technique is proposed for joint optimal precoding interference cancellation (SIC) scheme [256]. Each step of
and SIC decoding of multiple users. This work aims to the SIC process utilizes DNNs at the BS to decode users’
minimize the mean square error (MSE) between the desired signals. The DNN takes combined received signals and
and decoded signals of the users. The numerical results previously decoded symbols as inputs, outputting transmitted
demonstrate that the proposed scheme outperforms the con- bit sequences. Pilot symbols transmitted at the beginning
ventional SIC schemes regarding MSE performance. of each frame train the DNNs, demonstrating improved
In research [252], a backscatter-NOMA network is con- BER and computational complexity performance compared
sidered that consist of one full-duplex BS, multiple down- to conventional SIC methods. In another scenario, a MISO-
link users, and multiple uplink backscatter devices. This NOMA system utilizes DNNs to construct transmitted signal
work aims to maximize the sum rate of uplink backscatter symbols [257]. The softmax function enhances detector
communication while fulfilling the QoS requirement of the reliability instead of relying on hard decisions. Soft infor-
downlink users. To achieve this objective, the reflection mation from detected signals is used in the SIC process
coefficient of backscatter devices and the NOMA-based and as input for subsequent DNNs, leading to superior BER
resource allocation i.e. Tx power of downlink users are performance over conventional methods. Furthermore, in an
jointly optimized using a soft actor-critic (SAC)-based RL SIMO-NOMA-based network, a pilot-assisted model-driven
algorithm. In this algorithm, the SAC agent explores and DL algorithm enhances receiver SIC performance [258]. This
exploits the dynamic backscatter-NOMA network efficiently approach introduces new learnable parametersâ C”noise can-
by entropy regularization. Simulation results demonstrate cellation and interference cancellationâC”to improve channel
that the proposed scheme outperforms backscatter-OMA and estimation and SIC-based signal detection accuracy. Numeri-
random power allocation techniques regarding the average cal results highlight its superiority over conventional MMSE
sum rate of uplink backscatter devices. The authors in [253] and DNN approaches in terms of both BER performance and
consider the downlink multi-carrier NOMA system wherein computational complexity.
to maximize the sum rate and minimal rate of the network, A NOMA-based uplink communication scenario is con-
a DRL-based algorithm is proposed for the optimal channel sidered in a UAV-enabled IoT network in [259], where an
assignment to users. Specifically, the attention-based neural adaptive RL-based resource allocation algorithm is proposed
network having an encoder-decoder structure is used for to maximize the sum rate of the network. Specifically,
policy formulation of the DRL framework. It is assumed that the Mamdani fuzzy inference system (MFIS) is utilized
only two users can be served per channel using the power- to supervise the MAB-based policy of the RL algorithm
domain NOMA technique. The optimal power allocation is for adaptive exploration and exploitation. Both Joint max-
done for both users for the given channel assignment so imum likelihood detection (JMLD) and SIC are used for
proper SIC decoding can be performed at the user end. information decoding at the receiver. It is claimed that the
Simulation results show that the proposed scheme outper- proposed scheme outperforms the random exploration and ϵ-
forms the conventional joint resource allocation technique greedy policy-based non-adaptive RL counterpart regarding
regarding sum rate and minimal rate performance. sum rate, outage probability, and BER performance. Also,
it is demonstrated that JMLD performs better than the
SIC-based detection in terms of BER but at the cost of
2) NOMA-based uplink communication scenarios:
increased complexity. Further, A multi-cell uplink NOMA-
We now explore how ML techniques improve network IoT network is considered in [260], where multiple IoT users
performance in NOMA-based uplink communication scenar- use NOMA technique to share the same RBs. This work aims
ios. Researchers in [254] focus on grant-free NOMA IoT to maximize the sum rate of the network with constraints
networks, proposing a DL-based algorithm for MUD. An on transmit power, channel gains, and the number of users.
LSTM-based network is trained offline using pilot signals, To solve the constrained sum-rate maximization problem,
performing joint channel estimation and MUD. Numerical
38 VOLUME ,
SARSA and DRL algorithms are proposed for low traffic and [42], [43], Support Vector Machine [41], [42], [44], [45],
high traffic density, respectively. These algorithms provide and k-nearest neighbor [43], [45]. The main features of the
dynamic clustering solutions for connecting IoT users to signals used for jamming detection and classification are
BSs, and sub-channel allocation based on the received power PDR [41], [43], [44], BPR [44], and RSSI [43]–[45], [53].
levels of IoT users. It is asserted that compared to OMA, To train these models data is generated either by simula-
the proposed NOMA-based solution provides better sum tions [41], [43]–[45] or by real-time experiments [45]. The
rate performance, and both SARSA and DRL exhibit lower performance of different algorithms is compared by metrics
complexity than the optimal benchmark scheme. such as detection and classification accuracy. As reported by
In research [261], a NOMA-based uplink communication several works, the performance of random forest is superior
system is considered with an imperfect SIC. In this work, to that of other supervised learning algorithms in terms
an actor-critic learning-based DRL algorithm is proposed of jamming detection accuracy [43]–[45]. Furthermore, re-
to optimize the dynamic user access and optimal decoding searchers can utilize the GAN model to detect the anomalies
order for the SIC process to maximize the number of inside the radio spectrum [51] and to detect adversarial
successfully served users. Simulation results show that the jamming attacks [52]. Also, deep learning algorithms such
proposed algorithm outperforms the conventional access and as CNN can help detect jamming attacks using spectrogram
decoding schemes in terms of the number of successfully images [46] and ensure the transmitter’s legitimacy using
communicating users with heterogeneous QoS demands and RF fingerprinting [49]. Federated learning is also a viable
imperfect SIC. The authors in [262] consider an uplink candidate for detecting jamming attacks in a distributed
grant-free NOMA system to reduce signaling overhead and manner [47]. Publicly available datasets such as CRAWDAD
latency for massive access. To reduce the collisions of [48] and POWDER [50] are available for researchers to train
uncoordinated access and to improve the system throughput, and test their jamming detection algorithms.
first, subchannels and user devices are divided into different Regarding jamming mitigation: For all different types of
clusters. Then, to improve the long-term throughput of communication scenarios such as UAV-assisted communi-
the clusters, a DRL-based algorithm is used for channel cation [56]–[62], IoT communication [63]–[65], mmWave-
selection and power allocation. Here, discrete power control MIMO communication [66], [67], vehicular network com-
is adopted to reduce the computational complexity of DRL munication [69], 5G HetNet communication [68], and THz
algorithm. Numerical results show that the proposed DRL- communication [73], the main focus is on TX power control
based algorithm for grant-free NOMA outperforms slotted to mitigate the jamming attacks [57], [60]–[63], [66], [67],
ALOHA NOMA with 156% gain on system throughput [69]. Specifically for UAV-assisted communication, relay
when the number of devices is five times more than the selection [58], [59], [61] is also crucial for jamming mit-
number of subchannels. igation along with TX power control. The widely accepted
approach to address power control and relay section is the
XI. LESSONS LEARNED, CHALLENGES AND OPEN
utilization of RL techniques. The performance metrics to
ISSUES
evaluate jamming mitigation are presented in terms of BER
In this section, the intentional interference detection and mit- [54], [58], [59], [61], [62], throughput [64], [66], [68], [69],
igation works are reviewed for insightful outcomes, followed and energy consumption [59], [62], [63]. For eavesdropping
by a thorough review of unintentional interference manage- mitigation, channel allocation [70], [73] is also considered
ment across different domains of 5G and beyond networks. along with power control. Mainly, RL-based algorithms such
For each domain, the discussion is structured into two main as DQN [72], DDPG [71], D3QN [70] can be used for
parts. Initially, we analyze insights and lessons learned from resource (power and channel) allocation such that secrecy
the state-of-the-art works featured in that domain to provide rate [70]–[72] can be maximized.
cognizance for researchers on do’s and don’ts. Subsequently,
we unveil the opportunities by outlining the challenges of 2) Challenges and Open Issues:
utilizing the different ML-based techniques and delving Designing and implementing ML-based jamming detec-
into the open issues/future research directions concerning tion and mitigation techniques are acquainted with several
interference handling for that specific enabling technology challenges and open issues. Following are some challenges
within next-generation wireless communication networks. and future research directions related to ML-based jamming
detection and mitigation.
A. Intentional-interference/Jamming Detection and First, collecting a diverse dataset with multiple features
Mitigation and labeled examples of jamming instances (simulated and
1) Lessons Learned: injected into real data) via real-time setups for the training
For the jamming detection and classification: Supervised of ML models can be challenging. Data with limited features
learning is the most widely used ML algorithm category can obstruct the competence of ML models to get accepted
for jamming detection and classification. Most often used as a general solution for various jamming scenarios. Since
algorithms include random forest [43]–[45], Decision Tree the jamming types or techniques may advance with time,
VOLUME , 39
adaptive ML models should be designed to detect innovative learning approach named affinity propagation can be used
jamming attacks. Effective feature selection and jammed [80], [83], and for the UAVs positioning control, the k-means
signal representation are crucial for jamming detection since algorithm can be used as suggested by the related works
sparse and imprecise feature representation can cause in- [80], [81], [84]. MADRL algorithms such as MA-DQN [89]
accurate results for jamming detection. Second, ML-based and MA-DDQN [90] can be utilized for UAV trajectory
jamming detection and mitigation models should be of low control. Researchers can use equal power allocation [84],
computational complexity to tackle the problem in real-time random power allocation [87], pure Q-learning [83], [87],
processing for time-critical applications. Along with lower genetic [81], [83], and heuristic approach [81] as baselines
computational complexity, the EE of ML models is crucial to compare their solutions against.
for deploying those models on energy-constrained or battery- Scenarios with multiple UAVs acting as aerial users: The
operated devices. main objectives are to reduce the co-channel interference
Third, the practical applicability of the proposed models from other ground base stations (GBSs) to UAVs in downlink
is a phenomenal challenge. This is because a model trained communication [92]–[94], [96]–[99] and to mitigate UAV-
in one environment may not be suitable for another environ- to-GUE interference in uplink communication [92], [95].
ment. For example, the assumptions made in a simulated or Various strategies are available to mitigate the interference.
lab environment may differ from the practical field. Design- These strategies include BS association [92], [94]–[96],
ing a generalized model applicable to different operational power control of GBSs [92], [93], [95], [98], [99], and UAV
environments is critical for practical implementation. At last, path planning/ trajectory design [94], [95] so that the rate
the proposed ML- model should be robust enough to handle per UAV/GUE [92], [93], [95], [96], EE [92], [98] can be
adversarial attacks that can elude the model and hinder maximized, and Tx latency per UAV [95] connectivity outage
its deployment in real-world applications. Furthermore, a of UAV [94], [97] can be minimized. The most commonly
jamming detection model can access communication signals used ML techniques include RL such as DQN for Tx power
for analysis, which may raise privacy concerns. Therefore, control [93] and trajectory design [94], ESN (Echo state
considering privacy is essential while designing an effective network)-DRL [95] for path planning, BS-association, and
jamming detection and mitigation model. To address these power allocation, and Dueling DQN [96] for BS-association.
challenges, it is crucial to consider advanced ML algorithms,
refine data collection strategies, leverage domain adaptation
techniques, and encourage collaboration among researchers 2) Challenges and Open Issues:
and domain experts. Employing ML-based interference mitigation techniques in
UAV-assisted communication networks instigates many chal-
B. UAV-based Communication Networks lenges and open issues. UAVs’ flexibility in placement,
trajectory designing, and air-to-ground channel modeling
1) Lessons Learned: bring unique characteristics and operational challenges that
We summarize the lessons learned from AI-enabled in- call for proficient ML-based techniques to tackle interference
terference detection and mitigation in two steps: 1) lessons mitigation problems in this highly dynamic environment.
learned for scenarios with multiple UAVs serving as aerial Below are some key challenges and open issues related
base stations, 2) lessons learned for scenarios with multiple to AI/ML-enabled interference management in the UAV-
UAVs acting as aerial users. assisted communication networks domain.
Scenarios with multiple UAVs serving as aerial base sta- First, in multiple UAV scenarios, since UAVs can have
tions: To mitigate co-channel interference and to maximize three-dimensional mobility with varying altitudes and oper-
the throughput/sum rate [80], [82]–[86], [88], [89] and EE ate in dynamic environments, interference estimation, pre-
[80], [81], [85], [87], [89], [90] of UAVs-assisted networks, diction, and modeling become more complicated. Therefore,
the main focus remains on UAV positioning [80]–[82], [84], considering the temporal and spatial characteristics of UAV-
[85], trajectory control [86], [89], [90], Tx power control assisted networks is important for ML models for interfer-
[81]–[87], and switching the highly interfering drone small ence control. For example, RNNs can be leveraged to deal
cells (DSCs) into sleep mode [80], [81], [83]. Further, to with time-dependent data and predict the optimal trajectory
solve the UAV positioning and power control problem, re- of UAVs. Since UAV-assisted networks are highly dynamic
searchers can use both distributed approaches such as multi- with varying interference characteristics, ML models must
agent DRL [85], [89], [90] as well as centralized approaches be adaptive to real-time interference variations due to the
such as Q-learning [83], [84], DRL [87], SAC DRL [86] mobility of UAVs. Also, a realistic path-loss and signal
and unsupervised learning algorithms [80], [81], [84]. It propagation model (Air-to-ground) should be considered
should be noted that a coordinated distributed approach using for the design of interference mitigation techniques to be
federated learning performs better than an uncoordinated practically deployable in real-time scenarios.
distributed approach in terms of network throughput and EE As UAVs are energy and computational resource-
[85]. To switch the DSCs into sleep mode, an unsupervised constrained devices, the proposed ML models must be of
40 VOLUME ,
lower computational complexity, higher EE, and capable of resource allocation using various ML techniques such as
real-time interference mitigation with low latency in highly DNN [108], soft actor-critic [115], LSTM [111], as well
dynamic UAVs-assisted communication environments. In as DQN and DDPG [114]. While utilizing mmWave to
multi-UAV scenarios, the cooperative-distributed learning serve vehicles via V2I links in a highly mobile environ-
approaches must be investigated for effective interference ment, optimal beam selection is critical to make seamless
management via Tx power control, resource block allocation, connections of vehicles with infrastructure. For example, a
switching UAVs into sleep mode, trajectory control, and combinatorial MAB approach can be useful for optimal beam
user association. While using discriminative ML approaches selection to maximize the overall network throughput [110].
shows promising performance for UAV positioning and Age of information is also a critical metric in a vehicular
switching the drone’s small cells into sleep mode, limited environment when working with AI/ML models in real-time.
data may hamper the generalization of ML models to di- This problem can be addressed via decentralized resource
verse and dynamic UAV-assisted communication scenarios. allocation for vehicles using the Gaussian process regression
Lastly, ML models are expected to consider integrating UAV- technique [105].
assisted networks with existing cellular infrastructure and 2) Challenges and Open Issues:
advanced signaling techniques such as mmWave-MIMO, The use of machine learning techniques for interference
NOMA, and THz Communication. Furthermore, the pro- mitigation in cellular-assisted V2X communication networks
posed ML models must be robust enough to handle the pri- demonstrates several open issues and challenges due to the
vacy and security concerns of UAV-assisted communication high mobility of vehicles and the continuously changing
networks. topology of vehicular networks. Cellular-assisted vehicular
networks consist of simultaneous V2V, V2I, V2P, and cel-
C. Vehicle to Everything (V2X) Communication Networks lular BS to cellular users that communicate by utilizing the
same spectrum band, creating CCI with each other. Below
we list some key challenges and open issues in the V2X
1) Lessons Learned:
communication networks domain.
In this domain, researchers mainly focus on minimizing
Since vehicular networks experience high mobility and
the CCI among V2I links, V2V links, and cellular users
frequent topology changes, ML models should be able to
sharing the same spectrum bands. Below, we list key lessons
adapt to the dynamic vehicular environment in real-time
learned from the state-of-the-art works concerning AI-based
scenarios with acceptable time and space complexity. The
interference detection/mitigation in V2X communication net-
ML models must be time-efficient to tackle the safety-
works: To avoid co-channel interference, resource allocation
critical/time-critical applications with low latency in real-
in terms of resource blocks (RBs) and Tx power to V2V
time implementations. There are solutions with distributed
links and V2I links is a widely used approach. Some works
and centralized approaches for interference management in
focus on only resource block allocation [107], [112], [117],
vehicular networks. Proper investigation of the time and
[119], [120], some on Tx power allocation [104], [108],
space complexity of distributed, cooperatively distributed,
and others on both channel and Tx power allocation [105],
and fully centralized approaches is critical for practical
[106], [109], [111], [113]–[116], [118]. The most commonly
implementation because ultra-reliability and low latency are
used objective function is to maximize the sum capacity of
essential in 5GB-enabled vehicular networks. Considering
V2I links with latency and reliability constraints on V2V
the decentralized nature of V2V, V2I, V2P, and V2N com-
links [111], [113], [115], [116], [118]–[120]. In the cognitive
munications, leveraging edge computing and investigating
Internet-of-vehicles network systems main objective remains
advanced distributed ML approaches can be effective for
to maximize the spectral efficiency while minimizing the
interference management in vehicular networks.
interference to primary cellular users [102], [103]. It is
ML models should provide efficient solutions for re-
achieved with efficient spectrum sensing and intelligent RB
source allocation within limited bandwidth resources while
allocation (eg. via SVM classifier [102] and Q-learning
considering heterogeneous data demands for V2V, V2I,
[103]). Some other commonly used objectives are to maxi-
V2P, V2N, and cellular communication links and provide
mize the sum rate of cellular users [108], packet delivery rate
consistent reliable performance. In the model considering
of V2V links [109], sum rate of all vehicles [117], network
ML or DL approaches, the collected data must contain
throughput of V2I links [110], and data rate of V2V links
sufficient diverse instances to generalize the model. Since
and success Tx probability of V2I links [114].
vehicle/traffic density can vary depending on time and place,
To solve the problem of resource allocation to V2V and
which may change the interference pattern, an ML model
V2I links, multi-agent RL can be utilized with a distributed
demonstrating competent performance in one environment
approach (eg. MA-DDQN [119]), a collaborative distributed
may not be suitable in another. Therefore, to implement the
approach (e.g., using Federated learning with DQN [113],
ML model in real-time scenarios, the model is expected to
[118], and D3QN [109]), or with a centralized training
adapt to such variations. Lastly, the ML models should be
and distributed execution approach [106], [107], [120]. It
able to integrate advanced techniques such as mmWAVE-
is possible to pursue a complete centralized approach for
VOLUME , 41
MIMO, NOMA, IRS, backscatter, and THZ communication and intra-tier interference in HCNs. In case of dynamic
with vehicular communication technologies. Furthermore, network reconfiguration such as the addition, modification,
ML models must be resilient to potential threats, ensuring the or removal of cells, ML models must be adaptable to
privacy and security of vehicular communication networks adjust these network parameters to minimize the impact of
while implementing interference management techniques. interference dynamically. Design of new ML models for
dynamic spectrum access mechanisms along with Tx power
D. Heterogeneous Cellular Networks (HCNs) control among different cells with users’ QoS constraints
is an open research direction, which can be tackled by the
1) Lessons Learned: researchers in this field. Further, ML-based techniques may
In HCNs, ML techniques have been proven to be quite be investigated for optimal location prediction of small cells
effective in mitigating inter-tier and intra-tier interference while considering user density, traffic, and interference pat-
and enhancing the overall network performance. The types of terns to enhance network throughput and spectral efficiency.
network architecture considered in this domain are HetNets To handle user mobility and inter-tier handover, ML
with the coexistence of macro cells and small cells [125]– models should be designed with the capability of minimizing
[130], [133]–[140], HetNets with dense small cells [131], interference during the transition by predicting user mobility
[132], [141]–[143], and multiple small-cell scenarios [144]– and optimizing handover decisions. A multi-connectivity
[146]. Following are some highlights for lessons learned approach can also be considered to handle user mobility and
from the state-of-art works in this domain: Commonly used fulfill their QoS requirements. To mitigate the interference
network metrics to evaluate the performance of various in HCNs, energy-efficient ML-based techniques with low
interference mitigation approaches in HCNs are network computational complexity are required. In centralized ap-
throughput [125], [127], [133], [134], [138]–[140], [143], proaches, feedback overhead, and high time and space com-
[144], packet loss ratio [125], [134], EE [126], [128], [129], plexity may be serious concerns. Therefore, advanced dis-
[142], [143], [145], spectral efficiency [135]–[137], and tributed approaches can be investigated for interference man-
QoS constraints of UEs [125], [128]–[132], [140], [141]. agement and performance enhancement of HCNs. Lastly, the
Tx power control and cell association using CRE bias are ML model should be flexible to integrate new technologies
widely used parameters to control the interference in HCNs. into HCNs and must be robust enough to maintain the
To obtain the optimal values of these parameters a decen- security and privacy of HCNs.
tralized MARL approach [130], [139]–[141], [144], [145],
distributed Q-learning [131], [133], cooperative Q-learning E. Device-to-Device (D2D) Communication Networks
[132], [133], or a centralized power control approach such
as DDPG [137] and DRL [143] can be used. 1) Lessons Learned:
Other than Tx power control, a few more techniques can We present lessons learned for AI/ML-based D2D interfer-
be considered to mitigate interference in HCNs such as traffic ence management in two parts: 1) lessons learned from the
offloading [125], [135], [136], ABS ratio [127], user clas- scenario with overlay deployment of D2D communication
sification/clustering [125], channel allocation [142], [146], with existing cellular networks, 2) lessons learned from
BS ON-OFF switching [142], Dynamic TDD configuration the scenario with underlay D2D communication with the
[134], network load prediction [128], and beamforming existing cellular networks.
[138]. To control the ABS ratio and CRE bias factor, the Scenarios with overlay deployment of D2D communica-
Multi Arm bandit [127] technique can be used, and for high tion with existing cellular networks: To control the co-
and low interference users’ classification, the Random Forest channel interference among D2D pairs optimal TX policy
algorithm [125] is a feasible approach to be used. If RL is [152], [154] with efficient link scheduling [151]/channel
empowered to address interference mitigation, DQN [135], allocation [153] and Tx power control [153], [155] are of vi-
[143], DDQN [136], [146], SAC [141], [145], Mutli Actors tal importance. Both centralized [150]–[152] and distributed
Shared Critic (MASC) [144] have been shown efficient under [153]–[155] approaches can be used for defining optimal TX
different settings. schemes. However, the main issue with centralized schemes
2) Challenges and Open Issues: is large feedback overhead. To reduce the feedback overhead,
HCNs consist of multiple tiers of diverse elements such partial and binary feedback can be adopted if a DRL-based
as macro, small, pico, and femtocells. In HCNs and ultra- TX policy is chosen [152]. DNN can be effectively used in
dense N/w, using ML techniques for inter-tier and intra-tier a centralized manner to map geographical location of D2D
interference mitigation introduces challenges and open is- pairs to optimal link scheduling parameters [151], and for
sues. Following are some key considerations in this domain: RF or VLC band selection to mitigate CCI [150].
Tx power control, cell association control using CRE RL-based schemes such as Duelling DQN is used for dis-
bias, ABS ratio, network load prediction, traffic offloading, tributed TX schemes [154], and distributed DRL are useful
BSs ON-OFF switching, and user classification/clustering instruments for joint channel and power allocation [153].
are some important factors/parameters to control inter-tier Also, RNN is beneficial for interference feature extraction
42 VOLUME ,
to aid the distributed TX scheme [155]. It should be noted spectrum allocation, and Tx power control. Interference miti-
that the distributed approach in this scenario underperforms gation in in-band overlay and underlay D2D communication
in comparison to a centralized approach [154]. On the other networks using AI/ML models presents several challenges
hand, decentralized schemes can still be preferred in the and open issues as highlighted below.
design if the priority is to mitigate the feedback overhead. ML models for interference detection and mitigation in
To evaluate the performance of the proposed approach, D2D communication networks should be adaptive to real-
the following metrics should be used: sum rate/throughput time variation in user density, traffic load, and network
[151]–[155], EE [150], and outage probability [150]. conditions by predicting/learning the interference pattern in
Scenarios with underlay deployment of D2D communi- changing environments and dynamically reconsidering the
cation with existing cellular networks: Under this scenario, interference mitigation strategies with low latency. While
with D2D communications, a crucial design constraint is considering the centralized approach, sufficient data should
that the QoS of cellular users should not degrade due to be collected to train the model for its generalized applica-
co-channel interference generated by D2D communication bility to diverse D2D communication environments. Further-
sharing the same spectrum [158]–[160], [165]. Other com- more, feedback overhead is critical in applying the central-
mon performance metrics to evaluate co-channel interference ized approach in D2D communication networks considering
mitigation techniques are sum rate [157], [158], [160], the energy-constrained end devices.
[161], [163], [165], [166], spectral efficiency [159], [162], For a decentralized approach, EE and computational com-
throughput [162], [164], [168], [169], fairness among D2D plexity are important parameters to measure its efficacy for
users [161], [166], EE [163], [164], and channel capacity energy-constrained devices. The state-of-the-art calls for ML
[167]. Co-channel interference mitigation techniques primar- models that consider QoS requirements, such as throughput
ily center on Tx power control and channel allocation. Some latency and outage probability, and that dynamically adapt
methods exclusively address Tx power control [158], [160], to meet contextual requirements. ML models are expected to
[162], while others solely focus on channel allocation [161], consider the coexistence of cellular networks and D2D com-
[165]. Additionally, it is also viable to take into account munication while handling directional interference, optimal
both Tx power control and channel allocation simultaneously spectrum usage, and the scalability concerns of continuously
[157], [159], [163], [164], [166]–[169]. Centralized AI/ML- increasing user density and demands. Lastly, the design
based approaches for power control include Centralized of robust ML models against privacy and security threats
DQN [159], and DNN-based centralized approach [158]. For by considering authentication and privacy-conserving tech-
spectrum access/channel allocation, a centralized approach niques along with interference mitigation approaches in self-
can employ DDQN [161]. Furthermore, for joint resource governing as well as cellular under-laid D2D communication
allocation (Tx power control and channel allocation), cen- N/w remains an open issue.
tralized GNN and CNN [157], and centralized DNN [159]
can be employed. F. Internet-of-Things (IoT) Communication Networks
In distributed approaches, to map the location and Tx
power, distributed DL can be utilized [162]. For channel 1) Lessons Learned:
allocation, the Actor-Critic-based MADRL technique is a In the IoT communication domain, a vast variety of
feasible candidate for centralized training and distributed network systems are considered along with corresponding
execution process [165]. For joint channel and power alloca- state-of-the-art interference mitigation solutions. Key points
tion design, a distributed DDQN [163] can be adopted. For regarding different types of network scenarios are as follows:
joint channel and power allocation, the MADRL technique is In a cognitive IoT network, the main aim is to mitigate
widely implemented with different algorithms such as DQN the primary user (PU) interference and minimize BER at
for Channel allocation and DDPG for Power allocation [166], the secondary user’s receiver [177]–[179]. In such a setup,
federated learning-aided DQN [164], and spiking neural to support the SU receiver, PU interference is clustered
network with co-operated DRL (actor-critic and PPO-based) using GMM for single PU [177], Affinity Propagation (AP)
algorithms [169]. It should be noted that a co-operated for multi-PU interference [178], and HMM-AP for asyn-
distributed approach will result in better performance when chronous cognitive-IoT case [179]. For an IoT network
compared to a fully distributed approach regarding through- utilizing D2D communications, resource allocation in terms
put, power consumption, and convergence rate [164], [169]. of Tx power and channel allocation is done either in a
centralized (eg. using GNN and CCN [184]) or distributed
2) Challenges and Open Issues: manner (eg. using MADRL-DDQN [191]). Specifically, for
In D2D communication networks, direct communication LoRa-enabled IoT communication networks, a DNN-based
between devices offloads traffic from the cellular BSs and algorithm can be used to assign suitable spreading factors to
enhances the network’s spectral efficiency, EE, and cov- IoT devices for uplink communication to mitigate the effect
erage capacity. Achieving all these benefits needs efficient of interference and enhance the packet success ratio [182].
interference management techniques via proper scheduling, Random channel access probability for Slotted-ALOHA
VOLUME , 43
protocol can be controlled with RL [187] or DRL [190] privacy of IoT communication networks. Lastly, ML models
techniques for the IoT devices to access the shared channel should consider integrating IoT communication with cutting-
for uplink communication. edge technologies such as mmWave-MIMO, IRS, and THz
Tx power allocation to IoT devices is also an important communication, implementing corresponding interference
factor in controlling the interference, which can be achieved mitigation strategies.
by RL or DRL [189] techniques both in centralized [189] and
distributed manner. The utilization of interference modeling G. mmWave-MIMO Communication Networks
[180] and interference prediction [175] can play a crucial role
in interference mitigation. This can be effectively accom- 1) Lessons Learned:
plished through the application of the DBSCAN approach In mmWave-MIMO-based communication networks, the
[180] for high interference detection accuracy and an LSTM- optimal design of beamforming vectors is critical to avoid
based approach [175] for minimizing prediction errors. To inter-beam/intra-cell and inter-cell interference and improve
mitigate the NB-IoT interference on LTE, sparse learning the network capacity performance. Various ML-based algo-
is effective for sparse cross-entropy minimization [181]. rithms can solve beamforming and beam selection problems
Distributed channel selection by each IoT device using the with low computation complexity. Following are the impor-
MAB-RL technique is also a possible solution to mitigate the tant lessons learned from ML-based state-of-the-art works in
interference and increase the communication success proba- the mmWave-MIMO-based communication domain: In the
bility [176]. Some other commonly used objective functions mmWave-MIMO communication network, the beam alloca-
to evaluate the network performance in IoT communication tion problem to multiple users can be treated as a multi-label
domain are throughput/sum rate [183]–[185], [189], [190], classification problem which can be solved by a CNN-based
spectral efficiency [181], [187], [191], and low latency [191]. algorithm with very low computational complexity [203].
Also, a DNN-based framework can facilitate RSS-based
2) Challenges and Open Issues: mapping for beam selection for multiple users to reduce the
Employing ML techniques for interference mitigation via initial access time [201]. In a mmWave-MIMO communica-
Tx power control, channel selection, random channel access tion network, for beamforming design [208], beam width and
control, and interference modeling/prediction in IoT commu- beam direction control [202], Tx power control [202], [208],
nication networks unveils various challenges and open issues and interference coordination, DNN [202] and DRL-based
owing to the unique characteristics of IoT environments. [208] algorithms can be effective for sum-rate maximization.
Following are some open issues and opportunities for the For a high-mobility user scenario, fast-beamforming and
researchers in this domain. IoT N/w have applications in var- multiple AP association can be solved by using a centralized
ious fields such as smart cities, smart health, smart transport, DRL-based approach and distributed federated-DRL-based
smart energy, and smart industry consisting of devices with approach to maintain the QoS of mobile users and enhance
diverse characteristics in terms of energy and computational the EE of the network [210].
capability constraints and different communication technolo- For the D2D-mmWave-MIMO communication network,
gies such as LoRa, WiFi, SigFox, ZigBee, NB-IoT, and 5G- a GNN-based algorithm can be competent at joint beam
NR and so on. Hence, for effective interference mitigation, selection and link activation to control the interference
ML models must consider the diverse characteristics of IoT between communication pairs and maximize the sum rate of
devices and communication technologies. the network [200]. In UAV-assisted MIMO-based communi-
Since IoT devices have limited energy and computation cation networks, the Gaussian process-based ML technique
resources, designing low-complexity ML algorithms and uti- can be used for channel estimation [198] and Q-learning can
lizing edge computing with distributed approaches for inter- be used for hybrid beamforming [205] and beam steering
ference mitigation and resource allocation need exploration. [206] to improve the sum rate of the network. For a cell-
It is worth noting that ML models must be capable of pro- free environment, DRL-based dynamic clustering of users
viding interference mitigation solutions in real time to fulfill and APs and hybrid beamforming design is proposed to
the latency constraints of time-critical IoT applications and mitigate the inter-UE and inter-cluster interference in a
practical implementation of these algorithms in real-world distributed manner [209]. To improve the BER as a signal
scenarios. Not to mention the adaptability of the designed detection performance metric at the receiver side of the
models to the diverse network topologies and varying traffic MIMO communication system, a DL-aided signal detector
loads with mobile devices in IoT communication networks for MIMO-OFDM is a suitable solution [211]. Also, the joint
while considering the interference mitigation strategies. use of an MLD and a deep CNN for the MIMO system
Scalability is a unique challenge in IoT networks where having correlated interfering signals [212], and the joint use
a large number of devices can be connected within one of a linear detector and a deep CNN for the MIMO system
network. Therefore, ML models designed for interference over correlating noise environment [213] have been proved
detection and mitigation must handle scalability challenges very effective in comparison to conventional detectors.
in IoT while being robust to preserve the security and 2) Challenges and Open Issues:
44 VOLUME ,
Utilizing ML techniques for efficient beamforming, beam and tackle THz band challenges such as molecular ab-
selection, channel estimation, and Tx power control for inter- sorption loss, frequency-selectivity, and distance-dependent
ference coordination has been proven effective in enhancing bandwidth change to enhance the network capacity perfor-
the sum rate performance of mmWave-MIMO communica- mance. Various ML-based algorithms are capable of solving
tion networks. However, the use of ML techniques in this these challenges of THz communication systems. Below are
domain poses several challenges and open issues, as well as the important lessons learned from ML-based state-of-the-art
opportunities, which are listed as follows. works in the THz communication domain:
First, mmWave-MIMO communication possesses complex A Federated DRL-based framework with a distributed
channel characteristics with high path loss and directional DDPG model at different BSs can be employed for THz
beams and sparse channel knowledge. Therefore, accurate beam selection having limited CSI to mitigate inter-cell
channel estimation, beamforming design, and beam alloca- interference and maximize the throughput in multiple BSs
tion to multiple users make the design of ML-based tech- network scenarios. This approach can be adopted instead
niques complex. A diverse and large amount of data collec- of the conventional ZF and MMSE-based beamforming
tion is required for a classification-based ML model to effec- techniques when improved computational complexity and
tively solve the beamforming and beam selection/allocation network throughput are aimed at. [237]. The beam selection
problem to mitigate the interference and enhance the net- problem in the THz communication system can be solved
work capacity. However, transfer learning and synthetic data as a multi-class classification problem. Without seeking
generation can be exploited to deal with data scarcity issues. sophisticated models, it is viable to go with Random Forest
The next challenge is to ensure the compatibility of the ML algorithm for a viable tradeoff between the compu-
developed ML models with many beams and beamforming tational complexity of hybrid beam forming and sum-rate
configurations while optimizing the interference mitigation maximization [224]. Combined use of RF, mmWave, and
strategies to deal with scalability challenges. Besides, ML THz spectrum bands can serve the data streaming request
models must be designed adaptively to varying network of users in a multi-drone environment. A specific spectrum
conditions such as channel conditions, traffic load, and band can be selected to serve a particular user based on
QoS requirements by dynamically optimizing the beam its channel condition. A combined echo-state learning and
management, users’ clustering, and interference mitigation RL-based framework would be an effective approach to
strategies. In a mmWave-MIMO network, high energy con- maximize the throughput via proper flight control of drones,
sumption is a serious issue. Hence, the development of ML user association, and spectrum band selection [232].
models to jointly address EE and interference mitigation Making the physical layer flexible in terms of selecting
strategies in mmWave-MIMO networks remains an open the Tx power [230], bandwidth [227], [230], and modulation
issue for researchers in this field. scheme [230] for Tx signals is an effective solution to tackle
To deal with correlated noise and interfering signals the communication challenges in THz-band such as molec-
during signal detection at the receiver side in a mmWave- ular absorption loss, frequency-selectivity, and distance-
MIMO communication system, some advanced ML-based dependent bandwidth change. A CNN-based bandwidth and
techniques should be explored to enhance the BER perfor- modulation classification technique can introduce such flex-
mance of the receiver. With this in mind, the state-of-the- ibility to the physical layer where Tx and Rx can exchange
art is seeking ML models that are capable of achieving their PHY parameters (bandwidth and modulation) without
efficient interference alignment in mmWave environments coordinating with each other [227]. Further, a DNN-based
through which interference from multiple transmitters is adaptive sub-band bandwidth allocation and power control
deliberately aligned in an orthogonal subspace to the desired can achieve flexible TX and enhanced sum rate of the THz
signal, which results in improved signal detection quality communication network [230]. In addition, Re-configurable
at the receiver. Last but not least, designing ML models Intelligent Surfaces (IRS) can be effective in tackling the
that meet the latency requirements of real-time applications propagation attenuation and atmospheric absorption losses
and are robust enough to preserve the security and privacy and enhancing the coverage range in THz-band [225], [231],
of mmWave-MIMO communication networks is another [235]. Specifically, the joint design of analog beamforming
research direction in this field. at RISs and digital beamforming at BS bridged with a
DRL-based approach can improve the THz communication
H. Tera-Hertz (THz) Communication Networks network’s total throughput and coverage range [235].
On the receiver side, learning-aided demodulation can
1) Lessons Learned: effectively handle the phase noise, phase shift, and I/Q
In THz communication networks, along with the adaptive imbalance-related challenges in the THz communication
and dynamic design of beamforming and beam selection, system. In this direction, a multi-output deep feed-forward
making the physical layer flexible in choosing PHY parame- NN for nonlinear mapping between the Rx symbols and the
ters such as bandwidth and modulation schemes is of critical demodulation likelihood information vector can be preferred
importance to avoid inter-symbol/inter-carrier interference instead of MMSE equalization or SVM-based schemes to
VOLUME , 45
aim for low BER performance for THz communication While ML models must be energy-efficient to ascertain the
systems under severe hybrid distortions [229]. The use of sustainability of THz communication networks in resource-
the THz communication band for task offloading is also constrained scenarios, it is also worth noting that the com-
encouraged [236], [238]. Specifically for UAV-assisted complexity of the ML model should be compatible with the
munication systems, to optimize UAV placement/trajectory, hardware constraints of the THz communication system
computation offloading, and resource allocation, DDQN along with being robust to maintain the security and privacy
[236], DDPG [236], and PPO-based [238] DRL algorithms of the THz communication networks.
can provide better system capacity [236] and network utility
[238] performance. Multi-arm bandit-based RL algorithms
I. NOMA-based Communication Networks
are also effective tools for beam training at the Tx side
1) Lessons Learned:
to enhance spectral efficiency [233] and for time-domain
interference mitigation at the Rx side to enhance BER In NOMA-based communication systems, the optimal design
performance [234] in THz communication systems. for the SIC process at the Rx side and efficient resource
2) Challenges and Open Issues:
allocation (channel and power allocation) at the Tx side
are pivotal to mitigate the intra-cell, inter-cell, and inter-
The utilization of ML techniques for adaptive hybrid beam-
user interference and improve the network performance. The
forming, beam selection, and physical layer flexibility at
important lessons learned and key takeaways from the state-
Tx and Rx have been proven to be effective in dealing
of-art contributions of ML-based techniques in these aspects
with THz band challenges and interference management in
of NOMA-based downlink and uplink communication sce-
THz communication networks. However, the development of
narios are expressed as follows:
ML-based algorithms for interference mitigation in the THz
NOMA-based downlink communication scenario: The de-
communication domain consists of several unique challenges
tection of modulation order of interfering signal can be
and open issues which are introduced as follows.
quite useful to support the SIC process at the Rx side
First of all, considering the beam selection problem as
which in turn enhances the network throughput of the
a multi-class classification problem needs sufficient diverse
NOMA-based communication system. A feature extraction
labeled training data for THz communication environments
method followed by a logistic regression-based classification
so that the designed ML model can adapt to the real-time
algorithm can achieve modulation order detection of inter-
variation in channel conditions due to distance selectivity
fering signals [245]. To fulfill the QoS demand of users,
and frequency selectivity of path loss. Accurate channel
mitigate the interference, and enhance the overall network
modeling, estimation, or prediction is quite complex in THz
performance, users association [250], users clustering [246],
communication scenarios due to high atmospheric absorp-
channel allocation [248], [250], and power allocation [248],
tion, diffraction, reflection, scattering, and shadowing effects.
[250] are critical factors.
Thus, designing ML models that overcome this challenge
For channel allocation, regression-based [250], Deep RNN
in THz communication networks remains an open issue.
[248], attention-based DRL [253] algorithms can be em-
The ML models must be dynamic in changing the physical
ployed to achieve high EE [248], [250] and sum-rate
layer parameters such as modulation scheme, Tx power, and
maximization [248], [253]. For power allocation to user
BW allocation to adapt to the time and distance-dependent
groups in NOMA-based communication, DNN [250], Deep
variations in BW and propagation characteristics in real-
RNN [248], [249], SAC-RL [252]-based algorithms can be
world applications with ultra-low latency.
adopted as their pros and cons have been discussed earlier.
Next, the ML model should efficiently handle the inter-
To tackle the imperfect SIC problem and to minimize the
carrier interference due to phase noise, beam squint, and
MSE between the desired and decoded signals of the users, a
spatial frequency wideband problems via adaptive frequency-
Feed-forward NN-based [251] technique is proposed for the
dependent beamforming and beam-combining at the trans-
SIC decoding of multiple users. In a multi-cell environment
mitter side. On the receiver side, employing time-domain
to enhance the sum rate performance, a cluster-free-NOMA
interference mitigation and learning-aided adaptive received
framework is a viable solution for inter-cell and intra-cell
signal mapping with demodulation schemes is crucial to
interference mitigation by utilizing a distributed and coordi-
deal with phase noise and beam-squint issues in the THz
nated auto-learning GNN-based approach for beamforming
communication domain. Sixth, the use of the THz band
and cluster-free SIC decoding in different cells [247].
in mobile environments such as autonomous vehicles is
NOMA-based uplink communication scenario: To improve
feasible with frequent handover and optimal beam steering
the BER performance with lower computational complexity,
and beamforming designs. The use of ML techniques in such
a DNN-based algorithm can be employed to detect the
scenarios may be promising to deal with these issues.
optimal sequence for SIC detection [255]. Usually, received
Integrating RIS with the THz communication system is
signal information and Tx power information are used for
crucial for increasing the coverage range, where the ML
training the proposed DNN. Also, a DNN can be used at each
model should be efficient in beamforming while taking care
SIC step to decode each user’s signal [256], [257]. To max-
of the unique channel characteristics of THz communication.
imize the sum rate of NOMA-based uplink communication,
46 VOLUME ,
efficient resource allocation, in terms of channel [259], [260] various technologies such as mmWave-MIMO, IoT, and
and power allocation [259] is required. ML-based techniques massive machine-type communication where BSs and UEs
such as MAB-RL [259], SARSA [260], and DRL [260] can have different demands and characteristics, it is crucial to
be exploited to obtain better performance than conventional develop ML models for resource allocation and interference
resource allocation approaches. management for NOMA-based communication in diverse
To maximize the number of successfully served users network architectures by considering specific characteristics
having different QoS requirements, an actor-critic learning- of different technologies. Lastly, ML algorithms to realize
based DRL algorithm can be exploited to optimize the NOMA techniques should be energy efficient and less com-
dynamic user access and optimal decoding order for the im- putationally complex for real-time practical implementation.
perfect SIC process [261]. For an uplink grant-free NOMA However, an ML model must be robust enough to maintain
system, to reduce the signaling overhead, latency, and colli- the privacy and security of NOMA-based networks.
sions of uncoordinated access and to improve the system
throughput, first, subchannels and users are divided into J. LLMs for Interference Detection and Mitigation
different clusters. Then, a DRL-based algorithm is used The evolution of wireless N/w towards 5G and 6G paradigms
for channel selection to clusters and power allocation to brings unprecedented advancements in speed, connectivity,
users [262]. Besides, for a grant-free NOMA system, LSTM- and data handling capabilities. However, these benefits also
based network [254] and model-driven DL [258] algorithms introduce complex interference challenges. Large Language
can be trained by using the pilot signals so that they Models (LLMs) like GPT-4 offer innovative solutions for
enable multi-user signal detection. Adopting these models detecting and mitigating interference in these advanced net-
for channel estimation and multi-user signal detection can works. LLMs can analyze vast amounts of diverse data in 5G
lead to improved BER performance in comparison to the and 6G environments, where network architectures are more
conventional SIC process [254], [258]. dynamic and heterogeneous, from enhanced Mobile Broad-
band (eMBB) to massive Machine Type Communications
(mMTC). By training on specialized datasets, including net-
2) Challenges and Open Issues: work traffic, sensor logs, and real-time user equipment data,
ML techniques are widely being investigated for user clus- LLMs can develop nuanced understandings of interference
tering, user scheduling, power allocation, and SIC decoding patterns unique to 5G/6G networks [263], [264].
sequence detection for interference mitigation and enhancing The application of LLMs in 5G/6G networks for interfer-
the sum capacity in NOMA-based N/w. However, there are ence detection and mitigation represents a forward-thinking
some open issues and challenges in the development of ML- approach to addressing the complexities of modern wireless
based algorithms for interference mitigation in the NOMA- communications. Their ability to process vast and varied
based communication domain which are presented below. datasets, adapt to evolving network scenarios, and provide
ML-based dynamic user clustering, user scheduling, chan- actionable insights makes them invaluable. However, lever-
nel allocation, and power allocation are required to adapt aging these benefits fully demands addressing challenges
to changing channel conditions with low latency for a suc- related to reliability, computational efficiency, and seamless
cessful implementation in real-time scenarios. Since correct integration with 5G/6G technologies. Achieving the ultra-
channel estimation is crucial for optimal power allocation reliability and low latency requirements of 5G/6G is critical,
and user scheduling in a NOMA-based communication sys- particularly for LLMs handling real-time interference detec-
tem, the ML models must be competent in estimating the tion and mitigation [265]. The scalability of LLMs in vast
channel condition with high accuracy in a dynamic wireless 5G/6G networks is a concern, requiring efficient algorithms
environment. Besides, there is a need to resolve the issue and distributed processing approaches. Therefore, integrating
of high communication overhead for channel estimation for LLMs with other 5G/6G technologies like Massive MIMO,
the practical deployment of the ML model. Along with the mmWave, and edge computing requires careful design to
objective of sum rate maximization, ML algorithms should ensure seamless operation. With the increased number of
focus on user fairness while allocating power resources to connected devices, ensuring data privacy while utilizing
different users. In NOMA systems, a user with good channel LLMs becomes more challenging and essential. The dynamic
conditions may cause high interference to users with poor nature of 5G/6G networks necessitates continuous learning
channel conditions while communicating on same spectrum and adaptation capabilities in LLMs to stay effective [266].
resources. Hence, optimal power allocation is vital in NOMA
to mitigate intra-tier and inter-user interference. XII. CONCLUSION
Given that SIC decoding is an essential part of the To achieve the goals of next-generation wireless networks
NOMA receiver system, diverse and sufficient data should such as global wireless coverage, sustainable energy-efficient
be collected to train ML models for detecting the se- networking, massive connectivity, ultra-high reliability, and
quence for SIC decoding and for being adopted in real- low latency, various enabling technologies exist and comple-
time implementations. As NOMA is being integrated into ment each other such as UAV-assisted networks, vehicular
VOLUME , 47
networks, HCNs, IoT, D2D, mmWave-MIMO, NOMA, and [9] M. Mohri, A. Rostamizadeh, and A. Talwalkar, Foundations of Ma-
THz communications. However, every enabling technology chine Learning. The MIT Press, 2012.
[10] M. van Otterlo and M. Wiering, Reinforcement Learning and Markov
has its own specific characteristics to be taken care of Decision Processes. Berlin, Heidelberg: Springer Berlin Heidelberg,
for efficient deployment of that technology. In that line, 2012, pp. 3–42. [Online]. Available: https://doi.org/10.1007/978-3-
interference management is a critical issue to tackle for every 642-27645-3 1
[11] S. Pouyanfar, S. Sadiq, Y. Yan, H. Tian, Y. Tao, M. P. Reyes, M.-L.
technology for proficient utilization of scarce bandwidth and Shyu, S.-C. Chen, and S. S. Iyengar, “A Survey on Deep Learning: Al-
power resources and to improve the overall network perfor- gorithms, Techniques, and Applications,” ACM Comput. Surv., vol. 51,
mance and QoE of users. Integration of AI/ML techniques in no. 5, sep 2018. [Online]. Available: https://doi.org/10.1145/3234150
[12] N. I. Miridakis and D. D. Vergados, “A Survey on the Succes-
5GB networks is inevitable and being widely investigated to sive Interference Cancellation Performance for Single-Antenna and
deal with interference management in complex and dynamic Multiple-Antenna OFDM Systems,” IEEE Communications Surveys
environments. This survey paper has provided a detailed & Tutorials, vol. 15, no. 1, pp. 312–335, 2013.
[13] C. Kosta, B. Hunt, A. U. Quddus, and R. Tafazolli, “On Interference
study of AI/ML-based works for intentional jamming and Avoidance Through Inter-Cell Interference Coordination (ICIC) Based
unintentional interference management focusing on the phys- on OFDMA Mobile Systems,” IEEE Communications Surveys &
ical layer perspective. For tackling intentional interference, Tutorials, vol. 15, no. 3, pp. 973–995, 2013.
[14] A. S. Hamza, S. S. Khalifa, H. S. Hamza, and K. Elsayed, “A Sur-
the key characteristics of different types of jamming attacks vey on Inter-Cell Interference Coordination Techniques in OFDMA-
have been discussed, and then a detailed review of AI- Based Cellular Networks,” IEEE Communications Surveys & Tutorials,
enabled techniques has been provided for jamming detection vol. 15, no. 4, pp. 1642–1670, 2013.
[15] H. O. Kpojime and G. A. Safdar, “Interference Mitigation in
and mitigation in various wireless communication scenarios. Cognitive-Radio-Based Femtocells,” IEEE Communications Surveys &
For intentional interference, every enabling technology of 5G Tutorials, vol. 17, no. 3, pp. 1511–1534, 2015.
networks towards 6G is considered individually discussing [16] M. Peng, C. Wang, J. Li, H. Xiang, and V. Lau, “Recent Advances
in Underlay Heterogeneous Networks: Interference Control, Resource
their unique features, characteristics, and interference effects Allocation, and Self-Organization,” IEEE Communications Surveys &
followed by an in-depth review of AI-enabled interference Tutorials, vol. 17, no. 2, pp. 700–729, 2015.
management works along with the qualitative comparison of [17] N. Zhao, F. R. Yu, M. Jin, Q. Yan, and V. C. M. Leung, “Interference
Alignment and Its Applications: A Survey, Research Issues, and
the different ML-based approaches in terms of performance Challenges,” IEEE Communications Surveys & Tutorials, vol. 18,
and complexity. Furthermore, the challenges associated with no. 3, pp. 1779–1803, 2016.
developing and deploying AI/ML-based interference mitiga- [18] T. O. Olwal, K. Djouani, and A. M. Kurien, “A Survey of Resource
Management Toward 5G Radio Access Networks,” IEEE Communi-
tion techniques in different enabling technologies towards cations Surveys & Tutorials, vol. 18, no. 3, pp. 1656–1686, 2016.
6G networks are presented exhibiting the potential future [19] S. M. R. Islam, N. Avazov, O. A. Dobre, and K.-s. Kwak, “Power-
directions of using AI/ML-enabled techniques to enhance Domain Non-Orthogonal Multiple Access (NOMA) in 5G Systems:
Potentials and Challenges,” IEEE Communications Surveys & Tutori-
the performance of future-generation wireless networks. als, vol. 19, no. 2, pp. 721–742, 2017.
[20] E. CastaÃ±eda, A. Silva, A. Gameiro, and M. Kountouris, “An
Overview on Resource Allocation Techniques for Multi-User MIMO
REFERENCES Systems,” IEEE Communications Surveys & Tutorials, vol. 19, no. 1,
[1] X. Lin, J. Li, R. Baldemair, J.-F. T. Cheng, S. Parkvall, D. C. Larsson, pp. 239–284, 2017.
H. Koorapaty, M. Frenne, S. Falahati, A. Grovlen, and K. Werner, “5G [21] F. Jameel, Z. Hamid, F. Jabeen, S. Zeadally, and M. A. Javed, “A
New Radio: Unveiling the Essentials of the Next Generation Wireless Survey of Device-to-Device Communications: Research Issues and
Access Technology,” IEEE Communications Standards Magazine, Challenges,” IEEE Communications Surveys & Tutorials, vol. 20,
vol. 3, no. 3, pp. 30–37, 2019. no. 3, pp. 2133–2168, 2018.
[2] Ericsson, “5G wireless access: an overview,” White Paper, April [22] J. Yao, J. Xu, S. Luo, L. Wang, C. Yang, K. Wu, and W. Lou,
2020. [Online]. Available: https://www.ericsson.com/en/reports-and- “Comprehensive Study on MIMO-Related Interference Management in
papers/white-papers/5g-wireless-access-an-overview WLANs,” IEEE Communications Surveys & Tutorials, vol. 21, no. 3,
pp. 2087–2110, 2019.
[3] A. Ghosh, A. Maeder, M. Baker, and D. Chandramouli, “5G Evolution:
[23] P. S. Bithas, E. T. Michailidis, N. Nomikos, D. Vouyioukas, and A. G.
A View on 5G Cellular Technology Beyond 3GPP Release 15,” IEEE
Kanatas, “A Survey on Machine-Learning Techniques for UAV-Based
Access, vol. 7, pp. 127 639–127 651, 2019.
Communications,” Sensors, vol. 19, no. 23, p. 5170, 2019.
[4] Ericsson, “5G Advanced: Evolution towards 6G,” White Paper, June [24] R. Morales-Ferre, P. Richter, E. Falletti, A. de la Fuente, and E. S.
2023. [Online]. Available: https://www.ericsson.com/en/reports-and- Lohan, “A Survey on Coping With Intentional Interference in Satellite
papers/white-papers/5g-advanced-evolution-towards-6g Navigation for Manned and Unmanned Aircraft,” IEEE Communica-
[5] X. Lin, “An Overview of 5G Advanced Evolution in 3GPP Release tions Surveys & Tutorials, vol. 22, no. 1, pp. 249–291, 2020.
18,” IEEE Communications Standards Magazine, vol. 6, no. 3, pp. [25] M. U. A. Siddiqui, F. Qamar, F. Ahmed, Q. N. Nguyen, and R. Hassan,
77–83, 2022. “Interference Management in 5G and Beyond Network: Requirements,
[6] Qualcomm, “Setting off the 5G Advanced evolution,” Challenges and Future Directions,” IEEE Access, vol. 9, pp. 68 932–
Technical Report, January 2022. [Online]. Available: 68 965, 2021.
https://www.qualcomm.com/content/dam/qcomm-martech/dm- [26] S. Hashima, B. M. ElHalawany, K. Hatano, K. Wu, and E. M.
assets/documents/setting-off-the-5g-advanced-evolution-with-3gpp- Mohamed, “Leveraging Machine-Learning for D2D Communications
release-18.pdf in 5G/Beyond 5G Networks,” Electronics, vol. 10, no. 2, p. 169, 2021.
[7] ——, “Vision, market drivers, and research directions on the [27] Y. Xu, G. Gui, H. Gacanin, and F. Adachi, “A Survey on Resource
path to 6G,” White Paper, December 2022. [Online]. Available: Allocation for 5G Heterogeneous Networks: Current Research, Future
https://www.qualcomm.com/content/dam/qcomm-martech/dm- Trends, and Challenges,” IEEE Communications Surveys & Tutorials,
assets/documents/Qualcomm-Whitepaper-Vision-market-drivers- vol. 23, no. 2, pp. 668–695, 2021.
and-research-directions-on-the-path-to-6G.pdf [28] N. M. Elfatih, M. K. Hasan, Z. Kamal, D. Gupta, R. A. Saeed, E. S.
[8] W. Attaoui, E. Sabir, H. Elbiaze, and M. Guizani, “VNF and CNF Ali, and H. M. S., “Internet of Vehicleâ C™s resource management in
Placement in 5G: Recent Advances and Future Trends,” IEEE Trans. 5G networks using AI technologies: Current status and trends,” IET
on Network and Service Management, pp. 1–1, 2023. Communication, vol. 16, no. 5, pp. 400–420, 2022.
48 VOLUME ,
[29] O. Alzubaidi, M. Hindia, K. Dimyati, K. Noordin, A. Wahab, F. Qa- ing Under Impaired Channels With Interference and Noise,” IEEE
mar, and R. Hassan, “Interference Challenges and Management in Access, vol. 11, pp. 26 289–26 307, 2023.
B5G Network Design: A Comprehensive Review,” Electronics, vol. 11, [50] G. Reus-Muns, D. Jaisinghani, K. Sankhe, and K. Chowdhury, “Trust
no. 18, p. 2842, 2022. in 5G Open RANs through Machine Learning: RF Fingerprinting on
[30] Q. Mao, F. Hu, and Q. Hao, “Deep Learning for Intelligent Wireless the POWDER PAWR Platform,” in IEEE Globecom 2020-IEEE Global
Networks: A Comprehensive Survey,” IEEE Communications Surveys Communications Conference. IEEE, 2020.
& Tutorials, vol. 20, no. 4, pp. 2595–2621, 2018. [51] A. Toma, A. Krayani, M. Farrukh, H. Qi, L. Marcenaro, Y. Gao, and
[31] M. Chen, U. Challita, W. Saad, C. Yin, and M. Debbah, “Artificial C. S. Regazzoni, “AI-Based Abnormality Detection at the PHY-Layer
Neural Networks-Based Machine Learning for Wireless Networks: A of Cognitive Radio by Learning Generative Models,” IEEE Trans. on
Tutorial,” IEEE Communications Surveys & Tutorials, vol. 21, no. 4, Cognitive Communications and Networking, vol. 6/1, pp. 21–34, 2020.
pp. 3039–3071, 2019. [52] C. Zhong, M. C. Gursoy, and S. Velipasalar, “Learning-Based Robust
[32] N. C. Luong, X. Lu, D. T. Hoang, D. Niyato, and D. I. Kim, Anomaly Detection in the Presence of Adversarial Attacks,” in IEEE
“Radio Resource Management in Joint Radar and Communication: A Wireless Communications and Networking Conf. (WCNC), 2022, pp.
Comprehensive Survey,” IEEE Communications Surveys & Tutorials, 1206–1211.
vol. 23, no. 2, pp. 780–814, 2021. [53] S. Dinh-Van, T. M. Hoang, B. B. Cebecioglu, D. S. Fowler, Y. K. Mo,
[33] T. Oyedare, V. K. Shah, D. J. Jakubisin, and J. H. Reed, “Interference and M. D. Higgins, “A Defensive Strategy Against Beam Training
Suppression Using Deep Learning: Current Approaches and Open Attack in 5G mmWave Networks for Manufacturing,” IEEE Trans. on
Challenges,” IEEE Access, vol. 10, pp. 66 238–66 266, 2022. Information Forensics and Security, vol. 18, pp. 2204–2217, 2023.
[34] A. Mpitziopoulos, D. Gavalas, C. Konstantopoulos, and G. Pantziou, [54] N. Van Huynh, D. N. Nguyen, D. T. Hoang, T. X. Vu, E. Dutkiewicz,
“A survey on jamming attacks and countermeasures in WSNs,” IEEE and S. Chatzinotas, “Defeating Super-Reactive Jammers With Decep-
Communications Surveys & Tutorials, vol. 11, no. 4, pp. 42–56, 2009. tion Strategy: Modeling, Signal Detection, and Performance Analysis,”
[35] X. Lu, L. Xiao, P. Li, X. Ji, C. Xu, S. Yu, and W. Zhuang, IEEE Transactions on Wireless Communications, vol. 21, no. 9, pp.
“Reinforcement Learning-Based Physical Cross-Layer Security and 7374–7390, 2022.
Privacy in 6G,” IEEE Communications Surveys & Tutorials, vol. 25, [55] F. Yao and L. Jia, “A Collaborative Multi-Agent Reinforcement Learn-
no. 1, pp. 425–466, 2023. ing Anti-Jamming Algorithm in Wireless Networks,” IEEE Wireless
[36] W. Xu, K. Ma, W. Trappe, and Y. Zhang, “Jamming sensor networks: Communications Letters, vol. 8, no. 4, pp. 1024–1027, 2019.
attack and defense strategies,” IEEE Network, vol. 20, no. 3, pp. 41– [56] N. I. Mowla, N. H. Tran, I. Doh, and K. Chae, “AFRL: Adaptive
47, 2006. federated reinforcement learning for intelligent jamming defense in
[37] T. Erpek, Y. E. Sagduyu, and Y. Shi, “Deep learning for launching FANET,” Journal of Communications and Networks, vol. 22, no. 3,
and mitigating wireless jamming attacks,” IEEE Trans. on Cognitive pp. 244–258, 2020.
Communications and Networking, vol. 5, no. 1, pp. 2–14, 2019. [57] A. Krayani, A. S. Alam, L. Marcenaro, A. Nallanathan, and C. Regaz-
[38] K. Pelechrinis, M. Iliofotou, and S. V. Krishnamurthy, “Denial of zoni, “A Novel Resource Allocation for Anti-Jamming in Cognitive-
Service Attacks in Wireless Networks: The Case of Jammers,” IEEE UAVs: An Active Inference Approach,” IEEE Communications Letters,
Communications Surveys & Tutorials, vol. 13, no. 2, pp. 245–257, vol. 26, no. 10, pp. 2272–2276, 2022.
2011. [58] L. Xiao, X. Lu, D. Xu, Y. Tang, L. Wang, and W. Zhuang, “UAV Relay
[39] S. Bandaru, “Investigating the Effect of Jamming Attacks on Wireless in VANETs Against Smart Jamming With Reinforcement Learning,”
LANS,” International Journal of Computer Applications, vol. 99, pp. IEEE Trans. on Vehicular Technology, vol. 67, no. 5, pp. 4087–4097,
5–9, 08 2014. 2018.
[40] T. Hamza, G. Kaddoum, A. Meddeb, and G. Matar, “A Survey on [59] X. Lu, L. Xiao, C. Dai, and H. Dai, “UAV-Aided Cellular Communi-
Intelligent MAC Layer Jamming Attacks and Countermeasures in cations with Deep Reinforcement Learning Against Jamming,” IEEE
WSNs,” in 2016 IEEE 84th Vehicular Technology Conference (VTC- Wireless Communications, vol. 27, no. 4, pp. 48–53, 2020.
Fall), 2016, pp. 1–5. [60] L. Xiao, C. Xie, M. Min, and W. Zhuang, “User-Centric View of
[41] N. Venkata Abhishek and M. Gurusamy, “JaDe: Low Power Jamming Unmanned Aerial Vehicle TX Against Smart Attacks,” IEEE Trans.
Detection Using Machine Learning in Vehicular Networks,” IEEE on Vehicular Technology, vol. 67, no. 4, pp. 3420–3430, 2018.
Wireless Communications Letters, vol. 10, no. 10, pp. 2210–2214, [61] Z. Lv, L. Xiao, Y. Du, G. Niu, C. Xing, and W. Xu, “Multi-Agent
2021. Reinforcement Learning based UAV Swarm Communications Against
[42] S. Hong, K. Kim, and S.-H. Lee, “A Hybrid Jamming Detection Jamming,” IEEE Transactions on Wireless Communications, pp. 1–1,
Algorithm for Wireless Communications: Simultaneous Classification 2023.
of Known Attacks and Detection of Unknown Attacks,” IEEE Com- [62] W. Wang, Z. Lv, X. Lu, Y. Zhang, and L. Xiao, “Distributed rein-
munications Letters, pp. 1–1, 2023. forcement learning based framework for energy-efficient UAV relay
[43] Z. Feng and C. Hua, “Machine Learning-based RF Jamming Detection against jamming,” Intelligent and Converged Networks, vol. 2, no. 2,
in Wireless Networks,” in 2018 Third International Conference on Se- pp. 150–162, 2021.
curity of Smart Cities, Industrial Control System and Communications [63] Y. Chen, Y. Li, D. Xu, and L. Xiao, “DQN-Based Power Control for
(SSIC), 2018, pp.1–6. IoT TX against Jamming,” in 2018 IEEE 87th Vehicular Technology
[44] Y. Arjoune, F. Salahdine, M. S. Islam, E. Ghribi, and N. Kaabouch, Conference (VTC Spring), 2018, pp. 1–5.
“A Novel Jamming Attacks Detection Approach Based on Machine [64] X. Zhu, Y. Huang, S. Wang, Q. Wu, X. Ge, Y. Liu, and Z. Gao, “Dy-
Learning for Wireless Communication,” in 2020 International Confer- namic Spectrum Anti-Jamming With Reinforcement Learning Based
ence on Information Networking (ICOIN), 2020, pp. 459–464. on Value Function Approximation,” IEEE Wireless Communications
[45] B. Upadhyaya, S. Sun, and B. Sikdar, “Machine Learning-based Letters, vol. 12, no. 2, pp. 386–390, 2023.
Jamming Detection in Wireless IoT Networks,” in IEEE VTS Asia [65] X. Wang, Y. Xu, J. Chen, C. Li, X. Liu, D. Liu, and Y. Xu, “Mean
Pacific Wireless Communications Symp., 2019, pp. 1–5. Field Reinforcement Learning Based Anti-Jamming Communications
[46] Y. Li, J. Pawlak, J. Price, K. Al Shamaileh, Q. Niyaz, S. Paheding, and for Ultra-Dense Internet of Things in 6G,” in 2020 International Con-
V. Devabhaktuni, “Jamming Detection and Classification in OFDM- ference on Wireless Communications and Signal Processing (WCSP),
Based UAVs via Feature- and Spectrogram-Tailored Machine Learn- 2020, pp. 195–200.
ing,” IEEE Access, vol. 10, pp. 16 859–16 870, 2022. [66] Z. Xiao, B. Gao, S. Liu, and L. Xiao, “Learning Based Power Control
[47] N. I. Mowla, N. H. Tran, I. Doh, and K. Chae, “Federated Learning- for mmWave Massive MIMO against Jamming,” in 2018 IEEE Global
Based Cognitive Detection of Jamming Attack in Flying Ad-Hoc Communications Conference (GLOBECOM), 2018, pp. 1–6.
Network,” IEEE Access, vol. 8, pp. 4338–4350, 2020. [67] L. Xiao, Y. Li, C. Dai, H. Dai, and H. V. Poor, “Reinforcement
[48] O. PuÃ±al, C. Pereira, A. Aguiar, and J. Gross, “CRAW- Learning-Based NOMA Power Allocation in the Presence of Smart
DAD uportorwthaachen/vanetjamming2014,” 2022. [Online]. Avail- Jamming,” IEEE Transactions on Vehicular Technology, vol. 67, no. 4,
able: https://dx.doi.org/10.15783/C7Q306 pp. 3377–3389, 2018.
[49] O. M. Gul, M. Kulhandjian, B. Kantarci, A. Touazi, C. Ellement, and [68] H. Sharma, N. Kumar, and R. Tekchandani, “Mitigating Jamming At-
C. Dâ C™amours, “Secure Industrial IoT Systems via RF Fingerprint- tack in 5G Heterogeneous Networks: A Federated Deep Reinforcement
VOLUME , 49
Learning Approach,” IEEE Trans. on Vehicular Technology, vol. 72, Journal of Communications and Information Networks, vol. 7, no. 2,
no. 2, pp. 2439–2452, 2023. pp. 192–201, 2022.
[69] Y. Yao, J. Zhao, Z. Li, X. Cheng, and L. Wu, “Jamming and [87] L. Li, Q. Cheng, K. Xue, C. Yang, and Z. Han, “Downlink Transmit
Eavesdropping Defense Scheme Based on Deep Reinforcement Learn- Power Control in Ultra-Dense UAV Network Based on Mean Field
ing in Autonomous Vehicle Networks,” IEEE Trans. on Information Game and Deep Reinforcement Learning,” IEEE Trans. on Vehicular
Forensics and Security, vol. 18, pp. 1211–1224, 2023. Technology, vol. 69, no. 12, pp. 15 594–15 605, 2020.
[70] H. Sharma, N. Kumar, and R. K. Tekchandani, “SecBoost: Secrecy- [88] J. Ji, L. Cai, K. Zhu, and D. Niyato, “Decoupled Association with Rate
Aware Deep Reinforcement Learning Based Energy-Efficient Scheme Splitting Multiple Access in UAV-assisted Cellular Networks Using
for 5G HetNets,” IEEE Trans. on Mobile Computing, pp. 1–15, 2023. Multi-agent Deep Reinforcement Learning,” IEEE Trans. on Mobile
[71] H. Sharma, I. Budhiraja, N. Kumar, and R. K. Tekchandani, “Secrecy Computing, pp. 1–15, 2023.
Rate Maximization for THz-Enabled Femto Edge Users using Deep [89] G. Chen, X. B. Zhai, and C. Li, “Joint Optimization of Trajectory
Reinforcement Learning in 6G,” in IEEE INFOCOM 2022 - IEEE and User Association via Reinforcement Learning for UAV-Aided
Conference on Computer Communications Workshops (INFOCOM Data Collection in Wireless Networks,” IEEE Trans. on Wireless
WKSHPS), 2022, pp. 1–6. Communications, vol. 22, no. 5, pp. 3128–3143, 2023.
[72] F. Jameel, M. A. Javed, S. Zeadally, and R. JÃ¤ntti, “Secure TX [90] B. Omoniwa, B. Galkin, and I. Dusparic, “Optimizing Energy Effi-
in Cellular V2X Communications Using Deep Q-Learning,” IEEE ciency in UAV-Assisted Networks Using Deep Reinforcement Learn-
Transactions on Intelligent Transportation Systems, vol. 23, no. 10, ing,” IEEE Wireless Communications Letters, vol. 11, no. 8, pp. 1590–
pp. 17 167–17 176, 2022. 1594, 2022.
[73] W. Gao, C. Han, and Z. Chen, “DNN-Powered SIC-Free Receiver Arti- [91] G. Wu, “UAV-Based Interference Source Localization: A Multimodal
ficial Noise Aided Terahertz Secure Communications With Randomly Q-Learning Approach,” IEEE Access, vol. 7, pp. 137 982–137 991,
Distributed Eavesdroppers,” IEEE Trans. on Wireless Communications, 2019.
vol. 21, no. 1, pp. 563–576, 2022. [92] Y. Shi, M. Q. Hamdan, E. Alsusa, K. A. Hamdi, and M. W. Baidas,
[74] D. Han, A. Li, L. Zhang, Y. Zhang, J. Li, T. Li, T. Zhu, and “A Decoupled Access Scheme With Reinforcement Learning Power
Y. Zhang, “Deep Learning-Guided Jamming for Cross-Technology Control for Cellular-Enabled UAVs,” IEEE Internet of Things Journal,
Wireless Networks: Attack and Defense,” IEEE/ACM Transactions on vol. 8, no. 24, pp. 17 261–17 274, 2021.
Networking, vol. 29, no. 5, pp. 1922–1932, 2021. [93] A. Warrier, S. Al-Rubaye, D. Panagiotakopoulos, G. Inalhan, and
[75] M. Liu, H. Zhang, Z. Liu, and N. Zhao, “Attacking Spectrum Sensing A. Tsourdos, “Interference Mitigation for 5G-Connected UAV using
With Adversarial Deep Learning in Cognitive Radio-Enabled Internet Deep Q-Learning Framework,” in IEEE/AIAA 41st Digital Avionics
of Things,” IEEE Trans. on Reliability, vol. 72, no. 2, pp. 431–444, Systems Conf., 2022, pp. 1–8.
2023. [94] Y.-J. Chen and D.-Y. Huang, “Joint Trajectory Design and BS Asso-
[76] W. Jiang, H. Li, S. Liu, X. Luo, and R. Lu, “Poisoning and Evasion ciation for Cellular-Connected UAV: An Imitation-Augmented Deep
Attacks Against Deep Learning Algorithms in Autonomous Vehicles,” Reinforcement Learning Approach,” IEEE Internet of Things Journal,
IEEE Transactions on Vehicular Technology, vol. 69, no. 4, pp. 4439– vol. 9, no. 4, pp. 2843–2858, 2022.
4449, 2020. [95] U. Challita, W. Saad, and C. Bettstetter, “Interference Management
[77] P. Lohan and D. Mishra, “Utility-Aware Optimal Resource Allocation for Cellular-Connected UAVs: A Deep Reinforcement Learning Ap-
Protocol for UAV-Assisted Small Cells With Heterogeneous Coverage proach,” IEEE Trans. on Wireless Communications, vol. 18, no. 4, pp.
Demands,” IEEE Trans. on Wireless Communications, vol. 19, no. 2, 2125–2140, 2019.
pp. 1221–1236, 2020. [96] B. Galkin, E. Fonseca, R. Amer, L. A. DaSilva, and I. Dusparic,
[78] V. Yajnanarayana, Y.-P. Eric Wang, S. Gao, S. Muruganathan, and “REQIBA: Regression and Deep Q-Learning for Intelligent UAV
X. Lin Ericsson, “Interference Mitigation Methods for Unmanned Cellular User to Base Station Association,” IEEE Trans. on Vehicular
Aerial Vehicles Served by Cellular Networks,” in 2018 IEEE 5G World Technology, vol. 71, no. 1, pp. 5–20, 2022.
Forum (5GWF), 2018, pp. 118–122. [97] Y. Li and A. Hamid Aghvami, “Radio Resource Management for
[79] S. Chandrasekharan, K. Gomez, A. Al-Hourani, S. Kandeepan, Cellular-Connected UAV: A Learning Approach,” IEEE Trans. on
T. Rasheed, L. Goratti, L. Reynaud, D. Grace, I. Bucaille, T. Wirth, and Communications, pp. 1–1, 2023.
S. Allsopp, “Designing and implementing future aerial communication [98] S. Lee, S. Lim, S. H. Chae, B. C. Jung, C. Y. Park, and H. Lee,
networks,” IEEE Communications Magazine, vol. 54, no. 5, pp. 26–34, “Optimal Frequency Reuse and Power Control in Multi-UAV Wireless
2016. Networks: Hierarchical Multi-Agent Reinforcement Learning Perspec-
[80] L.-C. Wang, Y.-S. Chao, S.-H. Cheng, and Z. Han, “An Integrated tive,” IEEE Access, vol. 10, pp. 39 555–39 565, 2022.
Affinity Propagation and Machine Learning Approach for Interference [99] M. Z. Hassan, G. Kaddoum, and O. Akhrif, “Resource Allocation for
Management in Drone Base Stations,” IEEE Trans. on Cognitive Joint Interference Management and Security Enhancement in Cellular-
Communications and Networking, vol. 6, no. 1, pp. 83–94, 2020. Connected Internet-of-Drones Networks,” IEEE Trans. on Vehicular
[81] S.-H. Cheng, Y.-T. Shih, and K.-C. Chang, “Proactive Power Control Technology, vol. 71, no. 12, pp. 12 869–12 884, 2022.
and Position Deployment for Drone Small Cells: Joint Supervised and [100] J. B. Kenney, “Dedicated Short-Range Communications (DSRC)
Unsupervised Learning,” IEEE Access, vol. 9, pp. 126 735–126 747, Standards in the United States,” Proceedings of the IEEE, vol. 99,
2021. no. 7, pp. 1162–1182, 2011.
[82] M.-H. T. Nguyen, E. Garcia-Palacios, T. Do-Duy, L. D. Nguyen, S. T. [101] R. Molina-Masegosa and J. Gozalvez, “LTE-V for Sidelink 5G V2X
Mai, and T. Q. Duong, “Spectrum-Sharing UAV-Assisted Mission- Vehicular Communications: A New 5G Technology for Short-Range
Critical Communication: Learning-Aided Real-Time Optimisation,” Vehicle-to-Everything Communications,” IEEE Vehicular Technology
IEEE Access, vol. 9, pp. 11 622–11 632, 2021. Magazine, vol. 12, no. 4, pp. 30–39, 2017.
[83] S.-H. Cheng, J.-L. Liu, and L.-C. Wang, “Controlling Interference [102] R. Ahmed, Y. Chen, B. Hassan, L. Du, T. Hassan, and J. Dias, “Hy-
Structure and Transmit Power of Aerial Small Cells by Hybrid Affinity brid Machine-Learning-Based Spectrum Sensing and Allocation With
Propagation Clustering and Reinforcement Learning,” IEEE Open Adaptive Congestion-Aware Modeling in CR-Assisted IoV Networks,”
Journal of Vehicular Technology, vol. 2, pp. 412–418, 2021. IEEE Internet of Things J., vol. 9/24, pp. 25 100–25 116, 2022.
[84] Z. Kaleem, W. Khalid, A. Muqaibel, A. A. Nasir, C. Yuen, and [103] X. Liu, C. Sun, K.-L. A. Yau, and C. Wu, “Joint Collaborative Big
G. K. Karagiannidis, “Learning-Aided UAV 3D Placement and Power Spectrum Data Sensing and Reinforcement Learning Based Dynamic
Allocation for Sum-Capacity Enhancement Under Varying Altitudes,” Spectrum Access for Cognitive Internet of Vehicles,” IEEE Trans. on
IEEE Communications Letters, vol. 26, no. 7, pp. 1633–1637, 2022. Intelligent Transportation Systems, pp. 1–11, 2022.
[85] Z. Dai, Y. Zhang, W. Zhang, X. Luo, and Z. He, “A Multi-Agent [104] H. Zhu, Y. Zhou, X. Luo, and H. Zhou, “Joint Control of Power,
Collaborative Environment Learning Method for UAV Deployment Beamwidth, and Spacing for Platoon-Based Vehicular Cyber-Physical
and Resource Allocation,” IEEE Trans. on Signal and Information Systems,” IEEE Transactions on Vehicular Technology, vol. 71, no. 8,
Processing over Networks, vol. 8, pp. 120–130, 2022. pp. 8615–8629, 2022.
[86] C. Zhang, S. Liang, C. He, and K. Wang, “Multi-UAV Trajectory [105] M. K. Abdel-Aziz, S. Samarakoon, M. Bennis, and W. Saad, “Ultra-
Design and Power Control Based on Deep Reinforcement Learning,” Reliable and Low-Latency Vehicular Communication: An Active
50 VOLUME ,
Learning Approach,” IEEE Communications Letters, vol. 24, no. 2, [125] D. Anand, M. A. Togou, and G.-M. Muntean, “A Machine Learning
pp. 367–370, 2020. Solution for Video Delivery to Mitigate Co-Tier Interference in 5G
[106] O. Urmonov, H. Aliev, and H. Kim, “Multi-Agent Deep Reinforce- HetNets,” IEEE Transactions on Multimedia, pp. 1–13, 2022.
ment Learning for Enhancement of Distributed Resource Allocation in [126] J. A. Ayala-Romero, J. J. Alcaraz, A. Zanella, and M. Zorzi,
Vehicular Network,” IEEE Systems Journal, vol. 17, no. 1, pp. 491– “Contextual Bandit Approach for Energy Saving and Interference
502, 2023. Coordination in HetNets,” in 2018 IEEE International Conference on
[107] B. Gu, W. Chen, M. Alazab, X. Tan, and M. Guizani, “Multiagent Communications (ICC), 2018, pp. 1–6.
Reinforcement Learning-Based Semi-Persistent Scheduling Scheme in [127] J. A. Ayala-Romero, J. J. Alcaraz, and J. Vales-Alonso, “Data-Driven
C-V2X Mode 4,” IEEE Trans. on Vehicular Technology, vol. 71/11, Configuration of Interference Coordination Parameters in HetNets,”
pp. 12 044–12 056, 2022. IEEE Trans. on Vehicular Technology, vol. 67, no. 6, pp. 5174–5187,
[108] D. Ron and J.-R. Lee, “DNN-Based Dynamic Transmit Power 2018.
Control for V2V Communication Underlaid Cellular Uplink,” IEEE [128] J. A. Ayala-Romero, J. J. Alcaraz, and J. Vales-Alonso, “Energy
Trans. on Vehicular Technology, vol. 71/11, pp. 12 413–12 418, 2022. Saving and Interference Coordination in HetNets Using Dynamic
[109] X. Li, L. Lu, W. Ni, A. Jamalipour, D. Zhang, and H. Du, “Federated Programming and CEC,” IEEE Access, vol. 6, pp. 71 110–71 121,
Multi-Agent Deep Reinforcement Learning for Resource Allocation 2018.
of Vehicle-to-Vehicle Communications,” IEEE Trans. on Vehicular [129] J. A. Ayala-Romero, J. J. Alcaraz, A. Zanella, and M. Zorzi,
Technology, vol. 71, no. 8, pp. 8810–8824, 2022. “Online Learning for Energy Saving and Interference Coordination
in HetNets,” IEEE J. on Sel. Areas in Communications, vol. 37, no. 6,
[110] I. Nasim, A. S. Ibrahim, and S. Kim, “Learning-Based Beamforming
pp. 1374–1388, 2019.
for Multi-User Vehicular Communications: A Combinatorial Multi-
[130] R. Amiri, M. A. Almasi, J. G. Andrews, and H. Mehrpouyan,
Armed Bandit Approach,” IEEE Access, vol. 8, pp. 219 891–219 902,
“Reinforcement Learning for Self Organizationn and Power Control
2020.
of Two-Tier Heterogeneous Networks,” IEEE Trans. on Wireless
[111] Y.-H. Xu, J.-H. Li, W. Zhou, and C. Chen, “Learning-Empowered Communications, vol. 18, no. 8, pp. 3933–3947, 2019.
Resource Allocation for Air Slicing in UAV-Assisted Cellular V2X [131] M. U. Iqbal, E. A. Ansari, and S. Akhtar, “Interference Mitigation in
Communications,” IEEE Systems Journal, vol. 17, no. 1, pp. 1008– HetNets to Improve the QoS Using Q-Learning,” IEEE Access, vol. 9,
1011, 2023. pp. 32 405–32 424, 2021.
[112] B. Zhao, J. Liu, B. Mao, and S. Li, “Optimal Resource Allocation [132] M. U. Iqbal, E. A. Ansari, S. Akhtar, and A. N. Khan, “Improving the
for Random Multiple Access Oriented SCMA-V2X Networks,” IEEE QoS in 5G HetNets Through Cooperative Q-Learning,” IEEE Access,
Trans. on Vehicular Technology, pp. 1–12, 2023. vol. 10, pp. 19 654–19 676, 2022.
[113] G. Chai, W. Wu, Q. Yang, R. Liu, and F. R. Yu, “Learning-Based [133] M. U. Iqbal, E. A. Ansari, S. Akhtar, M. Farooq-I-Azam, S. R.
Resource Allocation for Ultra-Reliable V2X Networks With Partial Hassan, and R. Asif, “Optimal Learning Paradigm and Clustering for
CSI,” IEEE Transactions on Communications, vol. 70, no. 10, pp. Effective Radio Resource Management in 5G HetNets,” IEEE Access,
6532–6546, 2022. vol. 11, pp. 41 264–41 280, 2023.
[114] Y. Yuan, G. Zheng, K.-K. Wong, and K. B. Letaief, “Meta- [134] F. Tang, Y. Zhou, and N. Kato, “Deep Reinforcement Learning for
Reinforcement Learning Based Resource Allocation for Dynamic V2X Dynamic Uplink/Downlink Resource Allocation in High Mobility 5G
Communications,” IEEE Trans. on Vehicular Technology, vol. 70, HetNet,” IEEE Journal on Selected Areas in Communications, vol. 38,
no. 9, pp. 8964–8977, 2021. no. 12, pp. 2773–2782, 2020.
[115] H. Yang, X. Xie, and M. Kadoch, “Intelligent Resource Management [135] S. Singh, A. Kumbhar, I. Güvenç, and M. L. Sichitiu, “Distributed
Based on Reinforcement Learning for Ultra-Reliable and Low-Latency Approaches for Inter-Cell Interference Coordination in UAV-Based
IoV Communication Networks,” IEEE Trans. on Vehicular Technology, LTE-Advanced HetNets,” in 2018 IEEE 88th Vehicular Technology
vol. 68, no. 5, pp. 4157–4169, 2019. Conference (VTC-Fall), 2018, pp. 1–6.
[116] Q. Zhou, C. Guo, C. Wang, and L. Cui, “Radio Resource Manage- [136] ——, “Intelligent Interference Management in UAV-Based HetNets,”
ment for C-V2X Using Graph Matching and Actor-Critic Learning,” Telecom, no. 2, pp. 472–488, 2021.
IEEE Wireless Communications Letters, vol. 11, no. 12, pp. 2645– [137] C. Wang, D. Deng, L. Xu, W. Wang, and F. Gao, “Joint Interference
2649, 2022. Alignment and Power Control for Dense Networks via Deep Rein-
[117] L. Wang, H. Ye, L. Liang, and G. Y. Li, “Learn to Compress forcement Learning,” IEEE Wireless Communications Letters, vol. 10,
CSI and Allocate Resources in Vehicular Networks,” IEEE Trans. on no. 5, pp. 966–970, 2021.
Communications, vol. 68, no. 6, pp. 3640–3653, 2020. [138] Z. Zhang, J. Hou, X. Chu, H. Zhou, G. Wei, and J. Zhang, “Multi-
[118] X. Zhang, M. Peng, S. Yan, and Y. Sun, “Deep-Reinforcement- Agent Deep Reinforcement Learning Based Downlink Beamforming
Learning-Based Mode Selection and Resource Allocation for Cellular in Heterogeneous Networks,” IEEE Trans. on Wireless Communica-
V2X Communications,” IEEE Internet of Things Journal, vol. 7, no. 7, tions, vol. 22, no. 6, pp. 4247–4263, 2023.
pp. 6380–6391, 2020. [139] M. Simsek, M. Bennis, and A. Czylwik, “Dynamic Inter-Cell Interfer-
ence Coordination in HetNets: A reinforcement learning approach,” in
[119] A. D. Mafuta, B. T. J. Maharaj, and A. S. Alfa, “Decentralized
2012 IEEE Global Communications Conference (GLOBECOM), 2012,
Resource Allocation-Based Multiagent Deep Learning in Vehicular
pp. 5446–5450.
Network,” IEEE Systems Journal, vol. 17, no. 1, pp. 87–98, 2023.
[140] M. Simsek, M. Bennis, and I. Güvenç, “Learning Based Frequency-
[120] P. Xiang, H. Shan, Z. Su, Z. Zhang, C. Chen, and E.-P. Li, “Multi- and Time-Domain Inter-Cell Interference Coordination in HetNets,”
Agent Reinforcement Learning-Based Decentralized Spectrum Access IEEE Transactions on Vehicular Technology, vol. 64, no. 10, pp. 4589–
in Vehicular Networks With Emergent Communication,” IEEE Com- 4602, 2015.
munications Letters, vol. 27, no. 1, pp. 195–199, 2023. [141] Y. Wang, G. Feng, Y. Sun, S. Qin, and Y.-C. Liang, “Decentralized
[121] A. Damnjanovic, J. Montojo, Y. Wei, T. Ji, T. Luo, M. Vajapeyam, Learning Based Indoor Interference Mitigation for 5G-and-Beyond
T. Yoo, O. Song, and D. Malladi, “A survey on 3GPP heterogeneous Systems,” IEEE Transactions on Vehicular Technology, vol. 69, no. 10,
networks,” IEEE Wireless Communications, vol. 18, no. 3, pp. 10–21, pp. 12 124–12 135, 2020.
2011. [142] A. H. Arani, A. Mehbodniya, M. J. Omidi, F. Adachi, W. Saad, and
[122] P. Lohan, J.-B. Seo, and S. De, “Social Welfare Maximization in I. Güvenç, “Distributed Learning for Energy-Efficient Resource Man-
Two-Tier Heterogeneous Cellular Networks,” IEEE Communications agement in Self-Organizing Heterogeneous Networks,” IEEE Trans.
Letters, vol. 23, no. 10, pp. 1826–1830, 2019. on Vehicular Technology, vol. 66/10, pp. 9287–9303, 2017.
[123] W. Noh and K. Jang, “Downlink Interference Control in Hetero- [143] L. Xiao, H. Zhang, Y. Xiao, X. Wan, S. Liu, L.-C. Wang, and H. V.
geneous Cellular Networks: Macroscopic and Microscopic Control,” Poor, “Reinforcement Learning-Based Downlink Interference Control
IEEE Trans. on Vehicular Technology, vol. 66/7, pp. 5919–5932, 2017. for Ultra-Dense Small Cells,” IEEE Trans. on Wireless Communica-
[124] C. C. Coskun, K. Davaslioglu, and E. Ayanoglu, “Three-Stage tions, vol. 19, no. 1, pp. 423–434, 2020.
Resource Allocation Algorithm for Energy-Efficient Heterogeneous [144] L. Zhang and Y.-C. Liang, “Deep Reinforcement Learning for Multi-
Networks,” IEEE Trans. on Vehicular Technology, vol. 66, no. 8, pp. Agent Power Control in Heterogeneous Networks,” IEEE Trans. on
6942–6957, 2017. Wireless Communications, vol. 20, no. 4, pp. 2551–2564, 2021.
VOLUME , 51
[145] R. Raghunath, B. Peng, K.-L. Besser, and E. A. Jorswieck, “Rein- [166] I. Budhiraja, N. Kumar, and S. Tyagi, “Deep-Reinforcement-
forcement Learning-Based Global Programming for Energy Efficiency Learning-Based Proportional Fair Scheduling Control Scheme for
in Multi-Cell Interference Networks,” in ICC 2022 - IEEE Interna- Underlay D2D Communication,” IEEE Internet of Things Journal,
tional Conference on Communications, 2022, pp. 5499–5504. vol. 8, no. 5, pp. 3143–3156, 2021.
[146] M. Yan, J. Yang, K. Chen, Y. Sun, and G. Feng, “Self-Imitation [167] D. Shi, L. Li, T. Ohtsuki, M. Pan, Z. Han, and H. V. Poor, “Make
Learning-Based Inter-Cell Interference Coordination in Autonomous Smart Decisions Faster: Deciding D2D Resource Allocation via Stack-
HetNets,” IEEE Transactions on Network and Service Management, elberg Game Guided Multi-Agent Deep Reinforcement Learning,”
vol. 18, no. 4, pp. 4589–4601, 2021. IEEE Trans. on Mobile Computing, vol. 21/12, pp. 4426–4438, 2022.
[147] M. Agiwal, A. Roy, and N. Saxena, “Next Generation 5G Wireless [168] H. Xiang, Y. Yang, G. He, J. Huang, and D. He, “Multi-Agent
Networks: A Comprehensive Survey,” IEEE Communications Surveys Deep Reinforcement Learning-Based Power Control and Resource Al-
& Tutorials, vol. 18, no. 3, pp. 1617–1655, 2016. location for D2D Communications,” IEEE Wireless Communications
[148] A. M. H. Alibraheemi, M. N. Hindia, K. Dimyati, T. F. T. M. N. Letters, vol. 11, no. 8, pp. 1659–1663, 2022.
Izam, J. Yahaya, F. Qamar, and Z. H. Abdullah, “A Survey of [169] P.-G. Ye, Y.-G. Wang, and W. Tang, “S-MFRL: Spiking Mean Field
Resource Management in D2D Communication for B5G Networks,” Reinforcement Learning for Dynamic Resource Allocation of D2D
IEEE Access, vol. 11, pp. 7892–7923, 2023. Networks,” IEEE Trans. on Vehicular Technology, vol. 72, no. 1, pp.
[149] P. Osti, P. Lassila, and S. Aalto, “Performance of D2D underlay and 1032–1047, 2023.
overlay for multi-class elastic traffic,” Computer Communications, vol. [170] J. Lin, W. Yu, N. Zhang, X. Yang, H. Zhang, and W. Zhao, “A
117, pp. 147–163, 2018. [Online]. Survey on Internet of Things: Architecture, Enabling Technologies,
[150] M. Najla, P. Mach, and Z. Becvar, “Deep Learning for Selection Security and Privacy, and Applications,” IEEE Internet of Things
Between RF and VLC Bands in Device-to-Device Communication,” Journal, vol. 4, no. 5, pp. 1125–1142, 2017.
IEEE Wireless Communications Letters, vol. 9/10, pp. 1763–1767, [171] I. Ahmad, T. Kumar, M. Liyanage, J. Okwuibe, M. Ylianttila, and
2020. A. Gurtov, “Overview of 5G Security Challenges and Solutions,” IEEE
[151] L. Luo, Z. Liu, Z. Chen, M. Hua, W. Li, and B. Xia, “Age of Communications Standards Magazine, vol. 2, no. 1, pp. 36–43, 2018.
Information-Based Scheduling for Wireless D2D Systems With a Deep [172] L. Chettri and R. Bera, “A Comprehensive Survey on Internet of
Learning Approach,” IEEE Trans. on Green Communications and Things (IoT) Toward 5G Wireless Systems,” IEEE Internet of Things
Networking, vol. 6, no. 3, pp. 1875–1888, 2022. Journal, vol. 7, no. 1, pp. 16–32, 2020.
[152] T.-W. Ban, “A Deep Learning-Based TX Scheme Using Reduced [173] Y.-P. E. Wang, X. Lin, A. Adhikary, A. Grovlen, Y. Sui, Y. Blanken-
Feedback for D2D Networks,” IEEE Access, vol. 10, pp. 102 316– ship, J. Bergman, and H. S. Razaghi, “A Primer on 3GPP Narrowband
102 324, 2022. Internet of Things,” IEEE Communications Magazine, vol. 55, no. 3,
[153] J. Tan, Y.-C. Liang, L. Zhang, and G. Feng, “Deep Reinforcement pp. 117–123, 2017.
Learning for Joint Channel Selection and Power Control in D2D [174] R. A. Osman, S. N. Saleh, and Y. N. M. Saleh, “A Novel Interference
Networks,” IEEE Transactions on Wireless Communications, vol. 20, Avoidance Based on a Distributed Deep Learning Model for 5G-
no. 2, pp. 1363–1378, 2021. Enabled IoT,” Sensors, vol. 21, no. 19, 2021. [Online]. Available:
[154] T.-W. Ban, “An Autonomous TX Scheme Using Dueling DQN for https://www.mdpi.com/1424-8220/21/19/6555
D2D Communication Networks,” IEEE Trans. on Vehicular Technol-
[175] A. Nikoukar, Y. Shah, A. Memariani, M. GÃ¼neÅŸ, and B. Dez-
ogy, vol. 69, no. 12, pp. 16 348–16 352, 2020.
fouli, “Predictive Interference Management for Wireless Channels in
[155] J. Shi, Q. Zhang, Y.-C. Liang, and X. Yuan, “Distributed Deep Learn-
the Internet of Things,” in IEEE 31st Annual Intl. Symp. on Personal,
ing for Power Control in D2D Networks With Outdated Information,”
Indoor and Mobile Radio Communications, 2020, pp. 1–7.
IEEE Transactions on Wireless Communications, vol. 20, no. 9, pp.
[176] D. Yamamoto, H. Furukawa, A. Li, Y. Ito, K. Sato, K. Oshima,
5702–5713, 2021.
S. Hasegawa, Y. Watanabe, Y. Shoji, S.-J. Kim, and M. Hasegawa,
[156] B. Shang, L. Liu, R. M. Rao, V. Marojevic, and J. H. Reed, “3D
“Performance Evaluation of Reinforcement Learning Based Dis-
Spectrum Sharing for Hybrid D2D and UAV Networks,” IEEE Trans.
tributed Channel Selection Algorithm in Massive IoT Networks,” IEEE
on Communications, vol. 68, no. 9, pp. 5375–5389, 2020.
Access, vol. 10, pp. 67 870–67 882, 2022.
[157] T. Chen, X. Zhang, M. You, G. Zheng, and S. Lambotharan, “A
GNN-Based Supervised Learning Framework for Resource Allocation [177] Y. Liu, X. Kuai, X. Yuan, Y.-C. Liang, and L. Zhou, “Learning-Based
in Wireless IoT Networks,” IEEE Internet of Things Journal, vol. 9, Iterative Interference Cancellation for Cognitive Internet of Things,”
no. 3, pp. 1712–1724, 2022. IEEE Internet of Things Journal, vol. 6, no. 4, pp. 7213–7224, 2019.
[158] W. Lee and K. Lee, “Resource Allocation Scheme for Guarantee of [178] Y. Liu, X. Yuan, Y.-C. Liang, and Z. Han, “Machine Learning Based
QoS in D2D Communications Using Deep Neural Network,” IEEE Iterative Detection and Multi-Interference Cancellation for Cognitive
Communications Letters, vol. 25, no. 3, pp. 887–891, 2021. IoT,” IEEE Communications Letters, vol. 24, no. 9, pp. 1995–1999,
[159] W. Lee and R. Schober, “Deep Learning-Based Resource Alloca- 2020.
tion for Device-to-Device Communication,” IEEE Trans. on Wireless [179] Y. Liu and X. Yuan, “Temporal-Structure-Aware Interference Can-
Communications, vol. 21, no. 7, pp. 5235–5250, 2022. cellation for Asynchronous Cognitive IoT,” IEEE Wireless Communi-
[160] Z. Bi and W. Zhou, “Deep Reinforcement Learning Based Power cations Letters, vol. 11, no. 2, pp. 293–297, 2022.
Allocation for D2D Network,” in IEEE 91st Vehicular Technology [180] B. A. Homssi, A. Al-Hourani, Z. Krusevac, and W. S. T. Rowe,
Conf. (VTC-Spring), 2020, pp. 1–5. “Machine Learning Framework for Sensing and Modeling Interference
[161] J. Huang, Y. Yang, G. He, Y. Xiao, and J. Liu, “Deep Reinforcement in IoT Frequency Bands,” IEEE Internet of Things Journal, vol. 8,
Learning-Based Dynamic Spectrum Access for D2D Communication no. 6, pp. 4461–4471, 2021.
Underlay Cellular Networks,” IEEE Communications Letters, vol. 25, [181] S. Liu, L. Xiao, Z. Han, and Y. Tang, “Eliminating NB-IoT Interfer-
no. 8, pp. 2614–2618, 2021. ence to LTE System: A Sparse Machine Learning-Based Approach,”
[162] J. Kim, J. Park, J. Noh, and S. Cho, “Autonomous Power Allocation IEEE Internet of Things Journal, vol. 6, no. 4, pp. 6919–6932, 2019.
Based on Distributed Deep Learning for Device-to-Device Commu- [182] A. Farhad and J.-Y. Pyun, “AI-ERA: Artificial Intelligence-
nication Underlaying Cellular Network,” IEEE Access, vol. 8, pp. Empowered Resource Allocation for LoRa-Enabled IoT Applications,”
107 853–107 864, 2020. IEEE Trans. on Industrial Informatics, pp. 1–13, 2023.
[163] Y. Yuan, Z. Li, Z. Liu, Y. Yang, and X. Guan, “Double Deep Q- [183] W. Na, N. Kim, N.-N. Dao, and S. Cho, “Machine Learning-
Network Based Distributed Resource Matching Algorithm for D2D Based Communication Failure Identification Scheme for Directional
Communication,” IEEE Transactions on Vehicular Technology, vol. 71, Industrial IoT Networks,” IEEE Systems Journal, vol. 17, no. 1, pp.
no. 1, pp. 984–993, 2022. 1559–1568, 2023.
[164] Q. Guo, F. Tang, and N. Kato, “Federated Reinforcement Learning- [184] T. Chen, X. Zhang, M. You, G. Zheng, and S. Lambotharan, “A
Based Resource Allocation in D2D-Enabled 6G,” IEEE Network, pp. GNN-Based Supervised Learning Framework for Resource Allocation
1–7, 2022. in Wireless IoT Networks,” IEEE Internet of Things Journal, vol. 9,
[165] Z. Li and C. Guo, “Multi-Agent Deep Reinforcement Learning no. 3, pp. 1712–1724, 2022.
Based Spectrum Allocation for D2D Underlay Communications,” [185] S. Sakib, T. Tazrin, M. M. Fouda, Z. M. Fadlullah, and N. Nasser, “An
IEEE Trans. on Vehicular Technology, vol. 69/2, pp. 1828–1840, 2020. Efficient and Lightweight Predictive Channel Assignment Scheme for
52 VOLUME ,
Multiband B5G-Enabled Massive IoT: A Deep Learning Approach,” mization for mmWave Multi-UAV Links,” IEEE Trans. on Wireless
IEEE Internet of Things Journal, vol. 8, no. 7, pp. 5285–5297, 2021. Communications, vol. 20, no. 8, pp. 5481–5494, 2021.
[186] F. Hussain, R. Hussain, A. Anpalagan, and A. Benslimane, “A [206] L. Li, H. Ren, Q. Cheng, K. Xue, W. Chen, M. Debbah, and Z. Han,
New Block-Based Reinforcement Learning Approach for Distributed “Millimeter-Wave Networking in the Sky: A Machine Learning and
Resource Allocation in Clustered IoT Networks,” IEEE Trans. on Mean Field Game Approach for Joint Beamforming and Beam-
Vehicular Technology, vol. 69, no. 3, pp. 2891–2904, 2020. Steering,” IEEE Transactions on Wireless Communications, vol. 19,
[187] H. Cha and S.-L. Kim, “A Reinforcement Learning Approach to no. 10, pp. 6393–6408, 2020.
Dynamic Spectrum Access in Internet-of-Things Networks,” in IEEE [207] M. Elsayed, K. Shimotakahara, and M. Erol-Kantarci, “Machine
Intl. Conf. on Communications (ICC), 2019, pp. 1–6. Learning-based Inter-Beam Inter-Cell Interference Mitigation in
[188] F. Jameel, W. U. Khan, S. T. Shah, and T. Ristaniemi, “Towards Intel- mmWave,” in IEEE Intl. Conf. on Communications (ICC), 2020, pp.
ligent IoT Networks: Reinforcement Learning for Reliable Backscatter 1–6.
Communications,” in IEEE Globecom Workshops (GC Wkshps), 2019, [208] F. B. Mismar, B. L. Evans, and A. Alkhateeb, “Deep Reinforcement
pp. 1–6. Learning for 5G Networks: Joint Beamforming, Power Control, and
[189] W. Ahsan, W. Yi, Z. Qin, Y. Liu, and A. Nallanathan, “Resource Al- Interference Coordination,” IEEE Trans. on Communications, vol. 68,
location in Uplink NOMA-IoT Networks: A Reinforcement-Learning no. 3, pp. 1581–1592, 2020.
Approach,” IEEE Transactions on Wireless Communications, vol. 20, [209] Y. Al-Eryani and E. Hossain, “Self-Organizing mmWave MIMO Cell-
no. 8, pp. 5083–5098, 2021. Free Networks With Hybrid Beamforming: A Hierarchical DRL-Based
[190] S. Khairy, P. Balaprakash, L. X. Cai, and Y. Cheng, “Constrained Design,” IEEE Transactions on Communications, vol. 70, no. 5, pp.
Deep Reinforcement Learning for Energy Sustainable Multi-UAV 3169–3185, 2022.
Based Random Access IoT Networks With NOMA,” IEEE J. on Sel. [210] M. Fozi, A. R. Sharafat, and M. Bennis, “Fast MIMO Beamform-
Areas in Communications, vol. 39, no. 4, pp. 1101–1115, 2021. ing via Deep Reinforcement Learning for High Mobility mmWave
[191] B. Gu, X. Zhang, Z. Lin, and M. Alazab, “Deep Multiagent Connectivity,” IEEE Journal on Selected Areas in Communications,
Reinforcement-Learning-Based Resource Allocation for Internet of vol. 40, no. 1, pp. 127–142, 2022.
Controllable Things,” IEEE Internet of Things Journal, vol. 8, no. 5, [211] X. Zhou, J. Zhang, C.-W. Syu, C.-K. Wen, J. Zhang, and S. Jin,
pp. 3066–3074, 2021. “Model-Driven Deep Learning-Based MIMO-OFDM Detector: De-
[192] S. Xu, X. Zhang, C. Li, D. Wang, and L. Yang, “Deep Reinforcement sign, Simulation, and Experimental Results,” IEEE Trans. on Com-
Learning Approach for Joint Trajectory Design in Multi-UAV IoT munications, vol. 70, no. 8, pp. 5193–5207, 2022.
Networks,” IEEE Transactions on Vehicular Technology, vol. 71, no. 3, [212] J. Xia, K. He, W. Xu, S. Zhang, L. Fan, and G. K. Karagiannidis,
pp. 3389–3394, 2022. “A MIMO Detector With Deep Learning in the Presence of Correlated
[193] S. A. Busari, K. M. S. Huq, S. Mumtaz, L. Dai, and J. Rodriguez, Interference,” IEEE Trans. on Vehicular Technology, vol. 69, no. 4, pp.
“Millimeter-Wave Massive MIMO Communication for Future Wireless 4492–4497, 2020.
Systems: A Survey,” IEEE Communications Surveys & Tutorials, [213] K. He, Z. Wang, W. Huang, D. Deng, J. Xia, and L. Fan, “Generic
vol. 20, no. 2, pp. 836–869, 2018. Deep Learning-Based Linear Detectors for MIMO Systems Over
[194] S. Yang and L. Hanzo, “Fifty Years of MIMO Detection: The Road Correlated Noise Environments,” IEEE Access, vol. 8, pp. 29 922–
to Large-Scale MIMOs,” IEEE Communications Surveys & Tutorials, 29 929, 2020.
vol. 17, no. 4, pp. 1941–1988, 2015. [214] H. Tataria, M. Shafi, A. F. Molisch, M. Dohler, H. SjÃ¶land, and
[195] K. Zheng, L. Zhao, J. Mei, B. Shao, W. Xiang, and L. Hanzo, “Survey F. Tufvesson, “6G Wireless Systems: Vision, Requirements, Chal-
of Large-Scale MIMO Systems,” IEEE Communications Surveys & lenges, Insights, and Opportunities,” Proceedings of the IEEE, vol.
Tutorials, vol. 17, no. 3, pp. 1738–1760, 2015. 109, no. 7, pp. 1166–1199, 2021.
[196] S.-E. Chiu, N. Ronquillo, and T. Javidi, “Active Learning and CSI [215] Z. Chen, C. Han, Y. Wu, L. Li, C. Huang, Z. Zhang, G. Wang, and
Acquisition for mmWave Initial Alignment,” IEEE J. on Sel. Areas in W. Tong, “Terahertz Wireless Communications for 2030 and Beyond:
Communications, vol. 37, no. 11, pp. 2474–2489, 2019. A Cutting-Edge Frontier,” IEEE Communications Magazine, vol. 59,
[197] C. Feres and Z. Ding, “An Unsupervised Learning Paradigm for User no. 11, pp. 66–72, 2021.
Scheduling in Large Scale Multi-Antenna Systems,” IEEE Trans. on [216] C. Han, A. O. Bicen, and I. F. Akyildiz, “Multi-Wideband Waveform
Wireless Communications, vol. 22, no. 5, pp. 2932–2945, 2023. Design for Distance-Adaptive Wireless Communications in the Tera-
[198] J. Wang, R. Han, L. Bai, T. Zhang, J. Liu, and J. Choi, “Coordi- hertz Band,” IEEE Trans. on Signal Processing, vol. 64, no. 4, pp.
nated Beamforming for UAV-Aided Millimeter-Wave Communications 910–922, 2016.
Using GPML-Based Channel Estimation,” IEEE Trans. on Cognitive [217] S. Tarboush, H. Sarieddeen, H. Chen, M. H. Loukil, H. Jemaa, M.-S.
Communications and Networking, vol. 7, no. 1, pp. 100–109, 2021. Alouini, and T. Y. Al-Naffouri, “TeraMIMO: A Channel Simulator for
[199] M. Jafri, S. Srivastava, N. K. D. Venkategowda, A. K. Jagannatham, Wideband Ultra-Massive MIMO Terahertz Communications,” IEEE
and L. Hanzo, “Cooperative Hybrid Transmit Beamforming in Cell- Trans. on Vehicular Technology, vol. 70/12, pp. 12 325–12 341, 2021.
Free mmWave MIMO Networks,” IEEE Trans. on Vehicular Technol- [218] I. F. Akyildiz, C. Han, Z. Hu, S. Nie, and J. M. Jornet, “Terahertz
ogy, vol. 72, no. 5, pp. 6023–6038, 2023. Band Communication: An Old Problem Revisited and Research Direc-
[200] S. He, S. Xiong, W. Zhang, Y. Yang, J. Ren, and Y. Huang, “GBLinks: tions for the Next Decade,” IEEE Trans. on Communications, vol. 70,
GNN-Based Beam Selection and Link Activation for Ultra-Dense D2D no. 6, pp. 4250–4285, 2022.
mmWave Networks,” IEEE Trans. on Communications, vol. 70, no. 5, [219] J. L. Hernando, A. S. Tan, and A. Shojaeifard, “Mitigation of Phase
pp. 3451–3466, 2022. Noise-induced ICI at THz bands using CP-OFDM PT-RS signals,” in
[201] T. S. Cousik, V. K. Shah, T. Erpek, Y. E. Sagduyu, and J. H. 2022 IEEE Globecom Workshops (GC Wkshps), 2022, pp. 1766–1771.
Reed, “Deep Learning for Fast and Reliable Initial Access in AI- [220] B. Wang, F. Gao, S. Jin, H. Lin, and G. Y. Li, “Spatial- and
Driven 6G mm Wave Networks,” IEEE Trans. on Network Science Frequency-Wideband Effects in Millimeter-Wave Massive MIMO Sys-
and Engineering, pp. 1–12, 2022. tems,” IEEE Trans. on Signal Processing, vol. 66, no. 13, pp. 3393–
[202] P. Zhou, X. Fang, X. Wang, Y. Long, R. He, and X. Han, “Deep 3406, 2018.
Learning-Based Beam Management and Interference Coordination in [221] K. Dovelos, M. Matthaiou, H. Q. Ngo, and B. Bellalta, “Channel
Dense mmWave Networks,” IEEE Trans. on Vehicular Technology, Estimation and Hybrid Combining for Wideband Terahertz Massive
vol. 68, no. 1, pp. 592–603, 2019. MIMO Systems,” IEEE Journal on Selected Areas in Communications,
[203] P. Zhang and C. Qi, “Beam Allocation based on Deep Learning vol. 39, no. 6, pp. 1604–1620, 2021.
for Wideband mmWave Massive MIMO,” in ICC 2022 - IEEE [222] N. Kouzayha, M. A. Kishk, H. Sarieddeen, M.-S. Alouini, and
International Conference on Communications, 2022, pp. 913–918. T. Y. Al-Naffouri, “Coexisting Terahertz and RF Finite Wireless
[204] S. Huang, Y. Ye, and M. Xiao, “Learning-Based Hybrid Beamforming Networks: Coverage and Rate Analysis,” IEEE Trans. on Wireless
Design for Full-Duplex Millimeter Wave Systems,” IEEE Trans. on Communications, vol. 22, no. 7, pp. 4873–4889, 2023.
Cognitive Communications and Networking, vol. 7/1, pp. 120–132, [223] H. Zhang, H. Zhang, W. Liu, K. Long, J. Dong, and V. C. M.
2021. Leung, “Energy Efficient User Clustering, Hybrid Precoding and
[205] H.-L. Chiang, K.-C. Chen, W. Rave, M. Khalili Marandi, and Power Optimization in Terahertz MIMO-NOMA Systems,” IEEE J. on
G. Fettweis, “Machine-Learning Beam Tracking and Weight Opti- Sel. Areas in Communications, vol. 38, no. 9, pp. 2074–2085, 2020.
VOLUME , 53
[224] X. Ma, Z. Chen, Z. Li, W. Chen, and K. Liu, “Low Complexity [243] Z. Ding, X. Lei, G. K. Karagiannidis, R. Schober, J. Yuan, and
Beam Selection Scheme for Terahertz Systems: A Machine Learning V. K. Bhargava, “A Survey on Non-Orthogonal Multiple Access for
Approach,” in 2019 IEEE International Conference on Communica- 5G Networks: Research Challenges and Future Trends,” IEEE J. on
tions Workshops (ICC Workshops), 2019, pp. 1–6. Sel. Areas in Communications, vol. 35, no. 10, pp. 2181–2195, 2017.
[225] N. Abuzainab, M. Alrabeiah, A. Alkhateeb, and Y. E. Sagduyu, [244] B. Makki, K. Chitti, A. Behravan, and M.-S. Alouini, “A Survey of
“Deep Learning for THz Drones with Flying Intelligent Surfaces: NOMA: Current Status and Open Research Challenges,” IEEE Open
Beam and Handoff Prediction,” in IEEE Intl. Conf. on Communications Journal of the Communications Society, vol. 1, pp. 179–189, 2020.
Workshops (ICC Workshops), 2021, pp. 1–6. [245] N. Zhang, K. Cheng, and G. Kang, “A Machine-Learning-Based
[226] A. Alkhateeb, “DeepMIMO: A Generic Deep Learning Dataset for Blind Detection on Interference Modulation Order in NOMA Sys-
Millimeter Wave and Massive MIMO Applications,” arXiv preprint, tems,” IEEE Communications Letters, vol. 22/12, pp. 2463–2466,
arxiv:1902.06435, 2019. 2018.
[227] J. Hall, J. M. Jornet, N. Thawdar, T. Melodia, and F. Restuccia, “Deep [246] Q. N. Le, V.-D. Nguyen, O. A. Dobre, N.-P. Nguyen, R. Zhao, and
Learning at the Physical Layer for Adaptive Terahertz Communica- S. Chatzinotas,“Learning-Assisted User Clustering in Cell-Free Mas-
tions,” IEEE Trans. on Terahertz Science and Technology, vol. 13, sive MIMO-NOMA Networks,” IEEE Trans. on Vehicular Technology,
no. 2, pp. 102–112, 2023. vol. 70, no. 12, pp. 12 872–12 887, 2021.
[228] P. Sen, D. A. Pados, S. N. Batalama, E. Einarsson, J. P. Bird, and [247] X. Xu, Y. Liu, Q. Chen, X. Mu, and Z. Ding, “Distributed Auto-
J. M. Jornet, “The TeraNova platform: An integrated testbed for ultra- Learning GNN for Multi-Cell Cluster-Free NOMA Communications,”
broadband wireless communications at true Terahertz frequencies,” IEEE J. on Sel. Areas in Communications, vol. 41/4, pp. 1243–1258,
Computer Networks, vol. 179, p. 107370, 2020. 2023.
[229] D. He and Z. Wang, “Deep Learning-Assisted Demodulation for [248] M. Liu, T. Song, and G. Gui, “Deep Cognitive Perspective: Resource
TeraHertz Communications Under Hybrid Distortions,” IEEE Com- Allocation for NOMA-Based Heterogeneous IoT With Imperfect SIC,”
munications Letters, vol. 26, no. 2, pp. 325–329, 2022. IEEE Internet of Things Journal, vol. 6, no. 2, pp. 2885–2894, 2019.
[230] A. Shafie, C. Li, N. Yang, X. Zhou, and T. Q. Duong, “An Un- [249] A. Kumar and K. Kumar, “Deep Learning-Based Joint NOMA Signal
supervised Learning Approach for Spectrum Allocation in Terahertz Detection and Power Allocation in Cognitive Radio Networks,” IEEE
Communication Systems,” in IEEE Global Communications Conf., Trans. on Cognitive Communications and Networking, vol. 8, no. 4,
2022, pp. 3447–3453. pp. 1743–1752, 2022.
[231] H. Zarini, N. Gholipoor, M. R. Mili, M. Rasti, H. Tabassum, and [250] H. Zhang, H. Zhang, K. Long, and G. K. Karagiannidis, “Deep
E. Hossain, “Resource Management for Multiplexing eMBB and Learning Based Radio Resource Management in NOMA Networks:
URLLC Services Over RIS-Aided THz Communication,” IEEE Trans. User Association, Subchannel and Power Allocation,” IEEE Trans. on
on Communications, vol. 71, no. 2, pp. 1207–1225, 2023. Network Science and Engineering, vol. 7/4, pp. 2406–2415, 2020.
[232] S. Krishna Moorthy, M. Mcmanus, and Z. Guan, “ESN Reinforce- [251] J.-M. Kang, I.-M. Kim, and C.-J. Chun, “Deep Learning-Based
ment Learning for Spectrum and Flight Control in THz-Enabled Drone MIMO-NOMA With Imperfect SIC Decoding,” IEEE Systems Journal,
Networks,” IEEE/ACM Transactions on Networking, vol. 30, no. 2, pp. vol. 14, no. 3, pp. 3414–3417, 2020.
782–795, 2022. [252] A. Alajmi, W. Ahsan, M. Fayaz, and A. Nallanathan, “Intelligent
[233] A. Ahmadi and O. Semiari, “Reinforcement Learning for Optimized Resource Allocation in Backscatter-NOMA Networks: A Soft Actor-
Beam Training in Multi-Hop Terahertz Communications,” in ICC 2021 Critic Framework,” IEEE Trans. on Vehicular Technology, vol. 72,
- IEEE International Conference on Communications, 2021, pp. 1–6. no. 8, pp. 10 119–10 132, 2023.
[234] R. Barazideh, O. Semiari, S. Niknam, and B. Natarajan, “Reinforce- [253] C. He, Y. Hu, Y. Chen, and B. Zeng, “Joint Power Allocation and
ment Learning for Mitigating Intermittent Interference in Terahertz Channel Assignment for NOMA With Deep Reinforcement Learning,”
Communication Networks,” in IEEE Intl. Conf. on Communications IEEE J. on Sel. Areas in Communications, vol. 37/10, pp. 2200–2210,
Workshops (ICC Workshops), 2020, pp. 1–6. 2019.
[235] C. Huang, Z. Yang, G. C. Alexandropoulos, K. Xiong, L. Wei, [254] A. Emir, F. Kara, H. Kaya, and H. Yanikomeroglu, “DeepMuD:
C. Yuen, Z. Zhang, and M. Debbah, “Multi-Hop RIS-Empowered Ter- Multi-User Detection for Uplink Grant-Free NOMA IoT Networks
ahertz Communications: A DRL-Based Hybrid Beamforming Design,” via Deep Learning,” IEEE Wireless Communications Letters, vol. 10,
IEEE J. on Sel. Areas in Communications, vol. 39, no. 6, pp. 1663– no. 5, pp. 1133–1137, 2021.
1677, 2021. [255] J. Fu, Y. Xiao, H. Liu, P. Yang, and B. Zhang, “A Novel Intelligent
[236] H. Wang, H. Zhang, X. Liu, K. Long, and A. Nallanathan, “Joint SIC Detector for NOMA Systems Based on Deep Learning,” in IEEE
UAV Placement Optimization, Resource Allocation, and Computation 93rd Vehicular Technology Conf. (VTC-Spring), 2021, pp. 1–6.
Offloading for THz Band: A DRL Approach,” IEEE Trans. on Wireless [256] M. A. Aref and S. K. Jayaweera, “Deep Learning-aided Successive
Communications, vol. 22, no. 7, pp. 4890–4900, 2023. Interference Cancellation for MIMO-NOMA,” in IEEE Global Com-
[237] P.-C. Hsu, L.-H. Shen, C.-H. Liu, and K.-T. Feng, “Federated Deep munications Conf., 2020, pp. 1–5.
Reinforcement Learning for THz-Beam Search with Limited CSI,” in [257] P. Zhu, X. Wang, X. Jia, and Y. Xu, “Deep Learning-Based Signal
IEEE 96th Vehicular Technology Conf. (VTC-Fall), 2022, pp. 1–6. Detection with Soft Information for MISO-NOMA Systems,” in IEEE
[238] Y. M. Park, S. S. Hassan, Y. K. Tun, Z. Han, and C. S. Hong, 94th Vehicular Technology Conf. (VTC-Fall), 2021, pp. 1–5.
“Joint Trajectory and Resource Optimization of MEC-Assisted UAVs [258] X. Wang, P. Zhu, D. Li, Y. Xu, and X. You, “Pilot-Assisted SIMO-
in Sub-THz Networks: A Resources-Based Multi-Agent Proximal NOMA Signal Detection With Learnable Successive Interference
Policy Optimization DRL With Attention Mechanism,” IEEE Trans. Cancellation,” IEEE Communications Letters, vol. 25, no. 7, pp. 2385–
on Vehicular Technology, pp. 1–14, 2023. 2389, 2021.
[239] O. Maraqa, A. S. Rajasekaran, S. Al-Ahmadi, H. Yanikomeroglu, [259] S. K. Mahmud, Y. Liu, Y. Chen, and K. K. Chai, “Adaptive Re-
and S. M. Sait, “A Survey of Rate-Optimal Power Domain NOMA inforcement Learning Framework for NOMA-UAV Networks,” IEEE
With Enabling Technologies of Future Wireless Networks,” IEEE Communications Letters, vol. 25, no. 9, pp. 2943–2947, 2021.
Communications Surv. & Tutorials, vol. 22/4, pp. 2192–2235, 2020. [260] W. Ahsan, W. Yi, Z. Qin, Y. Liu, and A. Nallanathan, “Resource Al-
[240] S. M. R. Islam, N. Avazov, O. A. Dobre, and K.-s. Kwak, “Power- location in Uplink NOMA-IoT Networks: A Reinforcement-Learning
Domain Non-Orthogonal Multiple Access (NOMA) in 5G Systems: Approach,” IEEE Transactions on Wireless Communications, vol. 20,
Potentials and Challenges,” IEEE Communications Surveys & Tutori- no. 8, pp. 5083–5098, 2021.
als, vol. 19, no. 2, pp. 721–742, 2017. [261] Z. Shi, X. Xie, H. Lu, H. Yang, and J. Cai, “Deep Reinforcement
[241] L. Dai, B. Wang, Z. Ding, Z. Wang, S. Chen, and L. Hanzo, “A Learning Based Dynamic User Access and Decode Order Selection
Survey of Non-Orthogonal Multiple Access for 5G,” IEEE Communi- for Uplink NOMA System With Imperfect SIC,” IEEE Wireless
cations Surveys & Tutorials, vol. 20, no. 3, pp. 2294–2323, 2018. Communications Letters, vol. 10, no. 4, pp. 710–714, 2021.
[242] M. Elbayoumi, M. Kamel, W. Hamouda, and A. Youssef, “NOMA- [262] J. Zhang, X. Tao, H. Wu, N. Zhang, and X. Zhang, “Deep Reinforce-
Assisted Machine-Type Communications in UDN: State-of-the-Art ment Learning for Throughput Improvement of the Uplink Grant-Free
and Challenges,” IEEE Communications Surveys & Tutorials, vol. 22, NOMA System,” IEEE Internet of Things Journal, vol. 7, no. 7, pp.
no. 2, pp. 1276–1304, 2020. 6369–6379, 2020.
54 VOLUME ,
[263] A. Maatouk, F. Ayed, N. Piovesan, A. De Domenico, M. Deb- for IEEE Networking Letters, and an Associate Editor for Elsevier Vehicular
bah, and Z.-Q. Luo, “Teleqna: A benchmark dataset to assess large Communications.
language models telecommunications knowledge,” arXiv preprint
arXiv:2310.15051, 2023.
[264] L. Bariah, H. Zou, Q. Zhao, B. Mouhouche, F. Bader, and M. Debbah, MOHAMED AMINE FERRAGearned his Bache-
“Understanding Telecom Language Through Large Language Models,” lor’s, Master’s, Ph.D., and Habilitation degrees in
arXiv preprint arXiv:2306.07933, 2023. Computer Science from Badji Mokhtarâ C”Annaba
[265] M. Debbah, “Large language models for telecom,” in Intl. Conference University in Annaba, Algeria, completing his
on Fog and Mobile Edge Computing (FMEC). IEEE, 2023, pp. 3–4. studies in 2008, 2010, 2014, and 2019, respec-
[266] L. Bariah, Q. Zhao, H. Zou, Y. Tian, F. Bader, and M. Debbah, “Large tively. He served as an Associate Professor in
Language Models for Telecom: The Next Big Thing?” arXiv preprint the Department of Computer Science at Guelma
arXiv:2306.10249, 2023. University, Algeria, from 2014 until 2022. Con-
currently, from 2019 to 2022, he held the position
of Senior Researcher at the NAU-Lincoln Joint Re-
search Center of Intelligent Engineering, based at
POONAM LOHAN (SâC™19, Nanjing Agricultural University in China. As of 2022, Dr. Ferrag is the Lead
Mâ C™20)received the B.Tech. degree in Researcher at the Artificial Intelligence & Digital Science Research Center
Electronics and Communication Engineering at the Technology Innovation Institute in Abu Dhabi, United Arab Emirates.
(ECE) from the Guru Jambheshwar University Dr. Ferrag’s research is primarily focused on a spectrum of topics within
of Science and Technology, Hisar, India, in 2009, the cyber security domain, including wireless network security, network
the M.Tech. degree (Gold Medalist) in ECE from coding security, applied cryptography, blockchain technology, generative
the National Institute of Technology Kurukshetra AI, software security, and the application of AI in cyber security. His
(NITK), India, in 2011, and the Ph.D. degree scholarly output includes over 140 papers published in international journals
in Electrical Engineering from the IIT Delhi, and conference proceedings. Dr. Ferrag has spearheaded numerous projects
New Delhi, India, in November 2019. She was in research and development, fostering collaborative ties with academic
a Postgraduate Trainee with Magneti Marelli UM institutions in the UK, Australia, USA, Canada, and China. His contributions
Electronics Limited, Gurugram, India, from April 2011 to April 2012. She to the field include the creation of two cybersecurity datasets, namely, the
was an Assistant Professor at Punjab Engineering College (Deemed to be Edge-IIoT dataset and the FormAI dataset, which have become essential
University), Chandigarh, India, from Jan 2020 to Jan 2023, where she resources for AI researchers worldwide. His academic contributions have
received the best faculty award in Sep 2022. Currently, she is associated been recognized with the 2021 IEEE TEM Best Paper Award, the 2022
with the Smart Connected Vehicles Innovation Centre (SCVIC) at the Scopus Algeria Award, and many best paper conference awards. He has
University of Ottawa as a Postdoctoral Researcher. Her research interest consistently been named on Stanford University’s list of the world’s top 2%
pertains to AI/ML-enabled NextG wireless networks, Heterogeneous of scientists four times from 2020 through 2023. Dr. Ferrag also contributes
Cellular Networks, Vehicular Networks, and Unmanned Aerial Vehicle to the academic community as an associate editor for prestigious journals,
(UAV)-assisted networks in terms of coverage probability, capacity, such as the IEEE Internet of Things Journal and ICT Express (Elsevier). In
and secrecy data rate analysis, wireless channel modeling, interference addition to his research and editorial roles, Dr. Ferrag is a Senior Member
management, resource-allocation/scheduling for utility maximization of of the IEEE.
the network, VNF placement and SFC provisioning. She is an active
reviewer in the IEEE Internet of Things Journal, IEEE Trans. on Vehicular
Technology, IEEE Communication Letters, IEEE GLOBECOM, IEEE ICC,
and IEEE VTC conferences. NORBERT TIHANYIholds a B.Sc in Security En-
gineering, an M.Sc degree in Safety Engineering
and an another M.Sc degree in IT Engineering
(with honors). He received his Ph.D. in Infor-
mation Science and Technology from EÃ¶tvÃ¶s
BURAK KANTARCI (S’05, M’09, SM’12)is LorÃ¡nd University, Budapest, Hungary, in 2020.
a Full Professor and the Founding Director of He is a Public Body member of the Hungarian
the Smart Connected Vehicles Innovation Centre Academy of Science. Currently, he is a Lead
(SCVIC) and Next Generation Communications Researcher at the Technology Innovation Institute
and Computing Networks (NEXTCON) Research (TII), Abu Dhabi, United Arab Emirates. His re-
Lab at uOttawa. He holds a Ph.D. degree in search interests include cryptanalysis, security of
computer engineering and is the author/co-author embedded devices and cryptography-related prime number theory.
of 250+ publications in established journals and
conferences, and 15 book chapters. Continuously
listed among the top-cited scientists in telecom-
munications and networking based on the data YI SHIis a Research Associate Professor at Com-
reported by Stanford University since 2020, and since 2021, based on monwealth Cyber Initiative, Virginia Tech. He is
data collected from Microsoft Academic Graph, research.com has listed also a Research Associate Professor of Electrical
Dr. Kantarci among Canadaâ C™s top computer scientists. Dr. Kantarci and Computer Engineering (by Courtesy), Virginia
holds an Exemplary Editor Award from IEEE Communications Surveys and Tech. Prior to joining Virginia Tech, He was a
Tutorials (2021), and multiple best paper awards from various conferences, Senior Lead Research Scientist in BlueHalo Inc.
most recently from IEEE Globecom2021, Wireless World Research Forum Dr. Shi’s research focuses on algorithm design,
2022, and IEEE ICC2023, IEEE VCC2023. He is a recipient of the optimization, and machine learning for NextG
Minister’s Award of Excellence from Ontario Ministry of Colleges and wireless networks. He was a recipient of Test of
Universities (2021). He is the recipient of 2023 Technical Achievement Time Paper Award at IEEE INFOCOM 2023, a
Award of IEEE ComSoc Communications Software Technical Committee. recipient of Best Paper Award at IEEE INFOCOM
He was a Distinguished Speaker of the Association of Computing Machinery 2008, a recipient of the only Best Paper Award Runner-Up at IEEE INFO-
(ACM) in 2019-2021. Currently he is a Distinguished Lecturer of the COM 2011, a recipient of Best Student Paper Award at ACM WUWNet
IEEE Communications Society and IEEE Systems Council. He has been 2014, and a recipient of Best Paper Award at IEEE HST 2018. He was an
a keynote/invited speaker or panelist in 40 events. In 2019-2020, Dr. Exemplary Editor for IEEE Communications Surveys and Tutorials in 2014
Kantarci chaired the Communications Systems Integration and Modeling and a Distinguished TPC Member for IEEE INFOCOM in 2021. Dr. Shi is
Technical Committee of the Institute of Electrical and Electronics Engineers an Editor for IEEE Communications Surveys and Tutorials and an Editor
(IEEE). He has been a general chair, program chair or track chair in 30+ for IEEE Trans. on Cognitive Communications and Networking. He served
international conferences. He is an Editor of the IEEE Communications as a TPC chair for IEEE and ACM symposiums, tracks, and workshops.
Surveys and Tutorials, IEEE Internet of Things Journal, an Associate Editor
VOLUME , 55

From 5G To 6G Networks A Survey On AI-Based Jammin

Uploaded by

Copyright:

Available Formats

From 5G To 6G Networks A Survey On AI-Based Jammin

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

From 5G To 6G Networks A Survey On AI-Based Jammin

Uploaded by

Copyright:

Available Formats

This article has been accepted for publication in IEEE Open Journal of the Communications Society.

From 5G to 6G Networks, a Survey on

I. INTRODUCTION communication (mMTC), and ultra-reliable low latency com-

T HE third-generation partnership project (3GPP) intro-

TABLE 1. List of Abbreviations

Abbreviation Explanation Abbreviation Explanation Abbreviation Explanation

3GPP Release Timeline

REL-15 REL-16 REL-17 REL-18 REL-19 REL-20* REL-21*

FIGURE 2. 3GPP timeline for 5G advance and 6G based on [4]

AI/ML- Intentional THz

Section VII: IoT Networks Section VI: D2D Networks

FIGURE 3. Structure of the survey

Network Pulse Jamming

Jamming Attacks at Frequency

Non Smart Static Sweep

FIGURE 4. Types of jamming attacks at physical and MAC layers

of the frequencies, the output power of the jamming is

and effectively blocks communication. However, static hJR

the SINR value to exceed a threshold. Jamming signals Decision

to AI/ML-based research, specifying network systems, pa- dataset [47-48] Jamming

FIGURE 7. Summary of jamming mitigation related works

Scenario 1: UAVs as aerial users Scenario 2: UAVs as aerial base-stations

UAV positioning control/Optimal Deployment

UAVs as Aerial BSs [80-90] Power allocation Regression [96]

UAV to GUE co-channel interference control Drone Clustering

Rate per UAV/GUE

FIGURE 9. Summary of related works for interference mitigation in UAV-based networks

Cellular V2I V2P

In research [116], a cellular-V2X is considered where

cation to enable collaboration among themselves. For inter-

offering a diverse and dynamic environment where macro-

Heterogeneous Cellular Network

Network Objectives Wireless

Coexistence of Interference Supervised

Desired Cellular-D2D Intra D2D

Link Scheduling/Channel Selection DNN [150-151], [158-159]

Throughput/sum rate of D2D and Cellular links

D2D users fairness

Network Layer Communication Architecture Layer

FIGURE 16. IoT network architecture

bands, which determines the interference, a software-defined

Desired Signal Inter-cell Interference Intra-cell interference

mmWave-MIMO Inter-beam/ Precoding/

interference can be visualized in Fig. 18. Nowadays, with the

opportunities, they also pose unique challenges that are

Received Subtract Subtract Subtract

You might also like