Leveraging AI Approaches for Robust Wireless Security: From Physical to Network Layer

Wireless communication has become the backbone of modern information infrastructure, connecting billions of devices and supporting a wide range of services, including mobile broadband, industrial automation, autonomous systems, and the Internet of Things (IoT). The evolution of technologies from 5G to emerging 6G, alongside Wi-Fi, Bluetooth, and low-power IoT protocols, has transformed global connectivity but also introduced significant security risks. At the physical layer, the broadcast nature of the medium exposes systems to eavesdropping, interference, and sophisticated jamming. Radio frequency fingerprint identification (RFFI) offers lightweight authentication but struggles with receiver heterogeneity and open-set recognition, where unseen or rogue devices may appear. At the network layer, growing traffic volumes, limited labeled data, and heterogeneous deployment environments complicate anomaly detection, threatening confidentiality, integrity, and availability. This thesis addresses the challenges of wireless security in a holistic manner by designing novel methods tailored to the vulnerabilities of different layers. At the physical layer, we develop Capodoglio, which models adaptive multi-armed bandit (MAB) jammers and proposes three defense strategies, namely Speed-Up, Helper-Node, and Mirror-MAB, to stabilize packet delivery under adversarial scenarios. DOA-CGAN introduces a conditional generative adversarial network (GAN) that suppresses structured interference in the covariance matrix of received signals, enabling classical direction-of-arrival (DOA) estimators to recover accuracy under jamming. For authentication, Triad-GAN learns receiver-invariant and transmitter-discriminative embeddings, improving the robustness of RFFI in multi-receiver scenarios. To extend to open-set recognition, OS-TriadGAN integrates label smoothing, temperature scaling, and an F1-optimized threshold, ensuring reliable rejection of unknown devices while preserving closed-set accuracy. At the network layer, Catch-All leverages graph neural networks (GNNs) and topological features to detect anomalies in Wi-Fi traffic, with robustness validated against perturbations and adversarial manipulations. Complementarily, GANSec introduces a classification-oriented conditional GAN that synthesizes wireless time-series data, strengthening supervised anomaly detection under domain shifts and data scarcity. Overall, this thesis contributes a portfolio of resilient and realistic methods that reinforce wireless security at both signal and traffic levels. At the physical layer, we advance defense strategies against adaptive jamming, introduce a generative approach that operates in the covariance matrix to remove interference, and establish receiver-invariant authentication extended to open-set scenarios. At the network layer, we develop graph-based anomaly detection and GAN-based augmentation tailored to various distribution challenges. Collectively, these studies underscore the value of generative, adversarial, and calibration-aware techniques in guiding the design of resilient and trustworthy wireless networks.