Businesses are constantly under security threats, which not only costs billions of dollars in damage and recovery, but also detrimentally affects their reputation. A botnet-assisted attack is a widely known threat to these organizations. According to U.S. Federal Bureau of Investigation, “Botnets caused over $9 billion in losses to U.S. victims and over $110 billion globally. Approximately 500 million computers are infected each year, translating into 18 victims per second.” The most infamous attack, Rustock, infected 1 million machines, sending up to 30 billion spam emails a day. More recently, Mirai knocked offline 900,000 users of Deutsche Telekom. Thus, it is imperative to defend against botnet-assisted attacks. A botnet is a collection of bots, agents in compromised hosts, controlled by botmasters via command and control (C2) channels. A botmaster could be distributed across several agents that reside within or outside the network. Hence, a botnet can be used for tasks ranging from distributed denial-of-service (DDoS), to massive-scale spamming, to fraud and identify theft. Numerous measures are employed to fend off these threats and protect the network and its data from botnets. This project aims to devise an adaptive and robust botnet detection and mitigation system that leverages machine learning (ML). On the detection front, novel anomaly-based intrusion detection, employing host- and networkbased detection methods along with ML models adaptive to network dynamics and adversarial activities will be devised to build an advanced detection system that bots cannot easily evade. On the mitigation front, software-defined networking (SDN) will be leveraged to dynamically adapt the monitoring of the network, instigate root cause analysis, and automatically generate and enforce mitigation workflows. This project will broaden the scope of botnet detection and mitigation, including protection against zero-day threats. Advances made in collaboration with the industry partner, will have a lasting impact on the design principles and practices of cybersecurity for businesses and financial institutions.
H. Tsang, I. Akbari, M. A. Salahuddin, N. Limam and R. Boutaba. ATMoS+: Generalizable Threat Mitigation in SDN using Permutation Equivariant and Invariant Deep Reinforcement Learning. IEEE Communications Magazine. Accepted October 2021.