malicious traffic detection in cloud

Introduction

The design of this system plays a pivotal role in the successful realization of any complex computing project. In the context of this project, which focuses on the crucial task of malicious traffic detection in a cloud environment, the system design becomes the cornerstone upon which the efficacy and resilience of our solution stand

The design model is structured around the meticulous process of detecting and responding to malicious traffic. This is achieved through a systematic approach involving the collection of flow logs from the EC2 instance subnet. Furthermore, the integration of Amazon's cutting-edge AI tools, including Amazon Guard Duty, Amazon S3 buckets, and EventBridge, plays a pivotal role in the comprehensive analysis and storage of these logs.

Objectives

Design an efficient architecture/framework to accurately identify and differentiate malicious traffic in cloud environments
Implement the designed system by integrating necessary components and monitoring infrastructure to enable real-time detection of malicious activities
Evaluate the system's performance, accuracy, and scalability through rigorous experiments to ensure effective protection against threats in the cloud

Summary of the project

This project focuses on how to automatically log, detect and response to malicious traffic or vulnerabilities in a web server on EC2 instance cloud, using Amazon Guard duty

Project Architecture

The design model is structured around the meticulous process of detecting and responding to malicious traffic. This is achieved through a systematic approach involving the collection of flow logs from the EC2 instance subnet. Furthermore, the integration of Amazon's cutting-edge AI tools, including Amazon GuardDuty, Amazon S3 buckets, and EventBridge, plays a pivotal role in the comprehensive analysis and storage of these logs.

These meticulously collected flow logs are processed and analyzed to identify any signs of malicious activity, forming the cornerstone of our security strategy. In the event of a malicious event detection, the system is primed to respond with prompt and effective measures. These responses can range from immediate blocking of malicious activities to the proactive notification of administrators, ensuring that threats are addressed swiftly and effectively. This advanced architecture underscores the high level of sophistication and reliability of our approach in safeguarding the Amazon cloud environment against potential threats

Components and their Roles

Elastic Cloud Compute (EC2): Amazon Elastic Compute Cloud (Amazon EC2) offers the broadest and deepest compute platform, with over 700 instances and choice of the latest processor, storage, networking, operating system, and purchase model to help you best match the needs of your workload.
Amazon GuardDuty stands as a vigilant threat detection service that operates in a continuous fashion, diligently monitoring your Amazon Web Services accounts, workloads, and data stored within Amazon S3 to safeguard against malicious activity and unauthorized behavior. In cloud environments, the task of collecting and aggregating data related to account and network activities is made more accessible, but it can also become a time-consuming endeavor for security teams who must consistently scrutinize event log data for potential threats. With the introduction of GuardDuty, a prudent and cost-effective solution emerges for ongoing threat detection within the Amazon Web Services Cloud.This service leverages a combination of machine learning, anomaly detection, and the integration of threat intelligence to proficiently identify and prioritize potential threats. GuardDuty conducts a meticulous analysis of an immense volume of events, numbering in the billions, drawn from various data sources within Amazon Web Services, including event logs from Amazon CloudTrail, Amazon VPC Flow Logs, and DNS logs. What's more, enabling GuardDuty requires just a few clicks within the Amazon Web Services Management Console, without any need for software or hardware deployment or maintenance.
Amazon Event Bridge: Amazon EventBridge is a serverless event bus service by AWS that simplifies event-driven architectures. It facilitates the routing of events from various sources to different AWS services, enabling real-time responses and automation. EventBridge plays a pivotal role in orchestrating, scaling, and integrating applications and services in a seamless and efficient manner.
Simple storage service (S3) bucket: Amazon Simple Storage Service (Amazon S3) is a leading object storage solution, known for its exceptional scalability, data reliability, security, and speed. It caters to a diverse range of customers, regardless of their size or industry, allowing them to securely store and safeguard data for a wide array of purposes, including data lakes, cloud-native apps, and mobile applications. Its cost-efficient storage options and user-friendly management tools empower users to manage expenses, structure data, and establish precise access controls to align with specific business, organizational, and compliance needs.
VPC flow logs: VPC Flow Logs is a functionality designed to capture details concerning the IP traffic traversing network interfaces within your Virtual Private Cloud (VPC). Flow log data can be directed to several destinations, including Amazon CloudWatch Logs, Amazon S3, or Amazon Kinesis Data Firehose. Once a flow log is established, you can access and review the flow log records within the designated log group, storage bucket, or delivery stream. Flow logs serve various purposes, including:
Simple Notification Service (SNS): Amazon Simple Notification Service (Amazon SNS) is a web service that coordinates and manages the delivery or sending of messages to subscribing endpoints or clients. In Amazon SNS, there are two types of clients publishers and subscribers also referred to as producers and consumers. Publishers communicate asynchronously with subscribers by producing and sending a message to a topic, which is a logical access point and communication channel. Subscribers (web servers, email addresses, Amazon SQS queues, Lambda functions) consume or receive the message or notification over one of the supported protocols (Amazon SQS, HTTP/S, email, SMS, AWS Lambda) when they are subscribed to the topic.
Network Access control list (NACL): A network access control list (ACL) allows or denies specific inbound or outbound traffic at the subnet level. You can use the default network ACL for your VPC, or you can create a custom network ACL for your VPC with rules that are similar to the rules for your security groups in order to add an additional layer of security to your VPC.
VPC: Amazon Virtual Private Cloud (Amazon VPC) have the ability to deploy AWS resources within a logically segregated virtual network of your design. This virtual network closely mirrors a conventional network that one would typically manage in their on-premises data center, all while harnessing the advantages of AWS's scalable infrastructure.
VPC Subnet: A subnet within your VPC represents a defined range of IP addresses. This is where you deploy various AWS resources, including Amazon EC2 instances. Subnets can be connected to the internet, other VPCs, and your on-premises data centers, allowing you to manage traffic to and from your subnets efficiently via route tables.
AWS Security Group: A security group manages the flow of traffic permitted to enter and exit the resources it's linked to. Once associated with an EC2 instance, for instance, it governs the traffic entering and exiting that instance. When a VPC is established, it includes a default security group. Additional security groups can be generated for the VPC, each having its unique inbound and outbound regulations. For each inbound rule, you can define the source, port range, and protocol. Similarly, for each outbound rule, you can set the destination, port range, and protocol.

Problem Statement

What methods can be implemented to address the challenge of automating the logging process for activities on a web application server, with the aim of enhancing analysis and monitoring capabilities
The challenge lies in automating the detection of malicious scanning activities on a cloud-based server, aiming to prevent potential threats such as DOS, DDoS attacks, brute force attacks, and similar security risks.
What approaches can be employed to automate the response to detected malicious activities on a web server, enhancing the incident response process?

The Solution

Logs collection: All activity occurring in the VPC where the EC2 server hosting your web app is located is collected by the VPC flow logs. This includes all traffic entering and exiting the network.
Amazon Gard duty: The Amazon GuardDuty automatically detects findings (such as brute force, unprotected ports, port scanning, malware, etc.), vulnerabilities, and sends them to EventBridge to connect these findings to the channel for an immediate response.
Automate response: The response can either be automatic, wherein the EventBridge is connected to a Lambda function that invokes the Web Application Firewall (WAF) to block the traffic in the Network Access Control List (NACL).
Notify Security Admin: The EventBridge can also be connected to an SNS (Simple Notification Service) topic that will immediately alert and send malicious findings to an SNS subscriber, who is the security administrator. The email contains all information about the attacker, and the IP address can easily be blacklisted.

Implementation

STEP 1: Launching an EC2 Instance

To launch an EC2 instance on AWS, log in to the AWS Management Console, navigate to EC2, and click "Launch Instance." Choose an Amazon Machine Image (AMI) for your instance, select an instance type, configure instance details, add storage, and configure security groups. You can also add tags for organization. Review the instance configuration and choose an existing key pair or create a new one for secure access. After launching, monitor your instance's status, and once it's in the "running" state, you can connect to it using the private key you downloaded. Properly manage and secure your EC2 instances for effective cloud computing.

An EC2 instance running on Amazon. This Server is used to host the Web App

FIG 2: An EC2 instance running on Amazon. This Server is used to host the Web App

STEP 2: Installing Apache server on EC2 Instance

The following command used to install Apache server

sudo yum install httpd -y
To start the server

sudo systemctl start httpd
Configure Apache to run on system boot

sudo systemctl enable httpd

STEP 3 Installing WordPress on the Server

Download the latest WordPress installation package with the wget command. The following command should always download the latest release.

https://wordpress.org/latest.tar.gz
Unzip and unarchive the installation package. The installation folder is unzipped to a folder called wordpress.

tar -xzf latest.tar.gz

FIG 4: A wordpress server installed on EC2 instance

STEP 4: Creating a database user and database for the WordPress

Start the database server.

sudo systemctl start mariadb
Log in to the database server as the root user. Enter your database root password when prompted; this may be different than your root system password, or it might even be empty if you have not secured your database server.

mysql -u root -p
Create a user and password for your MySQL database. Your WordPress installation uses these values to communicate with your MySQL database.

CREATE USER 'wordpress-user'@'localhost' IDENTIFIED BY 'password';
Create your database. Give your database a descriptive, meaningful name, such as wordpress-db.

CREATE DATABASE wordpress-db;
Grant full privileges for your database to the WordPress user that you created earlier.

GRANT ALL PRIVILEGES ON wordpress-db.* TO "wordpress-user"@"localhost";
Flush the database privileges to pick up all of your changes and exit

FLUSH PRIVILEGES;
Exit

FIG5: A maria database configured for the web server on EC2 instance

STEP 5: To creating and editing the wp-config.php file

The WordPress installation folder contains a sample configuration file called wp-config-sample.php. In this procedure, you copy this file and edit it to fit your specific configuration.

Copy the wp-config-sample.php file to a file called wp-config.php. This creates a new configuration file and keeps the original sample file intact as a backup.

-> cp wordpress/wp-config-sample.php wordpress/wp-config.php
Edit the wp-config.php file with your favorite text editor (such as nano or vim) and enter values for your installation. If you do not have a favorite text editor, nano is suitable for beginners.

->nano wordpress/wp-config.php
Find the line that defines DB_NAME and change database_name_here to the database name that you created in step 4

-> define('DB_NAME', 'wordpress-db');
Find the line that defines DB_USER and change username_here to the database user that you created in step 4

-> define('DB_USER', 'wordpress-user');
Find the line that defines DB_PASSWORD and change password_here to the strong password that you created in step 4.

->define('DB_PASSWORD', 'password');
Save the file and exit your text editor.

STEP 6: Configuration on the Web page

To Navigate to the wordpress page on the browser the following path is pasted on the browser

-> https/ip-address/wp-admin

FIG : A wordPress Webserver successfully deployed on Amazon EC2 Instance

Data Collection and Processing

VPC flow logs for collection of event logs on Amazon EC2 instance

The VPC flow log is attached to the EC2 instance subnet to collect and log all the event happening on the Server. VPC (Virtual Private Cloud) flow logs capture network traffic data on Amazon EC2 instances within your VPC. These logs record information about inbound and outbound traffic, including details like source and destination IP addresses, ports, and protocols. VPC flow logs are valuable for network monitoring, security analysis, and compliance, providing insights into the flow of data in your AWS environment. You can use VPC flow logs to diagnose network issues, detect security threats, and gain a better understanding of the traffic patterns on your EC2 instances.

How to Enable VPC flow to Capture event on the EC2 instance hosting the web server

Open the Amazon VPC console.
In the navigation pane, choose Your VPCs. Select the checkbox for the VPC.
In the navigation pane, choose Subnets. Select the checkbox for the subnet.
Select the subnet which your EC2 instance is deployed
Choose Create Flow Logs.
Give a name to the VPC flow log
Select S3 bucket as the Destination
If you have an S3 bucket created already copy and paste the ARN else create a new S3 bucket
Steps To create S3 bucket would be discussed in “storage Management” section
Click on Create flow logs

FIG: A log file sample captured from EC2 instance hosting a Wordpress server

Malicious Traffic Detection

In this system design, Amazon GuardDuty, an AI-powered tool, is employed to autonomously identify and assess malicious threats targeting the EC2 WordPress server. It extracts this information from VPC flow logs, treating them as findings, and categorizes these threats based on their severity levels. GuardDuty additionally provides a visual representation of various threat types and their proportions in the form of a pie chart. In a broader context, Amazon GuardDuty can also be configured to automatically collect findings from diverse sources such as DNS logs, VPC flow logs, CloudTrail logs, S3 data events, among others. Furthermore, it features malware scanning capabilities to fortify EC2 instances against malware attacks.

GuardDuty offers features to manage trusted IP address lists and maintain a catalog of threat IP addresses. This functionality enables the automatic allowance of trusted IP addresses to access the EC2 instance while also promptly blocking any flagged IP address considered a threat. In the context of this project, a brute force attack is initiated on the EC2 instance responsible for hosting the WordPress blog. This deliberate action aims to trigger Amazon GuardDuty's automated detection mechanisms, which, in turn, swiftly communicate these findings to the EventBridge service. Subsequently, the EventBridge service can report these findings to the designated target destination, which may include an SNS topic.

The EventBridge Rule acts as a go-between, connecting the source of the findings (GuardDuty) and the destination, which might be either an AWS Lambda function or an SNS topic with subscribers. It is use to match event and send them to the target for processing To facilitate the collection of findings, EventBridge is equipped with a JSON role policy document that authorizes it to retrieve data from GuardDuty.

FIG8 : A Sample for an event findings captured with Guardy duty

Malicious Port Scanning using Nmap

Amazon GuardDuty can assist in identifying unauthorized scanning activities and automatically send notifications to the security administrator via the Simple Notification Service (SNS).

Nmap (Network Mapper) is a versatile network scanning tool that operates by sending packets to target hosts and analyzing their responses to perform tasks such as host discovery, port scanning, operating system and service detection, scripting, and network mapping. It is a valuable tool for network administrators and security professionals to assess network security, identify open ports and services, and gain insights into the network's structure and vulnerabilities.

In this project nmap tool was used to scan the EC2 instance to allow Guard duty to detect the malicious event.

FIG 9: Port scanning of the EC2 instance

After this port scanning was performed Amazon Guard duty was able to detect this threat and automatically send to the security administration mail so as to respond to the threat.

FIG 10: Notification sent to security Admin on port scanning detection

Security Measures

The security measures taken on this project

Security Group: The security group for the wordpress EC2 instance in such a way that it only allows SSH from the IP address of this Administrator. This implies that no other person can SSH into the server.

While experimenting the SSH port was first made open, the Amazon guard duty immediately send a notification email, to notify the security personnel that a threat has been detected and must be address.

FIG 11: Security group of the EC2 Instance. Allowing SSH traffic open to all

FIG 12: Guard Duty findings sent to SNS Notification and delivered to an email (EC2 instance has an unprotected port, which is probed by a known malicious host)

CONCLUSION

In conclusion, malicious intrusion detection is a critical aspect of maintaining security in cloud environments. While it offers numerous benefits, it also comes with some limitations that need to be carefully considered and managed to ensure the overall effectiveness of the security strategy.

Automate Malicious traffic detection in cloud using Amazon guard duty AI