Steps Overview:

1. Create a Google Chat Space and Webhook URL.

2. Configure CloudWatch Events to monitor pipeline failures.

3. Set up an SNS topic to handle notifications.

4. Process notifications with a Lambda function and send them to the Google Chat channel.

Step 1: Create a Google Chat Space and Webhook URL

1. Create a Google Chat Space

   In Google Chat, create a space where you want the pipeline notifications to appear

   

   .

2. Set Up Webhook Integration

   • Navigate to the space, click the dropdown menu, and go to Apps & Integrations.

   • Select Manage Webhooks to generate a webhook URL.Note: Webhooks can only be created in Workspace accounts (not personal Google accounts), and the

     

     administrator must enable this feature.

3. Save the Webhook URL

   Save the generated webhook URL. You'll use it later in the Lambda function.

Step 2: Configure CloudWatch to Monitor Pipeline Failures

AWS CodePipeline provides basic notifications for Slack, but for custom solutions like Google Chat, we’ll use CloudWatch.

1. Create a CloudWatch Event Rule

   • Go to CloudWatch > Buses > Rules > Create Rule.

   • 

   • Configure the rule to listen for CodePipeline execution state changes:

     • Event Source: AWS CodePipeline

     • Event Type: CodePipeline Action Execution State Change

     • Specific State: FAILE

       

       D

2. Set the Target as an SNS Topic

   • Under the Target section, create an SNS topic.

   • Allow CloudWatch to send events to this SNS topic.

Step 3: Process Notifications with a Lambda Function

The Lambda function will process the SNS notification and send the formatted message to Google Chat.

1. Create a Lambda Function

   • Go to the AWS Lambda Console and create a new function named FnPipelineNotificationProcessor.

   • Select Python as the runtime.

     

2. Deploy the Code

   • Clone the repository from:https://github.com/LajahShrestha/AWSPipelineNotificationProcessor

   • Modify the lambda_function.py file:

     • Replace the url value (line 7) with the Google Chat webhook URL you saved earlier.

   • Zip the code and upload it to an S3 bucket.

3. Upload Code to Lambda

   • In the Lambda console, upload the zipped file from S3.

   • Increase the timeout (under the Configuration tab) to around 10 minutes.

     

4. Add SNS as a Trigger

   • Add the SNS topic you created as a trigger for the Lambda function.

   • Ensure the Lambda execution role has sufficient permissions to read from SNS

     

     .

Step 4: Testing the Integration

1. Trigger a Pipeline Failure

   • Create a dummy pipeline or use an existing one.

   • Induce a failure to test the notification flow.

2. Verify Notifications

   • Check your Google Chat space for pipeline failure notifications.

   • If no notifications appear, review the Lambda function logs in CloudWatch.

Debugging Tips

• Webhook Issues: Ensure the Google Chat webhook URL is valid and accessible.

• Lambda Logs: Check CloudWatch logs for any processing errors.

• Permissions: Verify the Lambda execution role has permissions to subscribe to and process SNS messages.

• Event Rule: Ensure the CloudWatch event rule is correctly configured to capture the desired pipeline state changes.

Conclusion

With this setup, you can efficiently monitor AWS CodePipeline failures in Google Chat, ensuring prompt action when issues arise. This integration leverages the flexibility of CloudWatch, SNS, and Lambda to provide real-time notifications.

For further reference, check out this detailed tutorial on Medium.

In this blog we’ll be looking into how we can collect VPC flow logs from multiple accounts into a single S3 bucket. This use case is needed in an AWS Organizations having multiple AWS Accounts enabling a single S3 bucket as a source of logs for creating visual dashboards out of the data.

Prerequisite(Not mandatory):

1. AWS Organization setup

2. Log Archive Account setup by AWS Control tower

3. VPCs

Steps to follow:

1. Create a central S3 Bucket(sink) in Log Archive Account

2. Update the Destination Bucket Policy.

3. Create VPC

4. Enable VPC Flow logs & select the logs format

5. Choose central Bucket as Destination.

6. Create a central S3 Bucket(sink) in Log Archive Account

Create a central S3 Bucket(sink) in Log Archive Account

While managing AWS organization from AWS Control tower, it creates a Log Archive account which serves the purpose of storing different kinds of logs across the organization. We’ll also be centralizing the logs into a self created S3 bucket inside this account.

• Create an S3 bucket from the Management Console.

• After creating the S3 bucket, the bucket must be allowed to receive the logs files written into it from multiple sources. The sources in our use case are from VPC of all the aws accounts across the organization.

Bucket Policy for Centralized Flow log

Update the bucket Policy as the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AWSLogDeliveryWrite",
            "Effect": "Allow",
            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": [
                "arn:aws:s3:::DESTINATION_BUCKET_NAME",
                "arn:aws:s3:::DESTINATION_BUCKET_NAME/*"
            ],
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control",
                    "aws:SourceAccount": [
                        "SOURCE_ACCOUNT_NUMBER_1",
                        "SOURCE_ACCOUNT_NUMBER_2",
                        "SOURCE_ACCOUNT_NUMBER_3"
                    ]
                },
                "ArnLike": {
                    "aws:SourceArn": [
                        "arn:aws:logs:REGION:SOURCE_ACCOUNT_NUMBER_1:*",
                        "arn:aws:logs:REGION:SOURCE_ACCOUNT_NUMBER_2:*",
                        "arn:aws:logs:REGION:SOURCE_ACCOUNT_NUMBER_3:*"
                    ]
                }
            }
        },
        {
            "Sid": "AWSLogDeliveryCheck",
            "Effect": "Allow",
            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": [
                "s3:GetBucketAcl",
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::DESTINATION_BUCKET_NAME",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": [
                        "SOURCE_ACCOUNT_NUMBER_1",
                        "SOURCE_ACCOUNT_NUMBER_2",
                        "SOURCE_ACCOUNT_NUMBER_3"
                    ]
                },
                "ArnLike": {
                    "aws:SourceArn": [
                        "arn:aws:logs:REGION:SOURCE_ACCOUNT_NUMBER_1:*",
                        "arn:aws:logs:REGION:SOURCE_ACCOUNT_NUMBER_2:*",
                        "arn:aws:logs:REGION:SOURCE_ACCOUNT_NUMBER_3:*"
                    ]
                }
            }
        }
    ]
}

Here DESTINATION_BUCKET_NAME is the name of S3 we just creatd above. You may add more aws accounts as per need in the same format and replace the placeholders like "SOURCE_ACCOUNT_NUMBER_1" and "arn:aws:logs:REGION:SOURCE_ACCOUNT_NUMBER_1:*"

Enable VPC Flow logs & select the logs format

Create a VPC if not exists already. To enable VPC flow log, select the desired VPC and navigate to the flow logs tab under the VPC and select Create flow log.

Provide a desired name and select “ Send to an Amazon S3 bucket” as Destination and place the sink bucket arn(created in earlier step). Choose the desired format and select create

After sometime you can see the logs getting populated in the S3:

The logs are stored in the following format: Bucket_name/AWSLogs/ACCOUNT_NUMBER/vpcflowlogs/region/year/month/day/logfilename.log.gz

You can follow the same step of enabling vpc flow logs for all the required accounts and the logs will get centralized in one single bucket across the whole organization.

Reference:

https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-vpc-flow-logs-for-centralization-across-aws-accounts.html

In this blog, I narrate the story of Shuri and her journey of application development, transitioning from a local setup to a cloud-based solution at an industry scale, all while incorporating characters from the Marvel Cinematic Universe to add an engaging twist.

The Story:

Our story begins with our first Character Ms. Shuri. You guys must recognize her from the Marvel Cinematic Universe. But since we’re in the multiverse now, this is not MCU, This IS ACU, and you guys know what ACU mean. AWS Cinematic Universe. So Shuri is a college student and a part time developer at a startup company in her nation who happens to be a very creative problem solver.

Being a full time student, She needs to take notes for multiple subjects and keep track of assignments.Shuri struggles to manage and track the enormous amount of documents that need to be submitted as she needs to manage her professional job as well. So She comes up with an idea: Anote: A digital notebook application for taking notes and managing documents.

Phase 1: Local Development

Shuri develops the application locally on her own system. She uses Python with Flask for the backend, React for the frontend, and SQLite for the database.

Initial Features:

• Create, edit, and delete notes

• Share notes via unique links

• User authentication with basic username and password

Phase 2: Initial Deployment

Shuri realizes she can't use the same device every time and needs to access her notes from multiple devices.

To make the contents accessible from the web, she decides to host the application.

• She deploys it on her own using an EC2 instance with AWS's free tier service.

• She replaces SQLite with Amazon RDS for improved database management.

Phase 3: Growing User Base

• Shuri shares her application with college students, realizing she needs to create a login functionality.

• The application proves helpful, but it frequently goes down. Upon examining the nginx logs, Shuri notices that higher request volumes cause the server to crash, indicating insufficient server capacity.

• To address the curent issue, Given the small but growing user base and the server capacity issues, Shuri vertically scales the application. She upgrades the server from 2 GB to 8 GB of RAM, accommodating the fourfold increase in users.

Phase 4: CI/CD

• As the application gained popularity among students and attracted investors' attention, the user base expanded rapidly.

With this growth, data storage became a significant challenge. Shuri switched from EBS volumes to Amazon S3, a serverless object-level storage service. As demand increased, the need for continuous improvement in reliability and user-friendliness became apparent. Shuri would make changes locally, test them, and push the code to the repository.

New team members, including front-end developers, were brought on board as the application needed full upgrade.

The development process involved pulling the latest code from the repository and running build and run commands to reflect changes—even for minor updates like adjusting font sizes. This manual and repetitive process proved time-consuming and inefficient.

This demanded the need for automation to avoid time consuming manual and repetitive task. This is where our next character Peter comes into the story:

Automation: Peter addressed this challenge by implementing a CI/CD pipeline. This pipeline automates the repetitive build and deploy process, triggering automatically when code is pushed to the repository. (CodePipeline/CodeBuild)

With this automation in place, Shuri can now focus on adding new functionality without worrying about server deployment.

Phase 5: Need for a Standardized Deployment Process

Shuri's passion for her product and the positive user feedback drove her to continuously add features and address reported issues. However, a problem emerged: while the changes worked perfectly on her local machine, deploying them to the live server often caused the application to crash, resulting in downtime. This highlighted the need for a more robust deployment strategy.

2 major pre deployment procedure seems to be missing in the whole process.

Two key issues emerged:

Issue 1:

Inconsistency between development and live environments.

Solution By peter: Containerization

So in order to maintain the consistency the application was containerized using Docker.

Docker is a platform for developing, shipping, and running applications in containers. Containers are lightweight, portable, and self-sufficient units that can run consistently across different environments. Docker solves several key problems:

• Environment consistency: It ensures that the application runs the same way in development, testing, and production environments.

• Isolation: Containers isolate applications from each other and from the underlying system, reducing conflicts and improving security.

• Efficiency: Docker containers are more lightweight and use fewer resources compared to traditional virtual machines.

• Portability: Containerized applications can easily be moved between different systems and cloud platforms.

By using Docker, Shuri and her team can package the application and its dependencies into a container, ensuring that it behaves consistently across all environments and simplifying the deployment process.

Issue 2:

1. Lack of thorough testing and validation. Solution: Set up development, UAT, and production environments.

   

Peter, our DevOps hero, steps in. He establishes three crucial stages: Development → Staging → Production.

Peter implements a three-stage deployment process:

• Development: Where new features are built and initially tested

• Staging (UAT): A mirror of production for final testing and validation

• Production: The live environment accessed by users

This approach ensures thorough testing and reduces the risk of issues in the live environment, improving overall application stability and user experience.

And also to tackle the manual repetitive job of replicating the environment twice, Peter had created a CloudFormation Script as infrastructure as a code tool.

Infrastructure as Code (IaaC) is a practice of managing and provisioning computing infrastructure through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. Here's how it helps Peter:

• Consistency: IaaC ensures that the same environment is reproduced every time, eliminating discrepancies between development, staging, and production.

• Automation: Peter can automatically create and manage multiple environments without manual intervention, saving time and reducing human errors.

• Version Control: Infrastructure configurations can be versioned, allowing Peter to track changes and roll back if needed.

• Scalability: As Shuri's application grows, Peter can easily scale the infrastructure by modifying the CloudFormation script.

• Documentation: The CloudFormation script serves as living documentation of the infrastructure, making it easier for team members to understand the setup.

By using CloudFormation as an IaaC tool, Peter significantly streamlines the process of creating and managing multiple environments, ensuring consistency and reducing the time and effort required for infrastructure management.

Billing Hazard: Replicating environments and using more resources led to continuous, never-ending billing.

Peter's solution: Turn off EC2 instances and RDS during off-hours and at night using EventBridge Scheduler and auto-shutdown features. This significantly reduced costs by nearly half.

Phase 6: Gaining Popularity and the Need for Auto Scaling

As the application gained popularity, it attracted a large audience. During exam periods, there were sudden, unpredictable spikes in traffic. These huge surges caused the server to crash, even though it operated within normal limits on typical days.

To address this challenge, Peter implements an auto-scaling architecture that dynamically adjusts server capacity based on real-time metrics. He utilizes two key AWS services: Auto Scaling Groups (ASG) and Application Load Balancers (ALB). The ASG automatically increases or decreases the number of EC2 instances in response to fluctuating demand, while the ALB efficiently distributes incoming traffic across these instances. This setup ensures optimal performance during traffic spikes, such as exam periods, while maintaining cost-effectiveness during periods of lower usage.

Chain of Solutions: As we solve one problem, a new challenge often emerges, requiring another solution. This cycle of problem-solving and adaptation has been a key feature of peter, Shuri’s journey so far.

• Even after implementing autoscaling, the application was slowing down despite CPU and memory usage being within normal limits.

• Peter examined the statistics provided by AWS CloudWatch logs and alarms. He noticed that the database response time was increasing, indicating that the function responsible for fetching data from RDS was becoming a bottleneck due to numerous simultaneous requests.

• To address this, Shuri restructured the application into a microservice architecture. This allowed the module responsible for fetching data to be scaled and updated independently without causing system-wide downtime.

• The team deployed the new architecture to Amazon ECS using Fargate for automated, managed scaling.

Summary:

This story illustrates the overall DevOps process, demonstrating how DevOps culture accelerates software delivery to users. It specifically showcases the journey from basic to advanced implementations using AWS services.

In this blog, I’m excited to share my journey as both an organizer and speaker at Nepal’s inaugural Student Community Day, held on September 29, 2024. This event was a landmark gathering that brought together passionate students and professionals eager to connect, learn, and celebrate the power of technology and community.

Pre-Event Planning

Our team spent months meticulously preparing for the event. Originally scheduled for October 5, we realized that date fell during Dashain, a major festival in Nepal. Many students would likely have left the Kathmandu Valley to celebrate with family, so we decided to reschedule for Sunday, September 29, to ensure maximum participation and accommodate our VIP speakers.

Venue Challenges:

As we shifted the date, unexpected challenges arose. We had arranged to host the event at St. Xavier’s College in Maitighar, where a Saturday event would not interfere with regular classes. However, moving the event to a Sunday created logistical hurdles, as classes would be in session. The college was initially hesitant due to concerns about managing crowds, parking, and catering. After extensive planning and coordination with the venue, we finally secured the location.

Unexpected Obstacles: Just days before the event, Kathmandu experienced relentless rainfall, leading to widespread flooding. My home’s ground floor was flooded, and like many of our team members, I couldn’t reach the college on the 28th for final preparations. Additionally, heavy rainfall caused landslides, preventing out-of-valley students we had sponsored from traveling to Kathmandu. As the rain continued, our team felt the weight of months of preparation slipping out of reach, yet we persevered, determined to make the event a success.

Event Day

September 29 was the big day – the culmination of our hard work. I arrived at the college early, relieved to see that the venue setup team had completed their work despite the previous day’s rain. I was soon busy welcoming VIPs, guests, and speakers, ensuring everyone felt at ease.

As I made my way to the main event area, I was thrilled to see the venue filled with attendees. Every seat in the front section was taken, with the crowd extending back to the lower court. Despite the obstacles, the turnout was overwhelming and heartwarming.

The event commenced with the National Anthem, followed by a warm welcome from the college principal, who introduced our esteemed keynote speakers: Mr. Sambit Bhattrai and Mrs. Jen Looper from Amazon (AWS), and Mr. Sanjeev Pant from PM Square. The day was packed with inspiring talks, engaging activities, quizzes, prize and CTF(Capture the Flag) competition. Know more about it here.

My Session

I was honored to lead a breakout session on "Accelerating the Software Lifecycle with AWS and the Role of DevOps." As this was my first time delivering a technical session to such a large audience, I put in extra effort to create an engaging experience. Rather than a typical presentation, I crafted my session as a narrative, weaving in characters from the Marvel Cinematic Universe (MCU) to illustrate key concepts. This approach captured the audience’s attention, making complex ideas more relatable and enjoyable. Know more about the session on second part of the blog here.

Reflections on the Event :The event was a resounding success, with over 750 attendees. One of the standout moments amidst all the challenges was when, after my session, an audience member approached me to share that she previously held a negative view of DevOps due to a past experience. I was delighted to have changed her perspective, revealing the true potential and collaborative nature of DevOps.

In the end, despite the unexpected challenges, Nepal’s first Student Community Day surpassed our expectations. The passion and dedication of everyone involved made it a memorable experience, and I’m immensely proud of what we achieved. This event was more than just a gathering; it was a celebration of growth, learning, and the spirit of community.

Thank you for reading, and stay tuned for more insights into upcoming events!

Before we dive deep into the project lets breakdown the project into the phases in which the actions are to be performed:

Phase 1: Initial Setup and Deployment

Setup an EC2 Instance and run the application in docker container

Phase 2: Security

Implement Security checks using scanning tools

Phase 3: CI/CD Setup

1. Install Necessary Plugins: Jenkins plugins are installed for tools like SonarQube Scanner, NodeJS, and dependency management. Credentials for DockerHub are also added securely.

   1. Install Jenkins: This is used to automate the deployment process.

2. Configure CI/CD Pipeline: A Jenkins pipeline is created to automate the following stages:

   • Clean workspace

   • Checkout code from Git repository

   • SonarQube Analysis: Code is scanned for vulnerabilities and quality issues using SonarQube.

   • Quality Gate: The pipeline can be halted if quality thresholds are not met.

   • Install Dependencies: Any additional dependencies required by the application are installed.

   • OWASP FS SCAN: The code is scanned for potential vulnerabilities in dependencies using the OWASP Dependency-Check plugin.

   • TRIVY FS SCAN: The container image is scanned for vulnerabilities using Trivy.

   • Docker Build & Push: The application is built as a Docker image, tagged with your credentials, and pushed to a Docker registry.

   • TRIVY: The pushed image is scanned again for vulnerabilities using Trivy.

   • Deploy to container: The application container is deployed and runs on the server.

Phase 4: Monitoring

1. Install Prometheus and Grafana: These tools are used to monitor the application's performance and health. Prometheus collects metrics, while Grafana visualizes them for easy analysis.

2. Configure Prometheus: Prometheus is configured to scrape metrics from the application and other relevant sources like Node Exporter (which collects system metrics from the server).

3. Install Node Exporter: This tool is installed to collect system metrics from the server running the application.

4. Configure Prometheus Plugin Integration: Jenkins can be integrated with Prometheus to monitor the CI/CD pipeline itself.

5. Install Grafana: This is installed and configured to display visualizations of the collected metrics from Prometheus.

Phase 5: Notification

1. Implement Notification Services: This could involve setting up email notifications or other mechanisms to receive alerts when issues arise.

Phase 6: Kubernetes (Optional)

This phase covers deploying the application to a scalable environment using Kubernetes, a container orchestration platform. It also involves integrating monitoring with Prometheus and Node Exporter within Kubernetes.

1. Hardware Requirements: Server Configuration and port on which each service is configured to run:

Server 1(Application/Pipeline Instance) : T2.large
1. Jenkins(8080)
2. sonarqube(9000)
3. Netflix Container(8081)
---------------------------------------------------------------------=
On server 2(Monitoring Instance): T2.medium
3. Node Exporteer(9100)
4. Prometheus(9090) -> yaml configuration
5. Grafana(3000) data source

2. Application Architecture:

4. Implementation.

Phase 1: Initial Setup and Deployment

Instance: i-0a9b2965ff3babddd (devSecOps)

Step 1: Launch EC2 (Ubuntu 22.04):

• Provision an EC2 instance on AWS with Ubuntu 22.04.

• Connect to the instance using SSH.

Step 2: Clone the Code:

• Update all the packages and then clone the code.

• Clone your application's code repository onto the EC2 instance:

    git clone https://github.com/N4si/DevSecOps-Project.git
  

Step 3: Install Docker and Run the App Using a Container:

• Set up Docker on the EC2 instance:

    sudo apt-get update
    sudo apt-get install docker.io -y
    sudo usermod -aG docker $USER  # Replace with your system's username, e.g., 'ubuntu'
    newgrp docker
    sudo chmod 777 /var/run/docker.sock
  

• Build and run your application using Docker containers:

    docker build -t netflix .
    docker run -d --name netflix -p 8081:80 netflix:latest
  
    #to delete
    docker stop <containerid>
    docker rmi -f netflix
  

Once the container is run we can access the website but the page is empty because the contents are being delivered via an external API(TMDB) which were supposed to pass through the argument in our docker command.

TMDB: The movie database api credentials(personal) for the netflix clone website

Step 4: Get the API Key:

• Open a web browser and navigate to TMDB (The Movie Database) website.

• Click on "Login" and create an account.

• Once logged in, go to your profile and select "Settings."

• Click on "API" from the left-side panel.

• Create a new API key by clicking "Create" and accepting the terms and conditions.

• Provide the required basic details and click "Submit."

• You will receive your TMDB API key.

• 

Now recreate the Docker image with your api key:

docker build --build-arg TMDB_V3_API_KEY=<your-api-key> -t netflix .

sudo docker run -d -p 8081:80 netflix2
docker run -d --name sonar -p 9000:9000 sonarqube:lts-community

Phase 2: Security

1. Install SonarQube and Trivy:

   • Install SonarQube and Trivy on the EC2 instance to scan for vulnerabilities.

     sonarqube

       docker run -d --name sonar -p 9000:9000 sonarqube:lts-community
     

     To access:

     publicIP:9000 (by default username & password is admin)

     To install Trivy:

       sudo apt-get install wget apt-transport-https gnupg lsb-release
       wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
       echo deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main | sudo tee -a /etc/apt/sources.list.d/trivy.list
       sudo apt-get update
       sudo apt-get install trivy
     

     to scan image using trivy

       trivy image <imageid>
     

2. Integrate SonarQube and Configure:

   • Integrate SonarQube with your CI/CD pipeline.

   • Configure SonarQube to analyze code for quality and security issues.

Phase 3: CI/CD Setup

1. Install Jenkins for Automation:

   • Install Jenkins on the EC2 instance to automate deployment: Install Java

    sudo apt update
    sudo apt install fontconfig openjdk-17-jre
    java -version
    openjdk version "17.0.8" 2023-07-18
    OpenJDK Runtime Environment (build 17.0.8+7-Debian-1deb12u1)
    OpenJDK 64-Bit Server VM (build 17.0.8+7-Debian-1deb12u1, mixed mode, sharing)

    #jenkins
    sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
    https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
    echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
    https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
    /etc/apt/sources.list.d/jenkins.list > /dev/null
    sudo apt-get update
    sudo apt-get install jenkins
    sudo systemctl start jenkins
    sudo systemctl enable jenkins

• Access Jenkins in a web browser using the public IP of your EC2 instance.

  publicIp:8080

2. Install Necessary Plugins in Jenkins:

Goto Manage Jenkins →Plugins → Available Plugins →

Install below plugins

1 Eclipse Temurin Installer (Install without restart)

2 SonarQube Scanner (Install without restart)

3 NodeJs Plugin (Install Without restart)

4 Email Extension Plugin

Configure Java and Nodejs in Global Tool Configuration

Goto Manage Jenkins → Tools → Install JDK(17) and NodeJs(16)→ Click on Apply and Save

SonarQube

Create the token

Goto Jenkins Dashboard → Manage Jenkins → Credentials → Add Secret Text. It should look like this

After adding sonar token

Click on Apply and Save

The Configure System option is used in Jenkins to configure different server

Global Tool Configuration is used to configure different tools that we install using Plugins

We will install a sonar scanner in the tools.

Create a Jenkins webhook

1. Configure CI/CD Pipeline in Jenkins:

• Create a CI/CD pipeline in Jenkins to automate your application deployment.

Certainly, here are the instructions without step numbers:

Install Dependency-Check and Docker Tools in Jenkins

Install Dependency-Check Plugin:

• Go to "Dashboard" in your Jenkins web interface.

• Navigate to "Manage Jenkins" → "Manage Plugins."

• Click on the "Available" tab and search for "OWASP Dependency-Check."

• Check the checkbox for "OWASP Dependency-Check" and click on the "Install without restart" button.

Configure Dependency-Check Tool:

• After installing the Dependency-Check plugin, you need to configure the tool.

• Go to "Dashboard" → "Manage Jenkins" → "Global Tool Configuration."

• Find the section for "OWASP Dependency-Check."

• Add the tool's name, e.g., "DP-Check."

• Save your settings.

Install Docker Tools and Docker Plugins:

• Go to "Dashboard" in your Jenkins web interface.

• Navigate to "Manage Jenkins" → "Manage Plugins."

• Click on the "Available" tab and search for "Docker."

• Check the following Docker-related plugins:

  • Docker

  • Docker Commons

  • Docker Pipeline

  • Docker API

  • docker-build-step

• Click on the "Install without restart" button to install these plugins.

Add DockerHub Credentials:

• To securely handle DockerHub credentials in your Jenkins pipeline, follow these steps:

  • Go to "Dashboard" → "Manage Jenkins" → "Manage Credentials."

  • Click on "System" and then "Global credentials (unrestricted)."

  • Click on "Add Credentials" on the left side.

  • Choose "Secret text" as the kind of credentials.

  • Enter your DockerHub credentials (Username and Password) and give the credentials an ID (e.g., "docker").

  • Click "OK" to save your DockerHub credentials.

Now, you have installed the Dependency-Check plugin, configured the tool, and added Docker-related plugins along with your DockerHub credentials in Jenkins. You can now proceed with configuring your Jenkins pipeline to include these tools and credentials in your CI/CD process.

pipeline{
    agent any
    tools{
        jdk 'jdk17'
        nodejs 'node16'
    }
    environment {
        SCANNER_HOME=tool 'sonar-scanner'
    }
    stages {
        stage('clean workspace'){
            steps{
                cleanWs()
            }
        }
        stage('Checkout from Git'){
            steps{
                git branch: 'main', url: '<https://github.com/LajahShrestha/DevSecOps-Project.git>'
            }
        }
        stage("Sonarqube Analysis "){
            steps{
                withSonarQubeEnv('sonar-server') {
                    sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Netflix \\
                    -Dsonar.projectKey=Netflix '''
                }
            }
        }
        stage("quality gate"){
           steps {
                script {
                    waitForQualityGate abortPipeline: false, credentialsId: 'Sonar-token' 
                }
            } 
        }
        stage('Install Dependencies') {
            steps {
                sh "npm install"
            }
        }
        stage('OWASP FS SCAN') {
            steps {
                dependencyCheck additionalArguments: '--scan ./ --disableYarnAudit --disableNodeAudit', odcInstallation: 'DP-Check'
                dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
            }
        }
        stage('TRIVY FS SCAN') {
            steps {
                sh "trivy fs . > trivyfs.txt"
            }
        }
        stage("Docker Build & Push"){
            steps{
                script{
                   withDockerRegistry(credentialsId: 'docker', toolName: 'docker'){   
                       sh "docker build --build-arg TMDB_V3_API_KEY=8ddb5be8173d1f5790522f4d62bf3937 -t netflix ."
                       sh "docker tag netflix lajahshrestha/netflix-clone:latest "
                       sh "docker push lajahshrestha/netflix-clone:latest "
                    }
                }
            }
        }
        stage("TRIVY"){
            steps{
                sh "trivy image lajahshrestha/netflix-clone:latest > trivyimage.txt" 
            }
        }
        stage('Deploy to container'){
            steps{
                sh 'docker run -d -p 8081:80 lajahshrestha/netflix-clone:latest'
            }
        }
    }
}

Phase 4: Monitoring

Prometheus and Grafana are a powerful duo for monitoring applications and infrastructure. Prometheus, the unsung hero, acts as a data collector, constantly gathering metrics on various aspects like CPU usage, memory consumption, or API request counts. It keeps a watchful eye on these metrics, and if they stray from expected levels, it throws up an alert. But raw data can be overwhelming. This is where Grafana steps in. It acts as the visualizer, transforming the data collected by Prometheus into easy-to-understand graphs, charts, and dashboards. You can customize these dashboards to focus on key performance indicators (KPIs) that matter most to you. Together, they provide a clear picture of your application's health and performance, allowing you to identify and address potential issues before they snowball.

1. Install Prometheus and Grafana:

   Set up Prometheus and Grafana to monitor your application.

   Installing Prometheus:

   First, create a dedicated Linux user for Prometheus and download Prometheus:

    sudo useradd --system --no-create-home --shell /bin/false prometheus
    wget <https://github.com/prometheus/prometheus/releases/download/v2.47.1/prometheus-2.47.1.linux-amd64.tar.gz>
   

   Extract Prometheus files, move them, and create directories:

    tar -xvf prometheus-2.47.1.linux-amd64.tar.gz
    cd prometheus-2.47.1.linux-amd64/
    sudo mkdir -p /data /etc/prometheus
    sudo mv prometheus promtool /usr/local/bin/
    sudo mv consoles/ console_libraries/ /etc/prometheus/
    sudo mv prometheus.yml /etc/prometheus/prometheus.yml
   

   Set ownership for directories:

    sudo chown -R prometheus:prometheus /etc/prometheus/ /data/
   

   Create a systemd unit configuration file for Prometheus:

    sudo nano /etc/systemd/system/prometheus.service
   

   Add the following content to the prometheus.service file:

    [Unit]
    Description=Prometheus
    Wants=network-online.target
    After=network-online.target
   
    StartLimitIntervalSec=500
    StartLimitBurst=5
   
    [Service]
    User=prometheus
    Group=prometheus
    Type=simple
    Restart=on-failure
    RestartSec=5s
    ExecStart=/usr/local/bin/prometheus \\
      --config.file=/etc/prometheus/prometheus.yml \\
      --storage.tsdb.path=/data \\
      --web.console.templates=/etc/prometheus/consoles \\
      --web.console.libraries=/etc/prometheus/console_libraries \\
      --web.listen-address=0.0.0.0:9090 \\
      --web.enable-lifecycle
   
    [Install]
    WantedBy=multi-user.target
   

   Here's a brief explanation of the key parts in this prometheus.service file:

   • User and Group specify the Linux user and group under which Prometheus will run.

   • ExecStart is where you specify the Prometheus binary path, the location of the configuration file (prometheus.yml), the storage directory, and other settings.

   • web.listen-address configures Prometheus to listen on all network interfaces on port 9090.

   • web.enable-lifecycle allows for management of Prometheus through API calls.

Enable and start Prometheus:

    sudo systemctl enable prometheus
    sudo systemctl start prometheus

Verify Prometheus's status:

    sudo systemctl status prometheus

You can access Prometheus in a web browser using your server's IP and port 9090:

http://<your-server-ip>:9090

Installing Node Exporter:

Create a system user for Node Exporter and download Node Exporter:

    sudo useradd --system --no-create-home --shell /bin/false node_exporter
    wget <https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz>

Extract Node Exporter files, move the binary, and clean up:

    tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz
    sudo mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
    rm -rf node_exporter*

Create a systemd unit configuration file for Node Exporter:

    sudo nano /etc/systemd/system/node_exporter.service

Add the following content to the node_exporter.service file:

    [Unit]
    Description=Node Exporter
    Wants=network-online.target
    After=network-online.target

    StartLimitIntervalSec=500
    StartLimitBurst=5

    [Service]
    User=node_exporter
    Group=node_exporter
    Type=simple
    Restart=on-failure
    RestartSec=5s
    ExecStart=/usr/local/bin/node_exporter --collector.logind

    [Install]
    WantedBy=multi-user.target

Replace --collector.logind with any additional flags as needed.

Enable and start Node Exporter:

    sudo systemctl enable node_exporter
    sudo systemctl start node_exporter

Verify the Node Exporter's status:

    sudo systemctl status node_exporter

You can access Node Exporter metrics in Prometheus.

2. Configure Prometheus Plugin Integration:

   Integrate Jenkins with Prometheus to monitor the CI/CD pipeline.

   Prometheus Configuration:

   To configure Prometheus to scrape metrics from Node Exporter and Jenkins, you need to modify the prometheus.yml file. Here is an example prometheus.yml configuration for your setup:

    scrape_configs:
      - job_name: "prometheus"
        static_configs:
          - targets: ["localhost:9090"]
   
      - job_name: 'node_exporter'
        static_configs:
          - targets: ['localhost:9100']
   
      - job_name: 'jenkins'
        metrics_path: '/prometheus'
        static_configs:
          - targets: ['13.232.109.143:8080']
   

   Make sure to replace <your-jenkins-ip> and <your-jenkins-port> with the appropriate values for your Jenkins setup.

   Check the validity of the configuration file:

    promtool check config /etc/prometheus/prometheus.yml
   

   Reload the Prometheus configuration without restarting:

    curl -X POST <http://localhost:9090/-/reload>
   

   You can access Prometheus targets at:

   http://<your-prometheus-ip>:9090/targets

   

   

   

Install Grafana on Ubuntu 22.04 and Set it up to Work with Prometheus

Step 1: Install Dependencies:

First, ensure that all necessary dependencies are installed:

sudo apt-get update
sudo apt-get install -y apt-transport-https software-properties-common

Step 2: Add the GPG Key:

Add the GPG key for Grafana:

wget -q -O - <https://packages.grafana.com/gpg.key> | sudo apt-key add -

Step 3: Add Grafana Repository:

Add the repository for Grafana stable releases:

echo "deb <https://packages.grafana.com/oss/deb> stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Step 4: Update and Install Grafana:

Update the package list and install Grafana:

sudo apt-get update
sudo apt-get -y install grafana

Step 5: Enable and Start Grafana Service:

To automatically start Grafana after a reboot, enable the service:

sudo systemctl enable grafana-server

Then, start Grafana:

sudo systemctl start grafana-server

Step 6: Check Grafana Status:

Verify the status of the Grafana service to ensure it's running correctly:

sudo systemctl status grafana-server

Step 7: Access Grafana Web Interface:

Open a web browser and navigate to Grafana using your server's IP address. The default port for Grafana is 3000. For example:

http://<your-server-ip>:3000

You'll be prompted to log in to Grafana. The default username is "admin," and the default password is also "admin."

Step 8: Change the Default Password:

When you log in for the first time, Grafana will prompt you to change the default password for security reasons. Follow the prompts to set a new password.

Step 9: Add Prometheus Data Source:

To visualize metrics, you need to add a data source. Follow these steps:

• Click on the gear icon (⚙️) in the left sidebar to open the "Configuration" menu.

• Select "Data Sources."

• Click on the "Add data source" button.

• Choose "Prometheus" as the data source type.

• In the "HTTP" section:

  • Set the "URL" to http://localhost:9090 (assuming Prometheus is running on the same server).

  • Click the "Save & Test" button to ensure the data source is working.

Step 10: Import a Dashboard:

To make it easier to view metrics, you can import a pre-configured dashboard. Follow these steps:

• Click on the "+" (plus) icon in the left sidebar to open the "Create" menu.

• Select "Dashboard."

• Click on the "Import" dashboard option.

• Enter the dashboard code you want to import (e.g., code 1860).

• Click the "Load" button.

• Select the data source you added (Prometheus) from the dropdown.

• Click on the "Import" button.

You should now have a Grafana dashboard set up to visualize metrics from Prometheus.

Grafana is a powerful tool for creating visualizations and dashboards, and you can further customize it to suit your specific monitoring needs.

That's it! You've successfully installed and set up Grafana to work with Prometheus for monitoring and visualization.

1. Configure Prometheus Plugin Integration:

   • Integrate Jenkins with Prometheus to monitor the CI/CD pipeline.

Phase 5: Notification

1. Implement Notification Services:

   • Set up email notifications in Jenkins or other notification mechanisms.

    lajah.aws@gmail.com
    ktut rfsu nkog avvt

Phase 6: Kubernetes

Create Kubernetes Cluster with Nodegroups

In this phase, you'll set up a Kubernetes cluster with node groups. This will provide a scalable environment to deploy and manage your applications.

Monitor Kubernetes with Prometheus

Prometheus is a powerful monitoring and alerting toolkit, and you'll use it to monitor your Kubernetes cluster. Additionally, you'll install the node exporter using Helm to collect metrics from your cluster nodes.

Install Node Exporter using Helm

To begin monitoring your Kubernetes cluster, you'll install the Prometheus Node Exporter. This component allows you to collect system-level metrics from your cluster nodes. Here are the steps to install the Node Exporter using Helm:

1. Add the Prometheus Community Helm repository:

    helm repo add prometheus-community <https://prometheus-community.github.io/helm-charts>
   

2. Create a Kubernetes namespace for the Node Exporter:

    kubectl create namespace prometheus-node-exporter
   

3. Install the Node Exporter using Helm:

    helm install prometheus-node-exporter prometheus-community/prometheus-node-exporter --namespace prometheus-node-exporter
   

Add a Job to Scrape Metrics on nodeip:9001/metrics in prometheus.yml:

Update your Prometheus configuration (prometheus.yml) to add a new job for scraping metrics from nodeip:9001/metrics. You can do this by adding the following configuration to your prometheus.yml file:

  - job_name: 'Netflix'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['node1Ip:9100']

Replace 'your-job-name' with a descriptive name for your job. The static_configs section specifies the targets to scrape metrics from, and in this case, it's set to nodeip:9001.

Don't forget to reload or restart Prometheus to apply these changes to your configuration.

To deploy an application with ArgoCD, you can follow these steps, which I'll outline in Markdown format:

Deploy Application with ArgoCD

1. Install ArgoCD:

   You can install ArgoCD on your Kubernetes cluster by following the instructions provided in the EKS Workshop documentation.

2. Set Your GitHub Repository as a Source:

   After installing ArgoCD, you need to set up your GitHub repository as a source for your application deployment. This typically involves configuring the connection to your repository and defining the source for your ArgoCD application. The specific steps will depend on your setup and requirements.

3. Create an ArgoCD Application:

   • name: Set the name for your application.

   • destination: Define the destination where your application should be deployed.

   • project: Specify the project the application belongs to.

   • source: Set the source of your application, including the GitHub repository URL, revision, and the path to the application within the repository.

   • syncPolicy: Configure the sync policy, including automatic syncing, pruning, and self-healing.

4. Access your Application

   • To Access the app make sure port 30007 is open in your security group and then open a new tab paste your NodeIP:30007, your app should be running.

Conclusion

By implementing a DevSecOps approach with tools like Prometheus and Grafana, you gain a comprehensive view of your application's security, performance, and health. This allows for proactive management, ensuring a secure and well-functioning application throughout its lifecycle. Remember, DevSecOps is an ongoing process, so continuously refine your approach as your application evolves and make sure to cleanup the resources used while doing the process to cut off ongoing charges from the vendor. Happy learning.

AWS encompasses a myriad of cloud computing services, spanning computing power, storage, databases, machine learning, and more. These services collectively empower organizations to build, deploy, and scale applications with unparalleled flexibility and efficiency.

B. Understanding Deployment Needs

To navigate the AWS cloud effectively, it is crucial to discern the factors influencing deployment choices. Scalability, ease of management, and resource optimization are pivotal considerations in crafting a deployment strategy tailored to specific project requirements. As we delve into the intricacies of AWS deployment options, we will explore how each method aligns with these critical considerations.

AWS Deployment Methods

A. Traditional Deployment on EC2

Traditional deployment on Elastic Compute Cloud (EC2) instances involves manually deploying code on virtual machines. This method provides a high level of control over the infrastructure, making it suitable for scenarios where fine-tuning at the infrastructure level is paramount. It is particularly favored in applications where specific configurations and dedicated resources are essential.

Pros:

• Fine-Grained Control: Traditional deployment on EC2 provides developers with complete control over the underlying infrastructure, allowing for customized configurations and security settings.

• Versatility: Suitable for a wide range of applications, especially those with specific infrastructure requirements.

• Legacy Application Support: Ideal for hosting legacy applications that may not be easily containerized.

B. Containerization on EC2

Containerization, epitomized by Docker, has transformed the deployment landscape. AWS supports this paradigm through services like Docker on EC2. Containerization allows for packaging applications and their dependencies, ensuring consistency across different environments. This method is preferred in microservices architectures, providing agility and scalability by encapsulating services in lightweight, portable containers.

Pros:

• Portability: Containers encapsulate applications and dependencies, ensuring consistency across different environments and easing deployment across various platforms.

• Scalability: Well-suited for microservices architectures, allowing each service to scale independently, enhancing overall system scalability.

• Resource Efficiency: Containers share the host OS kernel, leading to more efficient resource utilization compared to traditional VMs.

C. AWS Elastic Beanstalk

AWS Elastic Beanstalk abstracts away infrastructure details, streamlining the deployment process for developers. It is an excellent choice for projects where simplicity and rapid deployment are prioritized. Elastic Beanstalk is particularly suitable for web applications and services where developers want to focus more on coding and less on infrastructure management.

Pros:

• Simplicity: Abstracts away infrastructure details, making it easy for developers to deploy applications without worrying about underlying infrastructure configurations.

• Rapid Deployment: Well-suited for projects that require quick iterations and frequent updates, enabling rapid deployment of web applications.

• Automatic Scaling: Elastic Beanstalk provides automated scaling capabilities based on application demand.

D. AWS Amplify

AWS Amplify is a platform tailored for developing and deploying web and mobile applications seamlessly. It excels in scenarios where quick iterations, continuous deployment, and scalability are essential. Amplify simplifies the deployment process, making it an ideal choice for projects with a strong focus on frontend and mobile development.

Pros:

• Streamlined Development: Simplifies the development and deployment of web and mobile applications, enabling developers to focus on code rather than infrastructure.

• Continuous Deployment: Integrates seamlessly with version control systems, allowing for continuous deployment and delivery.

• Backend Integration: Provides backend services, making it suitable for full-stack development and enabling developers to build scalable serverless applications.

E. Amazon ECS (Elastic Container Service)

Amazon ECS facilitates container orchestration at scale. This method is preferred in microservices architectures, offering efficient management and deployment of containerized applications. ECS allows developers to run, stop, and manage Docker containers on a cluster of EC2 instances, providing flexibility and scalability for containerized workloads.

Pros:

• Container Orchestration: Efficiently manages and deploys containerized applications at scale, ideal for microservices architectures.

• Integration with AWS Services: Seamless integration with other AWS services, enabling enhanced functionality for containerized applications.

• Customization: Provides flexibility in defining networking, security, and scaling configurations for containerized workloads.

F. Kubernetes Cluster on AWS (EKS)

For those delving into the realm of Kubernetes, AWS provides Elastic Kubernetes Service (EKS). Kubernetes is ideal for orchestrating and managing containerized applications in a microservices architecture. EKS simplifies Kubernetes deployment, making it suitable for projects that demand the advanced orchestration capabilities of Kubernetes, especially in large-scale applications.

Pros:

• Advanced Orchestration: Kubernetes offers advanced container orchestration capabilities, facilitating the management of complex microservices architectures.

• Community Support: Being an open-source platform, Kubernetes benefits from a vast and active community, ensuring continuous improvement and support.

• Multi-Cloud Deployment: EKS enables deploying Kubernetes clusters across multiple cloud providers, enhancing flexibility and avoiding vendor lock-in.

G. Serverless Deployments with AWS Lambda

The serverless architecture, exemplified by AWS Lambda, has gained prominence for its efficiency and cost-effectiveness. AWS Lambda allows developers to execute code without managing traditional server infrastructure. It is particularly favored in scenarios where event-driven, short-lived functions are sufficient, making it ideal for microservices architectures and applications with sporadic workloads.

Pros:

• Cost-Efficiency: Pay only for the compute time consumed, making it cost-effective for sporadically used functions and applications with variable workloads.

• Scalability: Scales automatically in response to incoming requests, ensuring optimal performance without the need for manual intervention.

• Event-Driven Architecture: Well-suited for event-driven architectures and microservices, allowing developers to focus on writing code rather than managing infrastructure.

In this comprehensive exploration of AWS deployment diversity, it becomes evident that each method caters to specific use cases and preferences. Whether you seek fine-grained control, simplicity, or scalability, AWS provides a deployment solution tailored to your project's unique demands.

Kubernetes has become the de facto standard for container orchestration, and setting up a Kubernetes cluster can be a crucial step in deploying and managing containerized applications. In this tutorial, we will guide you through the process of manually setting up a Kubernetes cluster on AWS or on-premises VMs using Rancher Kubernetes Engine (RKE). This step-by-step guide will help you deploy a three-node cluster with one master and two agent nodes.

Prerequisites

Before we begin, make sure you have the following resources available:

• Instances: 3 (Server 1, Server 2, Server 3)

• vCPUs: 4

• Memory: 8 GB

• Storage: 160 GB

Cluster Architecture

• k8s-1: Server 1 (Master node)

• k8s-2: Server 2 (Agent node)

• k8s-3: Server 3 (Agent node)

• 

Part-1: Master Node Setup (k8s-1)

Disable Firewall and Install RKE

sudo su

# Disable firewall
systemctl disable --now ufw

# Update and install required packages
apt update
apt install nfs-common -y
apt upgrade -y
apt autoremove -y

# Install RKE2
curl -sfL https://get.rke2.io | INSTALL_RKE2_CHANNEL=v1.26 INSTALL_RKE2_TYPE=server sh -
systemctl enable --now rke2-server.service

Configure kubectl and Check Node Status

# Symlink kubectl
ln -s $(find /var/lib/rancher/rke2/data/ -name kubectl) /usr/local/bin/kubectl

# Add kubectl configuration
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml

# Check node status
kubectl get node

Obtain Node Token for agent nodes to connect with master node

cat /var/lib/rancher/rke2/server/node-token

if in he case of fault and you require to reinstall the rke you may uninstall using the command: bash /usr/local/bin/rke2-uninstall.sh and then repeat the initial setup steps.

Part-2: Slave Nodes Setup (k8s-2 and k8s-3)

Disable Firewall and Install RKE

# Disable firewall
systemctl disable --now ufw

# Update and install required packages
apt update
apt install nfs-common -y
apt upgrade -y
apt autoremove -y

Add Configuration for VMs 2 and 3

# Export rancher1 IP and token
export RANCHER1_IP=10.0.4.196  # Change this!
export TOKEN=<TOKEN_FROM_SERVER_1>  # Change this as well.

# Install RKE2 as agent
curl -sfL https://get.rke2.io | INSTALL_RKE2_CHANNEL=v1.26 INSTALL_RKE2_TYPE=agent sh -

# Create config file
mkdir -p /etc/rancher/rke2/
echo "server: https://$RANCHER1_IP:9345" > /etc/rancher/rke2/config.yaml
echo "token: $TOKEN" >> /etc/rancher/rke2/config.yaml

# Enable and start
systemctl enable --now rke2-agent.service

Edit the configuration file (vim /etc/rancher/rke2/config.yaml) similarly for both Server 2 and Server 3.

Start RKE2 Services on Slave Nodes

bashCopy code# Master Node (k8s-1)
systemctl enable rke2-server.service
systemctl start rke2-server.service
systemctl restart rke2-server.service
systemctl status rke2-server.service

# Agent Nodes (k8s-2 and k8s-3)
systemctl enable rke2-agent.service
systemctl start rke2-agent.service
systemctl restart rke2-agent.service
systemctl status rke2-agent.service

Check Node Connection

bashCopy codekubectl get nodes -o wide -w

Setting up Rancher

Install Helm and Add Repositories

bashCopy code# Install Helm
curl -#L https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Add Helm repositories
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo add jetstack https://charts.jetstack.io

Configure Domain and Install Cert-Manager

bashCopy code# Install cert-manager
helm upgrade -i cert-manager jetstack/cert-manager -n cert-manager --create-namespace --set installCRDs=true

Install Rancher with Custom Domain

bashCopy code# Install Rancher
helm upgrade -i rancher rancher-latest/rancher --create-namespace --namespace cattle-system --set hostname=<yourdomain>--set bootstrapPassword=bootStrapAllTheThings --set replicas=1

Here I have mapped my custom domain with the public IP of Master VM using AWS Route53

Now if you access the domain you should obtain rancher UI.

You shall login using the bootstrap password using the one that you used during installation command. The site will be self certified once logged in for the first time.

Congratulations! You have successfully set up a Kubernetes cluster on AWS or on-premises VMs using Rancher Kubernetes Engine (RKE). You can now access Rancher using the specified domain and bootstrap password.

Architecture Diagram:

Requirements:

1. Must use a imported ssh key (Use ed25519 key)(done)

2. should have a elastic IP attached (done)

3. Must allow ssh key pair

Setup a Webserver

Domain to use: custom domain name(if you have)

Must have SSL/TLS enabled (Use certbot for this) Certbot

https://certbot.eff.org/favicon.ico

Must use NGINX as a webserver (done)

References:

How To Generate ed25519 SSH Key

https://www.unixtutorial.org/favicon.ico

1. Generating ssh key in ubuntu server

   

2. Importing ssh key

   

3. Deleting the Existing ssh key and updating with the self generated key pair (imported) using VIM editor.ssh/authorized_keys file on the ubuntu instance.

   

passphrase: mercantilesshpassphrase

1. Adding additional ssh key pair

   

   

2. Attaching Elastic IP

Elastic IP

An Elastic IP address is a static IPv4 address designed for dynamic cloud computing. An Elastic IP address is allocated to your AWS account, and is yours until you release it. By using an Elastic IP address, you can mask the failure of an instance or software by rapidly remapping the address to another instance in your account. Alternatively, you can specify the Elastic IP address in a DNS record for your domain, so that your domain points to your instance. For more information, see the documentation for your domain registrar, or Set up dynamic DNS on your Amazon Linux instance.

An Elastic IP address is a public IPv4 address, which is reachable from the internet. If your instance does not have a public IPv4 address, you can associate an Elastic IP address with your instance to enable communication with the internet. For example, this allows you to connect to your instance from your local computer.

Elastic IP: 13.200.34.252

1. Installing Certbot

sudo apt-get install certbot python3-certbot-nginx

1. Route S3 for personal Domain

   

Creating a custom record in route 53 for domain

www.lajahmercantile.test.mercantilecloud.com.np

Route53 successfully redirecting to the instance that is hosting webpage using Nginx server

1. Managing Website Files

Cloning the static website example into the instance working repository

Copying the website files to the /var/www/lajahmercantile@mercantilecloud.com.np directory

1. Configuring NGINX to serve my static website:

→Modifying the default configuration file /etc/nginx/sites-enable/default for the server to locate our custom website

sudo vim default

→ changing the default root location to the location containing out sample html file.

1. 

   Restarting Nginx

sudo systemctl restart restart nginx

1. Verifying the changes

Sample website Sucessfully hosted on nginx server sunning on Ubuntu webserver on custom domain created via Route 53