1. Running the application manually on your local machine.

2. Using Docker to containerize and run the application efficiently.

By the end, you’ll have a clear understanding of how to set up and run the application in both environments.

Part 1: Running the Application Manually

Running a .NET application manually involves installing necessary tools, building the project, and running it locally. Let’s break this down step-by-step.

Step 1: Install Prerequisites

Ensure you have the following installed:

• .NET SDK 6.0 to build and run the application.

• A terminal or command prompt (e.g., PowerShell, Bash).

Step 2: Navigate to the Project Directory

Assume the project structure is as follows:

QRCode_CoreMvc/
├── QRCode_UI/
│   ├── QRCode_UI.csproj
├── QRCode_CoreMvc.sln
├── Dockerfile

Navigate to the project directory:

cd QRCode_CoreMvc

Step 3: Restore Dependencies

Run the following command to restore the project dependencies:

dotnet restore

Explanation:
This command:

• Downloads the required NuGet packages.

• Resolves dependencies defined in the .csproj file.

Step 4: Build the Project

Compile the project using:

dotnet build

Explanation:
This step:

• Compiles the source code.

• Generates binaries for your application.

Step 5: Publish the Application

Prepare the application for deployment:

dotnet publish -c Release -o ./publish

Explanation:
This command:

• Builds the application (if not already built).

• Packages all necessary files into the ./publish directory.

Step 6: Run the Application

Navigate to the publish directory and start the application:cd publish

dotnet QRCode_UI.dll

Explanation:
This starts the .NET runtime to execute the application.

Step 7: Access the Application

Open a browser and visit the following URLs to access the application:

• http://localhost:5000 (HTTP)

• http://localhost:5001 (HTTPS)

You’ve successfully run the application manually!

Part 2: Running the Application in Docker

Now let’s containerize the application using Docker. This approach ensures portability and simplifies deployment across different environments.

Dockerfile Overview

The Dockerfile defines how the application is built and run in a container. Here’s the Dockerfile we’ll use:

# Stage 1: Build the application
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build-env
WORKDIR /app

# Restore dependencies
COPY *.sln ./
COPY QRCode_UI/*.csproj ./QRCode_UI/
RUN dotnet restore

# Build and publish the application
COPY . ./
WORKDIR /app/QRCode_UI
RUN dotnet publish -c Release -o /out

# Stage 2: Run the application
FROM mcr.microsoft.com/dotnet/aspnet:6.0
WORKDIR /app

# Copy the published output from the build stage
COPY --from=build-env /out .

# Expose the ports used by the application
EXPOSE 5000
EXPOSE 5001

# Configure the application to listen on all network interfaces
ENV ASPNETCORE_URLS="http://0.0.0.0:5000;http://0.0.0.0:5001"

# Run the application
ENTRYPOINT ["dotnet", "QRCode_UI.dll"]

This Dockerfile has two stages:

1. Build Stage: Compiles and publishes the application.

2. Runtime Stage: Runs the published application in a lightweight container.

Step 1: Build the Docker Image

Run the following command in the directory containing the Dockerfile:

docker build -t my-dotnet-app .

Explanation:

• docker build: Builds a Docker image.

• -t my-dotnet-app: Tags the image as my-dotnet-app.

• .: Specifies the current directory as the build context.

Step 2: Run the Docker Container

Start a container from the built image:

docker run -d -p 5000:5000 -p 5001:5001 --name my-dotnet-container my-dotnet-app

Explanation:

• docker run: Creates and starts a container.

• -d: Runs the container in detached mode (in the background).

• -p 5000:5000 -p 5001:5001: Maps container ports to your machine.

• --name my-dotnet-container: Names the container my-dotnet-container.

• my-dotnet-app: Specifies the image to use.

Step 3: Access the Application

Open a browser and navigate to:

• http://localhost:5000

• https://localhost:5001

The application is now running inside a Docker container!

Step 4: Manage the Container

To view the running containers:

docker ps

To stop the container:

docker stop my-dotnet-container

To remove the container:

docker rm my-dotnet-container

To delete the image:

docker rmi my-dotnet-app

Conclusion

We’ve demonstrated how to run a .NET application both manually and using Docker. The manual approach is straightforward for development and testing, while Docker simplifies deployment and ensures portability.

Key Benefits of Docker:

• Consistency across environments.

• Easy scaling and deployment.

With this knowledge, you can confidently run .NET applications in any environment. Happy coding!

Additional Resources:

• Docker Documentation

• Microsoft .NET Documentation

This blog provides a complete and well-structured guide for both manual and Docker-based application deployment.

Prerequisites

Before starting, ensure that you have the following installed:

• Java Development Kit (JDK): Make sure it’s compatible with the project.

• Apache Maven: Verify it’s installed by running mvn -v.

• Docker: To containerize and run the application in a container.

Step 1: Clone and Set Up the Project

Since you've already cloned the project, navigate to your project directory. Here’s a quick reminder:

git clone https://github.com/your-repository/BankApp.git
cd BankApp

You should see the following structure in the project:

BankApp/
├── pom.xml
└── src/
    ├── main/
    └── test/

pom.xml contains your project’s dependencies and build instructions, while the src/ folder contains the Java source code.

Step 2: Run the Application Manually with Maven

Let's start by building and running the application manually.

Build the Application

In your project directory, run:

./mvnw clean install

or if the mvnw script isn’t available, use:

mvn clean install

This will:

1. Download dependencies specified in pom.xml.

2. Compile the code and run any tests.

3. Package the application as a .jar file in the target/ directory.

After a successful build, you should see something like BankApp-0.0.1-SNAPSHOT.jar in the target/ folder.

Run the Application

To start the application, use:

java -jar target/BankApp-0.0.1-SNAPSHOT.jar

The application should now be running, and you can access it (if it’s a web app) by navigating to http://localhost:8080.

Step 3: Containerize the Application with Docker

Now, let’s make the application easier to deploy by containerizing it with Docker. A multi-stage Dockerfile will help us build and run the application more efficiently.

What is a Multi-Stage Dockerfile?

A multi-stage Dockerfile separates the build and runtime environments. We use a Maven-based image to compile the project, then copy only the compiled .jar file to a minimal Java runtime image, resulting in a smaller, more efficient Docker image.

Writing the Multi-Stage Dockerfile

In the project’s root directory, create a file named Dockerfile and add the following contents:

# Stage 1: Build the application
FROM maven:3.8.7-openjdk-17 AS build

# Set the working directory in the container
WORKDIR /app

# Copy the pom.xml and source code
COPY pom.xml .
COPY src ./src

# Build the application
RUN mvn clean install -DskipTests

# Stage 2: Run the application
FROM openjdk:17-jdk-alpine

# Set the working directory
WORKDIR /app

# Copy the jar file from the build stage
COPY --from=build /app/target/BankApp-0.0.1-SNAPSHOT.jar app.jar

# Expose the port (update if the app uses a different port)
EXPOSE 8080

# Command to run the application
CMD ["java", "-jar", "app.jar"]

Explanation of the Dockerfile

• Stage 1 (Build Stage):

  • We use maven:3.8.7-openjdk-17 as the base image.

  • The source code and pom.xml are copied into the container.

  • The mvn clean install -DskipTests command compiles the application and skips tests for faster builds.

  • The compiled .jar file is created in /app/target.

• Stage 2 (Runtime Stage):

  • We use a smaller image, openjdk:17-jdk-alpine, to run the app.

  • The .jar file from the build stage is copied over.

  • The container exposes port 8080 (adjust as needed).

  • The CMD instruction starts the app.

Step 4: Build and Run the Docker Image

Now that the Dockerfile is ready, let’s build and run the Docker image.

Build the Docker Image

Run the following command to build the Docker image:

docker build -t bankapp:latest .

The -t bankapp:latest option tags the image as bankapp with the latest tag.

Run the Docker Container

Start a container from the Docker image:

docker run -p 8080:81 bankapp:latest

• The -p 81:8080 option maps port 81 on your machine to port 8080 in the container.

• Your application should now be accessible at http://localhost:81.

Step 5: Verify the Application

To verify that everything is working:

1. Open a web browser or use a tool like curl to access http://localhost:81.

2. Check the logs in your terminal to ensure there are no errors.

Recap

Here’s what we covered in this guide:

1. Manual Build and Run: Built and ran the application locally using Maven.

2. Containerized Build and Run: Used a multi-stage Dockerfile to build a Docker image and run it in a container.

By containerizing the application, you’ve made it easier to deploy across environments consistently, without the need for a specific JDK or Maven installation. The multi-stage Dockerfile ensures the final image is optimized for size and performance, containing only what’s needed to run the application.

Table of Contents:

1. What is Nginx?

2. Installing Nginx

3. Installing Node.js

4. Setting Up PM2

5. Deploying the Node.js Application

6. Configuring Nginx as a Reverse Proxy

7. Running the Application

8. Using PM2 to Manage the Application

1. What is Nginx?

Nginx is a high-performance web server and reverse proxy server used to serve static content and forward dynamic requests to backend applications like Node.js. It also helps in load balancing, handling high concurrency, and caching, making it ideal for serving both static and dynamic content efficiently.

2. Installing Nginx

To install Nginx on a Linux-based server (like Ubuntu), follow these steps:

For Ubuntu/Debian:

cd
sudo apt update
sudo apt install nginx

For CentOS:

sudo yum install epel-release
sudo yum install nginx

After installation, you can start and enable Nginx to run on system boot:

sudo systemctl start nginx
sudo systemctl enable nginx

Check if Nginx is running:

sudo systemctl status nginx

You should now be able to visit http://your-server-ip in a browser and see the Nginx welcome page.

3. Installing Node.js

Next, you'll need to install Node.js on your server to run your Node.js application.

For Ubuntu/Debian:

1. Install the NodeSource repository for Node.js:

    # installs nvm (Node Version Manager)
    curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.0/install.sh | bash
   

2. Install Node.js:

    nvm install 22
   

3. Verify the installation:

    node -v
    npm -v
   

This should return the version of Node.js and npm installed.

4. Setting Up PM2

PM2 is a process manager for Node.js applications that allows you to keep applications alive forever, restart them on crashes, and even manage multiple instances.

Install PM2 globally:

sudo npm install pm2@latest -g

Verify the installation:

pm2 -v

Enable PM2 to start on boot:

pm2 startup

This command will generate a command that you need to run to configure PM2 to start on system reboot.

5. Deploying the Node.js Application

Now that Nginx and Node.js are set up, you can deploy your Node.js application. For this, you'll typically use Git to clone the repository, check out the desired branch, install dependencies, build the app, and start it using PM2.

Step-by-Step Guide:

1. Clone the Repository and Checkout a Specific Branch

Clone the Git repository of your Node.js application:

git clone https://github.com/your-repo/your-node-app.git
cd your-node-app

Checkout to the specific branch you want to deploy:

git checkout your-branch-name

2. Install Dependencies

Run the following command to install all the required dependencies for the Node.js applicatnpm install

3. Build the Application (Optional)

If your Node.js app requires a build step (e.g., for a frontend build or compiling assets), run:

npm run build

You can specify an environment for the build using the .env file. For example, to use a production environment, you might set:

npm run build:prod

4. Start the Application

Now, you can start your application. To run it normally (without PM2), use:

npm run start

However, to ensure that your app stays running even if the server restarts or crashes, we’ll use PM2.

6. Configuring Nginx as a Reverse Proxy

Nginx will serve as a reverse proxy to forward incoming HTTP requests to your Node.js application running on a specific port.

Configure Nginx:

1. Create a new Nginx server block (virtual host) configuration:

sudo nano /etc/nginx/sites-available/your-node-app

2. Add the following Nginx configuration to proxy requests to the Node.js app running on a specific port (e.g., port 3000):

# HTTP server block (port 80) to redirect to HTTPS
server {
    listen 80;
    server_name your-domain.com;

    # Redirect all HTTP requests to HTTPS
    return 301 https://$host$request_uri;
}

# HTTPS server block (port 443)
server {
    listen 443 ssl;
    server_name your-domain.com;

    # Self-signed SSL certificate paths
    ssl_certificate /etc/ssl/certs/selfsigned.crt;
    ssl_certificate_key /etc/ssl/private/selfsigned.key;

    location / {
        proxy_pass http://localhost:3000;  # Replace with your app's port if different
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

3. Create a symbolic link to enable this site:

sudo ln -s /etc/nginx/sites-available/your-node-app /etc/nginx/sites-enabled/

4. Test the Nginx configuration for syntax errors:

sudo nginx -t

5. Reload Nginx to apply the changes:

sudo systemctl reload nginx

Your Node.js application should now be accessible through your domain (e.g., http://your-domain.com).

7. Running the Application

After building the app, you can start it with PM2, which will ensure it runs in the background and automatically restarts if it crashes.

1. Start the app with PM2:

pm2 start server.js --name "my-app"

2. To check the app’s status:

pm2 status

3. To stop the app:

pm2 stop my-app

4. To restart the app:

pm2 restart my-app

5. To save the PM2 process list so that the app restarts automatically after reboot:

pm2 save

8. Using PM2 to Manage the Application

PM2 makes managing your Node.js application easier. Here are some common PM2 commands:

• List all processes:

    pm2 list
  

  

• View logs:

    pm2 logs your-node-app
  

• Monitor app resources:

    pm2 monit
  

PM2 will now ensure that your Node.js application runs smoothly and restarts automatically if there’s a failure or if the server reboots.

Multi-stage Dockerfile:

# Build Stage
FROM node:23-alpine3.19 AS builder

WORKDIR /app

# Install dependencies and build the project
COPY package.json package-lock.json ./
RUN npm ci --production

COPY . .
RUN npm run build

# Final Stage (Smaller image)
FROM node:23-alpine3.19 AS final

WORKDIR /app

# Copy the necessary files from the builder stage
COPY --from=builder /app/.next ./.next
COPY --from=builder /app/public ./public
COPY --from=builder /app/package.json ./package.json
COPY --from=builder /app/package-lock.json ./package-lock.json

# Install production dependencies in the final image
RUN npm install --production --silent

# Expose port and define the command to start the app
EXPOSE 3000
CMD ["npm", "run", "start"]

Explanation of the changes:

1. Builder stage:

   • This stage installs all dependencies and runs npm run build to generate the build output, which for a Next.js app will be in the .next directory.

2. Final stage:

   • Copies only the necessary files (i.e., .next directory, public folder, package.json, and package-lock.json) to the final image.

   • Installs production dependencies (npm install --production) in the final image.

How to use it:

1. Build the Docker image:

   Run the following command to build the image:

    sudo docker build -t node .
   

2. Run the container:

   After building the image, you can run the container using:

    sudo docker run -p 3000:3000 node
   

This setup will produce a smaller final image, as it only includes the necessary files for running the production application and avoids including unnecessary development dependencies and files. Let me know how it goes!

Before we begin, make sure you have the following:

1. An AWS account.

2. eksctl installed on your local machine.

3. kubectl installed on your local machine.

4. Git for version control.

5. https://archive.eksworkshop.com/intermediate/290_argocd/configure/

Step 1: Create an EKS Cluster

Use the following eksctl command to create an EKS cluster named "dev" in the EU (Stockholm) region with a managed node group:

eksctl create cluster --name dev --region eu-north-1 --nodegroup-name workers --node-type t3.medium --nodes 2 --nodes-min 2 --nodes-max 2 --managed

Wait for the cluster to be provisioned before proceeding.

Step 2: Install Argo CD

Create a namespace for Argo CD and apply the installation manifests:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Patch the Argo CD service to use a LoadBalancer:

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

Retrieve the external IP or DNS of the Argo CD server:

kubectl get svc argocd-server -n argocd -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Now, you can access the Argo CD UI using the provided external IP or DNS.

export ARGO_PWD=`kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d`
echo $ARGO_PWD

Step 3: Deploy Tetris Game with Argo CD

Clone the Tetris game repository:

git clone https://github.com/muhammadhassanb111/tetris-game/tree/main

Step1: Launch an Ec2 Instance

To launch an AWS EC2 instance with Ubuntu 22.04 using the AWS Management Console, sign in to your AWS account, access the EC2 dashboard, and click "Launch Instances." In "Step 1," select "Ubuntu 22.04" as the AMI, and in "Step 2," choose "t2.medium" as the instance type. Configure the instance details, storage, tags, and security group settings according to your requirements. Review the settings, create or select a key pair for secure access, and launch the instance. Once launched, you can connect to it via SSH using the associated key pair.

Step2A: Install Docker and Run Sonarqube Container

Connect to your Ec2 instance using Putty, Mobaxtreme or Git bash and install docker on it.

sudo apt-get update
sudo apt install docker.io -y
sudo usermod -aG docker ubuntu
newgrp docker
sudo chmod 777 /var/run/docker.sock

Pull the SonarQube Docker image and run it.

After the docker installation, we will create a Sonarqube container (Remember to add 9000 ports in the security group).

docker run -d --name sonar -p 9000:9000 sonarqube:lts-community

Now copy the IP address of the ec2 instance

<ec2-public-ip:9000>

Provide Login and password

login admin
password admin

Update your Sonarqube password & This is the Sonarqube dashboard

Step2B: Integrating SonarQube with GitHub Actions

Integrating SonarQube with GitHub Actions allows you to automatically analyze your code for quality and security as part of your continuous integration pipeline.

We already have Sonarqube up and running

On Sonarqube Dashboard click on Manually

Next, provide a name for your project and provide a Branch name and click on setup

On the next page click on With GitHub actions

This will Generate an overview of the Project and provide some instructions to integrate

Let's Open your GitHub and select your Repository

In my case it is Netflix-clone and Click on Settings

Search for Secrets and variables and click on and again click on actions

It will open a page like this click on New Repository secret

Now go back to Your Sonarqube Dashboard

Copy SONAR_TOKEN and click on Generate Token

Click on Generate

Let's copy the Token and add it to GitHub secrets

Now go back to GitHub and Paste the copied name for the secret and token

Name: SONAR_TOKEN

Secret: Paste Your Token and click on Add secret

Now go back to the Sonarqube Dashboard

Copy the Name and Value

Go to GitHub now and paste-like this and click on add secret

Our Sonarqube secrets are added and you can see

Go to Sonarqube Dashboard and click on continue

Now create your Workflow for your Project. In my case, the Netflix project is built using React Js. That's why I am selecting Other

Now it Generates and workflow for my Project

Go back to GitHub. click on Add file and then create a new file

Go back to the Sonarqube dashboard and copy the file name and content

Here file name (in my case only )

sonar-project.properties

The content to add to the file is (copied from the above image)

sonar.projectKey=Netflix

Add in GitHub like this

Let's add our workflow

To do that click on Add file and then click on Create a new file

Here is the file name

.github/workflows/build.yml  #you can use any name iam using sonar.yml

Copy content and add it to the file

name: Build,Analyze,scan

on:
  push:
    branches:
      - main


jobs:
  build-analyze-scan:
    name: Build
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
        with:
          fetch-depth: 0  # Shallow clones should be disabled for a better relevancy of analysis

      - name: Build and analyze with SonarQube
        uses: sonarsource/sonarqube-scan-action@master
        env:
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
          SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}

Click on commit changes

Now workflow is created.

Click on Actions now

Now it's automatically started the workflow

Let's click on Build and see what are the steps involved

Click on Run Sonarsource and you can do this after the build completion

Build complete.

Go to the Sonarqube dashboard and click on projects and you can see the analysis

If you want to see the full report, click on issues.

Step3: Let's scan files using Trivy

Add this code to your sonar.yml (I mean workflow)

- name: install trivy
  run: |
    #install trivy
    sudo apt-get install wget apt-transport-https gnupg lsb-release -y
    wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg > /dev/null
    echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list
    sudo apt-get update
    sudo apt-get install trivy -y
    #command to scan files
    trivy fs .

GitHub Actions workflow step that installs Trivy, a popular open-source vulnerability scanner for containers, and then uses it to scan the files.

I added a step in the workflow

Commit changes

Click on actions again

It started the workflow build

Click on Build, Analyze and scan

It installed Trivy version 0.46.0 and scanned files also. See report

The file scan is completed, this is another security check

Step4A: Docker build and push to Dockerhub

Create a Personal Access token for your Dockerhub account

Go to docker hub and click on your profile --> Account settings --> security --> New access token

It asks for a name Provide a name and click on generate token

Copy the token save it in a safe place, and close

Now Go to GitHub again and click on settings

Search for Secrets and variables and click on and again click on actions

It will open a page like this click on New Repository secret

Add your Dockerhub username with the secret name as

DOCKERHUB_USERNAME   #use your dockerhub username

Click on Add Secret.

Let's add our token also and click on the new repository secret again

Name

DOCKERHUB_TOKEN

Paste the token that you generated and click on Add secret.

Step4B: Create a TMDB API Key

Next, we will create a TMDB API key

Open a new tab in the Browser and search for TMDB

Click on the first result, you will see this page

Click on the Login on the top right. You will get this page.

You need to create an account here. click on click here. I have an account that's why I added my details there.

once you create an account you will see this page.

Let's create an API key, By clicking on your profile and clicking settings.

Now click on API from the left side panel.

Now click on create

Click on Developer

Now you have to accept the terms and conditions.

Provide basic details

Click on submit and you will get your API key.

Let's add the below step to the workflow

You have to add API at the Build command

Change your username also

- name: Docker build and push
  run: |
    #run commands to build and push docker images
    docker build --build-arg TMDB_V3_API_KEY=<USE YOUR API KEY> -t netflix .
    docker tag netflix hassanb111/netflix:${{ github.sha }}
    docker login -u ${{ secrets.DOCKERHUB_USERNAME }} -p ${{ secrets.DOCKERHUB_TOKEN }}
    docker push hassanb111/netflix:${{ github.sha }}
  env:
    DOCKER_CLI_ACI: 1

You can see the image, I already added

Let's commit changes

Click on actions again

It started the workflow build

Click on Build, Analyze and Scan. You will see this Docker image is building now

Build Succeeded

If you go to your Docker hub, you will find that the image is pushed to Dockerhub

Step5A: Add a self-hosted runner to Ec2

Go to GitHub and click on Settings --> Actions --> Runners

Click on New self-hosted runner

Now select Linux and Architecture X64

Use the below commands to add a self-hosted runner

Go to Putty or Mobaxtreme and connect to your ec2 instance

And paste the commands

NOTE: USE YOUR RUNNER COMMANDS (EXAMPLE CASE IAM USING MINE)

mkdir actions-runner && cd actions-runner

The command "mkdir actions-runner && cd actions-runner" is used to create a new directory called "actions-runner" in the current working directory and then immediately change the current working directory to the newly created "actions-runner" directory. This allows you to organize your files and perform subsequent actions within the newly created directory without having to navigate to it separately.

curl -o actions-runner-linux-x64-2.310.2.tar.gz -L https://github.com/actions/runner/releases/download/v2.310.2/actions-runner-linux-x64-2.310.2.tar.gz

This command downloads a file called "actions-runner-linux-x64-2.310.2.tar.gz" from a specific web address on GitHub and saves it in your current directory.

Let's validate the hash installation

echo "fb28a1c3715e0a6c5051af0e6eeff9c255009e2eec6fb08bc2708277fbb49f93  actions-runner-linux-x64-2.310.2.tar.gz" | shasum -a 256 -c

Now Extract the installer

tar xzf ./actions-runner-linux-x64-2.310.2.tar.gz

Let's configure the runner

./config.sh --url https://github.com/hassanb111/Netflix-clone --token A2MXW4323ALGB72GGLH34NLFGI2T4

Let's start runner

./run.sh

Step5B: Final workflow to run the container

Let's add a deployment workflow

deploy:    
    needs: build-analyze-scan  
    runs-on: [aws-netflix]  
    steps:
      - name: Pull the docker image
        run: docker pull hassanb111/netflix:${{ github.sha }}
      - name: Trivy image scan
        run: trivy image hassanb111/netflix:${{ github.sha }}
      - name: Run the container netflix
        run: docker run -d --name netflix -p 8081:80 hassanb111/netflix:${{ github.sha }}

1. deploy:: This is the name of a workflow or job, likely associated with a CI/CD pipeline. It specifies what should happen when this deployment job is triggered.

2. needs: build-analyze-scan: This line indicates that this deployment job depends on the successful completion of a previous job named "build-analyze-scan." In other words, it waits for "build-analyze-scan" to finish before starting.

3. runs-on: [aws-netflix]: This job is set to run on a specific type of runner or environment, labeled as "aws-netflix." Runners are the environments where jobs are executed, and "aws-netflix" suggests that this deployment might be intended for an AWS-based infrastructure.

4. steps:: This section lists the individual steps or tasks to be executed as part of the deployment job.

   • name: Pull the docker image: This step has a descriptive name. It uses the docker pull command to fetch a Docker image labeled "sevenajay/netflix:latest." This is a common step in container-based deployments, where it ensures that the latest version of the Docker image is available locally.

   • name: Trivy image scan: This step performs a security scan on the Docker image "sevenajay/netflix:latest" using a tool called Trivy. Trivy is used for vulnerability scanning of container images.

   • name: Run the container netflix: This step starts a Docker container named "netflix" using the image "sevenajay/netflix:latest." It runs the container in detached mode ("-d") and maps port 8081 on the host to port 80 in the container, making the service accessible via port 8081 on the host.

This workflow is designed to automate the deployment of a Docker container, with checks for the latest image, a security scan, and launching the container. The success of this job depends on the success of the preceding "build-analyze-scan" job, and it's meant to be executed on the specified runner, possibly in an AWS environment.

Commit changes

Click on actions again

You will see two different Jobs now

Click on Build and Push docker image Build (using the above image), you will see this once the first job completes

Now come back by clicking on Summary and click on Deploy now

You can see how it's pulling image and scanning image

It starts running the job on your Ec2 instance

Now it's completed running the container

You will see this in the instance

On GitHub, you will see this. the build succeeded

Now copy your ec2 instance ip and go to the browser

<Ec2-instance-ip:8081>

You will see Netflix app will run

Deployment is done.

FULL WORKFLOW

name: Build,Analyze,scan

on:
  push:
    branches:
      - main


jobs:
  build-analyze-scan:
    name: Build
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
        with:
          fetch-depth: 0  # Shallow clones should be disabled for a better relevancy of analysis

      - name: Build and analyze with SonarQube
        uses: sonarsource/sonarqube-scan-action@master
        env:
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
          SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}

      - name: install trivy
        run: |
           #install trivy
           sudo apt-get install wget apt-transport-https gnupg lsb-release -y
           wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg > /dev/null
           echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list
           sudo apt-get update
           sudo apt-get install trivy -y
           #scanning files
           trivy fs .

      - name: Docker build and push
        run: |
          #run commands to build and push docker images
          docker build --build-arg TMDB_V3_API_KEY=cc43e68ccd7edc1f4cbe88e891ad7059 -t netflix .
          docker tag netflix hassanb111/netflix:${{ github.sha }}
          docker login -u ${{ secrets.DOCKERHUB_USERNAME }} -p ${{ secrets.DOCKERHUB_TOKEN }}
          docker push hassanb111/netflix:${{ github.sha }}
        env:
          DOCKER_CLI_ACI: 1     

  deploy:    
    needs: build-analyze-scan  
    runs-on: [aws-netflix]  
    steps:
      - name: Pull the docker image
        run: docker pull hassanb111/netflix:${{ github.sha }}
      - name: Trivy image scans
        run: trivy image hassanb111/netflix:${{ github.sha }}
      - name: Run the container netflix
        run: docker run -d --name netflix -p 8081:80 hassanb111/netflix:${{ github.sha }}

Clear the instance.

I hope you found this blog insightful and that you've learned something new about how GitHub Actions can supercharge your Netflix deployments through the lens of DevSecOps. As technology evolves, staying informed and adaptable is key to thriving in the world of software development. If you have any questions or would like to share your thoughts, feel free to reach out. Your feedback and engagement are invaluable as we continue to explore and embrace the exciting innovations in the tech landscape. Thank you for joining me on this journey of discovery!

Phase 1: Initial Setup and Deployment

Step 1: Launch EC2 (Ubuntu 22.04):

• Provision an EC2 instance on AWS with Ubuntu 22.04.

• Connect to the instance using SSH.

Step 2: Clone the Code:

• Update all the packages and then clone the code.

• Clone your application's code repository onto the EC2 instance:

•     https://github.com/muhammadhassanb111/DevSecOps-Project.git
  

Step 3: Install Docker and Run the App Using a Container:

• Set up Docker on the EC2 instance:

•     sudo apt-get update
      sudo apt-get install docker.io -y
      sudo usermod -aG docker $USER  # Replace with your system's username, e.g., 'ubuntu'
      newgrp docker
      sudo chmod 777 /var/run/docker.sock
  

• Build and run your application using Docker containers:

•     docker build -t netflix .
      docker run -d --name netflix -p 8081:80 netflix:latest
  
      #to delete
      docker stop <containerid>
      docker rmi -f netflix
  

It will show an error cause you need API key

Step 4: Get the API Key:

• Open a web browser and navigate to TMDB (The Movie Database) website.

• Click on "Login" and create an account.

• Once logged in, go to your profile and select "Settings."

• Click on "API" from the left-side panel.

• Create a new API key by clicking "Create" and accepting the terms and conditions.

• Provide the required basic details and click "Submit."

• You will receive your TMDB API key.

Now recreate the Docker image with your api key:

docker build --build-arg TMDB_V3_API_KEY=<your-api-key> -t netflix .

Phase 2: Security

1. Install SonarQube and Trivy:

   • Install SonarQube and Trivy on the EC2 instance to scan for vulnerabilities.

     sonarqube

1. •      docker run -d --name sonar -p 9000:9000 sonarqube:lts-community
     

To access:

publicIP:9000 (by default username & password is admin)

To install Trivy:

sudo apt-get install wget apt-transport-https gnupg lsb-release
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update
sudo apt-get install trivy

to scan image using trivy

trivy image <imageid>

1. Integrate SonarQube and Configure:

   • Integrate SonarQube with your CI/CD pipeline.

   • Configure SonarQube to analyze code for quality and security issues.

Phase 3: CI/CD Setup

1. Install Jenkins for Automation:

   • Install Jenkins on the EC2 instance to automate deployment: Install Java

    sudo apt update
    sudo apt install fontconfig openjdk-17-jre
    java -version
    openjdk version "17.0.8" 2023-07-18
    OpenJDK Runtime Environment (build 17.0.8+7-Debian-1deb12u1)
    OpenJDK 64-Bit Server VM (build 17.0.8+7-Debian-1deb12u1, mixed mode, sharing)

    #jenkins
    sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
    https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
    echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
    https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
    /etc/apt/sources.list.d/jenkins.list > /dev/null
    sudo apt-get update
    sudo apt-get install jenkins
    sudo systemctl start jenkins
    sudo systemctl enable jenkins

1. • Access Jenkins in a web browser using the public IP of your EC2 instance.

     publicIp:8080

2. Install Necessary Plugins in Jenkins:

Goto Manage Jenkins →Plugins → Available Plugins →

Install below plugins

1 Eclipse Temurin Installer (Install without restart)

2 SonarQube Scanner (Install without restart)

3 NodeJs Plugin (Install Without restart)

4 Email Extension Plugin

Configure Java and Nodejs in Global Tool Configuration

Goto Manage Jenkins → Tools → Install JDK(17) and NodeJs(16)→ Click on Apply and Save

SonarQube

Create the token

Goto Jenkins Dashboard → Manage Jenkins → Credentials → Add Secret Text. It should look like this

After adding sonar token

Click on Apply and Save

The Configure System option is used in Jenkins to configure different server

Global Tool Configuration is used to configure different tools that we install using Plugins

We will install a sonar scanner in the tools.

Create a Jenkins webhook

1. Configure CI/CD Pipeline in Jenkins:

• Create a CI/CD pipeline in Jenkins to automate your application deployment.

pipeline {
    agent any
    tools {
        jdk 'jdk17'
        nodejs 'node16'
    }
    environment {
        SCANNER_HOME = tool 'sonar-scanner'
    }
    stages {
        stage('clean workspace') {
            steps {
                cleanWs()
            }
        }
        stage('Checkout from Git') {
            steps {
                git branch: 'main', url: 'https://github.com/muhammadhassanb111/DevSecOps-Project.git'
            }
        }
        stage("Sonarqube Analysis") {
            steps {
                withSonarQubeEnv('sonar-server') {
                    sh '''$SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Netflix \
                    -Dsonar.projectKey=Netflix'''
                }
            }
        }
        stage("quality gate") {
            steps {
                script {
                    waitForQualityGate abortPipeline: false, credentialsId: 'Sonar-token'
                }
            }
        }
        stage('Install Dependencies') {
            steps {
                sh "npm install"
            }
        }
    }
}

Certainly, here are the instructions without step numbers:

Install Dependency-Check and Docker Tools in Jenkins

Install Dependency-Check Plugin:

• Go to "Dashboard" in your Jenkins web interface.

• Navigate to "Manage Jenkins" → "Manage Plugins."

• Click on the "Available" tab and search for "OWASP Dependency-Check."

• Check the checkbox for "OWASP Dependency-Check" and click on the "Install without restart" button.

Configure Dependency-Check Tool:

• After installing the Dependency-Check plugin, you need to configure the tool.

• Go to "Dashboard" → "Manage Jenkins" → "Global Tool Configuration."

• Find the section for "OWASP Dependency-Check."

• Add the tool's name, e.g., "DP-Check."

• Save your settings.

Install Docker Tools and Docker Plugins:

• Go to "Dashboard" in your Jenkins web interface.

• Navigate to "Manage Jenkins" → "Manage Plugins."

• Click on the "Available" tab and search for "Docker."

• Check the following Docker-related plugins:

  • Docker

  • Docker Commons

  • Docker Pipeline

  • Docker API

  • docker-build-step

• Click on the "Install without restart" button to install these plugins.

Add DockerHub Credentials:

• To securely handle DockerHub credentials in your Jenkins pipeline, follow these steps:

  • Go to "Dashboard" → "Manage Jenkins" → "Manage Credentials."

  • Click on "System" and then "Global credentials (unrestricted)."

  • Click on "Add Credentials" on the left side.

  • Choose "Secret text" as the kind of credentials.

  • Enter your DockerHub credentials (Username and Password) and give the credentials an ID (e.g., "docker").

  • Click "OK" to save your DockerHub credentials.

Now, you have installed the Dependency-Check plugin, configured the tool, and added Docker-related plugins along with your DockerHub credentials in Jenkins. You can now proceed with configuring your Jenkins pipeline to include these tools and credentials in your CI/CD process.

pipeline{
    agent any
    tools{
        jdk 'jdk17'
        nodejs 'node16'
    }
    environment {
        SCANNER_HOME=tool 'sonar-scanner'
    }
    stages {
        stage('clean workspace'){
            steps{
                cleanWs()
            }
        }
        stage('Checkout from Git'){
            steps{
                git branch: 'main', url: 'https://github.com/muhammadhassanb111/DevSecOps-Project.git'
            }
        }
        stage("Sonarqube Analysis "){
            steps{
                withSonarQubeEnv('sonar-server') {
                    sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Netflix \
                    -Dsonar.projectKey=Netflix '''
                }
            }
        }
        stage("quality gate"){
           steps {
                script {
                    waitForQualityGate abortPipeline: false, credentialsId: 'Sonar-token' 
                }
            } 
        }
        stage('Install Dependencies') {
            steps {
                sh "npm install"
            }
        }
        stage('OWASP FS SCAN') {
            steps {
                dependencyCheck additionalArguments: '--scan ./ --disableYarnAudit --disableNodeAudit', odcInstallation: 'DP-Check'
                dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
            }
        }
        stage('TRIVY FS SCAN') {
            steps {
                sh "trivy fs . > trivyfs.txt"
            }
        }
        stage("Docker Build & Push"){
            steps{
                script{
                   withDockerRegistry(credentialsId: 'docker', toolName: 'docker'){   
                       sh "docker build --build-arg TMDB_V3_API_KEY=<yourapikey> -t netflix ."
                       sh "docker tag netflix hassanb111/netflix:latest "
                       sh "docker push hassanb111/netflix:latest "
                    }
                }
            }
        }
        stage("TRIVY"){
            steps{
                sh "trivy image hassanb111/netflix:latest > trivyimage.txt" 
            }
        }
        stage('Deploy to container'){
            steps{
                sh 'docker run -d --name netflix -p 8081:80 hassanb111/netflix:latest'
            }
        }
    }
}


If you get docker login failed errorr

sudo su
sudo usermod -aG docker jenkins
sudo systemctl restart jenkins

Phase 4: Monitoring

1. Install Prometheus and Grafana:

   Set up Prometheus and Grafana to monitor your application.

   Installing Prometheus:

   First, create a dedicated Linux user for Prometheus and download Prometheus:

1.   sudo useradd --system --no-create-home --shell /bin/false prometheus
     wget https://github.com/prometheus/prometheus/releases/download/v2.47.1/prometheus-2.47.1.linux-amd64.tar.gz
   

Extract Prometheus files, move them, and create directories:

tar -xvf prometheus-2.47.1.linux-amd64.tar.gz
cd prometheus-2.47.1.linux-amd64/
sudo mkdir -p /data /etc/prometheus
sudo mv prometheus promtool /usr/local/bin/
sudo mv consoles/ console_libraries/ /etc/prometheus/
sudo mv prometheus.yml /etc/prometheus/prometheus.yml

Set ownership for directories:

sudo chown -R prometheus:prometheus /etc/prometheus/ /data/

Create a systemd unit configuration file for Prometheus:

sudo nano /etc/systemd/system/prometheus.service

Add the following content to the prometheus.service file:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
User=prometheus
Group=prometheus
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/data \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.enable-lifecycle

[Install]
WantedBy=multi-user.target

Here's a brief explanation of the key parts in this prometheus.service file:

• User and Group specify the Linux user and group under which Prometheus will run.

• ExecStart is where you specify the Prometheus binary path, the location of the configuration file (prometheus.yml), the storage directory, and other settings.

• web.listen-address configures Prometheus to listen on all network interfaces on port 9090.

• web.enable-lifecycle allows for management of Prometheus through API calls.

Enable and start Prometheus:

sudo systemctl enable prometheus
sudo systemctl start prometheus

Verify Prometheus's status:

sudo systemctl status prometheus

You can access Prometheus in a web browser using your server's IP and port 9090:

http://<your-server-ip>:9090

Installing Node Exporter:

Create a system user for Node Exporter and download Node Exporter:

sudo useradd --system --no-create-home --shell /bin/false node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz

Extract Node Exporter files, move the binary, and clean up:

tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz
sudo mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
rm -rf node_exporter*

Create a systemd unit configuration file for Node Exporter:

sudo nano /etc/systemd/system/node_exporter.service

Add the following content to the node_exporter.service file:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/node_exporter --collector.logind

[Install]
WantedBy=multi-user.target

Replace --collector.logind with any additional flags as needed.

Enable and start Node Exporter:

sudo systemctl enable node_exporter
sudo systemctl start node_exporter

Verify the Node Exporter's status:

sudo systemctl status node_exporter

• You can access Node Exporter metrics in Prometheus.

• Configure Prometheus Plugin Integration:

  Integrate Jenkins with Prometheus to monitor the CI/CD pipeline.

  Prometheus Configuration:

  To configure Prometheus to scrape metrics from Node Exporter and Jenkins, you need to modify the prometheus.yml file. Here is an example prometheus.yml configuration for your setup:

•     global:
        scrape_interval: 15s
  
      scrape_configs:
        - job_name: 'node_exporter'
          static_configs:
            - targets: ['localhost:9100']
  
        - job_name: 'jenkins'
          metrics_path: '/prometheus'
          static_configs:
            - targets: ['<your-jenkins-ip>:<your-jenkins-port>']
  

Make sure to replace <your-jenkins-ip> and <your-jenkins-port> with the appropriate values for your Jenkins setup.

Check the validity of the configuration file:

promtool check config /etc/prometheus/prometheus.yml

Reload the Prometheus configuration without restarting:

curl -X POST http://localhost:9090/-/reload

1. You can access Prometheus targets at:

   http://<your-prometheus-ip>:9090/targets

####Grafana

Install Grafana on Ubuntu 22.04 and Set it up to Work with Prometheus

Step 1: Install Dependencies:

First, ensure that all necessary dependencies are installed:

sudo apt-get update
sudo apt-get install -y apt-transport-https software-properties-common

Step 2: Add the GPG Key:

Add the GPG key for Grafana:

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

Step 3: Add Grafana Repository:

Add the repository for Grafana stable releases:

echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Step 4: Update and Install Grafana:

Update the package list and install Grafana:

sudo apt-get update
sudo apt-get -y install grafana

Step 5: Enable and Start Grafana Service:

To automatically start Grafana after a reboot, enable the service:

sudo systemctl enable grafana-server

Then, start Grafana:

sudo systemctl start grafana-server

Step 6: Check Grafana Status:

Verify the status of the Grafana service to ensure it's running correctly:

sudo systemctl status grafana-server

Step 7: Access Grafana Web Interface:

Open a web browser and navigate to Grafana using your server's IP address. The default port for Grafana is 3000. For example:

http://<your-server-ip>:3000

You'll be prompted to log in to Grafana. The default username is "admin," and the default password is also "admin."

Step 8: Change the Default Password:

When you log in for the first time, Grafana will prompt you to change the default password for security reasons. Follow the prompts to set a new password.

Step 9: Add Prometheus Data Source:

To visualize metrics, you need to add a data source. Follow these steps:

• Click on the gear icon (⚙️) in the left sidebar to open the "Configuration" menu.

• Select "Data Sources."

• Click on the "Add data source" button.

• Choose "Prometheus" as the data source type.

• In the "HTTP" section:

  • Set the "URL" to http://localhost:9090 (assuming Prometheus is running on the same server).

  • Click the "Save & Test" button to ensure the data source is working.

Step 10: Import a Dashboard:

To make it easier to view metrics, you can import a pre-configured dashboard. Follow these steps:

• Click on the "+" (plus) icon in the left sidebar to open the "Create" menu.

• Select "Dashboard."

• Click on the "Import" dashboard option.

• Enter the dashboard code you want to import (e.g., code 1860).

• Click the "Load" button.

• Select the data source you added (Prometheus) from the dropdown.

• Click on the "Import" button.

You should now have a Grafana dashboard set up to visualize metrics from Prometheus.

Grafana is a powerful tool for creating visualizations and dashboards, and you can further customize it to suit your specific monitoring needs.

That's it! You've successfully installed and set up Grafana to work with Prometheus for monitoring and visualization.

1. Configure Prometheus Plugin Integration:

   • Integrate Jenkins with Prometheus to monitor the CI/CD pipeline.

Phase 5: Notification

1. Implement Notification Services:

   • Set up email notifications in Jenkins or other notification mechanisms.

Phase 6: Kubernetes

Create Kubernetes Cluster with Nodegroups

In this phase, you'll set up a Kubernetes cluster with node groups. This will provide a scalable environment to deploy and manage your applications.

Monitor Kubernetes with Prometheus

Prometheus is a powerful monitoring and alerting toolkit, and you'll use it to monitor your Kubernetes cluster. Additionally, you'll install the node exporter using Helm to collect metrics from your cluster nodes.

Install Node Exporter using Helm

To begin monitoring your Kubernetes cluster, you'll install the Prometheus Node Exporter. This component allows you to collect system-level metrics from your cluster nodes. Here are the steps to install the Node Exporter using Helm:

1. Add the Prometheus Community Helm repository:

1.   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   

Create a Kubernetes namespace for the Node Exporter:

kubectl create namespace prometheus-node-exporter

Install the Node Exporter using Helm:

helm install prometheus-node-exporter prometheus-community/prometheus-node-exporter --namespace prometheus-node-exporter

Add a Job to Scrape Metrics on nodeip:9001/metrics in prometheus.yml:

Update your Prometheus configuration (prometheus.yml) to add a new job for scraping metrics from nodeip:9001/metrics. You can do this by adding the following configuration to your prometheus.yml file:

  - job_name: 'Netflix'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['node1Ip:9100']

Replace 'your-job-name' with a descriptive name for your job. The static_configs section specifies the targets to scrape metrics from, and in this case, it's set to nodeip:9001.

Don't forget to reload or restart Prometheus to apply these changes to your configuration.

To deploy an application with ArgoCD, you can follow these steps, which I'll outline in Markdown format:

Deploy Application with ArgoCD

1. Install ArgoCD:

   You can install ArgoCD on your Kubernetes cluster by following the instructions provided in the EKS Workshop documentation.

2. Set Your GitHub Repository as a Source:

   After installing ArgoCD, you need to set up your GitHub repository as a source for your application deployment. This typically involves configuring the connection to your repository and defining the source for your ArgoCD application. The specific steps will depend on your setup and requirements.

3. Create an ArgoCD Application:

   • name: Set the name for your application.

   • destination: Define the destination where your application should be deployed.

   • project: Specify the project the application belongs to.

   • source: Set the source of your application, including the GitHub repository URL, revision, and the path to the application within the repository.

   • syncPolicy: Configure the sync policy, including automatic syncing, pruning, and self-healing.

4. Access your Application

   • To Access the app make sure port 30007 is open in your security group and then open a new tab paste your NodeIP:30007, your app should be running.

Phase 7: Cleanup

1. Cleanup AWS EC2 Instances:

   • Terminate AWS EC2 instances that are no longer needed.

https://github.com/muhammadhassanb111/solar-system

hashtag#nodejs hashtag#kubernetes hashtag#githubactions hashtag#devops

Picture this: you're creating a YouTube-like platform where users can explore videos and channels. It's a fantastic project, but managing the development workflow can be complex. That's where GitLab CI/CD comes in, offering a straightforward and powerful solution to automate your work.

This step-by-step tutorial is for developers and tech enthusiasts looking to make their projects more efficient with CI/CD. By the end, you'll know how to set up a robust CI/CD pipeline, including automated testing, code quality checks, and secure containerization. Let's embark on this journey together and see how CI/CD can transform your development experience while building a React YouTube app clone.

Ready? Let's simplify CI/CD and make your development life easier!

YouTube video:

https://youtu.be/YHPvCB3IQdI

Step 1: Create an API key for Youtube.

Step 2: Create a Repository and push it to GitLab.

Step 3: Launch an Ec2 instance and run Sonarqube on it.

Step 4A: Create a .gitlab-ci.yml File.

Step 4B: Add the required variables for the project.

Step 5: Install Gitlab Runner on Ec2.

Step 6: Run the Application on the Docker container.

Step 7: Access the Application on Browser.

Step 1: Create an API key from Rapid API

Open a new tab in the browser and search for rapidapi.com

It will automatically provide your mail and select a mail to create an account

Account is created

Now in the search bar search for YouTube and select YouTube v3

Copy API and save it for further use at the docker stage.

docker build --build-arg REACT_APP_RAPID_API_KEY=<API-KEY> -t ${imageName} .

Second way:

Open a new tab in the browser and search for rapidapi.com

You will see the page like this and click on signUp

Now click on Sign Up with Google

Select your mail here

It will automatically create your account now

Now in the search bar search for YouTube and select YouTube v3

Copy API and save it for further use at the docker stage.

docker build --build-arg REACT_APP_RAPID_API_KEY=<API-KEY> -t ${imageName} .

Step 2: Create a Repository and push it to GitLab

Go to GitLab.com and login to your account

Click on New Project

Click on Create Blank Project

Provide a name for the Project

Keep Visibility to the public

Uncheck the Readme and create the Project.

Use the below commands to push code to GitLab

Files pushed to GitLab

Step 3: Launch an Ec2 instance and run Sonarqube on it

Log into AWS Console: Sign in to your AWS account.

Launch an Instance:

Choose "EC2" from services. Click "Launch Instance."

Choose an AMI: Select an Ubuntu image.

Choose an Instance Type: Pick "t2.medium."

Key Pair: Choose an existing key pair or create a new one.

Configure Security Group:

Create a new security group. Add rules for HTTP, and HTTPS, and open all ports for learning purposes. Add Storage: Allocate at least 10 GB of storage.

Launch Instance: Review and launch the instance.

Access Your Instance: Use SSH to connect to your instance with the private key.

Keep in mind, that opening all ports is not recommended for production environments; it's just for educational purposes.

Connect to Your EC2 Instance and install docker:

Run the below commands to install the docker

sudo apt-get update
sudo apt-get install docker.io -y
sudo usermod -aG docker $USER   #my case is ubuntu
newgrp docker
sudo chmod 777 /var/run/docker.sock

After the docker installation, we will create a Sonarqube container (Remember to add 9000 ports in the security group).

Run this command on your EC2 instance to create a SonarQube container:

docker run -d --name sonar -p 9000:9000 sonarqube:lts-community

Now copy the IP address of the ec2 instance

<ec2-public-ip:9000>

Enter username and password, click on login and change password

username admin
password admin

Update New password, This is Sonar Dashboard.

Step 4A: Create a .gitlab-ci.yml File.

Now go to GitLab click on '+' and click on Newfile

File name .gitlab-ci.yml

Content

stages:
    - npm

Install dependecy:
    stage: npm    
    image:
        name: node:16
    script:
        - npm install

Commit the changes and it will automatically start the build

Now click on Build and Pipelines

Now click on Running.

Click on Install dependency

You will build output

Now add the Sonarqube stage to the pipeline

Go to the Sonarqube dashboard and click on Manually.

Provide the name of the Project and click on Setup

Select the CI tool as GitLab CI

Select Other because we are using the JS App

It will provide code and we need to create a file inside our repo

Go to Gitlab and click on + and Newfile

Filename is sonar-project.properties

Paste the content that you got from Sonarqube

The file looks like this and click on commit changes

Go to Sonarqube and click on continue

Now it will provide Variables to add to our GitLab

Step 4B: Add the required variables for the project.

Variables Generated

Now go to GitLab

Click on settings and CI/CD

Click on Expand in variables

Click on Add variable

Now go back to Sonarqube and copy the Key

Click on Generate a token

Again Click on Generate

Copy the token

Now come back to GitLab and add them like the below image and click on add variable.

Sonar token is added

Now go to the Sonarqube Dashboard again

Let's add another variable, copy them

Now go to GitLab and click on Add variable

Add the copied values like the below image

Two variables were added.

Now go back to the Sonarqube Dashboard

Click on continue

It will provide and CI configuration file copy it and use it inside our .gitlab-ci.yml file

Now go back to GitLab and edit the .gitlab-ci.yml file

Full file (update with your content)

stages:
    - npm
    - sonar

Install dependecy:
    stage: npm    
    image:
        name: node:16
    script:
        - npm install    

sonarqube-check:
  stage: sonar
  image: 
    name: sonarsource/sonar-scanner-cli:latest
    entrypoint: [""]
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"  # Defines the location of the analysis task cache
    GIT_DEPTH: "0"  # Tells git to fetch all the branches of the project, required by the analysis task
  cache:
    key: "${CI_JOB_NAME}"
    paths:
      - .sonar/cache
  script: 
    - sonar-scanner
  allow_failure: true
  only:
    - main

Commit changes and it will automatically start the build.

Click on Build --> Pipelines

Click on Running

Now click on Sonarqube-check

Build output

Now add the next stage of the Trivy file scan

Update the .gitlab-ci.yml file

stages:
    - npm
    - sonar
    - trivy file scan

Install dependecy:
    stage: npm    
    image:
        name: node:16
    script:
        - npm install    

sonarqube-check:
  stage: sonar
  image: 
    name: sonarsource/sonar-scanner-cli:latest
    entrypoint: [""]
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"  # Defines the location of the analysis task cache
    GIT_DEPTH: "0"  # Tells git to fetch all the branches of the project, required by the analysis task
  cache:
    key: "${CI_JOB_NAME}"
    paths:
      - .sonar/cache
  script: 
    - sonar-scanner
  allow_failure: true
  only:
    - main

Trivy file scan:
  stage: trivy file scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    - trivy fs .

Commit changes and go to pipeline stages

Click on the Trivy file scan

Build output

Add the Docker build and push stage

Before that Add docker credentials to GitLab Variables as secrets.

Go to the docker hub and create a Personal Access token

Click on your profile name and Account Settings

Now click on Security --> New Access Token

Provide a name --> Generate

Now copy the token and keep it safe

Now go back to Gitlab

Click on settings and CI/CD

Click on Expand in variables

Click on Add variable

Use your DockerHub username in value and Add variable

Key DOCKER_USERNAME

Click on Add variable again

Key DOCKER_PASSWORD

For value use the Generated Personal Access token and add a variable.

Variables added.

Now add the below stage to the Configuration .gitlab-ci.yml file

Added Docker and Trivy image scan stages

stages:
    - npm
    - sonar
    - trivy file scan
    - docker
    - trivy image scan

Install dependecy:
    stage: npm    
    image:
        name: node:16
    script:
        - npm install    

sonarqube-check:
  stage: sonar
  image: 
    name: sonarsource/sonar-scanner-cli:latest
    entrypoint: [""]
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"  # Defines the location of the analysis task cache
    GIT_DEPTH: "0"  # Tells git to fetch all the branches of the project, required by the analysis task
  cache:
    key: "${CI_JOB_NAME}"
    paths:
      - .sonar/cache
  script: 
    - sonar-scanner
  allow_failure: true
  only:
    - main

Trivy file scan:
  stage: trivy file scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    - trivy fs . 

Docker build and push:
  stage: docker
  image:
    name: docker:latest
  services:
    - docker:dind   
  script:
    - docker build --build-arg REACT_APP_RAPID_API_KEY=f0ead79813mshb0aa7ddf114a7dap1adb3djsn483b017de1a9 -t youtubev1 .    
    - docker tag youtubev1 hassanb111/youtubev1:latest
    - docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
    - docker push hassanb111/youtubev1:latest

Scan image:
  stage: trivy image scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    - trivy image hassanb111/youtubev1:latest

Added stages

Commit changes and it will automatically start building.

Go to Pipelines view

Now click on Docker build and push

Build view

Go to Dockerhub and see the image

Now come back to GitLab and click on Trivy image scan

Output raw

Step 5: Install Gitlab Runner on Ec2

Go to GitLab and Click on Settings and CI/CD

Click on Expand at Runners

Click on Three dots and then click on Show Runner installation

Click on Linux and amd64 and copy the commands

Now come back to Putty or Mobaxtreme

Create a new file

sudo vi gitlab-runner-installation

Paste the below commands into it

# Download the binary for your system
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64

# Give it permission to execute
sudo chmod +x /usr/local/bin/gitlab-runner

# Create a GitLab Runner user
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash

# Install and run as a service
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start

Provide executable permissions and run the script

sudo chmod +x <file-name>
./<file-name>

Installation completed

Start the GitLab runner

sudo gitlab-runner start

Now run the below command or your command to register the runner

Update the token is enough

sudo gitlab-runner register --url https://gitlab.com/ --registration-token <token>

Provide the details for registering the runner

1. Provide Enter for GitLab.com

2. For token we already added with token, so click on Enter again

3. Description as your wish

4. Tags also and you can use multiple tags by providing a comma after each tga

5. Maintenance note is just optional

6. For executors use Shell

Runner added successfully.

Start the GitLab runner

sudo gitlab-runner start

Run the GitLab runner

sudo gitlab-runner run

Go to GitLab and refresh the page once or click on Enable for this project

Now the runner is active and waiting for jobs

Click on the Pencil mark to edit

Click on the Check box to indicate whether this runner can pick jobs without tags.

Click on save changes.

Step 6: Run the Application on the Docker container

Now edit the .gitlab-ci.yml file for the deploy stage

The complete file

stages:
    - npm
    - sonar
    - trivy file scan
    - docker
    - trivy image scan
    - run container

Install dependecy:
    stage: npm    
    image:
        name: node:16
    script:
        - npm install    

sonarqube-check:
  stage: sonar
  image: 
    name: sonarsource/sonar-scanner-cli:latest
    entrypoint: [""]
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"  # Defines the location of the analysis task cache
    GIT_DEPTH: "0"  # Tells git to fetch all the branches of the project, required by the analysis task
  cache:
    key: "${CI_JOB_NAME}"
    paths:
      - .sonar/cache
  script: 
    - sonar-scanner
  allow_failure: true
  only:
    - main

Trivy file scan:
  stage: trivy file scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    - trivy fs . 

Docker build and push:
  stage: docker
  image:
    name: docker:latest
  services:
    - docker:dind   
  script:
    - docker build --build-arg REACT_APP_RAPID_API_KEY=f0ead79813mshb0aa7ddf114a7dap1adb3djsn483b017de1a9 -t youtubev1 .    
    - docker tag youtubev1 hassanb111/youtubev1:latest
    - docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
    - docker push hassanb111/youtubev1:latest

Scan image:
  stage: trivy image scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    - trivy image hassanb111/youtubev1:latest

deploy:
  stage: run container
  tags:
    - youtube        #use your own tags 
  script:
    - docker run -d --name youtube -p 3000:3000 hassanb111/youtubev1:latest

Commit changes, it will automatically start to build

Click on Build --> Pipelines

Click on Running

The last stage is added to the Pipeline

If you get an error like this

Go to GitLab and click on deploy job

Let's see what is the error

If you get an error like this, click on that link

It will open a new tab and provide a solution for that

Now go to Mobaxtreme and stop the Runner

Go to root and use the below commands

sudo su
sudo vi /home/gitlab-runner/.bash_logout

You will see file like this

Comment them

Save and exit from that file and restart GitLab runner

sudo gitlab-runner restart
exit #from root

Now start and Run the GitLab runner

sudo gitlab-runner start
sudo gitlab-runner run

Now go to GitLab --> Build --> Pipelines

Click on Run Pipeline

Again Click on Run Pipeline

Build completed and click on Deploy job

See the output it ran a container on ec2

Now go to MobaXtreme or Putty and Provide the below command to see running containers.

docker ps

Container running.

Step 7: Access the Application on Browser

Copy the Public IP of the ec2 instance and paste it into the Browser.

Don't forget to open the 3000 port in the Security Group

<Public-ip:3000>

Step 8: Termination

1. Delete the personal Access token of the docker.

2. Delete the containers.

1.     docker stop <container name>
       docker rm <container name>
   

2. Delete the Ec2 instance.

And there you have it, folks! You're now the DevOps master of your YouTube app deployment universe. We've journeyed through code, quality, security, containers, and automation, all with GitLab as our trusty sidekick. Now it's your turn to unleash the power of DevOps and take your YouTube app to the next level!

So, what's next on your development adventure? Whether it's conquering new projects, exploring even cooler tech, or simply celebrating with a well-deserved coffee break, remember that DevOps is all about making your life easier and your code better. Embrace the joy of seamless deployments, keep learning, and never stop having fun in the ever-evolving world of software development. 🚀😎

If you enjoyed this Blog, don't forget to hit that like button, share with your fellow tech enthusiasts, and subscribe for more exciting DevOps adventures. And as always, stay curious, stay creative, and keep coding with a smile! 😄✨

Jenkins Pipeline Overview:

Our Jenkins pipeline consists of several stages, each serving a specific purpose:

1. Code Stage:

    groovyCopy codestage('code') {
        steps {
            git url: 'https://github.com/muhammadhassanb111/node-todo-cicd.git', branch: 'master'
        }
    }
   

   In this stage, we fetch the latest code from our GitHub repository, ensuring that we're always working with the most up-to-date version of our application.

2. Build and Test Stage:

    groovyCopy codestage('build and test') {
        steps {
            script {
                echo '=== Building Docker Image ==='
                sh 'docker build . -t hassanb111/node-todo-app:latest'
                echo '=== Docker Image Built ==='
   
                echo '=== Docker Images List ==='
                sh 'docker images'
                echo '=== End of Docker Images List ==='
            }
        }
    }
   

   In this stage, we build a Docker image from our application's Dockerfile. The process is accompanied by informative messages, and we list the Docker images for visibility.

3. Login and Push Images Stage:

    groovyCopy codestage('login and push images') {
        steps {
            echo 'Logging into Docker Hub'
            withCredentials([usernamePassword(credentialsId: 'dockerhub', passwordVariable: 'dockerhubpassword', usernameVariable: 'dockerhubuser')]) {
                sh "docker login -u ${env.dockerhubuser} -p ${env.dockerhubpassword}"
                sh "docker push hassanb111/node-todo-app:latest"
            }
        }
    }
   

   This stage handles the secure login to Docker Hub using Jenkins credentials and subsequently pushes the Docker image to the Docker Hub repository.

4. Deploy Stage:

    groovyCopy codestage('deploy') {
        steps {
            script {
                echo '=== Stopping and Removing Containers ==='
                sh 'docker-compose down'
                echo '=== Containers Stopped and Removed ==='
   
                echo '=== Starting Containers ==='
                sh 'docker-compose up -d'
                echo '=== Containers Started ==='
            }
        }
    }
   

   The final stage involves stopping and removing existing Docker containers and then initiating fresh containers using Docker Compose.

   

By breaking down our CI/CD process into these stages, we ensure a systematic and automated approach to software development. This Jenkins pipeline, combined with Docker, provides a powerful foundation for building, testing, and deploying applications efficiently.