End-to-End Bank Application Deployment using DevSecOps on AWS EKS

• This is a multi-tier bank an application written in Java (Springboot).

Tech stack used in this project:

• GitHub (Code)
• Docker (Containerization)
• Jenkins (CI)
• OWASP (Dependency check)
• SonarQube (Quality)
• Trivy (Filesystem Scan)
• ArgoCD (CD)
• AWS EKS (Kubernetes)
• Helm (Monitoring using grafana and prometheus)

Steps to deploy:

Pre-requisites:

• root user access

sudo su

Note

This project will be implemented on North California region (us-west-1).

• Create 1 Master machine on AWS (t2.medium) and 29 GB of storage.

• Open the below ports in security group

• Create EKS Cluster on AWS

• IAM user with access keys and secret access keys

• AWSCLI should be configured (Setup AWSCLI)

  curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
  sudo apt install unzip
  unzip awscliv2.zip
  sudo ./aws/install
  aws configure

• Install kubectl(Setup kubectl )

  curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.19.6/2021-01-05/bin/linux/amd64/kubectl
  chmod +x ./kubectl
  sudo mv ./kubectl /usr/local/bin
  kubectl version --short --client

• Install eksctl(Setup eksctl)

  curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
  sudo mv /tmp/eksctl /usr/local/bin
  eksctl version

• Create EKS Cluster

  eksctl create cluster --name=bankapp \
                      --region=us-west-1 \
                      --version=1.30 \
                      --without-nodegroup

• Associate IAM OIDC Provider

  eksctl utils associate-iam-oidc-provider \
    --region us-west-1 \
    --cluster bankapp \
    --approve

• Create Nodegroup

  eksctl create nodegroup --cluster=bankapp \
                       --region=us-west-1 \
                       --name=bankapp \
                       --node-type=t2.medium \
                       --nodes=2 \
                       --nodes-min=2 \
                       --nodes-max=2 \
                       --node-volume-size=29 \
                       --ssh-access \
                       --ssh-public-key=eks-nodegroup-key 

Note

Make sure the ssh-public-key "eks-nodegroup-key is available in your aws account"

• Install Jenkins

sudo apt update -y
sudo apt install fontconfig openjdk-17-jre -y

sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
  
echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \
  https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
  
sudo apt-get update -y
sudo apt-get install jenkins -y

• After installing Jenkins, change the default port of jenkins from 8080 to 8081. Because our bankapp application will be running on 8080.
  • Open /usr/lib/systemd/system/jenkins.service file and change JENKINS_PORT environment variable
  • Reload daemon

  sudo systemctl daemon-reload 

  • Restart Jenkins

  sudo systemctl restart jenkins

• Install docker

sudo apt install docker.io -y
sudo usermod -aG docker ubuntu && newgrp docker

• Install and configure SonarQube

docker run -itd --name SonarQube-Server -p 9000:9000 sonarqube:lts-community

• Install Trivy

sudo apt-get install wget apt-transport-https gnupg lsb-release -y
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update -y
sudo apt-get install trivy -y

• Install and Configure ArgoCD
  • Create argocd namespace

  kubectl create namespace argocd

  • Apply argocd manifest

  kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

  • Make sure all pods are running in argocd namespace

  watch kubectl get pods -n argocd

  • Install argocd CLI

  curl --silent --location -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/download/v2.4.7/argocd-linux-amd64

  • Provide executable permission

  chmod +x /usr/local/bin/argocd

  • Check argocd services

  kubectl get svc -n argocd

  • Change argocd server's service from ClusterIP to NodePort

  kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "NodePort"}}'

  • Confirm service is patched or not

  kubectl get svc -n argocd

  • Check the port where ArgoCD server is running and expose it on security groups of a k8s worker node
  • Access it on browser, click on advance and proceed with

  <public-ip-worker>:<port>

  
  • Fetch the initial password of argocd server

  kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

  • Username: admin
  • Now, go to User Info and update your argocd password

Steps to add email notification

• Go to your Jenkins Master EC2 instance and allow 465 port number for SMTPS

• Now, we need to generate an application password from our gmail account to authenticate with jenkins
  • Open gmail and go to Manage your Google Account --> Security

Important

Make sure 2 step verification must be on

• Search for App password and create a app password for jenkins

• Once, app password is create and go back to jenkins Manage Jenkins --> Credentials to add username and password for email notification

• Go back to Manage Jenkins --> System and search for Extended E-mail Notification

• Scroll down and search for E-mail Notification and setup email notification

Important

Enter your gmail password which we copied recently in password field E-mail Notification --> Advance

• Go to Jenkins and click on Manage Jenkins --> Plugins --> Available plugins install the below plugins:
  • OWASP
  • SonarQube Scanner
  • Docker
  • Pipeline: Stage View

• Configure OWASP, move to Manage Jenkins --> Plugins --> Available pluginsb>

• After OWASP plugin is installed, Now move to Manage jenkins --> Tools

• Login to SonarQube server and create the credentials for jenkins to integrate with SonarQube
  • Navigate to Administration --> Security --> Users --> Token

• Now, go to Manage Jenkins --> credentials and add Sonarqube credentials:

• Go to Manage Jenkins --> Tools and search for SonarQube Scanner installations:

• Go to Manage Jenkins --> credentials and add Docker credentials to push updated the updated docker image to dockerhub.

• Again, add Github credentials to push updated code from the pipeline:

Note

While adding github credentials add Personal Access Token in the password field.

• Go to Manage Jenkins --> System and search for SonarQube installations:

• Now again, Go to Manage Jenkins --> System and search for Global Trusted Pipeline Libraries:</b

• Login to SonarQube server, go to Administration --> Webhook and click on create

• Go to Master Machine and add our own eks cluster to argocd for application deployment using cli
  • Login to argoCD from CLI

   argocd login 52.53.156.187:32738 --username admin

Tip

52.53.156.187:32738 --> This should be your argocd url

• Check how many clusters are available in argocd

argocd cluster list

• Get your cluster name

kubectl config get-contexts

• Add your cluster to argocd

argocd cluster add [email protected] --name bankapp-eks-cluster

Tip

[email protected] --> This should be your EKS Cluster Name.

• Once your cluster is added to argocd, go to argocd console Settings --> Clusters and verify it

• Go to Settings --> Repositories and click on Connect repo

Note

Connection should be successful

• Create BankApp-CI job

• Create BankApp-CD job, same as CI job.

• Provide permission to docker socket so that docker build and push command do not fail

chmod 777 /var/run/docker.sock

• Now, go to Applications and click on New App

Important

Make sure to click on the Auto-Create Namespace option while creating argocd application

• Congratulations, your application is deployed on AWS EKS Cluster

• Open port 30080 on worker node and Access it on browser

<worker-public-ip>:30080

• Email Notification

How to monitor EKS cluster, kubernetes components and workloads using prometheus and grafana via HELM (On Master machine)

• Install Helm Chart

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3

chmod 700 get_helm.sh

./get_helm.sh

• Add Helm Stable Charts for Your Local Client

helm repo add stable https://charts.helm.sh/stable

• Add Prometheus Helm Repository

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

• Create Prometheus Namespace

kubectl create namespace prometheus

kubectl get ns

• Install Prometheus using Helm

helm install stable prometheus-community/kube-prometheus-stack -n prometheus

• Verify prometheus installation

kubectl get pods -n prometheus

• Check the services file (svc) of the Prometheus

kubectl get svc -n prometheus

• Expose Prometheus and Grafana to the external world through Node Port

Important

change it from Cluster IP to NodePort after changing make sure you save the file and open the assigned nodeport to the service.

kubectl edit svc stable-kube-prometheus-sta-prometheus -n prometheus

• Verify service

kubectl get svc -n prometheus

• Now,let’s change the SVC file of the Grafana and expose it to the outer world

kubectl edit svc stable-grafana -n prometheus

• Check grafana service

kubectl get svc -n prometheus

• Get a password for grafana

kubectl get secret --namespace prometheus stable-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Note

Username: admin

• Now, view the Dashboard in Grafana

Clean Up

• Delete eks cluster

eksctl delete cluster --name=bankapp --region=us-west-1