Prometheus & Grafana

aws configure 
AKIA2NK3Yxxxxxxxxxxxxxxxxxxxxxxxx
/KK3WHDeeY1xxxxxxxxxxxxxxxxxxxxxx


- LATENCY
- TRAFFIC - NO OF REQUEST 
- ERRORS - PAGE TIME OUT , ERROR CODES 
- SATURATION LIMITS - CPU 80 

VALUES ( KEY - VALUE PAIR )  ( TIME - VALUE) (CRONOLOGICAL EVENT - TIME SERIES DATA) -- SHOUD KEEP IN TSDB ( PROMETHEUS) 
INFACT EVERY TRACEBLE PARAMETER U CAN MONITOR

--------

NOW USING MONITORING , WE CAN DETECT PROBLEM , BUT WHAT IS ROOT CAUSE . NOW OBSERVABILITY COMES INTO PICTURE .

CHECK -- LOGS 
      -- METRICS
	  -- TRACE 

--------------------------------

1. eks cluster --

  eksctl create cluster bat13
  
  (prerequisites -- awscli , eksctl , kubectl ...) 
  
 ==================================================================================
  
  
kubectl Installation on Linux :

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version

--------------------------------

eksctl installation on aws linux machine 

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
eksctl version

==============================================

  
3. heml Installation :

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

========================================================
ADD & UPDATE THE PROMETHEUS-COMMUNITY HELM REPOSITORY :
 
helm repo add stable https://charts.helm.sh/stable

Add Prometheus helm repo:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts


helm search repo prometheus-community

--------------
Update helm repo:

helm repo update
--------------------------------------------------------

kubectl create namespace prometheus

helm install stable prometheus-community/kube-prometheus-stack -n prometheus
(If error comes , try with with below command) 


helm install stable prometheus-community/kube-prometheus-stack -n prometheus --set prometheus.prometheusSpec.maximumStartupDurationSeconds=600

Expose Prometheus Service & Grafana service to Load balancer and access from browser. 
Port numbe for Prometheus : 9090
Port numbe for Grafana : 80


This is required to access prometheus-server using your browser.

================================
22 MAY 25:

MANUAL SETUP :

Prometheus setup:


wget https://github.com/prometheus/prometheus/releases/download/v2.53.1/prometheus-2.53.1.linux-amd64.tar.gz
tar -xvf prometheus-2.53.1.linux-amd64.tar.gz
cd prometheus-2.53.1.linux-amd64/
./prometheus &


================================================================================
grafana setup:


wget https://dl.grafana.com/enterprise/release/grafana-enterprise-11.1.0.linux-amd64.tar.gz
tar -xvf grafana-enterprise-11.1.0.linux-amd64.tar.gz
cd grafana-enterprise-11.1.0/
cd bin/
./grafana-server &



========================================================================

Node exporter :


wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar -xvf node_exporter-1.7.0.linux-amd64.tar.gz


sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin

sudo useradd --system --no-create-home --shell /bin/false node_exporter
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
sudo chmod +x /usr/local/bin/node_exporter

========================================================================================

node_exporter --version

node_exporter, version 1.7.0 (branch: HEAD, revision: 7333465abf9efba81876303bb57e6fadb946041b)
  build user:       root@35918982f6d8
  build date:       20231112-23:53:35
  go version:       go1.21.4
  platform:         linux/amd64
  tags:             netgo osusergo static_build

------------------
SET UP SERVICE FOR NODE EXPORTER :

sudo vim /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/node_exporter \
  --collector.logind

[Install]
WantedBy=multi-user.target
-------------------------------------------------------
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
systemctl status node_exporter.service

  
========================================================================================================================

cat prometheus-2.53.1.linux-amd64/prometheus.yml

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node_exporter"
    static_configs:
      - targets: ["localhost:9100"]

  - job_name: "node_exporter-k8sworker-1"
    static_configs:
      - targets: ["3.93.81.127:9100"]

  - job_name: "node_exporter-k8sworker-2"
    static_configs:
      - targets: ["44.222.117.122:9100"]

  - job_name: "kube-state-metrics-1"
    static_configs:
      - targets: ["3.93.81.127:32581"]
	  
  - job_name: "kube-state-metrics-2"
    static_configs:
      - targets: ["3.93.81.127:30893"]	  
	  

  - job_name: "jenkins"
    static_configs:
      - targets: ["localhost:8080"]
    metrics_path: '/prometheus'


=================================================================
nodeexporter - grafana dashboard - 1860	  
	  
=====================================================================================================
now to monitor extra functionality  at cluster level we need to setup kube-state-metrics server


git clone https://github.com/kubernetes/kube-state-metrics.git

kubectl apply -f kube-state-metrics/examples/standard

kubectl get all -n kube-system

kubectl expose service kube-state-metrics --type=NodePort --name kube-state-metrics-ext -n kube-system




=================