Stairway to the master of DevOps

6. Run Prometheus to monitor k8s in the cluster

Harry Hiyoshi's photo
Harry Hiyoshi
·

3 min read

There are several ways to set up Prometheus for this purpose, but kube-prometheus seems the easiest. With this repository, you can install Prometheus and Grafana into your cluster with default but enough configurations.

See https://github.com/prometheus-operator/kube-prometheus

1) For Minikube. Restart it with some parameters

minikube delete && minikube start --kubernetes-version=v1.23.0 --memory=6g --bootstrapper=kubeadm --extra-config=kubelet.authentication-token-webhook=true --extra-config=kubelet.authorization-mode=Webhook --extra-config=scheduler.bind-address=0.0.0.0 --extra-config=controller-manager.bind-address=0.0.0.0

minikube addons disable metrics-server

  1. $ minikube delete: This command deletes an existing minikube cluster, if one is already running. It ensures that you start with a clean slate before creating a new cluster.

  2. &&: This is a shell operator that means "and then." It ensures that the next command (minikube start) will only be executed if the previous command (minikube delete) is successful.

  3. $ minikube start: This command starts a new minikube cluster.

    • --kubernetes-version=v1.23.0: Specifies the Kubernetes version to be used. In this case, it sets the version to 1.23.0.

    • --memory=6g: Allocates 6 gigabytes of memory for the virtual machine that runs the Kubernetes cluster.

    • --bootstrapper=kubeadm: Specifies the bootstrapper to be used. In this case, it's set to use kubeadm, which is a tool for bootstrapping Kubernetes clusters.

    • --extra-config=kubelet.authentication-token-webhook=true: Configures the kubelet (a component running on each node) to use authentication token webhooks for authentication.

    • --extra-config=kubelet.authorization-mode=Webhook: Configures the kubelet to use webhook-based authorization mode.

    • --extra-config=scheduler.bind-address=0.0.0.0: Configures the Kubernetes scheduler to bind to all available network interfaces.

    • --extra-config=controller-manager.bind-address=0.0.0.0: Configures the Kubernetes controller manager to bind to all available network interfaces.

In summary, this command sequence deletes any existing minikube cluster and starts a new one with specific configurations, such as Kubernetes version, memory allocation, and various kubelet, scheduler, and controller manager settings. These settings are often used for development or testing purposes when running Kubernetes locally.

2) Clone the repository in your local directory

git clone https://github.com/prometheus-operator/kube-prometheus.git
cd kube-prometheus/

3) Check your version of Kubernetes

kubectl version -o yaml

The command kubectl version -o yaml is used to get the version information of both the Kubernetes client (kubectl) and the Kubernetes server (the cluster).

It needs to choose an appropriate version of Prometheus to the version of the cluster by checking a matrix here https://github.com/prometheus-operator/kube-prometheus?tab=readme-ov-file#compatibility

4) Create and switch to a new branch based on the version you checked above. Mine is 0.7 but you have to change to your version number.

git checkout -b release-0.7 origin/release-0.7
git branch

  • checkout -b release-0.7: This part of the command is creating a new branch named release-0.7. The -b option is used to create a new branch, and release-0.7 is the name given to this branch.

  • origin/release-0.7: This part specifies the starting point for the new branch. The branch release-0.7 is created based on the remote branch named release-0.7 from the remote repository named origin. This typically means that the new branch will be an identical copy of the remote release-0.7 branch.

5) Deploy it

kubectl create -f manifests/setup
kubectl create -f manifests/

6) Check if it started

It starts in the namespace "monitoring."

kubectl get svc --namespace monitoring

7) Access the UIs

I did port forwarding to access the UI of Prometheus, Alert Manager and Grafana from outside of the cluster

kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090 & \
kubectl --namespace monitoring port-forward svc/alertmanager-main 9093 & \
kubectl --namespace monitoring port-forward svc/grafana 3000 &

By accessing localhost:9090 and localhost:3000 on a Browser, you can see metrics scraped from your cluster.

You can also see that Grafana can collect metrics scraped by Prometheus by accessing localhost:3000.

Firstly, log in to Graana and then choose a menu of Dashboard-Default.

Kubernetes