K8s-fluentd-elasticsearch

For Chinese documentation, please visit Here

This is a custom fluentd image with elasticsearch plugin based on Kubernetes's official implementation.

Fluentd is deployed as a DaemonSet which spawns a pod on each node that reads logs, generated by kubelet, container runtime and containers and sends them to Elasticsearch.

Note: in order for Fluentd to work, every Kubernetes node must be labeled with beta.kubernetes.io/fluentd-ds-ready=true, as otherwise the Fluentd DaemonSet will ignore them.

Differences between my custom fluentd-elasticseach image and the official one:

• The Elasticseach address and port are injected into the container by using environment variables.
• Modify Dockerfile to solve some permission problems.
• The offical implementation also deploys the elasticseach on the kubernetes cluster. But in my company, we deploy elasticsearch cluster outside of kubernetes. We think this is a better idea because Elasticseach is a database that is stateful. Since our company uses kubernetes as a micro serice platform, we don't want to deploy anything stateful on kubernetes cluster.

logs are collected by this image:

• containers log: /var/log/containers/*.log
• docker log: /var/log/docker.log
• kubelet log: /var/log/kubelet.log
• kube-proxy log: /var/log/kube-proxy.log
• kube-apiserver log: /var/log/kube-apiserver.log
• kube-controller-manager log: /var/log/kube-controller-manager.log
• kube-scheduler log: /var/log/kube-scheduler.log For more details, please check yaml/fluentd-es-configmap.yaml

How to use

Before you deploy this log collector on your Kubernetes node, you have to label your node by this command:

kubectl label nodes nodename beta.kubernetes.io/fluentd-ds-ready=true

Then, you need to build the image by yourself. Simply go into the Image directory, and run:

docker build -t dockerhub.fengdidi:5000/fengdidi/fluentd-elasticsearch:1801 .
docker push dockerhub.fengdidi:5000/fengdidi/fluentd-elasticsearch:1801

dockerhub.fengdidi:5000/fengdidi/fluentd-elasticsearch:1801 is the image name and tag, use your own name and tag when you build by yourself. After you build the image, go to the yaml directory, create the config map on your kubernetes cluster.

kubectl create -f fluentd-es-configmap.yaml

Then modify fluentd-es-ds.yaml, change the image name to the one you built before. Assign the Elasticseach address and port. The Elasticseach address and port are injected into the container by using environment variables. After finishing your configuration, you can run the fluentd DaemonSet by using the command below:

kubectl create -f fluentd-es-ds.yaml