Fluentd + Elasticsearch + Kibana, your on-premise logging platform

TL;DR: How to set up an open source logging platform on Kubernetes

No, it’s not an Egyptian hieroglyph!

Motivation

Logs are mandatory.

We need a way to record and monitor our application's status in real production-grade applications.

In another post, I explored how you can use an OpenTelemetry-based stack to have traces of the requests which pass through our application.

Instrumenting your Python app under Istio

But, what happens if we need to analyze errors or warnings generated from our system?

In the past, the old school way was to connect with the remote server via SSH (or even telnet) and check for the access.log or error.log generated by Apache, Nginx, …

Nowadays we have a lot of solutions for logging capabilities. In the commercial/cloud stack we found AWS CloudWatch or Azure Monitor.

We have open source alternatives too: Sentry, ELK, …

Sentry is awesome… but it’s so heavy to run it for your “small” cluster.

But there is an open-source alternative very interesting: fluentd.

Indeed, fluentd is very used in the Kubernetes world.

I usually use CloudWatch day by day. I would like to achieve a similar experience on my Kubernetes cluster.

After an evaluation of several options, finally I have one…

The players

• Fluentd it’s a logging collector written in Ruby. It supports filtering, parsing, several output options using plug-ins, etc. I will use the plugin fluent-plugin-elasticsearch to write and index the logs in Elastic.
• ElasticSearch (ES) is very used to store and index JSON documents. Nothing new to tell about it, it’s well known.
• Kibana is a great tool for data discovery, visualization, metrics, … ElasticSearch and Kibana are two parts of the trident ELK (Elastic, Logstash and Kibana). We use it as UI to…