Fluentd + Elasticsearch + Kibana, your on-premise logging platform
TL;DR: How to set up an open source logging platform on Kubernetes
Logs are mandatory.
We need a way to record and monitor our application's status in real production-grade applications.
In another post, I explored how you can use an OpenTelemetry-based stack to have traces of the requests which pass through our application.
Instrumenting your Python app under Istio
How to add spans to your Istio traces in Python
But, what happens if we need to analyze errors or warnings generated from our system?
In the past, the old school way was to connect with the remote server via SSH (or even telnet) and check for the
error.log generated by Apache, Nginx, …
Nowadays we have a lot of solutions for logging capabilities. In the commercial/cloud stack we found AWS CloudWatch or Azure Monitor.
We have open source alternatives too: Sentry, ELK, …
Sentry is awesome… but it’s so heavy to run it for your “small” cluster.
But there is an open-source alternative very interesting: fluentd.
Indeed, fluentd is very used in the Kubernetes world.
I usually use CloudWatch day by day. I would like to achieve a similar experience on my Kubernetes cluster.
After an evaluation of several options, finally I have one…
- Fluentd it’s a logging collector written in Ruby. It supports filtering, parsing, several output options using plug-ins, etc. I will use the plugin
fluent-plugin-elasticsearchto write and index the logs in Elastic.
- ElasticSearch (ES) is very used to store and index JSON documents. Nothing new to tell about it, it’s well known.
- Kibana is a great tool for data discovery, visualization, metrics, … ElasticSearch and Kibana are two parts of the trident ELK (Elastic, Logstash and Kibana). We use it as UI to…