Elasticsearch-Logstash-Kibana (ELK) Overview — Part 1

In this story, I will walk you through my journey of last 2 months in a project. This is the project in which I had to implement ELK stack to collect application logs and AWS service logs and automation with AWS CloudFormation.

First, we will understand basics of ELK stack and then we will jump for advanced or production stack.

Elasticsearch

Elasticsearch is heart of ELK stack which is similar to database of ELK stack where all your data will be indexed, so it is very important. From the website, it is a distributed, RESTful search and analytics engine. Which means it is more then just a database.

Logstash

Logstash is open source, server side data processing pipeline that ingests data from multitude of sources simultaneously, transforms & parse it and sends to Elasticsearch (or other outputs like Kafka, MongoDB, AWS S3, etc).

Kibana

Kibana is used to explore, visualize and discover Elasticsearch data. This is mainly used by developers or end users to search, query and visualize data.