Elasticsearch-Logstash-Kibana (ELK) Overview — Part 2

In part 1, it was about E, L & K (theoretically). Now it is time to play around ELK stack. We will start with basics (which means this story is for understanding, not for production usage).

I am going to use AWS EC2 to setup ELK stack. You can use local machine or create VMs in GCP or Azure.

Play with Elasticsearch

• Create a machine in AWS EC2 for Elasticsearch. Use Ubuntu for AMI. For security group, open port 9200 with source 0.0.0.0/0.

9200 is default port for Elasticsearch. It is best practice to not give default port in production and source open to world. Example, you can give port like 19200 and CIDR range of your VPC as source.

• Install Java8 in this machine. Elasticsearch, Logstash and Kibana needs Java8.
• Download latest Elasticsearch from here. Right click on tar.gz format and save link address
• Go to Linux machine in home directory (/home/ubuntu). Wget to download Elasticsearch tar.gz file. Example,

Location: /home/ubuntu
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.1.2.tar.gz

• To untar above downloaded file, use tar command. Example,

Location: /home/ubuntu
$ tar -xzf elasticsearch-6.1.2.tar.gz

• Change basic configuration of Elasticsearch in elasticsearch.yml

Location: /home/ubuntu/elasticsearch-6.1.2/config/elasticsearch.yml

Make following changes for network.host and http.port:

# — — — — — — — — — — — — — — — — — Network — — — — — — — — — —
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200

• Boostrap configuration for Elasticsearch. Run following commands:

$ sudo sh -c "echo 'vm.max_map_count=262144' >> /etc/sysctl.conf"
$ sudo sh -c "echo '*    soft nofile 65536' >> /etc/security/limits.conf"
$ sudo sh -c "echo '*    hard nofile 65536' >> /etc/security/limits.conf"
$ sudo sh -c "echo '*    soft memlock unlimited' >> /etc/security/limits.conf"
$ sudo sh -c "echo '*    hard memlock unlimited' >> /etc/security/limits.conf"
$ sudo sh -c "echo 'root soft nofile 65536' >> /etc/security/limits.conf"
$ sudo sh -c "echo 'root hard nofile 65536' >> /etc/security/limits.conf"
$ sudo sh -c "echo 'session required pam_limits.so' >> /etc/pam.d/common-session"
$ sudo sh -c "echo 'session required pam_limits.so' >> /etc/pam.d/common-session-noninteractive"

• To make above changes in system, sign out of machine and re-login. (or reboot machine)
• Start Elasticsearch

Location: /home/ubuntu/elasticsearch-6.1.2
$ bin/elasticsearch