EC2 discovery with Elasticsearch

There are multiple nodes in Elasticsearch cluster. To have information of all the nodes to each other, there is node discovery available in Elasticsearch called Zen discovery. The zen discovery is built in discovery module for Elasticsearch and provides unicast discovery with extended support to cloud environments.

In this post, we will see how we can leverage EC2 to setup Elasticsearch cluster in AWS.

We are using Elasticsearch 6.0.0

Standard Way

To configure Elasticsearch cluster, we need to provide all the IPs/DNS of all the other nodes in cluster. Like this:

discovery.zen.ping.unicast.hosts: [“node1”, “node2”, “node3”]

or

discovery.zen.ping.unicast.hosts: [“10.120.0.1”, “10.120.1.2”, “10.120.3.4”]

EC2 Discovery way

Following configurations are required to setup cluster in EC2 instances:

1. All EC2 instances should have this policy attached via IAM role:

{
    “Statement”: [
        {
            “Action”: [
                “ec2:DescribeInstances”
            ],
            “Effect”: “Allow”,
            “Resource”: [
                “*”
            ]
        }
    ],
    “Version”: “2012–10–17”
}

2. All the nodes should have unique EC2 tags key and value. For example in our case it is

ec2discovery : elk

3. After downloading Elasticsearch in each EC2 machine. Run this command from Elasticsearch home directory to download EC2 discovery plugin

echo -y | bin/elasticsearch-plugin install discovery-ec2

4. For master node, use following configuration (considering all EC2 machines in US-West-2 region)

cluster.name: <CLUSTER_NAME>
node.name: <NODE_NAME>
network.host: [_ec2_,_local_]
http.port: 9200
discovery.zen.minimum_master_nodes: 2
discovery.zen.hosts_provider: ec2
bootstrap.memory_lock: true
discovery.ec2.endpoint: ec2.us-west-2.amazonaws.com
discovery.ec2.tag.ec2discovery: elk
discovery.ec2.availability_zones: us-west-2a,us-west-2b,us-west-2c
discovery.ec2.host_type: private_ip
node.master: true
node.data: false
node.ingest: false

5. For data nodes, use following configuration

cluster.name: <CLUSTER_NAME>
node.name: <NODE_NAME>
network.host: [_ec2_,_local_]
http.port: 9200
discovery.zen.minimum_master_nodes: 2
discovery.zen.hosts_provider: ec2
bootstrap.memory_lock: true
discovery.ec2.endpoint: ec2.us-west-2.amazonaws.com
discovery.ec2.tag.ec2discovery: elk
discovery.ec2.availability_zones: us-west-2a,us-west-2b,us-west-2c
discovery.ec2.host_type: private_ip
node.master: false
node.data: true
node.ingest: false

Here tags (in bold) are used to find which all nodes are in cluster.

To get more understanding of terminologies visit Elasticsearch documentation

Through this story, we can conclude that, no manual (or automated) individual IP configuration is required to create a cluster. This can even help in rolling update of Elasticsearch.