How to Setup Elastic Search in Docker Containers

Elastic Search and Docker

Elastic Search is great! Docker is great! Let’s use both of them together! Ok, seriously. I am working on project that is using a fairly large Elastic Search (ES) cluster. More than just a toy example. So we have nodes that run on different machines in different racks and in different datacenters. So the examples (here, here, here, and here) talk about some really great stuff but I couldn’t get them to really work with separate ES nodes. Maybe too, you are running in the cloud and have a nice plugin to use for your cloud provider. Ok, I don’t have that.

The problem.

Well, it lies in the fact that docker networking hides the containers from the networking stack on the host machine. The problem is that when you start elastic search, you have to host addresses:

• The bind host
• The publish host

The bind host is where we declare where we want the process to serve from. This allows us to define which network interface(s) we want to use. It is the basic requirement for a server process to listen to tcp traffic.

The publish host is like the “return address”. When we use the zen discovery with ES, we have to tell the other nodes how to talk to our node. They have important things to tell us and two-way communication is good.

The problem that arrises is that most of the documentation points to this nice parameter network.host (yep, the first parameter in the network documentation). The “cool” (or not) feature of this parameter is that it will set both the bind and publish host for you (source) And it has an even nicer thing where you can set it to 0.0.0.0 which tells ES, “hey, choose a non-loop back interface and just use that!” This shows up in the configuration for the official ES docker container!

Thhe problem hits us when we realize that the IP address in the docker container is not the same IP address that other processes outside of the server can see. Now, there is the GIANT HAMMER solve this. You can add the parameter --net="host" to your docker run command, but that is generally a bad idea (see here, and here).

So it took me some time to realize that I want to set my bind host to the “fancy pants” 0.0.0.0 and then set my publish host to the hostname of the host (or the IP address of my host). Since my docker container doesn’t know much of anything about the host, I have to tell it what is the external host. I do this as part of my docker run command; I add an environment variable --env "PUBLISH_IP=10.0.0.1". Then in my elasticsearch.yml, you have to set the following:

network.bind_host: 0.0.0.0
network.publish_host: "${PUBLISH_IP}"

Adding these two lines will allow you to have your ES containers talk to each other properly. Man, it feels kinda anticlimatic to have a solution that is so small.

Nothing big, but I hope you found this before too much hassle.