The Open Distro for Elasticsearch (odfe) in this article is Upstream of Amazon Elasticsearch Service (Amazon ES).
We are building a development environment with odfe because we plan to operate it on Amazon ES in the future. There is a part that got stuck when building an odfe development environment with multiple nodes, so I would like to summarize that this time.
The environment is basically constructed according to the official document of odfe. https://opendistro.github.io/for-elasticsearch-docs/docs/install/docker/
However, in the official documentation, the cluster consists of 2 nodes, and it is not recommended to create it with 2 nodes according to the documentation in terms of fault tolerance. Elastic explained as follows.
It is generally recommended to include three master-eligible nodes in the cluster. If one of the nodes fails, the other two can safely form quorum and continue. If there are no more than two master-eligible nodes in the cluster, none of them can tolerate the failure. Conversely, if there are four or more master-eligible nodes in the cluster, it can take a long time to elect the master and update the cluster status. (New Generation Elasticsearch Cluster Coordination)
So, this time I'm writing a setting in the docker-compose file to configure the cluster with 3 master-eligible nodes. Reference article: Important discovery and cluster formation settings
Also, up to version 6 series, it was necessary to set the master qualified node not to be an even number, but from version 7 series a new cluster coordination mechanism was introduced, and Elasticsearch even with an even number configuration It is designed to adjust automatically and avoid the occurrence of split brain.
For details, please see the Elastic Stack Practical Guide (Elasticsearch/Kibana edition) because the author Qiita is very easy to understand. "Is it okay to configure Elasticsearch Masternodes with an even number?" (2020 version)
The Docker file and docker-compose file created this time are as follows.
FROM amazon/opendistro-for-elasticsearch:1.11.0
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch analysis-kuromoji
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch analysis-icu
version: "3"
services:
elasticsearch-node1:
build:
context: .
dockerfile: dockerfiles/Dockerfile.es
container_name: elasticsearch-node1
environment:
- cluster.name=elasticsearch-cluster
- node.name=elasticsearch-node1
- discovery.seed_hosts=elasticsearch-node1,elasticsearch-node2,elasticsearch-node3
- cluster.initial_master_nodes=elasticsearch-node1,elasticsearch-node2,elasticsearch-node3
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- elasticsearch-data1:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
networks:
- elasticsearch-net
elasticsearch-node2:
build:
context: .
dockerfile: dockerfiles/Dockerfile.es
container_name: elasticsearch-node2
environment:
- cluster.name=elasticsearch-cluster
- node.name=elasticsearch-node2
- discovery.seed_hosts=elasticsearch-node1,elasticsearch-node2,elasticsearch-node3
- cluster.initial_master_nodes=elasticsearch-node1,elasticsearch-node2,elasticsearch-node3
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- elasticsearch-data2:/usr/share/elasticsearch/data
networks:
- elasticsearch-net
elasticsearch-node3:
build:
context: .
dockerfile: dockerfiles/Dockerfile.es
container_name: elasticsearch-node3
environment:
- cluster.name=elasticsearch-cluster
- node.name=elasticsearch-node3
- discovery.seed_hosts=elasticsearch-node1,elasticsearch-node2,elasticsearch-node3
- cluster.initial_master_nodes=elasticsearch-node1,elasticsearch-node2,elasticsearch-node3
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- elasticsearch-data3:/usr/share/elasticsearch/data
networks:
- elasticsearch-net
kibana:
image: amazon/opendistro-for-elasticsearch-kibana:1.11.0
container_name: elasticsearch-kibana
ports:
- 5601:5601
expose:
- "5601"
environment:
ELASTICSEARCH_URL: https://elasticsearch-node1:9200
ELASTICSEARCH_HOSTS: https://elasticsearch-node1:9200
networks:
- elasticsearch-net
volumes:
elasticsearch-data1:
elasticsearch-data2:
elasticsearch-data3:
networks:
elasticsearch-net:
The setting value is also explained in Official odfe document, so please refer to it.
By default, Amazon ES includes the ICU Analysis and Japanese (kuromoji) Analysis plugins in all domains (https://docs.aws.amazon.com/ja_jp/elasticsearch-service/latest/developerguide/aes-supported-plugins.html), so we also include the plugins in the odfe development environment.
When I started a container with docker-compose, when I tried to start it with multiple nodes, some containers ended with exit with 137.
If it exits with exit with 137 and some containers do not start, it is possible that the Docker Engine is out of memory.
Since it is also listed in Issue, if some containers do not start up like you, please try to increase the memory of Docker Engine from the setting screen of Docker Desktop.
In my environment, with this support, exit with 137 did not appear and all containers started up.
I introduced the part that got stuck when building an odfe development environment with multiple nodes. Since this is my first time using Elasticsearch, I would appreciate it if you could comment or request editing if there are any mistakes in the settings.
@ Mt-kage will be in charge of tomorrow's Advent calendar! looking forward to~!
Recommended Posts