Installing elastic search server

1

I’m trying to install Elasticsearch server on fly.io.

fly deploy -i docker.elastic.co/elasticsearch/elasticsearch:7.5.2

with 2gb of memory. It seems like it’s asking for more.

Screen Shot 2022-04-05 at 5.02.25 PM
Screen Shot 2022-04-05 at 5.02.25 PM1116×501 134 KB

2

I haven’t deployed it but I would suspect it’s not that the memory is not available, it’s that you need to configure elasticsearch to tell it to use it. By using its config options.

The memory error config option/solution:

The file descriptor error config option/solution:

Both of those example are for docker run. You’re running fly deploy (naturally) here. How to translate the command? Hmm. Well … fly deploy does support a --build-arg … so personally I’d give that flag a try. See what happens.

Else maybe try them in the fly.toml [env] block. If not, hopefully someone else can answer!

Here’s the deploy command docs:

3

Did you figure out this problem?

I have tried to set the sysctl -w vm.max_map_count=262144 but permissions issues every time. Also, tried it from the entrypoint and cmd.

4

@dragonfire1119 Maybe @jmarsh24 did (I didn’t end up trying).

If not, what I’d do is to use a custom Dockerfile. Base image would be the elastic one, as if deploying directly. But since you then have control over it, you can edit /etc/sysctl.conf. That way the setting is on the host and so I can’t imagine you would have permissions issues, the way you do with e.g cmd. Check the accepted answer below:

5

Thanks for the reply.

I’m using a custom docker image.

FROM bitnami/elasticsearch:7.12.0

# Tried to change 
USER 1001
ENTRYPOINT [ "/opt/bitnami/scripts/elasticsearch/entry.sh" ]

entry.sh

#!/bin/bash

# shellcheck disable=SC1091

set -o errexit
set -o nounset
set -o pipefail
#set -o xtrace

sysctl -w vm.max_map_count=262144

# Load libraries
. /opt/bitnami/scripts/libbitnami.sh
. /opt/bitnami/scripts/libelasticsearch.sh

# Load environment
. /opt/bitnami/scripts/elasticsearch-env.sh

print_welcome_page

if [[ "$1" = "/opt/bitnami/scripts/elasticsearch/run.sh" ]]; then
    info "** Starting Elasticsearch setup **"
    /opt/bitnami/scripts/elasticsearch/setup.sh
    info "** Elasticsearch setup finished! **"
fi

echo ""
exec "$@"

Error output

sysctl: permission denied on key 'vm.max_map_count'

I tried to copy a custom file to the docker container and it does not persist and change the vm.max_map_count.

COPY sysctl.conf /etc/sysctl.conf

sysctl.conf

vm.max_map_count=262144

Also tried the elasticsearch base image.

I’m also seeing that people say Docker can’t change those settings, only the host level can.

6

Ah. I’d hoped you could use COPY sysctl.conf /etc/sysctl.conf to apply it and it would persist. But seems not :frowning:

Er … that’s me out of ideas then! I wonder how other people do it. Maybe the other guy will get back with what they did.

7

Thanks for trying to help.

From what I see, vm.max_map_count has to be updated directly on the host system and not the container file, which does not work on Fly.io.

8

Anyone else had luck deploying ES to fly?

9

No, it needs permission on the node. From what I’ve read, the VM can’t do it.

10

Hey all. Are you currently using Elasticsearch in production?

11

I’m making an attempt at this currently to deploy sonarqube which uses elasticsearch. No dice so far.

12

What kids of problems are you having? Have you considered using a managed Elasticsearch provider?

13

I gave up and am just hosting it myself at home, I don’t think it’s possible on fly unfortunately.

14

Unforunately I don’t think it’s currently possible to deploy ElasticSearch on fly which is a huge disapointment, but I don’t think it’s realistically possible for Fly to rectify without causing possible issues.

The issue is ElasticSearch requires some non-default settings for various kernel parameters:

  • vm.max_map_count >= 262,144 to allow ES to create many many memory-mapped files.
  • fs.file-max >= 65535 to allow ES to open many fds

Because these are kernel parameters, they must be set on the container host system, they can’t be set from within a container.

As far as I’m aware these checks are hard-coded into the application and cannot be bypassed.

15

Machines run on Firecracker VMs, so each one has its own kernel. You are free to modify any sysctls:

❯ fly ssh console
Connecting to fdaa:1:d615:a7b:10d:fc64:91a9:2... complete
# sysctl vm.max_map_count
vm.max_map_count = 65530
# sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144
# sysctl vm.max_map_count
vm.max_map_count = 262144
1 Like
16

Thank you, @pavel, this works. However, we’ll need to apply the kernel settings for every new machine, so need some way to run sysctl as part of the deploy process.

Since it doesn’t work to run it as part of the docker image (permission issue), is there any way to run host-level command as part of the machine deploy process?

edit: I managed to set the vm.max_map_count=262144 kernel setting from a custom entrypoint.sh script. (Had to make sure the entrypoint script is executed with user root, but I can switch to the application using su inside the script) :white_check_mark:

1 Like
17

Hey @jfschwarz

You can set them via an entrypoint script - the important part is the command should be run as root.
Alternatively, you can configure kernel_args in fly.toml and set the respective sysctls there. I think something like this should work:

kernel_args = ["sysctl.max_map_count=262144"]
1 Like
18 
[[vm]]
memory = '8gb'
cpu_kind = 'shared'
kernel_args = ["vm.max_map_count=1048576", "fs.file-max=1048576"]
cpus = 4

Sorry for the ping, but tried these ones and couldn’t get it to work. Also tried with sysctl -w and says sysctl not found.