Benchmarking Elasticsearch cluster with Rally

Install Rally on each node

Prerequisites

$ yum update
$ yum install openssl-devel bzip2-devel libffi-devel
$ yum groupinstall "Development Tools"

Install Python 3.8+

$ wget https://www.python.org/ftp/python/3.8.15/Python-3.8.15.tar.xz
$ tar xf Python-3.8.15.tar.xz
$ vim Python-3.8.15/Modules/Setup
SSL=/usr/local/ssl
_ssl _ssl.c \
        -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
        -L$(SSL)/lib -lssl -lcrypto
$ mv Python-3.8.15 /usr/src
$ cd /usr/src/Python-3.8.15/
$ ./configure --enable-optimizations
$ make altinstall
$ python3.8 -m ssl
$ pip3.8 install --upgrade pip

Install Git (Not required for load generator node)

$ yum install libcurl-devel
$ wget https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.38.1.tar.xz
$ tar xvf git-2.38.1.tar.xz
$ cd git-2.38.1/
$ make configure
$ ./configure --prefix=/usr/local
$ make install
$ git --version
git version 2.38.1

Install JDK (Not required for load generator node)

$ yum install java
$ java -version
openjdk version "1.8.0_352"

Install esrally

$ pip3.8 install esrally
$ esrally --version
esrally 2.6.0

Elasticsearch can not be launched as root. Create a non-root user on each node.

$ groupadd es
$ useradd es -g es
$ passwd es
$ cd /home/es
$ su - es

Set JAVA_HOME path

$ vim .bash_profile
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre
export JAVA_HOME
$ source .bash_profile
$ echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre

Benchmarking a Single Node

Install Elasticsearch

$ esrally install --distribution-version=7.17.0 --node-name="rally-node-0" --network-host="127.0.0.1" --http-port=39200 --master-nodes="rally-node-0" --seed-hosts="127.0.0.1:39300"

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] Downloading Elasticsearch 7.17.0 (297.0 MB total size)                       [100%]
{
  "installation-id": "10735bfa-f1b8-44c4-8e7f-8932c8daa201"
}

--------------------------------
[INFO] SUCCESS (took 10 seconds)
--------------------------------

Start the Elasticsearch node

$ export INSTALLATION_ID=10735bfa-f1b8-44c4-8e7f-8932c8daa201
$ export RACE_ID=$(uuidgen)
$ esrally start --installation-id="${INSTALLATION_ID}" --race-id="${RACE_ID}"

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/


-------------------------------
[INFO] SUCCESS (took 3 seconds)
-------------------------------

Run a benchmark

$ esrally race --pipeline=benchmark-only --target-host=127.0.0.1:39200 --track=geonames --challenge=append-no-conflicts-index-only --on-error=abort --race-id=${RACE_ID}

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] Race id is [4e93324a-9326-49e8-be72-9f77cc837657]
[INFO] Downloading track data (252.9 MB total size)                               [100.0%]
[INFO] Decompressing track data from [/home/es/.rally/benchmarks/data/geonames/documents-2.json.bz2] to [/home/es/.rally/benchmarks/data/geonames/documents-2.json] (resulting size: [3.30] GB) ... [OK]
[INFO] Preparing file offset table for [/home/es/.rally/benchmarks/data/geonames/documents-2.json] ... [OK]
[INFO] Racing on track [geonames], challenge [append-no-conflicts-index-only] and car ['external'] with version [7.17.0].

Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running check-cluster-health                                                   [100% done]
Running index-append                                                           [100% done]
Running force-merge                                                            [100% done]
Running wait-until-merges-finish                                               [100% done]

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------

|                                                         Metric |         Task |           Value |   Unit |
|---------------------------------------------------------------:|-------------:|----------------:|-------:|
|                     Cumulative indexing time of primary shards |              |    12.5856      |    min |
|             Min cumulative indexing time across primary shards |              |     0.0190333   |    min |
|          Median cumulative indexing time across primary shards |              |     2.49065     |    min |
|             Max cumulative indexing time across primary shards |              |     2.61288     |    min |
|            Cumulative indexing throttle time of primary shards |              |     0.0081      |    min |
|    Min cumulative indexing throttle time across primary shards |              |     0           |    min |
| Median cumulative indexing throttle time across primary shards |              |     0           |    min |
|    Max cumulative indexing throttle time across primary shards |              |     0.0061      |    min |
|                        Cumulative merge time of primary shards |              |     6.75938     |    min |
|                       Cumulative merge count of primary shards |              |    55           |        |
|                Min cumulative merge time across primary shards |              |     0           |    min |
|             Median cumulative merge time across primary shards |              |     1.32735     |    min |
|                Max cumulative merge time across primary shards |              |     1.47518     |    min |
|               Cumulative merge throttle time of primary shards |              |     1.72393     |    min |
|       Min cumulative merge throttle time across primary shards |              |     0           |    min |
|    Median cumulative merge throttle time across primary shards |              |     0.323533    |    min |
|       Max cumulative merge throttle time across primary shards |              |     0.42085     |    min |
|                      Cumulative refresh time of primary shards |              |     2.97073     |    min |
|                     Cumulative refresh count of primary shards |              |   246           |        |
|              Min cumulative refresh time across primary shards |              |     0.00185     |    min |
|           Median cumulative refresh time across primary shards |              |     0.5943      |    min |
|              Max cumulative refresh time across primary shards |              |     0.599217    |    min |
|                        Cumulative flush time of primary shards |              |     0.237767    |    min |
|                       Cumulative flush count of primary shards |              |     8           |        |
|                Min cumulative flush time across primary shards |              |     0.00241667  |    min |
|             Median cumulative flush time across primary shards |              |     0.0473      |    min |
|                Max cumulative flush time across primary shards |              |     0.0492167   |    min |
|                                        Total Young Gen GC time |              |    14.009       |      s |
|                                       Total Young Gen GC count |              |  1159           |        |
|                                          Total Old Gen GC time |              |     3.491       |      s |
|                                         Total Old Gen GC count |              |    66           |        |
|                                                     Store size |              |     3.20482     |     GB |
|                                                  Translog size |              |     3.07336e-07 |     GB |
|                                         Heap used for segments |              |     1.00685     |     MB |
|                                       Heap used for doc values |              |     0.0602913   |     MB |
|                                            Heap used for terms |              |     0.77124     |     MB |
|                                            Heap used for norms |              |     0.104492    |     MB |
|                                           Heap used for points |              |     0           |     MB |
|                                    Heap used for stored fields |              |     0.0708237   |     MB |
|                                                  Segment count |              |   141           |        |
|                                    Total Ingest Pipeline count |              |     0           |        |
|                                     Total Ingest Pipeline time |              |     0           |      s |
|                                   Total Ingest Pipeline failed |              |     0           |        |
|                                                 Min Throughput | index-append | 86654           | docs/s |
|                                                Mean Throughput | index-append | 87091.9         | docs/s |
|                                              Median Throughput | index-append | 87152.4         | docs/s |
|                                                 Max Throughput | index-append | 87276.3         | docs/s |
|                                        50th percentile latency | index-append |   315.247       |     ms |
|                                        90th percentile latency | index-append |   573.935       |     ms |
|                                        99th percentile latency | index-append |  1139.89        |     ms |
|                                       100th percentile latency | index-append |  1153.22        |     ms |
|                                   50th percentile service time | index-append |   315.247       |     ms |
|                                   90th percentile service time | index-append |   573.935       |     ms |
|                                   99th percentile service time | index-append |  1139.89        |     ms |
|                                  100th percentile service time | index-append |  1153.22        |     ms |
|                                                     error rate | index-append |     0           |      % |


---------------------------------
[INFO] SUCCESS (took 255 seconds)
---------------------------------

Stop the Elasticsearch node

$ esrally stop --installation-id="${INSTALLATION_ID}"

If you only want to shutdown the node but don’t want to delete the node and the data, pass –preserve-install additionally.

Benchmarking a Cluster

Install and start Elasticsearch on each cluster node

$ esrally install --distribution-version=7.17.0 --node-name="rally-node-0" --network-host="10.10.10.2" --http-port=39200 --master-nodes="rally-node-0,rally-node-1,rally-node-2" --seed-hosts="10.10.10.2:39300,10.10.10.3:39300,10.10.10.4:39300"
[INFO] Downloading Elasticsearch 7.17.0 (297.0 MB total size)  [100%]
{
  "installation-id": "aa826112-d371-4f09-9b68-f9084e7c9e0b"
}

Generate a race id on one of the nodes

$ uuidgen
734bb4b3-8b7a-4c0b-9fa6-aaeb4659569f

Note: The same race id is set on all the nodes including the one where will generate load.

Start the cluster by running the following command on each node

$ export INSTALLATION_ID=aa826112-d371-4f09-9b68-f9084e7c9e0b
$ export RACE_ID=734bb4b3-8b7a-4c0b-9fa6-aaeb4659569f
$ esrally start --installation-id="${INSTALLATION_ID}" --race-id="${RACE_ID}"

Note: The INSTALLATION_ID is specific to each node and the RACI_ID is identical for all the nodes.

Once the cluster is started, check the cluster status with the _cat/health API

[es@node1 ~]$ curl http://10.10.10.2:39200/_cat/health\?v
epoch      timestamp cluster         status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1667015754 03:55:54  rally-benchmark green           3         3      6   3    0    0        0             0                  -                100.0%

On each cluster node, check the elastic process and port

$ ps -ef | egrep -i "rally|elastic" | grep -v grep
es        2258     1 91 20:52 ?        00:58:39 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,JRE -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-1322059221495755520 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/heapdump -XX:ErrorFile=/home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/logs/server/hs_err_pid%p.log -XX:+ExitOnOutOfMemoryError -XX:MaxDirectMemorySize=536870912 -Des.path.home=/home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/install/elasticsearch-7.17.0 -Des.path.conf=/home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/install/elasticsearch-7.17.0/config -Des.distribution.flavor=default -Des.distribution.type=tar -Des.bundled_jdk=true -cp /home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/install/elasticsearch-7.17.0/lib/* org.elasticsearch.bootstrap.Elasticsearch -d -p ./pid
es        2285  2258  0 20:52 ?        00:00:00 /home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/install/elasticsearch-7.17.0/modules/x-pack-ml/platform/linux-x86_64/bin/controller

$ netstat -anop | grep 39200
tcp6       0      0 10.10.10.2:39200       :::*                    LISTEN      2258/java            off (0.00/0/0)

Start the benchmark on the load generator node (remember to set the race id there)

[es@node1 ~]$ export RACE_ID=734bb4b3-8b7a-4c0b-9fa6-aaeb4659569f
[es@node1 ~]$ esrally race --pipeline=benchmark-only --target-host=10.10.10.2:39200,10.10.10.3:39200,10.10.10.4:39200 --track=geonames --challenge=append-no-conflicts --on-error=abort --race-id=${RACE_ID}
    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] Race id is [734bb4b3-8b7a-4c0b-9fa6-aaeb4659569f]
[INFO] Racing on track [geonames], challenge [append-no-conflicts] and car ['external'] with version [7.17.0].

[WARNING] merges_total_time is 420149 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] merges_total_throttled_time is 81765 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] indexing_total_time is 825388 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] refresh_total_time is 76340 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] flush_total_time is 10787 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running check-cluster-health                                                   [100% done]
Running index-append                                                           [100% done]
Running refresh-after-index                                                    [100% done]
Running force-merge                                                            [100% done]
Running refresh-after-force-merge                                              [100% done]
Running wait-until-merges-finish                                               [100% done]
Running index-stats                                                            [100% done]
Running node-stats                                                             [100% done]
Running default                                                                [100% done]
Running term                                                                   [100% done]
Running phrase                                                                 [100% done]
Running country_agg_uncached                                                   [100% done]
Running country_agg_cached                                                     [100% done]
Running scroll                                                                 [100% done]
Running expression                                                             [100% done]
Running painless_static                                                        [100% done]
Running painless_dynamic                                                       [100% done]
Running decay_geo_gauss_function_score                                         [100% done]
Running decay_geo_gauss_script_score                                           [100% done]
Running field_value_function_score                                             [100% done]
Running field_value_script_score                                               [100% done]
Running large_terms                                                            [100% done]
Running large_filtered_terms                                                   [100% done]
Running large_prohibited_terms                                                 [100% done]
Running desc_sort_population                                                   [100% done]
Running asc_sort_population                                                    [100% done]
Running asc_sort_with_after_population                                         [100% done]
Running desc_sort_geonameid                                                    [100% done]
Running desc_sort_with_after_geonameid                                         [100% done]
Running asc_sort_geonameid                                                     [100% done]
Running asc_sort_with_after_geonameid                                          [100% done]

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------

|                                                         Metric |                           Task |          Value |    Unit |
|---------------------------------------------------------------:|-------------------------------:|---------------:|--------:|
|                     Cumulative indexing time of primary shards |                                |   13.3055      |     min |
|             Min cumulative indexing time across primary shards |                                |    0           |     min |
|          Median cumulative indexing time across primary shards |                                |    2.68572     |     min |
|             Max cumulative indexing time across primary shards |                                |    2.74885     |     min |
|            Cumulative indexing throttle time of primary shards |                                |    0           |     min |
|    Min cumulative indexing throttle time across primary shards |                                |    0           |     min |
| Median cumulative indexing throttle time across primary shards |                                |    0           |     min |
|    Max cumulative indexing throttle time across primary shards |                                |    0           |     min |
|                        Cumulative merge time of primary shards |                                |    4.82182     |     min |
|                       Cumulative merge count of primary shards |                                |   57           |         |
|                Min cumulative merge time across primary shards |                                |    0           |     min |
|             Median cumulative merge time across primary shards |                                |    0.984917    |     min |
|                Max cumulative merge time across primary shards |                                |    1.06472     |     min |
|               Cumulative merge throttle time of primary shards |                                |    0.978367    |     min |
|       Min cumulative merge throttle time across primary shards |                                |    0           |     min |
|    Median cumulative merge throttle time across primary shards |                                |    0.195508    |     min |
|       Max cumulative merge throttle time across primary shards |                                |    0.265933    |     min |
|                      Cumulative refresh time of primary shards |                                |    1.20573     |     min |
|                     Cumulative refresh count of primary shards |                                |  148           |         |
|              Min cumulative refresh time across primary shards |                                |    3.33333e-05 |     min |
|           Median cumulative refresh time across primary shards |                                |    0.258775    |     min |
|              Max cumulative refresh time across primary shards |                                |    0.283433    |     min |
|                        Cumulative flush time of primary shards |                                |    0.172783    |     min |
|                       Cumulative flush count of primary shards |                                |   11           |         |
|                Min cumulative flush time across primary shards |                                |    1.66667e-05 |     min |
|             Median cumulative flush time across primary shards |                                |    0.0345      |     min |
|                Max cumulative flush time across primary shards |                                |    0.0385167   |     min |
|                                        Total Young Gen GC time |                                |   16.263       |       s |
|                                       Total Young Gen GC count |                                | 2821           |         |
|                                          Total Old Gen GC time |                                |    2.312       |       s |
|                                         Total Old Gen GC count |                                |   41           |         |
|                                                     Store size |                                |    3.03867     |      GB |
|                                                  Translog size |                                |    3.58559e-07 |      GB |
|                                         Heap used for segments |                                |    0.701981    |      MB |
|                                       Heap used for doc values |                                |    0.0314178   |      MB |
|                                            Heap used for terms |                                |    0.54541     |      MB |
|                                            Heap used for norms |                                |    0.0736694   |      MB |
|                                           Heap used for points |                                |    0           |      MB |
|                                    Heap used for stored fields |                                |    0.0514832   |      MB |
|                                                  Segment count |                                |  102           |         |
|                                    Total Ingest Pipeline count |                                |    0           |         |
|                                     Total Ingest Pipeline time |                                |    0           |       s |
|                                   Total Ingest Pipeline failed |                                |    0           |         |
|                                                     error rate |                   index-append |    0           |       % |
|                                                 Min Throughput |                    index-stats |   90.01        |   ops/s |
|                                                Mean Throughput |                    index-stats |   90.02        |   ops/s |
|                                              Median Throughput |                    index-stats |   90.02        |   ops/s |
|                                                 Max Throughput |                    index-stats |   90.04        |   ops/s |
|                                        50th percentile latency |                    index-stats |    5.16153     |      ms |
|                                        90th percentile latency |                    index-stats |    6.00114     |      ms |
|                                        99th percentile latency |                    index-stats |    6.61081     |      ms |
|                                      99.9th percentile latency |                    index-stats |   10.4064      |      ms |
|                                       100th percentile latency |                    index-stats |   10.8105      |      ms |
|                                   50th percentile service time |                    index-stats |    4.00402     |      ms |
|                                   90th percentile service time |                    index-stats |    4.6339      |      ms |
|                                   99th percentile service time |                    index-stats |    5.10083     |      ms |
|                                 99.9th percentile service time |                    index-stats |    9.17415     |      ms |
|                                  100th percentile service time |                    index-stats |    9.22474     |      ms |
[..]

[WARNING] No throughput metrics available for [index-append]. Likely cause: The benchmark ended already during warmup.

----------------------------------
[INFO] SUCCESS (took 4008 seconds)
----------------------------------

Shutdown the cluster on each node

$ esrally stop --installation-id="${INSTALLATION_ID}"

Note: If you only want to shutdown the node but don’t want to delete the node and the data, add the option “–preserve-install” additionally.

Troubleshooting

Elasticsearch start failure due to max virtual memory is too low

$ cat /home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/logs/server/rally-benchmark.log
[..]
[2022-10-28T16:58:41,041][ERROR][o.e.b.Bootstrap          ] [rally-node-0] node validation exception
[1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[..]

$ sysctl -a | grep max_map_count
vm.max_map_count = 65530

To fix this issue, change the kernel parameter

$ vim /etc/sysctl.conf
vm.max_map_count=1048576
$ sysctl -p
vm.max_map_count = 1048576

Restart Elasticsearch and verify the process and port

$ esrally stop --installation-id="${INSTALLATION_ID}" --preserve-install
$ esrally start --installation-id="${INSTALLATION_ID}" --race-id="${RACE_ID}"

$ netstat -anop | grep 39200
tcp6       0      0 10.10.10.2:39200       :::*                    LISTEN      23726/java           off (0.00/0/0)

$ ps -ef | egrep -i "rally|elastic" | grep -v grep
es       23726     1 18 18:01 ?        00:00:42 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,JRE -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-7969870787666953814 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/heapdump -XX:ErrorFile=/home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/logs/server/hs_err_pid%p.log -XX:+ExitOnOutOfMemoryError -XX:MaxDirectMemorySize=536870912 -Des.path.home=/home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/install/elasticsearch-7.17.0 -Des.path.conf=/home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/install/elasticsearch-7.17.0/config -Des.distribution.flavor=default -Des.distribution.type=tar -Des.bundled_jdk=true -cp /home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/install/elasticsearch-7.17.0/lib/* org.elasticsearch.bootstrap.Elasticsearch -d -p ./pid
es       23752 23726  0 18:01 ?        00:00:00 /home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/install/elasticsearch-7.17.0/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Elasticsearch start failure due to max file descriptors is too low

$ cat /home/es/.rally/benchmarks/races/aa826112-d371-4f09-9b68-f9084e7c9e0b/rally-node-0/logs/server/rally-benchmark.log
[..]
[2022-10-28T18:02:09,107][ERROR][o.e.b.Bootstrap          ] [rally-node-1] node validation exception
[1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
bootstrap check failure [1] of [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[..]

To fix this issue, change the max open files value in /etc/security/limits.conf

$ vim /etc/security/limits.conf
*              soft     nofile          1048576
*              hard     nofile          1048576

Exit and login back to the shell to see the changed value

ulimit -a  | grep "open files"
open files                      (-n) 1048576

Reference