Getting started with Elastic Rally benchmark
Rally is the macrobenchmarking framework for Elasticsearch. This post follows the instructions here to install Rally and run the very first benchmark(aka race).
Install python
Python 3.8+ including pip3 is required for Rally.
$ yum update
$ yum install openssl-devel bzip2-devel libffi-devel
$ yum groupinstall "Development Tools"
$ curl -O https://www.python.org/ftp/python/3.8.1/Python-3.8.1.tgz
$ tar zxf Python-3.8.1.tgz
$ mv Python-3.8.1 /usr/src
$ cd /usr/src/Python-3.8.1/
$ vim Modules/Setup
SSL=/usr/local/ssl
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
-L$(SSL)/lib -lssl -lcrypto
$ ./configure --enable-optimizations
$ make altinstall
$ python3.8 -m ssl
$ pip3 -V
pip 22.3 from /usr/local/lib/python3.8/site-packages/pip (python 3.8)
Note: If you do not uncomment the 4 lines in Modules/Setup, you would fail to install Rally with the ssl module unavailable issue as mentioned in the following troubleshooting section.
Install git
Git is not required if all of the following conditions are met:
You are using Rally only as a load generator (–pipeline=benchmark-only) or you are referring to Elasticsearch configurations with –team-path.
You create your own tracks and refer to them with –track-path.
In all other cases, Rally requires git 1.9 or better. Verify with git –version
$ yum -y remove git
$ yum -y remove git-*
$ yum install git
$ git version
git version 2.38.1
Install JDK
A JDK is required on all machines where you want to launch Elasticsearch. If you use Rally just as a load generator to benchmark remote clusters, no JDK is required. Refer to here to determine the appropriate JDK version to run Elasticsearch.
$ yum install java
$ java -version
openjdk version "1.8.0_345"
OpenJDK Runtime Environment (build 1.8.0_345-b01)
OpenJDK 64-Bit Server VM (build 25.345-b01, mixed mode)
To download and install a specific java version:
$ wget https://download.oracle.com/java/17/archive/jdk-17.0.5_linux-x64_bin.rpm
$ rpm -ivh jdk-17.0.5_linux-x64_bin.rpm
$ java -version
java version "17.0.5" 2022-10-18 LTS
Java(TM) SE Runtime Environment (build 17.0.5+9-LTS-191)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.5+9-LTS-191, mixed mode, sharing)
$ rpm -qi jdk-17-17.0.5-ga.x86_64
Name : jdk-17
Epoch : 2000
Version : 17.0.5
Release : ga
Architecture: x86_64
Install Date: Wed 02 Nov 2022 03:58:42 PM PDT
Group : Development/Tools
Size : 316751437
License : https://java.com/freeuselicense
Signature : RSA/SHA256, Tue 13 Sep 2022 09:36:17 AM PDT, Key ID 72f97b74ec551f03
Source RPM : jdk-17-17.0.5-ga.src.rpm
Build Date : Tue 13 Sep 2022 09:35:27 AM PDT
Build Host : java.com
Relocations : /usr/java
Vendor : Oracle Corporation
URL : http://www.oracle.com/technetwork/java/javase/overview/index.html
Summary : Java Platform Standard Edition Development Kit
Description :
The Java Platform Standard Edition Development Kit (JDK) includes both
the runtime environment (Java virtual machine, the Java platform classes
and supporting files) and development tools (compilers, debuggers,
tool libraries and other tools).
With java 1.8.0, you can run Elasticsearch 7.17.x or lower version.
To find the JDK, Rally expects the environment variable JAVA_HOME to be set on all targeted machines. To have more specific control, for example when you want to benchmark across a wide range of Elasticsearch releases, you can also set JAVAx_HOME where x is the major version of a JDK (e.g. JAVA8_HOME would point to a JDK 8 installation). Rally will then choose the highest supported JDK per version of Elasticsearch that is available.
$ export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre
$ echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre
Install Rally
$ pip3.8 install --upgrade pip
$ pip3.8 install esrally
$ esrally -h
usage: esrally [-h] [--version] {race,list,info,create-track,generate,compare,download,install,start,stop} ...
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
You Know, for Benchmarking Elasticsearch.
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
subcommands:
{race,list,info,create-track,generate,compare,download,install,start,stop}
race Run a benchmark
list List configuration options
info Show info about a track
create-track Create a Rally track from existing data
generate Generate artifacts
compare Compare two races
download Downloads an artifact
install Installs an Elasticsearch node locally
start Starts an Elasticsearch node locally
stop Stops an Elasticsearch node locally
Find out more about Rally at https://esrally.readthedocs.io/en/2.6.0/
$ esrally list tracks
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
Available tracks:
Name Description Documents Compressed Size Uncompressed Size Default Challenge All Challenges
---------------- ----------------------------------------------------------------------- ----------- ----------------- ------------------- ----------------------- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dense_vector Benchmark for dense vector indexing and search 10,000,000 7.2 GB 19.5 GB index-and-search index-and-search
elastic/endpoint Endpoint track 0 0 bytes 0 bytes default default
elastic/logs Track for simulating logging workloads 14,009,078 N/A N/A logging-indexing cross-clusters-search,logging-disk-usage,logging-indexing-querying,logging-indexing,logging-querying,logging-snapshot-mount,logging-snapshot-restore,logging-snapshot,many-shards-quantitative,many-shards-snapshots
elastic/security Track for simulating Elastic Security workloads 77,513,777 N/A N/A security-querying index-alert-source-events,security-indexing-querying,security-indexing,security-querying
eql EQL benchmarks based on endgame index of SIEM demo cluster 60,782,211 4.5 GB 109.2 GB default default,index-sorting
geonames POIs from Geonames 11,396,503 252.9 MB 3.3 GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts,significant-text
geopoint Point coordinates from PlanetOSM 60,844,404 482.1 MB 2.3 GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts
geopointshape Point coordinates from PlanetOSM indexed as geoshapes 60,844,404 470.8 MB 2.6 GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts
geoshape Shapes from PlanetOSM 84,220,567 17.0 GB 58.7 GB append-no-conflicts append-no-conflicts,append-no-conflicts-big
http_logs HTTP server log data 247,249,096 1.2 GB 31.1 GB append-no-conflicts append-no-conflicts,runtime-fields,append-no-conflicts-index-only,append-sorted-no-conflicts,append-index-only-with-ingest-pipeline,update,append-no-conflicts-index-reindex-only
metricbeat Metricbeat data 1,079,600 87.7 MB 1.2 GB append-no-conflicts append-no-conflicts
nested StackOverflow Q&A stored as nested docs 11,203,029 663.3 MB 3.4 GB nested-search-challenge nested-search-challenge,index-only
noaa Global daily weather measurements from NOAA 33,659,481 949.4 MB 9.0 GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,aggs,filter-aggs
nyc_taxis Taxi rides in New York in 2015 165,346,692 4.5 GB 74.3 GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts-index-only,update,append-ml,aggs
percolator Percolator benchmark based on AOL queries 2,000,000 121.1 kB 104.9 MB append-no-conflicts append-no-conflicts
pmc Full text benchmark with academic papers from PMC 574,199 5.5 GB 21.7 GB append-no-conflicts append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts,append-fast-with-conflicts,indexing-querying
so Indexing benchmark using up to questions and answers from StackOverflow 36,062,278 8.9 GB 33.1 GB append-no-conflicts append-no-conflicts,transform,frequent-items
so_vector Benchmark for vector search with StackOverflow data 2,000,000 12.3 GB 32.2 GB index-and-search index-and-search
sql SQL query performance based on NOAA Weather data 33,659,481 949.4 MB 9.0 GB sql sql
tsdb metricbeat information for elastic-app k8s cluster 116,633,698 N/A 123.0 GB append-no-conflicts append-no-conflicts,downsample
-------------------------------
[INFO] SUCCESS (took 3 seconds)
-------------------------------
$ esrally list cars
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
Available cars:
Name Type Description
----------------------- ------ --------------------------------------
16gheap car Sets the Java heap to 16GB
1gheap car Sets the Java heap to 1GB
24gheap car Sets the Java heap to 24GB
2gheap car Sets the Java heap to 2GB
4gheap car Sets the Java heap to 4GB
8gheap car Sets the Java heap to 8GB
defaults car Sets the Java heap to 1GB
basic-license mixin Basic License
debug-non-safepoints mixin More accurate CPU profiles
ea mixin Enables Java assertions
fp mixin Preserves frame pointers
g1gc mixin Enables the G1 garbage collector
parallelgc mixin Enables the Parallel garbage collector
trial-license mixin Trial License
unpooled mixin Enables Netty's unpooled allocator
x-pack-ml mixin X-Pack Machine Learning
x-pack-monitoring-http mixin X-Pack Monitoring (HTTP exporter)
x-pack-monitoring-local mixin X-Pack Monitoring (local exporter)
x-pack-security mixin X-Pack Security
zgc mixin Enables the ZGC garbage collector
-------------------------------
[INFO] SUCCESS (took 6 seconds)
-------------------------------
Run the first race with Rally
A “race” in Rally is the execution of a benchmarking experiment. You can choose different benchmarking scenarios (called tracks) for your benchmarks.
Rally should be run as a non-root user. We create a user “es” to run the following race.
$ groupadd es
$ useradd es -g es
$ passwd es
$ cd /home/es
$ su - es
$ export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre
$ esrally race --distribution-version=7.17.0 --track=geonames
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
[INFO] Race id is [d3102b91-ac10-4383-b6a4-7b98d2831af7]
[INFO] Preparing for race ...
[INFO] Downloading Elasticsearch 7.17.0 (297.0 MB total size) [100%]
[INFO] Downloading track data (252.9 MB total size) [100.0%]
[INFO] Decompressing track data from [/home/es/.rally/benchmarks/data/geonames/documents-2.json.bz2] to [/home/es/.rally/benchmarks/data/geonames/documents-2.json] (resulting size: [3.30] GB) ... [OK]
[INFO] Preparing file offset table for [/home/es/.rally/benchmarks/data/geonames/documents-2.json] ... [OK]
[INFO] Racing on track [geonames], challenge [append-no-conflicts] and car ['defaults'] with version [7.17.0].
Running delete-index [100% done]
Running create-index [100% done]
Running check-cluster-health [100% done]
Running index-append [100% done]
Running refresh-after-index [100% done]
Running force-merge [100% done]
Running refresh-after-force-merge [100% done]
Running wait-until-merges-finish [100% done]
Running index-stats [100% done]
Running node-stats [100% done]
Running default [100% done]
Running term [100% done]
Running phrase [100% done]
Running country_agg_uncached [100% done]
Running country_agg_cached [100% done]
Running scroll [100% done]
Running expression [100% done]
Running painless_static [100% done]
Running painless_dynamic [100% done]
Running decay_geo_gauss_function_score [100% done]
Running decay_geo_gauss_script_score [100% done]
Running field_value_function_score [100% done]
Running field_value_script_score [100% done]
Running large_terms [100% done]
Running large_filtered_terms [100% done]
Running large_prohibited_terms [100% done]
Running desc_sort_population [100% done]
Running asc_sort_population [100% done]
Running asc_sort_with_after_population [100% done]
Running desc_sort_geonameid [100% done]
Running desc_sort_with_after_geonameid [100% done]
Running asc_sort_geonameid [100% done]
Running asc_sort_with_after_geonameid [100% done]
------------------------------------------------------
_______ __ _____
/ ____(_)___ ____ _/ / / ___/_________ ________
/ /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \
/ __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
------------------------------------------------------
| Metric | Task | Value | Unit |
|---------------------------------------------------------------:|-------------------------------:|----------------:|--------:|
| Cumulative indexing time of primary shards | | 12.7087 | min |
| Min cumulative indexing time across primary shards | | 0.0196167 | min |
| Median cumulative indexing time across primary shards | | 2.5376 | min |
| Max cumulative indexing time across primary shards | | 2.58153 | min |
| Cumulative indexing throttle time of primary shards | | 0.0108667 | min |
| Min cumulative indexing throttle time across primary shards | | 0 | min |
| Median cumulative indexing throttle time across primary shards | | 0 | min |
| Max cumulative indexing throttle time across primary shards | | 0.00731667 | min |
| Cumulative merge time of primary shards | | 7.13778 | min |
| Cumulative merge count of primary shards | | 52 | |
| Min cumulative merge time across primary shards | | 0 | min |
| Median cumulative merge time across primary shards | | 1.20084 | min |
| Max cumulative merge time across primary shards | | 2.20237 | min |
| Cumulative merge throttle time of primary shards | | 1.59453 | min |
| Min cumulative merge throttle time across primary shards | | 0 | min |
| Median cumulative merge throttle time across primary shards | | 0.2181 | min |
| Max cumulative merge throttle time across primary shards | | 0.569333 | min |
| Cumulative refresh time of primary shards | | 2.89785 | min |
| Cumulative refresh count of primary shards | | 256 | |
| Min cumulative refresh time across primary shards | | 0.00186667 | min |
| Median cumulative refresh time across primary shards | | 0.577325 | min |
| Max cumulative refresh time across primary shards | | 0.599333 | min |
| Cumulative flush time of primary shards | | 0.2239 | min |
| Cumulative flush count of primary shards | | 14 | |
| Min cumulative flush time across primary shards | | 0.0023 | min |
| Median cumulative flush time across primary shards | | 0.0471833 | min |
| Max cumulative flush time across primary shards | | 0.05055 | min |
| Total Young Gen GC time | | 18.57 | s |
| Total Young Gen GC count | | 2243 | |
| Total Old Gen GC time | | 3.541 | s |
| Total Old Gen GC count | | 66 | |
| Store size | | 2.82211 | GB |
| Translog size | | 3.07336e-07 | GB |
| Heap used for segments | | 0.733753 | MB |
| Heap used for doc values | | 0.0489769 | MB |
| Heap used for terms | | 0.557007 | MB |
| Heap used for norms | | 0.0753784 | MB |
| Heap used for points | | 0 | MB |
| Heap used for stored fields | | 0.0523911 | MB |
| Segment count | | 103 | |
| Total Ingest Pipeline count | | 0 | |
| Total Ingest Pipeline time | | 0 | s |
| Total Ingest Pipeline failed | | 0 | |
| Min Throughput | index-append | 87315.7 | docs/s |
| Mean Throughput | index-append | 87373.7 | docs/s |
| Median Throughput | index-append | 87368.1 | docs/s |
| Max Throughput | index-append | 87440.9 | docs/s |
| 50th percentile latency | index-append | 316.067 | ms |
| 90th percentile latency | index-append | 458.448 | ms |
| 99th percentile latency | index-append | 1152.53 | ms |
| 100th percentile latency | index-append | 1316.07 | ms |
| 50th percentile service time | index-append | 316.067 | ms |
| 90th percentile service time | index-append | 458.448 | ms |
| 99th percentile service time | index-append | 1152.53 | ms |
| 100th percentile service time | index-append | 1316.07 | ms |
| error rate | index-append | 0 | % |
| Min Throughput | index-stats | 89.91 | ops/s |
| Mean Throughput | index-stats | 89.95 | ops/s |
| Median Throughput | index-stats | 89.95 | ops/s |
| Max Throughput | index-stats | 89.97 | ops/s |
| 50th percentile latency | index-stats | 4.36948 | ms |
| 90th percentile latency | index-stats | 5.06188 | ms |
| 99th percentile latency | index-stats | 5.51726 | ms |
| 99.9th percentile latency | index-stats | 7.79772 | ms |
| 100th percentile latency | index-stats | 9.64821 | ms |
| 50th percentile service time | index-stats | 3.16338 | ms |
| 90th percentile service time | index-stats | 3.67796 | ms |
| 99th percentile service time | index-stats | 3.86689 | ms |
| 99.9th percentile service time | index-stats | 4.13559 | ms |
| 100th percentile service time | index-stats | 6.9374 | ms |
[..]
----------------------------------
[INFO] SUCCESS (took 4199 seconds)
----------------------------------
You can save this report also to a file by using –report-file=/path/to/your/report.md and save it as CSV with –report-format=csv.
What did Rally just do?
- It downloaded and started Elasticsearch 7.17.0
- It downloaded the relevant data for the geonames track
- It ran the actual benchmark
- And finally it reported the results
Rally Configuration
Rally stores its configuration in the file .rally/rally.ini which is automatically created the first time Rally is executed.
$ pwd
/home/es/.rally/benchmarks
$ ls
data distributions races teams tracks
$ ls distributions/
elasticsearch-7.17.0-linux-x86_64.tar.gz
$ ls tracks/default/
download.sh elastic eql geonames geopoint geopointshape geoshape http_logs metricbeat nested noaa nyc_taxis percolator pmc README.md so sql
$ ls races/d3102b91-ac10-4383-b6a4-7b98d2831af7/
race.json rally-node-0
$ ls teams/default/
cars LICENSE NOTICE plugins README.md
$ ls -l data/geonames/
total 3723472
-rw-rw-r-- 1 es es 3547613828 Oct 27 17:01 documents-2.json
-rw-rw-r-- 1 es es 265208777 Oct 27 17:00 documents-2.json.bz2
-rw-rw-r-- 1 es es 4250 Oct 27 17:02 documents-2.json.offset
$ cat /home/es/.rally/rally.ini
[meta]
config.version = 17
[system]
env.name = local
[node]
root.dir = /home/es/.rally/benchmarks
src.root.dir = /home/es/.rally/benchmarks/src
[source]
remote.repo.url = https://github.com/elastic/elasticsearch.git
elasticsearch.src.subdir = elasticsearch
[benchmarks]
local.dataset.cache = /home/es/.rally/benchmarks/data
[reporting]
datastore.type = in-memory
datastore.host =
datastore.port =
datastore.secure = False
datastore.user =
datastore.password =
[tracks]
default.url = https://github.com/elastic/rally-tracks
[teams]
default.url = https://github.com/elastic/rally-teams
[defaults]
preserve_benchmark_candidate = false
[distributions]
release.cache = true
The benchmark data directory can be changed by modifying root.dir in rally.ini.
- root.dir (default: “~/.rally/benchmarks”): Rally uses this directory to store all benchmark-related data. It assumes that it has complete control over this directory and any of its subdirectories.
- src.root.dir (default: “~/.rally/benchmarks/src”): The directory where the source code of Elasticsearch or any plugins is checked out. Only relevant for benchmarks from sources.
Uninstall python3
The following are the optional commands in the case you need to uninstall python3 in CentOS.
$ whereis python3
python3: /usr/bin/python3 /usr/bin/python3.6 /usr/bin/python3.6m /usr/lib/python3.6 /usr/lib64/python3.6 /usr/local/bin/python3.11 /usr/local/bin/python3.11-config /usr/local/lib/python3.11 /usr/include/python3.6m /usr/share/man/man1/python3.1.gz
$ whereis pip3
pip3: /usr/local/bin/pip3 /usr/local/bin/pip3.8 /usr/local/bin/pip3.10
$ rpm -qa | grep python3 --> Only needed if you installed python3 by yum package installer.
$ whereis python3 |xargs rm -frv
$ whereis pip3 |xargs rm -frv
Troubleshooting
“WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.”
$ pip3.8 install esrally
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”)’: /simple/esrally/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”)’: /simple/esrally/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”)’: /simple/esrally/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”)’: /simple/esrally/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”)’: /simple/esrally/
Could not fetch URL https://pypi.org/simple/esrally/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host=’pypi.org’, port=443): Max retries exceeded with url: /simple/esrally/ (Caused by SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”)) - skipping
ERROR: Could not find a version that satisfies the requirement esrally (from versions: none)
ERROR: No matching distribution found for esrally
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host=’pypi.org’, port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”)) - skipping
WARNING: There was an error checking the latest version of pip.
To fix this, install the python after un-commenting the four lines in Modules/Setup.
$ curl -O https://www.python.org/ftp/python/3.8.1/Python-3.8.1.tgz
$ tar zxf Python-3.8.1.tgz
$ mv Python-3.8.1 /usr/src
$ cd /usr/src/Python-3.8.1/
$ vim Modules/Setup
SSL=/usr/local/ssl
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
-L$(SSL)/lib -lssl -lcrypto
$ ./configure --enable-optimizations
$ make altinstall
$ python3.8 -m ssl
Reference
- https://www.elastic.co/blog/announcing-rally-benchmarking-for-elasticsearch?spm=a2c65.11461447.0.0.e26a498cbHyowi
- https://github.com/elastic/rally
- https://github.com/elastic/rally-tracks
- https://esrally.readthedocs.io/en/stable/quickstart.html
- https://esrally.readthedocs.io/en/stable/install.html
- https://esrally.readthedocs.io/en/stable/configuration.html
- https://esrally.readthedocs.io/en/stable/race.html
- https://esrally.readthedocs.io/en/stable/faq.html
- https://www.python.org/ftp/python/
- https://www.elastic.co/support/matrix#matrix_jvm
- https://stackoverflow.com/questions/56552390/how-to-fix-ssl-module-in-python-is-not-available-in-centos
- https://blog.searchhub.io/how-to-setup-elasticsearch-benchmarking
- https://techviewleo.com/install-java-openjdk-on-rocky-linux-centos/