Kubernetes and Gluster performance

Kubernetes and Gluster Intro

Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.

Gluster is a scalable, distributed file system that aggregates disk storage resources from multiple servers into a single global namespace.

Gluster performance study

In this article, we will discuss the Gluster performance in a Docker container environment which is built on Kubernetes.


We use three Redhat Linux servers to form a Kubernetes cluster in this study. A Kubernetes cluster that handles production traffic should have a minimum of three nodes.

We have three Docker containers(application instances) provisioned within the Kubernetes cluster. Each container instance has its own Gluster storage pool.

[node1:root]~> kubectl get services
kubernetes   ClusterIP    <none>        443/TCP   6d14h
[node1:root]~> kubectl get nodes
node1   Ready    master   6d14h   v1.19.3
node2   Ready    master   6d14h   v1.19.3
node3   Ready    master   6d14h   v1.19.3
[node1:root]~> kubectl get pods --namespace ns-1
container1   1/1     Running   0          6d14h
container2   1/1     Running   0          6d14h
container3   1/1     Running   0          6d14h
[node1:root]~> gluster pool list
UUID                    Hostname    State
45d8ec04-4e7a-4442-bbb4-557256b864d6 Connected
875be270-ae69-45ea-b38e-2768b7c6ce05 Connected
f2696790-b305-4099-8dba-b31d23b0beac    localhost Connected

Workload and performance

We keep increasing the number of workload processes across three container instances and measure the throughput in MB/s. For each workload process, it ingests data from multiple clients through 10GbE bonding network and writes the data to the mounted Gluster filesystem.

There are two kinds of workloads. One is very I/O intensive and the other is CPU bound.


  • For the CPU bound workload, the performance scales very well.
  • For the I/O bound workload, the performance does not scale when the number of workload processes increases.


As we increase the number of workload processes, the workload can be distributed evenly across three instances(on three nodes). Thus, the CPU bandwidth from three nodes are available for the application computing.

Although the storage from three nodes are usable for three container instances, with the default disk allocation for Gluster filesystem, the I/O performance may not be optimal. It depends on how the disks are allocated to each Gluster filesystem bricks and how the bricks are assigned to the Gluster filesystems. Also, writing to remote disk would have worse performance due to network latency.

The following is the disk mapping to bricks which are used by one of the three Gluster filesystems. It shows four bricks are created on the same disk /dev/sde on the node Obviously, the I/O performance could be degraded if multiple processes write on it.

brickSize(GB)	device	node
3200	/dev/sdd
3200	/dev/sde
3200	/dev/sdd
3200	/dev/sdd
3200	/dev/sdc
3200	/dev/sdc
3200	/dev/sdb
3200	/dev/sdb
3200	/dev/sde
3200	/dev/sde
3200	/dev/sde
3200	/dev/sde


From this case study, we had basic understanding on how the Gluster file system works with storage across multiple nodes. The default Gluster filesystem layout is not fit for all use cases. A custom storage layout would be needed to meet the performance requirement.