Cloud Storage

GlusterFS - A distributed file syste

GlusterFS - A distributed file syste
Photo by Pietro Jeng / Unsplash

What is Gluster?

Gluster is a scalable, distributed file system that aggregates disk storage resources from multiple servers into a single global namespace.

Advantages

  • Scales to several petabytes
  • Handles thousands of clients
  • POSIX compatible
  • Uses commodity hardware
  • Can use any ondisk filesystem that supports extended attributes
  • Accessible using industry standard protocols like NFS and SMB
  • Provides replication, quotas, geo-replication, snapshots and bitrot detection
  • Allows optimization for different workloads
  • Open Source

Installation and configuration

  1. To install gluster and start gluster service:

    [root@centos83-1 ~]# cat /etc/centos-release
CentOS Linux release 8.3.2011

[root@centos83-1 ~]# systemctl stop firewalld
[root@centos83-1 ~]# systemctl disable firewalld

[root@centos83-1 ~]# yum install -y centos-release-gluster
[root@centos83-1 ~]# yum install -y glusterfs-server
[root@centos83-1 ~]# rpm -qa |grep gluster
glusterfs-cli-8.3-1.el8.x86_64
libvirt-daemon-driver-storage-gluster-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
glusterfs-client-xlators-8.3-1.el8.x86_64
qemu-kvm-block-gluster-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64
libglusterd0-8.3-1.el8.x86_64
glusterfs-8.3-1.el8.x86_64
pcp-pmda-gluster-5.1.1-3.el8.x86_64
glusterfs-fuse-8.3-1.el8.x86_64
centos-release-gluster8-1.0-1.el8.noarch
libglusterfs0-8.3-1.el8.x86_64
glusterfs-server-8.3-1.el8.x86_64

[root@centos83-1 ~]# systemctl enable glusterd
[root@centos83-1 ~]# systemctl restart glusterd

[root@centos83-1 ~]# systemctl status glusterd
 glusterd.service - GlusterFS, a clustered file-system server
 Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: enabled)
 Active: active (running) since Tue 2021-01-05 17:28:42 PST; 1 months 22 days ago
  Docs: man:glusterd(8)
 Main PID: 1420 (glusterd)
 Tasks: 26 (limit: 409792)
 Memory: 152.3M
 CGroup: /system.slice/glusterd.service

  2. To form a trusted storage pool with the second server:

    [root@centos83-1 ~]# gluster peer probe centos83-2
[root@centos83-1 ~]# gluster peer status
Number of Peers: 1

Hostname: centos83-2
Uuid: b07d3d6e-4d6e-42a9-ad21-018223843fd5
State: Peer in Cluster (Connected)

  3. To create brick on the first server:

    [root@centos83-1 ~]# lsblk | egrep "NAME|sdb"
NAME               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdb                  8:16   0    1T  0 disk

[root@centos83-1 ~]# pvcreate /dev/sdb
[root@centos83-1 ~]# vgcreate vg_bricks /dev/sdb
[root@centos83-1 ~]# lvcreate -L 800g -n gfslv1 vg_bricks
[root@centos83-1 ~]# mkfs.xfs /dev/vg_bricks/gfslv1
[root@centos83-1 ~]# mkdir -p /bricks/vm1_brick1
[root@centos83-1 ~]# vim /etc/fstab
/dev/vg_bricks/gfslv1 /bricks/vm1_brick1        xfs     defaults        0 0
[root@centos83-1 ~]# mount -a
[root@centos83-1 ~]# df -h |grep gfs
/dev/mapper/vg_bricks-gfslv1  800G  5.7G  794G   1% /bricks/vm1_brick1

  4. To create brick on the second server:

    [root@centos83-2 ~]# lsblk | egrep "NAME|sdb"
NAME               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdb                  8:16   0    1T  0 disk

[root@centos83-2 ~]# pvcreate /dev/sdb
[root@centos83-2 ~]# vgcreate vg_bricks /dev/sdb
[root@centos83-2 ~]# lvcreate -L 800g -n gfslv1 vg_bricks
[root@centos83-2 ~]# mkfs.xfs /dev/vg_bricks/gfslv1
[root@centos83-2 ~]# mkdir -p /bricks/vm2_brick1
[root@centos83-2 ~]# vim /etc/fstab
/dev/vg_bricks/gfslv1 /bricks/vm2_brick1        xfs     defaults        0 0
[root@centos83-2 ~]# mount -a
[root@centos83-2 ~]# df -h |grep gfs
/dev/mapper/vg_bricks-gfslv1  800G  5.7G  794G   1% /bricks/vm2_brick1

  5. To create distributed volume with the two bricks which are created on the two nodes:

    [root@centos83-1 ~]# gluster volume create gv0 centos83-1:/bricks/vm1_brick1/gv0 centos83-2:/bricks/vm2_brick1/gv0
[root@centos83-1 ~]# gluster volume start gv0

  6. To verify the volume status:

    [root@centos83-1 ~]#  gluster volume info gv0

Volume Name: gv0
Type: Distribute
Volume ID: ee08d16a-f940-4ec2-aba8-5f1fcfe41bd4
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: centos83-1:/bricks/vm1_brick1/gv0
Brick2: centos83-2:/bricks/vm2_brick1/gv0
Options Reconfigured:
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on

  7. To mount the distributed volume on one of the servers(treat it as client for simple demonstration):

    [root@centos83-1 ~]# mkdir /testmnt
[root@centos83-1 ~]# mount -t glusterfs centos83-2:/gv0 /testmnt
[root@centos83-1 ~]# df -h | grep testmnt
centos83-2:/gv0               1.6T   28G  1.6T   2% /testmnt

    As shown above, the usable storage size is the sum of the brick size from two nodes.

References

Read next