OpenZFS - A pooled storage file system
OpenZFS
The OpenZFS project is an open source derivative of the Oracle ZFS project. OpenZFS is an outstanding storage platform that encompasses the functionality of traditional filesystems, volume managers, and more, with consistent reliability, functionality and performance across all distributions.
ZFS and ZFS Pooled Storage
The ZFS file system is a revolutionary new file system that fundamentally changes the way file systems are administered, with features and benefits not found in any other file system available today. ZFS is robust, scalable, and easy to administer.
ZFS uses the concept of storage pools to manage physical storage. Historically, file systems were constructed on top of a single physical device. To address multiple devices and provide for data redundancy, the concept of a volume manager was introduced to provide a representation of a single device so that file systems would not need to be modified to take advantage of multiple devices. This design added another layer of complexity and ultimately prevented certain file system advances because the file system had no control over the physical placement of data on the virtualized volumes.
ZFS eliminates volume management altogether. Instead of forcing you to create virtualized volumes, ZFS aggregates devices into a storage pool. The storage pool describes the physical characteristics of the storage (device layout, data redundancy, and so on) and acts as an arbitrary data store from which file systems can be created. File systems are no longer constrained to individual devices, allowing them to share disk space with all file systems in the pool. You no longer need to predetermine the size of a file system, as file systems grow automatically within the disk space allocated to the storage pool. When new storage is added, all file systems within the pool can immediately use the additional disk space without additional work. In many ways, the storage pool works similarly to a virtual memory system: When a memory DIMM is added to a system, the operating system doesn’t force you to run commands to configure the memory and assign it to individual processes. All processes on the system automatically use the additional memory.
Install ZFS on CentOS
ZFS is not included by default in CentOS. We will learn how to install it on CentOS 7.9 in this post.
Add ZFS repository
$ cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)$ yum install https://zfsonlinux.org/epel/zfs-release.el7_9.noarch.rpm -y
DKMS vs. kABI
DKMS and kABI are the two methods ZFS module can be loaded into the kernel. We are going to use kABI since it doesn’t require kernel re-compilation in case of kernel update. We can enable it by editing the ZFS repository as below.
$ cat /etc/yum.repos.d/zfs.repo
[zfs]
name=OpenZFS for EL7 - dkms
baseurl=http://download.zfsonlinux.org/epel/7.9/$basearch/
enabled=1
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
[zfs-kmod]
name=OpenZFS for EL7 - kmod
baseurl=http://download.zfsonlinux.org/epel/7.9/kmod/$basearch/
enabled=0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
[zfs-source]
name=OpenZFS for EL7 - Source
baseurl=http://download.zfsonlinux.org/epel/7.9/SRPMS/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
[zfs-next]
name=OpenZFS for EL7 - dkms - Next upcoming version
baseurl=http://download.zfsonlinux.org/epel-next/7.9/$basearch/
enabled=0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
[zfs-testing]
name=OpenZFS for EL7 - dkms - Testing
baseurl=http://download.zfsonlinux.org/epel-testing/7.9/$basearch/
enabled=0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
[zfs-testing-kmod]
name=OpenZFS for EL7 - kmod - Testing
baseurl=http://download.zfsonlinux.org/epel-testing/7.9/kmod/$basearch/
enabled=0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
[zfs-testing-source]
name=OpenZFS for EL7 - Testing Source
baseurl=http://download.zfsonlinux.org/epel-testing/7.9/SRPMS/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
We can disable DKMS and enable KABI by editing the enable= in the following two sections.
$ vim /etc/yum.repos.d/zfs.repo
[zfs]
name=OpenZFS for EL7 - dkms
baseurl=http://download.zfsonlinux.org/epel/7.9/$basearch/
enabled=0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
[zfs-kmod]
name=OpenZFS for EL7 - kmod
baseurl=http://download.zfsonlinux.org/epel/7.9/kmod/$basearch/
enabled=1
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
- Install ZFS
Install ZFS using the following command:
$ yum install zfs -y
[..]
Installed:
zfs.x86_64 0:2.0.7-1.el7
Dependency Installed:
kmod-zfs.x86_64 0:2.0.7-1.el7 libnvpair3.x86_64 0:2.0.7-1.el7 libuutil3.x86_64 0:2.0.7-1.el7 libzfs4.x86_64 0:2.0.7-1.el7 libzpool4.x86_64 0:2.0.7-1.el7 lm_sensors-libs.x86_64 0:3.4.0-8.20160601gitf9185e5.el7
sysstat.x86_64 0:10.1.5-19.el7
Reboot the system to load zfs module:
$ reboot
$ lsmod | grep zfs
Use the following command to load the ZFS kernel module if it’s not loaded after reboot:
$ modprobe zfs
$ lsmod | grep zfs
zfs 4224878 0
zunicode 331170 1 zfs
zzstd 460780 1 zfs
zlua 151526 1 zfs
zcommon 94285 1 zfs
znvpair 94388 2 zfs,zcommon
zavl 15698 1 zfs
icp 301775 1 zfs
spl 96750 6 icp,zfs,zavl,zzstd,zcommon,znvpair
$ modinfo zfs
filename: /lib/modules/3.10.0-1160.11.1.el7.x86_64/weak-updates/zfs/zfs/zfs.ko
version: 2.0.7-1
license: CDDL
author: OpenZFS
description: ZFS
alias: devname:zfs
alias: char-major-10-249
retpoline: Y
rhelversion: 7.9
srcversion: CDFB8A7F2D3EE43324CF460
depends: spl,znvpair,icp,zlua,zzstd,zunicode,zcommon,zavl
vermagic: 3.10.0-1160.49.1.el7.x86_64 SMP mod_unload modversions
[..]
$ zfs version
zfs-2.0.7-1
zfs-kmod-2.0.7-1
Manage ZFS storage pool and file system
Create ZFS storage pool
$ zpool create zpooldemo /dev/sdb
Add disk to ZFS storage pool
$ zpool add zpooldemo /dev/sdc
Check ZFS pool status
$ zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zpooldemo 119G 174K 119G - - 0% 0% 1.00x ONLINE -$ zpool status
pool: zpooldemo
state: ONLINE
config:NAME STATE READ WRITE CKSUM zpooldemo ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0errors: No known data errors
Create ZFS file system
When you create a pool, a ZFS file system is created and mounted automatically. The whole file system space can be used as needed.
$ mount | egrep "zfs"
zpooldemo on /zpooldemo type zfs (rw,xattr,noacl)
$ df -h | egrep "Filesystem|zpool"
Filesystem Size Used Avail Use% Mounted on
zpooldemo 116G 128K 116G 1% /zpooldemo
$ touch /zpooldemo/testfile
$ ls -la /zpooldemo/
total 6
drwxr-xr-x 3 root root 4 Mar 17 22:49 .
dr-xr-xr-x. 19 root root 4096 Mar 17 22:37 ..
-rw-r--r-- 1 root root 0 Mar 17 22:49 testfile
Within a pool, additional file systems can be created. The new create file systems allow users to manage different sets of data within the same pool.
$ zfs create zpooldemo/zfsdemo
$ mount | egrep "zfs"
zpooldemo on /zpooldemo type zfs (rw,xattr,noacl)
zpooldemo/zfsdemo on /zpooldemo/zfsdemo type zfs (rw,xattr,noacl)
$ df -h | egrep "Filesystem|zpool"
Filesystem Size Used Avail Use% Mounted on
zpooldemo 116G 128K 116G 1% /zpooldemo
zpooldemo/zfsdemo 116G 128K 116G 1% /zpooldemo/zfsdemo
- Set ZFS file system properties
The file system property can be set when the ZFS is created.
$ zfs create -o atime=off zpooldemo/zfsdemo
$ mount | grep zfs
zpooldemo on /zpooldemo type zfs (rw,xattr,noacl)
zpooldemo/zfsdemo on /zpooldemo/zfsdemo type zfs (rw,noatime,xattr,noacl)
The storage pool and file system properties can be retrieved as below.
$ zpool get all zpooldemo
NAME PROPERTY VALUE SOURCE
zpooldemo size 59.5G -
zpooldemo capacity 0% -
zpooldemo altroot - default
zpooldemo health ONLINE -
zpooldemo guid 11167503015555961412 -
zpooldemo version - default
zpooldemo bootfs - default
zpooldemo delegation on default
zpooldemo autoreplace off default
zpooldemo cachefile - default
zpooldemo failmode wait default
zpooldemo listsnapshots off default
zpooldemo autoexpand off default
zpooldemo dedupratio 1.00x -
zpooldemo free 59.5G -
zpooldemo allocated 106K -
zpooldemo readonly off -
zpooldemo ashift 0 default
zpooldemo comment - default
zpooldemo expandsize - -
zpooldemo freeing 0 -
zpooldemo fragmentation 0% -
zpooldemo leaked 0 -
zpooldemo multihost off default
zpooldemo checkpoint - -
zpooldemo load_guid 10842965729770643306 -
zpooldemo autotrim off default
zpooldemo feature@async_destroy enabled local
zpooldemo feature@empty_bpobj enabled local
zpooldemo feature@lz4_compress active local
zpooldemo feature@multi_vdev_crash_dump enabled local
zpooldemo feature@spacemap_histogram active local
zpooldemo feature@enabled_txg active local
zpooldemo feature@hole_birth active local
zpooldemo feature@extensible_dataset active local
zpooldemo feature@embedded_data active local
zpooldemo feature@bookmarks enabled local
zpooldemo feature@filesystem_limits enabled local
zpooldemo feature@large_blocks enabled local
zpooldemo feature@large_dnode enabled local
zpooldemo feature@sha512 enabled local
zpooldemo feature@skein enabled local
zpooldemo feature@edonr enabled local
zpooldemo feature@userobj_accounting active local
zpooldemo feature@encryption enabled local
zpooldemo feature@project_quota active local
zpooldemo feature@device_removal enabled local
zpooldemo feature@obsolete_counts enabled local
zpooldemo feature@zpool_checkpoint enabled local
zpooldemo feature@spacemap_v2 active local
zpooldemo feature@allocation_classes enabled local
zpooldemo feature@resilver_defer enabled local
zpooldemo feature@bookmark_v2 enabled local
zpooldemo feature@redaction_bookmarks enabled local
zpooldemo feature@redacted_datasets enabled local
zpooldemo feature@bookmark_written enabled local
zpooldemo feature@log_spacemap active local
zpooldemo feature@livelist enabled local
zpooldemo feature@device_rebuild enabled local
zpooldemo feature@zstd_compress enabled local
$ zfs get all zpooldemo/zfsdemo
NAME PROPERTY VALUE SOURCE
zpooldemo/zfsdemo type filesystem -
zpooldemo/zfsdemo creation Thu Mar 17 23:00 2022 -
zpooldemo/zfsdemo used 24K -
zpooldemo/zfsdemo available 57.6G -
zpooldemo/zfsdemo referenced 24K -
zpooldemo/zfsdemo compressratio 1.00x -
zpooldemo/zfsdemo mounted yes -
zpooldemo/zfsdemo quota none default
zpooldemo/zfsdemo reservation none default
zpooldemo/zfsdemo recordsize 128K default
zpooldemo/zfsdemo mountpoint /zpooldemo/zfsdemo default
zpooldemo/zfsdemo sharenfs off default
zpooldemo/zfsdemo checksum on default
zpooldemo/zfsdemo compression off default
zpooldemo/zfsdemo atime off local
zpooldemo/zfsdemo devices on default
zpooldemo/zfsdemo exec on default
zpooldemo/zfsdemo setuid on default
zpooldemo/zfsdemo readonly off default
zpooldemo/zfsdemo zoned off default
zpooldemo/zfsdemo snapdir hidden default
zpooldemo/zfsdemo aclmode discard default
zpooldemo/zfsdemo aclinherit restricted default
zpooldemo/zfsdemo createtxg 20 -
zpooldemo/zfsdemo canmount on default
zpooldemo/zfsdemo xattr on default
zpooldemo/zfsdemo copies 1 default
zpooldemo/zfsdemo version 5 -
zpooldemo/zfsdemo utf8only off -
zpooldemo/zfsdemo normalization none -
zpooldemo/zfsdemo casesensitivity sensitive -
zpooldemo/zfsdemo vscan off default
zpooldemo/zfsdemo nbmand off default
zpooldemo/zfsdemo sharesmb off default
zpooldemo/zfsdemo refquota none default
zpooldemo/zfsdemo refreservation none default
zpooldemo/zfsdemo guid 10461278007032944398 -
zpooldemo/zfsdemo primarycache all default
zpooldemo/zfsdemo secondarycache all default
zpooldemo/zfsdemo usedbysnapshots 0B -
zpooldemo/zfsdemo usedbydataset 24K -
zpooldemo/zfsdemo usedbychildren 0B -
zpooldemo/zfsdemo usedbyrefreservation 0B -
zpooldemo/zfsdemo logbias latency default
zpooldemo/zfsdemo objsetid 136 -
zpooldemo/zfsdemo dedup off default
zpooldemo/zfsdemo mlslabel none default
zpooldemo/zfsdemo sync standard default
zpooldemo/zfsdemo dnodesize legacy default
zpooldemo/zfsdemo refcompressratio 1.00x -
zpooldemo/zfsdemo written 24K -
zpooldemo/zfsdemo logicalused 12K -
zpooldemo/zfsdemo logicalreferenced 12K -
zpooldemo/zfsdemo volmode default default
zpooldemo/zfsdemo filesystem_limit none default
zpooldemo/zfsdemo snapshot_limit none default
zpooldemo/zfsdemo filesystem_count none default
zpooldemo/zfsdemo snapshot_count none default
zpooldemo/zfsdemo snapdev hidden default
zpooldemo/zfsdemo acltype off default
zpooldemo/zfsdemo context none default
zpooldemo/zfsdemo fscontext none default
zpooldemo/zfsdemo defcontext none default
zpooldemo/zfsdemo rootcontext none default
zpooldemo/zfsdemo relatime off default
zpooldemo/zfsdemo redundant_metadata all default
zpooldemo/zfsdemo overlay on default
zpooldemo/zfsdemo encryption off default
zpooldemo/zfsdemo keylocation none default
zpooldemo/zfsdemo keyformat none default
zpooldemo/zfsdemo pbkdf2iters 0 default
zpooldemo/zfsdemo special_small_blocks 0 default
zfs set command can be used to set any dataset property.
$ zfs set checksum=off zpooldemo/zfsdemo
zfs get command can be used to retrieve any dataset property.
$ zfs get checksum zpooldemo
NAME PROPERTY VALUE SOURCE
zpooldemo checksum on default
$ zfs get checksum zpooldemo/zfsdemo
NAME PROPERTY VALUE SOURCE
zpooldemo/zfsdemo checksum off local
Destroy ZFS storage pool and file system
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
zpooldemo 194K 115G 25K /zpooldemo
zpooldemo/zfsdemo 24K 115G 24K /zpooldemo/zfsdemo$ zfs destroy zpooldemo/zfsdemo
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
zpooldemo 169K 115G 25K /zpooldemo$ zpool destroy zpooldemo
$ zpool list
no pools available
Kernel compatibility
When to install zfs on CentOS, it will check if the already installed kernel version matches the specified the release version. In the following example, the required kernel 3.10.0-1160 will be installed automatically during zfs-release.el7_9 installation.
$ yum install https://zfsonlinux.org/epel/zfs-release.el7_9.noarch.rpm -y
$ yum install zfs -y
Installed:
kernel.x86_64 0:3.10.0-1160.59.1.el7
zfs.x86_64 0:2.0.7-1.el7
Dependency Installed:
kmod-zfs.x86_64 0:2.0.7-1.el7
libnvpair3.x86_64 0:2.0.7-1.el7
libuutil3.x86_64 0:2.0.7-1.el7
libzfs4.x86_64 0:2.0.7-1.el7
libzpool4.x86_64 0:2.0.7-1.el7
$ reboot
$ uname -r
3.10.0-1160.59.1.el7.x86_64
$ lsmod | grep zfs
$ modprobe zfs
$ lsmod | grep zfs
zfs 4224878 0
zunicode 331170 1 zfs
zzstd 460780 1 zfs
zlua 151526 1 zfs
zcommon 94285 1 zfs
znvpair 94388 2 zfs,zcommon
zavl 15698 1 zfs
icp 301775 1 zfs
spl 96750 6 icp,zfs,zavl,zzstd,zcommon,znvpair
$ zfs version
zfs-2.0.7-1
zfs-kmod-2.0.7-1
Uninstall ZFS
Remove the installed rpms and remove the repository as below.
$ rpm -ev <pkg-rpm-name>
$ yum remove zfs-release