OpenZFS - A pooled storage file system
OpenZFS
The OpenZFS project is an open source derivative of the Oracle ZFS project. OpenZFS is an outstanding storage platform that encompasses the functionality of traditional filesystems, volume managers, and more, with consistent reliability, functionality and performance across all distributions.
ZFS and ZFS Pooled Storage
The ZFS file system is a revolutionary new file system that fundamentally changes the way file systems are administered, with features and benefits not found in any other file system available today. ZFS is robust, scalable, and easy to administer.
ZFS uses the concept of storage pools to manage physical storage. Historically, file systems were constructed on top of a single physical device. To address multiple devices and provide for data redundancy, the concept of a volume manager was introduced to provide a representation of a single device so that file systems would not need to be modified to take advantage of multiple devices. This design added another layer of complexity and ultimately prevented certain file system advances because the file system had no control over the physical placement of data on the virtualized volumes.
ZFS eliminates volume management altogether. Instead of forcing you to create virtualized volumes, ZFS aggregates devices into a storage pool. The storage pool describes the physical characteristics of the storage (device layout, data redundancy, and so on) and acts as an arbitrary data store from which file systems can be created. File systems are no longer constrained to individual devices, allowing them to share disk space with all file systems in the pool. You no longer need to predetermine the size of a file system, as file systems grow automatically within the disk space allocated to the storage pool. When new storage is added, all file systems within the pool can immediately use the additional disk space without additional work. In many ways, the storage pool works similarly to a virtual memory system: When a memory DIMM is added to a system, the operating system doesn't force you to run commands to configure the memory and assign it to individual processes. All processes on the system automatically use the additional memory.
Install ZFS on CentOS
ZFS is not included by default in CentOS. We will learn how to install it on CentOS 7.9 in this post.
-
Add ZFS repository
$ cat /etc/centos-release CentOS Linux release 7.9.2009 (Core) $ yum install https://zfsonlinux.org/epel/zfs-release.el7_9.noarch.rpm -y -
DKMS vs. kABI
DKMS and kABI are the two methods ZFS module can be loaded into the kernel. We are going to use kABI since it doesn't require kernel re-compilation in case of kernel update. We can enable it by editing the ZFS repository as below.
$ cat /etc/yum.repos.d/zfs.repo [zfs] name=OpenZFS for EL7 - dkms baseurl=http://download.zfsonlinux.org/epel/7.9/$basearch/ enabled=1 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux [zfs-kmod] name=OpenZFS for EL7 - kmod baseurl=http://download.zfsonlinux.org/epel/7.9/kmod/$basearch/ enabled=0 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux [zfs-source] name=OpenZFS for EL7 - Source baseurl=http://download.zfsonlinux.org/epel/7.9/SRPMS/ enabled=0 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux [zfs-next] name=OpenZFS for EL7 - dkms - Next upcoming version baseurl=http://download.zfsonlinux.org/epel-next/7.9/$basearch/ enabled=0 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux [zfs-testing] name=OpenZFS for EL7 - dkms - Testing baseurl=http://download.zfsonlinux.org/epel-testing/7.9/$basearch/ enabled=0 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux [zfs-testing-kmod] name=OpenZFS for EL7 - kmod - Testing baseurl=http://download.zfsonlinux.org/epel-testing/7.9/kmod/$basearch/ enabled=0 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux [zfs-testing-source] name=OpenZFS for EL7 - Testing Source baseurl=http://download.zfsonlinux.org/epel-testing/7.9/SRPMS/ enabled=0 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinuxWe can disable DKMS and enable KABI by editing the enable= in the following two sections.
$ vim /etc/yum.repos.d/zfs.repo [zfs] name=OpenZFS for EL7 - dkms baseurl=http://download.zfsonlinux.org/epel/7.9/$basearch/ enabled=0 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux [zfs-kmod] name=OpenZFS for EL7 - kmod baseurl=http://download.zfsonlinux.org/epel/7.9/kmod/$basearch/ enabled=1 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux -
Install ZFS
Install ZFS using the following command:
$ yum install zfs -y [..] Installed: zfs.x86_64 0:2.0.7-1.el7 Dependency Installed: kmod-zfs.x86_64 0:2.0.7-1.el7 libnvpair3.x86_64 0:2.0.7-1.el7 libuutil3.x86_64 0:2.0.7-1.el7 libzfs4.x86_64 0:2.0.7-1.el7 libzpool4.x86_64 0:2.0.7-1.el7 lm_sensors-libs.x86_64 0:3.4.0-8.20160601gitf9185e5.el7 sysstat.x86_64 0:10.1.5-19.el7Reboot the system to load zfs module:
$ reboot $ lsmod | grep zfsUse the following command to load the ZFS kernel module if it's not loaded after reboot:
$ modprobe zfs $ lsmod | grep zfs zfs 4224878 0 zunicode 331170 1 zfs zzstd 460780 1 zfs zlua 151526 1 zfs zcommon 94285 1 zfs znvpair 94388 2 zfs,zcommon zavl 15698 1 zfs icp 301775 1 zfs spl 96750 6 icp,zfs,zavl,zzstd,zcommon,znvpair $ modinfo zfs filename: /lib/modules/3.10.0-1160.11.1.el7.x86_64/weak-updates/zfs/zfs/zfs.ko version: 2.0.7-1 license: CDDL author: OpenZFS description: ZFS alias: devname:zfs alias: char-major-10-249 retpoline: Y rhelversion: 7.9 srcversion: CDFB8A7F2D3EE43324CF460 depends: spl,znvpair,icp,zlua,zzstd,zunicode,zcommon,zavl vermagic: 3.10.0-1160.49.1.el7.x86_64 SMP mod_unload modversions [..] $ zfs version zfs-2.0.7-1 zfs-kmod-2.0.7-1
Manage ZFS storage pool and file system
-
Create ZFS storage pool
$ zpool create zpooldemo /dev/sdb -
Add disk to ZFS storage pool
$ zpool add zpooldemo /dev/sdc -
Check ZFS pool status
$ zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zpooldemo 119G 174K 119G - - 0% 0% 1.00x ONLINE - $ zpool status pool: zpooldemo state: ONLINE config: NAME STATE READ WRITE CKSUM zpooldemo ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0 errors: No known data errors -
Create ZFS file system
When you create a pool, a ZFS file system is created and mounted automatically. The whole file system space can be used as needed.
$ mount | egrep "zfs" zpooldemo on /zpooldemo type zfs (rw,xattr,noacl) $ df -h | egrep "Filesystem|zpool" Filesystem Size Used Avail Use% Mounted on zpooldemo 116G 128K 116G 1% /zpooldemo $ touch /zpooldemo/testfile $ ls -la /zpooldemo/ total 6 drwxr-xr-x 3 root root 4 Mar 17 22:49 . dr-xr-xr-x. 19 root root 4096 Mar 17 22:37 .. -rw-r--r-- 1 root root 0 Mar 17 22:49 testfileWithin a pool, additional file systems can be created. The new create file systems allow users to manage different sets of data within the same pool.
$ zfs create zpooldemo/zfsdemo $ mount | egrep "zfs" zpooldemo on /zpooldemo type zfs (rw,xattr,noacl) zpooldemo/zfsdemo on /zpooldemo/zfsdemo type zfs (rw,xattr,noacl) $ df -h | egrep "Filesystem|zpool" Filesystem Size Used Avail Use% Mounted on zpooldemo 116G 128K 116G 1% /zpooldemo zpooldemo/zfsdemo 116G 128K 116G 1% /zpooldemo/zfsdemo -
Set ZFS file system properties
The file system property can be set when the ZFS is created.
$ zfs create -o atime=off zpooldemo/zfsdemo $ mount | grep zfs zpooldemo on /zpooldemo type zfs (rw,xattr,noacl) zpooldemo/zfsdemo on /zpooldemo/zfsdemo type zfs (rw,noatime,xattr,noacl)The storage pool and file system properties can be retrieved as below.
$ zpool get all zpooldemo NAME PROPERTY VALUE SOURCE zpooldemo size 59.5G - zpooldemo capacity 0% - zpooldemo altroot - default zpooldemo health ONLINE - zpooldemo guid 11167503015555961412 - zpooldemo version - default zpooldemo bootfs - default zpooldemo delegation on default zpooldemo autoreplace off default zpooldemo cachefile - default zpooldemo failmode wait default zpooldemo listsnapshots off default zpooldemo autoexpand off default zpooldemo dedupratio 1.00x - zpooldemo free 59.5G - zpooldemo allocated 106K - zpooldemo readonly off - zpooldemo ashift 0 default zpooldemo comment - default zpooldemo expandsize - - zpooldemo freeing 0 - zpooldemo fragmentation 0% - zpooldemo leaked 0 - zpooldemo multihost off default zpooldemo checkpoint - - zpooldemo load_guid 10842965729770643306 - zpooldemo autotrim off default zpooldemo feature@async_destroy enabled local zpooldemo feature@empty_bpobj enabled local zpooldemo feature@lz4_compress active local zpooldemo feature@multi_vdev_crash_dump enabled local zpooldemo feature@spacemap_histogram active local zpooldemo feature@enabled_txg active local zpooldemo feature@hole_birth active local zpooldemo feature@extensible_dataset active local zpooldemo feature@embedded_data active local zpooldemo feature@bookmarks enabled local zpooldemo feature@filesystem_limits enabled local zpooldemo feature@large_blocks enabled local zpooldemo feature@large_dnode enabled local zpooldemo feature@sha512 enabled local zpooldemo feature@skein enabled local zpooldemo feature@edonr enabled local zpooldemo feature@userobj_accounting active local zpooldemo feature@encryption enabled local zpooldemo feature@project_quota active local zpooldemo feature@device_removal enabled local zpooldemo feature@obsolete_counts enabled local zpooldemo feature@zpool_checkpoint enabled local zpooldemo feature@spacemap_v2 active local zpooldemo feature@allocation_classes enabled local zpooldemo feature@resilver_defer enabled local zpooldemo feature@bookmark_v2 enabled local zpooldemo feature@redaction_bookmarks enabled local zpooldemo feature@redacted_datasets enabled local zpooldemo feature@bookmark_written enabled local zpooldemo feature@log_spacemap active local zpooldemo feature@livelist enabled local zpooldemo feature@device_rebuild enabled local zpooldemo feature@zstd_compress enabled local $ zfs get all zpooldemo/zfsdemo NAME PROPERTY VALUE SOURCE zpooldemo/zfsdemo type filesystem - zpooldemo/zfsdemo creation Thu Mar 17 23:00 2022 - zpooldemo/zfsdemo used 24K - zpooldemo/zfsdemo available 57.6G - zpooldemo/zfsdemo referenced 24K - zpooldemo/zfsdemo compressratio 1.00x - zpooldemo/zfsdemo mounted yes - zpooldemo/zfsdemo quota none default zpooldemo/zfsdemo reservation none default zpooldemo/zfsdemo recordsize 128K default zpooldemo/zfsdemo mountpoint /zpooldemo/zfsdemo default zpooldemo/zfsdemo sharenfs off default zpooldemo/zfsdemo checksum on default zpooldemo/zfsdemo compression off default zpooldemo/zfsdemo atime off local zpooldemo/zfsdemo devices on default zpooldemo/zfsdemo exec on default zpooldemo/zfsdemo setuid on default zpooldemo/zfsdemo readonly off default zpooldemo/zfsdemo zoned off default zpooldemo/zfsdemo snapdir hidden default zpooldemo/zfsdemo aclmode discard default zpooldemo/zfsdemo aclinherit restricted default zpooldemo/zfsdemo createtxg 20 - zpooldemo/zfsdemo canmount on default zpooldemo/zfsdemo xattr on default zpooldemo/zfsdemo copies 1 default zpooldemo/zfsdemo version 5 - zpooldemo/zfsdemo utf8only off - zpooldemo/zfsdemo normalization none - zpooldemo/zfsdemo casesensitivity sensitive - zpooldemo/zfsdemo vscan off default zpooldemo/zfsdemo nbmand off default zpooldemo/zfsdemo sharesmb off default zpooldemo/zfsdemo refquota none default zpooldemo/zfsdemo refreservation none default zpooldemo/zfsdemo guid 10461278007032944398 - zpooldemo/zfsdemo primarycache all default zpooldemo/zfsdemo secondarycache all default zpooldemo/zfsdemo usedbysnapshots 0B - zpooldemo/zfsdemo usedbydataset 24K - zpooldemo/zfsdemo usedbychildren 0B - zpooldemo/zfsdemo usedbyrefreservation 0B - zpooldemo/zfsdemo logbias latency default zpooldemo/zfsdemo objsetid 136 - zpooldemo/zfsdemo dedup off default zpooldemo/zfsdemo mlslabel none default zpooldemo/zfsdemo sync standard default zpooldemo/zfsdemo dnodesize legacy default zpooldemo/zfsdemo refcompressratio 1.00x - zpooldemo/zfsdemo written 24K - zpooldemo/zfsdemo logicalused 12K - zpooldemo/zfsdemo logicalreferenced 12K - zpooldemo/zfsdemo volmode default default zpooldemo/zfsdemo filesystem_limit none default zpooldemo/zfsdemo snapshot_limit none default zpooldemo/zfsdemo filesystem_count none default zpooldemo/zfsdemo snapshot_count none default zpooldemo/zfsdemo snapdev hidden default zpooldemo/zfsdemo acltype off default zpooldemo/zfsdemo context none default zpooldemo/zfsdemo fscontext none default zpooldemo/zfsdemo defcontext none default zpooldemo/zfsdemo rootcontext none default zpooldemo/zfsdemo relatime off default zpooldemo/zfsdemo redundant_metadata all default zpooldemo/zfsdemo overlay on default zpooldemo/zfsdemo encryption off default zpooldemo/zfsdemo keylocation none default zpooldemo/zfsdemo keyformat none default zpooldemo/zfsdemo pbkdf2iters 0 default zpooldemo/zfsdemo special_small_blocks 0 defaultzfs set command can be used to set any dataset property.
$ zfs set checksum=off zpooldemo/zfsdemozfs get command can be used to retrieve any dataset property.
$ zfs get checksum zpooldemo NAME PROPERTY VALUE SOURCE zpooldemo checksum on default $ zfs get checksum zpooldemo/zfsdemo NAME PROPERTY VALUE SOURCE zpooldemo/zfsdemo checksum off local -
Destroy ZFS storage pool and file system
$ zfs list NAME USED AVAIL REFER MOUNTPOINT zpooldemo 194K 115G 25K /zpooldemo zpooldemo/zfsdemo 24K 115G 24K /zpooldemo/zfsdemo $ zfs destroy zpooldemo/zfsdemo $ zfs list NAME USED AVAIL REFER MOUNTPOINT zpooldemo 169K 115G 25K /zpooldemo$ zpool destroy zpooldemo $ zpool list no pools available
Kernel compatibility
When to install zfs on CentOS, it will check if the already installed kernel version matches the specified the release version. In the following example, the required kernel 3.10.0-1160 will be installed automatically during zfs-release.el7_9 installation.
$ yum install https://zfsonlinux.org/epel/zfs-release.el7_9.noarch.rpm -y
$ yum install zfs -y
Installed:
kernel.x86_64 0:3.10.0-1160.59.1.el7
zfs.x86_64 0:2.0.7-1.el7
Dependency Installed:
kmod-zfs.x86_64 0:2.0.7-1.el7
libnvpair3.x86_64 0:2.0.7-1.el7
libuutil3.x86_64 0:2.0.7-1.el7
libzfs4.x86_64 0:2.0.7-1.el7
libzpool4.x86_64 0:2.0.7-1.el7
$ reboot
$ uname -r
3.10.0-1160.59.1.el7.x86_64
$ lsmod | grep zfs
$ modprobe zfs
$ lsmod | grep zfs
zfs 4224878 0
zunicode 331170 1 zfs
zzstd 460780 1 zfs
zlua 151526 1 zfs
zcommon 94285 1 zfs
znvpair 94388 2 zfs,zcommon
zavl 15698 1 zfs
icp 301775 1 zfs
spl 96750 6 icp,zfs,zavl,zzstd,zcommon,znvpair
$ zfs version
zfs-2.0.7-1
zfs-kmod-2.0.7-1
Uninstall ZFS
Remove the installed rpms and remove the repository as below.
$ rpm -ev <pkg-rpm-name>
$ yum remove zfs-release
