OpenZFS - A pooled storage file system

OpenZFS - A pooled storage file system
Photo by Mr Cup / Fabien Barral / Unsplash

OpenZFS

The OpenZFS project is an open source derivative of the Oracle ZFS project. OpenZFS is an outstanding storage platform that encompasses the functionality of traditional filesystems, volume managers, and more, with consistent reliability, functionality and performance across all distributions.

Source1
Source2

ZFS and ZFS Pooled Storage

The ZFS file system is a revolutionary new file system that fundamentally changes the way file systems are administered, with features and benefits not found in any other file system available today. ZFS is robust, scalable, and easy to administer.

ZFS uses the concept of storage pools to manage physical storage. Historically, file systems were constructed on top of a single physical device. To address multiple devices and provide for data redundancy, the concept of a volume manager was introduced to provide a representation of a single device so that file systems would not need to be modified to take advantage of multiple devices. This design added another layer of complexity and ultimately prevented certain file system advances because the file system had no control over the physical placement of data on the virtualized volumes.

ZFS eliminates volume management altogether. Instead of forcing you to create virtualized volumes, ZFS aggregates devices into a storage pool. The storage pool describes the physical characteristics of the storage (device layout, data redundancy, and so on) and acts as an arbitrary data store from which file systems can be created. File systems are no longer constrained to individual devices, allowing them to share disk space with all file systems in the pool. You no longer need to predetermine the size of a file system, as file systems grow automatically within the disk space allocated to the storage pool. When new storage is added, all file systems within the pool can immediately use the additional disk space without additional work. In many ways, the storage pool works similarly to a virtual memory system: When a memory DIMM is added to a system, the operating system doesn't force you to run commands to configure the memory and assign it to individual processes. All processes on the system automatically use the additional memory.

Source

Install ZFS on CentOS

ZFS is not included by default in CentOS. We will learn how to install it on CentOS 7.9 in this post.

  1. Add ZFS repository

    $ cat /etc/centos-release
    CentOS Linux release 7.9.2009 (Core)
    
    $ yum install https://zfsonlinux.org/epel/zfs-release.el7_9.noarch.rpm -y
    
  2. DKMS vs. kABI

    DKMS and kABI are the two methods ZFS module can be loaded into the kernel. We are going to use kABI since it doesn't require kernel re-compilation in case of kernel update. We can enable it by editing the ZFS repository as below.

    $ cat /etc/yum.repos.d/zfs.repo
    [zfs]
    name=OpenZFS for EL7 - dkms
    baseurl=http://download.zfsonlinux.org/epel/7.9/$basearch/
    enabled=1
    metadata_expire=7d
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
    [zfs-kmod]
    name=OpenZFS for EL7 - kmod
    baseurl=http://download.zfsonlinux.org/epel/7.9/kmod/$basearch/
    enabled=0
    metadata_expire=7d
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
    [zfs-source]
    name=OpenZFS for EL7 - Source
    baseurl=http://download.zfsonlinux.org/epel/7.9/SRPMS/
    enabled=0
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
    [zfs-next]
    name=OpenZFS for EL7 - dkms - Next upcoming version
    baseurl=http://download.zfsonlinux.org/epel-next/7.9/$basearch/
    enabled=0
    metadata_expire=7d
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
    [zfs-testing]
    name=OpenZFS for EL7 - dkms - Testing
    baseurl=http://download.zfsonlinux.org/epel-testing/7.9/$basearch/
    enabled=0
    metadata_expire=7d
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
    [zfs-testing-kmod]
    name=OpenZFS for EL7 - kmod - Testing
    baseurl=http://download.zfsonlinux.org/epel-testing/7.9/kmod/$basearch/
    enabled=0
    metadata_expire=7d
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
    [zfs-testing-source]
    name=OpenZFS for EL7 - Testing Source
    baseurl=http://download.zfsonlinux.org/epel-testing/7.9/SRPMS/
    enabled=0
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    

    We can disable DKMS and enable KABI by editing the enable= in the following two sections.

    $ vim /etc/yum.repos.d/zfs.repo
    [zfs]
    name=OpenZFS for EL7 - dkms
    baseurl=http://download.zfsonlinux.org/epel/7.9/$basearch/
    enabled=0
    metadata_expire=7d
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
    [zfs-kmod]
    name=OpenZFS for EL7 - kmod
    baseurl=http://download.zfsonlinux.org/epel/7.9/kmod/$basearch/
    enabled=1
    metadata_expire=7d
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
  3. Install ZFS

    Install ZFS using the following command:

    $ yum install zfs -y
    [..]
    Installed:
    zfs.x86_64 0:2.0.7-1.el7
    
    Dependency Installed:
    kmod-zfs.x86_64 0:2.0.7-1.el7      libnvpair3.x86_64 0:2.0.7-1.el7     libuutil3.x86_64 0:2.0.7-1.el7     libzfs4.x86_64 0:2.0.7-1.el7     libzpool4.x86_64 0:2.0.7-1.el7     lm_sensors-libs.x86_64 0:3.4.0-8.20160601gitf9185e5.el7
    sysstat.x86_64 0:10.1.5-19.el7
    

    Reboot the system to load zfs module:

    $ reboot
    $ lsmod | grep zfs
    

    Use the following command to load the ZFS kernel module if it's not loaded after reboot:

    $ modprobe zfs
    
    $ lsmod | grep zfs
    zfs                  4224878  0
    zunicode              331170  1 zfs
    zzstd                 460780  1 zfs
    zlua                  151526  1 zfs
    zcommon                94285  1 zfs
    znvpair                94388  2 zfs,zcommon
    zavl                   15698  1 zfs
    icp                   301775  1 zfs
    spl                    96750  6 icp,zfs,zavl,zzstd,zcommon,znvpair
    
    $ modinfo zfs
    filename:       /lib/modules/3.10.0-1160.11.1.el7.x86_64/weak-updates/zfs/zfs/zfs.ko
    version:        2.0.7-1
    license:        CDDL
    author:         OpenZFS
    description:    ZFS
    alias:          devname:zfs
    alias:          char-major-10-249
    retpoline:      Y
    rhelversion:    7.9
    srcversion:     CDFB8A7F2D3EE43324CF460
    depends:        spl,znvpair,icp,zlua,zzstd,zunicode,zcommon,zavl
    vermagic:       3.10.0-1160.49.1.el7.x86_64 SMP mod_unload modversions
    [..]
    
    $ zfs version
    zfs-2.0.7-1
    zfs-kmod-2.0.7-1
    

Manage ZFS storage pool and file system

  • Create ZFS storage pool

    $ zpool create zpooldemo /dev/sdb
    
  • Add disk to ZFS storage pool

    $ zpool add zpooldemo /dev/sdc
    
  • Check ZFS pool status

    $ zpool list
    NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
    zpooldemo   119G   174K   119G        -         -     0%     0%  1.00x    ONLINE  -
    
    $ zpool status
    pool: zpooldemo
    state: ONLINE
    config:
    
        NAME        STATE     READ WRITE CKSUM
        zpooldemo   ONLINE       0     0     0
        sdb       ONLINE       0     0     0
        sdc       ONLINE       0     0     0
    
    errors: No known data errors
    
  • Create ZFS file system

    When you create a pool, a ZFS file system is created and mounted automatically. The whole file system space can be used as needed.

    $ mount | egrep "zfs"
    zpooldemo on /zpooldemo type zfs (rw,xattr,noacl)
    
    $ df -h | egrep "Filesystem|zpool"
    Filesystem      Size  Used Avail Use% Mounted on
    zpooldemo       116G  128K  116G   1% /zpooldemo
    
    $ touch /zpooldemo/testfile
    $ ls -la /zpooldemo/
    total 6
    drwxr-xr-x   3 root root    4 Mar 17 22:49 .
    dr-xr-xr-x. 19 root root 4096 Mar 17 22:37 ..
    -rw-r--r--   1 root root    0 Mar 17 22:49 testfile
    

    Within a pool, additional file systems can be created. The new create file systems allow users to manage different sets of data within the same pool.

    $ zfs create zpooldemo/zfsdemo
    
    $ mount | egrep "zfs"
    zpooldemo on /zpooldemo type zfs (rw,xattr,noacl)
    zpooldemo/zfsdemo on /zpooldemo/zfsdemo type zfs (rw,xattr,noacl)
    
    $ df -h | egrep "Filesystem|zpool"
    Filesystem         Size  Used Avail Use% Mounted on
    zpooldemo          116G  128K  116G   1% /zpooldemo
    zpooldemo/zfsdemo  116G  128K  116G   1% /zpooldemo/zfsdemo
    
  • Set ZFS file system properties

    The file system property can be set when the ZFS is created.

    $ zfs create -o atime=off zpooldemo/zfsdemo
    $ mount | grep zfs
    zpooldemo on /zpooldemo type zfs (rw,xattr,noacl)
    zpooldemo/zfsdemo on /zpooldemo/zfsdemo type zfs (rw,noatime,xattr,noacl)
    

    The storage pool and file system properties can be retrieved as below.

    $ zpool get all zpooldemo
    NAME       PROPERTY                       VALUE                          SOURCE
    zpooldemo  size                           59.5G                          -
    zpooldemo  capacity                       0%                             -
    zpooldemo  altroot                        -                              default
    zpooldemo  health                         ONLINE                         -
    zpooldemo  guid                           11167503015555961412           -
    zpooldemo  version                        -                              default
    zpooldemo  bootfs                         -                              default
    zpooldemo  delegation                     on                             default
    zpooldemo  autoreplace                    off                            default
    zpooldemo  cachefile                      -                              default
    zpooldemo  failmode                       wait                           default
    zpooldemo  listsnapshots                  off                            default
    zpooldemo  autoexpand                     off                            default
    zpooldemo  dedupratio                     1.00x                          -
    zpooldemo  free                           59.5G                          -
    zpooldemo  allocated                      106K                           -
    zpooldemo  readonly                       off                            -
    zpooldemo  ashift                         0                              default
    zpooldemo  comment                        -                              default
    zpooldemo  expandsize                     -                              -
    zpooldemo  freeing                        0                              -
    zpooldemo  fragmentation                  0%                             -
    zpooldemo  leaked                         0                              -
    zpooldemo  multihost                      off                            default
    zpooldemo  checkpoint                     -                              -
    zpooldemo  load_guid                      10842965729770643306           -
    zpooldemo  autotrim                       off                            default
    zpooldemo  feature@async_destroy          enabled                        local
    zpooldemo  feature@empty_bpobj            enabled                        local
    zpooldemo  feature@lz4_compress           active                         local
    zpooldemo  feature@multi_vdev_crash_dump  enabled                        local
    zpooldemo  feature@spacemap_histogram     active                         local
    zpooldemo  feature@enabled_txg            active                         local
    zpooldemo  feature@hole_birth             active                         local
    zpooldemo  feature@extensible_dataset     active                         local
    zpooldemo  feature@embedded_data          active                         local
    zpooldemo  feature@bookmarks              enabled                        local
    zpooldemo  feature@filesystem_limits      enabled                        local
    zpooldemo  feature@large_blocks           enabled                        local
    zpooldemo  feature@large_dnode            enabled                        local
    zpooldemo  feature@sha512                 enabled                        local
    zpooldemo  feature@skein                  enabled                        local
    zpooldemo  feature@edonr                  enabled                        local
    zpooldemo  feature@userobj_accounting     active                         local
    zpooldemo  feature@encryption             enabled                        local
    zpooldemo  feature@project_quota          active                         local
    zpooldemo  feature@device_removal         enabled                        local
    zpooldemo  feature@obsolete_counts        enabled                        local
    zpooldemo  feature@zpool_checkpoint       enabled                        local
    zpooldemo  feature@spacemap_v2            active                         local
    zpooldemo  feature@allocation_classes     enabled                        local
    zpooldemo  feature@resilver_defer         enabled                        local
    zpooldemo  feature@bookmark_v2            enabled                        local
    zpooldemo  feature@redaction_bookmarks    enabled                        local
    zpooldemo  feature@redacted_datasets      enabled                        local
    zpooldemo  feature@bookmark_written       enabled                        local
    zpooldemo  feature@log_spacemap           active                         local
    zpooldemo  feature@livelist               enabled                        local
    zpooldemo  feature@device_rebuild         enabled                        local
    zpooldemo  feature@zstd_compress          enabled                        local
    
    $ zfs get all zpooldemo/zfsdemo
    NAME               PROPERTY              VALUE                  SOURCE
    zpooldemo/zfsdemo  type                  filesystem             -
    zpooldemo/zfsdemo  creation              Thu Mar 17 23:00 2022  -
    zpooldemo/zfsdemo  used                  24K                    -
    zpooldemo/zfsdemo  available             57.6G                  -
    zpooldemo/zfsdemo  referenced            24K                    -
    zpooldemo/zfsdemo  compressratio         1.00x                  -
    zpooldemo/zfsdemo  mounted               yes                    -
    zpooldemo/zfsdemo  quota                 none                   default
    zpooldemo/zfsdemo  reservation           none                   default
    zpooldemo/zfsdemo  recordsize            128K                   default
    zpooldemo/zfsdemo  mountpoint            /zpooldemo/zfsdemo     default
    zpooldemo/zfsdemo  sharenfs              off                    default
    zpooldemo/zfsdemo  checksum              on                     default
    zpooldemo/zfsdemo  compression           off                    default
    zpooldemo/zfsdemo  atime                 off                    local
    zpooldemo/zfsdemo  devices               on                     default
    zpooldemo/zfsdemo  exec                  on                     default
    zpooldemo/zfsdemo  setuid                on                     default
    zpooldemo/zfsdemo  readonly              off                    default
    zpooldemo/zfsdemo  zoned                 off                    default
    zpooldemo/zfsdemo  snapdir               hidden                 default
    zpooldemo/zfsdemo  aclmode               discard                default
    zpooldemo/zfsdemo  aclinherit            restricted             default
    zpooldemo/zfsdemo  createtxg             20                     -
    zpooldemo/zfsdemo  canmount              on                     default
    zpooldemo/zfsdemo  xattr                 on                     default
    zpooldemo/zfsdemo  copies                1                      default
    zpooldemo/zfsdemo  version               5                      -
    zpooldemo/zfsdemo  utf8only              off                    -
    zpooldemo/zfsdemo  normalization         none                   -
    zpooldemo/zfsdemo  casesensitivity       sensitive              -
    zpooldemo/zfsdemo  vscan                 off                    default
    zpooldemo/zfsdemo  nbmand                off                    default
    zpooldemo/zfsdemo  sharesmb              off                    default
    zpooldemo/zfsdemo  refquota              none                   default
    zpooldemo/zfsdemo  refreservation        none                   default
    zpooldemo/zfsdemo  guid                  10461278007032944398   -
    zpooldemo/zfsdemo  primarycache          all                    default
    zpooldemo/zfsdemo  secondarycache        all                    default
    zpooldemo/zfsdemo  usedbysnapshots       0B                     -
    zpooldemo/zfsdemo  usedbydataset         24K                    -
    zpooldemo/zfsdemo  usedbychildren        0B                     -
    zpooldemo/zfsdemo  usedbyrefreservation  0B                     -
    zpooldemo/zfsdemo  logbias               latency                default
    zpooldemo/zfsdemo  objsetid              136                    -
    zpooldemo/zfsdemo  dedup                 off                    default
    zpooldemo/zfsdemo  mlslabel              none                   default
    zpooldemo/zfsdemo  sync                  standard               default
    zpooldemo/zfsdemo  dnodesize             legacy                 default
    zpooldemo/zfsdemo  refcompressratio      1.00x                  -
    zpooldemo/zfsdemo  written               24K                    -
    zpooldemo/zfsdemo  logicalused           12K                    -
    zpooldemo/zfsdemo  logicalreferenced     12K                    -
    zpooldemo/zfsdemo  volmode               default                default
    zpooldemo/zfsdemo  filesystem_limit      none                   default
    zpooldemo/zfsdemo  snapshot_limit        none                   default
    zpooldemo/zfsdemo  filesystem_count      none                   default
    zpooldemo/zfsdemo  snapshot_count        none                   default
    zpooldemo/zfsdemo  snapdev               hidden                 default
    zpooldemo/zfsdemo  acltype               off                    default
    zpooldemo/zfsdemo  context               none                   default
    zpooldemo/zfsdemo  fscontext             none                   default
    zpooldemo/zfsdemo  defcontext            none                   default
    zpooldemo/zfsdemo  rootcontext           none                   default
    zpooldemo/zfsdemo  relatime              off                    default
    zpooldemo/zfsdemo  redundant_metadata    all                    default
    zpooldemo/zfsdemo  overlay               on                     default
    zpooldemo/zfsdemo  encryption            off                    default
    zpooldemo/zfsdemo  keylocation           none                   default
    zpooldemo/zfsdemo  keyformat             none                   default
    zpooldemo/zfsdemo  pbkdf2iters           0                      default
    zpooldemo/zfsdemo  special_small_blocks  0                      default
    

    zfs set command can be used to set any dataset property.

    $ zfs set checksum=off zpooldemo/zfsdemo
    

    zfs get command can be used to retrieve any dataset property.

    $ zfs get checksum zpooldemo
    NAME       PROPERTY  VALUE      SOURCE
    zpooldemo  checksum  on         default
    
    $ zfs get checksum zpooldemo/zfsdemo
    NAME               PROPERTY  VALUE      SOURCE
    zpooldemo/zfsdemo  checksum  off        local
    
  • Destroy ZFS storage pool and file system

    $ zfs list
    NAME                USED  AVAIL     REFER  MOUNTPOINT
    zpooldemo           194K   115G       25K  /zpooldemo
    zpooldemo/zfsdemo    24K   115G       24K  /zpooldemo/zfsdemo
    
    $ zfs destroy zpooldemo/zfsdemo
    
    $ zfs list
    NAME        USED  AVAIL     REFER  MOUNTPOINT
    zpooldemo   169K   115G       25K  /zpooldemo
    
    $ zpool destroy zpooldemo
    $ zpool list
    no pools available
    

Kernel compatibility

When to install zfs on CentOS, it will check if the already installed kernel version matches the specified the release version. In the following example, the required kernel 3.10.0-1160 will be installed automatically during zfs-release.el7_9 installation.

$ yum install https://zfsonlinux.org/epel/zfs-release.el7_9.noarch.rpm -y
$ yum install zfs -y
Installed:
kernel.x86_64 0:3.10.0-1160.59.1.el7                                                                                       
zfs.x86_64 0:2.0.7-1.el7

Dependency Installed:
kmod-zfs.x86_64 0:2.0.7-1.el7                 
libnvpair3.x86_64 0:2.0.7-1.el7                 
libuutil3.x86_64 0:2.0.7-1.el7                 
libzfs4.x86_64 0:2.0.7-1.el7                 
libzpool4.x86_64 0:2.0.7-1.el7

$ reboot
$ uname -r
3.10.0-1160.59.1.el7.x86_64

$ lsmod  |  grep zfs
$ modprobe zfs
$ lsmod  |  grep zfs
zfs                  4224878  0
zunicode              331170  1 zfs
zzstd                 460780  1 zfs
zlua                  151526  1 zfs
zcommon                94285  1 zfs
znvpair                94388  2 zfs,zcommon
zavl                   15698  1 zfs
icp                   301775  1 zfs
spl                    96750  6 icp,zfs,zavl,zzstd,zcommon,znvpair
$ zfs version
zfs-2.0.7-1
zfs-kmod-2.0.7-1

Uninstall ZFS

Remove the installed rpms and remove the repository as below.

$ rpm -ev <pkg-rpm-name>
$ yum remove zfs-release

Reference