Home OpenZFS - A pooled storage file system
Post
Cancel

OpenZFS - A pooled storage file system

OpenZFS

The OpenZFS project is an open source derivative of the Oracle ZFS project. OpenZFS is an outstanding storage platform that encompasses the functionality of traditional filesystems, volume managers, and more, with consistent reliability, functionality and performance across all distributions.

Source1 Source2

ZFS and ZFS Pooled Storage

The ZFS file system is a revolutionary new file system that fundamentally changes the way file systems are administered, with features and benefits not found in any other file system available today. ZFS is robust, scalable, and easy to administer.

ZFS uses the concept of storage pools to manage physical storage. Historically, file systems were constructed on top of a single physical device. To address multiple devices and provide for data redundancy, the concept of a volume manager was introduced to provide a representation of a single device so that file systems would not need to be modified to take advantage of multiple devices. This design added another layer of complexity and ultimately prevented certain file system advances because the file system had no control over the physical placement of data on the virtualized volumes.

ZFS eliminates volume management altogether. Instead of forcing you to create virtualized volumes, ZFS aggregates devices into a storage pool. The storage pool describes the physical characteristics of the storage (device layout, data redundancy, and so on) and acts as an arbitrary data store from which file systems can be created. File systems are no longer constrained to individual devices, allowing them to share disk space with all file systems in the pool. You no longer need to predetermine the size of a file system, as file systems grow automatically within the disk space allocated to the storage pool. When new storage is added, all file systems within the pool can immediately use the additional disk space without additional work. In many ways, the storage pool works similarly to a virtual memory system: When a memory DIMM is added to a system, the operating system doesn’t force you to run commands to configure the memory and assign it to individual processes. All processes on the system automatically use the additional memory.

Source

Install ZFS on CentOS

ZFS is not included by default in CentOS. We will learn how to install it on CentOS 7.9 in this post.

  1. Add ZFS repository

    1
    2
    3
    4
    
     $ cat /etc/centos-release
     CentOS Linux release 7.9.2009 (Core)
    
     $ yum install https://zfsonlinux.org/epel/zfs-release.el7_9.noarch.rpm -y
    
  2. DKMS vs. kABI

    DKMS and kABI are the two methods ZFS module can be loaded into the kernel. We are going to use kABI since it doesn’t require kernel re-compilation in case of kernel update. We can enable it by editing the ZFS repository as below.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    
     $ cat /etc/yum.repos.d/zfs.repo
     [zfs]
     name=OpenZFS for EL7 - dkms
     baseurl=http://download.zfsonlinux.org/epel/7.9/$basearch/
     enabled=1
     metadata_expire=7d
     gpgcheck=1
     gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
     [zfs-kmod]
     name=OpenZFS for EL7 - kmod
     baseurl=http://download.zfsonlinux.org/epel/7.9/kmod/$basearch/
     enabled=0
     metadata_expire=7d
     gpgcheck=1
     gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
     [zfs-source]
     name=OpenZFS for EL7 - Source
     baseurl=http://download.zfsonlinux.org/epel/7.9/SRPMS/
     enabled=0
     gpgcheck=1
     gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
     [zfs-next]
     name=OpenZFS for EL7 - dkms - Next upcoming version
     baseurl=http://download.zfsonlinux.org/epel-next/7.9/$basearch/
     enabled=0
     metadata_expire=7d
     gpgcheck=1
     gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
     [zfs-testing]
     name=OpenZFS for EL7 - dkms - Testing
     baseurl=http://download.zfsonlinux.org/epel-testing/7.9/$basearch/
     enabled=0
     metadata_expire=7d
     gpgcheck=1
     gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
     [zfs-testing-kmod]
     name=OpenZFS for EL7 - kmod - Testing
     baseurl=http://download.zfsonlinux.org/epel-testing/7.9/kmod/$basearch/
     enabled=0
     metadata_expire=7d
     gpgcheck=1
     gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
     [zfs-testing-source]
     name=OpenZFS for EL7 - Testing Source
     baseurl=http://download.zfsonlinux.org/epel-testing/7.9/SRPMS/
     enabled=0
     gpgcheck=1
     gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    

    We can disable DKMS and enable KABI by editing the enable= in the following two sections.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    
     $ vim /etc/yum.repos.d/zfs.repo
     [zfs]
     name=OpenZFS for EL7 - dkms
     baseurl=http://download.zfsonlinux.org/epel/7.9/$basearch/
     enabled=0
     metadata_expire=7d
     gpgcheck=1
     gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
     [zfs-kmod]
     name=OpenZFS for EL7 - kmod
     baseurl=http://download.zfsonlinux.org/epel/7.9/kmod/$basearch/
     enabled=1
     metadata_expire=7d
     gpgcheck=1
     gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
    
  3. Install ZFS

    Install ZFS using the following command:

    1
    2
    3
    4
    5
    6
    7
    8
    
     $ yum install zfs -y
     [..]
     Installed:
     zfs.x86_64 0:2.0.7-1.el7
    
     Dependency Installed:
     kmod-zfs.x86_64 0:2.0.7-1.el7      libnvpair3.x86_64 0:2.0.7-1.el7     libuutil3.x86_64 0:2.0.7-1.el7     libzfs4.x86_64 0:2.0.7-1.el7     libzpool4.x86_64 0:2.0.7-1.el7     lm_sensors-libs.x86_64 0:3.4.0-8.20160601gitf9185e5.el7
     sysstat.x86_64 0:10.1.5-19.el7
    

    Reboot the system to load zfs module:

    1
    2
    
     $ reboot
     $ lsmod | grep zfs
    

    Use the following command to load the ZFS kernel module if it’s not loaded after reboot:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    
     $ modprobe zfs
    
     $ lsmod | grep zfs
     zfs                  4224878  0
     zunicode              331170  1 zfs
     zzstd                 460780  1 zfs
     zlua                  151526  1 zfs
     zcommon                94285  1 zfs
     znvpair                94388  2 zfs,zcommon
     zavl                   15698  1 zfs
     icp                   301775  1 zfs
     spl                    96750  6 icp,zfs,zavl,zzstd,zcommon,znvpair
    
     $ modinfo zfs
     filename:       /lib/modules/3.10.0-1160.11.1.el7.x86_64/weak-updates/zfs/zfs/zfs.ko
     version:        2.0.7-1
     license:        CDDL
     author:         OpenZFS
     description:    ZFS
     alias:          devname:zfs
     alias:          char-major-10-249
     retpoline:      Y
     rhelversion:    7.9
     srcversion:     CDFB8A7F2D3EE43324CF460
     depends:        spl,znvpair,icp,zlua,zzstd,zunicode,zcommon,zavl
     vermagic:       3.10.0-1160.49.1.el7.x86_64 SMP mod_unload modversions
     [..]
    
     $ zfs version
     zfs-2.0.7-1
     zfs-kmod-2.0.7-1
    

Manage ZFS storage pool and file system

  • Create ZFS storage pool

    1
    
      $ zpool create zpooldemo /dev/sdb
    
  • Add disk to ZFS storage pool

    1
    
      $ zpool add zpooldemo /dev/sdc
    
  • Check ZFS pool status

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    
      $ zpool list
      NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
      zpooldemo   119G   174K   119G        -         -     0%     0%  1.00x    ONLINE  -
    
      $ zpool status
      pool: zpooldemo
      state: ONLINE
      config:
    
          NAME        STATE     READ WRITE CKSUM
          zpooldemo   ONLINE       0     0     0
          sdb       ONLINE       0     0     0
          sdc       ONLINE       0     0     0
    
      errors: No known data errors
    
  • Create ZFS file system

    When you create a pool, a ZFS file system is created and mounted automatically. The whole file system space can be used as needed.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    
      $ mount | egrep "zfs"
      zpooldemo on /zpooldemo type zfs (rw,xattr,noacl)
    
      $ df -h | egrep "Filesystem|zpool"
      Filesystem      Size  Used Avail Use% Mounted on
      zpooldemo       116G  128K  116G   1% /zpooldemo
    
      $ touch /zpooldemo/testfile
      $ ls -la /zpooldemo/
      total 6
      drwxr-xr-x   3 root root    4 Mar 17 22:49 .
      dr-xr-xr-x. 19 root root 4096 Mar 17 22:37 ..
      -rw-r--r--   1 root root    0 Mar 17 22:49 testfile
    

    Within a pool, additional file systems can be created. The new create file systems allow users to manage different sets of data within the same pool.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    
      $ zfs create zpooldemo/zfsdemo
    
      $ mount | egrep "zfs"
      zpooldemo on /zpooldemo type zfs (rw,xattr,noacl)
      zpooldemo/zfsdemo on /zpooldemo/zfsdemo type zfs (rw,xattr,noacl)
    
      $ df -h | egrep "Filesystem|zpool"
      Filesystem         Size  Used Avail Use% Mounted on
      zpooldemo          116G  128K  116G   1% /zpooldemo
      zpooldemo/zfsdemo  116G  128K  116G   1% /zpooldemo/zfsdemo
    
  • Set ZFS file system properties

    The file system property can be set when the ZFS is created.

    1
    2
    3
    4
    
      $ zfs create -o atime=off zpooldemo/zfsdemo
      $ mount | grep zfs
      zpooldemo on /zpooldemo type zfs (rw,xattr,noacl)
      zpooldemo/zfsdemo on /zpooldemo/zfsdemo type zfs (rw,noatime,xattr,noacl)
    

    The storage pool and file system properties can be retrieved as below.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    
      $ zpool get all zpooldemo
      NAME       PROPERTY                       VALUE                          SOURCE
      zpooldemo  size                           59.5G                          -
      zpooldemo  capacity                       0%                             -
      zpooldemo  altroot                        -                              default
      zpooldemo  health                         ONLINE                         -
      zpooldemo  guid                           11167503015555961412           -
      zpooldemo  version                        -                              default
      zpooldemo  bootfs                         -                              default
      zpooldemo  delegation                     on                             default
      zpooldemo  autoreplace                    off                            default
      zpooldemo  cachefile                      -                              default
      zpooldemo  failmode                       wait                           default
      zpooldemo  listsnapshots                  off                            default
      zpooldemo  autoexpand                     off                            default
      zpooldemo  dedupratio                     1.00x                          -
      zpooldemo  free                           59.5G                          -
      zpooldemo  allocated                      106K                           -
      zpooldemo  readonly                       off                            -
      zpooldemo  ashift                         0                              default
      zpooldemo  comment                        -                              default
      zpooldemo  expandsize                     -                              -
      zpooldemo  freeing                        0                              -
      zpooldemo  fragmentation                  0%                             -
      zpooldemo  leaked                         0                              -
      zpooldemo  multihost                      off                            default
      zpooldemo  checkpoint                     -                              -
      zpooldemo  load_guid                      10842965729770643306           -
      zpooldemo  autotrim                       off                            default
      zpooldemo  feature@async_destroy          enabled                        local
      zpooldemo  feature@empty_bpobj            enabled                        local
      zpooldemo  feature@lz4_compress           active                         local
      zpooldemo  feature@multi_vdev_crash_dump  enabled                        local
      zpooldemo  feature@spacemap_histogram     active                         local
      zpooldemo  feature@enabled_txg            active                         local
      zpooldemo  feature@hole_birth             active                         local
      zpooldemo  feature@extensible_dataset     active                         local
      zpooldemo  feature@embedded_data          active                         local
      zpooldemo  feature@bookmarks              enabled                        local
      zpooldemo  feature@filesystem_limits      enabled                        local
      zpooldemo  feature@large_blocks           enabled                        local
      zpooldemo  feature@large_dnode            enabled                        local
      zpooldemo  feature@sha512                 enabled                        local
      zpooldemo  feature@skein                  enabled                        local
      zpooldemo  feature@edonr                  enabled                        local
      zpooldemo  feature@userobj_accounting     active                         local
      zpooldemo  feature@encryption             enabled                        local
      zpooldemo  feature@project_quota          active                         local
      zpooldemo  feature@device_removal         enabled                        local
      zpooldemo  feature@obsolete_counts        enabled                        local
      zpooldemo  feature@zpool_checkpoint       enabled                        local
      zpooldemo  feature@spacemap_v2            active                         local
      zpooldemo  feature@allocation_classes     enabled                        local
      zpooldemo  feature@resilver_defer         enabled                        local
      zpooldemo  feature@bookmark_v2            enabled                        local
      zpooldemo  feature@redaction_bookmarks    enabled                        local
      zpooldemo  feature@redacted_datasets      enabled                        local
      zpooldemo  feature@bookmark_written       enabled                        local
      zpooldemo  feature@log_spacemap           active                         local
      zpooldemo  feature@livelist               enabled                        local
      zpooldemo  feature@device_rebuild         enabled                        local
      zpooldemo  feature@zstd_compress          enabled                        local
    
      $ zfs get all zpooldemo/zfsdemo
      NAME               PROPERTY              VALUE                  SOURCE
      zpooldemo/zfsdemo  type                  filesystem             -
      zpooldemo/zfsdemo  creation              Thu Mar 17 23:00 2022  -
      zpooldemo/zfsdemo  used                  24K                    -
      zpooldemo/zfsdemo  available             57.6G                  -
      zpooldemo/zfsdemo  referenced            24K                    -
      zpooldemo/zfsdemo  compressratio         1.00x                  -
      zpooldemo/zfsdemo  mounted               yes                    -
      zpooldemo/zfsdemo  quota                 none                   default
      zpooldemo/zfsdemo  reservation           none                   default
      zpooldemo/zfsdemo  recordsize            128K                   default
      zpooldemo/zfsdemo  mountpoint            /zpooldemo/zfsdemo     default
      zpooldemo/zfsdemo  sharenfs              off                    default
      zpooldemo/zfsdemo  checksum              on                     default
      zpooldemo/zfsdemo  compression           off                    default
      zpooldemo/zfsdemo  atime                 off                    local
      zpooldemo/zfsdemo  devices               on                     default
      zpooldemo/zfsdemo  exec                  on                     default
      zpooldemo/zfsdemo  setuid                on                     default
      zpooldemo/zfsdemo  readonly              off                    default
      zpooldemo/zfsdemo  zoned                 off                    default
      zpooldemo/zfsdemo  snapdir               hidden                 default
      zpooldemo/zfsdemo  aclmode               discard                default
      zpooldemo/zfsdemo  aclinherit            restricted             default
      zpooldemo/zfsdemo  createtxg             20                     -
      zpooldemo/zfsdemo  canmount              on                     default
      zpooldemo/zfsdemo  xattr                 on                     default
      zpooldemo/zfsdemo  copies                1                      default
      zpooldemo/zfsdemo  version               5                      -
      zpooldemo/zfsdemo  utf8only              off                    -
      zpooldemo/zfsdemo  normalization         none                   -
      zpooldemo/zfsdemo  casesensitivity       sensitive              -
      zpooldemo/zfsdemo  vscan                 off                    default
      zpooldemo/zfsdemo  nbmand                off                    default
      zpooldemo/zfsdemo  sharesmb              off                    default
      zpooldemo/zfsdemo  refquota              none                   default
      zpooldemo/zfsdemo  refreservation        none                   default
      zpooldemo/zfsdemo  guid                  10461278007032944398   -
      zpooldemo/zfsdemo  primarycache          all                    default
      zpooldemo/zfsdemo  secondarycache        all                    default
      zpooldemo/zfsdemo  usedbysnapshots       0B                     -
      zpooldemo/zfsdemo  usedbydataset         24K                    -
      zpooldemo/zfsdemo  usedbychildren        0B                     -
      zpooldemo/zfsdemo  usedbyrefreservation  0B                     -
      zpooldemo/zfsdemo  logbias               latency                default
      zpooldemo/zfsdemo  objsetid              136                    -
      zpooldemo/zfsdemo  dedup                 off                    default
      zpooldemo/zfsdemo  mlslabel              none                   default
      zpooldemo/zfsdemo  sync                  standard               default
      zpooldemo/zfsdemo  dnodesize             legacy                 default
      zpooldemo/zfsdemo  refcompressratio      1.00x                  -
      zpooldemo/zfsdemo  written               24K                    -
      zpooldemo/zfsdemo  logicalused           12K                    -
      zpooldemo/zfsdemo  logicalreferenced     12K                    -
      zpooldemo/zfsdemo  volmode               default                default
      zpooldemo/zfsdemo  filesystem_limit      none                   default
      zpooldemo/zfsdemo  snapshot_limit        none                   default
      zpooldemo/zfsdemo  filesystem_count      none                   default
      zpooldemo/zfsdemo  snapshot_count        none                   default
      zpooldemo/zfsdemo  snapdev               hidden                 default
      zpooldemo/zfsdemo  acltype               off                    default
      zpooldemo/zfsdemo  context               none                   default
      zpooldemo/zfsdemo  fscontext             none                   default
      zpooldemo/zfsdemo  defcontext            none                   default
      zpooldemo/zfsdemo  rootcontext           none                   default
      zpooldemo/zfsdemo  relatime              off                    default
      zpooldemo/zfsdemo  redundant_metadata    all                    default
      zpooldemo/zfsdemo  overlay               on                     default
      zpooldemo/zfsdemo  encryption            off                    default
      zpooldemo/zfsdemo  keylocation           none                   default
      zpooldemo/zfsdemo  keyformat             none                   default
      zpooldemo/zfsdemo  pbkdf2iters           0                      default
      zpooldemo/zfsdemo  special_small_blocks  0                      default
    

    zfs set command can be used to set any dataset property.

    1
    
      $ zfs set checksum=off zpooldemo/zfsdemo
    

    zfs get command can be used to retrieve any dataset property.

    1
    2
    3
    4
    5
    6
    7
    
      $ zfs get checksum zpooldemo
      NAME       PROPERTY  VALUE      SOURCE
      zpooldemo  checksum  on         default
    
      $ zfs get checksum zpooldemo/zfsdemo
      NAME               PROPERTY  VALUE      SOURCE
      zpooldemo/zfsdemo  checksum  off        local
    
  • Destroy ZFS storage pool and file system

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    
      $ zfs list
      NAME                USED  AVAIL     REFER  MOUNTPOINT
      zpooldemo           194K   115G       25K  /zpooldemo
      zpooldemo/zfsdemo    24K   115G       24K  /zpooldemo/zfsdemo
    
      $ zfs destroy zpooldemo/zfsdemo
    
      $ zfs list
      NAME        USED  AVAIL     REFER  MOUNTPOINT
      zpooldemo   169K   115G       25K  /zpooldemo
    
    1
    2
    3
    
      $ zpool destroy zpooldemo
      $ zpool list
      no pools available
    

Kernel compatibility

When to install zfs on CentOS, it will check if the already installed kernel version matches the specified the release version. In the following example, the required kernel 3.10.0-1160 will be installed automatically during zfs-release.el7_9 installation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
$ yum install https://zfsonlinux.org/epel/zfs-release.el7_9.noarch.rpm -y
$ yum install zfs -y
Installed:
kernel.x86_64 0:3.10.0-1160.59.1.el7                                                                                       
zfs.x86_64 0:2.0.7-1.el7

Dependency Installed:
kmod-zfs.x86_64 0:2.0.7-1.el7                 
libnvpair3.x86_64 0:2.0.7-1.el7                 
libuutil3.x86_64 0:2.0.7-1.el7                 
libzfs4.x86_64 0:2.0.7-1.el7                 
libzpool4.x86_64 0:2.0.7-1.el7

$ reboot
$ uname -r
3.10.0-1160.59.1.el7.x86_64

$ lsmod  |  grep zfs
$ modprobe zfs
$ lsmod  |  grep zfs
zfs                  4224878  0
zunicode              331170  1 zfs
zzstd                 460780  1 zfs
zlua                  151526  1 zfs
zcommon                94285  1 zfs
znvpair                94388  2 zfs,zcommon
zavl                   15698  1 zfs
icp                   301775  1 zfs
spl                    96750  6 icp,zfs,zavl,zzstd,zcommon,znvpair
$ zfs version
zfs-2.0.7-1
zfs-kmod-2.0.7-1

Uninstall ZFS

Remove the installed rpms and remove the repository as below.

1
2
$ rpm -ev <pkg-rpm-name>
$ yum remove zfs-release

Reference

This post is licensed under CC BY 4.0 by the author.

Fio initial write, overwrite and append write

Useful tools in Linux operating system

Comments powered by Disqus.