How to uninstall ceph storage cluster

Check the pools, images and OSDs

[ceph: root@host1 /]$ ceph osd tree
ID  CLASS  WEIGHT    TYPE NAME             STATUS  REWEIGHT  PRI-AFF
-1         83.83411  root default
-3         27.94470      host host1
 0    ssd   3.49309          osd.0             up   1.00000  1.00000
 1    ssd   3.49309          osd.1             up   1.00000  1.00000
 2    ssd   3.49309          osd.2             up   1.00000  1.00000
 3    ssd   3.49309          osd.3             up   1.00000  1.00000
 4    ssd   3.49309          osd.4             up   1.00000  1.00000
 5    ssd   3.49309          osd.5             up   1.00000  1.00000
 6    ssd   3.49309          osd.6             up   1.00000  1.00000
 7    ssd   3.49309          osd.7             up   1.00000  1.00000
-5         27.94470      host host2
 8    ssd   3.49309          osd.8             up   1.00000  1.00000
 9    ssd   3.49309          osd.9             up   1.00000  1.00000
10    ssd   3.49309          osd.10            up   1.00000  1.00000
11    ssd   3.49309          osd.11            up   1.00000  1.00000
12    ssd   3.49309          osd.12            up   1.00000  1.00000
13    ssd   3.49309          osd.13            up   1.00000  1.00000
14    ssd   3.49309          osd.14            up   1.00000  1.00000
15    ssd   3.49309          osd.15            up   1.00000  1.00000
-7         27.94470      host host3
16    ssd   3.49309          osd.16            up   1.00000  1.00000
17    ssd   3.49309          osd.17            up   1.00000  1.00000
18    ssd   3.49309          osd.18            up   1.00000  1.00000
19    ssd   3.49309          osd.19            up   1.00000  1.00000
20    ssd   3.49309          osd.20            up   1.00000  1.00000
21    ssd   3.49309          osd.21            up   1.00000  1.00000
22    ssd   3.49309          osd.22            up   1.00000  1.00000
23    ssd   3.49309          osd.23            up   1.00000  1.00000

[ceph: root@host1 /]$ ceph osd lspools
1 device_health_metrics
2 datapool

[ceph: root@host1 /]$ rbd showmapped
id  pool      namespace  image    snap  device
0   datapool             rbdvol1  -     /dev/rbd0
1   datapool             rbdvol2  -     /dev/rbd1
2   datapool             rbdvol3  -     /dev/rbd2
3   datapool             rbdvol4  -     /dev/rbd3

Remove the images and pools

[ceph: root@host1 /]$ rbd unmap /dev/rbd0
[ceph: root@host1 /]$ rbd unmap /dev/rbd1
[ceph: root@host1 /]$ rbd unmap /dev/rbd2
[ceph: root@host1 /]$ rbd unmap /dev/rbd3

[ceph: root@host1 /]$ rbd showmapped

[ceph: root@host1 /]$ rbd rm datapool/rbdvol1
Removing image: 100% complete...done.
[ceph: root@host1 /]$ rbd rm datapool/rbdvol2
Removing image: 100% complete...done.
[ceph: root@host1 /]$ rbd rm datapool/rbdvol3
Removing image: 100% complete...done.
[ceph: root@host1 /]$ rbd rm datapool/rbdvol4
Removing image: 100% complete...done.

[ceph: root@host1 /]$ ceph osd pool rm datapool datapool --yes-i-really-really-mean-it
Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool

[ceph: root@host1 /]$ ceph tell mon.\* injectargs '--mon-allow-pool-delete=true'
mon.host1: mon_allow_pool_delete = 'true'
mon.host1: {}
mon.host3: mon_allow_pool_delete = 'true'
mon.host3: {}
mon.host2: mon_allow_pool_delete = 'true'
mon.host2: {}
[ceph: root@host1 /]$ ceph osd pool rm datapool datapool --yes-i-really-really-mean-it
pool 'datapool' removed

Remove the OSDs

[ceph: root@host1 /]$ for i in `seq 0 23`
> do
> ceph osd down $i && ceph osd destroy $i --force
> done
marked down osd.0.
destroyed osd.0
[omitted...]

[ceph: root@host1 /]$  ceph osd tree
ID  CLASS  WEIGHT    TYPE NAME             STATUS     REWEIGHT  PRI-AFF
-1         83.83411  root default
-3         27.94470      host host1
 0    ssd   3.49309          osd.0         destroyed   1.00000  1.00000
 1    ssd   3.49309          osd.1         destroyed   1.00000  1.00000
 2    ssd   3.49309          osd.2         destroyed   1.00000  1.00000
 3    ssd   3.49309          osd.3         destroyed   1.00000  1.00000
 4    ssd   3.49309          osd.4         destroyed   1.00000  1.00000
 5    ssd   3.49309          osd.5         destroyed   1.00000  1.00000
 6    ssd   3.49309          osd.6         destroyed   1.00000  1.00000
 7    ssd   3.49309          osd.7         destroyed   1.00000  1.00000
-5         27.94470      host host2
 8    ssd   3.49309          osd.8         destroyed   1.00000  1.00000
 9    ssd   3.49309          osd.9         destroyed   1.00000  1.00000
10    ssd   3.49309          osd.10        destroyed   1.00000  1.00000
11    ssd   3.49309          osd.11        destroyed   1.00000  1.00000
12    ssd   3.49309          osd.12        destroyed   1.00000  1.00000
13    ssd   3.49309          osd.13        destroyed   1.00000  1.00000
14    ssd   3.49309          osd.14        destroyed   1.00000  1.00000
15    ssd   3.49309          osd.15        destroyed   1.00000  1.00000
-7         27.94470      host host3
16    ssd   3.49309          osd.16        destroyed   1.00000  1.00000
17    ssd   3.49309          osd.17        destroyed   1.00000  1.00000
18    ssd   3.49309          osd.18        destroyed   1.00000  1.00000
19    ssd   3.49309          osd.19        destroyed   1.00000  1.00000
20    ssd   3.49309          osd.20        destroyed   1.00000  1.00000
21    ssd   3.49309          osd.21        destroyed   1.00000  1.00000
22    ssd   3.49309          osd.22        destroyed   1.00000  1.00000
23    ssd   3.49309          osd.23               up   1.00000  1.00000

Remove the cluster hosts

[ceph: root@host1 /]$ ceph orch host rm host3
Removed host 'host3'
[ceph: root@host1 /]$ ceph orch host rm host2
Removed host 'host2'
[ceph: root@host1 /]$ ceph orch host rm host1
Removed host 'host1'

Check if there is ceph daemon running

[ceph: root@host1 /]$ ceph orch ps host3
No daemons reported
[ceph: root@host1 /]$ ceph orch ps host2
No daemons reported
[ceph: root@host1 /]$ ceph orch ps host1
No daemons reported

Remove the ceph storage cluster

[root@host1 ~]$ cephadm rm-cluster --fsid fec2332e-1b0b-11ec-abbe-ac1f6bc8d268 --force
[root@host1 ~]$ cephadm ls
[]

Cleanup the ceph configuration files

[root@host1 ~]$ rm -rf /etc/ceph
[root@host1 ~]$ rm -rf /var/lib/ce
ceph/       cephadm/    certmonger/
[root@host1 ~]$ rm -rf /var/lib/ceph*

Cleanup the ceph block devices

Do the following on each cluster node.

[root@host1 ~]$ lsblk
NAME                                                                                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme0n1                                                                                               259:0    0  3.5T  0 disk
├─nvme0n1p3                                                                                           259:4    0  3.5T  0 part
│ ├─vgroot-lvswap01                                                                                   253:1    0    4G  0 lvm
│ └─vgroot-lvroot                                                                                     253:0    0  3.5T  0 lvm  /
├─nvme0n1p1                                                                                           259:2    0    1G  0 part /boot/efi
└─nvme0n1p2                                                                                           259:3    0  500M  0 part /boot
nvme3n1                                                                                               259:6    0  3.5T  0 disk
└─ceph--ab144c40--73d6--49bc--921b--65025c383bb1-osd--block--2b965e29--b194--4363--8c96--20ab5b97db33 253:3    0  3.5T  0 lvm
nvme2n1                                                                                               259:5    0  3.5T  0 disk
└─ceph--b1ffe76d--1043--43a2--848b--6ba117e71a75-osd--block--0d6ff85d--9c49--43a0--98a3--c519fbb20b9c 253:4    0  3.5T  0 lvm
nvme1n1                                                                                               259:1    0  3.5T  0 disk

[root@host1 ~]$ for i in `seq 2 9`; do dd if=/dev/zero of=/dev/nvme${i}n1 bs=1M count=1000; done
[root@host1 ~]$ reboot