Client set up for Rados Block Device ( RBD ) on Linux

Installation

Supported RHEL Based Distributions

AlmaLinux 8

First you must add the key for the Ceph RPM repositories.

rpm --import 'https://download.ceph.com/keys/release.asc'

Next you need to add the following repositories to /etc/yum.repos.d/ceph.repo.

[ceph]
name=Ceph packages for $basearch
baseurl=https://download.ceph.com/rpm-nautilus/el8/$basearch
enabled=1
priority=2
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-nautilus/el8/noarch
enabled=1
priority=2
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-source]
name=Ceph source packages
baseurl=https://download.ceph.com/rpm-nautilus/el8/SRPMS
enabled=0
priority=2
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

Finally, you need to install the ceph-utils package.

dnf install -y ceph-utils

CentOS 7

The CentOS 7 install is very similar to the AlmaLinux 8 install, except that the repository file specifies EL7.

First install the key for the repositories.

rpm --import 'https://download.ceph.com/keys/release.asc'

Next add the EL7 Ceph RPM repositories to /etc/yum.repos.d/ceph.repo.

[ceph]
name=Ceph packages for $basearch
baseurl=https://download.ceph.com/rpm-nautilus/el7/$basearch
enabled=1
priority=2
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-nautilus/el7/noarch
enabled=1
priority=2
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-source]
name=Ceph source packages
baseurl=https://download.ceph.com/rpm-nautilus/el7/SRPMS
enabled=0
priority=2
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

Finally, install the ceph-utils package.

yum install -y ceph-utils

Supported Debian Based Distributions

Debian 11 / Ubuntu 18.04 LTS / Ubuntu 20.04 LTS / Ubuntu 22.04 LTS

The installation process for the Debian based distribution that our team supports is much more simple.  The required package is ceph-base, which exists in the standard repositories. 

In older versions of Debian (version 10 or older) the base repositories don't provide up-to-date packages. As such, for those systems you'll need to acquire more recent versions of the RBD software from 

https://docs.ceph.com/en/latest/install/get-packages/

The following is all that is required to install RBD on the Debian based distributions listed in the title above:

apt update
apt install -y ceph-base

Configuration

Add the Ceph Configuration

This configuration must be put in /etc/ceph/ceph.conf exactly as it is below.

[global]
    mon host = monitor1.softiron.servercontrol.com.au,monitor2.softiron.servercontrol.com.au,monitor3.softiron.servercontrol.com.au

Add the RBD Keyring

You will also need to add the RBD keyring to authenticate with the block device.  In this example the image name (also the namespace) is cid-YYYY-SAU-XXXXX-RS.  You will find both your image name and RBD keyring in your mySAU portal, associated with your block storage product.  Below is an example of a key ring that would be saved at /etc/ceph/ceph.client.cid-YYYY-SAU-XXXXX-RS.keyring.  When saving your keyring, make sure you save the correct image name in it's path.

[client.cid-YYYY-SAU-XXXXX-RS]
        key = redactedforexample==
        caps mon = "profile rbd"
        caps osd = "profile rbd pool=rbd namespace=cid-YYYY-SAU-XXXXX-RS"

Mapping the Block Device

Manually Mapping the Block Device

You can manually map the block device on the server by running a command similar to what is below.  Once again, this example uses the image name cid-YYYY-SAU-XXXXX-RS, and your image name will be different.

rbd map --namespace cid-YYYY-SAU-XXXXX-RS -n client.cid-YYYY-SAU-XXXXX-RS --keyring=/etc/ceph/ceph.client.cid-YYYY-SAU-XXXXX-RS.keyring cid-YYYY-SAU-XXXXX-RS

Automatic Mounting of the Block Device on System Boot

Ideally, you will want the filesystem on the block device to automatically mount to the correct mount point during the boot process.  Please keep in mind that you will need to complete the next section, "Creating the Filesystem", before your system will be able to mount the block device at boot, so please do not reboot your system before completing the next section.

First you will need to create or append an entry for your block device in /etc/ceph/rbdmap.  Below is an example of an entry for rbdmap using cid-YYYY-SAU-XXXXX-RS as the image name.  Once again, your image name will be different, and you can find this information in your mySAU portal.

rbd/cid-YYYY-SAU-XXXXX-RS id=cid-YYYY-SAU-XXXXX-RS,keyring=/etc/ceph/ceph.client.cid-YYYY-SAU-XXXXX-RS.keyring,namespace=cid-YYYY-SAU-XXXXX-RS

After entering your entry to the rbdmap, you will need to start and enable the rbdmap service.

systemctl enable --now rbdmap

Finally, you will need to create an entry into the /etc/fstab file, so the system knows about the filesystem, and where to mount it.  There are three things to consider when creating an fstab entry, block device id, mount point, and filesystem.  In our example we assume the following:

• The block device is /dev/rbd0
• The mount point is /mnt/rbd0
• The filesystem is XFS

/dev/rbd0    /mnt/rbd0    xfs    defaults,noatime,nofail,_netdev    0 2

Creating the Filesystem

Choosing a Filesystem

There are two filesystems used by Linux that are supported by Servers Australia engineers and support technicians, and these are EXT4 and XFS.  Which filesystem is required depends on your use case.  Your choice of filesystem will also impact what command is required to expand the filesystem after you have ordered an upgrade to your block storage.  To help you make this decision, below is a brief summary of each filesystem.

EXT4

• Better for many small files
• Max single file size 16 TiB
• Max filesystem size 50TiB

XFS

• Better for larger files
• Max single file size 8EiB
• Max filesystem size 1PiB
• High scalability for I/O threads

Formatting the Block Device

The examples in this section assume that /dev/rbd0 is the target block device.  If you have multiple Rados Block Devices mapped to your server than rbd0 may not be the correct target when formatting the filesystem.  Formatting an incorrect device can result in the permanent loss of data.  To confirm which RBD image you need to format run the following.

rbd showmapped

The following are the commands to format the block device with EXT4 or XFS.

EXT4

mkfs.ext4 /dev/rbd0

XFS

mkfs.xfs /dev/rbd0

Upgrading RBD Size

After ordering an upgrade on the Rados Block Device, you should see the size change in the system when running lsblk.  Once the size of the block device has been confirmed, you will need to grow whichever filesystem you have chosen to use.  The following are the commands for resizing the two Linux filesystems supported in this documentation, and is assuming that the upgrade is to a filesystem on /dev/rbd0.

Increase EXT4 Filesystem

resize2fs /dev/rbd0

Increase XFS Filesystem

xfs_growfs /dev/rbd0