Setting up your CephFS share on Linux

Install the Ceph Client

Debian and Ubuntu

First, you'll want to make sure that you have the apt-add-repository command.

apt update && apt install -y software-properties-common

You will also need the gnupg2 package so that the Ceph repo key can be added.

apt install -y gnupg2

Next add the Ceph repository key.

wget -q -O- 'https://download.ceph.com/keys/release.asc' |  apt-key add -

Next add the Ceph repository.

# For Debian 10
apt-add-repository 'deb https://download.ceph.com/debian-pacific/ buster main'

# For Debian 11
apt-add-repository 'deb https://download.ceph.com/debian-quincy/ bullseye main'

# For Ubuntu 18.04
apt-add-repository 'deb https://download.ceph.com/debian-pacific/ bionic main'

# For Ubuntu 20.04
apt-add-repository 'deb https://download.ceph.com/debian-quincy/ focal main'

You will need to update your repos once the ceph repo has been added.

apt update

Finally you can install the ceph-common package.

apt install -y ceph-common

RHEL variants such as CentOS and AlmaLinux

First you should clean up any previous yum metadata

yum clean all

Next you will need to install epel-release to ensure that you will be able to install the correct dependencies for ceph-common.

yum install -y epel-release

Next add the Ceph repository key

rpm --import 'https://download.ceph.com/keys/release.asc'

Next install the Ceph repository.

For RHEL7 and CentOS7 versions the latest Ceph version available is octopus.  Add the following repo data to /etc/yum.repos.d/ceph.repo.

[ceph]
name=Ceph packages for $basearch
baseurl=https://download.ceph.com/rpm-octopus/el7/$basearch
enabled=1
priority=2
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-octopus/el7/noarch
enabled=1
priority=2
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-source]
name=Ceph source packages
baseurl=https://download.ceph.com/rpm-octopus/el7/SRPMS
enabled=0
priority=2
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

If you are on a RHEL 8 variant, such as AlmaLinux 8, use the following for the /etc/yum.repos.d/ceph.repo file.

[ceph]
name=Ceph packages for $basearch
baseurl=https://download.ceph.com/rpm-quincy/el8/$basearch
enabled=1
priority=2
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-quincy/el8/noarch
enabled=1
priority=2
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-source]
name=Ceph source packages
baseurl=https://download.ceph.com/rpm-quincy/el8/SRPMS
enabled=0
priority=2
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

At this point it is ideal to ensure that the system is up to date

yum update

Finally, you can install the ceph-common package.

yum install -y ceph-common

Adding your CephFS configuration and authentication files

The following Ceph configuration must be placed in /etc/ceph/ceph.conf.

[global]
    mon host = monitor1.softiron.servercontrol.com.au,monitor2.softiron.servercontrol.com.au,monitor3.softiron.servercontrol.com.au

Next add your authentication file to /etc/ceph/ceph.client.cephfs.{your-key-name}.keyring.  In this example {your-key-name} is cid-XXX-SAU-YYYYY-CF. Your CephFS key name will be different for each CephFS product you are using. Please keep in mind that the path to the keyring is case sensitive.

Below is the example for a key ring called cid-XXX-SAU-YYYYY-CF.  It is saved in /etc/ceph/ceph.client.cephfs.cid-XXX-SAU-YYYYY-CF.keyring.

[client.cephfs.cid-XXX-SAU-YYYYY-CF]
    key = redacted-for-example==
    caps mds = "allow rw path=cid-XXX-SAU-YYYYY-CF"
    caps mon = "profile fs-client"
    caps osd = "allow rw namespace=cid-XXX-SAU-YYYYY-CF"

Technically, the caps sections of the keyring are not required, as they are only used on the Ceph server itself; however, it is fine to leave these lines in the keyring file.

Manually mounting the CephFS share

After adding the configuration and authentication files you will be able to mount the remote CephFS share.  While it is usually ideal to have the share automatically mount at each boot, it is good to test that the mount is working by manually mounting the file system for the first time.  In the next section I will describe how to have the file system automatically mount on boot.

First lets create a directory in /mnt to mount the file system.  You can mount your CephFS share anywhere on your servers files system that you require, but for this example we will mount the file system in /mnt/cephfs/cid-XXX-SAU-YYYYY-CF.

Create the mount point:

mkdir -p /mnt/cephfs/cid-XXX-SAU-YYYYY-CF

Next we will manually mount the remote file system.  Keep in mind, like with the authentication file described earlier, your CephFS name will be different.

mount -t ceph :/cid-XXX-SAU-YYYYY-CF -o name=cephfs.cid-XXX-SAU-YYYYY-CF /mnt/cephfs/cid-XXX-SAU-YYYYY-CF

To check that the filesystem has mounted, you can grep from the mount command.

mount | grep ceph

# output:
100.64.15.8:6789,100.64.15.9:6789,100.64.15.10:6789:/cid-XXX-SAU-YYYYY-CF on /mnt/cephfs/cid-XXX-SAU-YYYYY-CF type ceph (rw,relatime,name=cephfs.cid-XXX-SAU-YYYYY-CF,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl)

Automatically mounting CephFS during boot

Now that we know the file system is mounting correctly, we can set up the servers fstab to ensure that the file system will mount when the server boots.

Add the following to your /etc/fstab file, and make sure that you replace cid-XXX-SAU-YYYYY-CF with the actual name of your CephFS product. Also, make sure that you use the correct mount point for your needs.

:/cid-XXX-SAU-YYYYY-CF	/mnt/cephfs/cid-XXX-SAU-YYYYY-CF	ceph	name=cephfs.cid-XXX-SAU-YYYYY-CF,noatime,_netdev,rbytes	0 2