Create a Static Route in Ubuntu 18.04 or Newer for Unmetered Bandwidth to the Storage Platform

Overview

Our HPE dedicated servers by default have a dedicated port for provisioning and storage. 

To utilize this port for storage traffic, you'll need to create static routes inside the operating system of the server. 

This tells the operating system to send traffic to/from the storage platform via the storage port, instead of via your default gateway (primary/monitored interface).

Requirements

• A HPE Dedicated Server ( Blaze or Enterprise Server ).
• You must already know your storage IP address, netmask, and gateway. To find those, please follow the guide here: Find Your Storage IP Address
• Root console access to your server, via SSH or IPMI console.

Steps

Configure your storage IP address

Use the "ip a" command to list all network interfaces. The storage/provisioning port is usually on the same physical network adapter as the primary IP address.

In the below example, you can see that the primary IP address is assigned to network adapter "ens10f0np0". 

As such, the storage/provisioning port will be the second interface on the same network adapter, "ens10f1np1".  

In this example, ens2f0np0 and ens2f1np1 are both on "ens2" which is a completely separate network adapter, and can be ignored.

Find your secondary network adapter name

root@SAU-ECC6A-OR:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens10f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 5c:6f:69:06:85:4c brd ff:ff:ff:ff:ff:ff
    inet 221.121.144.163/31 scope global ens10f0np0
       valid_lft forever preferred_lft forever
    inet6 fe80::5e6f:69ff:fe06:854c/64 scope link
       valid_lft forever preferred_lft forever
3: ens10f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 5c:6f:69:06:85:4d brd ff:ff:ff:ff:ff:ff
4: ens2f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 5c:6f:69:d6:12:30 brd ff:ff:ff:ff:ff:ff
5: ens2f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 5c:6f:69:d6:12:31 brd ff:ff:ff:ff:ff:ff

Now that we've identified "ens10f1np1" as the secondary network adapter, we can configure the storage IP address on it.

You'll have to note down the MAC address of the interface. In this case, it's "5c:6f:69:06:85:4d". 

Please save this to a notepad as we'll use this to configure the interface.

Configure the storage IP address as a static IP on the storage network adapter

Ubuntu 18 and newer use 'netplan' for their network configuration files. 

To find your netplan configuration file, run the following command:

ls /etc/netplan/
01-netcfg.yaml

Before proceeding, make sure you have the following information:

• Storage IP address
• MAC address of the storage interface

The following guide shows how to find your storage IP: Find Your Storage IP Address

Now let's configure the storage IP address, and add the static routes to the storage platform.

Open the netplan configuration in your preferred editor. If you're unsure, nano is probably the easiest for beginners.

nano /etc/netplan/01-netcfg.yaml

You can see the final configuration below. 

We added the "storageif:" line, and everything below it.

network:
  version: 2
  renderer: networkd
  ethernets:
    mainif:
      match:
        macaddress: "5c:6f:69:06:85:4c"
      addresses: [221.121.144.163/31]
      gateway4: 221.121.144.162
      nameservers:
        addresses: ['221.121.130.3', '221.121.134.9']
    storageif:
      match:
        macaddress: "5c:6f:69:06:85:4d"
      addresses: [100.64.24.78/30]
      routes:
          - to: 100.64.15.0/24
            via: 100.64.24.77
          - to: 27.50.66.224/28
            via: 100.64.24.77

You'll need to be careful with your spacing when editing a .yaml file. 

Make sure that the line 'storageif:' has the same number of spaces before it, as the 'mainif:' line.

The spacing should probably look like this:

storageif: 4 spaces

match: 6 spaces

macaddress: 8 spaces

addresses: 6 spaces

If you're unsure, just try to match the spacing exactly as the mainif configuration above what we added.

If you used Nano, you can save and exit by pressing: 

1. CTRL + O, then Enter (To save the file)
2. CTRL + X (To exit the editor)

Once you've completed editing the configuration file, apply the config with the following command:

netplan apply

Verify that the static routes applied correctly.

Check the routing table with the following command:

ip route

You should see the routes that we added to the configuration file in the output:

27.50.66.224/28 via 100.64.24.77 dev ens10f1np1 proto static
100.64.15.0/24 via 100.64.24.77 dev ens10f1np1 proto static

Next, you should do a few tests to make sure the static routes are working as expected.

1. Ping the storage endpoints..

First test the s3 endpoint:

ping s3.si.servercontrol.com.au
PING s3.si.servercontrol.com.au (27.50.66.227) 56(84) bytes of data.
64 bytes from 27-50-66-227.as45671.net (27.50.66.227): icmp_seq=1 ttl=61 time=0.353 ms
64 bytes from 27-50-66-227.as45671.net (27.50.66.227): icmp_seq=2 ttl=61 time=0.302 ms
64 bytes from 27-50-66-227.as45671.net (27.50.66.227): icmp_seq=3 ttl=61 time=0.308 ms
^C
--- s3.si.servercontrol.com.au ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2028ms
rtt min/avg/max/mdev = 0.302/0.321/0.353/0.022 ms

Next test one of the RBD/Ceph endpoints:

ping storage1.si.servercontrol.com.au
PING storage1.si.servercontrol.com.au (100.64.15.11) 56(84) bytes of data.
64 bytes from 100.64.15.11 (100.64.15.11): icmp_seq=1 ttl=61 time=0.087 ms
64 bytes from 100.64.15.11 (100.64.15.11): icmp_seq=2 ttl=61 time=0.100 ms
64 bytes from 100.64.15.11 (100.64.15.11): icmp_seq=3 ttl=61 time=0.091 ms
^C
--- storage1.si.servercontrol.com.au ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4089ms
rtt min/avg/max/mdev = 0.076/0.087/0.100/0.010 ms

Now we just need to make sure that the traffic is taking the correct path. To do this, use tracepath. We just need to verify that the first hop in the path is your storage gateway, and NOT the gateway of your public IP address.

You can press CTRL + C once you see the first hop, to quit the command.

tracepath -n s3.si.servercontrol.com.au
 1?: [LOCALHOST]                      pmtu 1500
 1:  100.64.24.77                                          0.636ms
 1:  100.64.24.77                                          0.439ms
 2:  no reply
^C

You can see that the first hop was correct, 100.64.24.77 which is my storage gateway.

Check for the Ceph/RBD subnet as well:

tracepath -n storage1.si.servercontrol.com.au
 1?: [LOCALHOST]                      pmtu 1500
 1:  100.64.24.77                                          0.953ms
 1:  100.64.24.77                                          0.640ms
 2:  no reply

The above tracepaths show that our static routes are working correctly.

If the static routes were not configured properly, you'd see the trace would be using your public IP address gateway instead. Something like this:

tracepath -n storage1.si.servercontrol.com.au
 1?: [LOCALHOST]                      pmtu 1500
 1:  221.121.144.162                                       0.658ms
 1:  221.121.144.162                                       0.519ms
 2:  100.64.105.74                                         1.567ms

If you see a public IP address as the first hop, then something has gone wrong and your static routes are not working correctly. If that's the case and you can't figure it out please feel free to submit a support case and we'll be happy to assist you further.