UBNT LACP & STP defaults that Bite

We've plunged into building out a fairly large laboratory network using Ubiquiti's ES-16-XG and ES-48-Lite. By default these switches have some defaults that can bite you.

LACP isn't a default

The first is when you set up Link Aggregation the default is to use the older static protocol instead of the more modern 802.3ad LACP standard. You'll have to set each link to use LACP with this parameter:

configure
interface 3/1
no port-channel static

If you do not do this, you'll see periodic drops between switches that don't make a lot of sense. This can be very frustrating to troubleshoot because you'll sometimes get a race condition where both switches just agree to work over the static aggregate, then at a later time you drop. Always use LACP between switches.

STP is a default, but you better have a root bridge

Once you start using LACP you'll see that STP is enabled by default. If you do not have a root bridge specified in your network then STP will adjudicate by selecting the oldest MAC in your network... yeah... So, assuming you're creating a "simple" network and trusting STP you will want to select a device to be the root bridge, this likely will be the device that is "one armed" to your router. On that switch you will want to:

configure
spanning-tree mst priority 0 0

I had created a pretty bad mess when a very old ES-48 was behaving as the root bridge of our network, while functionally it was a leaf unit.

Bonus... LACP and Vlans with systemd-networkd

This is well documented on the archwiki, however I'll re-document here for posterity sake. If you're wanting to connect up a server that uses trunking on one of these links you'd want to do the following:

  • dozer is a server that needs to be in four subnet/vlans:
    • 10 : MGMT - the management subnet for servers
    • 20 : OPRS - the operations subnet for administrators
    • 60 : SRGA - a subnet for users

On the switch we would configure the LAG interface:

vlan database
vlan 10,20,60
vlan name 10 "MGMT"
vlan name 20 "OPRS"
vlan name 60 "SRGA"
exit
configure
interface 0/1-0/4
addport 3/1
interface 3/1
description 'trunk to dozer'
no port-channel static
switchport mode trunk
switchport trunk allowed vlan 10,20,60
exit

On the server, assuming systemd-networkd we would first prevent the bonding module to create an initial bond0 interface, set the max_bonds option of the bonding module to 0 (default value is 1):

vim /etc/modprobe.d/bonding.conf
---
options bonding max_bonds=0

Enumerate the carrier adapters you're planning to use by using ip link to list them. We will assume enp3s0f0 and enp3s0f1 for this example. Specify the bond, in this case we'll use the name bondxg:

vim /etc/systemd/network/bondxg.netdev
---
[NetDev]
Name=bondname
Kind=bond

[Bond]
Mode=802.3ad
LACPTransmitRate=fast

You may want to specify the MAC address of the bond under [NetDev] with the MACAddress directive. I'd suggest doing this after you reboot the machine fully and find out which of the carrier adapters "wins" as the device is created by systemd. Now you'll create a network file for this netdev:

vim /etc/systemd/network/bondxg.network
---
[Match]
Name=bondxg

[Network]
VLAN=mgmt
VLAN=oprs
VLAN=srga
BindCarrier=enp3s0f0 enp3s0f1

You now need to slave up the carrier adapters, for each adapter you need a network file:

vim /etc/systemd/network/enp3s0f0.network
---
[Match]
Name=enp3s0f0

[Network]
Bond=bondxg

You now need to specify your vlans, for each vlan you'll need a network and netdev file:

vim /etc/systemd/network/mgmt.netdev
---
[NetDev]
Name=mgmt
Kind=vlan

[VLAN]
Id=10
vim /etc/systemd/network/mgmt.network
---
[Match]
Name=mgmt

[Network]
VLAN=mgmt

[Address]
Address=10.10.10.2/24

[Route]
Destination=0.0.0.0/0
Gateway=10.10.10.1

You can extrapolate from this example to create all the necessary files. You'll want to fully reboot after specifying the bondxg.netdev because systemd needs to create that device and it's best to verify its created at boot time.