Configure bonding for IEEE 802.3ad (Link aggregation)

For details about bonding in general, see the Linux Ethernet Bonding Driver HOWTO

This article is the counterpart of port-channel.

Debian

/etc/network/interfaces

auto bond0
iface bond0 inet static
address 192.168.0.10
netmask 255.255.255.0
slaves eth0 eth2
bond_miimon 100
bond_mode 802.3ad
bond_lacp_rate fast


allow-bond0 eth0
iface eth0 inet manual


allow-bond0 eth2
iface eth2 inet manual

RedHat (CentOS, Scientific Linux, Fedora, ...)

For users of a ixgbe (Intel) nic: Careful with kernel releases between 2.6.194-8.1.el5 and 2.6.18-229.el5, you might run into RHEL bug #619070

/etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
IPADDR=192.168.2.12
NETMASK=255.255.255.0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
BONDING_OPTS="miimon=100 mode=802.3ad lacp_rate=fast"

/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
HWADDR=xx:xx:xx:xx:xx:xx
MASTER=bond0
SLAVE=yes
USERCTL=no

/etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
HWADDR=xx:xx:xx:xx:xx:xx
MASTER=bond0
SLAVE=yes
USERCTL=no

OpenSUSE

Presumably the same issue with the kernel as for RedHat. I haven't got it to work with 2.6.27.7-9-default (openSUSE 11.1).

/etc/sysconfig/network/ifcfg-bond0

STARTMODE='auto'
BOOTPROTO='static'
BONDING_MASTER=yes
BONDING_SLAVE_1='eth0'
BONDING_SLAVE_2='eth1'
BONDING_MODULE_OPTS='mode=802.3ad miimon=100 lacp_rate=fast'
IPADDR='192.168.0.10/24'
NETWORK='192.168.0.0'
USERCONTROL='no'

/etc/sysconfig/network/ifcfg-eth0

STARTMODE='off'
BOOTPROTO='none'
USERCONTROL='no'

/etc/sysconfig/network/ifcfg-eth1

STARTMODE='off'
BOOTPROTO='none'
USERCONTROL='no'

Gather Cisco Discovery Protocol informations with tcpdump

No cdpr at hand, no problem just throw this little tcpdump line at the interface of your interest:

tcpdump -nn -v -s 1500 -c 1 'ether[20:2] == 0x2000' -i eth0

If there are more than only CDP packets you may narrow it down to the Broadcast MAC:

tcpdump -nn -v -s 1500 -c 1 'ether[20:2] == 0x2000 and ether dst 01:00:0c:cc:cc:cc' -i eth0

Oh, and don't forget to wait for up to 60 seconds.

Policy-based routing using multiple routing tables

Making a System accessible over two IPs (Interfaces) is a bit of a challenge. Say you have connected the Machine via

eth0 192.168.0.10/24 GW: 192.168.0.1 (old)
eth1 192.168.1.10/24 GW: 192.168.1.1 (new)

Normally all traffic to remote IPs would be routed to the default gateway. Further say your default gateway is 192.168.0.1, so it's reachable via eth0. IP packages coming in via the IP address of eth1 would be answered via the default gateway through eth0 and therefore be invalid. The Solution to this problem is called Policy-based routing (by cisco) the implementation for Linux is based on multiple routing tables. Those routing tables are registered in /etc/iproute2/rt_tables:

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
200   oldnet
201   newnet

In this case the oldnet and newnet tables are custom, here we want to migrate the Server from one Network (oldnet) to the other (newnet) and the Policy-based routing allows us to do this seamless on the server side. To use your defined tables you can simply use the ip command:

ip route add 192.168.0.0/24 dev eth0 table oldnet
ip route add default via 192.168.0.1 table oldnet

To actually use your new routing table you have to issue an ip rule command, which makes the kernel use the table according to the matched rule:

ip rule add from 192.168.0.0/24 table oldnet

Now to actually make a difference we also have to define the routing for the newnet so:

ip route add 192.168.1.0/24 dev eth1 table newnet
ip route add default via 192.168.1.1 table newnet
ip rule add from 192.168.1.0/24 table newnet

Now we have our server use the table oldnet whenever he sends out a package from the 192.168.0.0/24 subnet and use the table oldnet for packages from 192.168.1.0/24. It's important to understand the direction of the packages, it's NOT about packages comming in, they will reach the host anyhow it's all about the (answering) packages that need to go the same way back as they came from, so the from statement is the one to use here.

Now the commands ip rule list will show you the rules actually used and the ip route show table oldnet resp. ip route show table newnet will show you the actual tables. Btw. a ip route show table all will give you quite an intimate look at the routing behind the scenes, just in case you didn't know.

To make things happen at boot time there are different ways things are handle, to name the major ones:

Debian

Quite obviously you need to touch the almighty /etc/network/interfaces like this:

iface eth0 inet static
address 192.168.0.1
netmask 255.255.255.0
gateway 192.168.0.1
post-up ip route add 192.168.0.0/24 dev eth0 table oldnet
post-up ip route add default via 192.168.0.1 table oldnet
post-up ip rule add from 192.168.0.0/24 table oldnet
post-down ip rule del from 192.168.0.0/24 table oldnet


iface eth1 inet static
address 192.168.1.1
netmask 255.255.255.0
gateway 192.168.1.1
post-up ip route add 192.168.1.0/24 dev eth1 table newnet
post-up ip route add default via 192.168.1.1 table newnet
post-up ip rule add from 192.168.1.0/24 table newnet
post-down ip rule del from 192.168.1.0/24 table newnet

Redhat"ish" (Scientific, Centos, Fedora and the like)

You need to edit (or create) the files /etc/sysconfig/network-scripts/route-eth{0,1} to contain the following:

/etc/sysconfig/network-scripts/route-eth0

192.168.0.0/24 dev eth0 table oldnet
default via 192.168.0.1 dev eth0 table oldnet

/etc/sysconfig/network-scripts/route-eth1
```
192.168.1.0/24 dev eth1 table newnet
default via 192.168.1.1 dev eth0 table newnet
```
The rules are configured in /etc/sysconfig/network-scripts/rule-eth{0,1}:
/etc/sysconfig/network-scripts/rule-eth0
```
from 192.168.0.0/24 lookup oldnet
```
/etc/sysconfig/network-scripts/rule-eth0
```
from 192.168.1.0/24 lookup newnet
```

openSUSE 11.1

Sadly I was not able to find a "simple" way to configure it for openSUSE 11.1. As its only a transitional Situation I live with the routing not beeing reboot-safe. *fingers crossed*

Setup a bridge on a bond

Debian 5 or later: /etc/network/interfaces

# The bridge interface.
# The lines with pre-up and post-down are to bring up the
# bond before the bridge. The '--allow "$IFACE"' parameter to
# ifup/ifdown and the corresponding "allow-br0 bond0" line aren't
# really needed, but I like them.

auto br0
iface br0 inet static
    address 192.168.1.10
    netmask 255.255.255.0
    gateway 192.168.1.1
    bridge_ports bond0
    bridge_maxwait 0
    pre-up ifup --allow "$IFACE" bond0 
    post-down ifdown --allow "$IFACE" bond0

# The bond interface
# notice it's set as manual to allow the scripts in ifenslave-2.6
# package to create the master interface and enslave the slaves.

allow-br0 bond0
iface bond0 inet manual
    bond_slaves eth0 eth1
    bond_miimon 100
    bond_mode 802.3ad
    bond_lacp_rate fast

# Alternative modes: 
    #  balance-rr or 0
    #  active-backup or 1
    #  balance-xor or 2
    #  broadcast or 3
    #  802.3ad or 4
    #  balance-tlb or 5
    #  balance-alb or 6

# Physical interfaces.
# Notice they are set as manual to permit you to add up/down commands
# and special directives such as wifi configurations.
# Lines beginning with allow-bond0 are required if you want to do
# something with these interfaces, otherwise you can remove both of
# them as they are brought up when they are enslaved to their master.

allow-bond0 eth0
iface eth0 inet manual

allow-bond0 eth1
iface eth1 inet manual

thx to Marco Nenciarini

Using a Bridge as a Gateway

If you want to use bridge as a gateway for other hosts you don't need to use the usual iptables nat, but you have to disable the iptableusage of the bridge altogether:

net.ipv4.ip_forward = 1

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0