Network Development Team

What is a MAC Flap?

A MAC Flap is caused when a switch receives packets from two different interfaces with the same source MAC address. If this makes no sense, perhaps a  quick summary of how switching at layer 2 works will help.

Switches learn where hosts are by examining the source MAC address in frames received on a port, and populating its MAC address-table with an entry for that MAC address and port. Say a device ‘A’ with MAC aaaa.aaaa.aaaa (hereafter aaaa) sends a frame to device ‘B’ with MAC address bbbb. Assume A is on port 0/1 and B is on port 0/2. The switch populates it MAC address-table something like:

Port		Host
0/1		aaaa

and floods the frame out of all other ports. When B replies the MAC address table becomes:

Port		Host
0/1		aaaa
0/2		bbbb

and the switch forwards the frame to port 0/1 – there is no need to flood now since the location of A is known.

If the switch were to then receive a frame on port 0/2 with a source MAC address of aaaa, there would be clash and the switch would log something like this:

1664321: Nov 14 11:18:16 UTC: %MAC_MOVE-SP-4-NOTIF:
Host aaaa.aaaa.aaaa in vlan A is flapping between
port 0/1 and port 0/2

and the MAC address-table would become:

Port		Host
0/1
0/2		bbbb
0/2		aaaa

What happens when B tries to send A a frame now? The switch won’t flood the frame as it knows a destination and it won’t send the frame back down the link – it gets dropped.

Lab time…

Let’s see if we can mimic this. This isn’t an easy thing to replicate so please forgive the artificial nature of the lab. I configured a switch with three hosts directly connected on VLAN 30. The hosts could ping each other and the MAC address-table was as follows:

3750-1#show mac address-table dynamic vlan 30
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  30    0008.7c82.5409    DYNAMIC     Fa1/0/1
  30    001a.2f22.d0c2    DYNAMIC     Fa1/0/2
  30    0024.97f0.3a70    DYNAMIC     Fa1/0/3
Total Mac Addresses for this criterion: 3

Host A had an IP of 192.168.30.1 and was on port 1. Host B was 192.168.30.30 and on port 2. Host C was 192.168.30.254 and on port 3.

So, ping with host A:

Host A# ping 192.168.30.254
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.30.254,
timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5),
round-trip min/avg/max = 1/201/1000 ms

Ping with host B:

Host B#ping 192.168.30.254

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.30.254,
timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5),
round-trip min/avg/max = 1/2/8 ms

Next I manually set host A to have the same MAC address as host B (001a.2f22.d0c2). The results? Host B lost connectivity for a few seconds.

Host A# int vlan 30
Host A(config-if)# mac-address 001a.2f22.d0c2
Host A# ping 192.168.30.254
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.30.254,
timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Here is the switch mac address table after the clone:

3750-1#show mac address-table dynamic vlan 30
 Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
 30    0008.7c82.5409    DYNAMIC     Fa1/0/1
 30    001a.2f22.d0c2    DYNAMIC     Fa1/0/1
 30    0024.97f0.3a70    DYNAMIC     Fa1/0/3
Total Mac Addresses for this criterion: 3
3750-1#
*Mar 17 04:22:02.620: %SW_MATM-4-MACFLAP_NOTIF:
Host 001a.2f22.d0c2 in vlan 30 is flapping between
port Fa1/0/2 and port Fa1/0/1
3750-1#

Here is what happened to Host B:

Host B#ping 192.168.30.254

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.30.254,
timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
Host B#ping 192.168.30.254

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.30.254,
timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5),
round-trip min/avg/max = 1/2/8 ms
Host B#ping 192.168.30.254

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.30.254,
timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5),
round-trip min/avg/max = 1/1/1 ms

Yes, this is the same impact you would have if two hosts had the same MAC on your network – there is a reason they need to be unique!

What does all this mean?

When you have an annexe VLAN [1] the backbone can be thought of as a series of Layer 2 switches for that VLAN. The ‘Broadcast Domain’ stretches over the entire Backbone. This means the CPU of every host (including our core switches) on a VLAN will receive every broadcast from every other host – this is not ideal but the only way we can offer the same subnet at multiple sites in this generation of the backbone. Another term sometimes used is ‘Failure Domain’. That is, a failure in part of the VLAN could impact the entire core. It is because of this risk to other units that we are keen to make sure annexe VLANs are tightly managed.

[1] These are known as Layer 2 end-to-end VLANs as there is no routing involved. We have called them ‘switched’ VLANs in the past. VLANs with a Layer 3 interface or SVI on the backbone are known as Layer 3 Routed VLANs.

To return to the the issues MAC flaps will cause on your network, each switch in the backbone has a MAC address-table for your VLAN. If for some reason your MAC addresses appear from different locations you will get dropped packets and our logs will fill up with messages which cause issues when we raise a support case with Cisco as our network appears to have loops.

What could cause it?

There are two or three common causes that we see.

1. Local loops
2. NAC
3. Wireless

1. Local Loops

If you don’t run STP then you are far more likely to suffer from network loops. Here are a couple of resources: STP is your friend and Implementing Spanning Tree. The issue with an annexe VLAN is that a local loop is no longer so local and could cause problems everywhere, both for you and others.

2. NAC

There is a legitimate but ill-advised network design which can cause issues. If you have a L2 NAC which forces all traffic through itself then it is possible that a frame will need to leave site A, get switched through to site B only to return to site A, all with the same MAC address. See the image below. I’ve represented the Backbone as one red switch and the ingress and egress ports as tunnel entrances and exits. This design mustn’t be used with the current generation of the backbone.

3. Wireless

We used to run OWL and eduroam (Phase 1) over two VLANs which spanned the entire core. Due to the issues I’ve mentioned we changed this last year. Now the VLANs are local to the FroDos and routed through the core. Prior to doing this it was possible to roam from access points connected to different FroDos and cause MAC flaps.

What should I do next?

We’re going to keep an eye on the logs and will let Units know if they are causing MAC flaps. We’ll work with you as far as possible to locate the source of the issue and get things stable. If you aren’t yet running STP please can I urge you do consider doing so. The new backbone is still some years off so for the good of everyone we need to work together to reduce this. For units which cannot resolve this we may need to look at reverting to a fully routed connection, with each Annexe having its own subnet.

Do get in touch if you have any questions.