LTLnetworker | IT hálózatok, biztonság, Cisco

               IT networks, security, Cisco

Why is this huge traffic appearing here? Unknown unicast flood

Posted by ltlnetworker on October 5, 2014


Switches usually forward unicast frames to the necessary direction only. Selecting the egress port depends on the MAC address table that is populated by MAC learning. The switch has a chance to learn an address and keep it in the table only if frames are sent from that address regularly. Cisco switches’ default aging time is 300 s, a MAC address is dropped from the table if no frames arrive for 5 minutes.

Unknown unicast flood occurs if traffic is sent to a MAC address which was
a) never learned
b) already aged out
from the MAC address table. In this case, the frame is flooded out on all ports belonging to the VLAN just like a broadcast.

Why would a node send traffic to a nonexistent MAC address? Apart from malicious attacks, usually it is not the sender to blame. For IPv4 traffic, the destination MAC address is determined from the ARP information the receiver sends. Unfortunately there are some products placing a MAC address in the ARP reply that is not or rarely used for sending traffic:

Let’s see some cases where I have encountered unknown unicast floods:

VMware Fault Tolerance requires a vmk interface on both vSphere hosts. Each has an IPv4 address and the primary sends thousands of TCP/8100 packets to the secondary replicating the changes in VM states. For some reason in our network the real MAC address was not used in reply packets. (I did not capture it but a Wireshark file is welcome if you have one). When we pinged the secondary host’s FT IP address the ARP reply and the ICMP echo reply frame was sent from the right MAC address, the switch learned the address and the flood ceased. If you have an automated network monitoring tool you may want to add these IP addresses simply to guarantee a regular ping and prevent such floods.

Some Unix servers have a bonding or cluster setup where multiple NICs have a common virtual IP address. Apparently those techniques were designed by engineers unaware of switching and ARP basics or ignorant of flooding disadvantages. A customer had a TSM server on AIX with two interfaces that were combined somehow. I did not have the opportunity to examine the settings so I cannot estimate if the setup had a flaw. I was informed that either interface had a different IP address:
10.7.240.57         0011.2508.8f2e
10.7.247.125        0011.2508.920f

This information was confirmed by looking at the ARP table:
SERV-SUP720#sh ip arp | i 10.7.240.57|10.7.247.125|0011.2508.8f2e|0011.2508.920f
Internet  10.7.240.57          14   0011.2508.8f2e  ARPA   Vlan20
Internet  10.7.247.125          9   0011.2508.920f  ARPA   Vlan20

We experienced a traffic spike on all switchports on the same time every day which turned out to be a backup job. The packet capture file showed 9000-10000 packet/s from 10.7.254.235 to 10.7.247.125
But the destination MAC address was not in the MAC address table. The sender server must have cached in its ARP table for hours while the switch dropped the CAM entry after 300 seconds. After a ping, the entry appeared again (for 5 minutes):

SERV-SUP720#sh mac-address-table | i 0011.2508.920f|0011.2508.8f2e|1/1$|1/2$
*   20  0011.2508.8f2e   dynamic  Yes          5   Gi1/2
SERV-SUP720#ping 10.7.247.125

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.7.247.125, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
SERV-SUP720#sh mac-address-table | i 0011.2508.920f|0011.2508.8f2e|1/1$|1/2$
*   20  0011.2508.920f   dynamic  Yes          5   Gi1/1
*   20  0011.2508.8f2e   dynamic  Yes          0   Gi1/2
SERV-SUP720#sh mac-address-table | i 0011.2508.920f|0011.2508.8f2e|1/1$|1/2$
*   20  0011.2508.920f   dynamic  Yes        210   Gi1/1
*   20  0011.2508.8f2e   dynamic  Yes          0   Gi1/2
SERV-SUP720#sh mac-address-table | i 0011.2508.920f|0011.2508.8f2e|1/1$|1/2$
*   20  0011.2508.8f2e   dynamic  Yes          0   Gi1/2

The traffic is TCP/1500 so the problem is not the lack of reply packets from 10.7.247.125. The capture shows continuous reply packets (TCP ACKs) but with the src MAC address of the other NIC (0011.2508.8f2e)! AIX is hard to understand (and tolerate). But a ping from the L3 switch caused the second NIC to use the proper MAC address, perhaps due to an ARP request but I did not verify that with targeted testing. Flooding ceased immediately when I started pinging. So fortunately this MAC address could be tricked to appear so we added a ping probe to Nagios and eliminated the unicast flood this way.

But the most common case is Microsoft’s Network Load Balancing (NLB). It is usually used in unicast mode which should be totally banned in an enterprise network. Unicast NLB relies on unknown unicast flood as this guarantees that multiple servers receive the same packets. You cannot prevent the flood, all you can do is use a dedicated VLAN which is kept as small as possible and added only to the minimum necessary inter-switch 802.1Q trunks (VTP pruning may be your friend). If you network is capable of multicast and IGMP snooping you have the opportunity to make the server admins switch to NLB multicast IGMP mode but don’t forget to provide an IGMP querier in the VLAN.

The effects of unknown unicast floods are not always noticed. The network and the connected hosts can often cope with it and nobody is aware of it. If network slowness is experienced, a packet capture or a traffic graph usually reveals the flood. I have seen a Cisco IP phone disconnecting periodically due to hundreds of thousands of such packets arriving in every minute.

It is hard to anticipate when and where unknown unicast flood packets will appear in a network. It is more likely to happen in server VLANs. That’s another reason to avoid clients and servers share the same VLAN. If the affected server VLAN spreads to large network segments then all the 802.1Q tagged links will carry the huge traffic including inter-switch uplinks and even VMware vSphere switchports if you don’t specify the allowed VLAN list on 802.1Q trunk ports. Even worse if you were uncautious and built Layer 2 interconnected data centers.

I recommend these as best practice:

  • Do not let your VLANs spread to everywhere. Especially not to another site/DC.
  • Use dedicated VLANs for NLB, FT, backup and similar flood-prone applications.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: