How to Troubleshoot a Broadcast Storm

Broadcast storms are a major pain point for network administrators. Let’s talk a bit about what they are, what causes them, and what steps you can take to eliminate sources of broadcast storms on the networks you manage.

What’s a broadcast storm?

A broadcast storm is an abnormally high number of broadcast packets within a short period of time. They can overwhelm switches and endpoints as they struggle to keep up with processing the flood of packets. When this happens, network performance degrades.

What’s a broadcast packet?

There are three types of packet: broadcast, multicast, and unicast. Anycast is considered similar to multicast, but packets will be delivered to only one random host, instead of the entire group.

Think back to the old days of watching TV using rabbit ears. We’d catch a channel “over the air”. Broadcast, like its name implies, is an “over the air” packet sent from one source to any listener tuned into the same frequency. In terms of the OSI network model, a broadcast packet from a specific source will be heard by any interface logically placed within the same broadcast domain. For example, a broadcast packet sitting within VLAN X will be received by any network interface also logically residing on VLAN X.

Broadcast packets are typically sent by an attention seeker—a node on the network that wants others to know of its presence. A broadcast packet consists of the following Layer 2 and Layer 3 destination header:

• ff-ff-ff-ff-ff-ff (Layer 2 broadcast)
• 255.255.255.255 (Layer 3 limited broadcast)

Typical root causes of a broadcast storm

1. High volume of requests for an IP address via DHCP

The Dynamic Host Configuration Protocol (DHCP) is the most common way for a networked host to obtain an IP address from a network controller.

Think about when you take your laptop to a coffee shop and hop on WiFi. Aside from clicking “I agree” to a legal disclaimer or entering in some credentials, you don’t need to worry about statically assigning yourself an IP address. This is DHCP at work.

DHCP consists of a few key steps:

1. Discover – always broadcast
2. Offer
3. Request
4. Acknowledge

Depending on how things are configured, DHCP may be sent as broadcast or unicast (one-to-one) packets. A storm of broadcast packets is sometimes expected behavior—for example, when a network is brought back online after an outage and all clients are attempting to negotiate an IP address. But in normal cases, having a continuous stream of broadcast packets in a network segment or from a specific host is suspicious.

Some suggestions for dealing with DHCP-related broadcast storms:

• Stagger the enablement of a large group of devices that would otherwise all request an address through DHCP at the same time.
• Check to see if you’re using a DHCP relay between some or all of your VLANs. DHCP relay routes broadcast packets between broadcast domains.
• Set up a maintenance window in Auvik to silence broadcast-related alerts during known periods of high-volume DHCP requests.

2. The broadcast domain is too big

The amount of broadcast traffic you see within a broadcast domain is directly proportional to the size of the broadcast domain—the number of hosts within the L2 VLAN or L3 subnet.

Many times, we see MSPs bringing on board clients with all of their hosts in the same broadcast domain, sometimes as large as a /16 subnet which can accommodate just over 65,000 possible addresses. (Tip: try out our handy subnet calculator to help!

In this case, network segmentation is your friend. Place like groups of devices together in segments. This will reduce the number of “I’m here” broadcasts generated.

How to identify broadcast storms

Get an alert on storms in Auvik. There’s a preconfigured Auvik alert that tells you when a significant percentage of a switch port’s traffic is broadcast as opposed to unicast or multicast. You can lower the threshold of this alert in sensitive or troubleshooting scenarios to be more proactive and in the know.

Use the troubleshooting view to see if a broadcast storm may have caused other events within the same timeframe, such as a spike in CPU on the host or an adjacent switch.

Ideas for reducing broadcast storms

• Storm control and equivalent protocols allow you to rate-limit broadcast packets. If your switch has such a mechanism, turn it on.
• Ensure IP-directed broadcasts are disabled on your Layer 3 devices. There’s little to no reason why you’d want broadcast packets coming in from the internet going to a private address space. If a storm is originating from the WAN, disabling IP-directed broadcasts will shut it down.
• Split up your broadcast domain. Creating a new VLAN and migrating hosts into it will load and balance the broadcast traffic to a more acceptable level. Broadcast traffic is necessary and useful, but too much of it eventually leads to a poor network experience.
• Check how often ARP tables are emptied. The more frequently they’re emptied, the more often ARP broadcast requests occur.
• Sometimes, when switches have a hardware failure, their switch ports begin to spew out broadcast traffic onto the network. If you have a spare switch of the same or similar model, clone the config of the active switch onto the spare and swap the hardware and cables during a maintenance window. Does the storm subside? If it does, it was a hardware issue. If not, then you’ve gotta keep digging.
• Check for loops in switches. Say there was an unmanaged Layer 2 switch connected upstream to an unmanaged switch, and someone’s connected a cable between two ports on the same unmanaged switch (let’s say ports 1 and 2). The unmanaged switch will respond to all broadcasts multiple times and flood the broadcast domain with packets, causing a denial of service attack on the network.
  • BPDU and PortFast or equivalent features should be implemented as a best practice to prevent loops.
  • Discourage users from connecting unmanaged switches to managed switch ports by enforcing a maximum number of MAC addresses per port. This may be up to two MAC addresses if users have a computer plugged into an IP phone, which in turn is plugged into the switch.