Recovering A Bad Flash — Part 1

Some people are perfect, they never make mistakes, and they never put themselves in a position where they have to dig themselves out of whatever problem they’ve caused.

I’m not that person. I make mistakes, but I learn from them. I learn how to solve the problems those mistakes cause, and I learn more about the systems themselves. I have to think this is pretty normal — and as my mentor Tom Jacoby always says: “the definition of an expert is someone who has already made all the mistakes!”

So yesterday I made a mistake that made me feel like a junior tech all over again — if only until I knew how I was going to fix it. I was upgrading software on four pre-production routers, and one of these had a flash that was just big enough for one image at a time. Standard procedure here is to delete the old image, load a new one on and reload the system.

This was no problem — I had already done another identical machine and it came up fine… The problem was this was the last router I had to work on so while it was copying I started cleaning up and then I find myself holding the power cables for all the routers in my hands. They’re all offline.

And now the last router will not boot — it just drops into ROMMON. I’ve recovered hardware over console xmodem and that is not nice — it can take hours to transfer a 30 meg image. Thankfully these routers use CF cards, and thankfully I have a USB CF reader. I mounted the card on my laptop, copied the new image over, booted the system and I was off to the races.

So what did I learn? Don’t rush the job. Think about what you’re doing and if you’re in a hurry just put that hurry out of your mind. Small mistakes can cause huge amounts of pain — if these were production routers instead of pre-production it would have been a very difficult day for me.

All About Spanning Tree Protocol

Spanning Tree Protocol

I discussed STP in a couple of earlier articles, here and here, but I would like to go into a little more detail because I think this is really important.

Spanning Tree Protocol can help us design fault-tolerant networks in two ways; primarily by detecting and disabling port misconfigurations, and secondly by allowing administrators to build failover network links.

STP is a complicated protocol, and it comes with a suite of different applications that can help fine-tune the system. I highly recommend a network engineer studies the Cisco documentation on STP, and then builds a lab environment before deploying.

CIO.com has a fascinating article that describes how critical STP is, and how good network design can help eliminate the worst of these consequences.

STP Root

The first thing to remember about STP is to ensure that the root switch is defined properly. The root switch should be the most central switch on the network (not necessarily the most powerful), and it should be the least disturbed switch — this means the LAN core switches are often the best choice as a STP root.

STP can choose a root switch automatically, but unfortunately this is not usually what you want. In fact, one of STPs parameters is to select the switch with the lowest MAC address, which is usually the oldest switch on the network. That in itself wouldn’t be a problem, except that on many networks the oldest switches are pushed to the edge of the network.

STP Parameter Tuning

STP is designed with an average network in mind. But your network isn’t average — to get the best performance out of STP you will have to modify the parameters that make it work.

Here is a Cisco guide on this — Understanding and Tuning Spanning Tree Protocol Timers. Note Cisco’s caution note: If you make mistakes with tuning STP you risk “LAN meltdown”.

The simplest way to tune the STP timers is to set the STP diameter. The link above has a guide to help calculate the STP diameter of your network — but be wary while setting this parameter. If your network changes (future adds/moves) it may expand beyond the STP diameter that you have set and then bad things happen.

The diagram above duplicates what is in the Cisco document, but it identifies one of the calculations. You need to identify the worst-case scenario for the number of switches a packet has to cross — in this case it is CADBE, and DACBE; STP diameter here is 5.

My advice is to leave these parameters alone unless you have a good reason to make changes. The risk is high, and what is gained is a slightly more rapid recovery from a network failure — the long term consequence is that someone will have to remember that this was done.

Loop Protection

Loops in Layer 2 networks are very, very bad. The layer 2 header has no time-to-live value, so a looped frame can continue to loop forever. Add in some broadcast traffic and you have a recipe for disaster, Cisco calls it LAN meltdown.

A key element of STP is that it prevents loops — STP is designed to detect and resolve Layer 2 loops. It is not enough to run STP only on the core switches of a network (although it helps), to fully protect a LAN all switches must be running STP.

Redundant Connections

STP can be used to create failover links within the LAN, allowing the network administrator to design networks with multiple links between switches. Essentially the network administrator intentionally creates a loop, and allows STP to block one of the links. If the primary link were to fail, then STP would re-calculate the topology and would bring up the secondary link.

Host Port Configurations

Running STP on your network has some interesting side effects. It sure is great knowing that Layer 2 loops are automatically detected and mitigated, but sometimes running STP can make life difficult for users.

For example, when a port comes online STP does not trust the interface. That means it will listen to the port to see if any BDPUs come through, but it will not forward any traffic. The process from Blocking to Listening, to Learning and finally to Forwarding state can take up to a minute. Unfortunately, that means no traffic will pass the interface, which can cause some hosts that depend on DHCP for IP address assignment to fail. If you unplug and reconnect the network cable (or disable and enable the interface) the cycle starts again so that tried and true end-user approach to fixing network faults will not help. Thankfully most often a user can simply “Repair” the network connection in software, and the OS will make a successful DHCP transaction.

Cisco (and other network vendors) have created a clever solution for this problem. spanning-tree portfast disables STP on a port; so ports are always in a forwarding state.

Unfortunately, spanning-tree portfast leaves your network at risk from a user inadvertently connecting a single cable to two ports — creating a network loop that cannot be detected that can potentially eat up all the CPU processing on the switch. Cisco (and other network vendors) have come up with a solution for this too. spanning-­tree portfast bpduguard will automatically disable a port if a single BPDU frame is detected. This is not an intelligent protocol, so do NOT use spanning-tree portfast bpduguard on any switch uplinks as it will definitely shut those down as soon as it receives the first BPDU frame.

Usually when a port is shutdown, the user (or network administrator) will see the problem and resolve it. But if the port remains disabled, it requires an administrator to manually bring the port online. Cisco decided this is too much work for us network administrators, so they defined the errdisable recovery cause bpduguard command — this brings the port back online after 5 minutes (by default). If the loop still exists, it will be detected and the port will go offline again

Advanced Configurations

All of the below configurations are discussed in detail in the Cisco documentation on STP, but I will review them in brief.

Bpdufilter is an alternative to bpduguard. Like bpduguard it watches for incoming BPDU frames, but in addition it filters outbound BPDU frames (less traffic for hosts which discard these frames anyway), and if a BPDU frame is received on this port (a switching loop is created) then the port loses its portfast status and STP starts to monitor the port. There is some risk here in the event of a user looping two ports at their own desk, as the loop would not be detected.

Uplinkfast allows the network administrator to define redundant switch uplinks so they skip the Listening and Learning state in the event of a link failure. This allows the network to converge faster (Cisco says about 5 seconds) than it would have done otherwise.

Backbonefast allows switch backbone interfaces to detect and resolve indirect link failures faster than the normal STP timeouts would allow. Essentially, it allows a switch to detect a network link failure on a neighboring switch, and update its own topology very quickly.

Rootguard prevents a particular port from becoming a root port. This prevents a new switch connected to the edge of the network from becoming a root switch. Recall that the network administrator spends a lot of effort design the STP topology so the root switch is in a logical location — usually the core.

Loopguard detects a unidirectional link (usually a wiring fault) and moves the interface to the Listening state. This prevents STP calculation failures and neighboring switches will make incorrect assumptions about the network topology. There is another Layer 2 solution to this problem as well called udld which may provide more interoperability and flexibility than loopguard; Cisco has documentation on udld and loopguard here.

Rapid STP

Cisco has a white paper describing RSTP; the document is very concise and explains the differences between RSTP and STP quite clearly.

In short RSTP brings in the features of uplinkfast and backbonefast, (which were Cisco proprietary features in STP), updates the algorithm so that topology detection cascades across the network, instead of the slow plodding detection of STP, and finally it updates the timers so that convergence happens much faster.

Downgrade from Lightweight to Autonomous Using Linux

Hello world,

I recently came across a solution for a small problem I was working on, and I thought I would share.

I have a Cisco 1131 AG lightweight access point that I wanted to downgrade to autonomous mode. Now this is easy if you have a WLC (Wireless LAN Controller) that these APs are designed for, but if you don’t you have to perform a downgrade manually.

There are lots of instructions on the Internet, but this is the Cisco documentation for this issue.

What Cisco leaves out is what the AP is expecting out of your TFTP server. When the AP loads up into recovery mode it is going to be attempting to connect to a tftp server at the broadcast address: tftp://255.255.255.255 — essentially this means that the AP will attempt to connect to an address that every device on the LAN has.

My laptop is running Ubuntu (a flavour of Linux) and I have the tftpd-hpa package as my TFTP server. Normally I like the setup I have, but today it wasn’t enough — the AP wasn’t connecting.

The CLI was throwing this error:

[code]]czoyMTY6XCJpbWFnZV9yZWNvdmVyeTogRG93bmxvYWQgZGVmYXVsdCBJT1MgdGFyIGltYWdlIHRmdHA6Ly8yNTUuMjU1LjI1NS4yNTV7WyYqJl19L2MxMTMwLWs5dzctdGFyLmRlZmF1bHQNCmV4YW1pbmluZyBpbWFnZeKApg0KZXh0cmFjdGluZyBpbmZvICgyODAgYnl0ZXMpDQpQcntbJiomXX1lbWF0dXJlIGVuZCBvZiB0YXIgZmlsZQ0KRVJST1I6IEltYWdlIGlzIG5vdCBhIHZhbGlkIElPUyBpbWFnZSBhcmNoaXZlLlwiO3tbJiomXX0=[[/code]

And when I used wireshark to look at the traffic I found that the tftp server wasn’t responding to the requests from the AP. So I configured tftpd-hpa to specifically listen on the broadcast address like this:

[code]]czoxNDQ6XCIkIGNhdCAvZXRjL2RlZmF1bHQvdGZ0cGQtaHBhDQojRGVmYXVsdHMgZm9yIHRmdHBkLWhwYQ0KUlVOX0RBRU1PTj0mIzN7WyYqJl19NDt5ZXMmIzM0Ow0KT1BUSU9OUz0mIzM0Oy1jIC1sIC1zIC92YXIvbGliL3RmdHBib290IC1hIDI1NS4yNTUuMjU1LjI1NSYjMzQ7XCJ7WyYqJl19O3tbJiomXX0=[[/code]

And restarted tftpd with /etc/init.d/tftpd-hpa.

Once I did that, the AP was able to connect and all was well in the world.

High Availability — LAN — STP

The parent article on High Availability.

Switching on a LAN provides some of the most basic network connectivity options, and are often overlooked. Nonetheless most switches (Cisco, HP, Dell and others) support these configurations, but one thing I can guarantee is that you will find limitations on pretty much every platform. If you’re after inter-operability, do your testing so you can understand these limitations.

Spanning Tree Protocol

I discussed STP in an earlier article, but I would like to go into a little more detail here.

Spanning Tree Protocol can help us design fault-tolerant networks in two ways; primarily by detecting and disabling port misconfigurations, and secondly by allowing administrators to build failover network links.

STP is a complicated protocol, and it comes with a suite of different applications that can help fine-tune the system. I highly recommend a network engineer studies the Cisco documentation on STP, and then builds a lab environment before deploying.

I also recommend Cisco’s STP Problems and Design Considerations document. It just might help identify why things are happening the way they are.

Loop Protection

Loops in Layer 2 networks are very, very bad. The layer 2 header has no time-to-live value, so a looped frame can continue to loop forever. Add in some broadcast traffic and you have a recipe for disaster, Cisco calls it LAN meltdown.

A key element of STP is that it prevents loops — STP is designed to detect and resolve Layer 2 loops. It is not enough to run STP on the core switches of a network (although it helps), to fully protect a LAN all switches must be able to run STP.

Redundant Connections

STP Example

If you configure two switches with two network connections, STP will detect the loop and block one of the ports. There are calculations that help STP decide which interface to block, but that is for a more technical review of STP.

In the example on the left, I’ve configured two switches to use two links between them. As long as the configuration stays simple; this would actually work with a STP capable switch and a dumb switch or even a hub.

STP detects the second link and blocks the port. The calculation of which port to block is determined by an algorithm based on a interface speeds. This is customizable, so you can make sure that STP opens and fails predictably.

Failure Mode

If something happens to the active interface, STP detects this change and stops passing traffic so it can recalculate the topology. Once it is complete things look as they do on the left.

Here STP detected a failure on the active interface, and opened the secondary connection.

STP doesn’t depend on physical failures to detect network changes — each switch is constantly sending out Hello frames to the root switch. If any switch on the network is disconnected from the root switch for 3 Hello frames the entire network stops and recalculates the topology.

Advanced Configurations

When designing a network, you must always consider the complexity if your design and the requirements of your client. Sometimes for a client without a network savvy administrator to maintain the network, relying on STP for redundancy is a bad idea; and there are other options.

High Availability — LAN — NIC Bundling

The parent article on High Availability.

Switching on a LAN provides some of the most basic network connectivity options, and are often overlooked. Nonetheless most switches (Cisco, HP, Dell and others) support these configurations, but one thing I can guarantee is that you will find limitations on pretty much every platform. If you’re after inter-operability, do your testing so you can understand these limitations.

Bundle Network Links

We want to bundle network links for two reasons; to aggregate bandwidth (two links give twice the packet-passing capacity) and for failover (if one link fails a second is still running).

LACP

I discussed LACP in an earlier article, but I would like to go into a little more detail here. Make sure you review Cisco’s documentation on configuring LACP, and the Wikipedia article on link aggregation.

In my experience, I find LACP to be the best solution for link aggregation. It is a common protocol so interoperability between devices is almost always possible and the configuration is sensible enough that you can explain it to a lay-person.

In the example above, we have bundled two physical links into a single logical link between two switches.

LACP Virtual Adaptors

When we bundle network links with LACP, each host creates a virtual adaptor that represents the bundle. For example, on a Cisco switch we can create an interface called portchannel 1, that represents the two interfaces fastethernet0/1 and fastethernet0/2.

In this case, instead of making changes to or examining the configurations of the physical interfaces we can instead work with portchannel 1. Of course you can work with the physical interfaces, but you must take care to make sure all parameters match on all physical interfaces in the bundle.

LACP Load-balancing Flows

LACP is flow-aware, and it can be configured to load-balance based on MAC address or IP address; the default in Cisco switches is to load-balance based on MAC address.

Load-balancing only really works when the system is able to identify many unique flows; as each flow is established it is put on one of the bundled links and all subsequent traffic also follows that physical link.

Be aware, that load-balancing based on MAC address (the default behaviour) may not be what you want — if your traffic crosses a router the original source MAC address will be obscured.

In a routed environment (if you’re using VLANs) you will find that any traffic that crosses a routing boundary will have its source MAC address replaced by the router MAC address. This can make many hosts appear (to LACP) as if they’re coming from a single MAC address and will definitely skew the load-balancing calculations. A better approach is to use IP based LACP load-balancing, as each host will likely have a unique IP address.

A good rule of thumb is to use MAC address load-balancing if you’ve got a flat Layer 2 network. It is easier for the switch to identify MAC addresses and in a network like this, every flow should be coming from a unique MAC address.

Server Based Failover and Load Balancing

Some NIC manufacturers have provided software to accommodate NIC failover and load-balancing without using LACP. See HP’s document describing the options they offer.

These configurations do work, however they are complex (in terms of traffic flow) and are therefore harder to troubleshoot in the event of network problems. Use LACP where possible, and these server based methods where necessary.