Recovering A Bad Flash — Part 1

Some people are perfect, they never make mistakes, and they never put themselves in a position where they have to dig themselves out of whatever problem they’ve caused.

I’m not that person. I make mistakes, but I learn from them. I learn how to solve the problems those mistakes cause, and I learn more about the systems themselves. I have to think this is pretty normal — and as my mentor Tom Jacoby always says: “the definition of an expert is someone who has already made all the mistakes!”

So yesterday I made a mistake that made me feel like a junior tech all over again — if only until I knew how I was going to fix it. I was upgrading software on four pre-production routers, and one of these had a flash that was just big enough for one image at a time. Standard procedure here is to delete the old image, load a new one on and reload the system.

This was no problem — I had already done another identical machine and it came up fine… The problem was this was the last router I had to work on so while it was copying I started cleaning up and then I find myself holding the power cables for all the routers in my hands. They’re all offline.

And now the last router will not boot — it just drops into ROMMON. I’ve recovered hardware over console xmodem and that is not nice — it can take hours to transfer a 30 meg image. Thankfully these routers use CF cards, and thankfully I have a USB CF reader. I mounted the card on my laptop, copied the new image over, booted the system and I was off to the races.

So what did I learn? Don’t rush the job. Think about what you’re doing and if you’re in a hurry just put that hurry out of your mind. Small mistakes can cause huge amounts of pain — if these were production routers instead of pre-production it would have been a very difficult day for me.

One thought on “Recovering A Bad Flash — Part 1

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s