Some people are perfect, they never make mistakes, and they never put themselves in a position where they have to dig themselves out of whatever problem they’ve caused.
I’m not that person. I make mistakes, but I learn from them. I learn how to solve the problems those mistakes cause, and I learn more about the systems themselves. I have to think this is pretty normal — and as my mentor Tom Jacoby always says: “the definition of an expert is someone who has already made all the mistakes!”
So yesterday I made a mistake that made me feel like a junior tech all over again — if only until I knew how I was going to fix it. I was upgrading software on four pre-production routers, and one of these had a flash that was just big enough for one image at a time. Standard procedure here is to delete the old image, load a new one on and reload the system.
This was no problem — I had already done another identical machine and it came up fine… The problem was this was the last router I had to work on so while it was copying I started cleaning up and then I find myself holding the power cables for all the routers in my hands. They’re all offline.
And now the last router will not boot — it just drops into ROMMON. I’ve recovered hardware over console xmodem and that is not nice — it can take hours to transfer a 30 meg image. Thankfully these routers use CF cards, and thankfully I have a USB CF reader. I mounted the card on my laptop, copied the new image over, booted the system and I was off to the races.
So what did I learn? Don’t rush the job. Think about what you’re doing and if you’re in a hurry just put that hurry out of your mind. Small mistakes can cause huge amounts of pain — if these were production routers instead of pre-production it would have been a very difficult day for me.