Recovering A Bad Flash — Part 2

Today I encountered another flash problem. I was working on an older switch, a WS-C2950G-48-EI running 12.1(13)EA1c. My client had asked me to upgrade the software because he wanted access to the html interface on the switch.

Now I’m at least an hour away from this client so I wanted to do the upgrade from my office, but I don’t like to do TFTP over the greater Internet as there’s no error correction with this.

With newer hardware we can upgrade with HTTP — this is fantastic because I can just load the code onto a webserver here in my office, punch a hole in the firewall and upgrade the hardware. Easy as pie.

But this old switch only supports FTP and TFTP. I figured I’d give FTP a whirl — it is a bit more complex than http (okay a lot more complex) but it should work.

[code]]czo3NTpcImNvbmYgdA0KaXAgZnRwIHVzZXJuYW1lIHBhdWwNCmlwIGZ0cCBwYXNzd29yZCBzb21lcGFzcw0KZW5kDQpjb3B5IGZ0cCB7WyYqJl19Zmxhc2hcIjt7WyYqJl19[[/code]

And away you go! You can also do this:

[code]]czo1MzpcImNvcHkgZnRwOi8vcGF1bDpzb21lcGFzc0BzZXJ2ZXJpcC9maWxlbmFtZS5iaW4gZmxhc2g6XCI7e1smKiZdfQ==[[/code]

You can do this too if you’re working with Cisco’s tar files:

[code]]czo2ODpcImFyY2hpdmUgdGFyIC94dHJhY3QgZnRwOi8vcGF1bDpzb21lcGFzc0BzZXJ2ZXJpcC9maWxlbmFtZS50YXIgZmxhc2g6XCI7e1smKiZdfQ==[[/code]

I initally had a problem with FTP getting through the firewall — the client was throwing an error: “no such user”. That didn’t make sense because the username definitely works. I double checked locally, and remotely from another server to make sure my credentials worked.

The problem was that FTP uses a data channel (port 20) in addition to the control channel (port 21) and this can be a bit confusing for firewalls to track the sessions. So I turned on FTP inspection on my office firewall and things seemed to work, at least the debug suggested that the client was able to login.

You can debug FTP client sessions like this:

[code]]czo3MDpcImRlYnVnIGlwIGZ0cA0KdGVybSBtb24gIyB0aGlzIHdpbGwgcmVkaXJlY3QgZGVidWcgbG9ncyB0byB5b3VyIHNlc3Npb257WyYqJl19XCI7e1smKiZdfQ==[[/code]

But it still wasn’t working. The sessions would login, start a download but never finish. The debugs showed that the client sent an ABOR code — which aborts the download.

Interestingly, many clients don’t handle the ABOR command well. The site FTPGuide.com says: The abort command may require “special action” to force recognition from the server. Unfortunately the only special action I had at my immediate disposal was to kill the terminal session.

That was my fatal mistake.

Three FTP sessions later, and I can’t view the flash drive anymore. sh flash and dir both report:

[code]]czozOTpcIiVFcnJvciBvcGVuaW5nIGZsYXNoOi8gKE5vIHN1Y2ggZGV2aWNlKVwiO3tbJiomXX0=[[/code]

Now I’m in trouble. I’ve deleted the old image to make space for the new one, and now I can’t write the new image because the flash is non-responsive. I was able to erase flash and format flash but while these claimed to work, neither of them helped to make the flash usable again.

Still I suspected this problem was related to the FTP sessions I attempted, so I found a new command (new to me):

[code]]czoxOTU6XCIjc2ggZmlsZSBkZXNjcmlwdG9ycw0KRmlsZSBEZXNjcmlwdG9yczogRkQgUG9zaXRpb24gT3BlbiBQSUQgUGF0aA0KMCB7WyYqJl19MCAwMDAxIDUyIGZ0cDovL3NlcnZlci9jMjk1MC1pNmsybDJxNC10YXIuMTIxLTIyLkVBMTMudGFyDQoxIDAgMDAwMSA2NSBmdHA6L3tbJiomXX0vc2VydmVyL2h0bWwudGFyDQoyIDAgMDAwMSA2NiBmdHA6Ly9zZXJ2ZXIvZm9vLnR4dFwiO3tbJiomXX0=[[/code]

Ah ha! Clearly this is my problem. These FTP sessions are tying up the flash, if only I could kill the PIDs listed here, but after much searching I find that Cisco doesn’t allow us to kill individual PIDs. I guess they feel that if a PID doesn’t close nicely then something is so wrong with the device that it needs more help than just a kill.

But that doesn’t help me. I realize that I can clear tcp sessions that are originating from the system, if only I know the source and destination ports. I had my firewall log open and was able to pick out the ports and clear these sessions, but I could also have used show ip sockets to figure it out.

[code]]czo1MDpcIiNjbGVhciB0Y3AgbG9jYWwgc3dpdGNoSVAgMTEwMjAgcmVtb3RlIHNlcnZlcklQIDIxXCI7e1smKiZdfQ==[[/code]

After clearing a few of these I was able to get access to the flash drive and load an image back on. This time I asked my client to setup a TFTP server locally and I used that.

The moral of the story — don’t use FTP.

3 thoughts on “Recovering A Bad Flash — Part 2

  1. show ip sockets wouldn’t do the job. Better luck with show tcp brief

    regards,
    Jesus

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s