Work: Happy Halloween *warning post contains rant

So yesterday I'm reading my email from work and notice a large delay in my filtering rules moving a message to a folder. Off to the folder I go to see what's happening. Well it seems the little network hackers started trick-or-treating a little early this year, at least this little bast##rd (210.230.200.199) was lighting up my machines with ssh attempts. By this I mean that he generated around 111 MB of log files in failed messages. So in my infinite wisdom (cue angelic music) I thought I would add his little address to our blacklisted sites. After attempting to log into our firewall unsuccessfully I remember that we do not allow access to our firewall unless you are sitting in front of it. I called a co-worker and here is what transpired.
Me: Dude need a favor. I need you to add a firewall rule to the block this address.
Him: Ok tell me what to do.
Me: log on to the KVM and add this rule.
Him: dude the machine is not responding to key-strokes.
Me: Hold on. ( I quickly confirm that the firewall is still up performing it's task as gatekeeper.) Ummm ok just reboot it.
Him: reboot it? Are you sure.
Me: Yeah just reboot it. It'll be back up in a second and it will see the keyboard and be happy
Him: ok. click.
Meanwhile I start pinging the firewall looking for signs of live. Then the phone rings.

Him: Dude umm... it can't find the disk.
Me: what.... what do you mean.
Him: um it's trying to PXE boot off the network.
Me: SOB! ok just power it all the way down and wait a little while. Then power it back up. (I'm starting to remember when we reboot it last.... oh yeah about 1.5 YEARS AGO)
Him: Dude you know we're offline right now.
Me: No sh*t sherlock. Just power it down and wait a while.

Now I'm asking around the other office that I'm at if anyone wants to give me a ride to work so I can fix the mess I've made. I'm about 30 mintues away with traffic and the airport. Plus quickly compose an email message to our userbase letting them know we're offline and everything is under control..Ring

Me: Hell-o
New Him: Um dude I just lost all my sessions to your lab.
Me: Yeah I know... we're having a little issue with our firewall.
New Him: I can't even ping the firewall.
Me: That's one of the issues.
New Him: When are you going to fix it?
Me: Read your email... I just sent a message. click

Ring.
Him: Dude it's not finding the disk and it's PXE booting again.
Me: Go into the BIOS and disable PXE boot.
Him: Did that.
Me: DAAAMMMNNIIIITTTTTT!!!!!! ( a mild earthquake was felt in Marina del Rey at this time) I'm on my way.
Him: Good I've got a meeting at 2:00. click.

I spent the next 8 hours trying to fix the current firewall before giving up and re-building a new firewall. I the mist of rebuilding the new firewall I realized we have no backups of the firewall configuration. Of course we have the rules saved off but the addressing, the static routes which move traffic heither and therether, not so much. I was helped by my partner in crime from USC who was online IM-ing with me until we re-installed everything . He even provided a little humor by trying to hack in with usernames like "never-reboot-domass", "what-were-you-thinking" which of course show up in the logs I've got scrolling across my window during our...um... exposed time.

Lesson Learned: If it's not a Microsoft Windows machine rebooting it might not fix it and could very well kill it. I've also proposed to block all of the APNIC assigned ip segments.... that region attempts breakins more than any other.