BackBloggging: Travel back in time with me.

Ok so I've missed blogging for like the whole month of November. The next few weeks I'm going to backblog. Backblog you say (as you scratch your head). Well follow the link to become informed.

Now page down every so often and catchup on all that you've missed in "The Life of Coe".

Work: Google here I come

So I flew to San Jose today to visit with great friends Aaron and Jayita. Aaron is a long time buddy from USC who now works at Google. He married this sweetheart of a lady Jayita, who will be appearing later in the blog. Aaron picked me up from the airport and took me over to Google's campus for lunch. Wow all I can say is that Google does not suck. This place was made for people like me. Whiteboards everywhere, in the halls, in the offices and even in the lobby. They have sand volleyball courts in the middle of campus and even have...get this... an endless pool for swimming laps. They serve three meals a day all free of charge and have wireless access throughout the campus. The employees are young, fresh and spirited. To top it off there is an endless supply if coffee and my new favorite energy drink RockStar. This trip provides the most incentive I've had for finishing my PhD.

Work: Symbolic links,nsf mounting and enough rope to touch the ground

Today I was suppose to give a presentation on XCP on the IXP. I was hoping to actually have XCP packets flowing through the IXP processor but I was having this little problem with the drivers for the gigabit controllers. Montavista's 3.1 release shipped the drivers for the development board compiled for a different platform. bad bad bad. Professor Papadopoulos has a friend at CMU that had the previous 3.0 release with the drivers compiled correctly. So I had already booted the development board and nfs mounted the file system to the IXP2400 with the incorrect drivers in the path before I remembered that the correct drivers were over on this other path. Now for you non-computer people out there I'm going to take five minutes to explain symbolic links. Symbolic links are a lot like worm-holes. They connect some point A with some point B in another universe. The difference ends there though and actually it is at the end of the path that really matters, let me give an example.
I tell you on Wednesday to travel down a path(link) called "/erics/short-cut/to/get/there/fast". Now in the morning that "link" will connect you to "/this/has/the/old/drivers". Once I redirect the symbolic link in the afternoon that same link "/erics/short-cut/to/get/there/fast" drops you off over here "/these/are/the/drivers/that/work".
Herein lies the problem and solution. In the morning I told the computer process "nsfd" that "/erics/short-cut/to/get/there/fast" goes over to here "/this/has/the/old/drivers". So when I told the IXP that it should follow the "/erics/short-cut/to/get/there/fast" it checked with nsfd(worm-hole traffic cop) and happily went to "/this/has/the/old/drivers". Now in the afternoon I needed the IXP to goto over to the new drivers. So I changed the symbolic link to point over to "/these/are/the/drivers/that/work". When the IXP2400 rebooted it went to load it's file system and followed the symbolic link "/erics/short-cut/to/get/there/fast" to "/these/are/the/drivers/that/work" when the IXP2400 checked with the nfsd(worm-hole traffic cop) the nfsd process said 'NEIN' 'No link for you'........do you see the problem? nsfd thinks that "/erics/short-cut/to/get/there/fast" resolves to "/this/has/the/old/drivers" but the IXP2400 tells nsfd that it wants to go to "/these/are/the/drivers/that/work" and nfsd says NO link for you. I rebooted nsfd and life was happy, more importantly the IXP2400 forwarded packets from port 0 to port 1.

Work: Happy Halloween *warning post contains rant

So yesterday I'm reading my email from work and notice a large delay in my filtering rules moving a message to a folder. Off to the folder I go to see what's happening. Well it seems the little network hackers started trick-or-treating a little early this year, at least this little bast##rd (210.230.200.199) was lighting up my machines with ssh attempts. By this I mean that he generated around 111 MB of log files in failed messages. So in my infinite wisdom (cue angelic music) I thought I would add his little address to our blacklisted sites. After attempting to log into our firewall unsuccessfully I remember that we do not allow access to our firewall unless you are sitting in front of it. I called a co-worker and here is what transpired.
Me: Dude need a favor. I need you to add a firewall rule to the block this address.
Him: Ok tell me what to do.
Me: log on to the KVM and add this rule.
Him: dude the machine is not responding to key-strokes.
Me: Hold on. ( I quickly confirm that the firewall is still up performing it's task as gatekeeper.) Ummm ok just reboot it.
Him: reboot it? Are you sure.
Me: Yeah just reboot it. It'll be back up in a second and it will see the keyboard and be happy
Him: ok. click.
Meanwhile I start pinging the firewall looking for signs of live. Then the phone rings.

Him: Dude umm... it can't find the disk.
Me: what.... what do you mean.
Him: um it's trying to PXE boot off the network.
Me: SOB! ok just power it all the way down and wait a little while. Then power it back up. (I'm starting to remember when we reboot it last.... oh yeah about 1.5 YEARS AGO)
Him: Dude you know we're offline right now.
Me: No sh*t sherlock. Just power it down and wait a while.

Now I'm asking around the other office that I'm at if anyone wants to give me a ride to work so I can fix the mess I've made. I'm about 30 mintues away with traffic and the airport. Plus quickly compose an email message to our userbase letting them know we're offline and everything is under control..Ring

Me: Hell-o
New Him: Um dude I just lost all my sessions to your lab.
Me: Yeah I know... we're having a little issue with our firewall.
New Him: I can't even ping the firewall.
Me: That's one of the issues.
New Him: When are you going to fix it?
Me: Read your email... I just sent a message. click

Ring.
Him: Dude it's not finding the disk and it's PXE booting again.
Me: Go into the BIOS and disable PXE boot.
Him: Did that.
Me: DAAAMMMNNIIIITTTTTT!!!!!! ( a mild earthquake was felt in Marina del Rey at this time) I'm on my way.
Him: Good I've got a meeting at 2:00. click.

I spent the next 8 hours trying to fix the current firewall before giving up and re-building a new firewall. I the mist of rebuilding the new firewall I realized we have no backups of the firewall configuration. Of course we have the rules saved off but the addressing, the static routes which move traffic heither and therether, not so much. I was helped by my partner in crime from USC who was online IM-ing with me until we re-installed everything . He even provided a little humor by trying to hack in with usernames like "never-reboot-domass", "what-were-you-thinking" which of course show up in the logs I've got scrolling across my window during our...um... exposed time.

Lesson Learned: If it's not a Microsoft Windows machine rebooting it might not fix it and could very well kill it. I've also proposed to block all of the APNIC assigned ip segments.... that region attempts breakins more than any other.