RPC Hell

The beginning of the school year has finally arrived and thankfully, I’ve survived somehow. Since my last post, I’ve been busy getting ready to release all the cool new projects we’ve been working on all summer, but the whole end of summer rush only got worse because of RPC hell.

For those of you who have been living under a rock, I’m referring to the numerous RPC vulnerabilities on Windows NT, 2000 and XP machines and the unbelievably fast-spreading worms that have exploited them. As the computing organization that supports the approximately 10,000 on-campus housed students (almost all undergraduates and most graduates), we are responsible for a huge part of the campus network and total number of computers, especially considering 99% of students have their own computer. We had hundreds of computers hacked during the summer when there were very few people on campus and we knew that it would only get worse once school started and all 10,000 were back on campus– over 85% of our users run some flavor of Windows.

But we weren’t too worried.

The campus IT organization was (and still is) notifying administrators of hacked computers that they’ve been hacked and are throwing them into a DNS blackhole to further get their attention. They even provided a little tool that would scan (and clean) the most common worms as well as apply the latest hotfixes. Beyond that, we and our faithful student staff that live out in the residences could only work to educate students and help them get clean and patched when hacked. We figured that while fast-spreading, the worms weren’t doing TOO much damage and in the end, students are still adults and they should be expected to take care of their own machines.

The campus IT organization that essentially owns the wire we operate on didn’t see it the same way. Panic and hysteria ensued when they realized 1600 freshman would be arriving by the 18th and after that, the rest of the on-campus students that weren’t already here would arrive within days. Without getting into the details, I have spent A LOT of time over the past couple of weeks integrating tools to help ensure students get cleaned and patched before they get onto the network. Eighteen hour days for two weeks straight, especially when it’s something that we’re being forced to do, is not my idea of fun.

To summarize the situation, the analogy I use is this: there’s an STD going around. Our campus IT organization was treating it like HIV when really, it was just the clap– easily recognized and treatable. But more than that, it was like they were saying you couldn’t even come into casual contact with someone of the preferred sex until you had an STD test. In fact, better yet, they would have preferred everyone have their gonads removed and once checked, you could request to have them put back in/on. Draconian response, anyone?

Well, in the end, we’ll hopefully get some free X-Boxes out of this thing.