On the morning of December 20, 2004, I booted up my computer and visited DSL/Cable Webserver and I got this message:

Crap. 

Maybe only the front page got hacked?  I checked another page and that was defaced too. 

Crap.

I check the forums....

Crap.

I started to think, all in all, I've been pretty lucky over the years.  The site has has been up for 4 years without a major hacking.  Not too bad I figured.  Previously, a few commercial sites I work with had been hacked within 6 months.  4 years isn't too bad right?  Still a pain in the butt though.

The first thing on my mind was obviously: "Crap!".

the next thing was, "How do I fix this thing - fast?".  I did a search on Google and Google groups for "NeverEverNoSanity" and there were ZERO results.  Hmm... nobody ever heard of this worm?  That's just great. 

I was rather touched to see that I had received several dozen emails from visitors and friends to let me know that my site had been hacked.  That was cool and embarrassing at the same time.

I sat in front of my server and logged in.  Whew!!  I was afraid that the server was so compromised that I wouldn't be able to log in at all.  I noticed that the CPU usage was running around 80-90% constantly.  This is extremely unusual since typical CPU usage runs around 3-5%.  Either my server was still being attacked or my server was still in the process of spreading the worm and attacking other people.  Either senario was not good.

I checked out the files on my server and saw that all my files ending in .html and .php were overwritten with the defaced message.  What was really surprising was that files OUTSIDE of the web publishing directory were affected as well.  This really made me mad since I had saved several web pages from over the years and they were all now defaced!  All the articles and pages that I downloaded were gone.  ARRGGHHHHH!!!!!  There is a moral to this story.

I backup my website every week so luckily I had a recent version of my site.  I simply re-uploaded all the content and everything seemed to be okay.

But I knew that since I didn't fix the vulnerability in the site, I could get hacked again.  But I didn't know where the weakness was.  I searched some more on Google and saw 2-3 new entries about sites being attacked.  Finally, some news, but no real details on what was the real problem.  So I started to try to "fix" everything that I could think of.

I first ran the IIS lockdown tool again to make sure that everything was closed on that end.

Then I searched on the web and found that PHP had a recent vulnerability and that might be the problem.  So I download and installed the latest PHP engine.  After restarting the server, the CPU load was still extremely high.

The last thing I could think of was our forum software, PHPBB.  I was kind of doubtful that this was the problem since I didn't think that a flaw in PHPBB could affect all my files on the server.  But hey, I was desperate and willing to try anything. 

You might be asking.  Why wasn't your PHPBB updated before the attack?  Was I being extremely negligent?  Well, my answer is that no, I wasn't exactly being negligent, I was being lazy.  You see, I made a lot of customizations and modifications to PHPBB.  You couldn't see the mods as a forum user, but they were for search engine indexing and admin stuff.  If I were to update my PHPBB to a newer version, I would have lost many of those mods.  I was being lazy in updating the software since I would have to reinstate all those mods again, something I wasn't really looking forward to.

Back to our story, after updating PHPBB to the latest version, the CPU usage dropped back down to normal levels.  Bingo.  PHPBB was the culprit.  Well, I guess I was actually the culprit since PHPBB had issued a fix a few months earlier.

So aside from the personal files I had stored on the server, I didn't lose too much data from the website.

At the end of the day, I did a final search on Google and found thousands of entries for "NeverEverNoSanity".  A search on MSN found over 40,000 hits!  I guess that worm really got around.

Symantec released this page with detailed information about the worm: 

• http://securityresponse.symantec.com/avcenter/venc/data/perl.santy.html

By the next day all the major anti-virus companies released virus definitions for their anti-virus software. 

Google did their part as well by blocking requests from the worm to spread.  Although they probably could have blocked the worm much earlier.  They must have noticed the strange pattern of searches.  Oh well...

A few days later, there was another variant of the worm that tried to exploit the same hole in PHPBB.  My site was targeted, but it didn't succeed.  How did I know I got hit?  Well, I found it very suspicious that our forums suddenly got 500+ concurrent visitors.  A quick trip to the PHPBB website confirmed that other people were getting the same erroneous member count.  Good thing I was patched.  (finally)

• http://securityresponse.symantec.com/avcenter/venc/data/perl.santy.b.html

1. Keep your software up to date.  This includes the operating system, all the software (webserver, mail, ftp), all the scripting languages (perl, php), and scripting programs (phpbb, gallery).
2. Get on the mailing list for each of the products you use.  They will then let you know when a critical update comes around.
3. Make consistent backups.  The reason I was able to get the site back up so quickly was because I had a backup on hand.  Back up to something permanent like a cdr or dvdr, not to something that can be erased (hd, zip).
4. Don't store personal files on your server.  Assume everything on your server is public.  If you put stuff on the server, it could get damaged like my stuff was, or it might be viewed on the internet.  Either option is bad.
5. Use non standard ports whenever possible, especially for FTP and Remote Desktop.  Using non standard ports wouldn't of helped my case, but for other types of attacks where widely known port numbers are targeted, it would be good to use non standard port numbers. 

Brian
 

Take Me Home!