This is a read-only archive. Find the latest Linux articles, documentation, and answers at the new Linux.com!

Linux.com

Feature: News

Mystery infestation strikes Linux/Apache Web sites

By Joe Barr on January 24, 2008 (7:18:05 PM)

Share    Print    Comments   

According to a press release issued earlier this month by Finjan, a security research firm, compromised Web servers are infecting thousands of visitors daily with malware that turns their Windows machines into unwitting bots to do the bidding of an as yet unidentified criminal organization. Security firms ScanSafe and SecureWorks have since added their own takes on the situation, though with varying estimates on the number of sites affected. All reports thus far say the compromised servers are running Linux and Apache.

According to an article on ServerTune.com, the exploit involves a rootkit installed on the compromised server that replaces several system binaries with infected versions. When the system is booted, the infected binaries are executed, and as a result, dynamically created JavaScript payloads are randomly and intermittently served to site visitors. The malware JavaScript attempts to exploit vulnerabilites in Windows, QuickTime, and Yahoo! Messenger on the visitor's machine in order to infect them.

We asked the Apache Software Foundation if it had any advice on how to detect the rootkit or cleanse a server when it's found. According to Mark Cox of the Apache security team, "Whilst details are thin as to how the attackers gained root access to the compromised servers, we currently have no evidence that this is due to an unfixed vulnerability in the Apache HTTP Server."

We sent a similar query to Red Hat, the largest vendor of Linux, but all its security team could tell us was that "At this point in time we have not had access to any affected machines and therefore cannot give guidance on which tools would reliably detect the rootkit."

cPanel, a popular administration tool used by hosting companies that allows clients to manage their hosted sites, has posted a security note describing what the rootkit does after it's installed, and suggests two ways to check a server for the rootkit.

According to cPanel, if you are unable to create a directory name beginning with a numeral -- as in mkdir 1 -- you're infected. Another test is to monitor the packets from the server with the following tcpdump command:

tcpdump -nAs 2048 src port 80 | grep "[a-zA-Z]\{5\}\.js'"

One great unknown thus far is how the servers come to be infected. Absent any forensic evidence of break-ins, the current thinking is that the malware authors gained access to the servers using stolen root passwords. The earliest known victims, according to quotes by researchers in this ComputerWorld story, were sites run by large hosting companies, which could give attackers root access to hundreds or even thousands of Web sites when compromised.

Other than using and safeguarding secure root passwords, not much can be done at this time to be proactive in preventing servers from being compromised, so searching techniques similar to the tcpdump command above, which check to see if a server has already been compromised, is probably the best course of action available to administrators. We haven't found a good answer yet for disinfecting compromised servers, but a complete reinstall of Linux, Apache, and a new root password would certainly do the trick.

Share    Print    Comments   

Comments

on Mystery infestation strikes Linux/Apache Web sites

Note: Comments are owned by the poster. We are not responsible for their content.

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 129.33.49.251] on January 24, 2008 07:54 PM
sounds like FUD to me... especially since the only people who can be affected on the other end is windows users

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 64.106.131.250] on January 24, 2008 08:08 PM
Funny that the article focuses on the webservers and the unknowns and doesn't talk about the known vulnerabilities in the windows apps.

#

Re: Mystery infestation strikes Linux/Apache Web sites

Posted by: Joe Barr on January 24, 2008 08:26 PM
"Funny that the article focuses on the webservers and the unknowns and doesn't talk about the known vulnerabilities in the windows apps."

We're more interested in the Linux side of things here on Linux.com. Besides, malware on Windows isn't really news.

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 81.11.224.52] on January 24, 2008 08:18 PM
you forgot to mention that cropcircles are burned in the harddisks of the infected servers...

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 70.106.31.190] on January 24, 2008 08:33 PM
I'm pretty sure that the grep regex matches any file with AT LEAST 5 chars before the .js. A slash / at the begining of the regex should be included to limit to only 5 character names.

#

Re: Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 131.207.242.5] on January 25, 2008 06:52 AM
notice the bakcslashes before { and }
the regexp [a-zA-Z]\{5\}\.js' matches any string ending with [alpha]{5}.js' where [alpha] is letter from a-z or A-Z

#

Re(1): Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 70.106.31.190] on January 25, 2008 03:15 PM
My point is that the security note over at cpanel says that the file names that it creates are random 5 char names and then .js. Always exactly 5 chars. The regex posted will match any .js file with AT LEAST 5 chars. It will match abcde.js and it will also match abcdefghigklmnop.js. The article says that if you get any output with the posted regex then you are most likely infected. That's not true. In order to fix the regex, you would put a slash character at the beginning of the regex to match a directory , then have the [a-zA-Z]...

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 139.80.23.83] on January 24, 2008 08:37 PM
I think that in some cases, 'compromised' might be the wrong word. I imagine that more than a few of these Linux servers are intentionally configured to infect other computers. In such cases, the Windows-based computers on the client-side are the ones being compromised...whereas the server may actually be doing what its owners/operators intended.

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 82.192.250.149] on January 24, 2008 08:46 PM
An anonymous Microsoft spokesperson said, 'I think that in some cases, 'compromised' might be the wrong word. I imagine that more than a few of these Linux servers are intentionally configured to infect other computers. In such cases, the Windows-based computers on the client-side are the ones being compromised...whereas the server may actually be doing what its owners/operators intended.'.

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 195.174.15.254] on January 24, 2008 10:41 PM
There are very serious vulnerabilities in older versions of Yahoo Messenger and Quicktime. It seems the worm guy didn't make this for hobby or some 133t show off, there must be some money involved. Yahoo and Quicktime exploits aren't exploited first time and using a web server to try infecting them makes a perfect sense. I don't get how "windows applications targeted" disvalidates the entire security alert. I am sure the criminal purchased the rootkit (yes, like eBay now) can also purchase some previously unknown, huge list of compromised web hosting accounts data.
Malware, professionally coded for money running on Apache/Linux attacking Windows clients isn't news?

#

Old news

Posted by: Anonymous [ip: 72.185.64.219] on January 24, 2008 11:10 PM
Old news that dates back to November. Nice so called reporting.

http://www.channelregister.co.uk/2008/01/16/mysterious_web_infection_continues/

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 159.238.13.56] on January 24, 2008 11:11 PM
You know, if the got a few servers, and infected Windows hosts using them, there is a chance that one of the things running on the Windows clients is a keylogger, and that let them get some more, and so so. If you get root to a large number of machines at a large hosting company, then you could parlay that into more root passwords, that loading to more root passwords, etc. If they were smart, this has been going for a long time, and just flying under the radar until now.

#

Re: Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 204.137.64.112] on January 28, 2008 04:33 PM
That's a really scary question. Who would you put your money on for doing something like this?

#

does this affect windows pc only? unix, osx, linux affected or not?

Posted by: Anonymous [ip: 60.49.184.157] on January 25, 2008 02:57 AM
does this affect windows pc only? unix, osx, linux affected or not?

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 66.185.228.194] on January 25, 2008 03:02 AM
Umm. No one has heard of MD5 checks? Seems to me, given the nature of Linux, it shouldn't be that hard to figure out which files are the problem, generate correct MD5 checksums for the valid ones, then test the servers that could be infected using a liveCD. If it finds invalid MD5 checks for those files, replace them, reboot the server, and you are done. Or, if you are not sure about the compatibility with that, have your scripted process do a download of the source for the infected files, and their dependencies, then a 'make' on them. I barely know anything about Linux, have only for the most part used it from liveCDs to fiddle around, and even ***I*** can figure this much out. What kind of experts are these? Are they reading tea leaves to try to figure out the problem?

#

Re: Mystery infestation strikes Linux/Apache Web sites

Posted by: TK on January 25, 2008 03:56 PM
Using MD5 checksums is good to catch the issue AFTER it happens, actually an excellent step for admins to implement in different ways (perhaps tripwire?). However, most of these researchers know what you are talking about but are now trying to figure out HOW it happened, what the vector was. This way, security patches can be made, admins can be taught (especially if it's a matter of poor planning or implementation of security policy), and perhaps this can be prevented in the future. In other words, they've moved past just changing the blown tire and are now into making a run flat. :) [edited first sentence-TK]
[Modified by: TK on January 25, 2008 04:02 PM]

#

MD5 is not safe Re(1): Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 192.168.1.33] on January 28, 2008 07:02 PM

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 66.185.228.194] on January 25, 2008 03:07 AM
"does this affect windows pc only? unix, osx, linux affected or not?" Its worse than that, based on a different post on it. No one knows *what* is causing it, the only thing they have in common (since all sorts of versions exist and are effected), is something called cPanel, and reconfiguring the server seems, at least in some cases, to cause the problem to just vanish into thin air... Basically, no one has a clue at this point.

#

Re: Mystery infestation strikes Linux/Apache Web sites

Posted by: TK on January 25, 2008 04:04 PM
Right now, it appears to be LAMP sites (with CPanel being another common denominator) that are being exploited. These servers, in turn, send exploits to Windows PCs.

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 69.30.67.10] on January 25, 2008 04:23 AM
regarding md5, it won't do crap for you if the machines been rooted cause the md5 app will have already been compromised!

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 69.30.67.10] on January 25, 2008 04:25 AM
damn should have read the md5 paragraph more closely yup need to do it with a livecd!

#

Checksum or not..

Posted by: Anonymous [ip: 68.47.230.235] on January 25, 2008 05:56 AM
I understand that md5sums can be used to determine file modifcations, and yes you could configure any HIDS system to do that for you, but the question at this point is how does the intrusion occur? That's what people want to know.

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 212.179.141.1] on January 25, 2008 07:27 AM
I'm quite certain that once the Apache/Kernel devs see an infested machine + a network log they will tear it apart trying to see how it happened, IF it happened at all.

The problem is that until now no one knows anything about this thing other than "it infected some unknown server and installed a rootkit". well thanks a million for the great info, reminds my of my cousins asking me to fix their PC on the phone telling me "I get a blue screen with some white text on it"

Personally I still don't know IF this is true at all. Does anybody have the IP of an infected box? a name? ANYTHING? Call me ignorant but if someone wanted to make the L and A in LAMP look bad, this would be exactly the thing he/she'd do.

Note: I am running a LAMP server, as soon as I heard about this I made a list of MD5 checksums of all of my files using a liveCD(yes I took the server offline, security over uptime) and compared those to the checksums in the Debian Stable repository. Yesterday they checked out, I will check again tonight.

PS: How do you get a linebreak on Linux.com?
isn't working, neither is \n or a normal newline...

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 84.90.182.83] on January 25, 2008 11:05 AM
Anyone heard of tripwire?

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 163.206.119.91] on January 25, 2008 01:32 PM
There are some good posts here. I have always done the MD5 thing on a informal, personal list of libraries and apps (and, of course the kernel. ). I can boot from the install disks (or a live CD, as someone suggested) whenever I want, and check around. Once I know I can trust the basics, I can boot back into the OS, and continue looking around if needed. I'm not even a professional. Seems like some of these "professional" websites are not run by professionals. That's one of the downsides of all of these Windows shops migrating their servers to Linux. You get a bunch of admins that have never learned the basics.

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 128.186.4.84] on January 25, 2008 02:28 PM
'If it finds invalid MD5 checks for those files, replace them, reboot the server, and you are done'
---
Yeah, no, not so much.

You haven't even looked to see if a new service has been added, a new privileged account added, or some other back door installed. It's easier and simpler to make a copy of your configuration files and reinstall the system from scratch. Actually, it'll take less time than booting your liveCD and verifying the MD5 sums.

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 219.93.152.11] on January 25, 2008 03:20 PM
It happened to my server. All file under /bin were replaced. We believe it happen through brute force of SSH. The Apache get compromise throught CMS.

#

Re: Mystery infestation strikes Linux/Apache Web sites

Posted by: TK on January 25, 2008 04:07 PM
If this is the same exploit, can you submit an image of your compromised server to SANS or another trusted researcher for forensics? This would help tremendously!

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 151.205.99.143] on January 25, 2008 03:28 PM
I am not a tech person, but could this be the reason that Network Solutions LLC lost all my 90GB of video files and do not have the decency to reply?

#

Re: Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 76.110.132.102] on January 25, 2008 04:36 PM
Wow, that's a lot of pr0n.

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 207.99.52.161] on January 25, 2008 07:33 PM
My take from this article is that only the Windows clients are adversely affected.

This infestation dilemma only proves that those who use Linux/apache server-side with their Windows clients
can't have it both ways - security/reliability and familiar/popular access.

Either they need to get Microsoft IIS/SQL Server back end or Mac/Linux front ends.
Whichever way, they will come more quickly and clearly to their ultimate truth.

There is no question as to which is the better - in terms of reliability, good security, scalability, flexibility, interoperability/standards based and significantly reduced support/service and overall costs.

The only thing you really get with the other choice is the comfort that you are acquiring/using the same crappy
technology as everyone else that does not have a clue - i.e. most of society.

W. Anderson
wanderson@nac.net

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: googlingtingwana on January 25, 2008 10:58 PM
I run a small personal server that a few friends and family use; mainly for email and web hosting. When I read about this "infection" on slashdot I checked out my system based on the available information. The only symptom I found was: run tcpdump as suggested; access my server's home page; the tcpdump produced output like that expected from the infection, as follows: <script language='JavaScript' type='text/javascript' src='shfuy.js'></script>
. I searched the entire file system and there is no "shfuy.js" file, and I wasn't really expecting one. I looked for other discussed symptoms: 1) created a directory called "1" which was successful; 2) rebooted the system from a live CD and checked by /bin and /sbin directories - there were no renamed utilities such as ifconfig or route (nothing with a bunch of digits after the name); 3) there was no "bwlimited" module running within apache (output from apachectl -t -D DUMP_MODULES); 4) did wget on my server's home page from a couple of different IP addresses - no tcpdump regex matching output. This suggests to me that either I'm not compromised, or it is a different variation on the problem than others have seen.

[Modified by: googlingtingwana on January 25, 2008 04:00 PM]

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 213.37.69.211] on January 26, 2008 12:22 AM
Keywords: FUD FAKE HOAX

oops!, I have used 1234 as root pasword and now my server is infected with a rootkit.
Warning! Linux/Apache servers are not protected against human stupidity, use Windows instead.

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Martin Kaba on January 26, 2008 11:13 AM
I've read a couple of times of this plague on Apache, it is also said that intruders make use of stolen accounts to get in, together with the Apache feature – “dynamic module loading” that most web masters do not master.
Wrote on this a few days ago.

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 193.254.61.192] on January 26, 2008 11:27 AM
Stop that regexp stupidity. The cPanel example was meant as a quick check, not a perfect scan. And FWIW, [a-z]{5} will match exactly 5 characters, period. It will also match abcdefghktrj.js because it matches against the last 5 chars before .js, not because [a-z]{5} matches 5 or more chars. To get exact matches you'd have to add a ^ in front of the regexp, but then you'd have to match against the file name only; which is not practical since the input is from tcpdump.

#

Re: Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 72.218.28.122] on January 26, 2008 04:05 PM
it's not stupidity. Just put a freaking / at the beginning of the regex to match a directory marker and then you are guaranteed that you are matching a slash and then 5 chars. I simply don't want a bunch of admins wiping their servers because they think they've been rooted.

And btw,you can't put a ^ at the beginning because it's parsing a url. You are trying to parse http;//www.example.com/abcde.js or www.example.com/mydir/fghij.js. before you start with the stupidity crap, you should first know what the hell you're talking about.

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 89.211.38.190] on January 26, 2008 11:29 AM
It is the bloody rootkit! We where Infected to and we manage to fix it!

#

Beware of some Asterisk soft-phones

Posted by: Anonymous [ip: 72.147.45.77] on January 26, 2008 01:51 PM
Just the other day a co-worker's Trixbox (Asterisk VOIP server) got rootkited. How it happened was that he installed a soft-phone software on his PC. This software asked where the trixbox is and then asked for the root password to it. He thought that was a little odd but he trusted the soft-phone (which he says was from Amsterdam!). Days later we got complaints of port scanning. A complete reload of the server was in order. He's still kicking himself for this.

#

ARP Spoofing?

Posted by: Anonymous [ip: 127.0.0.1] on January 29, 2008 05:30 AM

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 75.165.41.202] on February 03, 2008 11:21 AM
Yes, it is probably brute force, not a ~real~ compromise, certain netblocks seem to get hit harder. I manage several servers in different locations; one server is pretty quiet, hardly ever gets bothered. But a new server I recently brought online in a different location is getting upwards of 50k-100k ssh attempts per week!!! yow!!! I put a 'scumblocker' program on there, seems like ssh does not place any limits on failed attempts, so I added a program that monitors the message log and starts blocking the ip after n fails. Yup, lots of brute force going on. but it seems to be pretty stupid stuff, very repetitive. one thing they do is test all the services accounts, don't you dare leave any with default passwords or shell access.


as for MD5 -- hey thanks for the link!! I've been hearing rumors about the compromise but this is the first time seeing the article. Here is the link again for any who missed it. http://www.mathstat.dal.ca/~selinger/md5collision/


The thing about md5 collisions is that the length is not the same. so if you compare byte count + md5 and for good measure do a crc32 as well. that combo will still detect the change.


However, you might find it simpler to use a Version Control System, yup that's what I did, but only for certain directories, it works great! Give www.bazaar-vcs.org a try, you might find that it does the trick. Even without virus worries, I find it is very useful for tracking changes to /etc and also to websites. Unlike most VCS, Bazaar does a decent job of dealing with binaries and renames.


codeslinger (compsalot)

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 75.165.41.202] on February 03, 2008 03:31 PM
strange stuff in my log

Any ideas what this is??? There are a bunch of these, looks like someone is using/spoofing IPv6 to access my server. strange thing is the server does not have an IPv6 address.

::1 - - [14/Jan/2008:17:52:14 -0000] "GET / HTTP/1.0" 200 2037

another strange entry, perhaps trying to see if my server is a proxy?
85.190.0.3 - - [15/Jan/2008:19:12:55 -0000] "CONNECT 213.92.8.7:31204 HTTP/1.0" 405 288
85.190.0.3 - - [15/Jan/2008:19:12:55 -0000] "POST http://213.92.8.7:31204/ HTTP/1.0" 404 261
85.190.0.3 - - [16/Jan/2008:23:31:47 -0000] "CONNECT 213.92.8.7:31204 HTTP/1.0" 405 288
85.190.0.3 - - [16/Jan/2008:23:31:47 -0000] "POST http://213.92.8.7:31204/ HTTP/1.0" 404 261
85.190.0.3 - - [17/Jan/2008:03:15:22 -0000] "CONNECT 213.92.8.7:31204 HTTP/1.0" 405 288
85.190.0.3 - - [17/Jan/2008:03:15:22 -0000] "POST http://213.92.8.7:31204/ HTTP/1.0" 404 261


sorry the formatting is all screwed up, this forum software is in dire need of improvement. The log entries start with 85.190 I have not changed the IPs I do not wish to protect the guilty. this was from a dev/test server, there should be no one on it other than myself. -- codeslinger

#

Re: Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 76.5.223.180] on February 24, 2008 02:05 AM
I found the exact same thing in my logs today.

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 75.165.41.202] on February 03, 2008 03:54 PM
never mind the question about the proxy probe, turns out to be sort of legit. I should have checked the ip before posting, just that it annoys me that anybody is connecting to my test box. The ip belongs to freenode irc which was scanning to see if I was spoofing them. However none of this explains the truly weird IPv6 address. ::1 since none of my boxes are configured to support it.

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 61.95.196.73] on February 06, 2008 01:00 PM
podey podey

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: [SwS]I'm So Freakin' Cool on February 11, 2008 11:14 AM
This is why I'm ditching Windows. One tiny little security threat and... THE WHOLE COMMUNITY FREAKS OUT. The same can't be said for Windows instead it's closer to like a gazillion hacks, viruses, and spyware threats on a regular basis and people are so used to it they just dismiss it as minor annoyance.

#

Mystery infestation strikes Linux/Apache Web sites

Posted by: Anonymous [ip: 190.80.148.219] on March 02, 2008 02:02 AM
AVG now has a free RootKit detection program for Windows but not Linux yet. They offer AVG Anti-virus 7.5 for Linux distros such as Debian, Ubuntu, SuSe, RedHat Fedora Core, Novell Linux and Mandrake/Mandriva. It might be a good idea to contact AVG support and see if they can soon offer a Linux RootKit detection tool. I have a dual boot Sony VAIO VGN-CR123E with a 2GHz Core 2 Duo and 2 gigs of RAM running Vista & Kubuntu Hardy Heron and on Vista I'm using their new free AVG Anti-virus and Anti-spyware and the RootKit Detection Tool and they work pretty decent. Here's a couple links to scope it all out...

http://free.grisoft.com/doc/5390/us/frt/0 AVG Downloads

http://free.grisoft.com/doc/5390/us/frt/0?prd=afl Linux Downloads

Later man...

#

This story has been archived. Comments can no longer be posted.



 
Tableless layout Validate XHTML 1.0 Strict Validate CSS Powered by Xaraya