System Environment: Hardware......: Generic PC i486dx 16Mhz dual-homed router (2 NICs) OS............: RedHat Linux 6.2 Zoot (base install) Primary Role..: NAT Router Other Services: dns,http,ftp,pop3,smtp,telnet,ssh Problem.......: See the line above. The Detection: This machine was performing the role of a NAT router for my network. It was on the "edge" of the network which is why it also performed the services listed above. All internet connectivity passed through this box, in both directions. I first noticed something was different when my downloads began to slow down significantly. From a Windows machine, I tried to ping an internet DNS server and over 80% of the packets sent were lost. I pinged an internal host and the packet loss was 0%. Something was wrong in the linux box. I logged into the linux box and tried to ping the same DNS server, similar results, 80% packet loss. Performing a traceroute to the DNS host resulted in slow response times from each router, no specific router was unusually slow. I determined from this that the problem was not routing or NAT related, I checked the hub (for flashing lights) to make certain it was not an internal host flooding the gateway. The conclusion was that there was something going on at the linux box that was unusual. Next, I used the "top" command to find what processes were consuming resources. A program called 'wscan' was using 89-92% of the CPU. I exited top and decided to find out exactly what wscan was. I typed 'man wscan' but there were no manual entries for wscan. I then tried to executed wscan with the --help argument but wscan was not a valid executable within my PATH variable (as root). At this point I realized something of significance was going on. Just to make sure, I checked the cron/ folders to make sure this was not some type of system maintaince script or exe. No luck. I decided to kill the process (killall wscan) and continue on downloading some files (after writing down wscan on a scrap sheet of paper). All was well after that. I did type "wscan linux" at google.com just to see what turned up, I became slightly more concerned when almost all of the sites containing a reference to wscan also had domain names similar to security.com, securify.com, insecure.org, etc. By this time, it was about 03:00 and time to go to bed. The next day, I was still suspicious of a security compromise but thought "that only happens to .MILs, .GOVs, and large transaction farms with ordering (credit card) databases". I logged onto the linux box just to have a look around. Things were slow again. I 'top'ed to see the system processes and at the very top consuming around 90% of the CPU was a program called "z0ne". Notice the 0 is not an uppercase O, it is a zero. Most all linux users and people who have researched unix security know what this means... So called 'elite' hackers often use this type of notation in their call signs and blackhat applications. As luck has it, linux programmers aren't that creative with their application naming. After discovering this 'z0ne' program running, I knew what was going on and immediately killed the process. What next? To see all outbound TCP/IP connections, I issued the 'netstat' command. The expected connections from a file server were present but also a connection to a DNS server named dns.something.uk. The uk (not being good old .com) represents the United Kingdom. DNS is a service best utilized only a few network hops away (atlanta... not the uk). I had not been surfing the internet and my bind DNS server is using forwarders instead of recursion. There is no reason my network should contact any DNS server in the UK. This 'z0ne' program was gathering DNS host IP addresses so that it could look for other systems to break into. Taking the linux box offline meant physically pulling the ethernet cable from the box. I had a feeling the 'cracker' was still in the system (for reasons I will explain later) and may be well ahead of me, I knew he couldn't outsmart a physical cable disconnect. Other Signs: Among the log files which are presented and explained later, as root, listing all running processes returned the following data (which has been trunicated): root@diamond[/] # ps -A 24347 ? 00:00:00 in.telnetd 24348 pts/1 00:00:00 lpr 24349 pts/1 00:00:00 bash 24374 pts/1 00:00:00 su 24375 pts/1 00:00:00 bash 24384 pts/1 00:00:00 ps As you can see, there are two bash shells running, this is because I telnet in as a user which is a member of the WHEEL group which can then su root. You can't login directly as root from a telnet session! There are two things of interest here but one is very obvious. Why is lpr running between in.telnetd and the initial bash shell? Telnet calls login, if the login program can successfully authenticate you, it forks the shell. Why is telnet calling lpr and lpr fork()ing the shell? First of all, lpr is a printing command, I don't have a printer on this PC and have explicitly turned off the lpd printing daemon to save resources. I never use the lpr (print) command so why is it running... And spawned by my pts/1 terminal... The reason is in the source code of the 'term' program which the cracker used to trojan the login program (explained later). The other thing that caught my attention is that I was on the pts/1 terminal. I checked with a 'w' command and I, as expected, was the only user logged in (or so I thought). As the only user, I should be in the pts/0 slot, the first available. As we can see, the in.telnetd was spawned on the pts/1 slot, why wasn't pts/0 available? It turns out, it is because the cracker was in the system and he was on pts/0, the ps command was hiding the processes spawned from pts/0. I imagine the cracker was amused watching me fumble around trying to dig up some answers (I would have been). If he was more than a 'script-kiddie' and knew what he was doing, he may have snooped pts/1, my terminal, watching every typo I made! The Research: Next, find this 'z0ne' program: root@diamond[/] # find / -name *z0ne* /var/lib/pgsql/wuftpd/z0ne Ok, z0ne is in the /var/lib/pgsql/wuftpd directory. Lets go there and see what all is in this directory. root@diamond[/] # ls -al /var/lib/pgsql drwxr-xr-x 3 866 401 1024 Mar 9 13:05 wuftpd/ -rw-r--r-- 1 postgres postgres 101876 Mar 9 02:12 wuftpd.tgz Ok, the folder has a tgz archive and a folder of the same name, obviously extracted with tar and gzip. As we can see, the tgz is owned by the user "postgres", the reason will surface later... root@diamond[/] # ls -al /var/lib/pgsql/wuftpd -rwxr-xr-x 1 866 401 108670 Feb 26 2001 7350wu* -rw-r--r-- 1 866 401 466 Feb 26 2001 FAQ -rw-r--r-- 1 866 401 7219 Feb 12 09:59 README -rw-r--r-- 1 866 401 41 Feb 12 10:00 VERSION -rwxr-xr-x 1 866 401 16628 Feb 9 12:10 eraser* -rwxr-xr-x 1 866 401 675 Feb 12 09:23 masswu* -rwxr-xr-x 1 866 401 16906 Feb 26 2001 scan* -rwxr-xr-x 1 866 401 1978 Feb 16 11:54 term* -rw-r--r-- 1 root root 16173 Mar 9 15:04 uk drwxr-xr-x 2 866 401 1024 May 29 2001 wuftp/ -rwxr-xr-x 1 866 401 11975 Feb 26 2001 z0ne* Here is the crackers hideout... Look at all those goodies! Note: the file owned by root. This file must have been created by a seperate process than the one which created the files owned by 866 belonging to group 401. 866 and 401 are the uids of the creater of this exploit package, since my linux box has no corresponding friendly name to these uids, they are displayed as their number value instead of their friendly name such as "coolkid21". This is because it was extracted from the tar archive. The file "uk" is a database from either the wscan or z0ne bin (which must have been run under superuser priveledge to be owned by root). I began to read the README you see in the folder listing, it was the full explanation of the exploit kit and what it does. This kit was an exploit of my FTP service (the Washington University FTP daemon) wuftpd (wu-2.6.0(1)). When you log into my FTP server, it prompts with the following: C:\WINDOWS>ftp diamond Connected to diamond. 220 diamond FTP server (Version wu-2.6.0(1) Mon ~~) ready. login> I really like that wuftp feature which broadcasts the daemon version. How much easier could it make things for a cracker to id my FTP daemon and it's vulnerabilities. I can't see (off hand, and for general recreational ftp purposes) why the version number is given even before the user logs in. I purposely disable sendmail from broadcasting all she knows. I will have to see if I can turn this off on ftpd as well. I read the source code in the 'term' script and see that it moved the 'login' program to 'lpr', that is why I was getting the lpr program in the listing, it was really the 'login' program as you would expect to see in the process vector. What was being called in place of the 'login' program was the trojan, if the cracker set his terminal to '0371' (TERM=0371) and then telnet my linux box, it immediately droped to shell with root prompt. If any a normal TERMinal emulation was set, the trojan exec()ed the real 'login' program running as 'lpr'. The trojan was not advanced enough to do anything more than drop to a shell, it could not perform authentication so it had to pass it to the system login running as lpr. That was a weakness which helped identify something odd on the system. (If you know your processes...) The Hard Evidence: Besides that nice little tool kit @/var/lib/pgsql/wuftpd, complete with a README... If I was going to use this little package, I would at least take the README files out to make it more stealth (it would also be more professional). The kit could also be re-archived to uid 0/root so that it blends in better when uncompressed. The log files...
The Analysis: Pre-attack Connections: Mar 4 08:48:36 in.ftpd: connect from 216.140.202.200 Mar 4 08:49:58 in.ftpd: connect from 216.140.202.200 Mar 4 08:56:19 in.ftpd: connect from 216.140.202.200 Mar 5 19:46:54 in.ftpd: connect from 198.59.140.200 Mar 5 19:48:11 in.ftpd: connect from 198.59.140.200 Mar 5 19:51:49 in.ftpd: connect from 198.59.140.200 Mar 6 03:11:50 in.ftpd: connect from 209.82.62.47 Mar 6 13:34:44 in.ftpd: connect from 151.26.55.112 Mar 6 15:36:39 in.ftpd: connect from 200.204.154.232 Mar 7 18:45:10 in.ftpd: connect from 195.58.186.187 Mar 8 14:12:57 in.ftpd: connect from 202.194.220.11 ftp ftpd21984 cof.sdust.edu.cn Fri Mar 8 14:13 still logged in Issuing the command (from his box or a compromised box): [root@securify wuftpd]# ./7350wu -r -h my.domain.com He is now in my system at a root prompt, this command exploited my wuftp daemon and made it fault to a root shell. This is what is know as a buffer overrun, probably b/c this version of ftpd uses the antiquated fgets(), for username and/or password string buffering. It can be broke and will just stream all input right into its buffer space (and then some) in RAM (where computer instructions are held). There is a pretty good chance that when wuftpd is put into memory and ran the instruction set to test a username and password is the next address in RAM (after the password buffer which was probably overran by good old linux ELF assembly which is the equivalent to a COM program in DOS, which is a file that is loaded directly into RAM and already has an runtime architecture, no EXE type interpretation). Anyway, enough on how easy it is to exploit a buffer overun, if you know two things: assembly and the characteristics of a service (RAM / CORE dump), you can exploit a service that uses weak malloc() techniques. Next log entry shows him deciding which username he will hijack in the process of installing a trojan. It looks as though he is going to go with the username 'postgres'. The next log entry shows him changing the password for user 'postgres'. This signals the initial backdoor compromise. The ftp daemon is attacked and left the cracker with a root shell (on my box). This is done from a telnet session on another unix based box with the '7350wu' utility (from the kit he used), as shown above. Mar 8 15:44:18 PAM_pwdb: password for (postgres/26) changed by ((null)/0) Immediately following this password change, a telnet connection. He has changed the password from the cripled ftp daemon and wants to login with the new password (getting out of the ripped ftpd, it is an ugly scene with lots of unreadable characters, but it takes simple commands such as changing passwords!): Mar 8 15:44:27 in.telnetd: connect from 203.130.250.18 Now a secure shell (ssh) connection from the same address (no success): I'm not sure why he wants to bother with a secure connection to my box? Mar 8 15:44:36 sshd2: connection from "203.130.250.18" Mar 8 15:44:37 sshd2: DNS lookup failed for "203.130.250.18". Mar 8 15:44:38 sshd2: FATAL ERROR: Executing ssh1 in compat mode. Now he is logging in from 167.205.26.28 (gc.tf.itb.ac.id) on the ftp service, he mistypes his password the first time but gets it correct the second time. It also looks as though he is getting superuser access (the uid=0) as the 'postgres' user.: Mar 8 15:45:34 in.ftpd[22074]: connect from 167.205.26.28 Mar 8 15:46:59 PAM_pwdb: authentication failure; (uid=0) -> postgres for ftp service Mar 8 15:46:59 PAM_pwdb: failed login from gc.tf.itb.ac.id [167.205.26.28] Mar 8 15:46:59 PAM_pwdb: FTP LOGIN FROM gc.tf.itb.ac.id [167.205.26.28], postgres postgres ftpd22074 gc.tf.itb.ac.id Fri Mar 8 15:47 - 15:48 (00:01) Mar 8 15:47:00 diamond ftpd: gc.tf.itb.ac.id: connected: IDLE Now that he has successfully connected and can upload files to the linux box, he uploads a file named 'term' (/var/lib/pgsql/term) which is the script which he will use to create the login trojan (backdoor access). Mar 8 15:47:12 ftpd: gc.tf.itb.ac.id: postgres Mar 8 15:47:52 15 gc.tf.itb.ac.id /var/lib/pgsql/term a _ i r postgres ftp 0 * c Mar 8 15:48:20 ftpd: gc.tf.itb.ac.id: postgres: QUIT He logs out of FTP... Here, email is sent to the creator of the exploit package (bigm@st3r.biz). The 'term' script (trojan creator) emails this person the password used in the trojan and the interface configuration of the machine (so he knows the address of the expolited machine). The user must be accessing the machine via telnet with the 'postgres' account (running root), since he was able to login as 'postgres' on the ftp service. He is able to execute the 'term' script (uploaded with ftp), via a telnet session authenticated under 'postgres', and create the trojan. Since the trojan sends email to the package creator (bigm@st3r.biz) when finished installing, it follows that the machine was completely compromised on March 8, 2002 @ 15:48. Mar 8 15:48:49 sendmail: to=bigm@st3r.biz The trojan is called by the system thinking it is the real 'login' program. The trojan is setup to give immediate root shell access if the TERMinal is set to '0371'. Since there is no such thing as a TERMinal of type '0371', it will never be discovered by a regular user using ANSI or VT100 TERMinal emulation. Here is what the cracker is now doing on his box. First setting the TERM to the non-exiting type of '0371', and then connecting to my computer. The trojaned login program sees that he is requesting the '0371' TERMinal emulation mode and knows to drop to root shell. With any other emulation type, such as the ones used by valid users, it execl() the real 'login' program renamed to 'lpr': [root@securify wuftpd]# export TERM=0371 [root@securify wuftpd]# telnet my.domain.com He now reconnects to the machine, using the trojaned backdoor. That is apparent because the plugable authentication module (PAM_pwdb) makes no log entries for these sessions. This is because the trojan never called the real 'login' program, it dropped to root shell. As you can see, he logs in once, and then again. He most likely created two simultaneous sessions, one to do his thing, one to watch for admins (or.. me). Mar 8 15:49:14 in.telnetd: connect from 203.130.250.18 Mar 8 16:08:54 in.telnetd: connect from 203.130.250.18 Some time passes... Now the cracker changes the password for the postgres user account. He changes it twice. Notice the log entries show the passwords were changed by user 'null' w/uid 0. He has a null username because he never authenticated with the real 'login' program. This shows he was using the backdoor and not the compromised postgres account. Mar 9 02:06:08 in.telnetd: connect from 203.130.254.26 Mar 9 02:08:01 PAM_pwdb: password for (postgres/26) changed by ((null)/0) Mar 9 02:08:19 PAM_pwdb: password for (postgres/26) changed by ((null)/0) As a note, here I am (spc) logging on... Thinking I am a security expert... I am actively in the system as it is being compromised. I had no idea... Mar 9 02:09:00 PAM_pwdb[22610]: (login) session opened for user spc by (uid=0) spc pts/0 sapphire Sat Mar 9 02:09 - 02:23 (00:14) He now changes the 'adm' user account's password. This is a user account that is not active on most linux boxes. I am not sure why he is activating the account (since he has backdoor access). Mar 9 02:09:14 PAM_pwdb: password for (adm/3) changed by ((null)/0) Here I am going to superuser mode while he is changing all my passwords: Mar 9 02:09:23 PAM_pwdb: (su) session opened for user root by spc(uid=500) Now he logs into the FTP service to upload the complete 'crackers' kit. The file is wuftpd.tgz which contains all of the utilities that will turn my linux box into a zombie. This kit scans internet addresses looking for unix machines that have the same problem this linux box had (an out of date ftp daemon with a known security hole). Mar 9 02:11:33 ftpd: FTP LOGIN FROM 61-222-173-226.HINET-IP.hinet.net, postgres postgres ftpd22660 61-222-173-226.H Sat Mar 9 02:11 - 02:12 (00:00) Mar 9 02:12:00 61-222-173-226.HINET-IP.hinet.net /var/lib/pgsql/wuftpd.tgz b _ i r postgres ftp 0 * c Mar 9 02:12:05 ftpd: FTP session closed At this point, the cracker is unpacking the kit and getting all setup. It wasn't long after this when I noticed the first sign of something being wrong, a slow internet connection dropping 80% of my ping packets. I look around and find the 'wscan' program running (as explained) and kill the process. It was late and I didn't want to accept that something was going on. So, I log off to go to bed: Mar 9 03:33:33 diamond PAM_pwdb[22900]: (su) session closed for user root The next morning... After only six to seven hours of sleep, I woke up knowing I had some research to do. Knowing things would make more sense to me than last night @3:30. I logged back in... Mar 9 11:39:34 PAM_pwdb[23500]: (su) session opened for user root by spc(uid=500) Mar 9 12:02:17 PAM_pwdb[23500]: (su) session closed for user root spc pts/1 sapphire Sat Mar 9 15:01 - 15:23 (00:21) This is when I see the 'z0ne' program running and know there is a big problem. I isolated the linux box from the internet connection and then began this research. In Conclusion: This attack was from a 'script-kiddie' who used a documented problem and a pre-made (by someone else) 'kit' to exploit the system. He was most likely trying to "prepare" for a Distributed Denial of Service attack which could be mounted in the event someone in a discussion group says something derogatory about South Park. At some point, all the computer he scaned and broke into with high-speed connections, would send large data packets to a specific computer, thus flooding it's network connection. Once the network connection is saturated, the cracker has successfully denied the target host network service, thus the term Denial of Service (DoS). Multiple hosts attacking this on internet host makes it distributed (Distributed Denial of Service or DDoS).