Sunday, April 20, 2014

Palmetto Cyber Defense Competition - 2014

Last weekend (April 12-13) was the second annual Palmetto Cyber Defense Competition (PCDC). Inspired by the national Collegiate Cyber Defense Competition (, PCDC is a computer security and business competition open to schools in South Carolina. On Saturday, college teams compete while Sunday is reserved for high schools. Unlike CCDC competitions which usually span a couple of days, PCDC competitors are pressured into one fast and furious day of competition. I was fortunate enough to be a part of the Red team this year. I wanted to make this blog post to supplement the short Red team presentation given to the students at the end of their competition day. Since PCDC competition sessions are only a day long, there is so much to do in that timeframe. As a result, the Red team doesn't really get to talk with the students as much as we would like.

For this year's PCDC, we had a relatively small Red team. At any given time, we were between 7-8 members. As a result, we decided it would be most effective if we each took a team to focus on. If anyone of us would find a way into a system, we'd shout it out and each Red team member would try that method on their target Blue team. We wanted to keep things as even as possible. In addition to all of our technical tools, the Red team was allowed to use social engineering this year. The Red team had access to a VOIP phone which we could use to call the Blue teams. In addition, the Blue teams' competition area was in a large reception style room. This meant that the Red team could walk around the Blue teams to try and pick up what was going on.

From a social engineering perspective, the Red team kept things pretty simple. We stole a couple of Blue team 'mentor' name badges that would have been used to help identify individuals who could provide the Blue teams with some extra technical support. One of the defining differences with PCDC is its goal in encouraging STEM outreach. As a result, Blue team mentors work with the competing high school teams months in advance to help prepare them for the competition. As a result, the Blue teams grow to trust the mentors. Although I personally offered my Linux subject matter expertise to the Blue teams, I was not asked to help. I felt completely under-utilized. This is a positive note to point out. All the Blue teams for both college and high school were extremely cautious with allowing personal access to their space. As a result, very few of the Red team members got information via personal interaction. But we had other plans.

One of the Red team members brought a high quality camera with him. Equipped with nothing more than a collared shirt and good personality, he set out to film the Blue teams as a member of the 'local news station'. He was successful in taking a picture of every single Blue team's password lists for both college and high school days. Only one team reported his actions. As you can see by the following picture, the results were awesome:

Blue Team Passwords

In addition to pictures, we also tried to add our own business inject. Prior to the competition, one of our Red team members made a fake tax calculator program and accompanying inject form. The inject stated that the students were to report the error code given by the tax software as the upgrade from Windows XP to Windows 7 has caused some unanticipated errors. What really happened is the program would crash every single time but open up a reverse shell to our waiting machines. We had some success with this approach including one team that send us over 30 shells back! Other notable events include a team that redialed their phone immediately following a Blue team runner who called the Red team asking to clarify if the weird behavior on the team's system was the Red team or an actual technical issues. The Blue team runner was not a competitor, but one of the helpers there for the Blue teams. Because we are a friendly Red team and we want to help everyone out, we would always pick up our phone pretending to be the Gold team. As a result, this particular Blue team thought we were the Gold team for most of the day.

The Red team's game plan for attacking the Blue teams was simple. Get access, establish persistence, then have fun. This is true with pretty much every Red team for all the CCDC style events. I will say that for both college and high school days, every team was pretty good at keeping us out by changing their default passwords. The only way we got their default passwords was with our photographer. What the Blue teams weren't told was that the default passwords given to them were not all their default passwords. One of the services the Blue teams had to keep up was a Kunagi server. Kunagi is deployed on a Java application server like Tomcat or JBoss. For this event, the Blue teams were running JBoss. What every single one of them failed to notice, however, was that their JBoss admin console was publicly available and still had default admin credentials. This is how we got our initial access for every team.

Through the admin console we were able to upload our own WAR file (the deployable version of the Java applications that run on JBoss) which would start a reverse meterpreter session back to our machines. With this we would establish simple persistence by using meterpreter's run persistence module. From here, I no longer needed my custom deployed WAR, so I deleted is. This would hopefully hide the fact that I had compromised their system. With this initial access, the next step is to try and move laterally through the network. It turns out, these system were all part of a domain. So, our next step was to pull credentials from the machine. We did this with mimikatz. For those of you that don't know what mimikatz is, it is a tool that allows for plaintext recovery of cached Windows credentials. The following is a screenshot of what mimikatz output looks like:

Plaintext credentials with mimikatz

Now, lucky for the Red team (and consequently, unlucky for the Blue teams), these credentials belong to the domain administrator. From here, we were able to use the psexec command to log into the other Windows boxes and consequently establish persistence on all the Windows machines.

In addition to Windows, the Blue teams also had a couple of Linux servers and an ESXi server. As a team, we really didn't have too much luck on getting on the Linux servers except with a default SSH credential and through the ESXi console. But let's talk about how we got to the ESXi servers. For those of you who have used ESXi and connected through a browser, you know that it uses HTTPS. It turns out, ESXi is using OpenSSL. Hmmm. PCDC 2014 was on the weekend of April 12th and 13th. Not even a week before, the security community was thrown on its head with the HeartBleed vulnerability present in most modern versions of OpenSSL. On a hunch, we decided to try HeartBleed on the ESXi servers. We set up a simple script that would constantly ping the Blue team's ESXi servers and look for a HeartBleed response that contained a password field. About an hour later, one of our Red team members jumps up yelling, "it worked!" And there on his terminal, was a username and password field. It turns out that HeartBleed will work on an ESXi server and spit out usernames and passwords after a login attempt. Here is a screenshot of was our output looked like:

HeartBleed works on standard ESXi 5.5

One of the goals of PCDC is to give students a taste of the real world. How much better could be get than launching an attack against a critical service that exploits a vulnerability less than a week old? Needless to say this certainly caught some buzz with the spectators. Unfortunately, we did not discover the effectiveness of HeartBleed until the second day. As a result, the college students missed out on this. With these credentials we could point vSphere ESXi clients at the ESXi servers and help the Blue team's administer their VMs. My favorite is when I open up the console to a Linux machine and just see a # sitting there. For my Blue team, I was a little quicker on the keyboard than they were. I opened the console and saw the root prompt. I quickly changed the root password, turned on SSH, and then quit the user session through the vSphere client. This effectively locks the Blue team out of their root account and allows me to login remotely via SSH. My favorite result of this was having the Gold team call up the Red team with the response, "One of the Blue teams just said they saw their root password change right before their eyes and then lock them out. Was that you?"

At this point, it was time to start having fun with the students. A couple of the Red team member came very prepared for fun. In the year leading up to PCDC 2014, we have been hearing a lot about ransom-ware in the news. Software that gets on your computer, encrypts important files, and then demands a ransom for the decryption key. Inspired by this, one Red team member made some hilarity-ware. This combined the ransom-ware with some amusing side effects. Let me introduce Ponyware:

Ponies demand a ransom

Ponyware would encrypt a lot of the folders found on the computer and then insert a small ransom program in the front of every legitimate program it could find. The goal was to get the Blue teams to call the Red team for the decryption key after they paid a ransom with some of their points they had earned throughout the day. We successfully deployed Ponyware, but no Blue team called us for the decryption key. Instead, most of them either opted to leave it on and have it play its music, leave the desktop background a checkered board of ponies, and enjoy the custom mouse cursor. Others opted to take a large point deduction and request a Gold team re-image. Ironically, the re-image cost more points then the ransom would have.

Our next trick was pretty nasty. Originally, we were planning to deploy this uniformly at 3:30pm, but our plans didn't really work out. The plan was to overwrite all the master boot records (MBRs) on as many Blue team machines as possible. What we wrote the the MBRs was an awesome 512 byte program that displayed an animation on Nyan cat.

Nyan cat animation written to boot sector

Once the MBR was overwritten, we would bounce the boxes and this would be the only thing the Blue teams would see. Nyan cat was a huge hit with just about everyone. Although the Blue teams would have preferred not having their MBRs overwritten, they appreciated the humor. By the end of the day, a couple of computers were just sitting there completely useless counting down till 3:30pm:

Waiting until 3:30pm

From a Red team member's perspective, there are a couple of things I would have suggested to the Blue teams. In general, most teams did a very good job, but no matter how many times we say it, default passwords are going to kill you. As a Blue team member, you need to be aware of every service you have running and if/what those service's default passwords are. You must be aware of service that come out-of-the-box with default credentials. Now, I have only been a Red teamer on 2 CCDC style events, but both times I've bean able to access the ESXi servers with my vSphere client. In the case of PCDC, every Blue team was provided an adaptive security appliance (ASA). This can be used to restrict the IP addresses allowed to access the ESXi servers. Blue team's should also be aware of authentication mechanisms that are required and not required. The Blue team I focused on for day 2 only allowed SSH login via key authentication. This is a great move, except when the Gold team needs to be able to login via SSH to score your service and they don't have an authorized key because Gold team is using a password. Over and over again the Gold team remarked about how the Blue teams did not read their information packets thoroughly. The information packets usually have detailed information about your network and systems. As a result, you should know what is on your network. It shouldn't be towards the end of the day when I see this as I'm watching your session..

What is an ESXi?

At almost 1:30pm, the competition has been going on for about 5 hours. Just now trying to learn what an ESXi server is might mean you're a little behind the game. Another lesson learned (the hard way) by some Blue teams is if you know something bad is going to happen at a specific time (say 3:30pm), do not try and adjust your system clock past that time. Trying this trick quite literally blew up in some teams' faces.

I just want to note that some individuals on the Blue teams felt that we were a little hard on them. It was our intention as the Red team to not be complete jerks. Our goal is to educate the students while having fun with them. As we tried to tell the Blue teams, we do this because we care about education. So, if you're a current or future member of a Blue team, know that we don't do what we do to be mean. We do it to help you. If you're competing and the frustration is building, just remember, Red team loves you!
Red Team <3's you!

Thursday, April 3, 2014

Red Team SE-CCDC 2014

Earlier this week I had the fantastic opportunity to participate in the South East regional Collegiate Cyber Defense Competition (CCDC). If you've never heard of CCDC before, here's the short and sweet. Teams of college students represent their schools by assuming the role of a Blue team which represents a business's IT branch. They are dropped into a preconfigured network after the previous IT employees were let go. They have no knowledge of what the systems, configurations, software, or even hardware look like before they get there. The Blue teams' goal is to ensure the daily operational aspects of the business remain up and operational. Here's the catch. The entire time they are trying to ensure their business stays online and available, the Red team (a team of security professionals playing the part of hackers) actively attack the Blue team's systems and try to take them offline. Not only must the Blue teams defend their networks, they must also complete a series of timed tasks administered by the judges (White team). The White teams gives the Blue teams business injects like opening up file sharing for the organization's employees or responding to potential incident response requests. Needless to say, the competition is a fast and furious exercise in information security and I was excited. Not only was this my first time as a CCDC Red team member, it was my first time at a CCDC period.

On Monday, a coworker and I drove out to this years location at Kennesaw State University. After the long drive, we setup in the Red team room and began scanning and enumeration. The rules for the first day were simple. We could scan the Blue teams all we wanted, but we were not allowed to perform any attacks. Our scans showed that there were a lot of systems and most of them looked pretty up to date. Every team had Windows 7 desktop boxes, a couple of Fedora 20  and CentOS Linux boxes, a Windows Server 2012 R2, and a couple of ESXi servers. By the end of the night, our best lead at an attack vector was an eCommerce site. I found a small SQL injection vulnerability in the search functionality and a couple other Red teamers found what looked to be a neat way of overwriting configurations by leveraging a misconfigured install directory. While we did our scanning, the Blue teams were hard at work configuring, administering, and hardening their infrastructure. To make things a little more interesting, the Red team worked with the competition's organizers (Gold team) to tip the scales a little bit more in our favor :).

Raphael Mudge (the creator of Armitage and Cobalt Strike - worked with the Gold team to drop 2 of his Beacon backdoors on every team's network. This was used to simulate the insider threat left behind by the old IT employees that use to operate the networks that each Blue team inherited. Beacons act as an advanced persistence agent capable of 'beaconing' back on a timed interval. Each time Beacon checks back, it will query the Red team's attack server looking for a queue of commands to act on. These Beacons are heavily embedded on the Blue team's systems and have multiple fall back communication channels. In my opinion, it is one of Cobalt Strike's best features.

Sure enough, the next morning as soon  as the competition started, the Beacons started calling back. The Red team decided that the best way to play the game was to split up into pairs and every pair would focus on a team. Every time a pair found something with one team, we would yell it out the rest of the Red team to try on their target Blue team. I got to pair up with Mudge and get a first hand education on Cobalt Strike and collaborative Red teaming. Mudge and I had Team 5 (team Echo) who we affectionally referred to as our children for the next 2 days. We decided early on, like good parents, we would watch over our children very carefully. As soon as we got our first Beacon, we queued up the 'keylogger' command. Now, with every checkin, Beacon would dump a series of cached keystrokes. Almost immediately we got a sequence that looked like 'root [TAB] @D4#c.L8'. SWEET! Root creds! But to what?! I tried a couple of the Linux boxes but it didn't work. So I saved them for a little bit and moved on. A couple minutes later, another Red team member shared that they pulled the competition rules off another Blue team's box. The rules detailed all the default user names and passwords for the Blue team's systems. Good news for us! And it was still early in the day.

Using a nifty feature of Cobalt Strike, we attempted a mass login with default credentials. SUCCESS! We got a couple of successful logins including a Linux box. Mudge went to work securing our foothold on the Linux system. Thankfully for us, this was made easier by the default user being in the '/etc/sudoers' file. So a quick 'sudo /bin/sh' and we were good to go! A little later I remembered that there were ESXi servers on the Blue team's network so I popped open my vSphere client and pointed it at the ESXi server's IP address. Hmmm, usually the user name is root. Lets give our first sniffed password a try...SUCCESS again! A couple minutes later I logged into the second ESXi server with the default password 'changeme'! It was less than an hour into the competition on Day 2 and we had root on 2 ESXi servers, a Linux box, and 2 Windows boxes hooked. It looked like it was going to be a good day.

Root on both ESXi servers

To maintain my foothold on the servers, I added a new user (vm-admin) with full administrative privileges. Then I sat and watched. Slowly it looked like the Blue team was moving more of their assets to the ESXi servers. How nice of them! I decided that I wouldn't make myself known till the very end, after they had given me all of their VMs. This was a little risky. If they would find my user account, I could be booted from the server. Luckily for me, they didn't know I was there until I wanted them to. Between getting root and the end of the day though, we managed another big win for the Red team.

One of Metasploit's features in meterpreter is the ability to grab screen shots. Using Cobalt Strike, we configured the screen shot functionality to snag a capture every 10 seconds. This was a slow, sneaky way of spying on team Echo. Eventually, we saw a member of Echo open up Internet Explorer to administer a spiffy new pfSense box through the browser administration client. Mudge lit up when he saw this. It was time to test browser pivot in a real scenario! Cobalt Strike's browser pivot essentially allows you to spawn a process inside of a current Internet Explorer browser session and then proxy the session back to the Red team's local browser all unknowingly to the victim. This means that the browser has already authenticated and we can piggy back off of that. So now we can help administer the pfSense box for team Echo :).

Helping team 'Echo' administer their pfSense box via a browser pivot

With this capability we quickly configure pfSense to allow SSH and best yet, we could inject commands via the pfSense web admin console's 'execute' feature. So we added our SSH key for easy access. Later on, we would keylog the root password to the pfSense box as well. It was ours.

After all of this the end of the day was approaching quickly. At 6:00pm, the Blue teams must be hands off keyboards. Team Echo had a pretty solid run all through the day by keeping most of their services up and running. But, at 5:55pm I decided to change that. The first thing I did was change the root passwords on both ESXi servers. This is not enough to boot the root user off if they are logged in, however. So I saved the state of all the VMs and quickly bounced the physical servers. Team Echo would be kicked off and after the reboot, their passwords wouldn't be correct. It worked. To put the servers further out of reach I changed the passwords for all the users (1 server had 3 default users and the other had 4) and made sure SSH was disabled. The next morning they would have physical access to the box, but no way in. Now, I saved the state of the VMs for a reason. If I would have bounced the ESXi server without saving VM state, then the VMs would have to reboot and more than likely, require a valid login. But, if I saved their state and they were already logged in, then I wouldn't need to authenticate at all. Lucky for me, this was absolutely the case for a Linux and Windows box.

The next morning the competition's director came into the Red team room and asked which one of us owned team Echo's ESXi servers. Mudge and I gladly took credit. He asked us to reset the root passwords to both ESXi servers and give it back to team Echo. Since all of their services were on the servers, they literally couldn't do anything. Mudge and I agreed on one condition. We could backdoor the VMs we had access to. So after backdooring the Windows and Linux box and deleting team Echo's VM snapshots and replacing them with our own, we handed team Echo their new root passwords. Since today was a much shorter day, Mudge and I focused on trying to figure out why our Beacons stopped calling out to us. After a couple of hours we determined that team Echo had found the Beacons and removed them. Good for them, bad for us. About half an hour before the very end of the competition I attempted to login to the ESXi servers with my old account I created when I first got on the box. I only gave team Echo the root account passwords, nothing else. To my surprise, I still had complete admin access with my vm-admin account!

Since it was the end of the day, it was time to go all it. I took all permissions away from root, deleted the other users, deleted all the VMs and datastores, and dropped the servers into maintenance mode. Literally game over.

Here's the recap. From all that I just wrote, it would seem like team Echo might have been off their game. This is simply not true. I only described our successes. Not failures. Team Echo seemed to boot us off the Linux boxes every time we gained access. Every. Time. Our Beacons stopped calling back on both Windows boxes. Really the only things we had were the ESXi servers and the pfSense box. But guess what. We didn't even truly have the pfSense box. Team Echo knew we were there and they weren't even using it! Now, Mudge and I noticed a lack of activity with the firewall, but overall, a great move by team Echo. In the end, it was a couple of critical mistakes that cost team Echo.

Team Echo's first mistake was not changing the default password on the second ESXi server. Next, there was a lack of user auditing on the ESXi servers and the principle of least privilege was not enforced. All administrative/root accounts on a system should be documented and accounted for. If a new one shows up, say vm-admin, action needs to be taken. Team Echo still had root, so they should have immediately changed their password and deleted my account. By doing that, they would have effectively locked me out of the system. Finally, in a situation like CCDC, the Blue teams can't always prepare for curve balls like judges allowing Mudge to install Beacons on their systems, but they can protect against other things. An ESXi server should never be available outside of a local management network. This is were the pfSense box should have been stood up. A simple firewall rule to deny access from any IP address (except ScoreBot which is off limits to the Red team) would have prevented me from accessing the ESXi servers with my vSphere client even if I had the root password. And finally, never, ever leave default usernames and passwords on your system. The Red team WILL find them and they will take advantage of them.

All in all I had a lot of fun and learned a lot. Mudge was awesome to work with and I appreciate the time he took to help show me the ropes. In the next week or so I'll be ramping up for a local CCDC called Palmetto Cyber Defense Competition (PCDC). Hopefully I'll have more to share (I'll try to do more screen shots and less text) from that event shortly after.