Red Team SE-CCDC 2014

Earlier this week I had the fantastic opportunity to participate in the South East regional Collegiate Cyber Defense Competition (CCDC). If you've never heard of CCDC before, here's the short and sweet. Teams of college students represent their schools by assuming the role of a Blue team which represents a business's IT branch. They are dropped into a preconfigured network after the previous IT employees were let go. They have no knowledge of what the systems, configurations, software, or even hardware look like before they get there. The Blue teams' goal is to ensure the daily operational aspects of the business remain up and operational. Here's the catch. The entire time they are trying to ensure their business stays online and available, the Red team (a team of security professionals playing the part of hackers) actively attack the Blue team's systems and try to take them offline. Not only must the Blue teams defend their networks, they must also complete a series of timed tasks administered by the judges (White team). The White teams gives the Blue teams business injects like opening up file sharing for the organization's employees or responding to potential incident response requests. Needless to say, the competition is a fast and furious exercise in information security and I was excited. Not only was this my first time as a CCDC Red team member, it was my first time at a CCDC period.

On Monday, a coworker and I drove out to this years location at Kennesaw State University. After the long drive, we setup in the Red team room and began scanning and enumeration. The rules for the first day were simple. We could scan the Blue teams all we wanted, but we were not allowed to perform any attacks. Our scans showed that there were a lot of systems and most of them looked pretty up to date. Every team had Windows 7 desktop boxes, a couple of Fedora 20  and CentOS Linux boxes, a Windows Server 2012 R2, and a couple of ESXi servers. By the end of the night, our best lead at an attack vector was an eCommerce site. I found a small SQL injection vulnerability in the search functionality and a couple other Red teamers found what looked to be a neat way of overwriting configurations by leveraging a misconfigured install directory. While we did our scanning, the Blue teams were hard at work configuring, administering, and hardening their infrastructure. To make things a little more interesting, the Red team worked with the competition's organizers (Gold team) to tip the scales a little bit more in our favor :).

Raphael Mudge (the creator of Armitage and Cobalt Strike - http://blog.strategiccyber.com/) worked with the Gold team to drop 2 of his Beacon backdoors on every team's network. This was used to simulate the insider threat left behind by the old IT employees that use to operate the networks that each Blue team inherited. Beacons act as an advanced persistence agent capable of 'beaconing' back on a timed interval. Each time Beacon checks back, it will query the Red team's attack server looking for a queue of commands to act on. These Beacons are heavily embedded on the Blue team's systems and have multiple fall back communication channels. In my opinion, it is one of Cobalt Strike's best features.

Sure enough, the next morning as soon  as the competition started, the Beacons started calling back. The Red team decided that the best way to play the game was to split up into pairs and every pair would focus on a team. Every time a pair found something with one team, we would yell it out the rest of the Red team to try on their target Blue team. I got to pair up with Mudge and get a first hand education on Cobalt Strike and collaborative Red teaming. Mudge and I had Team 5 (team Echo) who we affectionally referred to as our children for the next 2 days. We decided early on, like good parents, we would watch over our children very carefully. As soon as we got our first Beacon, we queued up the 'keylogger' command. Now, with every checkin, Beacon would dump a series of cached keystrokes. Almost immediately we got a sequence that looked like 'root [TAB] @D4#c.L8'. SWEET! Root creds! But to what?! I tried a couple of the Linux boxes but it didn't work. So I saved them for a little bit and moved on. A couple minutes later, another Red team member shared that they pulled the competition rules off another Blue team's box. The rules detailed all the default user names and passwords for the Blue team's systems. Good news for us! And it was still early in the day.

Using a nifty feature of Cobalt Strike, we attempted a mass login with default credentials. SUCCESS! We got a couple of successful logins including a Linux box. Mudge went to work securing our foothold on the Linux system. Thankfully for us, this was made easier by the default user being in the '/etc/sudoers' file. So a quick 'sudo /bin/sh' and we were good to go! A little later I remembered that there were ESXi servers on the Blue team's network so I popped open my vSphere client and pointed it at the ESXi server's IP address. Hmmm, usually the user name is root. Lets give our first sniffed password a try...SUCCESS again! A couple minutes later I logged into the second ESXi server with the default password 'changeme'! It was less than an hour into the competition on Day 2 and we had root on 2 ESXi servers, a Linux box, and 2 Windows boxes hooked. It looked like it was going to be a good day.

Root on both ESXi servers

To maintain my foothold on the servers, I added a new user (vm-admin) with full administrative privileges. Then I sat and watched. Slowly it looked like the Blue team was moving more of their assets to the ESXi servers. How nice of them! I decided that I wouldn't make myself known till the very end, after they had given me all of their VMs. This was a little risky. If they would find my user account, I could be booted from the server. Luckily for me, they didn't know I was there until I wanted them to. Between getting root and the end of the day though, we managed another big win for the Red team.

One of Metasploit's features in meterpreter is the ability to grab screen shots. Using Cobalt Strike, we configured the screen shot functionality to snag a capture every 10 seconds. This was a slow, sneaky way of spying on team Echo. Eventually, we saw a member of Echo open up Internet Explorer to administer a spiffy new pfSense box through the browser administration client. Mudge lit up when he saw this. It was time to test browser pivot in a real scenario! Cobalt Strike's browser pivot essentially allows you to spawn a process inside of a current Internet Explorer browser session and then proxy the session back to the Red team's local browser all unknowingly to the victim. This means that the browser has already authenticated and we can piggy back off of that. So now we can help administer the pfSense box for team Echo :).

Helping team 'Echo' administer their pfSense box via a browser pivot

With this capability we quickly configure pfSense to allow SSH and best yet, we could inject commands via the pfSense web admin console's 'execute' feature. So we added our SSH key for easy access. Later on, we would keylog the root password to the pfSense box as well. It was ours.

After all of this the end of the day was approaching quickly. At 6:00pm, the Blue teams must be hands off keyboards. Team Echo had a pretty solid run all through the day by keeping most of their services up and running. But, at 5:55pm I decided to change that. The first thing I did was change the root passwords on both ESXi servers. This is not enough to boot the root user off if they are logged in, however. So I saved the state of all the VMs and quickly bounced the physical servers. Team Echo would be kicked off and after the reboot, their passwords wouldn't be correct. It worked. To put the servers further out of reach I changed the passwords for all the users (1 server had 3 default users and the other had 4) and made sure SSH was disabled. The next morning they would have physical access to the box, but no way in. Now, I saved the state of the VMs for a reason. If I would have bounced the ESXi server without saving VM state, then the VMs would have to reboot and more than likely, require a valid login. But, if I saved their state and they were already logged in, then I wouldn't need to authenticate at all. Lucky for me, this was absolutely the case for a Linux and Windows box.

The next morning the competition's director came into the Red team room and asked which one of us owned team Echo's ESXi servers. Mudge and I gladly took credit. He asked us to reset the root passwords to both ESXi servers and give it back to team Echo. Since all of their services were on the servers, they literally couldn't do anything. Mudge and I agreed on one condition. We could backdoor the VMs we had access to. So after backdooring the Windows and Linux box and deleting team Echo's VM snapshots and replacing them with our own, we handed team Echo their new root passwords. Since today was a much shorter day, Mudge and I focused on trying to figure out why our Beacons stopped calling out to us. After a couple of hours we determined that team Echo had found the Beacons and removed them. Good for them, bad for us. About half an hour before the very end of the competition I attempted to login to the ESXi servers with my old account I created when I first got on the box. I only gave team Echo the root account passwords, nothing else. To my surprise, I still had complete admin access with my vm-admin account!

Since it was the end of the day, it was time to go all it. I took all permissions away from root, deleted the other users, deleted all the VMs and datastores, and dropped the servers into maintenance mode. Literally game over.

Here's the recap. From all that I just wrote, it would seem like team Echo might have been off their game. This is simply not true. I only described our successes. Not failures. Team Echo seemed to boot us off the Linux boxes every time we gained access. Every. Time. Our Beacons stopped calling back on both Windows boxes. Really the only things we had were the ESXi servers and the pfSense box. But guess what. We didn't even truly have the pfSense box. Team Echo knew we were there and they weren't even using it! Now, Mudge and I noticed a lack of activity with the firewall, but overall, a great move by team Echo. In the end, it was a couple of critical mistakes that cost team Echo.

Team Echo's first mistake was not changing the default password on the second ESXi server. Next, there was a lack of user auditing on the ESXi servers and the principle of least privilege was not enforced. All administrative/root accounts on a system should be documented and accounted for. If a new one shows up, say vm-admin, action needs to be taken. Team Echo still had root, so they should have immediately changed their password and deleted my account. By doing that, they would have effectively locked me out of the system. Finally, in a situation like CCDC, the Blue teams can't always prepare for curve balls like judges allowing Mudge to install Beacons on their systems, but they can protect against other things. An ESXi server should never be available outside of a local management network. This is were the pfSense box should have been stood up. A simple firewall rule to deny access from any IP address (except ScoreBot which is off limits to the Red team) would have prevented me from accessing the ESXi servers with my vSphere client even if I had the root password. And finally, never, ever leave default usernames and passwords on your system. The Red team WILL find them and they will take advantage of them.

All in all I had a lot of fun and learned a lot. Mudge was awesome to work with and I appreciate the time he took to help show me the ropes. In the next week or so I'll be ramping up for a local CCDC called Palmetto Cyber Defense Competition (PCDC). Hopefully I'll have more to share (I'll try to do more screen shots and less text) from that event shortly after.