The Southeast Collegiate Cyber Defense Competition is an annual competition where eight teams from various colleges have to defend their systems from Red Team attacks while also executing on management-type business challenges.
This is my second year helping Kennesaw State University run the SECCDC in AWS. This year we not only ran the Regional competition on-site at KSU, but we also hosted 26 teams for the preliminary round. In previous years the scenario was HALCORP, a fictional company that did nothing but generate compliance paperwork. Basically, AT&T without the telephones.
As we had moved into the Cloud in 2021, we decided to modernize our scenario and not just our Tech Stack. Thus was born Fooli.
Welcome to Fooli
I wrote this as the onboarding guide for the students in the preliminary round:
We’re glad you’re here to help Fooli fulfill its mission to provide the Internet’s dankest memes. As we mentioned during our interview, Fooli has gone through some restructuring recently and you’re now here to support our regional meme factory.
It’s important to know about Fooli’s history. Fooli was founded by our great leader Balvin Gelson, after he left HALCORP. Fooli has been funded by Bessmer Venture Partners, Greycroft, and Andreessen Horowitz. It recently secured a mezzanine funding round of $160M from Bain Capital Venture, placing Fooli’s valuation at over US$8 Billion dollars.
Taking a page from Jeff Bezos’s leadership principles which pushed Amazon’s great success, Fooli is a frugal company. From our CEO on down, Fooli will not spend money it doesn’t have to. This allows us to source the finest meme images and provide superior value to our customers.
Whatever the circumstances, whatever the sacrifices, the memes must keep flowing, i.e., they have to be available at all costs and at all times.
The core of the Fooli operations was the Fooli Meme Factory - a vulnerable by design web application I’ve been thinking about as a learning tool for a few years now. SECCDC finally gave me the impetus to build it.
The fun thing about my background is that I’ve worked in very large and diverse environments. I’ve seen things your mortal minds couldn’t possibly comprehend. These made for a fertile ground upon which I could channel my frustrations with developers (past, present, and future).
Overly permissive IAM Roles, publicly writable buckets, unsanitized user inputs, passwords in UserData, SSRFs, sensitive data leftover from ancient data center migrations. All of these got to be part of the Fooli Meme Factory.
(yeah, I apparently don’t know how to sanitize my inputs)
We put all 26 teams into one AWS account for the preliminary round, separated by their own VPCs and leveraging ABAC control on their IAM User. The scenarios were still very network and system-centric. Students had to keep Red Team out of their systems while managing a Palo Alto VM Series.
For the Regionals, we made the students go cloud-native. Each team got their own account - named after a character from Money Heist: Berlin, Denver, Helsinki, Lisbon, Moscow, Nairobi, Oslo, Palermo, Professor, Rio, and Tokyo (with apologies to Augusta University for getting stuck with Moscow). Lisbon and Palermo were our Dev and QA environments, and The Professor acted as a control environment during the competition.
New things for 2022
We made a bunch of improvements to the network infrastructure used in 2021. While we leveraged VPC Peering for the preliminary round, we moved to Transit Gateways to link the accounts for the Regionals. TGW Resource sharing made it super easy to join the networks. I’ve not had many opportunities to mess around with AWS VPC networking in the last several years, so it was good to experiment with some of these features.
Having upgraded the 2021 SECCDC Account to AWS Organizations, I did everything I outlined in my AWS Organizations Checklist for 2021. I also started heavily leveraging AWS SSO. Let me say how very disappointed I am that the AWS SSO Identity provider has no API. But that’s ok, Google Cloud Identity is free, and that was my first foray into creating a Google Workspace and Cloud Identity domain.
Since we would actually be onsite at KSU for the Regionals, we also established a Site to Site VPN via a pfSense machine to the TGWs. That allowed the teams direct access to their VPCs without worrying if they bricked their firewall they’d have no access to anything behind it.
Regional Injects
As part of the National CCDC scoring rubric, 50% of the points should come from service uptime, and 50% should come from the “injects” - business tasks the students get assigned to complete throughout the competition. Since we fully introduced AWS as in-scope for the regionals, we also created several AWS specific injects such as:
- Setup IAM Access Analyzer and report on the findings
- Setup Macie to scan all Buckets. I deliberately hid PII from the Fake Name Generator in a handful of S3 buckets.
- Analyze your GuardDuty Findings
- Investigate a Data Breach with CloudTrail & Splunk
- Present your Breach Investigation findings to “The Board,” which consisted of a handful of sponsors.
An important design principle for all of this was fairness. Werner says, “Everything breaks all the time,” so we needed to ensure that any breakage would be spread evenly across all teams. It’s one of the reasons we kept a Control Team running during both events. If the Control Team started failing, we could adjust overall scoring.
What worked well
All told, this year was a success. The feedback from students around having to defend an actual application was positive. Most of them appreciated the chance to learn something about AWS. My talk The Cloud is Dark and Full of Terrors, which I did for B-Sides Augusta, was intended as an OSINT opportunity for them to get an idea of what we’d throw at them.
One of the things I spent most of my time doing for last year’s regional competition and this year’s preliminary competition was restoring machines by deleting and recreating the CloudFormation Stack. For this year’s regional competition, I added AWS Backup and AWS Backup policies to the mix. I gave the students a short “here’s how to restore your machine” document on the first day, and they were easily able to recover their machines. It worked so well the Red Team complained that all their implants kept getting zapped, and they’d have to re-establish their persistence.
AWS License Manager was a convenient way to ensure the Marketplace AMIs were available in all the AWS accounts. If I do an “AWS Organizations for 2022” post, this will be part of it.
In previous years, different teams ran the qualifiers and the regionals. By using the same scenario across both, we could take the technical level of regionals up a notch. The regional teams appreciated that they weren’t walking into a brand new tech stack and all the new AWS stuff we threw at them.
What didn’t work
The biggest problem we faced was the campus network. We arrived early the day before to set up, and it quickly became apparent the pfSense firewall wasn’t going to work. We replaced the pfSense three times before we engaged campus networking. At 9 pm the night before, we gave up and went to bed - the plan was to abandon the site-to-site VPN and rely on the Campus Wifi and the Client VPN the students used for the preliminary competition. It wasn’t until halfway through opening remarks that the campus firewall person disabled some feature in the Campus PaloAlto, and our pfSense machine started working. That delayed our start time by about an hour and prevented the build team from getting things set up and tested.
A number of OpenVPN clients don’t work well with ClientVPN leveraging AWS SSO. While this was super easy to use, the Red Team found issues using it with Linux and especially Kali Linux. This led to an interesting mix of ingress to the Fooli network - Google-Identity-AWS-SSO Client VPN, AWS-AD-Client-VPN, and the Site-to-Site VPN. Students were able to identify red team activity quickly.
Meme Generator was fragile - When broken, students had a hard time fixing it. We need to introduce a student-fed CICD capability.
With 26 teams in a single account and AWS region, we quickly hit rate limiting on the EC2 APIs. At the opening bell, many teams could not get a list of their machines and explore the environment. Next year we will need to split the teams in the preliminary round across several AWS accounts. I’d be reluctant to split across regions because the latency or a regional outage could favor or disadvantage one team over another.
CCDC scoring is based on service uptime. We built a variety of systems of different OS versions and flavors. Unfortunately, many of these didn’t serve a purpose in the Fooli meme generation scenario. For next year’s scenario, the servers will be more integral to the overall Fooli business purpose. The risk here is that if the students can’t keep all the pieces of the meme supply chain working, they’ll fail to score any points. It will be essential to figure out how to offer partial credit. Something like: 1pt for accepting a meme order, 1pt for generating the meme, 1pt for delivery to customer, 3pt for recording in the accounting system, etc. In a real scenario, being able to accept and generate a product while experiencing a temporary failure to deliver isn’t a complete failure of the business. However, as we saw with the Colonial Pipeline ransomware attack, failure to account can trigger a complete shutdown of operations.
Plans for next year
We’ve not settled on the scenario for 2023. It may or may not leverage the Fooli meme factory. I’ll begin to hand over primary AWS build responsibilities for the qualifier round. Then for the regionals, I will join the Red Team as the AWS expert. I expect a slew of PRs to Pacu to come out of this effort.
Build Improvements
A few notes on how I did the build and things I want to improve:
AWS SSO was heavily leveraged, and for each team’s build, I did a direct AWS SSO login to their account. Next time I want to leverage a cross-account build-team role from the corporate or build account.
Similarly, as I prepped each account, I created a unique SSH key for the EC2 Instance launch. The AWS EC2 key-pair was for the build team, and each AMI had student and scoring credentials baked in. Next year I want to generate an SSH Key for Build Team and import it into each AWS account.
Finally, the Moscow environment will be the RedTeam proving ground / live-fire environment and not handed to a team.
