What it is
The Southeast Collegiate Cyber Defense Competition is an annual competition where 8 teams from various colleges have to defend their systems from Red Team attacks while also executing on management-type business challenges.
It is typically held in the spring. With the pandemic still in place, this year’s competition could not be held on the KSU campus like usual. Instead they decided to try and run the competition remotely in AWS, with teams communicating via Zoom, and working off AWS Workspaces.
I was sucked in to help with the AWS build out.
What I did
In previous years, each team was given a class-c network with a handful of systems to defend. Their systems were segmented from the main competition network via a firewall. KSU provided laptops the students could work from. One key was that internet access was severely limited. Unlike anyone in the real world, these poor students didn’t have access to google and stackoverflow. This was a key requirement to the AWS configuration, and one that created a ton of complexity I’d never implement in the real world.
For this year’s competition, we decided it would mirror scenarios from the previous years. A handful of Linux and Windows servers would be deployed in each team’s network. They would have a Palo Alto firewall to manage. Internet access would be limited.
The primary differences was the use of AWS Workspaces instead of laptops. Each team had their own VPC, and VPC Peering was used to allow their Workstations to manage their servers.
There was a plan to use as many native AWS services as possible to simplify the build-team’s job:
- AWS Simple AD
- AWS Client VPN
- Traffic Mirroring
- AWS Workspaces
- AWS Workdocs
- Transit Gateways
- AWS Ingress Routing
- Application Load Balancers & AWS WAF
- AWS Cloudformation
Lots of re-designing
The initial design called for each team’s VPC to connect to the central corporate VPC via Transit Gateways. The Palo Alto firewall would sit between corporate and the team’s infrastructure. The problem there is that all the design patterns for a Palo Alto + Transit Gateway leveraged a single PA, not eight. No normal enterprise would want 8x the management headache.
Turns out that dropping a firewall in the Team’s VPC to force traffic didn’t exactly work with the VPC Router. When I needed to tell the TGW to send all traffic in and out of the VPC via the PA’s ENI, the VPC router denied my ability to set a more specific route on the VPC’s CIDR range.
Furthermore - after doing the initial setup of Workspaces, I realized Workspaces could only be deployed in the same subnets as the AWS Directory Servers. Since I wasn’t an AD Admin, AWS Simple AD was the way to go. So while the initial plan was for a single competition Directory, with Workspaces in each team’s VPC, I was left with the choice of managing 9 (8 Teams + Corporate/GoldTeam) AD, or moving all the Workspaces into one subnet.
What went wrong
One of my original ideas was to mirror previous year’s networks as much as possible. Each Team was given IP space of 10.10.10X.0/24 where X was their team designation. 10.10.100.0/24 was for the corporate network. Previous year’s also had VOIP phones and video cameras, but this was pretty much replaced entirely with Slack and Zoom.
As this was a two day competition and I wanted to keep networking as simple as possible, I made the decision early on that this would be a single AZ competition. This pretty much failed as soon as I tried to use any AWS managed service. ALBs, Client VPN, and AWS Workspaces each required I supply two subnets in different AZs. Also fun trivia fact: AWS Workspaces isn’t available in all AZs in us-east-1. I probably burned down and rebuild the VPC half a dozen times as I iterated across getting these things setup correctly. Yay for CloudFormation - the burn/rebuild process to 20-40min and was 98% unattended.
Using a /24 for a VPC in AWS goes against almost all the AWS best practices I know. You always want to have extra breathing room in case a new subnet needed to be added (yup, did that a lot), or in case your application needed to scale. Since I knew we have a finite number of team, finite number of servers, and a finite number of contestants, I did the math and realized I easily had a 10% overhead and so was fine. Narrator voice: He was not fine
I’m reasonably spoiled with having AWS Enterprise Support. If I had a question about how I should use a specific service, I’d reach out to my dedicated account team and get the answers from them, or direct from the service team. So when we discovered a few days before the competition that the Simple AD was not forwarding traffic to the VPC Resolver like the docs said, we made the decision we’d just have the students set their DNS at the start of the competition.
Unfortunately, Between creating the build-team Workspaces and the student’s Workspaces, I disabled local-admin feature in Workspaces. So while we did have the students connect to their workstation ahead of time, we did not have them test anything in the workspace. Only the team captains, who had their Workspaces created prior to disabling local admin, could do any DNS. We also didn’t test our Workspaces fully with the “Corporate DLP rules” in place, so we may have missed some usability issues with WorkDocs.
The fix for the local admin problem was to rebuild the workspace. That was really easy to trigger via boto3. Unfortunately, the rebuild process consumes several IP addresses per machine. This wasn’t discovered during the build, but when the rebuild happened, we exhausted all the IPs in the /26!! Eight Workspaces failed to rebuild till we terminated some things to free up just enough addresses. Getting these last few Workspaces online took almost 2 hours. The ENIs spawned as part of the rebuild took several hours to free up.
Lots of other issues popped up:
- Using Panorama to manage the firewall website allowlist. This actually broke all of the Red Teams ability to beacon. “Corporate” had to whitelist all the RedTeam’s stuff. Since there were about 30 sites, we couldn’t let the students see the whitelist, since the Red Team C2 was kinda obvious.
- Failed to make Transit Gateway Work with PANs. PANs were a requirement since Palo Alto was a sponsor.
- Red Team expected certain things to work (like wireguard) which weren’t exactly friendly to AWS VPC’s built in security controls. EC2 Instances can’t just go claim any old IP address, and expect to get traffic.
- Due to licensing issues with the PAs, I couldn’t get NAT working with team firewalls at the outset (I did get it working on the Corp PA), so I used Nat Gateways.
- Leveraging a single PA in a single Subnet/AZ, but needing to talk to two ALB endpoints in the DMZ subnet required some custom routing in the PA config.
What went right
- CFT rebuilds of boxes worked well.
- CloudEndure worked well to import VMs
- PAN URL Filtering worked, we couldn’t make pfSense work.
- Students had fun and learned a lot about real-world scenarios
What I learned
- Never try and recreate on-prem in AWS. Just don’t. Even when you think you don’t need a cloud architecture, you need a cloud architecture.
- Palo Alto firewalls are kinda neat, but if you really want to automate their deployment, you need a VM-Series expert.
- Each Traffic Mirroring ENI destination can only support 10 sessions -
I’m going to be doing this again next year, with a deeper focus on making AWS part of the competition. Tooling like GuardDuty, Macie, IAM Access Analyzer and others will be in-scope for the scenario. That and the fictional company (fooli.wtf) is going to be Silicon Valley’s top producer of Dank memes, so the web app they need to defend will be fun.
