Chris Farris

Earlier this week, I did a big no-no. I deliberately published an AWS Access Key and its associated secret to GitHub.

While I suspect many of you now think, “what a horrible cloud security person he is”, I figured I’d share my learnings.

Timeline of events & quarantine

The first GetCallerIdentity came from an AWS IP Address (35.83.131.170). That could have been AWS performing a check of the user. The second GetCallerIdentity came from an OVH IP Address.

The CloudTrail event for the AttachUserPolicy was interesting:

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "AIDA2HONFREDACTED",
        "arn": "arn:aws:iam::703REDACTED:user/test1",
        "accountId": "703REDACTED",
        "accessKeyId": "ASIA2HONFREDACTED",
        "userName": "test1",
        "sessionContext": {
            "sessionIssuer": {},
            "webIdFederationData": {},
            "attributes": {
                "creationDate": "2023-03-23T12:53:07Z",
                "mfaAuthenticated": "false"
            }
        },
        "invokedBy": "AWS Internal"
    },
    "eventTime": "2023-03-23T12:53:07Z",
    "eventSource": "iam.amazonaws.com",
    "eventName": "AttachUserPolicy",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "AWS Internal",
    "userAgent": "AWS Internal",
    "requestParameters": {
        "userName": "test1",
        "policyArn": "arn:aws:iam::aws:policy/AWSCompromisedKeyQuarantineV2"
    }
}

We see from the userIdentity.arn that the compromised user test1 was the principal that performed AttachUserPolicy; however, the IAM User only had the ReadOnlyAccess policy applied. The userIdentity.accessKeyId was an ASIA, which means that somehow, AWS used my compromised account credentials to perform an action they did not have permissions to do.

The “Mumbai (Krishna Nagar) Threat Actor”

Up to this point, the activity was related to “Is this real” or quarantining the key. The 3.109.16.140 “Threat Actor” is the first attempt to enumerate the environment.

This is interesting behavior. They knew that IAM was a global service, but not S3 or Route53. They only listed SNS Topics in two regions, ap-south-1 and ap-southeast-1. A ControlTower SCP denied the ListTopics in ap-south-1, so perhaps their script looks for that? The eks:ListClusters were also denied by SCP, but the EKS service hasn’t yet done the work to notify if a deny is related to an SCP.

All in all, the first GetCallerIdentity occurred at 08:58:08 EDT, and the last call from this address was 08:58:20 - a 12-second duration.

This seems like a somewhat sophisticated actor. They’re using a go based tool. They enumerated a handful of interesting resources like ACM Certs and Route53 zones. Listing Identities in SES can be useful from both a phishing target perspective and from a “can I turn this account into a spam farm” perspective. I’d be curious if they were using an known open-source tool.

The "Beauharnois Threat Actor" GitGuardian

OVH hosts this IP Address, and Splunk iplocation tells me it’s Beauharnois Canada (which is southwest of Montreal).

This actor was the second IP to ping my access key at 08:56:50. It continued to do so at longer intervals until I disabled the access key before I went to dinner.

• 08:56:50 EDT
• 08:58:15 EDT
• 09:00:04 EDT
• 09:02:02 EDT
• 09:12:17 EDT
• 09:32:18 EDT
• 09:52:17 EDT
• 10:52:03 EDT
• 12:52:04 EDT
• 13:09:48 EDT
• 16:52:16 EDT
• 17:09:40 EDT

Update (2023-Apr-1): Turns out this IP range belongs to the good folks at GitGuardian. I didn’t mention it in my initial post, but I do have their free plan, and their email notice of my public key came in about a minute before the AWS notice that the Quarantine policy was applied. H/T to Jessie in the Cloud Security Forum Slack for the pointer.

The “Pollença Threat Actor”

By about 2 pm, I was sad that no one wanted to Crypto-mine with me. Then Pollença came along with their AWSPowerShell.Common/4.1.90.0 .NET_Core/6.0.5 OS/Microsoft_Windows_10.0.17763 PowerShellCore/7.-1 ClientAsync scripts.

Their MO was the check the region’s GetServiceQuota and DescribeAccountAttributes, then attempt to create a VPC. As that failed, it doesn’t look like they tried to create subnets or launch instances. Here is what they’d do in a typical region:

With both a ridiculous number of SCPs and AWS’s quarantine policy, this IP address was going to get nowhere. Still it tells me that I should give a little bit of attention to any abnormal GetServiceQuota or DescribeInstanceTypeOfferings API calls.

Breakdown of the initial actors

I only left the key active during the day while I was in front of a keyboard and watching in Splunk. It looks like maybe three to five “threat actors” (and I use that phrase a bit tongue-in-cheek) tried to use it.

I’m going to try this again in a week or so. I’ll be curious if it’s the same group, or different IPs that will jump on this key.

AWS’s Responses

In the sixty or so hours since I published the key, I’ve received three emails from AWS. The first one came within two minutes of publishing the key. The other two came 24 and 48 hrs after AWS opened the first case.

The first email contained a friendly reminder to add my Security Contact to the account:

AWS Health includes the full name fields of recipient primary and/or alternate contacts of accounts to help with your business operations. The account 703REDACTED currently does not have a full name for the security contact associated with it. To enable this feature, please populate the name field of the security contact (and alternate contacts if you would like them addressed in relevant communication) via the Billing console or can be managed programmatically. For more information please visit https://aws.amazon.com/blogs/mt/programmatically-managing-alternate-contacts-on-member-accounts-with-aws-organizations/

The other interesting detail is (emphasis mine):

Step 4: [IMPORTANT] You must respond to the existing Support Case or create a new one to confirm completion of the steps above in order to restore access to your account, prevent suspension, and apply for a billing adjustment, if applicable.

Emails two and three look like this (also emphasis mine):

Dear AWS Customer,

We are following up with you, as your AWS Account may still be compromised. Please review the previous notice we have sent and take immediate action to secure your account. If you believe that your account is secured and there is no unauthorized access or usage, please contact us immediately by responding to this Support case.

This account compromise poses a security risk to your account (including other account users), and could lead to excessive charges from unauthorized activity. To protect your account from excessive charges, we have temporarily limited your ability to use some AWS services. To remove the limits, please follow the instructions provided in our initial notice.

If the unauthorized usage is not stopped we may suspend your AWS account. To further protect your account from excessive charges, we may terminate any suspected unauthorized resources on your account.

If you have any questions, please contact us by responding through this Support case.

The first service I noticed I no longer have access to is billing.

That’s all I’ve got for now. On Saturday afternoon I re-enabled the Access Key and I’m curious to see two things:

1. What will AWS do as I continue to ignore their support cases?
2. Will someone come along and try the key again?

Stay tuned for the exciting conclusion of “Chris f–s around and finds out.”

Apr 4, 2023 updates:

Two folks have reached out to let me know that the “Beauharnois Threat Actor” was actually GitGuardian - a really nifty service that scans your public & private repos for secrets. Git Guardian actually notified me of the public key exposure before I got the email from AWS.

AWS sent me this nine days after the key was exposed and the first support case was opened:

We have not heard back from you regarding case 1234567890 in the last 7 days. We hope this means you have resolved your issue or question. If you need continued support, please contact us using the following URL: REDACTED

(If you will connect by federation, log in before following the link.)

Best Regards,

Amazon Web Services

After I re-enabled this key, only one GetCallerIdentity event occured on March 31st at midnight UTC from an AWS Address. No one is regularly scanning GH for keys and trying to use them, as this key is still public in my repo.

Second update today:

From: Amazon Web Services no-reply-aws@amazon.com

We have not heard back from you regarding case 1234567890 in the last 10 days. We hope this means you have resolved your issue or question. If this is not the case, please re-open case 1234567890 using the following URL and we will continue to support you: REDACTED

(If you will connect by federation, log in before following the link.) Please let us know if we helped address your issue:

If YES, click here: REDACTED

If NO, click here: REDACTED

*Please note: this e-mail was sent from an address that cannot accept incoming e-mail. Please use the link above if you need to contact us about this same issue.

Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc.