So I did it again. Proving I’m the most incompetent Security Hero EVER, I committed eight different access keys to a public GitHub repository for eight different AWS Accounts.
What is fascinating is the consistently inconsistent response of AWS support. Behold a tale of seven cities and a professor.
How is Fooli even still in business?
I pushed eight AKIAs and their secrets to GitHub for the Incident Response Class I’m teaching at BSides Augusta this week. As part of the class, the students need to trace back what an attacker did, and I figured a good first lab was the tried and true: user commits an access key, and a crypto-miner comes along and bills them more than their organs are worth on the black market.
Since this is Fooli, I decided that Dinesh would be the patsy for this exercise. So in 8 different accounts, I created a Dinesh IAM User and an Access Key. I put all of them into a nicely formatted ~/.aws/credentials file and pushed it to GitHub.
In every case but one, I got a support ticket opened, and for every support ticket, I responded with the following:
Greetings AWS.
The access key referenced in this support case was deliberately published as part of an incident response scenario. There is a Service Control Policy (p-REDACTED) in the Organizations Management Account (REDACTED) that limits this key’s access to a single IP Address (REDACTED).
This key will be deactivated shortly.
The Professor
We’ll start with the first account, the account I do development in, and the one I’ll dub “The Professor”. This account has seen access key leaks before. I used it for some testing, and the AKIA for this account already had a support case opened for it, and at the time of the commit, the support case was closed.
What’s interesting about this case is that AWS didn’t apply the quarantine policy on the second leak. Silence. No support case, nothing. I had to manually pretend to be AWS to get the class CloudTrail to look right.
Berlin
The response from “L”1 to the support case in Berlin was:
Hello,
I noticed that you contacted us while signed in as an AWS Identity and Access Management (IAM) user. The root account holder creates and maintains each IAM user’s permissions.
To proceed, do the following steps:
- Ask the root account holder or administrator to use the AWS account root user sign-in credentials to reply to this case.
- Have the root user review all resources present on the account and confirm if they are authorized
The root user can also give permission for us to discuss a specific issue with a specific IAM user. To do this, someone signed in as the root user must reply directly to this case and give permission.
For more information about IAM permissions, see the following links:
https://docs.aws.amazon.com/IAM/latest/UserGuide/access.html
https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-permissions-ref.html
For more information on how to sign in as a root account holder, see the following link:
https://docs.aws.amazon.com/accounts/latest/reference/root-user-sign-in.html
We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.
Best regards,
L
Amazon Web Services
Ok, weird. I guess we found another edge-case use for the root user: Responding to support inquiries about IAM Users. It makes sense that if the account’s IAM level is compromised, anyone can impersonate anyone, so only the root user can be trusted.
Let’s see what’s happening in Denver.
Denver
“F” gave me a much different read:
Hi Chris,
I’m contacting you about the security notice sent on Oct 1st. The Investigations team has confirmed that your account hasn’t been compromised and shows no risk of compromise.
We recommend rotating and deleting the exposed AWS access key that ends in US6H
- To delete IAM user access keys, go to the AWS Management Console: https://console.aws.amazon.com/iam/home#users .
- To delete root user access keys, go here: https://console.aws.amazon.com/iam/home#security_credential .
- If your application uses any exposed or compromised access keys, you must rotate the exposed key to re-secure your resources. See the following links for instructions on how to do this: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKeyhttps://aws.amazon.com/premiumsupport/knowledge-center/delete-access-key/
We have now removed the restrictions that were placed on your account. If you have another inquiry that isn’t related to unauthorized activity on your AWS account, then create a new support case from the following page:
https://docs.aws.amazon.com/awssupport/latest/user/case-management.html
Review the following resources to learn more about AWS Security best practices and policies:
Shared Responsibility Model: https://aws.amazon.com/compliance/shared-responsibility-model/
AWS CloudTrail: https://aws.amazon.com/cloudtrail/getting-started/
Amazon CloudWatch: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/gs_monitor_estimated_charges_with_cloudwatch.html#gs_creating_billing_alarm
AWS Trusted Advisor: https://aws.amazon.com/premiumsupport/technology/trusted-advisor/
AWS Identity and Access Management - MFA: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa.html
AWS Account Management - Best practices for AWS accounts: https://docs.aws.amazon.com/accounts/latest/reference/best-practices.html
GitHub - AWS Labs: https://github.com/awslabs/git-secrets
AWS Cost Anomaly Detection: https://docs.aws.amazon.com/cost-management/latest/userguide/manage-ad.html
For your convenience, I have resolved this case. No further action is required on your part.
We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.
Best regards,
F
Amazon Web Services
“The investigations team” is interesting. As is the part about “restrictions that were placed on your account.” While this was happening, “Gilfoyle” was removing the quarantine policy, and I was spinning up a fake t3g.nano crypto-miner.
Helsinki
R was the only support person to give me his last name (which I’m omitting for his privacy). That said, R wasn’t messing around. AWS Security Hero or not, I needed to “terminate” my access key!
Hello!
Thank you for the confirmation on the services!
As per AWS policies, once an access key is expossed, it needs to be terminated so we can remove the alert on it, I understand that you say it is on purpose and it is a controled enviroment but the policies still require the deletion of it so I would need to requst you to proceed with the steps below:
Step 1: Rotate and delete the exposed AWS access key ending on LPBV. –
- To delete IAM user access keys, go to the AWS Management Console: https://console.aws.amazon.com/iam/home#users .
- To delete root user access keys, go here: https://console.aws.amazon.com/iam/home#security_credential .
- If your application uses any exposed or compromised access keys, you must rotate the exposed key to re-secure your resources. See the following links for instructions on how to do this: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKeyhttps://aws.amazon.com/premiumsupport/knowledge-center/delete-access-key/
Once done please come back to us so we can complete the process.
Thank you!
We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.
Best regards,
R
Amazon Web Services
Lisbon
Ah, Lisbon, the city that cancer and COVID denied me visiting. “C1” was reading from the same run-book as L. I need to log in as root to comment on the support case.
Hello,
I noticed that you contacted us while signed in as an AWS Identity and Access Management (IAM) user. The root account holder creates and maintains each IAM user’s permissions.
To proceed, do the following steps:
Ask the root account holder or administrator to update your IAM permissions and grant you access. Or, ask the root account holder to reply to this case.
The root user can also give permission for us to discuss a specific issue with a specific IAM user. To do this, someone signed in as the root user must reply directly to this case and give permission.
For more information about IAM permissions, see the following links:
https://docs.aws.amazon.com/IAM/latest/UserGuide/access.html
https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-permissions-ref.html
For more information on how to sign in as a root account holder, see the following link:
https://docs.aws.amazon.com/accounts/latest/reference/root-user-sign-in.html
We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.
Best regards,
C1
Amazon Web Services
Oslo
Curious “C2” sought out more information.
Hi Chris,
C2 from AWS Suppot Team.
Thank you for the information provided.
In ordwr to our team to review your AWS account and to avoid any restrictions on it, we need more information from you. Respond to this case with answers to the following:
Which location are you trying to access the account from (country/city)?
Does anyone other than you have access to the account (for example, an IT developer)?
If you answered yes to the previous question, where is that person accessing the account from?
What AWS services and resources are you using on your account currently? To view details about your charges, see your Bills page:
https://console.aws.amazon.com/billing/home#/bill
- Are all active Identity and Access Management (IAM) users, roles, and policies, and IAM access keys and root access keys authorized by you?
To check IAM users, open the console here:
https://console.aws.amazon.com/iam/home#users
To check IAM roles, open the console here:
https://console.aws.amazon.com/iam/home#/roles
To check IAM policies, open the console here:
https://console.aws.amazon.com/iam/home#/policies
To check IAM user access keys, open the console here:
https://console.aws.amazon.com/iam/home#users
To check root user access keys, go here:
https://console.aws.amazon.com/iam/home#security_credential
- Confirm that you have reset your account password. This is required before we can continue with this process. For information on how to reset your password, see the following documentation:
After you respond with the requested information, I’ll forward it to our service team.
Your time and cooperation are greatly appreciated, we look forward hearing back from you soon!
We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.
Best regards,
C2
Amazon Web Services
Rio
This was by far the most exhaustive set of remediation instructions. Warning me to check into all the regions, even ones that aren’t enabled, and telling me to ensure there aren’t unauthorized resources across a number of services that cryptominers could use (such as sagemaker, lightsail, and ec2 spot). “M1” wins the most through award, but she did miss that there is now a Tel Aviv region. :(
Hello,
We detected an abnormal pattern in your AWS account that matches unauthorized activity.
This activity is related to your AWS access key DHRI , which might indicate that this access key and the corresponding secret key are compromised.
In accordance with the AWS Customer Agreement and/or other agreements with us governing your use of our service, and to protect your account from excessive charges, we may terminate any suspected unauthorized resources on your account and limit your ability to use the AWS service. However, this does not make your account secure.
FOLLOW THE INSTRUCTIONS BELOW TO SECURE YOUR ACCOUNT (Let us know after you complete the steps, and be sure to leave the support case open):
Step 1. Rotate and delete the exposed AWS access key DHRI:
- To delete IAM user access keys, open the AWS Management Console:
https://console.aws.amazon.com/iam/home#users .
- To delete root user access keys, see the following:
https://console.aws.amazon.com/iam/home#security_credential .
- If your application uses any exposed or compromised access keys, you must rotate the exposed key to re-secure your resources. See the following links for instructions on how to do this:
https://aws.amazon.com/premiumsupport/knowledge-center/delete-access-key/
Note: Rotating and deleting the exposed key alone might not be sufficient to protect your account. See step 2.
- Step 2. Check your AWS CloudTrail log in each AWS Region for unauthorized creation of IAM users, policies, roles, temporary security credentials, and any other unsanctioned activity. Delete any unauthorized IAM users, roles, and policies that you find. Revoke any temporary credentials that you find. For more information, see the following:
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html
https://aws.amazon.com/cloudtrail/pricing/
You can’t revoke temporary credentials obtained via the root user. For more information, see the following:
- To delete unauthorized IAM users, see the following:
https://console.aws.amazon.com/iam/home#users
- To delete unauthorized policies, see the following:
https://console.aws.amazon.com/iam/home#/policies
- To delete unauthorized roles, see the following:
https://console.aws.amazon.com/iam/home#/roles
- To update the password of unauthorized IAM user, see the following:
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_passwords_admin-change-user.html
- To revoke temporary credentials, see the following:
You can also revoke temporary credentials by deleting the IAM user.
NOTE: Deleting IAM users can impact production workloads and should be done with care. You cannot revoke temporary credentials obtained via the root user. For more information, see the following:
Take steps to prevent any new credentials from being publicly exposed. For more information, see Best Practices of Managing your Access Keys:
http://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html
Step 3. Check for unauthorized AWS usage.
Before you start:
Make sure to check all AWS Regions. Unauthorized usage can occur in any AWS Region and your console might show you only one Region at a time. To switch between Regions, you can use the drop-down in the upper-right corner of the console screen.
Make sure to check for unauthorized usage in the following AWS Regions that might be disabled:
Africa (Cape Town)
Asia Pacific (Hong Kong)
Asia Pacific (Jakarta)
Europe (Milan)
Middle East (Bahrain)
Middle East (UAE)
If you find unauthorized usage in any of the preceding Regions, make sure that those Regions are enabled. If they’re enabled, then you can terminate the unauthorized resources or activity. Confirm in your response if you already did this. For instructions on enabling Regions, see the following:
https://docs.aws.amazon.com/general/latest/gr/rande-manage.html
Keep in mind that disabling a Region does not terminate the resources.
- Make sure that no termination protection or any other back-up restoration system such as Elastic Load Balancing or AutoScaling groups is enabled on the resources you want to delete. For more information, see the following:
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-relaunched-after-termination/
https://aws.amazon.com/premiumsupport/knowledge-center/source-ec2-instances/
To check for unauthorized AWS usage, do the following:
- Check your CloudTrail logs. For information on how to do this, see the following documentation:
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html
- Check your Bills page. Choose the billing month.
https://console.aws.amazon.com/billing/home#/bill
- Check Cost Explorer. For information on how to do this, see the following documentation:
https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/ce-what-is.html
https://docs.aws.amazon.com/cost-management/latest/userguide/ce-modify.html
- Repeat steps 1 and 3 for all relevant AWS Regions.
Some common usage types to check for are Amazon EC2 instances, EC2 Spot bids, AWS Lambda functions, AMIs, Amazon EBS volumes, EBS snapshots, Amazon Lightsail instances, and Amazon SageMaker notebook instances. For help with deleting resources that are associated with these services, see the following:
EC2 instances:
https://aws.amazon.com/premiumsupport/knowledge-center/delete-terminate-ec2/
EC2 Spot bids:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html#terminating-a-spot-instance
Lambda functions:
AMIs:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/deregister-ami.html
EBS volumes:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-deleting-volume.html
EBS snapshots:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-deleting-snapshot.html#ebs-delete-snapshot
Lightsail instances:
https://lightsail.aws.amazon.com/ls/docs/en_us/articles/delete-an-amazon-lightsail-instance
SageMaker notebook instances:
https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-cleanup.html
ECS resources:
https://docs.aws.amazon.com/AmazonECS/latest/userguide/delete_cluster.html
Amazon S3 buckets:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/delete-bucket.html
Amazon RDS resources:
https://aws.amazon.com/premiumsupport/knowledge-center/delete-terminate-rds/
Amazon ECR resources:
https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-delete.html
For information about deleting resources associated with other AWS services, see the documentation for the specific service here:
https://docs.aws.amazon.com/index.html#lang/en_us
- Preventative actions:
- We recommend that you enable Amazon GuardDuty and AWS WAF Fraud Control account takeover prevention (ATP):
Amazon GuardDuty is an AWS threat detection service that helps you continuously monitor and protect your AWS accounts and workloads. To learn more, see the following:
https://aws.amazon.com/guardduty
ATP is a feature of AWS WAF (web application firewall) that protects your application’s login page against credential stuffing attacks, brute force attempts, and other anomalous login activities. To learn more, see the following:
https://docs.aws.amazon.com/waf/latest/developerguide/getting-started.html
https://docs.aws.amazon.com/waf/latest/developerguide/waf-atp.html
For additional information about re-securing your account, see the following:
https://aws.amazon.com/premiumsupport/knowledge-center/potential-account-compromise
If you believe that your account is secure and there is no unauthorized access or usage, then confirm this in writing. If you need help with this process, let us know.
To reply to this support case, sign in to the AWS Support Center:
https://console.aws.amazon.com/support/home
If your case needs urgent attention, then choose the “Phone” or “Chat” option.
We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.
Best regards,
M1
Amazon Web Services
Tokyo
Our final memefactory is Tokyo, and “M2” took the opposite approach to M1.
Hi, Chris
I am following up regarding your account and billing situation.
Thanks for your quick response toward this security event, and I’m glad you are already taking actions to secure the account.
The exposed key needs to be deleted so that your account is considered secured, so please, let me know once you have removed it to move on assisting you.
Once you have done so, feel free to reply to this case.
Looking forward to hearing back from you.
We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.
Best regards,
M2
Amazon Web Services
Wrap up.
So there you have it… Eight different access key leaks, seven different responses (or non-responses). I want to say that all of these support cases came in almost the same instant. All of the responses were within minutes of each other. I cannot imagine the number of support cases the average tech processes on a given evening, so I do not fault any of them.
The entire purpose of this post was mostly a way for me to pull each response and look at it a little deeper. It was an interesting enough set of data I figured I’d share it with the community.
Bella Ciao!
-
After drafting this post, I decided to anonymize the names of the support reps who responded to just their first letter. ↩︎