(With apologies for the click-bait headline. It seems to be what all the cool kids are doing when they’re not on TikTok)
(2024 update: Daniel Grzelak has penned an excellent write up of how S3 encryption works, and demonstrated that S3 encryption is nothing more than an additional layer of access control. Check out S3 Bucket Encryption Doesn’t Work The Way You Think It Works)
I have spent way too much of my life the last three years in the cloud compliance and cloud security posture monitoring space. One of the biggest issues I’ve struggled with is putting a priority on the issue of re-encrypting existing cloud resources.
Let me start off by saying that you should always check that little box to encrypt at rest. Because at some point, some one is going to audit your environment and come back with: “this thing isn’t encrypted”. You will then waste more time in meetings with officious people asking when you will encrypt that thing. Save yourself the grief and encrypt now.
Da da da dat da dat da da da da ya da
Da da dat dat da ya da!
I even made you some scripts to help with buckets and volumes.
So, having said you should always encrypt, this post is about discussing what to do when you discover a cloud resource isn’t encrypted. I will argue that re-encrypting things in the cloud is a high-effort, low-security-value exercise that should be a the bottom of your cloud posture hygiene priority list.
Encryption At Rest
Lets talk about the value of Encryption At Rest in public cloud (and again, I’ll use AWS as my example, but these principles apply to most CSPs).
What threat are you defending against? Obviously, Disclosure is the primary one. Alteration is another. What are the most likely vectors of information disclosure or alteration in the cloud? Is it Ninjas descending on us-east-1a to steal the hard-drives your data is stored on? Or is it Bob in accounting who has an overly-permissioned IAM Key so he can download the AWS bill each month?
Don’t be focusing on encryption before you first have your IAM and account governance houses in order!
It doesn’t matter how much encryption you have on your data. If I provide AWS the right AKIA and 40 character random gobbledygook, Amazon will happily decrypt and hand me your data.
Ok then, besides ninjas, what threats does encryption-at-rest protect against?
- Cloud Insider Threats - AWS has access to your data. As Rich Mogull wrote: “If you can see your data in a Web browser after entering only your account password, the odds are extremely high that your provider can read it as well.” However, there are segregation of duties at AWS, and the person in the S3 team with access to your bits is not the same person who has access to your KMS key. So the S3 engineer would need to meet and conspire with the KMS engineer at the nearby Starbucks and then decide they want to go after your data. Maybe what you have is so valuable it’s worth the risk to them. Perhaps you should keep that shit on-prem.
- Bad Code Paths - KMS, even AWS Managed KMS, is tied to your account. Assuming a software bug in the S3 service removed some amount of access controls, KMS access controls might prevent the decryption of the data. As far as anyone knows this has never happened, but it’s not outside the realm of possibility.
- Poor Cloud Hygiene - If you have an EBS Volume that is encrypted, and you snapshot that volume and accidentally share it to the world, access to the KMS key would be required to hydrate that snapshot into a volume in an adversary’s account. However if you have such crappy cloud hygiene, you’re probably also not encrypting things consistently, so focus on fixing the hygiene and stop having myopic focus on encryption.
- Encryption-in-Transit in the hypervisor-plane - AWS documentations states that EBS encrypt/decrypt operations happen in the hypervisor of the compute node (not on the node where the volume is stored). As a result, the EBS I/O is encrypted when going across that back-end AWS network.
Types of KMS
AWS has three types of Customer Managed Keys (CMK), where the last one comes in two flavors:
- AWS Owned CMK - this seems like a oxymoron “AWS Managed Customer Managed Key”. These keys are used by AWS for services across multiple accounts
- AWS Managed CMK - are in your account, but AWS manages the resource policy, not the customer.
- Customer managed CMKs - another silly named key - Customer Managed Customer Managed Key. Here you as the key owner can manage the resource policy and control who can use the key and in what ways.
- Customer Provided Customer managed CMKs - With these Customer managed CMKs, AWS does not generate the master key material. It is created by you and imported into the KMS service. Here is the $65 million dollar warning though: “In the unlikely event of certain regionwide failures that affect the service (such as a total loss of power), AWS KMS cannot automatically restore your imported key material."
AWS Owned CMKs exist to check a box on a compliance spreadsheet. Luckily most cases this encryption is enabled by default.
AWS Managed CMKs might protect against the “Bad Code Paths” threat outlined above, while also checking a box on a spreadsheet.
Customer Managed CMKs do provide an extra level of security if you’re IAM and/or account governance is a mess. But defining a resource policy on a specific CMK, you can prevent other IAM Principals in the account from using that key for specific decrypt operations. This is important when you have large multi-tenant accounts due to poor application separation.
Customer Provided CMKs are a bad idea. If you need that you should stay on-prem. Otherwise you need to regularly test your procedures for extracting the key material from your on-prem vault and re-importing it into AWS. Because if there is a disaster with KMS, you are completely dead in the water till that is done.
The cost of re-encrypting
If you didn’t heed my initial advice and you did create a data store without enabling encryption at rest, what do you do?
For EBS volumes, you need to:
- Stop your instance
- create a snapshot
- create a new volume (with encryption)
- attach new volume to a new instance
- Start instance
- Suffer performance impacts till all the blocks are copied from the snapshot to the volume on read.
(here is a script I wrote three years ago when a cloud migration failed to do the right thing in terraform.)
For S3, you’ll need to copy each S3 Object and re-encrypt. AWS Batch Operations can help there. But for a large enough volume of data, you’ll incur some costs. PutObject operations aren’t cheap at scale.
RDS also requires taking a snapshot, encrypting the snapshot when you copy it, then launching a new RDS from the encrypted snapshot. With another performance impact.
The issue here is: “Is this worth it?” - The production downtime, performance impacts, lost time on the sprint that could be delivering business value or fixing the AppSec issues?
Encryption in Transit
All AWS services support TLS Endpoints and if you’re using the SDKs the default behavior is to use the https endpoint. This means that most of your serverless applications get encrypt-in-transit out of the box. API Gateway invoking a lambda, which calls out to Lex is all encrypted-in-transit (with identity as the perimeter too).
Additionally, ACM makes it dirt-simple to drop a custom TLS Cert on your load balancer or CloudFront (which is how this site is https://).
Where you run into problems is when you’re inside of the VPC. Virtual Private Cloud does not mean virtual encrypted cloud. There is no encryption of traffic inside your VPC.
Your VPC exists across multiple AZs which means it also exists across multiple buildings. Traffic between instances across AZs travels unencrypted across metro areas. If your threat model includes adversaries that can compel warrantless wiretaps, you need to focus on intra-VPC Encryption. If that’s not your adversarial threat model go back and look at your other cloud hygiene issues first.
All that said, intra-VPC Encryption-in-Transit is undifferentiated heavy lifting. Managing keys on instances and containers is added complexity that interferes with builders. AWS should provide KMS support on ENIs for all intra-VPC traffic. [edit-1]
Conclusion
When evaluating your cloud security posture priorities, encryption should be at the bottom of your list. Yes, you should enable Default EBS encryption, and then enable Default S3 encryption on all your S3 Buckets.
But first get your IAM house in order. Get rid of IAM users and figure out least-privilege identity federation. Fix your InstanceProfile or Lambda policies so you’re not the next CapitalOne.
Fix those Launch Wizards that are open to the world. Close your S3 buckets, and turn on CloudTrail FFS!
Make sure you’re leveraging organizations to effectively protect your accounts from outside the account (via SCPs). Operationalize GuardDuty and IAM Access Analyzer. Pay attention to your AppSec, since those SSRFs are how you’re leaking keys to the world.
Once you’ve done all that, then consider re-encrypting the cloud resources you missed encrypting on creation.
Edit 1
2020-Aug-3 @0710 EDT - In a slack discussion with Scott Piper after this was published, he led me to some docs that show some ENIs do support transparent encryption. However the instance types are quite limited, and NAT and Transit gateways aren’t supported. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/data-protection.html#encryption-transit
Additionally AWS states (emphasis mine):
All data flowing across the AWS global network that interconnects our datacenters and regions is automatically encrypted at the physical layer before it leaves our secured facilities. Additional encryption layers exist as well; for example, all VPC cross-region peering traffic, and customer or service-to-service TLS connections.(link)