Welcome to the American Thanksgiving holiday, which for us cloud peeps is the quiet period between pre:Invent and re:Invent. Traditionally the run up to AWS re:Invent is chock full of feature releases (and some product releases) that don’t merit mention in Andy or Werner’s keynotes.
As I was slammed with work things, I wasn’t following pre:Invent (and will probably miss much of the lame online re:Invent), so I’m going back and reviewing all the announcements for things of note to a serverless nerd or security geek.
There were 279 announcements in November. Here they are graphed by day:
I suspect there may have been more slated for 11/25, but someone broke us-east-1 and took down the Internet.
Here are the 29 I thought interesting. Rather that review these by date (which is what you get from the AWS Feed), I thought I’d categorize them into useful buckets.
Watch out for these
Amazon EventBridge announces improved resource policies for event buses
Announced On: Nov 20, 2020
Amazon EventBridge announces improvements to event bus resource policies that make it easier to build applications that work across accounts. With this change, you can now send events to, and create rules on event buses in another account while relying on the event bus resource policy to manage your permissions.
Anytime I see Resource Policies I think “Great, my developers will figure out how to make this public”. Which is awesome because with EventBridge I can make my CloudTrail public. Or I can allow anyone to publish GuardDuty findings to my account. TODO - validate what these are
AWS Organizations Related
AWS Config supports delegated administrator account
Announced On: Nov 24, 2020
AWS Config lets you assess, audit, and evaluate how your AWS resources are configured, and helps you determine your overall compliance against the configurations specified in your internal guidelines. You can use AWS Config aggregators to collect your configuration and compliance data from the below sources, and aggregate that data into a single account and AWS Region to get a centralized view of your resource inventory and compliance.
Delegated Admin accounts came out this spring with GuardDuty and are supporting more and more features. If you read my rant against Multi-Account Config you’ll know this is a pretty good release. I’ve not had time to play with it (ask again after the Christmas break).
AWS Security Hub supports delegated administrator account
Announced On: Nov 23, 2020
AWS Security Hub is now integrated with AWS Organizations to simplify security posture management across all of your existing and future AWS accounts in an organization. With this launch, new and existing Security Hub customers can delegate any account in their organization as the Security Hub administrator and centrally view security findings from up to 5,000 AWS accounts. The integration with AWS Organizations allows you to automatically enable Security Hub and its automated security checks in any existing and newly created accounts in the organization. You can also now see AWS account names alongside account IDs in the Security Hub console. Customers using Security Hub’s existing multi-account management feature can transition to this new AWS Organizations-enabled multi-account management without any disruption to existing Security Hub usage.
Another service that relies on AWS Config supports organizations and delegated administrator, allowing the security team to manage Security Hub in all the child accounts. Open question: does this work like GuardDuty and all Security Hub events are visible in the child account? This would allow the security team to centrally manage the Security Hub configuration, while providing visibility back to account owners. This would finally make Security Hub a useful enterprise tool for CSPM.
AWS Firewall Manager supports AWS Network Firewall
Announced On: Nov 18, 2020
AWS Firewall Manager now supports AWS Network Firewall, making it easy for security administrators to centrally configure and deploy Network Firewall rules across their organization. AWS recently launched AWS Network Firewall, a highly available, managed firewall service that gives customers enhanced control and visibility of all traffic leaving and entering their network. With Firewall Manager support, customers can centrally manage the deployment of Network Firewall rules across accounts, organizational units (OUs), and Amazon virtual private clouds (VPCs) in their organization.
Centralized control of the new Network Firewall (discussed below) is cool to see at launch, rather that 9 months later after everyone has spent hours engineering a work-around.
Amazon S3 Storage Lens
Announced On: Nov 18, 2020
Amazon S3 Storage Lens delivers organization-wide visibility into your object storage usage and activity trends, and makes actionable recommendations to improve cost-efficiency and apply data protection best practices. S3 Storage Lens is the first cloud storage analytics solution to provide a single view of object storage usage and activity across tens to hundreds of accounts in an AWS organization, with drill-downs to generate insights at the account, bucket, or even prefix level. Drawing from more than 14 years of experience helping customers optimize storage, S3 Storage Lens analyzes organization-wide metrics to deliver contextual recommendations to find ways to reduce your storage costs and apply best practices on data protection.
I think this is more of a governance than security function, but I look forward to checking out how much this help with S3 compliance issues.
AWS Systems Manager Explorer now provides a multi-account, multi-region summary of AWS Config compliance
Announced On: Nov 11, 2020
Starting today, AWS Systems Manager Explorer provides a summary of AWS Config rules and associated resource compliance, to help you check overall compliance status and quickly find non-compliant resources. Systems Manager Explorer is an operations dashboard that provides a view of your operations data across your AWS accounts and Regions, helping you see where you may need to investigate and remediate operational issues. AWS Config enables you to assess, audit, and evaluate the configurations of your AWS resources.
Regional Services that don’t provide aggregation still means that an enterprise security team has to build or purchase external tools to aggregate the data from 16-18 sources. Getting data from all regions into a single place is a win.
AWS Trusted Advisor supports AWS Organizations
Announced On: Nov 17, 2020
Starting today, you can use the console to aggregate AWS Trusted Advisor recommendations across all accounts in your organization with AWS Organizations. The new Organizational View feature in Trusted Advisor allows you to generate reports with detailed check results across multiple accounts in your AWS organization. In addition, you can view a high level summary of check status within the console.
Another governance tool that allows you to aggregate data from all accounts.
AWS Backup and AWS Organizations bring cross-account backup feature
Announced On: Nov 19, 2020
AWS Backup now supports cross-account backup, enabling AWS customers to securely copy backups across accounts within their AWS Organizations.
I don’t use AWS Backup, but as we saw with CodeSpaces, I probably should - and if I did it should go to another AWS account.
New Security tools
Introducing the AWS Network Firewall - a new managed service to deploy network security across your Amazon VPCs with just a few clicks
Announced On: Nov 19, 2020
AWS Network Firewall is a new AWS-managed service that makes it easy to deploy essential network protections for all of your Amazon Virtual Private Clouds (VPCs). The service can be set up with just a few clicks and scales automatically with your network traffic, so you don’t have to worry about deploying and managing any infrastructure. AWS Network Firewall is for customers who want to inspect and filter traffic to, from, or between their Amazon VPCs.
Azure has had a managed firewall for quite some time now, so it’s good to see AWS catching up. TODO - Research this some
Introducing AWS Gateway Load Balancer
Announced On: Nov 11, 2020
Today AWS announced the availability of AWS Gateway Load Balancer, a new service that helps you deploy, scale, and manage third-party virtual network appliances such as firewalls, intrusion detection and prevention systems, analytics, visibility and others. An addition to the Elastic Load Balancer family, AWS Gateway Load Balancer combines a transparent network gateway (that is, a single entry and exit point for all traffic) and a load balancer that distributes traffic and scales your virtual appliances with the demand.
Shoving an on-prem firewall in-line to your highly elastic cloud-native architecture is downright stupid. But if your security polices were lift-and-shifted from on-prem like your applications, it could be a requirement that is shoved down the throats of cloud teams. Gateway Load Balancers seem to provide a more elastic and resilient architecture for inspection and potential blocking of all traffic into or out of your cloud network.
Announcing protection groups for AWS Shield Advanced
Announced On: Nov 16, 2020
AWS Shield Advanced now allows you to bundle resources into protection groups, giving you a self-service way to customize the scope of detection and mitigation for your application by treating multiple resources as a single unit. Resource grouping improves the accuracy of detection, reduces false positives, eases automatic protection of newly created resources, and accelerates the time to mitigate attacks against multiple resources. For example, if an application consists of four CloudFront distributions, you can add them to one protection group to receive detection and protection for the collection of resources as a whole. Reporting can be consumed at the protection group level, in addition to the resource level, giving a more holistic view of overall application health.
Shield Advanced is one of those services that costs $3k/month, so you don’t see many independent bloggers talk about it. That said I’m going to be doing a blog post about this at some point so I’ll cover this release then.
Amazon S3 Object Ownership is now generally available with AWS CloudFormation support
Announced On: Nov 9, 2020
Amazon S3 Object Ownership is now generally available with the addition of support for AWS CloudFormation. S3 Object Ownership is a new S3 feature that enables bucket owners to automatically assume ownership of objects that are uploaded to their buckets by other AWS Accounts. This helps you to standardize ownership of new objects in your bucket, and to share and manage access to these objects at scale via resource-based policies such as a bucket policy or an access point policy. Whether your S3 bucket receives data from other AWS accounts, or stores output from AWS services like AWS CloudTrail, S3 Object Ownership simplifies the work of creating and maintaining shared data sets on Amazon S3.
One of the biggest issues with centralized CloudTrail is providing access to an account’s generated events back to that account. Before S3 Object Ownership, the central logging account owned the bucket, but the CloudTrail Service owned the objects. This made it impossible to use bucket policies to provide an account access to its own events. Thus you had to build a stupidly-complex Rube Goldberg machines from Lambda, SNS, and S3 Events, or setup a second CloudTrail and pay the costs of that.
It’s good AWS Service teams aren’t incentivized on revenue, otherwise the CloudTrail team would be looking for revenge on the S3 team. Also, I like how the CloudFormation support is called out in the announcement for general availability. It means our complaints about cfn not being a launch requirement are being heard.
AWS Lambda code signing and verification. Plus AWS Signer!
Announced On: Nov 23, 2020
You can now ensure that only trusted and verified code is deployed in your AWS Lambda functions. With Code Signing for Lambda, administrators can configure Lambda functions to only accept signed code on deployment. When developers deploy signed code to such functions, Lambda checks the signatures to ensure the code is not altered or tampered. Additionally, Lambda ensures the code is signed by trusted developers before accepting the deployment.
This actually leverages another service I missed getting announced, AWS Signer:
This feature uses AWS Signer, a fully managed code signing service from AWS. Administrators create a Signing Profile, a resource in Signer that is used for creating signatures, and use AWS Identity and Access Management (IAM) to provide access to users. Within Lambda, administrators specify the permitted signing profiles for each function. Administrators can also configure whether to warn or reject if signature checks fail at deployment.
Hopefully they fix this soon:
https://twitter.com/jcfarris/status/1331973110002814976
Announcing Modules for AWS CloudFormation
Announced On: Nov 24, 2020
You can now define your infrastructure and applications in AWS CloudFormation with reusable building blocks called modules. A module encapsulates one or more resources and their respective configurations for reuse across your organization.
Modules are great to provide secure and compliant building blocks to your developers. They leverage the CloudFormation registry (a technology I’ve not played with yet). Currently, CloudFormation modules only support JSON templates.
Here is a good blog post to get your started.
AWS Single Sign-On enables attribute-based access control for workforce users to simplify permissions in AWS
Announced On: Nov 24, 2020
AWS Single Sign-On (SSO) now enables you to create fine-grained permissions for your workforce in AWS using attributes, such as cost center and department, defined in your AWS SSO identity source. Your administrators can now implement attribute-based access control (ABAC) with AWS SSO to centrally manage access to your AWS accounts and simplify permissions management at scale.
AWS SSO is a neat concept, but adoption required violating many best practices. It’s getting better and I highly recommend it when you don’t have anything better. It comes pre-setup with Control Tower. I need to configure it in my personal AWS organization, so that will be a future blog post.
AWS Single Sign-On enables administrators to require users to set up MFA devices during sign-in
Announced On: Nov 23, 2020
AWS Single Sign-On (SSO) administrators can now require users to self-enroll multi-factor authentication (MFA) devices during sign-in. For your users without a registered MFA device, you can require them to complete a self-guided MFA enrollment process following a successful password authentication. This allows administrators to secure their organization’s AWS environments with MFA without having to individually enroll and distribute authentication devices to users.
MFA GOOD.
AWS Security Hub adds five new integrations and a new consulting partner
Announced On: Nov 4, 2020
AWS Security Hub is now integrated with 3CORESec, cloudtamer.io, Prowler, StackRox, and ThreatModeler.
Seeing an open-source tool like Prowler on here is cool.
AWS Identity and Access Management introduces new policy defaults for IAM user passwords
Announced On: Nov 19, 2020
AWS Identity and Access Management (IAM) now introduces new policy defaults for passwords of IAM users. This policy improves the default security for all AWS customers by ensuring customers set stronger passwords for IAM users in their AWS accounts.
AWS is stepping over the line on Shared Responsibility to keep new accounts from doing something stupid. They need to do this more.
Managed Backup Retention for AWS CloudHSM
Announced On: Nov 25, 2020
AWS CloudHSM automatically takes a backup of your HSM cluster once a day and whenever an HSM is added to or removed from your cluster. Until today, however, customers were responsible for deleting old backups. Deleting out of date backups is important to prevent inactive users and expired login credentials from being used to access sensitive data on the HSM.
If you’re required to use CloudHSM, you have my deepest sympathies.
Serverless Technologies
I love doing things with serverless. I got started because I didn’t ever want to have to deal with my vulnerability management team at the last job (“tool say bad. you must fix."). Now it just makes more sense to me and allows me to handle events and multiple threads without having to know how to do any of that in Python.
AWS Step Functions now supports Synchronous Express Workflows
Announced On: Nov 24, 2020
AWS Step Functions now supports the synchronous executions of Express Workflows, allowing you to easily build web-based applications and orchestrate high-volume, short-duration microservices.
Best as I can tell, these don’t solve some of the concurrency issues I experienced processing a thousand AWS accounts via Map Iterator states in step functions (AWS had a non-configurable, not-described-as-a-limit hard limit of around 50 concurrent executions, regardless of how much concurrency the lambda supported). TBH, I’m not entirely sure what this does.
AWS Lambda can send logs to custom destinations
Announced On: Nov 12, 2020 (currently in Preview)
You can now send logs from AWS Lambda functions directly to a destination of your choice by using AWS Lambda Extensions. AWS Lambda Extensions are a new way for monitoring, observability, security, and governance tools to integrate with Lambda, and today, you can use extensions that send logs to the following providers: Datadog, New Relic, Sumo Logic, Honeycomb, Lumigo, and Coralogix.
This leverages another new Lambda feature called Lambda Extensions. They also have a blog post about the new Logging API extension.
I suspect a cloud security researcher could find some very interesting side effects from these new exposed APIs.
Amazon EventBridge introduces support for Event Replay
Announced On: Nov 6, 2020
Amazon EventBridge now supports Event Replay, which makes event-driven applications more durable and extensible by providing developers an easy way to replay past events. Event replay enables developers using Amazon EventBridge to build applications with the confidence that they can quickly recover from errors in their code, and the ability to easily extend their existing applications to add new functionality.
EventBridge has the ability to archive and then replay events. Pricing aint cheap at 10¢ per GB (plus S3 monthly storage rates). Nothing says you can’t archive and replay AWS Service events, so if you want to roll-your-own CloudTrail at way more the cost you can!!! This probably also explains why they announced Amazon EventBridge adds Server-Side Encryption (SSE) and increases default quotas on November 24th (not bad, new feature didn’t support KMS for only about 18 days).
Here is a blog post telling you more.
AWS Step Functions now supports Amazon EKS service integration
Announced On: Nov 16, 2020
AWS Step Functions is now integrated with Amazon EKS, making it easier to build resilient applications that orchestrate jobs running on Kubernetes with AWS services such as AWS Lambda, Amazon SNS, and Amazon SQS with minimal code. You can now build workflows including steps that launch tasks in Amazon EKS and wait for its completion without writing code to manage the state of the Kubernetes job.
If I ever build a serverless application that requires Kubernetes, please for the love of god, stage an intervention. I’m sure some customer asked for this, but AWS’s customer obsession didn’t talk them down from that cliff.
AWS ElasticSearch
I love me some ElasticSearch from my days building Antiope. Here are a few new features that look interesting from a security and usability standpoint.
New security features with ElasticSearch and Kibana
Announced On: Nov 24, 2020
Amazon Elasticsearch Service has introduced several security enhancements to the fine-grained access control feature that include a revamped and improved security workflow in Kibana, and integration with Open Distro for Elasticsearch Alerting and Anomaly Detection features.
Only available in version 7.9, these new features are centered around fine-grained access control features.
Amazon Elasticsearch Service custom domain names
Announced On: Nov 5, 2020
Amazon Elasticsearch Service now provides the ability to define a custom endpoint for your domain and associate an SSL certificate from AWS Certificate Manager (ACM). Defining a friendly name makes it easier for your users to access Kibana, and allows you to move to a new domain without updating your clients.
Mostly cosmetic, this also allows you to burn down your clusters easier. You don’t need to re-point all your users and clients.
Amazon Elasticsearch Service adds support for hot reload of dictionary files
Announced On: Nov 18, 2020
Amazon Elasticsearch Service now supports the ability to reload dictionary files without reindexing your data. Elasticsearch uses analyzers to convert string data into terms or tokens that power its search capabilities. These analyzers can do things like remove white space and stop words, perform stemming, handle compound words, and add synonyms. Previously, on Amazon Elasticsearch Service these analyzers could only process data as it was indexed. If you wanted to add some additional synonyms at a later time, you had to reindex your data with the new dictionary file.
I never mastered Elasticsearch, so I often had to delete, recreate, and then re-index all my data. This doesn’t necessarily do that, but it does let you change index setting at search.
Even more amazing, apparently you can use SQL syntax when searching ElasticSearch. That’s pretty dang awesome and something I need to play with in Antiope someday.
DynamoDB
You now can use a SQL-compatible query language to query, insert, update, and delete table data in Amazon DynamoDB
Announced On: Nov 23, 2020
You now can use PartiQL (a SQL-compatible query language)—in addition to already-available DynamoDB operations—to query, insert, update, and delete table data in Amazon DynamoDB. PartiQL makes it easier to interact with DynamoDB and run queries in the AWS Management Console. Because PartiQL is supported for all DynamoDB data-plane operations, it can help improve the productivity of developers by enabling them to use a familiar, structured query language to perform these operations.
Speaking of SQL… The hardest part of using DDB is figuring out how to actually use the boto3 client. AWS just created PartiQL. For ideas on what it can do see: this page. If you’re worried you don’t see boto3 on that page, it is supported with the execute_statement() method. Woot!
Now you can use Amazon Kinesis Data Streams to capture item-level changes in your Amazon DynamoDB tables
Announced On: Nov 23, 2020
With Amazon Kinesis Data Streams for Amazon DynamoDB, you can capture item-level changes in your DynamoDB tables as a Kinesis data stream. You can enable streaming to a Kinesis data stream on your table with a single click in the DynamoDB console, or via the AWS API or AWS CLI.
I can’t help but wonder if this is partially responsible for yesterdays issue. I guess is like DynanmoDB Streams, but directly to Kinesis. There are a few limitations (same region & same account), and apparently no CloudFormation support at launch. If your DDB Streams weren’t scaling for you, this might help.
Encrypt your Amazon DynamoDB global tables by using your own encryption keys
Announced On: Nov 6, 2020
With Amazon DynamoDB global tables, you can give massively scaled, global applications local access to DynamoDB tables for fast read and write performance. All of your data in DynamoDB is encrypted by default using the AWS Key Management Service (KMS). Starting today, you can now choose a customer managed key for your global tables, giving you full control over the key used for encryption of your DynamoDB data replicated using global tables. Customer managed keys also come with full AWS CloudTrail monitoring so you can view every time the key was used or accessed.
yawn Encryption… But good to know if you couldn’t do this before due to lack of encryption support.
That’s it. All the pre:Invent goodness. I’ll have another writeup of what gets announced at re:Invent. Be sure to stop by the AWS booth at the back of the exhibit hall to ask the AWS experts more about these new things. Oh, right. Shit. Nevermind….