“Architect for the AWS you have, not the AWS you want” –Chris Farris, 2017
AWS is constantly innovating. This is both a blessing and a curse. It’s a blessing because things are getting better, cheaper, and faster. It’s a curse because the best way to accomplish an objective is constantly changing.
I’m interested in auto-remediation. How do I revert a bad state to a good state without asking a human to fix something. But the key is that the human has to know the cloud healed itself. Otherwise the human doesn’t learn (and the IaC templates get all outta wack). There are some vendors out there doing things, but at my scale the pricing is pretty brutal. They also don’t always solve the “ghost-in-the-cloud” problem where the cloud heals itself but the human isn’t informed.
AWS Config is Amazon’s cloud-provider-native answer to this. I’ve poked at config in the past. Time to poke at it again as it has evolved since my last look at it.
Config Service
As of August 2020, Config Service’s recorder & delivery channel cannot be deployed via AWS Organizations like Macie, GuardDuty, and IAM Access Analyzer. This means CloudFormation Stacks. In each account. In each Region.
But what about…
Stack Sets
Luckily, AWS Organizations does support StackSets. It has some interesting capabilities, such as:
In addition to setting permissions, CloudFormation StackSets now offers the option for automatically creating or removing your CloudFormation stacks when a new AWS account joins or quits your Organization. You do not need to remember to manually connect to the new account to deploy your common infrastructure or to delete infrastructure when an account is removed from your Organization. When an account leaves the organization, the stack will be removed from the management of StackSets. However, you can choose to either delete or retain the resources managed by the stack.
StackSets for Organizations does have some limits. It will not deploy a stackset to the payer account, so you need to deploy that manually. StackSets themselves cannot be created via CloudFormation (but that may be changing soon), so there are some gnarly CLI commands you need to run when deploying stacksets. There are also the soft-limits: 100 StackSets per admin account, and 2000 stack instances per stackset. With 16 regions, StackSets can support an organization no larger than 125 Accounts. Note: Enterprise Support can probably help you there, you’ll need to ask your account team.
Another aspect of Organization StackSets is how long they take to deploy. In a small test organization of 9 accounts, it took about 20 minutes to deploy to the 128 regions (excluding the payer which needed to be deployed separate). Back to the topic at hand…
Deploying the Config Recorder via Stack Sets
Anyway, getting config deployed to an organization is a bit of an ordeal. You also need to create a ServiceLinkRole for Config. But only once. So you need a condition to deploy this in a specific region, then you need to make sure that region is deployed first.
Conditions:
cCreateServiceLinkedRole: !Equals [ !Ref "AWS::Region", "us-east-1"]
Resources:
AWSServiceRoleForConfig:
Type: AWS::IAM::ServiceLinkedRole
Condition: cCreateServiceLinkedRole
Properties:
AWSServiceName: config.amazonaws.com
Finally, the config recorder will record global services multiple times. So you want to make sure that the IncludeGlobalResourceTypes parameter is only true in one region (I use us-east-1 because).
Conditions:
cIncludeGlobalResourceTypes: !Equals [ !Ref "AWS::Region", "us-east-1"]
Resources:
ConfigRecorder:
Type: AWS::Config::ConfigurationRecorder
Properties:
Name: default
RecordingGroup:
AllSupported: true
IncludeGlobalResourceTypes: !If [ cIncludeGlobalResourceTypes, True, False ]
RoleARN: !Sub 'arn:aws:iam::${AWS::AccountId}:role/aws-service-role/config.amazonaws.com/AWSServiceRoleForConfig'
Ok, so after the better part of a Saturday afternoon (the first of three required to make the blog post) messing with this, I finally get it deployed around 8pm. Now to setup Aggregation, so the data from all the accounts in all the regions is available.
Config Service ConfigurationAggregator
AWS Config Service released support for aggregation of Config resources and rules around re:Invent 2018. This solves the single-pane-of-glass issue. The CloudFormation definition even allows you to specify an entire org:
ConfigAggregator:
Type: AWS::Config::ConfigurationAggregator
Properties:
ConfigurationAggregatorName: !Sub "${AWS::StackName}-Aggregator"
OrganizationAggregationSource:
AllAwsRegions: True
RoleArn: !GetAtt ConfigAggregatorRole.Arn
It provides similar function to Antiope in that it can solve my “Is this my bucket?” problem. The disadvantage is the Aggregator account MUST be the AWS Organization Master account. The account that best-practices say never log into unless you need to do something with AWS Organizations (or download your bill).
But, it allows me to see all my Config Rules in one place. Which leads next to…
Conformance Packs
Conformance Packs are sets of AWS Config Rules. The AWS Provided examples are aligned to CIS or NIST CSF guidance.
Conformance Packs can also be deployed across an entire AWS Organization. This seems to be independent of Aggregation.
NISTConformancePack:
Type: AWS::Config::OrganizationConformancePack
Properties:
OrganizationConformancePackName: NISTConformancePack
DeliveryS3Bucket: !Sub awsconfigconforms-${pOrganizationConformancePackBucketSuffix}
TemplateS3Uri: !Sub "s3://${pOrganizationConformancePackTemplateBucket}/conformance-packs/NISTConformancePack.yaml"
The TemplateS3Uri is the location of the YAML definition. It looks suspiciously like a CloudFormation Template full of AWS::Config:ConfigRule. Note: The DeliveryS3Bucket must begin with awsconfigconforms, and requires a special bucket policy.
There is a catch: while ConfigurationAggregator can be multi-region & pan-organizational, These OrganizationConformancePack resources are pan-organizational but single-region.
Back to the StackSets!
I’m not going to even figure out that. One region can POC this for me.
I mention that “DeliveryS3Bucket” with the special bucket name prefixed by awsconfigconforms. You’d expect the Delivery Bucket to have the results of your conformance pack evaluations, yes?
Nope. The DeliveryS3Bucket contains nothing more than copies of the ConformancePackTemplates. So how do I get the results of my organization conformance packs?
Apparently you don’t.
Here are the CLI commands for conformance packs:
[chris@Patton] chris@payer (us-east-1): aws configservice help | grep conformance
o delete-conformance-pack
o delete-organization-conformance-pack
o describe-conformance-pack-compliance
o describe-conformance-pack-status
o describe-conformance-packs
o describe-organization-conformance-pack-statuses
o describe-organization-conformance-packs
o get-conformance-pack-compliance-details
o get-conformance-pack-compliance-summary
o get-organization-conformance-pack-detailed-status
o put-conformance-pack
o put-organization-conformance-pack
get-organization-conformance-pack-detailed-status returns the deployment status. Not the compliance.
Nope, to pull the rules for a conformance pack you need to go straight to the Config Rules they deploy. Luckliy I can do that with my aggregator:
[chris@Patton] chris@payer (us-east-1): aws configservice get-aggregate-config-rule-compliance-summary --configuration-aggregator-name org-config-Aggregator
{
"AggregateComplianceCounts": [
{
"ComplianceSummary": {
"CompliantResourceCount": {
"CappedCount": 196,
"CapExceeded": false <-- Aw hell. What is this limit going to mean?
Not going there today, but I bet it will f--- something up at scale.
},
"NonCompliantResourceCount": {
"CappedCount": 249,
"CapExceeded": false
},
"ComplianceSummaryTimestamp": "2020-08-29T07:22:56.433000-04:00"
}
}
]
}
Finally! (after several hours messing around with the various aws configservice CLI command). This is the command I can use to get the status of all my rules:
[chris@Patton] chris@payer (us-east-1): aws configservice describe-aggregate-compliance-by-config-rules --configuration-aggregator-name org-config-Aggregator
{
"AggregateComplianceByConfigRules": [
{
"ConfigRuleName": "cloudformation-stack-notification-check-conformance-pack-8nd4su297",
"Compliance": {
"ComplianceType": "NON_COMPLIANT",
"ComplianceContributorCount": {
"CappedCount": 3,
"CapExceeded": false
}
},
"AccountId": "123456789012",
"AwsRegion": "us-east-1"
},
{
"ConfigRuleName": "iam-root-access-key-check-conformance-pack-8nd4su297",
"Compliance": {
"ComplianceType": "COMPLIANT"
},
"AccountId": "123456789012",
"AwsRegion": "us-east-1"
},
...
Dumping that full json file and doing a quick grep on the ConfigRuleName presents this bit of fun. THE RULES HAVE DIFFERENT NAMES IN EACH ACCOUNT!!!!
"ConfigRuleName": "restricted-ssh-conformance-pack-7w6d8cpan",
"ConfigRuleName": "restricted-ssh-conformance-pack-8nd4su297",
"ConfigRuleName": "restricted-ssh-conformance-pack-9sqtxkl5b",
"ConfigRuleName": "restricted-ssh-conformance-pack-eazno1oqj",
"ConfigRuleName": "restricted-ssh-conformance-pack-iemewqrpa",
"ConfigRuleName": "restricted-ssh-conformance-pack-juja9owj4",
"ConfigRuleName": "restricted-ssh-conformance-pack-lbfzjhgib",
"ConfigRuleName": "restricted-ssh-conformance-pack-mfumtkzxl",
"ConfigRuleName": "restricted-ssh-conformance-pack-wdpm2q4va",
Mapping these back to common requirement will be a pain. I’m not even going to bother to figure that out today.
The magic incantation to get the non-compliant resource ends up being this:
aws configservice get-aggregate-compliance-details-by-config-rule --configuration-aggregator-name org-config-Aggregator --config-rule-name sns-encrypted-kms-conformance-pack-juja9owj4 --account-id 123456789012 --aws-region us-east-1
{
"AggregateEvaluationResults": [
{
"EvaluationResultIdentifier": {
"EvaluationResultQualifier": {
"ConfigRuleName": "sns-encrypted-kms-conformance-pack-juja9owj4",
"ResourceType": "AWS::SNS::Topic",
"ResourceId": "arn:aws:sns:us-east-1:123456789012:StackSet-config-recorder-org-deploy-14e18186-2be1-49bc-8d7f-20c94b967ab8-us-east-1" <-- This is what I was looking for
},
"OrderingTimestamp": "2020-08-15T22:03:03.052000-04:00"
},
"ComplianceType": "NON_COMPLIANT",
"ResultRecordedTime": "2020-08-16T08:02:45.156000-04:00",
"ConfigRuleInvokedTime": "2020-08-16T08:02:44.978000-04:00",
"Annotation": "Amazon SNS Topic is not encrypted with KMS", <-- Along with this
"AccountId": "123456789012",
"AwsRegion": "us-east-1"
}
]
}
(Humm, probably should fix my CFT to support KMS on that topic)
Ok, so my logic is call describe-aggregate-compliance-by-config-rules, iterate that list for Compliance.ComplianceType == NON_COMPLIANT then call get-aggregate-compliance-details-by-config-rule with the rule name, account-id and region to get the ResourceId that is non-compliant. Yay! I’ve finally re-implemented CloudSploit.
Maybe I’ll write a python someday for this.
Replacing Antiope?
Could AWS Config Service replace Antiope?
I always intended for Config recorder to be an alternate for the resource inventory functionality of Antiope. And it probably would do a better job at some of the most ephemeral resources (ENI, EC2, etc.)
But could it entirely replace the multi-account discovery and search engine functionality. In theory the aggregator can pull across accounts and across regions. I could log into the payer (ugh) and go see if that bucket I’m looking for actually belongs to us.
Not sure it’s there yet.
From the payer account’s aggregator view you don’t see the compliance rules. Only within the account do you see the resource’s compliance.
Here is what a resource looks like via the aggregator:
And here is the same resource as seen in the child account. Notice how the Rules tab exists in addition to the tags?
One last bit - it doesn’t seem like you can deploy an Organizational Conformance Pack from the console. In some of my testing I tried and it just deployed a conformance pack for the payer, not the child accounts.
Costs of all this
What is this going to cost?
According to the pricing page
- Each resource change recorded costs $0.003.
- Each Rule evaluation costs $0.001.
- Because they are apparently evil, conformance pack evaluations cost $0.0012. Sure 2 one-hundredths of a penny isn’t a large amount of evil, but as we learned in Superman III it can add up.
Rule/Pack evaluations occur based on the MaximumExecutionFrequency parameter, which in the NIST conformance pack I used as my test was set to TwentyFour_Hours. 39 rules in 9 accounts, 16 regions, across a 30 day month should be 168k evaluations, or $202. My entire org currently runs me less than $15/mo.
The docs aren’t entirely clear if rule evaluations also occur with every resource change. I only have anecdotal recollection of a conversation with Corey Quinn at re:Inforce last year that makes me think they do (something about how if you have lots of changes, Rules cost more now than they did when they were $2 each). I may be completely mistaken. The RhinoSec party killed a lot of brain cells that week.
This blog post is at least three weeks in the drafting. I’ve got to stop at some point and publish. Note: I’ve not gotten any auto-remediations or human notices implemented. I’ve spent the better part of 6 weekend days messing with this (in between having a life and doing other things - this is not 60 hrs of work). Let me just throw the CloudFormation templates into gists, run a spell check and push this to production. Oh, after I find an ancient vacation picture as the banner graphic.
CloudFormation Templates
- Config Recorder Template (for StackSet)
- Org Aggregation Template - Deploy in the payer in one region.
- AWS CLI Commands to deploy the stackset