What do we say to the god of account compromise? Not Today!

Threat Hunting with CloudTrail and GuardDuty in Splunk

AWS , CloudSecurity , Technology , AWS re:Inforce

This post is the reference section of my dev-chat at the first ever AWS re:Inforce conference in Boston. You can find my slides here.

The purpose was to give the audience a brief overview of how to conduct basic threat hunting in their CloudTrail and GuardDuty. We throw in a bit of Vulnerability Hunting and awareness with Antiope at the end.

Tools

The tools we need here are:

  • Centralized CloudTrail
  • Centralized GuardDuty
  • Antiope
  • Splunk.

CloudTrail

We centralize all our CloudTrail events from all our accounts into a single bucket. Splunk then points at the bucket and all our events are automatically ingested. CloudTrail is deployed at account creation by a CloudFormation template created by the security team.

GuardDuty

We leverage the GuardDuty Master/Member account concept. New accounts are discovered by Antiope and auto-invited into the master. We have cross account role that allows the security account to accept the invitation. As a global company, detectors are deployed in all AWS Regions. A CloudWatch Event is configured in the GuardDuty master account to invoke a lambda that will push the event to a Splunk HTTP Event Collector (HEC) cluster.

Antiope

I’ve written about Antiope before. It functions both as an inventory gathering tool, and it helps us discover and manage our several hundred AWS accounts.

Splunk

Splunk is the central tool we use for all log gathering and analysis. I do not claim to be much of a Splunk expert.

CloudTrail

Anatomy of a CloudTrail Event

This is what a generic CloudTrail event looks like:

{
  "awsRegion": "us-east-1",
  "eventName": "CreateBucket",
  "eventSource": "s3.amazonaws.com",
  "eventTime": "2019-06-09T15:37:18Z",
  "eventType": "AwsApiCall",
  "recipientAccountId": "123456789012",
  "requestParameters": {},
  "responseElements": null,
  "sourceIPAddress": "192.168.357.420",
  "userAgent": "[S3Console/0.4, aws-internal/3 aws-sdk-java/1.11.526
        Linux/4.9.152-0.1.ac.221.79.329.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.202-b08
        java/1.8.0_202 vendor/Oracle_Corporation]",
  "userIdentity": {
    "accessKeyId": "ASIATFNORDFNORDAZQ",
    "accountId": "123456789012",
    "arn": "arn:aws:sts::123456789012:assumed-role/assume-rolename/first.last@company.com",
    "type": "AssumedRole"
  }
}

A few key elements from a threat hunting perspective are:

  • eventName - This is the API Call made
  • eventSource - This is the AWS service (ec2, s3, lambda, etc)
  • sourceIPAddress - IP address the call came from. It’s either an IP, or an AWS service like cloudformation.amazonaws.com
  • userIdentity.arn - Depending on type, the attributes of userIdentity change, but the arn is always present
  • userIdentity.type - Typically IAMUser or AssumeRole
  • recipientAccountId - AWS Account ID the event was for. In this post they will all be 123456789012

requestParameters will change based on the API call. Sometimes there are useful filters in there.

Root Login Detection

The first basic query is to find all the root login attempts. The query looks like:

index=cloudtrail "userIdentity.type"=Root AND eventName=ConsoleLogin

And the response:

{
  "additionalEventData": {
    "LoginTo": "https://console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true",
    "MFAUsed": "No",
    "MobileVersion": "No"
  },
  "eventName": "ConsoleLogin",
  "eventSource": "signin.amazonaws.com",
  "eventType": "AwsConsoleSignIn",
  "responseElements": {
    "ConsoleLogin": "Success"
  },
  "sourceIPAddress": "192.168.357.420",
  "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko)
        Chrome/74.0.3729.169 Safari/537.36",
  "userIdentity": {
    "accessKeyId": "",
    "accountId": "123456789012",
    "arn": "arn:aws:iam::123456789012:root",
    "principalId": "123456789012",
    "type": "Root"
  }
}

In this particular event, we see MFAUsed is No. Since we have compliance monitoring around MFA on root, and notification of discovery of new accounts, we can correlate this event back to a new account being created and our cloud team putting MFA on the account for the first time.

IAM Login with no MFA

The purpose of this query is to see who is logging as an IAMUser without supplying MFA. In theory there should be none of these, so if we see one, we want to reach out to the account owner to understand why this user isn’t required to have MFA. This query is adapted from the CIS Foundations Benchmark for AWS.

index=cloudtrail ConsoleLogin "additionalEventData.MFAUsed"!=Yes "userIdentity.type"=IAMUser
| dedup userIdentity.arn sourceIPAddress
| table "userIdentity.accountId" "userIdentity.arn" sourceIPAddress "responseElements.ConsoleLogin"

Additionally, if the sourceIPAddress seems suspect we can bump the urgency of the escalation to the account owner.

This is that same query but including the City and Country of the login:

index=cloudtrail ConsoleLogin "additionalEventData.MFAUsed"!=Yes "userIdentity.type"=IAMUser
| dedup userIdentity.arn sourceIPAddress
| iplocation sourceIPAddress
| search Country!="United States"
| table "userIdentity.accountId" "userIdentity.arn" sourceIPAddress,
  City, Country "responseElements.ConsoleLogin"

Unauthorized Calls

This one is also part of the CIS Benchmarks. Here it is translated to Splunk and aggregated across all the accounts.

index=cloudtrail errorCode="AccessDenied" OR errorCode="UnauthorizedOperation"
| stats count by eventName userIdentity.arn

Failed IAM Console Logins

index=cloudtrail errorMessage="Failed*" eventName=ConsoleLogin sourceIPAddress!="357.420.*"
| iplocation sourceIPAddress
| stats count by sourceIPAddress, Country

Another CIS Benchmark query. Here we can exclude our known IP ranges, and decorate the results with the Country of origin.

Money Wasting

This one will find all instances that are launched (for all instance types) that are 10xlarge or bigger. If big money instances are not your norm, this alarm can find abuse or devs who don’t know better.

index=cloudtrail eventName=RunInstances
| regex "requestParameters.instanceType"=\d{2}xlarge
| stats count by requestParameters.instanceType

Wall of Shame

Shame

This query shows who (userIdentity.arn) opened a security group (eventName=AuthorizeSecurityGroupIngress) to the world (cidrIp"="0.0.0.0/0") on port 22 or 3389 (fromPort=22 OR fromPort=3389).

index=cloudtrail  eventName = AuthorizeSecurityGroupIngress
"requestParameters.ipPermissions.items{}.ipRanges.items{}.cidrIp"="0.0.0.0/0"
"requestParameters.ipPermissions.items{}.fromPort"=22
OR "requestParameters.ipPermissions.items{}.fromPort"=3389
| stats count by userIdentity.arn

IAM User Creation by Country

A common method of persistence is to create a new IAM User. This query returns the list of countries where a CreatUser was done. We exclude CloudFormation from these results.

index=cloudtrail  eventName="CreateUser" sourceIPAddress!="*.amazonaws.com"
| iplocation sourceIPAddress
| stats count by Country

UserCreation

To dive deeper into what was happening in Hong Kong

index=cloudtrail eventName="CreateUser" sourceIPAddress!="*.amazonaws.com"
| iplocation sourceIPAddress | search Country="Hong Kong"

Note how we have to pipe back to search after piping to iplocation.

Recon

This finds all attempts to GetCallerIdentity that didn’t come from the US. GetCallerIdentity is a common recon call that returns the IAMUser name and account number for a API Key and Secret.

index=cloudtrail eventName=GetCallerIdentity "userIdentity.type"=IAMUser
| iplocation sourceIPAddress
| search Country!="United States"
| stats count by userIdentity.arn Country

Interesting Events

Here are a handful of other interesting eventNames you might want to search on. Many of these are based on the CIS Foundation Benchmarks for AWS.

CloudTrail:

  • DeleteTrail
  • StopLogging
  • UpdateTrail

GuardDuty:

  • DeleteDetector
  • DeleteMembers
  • DisassociateFromMasterAccount
  • DisassociateMembers
  • StopMonitoringMembers

KMS:

  • ScheduleKeyDeletion
  • DisableKey

AWS Config Service:

  • StopConfigurationRecorder
  • DeleteDeliveryChannel
  • PutDeliveryChannel
  • PutConfigurationRecorder

Security Groups & NACLs:

  • AuthorizeSecurityGroupEgress
  • RevokeSecurityGroupEgress
  • CreateNetworkAcl
  • DeleteNetworkAcl
  • CreateNetworkAclEntry
  • DeleteNetworkAclEntry
  • ReplaceNetworkAclEntry
  • ReplaceNetworkAclAssociation

VPCs:

  • AttachInternetGateway
  • CreateVpcPeeringConnection
  • AcceptVpcPeeringConnection
  • CreateClientVpnEndpoint

GuardDuty

Like CloudTrail, Centralizing GuardDuty is a must for a large multi-account environment. Luckily GuardDuty has a master-account/member-account model that works across AWS Organizations. We leverage that to pull all GuardDuty findings, in every region, back to that region in a central GuardDuty account. From there a CloudWatch Event fires an AWS Lambda which pushes the finding to a Splunk HTTP Event Collector (HEC). The master account has this Detector/CWE/Lambda combination deployed in all AWS Regions. We maintain a redundant HEC cluster.

The code we use to manage this is available here

Sample Finding

{
  "id": "d5b0fccf-THIS-IS-UNIQUE-PER-FINDING",
  "account": "987654321098", <-- SECURITY ACCOUNT
  "time": "2019-06-14T14:07:29Z",
  "region": "us-east-1",
  "detail": {
    "schemaVersion": "2.0",
    "accountId": "123456789012", <-- MONITORED ACCOUNT
    "region": "us-east-1",
    "partition": "aws",
    "type": "Recon:EC2/PortProbeUnprotectedPort", <-- AWS CLASSIFICATION
    "severity": 2,
    "resource": {}, <-- either AccessKey or Instance
    "service": {
      "action": {
        "actionType": "PORT_PROBE",
        "portProbeAction": {
          "portProbeDetails": [
            {
              "localPortDetails": {"port": 22, "portName": "SSH"},
              "remoteIpDetails": {
                "ipAddressV4": "116.112.202.89", <--USEFUL
                "organization": {"org": "China Unicom Neimeng"},  <-- ALSO USEFUL
                "country": {"countryName": "China"},
                "city": {"cityName": "Ordos"},
                "geoLocation": {"lat": 39.6, "lon": 109.7833 }
              }
            }
          ],
          "blocked": false
        }
      }
      "resourceRole": "TARGET",
      "additionalInfo": {"threatName": "Scanner", "threatListName": "ProofPoint"},
    },
    "createdAt": "2019-02-27T23:41:19.160Z",
    "updatedAt": "2019-06-14T13:59:41.042Z",
    "title": "Unprotected port on EC2 instance i-fnord is being probed.",
    "description": "EC2 instance has an unprotected port which is being probed by a known malicious host."
}

Count of Finding Types

Simple Query to get a summary of all the GuardDuty findings in a period of time. This is a good place to start.

index=guardduty  | dedup id | stats count by detail.type

Because findings can re-occur, I always dedup by the finding id.

Logins From New IP Addresses

GuardDuty will learn where your users normally login from, and if it sees an unusual login, it will create a finding. As people travel this will generate several false-positives. However you can use this query to display all the findings.

index=guardduty "detail.type"="UnauthorizedAccess:IAMUser/ConsoleLogin"
"detail.service.action.awsApiCallAction.remoteIpDetails.organization.org"!="EXCLUDE YOUR ORG HERE"
| dedup "detail.service.action.awsApiCallAction.remoteIpDetails.ipAddressV4"
| rename "detail.service.action.awsApiCallAction.remoteIpDetails.country.countryName" as Country
| rename "detail.service.action.awsApiCallAction.remoteIpDetails.city.cityName" as City
| rename "detail.service.action.awsApiCallAction.remoteIpDetails.organization.org" as Org
| rename "detail.resource.accessKeyDetails.userName" as UserName
| rename "detail.resource.accessKeyDetails.userType" as LoginType
| rename "detail.service.action.awsApiCallAction.remoteIpDetails.ipAddressV4" as IPAddr
| table UserName City Country IPAddr Org LoginType

ConsoleLogin

Note how we leverage rename to provide heading columns that aren’t 40 char long.

You can add this before the first dedup to exclude all the logins that occurred inside the US

"detail.service.action.awsApiCallAction.remoteIpDetails.country.countryName"!="United States"

Credential Exfiltration

Probably the most critical GuardDuty alert you can receive is UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration. This indicates EC2 Instance Profile credentials have been used outside of AWS.

index=guardduty UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration

RDP Brute Forcing

This query displays key info about the RDP Brute Force GuardDuty finding. The data about the attacker is buried in the action.networkConnectionAction.remoteIpDetails, except for the port which is inside the localPortDetails.

index=guardduty "detail.type"="UnauthorizedAccess:EC2/RDPBruteForce"
| dedup id
| rename "detail.service.action.networkConnectionAction.remoteIpDetails.country.countryName" as Country
| rename "detail.service.action.networkConnectionAction.remoteIpDetails.city.cityName" as City
| rename "detail.service.action.networkConnectionAction.remoteIpDetails.organization.org" as Org
| rename "detail.service.action.networkConnectionAction.localPortDetails.port" as Port
| rename "detail.service.action.networkConnectionAction.remoteIpDetails.ipAddressV4" as IPAddr
| rename "detail.resource.instanceDetails.instanceId" as instanceId
| table City Country Org IPAddr Port instanceId

ConsoleLogin

This is the difference between “you have a vulnerability” and “you are under attack”

PrivilegeEscalation:IAMUser/AdministrativePermissions

This one is an IAM User call where the information is in awsApiCallAction and not in networkConnectionAction like the RDP Example.

index=guardduty "detail.type"="PrivilegeEscalation:IAMUser/AdministrativePermissions"
| dedup id
| rename "detail.service.action.awsApiCallAction.remoteIpDetails.country.countryName" as Country
| rename  "detail.service.action.awsApiCallAction.remoteIpDetails.city.cityName" as City
| rename "detail.service.action.awsApiCallAction.remoteIpDetails.organization.org" as Org
| rename "detail.resource.accessKeyDetails.userName" as UserName
| rename "detail.resource.accessKeyDetails.userType" as LoginType
| rename "detail.resource.accessKeyDetails.principalId" as principalId
| rename "detail.service.action.awsApiCallAction.remoteIpDetails.ipAddressV4" as IPAddr
| table UserName City Country IPAddr Org LoginType principalId

Antiope

Support Cases

This gives you a quick view of all the open support cases in the organization:

index=antiope  resourceType="AWS::Support::Case" | dedup resourceId
| table awsAccountName configuration.serviceCode configuration.categoryCode
  configuration.status configuration.subject

And this one covers AWS account specific things:

index=antiope  resourceType="AWS::Support::Case" "configuration.serviceCode"="customer-account"
| dedup resourceId

Splunk tends to re-ingest Antiope resources every time they’re queried. De-duplicating on the resourceId fixes this.

Wide Open Elastic Search Clusters

This one has come up thanks to several data leaks starting last fall.

index=antiope resourceType="AWS::ElasticSearch::Domain"
    NOT configuration.VPCOptions.VPCId=*
    NOT "supplementaryConfiguration.AccessPolicies.Statement{}.Condition.IpAddress.aws:SourceIp{}"=*
    NOT "supplementaryConfiguration.AccessPolicies.Statement{}.Condition.IpAddress.aws:SourceIp"=*
    NOT "supplementaryConfiguration.AccessPolicies.Statement{}.Condition.StringEquals.aws:SourceVpc"=*
| regex "supplementaryConfiguration.AccessPolicies.Statement{}.Principal.AWS"="\*"
| dedup resourceId
| table configuration.Endpoint resourceName awsAccountName

We’re looking for:

  • ElasticSearch Domains
  • Not in a VPC
  • Not using SourceIP or SourceVpc Conditions
  • Where the Resource Policy is AWS:* (anonymous access)

SourceIp can be a single value or an array, so it’s in the query twice.