Wednesday, March 6, 2019

Cyber Threat Hunting in AWS Logs




As increasingly high number of apps moving into cloud, AWS is one of the top preferred venue for cloud hosting. AWS provides solid logging capability for hunting using Splunk. AWS like other cloud infrastructure provides blue teams with number of tools and API’s , but it also creates and increases the attack surface. This post covers some of the basic AW Hunt techniques that can be implemented in Splunk as dashboard for or monitoring.

Splunk App for AWS provides lot of useful information, but I would prefer a single page view of notable events for quick glance and then worry about details as needed. This post does not cover how to bring the logs into Splunk, but ways to hunt for notable events.There will be a link to full dashboard at the end from my Github page.

Some of the native AWS specific log source from security monitoring perspective are
  • aws:config  : Configuration snapshots and historical configuration data from the AWS Config service.
  • aws:config:notification  : Configuration change notifications from the AWS Config service.
  • aws:cloudtrail : AWS API call history from the AWS CloudTrail service.
  • aws:cloudwatchlogs : Logs from the CloudWatch Logs service, including VPC Flow Logs. VPC Flow Logs allow you to capture IP traffic flow data for the network interfaces in your resources.
  • aws:cloudwatchlogs:vpcflow : VPC flow logs from the CloudWatch Logs service.
  • aws:s3:accesslogs : S3 access logs.
  • aws:cloudfront:accesslogs  : CloudFront access logs.
  • aws:elb:accesslogs  : ELB access logs.


From ATT&CK perspective, following category techniques are applicable (but not limited) to the AWS events.
  • TA0001 - Initial Access
  • TA0002 - Execution
  • TA0005 - Defence Evasion
  • TA0006 - Credential access


TA0006 - Credential access - Console Login

Console login from unusual location and high number of login attempts can be indicative of either bruteforcing or password spraying attacks. Pay attention to the “responseElements.ConsoleLogin” and additionalEventData.MFAUsed fields.
index=<index_name> sourcetype=aws:cloudtrail (eventName=ConsoleLogin) | iplocation src Country | strcat userName " performed --> " eventName user_event | table userIdentity.accountId, user_event, responseElements.ConsoleLogin additionalEventData.MFAUsed sourceIPAddress, Country, userAgent

TA0005 - Defence Evasion - VPC Network Configuration changes

Search to monitor for notable network configuration changes to the API : AttachInternetGateway, AssociateRouteTable, CreateRoute, DeleteCustomerGateway, DeleteInternetGateway, DeleteRoute, DeleteRouteTable, DeleteDhcpOptions, DisassociateRouteTable
index=<index_name> sourcetype=aws:cloudtrail (eventName="AttachInternetGateway" OR eventName="AssociateRouteTable" OR eventName="CreateRoute" OR eventName="DeleteUserPolicy" OR eventName="DeleteCustomerGateway" OR eventName="DeleteInternetGateway" OR eventName="DeleteRoute" OR eventName="DeleteRouteTable" OR eventName="DeleteDhcpOptions" OR eventName="DisassociateRouteTable" )  | iplocation sourceIPAddress | table _time awsRegion, eventName, userName, sourceIPAddress, Country, userAgent, additionalEventData.LoginTo userIdentity.arn


TA0005 - Defence Evasion - Network Access List Modification

Search to monitor for notable network access list modifications for CreateNetworkAcl, CreateNetworkAclEntry, DeleteNetworkAcl, DeleteNetworkAclEntry, ReplaceNetworkAclEntry, ReplaceNetworkAclAssociation
index=<index_name> sourcetype=aws:cloudtrail (eventName="DeleteNetworkAcl" OR eventName="CreateNetworkAcl" OR eventName="CreateNetworkAclEntry" OR eventName="DeleteNetworkAclEntry" OR eventName="ReplaceNetworkAclEntry" OR eventName="ReplaceNetworkAclAssociation") |rename userIdentity.arn as arn | stats count min(_time) as firstTime max(_time) as lastTime values(errorMessage) values(errorCode) values(userAgent) values(userIdentity.*) by src userName arn eventName

TA0005 - Defence Evasion - Network ACLs created with Higher ports

VPC comes with a default Network ACL that allows all inbound and outbound rules. And if you create a custom NACL, both inbound and outbound rules are denied. If you have not created a custom network ACL, then the subnets will be associated with VPC’s default ACL automatically. This will ‘Allow’ all traffic to flow into and out of the network. For example AWS NACL to a public subnet with instances that can receive and send Internet traffic over port 80 (HTTP) and higher ports 1024-65535. Not blocking the traffic over port 2049 (NFS) or ports vulnerable to denial of service attacks.
index=<index_name> sourcetype=aws:cloudtrail eventName=CreateNetworkAclEntry | mvexpand requestParameters | mvexpand responseElements | search requestParameters.portRange.from=1024 requestParameters.portRange.to=65525 requestParameters.ruleAction=allow | rename userIdentity.arn as arn | rename requestParameters.networkAclId as networkAclId | table _time aws_account_id src userName arn networkAclId requestParameters.* responseElements.*

TA0002 - Execution - Suspicious API calls to Create, Run or Terminate Instance
Splunk search to monitor for attempts to RunInstances, CreateInstances, LaunchInstances, TerminateInstances
index=<index_name> sourcetype="aws:cloudtrail" (eventName="RunInstances" OR eventName="CreateInstances" OR eventName="RunInstances" OR eventName="LaunchInstances" OR eventName="TerminateInstances") NOT (userName="AWSServiceRoleForAutoScaling" OR userName="AWSServiceRoleForEC2Spot" OR userName="EMR_DefaultRole" OR userName="Platform-Provision") | stats earliest(_time) as earliest latest(_time) as latest by userName, eventName| eventstats max(latest) as maxlatest| eval isOutlier=if(earliest &gt;= relative_time(maxlatest, "-1d@d"), 1, 0)| eval earliest=strftime(earliest,"%d-%m-%Y - %H:%M:%S") | eval latest=strftime(latest,"%d-%m-%Y - %H:%M:%S") | eval maxlatest=strftime(maxlatest,"%d-%m-%Y - %H:%M:%S") | eval day=strftime(day,"%Y/%m/%d")

TA0002 - Execution - Suspicious Unusual Amount of Modifications to Security Groups
Dashboard to detect modifications to Security groups using the API calls and also to measure the volume of change in comparison to previous instance for outliers. API calls: AuthorizeSecurityGroupIngress, AuthorizeSecurityGroupEgress, RevokeSecurityGroupIngress, RevokeSecurityGroupEgress, CreateSecurityGroup, DeleteSecurityGroup
index=<index_name> sourcetype="aws:cloudtrail" (eventName="AuthorizeSecurityGroup*" OR eventName="RevokeSecurityGroup*" OR eventName="CreateSecurityGroup" OR eventName="DeleteSecurityGroup") | bucket _time span=1d| stats count by userName _time | eventstats max(_time) as maxtime | stats count as num_data_samples max(eval(if(_time &gt;= relative_time(maxtime, "-1d@d"), 'count',null))) as "count" avg(eval(if(_time&lt;relative_time(maxtime,"-1d@d"),'count',null))) as avg stdev(eval(if(_time&lt;relative_time(maxtime,"-1d@d"),'count',null))) as stdev by "userName" | eval earliest=strftime(earliest,"%d-%m-%Y - %H:%M:%S") | eval latest=strftime(latest,"%d-%m-%Y - %H:%M:%S") | eval day=strftime(day,"%Y/%m/%d")| eval lowerBound=(avg-stdev*2), upperBound=(avg+stdev*2)| eval isOutlier=if(('count' &lt; lowerBound OR 'count' &gt; upperBound) AND num_data_samples &gt;=7, 1, 0)

TA0005 - Defence Evasion - Security Groups - Authorise or Revoke Activity
Monitoring for Authorisation or revoking activity
index=<index_name> sourcetype=aws:cloudtrail eventName=RevokeSecurityGroup* OR eventName=AuthorizeSecurityGroup*| fields eventName eventTime sourceIPAddress userIdentity.arn requestParameters.* responseElements.*| foreach requestParameters.ipPermissions.items{}.ipRanges.items{}.* [eval <<MATCHSTR>>='<<FIELD>>'] | foreach requestParameters.ipPermissions.items{}.* [eval <<MATCHSTR>>='<<FIELD>>'] | eval ipProtocol=if(ipProtocol=-1, "ALL", ipProtocol) | eval portRange=mvzip(fromPort, toPort, "-") | eval direction = case(match(eventName, ".*Ingress"), "ingress", match(eventName, ".*Egress"), "egress") | rename userIdentity.arn as arn responseElements._return as return requestParameters.groupId as groupId | fields - *items* responseElements.* requestParameters.* toPort fromPort _time _raw | table eventName eventTime sourceIPAddress arn * | fillnull value="N/A" | rename eventName as "Event Name", eventTime as "Event Time", sourceIPAddress as "Source IP Address", arn as ARN, cidrIp as "CIDR IP", direction as "Direction", groupId as "Group ID", ipProtocol as "IP Protocol", portRange as "Port Range", return as "Return" 

TA0005 - Defence Evasion- Indicator Removal on Host
Adversaries may delete or alter generated artefacts on a host system, including logs and potentially captured files such as quarantined malware. API Calls : StopLogging, DeleteTrail, UpdateTrail
index=<index_name> sourcetype=aws:cloudtrail ( eventName="DeleteTrail" OR eventName="UpdateTrail" OR eventName="StopConfigurationRecorder" OR eventName="DeleteConfigurationRecorder" OR eventName="DeleteAssessmentRun" OR eventName="UpdateAssessmentTarget") | iplocation sourceIPAddress | table _time awsRegion, eventName, userName, sourceIPAddress, Country, userAgent, additionalEventData.LoginTo userIdentity.arn

TA0001 - Initial Access Suspicious Provisioning Activity from Unusual country
Monitoring for instance provisioning attempts from countries that you typically don’t expect is another indicator to watchout for .
index=<index_name> sourcetype="aws:cloudtrail" eventName=Create* OR eventName=Run* OR eventName=Attach* | iplocation src | search NOT (Country="United States" OR Country=Australia) | stats earliest(_time) as earliest latest(_time) as latest by src, Country, sourcetype | eventstats max(latest) as maxlatest | eval isOutlier=if(earliest &gt;= relative_time(maxlatest, "-1d@d"), 1, 0) | eval earliest=strftime(earliest,"%d-%m-%Y - %H:%M:%S") | eval latest=strftime(latest,"%d-%m-%Y - %H:%M:%S") | eval day=strftime(day,"%Y/%m/%d")

TA0002 - Execution - Abnormally High AWS Instances Launched by User
In a typical AWS enterprise environment, auto scaling is used, and provisioning are automated by service accounts. Any user accounts involved in provisioning servers needs to be monitored.
index=<index_name> sourcetype=aws:cloudtrail eventName=RunInstances errorCode=success | bucket span=10m _time | stats count AS instances_launched by _time userName | eventstats avg(instances_launched) as total_launched_avg, stdev(instances_launched) as total_launched_stdev | eval threshold_value = 4 | eval isOutlier=if(instances_launched &gt; total_launched_avg+(total_launched_stdev * threshold_value), 1, 0) | search isOutlier=1 AND _time &gt;= relative_time(now(), "-10m@m") | eval num_standard_deviations_away = round(abs(instances_launched - total_launched_avg) / total_launched_stdev, 2) | table _time, userName, instances_launched, num_standard_deviations_away, total_launched_avg, total_launched_stdev

TA0002 - Execution - IAM Error Activity

Monitoring for IAM error activity for notable error codes is another easy way to find events of interest.
index=<index_name> sourcetype=aws:cloudtrail errorCode eventSource=iam*| spath requestParameters | rename userIdentity.arn as arn responseElements._return as return | table _time userName eventName, eventTime, errorCode, errorMessage, sourceIPAddress arn requestParameters| rename eventName as "Event Name", eventTime as "Event Time", errorCode as "Error Code", errorMessage as "Error Message" sourceIPAddress as "Source IP Address", arn as ARN, requestParameters as "Request Parameters"

Other Notable API Calls

Other API calls that are worth looking into.
index=<index_name> sourcetype=aws:cloudtrail (eventName="AddUserToGroup" OR eventName="AddRoleToInstanceProfile" OR eventName="AttachPolicy" OR eventName="PutPolicy" OR eventName="CreateRole" OR eventName="CreateSAMLProvider" OR eventName="UpdateSAMLProvider" OR eventName="UpdateLoginProfile" OR eventName="CreateAccessKey" OR eventName="DeleteTrail" OR eventName="UpdateTrail" OR eventName="StopConfigurationRecorder" OR eventName="DeleteConfigurationRecorder" OR eventName="DeleteAssessmentRun" OR eventName="UpdateAssessmentTarget") | iplocation sourceIPAddress | table _time awsRegion, eventName, userName, sourceIPAddress, Country, userAgent, additionalEventData.LoginTo userIdentity.arn

Accepted vs. Rejected Traffic by Location (Map)

Geostats view of the VPC flow logs showing the top accepted and blocked connections is a good place to start. High number of blocked connections
index=<aws_vpc_index> | stats sum(packets) as packets by City Country Region lat lon vpcflow_action src_ip | geostats sum(packets) by vpcflow_action

Dashboard
Following Github repository has the XML Splunk dashboard. Change the index and sourcetypes as required.
https://github.com/Brainfold/CyberThreatHunting/blob/master/CTH_AWS

Reference
https://www.sans.org/cyber-security-summit/archives/file/summit-archive-1492554533.pdf
https://www.gorillastack.com/wp-content/uploads/2017/11/CloudTrailEventNames.pdf