Introduction
Amazon Web Services (AWS) stands as a prominent cloud provider, offering an array of services that generate valuable log data crucial for monitoring, security, and compliance in modern IT ecosystems. This guide will assist you in configuring Snare Central for the task of collecting and processing logs via the AWS Kinesis Data Stream.
This setup guide will cover only the basic required setup for the SNARE - AWS Cloud log collection to work, security related setup, charges you may incur and other intricacies related to AWS will not be covered in details on this guide.
Overview
In today's data-driven landscape, efficiently managing log data is imperative. AWS services, such as AWS CloudTrail, AWS Web Application Firewall (WAF), and AWS VPC Flow Logs, generate a wealth of log information during their operations. This services can be configured for their logs to be routed to AWS CloudWatch Logs, functioning as an initial repository.
However, the process doesn't stop there. AWS CloudWatch Logs can employ subscription filters to selectively forward or aggregate specific log data into AWS Kinesis Data Streams. These streams serve as dynamic conduits, ensuring real-time or near-real-time access to log data.
Enter Snare Central, a powerful log collection solution. Configured to periodically access AWS Kinesis Data Streams via the Kinesis Data Streams API, Snare Central automates the collection process, for a continuous flow of log data into its centralized repository and/or reflecting them to another Snare Central server, or to a third party SIEM server or collector.
Setup Supported AWS Services to Send Log to CloudWatch Logs
Amazon CloudWatch Logs enables you to centralize the logs from all of your systems, applications, and AWS services that you use. CloudWatch Logs enables you to see all of your logs, regardless of their source, as a single and consistent flow of events ordered by time, and you can query them and sort them based on other dimensions, group them by specific fields.
Currently Snare Central support logs from the following AWS Services:
AWS CloudTrail
AWS Web Application Firewall (WAF)
AWS VPC Flow Logs
For CloudTrail setup guide see: Sending Events to CloudWatch Logs - CloudTrail
For WAF setup guide see: Sending Events to CloudWatch Logs - WAF
For VPC Flow Logs setup guide see: Sending Events to CloudWatch Logs - VPC Flow Logs
Setting Up AWS Kinesis Data Stream
Amazon Kinesis Data Streams ingests a large amount of data in real time, durably stores the data, and makes the data available for consumption. The unit of data stored by Kinesis Data Streams is a data record. A data stream represents a group of data records. The data records in a data stream are distributed into shards.
A shard has a sequence of data records in a stream. It serves as a base throughput unit of a Kinesis data stream. A shard supports 1 MB/s and 1000 records per second for writes and 2 MB/s for reads in both on-demand and provisioned capacity modes
For more info see: Amazon Kinesis Data Streams
Also see: Amazon Kinesis Data Streams Pricing for more info on the possible charges you may incur.
Step 1. Sign in to the AWS Management Console and open the Kinesis console at Kinesis console - AWS Management Console.
Step 2. Choose Data Streams in the navigation pane.
Step 3. In the navigation bar, expand the Region selector and choose a Region.
Step 4. Click Create data stream.
Step 5. In Data stream name, enter a name for your stream ( for example snare ), then in Capacity mode select Provisioned and enter the number of shards that you will need in the Provisioned shards.
Step 6. Scroll down to the bottom and click Create data stream.
Step 7. Once you created the Kinesis Data Stream successfully, you should be able to see it in the list of Data streams.
For more info on how to create Kinesis Data Stream see: Create Amazon Kinesis Data Stream.
Setting Up AWS CloudWatch Logs Subscription Filter
You can use subscriptions to get access to a real-time feed of log events from CloudWatch Logs and have it delivered to other services such as an Amazon Kinesis Data Stream for custom processing, analysis, or loading to other systems.
A subscription filter defines the filter pattern to use for filtering which log events get delivered to your AWS resource, as well as information about where to send matching log events to.
Each log group can have up to two subscription filters associated with it.
When log events are sent to the receiving service, they are base64 encoded and compressed with the gzip format.
For more info see: Using CloudWatch Logs subscription filters.
Also see: Amazon CloudWatch Pricing for more info on the possible charges you may incur.
Step 1. Sign in to the AWS Management Console and open the CloudWatch console at CloudWatch console - AWS Management Console.
Step 2. Choose Log groups in the navigation pane.
Step 3. In the navigation bar, expand the Region selector and choose a Region.
Step 4. Select the Log group that you want logs to be streamed to Kinesis Data Stream and get collected by Snare Central’s AWS Log Collection.
Step 5. Select Subscription filters tab then click Create then select Create Kinesis subscription filter.
Step 6. Set Destination account and Kinesis data stream (in the list, select the name of the Kinesis data stream you previously setup).
Step 7. Set Grant permission. Click create a new role if you don’t have an existing role that grant CloudWatch Logs permission to put data into your Kinesis data stream or Select an existing role if you already have one.
Step 8. Set your desired Distribution method and Configure log format and filters.
Step 9. Optionally you can Test pattern if you set one. Then afterwards click Start streaming.
Step 10. Once the setup is successful, you should be able to see the created subscription filter in the list.
Setting Up Snare Central - Amazon Web Services(AWS) Cloud Log Collection
Starting from Snare Central v8.6.0, AWS Cloud Log Collection functionality will be available as long as you have the proper license for it. This guide will help you setup up your Snare Central and start collecting supported AWS logs in no-time by simply using the intuitive Cloud Log Collection Configuration Web UI of Snare Central.
For more info about the supported AWS Log types, see: Supported AWS Log Types.
Before you proceed, make sure that you are already done setting up the supported AWS Services to send logs to CloudWatch Logs, AWS Kinesis Data Stream and AWS CloudWatch Logs Subscription Filter.
Step 1. Go to Snare Central and navigate to System > Administrative Tools > Cloud Log Collection Configuration.
Step 2. Select Amazon Web Services and Click ADD CLOUD COLLECTION button.
Step 3. Input all the necessary AWS Cloud Collection Configuration Information and click Test Connection Button to check that the configuration correct and can properly connect to you created Kinesis Data Stream.
Name: Any name to easily identify this AWS Cloud Log Collector.
Enabled: Can be toggled ON/OFF. This will determine if the AWS Cloud Collector will be enabled and start log collection (This can also be toggled ON/OFF easily later after setup).
AWS Access Key ID: AWS Credential with permission to make programmatic calls/request to AWS API. see: Managing Access Keys for IAM users for more info.
AWS Secret Access Key: AWS Credential used to sign request to AWS API. see: Managing Access Keys for IAM users for more info.
AWS Region Code: Region code were you setup your AWS Kinesis Data Stream, e.g. us-east-1.
AWS Kinesis Data Stream Name: The Kinesis Data Stream Name you want to collect logs from. e.g. The name used in Setting Up AWS Kinesis Data Stream.
Polling Interval: Log collection interval (in millisecond) for each log collection request to specified AWS Kinesis Data Stream. (Actual request interval maybe greater that what is set, depending on the actual response time for each request).
Default Starting Position When Collecting Logs: This will be the default log collection starting position in AWS Kinesis Data Stream Specified when there is no valid sequence number yet. TRIM_HORIZON Start streaming at the last untrimmed record in the shard, which is the oldest data record in the shard. LATEST Start streaming just after the most recent record in the shard, so that you always read the most recent data in the shard. See API Starting Position for more info.
Note: Optional field that you may use to note any related information to this AWS Cloud Log Collector
Step 4. Click ADD button, then you should be able to see the added AWS Cloud Log Collector under the Amazon Web Services Cloud Collection List.
Updating/Deleting - AWS Cloud Log Collection Configuration
If you want to update or delete an existing Amazon Web Services - Cloud Log Collector that were previously configured, you can simply use the Snare Central’s Cloud Log Collection Configuration Web UI and follow the simple steps below.
Step 1. Go to Snare Central and navigate to System > Administrative Tools > Cloud Log Collection Configuration.
Step 2. Select Amazon Web Services and Click the AWS Log Collector that you want to update, then click the Edit icon on the right side.
Step 3. In the Edit screen, you can update the configuration and optionally do a Test Connection to check if the updated configuration can successfully connect with your AWS Kinesis Data Stream, then simply click the SAVE button to save the updated configuration.
Step 1. Go to Snare Central and navigate to System > Administrative Tools > Cloud Log Collection Configuration.
Step 2. Select Amazon Web Services and Click the AWS Log Collector that you want to delete, then click the Delete icon on the right side.
Troubleshooting Guide
This guide will be your resource for resolving common issues and challenges that you may encounter with Amazon Web Services (AWS) - Cloud Log Collection.
Amazon Web Services icon is gray in System > Administrative Tools > Cloud Log Collection Configuration Web UI.
When Amazon Web Services icon in Cloud Log Providers list is gray, it is possible that Snare Central license does not have IA_CLOUD or IA_CLOUD_AWS.
You can check it via navigating to Status > Snare Health Checker or simply click the heart icon the lower left corner of Snare Central and scroll down to Snare Central License and select Show Details to view the License Information.
If there is no IA_CLOUD or IA_CLOUD_AWS in the License Information, then you needed the correct license with IA_CLOUD or IA_CLOUD_AWS. Once you have the correct license, click License Page button.
In the License Update page, click Browse button and navigate to the correct license then click Load License button.
Wait for a while then navigate to System > Administrative Tools > Cloud Log Collection Configuration and you should be able to see Amazon Web Services icon is now green and you should be able to Add Cloud Collection.
AWS Cloud Log Collector icon is gray and the Status is Not Running (Disabled by configuration)
When your configured AWS Cloud Log Collector icon is gray, it is possible that the log collector is disabled during configuration or toggled off.
Select the AWS Cloud Log Collector and check if Status: Not Running (Disabled by configuration)
To enable AWS Cloud Log Collector, simply toggle on the Enable button besides beside the name in Cloud Log Providers or the one in the upper right corner besides the Edit and Delete icon. Then click Confirm in the pop-up dialog box.
Once toggled on, the configured AWS Cloud Log Collector icon should be green and enabled.
Snare Central will now start collecting AWS Logs.
AWS Cloud Log Collector icon is red and the Status is Not Running (message: The security token included in the request is invalid.)
When the AWS Cloud Log Collector icon is red and Status is Not Running (message: The security token included in the request is invalid), it is possible that the AWS Access Key ID is invalid or expired.
Go to AWS website and check if AWS Access Key ID is not yet expired and the value entered in the Snare Central configuration is correct.
If the value entered in the Snare Central Configuration is incorrect, you can simply edit it by clicking the Edit icon on the upper left corner. For more info, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection
AWS Cloud Log Collector icon is red and the Status is Not Running (message: The request signature we calculated does not match the signature you provided.)
When the AWS Cloud Log Collector icon is red and Status is Not Running (message: The request signature we calculated does not match the signature you provided), it is possible that the AWS Secret Access Key is invalid or expired.
Go to AWS website and check if AWS Secret Access Key is not yet expired and is valid.
If it is still valid and not yet expired, The value entered in the Snare Central Configuration maybe incorrect, you can simply edit it by clicking the Edit icon on the upper left corner. For more info, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection
AWS Cloud Log Collector icon is red and the Status is Not Running (message: Stream <streamname> under account <account number> not found.)
When the AWS Cloud Log Collector icon is red and Status is Not Running (message: Stream <streamname> under account <account number> not found), it is possible that the AWS Kinesis Data Stream Name you specified is not in the configured AWS Region Code or the AWS Kinesis Data Stream Name is wrong/does not exist.
Go to AWS website and check if the AWS Kinesis Data Stream Name exist in the AWS Region Code you specified.
If it exist in the specified AWS Region Code then the value entered in the Snare Central Configuration maybe incorrect. Double check the AWS Region Code entry and the AWS Kinesis Data Stream Name
Modify the wrong entry by simply clicking the Edit icon on the upper left corner. For more info, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection
AWS Cloud Log Collector did not collect the old logs in AWS Kinesis Data Stream.
When AWS Cloud Log Collector is not collecting the old logs in AWS Kinesis Data Stream, it is possible that the Default Starting Position When Collecting Logs is configured to LATEST or that the old logs were already expired based from the set retention period in AWS Kinesis Data Stream.
Go to AWS website and check if the old logs still exist in the AWS Kinesis Data Stream.
If it still exist, then check the configured Default Starting Position When Collecting Logs if it is set to LATEST.
If it is LATEST, then change it to TRIM_HORIZON to start collecting from the oldest log in the AWS Kinesis Data Stream. You can simply edit it by clicking the Edit icon on the upper left corner. For more info, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection