Introduction
Amazon Web Services (AWS) stands as a prominent cloud provider, offering an array of services that generate valuable log data crucial for monitoring, security, and compliance in modern IT ecosystems. This guide will assist you in configuring Snare Central for the task of collecting and processing logs via the AWS Kinesis Data Stream.
This setup guide will cover only the basic required setup for the SNARE - AWS Cloud log collection to work, security related setup, charges you may incur and other intricacies related to AWS will not be covered in details on this guide.
Overview
In today's data-driven landscape, efficiently managing log data is imperative. AWS services, such as AWS CloudTrail, AWS Web Application Firewall (WAF), and AWS VPC Flow Logs, generate a wealth of log information during their operations. This services can be configured for their logs to be routed to AWS CloudWatch Logs, functioning as an initial repository.
However, the process doesn't stop there. AWS CloudWatch Logs can employs subscription filters to selectively forward or aggregate specific log data into AWS Kinesis Data Streams. These streams serve as dynamic conduits, ensuring real-time or near-real-time access to log data.
Enter Snare Central, a powerful log collection solution. Configured to periodically access AWS Kinesis Data Streams via the Kinesis Data Streams API, Snare Central automates the collection process, for a continuous flow of log data into its centralized repository and/or reflecting them to another Snare Central server, or to a third party SIEM server or collector.
Setup Supported AWS Services to Send Log to CloudWatch Logs
Amazon CloudWatch Logs enables you to centralize the logs from all of your systems, applications, and AWS services that you use. CloudWatch Logs enables you to see all of your logs, regardless of their source, as a single and consistent flow of events ordered by time, and you can query them and sort them based on other dimensions, group them by specific fields.
Currently Snare Central support logs from the following AWS Services:
AWS CloudTrail
AWS Web Application Firewall (WAF)
AWS VPC Flow Logs
For CloudTrail setup guide see: Sending Events to CloudWatch Logs - CloudTrail
For WAF setup guide see: Sending Events to CloudWatch Logs - WAF
For VPC Flow Logs setup guide see: Sending Events to CloudWatch Logs - VPC Flow Logs
Setting Up AWS Kinesis Data Stream
Amazon Kinesis Data Streams ingests a large amount of data in real time, durably stores the data, and makes the data available for consumption. The unit of data stored by Kinesis Data Streams is a data record. A data stream represents a group of data records. The data records in a data stream are distributed into shards.
A shard has a sequence of data records in a stream. It serves as a base throughput unit of a Kinesis data stream. A shard supports 1 MB/s and 1000 records per second for writes and 2 MB/s for reads in both on-demand and provisioned capacity modes
For more info see: Amazon Kinesis Data Streams
Also see: Amazon Kinesis Data Streams Pricing for more info on the possible charges you may incur.
Step 1. Sign in to the AWS Management Console and open the Kinesis console at Kinesis console - AWS Management Console.
Step 2. Choose Data Streams in the navigation pane.
Step 3. In the navigation bar, expand the Region selector and choose a Region.
Step 4. Click Create data stream.
Step 5. In Data stream name, enter a name for your stream ( for example snare ), then in Capacity mode select Provisioned and enter the number of shards that you will need in the Provisioned shards.
Step 6. Scroll down to the bottom and click Create data stream.
Step 7. Once you created the Kinesis Data Stream successfully, you should be able to see it in the list of Data streams.
For more info on how to create Kinesis Data Stream see: Create Amazon Kinesis Data Stream.
Setting Up AWS CloudWatch Logs Subscription Filter
You can use subscriptions to get access to a real-time feed of log events from CloudWatch Logs and have it delivered to other services such as an Amazon Kinesis Data Stream for custom processing, analysis, or loading to other systems.
A subscription filter defines the filter pattern to use for filtering which log events get delivered to your AWS resource, as well as information about where to send matching log events to.
Each log group can have up to two subscription filters associated with it.
When log events are sent to the receiving service, they are base64 encoded and compressed with the gzip format.
For more info see: Using CloudWatch Logs subscription filters.
Also see: Amazon CloudWatch Pricing for more info on the possible charges you may incur.
Step 1. Sign in to the AWS Management Console and open the CloudWatch console at CloudWatch console - AWS Management Console.
Step 2. Choose Log groups in the navigation pane.
Step 3. In the navigation bar, expand the Region selector and choose a Region.
Step 4. Select the Log group that you want logs to be streamed to Kinesis Data Stream and get collected by Snare Central’s AWS Log Collection.
Step 5. Select Subscription filters tab then click Create then select Create Kinesis subscription filter.
Step 6. Set Destination account and Kinesis data stream (in the list, select the name of the Kinesis data stream you previously setup).
Step 7. Set Grant permission. Click create a new role if you don’t have an existing role that grant CloudWatch Logs permission to put data into your Kinesis data stream or Select an existing role if you already have one.
Step 8. Set your desired Distribution method and Configure log format and filters.
Step 9. Optionally you can Test pattern if you set one. Then afterwards click Start streaming.
Step 10. Once the setup is successful, you should be able to see the created subscription filter in the list.
Setting Up Snare Central - Amazon Web Services(AWS) Cloud Log Collection
Starting from Snare Central v8.6.0, AWS Cloud Log Collection functionality will be available as long as you have the proper license for it. This guide will help you setup up your Snare Central and start collecting supported AWS logs in no-time by simply using the intuitive Cloud Log Collection Configuration Web UI of Snare Central.
For more info about the supported AWS Log types, see: Supported AWS Log Types.
Before you proceed, make sure that you are already done setting up the supported AWS Services to send logs to CloudWatch Logs, AWS Kinesis Data Stream and AWS CloudWatch Logs Subscription Filter.
Step 1. Go to Snare Central and navigate to System > Administrative Tools > Cloud Log Collection Configuration.
Step 2. Select Amazon Web Services and Click ADD CLOUD COLLECTION button.
Step 3. Input all the necessary AWS Cloud Collection Configuration Information and click Test Connection Button to check that the configuration correct and can properly connect to you created Kinesis Data Stream.
Name: Any name to easily identify this AWS Cloud Log Collector.
Enabled: Can be toggled ON/OFF. This will determine if the AWS Cloud Collector will be enabled and start log collection (This can also be toggled ON/OFF easily later after setup).
AWS Access Key ID: AWS Credential with permission to make programmatic calls/request to AWS API. see: Managing Access Keys for IAM users for more info.
AWS Secret Access Key: AWS Credential used to sign request to AWS API. see: Managing Access Keys for IAM users for more info.
AWS Region Code: Region code were you setup your AWS Kinesis Data Stream, e.g. us-east-1.
AWS Kinesis Data Stream Name: The Kinesis Data Stream Name you want to collect logs from. e.g. The name used in Setting Up AWS Kinesis Data Stream.
Polling Interval: Log collection interval (in millisecond) for each log collection request to specified AWS Kinesis Data Stream. (Actual request interval maybe greater that what is set, depending on the actual response time for each request).
Default Starting Position When Collecting Logs: This will be the default log collection starting position in AWS Kinesis Data Stream Specified when there is no valid sequence number yet. TRIM_HORIZON Start streaming at the last untrimmed record in the shard, which is the oldest data record in the shard. LATEST Start streaming just after the most recent record in the shard, so that you always read the most recent data in the shard. See API Starting Position for more info.
Note: Optional field that you may use to note any related information to this AWS Cloud Log Collector
Step 4. Click ADD button, then you should be able to see the added AWS Cloud Log Collector under the Amazon Web Services Cloud Collection List.
Updating/Deleting - AWS Cloud Log Collection Configuration
If you want to update or delete an existing Amazon Web Services - Cloud Log Collector that were previously configured, you can simply use the Snare Central’s Cloud Log Collection Configuration Web UI and follow the simple steps below.
Step 1. Go to Snare Central and navigate to System > Administrative Tools > Cloud Log Collection Configuration.
Step 2. Select Amazon Web Services and Click the AWS Log Collector that you want to update, then click the Edit icon on the right side.
Step 3. In the Edit screen, you can update the configuration and optionally do a Test Connection to check if the updated configuration can successfully connect with your AWS Kinesis Data Stream, then simply click the SAVE button to save the updated configuration.
Step 1. Go to Snare Central and navigate to System > Administrative Tools > Cloud Log Collection Configuration.
Step 2. Select Amazon Web Services and Click the AWS Log Collector that you want to delete, then click the Delete icon on the right side.
