| Table of Contents | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
Introduction
Amazon Web Services (AWS) stands as a prominent cloud provider, offering an array of services that generate valuable log data crucial for monitoring, security, and compliance in modern IT ecosystems. This guide will assist you in configuring Snare Central for the task of collecting and processing logs via the AWS Kinesis Data Stream.
Note
This setup guide will cover only the basic required setup for the SNARE - AWS Cloud log collection to work, security related setup, charges you may incur and other intricacies related to AWS will not be covered
...
on this guide.
Please refer to official AWS documentation for detailed information related to AWS.
Overview
In today's data-driven landscape, efficiently managing log data is imperative. AWS services, such as AWS CloudTrail, AWS Web Application Firewall (WAF), and AWS VPC Flow Logs, generate a wealth of log information during their operations. This services can be configured for their logs to be routed to AWS CloudWatch Logs, functioning as an initial repository.
However, the process doesn't stop there. AWS CloudWatch Logs can employs employ subscription filters to selectively forward or aggregate specific log data into AWS Kinesis Data Streams. These streams serve as dynamic conduits, ensuring real-time or near-real-time access to log data.
...
Amazon CloudWatch Logs enables you to centralize the logs from all of your systems, applications, and AWS services that you use. CloudWatch Logs enables you to see all of your logs, regardless of their source, as a single and consistent flow of events ordered by time, and you can query them and sort them based on other dimensions, group them by specific fields.
Currently Snare Central support logs from the following AWS Services:
AWS CloudTrail
AWS Web Application Firewall (WAF)
AWS VPC Flow Logs
...
Note
For CloudTrail setup guide
...
, please refer to AWS official documentation: Sending Events to CloudWatch Logs - CloudTrail
For WAF setup guide
...
, please refer to AWS official documentation: Sending Events to CloudWatch Logs - WAF
For VPC Flow Logs setup guide
...
, please refer to AWS official documentation: Sending Events to CloudWatch Logs - VPC Flow Logs
Setting Up AWS Kinesis Data Stream
Amazon Kinesis Data Streams ingests a large amount of data in real time, durably stores the data, and makes the data available for consumption. The unit of data stored by Kinesis Data Streams is a data record. A data stream represents a group of data records. The data records in a data stream are distributed into shards.
...
Note
A shard has a sequence of data records in a stream. It serves as a base throughput unit of a Kinesis data stream. A shard supports 1 MB/s and 1000 records per second for writes and 2 MB/s for reads in both on-demand and provisioned capacity modes
For more
...
information, please refer to AWS official documentation: Amazon Kinesis Data Streams
Also
...
refer to: Amazon Kinesis Data Streams Pricing for more
...
information on the possible charges you may incur.
| Expand | ||||||
|---|---|---|---|---|---|---|
| ||||||
Step 1. Sign in to the AWS Management Console and open the Kinesis console at Kinesis console - AWS Management Console. Step 2. Choose Click Data Streams in the navigation pane.   Step 3. In the navigation bar, expand the Region selector and choose a the appropriate Region.   Step 4. Click Create data stream.   Step 5. In Data stream name, enter a name for your stream ( for example e.g. snare), then in Capacity mode select Provisioned and enter the number of shards that you will need in the Provisioned shards (e.g. 1).
  Step 6. Scroll down to the bottom and click Create data stream.   Step 7. Once you created the Kinesis Data Stream successfully, you should be able to see the Data stream summary.  Step 8. Click Data Streams in the navigation pane, then you should be able to see it in the list of Data streams with an Active status.  
|
...
You can use subscriptions to get access to a real-time feed of log events from CloudWatch Logs and have it delivered to other services such as an Amazon Kinesis Data Stream for custom processing, analysis, or loading to other systems.
...
Note
A subscription filter defines the filter pattern to use for filtering which log events get delivered to your AWS resource, as well as information about where to send matching log events to.
Each log group can have up to two subscription filters associated with it.
When log events are sent to the receiving service, they are base64 encoded and compressed with the gzip format.
For more
...
information, please refer to AWS official documentation: Using CloudWatch Logs subscription filters.
Also
...
refer to: Amazon CloudWatch Pricing for more
...
information on the possible charges you may incur.
| Expand | ||||
|---|---|---|---|---|
| ||||
Step 1. Sign in to the AWS Management Console and open the CloudWatch console at CloudWatch console - AWS Management Console. Step 2. Choose Click Log groups in the navigation pane.   Step 3. In the navigation bar, expand the Region selector and choose a the appropriate Region.   Step 4. Select Click the Log group that you want logs to be streamed to Kinesis Data Stream and get collected by Snare Central’s AWS Log Collection e.g.  aws-waf-logs-sampleLogGroup.  Step 5. Select Click Subscription filters tab then click Create then select Create Kinesis subscription filter.   Step 6. Set Destination account and Kinesis data stream (in the list, enter or select the name of the Kinesis data stream you previously setup).   Step 7. Set Grant permission. Click create a new role if you don’t have an existing role that grant CloudWatch Logs permission to put data into your Kinesis data stream or Select an existing role if you already have one.   Step 8. Set your desired Distribution method and Configure log format and filters.   Step 9. Optionally you can Test pattern if you set one. Then afterwards click Start streaming.   Step 10. Once the setup is successful, you should be able to see the created subscription filter in the list, this log group will stream the log data to your Kinesis data stream.  
|
Setting Up Snare Central - Amazon Web Services(AWS) Cloud Log Collection
Starting from Snare Central v8.6.0, AWS Cloud Log Collection functionality will be available as long as you have the proper license for it. This guide will help you setup up your Snare Central and start collecting supported AWS logs in no-time by simply using the intuitive Cloud Log Collection Configuration Web UI of Snare Central.
...
Note
For more
...
information about the supported AWS Log types, see: Supported AWS Log Types.
Before you proceed, make sure that you are already done setting up the supported AWS Services to send logs to CloudWatch Logs, AWS Kinesis Data Stream and AWS CloudWatch Logs Subscription Filter.
| Expand | ||
|---|---|---|
| ||
Step 1. Go to Snare Central and navigate to System > Administrative Tools > Cloud Log Collection Configuration.  Step 2. Select Amazon Web Services and Click ADD CLOUD COLLECTION button.  Step 3. Input all the necessary AWS Cloud Collection Configuration Information and click “Test Connection” Button to check that if the configuration correct and can properly connect to you your previously created Kinesis Data Stream.
 Step 4. Click ADD button, then you should be able to see the added AWS Cloud Log Collector under the Amazon Web Services Cloud Collection List.  |
...
| Expand | ||
|---|---|---|
| ||
Step 1. Go to Snare Central and navigate to System > Administrative Tools > Cloud Log Collection Configuration. Step 2. Select Amazon Web Services and Click the AWS Log Collector that you want to delete, then click the Delete icon on the right side.  |
Troubleshooting Guide
This guide will be your resource for resolving common issues and challenges that you may encounter with Amazon Web Services (AWS) - Cloud Log Collection.
Amazon Web Services icon is gray in System > Administrative Tools > Cloud Log Collection Configuration Web UI.
| Expand | ||
|---|---|---|
| ||
When Amazon Web Services icon in Cloud Log Providers list is gray, it is possible that Snare Central license does not have IA_CLOUD or IA_CLOUD_AWS.  You can check it via navigating to Status > Snare Health Checker or simply click the heart icon in the lower left corner of Snare Central and scroll down to Snare Central License and select Show Details to view the License Information.  If there is no IA_CLOUD or IA_CLOUD_AWS in the License Information, then you needed the correct license with IA_CLOUD or IA_CLOUD_AWS. Once you have the correct license, click License Page button.  In the License Update page, click Browse button and navigate to the correct license then click Load License button.  Wait for a while then navigate to System > Administrative Tools > Cloud Log Collection Configuration and you should be able to see Amazon Web Services icon is now green and you should be able to Add AWS Cloud Collection.  |
AWS Cloud Log Collector icon is gray and the Status is Not Running (Disabled by configuration)
| Expand | ||
|---|---|---|
| ||
When your configured AWS Cloud Log Collector icon is gray, it is possible that the log collector is disabled during configuration or toggled off.  Select the AWS Cloud Log Collector and check if Status: Not Running (Disabled by configuration)  To enable AWS Cloud Log Collector, simply toggle on the Enable button beside its name in Cloud Log Providers or the one in the upper right corner beside the Edit icon. Then click Confirm in the pop-up dialog box.  Once toggled on, the configured AWS Cloud Log Collector icon should be green and enabled. Snare Central will now start collecting AWS Logs. |
AWS Cloud Log Collector icon is red and the Status is Not Running (The security token included in the request is invalid.)
| Expand | ||
|---|---|---|
| ||
When the AWS Cloud Log Collector icon is red and Status is Not Running (The security token included in the request is invalid), it is possible that the AWS Access Key ID is invalid or expired.  Go to AWS website and check if AWS Access Key ID is not yet expired and the value entered in the Snare Central configuration is correct.  If the value entered in the Snare Central Configuration is incorrect, you can simply edit it by clicking the Edit icon on the upper left corner. For more information, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection |
AWS Cloud Log Collector icon is red and the Status is Not Running (The request signature we calculated does not match the signature you provided.)
| Expand | ||
|---|---|---|
| ||
When the AWS Cloud Log Collector icon is red and Status is Not Running (The request signature we calculated does not match the signature you provided), it is possible that the AWS Secret Access Key is invalid or expired.  Go to AWS website and check if AWS Secret Access Key is not yet expired and is valid. If it is still valid and not yet expired, The value entered in the Snare Central Configuration maybe incorrect, you can simply edit it by clicking the Edit icon on the upper left corner. For more information, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection |
AWS Cloud Log Collector icon is red and the Status is Not Running (Post "https://kinesis.us-east-11.amazonaws.com": dial tcp: lookup kinesis.us-east-11.amazonaws.com: no such host)
| Expand | ||
|---|---|---|
| ||
When the AWS Cloud Log Collector icon is red and Status is Not Running (Post "https://kinesis.us-east-11.amazonaws.com": dial tcp: lookup kinesis.us-east-11.amazonaws.com: no such host), it is possible that the AWS Region Code is invalid or does not exist.  Go to AWS website and check if AWS Region Code exist/valid and the value entered in the Snare Central configuration is correct.  If the value entered in the Snare Central Configuration is incorrect, you can simply edit it by clicking the Edit icon on the upper left corner. For more information, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection |
AWS Cloud Log Collector icon is red and the Status is Not Running (Stream <streamname> under account <account number> not found.)
| Expand | ||
|---|---|---|
| ||
When the AWS Cloud Log Collector icon is red and Status is Not Running (Stream <streamname> under account <account number> not found), it is possible that the AWS Kinesis Data Stream Name you specified is not in the configured AWS Region Code or the AWS Kinesis Data Stream Name is wrong/does not exist.  Go to AWS website and check if the AWS Kinesis Data Stream Name exist in the AWS Region Code you specified. If it exist in the specified AWS Region Code then the value entered in the Snare Central Configuration maybe incorrect. Double check the AWS Region Code entry and the AWS Kinesis Data Stream Name  Modify the wrong entry by simply clicking the Edit icon on the upper left corner. For more information, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection |
AWS Cloud Log Collector takes too long to get new logs.
| Expand | ||
|---|---|---|
| ||
When AWS Cloud Log Collector takes too long to get new logs, it is possible that the Polling Interval(ms) is set too high.  Modify the entry to the desired interval for getting new logs in millisecond by simply clicking the Edit icon on the upper left corner. For more information, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection |
AWS Cloud Log Collector did not collect the old logs in AWS Kinesis Data Stream.
| Expand | ||
|---|---|---|
| ||
When AWS Cloud Log Collector is not collecting the old logs in AWS Kinesis Data Stream, it is possible that the Default Starting Position When Collecting Logs is configured to LATEST or that the old logs were already expired based from the set retention period in AWS Kinesis Data Stream. Go to AWS website and check if the old logs still exist in the AWS Kinesis Data Stream. If it still exist, then check the configured Default Starting Position When Collecting Logs if it is set to LATEST.  If it is LATEST, then change it to TRIM_HORIZON to start collecting from the oldest log in the AWS Kinesis Data Stream. You can simply edit it by clicking the Edit icon on the upper left corner. For more information, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection |