Versions Compared

Old Version 3

changes.mady.by.user Richard Siman

Saved on

compared with

New Version Current

changes.mady.by.user Richard Siman

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Amazon Web Services (AWS) stands as a prominent cloud provider, offering an array of services that generate valuable log data crucial for monitoring, security, and compliance in modern IT ecosystems. This guide will assist you in configuring Snare Central for the task of collecting and processing logs via the AWS Kinesis Data Stream.

Note

  • This setup guide will cover only the basic required setup for the SNARE - AWS Cloud log collection to work, security related setup, charges you may incur and other intricacies related to AWS will not be covered

...

  • on this guide.

  • Please refer to official AWS documentation for detailed information related to AWS.

Overview

In today's data-driven landscape, efficiently managing log data is imperative. AWS services, such as AWS CloudTrail, AWS Web Application Firewall (WAF), and AWS VPC Flow Logs, generate a wealth of log information during their operations. This services can be configured for their logs to be routed to AWS CloudWatch Logs, functioning as an initial repository.

...

Amazon CloudWatch Logs enables you to centralize the logs from all of your systems, applications, and AWS services that you use. CloudWatch Logs enables you to see all of your logs, regardless of their source, as a single and consistent flow of events ordered by time, and you can query them and sort them based on other dimensions, group them by specific fields.

Currently Snare Central support logs from the following AWS Services:

  • AWS CloudTrail

  • AWS Web Application Firewall (WAF)

  • AWS VPC Flow Logs

...

Note

  • For CloudTrail setup guide

...

...

...

Setting Up AWS Kinesis Data Stream

Amazon Kinesis Data Streams ingests a large amount of data in real time, durably stores the data, and makes the data available for consumption. The unit of data stored by Kinesis Data Streams is a data record. A data stream represents a group of data records. The data records in a data stream are distributed into shards.

...

Note

  • A shard has a sequence of data records in a stream. It serves as a base throughput unit of a Kinesis data stream. A shard supports 1 MB/s and 1000 records per second for writes and 2 MB/s for reads in both on-demand and provisioned capacity modes

  • For more

...

...

...

  • information on the possible charges you may incur.

For more info image-20241122-093326.pngImage Added

Step 8. Click Data Streams in the navigation pane, then you should be able to see it in the list of Data streams with an Active status.

image-20241122-094341.pngImage Added

Expand
titleStep by Step Guide for Setting Up AWS Kinesis Data Stream
Step 1. Sign in to the AWS Management Console and open the Kinesis console at Kinesis console - AWS Management Console
Info

Before proceeding, ensure that you have set up one or more supported AWS services to send logs to CloudWatch Logs. If this has not been completed yet, please finish this step first. The necessary information can be found here: Setup Supported AWS Services to Send Log to CloudWatch Logs

Step 1. Sign in to the AWS Management Console and open the Kinesis console at Kinesis console - AWS Management Console.

Step 2. Choose Click Data Streams in the navigation pane.

Image Removedimage-20241122-083923.pngImage Added

Step 3. In the navigation bar, expand the Region selector and choose a the appropriate Region.

Image Removedimage-20241122-084748.pngImage Added

Step 4. Click Create data stream.

Image Removedimage-20241122-085454.pngImage Added

Step 5. In Data stream name, enter a name for your stream ( for example e.g. snare), then in Capacity mode select Provisioned and enter the number of shards that you will need in the Provisioned shards .

Image Removed

(e.g. 1).

Info

You may opt to use On-demand in the Capacity mode, depending on your use case.

image-20241122-090056.pngImage Added

Step 6. Scroll down to the bottom and click Create data stream.

Image Removedimage-20241122-092426.pngImage Added

Step 7. Once you created the Kinesis Data Stream successfully, you should be able to see it in the list of Data streamsstream summary.

Image Removed
Info
Info

For updated and more information on how to create Kinesis Data Stream see, please refer to AWS official documentation: Create Amazon Kinesis Data Stream.

...

You can use subscriptions to get access to a real-time feed of log events from CloudWatch Logs and have it delivered to other services such as an Amazon Kinesis Data Stream for custom processing, analysis, or loading to other systems.

...

Note

  • A subscription filter defines the filter pattern to use for filtering which log events get delivered to your AWS resource, as well as information about where to send matching log events to.

  • Each log group can have up to two subscription filters associated with it.

  • When log events are sent to the receiving service, they are base64 encoded and compressed with the gzip format.

  • For more

...

...

...

  • information on the possible charges you may incur.

Expand
titleStep by Step Guide for Setting Up AWS CloudWatch Logs Subscription Filter
Info

Before proceeding, ensure that you have set up AWS Kinesis Data Streams. If this has not been completed yet, please finish this step first. The necessary information can be found here: Setting Up AWS Kinesis Data Stream

Step 1. Sign in to the AWS Management Console and open the CloudWatch console at CloudWatch console - AWS Management Console.

Step 2. Choose Click Log groups in the navigation pane.

Image Removedimage-20241125-103606.pngImage Added

Step 3. In the navigation bar, expand the Region selector and choose a the appropriate Region.

Image Removedimage-20241125-104235.pngImage Added

Step 4. Select Click the Log group that you want logs to be streamed to Kinesis Data Stream and get collected by Snare Central’s AWS Log Collection e.g.

Image Removed

aws-waf-logs-sampleLogGroup.

image-20241125-104658.pngImage Added

Step 5. Select Click Subscription filters tab then click Create then select Create Kinesis subscription filter.

Image Removedimage-20241125-105054.pngImage Added

Step 6. Set Destination account and Kinesis data stream (in the list, enter or select the name of the Kinesis data stream you previously setup).

Image Removedimage-20241125-110014.pngImage Added

Step 7. Set Grant permission. Click create a new role if you don’t have an existing role that grant CloudWatch Logs permission to put data into your Kinesis data stream or Select an existing role if you already have one.

Image Removedimage-20241125-110346.pngImage Added

Step 8. Set your desired Distribution method and Configure log format and filters.

Image Removedimage-20241125-110730.pngImage Added

Step 9. Optionally you can Test pattern if you set one. Then afterwards click Start streaming.

Image Removedimage-20241125-111441.pngImage Added

Step 10. Once the setup is successful, you should be able to see the created subscription filter in the list, this log group will stream the log data to your Kinesis data stream.

Image Removed

Setting Up Snare Central - Amazon Web Services(AWS) Cloud Log Collection

Starting from Snare Central v8.6.0, AWS Cloud Log Collection functionality will be available as long as you have the proper license for it. This guide will help you setup up your Snare Central and start collecting supported AWS logs in no-time by simply using the intuitive Cloud Log Collection Configuration Web UI of Snare Central.

...

For more info about the supported AWS Log types, see: Supported AWS Log Types.

...

image-20241125-112712.pngImage Added

Info

To setup more log groups that you want logs to get collected by Snare Central’s AWS Log Collection, repeat the steps in this section and setup the additional log groups that has no subscription filter yet that streams in the previously created Kinesis Data Stream.

Setting Up Snare Central - Amazon Web Services(AWS) Cloud Log Collection

Starting from Snare Central v8.6.0, AWS Cloud Log Collection functionality will be available as long as you have the proper license for it. This guide will help you setup up your Snare Central and start collecting supported AWS logs in no-time by simply using the intuitive Cloud Log Collection Configuration Web UI of Snare Central.

Note

  • For more information about the supported AWS Log types, see: Supported AWS Log Types.

  • Before you proceed, make sure that you are already done setting up the supported AWS Services to send logs to CloudWatch Logs, AWS Kinesis Data Stream and AWS CloudWatch Logs Subscription Filter.

Expand
titleStep by Step Guide for Setting Up Snare Central - Amazon Web Services (AWS) Cloud Log Collection

Step 1. Go to Snare Central and navigate to System > Administrative Tools > Cloud Log Collection Configuration.

Step 2. Select Amazon Web Services and Click ADD CLOUD COLLECTION button.

Step 3. Input all the necessary AWS Cloud Collection Configuration Information and click “Test Connection” Button to check that if the configuration correct and can properly connect to you your previously created Kinesis Data Stream.

  • Name: Any name to easily identify this AWS Cloud Log Collector.

  • Enabled: Can be toggled ON/OFF. This will determine if the AWS Cloud Collector will be enabled and start log collection (This can also be toggled ON/OFF easily later after setup).

  • AWS Access Key ID: AWS Credential with permission to make programmatic calls/request to AWS API. see: Managing Access Keys for IAM users for more infoinformation.

  • AWS Secret Access Key: AWS Credential used to sign request to AWS API. see: Managing Access Keys for IAM users for more infoinformation.

  • AWS Region Code: Region code were you setup your AWS Kinesis Data Stream, e.g. us-east-1.

  • AWS Kinesis Data Stream Name: The Kinesis Data Stream Name you want to collect logs from. e.g. snare. (The name used in Setting Up AWS Kinesis Data Stream.)

  • Polling Interval: Log collection interval (in millisecond) for each log collection request to specified AWS Kinesis Data Stream. (Actual request interval maybe greater that than what is set, depending on the actual response time for each request).

  • Default Starting Position When Collecting Logs: This will be the default log collection starting position in AWS Kinesis Data Stream Specified when there is no valid sequence number yet.

    • TRIM_HORIZON Start streaming at the last untrimmed record in the shard, which is the oldest data record in the shard.

    • LATEST Start streaming just after the most recent record in the shard

    , so that you always read the most recent data in the shard info

    • information.

  • Note: Optional field that you may use to note any related information to this AWS Cloud Log Collector

Step 4. Click ADD button, then you should be able to see the added AWS Cloud Log Collector under the Amazon Web Services Cloud Collection List.

...

  1. AWS Cloud Log Collector icon is red and the Status is Not Running (message: The security token included in the request is invalid.)

Expand
titlePossible Cause and Resolution

When the AWS Cloud Log Collector icon is red and Status is Not Running (message: The security token included in the request is invalid), it is possible that the AWS Access Key ID is invalid or expired.

Image RemovedImage Added

Go to AWS website and check if AWS Access Key ID is not yet expired and the value entered in the Snare Central configuration is correct.

Image RemovedImage Added

If the value entered in the Snare Central Configuration is incorrect, you can simply edit it by clicking the Edit icon on the upper left corner. For more infoinformation, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection

  1. AWS Cloud Log Collector icon is red and the Status is Not Running (message: The request signature we calculated does not match the signature you provided.)

Expand
titlePossible Cause and Resolution

When the AWS Cloud Log Collector icon is red and Status is Not Running (message: The request signature we calculated does not match the signature you provided), it is possible that the AWS Secret Access Key is invalid or expired.

Image RemovedImage Added

Go to AWS website and check if AWS Secret Access Key is not yet expired and is valid.

If it is still valid and not yet expired, The value entered in the Snare Central Configuration maybe incorrect, you can simply edit it by clicking the Edit icon on the upper left corner. For more infoinformation, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection

  1. AWS Cloud Log Collector icon is red and the Status is Not Running (message: Stream <streamname> under account <account number> not found.Post "https://kinesis.us-east-11.amazonaws.com": dial tcp: lookup kinesis.us-east-11.amazonaws.com: no such host)

Expand
titlePossible Cause and Resolution

When the AWS Cloud Log Collector icon is red and Status is Not Running (message: Stream <streamname> under account <account number> not foundPost "https://kinesis.us-east-11.amazonaws.com": dial tcp: lookup kinesis.us-east-11.amazonaws.com: no such host), it is possible that the AWS Kinesis Data Stream Name you specified is not in the configured AWS Region Code or the AWS Kinesis Data Stream Name is wrong/is invalid or does not exist.

Image RemovedImage Added

Go to AWS website and check if the AWS Kinesis Data Stream Name exist in the AWS Region Code you specified.

If it exist in the specified AWS Region Code then the value entered in the Snare Central Configuration maybe incorrect. Double check the AWS Region Code entry and the AWS Kinesis Data Stream Name

Image Removed

Modify the wrong entry AWS Region Code exist/valid and the value entered in the Snare Central configuration is correct.

Image Added

If the value entered in the Snare Central Configuration is incorrect, you can simply edit it by clicking the Edit icon on the upper left corner. For more information, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection

  1. AWS Cloud Log Collector icon is red and the Status is Not Running (Stream <streamname> under account <account number> not found.)

Expand
titlePossible Cause and Resolution

When the AWS Cloud Log Collector icon is red and Status is Not Running (Stream <streamname> under account <account number> not found), it is possible that the AWS Kinesis Data Stream Name you specified is not in the configured AWS Region Code or the AWS Kinesis Data Stream Name is wrong/does not exist.

Image Added

Go to AWS website and check if the AWS Kinesis Data Stream Name exist in the AWS Region Code you specified.

If it exist in the specified AWS Region Code then the value entered in the Snare Central Configuration maybe incorrect. Double check the AWS Region Code entry and the AWS Kinesis Data Stream Name

Image Added

Modify the wrong entry by simply clicking the Edit icon on the upper left corner. For more information, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection

  1. AWS Cloud Log Collector takes too long to get new logs.

Expand
titlePossible Cause and Resolution

When AWS Cloud Log Collector takes too long to get new logs, it is possible that the Polling Interval(ms) is set too high.

Image Added

Modify the entry to the desired interval for getting new logs in millisecond by simply clicking the Edit icon on the upper left corner. For more infoinformation, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection

  1. AWS Cloud Log Collector did not collect the old logs in AWS Kinesis Data Stream.

Expand
titlePossible Cause and Resolution

When AWS Cloud Log Collector is not collecting the old logs in AWS Kinesis Data Stream, it is possible that the Default Starting Position When Collecting Logs is configured to LATEST or that the old logs were already expired based from the set retention period in AWS Kinesis Data Stream.

Go to AWS website and check if the old logs still exist in the AWS Kinesis Data Stream.

If it still exist, then check the configured Default Starting Position When Collecting Logs if it is set to LATEST.

If it is LATEST, then change it to TRIM_HORIZON to start collecting from the oldest log in the AWS Kinesis Data Stream. You can simply edit it by clicking the Edit icon on the upper left corner. For more infoinformation, see: Step by Step Guide for Updating Snare Central - Amazon Web Services (AWS) Cloud Log Collection

...