Amazon Web Services (AWS) - Cloud Log Collection Configuration

Owned by Richard Siman
Last updated: Nov 26, 2024
15 min read

 

 

Introduction

Amazon Web Services (AWS) stands as a prominent cloud provider, offering an array of services that generate valuable log data crucial for monitoring, security, and compliance in modern IT ecosystems. This guide will assist you in configuring Snare Central for the task of collecting and processing logs via the AWS Kinesis Data Stream.

Note

  • This setup guide will cover only the basic required setup for the SNARE - AWS Cloud log collection to work, security related setup, charges you may incur and other intricacies related to AWS will not be covered on this guide.

  • Please refer to official AWS documentation for detailed information related to AWS.

 

Overview

In today's data-driven landscape, efficiently managing log data is imperative. AWS services, such as AWS CloudTrail, AWS Web Application Firewall (WAF), and AWS VPC Flow Logs, generate a wealth of log information during their operations. This services can be configured for their logs to be routed to AWS CloudWatch Logs, functioning as an initial repository.

However, the process doesn't stop there. AWS CloudWatch Logs can employ subscription filters to selectively forward or aggregate specific log data into AWS Kinesis Data Streams. These streams serve as dynamic conduits, ensuring real-time or near-real-time access to log data.

Enter Snare Central, a powerful log collection solution. Configured to periodically access AWS Kinesis Data Streams via the Kinesis Data Streams API, Snare Central automates the collection process, for a continuous flow of log data into its centralized repository and/or reflecting them to another Snare Central server, or to a third party SIEM server or collector.

image-20241125-092739.png
AWS Cloud Log Collection Overview

 

 

Setup Supported AWS Services to Send Log to CloudWatch Logs

Amazon CloudWatch Logs enables you to centralize the logs from all of your systems, applications, and AWS services that you use. CloudWatch Logs enables you to see all of your logs, regardless of their source, as a single and consistent flow of events ordered by time.

Currently Snare Central support logs from the following AWS Services:

  • AWS CloudTrail

  • AWS Web Application Firewall (WAF)

  • AWS VPC Flow Logs

Note

 

 

Setting Up AWS Kinesis Data Stream

Amazon Kinesis Data Streams ingests a large amount of data in real time, durably stores the data, and makes the data available for consumption. The unit of data stored by Kinesis Data Streams is a data record. A data stream represents a group of data records. The data records in a data stream are distributed into shards.

Note

  • A shard has a sequence of data records in a stream. It serves as a base throughput unit of a Kinesis data stream. A shard supports 1 MB/s and 1000 records per second for writes and 2 MB/s for reads in both on-demand and provisioned capacity modes

  • For more information, please refer to AWS official documentation: Amazon Kinesis Data Streams

  • Also refer to: Amazon Kinesis Data Streams Pricing for more information on the possible charges you may incur.

 

 

Step 1. Sign in to the AWS Management Console and open the Kinesis console at Kinesis console - AWS Management Console.

 

Step 2. Click Data Streams in the navigation pane.

image-20241122-083923.png
Amazon Kinesis Navigation Pane (Left Side)

 

Step 3. In the navigation bar, expand the Region selector and choose the appropriate Region.

 

Step 4. Click Create data stream.

 

Step 5. In Data stream name, enter a name for your stream (e.g. snare), then in Capacity mode select Provisioned and enter the number of shards that you will need in the Provisioned shards (e.g. 1).

 

Step 6. Scroll down to the bottom and click Create data stream.

 

Step 7. Once you created the Kinesis Data Stream successfully, you should be able to see the Data stream summary.

 

Step 8. Click Data Streams in the navigation pane, then you should be able to see it in the list of Data streams with an Active status.

 

 

 

Setting Up AWS CloudWatch Logs Subscription Filter

You can use subscriptions to get access to a real-time feed of log events from CloudWatch Logs and have it delivered to other services such as an Amazon Kinesis Data Stream for custom processing, analysis, or loading to other systems.

 

 

Step 1. Sign in to the AWS Management Console and open the CloudWatch console at CloudWatch console - AWS Management Console.

 

Step 2. Click Log groups in the navigation pane.

 

Step 3. In the navigation bar, expand the Region selector and choose the appropriate Region.

 

Step 4. Click the Log group that you want logs to be streamed to Kinesis Data Stream and get collected by Snare Central’s AWS Log Collection e.g. aws-waf-logs-sampleLogGroup.

 

Step 5. Click Subscription filters tab then click Create then select Create Kinesis subscription filter.

 

Step 6. Set Destination account and Kinesis data stream (in the list, enter or select the name of the Kinesis data stream you previously setup).

 

Step 7. Set Grant permission. Click create a new role if you don’t have an existing role that grant CloudWatch Logs permission to put data into your Kinesis data stream or Select an existing role if you already have one.

 

Step 8. Set your desired Distribution method and Configure log format and filters.

 

Step 9. Optionally you can Test pattern if you set one. Then afterwards click Start streaming.

 

Step 10. Once the setup is successful, you should be able to see the created subscription filter in the list, this log group will stream the log data to your Kinesis data stream.

 

 

 

Setting Up Snare Central - Amazon Web Services(AWS) Cloud Log Collection

Starting from Snare Central v8.6.0, AWS Cloud Log Collection functionality will be available as long as you have the proper license for it. This guide will help you setup up your Snare Central and start collecting supported AWS logs in no-time by simply using the intuitive Cloud Log Collection Configuration Web UI of Snare Central.

 

Step 1. Go to Snare Central and navigate to System > Administrative Tools > Cloud Log Collection Configuration.

 

Step 2. Select Amazon Web Services and Click ADD CLOUD COLLECTION button.

 

Step 3. Input all the necessary AWS Cloud Collection Configuration Information and click “Test Connection” Button to check if the configuration can properly connect to your previously created Kinesis Data Stream.

  • Name: Any name to easily identify this AWS Cloud Log Collector.

  • Enabled: Can be toggled ON/OFF. This will determine if the AWS Cloud Collector will be enabled and start log collection (This can also be toggled ON/OFF easily later after setup).

  • AWS Access Key ID: AWS Credential with permission to make programmatic calls/request to AWS API. see: Managing Access Keys for IAM users for more information.

  • AWS Secret Access Key: AWS Credential used to sign request to AWS API. see: Managing Access Keys for IAM users for more information.

  • AWS Region Code: Region code were you setup your AWS Kinesis Data Stream, e.g. us-east-1.

  • AWS Kinesis Data Stream Name: The Kinesis Data Stream Name you want to collect logs from. e.g. snare. (The name used in Setting Up AWS Kinesis Data Stream)

  • Polling Interval: Log collection interval (in millisecond) for each log collection request to specified AWS Kinesis Data Stream. (Actual request interval maybe greater than what is set, depending on the actual response time for each request).

  • Default Starting Position When Collecting Logs: This will be the default log collection starting position in AWS Kinesis Data Stream Specified when there is no valid sequence number yet.

    • TRIM_HORIZON Start streaming at the last untrimmed record in the shard, which is the oldest data record in the shard.

    • LATEST Start streaming just after the most recent record in the shard. See API Starting Position for more information.

  • Note: Optional field that you may use to note any related information to this AWS Cloud Log Collector

 

Step 4. Click ADD button, then you should be able to see the added AWS Cloud Log Collector under the Amazon Web Services Cloud Collection List.

 

 

Updating/Deleting - AWS Cloud Log Collection Configuration

If you want to update or delete an existing Amazon Web Services - Cloud Log Collector that were previously configured, you can simply use the Snare Central’s Cloud Log Collection Configuration Web UI and follow the simple steps below.

 

Step 1. Go to Snare Central and navigate to System > Administrative Tools > Cloud Log Collection Configuration.

 

Step 2. Select Amazon Web Services and Click the AWS Log Collector that you want to update, then click the Edit icon on the right side.

 

Step 3. In the Edit screen, you can update the configuration and optionally do a Test Connection to check if the updated configuration can successfully connect with your AWS Kinesis Data Stream, then simply click the SAVE button to save the updated configuration.

 

 

 

 

 

Step 1. Go to Snare Central and navigate to System > Administrative Tools > Cloud Log Collection Configuration.

 

Step 2. Select Amazon Web Services and Click the AWS Log Collector that you want to delete, then click the Delete icon on the right side.

 

 

 

Troubleshooting Guide

This guide will be your resource for resolving common issues and challenges that you may encounter with Amazon Web Services (AWS) - Cloud Log Collection.

  1. Amazon Web Services icon is gray in System > Administrative Tools > Cloud Log Collection Configuration Web UI.

  1. AWS Cloud Log Collector icon is gray and the Status is Not Running (Disabled by configuration)

  1. AWS Cloud Log Collector icon is red and the Status is Not Running (The security token included in the request is invalid.)

  1. AWS Cloud Log Collector icon is red and the Status is Not Running (The request signature we calculated does not match the signature you provided.)

  1. AWS Cloud Log Collector icon is red and the Status is Not Running (Post "https://kinesis.us-east-11.amazonaws.com": dial tcp: lookup kinesis.us-east-11.amazonaws.com: no such host)

  1. AWS Cloud Log Collector icon is red and the Status is Not Running (Stream <streamname> under account <account number> not found.)

  1. AWS Cloud Log Collector takes too long to get new logs.

  1. AWS Cloud Log Collector did not collect the old logs in AWS Kinesis Data Stream.