Stream VPC Move Logs to Datadog by way of Amazon Kinesis Knowledge Firehose

It’s frequent to retailer the logs generated by buyer’s purposes and providers in numerous instruments. These logs are essential for compliance, audits, troubleshooting, safety incident responses, assembly safety insurance policies, and plenty of different functions. You possibly can carry out log evaluation on these logs to know customers’ software habits and patterns to make knowledgeable choices.

When working workloads on Amazon Net Companies (AWS), you want to analyze Amazon Digital Personal Cloud (Amazon VPC) Move Logs to trace the IP site visitors going to and from the community interfaces for the workloads of their VPC. Analyzing VPC stream logs helps you perceive how your purposes are speaking over the VPC community and acts as a essential supply of data to the community in your VPC.

You possibly can simply ship information to supported locations utilizing the Amazon Kinesis Knowledge Firehose integration with VPC stream logs. Kinesis Knowledge Firehose is a totally managed service for delivering near-real-time streaming information to varied locations for storage and performing near-real-time analytics. With its extensible information transformation capabilities, you too can streamline log processing and log supply pipelines right into a single Kinesis Knowledge Firehose supply stream. You possibly can carry out analytics on VPC stream logs delivered out of your VPC utilizing the Kinesis Knowledge Firehose integration with Datadog as a vacation spot.

Datadog is a monitoring and safety platform and AWS Associate Community (APN) Superior Know-how Associate with AWS Competencies in AWS Cloud Operations, DevOps, Migration, Safety, Networking, Containers, and Microsoft Workloads, together with many others.

Datadog allows you to simply discover and analyze logs to achieve deeper insights into the state of your purposes and AWS infrastructure. You possibly can analyze all of your AWS service logs whereas storing solely those you want, generate metrics from aggregated logs to uncover, and ship alerts about traits in your AWS providers.

On this put up, you learn to combine VPC stream logs with Kinesis Knowledge Firehose and ship it to Datadog.

Answer overview

This answer makes use of native integration of VPC stream logs streaming to Kinesis Knowledge Firehose. We use a Kinesis Knowledge Firehose supply stream to buffer the streamed VPC stream logs to a Datadog vacation spot endpoint in your Datadog account. You should utilize these logs with Datadog Log Administration and Datadog Cloud SIEM to investigate the well being, efficiency, and safety of your cloud assets.

The next diagram illustrates the answer structure.

We stroll you thru the next high-level steps:

1. Hyperlink your AWS account together with your Datadog account.
2. Create the Kinesis Knowledge Firehose stream the place VPC service streams the stream logs.
3. Create the VPC stream log subscription to Kinesis Knowledge Firehose.
4. Visualize VPC stream logs within the Datadog dashboard.

The account ID 123456781234 used on this put up is a dummy account. It’s used just for demonstration functions.

Conditions

It is best to have the next stipulations:

Hyperlink your AWS account together with your Datadog account for AWS integration

Comply with the instructions offered on the Datadog web site for AWS Integration. To configure log archiving and enrich the log information despatched out of your AWS account with helpful context, hyperlink the accounts. While you full the linking setup, proceed to the next step.

Create a Kinesis Knowledge Firehose stream

Now that your Datadog integration with AWS is full, you possibly can create a Kinesis Knowledge Firehose supply stream the place VPC Move Logs are streamed by following these steps:

1. On the Amazon Kinesis console, select Kinesis Knowledge Firehose within the navigation pane.
2. Select Create supply stream.
3. Select Direct PUT because the supply.
4. Set Vacation spot as Datadog.
   

5. For Supply stream identify, enter PUT-DATADOG-DEMO.
6. Preserve Knowledge transformation set to Disabled beneath Rework data.
7. In Vacation spot settings, for HTTP endpoint URL, select the specified log’s HTTP endpoint primarily based in your Area and Datadog account configuration.
   
8. For API key, enter your Datadog API key.

This enables your supply stream to publish VPC Move logs to the Datadog endpoint. API keys are distinctive to your group. An API key is required by the Datadog Agent to submit metrics and occasions to Datadog.

9. Set Content material encoding to GZIP to cut back the scale of information transferred.
10. Set the Retry period to 60.You possibly can change the Retry period worth if you want to. This is dependent upon the request dealing with capability of the Datadog endpoint.
    
    Below Buffer hints, Buffer dimension and Buffer interval are set with default values for Datadog integration.
    

11. Below Backup settings, as talked about within the stipulations, select the S3 bucket that you just created to retailer failed logs and backup with particular prefix.
12. Below S3 buffer hints part, set Buffer dimension to five and Buffer interval to 300.

You possibly can change the S3 buffer dimension and interval primarily based in your necessities.

13. Below S3 compression and encryption, choose GZIP for Compression for information data or one other compression technique of your alternative.

Compressing information reduces the required space for storing.

14. Choose Disabled for Encryption of the information data. You possibly can allow encryption of the information data to safe entry to your logs.
    

15. Optionally, in Superior settings, choose Allow server-side encryption for supply data in supply stream.
    You should utilize AWS managed keys or a CMK managed by you for the encryption kind.

16. Allow CloudWatch error logging.
17. Select Create or replace IAM position, which is created by Kinesis Knowledge Firehose as a part of this stream.
    

18. Select Subsequent.
19. Assessment your settings.
20. Select Create supply stream.

Create a VPC stream logs subscription

Create a VPC stream logs subscription for the Kinesis Knowledge Firehose supply stream you created within the earlier step:

1. On the Amazon VPC console, select Your VPCs.
2. Choose the VPC that you just to create the stream log for.
3. On the Actions menu, select Create stream log.
   

4. Choose All to ship all stream log data to the Firehose vacation spot.

If you wish to filter the stream logs, you would alternatively choose Settle for or Reject.

5. For Most aggregation interval, choose 10 minutes or the minimal setting of 1 minute should you want the stream log information to be out there for near-real-time evaluation in Datadog.
6. For Vacation spot, choose Ship to Kinesis Knowledge Firehose in the identical account if the supply stream is ready up on the identical account the place you create the VPC stream logs.

If you wish to ship the information to a distinct account, check with Publish stream logs to Kinesis Knowledge Firehose.

7. Select an possibility for Log report format:
8. When you go away Log report format because the AWS default format, the stream logs are despatched as model 2 format.
9. Alternatively, you possibly can specify the customized fields for stream logs to seize and ship it to Datadog.

For extra data on log format and out there fields, check with Move log data.

10. Select Create stream log.
    

Now let’s discover the VPC stream logs in Datadog.

Visualize VPC stream logs within the Datadog dashboard

Within the Logs Search possibility within the navigation pane, filter to supply:vpc. The VPC stream logs out of your VPC are within the Datadog Log Explorer and are robotically parsed so you possibly can analyze your logs by supply, vacation spot, motion, or different attributes.

Clear up

After you check this answer, delete all of the assets you created to keep away from incurring future prices. Discuss with the next hyperlinks for directions for deleting the assets:

Conclusion

On this put up, we walked by way of an answer of learn how to combine VPC stream logs with a Kinesis Knowledge Firehose supply stream, ship it to a Datadog vacation spot with no code, and visualize it in a Datadog dashboard. With Datadog, you possibly can simply discover and analyze logs to achieve deeper insights into the state of your purposes and AWS infrastructure.

Do that new, fast, and hassle-free method of sending your VPC stream logs to a Datadog vacation spot utilizing Kinesis Knowledge Firehose.

In regards to the Writer

Chaitanya Shah is a Sr. Technical Account Supervisor(TAM) with AWS, primarily based out of New York. He has over 22 years of expertise working with enterprise clients. He likes to code and actively contributes to the AWS options labs to assist clients clear up complicated issues. He gives steerage to AWS clients on greatest practices for his or her AWS Cloud migrations. He’s additionally specialised in AWS information switch and the information and analytics area.