Skip to content

Amazon MSK cluster logging is not enabled

Description

Amazon MSK enables you to build and run applications that use Apache Kafka to process streaming data. It also provides a control-plane for advanced operations, for example, creating, updating, and deleting clusters.

Consistent cluster logging helps you determine if a request was made with root or AWS Identity and Access Management (IAM) user credentials and whether the request was made with temporary security credentials for a role or federated user.

Fix - Runtime

AWS Console

New Cluster:

  1. Log in to the AWS Management Console at [https://console.aws.amazon.com/].
  2. Open the Amazon MSK console.
  3. Go to Broker Log Delivery in the Monitoring section.
  4. Specify the destinations to which you want Amazon MSK to deliver your broker logs.

Existing Cluster:

  1. In the Amazon MSK console choose the cluster from your list of clusters.
  2. Go to the Details tab. Scroll down to the Monitoring section and click Edit.
  3. Specify the destinations to which you want Amazon MSK to deliver your broker logs.

CLI Command

When you use the create-cluster or the update-monitoring commands, you can optionally specify the logging-info parameter and pass to it a JSON structure. In this JSON, all three destination types are optional.

{
  "BrokerLogs": {
    "S3": {
      "Bucket": "ExampleBucketName",
      "Prefix": "ExamplePrefix",
      "Enabled": true
    },
    "Firehose": {
      "DeliveryStream": "ExampleDeliveryStreamName",
      "Enabled": true
    },
    "CloudWatchLogs": {
      "Enabled": true,
      "LogGroup": "ExampleLogGroupName"
    }
  }
}

Fix - Buildtime

Terraform

  • Resource: aws_msk_cluster
  • Argument: logging_info - (Optional) Configuration block for streaming broker logs to Cloudwatch/S3/Kinesis Firehose. See below. 
resource "aws_msk_cluster" "example" {
  cluster_name           = "example"
  ...
+  logging_info {
+    broker_logs {
+      cloudwatch_logs {
+        enabled   = true
+        log_group = aws_cloudwatch_log_group.test.name
      }

CloudFormation

  • Resource:AWS::MSK::Cluster
  • ArgumentLoggingInfo. Configure your MSK cluster to send broker logs to different destination types. This is a container for the configuration details related to broker logs. 
{
  "Type" : "AWS::MSK::Cluster",
  "Properties" : {
      ...
+    "LoggingInfo" : {
+      "BrokerLogs" : {
+        "CloudWatchLogs" : CloudWatchLogs,
+        "Firehose" : Firehose,
+        "S3" : S3
      }
    }
  }
}