Amazon Web Services flow logs

Overview

The integration between Amazon Web Services (AWS) and Plixer Scrutinizer provides insight into network traffic destined for AWS, such as top AWS users, top AWS applications, as well as overall traffic load of AWS hosted applications. After configuring Amazon Web Services Flow Log integration, the following reports become available in Plixer Scrutinizer:

  • Action
  • Action with Interface
  • Action with Interface and Dst
  • Action with Interface and Src
  • Interface
  • Pair Interface
  • Pair Interface Action

Prerequisites

The following information is required to configure AWS flow logs integration:

  1. AWS IDs and secrets with full access permission to the FlowLog S3 buckets and necessary permissions to collect descriptions OR Plixer Scrutinizer needs to be running in AWS where the EC2 instance is assigned a role with those permissions.

Hint

The VPC(s) you want to monitor need to be configured to send flow logs to the S3 buckets Plixer Scrutinizer is configured to read.

Note

These S3 buckets are solely intended for Plixer Scrutinizer’s use. Plixer Scrutinizer will delete the logs from the buckets as it collects them.

  1. The region that hosts the S3 bucket.
  2. It is strongly recommended to include the two log fields listed below:

log-status

This is a version 2 default log field. Customized VPC logs without the field will not be collected until it is re-added.

Important

VPC flow logs that do not include this field will be discarded.

vpc-id

This is a version 3 custom log field. Users running the default version 2 logs will need to add it to start log collection. Plixer Scrutinizer will attempt to generate exporter IDs using alternate data if vpc-id is not present. These fallbacks will usually generate more exporters than using vpc-id. In addition to potentially exceeding license limits on exporter counts, egregious exporters may also impair Plixer Scrutinizer’s flow collection rates and reporting times.
  1. Exporter IDs will change for the upgraded deployments. v19.0.1 Plixer Scrutinizer sources exporter IDs from the vpc-id log field and the AWS account number in the directory path. Prior to v 19.0.1, exporter IDs were determined by the AWS S3 bucket name. They will stop exporting after an upgrade.
  2. Installations with several VPCs will have several new exporters. The new VPC-based exporters can potentially exceed license limits.

Note

Review the status of your exporters via the Admin>Definitions>Manage Exporters page if you are not collecting data.

Configuring AWS flow logs

  1. Navigate to the Admin > Settings > AWS Flow Logs S3 page.
  2. Click “Add” to create a new flow log source in Plixer Scrutinizer. A single S3 bucket can comprise data from several of the sources we consider to be exporters.
  3. Provide a unique name for the Flow Log source.
  4. Select the collector that will communicate with AWS to receive data for this bucket.
  5. Enter the bucket name, region, ID, and Secret.
  6. Save the entry.

Enabling IAM role-based authentication for S3 buckets

  1. Navigate to the Admin > Settings > AWS Flow Logs S3 page of your Plixer Scrutinizer AMI deployment.
  2. Click “Add” to create a new flow log source.
  3. Provide a unique name for the flow log source.
  4. Select the collector that will communicate with AWS to receive data for this bucket.
  5. Enter the bucket name and its region.
  6. Check the box to enable IAM role-based authentication
  7. Save the entry.

The necessary policy permissions that a role must have are as follows:

{ "Version": "2012-10-17",
  "Statement": \[
             { "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": \[ "s3:GetObject", "s3:DeleteObject" \],
               "Resource": \[ "arn:aws:s3:::<S3BUCKET>/\*" \]
             },
             { "Sid": "VisualEditor1",
               "Effect": "Allow",
               "Action": "s3:\*",
               "Resource": "arn:aws:s3:::<S3BUCKET>"
             }
  \]
}

Hint

<S3BUCKET> should be replaced with the name of the bucket you are using. The “Version” element is an AWS-provided version string that refers to the version of the policy specification it adheres to, i.e. only change this value to something that you know AWS will support and only if necessary.

Importing descriptions for AWS entity IDs

The AWS entity IDs import functionality provides reporting descriptions and filtering for any AWS entity identifiers, such as interface-id, vpc-id, subnet-id, and instance-id. With the feature configured, a user will not need to use the AWS console or documentation to determine the meaning of the identifier.

  1. The following user permissions are required:
ec2:DescribeInstances
ec2:DescribeSubnets
ec2:DescribeVpcs
ec2:DescribeNetworkInterfaces

For example, the policy below can be applied to a user/IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeVpcs",
                "ec2:DescribeSubnets"
            ],
            "Resource": "*"
        }
    ]
 }
  1. SSH into the Plixer Scrutinizer standalone or a primary reporter appliance in the distributed cluster. Run the interactive CLI command to sync IDs and descriptions from AWS.
SCRUTINIZER> awssync
AWS entities synced!
  1. Launch an AWS specific report. The identifiers will be replaced with their appropriate descriptions.

Note

The awssync task is scheduled to run hourly.

Helpful tips

Amazon flow logs are updated every 1 minute or every 10 minutes. Choosing every 10 minutes will result in data spikes and longer data update times. There will, however, be less processing load on the Plixer Scrutinizer server. Choosing every 1 minute will result in better reporting granularity.

If you are not seeing an exporter:

  • check the collector log for errors;
  • go to the AWS interface and make sure you see flow logs in the configured bucket;
  • edit the S3 profile in Plixer Scrutinizer and use the “Test” button to make sure the configuration is correct;
  • make sure the exporter is not disabled under Admin > Definitions > Manage Exporters.