AWS flow logs

Amazon Web Services (AWS) flow log integration allows Plixer Scrutinizer to report additional insight into network traffic destined for AWS, includ top AWS users and applications, as well as traffic load generated by AWS-hosted applications.

Once enabled and configured, the following AWS-flow-log based Reports become available in Plixer Scrutinizer:

  • Action
  • Action with Interface
  • Action with Interface and Dst
  • Action with Interface and Src
  • Interface
  • Pair Interface
  • Pair Interface Action

Prerequisites

The following information is required to configure AWS flow log integration:

  1. AWS IDs and Secrets with full access permission to FlowLog S3 buckets and necessary permissions to collect descriptions OR Plixer Scrutinizer running in AWS with the EC2 instance assigned a role with those permissions

Hint

The VPC(s) you want to monitor need to be configured to send flow logs to the S3 buckets Plixer Scrutinizer is configured to read.

Note

These S3 buckets are solely intended for Plixer Scrutinizer’s use. Plixer Scrutinizer will delete logs as they are collected.

  1. The region hosting the S3 bucket
  2. The following log fields should be included:

log-status

This is a version 2 default log field. Customized VPC logs without the field will not be collected until it is re-added.

Important

VPC flow logs that do not include this field will be discarded.

vpc-id

This is a version 3 custom log field. Users running the default version 2 logs will need to add it to start log collection. Plixer Scrutinizer will attempt to generate exporter IDs using alternate data if vpc-id is not present. These fallbacks will usually generate more exporters than using vpc-id. In addition to potentially exceeding license limits on exporter counts, egregious exporters may also impair Plixer Scrutinizer’s flow collection rates and reporting times.

interface-id

This is a version 2 custom log field that identifies the network interface whose traffic is being recorded.

flow-direction

This is a version 5 custom log field that specifies the flow direction (ingress or egress) relative to the interface where traffic is being recorded. If interface-id or flow-direction is not added, VPC flows will not be collected.

Important

Customers upgrading from older versions may need to delete and re-create their logging configuration to add the interface-id and flow-direction fields.

  1. Exporter IDs will change for the upgraded deployments. v19.0.1 Plixer Scrutinizer sources Exporter IDs from the vpc-id log field and the AWS account number in the directory path. Prior to v 19.0.1, Exporter IDs were determined by the AWS S3 bucket name. They will no longer be used after the upgrade.
  2. Installations with several VPCs will have several new Exporters, which may exceed license limits.

Note

Review the status of your Exporters via the Admin > Definitions > Manage Exporters page if data is not being collected.

Configuring AWS flow logs

  1. Navigate to the Admin > Settings > AWS Flow Logs S3 page.
  2. Click “Add” to create a new flow log source in Plixer Scrutinizer. A single S3 bucket can comprise data from multiple sources treated as Exporters.
  3. Provide a unique name for the Flow Log source.
  4. Select the Collector that will communicate with AWS to receive data for this bucket.
  5. Enter the bucket name, region, ID, and Secret.
  6. Save the entry.

Enabling IAM role-based authentication for S3 buckets

  1. Navigate to the Admin > Settings > AWS Flow Logs S3 page of your Plixer Scrutinizer AMI deployment.
  2. Click “Add” to create a new flow log source.
  3. Provide a unique name for the flow log source.
  4. Select the Collector that will communicate with AWS to receive data for this bucket.
  5. Enter the bucket name and its region.
  6. Check the box to enable IAM role-based authentication
  7. Save the entry.

The necessary policy permissions for the role are as follows:

{ "Version": "2012-10-17",
  "Statement": \[
             { "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": \[ "s3:GetObject", "s3:DeleteObject" \],
               "Resource": \[ "arn:aws:s3:::<S3BUCKET>/\*" \]
             },
             { "Sid": "VisualEditor1",
               "Effect": "Allow",
               "Action": "s3:\*",
               "Resource": "arn:aws:s3:::<S3BUCKET>"
             }
  \]
}

Hint

<S3BUCKET> should be replaced with the name of the bucket you are using. The “Version” element is an AWS-provided version string that refers to the version of the policy specification it adheres to, i.e. only change this to a value that you know AWS will support and only if necessary.

Importing descriptions for AWS entity IDs

The AWS entity ID import functionality provides description reporting and filtering by AWS entity identifiers, such as interface-id, vpc-id, subnet-id, and instance-id. Once set up, a user will no longer need to use the AWS console or documentation to determine the meaning of the identifier.

  1. Provision the user with the following permissions:
ec2:DescribeInstances
ec2:DescribeSubnets
ec2:DescribeVpcs
ec2:DescribeNetworkInterfaces

For example, the policy below can be applied to a user/IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeVpcs",
                "ec2:DescribeSubnets"
            ],
            "Resource": "*"
        }
    ]
 }
  1. Start an SSH session with either the Plixer Scrutinizer standalone deployment or the primary reporting server in a distributed cluster, and run the following command from the interactive scrut_util CLI to sync IDs and descriptions from AWS.
SCRUTINIZER> awssync
AWS entities synced!
  1. Launch an AWS-specific report. The identifiers will be replaced with their appropriate descriptions.

Note

The awssync task is scheduled to run hourly.

Additional tips

AWS flow logs can be updated every minute or every 10 minutes. Choosing every 10 minutes will result in data spikes and longer data update times but also reduce processing load on the Plixer Scrutinizer server/Collector. Choosing 1-minute updates will result in more granular reporting.

If you are not seeing an Exporter:

  • Check the collector log for errors.
  • Go to the AWS interface and make sure you see flow logs in the configured bucket.
  • Edit the S3 profile in Plixer Scrutinizer and use the “Test” button to make sure the configuration is correct.
  • Make sure the Exporter is not disabled under Admin > Definitions > Manage Exporters.