AWS VPC logs

AWS VPC logs#

When AWS Virtual Private Cloud (VPC) flow log ingestion is configured and enabled, Scrutinizer can report additional insights for network traffic destined for AWS, including top AWS users and applications, as well as traffic load generated by AWS-hosted applications.

The following AWS-specific report types become available to run:

AWS report types

Report Type

Description

Action

Aggregation based on the Action (ACCEPT, REJECT, or DROP) associated with the traffic

Action with Interface

Aggregation based on the action applied and the interface associated with the flow

Action with Interface and Dst

Aggregation based on the action applied, the associated interface, and the traffic’s destination

Action with Interface and Src

Aggregation based on the action applied, the associated interface, and the traffic’s source

Availability Zones

Aggregation based on the AWS Availability Zone associated with the traffic

Dst Service

Aggregation based on the AWS service the traffic was destined for

Interface

Aggregation based on the source or destination interface associated with the traffic

Pair Interface

Aggregation based on the source and destination interfaces associated with the traffic

Pair Interface Action

Aggregation based on the Action applied and the source and destination interfaces of the traffic

Src Service

Aggregation based on the AWS service the traffic originates from

Src Service-Dst Service

Aggregation based on AWS services the traffic originated from and was destined for

Traffic Path

Aggregation based on the traffic path used by egress traffic to reach its destination

VPCs

Aggregation based on the VPC ID associated with the traffic

This section covers the prerequisites and setup/configuration steps for AWS VPC flow log ingestion.

Setting up the AWS S3 storage bucket#

Before setting up AWS VPC flow log ingestion in Scrutinizer, the Amazon S3 storage bucket(s) that will be used should be configured as follows:

  • Set the VPC(s) to be monitored to send flow logs to the bucket. The flow log format must include the following fields:

    • log-status

    • vpc-id

    • interface-id

    • flow-direction

  • The bucket should be reserved for exclusive use by Scrutinizer. If the flow logs need to be archived or used for other purposes, send the flow logs to a separate S3 bucket, and then automate the replication/duplication of those logs to the bucket that will be used by Scrutinizer.

  • Versioning should be disabled.

Note

  • When upgrading from older versions of Scrutinizer, it may be necessary to delete the existing VPC flow log configuration and create a new one that includes the interface-id and flow-direction fields.

  • When creating a VPC flow log, leaving the Maximum Aggregation Interval setting at the default 10 minutes will minimize processing load on the Scrutinizer collector at the cost of longer update times and data spikes. Setting the maximum aggregation interval to 1 minute will result in more granular data but also increase resource utilization.

  • After an S3 bucket is first configured for ingestion, Scrutinizer purges all older flow logs from the bucket before starting to collect and delete the most recent 15 minutes of logs as normal. If any historical data needs to be retained, it should be copied off the bucket before the integration is configured. Manually clearing the bucket of any log data older than 15 minutes will also allow Scrutinizer to become current more quickly.

Configuring AWS VPC flow log ingestion in Scrutinizer#

To add an S3 bucket as a flow log ingestion source in Scrutinizer, follow these steps:

  1. In the Scrutinizer web interface, navigate to Admin > Integrations > Flow Log Ingestion.

  2. Click the + icon and select AWS VPC FlowLogs in the tray.

  3. In the secondary tray, configure the bucket details as follows:

    • Enter a name to identify the bucket/source by.

    • Select the Scrutinizer servers to use as log downloader(s) and collector(s) for the bucket (in distributed clusters, remote collectors are recommended for these roles).

    • Enter the name of the bucket.

    • Select the AWS region where the bucket is hosted from the dropdown.

    • Enter the credentials to use to access the bucket (AWS access key ID and secret access key; must have full access to the bucket)

  4. Click the Save button to add the bucket with the current settings.

Once added, the bucket will be listed in the main Admin > Integrations > Flow Log Ingestion view under the configured name. An exporter associated with the VPC will also be added to the device lists for Scrutinizer’s various functions (reports, network maps, etc.).

Note

  • After a bucket configuration has been saved, click on the name assigned to it in the main view to open the settings tray, and use the Test button to confirm that Scrutinizer is able to establish a connection to the bucket with the credentials entered.

  • To verify that an AWS VPC flow log source has been successfully added, look for an exporter labeled vpc- in the Explore > Exporters > By Exporters view or the Admin > Resources > Exporters page (after ~1 hour).

  • Flow log ingestion processes are divided between the log downloader (downloads the flow logs from the bucket) and the flow collector (collects and processes the downloaded logs). A different Scrutinizer server can be used for each role, and a single bucket can have multiple downloaders and collectors.

Troubleshooting#

If the Admin > Resources > Exporters view does not list exporters matching the virtual network(s) set up for flow ingestion, check the following for issues:

  • Open the tray for the ingestion source in the Admin > Integrations > Flow Ingestion view and use the Test button to verify that the collector/downloader is able to communicate with the data source using the details entered.

  • Verify that logs are correctly being sent to the source bucket.

  • Check the collector log file in /home/plixer/scrutinizer/files/logs/ for errors.

  • Check awss3_log.json for possible source-side issues.

For further assistance, contact Plixer Technical Support.

Overloaded collectors/downloaders#

The Unresourced - Enabled status in the Admin > Resources > Exporters view indicates that a log source is being temporarily disabled/paused due to insufficient resources.

The following are potential solutions for an overloaded collector:

  • If the collector is a VM, allocate additional resources (starting with CPU cores) to it.

  • If the collector is ingesting logs from only one bucket, distribute the logs across multiple buckets, which can then be assigned to different collectors.

  • If the collector is ingesting logs from multiple buckets, distribute the buckets across multiple collectors.

  • If the collector license has a flow rate limit, the license may need to be upgraded.

Note

  • Sources that are tagged as Disabled may have been automatically disabled (last-in/first-out order) due to the license exporter count limit.

  • In distributed deployments, it is recommended to start with a 1:1 pairing of sources and collectors.

Enabling role-based IAM for AWS deployments#

Role-based IAM can be enabled for Scrutinizer AMI instances by ticking the checkbox in the configuration tray. The role assigned to the EC2 instance should be provisioned with the following permissions:

{ "Version": "2012-10-17",
  "Statement": \[
             { "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": \[ "s3:GetObject", "s3:DeleteObject" \],
               "Resource": \[ "arn:aws:s3:::<S3BUCKET>/\*" \]
             },
             { "Sid": "VisualEditor1",
               "Effect": "Allow",
               "Action": "s3:\*",
               "Resource": "arn:aws:s3:::<S3_BUCKET_NAME>"
             }
  \]
}

Note

Role based authentication is only available when all log downloaders are hosted in AWS.

Importing AWS entity descriptions#

To allow description reporting and filtering by AWS entity identifiers (interface-id, vpc-id, etc.) directly in the Scrutinizer UI, follow these steps:

  1. Provision the user or IAM role with the following additional permissions:

    ec2:DescribeInstances
ec2:DescribeSubnets
ec2:DescribeVpcs
ec2:DescribeNetworkInterfaces

  2. Start an SSH session with the Scrutinizer server (or the primary reporter in distributed deployments), and run the following command via the scrut_util interactive CLI:

    SCRUTINIZER> awssync
      AWS entities synced!

Once entity descriptions have been synced, AWS entity identifiers will automatically be replaced with their descriptions whenever an AWS-specific report is run. The awssync task will automatically be run every hour thereafter.

Contents