AWS VPC and Transit Gateway flow logs

AWS VPC and Transit Gateway flow logs#

When AWS Virtual Private Cloud (VPC) flow log ingestion is configured and enabled, Scrutinizer can report additional insights for network traffic destined for AWS, including top AWS users and applications, as well as traffic load generated by AWS-hosted applications.

Scrutinizer supports ingestion of both AWS VPC Flow Logs (up to version 11) and AWS Transit Gateway (TGW) Flow Logs (up to version 6). While VPC Flow Logs and TGW Flow Logs are configured and ingested using the same mechanism (see the Setting up the AWS S3 storage bucket section), they are distinct log formats with different field schemas, and are covered separately below.

Note

Each Virtual Private Cloud (VPC) configured to send flow logs to Scrutinizer for AWS counts toward the license-defined exporter limit.
For Transit Gateways, only TGW attachments count toward the license-defined exporter limit; the Transit Gateway itself does not.

The following AWS-specific report types become available to run:

AWS report types

Report Type	Description
Action	Aggregation based on the Action (`ACCEPT`, `REJECT`, or `DROP`) associated with the traffic
Action with Interface	Aggregation based on the action applied and the interface associated with the flow
Action with Interface and Dst	Aggregation based on the action applied, the associated interface, and the traffic’s destination
Action with Interface and Src	Aggregation based on the action applied, the associated interface, and the traffic’s source
Availability Zones	Aggregation based on the AWS Availability Zone associated with the traffic
Dst Service	Aggregation based on the AWS service the traffic was destined for
Interface	Aggregation based on the source or destination interface associated with the traffic
Pair Interface	Aggregation based on the source and destination interfaces associated with the traffic
Pair Interface Action	Aggregation based on the Action applied and the source and destination interfaces of the traffic
Src Service	Aggregation based on the AWS service the traffic originates from
Src Service-Dst Service	Aggregation based on AWS services the traffic originated from and was destined for
Traffic Path	Aggregation based on the traffic path used by egress traffic to reach its destination
VPCs	Aggregation based on the VPC ID associated with the traffic

This section covers the prerequisites and setup/configuration steps for AWS VPC flow log ingestion.

Setting up the AWS S3 storage bucket#

A single S3 bucket can be used to ingest both VPC Flow Logs and TGW Flow Logs simultaneously — the two log types do not require separate buckets. Point both your VPC flow log subscriptions and your TGW flow log subscriptions at the same bucket, and Scrutinizer will ingest and correctly parse both.

If ingesting VPC Flow Logs, the flow log format sent to the bucket must include the following fields:

log-status
vpc-id
interface-id
flow-direction
start
end

If ingesting TGW Flow Logs, the flow log format sent to the bucket must include the following fields instead:

log-status
flow-direction
start
end
tgw-attachment-id
The bucket should be reserved for exclusive use by Scrutinizer. If the flow logs need to be archived or used for other purposes, send the flow logs to a separate S3 bucket, and then automate the replication/duplication of those logs to the bucket that will be used by Scrutinizer.
Versioning should be disabled.

Note

When upgrading from older versions of Scrutinizer, it may be necessary to delete the existing VPC flow log configuration and create a new one that includes the interface-id and flow-direction fields.
When creating a VPC flow log, leaving the Maximum Aggregation Interval setting at the default 10 minutes will minimize processing load on the Scrutinizer collector at the cost of longer update times and data spikes. Setting the maximum aggregation interval to 1 minute will result in more granular data but also increase resource utilization.
After an S3 bucket is first configured for ingestion, Scrutinizer purges all older flow logs from the bucket before starting to collect and delete the most recent 15 minutes of logs as normal. If any historical data needs to be retained, it should be copied off the bucket before the integration is configured. Manually clearing the bucket of any log data older than 15 minutes will also allow Scrutinizer to become current more quickly.

AWS Transit Gateway flow log fields#

Transit Gateway Flow Logs are created against a Transit Gateway or Transit Gateway attachment as the resource (rather than a VPC), and use a different field schema than VPC Flow Logs. In addition to the flow-direction and log-status fields shared with VPC Flow Logs, Scrutinizer supports the following TGW-specific fields:

Field	Description
`tgw-id`	The transit gateway the traffic crossed
`tgw-attachment-id`	The attachment that carried the flow
`tgw-pair-attachment-id`	The paired attachment (source/destination) associated with the flow
`resource-type`	The type of resource the flow log record is associated with
`tgw-src-vpc-id`	Source VPC ID
`tgw-dst-vpc-id`	Destination VPC ID
`tgw-src-vpc-account-id`	AWS account ID owning the source VPC (useful in multi-account environments)
`tgw-dst-vpc-account-id`	AWS account ID owning the destination VPC
`tgw-src-subnet-id`	Source subnet ID
`tgw-dst-subnet-id`	Destination subnet ID
`tgw-src-eni`	Source elastic network interface
`tgw-dst-eni`	Destination elastic network interface
`tgw-src-az-id`	Source Availability Zone ID
`tgw-dst-az-id`	Destination Availability Zone ID
`packets-lost-no-route`	Number of packets lost because no matching route was found
`packets-lost-blackhole`	Number of packets lost because they matched a blackhole route
`packets-lost-mtu-exceeded`	Number of packets lost because they exceeded the path MTU
`packets-lost-ttl-expired`	Number of packets lost because their TTL expired

Scrutinizer supports TGW Flow Log fields up to version 6. For the full AWS reference on TGW flow log fields, see the AWS Transit Gateway Flow Logs documentation.

Configuring AWS VPC flow log ingestion in Scrutinizer#

To configure flow log ingestion from an AWS S3 bucket, follow these steps:

In the Scrutinizer web interface, navigate to Admin > Integrations > Flow Log Ingestion.
Click the + icon and select AWS VPC FlowLogs in the tray.
In the secondary tray, configure the bucket details as follows:
- Enter a name to identify the bucket/source by.
- Select the Scrutinizer servers to use as log downloader(s) and collector(s) for the bucket (in distributed clusters, remote collectors are recommended for these roles).
- Enter the name of the bucket.
- Select the AWS region where the bucket is hosted from the dropdown.
- Enter the credentials to use to access the bucket (AWS access key ID and secret access key; must have full access to the bucket)
Click the Save button to add the bucket with the current settings.

Note

Configuration for TGW Flow Log ingestion is identical to VPC Flow Log ingestion, using the same AWS VPC FlowLogs integration type — there is no separate setup workflow for TGW. Once a bucket is configured and TGW flow logs are being sent to it, Scrutinizer will automatically begin ingesting them alongside any VPC flow logs sent to the same bucket.

Once added, the bucket will be listed in the main Admin > Integrations > Flow Log Ingestion view under the configured name. An exporter associated with the VPC will also be added to the device lists for Scrutinizer’s various functions (reports, network maps, etc.).

Note

After a bucket configuration has been saved, click on the name assigned to it in the main view to open the settings tray, and use the Test button to confirm that Scrutinizer is able to establish a connection to the bucket with the credentials entered.
To verify that an AWS flow log source has been successfully added, look for an exporter labeled vpc- (for VPC sources) or tgw- (for Transit Gateway sources) in the Explore > Exporters > By Exporters view or the Admin > Resources > Exporters page. This can take up to ~1 hour to appear depending on flow log configuration within AWS. Typical times are usually much shorter.
Flow log ingestion processes are divided between the log downloader (downloads the flow logs from the bucket) and the flow collector (collects and processes the downloaded logs). A different Scrutinizer server can be used for each role, and a single bucket can have multiple downloaders and collectors.

Troubleshooting#

If the Admin > Resources > Exporters view does not list exporters matching the virtual network(s) set up for flow ingestion, check the following for issues:

Open the tray for the ingestion source in the Admin > Integrations > Flow Ingestion view and use the Test button to verify that the collector/downloader is able to communicate with the data source using the details entered.
Verify that logs are correctly being sent to the source bucket.
Check the collector log file in /home/plixer/scrutinizer/files/logs/ for errors.
Check awss3_log.json for possible source-side issues, including errors or reports of unrecognized/unknown fields. This is useful for confirming that TGW field parsing is working as expected, in addition to general log delivery issues.

For further assistance, contact Plixer Technical Support.

Overloaded collectors/downloaders#

The Unresourced - Enabled status in the Admin > Resources > Exporters view indicates that a log source is being temporarily disabled/paused due to insufficient resources.

The following are potential solutions for an overloaded collector:

If the collector is a VM, allocate additional resources (starting with CPU cores) to it.
If the collector is ingesting logs from only one bucket, distribute the logs across multiple buckets, which can then be assigned to different collectors.
If the collector is ingesting logs from multiple buckets, distribute the buckets across multiple collectors.
If the collector license has a flow rate limit, the license may need to be upgraded.

Note

Sources that are tagged as Disabled may have been automatically disabled (last-in/first-out order) due to the license exporter count limit.
In distributed deployments, it is recommended to start with a 1:1 pairing of sources and collectors.

Enabling role-based IAM for AWS deployments#

Role-based IAM can be enabled for Scrutinizer AMI instances by ticking the checkbox in the configuration tray. The role assigned to the EC2 instance should be provisioned with the following permissions:

{ "Version": "2012-10-17",
  "Statement": \[
             { "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": \[ "s3:GetObject", "s3:DeleteObject" \],
               "Resource": \[ "arn:aws:s3:::<S3BUCKET>/\*" \]
             },
             { "Sid": "VisualEditor1",
               "Effect": "Allow",
               "Action": "s3:\*",
               "Resource": "arn:aws:s3:::<S3_BUCKET_NAME>"
             }
  \]
}

Note

Role based authentication is only available when all log downloaders are hosted in AWS.

Importing AWS entity descriptions#

To enable description reporting and filtering by AWS entity identifiers (interface-id, vpc-id, etc.) directly in the Scrutinizer UI, follow these steps:

Provision the user or IAM role with the following additional permissions:

ec2:DescribeInstances
ec2:DescribeSubnets
ec2:DescribeVpcs
ec2:DescribeNetworkInterfaces

Start an SSH session with the Scrutinizer server (or the primary reporter in distributed deployments), and run the following command via the scrut_util interactive CLI:
```
SCRUTINIZER> awssync
      AWS entities synced!
```

Once entity descriptions have been synced, AWS entity identifiers will automatically be replaced with their descriptions whenever an AWS-specific report is run. The awssync task will automatically be run every hour thereafter.

AWS VPC and Transit Gateway flow logs

Contents

AWS VPC and Transit Gateway flow logs#

Setting up the AWS S3 storage bucket#

AWS Transit Gateway flow log fields#

Configuring AWS VPC flow log ingestion in Scrutinizer#

Troubleshooting#

Overloaded collectors/downloaders#

Enabling role-based IAM for AWS deployments#

Importing AWS entity descriptions#