Amazon Web Services VPC flow logs¶

With AWS Virtual Private Cloud (VPC) flow log ingestion enabled, Plixer Scrutinizer is able to report additional insights for network traffic destined for AWS, including top AWS users and applications, as well as traffic load generated by AWS-hosted applications.

The following AWS-specific report types become available to run:

Action	Aggregation based on the Action (`ACCEPT`, `REJECT`, or `DROP`) associated with the traffic
Action with Interface	Aggregation based on the action applied and the interface associated with the flow
Action with Interface and Dst	Aggregation based on the action applied, the associated interface, and the traffic’s destination
Action with Interface and Src	Aggregation based on the action applied, the associated interface, and the traffic’s source
Availability Zones	Aggregation based on the AWS Availability Zone associated with the traffic
Dst Service	Aggregation based on the AWS service the traffic was destined for
Interface	Aggregation based on the source or destination interface associated with the traffic
Pair Interface	Aggregation based on the source and destination interfaces associated with the traffic
Pair Interface Action	Aggregation based on the Action applied and the source and destination interfaces of the traffic
Src Service	Aggregation based on the AWS service the traffic originates from
Src Service-Dst Service	Aggregation based on AWS services the traffic originated from and was destined for
Traffic Path	Aggregation based on the traffic path used by egress traffic to reach its destination
VPCs	Aggregation based on the VPC ID associated with the traffic

Setting up S3 storage

Before setting up AWS VPC flow log ingestion in Plixer Scrutinizer, the Amazon S3 storage bucket(s) that will be used should be configured as follows:

Set the VPC(s) to be monitored to send flow logs to the bucket. The flow log format must include the following fields:
- log-status
- vpc-id
- interface-id
- flow-direction
The bucket should be reserved for exclusive use by Plixer Scrutinizer. If the flow logs need to be archived or used for other purposes, send the flow logs to a separate S3 bucket, and then automate the replication/duplication of those logs to the bucket that will be used by Plixer Scrutinizer.
Versioning should be disabled.

Note

When upgrading from older versions of Plixer Scrutinizer, it may be necessary to delete the existing VPC flow log configuration and create a new one that includes the interface-id and flow-direction fields.
When creating a VPC flow log, leaving the Maximum Aggregation Interval setting at the default 10 minutes will minimize processing load on the Plixer Scrutinizer collector at the cost of longer update times and data spikes. Setting the maximum aggregation interval to 1 minute will result in more granular data but also increase resource utilization.
After an S3 bucket is first configured for ingestion, Plixer Scrutinizer purges all older flow logs from the bucket before starting to collect and delete the most recent 15 minutes of logs as normal. If any historical data needs to be retained, it should be copied off the bucket before the integration is configured. Manually clearing the bucket of any log data older than 15 minutes will also allow Plixer Scrutinizer to become current more quickly.

Configuring AWS VPC flow log ingestion

To add an S3 bucket as a flow log ingestion source in Plixer Scrutinizer, follow these steps:

Navigate to Admin > Integrations > Flow Log Ingestion in the web interface.
Click the + button and select AWS VPC FlowLogs in the tray.
In the secondary tray, configure the bucket for flow log ingestion as follows:
- Enter a name identify the bucket/source by.
- Select the Plixer Scrutinizer server to use as the log downloader for the bucket.
- Select the Plixer Scrutinizer server to use as the collector for the bucket.
- Enter the name of the bucket.
- Select the AWS region where the bucket is hosted from the dropdown.
- Enter the credentials to use to access the bucket (AWS access key ID and secret access key)
Click the Save button to add the bucket with the current settings.

Once added, the bucket will be listed in the main Admin > Integrations > Flow Log Ingestion view under the configured name. An exporter associated with the VPC will also be added to the device lists for Plixer Scrutinizer’s various functions (Flow Analytics, network maps, reports, etc.).

Note

The credentials entered must grant full access to the bucket.
To verify that an AWS VPC flow log source has been successfully added, look for an exporter labeled vpc- in the Explore > Exporters > By Exporters view or the Admin > Resources > Manage Exporters page (after ~1 hour).
Flow log ingestion processes are divided between the log downloader (downloads the flow logs from the bucket) and the flow collector (collects and processes the downloaded logs). A different Plixer Scrutinizer server can be used for each role. Multiple downloaders and collectors can also be assigned to a single bucket.
In distributed environments, avoid using the primary reporter as a log downloader or flow collector.

For assistance with any issues, consult the troubleshooting guide or contact Plixer Technical Support.

Enabling role-based IAM for AWS deployments

Role-based IAM can be enabled for Plixer Scrutinizer AMI instances by ticking the checkbox in the configuration tray. The role assigned to the EC2 instance should be provisioned with the following permissions:

{ "Version": "2012-10-17",
  "Statement": \[
             { "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": \[ "s3:GetObject", "s3:DeleteObject" \],
               "Resource": \[ "arn:aws:s3:::<S3BUCKET>/\*" \]
             },
             { "Sid": "VisualEditor1",
               "Effect": "Allow",
               "Action": "s3:\*",
               "Resource": "arn:aws:s3:::<S3_BUCKET_NAME>"
             }
  \]
}

Note

Role based authentication is only available when all Log Downloaders are hosted in AWS.

Importing AWS entity descriptions

To allow description reporting and filtering by AWS entity identifiers (interface-id, vpc-id, etc.) directly in the Plixer Scrutinizer UI, follow these steps:

Provision the user or IAM role with the following additional permissions:

ec2:DescribeInstances
ec2:DescribeSubnets
ec2:DescribeVpcs
ec2:DescribeNetworkInterfaces

Start an SSH session with the Plixer Scrutinizer server (or the primary Reporter in distributed deployments), and run the following command via the scrut_util interactive CLI:
```
SCRUTINIZER> awssync
      AWS entities synced!
```

Once entity descriptions have been synced, AWS entity identifiers will automatically be replaced with their descriptions whenever an AWS-specific Report is run. The awssync task will automatically be run every hour thereafter.