Configuring Amazon Web Services flow streaming

Overview

The integration between Amazon Web Services (AWS) and Scrutinizer provides insight into network traffic destined for AWS such as:

  • AWS connection latency
  • Top AWS users
  • Top AWS applications
  • Overall traffic load of AWS hosted applications

After configuring Amazon Web Services using Amazon Kinesis to stream logging data to Scrutinizer, the following AWS specific flow reports are available in Scrutinizer.

  • Action
  • Action with Interface
  • Action with Interface and Dst
  • Action with Interface and Src
  • Interface
  • Pair Interface
  • Pair Interface Action

Prerequisites

A minimum of Scrutinizer v16.10 is required for the configuration to successfully complete and for the AWS reports to be available.

Required Information

Before starting the AWS configuration in Scrutinizer, the following in information needs to be collected.

  1. AWS Account ID

    1. In the AWS interface, click on the name in the upper right hand corner.
    2. Select ‘My Account’.
    3. The Account ID is under ‘Account Settings’.
  2. What region will be monitored?

    1. Click the AWS console home link (AWS logo in upper left hand corner).
    2. The region will be in the url. Example: us-west-2
  3. What VPC will be monitored?

    1. Go to AWS > VPC > Your VPCs
    2. Make note of the VPC ID needed to collect flow data from.

Create an AWS User for Flow Streaming

The next step is to create an AWS user for flow streaming.

  1. Go to AWS > Security, Identity & Compliance > IAM

  2. Users > Add User

  3. Enter a user name. Example: ‘ScrutinizerConsumer’

  4. Select ‘Programatic access’

  5. Click ‘Next: Permissions’

  6. Select ‘Attach existing policies directly’, Check the following policies:

    • AmazonEC2ReadOnlyAccess

    • CloudWatchFullAccess

    • AmazonDynamoDBFullAccess

    • AmazonKinesisFullAccess

    • AdministratorAccess

      • This policy can be removed from this user after the configuration is complete.
  7. Click ‘Next: Review’

  8. Verify the policies and access type from above.

  9. Click ‘Create User’

  10. Click ‘Download .csv’ (to be used in ‘AWS Configuration in Scrutinizer’ below)

AWS Configuration in Scrutinizer

Next we’ll add the AWS configuration settings in the Scrutinizer user interface.

  1. Add the following ScrutinizerConfig credentials in Admin > Settings > AWS Configuration

    1. AWS access key ID

      This is included in the credentials downloaded for the flow streaming AWS user just created.

    2. AWS Region Name

      Gathered in the Required Information step.

    3. AWS secret access key

      This is included in the credentials downloaded for the flow streaming AWS user just created.

    4. AWS Stream Name

      Defaults to ‘ScrutinizerStream’ and is generated in step 5 (aws_config.exe).

    5. Flow Collector IP

      IP Address of the server where step 2 (–enable aws) will be run.

    6. Flow Collector Port

      Default 4739

  2. Run /home/plixer/scrutinizer/bin/scrut_util.exe –enable aws

Note

The first time this is run on a server that has been upgraded from an earlier version of Scrutinizer, it will take a few minutes to install all the required libraries.

  1. Change to the bin directory:

    1. cd /home/plixer/scrutinizer/bin/

Note

Step 4 is for versions 16.10 and 16.11 only. The aws_config.exe file is included with the install and upgrade of versions above 16.11.

  1. Download aws_config.exe to the /home/plixer/scrutinizer/bin/ directory.

    1. wget http://files.plixer.com/support/aws_config.exe
    2. chmod 750 aws_config.exe
  2. Run ./aws_config.exe –configure –region XXX –account_id YYY –vpc_id ZZZ

    1. region, account_id, and vpc_id are all from the information gathered in the Required Information section.
    2. This will create the AWS Stream Name ‘ScrutinizerStream’.
  3. Restart the AWS service

    1. Run: service plixer_aws restart

      • It can take 10 minutes before the exporter can be seen.

Note

If the exporter is not seen within 10 minutes, verify that the exporter is not disabled in Admin > Definitions > Manage Exporters.

  1. Detach the AdministratorAccess policy from the AWS user.

Frequently Asked Questions

Q: Why do I see “Unable to load AWS credentials” installing modules?
A: Amazon’s package tests assume AWS connectivity is already in place. These can be ignored.
Q: Why are there gaps in the data in 1m and 5m intervals?
A: Amazon flow logs are updated every 10 minutes.
Q: It isn’t working, how can I see what is going on?
A: Turn on debugging and review the log files (see the ‘Troubleshooting AWS’ section below).
Q: All commands ran successfully but I don’t see the exporter?
A: If no exporter appears after 10 minutes, verify that the exporter is not disabled in Admin > Definitions > Manage Exporters.
Q: How do I add resources to the AWS instance?
A: Visit the page on AWS Adding Resources .

Troubleshooting AWS

Overview

AWS integration involves a number of moving parts.

  • On the Amazon side of things:

    • Cloudwatch logs are streamed to Kinesis for consumption.
  • On the Scrutinizer side:

    • The plixer_aws daemon takes settings from ‘Admin > Settings > AWS Configuration’ and connects to the Kinesis stream. That data is then sent to Scrutinizer as IPFIX.

Enabling

If upgrading from a version prior to 16.10, required libraries will be installed the first time this command is run ‘./scrut_util.exe –enable aws’. This will also install the plixer_aws service.

Running

When the service is started all necessary configuration files are rebuilt. This means that changes to settings require a restart of the service.

service plixer_aws restart

A plixer_aws.log file is created under: /home/plixer/scrutinizer/files/logs/. It will show the service start time.

Debugging

  1. Stop the plixer_aws service.
  2. Go to /home/plixer/scrutinizer/bin/
  3. Run this command: ‘./scrut_util.exe –enable aws –debug’
  4. Go to /home/plixer/scrutinizer/files/
  5. Run this command: ‘./kinesis.sh’
  6. Collect plixer_aws_debug.log and plixer_aws.log files from /home/plixer/scrutinizer/files/logs