Microsoft Azure logs

When Azure virtual network flow log ingestion is configured and enabled, Scrutinizer can monitor and run reports on traffic traversing assets in the cloud.

Once flow data for network resources on Azure is being received, the following additional report types can be run:

Azure report types

Report Type	Description
Flow Decisions	Aggregation based on decision (accept or deny) applied to traffic via configured rules
Flow Decisions Count	Flow count aggregation for each traffic decision
Flow States	Aggregation based on distinct states reported for individual network flows
Flow States Count	Flow count aggregation for each network flow state
All Details	Aggregation based on full range of flow details, including the rule and application associated with the traffic
Resource IDs	Aggregation based on resource IDs

This section covers the prerequisites and setup/configuration steps for Azure flow log ingestion.

Changed in version 19.6.0: Scrutinizer now supports ingestion for VNet flow data in addition to NSG flow data.

Setting up the Azure blob storage container

Before setting up Azure flow log ingestion in Scrutinizer, the virtual networks to be monitored should be configured to send flow logs to the Azure Storage blob container(s) that will be used. Both v1 and v2 flow logs are supported, but the latter is recommended to enable volume-based reports.

Important

• Any containers used for this purpose should have versioning disabled and must be reserved for exclusive use by Scrutinizer. If the flow logs need to be archived or used for other purposes, send the flow logs to a separate blob container, and then automate the replication/duplication of those logs to the container that will be used by Scrutinizer.

• If any historical data needs to be retained, it will need to be copied off the container before ingestion is enabled/configured. Manually clearing the container of inactive log files will also allow Scrutinizer to become current more quickly.

Once a blob countainer is configured as a flow log source, Scrutinizer will periodically collect the most recent 15 minutes of logs and delete all inactive log files not updated in the past ~1 hour.

Configuring Azure flow log ingestion in Scrutinizer

To add an Azure Storage blob container as a flow log source in Scrutinizer, follow these steps:

1. In the Scrutinizer web interface, navigate to Admin > Integrations > Flow Log Ingestion.

2. Click the + icon, and then select Azure FlowLogs in the tray.

3. In the secondary tray, configure the container details as follows:

   • Enter a name to identify the bucket/source by.

   • Enter the container name:

     • For NSG flow logs, this will typically follow the format of insights-logs-networksecuritygroupflowevent

     • For VNet flow logs, this will typically follow the format of insights-logs-flowlogflowevent

   • Select the collector(s) to assign to the container from the dropdown (the primary reporter of a distributed cluster is not recommended).

   • Enter the storage account name and key to use to access the container (in most cases, the service URL host name without .blob.core.windows.net/ or another domain)

   • Enter the service URL for the container (in most cases, formatted as https://STORAGE-ACCOUNT-NAME.blob.core.windows.net/).

4. Click the Save button to add the container with the current settings.

Once added, the container will be listed in the main Admin > Integrations > Flow Log Ingestion view under the configured name. An exporter associated with the Azure virtual network will also be added to the device lists for Scrutinizer’s various functions (reports, network maps, etc.).

Note

• After a container configuration has been saved, click on the name assigned to it in the main view to open the settings tray, and use the Test button to confirm that Scrutinizer is able to establish a connection to the container with the credentials entered.

• To verify that the Azure flow log source has been successfully added, look for an exporter whose hostname matches the virtual network in the Explore > Exporters > By Exporters view or the Admin > Resources > Exporters page (after ~1 hour).

Troubleshooting

MFSNs and a buildup of log files in flow log source containers are indications that the rate of flow and/or log generation exceeds the capacity of the collector assigned to the flow log source.

The following are potential solutions for an overloaded collector:

• If the collector is a VM, allocate additional resources (starting with CPU cores) to it.

• If the collector is ingesting flow logs from only one source (bucket or container), distribute the logs across multiple sources, which can then be assigned to different collectors.

• If the collector is ingesting flow logs from multiple sources, reassign sources across multiple collectors.

• If the collector license has a flow rate limit, the license may need to be upgraded.

Note

• In distributed deployments, it is recommended to start with a 1:1 pairing of sources and collectors.

• The Unresourced - Enabled status in the Admin > Resources > Exporters view is another indication that flow log sources are being temporarily disabled/paused due to insufficient resources.

If the Admin > Resources > Exporters view does not list exporters that are associated with the virtual network(s) set up for flow ingestion, do the following:

1. Navigate to Admin > Integrations > Flow Ingestion, open the configuration tray for the collector it was assigned to, and then use the Test button to verify that the correct details were entered.

   Note

   The Test button only checks if the communication with the data source works.

2. Verify that flow logs are correctly being sent to the bucket or container.

3. Check the collector log file in /home/plixer/scrutinizer/files/logs/ for errors.

4. Check awss3_log.json (AWS), azure_log.json (Azure), or ocist_log.json for possible source-side issues.

Note

The Admin > Resources > Exporters view also displays exporters that have been disabled. Because each AWS, Azure, or OCI flow log source counts as an exporter, one or more sources may be disabled automatically (in last-in/first-out order) if the exporter count limit of the current license is reached.