Microsoft Azure logs

Microsoft Azure logs#

When Azure virtual network flow log ingestion is configured and enabled, Scrutinizer can monitor and run reports on traffic traversing assets in the cloud.

Once flow data for network resources on Azure is being received, the following additional report types can be run:

Azure report types

Report Type	Description
Flow Decisions	Aggregation based on decision (accept or deny) applied to traffic via configured rules
Flow Decisions Count	Flow count aggregation for each traffic decision
Flow States	Aggregation based on distinct states reported for individual network flows
Flow States Count	Flow count aggregation for each network flow state
All Details	Aggregation based on full range of flow details, including the rule and application associated with the traffic
Resource IDs	Aggregation based on resource IDs

This section covers the prerequisites and setup/configuration steps for Azure flow log ingestion.

Changed in version 19.6.0: Scrutinizer now supports ingestion for VNet flow data in addition to NSG flow data.

Setting up the Azure blob storage container#

Before setting up Azure flow log ingestion in Scrutinizer, the virtual networks to be monitored should be configured to send flow logs to the Azure Storage blob container(s) that will be used. Both v1 and v2 flow logs are supported, but the latter is recommended to enable volume-based reports.

Important

Any containers used for this purpose should have versioning disabled and must be reserved for exclusive use by Scrutinizer. If the flow logs need to be archived or used for other purposes, send the flow logs to a separate blob container, and then automate the replication/duplication of those logs to the container that will be used by Scrutinizer.
If any historical data needs to be retained, it will need to be copied off the container before ingestion is enabled/configured. Manually clearing the container of inactive log files will also allow Scrutinizer to become current more quickly.

Once a blob countainer is configured as a flow log source, Scrutinizer will periodically collect the most recent 15 minutes of logs and delete all inactive log files not updated in the past ~1 hour.

Configuring Azure flow log ingestion in Scrutinizer#

To add an Azure Storage blob container as a flow log source in Scrutinizer, follow these steps:

In the Scrutinizer web interface, navigate to Admin > Integrations > Flow Log Ingestion.
Click the + icon, and then select Azure FlowLogs in the tray.
In the secondary tray, configure the container details as follows:
- Enter a name to identify the bucket/source by.
- Enter the container name:
  - For NSG flow logs, this will typically follow the format of insights-logs-networksecuritygroupflowevent
  - For VNet flow logs, this will typically follow the format of insights-logs-flowlogflowevent
- Select the collector(s) to assign to the container from the dropdown (in distributed clusters, a remote collector is recommended for this role).
- Enter the storage account name and key to use to access the container (in most cases, the service URL host name without .blob.core.windows.net/ or another domain)
- Enter the service URL for the container (in most cases, formatted as https://STORAGE-ACCOUNT-NAME.blob.core.windows.net/).
Click the Save button to add the container with the current settings.

Once added, the container will be listed in the main Admin > Integrations > Flow Log Ingestion view under the configured name. An exporter associated with the Azure virtual network will also be added to the device lists for Scrutinizer’s various functions (reports, network maps, etc.).

Note

After a container configuration has been saved, click on the name assigned to it in the main view to open the settings tray, and use the Test button to confirm that Scrutinizer is able to establish a connection to the container with the credentials entered.
To verify that the Azure flow log source has been successfully added, look for an exporter whose hostname matches the virtual network in the Explore > Exporters > By Exporters view or the Admin > Resources > Exporters page (after ~1 hour).

Troubleshooting#

If the Admin > Resources > Exporters view does not list exporters matching the virtual network(s) set up for flow ingestion, check the following for issues:

Open the tray for the ingestion source in the Admin > Integrations > Flow Ingestion view and use the Test button to verify that the collector/downloader is able to communicate with the data source using the details entered.
Verify that logs are correctly being sent to the source container.
Check the collector log file in /home/plixer/scrutinizer/files/logs/ for errors.
Check azure_log.json for possible source-side issues.

For further assistance, contact Plixer Technical Support.

Overloaded collectors/downloaders#

The Unresourced - Enabled status in the Admin > Resources > Exporters view indicates that a log source is being temporarily disabled/paused due to insufficient resources.

The following are potential solutions for an overloaded collector:

If the collector is a VM, allocate additional resources (starting with CPU cores) to it.
If the collector is ingesting logs from only one container, distribute the logs across multiple containers, which can then be assigned to different collectors.
If the collector is ingesting logs from multiple containers, distribute the containers across multiple collectors.
If the collector license has a flow rate limit, the license may need to be upgraded.