Google Cloud VPC logs

Google Cloud VPC logs#

When GCP Virtual Private Cloud (VPC) flow log ingestion is configured and enabled, Scrutinizer can monitor and report on traffic data associated with GCP VPC assets.

This section covers the prerequisites and setup/configuration steps for GCP VPC flow log ingestion.

Setting up the Google Cloud Pub/Sub topic and subscription#

Scrutinizer uses the GCP Pub/Sub messaging service as an ingestion source for VPC flow logs.

To set up the Pub/Sub topic that will receive the log entries to be ingested, follow these steps:

Note

To ensure seamless access between components/services, it is highly recommended to set everything up under the project where flow logs will originate.

  1. Enable and configure VPC Flow Logs for the target resources.

  2. Next, navigate to the Pub/Sub Topics page and create a new topic with message retention enabled and set to at least one hour (other topic settings can be configured as desired).

  3. After the topic has been created, go to the Subscriptions page and create a pull subscription for the new topic (note the Subscription ID for later use).

  4. Next, go to the Log Router page and create a sink to route the log entries to the newly created topic and configure any inclusion/exclusion filters necessary.

  5. After adding the Pub/Sub topic as a sink, navigate to the Service Accounts page and select a service account associated with the sink/topic.

  6. Under the Keys tab, click the Add Key button and select JSON to download a file containing the credentials required to subscribe to the Pub/Sub topic.

Once the above steps have been completed, verify that log entries are being correctly routed to the Pub/Sub topic, and then proceed to configuring ingestion in Scrutinizer.

Configuring GCP VPC flow log ingestion in Scrutinizer#

Once the Pub/Sub topic is receiving log entries and the subscription has been set up, it can be added to Scrutinizer as follows:

  1. In the Scrutinizer web interface, navigate to Admin > Integrations > Flow Log Ingestion.

  2. Click the + icon, and then select Google Cloud Platform in the tray.

  3. In the secondary tray, configure the subscription details as follows:

    • Enter a name to identify the source by.

    • Select the Scrutinizer servers to use as the log downloader(s) and collector(s) (the primary reporter of a distributed cluster is not recommended for either role).

    • Enter the GCP project ID associated with the topic subscription.

    • Enter the subscription name/ID used.

    • Enter/paste the contents of the service account key JSON file.

  4. Click the Save button to add the subscription with the current settings.

Note

  • After a subscription configuration has been saved, click on the name assigned to it in the main view to open the settings tray, and use the Test button to confirm that Scrutinizer is able to establish a connection with the credentials entered.

  • To verify that an GCP VPC flow log source has been successfully added, look for an exporter whose hostname matches the GCP VPC in the Explore > Exporters > By Exporters view or the Admin > Resources > Manage Exporters page (after ~1 hour).

  • Flow log ingestion processes are divided between the log downloader (downloads the flow logs through the topic subscription) and the flow collector (collects and processes the downloaded logs). A different Scrutinizer server can be used for each role, and a single subscription can have multiple downloaders and collectors.

Troubleshooting#

MFSNs and a buildup of log files in flow log source containers are indications that the rate of flow and/or log generation exceeds the capacity of the collector assigned to the flow log source.

The following are potential solutions for an overloaded collector:

  • If the collector is a VM, allocate additional resources (starting with CPU cores) to it.

  • If the collector is ingesting flow logs from only one source (bucket or container), distribute the logs across multiple sources, which can then be assigned to different collectors.

  • If the collector is ingesting flow logs from multiple sources, reassign sources across multiple collectors.

  • If the collector license has a flow rate limit, the license may need to be upgraded.

Note

  • In distributed deployments, it is recommended to start with a 1:1 pairing of sources and collectors.

  • The Unresourced - Enabled status in the Admin > Resources > Exporters view is another indication that flow log sources are being temporarily disabled/paused due to insufficient resources.

If the Admin > Resources > Exporters view does not list exporters that are associated with the virtual network(s) set up for flow ingestion, do the following:

  1. Navigate to Admin > Integrations > Flow Ingestion, open the configuration tray for the collector it was assigned to, and then use the Test button to verify that the correct details were entered.

    Note

    The Test button only checks if the communication with the data source works.

  2. Verify that flow logs are correctly being sent to the bucket or container.

  3. Check the collector log file in /home/plixer/scrutinizer/files/logs/ for errors.

  4. Check awss3_log.json (AWS), azure_log.json (Azure), or ocist_log.json for possible source-side issues.

Note

The Admin > Resources > Exporters view also displays exporters that have been disabled. Because each AWS, Azure, or OCI flow log source counts as an exporter, one or more sources may be disabled automatically (in last-in/first-out order) if the exporter count limit of the current license is reached.

Contents