Elasticsearch / Kibana (ELK) Integration

Overview

What does ELK integration with Scrutinizer do?

What is ELK?

Elasticsearch - A searching service to look through the stored data collected by Logstash.

Logstash - A means to collect logs and events (like syslogs) and filter them in a specific way to be stored for later analysis.

Kibana - Front-end to present data by creating dashboards and visualizations, similar to Scrutinizer’s dashboards and gadgets.

Integration Prerequisites

Kibana 4.2

Scrutinizer v15.12+

Note

The following configuration instructions apply to Scrutinizer v16.7 and later. If an earlier version of Scrutinizer is installed, contact plixer for assistance.

Kibana Searches from Scrutinizer Reports and Alarms

The following steps will walk through how to search Kibana’s database from Scrutinizer reports.

  1. Select a Scrutinizer report that includes IP addresses

    1. Select a host of interest and click on it
    2. Select ‘Other Options’ from the Reports menu
    3. Select ‘Kibana (ELK)’
  2. The IP address and report timeframe are passed to Kibana’s search engine for detailed Kibana reporting.

For more detail (from Kibana) on Scrutinizer Alarms, follow these steps.

  1. Go to the Alarms tab in Scrutinizer and select either:

    1. Bulletin Board by Violator : Select a violator
    2. Bulletin Board by Policy : Select the policy
  2. In the Bulletin Board Events view that opens, click on the dropdown arrow to the left of the Message column for the alarm that was selected.

  3. Select ‘Kibana (ELK)’ from the Available Options menu and the Violator’s IP address and timeframe of the violation are passed to Kibana.

    • If the Violator’s IP address and an alarm time (not timeframe) are being passed, then 30 minutes before and after the alarm time is searched.

Scrutinizer Reporting from within Kibana

Within the Kibana (ELK) integration with Scrutinizer, dashboards can be setup which include:

  • Scrutinizer Vitals information
  • Flow Analytics TopN Algorithms

The Scrutinizer Vitals dashboard in Kibana can include:

  • CPU
  • Memory
  • Disk Usage
  • Flows per collector
  • Status per collector

Dashboards created with the TopN Algorithm gadgets from Flow Analytics include:

  • Top Applications
  • Top Countries
  • Top Rev 2nd lvl Domains (Top reverse 2nd level domains)
  • Top Flows
  • Top Hosts
  • Top Jitter
  • Top Networks

How to Configure ELK Integration with Scrutinizer

There are two components to the ELK Integration.

  1. Preparing Scrutinizer
  2. Importing the necessary files into Kibana’s UI

Preparing Scrutinizer

Note

Flow Analytics must be enabled and collecting statistics for the Top X Algorithms.

  1. Log on to the Scrutinizer server with administrative permissions

  2. Open the Interactive scrut_util.exe prompt with the following command:

    /home/plixer/scrutinizer/bin/scrut_util.exe

  3. And run:

    SCRUTINIZER> enable elk http://<ip:port>

    <ip:port> is the ELK server’s IP address and port

After a few moments, Scrutinizer will begin to send events to ELK.

  • To test the data export, from within the scrut_util shell, run:

    collect elk <elk_ip>

  • To disable the data export, run:

    disable elk http://<ip:port>

Preparing Kibana

Integrating ELK with Scrutinizer displays details in Kibana that have been collected and processed by Scrutinizer. For more information, visit Plixer’s Elasticsearch / Kibana Integration page.

  1. After enabling the ELK integration on Scrutinizer, refresh the index on Logstash in order to get Scrutinizer’s fields to show up. In Kibana, go to Indices > Logstash. Click on the Reload field list icon at the center top of the screen.

    ../_images/Kibana_image1.PNG
  1. Download the Kibana Integration Plugin from the Elasticsearch / Kibana Integration page.

    1. Extract the files from the scrutinizer-elk.zip file.
  2. In Kibana, go to Settings > Objects > Import > Visualizations and navigate to the elk-scrutinizer-visualizations.json file extracted in Step 2a above and click Open.

  3. Go to Settings > Objects > Import > Dashboards and navigate to the elk-scrutinizer-dashboards.json file extracted in Step 2a above and click Open.

  4. The Kibana dashboards and visualizations are now all imported and events have been configured to be coming from Scrutinizer.

  5. From the Visualize tab, scroll to the bottom and filter for a specific visualization. Typing Scrutinizer in the filter for example, will show all of the Scrutinizer visualizations.

  6. From the Dashboard tab, navigate around the various Scrutinizer dashboards imported.