Elasticsearch / Kibana (ELK) integration

The integration between ELK and Scruitnizer allows Scrutinizer users to launch searches for IP addresses in Kibana. From Kibana, users can view Scrutinizer Vitals and FA TopN gadget details.

What is ELK?

Elasticsearch is a searching service to look through the stored data collected by Logstash.

Logstash ia a means to collect logs and events (like syslogs) and filter them in a specific way to be stored for later analysis.

Kibana is front-end to present data by creating dashboards and visualizations, similar to Scrutinizer’s dashboards and gadgets.

Integration prerequisites

Kibana 4.2

Scrutinizer v15.12+

Note

The following configuration instructions apply to Scrutinizer v16.7 and later. If an earlier version of Scrutinizer is installed, contact plixer for assistance.

Kibana searches from Scrutinizer reports and alarms

Follow the steps below to search Kibana’s database from Scrutinizer reports.

  1. Select a Scrutinizer report that includes IP addresses
  1. Select a host of interest and click on it;
  2. Choose Other Options from the Reports menu;
  3. Click Kibana (ELK).
  1. The IP address and report timeframe are passed to Kibana’s search engine for detailed Kibana reporting.

To get more details regarding Scrutinizer alarms in Kibana:

  1. Go to the Alarms tab in Scrutinizer and select either:
  1. Bulletin Board by Violator : Select a violator;
  2. Bulletin Board by Policy : Select a policy;
  1. In the Bulletin Board Events view that opens, click on the dropdown arrow to the left of the Message column for the alarm that was selected.
  2. Select Kibana (ELK) from the Available Options menu and the Violator’s IP address and timeframe of the violation are passed to Kibana.

Hint

If the Violator’s IP address and an alarm time (not timeframe) are being passed, then 30 minutes before and after the alarm time is searched.

Scrutinizer reporting from within Kibana

Within the Kibana (ELK) integration with Scrutinizer, you can set up dashboards, such as Scrutinizer Vitals information and Flow Analytics TopN Algorithms.

The Scrutinizer Vitals dashboard in Kibana includes:

  • CPU
  • Memory
  • Disk Usage
  • Flows per collector
  • Status per collector

Dashboards created with the TopN Algorithm gadgets from Flow Analytics contain:

  • Top Applications
  • Top Countries
  • Top Rev 2nd lvl Domains (Top reverse 2nd level domains)
  • Top Flows
  • Top Hosts
  • Top Jitter
  • Top Networks

How to configure ELK integration with Scrutinizer

Preparing Scrutinizer

Note

Flow Analytics must be enabled and collecting statistics for the Top X Algorithms.

  1. SSH into the Scrutinizer server as the plixer user.
  2. Use the interactive scrut_util command:
  /home/plixer/scrutinizer/bin/scrut_util

  **SCRUTINIZER>** enable elk http://<ip:port>

where <ip:port> is the ELK server's IP address and port.
  1. After a few moments, Scrutinizer will begin to send events to ELK.
  • To test the data export, from within the scrut_util shell, run:
collect elk <elk_ip>
  • To disable the data export, run:
disable elk http://<ip:port>

Preparing Kibana

Integrating ELK with Scrutinizer displays details in Kibana that have been collected and processed by Scrutinizer. For more information, visit Plixer’s Elasticsearch / Kibana Integration page.

  1. After enabling the ELK integration for Scrutinizer, refresh the index on Logstash in order to get Scrutinizer’s fields to show up. In Kibana, go to Indices > Logstash. Click on the Reload field list icon at the center top of the screen.

    ../_images/Kibana_image1.PNG
  1. Download the Kibana Integration Plugin from the Elasticsearch / Kibana Integration page. Extract the files from the scrutinizer-elk.zip file.
  2. In Kibana, go to Settings > Objects > Import > Visualizations and navigate to the elk-scrutinizer-visualizations.json file extracted in Step 2a above and click Open.
  3. Go to Settings > Objects > Import > Dashboards and navigate to the elk-scrutinizer-dashboards.json file extracted in Step 2a above and click Open.
  4. The Kibana dashboards and visualizations are now all imported and events have been configured to be coming from Scrutinizer.
  5. In the Visualize tab, scroll to the bottom and filter for a specific visualization. Typing Scrutinizer in the filter for example, will show all of the Scrutinizer visualizations.
  6. In the Dashboard tab, navigate around the various Scrutinizer dashboards imported.