What is a Flow Replicator

Overview

Many routers, servers, and other systems can only send messages to a single log management system. The Flow Replicator™ allows a single stream of log data to be transparently replicated to multiple destinations.

By configuring these network devices to send their log and flow data to the Flow Replicator, users can control which management system(s) receive the replicated data.

A Flow Replicator eliminates the limitation of sending log and flow data to a small number of management systems.

The Basic Concepts

The Flow Replicator relies on Profiles that contain a list of devices (exporters) sending or streaming data to management systems (collectors). A packet is received by the Flow Replicator on a particular UDP port. The Flow Replicator references a list of profiles to determine if the data received from an exporter should be forwarded on to one or more collectors.

  • Collectors : A collector is a SIEM, Flow Collector, SNMPTrap Receiver, or other Network Management System that actively receives data from networked devices.
  • Exporters : An exporter is a networked device such as a router, switch, or server that generates different types of data and is capable of sending that data to a collector.
  • Profiles : A profile contains exporter(s), a listening port, collector(s), and sending UDP port.

How Profiles Work

+--------------------------------+-----------------------------------------+
| MyProfile                      | IN PORT 2002 -> OUT PORT 9996
+--------------------------------+-----------------------------------------+

Policies                   Exporters     ->    Collectors
(include) 10.3.1.1/32      10.3.1.1            10.11.1.165

When a packet is received from the exporter 10.3.1.1 on port 2002 it is replicated to the collector 10.11.1.165 on port 9996. Collectors interpret that packet’s origin as 10.3.1.1 and not the replicator.

Profiles can contain multiple exporters and collectors.

+--------------------------------+-----------------------------------------+
| distdev-63                     | IN PORT 2002 -> OUT PORT 2055
+--------------------------------+-----------------------------------------+

 Policies                   Exporters     ->    Collectors
 (include) 10.1.2.18/32     10.1.2.18           10.1.10.63
 (include) 10.4.1.1/32      10.4.1.1            10.1.4.203
 (include) 10.9.1.254/32    10.9.1.254          10.30.11.23
 (include) 192.168.0.17/32  192.168.0.17

If a packet is received from the exporters 10.1.2.18, 10.4.1.1, 10.9.1.254, or 192.168.0.7 on port 2002 it is replicated to 10.1.10.63, 10.1.4.203, and 10.30.11.23 on port 2055.

Policies

A policy is used to determine if a particular exporter should be included or excluded from the profile. Administrators add policies using CIDR notation to include or exclude exporters in a profile.

+--------------------------------+-----------------------------------------+
| steady-replays                 | IN PORT 2002 -> OUT PORT 9996
+--------------------------------+-----------------------------------------+

 Policies                  Exporters     ->    Collectors
 (include) 10.25.5.0/24    10.25.5.122         10.1.10.1
                           10.25.5.123
                           10.25.5.10
                           10.25.5.29
                           10.25.5.30

The replicator will automatically replicate packets from any exporter matching 10.25.5.x on port 2002 to collector 10.1.10.1 on port 9996.

Exclude policies can be used with include policies to exclude one or more exporters.

+--------------------------------+-----------------------------------------+
| steady-replays                 | IN PORT 2002 -> OUT PORT 9996
+--------------------------------+-----------------------------------------+

 Policies                     Exporters     ->    Collectors
 (include) 10.25.5.0/24       10.25.5.122         10.1.10.1
 (exclude) 10.25.5.10/32      10.25.5.123
                              10.25.5.29
                              10.25.5.30

The above example is identical to the previous example except it has an exclude policy. In this case, any exporter matching 10.25.5.x except 10.25.5.10 will be replicated to the collector 10.1.10.1.

Installation

There are two types of appliances available. A valid or evaluation key is required with either install. A key can be obtained from Plixer or a local reseller.

Hardware Appliance

Once the hardware appliance is installed in a network rack, power it on and follow the steps below.

  1. Using an SSH client, remotely login using the username root and password replicator. The hardware appliance will perform a quick setup and immediately reboot.
CentOS release 6.5 (Final)
Kernal 2.6.32-431.3.1.e16.x86_64 on an x86_64

localhost login: root
Passord: _
  1. Login to the hardware appliance again using the username root and password replicator. Input the answers to the configuration questions. The hardware appliance will reboot to apply the necessary settings.
Last Login: Tue Feb 11 11:067:45 on tty1

********************************
Replicator Virtual Appliance
Initial Configuration
********************************

What is the appliances static IP Address?
10.1.15.128

What is the appiances Netmask?
255.255.0.0

What is the appliances gateway?
10.1.1.1

What is the hostname for this appliance?
replicatorVA_
  1. Login to the hardware appliance command line with the replicator username and password configured in the previous step. Apply the license key by logging into the web UI or by issuing the license set command on the CLI:

    1. In the new window, under “license=”, paste in your license key.
    2. Press CTRL+x to save.

The replicator is now ready for configuration.

Virtual Appliance - ESX

The Replicator Virtual Appliance (RVA) is packaged as an all-in-one virtual machine template known as an OVF template.

For VMware deployments, ESX/ESXi 5 or higher is required. VMware Tools will be required to shut down the RVA through the VMware vSphere Client.

System Requirements

Component           Recommended Specifications
RAM                 2GB
Disks               100GB
Processor           2 CPU 2 Core 2GHz+
Operating System    ESXi5+

Deploying the OVF Template

  1. Connect to the ESX host using VMware vSphere, or vCenter.

  2. Select File then Deploy OVF Template

  3. Select Deploy from File, browse to the OVF Template, and click Next.

  4. Review the OVF template details and click Next.

  5. Define the name of the Replicator Virtual Appliance and click Next.

  6. Select a datastore and click Next.

  7. Select the disk format and click Next.

  8. Select the Network Mapping and click Next.

  9. Review the Virtual Settings and click Finish to import the OVF Template.

  10. Right click on the Flow Replicator virtual machine and power it on.

  11. Navigate to the Console tab and login using the username root and password replicator. The virtual appliance will perform a quick setup and immediately reboot.

  12. Login to the hardware appliance again using the username root and password replicator. Input the answers to the configuration questions. The virtual appliance will reboot to apply the necessary settings.

  13. Login to the hardware appliance command line with the replicator username and password configured in the previous step. Apply the license key by logging into the web UI or by issuing the license set command on the CLI:

    1. In the new window, under “license=”, paste in your license key.
    2. Press CTRL+x to save.

The replicator is now ready for configuration.

Installing VMware Tools

VMware Tools are not required for proper function of the virtual appliance. However, there are certain advantages to deploying it on each virtual appliance. See VMware’s documentation for more details.

VMware Tools are not installed by default because each version of ESX installs a different VMware Tools package. A script is included with the Virtual Replicator to simplify the install process.

  1. In the VMware vSphere Client, right click on the Replicator virtual machine and select Guest, then Install/Upgrade VMware Tools.
  2. Login to the console of the Replicator Virtual Appliance as the root user and run the command /home/replicator/conf/vmwareToolsInstall.sh

Upgrading the Virtual Machine Hardware Version

The Replicator Virtual Appliance is built on Virtual Machine Hardware Version 7 to maintain backwards compatibility with ESXi 5 hypervisors.

While the virtual machine is powered off, in vSphere (or vCenter) right click on the virtual machine and select Upgrade Virtual Hardware.

Virtual Machine – Hyper-V

System Requirements

Component   Recommended Specifications
RAM         2GB
Disks       100GB
Processor   2 CPU 2 Core 2GHz+

Importing Virtual Machine

  1. Download the latest Plixer Flow Replicator
  2. Unzip the file on your Hyper-V server
  3. Open Hyper-V Manager and select Import Virtual Machine
  4. Specify the Replicator System Folder
  5. Select the Virtual Machine
  6. Choose the import type
  7. Go to Settings
  8. Select your Network Adapter and assign it to the appropriate Virtual Switch
  9. Expand the Network Adapter section, select Advanced Features, set the MAC Address to Static, enter in a unique MAC Address, and then press “OK”.
  10. Start the Virtual Machine.
  11. Right Click on the Virtual Machine and click Connect to login to the Plixer Flow Replicator using root/replicator. The server will perform a quick setup and immediately reboot.

Virtual Machine – KVM

System Requirements

Component   Recommended Specifications
RAM         2GB
Disks       100GB
Processor   2 CPU 2 Core 2GHz+

Importing Virtual Machine

  1. Create a directory for your install
mkdir kvm/Scrut_VM_Guide/
  1. Download the latest Replicator Virtual Appliance to your KVM install
Command Line Example:
wget https://files.plixer.com/Replicator_KVM.tar.gz

Note

Contact support for latest image if the url above does not work

  1. Unzip the file on your KVM server to your new folder
sudo tar xvzf Replicator_KVM.tar.gz
  1. Run your script to install Replicator
sudo ./install.sh

At this point the machine has been created from the image that was deployed.

  1. Lastly, we just need to log into the machine now that it has been deployed. Run this command to get to the console.
virsh console Replicator

You will be prompted to login, default credentials are root/replicator. The machine will reboot and you will be asked to login again. This time, you will be presented with a shell script asking for networking information. Follow the on screen instructions and celebrate!