Replicator load balancing

The load balancing feature provides integration between a Scrutinizer distributed cluster and a Replicator appliance. Load balancing for multiple replicators is covered in the advanced configuration section.

Setting up load balancing

Load balancing requires you add credentials in the UI of Scrutinizer for your replicator, and running the autoreplicate binary from the command line of the reporter twice. The initial run will generate a config file, the second will start the processing of exporters.

Default Configuration

1. Fill out the replicator credentials to connect to, navigate Admin > Settings > Plixer Replicator
2. SSH as the plixer user into your Scrutinizer primary reporter.
3. Run the following command (Note this is NOT run inside interactive CLI):

scrut_util --autoreplicate

When you launch the command for the first time, it will create a configuration file called /home/plixer/scrutinizer/files/autoreplicate.conf. The file will be pre-populated with the necessary entries as JSON.

4. Edit the /home/plixer/scrutinizer/files/autoreplicate.conf file for your setup. More information about the configuration file is detailed in the Configuration section

5. Rerun the scrut_util –autoreplicate command. It will create a replicator policy for each collector and add exporters that are in the seed_profile to collector profiles up to collector_capacities. After you run the auto-replicate command, Scrutinizer will create an alarm that will contain the configuration details.

Note

If you would like Replicator to forward the traffic from all the exporters to Scrutinizer, add a policy for “0.0.0.0/0” to your seed profile.

If you are running in the default mode as scrut_util from the bin directory, the configuration file will be /home/plixer/scrutinizer/files/autoreplicate.conf

Advanced Configuration

Advanced options are available such as running multiple autoreplicate binaries and pooling.

With the help of Plixer Support, additional binaries can be created and a directory setup to allow for replicator pooling.

If the binary you are running is named ‘replicator_util’ or the directory path it is in contains ‘replicator_pools’ , the configuration file will be created / referenced in the same directory as the binary you are running.

Additional parameters added to the configuration file will take precendence over what was set in the UI

Those optional parameters are:

replicator_host

replicator_pass

replicator_seed_profile

replicator_receive_port

replicator_send_port

Passwords can be AES256 encrypted in the conf file or as plain text. To avoid using plain text password, generate an encrypted value use this command, then put the output in the configuration file as replicator_pass:

--autoreplicate --encrypt YOUR_PASSWORD

Editing the configuration file

Here are a few helpful tips for modifying the /home/plixer/scrutinizer/files/autoreplicate.conf file.

1. When editing the collector_capacities section, make sure you have an entry for each collector in the cluster. If you add a new collector, update the configuration accordingly. A policy is automatically created and managed for each collector in the Scrutinizer distributed cluster.
2. The exporters value is the maximum number of exporters that should be sending flows to a collector.
3. The flow_rate setting controls the maximum flows per second the collector should receive.
4. The seed_profile stands for the replicator profile that you add all exporters to on the replicator appliance. Any exporters added to a designated policy on a replicator will be sent to a collector in the distributed cluster.

Important

The collector_capacities setting limits the number of exporters. If you exceed the capacity, exporters will be removed.

FAQs

Q: If a new collector is added to the Scrutinizer cluster does Scrutinizer automatically start balancing exporters to it?
A: No. Configuration file needs to be updated. You can manually add the collector to the configuration file and then subsequently run the scrut_util –autoreplicate ( or wait for next scheduled* run, if configured.)

Q: How does auto-replicate handle a collector going offline? Can it detect a down collector and re-balance the cluster to send flows elsewhere?
A: No. It does not automatically detect down collectors.

Q: The collectors within my cluster are unbalanced. Why is auto-replicate not distributing the load evenly across all of the collectors?
A: Every exporter sends flows at a different rate; that rate changes throughout the day. When exporters are added to a collector, the assumed rate of 200 flows/second, is used. As the real rate is observed, the exporter will only be moved when a collector is pushed outside of its defined limits.

Q: How does auto-replicate determine where new exporters will be assigned?
A: Exporter placement is determined by a round-robin of all collectors defined in the autoreplicate.conf file, until a collector is over its limits.

Q: When a collector is over provisioned, how does Scrutinizer determine which collector the exporter will be moved to?
A: Exporter placement is determined by a round-robin of all collectors defined in the autoreplicate.conf file, until a collector is over its limits.

Q: My auto-replicate collector threshold is set to 40,000 flows per second but I’m seeing times every day where the collector is getting more than 40,000 flows/s. Why are exporters not being moved to other collectors when the peak threshold is being crossed?
A: Flowrate is based on a 24-hour average.

Q: How many Replicators does this functionality support?
A: One replicator can be defined in the Scrutinizer UI, however unlimited Replicators are supported with manual configuration. See the Advanced Configuration section of the manual regarding autoreplicate.

Q: How many Replicator seed profiles does this functionality support?
A: One per autoreplicate configuration. See Advanced Configuration section for details.

Q: How many Replicator unique listening ports does this functionality support?
A: One per autoreplicate configuration. See Advanced Configuration section for details.

Q: The manual says to run the scrut_util -autoreplicate command. Why do I need to do this manually and do I always need to do this manually?
A: Autoreplicate is not scheduled to run by default. Users have the option to cron this to run whenever they would like.

Q: How often does auto-replicate run?
A: Autoreplicate is not scheduled to run by default. Users have the option to cron this to run whenever they would like.

Q: Can I statically assign some exporters to some collectors with auto-replicate?
A: No. For static exporters create a new replicator profile, as you normally would, to send additional exporters to your collector.

Q: How do exporters that are sending directly to Scrutinizer and not going through a Replicator impact auto-balancing?
A: They are referred to as ‘Rogue Exporters’ and will count against the collectors exporter count and flow limits.

Q: What happens if all of the collectors are exceeding their defined collections rates and new exporters are added to the replicator? Where do they go?
A: Those exporters will not be placed anywhere.

Q: Where does Scrutinizer log the auto-replicate changes that are being made?
A: Output can be found at /home/plixer/scrutnizer/files/logs/ inside a epoch-stamped file which will contain a final output file after the autoreplicate command completes.

Q: Does auto-replicate take into account MFSN’s when balancing?
A: No it does not.

Q: Does old profiles are automatically removed after the configuration changes?
A: No it does not. The UI profile descriptions says not to edit by default, but it is safe to delete profiles no longer relevant to your config file.