Auto Replicate

Auto Replicate#

Auto Replicate is an additional Replicator integration for Scrutinizer/Plixer One deployments that allows flow streams to all be sent to a single Replicator instance, which will then automatically distribute them across specified collectors in the same Scrutinizer distributed cluster based on their available capacity.

Rebalancing process#

Auto Replicate uses two Replicator profile types to define exporters and collectors for autoreplication:

  • Seed profile: Contains inclusion and exclusion policies to define a pool of source exporters to be managed through Auto Replicate

  • Collector profile: Contains collector details, including maximum exporter count and flow rate, and must be associated with the seed profile to add destination collectors for an Auto Replicate cluster (one profile per collector)

Rebalancing action sequence#

Once the necessary profiles have been set up, enabling the seed profile will start the following rebalancing sequence:

  1. All exporters that are currently sending flow data but not assigned to a collector are identified.

  2. The unassigned exporters are assigned, prioritizing collectors with the most available capacity (exporter count and flow rate).

  3. Collectors are checked, and exporters are reassigned from any collectors that have exceeded their capacity limits (exporter count or flow rate).

  4. Actions are reported in the Alarm Monitor using the following policies:

    • Auto Replicate ran: Triggered whenever the Auto Replicate sequence is automatically or manually run

    • Auto Replicate exporter added: Triggered when a new exporter is discovered or when an exporter is assigned to a collector

    • Auto Replicate exporter removed: Triggered when an exporter is unassigned from a collector

    • Auto Replicate error: All configured collector limits reached (requires collector reconfiguration or additional collectors)

By default, the full sequence is automatically re-run every 24 hours, and a check to assign new and unassigned exporters is run every hour. The sequence can also be manually run from the web interface or via API.

Additional logic#

  • No additional exporters will be assigned to a collector that has reached its configured exporter count or flow rate limit. If either limit is exceeded, exporters will be reassigned from the collector when the load distribution sequence is next run (either automatically or manually).

  • Auto Replicate uses a weighted scoring system (see below) to assess collector utilization to optimize exporter assignments. Any unassigned exporter is always assigned to the collector with the lowest resulting composite score.

  • When reassigning exporters from an overloaded collector, the exporter that was most recently added is unassigned first (LIFO).

  • If an exporter is offline/inactive for 15 minutes (can be changed by editing active_exporter_minutes in /home/plixer/scrutinizer/files/conf/rebalance.yaml), it will be unassigned from its current collector.

Composite Utilization Score#

A collector’s Composite Utilization Score is the weighted sum of its current exporter count and flow rate utilization values (e.g., current exporters / max exporters).

By default, the system favors exporter count (70%) over flow rate (30%), but this weighting can be tuned by changing the exporter_count_weight value in /home/plixer/scrutinizer/files/conf/rebalance.yaml to one of the following:

Value

Behavior

1.0

Exporter count only; exporters are distributed as evenly as possible regardless of flow rate differences

0.7

Strongly favors even exporter distribution while still considering flow rates (Default)

0.5

Exporter count and flow rate are considered equally

0.3

Strongly favors even flow rate distribution while still considering exporter counts

0.0

Flow rate only; flow rates are kept as even as possible regardless of exporter distribution

Setup and configuration#

Auto Replicate is enabled by creating a collector profile for each destination collector and associating them with a seed profile with the necessary exporter inclusion/exclusion policies.

Profiles must be created on the local Replicator instance on a Scrutinizer primary reporter or a headless deployment registered with the primary reporter. Destination collectors for autoreplication must be registered with the same primary reporter, which will have access to all collector configurations and current loads.

Note

Multiple seed profiles can be created to enable autoreplication for separate collector groups. All seed profiles are automatically discovered and processed when rebalancing and assigning exporters.

Creating a collector profile#

To create a new collector profile:

View instructions

  1. Navigate to Replicator > Collectors, and then click the + icon to create a new collector profile.

  2. In the Add Replicator Profile tray, enter a name for the profile.

  3. Select Auto Replicate Collector as the profile type, and then select the Replicator instance to create the profile on.

  4. Enter the collector’s IP address and port number to use.

  5. Enter an exporter count limit and a flow rate limit for the collector.

  6. [OPTIONAL] Add a description for the collector/profile.

  7. Click Save.

Repeat the above steps to create a collector profile for each destination collector, and then proceed to create the seed profile.

Creating a seed profile

To create a new seed profile:

View instructions

  1. Navigate to Replicator > Profiles, and then click the + icon to create a new profile.

  2. In the Add Replicator Profile tray, enter a name for the profile.

  3. Select Auto Replicate Seed as the profile type in the dropdown, and then select the Replicator instance to create the profile on.

  4. [OPTIONAL] Add a description for the collector/profile.

  5. Click Save, and then return to the main Profiles view.

  6. Click on the newly created profile to open the configuration tray.

  7. Create inclusion and exclusion policies to define source exporters for autoreplication.

  8. Select the collector profiles of all destination collectors to associate with the seed profile (only collector profiles not currently associated with a seed profile can be selected).

  9. Click Save.

Once the seed profile has been configured, enabling it will start Auto Replicate and immediately run the load distribution sequence.

Manual configuration#

Autoreplication parameters can be manually configured by editing the following values in /home/plixer/scrutinizer/files/conf/rebalance.yaml:

Option

Type

Default

Description

api_port

int

8082

API server port

api_enabled

bool

true

Enable/disable the API server

rebalance_interval

int

1440 (24 hours)

Interval between full rebalances (in minutes)

add_exporter_interval

int

60 (1 hour)

Interval between adding unassigned exporters only (in minutes)

active_exporter_minutes

int

15

Exporter inactivity timeout (in minutes)

exporter_count_weight

float

0.7

Exporter count weighting in the Composite Utilization Score

log_level

string

info

Sets the logging level (debug, info, warn, error)

log_directory

string

/home/plixer/scrutinizer/files/logs

Specify the directory to save log files to

api_log_file

string

autoreplicate.log

Specify the name of the API log file

level_tolerance

float

0.05

Composite Utilization Score tolerance for the CLI -level runmode (0.0–1.0); collectors within this score range are considered balanced

assumed_exporter_rate

float

200

Fallback flow rate for exporters without observed metrics (auto-updated to the median observed rate after -level/-levelall is run)

generate_text_reports

bool

false

Enable/disable daily text summary report generation

text_report_directory

string

/home/plixer/scrutinizer/files/logs

Specify the directory to save text summary reports

Note

A different configuration file can be used by running:

/home/plixer/scrutinizer/bin/rebalance -config /PATH/TO/CUSTOM_CONFIG_FILENAME.yaml

Activity logging#

Auto Replicate uses Zerolog for structured, JSON-based logging and supports multiple logging levels for ease of monitoring and troubleshooting.

CLI mode#

When rebalance is run from the CLI, the logging level (via stderr) defaults to warn (see below).

To change the terminal logging level use the following runmodes:

info level:

/home/plixer/scrutinizer/bin/rebalance -verbose

debug level:

/home/plixer/scrutinizer/bin/rebalance -debug <-minutes [TIME_WINDOW_IN_MINUTES]>

API daemon mode#

When rebalance is running in API daemon mode, logs are saved to /home/plixer/scrutinizer/files/logs/rebalance.log.

Logging levels#

The logfile logging level for Auto Replicate activity can be modified by changing log_level in /home/plixer/scrutinizer/files/conf/rebalance.yaml to one of the following:

  • debug: Per-item details (individual policy changes, capacity calculations, function entry/exit), profile loading, exporter metrics

  • info: Process lifecycle (start/stop), per-collector capacity status, rebalance summaries, decision outcomes, policy update totals, manually managed exporters

  • warn: Only actionable events requiring attention (overloaded collectors, unassignable exporters, collectors without Auto Replicate profiles)

  • error: Only critical errors preventing operation

CLI operations#

Rebalancing operations can be manually initiated from the CLI by running:

/home/plixer/scrutinizer/bin/rebalance

CLI mode options:#

When running rebalance via CLI, the following additional options are available:

  • -debug: Enable debug-level logging

  • -verbose: Enable info-level logging

  • -minutes: Lookback window for exporter rate analysis in minutes (default: 2880 = 48 hours)

  • -seed-profile-id: Specify a seed profile ID to rebalance, -level, or -levelall (0 = all seed profiles)

  • -level: Reassign most recently assigned exporters from over-target collectors to under-target collectors (gentle redistribution)

  • -level-tolerance: Specify a score tolerance for -level mode (0.0–1.0, default: 0.05 = 5%); only collectors outside this tolerance band are considered over-/under-target

  • -levelall: Delete all policies automatically created by Auto Replicate and reassign all exporters by descending rate (full reset)

  • -help: Show help message

Note

  • The default -level-tolerance value can be set by changing level_tolerance in /home/plixer/scrutinizer/files/conf/rebalance.yaml.

  • After -level is run, assumed exporter flow rates are updated to the median observed rate and saved to /home/plixer/scrutinizer/files/conf/rebalance.yaml.

  • -level and -levelall will not run if any Auto Replicate policies were recently modified (cooldown timer is the same as the exporter inactivity timeout).

API daemon mode options:#

To run rebalance as an API daemon server (with scheduling), run the following:

/home/plixer/scrutinizer/bin/rebalance -daemon

In API daemon mode, the following additional options are available:

  • -api-port: Set the API server port (default: 8082)

  • -api-enabled: Enable API server (default: true)

  • -log-level: Set the logging level to debug, info, warn, error (default: info)

  • -config: Set the configuration file path (default: /home/plixer/scrutinizer/files/conf/rebalance.yaml)

  • -rebalance-interval: Set the interval between full rebalances in minutes (default: 1440)

  • -add-exporter-interval: Set the interval between adding unassigned exporters only in minutes (default: 60)

  • -active-exporter-minutes: Set the exporter inactivity timeout in minutes (default: 15)

  • -exporter-count-weight: Set the exporter count weighting in the Composite Utilization Score (default: 0.7)

  • -generate-text-reports: Enable daily summary text report generation (default: false)

  • -text-report-directory: Specify the directory for text summary reports (default: /home/plixer/scrutinizer/files/logs)

Note

The above settings can also be configured in rebalance.yaml or another specified configuration file.

API endpoints#

When Auto Replicate is running as an API daemon server, the following endpoints are available:

Note

API access requires an API authentication token.

  • GET /health - Get service health status (no authentication required)

  • GET /api/v1/status - Get service status and statistics

  • GET /api/v1/operations - List all operations (active and completed)

  • GET /api/v1/operations/{operation_id} - Get operation details for the specified operation_id

  • POST /api/v1/rebalance/trigger - Trigger a rebalance operation

  • GET /api/v1/rebalance/status/{operation_id} - Get operation status and progress for the specified operation_id

  • GET /api/v1/rebalance/history - Get rebalance operation history

  • GET /api/v1/profiles - List all seed profiles

  • GET /api/v1/profiles/{profile_id}/status - Get detailed profile status for the specified profile_id

API call examples#

View examples

Trigger rebalance for all seed profiles:

curl -X POST http://localhost:8082/api/v1/rebalance/trigger \
  -H "Content-Type: application/json" \
  -d '{"minutes": 2880}' \
  --cookie "sessionid=your_session_id"

Trigger rebalance for a specific profile:

curl -X POST http://localhost:8082/api/v1/rebalance/trigger \
  -H "Content-Type: application/json" \
  -d '{"seed_profile_id": 1, "minutes": 1440}' \
  --cookie "sessionid=your_session_id"

Check operation status:

curl http://localhost:8082/api/v1/rebalance/status/rebalance-20250814-093500-1 \
  --cookie "sessionid=your_session_id"

Get service status:

curl http://localhost:8082/api/v1/status \
  --cookie "sessionid=your_session_id"

Get rebalance history:

curl http://localhost:8082/api/v1/rebalance/history?limit=10 \
  --cookie "sessionid=your_session_id"