System Health#

The System Health section of the Admin Dashboard displays the current status of every server/collector (including ML Engines) in a Scrutinizer cluster. It allows administrators to check whether servers are running normally, spot resource problems, review the health of individual services, and dig into logs without needing to log in to each machine separately.

System health data is refreshed every 15 seconds, so the information displayed is always current within that window.

System Health gadget#

The Admin Dashboard’s System Health displays a summary of the following information for all nodes in the cluster:

Column

Description

Name

The server’s hostname and its role icon

Resource

The overall health of CPU, memory, pressure, and load metrics

Disk

The overall health of disk usage across all monitored partitions

Services

Whether all monitored services are running as expected

Health Checks

The combined health of database connections, cluster communication, and automated checks

CPU

Current CPU usage percentage

Memory

Current memory usage percentage

Disk %

Current disk usage percentage (highest across monitored partitions)

Last Seen

When health data was last successfully collected from this server

Clicking on a row in the table drills into the full server health view/page for the corresponding node.

System Health overview#

Clicking the stethoscope button or the Server Health virtual LED switches to a card-based overview.

In this view, each node has the following health details displayed in an expandable card:

  • The server hostname and its role name

  • The server’s current health status (Green / Yellow / Red) and status category badges (R, D, S, SYS)

  • Resource usage bars showing how much of each resource is currently in use:

    • CPU and Memory

    • CPU Pressure and IO Pressure

    • Swap

    • Load average shown as three numbers representing the last 1, 5, and 15 minutes, along with the total CPU core count

    • Disk usage for the root filesystem, database storage, and the flow spool directory

  • A Services counter (for example, 10 / 10) showing how many of the server’s monitored services are healthy

  • The time the data was last collected

  • Upload and Download buttons for downloading diagnostic data or sharing it with Plixer Support

Tip

The Grid View and List View toggles can be used to switch between the two System Health summary layouts.

Health Status#

Every server, and the cluster as a whole, is always in one of three states:

Status

Color

What it means

Healthy

Green

Everything is operating within normal ranges

Degraded

Yellow

Something is approaching a problem threshold and may need attention

Unhealthy

Red

Something has exceeded a critical threshold and needs immediate attention

Health status rolls up from the bottom. The most severe issue on any individual server determines that server’s status. The most severe status across all servers determines what the cluster-level indicator shows in the upper-right corner.

For example, if a single check on a secondary server goes critical, the top-level indicator turns red, even if all other servers are healthy.

Note

Individual health checks may also be shown in gray. This indicates that the check was skipped because it does not apply to the server’s platform, or that it is still collecting the data needed to report a result. Gray checks do not affect the server’s health status.

Status Category Badges#

Each server status card (grid view) displays up to four colored letter badges, one for each health category being monitored on that server. The badge color reflects the health status of that category.

Badge

Category

What it covers

R

Resource Status

CPU, memory, pressure, swap, and load metrics

D

Disk Status

Usage across all monitored disk partitions

S

Service Status

The running state of all monitored Scrutinizer services

SYS

Systems Status

Database connection health and cluster communication

Server Actions#

Each server status card (grid view) has four action buttons in the header row:

Button

What it does

View Details (→)

Opens the full detail page for that server

Device Performance (ⓘ)

Opens the System Performance page for that server, showing disk utilization, CPU and memory usage relative to exporters, and feature resource consumption

Upload to Support (↑)

Sends a diagnostic bundle for that server directly to Plixer Support

Download Bundle (↓)

Downloads a diagnostic bundle for that server to your computer

Device Performance#

Clicking the Device Performance button on a server card opens the System Performance page for that server.

The System Performance page is split into the Disk Utilization and Feature Resources subviews. Additional reference tables for CPU and RAM recommendations based on exporter count and flow volume are also available via the dropdown menu.

Disk Utilization#

The Disk Utilization subview displays the follow information related to Scrutinizer’s disk usage:

  • Current total vs. predicted utilization based on current disk capacity

  • Current vs. predicted maximum disk utilization, based on current flow volume and data retention settings.

  • Current disk utilization per roll-up interval vs. predicted maximum, based on the number of days the data is configured to be stored

The default Data Retention graph shows the number of days of historical data currently saved compared against the total number of days that will be retained based on the current data history settings.

The Chart dropdown can also be used to access recommended resource allocation tables as well as the Feature Resources summary/management view (see below) for the collector.

Feature Resources#

The Feature Resources subview can be used to inspect and manage resource usage by feature set.

The page’s main view lists all available feature sets, alongside the following details:

  • Current state (green: active, grey: inactive)

  • Importance

  • Number of active alarms indicating resource issues for the feature set

  • Expected CPU core usage per collector

  • Expected RAM usage per collector

  • Number of FA algorithms associated with the feature set

  • Number of alarm policies associated with the feature set

Users are also able to toggle between graphs showing algorithms, policies, CPUs, or RAM per feature set.

Enabling/disabling feature sets

To allow teams to better adapt Scrutinizer’s functions to monitoring and resource requirements, related FA algorithms and their associated alarm policies can be disabled/deactivated by feature set instead of individually disabling them via the respective management pages.

Clicking on a feature set name opens the information tray, where it can be activated or deactivated via a toggle. All FA algorithms and alarm policies included in the feature set are also listed in this tray.

Important

Deactivating services may result in loss of functionality and/or other issues. Contact Plixer Technical Support for assistance.

Use the Back button in the upper-right corner to return to the previous page.

Server Health view#

The full server health page provides a complete picture of a server/node’s health. At the top of the page, the server’s operating system, agent version, IP address, and the last time health data was successfully collected are displayed.

The rest of the page is divided into the following sections:

Resource#

This section provides an overview of system resource utilization and hardware performance metrics.

Metric

Description

Warning / Critical

CPU

Percentage of CPU currently in use. High values can slow flow processing, queries, and the UI.

70% / 90%

Memory

Percentage of physical memory currently in use. High values increase the risk of service instability or out-of-memory events.

80% / 90%

CPU Pressure (some)

Percentage of time at least one process was waiting for CPU resources. Indicates CPU contention.

25% / 50%

CPU Pressure (full)

Percentage of time all processes were waiting for CPU resources. Indicates severe CPU saturation.

5% / 10%

Mem Pressure (some)

Percentage of time at least one process was waiting for memory resources. Can cause slower query and UI performance.

25% / 50%

Mem Pressure (full)

Percentage of time all processes were waiting for memory resources. Indicates significant memory pressure.

5% / 10%

IO Pressure (some)

Percentage of time at least one process was waiting for disk I/O operations.

25% / 50%

IO Pressure (full)

Percentage of time all processes were waiting for disk I/O operations. Indicates disk I/O is a system bottleneck.

5% / 10%

Swap

Percentage of configured swap space currently in use. Higher values can significantly reduce performance.

50% / 90%

Scrutinizer categorizes resource pressure into two distinct operational states:

  • Some: At least one program/process had to wait for the resource. This is an early warning sign that the server is starting to get busy and delays could happen soon.

  • Full: Every running program was completely stopped, waiting for the resource. This is a serious condition that usually causes noticeable slowdowns in system performance.

Tip

On virtual machines, the CPU Pressure metrics only reflect contention inside the server. If flow processing, rollups, or reports are falling behind while CPU usage and pressure appear normal, check the Host CPU Starvation health check to determine whether the hypervisor or cloud host is withholding CPU from the server.

Disk#

This section shows how full each of the server’s key storage locations is.

Location

Data stored in the location

root (/)

Stores operating system files, logs, journals, and temporary data.

spools

Stores flow records temporarily before they are written to the database.

database

Stores PostgreSQL data used by Scrutinizer for flow and reporting data.

Services#

This section lists every Scrutinizer-related service on the server, along with its current state and memory usage. The services counter on the server card (for example, 10 / 10) reflects the number of healthy services shown here.

Service

Description

plixer_api

Backend API service used by Scrutinizer components and the web interface.

plixer_autoreplicate

Manages automated replication-related tasks between Scrutinizer systems.

plixer_db

Database management service used to monitor and maintain PostgreSQL operations.

plixer_collector

Receives, processes, and writes flow data to the database.

plixer_fcgi

FastCGI service that processes API and application requests.

plixer_host_index

Maintains host indexing information used for searching and reporting.

plixer_mcp

Internal management and control service used by Scrutinizer components.

plixer_snmp

Collects and processes SNMP data from monitored devices.

plixer_webapp

Scrutinizer web application service.

memcached

In-memory cache used to improve application performance by reducing database queries.

nginx

Web server and reverse proxy that handles incoming HTTP and HTTPS requests.

pgbouncer

PostgreSQL connection pooler that manages and reuses database connections.

replicator

Transfers data between Scrutinizer systems in a replication deployment.

Note

Some services are designed to run a task and stop cleanly. A status of active with a state of exited is expected for these services and does not indicate a problem. The state to watch for is failed in either the Status or State column.

Health Checks#

This section shows the database connection health, cluster communication metrics, and automated system checks in one place. Each item displays a color indicator and, where applicable, a current value.

Clicking a specific health check opens a tray displaying the full details for that check:

  • Value: The current result of the check

  • Warning / Critical: The thresholds at which the check turns yellow or red (shown only for checks with thresholds)

  • Scope: Where the check is run (e.g., Agent for checks run locally on the server)

  • Detail: Additional information about the current result

  • What it’s telling you: A description of what the check monitors and, where applicable, the possible impact and troubleshooting steps

Metric

Description

SSL Certificate Expiry

Number of days remaining before the server’s SSL certificate expires.

Time Synchronization

Amount of time (in seconds) the server’s clock has drifted from NTP. Clock drift can result in incorrect flow record timestamps, make replication and audit logs harder to correlate, and cause TLS, Kerberos, or TOTP authentication to fail. Warning and critical thresholds are 0.1 s and 0.5 s, respectively.

DNS Resolution

Verifies that hostnames can be resolved through the configured DNS server.

Critical Service Ports

Verifies that required service ports are listening and accepting connections.

Spool Directory

Verifies that the collector spool directory exists and is accessible.

Meta Sync

Verifies that metadata synchronization between cluster nodes is functioning correctly.

Time Sync

Clock offset (in milliseconds) from the configured time source, along with the time since the last successful sync and the address of the time source. This check is run at scheduled intervals (every 5 minutes by default).

History Maintenance

Number of failed history maintenance jobs responsible for rollups, retention, and partition management.

Expired Tables

Number of expired database partitions waiting to be removed.

Storage Artifacts

Number of orphaned or stray storage objects detected on disk.

Host CPU Starvation

Percentage of the server’s CPU demand that was not served by the hypervisor or cloud host. Applies to virtual machines only (see Host Starvation Checks).

Host Memory Starvation

Percentage of the server’s memory reclaimed by the VMware host through ballooning. Applies to VMware virtual machines only (see Host Starvation Checks).

Rollup Lag

Number of minutes the oldest closed interval has been waiting to be rolled up, and whether the lag is increasing or decreasing (see Rollup and Ingestion Lag Checks).

Ingestion Lag

Number of minutes the oldest flow data has been waiting to be inserted into the database, along with the number of payloads waiting (see Rollup and Ingestion Lag Checks).

api

Response time for health checks performed against the node’s API endpoint.

db_direct

Response time for direct database connectivity checks.

db_pgbouncer

Response time for database connectivity checks through PgBouncer.

PG Connections

Percentage of PostgreSQL connection slots currently in use.

PG Frozen XID Age

PostgreSQL transaction ID age used to monitor autovacuum health and prevent transaction wraparound.

Host Starvation Checks#

When the hypervisor or cloud host running a Scrutinizer server is oversubscribed, the server may not receive the CPU or memory it needs. This is not visible from inside the server: CPU usage may appear mostly idle and CPU pressure may stay low, while rollups, inserters, and reports fall behind.

The Host CPU Starvation and Host Memory Starvation checks report how much CPU or memory the host withheld from the server. When a VMware CPU or memory limit is configured for the virtual machine, it is also included at the end of the check’s message (e.g., CPU limit 500 MHz).

Check

Description

Warning / Critical

Host CPU Starvation

Percentage of the server’s CPU demand that waited on the host, averaged over the last 15 to 20 minutes

25% / 50%

Host Memory Starvation

Percentage of the server’s memory ballooned by the host. The amount of ballooned and host-swapped memory (in MB) is also shown.

25% / 50%

The thresholds are set high to avoid flagging hosts that are only slightly oversubscribed.

The data available to these checks depends on the server’s platform:

Platform

Host CPU Starvation

Host Memory Starvation

VMware with VMware Tools

Percentage (source: VMware Tools)

Percentage ballooned (source: VMware Tools)

KVM, OCI, GCP, AWS (Nitro and Xen)

Percentage (source: steal time)

Gray (no host memory counter on this platform)

Hyper-V, Azure

Gray (no host CPU counter on this platform (microsoft))

Gray (no host memory counter on this platform (microsoft))

VMware without VMware Tools

Gray

Gray

Bare metal (hardware)

Green, 0% (not virtualized)

Green, 0% (not virtualized)

Note

After the System Health service is restarted, Host CPU Starvation is shown in gray with the message collecting baseline, result available after the next run. A value is reported after the next collection cycle (up to 5 minutes).

Interpreting Host CPU Starvation

Host CPU Starvation should be reviewed together with the CPU Pressure metrics to determine where a CPU bottleneck is:

Host CPU Starvation

CPU Pressure

Meaning

High

Any

The host is not providing the CPU the server needs. The virtualization host or cloud instance should be reviewed by the team that manages it.

Low

High

The server’s own workload is exceeding its allocated CPU. Additional vCPUs or a reduction in workload may be required.

Low

Low

CPU is not the bottleneck.

Important

Adding vCPUs or increasing the number of processes on a starved server will not resolve the issue, since the additional CPU demand will also wait on the host.

Rollup and Ingestion Lag Checks#

The Rollup Lag and Ingestion Lag checks run on every collector and report how far behind the server is in processing flow data. Both checks report delays in minutes, allowing the same thresholds to apply to all collectors regardless of the number of exporters or templates.

  • Rollup Lag: Measures how long the oldest closed interval has been waiting to be rolled up. Only intervals that have already ended are considered. The check also indicates whether the lag is increasing or decreasing compared to 15 minutes earlier, to distinguish a server catching up after a restart from one that cannot keep up. Rollup rows that have been stuck in a running or unknown state for more than 30 minutes are also reported.

  • Ingestion Lag: Measures how long the oldest flow data has been waiting in the inserters’ queue, along with the number of payloads currently waiting.

Check

Warning

Critical

Rollup lag

10 min

30 min

Rollup rows stuck in running/unknown for more than 30 min

Any

—

Ingestion lag

5 min

10 min

Note

  • Sustained ingestion lag of 10 minutes or more may prevent the inserters from recovering on their own. If Ingestion Lag remains critical, contact Plixer Technical Support.

  • On virtual machines, sustained rollup or ingestion lag may be caused by the host withholding CPU from the server. Check Host CPU Starvation before adding vCPUs or rollers.

Log Files#

This section lists all the log files Scrutinizer monitors on a given server. This makes it easy to spot which component is generating errors without having to access the server directly.

Column

What it shows

Log File

The name of the log and where it lives on the server

Format

The type of log file (affects how errors and warnings are counted)

Size

How large the file currently is

Lines

The total number of log entries

Errors

How many error-level entries were written during the selected time window

Warnings

How many warning-level entries were written during the selected time window

Modified

When the log was last written to

The Errors and Warnings counts reflect activity within the currently selected time window, and not lifetime totals. As the window moves forward, counts from quieter periods will naturally drop. These numbers cannot be reset manually.

If a log file shows — for size and lines with a date of 0000-12-31, it means that feature is either not configured or has not yet produced any data on this server (for example, a cloud flow source that is not in use).

Clicking any log file name opens it in the log viewer. From there you can:

  • Filter by level: Show only errors, warnings, info, or debug entries, or view all levels together.

  • Search: Type to filter the loaded entries by keyword.

  • Tail: Click TAIL to watch the log update live. The viewer loads the most recent entries and streams new lines as they are written, scrolling automatically to keep up. Click Stop Tail to stop the live feed and return to normal browsing.

Health Events Log#

The Health Events log is a focused record of every time something changed in System Health (e.g. status transitions, automated responses, and skipped actions). This is the first place to check to understand what happened on a server and when.

There are three types of entries:

Entry

Description

Health state changed

A metric or check crossed a threshold and the server’s status changed (for example, from Green to Yellow)

Action executed

Scrutinizer automatically took a corrective action, with a note on whether it succeeded

Action skipped

A corrective action was triggered but not taken, along with the reason (for example, the action is still in its cooldown period)

Each entry is tagged with a severity that matches the resulting health state: Info for a return to green, Warning for a degraded state, and Error for an unhealthy state.

The Health Events log is separate from the general System Health log, which records the internal workings of the health monitoring service itself such as startup events, collection cycles, and internal errors. The Health Events log answers “what changed and when?”, while the System Health log is for deeper troubleshooting of the monitoring system itself.

Downloading and Uploading System Health Data#

If further assistance is needed, a support bundle can be sent directly from the System Health view. A support bundle is a compressed archive that contains the server’s current status, log files, and configuration.

To download a bundle to your computer: Click the Download icon on a server card. The bundle will be saved as a file you can attach to a support case.

To send a bundle directly to Plixer: Click the Upload icon on a server card. A loading spinner appears while the transfer is in progress, then changes to a checkmark when the upload is complete. No additional confirmation is required.

There are two available options for scope:

  • Single server: Use the Upload or Download icon on that server’s card to send data for that server only.

  • Entire cluster: Use the Upload or Download icon at the upper-right corner of the main System Health page to send data for all servers at once.