System Health

System Health#

The System Health section of the Admin Dashboard displays the current status of every server/collector (including ML Engines) in a Scrutinizer cluster. It allows administrators to check whether servers are running normally, spot resource problems, review the health of individual services, and dig into logs without needing to log in to each machine separately.

System health data is refreshed every 15 seconds, so the information displayed is always current within that window.

System Health gadget#

The Admin Dashboard’s System Health displays a summary of the following information for all nodes in the cluster:

Column	Description
Name	The server’s hostname and its role icon
Resource	The overall health of CPU, memory, pressure, and load metrics
Disk	The overall health of disk usage across all monitored partitions
Services	Whether all monitored services are running as expected
Health Checks	The combined health of database connections, cluster communication, and automated checks
CPU	Current CPU usage percentage
Memory	Current memory usage percentage
Disk %	Current disk usage percentage (highest across monitored partitions)
Last Seen	When health data was last successfully collected from this server

Clicking on a row in the table drills into the full server health view/page for the corresponding node.

System Health overview#

Clicking the stethoscope button or the Server Health virtual LED switches to a card-based overview.

In this view, each node has the following health details displayed in an expandable card:

The server hostname and its role name
The server’s current health status (Green / Yellow / Red) and status category badges (R, D, S, SYS)
Resource usage bars showing how much of each resource is currently in use:
- CPU and Memory
- CPU Pressure and IO Pressure
- Swap
- Load average shown as three numbers representing the last 1, 5, and 15 minutes, along with the total CPU core count
- Disk usage for the root filesystem, database storage, and the flow spool directory
A Services counter (for example, 10 / 10) showing how many of the server’s monitored services are healthy
The time the data was last collected
Upload and Download buttons for downloading diagnostic data or sharing it with Plixer Support

Tip

The Grid View and List View toggles can be used to switch between the two System Health summary layouts.

Health Status#

Every server, and the cluster as a whole, is always in one of three states:

Status	Color	What it means
Healthy	Green	Everything is operating within normal ranges
Degraded	Yellow	Something is approaching a problem threshold and may need attention
Unhealthy	Red	Something has exceeded a critical threshold and needs immediate attention

Health status rolls up from the bottom. The most severe issue on any individual server determines that server’s status. The most severe status across all servers determines what the cluster-level indicator shows in the upper-right corner.

For example, if a single check on a secondary server goes critical, the top-level indicator turns red, even if all other servers are healthy.

Status Category Badges#

Each server status card (grid view) displays up to four colored letter badges, one for each health category being monitored on that server. The badge color reflects the health status of that category.

Badge	Category	What it covers
R	Resource Status	CPU, memory, pressure, swap, and load metrics
D	Disk Status	Usage across all monitored disk partitions
S	Service Status	The running state of all monitored Scrutinizer services
SYS	Systems Status	Database connection health and cluster communication

Server Actions#

Each server status card (grid view) has four action buttons in the header row:

Button	What it does
View Details (→)	Opens the full detail page for that server
Device Performance (ⓘ)	Opens the System Performance page for that server, showing disk utilization, CPU and memory usage relative to exporters, and feature resource consumption
Upload to Support (↑)	Sends a diagnostic bundle for that server directly to Plixer Support
Download Bundle (↓)	Downloads a diagnostic bundle for that server to your computer

Device Performance#

Clicking the Device Performance button on a server card opens the System Performance page for that server.

The System Performance page is split into the Disk Utilization and Feature Resources subviews. Additional reference tables for CPU and RAM recommendations based on exporter count and flow volume are also available via the dropdown menu.

Disk Utilization#

The Disk Utilization subview displays the follow information related to Scrutinizer’s disk usage:

Current total vs. predicted utilization based on current disk capacity
Current vs. predicted maximum disk utilization, based on current flow volume and data retention settings.
Current disk utilization per roll-up interval vs. predicted maximum, based on the number of days the data is configured to be stored

The default Data Retention graph shows the number of days of historical data currently saved compared against the total number of days that will be retained based on the current data history settings.

The Chart dropdown can also be used to access recommended resource allocation tables as well as the Feature Resources summary/management view (see below) for the collector.

Feature Resources#

The Feature Resources subview can be used to inspect and manage resource usage by feature set.

The page’s main view lists all available feature sets, alongside the following details:

Current state (green: active, grey: inactive)
Importance
Number of active alarms indicating resource issues for the feature set
Expected CPU core usage per collector
Expected RAM usage per collector
Number of FA algorithms associated with the feature set
Number of alarm policies associated with the feature set

Users are also able to toggle between graphs showing algorithms, policies, CPUs, or RAM per feature set.

Use the Back button in the upper-right corner to return to the previous page.

Server Health view#

The full server health page provides a complete picture of a server/node’s health. At the top of the page, the server’s operating system, agent version, IP address, and the last time health data was successfully collected are displayed.

The rest of the page is divided into the following sections:

Resource#

This section provides an overview of system resource utilization and hardware performance metrics.

Metric	Description	Warning / Critical
CPU	Percentage of CPU currently in use. High values can slow flow processing, queries, and the UI.	70% / 90%
Memory	Percentage of physical memory currently in use. High values increase the risk of service instability or out-of-memory events.	80% / 90%
CPU Pressure (some)	Percentage of time at least one process was waiting for CPU resources. Indicates CPU contention.	25% / 50%
CPU Pressure (full)	Percentage of time all processes were waiting for CPU resources. Indicates severe CPU saturation.	5% / 10%
Mem Pressure (some)	Percentage of time at least one process was waiting for memory resources. Can cause slower query and UI performance.	25% / 50%
Mem Pressure (full)	Percentage of time all processes were waiting for memory resources. Indicates significant memory pressure.	5% / 10%
IO Pressure (some)	Percentage of time at least one process was waiting for disk I/O operations.	25% / 50%
IO Pressure (full)	Percentage of time all processes were waiting for disk I/O operations. Indicates disk I/O is a system bottleneck.	5% / 10%
Swap	Percentage of configured swap space currently in use. Higher values can significantly reduce performance.	50% / 90%

Scrutinizer categorizes resource pressure into two distinct operational states:

Some: At least one program/process had to wait for the resource. This is an early warning sign that the server is starting to get busy and delays could happen soon.
Full: Every running program was completely stopped, waiting for the resource. This is a serious condition that usually causes noticeable slowdowns in system performance.

Disk#

This section shows how full each of the server’s key storage locations is.

Location	Data stored in the location
root (/)	Stores operating system files, logs, journals, and temporary data.
spools	Stores flow records temporarily before they are written to the database.
database	Stores PostgreSQL data used by Scrutinizer for flow and reporting data.

Services#

This section lists every Scrutinizer-related service on the server, along with its current state and memory usage. The services counter on the server card (for example, 10 / 10) reflects the number of healthy services shown here.

Service	Description
plixer_api	Backend API service used by Scrutinizer components and the web interface.
plixer_autoreplicate	Manages automated replication-related tasks between Scrutinizer systems.
plixer_db	Database management service used to monitor and maintain PostgreSQL operations.
plixer_collector	Receives, processes, and writes flow data to the database.
plixer_fcgi	FastCGI service that processes API and application requests.
plixer_host_index	Maintains host indexing information used for searching and reporting.
plixer_mcp	Internal management and control service used by Scrutinizer components.
plixer_snmp	Collects and processes SNMP data from monitored devices.
plixer_webapp	Scrutinizer web application service.
memcached	In-memory cache used to improve application performance by reducing database queries.
nginx	Web server and reverse proxy that handles incoming HTTP and HTTPS requests.
pgbouncer	PostgreSQL connection pooler that manages and reuses database connections.
replicator	Transfers data between Scrutinizer systems in a replication deployment.

Note

Some services are designed to run a task and stop cleanly. A status of active with a state of exited is expected for these services and does not indicate a problem. The state to watch for is failed in either the Status or State column.

Health Checks#

This section shows the database connection health, cluster communication metrics, and automated system checks in one place. Each item displays a color indicator and, where applicable, a current value.

Metric	Description
SSL Certificate Expiry	Number of days remaining before the server’s SSL certificate expires.
Time Synchronization	Time difference between the server clock and its configured NTP source.
DNS Resolution	Verifies that hostnames can be resolved through the configured DNS server.
Critical Service Ports	Verifies that required service ports are listening and accepting connections.
Spool Directory	Verifies that the collector spool directory exists and is accessible.
Meta Sync	Verifies that metadata synchronization between cluster nodes is functioning correctly.
Time Sync	Clock offset from the configured time source, measured in milliseconds.
History Maintenance	Number of failed history maintenance jobs responsible for rollups, retention, and partition management.
Expired Tables	Number of expired database partitions waiting to be removed.
Storage Artifacts	Number of orphaned or stray storage objects detected on disk.
api	Response time for health checks performed against the node’s API endpoint.
db_direct	Response time for direct database connectivity checks.
db_pgbouncer	Response time for database connectivity checks through PgBouncer.
PG Connections	Percentage of PostgreSQL connection slots currently in use.
PG Frozen XID Age	PostgreSQL transaction ID age used to monitor autovacuum health and prevent transaction wraparound.

Log Files#

This section lists all the log files Scrutinizer monitors on a given server. This makes it easy to spot which component is generating errors without having to access the server directly.

Column	What it shows
Log File	The name of the log and where it lives on the server
Format	The type of log file (affects how errors and warnings are counted)
Size	How large the file currently is
Lines	The total number of log entries
Errors	How many error-level entries were written during the selected time window
Warnings	How many warning-level entries were written during the selected time window
Modified	When the log was last written to

The Errors and Warnings counts reflect activity within the currently selected time window, and not lifetime totals. As the window moves forward, counts from quieter periods will naturally drop. These numbers cannot be reset manually.

If a log file shows — for size and lines with a date of 0000-12-31, it means that feature is either not configured or has not yet produced any data on this server (for example, a cloud flow source that is not in use).

Clicking any log file name opens it in the log viewer. From there you can:

Filter by level: Show only errors, warnings, info, or debug entries, or view all levels together.
Search: Type to filter the loaded entries by keyword.
Tail: Click TAIL to watch the log update live. The viewer loads the most recent entries and streams new lines as they are written, scrolling automatically to keep up. Click Stop Tail to stop the live feed and return to normal browsing.

Health Events Log#

The Health Events log is a focused record of every time something changed in System Health (e.g. status transitions, automated responses, and skipped actions). This is the first place to check to understand what happened on a server and when.

There are three types of entries:

Entry	Description
Health state changed	A metric or check crossed a threshold and the server’s status changed (for example, from Green to Yellow)
Action executed	Scrutinizer automatically took a corrective action, with a note on whether it succeeded
Action skipped	A corrective action was triggered but not taken, along with the reason (for example, the action is still in its cooldown period)

Each entry is tagged with a severity that matches the resulting health state: Info for a return to green, Warning for a degraded state, and Error for an unhealthy state.

The Health Events log is separate from the general System Health log, which records the internal workings of the health monitoring service itself such as startup events, collection cycles, and internal errors. The Health Events log answers “what changed and when?”, while the System Health log is for deeper troubleshooting of the monitoring system itself.

Downloading and Uploading System Health Data#

If further assistance is needed, a support bundle can be sent directly from the System Health view. A support bundle is a compressed archive that contains the server’s current status, log files, and configuration.

To download a bundle to your computer: Click the Download icon on a server card. The bundle will be saved as a file you can attach to a support case.

To send a bundle directly to Plixer: Click the Upload icon on a server card. A loading spinner appears while the transfer is in progress, then changes to a checkmark when the upload is complete. No additional confirmation is required.

There are two available options for scope:

Single server: Use the Upload or Download icon on that server’s card to send data for that server only.
Entire cluster: Use the Upload or Download icon at the upper-right corner of the main System Health page to send data for all servers at once.