System Health#
The System Health section of the Admin Dashboard displays the current status of every server/collector (including ML Engines) in a Scrutinizer cluster. It allows administrators to check whether servers are running normally, spot resource problems, review the health of individual services, and dig into logs without needing to log in to each machine separately.
System health data is refreshed every 15 seconds, so the information displayed is always current within that window.
System Health gadget#
The Admin Dashboard’s System Health displays a summary of the following information for all nodes in the cluster:
Column |
Description |
|---|---|
Name |
The server’s hostname and its role icon |
Resource |
The overall health of CPU, memory, pressure, and load metrics |
Disk |
The overall health of disk usage across all monitored partitions |
Services |
Whether all monitored services are running as expected |
Health Checks |
The combined health of database connections, cluster communication, and automated checks |
CPU |
Current CPU usage percentage |
Memory |
Current memory usage percentage |
Disk % |
Current disk usage percentage (highest across monitored partitions) |
Last Seen |
When health data was last successfully collected from this server |
Clicking on a row in the table drills into the full server health view/page for the corresponding node.
System Health overview#
Clicking the stethoscope button or the Server Health virtual LED switches to a card-based overview.
In this view, each node has the following health details displayed in an expandable card:
The server hostname and its role name
The server’s current health status (Green / Yellow / Red) and status category badges (R, D, S, SYS)
Resource usage bars showing how much of each resource is currently in use:
CPU and Memory
CPU Pressure and IO Pressure
Swap
Load average shown as three numbers representing the last 1, 5, and 15 minutes, along with the total CPU core count
Disk usage for the root filesystem, database storage, and the flow spool directory
A Services counter (for example, 10 / 10) showing how many of the server’s monitored services are healthy
The time the data was last collected
Upload and Download buttons for downloading diagnostic data or sharing it with Plixer Support
Tip
The Grid View and List View toggles can be used to switch between the two System Health summary layouts.
Health Status#
Every server, and the cluster as a whole, is always in one of three states:
Status |
Color |
What it means |
|---|---|---|
Healthy |
Green |
Everything is operating within normal ranges |
Degraded |
Yellow |
Something is approaching a problem threshold and may need attention |
Unhealthy |
Red |
Something has exceeded a critical threshold and needs immediate attention |
Health status rolls up from the bottom. The most severe issue on any individual server determines that server’s status. The most severe status across all servers determines what the cluster-level indicator shows in the upper-right corner.
For example, if a single check on a secondary server goes critical, the top-level indicator turns red, even if all other servers are healthy.
Note
Individual health checks may also be shown in gray. This indicates that the check was skipped because it does not apply to the server’s platform, or that it is still collecting the data needed to report a result. Gray checks do not affect the server’s health status.
Status Category Badges#
Each server status card (grid view) displays up to four colored letter badges, one for each health category being monitored on that server. The badge color reflects the health status of that category.
Badge |
Category |
What it covers |
|---|---|---|
R |
Resource Status |
CPU, memory, pressure, swap, and load metrics |
D |
Disk Status |
Usage across all monitored disk partitions |
S |
Service Status |
The running state of all monitored Scrutinizer services |
SYS |
Systems Status |
Database connection health and cluster communication |
Server Actions#
Each server status card (grid view) has four action buttons in the header row:
Button |
What it does |
|---|---|
View Details (→) |
Opens the full detail page for that server |
Device Performance (ⓘ) |
Opens the System Performance page for that server, showing disk utilization, CPU and memory usage relative to exporters, and feature resource consumption |
Upload to Support (↑) |
Sends a diagnostic bundle for that server directly to Plixer Support |
Download Bundle (↓) |
Downloads a diagnostic bundle for that server to your computer |
Device Performance#
Clicking the Device Performance button on a server card opens the System Performance page for that server.
The System Performance page is split into the Disk Utilization and Feature Resources subviews. Additional reference tables for CPU and RAM recommendations based on exporter count and flow volume are also available via the dropdown menu.
Disk Utilization#
The Disk Utilization subview displays the follow information related to Scrutinizer’s disk usage:
Current total vs. predicted utilization based on current disk capacity
Current vs. predicted maximum disk utilization, based on current flow volume and data retention settings.
Current disk utilization per roll-up interval vs. predicted maximum, based on the number of days the data is configured to be stored
The default Data Retention graph shows the number of days of historical data currently saved compared against the total number of days that will be retained based on the current data history settings.
The Chart dropdown can also be used to access recommended resource allocation tables as well as the Feature Resources summary/management view (see below) for the collector.
Feature Resources#
The Feature Resources subview can be used to inspect and manage resource usage by feature set.
The page’s main view lists all available feature sets, alongside the following details:
Current state (green: active, grey: inactive)
Number of active alarms indicating resource issues for the feature set
Expected CPU core usage per collector
Expected RAM usage per collector
Number of FA algorithms associated with the feature set
Number of alarm policies associated with the feature set
Users are also able to toggle between graphs showing algorithms, policies, CPUs, or RAM per feature set.
Enabling/disabling feature sets
To allow teams to better adapt Scrutinizer’s functions to monitoring and resource requirements, related FA algorithms and their associated alarm policies can be disabled/deactivated by feature set instead of individually disabling them via the respective management pages.
Clicking on a feature set name opens the information tray, where it can be activated or deactivated via a toggle. All FA algorithms and alarm policies included in the feature set are also listed in this tray.
Important
Deactivating services may result in loss of functionality and/or other issues. Contact Plixer Technical Support for assistance.
Use the Back button in the upper-right corner to return to the previous page.
Server Health view#
The full server health page provides a complete picture of a server/node’s health. At the top of the page, the server’s operating system, agent version, IP address, and the last time health data was successfully collected are displayed.
The rest of the page is divided into the following sections:
Resource#
This section provides an overview of system resource utilization and hardware performance metrics.
Metric |
Description |
Warning / Critical |
|---|---|---|
CPU |
Percentage of CPU currently in use. High values can slow flow processing, queries, and the UI. |
70% / 90% |
Memory |
Percentage of physical memory currently in use. High values increase the risk of service instability or out-of-memory events. |
80% / 90% |
CPU Pressure (some) |
Percentage of time at least one process was waiting for CPU resources. Indicates CPU contention. |
25% / 50% |
CPU Pressure (full) |
Percentage of time all processes were waiting for CPU resources. Indicates severe CPU saturation. |
5% / 10% |
Mem Pressure (some) |
Percentage of time at least one process was waiting for memory resources. Can cause slower query and UI performance. |
25% / 50% |
Mem Pressure (full) |
Percentage of time all processes were waiting for memory resources. Indicates significant memory pressure. |
5% / 10% |
IO Pressure (some) |
Percentage of time at least one process was waiting for disk I/O operations. |
25% / 50% |
IO Pressure (full) |
Percentage of time all processes were waiting for disk I/O operations. Indicates disk I/O is a system bottleneck. |
5% / 10% |
Swap |
Percentage of configured swap space currently in use. Higher values can significantly reduce performance. |
50% / 90% |
Scrutinizer categorizes resource pressure into two distinct operational states:
Some: At least one program/process had to wait for the resource. This is an early warning sign that the server is starting to get busy and delays could happen soon.
Full: Every running program was completely stopped, waiting for the resource. This is a serious condition that usually causes noticeable slowdowns in system performance.
Tip
On virtual machines, the CPU Pressure metrics only reflect contention inside the server. If flow processing, rollups, or reports are falling behind while CPU usage and pressure appear normal, check the Host CPU Starvation health check to determine whether the hypervisor or cloud host is withholding CPU from the server.
Disk#
This section shows how full each of the server’s key storage locations is.
Location |
Data stored in the location |
|---|---|
root (/) |
Stores operating system files, logs, journals, and temporary data. |
spools |
Stores flow records temporarily before they are written to the database. |
database |
Stores PostgreSQL data used by Scrutinizer for flow and reporting data. |
Services#
This section lists every Scrutinizer-related service on the server, along with its current state and memory usage. The services counter on the server card (for example, 10 / 10) reflects the number of healthy services shown here.
Service |
Description |
|---|---|
plixer_api |
Backend API service used by Scrutinizer components and the web interface. |
plixer_autoreplicate |
Manages automated replication-related tasks between Scrutinizer systems. |
plixer_db |
Database management service used to monitor and maintain PostgreSQL operations. |
plixer_collector |
Receives, processes, and writes flow data to the database. |
plixer_fcgi |
FastCGI service that processes API and application requests. |
plixer_host_index |
Maintains host indexing information used for searching and reporting. |
plixer_mcp |
Internal management and control service used by Scrutinizer components. |
plixer_snmp |
Collects and processes SNMP data from monitored devices. |
plixer_webapp |
Scrutinizer web application service. |
memcached |
In-memory cache used to improve application performance by reducing database queries. |
nginx |
Web server and reverse proxy that handles incoming HTTP and HTTPS requests. |
pgbouncer |
PostgreSQL connection pooler that manages and reuses database connections. |
replicator |
Transfers data between Scrutinizer systems in a replication deployment. |
Note
Some services are designed to run a task and stop cleanly. A status of active with a state of exited is expected for these services and does not indicate a problem. The state to watch for is failed in either the Status or State column.
Health Checks#
This section shows the database connection health, cluster communication metrics, and automated system checks in one place. Each item displays a color indicator and, where applicable, a current value.
Clicking a specific health check opens a tray displaying the full details for that check:
Value: The current result of the check
Warning / Critical: The thresholds at which the check turns yellow or red (shown only for checks with thresholds)
Scope: Where the check is run (e.g., Agent for checks run locally on the server)
Detail: Additional information about the current result
What it’s telling you: A description of what the check monitors and, where applicable, the possible impact and troubleshooting steps
Metric |
Description |
|---|---|
SSL Certificate Expiry |
Number of days remaining before the server’s SSL certificate expires. |
Time Synchronization |
Amount of time (in seconds) the server’s clock has drifted from NTP. Clock drift can result in incorrect flow record timestamps, make replication and audit logs harder to correlate, and cause TLS, Kerberos, or TOTP authentication to fail. Warning and critical thresholds are 0.1 s and 0.5 s, respectively. |
DNS Resolution |
Verifies that hostnames can be resolved through the configured DNS server. |
Critical Service Ports |
Verifies that required service ports are listening and accepting connections. |
Spool Directory |
Verifies that the collector spool directory exists and is accessible. |
Meta Sync |
Verifies that metadata synchronization between cluster nodes is functioning correctly. |
Time Sync |
Clock offset (in milliseconds) from the configured time source, along with the time since the last successful sync and the address of the time source. This check is run at scheduled intervals (every 5 minutes by default). |
History Maintenance |
Number of failed history maintenance jobs responsible for rollups, retention, and partition management. |
Expired Tables |
Number of expired database partitions waiting to be removed. |
Storage Artifacts |
Number of orphaned or stray storage objects detected on disk. |
Host CPU Starvation |
Percentage of the server’s CPU demand that was not served by the hypervisor or cloud host. Applies to virtual machines only (see Host Starvation Checks). |
Host Memory Starvation |
Percentage of the server’s memory reclaimed by the VMware host through ballooning. Applies to VMware virtual machines only (see Host Starvation Checks). |
Rollup Lag |
Number of minutes the oldest closed interval has been waiting to be rolled up, and whether the lag is increasing or decreasing (see Rollup and Ingestion Lag Checks). |
Ingestion Lag |
Number of minutes the oldest flow data has been waiting to be inserted into the database, along with the number of payloads waiting (see Rollup and Ingestion Lag Checks). |
api |
Response time for health checks performed against the node’s API endpoint. |
db_direct |
Response time for direct database connectivity checks. |
db_pgbouncer |
Response time for database connectivity checks through PgBouncer. |
PG Connections |
Percentage of PostgreSQL connection slots currently in use. |
PG Frozen XID Age |
PostgreSQL transaction ID age used to monitor autovacuum health and prevent transaction wraparound. |
Host Starvation Checks#
When the hypervisor or cloud host running a Scrutinizer server is oversubscribed, the server may not receive the CPU or memory it needs. This is not visible from inside the server: CPU usage may appear mostly idle and CPU pressure may stay low, while rollups, inserters, and reports fall behind.
The Host CPU Starvation and Host Memory Starvation checks report how much CPU or memory the host withheld from the server. When a VMware CPU or memory limit is configured for the virtual machine, it is also included at the end of the check’s message (e.g., CPU limit 500 MHz).
Check |
Description |
Warning / Critical |
|---|---|---|
Host CPU Starvation |
Percentage of the server’s CPU demand that waited on the host, averaged over the last 15 to 20 minutes |
25% / 50% |
Host Memory Starvation |
Percentage of the server’s memory ballooned by the host. The amount of ballooned and host-swapped memory (in MB) is also shown. |
25% / 50% |
The thresholds are set high to avoid flagging hosts that are only slightly oversubscribed.
The data available to these checks depends on the server’s platform:
Platform |
Host CPU Starvation |
Host Memory Starvation |
|---|---|---|
VMware with VMware Tools |
Percentage ( |
Percentage ballooned ( |
KVM, OCI, GCP, AWS (Nitro and Xen) |
Percentage ( |
Gray ( |
Hyper-V, Azure |
Gray ( |
Gray ( |
VMware without VMware Tools |
Gray |
Gray |
Bare metal (hardware) |
Green, 0% ( |
Green, 0% ( |
Note
After the System Health service is restarted, Host CPU Starvation is shown in gray with the message collecting baseline, result available after the next run. A value is reported after the next collection cycle (up to 5 minutes).
Interpreting Host CPU Starvation
Host CPU Starvation should be reviewed together with the CPU Pressure metrics to determine where a CPU bottleneck is:
Host CPU Starvation |
CPU Pressure |
Meaning |
|---|---|---|
High |
Any |
The host is not providing the CPU the server needs. The virtualization host or cloud instance should be reviewed by the team that manages it. |
Low |
High |
The server’s own workload is exceeding its allocated CPU. Additional vCPUs or a reduction in workload may be required. |
Low |
Low |
CPU is not the bottleneck. |
Important
Adding vCPUs or increasing the number of processes on a starved server will not resolve the issue, since the additional CPU demand will also wait on the host.
Rollup and Ingestion Lag Checks#
The Rollup Lag and Ingestion Lag checks run on every collector and report how far behind the server is in processing flow data. Both checks report delays in minutes, allowing the same thresholds to apply to all collectors regardless of the number of exporters or templates.
Rollup Lag: Measures how long the oldest closed interval has been waiting to be rolled up. Only intervals that have already ended are considered. The check also indicates whether the lag is increasing or decreasing compared to 15 minutes earlier, to distinguish a server catching up after a restart from one that cannot keep up. Rollup rows that have been stuck in a
runningorunknownstate for more than 30 minutes are also reported.Ingestion Lag: Measures how long the oldest flow data has been waiting in the inserters’ queue, along with the number of payloads currently waiting.
Check |
Warning |
Critical |
|---|---|---|
Rollup lag |
10 min |
30 min |
Rollup rows stuck in |
Any |
— |
Ingestion lag |
5 min |
10 min |
Note
Sustained ingestion lag of 10 minutes or more may prevent the inserters from recovering on their own. If Ingestion Lag remains critical, contact Plixer Technical Support.
On virtual machines, sustained rollup or ingestion lag may be caused by the host withholding CPU from the server. Check Host CPU Starvation before adding vCPUs or rollers.
Log Files#
This section lists all the log files Scrutinizer monitors on a given server. This makes it easy to spot which component is generating errors without having to access the server directly.
Column |
What it shows |
|---|---|
Log File |
The name of the log and where it lives on the server |
Format |
The type of log file (affects how errors and warnings are counted) |
Size |
How large the file currently is |
Lines |
The total number of log entries |
Errors |
How many error-level entries were written during the selected time window |
Warnings |
How many warning-level entries were written during the selected time window |
Modified |
When the log was last written to |
The Errors and Warnings counts reflect activity within the currently selected time window, and not lifetime totals. As the window moves forward, counts from quieter periods will naturally drop. These numbers cannot be reset manually.
If a log file shows — for size and lines with a date of 0000-12-31, it means that feature is either not configured or has not yet produced any data on this server (for example, a cloud flow source that is not in use).
Clicking any log file name opens it in the log viewer. From there you can:
Filter by level: Show only errors, warnings, info, or debug entries, or view all levels together.
Search: Type to filter the loaded entries by keyword.
Tail: Click TAIL to watch the log update live. The viewer loads the most recent entries and streams new lines as they are written, scrolling automatically to keep up. Click Stop Tail to stop the live feed and return to normal browsing.
Health Events Log#
The Health Events log is a focused record of every time something changed in System Health (e.g. status transitions, automated responses, and skipped actions). This is the first place to check to understand what happened on a server and when.
There are three types of entries:
Entry |
Description |
|---|---|
Health state changed |
A metric or check crossed a threshold and the server’s status changed (for example, from Green to Yellow) |
Action executed |
Scrutinizer automatically took a corrective action, with a note on whether it succeeded |
Action skipped |
A corrective action was triggered but not taken, along with the reason (for example, the action is still in its cooldown period) |
Each entry is tagged with a severity that matches the resulting health state: Info for a return to green, Warning for a degraded state, and Error for an unhealthy state.
The Health Events log is separate from the general System Health log, which records the internal workings of the health monitoring service itself such as startup events, collection cycles, and internal errors. The Health Events log answers “what changed and when?”, while the System Health log is for deeper troubleshooting of the monitoring system itself.
Downloading and Uploading System Health Data#
If further assistance is needed, a support bundle can be sent directly from the System Health view. A support bundle is a compressed archive that contains the server’s current status, log files, and configuration.
To download a bundle to your computer: Click the Download icon on a server card. The bundle will be saved as a file you can attach to a support case.
To send a bundle directly to Plixer: Click the Upload icon on a server card. A loading spinner appears while the transfer is in progress, then changes to a checkmark when the upload is complete. No additional confirmation is required.
There are two available options for scope:
Single server: Use the Upload or Download icon on that server’s card to send data for that server only.
Entire cluster: Use the Upload or Download icon at the upper-right corner of the main System Health page to send data for all servers at once.