Backups

Backups#

The Scrutinizer filesystem includes utilities that automate the process of creating or restoring system backups.

Note

  • These utilities are recommended for most long-term backup scenarios, because they include all database configuration and historical data for a Scrutinizer instance. Native snapshots may still be used as a short-term recovery option when there is no need to store the data, e.g., when upgrading the instance.

  • For Scrutinizer instances deployed on AWS, backups should be created and/or restored using native AWS functionality.

These utilities allow several types of backup and restore operations to be performed by the user.

Full backups#

Full or comprehensive backups are disaster-recovery-grade images of a Scrutinizer instance and include the following elements of the filesystem:

  • Application data and collected NetFlow in the PostgreSQL database

  • Host index data in BadgerDB databases

  • Scrutinizer’s third-party encryption key - /etc/plixer.key

  • Web Server TLS certificate and key

Important

  • The license key (if the instance is a primary reporter) and the TLS certificates and keys generated by Scrutinizer are not backed up and cannot be restored.

  • Any files not included in full backups must be manually backed up and restored, including:

    • Custom threat lists created under /home/plixer/scrutinizer/files/threats

    • Custom notifications created under /home/plixer/scrutinizer/files

    • LDAP authentication certificates

Creating a backup#

The Scrutinizer filesystem includes the scrut_base_backup.sh utility, which automates the creation of backups.

Important

  • The initial run of scrut_base_backup.sh requires a brief service pause. Subsequent runs do not.

  • WARNING: skipping special file "./cstore_fdw" is expected during this process.

The following instructions cover the process of creating and saving Scrutinizer backups:

View instructions

  1. SSH to the Scrutinizer server to be backed up and create a base backup.

    To run interactively:

    ~plixer/scrutinizer/database/utils/scrut_base_backup.sh

    To run the backup automatically (skipping confirmation):

    ~plixer/scrutinizer/database/utils/scrut_base_backup.sh -y

  2. Sync the backup. Use scrut_base_sync.sh to sync the backup taken with scrut_base_backup.sh to a new location.

    The following variables can be set:

    Variable

    Description

    BACKUP

    The database backup to sync

    REMOTE

    The remote host to sync to (example: user@host.example)

    TARGET

    The target directory

    KEY

    The SSH key to use

    Note

    The directory specified by the TARGET environment variable must already exist on the remote system, and the user specified by REMOTE must have read/write permissions to it.

    To sync the backup and data to a remote host:

    REMOTE=user@host.example TARGET=/home/plixer/backup \
~plixer/scrutinizer/database/utils/scrut_base_sync.sh

    To sync the backup and data to an NFS mount:

    TARGET=/mnt/nfs ~plixer/scrutinizer/database/utils/scrut_base_sync.sh

    To sync a specific backup locally (unattended):

    BACKUP=/var/db/backup/20260722_1159 \
~plixer/scrutinizer/database/utils/scrut_base_sync.sh -y

  3. Once the backup has been synced, you may remove it from your local machine.

    Example:

    sudo rm -Rf /var/db/backup/20260721_1150

Note

The directory name after backup/ (e.g., 20260722_1159, 20260721_1150) is automatically generated each time scrut_base_backup.sh is run. Replace it with the actual directory name created during your backup.

For further details or assistance with issues, contact Plixer Technical Support.

Backing up additional files#

When creating a full backup of a Scrutinizer server, any files not covered by the script must be manually backed up and should be stored on an external host/system.

These files should also be manually restored, after running the restore script.

Restoring the backup#

To restore a Scrutinizer instance from a full backup file, use the scrut_base_restore.sh utility located under home/plixer/scrutinizer/files. Make sure the backup is accessible locally on your machine.

The script will fully restore all backed up elements of a Scrutinizer instance, provided the following conditions are met:

  • A valid full backup file is accessible by the plixer user at the specified ($BACKUPDIR) remote location.

  • The Scrutinizer instance to be used for the restore has been freshly deployed.

  • The version of the backup matches the version of the fresh Scrutinizer instance to restore to (e.g. a 19.3.0 backup can only be restored to a new 19.3.0 instance).

Important

  • A restore completely overwrites the state of the target instance and deletes the source backup file. It is highly recommended to always restore from a copy of a backup file.

  • If the restore target is the primary reporter in a distributed cluster, contact Plixer Technical Support for assistance.

The following instructions cover the process of restoring from a backup file on a remote host to a fresh Scrutinizer deployment:

View instructions

To restore using the interactive menu:

~plixer/scrutinizer/database/utils/scrut_base_restore.sh

To restore from a specific backup directory:

BACKUP=/var/db/backup/20231027_120000 \
~plixer/scrutinizer/database/utils/scrut_base_restore.sh

Note

The directory name after backup/ (e.g., 20231027_120000) is automatically generated each time scrut_base_restore.sh is run. Replace it with the actual directory name created during your backup.

Use -y to run the script unattended (skipping confirmation). The BACKUP environment variable is required in this mode.

Based on the role of the Scrutinizer instance, proceed to finalize setup of the restored server:

  • If the restored instance is a standalone server, run the following to restart all services and register it:

    scrut_util --services --name all --switch restart
scrut_util --set selfregister --reset

    These commands may take several minutes to complete.

  • If the restored instance is a remote collector in a distributed cluster, run the following on the primary reporter to register it:

    scrut_util --set registercollector --ip RESTORED_INSTANCE_IP

  • If the restored instance is a primary reporter in a distributed cluster or a standalone server, and its Machine ID is different from that of the backup file, contact Plixer Technical Support to obtain a new license key.

Configuration backups#

For more “lightweight” backup and restore operations, the scrut_conf_dump.sh and scrut_conf_restore.sh scripts (both located in /home/plixer/scrutinizer/database/utils) can be used to target only the application/configuration data of a Scrutinizer instance, including:

  • User-added maps

  • Dashboards

  • IP groups

  • Saved reports

  • 3rd-party integration settings

Configuration backups do not include any collected flow data.

Note

In distributed clusters, the primary reporter regularly syncs application/configuration data to remote collectors. Only the configuration backup of the primary reporter is needed to perform a restore for the cluster.

scrut_conf_dump.sh and scrut_conf_restore.sh use Postgres’s pg_dump and pg_restore utils and respect the same set of environment variables:

Variable

Description

Default

DUMP

Location of the backup file

./conf.dump

PGHOST

IP address or hostname of the PostgreSQL database

localhost

PGUSER

Role/user used to connect to PGHOST

plixer

PGDATABASE

The database to access at PGHOST

plixer

Backing up configuration data#

To create a backup of a Scrutinizer server’s current configuration data, follow these steps:

View instructions

  1. Run the backup script.

    To save the backup file to the default location:

    ~/scrutinizer/database/utils/scrut_conf_dump.sh

    To use a custom location/filename:

    mkdir /tmp/CONF_BACKUP_DIR
touch /tmp/CONF_BACKUP_DIR/CONF_BACKUP.dump
DUMP=/tmp/CONF_BACKUP_DIR/CONF_BACKUP.dump ~/scrutinizer/database/utils/scrut_conf_dump.sh

  2. Restart the stopped services:

    sudo systemctl restart scrutinizer

Restoring configuration data#

To restore configuration data to a Scrutinizer server from a backup file, follow these steps:

View instructions

  1. Stop the plixer_webapp and plixer_collector services:

    sudo systemctl stop plixer_webapp
sudo systemctl stop plixer_collector

  2. Run the restore script.

    To restore from the default backup location/file:

    ~/scrutinizer/database/utils/scrut_conf_dump.sh

    To restore from a specified location/file:

    PGHOST=SCRUTINIZER_IP
DUMP=/tmp/CONF_BACKUP_DIR/CONF_BACKUP.dump ~/scrutinizer/database/utils/scrut_conf_restore.sh

  3. Restart the stopped services:

    sudo /bin/systemctl start plixer_webapp
sudo /bin/systemctl start plixer_collector

  4. Resync the access table:

    psql -c "SELECT setval(pg_get_serial_sequence('plixer.access', 'access_id'), COALESCE(max(access_id) + 1, 1), false) FROM plixer.access;"

Note

scrut_conf_restore.sh should only be used for restoring configuration data for the same Scrutinizer server/appliance. Follow this guide to migrate configuration data from one Scrutinizer server to another.

Additional notes#

  • pg_restore errors typically only cause the restore to fail for the table associated with the error. Other tables should still be restored successfully.

  • Errors associated with duplicate keys usually indicate a conflict between existing rows in the table and the rows being restored.

    pg_restore: [archiver (db)] Error from TOC entry 51348; 0 17943 TABLE DATA exporters plixer
pg_restore: [archiver (db)] COPY failed for table "exporters": ERROR:  duplicate key value violates unique constraint "exporters_pkey"
DETAIL:  Key (exporter_id)=(\x0a4d4d0a) already exists.

    The conflicting keys should be removed from the table before attempting to restore again.

  • If you are swapping IP addresses, the database keys should be rotated using scrut_util --pgcerts --verbose, because the backed up keys will be associated with the old address.