Backups

Backups#

The Scrutinizer filesystem includes utilities that automate the process of creating or restoring system backups.

Note

  • These utilities are recommended for most long-term backup scenarios, because they include all database configuration and historical data for a Scrutinizer instance. Native snapshots may still be used as a short-term recovery option when there is no need to store the data, e.g., when upgrading the instance.

  • For Scrutinizer instances deployed on AWS, backups should be created and/or restored using native AWS functionality.

These utilities allow several types of backup and restore operations to be performed by the user.

Full backups#

Full or comprehensive backups are disaster-recovery-grade images of a Scrutinizer instance and include the following elements of the filesystem:

  • Application data and collected NetFlow in the PostgreSQL database

  • Host index data in BadgerDB databases

  • Scrutinizer’s third-party encryption key - /etc/plixer.key

  • Web Server TLS certificate and key

Important

  • The license key (if the instance is a primary reporter) and the TLS certificates and keys generated by Scrutinizer are not backed up and cannot be restored.

  • Any files not included in full backups must be manually backed up and restored, including:

    • Custom threat lists created under /home/plixer/scrutinizer/files/threats

    • Custom notifications created under /home/plixer/scrutinizer/files

    • LDAP authentication certificates

Creating full backups#

The Scrutinizer filesystem includes the backup.sh utility, which automates the creation of full backups. This script is located under home/plixer/scrutinizer/files.

Note

The default runmode of backup.sh saves the backup file locally. Due to the size of full backup files, however, the remote method outlined below is recommended.

The following instructions cover the process of creating and saving full Scrutinizer instance backups to a specified remote host:

View instructions

  1. SSH to the Scrutinizer server to be backed up and start a tmux session to prevent timeouts:

    tmux new -s backup

  2. Allow others to use FUSE mounts:

    sudo grep -Eq "^user_allow_other" /etc/fuse.conf || \
sudo sed -i '$ a user_allow_other' /etc/fuse.conf

  3. Create the backup directory locally and mount it to an empty directory on the remote host:

    BACKUPDIR=/mnt/backup
sudo mkdir -p $BACKUPDIR
sudo chown plixer:plixer $BACKUPDIR
sshfs -o allow_other -o reconnect REMOTE_USER@REMOTE_HOST:REMOTE_DIRECTORY $BACKUPDIR

    Important

    Verify that the remote directory to be used is empty and there is sufficient storage available, before running the backup script in the next step. For a rough estimate of the backup file size, run the following on the Scrutinizer instance:

    df -h /var/db | awk '!/^Filesystem/ {print "Space Required: "$3}'

  4. Run backup.sh as the plixer user, with the mounted remote directory set as the backup file location:

    BACKUPDIR=/mnt/backup ~plixer/scrutinizer/files/backup.sh

  5. Once the script confirms that the backup file has been saved, unmount the remote backup directory:

    BACKUPDIR=/mnt/backup
fusermount -u $BACKUPDIR
sudo rmdir $BACKUPDIR

Full backup files are created as scrutinizer-VERSION-backup-DATE.tar.gz at the specified location and owned by the plixer user.

Note

  • A second Scrutinizer instance can be used as the remote backup host, provided it has sufficient disk space available and is running the same Scrutinizer version as the instance to be backed up. However, doing so is only recommended for redundancy.

  • If no remote hosts are available, backups can be saved locally on the same Scrutinizer instance. However, this will limit the amount of storage available for system functions and is not recommended.

For further details or assistance with issues, contact Plixer Technical Support.

Backing up additional files#

When creating a full backup of a Scrutinizer server, any files not covered by the script must be manually backed up and should be stored on an external host/system.

These files should also be manually restored, after running the restore script.

Restoring from a full backup#

To restore a Scrutinizer instance from a full backup file, use the restore.sh utility located under home/plixer/scrutinizer/files.

The script will fully restore all backed up elements of a Scrutinizer instance, provided the following conditions are met:

  • A valid full backup file is accessible by the plixer user at the specified ($BACKUPDIR) remote location.

  • The Scrutinizer instance to be used for the restore has been freshly deployed.

  • The version of the backup matches the version of the fresh Scrutinizer instance to restore to (e.g. a 19.3.0 backup can only be restored to a new 19.3.0 instance).

Important

  • A restore completely overwrites the state of the target instance and deletes the source backup file. It is highly recommended to always restore from a copy of a backup file.

  • If the restore target is the primary reporter in a distributed cluster, contact Plixer Technical Support for assistance.

The following instructions cover the process of restoring from a backup file on a remote host to a fresh Scrutinizer deployment:

View instructions

  1. SSH to the target Scrutinizer server for the restore, and start a tmux session to prevent timeouts:

    tmux new -s restore

  2. Allow others to use FUSE mounts:

    sudo grep -Eq "^user_allow_other" /etc/fuse.conf || \
sudo sed -i '$ a user_allow_other' /etc/fuse.conf

  3. Create the backup directory locally and mount the remote directory containing the backup file(s):

    BACKUPDIR=/mnt/backup
sudo mkdir -p $BACKUPDIR
sudo chown plixer:plixer $BACKUPDIR
sshfs -o allow_other -o reconnect REMOTE_USER@REMOTE_HOST:REMOTE_DIRECTORY $BACKUPDIR

  4. Run restore.sh as the plixer user, with the remote directory set as the backup file location:

    BACKUPDIR=/mnt/backup ~plixer/scrutinizer/files/restore.sh

  5. When prompted, enter yes to select the backup file to use for the restore or no to have the script continue searching (if the backup file was not previously specified).

    Hint

    To specify the file to use for the restore, use BACKUPDIR=/mnt/backup BACKUP=restore_filename.tar.gz ~plixer/scrutinizer/files/restore.sh at the previous step instead.

  6. Once the script confirms that the restore has been completed, unmount the remote backup directory:

    BACKUPDIR=/mnt/backup
fusermount -u $BACKUPDIR
sudo rmdir $BACKUPDIR

Important

The restore.sh utility does not restart Scrutinizer services after it completes running.

Based on the role of the Scrutinizer instance, proceed to finalize setup of the restored server:

  • If the restored instance is a standalone server, run the following to restart all services and register it:

    scrut_util --services --name all --switch restart
scrut_util --set selfregister --reset

    These commands may take several minutes to complete.

  • If the restored instance is a remote collector in a distributed cluster, run the following on the primary reporter to register it:

    scrut_util --set registercollector --ip RESTORED_INSTANCE_IP

  • If the restored instance is a primary reporter in a distributed cluster or a standalone server, and its Machine ID is different from that of the backup file, contact Plixer Technical Support to obtain a new license key.

Alternative backup methods#

Because full backup files are extremely large and intended for use in disaster recovery scenarios, saving and storing backup files to remote hosts serving ssh is highly recommended.

In scenarios where this is not possible, the following alternative backup methods can be used:

Backup to a second Scrutinizer instance#

If a separate host is not available to save backups to, a second Scrutinizer instance can be used for backup file storage instead. The versions of the two instances must match.

Important

Due to how Scrutinizer is designed to optimize the use of all available disk space, it will likely be necessary to add more storage and/or modify the data retention settings of the second instance. For assistance, contact Plixer Technical Support.

The following instructions cover the additional steps required for creating backups on a second Scrutinizer instance (using the default location):

View instructions

  1. Set the location/directory to use for backup files:

    BACKUPDIR=${BACKUPDIR:='/var/db/big/pgsql/restore'}
REMOTE=YOUR_REMOTE_SCRUTINIZER_INSTANCE

  2. Create the backup directory on both instances:

    sudo mkdir -p $BACKUPDIR
sudo chown plixer:plixer $BACKUPDIR
ssh plixer@$REMOTE "sudo su -c 'mkdir -p $BACKUPDIR && chown plixer:plixer $BACKUPDIR'"

  3. Allow other users to use FUSE mounts:

    sudo grep -Eq "^user_allow_other" /etc/fuse.conf || \
sudo sed -i '$ a user_allow_other' /etc/fuse.conf

  4. Mount the remote instance’s backup directory on the local instance:

    sshfs -o allow_other -o reconnect plixer@$REMOTE:$BACKUPDIR $BACKUPDIR

  5. Run backup.sh using the directory mounted from the remote instance as the backup file location:

    BACKUPDIR=/var/db/big/pgsql/restore ~plixer/scrutinizer/files/backup.sh

    Important

    Before running the backup utility, verify that the remote directory to be used is empty and there is sufficient storage available. For a rough estimate of the backup file size, run the command df -h /var/db | awk '!/^Filesystem/ {print "Space Required: "$3}' on the Scrutinizer instance.

  6. After the backup is complete, unmount the remote Scrutinizer directory:

BACKUPDIR=${BACKUPDIR:='/var/db/big/pgsql/restore'}
fusermount -u $BACKUPDIR

Local backups#

By default, both backup.sh and restore.sh are set to use /var/db/big/pgsql/restore on the local Scrutinizer filesystem for full backup files. However, in most cases, the backup operation will likely fail unless additional disk space is allocated to or created on the Scrutinizer instance. Running the command df -h /var/db | awk '!/^Filesystem/ {print "Space Required: "$3}' will provide a rough estimate of the storage required for the backup.

To force a checkpoint, enter psql plixer -c "CHECKPOINT" after the script has finished running.

Note

  • In v19.2, the backup file path must be defined in the backup.sh and restore.sh scripts before they are run.

  • Storing backup files locally will severely limit the storage Scrutinizer can use for its primary functions. As such, backup files saved to the instance should be transferred to a separate resource as soon as possible.

Configuration backups#

For more “lightweight” backup and restore operations, the scrut_conf_dump.sh and scrut_conf_restore.sh scripts (both located in /home/plixer/scrutinizer/database/utils) can be used to target only the application/configuration data of a Scrutinizer instance, including:

  • User-added maps

  • Dashboards

  • IP groups

  • Saved reports

  • 3rd-party integration settings

Configuration backups do not include any collected flow data.

Note

In distributed environments, the primary reporter regularly syncs application/configuration data to remote collectors. Only the configuration backup of the primary reporter is needed to perform a restore for the cluster.

scrut_conf_dump.sh and scrut_conf_restore.sh use Postgres’s pg_dump and pg_restore utils and respect the same set of environment variables:

Variable

Description

Default

DUMP

Location of the backup file

./conf.dump

PGHOST

IP address or hostname of the PostgreSQL database

localhost

PGUSER

Role/user used to connect to PGHOST

plixer

PGDATABASE

The database to access at PGHOST

plixer

Backing up configuration data#

To create a backup of a Scrutinizer server’s current configuration data, follow these steps:

View instructions

  1. Run the backup script.

    To save the backup file to the default location:

    ~/scrutinizer/database/utils/scrut_conf_dump.sh

    To use a custom location/filename:

    mkdir /tmp/CONF_BACKUP_DIR
touch /tmp/CONF_BACKUP_DIR/CONF_BACKUP.dump
DUMP=/tmp/CONF_BACKUP_DIR/CONF_BACKUP.dump ~/scrutinizer/database/utils/scrut_conf_dump.sh

  2. Restart the stopped services:

    sudo systemctl restart scrutinizer

Restoring configuration data#

To restore configuration data to a Scrutinizer server from a backup file, follow these steps:

View instructions

  1. Stop the plixer_webapp and plixer_collector services:

    sudo systemctl stop plixer_webapp
sudo systemctl stop plixer_collector

  2. Run the restore script.

    To restore from the default backup location/file:

    ~/scrutinizer/database/utils/scrut_conf_dump.sh

    To restore from a specified location/file:

    PGHOST=SCRUTINIZER_IP
DUMP=/tmp/CONF_BACKUP_DIR/CONF_BACKUP.dump ~/scrutinizer/database/utils/scrut_conf_restore.sh

  3. Restart the stopped services:

    sudo /bin/systemctl start plixer_webapp
sudo /bin/systemctl start plixer_collector

  4. Resync the access table:

    psql -c "SELECT setval(pg_get_serial_sequence('plixer.access', 'access_id'), COALESCE(max(access_id) + 1, 1), false) FROM plixer.access;"

Note

scrut_conf_restore.sh should only be used for restoring configuration data for the same Scrutinizer server/appliance. To apply a configuration backup to a different server, follow the steps for backup migrations in the migration guides.

Additional notes#

  • pg_restore errors typically only cause the restore to fail for the table associated with the error. Other tables should still be restored successfully.

  • Errors associated with duplicate keys usually indicate a conflict between existing rows in the table and the rows being restored.

    pg_restore: [archiver (db)] Error from TOC entry 51348; 0 17943 TABLE DATA exporters plixer
pg_restore: [archiver (db)] COPY failed for table "exporters": ERROR:  duplicate key value violates unique constraint "exporters_pkey"
DETAIL:  Key (exporter_id)=(\x0a4d4d0a) already exists.

    The conflicting keys should be removed from the table before attempting to restore again.

  • If you are swapping IP addresses, the database keys should be rotated using scrut_util --pgcerts --verbose, because the backed up keys will be associated with the old address.

Contents