Upgrades and updates

Upgrades and updates#

To ensure a consistently feature-rich and secure experience, all supported versions of Scrutinizer will continuously be updated. When installed, update packages may add new features, improve existing functionality, and/or apply patches for emerging security threats. All update packages will have been applied to Plixer’s own QA servers and extensively tested before they are made available.

Important

While it is possible to install Scrutinizer update packages without assistance, it is highly recommended to contact Plixer Technical Support and allow our engineers to guide you through the process.

On this page:

Update preparations
Update preparations
Version upgrades
Version upgrades
General and CVE patches
General and CVE patches
Vulnerability patch verification
Vulnerability patch verification

Update preparations#

Before attempting to install any type of update package, the following procedures should be observed:

  1. Verify that the version currently installed can be upgraded to the target version (e.g., v18.20 or v19.x -> v19.4.0).

  2. Back up the current install:

    • Virtual appliances: Take a snapshot, ideally with the appliance powered off.

    • Hardware appliances: Perform a full or configuration backup. For further details, see the Backups subsection of this documentation or contact Plixer Technical Support.

  3. Hardware appliances only - Log in to iDRAC and perform a hardware health check. Any hardware issues discovered should be escalated to Dell for resolution. A reboot is also recommended as an additional check for underlying hardware issues.

  4. Confirm that all Scrutinizer collectors/servers have access to https://files.plixer.com. This check can be performed by downloading the checksum file using the following command:

    curl -O https://files.plixer.com/plixer-repo/scrutinizer/19.7.0/scrutinizer-install.run.sha256

    For Scrutinizer deployments that do not have internet access, download the file from the REPO_HOST_IP for the offline yum/dnf repository instead.

  5. Collect the following details and check the Scrutinizer sizing guide to confirm that sufficient resources will be available to the system after the upgrade:

    • Flows per second

    • Number of active Exporters

    • CPU (number of cores, clock speeds)

    • Amount of RAM

    • Disk speed and RAID type

    • Flow Analytics algorithms enabled

  6. Obtain a valid license key for the upgrade if one has not been acquired.

  7. Delete any older versions of scrutinizer-installer.run on the Scrutinizer instance. This will prevent them from being used instead of the correct installer.

  8. Run crontab -e and inspect the table for lines containing * * * * * /home/Plixer/scrutinizer/files/collector_restart.sh. These should be commented out by adding a # at the beginning of the line to prevent scheduled restarts from interfering with the upgrade process.

  9. Distributed cluster upgrades only - If there are Palo Alto firewalls configured for the cluster, whitelist the connections between the primary reporter and remote collectors. This will prevent the firewall from identifying the ~113 SSH connections created during the collector registration process as a threat. Alternatively, the rate at which the SSH connections are established can be slowed down by adding sleep 5 to the /home/plixer/.bashrc file on each remote collector.

  10. AWS flow log integration only - As of version 19.2, Scrutinizer requires four log fields to be configured for AWS flog log collection: log-status, vpc-id, interface-id, and flow-direction. For further details, see the AWS flow log integration guide.

These steps are meant to identify and resolve any underlying issues with the current Scrutinizer install and help ensure that the upgrade will be applied without issue.

Once completed, follow the appropriate upgrade guide to update Scrutinizer to the latest version.

Note

All install logs will be saved to /var/log/Scrutinizer-Install.log.

Changed in version 19.1: plixer is the recommended OS user for command line access. The root user is no longer required.

Version upgrades#

Version upgrades update Scrutinizer to the latest major or minor release (e.g., 19.4) and include significant improvements over the previous version. These upgrades may include additional functionality, performance enhancements, and/or QoL improvements, in addition to implementing fixes for certain types of issues.

Latest Scrutinizer release#

After completing the recommended update preparations, follow the instructions below to upgrade Scrutinizer to the latest version.

Note

  • Only deployments on v19.5.3 (v19.5.4 for AWS AMIs) can be upgraded directly to v19.6.0 and beyond. For older versions, follow the steps in these guides to upgrade to the required Scrutinizer 19.5.x release before upgrading to the latest version.

  • After Scrutinizer has been upgraded to v19.7.0 or higher, contact Plixer Technical Support for a new Plixer One Core or Plixer One Enterprise key to enable AI features.

  • If the Scrutinizer server being upgraded does not have Internet access, an internal NTP server can be configured by running the following:

    sed -i -e '/^pool/aserver NTP_ADDRESS' -e 's/^pool/#&/' /etc/chrony.conf

  • For AWS deployments, contact Plixer Technical Support to obtain the latest AMI installer.

Online upgrades#

To download and install the latest version upgrade for Scrutinizer, follow these steps:

View instructions

  1. SSH to the primary reporter as the plixer user:

    ssh plixer@SCRUTINIZER_IP

  2. Start a new tmux session (to maintain the upgrade session if the SSH connection is lost):

    tmux new -s upgrade

  3. Download the installer and checksum file for the latest version:

    curl -O https://files.plixer.com/plixer-repo/scrutinizer/19.7.0/scrutinizer-install.run
curl -O https://files.plixer.com/plixer-repo/scrutinizer/19.7.0/scrutinizer-install.run.sha256

    ```{note} For Scrutinizer deployments that do not have Internet access, use the REPO_HOST_IP for the offline yum/dnf repository instead.

  4. Verify the checksum:

    sha256sum -c scrutinizer-install.run.sha256

  5. Set the correct permissions for the installer:

    chmod 755 scrutinizer-install.run

  6. Run the installer as the plixer user:

    ./scrutinizer-install.run

    For offline upgrades, use:

    REPO_HOST=REPO_HOST_IP ./scrutinizer-install.run -- -k

  7. [Distributed cluster upgrades only] When prompted for the authentication method to use for remote collectors in the cluster, enter either existing (recommended) or passwords.

  8. After the installer finishes running, execute the following heartbeat checks to verify communication between nodes:

    scrut_util --check heartbeat --type database
scrut_util --check heartbeat --type api

If the heartbeat checks are successful, the upgrade is complete.

Offline upgrades#

To upgrade Scrutinizer collectors/servers that are unable to access the default default yum/dnf repository on https://files.plixer.com/plixer-repo/scrutinizer/19.7.0, an offline repository will need to be set up on the local network. The local repository can be hosted on the primary Scrutinizer server or another host on the network.

To set up the offline repository on the primary Scrutinizer server (with IP address REPO_HOST_IP), follow these steps:

View instructions

  1. Download the offline repo package and checksum file on a host with Internet access:

    curl -O https://files.plixer.com/plixer-repo/scrutinizer/19.7.0_offline.tgz
curl -O https://files.plixer.com/plixer-repo/scrutinizer/19.7.0_offline.tgz.sha256

  2. Start an SSH session with the primary reporter as the plixer user:

    ssh plixer@REPO_HOST_IP

  3. Confirm that /var/db/big has at least 84 GB of free disk space:

    df -h --output='avail' /var/db/big

  4. Create a new directory for the offline installation files and set the correct permissions to give the plixer user access to it:

    sudo mkdir -p /var/db/big/offline
sudo chown plixer:plixer /var/db/big/offline

  5. On the Internet-connected host, copy the offline bundle and checksum file downloaded in step 1 to the repo host:

    scp 19.7.0_offline.tgz* plixer@REPO_HOST_IP:/var/db/big/offline/

  6. On the repo host, validate the checksum on the repo host:

    (cd /var/db/big/offline/ ; sha256sum -c 19.7.0_offline.tgz.sha256)

  7. Extract the repository:

    tar -zxvf /var/db/big/offline/19.7.0_offline.tgz -C /var/db/big/offline

  8. Create a link to the offline repo in a directory accessible to the web server:

    sudo -u webapp ln -sf /var/db/big/offline/plixer-repo /home/webapp/html/

    ```{note} For older versions, use ln -sf /var/db/big/offline/plixer-repo /home/plixer/scrutinizer/html/plixer-repo instead.

  9. Export the repo host’s IP address:

    export REPO_HOST=REPO_HOST_IP

Once the offline repository has been set up, follow these steps to proceed with the upgrade.

Plixer ML Engine#

Review/complete the recommended update preparations, and then follow these steps to upgrade a ML Engine deployment to the latest version:

Note

Scrutinizer 19.7.0 requires Plixer ML Engine deployments to be upgraded to v19.5.0 or the latest available version.

View instructions

  1. SSH to the ML Engine VM (i.e., the host used for management/deployment) as the plixer user.

  2. Download the installer for the latest version:

    curl -o plixer-machine-learning-update.run https://files.plixer.com/scripts/plixer-machine-learning/release/19.5.0/plixer-machine-learning-update.run

  3. Download the checksum file and validate the integrity of plixer-machine-learning-update.run:

    curl -o plixer-machine-learning-checksums.txt https://files.plixer.com/scripts/plixer-machine-learning/release/19.5.0/plixer-machine-learning-checksums.txt
cat plixer-machine-learning-checksums.txt
sha256sum plixer-machine-learning-update.run

  4. Set the correct permissions for the installer:

    chmod +x plixer-machine-learning-update.run

  5. Run the installer as the plixer user:

    STAGE="release"
VERSION="UPGRADE_VER"
STAGE=$STAGE ./plixer-machine-learning-update.run

After the installer script completes running, setup.sh will automatically be run to pull in any configuration changes and redeploy pods with new images.

Note

If any changes were previously made to pxi-settings.yaml, azure.tfvars, aws.tfvars, or vsphere.tfvars, the file(s) will be retained even if the upgrade package includes a newer version of the file. The updated file will instead be saved with a .dpk-dist extension, and any necessary edits should be migrated before it is used to overwrite the old configuration/tfvars file.

Once the upgrade process is complete, wait for the rke2-server service to restart. This sequence can be monitored by running:

journalctl -xeu rke2-server -f

Additional notes for ML Engine upgrades from v19.4.0 to v19.5.0#

  • Scrutinizer 19.6.0 includes new management/configuration functions for the ML Engine, requiring all attached engine deployments to also be upgraded from v19.4.0 to v19.5.0 or higher.

  • After Scrutinizer is upgraded to v19.6.0 or higher, all previous settings related to attached ML Engine deployments will be reset. Engines will need to be re-registered (but not re-deployed) via the Scrutinizer web interface before being upgraded to v19.5.0.

  • When upgrading the ML Engine from v19.4.0 to v19.5.0, setup.sh --reconfigure will automatically be run (instead of setup.sh as described above) to initiate the new configuration process and collect all required information (including the authentication token generated by Scrutinizer).

  • If the ML Engine is deployed as a standalone VM, new Docker images will be downloaded (may take several minutes) after the package updates. This step is skipped for cloud deployments.

Pre-19.5.x Scrutinizer deployments#

The Scrutinizer 19.5.0 upgrade includes the migration to Oracle Linux 9, which will be required for all new versions/releases going forward. Deployments on older versions must first be upgraded to the latest v19.5.x release before being upgraded further.

The following guides provide instructions for the required upgrade(s):

Upgrading from older versions to Scrutinizer 19.4.0 (required to upgrade to v19.5.3/v19.5.4)#

View guide

Pre-v19.4.0 Scrutinizer deployments must first be upgraded to v19.4.0 before being upgraded to v19.5.3 (or v19.5.4 for AWS AMI appliances), which includes the migration to Oracle Linux 9.

Follow these instructions to download the v19.4.0 installer (replace 19.7.0 with 19.4.0 in the download URLs) and apply the update. Once done, proceed with upgrading to the latest v19.5.x release.

Note

  • When upgrading an appliance that was previously upgraded from v18.20, the installer script will ask whether to delete the data.old backup created during that upgrade. Since a more recent backup should be created before the current upgrade process, this file can safely be deleted.

  • If a distributed cluster is being upgraded from v18.20, the prompt to create a new Plixer control key should be left blank unless encrypted keys are required. Additionally, passwords should be selected in the next step, when prompted for the login method to use for remote collectors.

Upgrading from 19.4.0 to 19.5.3#

View guide

Follow the steps outlined below to upgrade a Scrutinizer deployment on v19.4.0 or above to v19.5.3.

To upgrade an AWS AMI from 19.4.0 to v19.5.4, follow this guide instead. For older versions, refer to this guide to upgrade to v19.4.0 before proceeding.

Note

  • The upgrade will take at least one hour to complete.

  • The plixer user SSH password will be needed during the upgrade. If necessary, it can be reset when the OS upgrade script is run.

  • If root SSH login is enabled on the Scrutinizer server, it will be disabled as part of the upgrade.

  • If upgrading from v19.5.0 or above, proceed directly to upgrading to Scrutinizer 19.5.3.

  • If the Scrutinizer server is able to access files.plixer.com, the REPO_HOST variable should be set to files.plixer.com for the steps outlined below. For offline upgrades, the IP address of the offline repo should be used instead.

For assistance or clarifications, contact Plixer Technical Support.

Upgrade process

The process of upgrading a v19.4.0 Scrutinizer server to v19.5.3 involves the following steps:

  • Backing up the current install’s database and server-specific files

  • Downloading the operating system upgrade script, olmigrate.run, and running it a total of four times (with a reboot between runs). This only applies if upgrading from v19.4.0.

  • Downloading and running the Scrutinizer v19.5.3 installation script (scrutinizer-install.run)

  • Verifying that the current install’s data has been successfully migrated after v19.5.3 is installed

Pre-upgrade preparation

Distributed cluster upgrades

Nodes in distributed environments must be reverted to standalone appliances before being individually upgraded to v19.5.3:

View instructions

  1. Navigate to Admin > Resources > Collectors and delete all remote collectors.

  2. SSH to each remote collector as the plixer user and register it as a standalone appliance:

    scrut_util --set selfregister --reset

  3. Verify that each appliance is now running in standalone mode (no other addresses under collector_ips):

    scrut_util --check dist_info

When done, proceed with the OS migration and v19.5.3 upgrade for each node, and then rebuild the distributed cluster.

OS migration

Once all preparation steps have been completed, follow these steps to migrate the v19.4.0 appliance to the new operating system:

View instructions

Important

  • For offline upgrades, REPO_HOST should point to the IP address of the offline repo instead of files.plixer.com.

  • In distributed clusters, complete the upgrade for all remote collectors before upgrading the primary reporter.

  • To verify the current progress of the OS upgrade at any time:

    cat /etc/motd

    or check versions between runs (NAME= and VERSION= lines):

    cat /etc/os-release

  • If any errors are encountered during the upgrade process, run the following to collect log files:

    sudo tar -czf /tmp/olmigrate_logs.tar.gz /var/log/olmigration/ /var/log/leapp/ /var/log/messages /var/log/Scrutinizer-Install.log

    Afterwards, move /tmp/olmigrate_logs.tar.gz off the server before reverting. Plixer Technical Support will require the logs to better assist you with any issues.

  1. SSH to the v19.4.0 server to be upgraded as the plixer user.

  2. Verify that the current working directory is correct (plixer):

    cd /home/plixer/

  3. Download the OS upgrade script and its checksum file:

    REPO_HOST=files.plixer.com
curl -k -o olmigrate.run https://$REPO_HOST/plixer-repo/scrutinizer/19.5.3/olmigrate.run
curl -k -o olmigrate.run.sha256 https://$REPO_HOST/plixer-repo/scrutinizer/19.5.3/olmigrate.run.sha256

  4. Validate the integrity of olmigrate.run:

    sha256sum -c olmigrate.run.sha256

  5. Update permissions for the OS upgrade script:

    chmod a+x olmigrate.run

  6. Run the olmigrate.run script a total of four times:

    REPO_HOST=files.plixer.com ./olmigrate.run -- -k

    Important

    Reboots between runs of the OS upgrade script (olmigrate.run) can take a long time. Before trying to reconnect to the server, start a PING to the Scrutinizer IP address and wait for it to become available again. Do NOT manually reboot the server.

After the fourth olmigrate.run run (there will be no reboot), the OS migration will be complete.

Upgrading to Scrutinizer 19.5.3

Once the appliance is on the new OS, Scrutinizer can be upgraded to v19.5.3 as follows:

View instructions

  1. Change directories to /tmp:

    cd /tmp/

  2. Download the Scrutinizer v19.5.3 installation script and its checksum file:

    REPO_HOST=files.plixer.com
curl -k -o scrutinizer-install.run https://$REPO_HOST/plixer-repo/scrutinizer/19.5.3/scrutinizer-install.run
curl -k -o scrutinizer-install.run.sha256 https://$REPO_HOST/plixer-repo/scrutinizer/19.5.3/scrutinizer-install.run.sha256

  3. Validate the integrity of scrutinizer-install.run:

    sha256sum -c scrutinizer-install.run.sha256

  4. Update permissions for the installation script:

    chmod a+x scrutinizer-install.run

  5. Run scrutinizer-install.run to begin the upgrade to Scrutinizer v19.5.3:

    REPO_HOST=files.plixer.com ./scrutinizer-install.run -- -k

  6. After the installation script finishes running, reboot the appliance:

    sudo shutdown -r now

  7. After the reboot, run the following commands to verify that the system is in working order:

    scrut_util --check heartbeat --type database
scrut_util --check heartbeat --type api

    Important

    For distributed environments, the heartbeat checks should only be run on remote collectors after the primary reporter has been upgraded, and the cluster has been reestablished.

If the heartbeat checks are successful, then the Scrutinizer appliance has been successfully upgraded to v19.5.3.

Offline upgrades to v19.5.3

The following instructions for setting up an offline repo are intended for upgrading to Scrutinizer v19.5.3 only.

View instructions

  1. Deploy a new Scrutinizer v19.4.0 VM and assign an IP address to it.

  2. SSH to the VM as the plixer user:

    ssh plixer@SCRUTINIZER_VM_IP

  3. Create the offline repo directory and assign it the correct permissions:

    sudo mkdir /var/db/big/offline
sudo chown plixer:plixer /var/db/big/offline

  4. Download the offline tar file for 19.5.3 and its checksum file:

    curl -o /var/db/big/offline/19.5.3_offline.tgz https://files.plixer.com/plixer-repo/scrutinizer/19.5.3_offline.tgz
curl -o /var/db/big/offline/19.5.3_offline.tgz.sha256 https://files.plixer.com/plixer-repo/scrutinizer/19.5.3_offline.tgz.sha256

  5. Validate the integrity of 19.5.3_offline.tgz:

    sha256sum -c /var/db/big/offline/19.5.3_offline.tgz.sha256

  6. Extract the offline tar file:

    cd /var/db/big/offline
tar xvf 19.5.3_offline.tgz

  7. Create a symlink in the html directory to the offline repo:

    ln -s /var/db/big/offline/plixer-repo /home/plixer/scrutinizer/html/plixer-repo

After the offline repo has been set up, the VM’s IP address should be used in place of files.plixer.com for REPO_HOST in the upgrade instructions.

Upgrading from Scrutinizer 19.4.0 to 19.5.4 (AMI only)#

View guide

Follow the steps outlined below to upgrade a Scrutinizer AMI on v19.4.0 or above to v19.5.4.

For older versions, refer to this guide to upgrade to v19.4.0 before proceeding.

Note

  • The upgrade will take at least one hour to complete.

  • The plixer user SSH password will be needed during the upgrade.

  • If root SSH login is enabled on the Scrutinizer server, it will be disabled as part of the upgrade.

  • The new v19.5.4 instance must be in the same availability zone as the original v19.4.0 machine. Volumes outside the current availability zone will not be accessible from the AWS console.

Distributed cluster upgrades

Nodes in distributed environments must be reverted to standalone appliances before being individually upgraded to v19.5.4:

View instructions

  1. Navigate to Admin > Resources > Collectors and delete all remote collectors.

  2. SSH to each remote collector as the plixer user and register it as a standalone appliance:

    scrut_util --set selfregister --reset

  3. Verify that each appliance is now running in standalone mode (no other addresses under collector_ips):

    scrut_util --check dist_info

When done, proceed with upgrading each node as described below, and then rebuild the distributed cluster.

For assistance or clarifications, contact Plixer Technical Support.

Upgrade process

The process of upgrading a Scrutinizer 19.4.0 Scrutinizer AMI to v19.5.4 involves the following steps:

  • Backing up the current Scrutinizer install by taking a VM snapshot

  • Deploying a new v19.5.4 AMI appliance

  • Copying the dbexport.sh file from the new v19.5.4 appliance to the current v19.4.0 appliance

  • Detaching the storage volume from the v19.4.0 instance (using dbexport.sh and running as the root user)

  • Attaching the storage volume to the new v19.5.4 instance (using dbimport.sh and running as the root user)

  • Verifying that the v19.4.0 data has been successfully migrated after v19.5.4 is installed

Expanding storage

AMI deployments will require additional storage to be upgraded to v19.5.4.

View instructions

To verify whether the Scrutinizer 19.4.0 AMI instance is running on the default sizing, run the following:

df -h

If the output does not list a line that includes vg_scrut-lv_db, contact Plixer Technical Support for assistance with expanding storage before proceeding.

Upgrading to Scrutinizer 19.5.4

View instructions

  1. Copy the following file from the new v19.5.4 appliance to your current v19.4.0 appliance:

    /home/plixer/scrutinizer/files/dbimport/dbexport.sh

  2. Run the following command to make dbexport.sh executable:

    sudo chmod +x dbexport.sh

  3. SSH to the 19.4.0 appliance as the plixer user, and then navigate to the location where dbexport.sh was saved.

  4. Run the script to prepare the 19.4.0 storage volume to be detached:

    sudo ./dbexport.sh exportdb

  5. Shut down the Scrutinizer v19.4.0 instance.

  6. In the AWS EC2 management page, navigate to the Volumes page.

  7. In the Volume Management page, select the storage volume, click the Actions menu, and then select Detach volume. It may take a minute for the storage volume to go from In use to Available.

  8. Once the detached storage volume(s) are marked as Available (it may take several minutes), attach it to the Scrutinizer v19.5.4 instance. Refer to STEP 6 of the storage expansion instructions.

  9. After the storage volume(s) have been attached, SSH to the 19.5.4 instance as the plixer user.

  10. Run the following to import and set up the database on the 19.5.4 instance.

/home/plixer/scrutinizer/files/dbimport/dbimport.sh importdb <device name for the storage volume, e.g. /dev/xvdg>

Use the lsblk and show partitions commands to get the correct partition/device name to use. Once the script completes running, Scrutinizer will run a self-register reset that requires user input for verification.

  1. Add a new license key to fully activate your Scrutinizer v19.5.4 instance.

Note

  • If there are multiple volumes listed after dbexport.sh completes running, all volumes will need to be detached from the v19.4.0 instance and attached to the v19.5.4 instance.

  • At the end of the output from dbexport, the volumes that are part of the volume group for the database are listed. If the volume group contains more than one volume, the output will list all of those volumes, which will need to be detached and then attached to the Scrutinizer v19.5.4 instance.

  • When you first log in to the v19.5.4 UI to add a new Scrutinizer license, you must use the UI admin password for the v19.4.0 AWS instance. Alternatively, you can reset the UI admin password in scrut_util first.

General and CVE patches#

From time to time, customers may be notified that general and/or CVE patches are available for the Scrutinizer version they are currently running. These patches typically address noncritical system issues and/or improve protections against new security threats.

Note

General and CVE patches do not increment the Scrutinizer version number.

To apply these updates, follow the version upgrade instructions to download and run the latest installer for the current Scrutinizer version. Going through the standard update preparations is also highly recommended.

When run, the installer will automatically download and apply all available patches.

Vulnerability patch verification#

Some vulnerability scanning and auditing solutions may report vulnerabilities that have already been patched in the most recent update. This is typically the combined result of a backported security patch and the tool only scanning for component version numbers.

If this happens, there are two ways to verify the validity of the vulnerability report:

  • Check the package changelog for the CVE identifier/number of the vulnerability (e.g., CVE-2017-3169)

  • Download and install the latest OVAL definitions from oval.cisecurity.org/repository, which will allow any compatible tools to determine the status of vulnerabilities, even when security patches have been backported.

For additional assistance, contact Plixer Technical Support.

Contents