Data migration

Data migration#

The scrut_util --migrate command can be used to migrate configuration and/or historical data from source Scrutinizer server to a destination server.

Note

Because the steps outlined here can potentially result in an irrecoverable state for the source and/or destination Scrutinizer server, it is highly recommended to contact Plixer Technical Support for assistance. In case of issues, the migration utility logs (/var/log/migrate.log) can be provided to help with troubleshooting.

Source and destination server support#

The migration utility includes separate runmodes for configuration data migration and historical data migration.

The following table shows supported migration modes between Scrutinizer versions:

View table

19.5.x destination

19.6.x destination

19.7.x destination

18.20 source

Data

Data

Data

19.4.0 source

Configuration and data

Configuration and data

Configuration and data

19.5.x source

Configuration and data

Configuration and data

Configuration and data

19.6.x source

Configuration and data

Configuration and data

19.7.x source

Configuration and data

Note

  • If a Scrutinizer 18.20 server is upgraded to v19.4.0, its configuration data can be migrated to servers running v19.5.0 and above.

  • The utility may report pg_restore errors after cross-server configuration migrations. These errors can safely be ignored, and any issues are addressed by re-running the installer as described in these configuration migration instructions.

Source and destination server setup#

Follow the steps below to prepare source and destination Scrutinizer servers for migration.

Source server#

Run the following on the source server to create a temporary migration role and allow connections from the destination server using the role:

DESTINATION=DESTINATION_IP
psql -c "CREATE USER migrator WITH SUPERUSER ENCRYPTED PASSWORD 'ROLE_PASSWORD'"
sudo sed -i -e "1ihost plixer migrator $DESTINATION/32 md5" /var/db/big/pgsql/data/pg_hba.conf
psql -c "SELECT pg_reload_conf()"

Note

For Scrutinizer 18.20 source servers, use the following commands instead:

DESTINATION=x.x.x.x
psql -Uroot plixer -c "CREATE USER migrator WITH SUPERUSER ENCRYPTED PASSWORD 'replace_me'"
sudo sed -i -e "1ihost plixer migrator $DESTINATION/32 md5" /var/db/big/pgsql/data/pg_hba.conf
psql -Uroot plixer  -c "SELECT pg_reload_conf()"

Destination server#

Before starting the migration, the destination server must be deployed, licensed, and already collecting/receiving data.

Important

  • If the source and destination servers are collecting data from the same exporters prior to the migration, reports can be run against the migrated data after the migration.

  • Configuration migrations will overwrite the destination server’s configuration.

  • When performing a data migration after the configuration migration, starting a tmux session beforehand is highly recommended, as the migration may take some time. The flow collection service should also be running during the data migration.

After verifying that the destination is collecting data, update /etc/migrate.ini with the details of the source server and the temporary migration role.

Configuration migrations#

Configuration migrations transfer application configuration and user preference data from the source Scrutinizer server to the destination.

After the source and destination servers have been set up, follow these steps to perform a configuration (and optional historical data) migration:

View instructions

  1. Run the following on the destination server to transfer the source server’s encryption key, pause collection services, and start the migration:

    scp $SOURCE:/etc/plixer.key /etc/plixer.key
sudo systemctl stop plixer_collector
scrut_util --migrate config

  2. Once the migration is complete, reinstall the package for the destination server’s current Scrutinizer version and restart the flow collection service:

    PKG=$(rpm -qa plixer-scrutinizer_pg\*)
sudo yum reinstall -y $PKG
sudo systemctl restart scrutinizer

  3. Log in to the web interface and verify that all configuration data and user preferences have been successfully migrated.

  4. [Optional] Run the following command on the destination server to also migrate historical data from the source:

    scrut_util --migrate data

  5. Return to the web interface and run any report to verify that the historical data migration was successful.

  6. Remove the hba rule and temporary migrator role on the source server:

    sudo sed -i '/migrator/d' /var/db/big/pgsql/data/pg_hba.conf
psql -c "SELECT pg_reload_conf()"
psql -c "REASSIGN OWNED BY migrator TO plixer"
psql -c "DROP ROLE migrator"

Note

  • Configuration migrations only include Scrutinizer application data and not the system configuration. E.g., If a custom listening port was added via the Scrutinizer admin menu, that setting will be migrated, but the corresponding port will not be opened on the system. In such cases, undo and re-apply the setting in Scrutinizer after the migration to update the configuration.

  • The distributed cluster configuration on the destination Scrutinizer server are retained by default. These settings can also be migrated (while still preserving the destination server’s IP address and machine ID) by including the --force_dist option as follows:

    scrut_util --migrate config --force_dist

  • To extend the utility to also migrate contributions to the Scrutinizer SQL codebase, update /home/plixer/scrutinizer/database/utils/scrut_conf_dump.sh to include them.

Historical data migration#

If only historical data needs to be migrated, follow these steps after completing the setup steps for the source and destination servers:

Important

  • The collector service (plixer_collector) should be running on the destination server before and during the migration. The data migration process may take some time, depending on the amount of NetFlow data to be migrated.

  • If the source and destination servers are collecting data from the same exporters prior to the migration, reports can be run against the migrated data after the migration.

View instructions

  1. Start a new tmux session on the destination server:

    tmux new -s migration

  2. Run the migration utility:

    scrut_util --migrate data

  3. Confirm to start the migration when prompted.

  4. Log in to the web interface and run any report to verify that the historical data migration was successful.

  5. Remove the hba rule and temporary migrator role from the source server:

    sudo sed -i '/migrator/d' /var/db/big/pgsql/data/pg_hba.conf
psql -c "SELECT pg_reload_conf()"
psql -c "REASSIGN OWNED BY migrator TO plixer"
psql -c "DROP ROLE migrator"
Contents