Data migration

Data migration#

The scrut_util --migrate command can be used to migrate configuration and/or historical data from source Scrutinizer server to a destination server.

Note

Because the steps outlined here can potentially result in an irrecoverable state for the source and/or destination Scrutinizer server, it is highly recommended to contact Plixer Technical Support for assistance. In case of issues, the migration utility logs (/var/log/migrate.log) can be provided to help with troubleshooting.

Source and destination server support#

The migration utility includes separate runmodes for configuration data migration and historical data migration.

The following table shows supported migration modes between Scrutinizer versions:

View table

19.5.x destination

19.6.x destination

19.7.x destination

18.20 source

Data

Data

Data

19.4.0 source

Configuration and data

Configuration and data

Configuration and data

19.5.x source

Configuration and data

Configuration and data

Configuration and data

19.6.x source

Configuration and data

Configuration and data

19.7.0 source

Configuration (19.7.1+ only) and data

Important

Configuration data cannot be migrated from a 19.7.0 source to a 19.7.0 destination.

Note

  • If a Scrutinizer 18.20 server is upgraded to v19.4.0, its configuration data can be migrated to servers running v19.5.0 and above.

  • The utility may report pg_restore errors after cross-server configuration migrations. These errors can safely be ignored, and any issues are addressed by re-running the installer as described in these configuration migration instructions.

Source and destination server setup#

Follow the steps below to prepare source and destination Scrutinizer servers for migration.

Source server#

Run the following on the source server to create a temporary migration role and allow connections from the destination server using the role:

Note

Replace ROLE_PASSWORD below with the desired password for the temporary migration user/role migrator. The password will also be required to prepare the destination server for the migration (see below).

DESTINATION=DESTINATION_IP
psql -c "CREATE USER migrator WITH SUPERUSER ENCRYPTED PASSWORD 'ROLE_PASSWORD'"
sudo sed -i -e "1ihost plixer migrator $DESTINATION/32 md5" /var/db/big/pgsql/data/pg_hba.conf
psql -c "SELECT pg_reload_conf()"

Important

For Scrutinizer 18.20 source servers, use the following commands instead:

DESTINATION=x.x.x.x
psql -Uroot plixer -c "CREATE USER migrator WITH SUPERUSER ENCRYPTED PASSWORD 'ROLE_PASSWORD'"
sudo sed -i -e "1ihost plixer migrator $DESTINATION/32 md5" /var/db/big/pgsql/data/pg_hba.conf
psql -Uroot plixer  -c "SELECT pg_reload_conf()"

Destination server#

Before starting the migration, the destination server must be deployed, licensed, and already collecting/receiving data.

Important

  • If the source and destination servers are collecting data from the same exporters prior to the migration, reports can be run against the migrated data after the migration.

  • Configuration migrations will overwrite the destination server’s configuration.

  • It is highly recommended to run any type of migration within a tmux session, as the operation may take some time.

  • The flow collection service should be running during data migrations.

After verifying that the destination is collecting data, update /etc/migrate.ini with the details of the source server as well as the migration role’s username (migrator) and password (ROLE_PASSWORD entered when the role was created).

Configuration migrations#

Configuration migrations transfer application configuration and user preference data from the source Scrutinizer server to the destination.

After the source and destination servers have been set up, follow these steps to perform a configuration (and optional historical data) migration:

View instructions

  1. Start a new tmux session on the destination server:

    tmux new -s migration

  2. Transfer the source server’s encryption key, pause collection services, and start the migration:

    SOURCE=SOURCE_SERVER_IP
scp $SOURCE:/etc/plixer.key /etc/plixer.key
sudo systemctl stop plixer_flow_collector
scrut_util --migrate config

  3. Once the migration is complete, reinstall the package for the destination server’s current Scrutinizer version and restart the flow collection service:

    PKG=$(rpm -qa plixer-scrutinizer_pg\*)
sudo yum reinstall -y $PKG
sudo systemctl restart scrutinizer

  4. Log in to the web interface and verify that all configuration data and user preferences have been successfully migrated.

  5. [Optional] Migrate historical data from the source server to the destination:

    1. Run the utility to migrate historical data:

      scrut_util --migrate data

    2. After the data migration is complete, return to the web interface and run one or more reports (time ranges must cover migrated data) to verify that the operation was successful.

  6. Remove the hba rule and temporary migrator role on the source server:

    sudo sed -i '/migrator/d' /var/db/big/pgsql/data/pg_hba.conf
psql -c "SELECT pg_reload_conf()"
psql -c "REASSIGN OWNED BY migrator TO plixer"
psql -c "DROP ROLE migrator"

Note

  • Configuration migrations only include Scrutinizer application data and not the system configuration, e.g., if a custom listening port was added via the Scrutinizer admin menu, that setting will be migrated, but the corresponding port will not be opened on the system. In such cases, undo and re-apply the setting in Scrutinizer after the migration to update the configuration.

  • The distributed cluster configuration on the destination Scrutinizer server are retained by default. These settings can also be migrated (while still preserving the destination server’s IP address and machine ID) by including the --force_dist option as follows:

    scrut_util --migrate config --force_dist

  • To extend the utility to also migrate contributions to the Scrutinizer SQL codebase, update /home/plixer/scrutinizer/database/utils/scrut_conf_dump.sh to include them.

Historical data migration#

If only historical data needs to be migrated, follow these steps after completing the setup steps for the source and destination servers:

Important

  • The collector service (plixer_flow_collector) should be running on the destination server before and during the migration. The data migration process may take some time, depending on the amount of flow data to be migrated.

  • If the source and destination servers are collecting data from the same exporters prior to the migration, reports can be run against the migrated data after the migration.

View instructions

  1. Start a new tmux session on the destination server:

    tmux new -s migration

  2. Run the migration utility:

    scrut_util --migrate data

  3. Confirm to start the migration when prompted.

  4. After the data migration is complete, log in to the web interface and run one or more reports (time ranges must cover migrated data) to verify that the historical data was successful.

  5. Remove the hba rule and temporary migrator role from the source server:

    sudo sed -i '/migrator/d' /var/db/big/pgsql/data/pg_hba.conf
psql -c "SELECT pg_reload_conf()"
psql -c "REASSIGN OWNED BY migrator TO plixer"
psql -c "DROP ROLE migrator"
Contents