(advanced-migration)= # Data migration The `scrut_util --migrate` command can be used to migrate configuration and/or historical data from source Scrutinizer server to a destination server. :::{note} Because the steps outlined here can potentially result in an irrecoverable state for the source and/or destination Scrutinizer server, it is highly recommended to contact {ref}`Plixer Technical Support ` for assistance. In case of issues, the migration utility logs (`/var/log/migrate.log`) can be provided to help with troubleshooting. ::: ## Source and destination server support The migration utility includes separate runmodes for configuration data migration and historical data migration. The following table shows supported migration modes between Scrutinizer versions: ````{dropdown} View table ```{list-table} :header-rows: 1 :widths: auto * - - 19.5.x destination - 19.6.x destination - 19.7.x destination * - 18.20 source - Data - Data - Data * - 19.4.0 source - Configuration and data - Configuration and data - Configuration and data * - 19.5.x source - Configuration and data - Configuration and data - Configuration and data * - 19.6.x source - - Configuration and data - Configuration and data * - 19.7.0 source - - - Configuration (19.7.1+ only) and data ``` ```` ```{important} Configuration data **cannot** be migrated from a 19.7.0 source to a 19.7.0 destination. ``` ```{note} - If a Scrutinizer 18.20 server is {ref}`upgraded to v19.4.0 `, its configuration data can be migrated to servers running v19.5.0 and above. - The utility may report **pg_restore** errors after cross-server configuration migrations. These errors can safely be ignored, and any issues are addressed by re-running the installer as described in these {ref}`configuration migration instructions `. ``` (src-dst-prep)= ## Source and destination server setup Follow the steps below to prepare source and destination Scrutinizer servers for migration. ### Source server Run the following on the source server to create a temporary migration role and allow connections from the destination server using the role: ```{note} Replace `ROLE_PASSWORD` below with the desired password for the temporary migration user/role `migrator`. The password will also be required to prepare the destination server for the migration (see below). ``` ```cfg DESTINATION=DESTINATION_IP psql -c "CREATE USER migrator WITH SUPERUSER ENCRYPTED PASSWORD 'ROLE_PASSWORD'" sudo sed -i -e "1ihost plixer migrator $DESTINATION/32 md5" /var/db/big/pgsql/data/pg_hba.conf psql -c "SELECT pg_reload_conf()" ``` ````{important} For Scrutinizer 18.20 source servers, use the following commands instead: ```cfg DESTINATION=x.x.x.x psql -Uroot plixer -c "CREATE USER migrator WITH SUPERUSER ENCRYPTED PASSWORD 'ROLE_PASSWORD'" sudo sed -i -e "1ihost plixer migrator $DESTINATION/32 md5" /var/db/big/pgsql/data/pg_hba.conf psql -Uroot plixer -c "SELECT pg_reload_conf()" ``` ```` ### Destination server Before starting the migration, the destination server must be [deployed, licensed](scrutinizer-deployment), and already collecting/receiving data. ```{important} - If the source and destination servers are collecting data from the same exporters prior to the migration, reports can be run against the migrated data after the migration. - Configuration migrations will overwrite the destination server's configuration. - It is highly recommended to run any type of migration within a [tmux session](resource-terminal), as the operation may take some time. - The flow collection service should be running during data migrations. ``` After verifying that the destination is collecting data, update `/etc/migrate.ini` with the details of the source server as well as the migration role's username (`migrator`) and password (`ROLE_PASSWORD` entered when the role was created). (migrate_configuration)= ## Configuration migrations Configuration migrations transfer application configuration and user preference data from the source Scrutinizer server to the destination. After the source and destination servers have been set up, follow these steps to perform a configuration (and optional historical data) migration: `````{dropdown} View instructions 1. Start a new [tmux session](resource-terminal) on the destination server: ```cfg tmux new -s migration ``` 2. Transfer the source server's encryption key, pause collection services, and start the migration: ```cfg SOURCE=SOURCE_SERVER_IP scp $SOURCE:/etc/plixer.key /etc/plixer.key sudo systemctl stop plixer_flow_collector scrut_util --migrate config ``` 3. Once the migration is complete, reinstall the package for the destination server's current Scrutinizer version and restart the flow collection service: ```cfg PKG=$(rpm -qa plixer-scrutinizer_pg\*) sudo yum reinstall -y $PKG sudo systemctl restart scrutinizer ``` 4. Log in to the web interface and verify that all configuration data and user preferences have been successfully migrated. 5. [Optional] Migrate historical data from the source server to the destination: 1. Run the utility to migrate historical data: ```cfg scrut_util --migrate data ``` 2. After the data migration is complete, return to the web interface and run one or more reports (time ranges must cover migrated data) to verify that the operation was successful. 6. Remove the hba rule and temporary migrator role on the source server: ```cfg sudo sed -i '/migrator/d' /var/db/big/pgsql/data/pg_hba.conf psql -c "SELECT pg_reload_conf()" psql -c "REASSIGN OWNED BY migrator TO plixer" psql -c "DROP ROLE migrator" ``` ````` ````{note} - Configuration migrations only include Scrutinizer application data and not the system configuration, e.g., if a custom listening port was added via the Scrutinizer admin menu, that setting will be migrated, but the corresponding port will not be opened on the system. In such cases, undo and re-apply the setting in Scrutinizer after the migration to update the configuration. - The distributed cluster configuration on the destination Scrutinizer server are retained by default. These settings can also be migrated (while still preserving the destination server's IP address and machine ID) by including the `--force_dist` option as follows: ```cfg scrut_util --migrate config --force_dist ``` - To extend the utility to also migrate contributions to the Scrutinizer SQL codebase, update `/home/plixer/scrutinizer/database/utils/scrut_conf_dump.sh` to include them. ```` (migrate_history)= ## Historical data migration If only historical data needs to be migrated, follow these steps after completing the [setup steps for the source and destination servers](src-dst-prep): ```{important} - The collector service (`plixer_flow_collector`) should be running on the destination server before and during the migration. The data migration process may take some time, depending on the amount of flow data to be migrated. - If the source and destination servers are collecting data from the same exporters prior to the migration, reports can be run against the migrated data after the migration. ``` `````{dropdown} View instructions 1. Start a new [tmux session](resource-terminal) on the destination server: ```cfg tmux new -s migration ``` 2. Run the migration utility: ```cfg scrut_util --migrate data ``` 3. Confirm to start the migration when prompted. 4. After the data migration is complete, log in to the web interface and run one or more reports (time ranges must cover migrated data) to verify that the historical data was successful. 5. Remove the hba rule and temporary migrator role from the source server: ```cfg sudo sed -i '/migrator/d' /var/db/big/pgsql/data/pg_hba.conf psql -c "SELECT pg_reload_conf()" psql -c "REASSIGN OWNED BY migrator TO plixer" psql -c "DROP ROLE migrator" ``` `````