Migration Utility

NAME

Plixer Scrutinizer DB Migration Utility

Note

As of version 18.7 You can find the migrator at this location: /home/plixer/scrutinizer/files/migrate.tar.gz

  1. Extract the files to use them:
cd /home/plixer/scrutinizer/files/
tar xvzf migrate.tar.gz
  1. Fill out your INI file before you begin:
vi migrate.ini

SYNOPSIS

migrate.exe [-d|-f|-m|-p|-r|-t] OPTIONS…

This program is used to migrate a Scrutinizer install between different hosts, OS, and DB platforms. Supported Database platforms are MySQL and PostgreSQL. Please READ and completely UNDERSTAND the documentation before using this util on any kind of production database. This utility requires one of the run mode flags.

Run Modes:

-d, -data         Migrate Scrutinizer historical data.  The system MUST be
                  collecting flows on the destination before doing this.
-f, -finish       Attempt to remove DB ROLEs listed in the migrate.ini and
                  outputs necessary SQL commands to do so if not.
-j, -juxtapose    Compare the Scrutinizer SCHEMA between two Databases.
-m, -meta         Migrate Scrutinizer meta data.  The plixer_flow_collector
                  service MUST be stopped on the destination.
-p, -push         Migrate a single, specific table.
                  ex: ./migrate.exe -p plixer.ip_groups
-r, -retreat      'Un-migrates' historical data at the destination server.
-t, -test         Tests connections and gathers some initial migration stats.

Options:

-c, -cross_ver,   Used for cross-version meta data migrations.  Stomps SCHEMA
-clobber_version  on the destination and re-CREATEs it using the source as
                  reference.  The command re-run MUST be performed on the destination's
                  Scrutinizer installer after the meta data migration.
-checksum         Performs a checksum per table for migrated historical data
                  and logs the results.
-dump             Performs a mysqldump or pg_dump of the Scrutinizer meta
                  data pre-migration to scrutinizer/files providing that either
                  mysqldump or pg_dump have been installed in the environment PATH.
-flip_ips         Will "flip" the source and destination ips in the
                  migrate.ini.  This is handy if re-IPing the destination
-h, -help         Display this help message.
-i, -inifile      Designate a path to a configuration file.
-l, -logfile      Designate the path to write a migration log to.
                  scrutinizer/files/logs/migrate.log by default.
-man, -info       Display the full documentation.
-q, -quiet        Suppress terminal progress messages.
-skip_clock       Skips source/destination clock test (as risky as it sounds)
-skip_lock        Skips table locking during migration (as risky as it sounds)

DESCRIPTION

The Plixer Scrutinizer DB Migration Utility is designed to ease the process of moving a Scrutinizer install between hosts and/or Database platforms. As with any kind of critical database work, a complete back-up should be made before attempting any kind of migration.

PRE-MIGRATION

The Migration Utility requires database ROLEs with SUPERUSER privileges that can connect remotely. Since this is an obvious security risk, temporary ROLEs should be created for the duration of the migration.

These Examples assume ‘tmp_user’ and ‘tmp_pass’ as the ROLE.

MySQL

SELECT PASSWORD('tmp_pass');
+-------------------------------------------+
| PASSWORD('tmp_pass')                      |
+-------------------------------------------+
| *00C233A85F04B58B5C9728B5BB411DC46F85BA54 |
+-------------------------------------------+
GRANT ALL ON *.* TO 'tmp_user'@'%' IDENTIFIED BY PASSWORD '*00C233A85F04B58B5C9728B5BB411DC46F85BA54';
FLUSH PRIVILEGES;

PostgreSQL

CREATE USER tmp_user WITH SUPERUSER ENCRYPTED PASSWORD 'tmp_pass';

The user will need to update /home/plixer/scrutinizer/pgsql/data/pg_hba.conf to allow ‘tmp_user’ to connect remotely. Add this line to pg_hba.conf:

host  all tmp_user  0.0.0.0/0 md5

Restart the plixer_db service

Warning

Be sure to remove these ROLEs once the migration is complete. The -finish run mode command will help with this.

CONFIGURATION

Most configuration parameters are set in the migration ini file, ‘migrate.ini’ by default. The Utility will look for this file in the present working directory if -inifile is not set.

../_images/Migrate-INI-Example.png

[source|destination database]

DB connection parameters for the migration hosts.

*_db_host

IP address of the migration host: Use the loopback address if running this util locally on one of the hosts.

*_db_user

The user needs to ensure that the db user named here has full GRANTs to the plixer SCHEMA as well as information_schema. The ‘root’ user is likely the best bet.

*_db_pass

Password for the Database User.

*_db_port

Database connection port. Default for MySQL is 3306, 5432 for PostgreSQL.

*_db_dialect

This is the Database connection driver. ‘mysql’ for a MySQL database. ‘pgsql’ for a PostgreSQL database.

*_db

The database to use upon connection. Leave this blank for MySQL. Default for PostgreSQL is ‘plixer’.

[exporters]

inclusions

A space separated list of exporters to include when performing a historical data migration with the -d or –data flag.

ex:
inclusions= 10.1.1.3 10.1.1.4

The migrate utility will attempt to migrate all exporters if it is left blank

exclusions

A space separated list of exporters to exclude when performing a historical data migration with the -d or –data flag.

ex:
exclusions= 6.6.6.6 7.7.7.7

The migrate utility will attempt to migrate all exporters if it is left blank

[data range]

intervals

A space separated list of flow history intervals to migrate.

Valid entries are:
1m 5m 30m 2h 12h 1d 1w

By default, 1 minute data is not migrated.

start/end_epoch

Limits the time frame of data migrations to a unix timestamp.

[scrutinizer data]

These are specific “categories” of Scrutinizer meta data a user may want to migrate from the source install to the destination install. This is done when calling the utility with the -m or –meta flag.

Any value for these parameters in the migrate.ini is considered true. To avoid migrating a particular category, leave the value blank.

alarms

Migrates Alarm Bulletin Boards, Alarm Policies, Notification Policies, and Flow Analytics Configuration.

dashboards

Migrates all Dashboards and Dashboard Gadgets.

ipgroups

Migrates custom Applications and IpGroups.

objects

Migrates Scrutinizer Maps and Device Tree Groups. If using any custom map background images, they will need to be uploaded manually.

reports

Migrates Saved Reports and any reports created in Report Designer.

systemprefs

Migrates Scrutinizer server preferences, scheduled tasks, Cross Check configuration, and any Third Party configuration.

users

Migrates Scrutinizer users, user groups, and user preferences.

EXAMPLES

Scrutinizer Cross Version Migration

These steps work for either MySQL or PostgreSQL. The source server MUST be version 16.3 or greater.

  1. Deploy and license the latest Scrutinizer VA for the destination.
  2. Following the PRE-MIGRATION section, create temporary database ROLEs.
  3. Get the migration utility binary and migrate.ini file onto the destination.
  4. Following the CONFIGURATION section, fill out the migrate.ini.
  5. Stop the plixer_flow_collector service on the destination.
  6. Test the connections. Only proceed if all green [OK] results are displayed.
migrate.exe -t
  1. Migrate the Scrutinizer meta data using the clobber_version flag.
migrate.exe -c -m
  1. Run the latest Scrutinizer installer on the destination.
  2. Verify the meta data migration was sucessful via the web interface.
  3. Verify collection has started again. The destination must be collecting before continuing.
  4. Migrate the Scrutinizer historical data within tmux.
tmux
migrate.exe -d --skip_version
  1. Once the migration is complete, be sure to remove the temporary ROLEs.
migrate.exe --finish