6.15.6. Upgrading to v8

Every effort has been made so that upgrading to v8 is a seamless and painless as possible. There are however a few caveats that should be taken into account depending on the starting version that you are upgrading from.

The following summarises the differences between v6, v7 and v8:

  • In both v7 and v8, Security is enabled by default. In v6, security is disabled by default.

  • In v8 it is mandatory to configure an admin user for the API. In v7 it was optional and in v6 the API didn't exist.

  • In v7.1.0, the jGroup libraries where upgraded, which means managers running versions prior to 7.1.0 will not communicate with managers running 7.1.0 or later (including 8+)

  • v8, like v7 enables full security by default, so the disable-security-controls flag will default to false when not specified.

    In addition this property also enables encrypted communication to the database. Setting this value to false, translates to the following settings being applied:

    file-protection-level=0027
    rmi-ssl=true
    thl-ssl=true
    rmi-authentication=true
    jgroups-ssl=true
    datasource-enable-ssl=true
    connector-ssl=true

If you are running v6 and wish to upgrade to v7, then you should follow the steps in the v7 upgrade documentation, see v7 specific documentation for more information.

If you are upgrading from v6 directly to v8, you should step through all of the sections of this upgrade document, taking note of specific steps and decisions to be made along the way with regards to security.

If you are upgrading from v7 and do not wish to make any changes to your configuration you should go straight to Section 6.15.6.6, “Steps to upgrade using tpm”

6.15.6.1. Upgrade Decisions

6.15.6.1.1. Keep existing level of security

This is the easiest and smoothest approach. tpm will process your configuration and do its best to maintain the same level of security. In order to achieve that, tpm will dynamically update your configuration with additional properties to adjust the level of security to match.

The properties that tpm will add to your configuration will be some or all of the following depending on the initial starting point of your configuration:

disable-security-controls
connector-rest-api-ssl
manager-rest-api-ssl
replicator-rest-api-ssl
datasource-enable-ssl
enable-connector-ssl

Before commencing with the upgrade, you should include the following properties to configure the API authentication:

rest-api-admin-user=apiuser
rest-api-admin-password=secret

You can now proceed with the upgrade, refer to Section 6.15.6.6, “Steps to upgrade using tpm” for the required steps

6.15.6.1.2. Apply new recommendations and setup security

The following security setting levels can be enabled, and will require user action prior to upgrading. These are:

  1. Internal Encryption and Authentication

  2. Tungsten to Database Encryption

  3. Application (Connector) to Database Encryption

  4. API SSL

  5. Configure API Authentication

Applying all of the above steps will bring full security, equivalent to the default.

The steps to enable will depend on what (if any) security is enabled in your existing installation. The following sections outline the steps required to be performed to enable security for each of the various layers. To understand whether you have configured any of the various layers of security, the following summary will help to understand your configuration:

No Security

If no security has been configured, the installation that you are starting from will have disable-security-controls=true (or it will not supplied at all if upgrading from v6) and no additional security properties will be supplied.

Partial Security

The installation that you are starting from will have partial security in place. This could be a combination of any of the following:

To upgrade and enable security, you should follow one or more of the following steps based on your requirements. At a minimum, the first step should always be included, the remaining steps are optional.

6.15.6.2. Setup internal encryption and authentication

Prior to running the upgrade, you need to manually create the keystore, to do this follow these steps on one host, and then copy the files to all other hosts in your topology:

db1> mkdir /etc/tungsten/secure
db1> keytool -genseckey -alias jgroups -validity 3650 -keyalg Blowfish -keysize 56 \
-keystore /etc/tungsten/secure/jgroups.jceks -storepass tungsten -keypass tungsten -storetype JCEKS

If this is the only level of security you plan on configuring you should now copy these new keystores to all other hosts in your topology.

db1> for host in db2 db3 db4 db5 db6; do 
ssh ${host} mkdir /etc/tungsten/secure
scp /etc/tungsten/secure/*.jceks ${host}:/etc/tungsten/secure
done

Enabling internal encryption and authentication will also enable API SSL by default.

If you need to enable encryption to the underlying database, now proceed to the next step Section 6.15.6.3, “Enable Tungsten to Database Encryption” before running the upgrade, otherwise you can then start the upgrade by following the steps in Section 6.15.6.6, “Steps to upgrade using tpm”.

The following additional configuration properties will need adding to your existing configuration.

disable-security-controls=false
rest-api-ssl=true
rest-api-admin-user=username
rest-api-admin-password=secret
java-jgroups-keystore-path=/etc/tungsten/secure/jgroups.jceks

6.15.6.3. Enable Tungsten to Database Encryption

The following prerequisite steps must be performed before continuing with this step

In this step, you pre-create the various keystores required and register the MySQL certificates for Tungsten. Execute all of the following steps on a single host, for example, db1. In the example below it is assumed that the mysql certificates reside in /etc/mysql/certs. If you use the example syntax below, you will also need to ensure the following directory exists: /etc/tungsten/secure

These commands will import the MySQL certificates into the required Tungsten truststores.

db1> keytool -importkeystore -srckeystore /etc/mysql/certs/client-cert.p12 -srcstoretype PKCS12 \
-destkeystore /etc/tungsten/secure/keystore.jks -deststorepass tungsten -srcstorepass tungsten

db1> keytool -import -alias mysql -file /etc/mysql/certs/ca.pem -keystore /etc/tungsten/secure/truststore.ts \
-storepass tungsten -noprompt

If you do not intend to configure SSL for your applications (via Connectors), or if your connectors reside on remote, dedicated hosts, you should now copy all of the generated keystores and truststores to all of the other hosts.

db1> for host in db2 db3 db4 db5 db6; do 
ssh ${host} mkdir /etc/tungsten/secure
scp /etc/tungsten/secure/*.jceks ${host}:/etc/tungsten/secure 
scp /etc/tungsten/secure/*.jks ${host}:/etc/tungsten/secure
scp /etc/tungsten/secure/*.ts ${host}:/etc/tungsten/secure
done

If you need to enable encryption to the underlying database from the connectors, now proceed to Section 6.15.6.4, “Enable Connector to Database Encryption” before running the upgrade, alternatively you can now follow the steps outlined in Section 6.15.6.6, “Steps to upgrade using tpm”

The following additional configuration properties will need adding to your existing configuration.

datasource-enable-ssl=true
java-truststore-path=/etc/tungsten/secure/truststore.ts
java-truststore-password=tungsten
java-keystore-path=/etc/tungsten/secure/keystore.jks
java-keystore-password=tungsten
datasource-mysql-ssl-cert=/etc/mysql/certs/client-cert.pem
datasource-mysql-ssl-key=/etc/mysql/certs/client-key.pem
datasource-mysql-ssl-ca=/etc/mysql/certs/ca.pem

6.15.6.4. Enable Connector to Database Encryption

The steps outlined in this section will need to be performed on all nodes where the Connector has been installed

If you are also enabling Internal Encryption, you would have followed the steps in Section 6.15.6.2, “Setup internal encryption and authentication” and you would have a number of files already in /etc/tungsten/secure. This next step will pre-create the keystore and truststore, and register the MySQL certificates for the Connectors. Execute all of the following steps on a single host, in this example, db1. In the example below it is assumed the mysql certificates reside in /etc/mysql/certs. If you use the example syntax below, you will need to ensure the following directory also exists: /etc/tungsten/secure

db1> keytool -importkeystore -srckeystore /etc/mysql/certs/client-cert.p12 \
-srcstoretype PKCS12 -destkeystore /etc/tungsten/secure/tungsten_connector_keystore.jks \
-deststorepass tungsten -srcstorepass tungsten

db1> keytool -import -alias mysql -file /etc/mysql/certs/ca.pem \
-keystore /etc/tungsten/secure/tungsten_connector_truststore.ts -storepass tungsten -noprompt

Now that all of the necessary steps have been taken to create the various keystores, you now need to copy all of these files to all other hosts in your topology.

db1> for host in db2 db3 db4 db5 db6; do 
ssh ${host} mkdir /etc/tungsten/secure
scp /etc/tungsten/secure/*.jceks ${host}:/etc/tungsten/secure
scp /etc/tungsten/secure/*.jks ${host}:/etc/tungsten/secure 
scp /etc/tungsten/secure/*.ts ${host}:/etc/tungsten/secure
done

Once the steps above have been performed, you can then continue with the upgrade, following the steps outlined in Section 6.15.6.6, “Steps to upgrade using tpm”

The following additional configuration properties will need adding to your existing configuration.

enable-connector-ssl=true
java-connector-keystore-path=/etc/tungsten/secure/tungsten_connector_keystore.jks
java-connector-keystore-password=tungsten
java-connector-truststore-path=/etc/tungsten/secure/tungsten_connector_truststore.ts
java-connector-truststore-password=tungsten

6.15.6.5. Enable MySQL SSL

A prerequisite to enabling full security, is to enable SSL within your database if this isn't already configured. To do this, we can use the mysql_ssl_rsa_setup tool supplied with most distributions of MySQL. If you do not have this tool, or require more detail, you can refer to Section 5.13.1, “Enabling Database SSL”. The steps below summarise the process using the mysql_ssl_rsa_setup

  1. The first step is to setup the directories for the certs, perform this on ALL hosts in your topology:

    shell> sudo mkdir -p /etc/mysql/certs
    shell> sudo chown -R tungsten: /etc/mysql/certs/

    NB: The ownership is temporarily set to tungsten so that the subsequent scp will work between hosts.

  2. This next step should be performed on just one single host, for the purpose of this example we will use db1 as the host:

    db1> mysql_ssl_rsa_setup -d /etc/mysql/certs/
    db1> openssl pkcs12 -export -inkey /etc/mysql/certs/client-key.pem \
    -name mysql -in /etc/mysql/certs/client-cert.pem -out /etc/mysql/certs/client-cert.p12 \
    -passout pass:tungsten

    Important

    When using OpenSSL 3.0 with Java 1.8, you MUST add the -legacy option to the openssl command.

    db1> for host in db2 db3 db4 db5 db6; do 
    scp /etc/mysql/certs/* ${host}:/etc/mysql/certs 
    done
  3. Next, on every host we need to reset the directory ownership

    shell> sudo chown -R mysql: /etc/mysql/certs/
    shell> sudo chmod g+r /etc/mysql/certs/client-*
  4. Now on every host, we need to reconfigure MySQL. Add the following properties into your my.cnf

    [mysqld]
    ssl-ca=/etc/mysql/certs/ca.pem
    ssl-cert=/etc/mysql/certs/server-cert.pem
    ssl-key=/etc/mysql/certs/server-key.pem
    
    [client]
    ssl-cert=/etc/mysql/certs/client-cert.pem
    ssl-key=/etc/mysql/certs/client-key.pem
    ssl-ca=/etc/mysql/certs/ca.pem
  5. Next, place your cluster(s) into MAINTENANCE mode

    shell> cctrl
    cctrl> set policy maintenance
  6. Restart MySQL for the new settings to take effect

    shell> sudo service mysqld restart
  7. Finally, return your cluster(s) into AUTOMATIC mode

    shell> cctrl
    cctrl> set policy automatic

6.15.6.6. Steps to upgrade using tpm

When you are ready to perform the upgrade, the following steps should be followed:

  1. Ensure you place your cluster(s) into MAINTENANCE mode

  2. Add the following options to the [default] stanza of your /etc/tungsten/tungsten.ini to configure API authentication. This step can be skipped if you are upgrading from v7 and these credentials where already defined.

    rest-api-admin-user=apiuser
    rest-api-admin-password=secret
  3. If no additional steps taken, and you wish to maintain the same level of security, skip skip the next step.

  4. Update your /etc/tungsten/tungsten.ini and include some, or all, of the options below depending on which steps you took earlier. All entries should be placed within the [defaults] stanza.

    disable-security-controls=false
    rest-api-ssl=true
    java-jgroups-keystore-path=/etc/tungsten/secure/jgroups.jceks

    If "Tungsten to Database Encryption" IS configured, also add:

    datasource-enable-ssl=true
    java-truststore-path=/etc/tungsten/secure/truststore.ts
    java-truststore-password=tungsten
    java-keystore-path=/etc/tungsten/secure/keystore.jks
    java-keystore-password=tungsten
    datasource-mysql-ssl-cert=/etc/mysql/certs/client-cert.pem
    datasource-mysql-ssl-key=/etc/mysql/certs/client-key.pem
    datasource-mysql-ssl-ca=/etc/mysql/certs/ca.pem

    If "Tungsten to Database Encryption" IS NOT configured, also add:

    datasource-enable-ssl=false

    If "Application (Connector) to Database Encryption" IS configured, also add:

    enable-connector-ssl=true
    java-connector-keystore-path=/etc/tungsten/secure/tungsten_connector_keystore.jks
    java-connector-keystore-password=tungsten
    java-connector-truststore-path=/etc/tungsten/secure/tungsten_connector_truststore.ts
    java-connector-truststore-password=tungsten

    If "Application (Connector) to Database Encryption" IS NOT configured, also add:

    enable-connector-ssl=false

    Important

    If start-and-report=true, remove this value or set to false

  5. Obtain the TAR or RPM package for your installation. If using a TAR file unpack this into your software staging tree on every host, typically /opt/continuent/software.

  6. Change into the directory for the software

    shell> cd /opt/continuent/software/tungsten-clustering-8.0.0-10
  7. Issue the following command on all hosts.

    shell> tools/tpm update --replace-release

    When upgrading the connectors, you could include the optional --no-connectors option if you wish to control the restart of the connectors manually

    Important

    If you are upgrading from any version PRIOR to 7.1.0 the managers on upgraded hosts will not function correctly until all hosts in the cluster are upgrade. This is due to an upgrade of the various JGroup libraries used for group communication.

    It is essential that you DO NOT issue a switch within the cluster until ALL hosts are upgraded to the same release.

  8. For Multi-Site/Active-Active topologies, you will also need to repeat the steps for the cross-site replicators

  9. Finally, before returning the cluster(s) to AUTOMATIC, you will need to sync the new certificates, created by the upgrade, to all hosts. This step will be required even if you have disabled security as these files will be used by the API and also, if you choose to enable it, THL Encryption.

    From one host, copy the certificate and keystore files to ALL other hosts in your topology. The following scp command is an example assuming you are issuing from db1, and the install directory is /opt/continuent:

    db1> for host in db2 db3 db4 db5 db6; do
    scp /opt/continuent/share/[jpt]* ${host}:/opt/continuent/share
    scp /opt/continuent/share/.[jpt]* ${host}:/opt/continuent/share
    done

    Note

    The examples assume you have the ability to scp between hosts as the tungsten OS user. If your security restrictions do not permit this, you will need to use alternative procedures appropriate to your environment to ensure these files are in sync across all hosts before continuing.

    If the files are not in sync between hosts, the software will fail to start!

  10. You will also need to repeat this if you have a Multi-Site/Active-Active topology for the cross-site replicators:

    db1> for host in db2 db3 db4 db5 db6; do
    scp /opt/replicator/share[jpt]* ${host}:/opt/replicator/share
    scp /opt/replicator/share.[jpt]* ${host}:/opt/replicator/share
    done
  11. Restart all tungsten components, one host at a time

    shell> manager restart
    shell> replicator restart
    shell> connector restart
  12. Return the cluster(s) to AUTOMATIC mode