A.1. Tungsten Clustering 8.0.0 GA (Not Yet Set)

Version End of Life. Not Yet Set

Release 8.0.0 contains a number of key bug fixes and improvements.

Deprecation Notices

The following features/tools have been deprecated from this release and will be removed in future releases. Any self written scripts that utilise these features should be updated.

  • Command-line Tools

    • tungsten_health_check and tungsten_provision_thl have been deprecated and will be removed in a future release.

      Issues: CT-2509

Behavior Changes

The following changes have been made to Tungsten Cluster and may affect existing scripts and integration tools. Any scripts or environment which make use of these tools should check and update for the new configuration:

  • Installation and Deployment

    • All Tungsten components now use log4j version 2

      Issues: CT-1410, CT-1412

    • Added a TPM check for MySQL 8+ to error out if the application user has CONNECTION_ADMIN privilege.

      Issues: CT-2325

    • The rest-api-admin-user and rest-api-admin-password TPM properties are now now required during a default installation or when explicitly setting the various manager-rest-api=true,connector-rest-api=true, replicator-rest-api=true or rest-apis=true properties.

      Issues: CT-2409

    • Upgraded the JGroup communication library to the v4.2.30

      Issues: CT-2435

  • Command-line Tools

    • Additional validation has been added to tprovision when parallel apply is in use.

      Issues: CT-2275

    • When provisioning a replica using mysqldump, the database will be set to offline mode until loading is complete to prevent accidental writes during the load.

      Issues: CT-2305

    • The tungsten_send_diag command has been updated with new security keys and the ability to accept security keys on the CLI.

      Issues: CT-2438

    • Removed sigtrap module dependency from tungsten_find_orphaned.

      Issues: CT-2451

    • The tpm command now allows hyphens in dataservice names.

      Issues: CT-2453

    • A new cli argument has been added (--apiver 8) to allow the tapi command to use APIv8. By default, tapi will continue to use APIv2 against v8 installations, using the --apiver 8 option is experimental and should be used with care. Full APIv8 functionality will be included in tapi in a future release.

      Issues: CT-2456

    • The tpm ask command now returns the appropriate values for keys: stagingdir, stagingpath, staginghost and staginguser.

      Additionally, the tpm ask staging command will return a JSON structure with all values. All values are the same as the output of the tpm query staging command, just presented differently for easier machine usage.

      Issues: CT-2510

    • The tpm cert info command now provides better error messages when a specified typeSpec file does not exist.

      Issues: CT-2530

  • Tungsten Connector

    • Connector now always reads its dynamic (auto-detect) configuration from primary node by default. Reading from a replica is still possible using property=readAutodetectConfigFromPrimary=false tpm flag.

      Issues: CT-2527

  • Core Clustering

    • Manager and Replicator now get higher CPU priority (-9 nice value) through systemd configuration file. When not using systemd, we advised to use the sudo renice <pid> command to avoid monitoring and processing pauses during high CPU usage periods. Note that this command has to be run as a privileged user, which is why the command cannot be part of the regular manager and replicator scripts

      Issues: CT-2330

    • Added protection against misconfigured applications or loadbalancers connecting repeatably to the Manager<>Connector port (11999). Such uncontrolled traffic would affect manager performance over time

      Issues: CT-2411

Improvements, new features and functionality

  • Installation and Deployment

    • Introduced a new set of tpm group aliases to control the rest api configuration when the same value needs aplying to multiple properties, as follows:

      • rest-apis will set connector-rest-api, manager-rest-api and replicator-rest-api

      • rest-api-address will set connector-rest-api-address, manager-rest-api-address and replicator-rest-api-address

      • rest-api-authentication will set connector-rest-api-authentication, manager-rest-api-authentication and replicator-rest-api-authentication

      • rest-api-ssl will set connector-rest-api-ssl, manager-rest-api-ssl and replicator-rest-api-ssl

      For example, a before and after comparisson of your /etc/tungsten/tungsten.ini may look like the following:

      BEFORE

      [defaults]
      ...
      connector-rest-api=true
      manager-rest-api=true
      replicator-rest-api=true
      connector-rest-api-adddress=0.0.0.0
      manager-rest-api-adddress=0.0.0.0
      replicator-rest-api-adddress=0.0.0.0
      connector-rest-api-authentication=true
      manager-rest-api-authentication=true
      replicator-rest-api-authentication=true
      connector-rest-api-ssl=true
      manager-rest-api-ssl=true
      replicator-rest-api-ssl=true
      ...

      AFTER

      [defaults]
      ...
      rest-apis=true
      rest-api-adddress=0.0.0.0
      rest-api-authentication=true
      rest-api-ssl=true
      ...

      Issues: CT-2459

  • Command-line Tools

    • The tpm post-process command has a new argument --statemap which processes property=statemap.* entries in the INI defaults section (only) to update the matching entries in the cluster-home/conf/statemap.properties.defaults file.

      Once done, tpm post-process --statemap will exit without invoking the old functionality.

      Issues: CT-1906

    • The tapi command is now able to provision a Connector database target (both MySQL Server and PostgreSQL) using a JSON payload or command-line arguments. For more information see Section 9.19, “The tapi Command”

      Issues: CT-2306

    • The new tpm ini command allows for easy creation, checking and administration of the /etc/tungsten/tungsten.ini configuration file.

      Issues: CT-2366

    • The tpm ask command now supports the -s|--service option for dsrole, dsstate, trrole and trstate keys. The option alias for --short is now -S (capitalized), was previously lower-case. Example: tpm ask dsrole --service east_from_west

      Issues: CT-2400

    • The tpm cert command will now handle public key files. The command tpm cert gen public will create $CONTINUENT_HOME/generated/tungsten_public.jks from the $CONTINUENT_HOME/share/tungsten_keystore.jks file and the tpm cert list public command will show the contents, etc.

      Issues: CT-2434

    • A new -z option has been added to tprovision which will enable compression when using the rsync method.

      Issues: CT-2489

    • The tpm diag command now gathers the output of cctrl system summary

      Issues: CT-2492

    • The tpm diag command now gathers the output of the tpm ini check -x command as the tpmINICheck call.

      Issues: CT-2499

    • Added missing help text for the router command within cctrl.

      Issues: CT-2500

    • The tpm ini command now handles all new tpm options.

      Issues: CT-2543

    • The tpm ask command will now convert values 0 and false to No and values 1 and true to Yes as human output when given --human or -n cli args.

      Issues: CT-2559

  • Core Replicator

    • Replicator now recognizes the new MySQL 8.4 show binary log status syntax (As well as show master status) and will run replacement configured request appropriately.

      Issues: CT-2291

    • A new applying field has been added to the output of trepctl status.

      ...
      applying : 3 / 0 (92.86%)
      ...

      In the above example, the applier is processing seqno 3 / fragno 0 and already applied 92.86% (this represents a percentage of rows and statements that were already applied).

      When parallel apply in enabled, there will be one applying field per channel configured, for example the following shows 2 channels:

      ...
      applying.0 : 
      applying.1 : 25 / 0 (92.86%)
      ...

      Issues: CT-2315

    • A new thl apply command is available which provides the ability to pull/push a range of transactions from source THL to a destination. The command can apply the THL either directly into the destination database (MySQL or PostgreSQL) or it can simply write the THL to the filesystem.

      The destination does not need to be running Tungsten Replicator services for this to be used and provides a way of applying data outside of normal Tungsten operations and can be used to extend backup/recovery operations in some scenarios.

      See Section 9.20.2, “thl apply Command” for more information.

      Issues: CT-2347

    • The trepctl perf output has been improved when parallel apply is enabled to show granular statistics per channel.

      Issues: CT-2517

  • Tungsten Connector

    • Listeners now support the protocol setting to switch between mysql and postgres dialects.

      Issues: CT-2324

    • Advanced Listeners can now be accessed from the Connector REST API v8.

      Issues: CT-2329

    • Fixed a bug where connections properly disconnecting with quit command would get an empty database change request.

      Issues: CT-2395

    • Connector now recognizes the new MySQL 8.4 show replica status syntax (As well as show slave status) and will run replacement configured request appropriately.

      Issues: CT-2401

  • Tungsten Manager

    • Upgrade of Drools libraries to v7.74.1.Final.

      Issues: CT-2206

  • Documentation

    • API documentation has been updated to reflect changes to many of the endpoints, including updated examples of expected inputs and outputs.

      Issues: CT-2339

  • API

    • Added /api/v8/manager/quorum REST API endpoint

      Issues: CT-2281

    • The api/v8/manager/control/service/<service>/datasource/<datasource>/shun call now checks if the datasource exists before shun. It returns an error if does not exist.

      Issues: CT-2447

Bug Fixes

  • Installation and Deployment

    • Hyphens are now allowed in service names by tpm.

      Issues: CT-2367

    • The tpm update command no longer displays the message "common::diffPasswordsDotStore: No custom source passwords.store file defined - SKIPPING"

      Issues: CT-2406

    • tpm now properly detects MariaDB default settings.

      Issues: CT-2424

    • tpm flags datasource-log-directory and repl-direct-datasource-log-directory will now use the value of each other as a default if only one is specified

      Issues: CT-2425

  • Command-line Tools

    • cctrl now validates input correctly for the set policy command

      Issues: CT-678

    • In certain instances, when provisioning from a primary using tprovision, the remote replication service would not get reset.

      Issues: CT-2138

    • tprovision now supports provisioning from a cluster to stand-alone replicator host

      Issues: CT-2199, CT-2483

    • New shotcut options are availabe with tprovision

      • --primary and --create-primary as an additional alias for -c

      • -C, --mysqlclone as aliases for -m mysqlclone

      • -D, --mysqldump as aliases for -m mysqldump

      • -R, --rsync as aliases for -m rsync

      • -X, --xtrabackup as aliases for -m xtrabackup

      Issues: CT-2311

    • The tpm diag command now properly gathers the uptime command.

      Issues: CT-2319

    • tprovision was incorrectly setting to the MySQL datadir to tungsten group ownership..

      Issues: CT-2322

    • When using the -m rsync option with tprovision, the rsync command now includes the oStrictHostKeyChecking=no options.

      Issues: CT-2348

    • The tpm report command now uses the correct API endpoint for Channel 3, Communications from the Connector to the Manager.

      Issues: CT-2354

    • The tpm diag command no longer displays a NOTICE for missing optional files.

      Issues: CT-2355

    • The tpm check ports and tpm check ports -c commands are now able to handle replicator-only nodes. By default, it will automatically try to discover the node type, and you may manually specify the node type with -r|--replicator or -c|--cluster.

      Issues: CT-2356

    • The tapi command now creates proper URL's for these standard endpoints when -C, -M or -R are specified along with one of the following strings: ping, tungstenVersion, createAdminUser, deleteAdminUser, listAdminUsers, rmiSecurity or user.

      Issues: CT-2391

    • The tpm diag command now gathers:

      • tungsten-replicator/log/replicator-api.log

      • tungsten-connector/log/connector-api.log

      • tungsten-manager/log/manager-api.log

      Issues: CT-2397

    • The tpm delete-service and tpm update commands now correctly remove Composite Active/Active cross-site subservices from cctrl

      Issues: CT-2404

    • The tungsten_get_status command now passes the correct number of arguments to the printf function in various places, preventing "Redundant argument" warnings.

      Issues: CT-2407

    • The tapi command now accepts non-standard ports on the command line, and also properly accepts a user/password pair on the CLI.

      Issues: CT-2420

    • The tpm update command now properly identifies and removes services with mixed-case service names.

      Issues: CT-2426

    • The tpm keep command is now properly called by tpm uninstall, and a regression in a call to the mysql cli has been fixed as well.

      Issues: CT-2462

    • executable-prefix is now applied to deployall and undeployall script names. For example, when executable-prefix=mm, deployall would be named mm_deployall.

      Issues: CT-2467

    • Fixed an issue where the database could potentially get locked in certain cases when using the rsync method with tprovision

      Issues: CT-2470

    • The cctrl ls command now correctly displays relative applied latency based on the use-relative-latency settings in the INI.

      Issues: CT-2472

    • tprovision will now correctly handle, and support, selinux if enabled.

      Issues: CT-2473

    • The tpm ask version and tpm ask versions commands now display values.

      Issues: CT-2475

    • During a tpm uninstall, when skipping data services not managed locally, tpm will now print a notice rather than a warning, since this is the expected behavior

      Issues: CT-2479

    • The cctrl system summary command now works correctly to display useful information regarding the host on which the command is executed.

      Issues: CT-2486

    • The tungsten_send_diag command now handles curl and AWS S3 errors separately and displays the proper message back.

      Issues: CT-2488

    • cctrl router invalid completion arguments have been removed.

      Issues: CT-2501

    • Fixes an issue where dsctl set with the -reset option could show warnings when run several times in a row, as shown:

      WARNING: Could not execute query DROP TABLE `tungsten_alpha`.`heartbeat` : Unknown table 'tungsten_alpha.heartbeat'

      Issues: CT-2502

    • The tpm cert aliases --running command now properly handles the tls_passwordstore and jmx_passwordstore typeSpecs.

      Issues: CT-2512

    • The tpm policy command now selects the correct service to act upon in Composite clusters.

      Issues: CT-2525

    • The tpm cert info command now properly handles all available typeSpecs.

      Issues: CT-2532

  • Core Replicator

    • Fixed an issue that would wrongly flag a SQL statement starting with a # comment as bidi_unsafe

      Issues: CT-2405

    • The replicator will now properly rollback an XA transaction in case of failure.

      Issues: CT-2410

    • Fixed an issue where a cluster could display a wrong latency after MySQL and the replicator were stopped and restarted. This would only show under no load conditions.

      Issues: CT-2485

  • Tungsten Connector

    • Fixed an exception when a user.map @default entry would not contain the expected username, password and dataservice parameters.

      Issues: CT-2368

    • Fixed an issue with PHP and caching_sha2_password when using proxy mode: the protocol part "fast path" was not handled properly by the connector, php would hang after reading an unexpected packet. Workaround for this issue (for earlier tungsten versions) is to set property=defaultAuthPlugin=mysql_native_password within /etc/tungsten/tungsten.ini.

      Issues: CT-2471

    • In active/active setups, no longer using global data service for show slave latency connector intercept.

      Issues: CT-2554

    • The tpm delete-service command now sets MAINTENANCE mode before operations, then sets back to AUTOMATIC if it was AUTOMATIC to start with.

      Issues: CT-2555

    • Closed a window where the Connector might let writes go to the old primary during a switch with networking issues.

      Issues: CT-2556

  • Tungsten Manager

    • Improved performance in manager client connection processing. The fix optimizes queue traversal time when searching for or deleting sessions, enhancing the handling of client (specifically manager or cctrl) connections.

      Issues: CT-2201

    • Under the conditions of a network outage between a primary site and a relay site, where a failover or switch operation was performed on the primary site, the relay replicator on the relay site was not being correctly updated at the time of the failover/switch.

      This issue has been resolved by detecting whether or not the relay replicator is correctly configured to replicate from the primary datasource on the primary site and, if not, reconfigures the replicator accordingly.

      Issues: CT-2225

    • There were cases where the state of a connector (router) was shown as UNKNOWN. The issue was resolved by either reporting the reason that the state is unknown i.e. a read timed out etc. or the connector will simply not be shown since it has been disconnected from all managers i.e. either is not running or not reachable due to a network issue.

      Issues: CT-2338

    • A datasource can now be removed from the manager configuration (including disk) when in MAINTENANCE mode.

      Note

      If the Replicator and Database are still online on the removed host, and the cluster returned to AUTOMATIC mode, the datasource will be recreated. Ensure at least one of the components are also stopped to prevent this.

      Issues: CT-2372

    • Fixed a manager state machine unhandled case when receiving a view update while joining the cluster.

      Issues: CT-2449

    • Fixed a race condition that aborted if an error happened when setting MySQL to read-only or read-write. Now the command retries until the MySQL is left in the desired state.

      Issues: CT-2452

    • The cctrl commands to set routers into the online and offline state were not functioning. This only affected manual execution of these commands from cctrl and did not affect other operations like switch, failover etc. This issue has been resolved.

      Issues: CT-2526

  • API

    • Fixed REST API ping method in a composite cluster.

      Issues: CT-2270

    • Fixed a bug when the /manager/cluster/status REST API call returned unknown for the replicator state.

      Issues: CT-2353

    • The manager REST API /api/v8/manager/cluster/status now correctly displays the connector active/total number of connections.

      Issues: CT-2423

    • Fixed an issue where a component (replicator/connector) stopped through through the API would wrongly indicate "unable to stop"

      Issues: CT-2515