Continuent Documentation

Manual Home
Tungsten Clustering (for MySQL) 7.0 Manual
Breadcrumbs
Chapters
Preface
1. Introduction
2. Deployment
3. Deployment: MySQL Topologies
4. Deployment: Advanced
5. Deployment: Security
6. Operations Guide
7. Tungsten Connector
8. Tungsten Manager
9. Command-line Tools
10. The tpm Deployment Command
11. Tungsten REST APIv2
12. Replication Filters
13. Performance, Tuning and Testing
A. Release Notes
B. Prerequisites
C. Troubleshooting
D. Files, Directories, and Environment
E. Terminology Reference
F. Internals
G. Frequently Asked Questions (FAQ)
Tungsten Clustering (for MySQL) 7.0 Manual
    Next

Tungsten Clustering (for MySQL) 7.0 Manual

Continuent Ltd

Copyright © 2025 Continuent Ltd

Abstract

This manual documents Tungsten Clustering (for MySQL), a high performance, High Availability and Disaster Recovery for MySQL clustering.

This manual includes information for 7.0, up to and including 7.0.3.

Build date: 2025-09-04 (35f1ba6e)

Up to date builds of this document: Tungsten Clustering (for MySQL) 7.0 Manual (Online), Tungsten Clustering (for MySQL) 7.0 Manual (PDF)

Table of Contents

Preface
1. Legal Notice
2. Conventions
3. Quickstart Guide
1. Introduction
1.1. Tungsten Replicator
1.1.1. Transaction History Log (THL)
1.2. Tungsten Manager
1.3. Tungsten Connector
2. Deployment
2.1. Host Types
2.1.1. Manager Hosts
2.1.2. Connector (Router) Hosts
2.1.3. Replicator Hosts
2.1.4. Active Witness Hosts
2.2. Deployment Sources
2.2.1. Using the TAR/GZipped files
2.2.2. Using the RPM package files
2.3. Common tpm Options During Deployment
2.4. Best Practices
2.4.1. Best Practices: Deployment
2.4.2. Best Practices: Upgrade
2.4.3. Best Practices: Operations
2.4.4. Best Practices: Maintenance
3. Deployment: MySQL Topologies
3.1. Deploying Standalone HA Clusters
3.1.1. Prepare: Standalone HA Cluster
3.1.2. Install: Standalone HA Cluster
3.2. Deploying Composite Active/Passive Clustering
3.2.1. Prepare: Composite Active/Passive Cluster
3.2.2. Install: Composite Active/Passive Cluster
3.2.3. Adding a remote Composite Cluster
3.3. Deploying Composite Active/Active Clusters
3.3.1. Prepare: Composite Active/Active Clusters
3.3.2. Install: Composite Active/Active Clusters
3.3.3. Best Practices: Composite Active/Active Clusters
3.3.4. Resetting a single dataservice
3.3.5. Resetting all dataservices
3.3.6. Dataserver maintenance
3.3.7. Adding a Cluster to a Composite Active/Active Topology
3.4. Deploying Composite Dynamic Active/Active
3.4.1. Enabling Composite Dynamic Active/Active
3.5. Deploying Multi-Site/Active-Active Clustering
3.5.1. Prepare: Multi-Site/Active-Active Clusters
3.5.2. Install: Multi-Site/Active-Active Clusters
3.5.3. Best Practices: Multi-Site/Active-Active Clusters
3.5.4. Resetting a single dataservice
3.5.5. Resetting all dataservices
3.5.6. Adding a new Cluster/Dataservice
3.5.7. Enabling SSL for Replicators Only
3.5.8. Dataserver maintenance
3.6. Deploying Tungsten Connector Only
3.7. Deploying Additional Datasources, Managers, or Connectors
3.7.1. Adding Datasources to an Existing Deployment
3.7.2. Adding Active Witnesses to an Existing Deployment
3.7.3. Replacing an Active Witness as a Full Cluster Node
3.7.4. Replacing a Full Cluster Node as an Active Witness
3.7.5. Adding Connectors to an Existing Deployment
3.7.6. Converting from a single cluster to a composite cluster
3.8. Replicating Data Into an Existing Dataservice
3.9. Replicating Data Out of a Cluster
3.9.1. Prepare: Replicating Data Out of a Cluster
3.9.2. Deploy: Replicating Data Out of a Cluster
3.10. Migrating and Seeding Data
3.10.1. Migrating from MySQL Native Replication 'In-Place'
3.10.2. Migrating from MySQL Native Replication Using a New Service
3.10.3. Seeding Data through MySQL
4. Deployment: Advanced
4.1. Deploying Parallel Replication
4.1.1. Application Prerequisites for Parallel Replication
4.1.2. Enabling Parallel Apply During Install
4.1.3. Channels
4.1.4. Parallel Replication and Offline Operation
4.1.5. Adjusting Parallel Replication After Installation
4.1.6. Monitoring Parallel Replication
4.1.7. Controlling Assignment of Shards to Channels
4.1.8. Disk vs. Memory Parallel Queues
4.2. Starting and Stopping Tungsten Components
4.2.1. Stopping/Starting Tungsten Replicator
4.2.2. Stopping/Starting Tungsten Connectors
4.2.3. Stopping/Starting Tungsten Managers
4.2.4. Restarting the Multi-Site/Active-Active Replicator Service
4.3. Configuring Startup on Boot
4.3.1. Configuring Multi-Site/Active-Active Replicator Startup on Boot
4.4. Removing Datasources, Managers or Connectors
4.4.1. Removing a Datasource from an Existing Deployment
4.4.2. Removing a Composite Datasource/Cluster from an Existing Deployment
4.4.3. Removing a Connector from an Existing Deployment
5. Deployment: Security
5.1. Enabling Security
5.1.1. Enabling Security using the Staging Method
5.1.2. Enabling Security using the INI Method
5.2. Disabling Security
5.3. Creating Suitable Certificates
5.3.1. Creating Tungsten Internal Certificates Manually
5.4. Installing from a Staging Host with Custom Certificates
5.4.1. Installing from a Staging Host with Manually-Generated Certificates
5.5. Installing via INI File with Custom Certificates
5.5.1. Installing via INI File with Manually-Generated Certificates
5.6. Installing via INI File with CA-Signed Certificates
5.7. Replacing the JGroups Certificate from a Staging Directory
5.8. Replacing the TLS Certificate from a Staging Directory
5.9. Removing JGroups Encryption from a Staging Directory
5.10. Removing JGroups Encryption via INI File
5.11. Removing TLS Encryption from a Staging Directory
5.12. Removing TLS Encryption via INI File
5.13. Enabling Tungsten<>Database Security
5.13.1. Enabling Database SSL
5.13.2. Configure Tungsten<>Database Secure Communication
5.13.3. Configuring Connector SSL
6. Operations Guide
6.1. The Home Directory
6.2. Establishing the Shell Environment
6.3. Checking Dataservice Status
6.3.1. Latency or Relative Latency Display
6.3.2. Getting Detailed Information
6.3.3. Understanding Datasource Roles
6.3.4. Understanding Datasource States
6.3.5. Understanding Replicator Roles
6.3.6. Checking Replication Status
6.3.7. Changing Datasource States
6.3.8. Datasource Statuses
6.3.9. Datasource States and Policy Mode Interactions
6.4. Policy Modes
6.4.1. Setting Policy Modes
6.4.2. Setting Policy Modes in a Composite cluster
6.5. Switching Primary Hosts
6.5.1. Automatic Primary Failover
6.5.2. Manual Primary Switch
6.6. Datasource Recovery Steps
6.6.1. Recover a failed Replica
6.6.2. Recover a failed Primary
6.6.3. Handling Multiple Host Failures
6.7. Composite Cluster Switching, Failover and Recovery
6.7.1. Composite Cluster Site Switch
6.7.2. Composite Cluster Site Failover (Forced Switch)
6.7.3. Composite Cluster Site Recovery
6.7.4. Composite Cluster Relay Recovery
6.8. Composite Active/Active Recovery
6.9. Managing Transaction Failures
6.9.1. Identifying a Transaction Mismatch
6.9.2. Skipping Transactions
6.10. Creating a Backup
6.10.1. Using a Different Backup Tool
6.10.2. Automating Backups
6.10.3. Using a Different Directory Location
6.10.4. Creating an External Backup
6.11. Restoring a Backup
6.11.1. Restoring a Specific Backup
6.11.2. Restoring an External Backup
6.11.3. Restoring from Another Replica
6.11.4. Manually Recovering from Another Replica
6.11.5. Reprovision a MySQL Replica using rsync
6.11.6. Rebuilding a Lost Datasource
6.11.7. Resetting an Entire Dataservice from Filesystem Snapshots
6.12. Migrating and Seeding Data
6.12.1. Migrating from MySQL Native Replication 'In-Place'
6.12.2. Migrating from MySQL Native Replication Using a New Service
6.13. Resetting a Tungsten Cluster Dataservice
6.13.1. Reset a Single Site in a Multi-Site/Active-Active Topology
6.13.2. Reset All Sites in a Multi-Site/Active-Active topology
6.14. Replicator Fencing
6.14.1. Fencing a Replica Node Due to a Replication Fault
6.14.2. Fencing Primary Replicators
6.15. Upgrading Tungsten Cluster
6.15.1. Upgrading using the Staging Method
6.15.2. Upgrading when using INI-based configuration
6.15.3. Upgrade/Convert: From Multi-Site/Active-Active (MSAA) to Composite Active/Passive (CAP)
6.15.4. Upgrade/Convert: From Multi-Site/Active-Active (MSAA) to Composite Active/Active (CAA)
6.15.5. Installing an Upgraded JAR Patch
6.15.6. Upgrading to v7
6.16. Performing Database or OS Maintenance
6.16.1. Performing Maintenance on a Single Replica
6.16.2. Performing Maintenance on a Primary
6.16.3. Performing Maintenance on an Entire Dataservice
6.16.4. Upgrading MySQL
6.16.5. Upgrading or Updating your JVM
6.16.6. Making Schema Changes
6.16.7. Updating Database User Passwords
6.17. Monitoring Tungsten Cluster
6.17.1. Managing Log Files with logrotate
6.17.2. Monitoring Status Using cacti
6.17.3. Monitoring Status Using nagios
6.17.4. Monitoring Status Using Prometheus Exporters
6.18. Managing THL Files
6.18.1. Moving the THL File Location
6.18.2. Changing the THL Retention Times
6.18.3. Rebuilding THL on the Primary
6.18.4. THL Encryption and Compression
7. Tungsten Connector
7.1. Connector/Application Basics
7.1.1. Connector Control Flow
7.2. Basic Connector Configuration
7.3. Clients and Deployment
7.3.1. Connection Pools
7.4. Routing Methods
7.4.1. Connector Routing Methods
7.4.2. Primary/Replica Selection
7.4.3. Connector Load Balancers
7.4.4. Specifying Required Latency
7.4.5. Setting Read Affinity and Direct Reads
7.4.6. Setting Read/Write Affinity in Composite Active/Active Environments
7.4.7. Setting Negative Affinity
7.4.8. Routing using embedded syntax in connect string
7.4.9. Connector Datasource Selection in Composite Clusters
7.4.10. Smartscale Routing
7.4.11. Direct Routing
7.4.12. SQL Routing
7.4.13. Host-based Routing
7.4.14. Port-based Routing
7.4.15. Read-only Routing
7.5. Using Bridge Mode
7.5.1. Enabling Bridge Mode
7.6. User Authentication
7.6.1. user.map File Format
7.6.2. Dual-Paswords and MySQL 8
7.6.3. user.map Direct Routing
7.6.4. user.map Host Options
7.6.5. user.map Updates
7.6.6. Generating user.map Entries from a Script
7.6.7. Encrypting user.map Data
7.6.8. Synchronizing user.map Data
7.6.9. user.map Limitations
7.6.10. Host-based Authentication
7.7. Testing Connectivity Via the Connector
7.7.1. Testing Connectivity in Bridge Mode
7.7.2. Testing Connectivity in Proxy Mode with No R/W Splitting Enabled
7.7.3. Testing Connectivity in Proxy Mode with R/W Splitting Enabled (SmartScale or @direct)
7.8. Connector Operational States
7.8.1. Connections During Automatic Failure/Failover
7.8.2. Connections During Manual Switch
7.8.3. Connections During Connection Failures
7.8.4. Other Errors
7.8.5. Connector Keepalive
7.8.6. Connector Change User as Ping
7.9. Connector/Manager Interface
7.10. Connector Command-Line Interface
7.11. Connector Inline Command Interface
7.11.1. Connector tungsten cluster status Command
7.11.2. Connector tungsten connection count Command
7.11.3. Connector tungsten connection status Command
7.11.4. Connector tungsten flush privileges Command
7.11.5. Connector tungsten gc Command
7.11.6. Connector tungsten help Command
7.11.7. Connector tungsten mem info Command
7.11.8. Connector tungsten show [full] processlist Command
7.11.9. Connector show slave status Command
7.11.10. Connector tungsten show variables Command
7.12. Advanced Configuration
7.12.1. Working with Proxy Protocol v1
7.12.2. Using Multiple Dataservices
7.12.3. Advanced Listeners
7.12.4. Connector Automatic Reconnect
7.12.5. Using the Connector with HAProxy
7.12.6. Using Fall-Back Bridge Mode
7.12.7. Using the Max Connections Feature
7.12.8. Adjusting the Client Disconnect Delay During Manual Switch
7.12.9. Adjusting the Bridge Mode Forced Client Disconnect Timeout
7.12.10. Adjusting the Connnector Response to Resource Losses
7.12.11. Connector Logging Configuration
7.12.12. Connector Audit Logging
7.12.13. Connector SSL Advertisement Configuration
7.12.14. Connector IP Address Configuration
7.12.15. Deploying a Connector through Docker
7.13. Connector General Limitations
8. Tungsten Manager
8.1. Tungsten Manager Introduction
8.1.1. How Does Failover Work?
8.1.2. Best Practices for Proper Cluster Failovers
8.1.3. Command-Line Monitoring Tools
8.2. Tungsten Manager Failover Behavior
8.2.1. Failover Replication State Scenarios
8.2.2. Recovery Behavior After Failover
8.2.3. Failover Response when MySQL Server Fails
8.2.4. Failover Response when Replica Applier is Latent
8.2.5. Understanding the FAILSAFE-SHUN state.
8.3. Tungsten Manager Failover Tuning
8.4. Tungsten Manager Failover Internals
8.4.1. Manual Switch Versus Automatic Failover
8.4.2. Switch and Failover Steps for Local Clusters
8.4.3. Switch and Failover Steps for Composite Services
8.5. Tungsten Manager Fault Detection, Fencing and Recovery
8.5.1. Tungsten Manager Definitions
8.5.2. Cluster Monitoring and Notification Events
8.5.3. Rule Organization - Detection, Investigation, Fencing, Recovery
8.6. Cluster State Savepoints
9. Command-line Tools
9.1. The cctrl Command
9.1.1. cctrl Command-line Options
9.1.2. cctrl Modes
9.1.3. cctrl Commands
9.2. The check_tungsten_latency Command
9.3. The check_tungsten_online Command
9.4. The check_tungsten_policy Command
9.5. The check_tungsten_progress Command
9.6. The check_tungsten_services Command
9.7. The clean_release_directory Command
9.8. The cluster_backup Command
9.9. The connector Command
9.10. The deployall Command
9.11. The dsctl Command
9.11.1. dsctl get Command
9.11.2. dsctl set Command
9.11.3. dsctl reset Command
9.11.4. dsctl help Command
9.12. env.sh Script
9.13. The manager Command
9.14. The multi_trepctl Command
9.14.1. multi_trepctl Options
9.14.2. multi_trepctl Commands
9.15. The query Command
9.16. The replicator Command
9.17. The startall Command
9.18. The stopall Command
9.19. The tapi Command
9.20. The thl Command
9.20.1. thl Position Commands
9.20.2. thl dsctl Command
9.20.3. thl index Command
9.20.4. thl info Command
9.20.5. thl list Command
9.20.6. thl purge Command
9.20.7. thl tail Command
9.20.8. thl help Command
9.21. The tmonitor Command
9.22. The tpasswd Command
9.23. The tprovision Script
9.24. The trepctl Command
9.24.1. trepctl Options
9.24.2. trepctl Global Commands
9.24.3. trepctl Service Commands
9.25. The tungsten_find_orphaned Command
9.26. The tungsten_find_position Command
9.27. The tungsten_find_seqno Command
9.28. The tungsten_get_mysql_datadir Script
9.29. The tungsten_get_status Script
9.30. The tungsten_get_ports Script
9.31. The tungsten_merge_logs Script
9.32. The tungsten_monitor Script
9.33. The tungsten_newrelic_event Command
9.34. The tungsten_post_process Command
9.35. The tungsten_prep_upgrade Script
9.36. The tungsten_purge_thl Command
9.37. The tungsten_reset_manager Command
9.38. The tungsten_send_diag Script
9.38.1. Updating tungsten_send_diag
9.39. The tungsten_show_processlist Script
9.40. The tungsten_skip_all Command
9.41. The tungsten_skip_seqno Script
9.42. The undeployall Command
9.43. The zabbix_tungsten_latency Command
9.44. The zabbix_tungsten_online Command
9.45. The zabbix_tungsten_progress Command
9.46. The zabbix_tungsten_services Command
10. The tpm Deployment Command
10.1. Processing Installs and Upgrades
10.2. tpm INI File Configuration
10.2.1. Creating an INI file
10.2.2. Installation with INI File
10.2.3. Upgrades with an INI File
10.2.4. Configuration Changes with an INI file
10.3. Converting from Staging to INI
10.3.1. Using the translatetoini.pl Script
10.4. tpm Staging Configuration
10.4.1. Configuring default options for all services
10.4.2. Configuring a single service
10.4.3. Configuring a single host
10.4.4. Reviewing the current configuration
10.4.5. Installation
10.4.6. Upgrades from a Staging Directory
10.4.7. Configuration Changes from a Staging Directory
10.5. tpm Commands
10.5.1. tpm ask Command
10.5.2. tpm check Command
10.5.3. tpm configure Command
10.5.4. tpm connector Command
10.5.5. tpm copy Command
10.5.6. tpm delete-service Command
10.5.7. tpm diag Command
10.5.8. tpm fetch Command
10.5.9. tpm firewall Command
10.5.10. tpm find-seqno Command
10.5.11. tpm generate-haproxy-for-api Command
10.5.12. tpm help Command
10.5.13. tpm install Command
10.5.14. tpm mysql Command
10.5.15. tpm policy Command
10.5.16. tpm post-process Command
10.5.17. tpm promote-connector Command
10.5.18. tpm purge-thl Command
10.5.19. tpm query Command
10.5.20. tpm report Command
10.5.21. tpm reverse Command
10.5.22. tpm uninstall Command
10.5.23. tpm update Command
10.5.24. tpm validate Command
10.5.25. tpm validate-update Command
10.6. tpm Common Options
10.7. tpm Validation Checks
10.8. tpm Configuration Options
10.8.1. A tpm Options
10.8.2. B tpm Options
10.8.3. C tpm Options
10.8.4. D tpm Options
10.8.5. E tpm Options
10.8.6. F tpm Options
10.8.7. H tpm Options
10.8.8. I tpm Options
10.8.9. J tpm Options
10.8.10. L tpm Options
10.8.11. M tpm Options
10.8.12. N tpm Options
10.8.13. O tpm Options
10.8.14. P tpm Options
10.8.15. R tpm Options
10.8.16. S tpm Options
10.8.17. T tpm Options
10.8.18. U tpm Options
10.8.19. W tpm Options
11. Tungsten REST APIv2
11.1. Getting Started with Tungsten REST API
11.1.1. Configuring the API
11.1.2. How to Access the API
11.1.3. Data Structures
11.1.4. Summary of all API TPM options
11.2. Proxy (Connector) API Specifics
11.2.1. Proxy States
11.2.2. List and Set Proxy Configuration
11.2.3. User Map
11.2.4. Cluster Configuration Manipulation
11.3. Manager API Specifics
11.3.1. Manager Status
11.3.2. Cluster Topology
11.3.3. Service Status
11.3.4. Datasource Status
11.3.5. Cluster Control Commands
11.4. Replicator API Specifics
11.4.1. Replicator Endpoints
11.4.2. Service Endpoints
11.4.3. Service THL Endpoints
12. Replication Filters
12.1. Enabling/Disabling Filters
12.2. Enabling Additional Filters
12.3. Filter Reference
12.3.1. ansiquotes.js Filter
12.3.2. BidiRemoteSlave (BidiSlave) Filter
12.3.3. breadcrumbs.js Filter
12.3.4. CaseTransform Filter
12.3.5. ColumnName Filter
12.3.6. ConvertStringFromMySQL Filter
12.3.7. DatabaseTransform (dbtransform) Filter
12.3.8. dbrename.js Filter
12.3.9. dbselector.js Filter
12.3.10. dbupper.js Filter
12.3.11. dropcolumn.js Filter
12.3.12. dropcomments.js Filter
12.3.13. dropddl.js Filter
12.3.14. dropmetadata.js Filter
12.3.15. droprow.js Filter
12.3.16. dropstatementdata.js Filter
12.3.17. dropsqlmode.js Filter
12.3.18. dropxa.js Filter
12.3.19. Dummy Filter
12.3.20. EnumToString Filter
12.3.21. EventMetadata Filter
12.3.22. foreignkeychecks.js Filter
12.3.23. Heartbeat Filter
12.3.24. insertsonly.js Filter
12.3.25. Logging Filter
12.3.26. MySQLSessionSupport (mysqlsessions) Filter
12.3.27. mapcharset Filter
12.3.28. NetworkClient Filter
12.3.29. nocreatedbifnotexists.js Filter
12.3.30. OptimizeUpdates Filter
12.3.31. PrimaryKey Filter
12.3.32. PrintEvent Filter
12.3.33. Rename Filter
12.3.34. Replicate Filter
12.3.35. ReplicateColumns Filter
12.3.36. Row Add Database Name Filter
12.3.37. Row Add Transaction Info Filter
12.3.38. SetToString Filter
12.3.39. Shard Filter
12.3.40. shardbyseqno.js Filter
12.3.41. shardbytable.js Filter
12.3.42. SkipEventByType Filter
12.3.43. TimeDelay (delay) Filter
12.3.44. TimeDelayMsFilter (delayInMS) Filter
12.3.45. tosingledb.js Filter
12.3.46. truncatetext.js Filter
12.3.47. zerodate2null.js Filter
12.4. Standard JSON Filter Configuration
12.4.1. Rule Handling and Processing
12.4.2. Schema, Table, and Column Selection
12.5. JavaScript Filters
12.5.1. Writing JavaScript Filters
12.5.2. Installing Custom JavaScript Filters
13. Performance, Tuning and Testing
13.1. Block Commit
13.1.1. Monitoring Block Commit Status
13.2. Improving Network Performance
13.3. Tungsten Replicator Block Commit and Memory Usage
13.4. Connector Memory Management
13.5. Functional Testing
13.5.1. Manual and Automatic Failover
13.5.2. Backup and Restore
13.5.3. Connectivity
13.5.4. Performance and Other Tests
A. Release Notes
A.1. Tungsten Clustering 7.0.3 GA (4 Apr 2023)
A.2. Tungsten Clustering 7.0.2 GA (9 Dec 2022)
A.3. Tungsten Clustering 7.0.1 GA (13 June 2022)
A.4. Tungsten Clustering 7.0.0 GA (29 Mar 2022)
B. Prerequisites
B.1. Requirements
B.1.1. Operating Systems Support
B.1.2. Database Support
B.1.3. RAM Requirements
B.1.4. Disk Requirements
B.1.5. Java Requirements
B.1.6. Cloud Deployment Requirements
B.1.7. Docker Support Policy
B.2. Staging Host Configuration
B.3. Host Configuration
B.3.1. Operating System Version Support
B.3.2. Creating the User Environment
B.3.3. Configuring Network and SSH Environment
B.3.4. Directory Locations and Configuration
B.3.5. Configure Software
B.3.6. sudo Configuration
B.3.7. SELinux Configuration
B.4. MySQL Database Setup
B.4.1. MySQL Version Support
B.4.2. MySQL Configuration
B.4.3. MySQL Configuration for Active/Active Deployments
B.4.4. MySQL User Configuration
B.4.5. MySQL Unprivileged Users
B.5. Prerequisite Checklist
C. Troubleshooting
C.1. Contacting Support
C.1.1. Support Request Procedure
C.1.2. Creating a Support Account
C.1.3. Open a Support Ticket
C.1.4. Open a Support Ticket via Email
C.1.5. Getting Updates for all Company Support Tickets
C.1.6. Support Severity Level Definitions
C.2. Support Tools
C.2.1. Generating Diagnostic Information
C.2.2. Generating Advanced Diagnostic Information
C.2.3. Using tungsten_upgrade_manager
C.3. Error/Cause/Solution
C.3.1. MySQL is incorrectly configured
C.3.2. WARNING: An illegal reflective access operation has occurred
C.3.3. element 'mysql_readonly' not found in path
C.3.4. Latency is high: master:ONLINE, progress=41331580333, THL latency=78849.733
C.3.5. Backup/Restore is not bringing my host back to normal
C.3.6. Attempt to write new log record with equal or lower fragno: seqno=3 previous stored fragno=32767 attempted new fragno=-32768
C.3.7. Triggers not firing correctly on Replica
C.3.8. ERROR backup.BackupTask Backup operation failed: null
C.3.9. cctrl reports MANAGER(state=STOPPED)
C.3.10. Starting replication after performing a restore because of an invalid restart sequence number
C.3.11. WARN [KeepAliveTimerTask] - Error while sending a KEEP_ALIVE query to connection.
C.3.12. [S1000][unixODBC][MySQL][ODBC 5.3(w) Driver]SSL connection error: unknown error number [ISQL]ERROR: Could not SQLConnect
C.3.13. MySQL 8.0+, User Roles and Smartscale
C.3.14. Connector shows errors with "java.net.SocketException: Broken pipe"
C.3.15. Services requires a reset
C.3.16. trepctl status hangs
C.3.17. Replicator reports an Out of Memory error
C.3.18. Backup agent name not found: xtrabackup-full
C.3.19. ERROR 1010 (HY000) at line 5094506: Error dropping database (can't rmdir './mysql-bin/', errno: 17)
C.3.20. ERROR >> host1 >> can't alloc thread
C.3.21. Too many open processes or files
C.3.22. Replicator fails to connect after updating password
C.3.23. Cluster remains in MAINTENANCE mode after tpm update
C.3.24. Replication latency very high
C.3.25. ERROR 2013 (HY000) at line 583: Lost connection to MySQL server during query
C.3.26. pendingExceptionMessage": "Unable to update last commit seqno: Incorrect datetime value: '2025-03-13 02:02:26' for column 'update_timestamp' at row 1
C.3.27. Lots of entries added to replicator log
C.3.28. Error: could not settle on encryption_client algorithm
C.3.29. Unable to update the configuration of an installed directory
C.3.30. ERROR 1580 (HY000) at line 5093787: You cannot 'DROP' a log table if logging is enabled
C.3.31. The Primary replicator stopped with a JDBC error.
C.3.32. Replicator runs out of memory
C.4. Known Issues
C.4.1. Triggers
C.5. Troubleshooting Timeouts
C.6. Troubleshooting Backups
C.7. Running Out of Diskspace
C.8. Troubleshooting SSH and tpm
C.9. Troubleshooting Data Differences
C.9.1. Identify Structural Differences
C.9.2. Identify Data Differences
C.10. Comparing Table Data
C.11. Troubleshooting Memory Usage
D. Files, Directories, and Environment
D.1. The Tungsten Install Directory
D.1.1. The backups Directory
D.1.2. The releases Directory
D.1.3. The service_logs Directory
D.1.4. The share Directory
D.1.5. The thl Directory
D.1.6. The tungsten Directory
D.2. Log Files
D.3. Environment Variables
E. Terminology Reference
E.1. Transaction History Log (THL)
E.1.1. THL Format
E.2. Generated Field Reference
E.2.1. Terminology: Fields masterConnectUri
E.2.2. Terminology: Fields masterListenUri
E.2.3. Terminology: Fields accessFailures
E.2.4. Terminology: Fields active
E.2.5. Terminology: Fields activeSeqno
E.2.6. Terminology: Fields appliedLastEventId
E.2.7. Terminology: Fields appliedLastGoodLatency
E.2.8. Terminology: Fields appliedLastGoodSeqno
E.2.9. Terminology: Fields appliedLastSeqno
E.2.10. Terminology: Fields appliedLatency
E.2.11. Terminology: Fields applier.class
E.2.12. Terminology: Fields applier.name
E.2.13. Terminology: Fields applyTime
E.2.14. Terminology: Fields autoRecoveryEnabled
E.2.15. Terminology: Fields autoRecoveryTotal
E.2.16. Terminology: Fields averageBlockSize
E.2.17. Terminology: Fields blockCommitRowCount
E.2.18. Terminology: Fields cancelled
E.2.19. Terminology: Fields channel
E.2.20. Terminology: Fields channels
E.2.21. Terminology: Fields clusterName
E.2.22. Terminology: Fields commits
E.2.23. Terminology: Fields committedMinSeqno
E.2.24. Terminology: Fields criticalPartition
E.2.25. Terminology: Fields currentBlockSize
E.2.26. Terminology: Fields currentEventId
E.2.27. Terminology: Fields currentLastEventId
E.2.28. Terminology: Fields currentLastFragno
E.2.29. Terminology: Fields currentLastSeqno
E.2.30. Terminology: Fields currentTimeMillis
E.2.31. Terminology: Fields dataServerHost
E.2.32. Terminology: Fields discardCount
E.2.33. Terminology: Fields doChecksum
E.2.34. Terminology: Fields estimatedOfflineInterval
E.2.35. Terminology: Fields eventCount
E.2.36. Terminology: Fields extensions
E.2.37. Terminology: Fields extractTime
E.2.38. Terminology: Fields extractor.class
E.2.39. Terminology: Fields extractor.name
E.2.40. Terminology: Fields filter.#.class
E.2.41. Terminology: Fields filter.#.name
E.2.42. Terminology: Fields filterTime
E.2.43. Terminology: Fields flushIntervalMillis
E.2.44. Terminology: Fields fsyncOnFlush
E.2.45. Terminology: Fields headSeqno
E.2.46. Terminology: Fields intervalGuard
E.2.47. Terminology: Fields lastCommittedBlockSize
E.2.48. Terminology: Fields lastCommittedBlockTime
E.2.49. Terminology: Fields latestEpochNumber
E.2.50. Terminology: Fields logConnectionTimeout
E.2.51. Terminology: Fields logDir
E.2.52. Terminology: Fields logFileRetainMillis
E.2.53. Terminology: Fields logFileSize
E.2.54. Terminology: Fields maxChannel
E.2.55. Terminology: Fields maxDelayInterval
E.2.56. Terminology: Fields maxOfflineInterval
E.2.57. Terminology: Fields maxSize
E.2.58. Terminology: Fields maximumStoredSeqNo
E.2.59. Terminology: Fields minimumStoredSeqNo
E.2.60. Terminology: Fields name
E.2.61. Terminology: Fields offlineRequests
E.2.62. Terminology: Fields otherTime
E.2.63. Terminology: Fields pendingError
E.2.64. Terminology: Fields pendingErrorCode
E.2.65. Terminology: Fields pendingErrorEventId
E.2.66. Terminology: Fields pendingErrorSeqno
E.2.67. Terminology: Fields pendingExceptionMessage
E.2.68. Terminology: Fields pipelineSource
E.2.69. Terminology: Fields processedMinSeqno
E.2.70. Terminology: Fields queues
E.2.71. Terminology: Fields readOnly
E.2.72. Terminology: Fields relativeLatency
E.2.73. Terminology: Fields resourcePrecedence
E.2.74. Terminology: Fields rmiPort
E.2.75. Terminology: Fields role
E.2.76. Terminology: Fields seqnoType
E.2.77. Terminology: Fields serializationCount
E.2.78. Terminology: Fields serialized
E.2.79. Terminology: Fields serviceName
E.2.80. Terminology: Fields serviceType
E.2.81. Terminology: Fields shard_id
E.2.82. Terminology: Fields simpleServiceName
E.2.83. Terminology: Fields siteName
E.2.84. Terminology: Fields sourceId
E.2.85. Terminology: Fields stage
E.2.86. Terminology: Fields started
E.2.87. Terminology: Fields state
E.2.88. Terminology: Fields stopRequested
E.2.89. Terminology: Fields store.#
E.2.90. Terminology: Fields storeClass
E.2.91. Terminology: Fields syncInterval
E.2.92. Terminology: Fields taskCount
E.2.93. Terminology: Fields taskId
E.2.94. Terminology: Fields timeInCurrentEvent
E.2.95. Terminology: Fields timeInStateSeconds
E.2.96. Terminology: Fields timeoutMillis
E.2.97. Terminology: Fields totalAssignments
E.2.98. Terminology: Fields transitioningTo
E.2.99. Terminology: Fields uptimeSeconds
E.2.100. Terminology: Fields version
F. Internals
F.1. Extending Backup and Restore Behavior
F.1.1. Backup Behavior
F.1.2. Restore Behavior
F.1.3. Writing a Custom Backup/Restore Script
F.1.4. Enabling a Custom Backup Script
F.2. Character Sets in Database and Tungsten Cluster
F.3. Understanding Replication of Date/Time Values
F.3. Best Practices
F.4. Tungsten Clustering Schemas
G. Frequently Asked Questions (FAQ)
G.1. General Questions
G.2. Cloud Deployment and Management

List of Figures

3.1. Topologies: Standalone HA Cluster
3.2. Topologies: Composite Active/Passive Cluster
3.3. Topologies: Composite Active/Active Clusters
3.4. Topologies: Composite Dynamic Active/Active Clusters
3.5. Topologies: Multi-Site/Active-Active Clusters
3.6. Topologies: Replicating into a Dataservice
3.7. Topologies: Replicating Data Out of a Cluster
3.8. Migration: Migrating Native Replication using a New Service
5.1. Security Internals: Cluster Communication Channels
6.1. Migration: Migrating Native Replication using a New Service
6.2. Cacti Monitoring: Example Graphs
7.1. Tungsten Connector Basic Architecture
7.2. Basic MySQL/Application Connectivity
7.3. Advanced MySQL/Application Connectivity
7.4. Using Tungsten Connector for MySQL/Application Connectivity
7.5. Tungsten Connector during a failed datasource
7.6. Tungsten Connector routing architecture
7.7. Tungsten Connector Bridge Mode Architecture
7.8. Tungsten Connector Authentication
8.1. Failover Scenario 1
8.2. Failover Scenario 2
8.3. Failover Scenario 3
10.1. Internals: Cluster Communication Channels
12.1. Filters: Pipeline Stages on Extractors
12.2. Filters: Pipeline Stages on Appliers
B.1. Tungsten Deployment

List of Tables

1.1. Key Terminology
2.1. Key Terminology
7.1. Routing Method Selection
7.2. Connector Command Line Sub-Commands
7.3. Inline Interface Commands
9.1. cctrl Command-line Options
9.2. cctrl Command-line Options
9.3. cctrl Command-line Options
9.4. cctrl Command-line Options
9.5. cctrl Command-line Options
9.6. cctrl Command-line Options
9.7. cctrl Command-line Options
9.8. cctrl Command-line Options
9.9. cctrl Command-line Options
9.10. cctrl Command-line Options
9.11. cctrl Command-line Options
9.12. cctrl Commands
9.13. cctrl datasource Commands
9.14. cctrl replicator Commands
9.15. check_tungsten_latency Options
9.16. check_tungsten_online Options
9.17. check_tungsten_policy Options
9.18. check_tungsten_progress Options
9.19. check_tungsten_services Options
9.20. cluster_backup Command-line Options
9.21. connector Commands
9.22. dsctl Commands
9.23. dsctl Command-line Options
9.24. dsctl Command-line Options
9.25. dsctl Command-line Options
9.26. manager Commands
9.27. multi_trepctl Command-line Options
9.28. multi_trepctl--output Option
9.29. multi_trepctl Commands
9.30. query Common Options
9.31. replicator Commands
9.32. replicator Commands Options for condrestart
9.33. replicator Commands Options for console
9.34. replicator Commands Options for restart
9.35. replicator Commands Options for start
9.36. tapi Generic Options
9.37. tapi CURL-related Options
9.38. tapi Nagios/NRPE/Zabbix-related Options
9.39. tapi Admin-related Options
9.40. tapi Filter-related Options
9.41. tapi API-related Options
9.42. tapi Status-related Options
9.43. tapi Backup and Restore-related Options
9.44. thl Options
9.45. tmonitor Common Options
9.46. tpasswd Common Options
9.47. tprovision Command-line Options
9.48. trepctl Command-line Options
9.49. trepctl Replicator Wide Commands
9.50. trepctl Service Commands
9.51. trepctl backup Command Options
9.52. trepctl clients Command Options
9.53. trepctl offline-deferred Command Options
9.54. trepctl online Command Options
9.55. trepctl pause Command Options
9.56. trepctl purge Command Options
9.57. trepctl reset Command Options
9.58. trepctl resume Command Options
9.59. trepctl setdynamic Command Options
9.60. trepctl setrole Command Options
9.61. trepctl shard Command Options
9.62. trepctl status Command Options
9.63. trepctl wait Command Options
9.64. tungsten_find_orphaned Options
9.65. tungsten_find_position Options
9.66. tungsten_get_mysql_datadir Command-line Options
9.67. tungsten_get_ports Options
9.68. tungsten_merge_logs Command-line Options
9.69. tungsten_monitor Command-line Options
9.70. tungsten_newrelic_event Command-line Options
9.71. tungsten_prep_upgrade Command-line Options
9.72. tungsten_reset_manager Options
9.73. tungsten_send_diag Command-line Options
9.74. tungsten_show_processlist Command-line Options
9.75. tungsten_skip_all Options
9.76. tungsten_skip_seqno Command-line Options
9.77. zabbix_tungsten_latency Options
9.78. zabbix_tungsten_online Options
9.79. zabbix_tungsten_progress Options
9.80. check_tungsten_services Options
10.1. tpm Core Options
10.2. tpm Commands
10.3. tpm ask Arguments
10.4. tpm ask Context Options
10.5. tpm copy Common Options
10.6. tpm delete-service Common Options
10.7. tpm diag Common Options
10.8. tungsten_find_seqno Options
10.9. tpm generate-haproxy-for-api Common Options
10.10. tpm policy Options
10.11. tpm-post-process Options
10.12. tpm purge-thl Options
10.13. tpm report Common Options
10.14. tpm Common Options
10.15. tpm Validation Checks
10.16. tpm Configuration Options
B.1. Tungsten OS Support
B.2. MySQL/Tungsten Version Support
C.1. tungsten_upgrade_manager Options
D.1. Continuent Tungsten Directory Structure
D.2. Continuent Tungsten tungsten Sub-Directory Structure
E.1. THL Event Format
   Next
   Preface
Copyright © 2025 Continuent, Ltd.1 (415) 529-4901Terms of UsePrivacy & CookiesLegal NoticeExhibits