Replicating from Cloud managed databases, such as Amazon Aurora or Google SQL, operates by directly accessing the binary log provided by the service and enables you to replicate from the remote instance, or to a standard instance within your cloud provider environment, or even out to self-hosted instances. The complexity with Cloud Managed instance is that there is no access to the host that is running the instance, or the MySQL binary logs.
To use this service, two aspects of the Tungsten Replicator are
required, direct mode and unprivileged user support. Direct mode reads the
MySQL binary log over the network, rather than accessing the binlog on the
filesystem. The unprivileged mode enables the user to access and update
information within Aurora (for example) without requiring
SUPER privileges, which are
unavailable within a Cloud Managed instance instance. For more information, see
Section B.4.5, “MySQL Unprivileged Users”.
The deployment requires a host for the extractor installation, this can be an instance within your cloud environment, or it could be a remote host in your own environment.
This deployment follows a similar model to an Offboard Installation
Before starting the installation, the prerequisites must have been completed (see Appendix B, Prerequisites) on both the Host designated for the installation of the extractor, and within the source database instance.
Install the Tungsten Replicator package (see Section 2.1.2, “Using the RPM package files”), or download the compressed tarball and unpack it to a staging directory:
shell>
cd /opt/continuent/softwareshell>tar zxf tungsten-replicator-6.1.25-6.tar.gzChange to the Tungsten Replicator staging directory:
shell>
cd tungsten-replicator-6.1.25-6Configure the replicator for extraction (In this example, the service name is alpha)
shell>
vi /etc/tungsten/tungsten.ini[defaults] install-directory=/opt/continuent user=tungsten profile-script=~/.bash_profile mysql-allow-intensive-checks=true skip-validation-check=InstallerMasterSlaveCheck skip-validation-check=MySQLPermissionsCheck skip-validation-check=MySQLBinaryLogsEnabledCheck skip-validation-check=MySQLMyISAMCheck skip-validation-check=RowBasedBinaryLoggingCheck rest-api-admin-user=apiuser rest-api-admin-password=secret replicator-rest-api-address=0.0.0.0 cloud-install=true[alpha] master=localhost members=localhost enable-heterogeneous-service=true privileged-master=false replication-host=rds.endpoint.url replication-port=3306 replication-user=tungsten_alpha replication-password=secret datasource-mysql-conf=/dev/null svc-extractor-filters=dropcatalogdata property=replicator.service.comments=trueConfiguration group
defaultsThe description of each of the options is shown below; click the icon to hide this detail:
install-directory=/opt/continuentPath to the directory where the active deployment will be installed. The configured directory will contain the software, THL and relay log information unless configured otherwise.
System User
profile-script=~/.bash_profileAppend commands to include env.sh in this profile script
mysql-allow-intensive-checks=trueFor MySQL installation, enables detailed checks on the supported data types within the MySQL database to confirm compatibility. This includes checking each table definition individually for any unsupported data types.
skip-validation-check=InstallerMasterSlaveCheckThe
skip-validation-checkdisables a given validation check. If any validation check fails, the installation, validation or configuration will automatically stop.Warning
Using this option enables you to bypass the specified check, although skipping a check may lead to an invalid or non-working configuration.
You can identify a given check if an error or warning has been raised during configuration. For example, the default table type check:
ERROR >> centos >> The datasource root@centos:3306 (WITH PASSWORD) » uses MyISAM as the default storage engine (MySQLDefaultTableTypeCheck)
The check in this case is
MySQLDefaultTableTypeCheck, and could be ignored usingskip-validation-check=MySQLDefaultTableTypeCheck.Values can be passed as a comma-separated list, or single
skip-validation-checkentries for each check to be skipped.Setting both
skip-validation-checkandenable-validation-checkis equivalent to explicitly disabling the specified check.skip-validation-check=MySQLPermissionsCheckThe
skip-validation-checkdisables a given validation check. If any validation check fails, the installation, validation or configuration will automatically stop.Warning
Using this option enables you to bypass the specified check, although skipping a check may lead to an invalid or non-working configuration.
You can identify a given check if an error or warning has been raised during configuration. For example, the default table type check:
ERROR >> centos >> The datasource root@centos:3306 (WITH PASSWORD) » uses MyISAM as the default storage engine (MySQLDefaultTableTypeCheck)
The check in this case is
MySQLDefaultTableTypeCheck, and could be ignored usingskip-validation-check=MySQLDefaultTableTypeCheck.Values can be passed as a comma-separated list, or single
skip-validation-checkentries for each check to be skipped.Setting both
skip-validation-checkandenable-validation-checkis equivalent to explicitly disabling the specified check.skip-validation-check=MySQLBinaryLogsEnabledCheckThe
skip-validation-checkdisables a given validation check. If any validation check fails, the installation, validation or configuration will automatically stop.Warning
Using this option enables you to bypass the specified check, although skipping a check may lead to an invalid or non-working configuration.
You can identify a given check if an error or warning has been raised during configuration. For example, the default table type check:
ERROR >> centos >> The datasource root@centos:3306 (WITH PASSWORD) » uses MyISAM as the default storage engine (MySQLDefaultTableTypeCheck)
The check in this case is
MySQLDefaultTableTypeCheck, and could be ignored usingskip-validation-check=MySQLDefaultTableTypeCheck.Values can be passed as a comma-separated list, or single
skip-validation-checkentries for each check to be skipped.Setting both
skip-validation-checkandenable-validation-checkis equivalent to explicitly disabling the specified check.skip-validation-check=MySQLMyISAMCheckThe
skip-validation-checkdisables a given validation check. If any validation check fails, the installation, validation or configuration will automatically stop.Warning
Using this option enables you to bypass the specified check, although skipping a check may lead to an invalid or non-working configuration.
You can identify a given check if an error or warning has been raised during configuration. For example, the default table type check:
ERROR >> centos >> The datasource root@centos:3306 (WITH PASSWORD) » uses MyISAM as the default storage engine (MySQLDefaultTableTypeCheck)
The check in this case is
MySQLDefaultTableTypeCheck, and could be ignored usingskip-validation-check=MySQLDefaultTableTypeCheck.Values can be passed as a comma-separated list, or single
skip-validation-checkentries for each check to be skipped.Setting both
skip-validation-checkandenable-validation-checkis equivalent to explicitly disabling the specified check.skip-validation-check=RowBasedBinaryLoggingCheckThe
skip-validation-checkdisables a given validation check. If any validation check fails, the installation, validation or configuration will automatically stop.Warning
Using this option enables you to bypass the specified check, although skipping a check may lead to an invalid or non-working configuration.
You can identify a given check if an error or warning has been raised during configuration. For example, the default table type check:
ERROR >> centos >> The datasource root@centos:3306 (WITH PASSWORD) » uses MyISAM as the default storage engine (MySQLDefaultTableTypeCheck)
The check in this case is
MySQLDefaultTableTypeCheck, and could be ignored usingskip-validation-check=MySQLDefaultTableTypeCheck.Values can be passed as a comma-separated list, or single
skip-validation-checkentries for each check to be skipped.Setting both
skip-validation-checkandenable-validation-checkis equivalent to explicitly disabling the specified check.- Optional: Must be specified along with rest-api-admin-pass if you wish to access the full API features and use the Dashboard GUI for cluster installations.
- Optional: Must be specified along with rest-api-admin-user if you wish to access the full API features.
replicator-rest-api-address=0.0.0.0Address for the API to bind too.
When set to
true(falseby default) tpm will skip checks for the endpoint hostname existing in the /etc/hosts file, allowing for much smoother installations against cloud based sources or targets such as Amazon Aurora and Google Cloud SQL.
Configuration group
alphaThe description of each of the options is shown below; click the icon to hide this detail:
The hostname of the primary (extractor) within the current service.
Hostnames for the dataservice members
enable-heterogeneous-service=trueOn a Primary
mysql-use-bytes-for-stringis set to false.colnamesfilter is enabled (in thebinlog-to-qstage to add column names to the THL information.pkeyfilter is enabled (in thebinlog-to-qandq-to-dbmsstage), with theaddPkeyToInsertsandaddColumnsToDeletesfilter options set to false.enumtostringfilter is enabled (in theq-to-thlstage), to translateENUMvalues to their string equivalents.settostringfilter is enabled (in theq-to-thlstage), to translateSETvalues to their string equivalents.
On a Replica
mysql-use-bytes-for-stringis set to true.pkeyfilter is enabled (q-to-dbmsstage).
Does the login for the Primary database service have superuser privileges
replication-host=rds.endpoint.urlHostname of the datasource where the database is located. If the specified hostname matches the current host or member name, the database is assumed to be local. If the hostnames do not match, extraction is assumed to be via remote access. For MySQL hosts, this configures a remote replication Replica (relay) connection.
The network port used to connect to the database server. The default port used depends on the database being configured.
replication-user=tungsten_alphaFor databases that required authentication, the username to use when connecting to the database using the corresponding connection method (native, JDBC, etc.).
The password to be used when connecting to the database using the corresponding
--replication-user.datasource-mysql-conf=/dev/nullMySQL config file
svc-extractor-filters=dropcatalogdataReplication service extractor filters
property=replicator.service.comments=trueThe
propertyoption enables you to explicitly set property values in the target files. A number of different models are supported:key=valueSet the property defined by
keyto the specified value without evaluating any template values or other rules.key+=valueAdd the value to the property defined by
key. Template values and other options append their settings to the end of the specified property.key~=/match/replace/Evaluate any template values and other settings, and then perform the specified Ruby regex operation to the property defined by
key. For exampleproperty=replicator.key~=/(.*)/somevalue,\1/will prependsomevaluebefore the template value forreplicator.key.
Once the prerequisites and configuring of the installation has been completed, the software can be installed:
shell>
./tools/tpm install
In the above examples,
enable-heterogeneous-service, is only required if the target applier is NOT a MySQL databasedatasource-mysql-conf, needs to be set as shown as we do not have access to themy.cnffile
If the installation process fails, check the output of the
/tmp/tungsten-configure.log file for
more information about the root cause.
Once the installation has been completed, you can now proceed to configure the Applier service following the relevant step within Chapter 4, Deploying Appliers.
Following installation of the applier, the services can be started. For information on starting and stopping Tungsten Cluster see Section 2.4, “Starting and Stopping Tungsten Components”; configuring init scripts to startup and shutdown when the system boots and shuts down, see Section 2.5, “Configuring Startup on Boot”.
Monitoring the extractor is the same as an extractor from MySQL, for information, see Section 3.2.1, “Monitoring the MySQL Extractor”.