6.7.2. Composite Cluster Site Failover (Forced Switch)

In the event the Active site goes down, and a graceful manual switch is not possible, the composite Active role can be failed over to the Passive cluster using cctrl. The failover command performs the forced switch operation. It will try to update the configuration of the east data service but will not fail if not successful.

In this example, hosts db4 (the composite Primary), db5 and db6 in cluster beta have been shut down. To force dataservice alpha to become the primary, login to a node in that cluster and get into cctrl:

shell> cctrl -multi
Tungsten Cluster 7.0.3 build 141
west: session established
[LOGICAL] / > use global
[LOGICAL] /global > ls

COORDINATOR[db2:AUTOMATIC:ONLINE]
   alpha:COORDINATOR[db2:AUTOMATIC:ONLINE]

ROUTERS:
+---------------------------------------------------------------------------------+
|connector@db1[43475](ONLINE, created=2, active=0)                                |
|connector@db2[75463](ONLINE, created=2, active=0)                                |
|connector@db3[43981](ONLINE, created=2, active=0)                                |
+---------------------------------------------------------------------------------+

DATASOURCES:
+---------------------------------------------------------------------------------+
|alpha(composite slave:ONLINE)                                                    |
|STATUS [OK] [2025/01/28 01:53:13 PM UTC]                                         |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|beta(composite master:SHUNNED(FAILSAFE_SHUN))                                    |
|STATUS [SHUNNED] [2025/01/28 02:21:56 PM UTC]                                    |
+---------------------------------------------------------------------------------+

Mark the beta data service as failed to prevent further actions:

[LOGICAL] /global > datasource beta fail

WARNING: This is an expert-level command:
Incorrect use may cause data corruption
or make the cluster unavailable.

Do you want to continue? (y/n)> y
COMPOSITE DATA SOURCE 'beta' IS NOW IN THE FAILED STATE

[LOGICAL] /global > ls

COORDINATOR[db2:AUTOMATIC:ONLINE]
   alpha:COORDINATOR[db2:AUTOMATIC:ONLINE]

ROUTERS:
+---------------------------------------------------------------------------------+
|connector@db1[43475](ONLINE, created=2, active=0)                                |
|connector@db2[75463](ONLINE, created=2, active=0)                                |
|connector@db3[43981](ONLINE, created=2, active=0)                                |
+---------------------------------------------------------------------------------+

DATASOURCES:
+---------------------------------------------------------------------------------+
|alpha(composite slave:ONLINE)                                                    |
|STATUS [OK] [2025/01/28 01:53:13 PM UTC]                                         |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|beta(composite master:FAILED(MANUALLY-FAILED))                                   |
|STATUS [CRITICAL] [2025/01/28 02:29:53 PM UTC]                                   |
|REASON[MANUALLY-FAILED]                                                          |
+---------------------------------------------------------------------------------+

Issue the failover command to force the alpha dataservice to become the composite Primary:

[LOGICAL] /global > failover
WARNING: DATA SERVICE 'beta' IS NOT AVAILABLE. CANNOT GET STATE
EXCEPTION=Unable to continue with command because no manager is available in service 'beta'.

Savepoint failover_1(cluster=global, source=db2, created=2025/01/28 14:35:37 UTC) created
SELECTED SLAVE: 'alpha@global'
ENSURING THAT WE CATCH UP WITH THE MOST ADVANCED RELAY
composite data source 'alpha@global' is now OFFLINE
PUT THE NEW MASTER 'alpha@global' ONLINE
REVERT POLICY: MAINTENANCE => AUTOMATIC
FAILOVER TO 'alpha@global' WAS SUCCESSFUL

[LOGICAL] /global > ls

COORDINATOR[db2:AUTOMATIC:ONLINE]
   alpha:COORDINATOR[db2:AUTOMATIC:ONLINE]

ROUTERS:
+---------------------------------------------------------------------------------+
|connector@db1[43475](ONLINE, created=2, active=0)                                |
|connector@db2[75463](ONLINE, created=2, active=0)                                |
|connector@db3[43981](ONLINE, created=2, active=0)                                |
+---------------------------------------------------------------------------------+

DATASOURCES:
+---------------------------------------------------------------------------------+
|alpha(composite master:ONLINE)                                                   |
|STATUS [OK] [2025/01/28 02:36:15 PM UTC]                                         |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|beta(composite master:SHUNNED(MANUAL-FAILOVER))                                  |
|STATUS [SHUNNED] [2025/01/28 02:35:46 PM UTC]                                    |
+---------------------------------------------------------------------------------+