In an Composite Active/Active topology, a switch or a failover not only promotes a Replica to be a new Primary, but also will require the ability to reconfigure cross-site communications. This process therefore assumes that cross-site communication is online and working. In some situations, it may be possible that cross-site communication is down, or for some reason cross-site replication is in an OFFLINE:ERROR state - for example a DDL or DML statement that worked in the local cluster may have failed to apply in the remote cluster.

If a switch or failover occurs and the process is unable to reconfigure the cross-site replicators, the local switch will still succeed, however the associated cross-site services will be placed into a SHUNNED(SUBSERVICE-SWITCH-FAILED) state.

The guide explains how to recover from this situation.

The examples are based on a 2-cluster topology, named nyc and london and the composite dataservice named global. The cluster is configured with the following dataservers:

The cross site replicators in both clusters are in an OFFLINE:ERROR state due to failing DDL.

A switch was then issued, promoting db3 as the new Primary in nyc and db5 as the new Primary in london

At the top-level, the switch was a success, however, in the cctrl output below, you can see that all dataources in the london_from_nyc sub-service are in the SHUNNED(SUBSERVICE-SWITCH-FAILED) state, and partial reconfiguration has happened. The same can also be observed in the nyc_from_london sub-service

shell> cctrl
Tungsten Cluster 7.0.3 build 141
nyc: session established
[LOGICAL:EXPERT] / > use london_from_nyc
london_from_nyc: session established, encryption=false, authentication=false
[LOGICAL:EXPERT] /london_from_nyc > ls

COORDINATOR[db6:AUTOMATIC:ONLINE]

DATASOURCES:
+---------------------------------------------------------------------------------+
|db4(relay:SHUNNED(SUBSERVICE-SWITCH-FAILED), progress=-1, latency=-1.000)        |
|STATUS [SHUNNED] [2025/01/29 11:55:42 AM UTC]                                    |
+---------------------------------------------------------------------------------+
|  MANAGER(state=ONLINE)                                                          |
|  REPLICATOR(role=relay, master=db3, state=SUSPECT)                              |
|  DATASERVER(state=ONLINE)                                                       |
|  CONNECTIONS(created=0, active=0)                                               |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|db5(slave:SHUNNED(SUBSERVICE-SWITCH-FAILED), progress=-1, latency=-1.000)        |
|STATUS [SHUNNED] [2025/01/29 11:55:42 AM UTC]                                    |
+---------------------------------------------------------------------------------+
|  MANAGER(state=ONLINE)                                                          |
|  REPLICATOR(role=slave, master=db4, state=SUSPECT)                              |
|  DATASERVER(state=ONLINE)                                                       |
|  CONNECTIONS(created=0, active=0)                                               |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|db6(slave:SHUNNED(SUBSERVICE-SWITCH-FAILED), progress=-1, latency=-1.000)        |
|STATUS [SHUNNED] [2025/01/29 11:55:42 AM UTC]                                    |
+---------------------------------------------------------------------------------+
|  MANAGER(state=ONLINE)                                                          |
|  REPLICATOR(role=slave, master=db4, state=SUSPECT)                              |
|  DATASERVER(state=ONLINE)                                                       |
|  CONNECTIONS(created=0, active=0)                                               |
+---------------------------------------------------------------------------------+

The role of db5 should relay and db4 should be slave.

The replicators for db4 and db6 should be Replicas of db5.

db5 has correctly configured to the new Primary in london and db3 in nyc.

The actual state of the cluster in each scenario maybe different depending upon the cause of the loss of cross-site communication. Using the steps below, apply the necessary actions that relate to your own cluster state, if in any doubt always contact Continuent Support for assistance.

In some situations the relay role may have switched, but the replicators are in a mixed state and not configured correctly, for example, db5 may be showing correctly as the new relay however the replicator may not have updated and may be configured to point to the old master in the nyc cluster (db1 instead of db3).

In that situation, the following steps should be followed:

Whichever process you followed above, you would now repeat on the other cluster if required.