When the site that was lost is returned to operation and all tungsten services have been restarted, if at all possible, the cluster will attempt automatic recovery ensuring the cluster is returned as a slave dataservice and all nodes online. For the automatic recovery to be attempted, the clusters must be in the AUTOMATIC policy mode.

If the nodes cannot be recovered, the first step in recovering the SHUNNED dataservice is to re-provision the nodes if the data has gotten out of sync. See Section 6.6.1.2, “Provision or Reprovision a Replica” for more information.

Once the failed site has been restored, the shunned/superseded dataservice can be brought back online using cctrl . The recover command performs this operation, annotating the progress.

...
DATASOURCES:
+---------------------------------------------------------------------------------+
|alpha(composite master:ONLINE)                                                   |
|STATUS [OK] [2025/01/28 02:36:15 PM UTC]                                         |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|beta(composite master:SHUNNED(SUPERSEDED))                                       |
|STATUS [SHUNNED] [2025/01/28 02:39:41 PM UTC]                                    |
+---------------------------------------------------------------------------------+
...

Use the recover to bring the SHUNNED dataservice back online as a composite Replica:

[LOGICAL] /global > recover
IDENTIFIED DATASOURCE 'beta@global' FOR RECOVERY
COULD NOT IDENTIFY ACTIVE PRIMARY FOR SERVICE 'beta'
ATTEMPTING TO IDENTIFY A FAILED PRIMARY FOR 'beta'
PHYSICAL DATA SERVICE 'beta' DOES NOT HAVE AN ACTIVE RELAY
FORCING THE PHYSICAL RELAY TO BE 'db4'
DATASOURCE 'db4@beta' IS NOW A RELAY
RECOVERED 2 DATA SOURCES IN SERVICE 'beta'
composite data source 'beta@global' role is now slave
composite data source 'beta' is now OFFLINE
REVERT SET POLICY AUTOMATIC
RECOVERY OF COMPOSITE SERVICE 'global' IS COMPLETE

[LOGICAL] /global > ls

COORDINATOR[db2:AUTOMATIC:ONLINE]
   alpha:COORDINATOR[db2:AUTOMATIC:ONLINE]
   beta:COORDINATOR[db4:AUTOMATIC:ONLINE]

ROUTERS:
+---------------------------------------------------------------------------------+
|connector@db1[43475](ONLINE, created=2, active=0)                                |
|connector@db2[75463](ONLINE, created=2, active=0)                                |
|connector@db3[43981](ONLINE, created=2, active=0)                                |
|connector@db4[2062](ONLINE, created=2, active=0)                                 |
|connector@db5[2079](ONLINE, created=2, active=0)                                 |
|connector@db6[2080](ONLINE, created=2, active=0)                                 |
+---------------------------------------------------------------------------------+

DATASOURCES:
+---------------------------------------------------------------------------------+
|alpha(composite master:ONLINE)                                                   |
|STATUS [OK] [2025/01/28 02:36:15 PM UTC]                                         |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|beta(composite slave:ONLINE)                                                     |
|STATUS [OK] [2025/01/28 02:40:06 PM UTC]                                         |
+---------------------------------------------------------------------------------+