In an unfortunate situation of your Galera cluster losing the quorum and having too much data located in different datacenters to perform an SST, it is possible to perform Virtual Machine Cloning and migration to different locations to restore the cluster with an IST.
Technical Expertise Required: Medium
This post will guide you through the steps required to perform a successful Quorum restoration via Virtual Machine cloning and migration to the correct datacenter.
So, let's assume you have a 5 node MariaDB cluster located in 2 datacenters:
10.0.0.4 < MariaDB 1 < Datacenter 1
10.0.0.5 < MariaDB 2 < Datacenter 1
10.1.0.4 < MariaDB 3 < Datacenter 2
10.1.0.5 < MariaDB 4 < Datacenter 2
10.1.0.6 < MariaDB 5 < Datacenter 2
10.0.0.6 < Your Management Node running ClusterControl (consistent with what we use at WooServers)
*At some point you lose your 4 nodes due to a datacenter outage and/or other reasons so that only 1 node remains Up: 10.1.0.5 in DC2. * If your cluster has a high amount of data (500GB+), you will need to perform a full SST for the cluster to get into sync, which may take hours and days.
We noticed, that when doing SST between different datacenters, the procedure may use the whole bandwidth capacity of your virtual machine and make even the only remaining node unavailable for queries.
So SST in this case is not the best solution to choose from.
In this case, it may be best to perform cloning of your virtual machine 10.1.0.5 into all other nodes that are currently down.
Let's do this step by step:
1) First, the cluster needs at least one more node to be able to form a quorum. Since there is only one node running, we will use "Galera Arbitrator" installed on at least one of the missing nodes (for example, 10.0.0.4).
GarbD can be installed using ClusterControl control panel or manually.
Here is the manual installation procedure for "Galera Arbitrator" on Ubuntu:
apt-get install software-properties-common
apt-key adv --keyserver keyserver.ubuntu.com --recv BC19DDBA
Add Galera repo in: /etc/apt/sources.list.d/galera.list
deb http://releases.galeracluster.com/ubuntu trusty main
apt-get install galera-3 galera-arbitrator-3
Next, create a configuration file for GarbD: /etc/garbd.cnf
address = gcomm://10.0.0.4:4567,10.0.0.5:4567,10.1.0.4:4567,10.1.0.5:4567,10.1.0.6:4567
group = mywsrepcluster
options = gmcast.listen_addr=tcp://0.0.0.0:4567
log = /var/log/garbd.log
Replace cluster name and IPs to those of your cluster.
Launch GarbD instance in Screen on one of the failed nodes:
garbd -c /etc/garbd.cnf
Check GarbD log that GarbD instance synced correctly to other node:
tail -f /var/log/garbd.log
2) Secondly, perform cloning and restoration of the virtual machine 10.1.0.5 into any of the down nodes, let's say 10.1.0.4
Virtual Machine cloning is allowed in most clouds like Amazon AWS, OpenStack or Microsoft Azure. If your datacenter does not allow VM cloning, ask their technical support team to perform cloning for you.
3) After cloning a virtual machine into another one, let's say 10.1.0.4, you need to change MariaDB configuration for it to recognise the new IP.
You need to change the following settings in /etc/mysql/my.cnf file:
Use your own settings of IPs of your cluster, where first 3 variables should have the IP of the CLONED vm, while Donor should be set to the only virtual machine running.
4) Finally, Galera stores the Server UUID key in a file called
This file will cause you trouble if you performed VM cloning, since it will contain the UUID of the alive server.
The cloned VM will be UNABLE to re-join cluster if this file is present.
Remove the file
gvwstate.dat from the Cloned VM.
Next, start Cloned VM as normal:
service mysql start
Once MariaDB starts, you should see the following:
150707 17:15:53 [Note] WSREP: Signalling provider to continue.
150707 17:15:53 [Note] WSREP: SST received: d38587ce-246c-11e5-bcce-6bbd0831cc0f:689767
150707 17:15:53 [Note] WSREP: Receiving IST: 71524 writesets, seqnos 689767-761291
This means that the cloned vm started receiving an IST and should shortly successfully sync with the cluster.
Once this is completed, check your Cluster Size variable by logging into mysql and running:
show status like 'wsrep_cluster_size';
| Variable_name | Value |
| wsrep_cluster_size | 3 |
It should show 3, because you are running cluster with 3 nodes - 1 main node, 1 Galera Arbitrator and 1 Cloned Virtual Machine that has successfully joined the cluster.
We recommend restoring at least one more machine before switching off Galera Arbitrator instance.
Contact our support team if you require assistance or outsourced MariaDB/Percona Galera cluster management.
Contact Severalnines about ClusterControl.