# Set up Percona XtraDB Cluster cross-site replication
The cross-site replication involves configuring one Percona XtraDB Cluster as *Source*, and another Percona XtraDB Cluster as *Replica* to allow an asynchronous replication between them:
![image](assets/images/pxc-replication.svg)
The Operator automates configuration of *Source* and *Replica* Percona XtraDB Clusters, but the feature itself is not bound to Kubernetes. Either *Source* or *Replica* can run outside of Kubernetes, be regular MySQL and be out of the Operators’ control.
This feature can be useful in several cases: for example, it can simplify migration from on-premises to the cloud with replication, and it can be really helpful in case of the disaster recovery too.
!!! note
Cross-site replication is based on [Automatic Asynchronous Replication Connection Failover :octicons-link-external-16:](https://dev.mysql.com/doc/refman/8.0/en/replication-asynchronous-connection-failover.html). Therefore it requires MySQL 8.0.22+ (Percona XtraDB Cluster 8.0.22+) to work.
Setting up MySQL for asynchronous replication without the Operator is out of the scope for this document, but it is described [here :octicons-link-external-16:](https://www.percona.com/blog/2021/04/14/what-you-can-do-with-auto-failover-and-percona-distribution-for-mysql-8-0-x/) and is also covered by [this HowTo](backups-move-from-external-db.md).
Configuring the cross-site replication for the cluster controlled by the Operator is explained in the following subsections.
## Creating a Replica cluster
Cross-site replication can be configured on two sibling Percona XtraDB Clusters.
That's why you should first make a fully operational clone of your main database cluster. After that your original cluster will be configured as *Source*, and a new one (the clone) will be configured as *Replica*.
The easiest way to achieve this is to use backups. You make a full backup of your main database cluster, and restore it to a new Kubernetes-based environment, following [this HowTo](backups-restore-to-new-cluster.md).
## Configuring cross-site replication on Source instances
You can configure *Source* instances for cross-site replication with `spec.pxc.replicationChannels` subsection in the `deploy/cr.yaml` configuration file. It is an array of channels, and you should provide the following keys for the channel in your *Source* Percona XtraDB Cluster:
* `pxc.replicationChannels.[].name` key is the name of the channel,
* `pxc.replicationChannels.[].isSource` key should be set to `true`.
Here is an example:
```yaml
spec:
pxc:
replicationChannels:
- name: pxc1_to_pxc2
isSource: true
```
You will also
need to expose every Percona XtraDB Cluster Pod of the *Source* cluster to
make it possible for the *Replica* cluster to connect. This is done through the
`pxc.expose` section in the `deploy/cr.yaml` configuration file as follows.
```yaml
spec:
pxc:
expose:
enabled: true
type: LoadBalancer
```
!!! note
This will create a LoadBalancer per each Percona XtraDB Cluster Pod.
In most cases, for cross-region replication to work this Load Balancer should
be internet-facing.
The cluster will be ready for asynchronous replication when you apply changes as usual:
``` {.bash data-prompt="$" }
$ kubectl apply -f deploy/cr.yaml
```
To list the endpoints assigned to PXC Pods list the Kubernetes Service objects by
executing `kubectl get services -l "app.kubernetes.io/instance=cluster1"` command
(don't forget to substitute `cluster1` with the real name of your cluster, if you
don't use the default name).
## Configuring cross-site replication on Replica instances
You can configure *Replica* instances for cross-site replication with `spec.pxc.replicationChannels` subsection in the `deploy/cr.yaml` configuration file. It is an array of channels, and you should provide the following keys for the channel in your *Replica* Percona XtraDB Cluster:
* `pxc.replicationChannels.[].name` key is the name of the channel,
* `pxc.replicationChannels.[].isSource` key should be set to `false`,
* `pxc.replicationChannels.[].sourcesList` is the list of *Source* cluster names from which Replica should get the data,
* `pxc.replicationChannels.[].sourcesList.[].host` is the host name or IP address of the Source,
* `pxc.replicationChannels.[].sourcesList.[].port` is the port of the source (`3306` port will be used if nothing specified),
* `pxc.replicationChannels.[].sourcesList.[].weight` is the *weight* of the source (in the event of a connection failure, a new source is selected from the list based on a weighted priority).
Here is the example:
```yaml
spec:
pxc:
replicationChannels:
- name: uspxc1_to_pxc2
isSource: false
sourcesList:
- host: pxc1.source.percona.com
port: 3306
weight: 100
- host: pxc2.source.percona.com
weight: 100
- host: pxc3.source.percona.com
weight: 100
- name: eu_to_pxc2
isSource: false
sourcesList:
- host: pxc1.source.percona.com
port: 3306
weight: 100
- host: pxc2.source.percona.com
weight: 100
- host: pxc3.source.percona.com
weight: 100
```
The cluster will be ready for asynchronous replication when you apply changes as usual:
``` {.bash data-prompt="$" }
$ kubectl apply -f deploy/cr.yaml
```
!!! note
You can also [configure SSL channel for replication :octicons-link-external-16:](https://dev.mysql.com/doc/refman/8.0/en/replication-encrypted-connections.html). Following
options allow you using replication over an encrypted channel.
Set the `replicationChannels.configuration.ssl` key to true, optionally
enable host name identity verification with the
`replicationChannels.configuration.sslSkipVerify` key, and set
`replicationChannels.configuration.ca` key to the path name of the
Certificate Authority (CA) certificate file:
```yaml
replicationChannels:
- isSource: false
name: uspxc1_to_pxc2
configuration:
ssl: true
sslSkipVerify: true
ca: '/etc/mysql/ssl/ca.crt'
...
```
SSL certificates on both sides should be signed by the same certificate
authority for encrypted replication channels to work.
## System user for replication
Replication channel demands a special [system user](users.md#system-users) with same credentials on both *Source* and *Replica*.
The Operator creates a system-level Percona XtraDB Cluster user named `replication` for this purpose, with
credentials stored in a Secret object [along with other system users](users.md#system-users).
!!! note
If the Replica cluster is not a clone of the original one (for example, it's outside of Kubernetes and is not under the Operator’s control) [the appropriate user with necessary permissions :octicons-link-external-16:](https://dev.mysql.com/doc/refman/8.0/en/replication-asynchronous-connection-failover.html) should be created manually.
If you need you can change a password for this user as follows:
=== "in Linux"
``` {.bash data-prompt="$" }
$ kubectl patch secret/cluster1-secrets -p '{"data":{"replication": "'$(echo -n new_password | base64 --wrap=0)'"}}'
```
=== "in macOS"
``` {.bash data-prompt="$" }
$ kubectl patch secret/cluster1-secrets -p '{"data":{"replication": "'$(echo -n new_password | base64)'"}}'
```
If you have changed the `replication` user’s password on the Source cluster, and you use the Operator version 1.9.0, you can have a *replication is not running* error message in log, similar to the following one:
``` {.text .no-copy}
{"level":"info","ts":1629715578.2569592,"caller":"zapr/zapr.go 69","msg":"Replication for channel is not running. Please, check the replication status","channel":"pxc2_to_pxc1"}
```
Fixing this involves the following steps.
1. Find the Replica Pod which was chosen by the Operator for replication, using the following command:
``` {.bash data-prompt="$" }
$ kubectl get pods --selector percona.com/replicationPod=true
```
2. Get the shell access to this Pod and login to the MySQL monitor as a [root user](users.md#system-users):
``` {.bash data-prompt="$" }
$ kubectl exec -c pxc --stdin --tty -- /bin/bash
bash-4.4$ mysql -uroot -proot_password
```
3. Execute the following three SQL commands to propagate the `replication` user password from the Source cluster to Replica:
```sql
STOP REPLICA IO_THREAD FOR CHANNEL '$REPLICATION_CHANNEL_NAME';
CHANGE MASTER TO MASTER_PASSWORD='$NEW_REPLICATION_PASSWORD' FOR CHANNEL '$REPLICATION_CHANNEL_NAME';
START REPLICA IO_THREAD FOR CHANNEL '$REPLICATION_CHANNEL_NAME';
```