# Set up Percona XtraDB Cluster cross-site replication The cross-site replication involves configuring one Percona XtraDB Cluster as *Source*, and another Percona XtraDB Cluster as *Replica* to allow an asynchronous replication between them: ![image](assets/images/pxc-replication.svg) The Operator automates configuration of *Source* and *Replica* Percona XtraDB Clusters, but the feature itself is not bound to Kubernetes. Either *Source* or *Replica* can run outside of Kubernetes, be regular MySQL and be out of the Operators’ control. This feature can be useful in several cases: for example, it can simplify migration from on-premises to the cloud with replication, and it can be really helpful in case of the disaster recovery too. !!! note Cross-site replication is based on [Automatic Asynchronous Replication Connection Failover :octicons-link-external-16:](https://dev.mysql.com/doc/refman/8.0/en/replication-asynchronous-connection-failover.html). Therefore it requires MySQL 8.0.22+ (Percona XtraDB Cluster 8.0.22+) to work. Setting up MySQL for asynchronous replication without the Operator is out of the scope for this document, but it is described [here :octicons-link-external-16:](https://www.percona.com/blog/2021/04/14/what-you-can-do-with-auto-failover-and-percona-distribution-for-mysql-8-0-x/) and is also covered by [this HowTo](backups-move-from-external-db.md). Configuring the cross-site replication for the cluster controlled by the Operator is explained in the following subsections. ## Creating a Replica cluster Cross-site replication can be configured on two sibling Percona XtraDB Clusters. That's why you should first make a fully operational clone of your main database cluster. After that your original cluster will be configured as *Source*, and a new one (the clone) will be configured as *Replica*. The easiest way to achieve this is to use backups. You make a full backup of your main database cluster, and restore it to a new Kubernetes-based environment, following [this HowTo](backups-restore-to-new-cluster.md). ## Configuring cross-site replication on Source instances You can configure *Source* instances for cross-site replication with `spec.pxc.replicationChannels` subsection in the `deploy/cr.yaml` configuration file. It is an array of channels, and you should provide the following keys for the channel in your *Source* Percona XtraDB Cluster: * `pxc.replicationChannels.[].name` key is the name of the channel, * `pxc.replicationChannels.[].isSource` key should be set to `true`. Here is an example: ```yaml spec: pxc: replicationChannels: - name: pxc1_to_pxc2 isSource: true ``` You will also need to expose every Percona XtraDB Cluster Pod of the *Source* cluster to make it possible for the *Replica* cluster to connect. This is done through the `pxc.expose` section in the `deploy/cr.yaml` configuration file as follows. ```yaml spec: pxc: expose: enabled: true type: LoadBalancer ``` !!! note This will create a LoadBalancer per each Percona XtraDB Cluster Pod. In most cases, for cross-region replication to work this Load Balancer should be internet-facing. The cluster will be ready for asynchronous replication when you apply changes as usual: ``` {.bash data-prompt="$" } $ kubectl apply -f deploy/cr.yaml ``` To list the endpoints assigned to PXC Pods list the Kubernetes Service objects by executing `kubectl get services -l "app.kubernetes.io/instance=cluster1"` command (don't forget to substitute `cluster1` with the real name of your cluster, if you don't use the default name). ## Configuring cross-site replication on Replica instances You can configure *Replica* instances for cross-site replication with `spec.pxc.replicationChannels` subsection in the `deploy/cr.yaml` configuration file. It is an array of channels, and you should provide the following keys for the channel in your *Replica* Percona XtraDB Cluster: * `pxc.replicationChannels.[].name` key is the name of the channel, * `pxc.replicationChannels.[].isSource` key should be set to `false`, * `pxc.replicationChannels.[].sourcesList` is the list of *Source* cluster names from which Replica should get the data, * `pxc.replicationChannels.[].sourcesList.[].host` is the host name or IP address of the Source, * `pxc.replicationChannels.[].sourcesList.[].port` is the port of the source (`3306` port will be used if nothing specified), * `pxc.replicationChannels.[].sourcesList.[].weight` is the *weight* of the source (in the event of a connection failure, a new source is selected from the list based on a weighted priority). Here is the example: ```yaml spec: pxc: replicationChannels: - name: uspxc1_to_pxc2 isSource: false sourcesList: - host: pxc1.source.percona.com port: 3306 weight: 100 - host: pxc2.source.percona.com weight: 100 - host: pxc3.source.percona.com weight: 100 - name: eu_to_pxc2 isSource: false sourcesList: - host: pxc1.source.percona.com port: 3306 weight: 100 - host: pxc2.source.percona.com weight: 100 - host: pxc3.source.percona.com weight: 100 ``` The cluster will be ready for asynchronous replication when you apply changes as usual: ``` {.bash data-prompt="$" } $ kubectl apply -f deploy/cr.yaml ``` !!! note You can also [configure SSL channel for replication :octicons-link-external-16:](https://dev.mysql.com/doc/refman/8.0/en/replication-encrypted-connections.html). Following options allow you using replication over an encrypted channel. Set the `replicationChannels.configuration.ssl` key to true, optionally enable host name identity verification with the `replicationChannels.configuration.sslSkipVerify` key, and set `replicationChannels.configuration.ca` key to the path name of the Certificate Authority (CA) certificate file: ```yaml replicationChannels: - isSource: false name: uspxc1_to_pxc2 configuration: ssl: true sslSkipVerify: true ca: '/etc/mysql/ssl/ca.crt' ... ``` SSL certificates on both sides should be signed by the same certificate authority for encrypted replication channels to work. ## System user for replication Replication channel demands a special [system user](users.md#system-users) with same credentials on both *Source* and *Replica*. The Operator creates a system-level Percona XtraDB Cluster user named `replication` for this purpose, with credentials stored in a Secret object [along with other system users](users.md#system-users). !!! note If the Replica cluster is not a clone of the original one (for example, it's outside of Kubernetes and is not under the Operator’s control) [the appropriate user with necessary permissions :octicons-link-external-16:](https://dev.mysql.com/doc/refman/8.0/en/replication-asynchronous-connection-failover.html) should be created manually. If you need you can change a password for this user as follows: === "in Linux" ``` {.bash data-prompt="$" } $ kubectl patch secret/cluster1-secrets -p '{"data":{"replication": "'$(echo -n new_password | base64 --wrap=0)'"}}' ``` === "in macOS" ``` {.bash data-prompt="$" } $ kubectl patch secret/cluster1-secrets -p '{"data":{"replication": "'$(echo -n new_password | base64)'"}}' ``` If you have changed the `replication` user’s password on the Source cluster, and you use the Operator version 1.9.0, you can have a *replication is not running* error message in log, similar to the following one: ``` {.text .no-copy} {"level":"info","ts":1629715578.2569592,"caller":"zapr/zapr.go 69","msg":"Replication for channel is not running. Please, check the replication status","channel":"pxc2_to_pxc1"} ``` Fixing this involves the following steps. 1. Find the Replica Pod which was chosen by the Operator for replication, using the following command: ``` {.bash data-prompt="$" } $ kubectl get pods --selector percona.com/replicationPod=true ``` 2. Get the shell access to this Pod and login to the MySQL monitor as a [root user](users.md#system-users): ``` {.bash data-prompt="$" } $ kubectl exec -c pxc --stdin --tty -- /bin/bash bash-4.4$ mysql -uroot -proot_password ``` 3. Execute the following three SQL commands to propagate the `replication` user password from the Source cluster to Replica: ```sql STOP REPLICA IO_THREAD FOR CHANNEL '$REPLICATION_CHANNEL_NAME'; CHANGE MASTER TO MASTER_PASSWORD='$NEW_REPLICATION_PASSWORD' FOR CHANNEL '$REPLICATION_CHANNEL_NAME'; START REPLICA IO_THREAD FOR CHANNEL '$REPLICATION_CHANNEL_NAME'; ```