# Transport layer security (TLS) The Percona Operator for PostgreSQL uses Transport Layer Security (TLS) cryptographic protocol for the following types of communication: * Internal - communication between PostgreSQL instances in the cluster * External - communication between the client application and the cluster The internal certificate is also used as an authorization method for PostgreSQL Replica instances. TLS security can be configured in following ways: * the Operator can generate long-term certificates automatically at cluster creation time, * you can generate certificates manually. !!! note Additionally, you can *force* your database cluster to use only encrypted channels for both internal and external communications. This effect is achieved by setting the `tlsOnly` Custom Resource option to `true`. ## Allow the Operator to generate certificates automatically The Operator is able to generate long-term certificates automatically and turn on encryption at cluster creation time, if there are no certificate secrets available. Just deploy your cluster as usual, with the `kubectl apply -f deploy/cr.yaml` command, and certificates will be generated. !!! note Starting from 2.5.0, the Operator creates root CA on per-cluster basis. In Operator versions before 2.5.0, autogenerated certificates for all database clusters were based on the same generated root CA. ## Check connectivity to the cluster You can check TLS communication using `psql`, the standard interactive terminal-based frontend to PostgreSQL. For this purpose, we will create a `pg-client` Deployment, which includes the necessary tools. We will use the existing Secret object with the TLS certificates, generated by the Operator. Follow these steps: 1. Export the namespace as the environment variable to simplify further configuration: ```bash export NAMESPACE= ``` 2. List the Secret objects: ```bash kubectl get secrets -n $NAMESPACE ``` ??? example "Expected output" ```text cluster1-cluster-ca-cert Opaque 2 37m cluster1-cluster-cert Opaque 3 37m cluster1-instance1-5nlp-certs Opaque 6 37m cluster1-instance1-8rks-certs Opaque 6 37m cluster1-instance1-z5tz-certs Opaque 6 37m cluster1-pgbackrest Opaque 5 37m cluster1-pgbouncer Opaque 6 37m cluster1-pguser-cluster1 Opaque 12 37m cluster1-replication-cert Opaque 3 37m ``` The secret with TLS certificates is `-cluster-ca-cert` (`cluster1-cluster-ca-cert` by default). 3. Create a deployment. Replace the placeholders in the following command with your values: * Replace the `` placeholder with your actual cluster name * Specify the Secret object with the TLS certificate from step 1 * Specify the CA certificate file for the `volumeMounts.name` and `volumes.name` options. This file is used to mount the root CA certificate into the container so that client tools like `psql` can verify the server's certificate. To view the file name, run `kubectl get secret -cluster-ca-cert -o yaml -n $NAMESPACE` command. ```bash cat <-cluster-ca-cert items: - key: root.crt path: root.crt mode: 0777 EOF ``` 4. Retrieve the `pgBouncer` URI to connect to PostgreSQL. It is stored in the Secret object with user credentials: `-pguser-`. The default value is `cluster1-pguser-cluster1`. Run the following command and replace the `cluster1-pguser-cluster1` Secret name with your value: ```bash kubectl get secret cluster1-pguser-cluster1 -o jsonpath='{.data.pgbouncer-uri}' -n $NAMESPACE | base64 --decode ``` ??? example "Sample output" ```text postgresql://cluster1:@cluster1-pgbouncer.default.svc:5432/cluster1 5. Now get shell access to the newly created container: ```bash kubectl exec -it deployment/pg-client -- bash -il ``` ??? example "Expected output" ```text [postgres@pg-client-54b449898f-ztlrh /]$ ``` 6. Launch the `psql` interactive terminal to check connectivity over the encrypted channel. Make sure to use your cluster name, the pgBouncer URL you retrieved at the previous step, and the CA certificate you provided when you created the `pg-client` deployment: ```bash PGSSLMODE=verify-ca PGSSLROOTCERT=/tmp/tls/root.crt psql 'postgresql://cluster1:@cluster1-pgbouncer.default.svc:5432/cluster1' ``` Now you should see the prompt of PostgreSQL interactive terminal: ```bash psql ({{ postgresrecommended }}) Type "help" for help. cluster1=> ``` ## Generate certificates manually You can customize TLS for the Operator by providing your own TLS certificates. To do this, you must create two Kubernetes Secret objects *before* deploying your cluster: * One for external communication, later referenced by the `spec.customTLSSecret` field in the `deploy/cr.yaml` * One for internal communication (used for replication authentication), referenced by the `spec.customReplicationTLSSecret` field in the `deploy/cr.yaml`. Each Secret must contain the following fields: - `tls.crt` (the TLS certificate) - `tls.key` (the TLS private key) - `ca.crt` (the Certificate Authority certificate) Note that you cannot use only one custom set of certificates. If you provide a custom TLS Secret, you **must** also provide a custom replication TLS Secret, and both must contain the same `ca.crt`. ### Provide pre-existing custom certificates For example, you have files named `ca.crt`, `my_tls.key`, and `my_tls.crt`. Run the following command to create a custom TLS Secret named `cluster1-tls`: ```bash kubectl create secret generic -n postgres-operator cluster1-tls \ --from-file=ca.crt=ca.crt \ --from-file=tls.key=my_tls.key \ --from-file=tls.crt=my_tls.crt ``` In the same way, create the custom TLS replication Secret, for example `replication1-tls`. Next, reference your Secrets in the `deploy/cr.yaml` Custom Resource manifest as follows: * add a Secret created for the external use to the `secrets.customTLSSecret.name` field * add a Secret created for internal communications to the `secrets.customReplicationTLSSecret.name` field Here's the sample configuration: ```yaml spec: ... secrets: customTLSSecret: name: cluster1-tls customReplicationTLSSecret: name: replication1-tls ... ``` Now you can create a cluster with your custom certificates: ```bash kubectl apply -f deploy/cr.yaml ``` ### Provide a pre-existing custom root CA certificate to the Operator You can also provide a custom root CA certificate to the Operator. In this case the Operator will not generate one itself, but will use the user-provided CA certificate. This can be useful if you would like to have several database clusters with certificates generated by the Operator based on the same root CA. To make the Operator use a custom root certificate, create a separate secret with this certificate and specify this secret in the Custom Resource options **before** you deploy a cluster. For example, if you have files named `my_tls.key` and `my_tls.crt` stored on your local machine, you could run the following command to create a Secret named `cluster1-ca-cert` in the `postgres-operator` namespace: ```bash kubectl create secret generic -n postgres-operator cluster1-ca-cert \ --from-file=tls.crt=my_tls.crt \ --from-file=tls.key=my_tls.key ``` You also need to specify details about this secret in your `deploy/cr.yaml` manifest: ```yaml ... secrets: customRootCATLSSecret: name: cluster1-ca-cert items: - key: "tls.crt" path: "root.crt" - key: "tls.key" path: "root.key" ``` Now, you can create the cluster with the `kubectl apply -f deploy/cr.yaml` command. The Operator should use the root CA certificate you had provided. !!! warning This approach allows using root CA certificate auto-generated by the Operator for some other clusters, but it needs caution. If the cluster with auto-generated certificate has `delete-ssl` finalizer enabled, the certificate will be deleted at the cluster deletion event even if it was manually provided to some other cluster. ### Generate custom certificates for the Operator yourself #### Understand certificate requirements To find out the certificates specifics needed for the Operator, view the certificates generated by the Operator automatically. For example, if you have a cluster deployed in some staging environment. Here's how to do it: 1. Check the secrets created by the Operator: ```bash kubectl get secrets ``` ??? example "Expected output" ```{.text .no-copy} cluster1-cluster-ca-cert Opaque 2 143m cluster1-cluster-cert Opaque 3 143m cluster1-instance1-frdm-certs Opaque 6 143m cluster1-instance1-qcqk-certs Opaque 6 143m cluster1-instance1-wq55-certs Opaque 6 143m cluster1-pgbackrest Opaque 5 143m cluster1-pgbouncer Opaque 6 143m cluster1-pguser-cluster1 Opaque 12 143m cluster1-replication-cert Opaque 3 143m ``` The Secrets of interest are `cluster1-cluster-cert` for external communication and `cluster1-replication-cert` for internal communication. 2. You can examine the auto-generated CA certificate (`ca.crt`) as follows: ```bash kubectl get secret/cluster1-cluster-cert -o jsonpath='{.data.ca\.crt}' | base64 --decode | openssl x509 -text -noout ``` ??? example "Expected output" ```{.text .no-copy} Certificate: Data: Version: 3 (0x2) Serial Number: ec:f3:d6:f5:35:5c:97:0c:66:cc:90:ed:e6:4b:0a:07 Signature Algorithm: ecdsa-with-SHA384 Issuer: CN = postgres-operator-ca Validity Not Before: Dec 24 13:58:21 2023 GMT Not After : Dec 21 14:58:21 2033 GMT Subject: CN = postgres-operator-ca Subject Public Key Info: ... ... ``` 3. You can check the auto-generated TLS certificate (`tls.crt`) in a similar way: === "External communication" ```bash kubectl get secret/cluster1-cluster-cert -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -text -noout ``` ??? example "Expected output" ``` {.text .no-copy} Certificate: Data: Version: 3 (0x2) Serial Number: 43:ac:81:65:4e:c6:1b:15:db:ca:36:c4:16:96:79:1b Signature Algorithm: ecdsa-with-SHA384 Issuer: CN=postgres-operator-ca Validity Not Before: Jul 22 08:15:42 2025 GMT Not After : Jul 22 09:15:42 2026 GMT Subject: CN=cluster1-primary.default.svc.cluster.local. Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:cd:06:b5:27:67:64:2b:a3:9e:84:e6:31:81:7f: 3f:a9:ae:c9:da:bd:b8:76:3e:f0:09:bd:b8:eb:03: 88:c2:d3:4b:2a:1f:e9:5b:97:cf:4e:7b:b3:12:2b: 47:ee:a6:24:fb:29:ae:01:74:e2:4c:5c:3e:f9:8d: cb:ff:0a:62:8d ASN1 OID: prime256v1 NIST CURVE: P-256 X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Basic Constraints: critical CA:FALSE X509v3 Authority Key Identifier: 59:98:FE:88:1B:54:A0:7D:DD:20:A0:F6:29:08:05:C7:18:38:7C:92 X509v3 Subject Alternative Name: DNS:cluster1-primary.default.svc.cluster.local., DNS:cluster1-primary.default.svc, DNS:cluster1-primary.default, DNS:cluster1-primary, DNS:cluster1-replicas.default.svc.cluster.local., DNS:cluster1-replicas.default.svc, DNS:cluster1-replicas.default, DNS:cluster1-replicas Signature Algorithm: ecdsa-with-SHA384 ... ``` === "Internal communication" ```bash kubectl get secret/cluster1-replication-cert -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -text -noout ``` ??? example "Expected output" ``` {.text .no-copy} Certificate: Data: Version: 3 (0x2) Serial Number: 31:1b:1e:ca:06:e6:98:4d:7e:de:6d:1b:68:d8:53:0e Signature Algorithm: ecdsa-with-SHA384 Issuer: CN=postgres-operator-ca Validity Not Before: Jul 22 08:15:42 2025 GMT Not After : Jul 22 09:15:42 2026 GMT Subject: CN=_crunchyrepl Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:b1:f7:9d:cd:33:0d:a5:19:a3:f2:fd:f6:b3:cd: e1:a5:e4:19:11:ec:18:db:fe:9c:a8:7e:eb:d2:27: 59:d1:ef:3b:09:24:58:21:6a:54:60:30:1c:be:b0: 7a:39:c5:91:6f:01:ee:d1:0b:23:86:0c:16:cf:fc: 7d:7e:39:cb:0e ASN1 OID: prime256v1 NIST CURVE: P-256 X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Basic Constraints: critical CA:FALSE X509v3 Authority Key Identifier: 59:98:FE:88:1B:54:A0:7D:DD:20:A0:F6:29:08:05:C7:18:38:7C:92 X509v3 Subject Alternative Name: DNS:_crunchyrepl Signature Algorithm: ecdsa-with-SHA384 ... ``` Both secrets share the same `ca.crt` certificate but have different `tls.crt` certificates. The `tls.crt` in the Secret for external communications should have a Common Name (CN) setting that matches the primary Service name (`CN = cluster1-primary.default.svc.cluster.local.` in the above example). Similarly, the `tls.crt` in the Secret for internal communications should have a Common Name (CN) setting that matches the preset replication user: `CN=_crunchyrepl`. #### Generate certificates One of the options to create certificates yourself is to use [CloudFlare PKI and TLS toolkit :octicons-link-external-16:](https://cfssl.org/). **You must generate certificates twice: one set is for external communications, and another set is for internal ones!** Let's say that your cluster name is `cluster1` and the desired namespace is `postgres-operator`. The commands to generate certificates may look as follows: 1. Set cluster context ```bash export CLUSTER_NAME=cluster1 export NAMESPACE=postgres-operator ``` 2. Generate the root CA certificate: ```bash cat < ca-config.json { "signing": { "default": { "expiry": "87600h", "usages": ["digital signature", "key encipherment", "content commitment"] } } } EOF ``` Explanation of the values: * `expiry` - sets the lifetime for the certificates * `usages` specifies what the certificate is valid for: * `digital signature`: for signing data * `key encipherment`: for secure key exchange * `content commitment`: ensures data integrity 4. Generate the custom TLS certificates for external communication and sign them using the previously created CA certificate. These certificates have the Common Name (CN) `cluster1-primary.postgres-operator.svc.cluster.local` ```bash cat < cluster1-cert.yaml ``` 3. Create a YAML manifest for the `cluster1-replication-cert` Secret. Run the following command to generate a YAML manifest (adjust file paths if needed): ```bash kubectl create secret generic cluster1-replication-cert \ --from-file=tls.crt=replica.pem \ --from-file=tls.key=replica-key.pem \ --from-file=ca.crt=ca.pem \ -n "$NAMESPACE" \ --dry-run=client -o yaml > cluster1-replication-cert.yaml ``` 4. Apply the manifests to update the Secrets: ```bash kubectl apply -f cluster1-cert.yaml -f cluster1-replication-cert.yaml -n "$NAMESPACE" ``` If you create new Secrets with new names and values, update the `spec.customTLSSecret` and `spec.customReplicationTLSSecret` fields in the `deploy/cr.yaml`. When you apply the new configuration,this causes the Operator to restart the cluster. ### Update a custom root CA certificate Here's what you need to know if you wish to update a custom root CA certificate: * If you change a root CA certificate, you must also change your custom TLS certificates for external and internal communications as these must be signed with the same root CA. * The new root CA and associated certs must be stored in new Secrets (not overwriting existing ones). This ensures rollback capability in case of misconfiguration or validation issues. * You must [pause the cluster](pause.md) before applying changes. This prevents the Operator from restarting or reconfiguring Pods mid-update. To update a custom root CA certificate, do the following: 1. Generate a new root CA certificate and key. For example, you have them in files named `new-ca.pem` and `new-ca-key.pem`. 2. Generate all dependent certificates for external and internal communication and sign them using the new root CA certificate. Check the [Generate certificates manually](#generate-certificates-manually) section for the steps. For example, you end up with the following certificates: * `server.pem` and `server-key.pem` for external communication * `replication.pem` and `replication-key.pem` for internal communication 3. Create a new Secret object for the new root CA certificate and define the new CA certificate and key within. Let's name it `cluster1-ca-cert-new`. ```bash kubectl create secret generic -n postgres-operator cluster1-ca-cert-new \ --from-file=ca.crt=new-ca.pem \ --from-file=ca.key=new-ca-key.pem ``` 4. Create new Secrets for external and internal communications, named `cluster1-tls` and `cluster1-replication-tls` respectively ```bash kubectl create secret generic -n postgres-operator cluster1-tls \ --from-file=ca.crt=ca.pem \ --from-file=tls.key=server-key.pem \ --from-file=tls.crt=server.pem ``` ```bash kubectl create secret generic -n postgres-operator cluster1-replication-tls \ --from-file=ca.crt=ca.pem \ --from-file=tls.key=replication-key.pem \ --from-file=tls.crt=replication.pem ``` 5. Pause the cluster to prevent the Operator to restart the Pods mid-update. ```bash kubectl patch pg cluster1 \ --type merge \ --patch '{"spec": {"pause": true}}' \ --namespace postgres-operator ``` 6. Specify details about new custom certificates in the `deploy/cr.yaml`. Since this is a provisioned cluster, apply the patch as follows: ```bash kubectl patch pg cluster1 \ --type merge \ --patch '{ "spec": { "secrets": { "customRootCATLSSecret": { "name": "cluster1-ca-cert-new", "items": [ { "key": "ca.crt", "path": "root.crt" }, { "key": "ca.key", "path": "root.key" } ] }, "customTLSSecret": { "name": "cluster1-tls" }, "customReplicationTLSSecret": { "name": "cluster1-replication-tls" } } } }' \ --namespace postgres-operator ``` 7. Unpause the cluster to resume the Operator control: ```bash kubectl patch pg cluster1 \ --type merge \ --patch '{"spec": {"pause": false}}' \ --namespace postgres-operator ``` ## Keep certificates after deleting the cluster In case of cluster deletion, objects, created for SSL (Secret, certificate, and issuer) are not deleted by default. If the user wants the cleanup of objects created for SSL, there is a [finalizers.percona.com/delete-ssl](operator.md#finalizers-delete-ssl) Custom Resource option, which can be set in `deploy/cr.yaml`: if this finalizer is set, the Operator will delete Secret, certificate and issuer after the cluster deletion event.