Internode (or node-to-node) encryption protects data traveling between nodes in a cluster
using TLS. This page explains how to enable Cassandra internode encryption using TLS on Edge for
Private Cloud. To perform these steps, you must be familiar with the details of your Cassandra
ring.

## Enable Cassandra internode encryption

Follow these steps to enable Cassandra internode encryption:

1. Generate server certificates by following the steps in the [Appendix](https://docs.apigee.com/private-cloud/v4.52.02/enable-cassandra-internode-encryption#appendix)
   to create a self-signed key
   and certificate.

   The following steps assume you have created `keystore.node0`
   and `truststore.node0`,
   as well as the keystore and truststore passwords, as explained in the [Appendix](https://docs.apigee.com/private-cloud/v4.52.02/enable-cassandra-internode-encryption#appendix).
   The keystore and truststore should be created as preliminary steps on each node before proceeding
   with next steps.
2. Add the following properties to the `/opt/apigee/customer/application/cassandra.properties` file. If the file does not exist, create it.

   ```
     conf_cassandra_server_encryption_internode_encryption=all
     conf_cassandra_server_encryption_keystore=/opt/apigee/data/apigee-cassandra/keystore.node0
     conf_cassandra_server_encryption_keystore_password=keypass
     conf_cassandra_server_encryption_truststore=/opt/apigee/data/apigee-cassandra/truststore.node0
     conf_cassandra_server_encryption_truststore_password=trustpass
     # Optionally set the following to enable 2-way TLS or mutual TLS
     conf_cassandra_server_encryption_require_client_auth=true
   ```
3. Ensure that the file `cassandra.properties` is owned by the apigee user:

   ```
   chown apigee:apigee \
   /opt/apigee/customer/application/cassandra.properties
   ```

Execute the following steps on each Cassandra node, one at a time, so the changes take effect
without causing any downtime for users:

3. Stop the Cassandra service:

   ```
   /opt/apigee/apigee-service/bin/apigee-service \
   apigee-cassandra stop
   ```
4. Restart the Cassandra service:

   ```
   /opt/apigee/apigee-service/bin/apigee-service \
   apigee-cassandra start
   ```
5. To determine if the TLS encryption service has started, check the system logs for the following message:

   ```
   Starting Encrypted Messaging Service on TLS port
   ```

## Perform certificate rotation

To rotate certificates, follow these steps:

1. Add the certificate for each unique generated key pair (see [Appendix](https://docs.apigee.com/private-cloud/v4.52.02/enable-cassandra-internode-encryption#appendix)) to an existing Cassandra node's truststore, such that both the old certificates and new certificates exist in the same truststore:

   ```
   keytool -import -v -trustcacerts -alias NEW_ALIAS \
   -file CERT -keystore EXISTING_TRUSTSTORE
   ```

   where `NEW_ALIAS` is a unique string to identify the entry, `CERT` is
   the name of the certificate
   file to add, and `EXISTING_TRUSTSTORE`
   is the name of the existing truststore on the Cassandra node.
2. Use a copy utility, such as scp, to distribute the truststore to all Cassandra nodes in the cluster replacing the existing truststore in use by each node.
3. Perform a rolling restart of the cluster to load the new truststore and establish trust for the new keys before they are in place:

   ```
   /opt/apigee/apigee-service/bin/apigee-service \
   apigee-cassandra restart
   ```
4. On each Cassandra node in the cluster, update the properties shown below to the new keystore values in the cassandra.properties file:

   ```
   conf_cassandra_server_encryption_keystore=NEW_KEYSTORE_PATH
   conf_cassandra_server_encryption_keystore_password=NEW_KEYSTORE_PASSWORD
   where `NEW_KEYSTORE_PATH` is the path to the directory where the keystore file is
   located and `NEW_KEYSTORE_PASSWORD` is the keystore password set when the certificates
   were created, as explained in the https://docs.apigee.com/private-cloud/v4.52.02/enable-cassandra-internode-encryption#appendix.
   ```
5. Stop the Cassandra service:

   ```
   /opt/apigee/apigee-service/bin/apigee-service \
   apigee-cassandra stop
   ```
6. Restart the Cassandra service:

   ```
   /opt/apigee/apigee-service/bin/apigee-service \
   apigee-cassandra start
   ```
7. When communication is successfully established between all nodes, proceed to the next Cassandra node. **Note:** Only proceed to the next node if communication is successfully established between all nodes.

## Appendix

The following example explains how to prepare server certificates needed to perform the
internode encryption steps. The commands shown in the example use the following parameters:

| Parameter | Description |
|---|---|
| `node0` | Any unique string to identify the node. |
| `keystore.node0` | A keystore name. The commands assume this file is in the current directory. |
| `keypass` | The keypass must be the same for both the keystore and the key. |
| `dname` | Identifies the IP address of `node0` as `10.128.0.39`. |
| `-validity` | The value set on this flag makes the generated key pair valid for 10 years. |

1. Go to the following directory:

   ```
   cd /opt/apigee/data/apigee-cassandra
   ```
2. Run the following command to generate a file named `keystore.node0` in the current directory:

   ```
   keytool -genkey -keyalg RSA -alias node0 -validity 3650 \
   -keystore keystore.node0 -storepass keypass \
   -keypass keypass -dname "CN=10.128.0.39, OU=None, \
   O=None, L=None, C=None"
   ```

   **Important:** Make sure that the key password is the same as the keystore password.
3. Export the certificate to a separate file:

   ```
   keytool -export -alias node0 -file node0.cer \
   -keystore keystore.node0
   ```
4. Ensure the file is readable by the apigee user only and by no one else:

   ```
   $ chown apigee:apigee \
   /opt/apigee/data/apigee-cassandra/keystore.node0
   $ chmod 400 /opt/apigee/data/apigee-cassandra/keystore.node0
   ```
5. Import the generated certificate `node0.cer` to the truststore of the node:

   ```
   keytool -import -v -trustcacerts -alias node0 \
   -file node0.cer -keystore truststore.node0
   ```

   The command above asks you to set a password. This is the truststore password and can
   be different from the keystore password you set earlier. If prompted to trust the certificate,
   enter `yes`.
6. Use openssl to generate a PEM file of the certificate with no keys. Note that `cqlsh` does not work with the certificate in the format generated.

   ```
   $ keytool -importkeystore -srckeystore keystore.node0 \
   -destkeystore node0.p12 -deststoretype PKCS12 -srcstorepass \
   keypass -deststorepass keypass
   $ openssl pkcs12 -in node0.p12 -nokeys -out node0.cer.pem \
   -passin pass:keypass
   $ openssl pkcs12 -in node0.p12 -nodes -nocerts -out node0.key.pem -passin pass:keypass
   ```
7. For node-to-node encryption, copy the `node0.cer` file to each node and import it to the truststore of each node.

   ```
   keytool -import -v -trustcacerts -alias node0 \
   -file node0.cer -keystore truststore.node1
   ```
8. Use `keytool -list` to check for certificates in the keystore and truststore files:

   ```
   $ keytool -list -keystore keystore.node0
   $ keytool -list -keystore truststore.node0
   ```