Internode (or node-to-node) encryption protects data traveling between nodes in a cluster using TLS. This page explains how to enable Cassandra internode encryption using TLS on Edge for Private Cloud. To perform these steps, you must be familiar with the details of your Cassandra ring. ## Enable Cassandra internode encryption Follow these steps to enable Cassandra internode encryption: 1. Generate server certificates by following the steps in the [Appendix](https://docs.apigee.com/private-cloud/v4.53.00/enable-cassandra-internode-encryption#appendix) to create a self-signed key and certificate. If you are using Edge for Private Cloud on a FIPS-enabled operating system, use a BouncyCastle FIPS Keystore (BCFKS). Refer to the [Appendix](https://docs.apigee.com/private-cloud/v4.53.00/enable-cassandra-internode-encryption#appendix) below for tips on working with BCFKS type keystore. The following steps assume you have created `keystore.node0` and `truststore.node0`, as well as the keystore and truststore passwords, as explained in the [Appendix](https://docs.apigee.com/private-cloud/v4.53.00/enable-cassandra-internode-encryption#appendix). The keystore and truststore should be created as preliminary steps on each node before proceeding with next steps. 2. Add the following properties to the `/opt/apigee/customer/application/cassandra.properties` file. If the file does not exist, create it. ``` conf_cassandra_server_encryption_internode_encryption=all conf_cassandra_server_encryption_keystore=/opt/apigee/data/apigee-cassandra/keystore.node0 conf_cassandra_server_encryption_keystore_password=keypass conf_cassandra_server_encryption_truststore=/opt/apigee/data/apigee-cassandra/truststore.node0 conf_cassandra_server_encryption_truststore_password=trustpass # Optionally set the following to enable 2-way TLS or mutual TLS conf_cassandra_server_encryption_require_client_auth=true # Set the following in FIPS enabled operating systems # With FIPS, older TLS protocols are disabled, so set to TLSv1.2 conf_cassandra_server_encryption_protocol=TLSv1.2 # With FIPS, use BCFKS keystores conf_cassandra_server_encryption_store_type=BCFKS ``` 3. Ensure that the file `cassandra.properties` is owned by the apigee user: ``` chown apigee:apigee \ /opt/apigee/customer/application/cassandra.properties ``` Execute the following steps on each Cassandra node, one at a time, so the changes take effect without causing any downtime for users: 3. Stop the Cassandra service: ``` /opt/apigee/apigee-service/bin/apigee-service \ apigee-cassandra stop ``` 4. Restart the Cassandra service: ``` /opt/apigee/apigee-service/bin/apigee-service \ apigee-cassandra start ``` 5. To determine if the TLS encryption service has started, check the system logs for the following message: ``` Internode messaging enabled TLS protocols Internode messaging enabled cipher suites ``` ## Perform certificate rotation To rotate certificates, follow these steps: 1. Add the certificate for each unique generated key pair (see [Appendix](https://docs.apigee.com/private-cloud/v4.53.00/enable-cassandra-internode-encryption#appendix)) to an existing Cassandra node's truststore, such that both the old certificates and new certificates exist in the same truststore: ``` keytool -import -v -trustcacerts -alias NEW_ALIAS \ -file CERT -keystore EXISTING_TRUSTSTORE ``` where `NEW_ALIAS` is a unique string to identify the entry, `CERT` is the name of the certificate file to add, and `EXISTING_TRUSTSTORE` is the name of the existing truststore on the Cassandra node. 2. Use a copy utility, such as scp, to distribute the truststore to all Cassandra nodes in the cluster, replacing the existing truststore in use by each node. 3. Perform a rolling restart of the cluster to load the new truststore and establish trust for the new keys before they are in place: ``` /opt/apigee/apigee-service/bin/apigee-service \ apigee-cassandra restart ``` 4. On each Cassandra node in the cluster, update the properties shown below to the new keystore values in the cassandra.properties file: ``` conf_cassandra_server_encryption_keystore=NEW_KEYSTORE_PATH conf_cassandra_server_encryption_keystore_password=NEW_KEYSTORE_PASSWORD ``` where `NEW_KEYSTORE_PATH` is the path to the directory where the keystore file is located and `NEW_KEYSTORE_PASSWORD` is the keystore password set when the certificates were created, as explained in the [Appendix](https://docs.apigee.com/private-cloud/v4.53.00/enable-cassandra-internode-encryption#appendix). 5. Stop the Cassandra service: ``` /opt/apigee/apigee-service/bin/apigee-service \ apigee-cassandra stop ``` 6. Restart the Cassandra service: ``` /opt/apigee/apigee-service/bin/apigee-service \ apigee-cassandra start ``` 7. When communication is successfully established between all nodes, proceed to the next Cassandra node. **Note:** Only proceed to the next node if communication is successfully established between all nodes. ## Appendix The following example explains how to prepare server certificates needed to perform the internode encryption steps. The commands shown in the example use the following parameters: | Parameter | Description | |---|---| | `node0` | Any unique string to identify the node. | | `keystore.node0` | A keystore name. The commands assume this file is in the current directory. | | `keypass` | The keypass must be the same for both the keystore and the key. | | `dname` | Identifies the IP address of `node0` as `10.128.0.39`. | | `-validity` | The value set on this flag makes the generated key pair valid for 10 years. | 1. Go to the following directory: ``` cd /opt/apigee/data/apigee-cassandra ``` 2. Run the following command to generate a file named `keystore.node0` in the current directory: ``` keytool -genkey -keyalg RSA -alias node0 -validity 3650 \ -keystore keystore.node0 -storepass keypass \ -keypass keypass -dname "CN=10.128.0.39, OU=None, \ O=None, L=None, C=None" ``` **Important:** Make sure that the key password is the same as the keystore password. 3. Export the certificate to a separate file: ``` keytool -export -alias node0 -file node0.cer \ -keystore keystore.node0 ``` 4. Ensure the file is readable by the apigee user only and by no one else: ``` $ chown apigee:apigee \ /opt/apigee/data/apigee-cassandra/keystore.node0 $ chmod 400 /opt/apigee/data/apigee-cassandra/keystore.node0 ``` 5. Import the generated certificate `node0.cer` to the truststore of the node: ``` keytool -import -v -trustcacerts -alias node0 \ -file node0.cer -keystore truststore.node0 ``` The command above asks you to set a password. This is the truststore password and can be different from the keystore password you set earlier. If prompted to trust the certificate, enter `yes`. 6. Use openssl to generate a PEM file of the certificate with no keys. Note that `cqlsh` does not work with the certificate in the format generated. ``` $ keytool -importkeystore -srckeystore keystore.node0 \ -destkeystore node0.p12 -deststoretype PKCS12 -srcstorepass \ keypass -deststorepass keypass $ openssl pkcs12 -in node0.p12 -nokeys -out node0.cer.pem \ -passin pass:keypass $ openssl pkcs12 -in node0.p12 -nodes -nocerts -out node0.key.pem -passin pass:keypass ``` 7. For node-to-node encryption, copy the `node0.cer` file to each node and import it to the truststore of each node. ``` keytool -import -v -trustcacerts -alias node0 \ -file node0.cer -keystore truststore.node1 ``` 8. Use `keytool -list` to check for certificates in the keystore and truststore files: ``` $ keytool -list -keystore keystore.node0 $ keytool -list -keystore truststore.node0 ``` ### Working with BCFKS keystores for FIPS-enabled operating systems When working with FIPS enabled operating systems, use BouncyCastle FIPS Keystores (BCFKS). The section below describes how to work with BCFKS type keystores. The other steps for working with keystores (like file ownership, file location, and others) remain the same as mentioned in this appendix. To generate a keystore of type BCFKS, use the command below: ``` keytool -genkeypair -keyalg RSA -alias node0 -validity 365 -keystore keystore.node0 \ -storepass keypass -keypass keypass -v \ -dname "EMAILADDRESS=youremail@domain.com, CN=yourcn, OU=yourou, O=youro, L=yourl, C=yourc" \ -storetype BCFKS -providerpath /opt/apigee/edge-gateway/lib/thirdparty/bc-fips-1.0.2.4.jar \ -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providername BCFIPS ``` **Note:** The keystore password and key password should be the same. **Note:** The `bc-fips` jar shipped by Apigee can be used, or the same jar can be downloaded from BouncyCastle's repositories. Refer to BouncyCastle's documentation for more details on how to generate a BCFKS keystore. Export the certificate to a separate file: ``` keytool -export -v -alias node0 -file node0.cer -keystore keystore.node0 -storepass keypass \ -storetype BCFKS -providerpath /opt/apigee/edge-gateway/lib/thirdparty/bc-fips-1.0.2.4.jar \ -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providername BCFIPS ``` Import the generated certificate `node0.cer` to the truststore of the node: ``` keytool -import -v -alias node0 -file node0.cer -keystore truststore.node0 -storepass storepass \ -storetype BCFKS -providerpath /opt/apigee/edge-gateway/lib/thirdparty/bc-fips-1.0.2.4.jar \ -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providername BCFIPS ```