Enable Cassandra internode encryption

Internode (or node-to-node) encryption protects data traveling between nodes in a cluster using TLS. This page explains how to enable Cassandra internode encryption using TLS on Edge for Private Cloud. To perform these steps, you must be familiar with the details of your Cassandra ring.

Enable Cassandra internode encryption

To enable Cassandra internode encryption, follow the procedure below on all nodes in the cluster. You need to distribute public certificates of each node to all nodes. After doing so, each node will contain certificates node0.cer, node1.cer etc. in its truststore. Each node will only contain its own private key in its keystore. For example, node0 will only contain node0.pem in its keystore. You need to enable encryption on each node, one at a time.

Follow these steps to enable Cassandra internode encryption:

  1. Generate server certificates by following the steps in the Appendix to create a self-signed key and certificate.

    The following steps assume you have created keystore.node0 and truststore.node0, as well as the keystore and truststore passwords, as explained in the Appendix. The keystore and truststore should be created as preliminary steps on each node before proceeding with next steps.

  2. Add the following properties to the /opt/apigee/customer/application/cassandra.properties file. If the file does not exist, create it. 
    conf_cassandra_internode_encryption=all
conf_cassandra_keystore=/opt/apigee/data/apigee-cassandra/keystore.node0
conf_cassandra_keystore_password=keypass
conf_cassandra_truststore=/opt/apigee/data/apigee-cassandra/truststore.node0
conf_cassandra_truststore_password=trustpass
# Optionally set the following to enable 2-way TLS or mutual TLS
# conf_cassandra_require_client_auth=true
  3. Ensure that the file cassandra.properties is owned by the apigee user: 
    chown apigee:apigee \
/opt/apigee/customer/application/cassandra.properties

Execute the following steps on each Cassandra node, one at a time, so the changes take effect without causing any downtime for users:

  1. Stop the Cassandra service: 
    /opt/apigee/apigee-service/bin/apigee-service \
apigee-cassandra stop
  2. Restart the Cassandra service: 
    /opt/apigee/apigee-service/bin/apigee-service \
apigee-cassandra start
  3. To determine if the TLS encryption service has started, check the system logs for the following message: 
    Starting Encrypted Messaging Service on TLS port

Perform certificate rotation

To rotate certificates, follow these steps:

  1. Add the certificate for each unique generated key pair (see Appendix) to an existing Cassandra node's truststore, such that both the old certificates and new certificates exist in the same truststore: 
    keytool -import -v -trustcacerts -alias NEW_ALIAS \
-file CERT -keystore EXISTING_TRUSTSTORE

    where NEW_ALIAS is a unique string to identify the entry, CERT is the name of the certificate file to add, and EXISTING_TRUSTSTORE is the name of the existing truststore on the Cassandra node.

  2. Use a copy utility, such as scp, to distribute the truststore to all Cassandra nodes in the cluster replacing the existing truststore in use by each node.
  3. Perform a rolling restart of the cluster to load the new truststore and establish trust for the new keys before they are in place: 
    /opt/apigee/apigee-service/bin/apigee-service \
apigee-cassandra restart
  4. On each Cassandra node in the cluster, update the properties shown below to the new keystore values in the cassandra.properties file: 
    conf_cassandra_keystore=NEW_KEYSTORE_PATH
conf_cassandra_keystore_password=NEW_KEYSTORE_PASSOWRD

  
    where NEW_KEYSTORE_PATH is the path to the directory where the keystore file is
  located and NEW_KEYSTORE_PASSWORD is the keystore password set when the certificates
  were created, as explained in the Appendix.
  5. Stop the Cassandra service: 
    /opt/apigee/apigee-service/bin/apigee-service \
apigee-cassandra stop
  6. Restart the Cassandra service: 
    /opt/apigee/apigee-service/bin/apigee-service \
apigee-cassandra start
  7. When communication is successfully established between all nodes, proceed to the next Cassandra node. Note: Only proceed to the next node if communication is successfully established between all nodes.

Appendix

The following example explains how to prepare server certificates needed to perform the internode encryption steps. The commands shown in the example use the following parameters:

Parameter Description
node0 Any unique string to identify the node.
keystore.node0 A keystore name. The commands assume this file is in the current directory.
keypass The keypass must be the same for both the keystore and the key.
dname Identifies the IP address of node0 as 10.128.0.39.
-validity The value set on this flag makes the generated key pair valid for 10 years.
  1. Go to the following directory: 
    cd /opt/apigee/data/apigee-cassandra
  2. Run the following command to generate a file named keystore.node0 in the current directory: 
    keytool -genkey -keyalg RSA -alias node0 -validity 3650 \
-keystore keystore.node0 -storepass keypass \
-keypass keypass -dname "CN=10.128.0.39, OU=None, \
O=None, L=None, C=None"

    Important: Make sure that the key password is the same as the keystore password.

  3. Export the certificate to a separate file: 
    keytool -export -alias node0 -file node0.cer \
-keystore keystore.node0
  4. Ensure the file is readable by the apigee user only and by no one else: 
    $ chown apigee:apigee \
/opt/apigee/data/apigee-cassandra/keystore.node0
$ chmod 400 /opt/apigee/data/apigee-cassandra/keystore.node0
  5. Import the generated certificate node0.cer to the truststore of the node: 
    keytool -import -v -trustcacerts -alias node0 \
-file node0.cer -keystore truststore.node0

    The command above asks you to set a password. This is the truststore password and can be different from the keystore password you set earlier. If prompted to trust the certificate, enter yes.

  6. Use openssl to generate a PEM file of the certificate with no keys. Note that cqlsh does not work with the certificate in the format generated. 
    $ keytool -importkeystore -srckeystore keystore.node0 \
-destkeystore node0.p12 -deststoretype PKCS12 -srcstorepass \
keypass -deststorepass keypass
$ openssl pkcs12 -in node0.p12 -nokeys -out node0.cer.pem \
-passin pass:keypass
$ openssl pkcs12 -in node0.p12 -nodes -nocerts -out node0.key.pem -passin pass:keypass
  7. For node-to-node encryption, copy the node0.cer file to each node and import it to the truststore of each node. 
    keytool -import -v -trustcacerts -alias node0 \
-file node0.cer -keystore truststore.node1
  8. Use keytool -list to check for certificates in the keystore and truststore files: 
    $ keytool -list -keystore keystore.node0
$ keytool -list -keystore truststore.node0