Enable Cassandra internode encryption

Internode (or node-to-node) encryption protects data traveling between nodes in a cluster using TLS. This page explains how to enable Cassandra internode encryption using TLS on Edge for Private Cloud. To perform these steps, you must be familiar with the details of your Cassandra ring.

Enable Cassandra internode encryption

Follow these steps to enable Cassandra internode encryption:

  1. Generate server certificates by following the steps in the Appendix to create a self-signed key and certificate.

    If you are using Edge for Private Cloud on a FIPS-enabled operating system, use a BouncyCastle FIPS Keystore (BCFKS). Refer to the Appendix below for tips on working with BCFKS type keystore.

    The following steps assume you have created keystore.node0 and truststore.node0, as well as the keystore and truststore passwords, as explained in the Appendix. The keystore and truststore should be created as preliminary steps on each node before proceeding with next steps.

  2. Add the following properties to the /opt/apigee/customer/application/cassandra.properties file. If the file does not exist, create it. 
    conf_cassandra_server_encryption_internode_encryption=all
conf_cassandra_server_encryption_keystore=/opt/apigee/data/apigee-cassandra/keystore.node0
conf_cassandra_server_encryption_keystore_password=keypass
conf_cassandra_server_encryption_truststore=/opt/apigee/data/apigee-cassandra/truststore.node0
conf_cassandra_server_encryption_truststore_password=trustpass
    
# Optionally set the following to enable 2-way TLS or mutual TLS
conf_cassandra_server_encryption_require_client_auth=true
  
# Set the following in FIPS enabled operating systems
# With FIPS, older TLS protocols are disabled, so set to TLSv1.2
conf_cassandra_server_encryption_protocol=TLSv1.2
# With FIPS, use BCFKS keystores
conf_cassandra_server_encryption_store_type=BCFKS
  3. Ensure that the file cassandra.properties is owned by the apigee user: 
    chown apigee:apigee \
/opt/apigee/customer/application/cassandra.properties

Execute the following steps on each Cassandra node, one at a time, so the changes take effect without causing any downtime for users:

  1. Stop the Cassandra service: 
    /opt/apigee/apigee-service/bin/apigee-service \
apigee-cassandra stop
  2. Restart the Cassandra service: 
    /opt/apigee/apigee-service/bin/apigee-service \
apigee-cassandra start
  3. To determine if the TLS encryption service has started, check the system logs for the following message: 
    Internode messaging enabled TLS protocols
Internode messaging enabled cipher suites

Perform certificate rotation

To rotate certificates, follow these steps:

  1. Add the certificate for each unique generated key pair (see Appendix) to an existing Cassandra node's truststore, such that both the old certificates and new certificates exist in the same truststore: 
    keytool -import -v -trustcacerts -alias NEW_ALIAS \
-file CERT -keystore EXISTING_TRUSTSTORE

    where NEW_ALIAS is a unique string to identify the entry, CERT is the name of the certificate file to add, and EXISTING_TRUSTSTORE is the name of the existing truststore on the Cassandra node.

  2. Use a copy utility, such as scp, to distribute the truststore to all Cassandra nodes in the cluster, replacing the existing truststore in use by each node.
  3. Perform a rolling restart of the cluster to load the new truststore and establish trust for the new keys before they are in place: 
    /opt/apigee/apigee-service/bin/apigee-service \
apigee-cassandra restart
  4. On each Cassandra node in the cluster, update the properties shown below to the new keystore values in the cassandra.properties file: 
    conf_cassandra_server_encryption_keystore=NEW_KEYSTORE_PATH
conf_cassandra_server_encryption_keystore_password=NEW_KEYSTORE_PASSWORD

    where NEW_KEYSTORE_PATH is the path to the directory where the keystore file is located and NEW_KEYSTORE_PASSWORD is the keystore password set when the certificates were created, as explained in the Appendix.

  5. Stop the Cassandra service: 
    /opt/apigee/apigee-service/bin/apigee-service \
apigee-cassandra stop
  6. Restart the Cassandra service: 
    /opt/apigee/apigee-service/bin/apigee-service \
apigee-cassandra start
  7. When communication is successfully established between all nodes, proceed to the next Cassandra node. Note: Only proceed to the next node if communication is successfully established between all nodes.

Appendix

The following example explains how to prepare server certificates needed to perform the internode encryption steps. The commands shown in the example use the following parameters:

Parameter Description
node0 Any unique string to identify the node.
keystore.node0 A keystore name. The commands assume this file is in the current directory.
keypass The keypass must be the same for both the keystore and the key.
dname Identifies the IP address of node0 as 10.128.0.39.
-validity The value set on this flag makes the generated key pair valid for 10 years.
  1. Go to the following directory: 
    cd /opt/apigee/data/apigee-cassandra
  2. Run the following command to generate a file named keystore.node0 in the current directory: 
    keytool -genkey -keyalg RSA -alias node0 -validity 3650 \
-keystore keystore.node0 -storepass keypass \
-keypass keypass -dname "CN=10.128.0.39, OU=None, \
O=None, L=None, C=None"

    Important: Make sure that the key password is the same as the keystore password.

  3. Export the certificate to a separate file: 
    keytool -export -alias node0 -file node0.cer \
-keystore keystore.node0
  4. Ensure the file is readable by the apigee user only and by no one else: 
    $ chown apigee:apigee \
/opt/apigee/data/apigee-cassandra/keystore.node0
$ chmod 400 /opt/apigee/data/apigee-cassandra/keystore.node0
  5. Import the generated certificate node0.cer to the truststore of the node: 
    keytool -import -v -trustcacerts -alias node0 \
-file node0.cer -keystore truststore.node0

    The command above asks you to set a password. This is the truststore password and can be different from the keystore password you set earlier. If prompted to trust the certificate, enter yes.

  6. Use openssl to generate a PEM file of the certificate with no keys. Note that cqlsh does not work with the certificate in the format generated. 
    $ keytool -importkeystore -srckeystore keystore.node0 \
-destkeystore node0.p12 -deststoretype PKCS12 -srcstorepass \
keypass -deststorepass keypass
$ openssl pkcs12 -in node0.p12 -nokeys -out node0.cer.pem \
-passin pass:keypass
$ openssl pkcs12 -in node0.p12 -nodes -nocerts -out node0.key.pem -passin pass:keypass
  7. For node-to-node encryption, copy the node0.cer file to each node and import it to the truststore of each node. 
    keytool -import -v -trustcacerts -alias node0 \
-file node0.cer -keystore truststore.node1
  8. Use keytool -list to check for certificates in the keystore and truststore files: 
    $ keytool -list -keystore keystore.node0
$ keytool -list -keystore truststore.node0

Working with BCFKS keystores for FIPS-enabled operating systems

When working with FIPS enabled operating systems, use BouncyCastle FIPS Keystores (BCFKS). The section below describes how to work with BCFKS type keystores. The other steps for working with keystores (like file ownership, file location, and others) remain the same as mentioned in this appendix.

To generate a keystore of type BCFKS, use the command below:

keytool -genkeypair -keyalg RSA -alias node0 -validity 365 -keystore keystore.node0 \
-storepass keypass -keypass keypass -v \
-dname "EMAILADDRESS=youremail@domain.com, CN=yourcn, OU=yourou, O=youro, L=yourl, C=yourc" \
-storetype BCFKS -providerpath /opt/apigee/edge-gateway/lib/thirdparty/bc-fips-1.0.2.4.jar \
-providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providername BCFIPS

Note: The keystore password and key password should be the same.

Note: The bc-fips jar shipped by Apigee can be used, or the same jar can be downloaded from BouncyCastle's repositories. Refer to BouncyCastle's documentation for more details on how to generate a BCFKS keystore.

Export the certificate to a separate file:

keytool -export -v -alias node0 -file node0.cer -keystore keystore.node0 -storepass keypass \
-storetype BCFKS -providerpath /opt/apigee/edge-gateway/lib/thirdparty/bc-fips-1.0.2.4.jar \
-providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providername BCFIPS

Import the generated certificate node0.cer to the truststore of the node:

keytool -import -v -alias node0 -file node0.cer -keystore truststore.node0 -storepass storepass \
-storetype BCFKS -providerpath /opt/apigee/edge-gateway/lib/thirdparty/bc-fips-1.0.2.4.jar \
-providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providername BCFIPS