Adding Cassandra nodes

When adding Cassandra nodes to a cluster, it is essential to consider the following two key points:

The existing positions of nodes in the Cassandra ring should not change to minimize streaming and maintain a balanced ring.
The number of nodes in all data centers must remain consistent.

To ensure the first objective, it is crucial to double the number of nodes in the Cassandra cluster each time you add new nodes.

For example, if you start with a standard 12-node cluster installation topology distributed across two data centers, you will have a total of six Cassandra nodes—three in each data center. To expand this cluster, you should add three nodes to each data center, increasing the total node count to 12 (six nodes in each data center). If further expansion is required, you should add six additional nodes to each data center, resulting in a total node count of 24 (12 nodes in each data center).

This document provides instructions for adding three new Cassandra nodes to an existing Edge for Private Cloud installation. The same steps can be followed to add additional nodes. Always ensure that when expanding your cluster, you double the number of nodes.

For a list of the system requirements for a Cassandra node, refer to the Installation Requirements section.

Existing Edge configuration

All the supported Edge topologies for a production system specify to use three Cassandra nodes. The three nodes are specified to the CASS_HOSTS property in the config file as shown below:

IP1=10.10.0.1
IP2=10.10.0.2
IP3=10.10.0.3
HOSTIP=$(hostname -i)
ADMIN_EMAIL=opdk@google.com
APIGEE_ADMINPW=Secret123
LICENSE_FILE=/tmp/license.txt
MSIP=$IP1 
USE_LDAP_REMOTE_HOST=n
LDAP_TYPE=1
APIGEE_LDAPPW=secret
MP_POD=gateway
REGION=dc-1
ZK_HOSTS="$IP1 $IP2 $IP3"
ZK_CLIENT_HOSTS="$IP1 $IP2 $IP3"
# Must use IP addresses for CASS_HOSTS, not DNS names.
CASS_HOSTS="$IP1:1,1 $IP2:1,1 $IP3:1,1" 
SKIP_SMTP=n
SMTPHOST=smtp.example.com
SMTPUSER=smtp@example.com
SMTPPASSWORD=smtppwd

Note that the REGION property specifies the region name as "dc-1". You need that information when adding the new Cassandra nodes.

Modifying the config file to add the three new Cassandra nodes

In this example, the three new Cassandra nodes are at the following IP addresses:

10.10.0.14
10.10.0.15
10.10.0.16

You must first update Edge configuration file to add the new nodes:

IP1=10.10.0.1
IP2=10.10.0.2
IP3=10.10.0.3
# Add the new node IP addresses.
IP14=10.10.0.14
IP15=10.10.0.15
IP16=10.10.0.16
HOSTIP=$(hostname -i)
ADMIN_EMAIL=opdk@google.com
...
# Update CASS_HOSTS to add each new node after an existing nodes. 
# Must use IP addresses for CASS_HOSTS, not DNS names.
CASS_HOSTS="$IP1:1,1 $IP14:1,1 $IP2:1,1 $IP15:1,1 $IP3:1,1 $IP16:1,1"

Note: Add each new Cassandra node to CASS_HOSTS after an existing node.

This ensure that the existing nodes retain their initial token settings, and the initial token of each new node is between the token values of the existing nodes.

Note: Some examples below:

A standard 12-node installation (6 Cassandra nodes):

CASS_HOSTS="$IP1:1,1 $IP2:1,1 $IP3:1,1 $IP7:2,1 $IP8:2,1 $IP9:2,1"

To expand the above cluster to double capacity with 12 Cassandra nodes (6 in each DC):

CASS_HOSTS="$IP1:1,1 $IP13:1,1 $IP2:1,1 $IP14:1,1 $IP3:1,1 $IP15:1,1 $IP7:2,1 $IP16:2,1 $IP8:2,1 $IP17:2,1 $IP9:2,1 $IP18:2,1"

To further expand the above cluster to double with 24 Cassandra nodes (12 in each DC):

CASS_HOSTS="$IP1:1,1 $IP19:1,1 $IP13:1,1 $IP20:1,1 $IP2:1,1 $IP21:1,1 $IP14:1,1 $IP22:1,1 $IP3:1,1 $IP23:1,1 $IP15:1,1 $IP24:1,1 $IP7:2,1 $IP25:2,1 $IP16:2,1 $IP26:2,1 $IP8:2,1 $IP27:2,1 $IP17:2,1 $IP28:2,1 $IP9:2,1 $IP29:2,1 $IP18:2,1 $IP30:2,1"

Configure Edge

After editing the config file, you must:

Reconfigure the existing Cassandra nodes
Install Cassandra on the new nodes
Reconfigure the Management Server

Reconfigure the existing Cassandra nodes

On the existing Cassandra nodes:

Rerun the setup.sh with the "-p c" profile and the new config file:
```
/opt/apigee/apigee-setup/bin/setup.sh -p c -f updatedConfigFile
```

Install Cassandra on the new nodes

Use the procedure below to install Cassandra on the new nodes.

Note: Cassandra uses the fixed data-center names dc-1, dc-2, and so on, irrespective of the region names specified in the REGION property of the configuration file for the regions in which you install the new nodes. In step 2 of the following procedure, you must use Cassandra's data center names (dc-1, dc-2, etc.) in the commands.

On each new Cassandra node:

Install Cassandra on the three nodes:
1. Install apigee-setup on the first node as described in Install the Edge apigee-setup utility.
2. Install Cassandra on the first node by using the updated config file:
```
/opt/apigee/apigee-setup/bin/setup.sh -p c -f updatedConfigFile
```
3. Repeat these two steps for the remaining new Cassandra nodes.
Rebuild the three new Cassandra nodes, specifying the region name to be the data center in which you are adding the node (dc-1, dc-2, and so on). In this example, it is dc-1:
1. On the first node, run:
```
/opt/apigee/apigee-cassandra/bin/nodetool [-u username -pw password] -h nodeIP rebuild dc-1
```
  Where nodeIP is the IP address of the Cassandra node.
  
  You only need to pass your username and password if you enabled JMX authentication for Cassandra.
2. Repeat this step on the remaining new Cassandra nodes.

Reconfigure the Management Server

On a Management-Server node

Rerun setup.sh to update the Management Server for the newly added Cassandra nodes:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ms -f updatedConfigFile
```

Restart all Routers and Message Processors

On all Routers:

/opt/apigee/apigee-service/bin/apigee-service edge-router restart

On all Message Processors:

/opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart

Free disk space on existing Cassandra nodes

After you add a new node, you can use the nodetool cleanup command on the pre-existing nodes to free up disk space. This command clears configuration tokens that are no longer owned by the pre-existing Cassandra node.

To free up disk space on pre-existing Cassandra nodes after adding a new node, execute the following command:

/opt/apigee/apigee-cassandra/bin/nodetool [-u username -pw password] -h cassandraIP cleanup

You only need to pass your username and password if you enabled JMX authentication for Cassandra.

Verify rebuild

Use the following commands to verify that the rebuild was successful:

nodetool [-u username -pw password] -h nodeIP netstats

This command should indicate MODE: Normal when the node is up and the indexes are built.

nodetool [-u username -pw password] -h nodeIP statusthrift

Should indicate that the thrift server is running, which allows Cassandra to accept new client requests.

nodetool [-u username -pw password] -h nodeIP statusbinary

Should indicate that the native transport (or binary protocol) is running.

nodetool [-u username -pw password] -h nodeIP describecluster

Should show new nodes are using the same schema version as the older nodes.

For more information on using nodetool, see the nodetool usage documentation.