Adding Cassandra nodes

This document describes how to add three new Cassandra nodes to an existing Edge for Private Cloud installation.

While you can add one or two Cassandra nodes to an existing Edge installation, Apigee recommends that you add three nodes at a time.

For a list of the system requirements for a Cassandra node, see Installation requirements.

Existing Edge configuration

All the supported Edge topologies for a production system specify to use three Cassandra nodes. The three nodes are specified to the CASS_HOSTS property in the config file as shown below:

IP1=10.10.0.1
IP2=10.10.0.2
IP3=10.10.0.3
HOSTIP=$(hostname -i)
ADMIN_EMAIL=opdk@google.com
APIGEE_ADMINPW=Secret123
LICENSE_FILE=/tmp/license.txt
MSIP=$IP1 
USE_LDAP_REMOTE_HOST=n
LDAP_TYPE=1
APIGEE_LDAPPW=secret
MP_POD=gateway
REGION=dc-1
ZK_HOSTS="$IP1 $IP2 $IP3"
ZK_CLIENT_HOSTS="$IP1 $IP2 $IP3"
# Must use IP addresses for CASS_HOSTS, not DNS names.
CASS_HOSTS="$IP1:1,1 $IP2:1,1 $IP3:1,1" 
SKIP_SMTP=n
SMTPHOST=smtp.example.com
SMTPUSER=smtp@example.com
SMTPPASSWORD=smtppwd

Note that the REGION property specifies the region name as "dc-1". You need that information when adding the new Cassandra nodes.

Modifying the config file to add the three new Cassandra nodes

In this example, the three new Cassandra nodes are at the following IP addresses:

10.10.0.14
10.10.0.15
10.10.0.16

You must first update Edge configuration file to add the new nodes:

IP1=10.10.0.1
IP2=10.10.0.2
IP3=10.10.0.3
# Add the new node IP addresses.
IP14=10.10.0.14
IP15=10.10.0.15
IP16=10.10.0.16
HOSTIP=$(hostname -i)
ADMIN_EMAIL=opdk@google.com
...
# Update CASS_HOSTS to add each new node after an existing nodes. 
# Must use IP addresses for CASS_HOSTS, not DNS names.
CASS_HOSTS="$IP1:1,1 $IP14:1,1 $IP2:1,1 $IP15:1,1 $IP3:1,1 $IP16:1,1"

Note: Add each new Cassandra node to CASS_HOSTS after an existing node.

This ensure that the existing nodes retain their initial token settings, and the initial token of each new node is between the token values of the existing nodes.

Configure Edge

After editing the config file, you must:

Reconfigure the existing Cassandra nodes
Install Cassandra on the new nodes
Reconfigure the Management Server

Reconfigure the existing Cassandra nodes

On the existing Cassandra nodes:

Rerun the setup.sh with the "-p c" profile and the new config file:
```
/opt/apigee/apigee-setup/bin/setup.sh -p c -f updatedConfigFile
```

Install Cassandra on the new nodes

Use the procedure below to install Cassandra on the new nodes.

Note: Cassandra uses the fixed data-center names dc-1, dc-2, and so on, irrespective of the region names specified in the REGION property of the configuration file for the regions in which you install the new nodes. In step 2 of the following procedure, you must use Cassandra's data center names (dc-1, dc-2, etc.) in the commands.

On each new Cassandra node:

Install Cassandra on the three nodes:
1. Install apigee-setup on the first node as described in Install the Edge apigee-setup utility.
2. Install Cassandra on the first node by using the updated config file:
```
/opt/apigee/apigee-setup/bin/setup.sh -p c -f updatedConfigFile
```
3. Repeat these two steps for the remaining new Cassandra nodes.
Rebuild the three new Cassandra nodes, specifying the region name to be the data center in which you are adding the node (dc-1, dc-2, and so on). In this example, it is dc-1:
1. On the first node, run:
```
/opt/apigee/apigee-cassandra/bin/nodetool [-u username -pw password] -h nodeIP rebuild dc-1
```
  Where nodeIP is the IP address of the Cassandra node.
  
  You only need to pass your username and password if you enabled JMX authentication for Cassandra.
2. Repeat this step on the remaining new Cassandra nodes.

Reconfigure the Management Server

On a Management-Server node

Rerun setup.sh to update the Management Server for the newly added Cassandra nodes:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ms -f updatedConfigFile
```

Restart all Routers and Message Processors

On all Routers:

/opt/apigee/apigee-service/bin/apigee-service edge-router restart

On all Message Processors:

/opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart

Free disk space on existing Cassandra nodes

After you add a new node, you can use the nodetool cleanup command on the pre-existing nodes to free up disk space. This command clears configuration tokens that are no longer owned by the pre-existing Cassandra node.

To free up disk space on pre-existing Cassandra nodes after adding a new node, execute the following command:

/opt/apigee/apigee-cassandra/bin/nodetool [-u username -pw password] -h cassandraIP cleanup

You only need to pass your username and password if you enabled JMX authentication for Cassandra.

Verify rebuild

Use the following commands to verify that the rebuild was successful:

nodetool [-u username -pw password] -h nodeIP netstats

This command should indicate MODE: Normal when the node is up and the indexes are built.

nodetool [-u username -pw password] -h nodeIP statusthrift

Should indicate that the thrift server is running, which allows Cassandra to accept new client requests.

nodetool [-u username -pw password] -h nodeIP statusbinary

Should indicate that the native transport (or binary protocol) is running.

nodetool [-u username -pw password] -h nodeIP describecluster

Should show new nodes are using the same schema version as the older nodes.

For more information on using nodetool, see the nodetool usage documentation.