Adding Cassandra nodes

Edge for Private Cloud v. 4.17.05

This document describes how to add three new Cassandra nodes to an existing Edge for Private Cloud installation.

While you can add one or two Cassandra nodes to an existing Edge installation, Apigee recommends that you add three nodes at a time.

For a list of the system requirements for a Cassandra node, see Installation Requirements.

Existing Edge configuration

All the supported Edge topologies for a production system specify to use three Cassandra nodes. The three nodes are specified to the CASS_HOSTS property in the config file as shown below:

IP1=10.10.0.1
IP2=10.10.0.2
IP3=10.10.0.3
HOSTIP=$(hostname -i)
ADMIN_EMAIL=opdk@google.com 
APIGEE_ADMINPW=Secret123
LICENSE_FILE=/tmp/license.txt 
MSIP=$IP1 
USE_LDAP_REMOTE_HOST=n 
LDAP_TYPE=1
APIGEE_LDAPPW=secret 
BIND_ON_ALL_INTERFACES=y
MP_POD=gateway
REGION=dc-1 
ZK_HOSTS="$IP1 $IP2 $IP3" 
ZK_CLIENT_HOSTS="$IP1 $IP2 $IP3" 
# Must use IP addresses for CASS_HOSTS, not DNS names.
CASS_HOSTS="$IP1:1,1 $IP2:1,1 $IP3:1,1" 
SKIP_SMTP=n
SMTPHOST=smtp.example.com 
SMTPUSER=smtp@example.com  
SMTPPASSWORD=smtppwd

Note that the REGION property specifies the region name as "dc-1". You need that information when adding the new Cassandra nodes.

Modifying the config file to add the three new Cassandra nodes

In this example, the three new Cassandra nodes are at the following IP addresses:

  • 10.10.0.14
  • 10.10.0.15
  • 10.10.0.16

You must first update Edge configuration file to add the new nodes:

IP1=10.10.0.1
IP2=10.10.0.2
IP3=10.10.0.3
# Add the new node IP addresses.
IP14=10.10.0.14
IP15=10.10.0.15
IP16=10.10.0.16
HOSTIP=$(hostname -i)
ADMIN_EMAIL=opdk@google.com 
...
# Update CASS_HOSTS to add each new node after an existing nodes. 
# Must use IP addresses for CASS_HOSTS, not DNS names.
CASS_HOSTS="$IP1:1,1 $IP14:1,1 $IP2:1,1 $IP15:1,1 $IP3:1,1 $IP16:1,1"

Important: Add each new Cassandra node to CASS_HOSTS after an existing node.

This ensure that the existing nodes retain their initial token settings, and the initial token of each new node is between the token values of the existing nodes.

Configure Edge

After editing the config file, you must:

  • Reconfigure the existing Cassandra nodes
  • Install Cassandra on the new nodes
  • Reconfigure the Management Server

Reconfigure the existing Cassandra nodes

On the existing Cassandra nodes:

  1. Rerun the setup.sh with the "-p c" profile and the new config file:
    > /opt/apigee/apigee-setup/bin/setup.sh -p c -f updatedConfigFile

Install Cassandra on the new nodes

On each new Cassandra node:

  1. Install Cassandra on the three nodes:
    1. Install apigee-setup on the first node as described in Install the Edge apigee-setup utility.
    2. Install Cassandra on the first node by using the updated config file:
      > /opt/apigee/apigee-setup/bin/setup.sh -p c -f updatedConfigFile
    3. Repeat these two steps for the remaining new Cassandra nodes.
  2. Rebuild the three new Cassandra nodes, specifying the region name set in the config file by the REGION property. In this example, it is "dc-1":
    1. On the first node, run:
      > /opt/apigee/apigee-cassandra/bin/nodetool -h nodeIP rebuild dc-1
      where nodeIP is the IP address of the Cassandra node.
    2. Repeat this step on the remaining new Cassandra nodes.

Reconfigure the Management Server

On a Management-Server node

  1. Rerun setup.sh to update the Management Server for the newly added Cassandra nodes:
    > /opt/apigee/apigee-setup/bin/setup.sh -p ms -f updatedConfigFile

Restart all Routers and Message Processors

  1. On all Routers:
    > /opt/apigee/apigee-service/bin/apigee-service edge-router restart
  2. On all Message Processors:
    > /opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart

Free memory on the existing Cassandra nodes

On the existing Cassandra nodes, run the nodetool cleanup command to free up memory:

> /opt/apigee/apigee-cassandra/bin/nodetool -h cassandraIP cleanup