When adding Cassandra nodes to a cluster, it is essential to consider the following two key points:

- The existing positions of nodes in the Cassandra ring should not change to minimize streaming and maintain a balanced ring.
- The number of nodes in all data centers must remain consistent.

To ensure the first objective, it is crucial to double the number of nodes in the Cassandra cluster each time you add new nodes.

For example, if you start with a standard [12-node cluster installation](https://docs.apigee.com/private-cloud/v4.53.01/installation-topologies#12hostclusteredinstallation) topology distributed across two data centers, you will have a total of six Cassandra nodes---three in each data center. To expand this cluster, you should add three nodes to each data center, increasing the total node count to 12 (six nodes in each data center). If further expansion is required, you should add six additional nodes to each data center, resulting in a total node count of 24 (12 nodes in each data center).

This document provides instructions for adding three new Cassandra nodes to an existing Edge for Private Cloud installation. The same steps can be followed to add additional nodes. Always ensure that when expanding your cluster, you double the number of nodes.

For a list of the system requirements for a Cassandra node, refer to the [Installation Requirements](https://docs.apigee.com/private-cloud/v4.53.01/installation-requirements) section.

## Existing Edge configuration

All the supported Edge topologies for a production system specify to use three Cassandra
nodes. The three nodes are specified to the `CASS_HOSTS` property in the config file
as shown below:

```javascript
IP1=10.10.0.1
IP2=10.10.0.2
IP3=10.10.0.3
HOSTIP=$(hostname -i)
ADMIN_EMAIL=opdk@google.com
APIGEE_ADMINPW=Secret123
LICENSE_FILE=/tmp/license.txt
MSIP=$IP1 
USE_LDAP_REMOTE_HOST=n
LDAP_TYPE=1
APIGEE_LDAPPW=secret
MP_POD=gateway
REGION=dc-1
ZK_HOSTS="$IP1 $IP2 $IP3"
ZK_CLIENT_HOSTS="$IP1 $IP2 $IP3"
# Must use IP addresses for CASS_HOSTS, not DNS names.
CASS_HOSTS="$IP1:1,1 $IP2:1,1 $IP3:1,1" 
SKIP_SMTP=n
SMTPHOST=smtp.example.com
SMTPUSER=smtp@example.com
SMTPPASSWORD=smtppwd
```

Note that the `REGION` property specifies the region name as "dc-1". You need that
information when adding the new Cassandra nodes.

## Modifying the config file to add
the three new Cassandra nodes

In this example, the three new Cassandra nodes are at the following IP addresses:

- 10.10.0.14
- 10.10.0.15
- 10.10.0.16

You must first update Edge configuration file to add the new nodes:

```javascript
IP1=10.10.0.1
IP2=10.10.0.2
IP3=10.10.0.3
# Add the new node IP addresses.
IP14=10.10.0.14
IP15=10.10.0.15
IP16=10.10.0.16
HOSTIP=$(hostname -i)
ADMIN_EMAIL=opdk@google.com
...
# Update CASS_HOSTS to add each new node after an existing nodes. 
# Must use IP addresses for CASS_HOSTS, not DNS names.
CASS_HOSTS="$IP1:1,1 $IP14:1,1 $IP2:1,1 $IP15:1,1 $IP3:1,1 $IP16:1,1" 
```

> [!NOTE]
> **Note** : Add each new Cassandra node to `CASS_HOSTS` *after an existing node*.

This ensure that the existing nodes retain their initial token settings, and the initial token
of each new node is between the token values of the existing nodes.
**Note:** Some examples below:

A standard 12-node installation (6 Cassandra nodes):
`CASS_HOSTS="$IP1:1,1 $IP2:1,1 $IP3:1,1 $IP7:2,1 $IP8:2,1 $IP9:2,1"`

To expand the above cluster to double capacity with 12 Cassandra nodes (6 in each DC):
`CASS_HOSTS="$IP1:1,1 $IP13:1,1 $IP2:1,1 $IP14:1,1 $IP3:1,1 $IP15:1,1 $IP7:2,1 $IP16:2,1 $IP8:2,1 $IP17:2,1 $IP9:2,1 $IP18:2,1"`

To further expand the above cluster to double with 24 Cassandra nodes (12 in each DC):
`CASS_HOSTS="$IP1:1,1 $IP19:1,1 $IP13:1,1 $IP20:1,1 $IP2:1,1 $IP21:1,1 $IP14:1,1 $IP22:1,1 $IP3:1,1 $IP23:1,1 $IP15:1,1 $IP24:1,1 $IP7:2,1 $IP25:2,1 $IP16:2,1 $IP26:2,1 $IP8:2,1 $IP27:2,1 $IP17:2,1 $IP28:2,1 $IP9:2,1 $IP29:2,1 $IP18:2,1 $IP30:2,1"`

## Configure Edge

After editing the config file, you must:

- Reconfigure the existing Cassandra nodes
- Install Cassandra on the new nodes
- Reconfigure the Management Server

### Reconfigure the existing Cassandra
nodes

On the existing Cassandra nodes:

1. Rerun the setup.sh with the "-p c" profile and the new config file:

   ```
   /opt/apigee/apigee-setup/bin/setup.sh -p c -f updatedConfigFile
   ```

### Install Cassandra on the new nodes

Use the procedure below to install Cassandra on the new nodes.

> [!WARNING]
> **Note:** Cassandra uses the fixed data-center names `dc-1`, `dc-2`, and so on, *irrespective* of the region names specified in the `REGION` property of the [configuration file](https://docs.apigee.com/private-cloud/v4.53.01/edge-configuration-file-reference) for the regions in which you install the new nodes. In step 2 of the following procedure, you must use Cassandra's data center names (`dc-1`, `dc-2`, etc.) in the commands.

On each new Cassandra node:

1. Install Cassandra on the three nodes:
   1. Install `apigee-setup` on the first node as described in [Install the
      Edge apigee-setup utility](https://docs.apigee.com/private-cloud/v4.53.01/install-edge-apigee-setup-utility).
   2. Install Cassandra on the first node by using the updated config file:

      ```
      /opt/apigee/apigee-setup/bin/setup.sh -p c -f updatedConfigFile
      ```
   3. Repeat these two steps for the remaining new Cassandra nodes.
2. Rebuild the three new Cassandra nodes, specifying the region name to be the data center in which you are adding the node (`dc-1`, `dc-2`, and so on). In this example, it is `dc-1`:
   1. On the first node, run:

      ```
      /opt/apigee/apigee-cassandra/bin/nodetool [-u username -pw password] -h nodeIP rebuild dc-1
      ```

      Where <var translate="no">nodeIP</var> is the IP address of the Cassandra node.

      You only need to pass your username and password if you
      [enabled JMX authentication for Cassandra](https://docs.apigee.com/private-cloud/v4.53.01/how-monitor#cassauthenable).
   2. Repeat this step on the remaining new Cassandra nodes.

### Reconfigure the Management Server

On a Management-Server node

1. Rerun setup.sh to update the Management Server for the newly added Cassandra nodes:

   ```
   /opt/apigee/apigee-setup/bin/setup.sh -p ms -f updatedConfigFile
   ```

### Restart all Routers and Message
Processors

1. On all Routers:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-router restart
   ```
2. On all Message Processors:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart
   ```

### Free disk space on existing
Cassandra nodes

After you add a new node, you can use the `nodetool cleanup` command on the
pre-existing nodes to free up disk space. This command clears configuration tokens that are no
longer owned by the pre-existing Cassandra node.

To free up disk space on pre-existing Cassandra nodes after adding a new node, execute the
following command:

```
/opt/apigee/apigee-cassandra/bin/nodetool [-u username -pw password] -h cassandraIP cleanup
```

You only need to pass your username and password if you
[enabled JMX authentication for Cassandra](https://docs.apigee.com/private-cloud/v4.53.01/how-monitor#cassauthenable).

## Verify rebuild

Use the following commands to verify that the rebuild was successful:

```
nodetool [-u username -pw password] -h nodeIP netstats
```

This command indicates whether the native transport (or *binary protocol*) is running, which allows Cassandra to accept new client requests.

```
nodetool [-u username -pw password] -h nodeIP statusbinary
```

This command indicates whether the native transport (or *binary protocol*) is running.

```
nodetool [-u username -pw password] -h nodeIP describecluster
```

This command shows whether the new nodes are using the same schema version as the older nodes.

> [!NOTE]
> **Note:** The `statusthrift` command is no longer available in Apache Cassandra 4.x because the Thrift API has been removed, as outlined in [CASSANDRA-11115](https://issues.apache.org/jira/browse/CASSANDRA-11115). In Apigee OPDK 4.53.00, port 9160 is deprecated and no longer required. Port 9042 is now the primary client port for Cassandra connections. For additional information, refer to the [Thrift API deprecation](https://lists.apache.org/thread/69yl7nnfbvp1o40p5s3jobjxv63fr7bv) announcement.

For more information on using `nodetool`, see the
[nodetool usage documentation](https://cassandra.apache.org/doc/latest/cassandra/troubleshooting/use_nodetool.html).