> [!CAUTION] > **Caution:** This documentation covers Edge for Private Cloud 4.19.06, which is no longer supported. Google does not provide security patches, bug fixes, or technical support for this version. The current supported version is 4.54.00. See the [Edge for Private Cloud documentation](https://docs.apigee.com/private-cloud/latest/overview) for the current version, or [All versions](https://docs.apigee.com/private-cloud/versions) to compare releases. > [!WARNING] > **CAUTION!** This section is for *advanced* users only. You should only follow the instructions in this section if you are familiar with configuring Cassandra rings. This section provides general guidance for scaling operations of Cassandra by making Cassandra on Apigee Edge for Private Cloud rack aware. > [!NOTE] > **NOTE:** The instructions in this section are for new installations only. Migrating to a rack aware Cassandra configuration from an existing installation is not currently documented or supported. For more information on why making your Cassandra ring rack aware is important, see the following resources: - [Replication](http://cassandra.apache.org/doc/latest/architecture/dynamo.html#replication) (Cassandra documentation) - [Cassandra Architecture \& Replication Factor Strategy](https://www.guru99.com/cassandra-architecture.html) ## What is a rack? A *Cassandra rack* is a logical grouping of Cassandra nodes within the ring. Cassandra uses racks so that it can ensure replicas are distributed among different logical groupings. As a result, operations are sent to not just one node, but multiple nodes, each on a separate rack, providing greater fault tolerance and availability. The examples in this section use three Cassandra racks, which is the number of racks that are supported by Apigee in production topologies. The default installation of Cassandra in Apigee Edge for Private Cloud assumes a single logical rack and places all the nodes in a data center within it. Although this configuration is simple to install and manage, it is susceptible to failure if an operation fails on one of those nodes. The following image shows the default configuration of the Cassandra ring: ![1 rack with 3 nodes](https://docs.apigee.com/static/private-cloud/images/cassandra-3nodes1rack.png) **(Figure 1)** *Default configuration: All nodes on a single rack* In a more robust configuration, each node would be assigned to a separate rack and operations would also execute on replicas on each of those racks. The following image shows a 3-node ring. This image shows the order in which operations are replicated across the ring (clockwise) and highlights the fact that no two nodes are on the same rack: ![3 racks, with 1 node in each rack](https://docs.apigee.com/static/private-cloud/images/cassandra-3nodering.png) **(Figure 2)** *Rack aware configuration: Three nodes, one on each rack* In this configuration, operations are sent to a node but are also sent to replicas of that node on other racks (in clockwise order). ## Add rack awareness (with 3 nodes) All production [Installation topologies](https://docs.apigee.com/private-cloud/v4.19.06/installation-topologies) of Apigee Edge for Private Cloud have at least three Cassandra nodes, which this section refers to as "IP1", "IP2", and "IP3". By default, each of these nodes is in the same rack, "ra-1". This section describes how to assign the Cassandra nodes to separate racks so that all operations are sent to replica nodes in separate logical groupings within the ring. **To assign Cassandra nodes to different racks during installation:** 1. Before you run the installer, log into the Cassandra node and open the following [silent configuration file](https://docs.apigee.com/private-cloud/v4.19.06/edge-configuration-file-reference) for edit: ``` /opt/silent.conf ``` Create the file if it does not exist and be sure to make the "apigee" user an owner. 2. Edit the `CASS_HOSTS` property, a space-separated list of IP addresses (not DNS or hostname entries) that uses the following syntax: ``` CASS_HOSTS="IP_address:data_center_number,rack_number [...]" ``` The default value is a three node Cassandra ring with each node assigned to rack 1 and data center 1, as the following example shows: ``` CASS_HOSTS="IP1:1,1 IP2:1,1 IP3:1,1" ``` 3. Change the rack assignments so that node 2 is assigned to rack 2 and node 3 is assigned to rack 3, as the following example shows: ``` CASS_HOSTS="IP1:1,1 IP2:1,2 IP3:1,3" ``` By changing the rack assignments, you instruct Cassandra to create two additional logical groupings (racks), which then provide replicas that receive all operations received by the first node. For more information on using the `CASS_HOSTS` configuration property, see [Edge Configuration File Reference](https://docs.apigee.com/private-cloud/latest/edge-configuration-file-reference). 4. Save your changes to the configuration file and execute the following command to install Cassandra with your updated configuration: ``` /opt/apigee/apigee-setup/bin/setup.sh -p c -f path/to/silent/config ``` For example: ``` /opt/apigee/apigee-setup/bin/setup.sh -p c -f /opt/silent.conf ``` 5. Repeat this procedure for each Cassandra node in the ring, in the order in which the nodes are assigned in the `CASS_HOSTS` property. In this case, you must install Cassandra in the following order: 1. Node 1 (IP1) 2. Node 2 (IP2) 3. Node 3 (IP3) After installation, you should [Check the Cassandra configuration](https://docs.apigee.com/private-cloud/v4.19.06/rack-support#check). ## Check the Cassandra configuration After installing a rack-aware Cassandra configuration, you can check that the nodes are assigned to the different racks by using the `nodetool status` command, as the following example shows: ``` /opt/apigee/apigee-cassandra/bin/nodetool status ``` (You execute this command on one of the Cassandra nodes.) The results should look similar to the following, where the **Rack** column shows the different rack IDs for each node: ``` Datacenter: dc-1 ======================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN IP1 737 MB 256 ? 554d4498-e683-4a53-b0a5-e37a9731bc5c ra-1 UN IP2 744 MB 256 ? cf8b7abf-5c5c-4361-9c2f-59e988d52da3 ra-2 UN IP3 723 MB 256 ? 48e0384d-738f-4589-aa3a-08dc5bd5a736 ra-3 ``` If you enabled JMX authentication for Cassandra, you must also pass your username and password to `nodetool`. For more information, see [Use nodetool to manage cluster nodes](https://docs.apigee.com/private-cloud/v4.19.06/how-monitor#cassandra-nodetool). ## Install a six-node ring For additional redundancy, you can expand the Cassandra ring to six nodes. In this case, you assign two nodes to each of the three racks. This configuration requires an additional three nodes: Node 4 (IP4), Node 5 (IP5), and Node 6 (IP6). The following image shows the order in which operations are replicated across the ring (clockwise) and highlights the fact that during replication, no two adjacent nodes are on the same rack: ![3 racks, with 2 nodes in each rack](https://docs.apigee.com/static/private-cloud/images/cassandra-6nodering.png) **(Figure 3)** *6-node Cassandra ring: Two nodes on each of three racks* In this configuration, each node has two more replicas: one in each of the other two racks. For example, node 1 in rack 1 has a replica in Rack 2 and Rack 3. Operations sent to node 1 are also sent to the replicas in the other racks, in clockwise order. To expand a three-node Cassandra ring to a six-node Cassandra ring, configure the nodes in the following way in your silent configuration file: ``` CASS_HOSTS="IP1:1,1 IP4:1,3 IP2:1,2 IP5:1,1 IP3:1,3 IP6:1,2" ``` As with a three-node ring, you must install Cassandra in the same order in which the nodes appear in the `CASS_HOSTS` property: 1. Node 1 (IP1) 2. Node 4 (IP4)\* 3. Node 2 (IP2) 4. Node 5 (IP5) 5. Node 3 (IP3) 6. Node 6 (IP6) \* Make your changes in the silent configuration file *before* running the setup utility on the fourth node (the second machine in the Cassandra installation order). > [!NOTE] > **NOTE:** The ordering and placement of nodes is very important. If done incorrectly, you might end up with a ring that is imbalanced or has hotspots that do not provide the desired amount of fault tolerance. ## Expand to 12 nodes To further increase fault tolerance and availability, you can increase the number of Cassandra nodes in the ring from six to 12. This configuration requires an additional six nodes (IP7 through IP12). The following image shows the order in which operations are replicated across the ring (clockwise) and highlights the fact that during replication, no two adjacent nodes are on the same rack: ![3 racks, with 4 nodes in each rack](https://docs.apigee.com/static/private-cloud/images/cassandra-12nodering.png) **(Figure 4)** *12-node Cassandra ring: Four nodes on each of three racks* The procedure for installing a 12-node ring is similar to installing a three or six node ring: set `CASS_HOSTS` to the given values and run the installer in the specified order. To expand to a 12-node Cassandra ring, configure the nodes in the following way in your silent configuration file: ``` CASS_HOSTS="IP1:1,1 IP7:1,2 IP4:1,3 IP8:1,1 IP2:1,2 IP9:1,3 IP5:1,1 IP10:1,2 IP3:1,3 IP11:1,1 IP6:1,2 IP12:1,3" ``` As with a three- and six-node rings, you must execute the installer on nodes in the order in which the nodes appear in the configuration file: 1. Node 1 (IP1) 2. Node 7 (IP7)\* 3. Node 4 (IP4) 4. Node 8 (IP8) 5. Node 2 (IP2) 6. Node 9 (IP9) 7. Node 5 (IP5) 8. Node 10 (IP10) 9. Node 3 (IP3) 10. Node 11 (IP11) 11. Node 6 (IP6) 12. Node 12 (IP12) \* You must make these changes before installing Apigee Edge for Private Cloud on the 7th node (the second machine in the Cassandra installation order). > [!NOTE] > **NOTE:** As with the three-node and six-node rings, the ordering and placement of nodes is very important. If done incorrectly, you might end up with a ring that is imbalanced or has hotspots that do not provide the desired amount of fault tolerance.