Scale down Cassandra

Apigee hybrid employs a ring of Cassandra nodes as a StatefulSet. These nodes act as the datastore which provides Apigee core persistence services (CPS) for the runtime plane.

Cassandra is a resource intensive service and cannot be deployed on a pod with any other hybrid service. As a result, you might want to scale the number of Cassandra nodes in the ring down in your cluster.

The general process for scaling down a Cassandra ring is:

  1. Scale down one Cassandra node at a time with the kubectl scale statefulset/apigee-cassandra command.
  2. Remove the "down" Cassandra node from the ring with the nodetool removenode command.
  3. Repeat the previous two steps for each Cassandra node in the ring that you want to remove.

Prerequisites

Before you scale down the number of Cassandra nodes in the ring, validate if the cluster is healthy and all the nodes are up and running, as the following example shows:

kubectl exec -it my-cluster nodetool status
Datacenter: dc-1
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens  Owns (effective)  Host ID                               Rack
UN  10.24.2.26  226.8 KiB  256     34.8%             abcdefgh-1942-4e31-6381-5195b617b473  ra-1
UN  10.24.2.27  337.97 KiB  256    33.2%             ijklmnop-1974-4f64-1620-4ac59f69f4bc  ra-1
UN  10.24.1.12  334.08 KiB  256    34.8%             qrstuvwx-2005-43f4-1492-720ab3c28d65  ra-1
UN  10.24.1.13  269.58 KiB  256    33.0%             yz123456-2003-4b73-4242-6d5783c8c269  ra-1
UN  10.24.0.15  225.35 KiB  256    35.0%             78901112-2001-4f7a-4340-ce9a6528e6b5  ra-1
UN  10.24.0.16  270.02 KiB  256    29.2%             31415461-1971-44cf-0723-0aa809292fd7  ra-1

Remove a Cassandra node from the ring

This section describes how to remove a Cassandra node from the ring.

To remove a Cassandra node from the ring:

  1. Scale down a Cassandra node by using the kubectl scale statefulset/apigee-cassandra command. When using this command:

    • Set the value of replicas to one less than the current number of replicas in the ring.
    • Note that statefulset/apigee-cassandra is the name of the Cassandra ring's StatefulSet. The default is apigee-cassandra.

    For example, if your Cassandra ring has six nodes, set the value of replicas to 5, as the followinge example shows:

    kubectl scale --replicas=5 statefulset/apigee-cassandra

    The kubectl command responds with the following:

    statefulset.apps/apigee-cassandra scaled
  2. Confirm that one node is down (indicated with DN) by using the nodetool status command, as the following example shows:
    kubectl exec -it my-cluster nodetool status
    Datacenter: dc-1
    ================
    Status=Up/Down
    |/ State=Normal/Leaving/Joining/Moving
    --  Address     Load       Tokens   Owns (effective)  Host ID                               Rack
    UN  10.24.2.26  226.8 KiB  256     34.8%             abcdefgh-1942-4e31-6381-5195b617b473  ra-1
    UN  10.24.2.27  337.97 KiB  256    33.2%             ijklmnop-1974-4f64-1620-4ac59f69f4bc  ra-1
    UN  10.24.1.12  334.08 KiB  256    34.8%             qrstuvwx-2005-43f4-1492-720ab3c28d65  ra-1
    UN  10.24.1.13  269.58 KiB  256    33.0%             yz123456-2003-4b73-4242-6d5783c8c269  ra-1
    UN  10.24.0.15  225.35 KiB  256    35.0%             78901112-2001-4f7a-4340-ce9a6528e6b5  ra-1
    DN  10.24.0.16  270.02 KiB  256    29.2%             31415461-1971-44cf-0723-0aa809292fd7  ra-1

    If the command completed successfully, nodetool changed one node's status from UN (Up/Normal) to DN (Down/Normal).

  3. Remove the "down" node from the ring by using the nodetool removenode command, as the following example shows:

    kubectl exec -it my-cluster nodetool removenode 31415461-1971-44cf-0723-0aa809292fd7
  4. Verify that the node has been removed from the Cassandra ring by using the nodetool status command:
    kubectl exec -it my-cluster nodetool status
    Datacenter: dc-1
    ================
    Status=Up/Down
    |/ State=Normal/Leaving/Joining/Moving
    --  Address     Load       Tokens   Owns (effective)  Host ID                               Rack
    UN  10.24.2.26  226.8 KiB  256     34.8%             abcdefgh-1942-4e31-6381-5195b617b473  ra-1
    UN  10.24.2.27  337.97 KiB  256    33.2%             ijklmnop-1974-4f64-1620-4ac59f69f4bc  ra-1
    UN  10.24.1.12  334.08 KiB  256    34.8%             qrstuvwx-2005-43f4-1492-720ab3c28d65  ra-1
    UN  10.24.1.13  269.58 KiB  256    33.0%             yz123456-2003-4b73-4242-6d5783c8c269  ra-1
    UN  10.24.0.15  225.35 KiB  256    35.0%             78901112-2001-4f7a-4340-ce9a6528e6b5  ra-1

    Notice that there are now only five nodes in the ring. The sixth node, with host ID beginning "31415461", has been removed.

  5. Repeat this process for each Cassandra node that you want to remove from the ring, one at a time.