Multi-region deployment

This topic describes a multi-region deployment for Apigee hybrid.

Use cases for a multi-region deployment topology include the following:

  • Active-Active: If you have applications deployed in multiple geographic locations and want to have low latency API response for your deployments. You have the option to deploy hybrid in multiple geographic locations nearest to your clients. For example: US West Coast, US East Coast, Europe, APAC.
  • Active-Passive: If you want to have a primary region and a failover or disaster recovery region, you can use this topology

The regions in a multi-region hybrid deployment communicate via Cassandra, as the following image shows:

Prerequisites

Before configuring hybrid for multi region, you must complete these prerequisites. Refer to the Kubernetes documentation for details.

  • Kubernetes clusters in multiple regions must be set up with different CIDR blocks.
  • Cross-region communication must be set up.
  • Cassandra ports 7000 and 7001 between Kubernetes clusters between all regions should be whitelisted.

Configure the multi-region seed host

This section describes how to expand the existing Cassandra cluster to a new region. This setup allows the new region to bootstrap the cluster and join the existing data center. Without this configuration, the multi region Kubernetes clusters would not know about each other.

  1. Run the following command to identify a seed host address for Cassandra in the current region. A seed host address allows a new regional instance to find the original cluster on the very first startup to learn the topology of the cluster. The seed host address is designated as the contact point in the cluster.
    kubectl get pods -o wide -n apigee
    
    NAME                      READY   STATUS      RESTARTS   AGE   IP          NODE                                          NOMINATED NODE
    apigee-cassandra-0        1/1     Running     0          5d    10.0.0.11   gke-k8s-dc-2-default-pool-a2206492-p55d
    apigee-cassandra-1        1/1     Running     0          5d    10.0.2.4    gke-k8s-dc-2-default-pool-e9daaab3-tjmz
    apigee-cassandra-2        1/1     Running     0          5d    10.0.3.5    gke-k8s-dc-2-default-pool-e589awq3-kjch
  2. Decide which of the IPs returned from the previous command will be the multi-region seed host.
  3. In data center 1, configure cassandra.multiRegionSeedHost and cassandra.datacenter in overrides.yaml, where multiRegionSeedHost is one of the IPs returned by the previous command:
    cassandra:
      multiRegionSeedHost: seed_host_IP
      datacenter: data_center_name
      rack: rack_name

    For example:

    cassandra:
      multiRegionSeedHost: 10.0.0.11
      datacenter: "dc-2"
      rack: "ra-1"
  4. In the new data center/region, before you install hybrid, set the same TLS certificates and credentials in overrides.yaml as you set in the first region.

Set up the new region

After you configure the seed host, you can set up the new region.

To set up the new region:

  1. Install hybrid in the new region. Be sure that the overrides-DC_name.yaml file includes the same TLS certificates that are configured in the first region, as explained in the previous section.

    The following command installs hybrid in the new region:

    apigeectl apply -v beta -f overridesDC_name.yaml

    For more information, see Installation overview.

  2. Expand all apigee keyspaces.

    The following steps expand the Cassandra data to the new data center:

    1. Open a shell in the Cassandra pod:
      kubectl.1.13 run -i --tty --restart=Never --rm --image-pull-policy=Never \
        --image gcr.io/jaibheemsenr/apigee-cassandra-client:beta bash
    2. Connect to the Cassandra server:
      cqlsh apigee-cassandra-0.apigee-cassandra.apigee.svc.cluster.local -u ddl --ssl
      Password:
      
      Connected to apigeecluster at apigee-cassandra-0.apigee-cassandra.apigee.svc.cluster.local:9042.
      [cqlsh 5.0.1 | Cassandra 3.11.3 | CQL spec 3.4.4 | Native protocol v4]
      Use HELP for help.
    3. Get the available keyspaces:
      SELECT * from system_schema.keyspaces ;
       keyspace_name              | durable_writes | replication
      ----------------------------+----------------+--------------------------------------------------------------------------------------------------------
                      system_auth |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '1', 'dc-2': '1'}
                    system_schema |           True |                                                {'class': 'org.apache.cassandra.locator.LocalStrategy'}
       cache_hybrid_test_7_hybrid |           True |                  {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3'}
         kms_hybrid_test_7_hybrid |           True |                  {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3'}
         kvm_hybrid_test_7_hybrid |           True |                  {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3'}
               system_distributed |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '1', 'dc-2': '1'}
                           system |           True |                                                {'class': 'org.apache.cassandra.locator.LocalStrategy'}
                           perses |           True |                  {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3'}
       quota_hybrid_test_7_hybrid |           True |                  {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3'}
                    system_traces |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '1', 'dc-2': '1'}
      
      (10 rows)
    4. Update/expand the apigee keyspaces:
      ALTER KEYSPACE cache_hybrid_test_7_hybrid WITH replication = {'class': 'NetworkTopologyStrategy', 'dc-1':3, 'dc-2':3};
      ALTER KEYSPACE kms_hybrid_test_7_hybrid WITH replication = {'class': 'NetworkTopologyStrategy', 'dc-1':3, 'dc-2':3};
      ALTER KEYSPACE kvm_hybrid_test_7_hybrid WITH replication = {'class': 'NetworkTopologyStrategy', 'dc-1':3, 'dc-2':3};
      ALTER KEYSPACE perses WITH replication = {'class': 'NetworkTopologyStrategy', 'dc-1':3, 'dc-2':3};
      ALTER KEYSPACE quota_hybrid_test_7_hybrid  WITH replication = {'class': 'NetworkTopologyStrategy', 'dc-1':3, 'dc-2':3};
    5. Validate the keyspace expansion:
      SELECT * from system_schema.keyspaces ;
       keyspace_name              | durable_writes | replication
      ----------------------------+----------------+--------------------------------------------------------------------------------------------------------
                      system_auth |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '1', 'dc-2': '1'}
                    system_schema |           True |                                                {'class': 'org.apache.cassandra.locator.LocalStrategy'}
       cache_hybrid_test_7_hybrid |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3', 'dc-2': '3'}
         kms_hybrid_test_7_hybrid |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3', 'dc-2': '3'}
         kvm_hybrid_test_7_hybrid |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3', 'dc-2': '3'}
               system_distributed |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '1', 'dc-2': '1'}
                           system |           True |                                                {'class': 'org.apache.cassandra.locator.LocalStrategy'}
                           perses |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3', 'dc-2': '3'}
       quota_hybrid_test_7_hybrid |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3', 'dc-2': '3'}
                    system_traces |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '1', 'dc-2': '1'}
      
      (10 rows)
      ddl@cqlsh>
  3. Run nodetool rebuild sequentially on all the nodes in new data center. This may take a few minutes to a few hours depending on the data size.
    kubectl exec apigee-cassandra-0 -n apigee  -- nodetool rebuild -- dc-1
  4. Verify the rebuild processes from the logs. Also, verify the data size using the nodetool status command:
    kubectl logs apigee-cassandra-0 -f  -n apigee

    The following example shows example log entries:

    INFO  01:42:24 rebuild from dc: dc-1, (All keyspaces), (All tokens)
    INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Executing streaming plan for Rebuild
    INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.1.45
    INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.1.45
    INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.4.36
    INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.432KiB), sending 0 files(0.000KiB)
    INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.1.45 is complete
    INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.4.36
    INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.5.22
    INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.693KiB), sending 0 files(0.000KiB)
    INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.4.36 is complete
    INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.5.22
    INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 3 files(0.720KiB), sending 0 files(0.000KiB)
    INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.5.22 is complete
    INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] All sessions completed
  5. Update the seed hosts. Remove multiRegionSeedHost: 10.0.0.11 from overrides-DC_name.yaml and reapply.
    apigeectl apply -v beta -f overrides-DC_name.yaml