Configure multiple data centers for Apigee mTLS

Apigee mTLS supports multiple data centers so that you can scale your configuration to include more complex topologies such as a 12-node clustered installation.

The installation process for mTLS on a multi-data center topology is the same as it is for simpler topologies. However, you must ensure that your installation meets the prerequisites and that you change your configuration files as described in the sections that follow.

Prerequisites

To use Apigee mTLS with multiple data centers, you must:

  • Uninstall apigee-mtls and reinstall it with the multiple data center configuration. You cannot modify an existing configuration. For more information, see Change an existing apigee-mtls configuration.
  • Open port 8302 on every host that is running mTLS.
  • Ensure that all mTLS cluster members have unique IP addresses, which are consistent for all members of the cluster.
  • When specifying configuration files, use absolute paths in your commands where ambiguity might exist.
  • Add multi-data center configuration properties, as described in Configuration files for multiple data centers.

Configuration files for multiple data centers

To use Apigee mTLS with multiple data centers, you create a separate configuration file for each data center.

In each of the configuration files:

  1. Change the value of the ALL_IP configuration property to include all host IP addresses in all regions.
  2. Ensure that the value of the REGION property is the name of the current region or data center. For example, "dc-1".
  3. Add the following properties:
    Property Description
    APIGEE_MTLS_MULTI_DC_ENABLE Whether or not you are using a multi-data center configuration. Set to "y" if you are configuring multiple data centers. Otherwise, omit or set to "n". The default is omitted.
    MTLS_LOCAL_REGION_IP A space-delimited list of all IP addresses used by the current region that you are configuring. For example, "10.0.0.1 10.0.0.2 10.0.0.3".

    For the second region in the configuration, use the MTLP_REMOTE_REGION_1_IP property.
    MTLS_REMOTE_REGION_1_NAME The name of the second region in a multi-data center configuration. For example, "dc-2".

    In the second region's configuration file, you'll use "dc-2" for REGION and "dc-1" for MTLS_REMOTE_REGION_1_NAME.
    MTLS_REMOTE_REGION_1_IP A space delimited list of all IP addresses used by the second region in a multi-data center configuration. For example, "10.0.0.4 10.0.0.5 10.0.0.6".

The following examples show the configuration files for two data centers ("dc-1" and "dc-2"). Properties that are specific to a multi-data center configuration are highlighted):

dc-1 Configuration File

ALL_IP="10.126.0.114 10.126.0.113 10.126.0.96 10.126.0.132 10.126.0.133 10.126.0.104 10.126.0.106 10.126.0.105 10.126.0.95 10.126.0.102 10.126.0.100 10.126.0.112"
LDAP_MTLS_HOSTS="10.126.0.114 10.126.0.106"
ZK_MTLS_HOSTS="10.126.0.114 10.126.0.113 10.126.0.96 10.126.0.106 10.126.0.105 10.126.0.95"
CASS_MTLS_HOSTS="10.126.0.114 10.126.0.113 10.126.0.96 10.126.0.106 10.126.0.105 10.126.0.95"
PG_MTLS_HOSTS="10.126.0.104 10.126.0.112"
RT_MTLS_HOSTS="10.126.0.113 10.126.0.96 10.126.0.105 10.126.0.95"
MS_MTLS_HOSTS="10.126.0.114 10.126.0.106"
MP_MTLS_HOSTS="10.126.0.113 10.126.0.96 10.126.0.105 10.126.0.95"
QP_MTLS_HOSTS="10.126.0.132 10.126.0.133 10.126.0.102 10.126.0.100"
ENABLE_SIDECAR_PROXY="y"
ENCRYPT_DATA="zRNQ9lhRySNTfegiLLLfIQ=="
PATH_TO_CA_CERT="/opt/consul-agent-ca.pem"
PATH_TO_CA_KEY="/opt/consul-agent-ca-key.pem"

APIGEE_MTLS_MULTI_DC_ENABLE="y"
REGION="dc-1"
MTLS_LOCAL_REGION_IP="10.126.0.114 10.126.0.113 10.126.0.96 10.126.0.132 10.126.0.133 10.126.0.104"
MTLS_REMOTE_REGION_1_NAME="dc-2"
MTLS_REMOTE_REGION_1_IP="10.126.0.106 10.126.0.105 10.126.0.95 10.126.0.102 10.126.0.100 10.126.0.112"

dc-2 Configuration File

ALL_IP="10.126.0.114 10.126.0.113 10.126.0.96 10.126.0.132 10.126.0.133 10.126.0.104 10.126.0.106 10.126.0.105 10.126.0.95 10.126.0.102 10.126.0.100 10.126.0.112"
LDAP_MTLS_HOSTS="10.126.0.114 10.126.0.106"
ZK_MTLS_HOSTS="10.126.0.114 10.126.0.113 10.126.0.96 10.126.0.106 10.126.0.105 10.126.0.95"
CASS_MTLS_HOSTS="10.126.0.114 10.126.0.113 10.126.0.96 10.126.0.106 10.126.0.105 10.126.0.95"
PG_MTLS_HOSTS="10.126.0.104 10.126.0.112"
RT_MTLS_HOSTS="10.126.0.113 10.126.0.96 10.126.0.105 10.126.0.95"
MS_MTLS_HOSTS="10.126.0.114 10.126.0.106"
MP_MTLS_HOSTS="10.126.0.113 10.126.0.96 10.126.0.105 10.126.0.95"
QP_MTLS_HOSTS="10.126.0.132 10.126.0.133 10.126.0.102 10.126.0.100"
ENABLE_SIDECAR_PROXY="y"
ENCRYPT_DATA="zRNQ9lhRySNTfegiLLLfIQ=="
PATH_TO_CA_CERT="/opt/consul-agent-ca.pem"
PATH_TO_CA_KEY="/opt/consul-agent-ca-key.pem"

APIGEE_MTLS_MULTI_DC_ENABLE="y"
REGION="dc-2"
MTLS_LOCAL_REGION_IP="10.126.0.106 10.126.0.105 10.126.0.95 10.126.0.102 10.126.0.100 10.126.0.112"
MTLS_REMOTE_REGION_1_NAME="dc-1"
MTLS_REMOTE_REGION_1_IP="10.126.0.114 10.126.0.113 10.126.0.96 10.126.0.132 10.126.0.133 10.126.0.104"

For information about the standard configuration properties, see Step 1: Update your configuration file.

Test a multi-data center configuration

The raft list-peers command displays a list of IP addresses associated with all ZooKeeper (ZK) nodes running a Consul process, as indicated by the consul agent process running on the server. These peers are not limited to a single data center but span all data centers in the configuration.

The following examples show sample output from a raft list-peers command for a 2-DC mTLS-enabled setup:

$ /opt/apigee/apigee-mtls-consul/bin/consul operator raft list-peers
Node              ID                                    Address            State     Voter  RaftProtocol
prc-test-2-20036  0f30e0fc-a33b-2cf9-91fc-3dd0c02711c7  10.126.0.116:8300  leader    true   3
prc-test-0-20036  9b00b259-5848-c72b-623e-7991ea1a5d1c  10.126.0.121:8300  follower  true   3
prc-test-1-20036  e92cd7b3-e2f2-30a1-2b06-c434160372f5  10.126.0.122:8300  follower  true   3
prc-test-6-20036  b196a6a1-2813-49d6-e141-b4f0cce19383  10.126.0.118:8300  follower  true   3
prc-test-7-20036  3708ca5f-a17f-a31e-42ed-bb1f92b1506c  10.126.0.120:8300  follower  true   3
prc-test-8-20036  ed7f1658-ea4b-abec-148b-427203d4200c  10.126.0.115:8300  follower  true   3

To troubleshoot or verify the raft list-peers output, ensure that the consul agent process is running on all ZK nodes. You can confirm this by running the following command:

$ ps -elf
4 S consul     77903       1  3  80   0 - 382742 -     Jan13 ?        00:41:10 /opt/apigee/apigee-mtls-consul/bin/consul agent -config-file=/opt/apigee/apigee-mtls-4.52.02-0.0.20256/conf/server.json

This command verifies that the Consul process is active and correctly configured on each node in the multi-data center setup.