Configure Apigee mTLS

After you install Apigee mTLS on all nodes in your cluster, you must configure and initialize the apigee-mtls component. You do this by generating a certificate/key pair and updating the configuration file on your administration machine. You then deploy the same generated files and configuration file to all nodes in the cluster and initialize the local apigee-mtls component.

Configure apigee-mtls (after initial installation)

This section describes how to configure Apigee mTLS directly after the initial installation. For information on updating an existing installation of Apigee mTLS, see Change an existing apigee-mtls configuration.

This section applies to installations in a single data center. For information on configuring Apigee mTLS in a multiple data center setup, see Configure multiple data centers for Apigee mTLS.

The general process for configuring apigee-mtls is as follows:

Update your configuration file: On your administration machine, update the configuration file to include the apigee-mtls settings.
Install Consul and generate credentials: Install Consul and (optionally) use it to generate TLS credentials (once only).
In addition, edit your Apigee mTLS configuration file to:
1. Add the credentials information
2. Define the cluster's topology
Note that you can use your existing credentials or generate them with Consul.
Distribute the credentials and configuration file: Distribute the same generated certificate/key pair and updated configuration file to all nodes in your cluster.
Initialize apigee-mtls: Initialize the apigee-mtls component on each node.

Each of these steps is described in the sections that follow.

Step 1: Update your configuration file

This section describes how to modify your configuration file to include mTLS configuration properties. For more general information about the configuration file, see Creating a configuration file.

After you update your configuration file with the mTLS-related properties, copy it to all nodes in the cluster before you initialize the apigee-mtls component on those nodes.

To update the configuration file:

On your administration machine, open the configuration file for edit.

Copy the following set of mTLS configuration properties and paste them into the configuration file:

ALL_IP="ALL_PRIVATE_IPS_IN_CLUSTER"
ZK_MTLS_HOSTS="ZOOKEEPER_PRIVATE_IPS"
CASS_MTLS_HOSTS="CASSANDRA_PRIVATE_IPS"
PG_MTLS_HOSTS="POSTGRES_PRIVATE_IPS"
RT_MTLS_HOSTS="ROUTER_PRIVATE_IPS"
MS_MTLS_HOSTS="MGMT_SERVER_PRIVATE_IPS"
MP_MTLS_HOSTS="MESSAGE_PROCESSOR_PRIVATE_IPS"
QP_MTLS_HOSTS="QPID_PRIVATE_IPS"
LDAP_MTLS_HOSTS="OPENLDAP_PRIVATE_IPS"
MTLS_ENCAPSULATE_LDAP="y"

ENABLE_SIDECAR_PROXY="y"
ENCRYPT_DATA="BASE64_GOSSIP_MESSAGE"
PATH_TO_CA_CERT="PATH/TO/consul-agent-ca.pem"
PATH_TO_CA_KEY="PATH/TO/consul-agent-ca-key.pem"
APIGEE_MTLS_NUM_DAYS_CERT_VALID_FOR="NUMBER_OF_DAYS"

Set the value of each property to align with your configuration.

The following table describes these configuration properties:

Property	Description
`ALL_IP`	A space separated list of the private host IP addresses of all nodes in the cluster. The order of IP addresses does not matter, except that it must be the same in all configuration files across the cluster. If you configure Apigee mTLS for multiple data centers, list all IP addresses for all hosts on all regions.
`LDAP_MTLS_HOSTS`	The private host IP address of the OpenLDAP node in the cluster.
`ZK_MTLS_HOSTS`	A space separated list of private host IP addresses on which ZooKeeper nodes are hosted in the cluster. Note that based on the requirements, there must be at least three ZooKeeper nodes.
`CASS_MTLS_HOSTS`	A space separated list of private host IP addresses on which Cassandra servers are hosted in the cluster.
`PG_MTLS_HOSTS`	A space separated list of private host IP addresses on which Postgres servers are hosted in the cluster.
`RT_MTLS_HOSTS`	A space separated list of private host IP addresses on which Routers are hosted in the cluster.
`MTLS_ENCAPSULATE_LDAP`	Encrypts LDAP traffic between the Message Processor and LDAP server. Set to `y`.
`MS_MTLS_HOSTS`	A space separated list of private host IP addresses on which Management Server nodes are hosted in the cluster.
`MP_MTLS_HOSTS`	A space separated list of private host IP addresses on which Message Processors are hosted in the cluster.
`QP_MTLS_HOSTS`	A space separated list of private host IP addresses on which Qpid servers are hosted in the cluster.
`ENABLE_SIDECAR_PROXY`	Determines whether Cassandra and Postgres should be aware of the service mesh. You must set this value to "y".
`ENCRYPT_DATA`	The base64-encoded encryption key used by Consul. You generated this key by using the `consul keygen` command in Step 2: Install Consul and generate credentials. This value must be the same across all nodes in the cluster.
`PATH_TO_CA_CERT`	The location of the certificate file on the node. You generated this file in Step 2: Install Consul and generate credentials. This location should be the same across all nodes in the cluster so that the configuration files are the same. The certificate must be X509v3 encoded.
`PATH_TO_CA_KEY`	The location of the key file on the node. You generated this file in Step 2: Install Consul and generate credentials. This location should be the same across all nodes in the cluster so that the configuration files are the same. The key file must be X509v3 encoded.
`APIGEE_MTLS_NUM_DAYS_CERT_VALID_FOR`	The number of days a certificate is good for when you generate a custom certificate. The default value is 365. The maximum value is 7865 days (5 years).

In addition to the properties listed above, Apigee mTLS uses several additional properties when you install it on a multi-data center configuration. For more information, see Configure multiple data centers.

Be sure that ENABLE_SIDECAR_PROXY is set to "y".
Update the IP addresses in the host-related properties. Be sure to use the private IP addresses when referring to each node, not the public IP addresses.
In later steps, you will set the values of the other properties such as ENCRYPT_DATA, PATH_TO_CA_CERT, and PATH_TO_CA_KEY. You do not set their values yet.

When editing the apigee-mtls configuration properties, note the following:
- All properties are strings; you must wrap the values of all properties in single or double quotes.
- If a host-related value has more than one private IP address, separate each IP address with a space.
- Use private IP addresses and not host names or public IP addresses for all host-related properties in the configuration file.
- The order of IP addresses in a property value must be in the same order in all configuration files across the cluster.
Save your changes to the configuration file.

Step 2: Install Consul and generate credentials

This section describes how to install Consul and generate the credentials that are used by the mTLS-enabled components.

You must choose one of the following methods to generate your credentials:

(Recommended) Create your own Certificate Authority (CA) using Consul, as described in this section
Use the credentials of an existing CA with Apigee mTLS (advanced)

About the credentials

The credentials consist of the following:

Certificate: The TLS certificate
Key: The TLS public key
Gossip message: A base-64 encoded encryption key

You generate a single version of each of these files once only. You then copy the key and certificate files to all the nodes in your cluster, and add the encryption key to your configuration file that you also copy to all nodes.

For more information about Consul's encryption implementation, see the following:

Install Consul and generate credentials

To generate credentials that Apigee mTLS uses for authenticating secure communications among the nodes in your Private Cloud cluster, use a local Consul binary . As a result, you must install Consul on your administration machine before you can generate credentials.

To install Consul and generate mTLS credentials:

On your administration machine, download the Consul 1.8.0 binary from the HashiCorp website.
Extract the contents of the downloaded archive file. For example, extract the contents to /opt/consul/.
NOTE: Continue with this procedure to use Consul to generate the credentials (recommended). If you want to use your existing credentials, see Integrate custom certificates with Apigee mTLS (advanced).
On your administration machine, create a new Certificate Authority (CA) by executing the following command:
```
/opt/consul/consul tls ca create
```
Consul creates the following files, which form a certificate/key pair:
- consul-agent-ca.pem (certificate)
- consul-agent-ca-key.pem (key)
By default, certificate and key files are X509v3 encoded.

Later, you will copy these files to all nodes in the cluster. At this time, however, you must only decide where on the nodes you will put these files. They should be in the same location on each node. For example, /opt/apigee/.
In the configuration file, set the value of PATH_TO_CA_CERT to the location to which you will copy the consul-agent-ca.pem file on the node. For example:
```
PATH_TO_CA_CERT="/opt/apigee/consul-agent-ca.pem"
```
Set the value of PATH_TO_CA_KEYto the location to which you will copy the consul-agent-ca-key.pem file on the node. For example:
```
PATH_TO_CA_KEY="/opt/apigee/consul-agent-ca-key.pem"
```
Create an encryption key for Consul by executing the following command:
```
/opt/consul/consul keygen
```
Consul outputs a randomized string that looks similar to the following:
```
QbhgD+EXAMPLE+Y9u0742X/IqX3X429/x1cIQ+JsQvY=
```
Copy this generated string and set it as the value of the ENCRYPT_DATA property in your configuration file. For example:
```
ENCRYPT_DATA="QbhgD+EXAMPLE+Y9u0742X/IqX3X429/x1cIQ+JsQvY="
```
NOTE: The value above is an example encryption key. It is not the actual string you will use. You must generate your own.
Save your configuration file.

The following example shows the mTLS-related settings in a configuration file (with example values):

...
IP1=10.126.0.121
IP2=10.126.0.124
IP3=10.126.0.125
IP4=10.126.0.127
IP5=10.126.0.130
ALL_IP="$IP1 $IP2 $IP3 $IP4 $IP5"
LDAP_MTLS_HOSTS="$IP3"
ZK_MTLS_HOSTS="$IP3 $IP4 $IP5"
CASS_MTLS_HOSTS="$IP3 $IP4 $IP5"
PG_MTLS_HOSTS="$IP2 $IP1"
RT_MTLS_HOSTS="$IP4 $IP5"
MS_MTLS_HOSTS="$IP3"
MP_MTLS_HOSTS="$IP4 $IP5"
QP_MTLS_HOSTS="$IP2 $IP1"
ENABLE_SIDECAR_PROXY="y"
ENCRYPT_DATA="QbhgD+EXAMPLE+Y9u0742X/IqX3X429/x1cIQ+JsQvY="
PATH_TO_CA_CERT="/opt/apigee/consul-agent-ca.pem"
PATH_TO_CA_KEY="/opt/apigee/consul-agent-ca-key.pem"
...

Step 3: Distribute the configuration file and credentials

Copy the following files to all nodes using a tool such as scp:

Configuration file: Copy the updated version of this file and replace the existing version on all nodes (not just the nodes running ZooKeeper).
consul-agent-ca.pem: Copy to the location you specified as the value of PATH_TO_CA_CERT in the configuration file.
consul-agent-ca-key.pem: Copy to the location you specified as the value of PATH_TO_CA_KEY in the configuration file.

Be sure that the locations to which you copy the certificate and key files match the values you set in the configuration file in Step 2: Install Consul and generate credentials.

Step 4: Initialize apigee-mtls

After you installed apigee-mtls on each node, updated your configuration file, and copied it and the credentials to all nodes in the cluster, you are ready to initialize the apigee-mtls component on each node.

To initialize apigee-mtls:

Log in to a node in the cluster as the root user. You can perform these steps on the nodes in any order you want.
Make the apigee:apigee user an owner of the updated configuration file, as the following example shows:
```
chown apigee:apigee config_file
```

Configure the apigee-mtls component by executing the following command:

/opt/apigee/apigee-service/bin/apigee-service apigee-mtls setup -f config_file

(Optional) Execute the following command to verify that your setup was successful:
```
/opt/apigee/apigee-mtls/lib/actions/iptables.sh validate
```
Start Apigee mTLS by executing the following command:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-mtls start
```
After installing Apigee mTLS, you must start this component before any other components on the node.

(Cassandra nodes only) Cassandra requires additional arguments to work within the security mesh. As a result, you must execute the following commands on each Cassandra node:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra setup -f config_file
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra configure
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restart

(Postgres nodes only) Postgres requires additional arguments to work within the security mesh. As a result, you must do the following on the Postgres nodes:
(Primary only)
1. Execute the following commands on the Postgres primary node:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup -f config_file
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql configure
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql restart
```
(Standby only)
1. Back up your existing Postgres data. To install Apigee mTLS, you must re-initialize the primary/standby nodes, so there will be data loss. For more information, see Set up primary/standby replication for Postgres.
2. Delete all Postgres data:
```
rm -rf /opt/apigee/data/apigee-postgresql/pgdata
```
3. Configure Postgres and then restart Postgres, as the following example shows:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup -f config_file
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql configure
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql restart
```
NOTE: Due to network issues, you might need to run this step more than once before it succeeds.

If you are installing on a multi-data center topology, use an absolute path for the configuration file.
Start the remaining Apigee components on the node in the start order, as the following example shows:
```
/opt/apigee/apigee-service/bin/apigee-service component_name start
```
Repeat this process for each node in the cluster.
(Optional) Verify that the apigee-mtls initialization was successful by using one or more of the following methods:
1. Validate the iptables configuration
2. Verify remote proxy status
3. Verify quorum status
Each of these methods is described in Verify your configuration.

Change an existing apigee-mtls configuration

To customize an existing apigee-mtls configuration, you must uninstall and reinstall apigee-mtls. You must also be sure to apply your customization across all nodes.

CAUTION: Apigee mTLS setup and configuration is not idempotent. As a result, the apigee-mtls configuration entries must be identical on all nodes: if you make a change on one node, you must make that same change on all nodes. If you do not, the service mesh may fail.

To reiterate this point, when changing an existing Apigee mTLS configuration:

If you change a configuration file, you must first uninstall apigee-mtls and re-run setup or configure:

# DO THIS:
/opt/apigee/apigee-service/bin/apigee-service apigee-mtls uninstall

# BEFORE YOU DO THIS:
/opt/apigee/apigee-service/bin/apigee-service apigee-mtls setup -f file
OR
/opt/apigee/apigee-service/bin/apigee-service apigee-mtls configure

You must uninstall and re-run setup or configure on all nodes in the cluster, not just a single node.

NOTE: Repeatedly executing setup or configure on apigee-mtls does not act the same as other Apigee services. It will not necessarily generate a new configuration (or modify the local configuration). As a result, you must uninstall mTLS when you change the configuration.