The Apigee mTLS feature adds security to communications between components in your Edge for the Private Cloud cluster. It provides an industry-standard way to configure and install the service mesh. It supports package management and configuration automation.
Architectural overview
To provide secure communications between components, Apigee mTLS uses a service mesh that establishes secure, mutually authenticated TLS connections between components.
The following image shows connections between Apigee components that Apigee mTLS secures (in red). The ports shown in the image are examples; refer to Port usage for a list of ranges that each component can use.
(Note that ports indicated with an "M" are used to manage the component and must be open on the component for access by the Management Server.)
As you can see in the diagram above, Apigee mTLS adds security to connections between most components in the cluster, including:
| Source | Destination | |
|---|---|---|
| Management Server | Router, MP, QPid, LDAP, Postgres, Zookeeper, and Cassandra nodes | |
| Router | Loopback; Qpid, Zookeeper, and Cassandra nodes | |
| Message Processor | Loopback; Qpid, Zookeeper, and Cassandra nodes | |
| ZooKeeper and Cassandra | Other Zookeeper and Cassandra nodes | |
| Edge UI | SMTP (for external IDP only) | |
| Postgres | Other Postgres, Zookeeper, and Cassandra nodes |
Message encryption/decryption
The Apigee mTLS service mesh consists of Consul servers that run on each ZooKeeper node in your cluster and the following Consul services on every node in the cluster:
- An egress proxy that intercepts outgoing messages on the host node. This service encrypts outgoing messages before sending them to their destination.
- An ingress proxy that intercepts incoming messages on the host node. This service decrypts incoming messages before sending them to their final destination.
For example, when the Management Server sends a message to the Router, the egress proxy service intercepts the outgoing message, encrypts it, and then sends it to the Router. When the Router's node receives the message, the ingress proxy service decrypts the message and then passes it to the Router component for processing.
This all happens transparently to the Edge components: they are unaware of the encryption and decryption process carried out by the Consul proxy services.
In addition, Apigee mTLS uses the iptables utility, a Linux firewall service that
manages traffic redirection.
Requirements
Before you can install Apigee mTLS, your environment must meet the following requirements:
- Version, platform and topology requirements
- Utilities installed and enabled
- User account with the appropriate level of permissions
- An administration machine (recommended)
- Port usage
The following sections describe each of these requirements in detail.
Version, platform, and topology
The following table lists the mTLS requirements:
| Requirement | Description | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Versions |
|
||||||||||||
| Topology | Must include at least three Zookeeper nodes. As a result, you can only install Apigee mTLS on topologies that use 5, 9, 12 (multi-data center), or 13 nodes. For more information, see Installation topologies. | ||||||||||||
| Platforms/Operating Systems | Use the following values to determine whether Apigee mTLS is supported on a particular OS:
Note that Apigee mTLS does not necessarily support all OSes that are supported by the corresponding version of Apigee Edge for Private Cloud on which it is running. For example, if v4.19.06 supports CentOS x and y, that does not necessarily mean that Apigee mTLS is supported on CentOS x and y for v4.19.06. |
||||||||||||
Utilities/packages
Apigee mTLS requires that you have the following packages installed and enabled on each machine in your cluster, including your administration machine, before you begin the installation process:
| Utility/package | Description | OK to Remove After Installation? |
|---|---|---|
base64 |
Verifies data within the installation scripts. | |
gnu-bashgnu-sedgnu-grep |
Used by the installation script and other common tools. | |
iptables |
Replaces the default firewall, firewalld. |
|
iptables-services |
Provides functionality to the iptables utility. |
|
lsof |
Used by the installation script. | |
nc |
Verifies iptables routes. |
|
openssl |
Signs certificates locally during the initial bootstrapping process. |
During installation, you also install the Consul package on the administration machine so that you can generate credentials and the encryption key.
The apigee-mtls package installs and configures the Consul servers including the
ingress and egress proxies on ZooKeeper nodes in the cluster.
User account permissions
Before installing, create a new user account or ensure that you have access to one that has elevated privileges.
The account that executes the Apigee mTLS installation on each node in the cluster must be able to:
- Start, stop, restart, and initialize Apigee components
- Set firewall rules
- Create a new OS/system user account
- Enable, disable, start, stop, and mask services with
systemctl
Administration machine (recommended)
Apigee recommends that you have a node within the cluster on which you can perform various administrative tasks described in this document, including:
- Install HashiCorp Consul 1.6.2.
- Generate and distribute a certificate/key pair and gossip encryption key.
- Update and distribute the configuration file.
When setting up the administration machine:
- Ensure that you have root access to it.
- Download and install the
apigee-serviceandapigee-setuputilities on it, as described in Install Edge apigee-setup utility. - Ensure that you can use
scp/sshto access to all nodes in the cluster from the administration machine. This is required so that you can distribute your configuration file and credentials.
Port usage and assignment
This section describes port usage and port assignments to support Consul communications with Apigee mTLS.
Port usage: All nodes running apigee-mtls
All nodes in the cluster that use the apigee-mtls service must allow connections
from services on the localhost (127.0.0.1). This lets the Consul proxies communicate with the
other services as they process incoming and outgoing messages.
Port usage: Consul server nodes (nodes running ZooKeeper)
You must open most of the following ports on the Consul server nodes (the nodes running ZooKeeper) to accept requests from all nodes in the cluster:
| Node | Consul Server Port | Description | Protocol | Allow external mtls-agents * |
|---|---|---|---|---|
| Consul Server (ZooKeeper nodes) | 8300 |
Connects all Consul servers in the cluster. | RPC | |
8301 |
Handles membership and broadcast messages within the cluster. | UDP/TCP | ||
8302 |
WAN port that handles membership and broadcast messages in a multi-data center configuration. | UDP/TCP | ||
8500 |
Handles HTTP connections to the Consul Server APIs from from processes on the same node.
This port is not used for remote communication or coordination; it listens on localhost only. |
HTTP | ||
8502 |
Handles gRPC+HTTPS connections to the Consul Server APIs from other nodes in the cluster. | gRPC+HTTPS | ||
8503 |
Handles HTTPS connections to the Consul Server APIs from other nodes in the cluster. | HTTPS | ||
8600 |
Handles the Consul server's DNS. | UDP/TCP | ||
* Apigee recommends that you restrict inbound requests to cluster members only
(including cross-datastore). You can do this with iptables.
|
||||
As this table shows, the nodes running the consul-server component (ZooKeeper nodes)
must open ports 8301, 8302, 8502, and 8503 to all members of the cluster that are running
the apigee-mtls service, even across data centers. Nodes that are not running ZooKeeper
do not need to open these ports.
Port assignments for all Consul nodes (including nodes running ZooKeeper)
To support Consul communications, nodes running the following Apigee components must allow external connections to ports within the following ranges:
| Apigee Component | Range | Number of Ports Required Per Node |
|---|---|---|
| Apigee mTLS | 10700 to 10799 | 1 |
| Cassandra | 10100 to 10199 | 2 |
| Message Processor | 10500 to 10599 | 2 |
| OpenLDAP | 10200 to 10299 | 1 |
| Postgres | 10300 to 10399 | 3 |
| Qpid | 10400 to 10499 | 2 |
| Router | 10600 to 10699 | 2 |
| ZooKeeper | 10000 to 10099 | 3 |
Consul assigns ports in a simple linear fashion. For example, if your cluster has two Postgres nodes, the first node uses two ports, so Consul assigns ports 10300 and 10301 to it. The second node also uses two ports, so Consol assigns 10302 and 10303 to that node. This applies to all component types.
As you can see, the actual number of ports depends on topology: If your cluster has two Postgres nodes, then you'll need to open four ports (two nodes times two ports each).
Note the following:
- Consul proxies cannot listen on the same ports as Apigee services.
- Consul has only one port address space. Consul proxy port assignments must be unique across the cluster, which includes data centers. This means that if proxy A on host A listens on port 15000, then proxy B on host B cannot listen on port 15000.
- The number of ports used varies based on the topology, as described previously.
In a multi-data center configuration, all hosts running mTLS must also open port 8302.
You can customize the default ports that Apigee mTLS uses. For information on how to do this, see Proxy port range customization.
Limitations
Apigee mTLS has the following limitations:
- Does not encrypt inter-node Cassandra communications (port 7000)
- Configuration and setup is not idempotent. This means that if you make one change on one node, you must make the same change on all nodes; the system does not apply that change for you on any other node. For more information, see Change an existing apigee-mtls configuration.
Terminology
This section uses the following terminology:
| Term | Definition |
|---|---|
| cluster | The group of machines that make up your Edge for the Private Cloud installation. |
| Consul | The service mesh used by Apigee mTLS. For information about how Consul secures your Private Cloud communications, see Consul's Security Model. |
| mTLS | Mutually Authenticated TLS. |
| service mesh | An overlay network (or a network within a network). |
| TLS | Transaction Layer Security. An industry-standard authentication protocol for secure communications. |