Adding ZooKeeper nodes

Edge for Private Cloud v. 4.17.01

This document describes how to add three new ZooKeeper nodes to an existing Edge for Private Cloud installation.

You can add one or two ZooKeeper nodes to an existing Edge installation, however, you must make sure that you always have an odd number of ZooKeeper voter nodes, as described below.

Existing Edge configuration

All the supported Edge topologies for a production system specify to use three ZooKeeper nodes. The three nodes are specified to the ZK_HOSTS and ZK_CLIENT_HOSTS properties in the config file as shown below:

IP1=10.10.0.1
IP2=10.10.0.2
IP3=10.10.0.3
HOSTIP=$(hostname -i)
ADMIN_EMAIL=opdk@google.com 
APIGEE_ADMINPW=Secret123
LICENSE_FILE=/tmp/license.txt 
MSIP=$IP1 
USE_LDAP_REMOTE_HOST=n 
LDAP_TYPE=1
APIGEE_LDAPPW=secret 
BIND_ON_ALL_INTERFACES=y
MP_POD=gateway
REGION=dc-1 
ZK_HOSTS="$IP1 $IP2 $IP3" 
ZK_CLIENT_HOSTS="$IP1 $IP2 $IP3" 
CASS_HOSTS="$IP1:1,1 $IP2:1,1 $IP3:1,1" 
SKIP_SMTP=n
SMTPHOST=smtp.example.com 
SMTPUSER=smtp@example.com  
SMTPPASSWORD=smtppwd

where:

  • ZK_HOSTS specifies the IP addresses or DNS names of the ZooKeeper nodes. The IP addresses or DNS names must be listed in the same order on all ZooKeeper nodes. In a multi-data center environment, list all ZooKeeper nodes from both data centers.
  • ZK_CLIENT_HOSTS specifies the IP addresses or DNS names of the ZooKeeper nodes used by this data center. The IP addresses or DNS names must be listed in the same order on all ZooKeeper nodes.

    In a single data center installation, these are the same nodes as specified by ZK_HOSTS. In a multi-data center environment, list only the ZooKeeper nodes in this data center.

Modifying the config file to add the three new ZooKeeper nodes

In this example, the three new ZooKeeper nodes are at the following IP addresses:

  • 10.10.0.14
  • 10.10.0.15
  • 10.10.0.16

You must first update Edge configuration file to add the new nodes:

IP1=10.10.0.1
IP2=10.10.0.2
IP3=10.10.0.3
# Add the new node IP addresses.
IP14=10.10.0.14
IP15=10.10.0.15
IP16=10.10.0.16
HOSTIP=$(hostname -i)
ADMIN_EMAIL=opdk@google.com 
...
# Update ZK_HOSTS to add each new node after an existing nodes. 
ZK_HOSTS="$IP1 $IP2 $IP3 $IP14 $IP15 $IP16:observer" 
# Update ZK_Client_HOSTS to add each new node after an existing nodes. 
ZK_CLIENT_HOSTS="$IP1 $IP2 $IP3 $IP14 $IP15 $IP16"

Mark the last node in ZK_HOSTS with the with “:observer” modifier. Nodes without the “:observer” modifier are called "voters". You must have an odd number of "voters" in your configuration. Therefore, in this configuration, you have 5 ZooKeeper voters and one observer.

Make sure to add the nodes to both ZK_HOSTS and ZK_CLIENT_HOSTS in the same order. However, omit the “:observer” modifier when setting ZK_CLIENT_HOSTS.

Configure Edge

After editing the config file, you must perform all of the following tasks.

Install ZooKeeper on the new nodes

  1. Install apigee-setup on the first node as described in Install the Edge apigee-setup utility.
  2. Install ZooKeeper on the first node by using the following commands:
    > /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper install
    > /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper setup -f updatedConfigFile
  3. Repeat steps 1 and 2 for the remaining new ZooKeeper nodes.

Reconfigure the existing ZooKeeper nodes

On the existing ZooKeeper nodes:

  1. Rerun the setup.sh with the "-p c" profile and the new config file:
    > /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper setup -f updatedConfigFile

Restart all Zookeeper nodes

On all ZooKeeper nodes:

  1. Restart the node:
    > /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper restart

    You must restart all ZooKeeper nodes, but order of restart does not matter.

Reconfigure the Management Server node

On the Management Server node:

  1. Run the setup command:
    > /opt/apigee/apigee-service/bin/apigee-service edge-management-server setup -f updatedConfigFile
  2. Restart the Management Server:
    > /opt/apigee/apigee-service/bin/apigee-service edge-management-server restart

Reconfigure all the Routers

On all Router nodes:

  1. Run the setup command:
    > /opt/apigee/apigee-service/bin/apigee-service edge-router setup -f updatedConfigFile
  2. Restart the Router:
    > /opt/apigee/apigee-service/bin/apigee-service edge-router restart

Reconfigure all the Message Processors

On all Message Processor nodes:

  1. Run the setup command:
    /opt/apigee/apigee-service/bin/apigee-service edge-message-processor setup -f updatedConfigFile
  2. Restart the Message Processor:
    > /opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart

Reconfigure all Qpid nodes

On all Qpid nodes:

  1. Run the setup command:
    > /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server setup -f updatedConfigFile
  2. Restart Qpid:
    > /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restart

Reconfigure all Postgres nodes

On all Postgres nodes:

  1. Run the setup command:
    > /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server setup -f updatedConfigFile
  2. Restart Postgres:
    > /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server restart

Validate the installation

You can validate the installation of the new ZooKeeper nodes by sending commands to port 2181 using netcat (nc) or telnet. For more info on ZooKeeper commands, see: http://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_zkCommands.

To validate:

  1. If it is not installed on the ZooKeeper node, install nc:
    > sudo yum install nc
  2. Run the following nc command:
    > echo stat | nc localhost 2181
  3. Repeat steps 1 and 2 on each ZooKeeper node.
    In the Mode line of the output for the nodes, one node should be designated as observer, one node as leader, and the rest as followers.