Moving Apigee Servers

Edge for Private Cloud v. 4.17.09

Apigee components use IP addresses to communicate with each other. Moving components from one machine to another may cause a configuration mismatch. To fix configuration mismatches, follow the relevant instructions below.

Changing the IP Address of a Cassandra Node

To change the IP address of a Cassandra node, perform the following steps:

For configurations with a single Cassandra node

  1. Edit /opt/apigee/customer/application/cassandra.properties on the system being modified. If the file does not exist, create it.
  2. Change the following parameters:
    • Set the conf_cassandra_seeds and conf_cassandra_listen_address parameters to specify the system’s new IP address.
    • Change the conf_cassandra_rpc_address to use either the new IP address or 0.0.0.0 (which allows Cassandra Thrift to listen on all interfaces).
  3. Open /opt/apigee/apigee-cassandra/conf/cassandra-topology.properties in an editor. You should see the old IP address and default setting in the form:
    192.168.56.101=dc-1:ra-1
    default=dc-1:ra-1

    Save that information.
  4. Edit /opt/apigee/customer/application/cassandra.properties to change the old IP address specified to the new IP address:
    conf_cassandra-topology_topology=192.168.56.103=dc-1:ra-1\ndefault=dc-1:ra-1\n

    Ensure that you insert “\n” after the IP address, and specify the same default settings as you found above in Step 3.
  5. Restart Cassandra:
    > /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restart
  6. If necessary also repair ZooKeeper (see below), else restart every Apigee platform component starting with Management Server.

For configurations with multiple Cassandra nodes (ring)

  1. If the node being changed is a seed node, edit /opt/apigee/customer/application/cassandra.properties file on each system in the ring, and change the conf_cassandra_seeds parameter to include the modified system’s new IP. If the cassandra.properties file does not exist, create it.
  2. Edit /opt/apigee/customer/application/cassandra.properties on the system being modified, and change the following parameters:
    • Set the conf_cassandra_listen_address to use the new IP address.
    • Set the conf_cassandra_rpc_address to use either the new IP address or 0.0.0.0 (which allows Cassandra Thrift to listen on all interfaces).
  3. Open /opt/apigee/apigee-cassandra/conf/cassandra-topology.properties in an editor. You should see all Cassandra IP addresses and default setting in the form:
    192.168.56.101=dc-1:ra-1
    192.168.56.102=dc-1:ra-1
    192.168.56.103=dc-1:ra-1
    default=dc-1:ra-1

    Save that information.
  4. Edit /opt/apigee/customer/application/cassandra.properties to change the old IP address specified to the new IP address:
    conf_cassandra-topology_topology=192.168.56.101=dc-1:ra-1\n192.168.56.102=dc-1:ra-1\n192.168.56.104=dc-1:ra-1\ndefault=dc-1:ra-1\n

    Ensure that you insert "\n" after each IP address, and use the same default settings as you recorded above in Step 3.
  5. Restart Cassandra on the modified system. If the modified system is a seed node, also restart each system that used the modified seed node.
    > /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restart
  6. Run the nodetool ring command on the modified node to ensure that the ring is complete. The utility can be found at opt/apigee/apigee-cassandra/bin.
    > nodetool -h localhost ring
  7. Run nodetool repair on the modified node. Note that this process may take some time, so it is highly recommended that this not be done during peak API traffic hours.
    > nodetool -h localhost repair
  8. If necessary, repair ZooKeeper (see below), then restart every Apigee platform component starting with Management Server

Update datastore registrations

  1. Find the UUIDs of datastore registrations specifying the old IP address by using the commands below. Take note of the "type" and "UUID" parameters:
    • curl -u $ADMINEMAIL:$PW "http://$MSIP:$port/v1/servers?pod=central&region=DC" | egrep -i '(type|internalip|uuid|region)'
    • curl -u $ADMINEMAIL:$PW "http://$MSIP:$port/v1/servers?pod=gateway&region=DC" | egrep -i '(type|internalip|uuid|region)'
    • curl -u $ADMINEMAIL:$PW "http://$MSIP:$port/v1/servers?pod=analytics&region=DC" | egrep -i '(type|internalip|uuid|region)'

      where DC is the data center name. In a single data center installation, the value is typically "dc-1".
  2. Register the new IP addresses using one of the commands below. The command needed will depend on the type of the changed node.

    Note: The REGION parameter below refers to the datacenter that the cluster is in. For example, for high availability you would generally have a cluster in dc-1 (Dater Center 1) and a cluster in dc-2 (Data Center 2). This parameter is defined at installation time. The default value is dc-1.
    • For type="application-datastore":
      curl -u $ADMINEMAIL:$PW "http://$MSIP:$port/v1/servers -d "Type=application-datastore&Type=audit-datastore&InternalIP=${NEWIP}&region=${REGION}&pod=central" -H 'content-type: application/x-www-form-urlencoded' -X POST
    • For type="kms-datastore":
      curl -u $ADMINEMAIL:$PW "http://$MSIP:$port/v1/servers -d "Type=kms-datastore&Type=dc-datastore&Type=keyvaluemap-datastore&Type=counter-datastore&Type=cache-datastore &InternalIP=${NEWIP}&region=${REGION}&pod=${GATEWAY_POD}" -H 'content-type: application/x-www-form-urlencoded' -X POST
    • For type="reportcrud-datastore":
      curl -u $ADMINEMAIL:$PW "http://$MSIP:$port/v1/servers" -d "Type=reportcrud-datastore&InternalIP=${NEW_IP}&region=${REGION}&pod=analytics" -H 'content-type: application/x-www-form-urlencoded' -X POST
  3. Delete old registrations for the UUID of the system on which the IP address was changed. For each of these UUIDs issue:
    curl -u $ADMINEMAIL:$PW "http://$MSIP:$port/v1/servers/${OLD_UUID}" -X DELETE

Changing the IP Address of a ZooKeeper Node

Follow the steps below to change the IP address of a ZooKeeper node:

Change the IP Address and restart the ZooKeeper ensemble (for multi-node ensemble configurations only)

  1. Open /opt/apigee/apigee-zookeeper/conf/zoo.cfg in an editor. You should see all ZooKeeper IP addresses and default setting in the form:
    server.1=192.168.56.101:2888:3888
    server.2=192.168.56.102:2888:3888
    server.3=192.168.56.103:2888:3888

    Save that information.
  2. On each ZooKeeper node, edit the file /opt/apigee/customer/application/zookeeper.properties file to set the conf_zoo_quorum property to the correct IP addresses. If the file does not exist, create it.
    conf_zoo_quorum=server.1=192.168.56.101:2888:3888\nserver.2=192.168.56.102:2888:3888\nserver.3=192.168.56.104:2888:3888\n

    Ensure that you insert “\n” after each IP address and that entries are in the same order on every node.
  3. Find the leader of the ZooKeeper ensemble by using the following command (replace <node> with the IP address of the Zookeeper machine):
    echo srvr | nc <node> 2181

    The Mode line in the output should say "leader".
  4. Restart one ZooKeeper after the other starting with the leader and ending with the node on which the IP address was changed. If more than one zookeeper node changed IP addresses it may be necessary to restart all nodes.
    > /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper restart
  5. Use the echo command described above to verify each ZooKeeper node.

Inform the Apigee nodes of the changed configuration

  1. On each Router node, edit the file /opt/apigee/customer/application/router.properties as follows. If the file does not exist, create it.
    • Change the conf_zookeeper_connection.string parameter to include the new IP address
    • Change the conf_zookeeper_zk1.host parameter to include the new IP address
  2. On every Message Processor node, edit the file /opt/apigee/customer/application/message-processor.properties as follows. If the file does not exist, create it.
    • Change the conf_zookeeper_connection.string parameter to include the new IP address
    • Change the conf_zookeeper_zk1.host parameter to include the new IP address
  3. On the Management Server node, edit the file /opt/apigee/customer/application/management-server.properties as follows. If the file does not exist, create it.
    • Change the conf_zookeeper_connection.string parameter to include the new IP address
    • Change the conf_zookeeper_zk1.host parameter to include the new IP address
  4. Restart all Apigee platform component by running the following command on each node:
    /opt/apigee/apigee-service/bin/apigee-all restart

Changing the IP Address of a LDAP Server (OpenLDAP)

To change the IP address of an OpenLDAP node, do the following:

  1. On the Management Server node, edit the file /opt/apigee/customer/application/management-server.properties file. If the file does not exist, create it.
  2. In the management-server.properties file, set the conf_security_ldap.server.host parameter to the new IP address.
  3. Restart the Management Server:
    > /opt/apigee/apigee-service/bin/apigee-service edge-management-server restart

Changing the IP Address of Other Apigee Node Types

To change the IP address of any of these node types (Router, Message Processor, Postgres Server (not postgresql) and Qpid Server (not qpidd):

  1. Use the following cURL command to register the new internal and external IP address:
    curl -u $ADMINEMAIL:$PW -X PUT http://$MSIP:8080/v1/servers/<uuid> -d ExternalIP=<ip>
    curl -u $ADMINEMAIL:$PW -X PUT http://$MSIP:8080/v1/servers/<uuid> -d InternalIP=<ip>

    where uuid is the UUID of the node.

If you do not know the UUID of the node, you can use the following command to display it:

  • Router: curl http://<router_IP>:8081/v1/servers/self
  • Message Processor: curl http://<mp_IP>:8082/v1/servers/self
  • Qpid: curl http://<qp_IP>:8083/v1/servers/self
  • Postgres: curl http://<pg_IP>:8084/v1/servers/self