Reinstall and restore components

This document covers re-installation and restoration of an Edge component. Use this procedure if you have to re-install the Edge component before you restore the backup.

Apache ZooKeeper

Restore one standalone node

  1. Stop ZooKeeper: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper stop
  2. Remove old ZooKeeper directories: 
    /opt/apigee/data/apigee-zookeeper
/opt/apigee/etc/apigee-zookeeper.d
  3. Re-install ZooKeeper: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper install
  4. Restore ZooKeeper: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper restore 2019.03.17,14.40.41

    Note that when restoring a component, you do not specify the directory path to the backup file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the date/time part of the backup file's name.

    You can optionally omit the backup file in the restore command and Edge will use the most recent backup file in the component's backup directory.

  5. Restart all components: 
    /opt/apigee/apigee-service/bin/apigee-all restart

Restore one cluster node

If a single ZooKeeper node fails that is part of an ensemble, you can create a new node with the same hostname/IP address and re-install ZooKeeper. When the new ZooKeeper node joins the ZooKeeper ensemble it will get the latest snapshots from the Leader and start to serve clients. You do not need to restore data in this instance.

  1. Re-install ZooKeeper: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper install
  2. Run setup on the ZooKeeper node using the same config file used when installing the original node: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper setup -f configFile
  3. Start ZooKeeper: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper start

Restore a complete cluster

  1. Stop the complete cluster.
  2. Restore all ZooKeeper nodes from the backup file as described above for a single node.
  3. Start the ZooKeeper cluster.
  4. Restart all components.

Apache Cassandra

Restore one standalone node

  1. Stop Cassandra: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop
  2. Remove old Cassandra directory: 
    /opt/apigee/data/apigee-cassandra
  3. Re-install Cassandra: 
    /apigee/apigee-service/bin/apigee-service apigee-cassandra install
  4. Restore Cassandra: 
    /apigee/apigee-service/bin/apigee-service apigee-cassandra restore 2019.03.17,14.40.41

    Note that when restoring a component, you do not specify the directory path to the backup file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the date/time part of the backup file's name.

    You can optionally omit the backup file in the restore command and Edge will use the most recent backup file in the component's backup directory.

  5. Restart all components:
    /apigee/apigee-service/bin/apigee-all restart

Restore one cluster node

If a single Cassandra node that is part of an ensemble fails, you can create a new node with the same hostname/IP address. You only need to re-install Cassandra and follow a few extra steps. You do not need to restore the data from backups. Replicas available in other nodes can be used to restore data on the failed node. These steps are typically applicable if a node in your Cassandra cluster crashes due to hardware malfunction, etc.

If the node that has failed is no longer accessible, destroy the node and isolate its network to ensure it cannot accidentally re-join the Cassandra cluster. Then bring up a new node with the same IP address as the original node and run the usual Apigee bootstrap steps on it.

If the node is still accessible and you would like to reuse it, perform the following steps:

  1. Stop the Cassandra process on the node:
    /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop
  2. Uninstall Cassandra software from the node:
    /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra uninstall
  3. Wipe out the data directory from the node:
    rm -rf /opt/apigee/data/apigee-cassandra

After completing the above steps, you should have a node with the same IP address as the node that failed and is ready for Apigee software installation. Perform the following steps:

  1. Identify the seed nodes of the data center.
    1. On any operational Cassandra node in the same data center as the node being restored, run the following command: 
      apigee-service apigee-cassandra configure -search conf_cassandra_seeds
    2. It will output something like this: 
      Found key conf_cassandra_seeds, with value, "127.0.0.1", in /opt/apigee/apigee-cassandra/token/default.properties
Found key conf_cassandra_seeds, with value, 127.0.0.1, in /opt/apigee/apigee-cassandra/token/application/cassandra.properties
Found key conf_cassandra_seeds, with value, "10.1.0.1,10.1.0.2", in /opt/apigee/token/application/cassandra.properties
apigee-configutil: apigee-cassandra: # OK
    3. Refer to the last line in the output. The comma-separated IP addresses listed are the seed nodes used by nodes in this data center.
  2. On the fresh node being restored, create or edit the following file:
    vi /opt/apigee/customer/application/cassandra.properties
  3. Add the following line:
    conf_jvm_options_custom_settings=-Dcassandra.replace_address=<ip-address-of-dead-node>
  4. If this node is one of the seed nodes (Node’s IP address shows up in the seed node list from step 1), add another line in the file:
    conf_cassandra_seeds="<comma-separated-seed-list-except-this-node’s-ip>"
    1. For example, in the instance above, the seed list has 2 IPs - 10.1.0.1 & 10.1.0.2. If the node being restored is 10.1.0.1, add a line like this:
      conf_cassandra_seeds="10.1.0.2"
  5. Save the file and ensure it is owned & readable by the Apigee user:
    chown apigee:apigee /opt/apigee/customer/application/cassandra.properties
  6. Install the Cassandra software:
    /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra install
  7. Run the Cassandra setup:
    /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra setup -f <configFile>
  8. The setup will internally try to bootstrap Cassandra service. This might take a long time based on how much data there is to stream from other nodes, your network throughput, etc.
  9. [Advanced option] You can expedite the streaming of data by running the following nodetool commands on the node:
    nodetool setstreamthroughput <value>
nodetool setcompactionthroughput <value>
  10. Once the Cassandra node has come back up, use standard commands like below to verify that the node has successfully joined the ring:
    /opt/apigee/apigee-cassandra/bin/nodetool ring
/opt/apigee/apigee-cassandra/bin/nodetool status
  11. Once you validate that the node is successfully up and running, edit the file /opt/apigee/customer/application/cassandra.properties again and remove the lines you added earlier in this procedure in steps 3 and 4.
  12. Restart the Cassandra service on the node:
    /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restart

Restore a complete cluster

  1. Stop the complete cluster.
  2. Restore all Cassandra nodes from the backup file.
  3. Start the Cassandra cluster.
  4. Restart all components.

PostgreSQL database

PosgreSQL running standalone or as Master

  1. Stop the Management Server, Qpid Server, and Postgres Server on all nodes: 
    /apigee/apigee-service/bin/apigee-service edge-management-server stop
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server stop
/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop
  2. Re-install PostgreSQL database: 
    /apigee/apigee-service/bin/apigee-service apigee-postgresql install
  3. Start PostgreSQL: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql start
  4. Restore PostgreSQL database from the backup file: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql restore 2019.03.17,14.40.41

    Note that when restoring the PostgreSQL component, you do not specify the directory path to the backup file, nor do you specify the ".dump" suffix. You only specify the date/time part of the backup file's name.

    You can optionally omit the backup file in the restore command and Edge will use the most recent backup file in the component's backup directory.

  5. Start the Management Server, Qpid Server, and Postgres Server on all nodes: 
    /opt/apigee/apigee-service/bin/apigee-service edge-management-server start
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server start
/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server start

PosgreSQL running as Standby

  1. Re-install PostgreSQL database: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql install
  2. Reconfigure PostgreSQL database using the same config file you used to install it:
    /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup -f configFile
  3. Start PostgreSQL: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql start

Postgres Server

  1. Stop Postgres Server on all master and standby nodes: 
    /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop
  2. Remove old Postgres Server directories: 
    /opt/apigee/data/edge-postgres-server /opt/apigee/etc/edge-postgres-server.d
  3. Re-install Postgres Server: 
    /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server install
  4. Restore Postgres Server from the backup file: 
    /opt/apigee/apigee-service/bin/apigee-service edge-postgre-server restore 2019.03.17,14.40.41

    Note that when restoring a component, you do not specify the directory path to the backup file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the date/time part of the backup file's name.

    You can optionally omit the backup file in the restore command and Edge will use the most recent backup file in the component's backup directory.

  5. Start Postgres Server on all master and standby nodes: 
    /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server start

Qpid Server and Qpidd

  1. Stop Qpidd, Qpid Server, and Postgres Server on all nodes: 
    /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server stop
/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop
/opt/apigee/apigee-service/bin/apigee-service apigee-qpidd stop
  2. Remove old Qpid Server and Qpidd directories: 
    /opt/apigee/data/edge-qpid-server
/opt/apigee/etc/edge-qpid-server.d
/opt/apigee/data/apigee-qpidd
/opt/apigee/etc/apigee-qpidd.d
  3. Re-install Qpidd: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-qpidd install
  4. Restore Qpidd: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-qpidd restore 2019.03.17,14.40.41

    Note that when restoring a component, you do not specify the directory path to the backup file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the date/time part of the backup file's name.

    You can optionally omit the backup file in the restore command and Edge will use the most recent backup file in the component's backup directory.

  5. Start Qpidd: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-qpidd start
  6. Re-install Qpid Server: 
    /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server install
  7. Restore Qpid Server: 
    /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restore 2019.03.17,14.40.41

    Note that when restoring a component, you do not specify the directory path to the backup file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the date/time part of the backup file's name.

    You can optionally omit the backup file in the restore command and Edge will use the most recent backup file in the component's backup directory.

  8. Restart Qpid Server, Qpidd, and Postgres Servers on all nodes: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-qpidd restart
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restart
/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server restart

OpenLDAP

  1. Stop OpenLDAP: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-openldap stop
  2. Re-install OpenLDAP: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-openldap install
  3. Remove old OpenLDAP directories: 
    /opt/apigee/data/apigee-openldap /opt/apigee/etc/apigee-openldap.d
  4. Restore OpenLDAP: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-openldap restore 2019.03.17,14.40.41

    Note that when restoring a component, you do not specify the directory path to the backup file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the date/time part of the backup file's name.

    You can optionally omit the backup file in the restore command and Edge will use the most recent backup file in the component's backup directory.

  5. Restart OpenLDAP: 
    /opt/apigee/apigee-service/bin/apigee-service apigee-openldap start
  6. Restart all Management Servers: 
    /opt/apigee/apigee-service/bin/apigee-service edge-management-server restart

Management Server

  1. Stop Management Server: 
    /opt/apigee/apigee-service/bin/apigee-service edge-management-server stop
  2. Remove old Management Server directories: 
    /opt/apigee/data/edge-management-server /opt/apigee/etc/edge-management-server.d
  3. Re-install Management Server: 
    /opt/apigee/apigee-service/bin/apigee-service edge-management-server install
  4. Restore Management Server from the backup file: 
    /opt/apigee/apigee-service/bin/apigee-service edge-management-server restore 2019.03.17,14.40.41

    Note that when restoring a component, you do not specify the directory path to the backup file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the date/time part of the backup file's name.

    You can optionally omit the backup file in the restore command and Edge will use the most recent backup file in the component's backup directory.

  5. Restart Management Server: 
    /opt/apigee/apigee-service/bin/apigee-service edge-management-server start

Message Processor

  1. Stop Message Processor: 
    /opt/apigee/apigee-service/bin/apigee-service edge-message-processor stop
  2. Remove old Message Processor directories: 
    /opt/apigee/data/edge-message-processor
/opt/apigee/etc/edge-message-processor.d
  3. Re-install Message Processor: 
    /opt/apigee/apigee-service/bin/apigee-service edge-message-processor install
  4. Restore Message Processor from the backup file: 
    /opt/apigee/apigee-service/bin/apigee-service edge-message-processor restore 2019.03.17,14.40.41

    Note that when restoring a component, you do not specify the directory path to the backup file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the date/time part of the backup file's name.

    You can optionally omit the backup file in the restore command and Edge will use the most recent backup file in the component's backup directory.

  5. Restart Message Processor: 
    /opt/apigee/apigee-service/bin/apigee-service edge-message-processor start

Router

  1. Stop Router: 
    /opt/apigee/apigee-service/bin/apigee-service edge-router stop
  2. Remove old Router directories: 
    /opt/apigee/data/edge-router
/opt/apigee/etc/edge-router.d
  3. Re-install Router:
    /opt/apigee/apigee-service/bin/apigee-service edge-router install
  4. Restore Router from the backup file: 
    /opt/apigee/apigee-service/bin/apigee-service edge-router restore 2019.03.17,14.40.41

    Note that when restoring a component, you do not specify the directory path to the backup file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the date/time part of the backup file's name.

    You can optionally omit the backup file in the restore command and Edge will use the most recent backup file in the component's backup directory.

  5. Restart Router:
    /opt/apigee/apigee-service/bin/apigee-service edge-router start

Edge UI

  1. Stop UI: 
    /opt/apigee/apigee-service/bin/apigee-service edge-ui stop
  2. Remove old UI directories: 
    /opt/apigee/data/edge-ui
/opt/apigee/etc/edge-ui.d
  3. Re-install UI:
    /opt/apigee/apigee-service/bin/apigee-service edge-ui install
  4. Restore UI from the backup file: 
    /opt/apigee/apigee-service/bin/apigee-service edge-ui restore 2019.03.17,14.40.41

    Note that when restoring a component, you do not specify the directory path to the backup file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the date/time part of the backup file's name.

    You can optionally omit the backup file in the restore command and Edge will use the most recent backup file in the component's backup directory.

  5. Restart UI:
    /opt/apigee/apigee-service/bin/apigee-service edge-ui start

Complete Site Recovery

  1. Stop all component nodes. Note that the order of stopping the subsystems is important: first stop all Edge nodes, and then stop all datastores nodes.
  2. Restore all components as described above.
  3. Now start all components in the following order. Note that the order of starting the subsystems is important:
    1. Start the ZooKeeper cluster
    2. Start the Cassandra cluster
    3. Ensure that OpenLDAP is up and running
    4. Start qpid
    5. Ensure that the PostgreSQL database is up and running
    6. Start Management Server
    7. Start Routers and Message Processors
    8. Start Qpid Server
    9. Start Postgres Server
    10. Start Apigee UI