This document covers re-installation and restoration of an Edge component. Use this procedure
if you have to re-install the Edge component before you restore the backup.

> [!NOTE]
> **Note:** These procedures all assume that you are re-installing the component onto a node with the same IP address or DNS name as the node that failed. For Cassandra, you can only use IP addresses.

## Apache ZooKeeper

### Restore one standalone node

1. Stop ZooKeeper:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper stop
   ```
2. Remove old ZooKeeper directories:

   ```
   /opt/apigee/data/apigee-zookeeper
   /opt/apigee/etc/apigee-zookeeper.d
   ```
3. Re-install ZooKeeper:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper install
   ```
4. Restore ZooKeeper:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper restore 2019.03.17,14.40.41
   ```

   Note that when restoring a component, you do not specify the directory path to the backup
   file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the
   date/time part of the backup file's name.

   You can optionally omit the backup file in the `restore` command and Edge will
   use the most recent backup file in the component's backup directory.
5. Restart all components:

   ```
   /opt/apigee/apigee-service/bin/apigee-all restart
   ```

### Restore one cluster node

If a single ZooKeeper node fails that is part of an ensemble, you can create a new node with
the same hostname/IP address and re-install ZooKeeper. When the new ZooKeeper node joins the
ZooKeeper ensemble it will get the latest snapshots from the Leader and start to serve clients.
You do not need to restore data in this instance.

1. Re-install ZooKeeper:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper install
   ```
2. Run setup on the ZooKeeper node using the same config file used when installing the original node:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper setup -f configFile
   ```
3. Start ZooKeeper:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper start
   ```

### Restore a complete cluster

1. Stop the complete cluster.
2. Restore all ZooKeeper nodes from the backup file as described above for a single node.
3. Start the ZooKeeper cluster.
4. Restart all components.

## Apache Cassandra

### Restore one standalone node

1. Stop Cassandra:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop
   ```
2. Remove old Cassandra directory:

   ```
   /opt/apigee/data/apigee-cassandra
   ```
3. Re-install Cassandra:

   ```
   /apigee/apigee-service/bin/apigee-service apigee-cassandra install
   ```
4. Restore Cassandra:

   ```
   /apigee/apigee-service/bin/apigee-service apigee-cassandra restore 2019.03.17,14.40.41
   ```

   Note that when restoring a component, you do not specify the directory path to the backup
   file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the
   date/time part of the backup file's name.

   You can optionally omit the backup file in the `restore` command and Edge will
   use the most recent backup file in the component's backup directory.
5. Restart all components:  

   ```
   /apigee/apigee-service/bin/apigee-all restart
   ```

### Restore one cluster node

If a single Cassandra node that is part of an ensemble fails, you can create a new node with the same hostname/IP address. You only need to re-install Cassandra and follow a few extra steps. You do not need to restore the data from backups. Replicas available in other nodes can be used to restore data on the failed node. These steps are typically applicable if a node in your Cassandra cluster crashes due to hardware malfunction, etc.

> [!NOTE]
> **Note:** When performing a restore on a non-seed node, ensure that at least one Cassandra seed node is up. When performing a restore on a seed node, ensure that at least one other seed node is up.

> [!NOTE]
> **Note:** Since a node failure in a Cassandra cluster could occur at any time, it is advised that you become familiar with these steps by simulating various types of Cassandra node failures in your non-prod environments. You should prepare your own standard operating procedures adhering to your practices which you can follow during production node failures.

> [!WARNING]
> **Caution:** Apigee's Cassandra setup, with a replication factor of 3, can only tolerate the loss of one node within a data center and rack. You should try to avoid having one node down in your Cassandra cluster for long periods of time as if even one additional node goes down, there will be an impact to your API traffic. You should follow the steps listed here to restore your node as soon as feasible.

If the node that has failed is no longer accessible, destroy the node and isolate its network to ensure it cannot accidentally re-join the Cassandra cluster. Then bring up a new node with the same IP address as the original node and run the usual Apigee bootstrap steps on it.

If the node is still accessible and you would like to reuse it, perform the following steps:

> [!WARNING]
> **Caution:**
>
> Following the steps below will result in data loss on the node. These steps assume that the node has failed and you want to cleanly add it back to the cluster, restoring its data from replicas. If there was data on this node that had not been replicated to at least two other nodes, such data will be lost if you proceed with the steps below.
>
> If you want to remove the node cleanly from the cluster, you should follow the steps for [decommissioning](https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/tools/toolsDecommission.html) a Cassandra node.

1. Stop the Cassandra process on the node:  
   `/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop`
2. Uninstall Cassandra software from the node:  
   `/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra uninstall`
3. Wipe out the data directory from the node:  
   `rm -rf /opt/apigee/data/apigee-cassandra`

After completing the above steps, you should have a node with the same IP address as the node that failed and is ready for Apigee software installation. Perform the following steps:

1. Identify the seed nodes of the data center.
   1. On any operational Cassandra node in the same data center as the node being restored, run the following command:

      ```
      apigee-service apigee-cassandra configure -search conf_cassandra_seeds
      ```
   2. It will output something like this:

      ```
      Found key conf_cassandra_seeds, with value, "127.0.0.1", in /opt/apigee/apigee-cassandra/token/default.properties
      Found key conf_cassandra_seeds, with value, 127.0.0.1, in /opt/apigee/apigee-cassandra/token/application/cassandra.properties
      Found key conf_cassandra_seeds, with value, "10.1.0.1,10.1.0.2", in /opt/apigee/token/application/cassandra.properties
      apigee-configutil: apigee-cassandra: # OK
      ```
   3. Refer to the last line in the output. The comma-separated IP addresses listed are the seed nodes used by nodes in this data center.
2. On the fresh node being restored, create or edit the following file:  

   ```
   vi /opt/apigee/customer/application/cassandra.properties
   ```
3. Add the following line:  

   ```
   conf_jvm_options_custom_settings=-Dcassandra.replace_address=<ip-address-of-dead-node>
   ```
4. If this node is one of the seed nodes (Node's IP address shows up in the seed node list from step 1), add another line in the file:  

   ```
   conf_cassandra_seeds="<comma-separated-seed-list-except-this-node’s-ip>"
   ```
   1. For example, in the instance above, the seed list has 2 IPs - 10.1.0.1 \& 10.1.0.2. If the node being restored is 10.1.0.1, add a line like this:  

      ```
      conf_cassandra_seeds="10.1.0.2"
      ```
5. Save the file and ensure it is owned \& readable by the Apigee user:  

   ```
   chown apigee:apigee /opt/apigee/customer/application/cassandra.properties
   ```
6. Install the Cassandra software:  

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra install
   ```
7. Run the Cassandra setup:  

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra setup -f <configFile>
   ```
8. The setup will internally try to bootstrap Cassandra service. This might take a long time based on how much data there is to stream from other nodes, your network throughput, etc.
9. **\[Advanced option\]** You can expedite the streaming of data by running the following nodetool commands on the node:  

   ```
   nodetool setstreamthroughput <value>
   nodetool setcompactionthroughput <value>
   ```
10. Once the Cassandra node has come back up, use standard commands like below to verify that the node has successfully joined the ring:  

    ```
    /opt/apigee/apigee-cassandra/bin/nodetool ring
    /opt/apigee/apigee-cassandra/bin/nodetool status
    ```
11. Once you validate that the node is successfully up and running, edit the file `/opt/apigee/customer/application/cassandra.properties` again and remove the lines you added earlier in this procedure in steps 3 and 4.
12. Restart the Cassandra service on the node:  

    ```
    /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restart
    ```

### Restore a complete cluster

1. Stop the complete cluster.
2. Restore all Cassandra nodes from the backup file.
3. Start the Cassandra cluster.
4. Restart all components.

## PostgreSQL database

### PosgreSQL running standalone or
as Master

1. Stop the Management Server, Qpid Server, and Postgres Server on all nodes:

   > [!NOTE]
   > **Note:** Your system can still handle requests to API proxies while these components are stopped.

       /apigee/apigee-service/bin/apigee-service edge-management-server stop
       /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server stop
       /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop

2. Re-install PostgreSQL database:

   ```
   /apigee/apigee-service/bin/apigee-service apigee-postgresql install 
   ```
3. Start PostgreSQL:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql start
   ```
4. Restore PostgreSQL database from the backup file:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql restore 2019.03.17,14.40.41
   ```

   Note that when restoring the PostgreSQL component, you do not specify the directory path to
   the backup file, nor do you specify the ".dump" suffix. You only specify the
   date/time part of the backup file's name.

   You can optionally omit the backup file in the `restore` command and Edge will
   use the most recent backup file in the component's backup directory.
5. Start the Management Server, Qpid Server, and Postgres Server on all nodes:

       /opt/apigee/apigee-service/bin/apigee-service edge-management-server start
       /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server start
       /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server start

### PosgreSQL running as Standby

1. Re-install PostgreSQL database:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql install
   ```
2. Reconfigure PostgreSQL database using the same config file you used to install it:  

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup -f configFile
   ```
3. Start PostgreSQL:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql start 
   ```

## Postgres Server

1. Stop Postgres Server on all master and standby nodes:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop
   ```
2. Remove old Postgres Server directories:

   ```
   /opt/apigee/data/edge-postgres-server /opt/apigee/etc/edge-postgres-server.d
   ```
3. Re-install Postgres Server:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server install
   ```
4. Restore Postgres Server from the backup file:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-postgre-server restore 2019.03.17,14.40.41
   ```

   Note that when restoring a component, you do not specify the directory path to the backup
   file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the
   date/time part of the backup file's name.

   You can optionally omit the backup file in the `restore` command and Edge will
   use the most recent backup file in the component's backup directory.
5. Start Postgres Server on all master and standby nodes:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server start
   ```

## Qpid Server and Qpidd

1. Stop Qpidd, Qpid Server, and Postgres Server on all nodes:   

       /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server stop
       /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop
       /opt/apigee/apigee-service/bin/apigee-service apigee-qpidd stop

2. Remove old Qpid Server and Qpidd directories:

   ```
   /opt/apigee/data/edge-qpid-server
   /opt/apigee/etc/edge-qpid-server.d
   /opt/apigee/data/apigee-qpidd
   /opt/apigee/etc/apigee-qpidd.d 
   ```
3. Re-install Qpidd:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-qpidd install 
   ```
4. Restore Qpidd:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-qpidd restore 2019.03.17,14.40.41
   ```

   Note that when restoring a component, you do not specify the directory path to the backup
   file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the
   date/time part of the backup file's name.

   You can optionally omit the backup file in the `restore` command and Edge will
   use the most recent backup file in the component's backup directory.
5. Start Qpidd:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-qpidd start 
   ```
6. Re-install Qpid Server:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server install
   ```
7. Restore Qpid Server:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restore 2019.03.17,14.40.41
   ```

   Note that when restoring a component, you do not specify the directory path to the backup
   file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the
   date/time part of the backup file's name.

   You can optionally omit the backup file in the `restore` command and Edge will
   use the most recent backup file in the component's backup directory.
8. Restart Qpid Server, Qpidd, and Postgres Servers on all nodes:

       /opt/apigee/apigee-service/bin/apigee-service apigee-qpidd restart
       /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restart
       /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server restart

## OpenLDAP

1. Stop OpenLDAP:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-openldap stop
   ```
2. Re-install OpenLDAP:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-openldap install
   ```
3. Remove old OpenLDAP directories:

   ```
   /opt/apigee/data/apigee-openldap /opt/apigee/etc/apigee-openldap.d
   ```
4. Restore OpenLDAP:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-openldap restore 2019.03.17,14.40.41
   ```

   Note that when restoring a component, you do not specify the directory path to the backup
   file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the
   date/time part of the backup file's name.

   You can optionally omit the backup file in the `restore` command and Edge will
   use the most recent backup file in the component's backup directory.
5. Restart OpenLDAP:

   ```
   /opt/apigee/apigee-service/bin/apigee-service apigee-openldap start
   ```
6. Restart all Management Servers:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-management-server restart
   ```

## Management Server

1. Stop Management Server:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-management-server stop
   ```
2. Remove old Management Server directories:

   ```
   /opt/apigee/data/edge-management-server /opt/apigee/etc/edge-management-server.d
   ```
3. Re-install Management Server:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-management-server install
   ```
4. Restore Management Server from the backup file:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-management-server restore 2019.03.17,14.40.41
   ```

   Note that when restoring a component, you do not specify the directory path to the backup
   file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the
   date/time part of the backup file's name.

   You can optionally omit the backup file in the `restore` command and Edge will
   use the most recent backup file in the component's backup directory.
5. Restart Management Server:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-management-server start
   ```

## Message Processor

1. Stop Message Processor:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-message-processor stop
   ```
2. Remove old Message Processor directories:

   ```
   /opt/apigee/data/edge-message-processor
   /opt/apigee/etc/edge-message-processor.d
   ```
3. Re-install Message Processor:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-message-processor install
   ```
4. Restore Message Processor from the backup file:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-message-processor restore 2019.03.17,14.40.41
   ```

   Note that when restoring a component, you do not specify the directory path to the backup
   file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the
   date/time part of the backup file's name.

   You can optionally omit the backup file in the `restore` command and Edge will
   use the most recent backup file in the component's backup directory.
5. Restart Message Processor:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-message-processor start
   ```

## Router

1. Stop Router:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-router stop
   ```
2. Remove old Router directories:

   ```
   /opt/apigee/data/edge-router
   /opt/apigee/etc/edge-router.d
   ```
3. Re-install Router:  

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-router install
   ```
4. Restore Router from the backup file:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-router restore 2019.03.17,14.40.41
   ```

   Note that when restoring a component, you do not specify the directory path to the backup
   file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the
   date/time part of the backup file's name.

   You can optionally omit the backup file in the `restore` command and Edge will
   use the most recent backup file in the component's backup directory.
5. Restart Router:  

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-router start
   ```

## Edge UI

1. Stop UI:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-ui stop
   ```
2. Remove old UI directories:

   ```
   /opt/apigee/data/edge-ui
   /opt/apigee/etc/edge-ui.d
   ```
3. Re-install UI:  

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-ui install
   ```
4. Restore UI from the backup file:

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-ui restore 2019.03.17,14.40.41
   ```

   Note that when restoring a component, you do not specify the directory path to the backup
   file, nor do you specify the "backup-" prefix or the ".tar.gz" suffix. You only specify the
   date/time part of the backup file's name.

   You can optionally omit the backup file in the `restore` command and Edge will
   use the most recent backup file in the component's backup directory.
5. Restart UI:  

   ```
   /opt/apigee/apigee-service/bin/apigee-service edge-ui start
   ```

## Complete Site Recovery

1. Stop all component nodes. Note that the order of stopping the subsystems is important: first stop all Edge nodes, and *then* stop all datastores nodes.
2. Restore all components as described above.
3. Now start all components in the following order. Note that the order of starting the subsystems is important:
   1. Start the ZooKeeper cluster
   2. Start the Cassandra cluster
   3. Ensure that OpenLDAP is up and running
   4. Start qpid
   5. Ensure that the PostgreSQL database is up and running
   6. Start Management Server
   7. Start Routers and Message Processors
   8. Start Qpid Server
   9. Start Postgres Server
   10. Start Apigee UI