Roll back Apigee Edge 4.52.02

If you encounter an error during an update to Edge 4.52.02, you can roll back the component that caused the error and then try the update again.

You can roll back Edge 4.52.02 to the following major release versions:

Version 4.52.01
Version 4.52.00
Version 4.51.00

Rolling back a version involves rolling back every component that you may have upgraded. Additionally, based on the version you started from, you may need to take into account special considerations before rolling back certain software components. Below table lists the various software for which special steps may be needed during rollback:

Rollback to version	Special consideration for software
4.52.01	Cassandra
4.52.00	Cassandra, Zookeeper, Qpid
4.51.00	Cassandra, Zookeeper, Qpid, Postgres

There are two scenarios where you might want to perform a rollback:

Roll back to a previous major or minor release. For example from 4.52.02 to 4.52.00.
Roll back to a previous patch release in the same release. For example, from 4.52.00.02 to 4.52.00.01.

For more information, see Apigee Edge release process.

Order of rollback

Rollback of components should be done in the reverse order they were upgraded, with the exception that management servers should be rolled back after Cassandra.

A typical general order of rollback for Private Cloud 4.52.02 will look like below:

Rollback Postgres, Qpid, and other analytics-related components
Rollback Routers and Message Processors
Rollback Cassandra, Zookeeper
Rollback Management server

For example, let’s say you had upgraded the entire Cassandra cluster, all your management servers, and a few RMPs to version 4.52.02 from version 4.52.01 and wish to rollback. In this case, you would:

Rollback all RMPs one by one
Rollback the entire Cassandra cluster using backups
Rollback Edge Management server nodes one by one

Who can perform a rollback

The user performing a rollback should be the same as the user who originally updated Edge, or a user running as root.

By default, Edge components run as the user "apigee". In some cases, you might be running Edge components as different users. For example, if the Router has to access privileged ports, such as those below 1000, then you have to run the Router as root or as a user with access to those ports. Or, you might run one component as one user, and another component as another user.

Components with common code

The following Edge components share common code. Therefore, to roll back any one of these components on a node, you must roll back all of these components that are on that node.

edge-management-server (Management Server)
edge-message-processor (Message Processor)
edge-router (Router)
edge-postgres-server (Postgres Server)
edge-qpid-server (Qpid Server)

For example, if you have the Management Server, Router, and Message Processor installed on the node, to roll back any one of them you must roll back all three.

Rollback of Cassandra

When a major upgrade of Cassandra is performed on a particular Cassandra node, Cassandra modifies the schema of the data stored on the node, making a direct rollback infeasible. There are two methodologies to rollback. You will use one of these methodologies based on the state of the upgrade you are rolling back from.

Methodologies to rollback

Wipe out and use existing nodes in the cluster to rebuild the node

This procedure can be followed if you have at least one fully functional data center in the cluster still on the older version of Cassandra (Cassandra 2.1.22). If you have upgraded the entire Cassandra cluster and want to rollback, you must follow the steps in Wipe out and restore backup or VM snapshot.

Steps for rollback

Start with the node you want to rollback:
- Stop Cassandra on the node:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop
```
- Uninstall Cassandra software from the node:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra uninstall
```
- Remove data directory from the node:
```
rm -rf /opt/apigee/data/apigee-cassandra
```
- Download and run the bootstrap of the older version of Edge for Private Cloud you want to rollback to.
Example: To rollback to 4.52.01,
- Download bootstrap of 4.52.01:
```
curl https://software.apigee.com/bootstrap_4.52.01.sh -o /tmp/bootstrap_4.52.01.sh -u ‘uName:pWord’
```
- Execute bootstrap of 4.52.01:
```
sudo bash /tmp/bootstrap_4.52.01.sh apigeeuser=uName apigeepassword=pWord
```
- Setup Cassandra on the node:
```
/opt/apigee/apigee-setup/bin/setup.sh -p c -f configFile
```
- Execute rebuild on the node by supplying the name of the functional data center:
```
/opt/apigee/apigee-cassandra/bin/nodetool rebuild -h <node-IP> <functional-dc>
```
- Repeat the above steps on each node you want to rollback, one at a time.
Once all nodes are rolled back and rebuilt:
- Run the setup of any management-server node. Ensure the management server is from the rolled-back version. If not, rollback the management server too.
- Stop the management server:
```
/opt/apigee/apigee-service/bin/apigee-service edge-management-server stop
```
- If you use monetization, uninstall monetization as well:
```
/opt/apigee/apigee-service/bin/apigee-service edge-mint-gateway uninstall
```
- Uninstall the management-server:
```
/opt/apigee/apigee-service/bin/apigee-service edge-gateway uninstall
```
- Download and execute bootstrap of the older version. For example, execute the following steps to download and execute bootstrap of version 4.52.01:
```
curl https://software.apigee.com/bootstrap_4.52.01.sh -o /tmp/bootstrap_4.52.01.sh -u ‘uName:pWord’
sudo bash /tmp/bootstrap_4.52.01.sh apigeeuser=uName apigeepassword=pWord
```
- Run the setup of one management-server node:
```
/opt/apigee/apigee-setup/bin/setup.sh -p mt -f configFile
```

Optimization after rebuild

In the above steps, all the data in the node is streamed from the remote data center during the rebuild. You can optimize this process by using a repair once all replicas have been streamed to the local data center. This avoids cross-DC streaming and should be quicker than rebuilding all nodes from a remote DC.

Example: Suppose you have six Cassandra nodes in the local data center. By default, Apigee’s replication factor is three, so every node possesses 50% of the data. In this case, you can rebuild nodes #1 and #4 by following the procedure above. For nodes #2, #3, #5, and #6, follow the steps below to restore the backup and run a repair.

Follow procedure up to step six as documented to rebuild replicas in the local data center.
For the remaining nodes, follow the steps below on each remaining node one at a time.
- Restore the backup you had captured on this node (note: this backup would likely have stale data as this backup was taken before you started the Cassandra upgrade):
```
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restore backup_file
```
- If you have a VM snapshot of the node, you can restore the snapshot instead of restoring the Cassandra backup.
- After the backup is restored, start the Cassandra service on the node:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra start
```
- Execute a repair on the node so that the latest data can be streamed from an existing data center:
```
/opt/apigee/apigee-cassandra/bin/nodetool -h <node-IP> repair -dc <local-dc-name>
```
- Repeat steps three to six on each node you want to repair.

Wipe out and restore backup/VM snapshot

This procedure is the only one available if you have upgraded the entire Cassandra cluster and wish to rollback.

Additionally, Apigee backups are node specific. It is not possible to restore a backup taken from one node into another. Cassandra backups include node metadata information (like IP address, ring position, etc.).

Start with 1 Cassandra node in the cluster:

Stop Cassandra service on the node:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop

Uninstall Cassandra software from the node:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra uninstall

Remove data directory from the node:

rm -rf /opt/apigee/data/apigee-cassandra

Download and run bootstrap of older version of Edge for Private Cloud you want to rollback to.

Example: To rollback to 4.52.01

Download bootstrap of 4.52.01:

curl https://software.apigee.com/bootstrap_4.52.01.sh -o /tmp/bootstrap_4.52.01.sh -u ‘uName:pWord’

Execute bootstrap of 4.52.01:

sudo bash /tmp/bootstrap_4.52.01.sh apigeeuser=uName apigeepassword=pWord

Setup Cassandra on the node:

/opt/apigee/apigee-setup/bin/setup.sh -p c -f configFile

Restore backup on the node using the procedure in Restore from a backup.

Stop Cassandra on the node:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop

Delete data directory:

rm -rf /opt/apigee/data/apigee-cassandra/data

Restore backup:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restore backup_file

Start Cassandra service on the node:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra start

Repeat steps on each Cassandra node one at a time.

Run repair across the nodes one at a time:

/opt/apigee/apigee-cassandra/bin/nodetool -h <node-IP> repair -pr

Rollback the Zookeeper 3.8.3 update

If you are rolling back to versions 4.52.00 or 4.51.00, you will need to refer to some special steps before rolling back Zookeeper. These steps are listed in Rollback.

If you are rolling back to version 4.52.01, roll back Zookeeper as you would roll back any software, as listed in the Roll back to a previous major or minor release section below.

Rollback Qpid

If you are rolling back to versions 4.52.00 or 4.51.00, you will need to refer to some special steps before rolling back Qpid. These steps are listed in Rollback.

If you are rolling back to version 4.52.01, rollback Qpid like you would rollback any software as listed in the Roll back to a previous major or minor release

Rollback the Postgres 10.17 update

If you are rolling back to version 4.51.00, you will need to refer to some special steps before rolling back Postgres. These steps are listed in Rollback.

If you are rolling back to version 4.52.01 or 4.52.00, roll back Postgres as you would roll back any software, as listed in the Roll back to a previous major or minor release section below.

Roll back to a previous major or minor release

To roll back to a previous major or minor release, do the following on each node that hosts the component:

Download the bootstrap.sh file for the version to which you want to roll back:
- To roll back to 4.51.00, download bootstrap_4.51.00.sh:
```
curl https://software.apigee.com/bootstrap_4.51.00.sh -o /tmp/bootstrap_4.51.00.sh 
```

Stop the component to roll back:

To roll back any of the components with common code on the node, you must stop them all, as the following example shows:

/opt/apigee/apigee-service/bin/apigee-service edge-management-server stop
/opt/apigee/apigee-service/bin/apigee-service edge-router stop
/opt/apigee/apigee-service/bin/apigee-service edge-message-processor stop
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server stop
/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop

To roll back any other component on the node, stop just that component:
```
/opt/apigee/apigee-service/bin/apigee-service component stop
```

If you are rolling back Monetization, uninstall it from all Management Server and Message Processor nodes:
```
/opt/apigee/apigee-service/bin/apigee-service edge-mint-gateway uninstall
```
Uninstall the component to roll back on the node:
1. To roll back any of the components with common code on the node, you must uninstall them all by uninstalling the edge-gateway component group, as the following example shows:
```
/opt/apigee/apigee-service/bin/apigee-service edge-gateway uninstall
```
2. To roll back any other component on the node, uninstall just that component, as the following example shows:
```
/opt/apigee/apigee-service/bin/apigee-service component uninstall
```
  Where component is the component name.
3. To roll back the Edge Router, you must delete the contents of the /opt/nginx/conf.d file in addition to uninstalling the edge-gateway component group:
```
cd /opt/nginx/conf.d
rm -rf *
```

Uninstall the 4.52.02 version of apigee-setup:

/opt/apigee/apigee-service/bin/apigee-service apigee-setup uninstall

Install the 4.51.00 version of the apigee-service utility and its dependencies. The following example installs the 4.51.00 version of the apigee-service:
```
sudo bash /tmp/bootstrap_4.51.00.sh apigeeuser=uName apigeepassword=pWord
```
Where uName and pWord are the username and password you received from Apigee. If you omit pWord, you will be prompted to enter it.

If you get an error, be sure you downloaded the bootstrap.sh file in step 1.

Install apigee-setup:

/opt/apigee/apigee-service/bin/apigee-service apigee-setup install

Install the older version of the component:
```
/opt/apigee/apigee-setup/bin/setup.sh -p component -f configFile
```
Where component is the component to install and configFile is your configuration file for the older version.
If you are rolling back Qpid, flush iptables:
```
sudo iptables -F
```
Repeat this process for each node that hosts the component you are rolling back.

Roll back to a previous patch release

To roll back a component to a specific patch release, do the following on each node that hosts the component:

Download the specific component version:
```
/opt/apigee/apigee-service/bin/apigee-service component_version install
```
Where component_version is the component and patch release to install. For example:
```
/opt/apigee/apigee-service/bin/apigee-service edge-ui-4.51.05-0.0.3749 install
```
If you are using the Apigee online repo, you can determine the available component versions by using the following command:
```
yum --showduplicates list comp
```
For example:
```
yum --showduplicates list edge-ui
```
Use apigee-setup to install the component:
```
/opt/apigee/apigee-setup/bin/setup.sh -p comp -f configFile
```
For example:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ui -f configFile
```
Note that you specify only the component name when you install it, not the version.
Repeat this process for each node that hosts the component you are rolling back.

Roll back mTLS

To roll back the mTLS update, do the following steps on all hosts:

Stop Apigee:
```
apigee-all stop
```
Stop mTLS:
```
apigee-service apigee-mtls uninstall
```

Reinstall mTLS:

apigee-service apigee-mtls install
apigee-service apigee-mtls setup -f /opt/silent.conf