Roll back Apigee Edge 4.52.02

If you encounter an error during an update to Edge 4.52.02, you can roll back the component that caused the error and then try the update again.

You can roll back Edge 4.52.02 to any of the following major release versions:

Version 4.52.01
Version 4.52.00
Version 4.51.00

Rolling back a version involves rolling back every component that you may have upgraded. Additionally, based on the version you started from, you may need to consider special steps before rolling back certain software components. The following table lists the various software components for which special steps may be needed during rollback:

Rollback to version	Special consideration for software
4.52.01	Cassandra
4.52.00	Zookeeper, Cassandra, Qpid
4.51.00	Zookeeper, Postgres, Cassandra, Qpid

There are two scenarios where you might want to perform a rollback:

Roll back to a previous major or minor release. For example, from 4.52.02 to 4.52.00.
Roll back to a previous patch release in the same release. For example, from 4.52.00.02 to 4.52.00.01.

For more information, see Apigee Edge release process.

Order of rollback

The rollback of components should follow the reverse order of their upgrade, with the exception that Management Servers should be rolled back after Cassandra. Cassandra, Runtime components, and the Management Server should all be rolled back using a data center-by-data center (DC-by-DC) approach, temporarily redirecting traffic to the functional data centers.

A typical general order of rollback for Private Cloud 4.52.02 will look like the following:

Single data center

For single data center setup, the rollback procedure will encounter a significant impact on runtime traffic and certain management APIs.

Rollback Qpid and other analytics-related components
Rollback Routers and Message Processors
Rollback Cassandra
Rollback Management server
Rollback Postgres and Zookeeper

Multiple data centers

In a multi-data center setup, rollbacks should follow a data center-by-data center (DC-by-DC) approach by temporarily redirecting traffic to the functional data centers. This ensures traffic continuity, avoids downtime, and enables a controlled rollback process for Cassandra, Management Server, and Runtime nodes.

Rollback Qpid and other analytics-related components across all the DCs.
Block traffic in the first data center and re-route the traffic to other DCs.
Rollback Routers and Message Processors in the first data center.
Rollback Cassandra in the first data center.
Rollback Management server in the first data center.
Unblock the traffic in the first data center and follow step #2 to step #6 until the last data center gets Runtime nodes, Cassandra, and Management server rollbacked.
Rollback Postgres, Zookeeper, and LDAP across all the DCs.

To put it into perspective, suppose you have upgraded your entire Cassandra cluster, all Management Servers, and a few Runtime Message Processors (RMPs) from version 4.52.01 to 4.52.02 and need to perform a rollback. In this case, the rollback should be carried out as follows:

Block traffic to the first data center (data center) and re-route traffic to the other active DCs to ensure service continuity.
Rollback Routers and Message Processors in the first data center.
Rollback Cassandra in the first data center by restoring from a backup or VM snapshot.
Rollback the Management Server in the first data center.
Unblock traffic to the first data center.
Repeat steps 1 to 5 for each remaining data center until all Runtime nodes, Cassandra, and Management Servers have been rolled back.

Who can perform a rollback

The user performing a rollback should be the same as the user who originally updated Edge, or a user running as root.

By default, Edge components run as the user "apigee". In some cases, you might be running Edge components as different users. For example, if the Router has to access privileged ports, such as those below 1000, then you have to run the Router as root or as a user with access to those ports. Or, you might run one component as one user, and another component as another user.

Components with common code

The following Edge components share common code. Therefore, to roll back any one of these components on a node, you must roll back all of these components that are on that node.

edge-management-server (Management Server)
edge-message-processor (Message Processor)
edge-router (Router)
edge-postgres-server (Postgres Server)
edge-qpid-server (Qpid Server)

For example, if you have the Management Server, Router, and Message Processor installed on the node, to roll back any one of them you must roll back all three.

Rollback of Cassandra

When a major upgrade of Cassandra is performed on a particular Cassandra node, Cassandra modifies the schema of the data stored on the node, making a direct rollback infeasible. There are two methodologies to rollback. You will use one of these methodologies based on the state of the upgrade you are rolling back from.

Methodologies to rollback

Rollback scenarios

Edge for Private Cloud 4.52.02 includes an upgrade in Cassandra and the driver used by the message processor and management server to connect to Cassandra. As a result, upgrades and rollback of these 3 components are closely tied together. The below table lists general examples of rollback scenarios for these three specific components. Rolling back other components should follow as per the order of the rollback section.

This section outlines various rollback scenarios along with the recommended methodology to be followed, based on the approaches described above.

Scenario	Rollback Strategy
Single data center, some Cassandra nodes upgraded	Backup restore
Single data center, all Cassandra nodes upgraded	Backup restore
Single data center, all nodes (Cassandra, Management server and Runtime nodes) upgraded	Runtime Nodes rollback Cassandra Backup restore Management rollback
Multiple data center, some/all Cassandra nodes in first data center upgraded	Rebuild from existing data center
Multiple data centers, all Cassandra nodes, Management server and Runtime nodes in first data center upgraded	Runtime Nodes rollback Cassandra rebuild restore Management rollback This should be performed on one data center at a time.
Multiple data centers, some/all Cassandra nodes of last data center are upgraded	Rollback the last data center using backup Rollback remaining data centers either using backup or rebuild, one data center at a time.
Multiple data centers, all Cassandra nodes, Management server and Runtime nodes upgraded in all the DCs	Runtime Nodes rollback Cassandra backup restore Management rollback This should be performed one data center at a time.

Generally, you should consider the following while rolling back the Cassandra:

Rollback of runtime or management components
If you need to roll back components such as the Edge Management Server or Edge Message Processor to a previous Edge Private Cloud version in any data center (DC), ensure that Cassandra is also rolled back in that specific data center at the same time. This is necessary to prevent management and runtime traffic failures.
Rollback using backups
Backups taken from Cassandra 3.11.x are not compatible with those from Cassandra 2.1.x. To enable rollback using backup restoration, ensure that backups of Cassandra 2.1.x are taken prior to performing the upgrade.
Isolating the Data Center for Rollback
To avoid downtime, ensure that traffic is redirected to fully functional data centers and blocked from the data center undergoing rollback.

Rollback Cassandra using rebuild

Prerequisites

You are operating an Edge for Private Cloud 4.51.00 / 4.52.00 / 4.52.01 cluster over multiple data centers
You are in the process of upgrading Cassandra from 2.1.X to 3.11.X and have encountered issues during upgrade
You have at least 1 fully functional data center in the cluster still on the older version of Cassandra (Cassandra 2.1.X)

High level steps

Choose one data center (either partially or fully upgraded) that you want to roll back. Redirect all application traffic from this data center to another fully functional data center.
If Router and Message Processor has been upgraded, rollback all Router and message processor nodes in the data center, one at a time.
Stop Cassandra on one node, uninstall it, and clean up all associated data.
Install the previous version bootstrap and set up Cassandra version 2.1.x on the cleaned node.
Rebuild the node from the existing functional data center that is still running Cassandra 2.1.x.
Perform steps 3 to 5 on each remaining Cassandra node in the data center, one node at a time.
Re-run the Management-Server setup in the data center.
Conduct testing to validate the rollback. Once verified, redirect application traffic back to the restored data center.
Repeat the above steps for the other data centers requiring a rollback, one at a time.

Detailed steps to wipe out and use existing nodes in the cluster to rebuild the node:

Start with the node you want to rollback

Ensure that traffic is redirected to fully functional data centers before proceeding with the next steps.
If Router and Message Processor has been upgraded, rollback all Router and message processor nodes to the previous version in the data center, one at a time.

Stop Cassandra on the node:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop

Uninstall Cassandra software from the node:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra uninstall

Remove data directory from the node:

rm -rf /opt/apigee/data/apigee-cassandra

Download and run the bootstrap of the older version of Edge for Private Cloud you want to rollback to:

Example: To rollback to 4.52.01

Download bootstrap of 4.52.01:

curl https://software.apigee.com/bootstrap_4.52.01.sh -o /tmp/bootstrap_4.52.01.sh -u ‘uName:pWord’

Execute bootstrap of 4.52.01:

sudo bash /tmp/bootstrap_4.52.01.sh apigeeuser=uName apigeepassword=pWord

Install Cassandra Software on the node:
```
apigee-service apigee-cassandra install
```

Add the below property in the /opt/apigee/apigee-cassandra/source/conf/cassandra-env.sh file.

JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=<cass_ip-address>"

Example:

JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=10.0.0.1"

Setup Cassandra on the node:
```
/opt/apigee/apigee-setup/bin/setup.sh -p c -f configFile
```
Note: In case the Cassandra setup fails, edit the /opt/apigee/apigee-cassandra/source/conf/cassandra-env.sh file and remove the line containing: JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=<cass_ip-address>" After making this change, re-run the above setup.sh script to retry the setup.
After Cassandra UP and RUNNING, remove the above CWC from the below file:/opt/apigee/apigee-cassandra/source/conf/cassandra-env.sh file.
Restart the Cassandra node
```
apigee-service apigee-cassandra restart
```

Execute rebuild on the node by supplying the name of the functional data center:

/opt/apigee/apigee-cassandra/bin/nodetool rebuild -h <node-IP> <functional-dc>

Example:

/opt/apigee/apigee-cassandra/bin/nodetool rebuild -h 10.0.0.1 dc-2

Repeat the above steps on each node you want to rollback in the data center, one at a time.

Once all Cassandra nodes in the data center are rolled back and rebuilt

Run the setup of any of the management-server nodes in the data center being rolled back. Ensure the management server is from the rolled-back version. If not, rollback the management server too.

Rollback the management server to older version

Stop the management server:

/opt/apigee/apigee-service/bin/apigee-service edge-management-server stop

If you use monetization, uninstall monetization as well:

/opt/apigee/apigee-service/bin/apigee-service edge-mint-gateway uninstall

Uninstall the edge-gateway and apigee-cassandra-client:

/opt/apigee/apigee-service/bin/apigee-service edge-gateway uninstall

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra-client uninstall

Download and execute bootstrap of the older version. For example, execute the following steps to download and execute bootstrap of version 4.52.01

curl https://software.apigee.com/bootstrap_4.52.01.sh -o /tmp/bootstrap_4.52.01.sh -u ‘uName:pWord’

sudo bash /tmp/bootstrap_4.52.01.sh apigeeuser=uName apigeepassword=pWord

Management server setup

Run the setup of one management-server node:

/opt/apigee/apigee-setup/bin/setup.sh -p mt -f configFile

After completing the above steps, redirect traffic back to the rolled-back data center.

Optimization after rebuild

In the above steps, all the data in the node is streamed from the remote data center during the rebuild. You can optimize this process by using a repair once all replicas have been streamed to the local data center. This avoids cross-data center streaming and should be quicker than rebuilding all nodes from a remote data center.

Example: Suppose you have six Cassandra nodes in the local data center. By default, Apigee’s replication factor is three, so every node possesses 50% of the data. In this case, you can rebuild nodes #1 and #4 by following the procedure above. For nodes #2, #3, #5, and #6, follow the steps below to restore the backup and run a repair.

Follow procedure up to above steps as documented to rebuild replicas in the local data center.
For the remaining nodes, follow the steps below on each remaining node one at a time.

Restore the backup you had captured on this node (note: this backup would likely have stale data as this backup was taken before you started the Cassandra upgrade):
```
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restore backup_file
```
If you have a VM snapshot of the node, you can restore the snapshot instead of restoring the Cassandra backup.
After the backup is restored, start the Cassandra service on the node:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra start
```

Execute a repair on the node so that the latest data can be streamed from an existing data center:

/opt/apigee/apigee-cassandra/bin/nodetool -h <node-IP> repair -dc <local-dc-name>

Example:

/opt/apigee/apigee-cassandra/bin/nodetool -h 10.0.0.1 repair -dc dc-1

Repeat all of the above steps mentioned under Step #2 on each node you want to repair

Rollback Cassandra using backup / VM snapshot

This procedure is the only one available if you have upgraded the entire Cassandra cluster and wish to rollback. Additionally, Apigee backups are node specific. It is not possible to restore a backup taken from one node into another. Cassandra backups include node metadata information (like IP address, ring position, etc.).

Prerequisites

You are in the process of upgrading Cassandra from 2.1.X to 3.11.X in the last data center and have encountered issues during upgrade.
You have backups for the node before the upgrade you are rolling back. The backup was taken before upgrade of 2.1.X to 3.11.X was attempted.

High level steps

Choose a data center (either partially or fully upgraded) to roll back. Redirect all runtime traffic from this data center to another fully functional data center.
If router and message processor has been upgraded, rollback all router and message processor nodes in the data center, one at a time
Stop Cassandra on one node, uninstall it, and clean up all associated data.
Install the previous version bootstrap and set up Cassandra version 2.1.x on the cleaned node.
Stop the Cassandra node and clean up all associated data.
Restore the Cassandra node from the backup taken prior to upgrade.
Repeat steps 3 to 6 for each of the remaining Cassandra nodes in the data center, one node at a time.
Re-run the Management-Server setup in the data center.
Perform testing to validate the rollback. Once verified, redirect runtime traffic back to the restored data center.
Repeat the above steps for the other data centers requiring a rollback, one at a time.
(Optional) Execute the repair command across all Cassandra nodes in all data centers if there is data inconsistency between them.

Detailed steps to rollback Cassandra using backups/VM snapshot

Start with 1 cassandra node in the cluster

Ensure that traffic is redirected to fully functional data centers before proceeding with the next steps.
If router and message processor has been upgraded, rollback all router and message processor nodes to the previous version in the data center, one at a time.

Stop Cassandra on the node:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop

Uninstall Cassandra software from the node:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra uninstall

Remove data directory from the node:

rm -rf /opt/apigee/data/apigee-cassandra

Download and run the bootstrap of the older version of Edge for Private Cloud you want to rollback to:

Example: To rollback to 4.52.01

Download bootstrap of 4.52.01:

curl https://software.apigee.com/bootstrap_4.52.01.sh -o /tmp/bootstrap_4.52.01.sh -u ‘uName:pWord’

Execute bootstrap of 4.52.01:

sudo bash /tmp/bootstrap_4.52.01.sh apigeeuser=uName apigeepassword=pWord

Setup Cassandra on the node:

/opt/apigee/apigee-setup/bin/setup.sh -p c -f configFile

Stop Cassandra on the node:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop

Delete data directory on the node:

rm -rf /opt/apigee/data/apigee-cassandra/data

Restore backup:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restore backup_file

Start Cassandra service on the node

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra start

Repeat steps on each Cassandra node one at a time.
Run the setup of any of the management-server nodes in the data center being rolled back. Ensure the management server is from the rolled-back version. If not, rollback the management server too.
After completing the above steps, redirect traffic back to the rolled-back data center.
(Optional) Execute the repair command across all Cassandra nodes in all data centers if there is data inconsistency between them.
```
/opt/apigee/apigee-cassandra/bin/nodetool -h <node-IP> repair -pr
```

Rollback the Zookeeper 3.8.3 update

If you are rolling back to versions 4.52.00 or 4.51.00, you will need to refer to some special steps before rolling back Zookeeper. These steps are listed in Rollback.

If you are rolling back to version 4.52.01, roll back Zookeeper as you would roll back any software, as listed in the Roll back to a previous major or minor release section below.

Rollback Qpid

If you are rolling back to versions 4.52.00 or 4.51.00, you will need to refer to some special steps before rolling back Qpid. These steps are listed in Rollback.

If you are rolling back to version 4.52.01, rollback Qpid like you would rollback any software as listed in the Roll back to a previous major or minor release

Rollback the Postgres 10.17 update

If you are rolling back to version 4.51.00, you will need to refer to some special steps before rolling back Postgres. These steps are listed in Rollback.

If you are rolling back to version 4.52.01 or 4.52.00, roll back Postgres as you would roll back any software, as listed in the Roll back to a previous major or minor release section below.

Roll back to a previous major or minor release

To roll back to a previous major or minor release, do the following on each node that hosts the component:

Download the bootstrap.sh file for the version to which you want to roll back:
- To roll back to 4.51.00, download bootstrap_4.51.00.sh

Stop the component to roll back:

To roll back any of the components with common code on the node, you must stop them all, as the following example shows:

/opt/apigee/apigee-service/bin/apigee-service edge-management-server stop
/opt/apigee/apigee-service/bin/apigee-service edge-router stop
/opt/apigee/apigee-service/bin/apigee-service edge-message-processor stop
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server stop
/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop

To roll back any other component on the node, stop just that component:
```
/opt/apigee/apigee-service/bin/apigee-service component stop
```

If you are rolling back Monetization, uninstall it from all Management Server and Message Processor nodes:
```
/opt/apigee/apigee-service/bin/apigee-service edge-mint-gateway uninstall
```
Uninstall the component to roll back on the node:
1. To roll back any of the components with common code on the node, you must uninstall them all by uninstalling the edge-gateway and apigee-cassandra-client component group, as the following example shows:
```
/opt/apigee/apigee-service/bin/apigee-service edge-gateway uninstall
```
```
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra-client uninstall
```
2. To roll back any other component on the node, uninstall just that component, as the following example shows:
```
/opt/apigee/apigee-service/bin/apigee-service component uninstall
```
  Where component is the component name.
3. To roll back the Edge Router, you must delete the contents of the /opt/nginx/conf.d file in addition to uninstalling the edge-gateway component group:
```
cd /opt/nginx/conf.d
rm -rf *
```

Uninstall the 4.52.02 version of apigee-setup:

/opt/apigee/apigee-service/bin/apigee-service apigee-setup uninstall

Install the 4.51.00 version of the apigee-service utility and its dependencies. The following example installs the 4.51.00 version of the apigee-service:
```
sudo bash /tmp/bootstrap_4.51.00.sh apigeeuser=uName apigeepassword=pWord
```
Where uName and pWord are the username and password you received from Apigee. If you omit pWord, you will be prompted to enter it.

If you get an error, be sure you downloaded the bootstrap.sh file in step 1.

Install apigee-setup:

/opt/apigee/apigee-service/bin/apigee-service apigee-setup install

Install the older version of the component:
```
/opt/apigee/apigee-setup/bin/setup.sh -p component -f configFile
```
Where component is the component to install and configFile is your configuration file for the older version.
If you are rolling back Qpid, flush iptables:
```
sudo iptables -F
```
Repeat this process for each node that hosts the component you are rolling back.

Roll back to a previous patch release

To roll back a component to a specific patch release, do the following on each node that hosts the component:

Download the specific component version:
```
/opt/apigee/apigee-service/bin/apigee-service component_version install
```
Where component_version is the component and patch release to install. For example:
```
/opt/apigee/apigee-service/bin/apigee-service edge-ui-4.51.05-0.0.3749 install
```
If you are using the Apigee online repo, you can determine the available component versions by using the following command:
```
yum --showduplicates list component
```
For example:
```
yum --showduplicates list edge-ui
```
Use apigee-setup to install the component:
```
/opt/apigee/apigee-setup/bin/setup.sh -p component -f configFile
```
For example:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ui -f configFile
```
Note that you specify only the component name when you install it, not the version.
Repeat this process for each node that hosts the component you are rolling back.

Roll back mTLS

To roll back the mTLS update, do the following steps on all hosts:

Stop Apigee:
```
apigee-all stop
```
Stop mTLS:
```
apigee-service apigee-mtls uninstall
```

Reinstall mTLS:

apigee-service apigee-mtls install
apigee-service apigee-mtls setup -f /opt/silent.conf