Roll back Apigee Edge 4.54.00

If you encounter an error during an update to Edge 4.54.00, you can roll back the component that caused the error and then try the update again.

You can roll back Edge 4.54.00 to any of the following major release versions:

Version 4.53.00
Version 4.53.01

For more information, see Apigee Edge release process.

Order of rollback

Rollback of components should be done in the reverse order they were upgraded. A typical general order of rollback for Private Cloud 4.54.00 will look like below:

Roll back Cassandra, Zookeeper.
Roll back Qpid, other analytics-related components and Postgres.
Roll back UI
Roll back LDAP.
Roll back management server.
Roll back routers and message processors.

For example, let's say you had upgraded all your Routers and Message Processors, all your management servers, LDAP, and a few UI nodes to version 4.54.00 from version 4.53.00 (or) 4.53.01 and wish to roll back. In this case, you would:

Roll back the upgraded UI nodes
Roll back the LDAP nodes
Roll back Management server nodes one by one
Roll back all RMPs one by one

Note: Apigee SSO in Apigee Edge for Private Cloud 4.54.00 doesn't support direct or partial rollbacks. To revert Apigee SSO, you must roll back all previously upgraded components and restore apigee-sso using the silent configuration file from your previous version.

Restart or reinstall apigee-sso:

If you only stopped apigee-sso during the upgrade, restart the service.
If you uninstalled apigee-sso during the upgrade, reinstall it using the silent configuration file from your previous version:
- 4.53.01 Installation steps
- 4.53.00 Installation steps

Who can perform a rollback

The user performing a rollback should be the same as the user who originally updated Edge, or a user running as root.

By default, Edge components run as the user "apigee". In some cases, you might be running Edge components as different users. For example, if the Router has to access privileged ports, such as those below 1000, then you have to run the Router as root or as a user with access to those ports. Or, you might run one component as one user, and another component as another user.

Roll back Cassandra

When a major upgrade of Cassandra is performed on a specific node, Cassandra modifies the schema of the data stored on that node. As a result, a direct in-place rollback is not feasible.

Rollback scenarios

The following table summarizes the rollback strategies that you can use:

Scenario	Rollback strategy
Single DC, some Cassandra nodes upgraded	Use backups
Multiple DC, some nodes in one DC upgraded	Rebuild from existing DC
Multiple DC, all Cassandra nodes in some DCs upgraded	Rebuild from existing DC
Multiple DC, Cassandra nodes of the last DC being upgraded	Try to finish the upgrade. If not feasible, roll back 1 DC using backup. Rebuild remaining DCs from the rolled-back DC.

General considerations

When considering a rollback, keep the following in mind:

Rollback of Cassandra during the Cassandra upgrade: If you encounter issues during the Cassandra upgrade, you may want to consider a rollback. Follow the rollback strategies in this article based on your state during the upgrade process.
Rollback using backups: Backups taken from Cassandra 5.0.X aren't compatible with backups of Cassandra 4.0.X. To roll back Cassandra using backup restoration, you must take backups of Cassandra 4.0.X before attempting the upgrade.

Roll back Cassandra using rebuild

Prerequisites

You are operating an Edge for Private Cloud 4.53.00/4.53.01 cluster across multiple data centers.
You are in the process of upgrading Cassandra from 4.0.X to 5.0.X and have encountered issues during the upgrade.
You have at least one fully functional data center in the cluster still running the older version of Cassandra (Cassandra 4.0.X).

This procedure relies on streaming data from an existing data center. It could take a significant amount of time, depending on how much data is stored in Cassandra. You should be prepared to divert your runtime traffic away from this data center while the rollback is ongoing.

High-level steps

Select one data center (either partially or fully upgraded) that you’d like to roll back. Divert runtime traffic to a different functioning data center.
Identify the seed node in the data center and start with one of the seed nodes.
Stop, uninstall, and clean up the Cassandra node.
Install the older version of Cassandra on the node and configure it as needed.
Remove the extra configurations that were added earlier.
Repeat the above steps for all seed nodes in the data center, one by one.
Repeat the above steps for all remaining Cassandra nodes in the data center, one by one.
Rebuild the nodes from the existing functional data center, one by one.
Restart all edge-* components in the data center that are connected to Cassandra.
Test and divert traffic back to this data center.
Repeat the steps for each data center, one by one.

Detailed steps

Pick one data center where all or some Cassandra nodes are upgraded. Divert all runtime proxy traffic and management traffic from this data center while the Cassandra nodes in this data center are being rolled back. Ensure all Cassandra nodes are in the UN (Up/Normal) state when the nodetool ring command is executed on the nodes. If certain nodes are down, troubleshoot the issue and bring those nodes back up before continuing.

See the example below:

/opt/apigee/apigee-cassandra/bin/nodetool status
Datacenter: dc-1
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens       Owns (effective)  Host ID                               Rack
UN  DC1-1IP1  456.41 KiB  1            100.0%            78fc4ddd-2ed9-4a8c-98a2-63a38c2f1920  ra-1
UN  DC1-1IP2  870.93 KiB  1            100.0%            160db01a-64ab-43a7-b9ea-3b7f8f66d52b  ra-1
UN  DC1-1IP3  824.08 KiB  1            100.0%            21d61543-d59e-403a-bf5d-bfe7f664baa6  ra-1
Datacenter: dc-2
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens       Owns (effective)  Host ID                               Rack
UN  DC2-1IP1   802.08 KiB  1            100.0%            583e0576-336d-4ce7-9729-2ae74e0abde2  ra-1
UN  DC2-1IP2   844.4 KiB   1            100.0%            fef794d5-f4c2-4a4e-bb05-9adaeb4aea4b  ra-1
UN  DC2-1IP3   878.12 KiB  1            100.0%            3894b3d9-1f5a-444d-83db-7b1e338bbfc9  ra-1

You can run nodetool describecluster on the nodes to understand the current state of the entire cluster. For example, the following shows an instance of a 2-data-center cluster where all DC-1 nodes are on Cassandra version 4, whereas all DC-2 nodes are on Cassandra version 3:

# On nodes where Cassandra is upgraded
/opt/apigee/apigee-cassandra/bin/nodetool describecluster
Cluster Information:
    Name: Apigee
    Snitch: org.apache.cassandra.locator.PropertyFileSnitch
    DynamicEndPointSnitch: enabled
    Partitioner: org.apache.cassandra.dht.RandomPartitioner
    Schema versions:
        2eadcd74-0245-309a-9992-3625afa70038: [DC-1-IP1, DC-1-IP2, DC-1-IP3]
        129dc15e-198e-3c11-b64c-701044a3a1ad: [DC-2-IP1, DC-2-IP2, DC-2-IP3]

Stats for all nodes:
    Live: 6
    Joining: 0
    Moving: 0
    Leaving: 0
    Unreachable: 0

Data Centers:
    dc-1 #Nodes: 3 #Down: 0
    dc-2 #Nodes: 3 #Down: 0

Database versions:
    5.0.8: [DC-1-IP1:7000, DC-1-IP2:7000, DC-1-IP3:7000]
    4.0.13: [DC-2-IP1:7000, DC-2-IP2:7000, DC-2-IP3:7000]

Keyspaces:
    system_schema -> Replication class: LocalStrategy {}
    system -> Replication class: LocalStrategy {}
    auth -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    cache -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    devconnect -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    dek -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    user_settings -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    apprepo -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    kms -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    identityzone -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    audit -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    analytics -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    keyvaluemap -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    counter -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    apimodel_v2 -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    system_distributed -> Replication class: SimpleStrategy {replication_factor=3}
    system_traces -> Replication class: SimpleStrategy {replication_factor=2}
    system_auth -> Replication class: SimpleStrategy {replication_factor=1}

# On nodes where Cassandra is not upgraded
/opt/apigee/apigee-cassandra/bin/nodetool describecluster
Cluster Information:
    Name: Apigee
    Snitch: org.apache.cassandra.locator.PropertyFileSnitch
    DynamicEndPointSnitch: enabled
    Partitioner: org.apache.cassandra.dht.RandomPartitioner
    Schema versions:
        2eadcd74-0245-309a-9992-3625afa70038: [DC-1-IP1, DC-1-IP2, DC-1-IP3]
        129dc15e-198e-3c11-b64c-701044a3a1ad: [DC-2-IP1, DC-2-IP2, DC-2-IP3]

Identify the seed nodes in the data center: Refer to the section How to identify seed nodes in the Appendix. Execute the steps below on one of the seed nodes:

Stop, uninstall, and clean up data from the node of Cassandra. Pick the first seed node on Cassandra version 5 in this data center. Stop it.

# Stop Cassandra service on the node
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop

# Uninstall Cassandra software
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra uninstall

# Wipe out Cassandra data
rm -rf /opt/apigee/data/apigee-cassandra

Install the older Cassandra software on the node and set some configurations. Execute the bootstrap file of Edge for Private Cloud 4.53.00/4.53.01.

# Download bootstrap of 4.53.00
curl https://software.apigee.com/bootstrap_4.53.00.sh -o /tmp/bootstrap_4.53.00.sh -u uName:pWord

# Execute bootstrap of 4.53.00
sudo bash /tmp/bootstrap_4.53.00.sh apigeeuser=uName apigeepassword=pWord

Create or edit the file /opt/apigee/customer/application/cassandra.properties.
Add the following contents to the file. ipOfNode is the IP address of the node that Cassandra uses to communicate with other Cassandra nodes:
```
conf_jvm_options_custom_settings=-Dcassandra.replace_address=ipOfNode -Dcassandra.allow_unsafe_replace=true
```

Ensure the file is owned and readable by the apigee user:

chown apigee:apigee /opt/apigee/customer/application/cassandra.properties

Install and set up Cassandra:

# Install cassandra version 4.0.X
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra install
# Setup cassandra while passing standard configuration file
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra setup -f configFile
# Ensure Cassandra version is correct and service is running
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra version
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra status

Verify that the node has started. Check the following command on this node and other nodes in the cluster. The node should report that it is in the "UN" (Up/Normal) state:
```
/opt/apigee/apigee-cassandra/bin/nodetool status
```
Remove the extra configurations added earlier from the file /opt/apigee/customer/application/cassandra.properties.
Repeat steps 3 to 10 on all Cassandra seed nodes in the data center, one by one.
Repeat steps 3 to 10 on all remaining Cassandra nodes in the data center, one by one.
Rebuild all the nodes in the data center from a data center running the older Cassandra version. Perform this step one node at a time:
```
/opt/apigee/apigee-cassandra/bin/nodetool rebuild -dc <name of working DC>
```
This procedure may take time. You can adjust streamingthroughput if needed. Check nodetool netstats for the status of operation completion.
(Optional) Execute the repair command in the Cassandra node if data is not getting rebuilt.
```
/opt/apigee/apigee-cassandra/bin/nodetool -h node-IP repair -pr
```

Restart all edge-* components in the data center, one by one:

/opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart
/opt/apigee/apigee-service/bin/apigee-service edge-router restart
/opt/apigee/apigee-service/bin/apigee-service edge-management-server restart
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restart
/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server restart

Validate and divert traffic back to this data center. Run some validations for runtime traffic and management APIs in this data center, and start rerouting proxy and management API traffic back to it.
Repeat the above steps for each data center you want to roll back.

Roll back Cassandra using Backup

Prerequisites

You are in the process of upgrading Cassandra from 4.0.X to 5.0.X and have encountered issues during the upgrade.
You have backups for the node you are rolling back. The backup was taken before the upgrade from 4.0.X to 5.0.X was attempted.

Steps

Select one node you want to roll back. If you are rolling back all nodes in a data center using backups, start with the seed nodes first. Refer to the section How to identify seed nodes in the Appendix.

Stop, uninstall, and clean up the Cassandra node:

# Stop Cassandra service on the node
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop

# Uninstall Cassandra software
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra uninstall

# Wipe Cassandra data
rm -rf /opt/apigee/data/apigee-cassandra

Install the older Cassandra software on the node and configure it:

Execute the bootstrap file for Edge for Private Cloud 4.53.00/4.53.01:

# Download bootstrap for 4.53.00
curl https://software.apigee.com/bootstrap_4.53.00.sh -o /tmp/bootstrap_4.53.00.sh -u uName:pWord

# Execute bootstrap for 4.53.00
sudo bash /tmp/bootstrap_4.53.00.sh apigeeuser=uName apigeepassword=pWord

Create or edit the file /opt/apigee/customer/application/cassandra.properties:

conf_jvm_options_custom_settings=-Dcassandra.replace_address=ipOfNode -Dcassandra.allow_unsafe_replace=true

Ensure the file is owned by the apigee user and is readable:

chown apigee:apigee /opt/apigee/customer/application/cassandra.properties

Install and set up Cassandra:

# Install Cassandra version 4.0.X
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra install

# Set up Cassandra with the standard configuration file
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra setup -f configFile

# Verify Cassandra version and check service status
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra version
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra status

Verify that the node has started. Check the following command on this node and other nodes in the cluster. Nodes should report that this node is in the "UN" state:

/opt/apigee/apigee-cassandra/bin/nodetool status

Stop the Cassandra service and restore the backup. Refer to the backup and restore documentation for more details:

# Stop Cassandra service on the node
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop

# Wipe the data directory in preparation for restore
rm -rf /opt/apigee/data/apigee-cassandra/data

# Restore the backup taken before the upgrade attempt
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restore backupFile

Once the backup is restored, remove the additional configurations:

Remove the configuration added earlier from the file /opt/apigee/customer/application/cassandra.properties.

Start the Cassandra service on the node:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra start

Repeat the steps on each Cassandra node you wish to roll back using backups, one at a time.

Once all Cassandra nodes are restored, restart all edge-* components one by one:

/opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart
/opt/apigee/apigee-service/bin/apigee-service edge-router restart
/opt/apigee/apigee-service/bin/apigee-service edge-management-server restart
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restart
/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server restart

Backup optimizations (advanced option)

You can potentially minimize (or eliminate) data loss while restoring backups if you have replicas available that contain the latest data. If replicas are available, after restoring the backup, run a repair on the node that was restored.

Appendix

How to identify seed nodes

On any Cassandra node in a data center, run the following command:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra configure -search conf_cassandra_seeds

The command will output multiple lines. Look for the last line of the output. The IP addresses listed in the last line are the seed nodes. In the example below, DC-1-IP1, DC-1-IP2, DC-2-IP1, and DC-2-IP2 are the seed node IPs:

Found key conf_cassandra_seeds, with value, "127.0.0.1", in /opt/apigee/apigee-cassandra/token/default.properties

Found key conf_cassandra_seeds, with value, 127.0.0.1, in /opt/apigee/apigee-cassandra/token/application/cassandra.properties

Found key conf_cassandra_seeds, with value, "DC-1-IP1, DC-1-IP2, DC-2-IP1, DC-2-IP2", in /opt/apigee/token/application/cassandra.properties
apigee-configutil: apigee-cassandra: # OK

Roll back ZooKeeper

Rollback

In case a rollback is required:

Perform rollback steps on observers and followers first. Please refer to Identify leader, followers and observers for selecting observers and followers.
Download and execute bootstrap of the version you're rolling back to—either 4.53.00/4.53.01. The process will likely vary depending on whether the node has an external internet connection or you're following offline installation.

Stop Zookeeper if it is running on the node:

  /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper stop

Uninstall existing zookeeper:

  /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper uninstall

Install Zookeeper as usual:

  /opt/apigee/apigee-setup/bin/setup.sh -p zk -f <silent-config-file>

Once all followers and observers have been rolled back, roll back the leader node by following steps 2 through 5 on the leader node.
After all the nodes have been rolled back, verify the cluster health and ensure there is a leader node in the cluster.

Restore backup

Refer to Restore from a backup.

Roll back Qpid

Note: Roll back apigee-qpidd first, before you attempt to roll back other Edge components (edge-qpid-server, edge-message-processor, edge-management-server).

To roll back to a previous Feature Release, make sure to download the bootstrap.sh file for the version to which you want to roll back. To roll back to v 4.53.00/4.53.01, download bootstrap_4.53.00.sh.

To roll back Qpid, follow these steps on all Qpid hosts:

Stop the existing Qpid broker:
```
apigee-service apigee-qpidd stop
```
Block incoming traffic on broker port 5672 from all message processors by applying a firewall. You can apply this firewall at the level of the Qpid node instance or some other external firewall/network component.
We recommend that you perform the same step for all the message processor IP addresses. For example, to DROP requests coming from message processor IP addresses to the Qpid node on port 5672 using IPTables, you could use a command like this:
```
iptables -A INPUT -p tcp --dport 5672 -s MESSAGE_PROCESSOR_IP -j DROP
```
Start the Qpid broker again to drain existing messages if any:
```
apigee-service apigee-qpidd start
```
Make sure the existing queues are empty. To check, log in to the Qpid management portal:
```
http://QPID_NODE_IP:8090
```
Note: In case this port 8090 on QPID node is not accessible, you can use alternate mechanisms like SSH port forwarding to access this URL.

After verifying queues have drained, stop and uninstall Qpid:

/opt/apigee/apigee-service/bin/apigee-service apigee-qpidd stop
/opt/apigee/apigee-service/bin/apigee-service apigee-qpidd uninstall

Delete the Qpid data directory:
```
rm -r APIGEE_ROOT/data/apigee-qpidd
```

Reinstall the Qpid broker:

/opt/apigee/apigee-setup/bin/setup.sh -p qs -f configFile

Once Qpid broker is reinstalled, remove firewall settings and list any existing settings using following commands:
```
iptables -F
iptables -L
```

Roll back Postgres

To downgrade from version 4.54.00 to version 4.53.00/4.53.01, you must roll back your Postgres update in addition to the Edge components.

To roll back the Postgres update when updating Postgres in a master-standby configuration:

Promote the new standby node to become the Postgres master. The new Postgres master will be the same version as your previous Edge installation.
Configure the old standby node to be a standby node of the new master. The old standby node will be the same version as your previous Edge installation.
Register the new master and standby nodes with the analytics and consumer groups.

When you are done with the rollback, the old master node will no longer be necessary. You can then decommission the old master node.

Before you begin

Before performing a rollback of Postgres 18 to 17/14, do the following steps on both the new standby host and old standby host, to update the max_locks_per_transaction property on apigee-postgresql:

If not present, create the file /opt/apigee/customer/application/postgresql.properties.

Change the ownership of this file to apigee:

sudo chown apigee:apigee /opt/apigee/customer/application/postgresql.properties

Add the following property to the file:

conf/postgresql.conf+max_locks_per_transaction=30000

Configure apigee-postgresql:

apigee-service apigee-postgresql configure

Restart apigee-postgresql:

apigee-service apigee-postgresql restart

Detailed rollback procedure

Make sure the new standby Postgres node is running:

/opt/apigee/apigee-service/bin/apigee-all status

If Postgres is not running, start it:

/opt/apigee/apigee-service/bin/apigee-all start

Make sure Postgres is stopped on the old master node and old standby node:

/opt/apigee/apigee-service/bin/apigee-all status

If Postgres is running, stop it:

/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop  
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql stop

If installed, start Qpid on the old standby node:
```
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server start
```
Note: In many configurations, the old standby node will only be hosting Postgres but not Qpid.
Promote the new standby node as the Postgres master:
1. Promote the new standby node to be the new master:
```
apigee-service apigee-postgresql promote-standby-to-master old_master_IP
```
  If prompted, enter the Postgres password for the 'apigee' user, which defaults to "postgres".
2. Edit the config file that you used to install your current version of Edge to specify the following:
```
# IP address of the new master:
PG_MASTER=new_standby_IP
# IP address of the old standby node
PG_STANDBY=old_standby_IP
```
3. Configure the new master:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup-replication-on-master -f configFile
```
If you have already upgraded the old standby node to the newer version, you must first downgrade Apigee software on the old standby node. If you still have the older version on the old standby node, you can skip this step and continue with step 6.
1. Stop Postgres on the old standby node:
```
apigee-service apigee-postgresql stop
apigee-service edge-postgres-server stop
```
2. Uninstall Postgres from the old standby node:
```
apigee-service apigee-postgresql uninstall
apigee-service edge-postgres-server uninstall
apigee-service edge-analytics uninstall
```
3. Delete the Postgres data directory from the old standby node:
```
cd /opt/apigee/data/apigee-postgresql/pgdata
rm -rf *
```
4. Download and run the older version bootstrap (for the Apigee version you are rolling back to) on the old standby node. The exact steps may vary based on whether you're using internet-based or offline installation. Running the older version Apigee bootstrap will set up the yum repositories with older version Apigee data.
5. Set up Postgres components on the old standby node:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ps -f configFile
```
6. Check and verify that Postgres components on the old standby node have been rolled back to the older version:
```
apigee-service apigee-postgresql version
apigee-service edge-postgres-server version
```

Note: Ensure that you remove the RPM postgresql18-libs-18.3-1PGDG if it still exists.

Rebuild the old standby node:

Edit the config file that you used to install your current version of Edge to specify the following:

# IP address of the new master:
PG_MASTER=new_standby_IP
# IP address of the old standby node
PG_STANDBY=old_standby_IP

Remove the data directory on the old standby node:

cd /opt/apigee/data/apigee-postgresql/pgdata
rm -rf *

Reconfigure the old standby node to be a standby node of the new master:

/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup-replication-on-standby -f configFile

Make sure Postgres is running on the old standby node:

/opt/apigee/apigee-service/bin/apigee-all status

If Postgres is not running, start it:

/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server start

Verify that the new standby node was added by viewing the /opt/apigee/apigee-postgresql/conf/pg_hba.conf file on the new master.

View the current analytics and consumer group information by running the following command on the Management Server:

curl -u sysAdminEmail:password http://ms_IP:8080/v1/analytics/groups/ax

This command returns the analytics group name in the name field, and the consumer group name in the name field under consumer-groups. It also returns the UUIDs of the old Postgres master and standby nodes in the postgres-server field, and in the datastores field. You should see output in the form:

{
  "name" : "axgroup-001",
  "properties" : {
  },
  "scopes" : [ "VALIDATE~test", "sgilson~prod" ],
  "uuids" : {
    "qpid-server" : [ "8381a053-433f-4382-bd2a-100fd37a1592", "4b6856ec-ef05-498f-bac6-ef5f0d5f6521" ],
    "postgres-server" : [
      "ab1158bd-1d59-4e2a-9c95-24cc2cfa6edc:27f90844-efab-4b32-8a23-8f85cdc9a256"
    ]
  },
  "consumer-groups" : [ {
    "name" : "consumer-group-001",
    "consumers" : [ "8381a053-433f-4382-bd2a-100fd37a1592", "4b6856ec-ef05-498f-bac6-ef5f0d5f6521" ],
    "datastores" :
      [ "ab1158bd-1d59-4e2a-9c95-24cc2cfa6edc:27f90844-efab-4b32-8a23-8f85cdc9a256" ],
      "properties" : {     }
    }
  ],
  "data-processors" : {
  }
}

Get the UUID address of the old master by running the following curl command on the old master node:
```
curl -u sysAdminEmail:password http://node_IP:8084/v1/servers/self
```
You should see the UUID of the node at the end of the output, in the form:
```
"type" : [ "postgres-server" ],
"uUID" : "599e8ebf-5d69-4ae4-aa71-154970a8ec75"
```
Note: The edge-postgres-server service must be running. If the Postgres server is not running, you can run the following command on the Management Server to determine the UUID:
```
curl -u sysAdminEmail:password http://ms_IP:8080/v1/servers?pod=analytics
```
The output of this command lists the UUID for the IP address of each Postres node.
Repeat previous step to get the IP addresses of the old standby node and the new master.
Remove old master and standby nodes from the consumer group:
```
curl -u sysAdminEmail:password -X DELETE \
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/consumer-groups/consumer-group-001/datastores/masterUUID,standbyUUID" -v
```
Where axgroup-001 and consumer-group-001 are the default names of the analytics and consumer groups. masterUUID,standbyUUID are in the same order as they appeared above when you viewed the current analytics and consumer group information above. You might have to specify them as standbyUUID,masterUUID.

The datastores property for consumer-groups should now be empty.

Remove the old master and standby nodes from the analytics group:

curl -u sysAdminEmail:password -X DELETE \
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/servers?uuid=masterUUID,standbyUUID&type=postgres-server" -v

The postgres-server property under uuids should now be empty.

curl -u sysAdminEmail:password -X POST -H "Content-Type: application/json" -d ''
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/servers?uuid=masterUUID,standbyUUID&type=postgres-server" -v
curl -u sysAdminEmail:password -X POST -H "Content-Type:application/json" -d ''
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/consumer-groups/consumer-group-001/datastores?uuid=masterUUID,standbyUUID" -v

Validate the analytics group:
```
curl -u sysAdminEmail:password http://ms_IP:8080/v1/analytics/groups/ax
```
You should see the UUIDs of the new master and standby nodes listed in the analytics group and the consumer group.

Restart the Edge Management Server:

/opt/apigee/apigee-service/bin/apigee-service edge-management-server restart

Restart all Qpid servers:

/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restart

Restart all Postgres servers:

/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server restart

Verify the replication status by issuing the following scripts on both servers. The system should display identical results on both servers to ensure a successful replication:
On the new master, run:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql postgres-check-master
```
Verify that it is the master. On the old standby node, run:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql postgres-check-standby
```
Verify that it is the standby.
Repeat the previous step after making several API requests to ensure that the nodes are in synch.
Decommission the old Postgres master:
Note: If the old master node was running Qpid, you can leave that server up to run Qpid. Ensure that it is running. If not, start it:
```
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server start
/opt/apigee/apigee-service/bin/apigee-service apigee-qpidd start
```
Alternatively, you can uninstall Qpid from the old master and install Qpid on the new master node. After uninstalling Qpid, you can decommission the old master node.

Roll back New Edge UI

Apigee Edge for Private Cloud 4.54.00 replaces the Classic UI (edge-ui) with the New Edge UI (edge-management-ui). Because the UI component changes in 4.54.00, rolling back to version 4.53.00 or 4.53.01 requires uninstalling edge-management-ui and reinstalling your previous UI profile:

Classic UI (edge-ui): Reinstall the ui profile.
New Edge UI (edge-ui and edge-management-ui): Reinstall both the ui and ue profiles in their respective nodes.

To roll back the UI on a node:

Stop and uninstall the 4.54.00 New Edge UI and install the previous version to roll back:

Stop and uninstall the 4.54.00 New Edge UI (edge-management-ui):

/opt/apigee/apigee-service/bin/apigee-service edge-management-ui stop
/opt/apigee/apigee-service/bin/apigee-service edge-management-ui uninstall
/opt/apigee/apigee-service/bin/apigee-service edge-management-ui-static uninstall

Execute the bootstrap file for Apigee Edge for Private Cloud 4.53.00/4.53.01:

# Download bootstrap for 4.53.00
curl https://software.apigee.com/bootstrap_4.53.00.sh -o /tmp/bootstrap_4.53.00.sh -u uName:pWord

# Execute bootstrap for 4.53.00
sudo bash /tmp/bootstrap_4.53.00.sh apigeeuser=uName apigeepassword=pWord

Roll back apigee-setup from 4.54.00 to the previous version:

# Uninstall apigee-setup
/opt/apigee/apigee-service/bin/apigee-service apigee-setup uninstall

# Install apigee-setup
/opt/apigee/apigee-service/bin/apigee-service apigee-setup install

Restore your 4.53.00/4.53.01 UI configurations:
- If you backed up the silent configuration file from your 4.53.00 or 4.53.01 installation, use that backup file for the rollback procedure. Otherwise, in your 4.54.00 silent configuration file, restore or uncomment the 4.53.00 or 4.53.01 UI properties that were removed or commented out during the upgrade. For reference, see the UI upgrade section.
- If /opt/apigee/customer/application/ui.properties was deleted during the upgrade, restore it from the backup directory:
```
cp -rf /opt/apigee/backup/ui/customer/application/ui.properties /opt/apigee/customer/application/
chown apigee:apigee /opt/apigee/customer/application/ui.properties
```
Reinstall and configure the previous UI version:
- Roll back to Classic UI (edge-ui):
  1. Install and set up Classic UI (for Basic Auth, no additional steps are required):
```
/opt/apigee/apigee-setup/bin/setup.sh -p ui -f configFile
```
  2. If SSO is enabled, configure SSO on Classic UI (ensure apigee-sso is running first):
```
/opt/apigee/apigee-service/bin/apigee-service edge-ui configure-sso -f configFile
```
- Roll back to 4.53.00 or 4.53.01 New Edge UI (edge-management-ui and Shoehorn edge-ui):
  1. Ensure EDGEUI_ENABLE_UNIFIED_UI=y is set in the silent configuration file, then install and set up the ui and ue profiles:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ui -f configFile
/opt/apigee/apigee-setup/bin/setup.sh -p ue -f configFile
```
  2. If SSO is enabled, configure SSO on both UI components (ensure apigee-sso is running first):
```
/opt/apigee/apigee-service/bin/apigee-service edge-ui configure-sso -f configFile
/opt/apigee/apigee-service/bin/apigee-service edge-management-ui configure-sso -f configFile
```
Note: In a multi-node topology in which the Classic UI and the 4.53.01 New Edge UI were hosted on separate nodes, execute the rollback commands on their respective hosts. The configure-sso command must be executed only if SSO was configured in the previous version.
- Node 1 (Classic UI node): Run the Classic UI commands:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ui -f configFile
/opt/apigee/apigee-service/bin/apigee-service edge-ui configure-sso -f configFile
```
- Node 2 (New Edge UI node): Run the New Edge UI commands (setup.sh -p ue and edge-management-ui configure-sso):
```
/opt/apigee/apigee-setup/bin/setup.sh -p ue -f configFile
/opt/apigee/apigee-service/bin/apigee-service edge-ui configure-sso -f configFile
/opt/apigee/apigee-service/bin/apigee-service edge-management-ui configure-sso -f configFile
```

Verify the rolled-back UI status and version:

For Classic UI (edge-ui):

/opt/apigee/apigee-service/bin/apigee-service edge-ui status
/opt/apigee/apigee-service/bin/apigee-service edge-ui version

For New Edge UI (edge-management-ui):

/opt/apigee/apigee-service/bin/apigee-service edge-management-ui status
/opt/apigee/apigee-service/bin/apigee-service edge-management-ui version

Note: In Apigee Edge for Private Cloud versions earlier than 4.54.00, the Classic Edge UI (edge-ui) and the New Edge UI (edge-management-ui) couldn't coexist on the same node. Administrators were required to provision separate, dedicated nodes for each UI component when running those earlier versions.

Roll back OpenLDAP

This section provides a step-by-step guide for rolling back Symas OpenLDAP 2.6.10 in Edge for Private Cloud 4.54.00 to either OpenLDAP 2.4.46 (Edge for Private Cloud 4.53.00) or Symas OpenLDAP 2.6.7 (Edge for Private Cloud 4.53.01). You must perform the rollback on the entire LDAP cluster using a valid pre-upgrade LDAP backup.

The primary goal is to revert the entire LDAP cluster to a consistent state from a known-good backup. This procedure is the same for all scenarios: single-server, active-active, and active-passive.

Before you begin

Backup incompatibility: Use a pre-upgrade backup from OpenLDAP 2.4.46 (Edge for Private Cloud 4.53.00) or Symas OpenLDAP 2.6.7 (Edge for Private Cloud 4.53.01) that was taken using apigee-service apigee-openldap backup before the upgrade attempt. Backups taken after the upgrade are incompatible and cannot be used.
Management API downtime: During the LDAP rollback, management APIs will be unavailable, which may impact administrative tasks. Shut down all edge-management-server and edge-ui instances before proceeding with the LDAP rollback.
Note: If you are using the New Edge UI (edge-management-ui), ensure that you stop both the Classic UI (edge-ui) and the New Edge UI (edge-management-ui).
Risk of data loss: Proceeding without a valid, compatible backup risks irreversible data loss.
Valid backup: A valid pre-upgrade LDAP backup is required.
Backup of silent.conf files: Retain backups of your silent.conf files from both the previous and current versions to prevent configuration mix-ups during the upgrade and rollback procedures.
Rollback sequence: Complete the rollback procedures for Cassandra, ZooKeeper, Qpid, PostgreSQL, apigee-sso, and the Edge UI before rolling back OpenLDAP.

Rollback procedure

Shut down the Management Server and UI: Shut down all edge-management-server and edge-ui instances before proceeding with the LDAP rollback:
```
apigee-service edge-management-server stop
apigee-service edge-ui stop
```
Record the current LDAP entry count (before stopping LDAP): Get the total record count for reference from all LDAP servers, replacing YOUR_PASSWORD with your current LDAP manager password. (The record count must match across all LDAP servers.)
```
# Note: Replace 'YOUR_PASSWORD' with your current LDAP manager password.
ldapsearch -o ldif-wrap=no -b "dc=apigee,dc=com" -D "cn=manager,dc=apigee,dc=com" -H ldap://:10389 -LLL -x -w 'YOUR_PASSWORD' | wc -l
```
Stop and uninstall new LDAP 2.6: Stop the apigee-openldap service and delete its configuration and data directories. Perform this step on all LDAP servers, one node at a time in the cluster, to ensure consistency:
```
apigee-service apigee-openldap stop
apigee-service apigee-openldap uninstall
rm -rf /opt/apigee/data/apigee-openldap/*
```

Install previous LDAP version 2.4.46 or 2.6.7:

Install old LDAP version:

/opt/apigee/apigee-setup/bin/setup.sh -p ld -f /opt/silent.conf

Validate LDAP version:

source ~/.bash_profile
ldapsearch -VV

# Expected output if rolled back to 4.53.00:
ldapsearch: @(#) $OpenLDAP: ldapsearch 2.4.46

# Expected output if rolled back to 4.53.01:
ldapsearch: @(#) $OpenLDAP: ldapsearch 2.6.7

Repeat the preceding substeps in step 4 on each LDAP node, one at a time.

Clean residual data: On all LDAP servers, one at a time, stop the newly installed service and clean its data directory to prepare for the restore from backup:
```
apigee-service apigee-openldap stop
rm -rf /opt/apigee/data/apigee-openldap/*
```
Restore LDAP data: For a single-server setup, proceed directly to step 6b. For a multi-server setup, proceed with step 6a.
1. Identify the active LDAP server: Before restoring LDAP data, identify the active server (provider) by running this command:
```
grep -i '^olcSyncrepl:' /opt/apigee/data/apigee-openldap/slapd.d/cn=config/olcDatabase*.ldif
```
  Note:
  1. If this command returns output, the server is a passive server.
  2. If this command returns no output, the server is the active server.
2. Restore LDAP data (ensure step 5 is completed across all servers before restoring):
  - On the single or active LDAP server (determined in step 6a), navigate to the backup directory and restore the pre-upgrade backup (for example, 2025.07.28,13.59.00, or omit the timestamp if the pre-upgrade backup is the latest backup in the directory):
```
cd /opt/apigee/backup/openldap

# To restore a specific backup, provide the timestamp as shown below:
apigee-service apigee-openldap restore 2025.07.28,13.59.00

# Note: If no timestamp is provided, the latest available backup will be restored by default:
apigee-service apigee-openldap restore

# It is recommended to specify the backup explicitly to avoid restoring an unintended version.
```
  - Start the apigee-openldap service:
```
apigee-service apigee-openldap start
```
Start remaining LDAP servers: If you have a multi-server setup, on each of the remaining LDAP servers, start the service:
```
apigee-service apigee-openldap start
```
Note: You do not need to restore data manually on these peer servers. Upon starting, they will connect to the first server (which now contains the restored data) and automatically synchronize the entire directory through replication.
Final validation:
- Verify that the rollback was successful and that data is consistent across the entire LDAP cluster.
- Run the validation command on all LDAP servers, replacing YOUR_PASSWORD with your current LDAP manager password. The record count must be identical across all servers and must match the count you captured in step 2:
```
# Note: Replace 'YOUR_PASSWORD' with your current LDAP manager password.
ldapsearch -o ldif-wrap=no -b "dc=apigee,dc=com" -D "cn=manager,dc=apigee,dc=com" -H ldap://:10389 -LLL -x -w 'YOUR_PASSWORD' | wc -l
```
- After you confirm that the data is correct and consistent, start edge-management-server, edge-ui, and any other dependent components according to your organization's standard procedure.

Roll back Edge Components

Components with common code

The following Edge components share common code. Therefore, to roll back any one of these components on a node, you must roll back all of these components that are on that node:

edge-management-server (Management Server)
edge-message-processor (Message Processor)
edge-router (Router)
edge-postgres-server (Postgres Server)
edge-qpid-server (Qpid Server)

For example, if you have the Management Server, Router, and Message Processor installed on the node, to roll back any one of them you must roll back all three.

To roll back to a previous major or minor release, do the following on each node that hosts the component:

Download the bootstrap.sh file for the version to which you want to roll back:
- To roll back to 4.53.00/4.53.01, download bootstrap_4.53.00.sh/bootstrap_4.53.01.sh

Stop the component to roll back:

To roll back any of the components with common code on the node, you must stop them all, as the following example shows:

/opt/apigee/apigee-service/bin/apigee-service edge-management-server stop
/opt/apigee/apigee-service/bin/apigee-service edge-router stop
/opt/apigee/apigee-service/bin/apigee-service edge-message-processor stop
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server stop
/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop

To roll back any other component on the node, stop just that component:
```
/opt/apigee/apigee-service/bin/apigee-service component stop
```

If you are rolling back Monetization, uninstall it from all Management Server and Message Processor nodes:
```
/opt/apigee/apigee-service/bin/apigee-service edge-mint-gateway uninstall
```
Uninstall the component to roll back on the node:
1. To roll back any of the components with common code on the node, you must uninstall them all by uninstalling the edge-gateway and apigee-cassandra-client component group, as the following example shows:
```
/opt/apigee/apigee-service/bin/apigee-service edge-gateway uninstall
```
```
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra-client uninstall
```
2. To roll back Nginx:
```
### Find the apigee-nginx RPM
rpm -qa | grep -i "apigee-nginx"

### Remove the apigee-nginx RPM
dnf remove apigee-nginx-1.30.x
```
3. To roll back any other component on the node, uninstall just that component, as the following example shows:
```
/opt/apigee/apigee-service/bin/apigee-service component uninstall
```
  Where component is the component name.
  Note: If edge-analytics doesn't automatically uninstall alongside edge-qpid-server and edge-postgres-server, uninstall it manually using the following command:
```
/opt/apigee/apigee-service/bin/apigee-service edge-analytics uninstall
```
4. To roll back the Edge Router, you must delete the contents of the /opt/nginx/conf.d file in addition to uninstalling the edge-gateway component group:
```
cd /opt/nginx/conf.d
rm -rf *
```

Uninstall the 4.54.00 version of apigee-setup:

/opt/apigee/apigee-service/bin/apigee-service apigee-setup uninstall

Install the 4.53.00 version of the apigee-service utility and its dependencies. The following example installs the 4.53.00 version of apigee-service:
```
sudo bash /tmp/bootstrap_4.53.00.sh apigeeuser=uName apigeepassword=pWord
```
Where uName and pWord are the username and password you received from Apigee. If you omit pWord, you are prompted to enter it.

If you get an error, be sure you downloaded the bootstrap.sh file in step 1.

Install apigee-setup:

/opt/apigee/apigee-service/bin/apigee-service apigee-setup install

Install the older version of the component:
```
/opt/apigee/apigee-setup/bin/setup.sh -p component -f configFile
```
Where component is the component to install and configFile is your configuration file for the older version.
If you are rolling back Qpid, flush iptables:
```
sudo iptables -F
```
Repeat this process for each node that hosts the component you are rolling back.

Note: Upon uninstallation of apigee-cassandra-client, apigee-provision and apigee-validate are automatically removed due to their dependency. If these are subsequently required, they must be reinstalled.

Roll back mTLS

To roll back the mTLS update, do the following steps on all hosts:

Stop Apigee:
```
apigee-all stop
```
Stop mTLS:
```
apigee-service apigee-mtls uninstall
```

Reinstall mTLS:

apigee-service apigee-mtls install
apigee-service apigee-mtls setup -f /opt/silent.conf