Roll back Apigee Edge 4.53.01

If you encounter an error during an update to Edge 4.53.01, you can roll back the component that caused the error and then try the update again.

You can roll back Edge 4.53.01 to any of the following major release versions:

Version 4.53.00
Version 4.52.02

Rolling back a version involves rolling back every component that you may have upgraded. Additionally, based on the version you started from, you may need to consider special steps before rolling back certain software components. The following table lists the various software components for which special steps may be needed during rollback:

Rollback to version	Special consideration for software
4.53.00	Zookeeper, Postgres, LDAP
4.52.02	Zookeeper, Cassandra, Postgres, LDAP

Here is the scenario where you might want to perform a rollback:

Roll back to a previous major or minor release. For example, from 4.53.01 to 4.53.00.

For more information, see Apigee Edge release process.

Order of rollback

Rollback of components should be done in the reverse order they were upgraded, with the exception that management servers should be rolled back after Cassandra. A typical general order of rollback for Private Cloud 4.53.01 will look like below:

Rollback Qpid, other analytics-related components and Postgres.
Rollback routers and message processors
Rollback Cassandra, Zookeeper
Rollback management server
Rollback LDAP

For example, let's say you had upgraded the entire Cassandra cluster, all your management servers, and a few RMPs to version 4.53.01 from version 4.52.02 and wish to rollback. In this case, you would:

Rollback all RMPs one by one
Rollback the entire Cassandra cluster using backups
Rollback Edge Management server nodes one by one

Who can perform a rollback

The user performing a rollback should be the same as the user who originally updated Edge, or a user running as root.

By default, Edge components run as the user "apigee". In some cases, you might be running Edge components as different users. For example, if the Router has to access privileged ports, such as those below 1000, then you have to run the Router as root or as a user with access to those ports. Or, you might run one component as one user, and another component as another user.

Components with common code

The following Edge components share common code. Therefore, to roll back any one of these components on a node, you must roll back all of these components that are on that node.

edge-management-server (Management Server)
edge-message-processor (Message Processor)
edge-router (Router)
edge-postgres-server (Postgres Server)
edge-qpid-server (Qpid Server)

For example, if you have the Management Server, Router, and Message Processor installed on the node, to roll back any one of them you must roll back all three.

Rollback of Cassandra

When a major upgrade of Cassandra is performed on a specific node, Cassandra modifies the schema of the data stored on that node. As a result, a direct in-place rollback is not feasible.

Rollback scenarios

Cassandra 4.0.X, available with Edge for Private Cloud 4.53.01, is compatible with other components of Private Cloud 4.52.02.

Please refer to the table below for a summary of the various rollback strategies you can use:

Scenario	Rollback strategy
Single DC, some Cassandra nodes upgraded	Use backups
Single DC, all Cassandra nodes upgraded	Do not rollback Cassandra. Other components can be rolled back.
Single DC, all nodes (Cassandra and others) upgraded	Do not rollback Cassandra. Other components can be rolled back.
Multiple DC, some nodes in one DC upgraded	Rebuild from existing DC
Multiple DC, all Cassandra nodes in some DCs upgraded	Rebuild from existing DC
Multiple DC, Cassandra nodes of the last DC being upgraded	Try to finish the upgrade. If not feasible, rollback 1 DC using backup. Rebuild remaining DCs from the rolled-back DC.
Multiple DC, all Cassandra nodes upgraded	Do not rollback Cassandra. Other components can be rolled back.
Multiple DC, all nodes (Cassandra and others) upgraded	Do not rollback Cassandra. Other components can be rolled back.

General considerations

When considering a rollback, keep the following in mind:

Rollback of runtime or management components: If you want to rollback components like edge-management-server, edge-message-processor, or any non-Cassandra component to Private Cloud version 4.52.02, it is recommended that you do NOT rollback Cassandra. Cassandra shipped with Private Cloud 4.53.00 is compatible with all non-Cassandra components of Edge for Private Cloud 4.52.02. You can rollback non-Cassandra components using the methodology listed here while Cassandra remains on version 4.0.13.
Rollback after the entire Cassandra cluster is upgraded to 4.0.X: If your entire Cassandra cluster is upgraded to version 4.0.X as part of the upgrade to Private Cloud version 4.53.00, it is recommended that you continue with this cluster setup and NOT rollback Cassandra. Components like edge-management-server, edge-message-processor, edge-router, etc., of Private Cloud version 4.52.02 are compatible with Cassandra version 4.0.X.
Rollback of Cassandra during the Cassandra upgrade: If you encounter issues during the Cassandra upgrade, you may want to consider a rollback. The rollback strategies listed in this article can be followed based on the state you are in during the upgrade process.
Rollback using backups: Backups taken from Cassandra 4.0.X are not compatible with backups of Cassandra 3.11.X. To rollback Cassandra using backup restoration, you must take backups of Cassandra 3.11.X before attempting the upgrade.

Rollback Cassandra using rebuild

Prerequisites

You are operating an Edge for Private Cloud 4.52.02 cluster across multiple data centers.
You are in the process of upgrading Cassandra from 3.11.X to 4.0.X and have encountered issues during the upgrade.
You have at least one fully functional data center in the cluster still running the older version of Cassandra (Cassandra 3.11.X).

This procedure relies on streaming data from an existing data center. It could take a significant amount of time, depending on how much data is stored in Cassandra. You should be prepared to divert your runtime traffic away from this data center while the rollback is ongoing.

High-level steps

Select one data center (either partially or fully upgraded) that you’d like to roll back. Divert runtime traffic to a different functioning data center.
Identify the seed node in the data center and start with one of the seed nodes.
Stop, uninstall, and clean up the Cassandra node.
Install the older version of Cassandra on the node and configure it as needed.
Remove the extra configurations that were added earlier.
Repeat the above steps for all seed nodes in the data center, one by one.
Repeat the above steps for all remaining Cassandra nodes in the data center, one by one.
Rebuild the nodes from the existing functional data center, one by one.
Restart all edge-* components in the data center that are connected to Cassandra.
Test and divert traffic back to this data center.
Repeat the steps for each data center, one by one.

Detailed steps

Pick one data center where all or some Cassandra nodes are upgraded. Divert all runtime proxy traffic and management traffic from this data center while the Cassandra nodes in this data center are being rolled back. Ensure all Cassandra nodes are in the UN (Up/Normal) state when the nodetool ring command is executed on the nodes. If certain nodes are down, troubleshoot the issue and bring those nodes back up before continuing.

See the example below:

/opt/apigee/apigee-cassandra/bin/nodetool status
Datacenter: dc-1
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens       Owns (effective)  Host ID                               Rack
UN  DC1-1IP1  456.41 KiB  1            100.0%            78fc4ddd-2ed9-4a8c-98a2-63a38c2f1920  ra-1
UN  DC1-1IP2  870.93 KiB  1            100.0%            160db01a-64ab-43a7-b9ea-3b7f8f66d52b  ra-1
UN  DC1-1IP3  824.08 KiB  1            100.0%            21d61543-d59e-403a-bf5d-bfe7f664baa6  ra-1
Datacenter: dc-2
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens       Owns (effective)  Host ID                               Rack
UN  DC2-1IP1   802.08 KiB  1            100.0%            583e0576-336d-4ce7-9729-2ae74e0abde2  ra-1
UN  DC2-1IP2   844.4 KiB   1            100.0%            fef794d5-f4c2-4a4e-bb05-9adaeb4aea4b  ra-1
UN  DC2-1IP3   878.12 KiB  1            100.0%            3894b3d9-1f5a-444d-83db-7b1e338bbfc9  ra-1

You can run nodetool describecluster on the nodes to understand the current state of the entire cluster. For example, the following shows an instance of a 2-data-center cluster where all DC-1 nodes are on Cassandra version 4, whereas all DC-2 nodes are on Cassandra version 3:

# On nodes where Cassandra is upgraded
/opt/apigee/apigee-cassandra/bin/nodetool describecluster
Cluster Information:
    Name: Apigee
    Snitch: org.apache.cassandra.locator.PropertyFileSnitch
    DynamicEndPointSnitch: enabled
    Partitioner: org.apache.cassandra.dht.RandomPartitioner
    Schema versions:
        2eadcd74-0245-309a-9992-3625afa70038: [DC-1-IP1, DC-1-IP2, DC-1-IP3]
        129dc15e-198e-3c11-b64c-701044a3a1ad: [DC-2-IP1, DC-2-IP2, DC-2-IP3]

Stats for all nodes:
    Live: 6
    Joining: 0
    Moving: 0
    Leaving: 0
    Unreachable: 0

Data Centers:
    dc-1 #Nodes: 3 #Down: 0
    dc-2 #Nodes: 3 #Down: 0

Database versions:
    4.0.13: [DC-1-IP1:7000, DC-1-IP2:7000, DC-1-IP3:7000]
    3.11.16: [DC-2-IP1:7000, DC-2-IP2:7000, DC-2-IP3:7000]

Keyspaces:
    system_schema -> Replication class: LocalStrategy {}
    system -> Replication class: LocalStrategy {}
    auth -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    cache -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    devconnect -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    dek -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    user_settings -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    apprepo -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    kms -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    identityzone -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    audit -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    analytics -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    keyvaluemap -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    counter -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    apimodel_v2 -> Replication class: NetworkTopologyStrategy {dc-2=3, dc-1=3}
    system_distributed -> Replication class: SimpleStrategy {replication_factor=3}
    system_traces -> Replication class: SimpleStrategy {replication_factor=2}
    system_auth -> Replication class: SimpleStrategy {replication_factor=1}

# On nodes where Cassandra is not upgraded
/opt/apigee/apigee-cassandra/bin/nodetool describecluster
Cluster Information:
    Name: Apigee
    Snitch: org.apache.cassandra.locator.PropertyFileSnitch
    DynamicEndPointSnitch: enabled
    Partitioner: org.apache.cassandra.dht.RandomPartitioner
    Schema versions:
        2eadcd74-0245-309a-9992-3625afa70038: [DC-1-IP1, DC-1-IP2, DC-1-IP3]
        129dc15e-198e-3c11-b64c-701044a3a1ad: [DC-2-IP1, DC-2-IP2, DC-2-IP3]

Identify the seed nodes in the data center: Refer to the section How to identify seed nodes in the Appendix. Execute the steps below on one of the seed nodes:

Stop, uninstall, and clean up data from the node of Cassandra. Pick the first seed node on Cassandra version 4 in this data center. Stop it.

# Stop Cassandra service on the node
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop

# Uninstall Cassandra software
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra uninstall

# Wipe out Cassandra data
rm -rf /opt/apigee/data/apigee-cassandra

Install the older Cassandra software on the node and set some configurations. Execute the bootstrap file of Edge for Private Cloud 4.52.02.

# Download bootstrap of 4.52.02
curl https://software.apigee.com/bootstrap_4.52.02.sh -o /tmp/bootstrap_4.52.02.sh -u uName:pWord

# Execute bootstrap of 4.52.02
sudo bash /tmp/bootstrap_4.52.02.sh apigeeuser=uName apigeepassword=pWord

Set Cassandra configs

Create or edit the file /opt/apigee/customer/application/cassandra.properties.
Add the following contents to the file. ipOfNode is the IP address of the node that Cassandra uses to communicate with other Cassandra nodes:
```
conf_jvm_options_custom_settings=-Dcassandra.replace_address=ipOfNode -Dcassandra.allow_unsafe_replace=true
```

Ensure the file is owned and readable by the apigee user:

chown apigee:apigee /opt/apigee/customer/application/cassandra.properties

Install and set up Cassandra:

Install Cassandra version 3.11.X:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra install

Set up Cassandra by passing the standard configuration file:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra setup -f configFile

Ensure that Cassandra 3.11.X is installed and the service is running:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra version

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra status

Verify that the node has started. Check the following command on this node and other nodes in the cluster. The node should report that it is in the "UN" (Up/Normal) state:
```
/opt/apigee/apigee-cassandra/bin/nodetool status
```
Remove the extra configurations added earlier from the file /opt/apigee/customer/application/cassandra.properties.
Repeat steps 3 to 6 on all Cassandra seed nodes in the data center, one by one.
Repeat steps 3 to 6 on all remaining Cassandra nodes in the data center, one by one.
Rebuild all the nodes in the data center from a data center running the older Cassandra version. Perform this step one node at a time:
```
/opt/apigee/apigee-cassandra/bin/nodetool rebuild -dc <name of working DC>
```
This procedure may take some time. You can adjust the streamingthroughput if necessary. Check the status using:
```
/opt/apigee/apigee-cassandra/bin/nodetool netstats
```

Restart all edge-* components in the data center, one by one:

/opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart
/opt/apigee/apigee-service/bin/apigee-service edge-router restart
/opt/apigee/apigee-service/bin/apigee-service edge-management-server restart
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restart
/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server restart

Validate and divert traffic back to this data center. Run some validations for runtime traffic and management APIs in this data center, and start rerouting proxy and management API traffic back to it.
Repeat the above steps for each data center you want to roll back.

Rollback Cassandra using Backup

Prerequisites

You are in the process of upgrading Cassandra from 3.11.X to 4.0.X and have encountered issues during the upgrade.
You have backups for the node you are rolling back. The backup was taken before the upgrade from 3.11.X to 4.0.X was attempted.

Steps

Select one node you want to roll back. If you are rolling back all nodes in a data center using backups, start with the seed nodes first. Refer to the section "How to Identify Seed Nodes" in the Appendix.

Stop, uninstall, and clean up the Cassandra node:

# Stop Cassandra service on the node
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop

# Uninstall Cassandra software
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra uninstall

# Wipe Cassandra data
rm -rf /opt/apigee/data/apigee-cassandra

Install the older Cassandra software on the node and configure it:

Execute the bootstrap file for Edge for Private Cloud 4.52.02:

# Download bootstrap for 4.52.02
curl https://software.apigee.com/bootstrap_4.52.02.sh -o /tmp/bootstrap_4.52.02.sh -u ‘uName:pWord’

# Execute bootstrap for 4.52.02
sudo bash /tmp/bootstrap_4.52.02.sh apigeeuser=uName apigeepassword=pWord

Create or edit the file /opt/apigee/customer/application/cassandra.properties:

conf_jvm_options_custom_settings=-Dcassandra.replace_address=ipOfNode -Dcassandra.allow_unsafe_replace=true

Ensure the file is owned by the apigee user and is readable:

chown apigee:apigee /opt/apigee/customer/application/cassandra.properties

Install and set up Cassandra:

# Install Cassandra version 3.11.X
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra install

# Set up Cassandra with the standard configuration file
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra setup -f configFile

# Verify Cassandra version and check service status
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra version
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra status

Verify that the node has started. Check the following command on this node and other nodes in the cluster. Nodes should report that this node is in the "UN" state:

/opt/apigee/apigee-cassandra/bin/nodetool status

Stop the Cassandra service and restore the backup. Refer to the backup and restore documentation for more details:

# Stop Cassandra service on the node
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra stop

# Wipe the data directory in preparation for restore
rm -rf /opt/apigee/data/apigee-cassandra/data

# Restore the backup taken before the upgrade attempt
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restore backupFile

Once the backup is restored, remove the additional configurations:

Remove the configuration added earlier from the file /opt/apigee/customer/application/cassandra.properties.

Start the Cassandra service on the node:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra start

Repeat the steps on each Cassandra node you wish to roll back using backups, one at a time.

Once all Cassandra nodes are restored, restart all edge-* components one by one:

/opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart
/opt/apigee/apigee-service/bin/apigee-service edge-router restart
/opt/apigee/apigee-service/bin/apigee-service edge-management-server restart
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restart
/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server restart

Backup optimizations (advanced option)

You can potentially minimize (or eliminate) data loss while restoring backups if you have replicas available that contain the latest data. If replicas are available, after restoring the backup, run a repair on the node that was restored.

Appendix

How to identify seed nodes

On any Cassandra node in a data center, run the following command:

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra configure -search conf_cassandra_seeds

The command will output multiple lines. Look for the last line of the output. The IP addresses listed in the last line are the seed nodes. In the example below, DC-1-IP1, DC-1-IP2, DC-2-IP1, and DC-2-IP2 are the seed node IPs:

Found key conf_cassandra_seeds, with value, "127.0.0.1", in /opt/apigee/apigee-cassandra/token/default.properties

Found key conf_cassandra_seeds, with value, 127.0.0.1, in /opt/apigee/apigee-cassandra/token/application/cassandra.properties

Found key conf_cassandra_seeds, with value, "DC-1-IP1, DC-1-IP2, DC-2-IP1, DC-2-IP2", in /opt/apigee/token/application/cassandra.properties
apigee-configutil: apigee-cassandra: # OK

Rollback the Zookeeper 3.8.4 update

Rollback

In case a rollback is required:

Perform rollback steps on observers and followers first.
Download and execute bootstrap of the version you're rolling back to—either 4.52.02 or 4.53.00. The process will likely vary depending on whether the node has an external internet connection or you're following offline installation.

Stop Zookeeper if it is running on the node:

  /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper stop

Uninstall existing zookeeper:

  /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper uninstall

Install Zookeeper as usual:

  /opt/apigee/apigee-setup/bin/setup.sh -p zk -f <silent-config-file>

Once all followers and observers have been rolled back, roll back the leader node by following steps 2 through 5 on the leader node.
After all the nodes have been rolled back, verify the cluster health and ensure there is a leader node in the cluster.

Restore backup

Refer to Restore from a backup. Note that backups of Zookeeper taken from earlier versions of Edge for Private Cloud like 4.52.02 or 4.53.00 should be compatible with the version of Zookeeper in Edge for Private Cloud 4.53.01

Roll back the Postgres 17 update

If you upgraded to 4.53.01 from version 4.52.02 or 4.53.00, you must roll back your Postgres update in addition to the Edge components.

To rollback the Postgres update when updating Postgres in a master-standby configuration:

Promote the new standby node to become the Postgres master. The new Postgres master will be the same version as your previous Edge installation.
Configure the old standby node to be a standby node of the new master. The old standby node will be the same version as your previous Edge installation.
Register the new master and standby nodes with the analytics and consumer groups.

When you are done with the rollback, the old master node will no longer be necessary. You can then decommission the old master node.

Make sure the new standby Postgres node is running:

/opt/apigee/apigee-service/bin/apigee-all status

If Postgres is not running, start it:

/opt/apigee/apigee-service/bin/apigee-all start

Make sure Postgres is stopped on the old master node and old standby node:

/opt/apigee/apigee-service/bin/apigee-all status

If Postgres is running, stop it:

/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop > /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql stop

If installed, start Qpid on the old standby node:
```
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server start
```
Note: In many configurations, the old standby node will only be hosting Postgres but not Qpid.
Promote the new standby node as the Postgres master:
1. Promote the new standby node to be the new master:
```
apigee-service apigee-postgresql promote-standby-to-master new_standby_IP
```
  If prompted, enter the Postgres password for the 'apigee' user, which defaults to "postgres".
2. Edit the config file that you used to install your current version of Edge to specify the following:
```
# IP address of the new master:
PG_MASTER=new_standby_IP
# IP address of the old standby node
PG_STANDBY=old_standby_IP
```
3. Configure the new master:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup-replication-on-master -f configFile
```
If you have already upgraded the old standby node to the newer version, you must first downgrade Apigee software on the old standby node. If you still have the older version on the old standby node, you can skip this step and continue with step 6.
1. Stop Postgres on the old standby node:
```
apigee-service apigee-postgresql stop
apigee-service edge-postgres-server stop
```
2. Uninstall Postgres from the old standby node:
```
apigee-service apigee-postgresql uninstall
apigee-service edge-postgres-server uninstall
```
3. Delete the Postgres data directory from the old standby node:
```
cd /opt/apigee/data/apigee-postgresql/pgdata > rm -rf *
```
4. Download and run the older version bootstrap (for the Apigee version you are rolling back to) on the old standby node. The exact steps may vary based on whether you're using internet based or offline installation. Running the older version Apigee bootstrap will set up the yum repositories with older version Apigee data.
5. Set up Postgres components on the old standby node:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ps -f configFile
```
6. Check and verify that Postgres components on the old standby node have been rolled back to the older version:
```
apigee-service apigee-postgresql version
apigee-service edge-postgres-server version
```

Rebuild the old standby node:

Edit the config file that you used to install your current version of Edge to specify the following:

# IP address of the new master:
PG_MASTER=new_standby_IP
# IP address of the old standby node
PG_STANDBY=old_standby_IP

Remove the data directory on the old standby node:

cd /opt/apigee/data/apigee-postgresql/pgdata > rm -rf *

Reconfigure the old standby node to be a standby node of the new master:

/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup-replication-on-standby -f configFile

Make sure Postgres is running on the old standby node:

/opt/apigee/apigee-service/bin/apigee-all status

If Postgres is not running, start it:

/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server start

Verify that the new standby node was added by viewing the /opt/apigee/apigee-postgresql/conf/pg_hba.conf file on the new master.

View the current analytics and consumer group information by running the following command on the Management Server:

curl -u sysAdminEmail:password http://ms_IP:8080/v1/analytics/groups/ax

This command returns the analytics group name in the name field, and the consumer group name in the name field under consumer-groups. It also returns the UUIDs of the old Postgres master and standby nodes in the postgres-server field, and in the datastores field. You should see output in the form:

{
  "name" : "axgroup-001",
  "properties" : {
  },
  "scopes" : [ "VALIDATE~test", "sgilson~prod" ],
  "uuids" : {
    "qpid-server" : [ "8381a053-433f-4382-bd2a-100fd37a1592", "4b6856ec-ef05-498f-bac6-ef5f0d5f6521" ],
    "postgres-server" : [
      "ab1158bd-1d59-4e2a-9c95-24cc2cfa6edc:27f90844-efab-4b32-8a23-8f85cdc9a256"
    ]
  },
  "consumer-groups" : [ {
    "name" : "consumer-group-001",
    "consumers" : [ "8381a053-433f-4382-bd2a-100fd37a1592", "4b6856ec-ef05-498f-bac6-ef5f0d5f6521" ],
    "datastores" :
      [ "ab1158bd-1d59-4e2a-9c95-24cc2cfa6edc:27f90844-efab-4b32-8a23-8f85cdc9a256" ],
      "properties" : {     }
    }
  ],
  "data-processors" : {
  }
}

Get the UUID address of the old master by running the following curl command on the old master node:
```
curl -u sysAdminEmail:password http://node_IP:8084/v1/servers/self
```
You should see the UUID of the node at the end of the output, in the form:
```
"type" : [ "postgres-server" ],
"uUID" : "599e8ebf-5d69-4ae4-aa71-154970a8ec75"
```
Note: The edge-postgres-server service must be running. If the Postgres server is not running, you can run the following command on the Management Server to determine the UUID:
```
curl -u sysAdminEmail:password http://ms_IP:8080/v1/servers?pod=analytics
```
The output of this command lists the UUID for the IP address of each Postres node.
Repeat previous step to get the IP addresses of the old standby node and the new master.
Remove old master and standby nodes from the consumer group:
```
curl -u sysAdminEmail:password -X DELETE \
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/consumer-groups/consumer-group-001/datastores/masterUUID,standbyUUID" -v
```
Where axgroup-001 and consumer-group-001 are the default names of the analytics and consumer groups. masterUUID,standbyUUID are in the same order as they appeared above when you viewed the current analytics and consumer group information above. You might have to specify them as standbyUUID,masterUUID.

The datastores property for consumer-groups should now be empty.

Remove the old master and standby nodes from the analytics group:

curl -u sysAdminEmail:password -X DELETE \
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/servers?uuid=masterUUID,standbyUUID&type=postgres-server" -v

The postgres-server property under uuids should now be empty.

curl -u sysAdminEmail:password -X POST -H "Content-Type: application/json" -d ''
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/servers?uuid=masterUUID,standbyUUID&type=postgres-server" -v
curl -u sysAdminEmail:password -X POST -H "Content-Type:application/json" -d ''
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/consumer-groups/consumer-group-001/datastores?uuid=masterUUID,standbyUUID" -v

Validate the analytics group:
```
curl -u sysAdminEmail:password http://ms_IP:8080/v1/analytics/groups/ax
```
You should see the UUIDs of the new master and standby nodes listed in the analytics group and the consumer group.

Restart the Edge Management Server:

/opt/apigee/apigee-service/bin/apigee-service edge-management-server restart

Restart all Qpid servers:

/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restart

Restart all Postgres servers:

/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server restart

Verify the replication status by issuing the following scripts on both servers. The system should display identical results on both servers to ensure a successful replication:
On the new master, run:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql postgres-check-master
```
Verify that it is the master. On the old standby node:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql postgres-check-standby
```
Verify that it is the standby.
Repeat the previous step after making several API requests to ensure that the nodes are in synch.
Decommission the old Postgres master using the procedure in Decommissioning a Postgres node.
Note: If the old master node was running Qpid, you can leave that server up to run Qpid. Ensure that it is running. If not, start it:
```
/opt/apigee/apigee-service/bin/apigee-service edge-management-server start
```
Alternatively, you can uninstall Qpid from the old master and install Qpid on the new master node. After uninstalling Qpid, you can decommission the old master node.

Rollback the LDAP 2.6 update

This section provides a comprehensive, step-by-step guide for rolling back a LDAP installation to a previous LDAP version. The rollback must be performed on the entire LDAP cluster and is only possible using a valid pre-upgrade LDAP backup.

The primary goal is to revert the entire LDAP cluster to a consistent state from a known - good backup. This procedure is the same for all scenarios - single server, active-active, and active-passive.

Prerequisites and considerations

Backup incompatibility: Use a backup from the older 2.4 LDAP version that you were running with Edge for Private Cloud 4.52.02 or 4.53.00. This backup must have been taken before the upgrade was attempted. Backups taken after the upgrade are incompatible and cannot be used.
Management API downtime: During the LDAP rollback, management APIs will be unavailable, which may impact administrative tasks. Please make sure to shutdown all edge-management-server and edge-ui before proceeding further with LDAP rollback.
Risk of data loss: Proceeding without a valid, compatible backup risks irreversible data loss.
Valid backup: A valid pre-upgrade LDAP backup is required.

Rollback procedure

Please make sure to shutdown all edge-management-server and edge-ui before proceeding further with LDAP upgrade.
```
apigee-service edge-management-server stop
apigee-service edge-ui stop
```

Backup existing LDAP Data (Before stopping LDAP)

Get total record count for reference from all ldap servers. (The record count should match across all LDAP servers)

      # Note: Replace 'YOUR_PASSWORD' with your current LDAP manager password. 
ldapsearch -o ldif-wrap=no -b "dc=apigee,dc=com" \ -D "cn=manager,dc=apigee,dc=com" -H ldap://:10389 -LLL -x -w 'YOUR_PASSWORD' | wc -l

Stop and uninstall new LDAP 2.6
Stop the apigee-openldap service and delete its configuration and data directories. This must be performed on all LDAP servers, one node at a time in the cluster to ensure consistency.
```
apigee-service apigee-openldap stop
apigee-service apigee-openldap uninstall
rm -rf /opt/apigee/data/apigee-openldap/*
```

Install old LDAP 2.4 version

Install old LDAP version:

/opt/apigee/apigee-setup/bin/setup.sh -p ld -f /opt/silent.conf

Validate LDAP version:

source ~/.bash_profile
ldapsearch -VV

#Expected output:
ldapsearch: @(#) $OpenLDAP: ldapsearch 2.4.46

Repeat the above Steps on each LDAP node, one at a time

Clean residual data
On all LDAP servers, one at a time, stop the newly installed service and clean its data directory to prepare for the restore from backup.
```
apigee-service apigee-openldap stop
rm -rf /opt/apigee/data/apigee-openldap/*
```

Restore LDAP data

For single-server setup, you can proceed directly to step 5b. For multi server setup, proceed with step 5a.

Identify the active LDAP server

Before restoring LDAP data, identify the active server (provider) by running this command.

grep -i '^olcSyncrepl:' /opt/apigee/data/apigee-openldap/slapd.d/cn=config/olcDatabase*\ldif

* **Note**:
* If this command returns output, the server is a passive server. 
* If it returns no output, the server is the active server.

Restore LDAP data (Ensure step 4 is completed across all servers before restoring.)

On the single and active LDAP server (determined in Step 5a), restore the backup.

cd /opt/apigee/backup/openldap

# To restore a specific backup, provide the timestamp as shown below:
apigee-service apigee-openldap restore 2025.07.28,13.59.00
# Note: If no timestamp is provided, the latest available backup will be restored by default.
apigee-service apigee-openldap restore

# It is recommended to specify the backup explicitly to avoid restoring an unintended version.

Start the apigee-openldap service.
```
apigee-service apigee-openldap start
```

Start remaining LDAP servers
If you have a multi server setup, on each of the LDAP servers, start the service:
```
apigee-service apigee-openldap start
```
Note: You do not need to restore data manually on these peer servers. Upon starting, they will connect to the first server (which now contains the restored data) and automatically synchronize the entire directory via replication.
Final validation
- The final step is to verify that the rollback was successful and that data is consistent across the entire LDAP cluster.
- Run the validation command on all LDAP servers. The record count should be identical across all servers and must match the count you captured in Step 1.
```
# Note: Replace 'YOUR_PASSWORD' with your current LDAP manager password.
ldapsearch -o ldif-wrap=no -b "dc=apigee,dc=com" \ -D "cn=manager,dc=apigee,dc=com" -H ldap://:10389 -LLL -x -w 'YOUR_PASSWORD' | wc -l
```
- Once you have confirmed that the data is correct and consistent, your LDAP rollback is complete. You may now proceed with starting the edge-management-server, edge-ui and any other dependent components as per your organization's standard upgrade procedure.

Roll back to a previous major or minor release

To roll back to a previous major or minor release, do the following on each node that hosts the component:

Download the bootstrap.sh file for the version to which you want to roll back:
- To roll back to 4.52.02, download bootstrap_4.52.02.sh

Stop the component to roll back:

To roll back any of the components with common code on the node, you must stop them all, as the following example shows:

/opt/apigee/apigee-service/bin/apigee-service edge-management-server stop
/opt/apigee/apigee-service/bin/apigee-service edge-router stop
/opt/apigee/apigee-service/bin/apigee-service edge-message-processor stop
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server stop
/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop

To roll back any other component on the node, stop just that component:
```
/opt/apigee/apigee-service/bin/apigee-service component stop
```

If you are rolling back Monetization, uninstall it from all Management Server and Message Processor nodes:
```
/opt/apigee/apigee-service/bin/apigee-service edge-mint-gateway uninstall
```
Uninstall the component to roll back on the node:
1. To roll back any of the components with common code on the node, you must uninstall them all by uninstalling the edge-gateway and apigee-cassandra-client component group, as the following example shows:
```
/opt/apigee/apigee-service/bin/apigee-service edge-gateway uninstall
```
```
/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra-client uninstall
```
2. To roll back any other component on the node, uninstall just that component, as the following example shows:
```
/opt/apigee/apigee-service/bin/apigee-service component uninstall
```
  Where component is the component name.
3. To roll back the Edge Router, you must delete the contents of the /opt/nginx/conf.d file in addition to uninstalling the edge-gateway component group:
```
cd /opt/nginx/conf.d
rm -rf *
```

Uninstall the 4.53.01 version of apigee-setup:

/opt/apigee/apigee-service/bin/apigee-service apigee-setup uninstall

Install the 4.52.02 version of the apigee-service utility and its dependencies. The following example installs the 4.52.02 version of the apigee-service:
```
sudo bash /tmp/bootstrap_4.52.02.sh apigeeuser=uName apigeepassword=pWord
```
Where uName and pWord are the username and password you received from Apigee. If you omit pWord, you will be prompted to enter it.

If you get an error, be sure you downloaded the bootstrap.sh file in step 1.

Install apigee-setup:

/opt/apigee/apigee-service/bin/apigee-service apigee-setup install

Install the older version of the component:
```
/opt/apigee/apigee-setup/bin/setup.sh -p component -f configFile
```
Where component is the component to install and configFile is your configuration file for the older version.
If you are rolling back Qpid, flush iptables:
```
sudo iptables -F
```
Repeat this process for each node that hosts the component you are rolling back.

Roll back mTLS

To roll back the mTLS update, do the following steps on all hosts:

Stop Apigee:
```
apigee-all stop
```
Stop mTLS:
```
apigee-service apigee-mtls uninstall
```

Reinstall mTLS:

apigee-service apigee-mtls install
apigee-service apigee-mtls setup -f /opt/silent.conf