Roll back Apigee Edge 4.51.00

If you encounter an error during an update to Edge 4.51.00, you can roll back the component that caused the error and then try the update again.

You can roll back Edge 4.51.00 to the following Feature Release versions:

Version 4.50.00
Version 4.19.06

There are two scenarios where you might want to perform a rollback:

Roll back to a previous Feature Release. For example from 4.51.00 to 4.50.00.
Roll back to a previous update version in the same release. For example, from 4.50.00.02 to 4.50.00.01.

For more information, see Apigee Edge release process.

Who can perform a rollback

The user performing a rollback should be the same as the user who originally updated Edge, or a user running as root.

By default, Edge components run as the user "apigee". In some cases, you might be running Edge components as different users. For example, if the Router has to access privileged ports, such as those below 1000, then you have to run the Router as root or as a user with access to those ports. Or, you might run one component as one user, and another component as another user.

Components with common code

The following Edge components share common code. Therefore, to roll back any one of these components on a node, you must roll back all of these components that are on that node.

edge-management-server (Management Server)
edge-message-processor (Message Processor)
edge-router (Router)
edge-postgres-server (Postgres Server)
edge-qpid-server (Qpid Server)

For example, if you have the Management Server, Router, and Message Processor installed on the node, to roll back any one of them you must roll back all three.

Roll back to a previous Feature Release

To roll back to a previous Feature Release, do the following on each node that hosts the component:

Download the bootstrap.sh file for the version to which you want to roll back:
- To roll back to 4.50.00, download bootstrap_4.50.00.sh:
```
curl https://software.apigee.com/bootstrap_4.50.00.sh -o /tmp/bootstrap_4.50.00.sh 
```
- To roll back to 4.19.06, download bootstrap_4.19.06.sh:
```
curl https://software.apigee.com/bootstrap_4.19.01.sh -o /tmp/bootstrap_4.19.01.sh 
```

Stop the component to roll back:

To roll back any of the components with common code on the node, you must stop them all, as the following example shows:

/opt/apigee/apigee-service/bin/apigee-service edge-management-server stop
/opt/apigee/apigee-service/bin/apigee-service edge-router stop
/opt/apigee/apigee-service/bin/apigee-service edge-message-processor stop
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server stop
/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop

To roll back any other component on the node, stop just that component:
- ```
/opt/apigee/apigee-service/bin/apigee-service component stop
```

If you are rolling back Monetization, uninstall it from all Management Server and Message Processor nodes:
```
/opt/apigee/apigee-service/bin/apigee-service edge-mint-gateway uninstall
```
Uninstall the component to roll back on the node:
1. To roll back any of the components with common code on the node, you must uninstall them all by uninstalling the edge-gateway component group, as the following example shows:
```
/opt/apigee/apigee-service/bin/apigee-service edge-gateway uninstall
```
2. To roll back any other component on the node, uninstall just that component, as the following example shows:
```
/opt/apigee/apigee-service/bin/apigee-service component uninstall
```
  Where component is the component name.
3. To roll back the Edge Router, you must delete the contents of the /opt/nginx/conf.d file in addition to uninstalling the edge-gateway component group:
```
cd /opt/nginx/conf.d
rm -rf *
```

Uninstall the 4.51.00 version of apigee-setup:

/opt/apigee/apigee-service/bin/apigee-service apigee-setup uninstall

Install the 4.19.06 or 4.50.00 version of the apigee-service utility and its dependencies. The following example installs the 4.50.00 version of the apigee-service:
```
sudo bash /tmp/bootstrap_4.50.00.sh apigeeuser=uName apigeepassword=pWord
```
Where uName and pWord are the username and password you received from Apigee. If you omit pWord, you will be prompted to enter it.

If you get an error, be sure you downloaded the bootstrap.sh file in step 1.

Install apigee-setup:

/opt/apigee/apigee-service/bin/apigee-service apigee-setup install

Install the older version of the component:
```
/opt/apigee/apigee-setup/bin/setup.sh -p component -f configFile
```
Where component is the component to install and configFile is your configuration file for the older version.
If you are rolling back Qpid, flush iptables:
```
sudo iptables -F
```
Repeat this process for each node that hosts the component you are rolling back.

Roll back to a previous update version

To roll back a component to a specific version of a release, do the following on each node that hosts the component:

Download the specific component version:
```
/opt/apigee/apigee-service/bin/apigee-service component_version install
```
Where component_version is the component and update version to install. For example:
```
/opt/apigee/apigee-service/bin/apigee-service edge-ui-4.18.05-0.0.3749 install
```
If you are using the Apigee online repo, you can determine the available component versions by using the following command:
```
yum --showduplicates list comp
```
For example:
```
yum --showduplicates list edge-ui
```
Use apigee-setup to install the component:
```
/opt/apigee/apigee-setup/bin/setup.sh -p comp -f configFile
```
For example:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ui -f configFile
```
Note that you specify only the component name when you install it, not the version.
Repeat this process for each node that hosts the component you are rolling back.

Roll back the Postgres 10.17 update

If you upgraded to 4.51.00 from version 4.19.06 or 4.50.00, you must roll back your Postgres update in addition to the Edge components.

To rollback the Postgres update when updating Postgres in a master-standby configuration:

Promote the new standby node to become the Postgres master. The new Postgres master will be the same version as your previous Edge installation.
Configure the old standby node to be a standby node of the new master. The old standby node will be the same version as your previous Edge installation.
Register the new master and standby nodes with the analytics and consumer groups.

When you are done with the rollback, the old master node will no longer be necessary. You can then decommission the old master node.

Make sure the new standby Postgres node is running:

/opt/apigee/apigee-service/bin/apigee-all status

If Postgres is not running, start it:

/opt/apigee/apigee-service/bin/apigee-all start

Make sure Postgres is stopped on the old master node and old standby node:

/opt/apigee/apigee-service/bin/apigee-all status

If Postgres is running, stop it:

/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop > /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql stop

If installed, start Qpid on the old standby node:
```
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server start
```
Note: In many configurations, the old standby node will only be hosting Postgres but not Qpid.
Promote the new standby node as the Postgres master:
1. Promote the new standby node to be the new master:
```
apigee-service apigee-postgresql promote-standby-to-master new_standby_IP
```
  If prompted, enter the Postgres password for the 'apigee' user, which defaults to "postgres".
2. Edit the config file that you used to install your current version of Edge to specify the following:
```
# IP address of the new master:
PG_MASTER=new_standby_IP
# IP address of the old standby node
PG_STANDBY=old_standby_IP
```
3. Configure the new master:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup-replication-on-master -f configFile
```
If you have already upgraded the old standby node to the newer version, you must first downgrade Apigee software on the old standby node. If you still have the older version on the old standby node, you can skip this step and continue with step 6.
1. Stop Postgres on the old standby node:
```
apigee-service apigee-postgresql stop
apigee-service edge-postgres-server stop
```
2. Uninstall Postgres from the old standby node:
```
apigee-service apigee-postgresql uninstall
apigee-service edge-postgres-server uninstall
```
3. Delete the Postgres data directory from the old standby node:
```
cd /opt/apigee/data/apigee-postgresql/pgdata > rm -rf *
```
4. Download and run the older version bootstrap (for the Apigee version you are rolling back to) on the old standby node. The exact steps may vary based on whether you're using internet based or offline installation. Running the older version Apigee bootstrap will set up the yum repositories with older version Apigee data.
5. Set up Postgres components on the old standby node:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ps -f configFile
```
6. Check and verify that Postgres components on the old standby node have been rolled back to the older version:
```
apigee-service apigee-postgresql version
apigee-service edge-postgres-server version
```

Rebuild the old standby node:

Edit the config file that you used to install your current version of Edge to specify the following:

# IP address of the new master:
PG_MASTER=new_standby_IP
# IP address of the old standby node
PG_STANDBY=old_standby_IP

Remove the data directory on the old standby node:

cd /opt/apigee/data/apigee-postgresql/pgdata > rm -rf *

Reconfigure the old standby node to be a standby node of the new master:

/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup-replication-on-standby -f configFile

Make sure Postgres is running on the old standby node:

/opt/apigee/apigee-service/bin/apigee-all status

If Postgres is not running, start it:

/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server start

Verify that the new standby node was added by viewing the /opt/apigee/apigee-postgresql/conf/pg_hba.conf file on the new master.

View the current analytics and consumer group information by running the following command on the Management Server:

curl -u sysAdminEmail:password http://ms_IP:8080/v1/analytics/groups/ax

This command returns the analytics group name in the name field, and the consumer group name in the name field under consumer-groups. It also returns the UUIDs of the old Postgres master and standby nodes in the postgres-server field, and in the datastores field. You should see output in the form:

{
  "name" : "axgroup-001",
  "properties" : {
  },
  "scopes" : [ "VALIDATE~test", "sgilson~prod" ],
  "uuids" : {
    "qpid-server" : [ "8381a053-433f-4382-bd2a-100fd37a1592", "4b6856ec-ef05-498f-bac6-ef5f0d5f6521" ],
    "postgres-server" : [
      "ab1158bd-1d59-4e2a-9c95-24cc2cfa6edc:27f90844-efab-4b32-8a23-8f85cdc9a256"
    ]
  },
  "consumer-groups" : [ {
    "name" : "consumer-group-001",
    "consumers" : [ "8381a053-433f-4382-bd2a-100fd37a1592", "4b6856ec-ef05-498f-bac6-ef5f0d5f6521" ],
    "datastores" :
      [ "ab1158bd-1d59-4e2a-9c95-24cc2cfa6edc:27f90844-efab-4b32-8a23-8f85cdc9a256" ],
      "properties" : {     }
    }
  ],
  "data-processors" : {
  }
}

Get the UUID address of the old master by running the following curl command on the old master node:
```
curl -u sysAdminEmail:password http://node_IP:8084/v1/servers/self
```
You should see the UUID of the node at the end of the output, in the form:
```
"type" : [ "postgres-server" ],
"uUID" : "599e8ebf-5d69-4ae4-aa71-154970a8ec75"
```
Note: The edge-postgres-server service must be running. If the Postgres server is not running, you can run the following command on the Management Server to determine the UUID:
```
curl -u sysAdminEmail:password http://ms_IP:8080/v1/servers?pod=analytics
```
The output of this command lists the UUID for the IP address of each Postres node.
Repeat previous step to get the IP addresses of the old standby node and the new master.
Remove old master and standby nodes from the consumer group:
```
curl -u sysAdminEmail:password -X DELETE \
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/consumer-groups/consumer-group-001/datastores/masterUUID,standbyUUID" -v
```
Where axgroup-001 and consumer-group-001 are the default names of the analytics and consumer groups. masterUUID,standbyUUID are in the same order as they appeared above when you viewed the current analytics and consumer group information above. You might have to specify them as standbyUUID,masterUUID.

The datastores property for consumer-groups should now be empty.

Remove the old master and standby nodes from the analytics group:

curl -u sysAdminEmail:password -X DELETE \
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/servers?uuid=masterUUID,standbyUUID&type=postgres-server" -v

The postgres-server property under uuids should now be empty.

curl -u sysAdminEmail:password -X POST -H "Content-Type: application/json" -d ''
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/servers?uuid=masterUUID,standbyUUID&type=postgres-server" -v
curl -u sysAdminEmail:password -X POST -H "Content-Type:application/json" -d ''
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/consumer-groups/consumer-group-001/datastores?uuid=masterUUID,standbyUUID" -v

Validate the analytics group:
```
curl -u sysAdminEmail:password http://ms_IP:8080/v1/analytics/groups/ax
```
You should see the UUIDs of the new master and standby nodes listed in the analytics group and the consumer group.

Restart the Edge Management Server:

/opt/apigee/apigee-service/bin/apigee-service edge-management-server restart

Restart all Qpid servers:

/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restart

Restart all Postgres servers:

/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server restart

Verify the replication status by issuing the following scripts on both servers. The system should display identical results on both servers to ensure a successful replication:
On the new master, run:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql postgres-check-master
```
Verify that it is the master. On the old standby node:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql postgres-check-standby
```
Verify that it is the standby.
Repeat the previous step after making several API requests to ensure that the nodes are in synch.
Decommission the old Postgres master using the procedure in Update Apigee Edge 4.16.01/4.16.05 to 4.17.09.
Note: If the old master node was running Qpid, you can leave that server up to run Qpid. Ensure that it is running. If not, start it:
```
/opt/apigee/apigee-service/bin/apigee-service edge-management-server start
```
Alternatively, you can uninstall Qpid from the old master and install Qpid on the new master node. After uninstalling Qpid, you can decommission the old master node.

Roll back mTLS

To roll back the mTLS update, do the following steps on all hosts:

Stop Apigee:
```
apigee-all stop
```
Stop mTLS:
```
apigee-service apigee-mtls uninstall
```

Reinstall mTLS:

apigee-service apigee-mtls install
apigee-service apigee-mtls setup -f /opt/silent.conf