Roll back Apigee Edge 4.19.01

Edge for Private Cloud v4.19.01

If you encounter an error during an update to Edge 4.19.01, you can roll back the component that caused the error and then try the update again.

You can roll back Edge 4.19.01 to the following Feature Release versions:

Version 4.18.05
Version 4.18.01
Version 4.17.09^*

^* To roll back from 4.19.01 to 4.17.09, you must roll back Postgres in addition to rolling back the components on each node. If you roll back to 4.18.01 or 4.18.05, you do not need to roll back Postgres as the upgrade process did not include a Postgres update.

There are two scenarios where you might want to perform a rollback:

Roll back to a previous Feature Release. For example from 4.19.01 to 4.18.05.
Roll back to a previous update version in the same release. For example, from 4.19.01.02 to 4.19.01.01.

For more information, see Apigee Edge release process.

Who can perform a rollback

The user performing a rollback should be the same as the user who originally updated Edge, or a user running as root.

By default, Edge components run as the user "apigee". In some cases, you might be running Edge components as different users. For example, if the Router has to access privileged ports, such as those below 1000, then you have to run the Router as root or as a user with access to those ports. Or, you might run one component as one user, and another component as another user.

Components with common code

The following Edge components share common code. Therefore, to roll back any one of these components on a node, you must roll back all of these components that are on that node.

edge-management-server (Management Server)
edge-message-processor (Message Processor)
edge-router (Router)
edge-postgres-server (Postgres Server)
edge-qpid-server (Qpid Server)

For example, if you have the Management Server, Router, and Message Processor installed on the node, to roll back any one of them you must roll back all three.

Roll back to a previous Feature Release

To roll back from 4.19.01 to 4.17.09, you must roll back Postgres in addition to rolling back the components on each node. If you roll back from 4.18.01 or 4.18.05, you do not need to roll back Postgres as the upgrade process did not include a Postgres update.

To roll back to a previous Feature Release, do the following on each node that hosts the component:

Download the bootstrap.sh file for the version to which you want to roll back:

To roll back to 4.18.05, download bootstrap_4.18.05.sh:

curl https://software.apigee.com/bootstrap_4.18.05.sh -o /tmp/bootstrap_4.18.05.sh

To roll back to 4.18.01, download bootstrap_4.18.01.sh:

curl https://software.apigee.com/bootstrap_4.18.01.sh -o /tmp/bootstrap_4.18.01.sh

To roll back to 4.17.09, download bootstrap_4.17.09.sh:

curl https://software.apigee.com/bootstrap_4.17.09.sh -o /tmp/bootstrap_4.17.09.sh

Stop the component to roll back:

To roll back any of the components with common code on the node, you must stop them all, as the following example shows:

/opt/apigee/apigee-service/bin/apigee-service edge-management-server stop
/opt/apigee/apigee-service/bin/apigee-service edge-router stop
/opt/apigee/apigee-service/bin/apigee-service edge-message-processor stop
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server stop
/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop

To roll back any other component on the node, stop just that component:
- ```
/opt/apigee/apigee-service/bin/apigee-service component stop
```

If you are rolling back Monetization, uninstall it from all Management Server and Message Processor nodes:
```
/opt/apigee/apigee-service/bin/apigee-service edge-mint-gateway uninstall
```
Uninstall the component to roll back on the node:
1. To roll back any of the components with common code on the node, you must uninstall them all by uninstalling the edge-gateway component group, as the following example shows:
```
/opt/apigee/apigee-service/bin/apigee-service edge-gateway uninstall
```
2. To roll back any other component on the node, uninstall just that component, as the following example shows:
```
/opt/apigee/apigee-service/bin/apigee-service component uninstall
```
  Where component is the component name.
3. To roll back Edge Router, you must delete the contents of the /opt/nginx/conf.d file in addition to uninstalling the edge-gateway component group:
```
cd /opt/nginx/conf.d
rm -rf *
```

Uninstall the 4.19.01 version of apigee-setup:

/opt/apigee/apigee-service/bin/apigee-service apigee-setup uninstall

Install the 4.18.05, 4.18.01, or 4.17.09 version of the apigee-service utility and its dependencies. The following example installs the 4.17.09 version of the apigee-service:
```
sudo bash /tmp/bootstrap_4.17.09.sh apigeeuser=uName apigeepassword=pWord
```
Where uName and pWord are the username and password you received from Apigee. If you omit pWord, you will be prompted to enter it.

If you get an error, be sure you downloaded the bootstrap.sh file in step 1.

Install apigee-setup:

/opt/apigee/apigee-service/bin/apigee-service apigee-setup install

Install the older version of the component:
```
/opt/apigee/apigee-setup/bin/setup.sh -p component -f configFile
```
Where component is the component to install and configFile is your configuration file for the older version.
If you are rolling back Qpid, flush iptables:
```
sudo iptables -F
```
Repeat this process for each node that hosts the component you are rolling back.

Roll back to a previous update version

To roll back a component to a specific version of a release, do the following on each node that hosts the component:

Download the specific component version:
```
/opt/apigee/apigee-service/bin/apigee-service component_version install
```
Where component_version is the component and update version to install. For example:
```
/opt/apigee/apigee-service/bin/apigee-service edge-ui-4.17.09-0.0.3749 install
```
If you are using the Apigee online repo, you can determine the available component versions by using the following command:
```
yum --showduplicates list comp
```
For example:
```
yum --showduplicates list edge-ui
```
Use apigee-setup to install the component:
```
/opt/apigee/apigee-setup/bin/setup.sh -p comp -f configFile
```
For example:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ui -f configFile
```
Note that you specify only the component name when you install it, not the version.
Repeat this process for each node that hosts the component you are rolling back.

Roll back the Postgres 9.6 update

If you upgraded to 4.19.01 from version 4.17.09, you must roll back your Postgres update in addition to the Edge components.

To rollback the Postgres update when updating Postgres in a master-standby configuration:

Promote the new standby node to become the Postgres master. The new Postgres master will be the same version as your previous Edge installation.
Configure the old standby node to be a standby node of the new master. The old standby node will be the same version as your previous Edge installation.
Register the new master and standby nodes with the analytics and consumer groups.

When you are done with the rollback, the old master node will no longer be necessary. You can then decommission the old master node.

Make sure the new standby Postgres node is running:

/opt/apigee/apigee-service/bin/apigee-all status

If Postgres is not running, start it:

/opt/apigee/apigee-service/bin/apigee-all start

Make sure Postgres is stopped on the old master node and old standby node:

/opt/apigee/apigee-service/bin/apigee-all status

If Postgres is running, stop it:

/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop > /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql stop

If installed, start Qpid on the old standby node:
```
/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server start
```
Note: In many configurations, the old standby node will only be hosting Postgres but not Qpid.
Promote the new standby node as the Postgres master:
1. Promote the new standby node to be the new master:
```
apigee-service apigee-postgresql promote-standby-to-master new_standby_IP
```
  If prompted, enter the Postgres password for the 'apigee' user, which defaults to "postgres".
2. Edit the config file that you used to install your current version of Edge to specify the following:
```
# IP address of the new master:
PG_MASTER=new_standby_IP
# IP address of the old standby node
PG_STANDBY=old_standby_IP
```
3. Configure the new master:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup-replication-on-master -f configFile
```

Rebuild the old standby node:

Edit the config file that you used to install your current version of Edge to specify the following:

# IP address of the new master:
PG_MASTER=new_standby_IP
# IP address of the old standby node
PG_STANDBY=old_standby_IP

Remove data directory on the old standby node:

cd /opt/apigee/data/apigee-postgresql/pgdata > rm -rf *

Reconfigure the old standby node to be a standby node of the new master:

/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup-replication-on-standby -f configFile

Make sure Postgres is running on the old standby node:

/opt/apigee/apigee-service/bin/apigee-all status

If it is not running, start it:

/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server start

Verify that the new standby node was added by viewing the /opt/apigee/apigee-postgresql/conf/pg_hba.conf file on the new master.

View the current analytics and consumer group information by running the following command on the Management Server:

curl -u sysAdminEmail:password http://ms_IP:8080/v1/analytics/groups/ax

This command returns the analytics group name in the name field, and the consumer group name in the name field under consumer-groups. It also returns the UUIDs of the old Postgres master and standby nodes in the postgres-server field, and in the datastores field. You should see output in the form:

{
  "name" : "axgroup-001",
  "properties" : {
  },
  "scopes" : [ "VALIDATE~test", "sgilson~prod" ],
  "uuids" : {
    "qpid-server" : [ "8381a053-433f-4382-bd2a-100fd37a1592", "4b6856ec-ef05-498f-bac6-ef5f0d5f6521" ],
    "postgres-server" : [
      "ab1158bd-1d59-4e2a-9c95-24cc2cfa6edc:27f90844-efab-4b32-8a23-8f85cdc9a256"
    ]
  },
  "consumer-groups" : [ {
    "name" : "consumer-group-001",
    "consumers" : [ "8381a053-433f-4382-bd2a-100fd37a1592", "4b6856ec-ef05-498f-bac6-ef5f0d5f6521" ],
    "datastores" :
      [ "ab1158bd-1d59-4e2a-9c95-24cc2cfa6edc:27f90844-efab-4b32-8a23-8f85cdc9a256" ],
      "properties" : {     }
    }
  ],
  "data-processors" : {
  }
}

Get the UUID address of the old master by running the following curl command on the old master node:
```
curl -u sysAdminEmail:password http://node_IP:8084/v1/servers/self
```
You should see the UUID of the node at the end of the output, in the form:
```
"type" : [ "postgres-server" ],
"uUID" : "599e8ebf-5d69-4ae4-aa71-154970a8ec75"
```
Note: The edge-postgres-server service must be running. If the Postgres server is not running, you can run the following command on the Management Server to determine the UUID:
```
curl -u sysAdminEmail:password http://ms_IP:8080/v1/servers?pod=analytics
```
The output of this command lists the UUID for the IP address of each Postres node.
Repeat previous step to get the IP addresses of the old standby node and the new master.
Remove old master and standby nodes from the consumer group:
```
curl -u sysAdminEmail:password -X DELETE \
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/consumer-groups/consumer-group-001/datastores/masterUUID,standbyUUID" -v
```
Where axgroup-001 and consumer-group-001 are the default names of the analytics and consumer groups. masterUUID,standbyUUID are in the same order as they appeared above when you viewed the current analytics and consumer group information above. You might have to specify them as standbyUUID,masterUUID.

The datastores property for consumer-groups should now be empty.

Remove the old master and standby nodes from the analytics group:

curl -u sysAdminEmail:password -X DELETE \
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/servers?uuid=masterUUID,standbyUUID&type=postgres-server" -v

The postgres-server property under uuids should now be empty.

curl -u sysAdminEmail:password -X POST -H "Content-Type: application/json" -d ''
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/servers?uuid=masterUUID,standbyUUID&type=postgres-server" -v
curl -u sysAdminEmail:password -X POST -H "Content-Type:application/json" -d ''
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/consumer-groups/consumer-group-001/datastores?uuid=masterUUID,standbyUUID" -v

Validate the analytics group:
```
curl -u sysAdminEmail:password http://ms_IP:8080/v1/analytics/groups/ax
```
You should see the UUIDs of the new master and standby nodes listed in the analytics group and the consumer group.

Restart the Edge Management Server:

/opt/apigee/apigee-service/bin/apigee-service edge-management-server restart

Restart all Qpid servers:

/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restart

Restart all Postgres servers:

/opt/apigee/apigee-service/bin/apigee-service edge-postgres-server restart

Verify the replication status by issuing the following scripts on both servers. The system should display identical results on both servers to ensure a successful replication:
On the new master, run:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql postgres-check-master
```
Validate that it says it is the master. On the old standby node:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-postgresql postgres-check-standby
```
Validate that it says it is the standby.
Repeat the previous step after making several API requests to ensure that the nodes are in synch.
Decommission the old Postgres master using the procedure in Update Apigee Edge 4.16.01/4.16.05 to 4.17.09.
Note: If the old master node was running Qpid, you can leave that server up to run Qpid. Ensure that it is running. If not, start it:
```
/opt/apigee/apigee-service/bin/apigee-service edge-management-server start
```
Alternatively, you can uninstall Qpid from the old master and install Qpid on the new master node. After uninstalling Qpid, you can decommission the old master node.

Uninstall Qpid from the old master and install Qpid on the new master

To uninstall Qpid from the old master and install it on the new master:

Block access to Qpid port 5672 on the old master from access by Message Processors by running the following command on all Message Processors:
```
iptables -A OUTPUT -p tcp -d 10.233.147.20 --dport 5672 -j DROP
```
Ensure that the Qpid message queue is empty by running the following command. You cannot uninstall Qpid until it has processed all pending messages:
```
qpid-stat -q
```
This command displays a table containing a count for msg, msgIn, and msgOut. All messages will have been processed when msg=0, and msgIn=msgOut.
Determine the UUID of the Qpid server on the old master by running the following command on the old master. Save this information for later in the procedure:
```
curl -u sysAdminEmail:password http://node_IP::8083/v1/servers/self
```

Stop Qpid on the old master:

/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server stop
/opt/apigee/apigee-service/bin/apigee-service apigee-qpidd stop

Uninstall Qpid server:

/opt/apigee/apigee-service/bin/apigee-service edge-qpid-server uninstall
/opt/apigee/apigee-service/bin/apigee-service apigee-qpidd uninstall

Remove the old Qpid server from the analytics and consumer groups:

curl -u sysAdminEmail:password -X DELETE -H "Content-Type: application/json" -d '' \
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/consumer-groups/consumer-group-001/consumers/qpid_UUID" -v
curl -u sysAdminEmail:password -X DELETE \
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/servers?uuid=qpid_UUID&type=qpid-server" -v

Remove the old Qpid server from Zookeeper:

curl -u sysAdminEmail:password -X DELETE \
  http://ms_IP:8080/v1/servers/qpid_UUID

Install Qpid on the new master:

/opt/apigee/apigee-setup/bin/setup.sh -p qs -f configFile

Determine the UUID of the Qpid server on the new master by running the following command on the new master. Save this information for later in the procedure:
```
curl -u sysAdminEmail:password
  http://node_IP::8083/v1/servers/self
```

curl -u sysAdminEmail:password -X POST -H "Content-Type: application/json" -d '' \
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/servers?uuid=qpid_UUID&type=qpid-server" -v
curl -u sysAdminEmail:password -X POST -H "Content-Type:application/json" -d '' \
  "http://ms_IP:8080/v1/analytics/groups/ax/axgroup-001/consumer-groups/consumer-group-001/consumers?uuid=qpid_UUID" -v

Restart all Message Processors:

/opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart

Run the following command on the new Qpid server to check that queues were created:
```
qpid-stat -q
```
Ensure that you see msg, msgIn, and msgOut being updated as the Qpid server processes messages.

Contact Apigee Edge Support if you encounter issues when rolling back.