4.17.05 Rollback Process

Edge for Private Cloud v. 4.17.05

In the event of an error during an update to Edge 4.17.05, you can roll back the component that caused the error and then try the update again. For example, if the update to Postgres 9.4 fails, you can rollback just the Postgres nodes and try the update again.

There are two scenarios where you might want to perform a rollback:

Rollback to an older release. For example from 4.17.05 to 4.17.01.
Rollback to an older version in the same release.

Use the procedure below to perform a rollback for both scenarios.

Who can perform the rollback

The user performing the rollback should be the same as the user who originally updated Edge, or a user running as root.

By default, Edge components run as the user "apigee". In some cases, you might be running Edge components as different users. For example, if the Router has to access privileged ports, such as those below 1000, then you have to run the Router as root or as a user with access to those ports. Or, you might run one component as one user, and another component as another user.

Which components can be rolled back

You should be aware of the following conditions when performing a rollback:

The five Edge components listed below share common code. Therefore, to rollback any one of the five components on a node, you must roll back any of the five installed on the node. For example, if you have the Management Server, Router, and Message Processor installed on the node, to roll back any one of them you must roll back all three.
The five components that share code are:
- Management Server
- Router
- Message Processor
- Qpid Server
- Postgres Server
If you are updating from Edge 4.16.01, do not rollback Cassandra. This release of Edge contains an updated version of Cassandra. If you rollback any components, leave Cassandra at the 4.17.05 version.

Rolling back 4.17.05

This section contains the procedure to rollback Edge 4.17.05 to a previous version. This section is divided into two parts:

If you are updating from 4.16.01 or 4.16.05 only - rolling back the Postgres update to version 9.4
The final part of every update procedure from 4.16.01 or 4.16.05 is to update Postgres nodes to version 9.4. If that update fails, then you can use this procedure to rollback the update.
Rolling back all other Edge components
Use this procedure to rollback any other Edge components.

To rollback the Postgres 9.4 update

To rollback the Postgres update when updating Postgres in a master-standby configuration, you:

Promote the new standby node to become the Postgres master. The new Postgres master will be the same version as your previous Edge installation.
Configure the old standby node to be a standby node of the new master. The old standby node will be the same version as your previous Edge installation.
Register the new master and standby nodes with the analytics and consumer groups.

When you are done with the rollback, the old master node will no longer be necessary. You can then decommission the old master node.

Make sure the new standby Postgres node is running:
> /opt/apigee/apigee-service/bin/apigee-all status

If Postgres is not running, start it:
> /opt/apigee/apigee-service/bin/apigee-all start
Make sure Postgres is stopped on the old master node and old standby node:
> /opt/apigee/apigee-service/bin/apigee-all status

If Postgres is running, stop it:
> /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop
> /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql stop
If installed, start Qpid on the old standby node:
> /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server start

Note: In many configurations, the old standby node will only be hosting Postgres but not Qpid.
Promote the new standby node as the Postgres master:
1. Promote the new standby node to be the new master:
  > apigee-service apigee-postgresql promote-standby-to-master new_standby_IP
  
  If prompted, enter the Postgres password for the 'apigee' user, which defaults to "postgres".
2. Edit the config file that you used to install your current version of Edge to specify the following:
  # IP address of the new master:
  PG_MASTER=new_standby_IP
  # IP address of the old standby node
  PG_STANDBY=old_standby_IP
3. Configure the new master:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup-replication-on-master -f configFile
Rebuild the old standby node:
1. Edit the config file that you used to install your current version of Edge to specify the following:
  # IP address of the new master:
  PG_MASTER=new_standby_IP
  # IP address of the old standby node
  PG_STANDBY=old_standby_IP
2. Remove data directory on the old standby node:
  > cd /opt/apigee/data/apigee-postgresql/pgdata
  > rm -rf *
3. Reconfigure the old standby node to be a standby node of the new master:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup-replication-on-standby -f configFile
4. Make sure Postgres is running on the old standby node:
  > /opt/apigee/apigee-service/bin/apigee-all status
  
  If it is not running, start it:
  > /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server start
Verify that the new standby node was added by viewing the /opt/apigee/apigee-postgresql/conf/pg_hba.conf file on the new master.
View the current analytics and consumer group information by running the following command on the Management Server:
> curl -u sysAdminEmail:password http://<ms_IP>:8080/v1/analytics/groups/ax

This command returns the analytics group name in the name field, and the consumer group name in the name field under consumer-groups. It also returns the UUIDs of the old Postgres master and standby nodes in the postgres-server field, and in the datastores field. You should see output in the form:

{
"name" : "axgroup-001",
"properties" : {
},
"scopes" : [ "VALIDATE~test", "sgilson~prod" ],
"uuids" : {
"qpid-server" : [ "8381a053-433f-4382-bd2a-100fd37a1592", "4b6856ec-ef05-498f-bac6-ef5f0d5f6521" ],
"postgres-server" : [ "ab1158bd-1d59-4e2a-9c95-24cc2cfa6edc:27f90844-efab-4b32-8a23-8f85cdc9a256" ]
},
"consumer-groups" : [ {
"name" : "consumer-group-001",
"consumers" : [ "8381a053-433f-4382-bd2a-100fd37a1592", "4b6856ec-ef05-498f-bac6-ef5f0d5f6521" ],
"datastores" : [ "ab1158bd-1d59-4e2a-9c95-24cc2cfa6edc:27f90844-efab-4b32-8a23-8f85cdc9a256" ],
"properties" : { }
} ],
"data-processors" : {
}
}
Get the UUID address of the old master by running the following cURL command on the old master node:
> curl -u sysAdminEmail:password http://<node_IP>:8084/v1/servers/self

You should see the UUID of the node at the end of the output, in the form:
"type" : [ "postgres-server" ],
"uUID" : "599e8ebf-5d69-4ae4-aa71-154970a8ec75"

Note: The edge-postgres-server service must be running. If the Postgres server is not running, you can run the following command on the Management Server to determine the UUID:
> curl -u sysAdminEmail:password http://<ms_IP>:8080/v1/servers?pod=analytics

The output of this command lists the UUID for the IP address of each Postres node.
Repeat previous step to get the IP addresses of the old standby node and the new master.
Remove old master and standby nodes from the consumer group:
> curl -u sysAdminEmail:password -X DELETE "http://<ms_IP>:8080/v1/analytics/groups/ax/axgroup-001/consumer-groups/consumer-group-001/datastores/masterUUID,standbyUUID" -v

where axgroup-001 and consumer-group-001 are the default names of the analytics and consumer groups. masterUUID,standbyUUID are in the same order as they appeared above when you viewed the current analytics and consumer group information above. You might have to specify them as standbyUUID,masterUUID.

The datastores property for consumer-groups should now be empty.
Remove the old master and standby nodes from the analytics group:
> curl -u sysAdminEmail:password -X DELETE "http://<ms_IP>:8080/v1/analytics/groups/ax/axgroup-001/servers?uuid=masterUUID,standbyUUID&type=postgres-server" -v

The postgres-server property under uuids should now be empty.
Register new PG master and standby nodes with the analytics and consumer groups:
> curl -u sysAdminEmail:password -X POST -H "Content-Type: application/json" -d '' "http://<ms_IP>:8080/v1/analytics/groups/ax/axgroup-001/servers?uuid=masterUUID,standbyUUID&type=postgres-server" -v

> curl -u sysAdminEmail:password -X POST -H "Content-Type: application/json" -d '' "http://<ms_IP>:8080/v1/analytics/groups/ax/axgroup-001/consumer-groups/consumer-group-001/datastores?uuid=masterUUID,standbyUUID" -v
Validate the analytics group:
> curl -u sysAdminEmail:password http://<ms_IP>:8080/v1/analytics/groups/ax

You should see the UUIDs of the new master and standby nodes listed in the analytics group and the consumer group.
Restart the Edge Management Server:
> /opt/apigee/apigee-service/bin/apigee-service edge-management-server restart
Restart all Qpid servers:
> /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restart
Restart all Postgres servers:
> /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server restart
Verify the replication status by issuing the following scripts on both servers. The system should display identical results on both servers to ensure a successful replication:

On the new master, run:
> /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql postgres-check-master

Validate that it says it is the master.

On the old standby node:
> /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql postgres-check-standby

Validate that it says it is the standby.
Repeat the previous step after making several API requests to ensure that the nodes are in synch.
Decommission the old Postgres master using the procedure in Update Apigee Edge 4.16.01/4.16.05 to 4.17.05.

Note: If the old master node was running Qpid, you can leave that server up to run Qpid. Ensure that it is running. If not, start it:
> /opt/apigee/apigee-service/bin/apigee-service edge-management-server start

Alternatively, you can uninstall Qpid from the old master and install Qpid on the new master node, as described below. After uninstalling Qpid, you can decommission the old master node.

Uninstal Qpid from the old master and install Qpid on the new master

Use the following procedure to uninstall Qpid from the old master and install it on the new master:

Block access to Qpid port 5672 on the old master from access by Message Processors by running the following command on all Message Processors:
> iptables -A OUTPUT -p tcp -d 10.233.147.20 --dport 5672 -j DROP
Ensure that the Qpid message queue is empty by running the following command. You cannot uninstall Qpid until it has processed all pending messages:
> qpid-stat -q

This command displays a table containing a count for msg, msgIn, and msgOut. All messages will have been processed when msg=0, and msgIn=msgOut.
Determine the UUID of the Qpid server on the old master by running the following command on the old master. Save this information for later in the procedure:
> curl -u sysAdminEmail:password http://<node_IP>::8083/v1/servers/self
Stop Qpid on the old master:
> /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server stop
> /opt/apigee/apigee-service/bin/apigee-service apigee-qpidd stop
Uninstall Qpid server:
> /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server uninstall
> /opt/apigee/apigee-service/bin/apigee-service apigee-qpidd uninstall
Remove the old Qpid server from the analytics and consumer groups:
> curl -u sysAdminEmail:password -X DELETE -H "Content-Type: application/json" -d '' "http://<ms_IP>:8080/v1/analytics/groups/ax/axgroup-001/consumer-groups/consumer-group-001/consumers/qpid_UUID" -v

> curl -u sysAdminEmail:password -X DELETE "http://<ms_IP>:8080/v1/analytics/groups/ax/axgroup-001/servers?uuid=qpid_UUID&type=qpid-server" -v
Remove the old Qpid server from Zookeeper:
> curl -u sysAdminEmail:password -X DELETE http://<ms_IP>:8080/v1/servers/qpid_UUID
Install Qpid on the new master:
> /opt/apigee/apigee-setup/bin/setup.sh -p qs -f configFile
Determine the UUID of the Qpid server on the new master by running the following command on the new master. Save this information for later in the procedure:
> curl -u sysAdminEmail:password http://<node_IP>::8083/v1/servers/self
Register the new Qpid server with the analytics and consumer groups:
> curl -u sysAdminEmail:password -X POST -H "Content-Type: application/json" -d '' "http://<ms_IP>:8080/v1/analytics/groups/ax/axgroup-001/servers?uuid=qpid_UUID&type=qpid-server" -v

> curl -u sysAdminEmail:password -X POST -H "Content-Type: application/json" -d '' "http://<ms_IP>:8080/v1/analytics/groups/ax/axgroup-001/consumer-groups/consumer-group-001/consumers?uuid=qpid_UUID" -v
Restart all Message Processors:
> /opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart
Run the following command on the new Qpid server to check that queues were created:
> qpid-stat -q

Ensure that you see msg, msgIn, and msgOut being updated as the Qpid server processes messages.

To rollback individual components from 4.17.05

As part of performing the rollback, you have to download the bootstrap.sh file for your current version of Edge:

For rolling back to 4.17.01, download bootstrap_4.17.01.sh
For rolling back to 4.16.09, download bootstrap_4.16.09.sh
For rolling back to 4.16.05, download bootstrap_4.16.05.sh
For rolling back to 4.16.01, download bootstrap.sh

For each node hosting a component to roll back:

Stop the component to rollback:
1. If you are rolling back any one of the following components on the node, you must stop them all: Management Server, Router, Message Processor, Qpid Server, or Postgres Server:
  - > apigee-service edge-management-server stop
  - > apigee-service edge-router stop
  - > apigee-service edge-message-processor stop
  - > apigee-service edge-qpid-server stop
  - > apigee-service edge-postgres-server stop
2. If you are rolling back any other component on the node, stop just that component:
  - > apigee-service comp stop
If you are rolling back Monetization, uninstall it from all Management Server and Message Processor nodes:
> apigee-service edge-mint-gateway uninstall
Uninstall the component to rollback on the node:
1. If you are rolling back any of the following components on the node, then uninstall them all: Management Server, Router, Message Processor, Qpid Server, or Postgres Server:
  > apigee-service edge-gateway uninstall
2. If you are rolling back any other component on the node, uninstall just that component:
  > apigee-service comp uninstall
3. If you are rolling back the Router, then you have to delete the contents of /opt/nginx/conf.d:
  > cd /opt/nginx/conf.d
  > rm -rf *
To rollback the component:
1. Uninstall the 4.17.05 version of apigee-setup:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-setup uninstall
2. Download bootstrap.sh for the 4.16.01, 4.16.05, 4.16.09, or 4.17.01 release:
  
  For example, for 4.16.09:
  > curl https://software.apigee.com/bootstrap_4.16.09.sh -o /tmp/bootstrap_4.16.09.sh
3. Install the 4.16.01, 4.16.05, or 4.16.09 apigee-service utility and dependencies.
  
  For example, for 4.16.09:
  > sudo bash /tmp/bootstrap_4.16.09.sh apigeeuser=uName apigeepassword=pWord
  
  where uName and pWord are the username and password you received from Apigee. If you omit pWord, you will be prompted to enter it.
4. Install the 4.16.01, 4.16.05, or 4.16.09 version of apigee-setup:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-setup install
5. Install the 4.16.01, 4.16.05, 4.16.09, or 4.17.01 version of the component:
  > /opt/apigee/apigee-setup/bin/setup.sh -p comp -f configFile
  
  where comp is the component to install and configFile is your 4.16.01, 4.16.05, or 4.16.09 configuration file.
6. If you are rolling back Qpid, flush iptables:
  > sudo iptables -F
To rollback the component to a specific version of the 4.17.05 release:
1. Download the specific component version:
  > /opt/apigee/apigee-service/bin/apigee-service comp-version install
  
  where comp-version is the component and version to install. For example:
  > /opt/apigee/apigee-service/bin/apigee-service edge-ui-4.17.05-0.0.3749 install
  
  If you are using the Apigee online repo, you can determine the available component versions by using the following command:
  > yum --showduplicates list comp
  
  For example:
  > yum --showduplicates list edge-ui
2. Use apigee-setup to install the component:
  > /opt/apigee/apigee-setup/bin/setup.sh -p comp -f configFile
  
  For example:
  > /opt/apigee/apigee-setup/bin/setup.sh -p ui -f configFile
  
  Note how you only specify the component name when you do the install.

Contact Apigee Support if you encounter issues when rolling back.