Update Apigee Edge to 4.16.09

Edge for Private Cloud v. 4.16.09

Which Edge versions can you update to 4.16.09

You can update Apigee Edge version 4.16.01.0x and 4.16.05.x to 4.16.09.

If you have a version of Edge previous to version 4.16.01, then you must first migrate to version 4.16.01.x and then update to version 4.16.09.

Who can perform the update

The user running the update should be the same as the user who originally installed Edge, or a user running as root.

After you install the Edge RPMs, any user can configure them.

Required upgrade to Java JDK Version 8

This release of Edge requires that you have installed Java JDK version 8 on all Edge processing nodes. You can install the Oracle JDK 8 or OpenJDK 8. If Java JDK 8 is not installed already, the update script can install it for you.

As part of the update to Java 8, some TLS ciphers are no longer available in Oracle JDK 8. For the complete list, see the section "Default Disabled Cipher Suites" at http://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html.

Downgrading Zookeeper if updating from 4.16.01

The version of the Zookeeper RPM in Edge for Private Cloud 4.16.01 is apigee-zookeeper-3.4.5-1.0.905.noarch.rpm. In subsequent versions of Edge, the Zookeeper version was changed back to apigee-zookeeper-3.4.5-0.0.94x. This prevents yum from upgrading Zookeeper to a later versions from 4.16.01. The way to corrct this situation is to run yum downgrade apigee-zookeeper before updating Zookeeper.

You can check your Zookeeper version by using the following command:

> rpm -qa |grep apigee-zookeeper

If this command returns Zookeeper version:

apigee-zookeeper-3.4.5-1.0.905

Then you must do the downgrade.

Automatic propagation of property settings

If you have set any properties by editing .properties files in /opt/apigee/customer/application then these values are retained by the update.

Update prerequisites

Take care of following prerequisites before upgrading Apigee Edge:

Backup all nodes
Before you update, it is recommended to perform a complete backup of all nodes for safety reasons. Use the procedure for your current version of Edge to perform the backup.

This allows you to have a backup plan, in case the update to a new version doesn’t function properly. For more information on backup, see Backup and Restore.
Ensure Edge is running
Ensure that Edge is up and running during update process by using the command:

> /<inst_root>/apigee/apigee-service/bin/apigee-all status

Handling a failed update

In the case of an update failure, you can try to correct the issue, and then run update.sh again. You can run the update multiple times and it continues the update from where it last left off.

If the failure requires that you roll back the update to your previous version, see 4.16.09 Rollback Process for more.

Logging update information

By default, the update.sh utility writes log information to:

/opt/apigee/var/log/apigee-setup/update.log

If the user running the update.sh utility does not have access to that directory, it writes the log to the /tmp directory as a file named update_username.log.

If the user does not have access to /tmp, the update.sh utility fails.

Required upgrade to Postgres 9.4

This release of Edge includes an upgrade to Postgres 9.4. As part of that upgrade, all Postgres data is migrated to the Postgres 9.4.

During the update process, while the Postgres nodes are down for update, analytics data is still written to the Qpid nodes. After the Postgres nodes are updated and back online, analytics data is then pushed to the Postgres nodes.

The additional Postgres standby node is required if you have to roll back the update for any reason. If you have to roll back the update, the new Postgres standby node becomes the master Postgres node after the rollback. Therefore, when you install the new Postgres standby node, it should be on a node that meets all the hardware requirements of a Postgres server, as defined in the Edge Installation Requirements.

Installing a new Postgres standby node

This procedure creates a Postgres standby server on a new node. Ensure that you install a new Postgres standby server for your existing version of Edge (4.16.01 or 4.16.05), not for version 4.16.09.

To perform the install, use the same config file that you used to install your current version of Edge.

To create a new Postgres standby node:

On the current Postgres master, edit the /opt/apigee/customer/application/postgresql.properties file to set the following token. If that file does not exist, create it:

conf_pg_hba_replication.connection=host replication apigee existing_slave_ip/32 trust\ \nhost replication apigee new_slave_ip/32 trust

where existing_slave_ip is the IP address of the current Postgres standby server and new_slave_ip is the IP address of the new standby node.
Restart apigee-postgresql on the Postgres master:
> /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql restart
Verify that the new standby node was added by viewing the /opt/apigee/apigee-postgresql/conf/pg_hba.conf file on the master. You should see the following lines in that file:
host replication apigee existing_slave_ip/32 trust
host replication apigee new_slave_ip/32 trust
Install the new Postgres standby server:
1. Edit the config file that you used to install your current version of Edge to specify the following:
  
  # IP address of the current master:
  PG_MASTER=192.168.56.103
  # IP address of the new standby node
  PG_STANDBY=192.168.56.102
2. Disable SELinux as described at Install the Edge apigee-setup utility.
3. Download the Edge bootstrap_4.16.05.sh file to /tmp/bootstrap_4.16.05.sh:
  > curl https://software.apigee.com/bootstrap_4.16.05.sh -o /tmp/bootstrap_4.16.05.sh
  
  Note: If you are updating from 4.16.01, download the Edge bootstrap.sh file.
4. Install the Edge apigee-service utility and dependencies:
  > sudo bash /tmp/bootstrap_4.16.05.sh apigeeuser=uName apigeepassword=pWord
5. Use apigee-service to install the apigee-setup utility:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-setup install
6. Install Postgres:
  > /opt/apigee/apigee-setup/bin/setup.sh -p ps -f configFile
7. On the new standby node run the following command:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql postgres-check-standby
  
  Validate that it says it is the standby.

Decommissioning a Postgres node

After the update completes, decommission the new standby node:

Make sure Postgres is running:
> /opt/apigee/apigee-service/bin/apigee-all status

If Postgres is not running, start it:
> /opt/apigee/apigee-service/bin/apigee-all start
Get the UUID of the new standby node by running the following cURL command on the new standby node:
> curl -u sysAdminEmail:password http://<node_IP>:8084/v1/servers/self

You should see the UUID of the node at the end of the output, in the form:
"type" : [ "postgres-server" ],
"uUID" : "599e8ebf-5d69-4ae4-aa71-154970a8ec75"
Stop the new standby node by running the following command on the new standby node:
> /opt/apigee/apigee-service/bin/apigee-all stop
On the Postgres master node, edit /opt/apigee/customer/application/postgresql.properties to remove the new standby node from conf_pg_hba_replication.connection:
conf_pg_hba_replication.connection=host replication apigee existing_slave_ip/32 trust
Restart apigee-postgresql on the Postgres master:
> /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql restart
Verify that the new standby node was removed by viewing the /opt/apigee/apigee-postgresql/conf/pg_hba.conf file on the master. You should see only the following line in that file:
host replication apigee existing_slave_ip/32 trust
Delete the UUID of the standby node from ZooKeeper by making the following Edge management API call on the Management Server node:
> curl -u sysAdminEmail:password -X DELETE http://<ms_IP>:8080/v1/servers/<new_slave_uuid>

Zero-downtime update

A zero-downtime update, or rolling update, lets you update your Edge installation without bringing down Edge.

Zero-downtime update is only possible with a 5-node configuration and larger.

The key to zero-downtime upgrading is to remove each Router, one at a time, from the load balancer. You then update the Router and any other components on the same machine as the Router, and then add the Router back to the load balancer.

Update the machines in the correct order for your installation as described below in "Order of machine update".
When it is time to update the Routers, select any one Router and make it unreachable, as described in Enabling/Disabling Server (Message Processor/Router) Reachability.
Update the selected Router and all other Edge components on the same machine as the Router. All Edge configurations show a Router and Message Processor on the same node.
Make the Router reachable again.
Repeat steps 2 through 4 for the remaining Routers.
Continue the update for any remaining machines in your installation.

Take care of the following before/after update:

On combined Router and Message Processor node:
- Before update – perform the following:
  1. Make the Router unreachable.
  2. Make the Message Processor unreachable.
- After update - perform the following:
  1. Make the Message Processor reachable.
  2. Make the Router reachable.
On single Router node:
- Before update, make the Router unreachable.
- After update, make the Router reachable.
On single Message Processor node:
- Before update, make the Message Processor unreachable.
- After update, make the Message Processor reachable.

Using a silent configuration file

You must pass a silent configuration file to the update command. The silent configuration file should be the same one you used to install Edge 4.16.01 or 4.16.05.

Procedure for updating to 4.16.09 on a node with an external internet connection

Use the following procedure to update the Edge components on a node:

If you are currently using Postgres master-standby replication, install a new Postgres standby node as described above in Installing a new Postgres standby node.
If present, disable any CRON jobs configured to perform a repair operation on Cassandra until after the update completes.
Log in to your node as root to install the Edge RPMs.
Note: While RPM installation requires root access, you can perform Edge configuration without root access.
Disable SELinux as described in Install the Edge apigee-setup utility.
Download the Edge 4.16.09 bootstrap_4.16.09.sh file to /tmp/bootstrap_4.16.09.sh:
> curl https://software.apigee.com/bootstrap_4.16.09.sh -o /tmp/bootstrap_4.16.09.sh
Install the Edge 4.16.09 apigee-service utility and dependencies:
> sudo bash /tmp/bootstrap_4.16.09.sh apigeeuser=uName apigeepassword=pWord

where uName:pWord are the username and password you received from Apigee. If you omit pWord, you will be prompted to enter it.

By default, the installer checks to see that you have Java 1.8 installed. If you do not, it installs it for you. Use the JAVA_FIX option to specify how to handle Java installation. JAVA_FIX takes the following values:

I = Install OpenJDK 1.8 (default)
C = Continue without installing Java
Q = Quit. For this option, you have to install Java yourself.
Use apigee-service to update the apigee-setup utility:
1. If you installed 4.16.01 by upgrading Edge version 4.15.07.0x, you must install the apigee-setup utility:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-setup install
  
  This command installs the update.sh utility in <inst_dir>/apigee/apigee-setup/bin.
  
  If you already installed the apigee-setup utility, then update it:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-setup update
2. If you installed 4.16.01 directly, meaning you did not perform an upgrade from 4.15.07.0x, you must update the apigee-setup utility:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-setup update
  
  This update to apigee-service installs the update.sh utility in <inst_dir>/apigee/apigee-setup/bin.
3. If you installed 4.16.05 directly or by update, you must update the apigee-setup utility:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-setup update
  
  This update to apigee-service installs the update.sh utility in <inst_dir>/apigee/apigee-setup/bin.
Depending on your current version of Edge, you must either install or update the apigee-validate utility on the Management Server.
1. If you are currently using Edge 4.16.05: update the apigee-validate utility on the Management Server:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-validate update
2. If you are currently using Edge 4.16.01: install the apigee-validate utility on the Management Server:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-validate install
  
  Note: If you have installed the apigee-validate utility on a Message Processor node when installing 4.16.01, you can update it by using the following command on that node:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-validate update
  
  However, as of 4.16.05 and later, Apigee recommends that you install and run the apigee-validate utility on the Management Server.
3. If you are upgrading from 4.16.01: Edit the config file passed to the apigee-validate utility. In the 4.16.01 Edge release, the config file used by apigee-validate required the following properties:
  APIGEE_ADMINPW=sysAdminPword
  MP_POD=gateway
  REGION=dc-1
  
  In this release, the config file only requires the APIGEE_ADMINPW property. You can remove the other two properties from the file.
Update the apigee-provision utility:
> /opt/apigee/apigee-service/bin/apigee-service apigee-provision update
Run the update utility on your nodes in the order described below in "Order of machine update" below:
> /opt/apigee/apigee-setup/bin/update.sh -c component -f configFile

The only requirement on the config file is that the configuration file must be accessible or readable by the "apigee" user.

Use the “-c” option to specify the component to update. The list of possible components includes:
ldap = OpenLDAP
cs = Cassandra
zk = Zookeeper
qpid = qpidd
ps = postgresql
edge =All Edge components except Edge UI: Management Server, Message Processor, Router, QPID Server, Postgres Server
ui = Edge UI
all = update all components on machine (only use for an Edge aio installation profile or an API BaaS asa installation profile)
e = ElasticSearch
b = API BaaS Stack
p = API BaaS Portal
ebp = ElasticSearch, API BaaS Stack, and API BaaS Portal on the same node
Test the update by running the apigee-validate utility on the Management Server, as described in Test the install.
If you installed a new Postgres standby node, decommission the node as described above in Decommissioning a Postgres node.

To later rollback the update, use the procedure described in 4.16.09 Rollback Process.

Procedure for updating to 4.16.09 from a local repo

If your Edge nodes are behind a firewall, or in some other way are prohibited from accessing the Apigee repository over the Internet, then you can perform the update from a local repository, or mirror, of the Apigee repo.

After you create a local Edge repository, you have two options for updating Edge from the local repo:

Create a .tar file of the repo, copy the .tar file to a node, and then update Edge from the .tar file.
Install a webserver on the node with the local repo so that other nodes can access it. Apigee provides the Nginx webserver for you to use, or you can use your own webserver.

To update from a local 4.16.09 repo:

If you are currently using Postgres master-standby replication, install a new Postgres standby node as described above in Installing a new Postgres standby node.
Create a local 4.16.09 repo as described in "Create a local Apigee repository" at Install the Edge apigee-setup utility.
Note: If you already have an existing 4.16.01 or 4.16.05 repo, you can add the 4.16.09 repo to it as described in "Update a local Apigee repository" at Install the Edge apigee-setup utility.
To install apigee-service from a .tar file:
1. On the node with the local repo, use the following command to package the local repo into a single .tar file named /opt/apigee/data/apigee-mirror/apigee-4.16.09.tar.gz:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-mirror package
2. Copy the .tar file to the node where you want to update Edge. For example, copy it to the /tmp directory on the new node.
3. On the new node, untar the file to the /tmp directory:
  > tar -xzf apigee-4.16.09.tar.gz
  
  This command creates a new directory, named repos, in the directory containing the .tar file. For example /tmp/repos.
4. Install the Edge apigee-service utility and dependencies from /tmp/repos:
  > sudo bash /tmp/repos/bootstrap_4.16.09.sh apigeeprotocol="file://" apigeerepobasepath=/tmp/repos
  
  Notice that you include the path to the repos directory in this command.
To install apigee-service using the Nginx webserver:
1. Configure the Nginx web server as described in "Install from the repo using the Nginx webserver" at Install the Edge apigee-setup utility.
2. On the remote node, download the Edge bootstrap_4.16.09.sh file to /tmp/bootstrap_4.16.09.sh:
  > /usr/bin/curl http://uName:pWord@remoteRepo:3939/bootstrap_4.16.09.sh -o /tmp/bootstrap_4.16.09.sh
  
  where uName:pWord are the username and password you set above for the repo, and remoteRepo is the IP address or DNS name of the repo node.
3. On the remote node, install the Edge apigee-service utility and dependencies:
  > sudo bash /tmp/bootstrap_4.16.09.sh apigeerepohost=remoteRepo:3939 apigeeuser=uName apigeepassword=pWord apigeeprotocol=http://
  
  where uName:pWord are the repo username and password.
Use apigee-service to update the apigee-setup utility:
1. If you installed 4.16.01 by upgrading Edge version 4.15.07.0x, you must install the apigee-setup utility:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-setup install
  
  This command installs the update.sh utility in <inst_dir>/apigee/apigee-setup/bin.
  
  If you already installed the apigee-setup utility, then update it:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-setup update
2. If you installed 4.16.01 directly, meaning you did not perform an upgrade from 4.15.07.0x, you must update the apigee-setup utility:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-setup update
  
  This update to apigee-service installs the update.sh utility in <inst_dir>/apigee/apigee-setup/bin.
3. If you installed 4.16.05 directly or by update, you must update the apigee-setup utility:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-setup update
  
  This update to apigee-service installs the update.sh utility in <inst_dir>/apigee/apigee-setup/bin.
Depending on your current version of Edge, you must either install or update the apigee-validate utility on the Management Server.
1. If you are currently using Edge 4.16.05: update the apigee-validate utility on the Management Server:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-validate update
2. If you are currently using Edge 4.16.01: install the apigee-validate utility on the Management Server:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-validate install
  
  Note: If you have installed the apigee-validate utility on a Message Processor node when installing 4.16.01, you can update it by using the following command on that node:
  > /opt/apigee/apigee-service/bin/apigee-service apigee-validate update
  
  However, as of 4.16.05 and later, Apigee recommends that you install and run the apigee-validate utility on the Management Server.
3. If you are upgrading from 4.16.01: Edit the config file passed to the apigee-validate utility. In the 4.16.01 Edge release, the config file used by apigee-validate required the following properties:
  APIGEE_ADMINPW=sysAdminPword
  MP_POD=gateway
  REGION=dc-1
  
  In this release, the config file only requires the APIGEE_ADMINPW property. You can remove the other two properties from the file.
Update the apigee-provision utility:
> /opt/apigee/apigee-service/bin/apigee-service apigee-provision update
Run the update utility on your nodes in the order described below in "Order of machine update" below:
> /opt/apigee/apigee-setup/bin/update.sh -c component -f configFile

The only requirement on the config file is that the configuration file must be accessible or readable by the "apigee" user.

Use the “-c” option to specify the component to update. The list of possible components includes:
ldap = OpenLDAP
cs = Cassandra
zk = Zookeeper
qpid = qpidd
ps = postgresql
edge =All Edge components except Edge UI: Management Server, Message Processor, Router, QPID Server, Postgres Server
ui = Edge UI
all = update all components on machine (only use for an Edge aio installation profile or an API BaaS asa installation profile)
e = ElasticSearch
b = API BaaS Stack
p = API BaaS Portal
ebp = ElasticSearch, API BaaS Stack, and API BaaS Portal on the same node
Test the update by running the apigee-validate utility on the Management Server, as described in Test the install.
If you installed a new Postgres standby node, decommission the node as described above in Decommissioning a Postgres node.

To later rollback the update, use the procedure described in 4.16.09 Rollback Process.

Order of machine update

The order that you update the machines in an Edge installation is important. The most important considerations to an update are:

You must update all Cassandra and ZooKeeper nodes before you update any other nodes.
For any machine with multiple Edge components (Management Server, Message Processor, Router, QPID Server but not Postgres Server), use the "-c edge" option to update them all at the same time.
If a step specifies that it should be performed on multiple machines, perform it in the specified machine order.
There is no separate step to update Monetization. It is updated when you specify the "-c edge" option.