Node by node operating system upgrade for Apigee Edge for Private Cloud

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

This document outlines an alternative method for upgrading the Operating System (OS) of the servers in your Apigee Edge for Private Cloud cluster. Instead of setting up an entirely new parallel data center, this node-by-node approach allows you to upgrade your OS in-place, one machine/node at a time.

Compatibility

Before initiating the node by node OS upgrade, verify that your current version of Apigee Edge for Private Cloud supports the target OS version. To verify, please refer to the OS compatibility matrix.

Prerequisites

Before you begin the node by node upgrade, confirm that you've completed the following tasks:

Back up all nodes

Before following the procedure, we recommend that you perform a complete backup of all nodes, such as at the VM-level backup, component-level backup, (or both), for safety reasons. This allows you to have a backup plan, in case any of the following steps does not function properly. For more information on component-level backup, see Back up and Restore.

Ensure Edge is running

Ensure that Edge is up and running during the update process by using the command:

/opt/apigee/apigee-service/bin/apigee-all status

Order of node replacement

We recommend upgrading nodes in the order listed below, based on component type:

Cassandra
Zookeeper
Postgres
LDAP
Qpid
Message-Processor
Router
Management-server
UI/UE
SSO

Additional scenarios

When performing a node-by-node OS upgrade, if any of the following configurations are present in the existing setup, please consider these additional steps:

apigee-mtls - If apigee-mtls is enabled, node by node OS upgrade (rolling) is not supported for this setup. For upgrading the same, please follow the Upgrading apigee-mtls guide.

Cassandra

When replacing Cassandra, consider replacing the non-seed nodes first, followed by the seed nodes.

Special cases to be considered for Cassandra are as follows:

Cassandra Native-mTLS - If native-mTLS is enabled and enforced, remove the enforcement on mTLS-enabled nodes by setting conf_cassandra_client_encryption_optional=true. For more information, see the native-mTLS documentation.
Cassandra internode encryption enabled - Refer to the Enable Cassandra Internode Encryption to ensure that the certificates of existing Cassandra nodes are included in the newly replaced node's truststore and vice versa.

Steps to replace Cassandra node:

Back up the node by taking virtual machine snapshots (if possible) and take Apigee backup for Cassandra on all the nodes using the following command:
```
apigee-service apigee-cassandra backup
```
Acquire new node with the upgraded OS.
Apply the OS-specific prerequisites.
Download and run the Edge file to install the apigee-service utility. The bootstrap of Apigee version in use, run bootstrap to get apigee-service and install apigee-setup.
NOTE: Ensure that the new node does not contain any data directory on the path of apigee-cassandra data directory.

List out the seed nodes:

apigee-service apigee-cassandra configure -search conf_cassandra_seeds

Update the configuration file by changing the existing Cassandra node's IP and replacing it with new node IP address.
NOTE: Ensure that the order of IP addresses being replaced is preserved. For example, if replacing IP2 with IP4.

Existing CASS_HOSTS
```
CASS_HOSTS="$IP1 $IP2 $IP3"
```
Newly Updated CASS_HOSTS
```
CASS_HOSTS="$IP1 $IP4 $IP3"
```
Reconfigure the existing nodes on the cluster with the updated new Cassandra node IP (except for the node to be replaced). Please use the configuration file reference.
```
/opt/apigee/apigee-setup/bin/setup.sh -p c -f updatedConfigFile
```

Stop and deregister Cassandra on the old node:

Stop Cassandra on the node to be replaced:
```
apigee-service apigee-cassandra stop
```

List the UUID of Cassandra node to be decommissioned (if multiple data centers consider accordingly):

apigee-adminapi.sh servers list -r dc-1 -p central -t application-datastore --admin AdminEmailID --pwd 'AdminPassword' --host localhost

Deregister the datastore types associated with the existing Cassandra UUID:

curl -u AdminEmailID:'AdminPassword' -X POST http://MS_IP:8080/v1/servers -d "type=cache-datastore&type=user-settings-datastore&type=scheduler-datastore&type=audit-datastore&type=apimodel-datastore&type=application-datastore&type=edgenotification-datastore&type=identityzone-datastore&type=auth-datastore&region=dc-1&pod=central&uuid=old_cassandra_uuid&action=remove"

Delete the server.
NOTE: Do not run this step if Cassandra and other Edge components, such as management-server, are on the same node. This may be the case if your configuration uses a standard Apigee 5-node setup.
```
curl -u AdminEmailID:AdminPassword -X DELETE http://MS_IP:8080/v1/servers/old_cassandra_uuid
```

Prepare a new node for apigee-cassandra host. Add the following line to the cassandra.properties file under /opt/apigee/customer/application/cassandra.properties (create if not already present) on the new Cassandra node:
```
conf_jvm_options_custom_settings=-Dcassandra.replace_address=IPOfOldCassandraNode -Dcassandra.allow_unsafe_replace=true
```
Confirm that the owner and group is set to apigee:apigee in the cassandra.properties file. If required, use the following command to update the owner and group:
```
chown apigee:apigee /opt/apigee/customer/application/cassandra.properties
```

Install Cassandra and run the setup on the newly added node:

apigee-service apigee-cassandra install

apigee-service apigee-cassandra setup -f updatedConfigFile

After setting up the new node, remove the additional config added to /opt/apigee/customer/application/cassandra.properties and restart apigee-cassandra:

conf_jvm_options_custom_settings=-Dcassandra.replace_address=IPOfOldCassandraNode -Dcassandra.allow_unsafe_replace=true

apigee-service apigee-cassandra restart

Check for the status of the new replaced node:

apigee-service apigee-cassandra status

/opt/apigee/apigee-cassandra/bin/nodetool status

Rebuild the new node added to the cluster:

/opt/apigee/apigee-cassandra/bin/nodetool rebuild -dc dc-1

Resetup the existing nodes of Router, Message-Processor, Qpid, Postgres, and Management servers respectively by updating the New Cassandra Node IP in the configuration file:
```
/opt/apigee/apigee-setup/bin/setup.sh -p r|mp|qs|ps|ms|mo -f updatedConfigFile
```
NOTE: If the Router and Message Processor are on the same node use -p rmp.

Remove/uninstall Cassandra from older node:

apigee-service apigee-cassandra uninstall

Repeat for other Cassandra nodes in cluster.

Zookeeper

Refer to the steps below to replace Zookeeper nodes. Consider replacing the follower nodes first and the leader nodes second, to ensure that the ensemble maintains quorum.

Back up the node by taking virtual machine snapshots (if possible) and take Apigee backup for Zookeeper on all nodes.

Stop and uninstall Zookeeper on existing node:

apigee-service apigee-zookeeper stop

apigee-service apigee-zookeeper uninstall

Acquire new node with the upgraded OS.
Apply the OS-specific prerequisites.
Download bootstrap of Apigee version in use, run bootstrap to get apigee-service and install apigee-setup.
Update the configuration file by removing the stale/deleted node IP addresses and replacing it with new node IP for Zookeeper. For more details, see the configuration file reference:
NOTE: Ensure that the order of IP being replaced is preserved. For example, if replacing IP2 with IP4.

Existing ZK configuration
```
ZK_HOSTS="$IP1 $IP2 $IP3"
ZK_CLIENT_HOSTS="$IP1 $IP2 $IP3"
    
```
Updated ZK configuration
```
ZK_HOSTS="$IP1 $IP4 $IP3"
ZK_CLIENT_HOSTS="$IP1 $IP4 $IP3"
    
```
Install, setup and start Zookeeper on new node using updated config file:
```
/opt/apigee/apigee-setup/bin/setup.sh -p zk -f updatedConfigFile
```
Make changes to the existing Zookeeper node with updatedConfigFile:
```
/opt/apigee/apigee-setup/bin/setup.sh -p zk -f updatedConfigFile
```
Resetup the existing nodes of Router, Message-Processor, Qpid, Postgres and Management servers respectively by updating the New Zookeeper node IP in the configuration file:
```
/opt/apigee/apigee-setup/bin/setup.sh -p r|mp|qs|ps|ms|mo -f updatedConfigFile
```
NOTE: If the Router and Message Processor are on the same node use -p rmp.
Repeat for all Zookeeper nodes.

Postgres

This section outlines a zero downtime approach to upgrading the Postgres nodes, where you will complete the outlined steps. Detailed instructions for completing each step are provided in the subsections to follow.

Replace any of the existing standby node with the targeted OS.
Promote the new standby node to master.
Decommission the old master.
Continue to replace additional standby nodes.

Before starting the replace, it's recommended to stop the edge-qpid-server to minimize any potential loss of analytics messages.

apigee-service edge-qpid-server stop

Standby

Get UUID of the edge-postgres-server nodes:

curl http://pg_standby_ip:8084/v1/servers/self

Stop Postgres:

apigee-service edge-postgres-server stop

apigee-service apigee-postgresql stop

Remove the existing Postgres standby from analytics group by following these steps.

Uninstall apigee-postgres and postgres-server:

apigee-service edge-postgres-server uninstall

apigee-service apigee-postgresql uninstall

Acquire new node with the upgraded OS.
Apply the OS-specific prerequisites.
Download bootstrap of Apigee version in use, run bootstrap to get apigee-service and install apigee-setup.
Setup master-standby replication on the existing master by updating the configuration file with the EXISTING_PG_MASTER_IP and NEW_PG_STANDBY_IP. For more details, see the configuration file reference:
```
PG_MASTER=EXISTING_PG_MASTER_IP
```
```
PG_STANDBY=NEW_PG_STANDBY_IP
```
```
apigee-service apigee-postgresql setup-replication-on-master -f updatedConfigFile
```

Install, setup and start Postgres with master-standby configs on the new standby node:

PG_MASTER=EXISTING_PG_MASTER_IP

PG_STANDBY=NEW_PG_STANDBY_IP

/opt/apigee/apigee-setup/bin/setup.sh -p ps -f updatedConfigFile

Verify the standby setup:

apigee-service apigee-postgresql postgres-check-standby

Master

Get UUID of the existing Postgres master node:
```
curl http://pg_ip:8084/v1/servers/self
```
Promote the new upgraded OS standby node to master.

Stop Postgres on the existing master:

apigee-service apigee-postgresql stop

apigee-service edge-postgres-server stop

Promote the new standby node to master:

apigee-service apigee-postgresql promote-standby-to-master EXISTING_PG_MASTER_IP

Remove the old master node from the analytics and consumer groups.
Add the new master node to the analytics and consumer groups.
If using monetization, update the new Postgres master node and perform the following steps.
Once the master node is replaced, the edge-qpid-server can be started with an updated master node configuration:
```
apigee-service edge-qpid-server start
```
If you are using monetization, restart the Management Server and Message Processor as well:
```
apigee-service edge-management-server restart
```

Stop and uninstall apigee-postgres and postgres-server on old master:

apigee-service edge-postgres-server uninstall

apigee-service apigee-postgresql uninstall

Adding new standby

Acquire new node with the upgraded OS.
Apply the OS-specific prerequisites.
Download bootstrap of Apigee version in use, run bootstrap to get apigee-service and install apigee-setup.
Setup master-standby replication using new master IP and new standby node IP in the new master & new standby node configuration file. For more details, see the configuration file reference:
```
apigee-service apigee-postgresql setup-replication-on-master -f updatedConfigFile
```

Install and setup on the new standby node:

/opt/apigee/apigee-setup/bin/setup.sh -p ps -f updatedConfigFile

Add the new standby to the analytics groups by following the steps for master-standby setup.
Note: Make sure while adding new Standby nodes to analytics groups, the existing Postgres needs to be removed first and then simultaneously added back with the standby.

LDAP

Replacing the LDAP node would require the Management Server to stop and any of the Management APIs would be considered unavailable for the duration of this node replacement.

Stop Management Server on all nodes:

apigee-service edge-management-server stop

Decommissioning the existing LDAP node:
NOTE: If using version 4.53.01 or higher, make sure to use mdb in any LDAP configurations. If youa are using version 4.53.00 or less, ensure to use bdb in LDAP configuration.
1. Take VM level backup/snapshot of the LDAP node.
2. Before stopping LDAP, make sure to capture the count of LDAP data, which can be verified later, for example:
```
ldapsearch -o ldif-wrap=no -b "dc=apigee,dc=com" -D "cn=manager,dc=apigee,dc=com" -H ldap://:10389 -LLL -x -w Secret123 | wc -l
```
3. For active-passive setup:
  1. Make the current passive to active by adding a file mark_writable.ldif on the passive node with the following content:
```
dn: olcDatabase={2}bdb,cn=config
changetype: modify
replace: olcReadOnly
olcReadOnly: FALSE
```
  2. Execute the following command on the passive node to make it active:
```
ldapmodify -a -x -w "$APIGEE_LDAPPW" -D "$CONFIG_BIND_DN" -H "ldap://:10389" -f mark_writable.ldif
```
  3. Continue with Step d) for active-active to decommission nodes.
4. In an Active-Active LDAP setup, break replication between the two LDAP nodes and update configs. Following the steps as listed in LDAP decommissioning document.
5. Stop and take backup of LDAP data and uninstall the existing LDAP node (skip for active-active setup if already done):
```
apigee-service apigee-openldap stop
```
```
apigee-service apigee-openldap backup
```
```
apigee-service apigee-openldap uninstall
```
Acquire new node with the upgraded OS.
Apply the OS-specific prerequisites.
Download bootstrap of Apigee version in use, run bootstrap to get apigee-service and install apigee-setup.
Setup new LDAP node:
1. Update the configFile for new node based on setup, you can refer Edge configuration file reference for, based on active only or active-active setup:
  - LDAP_TYPE
  - LDAP_PEER
  - LDAP_SID
  - USE_LDAP_REMOTE_HOST=y/n
    Note: Set USE_LDAP_REMOTE_HOST to 'y' or 'n' based on whether LDAP is configured on the same node as the Management Server or not.
2. Setup and start LDAP on new node with updatedConfigFile:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ld -f updatedConfigFile
```
3. Back up the LDAP data from older nodes (if active-active setup can be skipped):
  1. Take a sample backup on the new node, this is so that it generates a default backup folder/path:
```
apigee-service apigee-openldap backup
```
  2. Move your backed up data from old node, to the backup folder generated.
  3. Stop apigee-openldap before restoring the data:
```
apigee-service apigee-openldap stop
```
  4. Restore the backup taken from the older node:
```
apigee-service apigee-openldap restore backup_file
```
    For example:
```
apigee-service apigee-openldap restore 2026.02.09,10.28.20
```
Repeat for other LDAP node (if active-active).
Resetup and start Management Server with updated LDAP host configs:
```
/opt/apigee/apigee-setup/bin/setup.sh -p mt -f updatedConfigFile
```

Reset up and start SSO (if applicable):

apigee-service apigee-sso setup -f updatedConfigFile

apigee-service apigee-sso restart

(Optional) Configure the second active directory to read-only by following the documentation steps.

Qpid

To replace an existing Qpid node with an upgraded OS node, we need to make sure of the following steps:

Stop receiving any messages to the existing node queue from message processors.
Drain the existing messages in the queue.
Stop and Decommission the existing node.

Decommissioning existing nodes. To make sure that no messages are lost additional measures are listed along:

Stop Qpid broker:
```
apigee-service apigee-qpidd stop
```
Block incoming traffic on port 5672 from all the Message Processors:
```
iptables -A INPUT -p tcp --dport 5672 -s MP_IP -j DROP
```
Start Qpid broker again to drain existing messages:
```
apigee-service apigee-qpidd start
```
Make sure all the messages in the queue are drained to the consumer, before you start to decommission.

Get UUID of the edge-qpid-server node:

curl http://qpid_node_ip:8083/v1/servers/self

Complete the decommission or remove steps for Qpid from analytics for apigee-qpidd and edge-qpid-server.

Stop and Uninstall both apigee-qpidd and edge-qpid-server:

apigee-service apigee-qpidd stop

apigee-service edge-qpid-server stop

apigee-service apigee-qpidd uninstall

apigee-service edge-qpid-server uninstall

Add new node:

Acquire new node with the upgraded OS.
Apply the OS-specific prerequisites.
Download bootstrap of Apigee version in use, run bootstrap to get apigee-service and install apigee-setup.
Setup and run the Qpid services on the new node and add these to the consumer groups, following these steps. Consider the same steps to add other groups like for monetization.

If monetization service is enabled, restart Management Server and Message Processor:

apigee-service edge-management-server restart

apigee-service edge-message-processor restart

Repeat on all Qpid nodes.

Message-processor

Get UUID of the edge-message-processor node:
```
curl http://mp_ip:8082/v1/servers/self
```
Decommission and uninstall Message Processor node.
If monetization service enabled, uninstall the edge-mint-message-processor.

Uninstall Message Processor:

apigee-service edge-message-processor uninstall

Acquire new node with the upgraded OS.
Apply the OS-specific prerequisites.
Download bootstrap of Apigee version in use, run bootstrap to get apigee-service and install apigee-setup.
Add, setup and start Message Processor to the cluster, following the steps.
If monetization is enabled, set up the monetization service along with the Message Processor. For more information, refer the following documentation.

Router

Get UUID of the edge-router node:

curl http://router_ip:8081/v1/servers/self

Disable reachability and Verify if its reachable:

iptables -A INPUT -i eth0 -p tcp --dport 15999 -j REJECT

curl -vvv -X GET http://router_ip:15999/v1/servers/self/reachable

Decommission and uninstall Router node.
Uninstall Router:
```
apigee-service edge-router uninstall
```
Acquire new node with the upgraded OS.
Apply the OS-specific prerequisites.
Download bootstrap of Apigee version in use, run bootstrap to get apigee-service and install apigee-setup.
Add, setup and start router to the cluster.

Management server

Single management server only

Acquire new node with the upgraded OS.
Apply the OS-specific prerequisites.
Download bootstrap of Apigee version in use, run bootstrap to get apigee-service and install apigee-setup.
Add Management Server using the updated IP and configuration file which can be referred from here and run the setup command:
```
/opt/apigee/apigee-setup/bin/setup.sh -p mt -f updatedConfigFile
```

Resetup and restart UI with new Management IP configuration.

/opt/apigee/apigee-setup/bin/setup.sh -p ui -f updatedConfigFile

Resetup SSO with new Management IP configuration (if applicable):

/opt/apigee/apigee-setup/bin/setup.sh -p sso -f updatedConfigFile

Resetup New UI (UE) with new Management IP configuration (if applicable):
```
/opt/apigee/apigee-setup/bin/setup.sh -p ue -f updatedConfigFile
```
If monetization is enabled:
1. Resetup monetization MS with new Management IP configuration (if applicable), referring monetization configuration documentation.
```
/opt/apigee/apigee-setup/bin/setup.sh -p mo -f updatedConfigFile
```
2. Restart Management Server:
```
apigee-service edge-management-server restart
```
Decommission the Management Server on the existing/old node, following the steps.

If monetization is enabled, remove the Management Server UUID from the analytics groups. Example:

curl -v -u AdminEmail:AdminPassword -X DELETE "http://localhost:8080/v1/analytics/groups/ax/mint_axgroup/servers?uuid=MS_UUID&type=consumer-server"

Uninstall the monetization Management Server:

apigee-service edge-mint-management-server uninstall

Multiple management servers

Decommission one of the Management Servers using the other on existing node, following the steps.

Stop and uninstall Management Server:

apigee-service edge-management-server stop

apigee-service edge-management-server uninstall

Acquire new node with the upgraded OS.
Apply the OS-specific prerequisites.
Download bootstrap of Apigee version in use, run bootstrap to get apigee-service and install apigee-setup.
Add Management Server using the updated IP and configuration file which can be referred from here and run the setup command:
```
/opt/apigee/apigee-setup/bin/setup.sh -p mt -f updatedConfigFile
```

Resetup and restart UI with updated Management IP configuration:

/opt/apigee/apigee-setup/bin/setup.sh -p ui -f updatedConfigFile

(Only if SSO enabled) Resetup and restart SSO with updated Management IP configuration:
```
/opt/apigee/apigee-setup/bin/setup.sh -p sso -f updatedConfigFile
```
(Only if New UI enabled) Resetup and restart New UI with updated Management IP configuration:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ue -f updatedConfigFile
```
Repeat for other Management Server.

UI/UE

Take backup/VM snapshot of UI node.

Stop and uninstall UI:

apigee-service edge-ui stop

apigee-service edge-ui uninstall

(If using New UI only) Stop and uninstall New UI:

apigee-service edge-management-ui stop

apigee-service edge-management-ui uninstall

Acquire new node with the upgraded OS.
Apply the OS-specific prerequisites.
Download bootstrap of Apigee version in use, run bootstrap to get apigee-service and install apigee-setup.
Setup and run edge-ui on the new node, with updated IP configurations:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ui -f updatedConfigFile
```
(Only if SSO enabled) Resetup and run SSO with UI new node IP configurations as applicable:
```
/opt/apigee/apigee-setup/bin/setup.sh -p sso -f updatedConfigFile
```

(Only if New UI enabled) Setup and run New UI on the new node, with updated IP configurations:

/opt/apigee/apigee-setup/bin/setup.sh -p ue -f updatedConfigFile

apigee-service edge-management-ui configure-sso -f updatedConfigFile

apigee-service edge-ui configure-sso -f updatedConfigFile

SSO

Take backup/VM snapshot of UI node.

Stop and uninstall SSO:

apigee-service apigee-sso stop

apigee-service apigee-sso uninstall

Acquire new node with the upgraded OS.
Apply the OS-specific prerequisites.
Download bootstrap of Apigee version in use, run bootstrap to get apigee-service and install apigee-setup.
Add configurations and keys for SSO following the guide, considering the new node IP.

Setup and start SSO:

/opt/apigee/apigee-setup/bin/setup.sh -p sso -f updatedConfigFile

Update config with the new SSO node IPs and reconfigure SSO for UI:
```
apigee-service edge-ui configure-sso -f updatedConfigFile
```

(If using New UI) Run the configure-sso on the New UI node, referring the documentation:

apigee-service edge-management-ui configure-sso -f updatedConfigFile

apigee-service edge-ui configure-sso -f updatedConfigFile