> [!CAUTION] > **Caution:** This documentation covers Edge for Private Cloud 4.52.00, which is no longer supported. Google does not provide security patches, bug fixes, or technical support for this version. The current supported version is 4.54.00. See the [Edge for Private Cloud documentation](https://docs.apigee.com/private-cloud/latest/overview) for the current version, or [All versions](https://docs.apigee.com/private-cloud/versions) to compare releases. At times, you might need to decommission a data center. For example, if you are [upgrading your operating system](https://docs.apigee.com/how-to-guides/upgrade-rhel), you need to install the new operating system in a new data center and then decommission the old data center. The following sections present an example of decommissioning a data center, in which there are two data centers, dc-1 and dc-2, on a [12-node clustered installation](https://docs.apigee.com/private-cloud/v4.52.00/installation-topologies#12hostclusteredinstallation): - dc-1 is the data center to be decommissioned. - dc-2 is a second data center, which is used in the decommissioning procedure. If you are upgrading your operating system, dc-2 could be the data center in which you have installed the new version of the operating system (OS). However, installing a new OS isn't required to decommission a data center. > [!NOTE] > **Note:** The decommissioning produre described below is for two data centers. If you have more than two data centers, Apigee recommends that you work with a Customer Success partner to decommission a data center. ## Considerations before decommissioning a data center Keep the following considerations in mind when you decommission a data center: - Block all runtime and management traffic to the data center being decommissioned and redirect them to other data centers. - After decommissioning the data center, you will have reduced capacity in your Apigee cluster. To make up for it, consider increasing capacity in the remaining data centers or adding data centers after decommissioning. - During the decommission process, there is a potential for analytics data loss, depending on which analytics components are installed in the data center being decommissioned. You can find more details in [Add or remove Qpid nodes](https://docs.apigee.com/private-cloud/v4.52.00/add-or-remove-qpid-nodes). - Before you decommission a data center, you should understand how all the components are configured across all data centers, especially the OpenLDAP, ZooKeeper, Cassandra, and Postgres servers. You should also take backups of all components and their configurations. ## Before you start > [!CAUTION] > **Caution:** > > - If you restart a component after deregistering it, without uninstalling the component completely, the component will register itself again Edge. You must uninstall the component completely or isolate the node network once decommissioned. > - Uninstalling a component will not remove data from `/opt/apigee/data`. > - During the decommissioning process you will be required to reconfigure multiple Edge components. While an Edge component is being reconfigured, that specific component on that specific node will be unavailable. - Management Server: All the decommission steps are highly dependent on the Management Server. If you have only one Management Server available, we recommend that you install a new Management Server component on a data center other than dc-1 before decommissioning the Management Server on dc-1, and make sure that one of the Management Servers is always available. - Router: Before decommissioning a Router, disable reachability of Routers by blocking the port 15999. Ensure no runtime traffic is being directed at the Routers being decommissioned. - Cassandra and ZooKeeper: The sections below describe how to decommission dc-1 in a two data center setup. If you have more than two data centers, make sure to remove all references to the node being decommissioned (dc-1 in this case) from all silent configuration files across all the remaining data centers. For Cassandra nodes that are to be decommissioned, drop those hosts from `CASS_HOSTS`. The remaining Cassandra nodes should remain in the original ordering of `CASS_HOSTS`. - Postgres: If you decommission Postgres master, make sure to promote any one of the available standby nodes as a new postgres master. While the QPID server keeps a buffer in the queue, if Postgres master is unavailable for a longer time, you risk losing analytics data. > [!CAUTION] > **Caution:** You must remove and add Postgresql servers from the analytics group and the consumer group at the same time. You might lose some analytics data if there is a substantial delay between the two steps. > > Suggestion: The Message Processor has a small queue where it keeps > analytics data in memory > for a brief time, which you can usw to help avoid loss of data. Before you make changes to the analytics group, > block port 5672 in the data center that is taking traffic, > and make sure that you quickly re-enable this port after making the changes, > as there is a chance of data overflow. > > For example, to temporarily stop analytics traffic from > Message Processors to the Qpid server: > 1. Enter > > ``` > iptables -A INPUT -p tcp --destination-port 5672 ! -s `hostname` -i eth0 -j DROP > ``` > 2. Follow the axgroup change steps. > 3. Remove the iptable rule after axgroup changes: > > ``` > iptables -F > ``` ### Prerequisites - Before decommissioning any component, we recommend that you perform a complete backup of all nodes. Use the procedure for your current version of Edge to perform the backup. For more information on backup, see [Backup and restore](https://docs.apigee.com/private-cloud/v4.52.00/backup-and-restore). **Note:** If you have multiple Cassandra or ZooKeeper nodes, back them up one at a time, as the backup process temporarily shuts down ZooKeeper. - Ensure that Edge is up and running before decommissioning, using the command: ``` /opt/apigee/apigee-service/bin/apigee-all status ``` - Make sure that no runtime traffic is currently arriving at the data center you are decommissioning. ### Order of decommissioning components If you install Edge for Private Cloud on multiple nodes, you should decommission Edge components on those nodes in the following order: 1. [Edge UI](https://docs.apigee.com/private-cloud/v4.52.00/decommission-data-center#edge-ui) (edge-ui) 2. [Management Server](https://docs.apigee.com/private-cloud/v4.52.00/decommission-data-center#management-server) (edge-management-server) 3. [OpenLDAP](https://docs.apigee.com/private-cloud/v4.52.00/decommission-data-center#openldap) (apigee-openldap) 4. [Router](https://docs.apigee.com/private-cloud/v4.52.00/decommission-data-center#router) (edge-router) 5. [Message Processor](https://docs.apigee.com/private-cloud/v4.52.00/decommission-data-center#message-processor) (edge-message-processor) 6. [Qpid Server and Qpidd](https://docs.apigee.com/private-cloud/v4.52.00/decommission-data-center#qpid-server-and-qpidd) (edge-qpid-server and apigee-qpidd) 7. [Postgres](https://docs.apigee.com/private-cloud/v4.52.00/decommission-data-center#postgres-and-postgresql) and PostgreSQL database (edge-postgres-server and apigee-postgresql) 8. [ZooKeeper (apigee-zookeeper)](https://docs.apigee.com/private-cloud/v4.52.00/decommission-data-center#zookeeper-and-cassandra) 9. [Cassandra (apigee-cassandra)](https://docs.apigee.com/private-cloud/v4.52.00/decommission-data-center#zookeeper-and-cassandra) The following sections explain how to decommission each component. ## Edge UI To stop and uninstall the Edge UI component of dc-1, enter the following commands: ``` /opt/apigee/apigee-service/bin/apigee-service edge-ui stop ``` ``` /opt/apigee/apigee-service/bin/apigee-service edge-ui uninstall ``` ## Management Server > [!CAUTION] > **Caution:** If you have only one Management Server available, we recommend that you install a new Management Server component on a data center other than dc-1 before decommissioning the Management Server on dc-1, and make sure that one of the Management Servers is always available. To decommission the Management Server on dc-1, do the following steps: 1. Stop the Management Server on dc-1: ``` apigee-service edge-management-server stop ``` 2. Find the UUID of Management Server registered in dc-1: ``` curl -u :'' \ -X GET “http://{MS_IP}:8080/v1/servers?pod=central®ion=dc-1&type=management-server” ``` 3. Deregister server's type: ``` curl -u :'' -X POST http://{MS_IP}:8080/v1/servers \ -d "type=management-server®ion=dc-1&pod=central&uuid=UUID&action=remove" ``` 4. Delete the server. **Note:** If other components are also installed on this server, deregister all of them first before deleting the UUID. ``` curl -u :' -X DELETE http://{MS_IP}:8080/v1/servers/{UUID} ``` 5. Uninstall Management Server component on dc-1: ``` /opt/apigee/apigee-service/bin/apigee-service edge-management-server uninstall ``` ## Open LDAP This section explains how to decommission OpenLDAP on dc-1. **Note:** If you have more than two data centers, see [Setups with more than two data centers](https://docs.apigee.com/private-cloud/v4.52.00/decommission-data-center#setups-with-more-than-two-data-centers) below. To decommission OpenLDAP on dc-1, do the following steps: 1. Back up the dc-1 OpenLDAP node by following the steps in [How to back up](https://docs.apigee.com/private-cloud/v4.52.00/how-perform-backup). 2. Break the data replication between the two data centers, dc-1 and dc-2, by executing the following steps in both data centers. > [!NOTE] > **Note:** If your setup has more than two data centers, you need to: > 1. Identify which OpenLDAP node is replicating to the node being decommissioned. > 2. Change the replication for that node to replicate > to a different node to complete the replication circle. > > For example, suppose you have three data centers, in which node 1 replicates to node 2, > node 2 replicates to node 3, and node 3 replicates to node 1. If you are decommissioning > node 1, > you need to break the replication on node 1 and also modify replication for node 3 to > replicate to node 2. 1. Check the present state: ``` ldapsearch -H ldap://{HOST}:{PORT} -LLL -x -b "cn=config" -D "cn=admin,cn=config" -w {credentials} -o ldif-wrap=no 'olcSyncRepl' | grep olcSyncrepl ``` The output should be similar to the following: ``` olcSyncrepl: {0}rid=001 provider=ldap://{HOST}:{PORT}/ binddn="cn=manager,dc=apigee,dc=com" bindmethod=simple credentials={credentials} searchbase="dc=apigee,dc=com" attrs="*,+" type=refreshAndPersist retry="60 1 300 12 7200 +" timeout=1 ``` 2. Create a file `break_repl.ldif` containing the following commands: ``` dn: olcDatabase={2}bdb,cn=config changetype: modify delete: olcSyncRepl dn: olcDatabase={2}bdb,cn=config changetype: modify delete: olcMirrorMode ``` 3. Run the `ldapmodify` command: ``` ldapmodify -x -w {credentials} -D "cn=admin,cn=config" -H "ldap://{HOST}:{PORT}/" -f path/to/file/break_repl.ldif ``` The output should be similar to the following: ``` modifying entry "olcDatabase={2}bdb,cn=config" modifying entry "olcDatabase={2}bdb,cn=config" ``` 3. You can verify that dc-2 is no longer replicating to dc-1 by creating an entry in dc-2 LDAP and ensuring it doesn't show up in LDAP of dc-1. Optionally, you can follow the steps below, which create a read-only user in the dc-2 OpenLDAP node and then check whether the user is replicated or not. The user is subsequently deleted. 1. Create a file `readonly-user.ldif` in dc-2 with following contents: ``` dn: uid=readonly-user,ou=users,ou=global,dc=apigee,dc=com objectClass: organizationalPerson objectClass: person objectClass: inetOrgPerson objectClass: top cn: readonly-user sn: readonly-user userPassword: {testPassword} ``` 2. Add user with \`ldapadd\` command in dc-2: ``` ldapadd -H ldap://{HOST}:{PORT} -w {credentials} -D "cn=manager,dc=apigee,dc=com" -f path/to/file/readonly-user.ldif ``` The output will be similar to: ``` adding new entry "uid=readonly-user,ou=users,ou=global,dc=apigee,dc=com" ``` 3. Search for the user in dc-1 to make sure the user is not replicated. If the user is not present in dc-1, you will be sure that both LDAPs are no longer replicating: ``` ldapsearch -H ldap://{HOST}:{PORT} -x -w {credentials} -D "cn=manager,dc=apigee,dc=com" -b uid=readonly-user,ou=users,ou=global,dc=apigee,dc=com -LLL ``` The output should be similar to the following: ``` No such object (32) Matched DN: ou=users,ou=global,dc=apigee,dc=com ``` 4. Remove the read-only user you added previously: ``` ldapdelete -v -H ldap://{HOST}:{PORT} -w {credentials} -D "cn=manager,dc=apigee,dc=com" "uid=readonly-user,ou=users,ou=global,dc=apigee,dc=com" ``` 4. Stop OpenLDAP in dc-1: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-openldap stop ``` 5. Uninstall the OpenLDAP component on dc-1: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-openldap uninstall ``` ## Router This section explains how to decommission a Router. See [Remove a server](https://docs.apigee.com/private-cloud/v4.52.00/removing-server-management-servermessage-processorrouter) for more details about removing the Router. > [!NOTE] > **Note:** Before decommissioning the Router, disable reachability of Routers by blocking the port 15999. Ensure no runtime traffic is being directed at the Routers being decommissioned. The following steps decommission the Router from dc-1. If there are multiple Router nodes configured in dc-1, perform the steps in all Router nodes one at a time **Note:** Here it is assumed that the router's health-check port 15999 is configured in your load balancer, and that blocking port 15999 will make router unreachable. You may need root access to block the port. To decommission a Router, do the following steps: 1. Disable the reachability of routers by blocking port 15999, the health check port. Ensure that runtime traffic is blocked on this data center: ``` iptables -A INPUT -i eth0 -p tcp --dport 15999 -j REJECT ``` 2. Verify that the router is reachable: ``` curl -vvv -X GET http://{ROUTER_IP}:15999/v1/servers/self/reachable ``` The output should be similar to the following: ``` About to connect() to 10.126.0.160 port 15999 (#0) Trying 10.126.0.160... Connection refused Failed connect to 10.126.0.160:15999; Connection refused Closing connection 0 curl: (7) Failed connect to 10.126.0.160:15999; Connection refused ``` 3. Get the UUID of the Router, as described in [Get UUIDs](https://docs.apigee.com/private-cloud/v4.52.00/get-uuids). 4. Stop the router: ``` /opt/apigee/apigee-service/bin/apigee-service edge-router stop ``` 5. List the available gateway pods in the organization by the following command: ``` curl -u : -X GET "http://{MS_IP}:8080/v1/organizations/{ORG}/pods" ``` See [About Pods](https://docs.apigee.com/private-cloud/v4.18.05/about-planets-regions-pods-organizations-environments-and-virtual-hosts#aboutpods). 6. Deregister the server's type: ``` curl -u :'' -X POST http://{MS_IP}:8080/v1/servers \ -d "type=router&region=dc-1&pod=gateway-1&uuid=UUID&action=remove" ``` 7. Delete the server: ``` curl -u :'’ -X DELETE http://{MS_IP}:8080/v1/servers/UUID ``` 8. Uninstall `edge-router`: ``` /opt/apigee/apigee-service/bin/apigee-service edge-router uninstall ``` See [Remove a server](https://docs.apigee.com/private-cloud/v4.52.00/removing-server-management-servermessage-processorrouter). 9. Flush `iptables` rules to enable the blocked port 15999: ``` iptables -F ``` ## Message Processor This section describes how to decommission the Message Processor from dc-1. See [Remove a server](https://docs.apigee.com/private-cloud/v4.52.00/removing-server-management-servermessage-processorrouter) for more details about removing the Message Processor. Since we are assuming that dc-1 has a [12-node clustered installation](https://docs.apigee.com/private-cloud/v4.52.00/installation-topologies#12hostclusteredinstallation), there are two Message Processor nodes configured in dc-1. Perform the following commands in both the nodes. 1. Get the UUIDs of the Message Processors, as described in [Get UUIDs](https://docs.apigee.com/private-cloud/v4.52.00/get-uuids). 2. Stop the Message Processor: ``` apigee-service edge-message-processor stop ``` 3. Deregister the server's type: ``` curl -u :'' -X POST http://{MS_IP}:8080/v1/servers \ -d "type=message-processor&region=dc-1&pod=gateway-1&uuid=UUID&action=remove"/pre> ``` 4. Disassociate an environment from the Message Processor. **Note:** You need to remove the bindings on each org/env that associates the Message Processor UUID. ``` curl -H "Content-Type:application/x-www-form-urlencoded" -u :'' \ -X POST http://{MS_IP}:8080/v1/organizations/{ORG}/environments/{ENV}/servers \ -d "action=remove&uuid=UUID" ``` 5. Deregister the server's type: ``` curl -u :'' -X POST http://MS_IP:8080/v1/servers -d "type=message-processor®ion=dc-1&pod=gateway-1&uuid=UUID&action=remove" ``` 6. Uninstall the Message Processor: ``` /opt/apigee/apigee-service/bin/apigee-service edge-message-processor uninstall ``` 7. Deregister the server: ``` curl -u :' -X DELETE http://{MS_IP}:8080/v1/servers/UUID ``` ## Qpid server and Qpidd This section explains how to decommission Qpid Server (`edge-qpid-server`) and Qpidd (`apigee-qpidd`). There are two Qpid nodes configured in dc-1, so you must do the following steps for both nodes: 1. Get the UUID for Qpidd, as described in [Get UUIDs](https://docs.apigee.com/private-cloud/v4.52.00/get-uuids). 2. Stop `edge-qpid-server` and `apigee-qpidd`: /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server stop /opt/apigee/apigee-service/bin/apigee-service apigee-qpidd stop 3. Get a list of Analytics and consumer groups: ``` curl -u :'' -X GET http://{MS_IP}:8080/v1/analytics/groups/ax ``` 4. Remove Qpid from the consumer group: ``` curl -u :'' -H "Content-Type: application/json" -X DELETE \ "http://{MS_IP}:8080/v1/analytics/groups/ax/{ax_group}/consumer-groups/{consumer_group}/consumers/{QPID_UUID}" ``` 5. Remove Qpid from the analytics group: ``` curl -v -u :'' \ -X DELETE "http://{MS_IP}:8080/v1/analytics/groups/ax/{ax_group}/servers?uuid={QPID_UUID}&type=qpid-server" ``` 6. Deregister the Qpid server from the Edge installation: ``` curl -u :'' -X POST http://{MS_IP}:8080/v1/servers \ -d "type=qpid-server®ion=dc-1&pod=central&uuid={QPID_UUID}&action=remove" ``` 7. Remove the Qpid server from the Edge installation: ``` curl -u :'' -X DELETE http://{MS_IP}:8080/v1/servers/UUID ``` 8. Restart all edge-qpid-server components on all nodes to make sure the change is picked up by those components: ``` $ /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restart $ /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server wait_for_ready ``` 9. Uninstall edge-qpid-server and apigee-qpidd: ``` $ /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server uninstall $ /opt/apigee/apigee-service/bin/apigee-service apigee-qpidd uninstall ``` ## Postgres and Postgresql The data center you are decommissioning could have a Postgres master or a Postgres standby. The following sections explain how to decommission them: - [Decommissioning Postgres master](https://docs.apigee.com/private-cloud/v4.52.00/decommission-data-center#decommissioning-postgres-master) - [Decommissioning Postgres standby](https://docs.apigee.com/private-cloud/v4.52.00/decommission-data-center#decommissioning-postgres-standby) > [!CAUTION] > **Caution:** You must remove and add Postgresql servers from the analytics > group and the consumer group at the same time. You might lose some analytics data > if there is a substantial delay between the two steps. > > Suggestion: The Message Processor has a small queue where it keeps > analytics data in memory > for a brief time, which you can usw to help avoid loss of data. Before you make changes to the analytics group, > block port 5672 in the data center that is taking traffic, > and make sure that you quickly re-enable this port after making the changes, > as there is a chance of data overflow. > > For example, to temporarily stop analytics traffic from > Message Processors to the Qpid server: > > 1. Enter > > ``` > iptables -A INPUT -p tcp --destination-port 5672 ! -s `hostname` -i eth0 -j DROP > ``` > 2. Follow the axgroup change steps. > 3. Remove the iptable rule after axgroup changes: > > ``` > iptables -F > ``` ### Decommissioning Postgres master **Note:** If you decommission Postgres master, make sure to promote any one of the available standby nodes as a new postgres master. While the QPID queeues buffer data, if the Postgres master is unavailable for a long time, you risk losing analytics data. To decommission Postgres master: 1. Back up the dc-1 Postgres master node by following the instructions in the following links: - [Backup and restore](https://docs.apigee.com/private-cloud/v4.52.00/backup-and-restore) - [How to back up](https://docs.apigee.com/private-cloud/v4.52.00/how-perform-backup) 2. Get the UUIDs of the Postgres servers, as described in [Get UUIDs](https://docs.apigee.com/private-cloud/v4.52.00/get-uuids). 3. On dc-1, stop `edge-postgres-server` and `apigee-postgresql` on the current master: /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql stop 4. On the standby node on dc-2, enter the following command to make it the master node: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql promote-standby-to-master ``` **Note:** If you have more than one standby Postgres node, you must add host entries on the new master and update the replication setting for all available postgres standby nodes. To add host entries to the new Postgres master: follow the steps in the appropriate section below: #### If there is only one standby node remaining For example, suppose that before decommissioning, there were three Postgres nodes configured. You decommissioned the existing master and promoted one of the remaining postgres standby nodes to master. Configure the remaining standby node with the following steps: 1. On the new master, edit the configuration file to set: ``` PG_MASTER=IP_or_DNS_of_new_PG_MASTER PG_STANDBY=IP_or_DNS_of_PG_STANDBY ``` 2. Enable replication on the new master: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup-replication-on-master -f configFIle ``` #### If there is more than one standby node remaining 1. Add the following configuration in `/opt/apigee/customer/application/postgresql.properties`: ``` conf_pg_hba_replication.connection=host replication apigee standby_1_ip/32 trust \n host replication apigee standby_2_ip/32 trust ``` 2. Ensure the file /opt/apigee/customer/application/postgresql.properties is owned by apigee user: ``` chown apigee:apigee /opt/apigee/customer/application/postgresql.properties ``` 3. Restart `apigee-postgresql`: ``` apigee-service apigee-postgresql restart ``` 4. To update replication settings on a standby node: 1. Modify the configuration file `/opt/silent.conf` and update the `PG_MASTER` field with the IP address of the new Postgres master. 2. Remove any old Postgres data with following command: ``` rm -rf /opt/apigee/data/apigee-postgresql/ ``` 3. Set up replication on the standby node: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql setup-replication-on-standby -f configFile ``` 5. Verify that Postgres master is set up correctly by entering the following command in dc-2: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql postgres-check-master ``` 6. Remove and add Postgresql servers from the analytics group and the consumer group. > [!CAUTION] > **Caution:** Perform the next two steps together, as there may be a loss of analytics data between the two steps. 1. Remove the old Postgres server from the analytics group following the instructions in [Remove a Postgres server from an analytics group](https://docs.apigee.com/api-platform/troubleshoot/analytics/add-remove-postgres-nodes#remove-a-postgres-server-from-an-an-analytics-group). 2. Add a new postgres server to the analytics group following the instructions in [Add an existing Postgres server to an analytics group](https://docs.apigee.com/api-platform/troubleshoot/analytics/add-remove-postgres-nodes#add-server). 7. Deregister the old postgres server from dc-1: ``` curl -u :'' -X POST http://{MS_IP}:8080/v1/servers \ -d "type=postgres-server®ion=dc-1&pod=analytics&uuid=UUID&action=remove"< ``` 8. Delete the old postgres server from dc-1: ``` curl -u >AdminEmailID>:'>AdminPassword>' -X DELETE http://{MS_IP}:8080/v1/servers/UUID ``` 9. The old Postgres master is now safe to decommission. Uninstall `edge-postgres-server` and `apigee-postgresql`: ``` /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server uninstall /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql uninstall ``` ### Decommissioning Postgres standby **Note:** The documentation for a [12-node clustered installation](https://docs.apigee.com/private-cloud/v4.52.00/installation-topologies#12hostclusteredinstallation12-node) shows the dc-1 postgresql node as master, but for convenience, in this section it is assumed that the dc-1 postgresql node is standby and the dc-2 postgresql node is master. To decommission Postgres standby, do the following steps: 1. Get the UUIDs of the Postgres servers, following the instructions in [Get UUIDs](https://docs.apigee.com/private-cloud/v4.52.00/get-uuids). 2. Stop `apigee-postgresql` on the current standby node in dc-1: /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server stop /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql stop 3. Remove and add Postgresql servers from the analytics group and the consumer group. > [!CAUTION] > **Caution:** Perform the next two steps together, as there may be a loss of analytics data between the two steps. 1. Remove the old Postgres server from the analytics group following the instructions in [Remove a Postgres server from an analytics group](https://docs.apigee.com/api-platform/troubleshoot/analytics/add-remove-postgres-nodes#remove-a-postgres-server-from-an-an-analytics-group). 2. Add a new postgres server to the analytics group following the instructions in [Add an existing Postgres server to an analytics group](https://docs.apigee.com/api-platform/troubleshoot/analytics/add-remove-postgres-nodes#add-server). 4. Deregister the old postgres server from dc-1: ``` curl -u :'' -X POST http://{MS_IP}:8080/v1/servers \ -d "type=postgres-server®ion=dc-1&pod=analytics&uuid=UUID&action=remove"< ``` 5. Delete the old postgres server from dc-1: ``` curl -u >AdminEmailID>:'>AdminPassword>' -X DELETE http://{MS_IP}:8080/v1/servers/UUID ``` 6. The old Postgres master is now safe to decommission. Uninstall `edge-postgres-server` and `apigee-postgresql`: ``` /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server uninstall /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql uninstall ``` ## ZooKeeper and Cassandra This section explains how to decommission ZooKeeper and Cassandra servers in a two data center setup. If you have more than two data centers, make sure to remove all references to the node being decommissioned (dc-1 in this case) from all silent configuration files across all the remaining data centers. For Cassandra nodes that are to be decommissioned, drop those hosts from `CASS_HOSTS`. The remaining Cassandra nodes should remain in the original ordering of `CASS_HOSTS`. **Note on ZooKeeper:** You must maintain a quorum of voter nodes while modifying the `ZK_HOST` property in the configuration file, to ensure that the ZooKeeper ensemble stays functional. You must have an odd number of voter nodes in your configuration. For more information, see [Apache ZooKeeper maintenance tasks](https://docs.apigee.com/private-cloud/v4.52.00/apache-zookeeper-maintenance-tasks). > [!CAUTION] > **Caution:** If you decommission a data center that has a leader ZooKeeper node, there will be downtime. To decommission ZooKeeper and Cassandra servers: 1. Back up the dc-1 Cassandra and ZooKeeper nodes by following the instructions in the following links: - [Backup and restore](https://docs.apigee.com/private-cloud/v4.52.00/backup-and-restore) - [How to back up](https://docs.apigee.com/private-cloud/v4.52.00/how-perform-backup) > [!NOTE] > **Note:** If you have multiple ZooKeeper nodes, back them up one at a time. The backup process temporarily shuts down ZooKeeper. 2. List the [UUIDs](https://docs.apigee.com/private-cloud/v4.52.00/get-uuids) of ZooKeeper and Cassandra servers in the data center where Cassandra nodes are about to be decommissioned. ``` apigee-adminapi.sh servers list -r dc-1 -p central -t application-datastore --admin --pwd '' --host localhost ``` 3. Deregister the server's type: ``` curl -u :'' -X POST http://MS_IP:8080/v1/servers -d "type=cache-datastore&type=user-settings-datastore&type=scheduler-datastore&type=audit-datastore&type=apimodel-datastore&type=application-datastore&type=edgenotification-datastore&type=identityzone-datastore&type=user-settings-datastore&type=auth-datastore®ion=dc-1&pod=central&uuid=UUID&action=remove" ``` 4. Deregister the server: ``` curl -u :'' -X DELETE http://MS_IP:8080/v1/servers/UUID ``` 5. Update the configuration file with the IPs of decommissioned nodes removed from `ZK_HOSTS` and `CASS_HOSTS`. > [!NOTE] > **Notes:** > - You must maintain a quorum of voter nodes while modifying the `ZK_HOST` property in the configuration file, to ensure the ZooKeeper ensemble stays functional. You must have an odd number of voter nodes in your configuration. See [About ZooKeeper > and edge](https://docs.apigee.com/private-cloud/v4.52.00/zookeeper-maintenance#aboutzookeeperandedge). > - For Cassandra nodes that are to be decommissioned, drop those hosts from `CASS_HOSTS`. The remaining Cassandra nodes should remain in the original ordering of `CASS_HOSTS`. > - ZooKeeper observer nodes do not participate in the voting and are not counted when determining a quorum. Example: Suppose you have the IPs `$IP1 $IP2 $IP3` in dc-1 and `$IP4 $IP5 $IP6` in dc-2, and you are decommissioning dc-1. Then you should remove the IPs `$IP1 $IP2 $IP3` from the configuration files. - Existing configuration file entries: ``` ZK_HOSTS="$IP1 $IP2 $IP3 $IP4 $IP5 $IP6" CASS_HOSTS="$IP1:1,1 $IP2:1,1 $IP3:1,1, $IP4:2,1 $IP5:2,1 $IP6:2,1” ``` - New configuration file entries: ``` ZK_HOSTS="$IP4 $IP5 $IP6" CASS_HOSTS="$IP4:2,1 $IP5:2,1 $IP6:2,1" ``` 6. Update the silent configuration file (modified in step e) with the IPs of the removed decommissioned nodes and run the Management server profile on all nodes hosting Management Servers: ``` /opt/apigee/apigee-setup/bin/setup.sh -p ms -f updated_config_file ``` 7. Update the configuration file with IPs of the removed decommissioned nodes and run the MP/RMP profile on all Router and Message Processor nodes: - If Edge Router and Message Processor are configured on the same node, enter: ``` /opt/apigee/apigee-setup/bin/setup.sh -p rmp -f updated_config_file ``` - If Edge Router and Message Processor are configured on separate nodes, enter the following: For the Router: ``` /opt/apigee/apigee-setup/bin/setup.sh -p r -f updated_config_file ``` For the Message Processor: ``` /opt/apigee/apigee-setup/bin/setup.sh -p mp -f updated_config_file ``` 8. Reconfigure all Qpid nodes, with the IPs of decommissioned nodes removed from Response File: ``` /opt/apigee/apigee-setup/bin/setup.sh -p qs -f updated_config_file ``` 9. Reconfigure all Postgres nodes, with the IPs of decommissioned nodes removed from Response File: ``` /opt/apigee/apigee-setup/bin/setup.sh -p ps -f updated_config_file ``` 10. Alter `system_auth` keyspace. If you have Cassandra auth enabled on an existing Cassandra node, update the replication factor of the `system_auth` keyspace by running the following command: ``` ALTER KEYSPACE system_auth WITH replication = {'class': 'NetworkTopologyStrategy', 'dc-2': '3'}; ``` This command sets the replication factor to `'3'`, indicating three Cassandra nodes in the cluster. Modify this value as necessary. After completing this step, the Cassandra topology should not have `dc-1` in any of the keyspaces. 11. Decommission the Cassandra nodes on dc-1, one by one. > [!NOTE] > **Note:** The decommissioned nodes still have Cassandra > with seed information of the > original cluster on them. If by chance the Cassandra service should come up, the decommissioned > nodes will rejoin the cluster ring. To avoid this, > make sure you uninstall the Cassandra component after decommissioning and do not allow > any decommissioned nodes to come up and rejoin the cluster. To decommission the Cassandra nodes, enter the following command: ``` /opt/apigee/apigee-cassandra/bin/nodetool -h cassIP -u cassandra -pw '' decommission ``` **Note:** If a node becomes stuck in a decommissioning state, you can use [Cassandra `nodetool removenode`](https://docs.datastax.com/en/cassandra-oss/2.1/cassandra/tools/toolsRemoveNode.html) to remove it. For example: ``` /opt/apigee/apigee-cassandra/bin/nodetool removenode UUID ``` 12. Check the connection of the Cassandra nodes from dc-1 using one of the following commands: ``` /opt/apigee/apigee-cassandra/bin/cqlsh cassIP 9042 -u cassandra -p '' ``` Or secondary verification command to be run on the decommissioned node: ``` /opt/apigee/apigee-cassandra/bin/nodetool netstats ``` The above command should return: ``` Mode: DECOMMISSIONED ``` 13. Run the DS profile for all Cassandra and ZooKeeper nodes in dc-2: ``` /opt/apigee/apigee-setup/bin/setup.sh -p ds -f updated_config_file ``` 14. Stop `apigee-cassandra` and `apigee-zookeeper` in dc-1: apigee-service apigee-cassandra stop apigee-service apigee-zookeeper stop 15. Uninstall `apigee-cassandra` and `apigee-zookeeper` in dc-1: apigee-service apigee-cassandra uninstall apigee-service apigee-zookeeper uninstall ## Delete the bindings from dc-1 To delete the bindings from dc-1, do the following steps: 1. Delete the bindings from dc-1. 1. List all the available pods under organization: ``` curl -v -u : -X GET "http://MS_IP:8080/v1/o/ORG/pods" ``` 2. To check whether all the bindings have been removed, get the [UUIDs](https://docs.apigee.com/private-cloud/v4.52.00/get-uuids) of the servers associated with the pods: ``` curl -v -u : \ -X GET "http://MS_IP:8080/v1/regions/dc-1/pods/gateway-1/servers" ``` If this command doesn't return any UUIDs, the previous steps have removed all the bindings, and you can skip the next step. Otherwise, perform the next step. 3. Remove all the server bindings for the UUIDs obtained in the previous step: ``` curl -u :'' -X DELETE http://MS_IP:8080/v1/servers/UUID ``` 4. Disassociate Org from the pod: ``` curl -v -u : "http://MS_IP:8080/v1/o/ORG/pods" -d "action=remove®ion=dc-1&pod=gateway-1" -H "Content-Type: application/x-www-form-urlencoded" -X POST ``` 2. Delete the pods: ``` curl -v -u : "http://MS_IP:8080/v1/regions/dc-1/pods/gateway-1" -X DELETE ``` 3. Delete the region. ``` curl -v -u : "http://MS_IP:8080/v1/regions/dc-1" -X DELETE ``` **Note:** If you miss one of the steps deleting the servers, the above step will return an error message that a particular server in the pod still exists. So, delete them by following the troubleshooting steps below, while customizing the types in the `curl` command. At this point you have completed the decommissioning of dc-1. ## Appendix ### Troubleshooting If after performing the previous steps, there are still servers in some pods, do following steps to deregister and delete the servers. **Note:** Change the types and pod as necessary. 1. Get the [UUIDs](https://docs.apigee.com/private-cloud/v4.52.00/get-uuids) using the following command: ``` apigee-adminapi.sh servers list -r dc-1 -p POD -t --admin --pwd '’ --host localhost ``` 2. Deregister server's type: ``` curl -u :'' -X POST http://MP_IP:8080/v1/servers -d "type=TYPE=REGION=dc-1&pod=POD&uuid=UUID&action=remove" ``` 3. Delete the servers one by one: ``` curl -u :'' -X DELETE http://MP_IP:8080/v1/servers/UUID ``` ### Validation You can validate the decommissioning using the following commands. #### Management Server Run the following commands from the Management Servers on all the regions. ``` curl -v -u :'' http://MS_IP:8080/v1/servers?pod=central®ion=dc-1 curl -v -u :'' http://MS_IP:8080/v1/servers?pod=gateway®ion=dc-1 curl -v -u :'' http://MS_IP:8080/v1/servers?pod=analytics®ion=dc-1 ``` Run the following command on all components to check port requirements for all the management ports. ``` curl -v http://MS_IP:8080/v1/servers/self ``` Check the analytics group. ``` curl -v -u :'' "http://MS_IP:8080}/v1/o/ORG/e/ENV/provisioning/axstatus" curl -v -u :'' http://MS_IP:8080/v1/analytics/groups/ax ``` #### Cassandra/ZooKeeper nodes On all Cassandra nodes enter: ``` /opt/apigee/apigee-cassandra/bin/nodetool -h statusthrift ``` This will return a `running` or `not running` status for that particular node. On one node enter: ``` /opt/apigee/apigee-cassandra/bin/nodetool -h ring ``` ``` /opt/apigee/apigee-cassandra/bin/nodetool -h status ``` The above commands will return active data center information. On ZooKeeper nodes, first enter: ``` echo ruok | nc 2181 ``` This command will return `imok`. Then enter: ``` echo stat | nc 2181 | grep Mode ``` The value of `Mode` returned by the above command will be one of the following: `observer`, `leader`, or `follower`. In one ZooKeeper node: ``` /opt/apigee/apigee-zookeeper/contrib/zk-tree.sh >> /tmp/zk-tree.out.txt ``` On the Postgres master node, run: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql postgres-check-master ``` Validate that the response says the node is the master. On the standby node: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql postgres-check-standby ``` Validate that the response says the node is the standby. Login to PostgreSQL database using the command ``` psql -h localhost -d apigee -U postgres ``` When prompted, enter the 'postgres' user password as `'postgres'`. Select `max(client_received_start_timestamp)` from analytics. `"$org.$env.fact" limit 1`; #### Logs Check the logs on the components to make sure there are no errors.