This document describes the monitoring techniques of components supported by an on-premise deployment of Apigee Edge for Private Cloud. ## Overview Edge supports several ways for getting details about services as well as checking their statuses. The following table lists the types of checks you can perform on each eligible service: | | | **Mgmt API** ||| | | | | Component | Memory Usage \[JMX^\*^\] | **Service Check** | **User/Org/ Deployment Status** | **axstatus** | Database check | `apigee-service` Status | `apigee-monit`\*\* | |---|---|---|---|---|---|---|---| | Management Server | Yes | Yes | Yes | No | No | Yes | Yes | | Message Processor | Yes | Yes | No | No | No | Yes | Yes | | Router | Yes | Yes | No | No | No | Yes | Yes | | Qpid | Yes | Yes | No | Yes | No | Yes | Yes | | Postgres | Yes | Yes | Yes | Yes | Yes | Yes | Yes | | | [More Info](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#usejmx) | [More Info](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#perform-service-checks) | [More Info](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#user-org-deployment) | [More Info](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#postgres-axstatus) | [More Info](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#postgressqldatabase) | [More Info](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#apigee-service) | [More Info](https://docs.apigee.com/private-cloud/v4.52.01/apigee-monit) | | ^\* Before you can use JMX, you must enable it, as described in [Enable JMX](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#enable-jmx).^ ^\*\* The `apigee-monit` service checks if a component is up and will attempt to restart it if it isn't. For more information, see [Self healing with apigee-monit](https://docs.apigee.com/private-cloud/v4.52.01/apigee-monit).^ |||||||| ### Monitoring ports and configuration files Each component supports Java Management Extensions (JMX) and Management API monitoring calls on different ports. The following table lists the JMX and Management API ports for each type of server, and configuration file locations: > [!NOTE] > **Warning:**Apigee recommends that JMX ports be open for internal servers only. JMX ports should be closed from external access. | Component | JMX Port | Management API Port | Configuration file location | |---|---|---|---| | Management Server | 1099 | 8080 | `$APIGEE_ROOT/customer/application/management-server.properties` | | Message Processor | 1101 | 8082 | `$APIGEE_ROOT/customer/application/message-processor.properties` | | Router | 1100 | 8081 | `$APIGEE_ROOT/customer/application/router.properties` | | Qpid | 1102 | 8083 | `$APIGEE_ROOT/customer/application/qpid-server.properties` | | Postgres | 1103 | 8084 | `$APIGEE_ROOT/customer/application/postgres-server.properties` | ## Use JMX to monitor components The following sections explain how to use JMX to monitor Edge components. ### Enable JMX To enable JMX without authentication or SSL based communication, perform the steps below. **Note:** In production systems, both encrypted authentication and SSL should be enabled for security. 1. Edit the appropriate configuration file (see [Configuration file reference](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#configuration-file-reference)). Create the configuration file if it doesn't exist. ``` conf_system_jmxremote_enable=true ``` 2. Save the configuration file and make sure it is owned by `apigee:apigee`. 3. Restart appropriate Edge component ``` apigee-service edge-management-server restart ``` To disable JMX, either remove the property `conf_system_jmxremote_enable` or change its value to `false`. Then restart the appropriate Edge component. ### Authentication in JMX Edge for Private Cloud supports password based authentication using details stored in files. You can store passwords as a Hash for added security. 1. To enable JMX authentication in an edge-\* component, edit the appropriate configuration file (see [Configuration file reference](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#configuration-file-reference)). Create the configuration file if it doesn't exist: ``` conf_system_jmxremote_enable=true conf_system_jmxremote_authenticate=true conf_system_jmxremote_encrypted_auth=true conf_system_jmxremote_access_file=/opt/apigee/customer/application/management-server/jmxremote.access conf_system_jmxremote_password_file=/opt/apigee/customer/application/management-server/jmxremote.password ``` Save the configuration file and make sure it is owned by `apigee:apigee`. 2. Create a SHA256 hash of the password: ``` echo -n '' | openssl dgst -sha256 ``` 3. Create a `jmxremote.password` file with JMX user credentials: 1. Copy the following files from your `$JAVA_HOME` directory to the directory `/opt/apigee/customer/application//`: ``` cp ${JAVA_HOME}/lib/management/jmxremote.password.template $APIGEE_ROOT/customer/application/management-server/jmxremote.password ``` 2. Edit the file and add your JMX username and password using the following syntax: ``` USERNAME ``` 3. Make sure the file is owned by `apigee` and that the file mode is 400: ``` chown apigee:apigee $APIGEE_ROOT/customer/application/management-server/jmxremote.password chmod 400 $APIGEE_ROOT/customer/application/management-server/jmxremote.password ``` 4. Create a `jmxremote.access` file with JMX user permissions: 1. Copy the following files from your $JAVA_HOME directory to the directory `/opt/apigee/customer/application//` ``` cp ${JAVA_HOME}/lib/management/jmxremote.access$APIGEE_ROOT/customer/application/management-server/jmxremote.password/jmxremote.access ``` 2. Edit the file and add your JMX username followed by a permission (READONLY/READWRITE) ``` USERNAME READONLY ``` 3. Make sure the file is owned by `apigee` and that the file mode is 400: ``` chown apigee:apigee $APIGEE_ROOT/customer/application/management-server/jmxremote.password chmod 400 $APIGEE_ROOT/customer/application/management-server/jmxremote.access ``` 5. Restart the appropriate Edge component: ``` apigee-service edge-management-server restart ``` To disable JMX authentication, either remove the property `conf_system_jmxremote_authenticate` or change the value to `false` and restart appropriate Edge component. ### SSL in JMX To enable SSL based JMX in an edge-\* component: 1. Edit the appropriate configuration file (see [Configuration file reference](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#configuration-file-reference)). Create the configuration file if it doesn't exist: ``` conf_system_jmxremote_enable=true conf_system_jmxremote_ssl=true conf_system_javax_net_ssl_keystore=/opt/apigee/customer/application/management-server/jmxremote.keystore conf_system_javax_net_ssl_keystorepassword= ``` Save the configuration file and make sure it is owned by `apigee:apigee`. 2. Prepare a keystore containing the server key and place it at the path provided in the configuration `conf_system_javax_net_ssl_keystore` above. Ensure the keystore file is readable by `apigee:apigee`. 3. Restart the appropriate Edge component: ``` apigee-service edge-management-server restart ``` To disable SSL-based JMX, either remove the property `conf_system_jmxremote_ssl` or change the value to `false`. Restart the appropriate Edge component. ### Monitoring via Jconsole Monitoring instructions via jconsole remain the same as described in https://docs.apigee.com/private-cloud/v4.52.01/how-monitor#jconsole. One line can be added that "jconsole will need to be started with truststore and truststore password if SSL is enabled for JMX." Reference: https://docs.oracle.com/javase/8/docs/technotes/guides/management/jconsole.html ### Monitor with JConsole Use JConsole (a JMX compliant tool) to manage and monitor health check and process statistics. With JConsole, you can consume JMX statistics exposed by your servers and display them in a graphical interface. For more information, see [Using JConsole](https://docs.oracle.com/javase/8/docs/technotes/guides/management/jconsole.html). You need to start JConsole with truststore and truststore password if SSL is enabled for JMX. See [Using JConsole](https://docs.oracle.com/javase/8/docs/technotes/guides/management/jconsole.html). JConsole uses the following service URL to monitor the JMX attributes (MBeans) offered via JMX: ``` service:jmx:rmi:///jndi/rmi://IP_address:port_number/jmxrmi ``` Where: - IP_address is the IP address of the server you want to monitor. - port_number is the [JMX port number](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#ports) of the server you want to monitor. For example, to monitor the Management Server, issue a command like the following (assuming the server's IP address is 216.3.128.12): ``` service:jmx:rmi:///jndi/rmi://216.3.128.12:1099/jmxrmi ``` Note that this example specifies port 1099, which is the Management Server JMX port. For other ports, see [JMX and Management API monitoring ports](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#ports). The following table shows the generic JMX statistics: | JMX MBeans | JMX Attributes | |---|---| | Memory | HeapMemoryUsage | | Memory | NonHeapMemoryUsage | | Memory | Usage | | > [!NOTE] > ### NOTE > Attribute values are displayed in four values: committed, init, max, and used. || ### Configuration file reference The following sections describe changes you might need to make to Edge component configuration files for JMX related configurations. See [Monitoring ports and configuration files](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#ports) for more information. #### JMX configuration to be added to the appropriate component's configuration file - Enable JMX agent on the edge-component. False by default. ``` conf_system_jmxremote_enable=true ``` #### Configurations for password based authentication - Enable password based authentication. False by default. ``` conf_system_jmxremote_authenticate=true ``` - Path to access file. Should be owned and readable by Apigee user only. ``` conf_system_jmxremote_access_file=/opt/apigee/customer/application/management-server/jmxremote.access ``` - Path to password file. Should be owned and readable by Apigee user only. ``` conf_system_jmxremote_password_file=/opt/apigee/customer/application/management-server/jmxremote.password ``` - Enable storing password in encrypted format. False by default. ``` conf_system_jmxremote_encrypted_auth=true ``` #### Configurations for SSL based JMX - Enable SSL for JMX communication. False by default. ``` conf_system_jmxremote_ssl=true ``` - Path to keystore. Should be owned and readable by Apigee user only. ``` conf_system_javax_net_ssl_keystore=/opt/apigee/customer/application/management-server/jmxremote.keystore ``` - Keystore password: ``` conf_system_javax_net_ssl_keystorepassword=changeme ``` #### Optional JMX configurations Values listed are default values and can be changed. - JMX Port. Default values are listed in the table below. ``` conf_system_jmxremote_port= ``` - JMX RMI Port. By default, Java process will pick a random port. ``` conf_system_jmxremote_rmi_port= ``` - The host name for remote stubs. Default IP address of localhost. ``` conf_system_java_rmi_server_hostname= ``` - Protect JMX registry with SSL. Default false. Only applicable if SSL is enabled. ``` conf_system_jmxremote_registry_ssl=false ``` ## Monitor with the Management API Edge includes several APIs that you can use to perform service checks on your servers as well as check your users, organizations, and deployments. This section describes these APIs. ### Perform service checks The Management API provides several endpoints for monitoring and diagnosing issues with your services. These endpoints include: | Endpoint | Description | |---|---| | `/servers/self/up` | Checks to see if a service is running. This API call does not require you to authenticate. If the service is running, this endpoint returns the following response: ``` true ``` If the service is not running, you will get a response similar to the following (depending on which service it is and how you checked it): ``` curl: Failed connect to localhost:port_number; Connection refused ``` | | `/servers/self` | Returns information about the service, including: - Configuration properties - Start time and up time - Build, RPM, and UUID information - Internal and external hostname and IP address - Region and pod - `` property, indicating whether the service is running This API call requires you to authenticate with your Apigee admin credentials. | To use these endpoints, invoke a utility such as `curl` with commands that use the following syntax: curl http://host:port_number/v1/servers/self/up -H "Accept: [application/json|application/xml]" curl http://host:port_number/v1/servers/self -u username:password -H "Accept: [application/json|application/xml]" Where: - host is the IP address of the server you want to check. If you are logged into the server, you can use "localhost"; otherwise, specify the IP address of the server as well as the username and password. - port_number is the Management API port for the server you want to check. This is a different port for each type of component. For example, the Management Server's Management API port is 8080. For a list of Management API port numbers to use, see [JMX and Management API monitoring ports](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#ports) To change the format of the response, you can specify the `Accept` header as "application/json" or "application/xml". The following example gets the status of the Router on localhost (port 8081): ``` curl http://localhost:8081/v1/servers/self/up -H "Accept: application/xml" ``` The following example gets information about the Message Processor at 216.3.128.12 (port 8082): ``` curl http://216.3.128.12:8082/v1/servers/self -u sysAdminEmail:password -H "Accept: application/xml" ``` ### Monitor user, organization, and deployment status You can use the Management API to monitor user, organization, and deployment status of your proxies on Management Servers and Message Processors by issuing the following commands: curl http://host:port_number/v1/users -u sysAdminEmail:password curl http://host:port_number/v1/organizations -u sysAdminEmail:password curl http://host:port_number/v1/organizations/orgname/deployments -u sysAdminEmail:password Where port_number is either 8080 for the Management Server or 8082 for the Message Processor. This call requires you to authenticate with your system administration username and password. The server should return a "deployed" status for all calls. If these fail, do the following: 1. Check the server logs for any errors. The logs are located at: - Management Server: `opt/apigee/var/log/edge-management-server` - Message Processor: `opt/apigee/var/log/edge-message-processor` 2. Make a call against the server to check whether it is functioning properly. 3. Remove the server from the ELB and then restart it: ``` /opt/apigee/apigee-service/bin/apigee-service service_name restart ``` Where service_name is: - `edge-management-server` - `edge-message-processor` ## Check status with the `apigee-service` command You can troubleshoot your Edge services by using the `apigee-service` command when you are logged into the server running the service. **To check the status of a service with `apigee-service`:** 1. Log in to the server and run the following command: ``` /opt/apigee/apigee-service/bin/apigee-service service_name status ``` Where service_name is one of the following: - Management Server: `edge-management-server` - Message Processor: `edge-message-processor` - Postgres: `edge-postgres-server` - Qpid: `edge-qpid-server` - Router: `edge-router` For example: ``` /opt/apigee/apigee-service/bin/apigee-service edge-message-processor status ``` 2. If the service is not running, start the service: ``` /opt/apigee/apigee-service/bin/apigee-service service_name start ``` 3. After restarting the service, check that it is functioning, either by using the `apigee-service status` command you used previously or by using the Management API described in [Monitor with the Management API](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#managementapi). For example: ``` curl -v http://localhost:port_number/v1/servers/self/up ``` Where port_number is the [Management API port](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#ports) for the service. This example assumes you are logged into the server and can use "localhost" as the hostname. To check the status remotely with the Management API, you must specify the IP address of the server and include the system administrator username and password in your API call. > [!NOTE] > > ### NOTE > > Ensure that you specify the correct port when checking a service. Each service uses a different port for the Management API monitoring, as described in [JMX and Management API monitoring ports](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#ports). ## Postgres monitoring Postgres supports several utilities that you can use to check its status. These utilities are described in the sections that follow. ### Check organizations and environments on Postgres You can check for organization and environment names that are onboarded on the Postgres Server by issuing the following `curl` command: ``` curl -v http://postgres_IP:8084/v1/servers/self/organizations ``` The system should display the organization and environment name. ### Verify analytics status You can verify the status of the Postgres and Qpid analytics servers by issuing the following `curl` command: ``` curl -u userEmail:password http://host:port_number/v1/organizations/orgname/environments/envname/provisioning/axstatus ``` The system should display a success status for all analytics servers, as the following example shows: ``` { "environments" : [ { "components" : [ { "message" : "success at Thu Feb 28 10:27:38 CET 2013", "name" : "pg", "status" : "SUCCESS", "uuid" : "[c678d16c-7990-4a5a-ae19-a99f925fcb93]" }, { "message" : "success at Thu Feb 28 10:29:03 CET 2013", "name" : "qs", "status" : "SUCCESS", "uuid" : "[ee9f0db7-a9d3-4d21-96c5-1a15b0bf0adf]" } ], "message" : "", "name" : "prod" } ], "organization" : "acme", "status" : "SUCCESS" } ``` ### PostgreSQL database This section describes techniques that you can use specifically for monitoring the Postgres database. #### Use the `check_postgres.pl` script To monitor the PostgreSQL database, you can use a standard monitoring script, `check_postgres.pl`. For more information, see [http://bucardo.org/wiki/Check_postgres](https://bucardo.org/wiki/Check_postgres). **Before you run the script:** 1. You must install the check_postgres.pl script on each Postgres node. 2. Ensure that you have installed `perl-Time-HiRes.x86_64`, a Perl module that implements high resolution alarm, sleep, gettimeofday, and interval timers. For example, you can install it by using the following command: ``` yum install perl-Time-HiRes.x86_64 ``` 3. CentOS 7: Before using check_postgres.pl on CentOS v7, install the `perl-Data-Dumper.x86_64` RPM. **check_postgres.pl output** The default output of the API calls using `check_postgres.pl` is Nagios compatible. After you install the script, do the following checks: 1. Check the database size: ``` check_postgres.pl -H 10.176.218.202 -db apigee -u apigee -dbpass postgres -include=apigee -action database_size --warning='800 GB' --critical='900 GB' ``` 2. Check the number of incoming connections to the database and compares with maximum allowed connections: ``` check_postgres.pl -H 10.176.218.202 -db apigee -u apigee -dbpass postgres -action backends ``` 3. Check if database is running and available: ``` check_postgres.pl -H 10.176.218.202 -db apigee -u apigee -dbpass postgres -action connection ``` 4. Check the disk space: ``` check_postgres.pl -H 10.176.218.202 -db apigee -u apigee -dbpass postgres -action disk_space --warning='80%' --critical='90%' ``` 5. Check the number of organization and environment onboarded in a Postgres node: ``` check_postgres.pl -H 10.176.218.202 -db apigee -u apigee -dbpass postgres -action=custom_query --query="select count(*) as result from pg_tables where schemaname='analytics' and tablename like '%fact'" --warning='80' --critical='90' --valtype=integer ``` > [!NOTE] > > ### NOTE > > Please refer to [http://bucardo.org/check_postgres/check_postgres.pl.html](https://bucardo.org/check_postgres/check_postgres.pl.html) in case you need any help on using the above commands. #### Run database checks You can verify that the proper tables are created in PostgreSQL database. Log in to PostgreSQL database using the following command: ``` psql -h /opt/apigee/var/run/apigee-postgresql/ -U apigee -d apigee ``` Then run: ``` \d analytics."org.env.fact" ``` #### Check health status of postgres process You can perform API checks on the Postgres machine by invoking the following `curl` command: ``` curl -v http://postgres_IP:8084/v1/servers/self/health ``` > [!NOTE] > > ### NOTE > > Ensure that you use port 8084. This command returns the `ACTIVE` status when postgres process is active. If the Postgres process is not up and running, it returns the `INACTIVE` status. #### Postgres resources For additional information about monitoring the Postgres service, see the following: - [http://www.postgresql.org/docs/9.0/static/monitoring.html](https://www.postgresql.org/docs/9.0/static/monitoring.html) - [http://www.postgresql.org/docs/9.0/static/diskusage.html](https://www.postgresql.org/docs/9.0/static/diskusage.html) - [http://bucardo.org/check_postgres/check_postgres.pl.html](https://bucardo.org/check_postgres/check_postgres.pl.html) ## Cassandra monitoring JMX is enabled by default for Cassandra and remote JMX access to Cassandra does not require a password. ### Enable JMX authentication for Cassandra You can enable JMX authentication for Cassandra. After doing so, you will then be required to pass a username and password to all calls to the nodetool utility. **To enable JMX authentication for Cassandra:** 1. Create and edit the `cassandra.properties` file: 1. Edit the `/opt/apigee/customer/application/cassandra.properties` file. If the file does not exist, create it. 2. Add the following to the file: ``` conf_cassandra_env_com.sun.management.jmxremote.authenticate=true conf_cassandra_env_com.sun.management.jmxremote.password.file=${APIGEE_ROOT}/customer/application/apigee-cassandra/jmxremote.password conf_cassandra_env_com.sun.management.jmxremote.access.file=${APIGEE_ROOT}/customer/application/apigee-cassandra/jmxremote.access ``` 3. Save the `cassandra.properties` file. 4. Change the owner of the file to `apigee:apigee`, as the following example shows: ``` chown apigee:apigee /opt/apigee/customer/application/cassandra.properties ``` For more information on using properties files to set tokens, see [How to configure Edge](https://docs.apigee.com/private-cloud/v4.52.02/how-configure-edge). 2. Create and edit `jmx_auth.sh`: 1. Create a file at the following location if it does not exist: ``` /opt/apigee/customer/application/jmx_auth.sh ``` 2. Add the following properties to the file: ``` export CASS_JMX_USERNAME=JMX_USERNAME export CASS_JMX_PASSWORD=JMX_PASSWORD ``` 3. Save the `jmx_auth.sh` file. 4. Source the file: ``` source /opt/apigee/customer/application/jmx_auth.sh ``` 3. Copy and edit the `jmxremote.password` file: 1. Copy the following file from your `$JAVA_HOME` directory to `/opt/apigee/customer/application/apigee-cassandra/`: ``` cp ${JAVA_HOME}/lib/management/jmxremote.password.template $APIGEE_ROOT/customer/application/apigee-cassandra/jmxremote.password ``` 2. Edit the `jmxremote.password` file and add your JMX username and password using the following syntax: ``` JMX_USERNAME JMX_PASSWORD ``` Where JMX_USERNAME and JMX_PASSWORD are the JMX username and password you set previously. 3. Make sure the file is owned by "apigee" and that the file mode is 400: chown apigee:apigee /opt/apigee/customer/application/apigee-cassandra/jmxremote.password chmod 400 /opt/apigee/customer/application/apigee-cassandra/jmxremote.password 4. Copy and edit the `jmxremote.access` file: 1. Copy the following file from your `$JAVA_HOME` directory to `/opt/apigee/customer/application/apigee-cassandra/`: cp ${JAVA_HOME}/lib/management/jmxremote.access $APIGEE_ROOT/customer/application/apigee-cassandra/jmxremote.access 2. Edit the `jmxremote.access` file and add the following role: ``` JMX_USERNAME readwrite ``` 3. Make sure the file is owned by "apigee" and that the file mode is 400: chown apigee:apigee /opt/apigee/customer/application/apigee-cassandra/jmxremote.access chmod 400 /opt/apigee/customer/application/apigee-cassandra/jmxremote.access 5. Run `configure` on Cassandra: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra configure ``` 6. Restart Cassandra: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restart ``` 7. Repeat this process on all other Cassandra nodes. ### Enable JMX password encryption To enable JMX password encryption, do the following steps: 1. Open the file `source/conf/casssandra-env.sh`. 2. Create and edit the `cassandra.properties` file: 1. Edit the `/opt/apigee/customer/application/cassandra.properties` file. If the file does not exist, create it. 2. Add the following to the file: ``` conf_cassandra_env_com.sun.management.jmxremote.encrypted.authenticate=true ``` 3. Save the cassandra.properties file. 4. Change the owner of the file to apigee:apigee, as the following example shows: ``` chown apigee:apigee /opt/apigee/customer/application/cassandra.properties ``` 3. On the command line, generate SHA1 hash(es) of the desired password(s) by entering `echo -n 'Secret' | openssl dgst -sha1` 4. Set the password(s) against the username in `$APIGEE_ROOT/customer/application/apigee-cassandra/jmxremote.password` (created in in the previous section). 5. Run configure on Cassandra: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra configure ``` 6. Restart Cassandra: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restart ``` 7. Repeat this process on all other Cassandra nodes. ### Enable JMX with SSL for Cassandra Enabling JMX with SSL provides additional security and encryption for JMX-based communication with Cassandra. To enable JMX with SSL, you need to provide a key and a certificate to Cassandra to accept SSL-based JMX connections. You also need to configure nodetool (and any other tools that communicate with Cassandra over JMX) for SSL. SSL-enabled JMX supports both plaintext and encrypted JMX passwords. To enable JMX with SSL for Cassandra, use the following procedure: 1. [Enable JMX](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#enablingjmx). Enable password encryption if required. 2. [Enable JMX authentication for Cassandra](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#cassauthenable). as described above. Ensure nodetool is working with the configured username and password. ``` /opt/apigee/apigee-cassandra/bin/nodetool -u -pw ring ``` 3. Prepare keystore and truststore. - Keystore should contain a key and certificate, and is used to configure the Cassandra server. If keystore contains multiple key pairs, Cassandra uses the first key pair to enable SSL. Note that the passwords for keystore and the key should be the same (the default when you generate the key using keytool). - Truststore should contain the certificate only and is used by clients (apigee-service based commands or nodetool) to connect over JMX. After verifying the above requirements: 1. Place the keystore file in `/opt/apigee/customer/application/apigee-cassandra/`. 2. Ensure the keystore file is readable by Apigee user only by entering chown apigee:apigee /opt/apigee/customer/application/apigee-cassandra/keystore.node1 chmod 400 /opt/apigee/customer/application/apigee-cassandra/keystore.node1 4. Configure Cassandra for JMX with SSL by doing the following steps: 1. Stop the Cassandra node by entering ``` apigee-service apigee-cassandra stop ``` 2. Enable SSL in Cassandra by opening the file `/opt/apigee/customer/application/cassandra.properties` and adding the following lines: ``` conf_cassandra_env_com.sun.management.jmxremote.ssl=true conf_cassandra_env_javax.net.ssl.keyStore=/opt/apigee/customer/application/apigee-cassandra/keystore.node1 conf_cassandra_env_javax.net.ssl.keyStorePassword=keystore-password ``` 3. Change the owner of the file to apigee:apigee, as the following example shows: ``` chown apigee:apigee /opt/apigee/customer/application/cassandra.properties ``` 4. Run configure on Cassandra: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra configure ``` 5. Restart Cassandra: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restart ``` 6. Repeat this process on all other Cassandra nodes. 7. Start the Cassandra node by entering ``` apigee-service apigee-cassandra start ``` 5. Configure the `apigee-service` Cassandra commands. You need to set certain environment variables while running `apigee-service` commands, including the ones below: ``` apigee-service apigee-cassandra stop apigee-service apigee-cassandra wait_for_ready apigee-service apigee-cassandra ring apigee-service apigee-cassandra backup ``` There are several options for configuring `apigee-service` for JMX authentication and SSL. Choose an option based on usability and your security practices. - [Option 1 (SSL arguments stored in file)](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#option1) - [Option 2 (SSL arguments stored in environment variables)](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#option2) - [Option 3 (SSL arguments passed directly to `apigee-service`)](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#option3) #### Option 1 (SSL arguments stored in file) Set the following environment variables: ``` export CASS_JMX_USERNAME=ADMIN # Provide encrypted password here if you have setup JMX password encryption export CASS_JMX_PASSWORD=PASSWORD export CASS_JMX_SSL=Y ``` Create a file in Apigee user's home directory (`/opt/apigee`). ``` $HOME/.cassandra/nodetool-ssl.properties ``` Edit the file and add the following lines: ``` -Djavax.net.ssl.trustStore= -Djavax.net.ssl.trustStorePassword= -Dcom.sun.management.jmxremote.registry.ssl=true ``` Make sure the trustore file is readable by Apigee user. Run the following `apigee-service` command. If it runs without error, your configurations are correct. ``` apigee-service apigee-cassandra ring ``` #### Option 2 (SSL arguments stored in environment variables) Set the following environment variables: ``` export CASS_JMX_USERNAME=ADMIN # Provide encrypted password here if you have setup JMX password encryption export CASS_JMX_PASSWORD=PASSWORD export CASS_JMX_SSL=Y # Ensure the truststore file is accessible by Apigee user. export CASS_JMX_TRUSTSTORE= export CASS_JMX_TRUSTSTORE_PASSWORD= ``` Run the following `apigee-service` command. If it runs without error, your configurations are correct. ``` apigee-service apigee-cassandra ring ``` #### Option 3 (SSL arguments passed directly to `apigee-service`) Run any `apigee-service` command like the one below. You don't need to configure any environment variables. ``` CASS_JMX_USERNAME=ADMIN CASS_JMX_PASSWORD=PASSWORD CASS_JMX_SSL=Y CASS_JMX_TRUSTSTORE= CASS_JMX_TRUSTSTORE_PASSWORD= /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra ring ``` 6. Set up nodetool. Nodetool requires JMX parameters to be passed to it. There are two ways you can configure nodetool to run with SSL-enabled JMX, as described in the configuration options below: - [Configuration Option 1](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#config-option1) - [Configuration Option 2](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#config-option2) The options differ in the way SSL related configurations are passed to nodetool. In both cases, the user running nodetool should have READ permissions on the truststore file. Choose an appropriate option based on usability and your security practices. To learn more about nodetool parameters, see the [DataStax documentation](https://docs.datastax.com/en/archived/cassandra/2.1/cassandra/security/secureNodetoolSSL.html). #### Configuration Option 1 Create a file in the home directory of the user running nodetool. ``` $HOME/.cassandra/nodetool-ssl.properties ``` Add the following lines to the file: ``` -Djavax.net.ssl.trustStore= -Djavax.net.ssl.trustStorePassword= -Dcom.sun.management.jmxremote.registry.ssl=true ``` The truststore path specified above should be accessible by any user running nodetool. Run `nodetool` with the `--ssl` option. ``` /opt/apigee/apigee-cassandra/bin/nodetool --ssl -u -pw -h localhost ring ``` #### Configuration option 2 Run `nodetool` as a single command with the extra parameters listed below. ``` /opt/apigee/apigee-cassandra/bin/nodetool -Djavax.net.ssl.trustStore= -Djavax.net.ssl.trustStorePassword= -Dcom.sun.management.jmxremote.registry.ssl=true -Dssl.enable=true -u -pw -h localhost ring ``` ### Revert SSL configurations If you need to revert the SSL configurations described in the procedure above, do the following steps: 1. Stop `apigee-cassandra`by entering ``` apigee-service apigee-cassandra stop ``` 2. Remove the line `conf_cassandra-env_com.sun.management.jmxremote.ssl=true` from the file `/opt/apigee/customer/application/cassandra.properties`. 3. Comment out the following lines in `/opt/apigee/apigee-cassandra/source/conf/cassandra-env.sh` ``` # JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStore=/opt/apigee/data/apigee-cassandra/keystore.node0" # JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStorePassword=keypass" # JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.registry.ssl=true” ``` 4. Start `apigee-cassandra` by entering ``` apigee-service apigee-cassandra start ``` 5. Remove the environment variable `CASS_JMX_SSL` if it was set.
``` unset CASS_JMX_SSL ``` 6. Check that `apigee-service` based commands like `ring`, `stop`, `backup`, and so on, are working. 7. Stop using the `--ssl` switch with nodetool ### Disable JMX authentication for Cassandra **To disable JMX authentication for Cassandra:** 1. Edit `/opt/apigee/customer/application/cassandra.properties`. 2. Remove the following line in the file: ``` conf_cassandra-env_com.sun.management.jmxremote.authenticate=true ``` 3. Run configure on Cassandra: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra configure ``` 4. Restart Cassandra: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restart ``` 5. Repeat this process on all other Cassandra nodes. ### Use JConsole: Monitor task statistics Use JConsole and the following service URL to monitor the JMX attributes (MBeans) offered via JMX: ``` service:jmx:rmi:///jndi/rmi://IP_address:7199/jmxrmi ``` Where IP_address is the IP of the Cassandra server. ### Cassandra JMX statistics | JMX MBeans | JMX Attributes | |---|---| | ColumnFamilies/apprepo/environments ColumnFamilies/apprepo/organizations ColumnFamilies/apprepo/apiproxy_revisions ColumnFamilies/apprepo/apiproxies ColumnFamilies/audit/audits ColumnFamilies/audit/audits_ref | PendingTasks | | ColumnFamilies/apprepo/environments ColumnFamilies/apprepo/organizations ColumnFamilies/apprepo/apiproxy_revisions ColumnFamilies/apprepo/apiproxies ColumnFamilies/audit/audits ColumnFamilies/audit/audits_ref | MemtableColumnsCount | | ColumnFamilies/apprepo/environments ColumnFamilies/apprepo/organizations ColumnFamilies/apprepo/apiproxy_revisions ColumnFamilies/apprepo/apiproxies ColumnFamilies/audit/audits ColumnFamilies/audit/audits_ref | MemtableDataSize | | ColumnFamilies/apprepo/environments ColumnFamilies/apprepo/organizations ColumnFamilies/apprepo/apiproxy_revisions ColumnFamilies/apprepo/apiproxies ColumnFamilies/audit/audits ColumnFamilies/audit/audits_ref | ReadCount | | ColumnFamilies/apprepo/environments ColumnFamilies/apprepo/organizations ColumnFamilies/apprepo/apiproxy_revisions ColumnFamilies/apprepo/apiproxies ColumnFamilies/audit/audits ColumnFamilies/audit/audits_ref | RecentReadLatencyMicros | | ColumnFamilies/apprepo/environments ColumnFamilies/apprepo/organizations ColumnFamilies/apprepo/apiproxy_revisions ColumnFamilies/apprepo/apiproxies ColumnFamilies/audit/audits ColumnFamilies/audit/audits_ref | TotalReadLatencyMicros | | ColumnFamilies/apprepo/environments ColumnFamilies/apprepo/organizations ColumnFamilies/apprepo/apiproxy_revisions ColumnFamilies/apprepo/apiproxies ColumnFamilies/audit/audits ColumnFamilies/audit/audits_ref | WriteCount | | ColumnFamilies/apprepo/environments ColumnFamilies/apprepo/organizations ColumnFamilies/apprepo/apiproxy_revisions ColumnFamilies/apprepo/apiproxies ColumnFamilies/audit/audits ColumnFamilies/audit/audits_ref | RecentWriteLatencyMicros | | ColumnFamilies/apprepo/environments ColumnFamilies/apprepo/organizations ColumnFamilies/apprepo/apiproxy_revisions ColumnFamilies/apprepo/apiproxies ColumnFamilies/audit/audits ColumnFamilies/audit/audits_ref | TotalWriteLatencyMicros | | ColumnFamilies/apprepo/environments ColumnFamilies/apprepo/organizations ColumnFamilies/apprepo/apiproxy_revisions ColumnFamilies/apprepo/apiproxies ColumnFamilies/audit/audits ColumnFamilies/audit/audits_ref | TotalDiskSpaceUsed | | ColumnFamilies/apprepo/environments ColumnFamilies/apprepo/organizations ColumnFamilies/apprepo/apiproxy_revisions ColumnFamilies/apprepo/apiproxies ColumnFamilies/audit/audits ColumnFamilies/audit/audits_ref | LiveDiskSpaceUsed | | ColumnFamilies/apprepo/environments ColumnFamilies/apprepo/organizations ColumnFamilies/apprepo/apiproxy_revisions ColumnFamilies/apprepo/apiproxies ColumnFamilies/audit/audits ColumnFamilies/audit/audits_ref | LiveSSTableCount | | ColumnFamilies/apprepo/environments ColumnFamilies/apprepo/organizations ColumnFamilies/apprepo/apiproxy_revisions ColumnFamilies/apprepo/apiproxies ColumnFamilies/audit/audits ColumnFamilies/audit/audits_ref | BloomFilterFalsePositives | | ColumnFamilies/apprepo/environments ColumnFamilies/apprepo/organizations ColumnFamilies/apprepo/apiproxy_revisions ColumnFamilies/apprepo/apiproxies ColumnFamilies/audit/audits ColumnFamilies/audit/audits_ref | RecentBloomFilterFalseRatio | | ColumnFamilies/apprepo/environments ColumnFamilies/apprepo/organizations ColumnFamilies/apprepo/apiproxy_revisions ColumnFamilies/apprepo/apiproxies ColumnFamilies/audit/audits ColumnFamilies/audit/audits_ref | BloomFilterFalseRatio | ### Use nodetool to manage cluster nodes The nodetool utility is a command line interface for Cassandra that manages cluster nodes. The utility can be found at `/opt/apigee/apigee-cassandra/bin`. The following calls can be made on all Cassandra cluster nodes: 1. **General ring info** (also possible for single Cassandra node): Look for the "Up" and "Normal" for all nodes. ``` nodetool [-u username -pw password] -h localhost ring ``` You only need to pass your username and password if you [enabled JMX authentication for Cassandra](https://docs.apigee.com/private-cloud/v4.52.02/how-monitor#cassauthenable). The output of the above command looks as shown below: ``` Datacenter: dc-1 ========== Address Rack Status State Load Owns Token 192.168.124.201 ra1 Up Normal 1.67 MB 33,33% 0 192.168.124.202 ra1 Up Normal 1.68 MB 33,33% 5671...5242 192.168.124.203 ra1 Up Normal 1.67 MB 33,33% 1134...0484 ``` 2. **General info about nodes** (call per node) ``` nodetool [-u username -pw password] -h localhost info ``` The output of the above command looks like the following: ``` ID : e2e42793-4242-4e82-bcf0-oicu812 Gossip active : true Thrift active : true Native Transport active: true Load : 273.71 KB Generation No : 1234567890 Uptime (seconds) : 687194 Heap Memory (MB) : 314.62 / 3680.00 Off Heap Memory (MB) : 0.14 Data Center : dc-1 Rack : ra-1 Exceptions : 0 Key Cache : entries 150, size 13.52 KB, capacity 100 MB, 1520781 hits, 1520923 requests, 1.000 recent hit rate, 14400 save period in seconds Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Token : 0 ``` 3. **Status of the thrift server** (serving client API) ``` nodetool [-u username -pw password] -h localhost statusthrift ``` The output of the above command looks like the following: ``` running ``` 4. **Status of data streaming operations** : Observe traffic for cassandra nodes: ``` nodetool [-u username -pw password] -h localhost netstats ``` The output of the above command looks like the following: ``` Mode: NORMAL Not sending any streams. Read Repair Statistics: Attempted: 151612 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool Name Active Pending Completed Dropped Commands n/a 0 0 0 Responses n/a 0 0 n/a ``` For more information on using the nodetool, see [About the nodetool utility](https://cassandra.apache.org/doc/3.11/cassandra/tools/nodetool/nodetool.html). ### Cassandra resource Refer to the following URL: [Cassandra Monitoring](https://cassandra.apache.org/doc/3.11/cassandra/operating/metrics.html). ## Monitoring Apache Qpid Broker-J You can monitor Qpid Broker-J from the Qpid management console. This section explains how to access the console and use it to perform basic monitoring functions. For more detailed information on using the management console, see [Web Management Console](https://qpid.apache.org/releases/qpid-broker-j-8.0.6/book/Java-Broker-Management-Channel-Web-Console.html) in the Apache Qpid documentation. > [!NOTE] > **Note:** We recommend you avoid making changes to the broker through the Qpid management console. If you intend to make broker configuration changes, we recommend you manage such changes using Edge properties files (referred to as *code with config*, or CwC) rather than using the console. If you do make broker changes using the management console, we recommend you test your changes on non-production environments before applying them to production. See also [How to configure Edge](https://docs.apigee.com/private-cloud/v4.52.01/how-configure-edge). ### Access the management console The default management console port is 8090. To access the console on this default port, point your web browser to: ``` http://QPID_NODE_IP:8090 ``` > [!NOTE] > **Note:**If port 8090 is not open to access from your browser, you must extend or forward the port to a node with web browser support. To log in to the console, use the default credentials set by Apigee or the ones set in the Edge configuration file. For details, see [Edge configuration file reference](https://docs.apigee.com/private-cloud/v4.52.01/edge-configuration-file-reference). ### Monitor queues and messages In the left-navigation panel, navigate to **Java-Broker \> virtualhosts \> queues**. Select a queue to view its details in the main part of the UI. In the details view, you can see Queue Attributes and Statistics, including information on messages delivered, enqueued, message rates, and so on. ### View and download log files In the left-navigation panel, navigate to **Java-Broker \> brokerloggers \> logfile**. In the main UI details view, you can view log file details and download log files. ### Using the Qpid management API You can use the Apache Qpid Broker-J REST API to automate management tasks and monitor the broker. For details, see the [Apache Qpid Broker REST API documentation](https://qpid.apache.org/releases/qpid-broker-j-8.0.6/book/Java-Broker-Management-Channel-REST-API.html). You can also use command line tools to monitor the broker. For example: ``` curl "QPID_NODE_IP":"8090"/api/latest/broker -u "USERNAME":"PASSWORD" ``` ### Enable monitoring with SSL for Qpid To provide additional security for monitoring and management, enable SSL on Qpid's Management portal and Qpid's Management APIs. Follow the guidelines below to provide a key and certificate. Qpid provides an option for a File Keystore, enabled in Apigee. This type accepts the standard JKS keystore format understood by Java and Java tools such as keytool. #### Keystore preparation You need to provide the certificate file directly for use by clients of Qpidd, in addition to using the keytool. For generating keystores, see [Java Keytool Documentation.](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html) #### After verifying requirements 1. Place the keystore and certificate files in `/opt/apigee/customer/application/apigee-qpidd`. 2. Ensure the keystore file is readable only by the Apigee user: ``` chown apigee:apigee /opt/apigee/customer/application/apigee-qpidd/qpidd.keystore chmod 400 /opt/apigee/customer/application/apigee-qpidd/qpidd.keystore chown apigee:apigee /opt/apigee/customer/application/apigee-qpidd/qpidd-cert.pem chmod 400 /opt/apigee/customer/application/apigee-qpidd/qpidd-cert.pem ``` #### Enable SSL in Qpid Perform the steps below on one Qpid node at a time: Open the file `/opt/apigee/customer/application/qpidd.properties` and add the following lines: ``` conf_qpidd_qpid_management.https.enabled=true conf_qpidd_qpid.keystore.storeUrl=/opt/apigee/customer/application/apigee-qpidd/qpidd.keystore conf_qpidd_qpid.keystore.password=keystore-password conf_qpidd_qpid.keystore.certificateAlias=certificate-alias conf_qpidd_qpid.keystore.certificate=/opt/apigee/customer/application/apigee-qpidd/qpidd-cert.pem ``` 1. Change the owner of the file to apigee:apigee: ``` chown apigee:apigee /opt/apigee/customer/application/qpidd.properties ``` 2. Configure Qpidd: ``` apigee-service apigee-qpidd configure ``` 3. Restart Qpidd: ``` apigee-service apigee-qpidd restart ``` 4. Check the status with wait_for_ready: ``` apigee-service apigee-qpidd wait_for_ready ``` > [!NOTE] > **Note:** Repeat these steps on each QPID node one at a time to ensure proper configuration and synchronization. #### Revert SSL configuration Remove or comment out the properties from the file `/opt/apigee/customer/application/qpidd.properties`: ``` conf_qpidd_qpid_management.https.enabled=true conf_qpidd_qpid.keystore.storeUrl=/opt/apigee/customer/application/apigee-qpidd/keystore ``` 1. Configure Qpidd: ``` apigee-service apigee-qpidd configure ``` 2. Restart Qpidd: ``` apigee-service apigee-qpidd restart ``` 3. Check the status with wait_for_ready: ``` apigee-service apigee-qpidd wait_for_ready ``` ## Apache ZooKeeper ### Check ZooKeeper status 1. Ensure the ZooKeeper process is running. ZooKeeper writes a PID file to `opt/apigee/var/run/apigee-zookeeper/apigee-zookeeper.pid`. 2. Test ZooKeeper ports to ensure that you can establish a TCP connection to ports 2181 and 3888 on every ZooKeeper server. 3. Ensure that you can read values from the ZooKeeper database. Connect using a ZooKeeper client library (or `/opt/apigee/apigee-zookeeper/bin/zkCli.sh`) and read a value from the database. 4. Check the status: ``` /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper status ``` ### Use ZooKeeper four-letter words ZooKeeper can be monitored via a small set of commands (four-letter words) that are sent to the port 2181 using netcat (nc) or telnet. For more info on ZooKeeper commands, see: [Apache ZooKeeper command reference](https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_zkCommands). For example: - `srvr`: Lists full details for the server. - `stat`: Lists brief details for the server and connected clients. The following commands can be issued to the ZooKeeper port: 1. Run the four-letter command ruok to test if server is running in a non-error state. A successful response returns "imok". ``` echo ruok | nc host 2181 ``` Returns: ``` imok ``` 2. Run the four-letter command, `stat`, to list server performance and connected clients statistics: ``` echo stat | nc host 2181 ``` Returns: ``` Zookeeper version: 3.4.5-1392090, built on 09/30/2012 17:52 GMT Clients: /0:0:0:0:0:0:0:1:33467[0](queued=0,recved=1,sent=0) /192.168.124.201:42388[1](queued=0,recved=8433,sent=8433) /192.168.124.202:42185[1](queued=0,recved=1339,sent=1347) /192.168.124.204:39296[1](queued=0,recved=7688,sent=7692) Latency min/avg/max: 0/0/128 Received: 26144 Sent: 26160 Connections: 4 Outstanding: 0 Zxid: 0x2000002c2 Mode: follower Node count: 283 ``` > [!NOTE] > **Note**: It is sometimes important to see whether a ZooKeeper is in Mode: leader, follower or observer. 3. If netcat (nc) is not available, you can use the python as an alternative. Create a file named `zookeeper.py` that contains the following: ``` import time, socket, sys c = socket.socket(socket.AF_INET, socket.SOCK_STREAM) c.connect((sys.argv[1], 2181)) c.send(sys.argv[2]) time.sleep(0.1) print c.recv(512) ``` Now run the following python lines: python zookeeper.py 192.168.124.201 ruok python zookeeper.py 192.168.124.201 stat ## LDAP level test You can monitor OpenLDAP to see whether the specific requests are served properly. In other words, check for a specific search that returns the right result. 1. Use `ldapsearch` (`yum install openldap-clients`) to query the entry of the system admin. This entry is used to authenticate all API calls. ``` ldapsearch -b "uid=admin,ou=users,ou=global,dc=apigee,dc=com" -x -W -D "cn=manager,dc=apigee,dc=com" -H ldap://localhost:10389 -LLL ``` You are then prompted for the LDAP admin password: ``` Enter LDAP Password: ``` After entering the password, you see a response in the form: ``` dn: uid=admin,ou=users,ou=global,dc=apigee,dc=com objectClass: organizationalPerson objectClass: person objectClass: inetOrgPerson objectClass: top uid: admin cn: admin sn: admin userPassword:: e1NTSEF9bS9xbS9RbVNXSFFtUWVsU1F0c3BGL3BQMkhObFp2eDFKUytmZVE9PQ= = mail: opdk@google.com ``` 2. Check whether Management Server is still connected to LDAP with the following command: ``` curl -u userEMail:password http://localhost:8080/v1/users/ADMIN ``` Returns: ``` { "emailId" : ADMIN, "firstName" : "admin", "lastName" : "admin" } ``` You can also monitor the OpenLDAP caches, which help in reducing the number of disk accesses and hence improve the performance of the system. Monitoring and then tuning the cache size in the OpenLDAP server can heavily impact the performance of the directory server. You can view the log files (`opt/apigee/var/log`) to obtain information about cache.