503 Service Unavailable - NoActiveTargets - HealthCheckFailures

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation.
info

Videos

See the following videos for more information on 503 errors:

Video Description
Troubleshoot and resolve 503 Service Unavailable - NoActiveTargets Learn about the following:
  • Importance of Target Servers and Health Monitors
  • Troubleshooting and resolving a real-time 503 Service Unavailable - NoActiveTargets error caused due to health check failure

Symptom

The client application receives the HTTP response status code 503 with the message Service Unavailable and the error code NoActiveTargets for the API proxy requests.

Error message

You will see the following error response:

HTTP/1.1 503 Service Unavailable
  

You will see the following error message in the HTTP response:

{
   "fault": {
      "faultstring": "The Service is temporarily unavailable",
      "detail": {
           "errorcode": "messaging.adaptors.http.flow.NoActiveTargets"
       }
    }
}
  

Possible causes

The HTTP response 503 Service Unavailable with the error code NoActiveTargets is typically observed when you use one or more target servers in the target endpoint configuration in your API Proxy.

This playbook covers 503 Service Unavailable with the error code NoActiveTargets caused due to health check failures. Please refer to this playbook to learn about other causes for this error.

Health check failures

The health check failures will be observed only if you have configured a Health Monitor as part of the target server load balancing configuration in the target endpoint of your API Proxy.

When a target server fails a health check, Edge increments that server’s failure count. If the number of health check failures for that server reaches the predefined threshold (<MaxFailures>), The Message Processor logs the warning message as shown below into its log file:

Apigee-Timer-7 WARN  ADAPTORS.HTTP.FLOW - LBServer.incrementFailureCount() : Max failure count(10) reached for server : mocktarget2{Environment=<orgname>__prod,Application=mocktargetapigee__1,Target=default}
    

The warning message provides the following information. This helps you understand which target server reached the MaxFailure count:

  • Target Server name
  • Organization and Environment names
  • API Proxy name
  • Target Endpoint name

Thereafter, Edge stops sending any further requests to that specific server. Once all the target servers configured in the LoadBalancer configuration reach the MaxFailure count, the subsequent API requests are responded with 503 Service Unavailable with the error code NoActiveTargets.

Using Health Monitor helps Apigee Edge to automatically include a target server back into the rotation when it becomes healthy, without having to redeploy the API Proxy.

Here are the possible causes for the health check failures:

Cause Description Who can perform the troubleshooting steps
Connection Timeout Error The Message Processor is unable to connect to the target server within the specified timeout period in the LoadBalancer configuration. Edge Private Cloud users
Secure Request on Non-Secure Port
  1. If the target server is defined to be a secure server, but incorrectly configured with a non-secure port.
  2. If the target server is defined to be a secure server, but the health monitor is configured to perform health checks on a non-secure port.
Edge Private Cloud users
Non-Secure Request on Secure Port
  1. If the target server is defined to be a non-secure server, but incorrectly configured with a secure port.
  2. If the target server is defined to be a non-secure server, but the health monitor is configured to perform health checks on a secure port.
Edge Private Cloud users
Health Check API responds with an error If the health check API responds with an error or a response code, anything other than specified in the SuccessResponse element of the Health Monitor. Edge Private Cloud users

Common diagnosis steps

Determine the Message ID of the failing request

Trace tool

To determine the message ID of the failing request using the Trace tool:

  1. Enable the trace session, make the API call, and reproduce the issue - 503 Service Unavailable with error code NoActiveTargets.
  2. Select one of the failing requests.
  3. Navigate to the AX phase, and determine the message ID (X-Apigee.Message-ID) of the request by scrolling down in the Phase Details section as shown in the following figure.

    Message ID in Phase Details section

NGINX access logs

To determine the message ID of the failing request using the NGINX access logs:

You can also refer to NGINX Access logs to determine the message ID for the 503 errors. This is particularly useful if the issue has occurred in the past or if the issue is intermittent and you are unable to capture the trace in the UI. Use the following steps to determine this information from NGINX access logs:

  1. Check the NGINX access logs: (/opt/apigee/var/log/edge-router/nginx/ <org>~ <env>.<port#>_access_log)
  2. Search if there are any 503 Errors for the specific API proxy during a specific duration (if the problem happened in the past) or if there are any requests still failing with 503.
  3. If there are any 503 Errors with X-Apigee-fault-code messaging.adaptors.http.flow.NoActiveTargets, note the message ID for one or more such requests as shown in the following example:

    Sample Entry showing the 503 Error

    Sample entry showing status code, message ID, fault source, and fault code

Common error messages

When target servers are used and an error occurs while the Message Processor is attempting to connect with the backend server, then you will see a few common error messages in the Message Processor logs. These errors are logged after the actual exception/error message that led to the failure.

The common error messages observed in Message Processor logs (/opt/apigee/var/log/edge-message-processor/logs/system.log) for the 503 Service Unavailable with the error code NoActiveTargets are as follows:

org:myorg env:prod api:TestTargetServer rev:2 messageid:<messageid>  NIOThread@0 INFO  ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : Failed to send request to target servers : [demo-target] for default{Organization=myorgEnvironment=prod,Application=TestTargetServer__2}

org:myorg env:prod api:TestTargetServer rev:2 messageid:<messageid>  NIOThread@0 ERROR ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : No Active Target server Found for default{Organization=myorgEnvironment=prod,Application=TestTargetServer__2}

org:myorg env:prod api:TestTargetServer rev:2 messageid:<messageid>  NIOThread@0 ERROR ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : Unexpected error while sending request
com.apigee.errors.http.server.ServiceUnavailableException: The Service is temporarily unavailable
	at com.apigee.messaging.adaptors.http.flow.data.LBTargetRequestSender.sendRequest(LBTargetRequestSender.java:299)
	at com.apigee.messaging.adaptors.http.flow.data.LBTargetRequestSender.access$400(LBTargetRequestSender.java:57)
	…<snipped>

These error messages indicate that the request could not be sent to the backend server due to a failure. As a result, the Message Processor sends out 503 Service Unavailable with error code NoActiveTargets as its response to the client.

Cause: Connection timeout

Diagnosis

  1. Determine the message id of the failing request.
  2. Search for the message id in the Message Processor log (/opt/apigee/var/log/edge-message-processor/logs/system.log).
  3. You will see the common error messages corresponding to the message id. However, to get the actual cause for the health check failures, scroll above these common error messages and check for any HEALTH MONITOR Errors.

    For example, The following HEALTH MONITOR error message indicates that Message Processor failed with connection timed out error when making the health check API request:

    Apigee-Timer-6 ERROR SERVICES.HEALTH_MONITOR - HTTPMonitor.getResponseFromCache() : Error sending request Request URL : https://<BackendServer-Hostname>:443/status
    java.net.ConnectException: Connection timed out (Connection timed out)
    	at java.net.PlainSocketImpl.socketConnect(Native Method)
    	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    …<snipped>
            

    If this error repeats for MaxFailure number of times configured in the Health Monitor, then you will see a warning message like this:

    Apigee-Timer-7 WARN  ADAPTORS.HTTP.FLOW - LBServer.incrementFailureCount() : Max failure count(10) reached for server : mocktarget2{Environment=<orgname>__prod,Application=mocktargetapigee__1,Target=default}
            

    Read the information given in the warning message carefully. Ensure that the MaxFailure count has been reached for a target server used in the specific API Proxy for which you are experiencing the 503 response code with error code NoActiveTargets.

  4. In the example above, the health check failed with the connection timed out error. Check if you are able to connect to the specific backend server directly from each of the Message Processors using the telnet command:
  5. telnet <BackendServer-HostName> 443
          
  6. If you are able to connect to the backend server, then you might see a message like Connected to backend-server. Then the problem might be a temporary issue and it might be either resolved or is an intermittent issue. Repeat step 4 a few times (10+ times) and verify the output.
    1. If there are no errors with the telnet command consistently, then the issue is resolved. Recheck if the health check failures have stopped. If yes, then you don’t have to do anything further.
    2. If you are unable to connect to the backend server with the telnet command intermittently, then there might be a network issue or your backend server might be busy.
  7. If you are unable to connect to the backend server with the telnet command consistently, then this might be because the traffic is not allowed from the Message Processors on the specific backend server.

Resolution

If the connection timed out error is consistently observed, then ensure the backend server does not have any firewall restrictions and allows the traffic from the Apigee Edge Message Processors. For example, on Linux, you could use iptables to allow the traffic from the Message Processor's IP addresses on the backend server.

If the problem persists, work with your Network administrator to determine and fix the issue. If you need any further assistance from Apigee, contact Apigee Support.

Cause: Secure request on non-secure port

Diagnosis

  1. Determine the message id of the failing request.
  2. Search for the message id in the Message Processor log (/opt/apigee/var/log/edge-message-processor/logs/system.log).
  3. You will see the common error messages corresponding to the message id. However, to get the actual cause for the health check failures, scroll above these common error messages and check for any HEALTH MONITOR Errors.

    For example, you may see a HEALTH MONITOR error as shown below:

    Apigee-Timer-1 ERROR SERVICES.HEALTH_MONITOR - HTTPMonitor.getResponseFromCache() : Error sending request Request URL : https://mocktarget.apigee.net:80/status
    javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
            at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
            at sun.security.ssl.InputRecord.read(InputRecord.java:527)
            at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
            at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
            at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
            at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
    …<snipped>
            

    If this error repeats for MaxFailure number of times configured in the Health Monitor, then you will see a warning message like this:

    Apigee-Timer-7 WARN  ADAPTORS.HTTP.FLOW - LBServer.incrementFailureCount() : Max failure count(10) reached for server : mocktarget{Environment=<orgname>__prod,Application=mocktargetapigee__1,Target=default}
            

    Read the information given in the warning message carefully. Ensure that the MaxFailure count has been reached for a target server used in the specific API Proxy for which you are experiencing the 503 response code with error code NoActiveTargets.

  4. The health check failed with the error:
    Error sending request Request URL : https://mocktarget.apigee.net:80/statuscode/200
    javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
          

    The error message and the URL indicate the cause of this issue is that a secure call (HTTPS) was made on the non-secure port 80.

    This error can occur under the following two scenarios:

    • Secure target server defined with non-secure port
    • Secure target server defined but Health Monitor configured with a non-secure port

    Secure target non-secure port

    Scenario 1: Secure target server defined with non-secure port

    If you have defined a secure target server but with a non-secure port such as 80, then you get this error. Follow the steps below to verify if this is the cause for this issue:

    1. Check the definition of the target server used in the target endpoint configuration.
    2. Use the Get TargetServer API to get the target server definition.

      Target Server Definition Output

      <TargetServer name="mocktarget">
        <Host>mocktarget.apigee.net</Host>
        <Port>80</Port>
        <IsEnabled>true</IsEnabled>
        <SSLInfo>
            <Enabled>true</Enabled>
        </SSLInfo>
      </TargetServer>
                

      In the example above, the definition shows that the target server mocktarget is a secure server as indicated by the SSLInfo block. However, it is configured with a non-secure Port 80.

    3. Now, check the Health Monitor configuration for the target server in the target endpoint configuration:

      Health Monitor Configuration

      <HealthMonitor>
        <IsEnabled>true</IsEnabled>
        <IntervalInSec>5</IntervalInSec>
        <HTTPMonitor>
          <Request>
            <ConnectTimeoutInSec>10</ConnectTimeoutInSec>
            <SocketReadTimeoutInSec>30</SocketReadTimeoutInSec>
            <Verb>GET</Verb>
            <Path>/statuscode/200</Path>
          </Request>
          <SuccessResponse>
            <ResponseCode>200</ResponseCode>
          </SuccessResponse>
        </HTTPMonitor>
      </HealthMonitor>
                

      Notice that there is no <Port> element specified in the Health Monitor configuration above. In this case, the Edge’s Message Processor uses the port specified in the target server definition (which is 80) for making health check API calls.

    4. Based on the information above, the cause for this error is that the target server is defined as a secure server (as SSLInfo block is enabled), but with a non-secure port 80.

    Secure target non-secure HM port

    Scenario 2: Secure target server defined but Health Monitor configured with a non-secure port

    If you have defined a secure target server but the Health Monitor is configured with a non-secure port such as 80, then you get this error. Follow the steps below to verify if this is the cause for this issue:

    1. Check the definition of the target server used in the target endpoint configuration.

      Use the Get TargetServer API to get the target server definition.

      Target Server Definition Output

      <TargetServer name="mocktarget">
        <Host>mocktarget.apigee.net</Host>
        <Port>443</Port>
        <IsEnabled>true</IsEnabled>
        <SSLInfo>
            <Enabled>true</Enabled>
        </SSLInfo>
      </TargetServer>
              

      In the example above, the definition shows that the target server mocktarget is a secure server as indicated by the SSLInfo block.

    2. Next, check the Health Monitor configuration for the target server in the target endpoint configuration:

      Health Monitor Configuration

      <HealthMonitor>
        <IsEnabled>true</IsEnabled>
        <IntervalInSec>5</IntervalInSec>
        <HTTPMonitor>
          <Request>
            <ConnectTimeoutInSec>10</ConnectTimeoutInSec>
         	<SocketReadTimeoutInSec>30</SocketReadTimeoutInSec>
            <Port>80</Port>
            <Verb>GET</Verb>
            <Path>/statuscode/200</Path>
          </Request>
          <SuccessResponse>
            <ResponseCode>200</ResponseCode>
          </SuccessResponse>
        </HTTPMonitor>
              

      In the example above, the Health Monitor is configured with a non-secure Port 80 as indicated by <Port> element.

    3. Based on the information above, the cause for this error is that the target server is defined as a secure server (as SSLInfo block is enabled) and uses secure port 443, but the Health Monitor is configured to perform health checks with a non-secure port 80 (specified in <Port> element).

      That is, in this case, Edge makes the health check APIs as a secure call with non-secure port 80 and fails with the above-mentioned error.

Resolution

Secure target non-secure port

Scenario 1: Secure target server defined with non-secure port

To fix this error, update the target server definition to use an appropriate secure port.

Use the Update a TargetServer API to update the target server definition and ensure that a secure port (for example: 443) is used as shown in the example below:

<TargetServer name="mocktarget">
  <Host>mocktarget.apigee.net</Host>
  <Port>443</Port>
  <IsEnabled>true</IsEnabled>
  <SSLInfo>
      <Enabled>true</Enabled>
  </SSLInfo>
</TargetServer>
    

Secure target non-secure HM port

Scenario 2: Secure target server defined but Health Monitor configured with a non-secure port

To fix this error, follow the instructions below:

  1. Modify the Health Monitor configuration to use a secure port (for example: 443) to perform target server health checks in the target endpoint configuration of the failing API Proxy as shown below:
    <HealthMonitor>
      <IsEnabled>true</IsEnabled>
      <IntervalInSec>5</IntervalInSec>
      <HTTPMonitor>
        <Request>
          <ConnectTimeoutInSec>10</ConnectTimeoutInSec>
        <SocketReadTimeoutInSec>30</SocketReadTimeoutInSec>
          <Port>443</Port>
          <Verb>GET</Verb>
          <Path>/statuscode/200</Path>
        </Request>
        <SuccessResponse>
          <ResponseCode>200</ResponseCode>
        </SuccessResponse>
      </HTTPMonitor>
    </HealthMonitor>
            
  2. Save the changes to the API Proxy.

Cause: Non-secure request on a secure port

Diagnosis

  1. Determine the message id of the failing request.
  2. Search for the message id in the Message Processor log (/opt/apigee/var/log/edge-message-processor/logs/system.log).
  3. You will see the common error messages corresponding to the message id. However, to get the actual cause for the health check failures, scroll above these common error messages and check for any HEALTH MONITOR Errors.

    For example, you may see a HEALTH MONITOR error as shown below:

    Apigee-Timer-2 ERROR SERVICES.HEALTH_MONITOR - HTTPMonitor.getResponseFromCache() : Error sending request Request URL : http://mocktarget.apigee.net:443/status
    java.net.SocketException: Unexpected end of file from server
    	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:851)
    	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
    	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:848)
    	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
    	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
    …<snipped>
              

    If this error repeats for MaxFailure number of times configured in the Health Monitor, then you will see a warning message like this:

    Apigee-Timer-7 WARN  ADAPTORS.HTTP.FLOW - LBServer.incrementFailureCount() : Max failure count(10) reached for server : mocktarget{Environment=<orgname>__prod,Application=mocktargetapigee__1,Target=default}
              

    Read the information given in the warning message carefully. Ensure that the MaxFailure count has been reached for a target server used in the specific API Proxy for which you are experiencing the 503 response code with error code NoActiveTargets.

  4. The health check failed with the error:
    Error sending request Request URL : http://mocktarget.apigee.net:443/status
    java.net.SocketException: Unexpected end of file from server
          

    The error message and the URL indicate the cause of this issue is that a non-secure call (HTTP) was made on the secure port 443.

    This error can occur under the following two scenarios:

    • Non-secure target server defined with secure port
    • Non-secure target server defined but Health Monitor configured with a secure port

    Non-secure target secure port

    Scenario 1: Non-secure target server defined with secure port

    If you have defined a non-secure target server but with a secure port such as 443, then you get this error. Follow the steps below to verify if this is the cause for this issue:

    1. Check the definition of the target server used in the target endpoint configuration.

      Use the Get TargetServer API to get the target server definition.

      Target Server Definition Output

      <TargetServer name="mocktarget">
        <Host>mocktarget.apigee.net</Host>
        <Port>443</Port>
        <IsEnabled>true</IsEnabled>
      </TargetServer>
                    

      In the example above, the definition shows that the target server mocktarget is a non-secure server as there is no SSLInfo block. However, it is incorrectly configured with a secure Port 443.

    2. Now, check the Health Monitor configuration for the target server in the target endpoint configuration:

      Health Monitor Configuration

      <HealthMonitor>
        <IsEnabled>true</IsEnabled>
        <IntervalInSec>5</IntervalInSec>
        <HTTPMonitor>
          <Request>
            <ConnectTimeoutInSec>10</ConnectTimeoutInSec>
            <SocketReadTimeoutInSec>30</SocketReadTimeoutInSec>
            <Verb>GET</Verb>
            <Path>/statuscode/200</Path>
          </Request>
          <SuccessResponse>
            <ResponseCode>200</ResponseCode>
          </SuccessResponse>
        </HTTPMonitor>
      </HealthMonitor>
                      

      Notice that there is no <Port> element specified in the Health Monitor configuration above. In this case, the Edge’s Message Processor will use the port specified in the target server definition which is 443.

    3. Based on the information above, the cause for this error is that the target server is defined as a non-secure server (as SSLInfo block is not defined), but with a secure port 443.

      That is, Edge makes the health checks as a non-secure call with secure port 443 and fails with the above-mentioned error.

    Non-secure target secure HM port

    Scenario 2: Non-secure target server defined but Health Monitor configured with a secure port

    If you have defined a non-secure target server but the Health Monitor is configured with a secure port such as 443, then you get this error. Follow the steps below to verify if this is the cause for this issue:

    1. Check the definition of the target server used in the target endpoint configuration.

      Use the Get TargetServer API to get the target server definition.

      Target Server Definition Output

      <TargetServer name="mocktarget">
        <Host>mocktarget.apigee.net</Host>
        <Port>80</Port>
        <IsEnabled>true</IsEnabled>
      </TargetServer>
              

      In the example above, the definition shows that the target server mocktarget is a non-secure server (as there is no SSLInfo block) configured with a non-secure port 80 correctly.

    2. Next, check the Health Monitor configuration for the target server in the target endpoint configuration:

      Health Monitor Configuration

      <HealthMonitor>
        <IsEnabled>true</IsEnabled>
        <IntervalInSec>5</IntervalInSec>
        <HTTPMonitor>
          <Request>
            <ConnectTimeoutInSec>10</ConnectTimeoutInSec>
         	<SocketReadTimeoutInSec>30</SocketReadTimeoutInSec>
            <Port>443</Port>
            <Verb>GET</Verb>
            <Path>/statuscode/200</Path>
          </Request>
          <SuccessResponse>
            <ResponseCode>200</ResponseCode>
          </SuccessResponse>
        </HTTPMonitor>
      </HealthMonitor>
            

      In the example above, the Health Monitor is configured with a secure Port 443 as indicated by <Port> element.

    3. Based on the information above, the cause for this error is that the target server is defined as a non-secure server (as SSLInfo block is not defined) with non-secure port 80 correctly, but the Health Monitor is configured to perform health checks with a secure port 443 (specified in <Port> element).

      That is, in this case, Edge makes the health checks as a non-secure call with secure port 443 and fails with the above-mentioned error.

Resolution

Non-secure target secure port

Scenario 1: Non-secure target server defined with secure port

To fix this error, update the target server definition to use an appropriate secure port.

Use the Update a Target Server API to update the target server definition and ensure that a non-secure port (for example: 80) is used as shown in the example below:

<TargetServer name="mocktarget">
  <Host>mocktarget.apigee.net</Host>
  <Port>80</Port>
  <IsEnabled>true</IsEnabled>
</TargetServer>
              

Non-secure target secure HM port

Scenario 2: Non-secure target server defined but Health Monitor configured with a secure port

To fix this error, follow the instructions below:

  1. Either remove the <Port> element from the Health Monitor configuration or modify the Health Monitor configuration to use a non-secure port (for example: 80) to perform target server health checks in the target endpoint configuration of the failing API Proxy as shown below:
    <HealthMonitor>
      <IsEnabled>true</IsEnabled>
      <IntervalInSec>5</IntervalInSec>
      <HTTPMonitor>
        <Request>
          <ConnectTimeoutInSec>10</ConnectTimeoutInSec>
       	<SocketReadTimeoutInSec>30</SocketReadTimeoutInSec>
          <Port>80</Port>
          <Verb>GET</Verb>
          <Path>/statuscode/200</Path>
        </Request>
        <SuccessResponse>
          <ResponseCode>200</ResponseCode>
        </SuccessResponse>
      </HTTPMonitor>
    </HealthMonitor>
            
  2. Save the changes to the API Proxy.

Cause: Health check API responds with an error

Diagnosis

  1. Determine the message id of the failing request.
  2. Search for the message id in the Message Processor log (/opt/apigee/var/log/edge-message-processor/logs/system.log).
  3. You will see the common error messages corresponding to the message id. However, to get the actual cause for the health check failures, scroll above these common error messages and check for any HEALTH MONITOR Errors/Warnings.

    For example, you may see a HEALTH MONITOR warning as shown below:

    Apigee-Timer-7 INFO  SERVICES.HEALTH_MONITOR - HTTPMonitor.sendRequest() : HTTPMonitor.monitor() : Connecting to https://mocktarget.apigee.net:443/status/200
    Apigee-Timer-7 WARN  SERVICES.HEALTH_MONITOR - HTTPMonitor.monitor() : HTTP response code from health monitoring service does not match.Expected response code : [200]. Received response code : 404
            

    If this error repeats for MaxFailure number of times configured in the Health Monitor, then you will see a warning message like this:

    Apigee-Timer-7 WARN  ADAPTORS.HTTP.FLOW - LBServer.incrementFailureCount() : Max failure count(10) reached for server : mocktarget{Environment=<orgname>__prod,Application=mocktargetapigee__1,Target=default}
            

    Read the information given in the warning message carefully. Ensure that the MaxFailure count has been reached for a target server used in the specific API Proxy for which you are experiencing the 503 response code with error code NoActiveTargets.

  4. The health check returned the warning message:
    HTTP response code from health monitoring service does not match.Expected response code : [200]. Received response code : 404
          

    The warning message above states that the expected response code for the health check API was 200, but the actual response received is 404. Hence, this is treated as a failure.

  5. Before investigating the cause for error response from the health check API, determine why Edge expects the response code as 200 for the health check API. For this, check the Health Monitor configuration for the target server in the target endpoint configuration:

    Health Monitor Configuration

    <HealthMonitor>
      <IsEnabled>true</IsEnabled>
      <IntervalInSec>5</IntervalInSec>
      <HTTPMonitor>
        <Request>
          <ConnectTimeoutInSec>10</ConnectTimeoutInSec>
       	<SocketReadTimeoutInSec>30</SocketReadTimeoutInSec>
          <Port>443</Port>
          <Verb>GET</Verb>
          <Path>/status/200</Path>
        </Request>
        <SuccessResponse>
          <ResponseCode>200</ResponseCode>
        </SuccessResponse>
      </HTTPMonitor>
    </HealthMonitor>
            

    Notice that the Health Monitor configuration is configured with the 200 response code under the <SuccessResponse> element. This means that if Edge gets any response code (such as 400, 401, 404, 500) other than 200 from the health check API, it will be treated as an error and increments the failure count.

  6. Now, to investigate the cause for error response from the health check API follow the steps below:
    1. Look at the message prior to the warning message in the Message Processor log.
      Apigee-Timer-7 INFO  SERVICES.HEALTH_MONITOR - HTTPMonitor.sendRequest() : HTTPMonitor.monitor() : Connecting to https://mocktarget.apigee.net:443/status/200
                

      Make a note of the health check URL from this message.

    2. You can make a direct call to this URL from the Message Processor and check the actual response
      curl -i https://mocktarget.apigee.net:443/status/200
                

      The response from the call above gives 404 as seen in the Message Processor logs:

      < HTTP/2 404
                
    3. This shows that, even the direct call to the health check URL fails with the same response code 404. This means that the health check URL may be incorrect or the resource being accessed as part of the URL is no longer available.
    4. In the example health check API provided above, the issue occurs because an incorrect URL was used in the Health Monitor configuration. The correct URL was found to be https://mocktarget.apigee.net:443/statuscode/200 from Mock Target API.
  7. If you get any other error response, then determine the cause for the same by following the steps above. If required, work with your backend team.

Resolution

  1. Fix the issue with the health check API on your backend server.
  2. To fix the issue in the example discussed above:
    1. Modify the <Path> element in the Health Monitor configuration to /statuscode/200 as shown below:
      <Path>/statuscode/200</Path>
              
    2. Save the changes in the API Proxy.

If the problem still persists, go to Must Gather Diagnostic Information.

Diagnose issues using API monitoring

API Monitoring enables you to isolate problem areas quickly to diagnose error, performance, and latency issues and their source, such as developer apps, API proxies, backend targets, or the API platform.

Step through a sample scenario that demonstrates how to troubleshoot 5xx issues with your APIs using API Monitoring. For example, you may want to set up an alert to be notified when the number of messaging.adaptors.http.flow.NoActiveTargets faults exceeds a particular threshold.

Must gather diagnostic information

If the problem persists even after following the instructions above, please gather the following diagnostic information. Contact and share them to Apigee Support:

  1. If you are a Public Cloud user, provide the following information:
    1. Organization Name
    2. Environment Name
    3. API Proxy Name
    4. Complete curl command to reproduce the error
    5. Trace file containing the requests with 503 Service Unavailable with error code NoActiveTargets
  2. If you are a Private Cloud user, provide the following information:
    1. Complete Error Message observed
    2. Environment Name
    3. API Proxy bundle
    4. Trace file containing the requests with 503 Service Unavailable with error code NoActiveTargets
    5. NGINX Access Logs

      (/opt/apigee/var/log/edge-router/nginx/<org>~<env>.<port#>_access_log)

    6. Message Processor Logs

      (/opt/apigee/var/log/edge-message-processor/logs/system.log)