503 Service Unavailable - NoActiveTargets - HealthCheckFailures

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Videos

See the following videos for more information on 503 errors:

Symptom

The client application receives the HTTP response status code 503 with the message Service Unavailable and the error code NoActiveTargets for the API proxy requests.

Error message

You will see the following error response:

HTTP/1.1 503 Service Unavailable
  

You will see the following error message in the HTTP response:

{
   "fault": {
      "faultstring": "The Service is temporarily unavailable",
      "detail": {
           "errorcode": "messaging.adaptors.http.flow.NoActiveTargets"
       }
    }
}
  

Possible causes

The HTTP response 503 Service Unavailable with the error code NoActiveTargets is typically observed when you use one or more target servers in the target endpoint configuration in your API Proxy.

This playbook covers 503 Service Unavailable with the error code NoActiveTargets caused due to health check failures. Please refer to this playbook to learn about other causes for this error.

Health check failures

The health check failures will be observed only if you have configured a Health Monitor as part of the target server load balancing configuration in the target endpoint of your API Proxy.

When a target server fails a health check, Edge increments that server’s failure count. If the number of health check failures for that server reaches the predefined threshold (<MaxFailures>), The Message Processor logs the warning message as shown below into its log file:

Apigee-Timer-7 WARN  ADAPTORS.HTTP.FLOW - LBServer.incrementFailureCount() : Max failure count(10) reached for server : mocktarget2{Environment=<orgname>__prod,Application=mocktargetapigee__1,Target=default}
    

The warning message provides the following information. This helps you understand which target server reached the MaxFailure count:

• Target Server name
• Organization and Environment names
• API Proxy name
• Target Endpoint name

Thereafter, Edge stops sending any further requests to that specific server. Once all the target servers configured in the LoadBalancer configuration reach the MaxFailure count, the subsequent API requests are responded with 503 Service Unavailable with the error code NoActiveTargets.

Using Health Monitor helps Apigee Edge to automatically include a target server back into the rotation when it becomes healthy, without having to redeploy the API Proxy.

Here are the possible causes for the health check failures:

Common diagnosis steps

Determine the Message ID of the failing request

Common error messages

When target servers are used and an error occurs while the Message Processor is attempting to connect with the backend server, then you will see a few common error messages in the Message Processor logs. These errors are logged after the actual exception/error message that led to the failure.

The common error messages observed in Message Processor logs (/opt/apigee/var/log/edge-message-processor/logs/system.log) for the 503 Service Unavailable with the error code NoActiveTargets are as follows:

org:myorg env:prod api:TestTargetServer rev:2 messageid:<messageid>  NIOThread@0 INFO  ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : Failed to send request to target servers : [demo-target] for default{Organization=myorgEnvironment=prod,Application=TestTargetServer__2}

org:myorg env:prod api:TestTargetServer rev:2 messageid:<messageid>  NIOThread@0 ERROR ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : No Active Target server Found for default{Organization=myorgEnvironment=prod,Application=TestTargetServer__2}

org:myorg env:prod api:TestTargetServer rev:2 messageid:<messageid>  NIOThread@0 ERROR ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : Unexpected error while sending request
com.apigee.errors.http.server.ServiceUnavailableException: The Service is temporarily unavailable
	at com.apigee.messaging.adaptors.http.flow.data.LBTargetRequestSender.sendRequest(LBTargetRequestSender.java:299)
	at com.apigee.messaging.adaptors.http.flow.data.LBTargetRequestSender.access$400(LBTargetRequestSender.java:57)
	…<snipped>

These error messages indicate that the request could not be sent to the backend server due to a failure. As a result, the Message Processor sends out 503 Service Unavailable with error code NoActiveTargets as its response to the client.

Cause: Connection timeout

Diagnosis

1. Determine the message id of the failing request.
2. Search for the message id in the Message Processor log (/opt/apigee/var/log/edge-message-processor/logs/system.log).
3. You will see the common error messages corresponding to the message id. However, to get the actual cause for the health check failures, scroll above these common error messages and check for any HEALTH MONITOR Errors.

   For example, The following HEALTH MONITOR error message indicates that Message Processor failed with connection timed out error when making the health check API request:

   Apigee-Timer-6 ERROR SERVICES.HEALTH_MONITOR - HTTPMonitor.getResponseFromCache() : Error sending request Request URL : https://<BackendServer-Hostname>:443/status
   java.net.ConnectException: Connection timed out (Connection timed out)
   	at java.net.PlainSocketImpl.socketConnect(Native Method)
   	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   …<snipped>
           

   If this error repeats for MaxFailure number of times configured in the Health Monitor, then you will see a warning message like this:

   Apigee-Timer-7 WARN  ADAPTORS.HTTP.FLOW - LBServer.incrementFailureCount() : Max failure count(10) reached for server : mocktarget2{Environment=<orgname>__prod,Application=mocktargetapigee__1,Target=default}
           

   Read the information given in the warning message carefully. Ensure that the MaxFailure count has been reached for a target server used in the specific API Proxy for which you are experiencing the 503 response code with error code NoActiveTargets.

4. In the example above, the health check failed with the connection timed out error. Check if you are able to connect to the specific backend server directly from each of the Message Processors using the telnet command:

telnet <BackendServer-HostName> 443
      

5. If you are able to connect to the backend server, then you might see a message like Connected to backend-server. Then the problem might be a temporary issue and it might be either resolved or is an intermittent issue. Repeat step 4 a few times (10+ times) and verify the output.
1. If there are no errors with the telnet command consistently, then the issue is resolved. Recheck if the health check failures have stopped. If yes, then you don’t have to do anything further.
2. If you are unable to connect to the backend server with the telnet command intermittently, then there might be a network issue or your backend server might be busy.
6. If you are unable to connect to the backend server with the telnet command consistently, then this might be because the traffic is not allowed from the Message Processors on the specific backend server.

Resolution

If the connection timed out error is consistently observed, then ensure the backend server does not have any firewall restrictions and allows the traffic from the Apigee Edge Message Processors. For example, on Linux, you could use iptables to allow the traffic from the Message Processor's IP addresses on the backend server.

If the problem persists, work with your Network administrator to determine and fix the issue. If you need any further assistance from Apigee, contact Apigee Support.

Cause: Secure request on non-secure port

Diagnosis

1. Determine the message id of the failing request.
2. Search for the message id in the Message Processor log (/opt/apigee/var/log/edge-message-processor/logs/system.log).
3. You will see the common error messages corresponding to the message id. However, to get the actual cause for the health check failures, scroll above these common error messages and check for any HEALTH MONITOR Errors.

   For example, you may see a HEALTH MONITOR error as shown below:

   Apigee-Timer-1 ERROR SERVICES.HEALTH_MONITOR - HTTPMonitor.getResponseFromCache() : Error sending request Request URL : https://mocktarget.apigee.net:80/status
   javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
           at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
           at sun.security.ssl.InputRecord.read(InputRecord.java:527)
           at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
           at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
           at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
           at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
   …<snipped>
           

   If this error repeats for MaxFailure number of times configured in the Health Monitor, then you will see a warning message like this:

   Apigee-Timer-7 WARN  ADAPTORS.HTTP.FLOW - LBServer.incrementFailureCount() : Max failure count(10) reached for server : mocktarget{Environment=<orgname>__prod,Application=mocktargetapigee__1,Target=default}
           

   Read the information given in the warning message carefully. Ensure that the MaxFailure count has been reached for a target server used in the specific API Proxy for which you are experiencing the 503 response code with error code NoActiveTargets.

4. The health check failed with the error:

   Error sending request Request URL : https://mocktarget.apigee.net:80/statuscode/200
   javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
         

   The error message and the URL indicate the cause of this issue is that a secure call (HTTPS) was made on the non-secure port 80.

   This error can occur under the following two scenarios:

   • Secure target server defined with non-secure port
   • Secure target server defined but Health Monitor configured with a non-secure port

   Secure target non-secure port

   Scenario 1: Secure target server defined with non-secure port

   If you have defined a secure target server but with a non-secure port such as 80, then you get this error. Follow the steps below to verify if this is the cause for this issue:

   1. Check the definition of the target server used in the target endpoint configuration.

   Use the Get TargetServer API to get the target server definition.

   Target Server Definition Output

   <TargetServer name="mocktarget">
     <Host>mocktarget.apigee.net</Host>
     <Port>80</Port>
     <IsEnabled>true</IsEnabled>
     <SSLInfo>
         <Enabled>true</Enabled>
     </SSLInfo>
   </TargetServer>
             

   In the example above, the definition shows that the target server mocktarget is a secure server as indicated by the SSLInfo block. However, it is configured with a non-secure Port 80.

   2. Now, check the Health Monitor configuration for the target server in the target endpoint configuration:

      Health Monitor Configuration

      <HealthMonitor>
        <IsEnabled>true</IsEnabled>
        <IntervalInSec>5</IntervalInSec>
        <HTTPMonitor>
          <Request>
            <ConnectTimeoutInSec>10</ConnectTimeoutInSec>
            <SocketReadTimeoutInSec>30</SocketReadTimeoutInSec>
            <Verb>GET</Verb>
            <Path>/statuscode/200</Path>
          </Request>
          <SuccessResponse>
            <ResponseCode>200</ResponseCode>
          </SuccessResponse>
        </HTTPMonitor>
      </HealthMonitor>
                

      Notice that there is no <Port> element specified in the Health Monitor configuration above. In this case, the Edge’s Message Processor uses the port specified in the target server definition (which is 80) for making health check API calls.

   3. Based on the information above, the cause for this error is that the target server is defined as a secure server (as SSLInfo block is enabled), but with a non-secure port 80.

   Secure target non-secure HM port

   Scenario 2: Secure target server defined but Health Monitor configured with a non-secure port

   If you have defined a secure target server but the Health Monitor is configured with a non-secure port such as 80, then you get this error. Follow the steps below to verify if this is the cause for this issue:

   1. Check the definition of the target server used in the target endpoint configuration.

      Use the Get TargetServer API to get the target server definition.

      Target Server Definition Output

      <TargetServer name="mocktarget">
        <Host>mocktarget.apigee.net</Host>
        <Port>443</Port>
        <IsEnabled>true</IsEnabled>
        <SSLInfo>
            <Enabled>true</Enabled>
        </SSLInfo>
      </TargetServer>
              

      In the example above, the definition shows that the target server mocktarget is a secure server as indicated by the SSLInfo block.

   2. Next, check the Health Monitor configuration for the target server in the target endpoint configuration:

      Health Monitor Configuration

      <HealthMonitor>
        <IsEnabled>true</IsEnabled>
        <IntervalInSec>5</IntervalInSec>
        <HTTPMonitor>
          <Request>
            <ConnectTimeoutInSec>10</ConnectTimeoutInSec>
         	<SocketReadTimeoutInSec>30</SocketReadTimeoutInSec>
            <Port>80</Port>
            <Verb>GET</Verb>
            <Path>/statuscode/200</Path>
          </Request>
          <SuccessResponse>
            <ResponseCode>200</ResponseCode>
          </SuccessResponse>
        </HTTPMonitor>
              

      In the example above, the Health Monitor is configured with a non-secure Port 80 as indicated by <Port> element.

   3. Based on the information above, the cause for this error is that the target server is defined as a secure server (as SSLInfo block is enabled) and uses secure port 443, but the Health Monitor is configured to perform health checks with a non-secure port 80 (specified in <Port> element).

      That is, in this case, Edge makes the health check APIs as a secure call with non-secure port 80 and fails with the above-mentioned error.

Resolution

<TargetServer name="mocktarget">
  <Host>mocktarget.apigee.net</Host>
  <Port>443</Port>
  <IsEnabled>true</IsEnabled>
  <SSLInfo>
      <Enabled>true</Enabled>
  </SSLInfo>
</TargetServer>
    

Cause: Non-secure request on a secure port

Diagnosis

1. Determine the message id of the failing request.
2. Search for the message id in the Message Processor log (/opt/apigee/var/log/edge-message-processor/logs/system.log).
3. You will see the common error messages corresponding to the message id. However, to get the actual cause for the health check failures, scroll above these common error messages and check for any HEALTH MONITOR Errors.

   For example, you may see a HEALTH MONITOR error as shown below:

   Apigee-Timer-2 ERROR SERVICES.HEALTH_MONITOR - HTTPMonitor.getResponseFromCache() : Error sending request Request URL : http://mocktarget.apigee.net:443/status
   java.net.SocketException: Unexpected end of file from server
   	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:851)
   	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
   	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:848)
   	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
   	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
   …<snipped>
             

   If this error repeats for MaxFailure number of times configured in the Health Monitor, then you will see a warning message like this:

   Apigee-Timer-7 WARN  ADAPTORS.HTTP.FLOW - LBServer.incrementFailureCount() : Max failure count(10) reached for server : mocktarget{Environment=<orgname>__prod,Application=mocktargetapigee__1,Target=default}
             

   Read the information given in the warning message carefully. Ensure that the MaxFailure count has been reached for a target server used in the specific API Proxy for which you are experiencing the 503 response code with error code NoActiveTargets.

4. The health check failed with the error:

   Error sending request Request URL : http://mocktarget.apigee.net:443/status
   java.net.SocketException: Unexpected end of file from server
         

   The error message and the URL indicate the cause of this issue is that a non-secure call (HTTP) was made on the secure port 443.

   This error can occur under the following two scenarios:

   • Non-secure target server defined with secure port
   • Non-secure target server defined but Health Monitor configured with a secure port

   Non-secure target secure port

   Scenario 1: Non-secure target server defined with secure port

   If you have defined a non-secure target server but with a secure port such as 443, then you get this error. Follow the steps below to verify if this is the cause for this issue:

   1. Check the definition of the target server used in the target endpoint configuration.

      Use the Get TargetServer API to get the target server definition.

      Target Server Definition Output

      <TargetServer name="mocktarget">
        <Host>mocktarget.apigee.net</Host>
        <Port>443</Port>
        <IsEnabled>true</IsEnabled>
      </TargetServer>
                    

      In the example above, the definition shows that the target server mocktarget is a non-secure server as there is no SSLInfo block. However, it is incorrectly configured with a secure Port 443.

   2. Now, check the Health Monitor configuration for the target server in the target endpoint configuration:

      Health Monitor Configuration

      <HealthMonitor>
        <IsEnabled>true</IsEnabled>
        <IntervalInSec>5</IntervalInSec>
        <HTTPMonitor>
          <Request>
            <ConnectTimeoutInSec>10</ConnectTimeoutInSec>
            <SocketReadTimeoutInSec>30</SocketReadTimeoutInSec>
            <Verb>GET</Verb>
            <Path>/statuscode/200</Path>
          </Request>
          <SuccessResponse>
            <ResponseCode>200</ResponseCode>
          </SuccessResponse>
        </HTTPMonitor>
      </HealthMonitor>
                      

      Notice that there is no <Port> element specified in the Health Monitor configuration above. In this case, the Edge’s Message Processor will use the port specified in the target server definition which is 443.

   3. Based on the information above, the cause for this error is that the target server is defined as a non-secure server (as SSLInfo block is not defined), but with a secure port 443.

      That is, Edge makes the health checks as a non-secure call with secure port 443 and fails with the above-mentioned error.

   Non-secure target secure HM port

   Scenario 2: Non-secure target server defined but Health Monitor configured with a secure port

   If you have defined a non-secure target server but the Health Monitor is configured with a secure port such as 443, then you get this error. Follow the steps below to verify if this is the cause for this issue:

   1. Check the definition of the target server used in the target endpoint configuration.

      Use the Get TargetServer API to get the target server definition.

      Target Server Definition Output

      <TargetServer name="mocktarget">
        <Host>mocktarget.apigee.net</Host>
        <Port>80</Port>
        <IsEnabled>true</IsEnabled>
      </TargetServer>
              

      In the example above, the definition shows that the target server mocktarget is a non-secure server (as there is no SSLInfo block) configured with a non-secure port 80 correctly.

   2. Next, check the Health Monitor configuration for the target server in the target endpoint configuration:

      Health Monitor Configuration

      <HealthMonitor>
        <IsEnabled>true</IsEnabled>
        <IntervalInSec>5</IntervalInSec>
        <HTTPMonitor>
          <Request>
            <ConnectTimeoutInSec>10</ConnectTimeoutInSec>
         	<SocketReadTimeoutInSec>30</SocketReadTimeoutInSec>
            <Port>443</Port>
            <Verb>GET</Verb>
            <Path>/statuscode/200</Path>
          </Request>
          <SuccessResponse>
            <ResponseCode>200</ResponseCode>
          </SuccessResponse>
        </HTTPMonitor>
      </HealthMonitor>
            

      In the example above, the Health Monitor is configured with a secure Port 443 as indicated by <Port> element.

   3. Based on the information above, the cause for this error is that the target server is defined as a non-secure server (as SSLInfo block is not defined) with non-secure port 80 correctly, but the Health Monitor is configured to perform health checks with a secure port 443 (specified in <Port> element).

      That is, in this case, Edge makes the health checks as a non-secure call with secure port 443 and fails with the above-mentioned error.

Resolution

<TargetServer name="mocktarget">
  <Host>mocktarget.apigee.net</Host>
  <Port>80</Port>
  <IsEnabled>true</IsEnabled>
</TargetServer>
              

Cause: Health check API responds with an error

Diagnosis

1. Determine the message id of the failing request.
2. Search for the message id in the Message Processor log (/opt/apigee/var/log/edge-message-processor/logs/system.log).
3. You will see the common error messages corresponding to the message id. However, to get the actual cause for the health check failures, scroll above these common error messages and check for any HEALTH MONITOR Errors/Warnings.

   For example, you may see a HEALTH MONITOR warning as shown below:

   Apigee-Timer-7 INFO  SERVICES.HEALTH_MONITOR - HTTPMonitor.sendRequest() : HTTPMonitor.monitor() : Connecting to https://mocktarget.apigee.net:443/status/200
   Apigee-Timer-7 WARN  SERVICES.HEALTH_MONITOR - HTTPMonitor.monitor() : HTTP response code from health monitoring service does not match.Expected response code : [200]. Received response code : 404
           

   If this error repeats for MaxFailure number of times configured in the Health Monitor, then you will see a warning message like this:

   Apigee-Timer-7 WARN  ADAPTORS.HTTP.FLOW - LBServer.incrementFailureCount() : Max failure count(10) reached for server : mocktarget{Environment=<orgname>__prod,Application=mocktargetapigee__1,Target=default}
           

   Read the information given in the warning message carefully. Ensure that the MaxFailure count has been reached for a target server used in the specific API Proxy for which you are experiencing the 503 response code with error code NoActiveTargets.

4. The health check returned the warning message:

   HTTP response code from health monitoring service does not match.Expected response code : [200]. Received response code : 404
         

   The warning message above states that the expected response code for the health check API was 200, but the actual response received is 404. Hence, this is treated as a failure.

5. Before investigating the cause for error response from the health check API, determine why Edge expects the response code as 200 for the health check API. For this, check the Health Monitor configuration for the target server in the target endpoint configuration:

   Health Monitor Configuration

   <HealthMonitor>
     <IsEnabled>true</IsEnabled>
     <IntervalInSec>5</IntervalInSec>
     <HTTPMonitor>
       <Request>
         <ConnectTimeoutInSec>10</ConnectTimeoutInSec>
      	<SocketReadTimeoutInSec>30</SocketReadTimeoutInSec>
         <Port>443</Port>
         <Verb>GET</Verb>
         <Path>/status/200</Path>
       </Request>
       <SuccessResponse>
         <ResponseCode>200</ResponseCode>
       </SuccessResponse>
     </HTTPMonitor>
   </HealthMonitor>
           

   Notice that the Health Monitor configuration is configured with the 200 response code under the <SuccessResponse> element. This means that if Edge gets any response code (such as 400, 401, 404, 500) other than 200 from the health check API, it will be treated as an error and increments the failure count.

6. Now, to investigate the cause for error response from the health check API follow the steps below:
1. Look at the message prior to the warning message in the Message Processor log.

   Apigee-Timer-7 INFO  SERVICES.HEALTH_MONITOR - HTTPMonitor.sendRequest() : HTTPMonitor.monitor() : Connecting to https://mocktarget.apigee.net:443/status/200
             

   Make a note of the health check URL from this message.

2. You can make a direct call to this URL from the Message Processor and check the actual response

   curl -i https://mocktarget.apigee.net:443/status/200
             

   The response from the call above gives 404 as seen in the Message Processor logs:

   < HTTP/2 404
             

3. This shows that, even the direct call to the health check URL fails with the same response code 404. This means that the health check URL may be incorrect or the resource being accessed as part of the URL is no longer available.
4. In the example health check API provided above, the issue occurs because an incorrect URL was used in the Health Monitor configuration. The correct URL was found to be https://mocktarget.apigee.net:443/statuscode/200 from Mock Target API.
7. If you get any other error response, then determine the cause for the same by following the steps above. If required, work with your backend team.

Resolution

1. Fix the issue with the health check API on your backend server.
2. To fix the issue in the example discussed above:
1. Modify the <Path> element in the Health Monitor configuration to /statuscode/200 as shown below:

   <Path>/statuscode/200</Path>
           

2. Save the changes in the API Proxy.

If the problem still persists, go to Must Gather Diagnostic Information.

Diagnose issues using API monitoring

API Monitoring enables you to isolate problem areas quickly to diagnose error, performance, and latency issues and their source, such as developer apps, API proxies, backend targets, or the API platform.

Step through a sample scenario that demonstrates how to troubleshoot 5xx issues with your APIs using API Monitoring. For example, you may want to set up an alert to be notified when the number of messaging.adaptors.http.flow.NoActiveTargets faults exceeds a particular threshold.

Must gather diagnostic information

If the problem persists even after following the instructions above, please gather the following diagnostic information. Contact and share them to Apigee Support:

1. If you are a Public Cloud user, provide the following information:
1. Organization Name
2. Environment Name
3. API Proxy Name
4. Complete curl command to reproduce the error
5. Trace file containing the requests with 503 Service Unavailable with error code NoActiveTargets
2. If you are a Private Cloud user, provide the following information:
   1. Complete Error Message observed
   2. Environment Name
   3. API Proxy bundle
   4. Trace file containing the requests with 503 Service Unavailable with error code NoActiveTargets
   5. NGINX Access Logs

      (/opt/apigee/var/log/edge-router/nginx/<org>~<env>.<port#>_access_log)

   6. Message Processor Logs

      (/opt/apigee/var/log/edge-message-processor/logs/system.log)