503 Service Unavailable - Backend Server

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Videos

Watch the following video to learn more about solving 503 Service Unavailable Errors.

Video Description
503 Service Unavailable Error from Backend Server Learn about the following:
  • Introduction to 503 Service Unavailable Error in Apigee Edge
  • Troubleshooting and resolving a real-time 503 Service Unavailable from Backend Server

Symptom

The client application receives an HTTP response status 503 with the message Service Unavailable following an API proxy call.

Error messages

You can see one of the following error messages:

HTTP/1.1 503 Service Unavailable
 
HTTP/1.1 503 Service Unavailable: Back-end server is at capacity

You might also see an error message like the following in the HTTP response:

The server is temporarily unable to service your request due to
maintenance downtime or capacity problems. Please try again later.

Note: The above response code and error message are only examples. In some cases, you may receive only the error response code without any error message. The format and the content of the error response code and the error message may vary depending on the backend server implementation.

Causes

The HTTP status code 503 means that the server is currently unable to handle the incoming requests. Usually, this error occurs because the server is too busy or is temporarily down for maintenance.

Possible causes for the 503 Service Unavailable response are:

Cause Description Who can perform the troubleshooting steps
Overloaded Server The backend server is overloaded or beyond its capacity and cannot handle any new incoming client requests. Edge Public and Private Cloud users
Server under maintenance The backend server may be under maintenance temporarily. Edge Public and Private Cloud users

Cause: Overloaded Server/Server under Maintenance

In Apigee Edge, the 503 Service Unavailable Error can be returned from a backend server under either of the following circumstances:

  • A backend server is overloaded/busy and cannot handle any new requests.
  • The backend server is down for a temporary period due to maintenance.

Diagnosis

To diagnose the error, you can use any of the following three methods:

  • Trace tool
  • NGINX access logs
  • Direct call to backend server

Click the tabs below to learn about each method.

Trace tool

  1. Enable the trace session, and make the API call to reproduce the issue - 503 Service Unavailable.
  2. Select one of the failing requests and examine the trace.
  3. Navigate through various phases of the trace and locate where the failure occurred.
  4. If you find that the 503 error is returned as a response from the target server, the cause for 503 error is the target server.

    Here’s a sample trace screenshot showing 503 Service Unavailable response received from the target server:

  5. Click the Response received from target server phase and go through the Response Headers and Response Content sections to see if they have any useful information:
    • The Response Headers may contain the Server header, which indicates where the error response was sent from.
    • The Response Content may contain additional information about why the target server sent the 503 response code.
  6. Confirm that the 503 error is coming from the target server by checking the values of X-Apigee-fault-source and X-Apigee-fault-code in the AX (Analytics Data Recorded) Phase in the trace using the steps given below:
    1. Click on AX (Analytics Data Recorded) Phase as shown in the screenshot below:
    2. Scroll down the Phase Details to the Response Headers section and determine the values of X-Apigee-fault-code and X-Apigee-fault-source as shown below:
    3. If the values of X-Apigee-fault-source and X-Apigee-fault-code match the values shown in the below table, you can confirm that the 503 error is coming from the target server:
      Response Headers Value
      X-Apigee-fault-source target
      X-Apigee-fault-code messaging.adaptors.http.flow.ErrorResponseCode
  7. Check if you are using proxy chaining i.e. if the target server/target endpoint is invoking another proxy in Apigee. To determine this:
    1. Navigate back to the Request sent to target server phase and click the Show Curl button and determine the target server host alias.
    2. If the target server host alias is pointing to a virtual host alias, then it's proxy chaining. In this case, you need to repeat all the above steps for the chained proxy till you determine what is actually causing the 503 Service Unavailable error. In these cases 503 Service Unavailable may happen in other chained proxies at other stages as well which can be diagnosed using this playbook.
    3. If the target server host alias points to your backend server, then go to Resolution.

NGINX access logs

You can also refer to NGINX lccess logs to determine whether the 503 status code was sent by the backend server. This is particularly useful if the issue has occurred in the past or if the issue is intermittent and you are unable to capture the trace in UI. Use the following steps to determine this information from NGINX access logs:

  1. Check the NGINX access logs. 
    /opt/apigee/var/log/edge-router/nginx/<org>~<env>.<port#>_access_log
  2. Search for any 503 Errors for the specific API proxy during a specific duration (if the problem happened in the past) or for any requests still failing with 503.
  3. If there are any 503 Errors, then check if the error is coming from the backend server. If the values of X-Apigee-fault-source and X-Apigee-fault-code match the values shown in the table below, the 503 error is coming from the backend server:
    Response Headers Value
    X-Apigee-fault-source target
    X-Apigee-fault-code messaging.adaptors.http.flow.ErrorResponseCode

    Here's a sample entry showing the 503 error caused by the target server:

  4. Review the specific API Proxy and ensure that you are using proxy chaining i.e., if the target server/target endpoint is not invoking another proxy in Apigee. If you are using proxy chaining, you need to repeat all the above steps for the chained proxy until you determine what is actually causing the 503 Service Unavailable error. In these cases, 503 Service Unavailable may happen in other chained proxies at other stages as well, which you can diagnose using this playbook.
  5. If you confirm that you are not using proxy chaining, and the 503 error is coming from your backend server, then go to Resolution.

Call to Backend Server

You can make a direct call to the backend server and verify that you are getting the same 503 Service Unavailable response as received when the request was made through Apigee Edge.

  1. Ensure that you have all the required headers, query parameters and any credentials that need to be passed to the backend server as part of the request.
  2. If the backend service is publicly accessible, you can use the curl command, Postman or any other REST Client and invoke the backend server API directly.
  3. If the backend server is accessible only from the Message Processors, you can use the curl command, Postman or any other REST Client and invoke the backend server API directly from the Message Processor.
  4. Verify that the backend service is indeed returning 503 Service Unavailable error.

Resolution

If you ascertain that the 503 error is coming from the backend server, you can do the following to resolve the issue:

  • If the issue is caused because the backend server is down for maintenance, you can bring the backend server online after the maintenance period.
  • If the issue is caused because the backend server is overloaded, then fix the issue if you have access to the backend server. Otherwise you may need to work with your backend server team to fix the issue.

Diagnose Issues using API Monitoring

API Monitoring enables you to isolate problem areas quickly to diagnose error, performance, and latency issues and their source, such as developer apps, API proxies, backend targets, or the API platform.

Step through a sample scenario that demonstrates how to troubleshoot 5xx issues with your APIs using API Monitoring. For example, you may want to set up an alert to be notified when the number of messaging.adaptors.http.flow.ErrorResponseCode faults exceeds a particular threshold.

Must gather diagnostic information

If the problem persists even after following the above instructions, please gather the following diagnostic information, and then contact Apigee Support.

If you are a public cloud user, provide the following information:

  • Organization Name
  • Environment Name
  • API Proxy Name
  • Complete curl command to reproduce the 503 error
  • Trace file containing the requests with 503 Service Unavailable error
  • If the 503 errors are not occurring currently, provide the time period with the timezone information when 503 errors occurred in the past.

If you are a private cloud user, provide the following information:

  • Complete error message observed for the failing requests.
  • Organization, Environment Name and API Proxy Name for which you are observing 503 errors.
  • API Proxy Bundle.
  • Trace file containing the requests with 503 Service Unavailable error.
  • NGINX access logs. 
    /opt/apigee/var/log/edge-router/nginx/<org>~<env>.<port#>_access_log
  • Message Processor logs. 
    /opt/apigee/var/log/edge-message-processor/logs/system.log
  • The time period with the timezone information when the 503 errors occurred.