500 Internal Server Error - Backend Server

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Videos

Video Description
500 Internal Server Error - caused by backend Demonstrates a real-time 500 Internal Server Error caused by the backend server along with steps to troubleshoot and resolve the error.

Symptom

The client application gets an HTTP status code of 500 with the message Internal Server Error as a response for API calls.

The HTTP status code 500 is a generic error response. It means that the server encountered an unexpected condition that prevented it from fulfilling the request. This error is usually returned by the server when no other error code is suitable.

Error messages

Client application gets the following response code: 

HTTP/1.1 500 Internal Server Error

In addition, you may observe an error message similar to the one shown below:

Sample #1

Sample Backend Server Response #1

{"errorMessage":"Sorry either your e-mail or password didn't match.",
"errorParameters":"{}",
"errorCode":"500",
"errorKey":"INVALID_EMAILPASSWORD"}

Sample #2

Sample Backend Server Response #2

<Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
   <Body>
      <Error>
         <code>500</code>
         <message xml:lang="en-US">Not Authorised(e4138fa0-ec57).</message>
      </Error>
   </Body>
</Envelope>

Possible causes

The 500 Internal Server Error could be returned by the backend server due to a number of causes. This playbook explains how to troubleshoot using common steps and resolve this error irrespective of its cause.

The possible causes for this issue are as follows:

Cause Description Troubleshooting instructions applicable for
Error in backend server The backend server may fail for some reason. Edge Private and Public Cloud users

Common diagnosis steps

Use one of the following tools/techniques to diagnose this error:

API Monitoring

Procedure #1: Using API Monitoring

To diagnose the error using API Monitoring:

  1. Sign in to Apigee Edge UI as a user with an appropriate role.

  2. Switch to the organization in which you want to investigate the issue.

  3. Navigate to the Analyze > API Monitoring > Investigate page.
  4. Select the specific timeframe in which you observed the errors.

  5. Plot Fault Code against Time.

  6. Select a cell which has the fault code messaging.adaptors.http.flow.ErrorResponseCode as shown below:

    ( view larger image)

  7. Information about the fault code messaging.adaptors.http.flow.ErrorResponseCode is displayed as shown below:

    ( view larger image)

  8. Click View logs and expand the row for the failed request.

    ( view larger image)

  9. From the Logs window, note the following details:
    • Request Message ID
    • Status Code: 500
    • Fault Source: target
    • Fault Code: messaging.adaptors.http.flow.ErrorResponseCode

Trace

Procedure #2: Using Trace tool

To diagnose the error using the Trace tool:

  1. Enable the trace session and either
    • Wait for the 500 Internal Server Error error with error code messaging.adaptors.http.flow.ErrorResponseCode to occur, or
    • If you can reproduce the issue, make the API call to reproduce the issue 500 Internal Server Error

  2. Ensure Show all FlowInfos is enabled:

  3. Select one of the failing requests and examine the trace.
  4. Navigate through different phases of the trace and locate where the failure occurred.

  5. You will find the error typically in a flow after the Response received from target server phase as shown below:

    ( view larger image)

  6. Navigate to the AX (Analytics Data Recorded) Phase in the trace and click it.

  7. Scroll down to the Phase Details Response Headers section and determine the values of X-Apigee-fault-code and X-Apigee-fault-source, and X-Apigee-Message-ID as shown below:

    ( view larger image)

  8. Note the values of X-Apigee-fault-code, X-Apigee-fault-source, and X-Apigee-Message-ID:
    9. Response headers Value
    X-Apigee-fault-code messaging.adaptors.http.flow.ErrorResponseCode
    X-Apigee-fault-source target
    X-Apigee-Message-ID MESSAGE_ID

NGINX

Procedure #3: Using NGINX access logs

To diagnose the error using NGINX access logs:

  1. If you are a Private Cloud user, then you can use NGINX access logs to determine the key information about HTTP 500 Internal Server Error.

  2. Check the NGINX access logs:

    /opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log

  3. Search to see if there are any 500 errors with error code messaging.adaptors.http.flow.ErrorResponseCode during a specific duration (if the problem happened in the past) or if there are any requests still failing with 500.

  4. If you do find any 500 errors with the X-Apigee-fault-code matching the value of messaging.adaptors.http.flow.ErrorResponseCode, then determine the value of the X-Apigee-fault-source.

    Sample 500 error from NGINX access log:

    ( view larger image)

    The above sample entry from NGINX access log has the following values for X-Apigee-fault-code and X-Apigee-fault-source:

    Headers Value
    X-Apigee-fault-code messaging.adaptors.http.flow.ErrorResponseCode
    X-Apigee-fault-source target

Cause: Error in backend server

Diagnosis

The 500 Internal Server Error responded by the backend server can be caused by a number of reasons. You will need to diagnose each situation independently.

  1. Determine the Fault Code, Fault Source for the error observed using API Monitoring, Trace tool, or NGINX access logs as explained in Common diagnosis steps.
  2. If the Fault Source is target and Fault Code is messaging.adaptors.http.flow.ErrorResponseCode, then that indicates that the error is returned by the backend server.
  3. You can use one of the following steps to diagnose the cause of the issue:

    Trace

    Using Trace:

    If you have a Trace session for the failure, then perform the following steps:

    1. In the Trace, select the API request that has failed with 500 Internal Server Error.

    2. Select the Response received from target server phase from the failing API request as shown in the figure below:

      ( view larger image)

    3. Scroll down to the Phase Details section and check the Response Content, which contains the response from the backend server.

      Sample Response Content:

      <Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
   <Body>
      <Error>
         <code>500</code>
         <message xml:lang="en-US">Not Authorised(e4138fa0-ec57).</message>
      </Error>
   </Body>
</Envelope>

      In the above response, note that the error message from the backend server is Not Authorised. This indicates that the user may have passed invalid credentials and that is why they are getting this error.

    Call backend server

    Making direct call to backend server:

    You can make a direct call to the backend server and:

    • Validate if you are getting the same 500 Internal Server Error response as received when the request was made through Apigee Edge
    • Check the error message (response) received from the backend server

    Perform the following steps to make the direct call to the backend server:

    1. Ensure that you have all the required headers, query parameters, and any credentials that need to be passed to the backend server as part of the request.
    2. If the backend service is publicly accessible, then you can use the curl command, Postman, or any other REST Client and invoke the backend server API directly.

    3. If the backend server is accessible only from the Message Processors, then you can use the curl command, Postman, or any other REST Client and invoke the backend server API directly from the Message Processor.

    4. Validate if the backend service is indeed returning 500 Internal Server Error and check the error message (response) returned by the backend server and determine the cause for this error.

    Backend server logs

    Using backend server logs

    1. Review the backend server logs and try to get more details about the error and its cause.
    2. If possible, enable the debug mode on the backend server to get more details about the error and the cause.

  4. Check to see if you are using proxy chaining in the specific target endpoint of the failing API Proxy; that is, if the target server/target endpoint is invoking another proxy in Apigee Edge. To determine this:

    1. If you have the trace for the failing request, navigate to the Request sent to target server phase and click Show Curl.

    2. The Curl for Request Sent to Target Server window opens from which you can determine the target server host alias.
    3. Review the target endpoint of your API Proxy and check if the backend server URL or the host name in the target server points to another Proxy or your own backend server.
    4. If the target server host alias is pointing to a virtual host alias, then it is proxy chaining. In this case, you need to repeat all the above steps for the chained proxy until you determine what is actually causing the 500 Internal Server Error. In these cases 500 Internal Server Error may happen in other chained proxies at other stages as well which can be diagnosed and resolved using the instructions given in this playbook or in the 500 Internal Server Error playbook.
    5. If the target server host alias points to your backend server, then go to Resolution.

Resolution

If it is ascertained that the 500 error is coming from the backend server, then work with your backend server team to fix the issue appropriately.

In the example discussed above, you may have to request the users pass valid credentials to fix this issue.

Key points to note

  1. The actual error message returned by the backend server for 500 Internal Server Error can be viewed only if you have captured the trace session for the failing requests.
  2. The backend server response will not be logged in API Monitoring, NGINX Access Logs, or Message Processor logs for security reasons.
  3. You can review the backend server logs or enable debug mode on the backend to get more details about the 500 Internal Server Error and/or view the error message returned by the backend server.

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information and contact Apigee Edge Support.

If you are a Public Cloud user, provide the following information:

  • Organization name
  • Environment name
  • API Proxy name
  • Complete curl command to reproduce the 500 error
  • Trace file containing the requests with 500 Internal Server Error
  • If the 500 errors are not occurring currently then provide the time period with the timezone information when 500 errors occurred in the past.

If you are a Private Cloud user, provide the following information:

  • Complete error message observed for the failing requests
  • Organization, environment name, and API proxy name for which you are observing 500 errors
  • API proxy bundle
  • Trace file containing the requests with 500 Internal Server Error
  • NGINX access logs /opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log

    Where: ORG, ENV, and PORT# are replaced with actual values.

  • Message Processor system logs /opt/apigee/var/log/edge-message-processor/logs/system.log
  • The time period with the timezone information when the 500 errors occurred.