502 Bad Gateway - DuplicateHeader

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Symptom

The client application gets an HTTP status code of 502 Bad Gateway with error code protocol.http.DuplicateHeader as a response for API calls.

Error message

Client application gets the following response code: 

HTTP/1.1 502 Bad Gateway

In addition, you may observe an error message similar to the one shown below: 

{
   "fault":{
      "faultstring":"Duplicate Header \"Expires\"",
      "detail":{
         "errorcode":"protocol.http.DuplicateHeader"
      }
   }
}

Possible Causes

This error occurs if a specific HTTP header that is not allowed to have duplicates in Apigee Edge, appears more than once with same or different values, as part of the HTTP response sent by the backend server to Apigee Edge.

As per RFC 7230, section 3.2.2: Field Order, a sender MUST NOT generate multiple header fields with the same field name in a message unless either the entire field value for that header field is defined as a comma-separated list, [i.e., #(values)] or the header field is a well-known exception. If Apigee Edge finds that a specific same header, that is not allowed to have duplicates, is sent more than once in the HTTP response sent by the target/backend server, then it responds with 502 Bad Gateway and error code protocol.http.DuplicateHeader

Here are the possible causes for this error:

Cause Description Troubleshooting instructions applicable for
Duplicate header in response The response from the backend server contains duplicate headers. Edge Public and Private Cloud users

Common diagnosis steps

Use one of the following tools/techniques to diagnose this error:

API Monitoring

To diagnose the error using API Monitoring:

  1. Sign in to Apigee Edge UI as a user with an appropriate role.

  2. Switch to the organization in which you want to investigate the issue.

  3. Navigate to the Analyze > API Monitoring > Investigate page.
  4. Select the specific timeframe in which you observed the errors.
  5. Ensure that the Proxy filter is set to All.
  6. Plot Fault Code against Time.

  7. Select a cell which has the fault code protocol.http.DuplicateHeader as shown below:

    (view larger image)

  8. Information about the fault code protocol.http.DuplicateHeader is displayed as shown below:

    (view larger image)

  9. Ensure that the Status Code is 502 as shown in the above example.
  10. Click View logs and expand the row for the failed request.

  11. From the Logs window, note the following details:

    • Status Code: 502
    • Fault Source: target
    • Fault Code: protocol.http.DuplicateHeader.
  12. The Fault Source is target, which indicates that the response from the backend server contained duplicate headers.

Trace tool

To diagnose the error using the Trace tool:

  1. Enable the trace session and either
    1. Wait for the 502 Bad Gateway error to occur or
    2. If you can reproduce the issue, make the API call and reproduce the 502 Bad Gateway error

  2. Ensure Show all Flow Infos is enabled:

  3. Select one of the failing requests and examine the trace.
  4. Navigate through different phases of the trace and locate where the failure occurred.

  5. You will find the error typically in a flow after the Request sent to target server phase as shown below:

    (view larger image)

  6. Note the value of the error from the trace.

    The above sample trace shows the error as Duplicate Header "Expires". Since the error is raised by Apigee after the request was sent to the backend server, it indicates that the backend server sent the header Expires more than once.

  7. Navigate to the AX (Analytics Data Recorded) Phase in the trace and click on it.

  8. Scroll down to the Phase Details - Response Headers section and determine the values of X-Apigee-fault-code and X-Apigee-fault-source as shown below:

    (view larger image)

  9. You will see the values of X-Apigee-fault-code and X-Apigee-fault-source as protocol.http.DuplicateHeader and target, indicating that this error is caused because duplicate headers were passed by the backend server for the response header Expires.
    Response Headers Value
    X-Apigee-fault-code protocol.http.DuplicateHeader
    X-Apigee-fault-source target

  10. Check if you are using proxy chaining; that is, if the target server or target endpoint is invoking another proxy in Apigee.

    1. To determine this, navigate back to the Request sent to target server phase. Click Show Curl.

    2. The Curl for Request Sent to Target Server window opens from which you can determine the target server host alias.

    3. If the target server host alias is pointing to a virtual host alias, then it is proxy chaining. In this case, you need to repeat all the above steps for the chained proxy until you determine what is actually causing the 502 Bad Gateway error.
    4. If the target server host alias points to your backend server, then it indicates that your backend server is sending the duplicate headers in the response to Apigee.

NGINX

To diagnose the error using NGINX access logs:

  1. If you are a Private Cloud user, then you can use NGINX access logs to determine the key information about HTTP 502 errors.

  2. Check the NGINX access logs:

    /opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log

    Where: ORG, ENV and, PORT# are replaced with actual values.

  3. Search to see if there are any 502 errors during a specific duration (if the problem happened in the past) or if there are any requests still failing with 502.

  4. If you do find any 502 errors with the X-Apigee-fault-code matching the value of protocol.http.DuplicateHeader, then determine the value of the X-Apigee-fault-source.

    Sample 502 error from NGINX access log:

    The above sample entry from NGINX Access log has the following values for X- Apigee-fault-code and X-Apigee-fault-source:

    Response Headers Value
    X-Apigee-fault-code protocol.http.DuplicateHeader
    X-Apigee-fault-source target

Cause: Duplicate Header in response

Diagnosis

  1. Determine the Fault Code and Fault Source for the error observed using API Monitoring or NGINX access logs as explained in Common diagnosis steps.
  2. If the Fault Source has the value target, then this indicates that the response sent by the target server contains duplicate headers.

  3. You can determine the actual header that is sent more than once as part of the response using one of the following methods:

    Error message

    Using the error message:

    1. If you have access to the complete error message received from Apigee Edge, then refer to the faultstring. The faultstring contains the header name that has been sent more than once.

      Sample error message:

      "faultstring":"Duplicate Header \"Expires\""
    2. In the above error message, you can see that the header Expires is sent more than once as seen in the faultstring.

    Actual request

    Using the actual request:

    1. If you don’t have access to the actual request made to the target server, then get the corresponding curl command from Using the Trace tool step 10.a and step 10.b.

    2. If you have access to the actual request made to the target server application, then perform the following steps:

      1. Make a call to the target server.

        Sample Request for the target server used in this example:

        curl -X GET "https://BACKEND_SERVER_HOST/response-headers?Expires=Mon%2C%2021%20June%202021%2007%3A28%3A00%20GMT&Expires=Mon%2C%2021%20June%202021%2007%3A28%3A00%20GMT" -v

      2. Verify the list of headers seen in the response.

        Sample Response from the target server used in this example:

        * ...Trimmed...
> GET /response-headers?Expires=Mon%2C%2021%20June%202021%2007%3A28%3A00%20GMT&Expires=Mon%2C%2021%20June%202021%2007%3A28%3A00%20GMT HTTP/2
> Host: BACKEND_SERVER_HOST
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< date: Fri, 02 Jul 2021 05:29:07 GMT
< content-type: application/json
< content-length: 166
< server: gunicorn/19.9.0
< Expires: Mon, 21 June 2021 07:28:00 GMT
< Expires: Mon, 21 June 2021 07:28:00 GMT
< access-control-allow-origin: *
< access-control-allow-credentials: true
<
----<Response BODY>------
* Connection #0 to host httpbin.org left intact
* Closing connection 0

        In the above example request, the header Expires is sent more than once. Therefore, this request fails with the 502 Bad Gateway error and the error code: protocol.http.DuplicateHeader.

      3. If the header whose name appears in the faultstring appears more than once in the backend server’s response, then that’s the cause for this error. In the above case, the header Expires is sent more than once.

Resolution

Fix duplication

Option #1 [Recommended Option] Fix the backend server to not include duplicate headers

  1. Analyze the reason for the specific backend server to send duplicate header Expires and verify whether it is okay for the API proxies to accept that. In most cases, it will not be desirable as per HTTP specification RFC7230.
  2. If it is not desirable, modify your target server application not to send duplicate headers. In the example discussed above, it is noticed that header Expires is sent twice with the same value, which is not desirable. You can fix the issue by making sure that the target server passes the Expires header only once.
  3. If it is desirable and you want to allow the duplicate headers, go to Option #2 Using CwC property.

CwC

Option #2 Using CwC property

Apigee provides a CwC property HTTPHeader.<HeaderName> ,which allows client applications and target servers to send duplicate headers to API proxies in Apigee Edge.

CwC property Values
HTTPHeader.<HeaderName> allowDuplicates,multivalued

For example, the following property can be set on the Message Processors to allow duplicates and multiple values for the header Expires. 

HTTPHeader.Expires=allowDuplicates, multiValued
  1. If you are a Private Cloud user, you can configure the property to prevent Apigee Edge from raising a 502 Bad Gateway error, even if the request contains duplicate headers using the Configuring Message Processors to use duplicate headers how-to guide.
  2. If you are a Public Cloud user, contact Apigee Edge Support to configure this property for your organization.

Specification

Apigee responds with the 502 Bad Gateway error response as it expects that the backend server would behave according to the following RFC specifications:

Specification
RFC 7230, section 3.2.2: Field Order
RFC 7230, section 3.2: Header Fields

If you still need any assistance from Apigee Support, go to Must gather diagnostic information.

Must gather diagnostic information

Gather the following diagnostic information, and then contact Apigee Edge Support.

If you are a Public Cloud user, provide the following information:

  • Organization name
  • Environment name
  • API Proxy name
  • Complete curl command used to reproduce the 502 error
  • Trace file for the API requests

If you are a Private Cloud user, provide the following information:

  • Complete error message observed for the failing requests
  • Environment name
  • API proxy bundle
  • Trace file for the API requests

  • NGINX access logs:

    /opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log

    Where: ORG, ENV and PORT# are replaced with actual values.

  • Message Processor system logs /opt/apigee/var/log/edge-message-processor/logs/system.log