502 Bad Gateway - DecompressionFailureAtResponse

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation.
info

Symptom

The client application gets an HTTP status code of 502 Bad Gateway with error code messaging.adaptors.http.flow.DecompressionFailureAtResponse as a response to API calls.

Error message

Client application gets the following response code:

HTTP/1.1 502 Bad Gateway

In addition, you may observe an error message similar to the one shown below:

{
   "fault":{
      "faultstring":"Decompression failure at response",
      "detail":{
         "errorcode":"messaging.adaptors.http.flow.DecompressionFailureAtResponse"
      }
   }
}

Possible causes

This error occurs only if:

  • The encoding specified in the HTTP response (from the backend/target server) header Content-Encoding is valid and supported by Apigee Edge,
  • BUT

  • The payload format sent by the backend/target server as part of the HTTP response does not match the encoding format specified in the Content-Encoding header

This is because Apigee Edge fails to decode the payload using the specified encoding since the format of the payload is not in the same format as the encoding specified in the Content-Encoding header.

Here are few examples of supported Content-Encoding values and how Apigee Edge expects the payload representation to be in those cases:

Scenario Content-Encoding Payload representation
Single Encoding gzip

The Unix gzip format.

See RFC1952 GZIP Format.

Single Encoding deflate

This format uses zlib structure with deflate compression algorithm.

See RFC1950 and RFC1951.

Multiple Encoding

Multiple Encoding

For example, in cases when the encoding is done twice, it can be:

  • gzip, deflate
  • gzip, gzip
  • deflate, gzip
  • deflate, deflate
Multiple encoding applied to the payload in the given order as it appears in the header.

The possible causes for this error are as follows:

Cause Description Troubleshooting instructions applicable for
Response payload format does not match the Content-Encoding The format of the response payload sent by the backend/target server is either not encoded or does not match the encoding specified in the Content-Encoding header. Edge Public and Private Cloud users

Common diagnosis steps

Use one of the following tools/techniques to diagnose this error:

API Monitoring

To diagnose the error using API Monitoring:

  1. Sign in to Apigee Edge UI as a user with an appropriate role.
  2. Switch to the organization in which you want to investigate the issue.

  3. Navigate to the Analyze > API Monitoring > Investigate page.
  4. Select the specific timeframe in which you observed the errors.
  5. Ensure that the Proxy filter is set to All.
  6. Plot Fault Code against Time.
  7. Select a cell which has the fault code messaging.adaptors.http.flow.DecompressionFailureAtResponse as shown below:

    ( view larger image)

  8. Information about the fault code messaging.adaptors.http.flow.DecompressionFailureAtResponse is displayed as shown below:

    ( view larger image)

  9. Click View logs and expand the row failing with the 502 error.

    ( view larger image)

  10. From the Logs window, note the following details:
    • Status Code: 502
    • Fault Source: target
    • Fault Code: messaging.adaptors.http.flow.DecompressionFailureAtResponse.
  11. If the Fault Source has the value target, then that indicates that the response payload format did not match the supported encoding specified in the backend server's response header Content-Encoding.

Trace tool

To diagnose the error using the Trace tool:

  1. Enable the trace session and either:
    1. Wait for the 502 Bad Gateway error to occur, or
    2. If you can reproduce the issue, make the API call and reproduce 502 Bad Gateway.
  2. Ensure Show all FlowInfos is enabled:

  3. Select one of the failing responses and examine the trace.
  4. Navigate through different phases of the trace and locate where the failure occurred.
  5. You will typically find the error in a flow right after the Response Received from target server phase as shown below:

    ( view larger image)

  6. Note the values of the properties from the trace:

    • Content-Encoding: gzip
    • Response Content Body: {"fault":{"faultstring":"Decompression failure at response","detail":{"errorcode":"messaging.adaptors.http.flow.DecompressionFailureAtResponse"}}}
  7. Navigate to error phase just after the Response Received from target server phase:

    ( view larger image)

    Note the properties:

    • error: Decompression failure at response
    • error.class: com.apigee.errors.http.server.BadGateway
    • error.cause: Not in GZIP format

      The error.cause states that the response payload is Not in GZIP format. This means that Apigee Edge was expecting the response payload to be in GZIP format as was specified in the Content-Encoding header (determined in previous step).Therefore, Apigee Edge cannot decompress the payload using gzip and returns the error Decompression failure at response.

    Note that the response from the target/backend server is 200 in this case; however, the client application will receive a 502 response since the error is returned by Apigee Edge.

  8. Navigate to the Response Sent to Client phase in the trace and click it.

    ( view larger image)

    Note the following details from the trace:

    • Status code: 502 Bad Gateway.
    • Error Content: {"fault":{"faultstring":"Decompression failure at response","detail":{"errorcode":"messaging.adaptors.http.flow.DecompressionFailureAtResponse"}}}
  9. Navigate to the AX (Analytics Data Recorded) phase in the trace and click it.

  10. Scroll down to the Phase Details, Error Headers section and determine the values of X-Apigee-fault-code and X-Apigee-fault-source as shown below:

    ( view larger image)

  11. You will see the values of X-Apigee-fault-code and X-Apigee-fault-source as messaging.adaptors.http.flow.DecompressionFailureAtResponse and target, indicating that the response payload format did not match the encoding specified in the Content-Encoding header.
    Response headers Value
    X-Apigee-fault-code messaging.adaptors.http.flow.DecompressionFailureAtResponse
    X-Apigee-fault-source target

NGINX

To diagnose the error using NGINX access logs:

  1. If you are a Private Cloud user, then you can use NGINX access logs to determine the key information about HTTP 502 errors.
  2. Check the NGINX access logs:

    /opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log

    Where: ORG, ENV, and PORT# are replaced with actual values.

  3. Search to see if there are any 502 errors during a specific duration (if the problem happened in the past) or if there are any responses still failing with 502.
  4. If you do find any 502 errors with the X-Apigee-fault-code matching the value of messaging.adaptors.http.flow.DecompressionFailureAtResponse, then determine the value of the X-Apigee-fault-source.

    Sample 502 error from NGINX access log:

    The above sample entry from the NGINX access log has the following values for X-Apigee-fault-code and X-Apigee-fault-source:

    Response headers Value
    X-Apigee-fault-code messaging.adaptors.http.flow.DecompressionFailureAtResponse
    X-Apigee-fault-source target

Cause: Response payload format does not match the Content-Encoding

By default, Apigee Edge always decompresses the payload if the response header Content-Encoding contains a valid and a supported encoding. Therefore, it is expected that the format of the response payload should match the encoding specified in the response header Content-Encoding. If there is a mismatch, then you get this error.

Diagnosis

  1. Determine the Fault Code and Fault Source for the error observed using API Monitoring, Trace tool, or NGINX access logs as explained in Common diagnosis steps.
  2. If the Fault Code is messaging.adaptors.http.flow.DecompressionFailureAtResponse and the Fault Source has the value target, then this indicates that the format of the response payload sent by the backend/target server does not match the supported encoding specified in the response header Content-Encoding.
  3. You can determine the mismatch as part of the HTTP response using one of the following methods:

    Error message

    To validate using the error message:

    1. If you have access to the complete error message received from Apigee Edge, then refer to the faultstring.

      Sample error message:

      "faultstring":"Decompression failure at response"
      
    2. In the above error message, it displays "Decompression failure at response" which implies that the response could not be decompressed using the encoding specified in the Content-Encoding header.

    Trace

    To validate using the Trace:

    1. Determine the Content-Type and error.cause using Trace as explained in Common diagnosis steps.
    2. The values from the sample trace are as follows:

      • Content-Encoding: gzip
      • error.cause: Not in GZIP format

      The value in the response header Content-Encoding is gzip; however, the response payload is not in GZIP format (as indicated by error.cause). Therefore, Apigee Edge responds with 502 Bad Gateway and error code messaging.adaptors.http.flow.DecompressionFailureAtResponse.

    Actual request

    To validate using the actual request:

    If you have access to the actual request made to the target/backend server application, then perform the following steps:

    1. If you are a Public Cloud/Private Cloud user, make a request directly to the backend server from the backend server itself or any other machine from where you are allowed to make the request to the backend server.
    2. If you are a Private Cloud user, you can also make the request to the backend server from one of the Message Processors.
    3. Examine the response sent by the backend server and determine the value passed in the response header Content-Encoding.
    4. Determine the format of the payload sent as part of the request.
    5. If the value of Content-Encoding header is in the list of supported encoding but the format of the response payload does not match with the encoding specified in Content-Encoding header, then that is the cause of the issue.

      Sample:

      curl -v https://HOSTALIAS/test
      

      ***trimmed***
      >
      < HTTP/1.1 200 OK
      < Accept-Ranges: bytes
      < Content-Encoding: gzip
      < Date: Mon, 02 Aug 2021 08:17:35 GMT
      < Transfer-Encoding: chunked
      <
      < response_payload.zip Response Body(not in GZIP format)>
      

      The above sample response sends the value gzip to the Content-Encoding header which is a supported encoding in Apigee Edge. However, the response_payload.zip is sent as a zip file. Therefore, this response fails with a 502 Bad Gateway error with the error code: messaging.adaptors.http.flow.DecompressionFailureAtResponse.

    Message Processor logs

    To validate using Message Processor logs:

    If you are a Private Cloud user, then you can use Message Processor logs to determine the key information about HTTP 502 errors.

    1. Check the Message Processor log:

      /opt/apigee/var/log/edge-message-processor/logs/system.log

    2. Search to see if there are any 502 errors during a specific duration (if the problem happened in the past) or if there are any responses still failing with 502. You may use the following search string:

      grep -ri "ZipException"
      
    3. You will find lines from system.log similar to the following:

      Scenario #1

      Scenario #1: When API response has the header Content-Encoding: gzip

      2021-08-02 06:50:25,433  NIOThread@2 ERROR HTTP.CLIENT -
      HTTPClient$Context.onInputException() :  ClientInputChannel(ClientChannel[Connected:
      Remote:3.8.1.1:9000 Local:10.0.115.32:41298]@38140 useCount=1 bytesRead=0
      bytesWritten=203 age=469ms  lastIO=0ms  isOpen=true).onExceptionRead exception: {}
      java.util.zip.ZipException: Not in GZIP format
      ---trimmed--
      2021-08-02 06:50:25,433  NIOThread@2 INFO  HTTP.CLIENT -
      HTTPClient$Context.logContextDetails() : Request details : host=null
      path=/folder/testFile method=GET. Channel details : Bytes read=0
      2021-08-02 06:50:25,434  NIOThread@2 ERROR ADAPTORS.HTTP.FLOW -
      AbstractResponseListener.onException() : AbstractResponseListener.onError(HTTPResponse@4806fdab, Not in GZIP format)
      2021-08-02 06:50:25,434  NIOThread@2 INFO  HTTP.SERVICE -
      ExceptionHandler.handleException() : Exception
      java.util.zip.ZipException: Not in GZIP format
      occurred while writing to channel null
      2021-08-02 06:50:25,434  NIOThread@2 INFO  HTTP.SERVICE -
      ExceptionHandler.handleException() : Exception trace:
      java.util.zip.ZipException: Not in GZIP format
      

      The line java.util.zip.ZipException: Not in GZIP format in the above error message indicates that the response payload is not sent in GZIP format although the Content-Encoding is specified as gzip. Therefore, Apigee Edge throws the exception and returns a 502 status code with fault code messaging.adaptors.http.flow.DecompressionFailureAtResponse to client applications.

      Scenario #2

      Scenario #2: When API response has the header Content-Encoding: deflate

      2021-08-02 06:35:21,215  NIOThread@0 ERROR HTTP.CLIENT -
      HTTPClient$Context.onInputException() :  ClientInputChannel(ClientChannel[Connected:
      Remote:3.8.1.1:9000 Local:192.168.194.140:35224]@36014 useCount=1 bytesRead=0
      bytesWritten=202 age=439ms  lastIO=2ms  isOpen=true).onExceptionRead exception: {}
      java.util.zip.ZipException: incorrect header check
      ---trimmed----
      Caused by:
      java.util.zip.DataFormatException: incorrect header check
      ---trimmed---
      2021-08-02 06:35:21,215  NIOThread@0 INFO  HTTP.CLIENT -
      HTTPClient$Context.logContextDetails() : Request details :
      host=null path=/folder/testFile method=GET. Channel details : Bytes read=0
      2021-08-02 06:35:21,216  NIOThread@0 ERROR ADAPTORS.HTTP.FLOW -
      AbstractResponseListener.onException() : AbstractResponseListener.onError(HTTPResponse@3966e277,
      incorrect header check)
      2021-08-02 06:35:21,216  NIOThread@0 INFO  HTTP.SERVICE -
      ExceptionHandler.handleException() : Exception
      java.util.zip.ZipException: incorrect header check occurred while writing to channel null
      2021-08-02 06:35:21,217  NIOThread@0 INFO  HTTP.SERVICE -
      ExceptionHandler.handleException() : Exception trace:
      java.util.zip.ZipException: incorrect header check
      
      

      The lines java.util.zip.ZipException: incorrect header check and Caused by: java.util.zip.DataFormatException: incorrect header check in the above error message indicate that the response payload is not sent in deflate format and does not match the encoding specified in the Content-Encoding header of deflate. Therefore, Apigee Edge throws the exception and returns a 502 status code with fault code messaging.adaptors.http.flow.DecompressionFailureAtResponse to client applications.

Resolution

  1. If there's no need for the compressed response payload in the API proxy flow in Apigee Edge and in the backend server, then do not pass the header Content-Encoding. If there is a need to compress the response payload, go to step 2.
  2. If there's a need to compress the response payload, then ensure that the backend server always sends the following:
    • Any of the supported encoding as the value for the Content-Encoding header in the response
    • The response payload in the supported format to Apigee Edge matches the encoding format specified in the Content-Encoding header
  3. In the example discussed above, the response payload is in ZIP format but the response header specifies Content-Encoding: gzip. You can fix the issue by sending the response header as Content-Encoding: gzip and the response payload in gzip format:
    curl -v https://HOSTALIAS/v1/test
    
    >
    < HTTP/1.1 200 OK
    < Accept-Ranges: bytes
    < Content-Encoding: gzip
    < Date: Mon, 02 Aug 2021 08:17:35 GMT
    < Transfer-Encoding: chunked
    <
    < response_payload.gz Response Body(in GZIP format)>
    

Specification

Apigee Edge responds with the status code 502 Bad Gateway with error code messaging.adaptors.http.flow.DecompressionFailureAtResponse as per the following RFC specifications:

Specification
RFC 7231, section 6.5.1
RFC 7231, section 3.1.2.2

If you still need any assistance from Apigee Support, go to Must gather diagnostic information.

Must gather diagnostic information

Gather the following diagnostic information, and then contact Apigee Edge Support:

If you are a Public Cloud user, provide the following information:

  • Organization name
  • Environment name
  • API Proxy name
  • Complete curl command used to reproduce the 502 error
  • Trace file for the API responses

If you are a Private Cloud user, provide the following information:

  • Complete error message observed for the failing responses
  • Environment name
  • API Proxy bundle
  • Trace file for the API responses
  • NGINX access logs /opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log

    Where: ORG, ENV and PORT# are replaced with actual values.

  • Message Processor system logs /opt/apigee/var/log/edge-message-processor/logs/system.log