502 Bad Gateway - DecompressionFailureAtResponse

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Symptom

The client application gets an HTTP status code of 502 Bad Gateway with error code messaging.adaptors.http.flow.DecompressionFailureAtResponse as a response to API calls.

Error message

Client application gets the following response code:

HTTP/1.1 502 Bad Gateway

In addition, you may observe an error message similar to the one shown below:

{
   "fault":{
      "faultstring":"Decompression failure at response",
      "detail":{
         "errorcode":"messaging.adaptors.http.flow.DecompressionFailureAtResponse"
      }
   }
}

Possible causes

This error occurs only if:

• The encoding specified in the HTTP response (from the backend/target server) header Content-Encoding is valid and supported by Apigee Edge,

BUT

• The payload format sent by the backend/target server as part of the HTTP response does not match the encoding format specified in the Content-Encoding header

This is because Apigee Edge fails to decode the payload using the specified encoding since the format of the payload is not in the same format as the encoding specified in the Content-Encoding header.

Here are few examples of supported Content-Encoding values and how Apigee Edge expects the payload representation to be in those cases:

Scenario	Content-Encoding	Payload representation
Single Encoding	gzip	The Unix gzip format. See RFC1952 GZIP Format.
Single Encoding	deflate	This format uses zlib structure with deflate compression algorithm. See RFC1950 and RFC1951.
Multiple Encoding	Multiple Encoding For example, in cases when the encoding is done twice, it can be: gzip, deflate gzip, gzip deflate, gzip deflate, deflate	Multiple encoding applied to the payload in the given order as it appears in the header.

The possible causes for this error are as follows:

Cause	Description	Troubleshooting instructions applicable for
Response payload format does not match the Content-Encoding	The format of the response payload sent by the backend/target server is either not encoded or does not match the encoding specified in the Content-Encoding header.	Edge Public and Private Cloud users

Common diagnosis steps

Use one of the following tools/techniques to diagnose this error:

Cause: Response payload format does not match the Content-Encoding

By default, Apigee Edge always decompresses the payload if the response header Content-Encoding contains a valid and a supported encoding. Therefore, it is expected that the format of the response payload should match the encoding specified in the response header Content-Encoding. If there is a mismatch, then you get this error.

Diagnosis

1. Determine the Fault Code and Fault Source for the error observed using API Monitoring, Trace tool, or NGINX access logs as explained in Common diagnosis steps.
2. If the Fault Code is messaging.adaptors.http.flow.DecompressionFailureAtResponse and the Fault Source has the value target, then this indicates that the format of the response payload sent by the backend/target server does not match the supported encoding specified in the response header Content-Encoding.

3. You can determine the mismatch as part of the HTTP response using one of the following methods:

   Error message

   To validate using the error message:

   1. If you have access to the complete error message received from Apigee Edge, then refer to the faultstring.

      Sample error message:

      "faultstring":"Decompression failure at response"
      

   2. In the above error message, it displays "Decompression failure at response" which implies that the response could not be decompressed using the encoding specified in the Content-Encoding header.

   Trace

   To validate using the Trace:

   1. Determine the Content-Type and error.cause using Trace as explained in Common diagnosis steps.

   2. The values from the sample trace are as follows:

      • Content-Encoding: gzip
      • error.cause: Not in GZIP format

      The value in the response header Content-Encoding is gzip; however, the response payload is not in GZIP format (as indicated by error.cause). Therefore, Apigee Edge responds with 502 Bad Gateway and error code messaging.adaptors.http.flow.DecompressionFailureAtResponse.

   Actual request

   To validate using the actual request:

   If you have access to the actual request made to the target/backend server application, then perform the following steps:

   1. If you are a Public Cloud/Private Cloud user, make a request directly to the backend server from the backend server itself or any other machine from where you are allowed to make the request to the backend server.
   2. If you are a Private Cloud user, you can also make the request to the backend server from one of the Message Processors.
   3. Examine the response sent by the backend server and determine the value passed in the response header Content-Encoding.
   4. Determine the format of the payload sent as part of the request.
   5. If the value of Content-Encoding header is in the list of supported encoding but the format of the response payload does not match with the encoding specified in Content-Encoding header, then that is the cause of the issue.

      Sample:

      curl -v https://HOSTALIAS/test
      

      ***trimmed***
      >
      < HTTP/1.1 200 OK
      < Accept-Ranges: bytes
      < Content-Encoding: gzip
      < Date: Mon, 02 Aug 2021 08:17:35 GMT
      < Transfer-Encoding: chunked
      <
      < response_payload.zip Response Body(not in GZIP format)>
      

      The above sample response sends the value gzip to the Content-Encoding header which is a supported encoding in Apigee Edge. However, the response_payload.zip is sent as a zip file. Therefore, this response fails with a 502 Bad Gateway error with the error code: messaging.adaptors.http.flow.DecompressionFailureAtResponse.

   Message Processor logs

   To validate using Message Processor logs:

   If you are a Private Cloud user, then you can use Message Processor logs to determine the key information about HTTP 502 errors.

   1. Check the Message Processor log:

      /opt/apigee/var/log/edge-message-processor/logs/system.log

   2. Search to see if there are any 502 errors during a specific duration (if the problem happened in the past) or if there are any responses still failing with 502. You may use the following search string:

      grep -ri "ZipException"
      

   3. You will find lines from system.log similar to the following:

      Scenario #1

      Scenario #1: When API response has the header Content-Encoding: gzip

      2021-08-02 06:50:25,433  NIOThread@2 ERROR HTTP.CLIENT -
      HTTPClient$Context.onInputException() :  ClientInputChannel(ClientChannel[Connected:
      Remote:3.8.1.1:9000 Local:10.0.115.32:41298]@38140 useCount=1 bytesRead=0
      bytesWritten=203 age=469ms  lastIO=0ms  isOpen=true).onExceptionRead exception: {}
      java.util.zip.ZipException: Not in GZIP format
      ---trimmed--
      2021-08-02 06:50:25,433  NIOThread@2 INFO  HTTP.CLIENT -
      HTTPClient$Context.logContextDetails() : Request details : host=null
      path=/folder/testFile method=GET. Channel details : Bytes read=0
      2021-08-02 06:50:25,434  NIOThread@2 ERROR ADAPTORS.HTTP.FLOW -
      AbstractResponseListener.onException() : AbstractResponseListener.onError(HTTPResponse@4806fdab, Not in GZIP format)
      2021-08-02 06:50:25,434  NIOThread@2 INFO  HTTP.SERVICE -
      ExceptionHandler.handleException() : Exception
      java.util.zip.ZipException: Not in GZIP format
      occurred while writing to channel null
      2021-08-02 06:50:25,434  NIOThread@2 INFO  HTTP.SERVICE -
      ExceptionHandler.handleException() : Exception trace:
      java.util.zip.ZipException: Not in GZIP format
      

      The line java.util.zip.ZipException: Not in GZIP format in the above error message indicates that the response payload is not sent in GZIP format although the Content-Encoding is specified as gzip. Therefore, Apigee Edge throws the exception and returns a 502 status code with fault code messaging.adaptors.http.flow.DecompressionFailureAtResponse to client applications.

      Scenario #2

      Scenario #2: When API response has the header Content-Encoding: deflate

      2021-08-02 06:35:21,215  NIOThread@0 ERROR HTTP.CLIENT -
      HTTPClient$Context.onInputException() :  ClientInputChannel(ClientChannel[Connected:
      Remote:3.8.1.1:9000 Local:192.168.194.140:35224]@36014 useCount=1 bytesRead=0
      bytesWritten=202 age=439ms  lastIO=2ms  isOpen=true).onExceptionRead exception: {}
      java.util.zip.ZipException: incorrect header check
      ---trimmed----
      Caused by:
      java.util.zip.DataFormatException: incorrect header check
      ---trimmed---
      2021-08-02 06:35:21,215  NIOThread@0 INFO  HTTP.CLIENT -
      HTTPClient$Context.logContextDetails() : Request details :
      host=null path=/folder/testFile method=GET. Channel details : Bytes read=0
      2021-08-02 06:35:21,216  NIOThread@0 ERROR ADAPTORS.HTTP.FLOW -
      AbstractResponseListener.onException() : AbstractResponseListener.onError(HTTPResponse@3966e277,
      incorrect header check)
      2021-08-02 06:35:21,216  NIOThread@0 INFO  HTTP.SERVICE -
      ExceptionHandler.handleException() : Exception
      java.util.zip.ZipException: incorrect header check occurred while writing to channel null
      2021-08-02 06:35:21,217  NIOThread@0 INFO  HTTP.SERVICE -
      ExceptionHandler.handleException() : Exception trace:
      java.util.zip.ZipException: incorrect header check
      
      

      The lines java.util.zip.ZipException: incorrect header check and Caused by: java.util.zip.DataFormatException: incorrect header check in the above error message indicate that the response payload is not sent in deflate format and does not match the encoding specified in the Content-Encoding header of deflate. Therefore, Apigee Edge throws the exception and returns a 502 status code with fault code messaging.adaptors.http.flow.DecompressionFailureAtResponse to client applications.

Resolution

1. If there's no need for the compressed response payload in the API proxy flow in Apigee Edge and in the backend server, then do not pass the header Content-Encoding. If there is a need to compress the response payload, go to step 2.
2. If there's a need to compress the response payload, then ensure that the backend server always sends the following:
   • Any of the supported encoding as the value for the Content-Encoding header in the response
   • The response payload in the supported format to Apigee Edge matches the encoding format specified in the Content-Encoding header
3. In the example discussed above, the response payload is in ZIP format but the response header specifies Content-Encoding: gzip. You can fix the issue by sending the response header as Content-Encoding: gzip and the response payload in gzip format:

   curl -v https://HOSTALIAS/v1/test
   

   >
   < HTTP/1.1 200 OK
   < Accept-Ranges: bytes
   < Content-Encoding: gzip
   < Date: Mon, 02 Aug 2021 08:17:35 GMT
   < Transfer-Encoding: chunked
   <
   < response_payload.gz Response Body(in GZIP format)>
   

Specification

Apigee Edge responds with the status code 502 Bad Gateway with error code messaging.adaptors.http.flow.DecompressionFailureAtResponse as per the following RFC specifications:

Specification
RFC 7231, section 6.5.1
RFC 7231, section 3.1.2.2

If you still need any assistance from Apigee Support, go to Must gather diagnostic information.

Must gather diagnostic information

Gather the following diagnostic information, and then contact Apigee Edge Support:

If you are a Public Cloud user, provide the following information:

• Organization name
• Environment name
• API Proxy name
• Complete curl command used to reproduce the 502 error
• Trace file for the API responses

If you are a Private Cloud user, provide the following information:

• Complete error message observed for the failing responses
• Environment name
• API Proxy bundle
• Trace file for the API responses

• NGINX access logs /opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log

  Where: ORG, ENV and PORT# are replaced with actual values.

• Message Processor system logs /opt/apigee/var/log/edge-message-processor/logs/system.log