502 Bad Gateway - ResponseWithBody

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Symptom

The client application gets an HTTP status code of 502 Bad Gateway with the error code protocol.http.ResponseWithBody as a response for API calls.

Error message

Client application gets the following response code:

HTTP/1.1 502 Bad Gateway

In addition, you may observe one of the following error messages:

{
   "fault":{
      "faultstring":"Received 204 Response with message body",
      "detail":{
         "errorcode":"protocol.http.ResponseWithBody"
      }
   }
}

{
   "fault":{
      "faultstring":"Received 205 Response with message body",
      "detail":{
         "errorcode":"protocol.http.ResponseWithBody"
      }
   }
}

Possible causes

This error occurs if the HTTP response from the backend server to Apigee Edge is either 204 No Content or 205 Reset Content but it contains the response body and/or one or more of the following headers:

Content-Length
Content-Encoding
Transfer-Encoding

As per the specifications RFC 7231, section 6.3.5: 204 No Content and RFC 7231, section 6.3.6: 205 Reset Content, it is expected that no additional content should be sent as part of the response payload body with status code 204 No Content or 205 Reset Content by the origin server. The response headers such as Content-Length, Content-Encoding or Transfer-Encoding indicate the size, type or format of the response payload.

Therefore, Apigee Edge returns a 502 Bad Gateway status code with the error code protocol.http.ResponseWithBody to the client under the following circumstances:

	Status code from backend server
Response from backend server contains	204 No Content	205 Reset Content
Response Body	ERROR	ERROR
`Content-Length` header (set to non-zero)	ERROR	ERROR
`Content-Encoding` (set to supported encoding in Apigee Edge)	ERROR	NO ERROR
`Transfer-Encoding`	ERROR	ERROR
Note: In the above table: ERROR indicates Apigee Edge will return `502 Bad Gateway` in these conditions. NO ERROR indicates Apigee Edge will not return error, instead return the response received from backend server as-is.

Here are the possible causes for this error:

Cause	Description	Troubleshooting instructions applicable for
Response body or Headers with 204 response from backend server	The backend server sends a `204 No Content` or `205 Reset Content` response with a response body and/or one or more of headers `Content-Type`, `Content-Encoding` or `Transfer-Encoding`.	Edge Public and Private Cloud users

Common diagnosis steps

Use one of the following tools/techniques to diagnose this error:

API Monitoring

To diagnose the error using API Monitoring:

Sign in to Apigee Edge UI as a user with an appropriate role.
Switch to the organization in which you want to investigate the issue.
Navigate to the Analyze > API Monitoring > Investigate page.
Select the specific timeframe in which you observed the errors.
Plot Fault Code against Time.
Note: You can also plot Status Code against Time.
Select a cell which has the fault code protocol.http.ResponseWithBody as shown below:

( view larger image)
You will see the information about the fault code protocol.http.ResponseWithBody as shown below:

( view larger image)
Click View logs and expand the row for the failed request.

( view larger image)
From the Logs window, note the following details:
- Status Code: 502
- Fault Source: target
- Fault Code: protocol.http.ResponseWithBody.
If the Fault Source has the value target and the Fault Code has the value protocol.http.ResponseWithBody, then that indicates that the error occurred because the backend server sent a 204 No Content or 205 Reset Content status code with the response body and/or one of the headers mentioned in the Possible causes section.

Trace tool

To diagnose the error using the Trace tool:

Enable the trace session and either:
1. Wait for the 502 Bad Gateway error to occur. or
2. If you can reproduce the issue, make the API call and reproduce 502 Bad Gateway error.
Ensure Show all FlowInfos is enabled:
Select one of the failing requests and examine the trace.
Navigate through different phases of the trace and locate where the failure occurred.
You will typically find the error in the flowinfo Error just after the Request sent to target server phase as shown below:
Scenario #1

Scenario #1: Backend Server responds with status code 204 No Content containing response body and/or one of the headers listed in Possible causes.

Note the values of the following from the trace:
- error: Received 204 Response with message body
- error.class: com.apigee.rest.framework.BadGateway
Scenario #2

Scenario #2: Backend Server responds with status code 204 No Content containing response body and/or one of the headers listed in Possible causes.

Note the values of the following from the trace:
- error: Received 205 Response with message body
- error.class: com.apigee.rest.framework.BadGateway
Navigate to the AX (Analytics Data Recorded) phase in the trace and click it.
Scroll down to the Phase Details, Error Headers section and determine the values of X-Apigee-fault-code and X-Apigee-fault-source as shown below:

( view larger image)

Note that the values of X-Apigee-fault-code and X-Apigee-fault-source are protocol.http.ResponseWithBody and target respectively. This indicates that the error occurred because the backend server sent a 204 No Content or 205 Reset Content status code with the response body and/or one of the headers mentioned in Possible causes.

Error	Value
X-Apigee-fault-code	`protocol.http.ResponseWithBody`
X-Apigee-fault-source	`target`

NGINX

To diagnose the error using NGINX access logs:

If you are a Private Cloud user, then you can use NGINX access logs to determine the key information about HTTP 502 Bad Gateway.
Check the NGINX access logs:

/opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log

Where: ORG, ENV, and PORT# are replaced with actual values.
Search to see if there are any 502 errors with error code protocol.http.ResponseWithBody during a specific duration (if the problem happened in the past) or if there are any requests still failing with 502.

If you do find any 502 errors with the X-Apigee-fault-code matching the value of protocol.http.ResponseWithBody, then determine the value of the X-Apigee-fault-source.

Sample 502 error from NGINX access log:

The above sample entry from the NGINX access log has the following values for X- Apigee-fault-code and X-Apigee-fault-source:

Response Headers	Value
X-Apigee-fault-code	`protocol.http.ResponseWithBody`
X-Apigee-fault-source	`target`

Note that the values of X-Apigee-fault-code and X-Apigee-fault-source are protocol.http.ResponseWithBody and target respectively. This indicates that the error occurred because the backend server sent a 204 No Content or 205 Reset Content status code with the response body and/or one of the headers mentioned in Possible causes.

Cause: Response body or Headers with 204 response from backend server

Diagnosis

Determine the Fault Code and Fault Source for the error observed using API Monitoring, Trace tool, or NGINX access logs as explained in Common diagnosis steps.
If the Fault Code is protocol.http.ResponseWithBody and Fault Source has the value target, then this indicates that the backend server has responded with a 204 No Content or 205 Reset Content status code with the response body and/or one of the headers mentioned in Possible causes.
To validate if the backend server has indeed sent a response payload body and/or one or more of the headers mentioned in Possible causes, you can perform the following steps:
1. If you are a Public Cloud user, and if you can make the same API request to the backend server directly from any of your systems.
  
  Note: You can make a direct call to the backend server only if the traffic is allowed from your system to the backend server.
2. If you are a Private Cloud user, then you can make the same API request to the backend server directly from one of the Message Processors associated with the specific organization and environment where the failure is observed.
3. Review the response received from the backend server and verify that it contains a response payload body and/or one or more of the above mentioned headers. If yes, then that’s the cause for this error.
  Sample #1
  
  Sample #1: Backend Server Response 204 with Content-Encoding Header
```
curl -v "https://BACKEND_SERVER_HOST_NAME/PATH" -H "HEADER: VALUE" -X HTTP_REQUEST_METHOD
```
```
…
< HTTP/1.1 204 No Content
< Content-Encoding: gzip
< Date: Tue, 31 Jul 2021 21:41:13 GMT
< Connection: keep-alive
```
  In this sample, the backend server responded with 204 No Content status code and Content-Encoding: gzip
  Sample #2
  
  Sample #2: Backend Server Response 204 with Content-Length Header
```
curl -v "https://BACKEND_SERVER_HOST_NAME/PATH" -H "HEADER: VALUE" -X HTTP_REQUEST_METHOD
```
```
…
< HTTP/1.1 204 No Content
< Content-Length: 48
< Date: Tue, 31 Jul 2021 21:41:13 GMT
< Connection: keep-alive
```
  In this sample, the backend server responded with 204 No Content status code and Content-Length: 48
  Sample #3
  
  Sample #3: Backend Server Response 205 with Response Body
```
curl -v "https://BACKEND_SERVER_HOST_NAME/PATH" -H "HEADER: VALUE" -X HTTP_REQUEST_METHOD
```
```
…
< HTTP/1.1 205 Reset Content
< Date: Sat, 31 Jul 2021 17:14:09 GMT
< Content-Length: 12
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host X.X.X.X left intact
This is a sample Response
```
  In this sample, the backend server responded with 205 Reset Content status code with response body This is a sample Response.
4. In all of the above examples, the backend server sent 204 No Content or 205 Reset Content status code with the response body and/or one of the headers mentioned in Possible causes.
5. Therefore, Apigee Edge sent 502 Bad Gateway status code with error code protocol.http.ResponseWithBody.

Resolution

Ensure that the backend server always adheres to the Specification RFC 7231, section 6.3.6: 205 Reset Content, when sending the 204 No Content or 205 Reset Content response to Apigee Edge. That is, the backend server MUST NOT send the following as part of a 204 No Content or 205 Reset Content response:

Response payload body
And any of the following headers:
1. Content-Length
2. Content-Encoding
3. Transfer-Encoding

Specification

Apigee Edge responds with 502 Bad Gateway status code and the error code protocol.http.ResponseWithBody if the backend server sends a 204 No Content or 205 Reset Content response but does not adhere to the following RFC specifications:

Specification
RFC 7231, section 6.3.5: 204 No Content
RFC 7231, section 6.3.6: 205 Reset Content

Key points to note

The recommended solution is to fix the backend server to send the 204 No Content and 205 Reset Content status code without response body and any of the headers - Content-Length, Content-Encoding, and Transfer-Encoding and adhere to the specifications RFC 7231, section 6.3.5: 204 No Content and RFC 7231, section 6.3.6: 205 Reset Content.

If you still need any assistance from Apigee Support, go to Must gather diagnostic information.

Must gather diagnostic information

Gather the following diagnostic information, and then contact Apigee Edge Support:

If you are a Public Cloud user, provide the following information:

Organization name
Environment name
API Proxy name
Complete curl command used to reproduce the 502 error
Trace file for the API requests

If you are a Private Cloud user, provide the following information:

Complete error message observed for the failing requests
Environment name
API Proxy bundle
Trace file for the API requests
NGINX access logs /opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log

Where: ORG, ENV and PORT# are replaced with actual values.
Message Processor system logs /opt/apigee/var/log/edge-message-processor/logs/system.log