502 Bad Gateway - TooBigBody

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Symptom

The client application gets an HTTP status code of 502 Bad Gateway with error code protocol.http.TooBigBody as a response for API calls.

Error message

Client application gets the following response code:

HTTP/1.1 502 Bad Gateway

In addition, you may observe the following error message:

{
   "fault":{
      "faultstring":"Body buffer overflow",
      "detail":{
         "errorcode":"protocol.http.TooBigBody"
      }
   }
}

Possible causes

This error occurs if the payload size sent by the target/backend server to Apigee Edge as part of HTTP response is greater than the allowed limit in Apigee Edge.

Here are the possible causes for the error:

Cause	Description	Troubleshooting instructions applicable for
Response payload size is greater than allowed limit	The payload size sent by the target/backend server as part of HTTP response to Apigee is more than the allowed limit in Apigee.	Edge Public and Private Cloud users
Response payload size exceeds allowed limit after decompression	The payload size sent in compressed format by the target/backend server as part of HTTP response to Apigee is more than the allowed limit when decompressed by Apigee.	Edge Public and Private Cloud users

Common diagnosis steps

Use one of the following tools/techniques to diagnose this error:

API Monitoring

To diagnose the error using API Monitoring:

Sign in to Apigee Edge UI as a user with an appropriate role.
Switch to the organization in which you want to investigate the issue.
Navigate to the Analyze > API Monitoring > Investigate page.
Select the specific timeframe in which you observed the errors.
You may select the Proxy filter to narrow down the fault code.
Plot Fault Code against Time.
Select a cell which has the fault code protocol.http.TooBigBody as shown below:
You will see the information about the fault code protocol.http.TooBigBody as shown below:
Click View logs and expand the row for the failed request.
From the Logs window, note the following details:
- Status Code: 502
- Fault Source: target
- Fault Code: protocol.http.TooBigBody.
If the Fault Source has the value target and the Fault Code has the value protocol.http.TooBigBody, then that indicates that the HTTP response from the target/ backend server has a response payload size greater than the allowed limit in Apigee Edge.

Trace

To diagnose the error using the Trace tool:

Enable the trace session and either:
- Wait for the 502 Bad Gateway error to occur, or
- If you can reproduce the issue, make the API call and reproduce 502 Bad Gateway error.
Select one of the failing requests and examine the trace.
Navigate through different phases of the trace and locate where the failure occurred.
Navigate to the phase Error just after the Response received from target server phase as shown below:

Note the values of the error from the trace:
- error: Body buffer overflow
- error.class: com.apigee.errors.http.server.BadGateway
This indicates that Apigee Edge (Message Processor component) throws the error as soon as it receives the response from the backend server due to payload size exceeding the allowed limit.
You would see the failure in the Response Sent to Client phase as shown below:
Note the values of the error from the trace. The above sample trace shows:
- error: 502 Bad Gateway
- Error Content: {"fault":{"faultstring":"Body buffer overflow","detail":{"errorcode":"protocol.http.TooBigBody"}}}
Navigate to the Response Received from target server phase as shown below for different scenarios:
Uncompressed

Scenario #1: Response Payload sent in uncompressed form

Note the values of the error from the trace:
- Response received from target server: 200 OK
- Content-Length (from Response Headers section): ~11MB
Compressed

Scenario #2: Request Payload sent in compressed form

Note the values of the error from the trace:
- Response received from target server: 200 OK
- Content-Encoding: If you see this header in the Response Headers section, note the value. For example, in this example the value is gzip.
Note the Body under the Response Content section:
```
{"fault":{"faultstring":"Body buffer overflow","detail":{"errorcode":"protocol.http.TooBigBody"}}}
```
Note: Even though the target/backend server sends 200, the actual response sent to the client is 502 along with the above error message by Apigee Edge.
Navigate to the AX (Analytics Data Recorded) Phase in the trace and click on it to see the related details.

Scroll down in Phase Details to the Variables Read section and determine the values of target.received.content.length which indicates:

The actual size of the response payload when it is sent in uncompressed format and
The size of the response payload upon decompression by Apigee, when the payload is sent in compressed format. It will always be the same as the value of the allowed limit (10 MB) in this scenario.

Uncompressed

Scenario #1: Response Payload sent in uncompressed form

Note the value of target.received.content.length:

Request headers	Value
target.received.content.length	~11 MB

Compressed

Scenario #2: Request Payload sent in compressed form

Note the value of target.received.content.length:

Request Headers	Value
target.received.content.length	~10 MB

The following table explains why the 502 error is returned by Apigee under the two scenarios based on the value of target.received.content.length:

Scenario	Value of target.received.content.length	Reason for failure
Response Payload in uncompressed format	~11 MB	Size > allowed limit of 10 MB
Response Payload in compressed format	~10 MB	Size limit exceeded upon decompression Note: The actual size of payload upon decompression could be much more than 10 MB. However, Apigee stops decompressing as soon as the size of the uncompressed data exceeds the limit of 10 MB (default limit) and responds with the error.

NGINX

To diagnose the error using NGINX access logs:

If you are a Private Cloud user, then you can use NGINX access logs to determine the key information about HTTP 502 errors.
Check the NGINX access logs:
```
/opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log
```
Where: ORG, ENV, and PORT# are replaced with actual values.
Search to see if there are any 502 Errors during a specific duration (if the problem happened in the past) or if there are any requests still failing with 502.

If you do find any 502 errors with the X-Apigee-fault-code matching the value of protocol.http.TooBigBody, then determine the value of the X-Apigee-fault-source.

Sample 502 error from NGINX access log:

The above sample entry from NGINX access log has the following values for X-Apigee- fault-code and X-Apigee-fault-source:

Response Headers	Value
X-Apigee-fault-code	`protocol.http.TooBigBody`
X-Apigee-fault-source	`target`

Cause: Response Payload Size is greater than allowed limit

Diagnosis

Determine the Fault Code, Fault Source, and Response Payload Size for the error observed using API Monitoring, Trace tool, or NGINX access logs as explained in Common diagnosis steps with Scenario #1.
If the Fault Source has the value target, then this indicates that the response payload size sent by the target/backend server to Apigee is greater than the allowed limit in Apigee Edge.
Verify the Response Payload Size as determined from step 1.
- If the payload size > 10 MB allowed limit, then that’s the cause for the error.
- If the payload size ~ 10 MB allowed limit, then it’s possible that the response payload is passed in compressed format. Go to Cause: Response payload size exceeds allowed limit after decompression.
Validate that the response payload size is indeed > 10 MB allowed limit by checking the actual response using the following steps:
1. If you don’t have access to the actual request made to the target/backend server, then go to Resolution.
2. If you have access to the actual request made to the target/backend server, then perform the following steps:
  1. If you are a Public Cloud/Private Cloud user, make a request directly to the backend server from the backend server itself or any other machine from where you are allowed to make the request to the backend server.
  2. If you are a Private Cloud user, you can also make the request to the backend server from one of the Message Processors.
  3. Verify the size of the payload passed in the response by checking the Content-Length header.
  4. If you find that the size of the payload is more than the allowed limit in Apigee Edge, then that is the cause of the issue.
Sample Response from the backend server:
```
curl -v https://BACKENDSERVER-HOSTNAME/testfile
```
```
* About to connect() to 10.14.0.10 port 9000 (#0)
*   Trying 10.14.0.10...
* Connected to 10.14.0.10 (10.148.0.10) port 9000 (#0)
> GET /testfile HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.14.0.10:9000
> Accept: */*
>
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 11534336
< Content-Type: application/octet-stream
< Last-Modified: Wed, 30 Jun 2021 08:18:02 GMT
< Date: Wed, 30 Jun 2021 09:22:41 GMT
<
----snipped----
<Response Body>
```
In the above example, you can see that Content-Length: 11534336 (which is ~11 MB) which is the cause for this error as it exceeds the allowed limit in Apigee Edge.

Resolution

Refer to Resolution.

Cause: Response Payload Size exceeds allowed limit after decompression

If the response payload is sent in compressed format and the response header Content-Encoding is set to gzip, Apigee decompresses the response payload. During the decompression process, if Apigee finds the size of the payload to be greater than the allowed limit in Apigee Edge., then it stops further decompression and responds back immediately with 502 Bad Gateway and the error code protocol.http.TooBigBody.

Diagnosis

Determine the Fault Code, Fault Source and Response Payload size for the error observed using API Monitoring, Trace Tool or NGINX Access logs as explained in Common diagnosis steps with Scenario #2.
If the Fault Source has the value target, then this indicates that the response payload size sent by the target/backend application to Apigee is greater than the allowed limit in Apigee Edge.
Verify the Response Payload Size as determined from step 1.
- If the payload size > 10 MB allowed limit, then that’s the cause for the error.
- If the payload size ~ 10 MB allowed limit, then it’s possible that the response payload is passed in compressed format. In this case, check for the uncompressed size of the compressed response payload.
You can validate if the response from the target/backend was sent in compressed format and the uncompressed size was greater than the allowed limit using one of the following methods:
Trace

Using the Trace tool:
1. If you have captured a trace for the failing request, refer to steps detailed in Trace and
  1. Determine the value of target.received.content.length
  2. Verify if the request from the client contained the Content-Encoding: gzip header
2. If the value of target.received.content.length is around the 10 MB allowed limit, and the response header Content-Encoding: gzip, then that’s the cause for this error.
Actual request

Using actual request:
1. If you don’t have access to the actual request made to the target/backend server, then go to Resolution.
2. If you have access to the actual request made to the target/backend server, then perform the following steps:
  1. Verify the size of the payload passed in the response along with the Content-Encoding header sent in the response.
  2. If you find that the response header Content-Encoding is set to gzip and the uncompressed size of the payload is more than the allowed limit in Apigee Edge, then that’s the cause for this error.
    Sample response received from the backend server:
    
    curl -v https://BACKENDSERVER-HOSTNAME/testzippedfile.gz
    
    * About to connect() to 10.1.0.10 port 9000 (#0) * Trying 10.1.0.10... * Connected to 10.1.0.10 (10.1.0.10) port 9000 (#0) > GET /testzippedfile.gz HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 10.1.0.10:9000 > Accept: */* > < HTTP/1.1 200 OK < Accept-Ranges: bytes < Content-Encoding: gzip < Content-Type: application/x-gzip < Last-Modified: Wed, 30 Jun 2021 08:18:02 GMT < Testheader: test < Date: Wed, 07 Jul 2021 10:14:16 GMT < Transfer-Encoding: chunked < ----snipped---- <Response Body>
    
    In the above case, the header Content-Encoding: gzip is sent and the size of the file testzippedfile.gz in the response is less than the limit, however the size of uncompressed file testzippedfile was ~15 MB.
Note: If you don't have access to the actual request and response from the backend server, then you can check the backend server logs, check the response file sent by the backend server, and verify the size of the compressed file.
Message Processor logs

Using Message Processor logs:

Note: The instructions in this section are for Edge Private Cloud users only.
1. If you are a Private Cloud user, then you can use Message Processor logs to determine the key information about HTTP 502 errors.
2. Check the Message Processor logs
  
  /opt/apigee/var/log/edge-message-processor/logs/system.log
3. Search to see if there are any 502 errors during a specific duration (if the problem happened in the past) or if there are any requests still failing with 502. You may use the following search strings:
```
grep -ri "chunkCount"
```
```
grep -ri "BadGateway: Body buffer overflow"
```
4. You will find lines from system.log similar to as shown below (TotalRead and chunkCount may vary in your case):
```
2021-07-07 09:40:47,012  NIOThread@7 ERROR HTTP.SERVICE -
TrackingInputChannel.checkMessageBodyTooLarge() : Message is too large.
TotalRead 10489856 chunkCount 2571

2021-07-07 09:40:47,012  NIOThread@7 ERROR HTTP.CLIENT -
HTTPClient$Context.onInputException() :
ClientInputChannel(ClientChannel[Connected:
Remote:10.148.0.10:9000 Local:10.148.0.9:42240]@9155
useCount=1 bytesRead=0 bytesWritten=182 age=23ms  lastIO=0ms
isOpen=true).onExceptionRead exception: {}
com.apigee.errors.http.server.BadGateway: Body buffer overflow

2021-07-07 09:40:47,012  NIOThread@7 ERROR
ADAPTORS.HTTP.FLOW - AbstractResponseListener.onException() :
AbstractResponseListener.onError(HTTPResponse@77cbd7c4,
Body buffer overflow)
```
5. During the process of decompression, as soon as the Message Processor determines the total read bytes is > 10 MB, it stops and prints the following line:
  
  Message is too large. TotalRead 10489856 chunkCount 2571
  
  It implies that the Response Payload Size is more than 10 MB and Apigee throws the error when the size starts to exceed the limit 10 MB with fault code as protocol.http.TooBigBody
  
  Note: You will see the above line in the Message Processor logs, even if you send the response payload in uncompressed format. The only difference is that TotalRead will contain the actual size of the request payload in that scenario.

Resolution

Fix size

Option #1 [Recommended]: Fix the target server application not to send payload size exceeding the Apigee limit

Analyze the reason for the specific target server to send response / payload size more than allowed limit as defined in Limits.
If it is not desirable, modify your target server application so that it sends response / payload size less than the allowed limit.
If it is desirable and you want to send a response/payload more than the allowed limit, go to the next options.

Signed URL pattern

Option #2 [Recommended]: Use signed URLs pattern within an Apigee JavaCallout

For payloads larger than 10 MB, Apigee recommends using a signed URLs pattern within an Apigee JavaCallout, illustrated by the Edge Callout: Signed URL Generator example on GitHub.

Streaming

Option #3: Use Streaming

If your API proxy needs to handle very large requests and/or responses, then you can enable streaming in Apigee.

CwC

Option #4: Use CwC property to increase the buffer limit

This option should be used only when you cannot use any of the recommended options as there might be performance issues if the default size is increased.

Apigee provides a CwC property which allows it to increase the request and response payload size limit. For details, refer to Set the message size limit on the Router or Message Processor.

Limits

Apigee expects the client application and backend server to not send payload sizes greater than the allowed limit as documented for Request/response size in Apigee Edge Limits.

If you are a Public Cloud user, then the maximum limit for request and response payload size is as documented for Request/response size in Apigee Edge Limits.
If you are a Private Cloud user, then you may have modified the default maximum limit for request and response payload size (even though it is not a recommended practice). You can determine the maximum request payload size limit by following instructions in How to check the current limit.

How to check the current limit?

This section explains how to verify that the property HTTPResponse.body.buffer.limit has been updated with a new value on the Message Processors.

On the Message Processor machine, search for the property HTTPResponse.body.buffer.limit in the /opt/apigee/edge-message- processor/conf directory and check to see what value has been set as shown below:
```
grep -ri "HTTPResponse.body.buffer.limit" /opt/apigee/edge-message-processor/conf
```

The sample result from the above command is as follows:

/opt/apigee/edge-message-processor/conf/http.properties:HTTPResponse.body.buffer.limit=10m

In the example output above, notice that the property HTTPResponse.body.buffer.limit has been set with the value 10m in http.properties.

This indicates that the limit for the request payload size configured in Apigee for Private Cloud is 10 MB.

If you still need any assistance from Apigee Support, go to Must gather diagnostic information.

Must gather diagnostic information

Gather the following diagnostic information, and then contact Apigee Edge Support:

If you are a Public Cloud user, provide the following information:

Organization name
Environment name
API Proxy name
Complete curl command used to reproduce the 502 error
Trace file for the API requests
Complete output of the response from target/backend server along with the size of the payload

If you are a Private Cloud user, provide the following information:

Complete error message observed for the failing requests
Organization name
Environment name
API Proxy bundle
Trace file for the failing API requests
Complete curl command used to reproduce the 502 error
Complete output of the response from target/backend server along with the size of the payload
NGINX access logs /opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log

Where: ORG, ENV and PORT# are replaced with actual values.
Message Processor system logs /opt/apigee/var/log/edge-message-processor/logs/system.log

502 Bad Gateway - TooBigBody

Symptom

Error message

Possible causes

Common diagnosis steps

API Monitoring

Trace

Uncompressed

Compressed

Uncompressed

Compressed

NGINX

Cause: Response Payload Size is greater than allowed limit

Diagnosis

Resolution

Cause: Response Payload Size exceeds allowed limit after decompression

Diagnosis

Trace

Actual request

Message Processor logs

Resolution

Fix size

Signed URL pattern

Streaming

CwC

Limits

How to check the current limit?

Must gather diagnostic information