502 Bad Gateway - TooBigLine

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Symptom

The client application gets an HTTP status code of 502 Bad Gateway with error code protocol.http.TooBigLine as a response for API calls.

Error message

Client application gets the following response code:

HTTP/1.1 502 Bad Gateway

In addition, you may observe the following error message:

{
   "fault":{
      "faultstring":"response line size exceeding 2,048",
      "detail":{
         "errorcode":"protocol.http.TooBigLine"
      }
   }
}

Possible causes

This error occurs if the Response-Line size sent by the target/backend server to Apigee Edge as part of the HTTP response is greater than the maximum allowed limit in Apigee Edge.

Before we look at the possible causes for this error, let’s understand what the Response-Line means and how to check its size.

Understanding Response-Line

A typical HTTP response consists of three parts:

1. Status-Line (Referred to as Response-Line in Apigee)
2. ( Set of HTTP headers )
3. [ Body ]

The Response-Line consists of three parts: protocol version followed by a numeric status code and its associated textual phrase as shown below:

Response-Line   = <HTTP-Version> <Status-Code> <Reason-Phrase>

When an HTTP response is sent by the target/backend server application, the first line that is sent represents the Response-Line as described above. This is followed by the headers and response body/payload.The following sample screenshot shows a typical curl request, the Request part and the Response part (along with the Response-Line).

Understanding Response-Line size

1. In the sample discussed above, the start line (first line) in the response also referred as Response-Line is as follows:

   HTTP/1.1 200 OK
   

   This Response-Line has a size as ~15 bytes since it contains 15 ASCII characters. Since this is within the allowed limit in Apigee Edge, the response is sent back to the client without any errors by Apigee Edge.

2. Similarly if you look at the faultstring in the Error message shown above, it contains "response line size exceeding 2,048". This indicates that the Response-Line in the HTTP response sent by the target/backend server exceeded 2,048 bytes.

Understanding large Response-Line

As per the definition of Status-Line (referred to as Response-Line here) and typical HTTP requests and response, the size would be much smaller than the defined default limit of 2 K in Apigee Edge, so we may not hit the limit. However, here are some possible scenarios where you may exceed this limit:

1. The target/backend server is not an HTTP system. It could be responding with a non-HTTP response.
2. The target/backend server has issues and sends a long Response-Line as part of the HTTP response.

Read more about this in Getting error protocol.http.TooBigLine, "response line size exceeding 2,048.

Here are the possible causes for the error:

Cause	Description	Troubleshooting instructions applicable for
Response-Line size is greater than allowed limit	The size of the Response-Line sent by the target/backend server as part of the HTTP response to Apigee Edge is greater than the allowed limit in Apigee Edge	Edge Public and Private Cloud users

Common diagnosis steps

Use one of the following tools/techniques to diagnose this error:

Cause: Response-Line size is greater than allowed limit

Diagnosis

1. Determine the Fault Code and Fault Source for the error observed using API Monitoring, Trace tool or NGINX access logs as explained in Common diagnosis steps.
2. If the Fault Source has the value target, then this indicates that the Response-Line size sent by the target/backend server application to Apigee is greater than the allowed limit in Apigee Edge.

3. You can validate that the Response-Line size has exceeded the 2 KB allowed limit using one of the following methods:

   Error message

   To validate using the error message:

   If you have access to the complete error message received from Apigee Edge, then refer to the faultstring.

   Sample error message:

   "faultstring":"response line size exceeding 2,048"
   

   The above faultstring indicates that the Response-Line size exceeded the allowed limit of 2 KB.

   Actual request

   To validate using the actual request:

   If you have access to the actual request made to the target/backend server application, then perform the following steps:

   1. Verify the size of the Response-Line
   2. If you find that the size of the URI is more than the allowed limit in Apigee Edge, then that is the cause of the issue.

      Sample response from the target/backend server:

      curl -v http://HOSTALIAS/test
      

      *   Trying 3.2.1.4...
      * TCP_NODELAY set
      * Connected to <hostalias> (3.2.1.4) port 80 (#0)
      > GET /test HTTP/1.1
      > Host: HOSTALIAS
      > User-Agent: curl/7.64.1
      > Accept: */*
      >
      < HTTP/1.1 200 1111…<trimmed>...11111111
      < Date: Mon, 26 Jul 2021 07:07:18 GMT
      < Content-Type: application/json
      < Content-Length: 269
      < Connection: keep-alive
      < Server: gunicorn/19.9.0
      < Access-Control-Allow-Origin: *
      < Access-Control-Allow-Credentials: true
      <
      {
      <Response Body>
      }
      * Connection #0 to host <hostalias> left intact
      * Closing connection 0
      

      In the above case, the Response-Line HTTP/1.1 200 1111…<trimmed>...11111111 is larger than 2 KB, that is, it contains more than 2 K ASCII characters.

      If you are using some other client, you can review the client logs and try to find out the size of the Response-Line being sent to Apigee Edge.

   Message Processor logs

   To validate using Message Processor logs:

   If you are a Private Cloud user, then you can use Message Processor logs to validate if the Response-Line size has exceeded the allowed limit in Apigee Edge.

   1. Determine the message ID of the failing request using API Monitoring, Trace tool, or NGINX access logs as explained in Common diagnosis steps.

   2. Search for the message ID in the Message Processor log:

      /opt/apigee/var/log/edge-message-processor/logs/system.log

   3. You will find lines from system.log similar to the following:

      2021-07-26 06:45:41,451 org:myorg env:prod api:testtoobigline rev:1 messageid:r-5110240-1
      NIOThread@1 ERROR HTTP.CLIENT - HTTPClient$Context$3.onException() :
      ClientChannel[Connected: Remote:3.2.1.2:80 Local:192.168.205.251:44398]@20592
      useCount=1 bytesRead=0 bytesWritten=201 age=144ms  lastIO=0ms  isOpen=true.onExceptionRead
      exception: {}
      com.apigee.errors.http.server.BadGateway: response line size exceeding 2,048
      at <snipped>
      
      2021-07-26 06:45:41,451 org:myorg env:prod api:testtoobigline rev:1
      messageid:r-5110240-1  NIOThread@1 ERROR ADAPTORS.HTTP.FLOW -
      AbstractResponseListener.onException() : AbstractResponseListener.onError
      (HTTPResponse@6a5d6c33, response line size exceeding 2,048)
      

      The text message = response line size exceeding 2,048 in the above error message indicates that the Response-Line size is more than 2 KB. Therefore, Apigee Edge throws the exception and returns a 502 status code with fault code protocol.http.TooBigline to client applications.

Resolution

Limits

Apigee expects the client application and backend server do not send Request/Response-Lines whose sizes are greater than the allowed limit as documented for Request/Response Line Limit in Apigee Edge Limits.

1. If you are a Public Cloud user, then the maximum limit for Request and Response Line size is as documented for Request/Response-Line size in Apigee Edge Limits.
2. If you are a Private Cloud user, then you may have modified the default maximum limit for Request and Response-Line size (even though it is not a recommended practice). You can determine the maximum Response-Line size limit by following instructions in How to check the current limit.

How to check the current limit?

This section explains how to verify that the property HTTPResponse.line.limit has been updated with a new value on the Message Processors.

1. On the Message Processor machine, search for the property HTTPResponse.line.limit in the /opt/apigee/edge-message-processor/conf directory and check to see what value has been set as shown below:

   grep -ri "HTTPResponse.line.limit" /opt/apigee/edge-message-processor/conf
   

2. The sample result from the above command is as follows:

   /opt/apigee/edge-message-processor/conf/http.properties:HTTPResponse.line.limit=2k
   

3. In the example output above, notice that the property HTTPResponse.line.limit has been set with the value 2k in http.properties.

   This indicates that the limit for the Response-Line size configured in Apigee for Private Cloud is 2 KB.

If you still need any assistance from Apigee Support, go to Must gather diagnostic information.

Must gather diagnostic information

Gather the following diagnostic information, and then contact Apigee Edge Support:

If you are a Public Cloud user, provide the following information:

• Organization name
• Environment name
• API Proxy name
• Complete curl command used to reproduce the 502 error
• Trace file for the API requests

If you are a Private Cloud user, provide the following information:

• Complete error message observed for the failing requests
• Organization name
• Environment name
• API Proxy bundle
• Trace file for the failing API requests
• Complete curl command used to reproduce the 502 error

• NGINX access logs /opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log

  Where: ORG, ENV and PORT# are replaced with actual values.

• Message Processor system logs /opt/apigee/var/log/edge-message-processor/logs/system.log