502 Bad Gateway

Symptom

The client application gets an HTTP status code of 502 with the message "Bad Gateway" as a response for API calls.

The HTTP status code 502 means that the client is not receiving a valid response from the backend servers that should actually fulfill the request.

Error Messages

Client application gets the following response code:

HTTP/1.1 502 Bad Gateway

In addition, you may observe the following error message:

{
   "fault": {
      "faultstring": "Unexpected EOF at target", 
      "detail": {
           "errorcode": "messaging.adaptors.http.UnexpectedEOFAtTarget"
       }
    }
}

Possible Causes

One of the typical causes for 502 Bad Gateway Error is the "Unexpected EOF" error, which can be caused due to following reasons:

Cause Details Steps Given For
Incorrectly configured Target Server Target server is not properly configured to support TLS/SSL connections. Edge Private and Public Cloud Users
EOFException from Backend Server The backend server may send EOF abruptly. Edge Private Cloud Users Only

Incorrectly configured Target Server

Target server is not properly configured to support TLS/SSL connections.

Diagnosis

Diagnostic Steps for Public Cloud Users only

API Monitoring enables you to isolate problem areas quickly to diagnose error, performance, and latency issues and their source, such as developer apps, API proxies, backend targets, or the API platform.

Step through a sample scenario that demonstrates how to troubleshoot 5xx issues with your APIs using API Monitoring. For example, you may want to set up an alert to be notified when the number of messaging.adaptors.http.UnexpectedEOFAtTarget faults exceeds a particular threshold.

Diagnostic Steps for Private and Public Cloud Users

  1. Enable the trace in UI for the affected API.
  2. If the trace for the failing API request shows the following:
    1. The 502 Bad Gateway error is seen as soon as the Target Flow request started.
    2. The error.class displays messaging.adaptors.http.UnexpectedEOF.

      Then it is very likely that this issue is caused by an incorrect target server configuration.

  3. Get the target server definition using the Edge management API call:
    1. If you are a Public Cloud user, use this API:
      curl -v https://api.enterprise.apigee.com/v1/organizations/<orgname>/environments/<envname>/targetservers/<targetservername> -u <username>
      
    2. If you are a Private Cloud user, use this API:
      curl -v http://<management-server-host>:<port #>/v1/organizations/<orgname>/environments/<envname>/targetservers/<targetservername> -u <username>
      

      Sample Faulty Target Server Definition:

      <TargetServer  name="target1">
        <Host>mocktarget.apigee.net</Host>
        <Port>443</Port>
        <IsEnabled>true</IsEnabled>
      </TargetServer >
      
  4. The illustrated target server definition is an example for one of the typical misconfigurations which is explained as follows:

    Let's assume that the target server "mocktarget.apigee.net" is configured to accept secure (HTTPS) connections on port # 443. However, if you look at the target server definition, there are no other attributes/flags that indicate that it is meant for secure connections. This causes Edge to treat the API requests going to the specific target server as HTTP (non-secure) requests. So Edge will not initiate the SSL Handshake process with this target server.

Since the target server is configured to accept only HTTPS (SSL) requests on 443, it will reject the request from Edge or close the connection. As a result, you get an UnexpectedEOFAtTarget error on the Message Processor. The Message Processor will send "502 Bad Gateway" as a response to the client.

Resolution

Always ensure that the target server is configured correctly as per your requirements.

For the illustrated example above, if you want to make requests to a secure (HTTPS/SSL) target server, you need to include the SSLInfo attributes with the "enabled" flag set to true. While it is allowed to add the SSLInfo attributes for a target server in the target endpoint definition itself, it is recommended to add the SSLInfo attributes as part of the target server definition to avoid any confusion.

  1. If the backend service requires one-way SSL communication, then:
    1. You need to enable the TLS/SSL in the target server definition by including the SSLInfo attributes where the "enabled" flag is set to true as shown below:
      <TargetServer name="mocktarget">
        <Host>mocktarget.apigee.net</Host>
        <Port>443</Port>
        <IsEnabled>true</IsEnabled>
        <SSLInfo>
            <Enabled>true</Enabled>
        </SSLInfo>
      </TargetServer>
      
    2. If you want to validate the target server's certificate in Edge, then we also need to include the truststore (containing the target server's certificate) as shown below:
      <TargetServer  name="mocktarget">
          <IsEnabled>true</IsEnabled>
          <Host>mocktarget.apigee.net</Host>
          <Port>443</Port>
          <SSLInfo>
              <Ciphers/>
              <ClientAuthEnabled>false</ClientAuthEnabled>
              <Enabled>true</Enabled>
              <IgnoreValidationErrors>false</IgnoreValidationErrors>
              <Protocols/>
              <TrustStore>mocktarget-truststore</TrustStore>
          </SSLInfo>
      </TargetServer>
      
  2. If the backend service requires two-way SSL communication, then:
    1. You need to have SSLInfo attributes with ClientAuthEnabled, Keystore, KeyAlias, and Truststore flags set appropriately, as shown below:
      <TargetServer  name="mocktarget"> 
      
           <IsEnabled>true</IsEnabled> 
      
           <Host>www.example.com</Host> 
      
           <Port>443</Port> 
      
           <SSLInfo> 
      
               <Ciphers/> 
      
               <ClientAuthEnabled>true</ClientAuthEnabled> 
      
               <Enabled>true</Enabled> 
      
               <IgnoreValidationErrors>false</IgnoreValidationErrors> 
      
               <KeyAlias>keystore-alias</KeyAlias> 
      
               <KeyStore>keystore-name</KeyStore> 
      
               <Protocols/> 
      
               <TrustStore>truststore-name</TrustStore> 
      
           </SSLInfo> 
      
        </TargetServer >
      

References

Load balancing across backend servers

EOFException from the Backend Server

The backend server may send EOF (End of File) abruptly.

Diagnosis

Diagnostic Steps for Public Cloud Users only

API Monitoring enables you to isolate problem areas quickly to diagnose error, performance, and latency issues and their source, such as developer apps, API proxies, backend targets, or the API platform.

Step through a sample scenario that demonstrates how to troubleshoot 5xx issues with your APIs using API Monitoring. For example, you may want to set up an alert to be notified when the number of messaging.adaptors.http.UnexpectedEOFAtTarget faults exceeds a particular threshold.

Diagnostic Steps for Private Cloud Users only

  1. Check the Message Processor logs (/opt/apigee/var/log/edge-message-processor/logs/system.log) and search if you have got "eof unexpected" for the specific API or if you have the unique messageid for the API request, then you can search for it.

    Sample exception stack trace from Message Processor log

    "message": "org:myorg env:test api:api-v1 rev:10 messageid:rrt-1-14707-63403485-19 NIOThread@0 ERROR HTTP.CLIENT - HTTPClient$Context$3.onException() : SSLClientChannel[C:193.35.250.192:8443 Remote host:0.0.0.0:50100]@459069 useCount=6 bytesRead=0 bytesWritten=755 age=40107ms lastIO=12832ms .onExceptionRead exception: {} 
    java.io.EOFException: eof unexpected 
    at com.apigee.nio.channels.PatternInputChannel.doRead(PatternInputChannel.java:45) ~[nio-1.0.0.jar:na] 
    at com.apigee.nio.channels.InputChannel.read(InputChannel.java:103) ~[nio-1.0.0.jar:na] 
    at com.apigee.protocol.http.io.MessageReader.onRead(MessageReader.java:79) ~[http-1.0.0.jar:na] 
    at com.apigee.nio.channels.DefaultNIOSupport$DefaultIOChannelHandler.onIO(NIOSupport.java:51) [nio-1.0.0.jar:na] 
    at com.apigee.nio.handlers.NIOThread.run(NIOThread.java:123) [nio-1.0.0.jar:na]" 
    

    In the above example, you can see that the "java.io.EOFException: eof unexpected" error occurred while Message Processor is trying to read a response from the backend server. This exception indicates the end of file (EOF), or the end of stream has been reached unexpectedly.

    That is, the Message Processor sent the API request to the backend server and was waiting or reading the response. However, the backend server terminated the connection abruptly before the Message Processor got the response or could read the complete response.

  2. Check your backend server logs and see if there are any errors or information that could have led the backend server to terminate the connection abruptly. If you find any errors/information, then go to Resolution step and fix the issue appropriately in your backend server.
  3. If you don't find any errors or information in your backend server, collect the tcpdump output on the Message Processor(s):
    1. If your backend server host has a single IP address then use the following command:
      tcpdump -i any -s 0 host <IP address> -w <File name>
      
    2. If your backend server host has multiple IP addresses, then use the following command:
      tcpdump -i any -s 0 host <Hostname> -w <File name>
      

      Typically, this error is caused because the backend server responds back with [FIN,ACK] as soon as the Message Processor sends the request to the backend server.

  4. Consider the following tcpdump example.

    Sample tcpdump taken when 502 Bad Gateway Error (UnexpectedEOFAtTarget) occurred

  5. From the TCPDump output, you notice the following sequence of events:
    1. The Message Processor sends the API request to the backend server.
    2. The backend server immediately responds back with [FIN,ACK].
    3. This is followed by the Message Processor responding with [FIN,ACK] to the backend server.
    4. Eventually the connections are closed with [ACK] and [RST] from both the sides.
    5. Since the backend server sends [FIN,ACK], you get the exception "java.io.EOFException: eof unexpected" exception on the Message Processor.
  6. This can happen if there's a network issue at the backend server. Engage your network operations team to investigate this issue further.

Resolution

Fix the issue on the backend server appropriately.

If the issue persists and you need assistance troubleshooting 502 Bad Gateway Error or you suspect that it's an issue within Edge, contact Apigee Support.