Runtime issues

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Any errors, latency issues, or unexpected results observed during the execution of your API requests are referred to as runtime issues.

4XX/5XX errors

Playbook

This section provides information and guidance on some specific procedures that can be followed for troubleshooting and resolving runtime 4XX and 5XX errors.

Error response/message Error code Playbook
HTTP/1.1 500 Internal Server Error Varies by the actual error 500 Internal Server Error

and

500 Internal Server Error - Streaming Enabled
HTTP/1.1 503 Service Unavailable messaging.adaptors.http.flow.ServiceUnavailable 503 Service Unavailable
HTTP/1.1 503 Service Unavailable messaging.adaptors.http.flow.NoActiveTargets 503 Service Unavailable - NoActiveTargets
HTTP/1.1 503 Service Unavailable messaging.adaptors.http.flow.NoActiveTargets

(Cause is due to health check failures)

 503 Service Unavailable - NoActiveTargets Health Check Failures
HTTP/1.1 503 Service Unavailable

(from backend server)

messaging.adaptors.http.flow.ErrorResponseCode 503 Service Unavailable - Backend Server
HTTP/1.1 504 Gateway Timeout messaging.adaptors.http.flow.GatewayTimeout 504 Gateway Timeout
HTTP/1.1 504 Gateway Timeout

(from backend server)

messaging.adaptors.http.flow.ErrorResponseCode 504 Gateway Timeout - Backend Server

Diagnostics information

If you need any assistance from Apigee Edge Support on 4XX runtime errors (such as 400, 401, 404, and 499) or 5XX (such as 500, 503, and 504) errors, then gather and share the following diagnostic logs and information in the support case:

Diagnostic information Where can I gather this information? How do I gather this information?
Trace tool output capturing failed API requests Edge UI How to use Trace Tool
Router logs Router 
tar cvzf /tmp/router_logs_ORGNAME_ENVNAME_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /opt/apigee/var/log/edge-router/nginx/ORGNAME~ENVNAME.*
Message Processor logs Message Processor 
tar cvzf /tmp/rmp_systemlogs_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /opt/apigee/var/log/edge-message-processor/logs/system*
Compress all the diagnostic data 
>tar -cvzf /tmp/data_CASE#_$(hostname).tar.gz /tmp/router* /tmp/rmp_*

400 Bad Request Error - SSL Certificate Error

Playbook

This section provides information and guidance on some specific procedures that can be followed for troubleshooting and resolving 400 Bad Request - SSL Certificate Error.

Error message Playbook 
<html>
  <head>
    <title>400 The SSL certificate error</title>
  </head>
  <body bgcolor="white">
    <center> <h1>400 Bad Request</h1>
    </center>
    <center>The SSL certificate error</center>
    <hr>
    <center>nginx</center>
  </body>
</html>
400 Bad Request Error - SSL Certificate Error

Diagnostic information

If you need any assistance from Apigee Edge Support on the 400 Bad Request - SSL Certificate Error, then gather the following diagnostic information and share it in the support case:

Diagnostic information Where can I gather this information? How do I gather this information?
Router logs Router 
tar cvzf /tmp/router_logs_ORGNAME_ENVNAME_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /opt/apigee/var/log/edge-router/nginx/ORGNAME~ENVNAME.*
Tcpdumps Router

Capture network packets using the tcpdump command on the Router machine:

sudo tcpdump -s 0 -i any host CLIENT_HOST_IP_ADDRESS -w /tmp/router_tcpdump_$(hostname).pcap

Compress the tcpdump: 

tar cvzf /tmp/router_tcpdumps_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /tmp/router_tcpdump_$(hostname).pcap
Tcpdumps Client machine

Capture network packets using tcpdump command on client machine:

sudo tcpdump -s 0 -i any host VIRTUAL_HOST_ALIAS -w /tmp/client_tcpdump_$(hostname).pcap

Compress the tcpdump:

tar cvzf /tmp/client_tcpdumps_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /tmp/router_tcpdump_$(hostname).pcap
Compress all the diagnostic data Router 
tar -cvzf /tmp/data_CASE#_$(hostname).tar.gz /tmp/router*

404 Unable to identify proxy for host error

Playbook

This section provides information and guidance on some specific procedures that can be followed for troubleshooting and resolving 404 Unable to identify proxy for host error.

Error message or description Error code Playbook 
HTTP/1.1 404 Not Found

{
   "fault":{
      "faultstring":"Unable to identify proxy for host: VIRTUAL_HOST_NAME and url: PATH",
      "detail":{
         "errorcode":"messaging.adaptors.http.flow.ApplicationNotFound"
      }
   }
}
messaging.adaptors.http.flow.ApplicationNotFound 404 Unable to identify proxy for host

Diagnostic information

If you need assistance from Apigee Edge Support on the 404 Unable to identify proxy for host error, then gather the following diagnostic information and share it in the support case:

Diagnostic information Where can I gather this information? How do I gather this information?
Deployments API output Management Server 
curl -s http://MANAGEMENT_SERVER_HOST:8080/v1/organizations/ORGNAME/environments/ENVNAME/apis/APINAME/deployments > /tmp/deployments_output_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).json
API and Classification Tree output Message Processor

Get the environments loaded for a specific organization:

curl -s 0:8082/v1/runtime/organizations/ORGNAME/environments > /tmp/rmp_environments_list_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).txt

Get the revisions deployed for a specific API Proxy:

curl -s 0:8082/v1/runtime/organizations/ORGNAME/environments/ENVNAME/apis/APINAME/revisions > /tmp/rmp_api_APINAME_revisions_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).txt

Get the classification tree:

curl -s 0:8082/v1/classification/tree > /tmp/rmp_classification_tree_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).json
Message Processor logs Message Processor 
tar cvzf /tmp/rmp_systemlogs_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /opt/apigee/var/log/edge-message-processor/logs/system*
 
tar cvzf /tmp/rmp_transactionlogs_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /opt/apigee/var/log/edge-message-processor/logs/transactions*
 
tar cvzf /tmp/rmp_configurationlogs_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /opt/apigee/var/log/edge-message-processor/logs/configurations*
 
tar cvzf /tmp/rmp_system_monitor_config_mp_logs_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /opt/apigee/var/log/edge-message-processor/edge-message-processor* /opt/apigee/var/log/edge-message-processor/config* /opt/apigee/var/log/edge-message-processor/system-monitor*
Heap dumps on Message Processors Message Processor

Get the live heap dump:

sudo -u apigee jmap -dump:live,format=b,file=/opt/apigee/var/snapshot_$(hostname)-$(date +%Y.%m.%d_%H.%M.%S).hprof $(cat /opt/apigee/var/run/edge-message-processor/edge-message-processor.pid)

Compress the heap dump:

tar cvzf /tmp/rmp_heapdumps_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /opt/apigee/var/snapshot_$(hostname)-$(date +%Y.%m.%d_%H.%M.%S).hprof
Compress all the diagnostic data Message Processor 
tar -cvzf /tmp/data_CASE#_$(hostname).tar.gz /tmp/rmp_*

502 Bad Gateway - no live upstreams while connecting to upstream

Playbook

This section provides information and guidance on some specific procedures that can be followed for troubleshooting and resolving 502 Bad Gateway - no live upstreams while connecting to upstream.

Problem Error message in logs Playbook 
HTTP/1.1 502 Bad Gateway

<html>
<head>
<title>Error</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>An error occurred.</h1>
<p>Sorry, the page you are looking for is currently unavailable.<br/>
Please try again later.</p>
</body>
</html>

You will see the following error in NGINX error logs:

(/opt/apigee/var/log/edge-router/nginx/ORGNAME~ENVNAME._error_log)

[error] 4796#4796: *56357443 no live upstreams while connecting to upstream, client: ROUTER_IP_ADDRESS, server: HOST_ALIAS, request: "PUT BASE_PATH HTTP/1.1", upstream: "http://LISTOFMP_IP_R_MP_PORT/BASE_PATH", host: "HOST_ALIAS" 		502 Bad Gateway

Diagnostic information

If you need assistance from Apigee Edge Support on the 502 Bad Gateway - no live streams while connecting to upstream, then gather the following diagnostic information and share it in the support case:

Diagnostic information Where can I gather this information? How do I gather this information?
Router logs Router 
tar cvzf /tmp/router_logs_ORGNAME_ENVNAME_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /opt/apigee/var/log/edge-router/nginx/ORGNAME~ENVNAME.*
Message Processor logs Message Processor 
tar cvzf /tmp/rmp_systemlogs_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /opt/apigee/var/log/edge-message-processor/logs/system*
Top output, heap dump and thread dumps Message Processor

Get the top command output:

top -H -bn5 > /tmp/rmp_top_output_$(hostname)-$(date +%Y.%m.%d_%H.%M.%S).txt

Get the heap dump:

sudo -u apigee jcmd $(cat /opt/apigee/var/run/edge-message-processor/edge-message-processor.pid) GC.heap_dump /opt/apigee/var/rmp_heapdump_$(hostname)-$(date +%Y.%m.%d_%H.%M.%S).hprof

Get the thread dump:

sudo -u apigee jcmd $(cat /opt/apigee/var/run/edge-message-processor/edge-message-processor.pid) Thread.print > /tmp/rmp_thread_print_$(hostname)-$(date +%Y.%m.%d_%H.%M.%S).tdump
Compress all the diagnostic data 
tar -cvzf /tmp/data_CASE#_$(hostname).tar.gz /tmp/router* /tmp/rmp_* /opt/apigee/var/rmp_heapdump_*

502 Bad Gateway - Unexpected EOF At Target

Playbook

This section provides information and guidance on some specific procedures that can be followed for troubleshooting and resolving 502 Bad Gateway - Unexpected EOF At Target:

Error response/message Error code Playbook 
HTTP/1.1 502 Bad Gateway

{
   "fault": {
      "faultstring": "Unexpected EOF at target",
      "detail": {
           "errorcode": "messaging.adaptors.http.flow.UnexpectedEOFAtTarget"
       }
    }
}
messaging.adaptors.http.flow.UnexpectedEOFAtTarget 502 Bad Gateway Unexpected EOF

Diagnostic information

If you need assistance from Apigee Edge Support on the 502 Bad Gateway - Unexpected EOF At Target, then gather the following diagnostic information and share it in the support case:

Diagnostic information Where can I gather this information? How do I gather this information?
Trace tool output capturing failed API requests Edge UI How to use Trace Tool
Router logs Router 
tar cvzf /tmp/router_logs_ORGNAME_ENVNAME_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /opt/apigee/var/log/edge-router/nginx/ORGNAME~ENVNAME.*
Message Processor logs Message Processor 
tar cvzf /tmp/rmp_systemlogs_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /opt/apigee/var/log/edge-message-processor/logs/system*
Tcpdumps Message Processor 
sudo tcpdump -s 0 -i any host BACKENDSERVER_HOSTNAME -w /tmp/rmp_tcpdump_$(hostname).pcap
 
tar cvzf /tmp/rmp_tcpdumps_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /tmp/$(hostname).pcap
Compress all the diagnostic data Router/Message Processor 
tar -cvzf /tmp/data_CASE#_$(hostname).tar.gz /tmp/router* /tmp/rmp_*

TLS handshake failures

Playbook

This section provides information and guidance on some specific procedures that can be followed for troubleshooting and resolving TLS/SSL handshake failures:

Error message Playbook
Received fatal alert: handshake_failure TLS/SSL Handshake Failures
Received fatal alert: bad_certificate SSL Handshake Failures - Bad Client Certificate

Diagnostic information

If you need assistance from Apigee Edge Support on the TLS/SSL handshake failures, then gather the following diagnostic information and share it in the support case:

Diagnostic information Where can I gather this information? How do I gather this information?
Trace tool output capturing failed API requests Edge UI How to use Trace Tool
Router logs Router 
tar cvzf /tmp/router_logs_ORGNAME>_ENVNAME_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /opt/apigee/var/log/edge-router/nginx/ORGNAME~ENVNAME.*
Message Processor logs Message Processor 
tar cvzf /tmp/rmp_systemlogs_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /opt/apigee/var/log/edge-message-processor/logs/system*
OpenSSL Command Output Message Processor

Non SNI enabled Backend Server:

openssl s_client -connect BACKEND_SERVER_HOSTNAME:PORT -showcerts | tee /tmp/rmp_openssl_$(hostname)-$(date +%Y.%m.%d_%H.%M.%S).txt

SNI enabled Backend Server:

openssl s_client -connect BACKEND_SERVER_HOSTNAME:PORT -server BACKEND_SERVER_HOSTNAME -showcerts | tee /tmp/rmp_openssl_$(hostname)-$(date +%Y.%m.%d_%H.%M.%S).txt
Tcpdumps Message Processor 
sudo tcpdump -s 0 -i any host BACKEND_SERVER_HOSTNAME -w /tmp/$(hostname).pcap
 
tar cvzf /tmp/rmp_tcpdumps_$(hostname)_$(date +%Y.%m.%d_%H.%M.%S).tar.gz /tmp/$(hostname).pcap
Certificates from keystore and truststore of Message Processor Management Server

Get the certificate details from the Keystore:

curl -v http://MANAGEMENT_SERVER_HOST:PORT/v1/organizations/ORGNAME/environments/ENVNAME/keystores/KEYSTORENAME/certs/CERTNAME -u USERNAME

Get the certificate details from the Truststore:

curl -v http://MANAGEMENT_SERVER_HOSTPORT/v1/organizations/ORGNAME/environments/ENVNAME/keystores/TRUSTSTORENAME/certs/CERTNAME -u USERNAME
Compress all the diagnostic data Message Processor 
tar -cvzf /tmp/data_CASE#_$(hostname).tar.gz /tmp/router* /tmp/rmp_*