API requests not captured in the Trace UI

Symptom

API requests are not captured in the Edge UI Trace when a trace session is started.

Error Message

No error messages will be displayed in the Edge UI when this problem occurs.

Possible Causes

Cause Description Troubleshooting Instructions Applicable For
Requests not processed by the Message Processor API requests must be processed by Edge's component Message Processor in order to capture trace. If an API request fails to reach Apigee Edge, fails at the entry point to Edge (i.e., Router) or fails before it is processed by the Message Processor, then the trace cannot be captured. Edge Public and Private Cloud Users
API Proxy Not Found in The Classification Tree Apigee Message Processors use a Routing Rule definition called Classification Tree for dispatching requests based on the hostname, base path, revision, and environment of the incoming request. If the relevant API proxy is removed for some reason from the Classification Tree, the Trace transactions may not populate. Edge Private Cloud Users

Cause: Requests not processed by the Message Processor

Diagnosis

In order to capture an API request in a Trace session, the API request must be processed by Edge's component Message Processor. There are several reasons why an API request may not be captured in a Trace transaction.

For example, if an API request fails to reach Apigee Edge, fails at the entry point to Edge (i.e., Router) or fails before it gets processed by the Message Processor, then the trace cannot be captured. Each of these scenarios is outlined in greater detail below.

Scenario 1: Requests fail to reach Apigee Edge

  • Cause

    In this scenario, the error may be caused by a DNS resolution or network connectivity issue. If so, you may see the following error when running this command:

    curl https://hostName:port/apiProxyBasePath/requestPath
    
    curl: (6) Could not resolve host: hostName
    
  • Resolution

    You can verify DNS configuration with the following command:

    dig hostName

    You can verify network connectivity with the following command:

    telnet hostName port

Scenario 2: Requests fail at the Apigee Edge Router

Scenario 3: Requests cannot be processed by Message Processor

  • Cause

    In this scenario, the Apigee Message Processor cannot find the API Proxy for the specified virtual host and path. As a result, you may see one of the following errors:

    HTTP/1.1 404 Not Found
    
    {
      "fault":{
        "faultstring":"Unable to identify proxy for host: default and url: \/apiProxyBasePath/requestPath",
        "detail":{
          "errorcode":"messaging.adaptors.http.flow.ApplicationNotFound"
        }
      }
    }
    
    
  • Resolution

    Refer to this playbook to troubleshoot and resolve this issue: 404 Unable to identify proxy for host.

Cause: API Proxy Not Found in The Classification Tree

Diagnosis

If an API proxy is not found in the Classification Tree of one or more Message Processors, then any API requests to that specific proxy will not be shown in Trace sessions in the Edge UI.

Follow the steps below to determine if this is the case:

  1. Log in to each of the Message Processors and check to see if the specific revision of the requested API is deployed in the relevant environment of the Message Processor using the following command:

    curl -v http://localhost:8082/v1/runtime/organizations/orgName/environments/envName/apis/apiName/revisions
    

    Example Output:

    The command above will output a list of deployed revisions. For example, if revision 12 is deployed then you will see the following output:

    [ "12" ]
    

    Unless you are encountering intermittent HTTP 404 errors, you are likely to see that the specific revision is deployed.

  2. Read the Classification Tree and check for the existence of the API proxy name using the following command:

    curl -i http://localhost:8082/v1/classification/tree | grep apiName
    
  3. Repeat Steps 1 and 2 for each Message Processor. If the given API proxy name is not found in the Classification Tree on any of the Message Processors, then follow the resolution given below.

Resolution

Please follow the steps below to resolve this issue. Make sure to take any precautions necessary to avoid production outages that may occur from restarting Message Processors while experiencing high request loads.

  1. Log in to each of the Message Processor hosts that are missing the specific API proxy in the Classification Tree and use the command below to restart the Message Processor:

    /opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart
    
  2. Once restarted, use the command below to wait until it becomes active:

    /opt/apigee/apigee-service/bin/apigee-service edge-message-processor wait_for_ready
    
  3. Once the Message Processor is ready, verify the availability of the API proxy using the following command:

    curl -v http://localhost:8082/v1/runtime/organizations/orgName/environments/envName/apis/apiName/revisions
    

    Example Output:

    The command above will output a list of deployed revisions. For example, if revision 12 is deployed then you will see the following output:

    [ "12" ]
    

    Unless you are encountering intermittent HTTP 404 errors, you are likely to see that the specific revision is deployed.

  4. Read the Classification Tree and verify the existence of the API proxy name using the following command:

    curl -i http://localhost:8082/v1/classification/tree | grep apiName
    

    If the problem still persists, go to Must Gather Diagnostic Information.

Must Gather Diagnostic Information

If the problem persists after following the instructions above, please gather the following diagnostic information and share with Apigee Support:

  1. Output of the command:

    curl -v management-server-host:8080/v1/runtime/organizations/orgName/environments/envName/apis/apiProxyName/revisions/revisionNumber/debugsessions -u user
    
  2. Management Server log:

    /opt/apigee/var/log/edge-management-server/logs/system.log
    
  3. Message Processor logs:

    /opt/apigee/var/log/edge-message-processor/logs/system.log
    
  4. Output of telnet/nc commands from Management Server to the Message Processor:

    telnet  8082
    nc -vz  8082
    
  5. Output of the below netstat command on the Message Processor(s):

    netstat -an > netstat.txt
    
  6. Output of the following command listing the revisions deployed for the specific API Proxy on all the Message Processor(s):

    curl -v http://localhost:8082/v1/runtime/organizations/orgName/environments/envName/apis/apiName/revisions
    
  7. Classification tree output on all the Message Processor(s):

    curl -i http://localhost:8082/v1/classification/tree