Diagnostic tools and logs

This topic discusses network tools, JVM tools, and diagnostic logs that you can use to troubleshoot network and JVM-related issues on Apigee Edge.

TCP/IP packet sniffer (tcpdump) utility

The tcpdump tool is a command-line packet sniffer tool that allows you to capture or filter TCP/IP packets that are received or transferred over a network. It is available on Linux/Unix based operating systems. You can install this utility using yum as follows:

yum install tcpdump

The tcpdump tool is useful for troubleshooting network or SSL related issues. For example:

  • 502 Bad Gateway Errors (caused due to EOF Exception)
  • 503 Service Unavailable Errors
  • SSL Handshake Failures

To troubleshoot any of these problems, you must first determine the pair of components between which the error occurred. In the case of Edge, it can be one of the following pairs:

  • Client app and Router
  • Router and Message Processor
  • Message Processor and Backend server

Once you identify the troublesome pair of components, you can capture the network packets using tcpdump on one or both of these components.

Capturing packets sent to/received from a specific host using tcpdump

Use the following tcpdump command to capture all of the packets sent to or received from a specified host (IP address) and save the information in the specified file:

tcpdump -i any -s 0 host <IP address> -w <File name>

Where:

Parameter Description
-i (interface) specifies the interface from which the packets should be captured. Using the value of “any” allows to capture packets from all interfaces.
-s (snarf/snaplen) specifies the amount of each packet to capture. Using the value of 0 (zero) allows you to capture the entire packet.
IP address is the ip address of the host for which we want to capture the packets
File Name is the name of the file to which tcpdump has to be written to

Example

Let’s say you want to capture the packets between the Message Processor and Backend Server:

  1. Log in to the Message Processor machine.
  2. Determine the IP address of the Backend Server (assume it is 22.22.22.22) for which we want to capture the packets.

Use the following command to capture the network packets for a host with a specific IP address:

tcpdump -i any -s 0 host 22.22.22.22 -w rmp-123.pcap

If the backend server resolves to multiple IP addresses, then use the hostname of the backend server in the tcpdump command as shown below:

tcpdump -i any -s 0 host <Hostname> -w rmp-123.pcap

If there are multiple backend servers with different IP addresses (22.22.22.22, 33.33.33.33 and 44.44.44.44), then use the tcpdump command below:

tcpdump -i any -s 0 host 22.22.22.22 or host 33.33.33.33 or host 44.44.44.44 -w rmp-123.pcap

Analyzing tcpdumps

You can view or analyze tcpdumps using the tcpdump command or the GUI based tool Wireshark.

References

Heap dumps

Heap dumps are a snapshot of the memory of a Java process. They contain the information about the Java objects and classes in the heap at the moment the heap dump is collected. They are usually pretty large in size ranging anywhere between few 100MBs to few GBs.

Heap dump is very useful when a Java process such as Message Processor shows:

  • High Memory Usage
  • OutofMemoryError

Generating Heap dump for a Java process

Java provides a utility called jmap, which allows you to generate the memory statistics or heap dumps of a running Java process.

Use the following jmap command to generate the heap dump of a Java process:

sudo -u apigee <JAVA_HOME>/bin/jmap -dump:live,format=b,file=<filename> <pid>

Where:

Parameter Description
JAVA_HOME Is the installation directory of Java
filename Is the file name to which the heap dump will be written
pid Is the process id of the Java application whose head dump has to be captured

Example

Let’s say the Message Processor mp-east has high memory usage or is throwing OutOfMemory Errors. Determine the process id of the Message Processor using ps command on Unix operating system. Let’s say it is 24569.

Run the jmap utility as follows to generate the heap dump:

sudo -u apigee <JAVA_HOME>/bin/jmap -dump:live,format=b,file=mp-east-heapdump.bin 24569

Analyzing Heap Dumps

Heap Dumps can be analysed using Eclipse MAT (Memory Analyzer Tool) to determine the potential memory leaks or which Java objects are leading to high memory usage.

References

How to collect a heap dump

jmap utility

jmap man page

Memory Analyzer Tool (MAT)


Thread dumps

A thread dump is a snapshot of the state of all the threads of a running Java process. The state of each thread is presented with the contents of its stack, referred to as a stack trace. Some of the threads will be part of the Java application that is running, while others will be JVM internal threads.

A thread dump reveals information about each of the application’s threads activities. This information can be very useful to:

  • Diagnose problems such as CPU spikes, slow response times, or unresponsive Java applications
  • Optimize application and JVM performance

Generating Thread Dumps

The thread dump for a Java process can be generated using the jstack utility as shown below:

<JAVA_HOME>/bin/jstack -l <pid> > <filename>

Where:

Parameter Description
JAVA_HOME Is the installation directory of Java.
pid Is the process id of the Java application whose thread dump you want to capture.
filename Is the file name to which the thread dump will be written.

Example

To generate a thread dump for the process ID 37320 on Message Processor mp-east, use the following command:

<JAVA_HOME>/bin/jstack -l 37320 > /opt/apigee/edge-message-processor/mp-east-threadDump.txt

As per the above example, thread dump of the process would be saved to the /opt/apigee/edge-message-processor/mp-east-threadDump.txt file.

Analyzing Thread Dumps

You can view thread dumps in any text editor such as vi (Linux), notepad (Windows). Refer to Thread Dump for details on different sections of Thread Dump and how to interpret the information.

References