You're viewing **Apigee Edge** documentation. [Go to the **Apigee X** documentation](http://cloud.google.com/apigee/docs/hybrid/v1.9/diagnostic-collector). info This topic discusses network tools, JVM tools, and diagnostic logs that you can use to troubleshoot network and JVM-related issues on Apigee Edge. ## TCP/IP packet sniffer (tcpdump) utility The **tcpdump** tool is a command-line packet sniffer tool that allows you to capture or filter TCP/IP packets that are received or transferred over a network. It is available on Linux/Unix based operating systems. You can install this utility using yum as follows: ``` yum install tcpdump ``` The tcpdump tool is useful for troubleshooting network or SSL related issues. For example: - 502 Bad Gateway Errors (caused due to EOF Exception) - 503 Service Unavailable Errors - SSL Handshake Failures To troubleshoot any of these problems, you must first determine the pair of components between which the error occurred. In the case of Edge, it can be one of the following pairs: - Client app and Router - Router and Message Processor - Message Processor and Backend server Once you identify the troublesome pair of components, you can capture the network packets using tcpdump on one or both of these components. ### Capturing packets sent to/received from a specific host using tcpdump Use the following tcpdump command to capture all of the packets sent to or received from a specified host (IP address) and save the information in the specified file: ``` tcpdump -i any -s 0 host -w ``` Where: |---|---| | **Parameter** | **Description** | | -i | (interface) specifies the interface from which the packets should be captured. Using the value of "any" allows to capture packets from all interfaces. | | -s | (snarf/snaplen) specifies the amount of each packet to capture. Using the value of 0 (zero) allows you to capture the entire packet. | | IP address | is the ip address of the host for which we want to capture the packets | | File Name | is the name of the file to which tcpdump has to be written to | ### Example Let's say you want to capture the packets between the Message Processor and Backend Server: 1. Log in to the Message Processor machine. 2. Determine the IP address of the Backend Server (assume it is 22.22.22.22) for which we want to capture the packets. Use the following command to capture the network packets for a host with a specific IP address: ``` tcpdump -i any -s 0 host 22.22.22.22 -w rmp-123.pcap ``` If the backend server resolves to multiple IP addresses, then use the hostname of the backend server in the tcpdump command as shown below: ``` tcpdump -i any -s 0 host -w rmp-123.pcap ``` If there are multiple backend servers with different IP addresses (22.22.22.22, 33.33.33.33 and 44.44.44.44), then use the tcpdump command below: ``` tcpdump -i any -s 0 host 22.22.22.22 or host 33.33.33.33 or host 44.44.44.44 -w rmp-123.pcap ``` ### Analyzing tcpdumps You can view or analyze tcpdumps using the tcpdump command or the GUI based tool [Wireshark](https://www.wireshark.org/). ### References - [tcpdump man page](https://www.google.com/url?q=https://linux.die.net/man/8/tcpdump&sa=D&ust=1518037574623000&usg=AFQjCNHV60fl3UaYXG-GDp_iIkEtDfY83Q) - [TCPdump commands - A Network Sniffer Tool](https://www.google.com/url?q=https://www.tecmint.com/12-tcpdump-commands-a-network-sniffer-tool/&sa=D&ust=1518037574623000&usg=AFQjCNHJHkdDkvou3ijOTmzP1CbGUIm3hw) - [Wireshark](https://www.wireshark.org/) ## Heap dumps Heap dumps are a snapshot of the memory of a Java process. They contain the information about the Java objects and classes in the heap at the moment the heap dump is collected. They are usually pretty large in size ranging anywhere between few 100MBs to few GBs. Heap dump is very useful when a Java process such as Message Processor shows: - High Memory Usage - OutofMemoryError ### Generating Heap dump for a Java process Java provides a utility called **jmap**, which allows you to generate the memory statistics or heap dumps of a running Java process. Use the following jmap command to generate the heap dump of a Java process: ``` sudo -u apigee /bin/jmap -dump:live,format=b,file= ``` Where: |---|---| | **Parameter** | **Description** | | JAVA_HOME | Is the installation directory of Java | | filename | Is the file name to which the heap dump will be written | | pid | Is the process id of the Java application whose head dump has to be captured | ### Example Let's say the Message Processor mp-east has high memory usage or is throwing OutOfMemory Errors. Determine the process id of the Message Processor using ps command on Unix operating system. Let's say it is 24569. Run the jmap utility as follows to generate the heap dump: ``` sudo -u apigee /bin/jmap -dump:live,format=b,file=mp-east-heapdump.bin 24569 ``` ### Analyzing Heap Dumps Heap Dumps can be analysed using Eclipse MAT (Memory Analyzer Tool) to determine the potential memory leaks or which Java objects are leading to high memory usage. ### References [How to collect a heap dump](https://www.google.com/url?q=https://blogs.oracle.com/luzmestre/how-to-collect-a-heap-dump&sa=D&ust=1518037574628000&usg=AFQjCNFfKQ9mRAZLRuBrwJiSyolKhHitgQ) [jmap utility](https://www.google.com/url?q=https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr014.html&sa=D&ust=1518037574629000&usg=AFQjCNGx9oybyfXpF6F-sER24UOnJV3vpA) [jmap man page](https://www.google.com/url?q=https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jmap.html&sa=D&ust=1518037574629000&usg=AFQjCNFQYZQ0bwjhYmZZtNdFrDfNa9S3JA) [Memory Analyzer Tool (MAT)](https://www.google.com/url?q=http://www.eclipse.org/mat/&sa=D&ust=1518037574629000&usg=AFQjCNELLNXHAygSitTJQO3YIQp5agEeug) *** ** * ** *** ## Thread dumps A thread dump is a snapshot of the state of all the threads of a running Java process. The state of each thread is presented with the contents of its stack, referred to as a stack trace. Some of the threads will be part of the Java application that is running, while others will be JVM internal threads. A thread dump reveals information about each of the application's threads activities. This information can be very useful to: - Diagnose problems such as CPU spikes, slow response times, or unresponsive Java applications - Optimize application and JVM performance ### Generating Thread Dumps The thread dump for a Java process can be generated using the jstack utility as shown below: ``` sudo -u apigee /bin/jstack -l > ``` Where: |---|---| | **Parameter** | **Description** | | JAVA_HOME | Is the installation directory of Java. | | pid | Is the process id of the Java application whose thread dump you want to capture. | | filename | Is the file name to which the thread dump will be written. | ### Example To generate a thread dump for the process ID 37320 on Message Processor **mp-east**, use the following command: ``` sudo -u apigee /bin/jstack -l 37320 > /opt/apigee/edge-message-processor/mp-east-threadDump.txt ``` As per the above example, thread dump of the process would be saved to the /opt/apigee/edge-message-processor/mp-east-threadDump.txt file. ### Analyzing Thread Dumps You can view thread dumps in any text editor such as vi (Linux), notepad (Windows). Refer to [Thread Dump](https://www.google.com/url?q=https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr019.html%23BABBEFFC&sa=D&ust=1518037574634000&usg=AFQjCNEBSj8WSlDeN0D0bBiemx1UC28-Hg) for details on different sections of Thread Dump and how to interpret the information. > [!IMPORTANT] > **Tip:** Use top **-h** output along with thread dump to troubleshoot high CPU usage issues. ### References - [jstack utility](https://www.google.com/url?q=https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr016.html&sa=D&ust=1518037574635000&usg=AFQjCNGM-SqrsFf8BCw1I927HtZwanMJDw) - [Analyzing Thread Dump: CPU High Usage Issue](https://www.google.com/url?q=https://blogs.oracle.com/jiechen/analysis-against-jvm-thread-dump-cpu-high-usage-issue&sa=D&ust=1518037574635000&usg=AFQjCNGYv3n7Zu_vQ_DUBTxL3ca1HgC3tw) - [How to Analyze Java Thread Dumps](https://www.google.com/url?q=https://dzone.com/articles/how-analyze-java-thread-dumps&sa=D&ust=1518037574635000&usg=AFQjCNFkZeBObjIS1NuqwRqJnDnIaZflkQ)