Sosreport tool

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Sosreport is a system-level diagnostic and data collection tool, based on the open source software SoS. You can use the Sosreport tool to collect information to help you debug problems in nodes where an Apigee component is running. The tool collects and stores information locally on the node's file system.

The Sosreport tool collects configuration details, system information, and diagnostic information from your system, and stores the data in an archive. For Apigee components, Sosreport collects the following:

  • The running kernel version
  • Loaded modules
  • System and service configuration files

Sosreport also runs external programs to collect further information, including:

  • Apigee logs from all components.
  • Runtime diagnostics
  • Cassandra diagnostics collected using nodetool commands.

The Sosreport tool contains two main commands:

  • sosreport: The primary command for collecting data and creating reports.
  • apigee-sosreport: An auxiliary tool used to install sosreport and manage sosreport plugins specific to Apigee components.

Installation

This section explains how to install sosreport.

If the SoS version of sosreport is already installed

If the original SoS version of sosreport is already installed on your system, do not set up Apigee's version, as the installation paths may conflict. Instead, copy the Apigee-specific plugins from the directory /opt/apigee/apigee-sosreport/source/plugin to the appropriate Python site-packages directory using a command like the ones shown below. For example, to copy agigee-mgmt.py:

  • If you're using Python 2, enter: 
    sudo cp apigee-mgmt.py /usr/lib/python2.7/site-packages/sos/plugins/
  • If you're using Python 3, enter: 
    sudo cp apigee-mgmt.py /usr/local/lib/python3.6/site-packages/sos/plugins/

To verify that the plugins have been installed, enter

sudo sosreport --list-plugins

If you have questions, contact Apigee support.

Installing Apigee's version of sosreport

If the SoS version of sosreport is not already installed on your system, you can install Apigee's version by entering the following commands on all nodes where Edge for Private Cloud is installed: 

/opt/apigee/apigee-service/bin/apigee-service apigee-sosreport install
/opt/apigee/apigee-service/bin/apigee-service apigee-sosreport setup

To test that sosreport is installed, confirm that the plugins related to Apigee are located in the following directory:

/usr/lib/pythonx.x/site-packages/sos/plugins

where pythonx.x is your installed version of Python.

Alternatively, you can run one of the following commands:

  • To verify Apigee-specific plugins only: 
    apigee-service apigee-sosreport diagnose -l

    This returns the following list of Apigee-specific plugins.

    sosreport (version 3.9)

The following plugins are currently enabled:

apigee-mgmt          This is the apigee plugin for edge-management-server
apigee-mp            This is the apigee plugin for edge-message-processor
apigee-rrt           This is the apigee plugin for edge-router
apigee-sample        Main diagnostics gatehring class
  • To verify all plugins: 
    apigee-service apigee-sosreport diagnose -s -l

    This returns the following list of plugins available with sosreport:

    sosreport (version 3.9)

The following plugins are currently enabled:

 acpid                ACPI daemon information
 alternatives         System alternatives
 anacron              Anacron job scheduling service
 apigee-cassandra     This is the base class for sosreport plugins. Plugins should subclass
    this and set the class variables where applicable. ←truncated→
 apigee-mgmt          This is the base class for sosreport plugins. Plugins should subclass
    this and set the class variables where applicable. ←truncated→
 apigee-mp            Main diagnostics gathering class
 apigee-rrt           This is the base class for sosreport plugins. Plugins should subclass
    this and set the class variables where applicable. ←truncated→

………………………………………………….
………………………………………………….
………………………………………………….
←truncated→

Creating a report

To create a report using the Sosreport tool, run the sosreport command with the required plugin option (see Apigee plugins). The default directory where the report is stored is either tmp or /var/tmp. You can change the directory where the report is stored by passing a different directory to the tmp-dir option. For example, to create a report on the Management Server and save it a directory named my_report-dir, enter one of the following:

  • Using Apigee's version of sosreport: 
    sudo sosreport --config-file /opt/apigee/apigee-sosreport/conf/sos.conf -o apigee-mgmt  --batch --tmp-dir=my_report-dir
  • Using the SoS version of sosreport: 
    sudo sosreport -o apigee-mgmt  --batch --tmp-dir=my_report-dir

The report is stored in a .tar.xz file in the directory you specify using the tmp-dir option. The location of the .tar.xz file is shown on STDOUT, as well as the checksum. For example:

Your sosreport has been generated and saved in:
 /var/sosreport-prc-test-0-9613-2021-07-12-orwxufx.tar.xz
The checksum is: 5a8b97c6020346a688254c8b04ef86ec
For more available options for sosreport, read the man file:
sudo man sosreport

The report is stored in a .tar.xz file in the directory you specify using the tmp-dir option. The location of the .tar.xz file is shown on STDOUT, as well as the checksum. For example:

For more available options for sosreport, read the man file:

sudo man sosreport

Apigee plugins

The following sections describe Apigee plugins to the Sosreport tool for the following components:

Management Server

The apigee-mgmt plugin collects the following information from the Management Server:

  • server hostname (default)
  • cpuinfo (default)
  • meminfo (default)
  • Java virtual machine (JVM) settings, such as memory, Djdk.tls.allowUnsafeServerCertChange, and so on (default)
  • Current Management logs (default)
  • Ten sequential top and jstacks output (default)
  • Entire Management log folder (optional)
  • config-files (optional)

Examples:

To collect all default items, enter one of the following:

  • Using Apigee's version of sosreport: 
    sudo sosreport --config-file /opt/apigee/apigee-sosreport/conf/sos.conf  -o apigee-mgmt --batch my_report-dir
  • Using the SoS version of sosreport: 
    sudo sosreport -o apigee-mgmt --batch my_report-dir

To collect all default items and also the optional item(s) specified in the command, for example, configuration files and all logs, enter one of the following:

  • Using Apigee's version of sosreport: 
    sudo sosreport --config-file /opt/apigee/apigee-sosreport/conf/sos.conf  -o apigee-mgmt --batch -k apigee-mgmt.config-files -k apigee-mgmt.all-logs
  • Using the SoS version of sosreport: 
    sudo sosreport -o apigee-mgmt --batch -k apigee-mgmt.config-files -k apigee-mgmt.all-logs

To collect everything, enter one of the following:

  • Using Apigee's version of sosreport: 
    sudo sosreport --config-file /opt/apigee/apigee-sosreport/conf/sos.conf  -o apigee-mgmt --batch -k apigee-mgmt.config-files -k apigee-mgmt.all-logs

    Alternatively, you can enter:

    sudo sosreport --config-file /opt/apigee/apigee-sosreport/conf/sos.conf  -o apigee-mgmt --batch -a
  • Using the SoS version of sosreport: 
    sudo sosreport -o apigee-mgmt --batch -k apigee-mgmt.config-files -k apigee-mgmt.all-logs

    Alternatively, can enter:

    sudo sosreport -o apigee-mgmt --batch -a

Message Processor

The apigee-mp plugin collects the following information from the Management Processor (MP):

  • Server hostname (default)
  • cpuinfo (default)
  • meminfo (default)
  • buildinfo/release info (default)
  • apigee-all status (default)
  • server details (default)
  • JVM level settings, for example, memory, Djdk.tls.allowUnsafeServerCertChange, etc. (default)
  • Current MP logs (default)
  • Ten sequential top, jstacks and NIO Metrics output (default)
  • Entire MP log folder (optional)
  • Classification tree (optional)
  • Heap dump (optional)
  • config-files (optional)
  • Runtime trace (optional and only when org:env:api:rev detail is provided). Note: Plugin waits for 25 seconds to collect the runtime requests.

Examples:

To collect all default items, enter one of the following:

  • Using Apigee's version of sosreport: 
    sudo sosreport --config-file /opt/apigee/apigee-sosreport/conf/sos.conf  -o apigee-mp --batch
  • Using the SoS version of sosreport: 
    sudo sosreport -o apigee-mp --batch

You can also specify additional optional items to collect, in addition to all default items. For example, to specify the heap dump, using the flag -k apigee-mp.heap, enter one of the following:

  • Using Apigee's version of sosreport: 
    sudo sosreport --config-file /opt/apigee/apigee-sosreport/conf/sos.conf  -o apigee-mp --batch -k apigee-mp.heap
  • Using the SoS version of sosreport: 
    sudo sosreport -o apigee-mp --batch -k apigee-mp.heap

Similarly, you can specify the following optional items:

-k apigee-mp.deployments
-k apigee-mp.heap
-k apigee-mp.all-logs
-k apigee-mp.config-files
-k apigee-mp.trace=gsc-cps:test:httpbin:3

You can specify multiple options in a single command. For example:

  • Using Apigee's version of sosreport: 
    sudo sosreport --config-file /opt/apigee/apigee-sosreport/conf/sos.conf  -o apigee-mp --batch -k apigee-mp.deployments -k apigee-mp.all-logs
  • Using the SoS version of sosreport: 
    sudo sosreport -o apigee-mp --batch -k apigee-mp.deployments -k apigee-mp.all-logs

To collect everything except trace, enter one of the following:

  • Using Apigee's version of sosreport: 
    sudo sosreport --config-file /opt/apigee/apigee-sosreport/conf/sos.conf  -o apigee-mp --batch -a
  • Using the SoS version of sosreport: 
    sudo sosreport -o apigee-mp --batch -a

Router

The apigee-rrt plugin collects the following information from the Router:

  • Server hostname (default)
  • cpuinfo (default)
  • meminfo (default)
  • buildinfo/release info (default)
  • apigee-all status (default)
  • server details (default)
  • ps output. This will give JVM level settings, for example, memory, Djdk.tls.allowUnsafeServerCertChange, and so on (default).
  • Current Router logs (default)
  • List of vhost file names including bad file names (default)
  • Entire Router log folder (optional)
  • config-files (optional)

Examples:

To collect all default items, enter one of the following:

  • Using Apigee's version of sosreport: 
    sudo sosreport --config-file /opt/apigee/apigee-sosreport/conf/sos.conf  -o apigee-rrt --batch
  • Using the SoS version of sosreport: 
    sudo sosreport -o apigee-rrt --batch

You can also specify additional optional items to collect, in addition to all default items. For example, to specify the entire log folder, using the flag -k apigee-rrt.all-logs, enter one of the following:

  • Using Apigee's version of sosreport: 
    sudo sosreport --config-file /opt/apigee/apigee-sosreport/conf/sos.conf  -o apigee-rrt --batch -k apigee-rrt.all-logs
  • Using the SoS version of sosreport: 
    sudo sosreport -o apigee-rrt --batch -k apigee-rrt.all-logs

Similarly, you can specify the following optional items:

-k apigee-rrt.all-logs
-k apigee-rrt.config-files

To collect everything:

  • Using Apigee's version of sosreport: 
    sudo sosreport --config-file /opt/apigee/apigee-sosreport/conf/sos.conf  -o apigee-rrt --batch -a
  • Using the SoS version of sosreport: 
    sudo sosreport -o apigee-rrt --batch -a

Cassandra

The apigee-cassandra plugin collects the following information from Cassandra nodes:

  • Machine statistics: 
    "hostname -i"
"cat /proc/cpuinfo"
"cat /proc/meminfo"
"ulimit -a"
"ps auxw"
"df -kh"
"du -sh"
"netstat -ntlpu"
"free -h"
"ifconfig -h"
"ps -ef"
  • Apigee component's status and version apigee-all version && apigee-all status
  • apigee-cassandra logs: All files under /opt/apigee/var/log/apigee-cassandra/
  • Commit log count
  • Output of command Top
  • Thread dump
  • Configuration files: cassandra-topology.properties, cassandra.yaml, cassandra-env.sh
  • Nodetool Output for the following parameters: version, status, ring, info, gossipinfo, compactionstats -H, tpstats, netstats, cfstats, proxyhistograms

Example:

To collect all default items, enter one of the following:

  • Using Apigee's version of sosreport: 
    sudo sosreport --config-file /opt/apigee/apigee-sosreport/conf/sos.conf  -o apigee-cassandra
  • Using the SoS version of sosreport: 
    sudo sosreport -o apigee-cassandra