Writing analytics data to a file

By default, analytics data collected by the Message Processor is uploaded to Qpid and Postgres for processing. You can then view the analytics data in the Edge UI.

Alternatively, you can configure the Message Processor to write analytics data to disk. Then, you can upload that data to your own analytics system for analysis. For example, you could upload the data to Google Cloud BigQuery. You can then take advantage of the powerful query and machine learning capabilities offered by BigQuery and TensorFlow to perform your own data analysis.

You can also choose to use both options. That means you can upload the analytics data to Qpid/Postgres and also save the data to disk.

File names and location

By default, if you enable the writing of analytics data to disk files, the files are written to the following directory:

/opt/apigee/var/log/edge-message-processor/ax/tmp

Edge creates a new directory under /tmp for the data files, at one minute intervals. The format of the directory name is:

org~env~yyyyMMddhhmmss

For example:

myorg~prod~20190909163500
myorg~prod~20190909163600

Each directory contains a .gz file with the individual data files for that interval. The format of the .gz file name is:

4DigitRandomHex_StartTime.StartTimePlusInterval_internalHostIP_hostUUID_writer_index.txt.gz

At regular intervals, Edge moves the directory and the .gz file it contains from /tmp to either of the following directories, based on the setting of the uploadToCloud Message Processor configuration property:

• uploadToCloud = false: files moved to /opt/apigee/var/log/edge-message-processor/ax/staging
• uploadToCloud = true: (default): files move to /opt/apigee/var/log/edge-message-processor/ax/failed

Unzip the data from either the /staging or /failed directory to obtain the analytics data files.

Configuration properties

Use the following properties to configure the Message Processor to write analytics data to disk. All of these properties are optional:

To set these properties:

1. Open the message-processor.properties file in an editor. If the file does not exist, create it:

   vi /opt/apigee/customer/application/message-processor.properties

2. Set the properties as desired:

   # Enable writing analytics data to disk.
   conf_analytics_analytics.saveToDisk=true
   # Disable writing analytics data to Qpid/Postgres.
   conf_analytics_analytics.sendToQueue=false
   
   # Specify base directory for analytics data files.
   conf_analytics_analytics.baseDataDirectoryPath=/opt/apigee/var/smg
   
   # Set the disk space available for analytics files.
   conf_analytics_analytics.allocatedDiskSpaceInMBytes=3072
   # Move final analytics data to files to the /staging directory.
   conf_analytics_analytics.uploadToCloud=false

3. Save your changes.

4. Make sure the properties file is owned by the "apigee" user:

   chown apigee:apigee /opt/apigee/customer/application/message-processor.properties

5. Set the value of the consumer-type property to ax for the axgroup-001 analytics group:

   curl -X POST -H "Content-Type:application/json" \
   "http://ms-ip:8080/v1/analytics/groups/ax/axgroup-001/properties?propName=consumer-type&propValue=ax" \
   -u sysAdminEmail:sysAdminPWord

   By default, the name of the analytics group is axgroup-001. In the config file for an Edge installation, you can set the name of the analytics group by using the AXGROUP property. If you are unsure of the names of the analytics group, run the following command on the Management Server node to display it:

   apigee-adminapi.sh analytics groups list \
     --admin sysAdminEmail --pwd sysAdminPword --host localhost

   This command returns the analytics group name in the name field.

6. Restart the Message Processor:

   /opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart

   After the restart, the Message Processor writes analytics data to data files.

7. Repeat these steps for all Message Processors.