Writing analytics data to a file

By default, analytics data collected by the Message Processor is uploaded to Qpid and Postgres for processing. You can then view the analytics data in the Edge UI.

Alternatively, you can configure the Message Processor to write analytics data to disk. Then, you can upload that data to your own analytics system for analysis. For example, you could upload the data to Google Cloud BigQuery. You can then take advantage of the powerful query and machine learning capabilities offered by BigQuery and TensorFlow to perform your own data analysis.

You can also choose to use both options. That means you can upload the analytics data to Qpid/Postgres and also save the data to disk.

File names and location

By default, if you enable the writing of analytics data to disk files, the files are written to the following directory:

/opt/apigee/var/log/edge-message-processor/ax/tmp

Edge creates a new directory under /tmp for the data files, at one minute intervals. The format of the directory name is:

org~env~yyyyMMddhhmmss

For example:

myorg~prod~20190909163500
myorg~prod~20190909163600

Each directory contains a .gz file with the individual data files for that interval. The format of the .gz file name is:

4DigitRandomHex_StartTime.StartTimePlusInterval_internalHostIP_hostUUID_writer_index.txt.gz

At regular intervals, Edge moves the directory and the .gz file it contains from /tmp to either of the following directories, based on the setting of the uploadToCloud Message Processor configuration property:

uploadToCloud = false: files moved to /opt/apigee/var/log/edge-message-processor/ax/staging
uploadToCloud = true: (default): files move to /opt/apigee/var/log/edge-message-processor/ax/failed

Unzip the data from either the /staging or /failed directory to obtain the analytics data files.

Configuration properties

Use the following properties to configure the Message Processor to write analytics data to disk. All of these properties are optional:

Property	Description
`conf_analytics_analytics.saveToDisk`	Set to `true` to configure the Message Processor to write analytics data to disk files. The default value is `false`.
`conf_analytics_analytics.sendToQueue`	Set to `true` to configure the Message Processor upload the data to Qpid/Postgres. Set to `false` to disable the writing of analytics data to Qpid/Postgres. The default value is `true`.
`conf_analytics_analytics.baseDataDirectoryPath`	Specifies the base path where the analytics data files are written. The default value is `/opt/apigee/var/log/edge-message-processor/ax.`
`conf_analytics_analytics.allocatedDiskSpaceInMBytes`	Specifies the disk space, in megabytes, allocated for analytics files. The default value is `3072`. If you exceed the allocated disk space for the analytics data files, the Message Processor stops saving analytics data and writes an error message to its log files.
`conf_analytics_analytics.uploadToCloud`	Controls the final location of the analytics files. `false`: files moved to `/opt/apigee/var/log/edge-message-processor/ax/staging` `true` (default): files move to `/opt/apigee/var/log/edge-message-processor/ax/failed` Note: Even though this property is called `uploadToCloud`, no data is uploaded to the cloud. All analytics data remains local to your installation.

To set these properties:

Open the message-processor.properties file in an editor. If the file does not exist, create it:
```
vi /opt/apigee/customer/application/message-processor.properties
```

Set the properties as desired:

# Enable writing analytics data to disk.
conf_analytics_analytics.saveToDisk=true
# Disable writing analytics data to Qpid/Postgres.
conf_analytics_analytics.sendToQueue=false

# Specify base directory for analytics data files.
conf_analytics_analytics.baseDataDirectoryPath=/opt/apigee/var/smg

# Set the disk space available for analytics files.
conf_analytics_analytics.allocatedDiskSpaceInMBytes=3072
# Move final analytics data to files to the /staging directory.
conf_analytics_analytics.uploadToCloud=false

Save your changes.

Make sure the properties file is owned by the "apigee" user:

chown apigee:apigee /opt/apigee/customer/application/message-processor.properties

Set the value of the consumer-type property to ax for the axgroup-001 analytics group:
```
curl -X POST -H "Content-Type:application/json" \
"http://ms-ip:8080/v1/analytics/groups/ax/axgroup-001/properties?propName=consumer-type&propValue=ax" \
-u sysAdminEmail:sysAdminPWord
```
By default, the name of the analytics group is axgroup-001. In the config file for an Edge installation, you can set the name of the analytics group by using the AXGROUP property. If you are unsure of the names of the analytics group, run the following command on the Management Server node to display it:
```
apigee-adminapi.sh analytics groups list \
  --admin sysAdminEmail --pwd sysAdminPword --host localhost
```
This command returns the analytics group name in the name field.
Restart the Message Processor:
```
/opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart
```
After the restart, the Message Processor writes analytics data to data files.
Repeat these steps for all Message Processors.