Uploading API Traffic Data to Apigee - Beta Release

Edge for Private Cloud v. 4.17.09

All Edge for Private Cloud customers are required to submit to Apigee statistics about API proxy traffic. Apigee recommends that customers upload that information once a day, possibly by creating a cron job.

You must submit the data for your production API deployments, but not for APIs in development or testing deployments. In most Edge installations, you will define specific organizations or environments for your production APIs. The data that you submit is only for those production organizations and environments.

To assist in uploading this data, Apigee provides the Beta release of the apigee-analytics-collector command-line utility. This utility sends the API call volume report back to Apigee. Every Edge for the Private Cloud installation can use this utility to retrieve and report traffic data to Apigee.

Accessing the data uploaded by apigee-analytics-collector

After uploading data with the apigee-analytics-collector utility, you can view it in Apigee 360.

To view the data, log in to Apigee 360 and then select the Usage option. The Usage page displays a usage graph and a table of usage data for each or your Edge organizations.

Installing apigee-analytics-collector

The apigee-analytics-collector utility is an RPM that you install by using the apigee-service utility.

Where to install

The node on which you install the apigee-analytics-collector utility can be any node that can access the Edge management API on the Edge Management Server. You can install it directly on the Management Server, on another node of the Edge installation, or on a separate node as long as that node can make API requests to the Management Server.

Internet access requirements

Install the apigee-analytics-collector utility on a machine with external internet access. Then the apigee-analytics-collector utility can upload data directly to Apigee.

If there is no node with both access to the Edge management API on the Edge Management Server and with external internet access, then you can use the Edge management API to save the traffic data locally. You must then transfer the data to a machine with internet access for upload to Apigee. In this scenario, you do not need to use the apigee-analytics-collector utility. See Manually uploading data to Apigee below for more.

Installation

Use the following command to install the apigee-analytics-collector utility. Because you are installing an RPM file, this command must be run by the root user or by a user that has full sudo access. For full sudo access, that means the user has sudo access to perform the same operations as root.

> /opt/apigee/apigee-service/bin/apigee-service apigee-analytics-collector install

Running apigee-analytics-collector

This section describes how to run the apigee-analytics-collector utility.

Configuring a user to run apigee-analytics-collector

You must run apigee-analytics-collector as a non-root user. This user must have full sudo access to the "apigee" user.

To configure a user to have full sudo access to the "apigee" user, use the "visudo" command to edit the sudoers file to add:

analyticsUser        ALL=(apigee)      NOPASSWD: ALL

where analyticsUser is the username of the person running the apigee-analytics-collector utility.

After installing the apigee-analytics-collector utility and configuring the user, you can test the utility by running the help command for the apigee-analytics-collector utility:

> /opt/apigee/apigee-service/bin/ apigee-service apigee-analytics-collector export traffic --help

Required information to run apigee-analytics-collector

You need the following information to run the apigee-analytics-collector command and to forward data to Apigee:

  • apigee_mgmt_api_uri: The base URL of the Edge API on your Management Server. This URL is typically in the form:
    http://ms_IP:8080/v1

    Where ms_IP is the IP address or your Management Server and 8080 is the port used by the Edge API. If you created a DNS entry for the Edge API, then the URL is in the form:
    http://ms_DNS/v1

    If you enable TLS on the Edge management API, then it is in the form:
    https://ms_IP:8080/v1
    https://ms_DNS/v1
  • apigee_mgmt_api_email: The email address of an account with access to the Edge /stats APIs. Often this will be the Edge system administrator's email, or the email of the organization administrator for your production organizations.
  • apigee_mgmt_api_password: The Edge password for the account specified by apigee_mgmt_api_email.
  • apigee_analytics_client_id and apigee_analytics_secret: Your credentials for uploading data to Apigee. Please submit a ticket with Apigee Support to obtain apigee_analytics_client_id and apigee_analytics_secret.

Example commands

Shown below is an example command to retrieve traffic data for all organizations and environments in the Edge installation and upload that data to Apigee. Notice how you use apigee-service to run the apigee-analytics-collector command:

>  /opt/apigee/apigee-service/bin/apigee-service apigee-analytics-collector export traffic \
--apigee_mgmt_api_uri http://192.168.56.103:8080/v1 \
--apigee_mgmt_api_email $ae_username \
--apigee_mgmt_api_password $ae_password \
--apigee_analytics_client_id $apigee_analytics_client_id \
--apigee_analytics_secret $apigee_analytics_secret

Notice that the command contains all required information, such as your apigee_analytics_client_id and apigee_analytics_secret.

You should see results in the form:

[
  {
    "org": "myOrg",
    "env": "prod",
    "time_range_start": "08/27/2016 00:00",
    "time_range_end": "08/30/2016 00:00",
    "response": [
      {
        "store_org_env_metrics_hourly_v4": 1
      }
    ]
  },
  {
    "org": "VALIDATE",
    "env": "test",
    "time_range_start": "08/27/2016 00:00",
    "time_range_end": "08/30/2016 00:00",
    "response": [
      {
        "store_org_env_metrics_hourly_v4": 1
      }
    ]
  }
]

Use command-line options to the command to control its actions. Use the following options to specify the organizations and environments to include in the generated data:

  • -i, --include_orgs <comma-separated list of items>
  • -x, --exclude_orgs <comma-separated list of items>
  • -n, --include_envs <comma-separated list of items>
  • -e, --exclude_envs <comma-separated list of items>

For example, to specify only the production organizations and environments, use the -i (or --include_orgs) and -n (or --include_envs) options:

>  /opt/apigee/apigee-service/bin/apigee-service apigee-analytics-collector export traffic -i myOrg -n prod \
--apigee_mgmt_api_uri http://192.168.56.103:8080/v1 \
--apigee_mgmt_api_email $ae_username \
--apigee_mgmt_api_password $ae_password \
--apigee_analytics_client_id $apigee_analytics_client_id \
--apigee_analytics_secret $apigee_analytics_secret

In this example, you only collect data from the prod environment of myOrg.

To dump the data to a screen to examine it before sending it to Apigee, use the -S option:

>  /opt/apigee/apigee-service/bin/apigee-service apigee-analytics-collector export traffic -i myOrg -n prod -S \
--apigee_mgmt_api_uri http://192.168.56.103:8080/v1 \
--apigee_mgmt_api_email $ae_username \
--apigee_mgmt_api_password $ae_password \
--apigee_analytics_client_id $apigee_analytics_client_id \
--apigee_analytics_secret $apigee_analytics_secret

The -S option omits the data upload to Apigee. You can then rerun the command without the -S option to send the data to Apigee.

One reason to use the -S option is so that you can display different types of data locally. Apigee only requires that you upload API traffic data, but the -D option lets you display data about API products, developers, apps, or API proxies. The example below uses the -D and -S options to display developer data locally:

>  /opt/apigee/apigee-service/bin/apigee-service apigee-analytics-collector export traffic -i myOrg -n prod -S -D devs \
--apigee_mgmt_api_uri http://192.168.56.103:8080/v1 \
--apigee_mgmt_api_email $ae_username \
--apigee_mgmt_api_password $ae_password \
--apigee_analytics_client_id $apigee_analytics_client_id \
--apigee_analytics_secret $apigee_analytics_secret

Include the -v option to get verbose output, and the -R option to see the curl commands generated by apigee-analytics-collector:

 >  /opt/apigee/apigee-service/bin/apigee-service apigee-analytics-collector export traffic -i myOrg -n prod -S -R -v \
--apigee_mgmt_api_uri http://192.168.56.103:8080/v1 \
--apigee_mgmt_api_email $ae_username \
--apigee_mgmt_api_password $ae_password \
--apigee_analytics_client_id $apigee_analytics_client_id \
--apigee_analytics_secret $apigee_analytics_secret

The next section contains a complete list of command-line options.

Command parameters

The following table lists the complete set of options to the apigee-analytics-collector utility:

Command

Use

-h, --help

Output usage information

-D, --dimension <dimension>

The traffic dimension to collect. Valid dimensions: apiproducts, devs, apps, apiproxy (default)

-d, --days <days>

The past number of days, starting from the current date, of data to collect. The default is 3.

If you specify -d, do not also specify -s and -z to set a time range.

-m, --apigee_mgmt_api_uri <apigee_mgmt_api_uri>

URL to Edge management API.

-u, --apigee_mgmt_api_email <apigee_mgmt_api_email>

The email address of an account with access to the Edge /stats APIs. Often this will be the Edge system administrator's email, or the email of the organization administrator for your production organizations.

-p, --apigee_mgmt_api_password <apigee_mgmt_api_password>

Password associated to the Edge management API email account specified by -u.

-i, --include_orgs <items>

Comma-separated list of orgs to include in the output.

-x, --exclude_orgs <items>

Comma-separated list of orgs to exclude from the output.

-n, --include_envs <items>

Comma-separated list of environments to include in the output.

-e, --exclude_envs <items>

Comma-separated list of environments to include from the output.

-o, --output <path>

Path and filename to save output.

-s, --time_range_start <time_range_start>

Time range to start for querying traffic stats, in the form: "03/01/2016 00:00".

If you specify -d, do not also specify -s and -z to set a time range.

-z, --time_range_end <time_range_end>

Time range end for querying traffic stats, in the form: "04/01/2016 24:00".

If you specify -d, do not also specify -s and -z to set a time range.

-t, --time_unit <time_unit>

Time unit for traffic data. Default week. Default units by hour. Valid time units: second, minute, hour, day, week.

-S, --standard_output

Write output to the terminal (stdout), instead of uploading it to Apigee.

-c, --apigee_analytics_client_id <apigee_analytics_client_id>

Your ID for uploading data to Apigee. Please submit a ticket with Apigee Support to obtain.

-r, --apigee_analytics_secret <apigee_analytics_secret>

Your secret for upload data to Apigee. Please submit a ticket with Apigee Support to obtain.

-R, --include_curl_commands

Include the generated cURL commands in the output for debugging.

-v, --verbose

Display verbose output.

Manually uploading data to Apigee

Apigee recommends that you install the apigee-analytics-collector utility on a machine with external internet access. Then the apigee-analytics-collector utility can upload data directly to Apigee.

However, if the machine does not have external internet access, use the Edge management API to collect traffic data, and then use cURL commands to upload it to Apigee from a machine with internet access. You have to repeat this process for every production organization and environment in your Edge installation.

Use the following cURL command to gather traffic data for a specific organization and environment for a specified time interval:

curl -X GET -u apigee_mgmt_api_email:apigee_mgmt_api_password \
"http://<ms_IP>:8080/v1/organizations/{org_name}/environments/{env_name}/stats/?select=sum(message_count)&timeRange=MM/DD/YYYY%20HH:MM~MM/DD/YYYY%20HH:MM"

This command uses the Edge Get API message count API. In this command:

  • apigee_mgmt_api_email:apigee_mgmt_api_password specifies the email address of an account with access to the Edge /stats APIs.
  • <ms_IP> is the IP address or DNS name of the Edge Management Server.
  • {org_name} and {env_name} specifies the org and environment.
  • MM/DD/YYYY%20HH:MM~MM/DD/YYYY%20HH:MM specifies the time range of the stats to gather. Notice that the cURL command uses the hex code %20 for the spaces in the time range.

For example, to gather stats for a 24 hour period, use the following command:

curl -X GET -u apigee_mgmt_api_email:apigee_mgmt_api_password \
"http://192.168.56.103:8080/v1/organizations/myOrg/environments/prod/stats/?select=sum(message_count)&timeRange=08/29/2016%2000:00~08/30/2016%2000:00"

You should see a response in the form:

{
  "environments" : [ {
    "metrics" : [ {
      "name" : "sum(message_count)",
      "values" : [ "42.0" ]
    } ],
    "name" : "prod"
  } ],
  "metaData" : {
    "errors" : [ ],
    "notices" : [ "query served by:53dab80c-e811-4ba6-a3e7-b96f53433baa", "source pg:6b7bab33-e732-405c-a5dd-4782647ce096", "Table used: myorg.prod.agg_api" ]
  }
}

To then upload that data to Apigee from a machine with internet access, use the following cURL command:

curl -X POST -H 'Content-Type:application/json' \
-u apigee_analytics_client_id:apigee_analytics_secret \
https://nucleus-api-prod.apigee.com/v1/apigee-analytics-cli-api/traffic/orgs/{org_name}/apis -d '{"environments"...}'

where:

  • apigee_analytics_client_id:apigee_analytics_secret specify your credentials for uploading data to Apigee that you obtained from Apigee Support.
  • {org_name} specifies the org.
  • {"environments"...} contains the results of the cURL command that you used to gather stats above.