Uploading API Traffic Data to Apigee - Beta Release

All Edge for Private Cloud customers are required to submit to Apigee statistics about API proxy traffic. Apigee recommends that customers upload that information once a day, possibly by creating a cron job.

You must submit the data for your production API deployments, but not for APIs in development or testing deployments. In most Edge installations, you will define specific organizations or environments for your production APIs. The data that you submit is only for those production organizations and environments.

To assist in uploading this data, Apigee provides the Beta release of the apigee-analytics-collector command-line utility. This utility sends the API call volume report back to Apigee. Every Edge for the Private Cloud installation can use this utility to retrieve and report traffic data to Apigee.

Required: Contact Apigee Support before uploading data

Before you can upload data to Apigee, you must contact Apigee Support to complete the onboarding process.

Install apigee-analytics-collector

The apigee-analytics-collector utility is an RPM that you install by using the apigee-service utility.

Where to install

The node on which you install the apigee-analytics-collector utility can be any node that can access the Edge management API on the Edge Management Server. You can install it directly on the Management Server, on another node of the Edge installation, or on a separate node as long as that node can make API requests to the Management Server.

Internet access requirements

Install the apigee-analytics-collector utility on a machine with external internet access. Then the apigee-analytics-collector utility can upload data directly to Apigee.

If there is no node with both access to the Edge management API on the Edge Management Server and access to the external internet, then you can use the Edge management API to save the traffic data locally. You must then transfer the data to a machine with internet access for upload to Apigee. In this scenario, you do not need to use the apigee-analytics-collector utility. See Manually uploading data to Apigee for more.

Installation

Use the following command to install the apigee-analytics-collector utility. Because you are installing an RPM file, this command must be run by the root user or by a user that has full sudo access. For full sudo access, that means the user has sudo access to perform the same operations as root.

/opt/apigee/apigee-service/bin/apigee-service apigee-analytics-collector install

Running apigee-analytics-collector

This section describes how to run the apigee-analytics-collector utility.

Configuring a user to run apigee-analytics-collector

You must run apigee-analytics-collector as a non-root user. This user must have full sudo access to the "apigee" user.

To configure a user to have full sudo access to the "apigee" user, use the "visudo" command to edit the sudoers file to add:

analyticsUser        ALL=(apigee)      NOPASSWD: ALL

Where analyticsUser is the username of the person running the apigee-analytics-collector utility.

After installing the apigee-analytics-collector utility and configuring the user, you can test the utility by running the help command for the apigee-analytics-collector utility:

/opt/apigee/apigee-service/bin/apigee-service apigee-analytics-collector export traffic --help

Required information to run apigee-analytics-collector

You need the following information to run the apigee-analytics-collector command and to forward data to Apigee:

  • apigee_mgmt_api_uri: The base URL of the Edge API on your Management Server. This URL is typically in the following form:

    http://ms_IP:8080/v1

    Where ms_IP is the IP address or your Management Server and 8080 is the port used by the Edge API. If you created a DNS entry for the Edge API, then the URL is in the form:

    http://ms_DNS/v1

    If you enable TLS on the Edge management API, then it is in the form:

    https://ms_IP:8080/v1
    https://ms_DNS/v1
  • apigee_mgmt_api_email: The email address of an account with access to the Edge /stats APIs. Often this will be the Edge system administrator's email, or the email of the organization administrator for your production organizations.
  • apigee_mgmt_api_password: The Edge password for the account specified by apigee_mgmt_api_email.
  • apigee_analytics_client_id and apigee_analytics_secret: Your credentials for uploading data to Apigee. Please submit a ticket with Apigee Support to obtain apigee_analytics_client_id and apigee_analytics_secret.

Example commands

Shown below is an example command to retrieve traffic data for all organizations and environments in the Edge installation and upload that data to Apigee. Notice how you use apigee-service to run the apigee-analytics-collector command:

/opt/apigee/apigee-service/bin/apigee-service apigee-analytics-collector export traffic \
--apigee_mgmt_api_uri http://192.168.56.103:8080/v1 \
--apigee_mgmt_api_email $ae_username \
--apigee_mgmt_api_password $ae_password \
--apigee_analytics_client_id $apigee_analytics_client_id \
--apigee_analytics_secret $apigee_analytics_secret

Notice that the command contains all required information, such as your apigee_analytics_client_id and apigee_analytics_secret.

You should see results in the form:

[
  {
    "org": "myOrg",
    "env": "prod",
    "time_range_start": "08/27/2016 00:00",
    "time_range_end": "08/30/2016 00:00",
    "response": [
      {
        "store_org_env_metrics_hourly_v4": 1
      }
    ]
  },
  {
    "org": "VALIDATE",
    "env": "test",
    "time_range_start": "08/27/2016 00:00",
    "time_range_end": "08/30/2016 00:00",
    "response": [
      {
        "store_org_env_metrics_hourly_v4": 1
      }
    ]
  }
]

Use command-line options to the command to control its actions. Use the following options to specify the organizations and environments to include in the generated data:

  • -i, --include_orgs comma-separated list of items
  • -x, --exclude_orgs comma-separated list of items
  • -n, --include_envs comma-separated list of items
  • -e, --exclude_envs comma-separated list of items

For example, to specify only the production organizations and environments, use the -i (or --include_orgs) and -n (or --include_envs) options:

/opt/apigee/apigee-service/bin/apigee-service apigee-analytics-collector export traffic -i myOrg -n prod \
--apigee_mgmt_api_uri http://192.168.56.103:8080/v1 \
--apigee_mgmt_api_email $ae_username \
--apigee_mgmt_api_password $ae_password \
--apigee_analytics_client_id $apigee_analytics_client_id \
--apigee_analytics_secret $apigee_analytics_secret

In this example, you only collect data from the prod environment of myOrg.

To dump the data to a screen to examine it before sending it to Apigee, use the -S option:

/opt/apigee/apigee-service/bin/apigee-service apigee-analytics-collector export traffic -i myOrg -n prod -S \
  --apigee_mgmt_api_uri http://192.168.56.103:8080/v1 \
  --apigee_mgmt_api_email $ae_username \
  --apigee_mgmt_api_password $ae_password \
  --apigee_analytics_client_id $apigee_analytics_client_id \
  --apigee_analytics_secret $apigee_analytics_secret

The -S option omits the data upload to Apigee. You can then rerun the command without the -S option to send the data to Apigee.

One reason to use the -S option is so that you can display different types of data locally. Apigee only requires that you upload API traffic data, but the -D option lets you display data about API products, developers, apps, or API proxies. The example below uses the -D and -S options to display developer data locally:

/opt/apigee/apigee-service/bin/apigee-service apigee-analytics-collector export traffic -i myOrg -n prod -S -D devs \
  --apigee_mgmt_api_uri http://192.168.56.103:8080/v1 \
  --apigee_mgmt_api_email $ae_username \
  --apigee_mgmt_api_password $ae_password \
  --apigee_analytics_client_id $apigee_analytics_client_id \
  --apigee_analytics_secret $apigee_analytics_secret

Include the -v option to get verbose output, and the -R option to see the curl commands generated by apigee-analytics-collector:

/opt/apigee/apigee-service/bin/apigee-service apigee-analytics-collector export traffic -i myOrg -n prod -S -R -v \
  --apigee_mgmt_api_uri http://192.168.56.103:8080/v1 \
  --apigee_mgmt_api_email $ae_username \
  --apigee_mgmt_api_password $ae_password \
  --apigee_analytics_client_id $apigee_analytics_client_id \
  --apigee_analytics_secret $apigee_analytics_secret

The next section contains a complete list of command-line options.

Command parameters

The following table lists the complete set of options to the apigee-analytics-collector utility:

Command Description
-h, --help

Output usage information

-D, --dimension dimension

The traffic dimension to collect. Valid values are:

  • apiproducts
  • apiproxy (default)
  • apps
  • devs
-d, --days days

The number of past days of data to collect, starting from the current date. The default is 3.

If you specify -d, do not also specify -s and -z to set a time range.

-m, --apigee_mgmt_api_uri apigee_mgmt_api_uri

URL to Edge management API.

-u, --apigee_mgmt_api_email apigee_mgmt_api_email

The email address of an account with access to the Edge /stats APIs. This is typically the Edge system administrator's email, or the email of the organization administrator for your production organizations.

-p, --apigee_mgmt_api_password apigee_mgmt_api_password

The password associated with the Edge management API email account specified by -u.

-i, --include_orgs items

A comma-separated list of organizations to include in the output.

-x, --exclude_orgs items

A comma-separated list of orgs to exclude from the output.

-n, --include_envs items

A comma-separated list of environments to include in the output.

-e, --exclude_envs items

A comma-separated list of environments to include from the output.

-o, --output path

The path and filename to save output.

-s, --time_range_start time_range_start

Time range to start for querying traffic stats, in the form: "MM/DD/YYYY HH:MM".

If you specify -d, do not also specify -s and -z to set a time range.

-z, --time_range_end time_range_end

Time range end for querying traffic stats, in the form: "04/01/2016 24:00".

If you specify -d, do not also specify -s and -z to set a time range.

-t, --time_unit time_unit

Time unit for traffic data. The default value is hour. Valid time units are:

  • second
  • minute
  • hour (default)
  • day
  • week

If you set -t to week and -d to 365, then Apigee returns a large amount of data, grouped into 52 separate entries (one for each week).

-S, --standard_output

Write output to the terminal (stdout), instead of uploading it to Apigee.

-c, --apigee_analytics_client_id apigee_analytics_client_id

Your ID for uploading data to Apigee. Please submit a ticket with Apigee Support to obtain.

-r, --apigee_analytics_secret apigee_analytics_secret

Your secret for upload data to Apigee. Please submit a ticket with Apigee Support to obtain.

-R, --include_curl_commands

Include the generated curl commands in the output for debugging.

-v, --verbose

Display verbose output.

Manually uploading data to Apigee

Apigee recommends that you install the apigee-analytics-collector utility on a machine with external internet access. Then the apigee-analytics-collector utility can upload data directly to Apigee.

However, if the machine does not have external internet access, use the Edge management API to collect traffic data, and then use curl commands to upload it to Apigee from a machine with internet access. You must repeat this process for every production organization and environment in your Edge installation.

Use the following curl command to gather traffic data for a specific organization and environment for a specified time interval:

curl -X GET -u apigee_mgmt_api_email:apigee_mgmt_api_password \
"http://ms_IP:8080/v1/organizations/org_name/environments/env_name/stats/apiproxy?select=sum(message_count)&timeRange=MM/DD/YYYY%20HH:MM~MM/DD/YYYY%20HH:MM&timeUnit=hour"

This command uses the Edge Get API message count API. In this command:

  • apigee_mgmt_api_email:apigee_mgmt_api_password specifies the email address of an account with access to the Edge /stats APIs.
  • ms_IP is the IP address or DNS name of the Edge Management Server.
  • org_name and env_name specifies the org and environment.
  • apiproxy is the dimenesion that groups metrics by API proxies.
  • MM/DD/YYYY%20HH:MM~MM/DD/YYYY%20HH:MM&timeUnit=hour specifies the time range divided into time units of the metrics to gather. Notice that the curl command uses the hex code %20 for spaces in the time range.

For example, to gather API proxy message counts hour by hour over a 24 hour period, use the following API call.

curl -X GET -u apigee_mgmt_api_email:apigee_mgmt_api_password \
"http://192.168.56.103:8080/v1/organizations/myOrg/environments/prod/stats/apiproxy?select=sum(message_count)&timeRange=01%2F01%2F2018%2000%3A00~01%2F02%2F2018%2000%3A00&timeUnit=hour"

(Note that timeRange contains URL-encoded characters.)

You should see a response in the form:

{
  "environments" : [ {
    "dimensions" : [ {
      "metrics" : [ {
        "name" : "sum(message_count)",
        "values": [
                {
                  "timestamp": 1514847600000,
                  "value": "35.0"
                },
                {
                  "timestamp": 1514844000000,
                  "value": "19.0"
                },
                {
                  "timestamp": 1514840400000,
                  "value": "58.0"
                },
                {
                  "timestamp": 1514836800000,
                  "value": "28.0"
                },
                {
                  "timestamp": 1514833200000,
                  "value": "29.0"
                },
                {
                  "timestamp": 1514829600000,
                  "value": "33.0"
                },
                {
                  "timestamp": 1514826000000,
                  "value": "26.0"
                },
                {
                  "timestamp": 1514822400000,
                  "value": "57.0"
                },
                {
                  "timestamp": 1514818800000,
                  "value": "41.0"
                },
                {
                  "timestamp": 1514815200000,
                  "value": "27.0"
                },
                {
                  "timestamp": 1514811600000,
                  "value": "47.0"
                },
                {
                  "timestamp": 1514808000000,
                  "value": "66.0"
                },
                {
                  "timestamp": 1514804400000,
                  "value": "50.0"
                },
                {
                  "timestamp": 1514800800000,
                  "value": "41.0"
                },
                {
                  "timestamp": 1514797200000,
                  "value": "49.0"
                },
                {
                  "timestamp": 1514793600000,
                  "value": "35.0"
                },
                {
                  "timestamp": 1514790000000,
                  "value": "89.0"
                },
                {
                  "timestamp": 1514786400000,
                  "value": "42.0"
                },
                {
                  "timestamp": 1514782800000,
                  "value": "47.0"
                },
                {
                  "timestamp": 1514779200000,
                  "value": "21.0"
                },
                {
                  "timestamp": 1514775600000,
                  "value": "27.0"
                },
                {
                  "timestamp": 1514772000000,
                  "value": "20.0"
                },
                {
                  "timestamp": 1514768400000,
                  "value": "12.0"
                },
                {
                  "timestamp": 1514764800000,
                  "value": "7.0"
                }
              ]
            }
          ],
          "name" : "proxy1"
      } ],
    "name" : "prod"
  } ],
  "metaData" : {
    "errors" : [ ],
    "notices" : [ "query served by:53dab80c-e811-4ba6-a3e7-b96f53433baa", "source pg:6b7bab33-e732-405c-a5dd-4782647ce096", "Table used: myorg.prod.agg_api" ]
  }
}

To then upload that data to Apigee from a machine with internet access, use the following curl command:

curl -X POST -H 'Content-Type:application/json' \
  -u apigee_analytics_client_id:apigee_analytics_secret \
  https://nucleus-api-prod.apigee.com/v1/apigee-analytics-cli-api/traffic/orgs/org_name/apis -d '"environments"...'

Where:

  • apigee_analytics_client_id:apigee_analytics_secret specify your credentials for uploading data to Apigee that you obtained from Apigee.
  • org_name specifies the organization name.
  • "environments"... contains the results of the curl command that you used to gather stats above.