Metrics collection overview

Apigee hybrid collects operations metrics that you can use to monitor the health of hybrid services, to set up alerts, and so on. Apigee uses the industry-standard Prometheus add-on for metrics collection. Once collected, Hybrid sends the metrics data to Stackdriver, at which point you can use the Stackdriver console for viewing, searching, and analyzing metrics and managing alerts.

The following diagram shows the metrics collection process:

alt_text

As you can see in this diagram, there is one Prometheus server running per cluster, and it can run on any pod in the cluster. Prometheus scrapes application metrics data from all Hybrid services and sends the metrics data to Stackdriver. You can access metrics data through the Stackdriver console.

Metrics collection is enabled by default. To disable it, see Disable metrics collection.

About metrics

Application metrics data is made available on a port as an internal Kubernetes service. Data collected by this service is scraped by the hybrid metrics collector service. You can use the Stackdriver Metrics Explorer to select metrics you want to view, such as:

  • read request count
  • read request latency
  • write request count
  • write request latency

For example, you can create a dashboard in Stackdriver to show your metrics:

alt_text

Configure metrics collection

To send metrics to the Stackdriver application for your GCP account, you must set configuration properties in the overrides.yaml file. Do this configuration whether you are on a GKE or a non-GKE cluster.

To configure metrics collection:

  1. Create a GCP service account with the Monitoring Metrics Writer role. You can create the service account using the hybrid CLI command create-service-account:.
    ./tools/create-service-account my-metrics-svc-account apigee-metrics
    For more information about GCP service accounts, see Creating and managing service accounts.
  2. The create-service-account command saves a key on your system as a .json file. Note the path to the file. You will need the path in the following steps.
  3. Add the following configuration to your overrides.yaml file:
    overrides.yaml
    metrics:
      enableMetrics: true
      serviceAccountPath: sa_json_file_path
    
    project_id: GCP_project_ID
    k8s_cluster_name: cluster_name
    k8s_cluster_location: cluster_region (Must be a GCP region name. See the note below.)

    Where:

    • sa_json_file_path is the path on your filesystem to the service account JSON file the create-service-account command saved.
    • GCP_project_ID is your GCP project ID for the project where you want the metrics to be sent to.
    • cluster_name is the name of the cluster running hybrid.
    • cluster_region is a GCP region name. See the note below for more information:

    For example:

    metrics:
          enableMetrics: true
          serviceAccountPath: $HOME/hybrid/user-9c0dcb0c6c45.json
    
        project_id: myproject
        k8s_cluster_name: user-cluster
        k8s_cluster_location: us-west2
  4. GKE only: If you are on GKE, in addition to the mandatory overrides, you need to grant your own user the GKE cluster admin role:
    ACCOUNT=$(gcloud info --format='value(config.account)')
    kubectl create clusterrolebinding owner-cluster-admin-binding \
    --clusterrole=cluster-admin --user=$ACCOUNT
  5. Apply the configuration to your cluster.

View metrics on Stackdriver

You can use Stackdriver to view your metrics information. You do this by selecting a resource type and a metric that has data from that resource type.

To view metrics on Stackdriver:

  1. Open the Stackdriver Metrics Explorer in a browser. Alternatively, if you're already in the Stackdriver console, select Resources > Metrics Explorer.
  2. Select the metric you want to examine. For example, to see metrics for proxy 4xx errors:
    1. In Find resource type and metric, begin typing "proxy" in the search box.
    2. Select external.googleapis.com/prometheus/proxy_requests.
    3. Filter on label.
    4. Select GET from the menu.

    Stackdriver displays data for the selected metric.

  3. To save the selected metrics settings, click the Save Chart button.

For more information about metrics, see the following:

Disable metrics collection

Metrics collection is enabled by default for hybrid. To disable metrics collection, add the following attribute to overrides.yaml and apply the change to your cluster:

overrides.yaml
metrics:
  enableMetrics: false

Apply the change:

apigeectl apply -c metrics -v beta

Checking service account permissions

To check the permissions for a GCP service account:

  1. In the GCP console, be sure your project is selected.
  2. Go to the GKE Kubernetes clusters screen.
  3. Select your cluster.
  4. Select the Nodes tab. This tab shows you a list of all of the nodes in your cluster.
  5. Select one of the nodes from the list.
  6. Select the Details tab.
  7. Click the VM instance link.
  8. Locate the Service account and make a note of the ID (it looks like an email address).
  9. Go to IAM & Permissions in the GCP console.
  10. Select IAM.
  11. Locate the service account ID in the list.
  12. The service account roles are listed there.

For more information, see Granting roles to service accounts.