Logging

All of the Apigee Hybrid services that run in your Kubernetes cluster output operational log information. This log information is useful for troubleshooting and debugging. For example, if a service pod's status indicates a problem, you can look at the log files for that pod to gain insight into the issue. Apigee support may request you to provide log information to diagnose and solve a problem.

About logging

Each runtime plane service writes log data to stdout/stderr, and Kubernetes writes this data to the host filesystem. A logs collector for Hybrid runs on each Hybrid node. The collector is a DaemonSet (replicated once and only once per node) whose job it is to collect the written log data and export it to the Stackdriver Logging application associated with your GCP account.

The following diagram shows the log data collection process:

alt_text

Enable logging

This section describes how to enable logging for your Hybrid deployment.

Label nodes for logging

To enable logging, you must apply the label apigee.com/apigee-logger-enabled=true to each node that runs an Apigee component. If a label is not provided, logs will not be collected from any pod running in the node.

To apply the logging label to your nodes, follow these steps:

  1. Determine if you need to add the label. If a node already has the logging label, you do not need to take any further action:
    1. List the nodes in your cluster:
      kubectl get nodes -n my-namespace
    2. Check the node labeling by running this command:
      kubectl describe node NodeName
    3. If the node is labeled for logging, you will see something like this in the output:
      Name:           NodeName
      Role:
      Labels:         apigee.com/apigee-logger-enabled=true
      ...
    4. If you see the label apigee.com/apigee-logger-enabled=true, the node is enabled for logging.
    5. If you don't see the label, go to the next step.
  2. Use the following command to apply the logging label to the node:
    kubectl label node NodeName apigee.com/apigee-logger-enabled=true
    For example:
    kubectl label node gke-apigee-hybrid-beta-apigee-data-550ebb2e-1mzw \
        apigee.com/apigee-logger-enabled=true

Configure logs for a GKE deployment

Apigee Hybrid enables logging by default. However, note that GKE comes with a built-in log collector; therefore, if you are on GKE, you must disable logging:

  1. Create a GCP service account with the Logs Writer role. You can create the service account using the Hybrid CLI command create-service-account, as the following example shows:

    ./tools/create-service-account my-logger-svc-account apigee-logger

    For more information about GCP service accounts, see Creating and managing service accounts.

  2. Add the following attribute to overrides.yaml:
    logger:
      enable: false
  3. Apply the configuration change.

Configure logs for a non-GKE deployment

  1. Create a GCP service account with the Logs Writer role. You can create the service account using the Hybrid CLI command create-service-account, as the following example shows:

    ./tools/create-service-account my-logger-svc-account apigee-logger

    For more information about GCP service accounts, see Creating and managing service accounts.

  2. The create-service-account command saves a key on your system as a .json file. Note the path to the file. You will need the path in the following steps.
  3. Add the following configuration to overrides.yaml:
    overrides.yaml
    logger:
      enable: true
      serviceAccountPath: sa_json_file_path
    
    gcpProjectID: GCP_project-ID
    k8sClusterName: cluster_name
    gcpRegion: cluster_region

    Where:

    • sa_json_file_path is the path on your filesystem to the service account JSON file that you just downloaded from GCP.
    • GCP_project_ID is your GCP project ID for the project where you want the log entries to be sent to.
    • cluster_name is the name of the cluster running Hybrid.
    • cluster_region is a GCP region name. The name must be in the standard GCP region format. You must specify the closest GCP region to where your cluster is located.

      For example, if your cluster is located on premises in Seattle, Washington, USA, the closest GCP region is in The Dalles, Oregon, USA, with the name us-west2. In this case, you would set k8s_cluster_location to us-west2.

  4. Apply the configuration change.

Check service account permissions

To check the permissions for a GCP service account:

  1. In the GCP console, be sure your project is selected.
  2. Go to the GKE Kubernetes clusters screen.
  3. Select your cluster.
  4. Select the Nodes tab. This tab shows you a list of all of the nodes in your cluster.
  5. Select one of the nodes from the list.
  6. Select the Details tab.
  7. Click the VM instance link.
  8. Locate the Service account and make a note of the ID (it looks like an email address).
  9. Go to IAM & Permissions in the GCP console.
  10. Select IAM.
  11. Locate the service account ID in the list. The service account roles are listed there.

For more information, see Granting roles to service accounts.

View logs on Stackdriver

If you are on GKE, logs are automatically sent to Stackdriver provided your configuration is correct.

If you are not on GKE, you need to enable logging first. See Configure logs for a non-GKE deployment.

To view logs on Stackdriver:

  1. Go to Stackdriver Logs Viewer.
  2. In the resource-type drop-down list, select GKE Container, the name of your cluster, and your namespace ID. For example:

    alt_text

    For more information, see Viewing logs in the GCP documentation.

The following example shows Hybrid log output in the Log viewer:

alt_text

View log files directly

You can view the logs that are written to each pod's filesystem directly using kubectl, as follows:

kubectl logs pod_name -n namespace

For example:

kubectl logs apigee-mp-hybrid-docs-test-blue-6fb96f5b9-2k8hp -n my-namespace