View Cassandra logs

Cassandra is the runtime data store used to provide Apigee Core Persistence Services (CPS) for the Apigee hybrid runtime. The Cassandra database is deployed in Kubernetes in a StatefulSet node pool, as Cassandra is a distributed data system requiring state managed on a filesystem.

Entities stored in the datastore that the Message Processor needs to execute policies include the following:

  • Key management system (KMS) data, including companies, developers, developer apps, API products, and API key
  • Key value map (KVM) data
  • Response cache data
  • OAuth data, including access tokens, refresh tokens, and authorization codes.
  • Quota data, including buckets and counters

Cassandra log files

Logs are a good way to troubleshoot problems with your installation. See Logging for details.

Error troubleshooting

This section explains how to troubleshoot and correct issues with the hybrid Cassandra ring.

Log files

Logs are a good way to troubleshoot problems with your installation. See Logging for details.

Diagnose and correct a node failure

You can tell there's a node problem if you notice a Cassandra pod fails to start up and is stuck in the Pending status. This problem can indicate an underlying node failure. Follow these steps to diagnose and correct a node failure.

  1. Get the pods to see if any are not running:
    $ kubectl get pods -n your_namespace
        NAME          READY   STATUS    RESTARTS   AGE
        cassandra-0   0/1     Pending   0          13s
        cassandra-1   1/1     Running   0          8d
        cassandra-2   1/1     Running   0          8d
  2. Check the worker nodes. If one is in the NotReady state, then that is the node that has failed:
    kubectl get nodes -n your_namespace
    NAME                          STATUS   ROLES    AGE   VERSION
    ip-10-30-1-190.ec2.internal   Ready    <none>   8d    v1.13.2
    ip-10-30-1-22.ec2.internal    Ready    master   8d    v1.13.2
    ip-10-30-1-36.ec2.internal    NotReady <none>   8d    v1.13.2
    ip-10-30-2-214.ec2.internal   Ready    <none>   8d    v1.13.2
    ip-10-30-2-252.ec2.internal   Ready    <none>   8d    v1.13.2
    ip-10-30-2-47.ec2.internal    Ready    <none>   8d    v1.13.2
    ip-10-30-3-11.ec2.internal    Ready    <none>   8d    v1.13.2
    ip-10-30-3-152.ec2.internal   Ready    <none>   8d    v1.13.2
    ip-10-30-3-5.ec2.internal     Ready    <none>   8d    v1.13.2
  3. Remove the dead Cassandra pod from the cluster.
    $ kubectl exec -it apigee-cassandra-0 -- nodetool status
    $ kubectl exec -it apigee-cassandra-0 -- nodetool removenode deadnode_hostID
  4. Remove the VolumeClaim from the dead node to prevent the Cassandra pod from attempting to come up on the dead node because of the affinity:
    kubectl get pvc -n your_namespace
    kubectl delete pvc volumeClaim_name -n your_namespace
  5. Update the volume template and create PersistentVolume for the newly added node. The following is an example volume template:
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: cassandra-data-3
    spec:
      capacity:
        storage: 100Gi
      accessModes:
      - ReadWriteOnce
      persistentVolumeReclaimPolicy: Retain
      storageClassName: local-storage
      local:
        path: /apigee/data
      nodeAffinity:
        "required":
          "nodeSelectorTerms":
          - "matchExpressions":
            - "key": "kubernetes.io/hostname"
              "operator": "In"
              "values": ["ip-10-30-1-36.ec2.internal"]
  6. Replace the values with the new hostname/IP and apply the template:
    kubectl apply -f volume-template.yaml