Antipattern: Impact of non-existent KVMs on latency

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Key-Value Map (KVM) policies are a fundamental component in managing configurations and dynamic values within the system. However, the presence and lookup of non-existent keys within these KVMs can significantly impact performance and lead to increased latencies.

Antipattern

When a KVM policy attempts to retrieve a value for a key that does not exist in the Key-Value Map, it results in additional calls to the Cassandra database. This increased communication with the backend database, particularly for non-existent keys, consumes more resources and time, thereby escalating overall request latency.

Impact

The primary effect of this antipattern is elevated runtime latencies and datastore errors. The system's processing speed and data access are negatively affected, as the high percentage of non-existent key lookups is the main reason for the observed latencies. This is a runtime characteristic that is directly dependent on the nature of incoming requests.

Symptoms

  • System exhibits elevated runtime latencies.
  • Datastore errors, indicating issues with processing speed and data access.

Error Messages

Analysis of system.log file indicates the presence of recurring Cassandra-related errors, specifically concerning data retrieval and host pool status. Key error messages observed include:
  • ERROR DATASTORE.CASSANDRA - AstyanaxCassandraClient.get() : Exception while fetching rowKey : [7ccd1062-6e05-4d12-aa86-5c7bf944682d, app_credentials, kms] from column family:{} in keyspace:{}
  • ERROR DATASTORE.CASSANDRA - AstyanaxCassandraClient.logHostPoolInCaseOfErrors() : Cassandra Host Pool under use - All Hosts: xx.xx.xx.102(xx.xx.xx.102):9160,xx.xx.xx.101(xx.xx.xx.101):9160,xx.xx.xx.103(xx.xx.xx.103):9160. Active Hosts: xx.xx.xx.101(xx.xx.xx.101):9160,xx.xx.xx.102(xx.xx.xx.102):9160

Diagnosis

Analysis of the message processor system.log file reveals elevated L1MissL2Miss values within the 5-minute KVM metric reports.

For example: CACHE.org__env__KVM-Example.statistics [hitCount=215 L1MissL2Miss=14120 hitRate=5 L1MissL2Hit=212 getCount=556]

Best practice

To mitigate this issue, we recommend adding dummy values for non-existent keys in the KVMs. This approach should reduce the number of calls to Cassandra for non-existent keys, improving performance and reducing latency.

Further reading