Edge for Private Cloud v. 4.17.05
About the Cassandra replication factor
Cassandra stores data replicas on multiple nodes to ensure reliability and fault tolerance. The replication strategy for each Edge keyspace determines the nodes where replicas are placed.
The total number of replicas for a keyspace across a Cassandra cluster is referred to as the keyspace's replication factor. A replication factor of one means that there is only one copy of each row in the Cassandra cluster. A replication factor of two means there are two copies of each row, where each copy is on a different node. All replicas are equally important; there is no primary or master replica.
In a production system with three or more Cassandra nodes in each data center, the default replication factor for an Edge keyspace is three. As a general rule, the replication factor should not exceed the number of Cassandra nodes in the cluster.
Use the following procedure to view the Cassandra schema, which shows the replication factor for each Edge keyspace:
- Log in to a Cassandra node.
- Run the following command:
> /opt/apigee/apigee-cassandra/bin/cassandra-cli -h $(hostname -i) <<< "show schema;"
Where $(hostname -i) resolves to the IP address of the Cassandra node. Or you can replace $(hostname -i) with the IP address of the node.
For each keyspace, you will see output in the form:
create keyspace kms
with placement_strategy = 'NetworkTopologyStrategy'
and strategy_options = {dc-1 : 3}
and durable_writes = true;
You can see that for data center 1, dc-1, the default replication factor for the kms keyspace is three for an installation with three Cassandra nodes.
If you add additional Cassandra nodes to the cluster, the default replication factor is not affected.
For example, if you increase the number of Cassandra nodes to six, but leave the replication factor at three, you do not ensure that all Cassandra nodes have a copy of all the data. If a node goes down, a higher replication factor means a higher probability that the data on the node exists on one of the remaining nodes. The downside of a higher replication factor is an increased latency on data writes.
About the Cassandra consistency level
The Cassandra consistency level is defined as the minimum number of Cassandra nodes that must acknowledge a read or write operation before the operation can be considered successful. Different consistency levels can be assigned to different Edge keyspaces.
When connecting to Cassandra for read and write operations, Message Processor and Management Server nodes typically use the Cassandra value of LOCAL_QUORUM to specify the consistency level for a keyspace. However, some keyspaces are defined to use a consistency level of one.
The calculation of the value of LOCAL_QUORUM for a data center is:
LOCAL_QUORUM = (replication_factor/2) + 1
As described above, the default replication factor for an Edge production environment with three Cassandra nodes is three. Therefore, the default value of LOCAL_QUORUM = (3/2) +1 = 2 (the value is rounded down to an integer).
With LOCAL_QUORUM = 2, at least two of the three Cassandra nodes in the data center must respond to a read/write operation for the operation to succeed. For a three node Cassandra cluster, the cluster could therefore tolerate one node being down per data center.
By specifying the consistency level as LOCAL_QUORUM, Edge avoids the latency required by validating operations across multiple data centers. If a keyspace used the Cassandra QUORUM value as the consistency level, read/write operations would have to be validated across all data centers.
To see the consistency level used by the Edge Message Processor or Management Server nodes:
- Log in to a Message Processor node.
- Change to the /opt/apigee/edge-message-processor/conf directory:
> cd /opt/apigee/edge-message-processor/conf - For write consistency:
> grep -ri "write.consistencylevel" * - For read consistency:
> grep -ri "read.consistencylevel" * - Log in to the Management Server node.
- Change to the /opt/apigee/edge-management-server/conf directory:
> cd /opt/apigee/edge-management-server/conf - Repeat steps 3 and 4.
If you add additional Cassandra nodes to the cluster, the consistency level is not affected.