The following sections describe the known issues with Edge for Private Cloud.
Area
Known issues
Edge for Private Cloud 4.53.00
440148595: End of Life Popup Warning Displayed Excessively
In Edge for Private Cloud 4.53.00 and later, the UI displays an
"End of Life" (EOL) warning pop-up. This warning
appears
repeatedly and cannot be prevented or reduced in frequency.
There is currently no method available for users to disable or reduce the frequency
of this EOL warning.
Edge for Private Cloud 4.53.01
443272053: Datastore errors in edge components
In Edge for Private Cloud 4.53.00 or later, a specific type of interaction between Cassandra and application components (Management server, Message Processor or Router) may cause datastore errors. When such an error occurs, you'll observe logs of the following pattern in the specific application component's system logs:
com.datastax.driver.core.exceptions.ProtocolError: An unexpected protocol error occurred on host WW.XX.YY.ZZ:9042.
These errors occur when the application component is not configured to handle warnings generated by the Cassandra database. You can mitigate this issue by avoiding or suppressing warnings in your Cassandra nodes. In most cases, warnings are generated due to excessive tombstones. You can follow one of the following options or a combination of options listed:
Reduce gc_grace_seconds: For the table displayed in the log message associated with the error, reduce gc_grace_seconds by running the following command like the following, using cqlsh:
Below command sets gc_grace_seconds of kms.oauth_20_access_tokens to 1 day from default 10 days
ALTER TABLE kms.oauth_20_access_tokens WITH gc_grace_seconds = '86400';
Increase tombstone thresholds in Cassandra for generating warnings. For this, use the following instructions:
On a Cassandra node, create or edit file $APIGEE_ROOT/customer/application/cassandra.properties.
Increase Tombstone warn threshold to 100k from the default 10k or set larger values as appropriate by adding the following line:
conf_cassandra_tombstone_warn_threshold=100000
Ensure the file above is owned and readable by apigee user:
Repeat the above steps on each Cassandra node, one by one.
42733857: Latency in updating encrypted key value maps (KVMs)
When working with Encrypted Key Value Maps that contain a large number of entries, users may experience latencies when adding or updating entries, whether through management APIs or the PUT element within the KeyValueMapOperations policy . The extent of the performance impact is generally proportional to the total number of entries stored in the encrypted KVM.
To mitigate this issue, it is recommended that users avoid creating encrypted KVMs with an excessive number of entries. A viable solution is to divide a large KVM into multiple, smaller KVMs. Additionally, if the use case permits, migrating to a non-encrypted KVM can also serve as an effective mitigation strategy. Please note that Apigee is aware of this issue and plans to release a fix in a future patch.
Java Callouts
Customer Java callouts that attempt to load the Bouncy Castle cryptography provider using the name "BC" might fail because the default provider has been changed to Bouncy Castle FIPS to support FIPS. The new provider name to use is "BCFIPS".
Edge for Private Cloud 4.53.00
Java Callouts
Customer Java callouts that attempt to load the Bouncy Castle cryptography provider using the name "BC" might fail because the default provider has been changed to Bouncy Castle FIPS to support FIPS. The new provider name to use is "BCFIPS".
Edge for Private Cloud 4.52.01 Mint update
This issue affects only those who are using MINT or have MINT enabled in Edge for Private Cloud installations.
Component affected: edge-message-processor
Issue: If you have monetization enabled and are installing 4.52.01 as a fresh install or upgrading from previous Private Cloud versions, you will encounter an issue with message processors. There will be a gradual increase in open thread count leading to resource exhaustion. The following exception is seen in edge-message-processor system.log:
A Denial-of-Service (DoS) vulnerability was recently discovered in multiple
implementations of the HTTP/2 protocol (CVE-2023-44487), including in Apigee Edge for
Private Cloud. The vulnerability could lead to a DoS of Apigee API management functionality.
For more details, see Apigee Security Bulletin GCP-2023-032.
The Edge for Private Cloud router and management server components are exposed to the
internet and can potentially be vulnerable. Although HTTP/2 is enabled on the management
port of other Edge-specific components of Edge for Private Cloud, none of those components are
exposed to the internet. On non-Edge components, like Cassandra, Zookeeper and others,
HTTP/2 is not enabled. We recommend that you take the
following steps to address the Edge for Private Cloud vulnerability:
Apigee-postgresql is having issues with upgrading from Edge for Private Cloud
version 4.50 or 4.51 to version 4.52. The issues mainly
occur when the number of tables is greater than 500.
You can check the total number of tables in Postgres by running the SQL query below:
149245401: LDAP connection pool settings for JNDI configured through the
LDAP resource
are not reflected, and JNDI defaults cause single-use connections each time.
As a result, connections are being opened
and closed each time for single use, creating a large number of
connections per hour to the LDAP server.
Workaround:
In order to change the LDAP connection pool properties, do
the following steps to set a global change across all LDAP policies.
Create a configuration properties file if it does not already exist:
Add the following to the file (replace values of
Java Naming and Directory Interface (JNDI) properties
based on your LDAP resource configuration requirement).
Make sure the file
/opt/apigee/customer/application/message-processor.properties is
owned by apigee:apigee.
Restart each message processor.
To verify that your connection pool JNDI
properties are taking effect, you can
perform a tcpdump to observe the behavior of the LDAP connection pool
over time.
High Request Processing Latency
139051927: High proxy processing latencies found in the Message Processor
are affecting
all API Proxies. Symptoms include 200-300ms delays in processing times over normal
API response
times and can occur randomly even with low TPS. This can occur when than more than 50 target
servers in which a message processor makes connections.
Root cause:
Message processors keep a cache that maps target server URL to HTTPClient object for
outgoing connections to target servers. By default this setting is set to 50 which may be
too low for most deployments. When a deployment has multiple org/env combinations in a setup,
and have a large number of target servers that exceed 50 altogether, the target server URLs
keep getting evicted from cache, causing latencies.
Validation:
To determine if target server URL eviction is causing the latency problem, search the
Message Processor system.logs
for keyword "onEvict" or "Eviction". Their presence in the logs indicate that target server URLs
are getting evicted from the HTTPClient cache because the cache size is too small.
Workaround:
For Edge for Private Cloud versions 19.01 and 19.06, you can edit and configure the HTTPClient
cache, /opt/apigee/customer/application/message-processor.properties:
Then restart the message processor. Make the same changes for all message processors.
The value 500 is an example. The optimal value for your setup should be greater than
the number of target servers that the message processor would connect to. There are no side
effects from
setting this property higher, and the only affect would be an improved message processor
proxy request processing times.
Note: Edge for Private Cloud version 50.00 has the default setting of 500.
Multiple entries for key value maps
157933959: Concurrent inserts and updates to the same key value map (KVM) scoped to the
organization or environment level causes inconsistent data and lost updates.
Note: This limitation only applies to Edge for Private Cloud. Edge for Public Cloud
and Hybrid do not have this limitation.
For a workaround in Edge for Private Cloud, create the KVM at the
apiproxy scope.
