Known issues with Apigee

When the cachehit flow variable is used after the LookupCache policy, due to the way debug points are dispatched for asynchronous behavior, the LookupPolicy populates the DebugInfo object before the call back has executed, resulting in an error.

Workaround: Repeat the process (make second call) again right after the first call.

Setting PurgeChildEntries in the InvalidateCache policy should purge the KeyFragment element values only but clears the entire cache.

Workaround: Use the KeyValueMapOperations policy to iterate cache versioning and bypass the need for cache invalidation.

This can happen, for example, when concurrent runs of a CI/CD deployment pipeline occur using different revisions. To avoid this problem, avoid deploying API proxies or SharedFlows before the current deployment is complete.

Workaround: Avoid concurrent API proxy or SharedFlow deployments.

Edge API Analytics can sometimes contain duplicate data for API calls. In that case the counts shown for API calls in Edge API Analytics are higher than the comparable values shown in third-party analytics tools.

Workaround: Export the analytics data and use the gateway_flow_id field to de-duplicate the data.

• Apigee Edge supports OpenAPI Specification 3.0 when you create specifications using the spec editor and publish APIs using SmartDocs on your portal, though a subset of features are not yet supported.

  For example, the following features from the OpenAPI Specification 3.0 are not yet supported:

  • allOf properties for combining and extending schemas
  • Remote references

  If an unsupported feature is referenced in your OpenAPI Specification, in some cases the tools will ignore the feature but still render the API reference documentation. In other cases, an unsupported feature will cause errors that prevent the successful rendering of the API reference documentation. In either case, you will need to modify your OpenAPI Specification to avoid use of the unsupported feature until it is supported in a future release.

  Note: Because the spec editor is less restrictive than SmartDocs when rendering API reference documentation, you may experience different results between the tools.

• When using Try this API in the portal, the Accept header is set to application/json regardless of the value set for consumes in the OpenAPI Specification.
• 138438484: Multiple servers are not supported.

• Simultaneous portal updates (such as page, theme, CSS, or script edits) by multiple users is not supported at this time.
• If you delete an API reference documentation page from the portal, there is no way to recreate it; you'll need to delete and re-add the API product, and regenerate the API reference documentation.
• When configuring the content security policy, it may take up to 15 minutes for changes to fully apply.
• When customizing your portal theme, it may take up to 5 minutes for changes to fully apply.

• Search will be integrated into the integrated portal in a future release.

When you update Edge for Private Cloud from version 4.51.00, 4.52.00, or 4.52.01 to 4.52.02, expect additional impact on runtime and management APIs.

This impact occurs after Cassandra nodes are updated and lasts until all message processor and management server nodes are updated.

When this occurs, expect an impact in one of the following areas:

• Runtime APIs refreshing OAuth token
• Management APIs listing developer apps
• Management APIs listing products

Apigee will publish revised update documentation for Edge for Private Cloud 4.52.02 that addresses this issue.

This issue affects only those who are using MINT or have MINT enabled in Edge for Private Cloud installations.

Component affected: edge-message-processor

Issue: If you have monetization enabled and are installing 4.52.01 as a fresh install or upgrading from previous Private Cloud versions, you will encounter an issue with message processors. There will be a gradual increase in open thread count leading to resource exhaustion. The following exception is seen in edge-message-processor system.log:

Error injecting constructor, java.lang.OutOfMemoryError: unable to create new native thread

A Denial-of-Service (DoS) vulnerability was recently discovered in multiple implementations of the HTTP/2 protocol (CVE-2023-44487), including in Apigee Edge for Private Cloud. The vulnerability could lead to a DoS of Apigee API management functionality. For more details, see Apigee Security Bulletin GCP-2023-032.

The Edge for Private Cloud router and management server components are exposed to the internet and can potentially be vulnerable. Although HTTP/2 is enabled on the management port of other Edge-specific components of Edge for Private Cloud, none of those components are exposed to the internet. On non-Edge components, like Cassandra, Zookeeper and others, HTTP/2 is not enabled. We recommend that you take the following steps to address the Edge for Private Cloud vulnerability:

• Steps for Edge for Private Cloud versions 4.51.00.11 or newer
• Steps for Edge for Private Cloud versions older than 4.51.00.11

Follow these steps if you are using Edge Private Cloud versions 4.51.00.11 or newer:

1. Update the management server:

   1. On each management server node, open /opt/apigee/customer/application/management-server.properties
   2. Add this line to the properties file:

      conf_webserver_http2.enabled=false

   3. Restart the management server component:

      apigee-service edge-management-server restart

2. Update the message processor:

   1. On each message processor node, open /opt/apigee/customer/application/message-processor.properties
   2. Add this line to the properties file:

      conf_webserver_http2.enabled=false

   3. Restart the message processor component:

      apigee-service edge-message-processor restart

3. Update the router:

   1. On each router node, open /opt/apigee/customer/application/router.properties
   2. Add this line to the properties file:

      conf_webserver_http2.enabled=false

   3. Restart the message processor component:

      apigee-service edge-router restart

4. Update QPID:

   1. On each QPID node, open /opt/apigee/customer/application/qpid-server.properties
   2. Add this line to the properties file:

      conf_webserver_http2.enabled=false

   3. Restart the message processor component:

      apigee-service edge-qpid-server restart

5. Update Postgres:

   1. On each Postgres node, open /opt/apigee/customer/application/postgres-server.properties
   2. Add this line to the properties file:

      conf_webserver_http2.enabled=false

   3. Restart the message processor component:

      apigee-service edge-postgres-server restart

Follow these steps if you are using Edge for Private Cloud versions older than 4.51.00.11:

1. Update the management server:

   1. On each management server node, open /opt/apigee/customer/application/management-server.properties
   2. Add the following two lines to the properties file:

      conf_webserver_http2.enabled=false
      conf/webserver.properties+http2.enabled=false

   3. Restart the management server component:

      apigee-service edge-management-server restart

2. Update the message processor:

   1. On each message processor node, open /opt/apigee/customer/application/message-processor.properties
   2. Add the following two lines to the properties file:

      conf_webserver_http2.enabled=false
      conf/webserver.properties+http2.enabled=false

   3. Restart the message processor component:

      apigee-service edge-message-processor restart

3. Update the router:

   1. On each router node, open /opt/apigee/customer/application/router.properties
   2. Add the following two lines to the properties file:

      conf_webserver_http2.enabled=false
      conf/webserver.properties+http2.enabled=false

   3. Restart the message processor component:

      apigee-service edge-router restart

4. Update QPID:

   1. On each QPID node, open /opt/apigee/customer/application/qpid-server.properties
   2. Add the following two lines to the properties file:

      conf_webserver_http2.enabled=false
      conf/webserver.properties+http2.enabled=false

   3. Restart the message processor component:

      apigee-service edge-qpid-server restart

5. Update Postgres:

   1. On each Postgres node, open /opt/apigee/customer/application/postgres-server.properties
   2. Add the following two lines to the properties file:

      conf_webserver_http2.enabled=false
      conf/webserver.properties+http2.enabled=false

   3. Restart the message processor component:

      apigee-service edge-postgres-server restart

Apigee-postgresql is having issues with upgrading from Edge for Private Cloud version 4.50 or 4.51 to version 4.52. The issues mainly occur when the number of tables is greater than 500.

You can check the total number of tables in Postgres by running the SQL query below:

select count(*) from information_schema.tables

Workaround: When Updating Apigee Edge 4.50.00 or 4.51.00 to 4.52.00, be sure to perform the preliminary step before upgrading Apigee-postgresql.

149245401: LDAP connection pool settings for JNDI configured through the LDAP resource are not reflected, and JNDI defaults cause single-use connections each time. As a result, connections are being opened and closed each time for single use, creating a large number of connections per hour to the LDAP server.

Workaround:

In order to change the LDAP connection pool properties, do the following steps to set a global change across all LDAP policies.

1. Create a configuration properties file if it does not already exist:

   /opt/apigee/customer/application/message-processor.properties

2. Add the following to the file (replace values of Java Naming and Directory Interface (JNDI) properties based on your LDAP resource configuration requirement).

   bin_setenv_ext_jvm_opts="-Dcom.sun.jndi.ldap.connect.pool.maxsize=20
   -Dcom.sun.jndi.ldap.connect.pool.prefsize=2
   -Dcom.sun.jndi.ldap.connect.pool.initsize=2
   -Dcom.sun.jndi.ldap.connect.pool.timeout=120000
   -Dcom.sun.jndi.ldap.connect.pool.protocol=ssl"

3. Make sure the file /opt/apigee/customer/application/message-processor.properties is owned by apigee:apigee.
4. Restart each message processor.

To verify that your connection pool JNDI properties are taking effect, you can perform a tcpdump to observe the behavior of the LDAP connection pool over time.

139051927: High proxy processing latencies found in the Message Processor are affecting all API Proxies. Symptoms include 200-300ms delays in processing times over normal API response times and can occur randomly even with low TPS. This can occur when than more than 50 target servers in which a message processor makes connections.

Root cause: Message processors keep a cache that maps target server URL to HTTPClient object for outgoing connections to target servers. By default this setting is set to 50 which may be too low for most deployments. When a deployment has multiple org/env combinations in a setup, and have a large number of target servers that exceed 50 altogether, the target server URLs keep getting evicted from cache, causing latencies.

Validation: To determine if target server URL eviction is causing the latency problem, search the Message Processor system.logs for keyword "onEvict" or "Eviction". Their presence in the logs indicate that target server URLs are getting evicted from the HTTPClient cache because the cache size is too small.

Workaround: For Edge for Private Cloud versions 19.01 and 19.06, you can edit and configure the HTTPClient cache, /opt/apigee/customer/application/message-processor.properties:

conf/http.properties+HTTPClient.dynamic.cache.elements.size=500

Then restart the message processor. Make the same changes for all message processors.

The value 500 is an example. The optimal value for your setup should be greater than the number of target servers that the message processor would connect to. There are no side effects from setting this property higher, and the only affect would be an improved message processor proxy request processing times.

Note: Edge for Private Cloud version 50.00 has the default setting of 500.

157933959: Concurrent inserts and updates to the same key value map (KVM) scoped to the organization or environment level causes inconsistent data and lost updates.

Note: This limitation only applies to Edge for Private Cloud. Edge for Public Cloud and Hybrid do not have this limitation.

For a workaround in Edge for Private Cloud, create the KVM at the apiproxy scope.