4.19.06 Edge for Private Cloud release notes

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation.

This section describes version 4.19.06 of the Edge for Private Cloud Feature Release.

Release summary

The following table summarizes the changes in this release:

New Features

This release includes the following new features:

○ LDAP support for SSO
○ HTTP deployment support
○ Pluggable analytics
○ Cassandra rack support
○ IPv6 support
○ Simplified New Edge experience installation
○ Exclude INVALID_TSC transactions (Monetization)

For more information about each of these new features, see New features.

Backward Compatibility

The following issues affecting backward compatibility were introduced in this version of Apigee Edge for Private Cloud:

  • Due to the fix for issue 132443137 (as described in the 19.03.01 Public Cloud release notes), Message Processors now ignore headers that begin with X-Apigee-*. As a result, you must refactor any code that uses X-Apigee-* headers and replace those headers with supported ones.

  • The name of the Cassandra log files has changed. It is now:

Included Releases

Since the previous Edge for Private Cloud Feature Release, the following releases have occurred and are included in this release:

○ Edge:
   19.07.23 (UI)
○ Portal:
Retirements None

With the release of 4.19.06, the following are deprecated:

  • 4.18.01: Version 4.18.01 of Apigee Edge for Private Cloud is now deprecated.

For more information, see Apigee deprecations, retirements, and CPS changes.

Bug Fixes

This release includes the following bug fixes:

○ UseStringDeduplication not set (140270474)
○ MP errors (111445518)
○ Cassandra failures with JMX (113342838)
○ 13-node upgrades (138931337)
○ Proxies with the same name (119802562)
○ Keystores not loading (118851790)
○ Analytics failed (110988191)
○ Backup failures (110785400)
HEAD requests to Node.js (79993247)
○ apigee-postgres backup/restore (70291825)
○ Resetting sysadmin password (65737520)

For more information about each of these fixes, see Bug fixes.

Known Issues

This release includes the following known issues:

○ Mint not rating transactions initially (141628497)
○ Mint rating servers memory leak (137865184)
○ Metadata file format for apigee-sso install (135616498)
apigee-monit not supported on Amazon Linux 1 (122370980)
○ Message Processor backup not backing up the correct set of files (121095148)
○ DataAccessExceptions in multi data center configurations (76087166)
○ MessageLogging policy including extra information in the log message (68722102)

For more information about each of these known issues, including workarounds, see Known issues.

Upgrade paths

The following table shows the upgrade paths for this release:

From 4.19.01 Directly upgrade from 4.19.01 → 4.19.06
From 4.18.05 Directly upgrade from 4.18.05 → 4.19.06
From 4.18.01 Directly upgrade from 4.18.01 → 4.19.06

New features

This section lists new features in this release. Additionally, this release includes all features in the Edge UI, Edge Management, and Portal releases listed in Included Releases.

In addition to the following enhancements, this release also contains multiple usability, performance, security, and stability enhancements.

HTTP deployment support

By default, Edge uses RPC to deploy API proxies. While this mode works very well for most installations, larger topologies with many MPs might experience timeouts when a large number of concurrent calls are made via RPC.

As a result, Apigee recommends that larger deployments use HTTP rather than RPC for deployment. Enabling HTTP deployments are now supported.

For more information, see Enable HTTP deployment.

Cassandra rack support

Apigee now provides general guidance for scaling operations of Cassandra by making Cassandra on Apigee Edge for Private Cloud rack aware.

For more information, see Cassandra rack support.

Pluggable analytics

By default, analytics data collected by the Message Processor is uploaded to Qpid and Postgres for processing. You can then view the analytics data in the Edge UI.

With this Feature Release, you can configure the Message Processor to write analytics data to disk or you can choose to use both options.

For more information, see Writing analytics data to a file.

Simplified New Edge experience installation

Installation of the New Edge experience has been simplified with a new script.

For more information, see Install the New Edge Experience.

LDAP IDP support

Authentication of users with Edge UI and Edge management API calls now supports LDAP in addition to SAML. LDAP support includes both direct binding and indirect binding methods.

For more information, see Overview of external IDP authentication.

IPv6 support

By default, all runtime API calls to Apigee Edge for Private Cloud use IPv4. You can add IPv6 support to the Router with a new property.

For more information, see Use IPv6 on the Router.

Exclude INVALID_TSC transactions (Monetization)

You can now exclude INVALID_TSC transactions from being added to the Postgres database when using Monetization.

For more information, see Monetization server properties.

Supported software

This release includes the following changes to supported software:

Support Added No Longer Supported

This release adds support for the following platforms:

  • Red Hat Enterprise Linux (RHEL) 6.10
  • CentOS 6.10
  • Oracle Linux 6.9

The following platforms are no longer supported with this release:

  • Red Hat Enterprise Linux (RHEL) 6.9
  • CentOS 6.9
  • Oracle 6.8

For a complete list of supported platforms, see Supported software and supported versions.

Bug fixes

This section lists the Private Cloud bugs that were fixed in this release. Additionally, this release includes all bug fixes in the Edge UI, Edge Management, and Portal releases shown in Included Releases.

Issue ID Description

UseStringDeduplication not set

UseStringDeduplication was not getting set for the edge-message-processor components in production environments.

MP errors

Message Processors continuously logged the following error message every 5 min:
2019-06-29 00:02:05,775 Apigee-Timer-6 ERROR CMgmt -
  ConnectorServiceImpl.refreshToken() : Connector Management API auth token
  generation failed: 0 null

Cassandra failures with JMX

Cassandra operations were not working when JMX auth was enabled

13-node upgrades

Upgrades of 13-node installations from 17.09, 18.01, and 18.05 were failing.

Proxies with the same name

If two users in the same organization created proxies with the same names, the second one created would overwrite the first. This only occurred when different users could not see the other proxy.

Keystores not loading

When there were more than 100 keystores for an organization/environment, the keystore and reference section in Edge UI was not loading. Instead, the UI displayed an Internal Error (Error Fetching Keystores) error.

Analytics failed

Analytics failed for an environment when supplied with data from the Microgateway that was set to a date in the future.

Backup failures

ZooKeeper, Cassandra, and OpenLDAP intermittently failed to properly back up their data with the following error: File changed as we read it.

HEAD requests to Node.js

HEAD requests to a Node.js target could hang, which left connections pending.

apigee-postgres backup/restore

Executing an apigee-postgres backup/restore now backs up data for all components.

Resetting sysadmin password

Resetting the system admin password for the Management Server that contained special characters would fail.

Known issues

The following table lists known issues in this release:

Issue ID Description

Mint not rating transactions initially

Edge for the Private Cloud does not rate transactions initially when you first enable monetization for an organization. This applies to new and existing organizations.

The monetization enablement API responds with the job ID and, upon completion, the status of the job will indicate success. However, any monetized API calls made initially will not be processed by monetization and will not be rated. These transactions will only show up in regular analytics.


Restart any Message Processors (MPs), which can be done with the following:
apigee-service edge-message-processor restart

After monetization has been enabled, restarting the MPs causes Edge to load the configuration from ZooKeeper at startup. This happens on every restart, which means that the issue only applies initially.


Mint rating servers memory leak

When a new Qpid instance is registered in an mxgroup, the queues build without consumers, which results in the queue depth growing until it uses all available resources.



Metadata file format for apigee-sso install

When installing apigee-sso and using a metadata.xml file instead of a URL, if the metadata.xml file does not include a new line (CRLF) at the end of the file, the metadata.xml file is not read entirely.


To work around this issue, you can do one of the following:

  • Add a CRLF to the end of the metadata.xml file


  • Use a URL-based location for the metadata

apigee-monit not supported on Amazon Linux 1



Message Processor backup not backing up the correct set of files


Run the backup a second time and it should back up the correct set of files.


DataAccessException in multiple data center configurations

In multiple data center configurations, if one datastore becomes unavailable, then you might see the following error:

DataAccessException: Error while accessing datastore;
Please retry later

The result is that Management Server may not start because it is trying to connect to Cassandra nodes in both dc-1 and dc-2. The DataAccessExceptions occurs if a Cassandra node is down. This might also result in API traffic disruption, where Message Processors report DataAccessExceptions while trying to retrieve KVMs.

Note that the expected state is for the Management Server not to connect to datastore components across regions.


The workaround is to deregister the following Cassandra node types in the unavailable data center and then re-register them after the Cassandra nodes are available again:

  • kms-datastore
  • dc-datastore
  • keyvaluemap-datastore

To deregister and reregister these Cassandra node types:

  1. Get the UUIDs of the Cassandra nodes by using the following curl command:
    curl -u ADMIN_EMAIL:ADMIN_PW \
      "http://MS_IP:MS_PORT/v1/servers?region=REGION&pod=GATEWAY_POD \


    • ADMIN_EMAIL and ADMIN_PW are the credentials of your Apigee account.
    • MS_IP and MS_PORT are the Management Server's IP address and port number.
    • REGION is the name of the data center in which the Management Server is located.
    • GATEWAY_POD is the pod name, which is by default "gateway". You might have renamed it to something else, though, so check your implementation.
    • CASSANDRA_NODE_TYPE is one of kms-datastore, dc-datastore, and keyvaluemap-datastore.

    For example:

    curl -u nickdanger@google.com:myP@$$w0rD

    The response uses the following format:

      "internalIP" : "POD_IP_ADDRESS",
      "isUp" : [true|false],
      "pod" : "GATEWAY_POD",
      "reachable" : [true|false],
      "region" : "dc-1",
      "tags" : {
        "property" : [ ]
      "type" : [ "kms-datastore", "dc-datastore", "keyvaluemap-datastore" ],
        "uUID" : "POD_UUID"

    For example:

      "internalIP" : "",
      "isUp" : false,
      "pod" : "gateway",
      "reachable" : false,
      "region" : "dc-1",
      "tags" : {
        "property" : [ ]
      "type" : "dc-datastore",
      "uUID" : "13cee956-d3a7-4577-8f0f-1694564179e4"

    Note the values of the uUID field in the response. You will use these to deregister the nodes.

  2. Repeat step one for each Cassandra node type: kms-datastore, dc-datastore, and keyvaluemap-datastore. Be sure to take note of the UUIDs that are returned.
  3. Deregister the nodes using the following command:
    curl -u ADMIN_EMAIL:ADMIN_PW "http://MS_IP:MS_PORT/v1/servers/UUID" -X DELETE

    Where UUID is the UUID returned in the previous command's response.

  4. Repeat step 3 for each UUID you collected in steps 1 and 2.
  5. Re-register the nodes using the following command:
    curl -u ADMIN_EMAIL:ADMIN_PW "http://MS_IP:MS_PORT/v1/servers -d \
      "Type=kms-datastore&Type=dc-datastore&Type=keyvaluemap-datastore& \
      Type=counter-datastore&Type=cache-datastore&InternalIP=POD_IP_ADDRESS& \
      region=REGION&pod=GATEWAY_POD" -H \
      'content-type: application/x-www-form-urlencoded' -X POST

Note that these operations register and deregister nodes from Zookeeper and do not have any impact on the Cassandra cluster. For more information about these commands, see Update datastore registrations.


MessageLogging policy including extra information in the log message

The FormatMessage element of the MessageLogging policy controls the format of the logged message. When FormatMessage=false, the logged message is not supposed to include any Apigee-generated information. However, even if you set FormatMessage=false, the log message still includes the following information:

  • The priority score
  • The timestamp



Next step

To get started with Edge for Private Cloud 4.19.06, use the following links:

New installations:
New installation overview
Existing installations:
Upgrade paths