Changes in Edge for Private Cloud 4.53.01

Overview of changes

Edge for Private Cloud 4.53.01 has introduced multiple changes that enhance security posture of the platform, incorporating updated versions of software/Libraries. These changes affect:

  • OAS (OpenAPI specification) validation policy
  • Policies that support JSONPath queries
  • Java callout policy that relies on deprecated libraries
  • OpenLDAP

This section describes various types of changes introduced in version 4.53.01 that can disrupt your workflows during or after the upgrade. Methods to identify potential problem areas and methodologies of mitigation or workarounds are also covered.

OAS (OpenAPI specification) validation policy

Context

The OAS validation policy validates incoming requests or responses against the rules defined in OpenAPI 3.0 specification (JSON or YAML). Edge for Private Cloud 4.53.01 delivers enhancements to the OAS (OpenAPI specification) policy, focusing on stricter and more accurate validation of API response bodies.

Changes

The Edge for Private Cloud 4.53.01 introduces two important changes in how the OAS policy validates API responses, ensuring closer alignment with your OpenAPI specification:

  • Scenario 1:
    • Previous behavior: If your OpenAPI specification required a response body but the actual response from target or upstream policy did not include one , the policy would not flag this as a validation error.
    • Current behavior: The policy will now correctly return a validation error (example: defines a response schema but no response body found) in this scenario, indicating a mismatch between the expected and actual response.
  • Scenario 2:
    • Previous behavior: If your OpenAPI specification explicitly stated that no response body was expected, but the actual response from target or upstream policy included a body, the policy would not result in a failure.
    • Current behavior: The policy will now result in a failure (example: No response body is expected but one was found) in this scenario, ensuring that responses adhere strictly to the specified schema.

Mitigation

Identify any proxies or shared flows where OAS validation policy is configured with a Source tag set to response or those validating the response from any other policy that generates a response.

Once a proxy/shared flow is identified, ensure Alignment between Response and OAS Specification. The response must strictly align with your OpenAPI specification regarding the presence or absence of a response body. You can use standard Apigee trace to review your traffic patterns. If target is returning response intermittently then use other policies to validate it before OAS policy

  • If your OAS specification file defines a response body, the response from target or upstream policy must always provide one.
  • If your OAS specification file does not define a response body, the target or upstream policy must not send one.

Update the OAS validation policy or your target behavior as necessary before you attempt the upgrade to Private Cloud 4.53.01. You should validate such identified workflows in your non-production environments first to minimize risk of disruptions during production cluster upgrade.

JSON path

Context

The Edge for Private Cloud 4.53.01 has introduced changes to how JSON path expressions are used in various policies. JSONPath expressions can be used in policies like ExtractVariable policy, RegularExpressionProtection policy, Data masking to parse JSON content or store values in variables. JSONPath expressions can also be used in general message templating to replace variables with values dynamically during proxy execution. The new JSONPath expressions and formats follow the latest JSON expression standards.

Changes

It is important to review existing API proxies/shared flows for policies that utilize JSONPath expressions. This includes, but is not limited to, Extract Variables policy, Regular Expression Protection policy or any policy with Message Template using JSONPath.

Below json input is used for explaining the changes:

{
  "store": {
    "book": [
      {"category": "reference", "author": "Nigel Rees", "price": 8.95},
      {"category": "fiction", "author": "Evelyn Waugh", "price": 12.99},
      {"category": "fiction", "author": "Herman Melville", "price": 8.99}
    ],
    "bicycle": {
      "color": "red",
      "book": [
        {"author": "Abc"}
      ]
    }
  }
}
  1. JSONPath wildcard [*] behavior change for object values

    The behavior of the [*] wildcard has been altered when used to access all immediate values of a JSON object. Previously, $.object[*] would return the immediate values wrapped within a single JSON object. With the updated libraries, the output is now an array containing these values.

    For example, $.store[*]:

    Previous behavior: 
    {
  "bicycle": {
    "color": "red",
    "book": [{"author": "Abc"}]
  },
  "book": [
    {"price": 8.95, "category": "reference", "author": "Nigel Rees"},
    {"price": 12.99, "category": "fiction", "author": "Evelyn Waugh"},
    {"price": 8.99, "category": "fiction", "author": "Herman Melville"}
  ]
}
    Current behavior: 
    [
  [
    {"category": "reference", "author": "Nigel Rees", "price": 8.95},
    {"category": "fiction", "author": "Evelyn Waugh", "price": 12.99},
    {"category": "fiction", "author": "Herman Melville", "price": 8.99}
  ],
  {
    "color": "red",
    "book": [{"author": "Abc"}]
  }
]
    Action:

    Change the JSONPath expression to simply target the parent object (example: $.store) to directly target the items which were retrieved previously.

  2. JSONPath trailing dot (.) in paths causes error

    There is stricter validation for JSONPath expressions. Previously, paths ending with an invalid trailing dot (example: $.path.to.element.) would be silently ignored, and the query still returns a result if the preceding valid path segment matched. With the new version, such malformed paths are now correctly identified as invalid and will result in an error.

    For example, $.store.book.

    Previous behavior: 
    [
  {"price":8.95,"category":"reference","author":"Nigel Rees"},
  {"price":12.99,"category":"fiction","author":"Evelyn Waugh"},
  {"price":8.99,"category":"fiction","author":"Herman Melville"}
]
    Current behavior: 
    ERROR: com.jayway.jsonpath.InvalidPathException - Path must not end with a '.' or '..'

    Any existing policies that use JSONPath expressions with an unintentional trailing dot will now fail with an InvalidPathException.

    Action:

    Remove the trailing dot from any JSONPath expression that ends with one. For example, change $.store.book. to $.store.book.

  3. JSONPath recursive descent (..) output structure change

    There are changes to how results are returned when using the (..) (recursive descent) operator to locate all occurrences of a named element. Previously, all found elements were flattened into a single list. The updated libraries now return a list of lists, preserving the original grouping structure where elements were found, rather than a single flat list.

    For example, $..book

    Previous behavior: 
    [
  {"price":8.95,"category":"reference","author":"Nigel Rees"},
  {"price":12.99,"category":"fiction","author":"Evelyn Waugh"},
  {"price":8.99,"category":"fiction","author":"Herman Melville"},
  {"author":"Abc"}
]
    Current behavior: 
    [
  [
    {"category":"reference","author":"Nigel Rees","price":8.95},
    {"category":"fiction","author":"Evelyn Waugh","price":12.99},
    {"category":"fiction","author":"Herman Melville","price":8.99}
  ],
  [
    {"author":"Abc"}
  ]
]
    Action:

    Update your downstream processing logic to account for the new nested array structure. You will likely need to iterate through the outer JSONArray and then iterate through each inner JSONArray to access the individual elements.

  4. JSONPath indexing after multi-item selection or filter returns empty array

    There is a change in behavior when an index (example: [0]) is applied immediately after a multi-item selector (like [*]) or a filter ([?(condition)]). Previously, such expressions would attempt to select the item at the specified index from the combined results. With the new version, these expressions will now return an empty array ([]).

    For example, $.store.book[*][0]

    Previous behavior: 
    {"category": "reference", "price": 8.95, "author": "Nigel Rees"}
    Current behavior: 
    []
    Action:

    If there is a need to filter and then get a specific item from the filtered set, process the filtered array returned by the JSONPath for example, $..book[?(@.category == 'fiction')] and then take [0] from the previous result.

  5. JSONPath negative array slicing output change

    The new version has modified the behavior of negative array slicing (example: [-2:], [-1:]). Previously, when applying a negative slice to an array (indicating elements from the end of the array), the old version would incorrectly return only a single item from that slice. The new version now correctly returns a list (array) containing all elements that fall within the specified negative range.

    For example $.store.book[-2:]

    Previous behavior: 
    {"price":12.99,"category":"fiction","author":"Evelyn Waugh"}
    Current behavior: 
    [
  {"category":"fiction","author":"Evelyn Waugh","price":12.99},
  {"category":"fiction","author":"Herman Melville","price":8.99}
]
    Action:

    The downstream processing logic must now be updated to iterate through the returned JSON array to get the desired output.

  6. JSONPath stricter preceding dot

    There is stricter enforcement of syntax for elements accessed directly from the root. When the elements are accessed directly from the root without a preceding dot (example: $propertyelement), such syntax is now considered as an error and will prevent proxy deployment.

    For example $store,

    {
  "bicycle": {
    "color": "red",
    "book": [{"author": "Abc"}]
  },
  "book": [
    {"price": 8.95, "category": "reference", "author": "Nigel Rees"},
    {"price": 12.99, "category": "fiction", "author": "Evelyn Waugh"},
    {"price": 8.99, "category": "fiction", "author": "Herman Melville"}
  ]
}

    Current behavior:

    Proxy will fail to deploy.

    Action:

    Change your JSONPath to include the dot: $.propertyName (example: $.store). This will correctly target and retrieve the value.

  7. Dynamic JSONPath expressions

    Pay close attention to policies where the JSONPath expression itself is provided by a variable (example: {myJsonPathVariable} or {dynamicPath}). The value of these variables must also be checked against the potential behavioral changes outlined above.

Mitigation

To mitigate, a comprehensive strategy is necessary. This process involves deciding on the appropriate update path and applying the necessary fix for breaking JSONPath expressions.

Choose the Upgrade Path Method that works best for you:

  • Zero Downtime Migration

    This strategy involves procuring one or more new environments so that you can wire separate message processor nodes to it. Such message processor nodes can be set to install 4.53.01 and have proxies with modern JSONPath expressions. These can be used during upgrade and can be decommissioned after upgrade is complete. This strategy is seamless but involves temporarily procuring additional message-processor nodes to support a smooth upgrade. Details below:

    • Create a new environment and add new message-processor nodes of version 4.53.01 to this new environment.
    • Upload the proxy bundle for affected proxies to the new environment and apply necessary fixes explained in the remediation section & deploy the updated proxy bundle to the new environment.
    • Redirect the traffic to the new environment and un-deploy the affected proxies from the old environment.
    • Upgrade original message processor nodes to 4.53.01. Deploy proxies that have fixes for JSONPath in the original environment.
    • Switch traffic back to the old environment, which now has message processors on 4.53.01 and proxy that is modernized for new jsonpath expressions.
    • Delete and decommission the new environment and associated nodes.
  • Downtime and upgrade

    This strategy involves procuring downtime for API Proxies using faulty JSON Path expressions. It does not necessitate procuring additional message processor nodes but causes disruption to API traffic for impacted proxies.

    • Identify the affected proxies with affected policies and generate a new revision for all affected proxies.
    • Apply necessary fixes by Implementing the fixes explained in the remediation section in a new revision of the proxy. Don’t deploy this yet.
    • Procure downtime for the proxy/proxies with impact.
    • Upgrade all message processors to Edge for private cloud version 4.53.01. Note that the existing proxies may fail on the newly upgraded message processors.
    • Once all message processors are upgraded to Edge for private cloud version 4.53.01 , deploy the newly created proxy revision with fixed JSONPath expression.
    • Resume traffic on such proxies.
  • Redesign Proxy before upgrade

    You can redesign the proxy itself before upgrading to Edge for Private Cloud 4.53.01. Instead of relying on specific JSON path expressions, you can get the same result by using a different method.

    For example, if you are using an Extract Variable Policy with a JSON path, you could replace the policy with a Javascript policy that extracts similar data before upgrading to the newer version. After the upgrade is complete, you can change your proxy back to using JSON Paths with newer formats.

JavaCallout changes

Context

Edge for private cloud 4.53.00 and earlier used to contain a directory called deprecated ($APIGEE_ROOT/edge-message-processor/lib/deprecated) that contained a bunch of JAR libraries. These libraries were available for use in Java code in JavaCallout policy and could have been used by your custom Java code directly or indirectly.

Changes

The deprecated directory is now removed in Edge for private cloud version 4.53.01. In case your java code is relying on such libraries, proxies using such java callouts will fail when message processors are upgraded to version 4.53.01. In order to avoid such failures, follow the below mitigating steps before upgrading the message-processors to version 4.53.01.

Mitigation

  1. Review your Java-Callout policies and associated jars and identify if any of them reference or use any libraries present in the “deprecated” directory of your current message-processors. Note that java callouts may be using jars uploaded as organization or environment level resources. Consider these libraries too.
  2. Once you have identified such deprecated libraries , you can follow one of the below methods for mitigating the issue.
    • Resource placement (Recommended if you have a small number of jars / libraries from deprecated directory that are being referenced by the Java-Callout jars)
      • Upload the identified deprecated jars as a resource at the desired level: API proxy revision, environment or organization.
      • Proceed with Apigee software upgrade as usual.
    • Manual Placement (Recommended if you have a large number of jars / libraries that are being referenced by the Java-Callout jars)
      • On each message processor node, create a new directory named external-lib at the path $APIGEE_ROOT/data/edge-message-processor/.
      • Copy the identified JARs into this external-lib directory from deprecated directory: cp $APIGEE_ROOT/edge-message-processor/lib/deprecated/some.jar $APIGEE_ROOT/data/edge-message-processor/external-lib/some.jar
      • Ensure the directory and underlying jars are readable by Apigee user: chown -R apigee:apigee $APIGEE_ROOT/data/edge-message-processor/external-lib
      • Proceed with Apigee software upgrade as usual.

OpenLDAP change

Context

OpenLDAP can be used in Edge Private Cloud for both authentication and authorization. In Edge for Private Cloud 4.53.01, OpenLDAP software shipped by Apigee was upgraded from version 2.4 to 2.6.

Changes

In OpenLDAP 2.6, relative distinguished name (RDN) is limited to approximately 241 bytes/characters. This limitation is a hard cap enforced and cannot be modified.

Impact
  • Replication or import failures occur for entries with excessively large RDNs.
  • Attempting to create an entities like organizations,environments, custom roles, permissions etc. may results in the error message: "message": "[LDAP: error code 80 - Other]".
  • Any DN longer than 241 bytes in Apigee’s LDAP is impacted. Such DN’s will prevent successful upgrade of Apigee OpenLDAP software and you will have to follow mitigation strategies for such items before you can proceed with the upgrade.

Generally, in Apigee’s LDAP, long DNs are related to permissions since they are created by concatenating multiple entities. Such permission entries are especially prone to upgrade issues.

For example, 

dn: cn=@@@environments@@@*@@@applications@@@*@@@revisions@@@*@@@debugsessions,ou=resources,cn=businessuser,ou=userroles,o=orgname,ou=organizations,dc=apigee,dc=com

Typically, you would have organization, environment and role names of lengths such that RDNs in LDAP end up being smaller than 241 bytes.

Mitigation

Before upgrade to 4.53.01:

The following steps will help to verify the presence of long RDNs in your existing LDAP 2.4 cluster.

#1 - Extract LDAP data

Use ldapsearch command to find the distinguished name (dn) and redirect the output to a file: 

ldapsearch -o ldif-wrap=no -b "dc=apigee,dc=com" -D "cn=manager,dc=apigee,dc=com" -H ldap://:10389 -LLL -x -w LDAP_PASSWORD dn > /tmp/DN.ldif

Make sure the DN.ldif file above has LDAP entries.

#2 - Identify long RDNs

Find the RDNs exceeding 241 bytes/characters from the DN.ldif file above:

cat /tmp/DN.ldif |  grep '^dn:' | gawk -F',|dn: ' '{ rdn = $2; char_count = length(rdn); cmd = "echo -n \"" rdn "\" | wc -c"; cmd | getline byte_count; close(cmd); if (char_count > 241 || byte_count > 241) { print rdn, "(chars: " char_count ") (bytes: " byte_count ")"; }}'
o=VeryLongOrgNameWithMoreThan241Chars.... (chars: 245) (bytes: 245)
cn=VeryLongCustomRoleNameWithMoreThan241Chars.... (chars: 258) (bytes: 258)

If the above command produces no output, then no RDNs in the existing LDAP setup exceed 241 bytes/characters. You are safe to proceed with the upgrade as usual.

If the above command generates an output, it indicates the presence of RDNs exceeding 241 bytes/characters. For such items, follow mitigation steps as described in step#3 prior to proceeding with the Edge for Private Cloud 4.53.01 upgrade.

#3 - Handle long RDNs

If output is received from step#2, it indicates the presence of RDNs exceeding 241 bytes/characters and follow the below mitigation steps:

Review the LDAP entries that exceed 241 bytes.

  • If it is the custom role name, App, API product or other entities which is the primary factor causing the RDN to be long, migrate to using an alternate entity with a shorter name.
  • If it is the organization name or environment name that is the primary contributor to long RDN, you will have to migrate to a different organization or environment with a shorter name.

Keep repeating the steps above until your LDAP has no RDNs longer than 241 bytes. Once you reach this state, proceed with the private cloud version upgrade as usual.

Cryptography Provider changes

Context

This change is a carry-over from Edge for Private Cloud 4.53.00. In Edge for Private Cloud 4.53.00, the internal cryptography provider was updated from Bouncy Castle (BC) to Bouncy Castle FIPS (BCFIPS) to enable FIPS support.

Changes

If JavaCallout policies rely on using the original BC provider, especially when using security functionality that has been hardened in the BCFIPS provider (example, using a common key pair for both encryption and signing), such JavaCallout policies will need to be modernized. JavaCallout policies attempting to load the Bouncy Castle cryptography provider using the name BC may fail because the default provider has changed. Such policies using the BC provider can subsequently break. Any custom implementations relying on the old BC provider will no longer be accessible and will need to be reviewed and re-implemented.

Mitigation

The recommended workaround is to use the BCFIPS provider. Custom JavaCallout implementations that relied on the old provider will need to be reviewed and re-implemented using the Bouncy Castle FIPS provider, which can be accessed using the string "BCFIPS".

Automated change detection tool

A change detection tool is planned to be released soon. This tool will have the ability to scan and identify API proxies, shared flows, resources and LDAP RDNs that are potentially affected by various changes described in this article. This tool should help with identifying various entities that are prone to failure during or after upgrade to Edge for Private Cloud 4.53.01.