Configuring Message Processors to allow duplicate headers

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation.
info

As per the HTTP Specification RFC 7230, section 3.2.2: Field Order, Apigee Edge expects that the HTTP request from the client or HTTP response from the backend server do not contain the same header being passed more than once with the same or different values, unless the specific header has an exception and is allowed to have duplicates.

By default, Apigee Edge allows duplicates and multiple values to be passed to most of the HTTP headers. However, it does not allow certain headers which are listed in Headers that are not allowed to have duplicates and multiple values. Therefore:

  • You will get 400 Bad Request with error code protocol.http.DuplicateHeader if the client sends an HTTP request with particular header more than once or with multiple values for the HTTP headers which are not allowed to have duplicates/multiple values in Apigee Edge.
  • Similarly, you will get 502 Bad Gateway with error code protocol.http.DuplicateHeader if the backend server sends an HTTP response with particular header more than once or with multiple values for the HTTP headers which are not allowed to have duplicates or multiple values in Apigee Edge

The recommended solution to address these errors is to fix the client application and backend server to not send duplicate headers and adhere to the specification RFC 7230, section 3.2.2: Field Order as explained in the following troubleshooting playbooks:

However, in some cases you may want to add an exception to include duplicates and multiple values for some HTTP headers. In such situations, you can allow duplicate headers and multiple values for a specific HTTP header by setting a property HTTPHeader.HEADER_NAME at the Message Processor level.

This document provides information about this property, explains how to enable this property to avoid the above-mentioned errors and shares best practices around the same.

HTTP header properties to allow duplicates and multiple values

Apigee Edge provides the following two properties to control the behaviour of allowing duplicates and multiple values for HTTP headers. Note that these can be configured only on the Message Processors using the token syntax explained in How to configure Edge.

Property name Description Values allowed
HTTPHeader.ANY

This property indicates whether duplicates or multiple values are allowed for all HTTP headers including the custom headers sent as part of HTTP request made by the client or HTTP response sent by the backend server to Apigee Edge.

Default value:

multiValued, allowDuplicates,

  1. blank: Duplicates and multiple values for HTTP headers are not allowed.
  2. multiValued: Split the multi-valued header into multiple headers. Multiple values are allowed for HTTP headers, but duplicates are not allowed. The value multiValued is enabled, which implies that test-header=a,b would be converted to test-header=a and test-header=b.
  3. allowDuplicates: Allows multiple (duplicate) HTTP headers with the same name.
  4. multiValued, allowDuplicates: Both multiple values and duplicates are allowed for HTTP headers.

HTTPHeader.HEADER_NAME

This property is used to override the behavior of a specific header from what is specified by HTTPHeader.ANY

Same as above.

Headers that are not allowed to have duplicates and multiple values

As explained earlier, Apigee Edge allows duplicates and multiple values for most of the HTTP headers by default. This is because the property HTTPHeader.ANY is configured with the value multiValued, allowDuplicates.

Configuration overwritten

For some specific headers, the default configuration is overwritten using one of the following methods:

  • HTTPHeader.HEADER_NAME=multiValued, allowDuplicates

    This configuration does not change the default behaviour. That is, the specific header is allowed to have duplicates and multiple values

    .
  • HTTPHeader.HEADER_NAME=

    This configuration changes the default behaviour. That is, the specific header is not allowed to have duplicates and multiple values.

Determining headers that are not allowed to have duplicates and multiple values

This section describes how to identify the following:

  • The specific headers that are not allowed to have duplicates and multiple values on your Apigee Edge Private Cloud setup, and
  • The specific headers with pre-existing config
  1. On the Message Processor machine, search for the property HTTPHeader. in the /opt/apigee/edge-message-processor/conf directory as shown below:

    grep -ri "HTTPHeader." /opt/apigee/edge-message-processor/conf
    

    Sample output:

    # grep -ri "HTTPHeader" /opt/apigee/edge-message-processor/conf
    /opt/apigee/edge-message-processor/conf/http.properties:HTTPHeader.ANY=allowDuplicates, multiValued
    /opt/apigee/edge-message-processor/conf/http.properties:HTTPHeader.Connection=allowDuplicates, multiValued
    … <snipped>
    /opt/apigee/edge-message-processor/conf/http.properties:HTTPHeader.Host=
    /opt/apigee/edge-message-processor/conf/http.properties:HTTPHeader.Expires=
    /opt/apigee/edge-message-processor/conf/http.properties:HTTPHeader.Date=allowDuplicates
    …
    <snipped>
  2. As explained in the Configuration overwritten section, note the following information in the sample output above:
    1. The HTTP header Connection is overwritten, but allowed to have duplicates and multiple values
    2. The HTTP headers Host and Expires are overwritten and not allowed to have duplicates and multiple values
    3. The HTTP header Date is overwritten and allowed to have duplicates but not allowed to have multiple values
    4. All the headers which appear here (Connection, Host, Expires, and Date in the above sample) are referred to as headers with pre-existing config in this document.

Behaviour of Apigee Edge

The following table describes the behaviour of Apigee Edge when the headers are sent as duplicates and with multiple values depending on how the HTTPHeader properties are configured on the Message Processors with an example HTTPHeader of test-header.

Request Outgoing HEADERS based on value of conf/http.properties+HTTPHeader.test-header=
<Blank> allowDuplicates multiValued allowDuplicates, multiValued (DEFAULT)
test‑header=a,b test‑header=a,b test‑header=a,b

protocol.http.
DuplicateHeader

Internally we split test-header=a,b into:

  • test-header=a, and
  • test-header=b,

and then the DuplicateHeader error is thrown.

test‑header=a,b

Internally we split test-header=a,b into:

  • test-header=a, and
  • test-header=b,

but then the original form is sent to target.

test‑header=a
test‑header=b
protocol.http.
DuplicateHeader
test‑header=a
test‑header=b
protocol.http.
DuplicateHeader
test‑header=a
test‑header=b

Before you begin

Before you use the steps in this document, be sure you understand configuring properties for Edge on Private Cloud, described in How to configure Edge.

Configuring allowDuplicates and multiple values for headers

As explained in HTTP header properties to allow duplicates and multiple values, the value of the property HTTPHeader.ANY = allowDuplicates, multiValued implies that all headers are allowed to have duplicates and multiple values in Apigee Edge. However, there are certain headers whose values are overwritten explicitly to not allow duplicate headers or multiple values for thi using the property HTTPHeader.HEADER_NAME.

This section explains how to configure the property HTTPHeader.HEADER_NAME to allow duplicates and multiple values for any such HTTP headers on the Message Processors, using the corresponding token as per the syntax described in How to configure Edge.

In this section, we will use Expires (and myheader) as an example header for which we want to allow duplicates and multiple values as explained below:

  1. Determine the current value of the property HTTPHeaderHEADER_NAME to make sure it is not already enabled to allow duplicates and multiple values using the following command:
    grep -ri "HTTPHeader.HEADER_NAME" /opt/apigee/edge-message-processor/conf
    

    For example, if you are trying to set the property for the Expires header, then check the current value of the property HTTPHeader.Expires token on the Message Processor:

    grep -ri "HTTPHeader.Expires" /opt/apigee/edge-message-processor/conf
    

    The output of the above command results in one of the following:

    1. The property is set to blank, then it implies that the value is overwritten (and this is a header with pre-existing config) to NOT allow duplicate headers and multiple values. That is, you are not allowed to send the Expires header more than once as part of the HTTP request or HTTP response to Apigee.
    2. There are no hits for the specific property, then that means that value is not overwritten (and this is NOT a header with pre-existing config). This means that the specific header can be sent more than once (duplicates are allowed) as part of the HTTP request or HTTP response to Apigee Edge.
    3. The property is set with the value allowDuplicates, multiValued then that means that value is overwritten explicitly (and this is a header with pre-existing config). This means that the specific header can be sent more than once (duplicates are allowed) as part of the HTTP request or HTTP response to Apigee.

    Sample output of the search command:

    /opt/apigee/edge-message-processor/conf/http.properties:HTTPHeader.Expires=

    The above sample output shows that the property HTTPHeader.Expires is set to blank. This means that the property is overwritten to not allow duplicate or multiple values for the header Expires.

  2. If you notice that the property corresponding to the specific header is explicitly overwritten to not allow duplicate or multiple values as in the example output above, only then perform the following steps. If it is not explicitly overwritten, then skip the rest of the steps in this section.
  3. Edit. If it does not exist, you can create it:
    /opt/apigee/customer/application/message-processor.properties

    For example, to open the file using vi, enter the following:

    vi /opt/apigee/customer/application/message-processor.properties
    
  4. Add a line in the following format:
    conf_http_HTTPHeader.Expires=allowDuplicates, multiValued
  5. Save your changes.
  6. Ensure that the properties file is owned by the apigee user. If it is not, execute the following command:

    chown apigee:apigee /opt/apigee/customer/application/message-processor.properties
    
  7. Restart the Message Processor:

    /opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart
    

    To restart without traffic impact, see Rolling restart of Message Processors without traffic impact.

  8. If you have more than one Message Processor, repeat the above steps on all the Message Processors.

Verifying the header is configured to have duplicates and multiple values

This section explains how to verify that the property HTTPHeader.HEADER_NAME for a specific header has been updated successfully to allow duplicates on the Message Processors.

We will use Expires as an example header and check if the corresponding property HTTPHeader.Expires has been updated.

Even though you use the token conf_http_HTTPHeader.Expires to update the value on the Message Processor, you need to verify if the actual property HTTPHeader.Expires has been set with the new value.

  1. On the Message Processor machine, search for the property HTTPHeader.HEADER_NAME in the /opt/apigee/edge-message-processor/conf directory and check to see if it has been set with the new value as shown below:
    grep -ri "HTTPHeader.HEADER_NAME" /opt/apigee/edge-message-processor/conf
    

    For example, if you want to check that the property HTTPHeader.Expires is set with the new value, then run the following command:

    grep -ri "HTTPHeader.Expires" /opt/apigee/edge-message-processor/conf
    
  2. If the new value is successfully set for HTTPHeader.HEADER_NAME on the Message Processor, then the above command shows the new value in the http.properties file.
  3. The sample result from the above command after you have configured allowDuplicates and multiValued is as follows:

    /opt/apigee/edge-message-processor/conf/http.properties:HTTPHeader.Expires=allowDuplicates, multiValued
  4. In the example output above, note that the property HTTPHeader.Expires has been set with the new value allowDuplicates, multiValued in http.properties. This indicates that the behaviour to allow duplicates and multiple values in HTTPHeader is successfully configured on the Message Processor.
  5. If you still see the old value for the property HTTPHeader.HEADER_NAME, then verify that you have followed all the steps outlined in Configuring allowDuplicates and multiple values for headers correctly. If you have missed any step, repeat all the steps again correctly.

    Make sure your proxies are working as expected, especially if there is a functional logic to get and set the headers in the proxy.

  6. If you are still not able to modify the property, then contact Apigee Edge Support

Disabling allowDuplicates for headers

This section explains how to configure the property HTTPHeader.{Headername} to not allow duplicates and multiple values for a specific HTTP header on the Message Processors, using the corresponding token according to the syntax described in How to configure Edge.

In this section, we will use Expires (and myheader) as an example header for which we do not want to allow duplicates as explained below:

  1. Determine the current value of the property HTTPHeaderHEADER_NAME to make sure it is not already disabled to allow duplicates and multiple values using the following command:
    grep -ri "HTTPHeader.HEADER_NAME" /opt/apigee/edge-message-processor/conf
    

    For example if you are trying to set the property for Expires header then check the current value of the property HTTPHeader.Expires token on the Message Processor:

    grep -ri "HTTPHeader.Expires" /opt/apigee/edge-message-processor/conf
    

    The output of the above command results in one of the following:

    1. The property is set to blank, then it implies that the value is overwritten to NOT to allow duplicate headers and multiple values. That is you are not allowed to send the Expires header more than once as part of the HTTP request or HTTP response to Apigee.
    2. There are no hits for the specific property, then that means that value is not overwritten and this is a NOT header with pre-existing config. This means that the specific header can be sent more than once (duplicates are allowed) as part of the HTTP request or HTTP response to Apigee Edge.
    3. The property is set with the value allowDuplicates, multiValued then that means that value is overwritten explicitly and this is an existent config. However, this means that the specific header can be sent more than once (duplicates are allowed) as part of the HTTP request or HTTP response to Apigee.

    Sample output #1

    Sample Output #1 of the search command:

    /opt/apigee/edge-message-processor/conf/http.properties:HTTPHeader.Expires=allowDuplicates, multiValued

    The sample output shows that the property HTTPHeader.Expires is set to allowDuplicates, multiValued. This means that the property is overwritten to allow duplicate or multiple values for the header Expires.

    Sample output #2

    Sample Command and Output #2 of the search command

    grep -ri "HTTPHeader.myheader" /opt/apigee/edge-message-processor/conf
    

    The sample output shows no output, which implies that the property HTTPHeader.myheader is set to allowDuplicates, multiValued by default. This also implies that the property is not overwritten for the header myheader,

  2. If you notice one of the following, perform the rest of the steps in this section:
    1. The property corresponding to the specific header is overwritten to allow duplicates and multiple values as in the Sample output #1 above (header with pre-existing config)
    2. There are no hits for the property corresponding to the specific header as in the Sample output #2 above (not a header with pre-existing config)

    Otherwise, skip the rest of the steps in this section.

  3. Edit the following file. If it does not exist, you can create it.
    /opt/apigee/customer/application/message-processor.properties

    For example, to open the file using vi, enter the following:

    vi /opt/apigee/customer/application/message-processor.properties
    
  4. Add a line in the following format to the properties file:

    Pre-existing config

    Scenario #1: Header with pre-existing config:

    conf_http_HTTPHeader.Expires=

    No pre-existing config

    Scenario #2: Not a header with pre-existing config:

    conf/http.properties+HTTPHeader.myheader=
  5. Save your changes.
  6. Ensure that the properties file is owned by the apigee user. If it is not, execute the following:
    chown apigee:apigee /opt/apigee/customer/application/message-processor.properties
    
  7. Restart the Message Processor:
    /opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart
    

    To restart without traffic impact, see Rolling restart of Message Processors without traffic impact.

  8. If you have more than one Message Processor, repeat the above steps on all the Message Processors.

Verifying the header is configured to not allow duplicates and multiple values

This section explains how to verify that the property HTTPHeader.HEADER_NAME for a specific header has been updated successfully to not allow duplicates on the Message Processors.

We will use Expires (and myheader) as an example header and check if the corresponding property HTTPHeader.Expires (and HTTPHeader.myheader) has been updated.

  1. On the Message Processor machine, search for the property HTTPHeader.HEADER_NAME in the /opt/apigee/edge-message- processor/conf directory and check to see if it has been set with the new value as shown below:

    grep -ri "HTTPHeader.HEADER_NAME" /opt/apigee/edge-message-processor/conf
    

    For example, if you want to check the property HTTPHeader.Expires is set with the new value, then you can run the following command:

    Pre-existing config

    grep -ri "HTTPHeader.Expires" /opt/apigee/edge-message-processor/conf
    

    No pre-existing config

    grep -ri "HTTPHeader.myheader" /opt/apigee/edge-message-processor/conf
    
  2. If the new HTTP header value is successfully set for HTTPHeader.HEADER_NAME I on the Message Processor, then the above command shows the new value in the http.properties file.
  3. The sample result from the above command after you have disabled allowDuplicates is as follows:

    Pre-existing config

    Scenario #1: Expires Header (header with pre-existing config)

    /opt/apigee/edge-message-processor/conf/http.properties:HTTPHeader.Expires=

    No pre-existing config

    Scenario #2: myheader Header (not a header with pre-existing config)

    /opt/apigee/edge-message-processor/conf/http.properties:HTTPHeader.myheader=
  4. In the example output above, note that the property HTTPHeader.Expires ( and HTTPHeader.myheader ) has been set with the new value {blank} in http.properties. This indicates that the behaviour to allow duplicates and multiple values for the specific HTTP header Expires (and myheader) is successfully disabled on the Message Processor.
  5. If you still see the old value for the property HTTPHeader.Expires (or HTTPHeader.myheader), then verify that you have followed all the steps outlined in Configuring allowDuplicates and multiple values for headers correctly. If you have missed any step, repeat all the steps again correctly.

    Make sure your proxies are working as expected, especially if there is a functional logic to get and set the headers in the proxy.

  6. If you are still not able to modify the property, then contact Apigee Edge Support.