500 Internal Server Error - BadFormData

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation.
info

Symptom

The client application gets an HTTP status code of 500 Internal Server Error with the error code protocol.http.BadFormData as a response for API calls.

Error message

Client application gets the following response code:

HTTP/1.1 500 Internal Server Error

In addition, you may observe the following error message:

{
   "fault":{
      "faultstring":"Bad Form Data",
      "detail":{
         "errorcode":"protocol.http.BadFormData"
      }
   }
}

Form data

Before we get into the details of troubleshooting this issue, let’s understand what is form data.

Form data is the information provided by the user typically through an HTML form having elements such as a text input box, button, or a check box. The form data is generally sent as a series of key-value pairs as part of HTTP requests or responses.

Form data transmission

  1. Content-Type: application/x-www-form-urlencoded
    • If the size of the form data is small, then the data is sent as key-value pairs with:
      • The characters in both keys encoded as per the rules explained in Forms - Section 17.13.4.1
      • The header Content-Type: application/x-www-form-urlencoded

      Sample request with form data:

      curl https://HOSTALIAS/somepath -H "Content-Type: application/x-www-form-urlencoded" -d "username=abc@google.com&pasword=secret123"
      
    • Any non-alphanumeric characters in both keys and values are percent encoded, that is, they are represented as a character triplet %HH, consisting of a percent sign followed by two hexadecimal digits representing the ASCII code of the specific character.
    • Thus, even though the percent sign (%) is allowed in the form data, it is interpreted as the start of a special escape sequence. Therefore, if the form data needs to contain the percent sign (%) in the key or value, then it should be transmitted as %25, which represents the ASCII code for the percent sign (%) character.
  2. Content-Type: multipart/form-data

    If you want to transmit large quantities of binary data or text containing non-ASCII characters, then you can send the data with the Content-Type: multipart/form-data as explained in Forms - Section 17.13.4.2

Possible causes

This error occurs if and only if all of the following conditions are met:

  1. The HTTP request sent by the client to Apigee Edge contains:
    1. Content-Type: application/x-www-form-urlencoded, and
    2. Form data with the percent sign (%), or the percent sign (%) followed by invalid hexadecimal characters that are not allowed as per Forms - Section 17.13.4.1.
  2. The API proxy in Apigee Edge reads the specific form parameters containing any characters that are not allowed to be used in the request flow using the ExtractVariables or the AssignMessage policy.

    For example, if the form data contains the percent sign (%) as-is (without encoding) or the percent sign (%) followed by any invalid hexadecimal characters in the key and/or value, then you get this error.

    Here are the possible causes for this error:

    Cause Description Troubleshooting instructions applicable for
    Form Parameters in request has characters that are not allowed The form parameters passed as part of the HTTP request by the client contains any characters that are not allowed to be used. Edge Public and Private Cloud users

Common diagnosis steps

Use one of the following tools/techniques to diagnose this error:

API Monitoring

To diagnose the error using API Monitoring:

  1. Sign in to Apigee Edge UI as a user with an appropriate role.
  2. Switch to the organization in which you want to investigate the issue.

  3. Navigate to the Analyze > API Monitoring > Investigate page.
  4. Select the specific timeframe in which you observed the errors.
  5. Plot Fault Code against Time.

  6. Select a cell which has the fault code protocol.http.BadFormData as shown below:

    (view larger image)

  7. Information about the fault code protocol.http.BadFormData is displayed as shown below:

    (view larger image)

  8. Click View logs and expand the row for the failed request.

  9. From the Logs window, note the following details:
    • Status Code: 500
    • Fault Source: proxy
    • Fault Code: protocol.http.BadFormData
    • Fault Policy: extractvariables/EV-ExtractFormParams
  10. If the Fault Source is proxy, Fault Code is protocol.http.BadFormData and Fault Policy is non-empty, then it indicates that the error occurred while the specific policy indicated in Fault Policy was reading or extracting the form data (form parameters) which has any characters that are not allowed to be used.
  11. In this example, X-Apigee-fault-policy is extractvariables/EV- ExtractFormParams, which means that the ExtractVariables policy named EV-ExtractFormParams failed while reading or extracting the form parameters.

Trace tool

To diagnose the error using the Trace tool:

  1. Enable the trace session and either:
    • Wait for the 500 Internal Server Error error to occur, or
    • If you can reproduce the issue, make the API call to reproduce the issue 500 Internal Server Error
  2. Ensure Show all FlowInfos is enabled:

  3. Select one of the failing requests and examine the trace.
  4. Navigate through different phases of the trace and locate where the failure occurred.
  5. You will find the error typically in one of the policies as shown below:

    In the above sample trace, note that the failure occurred in the ExtractVariables policy named EV-ExtractFormParams.

  6. Navigate to the flow named Error after the specific policy that failed:

  7. Note the values of the following from the trace:

    error: Bad Form Data

    state: PROXY_REQ_FLOW

    error.class: com.apigee.rest.framework.BadRequestException

    • The value of the error Bad Form Data indicates that the form parameters had some characters that are not allowed to be used.
    • The value of the state PROXY_REQ_FLOW, indicates that the error occurred in the request flow of the API proxy.
  8. Navigate to the AX (Analytics Data Recorded) Phase in the trace and click it.
  9. Scroll down to the Phase Details - Error Headers section and determine the values of X-Apigee-fault-code, X-Apigee-fault-source, and X-Apigee-fault-policy as shown below:

  10. Note that the values of X-Apigee-fault-code and X-Apigee-fault-source are protocol.http.BadFormData and policy respectively and X-Apigee-fault-policy is non-empty. This indicates that the error occurred while the specific policy indicated in X-Apigee-fault-policy was reading or extracting the form data (form parameters), which had any characters that are not allowed to be used.

    Response headers Value
    X-Apigee-fault-code protocol.http.BadFormData
    X-Apigee-fault-source policy
    X-Apigee-fault-policy extractvariables/EV-ExtractFormParams
  11. In this example, X-Apigee-fault-policy is extractvariables/EV- ExtractFormParams, which means that the ExtractVariables policy named EV-ExtractFormParams failed while reading or extracting the form parameters.

NGINX

To diagnose the error using NGINX access logs:

  1. If you are a Private Cloud user, then you can use NGINX access logs to determine the key information about HTTP 500 Internal Server Error.
  2. Check the NGINX access logs:

    /opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log

  3. Search to see if there are any 500 errors with error code protocol.http.BadFormData during a specific duration (if the problem happened in the past) or if there are any requests still failing with 500.
  4. If you do find any 500 errors with the X-Apigee-fault-code matching the value of protocol.http.BadFormData, then determine the value of the X-Apigee-fault-source and X-Apigee-fault-policy.

    Sample 500 error from NGINX access log:

    The above sample entry from the NGINX access log has the following values for X-Apigee-fault-code and X-Apigee-fault-source:

    Headers Value
    X-Apigee-fault-code protocol.http.BadFormData
    X-Apigee-fault-source policy
    X-Apigee-fault-policy extractvariables/EV-ExtractFormParams
  5. Note that the values of X-Apigee-fault-code, X-Apigee-fault-source are protocol.http.BadFormData, policy respectively and X-Apigee-fault-policy is non-empty. This indicates that the error occurred while the specific policy indicated in X-Apigee-fault-policy, was reading or extracting the form data (form parameters), which had any characters that are not allowed to be used.
  6. In this example, X-Apigee-fault-policy is extractvariables/EV- ExtractFormParams, which means that the ExtractVariables policy named EV-ExtractFormParams failed while reading the form parameters.

Cause: Form parameters in request has characters that are not allowed

Diagnosis

  1. Determine the Fault Code, Fault Source, and Fault Policy for 500 Internal Server Error using API Monitoring, Trace tool or NGINX access logs as explained in Common diagnosis steps.
  2. If the Fault Code is protocol.http.BadFormData, Fault Source has the value proxy or policy, and Fault Policy is non- empty, then this indicates that policy specified in Fault Policy failed while reading or extracting the form data (form parameters).
  3. Examine the policy indicated in the Fault Policy and determine the following information:
    1. Source: Determine whether the policy is reading or extracting the data from request or response.
    2. Form parameters: Determine the specific form parameters that are being read in the policy.

      Sample #1

      Sample #1: ExtractVariables policy extracting form parameters:

            <ExtractVariables name="EV-ExtractFormParms">
               <DisplayName>EV-ExtractFormParams</DisplayName>
               <Source>request</Source>
               <FormParam name="username">
                  <Pattern ignoreCase="false">{username}</Pattern>
               </FormParam>
               <FormParam name="password">
                 <Pattern ignoreCase="false">{password}</Pattern>
               </FormParam>
               <VariablePrefix>forminfo</VariablePrefix>
             <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
            </ExtractVariables>
            

      In the above ExtractVariables policy:

      • Source: request

        This is indicated by the <Source> element

      • Form parameters: username and password

        This is indicated by the <Pattern> element within the <FormParam> element

      This indicates that the form parameters username and/or password passed as part of the HTTP request by the client to Apigee Edge contains characters that are not allowed to be used.

      Sample #2

      Sample #2: AssignMessage policy copying form parameters:

            <AssignMessage continueOnError="false" enabled="true" name="AM-CopyFormParams">
              <Copy source="request">
                <FormParams>
                  <FormParam name="username"/>
                  <FormParam name="password"/>
                </FormParams>
              </Copy>
              <AssignTo createNew="true" transport="http" type="request"/>
            </AssignMessage>
            

      In the above ExtractVariables policy:

      • Source: request

        This is indicated by the source attribute in the <Copy> element

      • Form parameters: username and password

        This is indicated by the name attribute in the <FormParam> element

      This indicates that the form parameters username or password or both passed as part of the HTTP request by the client to Apigee Edge contains any characters that are not allowed to be used.

  4. Check to see if there are any characters that are not allowed to be used characters in the form parameters identified in step 3 using one of the following methods:

    Trace tool

    To validate using the Trace tool:

    1. If you have captured the trace for the failing request as explained in Common diagnosis steps, then select one of the failing requests.
    2. If you have determined that the form parameters containing any characters that are not allowed to be used are part of the HTTP request in step 3 above, then
      1. Navigate to the phase Request Received from Client.
      2. Scroll down to the Phase Details section and review the Request Content.

        ( view larger image)

      3. In the above example, note that the form parameter password contains the percent sign (%).
      4. Since the percent sign (%) is also used for percent encoding the special characters, it cannot be used as-is in the form data.
      5. Therefore, Apigee Edge responds with 500 Internal Server Error with the error code protocol.http.BadFormData.

    Actual request

    To validate using the actual request:

    1. If you don’t have access to the actual request made to the target server, then go to Resolution.
    2. If you have access to the actual request made to Apigee Edge, then perform the following steps:
      1. Review the form data contents and see if it contains any characters that are not allowed to be used such as the percent sign (%) or the percent sign (%) followed by invalid hexadecimal characters.

        Sample #1

        Sample Request #1: Form data as part of the request

        curl -X GET "https://HOSTALIAS/myproxy -H "Content-Type: application/x-www-form-urlencoded" -d "client_id=123456abc123&client_secret=c23578%ZY"
        

        In this example, note that the element client_secret contains the percent sign (%) followed by invalid hexadecimal characters ZY.

        Sample #2

        Sample Request #2: Form data passed in a file:

        curl -X GET "https://HOSTALIAS/myproxy -H "Content-Type: application/x-www-form-urlencoded" -d @form_data.xml
        

        Contents of form_data.xml:

        xml=<user><username>abc1234@google.com</username><password>qwerty12345!@#$%</password></user>
        

        In this example, note that the element password contains the percent sign (%), which should not be passed as-is in the form data.

    3. In the above two examples, the form data sent as part of HTTP request to Apigee Edge contains characters that are not allowed to be used.
    4. Therefore, Apigee Edge responds with 500 Internal Server Error with the error code protocol.http.BadFormData.

Resolution

  1. Ensure that any special characters in both the keys and values of form data or parameters sent as part of HTTP request by the client are always encoded as explained in Form Data - application/x-www-form-urlencoded.
  2. For the examples discussed above, you can fix the issues as follows:

    Sample #1

    Sample #1: Form data passed as part of the request:

    Use valid hexadecimal characters that match the ASCII code for a specific character. For example, if you want to send the dollar sign ($), use %24 as shown below:

    curl -X GET "https://HOSTALIAS/myproxy -H "Content-Type: application/x-www-form-urlencoded" -d "client_id=123456abc123&client_secret=c23578%24"
    

    Sample #2

    Sample Request #2: Form data passed in a file:

    curl -X GET "https://HOSTALIAS/myproxy -H "Content-Type: application/x-www-form-urlencoded" -d @form_data.xml
    

    Contents of form_data.xml:

    Use the percent-encoding for the percent (%) sign, that is modify the file to have %25 as shown below:

    xml=<user><username>abc1234@google.com</username><password>qwerty12345!!@#$%25</password></user>
    

Specification

Apigee Edge expects that the form data to be sent as per the following specifications:

Specification
Form Data - application/x-www-form-urlencoded

If you still need any assistance from Apigee Support, go to Must gather diagnostic information.

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information, and then contact Apigee Edge Support:

If you are a Public Cloud user, provide the following information:

  • Organization name
  • Environment name
  • API proxy name
  • Complete curl command used to reproduce the 500 Internal Server Error with the error code protocol.http.BadFormData
  • Trace file for the API requests

If you are a Private Cloud user, provide the following information:

  • Complete error message observed for the failing requests
  • Environment name
  • API proxy bundle
  • Trace file for the API requests
  • NGINX access logs

    /opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log

    Where: ORG, ENV and PORT# are replaced with actual values.

  • Message Processor system logs

    /opt/apigee/var/log/edge-message-processor/logs/system.log

References