RegularExpressionProtection policy

You're viewing Apigee Edge documentation.
View Apigee X documentation.

What

Extracts information from a message (for example, URI Path, Query Param, Header, Form Param, Variable, XML Payload, or JSON Payload) and evaluates that content against predefined regular expressions. If any specified regular expressions evaluate to true, the message is considered a threat and is rejected.

Videos

Watch the following videos to learn more about the Regular Expression Protection policy.

Video Description
Protecting against SQL injection attacks (New Edge) Protect against SQL injection attacks using the Regular Expression Protection policy in the New Edge experience UI.
Protecting against SQL injection attacks (Classic Edge) Protect against SQL injection attacks using the Regular Expression Protection policy in the Classic Edge UI.

Samples

GitHub

The regex-protection sample on GitHub illustrates how to trap potential SQL injection attacks issued through a query parameter. Also, the sample illustrates a good practice of setting a generic 400 error status to prevent hackers from gaining any useful information from the response.

JavaScript include attack protection

<RegularExpressionProtection name="JsonPathRegExProtection">
    <DisplayName>Regular Expression Protection 1</DisplayName>
    <Source>request</Source>
    <JSONPayload escapeSlashCharacter="true">
       <JSONPath>
          <Expression>$.</Expression>
          <Pattern>&lt;\s*script\b[^&gt;]*&gt;[^&lt;]+&lt;\s*\/\s*script\s*&gt;
          </Pattern>
          <Pattern>n\s*\\\\\s*slash</Pattern>
          <Pattern>n\s*\/\s*slash</Pattern>
          <Pattern>n\s*\\"\s*quotes</Pattern>
          <Pattern>n\s*\\b\s*space</Pattern>
          <Pattern>n\s*\\f\s*forwardfeed</Pattern>
          <Pattern>n\s*\\n\s*newline</Pattern>
          <Pattern>n\s*\\r\s*carria</Pattern>
          <Pattern>n\s*\\t\s*tab</Pattern>
          <Pattern>n\s*\\uFFFF\s*hex</Pattern>
       </JSONPath>
    </JSONPayload>
 </RegularExpressionProtection>

The sample above illustrates how to use the RegularExpressionProtection policy to evaluate JSON payloads for JavaScript include attacks. Specifically, the content extracted by <JSONPath>/<Expression> is evaluated against the regular expression in <JSONPath>/<Pattern>.

If the regular expression in your <JSONPath>/<Pattern> includes XML-reserved characters (", &, ', <, or .), you must XML-encode it before you include it in the policy XML configuration file. For example, in the sample above, the regular expression <\s*script\b[^>]*>[^<]+<\s*\/\s*script\s*> has been XML-encoded as &lt;\s*script\b[^&gt;]*&gt;[^&lt;]+&lt;\s*\/\s*script\s*&gt;.

Additionally, if your regular expression includes forward slashes (/), you must escape them by setting the <JSONPayload> escapeSlashCharacter attribute to true.

Case insensitive matching

It's a common use case to do case-insensitive matching. Here's an example of how you can accomplish that in a regular expression using the construct (?i). In this example, for instance, DELETE, delete, and Delete will evaluate to true. 

<Pattern>[\s]*(?i)((delete)|(exec)|(drop\s*table)|(insert)|(shutdown)|(update)|(\bor\b))</Pattern>

About the Regular Expression Protection policy

Apigee Edge enables you to configure regular expressions that can be evaluated against API traffic at runtime to identify common content-level threats that follow certain patterns.

A regular expression, or regex for short, is a set of strings that specify a pattern in a string. Regular expressions enable content to be programmatically evaluated for patterns. Regular expressions can be used, for example, to evaluate an email address to ensure that it is properly structured. For more information, see Regular Expressions in the Java Tutorials.

The most common usage of RegularExpressionProtection is the evaluation of JSON and XML payloads for malicious content.

No regular expression can eliminate all content-based attacks, and multiple mechanisms should be combined to enable defense-in-depth. This section describes some recommended patterns for excluding content.

Example exclusion patterns

Regular expressions must be XML-encoded in the policy's XML configuration file.

Name Regular Expression
SQL Injection 
[\s]*((delete)|(exec)|(drop\s*table)|(insert)|(shutdown)|(update)|(\bor\b))
Server-Side Include Injection 
<!--#(include|exec|echo|config|printenv)\s+.*

XML encoded:

&lt;!--#(include|exec|echo|config|printenv)\s+.*
XPath Abbreviated Syntax Injection 
(/(@?[\w_?\w:\*]+(\[[^]]+\])*)?)+
XPath Expanded Syntax Injection 
/?(ancestor(-or-self)?|descendant(-or-self)?|following(-sibling))
JavaScript Injection 
<\s*script\b[^>]*>[^<]+<\s*/\s*script\s*>

XML encoded:

&lt;\s*script\b[^&gt;]*&gt;[^&lt;]+&lt;\s*/\s*script\s*&gt;
Java Exception Injection 
.*?Exception in thread.*

Set the Content-Type header in a request with an XML or JSON payload

The payload of the Regular Expression Protection policy can include the following elements:

  • <XMLPayload> element: Specifies that information needs to be extracted from an XML payload and evaluated against the regular expression provided.

    If you use <XMLPayload> in the policy, the Content-Type header of the request must be an XML content-type such as application/xml or text/xml.

  • <JSONPayload> element: Specifies that information needs to be extracted from a JSON payload and evaluated against the regular expression provided.

    If you use <JSONPayload> in the policy, the Content-Type header of the request must be a JSON content-type such as application/json.

Typically you design an API to accept either XML or JSON. However, there could be a scenario where the API accepted both. You could then define a Regular Expression Protection policy that uses both the <XMLPayload> and <JSONPayload> elements. Only one element would apply for a specific request based on the value of the Content-Type header.

Element reference

The element reference describes the elements and attributes of the RegularExpressionProtection policy.

<RegularExpressionProtection async="false" continueOnError="false" enabled="true" name="Regular-Expression-Protection-1">
   <DisplayName>Regular Expression Protection 1</DisplayName>
   <Source>response</Source>
   <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
   <URIPath>
     <Pattern>REGEX PATTERN</Pattern>
     <Pattern>REGEX PATTERN</Pattern>
   </URIPath>
   <QueryParam name="a-query-param">
     <Pattern>REGEX PATTERN</Pattern>
     <Pattern>REGEX PATTERN</Pattern>
   </QueryParam>
   <Header name="a-header">
     <Pattern>REGEX PATTERN</Pattern>
     <Pattern>REGEX PATTERN</Pattern>
   </Header>
   <FormParam name="a-form-param">
     <Pattern>REGEX PATTERN</Pattern>
     <Pattern>REGEX PATTERN</Pattern>
   </FormParam>
   <Variable name="request.content">
     <Pattern>REGEX PATTERN</Pattern>
     <Pattern>REGEX PATTERN</Pattern>
   </Variable>
   <XMLPayload>
     <Namespaces>
       <Namespace prefix="apigee">http://www.apigee.com</Namespace>
     </Namespaces>
     <XPath>
       <Expression>/apigee:Greeting/apigee:User</Expression>
       <Type>string</Type>
       <Pattern>REGEX PATTERN</Pattern>
       <Pattern>REGEX PATTERN</Pattern>
     </XPath>
   </XMLPayload>
   <JSONPayload>
     <JSONPath>
       <Expression>$.store.book[*].author</Expression>
       <Pattern>REGEX PATTERN</Pattern>
       <Pattern>REGEX PATTERN</Pattern>
     </JSONPath>
    </JSONPayload>
</RegularExpressionProtection>

<RegularExpressionProtection> attributes

<RegularExpressionProtection async="false" continueOnError="false" enabled="true" name="Regular-Expression-Protection-1">

The following table describes attributes that are common to all policy parent elements:

Attribute Description Default Presence
name

The internal name of the policy. The value of the name attribute can contain letters, numbers, spaces, hyphens, underscores, and periods. This value cannot exceed 255 characters.

Optionally, use the <DisplayName> element to label the policy in the management UI proxy editor with a different, natural-language name.

 N/A Required
continueOnError

Set to false to return an error when a policy fails. This is expected behavior for most policies.

Set to true to have flow execution continue even after a policy fails.

 false Optional
enabled

Set to true to enforce the policy.

Set to false to turn off the policy. The policy will not be enforced even if it remains attached to a flow.

 true Optional
async

This attribute is deprecated.

 false Deprecated

<DisplayName> element

Use in addition to the name attribute to label the policy in the management UI proxy editor with a different, natural-language name.

<DisplayName>Policy Display Name</DisplayName>
Default

N/A

If you omit this element, the value of the policy's name attribute is used.
Presence Optional
Type String

<Source> element

Indicates the message from which information needs to be extracted.

If the <Source> element is omitted, the value defaults to message. For example, <Source>message</Source>. When set to message, the policy uses the request message as source when attached to a request flow. Likewise, the policy uses the response message when attached to a response flow.

If the source message cannot be resolved or if it resolves to a non-message type, the policy returns an error.

<Source>response</Source>
Default: N/A
Presence: Optional
Type: String

<IgnoreUnresolvedVariables> element

Determines whether the policy returns an error when it encounters a variable that is unresolvable.

If set to false (the default), the policy returns an error when an unresolvable variable is encountered. If set to true, the unresolved variable is treated as empty string (Null).

<IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
Default: false
Presence: Optional
Type: Boolean

<URIPath> element

Specifies that information needs to be extracted from the request URI path and evaluated against the regular expressions provided. You must provide at least one <Pattern> element specifying a regular expression pattern to match.

<URIPath>
  <Pattern>REGEX PATTERN</Pattern>
  <Pattern>REGEX PATTERN</Pattern>
</URIPath>
Default: N/A
Presence: Optional
Type: N/A

<QueryParam> element

Specifies that information needs to be extracted from the request query parameter and evaluated against the regular expressions provided. You must provide at least one <Pattern> element specifying a regular expression pattern to match.

<QueryParam name="a-query-param">
  <Pattern>REGEX PATTERN</Pattern>
  <Pattern>REGEX PATTERN</Pattern>
</QueryParam>
Default: N/A
Presence: Optional
Type: N/A

Attributes

Attribute Description Default Presence
name Name of the request query parameter from which information needs to be extracted for evaluation against the regular expressions provided. N/A Required

<Header> element

Specifies that information needs to be extracted from the request and response headers and evaluated against the regular expressions provided. You must provide at least one <Pattern> element specifying a regular expression pattern to match.

<Header name="a-header">
  <Pattern>REGEX PATTERN</Pattern>
  <Pattern>REGEX PATTERN</Pattern>
</Header>
Default: N/A
Presence: Optional
Type: N/A

Attributes

Attribute Description Default Presence
name

Name of the request and response header from which information needs to be extracted for evaluation against the regular expressions provided.

 N/A Required

<FormParam> element

Specifies that information needs to be extracted from the request form parameter and evaluated against the regular expressions provided. You must provide at least one <Pattern> element specifying a regular expression pattern to match.

<FormParam name="a-form-param">
   <Pattern>REGEX PATTERN</Pattern>
   <Pattern>REGEX PATTERN</Pattern>     
</FormParam>
Default: N/A
Presence: Optional
Type: N/A

Attributes

Attribute Description Default Presence
name

Name of the request form parameter from which information needs to be extracted for evaluation against the regular expressions provided.

 N/A Required

<Variable> element

Specifies that information needs to be extracted from the given variable and evaluated against the regular expressions provided.

<Variable name="request.content">
   <Pattern>REGEX PATTERN</Pattern>
   <Pattern>REGEX PATTERN</Pattern>
</Variable>
Default: N/A
Presence: Optional
Type: N/A

Attributes

Attribute Description Default Presence
name

Name of the variable from which information needs to be extracted for evaluation against the regular expressions provided.

 N/A Required

<XMLPayload> element

Specifies that information needs to be extracted from an XML payload and evaluated against the regular expressions provided.

<XMLPayload>
   <Namespaces>
      <Namespace prefix="apigee">http://www.apigee.com</Namespace>
   </Namespaces>
   <XPath>
      <Expression>/apigee:Greeting/apigee:User</Expression>
      <Type>string</Type>
      <Pattern>REGEX PATTERN</Pattern>
      <Pattern>REGEX PATTERN</Pattern>
   </XPath>
</XMLPayload>
Default: N/A
Presence: Optional
Type: N/A

<XMLPayload>/<Namespaces> element

Specifies the namespaces to be used in the XPath evaluation.

<XMLPayload>
   <Namespaces>
      <Namespace prefix="apigee">http://www.apigee.com</Namespace>
   </Namespaces>
   <XPath>
      <Expression>/apigee:Greeting/apigee:User</Expression>
      <Type>string</Type>
      <Pattern>REGEX PATTERN</Pattern>
      <Pattern>REGEX PATTERN</Pattern>
   </XPath>
</XMLPayload>
Default: N/A
Presence: Optional
Type: String

<XMLPayload>/<Namespaces>/<Namespace> element

Specifies each namespace to be used in the XPath evaluation.
<Namespaces>
   <Namespace prefix="apigee">http://www.apigee.com</Namespace>
</Namespaces>
Default: N/A
Presence: Optional
Type: String

Attributes

Attribute Description Default Presence
prefix

Provides a prefix to help qualify a given namespace.

 N/A Required

<XMLPayload>/<XPath> element

Specifies the XPath to be evaluated.
<XPath>
   <Expression>/apigee:Greeting/apigee:User</Expression>
   <Type>string</Type>
   <Pattern>REGEX PATTERN</Pattern>
   <Pattern>REGEX PATTERN</Pattern>
</XPath>
Default: N/A
Presence: Optional
Type: N/A

<XMLPayload>/<XPath>/<Expression> element

Specifies the XPath expression defined for the variable. Only XPath 1.0 expressions are supported. For example, <Expression>/company/employee[@age>=$request.header.age]</Expression> extracts details for employees whose age is greater than or equal to the value specified in request.header.age.
<XPath>
   <Expression>/apigee:Greeting/apigee:User</Expression>
   <Type>string</Type>
   <Pattern>REGEX PATTERN</Pattern>
   <Pattern>REGEX PATTERN</Pattern>
</XPath>
Default: N/A
Presence: Optional
Type: String

<XMLPayload>/<XPath>/<Type> element

Specifies the datatype.
<XPath>
   <Expression>/apigee:Greeting/apigee:User</Expression>
   <Type>string</Type>
   <Pattern>REGEX PATTERN</Pattern>
   <Pattern>REGEX PATTERN</Pattern>
</XPath>
Default: string
Presence: Optional
Type: String
Valid values:

String. Valid values include string, boolean, int, long, float, double, and nodeset.

<XMLPayload>/<XPath>/<Pattern> element

Defines the regular expression pattern. If a regular expression in your<Pattern> element includes XML-reserved characters (", &, ', <, or .), you must XML-encode it before you include it.

<XPath>
   <Expression>/apigee:Greeting/apigee:User</Expression>
   <Type>string</Type>
   <Pattern>REGEX PATTERN</Pattern>
   <Pattern>REGEX PATTERN</Pattern>
</XPath>
Default: N/A
Presence: Required
Type: String

<JSONPayload> element

Specifies that information needs to be extracted from a JSON payload and evaluated against the regular expressions provided.

<JSONPayload>
   <JSONPath>
      <Expression>$.store.book[*].author</Expression>
      <Pattern>REGEX PATTERN</Pattern>
      <Pattern>REGEX PATTERN</Pattern>
   </JSONPath>
</JSONPayload>
Default: N/A
Presence: Optional
Type: N/A

Attributes

Attribute Description Default Presence
escapeSlashCharacter

Set to true to escape any forward slash (/) characters in the regular expressions in <JSONPath>/<Pattern>.

 true Optional

<JSONPayload>/<JSONPath>/<Expression> element

Specifies the JSONPath expression defined for the variable.

<JSONPath>
   <Expression>$.store.book[*].author</Expression>
   <Pattern>REGEX PATTERN</Pattern>
   <Pattern>REGEX PATTERN</Pattern>
</JSONPath>
Default: N/A
Presence: Optional
Type: String

<JSONPayload>/<JSONPath>/<Pattern> element

Defines the regular expression pattern. If a regular expression in your <Pattern> element includes XML-reserved characters (", &, ', <, or .), you must XML-encode it before you include it.

<JSONPath>
   <Expression>$.store.book[*].author</Expression>
   <Pattern>REGEX PATTERN</Pattern>
   <Pattern>REGEX PATTERN</Pattern>
</JSONPath>
Default: N/A
Presence: Required
Type: String

Error reference

This section describes the error codes and messages returned and fault variables set by Edge when this policy triggers an error. This information is important to know if you are developing fault rules to handle faults. If you want to capture an error and raise your own custom error, set the continueOnError="true" attribute on the policy root element. To learn more, see What you need to know about policy errors and Handling faults.

Errors returned from Edge policies follow a consistent format as described in the Error code reference.

Runtime errors

These errors can occur when the policy executes.

Error Code Message
ExecutionFailed Failed to execute the RegularExpressionProtection StepDefinition {0}. Reason: {1}
InstantiationFailed Failed to instantiate the RegularExpressionProtection StepDefinition {0}
NonMessageVariable Variable {0} does not resolve to a Message
SourceMessageNotAvailable {0} message is not available for RegularExpressionProtection StepDefinition {1}
ThreatDetected Regular Expression Threat Detected in {0}: regex: {1} input: {2}
VariableResolutionFailed Failed to resolve variable {0}

Deployment errors

Error Code Message Fix
CannotBeConvertedToNodeset RegularExpressionProtection {0}: Result of xpath {1} cannot be converted to nodeset. Context {2}
DuplicatePrefix RegularExpressionProtection {0}: Duplicate prefix {1}
EmptyJSONPathExpression RegularExpressionProtection {0}: Empty JSONPath expression
EmptyXPathExpression RegularExpressionProtection {0}: Empty XPath expression
InvalidRegularExpression RegularExpressionProtection {0}: Invalid Regular Expression {1}, Context {2}
JSONPathCompilationFailed RegularExpressionProtection {0}: Failed to compile jsonpath {1}. Context {2}
NONEmptyPrefixMappedToEmptyURI RegularExpressionProtection {0}: Non-empty prefix {1} cannot be mapped to empty uri
NoPatternsToEnforce RegularExpressionProtection {0}: No patterns to enforce in {1}
NothingToEnforce RegularExpressionProtection {0}: at least one of URIPath, QueryParam, Header, FormParam, XMLPayload, JSONPayload is mandatory
XPathCompilationFailed RegularExpressionProtection {0}: Failed to compile xpath {1}. Context {2}

Fault variables

These variables are set when this policy triggers an error. For more information, see What you need to know about policy errors.

Variables Where Example
fault.name="fault_name" fault_name is the name of the fault, as listed in the table above. fault.name Matches "ThreatDetected"
regularexpressionprotection.policy_name.failed policy_name is the user-specified name of the policy that threw the fault. regularexpressionprotection.Regular-Expressions-Protection-1.failed = true

Schemas

Related topics

JSON Threat Protection policy

XML Threat Protection policy