MessageLogging policy

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

What

One of the best ways to track down problems in the API runtime environment is to log messages. You can attach and configure a MessageLogging policy on your API to log custom messages to a local disk (Edge for Private Cloud only) or to syslog.

Samples

<MessageLogging name="LogToSyslog">
  <Syslog>
    <Message>[3f509b58 tag="{organization.name}.{apiproxy.name}.{environment.name}"] Weather request for WOEID {request.queryparam.w}.</Message>
    <Host>logs-01.loggly.com</Host>
    <Port>514</Port>
    <Protocol>TCP</Protocol>
    <FormatMessage>true</FormatMessage>
    <DateFormat>yyyy-MM-dd'T'HH:mm:ss.SSSZ</DateFormat>
  </Syslog>
  <logLevel>ALERT</logLevel>
</MessageLogging>

<MessageLogging name="LogToSyslog">
  <Syslog>
    <Message>[3f509b58 tag="{organization.name}.{apiproxy.name}.{environment.name}"] Weather request for WOEID {request.queryparam.w}.</Message>
    <Host>logs-01.loggly.com</Host>
    <Port>6514</Port>
    <Protocol>TCP</Protocol>
    <FormatMessage>true</FormatMessage>
    <SSLInfo>
        <Enabled>true</Enabled>
    </SSLInfo>
    <DateFormat>yyMMdd-HH:mm:ss.SSS</DateFormat>
  </Syslog>
  <logLevel>WARN</logLevel>
</MessageLogging>

<MessageLogging name="LogPolicy">
  <File>
    <Message>This is a test message. Message id : {request.header.messageid}</Message>
      <FileName>test.log</FileName>
      <FileRotationOptions rotateFileOnStartup="true">
        <FileRotationType>SIZE</FileRotationType>
        <MaxFileSizeInMB>10</MaxFileSizeInMB>
        <MaxFilesToRetain>10</MaxFilesToRetain>
      </FileRotationOptions>
  </File>
  <logLevel>ERROR</logLevel>
</MessageLogging>

<MessageLogging name="LogPolicy">
  <File>
    <Message>This is a test message. Message id : {request.header.messageid}</Message>
    <FileName>test.log</FileName>
    <FileRotationOptions rotateFileOnStartup="true">
      <FileRotationType>TIME</FileRotationType>
      <RotationFrequency unit="minute">10</RotationFrequency>
      <MaxFilesToRetain>10</MaxFilesToRetain>
    </FileRotationOptions>
  </File>
  <logLevel>ERROR</logLevel>
</MessageLogging>

<MessageLogging name="LogPolicy">
  <File>
    <Message>This is a test message. Message id : {request.header.messageid}</Message>
    <FileName>test.log</FileName>
    <FileRotationOptions rotateFileOnStartup="true">
      <FileRotationType>TIME_SIZE</FileRotationType>
      <MaxFileSizeInMB>10</MaxFileSizeInMB>
      <MaxFilesToRetain>10</MaxFilesToRetain>
      <RotationFrequency unit="minute">10</RotationFrequency>
    </FileRotationOptions>
  </File>
  <logLevel>ERROR</logLevel>
</MessageLogging>

<MessageLogging name="LogPolicy">
  <File>
  ....
  ....
  </File>
  <BufferMessage>true</BufferMessage>
</MessageLogging>

Element reference

Use the following elements to configure the MessageLogging policy type.

<SSLInfo>
    <Enabled>true</Enabled>
</SSLInfo>

Schemas

Usage notes

When attaching a MessageLogging policy to an API proxy flow, consider placing it in the ProxyEndpoint response, in a special flow called PostClientFlow. The PostClientFlow executes after the response is sent to the requesting client, which ensures that all metrics are available for logging. For details on using PostClientFlow, see API proxy configuration reference.

The PostClientFlow is special in two ways:

1. It only executed as part of the response flow.
2. It is the only flow executed after the proxy enters the error state.

Because it is executed regardless of whether the proxy succeeded or failed, you can put MessageLogging policies in the PostClientFlow and be guaranteed that they always execute.

The following Trace image shows a MessageLogging policy executing as part of the PostClientFlow, after the DefaultFaultRule executes:

In this example, the Verify API Key policy caused the fault because of an invalid key.

Shown below is the ProxyEndpoint definition that includes the PostClientFlow:

<ProxyEndpoint name="default">
  ...
  <PostClientFlow>
    <Response>
      <Step>
        <Name>Message-Logging-1</Name>
      </Step>
    </Response>
  </PostClientFlow>
  ...
</ProxyEndpoint>

Edge logs messages as simple text, and you can configure logging to include variables, such as the date and time when the request or response was received, the user identity on the request, the source IP address from which the request was sent, and so on. Edge logs message asynchronously, meaning that no latency that might be caused by blocking callouts is introduced to your API.

The MessageLogging policy writes logged messages in memory to a buffer. The message logger reads messages from the buffer and then writes to the destination that you configure. Each destination has its own buffer.

If the write rate to the buffer increases beyond the read rate, the buffer overflows and logging will fail. If this happens, you might find a message containing the following in the log file:

Log message size exceeded. Increase the max message size setting

If you encounter this issue in Edge for Private Cloud 4.15.07 and earlier, locate the message-logging.properties file and use this solution:

Increase the max.log.message.size.in.kb property (default value = 128 KB) in the message-logging.properties file.

For Edge for Private Cloud 4.16.01 and later, set the conf/message-logging.properties+max. log.message.size.in.kb property in the /opt/apigee/customer/application/message-processor.properties file and restart the Message Processor. Please note that this property is initially commented out by default.

Note: The response message variables in Edge are not available from the Error Flow. These variables are also not available in PostClientFlow if the preceding flow was the Error Flow. If you want to log response information from the PostClientFlow, use the message object. You can use this object to get at headers and other information from the response whether or not there was an error. See Message variables for more information and an example.

Controlling log message timestamp in Edge for Private Cloud

By default, the timestamp in all log messages has the format:

yyyy-MM-dd'T'HH:mm:ss.SSSZ

This system-wide default can be overridden for syslog destinations using the DateFormat element. The behavior of this template is described in the documentation for Java's SimpleDateFormat class. According to that definition, yyyy will be replaced with a 4-digit year, MM will be replaced with a 2-digit month number, and so on. The above format might result in a string of this form:

2022-09-28T22:38:11.721+0000

You can use the conf_system_apigee.syslogger.dateFormat property on the Edge Message Processor to control that format. For example, changing the message format to:

yy/MM/dd'T'HH:mm:ss.SSSZ

..replacing the dashes with slashes, and shortening to a 2-digit year, records a timestamp in the form:

22/09/28T22:38:11.721+0000

To change the format:

1. Open the message-processor.properties file in an editor. If the file does not exist, create it:
   > vi /opt/apigee/customer/application/message-processor.properties
2. Set the properties as desired:
   conf_system_apigee.syslogger.dateFormat=yy/MM/dd'T'HH:mm:ss.SSSZ
3. Save your changes.
4. Make sure the properties file is owned by the 'apigee' user:
   > chown apigee:apigee /opt/apigee/customer/application/message-processor.properties
5. Restart the Edge Message Processor:
   > /opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart

Log file location in Edge for Private Cloud

Edge for Private Cloud 4.16.01 and later

By default, Private Cloud message logs are located in the following directory on Message Processor nodes:

/opt/apigee/var/log/edge-message-processor/messagelogging/org_name/environment/api_proxy_name/revision/logging_policy_name/

You can change the default log location by modifying properties in the message-logging.properties file on the Message Processors:

• bin_setenv_data_dir - Sets the root path for log file storage. For example, bin_setenv_data_dir=/opt/apigee/var/log
• conf_message-logging_log.root.dir - If you set this to a relative path, such as conf/message-logging.properties+log.root.dir=custom/folder/, the path is appended to the bin_setenv_data_dir location.
  
  If you set this to an absolute path, such as conf/message-logging.properties+log.root.dir=/opt/apigee/var/log/messages, message logs will be stored in /opt/apigee/var/log/messages/messagelog/. An absolute path takes precedence over bin_setenv_data_dir.
  
  Note that you have to reference the property as conf/message-logging.properties+log.root.dir because it is commented out by default. See Setting a token that is currently commented out for more.

If you want to store log files in a flat file structure so that all log files are put in the same directory, set the conf/message-logging.properties+enable.flat.directory.structure to true in the message-logging.properties file. Messages are stored in the directory specified by the properties above, and the file names take the form of {org}_{environment}_{api_proxy_name}_{revision}_{logging_policy_name}_{filename}.

To set these properties:

1. Open the message-processor.properties file in an editor. If the file does not exist, create it:
   > vi /opt/apigee/customer/application/message-processor.properties
2. Set the properties as desired:
   conf/message-logging.properties+log.root.dir=/opt/apigee/var/log/messages
3. Save your changes.
4. Make sure the properties file is owned by the 'apigee' user:
   > chown apigee:apigee /opt/apigee/customer/application/message-processor.properties
5. Restart the Edge component:
   > /opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart

Edge for Private Cloud 4.15.07 and earlier

By default, message logs are located in the following location on message processors:

/opt/apigee4/var/log/apigee/message-processor/messagelog/{org}/{environment}/{api_proxy_name}/{revision}/{logging_policy_name}/

You can change the default log location by modifying the following properties in the message-logging.properties file on the message processors:

• data.dir - Sets the root path for log file storage. For example, data.dir=/opt/apigee4/var/log
• log.root.dir - If you set this to a relative path, such as log.root.dir=custom/folder/, the path is appended to the data.dir location.

For example, the combination of the two properties would set the logging directory at /opt/apigee4/var/log/custom/folder/messagelog/ (note that /messagelog is added automatically).

If you set this to an absolute path, such as log.root.dir=/opt/apigee4/var/log/messages, message logs will be stored in /opt/apigee4/var/log/messages/messagelog/. An absolute path in log.root.dir takes precedence over data.dir.

If you want to store log files in a flat file structure so that all log files are put in the same directory, set the enable.flat.directory.structure property to true in the message-logging.properties file on Message Processors. Messages are stored in the directory specified by the properties above, and the file names take the form of {org}_{environment}_{api_proxy_name}_{revision}_{logging_policy_name}_{filename}.

Default values for variables in message template

Default values can be specified for each variable in message template separately. For example, if the variable request.header.id cannot be resolved, then its value is replaced with the value unknown.

<Message>This is a test message. id = {request.header.id:unknown}</Message>

A common default value can be specified for all the unresolved variables by setting the defaultVariableValue attribute on the Message element:

<Message defaultVariableValue="unknown">This is a test message. id = {request.header.id}</Message>

Configuring third-party log management services

The MessageLogging policy lets you send syslog messages to third-party log management services, such as Splunk, Sumo Logic, and Loggly. If you want to send syslog to one of those services, see that service's documentation to configure the service's host, port, and protocol, then set the Syslog element on this policy accordingly.

See the following documentation for third-party log management configuration:

• Splunk (select the product version)
  Also see this Apigee Community post: https://community.apigee.com/content/kbentry/13298/log-messages-into-splunk.html
• Sumo Logic
  • Also see this Apigee Community post: https://community.apigee.com/questions/5226/setting-up-logging-with-sumo-logic-which-host-shou.html
  • For a complete example using Sumo Logic as the logging service, see the following Apigee Community post. The solution uses a single JavaScript policy to make HTTP POST requests to Sumo Logic HTTP Source Collector: https://community.apigee.com/articles/32286/logging-to-sumo-logic-using-javascript-and-http.html
• Loggly
  When using Loggly, <FormatMessage>true</FormatMessage> is required in the policy as a child of the <Syslog> element.
  Also see this Apigee Community post for more information about message logging to Loggly: https://community.apigee.com/content/kbentry/14798/log-messages-into-loggly.html

Error reference

This section describes the fault codes and error messages that are returned and fault variables that are set by Edge when this policy triggers an error. This information is important to know if you are developing fault rules to handle faults. To learn more, see What you need to know about policy errors and Handling faults.

Runtime errors

These errors can occur when the policy executes.

Deployment errors

These errors can occur when you deploy a proxy containing this policy.

Fault variables

These variables are set when a runtime error occurs. For more information, see What you need to know about policy errors.

Example error response

{  
   "fault":{  
      "detail":{  
         "errorcode":"steps.messagelogging.StepDefinitionExecutionFailed"
      },
      "faultstring":"Execution failed"
   }
}

Example fault rule

<FaultRule name="MessageLogging">
    <Step>
        <Name>ML-LogMessages</Name>
        <Condition>(fault.name Matches "StepDefinitionExecutionFailed") </Condition>
    </Step>
    <Condition>(messagelogging.ML-LogMessages.failed = true) </Condition>
</FaultRule>

Flow variables

The following variables are populated on policy failure.

• messagelogging.failed
• messagelogging.{stepdefinition-name}.failed

Related topics

• Variables exposed by Edge: Variables reference