Develop custom plugins

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Edge Microgateway v. 3.1.5 and later

Audience

This topic is intended for developers who wish extend Edge Microgateway features by writing custom plugins. If you wish to write a new plugin, experience with JavaScript and Node.js is required.

What is a custom Edge Microgateway plugin?

A plugin is a Node.js module that adds functionality to Edge Microgateway. Plugin modules follow a consistent pattern and are stored in a location known to Edge Microgateway, enabling them to be discovered and run automatically. Several predefined plugins are provided when you install Edge Microgateway. These include plugins for authentication, spike arrest, quota, and analytics. These existing plugins are described in Use plugins.

You can add new features and capabilities to the microgateway by writing custom plugins. By default, Edge Microgateway is essentially a secure pass-through proxy that passes requests and responses unchanged to and from target services. With custom plugins, you can programmatically interact with the requests and responses that flow through the microgateway.

Where to put custom plugin code

A folder for custom plugins is included as part of the Edge Microgateway installation here:

[prefix]/lib/node_modules/edgemicro/node_modules/microgateway-plugins

where [prefix] is the npm prefix directory as described in "Where is Edge Microgateway installed" in Installing Edge Microgateway.

You can change this default plugin directory. See Where to find plugins.

Reviewing the predefined plugins

Before you try to develop your own plugin, it's good to check that none of the predefined plugins meet your requirements. These plugins are located in:

[prefix]/lib/node_modules/edgemicro/node_modules/microgateway-plugins

where [prefix] is the npm prefix directory. See also "Where is Edge Microgateway installed" in Installing Edge Microgateway.

For details, see also Predefined plugins provided with Edge Microgateway.

Write a simple plugin

In this section, we'll walk through the steps required to create a simple plugin. This plugin overrides the response data (whatever it is) with the string "Hello, World!" and prints it to the terminal.

If Edge Microgateway is running, stop it now:
```
edgemicro stop
```
cd to the custom plugin directory:
cd [prefix]/lib/node_modules/edgemicro/plugins

where [prefix] is the npm prefix directory as described in "Where is Edge Microgateway installed" in Installing Edge Microgateway.
Create a new plugin project called response-override and cd to it:
```
mkdir response-override && cd response-override
```
Create a new Node.js project:
```
npm init
```
Hit Return multiple times to accept the defaults.
Use a text editor to create a new file called index.js.
Copy the following code into index.js, and save the file.
```
'use strict';
var debug = require('debug')

module.exports.init = function(config, logger, stats) {

  return {
   
    ondata_response: function(req, res, data, next) {
      debug('***** plugin ondata_response');
      next(null, null);
    },
    
    onend_response: function(req, res, data, next) {
      debug('***** plugin onend_response');
      next(null, "Hello, World!\n\n");
    }
  };
}
```
Note: The plugin functions (*_response) are event handlers. The ondata_response function is called whenever a chunk of data is received by Edge Microgateway. The onend_response function is called when the last byte of response data is received.

Note: Important: The next() function is required by all plugin functions. This function passes error (first argument) and data (second argument) to the next plugin function that is called. Note that in this example, the data argument in ondata_response is null. Setting this argument to null allows us to override the response data in onend_response. If we did not set the argument to null, then, by default, the data would be passed through to the response whether or not we override it in onend_response.
Now you've created a plugin, and you need to add it to the Edge Microgateway configuration. Open the file $HOME/.edgemicro/[org]-[env]-config.yaml, where org and env are your Edge organization and environment names.
Add the response-override plugin to the plugins:sequence element as shown below.
```
      ...
      
      plugins:
        dir: ../plugins
        sequence:
          - oauth
          - response-override
          
      ...
```
Note: Note: Plugins are executed in the order in which they appear in the plugins:sequence list.
Restart Edge Microgateway.
Call an API through Edge Microgateway. (This API call assumes you've set up the same configuration as the tutorial with API key security, as described in Setting up and configuring Edge Microgateway:
```
curl -H 'x-api-key: uAM4gBSb6YoMvTHfx5lXJizYIpr5Jd' http://localhost:8000/hello/echo
Hello, World!
```

Anatomy of a plugin

The following Edge Microgateway sample plugin illustrates the pattern to follow when developing your own plugins. The source code for the sample plugin discussed in this section is in the plugins/header-uppercase/index.js.

Plugins are standard NPM modules with a package.json and index.js in the root folder.
A plugin must export an init() function.
The init() function takes three arguments: config, logger, and stats. These arguments are described in Plugin init() function arguments.
init() returns an object with named function handlers that are called when certain events occur during the lifetime of a request.

Event handler functions

A plugin must implement some or all of these event handler functions. Implementation of these functions is up to you. Any given function is optional, and a typical plugin will implement at least a subset of these functions.

Request flow event handlers

These functions are called on request events in Edge Microgateway.

onrequest
ondata_request
onend_request
onclose_request
onerror_request

onrequest function

Called at the start of the client request. This function fires when the first byte of the request is received by Edge Microgateway. This function gives you access to the request headers, URL, query parameters, and HTTP method. If you call next with a truthy first argument (such as an instance of Error), then request processing stops and a target request is not initiated.

Example:

onrequest: function(req, res, next) {
      debug('plugin onrequest');
      req.headers['x-foo-request-start'] = Date.now();
      next();
    }

ondata_request function

Called when a chunk of data is received from the client. Passes request data to the next plugin in the plugin sequence. The returned value from the last plugin in the sequence is sent to the target. A typical use case, shown below, is to transform the request data before sending it to the target.

Example:

ondata_request: function(req, res, data, next) {
      debug('plugin ondata_request ' + data.length);
      var transformed = data.toString().toUpperCase();
      next(null, transformed);
    }

onend_request function

Called when all of the request data has been received from the client.

Example:

onend_request: function(req, res, data, next) {
      debug('plugin onend_request');
      next(null, data);
    }

onclose_request function

Indicates the client connection has closed. You might use this function in cases where the client connection is unreliable. It is called when the socket connection to the client is closed.

Example:

onclose_request: function(req, res, next) {
      debug('plugin onclose_request');
      next();
    }

onerror_request function

Called if there is an error receiving the client request.

Example:

onerror_request: function(req, res, err, next) {
      debug('plugin onerror_request ' + err);
      next();
    }

Response flow event handlers

These functions are called on response events in Edge Microgateway.

onresponse
ondata_response
onend_response
onclose_response
onerror_response

onresponse function

Called at the start of the target response. This function fires when the first byte of the response is received by Edge Microgateway. This function gives you access to the response headers and status code.

Example:

onresponse: function(req, res, next) {      
    debug('plugin onresponse');     
    res.setHeader('x-foo-response-time', Date.now() - req.headers['x-foo-request-start'])    
    next();    
}

ondata_response function

Called when a chunk of data is received from the target.

Example:

ondata_response: function(req, res, data, next) {
      debug('plugin ondata_response ' + data.length);
      var transformed = data.toString().toUpperCase();
      next(null, transformed);
    }

onend_response function

Called when all of the response data has been received from the target.

Example:

onend_response: function(req, res, data, next) {
      debug('plugin onend_response');
      next(null, data);
    }

onclose_response function

Indicates the target connection has closed. You might use this function in cases where the target connection is unreliable. It is called when the socket connection to the target is closed.

Example:

onclose_response: function(req, res, next) {
      debug('plugin onclose_response');
      next();
    }

onerror_response function

Called if there is an error receiving the target response.

Example:

onerror_response: function(req, res, err, next) {
      debug('plugin onerror_response ' + err);
      next();
    }

What you need to know about the plugin event handler functions

Plugin event handler functions are are called in response to specific events that occur while Edge Microgateway processes a given API request.

Each of the init() function handlers (ondata_request, ondata_response, etc) must call the next() callback when done processing. If you don't call next(), processing will stop and the request will hang.
The first argument to next() may be an error which will cause request processing to terminate.
The ondata_ and onend_ handlers must call next() with a second argument containing the data to be passed to the target or the client. This argument can be null if the plugin is buffering and has not enough data to transform at the moment.
Note that one single instance of the plugin is used to service all requests and responses. If a plugin wishes to retain per-request state between calls, it can save that state in a property added to the supplied request object (req), whose lifetime is the duration of the API call.
Be careful to catch all errors and call next() with the error. Failure to call next() will result in the API call hanging.
Be careful not to introduce memory leaks as that can affect the overall performance of Edge Microgateway and cause it to crash if it runs out of memory.
Be careful to follow the Node.js model by not doing compute-intensive tasks in the main thread as this can adversely affect performance of Edge Microgateway.

About the plugin init() function

This section describes the arguments passed to the init() function: config, logger, and stats.

config

Configuration data obtained by merging the Edge Microgateway configuration file with the data downloaded from Apigee Edge is placed in an object named: config.

To add a config parameter called param with a value of foo to a plugin called response-override, put this in the default.yaml file:

response-override:
    param: foo

Then, you can access the parameter in your plugin code, like this:

// Called when response data is received
    ondata_response: function(req, res, data, next) {
      debug('***** plugin ondata_response');
      debug('***** plugin ondata_response: config.param: ' + config.param);
      next(null, data);
    },

In this case, you will see foo printed in the plugin debug output:

Sun, 13 Dec 2015 21:25:08 GMT plugin:response-override ***** plugin ondata_response: config.param: foo

You can access the merged microgateway configuration and downloaded Apigee Edge data in the child object config.emgConfigs. For example, you can access this configuration data in the init function as follows:

module.exports.init = function(config, logger, stats) {
   let emgconfigs = config.emgConfigs;

Note: The emgConfigs object was added in v3.1.6. The key name emgConfigs is reserved and cannot be used for defining any other configuration under a plugin’s configuration in the YAML configuration file. If defined, the value will be overwritten at runtime.

Below is an example of data that emgConfigs contains:

{
    edgemicro:
    {
        port: 8000,
        max_connections: 1000,
        config_change_poll_interval: 600,
        logging:
        {
            level: 'error',
            dir: '/var/tmp',
            stats_log_interval: 60,
            rotate_interval: 24,
            stack_trace: false
        },
        plugins: { sequence: [Array] },
        global: { org: 'Your Org', env: 'test' }
    },
    headers:
    {
        'x-forwarded-for': true,
        'x-forwarded-host': true,
        'x-request-id': true,
        'x-response-time': true,
        via: true
    },
    proxies:
    [    {
                max_connections: 1000,
                name: 'edgemicro_delayed',
                revision: '1',
                proxy_name: 'default',
                base_path: '/edgemicro_delayed',
                target_name: 'default',
                url: 'https://httpbin.org/delay/10',
                timeout: 0
            }
    ],
    product_to_proxy: { EdgeMicroTestProduct: [ 'edgemicro-auth','edgemicro_delayed',] },
    product_to_scopes: {prod4: [ 'Admin', 'Guest', 'Student' ] },
    product_to_api_resource: { EdgeMicroTestProduct: [ '/*' ] },
    _hash: 0,
    keys: { key: 'Your key', secret: 'Your key ' },
    uid: 'Internally generated uuid',
    targets: []
  }

logger

The system logger. The currently employed logger exports these functions, where object can be a string, HTTP request, HTTP response, or an Error instance.

info(object, message)
warn(object, message)
error(object, message)
trace(object, message)
debug(object, message)

stats

An object that holds counts of requests, responses, errors, and other aggregated statistics related to the requests and responses flowing through a microgateway instance.

treqErrors - The number of target requests with errors.
treqErrors - The number of target responses with errors.
statusCodes - An object containing response code counts:

{
  1: number of target responses with 1xx response codes
  2: number of target responses with 2xx response codes
  3: number of target responses with 3xx response codes
  4: number of target responses with 4xx response codes
  5: number of target responses with 5xx response codes
  }

requests - The total number of requests.
responses - The total number of responses.
connections - The number of active target connections.

About the next() function

All plugin methods must call next() to continue processing the next method in the series (or the plugin process will hang). In the request life cycle, the first method called is onrequest(). The next method to be called is the ondata_request() method; however, ondata_request is only called if the request includes data, as in the case, for example, of a POST request. The next method called will be onend_request(), which is called when the request processing is completed. The onerror_* functions are only called in the event of an error, and they allow you to handle the errors with custom code if you wish.

Let's say data is sent in the request, and ondata_request() is called. Notice that the function calls next() with two parameters:

next(null, data);

By convention, the first parameter is used to convey error information, which you can then handle in a subsequent function in the chain. By setting it to null, a falsy argument, we are saying there are no errors, and request processing should proceed normally. If this argument is truthy (such as an Error object), then request processing stops and request is sent to the target.

The second parameter passes the request data to the next function in the chain. If you do no additional processing, then the request data is passed unchanged to the target of the API. However, you have a chance to modify the request data within this method, and pass the modified request on to the target. For example, if the request data is XML, and the target expects JSON, then you can add code to the ondata_request() method that (a) changes the Content-Type of the request header to application/json and converts the request data to JSON using whatever means you wish (for example, you could use a Node.js xml2json converter obtained from NPM).

Let's see how that might look:

ondata_request: function(req, res, data, next) {
  debug('****** plugin ondata_request');
  var translated_data = parser.toJson(data);
  next(null, translated_data);
},

In this case, the request data (which is assumed to be XML) is converted to JSON, and the transformed data is passed via next() to the next function in the request chain, before being passed to the backend target.

Note that you could add another debug statement to print the transformed data for debugging purposes. For example:

ondata_request: function(req, res, data, next) {
  debug('****** plugin ondata_request');
  var translated_data = parser.toJson(data);
  debug('****** plugin ondata_response: translated_json: ' + translated_json);
  next(null, translated_data);
},

About plugin handler execution order

If you write plugins for Edge Microgateway, you need to understand the order in which plugin event handlers are executed.

The important point to remember is that when you specify a plugin sequence in the Edge Microgateway config file, the request handlers execute in ascending order, while the response handlers execute in descending order.

The following example is designed to help you understand this execution sequence.

1. Create three simple plugins

Consider the following plugin. All it does is print console output when its event handlers are called:

plugins/plugin-1/index.js

module.exports.init = function(config, logger, stats) {

  return {

    onrequest: function(req, res, next) {
      console.log('plugin-1: onrequest');
      next();
    },

    onend_request: function(req, res, data, next) {
      console.log('plugin-1: onend_request');
      next(null, data);
    },

    ondata_response: function(req, res, data, next) {
      console.log('plugin-1: ondata_response ' + data.length);
      next(null, data);
    },

    onend_response: function(req, res, data, next) {
      console.log('plugin-1: onend_response');
      next(null, data);
    }
  };
}

Now, consider creating two more plugins, plugin-2 and plugin-3, with the same code (except, change the console.log() statements to plugin-2 and plugin-3 respectively).

2. Review the plugin code

The exported plugin functions in <microgateway-root-dir>/plugins/plugin-1/index.js are event handlers that execute at specific times during request and response processing. For example, onrequest executes the first byte of the request headers is received. While, onend_response executes after the last byte of response data is received.

Take a look at the handler ondata_response -- it is called whenever a chunk of response data is received. The important thing to know is that response data is not necessarily received all at once. Rather, the data may be received in chunks of arbitrary length.

3. Add the plugins to the plugin sequence

Continuing with this example, we'll add the plugins to the plugin sequence in the Edge Microgateway config file (~./edgemicro/config.yaml) as follows. The sequence is important. It defines the order in which the plugin handlers execute.

  plugins:
    dir: ../plugins
    sequence:
      - plugin-1
      - plugin-2
      - plugin-3

4. Examine the debug output

Now, let's look at the output that would be produced when these plugins are called. There are a few important points to notice:

The plugin sequence the Edge Microgateway config file (~./edgemicro/config.yaml) specifies the order in which event handlers are called.
Request handlers are called in ascending order (the order in which they appear in the plugin sequence -- 1, 2, 3).
Response handlers are called in descending order -- 3, 2, 1.
The ondata_response handler is called once for each chunk of data that arrives. In this example (output shown below), two chunks are received.

Here is sample debug output produced when these three plugins are in use and a request is sent through Edge Microgateway. Just notice the order in which the handlers are called:

  plugin-1: onrequest
  plugin-2: onrequest
  plugin-3: onrequest

  plugin-1: onend_request
  plugin-2: onend_request
  plugin-3: onend_request

  plugin-3: ondata_response 931
  plugin-2: ondata_response 931
  plugin-1: ondata_response 931

  plugin-3: ondata_response 1808
  plugin-3: onend_response

  plugin-2: ondata_response 1808
  plugin-2: onend_response

  plugin-1: ondata_response 1808
  plugin-1: onend_response

Summary

Understanding the order in which plugin handlers are called is very important when you try to implement custom plugin functionality, such as accumulating and transforming request or response data.

Just remember that request handlers are executed in the order in which the plugins are specified in the Edge Microgateway config file, and response handlers are executed in the opposite order.

About using global variables in plugins

Every request to Edge Microgateway is sent to the same instance of a plugin; therefore, a second request’s state from another client will overwrite the first. The only safe place to save plugin state is by storing the state in a property on the request or response object (whose lifetime is limited to that of the request).

Rewriting target URLs in plugins

Added in: v2.3.3

You can override the default target URL in a plugin dynamically by modifying these variables in your plugin code: req.targetHostname and req.targetPath.

Added in: v2.4.x

You can also override the target endpoint port and choose between HTTP and HTTPS. Modify these variables in your plugin code: req.targetPort and req.targetSecure. To choose HTTPS, set req.targetSecure to true; for HTTP, set it to false. If you set req.targetSecure to true, see this discussion thread for more information.

A sample plugin called eurekaclient has been added to Edge Microgateway. This plugin demonstrates how to use the req.targetPort and req.targetSecure variables and illustrates how Edge Microgateway can perform dynamic endpoint lookup using Eureka as a service endpoint catalog.

Sample plugins

These plugins are provided with your Edge Microgateway installation. You can find them in the Edge Microgateway installation here:

[prefix]/lib/node_modules/edgemicro/plugins

where [prefix] is the npm prefix directory as described in "Where is Edge Microgateway installed" in Installing Edge Microgateway.

accumulate-request

This plugin accumulates data chunks from the client into an array property attached to the request object. When all the request data is received, the array is concatenated into a Buffer which is then passed to the next plugin in the sequence. This plugin should be the first plugin in the sequence so that subsequent plugins receive the accumulated request data.

module.exports.init = function(config, logger, stats) {

  function accumulate(req, data) {

    if (!req._chunks) req._chunks = [];
    req._chunks.push(data);

  }

  return {

    ondata_request: function(req, res, data, next) {

      if (data && data.length > 0) accumulate(req, data);

      next(null, null);

    },


    onend_request: function(req, res, data, next) {

      if (data && data.length > 0) accumulate(req, data);

      var content = null;

      if (req._chunks && req._chunks.length) {

        content = Buffer.concat(req._chunks);

      }

      delete req._chunks;

      next(null, content);

    }

  };

}

accumulate-response

This plugin accumulates data chunks from the target into an array property attached to the response object. When all the response data is received, the array is concatenated into a Buffer which is then passed to the next plugin in the sequence. Because this plugin operates on responses, which are processed in reverse order, you should place position it as the last plugin in the sequence.

module.exports.init = function(config, logger, stats) {

  function accumulate(res, data) {
    if (!res._chunks) res._chunks = [];
    res._chunks.push(data);
  }

  return {

    ondata_response: function(req, res, data, next) {
      if (data && data.length > 0) accumulate(res, data);
      next(null, null);
    },

    onend_response: function(req, res, data, next) {
      if (data && data.length > 0) accumulate(res, data);
      var content = Buffer.concat(res._chunks);
      delete res._chunks;
      next(null, content);
    }

  };

}

header-uppercase plugin

Edge Microgateway distributions include a sample plugin called <microgateway-root-dir>/plugins/header-uppercase. The sample includes comments describing each of the function handlers. This sample does some simple data transformation of the target response and adds custom headers to the client request and target response.

Here's the source code for <microgateway-root-dir>/plugins/header-uppercase/index.js:

'use strict';

var debug = require('debug')('plugin:header-uppercase');

// required
module.exports.init = function(config, logger, stats) {

  var counter = 0;

  return {

    // indicates start of client request
    // request headers, url, query params, method should be available at this time
    // request processing stops (and a target request is not initiated) if
    // next is called with a truthy first argument (an instance of Error, for example)
    onrequest: function(req, res, next) {
      debug('plugin onrequest');
      req.headers['x-foo-request-id'] = counter++;
      req.headers['x-foo-request-start'] = Date.now();
      next();
    },

    // indicates start of target response
    // response headers and status code should be available at this time
    onresponse: function(req, res, next) {
      debug('plugin onresponse');
      res.setHeader('x-foo-response-id', req.headers['x-foo-request-id']);
      res.setHeader('x-foo-response-time', Date.now() - req.headers['x-foo-request-start']);
      next();
    },

    // chunk of request body data received from client
    // should return (potentially) transformed data for next plugin in chain
    // the returned value from the last plugin in the chain is written to the target
    ondata_request: function(req, res, data, next) {
      debug('plugin ondata_request ' + data.length);
      var transformed = data.toString().toUpperCase();
      next(null, transformed);
    },

    // chunk of response body data received from target
    // should return (potentially) transformed data for next plugin in chain
    // the returned value from the last plugin in the chain is written to the client
    ondata_response: function(req, res, data, next) {
      debug('plugin ondata_response ' + data.length);
      var transformed = data.toString().toUpperCase();
      next(null, transformed);
    },

    // indicates end of client request
    onend_request: function(req, res, data, next) {
      debug('plugin onend_request');
      next(null, data);
    },

    // indicates end of target response
    onend_response: function(req, res, data, next) {
      debug('plugin onend_response');
      next(null, data);
    },

    // error receiving client request
    onerror_request: function(req, res, err, next) {
      debug('plugin onerror_request ' + err);
      next();
    },

    // error receiving target response
    onerror_response: function(req, res, err, next) {
      debug('plugin onerror_response ' + err);
      next();
    },

    // indicates client connection closed
    onclose_request: function(req, res, next) {
      debug('plugin onclose_request');
      next();
    },

    // indicates target connection closed
    onclose_response: function(req, res, next) {
      debug('plugin onclose_response');
      next();
    }

  };

}

transform-uppercase

This is a general transformation plugin that you can modify to do whatever kind of transformation you wish. This example simply transforms the response and request data to uppercase.

 */
module.exports.init = function(config, logger, stats) {

  // perform content transformation here
  // the result of the transformation must be another Buffer
  function transform(data) {
    return new Buffer(data.toString().toUpperCase());
  }

  return {

    ondata_response: function(req, res, data, next) {
      // transform each chunk as it is received
      next(null, data ? transform(data) : null);
    },

    onend_response: function(req, res, data, next) {
      // transform accumulated data, if any
      next(null, data ? transform(data) : null);
    },

    ondata_request: function(req, res, data, next) {
      // transform each chunk as it is received
      next(null, data ? transform(data) : null);
    },

    onend_request: function(req, res, data, next) {
      // transform accumulated data, if any
      next(null, data ? transform(data) : null);
    }

  };

}