Kubernetes integration tasks

Adding a custom plugin

You can add new features and capabilities to the microgateway by writing custom plugins. Custom plugins let you interact programmatically with the requests and responses that flow through the microgateway.

This section explains how to package and deploy plugins to an Edge Microgateway instance running in your Kubernetes cluster. It explains how to add custom plugins for both standalone service and sidecar deployments

The rest of this section assumes that you are familiar with writing and configuring plugins for a standard Edge Microgateway setup. If not, see Develop custom plugins.

Package your plugin(s)

To package custom plugin(s), follow these steps:

  1. Write and test your plugin, according to the directions in Write a simple plugin.

  2. Place your plugin code in the proper directory structure. Plugin directories must follow a set structure. The following example shows the structure you must follow, where response-uppercase and request-headers are the names of folders containing custom plugin code (these names are examples only, your folder names may differ):

    plugin
      |
      |-- plugins
        |
        |- response-uppercase
        |     |- index.js
        |     |- package.json
        |- request-headers
        |     | - index.js
              | - package.json
    
  3. cd to the plugin folder.

  4. In the plugin folder, zip the entire plugins folder:

    zip -r plugins.zip plugins/

Create a Docker image

  1. In the same directory where the zip file is located, create a new file called Dockerfile.
  2. Add the following code to Dockerfile and save the file:

    FROM gcr.io/apigee-microgateway/edgemicro:latest
    RUN apt-get install unzip
    COPY plugins.zip /opt/apigee/
    RUN chown apigee:apigee /opt/apigee/plugins.zip
    RUN su - apigee -c "unzip /opt/apigee/plugins.zip -d /opt/apigee"
    EXPOSE 8000
    EXPOSE 8443
    ENTRYPOINT ["entrypoint"]
    
  3. Create a new Edge Microgateway Docker image with your plugins and push the image to your docker registry. You can use whichever registry you wish, such as docker.io or gcr.io:

    docker build -t edgemicroplugins .
    docker tag edgemicroplugins container-registry/your-project/edgemicroplugins
    docker push container-registry/your-project/edgemicroplugins
    

    For example:

    docker build -t edgemicroplugins .
    docker tag edgemicroplugins gcr.io/my-project/edgemicroplugins
    docker push   gcr.io/my-project/edgemicroplugins
    

Update the Edge Microgateway configuration

  1. Add the plugin(s) to your Edge Microgateway configuration file. You can find the config file here:

    $HOME/.edgemicro/org-env-config.yaml
    

    For example:

    $HOME/.edgemicro/myorg-test-config.yaml

In the following sample config, the custom plugin response-uppercase was added. The oauth plugin was already there by default.

  edgemicro:
    ...
    plugins:
      sequence:
        - oauth
        - response-uppercase

Update your Kubernetes cluster

The final step is to apply the configuration change to your Kubernetes cluster. Kubernetes will pull the new image with the plugin code that you pushed to the container registry and use it for any newly created pods. Follow the steps below for either a service or sidecar deployment, depending on your scenario.

If you deployed Edge Microgateway as a service

Use the edgemicroctl command to inject the updated Edge Microgateway configuration:

  1. Update the Edge Microgateway deployment with the new image. For example:

    kubectl apply -f <(edgemicroctl -org=your_organization -env=your_environment -key=configuration_key -sec=configuration_secret -conf=config_file_path -img=container-registry/your_project_name/image_name:latest)

    where:

    • your_organization - The Apigee organization you specified in the edgemicro configure command.
    • your_environment - The environment you specified in the edgemicro configure command.
    • configuration_key - The key returned from the edgemicro configure command.
    • configuration_secret - The secret returned from the edgemicro configure command.
    • config_file_path - The path to the Edge Micro configuration file returned from the edgemicro configure command.
    • container-registry - The Docker registry where you pushed the image. For example, gcr.io or docker.io.
    • your_project_name - The project name for the Docker repository where you pushed the Docker image.
    • image_name - The name of the Docker image you pushed.

    Example:

    kubectl apply -f <(edgemicroctl -org=jdoe -env=test -key=f2d2eaa52b758493d00cec656e574ac947bee1d701c5c5f3295e5eaa39a3b -sec=0c38cda3fac6c59152f15657052ba1728f8003c1a763cf08da2a -conf=/Users/jdoe/.edgemicro/apigeesearch-test-config.yaml -img=gcr.io/jdoe-project/edgemicroplugins:latest)
  2. Test the plugin. Call the API to see if you get the expected behavior. For example, for "response uppercase" plugin, the response text is converted to all uppercase, as shown below:

    curl $GATEWAY_IP -H 'x-api-key:3eqeedJRFLlCshwWBiXq4xKFoH1Se3xR'

    Output:

    HELLO WORLD
    

If you deployed Edge Microgateway as a sidecar

If you deployed Edge Microgateway as a sidecar, you have two options for updating the configuration. You can update the automatic sidecar injector configuration and apply it to the cluster, or you can manually inject the sidecar.

Manually injecting the new configuration

Manual injection is a straightforward approach, where you inject the new configuration from the command line.

  1. Execute the following command:

    kubectl apply -f <(edgemicroctl -org=your_org -env=your_env -key=your_key -sec=your_secret -conf=config_file_path -img=container-registry/your_project_name/image_name:latest -svc=service_deployment_file)

    where:

    • your_org - The Apigee organization you specified in the edgemicro configure command.
    • your_env - The environment you specified in the edgemicro configure command.
    • your_key - The key returned from the edgemicro configure command.
    • your_secret - The secret returned from the edgemicro configure command.
    • config_file_path - The path to the Edge Micro configuration file returned from the edgemicro configure command.
    • container-registry - The Docker registry where you pushed the image. For example, gcr.io or docker.io.
    • your_project_name - The project name for the Docker repository where you pushed the Docker image.
    • image_name - The name of the Docker image you pushed.
    • service_deployment_file - The path to the deployment file of the service for which the plugin(s) will apply. For example: samples/helloworld/helloworld.yaml.

    For example:

    kubectl apply -f <(edgemicroctl -org=myorg -env=test-key=0e3ecea28a64099410594406b30e54439af5265f8 -sec=e3919250bee37c69cb2e5b41170b488e1c1d -conf=/Users/jdoe/.edgemicro/myorg-test-config.yaml -img=gcr.io/myproject/edgemicroplugins:latest -svc=samples/helloworld/helloworld.yaml)
  2. Test the plugin. Call the service API to see if you get the expected behavior. For example, for "response uppercase" plugin, the response text is converted to all uppercase, as shown below:

    curl $GATEWAY_IP -H 'x-api-key:3eqeedJRFLlCshwWBiXq4xKFoH1Se3xR'

    Output:

    HELLO WORLD
    
Updating an automatic sidecar configuration

If you update the automatic sidecar injection configuration, any time a new pod starts, it will get the new Edge Microgateway configuration.

  1. Edit the file install/kubernetes/edgemicro-sidecar-injector-configmap-release.yaml and change the containers image to the path for the image you pushed to the docker repository.

    containers:
      - name: edge-microgateway
        image: gcr.io/your-project/edgemicroplugins:latest
    
  2. Apply the changes to the cluster:

    kubectl apply -f  install/kubernetes/edgemicro-sidecar-injector-configmap-release.yaml
  3. Go to the section Making configuration changes to complete the update.

Making Edge Microgateway configuration changes

In some cases, you may need to modify your Edge Microgateway configuration. For example, you might want to add a new plugin to Edge Microgateway or change a configuration parameter. This section explains how to make and apply configuration changes to Edge Microgateway running in Kubernetes.

  1. Create a configuration file secret.yaml as shown below:

    apiVersion: v1
    kind: Secret
    metadata:
      name: mgwsecret
    type: Opaque
    data:
      mgorg: EDGEMICRO_ORG
      mgenv: EDGEMICRO_ENV
      mgkey: EDGEMICRO_KEY
      mgsecret: EDGEMICRO_SECRET
      mgconfig: EDGEMICRO_CONFIG
    
  2. Specify the base64 encoded value of EDGEMICRO_ORG, EDGEMICRO_ENV, EDGEMICRO_KEY, EDGEMICRO_SECRET:

    echo -n "your-org" | base64 | tr -d '\n'
    echo -n "your-org-env" | base64 | tr -d '\n'
    echo -n "your-mg-key" | base64 | tr -d '\n'
    echo -n "your-mg-secret" | base64 | tr -d '\n'
    
  3. Make your changes to the Edge Microgateway configuration file for your organization and environment:

    $HOME/.edgemicro/your_org-your_env-config.yaml
  4. Base64 Encode twice the contents of config file:

    cat $HOME/.edgemicro/org-env-config.yaml | base64 | tr -d '\n' | base64  | tr -d '\n'
  5. Apply changes to kubernetes on the namespace where your service is running.

    kubectl apply -f secret.yaml -n 

These new changes are not picked up automatically by existing microgateway pods; however, the new pods will get the changes. You can delete the existing pod so that deployment creates a new pod that picks up the change.

Service example

The following example illustrates how to update a service deployment with a new

  1. Get the pods.

    kubectl get pods

    Example output:

    NAME                                 READY     STATUS    RESTARTS   AGE
    edge-microgateway-57ccc7776b-g7nrg   1/1       Running   0          19h
    helloworld-6987878fc4-cltc2          1/1       Running   0          1d
    
  2. Delete the edge-microgateway pod.

    kubectl delete pod edge-microgateway-57ccc7776b-g7nrg

    Example output:

    pod "edge-microgateway-57ccc7776b-g7nrg" deleted
    
  3. Get the pods again. A new pod spins up and gets your configuration changes.

    kubectl get pods

    Example output:

    NAME                                 READY     STATUS    RESTARTS   AGE
    edge-microgateway-57ccc7776b-7f6tc   1/1       Running   0          5s
    helloworld-6987878fc4-cltc2          1/1       Running   0          1d
    

Sidecar example

The following example illustrates how to update a sidecar deployment with a new configuration.

  1. Get the pods.

    kubectl get pods

    Example output:

    NAME                          READY     STATUS    RESTARTS   AGE
    helloworld-7d5f5b6769-vcq6m   2/2       Running   0          32m
    
  2. Delete the helloworld pod.

    kubectl delete pod helloworld-7d5f5b6769-vcq6m

    Example output:

    pod "helloworld-7d5f5b6769-vcq6m" deleted
    
  3. Get the pods again. A new pod spins up and gets your configuration changes.

    kubectl get pods

    Example output:

    NAME                          READY     STATUS        RESTARTS   AGE
    helloworld-7d5f5b6769-cr4z5   2/2       Running       0          5s
    helloworld-7d5f5b6769-vcq6m   0/2       Terminating   0          32m
    

Scaling your deployment

If you have deployed Edge Microgateway as sidecar, by default it comes with one replica. This section explains how you can use Kubernetes scaling principles to scale your deployments.

Scaling a sidecar deployment

  1. Check the deployments:

    kubectl get deployments

    Example output:

    NAME         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    helloworld   1         1         1            1           3m
    

    The output indicates one replica is deployed.

  2. Get the pods:

    kubectl get pods

    Example output:

    NAME                          READY     STATUS    RESTARTS   AGE
    helloworld-6987878fc4-gz74k   2/2       Running   0          2d
    
  3. Scale the deployment from 1 to as many replicas as you wish. In this example, the helloworld service is scaled:

    kubectl scale deployment helloworld --replicas=2
  4. (Optional) If you want to use autoscaling, use this command:

    kubectl autoscale deployment helloworld --cpu-percent=50 --min=1 --max=10
  5. Check the deployments to verify scaling is enabled:

    kubectl get deployments

    Example output:

    NAME         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    helloworld   2         2         2            1           5m
    

    The state has been changed to include two replicas.

  6. Check the pods:

    kubectl get pods

    Example output:

    NAME                          READY     STATUS    RESTARTS   AGE
    helloworld-7d5f5b6769-nm7zd   2/2       Running   0          30s
    helloworld-7d5f5b6769-vcq6m   2/2       Running   0          6m
    

    The output shows both replicas are in a running state.

Scaling a service deployment

  1. Check the deployments:

    kubectl get deployments

    Example output:

    NAME                DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    edge-microgateway   1         1         1            1           18h
    helloworld          1         1         1            1           1d
    

    The output indicates one replica is deployed.

  2. Scale the deployment from 1 to as many replicas as you wish. In this example, the edge-microgateway service is scaled.

    kubectl scale deployment edge-microgateway --replicas=2
  3. (Optional) If you want to use autoscaling, use this command:

    kubectl autoscale deployment edge-microgateway --cpu-percent=50 --min=1 --max=10
  4. Check the deployments to verify scaling is enabled:

    kubectl get deployments

    Example output:

    NAME                DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    edge-microgateway   2         2         2            2           18h
    helloworld          1         1         1            1           1d
    

    The state has been changed to include two replicas.

  5. Check the pods:

    kubectl get pods

    Example output:

    NAME                                 READY     STATUS    RESTARTS   AGE
    edge-microgateway-57ccc7776b-g7nrg   1/1       Running   0          18h
    edge-microgateway-57ccc7776b-rvfz4   1/1       Running   0          41s
    helloworld-6987878fc4-cltc2          1/1       Running   0          1d
    

    The output shows both replicas are in a running state.

Use namespaces for multiple Edge Microgateway configurations

You can deploy and configure multiple instances of Edge Microgateway services to your Kubernetes cluster. This use case allows you to configure each microgateway instance with its own set of plugins and parameters. For example:

  • Edge Microgateway Service A requires only the spike arrest plugin.
  • Edge Microgateway Service B requires the quota and oauth plugin, but not spike arrest.

To address this use case, use Kubernetes namespaces. For example, you can deploy Edge Microgateway Service A to namespace foo, and Edge Microgateway Service B to namespace bar.

In the following example, Edge Microgateway configured in organization OrgA is deployed as a service to namespace foo using the -n option:

kubectl apply -f <(edgemicroctl -org=myorgA -env=test-key=0e3ecea28a64099410594406b30e54439af5265f8 -sec=e3919250bee37c69cb2e5b41170b488e1c1d -conf=/Users/joed/.edgemicro/orgA-test-config.yaml -svc=samples/helloworld/helloworld.yaml) -n foo

Similarly, in the following example, Edge Microgateway configured in organization OrgB is deployed as a service to namespace bar using the -n option:

kubectl apply -f <(edgemicroctl -org=myorgB -env=test-key=0e3ecea28a64099410594406b30e54439af5265f8 -sec=e3919250bee37c69cb2e5b41170b488e1c1d -conf=/Users/joed/.edgemicro/orgB-test-config.yaml -svc=samples/helloworld/helloworld.yaml) -n bar

You can follow the exact same pattern to deploy Edge Microgateway as a sidecar using the manual injection technique.

Specifying the container ports for sidecar deployments

This section explains what to do if you run an Edge Microgateway container on a separate port than the service to which the microgateway proxies.

In a sidecar deployment, Edge Microgateway and a service run in the same pod, but in separate containers. Edge Microgateway serves as a local proxy to the service; therefore, it must know the container port on which it can create the local proxy.

The Edge Microgateway container is aware of the service port, but not the container port. If your container port is the same as the service port, the microgateway creates the local proxy on that port. The setup will work as expected.

In cases where you run your container on a separate port than the service port, Edge Microgateway needs to be aware of the container port. In such cases, Edge Microgateway looks for a containerPort label in your deployment template metadata and creates the local connection on that port.

In the following example, the container port is set to 8082:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: httpbin-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: httpbin-app
        containerPort: "8082"
    spec:
      containers:
      - name: httpbin
        image: gcr.io/apigee-microgateway/httpbin:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8082

Uninstalling Edge Microgateway deployments from Kubernetes

To disable automatic injection:

kubectl label namespace default edgemicro-injection-

To uninstall Edge Microgateway with sidecar injection:

kubectl delete -f samples/helloworld/helloworld.yaml --namespace=default
kubectl delete -f install/kubernetes/edgemicro-sidecar-injector-with-ca-bundle.yaml
kubectl -n edgemicro-system delete secret sidecar-injector-certs
kubectl delete csr edgemicro-sidecar-injector.edgemicro-system
kubectl label namespace default edgemicro-injection-
kubectl delete -f install/kubernetes/edgemicro-config-namespace-bundle.yaml
kubectl delete -f install/kubernetes/edgemicro-sidecar-injector-configmap-release.yaml

rm -fr  install/kubernetes/edgemicro-sidecar-injector-with-ca-bundle.yaml
rm -fr  install/kubernetes/config/*config.yaml
rm -fr  install/kubernetes/edgemicro-config-namespace-bundle.yaml

kubectl delete -f install/kubernetes/edgemicro-nginx-gke.yaml
kubectl delete -f install/kubernetes/edgemicro.yaml