Event hooks are a brand new feature we launched with Kong Gateway 2.5 that allows you to get notifications when certain events happen on your Kong Gateway deployment. If you want to keep an eye out for when your system creates new administrators or adds new plugins to a latency-sensitive route, this is the feature for you!

Event hooks is a Kong Gateway Enterprise feature. Interested in learning more? Contact our sales team.

How Do Event Hooks Work?

There are four types of actions you can use when an event hook runs:

1. webhook: Send a preformatted POST request to a system of your choosing.
2. webhook-custom: Build a custom HTTP request using any verb, headers and payload you need.
3. log: Log the event and payload in the Kong Gateway logs. This is useful for conditional logging.
4. lambda: Trigger a Lua function that you write and upload to the Kong Gateway runtime.

The data available is identical in all cases, so your use depends on your specific use case.

My Lua skills aren't particularly great, so I'm sticking to the webhook, webhook-custom and log handlers in the examples later in this post. Look out for a future post on using the lambda handler to write custom Lua functions to respond to event hooks!

Configuring Your Event Listeners

As the available events may change with each release, Kong Gateway offers the /event-hooks/sources endpoint that you can call to see all available sources, events and fields that are available for templating.

This endpoint returns a lot of data, so I'd recommend using a tool such as Insomnia, which allows you to collapse fields that you're not interested in to explore the response.

To learn more about the available endpoints, see the Kong documentation, the canonical reference for all available endpoints.

Examples

Health Check Failures (webhook-custom)

The first example I'd like to show is how you can get notifications any time an upstream in your application is not reachable. This usually happens when your application is not running, and Kong tries to forward a request.

Looking at the /event-hooks/sources endpoint, I can see the following data in the response:

{
  "data": {
    "balancer": {
      "health": {
        "fields": [
          "upstream_id",
          "ip",
          "port",
          "hostname",
          "health"
        ]
      },
      ....
    }
  }
}

Based on the documentation, we know that the source for this action is balancer, the event is health and that we have five fields available for use in webhook-custom payloads. Let's use them now to build a Slack notification when an upstream is offline.

From the documentation:

• 1st level = The source, which is the action that triggers the event hook

• 2nd level = The event, which is the Kong entity the event hook will listen to for events

• 3rd level = The available template parameters for use in webhook-custom payloads

That gives us the initial configuration to use when creating our event hook:

{
    "event": "health",
    "source": "balancer",
    "handler": "webhook-custom"
}

It also tells us which fields are available for use in the payload:

• upstream_id
• port
• ip
• host
• healthy

When sending an incoming webhook, Slack expects a POST request with a JSON body. That's why we tell Kong to send a JSON post using the config parameter. At the same time, we provide the payload to send (using the available fields) and configure our Slack webhook URL:

{
  "event": "health",
  "source": "balancer",
  "handler": "webhook-custom",
  "config": {
    "method": "POST",
    "headers": {
      "Content-Type": "application/json"
    },
    "payload": {
      "text": "Upstream {{upstream_id}} is {{health}}. Tried to connect to port {{port}} on {{hostname}}/{{ip}}"
    },
    "url": "https://hooks.slack.com/services/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"
  }
}

If you create this webhook then stop your upstream service, you should get a Slack message that looks like the following:

> Upstream 3d2aaac6-0400-401d-aac0-3f6a1bd3b6b7 is unhealthy. Tried to connect to port 3000 on web1.mycompany.net/10.16.0.2.

This works, but I wanted to take it one step further. Slack has the concept of Blocks, which allow you to build complex layouts for your messages. Using their Block Kit designer, I created a layout that shows the same data in a grid:

To create this layout, you need to send a blocks key in the payload. Kong expects a string in that field, so we need to encode it as JSON before sending the API request like so:

{
  "event": "health",
  "source": "balancer",
  "handler": "webhook-custom",
  "config": {
    "headers": {
      "Content-Type": "application/json"
    },
    "payload": {
      "text": "Upstream {{upstream_id}} is {{health}}. Tried to connect to port {{port}} on {{hostname}} or {{ip}}",
      "blocks": "[ { \"type\": \"header\", \"text\": { \"type\": \"plain_text\", \"text\": \"Upstream is {{health}}\", \"emoji\": true } }, { \"type\": \"section\", \"fields\": [ { \"type\": \"mrkdwn\", \"text\": \"*Upstream:*\\n{{upstream_id}}\" }, { \"type\": \"mrkdwn\", \"text\": \"*Port:*\\n{{port}}\" } ] }, { \"type\": \"section\", \"fields\": [ { \"type\": \"mrkdwn\", \"text\": \"*Hostname:*\\n{{hostname}}\" }, { \"type\": \"mrkdwn\", \"text\": \"*IP:*\\n{{ip}}\" } ] } ]"
    },
    "url": "https://hooks.slack.com/services/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",
    "method": "POST"
  }
}

Once that's been submitted, you'll start to receive formatted notifications whenever an upstream becomes healthy/unhealthy.

Audit Logging (log)

Next up, we want to keep an eye on the new admins the system is creating. We could send a webhook like the last example, but this time I'm going to use the log option to write the event to the Kong error log since the SRE team is already monitoring them.

Like before, we look at the /event-hooks/sources endpoint to determine which source and event to use for our configuration. This time, it's crud and admins:create, which gives us the following configuration:

{
  "event": "crud",
  "source": "admins:create",
  "handler": "log"
}

Much shorter than our webhook-custom config, but that's really all there is to it! If we POST that to /event-hooks, the system will log the data for inspection the next time you create an admin.

Here's a sample event:

[kong] event_hooks.lua:?:452 "log callback: " { "admins:create", "crud", {
  entity = {
    consumer = {
        id = "bdebb22d-eda0-4e12-b05b-06c4e35a0ca6"
    },
    created_at = 1630675686,
    email = "admin@example.com",
    id = "5e1ed9c4-d981-41c2-88a8-4237a80aedc0",
    rbac_token_enabled = true,
    rbac_user = {
      id = "8d8ddf7f-a0d6-4d38-91d5-c78751253cc9"
    },
    status = 4,
    updated_at = 1630675686,
    username = "hook_test_admin"
  }, 
  operation = "create",
  schema = "admins"
}

Policy Enforcement (webhook)

Taking the previous example, let's extend it to log when the system creates an admin and enforce that every admin created has a @konghq.com email address. If they have a different email, we'll delete their account immediately.

This time, we'll use the webhook handler as we're building a small application to handle the logic, so we don't need the custom templating features. Just like before, we provide the source and event, plus the URL to send the payload to:

{
  "source": "crud",
  "event": "admins:create",
  "handler": "webhook",
  "config": {
    "url": "http://<your host>/admin-created"
  }
}

I'm not going to share the whole application in this blog post as it's a little long, but it is available on GitHub if you'd like to see it.

Here are the key steps:

• Receive the event hook from Kong
• Check if the entity.email in the body ends with our required domain (konghq.com in this case)
  
  • If it does, the endpoint returns here and takes no further action
• Use the Kong Admin API to delete the admin account that we just created using our Kong Admin user and token
• Tell the caller that an admin was deleted

If you're not sure what information is in the payload, I'd recommend using the log event hook or outputting the whole request body into a log file so that you can inspect which fields are available.

We've Only Just Begun

This is already quite a long post, and we've barely scratched the surface of what you can do with event hooks! We built them with developer experience in mind, which means there's a snooze parameter to silence events within a specific window. If you're using the webhook transport, you'll also want to enable x-kong-signature, which will allow you to validate the HMAC signature of the request and verify that it came from Kong.

There's a ton you can do with event hooks, and I'd love to hear all about what you build with them. I'm @mheap on Twitter, or you can email me at michael.heap@konghq.com with your ideas.