As a Solutions Engineer here at Kong, one question that frequently comes across my desk is “how can I transform a Kong logging plugin message into a format that my `insert-observability-stack-here` understands, (e.g., ELK, Loki, Splunk, etc.)?”
In this blog, I’m going to show you how to easily accomplish converting a Kong logging payload to the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html)Elastic Common Schema.
In order to accomplish this task, we’re going to be running Kong Gateway in Kubernetes and using two Kong plugins.
If you don’t already have an instance of Kong running in a Kubernetes cluster, connect to your cluster and run the following the commands to get one in seconds.
% kubectl create ns kong
% kubectl apply -f https://bit.ly/kong-ingress-dbless
% kubectl get po -n kong -w
NAME READY STATUS RESTARTS AGE
ingress-kong-7c4b795d5d-f2lpt 0/2 ContainerCreating 0 1s
ingress-kong-7c4b795d5d-f2lpt 0/2 Running 0 1s
ingress-kong-7c4b795d5d-f2lpt 1/2 Running 0 10s
ingress-kong-7c4b795d5d-f2lpt 2/2 Running 0 20s
^C
See [how to install Kong in Kubernetes](https://docs.konghq.com/gateway/latest/install/kubernetes/kubectl/)how to install Kong in Kubernetes for more information. Once you have an available instance of the Kong Gateway, continue.
First, create an empty Kubernetes manifest file called, `elastic-common-schema.yaml`.
Next, let’s define our KongPlugin resources. The first plugin we will create is the serverless pre-function. From the Kong plugin docs, a serverless pre-function plugin:
*Runs before other plugins run during each phase. The pre-function plugin can be applied to individual services, routes, or globally.*
Since we’re logging, we’re concerned with the `log` phase or “context”. For more information on all available plugin contexts, read this [doc](https://docs.konghq.com/gateway/latest/plugin-development/custom-logic/#available-contexts)doc.
Paste the below yaml in your manifest.
The above resource definition creates a `KongPlugin` that executes before the logging phase of each plugin defined in scope. The `kong.ctx.shared.mystuff=kong.log.serialize()` is a single line of Lua code that stores the logging payload into a shared context. From the Kong docs, a shared context is:
*A [Lua] table that has the same lifetime as the current request. This table is shared between all plugins. It can be used to share data between several plugins in a given request.*
For more info on shared contexts, see this [doc](https://docs.konghq.com/gateway/latest/plugin-development/pdk/kong.ctx/#kongctxshared)doc.
The key to doing the transformation is the `custom_fields_by_lua` configuration. From the Kong docs, the `custom_fields_by_lua` is:
*A list of key-value pairs, where the key is the name of a log field and the value is a chunk of Lua code, whose return value sets or replaces the log field value.*
% kubectl create ns myblog
% kubectl apply -f https://bit.ly/k8s-httpbin -n myblog
% kubectl get po -n myblog -w
NAME READY STATUS RESTARTS AGE
httpbin-64cdb8c89c-7rxm2 1/1 Running 0 5s
^C
% kubectl logs ingress-kong-7c4b795d5d-pg2c6 -n kong -c proxy -f | grep "@timestamp"
# You should see something similar to the below
{"@timestamp":"1667427319","url":{"original":"/testing/anything"},"http":{"response":{"status_code":200},"request":{"body":{"bytes":93}}},"client_ip":"10.48.0.1"}
{
"latencies": {
"request": 515,
"kong": 58,
"proxy": 457
},
"service": {
"host": "httpbin.org",
"created_at": 1614232642,
"connect_timeout": 60000,
"id": "167290ee-c682-4ebf-bdea-e49a3ac5e260",
"protocol": "http",
"read_timeout": 60000,
"port": 80,
"path": "/anything",
"updated_at": 1614232642,
"write_timeout": 60000,
"retries": 5,
"ws_id": "54baa5a9-23d6-41e0-9c9a-02434b010b25"
},
"request": {
"querystring": {},
"size": 138,
"uri": "/log",
"url": "http://localhost:8000/log",
"headers": {
"host": "localhost:8000",
"accept-encoding": "gzip, deflate",
"user-agent": "HTTPie/2.4.0",
"accept": "*/*",
"connection": "keep-alive"
},
"method": "GET"
},
"tries": [
{
"balancer_latency": 0,
"port": 80,
"balancer_start": 1614232668399,
"ip": "18.211.130.98"
}
],
"client_ip": "192.168.144.1",
"workspace": "54baa5a9-23d6-41e0-9c9a-02434b010b25",
"upstream_uri": "/anything",
"response": {
"headers": {
"content-type": "application/json",
"date": "Thu, 25 Feb 2021 05:57:48 GMT",
"connection": "close",
"access-control-allow-credentials": "true",
"content-length": "503",
"server": "gunicorn/19.9.0",
"via": "kong/2.2.1.0-enterprise-edition",
"x-kong-proxy-latency": "57",
"x-kong-upstream-latency": "457",
"access-control-allow-origin": "*"
},
"status": 200,
"size": 827
},
"route": {
"id": "78f79740-c410-4fd9-a998-d0a60a99dc9b",
"paths": [
"/log"
],
"protocols": [
"http"
],
"strip_path": true,
"created_at": 1614232648,
"ws_id": "54baa5a9-23d6-41e0-9c9a-02434b010b25",
"request_buffering": true,
"updated_at": 1614232648,
"preserve_host": false,
"regex_priority": 0,
"response_buffering": true,
"https_redirect_status_code": 426,
"path_handling": "v0",
"service": {
"id": "167290ee-c682-4ebf-bdea-e49a3ac5e260"
}
},
"started_at": 1614232668342
}
*which gets transformed into:*
{
"@timestamp": "1667427319",
"url": {
"original": "/testing/anything"
},
"http": {
"response": {
"status_code": 200
},
"request": {
"body": {
"bytes": 93
}
}
},
"client_ip": "10.48.0.1"
}Congratulations, you have transformed a Kong log payload into an Elastic Common Schema format ready for ingestion! This pattern can be used to easily transform Kong logging messages into any format for ingestion with any observability stack.
Multi-agent systems are, at their core, context distribution systems. Every agent in your workflow is a consumer and producer of context. The interesting architectural questions are all about how that context moves. Two operations drive everything:
You wouldn't expose a REST API without a gateway. What about Kafka? Before we talk about an event gateway, it's worth grounding ourselves on what an event is. An event is simply a record of something that happened. In modern integration, there are
An AI control layer is the governance and observability infrastructure that sits between AI agents and enterprise applications, handling authentication, routing, rate limiting, and auditability to ensure secure, managed access. Unlike traditional in
When an AI application fails, knowing that an individual model call returned a 200 isn’t enough. You need to know what led to it. Kong stitches the interactions associated with an AI session together, giving developers a view across the sequence of
We're thrilled to announce that Kong, the AI connectivity company, has been recognized as a Leader in the Gartner Magic Quadrant for API Management for the seventh year in a row. Plus, Kong was positioned furthest on the Completeness of Vision axis
If you've run Kong at scale, you know the moment: a spike in 5xxs from one specific data plane node, a customer opening a ticket about intermittent failures, or a partner integration that fails calls no one else does. Debug-level logs would tell you
