Kong Gateway Enterprise 3.11 Makes APIs & Event Streams More Powerful
Update Includes Data Orchestration, CyberArk Support, Solace Integration, and Kafka Schema Validation
We’re excited to bring you Kong Gateway Enterprise 3.11 with compelling new features to make your APIs and event streams even more powerful, including:
- Support for advanced in-line data orchestration with the new Datakit plugin
- New vault type with CyberArk Secrets Manager
- New Event Gateway features: Solace message producer and validate schema before publish
We’ll also touch on what’s new with Konnect networking and Active Tracing. There’s a lot to unpack, so keep on reading for the full story!
Advanced data orchestration with Datakit
Sometimes you need just a little bit more flexibility to handle certain API request flows or transformations that aren’t handled by our extensive suite of plugins out of the box. You could augment Kong Gateway by building a custom plugin with our plugin development kit (PDK). Custom plugins are a powerful capability, but require developer time to build and maintain.
With Gateway 3.11, we’re giving you a native way to solve for many of these advanced data orchestration challenges with the Datakit plugin. Datakit was released as a tech preview in Gateway 3.9 back in December and is now generally available.
Perhaps you need to retrieve or validate an access token using an internal authentication or licensing service? Maybe you have a data orchestration use case where you need to combine payloads from multiple upstream APIs? Do you need to dynamically construct an upstream URL? These are all use cases Datakit was designed to tackle out of the box.
With Datakit, you combine different node types to help you accomplish the task at hand. So what do these nodes look like?
- call: Callout to other APIs to get additional data or context
- jq: Use the expressive jquery syntax to extract and transform request and response data, from both upstream APIs and API call nodes. Can also be used to format strings.
- static: Define arbitrary static variables that can be used for inputs to other nodes
- property: Get and set Kong Gateway configuration parameters
- exit: better control of when and how your API call returns and response codes
Datakit can be configured like any other plugin in Kong: through the UI, Admin API, or declarative config with decK or Terraform. With tight control over the order of operations and dependencies, you can now orchestrate complex workflows that may require data from a previous callout to fulfill subsequent callouts. We’ve applied our usual Kong engineering obsession on throughput and performance — but if you hit a snag, you can rely on Active Tracing in Konnect to quickly surface the root cause.
You can expect further improvements and a low-code UI to make it even easier to build and test Datakit flows in subsequent releases.
Please note that with this release, we've removed the previous implementation’s dependency on WASM. As of 3.11, we're additionally removing WASM support (Beta) from the Gateway as the new implementation addresses the same use cases and is considerably more performant and memory efficient.
CyberArk Secrets Manager
Kong already supports a range of secret vaults including HashiCorp, cloud service provider vaults, and our own built-in vault. Support for CyberArk Secrets Manager (formerly known as Conjur Cloud Secrets Manager) was a highly requested feature. Wherever you can dynamically retrieve a configuration value or secret from a vault in Kong, you can now use CyberArk Secrets manager as the target vault — SaaS, enterprise self-hosted, and open source versions.
Support legacy applications with Sticky Sessions
While typically our customers route traffic to workloads that follow the principles of the 12-factor application, sometimes we need to route traffic to applications that store a lot of user-specific context in memory. This might be a legacy application that was only ever designed to run as a single process – storing session context only in memory. Or it might be a long-lived computational task that you need to query the state on. In either scenario it is useful to route traffic to a specific upstream target based on the value of a session cookie.
To support these kinds of use cases, we’re delighted to announce support for sticky-sessions as a new upstream load balancer algorithm. When a request is proxied through an Upstream using the sticky-sessions algorithm, Kong Gateway sets a cookie on the response (via the Set-Cookie header). On subsequent requests, if the cookie is still valid and the original Target is available, traffic is routed to that same Target.
Kong Event Gateway Advances with Solace Integration and Kafka Schema Validation
Back in May, we announced Kong Event Gateway, significantly expanding our support for Kafka both in terms of brokering HTTP-based access to Kafka topics and our new Kong Native Event Proxy to manage Kafka-native protocol traffic. Kong Event Gateway makes your event-driven architecture more accessible, secure, and powerful — all while lowering costs. If you missed our Event Gateway announcements, read more about the new capabilities here, and find out more about why we think it is so game-changing here.
We promised to keep adding new Event Gateway capabilities, and with this release, we're building on that momentum, with some significant additions to our protocol mediation story.
Support for Solace PubSub+ messaging (BETA)
Our Event Gateway vision extends beyond Kafka, and we’re pleased to announce our support for Solace with our new Solace Upstream plugin. Solace PubSub+ is a leading event streaming and messaging platform, with fine-grained routing and event mesh capabilities.
With Kong’s Solace producer plugin, you can now publish messages to a Solace queue or topic using a standard HTTP API request, bringing new consumers into the scope of Solace’s EDA while re-using authentication, encryption, and other policies that you're already using in Kong Gateway. The plugin supports Solace’s Message VPNs, and OAuth/OIDC authentication. You can find more details in the plugin documentation.
Shift left on data quality in your event streams
Kafka has long supported adding schema metadata to messages, and the Confluent schema registry provides a widely used repository for event schemas. While Kafka passes along the schema metadata, it leaves it to the client application to actually perform the validation. With this new capability in Kong’s Kafka and Confluent Upstream plugins, Kong will validate the message before it is published to a topic. If schema validation fails, it will block the message and return an error to the Kafka client.
With this release, we're supporting the Confluent Schema registry for both AVRO and JSON message types, and we’ll add support for other schema registries in the future. Schemas in the registry are cached in the proxy for performance.
To get started, use the schema_registry parameter in your Kafka upstream or Confluent plugin configuration.
The Kafka Consume, Confluent Consume, and Kafka Log plugins can also use the schema to deserialize the payload.
When you're expanding the set of producers, perhaps even to external organizations, it’s important that all producers with access to the cluster respect the schema. Kong Event Gateway provides data owners the means to ensure adherence to the schema and block poison messages.
Fixing data quality problems at source means fewer failures and less time and pain trawling through logs and finding “needle” errors in the proverbial messaging haystack when things go wrong.
Konnect updates and teasers
We recently launched significant feature updates for Service Catalog and the Developer Portals in Konnect. Here’s some more Konnect news.
Dedicated Cloud Gateways support for VPC Peering and Route 53 Private DNS
Until now, AWS Transit Gateways were the only option in AWS to establish a private connection between Konnect-hosted Dedicated Cloud Gateways (DCGW) to your own AWS VPCs. Transit gateways are excellent for more complex network topologies as it allows you to avoid creating a full mesh of peering connections between VPCs.
VPC Peering, on the other hand, uses a point-to-point model. It's simpler and cheaper to set up when connecting DCGWs to a small number of VPCs.Now you can configure VPC Peering to route traffic from Konnect-hosted DCGW to specific VPCs, circumventing the complexity and cost of AWS Transit Gateways.
Another AWS networking improvement for DCGW is support for private DNS with the AWS Route 53 service. Kong’s private DNS support allows the gateway to query your hosted zone as if it were inside your VPC with no special changes needed to your DNS setup. This works well with both VPC peering and Transit Gateway, assuming the proper VPC association and DNS resolution settings are in place.
For more on this topic, I highly recommend this awesome deep-dive on Dedicated Cloud Gateways by Michael Field, which covers these updates and more.
Active Tracing improvements coming soon
We’re receiving A LOT of positive feedback on Active Tracing — a feature in Konnect that lets you analyze traces to build a sub-millisecond picture of exactly what is happening in your request flow, helping you to resolve issues faster. Customers are reporting hours, sometimes even days' worth, of time saved finding and resolving issues
Kong Gateway deployments in Kubernetes can also benefit from Active Tracing, using Kong Gateway Operator (1.6+) and Konnect, to quickly get to the root of performance bottlenecks.
We’ll soon be releasing some more time-saving features to Active Tracing, like side-by-side views of logs alongside traces, allowing you to instantly jump to logs aligned with the request timeline and the dataplane that actually served the request. Out-of-order trace spans can make it harder to follow the request lifecycle; we're post-processing traces to resolve issues and merge those out-of-order items to the correct parent span. Look out for more news on Active Tracing in July.
If this is your first time hearing of Active Tracing, it’s easy to get started with Active Tracing in Konnect.
That’s it for this release for core Kong Gateway and Kong Event Gateway. We also have a slew of new features in Kong AI Gateway – be sure to check out the announcement for more.
Power your APIs with Kong Gateway
