Orchestrating Konnect’s Great API Renaming with Kong Gateway Transformer Plugins
Before we dive in, we want to remind our readers that the Konnect engineering team actively uses Kong products like Kong Gateway, Kong Mesh, and Insomnia.
In this post, we'll showcase the power of Kong Gateway and two plugins — the JQ and Request Transformer Advanced Plugins — to govern and configure APIs, and explain how it played, and continues to play, a crucial role in the release of a new v2 Konnect API specification.
The key takeaway here is: don’t do more work than is necessary. With that in mind, let’s get started!
Introduction
For API Summit 2023, the Kong Konnect Engineering team quietly executed a project, codename: “The Great Renaming.” Our objective? To rename Runtime Manager and Runtime Groups to what you see today, Gateway Manager and Control Planes. From an engineering perspective, this boiled down to releasing a new major version of the Konnect APIs.
However, what is critical to understand is that the desired changes to the APISpec did not change any existing customer-facing behavior. It was purely a cosmetic change, and we wanted a solution that insulated us from refactoring Konnect for seemingly cosmetic changes — not only in this one particular instance but for future cases as well.
As Konnect evolves as a SaaS platform these “superficial” changes to the API spec are also likely to continue to evolve, to align with the language of the times. It was absolutely true in this case.
There is good reason to suspect that product management will have to request changes that arguably is just a terminology change.
While it’s true that our customers could have continued using Konnect with the v2 specs, these minor changes have a significant positive impact. The updated terminology makes it easier for both existing and potential customers to understand the key technical concepts of Konnect as a whole.
So with the product requirements in hand, the next task was to plan the engineering work. From a technical perspective, there were a number of requirements.
First, support both v2/runtime-group(s) and v2/control-plane(s) API specs simultaneously. These interfaces are the predominant way that customers interact with Konnect and it's critical that our existing customers have backwards compatibility.
Second, introduce the fewest changes possible to the backend system. The team really wanted to keep the blast radius of the change small, if at all possible, only customer facing, and not develop new APIs that matched the specification. In other words, our internal-facing microservices do not need to know that gateway-manager is synonymous to runtime-manager. The Konnect backend leans in heavily to the microservice architecture. So it was important that minor customer-facing changes do not require a cascading update to surrounding microservices to consume the new version of the APIs as well.
Debut of Kong Gateway and the transformer plugins
Because of the features of the Kong Gateway in conjunction with the plugins, a majority of the Konnect backend teams had zero work involved in the release of the new APIs.
With Kong Gateway, there are three main abstractions to concern yourself with for the sake of this story: gateway services, routes, and plugins. A gateway service is an abstraction of an upstream application, and routes define what APIs are exposed on that gateway service. A gateway service can have 1:N routes. Then plugins can be configured in various contexts depending on the granularity needed when that plugin is applied at runtime: Services, Routes, etc.
So from a high-level perspective, we used these abstractions to build the following gateway configuration:
- Create new routes for the APIspecs (v2/control-planes) and keep the old (v2/runtime-groups) API spec unchanged for backwards compatibility.
- Apply the jq plugin and the request-transformer-advanced plugin to support request/response body and query-string transformations isolated to the new routes.
The novelty in the design is two-fold. First is its simplicity and ease of maintenance. All work related to the new API is confined to the Kong Gateway configuration, avoiding unnecessary complexity in our microservices. Second, it establishes a clear separation of concerns between the client-facing API design and the internal workings of Konnect. As an API-first company, we ensure that our client-facing APIs are thoroughly vetted by our product teams, and this design supports that focus.
So, in conclusion, it's nice.
From a timeline perspective, once the product team reached a consensus on the new API specs, the gateway config was ready within a couple of weeks, giving a good lead time for the frontend and docs to begin refactoring as well.
Transformer plugin configuration
Let’s quickly give you a purview of our plugin configurations.
In practice, the jq plugin is highly powerful due to its ability to create complex JSON processors for transforming request and response bodies. It's also widely used in the open-source community and transformations are easily optimized because it's a CLI tool as well. Beyond the jq interface, the plugin offers several useful features. It supports separate transformations for requests and responses within the same configuration and can conditionally execute transformations based on response codes.
Take a look at the snippet from the Konnect deck config below where we leverage all of these features:
Meanwhile, the request-transformer-advanced plugin was specifically used to target transformations to the query parameters. The request-transformer-advanced plugin can also handle simple transformations such as removing, appending headers, query strings or to the body. But in order to replace the query string it really required the lua function you see below.
Just to really drive home the point of how convenient this setup is, the plugin deck config is exactly 30 lines of code.
Wrapping up
If you remember anything from the post it should be how we leveraged Kong Gateway's capabilities to expose new APIs to our customers and maintain backwards compatibility with minimal disruption to the underlying architecture.
But before we go, some shout-outs are in order. At the end of the day, the great renaming as a whole was a large undertaking. The plugin configuration described here was rather simple, however, the entire project itself was complex in nature due to the number of cross-cutting concerns: from getting consensus from the product team to executive leadership on the new API terminology, followed by building the API specs, a major refactor the Konnect frontend to catch every small possible terminology, updating the docs, updating our E2E integration on the fly, and release all assets at once.
So with that, whether you are an engineer, an engineering manager, or a product manager, we hope you learned of a new use case of Kong Gateway. Onwards!