How We Built It: Managing Konnect Entities from K8s Clusters with KGO
We recently released Kong Gateway Operator 1.4 with support for managing Konnect entities from within the Kubernetes clusters. This means users can now manage their Konnect configurations declaratively, through Kubernetes resources powered by Kong’s Custom Resource Definitions. For example, here’s how you can use a KongConsumer resource in Kubernetes to configure a Consumer in Konnect.
You can read more about using the Operator to declaratively manage Konnect in the Kong Gateway Operator docs. In this blog post, we’ll go through the design of this feature and how we’ve built it.
API design
In order to interact with Kubernetes using types that model Konnect entities, we have to define CRDs for each of the types that it supports. This will allow users to have strongly typed resources to work with.
There are several CRDs that are already supported by KIC. These are defined in configuration.konghq.com API group and could potentially be used for Konnect entities support but we’d need to clearly mark the line which types are supported in KIC, which are supported by the operator, managing Konnect’s objects and which are supported by both.
An alternative was to define a new, isolated API Group dedicated to Konnect. This would allow for a clear partition between Konnect and self-hosted Kubernetes configuration but it would require users to learn, install, and manage yet another set of CRDs. We wanted to avoid that so we’ve decided to use the first approach.
We’ve decided to set this line at the Konnect product level and following these two rules:
- Anything that lives as part of a control plane’s configuration is still going to use
configuration.konghq.comAPI Group, - Anything else that constitutes an entity on its own in Konnect will live in a brand new API Group:
konnect.konghq.com
This means that apart from the newly introduced CRDs, users will be able to use types that they are already familiar with, like KongConsumer, KongPlugin, etc.
These new CRDs are available at https://github.com/Kong/kubernetes-configuration. Types from configuration.konghq.com are backward compatible with KIC’s CRDs that many of you are already familiar with.
Control plane references
In order to satisfy the above-described approach, we’ve decided to extend each type that lives as part of the control plane’s configuration, with a control plane reference. To continue supporting resources without this field set, we made it optional with the default type set to kic — indicating that the object is to be reconciled by KIC. So a KongConsumer can still be used as:
In order to make your resources refer to a locally managed Konnect ControlPlane, you can use the control plane reference with type set to konnectNamespacedRef, like so:
Konnect SDK
The next piece of the puzzle is the actual communication with Konnect and the underlying Konnect API types. For this, we use our sdk-konnect-go package.
Thanks to speakeasy’s Go SDK generation, this SDK is almost entirely generated from our OpenAPI spec which is the source of truth for our APIs. This allows us to focus on the robustness of our kubernetes operator and leave the API specification to teams that provide those.
We leverage Go types generated from the OpenAPI specification in the underlying CRD types where possible so that we get autogenerated CRDs when there’s a change in API.
SDK types validation
For CRD definitions we utilize kubebuilder CRD markers.
While some of the types do not require additional validation in the form of those markers, some do. For those we resort to redefining the types and adding kubebuilder markers for CRD validation for native Kubernetes object validation.
We hope to solve this problem soon and to have one source of truth for our types.
Reconciliation loop’s design
Following Kubernetes operator pattern — as we’ve done so for every other controller in our operator —we’ve designed a control loop which takes the provided resource spec (the desired state) and aims to progress the Konnect entity state (the observed state) to match the former.
The idea behind it is pretty simple (simplified code):
Fortunately, making the controller continuously reconcile an object is trivial using controller-runtime’s Result.RequeueAfter.
One important thing to note is that the operator will always defer back to Konnect as the source of truth if conflicts arise. This is especially important with using cached clients in one’s controllers, which is the default using controller-runtime’s client.Client.
Code generation vs Go’s generics
In order to support the plethora of object types that Konnect APIs allow its users to configure, we needed to choose an approach to be type agnostic (to a degree) in our implementation and not implement the same controller for each type.
We’ve decided to use Go’s generics to leverage strongly typed code, have full control over type constraints but also to minimize the amount of code in our repository.
Our implementation uses two core type constraints:
and
The distinction between the two came from the fact that some functions ( that we had no control over) required pointers and some values to work with.
This in turn allowed us to write our reconciler like so:
We were able to then instantiate a reconciler for each of the supported types. Setting up watch criteria for each type at this point was a matter of:
Generics limitations
Generics can only get you so far. They're very useful for deduplicating code and can immensely help with making your codebase smaller but they also have their limitations.
The biggest drawback is that you can't specialize your code without resorting to type switches or interface assertions to verify whether a particular reconciliation step should be performed for the type in question.
For example: most of our types have a Control Plane reference but not all (Control Planes themselves do not reference each other, with the exception of Control Plane groups). To check this we can either check each type individually —which is error prone — or generate a GetControlPlaneRef() method on each of those types and perform a type assertion verifying the object implements it:
This is far from ideal but until Go implements generics specialization, we have to live with it.
Conclusion
We hope that this was insightful and now you know how Kong Gateway Operator manages Konnect entities from Kubernetes clusters.
If you'd like to read more about the topic, please visit the links below. We’re always looking for feedback so in case there’s anything you’d like to share with us please open an issue or reach out to us through the support channel.