Sending Traffic Across Namespaces with Gateway API

March 8, 2024

4 min read

Grzegorz Burzyński

Software Engineer, Kong

In this blog post, we’ll demonstrate how easy it is to use Gateway API HTTPRoutes to route traffic to workloads deployed in different namespaces in a single Kubernetes cluster — a process that’s easier than ever.

Previously, we only had Ingress API to define ingress routing rules. It served us well, but it also had its flaws and limitations that we had to overcome, sometimes in a hacky way. One such limitation was routing traffic to Services defined in namespaces different from the one our Ingress is defined in.

For the complete example, please refer to the Gist we’ve created.

Cross-namespace HTTPRoute and Service references with ReferenceGrants

Let’s say we wanted to split our Kubernetes cluster into three namespaces, one per team:

infra - to deploy our company’s API Gateway along with resources configuring its behavior (i.e. Ingress, Gateway, HTTPRoute, etc.).
apples - to deploy team apples-managed workloads.
bananas - to deploy team bananas-managed workloads.

In Ingress API, to use a Service, we had to make sure the Service we’re referring to in the Ingress rules is defined in the same namespace the Ingress is in. So, if we wanted to create an Ingress in the infra namespace that would use Services from both apples and bananas namespaces, we’d have to work this around using non-standard hacks — like creating an additional ExternalName Service in the Ingress namespace, pointing to an FQDN of a Service in the other namespace, for more see the Stack Overflow threads here and here.

The good news? Hacks are no longer needed with Gateway API as these use cases have been taken into account from the beginning.

In Gateway API, an equivalent of Ingress is an HTTPRoute resource. It can, by design, use multiple Services in a single rule as its backends to load-balance the traffic with user-defined weights. What’s more, those Services can be deployed in any namespace as long as there’s a ReferenceGrant object defined in the Service’s namespace that allows it to be referred in specified resources (e.g., an HTTPRoute).

Let’s say, we’d like to define an HTTPRoute in the infra namespace that would route 70 percent of the traffic coming to a /fruits endpoint to the team’s bananas Services (deployed in bananas namespace), and the rest of it shall go to the team’s apples Services (deployed in apples namespace). Such an HTTPRoute would look something like this:

Suppose we define such an HTTPRoute with no ReferenceGrants in bananas and apples namespaces. In that case, our implementation of the Gateway API (Kong Ingress Controller, in our case) will reject such references and will inform us about that in the HTTPRoute parent’s status’ ResolvedRefs condition:

For these references to be resolved, we simply need to create ReferenceGrants for both namespaces:

ReferenceGrant’s spec.from and spec.to lists define from what resources we are allowed to refer to what resources in a namespace we define the ReferenceGrant in. In our case, we allow HTTPRoutes defined in the namespace infra to refer to a Service named banana-echo in the bananas namespace (because the ReferenceGrant is defined herein).

With those in place, we can now ensure that references have been resolved correctly and our traffic is forwarded as expected.

First, let’s check that the references were resolved by the Gateway:

With the references resolved, now we can ensure the traffic is routed as expected:

As expected, our traffic is routed through a Gateway deployed in the infra namespace to Services deployed in the apples and bananas team’s namespaces.

Shared Gateway with cross-namespace HTTPRoutes

Now that we know ReferenceGrant works as a remedy to using Services from different namespaces as backends for Gateway API Routes, let’s now think about another, slightly different scenario for our company infrastructure.

Instead of having a separate namespace for infrastructure (Gateway, HTTPRoutes, etc.), we’d like to allow teams to define HTTPRoutes in their namespaces to give them more control over the ingress traffic rules. The characteristics of our namespaces will be as follows:

infra - to deploy our company’s API Gateway (i.e., Gateway, GatewayClass).
apples - to deploy team apples-managed workloads and Gateway API Routes allowing ingress traffic to them.
bananas - to deploy team bananas-managed workloads and Gateway API Routes allowing ingress traffic to them.

First, let’s create an HTTPRoute in the team’s apples namespace to see what will happen with a default Gateway configuration.

With the above HTTPRoute created, let’s see its status conditions.

We’re getting the Accepted condition with the status False and the reason NotAllowedByListeners. That means our Gateway’s listeners do not allow our HTTPRoute to be attached to them. By default, Gateways will accept only Routes from their namespace. If we want to overcome this limitation, we can use Gateway’s listeners allowedRoutes field.

In our example, we will modify Gateway’s configuration to allow HTTPRoutes from apples and bananas namespaces.

After changing the proxy listener’s allowRoutes as above, we should now be able to see our HTTPRoute successfully accepted by the Gateway.

To finally verify that our HTTPRoute created in the apples namespace was successfully attached to the Gateway and configured, we can call the endpoint that’s expected to respond with the 200 status code.

And that’s it!

Conclusion

To sum up what we’ve learned, both Gateway’s listeners’ allowedRoutes and ReferenceGrant are useful Gateway API tools. They allow explicit configuration of rules that the Gateway controller will follow when resolving cross-namespace references that may occur in the Gateway API objects definitions.

If you would like to read more about the topic, here are some reference links that might be helpful: