In this blog post, we’ll demonstrate how easy it is to use Gateway API HTTPRoutes to route traffic to workloads deployed in different namespaces in a single Kubernetes cluster — a process that’s easier than ever.

Previously, we only had Ingress API to define ingress routing rules. It served us well, but it also had its flaws and limitations that we had to overcome, sometimes in a hacky way. One such limitation was routing traffic to Services defined in namespaces different from the one our Ingress is defined in.

For the complete example, please refer to the [Gist](https://gist.github.com/czeslavo/3550bd52880f1a946075f4e2b82b3fe6)Gist we’ve created.

## Cross-namespace HTTPRoute and Service references with ReferenceGrants

Let’s say we wanted to split our Kubernetes cluster into three namespaces, one per team:

• - **infra** - to deploy our company’s API Gateway along with resources configuring its behavior (i.e. Ingress, Gateway, HTTPRoute, etc.).
• - **apples** - to deploy team apples-managed workloads.
• - **bananas** - to deploy team bananas-managed workloads.

In Ingress API, to use a Service, we had to make sure the Service we’re referring to in the Ingress rules is defined in the same namespace the Ingress is in. So, if we wanted to create an Ingress in the infra namespace that would use Services from both apples and bananas namespaces, we’d have to work this around using non-standard hacks — like creating an additional ExternalName Service in the Ingress namespace, pointing to an FQDN of a Service in the other namespace, for more see the Stack Overflow threads [here](https://stackoverflow.com/questions/59844622/ingress-configuration-for-k8s-in-different-namespaces)here and [here](https://stackoverflow.com/questions/51878195/kubernetes-cross-namespace-ingress-network)here.

The good news? Hacks are no longer needed with [Gateway API](https://konghq.com/blog/engineering/gateway-api-vs-ingress)Gateway API as these use cases have been taken into account from the beginning.

In Gateway API, an equivalent of Ingress is an [HTTPRoute](https://gateway-api.sigs.k8s.io/api-types/httproute/)HTTPRoute resource. It can, by design, use multiple Services in a single rule as its backends to load-balance the traffic with user-defined weights. What’s more, those Services can be deployed in any namespace as long as there’s a ReferenceGrant object defined in the Service’s namespace that allows it to be referred in specified resources (e.g., an HTTPRoute). 

Let’s say, we’d like to define an HTTPRoute in the infra namespace that would route 70 percent of the traffic coming to a /fruits endpoint to the team’s bananas Services (deployed in bananas namespace), and the rest of it shall go to the team’s apples Services (deployed in apples namespace). Such an HTTPRoute would look something like this:

kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: echo-route
  namespace: infra
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: kong
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /fruits
      backendRefs:
        - name: bananas-echo
          port: 1027
          kind: Service
          namespace: bananas
          weight: 30
        - name: apples-echo
          port: 1027
          kind: Service
          namespace: apples
          weight: 70

Suppose we define such an HTTPRoute with no ReferenceGrants in bananas and apples namespaces. In that case, our implementation of the Gateway API (Kong Ingress Controller, in our case) will reject such references and will inform us about that in the HTTPRoute parent’s status’ ResolvedRefs condition:

$ kubectl get httproute -n infra echo-route -o=jsonpath='{.status.parents[0].conditions}' | jq 
[
  ...
  {
    "observedGeneration": 1,
    "reason": "RefNotPermitted",
    "status": "False",
    "type": "ResolvedRefs"
  },
  ...
]

For these references to be resolved, we simply need to create ReferenceGrants for both namespaces:

kind: ReferenceGrant
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: allow-apples-echo-in-infra
  namespace: bananas
spec:
  from: 
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: infra
  to:
    - group: ""
      kind: Service
      name: bananas-echo # This could be left empty to allow ANY service in the namespace.
# A similar one for apples namespace and apples-echo service should be created as well.

ReferenceGrant’s spec.from and spec.to lists define **from** what resources we are allowed to refer **to** what resources in a namespace we define the ReferenceGrant in. In our case, we allow HTTPRoutes defined in the namespace infra to refer to a Service named banana-echo in the bananas namespace (because the ReferenceGrant is defined herein). 

With those in place, we can now ensure that references have been resolved correctly and our traffic is forwarded as expected. 

First, let’s check that the references were resolved by the Gateway:

$ kubectl get httproute -n infra echo-route -o=jsonpath='{.status.parents[0].conditions}' | jq
   [
      {
         "observedGeneration": 2,
         "reason": "ResolvedRefs",
         "status": "True",
         "type": "ResolvedRefs"
      },
   ]

With the references resolved, now we can ensure the traffic is routed as expected:

$ http 'http://${PROXY_IP}/fruits'
   HTTP/1.1 200 OK
   Connection: keep-alive
   Content-Length: 22
   Content-Type: text/plain; charset=utf-8
   Date: Mon, 06 Nov 2023 18:06:01 GMT
   Via: kong/3.4.2
   X-Kong-Proxy-Latency: 2
   X-Kong-Upstream-Latency: 3

   In namespace apples.

   $ http 'http://${PROXY_IP}/fruits'
   HTTP/1.1 200 OK
   Connection: keep-alive
   Content-Length: 23
   Content-Type: text/plain; charset=utf-8
   Date: Mon, 06 Nov 2023 18:06:02 GMT
   Via: kong/3.4.2
   X-Kong-Proxy-Latency: 0
   X-Kong-Upstream-Latency: 1

   In namespace bananas.

As expected, our traffic is routed through a Gateway deployed in the infra namespace to Services deployed in the apples and bananas team’s namespaces.

## Shared Gateway with cross-namespace HTTPRoutes

Now that we know ReferenceGrant works as a remedy to using Services from different namespaces as backends for Gateway API Routes, let’s now think about another, slightly different scenario for our company infrastructure. 

Instead of having a separate namespace for infrastructure (Gateway, HTTPRoutes, etc.), we’d like to allow teams to define HTTPRoutes in their namespaces to give them more control over the ingress traffic rules. The characteristics of our namespaces will be as follows:

• - **infra** - to deploy our company’s API Gateway (i.e., Gateway, GatewayClass).
• - **apples** - to deploy team apples-managed workloads and Gateway API Routes allowing ingress traffic to them.
• - **bananas** - to deploy team bananas-managed workloads and Gateway API Routes allowing ingress traffic to them.

First, let’s create an HTTPRoute in the team’s apples namespace to see what will happen with a default Gateway configuration.

kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: route
  namespace: apples
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: kong
      namespace: infra # We need to explicitly specify the namespace of the Gateway as now it is not in the same namespace 
                       # as our HTTPRoute (apples).
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /apples
      backendRefs:
        - name: apples-echo
          port: 1027
          kind: Service
          # We can omit the namespace as the Service is in the same namespace as our HTTPRoute (apples).

With the above HTTPRoute created, let’s see its status conditions.

$ kubectl get -n apples httproute route -o=jsonpath='{.status.parents[0].conditions}' | jq
[
    {
        "lastTransitionTime": "2023-12-07T12:31:56Z",
        "observedGeneration": 1,
        "reason": "NotAllowedByListeners",
        "status": "False",
        "type": "Accepted"
    },
]

We’re getting the *Accepted* condition with the status *False* and the reason *NotAllowedByListeners*. That means our Gateway’s listeners do not allow our HTTPRoute to be attached to them. By default, Gateways will accept only Routes from their namespace. If we want to overcome this limitation, we can use Gateway’s listeners *allowedRoutes* field. 

In our example, we will modify Gateway’s configuration to allow HTTPRoutes from apples and bananas namespaces.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kong
  namespace: infra
spec:
  gatewayClassName: kong
  listeners:
  - name: proxy
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchExpressions:
            # Allow attaching Routes from apples and bananas namespaces.
            - key: kubernetes.io/metadata.name
              operator: In
              values: [apples, bananas]

After changing the proxy listener’s *allowRoutes* as above, we should now be able to see our HTTPRoute successfully accepted by the Gateway.

$ kubectl get -n apples httproute route -o=jsonpath='{.status.parents[0].conditions}' | jq
 [
     {
         "lastTransitionTime": "2023-12-07T13:09:58Z",
         "observedGeneration": 1,
         "reason": "Accepted",
         "status": "True",
         "type": "Accepted"
     },
 ]

To finally verify that our HTTPRoute created in the apples namespace was successfully attached to the Gateway and configured, we can call the endpoint that’s expected to respond with the 200 status code.

$ http "http://${PROXY_IP}/apples?details=true"
HTTP/1.1 200 OK

In namespace apples.

And that’s it! 

## Conclusion

To sum up what we’ve learned, both Gateway’s listeners’ allowedRoutes and ReferenceGrant are useful Gateway API tools. They allow explicit configuration of rules that the Gateway controller will follow when resolving cross-namespace references that may occur in the Gateway API objects definitions.

If you would like to read more about the topic, here are some reference links that might be helpful:

• - [Cross-Namespace routing](https://gateway-api.sigs.k8s.io/guides/multiple-ns/)Cross-Namespace routing — Guide from the Gateway API official documentation
• - [Gateway Listener’s AllowedRoutes API reference](https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1.AllowedRoutes)Gateway Listener’s AllowedRoutes API reference
• - [ReferenceGrant API reference](https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1alpha2.ReferenceGrant)ReferenceGrant API reference