From the Kong API Gateway perspective, using Consul as its Service Discovery infrastructure is one of the most well-known and common integration use cases. With this powerful combination more flexible and advanced routing policies can be implemented to address Canary Releases, A/B testings, Blue-Green deployments, etc. totally abstracted from the Gateway standpoint without having to deal with lookup procedures.

This article focuses on integrating Kong for Kubernetes (K4K8S), the Kong Ingress Controller based on the Kong API Gateway, and Consul Service Discovery running on a Kubernetes EKS Cluster. Kong for Kubernetes can implement all sorts of policies to protect the Ingresses defined to expose Kubernetes services to external Consumers including Rate Limiting, API Keys, OAuth/OIDC grants, etc.

The following diagram describes the Kong for Kubernetes Ingress Controller and Consul Service Discovery implementing a Canary Release:

Consul and Kong for Kubernetes Installation Process

This section assumes you have a Kubernetes Cluster with both Consul and Kong for Kubernetes installed. This HashiCorp link can help you spin up a Consul Kubernetes deployment. Similarly, Kong provides the following link to install Kong for Kubernetes.

Consul Configuration Process

After getting your Kubernetes Cluster installed with Consul and Kong for Kubernetes deployed, we’re ready to start the 5-step configuration process:

1. Configure Kubernetes DNS Service
2. Deploy both Current and Canary application releases
3. Register a Consul Service based on both application releases
4. Create an External Kubernetes Service based on the Consul Service
5. Register a Kong for Kubernetes Ingress for the External Service

Configure Kubernetes DNS

First of all, let’s configure the Kubernetes in order to consume Consul’s primary query instance based on DNS. The configuration depends on the DNS provided by the Kubernetes engine you are using. Please, refer to this link to check how to configure KubeDNS or CoreDNS.

Once configured, DNS requests in the form <consul-service-name>.service.consul will resolve for Consul Services. As an example, here are the configuration steps for CoreDNS:

Get the Consul DNS’ Cluster IP:

kubectl get service consul-consul-dns -n hashicorp -o jsonpath='{.spec.clusterIP}'
10.105.175.26

Edit the CoreDNS ConfigMap to include a forward definition that points to the Consul DNS’s Kubernetes Services.

kubectl edit configmap coredns -n kube-system
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    consul {
        errors
        cache 30
        forward . 10.105.175.26
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2020-06-19T13:42:16Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:Corefile: {}
    manager: kubeadm
    operation: Update
    time: "2020-06-19T13:42:16Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "178"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 698c5d0c-998e-4aa4-9857-67958eeee25a

Deploy both Current and Canary application releases

For the purpose of this article we’re going to create our Kubernetes Deployments using basic Docker Images for both Current and Canary releases available in http://hub.docker.com. Both Images return the current datetime, differing from each other by the text used. As expected, after the deployment, you should see two Kubernetes Services: benigno-v1 and benigno-v2.

The Current application release can be deployed using the following declaration:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: benigno-v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: benigno
      version: v1
  template:
    metadata:
      labels:
        app: benigno
        version: v1
    spec:
      containers:
      - name: benigno
        image: claudioacquaviva/benigno
        ports:
        - containerPort: 5000
----
apiVersion: v1
kind: Service
metadata:
  name: benigno-v1
  labels:
    app: benigno-v1
spec:
  type: ClusterIP
  ports:
  - port: 5000
    name: http
  selector:
    app: benigno
    version: v1
EOF

The Canary Release is deployed using the command below:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: benigno-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: benigno
      version: v2
  template:
    metadata:
      labels:
        app: benigno
        version: v2
    spec:
      containers:
      - name: benigno
        image: claudioacquaviva/benigno_rc
        ports:
        - containerPort: 5000
----
apiVersion: v1
kind: Service
metadata:
  name: benigno-v2
  labels:
    app: benigno-v2
spec:
  type: ClusterIP
  ports:
  - port: 5000
    name: http
  selector:
    app: benigno
    version: v2
EOF

Register a Consul Service based on both application releases

Now, we have to register a Consul Service based on both Kubernetes Services we have deployed. The benigno1 Consul Service will have both Kubernetes Services’ Cluster IPs configured with different weights. So, any DNS request to it will return one of the IPs applying the weights defined.

In order to get the Kubernetes Services’ Cluster IPs run:

kubectl get service --all-namespaces
NAMESPACE   NAME           TYPE        CLUSTER-IP      EXTERNAL-IP  PORT(S      AGE
default     benigno-v1     ClusterIP   10.100.225.125               5000/TCP    116s
default     benigno-v2     ClusterIP   10.100.148.236               5000/TCP    12s
…

Then create two files as described below using the Cluster IPs. Notice the weights used saying that the Consul DNS will return the Canary Release IP address for only 20% of the requests:

ben0.json:

{
  "ID": "ben0",
  "Name": "benigno1",
  "Tags": ["primary"],
  "Address": "10.100.225.125",
  "Port": 5000,
  "weights": {
    "passing": 80,
    "warning": 1
  }
  "proxy": {
    "local_service_port": 5000
  }
}

ben1.json:

{
  "ID": "ben1",
  "Name": "benigno1",
  "Tags": ["secondary"],
  "Address": "10.100.148.236",
  "Port": 5000,
  "weights": {
    "passing": 20,
    "warning": 1
  }
  "proxy": {
    "local_service_port": 5000
  }
}

Expose Consul using port-forward so we can send requests to it and get the Consul Service registered. On one local terminal run:

kubectl port-forward service/consul-connect-consul-server -n hashicorp 8500:8500

Open another local terminal to send the requests using the files created before. We’re using HTTPie to send the requests. Feel free to use any other tool.

http put :8500/v1/agent/service/register < ben0.json
http put :8500/v1/agent/service/register < ben1.json

Create an External Kubernetes Service based on the Consul Service

After registering the Consul Service, any DNS request to benigno1.service.consul will return one of the IPs applying the weight policy described. Now, we create an External Service to define a specific Kubernetes reference to the Consul Service.

cat <<EOF | kubectl apply -f -
kind: Service
apiVersion: v1
metadata:
  name: benigno1
spec:
  ports:
  - protocol: TCP
    port: 5000
  type: ExternalName
  externalName: benigno1.service.consul
EOF

Register a Kong for Kubernetes Ingress for the External Service

Finally we’re going to expose the Canary Release through an Ingress managed by Kong for Kubernetes. Using the External Service created before we abstract both Application releases under the Consul Service benigno1.service.consul name.

cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: benignoroute
  annotations:
    kubernetes.io/ingress.class: kong
    konghq.com/strip-path: "true"
spec:
  rules:
  - http:
      paths:
        - path: /benignoroute
          backend:
            serviceName: benigno1
            servicePort: 5000
EOF

You can test the Ingress sending a request like this:

$ http <K4K8S-EXTERNALIP>:<K4K8S-PORT>/benignoroute
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 36
Content-Type: text/html; charset=utf-8
Date: Wed, 16 Sep 2020 20:37:22 GMT
Server: Werkzeug/1.0.1 Python/3.8.3
Via: kong/2.1.3
X-Kong-Proxy-Latency: 4
X-Kong-Upstream-Latency: 2

Hello World, Benigno, Canary Release

Start a loop to see the Canary Release in action:

while [ 1 ]; do curl http://<K4K8S-EXTERNALIP>:<K4K8S-PORT>/benignoroute; sleep 1; echo; done

Kong for Kubernetes provides CRDs not just to define Ingresses but also apply typical policies defined at the Ingress Controller layer. Feel free to experiment further policy implementations like caching, log processing, OIDC-based authentication, GraphQL integration and more with the extensive list of plugins provided by Kong.