In this series of posts, we will look closely at how Kong Konnect Data Planes can take advantage of Autoscalers running on Amazon Elastic Kubernetes Services (EKS) 1.29 to support the throughput the demands API consumers impose on it at the lowest cost. The series comprises four parts:

• VPA - Vertical Pod Autoscaler
• HPA - Horizontal Pod Autoscaler
• Cluster Autoscaler
• Karpenter

Introduction

One of the primary drivers for companies to migrate their infrastructure to the cloud is elasticity, the ability to dynamically scale, up and down, compute and container resources according to their business needs. The "Overview of Amazon Web Services" whitepaper outlines the advantages of cloud computing related to it: variable expense and no more capacity guessing.

In summary, companies can reduce their costs with the "pay-as-you-go" model provided by cloud elasticity.

It's important to clarify some fundamental concepts. Given an application, in order to support different workloads, basically, we have two main options:

• Vertical scalability (scale up/down): you are adding or subtracting hardware resources (memory, CPU, storage) to/from your application.
• Horizontal scalability (scale out/in): you are adding or subtracting more nodes or servers and distributing the workload across them.

Elasticity goes beyond that. It provides the capacity to automatically provision and deprovision hardware resources based on demand. In other words, you will be able to vertically scale up/down or horizontally scale in/out your applications in an automatic way. This is also called "autoscaling."

Autoscaling and Kubernetes

Autoscaling is one of the most compelling features of the Kubernetes platform. Basically, it is available in two different perspectives:

1. Pod Autoscaling

• Pod Vertical Scalability with VPA (Vertical Pod Autoscaler): it increases and decreases the CPU and memory allocation for your Pods.
• Pod Horizontal Scalability with HPA (Horizontal Pod Autoscaler): it adds and subtracts the number of Pods of a given Kubernetes Deployment.

The following diagram shows both Vertical and Horizontal mechanisms. Note that, they work with a single Node since they are Pod Autoscaling options.

2. Cluster Scalability

Also called Node Autoscaling, this adds and subtracts Kubernetes Nodes of your Cluster. lt can be achieved with two different autoscalers:

• Cluster Autoscaler (CAS) is a Kubernetes-native tool that adds or removes Nodes, based on pending Pods and Node usage metrics. Cluster Autoscaler lacks the capacity to provision Nodes based on the requirements a Pod requests to run. It just identifies an unscheduled Pod and replicates an existing Node from the NodeGroup with identical specifications. It relies on EC2 and Auto Scaling Groups (ASG)
• Karpenter is an open source node provisioning project built for Kubernetes. Differently from CAS, Karpenter doesn't manage NodeGroups, instantiating EC2s directly and adding them as regular group-less Nodes. The main benefit is that it chooses the right Instance Type to support the required throughput. It can also consolidate multiple Nodes into larger ones.

The following diagram compares the two Cluster Scalability options. The main difference highlighted here is the capacity provided by Karpenter to choose Instant Types other than the original one created.

All these components are maintained by the Kubernetes Autoscaling Special Interest Group (SIG).

Konnect Data Plane and Autoscaling

Kong Konnect is an API lifecycle management platform delivered as a service. The Management Plane, also called the Control Plane, is hosted in the cloud by Kong, while the Gateway nodes, called Data Planes, are deployed in your environments.

The Control Plane enables customers to securely execute API management activities such as create API routes, define services, etc. Runtime environments connect with the management plane using mutual transport layer authentication (mTLS), receive the updates and take customer-facing API traffic.

The diagram below illustrates the architecture:

This blog post describes basic Konnect Data Planes deployments on an Amazon EKS 1.29 Cluster, taking advantage of these 4 different autoscaling technologies.

Before we get started, it's important to make two critical comments:

• The use cases we are going to run are not meant to be applied for production environments. The main purpose is to demonstrate how to leverage the Autoscaling capabilities we have available for our workloads and applications.
• The Konnect Kong Gateway Data Plane deployments are not tuned to deliver the best performance they can get. For optimal Konnect Kong Gateway performance configuration check the official documentation.

Amazon EKS Cluster Creation

The first thing to do is create the EKS Cluster along with a NodeGroup. For the basic autoscaling use cases we are going to run, I have chosen the t3.2xlarge Instance Type which provides plenty of space for CPU and memory allocation.

You can use eksctl CLI like this:

eksctl create cluster --name kong35-eks129-autoscaling --version 1.29 --region us-west-1 --without-nodegroup --asg-access

The --asg-access is not used for VPA and HPA configurations but for the Cluster Autoscaler settings we are going to explore later on.

Now, create the Managed NodeGroup for the Cluster with:

eksctl create nodegroup --cluster kong35-eks129-autoscaling --region us-west-1 \
  --name nodegroup-kong \
  --node-labels="nodegroupname=kong" \
  --node-type t3.2xlarge \
  --nodes 1 \
  --nodes-min 1 --nodes-max 10 \
  --max-pods-per-node 50

Metrics Server

Both VPA and HPA require Kubernetes Metrics Server to work, so let's install it with

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Kong Konnect Control Plane Creation

The next step is to create a Konnect Control Plane (CP). We are going to use the Konnect Rest Admin API to do so. Later on, we will spin up the Konnect Data Plane (CP) in our EKS Cluster.

The CP/DP communication is based on mTLS so we need a Private Key and Digital Certification pair. To make that easy, let's use openssl to issue them.

openssl req -new -x509 -nodes -newkey ec:<(openssl ecparam -name secp384r1) \
-keyout ./kongcp1.key \
-out ./kongcp1.crt \
-days 1095 \
-subj "/CN=konnect-cp1"

To use the Konnect Rest Admin API, we need a Konnect PAT (Personal Access Token) in order to send requests to Konnect. Read the Konnect PAT documentation page to learn how to generate one.

Create a Konnect Control Plane with the following command. It configures the Pinned Mode for the CP and DP communication, meaning we are going to use the same Public Key to both CP and DP.

curl -X POST \
  https://us.api.konghq.com/v2/control-planes \
  --header 'Authorization: Bearer <your_pat>' \
  --header 'Content-Type: application/json' \
  --header 'accept: application/json' \
  --data '{
  "name": "cp1",
  "description": "Control Plane 1",
  "cluster_type": "CLUSTER_TYPE_HYBRID",
  "labels":{},
  "auth_type": "pinned_client_certs"
}'

The following command should return the Konnect Control Plane Id:

curl -s https://us.api.konghq.com/v2/control-planes \
  --header 'Authorization: Bearer <your_pat>' | jq -r '.data[] | select(.name=="cp1") | .id'  

Get the CP's Endpoints with:

% curl -s https://us.api.konghq.com/v2/control-planes/<your_control_plane_id> \
  --header 'Authorization: Bearer <your_pat>' | jq -r ".config"
{
  "control_plane_endpoint": "https://1234567890.us.cp0.konghq.com",
  "telemetry_endpoint": "https://1234567890.us.tp0.konghq.com",
  "cluster_type": "CLUSTER_TYPE_CONTROL_PLANE",
  "auth_type": "pinned_client_certs",
  "cloud_gateway": false
}

Now we need to Pin the Digital Certificate. Use the CP Id in your request:

cert="{\"cert\": $(jq -sR . ./kongcp1.crt)}"

curl -X POST https://us.api.konghq.com/v2/runtime-groups/<your_control_plane_id>/dp-client-certificates \
  --header 'Authorization: Bearer <your_pat>' \
  --header 'Content-Type: application/json' \
  --header 'accept: application/json' \
  --data $cert

Kong Service and Route

With the Konnect Control Plane defined, it's time to create a Kong Service and Route, so the Data Plane can have an application to consume. For this Autoscaling exploration, The Konnect Data Plane will consume an Upstream service (based on the httpbin echo application), running on the same Cluster, we are going to deploy later on. In fact, the Kong Service refers to the Kubernetes Service's FQDN.

The following command creates a Kong Service:

http https://us.api.konghq.com/v2/control-planes/<your_control_plane_id>/core-entities/services name=service1 url='http://httpbin.default.svc.cluster.local' Authorization:"Bearer <your_pat>"

The following command should return the Kong Service Id:

http https://us.api.konghq.com/v2/control-planes/<your_control_plane_id>/core-entities/services/service1 Authorization:"Bearer <your_pat>" | jq -r ".id"

With the Kong Service Id, create a Kong Route:

http https://us.api.konghq.com/v2/runtime-groups/<your_control_plane_id/core-entities/services/<your_service_id>/routes name='route1' paths:='["/route1"]' Authorization:"Bearer <your_pat>"

Kong Konnect Data Plane deployment

We are ready to deploy the Konnect Data Plane (DP) in the EKS Cluster. Create a specific namespace and a secret with the Private Key and Digital Certificate pair.

kubectl create namespace kong

kubectl create secret tls kong-cluster-cert -n kong --cert=./kongcp1.crt --key=./kongcp1.key

Next, create the values.yaml we are going to use to deploy the Konnect Data Plane. Use the Control Plane endpoints you got when you created it.

Note we are going to expose the Data Plane with a Network Load Balancer, which provides the best performance for it. Besides, with the nodeSelector config we are explicitly referring to the NodeGroup we created previously.

Again, keep in mind this blog post describes a simple environment for autoscaling tests. That's the reason we are using nodeSelector for Pod isolation. As a best practice, for a production environment, it should be used as an exception.

From the VPA perspective, the first autoscaling technology we are going to explore, the resources section is the main configuration. It sets the CPU and memory requests and limits for the Data Plane.

cat > values.yaml << 'EOF'
image:
  repository: kong/kong-gateway
  tag: "3.5"

secretVolumes:
- kong-cluster-cert

admin:
  enabled: false

manager:
  enabled: false

proxy:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb

env:
  role: data_plane
  database: "off"
  cluster_mtls: pki
  cluster_control_plane: 1234567890.us.cp0.konghq.com:443
  cluster_server_name: 1234567890.us.cp0.konghq.com
  cluster_telemetry_endpoint: 1234567890.us.tp0.konghq.com:443
  cluster_telemetry_server_name: 1234567890.us.tp0.konghq.com
  cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt
  cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key
  lua_ssl_trusted_certificate: system
  konnect_mode: "on"
  vitals: "off"

nodeSelector:
  nodegroupname: kong

ingressController:
  enabled: false
  installCRDs: false

resources:
  requests:
    cpu: 0.1
    memory: 200Mi
  limits:
    cpu: 0.2
    memory: 300Mi'
EOF

Deploy the Konnect Data Plane with the Helm Charts:

helm repo add kong https://charts.konghq.com
helm repo update

helm install kong kong/kong -n kong --values ./values.yaml

You should see the Data Plane running:

% kubectl get pod -n kong

For a basic consumption test send a request to the Data Plane. Get the NLB DNS name with:

% kubectl get service -n kong -o json | jq -r ".items[].status.loadBalancer.ingress[].hostname"
<your_nlb_dnsname>

http <your_nlb_dnsname>

You can also check the Data Plane's Pod resources:

% kubectl get pod -n kong -o json | jq ".items[].spec.containers[].resources"
{
  "limits": {
    "cpu": "200m",
    "memory": "300Mi"
  },
  "requests": {
    "cpu": "100m",
    "memory": "200Mi"
  }
}

VPA

VPA Architecture

The diagram below was taken from the VPA's project repository. VPA is fully documented in a specific vertical-pod-autoscaler directory. Please check the repo to learn more about VPA.

VPA consists of three main components:

• Recommender monitors the current and past resource consumption and, based on it, provides recommended values for the containers' CPU and memory requests.
• Updater checks which of the managed Pods have correct resources set and, if not, kills them so that they can be recreated by their controllers with the updated requests.
• Admission Plugin sets the correct resource requests on new Pods (either just created or recreated by their controller due to Updater's activity).

VPA is configured with a new CRD ("Custom Resource Definition") object called VerticalPodAutoscaler. It allows to specify which pods should be vertically autoscaled as well as if/how the resource recommendations are applied.

VPA Installation

VPA is not available in Kubernetes by default, so we need to install it manually. To install VPA in your EKS Cluster clone the repo and run the vpa-up.sh script.

Among typical Kubernetes objects, such as CRDs, the process installs three new Deployments in the kube-system namespace: Recommender, Updater and a new Admission Controller.

git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler/
./hack/vpa-up.sh

You should see three new Pods running, one per VPA component:

% kubectl get pod -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
aws-node-swxfx                             2/2     Running   0          3m58s
coredns-59b9f5f4d6-lhktm                   1/1     Running   0          13m
coredns-59b9f5f4d6-pvngh                   1/1     Running   0          13m
kube-proxy-bmdmp                           1/1     Running   0          3m58s
metrics-server-6db4d75b97-gt4hh            1/1     Running   0          5m31s
vpa-admission-controller-7467db745-p5mgm   1/1     Running   0          2m36s
vpa-recommender-597b7c765d-vljms           1/1     Running   0          2m40s
vpa-updater-884d4d7d9-mrdd6                1/1     Running   0          2m42s

Upstream and Load Generation Nodes

To have better control over the Kubernetes resources we are going to deploy each component (Load Generator, Konnect Data Plane and Upstream Service) in a specific NodeGroup.

Create a new NodeGroup and install the Upstream Service

To create the NodeGroup for the Upstream Service run the following command. The c5.2xlarge Instance Type has enough resources to run the service, ensuring it doesn't become a bottleneck.

eksctl create nodegroup --cluster kong35-eks129-autoscaling --region us-west-1 \
  --name nodegroup-httpbin \
  --node-labels="nodegroupname=httpbin" \
  --node-type c5.2xlarge \
  --nodes 1 \
  --nodes-min 1 --nodes-max 10 \
  --max-pods-per-node 50

Now, install the Upstream Application. Note we are deploying 5 replicas of the application to provide a better performance. We also refer to the new NodeGroup with the nodeSelector configuration.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 5
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
      - name: httpbin
        image: kong/httpbin
        ports:
        - containerPort: 80
      nodeSelector:
        nodegroupname: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  type: ClusterIP
  ports:
    - port: 80
      protocol: TCP
  selector:
    app: httpbin
EOF

You can check the application exposing its Service:

kubectl port-forward service/httpbin 8080:80

Consume the Data Plane with the LoadGenerator

As you can see the specific NodeGroup for the Load Generator is quite similar to the Upstream Service:

eksctl create nodegroup --cluster kong35-eks129-autoscaling --region us-west-1 \
  --name nodegroup-fortio \
  --node-labels="nodegroupname=fortio" \
  --node-type c5.2xlarge \
  --nodes 1 \
  --nodes-min 1 --nodes-max 10 \
  --max-pods-per-node 50

Now, let's consume the Data Plane. The Load Generator deployment is based on the Fortio load testing tool. Fortio is particularly interesting because it can be run at a fixed Query per Second (QPS) rate. For example, here is a Fortio deployment to send requests to the Data Plane. Fortio is configured to keep, for 3 minutes, a 1000 QPS rate across 800 parallel connections.

Also, note Fortio is consuming the Data Plane Kubernetes Service's FQDN:

kong-kong-proxy.kong.svc.cluster.local

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name:  fortio
  labels:
    app: fortio
spec:
  containers:
  - name: fortio
    image: fortio/fortio
    args: ["load", "-c", "800", "-qps", "1000", "-t", "3m", "-allow-initial-errors", "http://kong-kong-proxy.kong.svc.cluster.local:80/route1/get"]
  nodeSelector:
      nodegroupname: fortio
EOF

You can check Fortio's output with:

kubectl logs -f fortio

Eventually, Fortio presents a summary of the load test. The main data are:

• The P99 latency: for example, # target 99% 1.81244
• The number of requests sent along with the QPS: All done 52007 calls (plus 800 warmup) 2775.391 ms avg, 287.6 qps

You can stop Fortio, simply deleting its Pod:

kubectl delete pod fortio

As you can see, the single Konnect Data Plane instance was not able to handle a 1000 QPS rate as requested. The main reason for this is that its Pod is restricted to consume up to 0.2 units of CPU and 300Mi for memory as set in the resources section of the Data Plane deployment declaration.

That's a nice opportunity to put VPA to run and see what it recommends.

VPA Policy

The VPA policy is described using the VertificalPodAutoscaler CRD. For a basic test, declare a policy like this:

cat <<EOF | kubectl apply -f -
apiVersion: "autoscaling.k8s.io/v1"
kind: VerticalPodAutoscaler
metadata:
  name: kong-vpa
  namespace: kong
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: kong-kong
  updatePolicy:
    updateMode: "Auto"
    minReplicas: 1
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        minAllowed:
          cpu: "0.2"
          memory: "300Mi"
        maxAllowed:
          cpu: "2"
          memory: "2Gi"
        controlledResources: ["cpu", "memory"]
EOF

A couple of comments here:

• The updateMode, configured as Auto, requests VPA to assign resources on Pod creation and updatesate mechanism. Check the documentation to learn more.
• The default behavior of VPA Updater is to allow Pod eviction only if there are at least 2 live replicas, in order to avoid temporary total unavailability of a workload under VPA in Auto mode. This has been changed with minReplicas configuration (of course, this is not recommended for production environments).
• minAllowed and maxAllowed configs set the range of resources (CPU and memory) to be considered.
  

Consume the Data Plane again

Let's run the same load test now and see what happens.

If you check the Updater with, for example, kubectl logs -f vpa-updater-884d4d7d9-mrdd6, you should see messages saying it has evicted and recreated the Pod with new resource configurations:

I0124 20:13:47.953035       1 pods_eviction_restriction.go:219] overriding minReplicas from global 2 to per-VPA 1 for VPA kong/kong-vpa
I0124 20:13:47.953082       1 update_priority_calculator.go:143] pod accepted for update kong/kong-kong-84964cbc8d-6sbc6 with priority 1.8132136917114257
I0124 20:13:47.953099       1 updater.go:215] evicting pod kong-kong-84964cbc8d-6sbc6
I0124 20:13:47.990898       1 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kong", Name:"kong-kong-84964cbc8d-6sbc6", UID:"04baab83-9d75-4f14-8887-cb79c88e8cfd", APIVersion:"v1", ResourceVersion:"2350", FieldPath:""}): type: 'Normal' reason: 'EvictedByVPA' Pod was evicted by VPA Updater to apply resource recommendation.
I0124 20:14:06.345672       1 reflector.go:559] k8s.io/client-go/informers/factory.go:150: Watch close - *v1.LimitRange total 7 items received

This is due to the new recommendations provided by the Recommender. For example, kubectl logs -f vpa-recommender-597b7c765d-vljms, should show messages like these:

I0124 20:12:49.047281       1 recommender.go:155] Recommender Run
I0124 20:12:49.047315       1 cluster_feeder.go:317] Start selecting the vpaCRDs.
I0124 20:12:49.047320       1 cluster_feeder.go:352] Fetched 1 VPAs.
I0124 20:12:49.047384       1 cluster_feeder.go:362] Using selector app.kubernetes.io/component=app,app.kubernetes.io/instance=kong,app.kubernetes.io/name=kong for VPA kong/kong-vpa
I0124 20:12:49.054707       1 metrics_client.go:74] 15 podMetrics retrieved for all namespaces
I0124 20:12:49.054820       1 cluster_feeder.go:440] ClusterSpec fed with #36 ContainerUsageSamples for #18 containers. Dropped #0 samples.
I0124 20:12:49.054835       1 recommender.go:165] ClusterState is tracking 15 PodStates and 1 VPAs
I0124 20:12:49.101647       1 checkpoint_writer.go:114] Saved VPA kong/kong-vpa checkpoint for proxy
I0124 20:12:49.101676       1 recommender.go:175] ClusterState is tracking 11 aggregated container states

As usual, Fortio should report the test results. In my case, I got the following:

• The P99 latency: target 99% 1.02964
• The number of requests sent along with the QPS: All done 180000 calls (plus 800 warmup) 650.192 ms avg, 993.4 qps

As we can see, now the DP was able to handle the QPS we requested with a much better latency time.

Let's check the VPA policy now:

% kubectl get vpa kong-vpa -n kong
NAME       MODE   CPU    MEM         PROVIDED   AGE
kong-vpa   Auto   548m   548861636   True       6m17s

You can get a more detailed view with the following command. The output shows that VPA generates some recommendations. Target is the actual recommendation as Upper and Lower bounds are indicators of the recommendation.

% kubectl get vpa kong-vpa -n kong -o json | jq ".status.recommendation"
{
  "containerRecommendations": [
    {
      "containerName": "proxy",
      "lowerBound": {
        "cpu": "200m",
        "memory": "300Mi"
      },
      "target": {
        "cpu": "548m",
        "memory": "548861636"
      },
      "uncappedTarget": {
        "cpu": "548m",
        "memory": "548861636"
      },
      "upperBound": {
        "cpu": "2",
        "memory": "2Gi"
      }
    }
  ]
}

Most importantly, we should be able to see the VPA effects over the Data Plane Pod. Note that VPA recommendation keeps the proportionality of the resource requests and limits.

% kubectl get pod -n kong -o json | jq ".items[].spec.containers[].resources"
{
  "limits": {
    "cpu": "1096m",
    "memory": "823292454"
  },
  "requests": {
    "cpu": "548m",
    "memory": "548861636"
  }
}

In general, VPA is not expected to provide fast actions. In fact, it is more useful to get general trends. For quick reactions, Horizontal Pod Autoscaler (HPA) is a better approach. Check the next post in this series to see Konnect Data Plane working along HPA.