WHY GARTNER’S “CONTEXT MESH” CHANGES EVERYTHING AI CONNECTIVITY: THE ROAD AHEAD DON’T MISS API + AI SUMMIT 2026 SEPT 30 – OCT 1
We're Entering the Age of AI Connectivity [Read more](/blog/news/the-age-of-ai-connectivity)Read moreProducts & Agents:
[Engineering](/blog/engineering)Engineering
February 19, 2024
4 min read

# Kong Konnect Data Plane Node Autoscaling with Cluster Autoscaler on Amazon EKS 1.29

Claudio Acquaviva
Principal Architect, Kong

After getting our Konnect Data Planes vertically and horizontally scaled, with [VPA](https://konghq.com/blog/engineering/data-plane-elasticity-eks-1-29-pod-autoscaling-with-vpa)VPA and [HPA](https://konghq.com/blog/engineering/pod-autoscaling-with-hpa-on-eks-1-29)HPA, it's time to explore the Kubernete Node Autoscaler options. In this post, we start with the Cluster Autoscaler mechanism. (Part 4 in this series is dedicated to Karpenter.)

## Cluster Autoscaler

The [Kubernetes Cluster Autoscaler documentation](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/README.md)Kubernetes Cluster Autoscaler documentation describes it as a tool that automatically adjusts the size of the Kubernetes Cluster when one of the following conditions is true:

  • - There are pods that failed to run in the cluster due to insufficient resources.
  • - There are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.

The following Cluster Autoscaler diagram was taken from the [AWS Best Practices Guides](https://aws.github.io/aws-eks-best-practices/cluster-autoscaling/)AWS Best Practices Guides portal.

## Amazon EKS Cluster Autoscaler

The [Amazon EKS Cluster Autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md)Amazon EKS Cluster Autoscaler implementation relies on the EC2's Autoscaling Group capability to manage NodeGroups. That is the reason why we used the [eksctl](https://eksctl.io)eksctl --asg-access parameter when we created our EKS Cluster. Check the [eksctl autoscaling](https://eksctl.io/usage/autoscaling/)eksctl autoscaling page to know more.

In fact, you can check all ASGs your EKS Cluster has in place with the following command, which asks for the regular MinSize, MaxSize, and DesiredCapacity values.

aws autoscaling \
    describe-auto-scaling-groups \
    --region us-west-1 \
    --query "AutoScalingGroups[? Tags[? (Key=='eks:cluster-name') && Value=='kong35-eks129-autoscaling']].[AutoScalingGroupName, MinSize, MaxSize, DesiredCapacity]" \
    --output table

The expected output should look like this. As shown, each NodeGroup has its own ASG defined. Of course, we are interested in the kong NodeGroup.

--------------------------------------------------------------------------------
|                           DescribeAutoScalingGroups                          |
+--------------------------------------------------------------+----+-----+----+
|  eks-nodegroup-fortio-f6c6a3b5-d248-111b-e890-cb5114697cae   |  1 |  10 |  1 |
|  eks-nodegroup-httpbin-72c6a3b5-c047-c19d-fde0-2646caa270df  |  1 |  10 |  1 |
|  eks-nodegroup-kong-3ac6a3c8-86de-d97c-42e2-0d788168cbbb     |  1 |  10 |  1 |
+--------------------------------------------------------------+----+-----+----+

### EKS Cluster Autoscaler and IRSA (IAM Roles for Service Accounts)

EKS Cluster Autoscaler is deployed and it runs as a Kubernetes Deployment. As such, it's recommended to use [IRSA (IAM Roles for Service Accounts)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html)IRSA (IAM Roles for Service Accounts) to associate the Service Account that the Cluster Autoscaler Deployment runs as with an IAM role that is able to perform these functions.

#### IAM OIDC provider

IRSA requires the EKS Cluster to be associated with an IAM OpenID Connect Provider. This can be done with a specific eksctl command:

eksctl utils associate-iam-oidc-provider --cluster kong35-eks129-autoscaling --region=us-west-1 --approve

You can check the IAM OIDC provider with the following command. You should see your EKS Cluster OIDC Issuer endpoint listed as an ARN.

aws iam list-open-id-connect-providers

If you will, you can check the OIDC Issuer endpoint with:

aws eks describe-cluster --name kong35-eks129-autoscaling --region us-west-1 | jq -r ".cluster.identity"

#### IAM Policy

IRSA is based on a Kubernetes Service Account and IAM Role pair. In turn, the Kubernetes Deployment should refer to the Service Account, which has the IAM Role set as an Annotation. Finally, the IAM Role allows the Deployment to access AWS Services, including, in our case, Auto Scaling Group (ASG) services.

Here is an IAM Policy example that allows Roles to access the ASG Services:

aws iam create-policy \
  --policy-name AmazonEKSClusterAutoscalerPolicy \
  --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeAutoScalingInstances",
        "autoscaling:DescribeLaunchConfigurations",
        "autoscaling:DescribeScalingActivities",
        "autoscaling:DescribeTags",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeLaunchTemplateVersions"
      ],
      "Resource": ["*"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:SetDesiredCapacity",
        "autoscaling:TerminateInstanceInAutoScalingGroup",
        "ec2:DescribeImages",
        "ec2:GetInstanceTypesFromInstanceRequirements",
        "eks:DescribeNodegroup"
      ],
      "Resource": ["*"]
    }
  ]
}'

#### Service Account and IAM Role

With the IAM Policy created we can run another eksctl command to create both Kubernetes Service Account and IAM Role:

eksctl create iamserviceaccount \
  --name cluster-autoscaler \
  --namespace kube-system \
  --cluster kong35-eks129-autoscaling \
  --region us-west-1 \
  --approve \
  --role-name kong35-eks129-autoscaling-role \
  --override-existing-serviceaccounts \
  --attach-policy-arn $(aws iam list-policies --query 'Policies[?PolicyName==`AmazonEKSClusterAutoscalerPolicy`].Arn' --output text)

If you check the Service Account, you will see it has the required annotation referring to the Role created by the iamserviceaccount command:

kubectl describe sa cluster-autoscaler -n kube-system

You can check the Role also.

aws iam get-role --role kong35-eks129-autoscaling-role

As expected the Role refers to our Policy:

aws iam list-attached-role-policies --role kong35-eks129-autoscaling-role

## Deploy Cluster Autoscaler with Helm

Now we are ready to install Cluster Autoscaler in our EKS Cluster. This can be done using its [Helm Charts](https://artifacthub.io/packages/helm/cluster-autoscaler/cluster-autoscaler)Helm Charts. Note the command refers to the Service Account and IAM Role created previously.

helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm repo update

helm install cluster-autoscaler autoscaler/cluster-autoscaler \
  -n kube-system \
  --set autoDiscovery.clusterName=kong35-eks129-autoscaling \
  --set cloudProvider=aws \
  --set awsRegion=us-west-1 \
  --set image.tag=v1.29.0 \
  --set rbac.serviceAccount.create=false \
  --set rbac.serviceAccount.name=cluster-autoscaler \
  --set rbac.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::<your_aws_account>:role/kong35-eks129-autoscaling-role

You can check the Cluster Autoscaler log files to see how the installation went:

kubectl logs -f -l app.kubernetes.io/name=aws-cluster-autoscaler -n kube-system

### Check Cluster Autoscaler at work

With the existing Load Generator still running (if you don't have it, please start one), you should see a new status for both ASG and the Konnect Data Plane Kubernetes Deployment:

aws autoscaling \
    describe-auto-scaling-groups \
    --region us-west-1 \
    --query "AutoScalingGroups[? Tags[? (Key=='eks:cluster-name') && Value=='kong35-eks129-autoscaling']].[AutoScalingGroupName, MinSize, MaxSize, DesiredCapacity]" \
    --output table
--------------------------------------------------------------------------------
|                           DescribeAutoScalingGroups                          |
+--------------------------------------------------------------+----+-----+----+
|  eks-nodegroup-fortio-f6c6a3b5-d248-111b-e890-cb5114697cae   |  1 |  10 |  1 |
|  eks-nodegroup-httpbin-72c6a3b5-c047-c19d-fde0-2646caa270df  |  1 |  10 |  1 |
|  eks-nodegroup-kong-3ac6a3c8-86de-d97c-42e2-0d788168cbbb     |  1 |  10 |  2 |
+--------------------------------------------------------------+----+-----+----+

ASG has requested a new instance of the Node:

% kubectl top node --selector='eks.amazonaws.com/nodegroup=nodegroup-kong'
NAME                                           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-192-168-11-188.us-west-1.compute.internal   2000m        103%   6641Mi          93%       
ip-192-168-53-177.us-west-1.compute.internal   54m          2%     1256Mi          17%

And then, the Pending Pods, after getting scheduled, are finally running:

% kubectl get pod -n kong
NAME                         READY   STATUS    RESTARTS   AGE
kong-kong-789ffd6f86-bmgh8   1/1     Running   0          15m
kong-kong-789ffd6f86-fcnts   1/1     Running   0          15m
kong-kong-789ffd6f86-k7cnm   1/1     Running   0          16m
kong-kong-789ffd6f86-nsqnn   1/1     Running   0          5m14s
kong-kong-789ffd6f86-sf65t   1/1     Running   0          7m10s
kong-kong-789ffd6f86-tv96x   1/1     Running   0          22m
kong-kong-789ffd6f86-twzd8   1/1     Running   0          16m

## Submit the Konnect Data Plane to an even higher throughput

So, what happens if we submit our Konnect Data Plane Deployment to a longer and higher throughput? For instance, we change the HPA policy like this, allowing up to 20 replicas to run:

cat <<EOF | kubectl apply -f -
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: kong-hpa
  namespace: kong
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: kong-kong
  minReplicas: 1
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 75
EOF

Also, we can submit a heavier load test with:

kubectl delete pod fortio

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name:  fortio
  labels:
    app: fortio
spec:
  containers:
  - name: fortio
    image: fortio/fortio
    args: ["load", "-c", "800", "-qps", "5000", "-t", "60m", "-allow-initial-errors", "http://kong-kong-proxy.kong.svc.cluster.local:80/route1/get"]
  nodeSelector:
      nodegroupname: fortio

The expected result is to have extra Nodes getting created along the way. Here are the HPA, ASG and Kubernetes Nodes outputs:

% kubectl get hpa -n kong
NAME       REFERENCE              TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
kong-hpa   Deployment/kong-kong   73%/75%   1         20        14         35m

aws autoscaling \
    describe-auto-scaling-groups \
    --region us-west-1 \
    --query "AutoScalingGroups[? Tags[? (Key=='eks:cluster-name') && Value=='kong35-eks129-autoscaling']].[AutoScalingGroupName, MinSize, MaxSize, DesiredCapacity]" \
    --output table
--------------------------------------------------------------------------------
|                           DescribeAutoScalingGroups                          |
+--------------------------------------------------------------+----+-----+----+
|  eks-nodegroup-fortio-f6c6a3b5-d248-111b-e890-cb5114697cae   |  1 |  10 |  1 |
|  eks-nodegroup-httpbin-72c6a3b5-c047-c19d-fde0-2646caa270df  |  1 |  10 |  1 |
|  eks-nodegroup-kong-3ac6a3c8-86de-d97c-42e2-0d788168cbbb     |  1 |  10 |  4 |
+--------------------------------------------------------------+----+-----+----+


% kubectl top node --selector='eks.amazonaws.com/nodegroup=nodegroup-kong'
NAME                                           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-192-168-11-188.us-west-1.compute.internal   1250m        64%    5870Mi          82%       
ip-192-168-28-192.us-west-1.compute.internal   1254m        64%    6366Mi          89%       
ip-192-168-48-1.us-west-1.compute.internal     839m         43%    670Mi           9%        
ip-192-168-53-177.us-west-1.compute.internal   1212m        62%    6268Mi          88%

The problem is that all Nodes are based on the same original t3.large Instance Type. You can check them out with:

% aws autoscaling describe-auto-scaling-groups --region us-west-1 | jq '.AutoScalingGroups[] | select (.Tags[0].Value == "kong35-eks129-autoscaling") | { AutoScalingGroupName }' | jq -r '.AutoScalingGroupName'
eks-nodegroup-fortio-f6c6a3b5-d248-111b-e890-cb5114697cae
eks-nodegroup-httpbin-72c6a3b5-c047-c19d-fde0-2646caa270df
eks-nodegroup-kong-3ac6a3c8-86de-d97c-42e2-0d788168cbbb

% aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name eks-nodegroup-kong-3ac6a3c8-86de-d97c-42e2-0d788168cbbb --region us-west-1 | jq ".AutoScalingGroups[].Instances[].InstanceType"
"t3.large"
"t3.large"
"t3.large"
"t3.large"

We could address that by deploying multiple instances of Cluster Autoscaler, each configured to work with a specific set of Node Groups. However, that's not recommended, as Cluster Autoscaler was not designed for this kind of configuration and it could lead to a situation where multiple Autoscaler would attempt to schedule a Pod. Please, check the [EKS Best Practices Guide for Cluster Autoscaler](https://aws.github.io/aws-eks-best-practices/cluster-autoscaling/#sharding-across-node-groups)EKS Best Practices Guide for Cluster Autoscaler provided by AWS.

That's one of the main reasons to switch to a more flexible and smarter cluster autoscaler as Karpenter implements. Let's check it out in the next (and final) part of this series, [Kong Konnect Data Plane Node Autoscaling with Karpenter on Amazon EKS 1.29](https://konghq.com/blog/engineering/dp-node-autoscaling-with-karpenter-aws-eks-1-29)Kong Konnect Data Plane Node Autoscaling with Karpenter on Amazon EKS 1.29.

**Topics**
- [API Testing](/blog/tag/api-testing)API Testing- [AWS](/blog/tag/aws)AWS
Claudio Acquaviva
Principal Architect, Kong

Recommended posts

## Ready to see Kong in action?

Get a personalized walkthrough of Kong's platform tailored to your architecture, use cases, and scale requirements.

[Get a Demo](/contact-sales)Get a Demo