Kong Konnect Data Plane Node Autoscaling with Cluster Autoscaler on Amazon EKS 1.29
After getting our Konnect Data Planes vertically and horizontally scaled, with VPA and HPA, it's time to explore the Kubernete Node Autoscaler options. In this post, we start with the Cluster Autoscaler mechanism. (Part 4 in this series is dedicated to Karpenter.)
Cluster Autoscaler
The Kubernetes Cluster Autoscaler documentation describes it as a tool that automatically adjusts the size of the Kubernetes Cluster when one of the following conditions is true:
- There are pods that failed to run in the cluster due to insufficient resources.
- There are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.
The following Cluster Autoscaler diagram was taken from the AWS Best Practices Guides portal.
![](https://prd-mktg-konghq-com.imgix.net/images/2024/02/65c507d3-image1-6.png?auto=format&fit=max&w=2560)
Amazon EKS Cluster Autoscaler
The Amazon EKS Cluster Autoscaler implementation relies on the EC2's Autoscaling Group capability to manage NodeGroups. That is the reason why we used the eksctl
--asg-access
parameter when we created our EKS Cluster. Check the eksctl autoscaling page to know more.
In fact, you can check all ASGs your EKS Cluster has in place with the following command, which asks for the regular MinSize, MaxSize, and DesiredCapacity values.
aws autoscaling \
describe-auto-scaling-groups \
--region us-west-1 \
--query "AutoScalingGroups[? Tags[? (Key=='eks:cluster-name') && Value=='kong35-eks129-autoscaling']].[AutoScalingGroupName, MinSize, MaxSize, DesiredCapacity]" \
--output table
The expected output should look like this. As shown, each NodeGroup has its own ASG defined. Of course, we are interested in the kong
NodeGroup.
--------------------------------------------------------------------------------
| DescribeAutoScalingGroups |
+--------------------------------------------------------------+----+-----+----+
| eks-nodegroup-fortio-f6c6a3b5-d248-111b-e890-cb5114697cae | 1 | 10 | 1 |
| eks-nodegroup-httpbin-72c6a3b5-c047-c19d-fde0-2646caa270df | 1 | 10 | 1 |
| eks-nodegroup-kong-3ac6a3c8-86de-d97c-42e2-0d788168cbbb | 1 | 10 | 1 |
+--------------------------------------------------------------+----+-----+----+
EKS Cluster Autoscaler and IRSA (IAM Roles for Service Accounts)
EKS Cluster Autoscaler is deployed and it runs as a Kubernetes Deployment. As such, it's recommended to use IRSA (IAM Roles for Service Accounts) to associate the Service Account that the Cluster Autoscaler Deployment runs as with an IAM role that is able to perform these functions.
IAM OIDC provider
IRSA requires the EKS Cluster to be associated with an IAM OpenID Connect Provider. This can be done with a specific eksctl
command:
eksctl utils associate-iam-oidc-provider --cluster kong35-eks129-autoscaling --region=us-west-1 --approve
You can check the IAM OIDC provider with the following command. You should see your EKS Cluster OIDC Issuer endpoint listed as an ARN.
aws iam list-open-id-connect-providers
If you will, you can check the OIDC Issuer endpoint with:
aws eks describe-cluster --name kong35-eks129-autoscaling --region us-west-1 | jq -r ".cluster.identity"
IAM Policy
IRSA is based on a Kubernetes Service Account and IAM Role pair. In turn, the Kubernetes Deployment should refer to the Service Account, which has the IAM Role set as an Annotation. Finally, the IAM Role allows the Deployment to access AWS Services, including, in our case, Auto Scaling Group (ASG) services.
Here is an IAM Policy example that allows Roles to access the ASG Services:
aws iam create-policy \
--policy-name AmazonEKSClusterAutoscalerPolicy \
--policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeScalingActivities",
"autoscaling:DescribeTags",
"ec2:DescribeInstanceTypes",
"ec2:DescribeLaunchTemplateVersions"
],
"Resource": ["*"]
},
{
"Effect": "Allow",
"Action": [
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup",
"ec2:DescribeImages",
"ec2:GetInstanceTypesFromInstanceRequirements",
"eks:DescribeNodegroup"
],
"Resource": ["*"]
}
]
}'
Service Account and IAM Role
With the IAM Policy created we can run another eksctl
command to create both Kubernetes Service Account and IAM Role:
eksctl create iamserviceaccount \
--name cluster-autoscaler \
--namespace kube-system \
--cluster kong35-eks129-autoscaling \
--region us-west-1 \
--approve \
--role-name kong35-eks129-autoscaling-role \
--override-existing-serviceaccounts \
--attach-policy-arn $(aws iam list-policies --query 'Policies[?PolicyName==`AmazonEKSClusterAutoscalerPolicy`].Arn' --output text)
If you check the Service Account, you will see it has the required annotation referring to the Role created by the iamserviceaccount
command:
kubectl describe sa cluster-autoscaler -n kube-system
You can check the Role also.
aws iam get-role --role kong35-eks129-autoscaling-role
As expected the Role refers to our Policy:
aws iam list-attached-role-policies --role kong35-eks129-autoscaling-role
Deploy Cluster Autoscaler with Helm
Now we are ready to install Cluster Autoscaler in our EKS Cluster. This can be done using its Helm Charts. Note the command refers to the Service Account and IAM Role created previously.
helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm repo update
helm install cluster-autoscaler autoscaler/cluster-autoscaler \
-n kube-system \
--set autoDiscovery.clusterName=kong35-eks129-autoscaling \
--set cloudProvider=aws \
--set awsRegion=us-west-1 \
--set image.tag=v1.29.0 \
--set rbac.serviceAccount.create=false \
--set rbac.serviceAccount.name=cluster-autoscaler \
--set rbac.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::<your_aws_account>:role/kong35-eks129-autoscaling-role
You can check the Cluster Autoscaler log files to see how the installation went:
kubectl logs -f -l app.kubernetes.io/name=aws-cluster-autoscaler -n kube-system
Check Cluster Autoscaler at work
With the existing Load Generator still running (if you don't have it, please start one), you should see a new status for both ASG and the Konnect Data Plane Kubernetes Deployment:
aws autoscaling \
describe-auto-scaling-groups \
--region us-west-1 \
--query "AutoScalingGroups[? Tags[? (Key=='eks:cluster-name') && Value=='kong35-eks129-autoscaling']].[AutoScalingGroupName, MinSize, MaxSize, DesiredCapacity]" \
--output table
--------------------------------------------------------------------------------
| DescribeAutoScalingGroups |
+--------------------------------------------------------------+----+-----+----+
| eks-nodegroup-fortio-f6c6a3b5-d248-111b-e890-cb5114697cae | 1 | 10 | 1 |
| eks-nodegroup-httpbin-72c6a3b5-c047-c19d-fde0-2646caa270df | 1 | 10 | 1 |
| eks-nodegroup-kong-3ac6a3c8-86de-d97c-42e2-0d788168cbbb | 1 | 10 | 2 |
+--------------------------------------------------------------+----+-----+----+
ASG has requested a new instance of the Node:
% kubectl top node --selector='eks.amazonaws.com/nodegroup=nodegroup-kong'
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
ip-192-168-11-188.us-west-1.compute.internal 2000m 103% 6641Mi 93%
ip-192-168-53-177.us-west-1.compute.internal 54m 2% 1256Mi 17%
And then, the Pending Pods, after getting scheduled, are finally running:
% kubectl get pod -n kong
NAME READY STATUS RESTARTS AGE
kong-kong-789ffd6f86-bmgh8 1/1 Running 0 15m
kong-kong-789ffd6f86-fcnts 1/1 Running 0 15m
kong-kong-789ffd6f86-k7cnm 1/1 Running 0 16m
kong-kong-789ffd6f86-nsqnn 1/1 Running 0 5m14s
kong-kong-789ffd6f86-sf65t 1/1 Running 0 7m10s
kong-kong-789ffd6f86-tv96x 1/1 Running 0 22m
kong-kong-789ffd6f86-twzd8 1/1 Running 0 16m
Submit the Konnect Data Plane to an even higher throughput
So, what happens if we submit our Konnect Data Plane Deployment to a longer and higher throughput? For instance, we change the HPA policy like this, allowing up to 20 replicas to run:
cat <<EOF | kubectl apply -f -
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: kong-hpa
namespace: kong
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: kong-kong
minReplicas: 1
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 75
EOF
Also, we can submit a heavier load test with:
kubectl delete pod fortio
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: fortio
labels:
app: fortio
spec:
containers:
- name: fortio
image: fortio/fortio
args: ["load", "-c", "800", "-qps", "5000", "-t", "60m", "-allow-initial-errors", "http://kong-kong-proxy.kong.svc.cluster.local:80/route1/get"]
nodeSelector:
nodegroupname: fortio
The expected result is to have extra Nodes getting created along the way. Here are the HPA, ASG and Kubernetes Nodes outputs:
% kubectl get hpa -n kong
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
kong-hpa Deployment/kong-kong 73%/75% 1 20 14 35m
aws autoscaling \
describe-auto-scaling-groups \
--region us-west-1 \
--query "AutoScalingGroups[? Tags[? (Key=='eks:cluster-name') && Value=='kong35-eks129-autoscaling']].[AutoScalingGroupName, MinSize, MaxSize, DesiredCapacity]" \
--output table
--------------------------------------------------------------------------------
| DescribeAutoScalingGroups |
+--------------------------------------------------------------+----+-----+----+
| eks-nodegroup-fortio-f6c6a3b5-d248-111b-e890-cb5114697cae | 1 | 10 | 1 |
| eks-nodegroup-httpbin-72c6a3b5-c047-c19d-fde0-2646caa270df | 1 | 10 | 1 |
| eks-nodegroup-kong-3ac6a3c8-86de-d97c-42e2-0d788168cbbb | 1 | 10 | 4 |
+--------------------------------------------------------------+----+-----+----+
% kubectl top node --selector='eks.amazonaws.com/nodegroup=nodegroup-kong'
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
ip-192-168-11-188.us-west-1.compute.internal 1250m 64% 5870Mi 82%
ip-192-168-28-192.us-west-1.compute.internal 1254m 64% 6366Mi 89%
ip-192-168-48-1.us-west-1.compute.internal 839m 43% 670Mi 9%
ip-192-168-53-177.us-west-1.compute.internal 1212m 62% 6268Mi 88%
The problem is that all Nodes are based on the same original t3.large
Instance Type. You can check them out with:
% aws autoscaling describe-auto-scaling-groups --region us-west-1 | jq '.AutoScalingGroups[] | select (.Tags[0].Value == "kong35-eks129-autoscaling") | { AutoScalingGroupName }' | jq -r '.AutoScalingGroupName'
eks-nodegroup-fortio-f6c6a3b5-d248-111b-e890-cb5114697cae
eks-nodegroup-httpbin-72c6a3b5-c047-c19d-fde0-2646caa270df
eks-nodegroup-kong-3ac6a3c8-86de-d97c-42e2-0d788168cbbb
% aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name eks-nodegroup-kong-3ac6a3c8-86de-d97c-42e2-0d788168cbbb --region us-west-1 | jq ".AutoScalingGroups[].Instances[].InstanceType"
"t3.large"
"t3.large"
"t3.large"
"t3.large"
We could address that by deploying multiple instances of Cluster Autoscaler, each configured to work with a specific set of Node Groups. However, that's not recommended, as Cluster Autoscaler was not designed for this kind of configuration and it could lead to a situation where multiple Autoscaler would attempt to schedule a Pod. Please, check the EKS Best Practices Guide for Cluster Autoscaler provided by AWS.
That's one of the main reasons to switch to a more flexible and smarter cluster autoscaler as Karpenter implements. Let's check it out in the next (and final) part of this series, Kong Konnect Data Plane Node Autoscaling with Karpenter on Amazon EKS 1.29.