We’re excited to announce a new addition to our Kong Konnect EKS Blueprint Family: the Kong Konnect Mesh EKS Blueprint Add-on to deploy your Mesh Zones. Deploy your zones securely on AWS with the new Mesh Manager offering in Kong Konnect. It gives you an opinionated install of Kong Mesh to quickly ramp up on Kong Konnect’s Mesh Manager on Amazon EKS. Let’s dive in.

### Introduction 

Earlier this year we announced two EKS Blueprint Add-ons:

1. - A Kong Konnect EKS Blueprint Add-on for Kong Gateway data planes 
2. - A Kong Konnect EKS Blueprint Add-on for Kong Ingress Controller 

(You can check out our [quickstart](https://konghq.com/blog/engineering/eks-terraform-blueprints-addons)quickstart for more on these add-ons.)

Today, we're releasing a new add-on into the Kong Konnect fold: **the Kong Konnect EKS Blueprint Add-on for Mesh Zones.**

## Using the Mesh Add-on on Amazon EKS

The purpose of the add-on is to provide an express lane for deploying and leveraging Kong Konnect’s Mesh Manager / Kong Mesh offering on EKS.

It provides an opinionated install of the Mesh Zones to quickly ramp up on the technology on Amazon EKS, while seamlessly integrating with AWS Secrets Manager and leveraging Terraform as the Infrastructure as Code best practices.

## A quick refresher on Mesh Manager and zones

Before diving into what the add-on does, it's important to understand the architecture pattern of Kong Mesh and the Mesh Manager offering in Kong Konnect. 

Kong Mesh within Konnect’s Mesh Manager leverages the multi-zone deployment topology. With this model, within Mesh Manager, customers are allowed to deploy global control planes (global CP). In turn, zones are deployed into infrastructure such as EKS, ECS, or even EC2. The zones act as a remote control plane receiving their configuration from a global control plane that resides in Mesh Manager. The zone control planes will proxy configuration from the global control plane down to data planes (Kong Mesh side cars) and vice versa. 

*Figure 1. High-level Kong Mesh Zone Architecture*

This is a bit of an oversimplification. Zones are not just proxies between the global control plane and data planes. The goal of Kong Mesh is **automatic connectivity regardless of physical infrastructure barriers**. 

With zones in place, engineers can easily construct service meshes that can span across the zones, essentially building a hybrid service mesh. The outcome of this is interconnectivity of microservices across those zones, interoperating as if they were in one network, none the wiser.

## How does the Kong Konnect Mesh Zone Add-on work?

The Kong Konnect Add-ons play a crucial role in seamlessly integrating Konnect deployments with AWS Secrets Manager via External Secrets Operator. Our add-ons are purpose-built to conform to the [AWS Well-Architected Framework ](https://docs.aws.amazon.com/wellarchitected/2022-03-31/framework/sec-design.html)AWS Well-Architected Framework security principles around least privilege.

From a deployment perspective, three key attributes are required to get a zone up and running:

1. - **Kong Konnect region **— Kong Konnect has multi-geo support in US, EU, and AU 
2. - **Global CP ID** — because we can have multiple global CPs within Mesh Manager 
3. - **Zone JWT Token** — this proves the identity of the zone to the global CP

### AWS Secrets Manager

In the context of our add-ons, there's a necessity to securely store and closely manage access to elements like zone JWT tokens or the data plane certificates required by the Kong Gateway Add-on.

AWS Secrets Manager extends beyond mere access control functionalities, making it indispensable for any production-ready workload, with automatic secret rotation, encryption in transit and at rest, programmatic access via APIs, and auditing features.

However, AWS Secrets Manager itself doesn't directly interface with Kubernetes to deposit secrets. That’s the role of the External Secrets Operator.

### External Secrets Operator

The External Secrets Operator (ESO) reads and automatically injects AWS Secrets into Kubernetes clusters while ensuring fine-grain access of AWS secrets to specific service accounts.

Again, in the context of the Kong Konnect Add-ons, this integration is purpose-built for retrieving secrets for Kong — whether it be the zone JWT token or data plane certificates.

### Bringing it together

Configuring the External Secrets Operator to interface with AWS Secrets Manager requires significant prerequisite knowledge of AWS IRSA, IAM Policies, as well as the External Secrets Operator configuration itself, and how to apply it to Kong-related assets.

The add-ons streamline this process for you. Specifically in the context of the Mesh Add-on, the primary concern is hosting and retrieving the zone JWT token.

At a high level, the add-on abstracts away the three major activities (Figure 2):

1. -

   Configuring AWS IRSA and IAM policies with the necessary access to the AWS secrets.

2. -

   Installing the [External Secrets Operator](https://external-secrets.io/latest/)External Secrets Operator and configuring all needed CRDs to properly deposit the zone token into EKS.

3. -

   Finally, deploying the Mesh zone into the EKS cluster, with the configuration necessary to integrate with a global CP residing in Mesh Manager. 

Under the hood, the Kong Mesh Helm chart is deployed by the AWS EKS Add-on framework with the AWS-specific configuration abstracted away. 

All that's required is a small subset of values to understand where your Konnect Mesh Manager global CP is located, and how to locate the AWS secret.

*Figure 2. Mesh Zone Add-on *

So, I've thrown a lot of information at you. Let’s cut to a demo to understand how to run the add-on.

## Let’s do a demo

There are three phases we need to run through.

### Step 1: Prerequisites

There are three prerequisites that need to be in place: 

1. -

   Create or have an existing global CP within Konnect’s Mesh Manager

2. -

   Within the global CP, create the zone name and respective zone JWT token 

3. -

   Host the JWT token in AWS Secrets Manager (in the same region as your EKS cluster)

To get you started, we've extended the [kong-konnect-runtime-instance-cert-generator](https://github.com/aws-samples/kong-konnect-runtime-cert-generator/)kong-konnect-runtime-instance-cert-generator CLI tool. The CLI tool requires you to be logged into the AWS via the CLI and defaulted to the AWS Region where the EKS Cluster is deployed.

Execute the CLI cmd like below to complete the prerequisites:

./kong-konnect-runtime-cert-generator mesh-manager create-zone --control-plane-name=eks-blueprint --zone-name=zone-one --personal-access-token=<YOUR_PAT>

The expected output should look similar to the following:

Global Control Plane created
Zone created
{
  "global_control_plane_name": "eks-blueprint",
  "Global_control_plane_id": "4e296a13-7bd8-4514-af64-19e7930434df",
  "Zone_name": "zone-one",
  "Zone_token_secret_name": "4e296a13-7bd8-4514-af64-19e7930434df-zone-one"
}

Save all these attributes for inputs to the terraform script in the next step.

One last item: navigate to AWS Secrets Manager to validate you can locate the zone token secret.

### Step 2: Build the Terraform Script

Here we're following the example found in [add-on git repo](https://github.com/Kong/terraform-aws-eks-blueprint-konnect-kmesh-zone/tree/main/examples)add-on git repo. 

There are numerous ways to slice and dice this, so we’ll give an abridged version here. For examples of best practices in setting up an AWS EKS cluster with the Kong add-ons, look to the [examples repo](https://github.com/aws-samples/terraform-eks-blueprints-kong-samples/blob/main/README.md)examples repo for all Kong Konnect blueprints. 

locals {

  konnect_kds_global_address = "grpcs://${var.konnect_region}.mesh.sync.konghq.com:443"

}

module "eks-blueprint-konnect-mesh-zone" {
  source = "../"
  version = "1.0.0"

  cluster_name      = var.eks_cluster_name
  cluster_endpoint  = data.aws_eks_cluster.eks.endpoint
  cluster_version   = data.aws_eks_cluster.eks.version
  oidc_provider_arn = local.oidc_provider_arn

  tags = local.tags

 # REQUIRED CONFIGURATION
  kong_config = {
    zone                                                 = var.zone_name
    cpId                                                  = var.konnect_mesh_global_cp_id
    kdsGlobalAddress                            = local.konnect_kds_global_address
    kmesh_ingress_enabled                  = true
    kmesh_egress_enabled                   = true
    kmesh_k8sServices_experimental_enabled = true
    cp_token_aws_secret_name               = var.cp_token_aws_secret_name

    add_ons = {
      enable_external_secrets = true
    }

    # OPTIONAL CONFIGURATION
    values = [templatefile("${path.module}/kong_mesh_values.yaml", {})]
  }
}

The required configuration is defined via the **kong_config**, any optional configuration can be passed as helm values file to the **values** attribute.

#konnect vars
konnect_region = "us"

#mesh manager vars
konnect_mesh_global_cp_id = "4e296a13-7bd8-4514-af64-19e7930434df"
zone_name                 = "zone-one"
cp_token_aws_secret_name  = "4e296a13-7bd8-4514-af64-19e7930434df-zone-one"

#eks cluster vars
eks_cluster_name = "df-eksctl"
aws_region       = "us-west-2"

The key takeaway from all of this is the install of a Mesh zone has been simplified down to the core attributes needed to install the zone. 

### Step 3: Deploy and validation

Now, we’ll run `terraform apply` to deploy the zone. Let's validate our setup.

kubectl get pods -n kong-mesh-system
NAME                                       READY   STATUS    RESTARTS   AGE
kong-mesh-control-plane-5b7d55cdc7-dvmlm   1/1     Running   0          69s
kong-mesh-control-plane-5b7d55cdc7-ps9gs    1/1     Running   0          69s
kong-mesh-control-plane-5b7d55cdc7-vrkk7     1/1     Running   0          69s
kong-mesh-egress-7d9bd8d767-thzj4                1/1     Running   0          69s
kong-mesh-ingress-7cd46bdddc-d7kv2              0/1     Running   0          69s

In this example, I have 3 zone pods, 1 egress, and 1 ingress pod. All are healthy.

kubectl get secret cp-token -n kong-mesh-system
NAME       TYPE     DATA   AGE
cp-token   Opaque   1      4m28s

NAME       STORE                   REFRESH INTERVAL   STATUS         READY
cp-token   kong-mesh-secretstore   1h                 SecretSynced   True

Lastly, let’s make sure everything is happy in Mesh Manager. Navigate up to your Konnect Console. 

In the Mesh Manager Overview Page, navigate to your Global Control Plane. Mine for this example is **eks-blueprint**, and you should see a list of your zones and their status. Similarly, if you view all zones, you can see more details on the zone.

### Step 4: Clean up

Clean-up is easy. We’ll execute `terraform destroy –auto-approve`:

…
module.eks-blueprint-konnect-mink.kubernetes_namespace_v1.kong[0]: Still destroying... [id=kong-mesh-system, 20s elapsed]
module.eks-blueprint-konnect-mink.kubernetes_namespace_v1.kong[0]: Still destroying... [id=kong-mesh-system, 30s elapsed]
module.eks-blueprint-konnect-mink.kubernetes_namespace_v1.kong[0]: Destruction complete after 34s

Destroy complete! Resources: 14 destroyed.

## Wrapping up

There you have it. From zero to hero in 30 minutes!

The purpose behind all of our Kong Konnect add-ons is to provide you with a quick and opinionated install of our products on Amazon EKS. 

They're here to quickly get started with AWS best practices in mind — to abstract secrets to AWS Secrets Manager and abstract away the complexity of integrating External Secrets Operator and the corresponding AWS configuration — but still deploy Kong Mesh zones quickly. 

Give the add-on a go and let us know what you think! We’re excited to have the Kong Mesh Add-on included in the family of Kong Konnect EKS Blueprint Add-ons.

## Relevant resources

• - [Terraform link ](https://registry.terraform.io/modules/Kong/eks-blueprint-konnect-kmesh-zone/aws/latest)Terraform link 
• - [AWS Examples - Kong Konnect EKS Blueprint Add-ons](https://github.com/aws-samples/terraform-eks-blueprints-kong-samples)AWS Examples - Kong Konnect EKS Blueprint Add-ons
• - [Runtime CLI](https://github.com/aws-samples/kong-konnect-runtime-cert-generator)Runtime CLI