New Kong Settings for Service Mesh
This post is the last in a three-part series about deploying Kong as a service mesh. The first post discussed how we define the term “service mesh” here at Kong, and the second post explained the architectural pattern used to deploy Kong as a mesh. In this post, we will talk about the new features and configuration options we added to give Kong its mesh capabilities.
Deploying Kong as a service mesh requires Kong nodes to understand which service is local to them, establish mutual Transport Layer Security (mTLS) with each other, understand the original destination for requests and know on which end of a transaction plugins should run. We added a number of new configurations and features to Kong 1.0 and 0.15 that enable these capabilities. They include:
- TCP proxying
- <code>transparent</code> and <code>iptables</code>
- A Kong Certificate Authority (CA)
Kong can now proxy Transport Control Protocol (TCP) traffic in addition to HTTP traffic. This capability was added to support mTLS. The TLS handshake involves creating a TCP streaming session and assigning a key to that session that encrypts traffic for as long as the session lasts. Although Kong uses mTLS for service mesh deployments, Kong users can now take advantages of added TCP support, no matter where their Kong nodes are deployed.
The new configuration setting <code>origins</code> tells a Kong node which instance of a service it should proxy outbound traffic to. In a mesh, it causes a given Kong node to route the traffic it receives to its local service instance rather than back over the network to a non-local service instance. Specifically, <code>origins</code> overrides outbound routing from Kong to one origin (any instance of a service) with a different origin (the local instance of the service).
iptables and transparent
Together, setting <code>iptables</code> and <code>transparent</code> cause a given service to communicate only with a local Kong proxy (instead of bypassing it) without changing the service itself. Each service could be modified so that it only communicated with its local proxy. However, we intended Kong to function as a service mesh without requiring any changes to the services it proxies. Using <code>iptables</code> and <code>transparent</code> make that possible.
The user-space utility, <code>iptables</code>, allows system administrators to send traffic destined for one set of IPs or ports to different ones. When traffic comes into a host destined for a service, <code>iptables</code> will send that traffic through Kong instead.
A new, optional suffix for <code>proxy_listen</code> and <code>stream_listen</code>, <code>transparent</code> lets Nginx (which Kong is built on) read original destinations and ports that <code>iptables</code> have changed and answer requests. This enables Kong to listen to and respond from the IP addresses and ports you configure in iptables.
- <code>iptables</code> on a host/vm/pod running Kong is configured with a rule like: “send everything that was addressed to 220.127.116.11/24 on any port to the program listening on 18.104.22.168:80 instead”
- For Kong to accept the HTTP traffic sent to 22.214.171.124:80, kong.conf includes <code>#proxy_listen = 126.96.36.199:80 transparent</code>
- OR if you want Kong to accept TCP traffic sent to 188.8.131.52:80, kong.conf would need to include <code>#stream_listen = 184.108.40.206:80 transparent</code>
- Kong/nginx will fail to start if both <code>stream_listen</code> and <code>proxy_listen</code> contain the same ip+port
The <code>run_on</code> property controls the Kong instance that a plugin executes on when Kong is running as a service mesh. Without <code> run_on</code>, configured plugins always run on both nodes in a Kong-to-Kong connection, which can produce undesirable results. For example, consider a Kong-to-Kong connection with a rate-limiting plugin enabled. The first Kong node would increment a rate-limiting counter, and then the second Kong node would also increment the counter, resulting in one request being counted twice. The <code>run_on</code> setting requires a mutual TLS connection between the Kong nodes, with a certificate issued by Kong’s built-in certificate authority.
Kong Certificate Authority (CA)
Kong-generated certificates are required for a mesh deployment to work correctly because of an ALPN (Application Layer Protocol Negotiation) tweak that the Kong CA includes in client certificates. The tweak enables Kong nodes to recognize each other as part of the same mesh and apply mesh-specific settings. Although you can deploy Kong nodes in a mesh pattern and establish mTLS connections between them using certificates from an outside CA, this configuration won’t function fully because mesh settings like <code>run_on</code> won’t work without Kong-generated certificates.
The Kong CA is generated and Kong-to-Kong mTLS connections (KmTLS) are established in the following steps:
- The first Kong node in your cluster starts up, checks the datastore, notices there is no Certificate Authority (CA), creates a CA and stores it in the datastore. This CA stays in the datastore and is valid indefinitely.
- That “first” Kong node and all future Kong nodes that start in that cluster get the CA from the datastore, use it to generate a certificate, which the nodes store locally. Each Kong node in the cluster now has a certificate signed by the same CA.
- When a given Kong node makes a TLS connection, it utilizes its certificate. When that connection is made to a second Kong node in the cluster, the connection is automatically upgraded to KmTLS.
- Now that the connection is KmTLS, the Kong nodes can correctly run plugins as directed by the <code>run_on</code> settings at the configured points in the connection (on the first Kong node, on the second or on both).
Note that if a Kong node connects to another Kong node without TLS (eg. via HTTP or TCP), there is no opportunity for a TLS connection to be automatically upgraded to mTLS. Therefore unencrypted connections between Kong nodes can not form fully functional service meshes. TLS is not only available when Kong is run as a service mesh; it is required.
We’re excited to continue developing Kong’s service mesh capabilities. For now, here are a few things to make sure of and look out for when deploying Kong in a service mesh pattern:
- Kong nodes in the cluster must be version 1.0.0 or above
- Kong nodes must all belong to a single Kong cluster (Kong does not yet support KmTLS when connecting across multiple Kong clusters)
- Kong’s Service entities must be configured with TLS or HTTPS (not TCP or HTTP).
- Some plugin configurations could cause the first Kong node that requests pass through to strip consumer information from the request before sending it to the second Kong node. Plugins that run on the second Kong node could have trouble with these requests if they need to know about the consumer.
Deploy Kong as a Service Mesh