Faster Config Updates in Hybrid Mode with Incremental Config Sync

In Kong Gateway 2.0, we released Hybrid Mode, also known as Control Plane/Data Plane separation. With it, our customers could efficiently and securely deploy clusters of Kong Gateway Dataplanes on their on-prem, private, and public clouds in any combination they wanted, and they could control the entire cluster from a single point, the Control Plane.

Hybrid Mode became instantly popular with all our customers with large and varied deployments, due to the increased flexibility and ease of management. Over time those customers grew their configuration sets to sizes in the order of hundreds of thousands of configuration entries, which brought a new challenge to Hybrid Mode.

In Hybrid Mode, when the Admin API is used at the control plane level to change the active configuration, it immediately triggers a cluster-wide update of all data plane configurations. In these updates, the entire configuration set is sent to the data planes; a bigger configuration set means more data is sent down the wire, more time needed to travel through the network, and more time needed to process the new configuration set on each data plane. All this processing time could translate into latency spikes and loss in throughput for high-traffic dataplanes under certain conditions. Additionally, the processing of large configuration sets causes Kong Gateway Control Plane and Data Plane to consume extra memory proportional to configuration size.

To fix this problem, we worked on redesigning the way configuration updates are handled in Hybrid Mode: Incremental Configuration Sync.

Incremental Config Sync

The main concept behind Incremental Configuration Sync is very simple: when a configuration changes, instead of sending the entire configuration set result of the change to each data plane, we only send the parts of the configuration that have changed. This means less data traveling over the network, less data for the data plane to process to rebuild the new configuration, and the end result is almost instantaneous propagation of configuration changes in the majority of cases*, even with configuration sets with hundreds of thousands of configuration entries.

*The exceptions would be Konnect Control Plane opportunistically coalescing multiple changes into one batch (still significantly smaller that the whole configuration), and instances where the CP might determine it’s safer to resync the whole configuration to a data plane, e.g., for a data plane that hasn’t connected for a time over a certain threshold.

Performance differences

To measure how much improvement incremental sync could bring to our users, we've performed a benchmark of Hybrid mode under incremental sync using the following config setup:

We performed two different sets of tests, one using the Konnect Control Plane, and one using the on-prem control plane distributed with Kong Gateway.

For the Data Planes, we performed the tests on two AWS EC2 t2.medium instances, one running the DP on kong/kong-gateway:3.8.0.0, and one running the incremental sync technical preview build, with each DP running a single worker process.

Memory usage tests

For the memory usage tests, we created the 90,000 entities described above in the Control Planes using a script.

After verifying that the configuration was up to date in each Data Plane, we performed the following additional operations:

• create one additional test service and route
• hit the route until getting a response with 200 status code
• delete test service and route
• hit the route until getting a response different from one with 200 status code

Results With Konnect

For the Konnect test, the memory usage is measured only on the Data Planes.

The results show that while with incremental sync the memory usage stays relatively constant before and after performing a configuration sync, without incremental sync the memory usage balloons significantly after performing a configuration sync.

Results with on-prem Control Plane

For this test, the memory usage is measured on the Control Planes and on the Data Planes.

Incremental Sync is OFF

Incremental Sync is ON

The results show that for both Control Plane and Data Plane nodes, the memory usage is significantly lower when using Incremental Config Sync. 

CPU usage tests

For the CPU Usage Tests, we used the same configuration and setup used for the memory tests, and then we measured the CPU usage of the container during the config sync of a newly created consumer entity.

Results with Konnect

Results with on-prem Control Plane

Without Incremental Config Sync

With Incremental Config Sync

Try it out

As demonstrated above, incremental sync achieves significant memory savings as well as CPU savings. This means lower total cost of ownership for Kong users, shorter config propagation delay, and less impact to proxy latency. You can see it for yourself by testing it using the Konnect Control Plane or the on-prem Control Plane.

To test Incremental Configuration Sync on Konnect, log into one of your Organizations and create a new Self-Managed Gateway in the Gateway Manager.

After that, create a new Data Plane Node and click on the Generate certificate button. At this point, you'll see a quick start docker command generated, something like:

To test incremental config you will need to make two changes to that script:

1. Use the kong/kong-gateway-dev:incremental-preview-01 image

2. Add an additional flag to enable the incremental config: KONG_CLUSTER_INCREMENTAL_SYNC=on

With the applied changes, the previous script would become:

If you want to test Incremental Configuration Sync without Konnect, just use the tech preview images (a special image of Kong Gateway 3.8) that we published in Kong Docker Hub:

Then follow the normal setup steps for a Hybrid mode setup.

To turn on Incremental Config Sync, add the additional flag KONG_CLUSTER_INCREMENTAL_SYNC=on, for both Data Plane and Control Plane.

These images are for testing purposes only and are not meant for production use. When testing incremental sync, please ensure the same image is deployed for both CP and DP nodes, as the preview feature has no compatibility guarantees. If you find any bugs, please let us know by filling an issue on the Kong Gateway GitHub repository or contact support if you’re an Enterprise customer.

Please note the following known limitations:

• Existing custom plugins that use custom cache keys may not be compatible with Incremental Configuration Sync due to changes in event handling.
• When using the Developer Portal, you won’t be able to enable or disable application registration for an API product version.
• The Konnect Config Store is currently incompatible with incremental configuration synchronization.
• Rate-limiting plugins might exhibit inconsistencies in sync behavior during rapid configuration updates, impacting performance for API traffic control.

All these known limitations are temporary and will be addressed in the GA release of Incremental Configuration Sync.

What’s next?

We're working hard to get Incremental Config Sync ready for production usage. While we get there, please try it out, and don't hesitate to share your thoughts and experiences with us by opening a discussion in our GitHub repo.

We'll keep you updated as we make changes to the tech preview, and stay tuned for an upcoming announcement of the general availability of the Incremental Configuration Sync!