In Kong Gateway 2.0, we released [Hybrid Mode](https://docs.konghq.com/gateway/latest/production/deployment-topologies/hybrid-mode/)Hybrid Mode, also known as Control Plane/Data Plane separation. With it, our customers could efficiently and securely deploy clusters of Kong Gateway Dataplanes on their on-prem, private, and public clouds in any combination they wanted, and they could control the entire cluster from a single point, the Control Plane.
Hybrid Mode became instantly popular with all our customers with large and varied deployments, due to the increased flexibility and ease of management. Over time those customers grew their configuration sets to sizes in the order of hundreds of thousands of configuration entries, which brought a new challenge to Hybrid Mode.
In Hybrid Mode, when the Admin API is used at the control plane level to change the active configuration, it immediately triggers a cluster-wide update of all data plane configurations. In these updates, the entire configuration set is sent to the data planes; a bigger configuration set means more data is sent down the wire, more time needed to travel through the network, and more time needed to process the new configuration set on each data plane. All this processing time could translate into latency spikes and loss in throughput for high-traffic dataplanes under certain conditions. Additionally, the processing of large configuration sets causes Kong Gateway Control Plane and Data Plane to consume extra memory proportional to configuration size.
To fix this problem, we worked on redesigning the way configuration updates are handled in Hybrid Mode: Incremental Configuration Sync.
The main concept behind Incremental Configuration Sync is very simple: when a configuration changes, instead of sending the entire configuration set result of the change to each data plane, we only send the parts of the configuration that have changed. This means less data traveling over the network, less data for the data plane to process to rebuild the new configuration, and the end result is almost instantaneous propagation of configuration changes in the majority of cases*, even with configuration sets with hundreds of thousands of configuration entries.
**The exceptions would be Konnect Control Plane opportunistically coalescing multiple changes into one batch (still significantly smaller that the whole configuration), and instances where the CP might determine it’s safer to resync the whole configuration to a data plane, e.g., for a data plane that hasn’t connected for a time over a certain threshold.*
To measure how much improvement incremental sync could bring to our users, we've performed a benchmark of Hybrid mode under incremental sync using the following config setup:
We performed two different sets of tests, one using the Konnect Control Plane, and one using the on-prem control plane distributed with Kong Gateway.
For the Data Planes, we performed the tests on two AWS EC2 `t2.medium` instances, one running the DP on `kong/kong-gateway:3.8.0.0`, and one running the incremental sync technical preview build, with each DP running a single worker process.
For the memory usage tests, we created the 90,000 entities described above in the Control Planes using a script.
After verifying that the configuration was up to date in each Data Plane, we performed the following additional operations:
For the Konnect test, the memory usage is measured only on the Data Planes.
The results show that while with incremental sync the memory usage stays relatively constant before and after performing a configuration sync, without incremental sync the memory usage balloons significantly after performing a configuration sync.
For this test, the memory usage is measured on the Control Planes and on the Data Planes.
Incremental Sync is **OFF**
Incremental Sync is **ON**
The results show that for both Control Plane and Data Plane nodes, the memory usage is significantly lower when using Incremental Config Sync.
For the CPU Usage Tests, we used the same configuration and setup used for the memory tests, and then we measured the CPU usage of the container during the config sync of a newly created consumer entity.
Without Incremental Config Sync
With Incremental Config Sync
As demonstrated above, incremental sync achieves significant memory savings as well as CPU savings. This means lower total cost of ownership for Kong users, shorter config propagation delay, and less impact to proxy latency. You can see it for yourself by testing it using the Konnect Control Plane or the on-prem Control Plane.
To test Incremental Configuration Sync on Konnect, log into one of your Organizations and create a new Self-Managed Gateway in the Gateway Manager.
After that, create a new Data Plane Node and click on the **Generate certificate** button. At this point, you'll see a quick start docker command generated, something like:
docker run -d \
-e "KONG_ROLE=data_plane" \
-e "KONG_DATABASE=off" \
-e "KONG_VITALS=off" \
-e "KONG_CLUSTER_MTLS=pki" \
-e "KONG_CLUSTER_CONTROL_PLANE=<host-prefix>.cp0.konghq.com:443" \
-e "KONG_CLUSTER_SERVER_NAME=<host-prefix>.cp0.konghq.com" \
-e "KONG_CLUSTER_TELEMETRY_ENDPOINT=<host-prefix>.tp0.konghq.com:443" \
-e "KONG_CLUSTER_TELEMETRY_SERVER_NAME=<host-prefix>.tp0.konghq.com" \
-e "KONG_CLUSTER_CERT=-----BEGIN CERTIFICATE-----
<actual certificate here>
-----END CERTIFICATE-----" \
-e "KONG_CLUSTER_CERT_KEY=-----BEGIN PRIVATE KEY-----
<actual key here>
-----END PRIVATE KEY-----" \
-e "KONG_LUA_SSL_TRUSTED_CERTIFICATE=system" \
-e "KONG_KONNECT_MODE=on" \
-p 8000:8000 \
-p 8443:8443 \
kong/kong-gateway:3.8.0.0To test incremental config you will need to make two changes to that script:
Use the `kong/kong-gateway-dev:incremental-preview-03 image`
Add two additional flags to enable the incremental config: `KONG_CLUSTER_RPC_SYNC=on` and `KONG_CLUSTER_RPC=on`
***Please follow the **[**instructions here**](https://docs.konghq.com/gateway/latest/production/deployment-topologies/hybrid-mode/incremental-config-sync/)**instructions here**** if you use Kong Gateway 3.10.x to enable the feature.***
With the applied changes, the previous script would become:
docker run -d \
-e "KONG_ROLE=data_plane" \
-e "KONG_DATABASE=off" \
-e "KONG_VITALS=off" \
-e "KONG_CLUSTER_MTLS=pki" \
-e "KONG_CLUSTER_CONTROL_PLANE=<host-prefix>.cp0.konghq.com:443" \
-e "KONG_CLUSTER_SERVER_NAME=<host-prefix>.cp0.konghq.com" \
-e "KONG_CLUSTER_TELEMETRY_ENDPOINT=<host-prefix>.tp0.konghq.com:443" \
-e "KONG_CLUSTER_TELEMETRY_SERVER_NAME=<host-prefix>.tp0.konghq.com" \
-e "KONG_CLUSTER_CERT=-----BEGIN CERTIFICATE-----
<actual certificate here>
-----END CERTIFICATE-----" \
-e "KONG_CLUSTER_CERT_KEY=-----BEGIN PRIVATE KEY-----
<actual key here>
-----END PRIVATE KEY-----" \
-e "KONG_LUA_SSL_TRUSTED_CERTIFICATE=system" \
-e "KONG_KONNECT_MODE=on" \
-e "KONG_CLUSTER_RPC_SYNC=on" \
-e "KONG_CLUSTER_RPC=on" \
-p 8000:8000 \
-p 8443:8443 \
kong/kong-gateway-dev:incremental-preview-03If you want to test Incremental Configuration Sync without Konnect, just use the tech preview images (a special image of Kong Gateway 3.8) that we published in [Kong Docker Hub](https://hub.docker.com/r/kong/kong-gateway-dev)Kong Docker Hub:
docker pull kong/kong-gateway-dev:incremental-preview-03To turn on Incremental Config Sync, add the additional flags `KONG_CLUSTER_RPC_SYNC=on` and `KONG_CLUSTER_RPC=on`, for both Data Plane and Control Plane.
These images are for testing purposes only and are not meant for production use. When testing incremental sync, please ensure the same image is deployed for both CP and DP nodes, as the preview feature has no compatibility guarantees. If you find any bugs, please let us know by [filling an issue](https://github.com/Kong/kong/issues)filling an issue on the Kong Gateway GitHub repository or contact support if you’re an Enterprise customer.
Please note the following known limitations:
All these known limitations are temporary and will be addressed in the GA release of Incremental Configuration Sync.
We're working hard to get Incremental Config Sync ready for production usage. While we get there, please try it out, and don't hesitate to share your thoughts and experiences with us by opening a [discussion in our GitHub repo](https://github.com/Kong/kong/discussions)discussion in our GitHub repo.
We'll keep you updated as we make changes to the tech preview, and stay tuned for an upcoming announcement of the general availability of the Incremental Configuration Sync!
Managing gateway configurations at scale is harder than it looks. When a plugin needs to apply to most routes, but not all, teams could either duplicate configuration across routes and violate DRY (“Don’t Repeat Yourself”) principles, or write custo
[](https://konghq.com/blog/product-releases/kong-gateway-3-14)
Imagine you have a single Service, order-api . You want to apply a strict rate limit to most traffic, but you want to bypass that limit—or apply a different one—if the request contains a specific X-App-Priority: High header. Previously, you had t
[](https://konghq.com/blog/engineering/conditional-policy-execution)
How OAuth 2.0 Token Exchange Reshapes Trust Between Services — and Why the API Gateway Is Exactly the Right Place to Enforce It Modern applications don’t run as a single monolithic. They are composed of services — frontend APIs, backend microservi
[](https://konghq.com/blog/engineering/token-exchange-at-the-gateway)
How Kong Gateway 3.14 closes the consistency gap in IAM-based authentication across AWS, Azure and GCP — and what it means for your production deployments Starting with 3.13 (which addressed Redis support) and completed in 3.14, Kong now presents
[](https://konghq.com/blog/engineering/cloud-native-authentication)
Traditional APIs are, in a word, predictable. You know what you're getting: Compute costs that don't surprise you Traffic patterns that behave themselves Clean, well-defined request and response cycles AI APIs, especially anything that runs on LLMs
[](https://konghq.com/blog/engineering/monetize-ai-apis)
It’s been almost a year since we released our Konnect Terraform provider . In that time we’ve seen over 300,000 installs, have 1.7 times as many resources available, and have expanded the provider to include data sources to enable federated managem
[](https://konghq.com/blog/product-releases/terraform-provider-konnect-v3)
When speaking with our customers, and particularly with platform teams, we repeatedly hear about how difficult it is to discover and govern all the services and APIs that actively run on their infrastructure. In ever-expanding and changing environm
[](https://konghq.com/blog/product-releases/create-an-internal-api-and-service-inventory)