How McAfee Leverages Kong Gateway for Data Ingestion at Scale
McAfee, a global leader in online protection for consumers, leveraged Kong Gateway to scale data ingestion for its consumer data platform — all while reducing overhead costs. At API Summit 2023, Wisdom Koi, Tech Lead of Data Platform Engineering at McAfee, discussed how Kong enabled its team to achieve significant performance improvements and cost savings.
Background
McAfee is a worldwide leader in online protection that focuses on safeguarding people, not just devices, through personalized privacy and security. McAfee's approach seeks to understand customer journeys and utilize data to drive engagement, product outcomes, and holistic, intelligent protections for safe online experiences.
McAfee ingests high volumes of data from its services and applications to inform business insights and use cases — up to 150 megabytes per second from over 80 different sources. As McAfee's data volumes and sources continue to grow, it's critical for them to maintain a cost-effective and scalable data ingestion process.
The Challenge
“The question,” Koi said, “is how can we scale more cost-effectively as data volume increases and new data sources are added?”
McAfee had previously tried third-party managed ingestion solutions and found they had little to no control over their costs when scaling alongside data volumes and sources. This outcome prompted their decision to redesign the ingestion system using a combination of open-source and cloud technologies.
Beyond scalability and cost, McAfee also needed to redesign the ingestion system for better monitoring, alerting, and data enrichment capabilities within their ingestion layer. Additionally, their teams were seeking centralized gateway functions to ingest and serve data for their entire platform.
The Solution
McAfee deployed Kong Gateway with a hybrid control and data plane architecture, reducing disruption, strengthening application security, and boosting development efficiency with increased configurability. On top of that, the hybrid deployment decreases traffic to the database since it only connects to the control plane and provides additional security by isolating the two planes.
Kong's deployment leverages plugins to integrate with other systems, like McAfee's queueing solution. The plugin acts as an intermediary enrichment and messaging layer to handle data processing, including encoding or decoding traffic requests. Kong Gateway enables configurable multi-tenant deployment to meet McAfee's availability requirements across different data sources. Isolation between the data and control planes creates greater reliability because if one plane is down, the other can still process traffic by keeping tenants separate. Lastly, the Kong plugins allow teams to handle errors properly and transform responses before they get back to the client.
By utilizing Kong Gateway and a distributed queuing system, McAfee can efficiently manage traffic, authentication, authorization, data processing, and storage.
Results
Kong Gateway achieved low latency numbers and a high throughput of 150 megabytes per second for McAfee. On top of that, it resulted in a 50% cost reduction compared to previous ingestion systems. Kong has demonstrated that it flexibly scales to meet McAfee’s data volumes, and its microservices plugin model accelerates new use cases to serve emerging needs.
“We continue to monitor the system for areas that we can optimize to save more costs and to improve performance,” Koi said.
McAfee saw more wins than just cost efficiency and scalability. The hybrid deployment with Kong was highly configurable, and its resource isolation increased reliability across multiple data sources. Additionally, its extensibility made innovation much more straightforward for development teams.
Lessons Learned
Koi shared that, as with any other solution, there are changes to keep in mind when turning toward an ingestion system that uses open-source and cloud technologies.
He advised development teams to be mindful of system upgrades that require migrating from one version to another. He also said teams should prepare for intentional infrastructure management, including system upgrades, node patching, and monitoring.
Overall, migrating to a hybrid architecture with Kong skyrocketed performance and plummeted costs while adapting to McAfee’s specific needs: “It makes the whole deployment process highly configurable," Koi said.
Conclusion
By jumping headfirst into a new hybrid ingestion solution using Kong, McAfee obtained the high-performing, cost-efficient, and scalable data ingestion capabilities they had been searching for. While no transition comes without growing pains, optimizing hybrid deployment opens the door for greater savings and efficiency over time.
As McAfee's data volume and source landscape continue to rapidly expand, the configurable Kong implementation will adapt to serve their unique consumer data needs.