Companies are charging headfirst into AI, with research around agentic AI in the enterprise finding as many as 9 out of 10 organizations are actively working to adopt AI agents. 

LLMs are being deployed, agentic workflows are getting created left and right, and the promises of what this technology will bring are endless. But behind the scenes, our customers tell us that there's another piece of the story, with shadow AI, runaway costs, security nightmares, and organizational uncertainty.

We recently spoke with some of our customers about AI governance – the good, the bad, and the ugly – and the message was unanimous: the opportunity is massive, but the risks are too.

Here's what we learned about the real challenges around AI governance that companies are facing today — and how forward-thinking organizations are starting to solve them.

1. Shadow AI is spreading faster than governance can catch up

One fear we hear about again and again? Teams are moving independently while leadership scrambles to establish guardrails.

Engineers are spinning up their own MCP servers, calling LLMs directly, and creating hordes of agents while platform teams worry about future cleanup nightmares.

One of our 2025 Kong Innovator Award winners, H&M, experienced this firsthand. As teams across the organization began adopting LLMs independently, the company found itself facing security risks, operational inefficiencies, and a complete lack of financial governance. Without centralized visibility, it was nearly impossible to scale AI safely across the enterprise.

The root cause? Teams are unsure whether governance belongs to AI squads, platform engineering, security, or a new AI Center of Excellence, leading to slow decision-making while engineers move fast on their own.

2. Your APIs probably aren't agent-ready (and that's a problem)

Elizabeth Brand, VP and Global Head of Cloud at Prudential, speaking at our API Summit 2025, described AI as a "once-in-a-decade opportunity" to solve previously difficult and time-consuming problems. Prudential is using AI to refactor legacy applications, decompose them into smaller components, and eliminate duplicate APIs — transforming modernization timelines from years to weeks or months.

But here's the catch: most organizations we talk to admit that their APIs are far from agent-ready, with missing specs, inconsistent design, and poor observability remaining as blockers.

The challenge is compounded by data privacy and security fears, with everyone anxious about PII exposure, accidental data leaks, and targeted attacks.

3. Token costs are the new cloud bill shock

Remember when cloud costs spiraled out of control because nobody was watching? AI governance faces the same risk — but faster.

Token spending is unpredictable and already scaring finance teams, with customers desperate for centralized limits, multi-model routing, and semantic caching to avoid these runaway costs.

Another Kong Innovator Award winner, SeatGeek, created a solution to address this challenge. 

As LLMs became central to their operations, SeatGeek faced a challenge: these LLM requests looked just like ordinary HTTP traffic, making it nearly impossible to ensure trust and safety in AI-driven integrations. The team at SeatGeek used Kong to create centralized LLM request validation at the API gateway layer, which eliminated the risk of spoofed traffic, avoided duplicating code across more than a dozen microservices, and reduced engineering time by 2–3 weeks per service.

4. It's a people problem just as much as a tech problem

Perhaps the most interesting insight from our customers: API teams speak in terms of requests and responses, while AI teams talk about tokens and models, making bridging that language gap part of governance itself.

Education, change management, and internal alignment repeatedly came up as the largest blockers to effective AI governance. But this is nothing new. For any large-scale, enterprise-wide project, the people piece will always be one of the major challenges, even more so than the technology.

1. Centralize before it's too late

Customers see the risks of decentralization and are actively exploring AI gateways and MCP gateways to regain control.

H&M's transformation illustrates this perfectly. By implementing Kong AI Gateway, they pivoted from a fragmented, high-risk model to a secure, scalable, and governed AI platform with centralized control, observability, and governance for all AI traffic. 

The results? AI service onboarding time dropped from weeks to days, they gained immediate financial oversight and cost visibility, and they established 100% centralized logging and auditability for compliance.

2. Treat API hygiene as a must-have, not a nice-to-have

Good specs, security scopes, and predictable behavior aren't nice-to-haves anymore — they're prerequisites for safe agentic workloads.

Prudential's approach demonstrates this. By breaking down applications into smaller components and eliminating API duplication, they're reducing ecosystem complexity while preparing their infrastructure for AI-driven workflows.

3. Make cost governance part of the conversation now, not later

Companies are already looking for ways to control token consumption and avoid bill shock. Implementing AI governance is an essential part of keeping your finance team happy, as well as your developers.

As H&M was building its centralized AI governance platform, the team implemented Kong AI Gateway features like intelligent LLM routing and centralized rate limiting for AI traffic to avoid incurring runaway costs. The team anticipates seeing major savings going forward — and having the visibility to predict the costs they will have.

4. Embrace the learning curve

Here's the surprising silver lining: none of our customers claim to have figured out the one simple solution to solving AI governance problems. The playbooks are still being written, so everyone has the chance to think creatively and put innovative solutions in place.

As Liz Brand from Prudential put it, companies in the successful 5% are those that obsessively learn and relentlessly pursue different solutions than what worked a decade ago. This is a "wipe the slate clean" moment.

AI governance isn't a future problem: it's the AI reality. The organizations that will succeed aren't necessarily the ones with the most AI projects or the biggest budgets. They're the ones who recognize that governance, security, and cost control need to be built into the foundation, not bolted on after the fact.

84% of companies report a hit to gross margins from AI costs. Sustainable AI businesses avoid the hidden AI fragmentation tax by being proactive.

The good news? You're not alone in figuring this out. Every organization — from global retailers to financial services giants — is navigating the same challenges. The difference between chaos and control comes down to one thing: centralizing governance before shadow AI forces your hand.

Ready to take control of your AI governance strategy? Learn more about how Kong AI Gateway is helping enterprises scale AI securely and cost-effectively. Contact us to get started.