Everyone's telling you to innovate faster with AI. Move quicker. Ship more features. Deploy more agents. But before we sprint headlong into the AI revolution, we need to have a proper dollars-and-cents conversation that most companies are avoiding.
Disclaimer: Statistics cited in this post were reported in the 2025 State of AI Cost Governance Report by Mavvrik.
The data is stark: 84% of companies report more than a 6% hit to gross margin from AI costs.
Within that alarming statistic, 58% see a 6–15% reduction, and 26% report erosion of 16% or more. This isn't a rounding error: this is a fundamental business problem threatening the viability of AI-powered products and services.
According to the report, AI costs are already eroding the margins that businesses have worked years to build. Yet most organizations are racing ahead without understanding the true cost of their AI infrastructure.
Why — with all of the promise that AI brings — is this happening at such an alarming rate?
Here's a sobering reality: only about 35% of companies include on-premises components in their AI cost reporting. Even more concerning, roughly half don't include LLM API costs in their tracking — even when AI is a core product component.
When teams are asked what would most improve their cost management, the answer is clear: unified visibility across environments is the top priority, followed closely by clear cost attribution. You can't optimize what you can't see.
Only 15% of companies can forecast AI costs within ±10% accuracy. Let that sink in. The majority (56%) miss their forecasts by 11–25%, and nearly one in four companies miss by more than 50%.
For CFOs and budget owners trying to protect gross profit targets, this level of unpredictability is a nightmare. As AI grows as a share of cost of goods sold (COGS), the inability to accurately forecast becomes an existential threat to financial planning.
A full 61% of companies run AI workloads across a combination of public and private environments and different tools. This hybrid pattern spans all company sizes, including small businesses, and creates exponentially greater difficulty in achieving unified cost reporting and governance.
All of this — mostly driven by lack of a unified approach to managing AI resources and AI connectivity — results in the hidden AI fragmentation tax. Let’s break it down.
Source: 2025 State of AI Cost Governance Report by Mavvrik
Let’s take an MCP-enabled agentic workflow as our example.
Each MCP client connects to MCP servers. Those servers interact with LLMs. Your end consumers trigger these interactions. And every single one of those touchpoints generates LLM API token consumption that quietly eats into your margins.
The 2025 State of AI Cost Governance Report reveals a critical insight: "Even companies that do not charge for AI-enabled products are heavy users of third-party LLMs (73%), meaning token-based costs are quietly reducing gross margins without being offset by direct revenue."
Think about that. Nearly three-quarters of companies are hemorrhaging margin to LLM providers without generating corresponding revenue.
LLMs aren’t the only resource that AI consumes, and they certainly aren’t the only driver of excess networking / egress charges, which is the second most common unexpected AI cost according to the report.
Source: 2025 State of AI Cost Governance Report by Mavvrik
Whether the AI application itself directly consumes them, or the AI application consumes an MCP server that consumes them, there is a massive amount of agentic consumption of both APIs and real-time data that organizations must be aware of.
So, when you map out the full architecture of modern AI applications, you see a sprawling web of connectivity:
And of course all of this exists — for many businesses — accross multiple on-prem, hybrid, cloud, and multi-cloud vendor environments.
Are you starting to see the looming problem? It’s time for a reality check.
The typical AI infrastructure today suffers from three critical problems:
This is why — when the report explores “Tactics for improvement,” it finds:
But, given this mess of connectivity to manage, how is this possible?
The solution isn't to stop innovating or slow down. It's to build a unified AI cost governance layer that can unify real-time cost visibility & enforcement.
From a technical implementation perspective, this means building a single cost control plane that provides:
This governance layer needs to span your entire AI infrastructure:
This is why we are so excited about OpenMeter and what it brings to the Konnect platform. By combining the existing LLM, MCP, API, Eventing, and microservices runtime infrastructure already in Konnect with the real-time metering, billing, and usage attribution of OpenMeter (now powering Konnect Metering and Billing), you have a single platform that can:
And all of this is, of course, on top of the security, reliability, performance, and developer experience benefits of using Konnect to build, run, discover, and govern the resources that drive AI, API, eventing, and microservices innovation.
Today, Konnect Metering & Billing is in early access. If you’d like to inquire into the early access program, fill out the form here. We’ll get back to you soon.
Take it from the report itself: "As AI transforms from 'nice to have' to 'must have,' the companies that master cost visibility and control will protect their margins while competitors watch profits disappear into untracked infrastructure costs."
Here's what this means practically:
In a world where everyone has access to similar AI capabilities, competitive advantage comes from operational excellence. The companies that can deliver AI-powered experiences profitably will win.
This isn't about choosing between speed and control. Modern AI Connectivity platforms must deliver both — fast deployment cycles with comprehensive cost governance built in from day one.
Seeing your costs isn't enough. You need real-time enforcement mechanisms that prevent overruns before they impact margins, not months later in a financial review.
Every day without unified AI cost governance is another day of margin erosion. The longer you wait, the more deeply embedded these costs become in your COGS structure, making them exponentially harder to unwind.
AI innovation is critical. Speed matters. But sustainable AI businesses are built on a foundation of cost visibility and control. Before you accelerate, make sure you can see where you're going—and how much it's actually costing you to get there.
The companies that figure this out will thrive. The ones that don't will find themselves with impressive AI capabilities and vanishing margins.
At Kong, we’re here to help you win the AI cost governance battle. Just give us a ring.
While many organizations mistakenly believe a single tool can solve all their API security woes, the truth is far more complex. This blog post will dismantle the myth of the "silver bullet" and demonstrate how a comprehensive, defense-in-depth strat
Ever feel like you're fighting an uphill battle with your API strategy? You're building APIs faster than ever, but somehow everything feels harder. Wasn’t API-first supposed to make all this easier? Well, you're not alone. And now industry analys
Kong-quer the Agentic AI Hackathon 🚀 Calling all builders, tinkerers, and API innovators. The Kong Hackathon is back for API Summit 2025 ! This year, we’re challenging developers worldwide to create projects that don’t just react, they think , a
Picture this: It's 3 AM, and your phone erupts with alerts. Within minutes, you're drowning in a tsunami of notifications—hundreds of them—while your company's critical services hang by a thread. Your monitoring dashboard looks like a Christmas tree
Organizations are going all-in on large language models (LLMs), with research finding 72% anticipate increased LLM spending in the coming year (and about 40% are already investing more than $250,000 USD per year). As enterprises rapidly adopt LLMs
In the last two parts of this series, we discussed How to Strengthen a ReAct AI Agent with Kong AI Gateway and How to Build a Single-LLM AI Agent with Kong AI Gateway and LangGraph . In this third and final part, we're going to evolve the AI Agen
In my previous post, we discussed how we can implement a basic AI Agent with Kong AI Gateway. In part two of this series, we're going to review LangGraph fundamentals, rewrite the AI Agent and explore how Kong AI Gateway can be used to protect an LLM
Get a personalized walkthrough of Kong's platform tailored to your architecture, use cases, and scale requirements.