KSAI began as a plugin marketplace for reusable agent skills. But it quickly evolved into a platform on which engineers can build agentic workflows.

Out of simplicity we decided to build KSAI around GH actions as a runtime engine. We didn't need any extra infrastructure we would need to worry about. Our architecture provides several benefits: 

• - Each repository adds one workflow file. 
• - KSAI Control Plane (CP) is responsible for triggering actions. 
• - We maintain the execution code in the shared workflows, and teams choose when to adopt a new version. 
• - Workload identities are provided by GitHub, so  our code never actually leaves GitHub.

Thanks to KSAI CP, we can have granular control over what model we choose for the work; we can do AB testing on models and prompts. We can have more control over concurrency than simply using GitHub Actions. This infrastructure allows engineering teams to easily conduct A/B tests on different LLM prompts in CI, ensuring prompt engineering changes actually improve precision and recall before a fleet-wide rollout.

The runtime and the model are separate choices. KSAI started by running Claude Code and Anthropic models, as we were widely using them in engineering. It was serving its purpose, but we quickly realised that we need to keep this vendor neutral if we want to easily incorporate the OSS models into the mix. 

Before we fully switched to OpenCode, we wanted to gather some data from actual runs. We have set up an A/B testing scenario, where 10% of the runs run both Claude and OpenCode reviews at the same time. Both engines run the same model and effort. We didn't see much of a difference in the cost of Opus runs, but surprisingly Sonnet runs were 40% cheaper on OpenCode.

The agent's tools inspect the repository. Kong AI Gateway handles its model traffic, including routing and usage attribution, and enforces budgets. We handle provider compatibility there, so individual repositories don't have to.

When building KSAI, we knew that we cannot let agents do what they want, and we need to keep them in check. We took a few countermeasures to prevent agents from roaming freely.

KSAI runs OpenCode in GitHub Actions inside a `bubblewrap` sandbox. The root filesystem is mounted read-only, `/tmp` is an isolated `tmpfs`, and user and process namespaces are separated. The repository workspace is writable where implementation requires it, but KSAI’s trusted checkout under `_ksai` is remounted read-only. OpenCode itself adds a second control layer. Its permission map denies tools, shell commands, and external-directory access by default, then enables only those required for the current phase. 

Secrets are constrained independently of filesystem access. GitHub tokens, OIDC variables, and runner credentials are removed from the agent’s environment. The model-access token is brokered outside the sandbox and exposed through a read-only file. Trusted scripts and runtime configuration are also mounted read-only; only OpenCode’s own cache and state directories remain writable.

Authorisation and approval remain separate boundaries. KSAI checks whether the requester may start a paid or writing run, and its approval action prevents the pull-request author, commit authors, and recognised co-authors from approving their own changes.

At the beginning KSAI was built on top of skills we already had. So it was natural to just create subagents for specific reviews. The first version of KSAI looked like this:

The orchestrator had access to all subagents and was supposed to decide which subagent it needed to do a quality review on. The biggest issue with this approach was that each subagent was exploring the codebase and changes on its own. In effect we were burning tokens multiple times for the same thing.

While analysing the cost and performance of our current setup, we quickly realised that this is not the best solution. We moved the main decisions from the LLM orchestrator to a static step that triages review requests. It decides what skill the agent will need, what model and effort it should use and if review is even worth starting. Thanks to this, we were able to flatten our structure and gain more control over the review process. The new flow with triage step looks like this

When we started, we ran every review using the Opus model with high effort. It gave us great results quality-wise but was super expensive. Thanks to the addition of the triage step, we were able to model and effort based on the complexity of a task. For simpler and smaller reviews we were able to use the Sonnet model, which helped us reduce the cost of reviews.

Triage was our first win on review cost and speed. In our early measurements, median review cost fell from $3.93 to $2.47, so about 37% of overall spend. This triage phase acts as our primary defense against unnecessary token burn, establishing best practices for AI code review triage and significantly reducing LLM token spend in continuous integration.

The cost of running KSAI is a major concern for us, and LLM tokens can quickly pile up into a significant bill. The problem is that it is hard to optimize cost without switching models. Our triage strategy already proved itself, but we started looking at OSS models.

We started by creating our internal benchmarks to measure how models perform and how much they cost. We created a synthetic repo with proper history and opened 4 PRs on which we were comparing models. On each model we compute `blocking recall `(how many blocking defects the model found) and `precision` (how many findings are accepted by the judge), which allows us to compute F1 which balances missed defects against noisy findings. These metrics—`blocking recall` and `precision`—are the most critical metrics that matter when benchmarking LLMs for code quality, as they directly measure the agent's ability to catch real bugs without overwhelming developers with false positives.

From these we compute a model score that takes into account the cost of a run.

`model score = 100 × (F1 − review cost / V)`

By calculating a model score combining F1 and cost, we can make objective decisions about when to trade slight dips in precision for massive cost savings.

Here, `V` is the dollar value assigned to moving from F1 0 to F1 1. We show cost-first (V=$1), balanced (V=$5), and quality-first (V=$20) views.

Here are the results of our first benchmarks:

As you can see, Opus was still a winning model, but its price per review is 7x higher than GLM 5.3 and 80x higher than GLM 5.3 flash. When we plot F1 over cost we can quickly see the differences:

For now we picked GLM 5.3 and GLM 5.3 Flash as our default reviewer, which reduced our cost of reviews significantly while keeping similar quality:

By September 17, 22 teams were enrolled, covering 135 distinct repositories. KSAI already reviewed 7.8M lines of code and found more than 4k findings. 

To streamline adoption of KSAI, we have built full self-service for it. Engineers can easily request access for their repositories, and KSAI will create onboarding PRs for them.This self-service model is the key to rolling out AI tooling to 100+ repositories smoothly without bottlenecking the platform team.

Since KSAI changes rapidly, we needed a simple way of migrating people to new versions. For easy updates that are just version bumps, we just use Renovate to open PRs against consumer repos. For structural changes we build an internal mechanism that helps us to keep the whole fleet up to date. 

Our KSAI app has access to all enrolled repositories in GitHub. Thanks to this, we can monitor versions of KSAI used by consumers. KSAI CP understands action versioning and API changes made in different versions of it. Thanks to this we can create automated PRs to consumer repositories with updates to the new version. Automating version bumps of shared workflows across repos makes customer migration simple for us and ensures all teams are using the latest, most secure agent configurations.

We know that dogfooding is super important, so we wanted to start using KSAI to build KSAI as soon as possible. We started using KSAI as a default reviewer in KSAI project and in all team projects as soon as possible. To immediately catch bugs and see if we like the actual UX of it.

KSAI's bot authored 901 of the 1,573 commits in our September 17 repository snapshot, so more than a half. We use it to help build the system it runs in. It also now reviews all PRs. That's the handoff we want to trust at the end of the day: leave an agent a task, close the laptop, and come back to work where we can inspect or make a decision.

**How does Kong use AI for automated code reviews?**
Kong uses a custom-built platform called KSAI (Kong Skills for Agentic Intelligence). Rather than acting as an inline autocomplete tool, KSAI deploys AI agents directly into GitHub Actions CI/CD pipelines. These agents asynchronously review pull requests, enforce coding standards, and identify blocking defects without requiring developers to keep their local environments running.

**Why add a triage phase before running LLM code reviews?**
Adding a static triage step prevents burning expensive LLM tokens on simple tasks. The triage phase evaluates the complexity of a review request and dynamically selects the appropriate model and effort level (e.g., routing a small PR to a cheaper, faster model like Sonnet or GLM 5.3 Flash instead of Opus). At Kong, this triage step reduced median review costs by roughly 37%.

**What is the cost difference between commercial models (like Claude) and open-source models (like GLM)?**
While high-tier commercial models like Claude Opus provide excellent quality, they can be cost-prohibitive at scale. In our benchmarks, Opus was 7x more expensive per review than the open-source GLM 5.3 model, and 80x more expensive than GLM 5.3 Flash. Switching to these open-source models allowed us to drastically cut costs while maintaining comparable precision and blocking recall.

**What security controls sandbox AI agents in a pipeline?**
To prevent AI agents from executing unauthorized commands or leaking data, KSAI employs strict sandboxing in GitHub Actions. This includes mounting the root filesystem as read-only, isolating namespaces, denying shell commands by default via an OpenCode permission layer, and brokering model-access tokens outside the sandbox so they are only exposed through a read-only file.

**How do you get observability on per-repo LLM usage costs?**
We route all AI agent model traffic through the Kong AI Gateway. This allows us to abstract the underlying LLM providers and gain granular observability, cost tracking, and token usage attribution on a per-repository basis, enforcing budgets automatically.

**Should we switch from Claude Opus to GLM 5.3 for code reviews?**
If cost optimization is a priority, switching to GLM 5.3 is highly recommended based on our F1 vs. cost model scores. While Opus slightly edges out open models in raw blocking recall, GLM 5.3 and 5.3 Flash offer a much better balance of cost-efficiency and quality for the vast majority of standard CI/CD code reviews.