While monitoring is an important part of any robust application deployment, it can also seem overwhelming to get a full application performance monitoring (APM) stack deployed. In this post, we'll see how operating a Kubernetes environment using the open-source Kong Ingress Controller can simplify this seemingly daunting task! You’ll learn how to use Prometheus and Grafana on Kubernetes Ingress to simplify APM setup.
The APM stack we're going to deploy will be based on Prometheus and Grafana, and it'll make use of Kong's Grafana dashboard. Together, the solution will give us access to many important system details about Kong and the services connected to it—right out of the box. We'll simulate traffic going through the deployed stack and then observe the impact of that traffic using our monitoring tools.
It's important to note that this implementation will be more of a demonstration than an instruction manual on deploying to a production environment. We're going to use kind to create our Kubernetes cluster, making it easy to follow along on your local machine. From here, you'll have a foundation that will translate easily to your production-ready business applications.
Since we're using kind, there are only a few things we need to get started. Follow the installation instructions for your platform. We'll also use the kubectl command-line tool and Helm, so make sure you have up-to-date versions of all tools to follow along.
First, we'll create a named Kubernetes cluster:
Now, we're ready to get our monitoring stack deployed. It may seem a little strange to start with the monitoring rather than our Kubernetes service. Still, this way, we will know immediately if Kong is connected properly to Prometheus and Grafana. We'll start by adding repos for Prometheus and Grafana:
If you've already installed these repositories, just make sure to run helm repo update to ensure you have up-to-date Helm charts.
We'll start with Prometheus configuration. The first step is to configure the scrape interval for Prometheus. We want our system to check for metrics every 10 seconds. We can do that with the following YAML:
Put that file in your working directory, and let's get helming!
That should display a message saying, "For more information on running Prometheus, visit: https://prometheus.io/." Notice that we created a new namespace for cluster monitoring. This will allow us to keep things clean in our K8s cluster.
While our Prometheus installation is configuring, we'll set up Grafana as well. Let's start with this YAML:
A lot is going on in this configuration file, but the comments should give a sufficient explanation. Essentially, when Grafana starts up, we want the Prometheus connection and the Kong dashboard already set up, too, rather than having to go through those steps manually. This file gets us that setup.
The Helm command is very similar to the one for Prometheus:
After running this command, you'll see instructions for getting the admin password for your installation. We'll come back to that soon.
Now that our monitoring stack is getting going, we can configure Kong to do the heavy lifting in our application. First, we'll make sure to add the Kong Helm chart repo:
You'll also need to create a YAML for the annotations we want to add to our Kong pod:
With the YAML in place, let's get Kong installed via Helm:
Once you've gotten that kicked off, create the following YAML file to connect Kong and Prometheus:
Apply that configuration with kubectl:
With our stack fully built out in our kind cluster, we need to see what would happen if we used this sort of stack in production. In particular, we need to set up access to these services from a browser. We'll do that with a few port forwarding rules. In a new terminal session, we use kubectl to look up each piece we've created so far. Then, we set up a rule to visit each service as a localhost application in our browser of choice.
Without closing the terminal session, you should now be able to access different services based on the port:
One thing you'll notice is the login screen for Grafana, where you'll need a password. To obtain that password, use the following command in your original terminal session:
This should give you a 40-character alphanumeric password that you can copy and paste into your browser at http://localhost:3000/login as the password, using "admin" as the username. If things are going according to plan, you'll be able to click on the "Kong (official)" Grafana dashboard link and see the Grafana UI:
If your dashboard doesn't say "No data" all over the place, you're in good shape. Otherwise, you might want to review the above steps again to ensure that everything is configured correctly.
Finally, let's get our ingress point set up and expose a few sample services. If you were really trying to use this monitoring stack, this is where the deployment of your application would go. For now, we're going to make a few dummy services with the following endpoints:
We'll use httpbin, which will allow us to simulate a fully functioning service with response codes as we ask for them. We're doing this so we can quickly generate some traffic to our system. That way, things in the dashboard will look like they would if you deployed a real production system to your Kong instance.
Create the following two YAML files:
Apply these configurations with kubectl:
After a few minutes, those pods should be available, and you can start sending traffic to your services:
After letting your traffic simulation run for just a few minutes, return to your Grafana instance in your browser (http://localhost:3000) and zoom in on the last few minutes of activity. You should see lots of beautiful visualizations ready for your analysis, with just a little bit of Kubernetes know-how.
If you've been following along, now you're free to play around with all the out-of-the-box monitoring solutions. You can even introduce new Prometheus metrics in your instance or create Grafana dashboards of your own. Since this is a custom installation of these powerful tools, there is no limit to what you can accomplish with your new APM stack.
When you finish the experiments, cleanup is simple:
Once you've finished this Prometheus and Grafana on Kubernetes Ingress tutorial, you may find these other Kubernetes tutorials helpful:
Have questions or want to stay in touch with the Kong community? Join us wherever you hang out:
🌎 Join the Kong Community🍻 Join our Meetups💯 Apply to become a Kong Champion📺 Subscribe on YouTube🐦 Follow us on Twitter⭐ Star us on GitHub❓ ️Ask and answer questions on Kong Nation
We're very excited to announce Kong Mesh 2.12 to the world! Kong Mesh 2.12 delivers two very important features: SPIFFE / SPIRE support, which provides enterprise-class workload identity and trust models for your mesh, as well as a consistent Kuma R
Meet Emily. She’s an API product manager at ACME, Inc., an ecommerce company that runs on dozens of APIs. One morning, her team lead asks a simple question: “Who’s our top API consumer, and which of your APIs are causing the most issues right now?”
In the last two parts of this series, we discussed How to Strengthen a ReAct AI Agent with Kong AI Gateway and How to Build a Single-LLM AI Agent with Kong AI Gateway and LangGraph . In this third and final part, we're going to evolve the AI Agen
In my previous post, we discussed how we can implement a basic AI Agent with Kong AI Gateway. In part two of this series, we're going to review LangGraph fundamentals, rewrite the AI Agent and explore how Kong AI Gateway can be used to protect an LLM
This is part one of a series exploring how Kong AI Gateway can be used in an AI Agent development with LangGraph. The series comprises three parts: Basic ReAct AI Agent with Kong AI Gateway Single LLM ReAct AI Agent with Kong AI Gateway and LangGr
What Is RAG, and Why Should You Use It? RAG (Retrieval-Augmented Generation) is not a new concept in AI, and unsurprisingly, when talking to companies, everyone seems to have their own interpretation of how to implement it. So, let’s start with a r
In February 2024, Kong became the first API platform to launch a dedicated AI gateway, designed to bring production-grade performance, observability, and policy enforcement to GenAI workloads. At its core, Kong’s AI Gateway provides a universal API