In this blog post, we will explore how organizations can leverage Kong and OpenTelemetry to establish and monitor Service Level Objectives (SLOs) and manage error budgets more effectively. By tracking performance metrics and error rates against predefined thresholds, teams can prioritize their work based on the impact on user experience and business objectives. This approach helps optimize the balance between innovation and reliability.

To achieve our goal of tracking SLOs, we will utilize Kong Gateway in conjunction with the OpenTelemetry plugin, OTEL collector, and Dynatrace.

## What is an SLA, SLI, and SLO?

Before diving in, it's essential to understand the definitions of SLA, SLI, and SLO, and why they are important to grasp before we delve into the practical implementation of "Tracking SLOs using Kong and OpenTelemetry."

### Service level agreement (SLA)

Often, you might hear questions like, "What is the SLA to resolve this ticket? Should it be resolved within an hour, a day, or a week?" The answers depend on the severity of the ticket and the resolution time defined in the SLA document. An SLA is a formal agreement between a service provider and a customer that outlines the expected level of service. In our context, we will define SLAs for API/service availability and error rates.

### Service level indicator (SLI)

When implementing services using APIs, we measure their performance using metrics such as latency, throughput, and error rates to ensure the application system meets customer expectations. An SLI is a quantitative metric used to measure the performance of a service. SLIs can be expressed as rates, averages, percentiles, etc.

### Service level objective (SLO)

Once we define the SLIs, we need to set target values (thresholds) or ranges of target values that represent the desired level of service expected by the customer. For example, the number of errors in a month should be less than 1%. We will explore detailed examples in the following sections.

## Understanding components required to track SLO

To track SLO using Kong and OpenTelemetry, we need to understand the components involved. Below is an overview of the end-to-end flow: 

• - **Design overview**: The design illustrates the flow of SLO tracking using Kong, OpenTelemetry, and Dynatrace. You can use any supported OpenTelemetry (OTel) collector and backend servers to collect and capture traces, logs, and metrics. Kong provides excellent documentation on supported [OpenTelemetry collectors and backends](https://docs.konghq.com/hub/kong-inc/opentelemetry/how-to/getting-started/)OpenTelemetry collectors and backends.

• - **API exposure through Kong:** APIs are exposed through the Kong Enterprise gateway, and we use the OpenTelemetry protocol to generate logs, traces, and metrics for our APIs. To achieve this, we need to add Kong's either globally or locally (specific to a route or service). In the plugin configuration, we specify the endpoints for logs and traces receivers. Don't worry, I will show you the practical steps in the next section.
• - **OTel Collector components**: The OTel Collector consists of three main parts: receiver, processor, and exporter. The receiver collects logs, traces, and metrics from the Kong OpenTelemetry plugin. The processor performs necessary transformations, such as generating metrics from traces. Finally, the exporter sends the logs, traces, and/or metrics to the OTel backend server. In our case, we are using the Dynatrace collector. To enable metrics, we need to use the supported [contrib versions](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter)contrib versions of the collector.
• - **Visualization and monitoring with Dynatrace**: To visualize and monitor the logs, traces, and performance metrics of our APIs, we will install Dynatrace. We can monitor logs and traces using Dynatrace's [Logs](https://docs.dynatrace.com/docs/analyze-explore-automate/logs/lma-analysis/logs-app)Logs and [Distributed Traces](https://docs.dynatrace.com/docs/analyze-explore-automate/distributed-traces/analysis/get-started)Distributed Traces applications. Additionally, we can set up email alerts, for example, if the error count reaches a defined threshold.

 Enough about the theoretical part — now it's time to perform some practical steps and see everything in action. Are you ready? 

## Setting up SLO tracking platform using Kong, OpenTelemetry, and Dynatrace

We've already discussed the importance and functionality of each component and tool required to track SLOs using Kong and OpenTelemetry. Now, let's get hands-on with the practical steps. 

### Define threshold for SLO

First things first, we need to define the thresholds (SLOs) for the APIs. These definitions should align with customer expectations and organizational SLAs. For simplicity, I've defined two SLOs:

• - **Request Latency**: We have set a target value 600ms for request latency per API. This means we expect the processing time for each API in our application to be no more than 600 milliseconds.
• - **Availability**: We have defined a 99% availability metric per hour for our application. This means that, on average, all APIs should not be unavailable for more than 1%. For example, if our system receives 100 API requests in an hour and only 1 request fails, this represents a 1% failure rate, which is acceptable according to our Availability SLO.

Now, let's proceed with the installation. The diagram below shows the installation sequence.

### Install Dynatrace

We assume you already have Dynatrace installed. If not, you can install Dynatrace on your server or create an account on the SaaS version of Dynatrace. 

Since the installation of Dynatrace is not our main focus, I will skip the detailed installation steps. However, you can follow these steps to set up your SaaS Dynatrace account:

1. - **Visit the Dynatrace website**
2. - **Start your free trial**: Click on the "Start free trial" button. You will be prompted to fill out a registration form with your details.
3. - **Fill out the registration form**: Provide the necessary information, such as your name, email address, company name, and other relevant details.
4. - **Verify your email**: After submitting the form, you will receive a verification email. Click on the link in the email to verify your account.
5. - **Set up your environment**: Once your account is verified, you can log in to the Dynatrace platform and start setting up your monitoring environment.

After installing Dynatrace, you should have Dynatrace Account with URL [https://ENV_ID.apps.dynatrace.com/](https://env_id.apps.dynatrace.com/)https://ENV_ID.apps.dynatrace.com/. Here, `ENV_ID` is your Dynatrace environment ID

### Install Dynatrace OTel Collector

I am using an Ubuntu-based EC2 instance and Docker compose to install the Dynatrace OTel Collector. Follow the steps below to install the appropriate Dynatrace OTel Collector:

**docker-compose.yml**

version: "3.8"
 services:
   dynatrace-otel-collector:
 	image: ghcr.io/dynatrace/dynatrace-otel-collector/dynatrace-otel-collector:0.21.0
 	command: ["--config=/etc/otelcol/otel-collector-config.yaml"]
 	volumes:
   	- ./otel-collector-config.yaml:/etc/otelcol/otel-collector-config.yaml
 	ports:
   	- "4317:4317"   # OTLP gRPC  We are using only HTTP protocol, so you can ignore gRPC
   	- "4318:4318"   # OTLP HTTP 

**otel-collector-config.yaml**

receivers:
   otlp:
 	protocols:
   	http:
     	endpoint: 0.0.0.0:4318  ## OTEL backend server. Data is being received at this server. You can replace 0.0.0.0 with appropriate hostname
exporters:
   otlphttp:
 	endpoint: "https://DYNATRACE_ENVIRONMENT_ID.live.dynatrace.com/api/v2/otlp"	
 	## Endpoint is a Dynatrace exporter endpoint, data is being exported at Dynatrace
 	## DYNATRACE_ENVIRONMENT_ID can be found in the environment section of your Dynatrace profile
 	headers:
   	"Authorization": "Api-Token API_TOKEN_FROM_DYNATRACE"
 	## You have to create an API access token in Dynatrace.
service:
   pipelines:
 	traces:
   	receivers: [otlp]
   	processors: []
   	exporters: [otlphttp]
 	logs:
   	receivers: [otlp]
   	processors: []
   	exporters: [otlphttp]

Run the following command to start the Dynatrace OTel Collector:

docker-compose up -d

You should see output like this:

[+] Running 1/2
  ⠧ Network dynatrace-collector_default                                     Created                                                               0.7s
  ✔ Container dynatrace-collector-dynatrace-otel-collector-1  Started                                                                0.6s

Verify the running container:

docker ps

You should see output like this:

CONTAINER ID   IMAGE            	COMMAND              	CREATED     	STATUS     	PORTS                             	NAMES
 d8780a812137   ghcr.io/dynatrace/dynatrace-otel-collector/dynatrace-otel-collector:0.21.0   "/dynatrace-otel-col…"   5 seconds ago   Up 4 seconds   0.0.0.0:4317-4318->4317-4318/tcp, :::4317-4318->4317-4318/tcp   dynatrace-collector-dynatrace-otel-collector-1

For detailed documentation on how to install the Dynatrace collector, refer to the Dynatrace Collector [Official Documentation](https://docs.dynatrace.com/docs/ingest-from/opentelemetry/collector)Official Documentation.

Note: Ensure that port 4318 is accessible from the Kong Gateway server. In this case, I modified the EC2 Security Group to open port 4318 for inbound traffic.

### Configure Kong Gateway

Assuming you already have Kong Gateway installed, you need to modify the kong.conf file with the following parameters and redeploy the Kong Gateway. This enables tracing in Kong Gateway and should be done before using the OpenTelemetry plugin.

 vi /etc/kong/kong.conf
tracing_instrumentations = all
 tracing_sampling_rate = 1.0

Save and exit the file:

:wq!

Restart Kong:

kong restart -c /etc/kong/kong.conf

To capture debug-level logs, you can also enable:

log_level = debug

If you have made all the changes correctly, great job! Now it's time to add the OpenTelemetry plugin to a route, service, or at the global level.

### Configure Route/Service with OpenTelemetry Plugin

Add the OpenTelemetry plugin to your route or service using Kong Manager or the Admin API. Here is the Admin API command to do so: 

curl -X POST https://KONG_HOSTNAME:PORT/services/{serviceName|serviceId|routeName|routeId}/plugins \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --data '
    {
  	"name": "opentelemetry",
  	"config": {
    	"traces_endpoint": "http://COLLECTOR_HOSTNAME:4318/v1/traces",
    	"logs_endpoint": "http://COLLECTOR_HOSTNAME:4318/v1/logs"
  	}
    }

Ensure that the logs and traces endpoints are OpenTelemetry collector endpoints and not the direct endpoints of Dynatrace.

### Tracking performance and error count as per SLO

I configured the OpenTelemetry plugin for my "HealthService" service and triggered a few requests through Postman. Now, it's time to check if the traces are being captured in the Dynatrace "**Distributed Traces**" application service, as shown below.

• - By monitoring the performance of the above APIs, we noticed that three API requests had processing times greater than 600ms. This means these service requests are not reliable according to our defined **Request Latency SLO.**
• - By monitoring the logs of the Kong Gateway through Dynatrace's "**Logs**" application service, as shown below, we noticed several 404 error messages. This clearly indicates that these service requests are not reliable according to our defined **Availability SLO**.

• - Based on the above monitoring, relevant teams need to prioritize API performance improvement work as soon as possible.
• - We can also use Dynatrace's "**Service Level Objective Definitions**" service to automate SLO monitoring, or we can use any other supported tools which your organization prefers.

## Conclusion

Let's summarize what we have achieved:

1. - **Understanding SLA, SLI, and SLO**: We began by learning the definitions and importance of Service Level Agreements (SLA), Service Level Indicators (SLI), and Service Level Objectives (SLO).
2. - **Components required to track SLOs**: We explored the necessary components for tracking SLOs, including Kong, OpenTelemetry, and Dynatrace.
3. - **Defining SLOs**: We defined target values for Request Latency and Availability.
4. - **Installation and configuration**: We installed Dynatrace, set up the OpenTelemetry Collector, and modified the Kong configuration to export logs, traces, and metrics to Dynatrace.
5. - **Tracking and monitoring**: Finally, we tracked, visualized, and monitored the defined service performance according to the SLOs.

In summary, organizations can effectively use Kong Gateway with the OpenTelemetry plugin and monitoring and alerting tools to track and manage SLOs, ensuring a balance between innovation and reliability.