For the last couple of years, Retrieval-Augmented Generation (RAG) architectures have become a rising trend for AI-based applications. Generally speaking, RAG offers a solution to some of the limitations in traditional generative AI models, such as accuracy and hallucinations, allowing companies to create more contextually relevant AI applications. The combination of retrieval-based systems like RAG and generative models such as Large Language Models (LLMs) will likely become more prevalent as companies seek to leverage their data for their new AI applications.

This blog post describes a RAG implementation using LangChain as the application orchestrator, driving Amazon Bedrock, protected by Kong AI Gateway, and Redis as the vector database.

## Kong AI Gateway Introduction

Last February, Kong announced Kong Gateway 3.6 with the [first version](https://konghq.com/blog/product-releases/announcing-kong-ai-gateway)first version of Kong AI Gateway with brand new capabilities to address AI-based use cases like multi-LLM (Large Language Models) integrations, Prompt Engineering, including templating, decoration and prompt guard, Request and Response Transformation based on LLM responses, AI analytics, and more. The next [Kong Gateway 3.7 version](https://konghq.com/blog/product-releases/ai-gateway-goes-ga)Kong Gateway 3.7 version, launched last May, included new features, such as support for OpenAI SDK, Rate Limiting policies based on tokens, streaming support, etc.

Now, with the latest [Kong Gateway 3.8 release](https://konghq.com/blog/product-releases/ai-gateway-3-8)Kong Gateway 3.8 release, announced during our [API Summit 2024](https://konghq.com/events/conferences/api-summit)API Summit 2024, new critical capabilities have been released:

• - Semantic caching and prompt guard.
• - Six different Load Balancing algorithms, including semantic routing, to support multiple LLMs.
• - Official support to [Amazon Bedrock](https://aws.amazon.com/bedrock)Amazon Bedrock and GCP Vertex/Gemini (in addition to all the others).

The Kong AI Gateway provides AI infrastructure for every team building new GenAI applications, and it combines the Kong API Gateway features with a set of AI-focused plugins that allow us to ensure data security and privacy, response quality and accuracy, and finally data and infra readiness for AI, while also providing out of the box observability for our LLM traffic and the ability to manage access tiers to the LLMs. This approach allows applications to adopt LLM infrastructures within their existing API traffic flow without having to rebuild common requirements that all AI applications need when they go into production. 

The diagram below represents the architecture:

*The Kong AI Gateway sits in between the GenAI applications we build and the LLMs we consume.*

### RAG Applications and LLM Frameworks

Concurrently, customers started to leverage their AI infrastructures to build LLM-based applications. One of the main processes used for such applications is Retrieval-Augmented Generation, or RAG, for short.

Fundamentally, LLMs are trained, for a long period of time, on massive public datasets. Although that turns into a powerful solution, it lacks context. That's where RAG comes in: [as described in its original paper](https://arxiv.org/abs/2005.11401)as described in its original paper, it enhances the capabilities of LLMs by incorporating internal and domain-specific customer data optimizing the LLM output with a knowledge base outside the data sources used for the LLM training. That assimilation is solved during query time, meaning it's done without retraining the model.

The picture describes the process:

Basically, the RAG application comprises two main processes:

1. - The RAG application converts the Prompt into an embedding, calling the Embedding Model.
2. - Leveraging Semantic Search, RAG matches the Prompt Embedding with the most relevant information and retrieves that Vector Database.
3. - The Vector Database returns relevant data as a response to the search query.
4. - The RAG application sends a query to the Prompt/Chat LLM Model combining the Prompt with the Relevant Data returned by the Vector Database.
5. - The LLM Model returns a response.

As you can see, there are multiple components in a typical RAG application implementation. You might see a need for a new one responsible for orchestrating the components. That's the main responsibility of emerging LLM frameworks like [LangChain](https://www.langchain.com/langchain)LangChain.

AWS provides nice introductions about [RAG](https://aws.amazon.com/what-is/retrieval-augmented-generation/)RAG and [LangChain](https://aws.amazon.com/what-is/langchain/)LangChain.

## Kong AI Gateway and Amazon Bedrock

We should get started discussing the benefits Kong AI Gateway brings to Amazon Bedrock for an LLM-based application. As we stated before, Kong AI Gateway leverages the existing Kong API Gateway extensibility model to provide specific AI-based plugins. Here are some values provided by Kong AI Gateway and its plugins:

• - AI Proxy and AI Proxy Advanced plugins: the Multi-LLM capability allows the AI Gateway to abstract Amazon Bedrock (and other LLMs as well) load balancing models based on several policies including latency time, model usage, semantics etc. These plugins extract LLM observability metrics (like number of requests, latencies, and errors for each LLM provider) and the number of incoming prompt tokens and outgoing response tokens. All this is in addition to hundreds of metrics already provided by Kong Gateway on the underlying API requests and responses. Finally, Kong AI Gateway leverages the observability capabilities provided by Konnect to track Amazon Bedrock usage out of the box, as well as generate reports based on monitoring data.
• -

  Prompt Engineering:

  • - AI Prompt Template plugin, responsible for pre-configuring AI prompts to users
  • - AI Prompt Decorator plugin, which injects messages at the start or end of a caller's chat history.
  • - AI Prompt Guard plugin lets you configure a series of PCRE-compatible regular expressions to allow and block specific prompts, words, phrases, or otherwise and have more control over a LLM service, controlled by Amazon Bedrock.
  • - AI Semantic Prompt Guard plugin to self-configurable semantic (or pattern-matching) prompt protection.
• - AI Semantic Cache plugin caches responses based on threshold, to improve performance (and therefore end-user experience) and cost.
• - AI Rate Limiting Advanced, you can tailor per-user or per-model policies based on the tokens returned by the LLM provider under Amazon Bedrock management or craft a custom function to count the tokens for requests.
• - AI Request Transformer and AI Response Transformer plugins seamlessly integrate with the LLM on Amazon Bedrock, enabling introspection and transformation of the request's body before proxying it to the Upstream Service and prior to forwarding the response to the client.

Besides, the Kong AI Gateway use cases can combine policies implemented by hundreds of Kong Gateway plugins, such as:

• - Authentication and Authorization: OIDC, mTLS, API Key, LDAP, SAML, Open Policy Agent (OPA)
• - Traffic Control: Request Validator and Size Limiting, WebSocket support, Route by Header, etc. 
• - Observability: OpenTelemetry (OTel), Prometheus, Zipkin, etc.

Also, from the architecture perspective, in a nutshell, the Konnect Control Plane and Data Plane nodes topology remains the same.

*By leveraging the same underlying core of Kong Gateway, we're reducing complexity in deploying the AI Gateway capabilities as well. And of course, it works on Konnect, Kubernetes, self-hosted, or across multiple clouds.*

### Amazon Bedrock Foundation Models

This blog post consumes some [Foundation Models](https://aws.amazon.com/what-is/foundation-models/)Foundation Models, including Prompt/Chat and Embeddings Models. Please, make sure you have requested access to the following models:

• - meta.llama3-70b-instruct-v1:0
• - amazon.titan-embed-text-v1

## Implementation Architecture

The RAG implementation architecture should include components representing and responsible for the functional scope described above. The architecture comprises:

• - Redis as the Vector Database
• - Amazon Bedrock supporting both Prompt/Chat and Embeddings Models
• - Kong AI Gateway to abstract and protect Amazon Bedrock as well as implement LLM-based use cases.

Kong AI Gateway, implemented as a regular Konnect Data Plane Node, and Redis run on an Amazon EKS 3.1 Cluster.

The architecture can be analyzed from both perspectives: Data Preparation and actual Query time. Let's take a look at the Data Preparation time. 

### Data Preparation time

The Data Preparation time is quite simple: the Document Loader component sends data chunks, based on the data taken, and calls Amazon Bedrock asking the Embedding Model to convert them into embeddings. The embeddings are then stored in the Redis Vector Database. Here's a diagram describing the flow:

For the purpose of this blog post, we're going to use the "amazon.titan-embed-text-v1" Embedding Model.

### Query time

Logically speaking we can break the Query time into some main steps:

1. - The AI Application builds a Prompt and, using the same "amazon.titan-embed-text-v1" Embedding Model, generates embeddings to it.
2. - The AI Application takes the embeddings and semantically searches the Vector Database. The database returns relevant data related to the context.
3. - The AI Application sends the prompt, augmented with the retrieved data received from the Vector Database, to Kong AI Gateway.
4. - Kong AI Gateway, using the AI Proxy (or AI Proxy Advanced) plugin, proxies the request to Amazon Bedrock and returns the response to the AI Application. Kong AI Gateway enforces eventual policies previously enabled.

## Kong AI Gateway and Amazon Bedrock Integration and APIs

Let's take a look at the specific integration point between Kong AI Gateway and Amazon Bedrock. To get a better understanding, here's the architecture cut isolating both:

The consumer can be any RESTful-based component, in our case will be a LangChain application.

As you can see, there are two important topics here:

• - OpenAI API specification
• - Amazon Bedrock Converse API and EKS Pod Identity

Let's discuss each one of them.

### OpenAI API support

Kong AI Gateway supports the [OpenAI API specification](https://platform.openai.com/docs/overview)OpenAI API specification. That means the consumer can send standard OpenAI requests to the Kong AI Gateway. As a basic example, consider this OpenAI request:

curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "Hello!"
      }
    ]
  }'

When we add Kong AI Gateway, sitting in front of Amazon Bedrock, we're not just exposing it but also allowing the consumers to use the same mechanism — in this case, OpenAI APIs — to consume it. That leads to a very flexible and powerful capability when we come to development processes. In other words, Kong AI Gateway normalizes the consumption of any LLM infrastructure, including Amazon Bedrock, Mistral, OpenAI, Cohere, etc.

As an exercise, the new request should be something like this. The request has some minor differences:

• - It sends a request to the Kong API Gateway Data Plane Node.
• - It replaces the OpenAI endpoint with a Kong API Gateway route.
• - The API Key is actually managed by the Kong API Gateway now.
• - We're using an Amazon Bedrock Model, meta.llama3-70b-instruct-v1:0.

curl http://$DATA_PLANE_LB/bedrock-route \
  -H "Content-Type: application/json" \
  -H "apikey: $KONG_API_KEY" \
  -d '{
     "model": "meta.llama3-70b-instruct-v1:0",
     "messages": [
       {
         "role": "user",
         "content": "Hello!"
       }
     ]
   }'

### Amazon EKS Pod Identity

The Konnect Data Plane Node, where the Kong AI Gateway runs, has to send requests to Amazon Bedrock on behalf of the Gateway consumer. In order to do it, we need to grant permissions to the Data Plane deployment to access the Amazon Bedrock API, more precisely the [Converse API](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html)Converse API, used by the AI Gateway to interact with it.

For example, here's a request to Amazon Bedrock, using the AWS CLI. Use "aws configure" first to set your Access and Secret Key as well as the AWS region you want to use then run the command. 

aws bedrock-runtime converse \
  --model-id amazon.titan-text-express-v1 \
  --messages '[{"role":"user","content":[{"text":"Tell me about John Coltrane"}]}]'

It's reasonable to consume Bedrock service with local CLI commands. However, for Amazon EKS deployments, the recommended approach is [EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html)EKS Pod Identity, instead of simple long-term credentials like AWS Access and Secret Keys. In a nutshell, EKS Pod Identity allows the Data Plane Pod's container to use the AWS SDK and send API requests to AWS services using AWS Identity and Access Management (IAM) permissions.

Amazon EKS Pod Identity associations provide the ability to manage credentials for your applications, similar to the way that Amazon EC2 instance profiles provide credentials to Amazon EC2 instances.

As another best practice, we recommend the Private Key and Digital Certificate pair used by the Konnect Control Plane and the Data Plane connectivity to be stored in AWS Secrets Manager. In this sense, the Data Plane deployment refers to the secrets to get installed.

## Amazon Elastic Kubernetes Service (EKS) installation and preparation

Now, we're ready to get started with our Kong AI Gateway deployment. As the installation architecture defines, it will be running on an EKS Cluster.

In order to create the EKS Cluster, you can use [eksctl](https://eksctl.io/)eksctl, the official CLI for Amazon EKS, like this:

eksctl create cluster --name konnectaigateway-eks131 \
  --version 1.31 \
  --region us-east-2 \
  --nodegroup-name kong-node \
  --node-type g6.xlarge \
  --nodes 1

Note command creates an EKS Cluster, version 1.31, with a single node based on the g6.xlarge instance type, powered by NVIDIA GPUs. That is particularly interesting if you're planning to deploy and run LLMs locally in the EKS Cluster.

### Cluster preparation

After the installation, we should prepare the Cluster to receive the other components:

• - [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/)AWS Load Balancer Controller to expose both Kong AI Gateway with public [Network Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html)Network Load Balancers (NLB).
• - [EKS Pod Identity Agent](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html)EKS Pod Identity Agent to be able to define Pod Identity Association and grant permissions to the Kong AI Gateway Data Plane Pod to access both Amazon Bedrock and AWS Secrets Manager.

Please, refer to the official documentation to learn more about the components and their installation processes.

## Redis installation

As defined in the implementation architecture, Redis plays the Vector Database role. It's going to be deployed and running in a specific EKS namespace and exposed with an NLB.

For its deployment, we're going to be using its [Helm Charts](https://github.com/redis-stack)Helm Charts. Get started adding its repo:

helm repo add redis-stack https://redis-stack.github.io/helm-redis-stack
help repo update

helm install redis-stack redis-stack/redis-stack -n redis --create-namespace

Expose the Redis Service with an ELB:

kubectl patch service redis-stack -n redis -p "{\"spec\": { \"type\" : \"LoadBalancer\"}}"

Get the ELB hostname with:

% kubectl get service redis-stack -n redis -o json | jq -r '.status.loadBalancer.ingress[].hostname'
a7c63f0c9a740483987398a6c0b6e444-967754848.us-east-2.elb.amazonaws.com

You can check Redis Server with a simple command:

% kubectl exec $(kubectl get pod -n redis -o json | jq -r '.items[].metadata.name') -n redis -- redis-server --version
Redis server v=7.4.1 sha=00000000:0 malloc=jemalloc-5.3.0 bits=64 build=5918102922f65a0d

Redis' dashboard is also available:

[http://a7c63f0c9a740483987398a6c0b6e444-967754848.us-east-2.elb.amazonaws.com:8001](http://a7c63f0c9a740483987398a6c0b6e444-967754848.us-east-2.elb.amazonaws.com:8001)http://a7c63f0c9a740483987398a6c0b6e444-967754848.us-east-2.elb.amazonaws.com:8001

## [](http://a7c63f0c9a740483987398a6c0b6e444-967754848.us-east-2.elb.amazonaws.com:8001)Kong Data Plane installation

The second component to be installed is the Kong AI Gateway. The process can be divided into two steps:

• - Pod Identity configuration
• - Kong Data Plane deployment

### Pod Identity configuration

In this first step, we configure EKS Pod Identity describing which AWS Services the Data Plane Pods should be allowed to access. In our case, we need to consume Amazon Bedrock and AWS Secrets Manager.

#### IAM Policy

Pod Identity relies on IAM policies to check which AWS Services can be consumed. Our policy should allow access to AWS Bedrock actions so the Data Plane will be able to send requests to Bedrock APIs, more precisely, [Converse](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html)Converse and [ConverseStream](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ConverseStream.html)ConverseStream APIs. The Converse API requires permission to the InvokeModel action as ConverserStream needs access to InvokeModelWithResponseStream.

Also, we're going to use AWS Secrets Manager to store our Private Key and Digital Certificate pair the Konnect Control Plane and Data Plane used to communicate.

Considering all this, let's create the IAM policy with the following request:

aws iam create-policy \
 --policy-name bedrock-policy \
 --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "bedrock:InvokeModel",
                "bedrock:InvokeModelWithResponseStream",
                "secretsmanager:ListSecrets",
                "secretsmanager:GetSecretValue"
            ],
            "Resource": "*"
        }
    ]
}'

#### Pod Identity Association

Pod Identity takes a Kubernetes Service Account to manage the permissions. So create the Kubernetes namespace for the Kong Data Plane deployment and a simple Service Account inside of it.

kubectl create namespace kong
kubectl create sa kaigateway-podid-sa -n kong

Now we're ready to create the Pod Identity Association. We use the same eksctl command to do it:

eksctl create podidentityassociation \
  --cluster konnectaigateway-eks131 \
  --region us-east-2 \
  --namespace kong \
  --service-account-name kaigateway-podid-sa \
  --role-name kaigateway-podid-role \
  --permission-policy-arns arn:aws:iam::<your_aws_account>:policy/bedrock-policy

The command above is responsible for:

• - IAM Role creation based on the IAM Policy we previously defined
• - Associating the IAM Role to the existing Kubernetes Service Account

You can check the Pod Identity Association with:

eksctl get podidentityassociation \
  --cluster konnectaigateway-eks131 \
  --region us-east-2 \
  --namespace kong

Check the IAM Role and Policies attached with:

aws iam get-role --role-name kaigateway-podid-role

aws iam list-attached-role-policies --role-name kaigateway-podid-role

### Kong Data Plane deployment

The Data Plane deployment comprises the following steps:

• - Konnect subscription and Control Plane creation
• - Create a Konnect Vault for AWS Secrets Manager
• - Private Key and Digital Certification pair creation
• - Store the pair in AWS Secrets Manager
• - Data Plane deployment

#### Konnect subscription and Control Plane creation

This fundamental step is required to get access to Konnect. Click on the [Registration](https://konghq.com/products/kong-konnect/register)Registration link and present your credentials. Or, if you already have a Konnect subscription, [log in to it](https://cloud.konghq.com/us)log in to it.

Any Konnect subscription has a "default" Control Plane defined. You can proceed using it or optionally create a new one. The following instructions are based on a new Control Plane. You can create a new one by clicking on the "Gateway Manager" menu option. Then click on "New Control Plane" and choose the Kong Gateway option. Inside the "Kong Gateway" page type "AI Gateway" for the name and leave the "Self-Managed Hybrid Instances" option set.

#### Create a Konnect Vault for AWS Secrets Manager

Now, inside the new "AI Gateway" Control Plane, click on the "Vaults" menu option and create a Vault referring to AWS Secret Manager where the Private Key and Digital Certificate pair is going to be stored. Inside the "New Vault" page, define "Ohio (us-east-2)" for Region and "aws-secrets" for Prefix. Click on "Save."

#### Private Key and Digital Certificate pair creation

Still inside the "AI Gateway" Control Plane, click on "Data Plane Nodes" menu option and "New Data Plane Node" button.

Choose Kubernetes as your platform. Click on "Generate certificate", copy and save the Digital Certificate and Private Key as tls.crt and tls.key files as described in the instructions. Also copy and save the configuration parameters in a values.yaml file.

#### Store the Private Key and Digital pair in AWS Secrets Manager

Store the pair in AWS Secret Manager with the following commands:

aws secretsmanager create-secret --name kongcp1-crt --region us-east-2 --secret-string "{\"cert\": \"$(cat ./tls.crt)\"}"

aws secretsmanager create-secret --name kongcp1-key --region us-east-2 --secret-string "{\"key\": \"$(cat ./tls.key)\"}"

#### Data Plane deployment

Finally, the last step will deploy the Data Plane. Take the values.yaml file and change it adding the AWS Secret Manager references and the Kubernetes Service Account used by the Pod Identity Association. The updated version should be something like this. Replace the cluster_* endpoints and server names with yours:

image:
  repository: kong/kong-gateway
  tag: "3.8"

admin:
  enabled: false

manager:
  enabled: false

env:
  role: data_plane
  database: "off"
  cluster_mtls: pki
  cluster_control_plane: abc71c0977.us.cp0.konghq.com:443
  cluster_server_name: abc71c0977.us.cp0.konghq.com
  cluster_telemetry_endpoint: abc71c0977.us.tp0.konghq.com:443
  cluster_telemetry_server_name: abc71c0977.us.tp0.konghq.com
  cluster_cert: "{vault://aws/kongcp1-crt/cert}"
  cluster_cert_key: "{vault://aws/kongcp1-key/key}"
  lua_ssl_trusted_certificate: system
  konnect_mode: "on"
  vitals: "off"
  nginx_worker_processes: "1"
  upstream_keepalive_max_requests: "100000"
  nginx_http_keepalive_requests: "100000"
  proxy_access_log: "off"
  dns_stale_ttl: "3600"

ingressController:
  enabled: false
  installCRDs: false

resources:
  requests:
    cpu: 1
    memory: "2Gi"


proxy:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

deployment:
  serviceAccount:
    create: false
    name: kaigateway-podid-sa

Basically, the changes are:

• - Since the Private Key and Digital Certificate pair has been stored in AWS Secrets, remove the "secretVolumes" section.
• - Replace the "cluster_cert" and "cluster_cert_key" fields with Konnect Vault references and AWS Secret Manager secrets.
• - Add the proxy annotation to request a public NLB for the Data Plane.
• - Add the deployment section referring to the Kubernetes Service Account that has been used to create the Pod Identity Association.

Use the Helm command to deploy the Data Plane:

helm repo add kong https://charts.konghq.com
helm repo update

helm install kong kong/kong -n kong --values ./values.yaml

### Checking the Data Plane

Use the Load Balancer created during the deployment:

DATA_PLANE_LB=$(kubectl get service -n kong kong-kong-proxy --output=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

You should get a response like this:

% curl -i $DATA_PLANE_LB
HTTP/1.1 404 Not Found
Date: Thu, 03 Oct 2024 12:36:53 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Content-Length: 103
X-Kong-Response-Latency: 0
Server: kong/3.8.0.0-enterprise-edition
X-Kong-Request-Id: c655f65a2d1fef7c7ed0550e3f7dc553

{
  "message":"no Route matched with those values",
  "request_id":"c655f65a2d1fef7c7ed0550e3f7dc553"
}       

Now we can define the Kong Objects necessary to expose and control Bedrock, including Kong Gateway Service, Routes, and Plugins.

## decK

With [decK](https://docs.konghq.com/deck/latest/)decK (declarations for Kong) you can manage Kong Konnect configuration and create Kong Objects in a declarative way. decK state files describe the configuration of Kong API Gateway. State files encapsulate the complete configuration of Kong in a declarative format, including services, routes, plugins, consumers, and other entities that define how requests are processed and routed through Kong. Please check the decK documentation to learn how to [install](https://docs.konghq.com/deck/latest/installation/)install it.

### Konnect PAT

Once you have decK installed, you should ping Konnect to check if the connection is up. In order to do it you need a [Konnect Personal Access Token (PAT)](https://docs.konghq.com/konnect/gateway-manager/declarative-config/#generate-a-personal-access-token)Konnect Personal Access Token (PAT). To generate your PAT, go to Konnect UI, click on your initials in the upper right corner of the Konnect home page, then select "Personal Access Tokens." Click on "+ Generate Token," name your PAT, set its expiration time, and be sure to copy and save it as an environment variable also named as PAT. Konnect won’t display your PAT again.

You can ping Konnect with:

% deck gateway ping --konnect-token $PAT
Successfully Konnected to the Kong organization!

### Kong declarations for the RAG Application

Now, create a file named "kong_bedrock.yaml" with the following content:

cat > kong_bedrock.yaml << 'EOF'
_format_version: "3.0"
_info:
  select_tags:
  - rag-bedrock
_konnect:
  control_plane_name: AI Gateway
services:
- name: bedrock-rag-service
  host: httpbin.default
  port: 8000
  routes:
  - name: bedrock-rag-route1
    paths:
    - /bedrock-route
    plugins:
    - name: ai-proxy
      instance_name: "ai-proxy-bedrock-rag"
      enabled: true
      config:
        auth:
          param_name: "allow_override"
          param_value: "false"
          param_location: "body"
        route_type: "llm/v1/chat"
        model:
          provider: "bedrock"
          options:
            bedrock:
              aws_region: "us-east-1"
    - name: ai-prompt-decorator
      instance_name: ai-prompt-decorator-rag
      enabled: false
      config:
        prompts:
          prepend:
          - role: system
            content: "You will always respond in the Portuguese (Brazil) language."
    - name: key-auth
      instance_name: key-auth-bedrock
      enabled: true
consumers:
- keyauth_credentials:
  - key: "123456"
  username: user1
  plugins:
  - name: ai-rate-limiting-advanced
    instance_name: ai-rate-limiting-advanced-consumer
    enabled: false
    config:
      llm_providers:
      - name: bedrock
        window_size: 60
        limit: 200
EOF

The declaration defines multiple Kong Objects:

• - [Kong Gateway Service](https://docs.konghq.com/gateway/latest/get-started/services-and-routes/)Kong Gateway Service named "bedrock-rag-service". The service doesn’t need to map to any real upstream URL. In fact, it can point somewhere empty, for example, http://localhost:32000. This is because the AI Proxy plugin, also configured in the declaration, overwrites the upstream URL. This requirement will be removed in a later Kong revision.
• - Kong Route: the Gateway Service has a route defined with the "/bedrock-route" path. That's the route we're going to consume to reach out to Bedrock.
• -

  Kong Route Plugins: the Kong Route has some plugins configured. Note that only the AI Proxy and Key Auth Plugins are enabled. The other ones are configured but disabled.

  • - Kong [AI Proxy Plugin](https://docs.konghq.com/hub/kong-inc/ai-proxy/)AI Proxy Plugin: that's the Plugin that allows us to connect to the LLM infrastructure. For [Bedrock](https://docs.konghq.com/hub/kong-inc/ai-proxy/how-to/machine-learning-platform-integration-guides/bedrock/)Bedrock, among other things, we need to configure which AWS region we should connect to.
  • - Kong [AI Prompt Decorator Plugin](https://docs.konghq.com/hub/kong-inc/ai-prompt-decorator/)AI Prompt Decorator Plugin: the Plugin instructs Bedrock to manage the prompt and response in a particular way.
  • - Kong [Key Auth Plugin](https://docs.konghq.com/hub/kong-inc/key-auth/)Key Auth Plugin: the Plugin requires the consumer to inject an API Key, named by default "apikey", to the requests in order to consume the Kong Route.
• - Kong Consumer: the declaration also defines a Kong Consumer, named "user1" with an API Key based credential. The Kong Consumer will allow the Kong AI Gateway to map the incoming requests with the API Key "apikey" (required by the Key Auth Plugin) with the credential "123456" to the Kong Consumer "user1". Once the Kong Consumer is mapped, the Kong AI Gateway can apply policies, defined with other Plugins, to this specific Kong Consumer.
• - Kong Consumer Plugins: the Kong Consumer "user1" has the [AI Rate Limiting Advanced Plugin](https://docs.konghq.com/hub/kong-inc/ai-rate-limiting-advanced/)AI Rate Limiting Advanced Plugin enabled. This Plugin, an extension of the standard Kong API Gateway [Rate Limiting Advanced Plugin](https://docs.konghq.com/hub/kong-inc/rate-limiting-advanced/)Rate Limiting Advanced Plugin, implements rate limiting policy based on the number of tokens returned by the LLM provider.

The declaration has been tagged as "rag-bedrock" so you can manage its objects without impacting any other ones you might have created previously. Also, note the declaration is saying it should be applied to the "AI Gateway" Konnect Control Plane.

You can submit the declaration with the following decK command:

$ deck gateway sync --konnect-token $PAT kong_bedrock.yaml
creating consumer user1
creating service bedrock-rag-service
creating key-auth 123456 for consumer user1
creating route bedrock-rag-route1
creating plugin key-auth for route bedrock-rag-route1
creating plugin ai-prompt-decorator for route bedrock-rag-route1
creating plugin ai-proxy for route bedrock-rag-route1
creating plugin ai-rate-limiting-advanced for consumer user1
Summary:
  Created: 8
  Updated: 0
  Deleted: 0

### Consume the Kong AI Gateway Route

You can now send a request consuming the Kong Route. Note we have injected the API Key with the expected name, "apikey", and with a value that corresponds to the Kong Consumer, "123456". If you change the value or the API Key name, or even not add it, you get a 401 (Unauthorized) error.

curl -s -X POST  http://$DATA_PLANE_LB/bedrock-route \
  -H "Content-Type: application/json" \
  -H "apikey: 123456" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "What is niilism?"
      }
    ],
    "model": "meta.llama3-70b-instruct-v1:0"
  }' | jq

To enable the other Plugins you can change the declaration as "enabled: true" or use the Konnect UI.

## LangChain

With all the components we need for our RAG application in place, it's time for the most exciting part of this blog post: the RAG application.

The minimalist app we're going to present focuses on the main steps of the RAG processes. It's not intended to be used other than learning processes and maybe lab environments.

Basically, the application comprises two Python scripts, using [LangChain](https://www.langchain.com/langchain)LangChain, one of the main frameworks available today to write AI applications including LLMs and RAG. The scripts are the following:

• - **rag_redis_bedrock_langchain_embeddings.py**: it implements the Data Preparation process with Redis and Bedrock's "amazon.titan-embed-text-v1" Embedding Model.
• - **rag_redis_bedrock_langchain_kong_ai_gateway.py**: responsible for the Query time implementation combining Kong AI Gateway, Redis, and Bedrock.

From the RAG perspective, the application is based on an interview Marco Palladino, Kong's co-founder and CTO, gave discussing Kong AI Gateway and our current technology environment.

### Data Preparation

The first thing the code does is to download the transcript of the interview in a local file. Then it uses the LangChain Community's [DirectoryLoader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.directory.DirectoryLoader.html)DirectoryLoader and [TextLoader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.text.TextLoader.html)TextLoader as well as [CharacterTextSplitter](https://python.langchain.com/api_reference/text_splitters/character/langchain_text_splitters.character.CharacterTextSplitter.html)CharacterTextSplitter Python packages to load and break the original text into chunks.

Next, the code uses the [BedrockEmbeddings](https://python.langchain.com/api_reference/community/embeddings/langchain_community.embeddings.bedrock.BedrockEmbeddings.html)BedrockEmbeddings and [Redis](https://python.langchain.com/api_reference/community/vectorstores/langchain_community.vectorstores.redis.base.Redis.html)Redis packages, also provided by LangChain, to generate the embeddings for each chunk and store all of them in Redis Vector Database's collection named "docs-bedrock". Note we hit Redis through the ELB we provisioned during its deployment. Note also the script forces the index recreation.

Before running the code make sure you have a Python3 environment set and the following packages installed:

• - requests
• - langchain
• - langchain-community
• - langchain-aws
• - langchain-redis

It's important to note that this Python script assumes you have your AWS credentials set locally in order to connect to Bedrock. Please refer to the [Boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html)Boto3 documentation to learn how to manage them.

You can run the script with:

python3 rag_redis_bedrock_langchain_embeddings.py

After the execution, you can check the Redis database using its dashboard or with the following command. You should see 52 points keys:

% kubectl exec $(kubectl get pod -n redis -o json | jq -r '.items[].metadata.name') -n redis -- redis-cli info keyspace
# Keyspace
db0:keys=52,expires=0,avg_ttl=0,subexpiry=0

#### Python script code

Here's the code of the first Python script:

import requests

#from langchain_redis import RedisConfig, RedisVectorStore
from langchain_community.vectorstores.redis import Redis
from langchain_aws import BedrockEmbeddings

from langchain.text_splitter import CharacterTextSplitter
from langchain_community.document_loaders import TextLoader
from langchain_community.document_loaders import DirectoryLoader


# Download transcript text and split it into chunks
# https://softwareengineeringdaily.com/2024/06/20/its-apis-all-the-way-down-with-marco-palladino

url = "http://softwareengineeringdaily.com/wp-content/uploads/2024/06/SED1683-Kong.txt"
headers = {'User-Agent': 'LangChain'}
res = requests.get(url, headers=headers)
with open("ragdoc.txt", "w") as f:
    f.write(res.text)

loader = DirectoryLoader(".", "ragdoc.txt", loader_cls=TextLoader)
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=50)
chunks = text_splitter.split_documents(documents)


# Generate embeddings based on the text chunks using Bedrock's Embedding Model and store them in Redis

redis_url = "redis://a7c63f0c9a740483987398a6c0b6e444-967754848.us-east-2.elb.amazonaws.com:6379"
embeddings = BedrockEmbeddings(region_name="us-east-1", model_id="amazon.titan-embed-text-v1")
index_name = "docs-bedrock"


vectorstore = Redis(
    redis_url=redis_url,
    embedding=embeddings,
    index_name=index_name,
)

vectorstore.drop_index(redis_url=redis_url, index_name=index_name, delete_documents=True)

vectorstore.add_documents(documents=chunks)

### LangChain Chain

Before exploring the second Python script, responsible for the actual RAG Query, it's important to get a better understanding of what a LangChain Chain is. Here's a simple example including Kong AI Gateway and Amazon Bedrock. Note that the code does not implement RAG, restricted to explore the Chain concept. The next section will evolve the example to include RAG.

The code requires the installation of the "langchain_openai" package.

from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

kong_dp = "http://k8s-kong-kongkong-cf4df736b2-f8ba17426cc444cc.elb.us-east-2.amazonaws.com"
bedrock_rag_url = kong_dp + "/bedrock-route"

prompt = ChatPromptTemplate.from_template("tell me what you think about {topic}")

llm = ChatOpenAI(base_url=bedrock_rag_url, model_name="meta.llama3-70b-instruct-v1:0", temperature=0, api_key="dummy", default_headers={"apikey": "123456"})


chain = prompt | llm


print("\nanswer: ", chain.invoke({"topic": "Ozempic"}).content)
print("\nanswer: ", chain.invoke({"topic": "Fantastic Negrito"}).content)

The most important line of the code is the creation of the LangChain Chain: chain = prompt | llm.

A Chain, the main LangChain component, can be created with [LCEL](https://python.langchain.com/docs/concepts/#langchain-expression-language-lcel)LCEL (LangChain Expression Language), a declarative language. In fact, a Chain is responsible for executing a sequence of steps, "chained" together and separated by the [Pipe ("|") Operator](https://python.langchain.com/docs/tutorials/llm_chain/)Pipe ("|") Operator. All steps of a Chain implement the [Runnable](https://python.langchain.com/api_reference/core/runnables/langchain_core.runnables.base.Runnable.html)Runnable LangChain Class.

In our case, the Chain takes the output of the Prompt, passes it to the LLM, which, as expected, sends the prompt to the LLM and gets the response.

Specifically, for the prompt object, it has been created with a {topic} template. So it needs to be informed by the chain.invoke Method.

On the other hand,  the llm object actually refers to Kong AI Gateway. As we stated before, Kong AI Gateway supports the OpenAI API specification, meaning that the consumer, in our case the LangChain code, should interact with the Gateway through the standard LangChain [ChatOpenAI chat model class](https://python.langchain.com/api_reference/openai/chat_models/langchain_openai.chat_models.base.ChatOpenAI.html)ChatOpenAI chat model class.

Here's the line:

llm = ChatOpenAI(base_url=bedrock_rag_url, model_name="meta.llama3-70b-instruct-v1:0", temperature=0, api_key="dummy", default_headers={"apikey": "123456"})

Note that the Class refers to the Kong AI Gateway Route. We're asking the Gateway to consume the Bedrock model meta.llama3-70b-instruct-v1:0. Besides, since the Route has the Key Auth plugin enabled, we've added the API Key using the default_headers parameter. Finally, the standard and required api_key parameter should be passed but it's totally ignored by the Gateway.

### RAG Query

Now, let's evolve the script to add RAG. This time, LangChain Chain will be updated to include the Vector Database.

First, remember that, in a RAG application, the Query should combine a Prompt and contextual and relevant data, related to the Prompt, coming from the Vector Database. Something like this:

Considering this flow we should implement a parallel processing with both sources. That's what the [LangChain's RunnableParallel](https://python.langchain.com/v0.2/api_reference/core/runnables/langchain_core.runnables.base.RunnableParallel.html)LangChain's RunnableParallel class does. So, the new Chain would look like this:

And the code snippet with the new Chain should be like this. Fundamentally, the Chain has included two new steps: one before and another after the original prompt and llm steps.

rag_chain = (
    retriever_prompt_setup
    | prompt 
    | llm
    | StrOutputParser() 
)

So let's review all of them. However, before jumping to the Chain, we have to describe a bit how it is going to hit the Redis Vector Database to get the Relevant Data coming from it.

#### retriever

The first line gets the reference to the Redis index we created in the first Python script.

vectorstore = Redis(
    redis_url = "redis://a7c63f0c9a740483987398a6c0b6e444-967754848.us-east-2.elb.amazonaws.com:6379",
    embedding = BedrockEmbeddings(region_name="us-east-1", model_id="amazon.titan-embed-text-v1"),
    index_name = "docs-bedrock",
)

The second line calls the ["as_retriever"](https://python.langchain.com/api_reference/community/vectorstores/langchain_community.vectorstores.redis.base.Redis.html#langchain_community.vectorstores.redis.base.Redis.as_retriever)"as_retriever" method of the Redis class so we can start sending queries to Redis with a [RedisVectorStoreRetriever](https://python.langchain.com/api_reference/community/vectorstores/langchain_community.vectorstores.redis.base.RedisVectorStoreRetriever.html#langchain_community.vectorstores.redis.base.RedisVectorStoreRetriever)RedisVectorStoreRetriever object. For example, here's the "retriever" object construction set with "similarity distance threshold" search type:

retriever = vectorstore.as_retriever(
    search_type="similarity_distance_threshold",
    search_kwargs={"distance_threshold": 0.6}
)

print("Redis query: \n" + str(retriever.invoke(input="what did Marco say about Ingress Controller?")[0]))

Check this [how-to guide](https://python.langchain.com/docs/how_to/vectorstore_retriever/)how-to guide to learn more about Retrievers.

### New LangChain Chain

Let's check the steps of the new Chain now.

#### retriever_prompt_setup

retriever_prompt_setup = RunnableParallel({"context": retriever, "question": RunnablePassthrough()})

The line calls the [RunnableParallel](https://python.langchain.com/v0.2/api_reference/core/runnables/langchain_core.runnables.base.RunnableParallel.html)RunnableParallel class which launches, in parallel, one task per parameter:

• -

  "context": refers to the retriever which is, in our case, the Redis Vector Database.

• -

  "question": uses the [RunnablePassthrough](https://python.langchain.com/v0.2/api_reference/core/runnables/langchain_core.runnables.passthrough.RunnablePassthrough.html)RunnablePassthrough class which takes the Chain's input and passes it with no updates. The RunnablePassthrough() call basically says the "question" parameter is not defined yet but it will be provided eventually. Check the following [how-to guide](https://python.langchain.com/docs/how_to/passthrough/)how-to guide to learn more about the RunnablePassthrough class

Both RunnableParallel and RunnablePassthrough classes are subclasses of the fundamental [Runnable](https://python.langchain.com/v0.2/api_reference/core/runnables/langchain_core.runnables.base.Runnable.html)Runnable class.

The Chain's flow will proceed only when the two tasks have been completed. The next step, "prompt", will receive both data as follows:

{"context": <data_coming_from_Redis>, "question": <question_from_the_chain's_invoke_method>}

#### prompt

Considering that it will receive the output of the "retriever_prompt_setup" step, it should be modified. Here it is:

prompt = ChatPromptTemplate.from_template(template)

And the template needs to refer to both context and question:

template = """
Answer the question based only on the following context:
{context}
Question: {question}
"""

#### llm

The "llm" step remains exactly the same. It calls the Kong AI Gateway route which is responsible for routing the request to Amazon Bedrock:

llm = ChatOpenAI(base_url=bedrock_rag_url, model_name="meta.llama3-70b-instruct-v1:0", temperature=0, api_key="dummy", default_headers={"apikey": "123456"})

#### StrOutputParser()

The [StrOutParser](https://python.langchain.com/api_reference/core/output_parsers/langchain_core.output_parsers.string.StrOutputParser.html)StrOutParser class parses the output coming from the "llm" step in a readable way.

### Invoking the Chain

We're now ready to invoke the Chain. Here's one example:

query = "What did Marco say about AI Gateway?"
print("\nRAG query: " + query)
print("\nRAG answer: ", rag_chain.invoke(query))

Of course, the response will be totally contextualized considering the data we have in our Vector Database.

LangChain provides a nice [tutorial](https://python.langchain.com/docs/tutorials/rag/)tutorial on how to build a RAG application.

## Kong AI Gateway Plugins

Once we have the RAG Application done, it'd be interesting to start enabling the AI Plugins we have created.

### Kong AI Prompt Decorator plugin

For example, besides the Key Auth plugin, you can enable the AI Prompt Decorator, configured with decK, to see the response in Brazilian Portuguese.

You can use decK again, if you will or go to Konnect UI and look for the Kong Route's Plugins page. Use the toggle button to enable the Plugin.

### Kong AI Rate Limiting Advanced plugin

You can also enable the AI Rate Limiting Advanced plugin we have created for the user. Notice that, differently to the AI Prompt Decorator plugin, this plugin has been applied to the existing Kong Consumer created by the decK declaration. If you create another Kong Consumer (and a different Credential), the AI Rate Limiting plugin will not be enforced.

Also note that the AI Rate Limiting Advanced defines policies based on the number of tokens returned by the LLM provider, not the number of requests as we usually have for regular rate limiting plugins.

### Python script code

Here's the code of the second Python script:

from langchain_community.vectorstores.redis import Redis
from langchain_aws import BedrockEmbeddings

from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

from langchain_core.runnables import RunnablePassthrough, RunnableParallel
from langchain_core.output_parsers import StrOutputParser



kong_dp = "http://k8s-kong-kongkong-af48daf85f-3e01263abaeeb50a.elb.us-east-2.amazonaws.com"
bedrock_rag_url = kong_dp + "/bedrock-route"
redis_url = "redis://a7c63f0c9a740483987398a6c0b6e444-967754848.us-east-2.elb.amazonaws.com:6379"
embeddings = BedrockEmbeddings(region_name="us-east-1", model_id="amazon.titan-embed-text-v1")
index_name = "docs-bedrock"


# Step 1: Get Redis Index

vectorstore = Redis(
    redis_url=redis_url,
    embedding=embeddings,
    index_name=index_name,
)


# Step 2: Retrieve & Augment
# Once you have the vector database reference, you can define it as the retriever component,
# which fetches the additional context based on the semantic similarity between the user query and the embedded chunks.

retriever = vectorstore.as_retriever(
    search_type="similarity_distance_threshold",
    search_kwargs={"distance_threshold": 0.6}
)

# Next, to augment the prompt with the additional context, you need to prepare a prompt template.
# The prompt can be easily customized from a prompt template, as shown below.

template = """
Answer the question based only on the following context:
{context}
Question: {question}
"""

prompt = ChatPromptTemplate.from_template(template)



# Step 3: Generate
# Finally, you can build a chain for the RAG pipeline, chaining together the retriever, the prompt template and the LLM.
# Once the RAG chain is defined, you can invoke it.

llm = ChatOpenAI(base_url=bedrock_rag_url, model_name="meta.llama3-70b-instruct-v1:0", temperature=0, api_key="dummy", default_headers={"apikey": "123456"})


retriever_prompt_setup = RunnableParallel({"context": retriever, "question": RunnablePassthrough()})

rag_chain = (
    retriever_prompt_setup
    | prompt 
    | llm
    | StrOutputParser()
)



query = "What did Marco say about AI Gateway?"
print("\nRAG query: " + query)
print("\nRAG answer: ", rag_chain.invoke(query))


query = "Is there any reference to Kubernetes in the context?"
print("\nRAG query: " + query)
print("\nRAG answer: ", rag_chain.invoke(query))


query = "What did MP say about Chopin?"
print("\nRAG query: " + query)
print("\nRAG answer: ", rag_chain.invoke(query))

## Conclusion

This blog post has presented a basic RAG Application using Kong AI Gateway, LangChain, Amazon Bedrock, and Redis as the Vector Database. It's totally feasible to implement advanced RAG apps with query transformation, multiple data sources, multiple retrieval stages, RAG Agents, etc. Moreover, Kong AI Gateway provides other plugins to enrich the relationship with the LLM providers, including Semantic Cache, Semantic Routing, Request and Response Transformation, etc.

Also, it's important to keep in mind that, just like we did with the Key Auth plugins, we can continue combining other API Gateway plugins to your AI-based use cases like using the OIDC plugin to secure your Foundation Models with AWS Cognito, using the Prometheus plugin to monitor your AI Gateway with Amazon Managed Prometheus and Grafana, and so on.

Finally, the architect flexibility provided natively by Konnect and Kong API Gateway allows us to deploy the Data Planes in a variety of platforms including AWS EC2 VMs, Amazon ECS, and Kong Dedicated Cloud Gateway, Kong's SaaS service for the Data Planes running in AWS.

You can discover all the features available on the [Kong AI Gateway](https://konghq.com/products/kong-ai-gateway)Kong AI Gateway product page, or you can check the Kong and AWS landing page at [https://konghq.com/partners/aws](https://konghq.com/partners/aws)https://konghq.com/partners/aws to learn more.