We introduced the concept of Secrets Management in the Kong Gateway 2.8 release, and we’re happy to share that as of the recent Kong Gateway 3.0 release we’re giving it the Kong seal of approval! That means that you can rely on Secrets Management in production to manage all of your sensitive information.

Kong Gateway relies on lots of secrets to operate — everything from your database passwords to API keys used in plugins. You’ve previously been able to use Role Based Access Control (RBAC) to limit access to sensitive information in the admin API and Kong Manager, but it’s an “all or nothing” approach. Contributors can manage plugin configuration, or they can’t. Wouldn’t it be great if they could manage the configuration without seeing any secret values?

This is what Secrets Management enables.

Which Vaults are supported?

With this announcement, we officially support the following data sources for secrets:

• Environment variables (OSS)
• HashiCorp Vault (Enterprise)
• AWS Secrets Manager (Enterprise)
• Google Cloud Secrets Engine (Enterprise, Beta)

Kong abstracts each of the above systems into a set of nested keys. The only thing that changes is the vault identifier (hcv, aws or env) For example, to access the password field of the Postgres secret in the HashiCorp vault, you would use the following reference:

{vault://hcv/postgres/password}

The same secret stored in AWS Secrets Manager would look almost identical:

{vault://aws/postgres/password}

Finally, let’s take a look at what this would look like using the env vault:

export POSTGRES='{"username":"user", "password":"pass"}'
{vault://env/postgres/password}

As you can see, Kong supports setting a JSON payload to provide nesting whilst using environment variables.

Understanding “Referenceable”

In order to keep Kong Gateway performant, we’ve limited the fields that accept using vault references to refer to secrets. To help you understand where you can use values from a vault, we’ve tagged any fields that support secrets with “referenceable" in our plugin documentation.

As an example, take a look at the proxy-cache-advanced documentation and you’ll see the following in the config.redis.password description:

This field is referenceable, which means it can be securely stored as a secret in a vault. References must follow a specific format

This means that you can set a value of {vault://hcv/redis/password} and it will be resolved as expected.

Securing Redis with Secrets Management

We’ve done a lot of talking about Secrets Management, but what really made it click for me was to see an example. Let’s take a look at how to store our Redis password in HashiCorp Vault when using the Proxy Cache Advanced plugin.

Running Redis

As we’ll be using Redis, let’s start by running a server locally. I already have Redis installed, and start the server using a configuration file provided as stdin to set a server password:

echo 'requirepass demo' | redis-server -

Running HashiCorp Vault

Next, I need a Vault server running to store our secret. To keep things simple, I’m running the server on my local machine with vault server -dev which starts Vault, creates a new kv store named secret and returns the root key for authentication (which looks like hvs.x4abajxI7TWduo0GQMnd5N8Q in my case).

Once we have a vault, we need to store some data in there. Create a redis secret by running the following:

export VAULT_ADDR="http://localhost:8200"
vault kv put -mount secret redis password=demo

Start Kong Gateway using Docker

Once Vault is running, I need to run a Kong Gateway container locally. To do this, I followed the Docker instructions on the Kong documentation with one big change — I enabled the HashiCorp vault using environment variables by adding the KONG_VAULT_HCV_ values:

docker run -d --name kong-gateway \
  --network=kong-net \
…snip…
  -e KONG_LICENSE_DATA \
  -e KONG_VAULTS=bundled \
  -e KONG_VAULT_HCV_PROTOCOL=http \<
  -e KONG_VAULT_HCV_HOST="host.docker.internal" \
  -e KONG_VAULT_HCV_PORT=8200 \
  -e KONG_VAULT_HCV_MOUNT=secret \
  -e KONG_VAULT_HCV_KV=v2 \
  -e KONG_VAULT_HCV_TOKEN="YOUR_TOKEN"  \
…snip…
  kong/kong-gateway:3.0.0

At this point, we have everything we need to test out secret management!

Using Proxy Caching Advanced

We’ll be using the Proxy Caching Advanced plugin to test our vault configuration. To enable the plugin, we first need to create a service and a route. Let’s proxy our test requests to mockbin.org:

curl -i -X POST \
  --url http://localhost:8001/services/ \
  --data 'name=example-service' \
  --data 'url=http://mockbin.org'

curl -i http://localhost:8001/services/example-service/routes -d paths="/mock"

We’ll also need to configure the proxy-cache-advanced plugin. I’m using the default values from the documentation for most fields, but take a look at config.redis.password. This is where we reference the value from our vault:

curl localhost:8001/services/example-service/plugins \
--data "name=proxy-cache-advanced"  \
--data "config.response_code=200" \
--data "config.request_method=GET" \
--data "config.content_type=application/json; charset=utf-8" \
--data "config.cache_control=false" \
--data "config.strategy=redis" \
--data "config.redis.host=host.docker.internal" \
--data "config.redis.port=6379" \
--data "config.redis.password={vault://hcv/redis/password}"

Keep an eye on your Kong Gateway logs at this point, as they’ll contain an error if your vault isn’t responding correctly. Here’s an error I received after setting the wrong HCV_TOKEN:

unable to resolve reference {vault://hcv/redis/password}

Finally, it’s time to make a request to our route. The first time you make a request the response will come from mockbin.org and the X-Cache-Status header in the response will be Miss.

curl -i localhost:8000/mock/request/hello

If you make the same request again, the X-Cache-Status header will return Hit. You can check that the cache is being populated by checking the keys in Redis too:

echo "KEYS *" | redis-cli -a demo

Conclusion

Congratulations! You just learned how Secret Management works in Kong. Sensitive information is sensitive for a reason and using Kong’s vault functionality you can keep the values away from prying eyes.

The environment, HashiCorp Vault, and AWS Secrets Manager drivers are production-ready today, but there’s one more thing I wanted to share with you. We’re also announcing support for Google Cloud Secrets Engine, so if you’re a GCP user don’t worry – we’ve got you covered.

I’m excited about this release, and I hope you are too. If you’ve got any questions, you can find me on Twitter at @mheap or on the Kong Community Slack.