4 Ways to Leverage Kong’s jq Plugin
As part of the Kong Gateway 2.6 release, we shipped a brand new jq plugin for anyone with an enterprise license to use. It’s like we combined the request and response transformer plugins to form a single, more powerful plugin—supercharging the way we work with request and response bodies.
If you're not familiar with jq, it's a JSON processing language that allows you to manipulate any JSON document and transform it however you need. We won't cover everything that jq can do in this post, but if you'd like a step-by-step introduction, I recommend this post on earthly.dev.
The Kong Gateway jq plugin allows you to run your transforms conditionally, depending on the request or response media type (it defaults to application/json) and response code. This means that you won't try to transform a response body if you get an error response by default.
To configure your jq filter, you need to provide the request_jq_program or response_jq_program options in your plugin config, or you can specify both to transform both the request and the response.
The Kong documentation has a comprehensive overview of the available options, which can be provided to the admin API as an HTTP request or via declarative configuration. Here's a sample declarative configuration file that uses jq to extract the title key from multiple items in a response:
Or using the admin API to configure the plugin:
Example Use Cases
The best way to learn jq is to see it used in some real-world scenarios. In this post, we'll cover three real-world scenarios and one fun example to show the power of jq:
- Transform a request body from one structure to another
- Remove sensitive information from a response
- Convert a JSON API to CSV
- Convert Celsius to Fahrenheit (this is the fun one!)
Let's dive in with how to restructure a request body before sending it to the upstream.
1. Transform a Request Body to Maintain Backwards Compatibility
Imagine that we've shipped an API to a handful of early access customers, and they've integrated it and are using the API in production. Then disaster strikes—we realize that we've made a typo in one of the field names! We accidentally named a field referer rather than referrer.
How could we fix this without impacting our early access customers? jq to the rescue!
By assigning a field using = and the del() function, we can create a new field containing the value of referer and delete the old key. This has the effect of renaming a key in the request body:
Although this program contains spaces, there is no need to encode the request:
2. Delete Sensitive Information From a Response
The next scenario is a backend team built an API for us, but they've left in some sensitive information that we don't want to share with customers. We asked for it to be removed, but the API should have been launched last week, and we can't wait another two weeks for the team to make the changes.
We want to launch today but only show the standard pricing field:
How do we remove these sensitive fields from the response dynamically? Once again,jq to the rescue! There are a couple of ways to achieve what we want to do. The first is to call del() multiple times:
This works but isn't the most efficient way to accomplish our goal. Instead, we can step in to pricing then delete multiple keys within that object:
Once we apply this transform using response_jq_program, we'll get the following response body:
The data we wanted to remove is gone, and we can ship our new API today!
3. Convert a JSON Response to CSV
Imagine that we built a lovely JSON API for our customers that works great, but then one of our valuable customers comes back to us saying, "But it doesn't work in my spreadsheets! Can you provide the data as CSV too?"
We don't want to invest more development time for just one customer, but we want to make them happy. How do we tackle it? That's right! We can use jq.
In this example, we'll be returning a list of sessions presented at Kong Summit 2021 through an API. Here's a sample of what the API looks like:
This is a slightly more complicated processing pipeline than the examples shown so far, so we'll build it up section by section. The first row we need to output is a list of field names. We can do this by fetching the keys from the first entry in the array:
Next, we need to process the payload to remove any complex data structures as they can't be represented in a CSV. In our API, the only complex data is the list of presenters, which we'll convert to a string by mapping over each entry and joining the list with a comma:
Now that the data only contains strings, we need to remove all key names. We can do that using map and $keys. It looks odd, but what we're saying is "add a row containing all of the available keys, then return a list of values, one for each available key":
At this point, our data looks like the following:
The final thing to do is to remove the wrapper array and convert it to CSV:
Putting it all together, we get the following:
Which returns a CSV response:
We're almost there, but not quite! We don't want to return a CSV payload to everyone, so we need to register a new route using Kong. We expect consumers to send an Accept: text/csv header to get the CSV response. Here's the configuration I used to create this route:
Finally, we have to enable the jq plugin on our new route. We can use the jq expression above as config.request_jq_program like normal, but we need a few other configuration options when working with CSV. The default options for jq are to compress the output and escape any quotes. As we're returning CSV data, we don't want this to happen, so we need to set config.response_jq_program_options.compact_output to false and config.response_jq_program_options.raw_output to true.
Once we set those configuration options, we're all good to go! We now have an endpoint that returns CSV data for our high-profile customer with zero development effort.
4. Convert Celsius to Fahrenheit
One final example before you go and try the jq plugin out yourself.
Imagine this—we've got a wildly successful greenhouse temperature monitoring product that's working really well in the EU, and we want to launch the product in the US. We're unsure if it will do as well, so we do some initial user testing. Feedback is positive except for one thing—they'd prefer to see the temperature in Fahrenheit rather than Celsius.
As it's a small batch of testers that we're very hands on with, we decide to convert on the fly using jq to validate our hypothesis rather than getting our engineering team to build the functionality. An API that returns data in the following format powers our application:
Our UI uses the units and temperature from the API, so we need to rewrite those two fields in both current and previous to be in Fahrenheit. Once again, we'll be using map to apply to each entry in the response.
This will convert the temperature and units values in every key in the returned object. This works for the simple use case above, but what if our response was a little more complex? What if it also contained a location and an account ID?
Using the jq expression above would result in an error:
Thankfully, jq also supports if statements that allow us to run our transform conditionally. Here's a transformation that checks that the units key has a value of celsius before trying to convert to Fahrenheit. If it doesn't equal celsius then it returns the original object:
Once we apply this jq plugin to a specific route that we're using in our beta testing, we can go ahead and test with US customers once again - this time with the units they're familiar with.
Give It a Go!
That's all we've got time for today, but hopefully, it's given you an idea of how the jq plugin works. You can learn about the available plugin options on Kong's plugin hub or learn more about jq itself by reading the jq manual.
We'd love to hear more about what you're doing with jq and Kong. You can tweet us at @thekonginc, and who knows, maybe we'll get together and do a jq themed episode of Kong Builders.