Using Kong Gateway to Adapt SOAP Services to the JSON World

While JSON-based APIs are ubiquitous in the API-centric world of today, many industries adapted internet-based protocols for automated information exchange way before REST and JSON became popular. One attempt to establish a standardized protocol suite for automated data interchange within and between enterprises was SOAP, originally an acronym for Simple Object Access Protocol.

SOAP APIs are similar to JSON APIs in that they use web technologies and protocols as transport mechanisms. But where modern APIs generally use JSON as the data format, SOAP uses XML. XML is more complex than JSON and generally requires specialized tools and libraries to be processed correctly and efficiently, while JSON is much more accessible and well-supported in all contemporary programming languages.

Enterprises that have invested in SOAP infrastructure are thus often faced with the challenge of making existing SOAP services available to clients that don't support XML. One way to go is to implement a protocol translator which provides clients with a JSON-based API that is automatically and transparently translated to SOAP invocations. This approach protects investments made in SOAP infrastructure while opening new integration opportunities with modern, JSON API-based systems.

How Kong Gateway can help

The Kong Gateway can support a number of different use cases for the transformation of client requests and server responses — and REST to SOAP is no exception! As the world's most popular gateway, the Kong Gateway is uniquely positioned to help with these kinds of transformations. Gateways typically broker traffic from external clients to internal services. This places Kong Gateway in the perfect position to offer these kinds of transformations to users.

In this post, we detail how Kong Gateway can be the translation layer between a REST (i.e., JSON) based client, and a SOAP-based backend. Approaching the problem this way helps organizations modernize their APIs without reworking their entire backend architecture. This approach should minimize the effort and cost required to ensure SOAP-based backends can work with REST-based clients.

Before diving into the details, we'll provide a brief primer on XML (i.e., SOAP) and JSON (i.e., REST) and the tools typically used to manipulate XML. From there, we'll show you how to start modernizing SOAP APIs to return JSON using a Celcius to Fahrenheit converter SOAP service.

How XML and JSON differ

JSON is a fairly simple data format. It only supports a couple of atomic data types, arrays, and hash tables. This makes it easy to process in most programming languages and converting a JSON document into a native data structure of a given language is usually a matter of calling a single function.

XML, on the other hand, is a complex format that can be used for a variety of use cases and provides a lot of options as to how data can be represented in a document.

Here's how the same information structure can be represented in two different ways in XML, and how each of the encodings require different methods to access the same information in the document.

By the nature of its extensibility and flexibility, mapping XML documents to a native, easy-to-use data structure of a programming language isn't generally possible. This also means that a direct, generic translation from XML to JSON or back isn't a viable solution when implementing a protocol translator.

Processing XML using XSLT

As processing XML is often found to be inconvenient using conventional programming languages, XSLT has been developed as a specialized language to process and transform XML. There's an old joke: "XML is like violence: If it does not work, use more of it." In that vein, XSLT itself is based on XML, and XSLT programs are also XML documents.

Here's a little XSLT program that performs the same function as the previous example:

This example also shows some additional complexity of XML that we haven't discussed so far: there's a preamble line that defines the file's encoding (UTF-8), and elements are prefixed with namespace identifiers, which allow elements from multiple applications to be mixed in one document.

While the XSLT program may seem convoluted, the language is very efficient at processing XML documents and has several features that make it suitable for tasks like the one we're interested in, protocol translation.

A taste of SOAP

SOAP, like XSLT, defines an XML vocabulary — not for specifying transformation programs, but for API interactions. Requests to a SOAP server and its responses follow strictly defined schemas. These schemas (you guessed it) are again XML documents that can be used to automatically validate documents, similar to what JSON Schema can do to JSON documents. SOAP requests and responses are XML documents that contain an envelope part with protocol information defined by the SOAP Schema itself, and a body part that contains application-specific data. To be compliant with SOAP standards, applications need to publish request and response XML Schema definitions, which allows clients, servers, and intermediate systems to validate documents as they are processed.

For the purpose of this blog post, we're going to implement a protocol translator for a simple demonstration web service that converts between degrees Celsius and Fahrenheit. A request to the SOAP service to convert 20 degrees Celsius to Fahrenheit will look like this:

Note that in this simple demonstration. The SOAP envelope doesn't contain any additional elements. In real-world applications, it would typically carry authentication and other protocol-level information.

This is what the corresponding response of the SOAP server would look like:

A modern API

Our goal is to create a protocol translator that makes this SOAP-based service available as a JSON-based API that can be used like so:

It will be the task of a custom Kong plugin to translate between the modern API and the upstream SOAP service.

Introducing Saxon

For our demonstration, we're going to use Saxon, an implementation of XSLT that has been around for several decades and that has been updated to support each new language standard as it became available. Saxon is supported by the company Saxonica and is available both as open source and commercially supported editions. It's written in Java, but the SaxonC version can be integrated into non-JVM programs. For the purpose of this blog post, we're using the "Home Edition" of SaxonC which can be downloaded and used without cost.

Transforming JSON requests to SOAP

To process JSON using XSLT, the JSON document is converted to XML so that it can be easily processed. The conversion is performed by the fn:json-to-xml function, which is part of the XSLT 3.0 standard library. Given this JSON document:

fn:json-to-xml returns this XML document:

Notice how the map element has an "xmlns" attribute, which defines its own namespace and that of its children. Using the correct namespace is required when processing the document with XML.

Here is the XSLT transformation that converts from JSON to a SOAP request:

You'll notice that multiple namespace prefixes are declared and used in this piece of code. The "xsl" namespace is the actual XSLT program, the "fn" namespace contains utility functions like json-to-xml and "soap12" is the name service that describes the SOAP WebService infrastructure elements. Finally, a default namespace with the URL "https://www.w3schools.com/xml/" is declared at the "CelsiusToFahrenheit" element which is also used for all unqualified child elements, and it defines the elements used by this particular application. Inside of the "Celsius" element, the "xsl:value-of" element is used to evaluate an XPath expression at run time. This is part of how templating works in XSLT.

Transforming SOAP responses to JSON

The response transformation works in the inverse way: it converts the SOAP response to an XML document representing the JSON response and then calls the "fn:xml-to-json" function to perform the conversion to a JSON string. A template is called when a specific element of the response is encountered to perform the conversion.

JSON<>XML and back in the Kong Gateway

Now that we know how we can transform between JSON and SOAP using XSLT, we need to figure out how we can make things happen inside of Kong Gateway.

This is done using a custom plugin. The plugin code loads the ffi and a glue library which makes SaxonC's C++ API available as C functions. It then declares the functions present in that library using "ffi.cdef" so that they can be called from Lua code:

With this, we're ready to take a look at the actual plugin code that runs in the proxy path when a request is being handled by Kong Gateway. In the "access" phase, the request body as sent by the proxy client is converted from JSON to a SOAP request before being passed to the upstream server:

In the "body_filter" phase, the SOAP response is converted back to JSON before it is passed to the client:

Wrapping it up

The most popular way to run Kong Gateway is by using a Docker container, and that's also our preferred method of distributing releases. When custom plugins like our protocol converter are used, a custom container image that includes the required shared external libraries and the plugin code needs to be created. In our case, there also is the need to compile the C++ to C glue code as part of the container-building process. All the required steps can be automated in a single Dockerfile that builds the custom container image.

Summary

This post serves as a demonstration of how Kong Gateway can be used with XSLT to implement a protocol converter to make a SOAP Web Service available to modern JSON API clients. If you're interested in learning more and seeing the whole codebase that enables this functionality, please reach out to our support team.