This post highlights how you could use Kong Gateway to implement a solution for the Australian Consumer Data Standards (CDS), which is part of the Consumer Data Right legislation introduced by the Australian Government in November 2017.

As detailed on the Australian ACCC website:

CDR will give consumers greater access to and control over their data and will improve consumers’ ability to compare and switch between products and services. It will encourage competition between service providers, leading not only to better prices for customers but also more innovative products and services. CDR will first apply to the banking sector, followed by the energy sector. The telecommunications sector is currently proposed to follow.  

Kong Gateway covers many of the CDS requirements using out-of-the-box features in conjunction with using Kong plugins, such as the OpenID Connect (OIDC) and CORS. 

This blog post will focus on specific features provided by Kong to support CDS and CDR, including CDS Versioning and CDS Traffic Thresholds. These implementation details target the initial rollout of CDS to the banking sector. Some of Kong’s banking customers in Australia are currently using these CDS artifacts in production. We can also utilize this solution to support telecommunications and energy sectors in the future.

CDS Standards and Non-Functional Requirements

Two major CDS requirements that are the focus of this blog post are Traffic Thresholds and Versioning, as detailed in the Standards and Non-functional Requirements sections of the Consumer Data Standards.

Traffic Thresholds

Calls above the CDS Traffic Thresholds will be freely throttled or rejected. Limits are based on the number of individual sessions per day, transactions per second (TPS) and calls for a specified duration. These limits are applied independently for different consumer types, as explained below.

Data Recipient is a configured application that is presented in the register metadata. This acknowledges that a single accredited entity may register multiple independent services (or apps) that can obtain authorizations from consumers independently of each other.

A Session is defined as the lifespan of a unique Access Token. Multiple API requests made with a single valid Access Token would be considered part of a single Session.

Customer Present are authenticated API requests made in direct response to interactions by the end customer using the digital services of the data recipient. Technically, a data holder will define an API request as “Customer Present” if the “x-fapi-customer-ip-address” header is populated with a valid IP address of the end customer’s device.

Customer Not Present is authenticated API requests that are not deemed to be “Customer Present.”

Unattended is a synonym of “Customer Not Present.”

Kong Implementation: Kong CDS Traffic Thresholds Plugin

Kong’s CDS Traffic Thresholds plugin extends the enterprise functionality of the Advanced Rate Limiting plugin to cater to the CDS Traffic Threshold requirements. This plugin implements the sliding window algorithm for high accuracy rate limits. See this blog post for more information on the benefits of the sliding window algorithm.

The high-level logic for determining both rate and session limits is detailed below. Note that for security, the plugin expects HTTP headers to be passed in as internal variables using kong.ctx.shared, and in the case of the session ID, hashed so that it never appears as clear text.

1. Check for the existence of a Customer ID. If this does not exist, then apply Maximum Public TPS because it’s Public Traffic.
2. If a Customer ID exists, check for both Session ID and Data Recipient. If none exists, then reject the request.
3. Check the request is classified as “Attended.” Distinguishing between the customer (including unattended) vs. public traffic is done by looking for the presence of a “x-fapi-customer-ip-address” header. If present, apply the Maximum Attended Customer TSP limit.
4. If the request is “Unattended,” check if the request was received during a defined high traffic period, and if not, apply the maximum Unattended Low Session Low TPS limit.
5. For all of the above, keep track of the number of “Sessions” for Customers and Data Recipients and apply limits.

The following table displays the available plugin schema properties applicable to the CDS Session and rate limits described above:

CDS Traffic Thresholds defines the following limits:

• Number of sessions per day–the number of individual sessions initiated on a calendar day.
• Transactions per second (TPS)–the number of concurrent transactions each second.
• Number of calls–the number of endpoint calls initiated for a specified duration.

Customer present and authorization traffic thresholds:

• Unlimited sessions per day
• 10 TPS per customer
• 50 TPS per data recipient

Unattended traffic thresholds will apply for low traffic periods:

• 20 sessions per day, per customer, per data recipient
• 100 total calls per session
• 5 TPS per session
• 50 TPS per data recipient

For Unattended traffic during high-traffic periods, only best-effort support is required.

Secure traffic (both Customer Present and Unattended) thresholds:

• 300 TPS total across all consumers

Public traffic (i.e., traffic to unauthenticated endpoints) thresholds:

• 300 TPS total across all consumers (additive to secure traffic)

Here’s how easy it is to apply the above limits to the plugin:

curl -s http://localhost:8001/services/example-service/plugins/ \
   -d name=cds-traffic-thresholds \
   -d config.max_attended_customer_tps=10 \
   -d config.max_data_recipient_tps=50 \
   -d config.max_unattended_sessions_low=100 \
   -d config.max_unattended_per_session_low=20 \
   -d config.max_unattended_session_tps_low=5 \
   -d config.max_private_tps=300 \
   -d config.max_public_tps=300 \
   -d config.sync_rate=0 \
   -d config.redis.host=kong-ee-redis \
   -d config.redis.port=6379 \

Endpoint Versioning

The CDS has adopted a two-level versioning strategy. The high-level standards, including principles, Uniform Resource Identifier (URI) structure, endpoint versioning, payload naming conventions, etc., can be versioned. Each API endpoint will have an additional version specific to that endpoint. For endpoint versioning, each endpoint will have multiple versions independent of other endpoints. A consumer will request a specific endpoint using an HTTP header:

x-v (Mandatory)

• A version of the API endpoint requested by the consumer. Must be set to a positive integer. The holder should respond with the highest supported version between x-min-v and x-v. If the value of x-min-v is equal to or higher than the value of x-v, then the x-min-v header should be treated as absent.
• If all versions requested are not supported, then the holder must respond with a 406 Not Acceptable.

x-min-v (Optional)

• Minimum version of the API endpoint requested by the client. Must be set to a positive integer if provided. The holder should respond with the highest supported version between x-min-v and x-v. If the value of x-min-v is equal to or higher than the value of x-v, then the x-min-v header should be treated as absent.
• If all versions requested are not supported, then the holder must respond with a 406 Not Acceptable

Kong Implementation: Kong CDS Versioning Plugin

The plugin works similarly to the Kong Route By Header plugin in that it overrides the normal upstream selection logic based on request HTTP headers.

The CDS requirement is to support multiple major and minor versions. In facilitate request routing, major (x-v) and minor (x-min-v) version header values will determine the exact application route based on a logic, such as, if x-min-v > x-v  is an error case, while invalid x-v and valid x-min-v responds with resource supporting x-min-v. This logic is built into the CDS versioning plugin.

The following table displays the available plugin schema properties applicable to the CDS Versioning requirements described above:

This curl command shows how straightforward it is to configure the plugin:

curl -i http://localhost:8001/routes/example-v1/plugins \
   -H 'Content-Type: application/json' \
   -d '{"name": "cds-versioning", "config": {"versions":[{"major": 1, "minor": 3, "upstream_name": "myapi"}]}}'

Conclusion

The specific requirements for Traffic Thresholds and Versioning are complex. Traffic Thresholds, in particular, require two types of limits for request rate limiting, tracking daily sessions and logic for the various request classifications. Using the Kong CDS plugins hides this complexity, and together with Kong Gateway Enterprise’s features, provides a real-world solution to the Australian Consumer Data Standards.

If you are an existing customer and want more information on the CDS plugins, contact your Kong customer experience manager. If you’re not a customer, then reach out to your local Kong Sales Team.